Title: Submodular Sum-product Networks for Scene Understanding
Authors: Abram L. Friesen
Pedro Domingos
Venue: ICLR 2017
Keywords: Computer vision
Structured prediction

Abstract: Sum-product networks (SPNs) are an expressive class of deep probabilistic models in which inference takes time linear in their size, enabling them to be learned effectively. However, for certain challenging problems, such as scene understanding, the corresponding SPN has exponential size and is thus intractable. In this work, we introduce submodular sum-product networks (SSPNs), an extension of SPNs in which sum-node weights are defined by a submodular energy function. SSPNs combine the expressivity and depth of SPNs with the ability to efficiently compute the MAP state of a combinatorial number of labelings afforded by submodular energies. SSPNs for scene understanding can be understood as representing all possible parses of an image over arbitrary region shapes with respect to an image grammar. Despite this complexity, we develop an efficient and convergent algorithm based on graph cuts for computing the (approximate) MAP state of an SSPN, greatly increasing the expressivity of the SPN model class. Empirically, we show exponential improvements in parsing time compared to traditional inference algorithms such as alpha-expansion and belief propagation, while returning comparable minima.


TL;DR: A novel extension of sum-product networks that incorporates submodular Markov random fields into the sum nodes, resulting in a highly expressive class of models in which efficient inference is still possible.

Review discussion:
InferSSPN looks to be a good algorithm for solving MAP inference problems in Submodular Sum-Product Networks (SSPN).
However, I'm missing the experimental justification for why SSPN is a good model. This leads to some questions:
- Why evaluate on training data?
- How do the energy values achieved by the various algorithms correlate with segmentation quality?
- If the pairwise terms were dropped from SSPNs, would this simplify to some existing model? How much modelling benefit comes from the pairwise terms?
- Similarly, if the grammar were dropped from the model to get a standard MRF, how would the segmentation quality compare? I.e., how much benefit comes from using the grammar?
Also,
- Can you give a ballpark number for how expensive running inference would be relative to ConvNet operations if one were trying to jointly learn SSPN and ConvNet parameters?
Thanks for your questions, I've responded to them below.
1) Why evaluate on training data?
This paper is about representation and inference; learning is future work.
As such, we are only evaluating inference performance and not generalization performance. Evaluating inference performance on test data would confuse inference performance with generalization performance, so we evaluated on the training data to ensure the two measures are kept separate.
2) How do the energy values achieved by the various algorithms correlate with segmentation quality?
Mean pixel accuracies are in the high 80s to mid 90s on the training data, but we did not report them because they’re not comparable to previous results reported in the literature (due to evaluation on a subset of the training data). Here we’re just evaluating our model and inference algorithm.
However, we could include a figure or brief discussion demonstrating that the resulting segmentations are good and that lower energy correlates with improved accuracy.
3) If the pairwise terms were dropped from SSPNs, would this simplify to some existing model? How much modelling benefit comes from the pairwise terms?
Dropping the pairwise terms would result in a model that independently parsed each pixel by simply choosing the lowest energy path in the grammar for that pixel (where E(path) = the sum of production costs along the path plus the unary term at the terminal), which would simply correspond to a structured form for the unaries that does not provide any smoothing or label-space structure between pixels. The reason for the pairwise terms is that a regular SPN (defined from the productions of a grammar) would need to explicitly consider an exponential number of splits for each production, which is computationally infeasible and thus untestable. Incorporating (submodular) MRFs into the SPN allows us to perform inference in an otherwise highly-intractable model.
4) Similarly, if the grammar were dropped from the model to get a standard MRF, how would the segmentation quality compare? I.e., how much benefit comes from using the grammar?
In general, the reason to use an SSPN (or a grammar in natural language processing) is that it simultaneously segments different regions and objects at multiple levels of abstraction and, in particular, considers multiple ways in which a region can be realized via its productions in the grammar.
In addition, this top-down information can actually improve the bottom-level segmentation, e.g., segmenting a roof from a wall may be more accurate knowing that they’re both part of a house, in contrast with an MRF, which only considers a single production and a single level of abstraction.
We also note that because the flat model is a special case of an SSPN, we can always include a production in the grammar corresponding to the flat MRF, which will be chosen if it has the lowest energy, so the SSPN will never produce worse segmentations. For the experiments in the paper, the accuracies of the bottom-level segmentations are essentially identical to those resulting from segmenting the image with a flat MRF, as one would expect.
5) Can you give a ballpark number for how expensive running inference would be relative to ConvNet operations if one were trying to jointly learn SSPN and ConvNet parameters?
Yes, although we first note that our code is not highly-optimized and has not been extended to run on the GPU, so these numbers can definitely be improved.
Further, there are many techniques (such as warm-starting) that can be used to speed up inference during learning.
Nonetheless, in the experiments in the paper, total SSPN inference time ranged from less than 1 second to a maximum 287 seconds;
however, for the depth experiments (which were the most time consuming), only 5/60 of the runs took over 200 seconds.
Thus, inference is in general fairly fast, albeit slower than with just a flat MRF, but only by a constant factor (|G|) per iteration.
Please let us know if you have any further questions.
This paper develops Submodular Sum Product Networks (SSPNs) and
an efficient inference algorithm for approximately computing the
most probable labeling of variables in the model. The main
application in the paper is on scene parsing. In this context,
SSPNs define an energy function with a grammar component for
representing a hierarchy of labels and an MRF for encoding
smoothness of labels over space. To perform inference, the
authors develop a move-making algorithm, somewhat in the spirit
of fusion moves (Lempitsky et al., 2010) that repeatedly improves
a solution by considering a large neighborhood of alternative segmentations
and solving an optimization problem to choose the best neighbor.
Empirical results show that the proposed algorithm achieves better
energy that belief propagation of alpha expansion and is much faster.
This is generally a well-executed paper. The model is interesting
and clearly defined, the algorithm is well presented with proper
analysis of the relevant runtimes and guarantees on the
behavior.
Overall, the algorithm seems effective at minimizing
the energy of SSPN models.
Having said that, I don't think this paper is a great fit for
ICLR. The model is even somewhat to the antithesis of the idea of
learning representations, in that a highly structured form of
energy function is asserted by the human modeller, and then
inference is performed. I don't see the connection to learning
representations. One additional issue is that while the proposed
algorithm is faster than alternatives, the times are still on the
order of 1-287 seconds per image, which means that the
applicability of this method (as is) to something like training
ConvNets is limited.
Finally, there is no attempt to argue that the model produces
better segmentations than alternative models. The only
evaluations in the paper are on energy values achieved and on
training data.
So overall I think this is a good paper that should be published
at a good machine learning conference, but I don't think ICLR is
the right fit.
Thank you for your detailed comments and feedback. We disagree that SSPNs are antithetical to ICLR.
While our results do not currently include learning, SSPNs are absolutely learnable, just as SPNs, MRFs, and CRFs, and convolutional neural networks are learnable.
While structure learning is possible for most of these models, the vast majority of learning (in ICLR and in other conferences) is parameter learning, meaning that the aforementioned models all have equivalent amounts of human-specified structure. The only difference between SSPNs and these other models (ignoring SPNs) is that SSPNs can be (but do not need to be) defined with respect to a grammar, as a convenient method for specifying the architecture of the network and to provide them with additional semantic information such as constituency and sub-categorization relationships.
With respect to inference time, many convolutional neural network architectures for semantic segmentation end up down-sampling images quite extensively (e.g., Deeplab, FCN) due to the use of max-pooling or strided convolution. In our preliminary work on end-to-end training of SSPNs with deep networks, the input image to the SSPN ends up being much smaller than the full image and thus inference only takes a small fraction of a second.
As such, inference time during training is similar to that of a forward pass through resnet101, making end-to-end training quite realistic.
See our general response to this point above.
This paper is about submodular sum product networks applied to scene understanding. SPNs have shown great success in deep linear models since the work of Poon 2011.  The authors propose an extension to the initial SPNs model to be submodular, introducing submodular unary and pairwise potentials.  The authors propose a new inference algorithm. The authors evaluated their results on Stanford Background Dataset and compared against multiple baselines.
Pros:
+ New formulation of SPNs
+ New inference algorithm
Cons:
- The authors did not discuss how the SSPN structure is learned and how the generative process chooses the a symbol (operation) at each level)
- The evaluations is lacking. The authors only showed results on their own approach and baselines, leaving out every other approach. Evaluations could have been also done on BSD for regular image segmentation (hierarchical segmentation).
The idea is great, however, the paper needs more work to be published.  I would also recommend for the authors to include more details about their approach and present a full paper with extended experiments and full learning approach.
Thank you for your review.
1. We do not do any learning in this paper, which is why it is not discussed. The grammar structures used in the experiments are algorithmically defined and the corresponding weights are generated heuristically by counting symbol appearances in the evaluated images. The same structures and parameters are used when evaluating each inference algorithm.
The generative process does not select symbols, instead it always begins at the start symbol over the entire image, chooses a production of the start symbol (from a categorical distribution over said productions – the paper has been updated to make this more explicit), chooses a partition of the current region into a labeling based on the constituent symbols of the chosen production (by sampling from the MRF defined by this production over the pixels in the current region), and then recurses on each (constituent, subregion) pair. The production-selection process is equivalent to that found in PCFGs.
2. See our general response to this point above. It’s not clear to us how evaluating on the Berkeley segmentation dataset (BSD) would be meaningful, as our model and inference algorithm are about identifying semantic classes and relationships such as part-subpart and class-subclass, not about identifying image contours or segmenting based on intensity values.
The paper discusses sub modular sum-product networks as a tractable extension for classical sum-product networks. The proposed approach is evaluated on semantic segmentation tasks and some early promising results are provided.
Summary:
I think the paper presents a compelling technique for hierarchical reasoning in MRFs but the experimental results are not yet convincing.
Moreover the writing is confusing at times. See below for details.
Quality: I think some of the techniques could be described more carefully to better convey the intuition.
Clarity: Some of the derivations and intuitions could be explained in more detail.
Originality: The suggested idea is great.
Significance: Since the experimental setup is somewhat limited according to my opinion, significance is hard to judge at this point in time.
Detailed comments:
1. I think the clarity of the paper would benefit significantly from fixes to inaccuracies. E.g., \alpha-expansion and belief propagation are not `scene-understanding algorithms’ but rather approaches for optimizing energy functions. Computing the MAP state of an SSPN in time sub-linear in the network size seems counterintuitive because it means we are not allowed to visit all the nodes in the network. The term `deep probabilistic model’ should probably be defined. The paper states that InferSSPN computes `the approximate MAP state of the SSPN (equivalently, the optimal parse of the image)’ and I’m wondering how the `approximate MAP state' can be optimal. Etc.
2.
Albeit being formulated for scene understanding tasks, no experiments demonstrate the obtained results of the proposed technique. To assess the applicability of the proposed approach a more detailed analysis is required. More specifically, the technique is evaluated on a subset of images which makes comparison to any other approach impossible. According to my opinion, either a conclusive experimental evaluation using, e.g., IoU metric should be given in the paper, or a comparison to publicly available results is possible.
3. To simplify the understanding of the paper a more intuitive high-level description is desirable. Maybe the authors can even provide an intuitive visualization of their approach.
Thanks for your comments and feedback.
1. We’ve edited and re-written parts of the paper to fix inaccuracies.
For example, with respect to alpha-expansion and BP as scene-understanding algorithms, we simply meant that they are used for this task, not that they are explicitly designed for it. This has been clarified, as has the language regarding the approximate MAP state (where the intention was that the MAP state is the optimal parse, but this was poorly worded). We note, however, that inference in time sub-linear in network size is not uncommon, depending on the definition of the network. Techniques such as A* and branch-and-bound are able to explore graphs and networks in time sub-linear in their size.
2. See our general response to this point above.
3. In editing the paper, we’ve re-structured and re-written parts of the inference section to try to make the intuitions more clear and have included a figure to help visualize the key parts of InferSSPN.
The main concern raised in each review is the lack of a direct comparison of SSPNs to existing semantic segmentation benchmarks. We address this point here and then respond to each reviewer's other points directly. We’ve also made significant revisions to the structure and writing of the paper, particularly in the inference section, which we hope more clearly explains the intuitions behind InferSSPN.
Despite not having comparisons on semantic segmentation benchmarks, SSPNs are a significant step forward in the state-of-the-art on scene understanding. SSPNs are (provably) more expressive than (and contain as a special case) MRFs and SPNs as commonly used for scene understanding and semantic segmentation, meaning that SSPNs are a richer model class than those already in use.
Despite this increase in expressivity and complexity, InferSSPN is a very efficient (approximate) inference algorithm for the problem of parsing an image with respect to a grammar, which has been addressed by many previous works, but their approaches have all been very restrictive or inefficient. InferSSPN provably achieves low-order-polynomial time complexity for a problem with a state space that has size exponential in both the number of pixels and the height of the grammar.
While the first version of the paper did not include accuracy results, as we thought the energy results were sufficient, we’ve since updated the paper to include all accuracy, energy, and time complexity results for all test cases. These show empirically that InferSSPN provides exponential improvements in time-complexity with no loss in accuracy relative to alpha-expansion, which provably returns local optima that are within a constant factor of the global optimum.
Further, while we have characterized SSPNs in terms of scene understanding, they are as broadly applicable as SPNs and MRFs, and scene understanding is just the initial application we chose to focus on. Future applications may include activity recognition or social network modeling.
Nonetheless, this paper presents a powerful new representation for efficiently modeling and reasoning about scenes at multiple levels of abstraction and is well within the scope of ICLR, which includes “hierarchical models” and “applications to vision” in its list of relevant topics.
Learning SSPNs is important – both for evaluating SSPNs on existing semantic segmentation benchmarks and for their adoption in general – and is something that we are already working on. Unfortunately, there do not yet exist any image grammars or relevant datasets with which to train SSPNs, so learning requires either creating such datasets or learning with a large number of hidden variables. This is ongoing and future work.
However, we believe that one benefit of our contributions will be to spur the creation of such datasets and open up new research problems where reasoning about multiple levels of abstraction in scenes and higher-level relationships between objects is crucial.
Finally, it’s common in AI and ML for new representations to be introduced well before learning algorithms are properly developed for them. Notable cases include graphical models and previous work on image grammars, and this paper does the same for SSPNs.

Meta-review:
role: PC
data: comment: This paper was reviewed by three experts. While they all find merits in the paper (interesting new model class SSPN, new MAP inference algorithm), they all consistently point to deficiencies in the current manuscript (lack of parameter learning, emphasis on evaluation on energies, lack of improvements in accuracy). 
 
 One problem that (I believe) is that manuscript as it stands makes neither a compelling impact on the chosen application (semantic segmentation) nor does it convincing establish the broad applicability of the proposed model (how do I run SSPNs on activity recognition or social network modeling). 
 
 To be clear, we all agree that there is promising content here. However, I agree with the reviewers that the significance has not been established.

Decision: Reject