Title: Deep Learning with Sets and Point Clouds
Authors: Siamak Ravanbakhsh
Jeff Schneider
Barnabas Poczos
Venue: ICLR 2017
Keywords: Deep learning
Structured prediction
Computer vision
Supervised Learning
Semi-Supervised Learning

Abstract: We introduce a simple permutation equivariant layer for deep learning with set structure. This type of layer, obtained by parameter-sharing, has a simple implementation and linear-time complexity in the size of each set. We use deep permutation-invariant networks to perform point-could classification and MNIST digit summation, where in both cases the output is invariant to permutations of the input. In a semi-supervised setting, where the goal is make predictions for each instance within a set, we demonstrate the usefulness of this type of layer in set-outlier detection as well as semi-supervised learning with clustering side-information.

TL;DR: Parameter-sharing for permutation-equivariance and invariance with applications to point-cloud classification.

Review discussion:
If I understand correctly, one issue with the invariance to permutation of the graph is that one can loose discriminative information (example 4.3). Might it be possible to recover the loss of information due to the averaging?
I understand why one needs a set structure for the set anomaly detection, but not for the point cloud detection.
Indeed, in the anomaly detection case, the objective of the problem is to seprate the set E of faces into E\{x} and {x} where x is the anomaly, and E\{x} is invariant by permutations.
But, in the point cloud detection, I would have expected the final objective to be invariant by any permutations of the points. I guess I am doing a logical error, could you correct me? Is it to avoid discriminability issues? Why did you chose this graph? In this case, can we interpret your model as a graph convolution, with the edges at x being {(x,x'),x'\neq x}?
If the relational dependencies are not given, could they be learned?
Thanks.
When the invariance is not "minimal" (as defined in section 4), one could lose discriminative information. The only place this lack of minimal invariance comes up in our paper is as an easy way to handle the product structure. Example 4.3 gives an extreme scenario to see the possible issues with using this trick with product structures. This inadvertent loss of information does not happen with correct (but possibly cumbersome) way of handling a product structure using minimally invariant layer (of eq. 1).
This means the set-invariat layer (as well-as other structured discussed, such as graph-convolution) is minimally invariant -- e.g.,
the output of point-cloud classification is indeed invariant to the permutation of points. You are correct in relating the set structure to graph structure.
However, note that our proposed set-layer, eq(4), is different from what you get from graph-convolution in using the max operation and its linear (rather than quadratic) complexity.
Thank you for your answers, please read carefully my final questions as I will use them for my final review.
# When the invariance is not "minimal" (as defined in section 4), one could lose discriminative information. The only place this lack of minimal invariance comes up in our paper is as an easy way to handle the product structure. Example 4.3 gives an extreme scenario to see the possible issues with using this trick with product structures. This inadvertent loss of information does not happen with correct (but possibly cumbersome) way of handling a product structure using minimally invariant layer (of eq. 1).
>>>>> Product structures are extremely important since they permit to combine progressively more complex invariants. My understanding of this invariance is that you are either dealing between two extreme cases: a fully connected layer(null set) or a global average pooling(all set). In the linear case, the agregation(the structure S) of the elements correspond to local pooling of groups of nodes(the x_i). It is totally possible to go over this issue with techniques such as lifting scheme ( https://en.wikipedia.org/wiki/Generalized_lifting ) where you could recover the loss of information due to an average of two nodes, via for instance the difference between those two nodes. Do you think this might make sens in your framework? I do not understand how it is possible to recover the loss of information of a projection otherwise, even with a non-trivial minimal invariance.
# This means the set-invariat layer (as well-as other structured discussed, such as graph-convolution) is minimally invariant -- e.g., the output of point-cloud classification is indeed invariant to the permutation of points. You are correct in relating the set structure to graph structure.
However, note that our proposed set-layer, eq(4), is different from what you get from graph-convolution in using the max operation and its linear (rather than quadratic) complexity.
>>>>> If I understand correctly, you have a better implementation and you apply it on a new dataset, which is a pros of your work.
However, the properties you have introduced are not fully exploited and the set data-structures are very specific, and one can wonder why introducing them: indeed, a reader could start its reading at the Section 5 and its understanding of the numerical experiments should be OK. I believe you should apply your theoretical results on others objects than set data-structures. Obviously, one of the difficulty would be to build the structure S. Do you have some suggestions to obtain(learn?) it, or do you know different structures and datasets that could fit your paper?
Thank you!
Thank you for your follow up!
The product structure is indeed central in our approach to handling of structure. According to our partial ordering of structures, they appear between two extreme cases of “all” structure --that requires using fully connected layer-- and “null” structure of the mini-batch. Note that the set-structure (handled by set-invariant layer) is not at either extremes of this partial order.
Now, to handle the product of any structures (not limited to “null”, “all” or “set” structures), one could obtain the product (as defined on page 3) and proceed by defining the parameter-sharing of the layers according to eq(1). A faster “heuristic” is to handle one structure at a time, which as our extreme example 4.3 shows could lead to trouble.
Let me clarify where and why we use this heuristic:
We present three sets of experiments on 1) point-cloud-classification; 2) face outlier detection and; 3) semi-supervised learning with sets. The product-structures in experiments (1) and (3) are the product of “null”, “set” and “all” structures, which only require using the set-invariant layer of section 5.1. Here, note that multiple input channels correspond to the “all” structure in the product structure.
However, in the face outlier detection experiment (2), we also have the 2D grid structure that requires using the parameter-sharing of the 2D-convolution layer. Here the complete product structure is:
S = null \times set \times all \times 2D-Grid
Where “null” structure reflects the mini-batch, set structure is due to having a set of 16 face images, “all” structure is due to multiple input channels (e.g. R, G, B) and 2D-grid reflects the structure of 2D image. To correctly handle this product structure, we need to use the parameter-sharing implied by the resulting product.
However, for handling the product of 2D-grid and set structures, we could not use the efficient implementation of convolution layer.
That is why we use the heuristic of section 4.1 for handling one structure at a time: first we define multiple convolution-pooling layers to capture the spatial/grid structure and then use the set-invariant layer of section 4.1 to handle the product of “null”, “all” and “set” structures (details are in Appendix D.1.). As our result for face outlier detection indicates, handling one structure at a time, works well here.
Here, while one could use the heuristic of section 4.1 followed by some other technique (as you suggest) to reduce the “information loss”, for best results one might as well avoid the heuristic of section 4.1 altogether.
Although we repeatedly acknowledge the fact that our experiments are limited to the set structure, our framework is more general; demonstrated by deriving other types of layers as special cases. We believe this alternative approach to structure based on parameter-sharing is useful on its own -- for example our derivation of graph convolution is much simpler and gives complementary view to current derivation. We are currently investigating applications of this framework for more complex structures such as nested sets and graph of graphs for multi-resolution handling of point-clouds and graphs. We agree with you that it would be very useful to be able to automatically infer the structure from data, however we have no idea for an efficient solution at the moment.
I hope this addresses your questions and comments.
Thank you very much for your detailed explanation.
One more comment: your framework handle 2D convolutions, but I think mentionning it can be slightly confusig in the following sens. Your framework permits to build invariance with respect to permutations to the relation. A linear averaging builds an invariant to translation, and permutations.
However, from your framework, one can not derive the fact that the layers need to be covariant with the action of translation to build this invariance. Here, (please correct me if I'm wrong), the invariance which is obtained is different: a convolution at a position i of an image x can be written as a dot product with a filter f, such that the resulting filtered image is result[i]=sum_j x[i+j]f[j] In your frame work, the structure is given by the neighbors(they are as uch relations as the number of elements in the support of the filters), f is the weights, and the invariance here means that the sum over j is commutative, which is not much informative.
Thanks for your follow up!
In the case of convolution, "permutation over relations" does not result in the final translation equivariance.
That is the commutative summation plays no role here. What does the trick for 2D-grid is the fact that individual functions f^{i,j}(x_{S}) defined on the neighborhood of (i,j) pixel, share their parameters -- i.e. f^{i',j'} = f^{i,j} = f_{theta} for all 0<i,j<N associated with pixels; see figure 2(a)-left.
In general there are two factors at play in producing invariances of a structure: 1) invariance to permutation over relations in the structure S; 2) parameter-sharing across functions f(x_S) for x \in \mathcal{X}, also encoded in the structure S.
This review is only an informed guess - unfortunately I cannot assess the paper due to my lack of understanding of the paper.
I have spent several hours trying to read this paper - but it has not been possible for me to follow - partially due to my own limitations, but also I think due to an overly abstract level of presentation. The paper is clearly written, but in the same way that a N. Bourbaki book is clearly written.
I would prefer to leave the accept/reject decision to the other reviewers who may have a better understanding - even if the authors had made a serious mistake, I would not be able to tell. My proposal is positive because the paper is apparently clearly written and the empirical evaluation is quite promising.
But some effort will be needed in order to address the broader audience that could potentially be interested in the topic.
I therefore would like to provide feedback only at the level of presentation.
My main source of problems is that the authors do not try to ground their abstract formalism with concrete examples;
when the examples show up it is by "revelation" rather than by explaining how they connect to the previous concepts.
The one example that could unlock most people's understanding is how convolution, or inner product operations connect with the setting described here. For what I know convolution is tied with space (or time) and is understood as an equivariant operation - shifting the signal shifts the output.
It is not explained how the '(x, x')' pairs used by the authors in order to build relations, structures and then to define invariance relate to this setting.
Going from sets, to relations, to functions, to operators, and then to shift-invariant operators (convolutions) involves many steps, and some hand-holding is needed.
Why is the 3x3 convolution associated to 9 relations?
Are these relations referring to the input at a given coordinate and its contribution to the output? (w_{offset} x_{i-offset})? In that case, why is there a backward arrow from the center node to the other nodes? And why are there arrows across nodes?
What is a Cardinal and what is a Cartesian convolution in signal processing terms? (clearly these are not standard terms).
Are we talking about separable filters?
What are the X and Square symbols in Figure 2? And what are the horizontal and vertical sub-graphs standing for? What is x_1 and what is x_{11},x_{1,2},x_{1,3} and what is the relationship between them?
I realize that to the authors these questions may seem to be trivial and left as  homework for the reader.
But I think part of publishing a paper is doing a big part of the homework for the readers so that it becomes easy to get the idea.
Clearly the authors target the more general case - but spending some time to explain how the particular case is an instance of the the general case would be a good use of space.
I would propose that the authors explain what are  x, x_{I}, and x_{S} for the simplest possible example, e.g. convolving a 1x5 signal with a 1x3 filter, how the convolution filter parameters show up in the function f, as well as how the spatial invariance (or, equivariance) of convolution is reflected here.
Thank you very much for your feedback! We appreciate the time you have spent on our paper and based on your comments we have extensively rewritten the paper. We have removed all the abstract and general treatments of invariance and focused solely on the case of sets and point-clouds. We have included more experimental results and baseline evaluations as well. Any further feedback is appreciated.
This paper discusses ways to enforce invariance in neural networks using weight sharing.  The authors formalize a way for feature functions to be invariant to a collection of relations and the main invariance studied is a “set-invariant” function, which is used in an anomaly detection setting and a point cloud classification problem.
“Invariance” is, at a high level, an important issue of course, since we don’t want to spend parameters to model spurious ordering relationships, which may potentially be quite wasteful and I like the formalization of invariance presented in this paper.
However, there are a few weaknesses that I feel prevent this from being a strong submission.
First, the exposition is too abstract and this paper could really use a running and *concrete* example starting from the very beginning.
Second, “set invariance”, which is the main type of invariance studied in the paper is defined via the author’s formalization of invariance, but is never explicitly related to what I might think of as “set invariance” — e.g. to permutations of input or output dimensions.  Explicitly defining set invariance in some other way, then relating it to the  “structural invariance” formulation may be a better way to explain things.  It is never made clear, for example, why Figure 1(b) is *the* set data-structure.
I like the discussion of compositionality of structures (one question I have here is: are the resulting compositional structures are still valid as structures?).
But the authors have ignored the other kind of compositionality that is important to neural networks — specifically that relating the proposed notion of invariance to function composition seems important — i.e. under what conditions do compositions of invariant functions remain invariant?  And  It is clear to me that just by having one layer of invariance in a network doesn’t make the entire network invariant, for example.
So if we look at the anomaly detection network at the end for example, is it clear that the final predictor is “set invariant” in some sense?
Regarding experiments, there are no baselines presented for anomaly detection.  Baselines *are* presented in the point cloud classification problem, but the results of the proposed model are not the best, and this should be addressed.  (I should say that I don’t know enough about the dataset to say whether these are exactly fair comparisons or not).  It is also never really made clear why set invariance is a desirable property for a point cloud classification setting.  As a suggestion: try a network that uses a fully connected layer at the end, but uses data augmentation to enforce set invariance.
Also, what about classical set kernels?
Other random things:
* Example 2.2: Shouldn’t |S|=5 in the case of left-right and up-down symmetry?
* “Parameters shared within a relation” is vague and undefined.
* Why is “set convolution” called “set convolution” in the appendix?  What is convolutional about it?
* Is there a relationship to symmetric function theory?
Thank you very much for your feedback! Based on your comments, we have extensively revised the paper.
In particular, we have removed the general treatment of invariances from this paper in favour of clarity and focused solely on the case of permutation-invariance/equivariance for set structure.
Regarding the choice of set-layer, a new theorem in our paper shows that parameter-sharing of our set-layer is the only way of achieving permutation-equivariance in “standard” neural network layers.
WRT baselines: We have added new experiments as well as more baseline results for “all” settings to address your concern.
WRT composition: in the revised paper, we show that functional composition preserves permutation-equivariance.
WRT connections to symmetric function theory: the permutation-invariant function in our case is indeed a so-called “symmetric function”. We have added citations for this.
However, symmetric function theory is concerned with polynomial forms while we study symmetric functions in the form of abstract neurons.
Pros :
- New and clear formalism for invariance on signals with known structure
- Good numerical results
Cons :
- The structure must be specified.
- The set structure dataset is too simple
- There is a gap between the large (and sometimes complex) theory introduced and the numerical experiments ;
consequently a new reader could be lost since examples might be missing
Besides, from a personal point of view, I think the topic of the paper and its content could be suitable for a big conference as the author improves its content.
Thus, if rejected, I think you should not consider the workshop option for your paper if you wish to publish it later in a conference, because big conferences might consider the workshop papers of ICLR as publications. (that's an issue I had to deal with at some points)
Thanks for your feedback!
To remove the gap between our general formalism and the experimental treatment that was limited to sets and point-clouds, we have extensively revised the paper, solely focusing on the application of deep permutation-equivariant/invariant models. We have added new experimental results and clarified the formalism for sets.
Hi,
Could you add some baseline accuracies to your redshift prediction dataset? The improvement is demonstrated, however I do not know well those problems and I wish you could add some numbers to see how good you perform.(as you did with mnist)
One more comment (and please let me know if I am doing a mistake). In the way you wrote this new version, I see some connexions between your work and a haar transform(or lifting scheme) on graphs.
For example this paper: https://arxiv.org/pdf/1406.2390v2.pdf . Basically, they can learn a graph associated to the data and find out an appropriate representation for classification, yet in an unsupervised setting. And, as far as I understand, they build as well an invariant representation to permutations. (the fact they use filters with support 2^j is not a limitation, as far as I understand) However, they are still discriminative because they can recover some missing information lost after the averaging. As far as I understand your equation (2) in a convolutioonal setting, you combine a dirac and an averaging (say A). In some sens, you are at the first step to build an isometric operator(eg {A,I-A}). Interestingly, you set up this in a supervised setting.
This being said, I find the applications interesting.
Thanks for your comments! We have added a paragraph explaining the relation of Chen et al.’14 to our work. Scattering features are permutation invariant, however, they cannot be used for semi-surpervised learning where we need “equivariance”. Their other requirement seems to be an underlying graph structure to guide the partitioning of the nodes.
Also note that we believe the scrambled MNIST results in their paper is produced by permuting the pixels with an “identical” permutation matrix for all instances and their result on this dataset is not permutation-invariant by our definition. They only intend to show that they are able to find the underlying pixel neighborhood and leverage it in supervised learning (We intended to compare our method against their method on this task before we noticed this issue.)
We have also added another baseline for the red-shift improvement experiments from the original redMaPPer catalog, per your request.
I slightly disagree. Basically, they do not know the graphs, and they learn it.
However, their representation is invariant w.r.t. a large family of permutations that is defined by the graph. (for example, if two nodes are paired, the representation invariant by this permutation) The framework they present is more general, in the sens that given a permutation that has been averaged, they recover the missing information by using the corresponding wavelets. The construction of such wavelets is very well introduced in this paper: https://pdfs.semanticscholar.org/d6b6/3e12c44129b0dc06bb242ab1777c152cff89.pdf
At the end, your two representations want to be invariants to several class of permutations. They showed it is possible to recover the grid, but as well building a representation that is discriminative without the knowledge of it.
However, I totally agree that they did it in an unsupervised setting.
It would be interesting, to use their techniques to discover graphs with your techniques to design the filters. Anyway, that's more an open suggestion than a requirement.
Thanks again for your answers and the numbers.
We thank all reviewers for their comments! A commonl issue in reviews was regarding the disconnect between our general treatment of invariance and experimental results. The other major issue was regarding the clarity of the first part.
To resolve both issues we have removed our general treatment and focused on "deep learning with sets and point-clouds", incorporating all reviewer feedback on this part. We welcome any further comments on the revised version.

Meta-review:
role: PC
data: comment: This paper studies neural models that can be applied to set-structured inputs and thus require permutation invariance or equivariance. After a first section that introduces necessary and sufficient conditions for permutation invariance/equivariance, the authors present experiments in supervised and semi-supervised learning on point-cloud data as well as cosmology data.
 
 The reviewers agreed that this is a very promising line of work and acknowledged the effort of the authors to improve their paper after the initial discussion phase. However, they also agree that the work appears to be missing more convincing numerical experiments and insights on the choice of neural architectures in the class of permutation-covariant. 
 
 In light of these reviews, the AC invites their work to the workshop track. 
 Also, I would like to emphasize an aspect of this work that I think should be addressed in the subsequent revision.
 
 As the authors rightfully show (thm 2.1), permutation equivariance puts very strong constraints in the class of 1-layer networks. This theorem, while rigorous, reflects a simple algebraic property of matrices that commute with permutation matrices. It is therefore not very surprising, and the resulting architecture relatively obvious. So much so that it already exists in the literature. In fact, it is a particular instance of the graph neural network model of Scarselli et al. '09 (http://ieeexplore.ieee.org/abstract/document/4700287/) when you consider a complete graph, which has been used in the setup of full set equivariance for example in 'Learning Multiagent communication with backpropagation', Sukhbaatar et al NIPS'16; see also 'Order Matters: sequence to sequence for sets', Vinyals et al. https://arxiv.org/abs/1511.06391. 
 The general question of how to model point-cloud data, or more generally data defined over graphs, with neural networks is progressing rapidly; see for example https://arxiv.org/abs/1611.08097 for a recent survey.
 
 The question then is what is the contribution of the present work relative to this line of work. The authors should answer this question explicitly in the revised manuscript, either with a new application of the model, or with theory that advances our understanding of these models, or with new numerical applications.

Decision: Invite to Workshop Track