Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes to make permutation-equivariant set autoencoders, by introducing a permutation invariant encoder (based on feature-wise sorting) and at the same time an 'inverse' operation (that undoes the sorting) in the decoder. The achieved equivariance allows the autoencoders to be trained end-2-end without a matching-based loss (like Chamfer) which I think is a very neat result. Also, to address potentially variable-sized sets as inputs/outputs, the authors propose to use a calibrator function that effectively 'samples' the produced features on the set's elements in fixed intervals, that are auto-adaptable in the (variable) set size. 
The results shown are well placed in the current literature and form an important novelty that should have non-trivial impact in the sub-field. Also, the experiments done are well executed, with ample variability in the nature of the datasets used and I expect them to also be easily reproducible (given the authors' provided code).

Some points I would appreciate to see an improvement:
a. The intro is rough (please see minor-specifics at places that you can improve the exposition).
b. The Figure1 is not easy to read. The 90degree rotation results in the same set, but as a 'pictorial' image, obviously not.
c. The "responsibility problem" is already very well explained in the Zhang19a. I would appreciate to tone-down in this paper, the "discovery" of it as a main contribution.
d. Experiments 6.2. it appears that the FSPool/unpool model is better *only* when the mask features are been considered. What are these mask-features? Why do they matter?
e. The way you describe the responsibility problem (discontinuity) is very hand-wavy. It would be nice to explicitly it write it in rigorous math.
f. Why using the relaxation of Grover et al. helped you to avoid the discontinuity that would be otherwise introduced via standard sorting? (I am not familiar with their exact relaxation, but intuitively, their method been a good proxy for sorting, should suffer from it as well). 


Explicit Minor Comments on writing:
(all in introduction)
-4rth line: "this" -> this problem
-"Methods like by" -> Methods like those in 
-"In this paper, we introduce a set pooling method for neural networks that addresses both issues" -> which issues? the encoder's collapse and the decoder's inneficiency in matching? Please explain.
-"good baselines" -> sophisticated/non-trivial baselines

Appendix. Table 9, max-pool at \sigma=0, seems to be the best (please use boldface to indicate it).
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper presents a new approach to representing sets of inputs and a strategy for set auto-encoders. Multiple experiments demonstrate that the new approach performs better than baseline models from recent literature around set representations. The forward encoding strategy is simple enough for a practitioner to implement, and likely to perform better (in terms of training time/gradient flow; if not also test metrics) than existing sum, min, max, mean pooling strategies. It is not substantially more expensive.

The central trick here is to use a per-feature sort across the set as the representation of the set. In cases of arbitrarily sized sets, the authors provide and use a piecewise linear interpolation strategy, and suggest other possibilities (splines, etc), to induce a uniform representation shape regardless of the input set.

The decoder uses a similar trick to expand a latent value back to input-set-size, and then leverages the argsort from the input to re-permute the expanded set. They point out that this helps to avoid discontinuities otherwise caused by the 'responsibility problem', i.e. which feature is responsible to describe which input element[s].

The experiments seem to cover a lot of ground:
- toy polygon dataset demonstrated to be hard for existing sota in set representations
- sets of points from mnist images
- graph classification (competitive with a recent graph convolution approach)
- integrate with Relation Nets for deep set prediction

The authors acknowledge (sec 7) the limitation imposed by requiring the input argsort (and size) at the decoder, but point out that even as a representational pre-training or regularization, this strategy can help to improve set prediction strategies not subject to the same constraint (like RN, as in 6.5).

I found the work to be well presented, the experiments to be strong, and I think it will be interesting to the community. Recommend accepting.

FYI: It seems this work has been previously shared, presumably on arxiv, judging from some amount of back-and-forth citations, building upon Zhang et al 2019.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors point out an interesting problem that the responsibility between target and input is not continuous and propose an FSPool method to alleviate this problem. The method is simple and easy to implement. 

However, there are still some questions the authors don't answer. 
Firstly, the authors point out that this issue also exists in some tasks like object detection, but I have no idea how to apply this method in object detection to fix this issue. 

Secondly, the authors make a comparison between FSPool and sum pool, average pool, and max pool, however, what about the weighted sum pool? I think it's most similar to FSPool. Maybe I misunderstand here. But please respond at my concerns and I'll change the score accordingly. 
review_title: Official Blind Review #1

Meta-review: Overall, this paper got strong scores from the reviewers (2 accepts and 1 weak accept).  The paper proposes to address the responsibility problem, enabling encoding and decoding sets without worrying about permutations.  This is achieved using permutation-equivariant set autoencoders and an 'inverse' operation that undoes the sorting in the decoder.  The reviewers all agreed that the paper makes a meaningful contribution and should be accepted.  Some concerns regarding clarity of exposition were initially raised but were addressed during the rebuttal period.  I recommend that the paper be accepted.
Decision: Accept (Poster)