Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Authors of this paper propose train a model by predicting the size of the symmetric difference between pairs of multisets. With the motivation from fuzzy set theory, both the multiset representation and predicting symmetric difference sizes given these representations are formulated.

In Section 3.3, authors stated that theorem 3.3.1 provides the compelling reason to use symmetric difference over intersection or non-symmetric difference. The statement seems not so straightforward, and how it works as the learning criterion for semi-supervised clustering in the experiments. 

For the relaxation used for the normalization of \phi(a), does this restrict the feasible space of the standard basis vectors? In Section 4.3, authors claimed that in the case of n=3, 98.9% classification accuracy can be obtained by simply picking the maximum valued coordinate of the representation of each object. A systematic comparison in terms of the classification accuracy is important for evaluating the semi-supervised clustering problem. 

In Section 4.2, authors directly model the mean absolute errors in symmetric difference size prediction. It might be more interesting to see what real problems the proposed model can naturally be applied.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a framework for learning representations of multisets.  The approach is built on top of the DeepSets (Zaheer et al., 2017) model, which already is capable of learning representations of multisets, with a new objective of predicting the size of symmetric differences between two multisets, plus a small change in the implementation of the DeepSets model by restricting the per-element module to output a point in the probability simplex.

I liked the background on fuzzy sets and the development of fuzzy multisets and different operations on these multisets.  The definitions and arguments are quite clear and helpful.  One small suggestion for page 4 is that I can understand why the formulation is the only sensible choice for multisets with desired properties, but a claim like this deserves a proper proof.

Model-wise the paper made two contributions for learning representations for multisets as mentioned above: (1) proposed the symmetric difference prediction as a task for learning representations, the argument for this task is that predicting symmetric difference implicitly encourages the model to learn about containment; (2) a slight change in the DeepSets model architecture where the outer rho function is identity and the inner phi function has to output a point in the simplex.

I found these technical contributions to be a bit small.  In addition to this, the paper only presents results on MNIST in a toyish setting, this makes me feel the paper may be more suited for publication in a workshop (idea is interesting, small scale experiments to illustrate the insights, but not complete enough to be published at a conference).

Regarding contribution (1), I can see why predicting symmetric difference makes sense as argued in the paper, but I’m not convinced that this is better than other alternatives.  In order to show that this is a reasonable approach for learning representations, some results that compare this with other possible learning objectives would be necessary.  But I don’t see any such results in this paper.

Regarding contribution (2), I feel the restriction of the phi function to output points in simplex is not very well motivated and confusing in the first read.  Again I can understand why we may want to do this but don’t see why we need to do this.  I’m also concerned that such an architecture may only be good for the task of predicting symmetric difference as it is customized for this task.  Figure 3 shows that an unrestricted model seems to learn better representations despite a worse symmetric difference prediction error, which again confirms the concern.

Another thing about the experiment setup: the second baseline, labeled “DeepSets” in Table 1 actually changed two things compared to the proposed approach: (1) changing the psi function and (2) also changed the symmetric difference function.  It would be good to isolate the contribution of the two.

Overall I feel this paper is not yet ready to be published at ICLR.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new task of learning from sets, predicting the size of the symmetric difference between multisets, and gives a new method to solve the task based on the fuzzy set theory.
Although the topic of learning from sets is relevant and using the fuzzy set theory for the task is interesting, I have the following concerns regarding with the clarity, significance, and evaluation.

- Motivation is not clearly presented. The new task of predicting the size of the symmetric difference between multisets is proposed, while its application is not well discussed.
    Although Theorem 1 characterizes the task using the subset inclusion relationship, its relevance to applications is still not clear.
- The problem to be solved is not mathematically formulated. In particular, what are input and output?
- More detailed explanation of the data preparation would be required.
    How to transform images to pairs of multisets?
    Is the label (number) of each is used as an element of a multiset?
- For the second comparison partner, why is \Delta(A, B) defined as \rho_2(\Psi(A) + \Psi(B))?
    For fair comparison, this function should be the same with the proposed method, that is, ||\Psi(A) - \Psi(B)||_1 for the learned \Psi by DeepSets.
- In experiments, one of the most straightforward ways is to first predict labels for each image, followed by computing the symmetric difference from the predicted labels. Comparison with such baseline should be performed.

Minor comments:
- P.1, L.1 of the second paragraph: "The the" -> "The"

review_title: Official Blind Review #3

Meta-review: While the reviewers appreciated the problem to learn a multiset representation, two reviewers found the technical contribution to be minor, as well as limited experiments. The rebuttal and revision addressed concerns about the motivation of the approach, but the experimental issues remain. The paper would likely substantially improve with additional experiments.
Decision: Reject