Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper describes a way to train functions that are able to represent the union of classes as well as to query if the classes in an image subsume the classes in another image. This is done throughly jointly training embedding functions, a set union function and a query function. The paper reads well.

While the approach is reasonable, the experiments seem to be quite incomplete and no explanation is given why a trivial solution cannot be used instead of the learnt functions.

The paper argues for learning a set union function however much of the evaluation focuses on quite small sets of 2 or 3 items. On the evaluation that utilises larger sets, e.g. COCO, there isn't any analysis of how performance of the technique scales with the size of the set since that would be one of the defining characteristics of a set union function. The COCO experiment is also lacking in detail, for example, how many items are there in the positive and negative sets and how the test set is balanced. Finally, it seems that f, g and h could be trivial non-learnt functions. For example, f could be a function that maps an image to a binary representation of its classes (this could be a typical ResNet image classifier), g could be a function that does a binary OR of its two arguments and h could be a function that uses a binary AND and equality test on its two arguments. In this case, g and h don't need to be learnt at all. This may not be possible in the COCO experiment where the individual labels are not known but it seems quite unrealistic to have a dataset where only pairwise subset relationships are known.

It also seems that the f is always different between that used with g and that used with h, is this the case? SimRef also doesn't do data augmentation but there's no explanation why it is done for the proposed method and not for this baseline. The MF baseline in experiment 1 seems to be a straw man especially since the baselines in experiment 2 are much stronger.

================================================================================
Update after rebuttal:

Thanks for answering my questions and performing the additional experiments with a ResNet baseline and performing an additional analysis based on the number of subclasses in figure 5. I think these provide a substantially better analysis of the algorithm so I've increased my score correspondingly. For the final paper, I think it would be good to add TradEmb/ResNet to figure 5 as well to understand how those methods scale worse/better with the number of subclasses.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose a joint/compositional embedding procedure where a single instance can be mapped/embedded to multiple classes while preserving the class-specific information in the embedded representations. The authors look at class union and class query criteria for the composite embeddings. The proposed approach is evaluated appropriately. There are several issues with the work. 

Does the proposal mean each embedding eventually corresponds to multiple classes/subclasses ie., one can learn something on-trivial about each class from these embeddings that is different from class-specific embedding? How do you avoid the trivial solution problem here i.e., the embeddings are going to be average of the class-specific embeddings --- as we see in the evaluations this is in fact happening (figure 1b)? Also, is this behaviour desired i.e., tending towards mean? 

And continuing along these lines, a clear choice of baseline for the proposal is to choose mean embeddings i.e., men of independent embeddings? Or is this not appropriate? Why is ML the best baseline? We can use the probability map (the input to final softmax) instead as the embedding as well correct? 

"... x_a containing objects in another image " -- this statement is not making sense, is it objects in x_a also present in another image x_b?

It is rather difficult to interpret the usefulness of g(.) when it is a nonlinear model like neural network. Simpler models (like Symm(a,b,.) i.e., just the first layer of what is being used now) should be evaluated instead to get better understanding of what is going on! 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
=======
This paper proposes compositional embeddings i.e. embeddings that can be used to infer multiple classes from the data. In particular, the paper deals with two types of composite functions for embeddings, one that computes union of the different classes represented by each embedding vector, and the other where the class of one of the embeddings is subsumed by the class of the other embedding. The actual composition functions are parameterized by neural networks whose parameters are learned from data. Results on synthetic as well as several real-world datasets highlight the superiority of the learned composite embeddings. 



Comments:
==========
1) This paper presents a welcome contribution to the saturated literature on embeddings. The whole idea of compositionally and its application to speaker diarization and multi-object detection is novel. 

2) The execution of the idea is also excellent and thorough. Further, the paper is very well written and puts itself nicely in context of previous work. I think this should inspire future work on other kinds of composite functions other than the two considered here. 

3) The results on both the synthetic and real-world omniglot and COCO datasets are impressive and mostly well executed and show significant improvement over the "most frequent" baseline. 


4) My only concern regarding the paper is w.r.t some arbitrary decisions made in the experiments e.g. how was the exact neural architecture for f in section 3.2 chosen? It seems contrived. Is it possible to do some ablation studies? Also, I think it will be nice to provide some more details regarding the neural network training in Section 3.1.

review_title: Official Blind Review #1

Meta-review: The authors propose a new type of compositional embedding (with two proposed variants) for performing tasks that involve set relationships between examples (say, images) containing sets of classes (say, objects).  The setting is new and the reviewers are mostly in agreement (after discussion and revision) that the approach is interesting and the results encouraging.  There is some concern, however, that the task setup may be too contrived, and that in any real task there could be a more obvious baseline that would do better.  For example, one task setup requires that examples be represented via embeddings, and no reference can be made to the original inputs; this is justified in a setting where space is a constraint, but the combination of this setting with the specific set query tasks considered seems quite rare.  The paper may be an example of a hammer in search of a nail.  The ideas are interesting and the paper is written well, and so the authors can hopefully refine the proposed class of problems toward more practical settings.
Decision: Reject