Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This is an interesting paper, in the sense of looking at a problem such as multiset prediction in the context of sequential decision making (using a policy).  

In more detail, the authors construct an oracle policy, shown to be optimal (in terms of precision and recall).  A parametrized policy instead of the oracle policy is utilized in the proposed multiset loss function, while furthermore, a termination policy facilitates the application on variable-sized multiset targets.  The authors also study other loss functions, ordered sequence prediction as well as reinforcement learning.

Results show that the proposed order-invariant loss outperforms other losses, along with a set if experiments evaluating choice of rank function for sequence prediction and selection strategies.  The experiments seem rather comprehensive, as well as the theoretical analysis.   The paper describes an interesting approach to the problem.

While the paper is comprehensive it could be improved in terms of clarity & flow (e.g., by better preparing the reader on what is to follow)
review_title: multiset prediction as sequential decision making

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a type of loss functions for the problem of multiset prediction. A detailed discussion on the intuition is provided and extensive experiments are conducted to show that this loss function indeed provides some performance gain in terms of Exact Matching and F1-score.

The idea of this paper is as follows: instead of viewing the multiset prediction task as a classification problem, this paper models it as a sequential decision problem (this idea is not new, see Welleck et al., 2017, as pointed out by the authors).  Define pi* to be the optimal policy that outputs the labels of input x in a certain way. We learn a parameterized policy pi(theta) which takes x and all previous predictions as input, and outputs a label as a new prediction. At each time step t, pi* and pi(theta) can be viewed as distributions over all remaining labels; the KL divergence is then used to calculate the difference between those two distributions. Finally, the loss function sums those KL divergences over all t. Computing this loss function directly can be intractable, so the author suggested that one can compute the entire trajectory of predictions and then do aggregation.

Admitted that such construction is quite intuitive, and can possibly be useful, the technical part of this paper seems to be rather straightforward. In particular, I think the solution to the issue of unknown T is very heuristic, making the proposed loss function less principled. 


Other Detailed Comments/Issues:

-- It seems to me that models that can utilize this loss function must support varying-length inputs, e.g. LSTM. Any idea how to apply it to models with fixed-length inputs?

-- The proposed loss function assumes that T (i.e. the size of the output set) is known, which is unrealistic. To handle this, the authors simply trained an extra binary classifier that takes x and all previous predictions as the input at each time step, and decides whether or not to terminate. I think this solution is rather hacky and I’d like to see a more elegant solution, e.g. incorporate the loss on T into the proposed loss function.

-- Could you formally define “Exact Match”?

-- In Section 4.4, maybe it is better to run the stochastic sampling multiple times to reduce the variance? I would expect that the result can be quite different in different runs.

-- Any discussion on how will the classifier for T affect the experimental results? 

review_title: While the proposed solution is intuitive, and can possibly be useful, the technical part of this paper is a bit weak.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 
The paper considers the prediction problem where labels are given as multisets. The authors give a definition of a loss function for multisets and show experimental results. The results show that the proposed methods optimizing the loss function perform better than other alternatives.

Comments: 
The problem of predicting multisets looks challenging and interesting. The experimental results look nice. On the other hand, I have several concerns about writing and technical discussions. 

First of all, the goal of the problem is not exactly stated. After I read the experimental section, I realized that the goal is to optimize the exact match score (EM) or F1 measure w.r.t. the ground truth multisets. This goal should be explicitly stated in the paper. Now then, the approach of the paper is to design surrogate loss functions to optimize these criteria. 

The technical discussions for defining the proposed loss function seems not reliable for the reasons below. Therefore, I do not understand the rationale of the definition of the proposed loss function.:  
- An exact definition of the term multiset is not given. If I understand it correctly, a multiset is a “set” of instances allowing duplicated ones. 
- There is no definition of Prec or Rec (which look like Precision and Recall) in Remark 1. The definitions appear in Appendix, which might not be well-defined. For example, let y, Y be mutisets , y=[a, a, a] and Y = [a, b]. Then, by definition, Prec(y,Y)=3/3 =1. Is this what you meant? (Maybe, the ill-definedness  comes from the lack of definition of inclusion in a mutiset.) 
- I cannot follow the proof of Remark 1 since it does not seem to take account of the randomness by the distribution \pi^*. 
- I do not understand the definition of the oracle policy exactly. It seems to me that, the oracle policy knows the correct label (multi-set) \calY for each instance x and use it to construct \calY_t. But, this implicit assumption is not explicitly mentioned. 
- In (1), (2) and Definition 3, what defines \calY_t? If \calY_t is determined by some “optimal” oracle, you cannot define the loss function in Def. 3 since it is not known a priori. Or, if the learner determines \calY_t, I don’t understand why the oracle policy is optimal since it depends on the learner’s choices. 

Also, I expect an investigation of theoretical properties of the proposed loss function, e.g., relationship to EM or F1 or other loss functions. Without understanding the theoretical properties and the rationale, I cannot judge the goodness of the experimental results (look good though). In other words, I cannot judge the paper in a qualitative perspective, not in a quantitative view. 

As a summary, I think the technical contribution of the paper is marginal because of the lack of reliable mathematical discussion or investigation.

review_title: Technical contribution of the paper is marginal because of the lack of reliable mathematical discussion or investigation.

Meta-review: The submission addresses the problem of multiset prediction, which combines predicting which labels are present, and counting the number of each object.  Experiments are shown on a somewhat artificial MNIST setting, and a more realistic problem of the COCO dataset.

There were several concerns raised by the reviewers, both in terms of the clarity of presentation (Reviewer 1), and that the proposed solution is somewhat heuristic (Reviewer 3).  On the balance, two of three reviewers did not recommend acceptance.
Decision: Reject