Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the problem of modeling sets via sequence models via generative approaches. Mainly this paper considers all the possible state-action paths that would yield the same set and approximately marginalizes over them for MLE estimation. An approach similar to the popular "REINFORCE"/score-matching trick is proposed to define a Monte-Carlo based estimator for the gradient of the problem and importance sampling is done to efficient estimate the gradient for training. Two instantiations of the model are proposed -- GRU2set and SetNN. The first uses an autoregressive decoder's states to represent items in the sets, the second approach defines an equivariant MLP based function to represent items generated so far in the set. Size-bias trick is also introduced which biases the learning of set generation according to the size. The experiments are done on ecommerce order datasets, and compared with histogram based baselines and other relevant models
strengths_and_weaknesses: This paper tackles an important problem which would be of great interest to retro fit essentially sequnce-based architecture for modelling sets. The main approach is clearly explained and is sound.

My main concern is that the empirical results show that the simple histogram-based baseline outperforms most of the models. The proposed approach trails until it applies the size bias trick to see modest gains over the baseline. 

Analysis of performance variation according to the size-wise variation of the set would improve the paper.

Why not use transformers? A simple transformer model can come close to modelling set properties by disabling positional embeddings. Comparison to transformers/SetTransofrmers is crucial given their ubiquity.

Related work: multilabel classification literature faces the same issues, a discussion about the approaches in this paradigm would be useful. For example: "Orderless Recurrent Models for Multi-label Classification", Yazici et al.

The paper is awkwardly written in parts – the size bias trick is very difficult to understand.

Line 235/236 is very confusing
questions: How does “sparse item graph" constructed form training data generalize to test sets?
Confused about their definition of "induce"


limitations: n/a
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors in this paper propose Sequence-to-Set that aims to convert any likelihood based sequence generative model to a set generative model. The proposed model first introduces a probability (likelihood) over a set $S$ induced from the generating path of the sequence generative model. Then an efficient importance sampling method is devised to tackle the training challenge of maximizing the set likelihood. The authors also present two instance of the proposed model, GRU2Set and SetNN to capture the complex relations among data points. Besides, a size-bias trick is designed to learn better set distribution from data. The proposed model is evaluated on two e-commerce order datasets, and the results show that the proposed model outperforms its baselines.
strengths_and_weaknesses: Strengths:

(1) The paper clearly identifies the problems with previous works and presents the idea of the proposed method.

(2) The paper is well-written and easy to follow. 

(3) The paper provides a strong baseline for the set generative model, and several attempts can be tried base on the proposed method.

(4) The experiments are well-designed.


Weaknesses:

(1) More deeper analysis about the experiments. for example, without Size-bias, the proposed model has different performance on the two datasets. The authors should give more discussion.

(2) Empirically, the Size-Bias trick is the key for the proposed method and it can be used as a plug-and-play module to improve any generative model. It seems that the Size-Bias trick improves the proposed two models larger than baselines. The authors should give detailed explanation.




questions: (1) I don't quite understand the definition of the indicator function $\tau$, is there any minor in $s_i=s \land a_i = a$ ?

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper tackles the challenge of developing generative models for sets, which have the property of permutation invariance. The authors first present a method for converting a sequence generative model to a set generative model based on maximum likelihood.  Additionally, they also propose an importance sampling algorithm to estimate the gradient of the log-likelihood function to improve the efficiency.  The authors further propose two parameterizations of the sequence-to-set model, which are GRU2Set and SetNN.  Finally, the authors propose a size-bias trick that can help reduce the distance between the ground truth set distribution and the learned distribution.

Empirical evaluations have been performed on two real-world datasets to convey the effectiveness of their method.
strengths_and_weaknesses: Strengths:
1. The paper studies an important and understudied problem and proposed an interesting method 
3. The empirical results are strong against baseline methods

Weaknesses:
1. The presentation of the paper is a bit hard to follow.  I would suggest the authors to illustrate the size-bias trick with a running example.
questions: 1. The formats of the citations are incorrect in a few places including the very first sentence in the introduction, making the text hard to read.
2. What are the authors boldfacing in the tables?
3. It’s a bit hard for me to understand why the performance for the > 7 group is even better than the smaller sizes.  Learning the set distribution for more items is an easier problem?  What’s the intuition here.
limitations: No ethical concerns.  Other limitations discussed in earlier sections.
review_title: null

Meta-review: metareview: Reviewers found the problem to be well-motivated and the results convincing. They objected to not using transformers and some difficulties to outperform the strong histogram baseline and presentation of the key size-bias trick.

The authors addressed some of the issues raised and all reviewers rated weak accept.
confidence: Certain
Decision: Accept