Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors present PROCAT dataset containing 11,063 product catalogs. Each product catalogs consist of several sections (pages) and many product offers. This dataset is made up of 1,613,686 product offers (instances). Each offer instance consists of its unique id, related section, catalog ids, text heading, and description. The proposed task is a set-to-sequence problem, where the ordered list of offers (the ranking) is predicted given the set of offer descriptions. The main contributions are:
1. Defining a new task for learning the correct permutation of offers in product catalogs.
2. Providing a new dataset containing more than 1 million product offers.
3. Introducing a synthetic dataset for the same task.

strengths: 1. This dataset is the first of its kind and can open up new research directions.
2. The dataset contains more than 1.5 million individual product offers which can be used for training large neural networks.
3. Basic baselines are provided for both real and synthetic datasets, and the code is also shared.

weaknesses: 1. Only one "right" order is provided for each product catalog. At the same time, there are usually many reasonable ways of designing the catalogs, and the human-designed one might be far from optimal. This formulation makes the problem too hard. Even on the synthetic dataset, the performance is very low. 
2. The dataset is not diverse as it only includes text in Danish. However, designing product catalogs is a very subjective task, and the preferences might be widely different in other parts of the world.
3. The image of the products are not included in this dataset. Sometimes the relative ordering of the product offers in the same section is determined by their pictures (e.g., similar-looking products might be located close to each other). Without that information finding the optimal ordering seems infeasible. 
correctness: The experiments and evaluation methods seem sound to me.
clarity: The paper is mostly clear. However, the authors could do a much better job in including better figures. The main structure of the dataset and the input outputs of the neural network are not clearly explained. 
relation to prior work: This is a new task and a new dataset designed for it, but related datasets are discussed in section 2.2.
additional feedback: The authors can make their dataset and work stronger by including:
1. More than one alignment for each product catalog (for example, three possible orderings by three different human designers)
2. Product catalogs in other languages and other parts of the world to ensure a diverse dataset.
3. Product photo for each offer to better predict the relative order within sections.
review_title: Not a well-defined problem formulation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a new large-scale dataset for products catalogue with both textual descriptions and structured information as well as meaningful ordering between the items in the catalogue. I'm not aware of many public datasets with structured/ordered information in this realm and so that is great to have a new one that covers that. In addition, it adds also baseline experiments on the task of set-to-sequence prediction over the data as well as means to generate additional synthetic catalogue data. Finally, The overall paper is written clearly and the work is well-presented, motivated and justified.

Update: After reading the author's response I would like to keep my score and overall I think it reflects my thoughts well!
strengths: - The new dataset is large scale, with over 1.5 millions items collected over the course of 4 years, and collected from 10k different catalogues (offering diversity). The items and offers are augmented by multiple fields, such as prices, sections, priority classes, and ordering. 
- The dataset has multiple potential practical applications and uses in the field of e-commerce and is thus a great contribution to the research community. The paper also does great in terms of motivating the need for the dataset and describing these potential uses, both from commerce perspective, as well as in terms of challenges for the machine learning community.
- It's great to see that the authors show awareness and work to address ethical aspects related to constructing and releasing the dataset (more details below).
- As discussed above it's good that the dataset comes with performance analysis of baseline models for related prediction tasks, and in addition to that, it also provides a library for auto-generation of synthetic data that supports new evaluation metrics. I find the balance between having both real-world data but also providing means for a more controlled analysis and evaluation through synthesized data to be both practically useful and academically interesting.
- The paper does well in presenting in detail both the collection and preparing process of the data, and what each items 

weaknesses: - I find it a bit surprising that for the experiments the authors choose to focus only on the set-to-sequence prediction use of the dataset. This sounds like a very specific task, while I'm certain there could be a larger variety of supervised and unsupervised tasks on top of the data, in the areas of clustering, generating textual descriptions given the product image, estimating prices, predicting recommendation for new items based on previously purchased ones, etc etc.
- Following up on the previous point, for some of these tasks, additional annotation within the dataset will be necessary. I think that the underlying dataset is a great and quite large resource, and it could be really nice if it was augmented with additional annotations of the items (some potential examples: prices, recommendations, properties of users that bought them, correlation between purchasing different items, etc. etc.). I don't know if these details could be easily collected but would be great to explore this direction!
- Not detailed enough online documentation of the dataset. Will be helpful to have a website with some information about the dataset, its key features, and the files/directory structure and format.
correctness: The claims in the paper seem to me sound and they are corroborated by details, explanations, experiments (for the baseline models), and examples.
clarity: The paper is clear, easy-to-follow and written in an interesting compelling manner. The writing quality is good and there are multiple helpful visualizations. It would be helpful if the website did a better job in presenting the dataset and providing more details about its format and main features.
relation to prior work: The paper does a good job in comparing to prior work and contextualizing the new dataset within the current field. It motivates the general task, provides background in the area including prior works, and explaining how the new dataset is distinguished from them.  It also discusses various set-to-sequence methods (which are relevant for the experiments section).
additional feedback: Nope, all my feedback is above.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes an e-commerce dataset consisting of over 1.5 million set items across a 4-year period, in both raw text form and with pre-processed features containing information about relative visual placement.
strengths: The introduction of data collection, statistical information, and application in this paper is very detailed. And the paper is well written.
weaknesses: I have two concerns.
First, the diversity of the dataset is relatively limited. Because the language and number of users involved in the data collection process are not enough to show the diversity of the dataset.
Then, whether the challenge of the dataset that the authors emphasize is caused by the quality of the dataset.
correctness: The paper did not propose a benchmark method, and the baseline methods for verification are also very limited.
clarity: The paper is well written.
relation to prior work: The comparison with related work is very limited in this paper. This is not enough to highlight the difference and contribution of the paper relative to the previous work.
additional feedback: See the discussion in "Weaknesses".
review_title: PROCAT: Product Catalogue Dataset for Implicit Clustering, Permutation Learning and Structure Prediction

Meta-review: In this paper the authors offer a dataset of many product catalogues and the task of predicting the order of products based on the set of items and their features. The task is fairly novel and the dataset is notably quite large. As with many datasets containing text, it could be improved by covering a broader swath of languages. Further, there were ideas raised of how to improve or expand the uses of this dataset, e.g. for new tasks or to allow flexibility beyond a single correct answer, and these would likely be valuable improvements.

In discussing with other ACs, we agreed that while the paper could be improved by covering more languages, this dataset is not unique in only covering one language and thus not grounds for rejection. With respect to including more than a single "correct" ranking per example, we felt that the scale of the dataset outweighed this downside. It'd be great if the authors could improve along these dimensions, but even with these limitations we feel the benefits outweigh the limitations.
Decision: Accept