Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper provides a large scale dataset for search containing samples from Baidu’s search logs. The dataset contains a lot of new information about aspects of the search results page or user interaction that were previously ignored by other papers or were too small to train large transformer models. The information is also released in an anonymized way reducing the risk of user privacy exposure.
strengths: (1) The dataset itself is quite large for a search dataset. 1.2B sessions and 7k+ expert annotations. This would be very useful for researchers not working at giant search engine companies.
(2) Most search papers/datasets focus on solely the rank part of search result pages. With modern search evolving to detailed abstracts and rich multimedia content within the results page, it is important to understand their effect on user engagement. 

weaknesses: The paper does not have many weaknesses. It is a large scale search dataset that would be pretty useful. The only concern I have is with the expert annotations and what kind of biases raters may have introduced while providing the relevance labels. I think a brief section about the rater guidelines and potential problems they could introduce into the dataset would be useful. With the data being anonymized we may never really know if experts tend to favor certain kinds of documents over others.
correctness: The data collection process and baseline methods seem to be correct. In particular Fig. 3, 4 are typical of search datasets and the authors provide DCG scores with a few different ULTR methods.
clarity: The paper is well structured and easy to follow.
relation to prior work: Prior work is well discussed. I agree with the authors, there are no/few datasets out there that contain detailed SERP level info along with user engagement metrics. Also most works before do not release query/document text with user feedback or are too small.
additional feedback: Sentences do have a lot of grammatical mistakes but they don’t affect the overall flow of information. Would urge the authors to reword some of them if possible, for ex:
(1) “BERT cannot be exploited due to the missing of the original text” -> “BERT models cannot be used because of missing original text”
(2) “missing the displayed abstract of documents for analyzing the click necessary bias” -> “missing abstract information useful for analyzing click bias”
(3) “vogue of deep learning” ? does this mean “rise of deep learning”
(4) “consider the border impact” -> broader*
review_title: Good paper, large dataset.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a large-scale unbiased learning to rank dataset for mobile web search, Baidu-ULTR, which randomly sampled 1.2 billion searching sessions from the largest Chinese search engine Baidu, and 7,008 expert annotated queries for validation and test. Baidu-ULTR provides: (1) the original semantic feature and a pre-trained language model for easy usage; (2) sufficient display information for enabling the comprehensive study of different biases; and (3) rich user feedback on search result pages (SERPs) for user engagement optimization and promoting the exploration of multi-task learning in ULTR. In addition, this paper presents the design principle of Baidu-ULTR and the performance of benchmark ULTR algorithms on the new data resource, favoring the exploration of ranking for long-tail queries and pre-training tasks for ranking.
strengths: 1.The idea of proposing a large-scale search dataset Baidu-ULTR from industry for unbiased learning to rank is particularly meaningful and necessary.
2.This paper presents the design principle of Baidu-ULTR and the performance of benchmark ULTR algorithms on the new dataset in detail, which has the potential to become the gold benchmark in the field of ULTR.
3.In general, this paper is well-organized and is a good work that can greatly promote the development of the field of ULTR.
weaknesses: 1.The presentation of the paper should be improved. The expressions in current version are not natural and elegant enough, and it would be better to improve it with the help of an English native speaker.
2.The layout of the paper should be optimized, especially the tables and figures in current version (e.g., compactness, color, etc.).
3.Because Baidu-ULTR is a large scale real-world dataset from industry, the balance of open source and privacy protection requires extra care. And some details of dataset collection and data pre-processing, including data desensitization, should be more clear.
4.As paper claims, data limitations, including inconvenient model sharing and mismatch between training and test, should be discussed more.
correctness: From my perspective, the dataset Baidu-ULTR is constructed in a sound way and the performance of benchmark ULTR algorithms on it is correct.
clarity: In general, this paper is well-organized and well-written. And it would be better to improve the presentation of it with the help of an English native speaker.
relation to prior work: As far as I can tell, this paper has clearly discussed how its proposed Baidu-ULTR differs from previous contributions.
additional feedback: In general, this paper is a good work that can greatly promote the development of the field of ULTR, and the proposed dataset Baidu-ULTR has great potential to become the gold benchmark in this field.
review_title: Baidu-ULTR: A good work to promote the development of the field of ULTR

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a large scale dataset of search queries from Baidu search engine, they added behavioral features and page presentation features. 
The contributions are as following:
- largest-scale search dataset
- added behavioral features
- added page presentation features
- expert annotations
- comparison of various ranking benchmarks 

strengths: - The search dataset covers a large range of web searches and queries
- The authors extracted a different kind of information from these web pages
- The authors tried to utilize the literature loss equations but adapted for the user’s implicit feedback for unbiased learning.
- The authors compared different benchmarks for showing the performance of the dataset using unbiased learning.
- I liked that they showed the tail queries performance and showed how the performance degrades for such queries. 
weaknesses: - The work is based on extracting pages from Baidu search engine, which targets mainly the Chinese world, this causes the dataset to lack diversity and lack different users' queries.
- The work gives high importance to the users' clicks in the introduction and building of the dataset, however in the discussion and the results we can not reach this importance in the end, the naive benchmark with the standard features is performing really well. This needs more illustration how these features can impact the performance.
- Similarly, the discussion of the performance tables needs more illustration, on how the display information and user behavior impact the performance, also more illustration of the different accuracy results of DCG@1, DCG@3, ....etc.

correctness: The claims are correct, and the construction of the dataset seems sound.
There are external annotators to annotate data to be useful for evaluation.
The experimental design seems appropriate when I revised the experimental design in this area. However, I am not quite familiar with the experimental setup in this area, this should be revised by another reviewer.
clarity: The paper is generally well written, except for the sections on results and discussion, which need paraphrasing and extra illustration in some parts.
Also, the arrangement of the experimental framework is a bit confusing, I would move the evaluation metrics to the end and speak first about the models and the baselines. 
As far as I understand the annotated data can be used for evaluation but is all the data annotated with enough external evaluators? 
The tables did not illustrate this well in my point of view.
relation to prior work: Mainly added features and a much larger dataset, also an enhancement to the loss function used for training the model for better unbiased learning.
Here, too, I would like to refer to another reviewer for the writing of the adaptation of the loss function and if this adaptation was essential and useful.
additional feedback: - The dataset is very useful to the Chinese research community, however, to extend its usefulness to further communities, it would be great if we have diverse search queries from different engines that do not focus on the Chinese user. For other research communities, the usefulness is questionable.

- The paper is a nice contribution and opens research questions about the usefulness of added features.
review_title: nice contribution for building large scale search dataset with useful features, from certain type of users 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a new real database for ULR based on Baidu chinese application. This database is large in size. It is compared to other databases of this type which are already numerous.
The database is available free of charge. The textual part is in Chinese, which does not facilitate its use. 
strengths: The main interest of this paper is to provide a database for unbiased learning to rank. 
weaknesses: The contribution on the collected data remains for me rather weak. The annotation by experts is quite basic. The use of this database, even if it is made available, will probably not be easy. 
correctness: For this dataset, queries are randomly sampled from search sessions of the Baidu search engine in April
2022. Expert annotation was made with 5 relevance labels of (bad, fair, good, excellent, perfect). Half are bad. 
clarity: The paper is well written and easy to read.
relation to prior work: The link with other database is good.
additional feedback: It could be interesting to propose a less strict annotation of the experts, for example a fuzzy annotation.
review_title: A new database for ULR

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: As the paper states in its abstract, the proposed dataset provides original semantic features and pre-trained language model. It also provides sufficient display info and rich user feedback.
strengths: First, this paper has the contributions of providing the original semantic
feature and a pre-trained language model, sufficient display
information and rich user feedback on search result
pages. Besides, this dataset is general enough, and can be used in other research areas. Also, the paper provides enough metric results.
weaknesses: The amount of expert annotations queries is far less than the amount of search session. The authors did not provide enough explanation why 7,008 expert annotations are sufficient for 1.2 billions searching sessions. Besides the expert annotation distribution is not balanced. The perfect only takes 1.8% of total expert annotations. It will be better if the authors could provide more expert annotated queries with balanced distribution.
correctness: The dataset is constructed in a sound way. The evaluation methods is appropriate.
clarity: The paper is well written.
relation to prior work: Yes, this paper clearly discussed previous the disadvantages of previous similar datasets, and clear shows what they provides. 
additional feedback: Generally, this is a great paper with remarkable contribution and enough metric results.
review_title: This is a great paper with lots of contributions. The proposed datasets could dramatically benefit other researchers' work.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces the Baidu-ULTR dataset for unbiased learning to rank. 

The contributions of this work include:
1. It introduces original text in the form of token ids for queries and documents so that the pre-trained language models can be used
2. It provides more diverse display information in addition to the position information, which is an improvement over previous works
3. It also offers rich user behaviors like dwelling time, displayed time, etc., in addition to the click behavior
4. Compared with previous works, this is a large-scale dataset, with real user feedback and expert annotations. The long-tailed query research is also supported by design.
5. More tasks can be done in addition to the click modeling, e.g., dwelling time prediction, etc.

In addition to the contributions, the strengths of this work include:
1. Unbiased learning to rank is an important research topic, I believe this dataset can greatly accelerate this direction's development and bring benefits to the broader research community
2. The dataset is publicly available, baselines, and pre-trained models are available

The weaknesses of this work include:
1. As mentioned in the paper, due to the privacy issue, the texts are transformed into token ids, making the most commonly used pre-trained models not applicable to this dataset, thus limiting the impact of this dataset. Also, it makes the data analysis hard, as a result, the following researchers can only work on the algorithm or model side
2. Also as mentioned in the paper, there is a mismatch between training and test data distributions. 
strengths: See "Summary And Contributions"
weaknesses: See "Summary And Contributions"
correctness: See "Summary And Contributions"
clarity: The paper is well written.
relation to prior work: See "Summary And Contributions"
additional feedback: See "Summary And Contributions"
review_title: A good dataset for unbiased learning to rank research

Meta-review: metareview: The reviewers seem to be in agreement on many aspects, including the strengths, the weaknesses, and the final assessment. 

As strengths, the reviewers highlight the following points:
- the size of the dataset, 
- the usefulness to the community

As weaknessess, the reviewers highlight the following points:
- the paper in layout and grammer
- potential bias towards languages and industry
- biases with respect to the annotaters

The reviewers provide scores ranging from 5 (although the reviewer concedes that they are not certain about their assessment) until 9 (rather confident), averaging at a 7. I agree with the aforementioned strong points, and do not see any showstoppers in the recurring weaknesses, so will therefore recommend acceptance of the paper. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept