Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
In this paper, the authors study the problem of neural LTR models. They discuss why neural LTR models are worse than gradient boosted decision tree-based LTR models, and introduce some directions to improve neural LTR models.

Pros:
This paper discusses potential reasons why neural LTR models are worse than gradient boosted decision tree-based LTR models, and uses empirical results to show the effectiveness of the proposed solutions.

Concerns:
Regarding the proposed solutions, the authors use data augmentation to improve neural LTR models. It seems that some feature engineering work can help improve performance. Comparing to traditional gradient boosted tree-based LTR models, is it really worth putting efforts into studying neural LTR models?



review_title: Review comment regarding #952

final_score: rating: 2: Strong rejection
review_content: - The paper argues that neural models perform significantly worse than GBDT models on some learning to rank benchmarks. It first conducts a set of experiments to show that GBDT outperforms some neural rankers. Then, it presents a few tweaks related to feature transformation and data augmentation to improve the performance of neural models. The resulting neural models perform on par with the state-of-the-art GBDT models.
- I think this paper establishes an unfair comparison between GBDT and neural-based models. As known,  neural models are good at learning great representations from the raw inputs, such as audio, images, and texts, while GBDT models are good at dealing with sparse features. A more fair comparison could be having the neural models to learn feature representations, which will be concatenated with the normalized sparse features.
- Finally, the technical contribution of the paper is also quite limited.
review_title: Unfair comparison and limited novelty

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Thanks to the authors for their hard work and the nice paper. I both enjoyed this paper and think it's a strong contribution to the ranking literature. It was well written, clear, and nicely organized. The appendices are full of useful experiments.

There is some room for improvement, particularly in the thoroughness of the experiments:

- Eq (6) - It's nice to have a single transform for everything but what if the data is already, say, standard normal? Or categorical? Do any of your benchmark datasets have categorical variables? How were they dealt with? 
- Sec 3.2 - Hyperparameters for neural networks - learning rate and batch size are usually crucial for neural networks, but are not tuned for any of of the baselines (as far as I could tell). You are likely to get higher performance if these are tuned as well - please do so, especially for the baselines that you implemented yourself.
- It appears from the ablation study that some of the the bigger performance boosts came from the feature transformation and data augmentation. As this was meant to address issues with the benchmark methods, it would be very worthwhile to apply these techniques to those methods, perhaps while increasing the network capacity.
- Sec 5 Main Result - for fairness, you should compare to an ensemble of lambdaMARTs.
- What about training time? LightGBM is very fast. How long does it take to train your model? Is the few percent improvement worth it?
- Does lambdaMART also get better with Gaussian noise data augmentation?

One final note: I'm not sure the word "hitherto" makes sense in the title.
review_title: Great focused contribution for ranking 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Originality, Significance: This paper establishes reference points for modern LTR research. The fact that RankLib is a very popular but also a weak baseline has been exploited by too many researchers for too long. When I was reviewing other LTR papers, I often had to point out that the proposed method significantly underperforms LightGBM. This fact has been fairly well known, but apparently not widespread enough. Having an ICLR paper published on this issue will help spreading the fact, which is significant on its own.

Quality: Considering the popularity of RankLib, deeper analysis of why LightGBM outperforms RankLib would've been very nice, however. Authors do mention that LightGBM has more features, but it is unclear which exact feature of LightGBM contributed to such a significant difference between the two. Understanding the reason for LightGBM's superiority could potentially help us to develop better LTR models, neural or not neural. Comparison against Catboost https://github.com/catboost/catboost would've also been useful, as it is often claimed to outperform LightGBM.

Proposed DASALC framework is quite simple and uses mostly standard techniques, and this is an advantage as a reference point. Still, DASALC significantly outperforms previous neural LTR approaches. Also, although the idea of applying these standard techniques on LTR seems straightforward, but I argue that's only due to the benefit of hindsight; neural LTR has been a fairly active area of research, yet these techniques haven't yet been widely used in LTR literature. It would've been very interesting to see how these techniques improve the performance of previous neural LTR models; log1p transformation, data augmentation, and model ensembling would straightforwardly apply to other neural models as well.

In summary, I believe this paper will foster more productive research by establishing the strong baseline on both decision-tree based method (although it has been known) and neural method (on which authors make good technical contributions).

Clarity: The paper is quite easy to follow.
review_title: Much needed work for neural LTR research to make progress

Meta-review: Three reviewers are positive, while one is negative. The negative reviewer is well-qualified, but the review is not persuasive. Overall, this paper should be published as a wake-up call to the research community. Unfortunately, the lesson of this paper is similar to that of several previous papers, in particular

Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009, November). Improvements that don't add up: ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 601-610).

This submission should be a spotlight, to maximize the chance that future researchers learn its lesson.
Decision: Accept (Spotlight)