NIAQUE: Neural Interpretable Any-Quantile Estimation - Towards Large Probabilistic Regression Models

Authors: Boris N. Oreshkin, Dmitry Efimov, Siamak Gilan, James Nordlund
Primary area: foundation or frontier models, including LLMs
Keywords: deep probabilistic regression, large regression models

Abstract:
State-of-the-art computer vision and language models largely owe their success to the ability to represent massive prior knowledge contained in multiple datasets by learning over multiple tasks. However, large-scale cross-dataset studies of deep probabilistic regression models are missing, presenting a significant research gap. To bridge this gap, in this paper we propose, analyze, and evaluate a novel probabilistic regression model, capable of solving multiple regression tasks represented by different datasets. To demonstrate the feasibility of such operation and the efficacy of our model, we define a novel multi-dataset probabilistic regression benchmark LPRM-101. Our results on this benchmark imply that the proposed model is capable of solving a probabilistic regression problem jointly over multiple datasets. The model, which we call NIAQUE, learns a meaningful cross-dataset representation, scores favorably against strong tree-based baselines and Transformer and exhibits positive transfer on unseen datasets after fine-tuning.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730578748802

thread_type:
official_review

<thread_object>
creator: reviewer_jsss

<content>
summary:
This paper aims to narrow the gap between deep probabilistic regression models with current powerful deep neural networks. It proposes a large unified model to handle regression tasks across different datasets. Correspondingly, it introduces a framework called NIAQUE, to address probabilistic regression problem. Compared with several baseline methods, the proposed model achieves better empirical results.

strengths:
1. This paper aims to narrow the gap between current neural networks and probability regression models, which is still under explored.
2. It proposes a complete working pipeline from building benchmark, proposing new model with theoritical analysis, and empirical analysis.
3. Overall, the draft is easy to read and follow.

weaknesses:
1. The problem statement is still not clear, why use transformer model to handle such tasks, what kind of real-world applications to illustrate the model practices? more discussions and intuitions are needed to support the motivation of this work.
2. Some table and figure formats are not well prepared such as figure 3. Necessary descriptions are needed.
3. Based on the table 1, there is not clear difference of final results compared with previous works, what is the benefit of proposing such a learning pipeline? how to illustrate the proposed model superiority?

questions:
Please refer to the weaknesses section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582553614

thread_type:
official_review

<thread_object>
creator: reviewer_njbn

<content>
summary:
The current manuscript introduces NIAQUE, a neural network model for probabilistic regression that performs any-quantile estimation across multiple datasets.  NIAQUE takes a training batch of the variable number of observations and generates a quantile of uniformly at random in the [0,1] range for each training sample. The quantile is used as the input of the quantile decoder to modulate
the observation representation and as the supervision signal in the quantile loss.

Besides this, the paper also introduces the LPRM-101 Benchmark which is a new benchmark with 101 diverse regression datasets from different sources like UCI, PMLB, OpenML, and Kaggle, aimed at assessing the performance of models on probabilistic regression tasks. The proposed method enables to handle a range of quantiles in a single framework. Its encoder-decoder design effectively manages variable data formats across datasets, making it flexible for multi-dataset.

strengths:
* The authors provided the LPRM-101 benchmark which allows for rigorous evaluation across diverse regression tasks and data sources.

* The proposed NIAQUE seems a new probabilistic regression that learns to approximate the inverse of the posterior distribution during training.

weaknesses:
* W1: co-training across datasets could introduce biases that are absent in single-dataset models. This is especially relevant if certain datasets dominate the training process, potentially leading to skewed representations and reduced performance on minority datasets. It would be interesting to see the performance of your model in individual datasets when trained jointly vs. separately.  

* W2: While the any-quantile framework is flexible, the performance of the model may vary depending on the choice of quantiles.The lack of specific guidance on the choice of quantiles may prevent users from achieving optimal results in different use cases.

* W3:NIAQUE relies on specific probabilistic assumptions for feature importance and distributional modeling, which may not be suitable for all regression tasks, particularly those with non-standard distributional characteristics. This could limit the model’s applicability to a narrower range of tasks than intended.

questions:
* Q1) I am wondering how NIAQUE's performance varies with different quantile choices.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685199706

thread_type:
official_review

<thread_object>
creator: reviewer_p8dt

<content>
summary:
The authors focus on the problem of multi-task probabilistic regression over tabular datasets. They contribute a benchmark of 101 regression datasets drawn from public sources, and develop a neural network architecture trained with quantile regression for this setting. They demonstrate that their architecture, NIAQUE, outperforms tree-based and Transformer baselines in both point and distributional metrics. The authors also provide a theoretical result demonstrating the consistency of their any-quantile training algorithm.

strengths:
* Good motivation: The problem setting of multi-task learning across tabular regression datasets is a promising and important one, so the paper is well-motivated.
* NIAQUE compares reasonably well to strongest baseline: The NIAQUE architecture appears to have decent empirical results, performing slightly better than the Transformer with ~2x reduction in wall-clock training time (L254).
* Strong benchmark and set of evaluation datasets: The contributed LPRM-101 benchmark is expansive.
* The method (the neural network architecture and quantile regression objective) seem sensible and novel.
* The consistency result for NIAQUE appears to be correct.

weaknesses:
* Minimal gains from multi-task transfer: the core pitch of the paper is to contribute an architecture that works well for multi-task learning, but the gains from learning across datasets appear to be pretty marginal (comparing NIAQUE-local and NIAQUE-global in Table 1).
* Unclear presentation: Some parts of the presentation of NIAQUE were confusing or unclear. How exactly is the model used for prediction if the encoder takes a variable number of inputs? At test time, do you concatenate each query point with the training dataset (or a subsample of training points) and feed them to the encoder to obtain the encoder representation? If so, have you considered various heuristics for selecting these points (e.g., nearest neighbors to the query input)?

questions:
My main concern is minimal gains from multi-task transfer. 
* One way to address this: is it possible to evaluate this model in the standard meta-learning setup, with entirely held-out datasets, as in [1, 2]? If these results are strong, it might better justify NIAQUE even when in the "held-in" dataset setting, NIAQUE-local and NIAQUE-global perform similarly. In the meta-learning setting, Neural Processes are the relevant baseline. If NIAQUE is competitive with them, it would encourage me to raise my score.

Other medium-priority concerns:
* In Table 1, why is Transformer-local not included? How does it perform?
* The architecture appears to resemble Neural Processes [1, 2], another neural architecture for multi-task learning / meta-learning. Could you contextualize NIAQUE with regard to them?
* The nomenclature of "co-training" seems to clash with the old-school pseudo-label style setting as in [3]. Could you instead use the established nomenclature for this setup (multi-task learning, pretraining, or meta-learning depending on the context?).

[1] Garnelo et al., 2018. Neural Processes. https://arxiv.org/abs/1807.01622

[2] Garnelo et al., 2018. Conditional Neural Processes. In ICML. https://proceedings.mlr.press/v80/garnelo18a.html

[3] Blum and Michell, 1998. Combining Labeled and Unlabeled Data with Co-Training. In COLT. https://www.cs.cmu.edu/~avrim/Papers/cotrain.pdf
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718372828

thread_type:
official_review

<thread_object>
creator: reviewer_ukfu

<content>
summary:
This paper introduces NIAQUE, a new model for probabilistic regression that learns to approximate the inverse of the posterior distribution. It also introduces a new benchmark LPRM-101 with 101 datasets for multi-dataset probabilistic regression. The authors provide theoretical analysis for NIAQUE and also show that it is competitive with (or better than) prior baselines (trees, transformers) on the new benchmark. Additionally, the authors argue that NIAQUE improves over existing baselines by being more interpretable.

strengths:
- To my knowledge, the benchmark is novel because it focuses on multi-dataset probabilistic regression
- NIAQUE design decisions are explained and also include some theoretical analysis
- NIAQUE is more interpretable while having competitive performance when compared with standard approaches
- Experimental results sweep over parameters and include confidence intervals

weaknesses:
- While the paper introduces a new benchmark, it lacks detail or tooling that would allow others to use it. I would like to better understand how easy it would be for another researcher to use it and build off this work.
- This is a multi-dataset benchmark, but I would like to understand how the best existing single dataset approaches perform, or if the best approaches are the ones in Table 1. For example, are the top submissions to the Kaggle datasets' respective competitions significantly different from Transformers or XGBoost? How do those numbers compare with the "local" approaches in Table 1?

questions:
See weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces NIAQUE, a neural network model for probabilistic regression problem that excels in any-quantile estimation across multiple datasets. It also presents a new benchmark LPRM-101, with 101 diverse regression datasets, showing that NIAQUE outperforms existing models like trees and transformers while being more interpretable and consistent in its training. 

This paper got a mixed rating with borderline accepts and rejects. The authors acknowledge that NIAQUE is more interpretable with theoretical analysis and can save significant training time. The proposed LPRM-101 benchmark allows for rigorous and expansive evaluation across diverse regression tasks. Several questions and concerns are raised in the initial round, including 1. Experimental performance and comparison. Marginal gains from multi-task transfers. It is also suggested to add comparison with the best single dataset approaches. 2. Presentation and clarity issue. 3. Bias concern in co-training across datasets. After rebuttal, some reviewers raise the ratings, but it is still a borderline (e.g. after raising the rating by p8Dt, it is still a 5). Some concerns like the marginal gain from multitask transfer and the challenges of real-world application still remain. Given all these, I tend to suggest a reject given the current paper status and the ICLR standard.