Informed Exploration via Generative Modeling

Authors: Kelly W. Zhang, Tiffany Cai, Hongseok Namkoong, Daniel Russo
Primary area: reinforcement learning
Keywords: bandit algorithms, Thompson sampling, bayesian inference, generative models

Abstract:
Conventionally trained neural networks excel at prediction but often struggle to model uncertainty in their own predictions. We explore this challenge in a meta-learning bandit decision-making problem for news recommendations; this setting require decision-making algorithms to incorporate pretrained language models to process text data for the best performance. We present a scalable approach to Bayesian uncertainty quantification by posing it as a problem of autoregressive generative modeling of future rewards. First, we use historical data on previously released news articles to pre-train a generative model to predict sequences of future potential rewards. At inference time, our algorithm makes decisions based on limited previous rewards and autoregressively generated future rewards. Far from a heuristic, we synthesize insights from the literature to show our method is a novel implementation of Thompson (posterior) sampling, a prominent bandit algorithm. We prove our pretraining loss directly controls online decision-making performance, and we demonstrate our framework on a news recommendation task where we integrate end-to-end fine-tuning of a pretrained language model to process news article headline text to improve performance.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730571361063

thread_type:
official_review

<thread_object>
creator: reviewer_vlme

<content>
summary:
This paper studies exploration under general neural networks. They propose the method kind of like "generative Thompson Sampling", which does not explicitly model the unknown latent variables, but aims at modeling the missing rewards. The procedure of the algorithm starts with implicitly learns the Bayesian model via pre-training by minimizing a sequence prediction loss on successive reward using historical data, then at inference time, it uses the both the observed and imputed reward to make decisions. This basically reduces 
 the sequential decision-making problem to a sequence prediction training with low loss. Experiments are done on synthetic and news recommendation dataset.

strengths:
- This paper tackles the important issue of exploration in general neural networks, a challenge with broad implications for real-world applications like recommendation systems, education, and even LLM generation. Informed exploration are critical for these complex decision-making systems.
- The authors effectively bridge the gap between sequence modeling and Bayesian inference, introducing the concept of "generative Thompson sampling". This provides a strong theoretical justification for their approach and nicely clarifies the role of pre-training as a form of empirical Bayes.
- The proposed method is easy to implement, and the experimental results on recommendation tasks demonstrate the effectiveness of the proposed approach, in terms of both reducing regret and providing valid confidence intervals.

weaknesses:
- Lack of Clarity and Potential Inconsistencies: The paper would benefit from increased clarity in its methodological details and problem formulation. Specifically:

1.  While the motivating example of news recommendation emphasizes no overlap in the action set across time steps, this constraint seems not to be the case  in the formal problem definition and Algorithm 1.
2. The role of different users in Equation 2 and Algorithm 1 remains unclear. It appears to estimate a population-level mean reward for each action, over all users. This then seems a multi-arm bandit problem than the contextual bandit one? The authors should explicitly state whether they are addressing a multi-arm bandit or a contextual bandit problem and revise the relevant sections accordingly. It is hard to estimate the context-dependent reward based on the current alg.

- Computational Cost: The proposed method's computational complexity scales linearly with both the action set size and the "overlap" between new and historical actions, and time-step T. This could become prohibitive in applications with large action spaces or significant small overlap, even disjoint support. The authors should discuss more on the scaling of the method.

- Limited Novelty: The proposed approach is very similar to reward models [1] employed in the offline bandit and RL literature. While subtle differences may exist, the authors need to provide a more thorough comparison with existing reward modeling techniques, explicitly highlighting their novel contributions here.

Ref: https://arxiv.org/pdf/1103.4601

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730976428768

thread_type:
official_review

<thread_object>
creator: reviewer_uukg

<content>
summary:
This paper aims to address the challenge of modeling uncertainty in decision-making in news recommendation systems. The authors propose a method that leverages autoregressive generative models to predict the missing or future rewards based on the historical, and then use the Thompson sampling for final decision-making with the previous and generated rewards. This paper proposes several theory analyses of the advantages of using sequential models (like transformers). The authors demonstrate their approach through both synthetic and semi-realistic news recommendation experiments, showing that their method can effectively incorporate pretrained language models and achieve strong performance in terms of regret minimization and uncertainty quantification.

strengths:
1) The idea of incorporating sequential models into decision-making makes sense to me. Pre-trained on previous data, the autoregressive model has the ability to predict the missing reward values, which is benefit for the downstream algorithm.

2) The authors provide several theory analyses to support the efficiency of the proposed modules.

3) Experiments on both synthetic and real-world settings show the improvements of the proposed model.

weaknesses:
1) One of the main concerns comes from the experiments. The authors only test their models with the base models (such as PS Neural Linear, PS Beta-Bernoulli), which may reduce the convincingness of the proposed method.

2) The authors claim that they use the language foundation model to improve the uncertainty quantification, which is not precise. Because they only pre-trained sequential models on historical data, the trained models are not actually foundation models.

3) The Thompson sampling at the inference stage relies heavily upon the generated missing reward values, which may limit its application when sequence models cannot precisely predict missing rewards.

questions:
Please see the above Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731168433342

thread_type:
official_review

<thread_object>
creator: reviewer_gvpa

<content>
summary:
The paper explores uncertainty in neural networks for decision-making, focusing on a meta-learning bandit setup applied to news recommendation. It introduces a novel approach using autoregressive generative modeling to quantify Bayesian uncertainty by predicting future rewards based on historical data. This method involves pre-training a generative model to forecast potential rewards, which helps balance exploration and exploitation without directly modeling latent variables. The approach is demonstrated through theoretical bounds and experiments, showing it enhances decision-making accuracy by effectively integrating neural networks.

strengths:
* The paper presents a new method for Bayesian uncertainty quantification through autoregressive generative modeling, which provides a scalable and effective alternative to traditional methods that rely on complex posterior approximations.

* By applying this method to a news recommendation task, the paper demonstrates the approach's potential to improve real-world decision-making where uncertainty and user interactions play a crucial role.

* The paper provides a formal regret bound, showing that the approach’s decision-making performance directly relates to the sequence model’s training loss, solidifying the theoretical basis for the proposed method.

weaknesses:
* A major concern is that this paper lacks a thorough discussion of related work, despite several highly relevant and overlapping studies. There is existing research that uses generative modeling for posterior approximations【1】and in recommendation tasks【2】. Additionally, multiple works have explored connections between autoregressive generative modeling and approximate Bayesian inference【3,4】. Furthermore, meta-learning with autoregressive generative networks has been studied in contexts such as classification【5】, Bayesian Optimization (BO)【6】, and decision-making in sequential prediction【7】.


1. Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. PMLR, 2016.
2. Da Tsai, Yun, and Shou De Lin. "Fast online inference for nonlinear contextual bandit based on generative adversarial network." arXiv preprint arXiv:2202.08867 (2022).
3. Hollmann, Noah, et al. "Tabpfn: A transformer that solves small tabular classification problems in a second." arXiv preprint arXiv:2207.01848 (2022).
4. Müller, Samuel, et al. "Transformers Can Do Bayesian Inference." International Conference on Learning Representations.
5. Bonet, David, et al. "HyperFast: Instant Classification for Tabular Data." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.
6. Müller, Samuel, et al. "Pfns4bo: In-context learning for bayesian optimization." International Conference on Machine Learning. PMLR, 2023.
7. Lee, Jonathan, et al. "Supervised pretraining can learn in-context reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

questions:
* Could the authors clarify the differences between their approach and existing works, highlighting the novelty and contribution of this work?
* Could the authors clarify if the proposed methods are used to address the delayed feedback issue (missing outcomes) rather than used for building a recommendation policy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732111243064

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their feedback. As the reviewers had different questions and concerns, we address those in each response separately. In the meantime, we emphasize our contribution and its significance: 

**Our primary contribution is our scalable algorithm for decision-making under uncertainty.** The novelty and contribution in our work is in connecting sequential decision-making and generative sequence models for modeling unobserved rewards, in a theoretically grounded way. We have 
1. Formulated and instantiated a meta-bandit problem with rich (unstructured) arm-features that requires interfacing with foundation models, that is common in industry (e.g. recommendation systems) but that does not yet exist in the literature, 
2. Proposed a correct and novel implementation of Thompson sampling in this problem that uses auto-regressive generation of missing rewards to power exploration,
3. Demonstrated the effectiveness of this method on both synthetic and real-world data, and 
4. Provided a novel theorem that controls the quality of interactive decision-making (i.e. regret) by the quality of offline prediction (near-optimality in terms of next-token prediction loss).

This combination of contributions is unique to our work.
</content>
</thread_object>

</thread>


Metareview: