In-Context Learning for Pure Exploration
Authors: Alessio Russo
Ryan Welch
Aldo Pacchiano
Venue: ICLR 2026 Poster (2026)
Keywords: active sequential hypothesis testing
pure exploration
reinforcement learning
in-context learning
best arm identification
Primary area: reinforcement learning

Abstract: We study the _active sequential hypothesis testing_ problem, also known as _pure exploration_: given a new task, the learner _adaptively collects data_ from the environment to  efficiently determine an underlying correct hypothesis. A classical instance of this problem is the task of identifying the best arm in a multi-armed bandit problem (a.k.a. BAI, Best-Arm Identification), where actions index hypotheses. Another important case is generalized  search, a problem of determining the correct label through a sequence of strategically
selected queries that indirectly reveal information about the label.
In this work, we introduce _In-Context Pure Explorer_ (ICPE), which meta-trains Transformers to map _observation histories_ to _query actions_ and a _predicted hypothesis_, yielding a model that transfers in-context. At inference time,  ICPE actively gathers evidence on new tasks and infers the true hypothesis without parameter updates.
Across deterministic, stochastic, and structured benchmarks, including BAI  and generalized search,  ICPE is competitive with  adaptive baselines while requiring no explicit modeling of information structure. Our results support Transformers as practical architectures for _general sequential testing_.

Published as a conference paper at ICLR 2026
IN-CONTEXT LEARNING FOR PURE EXPLORATION
Alessio Russo" Boston University a Brusso2@bu.ess
Ryan Welch Stanford University r stannllcdeaa
Aldo Pacchiano
Boston University
Broad Institute of MIT and Harvard pacchian@bu. edu
ABSTRACT
We study the active sequential hypothesis testing problem, also known as pure exploration: given new task, the learner adaptively collects data from the environ- ment to efficiently determine an underlying correct hypothesis. A classical instance of this problem is the task of identifying the best in multi- bandit problem (a.k.a. BAI, Best-Arm Identification), where actions index hypotheses. Another important case generalized search, a problem of determining the correct label through sequence of strategically selected queries that indirectly reveal information about the label. In this work, we introduce In-Context Pure Explorer (ICPE), which meta-trains Transformers to map observation histories to query actions and predicted hypothesis, yielding model that transfers in-context, At inference time, ICPE actively gathers evidence on new tasks and infers the true hypothesis without parameter updates. Across deterministic, stochastic, and struc- tured benchmarks, including BAI and generalized search, ICPE is competitive with adaptive baselines while requiring no explicit modeling of information structure. Our results support Transformers as practical architectures for general sequential testing. Code repository https:/ /tte:://ggiiie gighhbb.bb.... www.rssaaessiiieoooooooooooooooe icpe.
INTRODUCTION
Sequential architectures have shown striking in-context learning (ICL) abilities: given a short sequence of examples, they can infer task structure and act without parameter updates (Lee et al., 2023; Schaul & Schmidhuber, 2010; Bengio et al., 1990). While this behavior is well documented for supervised input-output tasks, as well as regret minimization problems, many real problems demand sequential experiment design: how do we allocate experiments to reliably infer an hypothesis? For instance, imagine a librarian trying to figure out which book you want by asking series of questions. Similarly, in generalized search (Nowak, 2008), the learner adaptively chooses which tests to run, each partitioning the hypothesis class, to identify the true hypothesis as quickly as possible. This raises natural question: can we leverage ICL for adaptive data collection and hypothesis identification across a family of problems?
We study this question through the lens of Active Sequential Hypothesis Testing (ASHT) (Chernoff, 1992; Cohn et al., 1996). a.k.a. pure exloration (Degenne & Koolen, 2019). where an agent adaptively performs measurements in environment identify ground-truth hypothesis. In particular, study a Bayesian formulation of ASHT, where each environment is drawn from a family of possible problems M.
Classically, ASHT has been studied either (i) with a fixed confidence (i.e., stop as soon as the predicted hypothesis is correct with error probability most 6) (Jang et al., 2024) or (ii) a fixed sampling budget (use N samples to predict the correct hypothesis) (Atsidakou et al., 2022). For example, in the fixed-confidence setting one can use ASHT to minimize the number of DNA-based tests performed to accurately detect cancer (Gan et al., 2021). Another canonical instantiation is Best-Arm Identification (BAI) in stochastic multi-armed bandits (Audibert & Bubeck, 2010). this problem the agent sequentially selects action (the query) and observes noisy reward: the task is to identify the action with the highest mean reward', Other applications include medical diagnostics
"Equal contribution (alphabetical order).
Note that, in this particular case, the hypothesis space coincides with the query space of the agent.

---

Published as a conference paper at ICLR 2026
Exploration Phase
Infermmee Phase
DN
HN Ix(-PN)
(a)
(b)
Figure I: (a) Generalized search example: ICPE starts from a masked image (left), and sequentially reveals patches expected reduce the posterior entropy labels. It stops once the inferred label is 5-correct (right). (b) After executing an action a₁, the agent observes 2 At inference time, the data collected is used to infer hypothesis.
(Berry al., 2010), sensor management (Hero & Cochran, 2011) and recommender systems (Resnick & Varian, 1997).
Despite substantial progress (Ghosh, 1991; Naghshvar & Javidi, 2013; Naghshvar et al., 2012: Mukherjee et al., 2022), solving ASHT problems remains difficult. Even in simple tabular environ- ments, computing optimal sampling policies often requires strong modeling assumptions (known observation models that do not depend the history, and/or known inference rules) and solving challenging (often nonconvex) programs (Al Marjani et al., 2021). This leaves open whether one can learn, in simple way, both gather informative data and infer the correct hypothesis without such assumptions.
To answer this question, we introduce In-Context Pure Explorer (ICPE). Transformer-basee architecture meta-trained on a family of tasks to jointly learn a data-collection policy and an inference rule, in both fixed-confidence and fixed-budgef regimes. ICPE is model that transfers in-context: inference time, ICPE gathers evidence on new tasks and infers the true hypothesis without parameter updates (Schaul & Schmidhuber, 2010; Bengio al., 1990).
The practical implementation of ICPE emerges naturally from the theory alone, showing how principled nntrmatiionttheoreeee reward function can be used to train, using Reinforcement Learning (RL), an optimal data-collection policy. Additionally. ICPE relaxes classical assumptions: the data-generation mechanism unknown and may be history-dependent, and the mapping from data to hypotheses is also unknown (we do not assume known likelihood or a hand-designed test). These facts, combined with the simple and practical implementation of ICPE, offer a new way to design efficient ASHT methods in more general environments.
On BAI and generalized search tasks (deterministic, stochastic, structured), ICPE efficiently explores and achieves performance comparable to instance-dependent algorithms, while requiring only forward pass at test time, and without requiring solving any complex optimization problem.
2 PROBLEM SETTING
The problem we consider is as follows: on an environment instance M ~ P, sampled from prior over an environment class M, the learner chooses actions (queries") a; in rounds 1, and observes outcomes Z4+1- The aim is to gather a trajectory D, = (x) 01,.... (4-1, that informative enough identify an environment-specific ground-truth hypothesis H with high probability.
Informally, we seek to answer the following question:
Given an environment M drawn from a prior P, how can we learn (i) sampling policy that collects data D from M and (ii) an inference rule I such that I(D) reliably predicts H*?
Environments, sampling policy and hypotheses. We consider environments (M (X, p, P. H*) with observation space X, action set A, initial observation law p A and (possibly
generative mechanism P = (Pt) such that 2 ~ Pt-- az). All M E M share the same X' and The learner uses a (possibly randomized) policy π (
The reason why denote "queries" as "actions" stems from the fact that the prohlem can he modeled similarly Markov Decision Process (MtPP)Puterman, 2014), and queries correspond actions MDP.
2

---

Published as a conference paper at ICLR 2026
with dr Ie (-),,,,, and sequence of inference rules I hypothesis set H. We assume throughout that H* is induced by the environment via a measurable functional h", i..., H == h² (p. P), and is almost surely unique under P.
(1)) with L. Di - H for a finite
Example 2.1 (Best Arm Identification). In BAI an agent seeks to identify the best arm among K arms. Upon selecting an action time t, it observes a random reward 2, distributed according to distribution P(-(a)). The goal is to identify a² arg max, Pe>a((-a)) (so H* a²). Many algorithms exist for specific assumptions (Garivier & Kaufmann, 2016; Jedra Proutiere, 2020), but designs change drastically with the model, and extensions to richer settings can be difficult and often non-convex (Marjani & Proutiere, 2021).
Fixed confidence and fixed budget regimes. Two regimes usually considered in pure exploration:
Given target error level 5 E (0, 1), the learner chooses: (i) stopping time N that denotes the total number of queries and marks the random moment data collection stops, (ii) data-collection policy (iii) and an inference rule I that minimize the expected total number queries while meeting a correctness guarantee:
Fixed confidence:
inf
E [r]
s.t.
PP (L, HM) 2 1 1-d 6.
T_m.
where IPT(() denotes the probability of the underlying data collection process when π gathers data from M, and M is sampled from prior P.
Fixed budget: For given horizon N E N. the learner chooses и and I to maximize the chance of predicting the correct hypothesis after exactly N queries:
sup JP =([N))H")) x,1
These two objectives capture the main operational modes of pure exploration: "stop when certain" and "maximize accuracy over a fixed horizon". Further note that the problem we propose to solve extends classical ASHT by allowing P, a.). Standard formulations assume memoryless dependence only on (H" .a, aa)(Naghs & Javidi, 2013; Garivier & Kaufmann, 2016). Moreover, whereas ASHT/BAI typically known estimators (e.g., maximum likelihood), we learn the inference rule from data. Consequently, both the sampling policy æ and the inference rule can depend on entire histories.
history-dependent observation kernels: F++1
ICPE: IN IN-CONTXT PURE EXPLORER
In this section we describe ICPE, meta-RI. approach for solving eqs. (1) and (2). The implementa- tion of ICPE motivated from the theory. We first show that learning an optimal inference rule amounts computing a posterior distribution. Secondly, the policy can be learned using RL with an appropriate reward function.
Importantly, the reward function used for training म emerges naturally from the problem formulation, and it is not a user-chosen criterion, making it principled information-theorrtica reward function. We now describe the theory, and then describe the practical implementation of ICPE.
3.1 THEORETICAL RESULTS
Our theoretical results highlight that the main quantity of interest, in both regimes in eqs. (1) and (2), the posterior distribution over the true hypothesis PP H = H(D.). First, the optimal inference rule I* is based on this posterior. Secondly, this posterior naturally defines a reward function that can characterize the optimality of a data-collection policy.
Throughout this section, we assume that X' C IR is compact and A, 76 are finite. We instantiate M via a parametrized family {(Pw., (P..,p.) : w E 22 with Q compact and w + (Pu, Pw) continuous, so prior on Ω induces a prior on M. For the sake of brevity, we provide informal statements here, and refer the reader to app. B. 1 for all the details.
We have the following result about the optimality of the inference, proved in app. B.1.2.
Proposition 3.1 (Inference Rule Optimality). Let 2 1 and a policy π. The optimal inference rule to sup,, (H* (D.)) is given by (=) arg maxHeh P(H HDD: z).
3

---

Published as a conference paper at ICLR 2026
Concretely, prop. 3.1 identifies the optimal inference rule the maximum posteriori estimator based on PH = Di), now differentiate between the two settings.
) so that learning Is amounts to learning this posterior. Based on this we
Fixed budget. We begin with the simpler fixed budget case. The key idea is to show that the optimal policy '' maximizes action-vale value function Q (Sutton & Barto, 2018). First, define the following reward function: for < let ry (Dr) = O, and for = N set (D((D) = maxH P(H = N In words, we assign a reward equal to the maximum value of the posterior distribution at the last time step and 0 otherwise.
Then, define Vv(DN) (Dx) to be the optimal value the last timestep 1 definition, we can recursively define the Q-function as follows:
N. From this
Qe a) = (c ((te,
(Dz, a, (++1)) and Vi(D,) max Qr aEA
Vt N 1.
=D411
where "xie+1 (Drt, a)" denotes the posterior distribution of Ze+1 given a). Optimizing with respect this reward function yields an optimal solution to (2), which we formalize the following result proved in app. B.1.3.
Theorem 3.2 (Policy Optimality for Fixed Budget). For all 2 1, define the policy oं (D:) = arg a Qi (D, a). Then, (xt ,1x) (where Iyy is as in prop. 3.1) are an optimal solution of eg. (2), and we have that
sup P (IN(DN) =1) = [rx(DN)] I
Simply speaking, thm. 3.2 indicates that an optimal exploration policy in the fixed-budget setting is obtained by greedy policy with respect to a Q-function whose terminal reward is the maximum posterior mass VN (Dx) max P(H* H 'Dw) (and zero reward for all other timesteps). A similar principle also holds for the fixed confidence setting.
Fixed confidence. In the fixed confidence setting, we first simplify the problem by noting that the stopping time T can be simply embedded as a stopping action Ostop in the policy π (see app. B. 1.4 for formal justification). Hence, we extend the action set as A + -AU {astop] and T inf E N a, = Astop]- Then, as in classical ASHT literature (Naghshvar & Javidi, 2013), we study the dual problem of eq. (1), that is:
inf sup Vx( 7), where VA 7) = - [E + A (I+(D+) = - 1 6]. A20 x./
To show optimality of policy, and satisfaction of the correctness constraint, there are 2 key obser- vations to make: (1) can show that the optimal inference rule I* remains in prop. B.2; (2) solving eq. (4) amounts to solving an RL problem in π.
Indeed, similarly the the fixed budget setting, for t 2 define the reward model
Te. (Dr, a) - (aykoaop)) + Al (amastop max P(H" = H (D:), H
(5)
which simply penalizes the policy for each extra timestep, accompanied by a reward proportional to the maximum posterior value at the stopping time. Accordingly, we define the Q-function as
a, (D., (D,, (aptasup Er,, Il(D..a) max Q4+1. (.
(6)
Then, we have the following result (see app. 1.1.5 and app. B.1.6 for proof) indicating that optimizing with respect to this reward function yields an optimal solution to (1).
Theorem 3.3 (Policy Optimality for Fixed Confidence). Let TiA and 5 = (Tt))) Then, for X > 0 the pair (I*, mz)) with I" === (I;), defined prop. 3.1, is an optimal solution of sup,..1 Vx I). Furthermore, under suitable identifibillity conditions (see assum. 2), any maximizer X of eq. (4) guarantees that TīA. sutisfies the ô-correctness criterion.
arg max Qe, a)
Intuitively, for the fixed-confidence setting, first recast the constrained problem in eq. (1) via Lagrangian dual, then prove that any admissible stopping rule 7 can be represented as the selection time of an absorbing stopping action astop- In thm. 3.3, we show that the resulting dual problem is
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 ICPE (In-Context Pure Explorer)
I: Input: Tasks distribution P: confidence ó; horizon N; initial À and hyper-parameter Ta To-
// Training phase
2:
Initialize buffer B, networks Qa, and set 0
while Training not over
4: Sample environment M P with hypothesis H" observe #1 ~ and set + 1.
5: repeat
6:
Execute action a, arg max, Qo (D in M and observe 71+1- and set
Add partial trajectory (D,, ús; 1t++1 H*) until 01-1 = dstop (fixed confidence) or t N (fixed budget).
9:
In the fixed confidence, update À according to eq. (11).
10: Sample batch B ~ B and update 0, é using Lint (B; Φ) (eq. (7)) and Cpolicy((BB))))) (eq. (8) or eq. (9)). 0).
11: Every Ta steps set á (similarly, every Ta steps
12: end while
// Inference phase
13: Sample unknown environment M ~ P.
14: Collect trajectory Dy D, fixed confidence) according a policy TP())) arg max, Qo until f . (or a; = (stop).
15: Return fī arg m Ie (H) (or ft, arg max [D(H)D=) in the fixed confidence)
solved by greedy policy on the Q-function defined via the reward in eq. (5), and that such policy achieves the desired level of correctness, 6. This result establishes that both an optimal 6-aware stopping rule and exploration strategy can be learned on the extended action space U {dstop): J. In the next section, we describe the practical implementation of ICPE based on these results using the Transformer architecture.
3.2 PRACTICAL IMPLEMENTATION THE ICPE ALGORITHM
We instantiate ICPE with two learners: an inference network (H,,), parametrized by é, that approximates the posterior P H = H (D,) (cf. prop. 3.1) and a Q-network Qo a). parametrized by whose greedy policy defines my (and includes Ostop the fixed confidence setting only). Both networks are implemented using Transformer architectures, and, for practical reasons, we impose a maximum trajectory length of N. This architecture handles both fixed budget (eq. (2)) and fixed confidence (eq. (I)) settings. However, we find it important to explicitly note that while algorithm I abstracts the main ideas of ICPE unified way, in practice we train separate models for the fixed- fixedbbudget and fixed-confidence regimes, each with their own reward and Q-function derived in section 3.1.
Training phase. At training time ICPE interacts with online environment: each episode draws an instance P and generates a trajectory. We maintain a buffer B with tuples (Dt, at x++1, Hír). where H is the true hypothesis for the sampled environment M (from single tuple we also obtain Di+1 (Dr, at, 1)). This buffer is used to sample mini-batches B C B to train (0, ø). Lastly, we treat each optimization in (Ø, 0) (and À for the fixed confidence) separately, treating the other variables as fixed.
Training of Ig. We train Is learn the posterior by SGD on the negative log likelihood on a a batch B C B of partial trajectories sampled from the buffer:
Line (#)
Σ (D,,0
log (H
In expectation this is (up to an additive constant) equivalent to minimizing the KL-ddvergence between P(H HDD) and Ts (D) (a similar loss is also used in (Lee al., 2023)). Lastly, we also set f. arg max (D.)) to be predicted hypothesis with data D..
Training in the Fixed Budget. In the fixed budget train 0 using DQN (Mnih et al., 2015) and the rewards defined in the previous section. We denote the target network Qg. which is parameterized by 0. Since rewards defined in terms of 1g. improve training stability we introduce separate target inference network 1ž. parameterized by ő. which provides feedback for training 0. These target networks are periodically updated, setting @ @ every T steps (similarly, 0 every To steps).

---

Published as a conference paper at ICLR 2026
Hence, in the fixed budget, for batch B ~ B, we update by performing SGD the following loss
ppolicy (B:0)= (DL.
Σ
max ( 1 + max Qo(Di+1, Qu(De ai)
Training in the Fixed Confidence. In this setting we train & similarly to the fixed budget setting. However, we also have a dedicated stop-action astop whose value depends solely on history. Thus, its Q-value can be updated any time, allowing retrospective evaluation of stopping. In other words, QD (stop) can be updated for any sampled transition 2 E B, even the logged action a A astop (L..., a "pretend to stop" update). This allows the model to retro-actively evaluate the quality of stopping earlier a trajectory.
Then, based eq. (6), update by performing SGD on the following Q-loss
ppolicy (B;
Σ
(achonoo) (-1 max (ū(((t1,)) -Qo((,,)))) (9)
Qo((
[B]
(Dr.
(Amax [AmaxI5(H|pp)) - QD(D (stpp)
(10)
and note that the loss depends on We learn À using gradient descent update, which depends on the correctness of the predicted hypothesis. We sample K trajectories {(D!), H;) with fixed (0, Φ) and update Å with a small learning rate 8:
À - max (0, A - 1 8)1, where P
1 K
Σ1 (arg max =
(11)
The quantity أ can be used to assess when to stop training by checking its empirical convergence. In the fixed confidence, in practice we can stop whenever P 2 1 - 5 is stable and À is almost constant. However, obtain rigorous guarantees care must be taken. app. 1. discuss how provide formal guarantees on the 8-correctness of the resulting method, bsaed on sequential testing procedure.
Inference phase. At inference time ICPE operates by simple forwards passes. An unknown task M P is sampled, and actions are selected according to a = arg max, (D. a). At the last timestep a hypothesis is predicted using HN = arg max I (H)))) (or A, = arg max,, Is (H)D) DD)) at the stopping time for the fixed confidence setting).
Theoretical guarantees and training correctness. In prop. B. we describe how to decide when to stop training in order to guarantee that the resulting (mo, L&) are d-correct. Furthermore, derive finite-sample guarantees for the fixed-budget ICPE meta-learning phase in a stylized setting app. B.2, In thm. B. we derive a bound on the sub-optimality of the policy =k) at training epoch k in terms of stage-wiss wise Bellman residuals and concentrability coefficients. In thm. B.15, we additionally show how these residuals are controlled by an approximation term (capturing how well the function class can represent the Bellman update) and estimation term that decays with the number and size of training batches. Together, these results yield an explicit finite-sample performance bound for ICPE in ideal scenario.
EMPIRICAL EVALUATION
We evaluate ICPE on range of tasks: BAI on bandit problems, hypothesis testing in MDPs, and general search problems (pixel sampling and binary search). For bandits, we consider different reward structures: deterministic, stochastic, with feedback graphs and with hidden information. Due to space limitations, refer to app. for the results on bandit problem with feedback graphs and MDPs. Also refer to app. for details on the algorithms. In all experiments we target accuracy value of = 0.1, and shaded areas indicate 95% confidence intervals computed via hierarchical bootstrapping (see app. D for details).

---

Published as a conference paper at ICLR 2026
4.1 BANDIT PROBLEMS
We now apply ICPE to the classical BAI problem within MAB tasks. For the MAB setting we have corresponding to the actions in the MAB problem M. For
finite number of actions A = f1....,A .....,,, } each action a, we define corresponding reward distribution P(-la) from which rewards are sampled i.i.d. Then, P is a prior distribution on the actions' rewards distributions. For the BAI problem, we let the true hypothesis be H* arg max, Be- [a]] so that the goal is to identify the best action (and thus H = A).
Stochastic Bandit Problems. We evaluate ICPE on stochastic bandit environments for both the fixed confidence and fixed budget setting (with N = 30). Each action's 's reward distribution is normally distributed V4 (pa 0.5²), with (Ha)) drawn from P. In this case P uniform distribution over problems with minimum gap max, Pa maXpy4a Ha 2 Ao. with Ao = 0.4. Hence, an algorithm could exploit this property to infer H more quickly. For this case, we also derive some sample complexity bounds in app. B.
TaS
Stopping
co
TTPS
SPPT
6.3
T45
1CEE
0.2 Dvbym
T1P5
Number of Actions
Time
Number of Actions
(b)
(c)
Figure 2: Results for stochastic MABs with fixed confidence 8 = 0.1 and N 100: (a) average stopping time r; (b) survival function of (c) probability of correctness 1 (Hr H
We compare against pure exploration baselines: TaS (Track-un (Track-and-Soc) and-Stop) (Garivier & Kaufmann, 2016) and TTPS (Top-TToo) (Russo et al., 2018), which are principled choices for hypothesis testing (asymptotically optimal or close to optimal allocations that target the most confusable hypotheses). We also include an ablation, "T-DPT", which uses our learned inference I (,) as in DPT (Lee et al., 2023) and acts greedily with respect to the posterior (and a simple confidence-threshold stop); this isolates the value of learning query policy versus relying on posterior-driven greedy control. Details for I-DPT are in app. C.
In fig. 2 are reported results for the fixed confidence. In fig. 2a see how ICPE is able find an efficient strategy compared to other techniques. Interestingly, also I-PPT seems to achieve relatively small sample complexities. However, the tail distribution of its is rather large compared ICPE (fig. 2b) and the correctness is smaller than 5 for large values of K. Methods like TaS and TTPS achieve larger sample complexity, but also larger correctness values (fig. 2c). This is well known fact: theoretically-somm stopping rules, such as the ones used by TaS and TTPS, tend to be are overly conservative (Garivier & Kaufmann, 2016).
Lastly, we verified the robustness of ICPE to distribution shifts. We trained ICPE in the stochastic fixed-confidence bandit setting as described above, and then evaluated the trained model on bandit instances drawn from shifted environment distributions. We report the results in app. D.1.2. Across all experiments, we observed that both correctness and stopping time remain remarkably stable, with only minor fluctuations within the reported confidence intervals. This suggests that ICPE is not excessively sensitive to moderate shifts in the environment distribution around the training family.
Finally, for the sake of space, we refer the reader to app. D.1.2 for the results in the fixed budget setting.
Deterministic Bandits. We also evaluated ICPE in deterministic bandit environments with fixed budget K, equal to the number of actions. Thus, ICPE needs to learn to select each action only once to determine the optimal action. Since the rewards are deterministic, we cannot compare to classical BAI methods, which are tailored for stochastic environments. Instead, we compare to: (i) uniform policy that uses maximum likelihood estimator estimate the best arm; (ii) DQN (Mnih et al., 2013), which uses D, as the state, and trains an network to infer the true hypothesis; (iii) and T-DPT, acts greedily with respect the posterior of I as in DPT. Figure
7

---

Published as a conference paper at ICLR 2026
DQN Unform
Poosabity of Carmatriee
0.8
0.4
Fraction of
0.4
Number
Actione
Number Actiona
Figure 3: Deterministic bandits: (left) probability of correctly identifying the best action K; (right) average fraction of unique actions selected during exploration vs. K.
reports the results: ICPE consistently identifies optimal actions (correctness 1) and learns optimal sampling strategies (fraction of unique actions x 1). Without being explicitly instructed to "choose each action exactly once", ICPE discovers on its own that sampling every action exactly what yields enough information to identify the best. While the optimal exploration strategy in this setting intuitive, baseline performance degrades sharply as the number of actions grows, illustrating that existing exploration methods can fail even in such simple environments.
Bandit Problems with Hidden Information. evaluate ICPE in structured settings, we introduce bandit environments with latent informational dependencies, termed magic actions. In the single magic action case, the magic action am 's reward is distributed according to N( on)). where Om (0, 1) and aam, (arg a (a) encodes information about the optimal action's identity through an invertible mapping & that is unknown to the learner. The index an is fixed, and the sampled from P, uniform distribution over models guaranteeing that am. as defined above, is not optimal (see apps. B.5 and D.1.3 for more details). Then, define the reward distribution of the non-magic actions M(uas (1 o )
mean rewards of the other actions (Ja)aaaa,,
20.2
11.3
15.0
Beaell Dhraeers R.1g
Reast (fleww K.1))
Cunulatvv Rugrett Rugret
15.
17.5
UEB
Stoophng
30.0
00.0
7.5
2.5
5.0
0.8
2.5
0.8
06 Standard Deviarion Magic Action
Number Magic Actions
Time
(a)
(b)
(c)
Figure 4: (a) Single magic action: average stopping time and the theoretical lower bound across varying om- (b) Magic chain: average stopping time between ICPE, T-IDS vs. number of magic regret minimization task, with om 0.1.
actions. (c) ICPE
In our first experiment, we vary the standard deviation o. [0, 1]. This problem isolates whether ICPE can detect and exploit latent informational dependencies (via single diagnostic action that encodes the optimal arm) and balance sampling across action based on varying uncertainty levels.
Regarding the baselines, applying classical baselines (e.g., TaS) here is nontrivial: the magic action is coupled to the optimal arm via an unknown map é, which would need to be encoded as inductive bias. Instead, we compare ICPE to "I-IDS", which is standard pure exploration IDS (Russo & Van Roy, 2018) instantiated on top of ICPE's trained inference Is for exploiting the magic action.
We evaluate in fixed-confidence setting with emror rate 8 0.1. Figure 4a compares ICPE's sample complexity against a theoretical lower bound (see app. B). ICPE achieves sample complexities close to the theoretical bound across all tested noise levels, consistently outperforming I-IDS.
Additionally, in fig. 4c we evaluate ICPE in a cumulative regret minimization setting, despite not being explicitly optimized for regret minimization. At the stopping ICPE commits to the identified best action (i.e., explore-then-commee algorithms such as UCB, Thompson Sampling, and standard IDS initialized with Gaussian priors.
strategy). As shown in the results, ICPE outperforms classic
8

---

Published as a conference paper at ICLR 2026
ICPE
Beep EMAR
torm
Agent
Accuracy
Arg. Regions Used
ICPE
0.03
Deep CMAB
0.66 1 0.04
7.90 1. 0.99
Uniform
0.04
(b)
(a)
Figure 5: MNIST pixel-sampling task: (a) A chord between two digits indicates that their distributions not significantly different (alvelue 0.05, based pairwise chi-squared test), with thicker chords representing higher p-values; (b) accuracy and performance (mean + 95% CI)
To further challenge ICPE, we introduce multi-layered "magic chain" bandit environment, where there is sequence of n magic actions A... == {ai,..... Us,, C A such that Mai, = * ), and Moin (arg ga Pa). The first index i known, and by following the chain, an agent can uncover the best action in n steps. However, the optimal sample complexity depends on the ratio of magic actions non-magic arms. Varying the number of magic actions from I to environment, Figure 4b demonstrates ICPE's empirical performance, outperforming I-IDS.
10-actions
4.2 GENERAL SEARCH PROBLEMS: PIXEL SAMPLING AND PROBABILISTIC BINARY SEARCH
We now evaluate the applicability of ICPE to general search problems, including structured real-world examples.
Pixel sampling as generalized search. We introduce a classification task inspired by active per- ception settings. We consider the MNIST images (LeCun al., 1998), each partitioned into of 36 distinct pixel patches, corresponding to the query space A = f1.....300 36). The agent starts from blank (masked) image and, patch by patch, reveals pixels to quickly discover "what the image is about." After choosing a query accumulates partially observed image. After a budget HN E (0.....9). 9}.
AA the agent observes (the revealed patch) and
12, the agent outputs the predicted digit
For this setting we consider slight variation of ICPE that may be of interest: we consider an inference net I that a pre-trained classifier, trained on fully revealed images from P. Using this network, benchmark ICPE against two baselines: standard uniform random sampling and Deep Contextual Multi-Armed Bandit (Deep CMAB) (Collier & Llorens, 2018), which employs Bayesian neural networks to sample from a posterior distribution (Deep CMAB uses rewards the correctness probabilities computed by I). Importantly, we cannot compare to methods such as DPT since A * H. the hypothesis space is different from the query space.
Table 5b reports the classification accuracy and number of regions sampled. ICPE achieves sub- stantially better performance than both baselines using fewer regions. However, analyze whether ICPE learns a sampling strategy that adapts to the context of the task, we compare region selection distributions across digit classes using pairwise chi-squared tests. ICPE exhibits significantly more variation across classes than either baseline, visualized in Figure 5a. This suggests ICPE adapts its exploration to condiiooni structure, rather than applying a generic sampling policy.
K
Min Accuracy
Mean Stop Time
Max Stop Time
log, K
8
1.00)
2.13 + 0.12
3
16
1.00
2.93 0.12
32
1.00
3.71 + 0.15
5
1.00
4.50 0.21
128
1.00
5.49 0.23
256
1.00
6.61 + 0.26
8
Table 1: ICPE performance on the binary search task as K increases.
Probabilistic binary search. We also evaluated ICPE's capabilities to autonomously meta-learn binary search. We define action space of A {1,.... K}. with H* A. Pulling above or below H" yields a observation x, -1 or I, = +1, respectively, providing directional feedback. In tab. report results 100 held-out tasks per setting. ICPE consistently achieves perfect

---

Published as a conference paper at ICLR 2026
accuracy with worst-case stopping times that match the optimal (K) rate, demonstrating that it has successfully learned binary search. While simple, this task illustrates ICPE's broader potential to learn efficient search strategies in domains where no hand-designed algorithm is available.
5 DISCUSSION AND CONCLUSIONS
Our results position ICPE within a broader line of work on active sequential hypothexis test- ing (Naghshvar& & Javidi, 2013) and its close ties to exploration in RL (Sutton & Barto, 2018). Regarding exploration, note that classical regret- reggretminimizatiinn methods, including UCB variants (Auer al., 2002), posterior sampling (Osband et al., 2013; Russo & Van Roy, 2014), and regret-focused IDS (Russo et al., 2018), optimize long-run reward, not hypothesis identification. On the other hand, pure-exploration formulations in BAT (Audibert & Bubeck, 2010) yield sharp, instance-dependent procedures for hypothesis testing in fixedccotidence regimes (c.g., Track-and-Stop, Garivier & Kaufmann, 2016). However, these approaches assume to know the problem structure, which is not always possible if the user is not aware of such structure. Furthermore, computing an optimal data-collection policy remains challenge in more general scenarios (Al Marjani et al., 2021), and we discuss some of these challenges in app. C.3.1.
ICPE uses Transformers (Vaswani et al., 2017) to learn, in-context, data collection policy and and inference rule. Transformers have demonstrated remarkable in-context learning capabilities (Brown et al., 2020; Garg et al., 2022). In-context learning (Moeini et al., 2025) is a form of meta-RL (Beck et al., 2023), where agents can solve new tasks without updating any parameters by simply conditioning on histories. Building on this approach, Transformers can mimic posterior sampling from offline data, as in DPT (Lee al., 2023), or perform return- condiitnniiionnn for regret minimization (e.g., ICEE Dai et al., 2024). However, these approaches primarily target cumulative reward and typically lack learned, 6-aware stopping rule; applying them to hypothesis testing would require altering objectives, data-collection protocol, and add stopping semantics. Moreover, in generalized search where A + H. additional modeling is needed to map hypotheses to actions.
ICPE addresses these gaps by learning to acquire information in-context. ICPE targets pure exploration for identification: it splits inference and control, using supervised inference network to provide task-relevant information signals, while an RL-trained Transformer learns acquisition policies that maximize information gain. This separation makes it possible to exploit rich, non-tabular structures that are difficult to encode in hand-designed tests or confidence bounds.
Empirically, ICPE is competitive on unstructured bandits and extends naturally to structured and deterministic settings. The results on the MNIST dataset highlights key strength: ICPE adapts sampling to the class-conditional structure. More broadly, ICPE suggests a path for data-driven generalized search.
Limitations point to concrete avenues for future work. First, scaling to continuous or combinatorial hypothesis spaces to deal with more general scenarios is an important direction. However, such extensions require substantial further theoretical development, as rigorous formalisms for continuous hypothesis-testing frameworks remain an active area of research, even in classical pure-exploration settings (see, e.g., (Garivier & Kaufmann, 2021)). Second, extending ICPE to offline datasets is also promising research direction. When offline data be used construct reliable simulator, ICPE can already be applied directly. Moreover, even without such a simulator, ICPE could in principle be meta-trained purely from logged data using offline RL methods (e.g.. IQL, CQL), and a systematic study of this offline regime is an important question for future work. Third, while the main focus of this work to introduce and analyze ICPE as general framework that can address a broad family of pure exploration problems, and we validate it on numerous BAI and active search tasks, we view real-worke experiments as a natural next step. ICPE holds the promise to discover novel exploration and search algorithms in complex domains that do not offer concrete way of finding an optimal solution a priori, such as determining efficient sequences of proteins to test in a lab (Amin et al., 2024), minimizing the number of tests required to detect cancer (Gan et al., 202 I), and expediting the design of materials with desired properties (Talapatra et al., 2018). 8). In sum, we believe ICPE advances pure exploration by leveraging in-context learning to discover aask-adaetive acquisition strategics, and it opens a route toward unifying classical sequential testing with learned, structure-awan search policies that scale to real problems.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our results. All model architectures, optimization procedures, and hyperparameters are described in detail in the paper (see Sections 2-3 and Appendix C-D). Experiments were conducted using Python 3.10.12 and standard libraries including NumPy, SciPy, PyTorch, Pandas, Seaborn, Matplotlib, CVXPY, and Gurobi.
To facilitate replication,
full source code at https:/ / /gi thub. com/ rssalessio/ /cpe under the MIT license. The code contains (i) implementations of ICPE and all baselines, (ii) configuration files specifying the hyperparameters for each experiment, and (iii) detailed instructions in the README. file for installing dependencies and running all experiments. Running the provided scripts will reproduce the main results reported in the paper, including bandit, MDP, and generalized search benchmarks.
provide
ACKNOWLEDGMENTS
The authors are pleased to acknowledge that the computational work reported on in this paper was performed on the Shared Computing Cluster administered by Boston University's Research Computing Services and computing resources from the Laboratory for Information and Decision Systems at MIT. R. W. was supported by Master of Engineering fellowship by the Eric and Wendy Schmidt Center the Broad Institute.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08771 2023.
Aymen Al Marjani, Aurélien Garivier, and Alexandre Proutiere. Navigating to the best policy in markov decision processes. Advances Neural Information Processing Systems, 34:25852-2555 25864, 2021.
Alan Nawzad Amin, Nate Gruver, Yilun Kuang, Lily Li, Hunter Elliott, Calvin McCarter, Aniruddh Raghu, Peyton Greenside, and Andrew Gordon Wilson. Bayesian optimization of antibodies informed by a generative model of evolving sequences. arXiv preprint arXiv:2412.0776. 2024.
Kaito Ariu, Po-An Wang, Alexandre Proutiere, and Kenshi Abe. Policy testing in markov decision processes. arXiv preprint arXiv:2505. 2025.
Dilip Arumugam and Thomas L Griffiths. Toward efficient exploration by large language model agents. arXiv preprint arXiv:2504.2099) 20997, 2025.
Alexia Atsidakou, Sumeet Katariya, Sujay Sanghavi, and Branislav Kveton. Bayesian fixed-budget best-arm identification. arXiv preprint (tti:22//100877: 2022.
Jean-Yves Audibert and Sébastien Bubeck. Best arm identification multi-armes bandits. In COLT-23th Conference on learning theory-2010, pp. 13-p. 2010.
Peter Auer. Using confidence bounds for exploitation-exxloration trade-offs. Journal of Machine Learning Research, 30N44):397-422, 2002.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-255 2002.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 21, 2008
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcemmnt learning. arXiv preprini arXiv:2301.08026 2023.
11

---

Published as a conference paper at ICLR 2026
Gowtham Bellala, Suresh Bhavnani, and Clayton Scott. Extensions of generalized binary search to group identification and exponential costs. Advances in Neural Information Processing Systems, 23, 2010.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning synaptic learning rule. Citeseer, 1990.
Scott M Berry, Bradley Carlin, J Jack Lee, and Peter Muller, Bayesian adaptive methods for clinical trials. CRC press, 2010.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL htttps://wee wandb. com/. Software available from wandb.com.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in finitely- armed and continuous- armed bandits. Theoretical Computer Science, 412(199:1832-1888 1852, 2011.
Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, pp. 1516-1541, 2013.
Emil Carlsson, Debabrota Basu, Fredrik Johansson, and Devdatt Dubhashi. Pure exploration in bandits with linear constraints. In International Conference on Artificial Intelligence and Statistics, pp. 334-342. PMLR, 2024.
Fabio Cecchi and Nidhi Hegde. Adaptive active hypothesis testing under limited information. Advances in Neural Information Processing Systems, 30, 2017.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097, 2021.
Xi Chen, Quanquan Liu, and Yining Wang. Active learning for contextual search with binary feedback. Management Science, (9(4):2165-218) 2023.
Herman Chernoff. Sequential design of experiments. The Annals of Mathematical Statistics, 30(3): 755-770, 1959.
Herman Chernoff. Sequential design of experiments. Springer, 1992.
Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta- in-context learning large language models. Advances in Neural Information Processing Systems, 2023.
David A Cohn, Zoubin Ghahramani, and Michael Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:12 1996.
Mark Collier and Hector Urdiales Llorens. Deep contextual multi-armed bandits. arXiv preprint arXiv:1807.09809 2018
Zhenwen Dai, Federico Tomasi, and Sina Ghiassian. In-context exploration-exxloitaiio for rein- forcement learning. arXiv preprint arXiv:2403.0688 2024.
Sanjoy Dasgupta. Analysis of greedy active learning strategy. In Advances in Neural Information Processing Systems, volume 17. 2004.
Rémy Degenne and Wouter M Koolen. Pure exploration with multiple correct answers. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2026
Rémy Degenne, Wouter M Koolen, and Pierre Ménard. Non-asympiotic pure exploration by solving games. Advances in Neural Information Processing Systems, 32, 2019.
Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI²; Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.0277,, 2016.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint 12X15221...0755, 2021.
Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
Kyra Gan, Su Jia, and Andrew Li. Greedy approximation algorithms for active sequential hypothesis testing, Advances in Neural Information Processing Systems, 34:5012-5024, 2021.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-305988, 2022.
Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory, pp. 998-1027. PMLR,, 2016.
Aurélien Garivier and Emilie Kaufmann. Nonasymptotic sequential tests for overlapping hypotheses applied near-optimal arm identification in bandit models. Sequential Analysis, 40(1):61-96, 2021.
Bashkar K Ghosh. brief history of sequential analysis. Handbook of sequential analysis, 1, 1991.
Debamita Ghosh, Manjesh Kumar Hanawal, and Nikola Zlatanov. Fixed budget best arm identification in unimodal bandits. Transactions on Machine Learning Reseurch, 2024. ISSN 2835-8856.
Daniel Golovin and Andreas Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal of Artificial Intelligence Research, 42:427-4866 2011.
Daniel Golovin, Andreas Krause, and Debajyoti Ray. Near-optimal bayesian active learning with noisy observations. Advances in Neural Information Processing Systems, 23, 2010.
Aditya Gopalan, Shie Mannor, and Yishay Mansour. Thompson sampling for complex online
problems. In International conference machine learning, pp. 100-108. PMLR, 2014.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024. URL https: /www.
gurobi. com.
László Györfi, Michael Kohler, Adam Krzyžak, and Harro Walk. Distribution-Fre Theory of Nonparametric Regression. Springer Series in Statistics. Springer, New York, NY, 2002. ISBN 978- 0-387-95441-7 978-0-387-222422-- com/ 10n/100.1 1007/b97848.
doi: 10. 1007/697848. URL http::/limn springer.
A Hantoute and MA López. Characterizations of the subdifferential of the supremum of convex functions. Journal of Comvex Analysis, -588,18858, 2008
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825) September 2020. doi: 10.1038//41586-000 224-224--2 URL https: //ooi org 038 58 0200 2200-266--22
Kevin Sheppard, Tyler Reddy,
13

---

Published as a conference paper at ICLR 2026
Keegan Harris and Aleksandrs Slivkins. Should you your large language model to explore exploit? arXiv preprint arxiv:2002.00225, 2025.
Alfred O. Hero and Douglas Cochran. Sensor management: Past, present, and future. IEEE Sensors Journal, 11(2))3064-3075, 201 doi: 1091/S9/SN 20112117944
John Hunter. Matplotlib: A 2d graphies environment. Computing in science engineering, 9()): 90-95, 2007.
Kyoungseok Jang, Junpei Komiyama, and Kazutoshi Yamazaki. Fixed confidence best arm identifica- tion in the bayesian setting. Advances in Neural Information Processing Systems, 37, 2024.
Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification linear bandits. Advances in Neural Information Processing Systems, 33: 13007-10011,, 2020.
Marc Jourdan, Rémy Degenne, Dorian Baudry, Rianne de Heide, and Emilie Kaufmann. Top two algorithms revisited. Advances in Neural Information Processing Systems, 35:26791 15:26791-26803, 2022.
Jerome Kagan. Motives and development. Journal of personality and social psychology. 22(1):51, 1972.
Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in aumei-armedd bandits. In International conference machine learning, pp. 1238-1244 1246. PMLR, 2013.
PN Karthik, Vincent YF Tan, Arpan Mukherjee, and Ali Tajer. Optimal best arm identification with fixed confidence in restless bandits. IEEE Transactions Information Theory, 70(10))7349-7384, 2024.
Emilic Kaufmann and Wouter M Koolen. Mixture martingales revisited with applications sequential tests and confidence intervals. Journal of Machine Learning Research, 22(246);1 2021.
Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In International conference on algorithmic learning theory, pp. 199-213. Springer, 2012.
Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. The Journal of Machine Learning Research, 17(1):1-42, 2016.
Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In Advances in Neural Information Processing Systems, volume 37, 2024.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint 191.v:1912.13465 2019.
Tor Lattimore and Marcus Hutter. Pac bounds for discounted mdps. In Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23, pp. 334. Springer, 2012.
Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Finite- Finite-sample analysis of least- squares policy iteration. Journal of Machine Learning Research, (3498))3041-3774 2012. URL http://jmm jmlr. orgg hrggppperrsssaee vv33/ /013//azzzr aaaariiallll..... aa. html.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228-2322, 1998.
Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn co-tentext reinforcement learning. Advances in Neural Information Processing Systems, 36:43057-43083, 2023.
Dennis V Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 2764))98-11055 1956.
14

---

Published as a conference paper at ICLR 2026
John Lisman, György Buzsáki, Howard Eichenbaum, Lynn Nadel, Charan Ranganath, and A David Redish. Viewpoints: how the hippocampus contributes to memory, navigation and cognition. Nature neuroscience, (00(1)):14441460 2017.
Grace Liu, Michael Tang, and Benjamin Eysenbach. single goal is all you need: Skills and exploration emerge from contrastive rl without rewards, demonstrations, or subgoals. arXiv preprint arXiv:2408.0588 2024.
Shie Mannor and Ohad Shamir. From bandits to experts: On the value of side-observations. In Advances in Neural Information Processing Systems, volume 24, 2011.
Aymen Al Marjani and Alexandre Proutiere. Adaptive sampling for best policy identification in markov decision processes. In International Conference on Machine Learning, pp. 7459-7468. PMLR, 2021.
Aymen Al Marjani, Aurélien Garivier, and Alexandre Proutiere. Navigating the best policy in markov decision processes. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Wes McKinney al. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, volume 445, pp. 51-56. Austin, TX, 2010.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-333, 2015.
Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang. A survey of in-context reinforcement learning, arXiv preprint arXiv:2502.07997, 2025.
Giovanni Monea, Antoine Bosselut, Kianté Brantley, and Yoav Artzi. Llms are in-context reinforce- ment learners. arXiv preprint arXiv:2410.05362, 2024.
Laurel S Morris, Mora M Grehl, Sarah B Rutter, Marishka Mchta, and Margaret L Westwater. On what motivates us: a detailed review of intrinsic v. extrinsic motivation. Psychological medicine, 52(10):180111866, 2022.
Subhojyoti Mukherjee, Ardhendu S Tripathy, and Robert Nowak. Chernoff sampling for active testing and extension to active regression. In International Conference on Artificial Intelligence and Statistics, pp. 7384-7432. PMLR, 2022.
Lynn Nadel. The hippocampus and space revisited. Hippocampus, 1(3):221-229, 1991.
Lynn Nadel and Mary A Peterson. The hippocampus: part an interactive posterior representational system spanning perceptual and memorial systems. Journal Experimental Psychology: General, 142(4): 2013.
Mohammad Naghshvar and Tara Javidi. Active sequential hypothesis testing. The Annals of Statistics, 41(6):2703-2738, 2013.
Mohammad Naghshvar, Tara Javidi, and Kamalika Chaudhuri. Noisy bayesian active learning. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1626 1633. IEEE, 2012.
Nicolas Nguyen, Imad Aouali, András György, and Claire Vernade. Prior-Dependent Allocations for Bayesian Fixed- Budget Best-Arm Identification in Structured Bandits. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, pp. 379-387. PMLR, April 2025.
Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed Chi, Quoe V Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint 1r.iv:24100062.56 2024.
15

---

Published as a conference paper at ICLR 2026
Robert Nowak. Generalized binary search. In 2008 46th annual Allerton conference communica-- tion, control, and computing. pp. 568-574. IEEE, 2008
Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070, 2020.
John O' O'keefe and Lynn Nadel. Précis of o' o'keefe & nadel's the hippocampus as a cognitive map. Behavioral and Brain Sciences, 2(4))487-494, 1979.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems (NeurIPS), 26, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perfie perffreefrrmece deep learning library. Advances Neural Information Processing Systems (NeurIPS), 32, 2019.
Riccardo Poiani, Marc Jourdan, Emilie Kaufmann, and Rémy Degenne. Best-Arm Identification in Unimodal Bandits. In Proceedings of The 28th International Conference on Artificial Intelligence and Stafistics, pp. 2233-2241, PMLR, April 2025.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith. experimental design. Statistical Science, 3901):000-114, 2024.
Modern bayesian
Paul Resnick and Hal R Varian. Recommender systems. Communications of the ACM, 40(3):56-58. 1997.
Chloé Rouyer, Dirk van der Hoeven, Nicolò Cesa-Bianchi Bianchi, and Yevgeny Seldin. near-optimal best-of-both-worldd algorithm for online learning with feedback graphs. Advances in Neural Information Processing Systems, 35:35035-350444 2022.
Alessio Russo and Aldo Pacchiano. Adaptive Exploration for Multi-Reward Multi-Policy Evaluation. In Proceedings of the 42nd International Conference Machine Learning, pp. 52382-52421.
PMLR, October 2025.
Alessio Russo and Alexandre Proutiere. Model-free active exploration in reinforcement learning.
Advances in Neural Information Processing Systems, 544554440-54533, 2023a.
Alessio Russo and Alexandre Proutiere. On the sample complexity of representation learning in multi-task bandits with global and local structure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9658-9667, 2023b.
Alessio Russo and Filippo Vannella. Fair best identification with fixed confidence. In 2024 IEEE 63rd conference on decision and control (CDC), pp. 1173-1180, 2024. doi: 10.1109/CDDC56224 4024.10886570
Alessio Russo and Filippo Vannella. Multi-reward best policy identification. Advances in Neural Information Processing Systems, 37: 37:105533-05566,, 2025.
Alessio Russo, Yichen Song, and Aldo Pacchiano. Pure exploration with feedback graphs. In Pro- ceedings of The 28th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine I Learning Research. PMLR, 2025.
Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 19(4)::2211122)) 2014.
Daniel Russo and Benjamin Van Roy. Learning optimize via information-directed sampling. Operations Research, 66(1):230 2018.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018.
16

---

Published as a conference paper at ICLR 2026
Tom Schaul and Jürgen Schmidhuber. Metalearning. Scholarpedia, 5(6):4650, 2010.
Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, and Matthieu Geist. Approximate modified policy iteration. In Proceedings of the 29th international coference on international conference on machine learning, pp. 1889-1896, 2012.
Apurv Shukla and Debabrota Basu. Preference-basee pure exploration. Advances in Neural Information Processing Systems, volume 37, pp. 17313-17347, 2024.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaskowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. CoRR, abs/1912.028877, 2019. URL
http:/ / rxiv. org/ harg/abs///ee aww./////11 02877.
Jiahang Sun, Zhiyong Wang, Runhan Yang, Chenjun Xiao, John Lui, and Zhongxiang Dai. Large language model-enhanced multi-armed bandits. arXiv preprint arXiv:2502.01118, 2025.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Anjana Talapatra, Shahin Boluki, Thien Duong, Xiaoning Qian, Edward Dougherty, and Raymundo Arróyave. Autonomous efficient experiment design for materials discovery with bayesian model averaging. Physical Review Materials, 2(11): 133003, 2018.
Adrienne Tuynman, Rémy Degenne, and Emilie Kaufmann. Finding good policies in average- reward Markov Decision Processes without prior knowledge. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 109948-109979. Curran Associates, Inc., 2024, doi: 10.52202/ 079017-3489.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde Informatica Amsterdam, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Îlhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi:
Po-An Wang, Ruo-Chun Tzeng, and Alexandre Proutiere. Best arm identification with fixed budget: A large deviation perspective. Advances in Neural Information Processing Systems, 36: 18116664161115 2023.
Michael Waskom, Olga Botvinnik, Drew O'Kane, Paul Hobson, Saulius Lukauskas, David C Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Julian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesbeck, Antony Lee, and Adel Qalieh. mwaskom/ssaboon:: v0.8.1 (september 2017), September 2017.
Alice X Zheng, Irina Rish, and Alina Beygelzimer. Efficient test selection in active diagnosis via entropy approximation. In Conference on Uncertainty Artificial Intelligence, 2005.
17


Reviews:
[{"review_id": "UuJL0PqYea", "reviewer": "Reviewer_J4j4", "summary": "The paper introduces In-Context Pure Exploration, a Transformer-based meta-RL framework that learns how to collect information adaptively and identify hypotheses without any parameter updates at test time. ICPE shows that in-context learning can implement adaptive experiment design—learning both data-acquisition and inference rules from experience.", "questions": "(1) How sensitive is ICPE to the distribution of training environments?\n(2) How would ICPE extend to continuous or combinatorial hypothesis spaces?\n(3) The method involves several interacting components (Q-network, inference network, and λ updates). How sensitive is performance to the stability of these updates or to hyperparameters like λ’s learning rate, buffer size, and synchronization intervals (Tθ, Tϕ)?\n(4) In the fixed-confidence setting, λ and the stop-action encode correctness constraints. How robust is this mechanism in practice?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The theoretical formulation is principled and elegant. The authors derive an information-theoretic reward function directly from the posterior optimality conditions, avoiding ad hoc design choices.\n(2) By showing that Transformers can meta-learn to explore—learning both when and how to query—ICPE opens a new research direction connecting in-context learning with active learning and sequential testing.", "weaknesses": "(1) It remains unclear how ICPE scales to unseen environment distributions or how its meta-training distribution affects performance.\n(2) The experimental suite, though diverse, primarily focuses on low-dimensional or discrete problems. These settings validate proof-of-concept behavior but do not test ICPE’s limits in large or continuous hypothesis spaces.\n(3) Many experiments rely on synthetic or well-defined priors over tasks, where the environment distribution and hypotheses are known. In more realistic scenarios, such priors are unavailable or nonstationary.\n(4) While ablations such as I-DPT and I-IDS isolate the inference component, the paper lacks a deeper analysis of what the Transformer learns in-context.\n(5) Meta-training Transformers on families of ASHT tasks could be expensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHwqUNEFky", "reviewer": "Reviewer_fitX", "summary": "The pure exploration problem in sequential decision making is the task of identifying a correct hypothesis based on sequential testing. Applications included best arm identification in multi-armed bandits. This paper presents a method for using in context learning to solve pure exploration problems. Within a multi-task setting, this paper proposes to learn two transformer networks: one for action selection conditioned on history and one for posterior inference conditioned on history. The approach is evaluated on both synthetic multi-armed bandits and real-world inspired domains where it is shown that the approach has a higher probability of returning the correct hypothesis or can reach a set confidence threshold faster. Overall, this paper provides an interesting exploration of learning to learn for the pure exploration problem setting.", "questions": "Let's discuss the weaknesses raised above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is written very well in my opinion. I am not deeply familiar with the pure exploration problem but I still feel like the paper was accessible and did a good job of conveying necessary background and its own technical contribution.\n- The experiments have a good breadth of domains covered, including both toy and real-world inspired. Across these domains, the proposed method improves the key performance criteria compared to baselines.\n- I find the approach of meta-learning both an inference function and sampling function to be quite interesting. I'm not sure if it entirely novel in the field of meta- or in-context RL but, if it is, it is a nice contribution.", "weaknesses": "- The empirical study only considers 3-5 seeds per baseline ran. This seems much too little for understanding the true spread of results. Confidence intervals are computed with hierachical bootstrapping but no explanation for why this method was chosen is given.\n- The importance of the theoretical results in this paper is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DAx5LyeQgj", "reviewer": "Reviewer_Ncbv", "summary": "This paper studies the pure exploration problem in both fixed-confidence and fixed-budget settings. It introduces an in-context pure exploration approach. The approach meta-trains Transformers to map observation histories to query actions and a predicted hypothesis. Experiments on various multi-armed bandit problems, general search problems illustrate the advantage of the approach in both the averaged stopping time and correctness of the selected hypothesis.", "questions": "Please see the last part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using a transformer-based approach to solve the pure exploration problem is novel and interesting.\n2. The approach is theoretically sound. For both the fixed confidence and fixed budget settings, the paper defines an MDP structure with corresponding reward functions. The paper shows that achieving the pure exploration objective is equivalent to finding an optimal policy for the corresponding MDP problem. \n3. Compared with existing approaches, the proposed in-context pure exploration does not pose structural assumptions in the underlying problem structure and can perform well without prior knowledge of the underlying problem. \n4. Extensive experiments in different BAI problem settings show that the proposed approach achieves better stopping time and a guaranteed correctness level.", "weaknesses": "1. My main concern lies in the setup of the training set, as the experimental section does not disclose these details. In general, a core challenge in online learning is that data arrive sequentially. If the method requires an extensive training set, along with prior knowledge of $H^*$ on that training set, and assumes that for each chosen $a_t$ during the training process, the corresponding $x_{t+1}$ can be observed, this may significantly limit its applicability in real-world scenarios. I hope the authors can provide more discussion on this aspect.\n2. In experiments, the authors use different metrics in different settings, e.g., Figure 2 presents $P(\\tau>t)$ and Figure 4 presents cumulative regret. I think it would be better to compare algorithms in consistent metrics. \n3. For deterministic bandits under a fixed-budget setting, it seems natural—even for traditional algorithms—to enumerate all possible actions and then select the one that performs best. What is the author trying to illustrate through this example?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdVENprXLe", "reviewer": "Reviewer_HFZw", "summary": "This paper studies Active Sequential Hypothesis Testing (ASHT) in Bayesian formulation, where each environment is drawn from a prior $M \\sim \\mathcal{P}$. This problem is also known as pure exploration where actions index hypotheses. There are two settings (i) with a fixed confidence $\\delta$ (i.e., stop as soon as the predicted hypothesis is correct w.p. at least  1- $\\delta$) and (ii) a fixed budget (use $N$ samples to predict the correct hypothesis).\nIn both settings, the paper introduces In-Context Pure Explorer (ICPE), a Transformer-based\narchitecture meta-trained on a family of tasks to jointly learn a sampling  policy  $\\pi$ that collects data $\\mathcal{D}$ and an inference rule $I(\\mathcal{D})$ to identify an environment-specific ground-truth hypothesis $H^*$. \nThe training phase is implemented using the formulation of Q-learning.  The inference phase consists of the $a  \\mathrm{argmax} Q_\\theta (D_t,a)$ and the final hypothesis $\\hat{H}_N= \\mathrm{argmax}_H I_{\\phi}(H \\mid D_N)$.", "questions": "- Is the stopping condition in fixed confidence setting related to existing principles in the BAI literature or completely independent? \n-  (Regarding the fixed-budget setting) Could the authors elaborate on ICPE's performance sensitivity to the budget N, given that it did not outperform all baselines when varying the number of actions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- When the generative mechanism $P_t$  is not history-dependent, this coincides with Best Arm Identification of multi-armed bandits. The paper  proposes to go beyond  classical ASHT by allowing environment-specific, history-dependent observation kernels $x_{t+1} \\sim $P_t (\\cdot \\mid D_t, a_t)$. The model is very realistic and may be useful in practical cases in real-world settings.\n\n\n\n- Policy optimality for both settings using RL is shown.\nIn the fixed budget setting, the reward is assigned to $r_N(D_N)=\\max_H P(H^*=H \\mid D_N)$ at the last time step and $0$ otherwise. By defining $V_T(D_T)=r_T(D_T)$, we can define $Q$-function and  the policy $\\pi^t(D_t)= \\mathrm{argmax}_a Q_t(D_t,a)$. Optimizing with\nrespect to this reward function yields an optimal solution (Theorem 3.2). Similar results can be shown in fixed confidence settings (Theorem 3.3).\n\n\n- The literature review is thorough and provides an excellent summary connecting the work to other online learning problems. The connections with BAI and RL, in particular, are well-discussed.\nThe experiments are also thorough, including several widely used baselines from BAI.", "weaknesses": "- Description of the stopping action is not fully specified in the paper. I am assuming a_stop = \\mathrm{argmax}_a Q_\\theta (s_t,a). \n- In the fixed-budget setting experiment, the probability of correctness is evaluated by varying the number of actions. In this test, ICPE’s performance does not consistently surpass the other baselines. What is the performance sensitivity to the budget size N?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ppRkre4NdK", "reviewer": "Reviewer_g3VE", "summary": "This paper introduces In-Context Pure Exploration (ICPE), a meta-learning approach for active sequential hypothesis testing that connects the pure exploration theory in bandits to sequential modeling using transformers. The key idea is to adaptively collect data from an environment M to identify a ground-truth hypothesis H* with minimal samples in either fixed-confidence or fixed-budget (N samples) settings. They have extensive theoretical analysis, starting with Proposition 3.1 that establishes that the optimal inference rule is $I^{\\star}(z) = arg \\max_H P(H^{\\star}=H|D_t=z)$, making posterior computation central. For policy optimization, the paper derives reward functions that emerge from the dual formulation - in fixed-budget, rewards equal the maximum posterior value at termination, while fixed-confidence uses a Lagrangian formulation with rewards $r_t, \\lambda = -1 + \\lambda\\cdot1_{stop}·\\max_H P(H^{\\star}=H|D_t)$. Note that this is heavily influenced by the Best-Arm Identification (BAI) setting in MAB. This is a special case where hypotheses coincide with actions (H=A) and $H^{\\star}=\\arg \\max_a E[\\text{reward}|a]$. They also discuss how ICPE generalizes beyond standard BAI by handling history-dependent observation kernels $P_t(·|D_t,a)$ rather than memoryless $P(·|H^{\\star},a)$, and learning the inference rule rather than assuming known likelihood functions. The paper proves that optimal policies satisfy Bellman equations with Q-functions incorporating posterior distributions, establishing the MDP formulation's validity. ICPE uses two Transformer networks: $I_{\\phi}$ approximating posteriors via cross-entropy loss, and $Q_{\\theta}$ learning the exploration policy via DQN with information-theoretic rewards. This architecture handles the credit assignment problem in sequential exploration. The approach unifies several exploration settings - it recovers classical algorithms like binary search, exploits structured dependencies in \"magic action\" bandits where certain arms reveal information about others, and generalizes to non-bandit settings like image classification, where actions (pixel regions) differ from hypotheses (digit classes). Experiments demonstrate that ICPE matches theoretical lower bounds in structured bandits (Theorem B.20) while maintaining competitive performance in standard settings, validating both theoretical predictions and practical effectiveness.", "questions": "1. Can a single model trained on the trajectory-level data handle both settings at test time? How does performance degrade when using a fixed-budget trained model on fixed-confidence tasks (or vice versa)?\n2. Given the theoretical nature of the paper, the above question should also be answered from a theoretical standpoint. For example, see https://papers.nips.cc/paper_files/paper/2012/file/8b0d268963dd0cfb808aac48a549829f-Paper.pdf.\n3. Why can't the stopping decision emerge from the value function without an explicit action ($a_{stop}$)? How does forcing stop as an action affect the exploration strategy compared to value-based stopping?\n4. How many training tasks are needed for $\\epsilon$-optimal performance? The paper provides no sample complexity for the meta-learning phase.\n5. More rigorous real-world experiments should be done.\n\nIf my questions and concerns are addressed, I am willing to raise my score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides a rigorous theoretical justification showing how the optimal inference rule and policy come from existing bandit theory. The problem formulation, along with the reward function, is principled rather than heuristic. The connection between posterior maximization and Q-learning is well established.\n2. The evaluation spans diverse domains from classical bandits to structured environments with hidden information, demonstrating ICPE's versatility. The magic action experiments particularly showcase the method's ability to discover and exploit non-obvious information structures.\n3. ICPE learns both components from data, handling history-dependent observations and unknown likelihood functions, making it more applicable to (some) real-world scenarios.", "weaknesses": "1. My main concern is that it is not clear to me how a single approach works for both best confidence and best budget setting and is simultaneously optimal. Is this a best-of-both-worlds approach? Or is there a tradeoff? This requires more discussion.\n2. The writing needs more improvement. In the discussion of Algorithm 1, the stop action was suddenly introduced in lines 258-262. It is not clear to me why it is necessary. Also, the stop action is not directly motivated from theory, and seems to be heuristic. Each theorem requires a dedicated discussion section to bring out the novelty in the proofs. Again, the novelty of the theoretical proofs is not clear to me. I have not checked the appendix.\n3. While the paper proves optimality under certain conditions, it doesn't provide finite-sample complexity bounds for ICPE or formal guarantees on the meta-learning convergence, leaving uncertainty about when the approach will succeed. This deviates from the traditional BAI analysis under the bandit framework", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)