Title: Adaptive Frontier Exploration on Graphs with Applications to Network-Based Disease Testing
Venue: NeurIPS 2025
Accepted: yes
Keywords: sequential decision making

graphs

branching bandits

gittins index

network-based testing

public health
Abstract: We study a sequential decision-making problem on a $n$-node graph $\mathcal{G}$ where each node has an unknown label from a finite set $\mathbf{\Omega}$, drawn from a joint distribution $\mathcal{P}$ that is Markov with respect to $\mathcal{G}$. At each step, selecting a node reveals its label and yields a label-dependent reward. The goal is to adaptively choose nodes to maximize expected accumulated discounted rewards. We impose a frontier exploration constraint, where actions are limited to neighbors of previously selected nodes, reflecting practical constraints in settings such as contact tracing and robotic exploration. We design a Gittins index-based policy that applies to general graphs and is provably optimal when $\mathcal{G}$ is a forest. Our implementation runs in $\mathcal{O}(n^2 \cdot |\mathbf{\Omega}|^2)$ time while using $\mathcal{O}(n \cdot |\mathbf{\Omega}|^2)$ oracle calls to $\mathcal{P}$ and $\mathcal{O}(n^2 \cdot |\mathbf{\Omega}|)$ space. Experiments on synthetic and real-world graphs show that our method consistently outperforms natural baselines, including in non-tree, budget-limited, and undiscounted settings. For example, in HIV testing simulations on real-world sexual interaction networks, our policy detects nearly all positive cases with only half the population tested, substantially outperforming other baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the adaptive frontier graph exploration problem, whereby one wants to explode the nodes of a graph in which each node has a label, each label a reward. The goal is to maximize total expected (discounted) reward. The formulation essentially combines a Markov decision process with dynamic graph exploration. \n\nThey develop a Gittins-index approach, proving that their algorithm is optimal in the case of trees, and showing experimentally that it performs well in more complicated, real-world graphs based on disease networks. They also prove several new results concerning the computational tractability of the Gittins-index in their setting.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n- Good results, both on synthetic data and real-world data \n- Well motivated problem setting. The authors did a good job (modulo one complaint below) of tying the formalism back to a real world setting. \n- New results on the computational tractability of the Gittins-index. \n- Well written \n\n# Weaknesses \n- The mathematical details are a bit hard to follow for a non-expert in the area. The Gittins formulation in particular is not intuitive, nor do the authors spend that much time providing intuition. But I understand that space constraints make this challenging. \n- I find the generality of the result a bit hard to parse (see below). It seems that they solve a problem that may be more general than adaptive graph exploration. (This is a minor weakness; could even be a strength if addressed.) \n- I'd like to see the experimental results stretched a bit more - when does their algorithm start being worse than its competitors? Surely there are some scenarios in which it is not the best.\n\nQUESTIONS:\n1. Does the agent know the structure of the graph a priori? There are flavors of the graph exploration problem where the graph structure has to be discovered by exploring. \n\n2. I find the formulation of $\\pi^*$ a bit confusing. You seem to be using $X_\\pi$ to denote both the node itself and the node’s label (otherwise how can we have $X_\\pi = v$?). But the reward function treats $X_\\pi$ as the node itself, not the label. (Eg  in the public health example at the bottom of page 2 you treat $X$ as an individual). I’m not sure you should necessarily change anything, but if you agree with this, it might be worth commenting on. \n\n3. Is it realistic to assume that n is known in advance? In the public health example, it seems you might start exploring and only decide to stop based on what you find, or external factors such as budget etc. But it doesn’t seem likely that you’d commit to only testing n = 10,000 and, regardless of the outcomes, stop after that. \n\n4. You say: “Our work addresses this gap by presenting the first efficient implementation of Gittins-based policies in  discrete branching bandits with history-dependent rewards, enabling their use in structured settings like network-based disease testing.” (Pg 4). How general are the structured settings that you’re alluding to? If very general, one wonders why you’re focused on adaptive graph exploration the first place, and didn’t write the paper about the more general setting of discrete branching bandits. A sentence or two about this would be helpful for readers like me, who aren’t extremely familiar with this area. \n\n5. In Figure 4, it’s interesting that greedy outperforms DQN in ~half the experiments. This seems counterintuitive? Also, if you increase the number of extra edges, does the Gittins performance ever fall below greedy and/or DQN, or do they simply converge? \n\n**Minor comments**: \n- Researchers typo in line 200.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

comments: SUMMARY:
Motivated by network-based disease testing, the paper formalizes Adaptive Frontier Exploration on Graphs (AFEG) as a sequential decision problem on a probabilistic graph. When the graph is a forest, the proposed Gittins index-based policy for AFEG has a provable convergence guarantee. Experiments on both synthetic and real-world graphs show the superiority of the proposed method.

STRENGTHS AND WEAKNESSES:
# Strengths
1. The problem of network-based disease testing is important. The proposed AFEG model correctly captures realistic locality constraints (e.g., contact-tracing). 
2. The piecewise-linear analysis of the recursive value functions appears original to me and leads to the first practical algorithm for discrete branching-bandit indices.

# Weaknesses 
1. Assumed access to the joint distribution $\mathcal{P}$. I think the $\mathcal{P}$ is usually learned from data in practice. How sensitive are results to model misspecification? Empirical ablations with learned vs. true $\mathcal{P}$ would strengthen the work.
2. Scalability to very large graphs. $\mathcal{O}(n^2 |\Omega|^2)$ is quadratic on $n$. Can sparsity or incremental computation be exploited to reach web-scale graphs?

QUESTIONS:
1. The definitions of two potential functions $\phi_{X,b}$ and $\Phi_{S,b}$ in Eq. 1 and 2 are confusing me, although there are some attempts of explanation on L176-178. What is the difference between these potentials?
2. I am curious why the author only uses a single-step DQN as a deep RL comparison, but omits more recent graph RL or planning methods (e.g., policy-gradient, Monte-Carlo tree search). Why were they not considered?

LIMITATIONS:
1. Optimality only holds for forests and relies on exact $\mathcal{P}$
2. Scalability to a very large graph is limited.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the problem of sequential decision-making over a graph where each node is associated with a label from a finite set of labels and a label-dependent \nreward whose distribution is given by a Markov model. The goal is to find the optimal policy that collects the maximum expected discounted reward. The work investigates an action constraint defined here as the adaptive frontier exploration on graphs (AFEG) constraint where the next node to visit in the graph has to be chosen from the set of non-visited neighbours of the already visited nodes. The authors model network based disease testing using this framework motivating such a model in scenarios where tested individuals might be interviewed to trace contact with other individuals so that these individuals can subsequently be tested. The problem of network based disease testing is an important problem in public health and studies have shown better algorithms lead to better outcomes. In light of this, the current work reformulates AFEG as a branching bandit problem and proves optimality of Gittins index when the disease network is a forest. In case of general graphs, the authors use the BFS tree of the graph to run the Gittins index based policy, thus providing a heuristic for the general case. Moreover, the authors show that their algorithm runs in time polynomial in the size of the graph and the number of elements in the label space. Finally, the work demonstrates empirically the performance of Gittins index based policy for synthetic as well as real-world disease network datasets. The experiments show that Gittins index based policy is generally faster than other baselines while giving superior or competitive performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The formulation of network based disease testing as an AFEG problem is interesting. Network based disease testing is an important problem in public health and has shown merits in multiple studies. Therefore, this work is an important contribution in the literature of network based disease testing.\n\n2. The computational efficiency results for Gittins index based policy which is generally intractable due to the curse of dimensionality is another novel aspect of the work.\n\n3. The experiments performed are extensive with several real-world disease datasets considered on top of synthetic experiments. In most of the experiments on real data, the Gittins index based policy is either slightly superior or competitive. The experiment set up is also natural and well motivated.\n\n4. Lastly, the writing is very clear and cogent with good organization and discussions making it easy for the reviewer to understand the work. Overall it was a pleasure to read.\n\nWeaknesses:\n\n1. Firstly, Theorem 1 is not very novel. It is a straightforward adaptation of the work [1]. Moreover, the guarantee works for the forest case which though is a good starting point, is theoretically a limiting assumption and may not hold in real world networks. However, to the authors’ defence the Gittins index based policy on the BFS tree of general graphs seems to give good empirical performance. Here however the question is why BFS tree and not other trees? This work might benefit if the authors can give provable guarantees on the global graph when Gittins based policy is run on an appropriately chosen tree of the graph.\n\n2. The work considers a case where the label space is finite. The main computational result (Theorem 6) also banks on this assumption. This may be limiting in scenarios where richer data is available, for example in disease networks, not just discrete labels but also real valued intensity may be available. While general label spaces may be hard to compute, it would be interesting to see some generalization, for example, can it be done for continuous label spaces with an $\\varepsilon$-discretization where the computational efficiency scale as $poly \\log (\\frac{1}{\\varepsilon}$ whereas the optimal policy over the discretization may differ by $\\varepsilon$ from optimal?\n\n3. The current work relies on access to the graph structure and the distribution $\\mathcal{P}$. While the assumption on graph structure is still fine, the assumption of access to distribution $\\mathcal{P}$ exactly is a limiting one. Particularly, one of the baselines considered is DQN which relies only on sample access to $\\mathcal{P}$. Instead of assuming exact oracle for $\\mathcal{P}$, it would be interesting to see the theory developed for sample access to the distribution. Indeed, a more reinforcement learning style with finite sample convergence guarantees will be useful to the work. This is because currently the work uses existing data to first learn an approximate $\\mathcal{P}$ and then use it for policy. However, this learning step makes certain strong structural assumptions which may be bypassed if an RL style algorithm can be given.\n\n4. Figure 14 seems to show that the DQN policy is much better than the Gittins policy with the later having constant suboptimal gap for all the disease networks considered. This is in contradiction to the experimental results in the remaining paper. Moreover, the motivation for the smaller dataset created by subsampling with threshold $\\tau = 300$ is not clear. Indeed, Fig. 14 result on the full dataset seems contradictory to the result on this subsample. Moreover, the authors’ algorithm does not seem to provide much performance benefit over greedy/DQN in the low $\\beta$ regime on synthetic dataset. In light of this, it is not clear why the authors do not repeat experiments with low $\\beta$ on the real world dataset.\n\n[1] Godfrey Keller, Alison Oldale. Branching bandits: a sequential search process with correlated pay-offs, Journal of Economic Theory, Volume 113, Issue 2, 2003, Pages 302-315.\n\nQUESTIONS:\nSee weaknesses above. Minor suggestion: please consider changing the notation $X_{\\pi(S_t)}$ that indicates both label and the node currently. It would help the reader if this notation was disambiguated.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConsider the problem of sequential decision making on a graph, where the arms that can be pulled are only be neighbours of previously pulled arms. This is the frontier of exploration. Specifically, this paper addresses the \"Adaptive Frontier Exploration on Graphs\" (AFEG) problem, a sequential decision-making task relevant to high-impact applications like network-based disease testing. \n\nThe core contribution is a novel, efficient, polynomial-time algorithm for computing the Gittins index policy, which is provably optimal for AFEG instances on tree-structured graphs. The authors connect this deep theoretical result to a pressing real-world problem, demonstrating on real-world HIV interaction networks that their method can improve testing efficiency compared to strong baselines, including a DQN (deep Q net) approach. The work is technically deep, rigorously presented, and its claims are backed by strong empirical evidence.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. One of the papers I was excited to read, and I've been reading MAB papers for a long time! Very interesting and practical problem formulation.\n2. The paper's primary contribution is arguably proving the first poliynomial time algorithm for the provably optimal policy in discrete branching bandits. This is a foundational contribution to the MAB and RL literature, with implications beyond the disease testing/contact tracing problem.\n3. I was very impressed with the empirical results which indicate that nearly all HIV-positive individuals were identified by testing only half the population in the ICPSR dataset, and for example works better than DQN (deep Q networks). Given decreasing medical and research budgets, this could be an important method to release to real-world use cases.\n\n### Weakness\n\nThe model assumes an oracle access to joint probability distribution P, which can provide answers about the conditional probability distributions over Bayesian Nets. \n\nNote: But in Appendix B in the paper, they convert the weakness into a strength by actually building this oracle out from a model using maximum pseudo-likelihood estimation (MPLE). This makes the oracle assumption plausible. Yet the authors acknowledge that such a parameter fitting may not recover the exact real world dynamics (Line 631).\n\nQUESTIONS:\n- Are real world graphs truly forests (for which the optimality results hold)? In the real world, are people not fully connected to each other?\n- Where does this come from (line 701 in appendix) $θ_1 \\in R^{2+2d}$ and $θ_2 \\in R^{4+5d}$\n- How valid is the assumption on non-decreasing piecewise linear function?\n\nLIMITATIONS:\nPlease include a more detailed limitations section in your work. Since this paper has potential to be applied to medical domains, it would be prudent to state the limitations in greater detail to potential users.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}