A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health
Authors: Nikhil Behari
Edwin Zhang
YUNFAN ZHAO
Aparna Taneja
Dheeraj Mysore Nagaraj
Milind Tambe
Venue: NeurIPS 2024 poster (2024)
Keywords: public health
multi-agent systems
multi-armed bandits
LLMs
Primary area: machine_learning_for_healthcare

Abstract: Restless multi-armed bandits (RMAB) have demonstrated success in optimizing resource allocation for large beneficiary populations in public health settings. Unfortunately, RMAB models lack flexibility to adapt to evolving public health policy priorities. Concurrently, Large Language Models (LLMs) have emerged as adept automated planners across domains of robotic control and navigation. In this paper, we propose a Decision Language Model (DLM) for RMABs, enabling dynamic fine-tuning of RMAB policies in public health settings using human-language commands. We propose using LLMs as automated planners to (1) interpret human policy preference prompts, (2) propose reward functions as code for a multi-agent RMAB environment, and (3) iterate on the generated reward functions using feedback from grounded RMAB simulations. We illustrate the application of DLM in collaboration with ARMMAN, an India-based non-profit promoting preventative care for pregnant mothers, that currently relies on RMAB policies to optimally allocate health worker calls to low-resource populations.  We conduct a technology demonstration in simulation using the Gemini Pro model, showing DLM can dynamically shape policy outcomes using only human prompts as input.

A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health
Nikhil Behari MIT, Harvard University
Edwin Zhang Harvard University
Yunfan Zhao
GE Healthcare, Harvard University
Aparna Taneja
Dheeraj Nagaraj
Milind Tambe
Google
Google
Harvard University, Google
Abstract
Restless multi-armed bandits (RMAB) have demonstrated success in optimizing resource allocation for large beneficiary populations in public health settings. Un- fortunately, RMAB models lack flexibility to adapt to evolving public health policy priorities. Concurrently, Large Language Models (LLMs) have emerged adept automated planners across domains of robotic control and navigation. In this paper, we propose Decision Language Model (DLM) for RMABs, enabling dynamic fine-tuning of RMAB policies in public health settings using human- lanaanlanguage commands. We propose using LLMs as automated planners to (1) interpret human policy preference prompts, (2) propose reward functions as code for multi-agent RMAB environment, and (3) iterate the generated reward functions using feed- back from grounded RMAB simulations. We illustrate the application of DLM in collaboration with ARMMAN, care for pregnant mothers, that currently relies on RMAB policies to optimally allocate health worker calls to low-resource populations. We conduct technology demonstration in simulation using the Gemini Pro model [I], showing DLM can dynamically shape policy outcomes using only human prompts as input.
India-based non-profit promoting preventative
Introduction
Limited resource allocation is a frequent challenge in public health settings. For instance, in the maternal and child health domain, preventative reducing maternal mortality, where global rates are currently more than double the UN Sustainable Development Goal target of fewer than 70 deaths per 100K live births 22. These essential programs help avoid preventable deaths 43. yet are often operated by non- non-profits supporting low-resource communities [5] 6) and thereby face two key challenges in distributing resources large beneficiary populations. First, programs typically operate with insufficient financial and human resources, necessitating effective resource allocation strategies to maximize health outcomes 7 [ Second, these programs must frequently adapt to changing population needs and intervention priorities (3). Adaptability is especially important for prioritizing care for known subpopulations with higher risk 9 and for adjusting strategies using community or expert-driven insights [00
awareness programs play a key role in
To address the first problem, restless multi-armed bandits (RMAB) have proven effective in real- world deployment for optimally allocating limited resources critical public health domains [1] In the classical RMAB formulation, central planner chooses arms allocate resources, observing a per-arm state-dependent reward. However, while RMABs are well-suited for resource allocation, a significant challenge persists in developing models capable of adapting to changing policy objectives. For instance: public health experts must often dynamically allocate resources to
13 14.
Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

specific underprivileged socioeconomic groups [5] 16, 17,, yet existing works in RMABs largely focus on fixed objectives, requiring significant human design achieve new desired health outcomes.
Recently, large language models (LLMs) have emerged as adept planners in tasks such as navigation 18]. spatio-temporal reasoning [99 and interactive decision-making (20]. Recent works have also demonstrated that LLMs, from language prompts, can generate reward functions as code, automating complex robotic manipulation tasks in reinforcement learning (RL) [2I]. While there growing research in LLMs for healthcare [22]. the potential of LLMs to dynamically adapt resource allocation
using language prompts potentially enabling automated policy tuning using expert and community
health insights [3 remains unstudied.
In this work, we propose Decision-Languee Model (DLM) for RMABs, enabling dynamic fine- tuning of resource allocation policies for public health using human language prompts. We propose: 1) using L.MMs to disambiguate language-expreseee policy preferences, 2) using L.LMs directly propose reward functions as code to reach desired RMAB objectives, and 3) fully automated self-reflection stage iteratively refine LLM-generated reward functions using grounded RMAB simulations, without requiring ground truth feedback. Stepping beyond existing work in LLMs for health, we uniquely propose using LLMs tune RMAB-driven resource allocation through reward function design, enabling: 1) more principled resource allocation using RMABBs as central planner, rather than direct LLM output, 2) continual alignment of reward functions to specified prompts using simulated RMAB allocation outcomes to refine LLM proposals, and 3) improved interpretability of proposed rewards through the use of code for reward function output.
We assess our proposed method within a simulated public health environment created in partnership with ARMMAN, India-based non-profit that spreads preventative care awareness pregnant women and new mothers through an automated call service. To increase program listenership. ARM- MAN employs health workers to provide live service calls to beneficiaries; however, due to limited support staff, ARMMAN faces a resource allocation problem to improve program listenership through optimal assignment of health workers. Prior works modeling the ARMMAN setting with RMABs have shown that RMAB policies can reduce engagement drops by ~ 30% in real-world deployment (4 25 26). However, these prior works fail address a persistent challenge within ARMMAN: tuning policies dynamically to prioritize disadvantaged groups, a common issue when targeting new regions or shifting population dynamics (27 28 We utilize an anonymized (Appendix B) dataset. from ARMMAN to develop simulation (Appendix C) representing changing priorities within the maternal health setting, and evaluate our method within this simulated public health environment. We provide detailed discussion of our consideration of ethical guidelines during the design process of this work in Appendix A| B
In summary, our key contributions are as follows:
To the best of our knowledge, we are the first to propose using LLMs to adapt changing resource allocation objectives in public health through reward design in the RMAB setting.
We introduce a reward proposal loop that enhances LLM-generated reward functions us- ing feedback from restless multi-armed bandit (RMAB) simulations, enabling LLMs to iteratively refine reward design to achieve specific, human-specified policy outcomes.
To assess the feasibility of our system in simulation, we evaluate our algorithms' performance using the Gemini Pro model [1 in a real-world inspired task of resource allocation in maternal and child care, demonstrating near human-level policy tuning to achieve human- specified outcomes using only language prompts as input.
2
Related Work
RMABs:
The RMAB problem, introduced by Whittle (29). is classically solved through the Whittle
index heuristic policy [30 Subsequent works have generalized to multi-action RMABs [32] 33
RMABs gained prominence in public health domains and are deployed disseminate preventative healthcare information, monitor health program adherence, and model disease spread 334 24 35)
However, existing works focus fixed reward functions and fail to consider that public health planners often have evolving priorities B
Reward Design: Designing reward functions that effectively condense long-term agent goals into immediate behavioral signals a central problem in RL (36). Manual designs are prone to task misspecification and overfitting [37]. Brys et al. [38] and Hussein et al. [3 reshape reward

---

Legend
Language Command
LLM
CLM inpus context
priontbe income mothass
"optimize over sightly
Vissig lire revarna functions my probiom...
R. o state/2 footo[21)/2
canaldate reward
Rg footer
Sim. under &
Context
LLM Reflection
1. Age, in years (Int)
VVom the rewara diis bitriibti choose best roward candidote Index
2: Incalie, oillios (in))
3. Multi-Agent Simulations
(
Codebase Context
Policy Outcome Comparison
R.
class Mettermak iootthErod: state: Mftoet value D-1" teatures: "skot?"
R
R2
TE
pen-agent pokky cot
incore
Figure I: Overview of the DLM ceneuuiieoood reward design loop. We provide three context descriptions to the I.IM: language command (full list of commands in Table 3). list of per-arm demographic features available for proposed reward functions, and syntax cues enabling LLM reward function output directly in code. From this context, the 1) LLM then proposes 2) candidate reward functions which are used to train 3) optimal policies under proposed rewards. Trained policies are simulated to generate 4) policy outcome comparisons showing state-feature distributions over key demographic groups. Finally, we query an LLM to perform 5) self-reflection 43 21 by choosing the best candidate reward aligning with the original language command; selected candidates are used as context to guide future reward generation.
through expert behavior observation; however, expert examples may not be available in resource and data-limited public health domains. Using LLMs as dense reward signals has been investigated 40 41. Ma et al. (21) and Li et al. [42] further use LLMs to output reward function code from language-specified goals. However, this work fails to address multi-agent settings, where a central planner could prioritize certain groups through reward design, and where the ground truth return signal is unknown.
3 Background
We consider RMAB problem with N arms, each representing public health program beneficiaries, following prior work in the ARMMAN setting [24, [3]] [4]. Each arm n E [N] follows a Markov decision process As, C,, T.. R,, 3,) with state space S,,, action space A,,. action costs C, An - RR, and unknown transition probabilities Tn S. x An - A(S,). We define reward function R,, Sn IR. and discount factor / 10,
When Sn, An, and Cn are the same for all arms n E [N], such as the public health setting we focus Note our algorithms still apply to more general settings. A planner
omit the subscript learns policy π that maps states 8 E S to actions in the set A, given the constraint that the sum cost of actions does not exceed budget B. The objective is to learn policy that maximizes the Bellman
equation: max {
J(s) = Σn R (8x) + 3 EggTTT s.m(s)) [J (s')] s.t. ((.) E A, Vn and
Ex C=(s..) B for cost of action C=(s..) E C.. We take the Lagrangian relaxation 255 fixing an action charge À to decouple the value function across arms:
AB
N Σ
(s, min X20
max {Qn (Sn, an an: E.EA
(1)
1 1- 3
s.t. Qn (Sn, On, A) R (s,) Aco, + +3E1Qn (8 any A) π (A)].
Defined for Q-function Q, arm action an, transitioning from state 8,, under action a, to 8 and (A) an optimal policy under X. 43 21. We additionally build on deep RL approaches for RMABs
33 which we discuss in Appendix
Decision-Language Model for RMABs
Below, we provide an overview of DLM (see Figure T). We first describe the reward generation setting and the context provided to the LLM for initial reward candidate generation (Figure Steps 1,2). We
3

---

next discuss the independent simulation stage (Step 3) and subsequent policy outcome comparison stage that simulates each policy reward outcomes (Step 4). We then discuss LLM reflection, which identifies top candidates to guide future in-context learning for L.M reward generation (Step 5).
Finally, we motivate the use of LLMMs for reward generation from a theoretical perspective.
4.1 Problem Setting: Reward Generation for RMABs
first define the goal of language-conditioned reward design in the RMAB setting. consider a human-specified language policy goal & with corresponding base reward function that maps policies т to real, scalar values: F; II + R. This base reward function establishes a scalar "ground truth" evaluation that corresponds directly to the human- -uman-specified command. Then, key objective is, given a huwan-speciiii speeii-pe language prompt i' and arm features z, to generate the reward function that maximizes base function F' through optimal policy x ft for reward R:
max F(TT')) where: arginax *Ell
R (sn z) (J 8, Σ
NER
s.t. (8x)) E A,, Vn and Σ Cr(an) S B. 1=1
Note that we not assume access to the ground truth reward function F during training; however, may query this ground truth "Base" reward at test time to evaluate proposed policy. We propose using LLMs to: 1) propose reward function R, such that optimal policy π Rt when evaluated via the unknown base function F, maximizes reward output, and 2) automatically refine R through self-reflection.
4.2 Provided DLM Context
In the reward function generation phase, we prompt the LLM with three key contextual components (shown in Alg. [ line 1). First, we include a human-language command that describes a desired policy outcome targeting feature-based group (see Table for examples). practice, this the only human input required for the proposed DLM technique. Second, we provide arm feature information, denoted z. In the public health setting, features may include demographic information, socioeconomic data, or other descriptors relevant to policy outcomes; we provide the features used in our simulated setting in Appendix Figure 4 We propose that LLMs, given a language prompt f, may be used to extract the relevant features Ze S from to design a reward function R. Third, we provide context regarding the RMAB setting and code implementation. Specifically, we include information describing state, a per-arm binary value, and syntax for accessing feature information (example prompt shown Appendix Tp. Using this context, we then query an LLM to propose reward functions as code (following [1] to achieve language-specified outcomes; to improve the alignment of proposed functions to these stage within the DLM loop.
goals, we then introduce a key multi-agent simulation
Algorithm 1 DLM
1: Input: Task f, feature z, and code Ç context, Output: RDLM
Hyperparams: Loop iters. I, proposal batch K, sim. epochs ne, sim. steps n., num. arms
3: for iteration to I do
4: ##f LLM reward proposal: Sample LLM reward funcs: RI.K ~ LLMME,, z, (), init. reflection string y
5: ## Multi Multi-agent agent simulations: Init. policy and critic net. 0,
6:
for reward = to K do
7: Init. X-net. A, buffer D, states = So. features z and clear reflection string y = [
8:
for epoch = 1 to n., do
Get lambda for current states: A(s)
10:
for timestep t= to n,,
11:
Sample actions: a. ~ 0(s.. X) Vn E [NV, simulate: s', Simulate(s, a, R., z).
12:
update buffer D + DU ((s, a, r's', X)). update states:
s'
13:
Update (0, Φ) with PPO using 2 and update A with D
14:
## Outcome comparison (Alg. 2 Update y + y OutAnalysis(f. 4, Z)
15:
## Top candidate selection:
ROLM + LLM(E, V,, "choose best..."), "). + Ç (RDm)

---

4.3 Multi-Agent Simulation
We evaluate each LLM -L--pooposed reward function RI.K by training a policy network under each proposed reward R2 (shown Alg. lines 5:13). We consider simulation space which defines the number of arms N, each with fixed, but hidden, transition dynamics. Following the procedure defined in Alg. for a given reward R,, we first sample an action-charge X, which is used to decouple the learning of arm policies (Fq. [ Alg. [lie 9). Then, we simulate n, timesteps, sampling trajectories for each arm under the action-charge (Alg. [ines 10:12), following [25] 45). These trajectories are used in a buffer D (Alg. I line 12) to eventually update the policy and critic network(0, )) (Alg. [line 13). Note that we compute PPO advantage estimates for the actor from estimated Q-values.
4.4 Reflection Stage
We propose enabling LLM self-reflection using policy outcome comparison procedure described in Appendix D Algorithm 2 Unlike prior works (21). we do not assume access to a numerical, scalar fitness function F to validate reward functions. In the absence of this scalar feedback, we propose showing state-feature distributions, described below, for LLMs to select top candidate reward functions, and use these top candidates to guide future generations, Given trained policy and critic network 0, O learnt lambda value X, and feature matrix Z E R xm we first simulate over n. evaluation timesteps (Alg. 2 lines 5:7). For stored accumulated agent states S, we designate preferred "positive states"; this may be customized depending on the setting, but we consider a binary per-agent state with preferred "positive" state Using accumulated states, we then compute the percentage of positive states, out of all positive states observed, accrued from each key feature group E G over the evaluation timesteps, as a signal for intervention effect (Alg. 2 linne 9:11). These state-feature distributions quantify the distribution of accumulated positive states over key demographic features, ultimately describing the percentage of total positive intervention effects attributed to each discretized age range, education level, or income bracket, for example. Each distribution is reported as an output string (see Appendix M for examples).
We then query an LLM to select the best candidate example given the original prompt l and the outcome distribution string y (Alg. line 15). This selected example is then added as context in the prompt of the next iteration of reward generation (Alg. [ line 15). Intuitively, because we focus on resource allocation in the public health domain, we propose that simulated state-feature distributions provide sufficient signal for the LLM to analyze proposed rewards and select top candidate examples for future in-context learning. This approach has two key strengths. First, we enable greater transparency through state-feature distribution analysis. By providing only simulated outcomes of policies under specified rewards π R for reflection, we ensure that self- relf-reflection focuses purely on alignment between the stated language goal and the health resource allocation outcomes, without assuming any access to ground truth reward. Second, we allow for greater flexibility; by investigating only the outcome distributions of the proposed rewards, rather than original policies themselves, we enable a model-agnostic approach to self-iteration in reward design.
4.5 LLM Reward Generation Capability
We theoretically justifyy our method and provide further insight into how LLMs generate reward function via reflection. We propose that the LLM can be interpreted as following a hyperparameter optimization algorithm: (1) Find the relevant features from 2 using world knowledge (2) Generate a reward function which is a weighted sum of the relevant features and observe the outcome of the optimal policy corresponding to this reward via feature distribution feedback (see Section (3) Find the state-feature distribution that best conforms to the human language command (4) Optimize the weights to obtain a better candidate reward function.
We find empirically in Table and Appendix H that the LLM indeed implements the first step of the proposed algorithm, by analyzing the precision and recall of finding relevant features through the usage of world knowledge and logical reasoning. This finding corroborates the results found in the logical reasoning analysis of language models done by Clark et al. [46]. We assume that the LLM can evaluate reward functions by assigning a value to their state-feature distribution. In this section, we prove by construction that transformer implement the second step of this algorithm and give complexity bounds and correctness guarantees of this optimization process.
In our setting, we consider the embedding space IRd, whose elements encode states and features of an arm. We assume that there is an embedding function e which maps features of an arm (denoted by 2) and its state to vector in Rd That is J6 0(s, 2) Rd We consider each entry of the vector ((s,a)
5

---

to correspond to some simple single feature of the arm such as (age group) or complex combination
of simple features such as (age group, education level). Let 1( denote the indicator function. For
example, the first entry of the vector & could be Φ1 (8, 2) = 1 (20- v Age(=) ) I 25, = 0) allowing us to set the rewards to arms in state and age between 20 and 25. The 10-th entry could be
$10(8, =) = 1 1(20 Σ Age(z) I 25, Education (z) = High School, s = 1), allowing us to set the rewards for arms in state 1, ages between 20 and 25 and education level being high school.
Based on empirical evidence of the LLM generated rewards, we consider the reward function for arm with features in state s of the form R(s, 2) = w" o(s. 2) for some dimensional vector w E IR with non negatvgte entries. This represents rich class of rewards. In the example above, if w1 = 1 and Who 10 and all other entries of w are zeros, the reward function is:
20 < (z) 1 25, 8 = 0 1f < Age (z) S 25, Education(z) = High School, s = 1
2)
0 otherwise
We let the true reward function to be R (s, 2) (T 2) for some w" E IK" with non-negative entries. Assume that w" is sparse vector i.e, the number of non-zero entries is constant independent of n. This roughly means that the reward depends on only a few features. Let Supp(w") be the set of indices where w" has non-zero entries. Let [lo == (Supp(w"))
Finding Supp(w") can be a challenging task, since n can be very large. Based on the discussion above, in Step 1, the LLM is able to use its world knowledge and find Supp(w"). ). We now try to understand how the LLM approximates the corresponding entries of w² via steps 2,3 and 4. Based on empirical analysis of LLM outputs, we propose the following hyperparameter search algorithm in the log-space to model its behavior. For some a > 1, denote the search space to be S = fa* : K < k S K} for some integer K. The algorithm searches for (w; (wx)):ssp))(w") E S)Supp(w")] II Here, a represents the base of the logspace, or level of granularity of the discretization. In practice, we found that a was often set to be 10 by the L.M.. Let V(w, w²) denote the value assigned by the L.M to the state-feature distribution corresponding the reward w ((s, z) (as proposed in Step 3). We assume that the V(. has the following monotonicity property, which roughly states that if w comes closer to w", then Vww, w²) increases.
Definition (Monotonicity). We say that the return function V is monotone for any u, v E R", if [u: w > p - - for every i E [n), then (u, w²) S V (ee w*). Proposition 1. Assume monotonicity for V(,, and let v == arg max w gispstp V(w, w²).
There exists transformer T with constant depth D and width (oo) which can find i with O[I[w*|ooK] K) samples, with oracle access to V(..=).
This shows that the LLM has the capability correctly optimize reward weights and converge to good reward function, under certain assumptions. We give a detailed proof in Appendiix G
5 Experimental Evaluation
5.1 Simulated Public Health Setting
We evaluate DLM in simulated setting developed in collaboration with ARMMAN, an India-based non- non-profit that enrolls pregnant women (beneficiaries) from low-income communities [33 24 into an automated messaging service sharing preventative healthcare information. ARMMAN employs health workers to provide live service calls to beneficiaries with a higher risk of program drop-out, however, the beneficiary population far outnumbers the available health workers. Currently, deployed RMABs in the ARMMAN setting model beneficiaries as arms, live health worker calls as intervention a preferred "engaged" (positive)
actions (4 49 (5), and binary arm states as "not engaged" and state when beneficiaries listen to an automated message for > 30 seconds. The RMAB policy then identifies individuals across the entire beneficiary population to live call such that overall program engagement is maximized [31 26). We consider this the old, fixed "Default" policy goal. However, a key challenge for ARMMAN is training dynamic policies to support specific underrepresented groups, for example when the program is deployed in different regions (27]. We simulate these dynamic policy goals with 16 distinct prompts (Table 3) that describe demographic feature-based subpopulation priorities. We simulate this environment using a fully anonymized dataset collected by ARMMAN in January 2022, created from a service quality improvement study of 7,668 mothers; we compute transition probabilities from this dataset to simulate actions of real- world beneficiaries.

---

Random
No Action
Base
Defaut
DUN (No Refection
DLM (Refrection)
Wo3 homa tos Reverre
0.0
Income
Hindi
Low Education Ries
Age Deenibution
inoom
Noon homo soca Rowoo
0.5
0.0
Morning MSD Registored
Youngor Blas
Nidolc Aged
Late Workerss
Nosn nommo Revo Revord
sver glven
1.0
tispinet
DLM (io Reflection
0.87 = 008
DLM (tullaction
c.85 0.00e
Murring Eall NGO Registered
Aftieraee NGO Registered
Call NGO Registered
leclrically Ehallerged
Figure 2: Main results. We compute normalized reward (Section 5.2) for each method over 200 seeds, and report the interquartile mean (IQM) and standard error of the IQM across all runs [47]. We compare the topline Base reward policy to the performance of DLM with No Reflection and with Reflection. We also compare to a No Action and Random policy, and a Default policy that demonstrates how the original (fixed) reward function would perform for each new task. Our method is able to achieve near-base reward performance across tasks, and consistently outperform the fixed Default reward policy in completely automated fashion. For some tasks, DLM with Reflection is also able to significantly improve upon zero-shot proposed reward.
Data usage and safety: We use fully anonymized dataset to simulate the ARMMAN public health setting; all experiments are a secondary analysis using this anonymized dataset, and are conducted with approval from the ARMMAN board of ethics. There is actual deployment of this technology demonstration in the real-world A ARMMAN setting. Results from a simulated setting are shown to highlight potential use cases of our proposed system. For details about the dataset and consent of data collection, please see Appendix Section B and Section C
5.2 Tasks, Baselines, and Metrics
We provide experimental results for 16 prompts Table 4)
which include examples emphasizing both single and multiple combined feature categories. We compute per-prompt mean normalized reward (described below) over 200 seeds. We evaluate DLM against several baseline reward functions:
This policy samples arms uniformly random, and has an MNR score of through normalization. No Action: This policy samples no arms at each timestep, typically yielding negative MNR or near-random MNR certain sparse reward tasks. Default: This policy is trained with reward prioritizing all beneficiaries equally, representing the old, fixed ARMMAN goal; thus, reward=1 for state and reward=0 for state 0. Most prior work RMABs assume this reward. Base: Reward is equal to the ground truth fitness function, nluman-speciffe goal function described by the potentially ambiguous language prompt. At test time, we evaluate all methods using Base reward, and consider Base-trained policy as topline (MNR=1), To be clear, each method uses different reward at train time, but uses Base reward at evaluation time. DLM (No Reflection): This version of our proposed method ablates the reflection and iterative refinement process, allowing assess the importance of reflection in improving zeroshot LLM M-geeerattedd reward functions. DLM (Reflection): This the full version of proposed method.
Random:
compute mean normalized reward (MNR) as the average reward achieved per policy, over 200 training seeds, using base reward during test-time evaluation. We normalize each reward R as

---

Task 0: Older Bias
Task Age Distribution Tail Emphasis
Task 15: Technically Challenged
Reward
(oldest_age)
(oldest_age youngest_age)
0.1s Dhusband_ seahaand_phone family_phone)
Reflection)
(oldest_age)
48 (2MooldestLage or oldestagee)
176 (2**youngeet_ag or youngestage)
0.1s (Illterate or husband_phone or evening_calll
0.1s @husband_phane and (NGO ARMMAN or PHC affilation))
DLM (Reflection)
0.28 (oldest, age)
0.18 10s x (oldest.age) age) 1 148 x (youngest. age)
Figure 3: Examples of DLM-generated reward functions vs. ground truth Base reward. Rewards reformatted for clarity; s represents the binary state, numbers are scalar multiplier quantities, and named features, each binary quantities, are shown. In some cases (ex: Older Bias) DLM may identify relevant features zeroshot, and use reflection to refine weights. Alternatively, reflection may help refine features (ex: Age Distribution Tail Emphasis). However, when prompts are ambiguous (ex: Technically Challenged), reflection may not have sufficient signal to effectively iterate; in these cases, additional human feedback may be required.
(R - Rrand)/ (Reuse R. An MNR of O therefore implies a policy's performance equivalent to random allocation, while an MNR of implies performance at the level of base reward policy.
5.3 Training Details
We use the Gemini Pro L.M model [1] from Google to generate reward functions, and train our downstream policy with RL using PPO (Alg. Tline 13) [52 45 and the generated reward function. The downstream policy is trained for 5 epochs, with each epoch containing 100 simulation steps accumulated in a buffer. Thus, we train on a total of 500 samples for each arm. Each such training is run for each proposed reward function, over rounds of reflection and 2 candidate reward functions per round. We evaluate each prompi over 200 seeds, with 50 separate trials per seed and 10 trajectory steps per trial, totaling 100, 000 total steps per task.
5.4 Evaluation of DLM Performance
We present the experimental evaluation of DLM in Figure 2 To the best of our knowledge, we are the first demonstrate LLMs as capable policy designers for resource allocation tasks in public health. We make several observations on the results shown in Figure 2
DLM approaches Base reward performance: We find DLM approaches Base reward performance across a broad range of tasks. Note that mean normalized reward (MNR) scores reported in Figure 2 are normalized as described in Section [..2 this score therefore represents performance above random policy (MNR=0) compared to the topline Base reward policy (MNR=1). We find that DLM (Reflection) achieves average MNR score of 0.93 = 0.006, achieving 90% or higher of base level performance in 1 1/16 tasks, demonstrating to the best of our knowledge the first example of using LLMs to adapt to changing resource allocation objectives in public health through reward design.
Reflection improves performance for many tasks: We find that over the 16 tested prompts DLM (No Reflection) achieves an average MNR of 0.87 + 0.008, while DLM (Reflection) achieves an average MNR of 0.93 0.006. Further, 8/16 of the prompts showed a significant increase in performance after reflection (Appendix Table 5). Notably, we observe that in cases where reflection does not improve performance, DLM (No Reflection) often achieves high zero-shot reward, obviating the need for the reflection stages. For example, we observe for the Age Distribution Tail Emphusis prompt that DLM (No Reflection) achieves an MNR score approximately equal to the normalized Base score. Thus, DI. (Reflection) may unnecessarily iterate on functions that already achieve Base reward, resulting in degraded performance. Such occurrences are rare, and may have little practical difference in the resulting performance, which still achieves near- near-Base score.
We observe that DLM consistently outperforms the Default reward policy, which represents the performance of the previously-used. fixed reward function across the given prompts. We observe an average Default reward MNR score of 0.57 = 0.027, compared to 0.87 = 0.008 for DLM (No Reflection) and 0.93 + 0.006 for DLM (Reflection). Furthermore, we observe that in 11/16 prompts DLM (No Reflection) significantly outperforms Default policy, while in 16/16 prompts DIM (Reflection) outperforms Default (Table in Appendix). In summary, we find that DIM can meaningfully adjust policies, using only language imput, to align more closely to human preferences. Additionally, this highlights that the previous fixed Default
DLM consistently outperforms Default reward:
8

---

Table 1: Average precision/recall of features used in -ropossed reward functions compared to ground truth Base reward function. Comparison between zeroshot DLM (No Reflection) and DLM (Reflection). Cells yellow showed improvement from Reflection with p < 0.1; cells in green showed improvement from Reflection with p < 0.05. Results indicate LLMS are very effective feature extractors for reward function generation. Furthermore, the Reflection module is particularly useful for improving recall rates, as 13/16 tasks showed significant recall improvement with Reflection.
Task
2
5
Prec. (zerushot)
0.49 = 0.028 0.54 + 0.026
0.014
0.45 + 0.022
+ 0.019
0.07
0.07 0.014
0.33 # 0.017 0.38 + 0.015
Prec. (reflection)
0.29 + 0.015
0.87 + 0.017
0.45 + 0.022
0.84 + 0.018
0.54 0.016
0.11 + 0.017
Rec. (zeroshott
0.72 1 0.032
0.64 0.034
1 0.000
0.87 a 0.024
1
0,025
0.11 0.021
0.67 1
Rec. (reflection)
0.84 = 0.026
0.72 + 0.032
1.00 + 0.000
0.92 + 0.020
0.94 + 0.013
0.020 0.84
0.17 = 0.026
0.80 + 0.020
Task
9
10
11
12
13
14
15
Prec. (zerushot)
0.78 0.018
0.014
0.87 0.015
0.93 + 0.012
0.009
0.96
0.010
0.04
0.009
Prec. (rellection)
0.83 a 0.015
0.42 a 0.010
0.93 a 0.013
0.84 a 0.013
0.95 a 0.011
0.96 0.009
0.94 = 0.011
0.06 + 0.010
Rec. (zeroshott
0.65 1
0.014
0.43 0.013
0.97 a: 0.009
0.96 0.009
0.006
0.98
0.006
0.09 1:
Rec. (reflection)
0,74 = 0.013
0.51 + 0.011
0.54 + 0.015
0.98 + 0.006
0.99 4 0.004
1,00 0.002
0.99 0.005
0.13 + 0.020
reward is ineffective in accommodating the diverse prompts, and hence policy goals, described by humans in our simulated setting.
5.5 LLM-Generated Rewards and Reflection in Public Health Settings
In this section we provide insight into LLM-generated reward functions and the reward Reflection module. We find that LLMs can reliably design and improve upon reward functions for public health goals, enabling automated policy tuning in resource-limited settings.
LLMs accurately interpret language prompts for policy shaping: We provide examples of LLM- generated reward functions compared to ground truth Base rewards in Figure [3] We find that DLM can consistently capture the ground truth Base reward, and use reflection to improve upon these proposed functions. However, in cases where language is ambiguous (e.g. "Technically Challenged" example), inherent language ambiguities may result in misaligned proposals. These findings suggest that LLMS can automatically tune public health allocation policies using only sanguage-descriher preferences. Ultimately, while human input may be required to overcome ambiguities of language prompts, DLM allows human mecision-maker monitor state-feature distributions and iteratively provide expert opinion to guide LLM- -rroposdd policies.
LLMs feature extractors for reward proposal in resource allocation tasks: We find LLMs can accurately extract relevant features from language-deseriinn demonstrate these results in Table
resource allocation preferences. We -.ooppood rewards have consistently high precision and recall, effectively capturing the features used in the Base reward function. We find in some cases, for instance Task 15 (Technically Challenged), that ambiguous prompts (see Appendix Table 4) may lead to lower recall ability. Interestingly, however, we still observe reasonable performance in these cases (Figure 2}. suggesting that DLM may identify potentially correlated substitute features even for challenging prompts. Note, additionally, that effective feature extraction does not necessarily imply the correct usage of features; however, the findings suggest that LLMs are effective at the first critical "filtering" step of reward design, particularly in our task of public health resource allocation.
5.6 Limitations
We test our method in purely simulated environment; any potential consideration of real- world deployment would require comprehensive field testing after approvals from relevant ethics boards. Additionally, we test with prompts in English; further testing in local (Indian) languages is required. Furthermore, whereas we present results on DLM with Mfain-of-Thoout
(CoT) reasoning in Table 6, indicating no significant improvement due to CoT, recognize that additional LLM reflection techniques have the potential to improve DLM. Finally, in any potential deployment of the proposed method, we expect that health experts should monitor state-feature distributions intervene on proposed policies, ensuring safety and proper disambiguation of unclear input prompts.
5.7 Ethical Considerations in the Use of DLM
Extending our discussion of the ethical considerations we take in developing DLM (Appendix A C).
we further consider the broader implications of algorithmic resource allocation, particularly public health. We note the need for mitigating data bias in the health setting, especially to

---

minimize harmful discrimination for underrepresented groups [53] and to avoiding data bias by enabling participatory design [34], all key considerations we make in collaboration with our partner NGO for this work. We further highlight the importance of democratized decision-making criteria for algorithmic allocation techniques [55] 56], as well as the importance of complete beneficiary autonomy through guaranteed consent and the opportunity to deny allocations [57]. as we do in this work. Finally, one extension this work that future studies may consider is to directly incorporate
prior work in fairness guarantees in resource allocation settings [58 55 60 61.
Conclusion
In this paper we introduce a Decision-Language Model (DLM) for resource allocation in public health, which can take language prompts describing complex, potentially ambiguous public health policy goals as input, and generate downstream control policies specialized for such goals. We uniquely demonstrate, beyond existing work in LLMs for public health, the strength of LLMs to shape principled, RMAB-based resource allocation strategies, potentially enabling rapid community- driven policy adjustment in critical, essurce-constrainee public health settings. For all future work considering potential deployment, DLM enables health experts to monitor seate-feature distributions intervene on proposed policies and guide LLM generation ensure safety.
Acknowledgments and Disclosure of Funding
This work was supported by the Harvard Data Science Initiative.
References
[1] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[2] Department of Economic United Nations and Social Affairs Sustainable Development. Trans- forming our world: the 2030 agenda for sustainable development, 2015.
[3] World Health Organization et al. Strategies towards ending preventable maternal mortality (epmm). World Health Organization, 2015.
[4] Avishek Choudhury, Onur Asan, and Murari M Choudhury. Mobile health technology to improve maternal health awareness in tribal populations: mobile for mothers. Journal of the American Medical Informatics Association, 28 (6(11):2467-2474 2021.
[5] HelpMum. Helpmum: Preventing maternal mortality in nigeria, 2023. Accessed: 2023-12-30.
[6] ARMMAN. Assessing the impact of mobile-based intervention on health literacy among pregnant women in urban india, 2019. Accessed: 2022-08-12 12.
[7] Margaret E Kruk, Anna D Gage, Catherine Arsenault, Keely Jordan, Hannah H Leslie, Sanam Mader-DeWann Olusoji Adeyi, Pierre Barker, Bernadette Daelmans, Svetlana V Doubova, et al. High-quality health systems in the sustainable development goals era: time for a revolution. The Lancet global health, 6(11)eel 195-e1252, 2018
[8] Rob Baltussen and Louis Niessen. Priority setting of health interventions: the need for multi- criteria decision analysis. Cost effectiveness and resource allocation, 4(1):1-9, 2006.
[9] Leonoor Gräler, Leonarda G. M. Bremmers, Pieter Bakx, Job van Exel, and Marianne E van Bochove. Informal care in times of public health crisis: Objective burden, subjective burden and quality of life of caregivers in the netherlands during the covid- 19 pandemic. Health & Social Care the Community, 2022.
[10] Katrina Deardorff, Arianna Rubin Means, Kristjana Hrönn Ásbjörnsóóttir, and Judd L. Walson. Strategies to improve treatment coverage in community- public health programs: systematic review of the literature. PLoS Neglected Tropical Diseases, 12, 2018.
10

---

[11] Turgay Ayer, Can Zhang, Anthony Bonifonte, Anne C Spaulding, and Jagpreet Chhatwal. Prioritizing hepatitis treatment in us prisons. Operations Research, (113):853-773) 2019.
[12] Siddharth Nishtala, Lovish Madaan, Aditya Mate, Harshavardhan Kamarthi, Anirudh Grama, Divy Thakkar, Dhyanesh Narayanan, Suresh Chaudhary, Neha Madhiwalla, Ramesh Padman- abhan, et al. Selective intervention planning using restless multi-armed bandits to improve maternal and child health outcomes. Proceedings of the 2023 International Conference Autonomous Agents and Multiagent Systems, page 1312- 1320, 2023.
[13] Sarang Deo, Seyed Iravani, Tingting Jiang, Karen Smilowitz, and Stephen Samuelson. Im- proving health outcomes through better capacity allocation in community-bad based chronic care model. Operations Research, 61(6):177-12244 2013.
[14] Shresth Verma, Gargi Singh, Aditya Mate, Paritosh Verma, Sruthi Gorantla, Neha Madhiwalla, Aparna Hegde, Divy Thakkar, Manish Jain, Milind Tambe, et al. Expanding impact of mobile health programs: Saheli for maternal and child care. Al Magazine, 44(4):363-376 2023.
[15] Lyndsay A Nelson, Shelagh A Mulvaney, Tebeb Gebretsadik, Yun-Xian Ho, Kevin Johnson, and Chandra Y Osborn. Disparities in the use of a mhealth medication adherence promotion intervention for low-income adults with type 2 diabetes. Journal of the American Medical Informatics Association, 23(1):12-18, 2016.
[16] Yan Jin, Lucinda Austin, Santosh Vijaykumar, Hyoyeun Jun, and Glen Nowak. Communicating about infectious disease threats: Insights from public health information officers. Public Relations Review, 45(1):167-177, 2019.
[17] Jimmy Phuong, Patricia Ordóñez, Jerry Cao, Mira Moukheiber, Lama Moukheiber, Anat Caspi, Bonnielin K Swenor, David Kojo N Naawu, and Jennifer Mankoff. Telehealth and digital health innovations: A mixed landscape of access. PLOS Digital Health, 2(12):e0004401 2023.
[18] Dhruv Shah, Blažej Osiáski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, pages 492-504. PMLR, 2023.
[19] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
[20] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du Clinton Wang. Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making Advances in Neural Information Processing Systems, 35:31 199-3122,,,, 2022.
[21] Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations, 2024.
[22] Malik Sallam, Nesreen A Salim, Muna Barakat, and B Ala'a a. Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the ad advantages and limitations. Narra J, 3(1), 2023.
[23] AAKANKSHA NAIK, CARLA ALVARADO, LUCY LU WANG, and IRENE CHEN. Nip for maternal healthcare: Perspectives and guiding principles in the age of llms. arXiv preprint arXiv:2312.11880) 2023.
[24] Aditya Mate, Lovish Madaan, Aparna Taneja, Neha Madhiwalla, Shresth Verma, Gargi Singh, Aparna Hegde, Pradeep Varakantham, and Milind Tambe. Field study in deploying restless multi- armed bandits: Assisting non-profits in improving maternal and child health. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12017-12025. 2022.
[25] Jackson Killian, Arpita Biswas, Lily Xu, Shresth Verma, Vineet Nair, Aparna Taneja, Aparna Hegde, Neha Madhiwalla, Paula Rodriguez Diaz, Sonja Johnson- Yu, et al. Robust planning over restless groups: engagement interventions for a large-scale maternal telehealth program. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, 2023,
11

---

[26] Kai Wang, Shresth Verma, Aditya Mate. Sanket Shah, Aparna Taneja, Neha Madhiwalla, Aparna Hegde, and Milind Tambe. Scalable Decision-focusees learning in restless with application to maternal and child health. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12138-12146, 2023.
bandits
[27] ARMMAN. Annual report 2020-2021, 2021.
[28] Sawan Rathi, Anindya S Chakrabarti, Chirantan Chatterjee, and Aparna Hegde. Pandemics and technology engagement: New evidence from m-health intervention during covid-19 india. Review of Development Economics, 26(4):2184-23) 221172 2022.
[29] Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of applied probability, 25(A))287-295 1988.
[30] Richard R Weber and Gideon Weiss. On an index policy for restless bandits. Journal of applied probability, (7(3):637-488, 1990.
[31] Kevin D Glazebrook, Diego Ruiz-Hernande,, and Christopher Kirkbride. Some indexable families of restless bandit problems. Advances Applied Probability, 38(3):643 672, 2006.
[32] Kevin D Glazebrook, David Hodge, and Chris Kirkbride. General notions of indexability for queueing control and asset management. The Annals of Applied Probability, 2011.
[33] Jackson Killian, Lily Xu, Arpita Biswas, and Milind Tambe. Restless and uncertain: Robust policies for restless bandits via deep multi-agent reinforcement learning. In Uncertainty in Artificial Intelligence, pages 990-1000. PMLR, 2022.
[34] Milind Tambe. Ai for social impact: Results from deployments for public health and conversa- tion. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
[35] Niclas Boehmer, Yunfan Zhao, Guojun Xiong, Paula Rodriguez-Diaz, Paola Del Cueto Cibrian, Joseph Ngonzi, Adeline Boatin, and Milind Tambe. Optimizing vital sign monitoring in resource-constrained maternal care: An rl-based restless bandit approach. arXiv preprint arXiv:2410.083377, 2024.
[36] Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pages 2601-2606. Cognitive Science Society, 2009.
[37] Serena Booth, W Bradley Knox, Julie Shah, Scott Nickum, Peter Stone, and Alessandro Allievi, The perils of tril--anderroo reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5920-5929 2023.
[38] Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, and Matthew E Taylor. Reinforcement Learning from Demonstration through Shaping. International Joint Conference on Artificial Intelligence, 2015.
[39] Ahmed Hussein, Eyad Elyan, Mohamed Medhat Gaber, and Chrisina Jayne. Deep reward shaping from demonstrations. In 2017 International Joint Conference on Neural Networks (IJCN), pages 510-517. IEEE, 2017.
[40] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2022.
[41] Yuqing Du Olivia Watkins, Zihan Wang. Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint Xrxi::2302.06692. 2023.
[42] Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. Auto me-reward: Automated dense reward design with large language models for minecraft. arXiv preprint arXiv:2312.0923:8 2023.
12

---

[43] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[44] Khaled Nakhleh, Santosh Ganji, Ping-Chun Hsieh, I Hou, Srinivas Shakkottai, et al. Neurwin: Neural whittle index network for restless bandits via deep rl. Advances in Neural Information Processing Systems, 34:828-835 -3988 2021.
[45] Yunfan Zhao, Nikhil Behari, Edward Hughes, Edwin Zhang, Dheeraj Nagaraj. Karl Tuyls, Aparna Taneja, and Milind Tambe. Towards a pretrained model for restless bandits via multi- arm generalization. International Joint Conference Artificial Intelligence, 2024.
[46] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language, 2020.
[47] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
[48] Shresth Verma, Gargi Singh, Aditya Mate, Paritosh Verma, Sruthi Gorantla, Neha Madhiwalla, Aparna Hegde, Divy Thakkar, Manish Jain, Milind Tambe, al. Increasing impact of mobile health programs: Saheli for maternal and child care. In Proceedings of the Conference on Innovative Applications of Artificial Intelligence (IAAI), 2023.
[49] Roderick Seow, Yunfan Zhao, Duncan Wood, Milind Tambe, and Cleotilde Gonzalez. Improving the prediction of individual engagement in recommendations using cognitive models. arXiv preprint arXiv:2408. 16147, 2024.
[50] Yunfan Zhao, Tonghan Wang, Dheeraj Nagaraj, Aparna Taneja, and Milind Tambe. The bandit whisperer: Communication learning for restless bandits. arXiv preprint arXiv:2408.05688 05686, 2024.
[51] Shresth Verma, Aditya Mate, Kai Wang. Neha Madhiwalla, Aparna Hegde, Aparna Taneja, and Milind Tambe. Restless multi-armed bandits for maternal and child health: Results from Docision-focusee learning. In AAMAS, pages 1312-1320, 2023.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633 7074064477777 2017.
[53] Haylee Lane, Mitchell Sarkies, Jennifer Martin, and Terry Haines. Equity in healthcare resource allocation decision making: systematic review. Social science & medicine, 175:11-27, 2017.
[54] Alvin Rajkomar, Michaela Hardr, Michael D Howell, Greg Corrado, and Marshall H Chin. Ensuring fairness machine learning to advance health equity. Annals of internal medicine, 169(12):666-822 2018
[55] Lalla Aida Guindo, Monika Wagner, Rob Baltussen, Donna Rindress, Janine van Til, Paul Kind, and Mireille M Goetghebeur. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost effectiveness and resource allocation, 10:1-13, 2012.
[56] Norman Daniels. Accountability for reasonableness: Establishing a fair process for priority setting is easier than agreeing on principles, 2000.
[57] Hellen Ransom and John M Olsson. Allocation of health care resources: Principles for decision- making. Pediatrics in Review, 38(7):300-32,, 2017.
[58] Dexun Li and Pradeep Varakantham. Efficient resource allocation with fairness constraints in restless multi-armed bandits. In Uncertainty in Artificial Intelligence, pages 1158-11677 PMIR, 2022.
[59] Shufan Wang. Guojun Xiong, and Jian Li. Online restless multi-armed bandits with long- term fairness constraints. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15616-15624, 2024.
13

---

[60] Shresth Verma, Yunfan Zhao. Sanket Shah, Niclas Boehmer, Aparna Taneja, and Milind Tambe. Group fairness in Uncertainty in Artificial Intelligence.
settings for restless bandits. In The 40th Conference on
[61] Shresth Verma, Niclas Boehmer, Lingkai Kong, and Milind Tambe. Balancing act: Prioritization strategies for llm-designed restless bandit rewards. arXiv preprint arXiv:2408. 12/12, 2024.
[62] David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers laboratory for interpretability, 2023.
[63] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. hutps//arrv..rg/bb//211. 1.666lvv3,. November 2022.
[64] Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking Like Transformers, July 2021.
14

---

Appendix
A Social Impact Statement
The presented methods carries a positive societal impact, allowing non-profit organizations in the public health domain efficiently allocate limited health resources to large beneficiary populations, and to effectively adapt policy outcomes. We illustrate the application of our method in collaboration with an India-based public health organization promoting preventive care for pregnant mothers, that currently relies on RMAB policies for ensuring spread of important health information to low resource populations.
Our proposed methods, tested purely as a technology demonstration and entirely in simulation, do not have direct negative societal implications. However, reinforcement learning agent training should be done responsibly, especially given the safety concerns associated with agents engaging in unsafe uninformed exploration strategies. While the public health domain we considered does not have concerns associated with extreme environments, when adapting our proposed methods for other domains, ensuring robust approach to training reinforcement models is critical. All work in simulation was conducted in collaboration with ARMMAN's team of ethics, and any future consideration of deployment would require a real-world evaluation together with the domain partner, which may reveal further challenges to be addressed.
B ARMMAN Dataset Description
To conduct the secondary analyses presented this study, we use a real dataset collected by ARM- MAN (2022). This dataset was created from comprehensive quality improvement study carried out by ARMMAN in 2022; the study involved 7,668 beneficiaries who were enrolled over a study period of 7 weeks. Throughout this period, 20% of participants were allocated at least one direct interaction, such as live service call from a healthcare volunteer. Engagement during this period was used to compute transition dynamics.
B.1 Associated Features and Protected Information
Each beneficiary enrolled in the ARMMAN study was described by 43 features, which include data on age, income, education, preferred call times, and ownership of a phone, among other factors. These features are entirely anonymized. ARMMAN, through the mobile voice call service initiative, works specifically with socially disadvantaged populations. However, ARMMAN does not collect constitutionally protected and sensitive categories such as cast and religion the recorded beneficiary features. In addition to such categories not being available in the ARMMAN dataset, we further ignore -anoymmized information regarding pregnancy history in our study to focus primarily on broader neciomcconomic categorizations such as age and income. We additionally worked with ARMMAN to ensure that the evaluated prompts, tested only in simulation, are closely aligned with health expert goals and challenging real-world scenarios where prioritizing specific underrepresented subpopulations is desired.
C Dataset Consent for Collection and Analysis
We provide further information regarding consent for data collection and analysis below.
C.1 Secondary Analysis
In this study, we conduct secondary analysis of the ARMMAN dataset. We use trajectories, which contain the historical engagement behavior for beneficiaries enrolled in the ARMMAN study. Engagement states are computed as binary values, which are determined as "engaged" if a beneficiary listens to a voice message for more than 30 seconds. The average length of each voice message is 115 seconds. For each beneficiary, we utilize only an estimated transition dynamic matrix, computed using the historical trajectories for T = 7 weeks. We only interface with this estimated transition matrix per beneficiary; for each sampled arm then, we observe only the provided transition matrix and the associated, anonymized, arm features. We then use these estimated transition matrices to
15

---

simulate agent behavior; we do not deploy the proposed method to the ARMMAN program. These described secondary experiments are conducted completely in simulation using estimated transition matrices to simulate plausible real-world agents.
C.2 Data Collection Consent
Full consent for data collection is obtained from each ARMMAN study participant; the process of data collection is explained in detail to each participant when obtaining consent, prior to data collection. The dataset was completely anonymized by ARMMAN, and data exchange from ARMMAN to the researchers was regulated through clearly defined exchange protocols including amonymization, read-only researcher access, restricted use of data for research purposes only, and approval by the ARMMAN ethics review committee.
C.3 Simulated Findings and Equal Distribution of Resources
We note again that all experiments presented in this work were conducted in simulation: we attempt simulate a plausible real-world public health setting to gain insight into strengths and weaknesses of the proposed approach. We additionally note that the intended goal of this research is to identify techniques more easily align algorithmic resource allocation methods to specific goals of public health experts. However, the service call program is always equally accessible to all beneficiaries (e.g.. participants can always request service calls via free missed call service) and health information disseminated through the program will never be restricted for any individual enrollee. That is, all participants will receive the same weekly health information by automated message regardless of whether they are scheduled to receive service calls or not. The proposed system is intended for use only cases where separate, additional service call resources are available to help especially underrepresented groups. In these cases, the proposed system may help align algorithmic resource allocation of the sepurate additional resources more closely to desired public health expert policy goals.
D Outcome Analysis Algorithm
Below, we provide the algorithm for the state-feature distribution analysis used in the DLM reflection stage to select top candidate rewards. Please see subsscciiio 4.4 for more details on the Reflection module and state-feature analysis procedure.
Algorithm 2 OUTCOMEANALYSIS
Input: Trained policy, critic net. 0, Q, action-charge X, feature matrix Z E R2 Xm Return: OutString
2: Hyperparameters: Sim. steps rts
## Step 1: Simulate nı, timesteps under 0
4: Init. evaluation totals + [0] x N, percentage distribution P G OutString empty string
5: for timestep 1 n,
Sample actions: a, ~ (s. X) E [N], simulate actions: s', Simulate(s, a, R. z),
7: accumulate states: $ < S + s, update states: s <- s'
8: ## Step 2: Compute state outcome distributions
9: Define feature groups within
10: for each group E G do
Compute state distribution over feats: P(g) + Eee ES - append to OutString: "y: P())% of reward;"
11:
E Compute Resources
use the Gemini Pro LLM model from Google to generate reward functions. We use 8 CPUs cores (Intel Cascade Lake Processors) for all tasks.
16

---

Deep RL for RMABs
The Lagrangian relaxation allows to disentangle the learning of arm policies and action charge A. To learn arm policies, it is common to use RL approaches such as tabular Q-learning, deep Q-learning. or policy gradient algorithms/44) In this paper, we use PPO [52]. policy gradient algorithm with an actor-critic structure. Specifically, the Q-function can be updated as follows:
(2 and 5- -1 (sn) an)
+ 0g (R(sn) ACa, max Q(5) a))
where Og is the learning rate. For an actor-critic methods, after performing an update of the Q-function in the critic, one may compute the advantage estimate:
ass -
In practice, the values A(s, a), (ss,a), V(s) will be evaluated based on the current policy æ. Intu- itively, the advantage estimate tells us how much better a is compared the current policy æ. The advantage estimate is then plugged into the policy gradient to update the actor. To learn an action charge X, we follow an updating rule (33
N ΣΣ
B
A - aλ
+
E
Cn,b
1-
n=1
t=
where Cn, is the cost the action taken by the optimal policy on arm n in round t, and a the step size.
G Proof of Proposition 1
Here we give a proof of Proposition Proof sketch. To prove this, first show that there exists parameters O which can implement the optimization step of a simple and efficient gridsearch algorithm ITERATED in the discretized weight space. We can do this by leveraging previous work connecting transformers
64 Specifically, the RASP 664 programming language and TRACR compiler [62] enables compiling program into transformer weights, giving direct proof of existence by construction. We then prove the correctness of this algorithm using the monotonicty property. Finally, prove upper bounds the runtime of this algorithm, yielding the desired sample complexity bound in Proposition] [
and traditional algorithms [62
63
First we state gridsearch algorithm ITERATEDLINESEARCHE In notation, w; is the entry of w E R" corresponding to index E [n). Recall that there is a total of (Spp(w")) weights corresponding to the non-zero entries of w", and that searching over 2K possible values for each weight. This means in total there (2K) (Supp(w")) total possible weight combinations.
17

---

Algorithm 3 ITER ATEELDTSAAEEEER
1: Input: Discretized grid of possible weights S, discretization level a evaluation function V(w) =
V(., w²), support Supp(u' w²) )
2: Return: Optiizzdd weights maximizing (ww).
3: ## Step 1: Set w min value
4: w4, w/ min S Vi Supp(w")
5: i=0
6: converged = False
7: while not converged do
8:
## Step 2: Call external oracle V(w)
9: Vwn Vw', + V(w, w"), ((w,, w²)
10: Wi, W/ converged, i &- OPTIMIZESTEP(S w, w', Va, Va 0,
11:
## Step 3: Return w as output
12:
wut
13:
return uout
We now state the actual optimization subroutine of INTRATEEDLLNESSARCHH OPTIMIZESTEP.
Algorithm 4 OPTIMIZESTEP
I: Input: {(5upp(=6)), w, w', ((w, w"), V(w', w²)
Return: Updated weights w, w', where Vww,, w²)) > V(rr w²), convergence status, and current index
3: if u(w') > n(w) and 0o; < max S then
4:
+ w/ ## update w' if has higher value
5: wo W-a
6: return w. w., False,
return w. w' and converged False,
7: else
9: ir > 1| (lo then
10:
return w', True, ## return ws, w' and converged True, i
11: else
return w. w', False. ## return w. w' and converged Fulse,
Given oracle access to a valuation function V(w, w²), OPTIMIZESTEP can be implemented in transformer with constant depth and OQI (loK) ([ll) width.
Proposition 2.
Proof. Using the RASP programming language 64], we show that OPTIMIZESTEP can be imple- mented in transformer. In each forward pass of the transformer, we take new weight combination and pass this to the loss function in order to evaluate its performance on line Algorithm(]] Note that the only state needed be kept track of between forward passes of the transformer is the current hyperparameter dimension can be stored in nK hidden dimensions, the size of g(Supp((0)) upper bounds every else as n.. The conditional statements on line and line 9 of Algorithm 4can be implemented in two sclect-aggregate blocks [64), corresponding two attention layers. Line 8 requires an additional fully connected layer after the first attention layer, and a finally an output layer is needed to implement each return statement, giving four layers in total. The select block roughly corresponds to the attention matrix in the transformer, and is a function that allows calculating conditionals by taking in three arguments: keys, queries, and boolean predicate p. Here, we would pass a, (/Supp(w5)), w, w', V (w, w*), V(w', w²) the input sequence, and construct three selectors corresponding to the boolean predicates on line 5, 6, and 8, respectively. An aggregate can then be performed to get the result of the predicate, controlling the program flow for selecting.
S[Supp(w;)),
w', V(w, w²) V(w', ww). This information
Proposition 3. ITERATEDL converges in O(I]w" (lo time.
Proof. Inside each iteration of the while loop, line either increases to i++1 (but never decreases) or w/ increases to w'a. There are 2K possible values for wi for every given i. There are |["" llo possible
18

---

values for i and 2K possible values for w', ensuring that the algorithm converges in (1I llo time.
Proof of Proposition 1 We consider iterated line search (Algorithm B above with search space S and scale a. In Proposition 3 we have shown that this algorithm converges in time ( and that this algorithm can be implemented by transformer. remains to show that it returns v = arg max w E [5(pp(((")) V (we, w*).
By monotonicity, it is clear that if the final output is such that w arg minges g (w"), for every i E Supp(w*), ", then whut wh Therefore, we prove the coordinate-ve wise optimality below.
Consider the value of w in the while loop, line 10 when is increased to i++1 by the OPTIMIZESTEP function.
Case I: Suppose the coordinate wi max 5 then we have V(w, w²) 2 V V(w, w") for every t such that tú; = Wy for i j and w; E S. Therefore, by the contrapositive of the monotonicity property, this implies wy arg minges lg (w")) This establishes optimality for the co-orindate since w W/g.
Case 2: Now suppose that w; < max S. A similar argument as above establishes that [oy - w I lg - w for every g S such that g S w². The loop breaks since the condition in line 8 is false. Again, by the contrapositive This implies that w < aww and hence w; will S larw; w S (a"w;= - -w;l for E N. Thus, we demonstrate that even in this case w, arg mittges lg w
the monotonicity property, we must have [u; w/'l & Jawu; w.l-
Therefore, coordinate wise optimality holds for every Supp(w*), completing the proof of the claim.
H Recall of Logical Combinations
We analyze the logical reasoning ability of the LLM to combine atomic features together in order to recover the ground-truth reward in Table 2
Table 2: Recall of logical combinations of features for multi-feature prompts. We consider multi- feature prompts 4-15, and report the recall compared to Base reward for accurately emulating behavior of Base reward, Note that we consider only LLM generations with high feature recall, i.c. those proposed rewards that include, at minimum, the features used in the corresponding ground truth Basc reward.
Task
6
7
8
9
Logic Recall
0.85 # 0.017
0.75 0.023
0.69 + 0.037
0.69 = 0.027
0.84 = 0.043
0.66 0.04
Task
10
11
12
13
14
15
Logic Recall
0.64 + 0.053
0.85 + 0.014
0.87 + 0.013
0.9 a 0.01
0.87 + 0.011
0.62 + 0.102
I Sample Input Prompt
A sample input prompt with full context inputs is shown below.
Create a Python revard function for AL in phone call resource allocation to India, with objective of prioritizing higher states While
still prioritiziag all, elightly focus on the oldest by age distribution... The function should
"state' (value either 0,1) features 'agent_feate (langth 43 array) to direct the RL agent. Here in description of the features you may use:
Indez Name DataType [rensitive feature hidden]
Agee 10-20 Binary
8. Ages 21-30 Binary
9. Ages 31-40 Binary
10. Agea 41-50 Binary
11. Agea 51-60
Binary
12. Speaks Bindi Binary
13. Speaks Marathi Binary
Speaks Gujurati Binary
19

---

15. Speaks Kanzada Binary
16. Education level 1/7
illiterate Binary
Education
Grade Completed Binary
18. Education level 3/7
6-9th Grade Completed Binary
19. Education
4/7
Passed Binary
20. Education level 5/7 12th Grade Passed Binary
21. Education level 8/7 -- Graduate Binary
22. Education level 7/7
graduate Binary
23. Phone ovner (a.g.. uoman) Binary
(e.g., husband) Binary
26. Phone owner (e.g., family) Binary
26. To called from 8:30am-10:304
Binary
27. To be ealled fron 10:30am-12:30pm Binary
28. To be called from 3:330p Binary
To called fron 3:30pm-6:30pp Binary
30. To be called fron 5:30pn-7:30pm Binary
7:30pa-9:30pp Binary
32. NGO Binary
33. ARMMAH Binary
34. PBC Binary
35. Incone bracket -1 (no incone) Binary
Income bracket 1 (e.g., 0-5000) Binary
37 Incone bracket 2 (e.g., 5001-10000) Binary
(e.g., 10001-15000) Binary
39. Incone bracket 4 (e.8., 15001-20000) Binary
40. Incone bracket 5 (e.g., 20001-25009) Binary
41. Incone bracket 6 (e.g.. 25001-30000) Binary
42. Incone bracket 7 (e.g., 90990-999999)
Binary
1. Write a simple, single- Python reward function. Exclude the word 'return' your code triple sigus:
exclude son-standard libraries.
S$8 (YOUR FUNCTION]$6$.
2. Provide an explanation on how this function prioritizes the specified age group Fornat your explanation with triple & signs: MEALYOUR EXPLAKATION] Note that HIGHER states are alvays proferred, sa ansure revard increases as revard
alvays positive Avoid using bitwise operators &, 1. Using and, or instead. emphasize agents
increasing
Example Prompt: While prioritizing
older
richer
Let's think about this step by step. We want to give reward only for agents that older, which correspands feature reature 42. This carraspanda to condition of (agent_ agent_,eate [42]).
and rich which corresponds
give and
addition, alvays only is 1, since the agent gets reward only when it is in a listening state. Therefore, agent_feats[42]] [42]).
should be:
(agent_
Exanple Respense:
Pythan Code: '$8$
0.1
(agent_ and
agent_ [42]) $88
Explenation: KRRThis function gives higher revarde higher states higher ages, aligning with the goal to revard older individuals with higher states . XXX for apecified geal: prioritizing all, alightly rocus the oldeat distribution.. Here are your best previous attempts:
Cone up with unique new
Prompts and Base Reward Functions
provide a full list of the tested prompts and the ground truth Base reward functions in Table 3 We additionally include a full list of the features available for each arm (and used within proposed
reward functions) in Figure 4
We note that we evaluate on a wide variety of tasks within the real of resource allocation for maternal health. In particular, we test across feature characteristics such as age, income, spoken languages (an important characteristic indicating ethnicity in the Indian subcontinentuuh), education, and other relevant features that may be considered in real-world allocation tasks. We provide these tasks directly as context to the LLM, in addition to relevant feature characteristics and their corresponding descriptions. Per seed, we the randomness in sampled set of arms from ARMMAN'S dataset, the deep RI. neural network initialization, batch sampling from the dataset, and the sampling of states for each arm during evaluation. However please note that we cannot control the randomness of the Gemini LLM generation only have access through the Google API interface.
20

---

[senaitive featurea 0-6 hidden, not used in proupts]
7. Ages 10-20 Binary
8. Ages 21-30 Binary
9. Agea 31-40 Binary
10. Agea 41-50 Binary
Ages
Binary
12. Speaks Rindi
Binary
13. Speake Marathi Binary
14. Speaks Gujursti Binary
15. Speaks Kannada Binary
Education
illiterate Binary
17. Education level 2/7
1-5th Grade Completed Binary
Education
Completed Binary
19. Education level 4/7
10th Grade Passed Binary
20. Education level 5/7
12th Grade Passed Binary
21. Education level 6/7
Graduate Binary
22. Education level 7/7
Post graduate Binary
(e.g., vonan) Binary
24. Phone owner (e.g., busband) Binaryy
Phone owner (e.g., family) Binary
26. To be called fron 1:30am-10:300m
Binary
27. Te be called fron 10:30am p2
Binary
28. To called
12:30pm-3:30pm Binary
29. To be called fron 3:30pn-5:30pm Binary
5:30pm-7:30pm Binary
31. Tо be called from 7:30pm-9:30pa Binary
Binary
33. ARMNAX Binary
34. PHC Binary
Incone bracket
(no income)
36. Incone bracket 1 (e.g., 0-5000) Binary
(e.g., 5001-10000) Binary
38. Incone bracket 3 (e.g., 15000)
Binary
Incone
(e-g., 15001-20000)
Binary
40.
Incone bracket 5 (e.g., 20001 -20000
Binary
41.
Incone bracket (e.g., 26001-30000
Binary
bracket (a.g., 30000-999999)
Binary
Figure 4: Full list of features with corresponding feature indices used Base and LLM-proposed reward functions. See Table for full list of language prompts and ground truth Base reward functions.
K Extended Results and Statistical Tests
K.1 Numerical Results
We show the full set of tasks and numerical results for the mean normalized reward score in Table 4
21

---

Table Full task list and ground truth Base reward functions.
idx
Label
Full Prompt
Base Reward Function
Older
While prioritizing all,
lambda feature, state: state 0.1 + +f (statt) 2.0 featur[[1]]
Slighlly focus on the oldest by distribution.
Low Income
While still prioritizing all, slightly focus on the bottom 12.5%
lambda feature, state: state 0.1 Hi (state) 2.0 fenture] 36]
bracket
diarriuution
2 Hindi Speakers
While still prioritizing all. who
lambda feature, state: state 0.1 if (state) * 2.0 (ffaeeeere)112)
slightly spcak Hindi.
3 Low Education Bius
While still pricritizing all, slightly weight those had low education.
lamhda feuture, 0.1 [f_(sttte) 2.0 (f(aturur((11))
Age Distribution Tail Emphasis
While still prioritizing all, slightly focus on both the youngest and oldest by age. While pricritizing slightly prefer the income bracket bounds for the middle 40% population. While still prioritizing all, slighlly faver those do not own their own phone.
lamhda feature, [eature]7]] (7))
if 2.0 (feature(
Middle Income
lambda,
(state) (festuur]]]8 feature| 39] or feuture(40])
6 Non-Phone Owner
lambda feature, state: state 0.1 (state) 2.0 if (feature([2) or feature (25)
who
7
Low Inceme + Younger Bias
While still prictitizing all, slightly prioritize impoverished younger mothers by combining the distribations of 'age' and "education".
lambda feature, state: state 0.1 + [ftatt))) 2.0 [f_[ttaure]]] and feature [6])
8 Marathi Speakers Middle
While still prioritizing all, slightly focus those Marathi speakers middlc-aged mothers. While still prioritizing all. slightly emphasäze heneficiaries who likely work early in the morning late night. While still priocitizing all. slightly weight the lowest income, bracket groups, absolute lowest earners the population.
lamada feature, state: state 0.1 (state) 2.0 (e:DD=ff_fffaaa (fenture(13] and (feature[9) or (attre[[0]]
Aged
9 Early and Late Workers
lambda feature, feature (28))
0.1 if (stane) 2.0 [[(fetnure]266
Critical Low Income
lambda feature, statc: state 0.1 +I (statt))) 2.0 it [[eaturre]]3] or featurel [3] or feuture[37])
Early Morning Call Registered
While prioritizing
feature, stale: state (state) "af===ffeuuaaaa (Feature(26) feature (32)
slightly advantage those who prefer being called before 10:30am 'slet' at an NGO.
registered
Morning Call NGO Registered
While still prioritizing all. slightly advantage prefer heing called hetween
lambda feuture, state: state * 0.1 (state) + 2.0 2.000 [f_(feeture]21 and feature[S3]
3::30m2m:300pp 2:30pm and registered at an NGO.
13 Afternoon Call NGO Registered
While still prioritizing all. slightly advantage prefer being called hetween 12:30pm-3:33pp and
lambdda feature, state: state * 0.1 (st(ttt)) * 2.0 (f(aaauure[88) and feature] 321)
registered
Evening Call + NGO Registered
While still prioritizing all. slightly advantage those who prefer being called after 7PM "soo'' registered at an NGO. infer While siill pricritizing technical challenges in reaching the phone that could indicate beneficiarics give slight preference.
lambda feature, state: state * 0.1 + (statc) * 2.0 (fcamur[33] and Reature]32])
Technically Challenged
lamhda fenture, feature|25] (25)
0.1 (state) 2.0 if (feature(ee44
22

---

Table 4: Full prompts and numerical results.
Idx.
Label
Full Prompt
Base (topline)
No Action
Default
DLM (No Re-
DLM (Rellec-
flection)
tion)
Older Bins
While still priortizing all, slighly focus the oldest by age distibuttoon
1.00
0.02 + 0.072
0.41 +
0.91 + +00029
0.99 = 0.005
Low Income
While still prioritizing all. slightly focus on bottom 12.5% of the income income_brackkt bracket distribution.
1.00
-0.02 + 0.073
0.51 + 0.122
0.83 + 0.046
0.97 1 0,014
Speakers
still prioritizing all. slightly focus on those who speak Hindi.
1.00
-0.12 0.067
+ 0.093
0.79 + 0.058
0.98
3 Low Education
While still priocitizing all. slightly weight those who have
1.00
-0.00 + 0.073
0.46 0.124
0.78 + 0.063
0.74 1 0.058
Bias
had low education.
Age Distribution
While still priotitizing all, slightly focus on both the youngest oldest age.
1.00
00002 # 0.069
0.111
1.00 0.018
0.92 = 0.018
Thil Emphasis
5 Midklee Income
While still prioritizing all. slighll prefer the income bracket bounds the middle 40% of the population.
1.00
-0.08 + 0.067
0.65 + 0.097
0.77 + 0.057
0.88 + 0.044
6
Non- Phn-P
While still priottiizing all, slightly favor those women who do not own their own phone.
1.00
-0.07 0.067
0.50
0.87 + 0.028
0.97 = 0.019
Owner
Low Income Younger Blas
While still prioritizing all. elightyy prioritize impoverished younger mothers by combining the distributions "age" and "education.
1.00
0.03 0.075
0.40
0.95 0.014
0.98 0.006
8 Marathi Speakers Middle Aged
While still priotitizing all. Marathi-
1.00
0.074
4.511
0.82 0.036
0.89 0.023
slightly
speakers with middle-aged mathers.
9 Early and Late
While still priocitizing all. slighlly emphasize beneficiaries who likely week early the morning and late at night.
1.00
-0.11 : 0.065
0.69 + 0.093
0.81 + 0.038
0.87 = 0.038
Workers
10 Crirical Low Income
While still prioritizing all, 1.00 slightly weight the lowest income bracket grops, the absolate lowest camers in the population.
.0.08 + 0.068
0.100
0.93 0.002
0.93 + 0.015
11 Early Morning Call NGO Registered
While still prieritizing all. slightly advamtage those who prefer heing called before 10:30am 'slat' and are registered NGO.
1.00
-0.12 0.065
0.70
0.87 0.003
0.96 2 0.026
Morning Call NGO Registered
While still priocitizing all. 1.00 slightly advantage those who peefer being called between 10:30am- 12:3ppm and are registered at an NGO.
-0.07 1 0.066
0.1099
0.020
0.87 0.025
Afterooon Call NGO Registered
While still prioritizing all, 1.00 slightly advantage those who prefer being called between 12: 3:000mmm and are regis- tered at an NGO.
.0.11 0.062
0.72 0.085
0.92
0.024
Evening Call NGO Registered
While still prioritizing all. 1.00 slightly advantage those who prefer being called after 7PM NGO.
-0.06 + 0.070
0.511
0.67 + 0.057
0.90 0.031
"sloc' registered
Technically Challenged
While still prioritizing all, infer 1.00 technical challenges reaching the phone could indicate risk' beneficiaries and give slight
-0.14 a 0.060
0.64 a 0.097
0.79 a 0.049
0.92 a 0.019
preference.
Avg.
Average MNR
Average MNR for given 16 tasks 1.00
-0.06 0.017
0.57 1 0.027
0.85 1 0.008
0.92 1 0.006
23

---

K.2
Statistical Tests
Table 5: One-tailed I-tests comparing MNR scores against the null hypothesis that the mean MNR
scores are less than or equal to the comparison group.
Label
Default
DLM (No Rellec- tion)
DLM (Reflection
DLM (No Reflec- tion) >Deaaul??
DLM (Reflection) >Default?
DLM (Rellection)
SDLM (No Re-
flection)?
Older Bias
0.41 + 0.130
0.91 0.029
0.99 + 0.005
P p-00.001
p p-00.01
P 0.004
Low Income
0.51 + 0.122
0.83 0.046
0.97 = 0.014
0.007
p p=00001
p 0.002
Hindi Speakers
0.71 + 0.098
0.058
0.98 + 0.042
p= 0.233
p= 0.004
p 0,004
Low Education
0.46 4: 0.124
0.78 a 0.063
0.74 a 0.058
p= 0.011
p= 0.021
0.680
Bias
Age Distribution
0.52 0.111
1.00 0.018
0.92 2 0.018
p p000001
P p000001
0,99
Tail Emphasis
Middle Income
0.65
0.057
0.88
p 0.144
p=
p=0.064
Phon-
0.114
0.87 + 0.028
0.97 + 0.019
p <0.001
<0.001
p 0.002
Owner
Low Income
0.134
0.95
0.98 0.006
p x0.001
P =0.001
P 0,025
Younger Bias
Maralhi Speakers
0.51 0.115
0.82 0.036
0.89 a 0.023
P 0.005
--00001
p 0.051
Middle Aged
Early and Late
0.69 + 0.093
0.81 0.038
0.87 = 0.038
p 0.117
p= 0.037
p= 0.133
Workers
Critical Low
0.60 = 0.104
0.03:00022
0.93 = 0.015
p= 0.001
p P000001
p= 0,500
Income
Early Morning
0.70 0.085
0.037
0.96 0.026
p= 0.034
0=0.00
p= 0.024
Call + NGO
Registered
Morning
0.52 0.109
0.87
P p=00.001
p= 0.001
p P-9985
NGO Registered
Afternoon Call
0.72 + 0.085
0.023
0.91 = 0.024
0=00112
p= 0.016
0=0.618
NGO Registered
Evening Call +
0.55 + 0.111
0.67 0.057
0.90 = 0.031
0.169
p= 0.001
P =0.000
NGO Registered
Technically Challenged
0.64 + 0.097
0.79 a 0.049
0.92 0.019
p= 0.085
p= 0.003
p p= 0.007
Here we include the full prompt list for each task, base reward functions, as well as the numerical results given in Figure 2
We show the full set of statistical tests in Table 3
L Chain-of-Thoughe Experiments
We note that methods that improve LLM reasoning, such as chain-of-thought (CoT), may be in- corporated to improve the base LLM reasoning in the DLM pipeline. Below, we include results using chain-of-thought reasoning for our model outputs; while we did not find that CoT significantly improved reward, we nevertheless emphasize the potential for additional LLM techniques to improve DLM outputs.
M Sample Reward Reflection Output
Here we provide the full output of sample reward reflection procedure, in addition to the corre- sponding selected LLM candidate reward.
M.I Task: Older Bias
We add a full reflection output string for the Idx. O: Older Bias task below. We make several observations the proposed rewards and selected best candidate index. First, we note that the original base function uses only agent_feats [11], which is the binary feature for the oldest age group Age 51-60. We see that the first proposed reward (Index 0) proposes using the top three oldest age groups, agent_feats [9] agent_feate [10] wwwetnneeattttateeeeeeeeeaaaaeee [11], while the second proposed reward (Index 1) uses only the top two oldest age groups agent_ feats [9]] agent_feat [10]. We additionally observe, in the state-feature distributions, that the Index 1 reward observes a higher percentage of positive state accumulated for the oldest age group Ages 51-60, with 31.65% accumu- lated state vs 28. 18%17% for Index O. Finally, the LLM selects reward at Index 1 as the top candidate reward, demonstrating that there is some reasonable similarity between the selection of the LLM and the intuitively preferred reward function. Additionally. in this case, the LLM reflection was able to help guide the selection of reward functions that prioritize the very oldest age groups, demonstrating some positive effecc on feature extraction through reflection.
24

---

Table 6: Full prompts and numerical results with chain-of-thought (CoT) reasoning for DLM.
iix
Label
Pull Promit
Base (leplin))
No Actio
Defoslt
PLM Refection)
with
DLM
with
(Rdicc-
D Older Bixs
While sill p.lorttiigg dl, 100)
4012 60002
041 0.30
039 0013
sightly Socaa on the olded by
sge dienbeaioon
Low
0.1 20003
slishls focms the button
12.5% of the income bracket
White
100
-.122
0
alightt foose thosr who speat itrch
Flluosionn
White eill prientiing
40000+ 0.003
0-46- 0:15
-009
had low odkcation
Age thaanhetion
White eill priontiing all
1000
4000 0109
0.52 0.11
164--003
Tail Explasi
siihily focss or both tie
dllex by
Midile fncone
White sill
100
4006 0.06
0,65 DO97
0.07
D666 0L003
slights prefer the ioons
brackei bounds for Bhe modde popalation
Nou Phoce
White will pelttiiing
100
0.00
0.50
0022
D155 0000
Owner
slgaaly lavee these worze sho do not ove ther ove plane
Low
will maenusns
LO0
00075
0
0.9
0L56 0.01
motier conkingg
the diaributou of uuse" und
"dlcasina'-
Midrle
slighly on thos Menitt-
speakers with midiiiogge
priontinn
093
Workers
slghtly emppasiie benericerve
who liely wark ealy fhe
mouiig and late at siglt.
slighty will prltiiing
035
Inconee
incoee booket
the
abodure lovest camen De
popelation.
will
2.122
0.70
0.42
1094
Cull NOO
slighll advantage thesz who
Ressstered
peeter beise calied before
resitaed
Morning Cull NCO Regiaaed
White will pieetio dll
100
4L07 2.000
0,52 0.100
0.93 0/126
niihhty shanigg thees who
peter being called hetwon
regramrad White eill prisniing all, 1000
NDO. NDo.
1S
Alforanon Call a NCO Regaimed
4011 A0/02
072 DOBS
0.76 0.05
A
slghtls atvarage
being called who
12 12:00p0:3
tered in NGO
Eveaing Call e Registnedd
While sill pokoriiingg all
1.00
00.00
0,55 R:11
033 0.3-0046
slightly colled TPM
fhoee
Stet regianed ot NGO.
Techrically Challeazed
White prodiizzgg all wnter
1.00
0.4 BO97
0.00 0015
de challennes is
tall benefaariic and dight
prolocacce
Python revard function far RL. with the objective of: While still prioritizing all, alightly focua on the oldest by age distribution. tried several revard functions for this task. Belov, I have the given revard function, and the corresponding distribution of revard achieved acroas 44 agant features. description follous:
My goal
allocation,
features
Indez Neme DataType
[sensitive features hidden]
7. Ages 10-20 Binary
Ages
Binary
9. Ages 31-40 Binary
10. Ages 41-50 Binary
11. Ages 51-60 Binary
12. Speaks Hindi Binary
Speaks Narathi Binary
14. Speaks Gujurati Binary
Speaks Kannada Binary
16. Education level 1/7 illiterate BBnnary
17. Education level 2/7 1-5th Grade Completed Binary
18. Education level 3/7
6-9th Grade Completed Binary
19. Education level 4/7
10th Grade Passed Binary
Education level 5/7 12th Grade Paesed Binery
21. Education level 6/7
Graduate Binary
Education level 7/7 Post graduate Binary
23. Phone ovner 0 (e.g., xonan) Binary
24. Phone ovner 1 (e.g., husband) BBiaary
25. Phone ovner 2 (e.g., family) Binery
28. To be called from 8:30am 10:30am Binary
called
10:30am 12:30pm Binary
25

---

28. be called irom 12:30pn-3:300 Binary
29. be called from 3:30pm-6:30pm Binary
30. called 5:30pm 5:30pm-7:300p -7:30pm Binary
31. To be called from 7:30pm-9:30pp Binary
Binary
33. ARNMAN Binary
34. PHC Binary
35. Income bracket -1 (no income) Binary
36. Income bracket (e.g., 0-5000) Binary
bracket (e.g., 5001-10000) Binary
38. Income bracket (e.g-, 10001-15900
Binary
39. Income bracket (e.g., 15001-20000)
Binary
40. Income bracket (e.g., 20001-25000 Binary
41. Income bracket (e.g.. 25001-30000
Binary
42. Income bracket (e.g., 30000 9999999 Binary
Boloy are the rezard functions I used and their corresponding rezard diatributions:
Index 0:
Ravard Function: (state) 4 (atata) Sggent_faata [9]
or agent_feata [11])
Reflection:
[sensitive features hidden]
Category: Agea
Ages 18.18%
Ages 10-20:
Ages 31-40: 22.95%
Agez 41-50: 2.95%
Ages 61-60: 28.17%
Category:
Income bracket -1 (no income): 0.00%
(e.g.. 0-5000):
Incone bracket 2 (a.g., (001-10000).
86.81X
Income bracket (e.g., 10001-150001
23.35%
Income bracket (e.g., 15001-20000)
6.86%
Incone bracket (a.g.> 20001
5.87%
(e.g., 25001-30000): 0.00%
Income bracket (e.g.. 30000
0.00%
Catagory: Calling Times
1:30am-10:30am: 7.45%
10:30am-12:30pa 4.68%
12:30pm-3:30 53.17% 3::0pm-5:30pm: 3:30pm 9.48%
7:30pm 99:0pmm 11.48%
Category: Education Levels
illiterate: 4.75%
1-5th Grade Completed: 6.22%
Completed: 31.70%
10th Grade Passed: 20. 22%
Grade Paased: 301
Graduate: 0.00%
Post graduste: 0.00%
Category: Langusgee Spoken
Speaks Hindi:
Speaks Narathi: 64.77%
Speaks Gujurati: 0.0
Speaks Kenuada: 0.00%
Category:
Phone ovner Nonan: 91.23%
Phone
Phone ouner Family: 3.05%
Category: Organizations MGD: 75.58%
PHC: 0.00%
Index 1:
Revard Function: state 0.1 state (agent_ [10] or agent, (11l) Reflection:
26

---

[sensitive
hidden]
catagory: Agea
Ages 10-20: 17.42%
Ages 21-30: .4%
Ages 31-40: 21. 11.78%
Ages 41-50: 2.907
Ages 51-60: 65%
category: Incone
Incone bracket (no income): 0.00%
Income bracket (e.g., 0-5000): 8.77%
Income bracket (e.g., 5001 1001-10000): 58.06%
Incone bracket 3 (a-g.> 10001-15000) 21.70%
bracket (e.g., 15001-20000):
Incone bracket 5 (a.g.> 20001-25000) 6.75%
Incone bracket (e.g.> 25001 0.00%
Income bracket (e.g., 30000 :999999: 0.00%
Category: Calling Times
8:30an-10:30am: 7.32%
10:30am-12:30pm
12:30pm-3:30pm 55.10%
3:30pm-S:30pm: 12.90%
5:30pm-7:300 8.73X
7:30pm 99:3pmm: 11.56%
Category: Education Levels
illiterate: 4.56%
1-5th Grade Completed: 5.86%
8-9th Grade Completed: 38%
10th Grade Passed: 19.07%
12th Grade Passad: 40.
Graduate:
Post graduate: 0.00%
Category: Languages Spoken
Speaks Hindi: 33.67%
Speaks Narathi: 66.
Speaks Gujurati: 0.00%
Speaks Kenneda: 0.00%
Category:
Phone ouner Woman: 91.34%
Phone ovner Husband: 6.76%
Phona
Family: 2.89%
Category: Organizatione NGO: 76.50% 23.50%
PHC: 0.00%
Based on the above revard distributione and the given goal: While still prioritizing all, elightly adentify the index of the most effective revard function. Provide your ansver following format:
by age distribution. please
[INDEX]
The best reward function at index:
M.2 Marathi Speakers + Middle Aged
We next analyze sample reward reflection for the Idx. 8: Marathi Speakers + Middle aged task below. Observe that the base reward function in Table 3 uses the features agent featsffeetss [13], the binary Marathi-speaking feature, and agent_ feats [9] agent_feate [10], the middle ages in the age range Ages 31-40 and Ages 41-50. Below, we observe that the first proposed reward includes these exact features, while the second proposed reward only includes one of the middle-income features agatt [9]. However, there is key difference in the feature weightings applied in both reward functions, where the first proposed reward function weights the prioritized features much more with scalar multipliers than the first proposed function. In this case, the L.M selects the first proposed reward function, which also happens to be closer to the scaling used in the original Base reward function. While this not known to the LLM. the first proposed reward function also results
27

---

in greater accumulate state percentage in the relevant middle-age feature groups, possible reason for the LLM selecting the first reward. thus demonstrating an example of reflection aiding in tuning feature weighting in reward functions.
Python revard function for RL with the objective of: While still prioritizing all, alightly focus an those Sapethi-
My
alloration,
middle-aged mothers. tried several this tack. Below, I have the given reward fuaction, and the corresponding diatribution revard achieved acroas 44 agent Ieaturea. description of the features is follows:
functions
Indez Name Datalype
[sensitive features hidden]
7. Ages 10-20 Binary
Ages
Binary
9. Ages 31-40 Binary
10. Ages 41-50 Binary
11. Ages 51-60 Binsry
12. Speaks Bindi Binary
Spaaks Narathi Binary
14. Speaks Gujurati - Binary
Speaks Kamaads Binary
16. Education level 1/7 illiterate Binary
17. Education level 2/T 1-5th Grade Completed Binary
18. Education level 3/7
6-9th Grade Completed Binary
19. Education level 4/7
10th Grade Passed Binary
Education level 5/7 12th Grade Passed Binary
21. Education level 6/T
Graduate Binary
Education level
Post graduate Binary
23. Phone ovner 0 (e.g.. vonan) Binary
24. Phone ovner 1 (e.g., husband) Binaary
25. Phone
(e.g.. family) Binary
26. To be called from 8:30am-
Binary
called
10:30an-12:30pp
Binary
28 To be called from 12:30pa-3:30pa Binary
called
5:30pm Binary
30. To be called from 6:30pm-7:30pa Binary
31. To be called from 7:30pm 9:30pm Binary
32. 8GO Binary
33. ARMMAN Binary
Binary
35 Income bracket -1 (no income) Binary
bracket (e.g.. 0-5000) Binary
37 Income bracket (e.g., 5001-10000) Binary
38
Income bracket (e.g., 10001-15000)
Binary
39.
Income bracket (e.g., 15001-20000)
Binary
40.
Income bracket (e.g., 20001-25000)
Binary
(e.g., 25001-30000)
Binary
42. Income bracket (e.g.. 30000-999998
Binary
Below are the rexard functions used and their corresponding revard distributions:
Index
Revard Function: state 0.1 + 3.5 state + ((agent_feate (9) or agant_feats [10]) and agent_feata [13])
Reflection:
[sensitive features hidden]
Category: Ages
Ages 10-20: 4.02%
Ages 21-30: 12.04%
Ages 31-40: 82,75 79%
Ages 41-50: 1. 167
Ages
00%
Category:
Incone bracket -1 (na incoma): 0.00%
Incone bracket (e.g., 0-5000): 2.49%
Income bracket (e.g., -001-10000): 28%
Income bracket (o.g., 10001-150000
32.
(e.g., 15001-20000)
Incone bracket 5 (e.g.. 20001
5.56%
(e.g., 25001 30000)
Income bracket 7 (a.g., 30009-999999): 0.00%
Category:
Calling Times
8:30am 30 6.46%
10:30 -1: 50%
28

---

12:30pm p::30pm 32.12X
3:30pm 3:30pm-6:30pm: 17.42%
5:30pm -m:30pm: 1.74X
7:30pm m9:30mm: 29.77%
Category: Education Levels
Illiterate: 1.13%
1-5th Grade Complated: 18.09%
6-9th drade Complated: 32.
Grade Passed: 30.49%
12th Grade Passed: 14.
Graduate: 0.59%
Post graduate: 2.96%
Category: Languages Spoken
Speaks Hindi: 16.86%
Speaks Narathi:
Speaks Gujurati: 0.00%
Speaks Kanuada: 0.00%
Category: Phone Oxners
Phone
Noman: 92.
Phone ooner Husband: 3.02%
Phone
Family:
Category: Organizatione MGO: 92.87%
ARNMAN: 7.13%
PHC: 0.00%
Index Revard Function: 2 state + 2 (state and (agent_feate [9] and aeatt_ (13)))
Reflection:
[sensitive features hidden]
Category: Ages
Ages 10-20: 4.33%
Ages 21-30: 12.53%
Ages 31-40: 81.90 11.
Ages 41-50: 1.23%
Ages
0.00%
Category:
Incone bracket -1 (no incoma): 0.00%
Income bracket (a.g., 0-5000): 2.48%
Income bracket (e.g., 5001-10000)- 63.04%
Income bracket 3 (e.g., 10001-150001
31 31.35%
(e.g., 15001-20000):
Income bracket 5 (e.g.. 20001
0.
(e.g., :3001-30000):
Income bracket 7 (a.g., 30000
0.00%
Category: Calling Times
-:30an-10:30am: 5.56%
=0:30am=
12:30pm 3:30pmm: 34.84%
3:80pm-6:30pm: 17. 17.14%
5::0pm-7:30pm: 1.81%
7:30pm-9:30pm 27.70%
Category: Education Levels
Illiterste:
1-5th Grade Completed: 17.84%
5-9th
completed: 34.92%
10th Grade Passed: 29.50%
12th Grade Passed: 12.58%
Graduate:
Post graduate: 3.35%
Category: Languages Spoken
Speaks Hindi: 18.26%
Speeks Narathi: 81.74% 74%
Speaks Gujurati: 0.00%
Speaks Kanaeda: 0.00%
Category:
Phone ouner Woman: 94.85%
Phone ovner Husband: 3.2
Phone Family: 1.91%
29

---

Category: Organizations NGD: .2.32%
:RNMAN: 7.68%
0.00%
Based the above rezard distributions and the given While still prioritizing all, alightly focua an those Narathi -spakkers with middle-aged nothers... please identify
effective revard function. Provide your anszer EXACTLY IN the following format: 'Te best revard function da at Index: [INDEX]
at
M.3 Technically Challenged
In the final example, compare sample of reward reflection for the challenging, ambiguous "Technically Challenged" prompt. The ground truth Base reward for this task considers only the features of phone ownership, interpreting "Technically Challenged" those women who do not own their own phone (using features agent_feate [24] agent_feats [25]. In this case, both proposed reward functions have low precision, including auxillary features that relevant to the Base reward. We observe, in this case, that although the second proposed reward, at Index has utilized only of the relevant features agent_feats feats [25]. it is ultimately selected in the reflection stage. For this highly ambiguous tasks, additional external input may be required, as we observe in this case that reflection does not align with what we may desire given the known Base reward function.
directly
My goal vas to create a Python reward function for RL in resource allocation, objective
prioritizing all, infer technical challeages in reaching the phone that could indicate 'at-risk' benefictaries and give slight preference. tried several revard functions for this task. Belox, have the given revard function, acroas 44 agent features. deacription of the features is as follous: DataType
While
the correspending distribution of reward achievad
Index
(sensitive features hidden]
Ages
Binary
8. Ages 21-30 Binary
9. Ages 31-40 Binary
10. Ages 41-50 Binary
11. Ages 51-60 Binary
Speaks
Binary
13. Speaks Narathi Binary
Speaks Gujurati Binary
15. Speaks Kannada Binary
16. Education level 1/7 illiterate - Binary
Education level
1-5th Grade Completed Binary
18. Education level 3/7 6-9th Grade Completed Binary
level
Passed Binary
20. Education level 6/7 12th Grade Passed Binary
Education
Graduate Binary
22. Education level 7/T Poat graduate Binary
23. Phone owner 0 (e.g., wonan) Binary
(e.g., husband) Binaay
25. Phone ovner 2 (e.g., family) Binary
called
8:30am-10:30as 10:300mm
Binary
27. To be called from
BBinary
28. be called frem 12:30pm
Binary
29. be called from 3:30pm-5:30ps Binary
30. To be called from 5:30pm-7:330
Binary
called
1::00p--::30pm Binary
32 RGO Binary
Binary
34. PHC Binary
35. Income bracket -1 income) Binary
36 Income bracket (e.g., 0-5000) Binary
37. Income bracket (e.g.. 5001 10000) Binary
bracket (e.g., 10001-15000) Binary
39. Incone bracket (e.g.. 15001-20000
Binary
bracket (e.g.. 20001-26000
Binary
41.
Income bracket (a.g., 25001-30000)
Binary
42.
Income bracket (e.g., 30000-999909) Binary
Below rexard functione used
correspeading revard
30

---

distributions:
Index
Revard Function: state 0.1 + 2 state ((agent_feate) [9] or agent_feats (101)
fagent_ [11] or agant_ffata [24] agent_feate (251))
Reflection:
[sensitive features hiddan]
Category: Ages
Agea 10-20: 5.38%
Ages 21-30: 84.35%
Ages 31-40: 6.312
Ages 41-50: 3.96%
Ages 51-60: 0.00%
Category: Incone
Incone bracket (no income): 0.00%
Income bracket (e.g., 0-5000) 6. 98%
Income bracket (e.g., 5001 10000): 44.30%
Income bracket (a.g., 10001-15000): 43.07%
Income bracket 4 (e.g.> 15001-20000) 4.83%
(e.g., 20001-25000):
Income bracket 6 (e.g.> 25001-300001
0.00%
Income bracket ge.g., 30000-999999) 0.00%
Category: Calling Times
8:30an-10:30am: 10.20X
10:30am 3.16%
12:30pa-9:30pm: 9.29%
9:30pm-6:30pm: 19.40%
7:30pm 21.35%
1:30pm-9:3000 36.59%
Category: Education
Illiterate: 2.42%
Completed: 20.64%
6-9th Grade Completed: 74.
10th Grade Passed: 24.80% 80%
12th Grade Passed: 4.62%
Graduate: 1.66%
graduate:
Category: Languages Spoken
Speaks Hind1: 37 45%
Speaks Marathi: 62.65%
Speaks dujurati: 0.00%
Speaks Kannada: 0.00%
Category: Phone Oxners
Phone ooner
34.69%
Phone ouner Husband: 31.75%
Phone ovner Family: 33.56%
Category: Organizatione MGD:
ARNMAN: 56.46%
0.00%
Index 1: Revard Fonction: agent_feates [7]
agent_ [8]
agent_feate [14]
2 + agatt_feats [25] 1) state
Reflection:
[sensitive features hidden]
Category: Ages
Ages 10-20: 5.21%
Ages
Ages 31-40- 6.00%
Ages 41-50: 4.01X
Ages 51-60: 00%
Category: Incone
Incone bracket -1 (no income): 0.00%
(e.g., 0-5000):
Incone bracket 2 (a.g., -0000)) 44.
Income bracket (e.g., 10001 42.90%
Income bracket (e.g., 15001-20000) 4.65%
31

---

Income bracket (e.g.. 20001 20000) 0.83%
Incone bracket (e.g., 25001 30000); 0.00%
bracket (a.g.> 30000 -9000-999999) 0.00%
catagory: Calling Times
3:30an-10:30am: 10.03%
1o:30am-12:30pp 2.98%
12:30pm-3:30pm 9.01%
3:30pn-6:30pm: 19.46%
5:30pa-7:30pm: 23.43%
7:30pm-9:300p 37.165
Category: Education Levels
Illiterate: 2.30%
1-5th Grade Complated: 21.66%
5-9th drade Complated: 27.61X
Passed: 72%
12th Grade Passed: 4.53%
Graduate: 1.53%
Post graduate: 17.67%
Category: Languages Spoken
Speaks Hindi: 36.94%
Speaks
Speaks Gujurati: 0.00%
Speaks Kenuada: 0.00%
Category: Phone Oxners
Phone
33.76%
Phone ooner Husband: 32.50%
Family:
Category: Organizatione NCO: 43.09 ARNMAN: 56.91X
PHC: 0.00%
Based the above rezard distribations and the given goal: While still prioritizing all, infer technical challenges reaching the phone that could andicate 'at-risk' beneficiaries and slight preference.. please identify the index of the most effactive ravard function. Provide your anaver EXACTLY following
function index: [INDEX]
N Generated Reward Functions
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction elearly state and claims, and the claimed made in the abstract and introduction match the theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The scope of the claims are clearly stated (see abstract, introduction, and experiments). Our approach is designed for restless multi-arm bandits tasks, and the experiments focused on public health domains.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
33

---

Answer: [Yes]
Justification: We provide all details on theoretical results, including necessary assumptions
and detailed derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all information needed reproduce the experimental results, including training details (Section [.3). a full task list and ground truth base reward function (see Appendix E), full output of sample reward reflection procedure (see Appendix F).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
34

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We use a real world dataset collected by a non-profit organization in public health domains. We are not allowed to release the data. However, we will release the code, and we describe dataset details (e.g. dataset size, features available) and dataset consent for collection and analysis in Appendix and C
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Sooeeee//ddeSSbbbm popoddSSbmisssooPoo//...o ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section on experiments, we provide detailed of the experimental setting, including details of the simulated public health setting, tasks, baselines, metries, training details, and evaluation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report error bars the main experimental results (see Figure 2 We report how these errors bars are calculated (number of trails, number of samples per steps, etc) in Figure caption. We also report results on statistical significance, specifically one-tailed t-test results, in Table 5 Appendix.
Guidelines:
35

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report information computing resources in Appendix Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wuripppccccccbp////aaaaa
Answer: [Yes]
Justification: Our experiments confirm with NeurlPS code of ethics. For experiments on real-world data, we describe dataset consent for collection and analysis Appendix C
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
36

---

Justification:
We discuss societal impacts of this work in the social impact statement (see Section[A]in Appendix).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
We discuss safeguards in place for responsible and safe of data and models in Section The public health information dataset is completely anonymized by the non con-profit organization collaborate with (see Section C for more details).
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers that produce the code and models used. Specifically, in Section we clearly state that we use the Gemini Pro LLM model [I] from Google to generate reward functions, and train our downstream policy with RL using PPO [52]
Guidelines:
37

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. We conduct secondary analysis of the ARMMAN dataset (see Appendix C for details).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
38

---

The data exchange from ARMMAN to the researchers was regulated through clearly defined exchange protocols including anonymization, read-only researcher access, restricted use of data for research purposes only, and approval by the ARMMAN ethics review committee (see Appendix C for details).
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "nCbYCaXJTv", "reviewer": "Reviewer_8Afe", "summary": "Restless multi-armed bandits (RMAB) are effective for resource allocation in public health but lack adaptability to changing policies. Large Language Models (LLMs) have potential in healthcare for dynamic resource allocation through language prompts but are understudied in this area. This work introduces a Decision-Language Model (DLM) for RMABs to fine-tune resource allocation policies using language prompts. It proposes using LLMs to clarify policy preferences, propose reward functions, and iteratively refine these functions through simulations. Key contributions include pioneering the use of LLMs for adapting public health resource allocation and demonstrating near-human-level policy tuning in a maternal and child care task. They introduce a reward proposal loop that improves LLM-generated reward functions using feedback from restless multi-armed bandit (RMAB) simulations. This allows LLMs to iteratively refine reward designs to achieve specific, human-specified policy outcomes.", "questions": "- What are the feature z exactly? It is very helpful to bring some clear examples in section or the beginning of section 4.\n\n- How the buffer is used in the Algorithm? It is not clear\n\n- Figure 2 is not a clear way to show the summary of the results. Maybe a table is better since now the difference is not clear. Now the question is why default at some cases has the same performance as DLM (no reflection)?\nWould not it because of the prompt?\n\n- Why some methods like CoT is not used as a baseline? Or ReAct or Reflexion?", "limitations": "It still needs human prompt at the algorithm which make it less practical in terms of type of the community. \n\nThe amount of its automation is not very clear. It is good to add it to the discussion. \n\nThey need to compare more baselines.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is very well written and fluent. \n- They highlighted the comments and important parts which is really helpful to follow the context. \n- The method is new in this application.", "weaknesses": "- The related works are not comprehensive. It should bring some works in healthcare from other approach and also the application of LLM in other healthcare examples. \n\n- The policy and critic is not identified till in the algorithm 1 and that is confused the reader. Since from the beginning it seems, the policy is also LLM. It requires more clarification and adjustment. \n\n- It is true that DLM (with reflection) works better but it is not a good fair comparison. Justifying why without reflection works good or bad is necessary. \n\n- The baseline are not fair. It is true that might be other methods with LLM is not yet applied to this healthcare problem but to claim why they choose this algorithm, comparing with other algorithms that use LLM is necessary. They need to bring enough evidence for their selection.  In this case they need to bring baseline from the work with LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) “The related works section is not comprehensive…”\n\nWe thank the reviewer for the comment. There is growing research in LLMs for healthcare (see Sec. 1). In particular, a question summarization framework in healthcare domains has been proposed [1, 2]. Methods based on contrastive language image pretraining (CLIP) and LLMs to identify medical disorders, generate relevant context, and craft summarizes with visual information have been studied [3, 4]. ChatGPT applications in medical, dental, pharmacy, and public health have been investigated [5, 6]. However, the potential of LLMs to dynamically adapt resource allocation using language prompts, potentially enabling automated policy tuning using expert and community health insights, remains unstudied (see Sec. 1). We will highlight healthcare related works in the paper. \n\n2) Clarify \"policy and critic\", \"features (z)\", and \"buffer\"\n \nWe describe the details of actor-critic algorithms in Appendix F. To improve clarity, we will move this explanation to earlier in Section 3.  Our novel reward proposal loop enhances LLM-generated reward functions using RMAB simulation feedback, and is compatible with various policy learning algorithms, including the widely used actor-critic algorithms. The features describe demographic information such as age range, education level, and income; examples of features (z) are shown in Sec. 4.2, Sec. 4.4 and Sec. 5. We will provide examples earlier at the beginning of Section 4. We train each policy network using RMAB simulations following the concept of experience replay buffers first introduced by [7], widely used to train RL agents using previously observed transitions [8]. We record per-arm state, action, reward, next state, and lambda over n_s timesteps (see Alg. 1 line 11), and store these “trajectories” of arms in the buffer D, which is then used to update our actor-critic algorithms (see Alg. 1 line 13, App. F).\n\n3) “The DLM with reflection works better, but the comparison is not a good fair comparison. Justifying why without reflection works good or bad is necessary.”\n\nWe appreciate the reviewer’s feedback here, but are unclear on why the comparison appears unfair. Could the reviewer clarify their concerns? We refer to DLM (No Reflection) as DLM-NR below. As highlighted (see Sec. 4.5, App. C), LLMs are trained on real-world data, and can therefore recognize and retrieve features relevant to natural language prompts. This is confirmed by our empirical findings: DLM-NR outperforms Default reward policy (see Fig. 2 and Sec. 5.4b), achieving a mean normalized reward (MNR) score of 0.87 +- 0.008 vs. Default score 0.57 +-0.027, with DLM-NR significantly outperforming Default in 11/16 tasks (see Sec. 5.4c). We justify the strong performance of DLM-NR in Table 1, demonstrating more than 70% feature recall in 9/16 tasks; thus, while DLM with Reflection potentially helps fine-tune feature selection and usage (see Fig. 2), DLM-NR still captures relevant features for reward functions without reflection. \n\n4) “Figure 2 does not clearly summarize results… not clear why Default method sometimes matches the performance of DLM (no reflection).”\n\nWe present numerical results in Tab. 4, and observe that DLM-NR significantly outperforms Default in 11/16 tasks. In the remaining tasks, while DLM-NR still performs well in feature selection for reward proposals (see Tab. 1), reflection iterations may be required to fine-tune the usage of these features (see Fig. 3). In some cases, a prompt may have a ground truth Base reward that is close to the original Default reward, yielding higher Default performance; yet even in these cases (see Tab. 4, Idx. 2 “Hindi Speakers”) we find that DLM (Reflection) can help fine-tune features and feature weights to improve over DLM-NR and Default. \n\n5) “The baselines are not fair … [why not use] CoT, ReAct, or Reflexion”\n\nWe note that DLM is a full framework for the use of LLMs in the reward design of an RMAB.  Other cited methods such as  Chain-of-Thought Reasoning (CoT) improve base LLM reasoning and, unlike our proposed model, are not a full framework. The listed methods are in fact compatible to use with our framework; we indeed attempted using CoT, but found it did not improve DLM reasoning by any significant margin (see attached PDF for results). We further emphasize that we compare against random (Fig. 2 red line) and no-action (Fig. 2 purple bar) allocation policy baselines. Thus, we compare against five other baselines in total. ReAct and Reflexion frameworks do not specify how to choose the state space, action space, and reward function for the LLM, which are all significant design decisions that DLM solves. It is unclear how they would be directly comparable to our multi-agent RMAB reward design loop. \n\n6) “More baseline comparisons are needed” \n\nSee (5). We respectfully disagree. As no existing RMAB reward design loop exists before our method, all other comparisons would be additions on top of the proposed DLM framework. We also would emphasize to clarify that in Figure 2, our main results, we indeed compare against 5 other baselines:“No Action”, “Random”, “No Reflection”, “Base”, and “Default”. We clarify that the “Random” baseline is the red line in Fig. 2. \n\n7) “The level of automation is not clear… the need for human prompts reduces its practicality”\n\nThe level of automation is akin to interacting with a chatbot, requiring only prompting and evaluation. Our method is significantly more automated than human design, allowing tasks to be specified via natural language. This significantly reduces the burden that nonprofits face supporting low-resource communities by enabling rapid policy adaptability without significant additional human effort. Note: in an actual deployment (see Sec. 5.5a, Sec. 5.6 and global comment above) a human operator would approve DLM-generated policy outcomes using output state-feature distributions, providing an additional layer of system scrutiny and safety.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQDwe3KgcD", "reviewer": "Reviewer_R6Uk", "summary": "This paper proposes using a decision-language model for restless multi-armed bandit tasks (RMAB) in the public health domain. The authors evaluated their method in a simulation environment developed from a real-world dataset. The authors conducted experiments with 16 different prompts and compared their approach with baselines, demonstrating the effectiveness of their method. Their study provided insights into the generated reward design for RMABs.", "questions": "1. Why use the ARMMAN dataset exclusively in this study? What language is used in this dataset? Given that the test involves prompts in English, was there any preprocessing of the dataset required?\n2. In some parts of the training process, it is crucial to explain why specific numbers are chosen. For instance, the downstream policy is trained for 5 epochs,  why 5?  why are 100 simulation steps selected?", "limitations": "The approach provided in this paper lacks real-world validation. This paper should also discuss the ethical implications of using AI for decision-making in health resource allocation.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors evaluated their method in a simulation environment developed from a real-world dataset and provided comparisons in the DLM-generated model with the baseline model. \n2. The author provides the parameters of the model training process, which helps other researchers reproduce the process.", "weaknesses": "1. Avoid citing sources in the Abstract\n2. The approach provided in this paper lacks real-world validation. \n3. This paper did not discuss the ethical implications of using AI for decision-making in health resource allocation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) Abstract Citations: “Avoid citing sources in the Abstract.”\n\nWe thank the reviewer for the suggestion. We will remove this citation from the abstract. We include this citation at the end of the introduction. \n\n2) Real-World Validation: “The approach provided in this paper lacks real-world validation.”\n\nThis work will be followed by a real world evaluation. Domain experts from the NGO played an integral role throughout the model iteration and design process. And we would also like to re-emphasize (see Section 5 and Appendix A)  that any subsequent real-world evaluations are only undertaken after securing ethics approval from the NGO's ethics board which is registered with the ICMR (Indian Council of Medical Research). Please see the global Note on Ethical Considerations above for more information. Obtaining approvals and designing and conducting real world evaluation require significant effort from the NGO and may take up to several months to complete due to the complexities involved in real-world trials. Success in simulations are required for any subsequent approvals, and our simulations indicate substantial potential gains from the system (as demonstrated in this paper). \n\n3) Training Process Explanation: “In some parts of the training process, it is crucial to explain why specific numbers are chosen. For instance, the downstream policy is trained for 5 epochs, why 5? Why are 100 simulation steps selected?”\n\nWe thank the reviewer for this suggestion and will include additional details on the selection of training hyperparameters in the Appendix. Choices for these hyperparameters were made following quantitative assessments over a hyperparameter search which showed improved stability and low overfitting for the selected values for our given dataset. Our choices were additionally qualitatively guided by prior works studying network-based multi-armed bandit policy training, which similarly use 100 steps in buffer for simulated settings [9]. We will include this information on hyperparameter selection in the Appendix.\n\n4) Ethical Implications: “This paper should also discuss the ethical implications of using AI for decision-making in health resource allocation.”\n\nWe discuss this extensively in Appendix A, Social Impact Statement, as well as Appendix B, Dataset Description and Appendix C, Dataset Consent for Collection and Analysis. We summarize these points discussed in our paper in the global comment above, including an overview of the ethical considerations taken prior to and during this work with our partner NGO. \n\n5) Dataset Usage: “Why use the ARMMAN dataset exclusively in this study? What language is used in this dataset? Given that the test involves prompts in English, was there any preprocessing of the dataset required?”\n\nWe focus on the ARMMAN dataset because it is vetted, real-world data collected with consent ethically. Such a dataset, created from a study over 7,668 mothers, has the potential for great impact, helping us gain insight into key policy design techniques that can help improve the deployment of public health resources across millions of beneficiaries. Furthermore, the partner organization ARMMAN has an operational AI/RMAB deployment that works to allocate resources. The collected dataset is also rich and not sparse, allowing a diverse set of tasks to be tested within it. Thus, there are multiple important reasons to work with this dataset. The dataset was already collected in English. Thus, no preprocessing of the dataset was required. Due to these key strengths and benefits, we focus our analysis entirely on this highly valuable source of data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFIJ2VZmfv", "reviewer": "Reviewer_zXh7", "summary": "The paper titled \"A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health\" introduces a novel method for improving public health resource allocation. By combining Restless Multi-Armed Bandit (RMAB) models with the interpretive power of Large Language Models (LLMs), the authors have created a system that can adaptively fine-tune health policies based on human-language commands. This innovation allows for more flexible and responsive policy adjustments, crucial for addressing the changing needs of public health programs.\nA significant contribution of this work is showing how LLMs can generate and refine reward functions for RMABs, enhancing decision-making in resource-limited settings. The authors demonstrate this through a collaboration with ARMMAN, an Indian non-profit focused on maternal health. Their simulations reveal that the DLM can significantly improve the allocation of health workers, leading to better engagement and outcomes. This approach promises to make public health interventions more effective and adaptable, closely aligning with the evolving needs of the community.", "questions": "Is there any theoretical foundation you rely on for designing the prompts used in your model? Do you think referring to sociological theories or frameworks could help in designing more effective and contextually relevant prompts?\nThe paper mentions issues with ambiguous language prompts leading to misaligned policies. Do you have any specific methods or future plans to quantify and address these ambiguities to improve the model's reliability?", "limitations": "The simulations in this work are impressive, but to make a real-world impact, the authors need to address some key areas. These include transitioning to actual field trials, expanding language support, ensuring the model can handle larger datasets, and dealing with ambiguous prompts. It's also important to consider potential negative societal impacts like data privacy, bias, and the effects on vulnerable groups. By tackling these issues and outlining a clear path for real-world use, the authors can significantly enhance the practical and ethical value of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nI think the originality of this paper quite impressive. The way it combines Large Language Models (LLMs) with Restless Multi-Armed Bandits (RMABs) to dynamically fine-tune public health policies is really innovative. Using human-language commands to adjust these policies is a clever idea that bridges advanced AI techniques and practical decision-making in a unique way. This creative approach brings a fresh perspective to public health, making it much more adaptable and responsive to changing needs.\nQuality:\nThe quality of the research really stands out. The authors did a good job detailing their methodology, from the reward proposal loop to the simulation stages and the reflection mechanism for refining reward functions. For example, they explain how LLMs interpret policy preferences and generate reward functions, which are then fine-tuned through simulations. The experiments are well-thought-out and use real-world data from ARMMAN, which adds a lot of credibility. The results are impressive, demonstrating DLM can achieve near human-level performance. The authors also provide a comprehensive analysis, comparing their model to baseline methods using clear performance metrics. This thorough validation highlights the potential for further real-world applications, showcasing how the approach can dynamically adjust policies to meet evolving public health needs.\nClarity:\nThe paper is generally well-organized and easy to follow. The tables and figures do a great job of illustrating the key concepts and results. However, some parts could be more engaging and less technical, making it easier for a wider audience to understand. Simplifying some of the technical language and adding short summaries at the end of sections would help readers quickly get the main points without getting lost in the details. For example, a brief recap of the key findings at the end of the results section would be really helpful.\nSignificance:\nDespite being based on simulations, this work shows great potential for real-world impact. The DLM can dynamically adjust RMAB policies to meet changing public health needs, which is incredibly important. The evaluation on real-world data show that this approach isn't just theoretical but has practical relevance. The findings suggest that the DLM could significantly improve how resources are allocated and how effective policies are, which is very promising. Being able to dynamically prioritize different demographic groups or regions based on evolving needs can lead to more targeted and efficient use of resources. This approach could have wide applications in various public health interventions.", "weaknesses": "One major limitation of the paper is its reliance on a simulated environment for validation, which is understandable given the complexity of real-world testing. While simulations using real-world data, the findings would be significantly strengthened by real-world trials. A practical next step would be to outline a detailed plan for field testing the Decision-Language Model (DLM) in actual public health settings. This would include partnerships with health organizations for pilot studies, addressing potential ethical and logistical challenges, and establishing metrics for real-world success.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1) Simulated Environment Reliance: “The paper relies on simulations for validation. Real-world trials would strengthen findings. A plan for field testing DLM in actual public health settings, including partnerships for pilot studies and addressing ethical and logistical challenges, is needed.”\n\nThis work will be followed by a real world evaluation. Domain experts from the NGO played an integral role throughout the model iteration and design process. And we would also like to re-emphasize (see Section 5 and Appendix A)  that any subsequent real-world evaluations are only undertaken after securing ethics approval from the NGO's ethics board which is registered with the ICMR (Indian Council of Medical Research). Please see the global Note on Ethical Considerations above for more information. Obtaining approvals and designing and conducting real world evaluation require significant effort from the NGO and may take up to several months to complete due to complexities involved in real-world trials. Success in simulations are required for any subsequent approvals, and our simulations indicate substantial potential gains from the system (as demonstrated in this paper). \n\n2) Field Trials and Testing: “Transitioning to real-world testing is essential. This includes expanding language support, handling larger datasets, and addressing ambiguous prompts.”\n\nWe thank the reviewer for this comment and agree that these are critical steps for future envisioned deployment. We discuss potential strategies to handle ambiguous prompts below. As referenced in Sec. 5.6, we agree also that expanded testing across different languages is critical. This may intersect with ongoing research in multilingual LLM evaluation [10], especially in LLM reasoning from Indic languages [11] which are highly represented in the ARMMAN deployment regions. Please see above global comment for additional details in consideration of future envisioned real-world testing.\n\n3) Ethical Considerations: “Address potential negative societal impacts such as data privacy, bias, and effects on vulnerable groups. Tackling these issues and outlining a clear path for real-world use will enhance the practical and ethical value of the work.”\n\nWe thank the reviewer for the comment. We summarize the ethical considerations discussed in our paper in the global comment above, and highlight these key sections below. \n\nWe include a discussion on societal impacts in Appendix A and a discussion on data usage in Appendix C. The real world dataset is completely anonymized by ARMMAN. Data exchange from ARMMAN to the researchers was regulated through clearly defined exchange protocols including anonymization, read-only researcher access, restricted use of data for research purposes only, and approval by the ARMMAN ethics review committee. Our method helps ARMMAN in training dynamic policies to support vulnerable groups and allows ARMMAN program managers to monitor state-feature distributions over underrepresented demographic groups (see Figure 1 description and Section 5). We will further emphasize above mentioned points on ethical considerations and societal impacts in the paper.  \n\n4) Prompt Design Foundation: “Is there a theoretical foundation for designing the prompts used in the model? Referring to sociological theories or frameworks could help in creating more effective and relevant prompts.”\n\nThe prompts in our model are primarily designed to capture the practical, real-world scenarios that could be encountered through callership patterns amongst mothers in India and in collaboration with ARMMAN. While there isn't a direct theoretical foundation in prompt design, the prompts are inspired by common decision-making processes and challenges faced by ARMMAN. In addition, we design them in such a way that they become progressively more difficult to reason about, where we include a mixture of tasks that require just recovering one feature, multiple features, and inferring the correct due to ambiguity. Thus, a substantial amount of time was spent thinking about designing the prompt, although the current prompts could benefit from principled approaches.\n\nFor example, we believe that integrating sociological theories or frameworks is a wonderful idea, and could indeed enhance the contextual relevance and effectiveness of prompts. For instance, a list of criteria such as the Social Determinants of Health [12] could be used to help guide the development of prompts that would  likely improve community needs and values. In addition, a large body of literature is now emerging along the direction of “prompt engineering” [13, 14], or finding methods for improving prompt efficacy.  \n\n5) Ambiguous Language Prompts: “The paper notes issues with ambiguous prompts leading to misaligned policies. Are there methods or plans to quantify and address these ambiguities to improve reliability?”\n\nOne simple idea is to reject ambiguous prompts and ask for further clarifying information, which is tackled in other work that uses the disagreement between sampled  outputs of the model [15] to quantify ambiguity. This is a well-studied area [16, 17], and there should be several strategies available that effectively mitigate ambiguous questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Thank You**

We thank the reviewers for their insightful feedback and comments. We are encouraged to find that the reviewers recognized the paper as novel, introducing "a new method in this application" (R1), and specifically highlighting its “impressive originality” (R3) and “pioneering” (R1) use of Large Language Models (LLMs) with Restless Multi-Armed Bandits (RMABs) in the public health setting, described as "a clever idea that bridges advanced AI techniques and practical decision-making in a unique way" (R3). We are also grateful that the reviewers appreciated the evaluation of our method in "a simulation environment developed from a real-world dataset," adding credibility to our findings (R2, R3). We are pleased to hear that the reviewers found the paper "very well written and fluent" (R1), with effective use of tables and figures (R3). We also appreciate that the described methodology was noted as aiding reproducibility, including that "the author provides the parameters of the model training process, which helps other researchers reproduce the process" (R2).

**Note on Ethical Considerations** 

We appreciate the reviewers raising important points regarding ethical considerations of AI solutions in data consent and privacy. We would like to highlight key sections from the paper that describe our deep consideration of these ethical guidelines during the design process of this work. All of the data collected for use in our _simulated_ public health setting was gathered in close collaboration with our partner NGO, following strict dataset anonymity guidelines (see Sec. 5.1, App. B) and receiving full consent for data collection from each ARMMAN study participant (see App. C.2). In our study, we conduct a _secondary_ analysis of this dataset, representing _simulated_ transition dynamics (see Sec. 5.1, App. C.1). ARMMAN retains full ownership and control over all data; we access _only_ an anonymized version of the data through clearly defined exchange protocols including anonymization, read-only researcher access, and restricted use of data for our secondary analysis (see App. C.2), which was approved by the ARMMAN Board of Ethics (see App. C), registered with the Indian Council of Medical Research (ICMR). 

With respect to the use of AI for allocation of resources in _any future deployment_, we note that the proposed system is intended for use _only_ in cases where separate, _additional_ live call resources are available to help underrepresented groups (see Sec. 5.6, App. C.3). These additional live calls provide motivation for users to listen to information in the original voice calls; they do _not_ provide any new information. We therefore do _not_ withhold any original voice call resources through our system, and consider _only_ the additional resources available for the specific use case of dynamic allocation to underrepresented groups as desired by the NGO. This ensures that all the original health voice messages are always available to all participants in the program in any future deployment scenarios (App. C.3). The ARMMAN ethics board has previously approved _deployed_ studies testing RMAB policies for these described additional call resources \[18\], providing evidence of approval of such AI allocation policies (Sec. 1); however, in these cases the RMAB objective was fixed with hand-set objectives, rather than the adaptive policy design we present.

We further note: we do _not_ deploy the proposed method to the ARMMAN program. In any envisioned deployment, we expect that health experts can monitor outcome state-feature distributions of proposed policies, allowing an ethics board to decide whether to adopt system policy suggestions (see Sec. 5.6). Additionally, we note that any potential deployment would follow an extensive real-world evaluation undertaken after securing approval from the ARMMAN Board of Ethics (see Sec. 5.1). Recognizing the significant effort required from the NGO to obtain these approvals and design and conduct such studies, we follow standard deployment protocol to proceed with field trials _only_ when simulations indicate substantial utility to our NGO partner and their ethics board, as demonstrated in this paper. We will ensure that these considerations are emphasized in the paper. 

**Response Citations**

[1] Lu et al., "Medical Question Summarization...," ACM TALLIP, 2024.

[2] Caciularu et al., "Long Context Question Answering...," NAACL, 2022.

[3] Ghosh et al., "Clipsyntel: clip and LLM synergy...," AAAI, 2024.

[4] Tiwari et al., "Dr. can see...," CIKM, 2022.

[5] Sallam et al., "ChatGPT applications in medical...," Narra J, 2023.

[6] Fatani, "ChatGPT for future medical...," Cureus, 2023.

[7] Lin, "Self-improving reactive agents...," Machine learning, 1992.

[8] Zhang & Sutton, "A deeper look at experience replay...," arXiv, 2017.

[9] Killian et al., "Restless and uncertain...," UAI, 2022.

[10] Bang et al., "A multitask, multilingual, multimodal evaluation...," arXiv, 2023.

[11] Singh et al., "IndicGenBench: A Multilingual Benchmark...," arXiv, 2024.

[12] WHO, "Social Determinants of Health."

[13] OpenAI, "Prompt Engineering."

[14] OpenAI, "Related Resources Around the Web."

[15] Cole et al., "Selectively answering ambiguous questions...," arXiv, 2023.

[16] Keyvan & Huang, "How to approach ambiguous queries...," ACM CS, 2022.

[17] Min et al., "AmbigQA: Answering ambiguous open-domain questions...," arXiv, 2020.

[18] Verma et al., "Restless Multi-Armed Bandits...," AAMAS, 2023.

Meta-review: After a review/rebuttal/discussion phase that involved the authors and all reviewers we have come to the recommendation of accepting this paper for NeurIPS.

While there were some ethical concerns, for which the authors received an ethical review whose recommendations need to be followed for the final versions of the paper, the overall consensus for this paper was positive.

In general the reviewers agree that the paper is very well written and clear, the questions are novel and a good application of AI in health care, and the experiments (though limited by only simulation data) are compelling and well presented.
Decision: Accept (poster)