Title: A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health
Venue: NeurIPS 2024
Accepted: yes
Keywords: public health

multi-agent systems

multi-armed bandits

LLMs
Abstract: Restless multi-armed bandits (RMAB) have demonstrated success in optimizing resource allocation for large beneficiary populations in public health settings. Unfortunately, RMAB models lack flexibility to adapt to evolving public health policy priorities. Concurrently, Large Language Models (LLMs) have emerged as adept automated planners across domains of robotic control and navigation. In this paper, we propose a Decision Language Model (DLM) for RMABs, enabling dynamic fine-tuning of RMAB policies in public health settings using human-language commands. We propose using LLMs as automated planners to (1) interpret human policy preference prompts, (2) propose reward functions as code for a multi-agent RMAB environment, and (3) iterate on the generated reward functions using feedback from grounded RMAB simulations. We illustrate the application of DLM in collaboration with ARMMAN, an India-based non-profit promoting preventative care for pregnant mothers, that currently relies on RMAB policies to optimally allocate health worker calls to low-resource populations.  We conduct a technology demonstration in simulation using the Gemini Pro model, showing DLM can dynamically shape policy outcomes using only human prompts as input.

A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health

Abstract

Restless multi-armed bandits (RMAB) have demonstrated success in optimizing resource allocation for large beneficiary populations in public health settings. Unfortunately, RMAB models lack flexibility to adapt to evolving public health policy priorities. Concurrently, Large Language Models (LLMs) have emerged as adept automated planners across domains of robotic control and navigation. In this paper, we propose a Decision Language Model (DLM) for RMABs, enabling dynamic fine-tuning of RMAB policies in public health settings using human-language commands. We propose using LLMs as automated planners to (1) interpret human policy preference prompts, (2) propose reward functions as code for a multi-agent RMAB environment, and (3) iterate on the generated reward functions using feedback from grounded RMAB simulations. We illustrate the application of DLM in collaboration with ARMMAN, an India-based non-profit promoting preventative care for pregnant mothers, that currently relies on RMAB policies to optimally allocate health worker calls to low-resource populations. We conduct a technology demonstration in simulation using the Gemini Pro model [1], showing DLM can dynamically shape policy outcomes using only human prompts as input.

1 Introduction

Limited resource allocation is a frequent challenge in public health settings. For instance, in the maternal and child health domain, preventative care awareness programs play a key role in reducing maternal mortality, where global rates are currently more than double the UN Sustainable Development Goal target of fewer than 70 deaths per 100K live births [2]. These essential programs help avoid preventable deaths [3, 4], yet are often operated by non-profits supporting low-resource communities [5, 6], and thereby face two key challenges in distributing resources to large beneficiary populations. First, programs typically operate with insufficient financial and human resources, necessitating effective resource allocation strategies to maximize health outcomes [7, 8]. Second, these programs must frequently adapt to changing population needs and intervention priorities [3]. Adaptability is especially important for prioritizing care for known subpopulations with higher risk [9], and for adjusting strategies using community or expert-driven insights [10].

To address the first problem, restless multi-armed bandits (RMAB) have proven effective in realworld deployment for optimally allocating limited resources in critical public health domains [11, 12, 13, 14]. In the classical RMAB formulation, a central planner chooses arms to allocate resources, observing a per-arm state-dependent reward. However, while RMABs are well-suited for resource allocation, a significant challenge persists in developing models capable of adapting to changing policy objectives. For instance: public health experts must often dynamically allocate resources to

specific underprivileged socioeconomic groups [15, 16, 17], yet existing works in RMABs largely focus on fixed objectives, requiring significant human design to achieve new desired health outcomes.

Recently, large language models (LLMs) have emerged as adept planners in tasks such as navigation [18], spatio-temporal reasoning [19] and interactive decision-making [20]. Recent works have also demonstrated that LLMs, from language prompts, can generate reward functions as code, automating complex robotic manipulation tasks in reinforcement learning (RL) [21]. While there is growing research in LLMs for healthcare [22], the potential of LLMs to dynamically adapt resource allocation using language prompts-potentially enabling automated policy tuning using expert and community health insights [23]-remains unstudied.

In this work, we propose a Decision-Language Model (DLM) for RMABs, enabling dynamic finetuning of resource allocation policies for public health using human language prompts. We propose: 1) using LLMs to disambiguate language-expressed policy preferences, 2) using LLMs to directly propose reward functions as code to reach desired RMAB objectives, and 3) a fully automated self-reflection stage to iteratively refine LLM-generated reward functions using grounded RMAB simulations, without requiring ground truth feedback. Stepping beyond existing work in LLMs for health, we uniquely propose using LLMs to tune RMAB-driven resource allocation through reward function design, enabling: 1) more principled resource allocation using RMABs as a central planner, rather than direct LLM output, 2) continual alignment of reward functions to specified prompts using simulated RMAB allocation outcomes to refine LLM proposals, and 3) improved interpretability of proposed rewards through the use of code for reward function output.

We assess our proposed method within a simulated public health environment created in partnership with ARMMAN, an India-based non-profit that spreads preventative care awareness to pregnant women and new mothers through an automated call service. To increase program listenership, ARMMANemploys health workers to provide live service calls to beneficiaries; however, due to limited support staff, ARMMAN faces a resource allocation problem to improve program listenership through optimal assignment of health workers. Prior works modeling the ARMMAN setting with RMABs have shown that RMAB policies can reduce engagement drops by ∼ 30% in real-world deployment [24, 25, 26]. However, these prior works fail to address a persistent challenge within ARMMAN: tuning policies dynamically to prioritize disadvantaged groups, a common issue when targeting new regions or shifting population dynamics [27, 28]. We utilize an anonymized (Appendix B) dataset from ARMMAN to develop a simulation (Appendix C) representing changing priorities within the maternal health setting, and evaluate our method within this simulated public health environment. We provide a detailed discussion of our consideration of ethical guidelines during the design process of this work in Appendix A, B, C.

In summary, our key contributions are as follows:

To the best of our knowledge, we are the first to propose using LLMs to adapt to changing resource allocation objectives in public health through reward design in the RMAB setting.

We introduce a reward proposal loop that enhances LLM-generated reward functions using feedback from restless multi-armed bandit (RMAB) simulations, enabling LLMs to iteratively refine reward design to achieve specific, human-specified policy outcomes.

To assess the feasibility of our system in simulation, we evaluate our algorithms' performance using the Gemini Pro model [1] in a real-world inspired task of resource allocation in maternal and child care, demonstrating near human-level policy tuning to achieve humanspecified outcomes using only language prompts as input.

2 Related Work

RMABs : The RMAB problem, introduced by Whittle [29], is classically solved through the Whittle index heuristic policy [30, 31]. Subsequent works have generalized to multi-action RMABs [32, 33]. RMABs gained prominence in public health domains and are deployed to disseminate preventative healthcare information, monitor health program adherence, and model disease spread [34, 24, 35]. However, existing works focus on fixed reward functions and fail to consider that public health planners often have evolving priorities [3].

Reward Design : Designing reward functions that effectively condense long-term agent goals into immediate behavioral signals is a central problem in RL [36]. Manual designs are prone to task misspecification and overfitting [37]. Brys et al. [38] and Hussein et al. [39] reshape reward

Legend

LanguageCommand

1.LLM

2.LLMCandidateRewards

→

LLMinput context

prioritizelowincomemothers"

"optimizeoverall,butslightly

RMABproblem...""

"designthreerewardfunctionsformy

R1

state/2 + feats[11]/2

Ri

candidatereward

R2

state*feats[11]+feats[33]

Sim.under R=R3

Z

ArmFeatureContext

5.LLMReflection

R3

(state+feats[11]*feats[37])2

1.Age, in years (Int)

"fromthereward distributions,choose

2. Income,in dollars (Int)

thebestrewardcandidateindex...

3.Multi-AgentSimulations

>

4.PolicyOutcomeComparison

R

R2

R3

classMaternalHealthEnv():

R

R2

R3

π

π

π

per-agent policy opt.

state:*floatvalue0-1*

features:*dict{*

Age

Income

Income

Age

Age

Income

00 0

入

update

lambda

入

through expert behavior observation; however, expert examples may not be available in resource and data-limited public health domains. Using LLMs as dense reward signals has been investigated [40, 41]. Ma et al. [21] and Li et al. [42] further use LLMs to output reward function code from language-specified goals. However, this work fails to address multi-agent settings, where a central planner could prioritize certain groups through reward design, and where the ground truth return signal is unknown.

3 Background

We consider an RMAB problem with N arms, each representing public health program beneficiaries, following prior work in the ARMMAN setting [24, 33, 14]. Each arm n ∈ [ N ] follows a Markov decision process ( S n , A n , C n , T n , R n , β n ) with state space S n , action space A n , action costs C n : A n → R , and unknown transition probabilities T n : S n ×A n → ∆( S n ) . We define reward function R n : S n → R and discount factor β ∈ [0 , 1) .

When S n , A n , and C n are the same for all arms n ∈ [ N ] , such as the public health setting we focus on, we omit the subscript n . Note our algorithms still apply to more general settings. A planner learns a policy π that maps states s ∈ S to actions in the set A , given the constraint that the sum cost of actions does not exceed budget B . The objective is to learn a policy that maximizes the Bellman equation: max π ∈ Π { J ( s ) := ∑ N n =1 R ( s n ) + β E s ′ ∼ T ( ·| s,π ( s )) [ J ( s ′ )] } s.t. π ( s n ) ∈ A , ∀ n and ∑ N n =1 c π ( s n ) ≤ B for cost of action c π ( s n ) ∈ C n . We take the Lagrangian relaxation [25], fixing an action charge λ to decouple the value function across arms:

Defined for Q-function Q , arm n , action a n , transitioning from state s n under action a n to s ′ n , and π ( λ ) an optimal policy under λ . [43, 21]. We additionally build on deep RL approaches for RMABs [44, 33], which we discuss in Appendix F.

4 Decision-Language Model for RMABs

Below, we provide an overview of DLM (see Figure 1). We first describe the reward generation setting and the context provided to the LLM for initial reward candidate generation (Figure 1 Steps 1,2). We

next discuss the independent simulation stage (Step 3) and subsequent policy outcome comparison stage that simulates each policy reward outcomes (Step 4). We then discuss LLM reflection, which identifies top candidates to guide future in-context learning for LLM reward generation (Step 5). Finally, we motivate the use of LLMs for reward generation from a theoretical perspective.

4.1 Problem Setting: Reward Generation for RMABs

We first define the goal of language-conditioned reward design in the RMAB setting. We consider a human-specified language policy goal ℓ with a corresponding base reward function that maps policies π ∈ Π to real, scalar values: F : Π → R . This base reward function F establishes a scalar 'ground truth" evaluation that corresponds directly to the human-specified command. Then, our key objective is, given a human-specified language prompt ℓ and arm features z , to generate the reward function R that maximizes base function F through optimal policy π R ∗ for reward R :

Note that we do not assume access to the ground truth reward function F during training; however, we may query this ground truth 'Base" reward at test time to evaluate a proposed policy. We propose using LLMs to: 1) propose reward function R , such that optimal policy π R ∗ , when evaluated via the unknown base function F , maximizes reward output, and 2) automatically refine R through self-reflection.

4.2 Provided DLM Context

In the reward function generation phase, we prompt the LLM with three key contextual components (shown in Alg. 1, line 1). First, we include a human-language command ℓ that describes a desired policy outcome targeting a feature-based group (see Table 3 for examples). In practice, this is the only human input required for the proposed DLM technique. Second, we provide arm feature information, denoted z . In the public health setting, features may include demographic information, socioeconomic data, or other descriptors relevant to policy outcomes; we provide the features used in our simulated setting in Appendix Figure 4. We propose that LLMs, given a language prompt ℓ , may be used to extract the relevant features z ℓ ⊆ z from ℓ to design a reward function R . Third, we provide context regarding the RMAB setting and code implementation. Specifically, we include information describing state, a per-arm binary value, and syntax for accessing feature information (example prompt shown in Appendix I). Using this context, we then query an LLM to propose reward functions as code (following [21, 42]) to achieve language-specified outcomes; to improve the alignment of proposed functions to these language-specified goals, we then introduce a key multi-agent simulation stage within the DLM loop.

Algorithm 1 DLM

4.3 Multi-Agent Simulation

We evaluate each LLM-proposed reward function R 1: K by training a policy network θ under each proposed reward R i (shown Alg. 1, lines 5:13). We consider a simulation space which defines the number of arms N , each with fixed, but hidden, transition dynamics. Following the procedure defined in Alg. 1, for a given reward R i , we first sample an action-charge λ , which is used to decouple the learning of arm policies (Eq. 1, Alg. 1 line 9). Then, we simulate n s timesteps, sampling trajectories for each arm under the action-charge λ (Alg. 1 lines 10:12), following [25, 45]. These trajectories are used in a buffer D (Alg. 1 line 12) to eventually update the policy and critic network ( θ, Φ) (Alg. 1 line 13). Note that we compute PPO advantage estimates for the actor from estimated Q-values.

4.4 Reflection Stage

We propose enabling LLM self-reflection using a policy outcome comparison procedure described in Appendix D Algorithm 2. Unlike prior works [21], we do not assume access to a numerical, scalar fitness function F to validate reward functions. In the absence of this scalar feedback, we propose showing state-feature distributions, described below, for LLMs to select top candidate reward functions, and use these top candidates to guide future generations. Given trained policy and critic network θ, Φ , learnt lambda value λ , and feature matrix Z ∈ R N × m , we first simulate over n s evaluation timesteps (Alg. 2 lines 5:7). For stored accumulated agent states S , we designate preferred 'positive states"; this may be customized depending on the setting, but we consider a binary per-agent state with preferred "positive" state 1. Using accumulated states, we then compute the percentage of positive states , out of all positive states observed, accrued from each key feature group g ∈ G over the evaluation timesteps, as a signal for intervention effect (Alg. 2 lines 9:11). These state-feature distributions quantify the distribution of accumulated positive states over key demographic features , ultimately describing the percentage of total positive intervention effects attributed to each discretized age range, education level, or income bracket, for example. Each distribution is reported as an output string (see Appendix M for examples).

We then query an LLM to select the best candidate example given the original prompt ℓ and the outcome distribution string Y (Alg. 1 line 15). This selected example is then added as context in the prompt of the next iteration of reward generation (Alg. 1 line 15). Intuitively, because we focus on resource allocation in the public health domain, we propose that simulated state-feature distributions provide sufficient signal for the LLM to analyze proposed rewards and select top candidate examples for future in-context learning. This approach has two key strengths. First, we enable greater transparency through state-feature distribution analysis. By providing only simulated outcomes of policies under specified rewards π R ∗ for reflection, we ensure that self-reflection focuses purely on alignment between the stated language goal and the health resource allocation outcomes, without assuming any access to ground truth reward . Second, we allow for greater flexibility; by investigating only the outcome distributions of the proposed rewards, rather than original policies themselves, we enable a model-agnostic approach to self-iteration in reward design.

4.5 LLMReward Generation Capability

We theoretically justify our method and provide further insight into how LLMs generate a reward function via reflection. We propose that the LLM can be interpreted as following a hyperparameter optimization algorithm: (1) Find the relevant features from z using world knowledge (2) Generate a reward function which is a weighted sum of the relevant features and observe the outcome of the optimal policy corresponding to this reward via state-feature distribution feedback (see Section 4.4) (3) Find the state-feature distribution that best conforms to the human language command (4) Optimize the weights to obtain a better candidate reward function.

We find empirically in Table 1 and Appendix H that the LLM indeed implements the first step of the proposed algorithm, by analyzing the precision and recall of finding relevant features through the usage of world knowledge and logical reasoning. This finding corroborates the results found in the logical reasoning analysis of language models done by Clark et al. [46]. We assume that the LLM can evaluate reward functions by assigning a value to their state-feature distribution. In this section, we prove by construction that a transformer can implement the second step of this algorithm and give complexity bounds and correctness guarantees of this optimization process.

In our setting, we consider the embedding space R d , whose elements encode states and features of an arm. We assume that there is an embedding function ϕ which maps features of an arm (denoted by z ) and its state s to a vector in R d . That is ∃ ϕ : ϕ ( s, z ) ∈ R d . Weconsider each entry of the vector ϕ ( s, z )

to correspond to some simple single feature of the arm such as (age group) or a complex combination of simple features such as (age group, education level). Let ✶ () denote the indicator function. For example, the first entry of the vector ϕ could be ϕ 1 ( s, z ) = ✶ (20 ≤ Age ( z ) ≤ 25 , s = 0) allowing us to set the rewards to arms in state 0 and age between 20 and 25. The 10 -th entry could be ϕ 10 ( s, z ) = ✶ (20 ≤ Age ( z ) ≤ 25 , Education ( z ) = High School , s = 1) , allowing us to set the rewards for arms in state 1, ages between 20 and 25 and education level being high school.

Based on empirical evidence of the LLM generated rewards, we consider the reward function for arm with features z in state s is of the form R ( s, z ) = w T ϕ ( s, z ) for some d dimensional vector w ∈ R d with non-negative entries. This represents a rich class of rewards. In the example above, if w 1 = 1 and w 10 = 10 and all other entries of w are zeros, the reward function is:

We let the true reward function to be R ∗ ( s, z ) = ( w ∗ ) T ϕ ( s, z ) for some w ∗ ∈ R n with non-negative entries. Assume that w ∗ is a sparse vector - i.e, the number of non-zero entries is a constant independent of n . This roughly means that the reward depends on only a few features. Let Supp ( w ∗ ) be the set of indices where w ∗ has non-zero entries. Let ∥ w ∗ ∥ 0 := | Supp ( w ∗ ) | .

Finding Supp ( w ∗ ) can be a challenging task, since n can be very large. Based on the discussion above, in Step 1, the LLM is able to use its world knowledge and find Supp ( w ∗ ) . We now try to understand how the LLM approximates the corresponding entries of w ∗ via steps 2,3 and 4. Based on empirical analysis of LLM outputs, we propose the following hyperparameter search algorithm in the log-space to model its behavior. For some α > 1 , denote the search space to be S = { α k : -K ≤ k ≤ K } for some integer K . The algorithm searches for ( w i ) i ∈ Supp ( w ∗ ) ∈ S | Supp ( w ∗ ) | . Here, α represents the base of the logspace, or level of granularity of the discretization. In practice, we found that α was often set to be 10 by the LLM. Let V ( w,w ∗ ) denote the value assigned by the LLM to the state-feature distribution corresponding to the reward w T ϕ ( s, z ) (as proposed in Step 3). We assume that the V ( , ) has the following monotonicity property, which roughly states that if w comes closer to w ∗ , then V ( w,w ∗ ) increases.

Definition 1 (Monotonicity) . We say that the return function V is monotone if for any u, v ∈ R n , if | u i -w ∗ i | ≥ | v i -w ∗ i | for every i ∈ [ n ] , then V ( u, w ∗ ) ≤ V ( v, w ∗ ) .

Proposition 1. Assume monotonicity for V ( · , w ∗ ) and let ˆ w := arg max w ∈ S | Supp ( w ∗ ) | V ( w,w ∗ ) . There exists a transformer T with constant depth D and width O ( ∥ w ∗ ∥ 0 K ) which can find ˆ w with O ( ∥ w ∗ ∥ 0 K ) samples, with oracle access to V ( · , w ∗ ) .

This shows that the LLM has the capability to correctly optimize reward weights and converge to a good reward function, under certain assumptions. We give a detailed proof in Appendix G.

5 Experimental Evaluation

5.1 Simulated Public Health Setting

We evaluate DLM in a simulated setting developed in collaboration with ARMMAN, an India-based non-profit that enrolls pregnant women (beneficiaries) from low-income communities [33, 24] into an automated messaging service sharing preventative healthcare information. ARMMAN employs health workers to provide live service calls to beneficiaries with a higher risk of program drop-out, however, the beneficiary population far outnumbers the available health workers. Currently, deployed RMABs in the ARMMAN setting model beneficiaries as arms, live health worker calls as intervention actions [48, 49, 50], and binary arm states 0 as "not engaged" and 1 as a preferred "engaged" (positive) state when beneficiaries listen to an automated message for > 30 seconds. The RMAB policy then identifies individuals across the entire beneficiary population to live call such that overall program engagement is maximized [51, 26]. We consider this the old, fixed 'Default" policy goal. However, a key challenge for ARMMAN is training dynamic policies to support specific underrepresented groups, for example when the program is deployed in different regions [27]. We simulate these dynamic policy goals with 16 distinct prompts (Table 3) that describe demographic feature-based subpopulation priorities. We simulate this environment using a fully anonymized dataset collected by ARMMANin January 2022, created from a service quality improvement study of 7,668 mothers; we compute transition probabilities from this dataset to simulate actions of real-world beneficiaries.

Random

No Action

Base

Default

DLM (No Reflection)

DLM (Reflection)

Mean Normalized Reward

1.0

0.5

0.0-

OlderBias

Low Income

Low Education Bias

AgeDistribution

MiddleIncome

Tail Emphasis

Mean Normalized Reward

1.0

0.5

0.0

Non-Phone Owner

Low Income +

Marathi Speakers +

Early and

Critical Low Income

Early Morning Call +

Middle Aged

Late Workers

NGO Registered

Mean Normalized Reward

Avg.over

given prompts

1.0

Base

1.00 (topline)

No Action

-0.06 ± 0.017

0.5

Default

0.57 ± 0.027

DLM (No Reflection)

0.87 ± 0.008

0.0

DLM (Reflection)

0.93 ± 0.006

Morning Call +

Evening Call +

Technically Challenged

NGO Registered

NGO Registered

NGO Registered

Data usage and safety : We use a fully anonymized dataset to simulate the ARMMAN public health setting; all experiments are a secondary analysis using this anonymized dataset, and are conducted with approval from the ARMMAN board of ethics. There is no actual deployment of this technology demonstration in the real-world ARMMAN setting. Results from a simulated setting are shown to highlight potential use cases of our proposed system. For more details about the dataset and consent of data collection, please see Appendix Section B and Section C.

5.2 Tasks, Baselines, and Metrics

We provide experimental results for 16 prompts (Table 4) which include examples emphasizing both single and multiple combined feature categories. We compute per-prompt mean normalized reward (described below) over 200 seeds. We evaluate DLM against several baseline reward functions:

Random : This policy samples arms uniformly at random, and has an MNR score of 0 through normalization. No Action : This policy samples no arms at each timestep, typically yielding negative MNRornear-random MNR in certain sparse reward tasks. Default : This policy is trained with reward prioritizing all beneficiaries equally , representing the old, fixed ARMMAN goal; thus, reward=1 for state 1 and reward=0 for state 0. Most prior work in RMABs assume this reward. Base : Reward is equal to the ground truth fitness function, a human-specified goal function described by the potentially ambiguous language prompt. At test time, we evaluate all methods using Base reward, and consider Base-trained policy as topline (MNR=1). To be clear, each method uses a different reward at train time, but uses Base reward at evaluation time. DLM(No Reflection) : This version of our proposed method ablates the reflection and iterative refinement process, allowing us to assess the importance of reflection in improving zeroshot LLM-generated reward functions. DLM(Reflection) : This is the full version of our proposed method.

We compute mean normalized reward (MNR) as the average reward achieved per policy, over 200 training seeds, using base reward during test-time evaluation . We normalize each reward R as

TaskO:OlderBias

Task4:AgeDistributionTailEmphasis

Task15:TechnicallyChallenged

BaseReward

0.1s + 2s x(oldest_age)

0.1s + 2s x (oldest_age or youngest_age)

0.1s + 2s ×(husband_phone or family_phone)

4s2+10s x(2ndoldest_age or oldest_age)+

0.1s+1.5sx(illiterateorhusband_phone

DLM(NoReflection)

0.6s +0.4sx(oldest_age)

17s x (2ndyoungest_age or youngest_age)

orevening_call)

DLM(Reflection)

s+0.2sx(oldest_age)

0.1s+2sx(husband_phone and (NGOor

ARMMANorPHCaffiliation))

( R -R rand ) / ( R base -R rand ) . An MNR of 0 therefore implies a policy's performance equivalent to random allocation, while an MNR of 1 implies performance at the level of base reward policy.

5.3 Training Details

We use the Gemini Pro LLM model [1] from Google to generate reward functions, and train our downstream policy with RL using PPO (Alg. 1 line 13) [52, 45] and the generated reward function. The downstream policy is trained for 5 epochs, with each epoch containing 100 simulation steps accumulated in a buffer. Thus, we train on a total of 500 samples for each arm. Each such training is run for each proposed reward function, over 2 rounds of reflection and 2 candidate reward functions per round. We evaluate each prompt over 200 seeds, with 50 separate trials per seed and 10 trajectory steps per trial, totaling 100 , 000 total steps per task.

5.4 Evaluation of DLM Performance

We present the experimental evaluation of DLM in Figure 2. To the best of our knowledge, we are the first to demonstrate LLMs as capable policy designers for resource allocation tasks in public health. We make several observations on the results shown in Figure 2.

DLMapproaches Base reward performance : We find DLM approaches Base reward performance across a broad range of tasks. Note that mean normalized reward (MNR) scores reported in Figure 2 are normalized as described in Section 5.2; this score therefore represents performance above random policy (MNR=0) compared to the topline Base reward policy (MNR=1). We find that DLM (Reflection) achieves an average MNR score of 0 . 93 ± 0 . 006 , achieving 90% or higher of base level performance in 11/16 tasks, demonstrating to the best of our knowledge the first example of using LLMs to adapt to changing resource allocation objectives in public health through reward design.

Reflection improves performance for many tasks : We find that over the 16 tested prompts DLM (No Reflection) achieves an average MNR of 0 . 87 ± 0 . 008 , while DLM (Reflection) achieves an average MNR of 0 . 93 ± 0 . 006 . Further, 8/16 of the prompts showed a significant increase in performance after reflection (Appendix Table 5). Notably, we observe that in cases where reflection does not improve performance, DLM (No Reflection) often achieves high zero-shot reward, obviating the need for the reflection stages. For example, we observe for the Age Distribution Tail Emphasis prompt that DLM (No Reflection) achieves an MNR score approximately equal to the normalized Base score. Thus, DLM (Reflection) may unnecessarily iterate on functions that already achieve Base reward, resulting in degraded performance. Such occurrences are rare, and may have little practical difference in the resulting performance, which still achieves near-Base score.

DLMconsistently outperforms Default reward : We observe that DLM consistently outperforms the Default reward policy, which represents the performance of the previously-used, fixed reward function across the given prompts. We observe an average Default reward MNR score of 0 . 57 ± 0 . 027 , compared to 0 . 87 ± 0 . 008 for DLM (No Reflection) and 0 . 93 ± 0 . 006 for DLM (Reflection). Furthermore, we observe that in 11/16 prompts DLM (No Reflection) significantly outperforms Default policy, while in 16/16 prompts DLM (Reflection) outperforms Default (Table 5 in Appendix). In summary, we find that DLM can meaningfully adjust policies, using only language input, to align more closely to human preferences. Additionally, this highlights that the previous fixed Default

reward is ineffective in accommodating the diverse prompts, and hence policy goals, described by humans in our simulated setting.

5.5 LLM-Generated Rewards and Reflection in Public Health Settings

In this section we provide insight into LLM-generated reward functions and the reward Reflection module. We find that LLMs can reliably design and improve upon reward functions for public health goals, enabling automated policy tuning in resource-limited settings.

LLMs accurately interpret language prompts for policy shaping : We provide examples of LLMgenerated reward functions compared to ground truth Base rewards in Figure 3. We find that DLM can consistently capture the ground truth Base reward, and use reflection to improve upon these proposed functions. However, in cases where language is ambiguous (e.g. 'Technically Challenged" example), inherent language ambiguities may result in misaligned proposals. These findings suggest that LLMs can automatically tune public health allocation policies using only language-described preferences. Ultimately, while human input may be required to overcome ambiguities of language prompts, DLM allows a human decision-maker to monitor state-feature distributions and iteratively provide expert opinion to guide LLM-proposed policies.

LLMs as feature extractors for reward proposal in resource allocation tasks : We find LLMs can accurately extract relevant features from language-described resource allocation preferences. We demonstrate these results in Table 1. LLM-proposed rewards have consistently high precision and recall, effectively capturing the features used in the Base reward function. We find in some cases, for instance Task 15 (Technically Challenged), that ambiguous prompts (see Appendix Table 4) may lead to lower recall ability. Interestingly, however, we still observe reasonable performance in these cases (Figure 2), suggesting that DLM may identify potentially correlated substitute features even for challenging prompts. Note, additionally, that effective feature extraction does not necessarily imply the correct usage of features; however, the findings suggest that LLMs are effective at the first critical 'filtering" step of reward design, particularly in our task of public health resource allocation.

5.6 Limitations

We test our method in a purely simulated environment; any potential consideration of real-world deployment would require comprehensive field testing after approvals from relevant ethics boards. Additionally, we test with prompts in English; further testing in local (Indian) languages is required. Furthermore, whereas we present results on DLM with Chain-of-Thought (CoT) reasoning in Table 6, indicating no significant improvement due to CoT, we recognize that additional LLM reflection techniques have the potential to improve DLM. Finally, in any potential deployment of the proposed method, we expect that health experts should monitor state-feature distributions to intervene on proposed policies, ensuring safety and proper disambiguation of unclear input prompts.

5.7 Ethical Considerations in the Use of DLM

Extending our discussion of the ethical considerations we take in developing DLM (Appendix A, B, C), we further consider the broader implications of algorithmic resource allocation, particularly in public health. We note the need for mitigating data bias in the health setting, especially to

minimize harmful discrimination for underrepresented groups [53], and to avoiding data bias by enabling participatory design [54], all key considerations we make in collaboration with our partner NGO for this work. We further highlight the importance of democratized decision-making criteria for algorithmic allocation techniques [55, 56], as well as the importance of complete beneficiary autonomy through guaranteed consent and the opportunity to deny allocations [57], as we do in this work. Finally, one extension of this work that future studies may consider is to directly incorporate prior work in fairness guarantees in resource allocation settings [58, 59, 60, 61].

6 Conclusion

In this paper we introduce a Decision-Language Model (DLM) for resource allocation in public health, which can take language prompts describing complex, potentially ambiguous public health policy goals as input, and generate downstream control policies specialized for such goals. We uniquely demonstrate, beyond existing work in LLMs for public health, the strength of LLMs to shape principled, RMAB-based resource allocation strategies , potentially enabling rapid communitydriven policy adjustment in critical, resource-constrained public health settings. For all future work considering potential deployment, DLM enables health experts to monitor state-feature distributions to intervene on proposed policies and guide LLM generation to ensure safety.

Acknowledgments and Disclosure of Funding

This work was supported by the Harvard Data Science Initiative.

References

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Department of Economic United Nations and Social Affairs Sustainable Development. Transforming our world: the 2030 agenda for sustainable development, 2015.

World Health Organization et al. Strategies towards ending preventable maternal mortality (epmm). World Health Organization , 2015.

Avishek Choudhury, Onur Asan, and Murari M Choudhury. Mobile health technology to improve maternal health awareness in tribal populations: mobile for mothers. Journal of the American Medical Informatics Association , 28(11):2467-2474, 2021.

HelpMum. Helpmum: Preventing maternal mortality in nigeria, 2023. Accessed: 2023-12-30.

ARMMAN. Assessing the impact of mobile-based intervention on health literacy among pregnant women in urban india, 2019. Accessed: 2022-08-12.

Margaret E Kruk, Anna D Gage, Catherine Arsenault, Keely Jordan, Hannah H Leslie, Sanam Roder-DeWan, Olusoji Adeyi, Pierre Barker, Bernadette Daelmans, Svetlana V Doubova, et al. High-quality health systems in the sustainable development goals era: time for a revolution. The Lancet global health , 6(11):e1196-e1252, 2018.

Rob Baltussen and Louis Niessen. Priority setting of health interventions: the need for multicriteria decision analysis. Cost effectiveness and resource allocation , 4(1):1-9, 2006.

Leonoor Gräler, Leonarda G. M. Bremmers, Pieter Bakx, Job van Exel, and Marianne E van Bochove. Informal care in times of a public health crisis: Objective burden, subjective burden and quality of life of caregivers in the netherlands during the covid-19 pandemic. Health & Social Care in the Community , 2022.

Katrina Deardorff, Arianna Rubin Means, Kristjana Hrönn Ásbjörnsdóttir, and Judd L. Walson. Strategies to improve treatment coverage in community-based public health programs: A systematic review of the literature. PLoS Neglected Tropical Diseases , 12, 2018.

Turgay Ayer, Can Zhang, Anthony Bonifonte, Anne C Spaulding, and Jagpreet Chhatwal. Prioritizing hepatitis c treatment in us prisons. Operations Research , 67(3):853-873, 2019.

Siddharth Nishtala, Lovish Madaan, Aditya Mate, Harshavardhan Kamarthi, Anirudh Grama, Divy Thakkar, Dhyanesh Narayanan, Suresh Chaudhary, Neha Madhiwalla, Ramesh Padmanabhan, et al. Selective intervention planning using restless multi-armed bandits to improve maternal and child health outcomes. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , page 1312-1320, 2023.

Sarang Deo, Seyed Iravani, Tingting Jiang, Karen Smilowitz, and Stephen Samuelson. Improving health outcomes through better capacity allocation in a community-based chronic care model. Operations Research , 61(6):1277-1294, 2013.

Shresth Verma, Gargi Singh, Aditya Mate, Paritosh Verma, Sruthi Gorantla, Neha Madhiwalla, Aparna Hegde, Divy Thakkar, Manish Jain, Milind Tambe, et al. Expanding impact of mobile health programs: Saheli for maternal and child care. AI Magazine , 44(4):363-376, 2023.

Lyndsay A Nelson, Shelagh A Mulvaney, Tebeb Gebretsadik, Yun-Xian Ho, Kevin B Johnson, and Chandra Y Osborn. Disparities in the use of a mhealth medication adherence promotion intervention for low-income adults with type 2 diabetes. Journal of the American Medical Informatics Association , 23(1):12-18, 2016.

Yan Jin, Lucinda Austin, Santosh Vijaykumar, Hyoyeun Jun, and Glen Nowak. Communicating about infectious disease threats: Insights from public health information officers. Public Relations Review , 45(1):167-177, 2019.

Jimmy Phuong, Patricia Ordóñez, Jerry Cao, Mira Moukheiber, Lama Moukheiber, Anat Caspi, Bonnielin K Swenor, David Kojo N Naawu, and Jennifer Mankoff. Telehealth and digital health innovations: A mixed landscape of access. PLOS Digital Health , 2(12):e0000401, 2023.

Dhruv Shah, Bła˙ zej Osi´ nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning , pages 492-504. PMLR, 2023.

Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res , 2:20, 2023.

Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems , 35:31199-31212, 2022.

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In The Twelfth International Conference on Learning Representations , 2024.

Malik Sallam, Nesreen A Salim, Muna Barakat, and B Ala'a. Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J , 3(1), 2023.

AAKANKSHA NAIK, CARLA S ALVARADO, LUCY LU WANG, and IRENE CHEN. Nlp for maternal healthcare: Perspectives and guiding principles in the age of llms. arXiv preprint arXiv:2312.11803 , 2023.

Aditya Mate, Lovish Madaan, Aparna Taneja, Neha Madhiwalla, Shresth Verma, Gargi Singh, Aparna Hegde, Pradeep Varakantham, and Milind Tambe. Field study in deploying restless multiarmed bandits: Assisting non-profits in improving maternal and child health. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 12017-12025, 2022.

Jackson A Killian, Arpita Biswas, Lily Xu, Shresth Verma, Vineet Nair, Aparna Taneja, Aparna Hegde, Neha Madhiwalla, Paula Rodriguez Diaz, Sonja Johnson-Yu, et al. Robust planning over restless groups: engagement interventions for a large-scale maternal telehealth program. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, 2023.

Kai Wang, Shresth Verma, Aditya Mate, Sanket Shah, Aparna Taneja, Neha Madhiwalla, Aparna Hegde, and Milind Tambe. Scalable decision-focused learning in restless multi-armed bandits with application to maternal and child health. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12138-12146, 2023.

ARMMAN. Annual report 2020-2021, 2021.

Sawan Rathi, Anindya S Chakrabarti, Chirantan Chatterjee, and Aparna Hegde. Pandemics and technology engagement: New evidence from m-health intervention during covid-19 in india. Review of Development Economics , 26(4):2184-2217, 2022.

Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of applied probability , 25(A):287-298, 1988.

Richard R Weber and Gideon Weiss. On an index policy for restless bandits. Journal of applied probability , 27(3):637-648, 1990.

Kevin D Glazebrook, Diego Ruiz-Hernandez, and Christopher Kirkbride. Some indexable families of restless bandit problems. Advances in Applied Probability , 38(3):643-672, 2006.

Kevin D Glazebrook, David J Hodge, and Chris Kirkbride. General notions of indexability for queueing control and asset management. The Annals of Applied Probability , 2011.

Jackson A Killian, Lily Xu, Arpita Biswas, and Milind Tambe. Restless and uncertain: Robust policies for restless bandits via deep multi-agent reinforcement learning. In Uncertainty in Artificial Intelligence , pages 990-1000. PMLR, 2022.

Milind Tambe. Ai for social impact: Results from deployments for public health and conversation. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022.

Niclas Boehmer, Yunfan Zhao, Guojun Xiong, Paula Rodriguez-Diaz, Paola Del Cueto Cibrian, Joseph Ngonzi, Adeline Boatin, and Milind Tambe. Optimizing vital sign monitoring in resource-constrained maternal care: An rl-based restless bandit approach. arXiv preprint arXiv:2410.08377 , 2024.

Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society , pages 2601-2606. Cognitive Science Society, 2009.

Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 5920-5929, 2023.

Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, and Matthew E Taylor. Reinforcement Learning from Demonstration through Shaping. International Joint Conference on Artificial Intelligence , 2015.

Ahmed Hussein, Eyad Elyan, Mohamed Medhat Gaber, and Chrisina Jayne. Deep reward shaping from demonstrations. In 2017 International Joint Conference on Neural Networks (IJCNN) , pages 510-517. IEEE, 2017.

Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations , 2022.

Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692 , 2023.

Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. Auto mc-reward: Automated dense reward design with large language models for minecraft. arXiv preprint arXiv:2312.09238 , 2023.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Khaled Nakhleh, Santosh Ganji, Ping-Chun Hsieh, I Hou, Srinivas Shakkottai, et al. Neurwin: Neural whittle index network for restless bandits via deep rl. Advances in Neural Information Processing Systems , 34:828-839, 2021.

Yunfan Zhao, Nikhil Behari, Edward Hughes, Edwin Zhang, Dheeraj Nagaraj, Karl Tuyls, Aparna Taneja, and Milind Tambe. Towards a pretrained model for restless bandits via multiarm generalization. International Joint Conference on Artificial Intelligence , 2024.

Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language, 2020.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems , 34:29304-29320, 2021.

Shresth Verma, Gargi Singh, Aditya Mate, Paritosh Verma, Sruthi Gorantla, Neha Madhiwalla, Aparna Hegde, Divy Thakkar, Manish Jain, Milind Tambe, et al. Increasing impact of mobile health programs: Saheli for maternal and child care. In Proceedings of the Conference on Innovative Applications of Artificial Intelligence (IAAI) , 2023.

Roderick Seow, Yunfan Zhao, Duncan Wood, Milind Tambe, and Cleotilde Gonzalez. Improving the prediction of individual engagement in recommendations using cognitive models. arXiv preprint arXiv:2408.16147 , 2024.

Yunfan Zhao, Tonghan Wang, Dheeraj Nagaraj, Aparna Taneja, and Milind Tambe. The bandit whisperer: Communication learning for restless bandits. arXiv preprint arXiv:2408.05686 , 2024.

Shresth Verma, Aditya Mate, Kai Wang, Neha Madhiwalla, Aparna Hegde, Aparna Taneja, and Milind Tambe. Restless multi-armed bandits for maternal and child health: Results from decision-focused learning. In AAMAS , pages 1312-1320, 2023.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Haylee Lane, Mitchell Sarkies, Jennifer Martin, and Terry Haines. Equity in healthcare resource allocation decision making: a systematic review. Social science & medicine , 175:11-27, 2017.

Alvin Rajkomar, Michaela Hardt, Michael D Howell, Greg Corrado, and Marshall H Chin. Ensuring fairness in machine learning to advance health equity. Annals of internal medicine , 169(12):866-872, 2018.

Lalla Aïda Guindo, Monika Wagner, Rob Baltussen, Donna Rindress, Janine van Til, Paul Kind, and Mireille M Goetghebeur. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost effectiveness and resource allocation , 10:1-13, 2012.

Norman Daniels. Accountability for reasonableness: Establishing a fair process for priority setting is easier than agreeing on principles, 2000.

Hellen Ransom and John M Olsson. Allocation of health care resources: Principles for decisionmaking. Pediatrics in Review , 38(7):320-329, 2017.

Dexun Li and Pradeep Varakantham. Efficient resource allocation with fairness constraints in restless multi-armed bandits. In Uncertainty in Artificial Intelligence , pages 1158-1167. PMLR, 2022.

Shufan Wang, Guojun Xiong, and Jian Li. Online restless multi-armed bandits with longterm fairness constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 15616-15624, 2024.

Shresth Verma, Yunfan Zhao, Sanket Shah, Niclas Boehmer, Aparna Taneja, and Milind Tambe. Group fairness in predict-then-optimize settings for restless bandits. In The 40th Conference on Uncertainty in Artificial Intelligence .

Shresth Verma, Niclas Boehmer, Lingkai Kong, and Milind Tambe. Balancing act: Prioritization strategies for llm-designed restless bandit rewards. arXiv preprint arXiv:2408.12112 , 2024.

David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability, 2023.

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? Investigations with linear models. https://arxiv.org/abs/2211.15661v3, November 2022.

Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking Like Transformers, July 2021.

Appendix

A Social Impact Statement

The presented methods carries a positive societal impact, allowing non-profit organizations in the public health domain to efficiently allocate limited health resources to large beneficiary populations, and to effectively adapt policy outcomes. We illustrate the application of our method in collaboration with an India-based public health organization promoting preventive care for pregnant mothers, that currently relies on RMAB policies for ensuring spread of important health information to low resource populations.

Our proposed methods, tested purely as a technology demonstration and entirely in simulation, do not have direct negative societal implications. However, reinforcement learning agent training should be done responsibly, especially given the safety concerns associated with agents engaging in unsafe or uninformed exploration strategies. While the public health domain we considered does not have concerns associated with extreme environments, when adapting our proposed methods for other domains, ensuring a robust approach to training reinforcement models is critical. All work in simulation was conducted in collaboration with ARMMAN's team of ethics, and any future consideration of deployment would require a real-world evaluation together with the domain partner, which may reveal further challenges to be addressed.

B ARMMANDataset Description

To conduct the secondary analyses presented in this study, we use a real dataset collected by ARMMAN(2022). This dataset was created from a comprehensive quality improvement study carried out by ARMMAN in 2022; the study involved 7,668 beneficiaries who were enrolled over a study period of 7 weeks. Throughout this period, 20% of participants were allocated at least one direct interaction, such as a live service call from a healthcare volunteer. Engagement during this period was used to compute transition dynamics.

B.1 Associated Features and Protected Information

Each beneficiary enrolled in the ARMMAN study was described by 43 features, which include data on age, income, education, preferred call times, and ownership of a phone, among other factors. These features are entirely anonymized. ARMMAN, through the mobile voice call service initiative, works specifically with socially disadvantaged populations. However, ARMMAN does not collect constitutionally protected and sensitive categories such as cast and religion in the recorded beneficiary features. In addition to such categories not being available in the ARMMAN dataset, we further ignore already-anonymized information regarding pregnancy history in our study to focus primarily on broader socio-economic categorizations such as age and income. We additionally worked with ARMMANto ensure that the evaluated prompts, tested only in simulation, are closely aligned with health expert goals and challenging real-world scenarios where prioritizing specific underrepresented subpopulations is desired.

C Dataset Consent for Collection and Analysis

We provide further information regarding consent for data collection and analysis below.

C.1 Secondary Analysis

In this study, we conduct a secondary analysis of the ARMMAN dataset. We use trajectories, which contain the historical engagement behavior for beneficiaries enrolled in the ARMMAN study. Engagement states are computed as binary values, which are determined as 'engaged" if a beneficiary listens to a voice message for more than 30 seconds. The average length of each voice message is 115 seconds. For each beneficiary, we utilize only an estimated transition dynamic matrix , computed using the historical trajectories for T = 7 weeks. We only interface with this estimated transition matrix per beneficiary; for each sampled arm then, we observe only the provided transition matrix and the associated, anonymized, arm features. We then use these estimated transition matrices to

simulate agent behavior; we do not deploy the proposed method to the ARMMAN program . These described secondary experiments are conducted completely in simulation using estimated transition matrices to simulate plausible real-world agents.

C.2 Data Collection Consent

Full consent for data collection is obtained from each ARMMAN study participant; the process of data collection is explained in detail to each participant when obtaining consent, prior to data collection . The dataset was completely anonymized by ARMMAN, and data exchange from ARMMAN to the researchers was regulated through clearly defined exchange protocols including anonymization, read-only researcher access, restricted use of data for research purposes only, and approval by the ARMMANethics review committee.

C.3 Simulated Findings and Equal Distribution of Resources

We note again that all experiments presented in this work were conducted in simulation; we attempt to simulate a plausible real-world public health setting to gain insight into strengths and weaknesses of the proposed approach. We additionally note that the intended goal of this research is to identify techniques to more easily align algorithmic resource allocation methods to specific goals of public health experts. However, the service call program is always equally accessible to all beneficiaries (e.g., participants can always request service calls via a free missed call service) and health information disseminated through the program will never be restricted for any individual enrollee. That is, all participants will receive the same weekly health information by automated message regardless of whether they are scheduled to receive service calls or not. The proposed system is intended for use only in cases where separate, additional service call resources are available to help especially underrepresented groups. In these cases, the proposed system may help align algorithmic resource allocation of the separate additional resources more closely to desired public health expert policy goals.

D Outcome Analysis Algorithm

Below, we provide the algorithm for the state-feature distribution analysis used in the DLM reflection stage to select top candidate rewards. Please see subsection 4.4 for more details on the Reflection module and state-feature analysis procedure.

E Compute Resources

We use the Gemini Pro LLM model [1] from Google to generate reward functions. We use 8 CPUs cores (Intel Cascade Lake Processors) for all tasks.

F Deep RL for RMABs

The Lagrangian relaxation allows us to disentangle the learning of arm policies and an action charge λ . To learn arm policies, it is common to use RL approaches such as tabular Q-learning, deep Q-learning, or policy gradient algorithms[44]. In this paper, we use PPO [52], a policy gradient algorithm with an actor-critic structure. Specifically, the Q-function can be updated as follows:

where α q is the learning rate. For an actor-critic methods, after performing an update of the Q-function in the critic, one may compute the advantage estimate:

In practice, the values A ( s, a ) , Q ( s, a ) , V ( s ) will be evaluated based on the current policy π . Intuitively, the advantage estimate tells us how much better a is compared to the current policy π . The advantage estimate is then plugged into the policy gradient to update the actor. To learn an action charge λ , we follow an updating rule [33]:

where c n,t is the cost of the action taken by the optimal policy on arm n in round t , and α Λ is the step size.

G Proof of Proposition 1

Here we give a proof of Proposition 1.

Proof sketch . To prove this, we first show that there exists parameters θ which can implement the optimization step of a simple and efficient gridsearch algorithm ITERATEDLINESEARCH in the discretized weight space. We can do this by leveraging previous work connecting transformers and traditional algorithms [62, 63, 64]. Specifically, the RASP [64] programming language and TRACR compiler [62] enables compiling a program into transformer weights, giving a direct proof of existence by construction. We then prove the correctness of this algorithm using the monotonicty property. Finally, we prove upper bounds on the runtime of this algorithm, yielding the desired sample complexity bound in Proposition 1.

First we state our gridsearch algorithm ITERATEDLINESEARCH. In our notation, w i is the entry of w ∈ R n corresponding to index i ∈ [ n ] . Recall that there is a total of | Supp ( w ∗ ) | weights corresponding to the non-zero entries of w ∗ , and that we are searching over 2 K possible values for each weight. This means in total there are (2 K ) | Supp ( w ∗ ) | total possible weight combinations.

Algorithm 3 ITERATEDLINESEARCH

We now state the actual optimization subroutine of ITERATEDLINESEARCH, OPTIMIZESTEP.

Proof. Using the RASP programming language [64], we show that OPTIMIZESTEP can be implemented in a transformer. In each forward pass of the transformer, we take one new weight combination and pass this to the loss function in order to evaluate its performance on line 8 of Algorithm 4. Note that the only state needed to be kept track of between forward passes of the transformer is the current hyperparameter dimension i , α , S | Supp ( w ∗ 0 ) | , w , w ′ , V ( w,w ∗ ) , V ( w ′ , w ∗ ) . This information can be stored in nK hidden dimensions, as the size of S | Supp ( w ∗ 0 ) | upper bounds everything else as nK . The conditional statements on line 3 and line 9 of Algorithm 4can be implemented in two select-aggregate blocks [64], corresponding to two attention layers. Line 8 requires an additional fully connected layer after the first attention layer, and a finally an output layer is needed to implement each return statement, giving us four layers in total. The select block roughly corresponds to the attention matrix in the transformer, and is a function that allows calculating conditionals by taking in three arguments: keys, queries, and a boolean predicate p . Here, we would pass α , S | Supp ( w ∗ 0 ) | , w , w ′ , V ( w,w ∗ ) , V ( w ′ , w ∗ ) as the input sequence, and construct three selectors corresponding to the boolean predicates on line 5, 6, and 8, respectively. An aggregate can then be performed to get the result of the predicate, controlling the program flow for selecting.

Proof. Inside each iteration of the while loop, line 10 either i increases to i +1 (but never decreases) or w ′ i increases to w ′ i α . There are 2 K possible values for w ′ i for every given i . There are ∥ w ∗ ∥ 0 possible

values for i and 2 K possible values for w ′ , ensuring that the algorithm converges in O ( ∥ w ∗ ∥ 0 K ) time.

Proof of Proposition 1. We consider iterated line search (Algorithm 3) above with search space S and scale α . In Proposition 3, we have shown that this algorithm converges in time O ( nK ) and that this algorithm can be implemented by a transformer. It remains to show that it returns ˆ w := arg max w ∈ S | Supp ( w ∗ ) | V ( w,w ∗ ) .

By monotonicity, it is clear that if the final output is such that w out i = arg min g ∈ S | g -( w ∗ ) i | for every i ∈ Supp ( w ∗ ) , then w out = ˆ w . Therefore, we prove the coordinate-wise optimality below.

Consider the value of w in the while loop, line 10 when i is increased to i +1 by the OPTIMIZESTEP function.

̸

Case 1: Suppose the coordinate w i = max S then we have V ( w,w ∗ ) ≥ V ( ¯ w,w ∗ ) for every ¯ w such that ¯ w j = w j for i = j and ¯ w i ∈ S . Therefore, by the contrapositive of the monotonicity property, this implies w i = arg min g ∈ S | g -( w ∗ ) i | . This establishes optimality for the co-orindate i since w out i = w i .

Case 2: Now suppose that w i < max S . A similar argument as above establishes that | w i -w ∗ i | ≤ | g -w ∗ i | for every g ∈ S such that g ≤ w i . The loop breaks since the condition in line 8 is false. Again, by the contrapositive of the monotonicity property, we must have | w i -w ∗ i | ≤ | αw i -w ∗ i | . This implies that w ∗ i < αw i and hence | w i -w ∗ i | ≤ | αw i -w ∗ i | ≤ | α h w i -w ∗ i | for all h ∈ N . Thus, we demonstrate that even in this case w i = arg min g ∈ S | g -w ∗ i | .

Therefore, coordinate-wise optimality holds for every i ∈ Supp ( w ∗ ) , completing the proof of the claim.

H Recall of Logical Combinations

We analyze the logical reasoning ability of the LLM to combine atomic features together in order to recover the ground-truth reward in Table 2.

I Sample Input Prompt

A sample input prompt with full context inputs is shown below.

J Prompts and Base Reward Functions

We provide a full list of the tested prompts and the ground truth Base reward functions in Table 3. We additionally include a full list of the features available for each arm (and used within proposed reward functions) in Figure 4.

We note that we evaluate on a wide variety of tasks within the real of resource allocation for maternal health. In particular, we test across feature characteristics such as age, income, spoken languages (an important characteristic indicating ethnicity in the Indian subcontinentuuh), education, and other relevant features that may be considered in real-world allocation tasks. We provide these tasks directly as context to the LLM, in addition to relevant feature characteristics and their corresponding descriptions. Per seed, we fix the randomness in the sampled set of 48 arms from ARMMAN's dataset, the deep RL neural network initialization, batch sampling from the dataset, and the sampling of states for each arm during evaluation. However please note that we cannot control the randomness of the Gemini LLM generation as we only have access through the Google API interface.

K Extended Results and Statistical Tests

K.1 Numerical Results

We show the full set of tasks and numerical results for the mean normalized reward score in Table 4.

slight preference.

K.2 Statistical Tests

Here we include the full prompt list for each task, base reward functions, as well as the numerical results given in Figure 2. We show the full set of statistical tests in Table 5.

L Chain-of-Thought Experiments

We note that methods that improve LLM reasoning, such as chain-of-thought (CoT), may be incorporated to improve the base LLM reasoning in the DLM pipeline. Below, we include results using chain-of-thought reasoning for our model outputs; while we did not find that CoT significantly improved reward, we nevertheless emphasize the potential for additional LLM techniques to improve DLM outputs.

M Sample Reward Reflection Output

Here we provide the full output of a sample reward reflection procedure, in addition to the corresponding selected LLM candidate reward.

M.1 Task: Older Bias

We add a full reflection output string for the Idx. 0: Older Bias task below. We make several observations on the proposed rewards and selected best candidate index. First, we note that the original base function uses only agent_feats[11] , which is the binary feature for the oldest age group Age 51-60. We see that the first proposed reward (Index 0) proposes using the top three oldest age groups, agent_feats[9],agent_feats[10],agent_feats[11] , while the second proposed reward (Index 1) uses only the top two oldest age groups agent_feats[9],agent_feats[10] . We additionally observe, in the state-feature distributions, that the Index 1 reward observes a higher percentage of positive state accumulated for the oldest age group Ages 51-60, with 31.65% accumulated state vs 28.17% for Index 0. Finally, the LLM selects reward at Index 1 as the top candidate reward, demonstrating that there is some reasonable similarity between the selection of the LLM and the intuitively preferred reward function. Additionally, in this case, the LLM reflection was able to help guide the selection of reward functions that prioritize the very oldest age groups, demonstrating some positive effect on feature extraction through reflection.

M.2 Marathi Speakers + Middle Aged

We next analyze a sample reward reflection for the Idx. 8: Marathi Speakers + Middle aged task below. Observe that the base reward function in Table 3 uses the features agent_feats[13] , the binary Marathi-speaking feature, and agent_feats[9],agent_feats[10] , the middle ages in the age range Ages 31-40 and Ages 41-50. Below, we observe that the first proposed reward includes these exact features, while the second proposed reward only includes one of the middle-income features agent_feats[9] . However, there is a key difference in the feature weightings applied in both reward functions, where the first proposed reward function weights the prioritized features much more with scalar multipliers than the first proposed function. In this case, the LLM selects the first proposed reward function, which also happens to be closer to the scaling used in the original Base reward function. While this is not known to the LLM, the first proposed reward function also results

in a greater accumulate state percentage in the relevant middle-age feature groups, a possible reason for the LLM selecting the first reward, thus demonstrating an example of reflection aiding in tuning feature weighting in reward functions.

M.3 Technically Challenged

In the final example, we compare a sample of reward reflection for the challenging, ambiguous "Technically Challenged" prompt. The ground truth Base reward for this task considers only the features of phone ownership, interpreting "Technically Challenged" as those women who do not own their own phone (using features agent_feats[24], agent_feats[25] . In this case, both proposed reward functions have low precision, including auxillary features that are not directly relevant to the Base reward. We observe, in this case, that although the second proposed reward, at Index 1, has utilized only one of the relevant features agent_feats[25] , it is ultimately selected in the reflection stage. For this highly ambiguous tasks, additional external input may be required, as we observe in this case that reflection does not align with what we may desire given the known Base reward function.

N Generated Reward Functions

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state and claims, and the claimed made in the abstract and introduction match the theoretical and experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The scope of the claims are clearly stated (see abstract, introduction, and experiments). Our approach is designed for restless multi-arm bandits tasks, and the experiments are focused on public health domains.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide all details on theoretical results, including necessary assumptions and detailed derivations.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all information needed to reproduce the experimental results, including training details (Section 5.3), a full task list and ground truth base reward function (see Appendix E), full output of sample reward reflection procedure (see Appendix F).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We use a real world dataset collected by a non-profit organization in public health domains. We are not allowed to release the data. However, we will release the code, and we describe dataset details (e.g. dataset size, features available) and dataset consent for collection and analysis in Appendix B and C.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section 5 on experiments, we provide detailed of the experimental setting, including details of the simulated public health setting, tasks, baselines, metrics, training details, and evaluation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars in the main experimental results (see Figure 2). We report how these errors bars are calculated (number of trails, number of samples per steps, etc) in Figure 2 caption. We also report results on statistical significance, specifically one-tailed t-test results, in Table 5 in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report information on computing resources in Appendix Section E

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our experiments confirm with NeurIPS code of ethics. For experiments on real-world data, we describe dataset consent for collection and analysis in Appendix C.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss societal impacts of this work in the social impact statement (see Section A in Appendix).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We discuss safeguards in place for responsible and safe use of data and models in Section 5. The public health information dataset is completely anonymized by the non-profit organization we collaborate with (see Section C for more details).

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers that produce the code and models used. Specifically, in Section 5.3, we clearly state that we use the Gemini Pro LLM model [1] from Google to generate reward functions, and train our downstream policy with RL using PPO [52].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. We conduct a secondary analysis of the ARMMAN dataset (see Appendix C for details).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: The data exchange from ARMMAN to the researchers was regulated through clearly defined exchange protocols including anonymization, read-only researcher access, restricted use of data for research purposes only, and approval by the ARMMAN ethics review committee (see Appendix C for details).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health\" introduces a novel method for improving public health resource allocation. By combining Restless Multi-Armed Bandit (RMAB) models with the interpretive power of Large Language Models (LLMs), the authors have created a system that can adaptively fine-tune health policies based on human-language commands. This innovation allows for more flexible and responsive policy adjustments, crucial for addressing the changing needs of public health programs.\nA significant contribution of this work is showing how LLMs can generate and refine reward functions for RMABs, enhancing decision-making in resource-limited settings. The authors demonstrate this through a collaboration with ARMMAN, an Indian non-profit focused on maternal health. Their simulations reveal that the DLM can significantly improve the allocation of health workers, leading to better engagement and outcomes. This approach promises to make public health interventions more effective and adaptable, closely aligning with the evolving needs of the community.\n\nSTRENGTHS:\nOriginality:\nI think the originality of this paper quite impressive. The way it combines Large Language Models (LLMs) with Restless Multi-Armed Bandits (RMABs) to dynamically fine-tune public health policies is really innovative. Using human-language commands to adjust these policies is a clever idea that bridges advanced AI techniques and practical decision-making in a unique way. This creative approach brings a fresh perspective to public health, making it much more adaptable and responsive to changing needs.\nQuality:\nThe quality of the research really stands out. The authors did a good job detailing their methodology, from the reward proposal loop to the simulation stages and the reflection mechanism for refining reward functions. For example, they explain how LLMs interpret policy preferences and generate reward functions, which are then fine-tuned through simulations. The experiments are well-thought-out and use real-world data from ARMMAN, which adds a lot of credibility. The results are impressive, demonstrating DLM can achieve near human-level performance. The authors also provide a comprehensive analysis, comparing their model to baseline methods using clear performance metrics. This thorough validation highlights the potential for further real-world applications, showcasing how the approach can dynamically adjust policies to meet evolving public health needs.\nClarity:\nThe paper is generally well-organized and easy to follow. The tables and figures do a great job of illustrating the key concepts and results. However, some parts could be more engaging and less technical, making it easier for a wider audience to understand. Simplifying some of the technical language and adding short summaries at the end of sections would help readers quickly get the main points without getting lost in the details. For example, a brief recap of the key findings at the end of the results section would be really helpful.\nSignificance:\nDespite being based on simulations, this work shows great potential for real-world impact. The DLM can dynamically adjust RMAB policies to meet changing public health needs, which is incredibly important. The evaluation on real-world data show that this approach isn't just theoretical but has practical relevance. The findings suggest that the DLM could significantly improve how resources are allocated and how effective policies are, which is very promising. Being able to dynamically prioritize different demographic groups or regions based on evolving needs can lead to more targeted and efficient use of resources. This approach could have wide applications in various public health interventions.\n\nWEAKNESSES:\nOne major limitation of the paper is its reliance on a simulated environment for validation, which is understandable given the complexity of real-world testing. While simulations using real-world data, the findings would be significantly strengthened by real-world trials. A practical next step would be to outline a detailed plan for field testing the Decision-Language Model (DLM) in actual public health settings. This would include partnerships with health organizations for pilot studies, addressing potential ethical and logistical challenges, and establishing metrics for real-world success.\n\nQUESTIONS:\nIs there any theoretical foundation you rely on for designing the prompts used in your model? Do you think referring to sociological theories or frameworks could help in designing more effective and contextually relevant prompts?\nThe paper mentions issues with ambiguous language prompts leading to misaligned policies. Do you have any specific methods or future plans to quantify and address these ambiguities to improve the model's reliability?\n\nLIMITATIONS:\nThe simulations in this work are impressive, but to make a real-world impact, the authors need to address some key areas. These include transitioning to actual field trials, expanding language support, ensuring the model can handle larger datasets, and dealing with ambiguous prompts. It's also important to consider potential negative societal impacts like data privacy, bias, and the effects on vulnerable groups. By tackling these issues and outlining a clear path for real-world use, the authors can significantly enhance the practical and ethical value of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This paper proposes using a decision-language model for restless multi-armed bandit tasks (RMAB) in the public health domain. The authors evaluated their method in a simulation environment developed from a real-world dataset. The authors conducted experiments with 16 different prompts and compared their approach with baselines, demonstrating the effectiveness of their method. Their study provided insights into the generated reward design for RMABs.

STRENGTHS:
1. The authors evaluated their method in a simulation environment developed from a real-world dataset and provided comparisons in the DLM-generated model with the baseline model. 
2. The author provides the parameters of the model training process, which helps other researchers reproduce the process.

WEAKNESSES:
1. Avoid citing sources in the Abstract
2. The approach provided in this paper lacks real-world validation. 
3. This paper did not discuss the ethical implications of using AI for decision-making in health resource allocation.

QUESTIONS:
1. Why use the ARMMAN dataset exclusively in this study? What language is used in this dataset? Given that the test involves prompts in English, was there any preprocessing of the dataset required?
2. In some parts of the training process, it is crucial to explain why specific numbers are chosen. For instance, the downstream policy is trained for 5 epochs,  why 5?  why are 100 simulation steps selected?

LIMITATIONS:
The approach provided in this paper lacks real-world validation. This paper should also discuss the ethical implications of using AI for decision-making in health resource allocation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRestless multi-armed bandits (RMAB) are effective for resource allocation in public health but lack adaptability to changing policies. Large Language Models (LLMs) have potential in healthcare for dynamic resource allocation through language prompts but are understudied in this area. This work introduces a Decision-Language Model (DLM) for RMABs to fine-tune resource allocation policies using language prompts. It proposes using LLMs to clarify policy preferences, propose reward functions, and iteratively refine these functions through simulations. Key contributions include pioneering the use of LLMs for adapting public health resource allocation and demonstrating near-human-level policy tuning in a maternal and child care task. They introduce a reward proposal loop that improves LLM-generated reward functions using feedback from restless multi-armed bandit (RMAB) simulations. This allows LLMs to iteratively refine reward designs to achieve specific, human-specified policy outcomes.\n\nSTRENGTHS:\n- It is very well written and fluent. \n- They highlighted the comments and important parts which is really helpful to follow the context. \n- The method is new in this application.\n\nWEAKNESSES:\n- The related works are not comprehensive. It should bring some works in healthcare from other approach and also the application of LLM in other healthcare examples. \n\n- The policy and critic is not identified till in the algorithm 1 and that is confused the reader. Since from the beginning it seems, the policy is also LLM. It requires more clarification and adjustment. \n\n- It is true that DLM (with reflection) works better but it is not a good fair comparison. Justifying why without reflection works good or bad is necessary. \n\n- The baseline are not fair. It is true that might be other methods with LLM is not yet applied to this healthcare problem but to claim why they choose this algorithm, comparing with other algorithms that use LLM is necessary. They need to bring enough evidence for their selection.  In this case they need to bring baseline from the work with LLMs.\n\nQUESTIONS:\n- What are the feature z exactly? It is very helpful to bring some clear examples in section or the beginning of section 4.\n\n- How the buffer is used in the Algorithm? It is not clear\n\n- Figure 2 is not a clear way to show the summary of the results. Maybe a table is better since now the difference is not clear. Now the question is why default at some cases has the same performance as DLM (no reflection)?\nWould not it because of the prompt?\n\n- Why some methods like CoT is not used as a baseline? Or ReAct or Reflexion?\n\nLIMITATIONS:\nIt still needs human prompt at the algorithm which make it less practical in terms of type of the community. \n\nThe amount of its automation is not very clear. It is good to add it to the discussion. \n\nThey need to compare more baselines.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}