Title: Leveraging automatic strategy discovery to teach people how to select better projects
Venue: NeurIPS 2023
Accepted: no
Keywords: metareasoning

automatic strategy discovery

decision-making

reinforcement learning

partially observable Markov decision process

project selection

metalevel Markov decision process

judge advisor systems

resource rationality
Abstract: Human decisions are often suboptimal due to limited cognitive resources and time constraints. Prior work has shown that errors in human decision-making can in part be avoided by leveraging artificial intelligence to automatically discover efficient decision strategies and teach them to people. So far, this line of research has been limited to simplified decision problems that are not directly related to the problems people face in the real world. Current methods are mainly limited by the computational difficulties of deriving efficient decision strategies for complex real-world problems through metareasoning. To bridge this gap, we model a real-world decision problem in which people have to choose which project to pursue, and develop a metareasoning method that enables us to discover and teach efficient decision strategies in this setting. Our main contributions are: formulating the metareasoning problem of deciding how to select a project, developing a metareasoning method that can automatically discover near-optimal project selection strategies, and developing an intelligent tutor that teaches people the discovered strategies. We test our strategy discovery method on a computational benchmark and experimentally evaluate its utility for improving human decision-making. In the benchmark, we demonstrate that our method outperforms PO-UCT while also being more computationally efficient. In the experiment, we taught the discovered planning strategies to people using an intelligent tutor. People who were trained by our tutor showed a significant improvement in their decision strategies compared to people who tried to discover good decision strategies on their own or practiced with an equivalent tutor that did not reveal the optimal strategy. Project selection is a very consequential high-stakes decision regularly faced by organizations, companies, and individuals. Our results indicate that our method can successfully improve human decision-making in naturalistic settings similar to the project selection decisions people face in the real-world. This is a first step towards applying strategy discovery methods to improve people's decisions in the real-world. 

Abstract

Human decisions are often suboptimal due to limited cognitive resources and time constraints. Prior work has shown that errors in human decision-making can in part be avoided by leveraging artificial intelligence to automatically discover efficient decision strategies and teach them to people. So far, this line of research has been limited to simplified decision problems that are not directly related to the problems people face in the real world. Current methods are mainly limited by the computational difficulties of deriving efficient decision strategies for complex real-world problems through metareasoning. To bridge this gap, we model a realworld decision problem in which people have to choose which project to pursue, and develop a metareasoning method that enables us to discover and teach efficient decision strategies in this setting. Our main contributions are: formulating the metareasoning problem of deciding how to select a project, developing a metareasoning method that can automatically discover near-optimal project selection strategies, and developing an intelligent tutor that teaches people the discovered strategies. We test our strategy discovery method on a computational benchmark and experimentally evaluate its utility for improving human decision-making. In the benchmark, we demonstrate that our method outperforms PO-UCT while also being more computationally efficient. In the experiment, we taught the discovered planning strategies to people using an intelligent tutor. People who were trained by our tutor showed a significant improvement in their decision strategies compared to people who tried to discover good decision strategies on their own or practiced with an equivalent tutor that did not reveal the optimal strategy. Project selection is a very consequential high-stakes decision regularly faced by organizations, companies, and individuals. Our results indicate that our method can successfully improve human decision-making in naturalistic settings similar to the project selection decisions people face in the real-world. This is a first step towards applying strategy discovery methods to improve people's decisions in the real-world.

1 Introduction

Corporations and individuals commonly have to select a project to pursue out of multiple alternatives. 29

These project selection problems are usually high-stakes decisions that can be highly impactful for the 30

future of an organization. For example, an organization looking for a sustainable investment project 31

could benefit both financially and by improving its public image by selecting an impactful and 32

profitable project, or incur major losses by selecting an unsuitable project. 33

Previous research on project selection recommends that candidate projects should be evaluated by 34

multiple experts [9, 17, 21], and many structured approaches to integrate the experts' opinions exist 35

[11]. However, structured project selection techniques are not well utilized in the real-world [28, 21], 36

and decision-makers often rely on their intuition and much simpler techniques like brainstorming [18]. 37 This is concerning because the intuitive decisions of groups and individuals are highly susceptible 38 to biases and unsystematic error [16]. However, people's errors in decision-making can partly be 39 prevented by teaching them better decision strategies. This approach is known as boosting [15]. 40

To identify appropriate decision strategies, we can score candidate strategies by their resource41 rationality , that is the degree to which they make rational use of people's limited time and bounded 42 cognitive resources [19]. In the resource-rational framework, the decision operations people can 43 perform to arrive at a decision are modeled explicitly and assigned a cost. The overall efficiency of 44 a decision strategy h in an environment e can then be computed by subtracting the expected costs 45 λ of the N used decision operations from the expected utility R total of the resulting decision (see 46 Equation 1) [10]. This measure is called resource-rationality score ( RR-score ) [10]. People are 47 usually not fully resource-rational, and identifying decision strategies would enable people to perform 48 as well as possible is an important open problem [6, 10, 14, 22]. 49

Recent work has demonstrated that the theory of resource rationality makes it possible to leverage AI to automatically discover and teach decision strategies that enable real people to make their decisions as well as possible [6, 10, 1, 32, 22]. This approach has been dubbed AI-powered boosting . The first step of AI-powered boosting is to compute resource-rational decision strategies. Automatic strategy discovery methods [6, 10, 14, 32, 22] can discover efficient decision strategies by solving the metareasoning problem of deciding which decision operations to perform. While recent work has extended automatic strategy discovery methods to larger [10] and partially observable environments [14], so far, they have not been applied to real-world decisions such as project selection.

In this article, we extend AI-powered boosting to improve how people select projects. Project selection is challenging because many crucial attributes of the candidate projects, such as their expected profitability, cannot be observed directly. Instead, they have to be inferred from observations that are not fully reliable. We, therefore, formalize project selection strategies as policies for solving a particular class of partially observable Markov Decision Processes (POMDPs). This formulation allows us to develop the first algorithm for discovering resource-rational strategies for human project selection.To achieve this, we model a realistic project selection task as a metareasoning problem. The metareasoning consists in deciding which information one should request from which advisors when information is highly limited, uncertain, and costly. We develop an efficient algorithm for solving this problem and apply it to derive the optimal decision strategy for a project selection problem a financial institution faced in the real world [17]. Finally, we develop an intelligent tutor [6] that teaches the decision strategy discovered by our method to people. We evaluated our approach by letting our intelligent tutor teach the automatically discovered project selection strategy to about 100 people, and then evaluated the quality of their decisions in realistic project selection problems against two control groups. Our results indicate that our approach can successfully improve human decisions in real-world problems where people are reluctant to let machines decide for them.

2 Background

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

Project selection In the project selection problem, a decision-maker aims to select the best-fitting 75 project out of several candidates [27]. Apart from a project's profitability, the evaluation usually also 76 considers other factors, such as the alignment with organizational goals [7]. This problem can be 77 formalized as multi-criteria decision-making (MCDM) [11, 23]. Projects can be evaluated using a 78 scoring technique, which evaluates relevant criteria and then combines them to a weighted sum [27]. 79 Common approaches to solving the project selection problem include techniques such as the analytic 80 hierarchy process, the analytic network process, real options analysis, and TOPSIS (see [11] for a 81 review). These methods are commonly combined with fuzzy sets to account for uncertainty [17]. 82 However, these methods are rarely used in real-world problems because implementing them would 83 require gathering and integrating a lot of information through a time-consuming process, which is 84 often incompatible with the organizational decision process [28, 21]. Instead, organizations often 85 rely on simpler, less structured methods like brainstorming [18]. In addition, the detailed information 86 required by these methods can be costly to acquire in real-world settings. 87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

Judge-advisor systems In a Judge Advisor System (JAS) [2], typically, a single decision-maker has to make a decision, and multiple advisors support the decision-maker by offering advice. Variations of the task can include costly advice [33, 12] , or advisors with varying reliability [24]. This is a common situation when CEOs decide which project their company should launch next. Unfortunately, decisionmakers are known to be highly susceptible to systematic errors, such as weighing one's own opinion too strongly, overconfidence, egocentric advice discounting, and weighting the recommendations of advisors by their confidence rather than their competence [2, 25, 33].

Strategy discovery methods Discovering resource-rational planning strategies can be achieved by solving a meta-level Markov decision process [13, 4, 5], which models the metareasoning process as a Markov Decision Process (MDP), which state represents the current belief about the environment and actions represent decision operations. Performing a decision operation results in a negative cost and results in an update to the belief state. A special termination action represents exiting the metareasoning process and making a real-world decision, guided by the current beliefs [13]. Multiple methods for solving meta-level MDPs exist (e.g. [26, 4, 13, 10]). We refer to these algorithms as strategy discovery methods [4, 6, 32, 10, 14, 22]. They learn or compute policies for selecting sequences of cognitive operations (i.e., computations) people can perform to reach good decisions.

MGPO [14] is currently the only strategy discovery algorithm that can efficiently approximate resource-rational strategies for decision-making in partially observable environments. MGPO chooses decision operations by approximating their value of computation [26] in a myopic manner: it always selects the computation whose execution would yield the highest expected gain in reward if a decision had to be made immediately afterward, without any additional planning.

Cognitive tutors Past work has developed cognitive tutors that teach automatically discovered planning strategies to people [6, 32, 22, 10]. Training experiments indicated that training with these cognitive tutors could significantly boost the quality of people's planning and decision-making [6, 32, 10, 14, 22]. These cognitive tutors teach efficient decision strategies in an automated manner, usually by computing the value of available decision operations using strategy discovery methods, and providing the learner feedback on the quality of the computations they select. Initially limited to small planning tasks due to the computational complexity of solving meta-level MDPs [20, 6], recent work has extended existing methods to larger [10] and partially observable [14] settings. However, none of these methods have been applied to naturalistic problems so far.

A crucial obstacle is that these methods are limited to settings where all decision-relevant information comes from the same source. By contrast, in the real world, people have to choose between and integrate multiple different sources of information. In doing so, they have to take into account that some information sources are more reliable than others. Additionally, current strategy discovery methods are limited to artificial settings where each piece of information is an estimate of a potential future reward. By contrast, in the real world, most information is only indirectly related to future rewards, and different pieces of information have different units (e.g., temperature vs. travel time).

3 Formalizing optimal decision strategies for human project selection as the solution to a meta-level MDP

In this section, we introduce explain our general resource-rational model of project selection, which we expect to be widely applicable to concrete, real-world project selection problems.

Our model of project selection consists of two decision problems, an object-level decision-problem and a meta-level MDP [4, 13]. The two decision problems separate the actions the decision-maker has to choose between (object level), such as executing one project versus another, from decision operations that represent thinking about which of those object-level actions to perform (meta-level), such as gathering information about the projects' attributes. This allows us to solve both problems separately. The object-level decision problem is a MCDM problem, where a set of N P potential projects P = { p 1 , ...p N P } are evaluated using N C relevant criteria C = [ c 1 , ...c N C ] weighted by fixed predetermined weights W = [ w 1 , ...w N C ] . Actions in the object-level problem represent selecting the corresponding project ( A = { a 1 , ..., a N P } ). The reward of a selecting a project is computed by summing the weighted criteria scores of the selected project: r O ( a i ) = ∑ c w c c c,i [9].

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

While the object-level decision problem is our model of the project selection task, the meta-level

MDP is our formalization of the problem of discovering resource-rational project selection strategies.

It models the task of gathering information about deciding which project to select. States in the meta- level MDP are belief states that represent the current information about each project's attributes. We

model belief states using a multivariate Normal distribution to quantify the estimated value and uncer- tainty about the N P projects' scores on the N C criteria: b = [( µ 1 , 1 , σ 1 , 1 ) , · · · , ( µ N P ,N C , σ N P ,N C )] . The actions (decision operations) of the meta-level MDP gather information about the different attributes of projects by asking one of the N E experts for their estimate of how one of the projects scores on one of the criteria. Experts provide discrete estimates from min obs to max obs , and expert estimates can differ in their reliability and their cost. Specifically, the available actions are A M = { a 1 , 1 , 1 , · · · , a N P ,N C ,N E , ⊥} , where the meta-level action a i,j,k represents asking the expert e k for their estimate of criterion c j of project p i . After receiving information obs from an expert, the current belief state b p i c j = N ( µ, σ ) is updated by applying the update equation for a Gaussian likelihood function with standard deviation σ e (i.e. the expert's reliability) and a conjugate Gaussian prior (i.e., the current belief), that is ˆ µ ← ( w c i · µ ( w c i · σ ) 2 + w c i · obs ( w c i · σ e ) 2 ) · ( ( w c i · σ ) 2 +( w c i · σ e ) 2 ) and

ˆ

σ

√

1

+

(

σ

1

·

s

) 2

(

1

σe

·

s

) 2

The reward of these meta-level actions is the negative cost r M ( a i,j,k ) = -λ e k of asking the expert e k for help. Finally, the meta-level action ⊥ is the termination action, which corresponds to terminating the decision-making process and selecting a project. The reward of the termination action is the expected reward of selecting the best project according to the current belief-state. An optional budget parameter N a specifies the maximum number of available meta-level actions, after which the termination action is performed automatically.

Meta-level MDPs are notoriously difficult to solve due to their extremely large state space [10, 13]. In the project selection task, the meta-level MDP has ( max obs -min obs +2) N P · N C · N E possible belief states and N P · N C · N E +1 possible meta-level actions. Our meta-level MDP introduces multiple new intricacies that current metareasoning methods like MGPO [14] aren't equipped to handle, making strategy discovery in this setting especially difficult. Compared to previously formulated meta-level MDPs [4, 13, 10, 14, 22], our meta-level description of project selection differs in the following ways: (1) the maximum amount of meta-level actions is constrained with a budget, (2) the project selection task features multiple consultants who differ in both the quality of their advice and the cost of their services, (3) consultants in the project selection task offer discrete estimates of each criterion , requiring that (4) criteria ratings are scaled to allow weighting the criteria according to their importance.

4 A new metareasoning algorithm for discovering optimal decision strategies for human project selection

Previous metareasoning methods are unable to handle some of the intricacies of the project selection problem. Therefore, we developed a new strategy discovery method based on MGPO [14], which overcomes the limitations that prevent MGPO from being applicable to project selection. To reflect the commonalities and innovations, we call our new strategy discovery method MGPS (meta-greedy policy for project selection). Similar to MGPO, our method approximates the value of computation (VOC) [26] by myopically estimating the immediate improvement in decision quality. Improving upon MGPO, MGPS calculates the myopic approximation to the VOC in a way that accounts for discrete criteria ratings, criteria scaling, and multiple sources of information with different costs and reliabilities.

MGPS calculates a myopic approximation to the VOC of asking an expert about a specific criterion of a single project according to Algorithm 1. To account for discrete advisor outputs, Algorithm 1 iterates over the discrete set of possible ratings the expert might give and estimates the probability p obs of each rating ( obs ) and the belief update that would result from it ˆ µ obs . The probability of each rating is computed using the cumulative distribution function ( Φ ) of the belief state for the selected project's criterion score (see Line 7). Here, the standard deviation σ e of the likelihood function encodes the expert's reliability, the prior ( N ( µ, σ ) ) is the current belief about the project's score on the evaluated criterion, and the weights w c i convert the criteria into a common currency. The belief update that would result from the observation ( ˆ µ obs ) is computed by applying the belief state update

.

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

(see Line 13 and Equation 3). For the highest and lowest possible ratings, we instead calculate p obs using the open interval (see Lines 9 and 11). The updated expected value of the belief state according to an observation obs is then calculated using Bayesian inference to integrate the new observation into the belief state (see Line 13).

The VOC calculation depends on the posterior belief states that would result from the different possible observations ( ˆ µ obs ), weighted by their probabilities. If the evaluated project currently has the highest expected reward (see Line 15), the VOC calculation expresses the probability of observing a value low enough that the second-best project becomes the most promising option (see Line 16). In the case where the evaluated project did not have the highest expected termination reward, the VOC calculation expresses the probability of observing a value high enough to make the evaluated project the most promising option (see Line 18). Finally, the cost of the requested expert λ e i is weighted using a free cost weight parameter w λ and subtracted from the VOC estimate (see Line 20).

The full meta-greedy policy consists of calculating the VOC for all possible meta-level actions and iteratively selecting the meta-level action with the highest VOC. If no action has a positive VOC, the termination action ⊥ is chosen.

5 Improving human project selection

Having developed a general metareasoning method for discovering resource-rational strategies for human project selection, we now extend it to an intelligent cognitive tutor for teaching people how to select better projects. Our goal is to provide a proof of concept for a general AI-powered boosting approach that can be used to improve human decision-making across a wide range of project selection problems. We first introduce a general approach for teaching people the project selection strategies discovered by MGPS, and then apply it to a real-world project selection problem.

5.1 MGPS Tutor: An intelligent tutor for teaching people how to select better projects

Our intelligent tutor for project selection ( MGPS Tutor ) trains people to select the near-optimal decision operations identified by MGPS. To achieve this, it lets people practice on a series of project

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

Project 1

Project 2

Expert 1Expert 2Expert 3 Expert 4 Expert 5Expert 6Current Expert 1Expert 2Expert 3Expert 4 Expert 5 Expert6Current

★1

★

★1

★1

★1

★

Estimate

★

★1

★1

★1

★

Estimate

Economiceffects

3.6

3.6

scale: 0.02

★

Social effects

3.17

3.17

scale: 0.07

★★

★★

Environmentaleffects

3.6

3.6

scale: 0.22

★★

Strategic alliance

3.13

3.13

scale: 0.11

★★

Organizationalreadiness

3.25

3.67

scale: 0.47

5

★★#

★1

Riskofinvestment

2.3

2.3

scale: 0.12

★★1

★★1

Estimated

3.21

3.4

performance

★★★1

★★★

selection problems and gives them feedback. MPGS Tutor leverages MPGS in two ways: i) to pedagogically construct the set of queries the learner is asked to choose from, and ii) to give the learner feedback on their chosen query.

Building on the choice tutor by [14], our tutor repeatedly asks the learner to choose from a pedagogically chosen set of decision operations (see Figure 1) that always includes the query that MGPS would have performed. Moreover, it leverages MGPS's VOC calculation (Algorithm 1) to score the chosen query, and then provides binary feedback on its quality. If learners select a suboptimal expert, project, or criterion, they receive feedback indicating the correct choice and have to wait for a short time. The unpleasantness of having to wait serves as a penalty [6]. Otherwise, they receive positive reinforcement and the next choice is displayed. To receive positive reinforcement, the learner must select a query whose VOC is sufficiently close, as determined by a tolerance parameter t , to the VOC of the optimal action. We set the tolerance to t = 0 . 001 .

Our tutor teaches the strategy discovered by MGPS using a novel sophisticated training schedule, which fosters learning by incrementally increasing the complexity of the training task. This learning methodology is also known as shaping [31], and has been successfully applied to teach decision strategies to humans [14]. Our training schedule varies the numbers of projects, how many different expert assessments learners have to choose between, and the specific types of expert assessments offered as choices. In total, our tutor teaches the discovered project selection strategy using ten training trials. The first seven training trials use a smaller version of the project selection task with only two projects, while the last three trials use the full environment with five projects. The number of choices gradually increases throughout training from 1 in the first training trial to 9 in the last three training trials. The tutor varies the types of choices across trials. After an initial trial with only a single choice, the tutor offers choices that focus on different criteria within the same project for two trials. Then, the tutor offers choices that focus on different experts within the same project for two trials. The remaining trials combine both types of highlights while sometimes also featuring queries about different projects and also increasing the overall number of choices.

5.2 Evaluating the effectiveness of MGPS Tutor in a training experiment

To evaluate if AI-powered boosting can improve human project selection, we tested the MPGS tutor 244 in a training experiment. We tested if people trained by MPGS tutor learn more resource-rational 245 project selection strategies. To make our assessment task as naturalistic as possible, we modelled it 246 on a real project selection problem that was faced by an Iranian financial institution [17]. We will 247 first describe how we modeled this real-world problem, and then the training experiment. 248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

Aproject-selection problem from the real world Khalili-Damghani and Sadi-Nezhad [17] worked on the real-world problem of helping a financial institution select between five potential projects with an eye to sustainability. Each project was evaluated by six advisors, who assigned scores from one to five on six different criteria. For our model of the task, we use the same number of experts, criteria, and projects, and the same criteria weights as the financial institution. The remaining parameters of the meta-level MDP were estimated as follows. We initialized the beliefs about the project's attributes by calculating the mean and the standard deviation of all expert ratings for each criterion according to [17]. We estimated the reliability of each expert by calculating the standard deviation from the average distance of their ratings from the average rating of all other experts, weighted by the number of occurrences of each guess. We estimated the cost parameter λ of the meta-level MDP by λ = cost stakes · r ( ⊥ ) to align the meta-level MDP's cost-reward ratio to its real-world equivalent. Using the expected termination reward of the environment r ( ⊥ ) = 3 . 4 and rough estimates for the stakes stakes = $10000000 and expert costs cost = $5000 , this led to λ = 0 . 002 . While [17] assumed all expert ratings are available for free, this is rarely the case. To make our test case more representative, we assumed that advisor evaluations are available on-request for a consulting fee. To capture that real-world decisions often have deadlines that limit how much information can be gathered, we set the maximum number of sequentially requested expert consultations to 5 .

Methods of the experiment We recruited 301 participants for an online training experiment on Prolific. The average participant age was 29 years, and 148 participants identified as female. Participants were paid £3.50 for completing the experiment, plus an average bonus of £0.50. The median duration of the experiment was 22 minutes, resulting in a median pay of £10.9 per hour. Our experiment was preregistered on AsPredicted and approved by the ethics commission of [removed] under IRB protocol [removed].

Each participant was randomly assigned to one of three conditions: (1) the No tutor condition, in which participants did not receive any feedback and were free to discover efficient strategies on their own; (2) the MGPS tutor condition, in which participants practiced with our cognitive tutor that provided feedback on the resource-rationality score MGPS assigns to the selected planning actions; and (3) the Dummy tutor condition, an additional control condition in which participants practiced with a dummy tutor comparable to the MGPS tutor, albeit with randomized feedback on which planning actions are correct. All participants practiced their planning strategy in 10 training trials and were then evaluated across 10 test trials.

We evaluated the participants' performance using two measures: their RR-score and click agreement . RR-score's are normalized by subtracting the average reward of a random baseline algorithm and dividing by the participant scores' standard deviation. The random baseline algorithm is defined as the policy that chooses meta-level actions at random until the maximum number of decision operations is reached. Click agreement measures, how well participants learned to follow the near-optimal strategy discovered by our method. Specifically, we computed for each participant, which proportion of their information requests matched the action taken by the approximately resource-rational strategy discovered by MGPS.

Experiment results Table 1 shows the results of the experiment. To determine whether the condition of participants had a significant effect on the RR-score and click agreement, we used an ANOVA analysis with Box approximation [3]. The ANOVA revealed a significant effect of condition on both RR-score ( F (1 . 99 , 293 . 57) = 10 . 48 , p < . 0001 ) and click agreement ( F (1 . 99 , 291 . 48) = 15 . 5 , p < . 0001 ). We further compared the performance of participants in the MGPS tutor condition to participants in the two control conditions with post hoc ANOVA-type statistics and used Cohen's d

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

[8] to evaluate the size of the effects. The post hoc tests revealed that participants in the MGPS tutor condition achieved a significantly higher RR-score than participants in the No tutor ( F (1) = 17 . 88 , p < . 0001 , d = . 35 ) and Dummy tutor ( F (1) = 13 . 4 , p = . 0002 , d = . 31 ) conditions. Additionally, participants in the MGPS tutor reached a higher click agreement with our pre-computed near optimal strategy than participants in the No tutor ( F (1) = 25 . 08 , p < . 0001 , d = . 58 ) and Dummy tutor ( F (1) = 19 . 3 , p < . 0001 , d = . 56 ) conditions.

When evaluated on the same test trials and normalizing against the baseline reward and the standard deviation of the experiment results, MGPS achieves a mean reward of 1 . 17 , demonstrating that MGPS discovers more resource-rational strategies than participants across all conditions. Although participants in the MGPS tutor condition performed significantly the better than participants in the other conditions, they did not learn to follow the strategy taught by the tutor exactly. Participants in the other conditions only discovered strategies with a similar RR-score to the random baseline strategy, with participants in the No tutor condition performing even worse than the random baseline strategy, and participants in the Dummy tutor condition outperforming the random baseline only by a small margin.

6 Performance evaluation

The results reported in the previous section show that MPGS can discover project selection strategies that are more effective than the strategies people discover on their own. But how does its performance compare to other strategy discovery algorithms? To answer this question, we evaluated MGPS on a computational benchmark. We chose PO-UCT [29] for comparisons because it is an established baseline for metareasoning algorithms in partially observable environments [14] and the more specialized MGPO algorithm is not applicable to project selection. PO-UCT utilizes Monte Carlo tree search to simulate the effects of different actions, resulting in more accurate results with increased computation time, making it a useful baseline for MGPS's computational efficiency and performance.

Method We evaluated the effectiveness of our method in the project selection task by comparing it against PO-UCT [29] with different numbers of steps. All methods were evaluated across the same 5000 randomly generated instances of the project selection environment.

Our main performance measure was the resource-rationality score ( RR-Score defined in Equation 1). To highlight the achieved improvements over a baseline algorithm that performs random meta-level actions, we normalized the reported RR-scores . Specifically, we applied a z-score transformation, subtracting the average reward of the random baseline algorithm (see Section 5.2) from the RRScores and dividing by the evaluated algorithm's RR-Scores ' standard deviation. We analyze the differences in RR-Scores with an ANOVA and evaluate the size of statistical effects with Cohen's d [8]. Additionally, we compare the computational efficiency of the different methods, which is crucial for being able to provide real-time feedback in our cognitive tutor.

Results As shown in Table 2, MGPS outperformed all tested versions of PO-UCT and the random baseline strategy. 1 An ANOVA revealed significant differences in the RR-scores of the strategies discovered by the different methods ( F (4 , 24995) = 2447 , p < . 0001 ). Hukey-HSD post-hoc

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

comparisons indicated that the strategies discovered by MGPS are significantly more resourcerational than the strategies discovered by PO-UCT with 10 steps ( p < . 0001 , d = 2 . 18 ), 100 steps ( p < . 0001 , d = . 27 ), 1000 steps ( p < . 0001 , d = . 14 ), or 5000 steps ( p < . 0001 , d = . 11 ). While MGPS achieves significantly higher RR-scores than all PO-UCT variants, the size of the effect decreases from a very large effect to a small effect when increasing PO-UCT's computational budget sufficiently. We therefore expect that PO-UCT with a much more than 5000 steps would ultimately achieve comparable RR-scores to MGPO, albeit at a much higher computational cost. Moreover, MGPS was substantially faster than PO-UCT with a computational budget of 1000 steps or more, which is when PO-UCT's performance starts to approach that of MGPS. With a computational budget of 100 steps or fewer, PO-UCT is faster than MGPS. However, such a small computational budget is not enough for PO-UCT to discover strategies with a RR-score anywhere near that of the strategy discovered by MGPS. Critically, the high amount of computation required for PO-UCT to achieve an approximately similar level of resource-rationality would render PO-UCT unusable for a cognitive tutor that computes feedback in real time.

7 Conclusion

People's decision-making is prone to systematic errors [16], and although people are happy to delegate some decisions, most CEOs are unlikely to let AI decide which projects their company should work on. Moreover, people are reluctant to use the more accurate technical decision procedures because they tend to be more tedious [28, 21, 18]. Motivated by people's insistence on making their own decisions with simple heuristics, we leveraged AI to discover and teach decision strategies that perform substantially better than people's intuitive strategies but are nevertheless simple enough that people use them. To this end, we introduced a metareasoning method for leveraging AI to discover optimal decision strategies for human project selection. Modeling project selection through the lens of resource rationality allowed is to formulate a mathematically precise criterion for the quality of decision strategies for human project selection. We further develop an efficient automatic strategy discovery algorithm automatically discovers efficient strategies for human project selection. Our algorithm discovered decision strategies that are much more resource-rational than the strategies humans discovered on their own and the strategies discovered by a general-purpose algorithm for solving POMDPs (PO-UCT). Using the efficient decision strategies discovered by our algorithm, we create a cognitive tutor that uses a shaping schedule and metacognitive feedback to teach the strategies to humans. In the training experiment, our cognitive tutor fostered significant improvements in participants' resource rationality.

A main limitation of our method is that it is unknown how precisely the environment parameters need to be estimated to construct the metareasoning task, which can prove especially problematic when there isn't much data on past decisions. Future work could investigate and potentially address it by extending MGPS with a Bayesian inference approach to estimate the environment structure. Encouraged by the promising results from successfully teaching humans in our naturalistic model of project selection, we are excited about future work assessing the real-world impact of improving people's decision-making by evaluating their decisions directly in the real world. Additionally, we are also excited about potential future work that combines MGPS with AI-Interpret [32] to automatically derive human-legible recommendations for how to make project selection decisions. Lastly, although MGPS performed very well on our benchmarks, MGPS's myopic approximation could fail in more complicated scenarios with interdependent criteria. Such challenges could be addressed by solving meta-level MDPs with methods from deep reinforcement learning, for example by utilizing AlphaZero [30].

Our results indicate that it is possible to use resource-rational analysis combined with automatic strategy discovery to improve human decision-making in a realistic project selection problem. As selecting projects is a common problem faced by both organizations and individuals, improving their decision strategies in this setting would have a direct positive impact. For example, a project-selection tutor could be integrated into MBA programs to teach future decision-makers efficient decision strategies as part of their education. We are optimistic that our general methodology is also applicable to other real-world problems, offering a promising pathway to teach people efficient strategies for making better decisions in other areas as well. Besides project selection problems, we believe our approach could be used to improve real-world decision-making in areas such as career choice, grant making, and public policy.

387

388

389

References

F. Becker, J. Skirzy´ nski, B. van Opheusden, and F. Lieder. Boosting human decision-making with ai-generated decision aids. Computational Brain & Behavior , pages 1-24, 2022.

S. Bonaccio and R. S. Dalal. Advice taking and decision-making: An integrative literature review, and 390 implications for the organizational sciences. Organizational behavior and human decision processes , 101 391 (2):127-151, 2006. 392

G. E. Box et al. Some theorems on quadratic forms applied in the study of analysis of variance problems, i. 393 effect of inequality of variance in the one-way classification. The annals of mathematical statistics , 25(2): 394 290-302, 1954. 395

F. Callaway, S. Gul, P. M. Krueger, T. L. Griffiths, and F. Lieder. Learning to select computations. 396 Uncertainty in Artificial Intelligence , 2018. 397

F. Callaway, F. Lieder, P. Das, S. Gul, P. M. Krueger, and T. Griffiths. A resource-rational analysis of 398 human planning. In CogSci , 2018. 399

F. Callaway, Y. R. Jain, B. van Opheusden, P. Das, G. Iwama, S. Gul, P. M. Krueger, F. Becker, T. L. Griffiths, 400 and F. Lieder. Leveraging artificial intelligence to improve people's planning strategies. Proceedings of the 401 National Academy of Sciences of the United States of America , Mar. 2022. doi: 10.1073/pnas.2117432119. 402 URL https://www.pnas.org/doi/10.1073/pnas.2117432119 . 403

A. F. Carazo, I. Contreras, T. Gómez, and F. Pérez. A project portfolio selection problem in a group 404 decision-making context. Journal of Industrial & Management Optimization , 8(1):243, 2012. Publisher: 405 American Institute of Mathematical Sciences. 406

J. Cohen. Statistical power analysis for the behavioral sciences . Academic press, 2013. 407

S. Coldrick, C. Lawson, P. Ivey, and C. Lockwood. A decision framework for r&d project selection. In 408 IEEE International Engineering Management Conference , volume 1, page 413-418. IEEE, 2002. 409

S. Consul, L. Heindrich, J. Stojcheski, and F. Lieder. Improving human decision-making by discovering 410 efficient strategies for hierarchical planning. Computational Brain & Behavior , 5(2):185-216, 2022. 411

D. G. B. de Souza, E. A. dos Santos, N. Y. Soma, and C. E. S. da Silva. MCDM-Based R&D Project 412 Selection: A Systematic Literature Review. Sustainability , 13(21):11626, 2021. Publisher: MDPI. 413

F. Gino. Do we listen to advice just because we paid for it? the impact of advice cost on its use. 414 Organizational Behavior and Human Decision Processes , 107:234-245, 2008. 415

N. Hay, S. Russell, D. Tolpin, and S. E. Shimony. Selecting computations: Theory and applications. arXiv 416 preprint arXiv:1408.2048 , 2014. 417

L. Heindrich, S. Consul, and F. Lieder. Leveraging ai to improve human planning in large partially 418 observable environments. arXiv preprint arXiv:2302.02785 , 2023. 419

R. Hertwig and T. Grüne-Yanoff. Nudging and boosting: Steering or empowering good decisions. 420 Perspectives on Psychological Science , 12(6):973-986, 2017. 421

D. Kahneman, S. P. Slovic, P. Slovic, and A. Tversky. Judgment under uncertainty: Heuristics and biases . 422 Cambridge university press, 1982. 423

K. Khalili-Damghani and S. Sadi-Nezhad. A hybrid fuzzy multiple criteria group decision making approach 424 for sustainable project selection. Applied Soft Computing , 13(1):339-352, 2013. 425

426

427

B. Kornfeld and S. Kara. Selection of Lean and Six Sigma projects in industry. International Journal of Lean Six Sigma , 2013. Publisher: Emerald Group Publishing Limited.

428

429

430

431

432

433

434

435

436

437

438

439

440

F. Lieder and T. L. Griffiths. Resource-rational analysis: understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences , 43, 2020.

F. Lieder, F. Callaway, Y. Jain, P. Krueger, P. Das, S. Gul, and T. Griffiths. A cognitive tutor for helping people overcome present bias. In RLDM 2019 , 2019.

F. Liu, W.-d. Zhu, Y.-w. Chen, D.-l. Xu, and J.-b. Yang. Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach. Scientometrics , 111(3):1501-1519, 2017. Publisher: Springer.

A. Mehta, Y. R. Jain, A. Kemtur, J. Stojcheski, S. Consul, M. Toši´ c, and F. Lieder. Leveraging machine learning to automatically derive robust decision strategies from imperfect knowledge of the real world. Computational Brain & Behavior , 5(3):343-377, 2022.

V. Mohagheghi, S. M. Mousavi, J. Antucheviˇ cien˙ e, and M. Mojtahedi. Project portfolio selection problems: a review of models, uncertainty approaches, solution techniques, and case studies. Technological and Economic Development of Economy , 25(6):1380-1412, 2019.

K. Olsen, A. Roepstorff, and D. Bang. Knowing whom to learn from: individual differences in metacogni441 tion and weighting of social information. PsyArXiv , 2019. 442

D. Ronayne, D. Sgroi, et al. Ignoring good advice . University of Warwick, Centre for Competitive 443 Advantage in the Global . . . , 2019. 444

S. Russell and E. Wefald. Principles of metareasoning. Artificial intelligence , 49(1-3):361-395, 1991. 445

S. Sadi-Nezhad. A state-of-art survey on project selection using mcdm techniques. Journal of Project 446 Management , 2(1):1-10, 2017. 447

R. L. Schmidt and J. R. Freeland. Recent progress in modeling R&D project-selection processes. IEEE 448 Transactions on Engineering Management , 39(2):189-201, 1992. Publisher: IEEE. 449

D. Silver and J. Veness. Monte-carlo planning in large pomdps. Advances in neural information processing 450 systems , 23, 2010. 451

452

453

454

455

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. P. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. CoRR , abs/1712.01815, 2017. URL http://arxiv.org/abs/ 1712.01815 .

B. Skinner. Shaping and maintaining operant behavior , pages 91-106. Free Press New York, 1953. 456

J. Skirzy´ nski, F. Becker, and F. Lieder. Automatic discovery of interpretable planning strategies. Machine 457 Learning , 110:2641-2683, 2021. 458

I. Yaniv and E. Kleinberger. Advice taking in decision making: Egocentric discounting and reputation 459 formation. Organizational behavior and human decision processes , 83(2):260-281, 2000. 460

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works addresses two problems: first, the problem of solving a specific class of information-gathering decision processes, and second, applying the resulting algorithm in the real world by creating an intelligent tutoring system. The algorithm is a straightforward greedy optimizer that gathers information based on maximizing one-step value of information.  The intelligent tutor blends the algorithm with a UI that gives humans feedback on their decisions by comparing them to the algorithm's; the paper shows that the humans implicitly learn from this feedback and improve their decision making abilities.  The paper concludes with a few small comparisons of the algorithm to other algorithms.\n\nSTRENGTHS:\n+ The paper is well-written\n+ The problem seems reasonably well formulated\n+ The algorithm is natural and reasonable\n+ The algorithm seems to perform well compared to alternatives\n+ The \"intelligent tutor\" is a bit of a misnomer, but does seem to improve human decision making ability\n\nWEAKNESSES:\n- This paper blends two distinct ideas, and does not take either very deep. The MGPS algorithm is straightforward, and the authors spend very little time (for example) analyzing its properties or connecting it seriously with the vast literature on decision processes.  (For example, I'm surprised not to see any connection to bandits).  I think you could have written a whole paper about MGPS -- establishing properties, contrasting with other algorithms, etc.  For example, a regret bound or similar form of theoretical analysis would have been nice.\n\nOn the other hand, the \"intelligent tutor\" seems like a very simple UI that gives very basic feedback.  While the authors do demonstrate some effectiveness, it too is not investigated deeply.  There is no comparison, for example, to other forms of UI, to other forms of feedback, etc.\n\nSo, it's hard to say: is this paper about MGPS, or intelligent tutor UI/UX design?  And while I appreciate that the authors probably have a vision of \"solving a real-world problem\" with this combination of ideas, I have to say that the problem formulation seems pretty far away from something that could actually be used in a business decision support system.\n\nQUESTIONS:\n1) I am surprised by the findings of the paper.  While the authors repeatedly refer to MGPS as generating \"optimal strategies\" and humans as using \"simple heuristics\", this terminology is hard to justify, since MGPS is itself a greedy heuristic.  I would have intuitively thought that humans would also try to maximize one-step information gain.  My question is this: do the authors have any sense of what strategy humans were using before exposure to the tutor? Why is their intuitive heuristic so bad? Can they (or you) somehow articulate what the humans are learning?\n\nLIMITATIONS:\nThe authors do not explicitly discuss the limitations of their algorithm/UI in a dedicated section.\n\nI do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe problem this paper tries to tackle is how to improve human decision making in the specific problem of selecting a project between a candidate set of existing projects. The strategy to improve the human's decision making is to build an agent that can solve the project selection itself by framing as a POMDP and then having that agent serve as a tutor during a teaching phase. The agent is learned by first framing the problems as a POMDP with Guassian states and the paper proposes a myopic algorithm to select actions (actions allow you to acquire information about each project on a specific dimension of the Guassian). The main evaluation of the paper is with a user study where human participants are trying to select between multiple projects in terms of estimated performance. The authors show that participants who received training with the tutor performed better than participants without a tutor or with a dummy tutor.\n\nSTRENGTHS:\n- very well-designed and rigorous user study that shows that the agent and the tutoring was effective.\n\nWEAKNESSES:\n- (main reason for low rating) out of scope of NeurIPS: this paper seems like a bad fit for NeurIPS, the paper does not contribute new algorithms that are broadly applicable and does not evaluate methods on broadly recognized datasets or benchmarks.  This papers models the problem of human project selection as a PODMP and proposes a relatively straightforward procedure and evaluates on a synthetic task of project selection. I don't see any insights that the community might benefit from. I think this would be a strong paper for a conference that suits it more. \nA bad (but still useful) heuristic is to look at the references where I count a single NeurIPS paper (from 2010) and one UAI paper (among a lot of management and behavioral science citations that don't include other human-ai conferences like CHI). \n\n- why not display the recommendation of the AI agent during test time as a baseline? since the AI agent performs well at the task, participants should be able to see its recommendations at test time. \n\n- the proposed algorithm while adequate for the problem is not a generalizable solution for the broad family of problems that are of interest in the human-AI space. In particular, the myopic approximation is very limiting. While section 6 does evaluate it against (a relatively old) baseline in PO-UCT, it limits the baseline to 5000 steps because of runtime constraints, however, it might be possible to optimize the performance of PO-UCT to be faster. \n\n- novelty: the method is a modification over MGPO [14] which introduces the myopic approximation, the proposed method MGPS modifies it to make it suitable for project selection. The authors do a good job of comparing to MGPO, but novelty overall is limited.\n\n- value of tutoring for project selection: why not just follow the recommendations of MGPS? the tutoring is specific to the problem domain, thus I don't understand anything that the human can take away from the tutoring to future tasks except the one they will encounter. Moreover, for the real world problem presented, it is a one shot decision problem, so tutoring is not as well motivated.\n\nQUESTIONS:\n(from weaknesses) why not display the recommendation of the AI agent during test time as a baseline? since the AI agent performs well at the task, participants should be able see it's recommendations at test time. \n\nfor section 6, how are the different environments of the project selection task generated? \n\nComments:\n- in title and sections, first letter of each word must be in capital\n\nLIMITATIONS:\nlimitations are well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of project selection (how does a human choose which, among a set of possible projects, is the best one to pursue). To address this problem, they develop an algorithm called MGPS that discovers a rational greedy strategy for solving this problem, and then they attempt to teach this strategy to a human via an intelligent tutoring system. The approach is evaluated in a real world project selection scenario.\n\n- page 2, line 64: \"selection.To\" -> \"selection. To\"\n- page 3, line 127: \"we introduce explain our general\" -> \"we introduce our general\"\n\nSTRENGTHS:\nIt is interesting to see a paper that goes all the way from finding an algorithm to solve a problem, to teaching humans based on that strategy and then evaluating human performance after being taught.\n\nWEAKNESSES:\nThe same strength of the paper seems to also be its main weakness: because the authors try to tackle a whole large problem end to end, I did not find each if the individual pieces that ground breaking. Perhaps if the paper focused on just one of the problems (finding optimal strategies, or just better techniques for teaching the learned strategies), maybe a stronger contribution would arise, by more systematically addressing one problem. But as it stands, the paper seems two-headed, with limited contributions, as each of the two problems is only dealt with shallowly.\n\nQUESTIONS:\n- Why limit yourself to a greedy approach? Given Algorithm 1, it is trivial to define a version that does a fixed amount of look-ahead (say, k steps) to find the optimal action considering the next k actions, rather than just the next (k=)1, as the current one does.\n- How come PO-UCT took so long to run for just 5000 steps? That is ~500 steps per second, which is extremely slow for what I would expect. Was a standard implementation used? \n- It also surprises me that PO-UCT underperforms MGPS, is there is something I do not understand. If PO-UCT is using the UCB bandit strategy, and there are, say, 500 possible actions at the root node (which are the 500 actions that MGPS will consider, and then select the best), during the first 500 iterations, PO-UCT will systematically select each of those 500, until each of them is expanded at least once, before moving on to the exploration/exploitation phase of UCB. Hence, at this point, shouldn't it be equivalent to MGPS? (they both have explored all actions), and then PO-UCT become better and better after this point? What am I missing?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors pose the problem of teaching decision-makers how to take a single action (picking a project from among a set of projects) based upon costly advice from experts across different, weighted criteria. The authors develop a reinforcement learning approach to creating a tutor that approximately solves the MDP (using a myopic approximation). The proposed tutor outperforms a baseline on a banking dataset. The tutor is also shown to improve decision-making of subjects in an online study, who get to learn from the tutor by watching the tutor make decisions.\n\nSTRENGTHS:\n+Section 3 is very clearly written. Well done (though Line 162 could have used O-notation).\n+The paper clearly presents the algorithm.\n+The paper is addressing an important problem of developing a tutoring system for solving MDPs\n+The paper has strong empirical results vs. PO-UCT.\n+The paper shows statistically significant results in a user-study. It is nice that a user-study was done.\n\nWEAKNESSES:\n-The terms h, e, \\lambda, N, and R_{total} are not sufficiently defined in Lines 41-50. Perhaps it would be better to give a more complete description later in the paper and abstract the presentation here to make it more intuitive just with words.\n-The comment on brainstorming in Line 86 ignores relevant literature on the wisdom of the crowd, the science behind brainstorming and focus groups, etc.\n-The statistical analysis does not report testing for the meeting of a Gaussian assumption for the confidence intervals. Details of the Box approximation should be provided. Further, it would have been better to also report p-values in that table. \n-For the user study, Table 1 should report how optimal (the RR) the MGPS tutor would be if run automatically (no human intervention) and how poorly a random, automatic system would be for the RR-score.\n-I am uncertain that the authors are reporting the degrees of the freedom of the F-test. The F-test has two degrees of freedom, but only 1 seems to be indicated in Lines 295-299.\n-I can appreciate that the authors might have thought that the results in lines 329-345 indicate that MGPS > PO-UCT and therefore not included PO-UCT as a baseline in the user study; however, that is a debatable decision. It could be that the behavior of PO-UCT is more intelligible by users, and users with PO-UCT could have outperformed those trained by MGPS. As such, I recommend the user study be re-ran (to account for cohort effects) with the PO-UCT baseline and randomizing the allocation of participants to the conditions.\n-The paper isn't exactly \"tutoring\" participants. Rather, the system is providing recommendations (or making decisions) and the users have to reverse engineering the actual strategy. Considering that the strategy itself is not constrained to be a set of if-then rules, it is unclear what exactly is being learned or how. It would make the paper better to have actually analyzed (by collecting the data from users) what users are learning and thinking. I would recommend looking at methods in explainable Artificial Intelligence.\n\nNote:\n-I recommend the \"Dummy\" tutor be called a random tutor -- a \"random tutor\" is a more clear description of that condition.\n\nQUESTIONS:\n-Arguably, the object-level decision is not \"really\" an MDP as there is no sequential decision-making. Could the authors comment on why the MDP formulation was adopted for this formulation rather than a multi-arm bandit formulation?\n\nLIMITATIONS:\nLimitations are reasonably discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problems of (1) sequential decision-making of information gathering through asking experts for information about the rewards for different projects (as meta-reasoning towards project selection), and (2) teaching people how to make near optimal decisions in the same problem through training in an intelligent tutoring system.  Specifically, novel problem aspects are considered compared to recent work on similar problems, including the availability of different experts of different reliabilities (instead of one information source) and multiple criteria for evaluating the quality/reward of a project.  An algorithm is developed for generating solutions guiding information gathering.  That solution is used to guide the training of humans through an Intelligent Tutoring System (ITS).  Experimental results highlight the benefits of the approach through both better training of humans in an ITS (measured through better rewards earned) and better performance in simulation of agent reasoning than the problem modeled as a belief MDP solved by PO-UCT.\n\nSTRENGTHS:\nThe primary strengths of this paper include:\n\nS1) The problem addressed is one that is relevant in the human-AI applications and it considers novelties not previously addressed that are important real-world complexities.  The problem of sequential decision-making will be of interest to the planning community at NeurIPs.\n\nS2) The paper is well written and easy to follow.\n\nS3) Modeling the problem as a POMDP is appropriate, and the MCDM component seems to be appropriately modeled.\n\nS4) I appreciated the use of two very different sets of experiments -- both training humans within an ITS and simulating the approach directly.  The ITS experiment was well designed and the evaluation was very carefully conducted and convincing.\n\nWEAKNESSES:\nThe primary weaknesses of this paper include:\n\nW1) The main contributions identified in the abstract are incremental, seemingly adding some environment complexity and solution adaption to [14], rather than being entirely novel.\n\nW2) While the choice of a POMDP was appropriate, I wasn't entirely sure why the authors relied more on the belief MDP formulation rather than a true POMDP with observations separate from belief state transitions.\n\nThe problem being solved is gathering information to ultimately choose the best task to accomplish.  This is very closely related to prior POMDP usage for problems like preference elicitation where the agent gathers information about which is the user's main preference or task they want the agent to perform.\n\nBoutilier,C. 2002.A POMDP Formulation of Preference Elicitation Problems.In Proceedings of AAAI'02, pp. 239–246.\n\nDoshi, F., & Roy, N. 2008. The Permutable POMDP: Fast Solutions to POMDPs for Preference Elicitation. In Proceedings of AAMAS'08, pp. 493–500.\n\nIn those models, the state space is the set of possible tasks/preferences, and actions either (1) query an information source (e.g., the user) for observations used to update the agent's Bayesian beliefs about the top preference/task, or (2) end information gathering to perform the perference/task the agent thinks is the top.  That is fundamentally the same as the problem being addressed here, but the details are slightly different.  Instead, your state space is belief states over the details of the tasks, from which a top one is selected.  It's not clear to me why the former wouldn't work in this situation and what the advantage is in your formulation, which would help strengthen the novelty of the work. (Note: what makes your paper different from [14] also makes it different from those works).\n\nInformation gathering in POMDPs in general also have special formulations, such as the \\rho-POMDP and equivalent POMDP-IR, and I would think your problem model would also fit nicely in the POMDP-IR, but neither is considered in your related work.\n\n\\rho-POMDP: Araya-Lopez, M., Buffet, O., Thomas, V., & Charpillet, F. 2010. A POMDP Extension with Belief-Dependent Rewards. In Proceedings of NIPS'10.\n\nPOMDP-IR: Spaan, M.T.J., Veiga, T.S., & Lima, P.U. Decision-theoretic planning under uncertainty with information rewards for active cooperative perception. Journal of Autonomous Agents and Multiagent Systems, 29(6):1157-1185.\n\nW3) I also wasn't sure why you chose PO-UCT as your baseline?  Its the simpler version of POMCP (whereas POMCP would have been more appropriate if you had explicit observations in your model).  Belief MDPs can also be considered continuous state MDPs, and there have been many advancements in Monte Carlo Tree Search planning in that area since PO-UCT, such as:\n\nSunberg, Z. & Kochenderfer, M. 2017. Online algorithms for POMDPs with continuous state, action, and observation spaces. arXiv, 2017. doi: 10.48550/ ARXIV.1709.06196. URL https://arxiv.org/ abs/1709.06196.\n\nFinally, the \\rho-POMDP has a MCTS solution that would also be relevant that is more recent:\n\nThomas, V., Hutin, G., & Buffet, O. 2020.. Monte information-oriented planning. In Proceedings ECAI’20.\n\nW4) I also didn't quite understand the novelty of the ITS experiment compared with [14].  Was the only difference the change in the meta-reasoning algorithm, or were there other differences?\n\nQUESTIONS:\nQ1) Why did you use your specific formulation of the POMDP problem instead of the state-of-the-art \\rho-POMDP or POMDP-IR for guiding information gathering?\n\nQ2) Why did you choose PO-UCT as your baseline?\n\n### Post-Rebuttal ###\n\nI thank the authors' for their rebuttal and the ensuing conversation.  They helped me better understand the research and its place in the literature.\n\nLIMITATIONS:\nThese were appropriate addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}