Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors formulate the optimal commitment problems (OCP), which can be used to model problems in which the agent determines whether to continue with some long-term actions like experiments until a signal of success or failure is received or to stop it in the middle. The agent is also allowed to change to a different (long-term) action during the process. On the theoretical side, the authors characterizes some fundamental properties of the optimal solution in a simplified OCP problem that has a single commitment. On the empirical side, the authors then propose an algorithm, Bayes-OCP that can be used to solve more general OCP problems. The authors also conduct  experiments to demonstrate the advantage of the proposed model and algorithm over existing models/algorithms in the literature.
strength_and_weaknesses: Strengths:
* The framework is well motivated and the applicability of the model to different domains like healthcare, finance and energy is clearly stated and demonstrated.
* The theoretical results of the special case OCP solution characterization are inspiring. The numerical results are also generally convincing and supportive. 
* The appendix also contains many interesting discussions, including the POMDP formulation of OCP, RL benchmark, the Bonferroni correction to improve FWER and the robustness to outcome model mis-specifications.

Weaknesses:
* If I understand correctly, it is assumed that the $R_{\psi}$ and $C_{\psi}$ coefficients in the utility function are known a priori for each experiment design $\psi$, which again limits the applicability and practicality of the proposed framework. Also some explanations are needed to characterize when such an assumption holds and when not.
* The figures (Figures 3 and 4) in the numerical experiments are not very easy to follow (the tables are good and clear, though). Some more explanations on them (especially how the switching points are determined, and how to associate each algorithm to each of the curves, etc.) would be very helpful.
* The discretized benchmark for solving POMDP formulation is not a very fair comparison. There are lots of more efficient POMDP algorithms (both with and without fully known models) that can be used (and many off-the-shelf solvers are available such as https://github.com/JuliaPOMDP/POMDPs.jl), and the authors should better compare with them in order to argue that the POMDP + RL approach does not work (or not). 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. I also don't have real concerns about the quality, novelty and reproducibility of this paper. However, there are also some weaknesses as detailed in the previous section of the review. And also I have the following questions and suggestions for the authors.
* Is it correct that throughout the paper, the authors are considering targeting a single/unique subpopulation within $\mathcal{X}$? Can this be generalized to multiple targeting subpopulations? This might be helpful to include A/B testing, which is a very important experiment design example, as a special case. Or can A/B testing already be modeled with a single targeting subpopulation?
* Why isn't the adaptive signature design mentioned in Table 2 compared in the experiments?
* When computing the average metrics (e.g., utility) over multiple instances, some normalization might be needed to make it a fairer comparison (since otherwise an instance with a very large metric/utility will dominate).
* As mentioned above, Figures 3 and 4 need more explanations and many details of the figures are not easy to understand. For example, in Figure 3, why is it the case that only greedy Bayes-OCP (but not the non-greedy Bayes-OCP) is compared in the top figure? Why is futility stopping not compared? Are the green, amber and red curves all for greedy Bayes-OCP? And is the top figure an amber instance and the bottom one a red instance? Just to give some example questions the readers may have so that the authors can explain in the paper to improve the readability. 
* Just to check, the main setting of this paper is online decision making, but the discretized RL algorithm in Appendix A needs to know the full model, right? This is not an issue but just want to clarify. 
* In Appendix B.1, it would be nice if the authors can provide a detailed algorithm frame for the Bonferroni correction variant for better reproducibility.


summary_of_the_review: Overall I think the paper contains non-trivial contribution in terms of both modeling, theory and practice. It's also clearly written and the reproducibility is also good given the detailed experiments. But the authors should also take a further look at the potential weaknesses detailed above and try to clarify and/or fix them.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the problem of online decision making, in which the decision maker decides whether to commit to long-term actions until success or failure, or stop it and/or switch to an alternative. The authors formulate it as an optimal commitment problem (OCP), which is a new type of optimal stopping/switching problem. For a simplified case of OCP with a single commitment, the authors provide theoretical analysis to obtain the characteristics of the optimal solution, and then propose a practical algorithm called Bayes-OCP to solve the general problems. Numerical experiments are also provided to validate the superiority of the proposed framework compared to related existing methods in the literature. 
strength_and_weaknesses: Strengths:
* The theoretical analysis of the simplified OCP with a single commitment is insightful, which not only motivates the general algorithm but also provides a better understanding of the OCP framework.
* The numerical experiments are relatively thorough and the discussion about green, amber and red instances are also inspiring. 

Weaknesses:
* The proposed algorithm, Bayes-OCP can only handle experiment designs with fixed time horizon $\tau$ and fixed success criterion $\rho$. The only decision variable becomes the targeting subpopulation $X\in\mathcal{X}$. This limits the applicability and practicality.
* The numerical experiments are conducted only on a synthetic/simulation environment with only two candidate targeting atomic populations. It would be more convincing if some experiments on more candidate targeting atomic populations and real-world datasets are included.


clarity,_quality,_novelty_and_reproducibility: In general, I find the paper relatively easy to follow. The results are also solid and cover many different facets ranging from theory to practice. The framework and algorithm are also new to my knowledge, and the experiment setup is also relatively clear for reproduction. These being said, there are several weaknesses as mentioned above. In addition, I have some questions and suggestions for the authors, as listed below.
* Is it true that in Bayes-OCP, we assume that we know $R_X$ and $C_X$ for every subpopulation $X$, but we don't know $\theta_X$? It would be helpful to more clearly state what are known and what are not.
* In line 15 of Bayes-OCP, should there be a reset of $t$ to $0$, or should $\mathcal{D}_0$ be $\mathcal{D}_t$ instead? Otherwise when going back to line 5 for the next loop, the assignment of $\mathcal{D}_0$ is meaningless.
* When comparing different algorithms, samples and runtime are not mentioned. These should also be discussed for fairer comparisons.
* In Table 3, it would be helpful to highlight other metrics (in addition to utility) as well, especially given that some metrics are the lower the better while the others are the higher the better if I understand correctly. 


summary_of_the_review: Overall, I find the paper is generally well-written and contains some good results both in theory and practice. But there are also several points for improvements as detailed above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces the _optimal commitment problem_, where an agent must decide, at each time step $t$, whether to continue running the current experiment (i.e., _commit_ to the current experiment), or terminate the current experiment, switching to a more promising experiment (or to no experiment at all, if one does not exist). The paper first analyzes the simple case, where the decision is just whether to continue or stop the current experiment. The paper establishes that the optimal policy, in this case, is a threshold policy (i.e., if the outcome of the current experiment is above some threshold continue, otherwise stop) that becomes increasingly close to a greedy policy as the experiment approaches its end. 

The paper then proposes Bayes-OCP, an algorithm that maintains a posterior distribution over the mean outcome for each possible atomic population $x\in\mathcal{X}$. Then, at each iteration, assesses whether switching from the current population $X\subset\mathcal{X}$ to an alternative population $X’\subset\mathcal{X}$ (including the “empty” population) is likely to lead to a higher utility. In that case, the agent stops the current experiment and starts a new experiment involving the population $X’$.

The results in the paper show that, when compared with alternative approaches, Bayes-OCP strikes a good balance between optimism (i.e., not stopping the experiments too soon) and timeliness (not stopping the experiments too late).
strength_and_weaknesses: The paper is somewhat outside my comfort area, so it is possible that I misjudge some of the contributions of the paper.

**Strengths:**

The paper is very well written, and a pleasure to read. It makes an excellent work of motivating and setting up the problem, the contributions are clearly exposed and presented in a constructive and intuitive manner. The experiments, although involving mostly simulation, do support the claims of the paper adequately. 

**Weaknesses:**

I have, perhaps, two main complaints regarding the paper. The first is that I would have liked to grasp more clearly the relation between the proposed work and other decision-theoretic problems. The paper properly provides the connection between the proposed framework and POMDPs, and also establishes early in the discussion that the OCP problem is, at its essence, one of exploration vs exploitation. This said, it would seem to me that approaches based on sequential prediction/multi-armed bandits (particularly using experts) could also be relevant alternatives to consider, but I found little discussion on this problem. Since $\mathcal{X}$ is finite, an approach that seeks to identify which $x\in\mathcal{X}$ to select that maximizes the utility would be a relevant competitor. I’m probably missing something here, but some discussion on this would certainly help to clarify my possible confusion.

My second complaint is more on form than on content. Personally, I found that laying out the tables and figures in the middle of the text to be disruptive while reading the paper. I realize that this is probably done for a matter of space, but perhaps moving some of the tables/plots to the appendix may help (for example, Figs. 3 and 4 can probably make it to the appendix with no significant loss in the exposition).
clarity,_quality,_novelty_and_reproducibility: **Clarity**

Overall, I believe that the paper is of high quality, very well written an well organized.

**Quality**

Although I haven’t checked the proofs in the appendices in detail, the results in the paper strike me as reasonable and the presentation as technically sound, in general.

**Novelty**

As mentioned earlier, the paper is somewhat outside my area of research, so I am not properly able to assess the originality of the work. However, from the discussion in the paper, the contributions appear novel and impactful.

**Reproducibility**

The paper provides a discussion on the reproducibility of the results in the paper. I believe that, in general, the results are reproducible.
summary_of_the_review: The paper is, in my opinion, very well written and provides a novel and interesting contribution. I would like to understand better the applicability of multi-armed bandit problems to the setting considered in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers a problem setup in which decision makers launch experiments (say over target population of agents) and have to wait a period of time before receiving the outcomes. These outcomes may or may not be conclusive, and accordingly the experiment launched may or may not be useful. If found inconclusive, it may be useful to relaunch the experiment with a new sample population. This paper studies the problem of determining when it is optimal to continue or break the commitment to a particular instance of an experiment. 

The paper formulates this as an “Optimal COmmitment Problem (OCP)” and provide some theoretical analysis. The paper also proposes a practical algorithm for solving the OCP problem and evaluates it empirically in the context of clinical trials. 

strength_and_weaknesses: Strengths

— Well motivated: The paper describes application of this problem formulation to multiple domains and motivates the applicability of these techniques well.

– Well written: The paper is well written overall and is easy to follow. Inclusion of warm-up section for example is useful in building understanding. 

– New formulation: The OCP problem formulation seems novel to the best of my knowledge.

– Good results: The proposed methodology and algorithm is shown to perform well.

Weaknesses

– Theoretical guarantees on algorithm performance would be valuable

– Related work: There is some related work on queuing theory and optimization in cloud computing (completely different application domain) such as “Efficient Straggler Replication in Large-scale Parallel Computing” by Wang et al. which focuses on the problem of when to kill existing tasks and relaunch. It may be interesting to discuss if these techniques are relevant or if there are fundamental shortcomings that prohibit their use for the OCP problem.

clarity,_quality,_novelty_and_reproducibility: Paper is clearly written, results seem high quality and the problem being considered is new as far as I know. 
summary_of_the_review: New problem domain, well-motivated, first results in terms of theory and algorithm.

Good paper overall
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the setting of optimally breaking commitments. In particular, it considers a decision making problem in which a decision maker has to decide on whether to break a commitment and if so, whether to select an alternative commitment or stop the making commitments. The paper formally characterizes a simplified version of the problem, which leads to three key insights that are later on used to design a novel algorithm for a more general setting. This novel algorithm is validated in a simulation-based test-bed. 
strength_and_weaknesses: **Strengths**:
- Arguably, the main strength of this paper is the novelty of the paper is the problem setting. To my knowledge, this particular setting has not been studies in the literature.  
- The characterization results on a simpler instance of the problem setting are intuitive, but provide insights that one can use to design effective algorithms. These results also inspired the design of the propose algorithm for optimally deciding on commitments. 
---
**Weaknesses**: 
- The more complex, and presumably more interesting setting is not formally analyzed. This makes the set of characterization results somewhat incomplete. 
- The algorithm assumes that the outcomes of interest come from a specific distribution, which does not affect the soundness of the algorithm, but may affect its performance. 
- The proposed algorithm doesn't seem to have any provable guarantee that it converges to an optimal solution, and it is not clear how it would fare against e.g. policies learned via RL with function approximation. The paper only reported results with relatively simple RL baseline.  
- Benchmarks appear to be rather simple with only two atomic population and carefully chosen parameters. Although the experimental results indicate that the proposed approach is more effective than the baselines, more extensive experimentation would be quite useful. For example, the experimental setup where the distributional assumption on the outcome could fail to hold would be informative about the robustness of the proposed approach. 
clarity,_quality,_novelty_and_reproducibility: *Clarity & Quality:*
The paper is clearly written and easy to follow. The set of results represent a good starting point to exploring this problem. At the same time, the experimental results could be expanded, while the analysis could focus on more general, less constrained settings. E.g., are there other instances of the problem setting for which one can obtain similar results? If so, those results could be informative. 
---
*Novelty & Reproducibility:*
The problem setting appears to be novel and, in my opinion, this is the main strength of the paper. The results look reproducible: the paper documents well the experimental testbed, while the appendix includes proofs of the claims from the main text. 
summary_of_the_review: While I believe that this paper studies an interesting topic, and provides an interesting set of initial results, the current results look preliminary at the first glance and could be further extended. More concretely, the theoretical analysis focuses on a simpler instance of the problems setting, while the experiments are based on simple benchmarks. In my opinion, the paper could greatly benefit from adding experiments that tests if the proposed algorithm is brittle to the deviation from the distributional assumption it relies on, and potentially adding more sophisticated baselines.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks at some optimal stopping problem where the objective is to stop a task and start a new one if the former is not "successful" (as starting new task has some cost).

If there is one task, the problem is not that difficult, but it gives insight to the cases of multiple tasks, where the analysis is more empirical than theoretical, which is a slight concern (I believe the latter is feasible and would provide more insights than the mere empirical one).

Anyway, the reviewers are rather positive, and I find this paper interesting as well (the fact of having non-standard repeated problems, i.e., interesting new potential variants of bandits is quite nice). Hence I recommend acceptance
justification_for_why_not_higher_score: Results are a bit contrived
justification_for_why_not_lower_score: The weird setting of bandits/online learning is nice (though the literature survey misses many relevant references)
Decision: Accept: poster