## 1 Introduction  
There is growing interest in using online reinforcement learning (RL) to optimize the delivery of messages or other forms of prompts in digital interventions. In mobile health, RL algorithms have been used to increase the effectiveness of the content and timing of intervention messages designed to promote physical activity [1, 2] and to manage weight loss [3]. In other areas, including the social sciences and education, RL algorithms are used to provide pretrial nudges to encourage court hearing attendance [4], to personalize math explanations [5], and to deliver quiz questions during lecture videos [6]. Unlike games and work in some areas of robotics, digital intervention studies can be extremely costly to run. Furthermore, when the study is a preregistered clinical trial, once initiated, the trial protocol (including any online algorithms) cannot be altered without jeopardizing trial validity. Thus, design decisions are a \"one-way door\" [7]; once we commit to a set of design decisions, they are irreversible for the duration of the trial. To prevent poor design decisions that could be detrimental to the effectiveness and the validity of study results, RL algorithms must undergo a thorough design and testing process before deployment.  
The development of an RL algorithm for digital interventions requires a multitude of design decisions. These decisions include how best to accommodate the lack of mechanistic models for dynamic human responses to digital interventions and how to ensure the robustness of the algorithm to potentially nonstationary/non-Markovian outcome distributions. Furthermore, one must ensure not only that the RL algorithm learns and quickly optimizes interventions but also that the algorithm runs stably and autonomously online within constrained amounts of time. One must also ensure that the RL algorithm can obtain data in a timely manner. Time and budgetary considerations may restrict the complexity of the RL algorithm that can be implemented. Furthermore, it is important to ensure the data collected by the RL algorithm can be used to inform future studies and address scientific, causal inference questions. Addressing these challenges in a reproducible, replicable manner is critical if RL algorithms are to play a role in optimizing digital interventions. Therefore, we need a framework for making design decisions for RL algorithms intended to optimize digital interventions.  
The primary contributions of this work are twofold:  
1. **Framework for Guiding Design Decisions in RL Algorithms for Digital Interventions:**_We provide a framework for evaluating the design of an online RL algorithm to increase confidence that the RL algorithm will improve the digital intervention's effectiveness in real-life implementation and maintain the intervention's reproducibility and replicability._ Specifically, we extend the PCS (predictability, computability, stability) data science framework of [PERSON] (2018) to address specific challenges in the development and evaluation of online RL algorithms for personalizing digital interventions.
2. **Case Study:** This case study concerns the development of an RL algorithm for Oralyttics, a mobile health intervention study designed to encourage oral self-care behaviors. The study is planned to go into the field in late 2022. _This case study provides a concrete example of implementing the PCS framework to inform the design of an online RL algorithm._