Abstract: As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about-summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts  [63]  and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles  [22] , producing summaries nearly as good as the human reference without any news-specific fine-tuning.  2  We conduct extensive analyses to understand our human feedback dataset and fine-tuned models.  3  We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want. * This was a joint project of the OpenAI Reflection team. Author order was randomized amongst {LO, JW, DZ, NS}; CV and RL were full-time contributors for most of the duration. PC is the team lead. 2 Samples from all of our models can be viewed on our website.  3  We provide inference code for our 1.3B models and baselines, as well as a model card and our human feedback dataset with over 64k summary comparisons, here.

Nisan Stiennon
Long Ouyang
Jeff Wu
Daniel M Ziegler
Ryan Lowe
Chelsea Voss
Alec Radford
Dario Amodei
Paul Christiano
Openai

Learning to summarize from human feedback

Introduction Large-scale language model pretraining has become increasingly prevalent for achieving high performance on a variety of natural language processing (NLP) tasks. When applying these models to a specific task, they are usually fine-tuned using supervised learning, often to maximize the log probability of a set of human demonstrations. While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective-maximizing the likelihood of human-written text-and what we care about-generating high-quality outputs as determined by humans. This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g. making up facts  [41] ) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models
 Supervised learning
 Human feedback
 Pretrain only
 Reference summaries Figure  1 : Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset.  4  Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in Appendix F; length differences explain about a third of the gap between feedback and supervised learning at 6.7B. are incentivized to place probability mass on all human demonstrations, including those that are low-quality; and distributional shift during sampling can degrade performance  [56, 52] . Quality can often be improved significantly by non-uniform sampling strategies such as beam search  [51] , but these can lead to repetition and other undesirable artifacts  [69, 23] . Optimizing for quality may be a principled approach to overcoming these problems. Our goal in this paper is to advance methods for training language models on objectives that more closely capture the behavior we care about. To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community  [16, 8, 54, 59, 50] , and is a subjective task where we believe it is difficult to quantify summary quality without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE  [39] , have received criticism for poor correlation with human judgments  [55, 45, 6, 33] . We follow the works of  [3, 73] , who fine-tune language models from human feedback using reward learning  [35] . We first collect a dataset of human preferences between pairs of summaries, then train a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each 'time step', and is updated using the PPO algorithm  [58]  based on the RM 'reward' given to the entire generated summary. We can then gather more human data using samples from the resulting policy, and repeat the process. We follow the works of  [48, 4]  and use large pretrained GPT-3 models with as many as 6.7 billion parameters. Our main contributions are four-fold. (1) We show that training with human feedback significantly outperforms very strong baselines on English summarization. When applying our methods on a version of the Reddit TL;DR dataset  [63] , we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure  1 ). (2) We show human feedback models generalize much better to new domains than supervised models. Our Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any news-specific fine-tuning, almost matching the quality of the dataset's reference summaries. We perform several checks to ensure that these human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section C.2), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices F and G.7). (3) We conduct extensive empirical analyses of our policy and reward model. We examine the impact of model and data size (Figure  6 ), study performance as we continue to optimize a given reward model (Section 4.3), and analyze reward model performance using synthetic and humanwritten perturbations of summaries (Section 4.3). We confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences, and that optimizing our reward model directly results in better summaries than optimizing ROUGE according to humans (Section 4.4). (4) We publicly release our human feedback dataset for further research. The dataset contains 64,832 summary comparisons on the TL;DR dataset, as well as our evaluation data on both TL;DR (comparisons and Likert scores) and CNN/DM (Likert scores). The methods we present in this paper are motivated in part by longer-term concerns about the misalignment of AI systems with what humans want them to do. When misaligned summarization models make up facts, their mistakes are fairly low-risk and easy to spot. However, as AI systems become more powerful and are given increasingly important tasks, the mistakes they make will likely become more subtle and safety-critical, making this an important area for further research.
 Related work Most directly related to our work is previous work using human feedback to train summarization models with RL  [3, 73] . Bohm et al.  [3]  learn a reward function from a dataset of human ratings of 2.5k CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing ROUGE. Our work is most similar to  [73] , who also train Transformer models  [62]  to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets. Unlike us, they train in an online manner and find the model highly extractive. They note that their labelers prefer extractive summaries and have low agreement rates with researchers. Compared to  [73] , we use significantly larger models, move to the batch setting for collecting human feedback, ensure high labeler-researcher agreement, and make some algorithmic modifications, such as separating the policy and value networks. Human feedback has also been used as a reward to train models in other domains such as dialogue  [25, 68, 21] , translation  [32, 1] , semantic parsing  [34] , story generation  [72] , review generation  [7] , and evidence extraction  [46] . Our reward modeling approach was developed in prior work on learning to rank  [40] , which has been applied to ranking search results using either explicit feedback  [2, 18]  or implicit feedback in the form of click-through data  [29, 30] . In a related line of research, human feedback has been used to train agents in simulated environments  [10, 24] . There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization  [50, 65, 45, 15, 19] , BLEU for translation  [50, 66, 1, 43] , and other domains  [61, 27, 26] . Finally, there has been extensive research on modifying architectures  [22, 59]  and pre-training procedures  [70, 36, 49, 60, 53, 14]  for improving summarization performance. 3 Method and experiment details
 High-level methodology Our approach is similar to the one outlined in  [73] , adapted to the batch setting. We start with an initial policy that is fine-tuned via supervised learning on the desired dataset (in our case, the Reddit TL;DR summarization dataset). The process (illustrated in Figure  2 ) then consists of three steps that can be repeated iteratively. Step 1: Collect samples from existing policies and send comparisons to humans. For each Reddit post, we sample summaries from several sources including the current policy, initial policy, original reference summaries and various baselines. We send a batch of pairs of summaries to our human evaluators, who are tasked with selecting the best summary of a given Reddit post. Step 2: Learn a reward model from human comparisons. Given a post and a candidate summary, we train a reward model to predict the log odds that this summary is the better one, as judged by our labelers. Step 3: Optimize a policy against the reward model. We treat the logit output of the reward model as a reward that we optimize using reinforcement learning, specifically with the PPO algorithm  [58] .
 Collect human feedback "j is better than k" "j is better than k" A Reddit post is sampled from the Reddit TL;DR dataset. Various policies are used to sample a set of summaries. Two summaries are selected for evaluation. A human judges which is a better summary of the post.
 Train reward model One post with two summaries judged by a human are fed to the reward model. The reward model calculates a reward r for each summary. The loss is calculated based on the rewards and human label, and is used to update the reward model.
 Train policy with PPO A new post is sampled from the dataset. The reward model calculates a reward for the summary. The reward is used to update the policy via PPO. We provide a more thorough description of our procedure, including details of the reward model and policy training and our quality control process, in the following sections. In practice, rather than precisely iterating this sequence of three steps, we updated our data collection and training procedures over the course of the project while accumulating labels (see Appendix C.6 for details).
 Datasets and task Datasets. We use the TL;DR summarization dataset  [63] , which contains ~3 million posts from reddit.com across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs). We additionally filter this dataset (see Appendix A) to ensure quality, including using a whitelist of subreddits that are understandable to the general population. Crucially, we also filter to include only posts where the human-written summaries contain between 24 and 48 tokens, to minimize the potential effect of summary length on quality (see Section 4.1 and Appendix F). Our final filtered dataset contains 123,169 posts, and we hold out ~5% as a validation set. For the remainder of this paper, we refer to this dataset simply as TL;DR. We chose the TL;DR dataset over the more commonly used CNN/DM dataset primarily because very strong performance can be attained on CNN/DM with simple extractive baselines. We find in Section 4.2 that our labelers prefer lead-3 over the CNN/DM reference summaries,  5  and that the supervised T5 model  [49]  with low-temperature sampling already surpasses the reference summary quality, while copying extensively from the article. On the other hand, simple extractive baselines perform poorly on TL;DR in our human evaluations (see Appendix G.2). Instead of training on CNN/DM, we study the transfer performance of our human feedback models to CNN/DM after being trained to summarize Reddit posts. Task. We define our ground-truth task as producing a model that generates summaries fewer than 48 tokens long that are as good as possible, according to our judgments. We judge summary quality by how faithfully the summary conveys the original post to a reader who can only read the summary and not the post (see Appendix C.5 for further discussion of criteria). Since we have limited capacity to do comparisons, we hire labelers to do the comparisons for us. We rely on detailed procedures to ensure high agreement between labelers and us on the task, which we describe in the next section. [r/dating_advice] First date ever, going to the beach. Would like some tips Hey Reddit! I (20M) would like some tips, because I have my first ever date tomorrow (although I've had a gf for 3 years, but no actual dating happened), and we're going to the beach. I met this girl, we have mutual friends, at a festival a few days ago. We didn't kiss, but we talked, held hands, danced a bit. I asked her to go on a date with me, which was super hard as it is the first time I've asked this to anybody. What I mean to say is, it's not like a standard *first* date because we already spent some time together. I'm really nervous and excited. I'm going to pick her up tomorrow, we're cycling to the beach which will take  30     1 : Example of post and samples on the TL;DR dataset, chosen to be particularly short. For random samples (along with posts), see Appendix H and our website.
 Collecting human feedback Previous work on fine-tuning language models from human feedback  [73]  reported "a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated", leading to model-generated summaries that were high-quality according to the labelers, but fairly low-quality according to the researchers. Compared to  [73] , we implement two changes to improve human data quality. First, we transition entirely to the offline setting, where we alternate between sending large batches of comparison data  6  to our human labelers and re-training our models on the cumulative collected data. Second, we maintain a hands-on relationship with labelers:  7  we on-board them with detailed instructions, answer their questions in a shared chat room, and provide regular feedback on their performance. We train all labelers to ensure high agreement with our judgments, and continuously monitor labeler-researcher agreement over the course of the project. See Appendix C.1 and C.5 for details. As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of comparison tasks, labelers agree with researchers 77% ± 2% of the time, while researchers agree with each other 73% ± 4% of the time. We provide more analysis of our human data quality in Appendix C.2.
 Models All of our models are Transformer decoders  [62]  in the style of GPT-3  [47, 4] . We conduct our human feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters. Pretrained models. Similarly to  [12, 47] , we start with models pretrained to autoregressively predict the next token in a large text corpus. As in  [48, 4] , we use these models as 'zero-shot' baselines by padding the context with examples of high-quality summaries from the dataset. We provide details on pretraining in Appendix B, and on our zero-shot procedure in Appendix B.2. Supervised baselines. We next fine-tune these models via supervised learning to predict summaries from our filtered TL;DR dataset (see Appendix B for details). We use these supervised models to sample initial summaries for collecting comparisons, to initialize our policy and reward models, and as baselines for evaluation. In our final human evaluations, we use T=0 to sample from all models, as we found it performed better than higher temperatures or nucleus sampling (see Appendix B.1). To validate that our supervised models are indeed strong baselines for comparison, we run our supervised fine-tuning procedure with our 6.7B model on the CNN/DM dataset, and find that we achieve slightly better ROUGE scores than SOTA models  [71]  from mid-2019 (see Appendix G.4). Reward models. To train our reward models, we start from a supervised baseline, as described above, then add a randomly initialized linear head that outputs a scalar value. We train this model to predict which summary y ∈ {y 0 , y 1 } is better as judged by a human, given a post x. If the summary preferred by the human is y i , we can write the RM loss as: loss(r θ ) = E (x,y0,y1,i)∼D [log(σ(r θ (x, y i ) − r θ (x, y 1−i )))] where r θ (x, y) is the scalar output of the reward model for post x and summary y with parameters θ, and D is the dataset of human judgments. At the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0. Human feedback policies. We want to use the reward model trained above to train a policy that generates higher-quality outputs as judged by humans. We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm  [58] , where each time step is a BPE token.  8  We initialize our policy to be the model fine-tuned on Reddit TL;DR. Importantly, we include a term in the reward that penalizes the KL divergence between the learned RL policy π RL φ with parameters φ and this original supervised model π SFT , as previously done in  [25] . The full reward R can be written as: R(x, y) = r θ (x, y) − β log[π RL φ (y|x)/π SFT (y|x)] This KL term serves two purposes. First, it acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode. Second, it ensures the policy doesn't learn to produce outputs that are too different from those that the reward model has seen during training. For the PPO value function, we use a Transformer with completely separate parameters from the policy. This prevents updates to the value function from partially destroying the pretrained policy early in training (see ablation in Appendix G.1). We initialize the value function to the parameters of the reward model. In our experiments, the reward model, policy, and value function are the same size.
 Results
 Summarizing Reddit posts from human feedback Policies trained with human feedback are preferred to much larger supervised policies. Our main results evaluating our human feedback policies on TL;DR are shown in Figure  1 . We measure policy quality as the percentage of summaries generated by that policy that humans prefer over the reference summaries in the dataset. Our policies trained with human feedback significantly outperform our supervised baselines on this metric, with our 1.3B human feedback model significantly outperforming a supervised model 10× its size (61% versus 43% raw preference score against reference summaries). Our 6.7B model in turn significantly outperforms our 1.3B model, suggesting that training with human feedback also benefits from scale. Additionally, both of our human feedback models are judged by humans to be superior to the human demonstrations used in the dataset. Controlling for summary length. When judging summary quality, summary length is a confounding factor. The target length of a summary is implicitly part of the summarization task; depending on the desired trade-off between conciseness and coverage, a shorter or longer summary might be better. Since our models learned to generate longer summaries, length could account for much of our quality improvements. We find that after controlling for length (Appendix F), the preference of our human feedback models vs. reference summaries drops by ~5%; even so, our 6.7B model summaries are still preferred to the reference summaries ~65% of the time. How do our policies improve over the baselines? To better understand the quality of our models' summaries compared to the reference summaries and those of our supervised baselines, we conduct an additional analysis where human labelers assess summary quality across four dimensions (or "axes") using a 7-point Likert scale  [38] . Labelers rated summaries for coverage (how much important information from the original post is covered), accuracy (to what degree the statements in the summary are stated in the post), coherence (how easy the summary is to read on its own), and overall quality.  The results (Figure  3 ) indicate that our human feedback models outperform the supervised baselines across every dimension of quality, but particularly coverage. Although our human labelers had a high bar for giving perfect overall scores, summaries from our 6.7B PPO model achieve a 7/7 overall score 45% of the time (compared to 20% and 23% for the 6.7B supervised baseline and reference summaries, respectively).
 Transfer to summarizing news articles Our human feedback models can also generate excellent summaries of CNN/DM news articles without any further training (Figure  4 ). Our human feedback models significantly outperform models trained via supervised learning on TL;DR and models trained only on pretraining corpora. In fact, our 6.7B human feedback model performs almost as well as a 6.7B model that was fine-tuned on the CNN/DM reference summaries, despite generating much shorter summaries. Since our human feedback models transferred to CNN/DM have little overlap in summary length distribution with models trained on CNN/DM, with about half as many tokens on average, they are difficult to compare directly. Thus our evaluations in Figure  4  use a 7-point Likert scale on four quality dimensions, as in Section 4.1 (see Appendix C.5 for labeler instructions). In Figure  4b  we show the average overall score at different summary lengths, which suggests our human feedback models would perform even better if they generated longer summaries. Qualitatively, CNN/DM summaries from our human feedback models are consistently fluent and reasonable representations of the article; we show examples on our website and in Appendix H.
 Understanding the reward model What happens as we optimize the reward model? Optimizing against our reward model is supposed to make our policy align with human preferences. But the reward model isn't a perfect representation of our labeler preferences, as it has limited capacity and only sees a small amount of comparison data from a relatively narrow distribution of summaries. While we can hope our reward model generalizes to summaries unseen during training, it's unclear how much one can optimize against the reward model until it starts giving useless evaluations. To answer this question, we created a range of policies optimized against an earlier version of our reward model, with varying degrees of optimization strength, and asked labelers to compare samples from them to the reference summaries. Figure  5  shows the results for PPO at a range of KL penalty coefficients (β). Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences. Though this is clearly undesirable, we note that this over-optimization also happens with ROUGE (see  [45]  and Appendix G.3). Similar behavior has been observed in learned reward functions in the robotics domain  [5] . How does reward modeling scale with increasing model and data size? We conduct an ablation to determine how data quantity and model size affect reward modeling performance. We train 7 reward models ranging from 160M to 13B parameters, on 8k to 64k human comparisons from our dataset. We find that doubling the training data amount leads to a ~1.1% increase in the reward model validation set accuracy, whereas doubling the model size leads to a ~1.8% increase (Figure  6 ). What has the reward model learned? We probe our reward model by evaluating it on several validation sets. We show the full results in Appendix G.6, and highlight them here. We find that our reward models generalize to evaluating CNN/DM summaries (Appendix G.7), agreeing with labeler preferences 62.4% and 66.5% of the time (for our 1.3B and 6.7B models, respectively). Our 6.7B reward model nearly matches the inter-labeler agreement value of 66.9%. We also find that our reward models are sensitive to small but semantically important details in the summary. We construct an additional validation set by having labelers make minimal edits to summaries to improve them. Our RMs prefer the edited summaries almost as often (79.4% for 1.3B and 82.8% for 6.7B) as a separate set of human evaluators (84.1%). Further, when comparing the reference summaries to perturbed summaries where the participants' roles are reversed, our models reliably select the original summary (92.9% of the time for 1.3B, 97.2% for 6.7B). However, our RMs are biased towards longer summaries: our 6.7B RM prefers improving edits that make the summary shorter only 62.6% of the time (vs. 76.4% for humans).
 Analyzing automatic metrics for summarization Evaluation. We study how well various automatic metrics act as predictors for human preferences, and compare them to our RMs. Specifically, we examine ROUGE, summary length, amount of copying from the post,  9  and log probability under our baseline supervised models. We present a full matrix of agreement rates between these metrics in Appendix G.7. We find that our learned reward models consistently outperform other metrics, even on the CNN/DM dataset on which it was never trained. We also find that ROUGE fails to track sample quality as our Figure  7 : Summary quality as a function of metric optimized and amount of optimization, using best-of-N rejection sampling. We evaluate ROUGE, our main reward models, and an earlier iteration of the 1.3B model trained on approximately 75% as much data (see Table  11  for details). ROUGE appears to peak both sooner and at a substantially lower preference rate than all reward models. Details in Appendix G.3. models improve. While ROUGE has ~57% agreement with labelers when comparing samples from our supervised baseline models, this drops to ~50% for samples from our human feedback model. Similarly, log probability agreement with humans drops to ≤50% on comparisons between samples from our human feedback models, while our RMs still perform above chance (62%). Scaling up the size of the supervised model does not reliably improve log probability's agreement with labelers. Optimization. In Figure  7 , we show that optimizing ROUGE using a simple optimization scheme doesn't consistently increase quality, as has been noted in  [45] . Optimization against ROUGE peaks both sooner and at a substantially lower quality rate than optimization against our reward models.
 Discussion Limitations. One limitation of our work is the time and cost required to produce our final models. Notably, fine-tuning our 6.7B model with RL required approximately 320 GPU-days. Our data collection procedure is also expensive compared to prior work -the training set took thousands of labeler hours and required significant researcher time to ensure quality. For this reason, we were unable to collect baselines such as an equivalent amount of high-quality human demonstrations for supervised baselines. See D for more discussion. We leave this ablation to future work. Nevertheless, we believe reward modeling is more likely to scale to tasks where it is extremely skill-intensive or time-consuming to provide good demonstrations. Future directions. The methods in this paper could be applied to any task where humans can compare samples, including dialogue, machine translation, question answering, speech synthesis, and music generation. We expect this method to be particularly important for generating long samples, where the distributional shift and degeneracy of maximum likelihood samples can be problematic. It may be possible to improve sample efficiency by training to predict feedback across many tasks  [42] . We are particularly interested in scaling human feedback to tasks where humans can't easily evaluate the quality of model outputs. In this setting, it is particularly challenging to identify whether an ML system is aligned with the human designer's intentions. One approach is to train ML systems to help humans perform the evaluation task quickly and accurately  [9] . There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models  [28, 17, 44, 64] . For example, we could solicit high-quality demonstrations from labelers, have labelers edit model outputs to make them better, or have labelers provide explanations for why they preferred one model output over another. All of this feedback could be leveraged as a signal to train more capable reward models and policies. Broader impacts. The techniques we explore in this paper are generic techniques that could be used in a wide variety of machine learning applications, for any task where it is feasible for humans to evaluate the quality of model outputs. Thus, the potential implications are quite broad. Our research is primarily motivated by the potential positive effects of aligning machine learning algorithms with the designer's preferences. Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait  [11] . In the short term, improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human well-being. In the long term, as machine learning systems become more capable it will likely become increasingly difficult to ensure that they are behaving safely: the mistakes they make might be more difficult to spot, and the consequences will be more severe. For instance, writing an inaccurate summary of a news article is both easy to notice (one simply has to read the original article) and has fairly low consequences. On the other hand, imitating human driving may be substantially less safe than driving to optimize human preferences. We believe that the techniques we explore in this paper are promising steps towards mitigating the risks from such capable systems, and better aligning them with what humans care about. Unfortunately, our techniques also enable malicious actors to more easily train models that cause societal harm. For instance, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs, or to induce dependence of humans on the technology, or to generate large amounts of toxic or hurtful content intended to harm specific individuals. Avoiding these outcomes is a significant challenge for which there are few obvious solutions. Large-scale models trained with human feedback could have significant impacts on many groups. Thus, it is important to be careful about how we define the 'good' model behavior that human labelers will reinforce. Deciding what makes a good summary is fairly straightforward, but doing this for tasks with more complex objectives, where different humans might disagree on the correct model behavior, will require significant care. In these cases, it is likely not appropriate to use researcher labels as the 'gold standard'; rather, individuals from groups impacted by the technology should be included in the process to define 'good' behavior, and hired as labelers to reinforce this behavior in the model. We chose to train on the Reddit TL;DR dataset because the summarization task is significantly more challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with minimal moderation, they often contain content that is offensive or reflects harmful social biases. This means our models can generate biased or offensive summaries, as they have been trained to summarize such content. For this reason, we recommend that the potential harms of our models be thoroughly studied before deploying them in user-facing applications. Finally, by improving the ability of machine learning algorithms to perform tasks that were previously only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially leading to significant job loss. Without suitable policies targeted at mitigating the effects of large-scale unemployment, this could also lead to significant societal harm.  Figure 2 : 2 Figure 2: Diagram of our human feedback, reward model training, and policy training procedure.
 Figure 4 : 4 Figure 4: Transfer results on CNN/DM. (a) Overall summary quality on CNN/DM as a function of model size. Full results across axes shown in Appendix G.2. (b) Overall scores vs. length for the 6.7B TL;DR supervised baseline, the 6.7B TL;DR human feedback model, and T5 fine-tuned on CNN/DM summaries. At similar summary lengths, our 6.7B TL;DR human feedback model nearly matches T5 despite never being trained to summarize news articles.
 Figure 3 : 3 Figure 3: Evaluations of four axes of summary quality on the TL;DR dataset.
 Table minutes, and then what? I'm a bit scared. Should I bring something (the weather, although no rain and sunny, is not super so no swimming), should we do something. I'd like all the tips I can get. Thanks! Human written reference TL;DR 6.7B supervised model 6.7B human feedback model First date after 3 years in a relation- Going on a date with a girl I met Going on my first ever date tomor- ship, going to the beach, terrified. a few days ago, going to the beach. row, cycling to the beach. Would What to bring with me, what to do? What should I bring, what should like some tips on what to do and we do? bring. I'm a bit nervous and excited. Thanks!
 Uifalean, Morris Stuttard, Russell Bernandez, Tasmai Dave, Rachel Wallace, Jenny Fletcher, Jian Ouyang, Justin Dill, Maria Orzek, Megan Niffenegger, William Sells, Emily Mariner, Andrew Seely, Lychelle Ignacio, Jelena Ostojic, Nhan Tran, Purev Batdelgar, Valentina Kezic, Michelle Wilkerson, Kelly Guerrero, Heather Scott, Sarah Mulligan, Gabriel Ricafrente, Kara Bell, Gabriel Perez, and Alfred Lee.
			 Throughout the paper, error bars represent 1 standard error.
			 We manually check this result in Appendix E and find we generally agree with labeler ratings.
			 Our decision to collect comparisons rather than Likert scores is supported by recent work, e.g. [37] . 7  We recruited labelers from a freelancing platform, Upwork, and two labeling services, Scale and Lionbridge.
			 Note that the reward model only gives rewards for entire summaries, and not at intermediate time steps. In RL terminology, each episode terminates when the policy outputs the EOS token, and the discount factor γ = 1.
			 We measure copying by computing the longest common subsequence of bigrams with the original Reddit post or news article, and dividing by the number of bigrams in the summary.


Abstract:
When we are faced with a choice among acts, but are uncertain about the true state of the world, we may be uncertain about the acts' "choiceworthiness". Decision theories guide our choice by making normative claims about how we should respond to this uncertainty. If we are unsure which decision theory is correct, however, we may remain unsure of what we ought to do. Given this decision-theoretic uncertainty, metatheories attempt to resolve the conflicts between our decision theories…but we may be unsure which meta-theory is correct as well. This reasoning can launch a regress of ever-higher-order uncertainty, which may leave one forever uncertain about what one ought to do. There is, fortunately, a class of circumstances under which this regress is not a problem. If one holds a cardinal understanding of subjective choiceworthiness, and accepts certain other criteria (which are too weak to specify any particular decision theory), one's hierarchy of metanormative uncertainty ultimately converges to precise definitions of "subjective choiceworthiness" for any finite set of acts. If one allows the metanormative regress to extend to the transfinite ordinals, the convergence criteria can be weakened further. Finally, the structure of these results applies straightforwardly not just to decision-theoretic uncertainty, but also to other varieties of normative uncertainty, such as moral uncertainty.
 Keywords Decision theory • Moral uncertainty • Normative uncertainty • Uncertaintism My thanks to Anubav Vasudevan, Christian Tarsney, and two anonymous referees for helpful comments and suggestions. All errors are my own.

Fixed-point solutions to the regress problem in normative uncertainty

Introduction People sometimes make claims about how we ought to act in the face of empirical uncertainty. A "decision theory" is a collection of such claims. Because they make demands on our behavior, decision theories are "norms". (Descriptive claims about how actual agents act under uncertainty are sometimes called "positive decision theories". For the purposes of this paper, the term "decision theory" will refer exclusively to normative decision theories, as defined above.) Moral theories are also norms, for example, because they too are collections of claims about how we "ought" (albeit in another sense) to act. Though I hope that the formal results of this paper are of interest to readers of many metaethical persuasions, the language used throughout will assume a realist position. I will say, then, that we suffer decision-theoretic uncertainty when we assign positive probability to the truth-values of conflicting decision theories, and, more generally, some form of normative uncertainty whenever we assign positive probability to the truth-values of conflicting norms. The most widely accepted normative decision theory, by far, is expected utility theory. As a normative theory, expected utility theory can be interpreted as the claim that we have a cardinal utility function whose value depends on what act we choose and on the state of the world, and that we ought to act so as to maximize the expected value of that function, given our uncertainty across states. Alternatively, expected utility theory can be interpreted as the claim that we have an ordinal utility function whose value depends on what act we choose and on the state of the world, and that we ought to act in such a way as satisfies various assumptions (the von Neumann-Morgenstern axioms, perhaps), which together happen to entail that we will be acting as if we were maximizing the expected value of a cardinal utility function. Either way, expected utility theory, as a normative ideal, is a set of claims about what we ought to do. Expected utility theory is not without critics. Soon after its foundations were laid in the 1940s and 1950s by John von Neumann, Oskar Morgenstern, and Leonard Savage, others, such as Maurice Allais, raised objections to the claim that expected utility maximization is a model of ideal behavior. It is true that, in the apparent absence of plausible alternatives, expected utility theory came to serve as the unchallenged basis for almost all of economic theory. But the debate over the normative claims that underlie it continued among some economists and philosophers, and has recently received new attention with Lara Buchak's 2013 publication of Risk and Rationality, which argues that it can be rational to violate the von Neumann-Morgenstern "independence" axiom so as to act on explicit risk preferences. In sum, the normative claims made by expected utility theory-even if sometimes taken for granted-are claims about whose truth-value an agent can be uncertain. What "should" one do in the face of decision-theoretic uncertainty? Is there even a coherent way to interpret that question? The position that there is-that we "ought" to act on the basis of our normative uncertainty in general-has been called "uncertaintism" (or, less frequently, "metanormativism"). The uncertaintist, presumably, must then offer an account of what one should do when one assigns, say, a seventy percent chance to the truth of expected utility theory, and a thirty percent chance to the truth of Buchak's alternative. As we can see, questions about how to deal with empirical uncertainty give rise to questions about how to deal with decision-theoretic uncertainty. But our regress does not stop there. Just as decision theories are theories about how to act in the face of empirical uncertainty, let us use the term "metanormative theories" for collections of claims about how we ought to act in the face of normative uncertainty. It seems that, just as we can suffer normative uncertainty, we can suffer metanormative uncertainty as well: we can assign positive probability to conflicting metanormative theories. Metametanormative theories, then, are collections of claims about how we ought to act in the face of metanormative uncertainty. And so on. In the end, it seems that the very existence of normative claims-the very notion that there are, in some sense or another, ways "one ought to behave"-organically gives rise to an infinite hierarchy of metanormative uncertainty, with which an agent may have to contend in the course of making a decision. Postulating such a hierarchy may seem like a strange and unneccesarily complex solution to a rather small and obscure problem. By analogy, therefore, consider three other structures explored in detail by philosophers and economic theorists: belief hierarchies, preference hierarchies, and hierarchies of deliberation procedures. We have beliefs about the world; and when one reflects on the fact that we can also have beliefs about people's beliefs, one can hardly help but document the emergence of a "belief hierarchy", constituted of beliefs about the world, beliefs about beliefs (2nd-order beliefs), beliefs about beliefs about beliefs (3rd-order), and so on [For a rigorous exploration of the structure, see  Mertens and Zamir (1985) ]. Likewise, we have preferences over features of the world, and the fact we can also have preferences over the contents of people's preferences (2nd-order preferences), and so on, gives rise to a "preference hierarchy" [see for example  Bergemann et al. (2017) ]. Finally, when we are not sure which act would maximize our utility, we may find it useful to ponder our options, even if doing so would come at a cost. Since pondering is in some sense just another option, we must then ponder whether to take one of our firstorder options or to take the option of pondering among them, and so on.  Lin (2014)  concludes that this deliberation must stop after finite steps, and will do so justifiably so long as the reasoner reaches the point that further meta-deliberation no longer obviously dominates taking one of the acts available.  Lipman (1991)  also explores this problem, in terms more similar to those we will use below: by constructing a (transfinite) hierarchy of beliefs over decision-making procedures, which justifies an act whenever it is judged to be (judged to be…) at least as good as the other available options, including the options of further deliberation at any order. Similarly, normative decision theories tell us how to act in the face of uncertainty about the true state of the world. The fact that we may have to act in the face of uncertainty about the true decision theory thus seems plausibly to give rise to a "metanormative hierarchy", similar in many respects to the hierarchies above. The infinite hierarchy of metanormative uncertainty, furthermore, is more than hypothetical. I myself, for instance, do not currently know the correct decision theory. Expected utility theory seems highly plausible to me, but I cannot fully rule out Buchak's arguments for some sort of risk-weighted expected utility theory, or arguments for theories that evade "Pascal's muggings" by giving special treatment to low-probability but high-expected-value events, to name just two examples. Likewise, I believe that there is a "true metatheory"-a correct way to act in the face of my decision-theoretic uncertainty-but I am not certain of what it is, nor of the true theory at any order of the above hierarchy. Despite all this uncertainty, however, my efforts to be moral or rational do ultimately result in conclusions about how to act. Somehow, from my infinite hierarchy of metanormative uncertainty, it seems I can be guided by normative concerns. The process by which this happens seems potentially interesting. Questions of decision-theoretic uncertainty have received little attention so far, however-research on normative uncertainty has focused primarily on moral uncertainty-and even the most recent inquiries into normative uncertainty (again, usually presented in the context of moral uncertainty) have generally left such questions unanswered.  Weatherson (2014)  avoids the problem by positing that the only norms are "firstorder"; that there simply is no right or wrong way to deal with normative uncertainty.  Sepielli (2014)  allows for "orders of rationality" and the corresponding "iterated normative uncertainty" (526). He constructs a metanormative hierarchy much like that we will define below, to serve a bridge between the mind-independent norms of which we are uncertain and the action-guiding, "all-things-considered" decision rule (given by what he calls "global systemic rationality") to which we are more plausibly all accountable. Then, given an agent who assigns some positive credence to theories of rationality, at some order, which conflict as to which act is most rational for her (at that order), he expresses the suspicion that that appealing to a higher-order rationality concept will always allow the agent to eliminate at least one action available. Posit "actions A…Z", he writes; "[p]erhaps my uncertainty regarding objective normativity will intentionally explain my doing any of A…R…. My hope is to show that, as a general matter, potential actions will be hived off with each stepping-back" (and permanently so). Given a finite act-set, this procedure would ultimately result in a coherent definition of what the agent subjectively ought to do. Unfortunately, as for whether the procedure succeeds, he concludes, "I'll have to make good on that suspicion elsewhere" (539). Tarsney (2017) reiterates Sepielli's hope for what he calls "fixed point solutions" to the regress problem. He acknowledges, however, that such fixed points will not necessarily exist (240-241), and concludes that a general solution to the regress problem must thus be found altogether elsewhere. In particular-motivated also by the thought that finite, boundedly rational agents cannot be normatively required to work out the fixed point of an infinite hierarchy, even if it exists (238) 1 -he argues for what he calls the "Enkratic Principle" (246). This is the broad, mind-independent principle that, just as long as one is in some sense "trying one's best" to do the first-order right thing in the face of normative uncertainty, one's decision is justified. On Tarsney's account, the regress is thus cut off at the second order. Finally, many accept that uncertaintism does indeed require some sort of fixedpoint solution to the regress problem-but conclude from this that there simply must be something deeply wrong with uncertaintism. The following sentiment, described in Volume 7 of Oxford Studies in Normative Ethics (2017), is typical: But a regress problem looms. Let us suppose that I am uncertain among some ordinary moral theories,  [and]  I ask what to do given the probability distribution over T1…Tn. But I am uncertain as to the answer, assigning some probability to each of U1…Un. This prompts me to ask what I ought to do given this probability distribution…. We can imagine this process iterating indefinitely…. The possibility of normative uncertainty all the way up makes the uncertaintist project look pointless. Does this possibility in fact render the uncertainist project pointless? Or can one accept the possibility of normative uncertainty all the way up, and still be norm-guided in some important classes of circumstances? I here argue the latter. To begin to answer the above questions more precisely, Sect. 2 presents a formal framework that aims to capture our intuitions about the concept of decision-theoretic uncertainty. Within this framework, Sect. 3 specifies various conditions under which two similar convergence results follow, as given in Sect. 4. Section 5 considers the earlier sections' implications for normative uncertainty more generally.
 Framework
 Choiceworthiness A finite set A = {a 1 , . . . , a |A| } of "feasible acts" presents itself. There is a finite set of "possible states" S = {s 1 , . . . , s |S| } to which I assign positive probability. 2 I assign utilities to performing each act in each state, as represented by the utility function u(A, s), where the value assigned to each act is not necessarily independent of the alternatives in A. 3 I also find myself in an overall finite epistemic position e, specifying the probabilities I assign to all relevant claims.  4  Let us call π = A, u, e my "choice problem". Definition 2.1 A choice problem π is a triple of (i) a set of acts A, (ii) an epistemic position e, and (iii) a state-contingent utility function u over A. We will say that my utility function specifies the "objective choiceworthiness" (or simply "choiceworthiness") of each act, conditional on each state. That is, given s, the choiceworthiness of a i is u(A, s) i -the ith element of the |A|-vector u(A, s). From the probabilities I assign to the states in S, therefore, I also assign probabilities to potential values of the objective choiceworthiness of each act. Definition 2.2 A (finite) choiceworthiness distribution is a (finite) probability distribution over choiceworthiness values for some (finite) set of acts. Let D n denote the set of all finite probability distributions in R n , and let some d(π ) ∈ D |A| represent the choiceworthiness distribution entailed by π . Many other finite probability distributions over R |A| might do just as well as the chosen d at representing my finite choiceworthiness distribution. Exactly which others depends on how much structure is contained in our understanding of "utility". If we understand utility to be a merely "ordinal" quantity, for instance, then any transformation of d that is monotonic in choiceworthiness (and constant in probability) represents the same choiceworthiness distribution. We are here assuming nothing about utility except that it at least partially orders act-state pairs from a given {feasible set × possible set}, and that R is "rich enough" to capture any potential difference between the choiceworthiness values of particular act-state pairs-that choiceworthiness cannot be lexicographic, for instance. As discussed at the end of Sect. 2.5, these assumptions about choiceworthiness, here made implicitly, will follow from similar assumptions made explicitly about subjective choiceworthiness.
 Subjective choiceworthiness I am uncertain about acts' choiceworthinesses. Even so, I may know that one act is the most appropriate for me to choose, given my epistemic position. As I write this, for instance, I assign high probability to the event that, if I go to the doctor, I will swiftly be cured of my back injury (an outcome I would prefer immensely to the status quo), and low probability to the roughly complementary event that, if I go to the doctor, I will waste some time and remain injured (an outcome to which I would slightly prefer the status quo). Despite this uncertainty, and all my other uncertainty, I am in fact certain that going to the doctor is the "better choice" for me right now (by far!). There is thus some scale on which the act of going to the doctor scores higher for me than the act of not going-and would score higher for anyone with the same utility function, in the same overall epistemic position, facing the same set of feasible acts. Let us call this scale "subjective chiceworthiness". It is my intuition that subjective choiceworthiness c, when well-defined, is fundamentally a cardinal scale. That is, I would maintain that a representation of acts' subjective choiceworthinesses (for an agent in a given situation) in R |A| would be unique at least up to affine transformation. If my feasible act set A = {a 1 , a 2 , a 3 } consists of going to the doctor (a 1 ), going to a very slightly less competent doctor (a 2 ), or not going at all (a 3 ), then there is some important and foundational sense in which, given my epistemic position and my preferences, the distance between c(π ) 1 and c(π ) 2 is less than the distance between c(π ) 2 and c(π ) 3 . It might be objected that I will always do whatever winds up being most subjectively choiceworthy; that therefore, in the absence of a specified theory of decision-making under uncertainty, no information is conveyed by postulated differences between the acts not chosen; and that c is therefore better understood as merely an ordering, or perhaps even as a choice relation. To this it might be replied that, under certain circumstances, differences in subjective choiceworthiness could bear some relationship to the subjective probability with which a subjectively sub-optimal act would become optimal upon further reflection. Or that cardinal subjective choiceworthiness takes on a clearer meaning in other situations of normative uncertainty (i.e. one might not choose the most subjectively morally choiceworthy act, and might in some sense be more blameworthy the less subjectively morally choiceworthy one's act was)-and that it would be strange for subjective choiceworthiness to be fundamentally cardinal in one of these situations but not the other. Or that our models of the world are generally simpler when we extend our intuitions regarding quantities' cardinality beyond the domains in which they happen to be testable-such as our intuition that temperature is generally cardinal, even on some cold, distant star that we will only discover if its temperature rises above some threshold. Furthermore, cardinal subjective choiceworthiness allows for the convergence results described below, and less structured interpretations of subjective choiceworthiness would not. If we are otherwise persuaded that the regress problem must have some solution or other, it is not circular to allow this observation itself to lend credibility to the concept of cardinal subjective choiceworthiness. In any event, for the purposes of this analysis, we will understand subjective choiceworthiness (again, when well-defined) to be cardinal. We will represent it by a "subjective choiceworthiness function" c(π ), where c assigns a real number to the subjective choiceworthiness of each of the acts in a feasible set A, for an agent with a utility function u, in epistemic position e. Note that by having c map into the real numbers, we are assuming that all information about differences in subjective choiceworthiness (and therefore utility) can be captured by ratios of differences in real numbers. We are here explicitly assuming for subjective choiceworthiness what we provisionally assumed above for objective choiceworthiness-that, for instance, it cannot be lexicographic. Like probability theories that let us condition on probability 0 events, utility theories that let us distinguish between acts that differ infinitesimally in choiceworthiness may also be interesting to consider in light of the regress problem. However, we will not touch them here.
 Metachoiceworthiness In general, if I am to translate a choiceworthiness distribution d into a determination of how to act, I must invoke a "decision theory": a collection of claims concerning how to evaluate acts in light of one's choiceworthiness distribution. For example, using this terminology, one decision theory is "Expected Choiceworthiness Theory" (EC). EC is characterized by the fact that, if I am certain that it is the correct decision theory, then each act's subjective choiceworthiness for me is its expected choiceworthiness under d.  5  Another decision theory would be "minimum choiceworthiness"-a theory characterized by the fact that, if I am certain that it is the correct decision theory, then each act's subjective choiceworthiness for me is its minimum possible choiceworthiness under d. Just as I am uncertain about the true state of the world, I may also be uncertain about the correct decision theory. To come to a determination of how to act, therefore, I may have to invoke a sort of "meta decision theory" (or, "2-metatheory"): a collection of claims concerning how to respond to one's uncertainty over decision theories. Note that, since this is so, the decision theories (we will awkwardly call these "1metatheories", for ease of indexing) cannot themselves be claims about subjective choiceworthiness. This is perhaps a surprising claim, so it bears repeating: expected utility theory (for example) is not, in this language, a theory about what subjective choiceworthiness is, or even about what it ought to be "all things considered". It is, rather, a theory about what subjective choiceworthiness "1-ought" to be, for someone with a given objective choiceworthiness distribution over his feasible set-or, a theory about what subjective choiceworthiness is for someone with a given objective choiceworthiness distribution over his feasible set, if he knows the true 1-metatheory. Suppose, for instance, that I am faced with three feasible acts, that I assign probability to each of two 1-metatheories, t 1 and t 2 , and that I am certain of "2-metatheory" m. The theories are such that if I were certain of t 1 , the subjective choiceworthinesses of the acts would be ordered a 1 a 2 a 3 ; if I were certain of t 2 , the subjective choiceworthinesses of the acts would be ordered a 3 a 2 a 1 ; and, given the probabilities I assign to t 1 and t 2 , but my certainty about m, the subjective choiceworthinesses of the acts are in fact ordered a 2 a 3 a 1 . Although I assign probability 1 2 to t 1 , I assign no positive probability to the event that a 1 is more subjectively choiceworthy than a 2 from my epistemic position. The 1-metatheories' claims, therefore, are not claims about the acts' subjective choiceworthinesses given my empirical uncertainty, but about how the acts score on an altogether different scale. Let us call this scale "metachoiceworthiness", or "1-metachoiceworthiness". Of course, metachoiceworthiness must be constructed such that, if I know that an act's 1-metachoiceworthiness is x, then the act's subjective choiceworthiness for me is also x. We might therefore informally think of 1-metachoiceworthiness as "whatever subjective choiceworthiness is, for someone who knows the correct 1-metatheory". But since, again, decision theories are not actually claims about subjective choiceworthiness, let us begin by thinking about 1-metachoiceworthiness on its own terms, and only afterward consider its relationship to subjective choiceworthiness. In any event, the elusiveness of subjective choiceworthiness is not restricted to "order 1". Just as I may be uncertain as to the correct 1-metatheory, I may be uncertain as to the correct 2-metatheory; I may therefore have to appeal to a "3-metatheory"; and the 2-metatheories are therefore making claims not about acts' subjective choiceworthiness given beliefs about their 1-metachoiceworthiness, but about acts', say, "2-metachoiceworthiness" given beliefs about their 1-metachoiceworthiness. So our regress begins.
 k-metachoiceworthiness Let us call choiceworthiness "0-metachoiceworthiness", choiceworthiness distributions "0-metachoiceworthiness distributions", and decision theories "1-metatheories". The concepts of k-choiceworthiness, k-metachoiceworthiness distributions, and kmetatheories can then together be defined recursively.
 Definition 2.3 The k-metachoiceworthiness c k of an act a i , for an agent facing finite choice problem π , is a i 's subjective choiceworthiness for an agent with the same (k − 1)-metachoiceworthiness distribution as that entailed by π , but who knows the correct k-metatheory. Let us denote acts' relative k-metachoiceworthiness by the two-place relation π,k . Definition 2.4 A (finite) k-metachoiceworthiness distribution d k ∈ D |A| is a probability distribution over k-metachoiceworthiness values for some (finite) set of acts A. Definition 2.5 A k-metatheory, applied to a finite set of acts A, is a function t k : D |A| → R |A| , representing claims about the k-metachoiceworthiness of the acts in A given (k − 1)-metachoiceworthiness distribution d k−1 ∈ D |A| . Note that, strictly speaking, if we want our k-metatheories to make k-metachoiceworthiness claims over finite act-sets of arbitrary size, we would have to say that a k-metatheory is a family of functions {t n k } from D n to R n , with one for each n ∈ N. For simplicity, however, we will take n = |A| as given and interpret our project only as an attempt to find criteria under which the subjective choiceworthinesses of any n acts will be well-defined-with the understanding that identical reasoning would apply to any other n. We can now define a few aditional terms.
 The relationship of k-metachoiceworthiness to subjective choiceworthiness Upon introducing the cardinal subjective choiceworthiness function c(π ) above, we placed no restrictions on what it could be. Now that we have documented the emergence of an elaborate web of concepts concerning π , however, we can consider how it relates to c. Recall that k-metachoiceworthiness claims are defined so that, if I know that an act's k-metachoiceworthiness for me is x, the act's subjective choiceworthiness for me is x. Let us now introduce a compatible, minimally restrictive principle with which one's subjective choiceworthiness function might comply in the face of uncertainty about an act's k-metachoiceworthiness. Definition 2.9 The Dominance Principle is the principle that • If b ≥ x ∀b ∈ [ c k ] i , and b * > x for some b * ∈ [ c k ] i , then c(a i ) > x. • If b ≤ x ∀b ∈ [ c k ] i , and b * < x for some b * ∈ [ c k ] i , then c(a i ) < x. Note that if I accept the Dominance Principle, it follows immediately that my subjective choiceworthiness for an act a i is well-defined whenever | ∩ k∈N [min([ c k ] i ), max([ c k ] i )]| = 1. That is, whenever exactly one number lies in the ranges of "admissible" (not dominated) k-metachoiceworthiness values, across all k, for an act, that number must be the act's subjective choiceworthiness. Note also that any claim about subjective choiceworthiness itself, such as the Dominance Principle, in some sense takes on both a positive and a normative interpretation. One could interpret the Principle normatively as asserting that one's subjective choiceworthiness always ought to obey the above pattern. In this case, if one accepts the Principle, one's subjective choiceworthiness also does obey it, since to hold that an act should be ranked highly for someone in your epistemic position is simply another way to say that it is highly subjectively choiceworthy. Alternatively, one could interpret the Principle positively as asserting that, as a matter of fact, subjective choiceworthiness always obeys the above pattern. If one accepts this claim (and that "ought implies can"), one must also accept that subjective choiceworthiness always ought to obey the above pattern. Either way, if one accepts the Principle, one cannot assign positive probability to k-metatheories that claim that the k-metachoiceworthiness of an act lies outside the admissible range imposed by one's k -metachoiceworthiness distribution for the act for lower orders k < k. Finally, note that the framework outlined here differs from other approaches to subjective choiceworthiness in the following respect. Some other approaches [e.g. that of MacAskill (2016a)] begin with the normative theories in all their diversity; work through problems of intertheoretic comparability; and then try to define subjective choiceworthiness with no more structure than necessary. On some accounts, this minimal structure allows only for a binary classification of acts into the "permissible" and the "impermissible" [as recommended, for instance, by  Barry and Tomlin (2016) ]. The above approach, by contrast, begins by assuming that subjective choiceworthiness is a single cardinal scale, and it characterizes k-metachoiceworthiness claims, and the k-metatheories that make them, in terms of the subjective choiceworthinesses that they would induce if they were known. This approach has the cost of assuming cardinal subjective choiceworthiness, but it has the benefit of immediately giving all my kmetachoiceworthiness claims both unit and level comparability, without requiring any further assumptions. Thus, from a cardinal definition of subjective choiceworthiness, we also get a cardinal definition of utility, without having to assume it explicitly. By similar reasoning, we also get cardinal definitions of k-metachoiceworthiness for all k. Note that we are not taking the Von Neumann-Morgenstern approach of defining my utility function so that it represents the choices I would make if I were maximizing expected utility; indeed, our project is to explore how far I can stray from certainty about expected utility theory while still knowing how I subjectively ought to act.
 Conditions
 Completeness A "partial k-metatheory" would be one that makes claims about the k-metachoiceworthinesses of some acts under some (k − 1)-metachoiceworthiness distributions, but not of all acts under all (k − 1)-metachoiceworthiness distributions. A partial decision theory of "strict dominance", for instance, claims that a i π,1 a j ⇐⇒ u(A, s) i > u(A, s) j ∀s ∈ S, and makes no other claims at all. That is, it claims that an act a i is more 1-metachoiceworthy than an act a j if and only if a i is more objectively choiceworthy than a j in all the states to which I assign positive probability, and it is silent about acts' relative 1-metachoiceworthinesses in all other cases. Conversely, Definition 3.1 A k-metatheory, applied to a finite set of acts A, is complete if it is defined throughout D |A| . One condition for the results below is that I assign positive probability only to complete decision theories. Believing that the true decision theory is complete is, I think, reasonably well motivated by the sense that, just as I know acts' 0metachoiceworthiness (i.e. objective choiceworthiness) if I know the true state, I should be able to know acts' 1-metachoiceworthiness if I know the true decision theory (and so on up the hierarchy). In any event, we will remove partial decision theories from consideration so as to separate the regress problem from the problems of incomparability that can plague normative uncertainty in their own right. Note that the framework laid out in Sect. 2 does not allow us to assign positive probability to the "nihilistic decision theory" (one that makes no claims about acts' 1-metachoiceworthinesses under any choiceworthiness distribution). Since a decision theory is a collection of claims determining what acts' subjective choiceworthinesses would be for me if I knew how to respond to my empirical uncertainty, and since my subjective choiceworthiness is already defined in the degenerate case of empirical certainty, all my decision theories at least claim that an act's subjective choiceworthiness is its objective choiceworthiness, when my objective choiceworthiness distribution is degenerate.
 Continuity We will say that Definition 3.2 A k-metatheory t k is continuous if slight changes to the individual (k−1)-metachoiceworthiness claims to which one assigns positive probability produce only slight changes to the k-metachoiceworthiness claims made by t k . To be called continuous, t k need not respond continuously to the probability one assigns to some (k − 1)-metachoiceworthiness claim. A more formal definition of continutity, as we are using the term here, is given in the "Appendix". A second condition, necessary for only the first of the results below, is that I assign positive probability only to continuous decision theories.
 The Analog Principle MacAskill (2014) argues that, when we are facing both empirical and normative uncertainty over a set of acts, there is a sense in which we should treat our empirical and normative uncertainty "analogously". If I am uncertain which act is objectively best, it may seem unlikely that the appropriate response to my uncertainty would depend on the reason (i.e., empirical or normative) for my uncertainty-especially upon considering that I might have uncertainty about how to behave without even knowing the reason for my uncertainty. In the context of the regress problem, one might likewise argue that we should treat our empirical and k-metatheoretic uncertainty analogously. More formally: Definition 3.3 Let t * k : D |A| → R |A| denote the true k-metatheory. The Analog Principle is the claim that t * k = t * 1 ∀k ≥ 1. One might wonder if it matters whether my beliefs about the k-metatheories are correlated across different orders k (as of course they are-very strongly!-if I accept the Analog Principle), or whether they are correlated my beliefs about the state of the world. In fact, it does not. A k-metatheory is simply a function of my (k-1)metachoiceworthiness distribution; a k-metatheory's output therefore does not depend on the probability that it is the true k-metatheory, nor on its probability conditional on some state or k -metatheory. A final condition, necessary only for the first of the results below, is that I accept the Analog Principle.
 Convergence
 Intuition In the context of the framework above, the commonness of well-defined subjective choiceworthiness is not surprising. If I assign positive probability to a finite number of theories, and they disagree about how subjectively choiceworthy some act should be for me, there will be a minimum and a maximum to that range of values. In the face of that uncertainty, my subjective choiceworthiness should lie somewhere in the interior of the range. Where, exactly? I will assign positive probability to different answers as to where it should lie, producing a smaller range. And so on. Given a few other assumptions (either the continuity of my theories and my acceptance of the Analog Principle, or the possibility of transfinite hierarchies), this process will not "get stuck" by shrinking the range of potential subjective choiceworthiness values for each act merely from a larger range to a smaller range. Instead, the process is guaranteed ultimately to shrink said range to a single point. In other words, nothing very counterintuitive falls out of the mathematical setup of the problem. The point of this exercise is simply to formally illustrate a coherent framework whereby our intuitions about normative uncertainty-including about the infinite regress that it threatens-can be reconciled with the understanding that, at the end of the day, we make norm-guided decisions. With that said, the convergence results can be stated as follows.
 Natural hierarchies Theorem 1 If one assigns positive probability only to a finite set of decision theories all of which are complete and continuous, and if one accepts the Dominance Principle and the Analog Principle, then one's subjective choiceworthiness is well-defined over any finite set A of acts. The proof can be found in the "Appendix". Proposition 4.1 If two acts are equally subjectively choiceworthy, this fact will not necessarily be revealed by the iterated application of one's distribution over kmetatheories. Proof Suppose for example that I assign probability 1 2 to Expected Choiceworthiness Theory, and to the analogous hierarchy (T 1 ) according to which the (k+1)metachoiceworthiness of an act is its expected k-metachoiceworthiness. I assign probability 1 2 to a risk-averse hierarchy of theories (T 2 ) according to which the (k+1)metachoiceworthiness of an act is the average of its expected k-metachoiceworthiness and its minimum possible k-metachoiceworthiness. I am deciding between two acts, a 1 and a 2 . I assign probability 1 2 to a state in which a 1 has objective choiceworthiness 0 and probability 1 2 to a state in which a 1 has objective choiceworthiness 1. Act a 2 has objective choiceworthiness 1 3 in both states. The subjective choiceworthiness of a 2 is, of course, 1 3 . The k-metachoiceworthiness claim about a 1 made by T 1 is [ c k ] 1,1 = 1 − k n=1 1 2 2n−1 ; that made by T 2 is [ c k ] 1,2 = k n=1 1 4 n . This can be seen by verifying algebraically that these summations satisfy the intitial conditions [ c 1 ] 1,1 = 1 2 and [ c 1 ] 1,2 = 1 4 , and the recursive conditions [ c k ] 1,1 = 1 2 [ c k−1 ] 1,1 + 1 2 [ c k−1 ] 1,2 and [ c k ] 1,2 = 1 4 [ c k−1 ] 1,1 + 3 4 [ c k−1 ] 1,2 (k > 1). As k increases, {[ c k ] 1,1 } converges to 1 3 from above, while always strictly greater, and {[ c k ] 1,2 } converges to 1 3 from below, while always strictly less. Thus the acts' subjective choiceworthinesses are precisely equal, even though any formal comparison of the acts is undefined if we halt our deliberation after finite k. This example does not imply the potentially concerning conclusion that one might not be able to infer acts' relative subjective choiceworthinesses, from one's choiceworthiness distribution and one's hierarchy distribution, in finite time. Indeed, since the subjective choiceworthiness of each act is well-defined under the above conditions, and is a logical consequence of one's finite choice problem, we know from the completeness of first-order logic that the value of c(π ) i can be determined for all i by some finite proof-such as the one given above. The example does, however, demonstrate that there may be facts about subjective choiceworthiness that are not discovered by the "iterate a few times and hope my uncertainty is more or less resolved" algorithm. In particular, therefore, I believe it demonstrates that fixed-point solutions to the regress problem need not take the form either of convergence after finite k or of monotonic decreases in the set of maximally kchoiceworthy acts-as hoped for in  Sepielli (2014) , and as claimed by  Tarsney (2017, p. 239) .
 Transfinite hierarchies Suppose that the k-metatheories to which I assign positive probability are all complete and compatible with the Dominance Principle, but that they are not continuous, or that I reject the Analog Principle. It is then possible that my subjective choiceworthiness for some act a i does not converge, even after infinite steps. That is, though lim k→∞ min([ c k ] i ) and lim k→∞ max([ c k ] i ) do both exist (by the fact that the respective sequences in k are monotonic and bounded), lim k→∞ min([ c k ] i ) < lim k→∞ max([ c k ] i ). In such a situation, it seems, someone could still claim that there is a "right way" for me to act. Someone could claim that my subjective choiceworthiness for a i should be the average of lim k→∞ min([ c k ] i ) and lim k→∞ max([ c k ] i ), for example. If I assign positive probability to competing theories of how to act in situations like the above, I must appeal to a theory about how to act in the face of this uncertainty. So our regress extends beyond the natural numbers, into the transfinite ordinals. The definitions given in Sects. 2 and 3 can generally be reinterpreted so that k (or, κ) is any ordinal, not just any natural number. Two, however, will require slight tweaks: Definition 4.1 (Definition 2.3, revised) The κ-metachoiceworthiness c κ of an act a i , for an agent facing finite choice problem π , is a i 's subjective choiceworthiness for an agent with the same κ -metachoiceworthiness distribution as that entailed by π for all κ < κ, but who knows the correct κ-metatheory. Definition 4.2 (Definition 2.5, revised) A κ-metatheory, applied to a finite set of acts A, is a function t κ : D κ|A| → R |A| , representing claims about the κ-metachoiceworthiness of the acts in A given κ -metachoiceworthiness distributions d κ ∈ D |A| for all κ < κ. Note that this definition of a higher-order metatheory is strictly more general than the original, even with respect to finite k, because it allows k-metatheories to be functions of one's beliefs not only about (k − 1)-metachoiceworthiness but about kmetachoiceworthiness at all lower orders k < k. The following result thus holds, as Theorem 1 does not, regardless of whether we want to allow for this possibility. We can now state the following: Theorem 2 If one assigns positive probability only to a finite set of complete κmetatheories for each ordinal κ, and one accepts the Dominance Principle, then one's subjective choiceworthiness is well-defined over any finite set A of acts. The proof can be found in the "Appendix".
 The infectiousness of stubbornness Definition 4.3 The Weak Dominance Principle is the principle that • If b ≥ x ∀b ∈ [ c κ ] i , for some κ, then c(a i ) ≥ x. • If b ≤ x ∀b ∈ [ c κ ] i , for some κ, then c(a i ) ≤ x. Let us call a κ-metatheory "compromising" if it is compatible with the (strong) Dominance Principle, and "stubborn" if it is not. Expected Choiceworthiness Theory and risk-weighted variants of it are examples of "compromising" theories. Minimax Theory, according to which an act's metachoiceworthiness is its minimum possible choiceworthiness, is an example of a "stubborn" theory. However, it is compatible with the Weak Dominance Principle. Both the theorems above demonstrate that, when all the decision theories (or κmetatheories) to which I assign positive probability are compromising (along with some other conditions), the range of potential subjective choiceworthiness values for each act shrinks to point. In both cases, this is demonstrated roughly by the fact that, when the range of potential subjective choiceworthiness values for some act is a non-degenerate interval at some order k (or κ), the application of even higher-order metatheories shrinks this interval by increasing its minimum. One might notice that, strictly speaking, neither of the proofs requires that all my decision theories (or κ-metatheories) be compromising. Suppose, for example, that I reject the Dominance Principle, but accept the Weak Dominance Principle. Suppose further that just one decision theory (or one κ-metatheory for each κ) to which I assign positive probability is "stubborn"-or, that all the stubborn theories to which I assign positive probability are one-sidedly pessimistic (like Minimax) or optimistic (like Maximax) at each order. Then our proofs can go through with only slight modifications. We just need to shrink our interval exclusively from the top or from the bottom, at a given order, to avoid asking concessions of our stubborn theories. The plausibility of stubborn theories, however, poses two challenges for this project in general. First, stubborn theories are "infectious": they can determine our behavior regardless how little positive credence we give them. Suppose I am deciding between two acts, a 1 and a 2 . I assign probability 0.99 to a state in which a 1 has objective choiceworthiness 1, and probability 0.01 to a state in which a 1 has objective choiceworthiness 0. Act a 2 has objective choiceworthiness 0.0001 in both states. Furthermore, I assign probability 0.99 to Expected Choiceworthiness, and probability 0.01 to Minimax, at every order of the hierarchy. It would be deeply counterintuitive to conclude that, from such a position, a 2 is more subjectively choiceworthy than a 1 . It would be perhaps even more counterintuitive to conclude that the acts' subjective choiceworthinesses were equal, if a 2 's objective choiceworthiness were known to be 0. But of course we do reach both conclusions, as repeated applications of my distribution over decision theories bring a 1 's higher-order metachoiceworthiness arbitrarily close to 0-according even to the sequence of "Expected Choiceworthiness"-analogous theories. Second, stubborn theories can clash with each other. If we are going to give some weight to Maximin, at every order, it seems only fair to give some weight to Maximax at every order as well. But if we do, then each act's range of potential subjective choiceworthiness values never shrinks at all; min([ c κ ] i ) = min([ c 0 ] i ), and max([ c κ ] i ) = max([ c 0 ] i ), for all κ. It does not feel as though I can fully rule out stubborn theories. Thus, despite all our progress, I am still left with the original motivating question: how do I so regularly wind up with well-defined subjective choiceworthiness? One encouraging thought is the observation that, though stubbornness is infectious in one sense, there is a sense in which compromise is infectious as well. For example, suppose I assign positive probability to stubborn κ-metatheories (or even, only to stubborn κ-metatheories) at almost all κ, but assign positive probability only to compromising theories at a relatively sparse class of orders-at the limit ordinals, perhaps. Then, even though there is a sense in which I believe in stubborn theories "almost everywhere" up the hierarchy, the scattered all-compromising orders will still force my subjective choiceworthiness range for each act down to a point. (A minimally modified version of the proof of Theorem 2 presented in the "Appendix" will hold so long as our class of "allcompromising" orders is such that, for every set M of ordinal numbers, there is an ordinal number γ ∈ : γ > μ ∀ μ ∈ M.) Similar reasoning applies to the case of merely natural hierarchies. In short, my hierarchy distribution can handle a lot of stubbornness; as long as an all-compromising order comes along every now and then to shrink my subjective choiceworthiness range for each act, there are reasonable conditions under which subjective choiceworthiness will generally be well-defined.
 Rescaling MacAskill (2014) offers the following example of undefined subjective choiceworthiness. Suppose an agent faces a choice problem: Order 0 s 1 (Pr. 18 23 ) s 2 (Pr. 5 23 ) a 1 0 4 a 2 1 0 She must choose among acts A = {a 1 , a 2 }. She assigns probability 18 23 to a state s 1 in which a 1 has objective choiceworthiness 0 and a 2 has objective choiceworthiness 1, and probability 5 23 to a state s 2 in which c 0 (a 1 ) = 4 and c 0 (a 2 ) = 0. In evaluating the at order 1, she assigns probability 18 23 to Expected Choiceworthiness Theory (t 1 ), and probability 5 23 to "Square Root Theory" (t 2 ), according to which an act's 1-metachoiceworthiness is the expectation of the square root of the difference between its objective choiceworthiness and the objective choiceworthiness of the least objectively choiceworthy act in A. As we can see, after rescaling, the 1-metachoiceworthiness distribution of a 1 is precisely what the 0-metachoiceworthiness distribution of a 2 had been, and the 1metachoiceworthiness distribution of a 2 is precisely what the 0-metachoiceworthiness distribution of a 1 had been. Therefore, if we obey the Analog Principle-that is, if our distribution over k-metatheories is the same at every order-and if we rescale after every step, our k-metachoiceworthiness distribution for the acts will flip forever between that of "Order 0" and that of "Order 1, rescaled", without converging. To my mind, however, k-metachoiceworthiness claims are intuitively characterized by the property that, if one believes them, they define one's subjective choiceworthiness. If an agent faces empirical uncertainty over two acts' objective choiceworthiness as represented above, we want to say that, in the event that she learns the truth of s 2 , act a 1 's subjective choiceworthiness for her is 4. In precisely the same language, I think, we want to say that in the event that she learns the truth of Expected Choiceworthiness Theory (but does not learn the true state), a 1 's subjective choiceworthiness for her is 20 23 -and likewise all up the hierarchy. To keep these claims "in line", the framework of Sect. 2 permits real-valued representations of the subjective choiceworthiness values and k-metachoiceworthiness distributions associated with a given choice problem to be rescaled only in conjunction, not independently. Furthermore, if this is the right way to think about k-metachoiceworthiness, then Square Root Theory (SR) is, as stated, incoherent. SR does not specify which real-valued representations of objective choiceworthiness to use as inputs, so its claims should be independent to rescaling. But they are not. Using our agent's 0-metachoiceworthiness as represented above, SR claims that the 1metachoiceworthiness of a 1 is 20 23 , and EC claims that the 1-metachoiceworthiness of a 1 is lower (just 10 23 ). But if we had represented her 0-metachoiceworthiness distribution differently, Order 0 s 1 (Pr. 18 23 ) s 2 (Pr. 5 23 ) a 1 0 1 a 2 1 4 0 we would conclude that EC claims that the 1-metachoiceworthiness of A is 5 23 for her, and that SR claims the same. To ensure that an act's true k-metachoiceworthiness for an agent be independent of the scale she arbitrarily uses to represent her (k−1)-metachoiceworthiness distribution, all our k-metatheories have to be "affine" (unique up to affine transformation). Though this condition closes the door to "Square Root Theory", it permits a wide array of other risk-averse theories, including Buchak's REU Theory and the risk-averse theory presented in Proposition 4.1.
 Applications to moral uncertainty If I assign positive probability only a finite set of complete, cardinal, comparable moral theories-or, if I at least know the right way to represent all my moral theories' choiceworthiness claims on the same cardinal scale-then the results above can be applied almost directly to my moral choice problems under empirical certainty. "Almost", because, to avoid wading into a sea of hopeless complexity, we must assume that my moral theories make no claims about how to respond to uncertainty per se. That is, we must say that, for example, among varieties of utilitarianism, I assign positive probability only to those that claim that an act's objective choiceworthiness is (something along the lines of) its objective impact on total utility. I must assign no positive probability to a variety of utilitarianism that claims that an act's objective choiceworthiness is, say, my expectation of its impact on total utility. Utilitarianism is so often described as the idea that we ought to maximize the world's expected utility-see  Parfit (1984, pp. 25, 26) , for instance-that one might easily come to believe that Expected Choiceworthiness is the only way utilitarians are allowed to deal with uncertainty. In this context, however, we should be careful to separate the unique moral claim of utilitarianism (that value is identified with utility) from the independent decision-theoretic claim (that one ought to maximize expected value). Moral theories that explicitly incorporate such decision-theoretic claims may also be interesting to consider in light of the regress problem, but we will not discuss them here. With that restriction, suppose I am certain about the state of the world. I then simply have to swap out our language about objective choiceworthiness being "my utility function, contingent on the true state of the world" for language about objective choiceworthiness being "moral value, contingent on the true moral theory", and Sects. 2-4 apply to cases of moral uncertainty, under empirical certainty, in full. If I face both empirical uncertainty and moral uncertainty, however, my situation is more complex. One approach would be for me to take "objective choiceworthiness" to be a function of both the true state and the true moral theory, to consider my probability distribution over {states} × {moral theories}, and then to apply my hierarchy distribution. Another approach, however, would be for me first to work out the subjective choiceworthiness of each act, conditional on each state, in light of my distribution over moral theories, and then to apply my hierarchy distribution a second time, to work out the subjective choiceworthiness of each act in light of my distribution over states. A third approach, symmetrical to the second, would be for me first to work out the subjective choiceworthiness of each act, conditional on each moral theory, in light of my distribution over states, and then to apply my hierarchy distribution a second time, to work out the subjective choiceworthiness of each act in light of my distribution over moral theories (These approaches have the disadvantage that they would not be able to account for any dependence between my distribution over states and my over moral theories. They have the advantage, however, that they would be able to account for the possibility that my hierarchy distribution over ways of dealing with moral uncertainty differs from my hierarchy distribution over ways of dealing with empirical uncertainty). And other conceivable approaches abound. Unfortunately, these approaches will not necessarily all yield the same subjective choiceworthiness values, or even the same act recommendations-not even under decision-theoretic certainty, and not even when I believe that the same theory should be used in the face of emprical uncertainty as in the face of moral uncertainty. Consider the following situation. I assign positive probability to a set of moral theories M = {m 1 , m 2 , m 3 } and to a set of states S = {s 1 , s 2 , s 3 }. I have two feasible acts, a 1 and a 2 . Their objective choiceworthinesses, conditional on each state and moral theory, are as follows: Furthermore, I am certain that an act's k-metachoiceworthiness is its second-lowestpossible (k−1)-metachoiceworthiness. If I apply this decision theory to my uncertainty over {states} × {moral theories}, I get c(a 1 ) = 2 and c(a 2 ) = 3, so a 2 a 1 . However, if I apply this decision theory first over states (conditional on each moral theory) and then over moral theories-or, first over moral theories (conditional on each state) and then over states-then I get c(a 1 ) = 4 and c(a 2 ) = 3, so a 1 a 2 . These complications only worsen when |M| = |S|, in which case even the "same" decision theory can aggregate across moral theories and across states arbitrarily differently. There is another way in which decision-theoretic uncertainty can interact with moral uncertainty. It is often argued that morality requires us to make decisions as if from behind a "veil of ignorance" about our own identity among those affected by our actions. If so, the moral choiceworthiness of an act depends directly on its decisiontheoretic 1-metachoiceworthiness. Suppose that I ought to act toward a group as if my identity is, in probability, distributed uniformly over the group. Then, if Expected Choiceworthiness is the correct decision theory, the Veil of Ignorance argument points toward classical utilitarianism as the correct moral theory; if Minimax is the correct decision theory, toward Rawls's "maximin criterion"; if some risk-weighted theory is the correct decision theory, toward the corresponding version of prioritarianism; and so on. 7 But we will not explore this interaction further here. Finally, the above thoughts about how to integrate the results of Sects. 2-4 into situations of moral uncertainty can apply straightforwardly to normative uncertainty in other domains, so long as one assigns positive probability finite set of theories which are in some analogous sense complete, cardinal, and intertheoretically comparable. But we will not explore such applications further here.
 Conclusion We are often uncertain about the moral and decision-theoretic norms which we believe should guide our behavior. Even when these norms conflict, however, we often have a subjective understanding of whether some act would be rationally or morally permissible for us, from our position of normative uncertainty. "Uncertaintism" might be understood as the project of unraveling how this uncertainty translates into the subjective choiceworthiness on which we ultimately feel justified in acting. When the uncertaintist tries to specify any particular mechanism for translating the uncertainty over choiceworthiness into an appropriate characterization of subjective choiceworthiness, however, we find that, just as we are not certain of our acts' objective choiceworthinesses, we are not certain of his proposed mechanism either. Nor are we certain about how to how to deal with our uncertainty about such a mechanism. Indeed, our certainty about subjective choiceworthiness seems to stand strangely on its own. In general, when we try to ground our certainty about subjective choiceworthiness in metanormative certainty at some order, we find that the hoped-for ground of certainty does not exist. For some, as cited above, this "possibility of normative uncertainty all the way up makes the uncertaintist project look pointless". The results presented here demonstrate that, as stated, the quoted worry is not justified. We can reliably have well-defined subjective choiceworthiness without being certain about the correct first-order normative theory or about any higher-order metatheory. We only have to commit to a weaker family of assumptions, such as the Let |d t k | and |d T | denote the number of k-metatheories and hierarchies, respectively, to which I assign positive probability. Let c k ∈ R |d t k ||A| represent the claims made by my |d t k | k-metatheories about the k-metachoiceworthinesses of the |A| acts in A. Let p k ∈ |d t k |−1 represent the probabilities I assign to these k-metatheories. We can now represent my kmetachoiceworthiness distribution by d k = c k , p k . 6 Definition 2.8 A hierarchy distribution d T is a probability distribution over hierar- chies. Definition 2.6 A k-metatheory distribution d t k is a probability distribution over kmetatheories.Definition 2.7 A metatheoretic hierarchy (or simply "hierarchy") T is a collection of k-metatheories t k with one for each k > 0.
			 Synthese (2021Synthese ( ) 198:1177Synthese ( -1199
			 In the process, he distinguishes between the "ideal regress problem", which an ideal agent with perfect reasoning ability might face, and the "non-ideal regress problem". In order to separate the issue of normative uncertainty from the issue of bounded rationality, here we will consider only what he calls the "ideal regress problem". Note that we are implicitly assuming that normative facts are not logical facts; if they are, then it is impossible for an agent with perfect reasoning ability to suffer normative uncertainty.
			 Here and elsewhere, we will assume that all credences satisfy the Kolmogorov probability axioms. Note that this implies that all the sets over which I have probability distributions are nonempty.3  We will assume that the probability of each state is independent of the chosen act. We will thus bypass the question of how to act in the face of such dependency (i.e. causal decision theory vs. evidential decision theory and other alternatives), and focus entirely on the question of how to act in the face of uncertainty over states (i.e. expected utility theory vs. its alternatives). For an analysis of how to approach uncertainty between causal and evidential decision theory, see MacAskill (2016b).4  More precisely, let e specify my probability distribution over the set of [{states of the world} × {decision theories (or, 1-metatheories)} × {2-metatheories} × {3-metatheories} × • • • ]. The concept of a "k-metatheory" is defined in Sect. 2.4.
			 Let us distinguish EC from "Maximize Expected Choiceworthiness" (MEC). MEC is the weaker theory characterized only by the fact that, if I am certain that it is correct, then the acts with the highest subjective choiceworthiness for me are the acts with the highest expected objective choiceworthiness under d.
			 This is not to say that a given distribution can only be represented by one particular vector pair. Multiple k-metatheories may make the same k-metachoiceworthiness claims in some situation, for instance.
			 The implications of risk-weighted expected utility theory for decisions made on behalf of groups, rather than individuals, are further discussed byBuchak (2013, pp. 167, 168).