Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations

Authors: Aryan Shrivastava, Jessica Hullman, Max Lamparth
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Language Models, AI Safety, Natural Language Processing, Inconsistency, Transparency, Military

Abstract:
There is an increasing interest in using language models (LMs) for automated decision-making, with multiple countries actively testing LMs to aid in military crisis decision-making. To scrutinize relying on LM decision-making in high-stakes settings, we examine the inconsistency of responses in a crisis simulation (``wargame"), similar to reported tests conducted by the US military. Prior work illustrated escalatory tendencies and varying levels of aggression among LMs but were constrained to simulations with pre-defined actions. This was due to the challenges associated with quantitatively measuring semantic differences and evaluating natural language decision-making without relying on pre-defined actions. In this work, we query LMs for free-form responses and use a metric based on BERTScore to quantitatively measure response inconsistency. We show that the inconsistency metric is robust to linguistic variations that preserve semantic meaning in a question-answering setting across text lengths. We first study the impact of different prompt sensitivity variations on wargame decision-making inconsistency at temperature $T = 0$. We find that all models exhibit levels of inconsistency indicative of semantic differences, even if answering to semantically identical prompts. We also study models at $T > 0$ under fixed prompts. We find that all studied models still exhibit high levels of inconsistency, even when adjusting the wargame setting, anonymizing involved conflict countries, or adjusting the sampling temperature parameter $T$. Further qualitative evaluation shows that models recommend courses of action that share few to no similarities. We find that inconsistency due to semantically equivalent prompt variations can exceed inconsistency from temperature sampling for most studied models across different levels of ablations. Given the high-stakes nature of military deployment, we recommend further caution be taken before using LMs to inform military decisions or other cases of high-stakes decision-making.

Decision: reject
Reviewer scores: 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730412400192

thread_type:
official_review

<thread_object>
creator: reviewer_xyfm

<content>
summary:
This paper proposes to examine how reliable LLMs are in high stakes decision making situation. For this, the authors conduct crisis simulations with free form responses and using bertscore and measure the inconsistencies across 5 LLMs. Across experiments conducted, this study shows that there inconsistencies even after fixing parameters like conflict anonymization and temperature, and show that prompt variations can lead to greater inconsistency than temperature adjustments.

strengths:
- the paper is well written and the methodology is easy to follow
- with the increasing use of LLMs in different areas, the focus on high-stakes decision-making contexts is timely and of importance

weaknesses:
- The framework for measuring inconsistency uses only BertScore. This potentially limits the evaluation setting to the discrepancies found through semantic similarity missing other forms of inconsistency.
- There is no human evaluation or correlation to human judgement of the inconsistency score. 
- I understand that the objective of this study is to probe LLMs for inconsistency in this particular context. While this study underlines a problem, it does not suggest possible mitigations for the issue of inconsistency.

questions:
Besides inconsistency, is the proposed evaluation framework able to highlights other types of errors or discrepancies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730571848195

thread_type:
official_review

<thread_object>
creator: reviewer_cn3c

<content>
summary:
This paper presents an investigation of the consistency of LLM responses in a military strategy scenario. Authors invert the BERTScore measure of semantic similarity to compute an inconsistency score, which they validate in a synthetic free-form QA setting based on Truthful QA. In the experiments, the paper shows that LLMs' answers are generally quite inconsistent in both types of generations for their scenario (initial, continuations). They also explore the effect of temperature on inconsistency, as well as the effect of prompt variations.

strengths:
- I appreciate the paper's main goal of investigating LLM inconsistencies in high-stake military scenarios
- I liked that the authors performed a validation of the inconsistency score using synthetic data with TruthfulQA, which also allows readers to calibrate on what score values mean.
- The main experiments were well executed.
- I liked the investigations into the prompt variations.
- I also appreciated the disclaimer at the end of the introduction.

weaknesses:
- The paper's main weakness is the conceptual problem definition.
    - I feel like in realistic high-stakes settings, the temperature should probably be set to 0, which is similar to greedy decoding. Authors should probably focus their experiments on that temperature, though I'm expecting very low inconsistency. More importantly, authors should make a clearer argument for why t=0 should not be used in these high-stakes settings, or why t>0 should be studied.
     - Concretely, I feel like section 6 was most relevant to the realistic way that LMs should be used. I wish authors had expanded on such experiments, possibly exploring various types of rephrasing and exploring exactly how the inconsistencies were affected.
- Second, I feel like the paper's results might be limited in terms of generalizability due to there being only one wargame scenario considered. I understand that this is a relevant vignette, but it would be very interesting to have at least 3-5 high-stakes scenarios (either all wargame or some other high-stakes domains like healthcare). This would ensure that the results aren't an artefact of the topics or domain chosen in the one scenario.
- Third, I felt like the paper could use more in-depth analyses w.r.t. how model responses are inconsistent. The measure at hand is quite coarse-grained, and might not be able to capture more nuanced consistent/inconsistent outputs (e.g., LLM outputs offering two alternatives, one of which is similar between two outputs). Given the specific and high-stakes nature of the scenario, it'd be really useful to have more insights into how the outputs differ, as currently, the coarse-grained measure yields very little information about how inconsistencies should be mitigated (other than lowering the temperature).
- Missing citations: http://arxiv.org/abs/2203.12258, https://arxiv.org/abs/2311.00059, https://arxiv.org/abs/2310.11324
- L204: Authors mention a bi-directional entailment clustering method, but without more details, it seems very confusing why the authors mentioned that... I would remove that sentence or specify why they needed to test that method and why they didn't include the results in the final main text.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1731046816800

thread_type:
official_review

<thread_object>
creator: reviewer_mcsh

<content>
summary:
This paper investigates the inconsistency of large language models (LLMs) in terms of ablations like sentence order and semantics when applied in war games. The authors focus on measuring free-form response variability using BERTScore to quantify inconsistencies. The findings indicate that the LLMs exhibit considerable inconsistency in their response, which raises concerns about their reliability in high-stake decision-making contexts.

strengths:
1. The overall presentation is very clear and intuitive
2: The experimental setups are rigorous - experiments are run with multiple LLMs, variations in prompt scenarios, and statistical controls to examine the consistency across different levels of temperature settings and prompt structures.
3. Besides quantitative measures, the paper provides qualitative examples that provides valuable insights
4. The prompts are fully provided for ease of reproduction

weaknesses:
1. The paper focuses on a very specific hypothetical military scenario. It’s also uncertain whether the observed inconsistency is unique to the wargame setup or would generalize to other critical decision-making applications. This might limit the generalizability to other high-stakes applications.
2. The paper’s main innovation centers on using BERTScore as an inconsistency measure, which may not offer significant novelty in approach. 
3. The study also did not sufficiently compare this approach with other potential inconsistency measurements. 
4. The choice of a default temperature setting of 1.0 in Section 5 may not be appropriate, as it introduces significant response inconsistency by design.
5. The comparison are limited to a few closed-source LLMs
6. While the study demonstrates that LLMs can produce inconsistent responses, it would be more impactful if it included strategies for reducing such variability.

questions:
1. Why was the wargame scenario chosen as the primary setting for examining inconsistency in high-stake decision making? Do you believe the inconsistency findings would generalize to other types of critical scenarios, or are they specific to the military context?
2. In Section 5, why was a temperature of 1.0 chosen as the default setting?
3. Could this research potentially inform new methods to improve LLM consistency in decision-making contexts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731884877623

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/2)

comment:
Dear reviewers and chairs,

Thank you to the reviewers for your constructive reviews and everyone for their consideration! Below, we address common concerns shared by all reviewers.

### *Why we evaluate LMs at temperature T > 0*
We have consulted academic scholars in international and national security throughout the course of this research. They said militaries cannot afford their systems to give deterministic decisions in case of cybersecurity failures leading to  predictable decision-making. Thus, we have strong reason to believe militaries will not set the temperature to 0.0. We agree this is unintuitive, and we will clarify this in the paper. We uphold that this fact makes the results all the more pressing - those implementing LMs in high-stakes settings should find a way to robustly parse through their inconsistency or scrap their use altogether.

Additionally, related works have analyzed LM inconsistency as a method for hallucination detection [e.g., 1, 2]. These works hinge on the assumption that high inconsistency levels imply low model confidence, and often set T = 1.0 for baseline experiments. Therefore, by setting T = 1.0, we are able to proxy a notion of model confidence and reliability when responding to fixed prompts. This establishes a necessary comparison point for the experiments in Section 6, where we set T = 0 and find that model inconsistency due to prompt variations when responding with T = 0 can lead to inconsistency levels comparable to model responses to fixed prompts at T > 0 (and up to 1.2 for some models).

Lastly, previous work has shown that there are limitations to greedy decoding [e.g., 3, 4, 5]. Thus, it is reasonable to expect that LMs be deployed at T > 0 even if it comes at the cost of less consistency.

### *Regarding generalizability concerns*
We see the reviewers’ concerns regarding generalizability and agree that applying our framework to more applications would strengthen our results. To this end, we run additional experiments on free-form responses of chatbots interacting with users in mental health emergencies using the public dataset from [6]. From that dataset, we use the responses from frontier closed-source models and open-source models (ChatGPT-3.5, ChatGPT-4, Mistral-7b, Llama2-7b, Llama2-13b, Claude-3-opus, Gemini) which are expert human labeled as either “safe,” “unsafe,” or “borderline”.

We find that models still exhibit high levels of inconsistency. Additionally, we find that our inconsistency metric is able to distinguish between safe and unsafe responses with statistical significance. We also find that borderline responses were significantly closer to safe responses than unsafe responses. We will add this evaluation to the appendix in the revised pdf. These results show that our inconsistency evaluation framework generalizes to a different context and different models not present in our initial analysis.

### *Further analysis and human evaluation for inconsistency score*
We have conducted another set of new experiments to test how our inconsistency score behaves under synthetic, on-distribution ablations specific to the experimental scenario. We selected example responses and changed between one to five “actions” conveyed in the response, while keeping all other text (including other actions) in the response the same. Keeping the other actions unablated and text exactly identical creates more stringent evaluation conditions for the inconsistency score as compared to the true conditions present in the main experiments.

We find 1) inconsistency increases approximately linearly as we change additional actions, 2) Changing just 2 actions while keeping the rest of the text exactly the same yields inconsistency scores indicative of substantial semantic difference. Because of the stringent test conditions, the observed inconsistency scores are a lower bound to those we would see when evaluating on the true dataset. We will add these results to the appendix in the revised pdf.

Additionally, we qualitatively evaluated many of the sample responses and their corresponding inconsistency scores. We will include a representative sample of response pairs and their corresponding inconsistency scores in the appendix of the revised pdf. Finally, as mentioned in the paper, we use BERTScore based on the DeBERTa xlarge model, which achieved a pearson correlation with human judgment of 0.7781 [7], the highest of all supported models. We will be sure to include this in the revised pdf as well.

*Continued*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732749925448

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We thank the reviewers for their continued engagement with our work! In response to the original comments and points brought up during discussion, we have updated the paper with a few revisions.

In the revised version, you will see text in *red* and text in *blue*. The text in *blue* is content that was exactly present in the original version (bar minor rephrasing) that was moved into a new section in the revision. The text in *red* is new content or content that we majorly changed.

**Summary of major changes:**
1. We added a discussion of our new experiments measuring inconsistency on chatbot responses to mental health crises. The full discussion is provided in Appendix E, with mentions in Section 4.2 and 7. This aims to address reviewer concerns regarding whether high inconsistency generalizes to other critical task domains, as well as whether our inconsistency score (and generally, BERTScore) can be used in other evaluation frameworks in other contexts.
2. We added a discussion of our more fine-grained analysis of BERTScore on synthetic, wargame-specific responses. The full discussion is provided in Appendix C, with a mention in Section 4.3 We also added a representative sample of response pairs taken from the experiments  alongside their inconsistency scores to provide further human evaluation (Appendix D). Finally, we added that the BERTScore model we implement achieved a Pearson correlation with human judgment of 0.7781 as found by the original authors of the BERTScore paper to Section 3. This aims to address reviewer concerns pertaining to the interpretation and validity of the inconsistency score.
3. We re-order Section 5 and Section 6 as recommended by Reviewer xYfm. That is, we moved our prompt sensitivity experiments above our evaluation at T = 1. This is in response to reviewer concerns regarding our conceptual problem definition and belief that the paper would be better framed if T = 0 was used as the default temperature. To that end, in the new Section 6, we write a bit about *why* we believe T = 1 to be well-motivated and worth evaluating. Because we acknowledge that is an unintuitive concept, particularly to those removed from national security circles, we provide a full discussion of this motivation in Appendix G.
4. We demote the mention of bi-directional entailment clustering to a footnote so as to not overstate its importance to the present work and maintain the focus on BERTScore and our contributions pertaining to the validation of it for inconsistency evaluation. We add a sentence explaining why we do not include it in the main body. We maintain the full discussion of our tests with the method and why exactly we do not include it in Appendix F. This is in response to reviewer recommendations to clarify the mention of bi-directional entailment clustering and to better position an aspect of our contribution. 

We also made some minor changes. For example, to accommodate the re-ordering of sections, we slightly change the abstract and introduction to ensure that our presentation stays strong and true to the present paper. We also added some missing citations, including those recommended by Reviewer cN3C. 

We appreciate all the reviewers' feedback and time! We believe our paper is much stronger as a result. We have conducted additional experiments and addressed all concerns raised in the reviews to the best of our ability. We would greatly appreciate it if the reviewers could consider revising their scores in light of these updates or kindly provide further details if any concerns remain unaddressed.

Sincerely,\
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary:**

The authors analyze how previously observed inconsistency in outputs of LLMs influences their behavior during military decision-making and mental health simulations. They use BERTScore for identifying model inconsistency. Their findings indicate LLMs may be too inconsistent for deployment in high-stakes scenarios. 

**Strengths:**

- Rigorous experimentation with BERTScore and good qualitative examples.

- Good comparison with synthetic perturbations of TruthQA.

**Weaknesses:**

- The point of the paper's main findings is unclear given the applications, and the authors' rebuttal further adds confusion by arguing non-determinism is favorable in wargames. What is being measured with BERTScore is not internal reliability. I would argue that the inconsistency being measured here isn't inherently bad. The authors really need a more nuanced metric capturing inconsistency in outcomes: in other words, they need to determine if multiple dissimilar decision paths lead to similarly or dissimilarly desirable outcomes. 

- The writing of this paper is confusing. Despite the title, it seems like the authors focus primarily on justifying BERTScore as a consistency metric rather than wargame simulation, which becomes an afterthought. I think the paper would benefit from a broader focus, with more applications, comparison across different metrics of consistency, and deeper analysis of the implications. 

- Lack of discussion around mitigation of undesirable inconsistency.

The core promise of measuring how LLM consistency impacts high stakes decision-making, particularly military decision-making, is a great idea. However, I believe the paper is far from being in a publishable state and is not appropriate for ICLR.