Can You Follow Me? Testing Situational Understanding for ChatGPT
Authors: Chenghao Yang
Allyson Ettinger
Venue: EMNLP 2023 Main (2023)
Keywords: Situational Understanding
Analysis of Models
ChatGPT
TL;DR: In this paper, we propose a novel synthetic testing environment for testing situational understanding ability to test ChatGPT. We instruct ChatGPT to process a series of sparse environment updates across time in a dialogue history.

Abstract: Understanding sentence meanings and updating information states appropriately across time---what we call ``situational understanding'' (SU)---is a critical ability for human-like AI agents. SU is essential in particular for chat models, such as ChatGPT, to enable consistent, coherent, and effective dialogue between humans and AI. Previous works have identified certain SU limitations in non-chatbot Large Language models (LLMs), but the extent and causes of these limitations are not well understood, and capabilities of current chat-based models in this domain have not been explored. In this work we tackle these questions, proposing a novel synthetic environment for SU testing which allows us to do controlled and systematic testing of SU in chat-oriented models, through assessment of models' ability to track and enumerate environment states. Our environment also allows for close analysis of dynamics of model performance, to better understand underlying causes for performance patterns. We apply our test to ChatGPT, the state-of-the-art chatbot, and find that despite the fundamental simplicity of the task, the model's performance reflects an inability to retain correct environment states across time. Our follow-up analyses suggest that performance degradation is largely because ChatGPT has non-persistent in-context memory (although it can access the full dialogue history) and it is susceptible to hallucinated updates---including updates that artificially inflate accuracies. Our findings suggest overall that ChatGPT is not currently equipped for robust tracking of situation states, and that trust in the impressive dialogue performance of ChatGPT comes with risks. We release the codebase for reproducing our test environment, as well as all prompts and API responses from ChatGPT, at https://github.com/yangalan123/SituationalTesting.

Can You Follow Me?
Testing Situational Understanding in ChatGPT
Chenghao Yang
Allyson Ettinger Allen Institute for AI al allysone@alenni..og
University of Chicago
chenghao@@uhiicao..oo edu
Abstract
tial for building communicative human-like Al agent (e-g., ChatGPT (OpenAT, 2022)). This ca- pability requires an agent to know truth con- ditions (Lewis, 1976; Heim and Kratzer, 1998) of a given input, and how that input updates the context over time (Veltman, 1996). For in- stance, the natural language input "Open BOX- 4, obtain KEY-1" in game environment should assert updates DOPENED(BOX-4 "OBTAINED(KEY-I
Understanding sentence meanings and updat- ing information states appropriately across time-what we call "situational understanding" (SU)-is is a critical ability for human-like Al agents. SU is essential in particular for chat models, such as ChatGPT, to enable consis- tent, coherent, and effective dialogue between humans and AI. Previous works have identi- fied certain SU limitations in non-chatbot Large Language models (LLMs), but the extent and causes of these limitations are not well under- stood, and capabilities of current chat-based models in this domain have not been explored. In this work we tackle these questions, propos- ing novel synthetic environment for SU test- ing which allows us to do controlled and sys- tematic testing of SU in chat-oriented models, through assessment of models' ability track and enumerate environment states. Our envi- ronment also allows for close analysis dy- namics of model performance, to better under- stand underlying causes for performance pat- terns. We apply our test to ChatGPT, of-the-art chatbot, and find that despite the fun- damental simplicity of the task, the model's per- formance reflects inability to retain correct environment states across time. Our follow-up analyses suggest that performance degradation is largely because ChatGPT has non-persistent in-context memory (although it can access the full dialogue history) and is susceptible to hal- lucinated updates- including updates that arti- ficially inflate accuracies. Our findings suggest overall that ChatGPT not currently equipped for robust tracking of situation states, and that trust in the impressive dialogue performance of ChatGPT comes with risks. We release the codebase for reproducing our test environ- ment, as well as prompts and API responses from ChatGPT, at https: hht..tb......... con/ yangalan123 http:/sitaaiiinnaaaaaaaeae
True" and True" in the agent's rep- resentation of the environment state. We describe this ability as situational understanding (SU), as the agent needs to ground language in situations and understand situational changes.
Recent research indicates that despite the tremen- dous success of Large Language Models (LLMs) (e.g.. GPT-3 (Brown et al., 2020)), these models still fail to understand situational changes, and can- not serve as human-like agents accomplish real- world tasks. For example, evidence suggests that models fail to detect sarcasm expressed under- lying context (Suzgun et al., 2022), infer incorrect logic states when context changes in games (Li et al., 2022) and fail to track entity state changes in discourse (Kim and Schuster, 2023). While these works have shown important limitations in LLMs, it remains unclear why these models show these limitations, and there less work that shows the extent to which these limitations persist more recent, chat-trained models like ChatGPT.
state-
apdates-including
this work we seek to shed light on both of these questions. To test the situation tracking abil- ity of ChatGPT and other chat models, we design synthetic environment for controlled testing of models' ability to follow instructions and main- tain consistent understanding of situation states. We design multi-stage testing framework, and include various controls avoid models relying on mappings memorized from pre-training rather than doing real-time state tracking. Our environment al- lows for flexible testing under different conditions,
1 Introduction
Understanding the meaning of language inputs and their impact on information states is essen-

---

well controlled follow-up analyses enable closer examination of causal factors.
ical underlying situational understanding to gen- crate coherent, consistent and effective dialogue (Karttunen, 1969; Kamp et al., 2011; Wu et al., 2019). Second, tests used in previous work are susceptible to interference from confounds present when fine-tuning linear probes, or shortcuts in the testing environment that can be utilized to bypass situational understanding tests (Kim and Schuster, 2023). We aim to address some of these concerns by introducing a number of additional controls in our tests. Finally, we also carry out more in-depth exploration model update dynamics, to better un- derstand causes for patterns of performance.
We apply our tests primarily to ChatGPT, which we find to outperform other models fine-tuned on chats. Our results show that ChatGPT"s perfor- mance reflects failure to retain coherent and cor- rect environment states across time, despite the simplicity of the task and the fact that ChatGPT has access to the full dialogue history in its in- put window. Our follow-up analyses suggest that a major contributor to this degradation is failure to retain prior states in memory, as well as sus- ceptibility to spurious hallucinated updates (which can also contribute to accidentally correct outputs). Our findings suggest overall that ChatGPT lacks the ability track situation state changes robustly.
3 Building a Situational Testing Environment
What we want to test is models' ability to pro- cess situational changes conveyed by language in- puts, and to maintain internal representations of the corresponding situation states. To test this, we use synthetic box- mox-moving environment like TextGame (Li et al., 2021), where we have full access to underlying game states, but climinate complicated reward pursuing and route branching. Having this kind of full-information synthetic en- vironment is helpful to test models' evolving un- derstanding of environment states the input de- scribes progressively more situational changes.
2 Related work
Situational understanding ability is essential for building agent-like real-world intelligent systems. comprehensive benchmarks, Big-Bench-HHi (BBH, Suzgun et al. (2022)) finds that for GPT- family models and PaLM 540B models (Chowd- hery et al., 2022), even equipped with the state- of-the-art Chain-of-Though prompting, still fail on tasks that require situational information (e.g.. detect sareasm). HELM (Liang et al., 2022) also points out LLMs can lack state tracking abil- ity based on evaluation results for 30 models on bAbl (Weston et al., 2016) environment.
3.1 Environment Setup
The environment includes two basic components:
In synthetic and controlled environments, Li al. (2021); Andreas (2022); Li et al. (2022) does probing on BART (Lewis et al., 2020), T5 (Raffel et al., 2020) and GPT-3 using the SCONE (Long et al., 2016) and TexIGame (Côté et al., 2019) datasets, and show that these models lack the abil- ity to track and infer information states. Kim and Schuster (2023) re-analyze the results in Li et al. (2021) and find that GPT-3/3.5 and Flan-T5 (Chung et al., 2022) cannot track entity state changes in discourse. Toshniwal al. (2022) instruct GPT- 2 (Radford et al., 2019) to play chess and find difficult to track board states per move.
Instructions. Instructions directed to an agent, defining the agent's quest and provid- ing information about the environment. As we will describe below, this component also sometimes contains explanations of the mean- ings of non-language functors and arguments to be used in the output.
2. Steps and queries. Descriptions of steps taken by the agent, followed by query that prompts for enumeration of all relevant en- vironment states after the action taken. Queries and answers to the queries take the format of sets of logical representations of states with corresponding truth values.
We build on these previous works in two pri- mary ways. First, these works mainly use LLMs feature extractors (c.g., train probe models over intermediate representations or final-layer repre- sentations), and none of them discuss whether situ- ational understanding limitations still exist in the recent powerful and widely used ChatGPT system. We investigate here whether ChatGPT has the crit-
In our tests, the input provided the model includes the task instructions, along with n few- shot Step-Query Answr examples to demonstrate the task. the zero-th step, we define the step

---

"Do nothing" and provide in the Answer the full set of correct truth values as an initialization. The иt few-shot Step-Query-Ansee then followed by series of Steps without Query- Answer components, followed by a Test Step and Test Query that the model is expected to complete with the full enumeration of environment states that hold after that step is taken. An example of our environment format, to be further explained below, is shown in Example 1.
actions, without needing convert to abstract sit- uation states. To control for this possibility, in addition to using natural language (NL) functors and arguments for our environment states, we also adopt settings in which synthetic language (SL) is used to specify the functors and arguments of the environment states. This allows us to disen- tangle our target task from pattern memorization and copying capabilities, better ensuring that mod- els must rely on combination of the instructions and the changes caused by the actions taken. To build synthetic functors and arguments, we use ran- domly selected ASCII characters. The length for each functor or argument is a random sample from (1, In the instructions, we include expla- nations of the meanings of these synthetic func- tors and arguments. An example of our synthetic language setting is shown in Example 1 (example instructions from our NL setting can be seen Example and Example 3). To counteract ran- domness effects and reduce any bias from specific synthetic language, for each test case (an instance), we generate different set of synthetic functors and arguments.
examples are
Instructions: As an agent, you need find the way go out of this quest. Currently, there are several boxes in front you and there is key inside each box. You can use only one of these keys to open the door and fimish this quest. There are boxes and keys here. Boxes are identified as jqC-X and Keye are identified as baS-X.
means that JqC-2 has been opened. B(bsS-3)=True means that baS-3 has been obtained. NvSWxzvJb (jqC-2)=False means that jqC-2 has not been opened. B(bsS-3)-False means that baS-3 has not been obtained.
Step-O: Initialization. Do nothing.
Question: (vsWwzvvJb(jaaCO0=== B....BBB))-)BB bas-1)=?...
Answer: NvSWxzvJb ()(0aa)vvv(()))))))))00))))) F0aaa00 (abasa.............))
False...
Step-1: Open JqC-3 and retrieve baS-2.
Counterintuitive instructions To further control for potential memorization of mappings between language and logical states from pre-training, we include one additional manipulation involving what call counterintuitive instruction. In counterin- tuitive instruction settings, the instructions define mappings that reverse the standard usage of logi- cal statements and truth values. The two versions we use are counterintuitive ouipul format (Exam- ple 2), and counterintuitive language instruction (Example 3). These manipulations draw on the tradition of negation testing (McCoy et al., 2019; Ribeiro et al., 2020; Hossain et al., 2020; Ravichan- der et al., 2022), and allow us to further disentangle pre-training memorization from understanding of our particular instructions.
Question: (jSW)avbb(jgC-0)).....
Example I: Running example of test environment.
For the experiments below, we set the number of boxes and keys to 10. This corresponds to a total of 20 environment states (boxes and keys have separate states) to be enumerated after each step. We sample the number of steps randomly from uniform distribution (1,, 10). We then keep in- structions almost entirely the same across samples. except that in the Synthetic Language settings (see Section 3.2) the state predicates are defined ran- domly and therefore vary between samples (e.g., in one instance we use NuSWacwhbiigg--))=HHi to refer to the box jqC-0 having been opened, while in another instance we use Abc(bb-O)=True to rep- resent the box bb-0 having been opened).
3.3 Evaluation Metrics
We mainly evaluate models' success in our syn- thetic environment by measuring two metrics: State-EM, and Step-EM.
3.2 Robustness Checks for State Tracking
Synthetic language When testing models' abil- ity to map to logical representations of environment states, a concern with using language-based logi- cal symbols (such as OPENED (BOX-4)) that the models may be able to leverage pre-training on similar language- logical symbols, or sim- ply copy from the input language describing the
State-EM is the proportion of all predicted
'There are other possible perturbations that we can do, such as adding distractor at each step (Appendix A) or apply- ing synthetic language only on functors/ only on arguments (Appendix B). For simplicity, we omit the discussion of other perturbation types and only focus on the two robustness checks explained in this section.

---

Instructions: OPENED (BOX-3)=Trus means that BOX-3 has Not been opened that XEY-1 has Not been obtained. OPENED ( that BOX-3 has BOX-3)-False OBTAINED (KEY-1)= been opened. False means that KEY-1 has not been obtained.
Instructions: OPENED (BOX-3) that BOX-3 has been opened. (KE-)) =Fasse means that KEY -1 has been obtained. (BOX)) True means that BOX-3 has not been opened. (BTAINED(KEY-1) that KEY-1 has not been obtained.
(KEY))
Example 2: CounterIntuitive Output Format
Example 3: Counter-Intuittve Language Instruction
Figure Counter-Intuidive Task Definition Examples
states that match the expected states. This metric is useful to check to what extent the model devel- ops correct partial understanding in response to situational changes.
truth values and corresponding states from model outputs. Details can be found in Appendix D.
Existence of Shortcuts It is clear that simple string automata plus a status tracking table should already be sufficient to solve every instance of our tasks. However, the simplicity of the task is part of its value: if models have basic capacity to track and maintain environment states, this task should be straightforward. Nonetheless, we will see in the experiments below that ChatGPT still struggles to solve these tasks reliably, despite the existence such simple solutions, indicating the presence of fundamental limitations in this class of capability.
State-EM #( Matched States)
# (Queried States)
Step-EM is a stricter metric than State-EM. the proportion of steps for which the full set pre- dicted states at that step have an exact match with the expected states, including all the truth values and the number of predicted states. This allows us to check whether the model can maintain consis- situational
tent and coherent understanding changes. This metric is also important given the sparsity of updates each step, to ensure that mod- els cannot achieve strong performance simply by copying previous states.
3.4 Discussion: Synthetic Environment as Simulation for Real-World Application
At its core, our synthetic environment is simpli- fied simulation for real-world state-tracking tasks (usually in the form of slot-filling critical ca- pability of dialogue systems / chatbots (Williams et al., 2014; Henderson et al., 2014; Wu al., 2019). By prompting the model to update states, we are gradually giving the model more contex- tual information and testing whether ChatGPT, the state-of-the--e chatbot model, can closely follow users' prompts and keep track of the full interaction history.
{
Truth States Matched Predicted States
Step-FM
otherwise
(2)
We illustrate the computation of these met- rics for the following simplified case (2 boxes and keys, synthetic language, no counterintuitive in- structions):
[Instructions and some previous stepe] Question: NvSWwzvub(jgC-0)= B(beS-0)=?
in
Our work has important potential implications as the usage of LLMs continues proliferate. In- structing LLMs remember many initial states, operate over synthetic languages, and keep track of interaction history can be seen as an important step in eventually teaching highly-capable agent to follow social norms and policies. Our initial set of environment states is similar to establishing basic rules about dos-and-ed and--ann-dn'' ts the beginning of human-Al conversations. The usage of synthetic languages likewise similar teaching Al agents about specific tones or styles of communication, ter- minologies, jargon, or perhaps even low-resource languages. Analyzing whether the model can keep
Nv$WkxvvJb(jeC-DDee B(bsS-1)=?
u Correct answer: 1iskkerrbb(jc--)=TTrue B( be$-0)-False, wwSwxaawlb(jgc--D-aaaaee
(bsS-1)=Palse
12 Model Answer: (rSlitzwJb(jqC-O))
B (baS
-0)=True, Nv JvSikeerbb(jcC-1)=aaa
B (bas
-1)=Falae
Example 4: Example for Metrics Computation
State-EM in this case is 3/4 4- 75% (only 3 out of 4 states are correct), while Step-EM is 0 because the step contains incorrect state.
When computing these metrics, we find that at times the generated output does not strictly fol- low the given format from the in-context samples. Therefore, we use regular expressions to extract the

---

track of environment states can allow draw conclusions about the model's ability follow in- structions or rules. In this sense, our work also has implications with respect to recent trends of Constitutional AT or Sule-based/Princille--e models (Bai et al., 2022; OpenAI, 2023; Sun et al., 2023), in which human social rules ("constitution") are explicitly written at the beginning of prompt align AT agents' behavior to human values. Our initial environment states are like constitution poli- cies that AT agents should obey. The steps and queries in our environments are reminiscent of a situation in which certain policies can be allowed to be updated with user permission. For example. initially an Al agent may be programmed to try its best to answer every question and disallow overly conservative responses like refuse-to-and to-nsseer---------- but under certain situations, the user could update with permission to the agent to refuse to answer for pri- vacy, fairness or other social reasons.
4.1 Results
Results for these tests on ChatGPT are shown in Table 1. We can see number of patterns.
Failures on Step-EM Under normal instructions, though the model achieves high State-EM (i.e., 90%), ) the Step-EM is generally much lower (up to 70% lower in 2-shot and 40 - 50% lower in 5- shot). This indicates that although models are able to take advantage of the sparsity of the updates to get a majority of states correct, they are much less successful in accurately characterizing the entire environment state at given step. As we will see in Section 5.2 State-EM may also be skewed by accidentally-correct states, and should in general be interpreted with some caution.
ChatGPT has limited capability to understand and follow counterintuitive instruction. From Table 1, we see that applying our counterintuitive instruction manipulation leads to significantly de- graded performance. Especially on Step-EM, per- formance in most settings is 10 20%, which is much worse than with normal instructions (> 20%. or even > 30% in most cases) This suggests that the model indeed to some extent relying on stan- dard language-to-loge mapping from pre-training, rather than fully understanding the instructions.
As we will see in the experiments below, when more interactions occur, the model will gradually lose track of states, propagate errors, and even generate hallucinations, despite all updates falling within the input window. By design, a super- capable AI agent like ChatGPT should have the ability to read and use all information within the input window-but our results suggest that this is not the case. Our research thus calls for further study, and for caution when implementing Con- stitutional Al approach.
Despite this inconsistency, the model still shows some success with the negation or the new rule with flipped truth values, it still manages achieve 70 80% State-EM. Though the State-EM values should be taken with caution, these accuracies are substantially stronger than would be expected from random guessing (50%) or completely ignoring the rule (0%), suggesting some level of capability in making use of the counterintuitive instructions.
4 Experiments: testing model sensitivity to instructions and situation change
We use our environment to test ChatGPT in its ability to track situation changes, and we report the results below. We also compare against the performance of additional chat models, which we find underperform ChatGPT. Those results can be found in Appendix E.
Effects of synthetic language Use of synthetic language affects model performance, but not al- ways in the predicted directions. Though perfor- mance is occasionally worse with synthetic lan- guage, it is more often better than with natural lan- guage. This suggests that the of synthetic lan- guage may at times help the models detach from unhelpful biases from pre-training, and rely more robustly on the in-context information. This fol- lows the analysis presented in Liang et al. (2022).
We try 2-shot, 3-shot and 5-shot settings (in which 2, 3 and 5 example steps with enumerated states are provided). We use 50 samples with the number of additional steps randomly sampled from (1,..., 10- - # (num-shots)} for each setting. We follow the official OpenAl 2ookbook2 to prepare the request and parse the response when interacting with ChatGPT API. Details are in Appendix C.
More in-context samples do not necessarily help. We also see that even if we are already providing answers for approximately 50% of states (consider- ing that we only have 10 boxes and 10 keys in the
Pnttps://gt thub. com/ com/open1///// openai- trenai-cookbook/ www.bookkbboooooooooooooooooookk ttte//reee/aaeeeeeee /main.

---

Step-EM/ / BMa
Normal Instruction
2-shot
3-shot
5-shot
NL Functor + NL Argument
22% 22%%/925 /92%
34%
36%/955%
Functor SL. Argument
19%/92'9
22%
52%/96%
Counter-Intuitive Instruction (On NL)
NL Functor NL Argument
00%%
6%%/75%
0%
SL Functor SL Argument
13%%/89%
20%/990%
12%/90%
Countcc-Intuitive Instruction (Truth Values Switching)
NL Functor NL Argument
6%%/72%
2% 69%
29%
SL Functor SL. Argument
19%/859%
1 14% 87%
12% 29%
Table 1: Experiment results ChatGPT Robustness check for state tracking in 10-box environment. Metrics here are presented in the format of "Step-EM State-FMM". We use 50 samples for each experiment setting.
environment, and each step will change exactly two states permanently within the dialog), the model does not make improvement in most cases.
steps. If performance after steps in this set- ting comparable performance after k k- steps in the previous setting, this will suggest that the degradation is indeed due to the complexity of the environment states. If, however, performance after steps in this setting is substantially better than performance at k + n steps in the prior analysis, this suggests that the degradation in performance is due to failure to handle the growing number of steps.
5 Analysis of model performance
In the previous section we tested the capacity of ChatGPT in tracking and enumerating states within our environment, and we found that the model showed clear limitations. this section, we an- alyze model performance further, to better under- stand the source of these limitations.
Step-EM / State-FM
Normal Initialization
5.1 Tracing Errors: State Tracking over Steps
2-shot
5-shot
NL Functor + NL Argument
22% / 99%% 36% 95%
To understand why ChatGPT shows relatively poor performance, straightforward way is reusing the instances created in Section 4, but querying for the environment states after each intermediate step to see where the errors emerge. Specifically, rather than only querying after the final step, we make queries at all steps (excluding "Step-0"), includ- ing those within the in-context example window (for querying each step in the in-context example window, in-context demonstrations of environment states are given only through step 1). We evalu- ate State-EN and Step-EM at every step. We refer this test as Intermediate State Probing.
St. Functor Sl. Argument
19% / 92%
52% / 966
Compressed Initialization
NL Functor + NL Argument
46% 99%
60% 99%
SL Functor SL Argument
48% / 95%
54% 997%
Table 2: Compressed Initialization Testing experiment results for 10-box environment on ChatGPT. Metrics here are shown in the format of "Step-EM/Sttte- EM"". We 50 samples with various number of steps for experiments.
The experiment results for the Intermediate State Probing and Compressed Initialization Test are shown in Figure 2 and Table 2, respectively. From these results make the following observations:
Potential confounder: state complexity When interpreting performance trajectory across increas- ing numbers of steps, a potential confounder is that performance may degrade simply because the set of environment states has become more complex, and not because there too many steps of updates involved. To investigate this possibility, we also run a test in which we compress and skip & of the early steps, and initialize in the state that would have resulted from those steps. We then test the trajectory of model performance on the subsequent
Degraded performance over steps. We see Figure that with increasing number of situational changes, both State-EM and Step-EM degrade. This degradation is particularly true for Step-EM, which decreases dramatically steps increase.
State complexity does not explain the degrada- tion. Additionally, we see in Table 2 that skipping steps and starting with more complex initialization leads to improved performance, indicating that the

---

Intermediate State Probing (2-shot)
Intermediate State Probing (5-shot)
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
In Context
Testing
Steps
In Context
Steps
Testing
Step-EM (SL)
Step-EM (NL)
Step-EM (SL)
Step-EM (NL)
State-EM (SL)
State-EM (NL)
State-EM (SL)
State-EM (NL)
(a) 2-shot
(b) S-shot
Figure 2: Results for Intermediate State Probing. Purple vertical lines indicate where in-context demonstrations (for steps prior to test step) end.
Dirty Read (DR)
Updates,
[Untouched] Error State
Accidentally Correct (HU-AC)
Error Propagation (EP)
T-1
Dirty (DW)
Void Action (MA)
Error T-1
Update
Fallowing Updates (NFU)
Updates
Correct State
Update at Step
Correct (CU)
Halluzinated Updates,
Incorrect Outcome (HU-H)
Maintain Correctness (MC)
(Untouched) Correst State at T-1
States at T-1
States at T
Category ID
Figure 3: Diagram for fine- ana-nneined error analysis
degradation across steps is not attributable to state complexity alone.
in tracking state changes- but only temporarily or over a limited number of steps.
More in-context demonstration mitigates degra- dation. In Section 4 we found that providing more in-context samples does not cause sig- nificant improvement in the performance of the model the final Test Query. With intermediate probing we gain finer-graines picture of the im- pacts of number of in-context samples across steps. First, we see that when prompting with more in- context samples (5-shot vs 2-shot), there is less rapid degradation in model accuracy after the end of the in-context window. In the 2-shot (Figure 2a) case, Step-3 (the second step preceded by non- demonstration step), Step-EM quickly drops from 60% 40%. By contrast, in 5-shot (Figuree 2b), Step-6, ChatGPT can still maintain Step-EM value of 60%. This suggests that having more in- context samples does strengthen models' accuracy
Though we see that having more demonstra- tions can mitigate degradation after the demon- strations end, when look within the in-context sample window itself, we see that on steps that directly follow in-context demonstrations (Step-1 for 2-shot, Step-1,2,3,4 for 5-shot), the model's 's performance does not monotonically increase response to the accumulating demonstrations. A similar phenomenon is also discovered in other few- shot reasoning benchmark works (Suzgun et al., 2022: Liang et al., 2022), despite the fact that in traditional fine-tuning, usually, more training in- stances yield better generalization ability. This suggests that although adding more demonstrations can briefly mitigate loss of accuracy, it does not straightforwardly translate to gains in accuracy.

---

#MMaintai Coreect States)
Analysis for Incorrect Predictions
Analysis for Correct Predictions
2.0
17.5
T400 occurrence
Type occurrence
1.5
1.5
17.0a
1.0
1.0
16.5
16.00
0.5
0.5
0.0
0.0
Steps
Steps
DR
DW
NFU
+ HU (10)
HU (AC)
VA
CU
MC
(a) Analysis for Incorrect outcome, 2shot, SL
(b) Analysis for Correct outcome, 2shot, SL
#(Mainta Coredd Beliel States)
Analysis for Incorrect Predictions
Analysis for Correct Predictions
2.0
Type occurrence
Type occurrence
1.5
1.5
1.0
1.0
0.5
14
0.0
Steps
Steps
DR
DW
NFU
HU (10)
HU (AC)
VA
CU
MC
(c) Analysis for Incorrect outcome, 2shot, SL Counter- (d) Analysis for Correct outcome, 2shot, SL Counter-Intuitive Intuitive Instruction (On NL)
Instruction (On NL)
Figure Fine-Grained Error Analysis for Logical Inconsistency
Interim discussion These patterns of degrada- tion over time occur, in both NL and SL settings, despite the fact that ChatGPT read the full dialogue in its input window. This suggests that ChatGPT cannot effectively utilize the full infor- mation its input window, and that claims about maximum input length capabilities (e.g.. ChatGPT can model 4k tokens as introduced the official announcement (OpenAI, 2022)) should be taken with grain of salt.
The experiment results are shown in Figure 4 (we only show 2-shot SL, for reasons of space).
Examining first the patterns for states which models make incorrect predictions, we see that the rise errors driven by states that should be untouched at that step. We see that as steps in- crease there are rapid increases in both Dirty Read (DR) transitions, where models retain previous ror, and Hallucinated Update (HU-IO) transitions, where models change state from correct incor- rect despite there being no change to that state the step description. These patterns indicate that the rise in errors over time can be attributed both to retention and propagation of errors from previous steps, but also to failures in retaining the memory of a previous step that should not change.
5.2 Fine-grained analysis of update patterns
In the above section, studied the trajectory of model performance as the number of steps in- creases, finding evidence that ChatGPT degrades state tracking with increased number of steps. In this section, we do a finer-grained analysis of the update dynamics in these experiments, in order to examine more closely the causal factors leading to both erroneous and accurate predictions. For the purpose of this analysis, define categories of state transitions, summarized in Figure 3. These categories allow us analyze the relationships between states at the analyzed step and the corre- sponding prior state, both when updates should be made to those states and when they should not.
Examining now the transitions associated with correct model predictions, we see that over time there is noteworthy decrease in Correct Update (CU) cases however, there is much more dra- matic decrease in Maintain Correctness (MC) cases, indicating that the model increasingly fails retain the memory of previously correct states. Over time we also see, particularly in the case of counterin- tuitive instructions, a rise Accidentally Correct (HU-AC) cases, which the model switches from

---

incorrect state back to the correct state, despite the fact that no update that state was described in the step. Both of these patterns are indicative of memory limitations and susceptibility to random noise in changes to enumerated states.
hallucinated updates. These findings indicate over- all that ChatGPT does not have robust situational state tracking ability.
Our proposed synthetic environment and the find- ings that it generates can have noteworthy real- world implications. First, it can diagnose the po- tential limitations of current chatbot systems in multi-round interactions. Second, reflect potential problem for the model's ability to follow instructions and remain consistent with rules/norms established in its context, which es- pecially important for responsible Al safety and human-AI alignment research.
These results yield several conclusions:
ChatGPT has non-persistent in-context mem- ory. A recurring observation above is that many of the model errors that increase over time can be attributed to limitations retaining states in memory-and in fact, some states marked as cor- rect also reflect accidental correctness arising due to similar failures to retain prior states.
findings also
A Acknowledgements
States can also be retained, but potentially by chance. In addition to memory retention failures, we also see propagation of errors between steps- which in theory indicative of successful memory retention, by contrast to the retention failures cited above. However, considering the prevalence of hallucinated updates, and the limited options for state values, we can expect that at least some of these retained updates in fact occur by chance.
We are grateful for the insightful discussion with Chih-chan Tien (UChicago), Kanishika Misra (UT Austin), Hao Zhu (CMU) and Zhaofeng Wu (MIT) at the early stage of this work (names not listed in particular order). We also thank the anonymous EMNLP reviewers and chairs for providing insight- ful and constructive feedback make this work more solid.
Couterintuitive instruction exacerbates non- robust behavior. As we mentioned above, the drop in correct CU updates is more dramatic- and the rise in spurious correct updates HU more substantial- in the case of counterintuitive instruc- tions. This suggests that the inability of the model to rely on memorized language-to-logtt mappings generally reduces the model's ability to execute and maintain correct state updates.
Limitations
In this work, we propose a controlled synthetic environment to investigate ChatGPT's situational understanding ability. While we believe our syn- thetic environment has important real-world impli- cations, as discussed in Section 3.4, for certain real-world applications our findings may not apply. Another limitation is that we only focus on the eval- uation of ChatGPT as the state-of-the-ie chatbot model (at least in terms of mainstream media cov- erage). There are other commercial chatbot mod- els that could show stronger performance on our tasks, as they may have more complicated system designs (e.g., multi-module systems in Blender- Bot (Shuster et al., 2022)) that could be better at dealing with multi-round dialogue history and extremely long inputs. As we do have sufficient budget or open access to test many such systems, we leave a comprchensive benchmark evaluation of situational understanding ability for Chat-Tuned LLMss for future work. Our experiments closely fol- low the OpenAl official cookbook for interacting with ChatGPT, but it is possible that there could be more optimal prompts to fully unlock the capability of ChatGPT.
6 Conclusion
In this paper, we propose novel synthetic test- ing environment for testing situational understand- ing capabilities, which apply to test ChatGPT, the state-of-the-art chatbot. We instruct ChatGPT process series of sparse environment updates across time a dialogue history. With careful en- vironmental designs, we reduce the possibility of data contamination and other artifacts typically in- troduced by traditional probing methods. We find that despite the simplicity of our task, and even with ChatGPT having full access to the complete dia- logue history within the input window, the model fails retain coherent and correct environment states over time. Further analysis suggests that this failure is largely because ChatGPT does not have persistent in-context memory, and susceptible to
There are many other synthetic environments like TextWorld (Côté et al., 2019) that may be pro-

---

grammed to do situational testing as in our work (though it may not be easy to assert full controls), and is would be interesting to establish whether in different environments we can still draw the same conclusions. Our work mainly focuses on our pro- posed environment as a case study, but we plan to extend our testing framework to other environ- ments in the future.
Matthew Hausknecht, Layla Asti, Mahmoud Adada, et al. 2019. Textworld: A learning environ- ment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial In- telligence, ICAI 2018, Slockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41-75. Springer.
Heim and A. Kratzer. 1998. Semantics in Genera- live Grammar. Blackwell Textbooks in Linguistics. Wiley.
References
Matthew Henderson. Blaise Thomson, and Jason D. Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meet- ing of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272, Philadelphia. PA, U.S.A. Association for Computational Linguis- tics.
Jacob Andreas. 2022. Language models as agent mod- els. In Findings the Association for Computational Linguistics: EMNLP 2022, pages 5769-5779, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et 2022. Constitutional Al: Harmlessness from AI feedback. ArXiv preprint, abs/2212.08073
Md Mosharaf Hossain, Antonios Anastasopoulos, Ed- uardo Blanco, and Alexis Palmer. 2020. It's not non-issue: Negation as source of error in ma- chine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2869-3885, Online. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Matcusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, lla Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neurul Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Hans Kamp, Josef Van Genabith, and Uwe Reyle. 2011. Discourse representation theory. Handbook of Philo- sophical Logic: Volume 15, pages 125-394.
Lauri Karttunen. 1969. Discourse referents. In Inter- national Conference on Computational Linguistics COLING 1969: Preprint No. 69: Collection of Ab- stracts of Pupers, Sånga Säby, Sweden.
Najoung Kim and Sebastian Schuster. 2023. Entity
tracking in language models. In Proc. of ACL. Lo- cation of the conference. Association for Computa- tional Linguistics.
Henry Kučera and Winthrop Nelson Francis. 1967. Computational analysis of drasent-day American En- glish. Brown university press.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chathot impressing GPT-4 with 90%* Chat- GPT quality.
David Lewis. 1976. General semantics. In Montague grammar, pages 1-50. Elsevier.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zeutlemoyer. 2020. BART: Denoising
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint. abs/2204.02311.
to-suquee--eseeeeeeeeeeeeeeeeeee pre-training for natural language generation, translation, and com- prehension. In Proc. of ACL, pages 7871-7880 On- line. Association for Computational Linguisties
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language models.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning neural lan- guage models. In Proc. of ACI., pages 1813-1827, Online. Association for Computational Linguistics.
2022. Scaling instruction-finete ArXiv preprint, abs/2210.1 11416..
Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2022. Language modeling with latent situations. ArXiv preprint, abs/2212. 10012.
Mlee-Alex Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore,

---

Percy Liang, Rishi Bommasani, Tony Lee. Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. ArXiv preprini, abs/2211.09110
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Se- bastian Gehrmann, Yi Tay. Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le. Ed H Chi, Denny Zhou, al. 2022. Challenging BIG-Bench tasks and whether thain-of-thoughe can solve them. ArXiv preprint, abs/22 00.09261.
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proc. of ACL, pages 1456- 1465, Berlin, Germany, Association for Computa- tional Linguistics.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. 2022. Chess as testbed for language model state tracking. In Thirty-Sixhh AAAI Conference Artificial Intelligence, AAAI, 2022, Thirty- Foorth Conference on Innovative Applications of Artificial, Intelligence, IAAI 2022, The Twelveth Symposium Educational Advances, in Artificial Intelligence, EAAI 2022 Virtall Event, February 22, March I, 2022, pages 11385-11393. AAAI Press.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- ties in natural language inference. In Proc. ACL. pages 3428 3448, Florence, Italy, Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. ArXiv preprint, abs/2302.1397/
OpenAI. 2022. Introducing ChatGPT. Accessed: 2023- 04-24.
OpenAI. 2023. GPT-4 technical report. ArXiv preprint, abs/2303.08774.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models unsupervised multitask learners.
Frank Veltman. 1996. Defaults in update semantics. Journal of philosophical logic. 25(3):21-2611.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn, Res., 21:140:1-140:67
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016. Towards AI-Complete ques- tion answering: set prerequisite toy tasks. Proc. of ICLR.
Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachan- dran. 2014. The dialog state tracking challenge series. Al Magazine, 35(4):121-15
Abhilasha Ravichander, Matt Gardner, and Ana Maraso- vic. 2022. CONDAQA: contrastive reading com- prehension dataset for reasoning about negation. In Proc. of EMNLP, pages 8729-8755, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini- Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transfcrable multi-domain state gener- ator for task mask-oriented dialogue systems. In Proc. of ACI., pages 808-819, Florence, Italy. Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behav- ioral testing of NL.P models with CheckList. In Proc. of ACI., pages 4902-4912, Online. Association for Computational Linguistics.
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. 2023. Judging LLM-as- aal--ss--jjuee assaa- with MT--eench and chatbot arena.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Fdd H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pages 11210-31227. PMLR.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, al. 2022. Blenderbot 3: A deployed conversational agent that continually learns to responsibly engage. ArXiv preprint, abs/2208.03188.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang. Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. 2023. Principle-driven self- alignment of language models from scratch with minimal human supervision. ArXiv preprint, abs/2305.03047.

---

A Distractor on Action
B Partial Usage of Synthetic Languages
We can do more fine-gratd fine-grained usage of synthetic langauges can choose to apply only func- tors (e.g.. "Opened") or arguments (e.g.. "BOX-1"), The full set of experiment results is shown in Ta- ble 4. We notice that the results are very similar to the full usage of synthetic languages so to save space, we move the results in Appendix.
addition to instruction understanding, the model should also learn to correctly understand and select key information every time new step is taken. To examine the robustness of this capability, in addi- tion to the inclusion of normal step descriptions, we also include distractor conditions (Shi al.. 2023), which add one randomly selected sen- tence from the Brown Corpus (Kučera and Francis, 1967) as distractor. An example of applying such distractor shown in Example 5. The experiment results are shown in Table 3.
C Interaction with OpenAl API and Prompt Format
According to the official OpenAl cookbook, to send requests to ChatGPT, it is advised to first add "system message" to ChatGPT (We use "You are helpful assistant" as this is one of the most used system messages). Then there two typical ways send the prompt, it contains in-context samples:
1s| Step-1: Open jqC-3 and zetrieve baS-2. It is nice day!
14] Question: vv
Example 5: Action Distractor Example
Instruction
Step-EM/ State-EM
Traditional Format Just put your prompt contents one-round interaction (as in prompt- ing GPT-3-Davinci series models), as shown Example 6:
2-shot
5-ahot
NI. Functor NI. Argument
22% 992%
36% / 95%
Synthetic Functor Synthetic Argument
19% 92%
52% / 96%
Actions w/ Distractor
NL. Functor + NI. Argument
18% 91%
44% 96%
Is response = openai. paatComplation.craatt model=....
Synthetic Funetor Synthetic Argument
34%
messages=[
("role": "system", "content": "You are helpful assistant."),
Table 3: Action understanding experiment results for 10- box environment on ChatGPT. Metrics here are shown the format of "Step-EM/State-EMM ples with various number of steps for experiments.
("role": "user", "content"; "[
use 50 sam-
Instruction + In-Context Samples] "}, },
From Table can after adding distrac- tors, the State-EM performance does not degrade as much as in counter-intuitive instructions, though Step-EM performance degrades a bit (but not con- sistently). These findings hold both in NL and synthetic languages. This suggests that ChatGPT does have the ability to understand the interaction happened at each step and can pick out useful in- formation.
Example 6: Traditional Input Format
Faked Multi-Round Format Another way is to synthesize fake multi-rount conversation and pre- tend the answers for in-context samples are gener- ated by ChatGPT, as shown in Example 7: response openai ChatCompletion. create( model=....
mesaages-
("role": "system", "content": "You are helpful assistant. "}, ("role": "user", "content"; "[
We also see that when adding distractors on ac- tions, within the 2-shot condition, Step-EM in syn- thetic language environment is better than the one in NL environment. This is probably because the usage of synthetic language helps the model better distinguish useful information as they look very different from synthetic languages. However, when there are more in-context samples, the model will gradually learn extract useful information each step and the usage of synthetic language does not help that much.
Instruction]
("role": "user", "content"; "[
29 Sample-I-Input]%;
("role": "assistant", "content": "[ Sample-1-GrromdTTeuhh Ansuer]
3 hntpps///atthuubccc//ommo wttp://///waaaieeee//eeeeeeeeeee operai- pagrraaicooobboot//bb///mmaaaaaa to Se_ffrmat_inppts_teee ofte
ipynb
4 "A of the time the project is initialized. OpenAl may change hit in the documentation and examples in the GitHuh repository.

---

Step-EM State-BM
Normal Instruction
2-shot
3-shot
5-shot
NL Functor + NL Argument
22% / 992%
34%
36%/955%
Synthetic Functor + NI. Argument
49%%
20%//93%
30%/994%
NL Functor Synthetic Argument
/95%
30%//4%
74%/99%
Synthetic Functer + Synthetic Argument
19% /92%
22%//93%
52%
Counter-Intuitive Instruction (On NL)
NL Functor + NL Argument
10%
6% /75%
0%%88%%
Synthetic Functor NI. Argument
18%/88
10% 88%
14%, 990%
NI. Functor Synthetic Argument
20%/881%
10% 18%%
88%
Synthetic Functor + Synthetic Argument
13%/89%
20%
12% 990%
Counter-Intuitive Instruction (Truth Values Switching)
Functor + NL Argument
6%%/72%
2% 69%
2%%/79%
Synthetic Functor NL Argument
10%
10% 84%
68%
NL Functor Synthetic Argument
8%/71%
8%/73%
0%/83%% 83%
Synthetic Functor + Synthetic Argument
19%%
14%/87%
12% 789%
Table 4: Experiment results on ChatGPT Robustness check for state tracking in 10-box environment. Metrics here are presented the format of "Step-EM/ State-EM". We use samples with various number steps for experiments.
30
("role": "user", "content"; "[
Normal Instraction
State State-MM, 2-shor
Mumple--IInutt]""
ChaiGPT
Text-Davinci-OUF
31
("role": "assistant", "content"; "[ Sample-2 Trutt-----ooo-rr---aaaaa
Arguneet
9/%
96%
Synthetic Functor NL Argument
90%
94%
32
NL Functor Synthetic Aggament
95%
94%
35
1.
Systhetic Functor Synthetic Arguneent
52%
88%
34
35
Table 5: Experiment Results Comparing ChatGPT and Davinci-003. Metries are State-EM. We use samples with 2-shot setting and various number of steps for experiments.
Example 7: Faked Multi-Round Input Format
D Post-Processing to Extract Answers from OpenAl API Responses
We do a prior study on experimenting with these two formats and find they give pretty similar Step- EM and -tate-EM performance in fully NL and fully Synthetic Language settings (usually the per- formance difference is less than 5%).. But faked multi-round format would insert many more tokens (leading higher cost) in the requests and can make the OpenAl server reject respond, leading to low response rates. Therefore, to save budgets, in the main text of the paper, we only report the results using the traditional format. Another reason is that to see how much our findings on ChatGPT can generalize to other models, also replicate some of our experiments when comparing with other models in Appendix E. Some models there may not support faked multi-round input format.
We note that ChatGPT return answers not strictly follow the given format. So we do light- weight postprocessing mainly using regular expres- sions to make sure we parse ChatGPT results appro- priately. Specifically, we first clean out the space and newline characters at the beginning and end of the answer. We then use the following regular ex- pression match all logical statements and extract groups of functors, arguments and truth values: ([a-zA-Z0-9]] + N [[a-zA-Zo-a -- True Srueltruel False (false)
E Comparison with other models
To clarify whether our observations can be gen- eralized other popular models, choose strong open-sourced competitor chatbot (according to ChatArena (Zheng et al., 2023)): Vicuna- 13bb and we also compare against the performance of GPT-3-davinci-005 (it is believed that ChatGPT is a variant fine-tuned over GPT-3).
For response parsing, just follow the instruc- tion parse the JSON-style response described in the same notebook and obtain the model output. For decoding parameters, we just follow default OpenAl API setting.

---

Comparison between Vicuna-13B On ChatArena (Zheng et al., 2023), Vicuna- 13B (Chiang et al., 2023) is voted to be the most close-to- Pose-to-ChatGPY chat models based on LLAMA (Touvron et al., 2023). We feed the same input to Vicuna and find the outputs are hard to parse and often incomplete. Even when there are no counter-intuitive instructions and we give in-context samples, the best observed State-EM (in SL enviromment) only 32%. Compared with ChatGPT (94%), there seems big gap on the capability situational understanding for open-source models.
Comparison between GPT-3.55 and ChatGPT Due to the budget limit and context length limit of GPT-3.5, we only compare ChatGPT performance with GPT-3.5 (Davinci-003) on 2-shot. The experi- ment results are in Table 5. We can see Davinci-003 achieves a similar performance as ChatGPT.


Reviews:
[{"review_id": "QIwHsM6sov", "reviewer": "Reviewer_xaGB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful comments from the reviewer. We have responded to the issues presented below.\n\n**Re: Conversation History Forgetting, Input Length Mismatch from Training Time**\n\nThanks for pointing this out. We agree that the length extrapolation issue (i.e., that performance degrades when testing on inputs of longer length compared with the training phase) is a well-known problem for LLMs. However, in this paper, we focus on situations in which test inputs are of the same length as those the model is trained to receive. We make special efforts to control input length *not* to be a confounder in our experiments. \n\nFirst, with regard to the input length that the model can handle, according to the OpenAI ChatGPT [official API](https://platform.openai.com/docs/guides/gpt/managing-tokens) and [most recent release page](https://openai.com/blog/function-calling-and-other-api-updates), for the current ChatGPT version (“gpt-3.5-turbo”), 4k context length is within the model capability and the model should be able to handle it properly. To avoid the confounder effect of length extrapolation, we carefully followed [OpenAI CookBook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb) to count tokens using the `tiktoken` package, and we also controlled the length of our inputs (including our instructions – note that our instructions are much simpler compared with prompts in InstructGPT [1], which can easily go beyond 1k tokens) plus our expected outputs to ensure that we avoid going beyond the recommended context length 4k (e.g., via controlling the number of rounds we prompt ChatGPT). \n\nHowever, we find that even within the designated 4k input length, where the length extrapolation problem should not apply, we clearly find that ChatGPT will gradually lose track of situational changes and maintain broken internal states. We believe this is a substantive new finding that is different from previous works, and users should be made aware of this risk. (A contemporary work is “Lost in the Middle” [2], which was released after our EMNLP submission and takes a different approach to study a similar problem that we do, while focusing more on positional bias). \n\n**Re: The paper does not explore why the model has these limitations and lacks an in-depth explanation and exploration of these situations.**\n\nWe respectfully disagree with the criticism that the paper lacks explanation and exploration of the situational understanding problems. First, in Section 5.1, we do interactive debugging to prompt each intermediate step for failure situations, and we find out 1) the model performance degrades over steps; 2) state complexity does not explain performance degradation; 3) more in-context samples help mitigate the performance drop problem in muti-round interactions. Then, in Section 5.2, we do a fine-grained error analysis to further illustrate the problem of error propagation, failure to maintain consistency, and the increasing risk of hallucination to gain better performance numbers. We believe that the insights from these explorations can help both in informing users about possible reasons for an undesired chat experience, and also ultimately in helping to build a more controllable, interactive, and trustworthy chat model in the future, by helping to zero in on the circumstances and characteristics of model failures. \n\n**Re: Confounders in our synthetic environmental tests**\n\nWe appreciate the reviewer’s attention to this concern. We fully agree with the importance of controlling for confounders, and we have carefully designed our experiments to exert this control to the best of our ability. A non-exhaustive list of confounders we have addressed includes 1) as stated in the first item in this response, we carefully control the input length to avoid the length extrapolation problem; 2) we do a natural language - synthetic language comparison to reduce data contamination problems; 3) we do a counterfactual verification (Section 5.1) to examine whether state complexity would interfere with the model performance drop instead of solely multi-round interactions. These careful experimental designs are part of the main value of our work. \n\nIf the reviewer could specify any particular confounding variables they believe were not controlled in our experiments, we would greatly appreciate the additional insight.\n\n\n**Re: The research focus of this paper is mainly on the ChatGPT model, and there is a lack of evaluation of other chat models, so the discussion on the generalization ability of the overall situational understanding ability of the chat model is limited.**\n\nAs stated in Appendix E, we also compare with Vicuna-13B [3], which was, at our submission time, an open-sourced state-of-the-art (in terms of ChatArena Benchmark [4]) chat model via fine-tuning on LLAMA [5]. Our experiment results reveal that this model has more significant problems in understanding situational changes compared with ChatGPT, even in simplified environmental settings (e.g., in a 5-state environment, where ChatGPT can easily achieve 94% State-EM but Vicuna only achieves 32% and can produce very hard-to-understand outputs). We also compare with the non-chat model OpenAI Davinci-003 by giving the same prompts and we find the performance number is similar to ChatGPT. Therefore, we focus our evaluation mainly on ChatGPT to expose the situational understanding limitation for the strongest state-of-the-art Chat model. We recognize that there also exist commercial model-based Chat systems that may be able to use memory modules to handle our tasks, but as we do not have access to APIs for these models, we leave the benchmarking on these models for future work, and make a note about this in the Limitations section. We believe that our environmental tests are general enough for testing any chat models, laying a strong foundation for future comprehensive benchmarking works.\n\n**Re: The paper does not provide specific suggestions and methods on how to solve or improve the situational understanding of the chat model.**\n\nIn this paper, our main focus is to expose these situational understanding issues in state-of-the-art chat models (i.e., ChatGPT), and to provide thorough in-depth analysis, providing insight and explanations with respect to model failures. We do provide some insights into how one might solve or improve the situational understanding of the chat model (e.g., via giving more in-context samples, or perhaps adding a memory module could help as our task can be solved by string automata + lookup tables), but proposing major methodological innovation goes beyond the scope of the paper. We believe, however, that our findings and evaluation can help model developers better design a more capable chat model with improved situational understanding ability, by assisting in pinpointing where and how these types of model failures are arising. \n\nReferences:\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\n\n[2] Liu, Nelson F., et al. \"Lost in the middle: How language models use long contexts.\" arXiv preprint arXiv:2307.03172 (2023).\n\n[3] Chiang, Wei-Lin, et al. \"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.\" See https://vicuna. lmsys. org (accessed 14 April 2023) (2023).\n\n[4] Zheng, Lianmin, et al. \"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.\" arXiv preprint arXiv:2306.05685 (2023).\n\n[5] Touvron, Hugo, et al. \"Llama: Open and efficient foundation language models.\" arXiv preprint arXiv:2302.13971 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper reveals the limitations of natural language models in understanding situational changes and becoming human-like agents to accomplish real-world tasks; There is a discussion about the risk of trust in ChatGPT's impressive conversation performance; The findings have potential implications for teaching highly effective agents to follow social norms and policies.", "reasons_to_reject": "In my opinion, the contribution of this paper is quite incremental:\n\n0. LLMs gradually forget the conversation history, limited on the one hand by the length of its hard prompt and on the other by the length of the window used in its pre-training. This is what we already know, so the idea that comes out of the paper doesn't seem interesting.\n\n1. The paper does not explore why the model has these limitations, and lacks in-depth explanation and exploration of these situations.\n\n2. The synthetic environmental tests mentioned in the paper can be interfered with in some cases, and there are some potential confounding factors that could lead to conclusions.\n\n3. The research focus of this paper is mainly on the ChatGPT model, and there is a lack of evaluation of other chat models, so the discussion on the generalization ability of the overall situational understanding ability of the chat model is limited.\n\n4. The paper does not provide specific suggestions and methods on how to solve or improve the situational understanding of the chat model.", "questions_to_authors": "", "paper_topic_and_main_contributions": "### Topic\nThis paper mainly focuses on the ability to situational understanding of chatbots, mainly ChatGPT. The paper proposes a novel synthetic environment for controlled testing of models’ ability to track and enumerate environment states, which reflects an agent's capacity to ground language in situations and comprehend situational changes. The research sheds light on the limitations that natural language models face, particularly in understanding situational changes and serving as human-like agents to accomplish real-world tasks. \n\n### Contributions\nThe authors state that ChatGPT fails to retain prior states in memory and has susceptibility to spurious hallucinated updates, and concluded that chatGPT cannot currently track environmental changes.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c7PBPYIHrS", "reviewer": "Reviewer_rJkr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're grateful for the constructive feedback provided by the reviewer. In the following section, we address the mentioned concerns and questions in turn.\n\n**Re: Missing Implementation Details**\n\nThanks for raising this concern. For designing step instructions for each of the 50 samples, we sample the number of steps randomly from a uniform distribution U(1, 10). We then keep instructions almost entirely the same across samples, except that in the Synthetic Language settings the state predicates are defined randomly, and therefore vary between samples (e.g., in one instance we use “NvSWxzvJb(jqC-0)=True“ to refer to the box jqC-0 has been opened (“NvSWxzvJb”-ed), while in the other instance, we use “Abc(bb-0)=True” to represent the box bb-0 has been opened (“Abc”-ed) ). We will add these clarifications in the revised version. \n\n**Re: Reproduction Issues on the release of all prompts**\n\nWe will release all prompts and API responses from ChatGPT for reproduction in the formal version. We will also release the environment building codes for users to build different testing environments and generate test examples. \n\n**Re: The difficulty of the task, especially given two different evaluation metrics, is not clear to me. The authors compare ChatGPT with other chatbots. But how does it compare to the fine-tuned models? If large language models such as T5 were trained on this task, will they perform better or worse than ChatGPT?**\n\n**Regarding Task difficulty**: In Section 3.3, we briefly discuss the task difficulty and the design consideration in the “Existence of Shortcuts” paragraph: we believe this task is in principle very simple, as long as the tested agent has the fundamental situational understanding ability – note that we only need a string automaton and a state-tracking table to achieve 100% EM on both metrics. However, we carefully design our testing environment and control for confounders, so that the task is *not* simple for models that lack this situational understanding ability – in particular, we work to avoid models being able to solve the task via easier, shallower solutions that don’t require the target situational understanding ability. In this way, our tests allow us to draw rigorous conclusions about situational understanding capabilities.\n\n**Regarding Comparison against fine-tuned T5**: We do expect that we could see improved performance by actively fine-tuning a model (like T5) to do our task, but this would be somewhat orthogonal to our purpose in this submission, which is to test the existing situational understanding capabilities of these state-of-the-art chat models out of the box. An important component of our test is that it taps into models’ fundamental situational understanding capabilities as they are, by providing a small number of demonstrations of the task format, but otherwise not doing any fine-tuning of the model to map between these specific inputs and outputs. \n\nAs for a scenario in which we compare against T5 in the few-shot setting that our test is designed for: we do not have results for this comparison on hand, but we speculate that because T5 does not go through the RLHF process, it may be difficult for T5 models to follow instructions and handle multi-round interactions in our environment – so we hypothesize that T5 will perform much worse than ChatGPT on our task as designed. \n\nAs a final note, in Appendix E we also compare ChatGPT with Davinci-003, which is reported to have the same base model as ChatGPT but is optimized more using multi-task instruction tuning, similar to T5 (See [OpenAI official Model Index](https://platform.openai.com/docs/model-index-for-researchers) for reference). We find that this model behaves similarly to ChatGPT, though at a much higher cost. So we focus on the evaluation of ChatGPT in this paper, but we ensure that it is straightforward to extend our evaluation to T5 or other fine-tuned models, as desired, since our evaluation is agnostic as to the model being prompted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper is well-written and properly organized.\n\n2) The paper introduces a novel synthetic environment for testing situational understanding, which provides a good perspective for evaluating large language models.\n\n3) The fine-grained error analysis provides a great understanding of the dynamics of the model performance, illustrating how correct or wrong transitions were made. It is particularly interesting to see the accidental correctness goes up as steps increase.", "reasons_to_reject": "1) Some implementation details of the synthetic environment are missing. For example, how do you design the step instructions for each of the 50 samples? Do they come from random permutations?\n\n2) Reproducibility might be an issue. Are you going to release all your prompts so others can replicate the whole environment and results?\n\n3) The difficulty of the task, especially given two different evaluation metrics, is not clear to me. The authors compare ChatGPT with other chatbots. But how does it compare to the fine-tuned models? If large language models such as T5 were trained on this task, will they perform better or worse than ChatGPT?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a simple synthetic testing environment for testing the ability of situational understanding (SU) of large language models. Specifically, they use ChatGPT as a case study. The authors propose a synthetic box-moving environment like TextGame (Li et al., 2021) to assess the models’ ability to track and enumerate environment states and analyze the causes for performance patterns. They find that ChatGPT struggles to retain coherent environment states over time, mainly due to non-persistent in-context memory and susceptibility to hallucinated updates. The authors also conduct a fine-grained analysis of the updated dynamics to illustrate the causal factors of error predictions. The paper contributes to the understanding of the limitations of the current ChatGPT model and highlights the need for improved situation tracking abilities.\n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I suggest using 10 instead of 5 in Example 1 to avoid confusion.\n"}, "extra_text": {}}, {"review_id": "XePanGayAS", "reviewer": "Reviewer_q8HG", "summary": "", "questions": "A: I do not fully understand the difference between the two evaluation metrics, Step-EM and State-EM, could you please give an example?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. Below we address the concern and question raised.\n\n**Re: Code Release**\n\nWe will release all experimental code in the formal version to help others build the test environment and generate test examples. \n\n**Re: Two Metrics Design**\n\nThank you for pointing out this confusion. We will clarify here, and we will add clarification in the revised version as well. Basically, State-EM assesses the percentage of states within a step that the model gets correct. So for this measure, at a given step, the model gets credit for all states that it got correct, and it is penalized for all states that it missed. By contrast, Step-EM evaluates the model’s ability to maintain a fully consistent and correct situational state – so for this measure, at a given step, the model only gets credit if it gets all states in that step correct. If any states are incorrect, the model does not get credit for that step. \n\nTo illustrate now with an example, consider the following made-up case:\n\n[Instructions and some previous steps]\n\nQuestion: NvSWxzvJb(jqC-0)=? B(bsS-0)=?\n\nCorrect answer: NvSWxzvJb(jqC-0)=True, B(bsS-0)=False\n\nModel Answer: NvSWxzvJb(jqC-0)=True, B(bsS-0)=True\n\nState-EM for this case: ½ = 50%\n\nStep-EM for this case: 0 (as there contains one error state)\n\n**Regarding Presentations, Typos and Styles**\n\nThank you for catching these! We will fix these in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The presented evaluation setup is carefully designed. The task is well chosen and argued for and measures are taken to minimize the tested models' reliance on common patterns. For examples, random string are used for state predicates. The task is both simple enough to test basic capabilities and complex enough to provide insight into a fundamental dialog capability.\n\n- The setup and results are discussed in detail, providing value to others who want to build on this work and understand the phenomenon.\n\n- The results are interesting and detailed, showing how correct and incorrect states happen at different stages of the tested dialogs.\n\n- The paper is well-written and structured. I have some comments in the presentation section below.", "reasons_to_reject": "- The paper does not mention whether the code would be made public in order for others to replicate and evaluate other LMs and extend the settings.", "questions_to_authors": "A: I do not fully understand the difference between the two evaluation metrics, Step-EM and State-EM, could you please give an example?", "paper_topic_and_main_contributions": "The paper describes an evaluation setup for LLMs that tests their ability to keep track of situational changes in a dialog. The LLMs are tasked to keep track of objects whose state can change with certain actions at each step and is then asked to present the new set of states. The paper describes the setup in detail and presents detailed discussions of the evaluation run with ChatGPT, showing its shortcomings in tracking dialog states.\n\nThe main contributions are\n- the design and description of the evaluation setup that can be applied to any language or dialog model\n- the application of the setup to ChatGPT\n- the analysis and discussion of the results that show that the model has trouble to remain consistent over time", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 1: I find the tight format of the numbers hard to read. You seem to have enough space to add spaces around the slash. Right-aligning the numbers makes the single digits stand out more.\n\n- Section 4.1, Failures on Step-EM: I think you mean that the results are 70 percentage points lower, not 70% lower\n\n- Section 4.1, Effects of synth langugae: \"better then\" -> \"better than\""}}]

Author rebuttal:

Meta-review: The paper at pointer delves into situational grasp within LLMs, emphasizing the ChatGPT model. Critical facets of the study include:

- The introduction of a synthetic environment akin to TextGame (as cited in one of the reviews), aimed at examining models' capability to chronicle and enumerate environment states.

- An extensive evaluation of ChatGPT's limitations in maintaining consistent environment states over extended periods attributed to transient in-context memory and proneness to hallucinated updates.

- Detailed error analyses reveal how LLMs transition from one state to another, encompassing correct and incorrect state transitions.

**Reasons to Accept**

- The paper uncovers the inherent limitations of LLMs, particularly ChatGPT, in tracking situational changes, emphasizing the challenges of LLMs in emulating human-like contextual comprehension and interactions.
- The synthetic environment introduced offers a novel paradigm to evaluate the situational understanding of LLMs. This environment sheds light on the intricate dynamics governing model performance, presenting invaluable insights into the nuances of state transitions.
- An in-depth discussion, spanning the risks associated with an unnecessary dependency on ChatGPT's conversational prowess to the broader importance of designing agents that align with societal norms and policies, lends a holistic perspective to the paper.

**Reasons to Reject**

- Reproducibility Concerns: The paper needs to include essential implementation details of the synthetic environment. Furthermore, there needs to be more clarity about the release of necessary prompts and the step instructions' design, making it challenging for others to replicate the results and potentially question the study's validity.
- Limited Scope: The research predominantly concentrates on ChatGPT, neglecting a broader examination of other chat models. This limited focus restricts the paper's applicability and generalizability in the broader context of chatbot situational understanding.
- Evaluation Metrics Ambiguity: The difference between specific evaluation metrics, specifically Step-EM and State-EM, must be articulated, leading to potential misunderstandings about the study's conclusions.
Decision: Accept-Main