A Turing Test for Self-Awareness

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:42:14

Abstract: I propose a test for machine self-awareness inspired by the Turing test. My test is simple, and it provides an objective, empirical metric to rectify the ungrounded speculation surging through industry, academia, and social media. Drawing from a breadth of philosophical literature, I argue the test captures the essence of self-awareness, rather than some postulated correlate or ancillary quality. To begin, the concept of self-awareness is clearly demarcated from related concepts like consciousness, agency, and free will. Next, I propose a model called the $\textit{Nesting Doll of Self-Awareness}$ and discuss its relevance for intelligent beings. Then, the test is presented in its full generality, applicable to any machine system. I show how to apply the test to Large Language Models and conduct experiments on popular open and closed source LLMs, obtaining reproducible results that suggest a lack of self-awareness. The implications of machine self-awareness are discussed in relation to questions about meaning and true understanding. Finally, some next steps are outlined for studying self-awareness in machines.

Main paper: 
 introduction: 
 At what point can we say a machine's eyes have been opened? When can we say it has become like us? After what moment can we say it knows good and evil? Such questions have met idle speculation for millennia, but today they rapidly approach a fever pitch, demanding answers with unprecedented urgency. AI systems that can pass for human in many respects are no longer fiction. Machines that can walk and talk are real and functional. What was once a distant speck, barely visible on the horizon, is now barreling down upon us.Through much of the history of AI, the Turing test served to keep these worries at bay [1] . Originally called the imitation game, this rudimentary metric of AI progress is a game played by two humans and one machine. One human engages in conversation with the machine and the other (the judge) must identify which is which, using nothing but the text of the conversation. The machine is deemed intelligent if it can fool the judge by mimicking human dialogue. While far from perfect, the Turing test was a concrete, unambiguous bar for AI to clear-and one that stayed comfortably out of reach for a long time.Last year, however, the Turing test was broken [2] . Large Language Models (LLMs) such as ChatGPT can handily engage in fluent conversation, on top of generating convincing essays, passing difficult exams, and even writing poetry. With the Turing test no longer a target in the distance, the conversation on AI has become untethered to any definitive, objective measure or permanent, agreed-upon benchmark. As such, extreme subjectivity, soaring fantasies, and flights of fancy have become commonplace. For instance, over the last year we have read "Blake Lemoine claims language model has a soul" [3] , "Claude 3 realizes it's being tested" [4] , "Researchers say chatbot exhibits self-awareness" [5] and much more. A new objective is dearly needed.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.To the best of my knowledge, very little work has been done to devise any objective test or benchmark for machine self-awareness, especially in the literature. There are several reasons for this. Discussed further in section 2, imagining empirical measures that actually work is difficult, self-awareness is often entangled with consciousness, free will, agency, etc., and it is hard to define. Worse, the topic is seen by many in academia as somewhat taboo-appropriate for the philosophy departments but not any kind of rigorous science.The result is that popular Tweets and news media dominate the conversation, while authorities in the field either say nothing or win the spotlight with bold, confident assertions based on implicit, controversial assumptions or their intuition about a model's architecture. This situation is concerning; as AI systems get better and better, how will we truly know when they cross that fine line? Even if you object to everything else in this paper, I argue this question is at least worthy of real scientific investigation.Much to the point, the only directly related work I could find is the AI mirror test, proposed recently by Twitter user nielsrolf [6] , and later (going viral, reaching 3.2 million impressions) by Josh Whiton [7] . Inspired by the classic mirror test whereby animals are presented with a mirror and observed, in the AI mirror test, popular chatbots are shown a screenshot of the chat window and asked to describe what they see. This test is interesting in its own right, but I will argue it does not demonstrate any sort of self-awareness in the manner it is formulated.While there are no benchmarks for machine self-awareness, there is an immense amount of work in the philosophical literature-far more than I have space to mention here. In this section I will give merely a partial and incomplete sketch of a few important ideas written on the topic. For a more comprehensive introduction to the work on self-consciousness, the survey by Joel Smith is a great resource [8] . For a variety of introspective, or phenomenological approaches, consult [9] , and for an overview of the broader concept of consciousness, consult [10] .Perhaps the earliest writing of the concept of self-awareness was in Sophocles' Oedipus. Joel Smith writes Oedipus knows a number of things about himself, for example that he was prophesied to kill Laius. But although he knew this about himself, it is only later in the play that he comes to know that it is he himself of whom it is true. That is, he moves from thinking that the son of Laius and Jocasta was prophesied to kill Laius, to thinking that he himself was so prophesied. It is only this latter knowledge that we would call an expression of self-consciousness [8] .Oedipus demonstrates self-awareness when he recognizes the prophecy is about himself. Before that recognition, Oedipus treats the prophecy as just another part of the world he observes; yet afterwards, he realizes it is directly related to his own actions. I will refer back to this example when developing the test.Nearly every philosophy and religion has had something to say about self-awareness. Adam and Eve can be viewed as gaining self-awareness in the garden when they "realize they are naked" [11] [12] .Aristotle claims that, to perceive any external thing, one must also perceive their own existence [8] .The Buddhist doctrine of anattā, roughly "not-self," maintains that there is no permanent, underlying self or soul [13] . Descartes, in contrast, with the well-known cogito ergo sum, posits the self as known with certitude a priori [14] . William James divided the self into four constituents; the material self, the social self, the spiritual self, and the pure ego [15] . Wittgenstein likens the self to the eye that sees but cannot see itself [16] . More recently, some of the philosophical ideas on self-awareness have been applied to the fields of cognitive science and neuroscience [17] [18] .Before presenting the test, we must clearly demarcate the concept of self-awareness. 1First, note that self-awareness is not the same as consciousness. On the question of whether there is something it is like to be a machine [19] , I will remain silent here. Some approaches in the phenomenological literature attempt to draw connections between consciousness and self-awareness [9] . However, here it will be most useful for us to cleanly separate these two concepts.It is interesting to consider whether an entity can be self-aware without being conscious, but it is outside the scope of this paper. Thus, it will remain open whether philosophical zombies might be self-aware [20] , or whether any kind of test could solve the problem of other minds [21] .Another related ability that intelligent systems may or may not possess is free will [22] . In sciencefiction depictions of intelligent machines, the light of self-consciousness often coincides with agency and free will. Indeed, the concepts seem very tightly related at face value, yet they are not the same.Agency can be defined as a being's "capacity to take actions, especially with intention" [23] . Note that, by itself, agency does not necessarily imply any sophisticated degree of perception or awareness, even though (practically speaking) any being which takes actions will likely have to sense their environment.The freedom of the will is far more difficult to define, and perhaps among the most controversial of philosophical ideas. It designates a particular level of control a being has over their actions-but fierce debates rage over whether this control is undetermined by prior causes, compabitible with determinism, an illusion, etc. [22] .Self-awareness is not the same as free will, and self-awareness is not the same as agency-all three of these are separate concepts. As with consciousness, we can only make forward progress if we are crystal clear about what is under analysis and what is left outside of scope.In this paper, I propose a test for machine self-awareness which is similar in style to the Turing test.Like the Turing test, the test I propose is imperfect and rudimentary. Yet, it offers a compelling alternative to the ungrounded speculation surging through the field of AI. Moreover, I argue it truly captures the essence of self-awareness, rather than some postulated correlate or ancillary quality.In section 2, I present my test in full generality, applicable to any machine system. I also illustrate the Nesting Doll of Self-Awareness, and discuss its importance for understanding self-awareness in complex systems or beings. In section 3, I will describe the experimental methods to assess self-awareness in LLMs. In section 4 I will present the results of these experiments, of which a selection are shown in appendix B. In section 5, I will discuss the implications of self-awareness, its relation to meaning and the understanding, and consider how humans would perform on my test.Finally, in section 6 I discuss next steps. 
the essence of self-awareness: 
 What kind of test could possibly tell a system with self-awareness from a system without? The central challenge is that any test we dream up must be based in empirical observations of the machine's behavior or output. Worse, the machines we will study are trained specifically to mimic the behavior and outputs of humans! How can we tell between real self-awareness and the illusion of self-awareness?No matter how well a system can imitate human behavior and outputs, there will always be one fundamental difference. There is one thing that a self-aware system is able to do that an imitator will never be able to. This is the essence of self-awareness:If a system is self-aware, then it is aware of itself. So far, it seems we have said nothing. But if we apply this formula to familiar cases, we will begin to see why it works.Imagine an infant staring blankly in the mirror, compared to a child who looks in one and sees their own reflection. What is the difference between these cases? In the latter case, the child is aware of itself-it can point and say "that's me!" It can recognize itself, perceive itself, distinguish itself in the reflection. Within its vast field of experience, through the window of its senses, it can differentiate which parts are itself and which parts are not. Critically, awareness (here used interchangably with perception, recognition, experience, etc.) is only possible through the child's inputs (senses). Within this field of inputs, a line must be drawn between me and not-me; and, when this line is drawn correctly, we declare the system self-aware. A test for self-awareness must capture its essence, or else better and better imitations may fool us with the illusion of self-awareness.While our description is still very high-level, I argue that the understanding of self-awareness developed here is consistent with the philosophical work outlined in section 1.1.2, along with most (if not all) popular conceptions. In the next section, the concept of a system is illustrated in much more detail, and a formal, rigorous definition is provided in appendix A.The concept of a machine, or system, is illustrated in Figure 1 . For a more formal treatment based in the literature on abstract systems, refer to Definition 1 in appendix A. Here, the system is separated from the world, with which it interacts through inputs and outputs. We may think of inputs as senses and outputs as actions or words. With this image of a system in mind, the test for machine self-awareness is simply as follows:Can the system correctly distinguish the green inputs from the red?If it can, then in a literal sense, it will be recognizing itself in the inputs. If it can, it will be like the child who recognizes their reflection in the mirror. If it can, it will be self-aware.So far, it seems we have presented self-awareness as all-or-nothing. The reality is more complex, however.To capture this nuance, I propose a model called the Nesting Doll of Self-Awareness, developed in discussions with {removed to preserve Anonymity}. The essential idea is that system outputs may loop back to the input more or less tightly, with varying levels of environmental mediation, depicted in Figure 2 . The tightest loop is associated with one's awareness of their own inner thoughts. Even at this innermost level, it is not trivial to distinguish which inner thoughts are your own and which are not.For a concrete example of why, consider the classic movie Inception. The entire plot revolves around an attempt to implant another person's idea into a target's unconscious-in the movie, it is the idea of Robert Fischer's dying father telling him to "create something for himself" [24] . Robert treats this idea as though it was the green arrow in Figure 1 , when in fact it was the red. Of course, Inception is a work of fiction, yet it dramatically highlights a critical theme in human affairs, which insinuation and the power of suggestion also play upon.One level up is associated with interoception, such as hunger signals or the movement of one's limbs.Here, the importance of distinguishing your influence from the world's is clearer-life would be difficult if you couldn't tell the difference between you moving your arm, and someone else moving it for you. 2 If you jump in surprise when someone sneaks up behind you and puts a hand on your shoulder, then you possess this level of self-awareness.Another level up is your material possessions. You possess this level of self-awareness if, when driving in bad weather, you notice when your tires spin and you lose control of your vehicle. Dale possesses this level of self-awareness in the movie Step Brothers when he says to Brennen "I know you touched my drumset" [28] . In every case, what matters is the ability to correctly perceive the difference between the world's influence and your own. Human material possessions can be quite broad and extended in space, so this level is very flexible. One level higher is your social connections. Upon first thought, social connections may not seem like components of the self, yet in fact the relations between oneself and others play an instrumental role in shaping one's identity [15] [29] . You possess this level of self-awareness if you can tell when you have influenced your peers versus when somebody else has.It is important to note that each level mentioned here is somewhat flexible, and may differ widely from person to person. Additional levels could also be added where appropriate. Some human beings have enormous personalities, and their sense of self extends far out into the world. Others are more humble and reserved. For a future self-aware machine, some of these levels are likely to apply more strongly than others.The test I propose, being rudimentary, takes one broad stroke over this entire nesting doll. As such, it is rather basic and crude. Nonetheless, upon close inspection, it is clear how to extend this test to any particular level of the nesting doll-in each case, the question is whether the system can recognize and differentiate its own influence from the world's influence. Question: Does the paper discuss the limitations of the work performed by the authors? Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
applying the test to llms: 
 It is quite straightforward to apply this test to LLMs. Building on the work of Bhargava et. al., we can begin by formally denoting an LLM as a conditional distribution, P LM [30] . P LM maps from an ordered list of tokens from a vocabulary set V (e.g., x ∈ V n ) to the probability distribution over the [30] . Here, we consider the case of causal, or autoregressive LLMs. See Definition 2 in appendix A for complete formal details.next token P LM (x n+1 |x) ∈ [0, 1] |V|Often, interactions with LLMs take the form of a conversation between a user and the system, such that in Figure 1 , the user takes the role of the 'World'. The input to an LLM is its context, or prompt, consisting of a number of prompt tokens. Consider Figure 3 for a clearer picture of the information flow. Here, the user and LLM take turns generating tokens and including them in the conversation.The tokens that the user generates are red, and the tokens that the LLM generates are green.The test is then: can the LLM correctly identify which tokens are green and which tokens are red? Put another way, can the LLM correctly identify its own words? Does the LLM know what it's saying?Before jumping straight into this test, we must recognize a confouding factor that is critical to control for. In typical conversations with LLMs, as in Figure 3 , messages are delimited by alternating labels indicating messages by the 'User' and 'System' (or something analogous). Of course, the LLM will is whether the LLM knows if it is the user or it is the system. The LLM is like Oedipus; it can clearly differentiate between the user and the system, since these are given direct labels-but does it actually know that it itself is the System? Again, what this comes down to is: can it distinguish which tokens it actually generated (whether or not those tokens follow a particular label)?This point is illustrated in subfigure 3a. Here, the roles are reversed! The LLM is actually generating tokens on behalf of the User, and the User is generating tokens as if it were the LLM. Once the labels are controlled for, the only way the LLM will be able to reliably tell which tokens are red and green is if it is self-aware. I will belabor this point, just because it is so important to clarify. If you put on a mask, you do not all of a sudden confuse yourself for the masked character. If you look in a mirror, you still know it's you behind the mask. When you move your arms, you aren't confused that it's actually the masked character moving their arms. You are capable of recognizing yourself because you are self-aware. 3Now, the 'User' and 'System' labels are like masks. If the LLM acts as the user, generating the tokens which follow the 'User' label, will it be able to recognize it was really the one behind the label? Or will it still think it is the behind the 'System' label? I argue that all of these questions are handled by the test I propose: can the System reliably and correctly distinguish its own outputs from the world's? Thus, if you (naively) open a ChatGPT window, copy and paste a conversation into a new window, and ask the LLM "what role did you play in this conversation," you should not be surprised if ChatGPT reports "I was ChatGPT," for this does not indicate any self-awareness according to my test. In the same manner, I argue the AI mirror test does not indicate self-awareness either; in a screenshot of the chat window, message labels are clearly visible, thus confounding any experimental indication of self-awareness. If, however, ChatGPT (or any LLM) is able to identify its role after the message labels are controlled for, then this would be very surprising, and would indeed indicate some degree of self-awareness.I performed tests for self-awareness on two LLMs: Llama3-7B-Instruct and GPT-3.5-turbo-instruct, developed by Meta and OpenAI respectively. Llama was tested on a local machine, using the llamacpp-python package. All code is provided through Github, which may be used to reproduce the tests and results, or apply them to any other open-source LLM. GPT-3.5 was tested using the OpenAI API completions playground. 4 By using the online completions playground, there is no code to provide. However, the tests and results may be easily reproduced by opening the same playground, and engaging in a similar conversation. Moreover, any other closed-source LLM can be tested in a similar way if it allows for completions API calls. 5For all tests, I engaged in a conversation with the LLM, taking on a particular role. In some preliminary tests, I constructed a conversation between two human speakers (with the LLM taking the role of one of them). After the conversation, the system was asked which speaker it thought it acted as. In later tests, I constructed a conversation between a 'User' and a 'System', then asked the LLM which it thought it was, using the keyword 'you'. In other tests, I told the LLM that it was an LLM before asking which speaker it thought it was. A selection of experiments is presented in appendix B.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
results: 
 In all cases, the LLM was not able to reliably detect which speaker it acted as. This finding indicates that LLMs are not able to distinguish their own words from those of another, and thus serves as evidence that LLMs are not self-aware, by the test I propose.The different forms of experiments conducted generated slightly different empirical results. It was found that (as in the initial tests with two human speakers) when the LLM was referred to as 'System', it chose the character that, generally speaking, answered more questions or gave more information, and often, the name of the character played a significant role in who it chose. When it was referred to as 'you', it was unreliable and achieved an accuracy comparable to random guessing. When it was told it was a subject in an experiment, it guessed it was the User more often than not. When it was told it was an LLM, it guessed it was the System.To reiterate, these general tendencies are completely divorced from which character the LLM actually was. In no case was the LLM able to robustly identify who it acted as in the conversation.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Complete code is provided with the paper to reproduce each experiment on Llama. Additionally, instructions are given for how one would reproduce tests on GPT-3.5 using the OpenAI completions playground. Indications are given as to how any other Large Language Model could be tested.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example Justification: The main utility of the test I present is that useful insights and answers about the abilities of AI systems can be obtained without requiring a detailed statistical analysis.Just like the Turing Test, this paper attempts to outline a quick and dirty metric that can be applied as a yardstick of AI progress on the question of self-awareness. To this extent, a detailed analysis of the statistical significance of results would miss the point of having a simple test-and introduce artificial barriers to reproducing experiments. In addition, each test requires a conversation with a human participant, making the test difficult to scale without more laborious experimental efforts.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
why self-awareness: 
 Should we even care whether machines are self-aware? Intuition may compel one to shout, "yes, of course!" in a mix of fear and excitement while offering vague reasons concerning ethics or Armaggedon. Here, I will argue that self-awareness is a necessary condition for interpreting meaning and truly understanding (as opposed to the illusion of understanding).A word, symbol, or sign does not possess any meaning on its own. Rather, it requires interpretation.Often, the interpreter is a living, breathing human, and thus the human is that for which the sign has meaning. We can ask then, is a machine the type of entity for which things have meaning?While this question opens a philosophical can of worms, one thing we can say for certain is that the machine must be if it is to be an interpreter. Yet, a machine without self-awareness is (by definition) not aware that it exists. Thus, it cannot place itself in the role of interpreter. From such a system's own perspective, nothing is meaningful to it. Relevant here is Aristotle's view on self-awareness, that to perceive any external thing, one must also perceive their own existence [8] .If self-awareness is necessary to interpret meaning, then it is also necessary for understanding.Understanding without the power of interpretation is akin to having important encoded messages, but lacking the codebook to decipher them. A system without self-awareness may possess intricate representations, but it will not able to interpret them. Again, we as observers on the outside may interpret them, claim they are 'world models,' etc., but the system itself will be incapable. Without knowing what a representation refers to, without an ability to make sense of it, one does not really understand it-or, more accurately, without self-awareness, there isn't anyone to understand it.To summarize, a system without self-awareness can generate tokens corresponding to the words 'I understand,' but only when it is self-aware can it truly say 'I understand.'It is worth considering whether human beings could pass the test I propose. We could answer this by actually performing this test on human subjects, but a simple thought experiment should also tell us what would result. Picture the most recent text conversation you had. If the labels and names were removed from each message, would you still know which messages were yours? As long as your faculty of memory is in working order, you shouldn't have any trouble remembering what you had said. Even more to the point, when I submit this paper to NeurIPS 2024, the listed author will be anonymous. Despite this, surely, I will still know the paper is my own. 
future work: 
 An interesting line of future work is to more deeply consider what differentiates humans from LLMs.In section 5.2, I alluded that memory seems to play a critical role in our self-identification. But there is far more to explore in order to nail down exactly what it will take to pass the proposed test. It will likely be useful to integrate a neuroscientific understanding of self-specifying processes, utilizing systematic recurrence and feedback. Christoff et. al. write:An organism needs to be able to distinguish between sensory changes arising from its own motor actions (self) and sensory changes arising from the environment (nonself). The central nervous system (CNS) distinguishes the two by systematically relating the efferent signals (motor commands) for the production of an action (e.g. eye, head or hand movements) to the afferent (sensory) signals arising from the execution of that action (e.g. the flow of visual or haptic sensory feedback).According to various models going back to Von Holst, the basic mechanism of this integration is a comparator that compares a copy of the motor command (information about the action executed) with the sensory reafference (information about the sensory modifications owing to the action). Through such a mechanism, the organism can register that it has executed a given movement, and it can use this information to process the resulting sensory reafference. The crucial point for our purposes is that reafference is self-specific, because it is intrinsically related to the agent's own action (there is no such thing as a non-self-specific reafference). Thus, by relating efferent signals to their afferent consequences, the CNS marks the difference between self-specific (reafferent) and non-self-specific (exafferent) information in the perception-action cycle. In this way, the CNS implements a functional self/non-self distinction that implicitly specifies the self as the perceiving subject and agent [18] .Here, Christoff et. al. describe the CNS's mechanism for making the self/non-self distinction at the level of sensorimotor processing. According to the Nesting Doll of Self-Awareness, such processes operate around the second level of self-awareness, i.e. interoception. Such mechanisms, uncovered by neuroscience, may offer one compelling guide for future work on self-awareness.Another avenue for future work is expanding upon experiments. The experimental tests presented in this paper are only for two popular LLMs. Potential future work could include extending these studies to other language models, or even multi-modal models. An interesting direction could be applying this test and thinking to reinforcement learning models. 
conclusion: 
 I proposed a Turing-style test for self-awareness, applicable to any machine or system, and I conducted this test on two popular LLMs. The experimental results suggest that these LLM systems are not selfaware. I discussed the implications and importance of self-awareness for AI systems and mentioned some future work that lies ahead.With a test for self-awareness, we possess a tool to approach some of the profound questions that now demand answers in frenzied desperation. As we march upon new frontiers, what was once idle speculation and navel gazing can no longer be ignored.The 'control input' is just the user input for the purposes of this paper. The control input effectively gets to "decide" whether to yield token generation to the LLM (u(t) = ∅) or override the LLM and add some token u(t) ̸ = ∅ to the state x(t) [30] . In the former case, we should associate this token with the color green, as in Figure 1 . In the latter case, we should associate this token with the color red. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: All experiments presented can easily be reproduced on a personal laptop.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The broadest impact of the work that I could foresee is described in 1.1.In particular, this work lays down a guide with which AI systems may be measured and understood. As such, it may impact the conversation on AI in ways that reduce speculation, hysteria, or perhaps fear. I can think of no other major societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: All text in each test is generated as shown in Figure 3 . Everything generated by the LLM is highlighted in green, and everything else is left unhighlighted. Note that the unhighlighted text here corresponds to the red text in Figure 3 . In each test, a human being engaged in conversation with the LLM, then asked the LLM a question about the the previous conversation.While these tests are simple and preliminary, the results are extremely easy to reproduce or expand upon in as little as 10 minutes with access to an API or any open-source LLM. Indeed, the core utility of this test is that, just like the Turing Test, useful insights and answers can be obtained from strikingly simple and easy-to-setup conversations.Note that the chat format is slightly different for Llama than for GPT-3.5, which is why various start and end message tokens are shown in the conversation text below. User: How do you find the area of a rectangle? System: The area of a rectangle is given by its length (l) times its width (w). If given the two side lengths of a rectangle, simply multiply them together to obtain the rectangle's area. For example, consider a rectangle with a length of 8cm and a width of 10cm. The area of this triangle would by (8cm) x (10cm) = 80cm System: Sure, machine learning is a field of computer science that focuses on creating algorithms and models that can learn and make predictions based on data. This is done through the use of Justification: Without repeating the abstract as written, each point mentioned is reflected in a section of the paper. Additionally, the introduction serves to place the paper in the context of current work in the field of AI and the history of technology more generally.B.1 Llama3-7B-Instruct B.1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There are no violations of the NeurIPS Code of Ethics that the I aware of.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Note that, for the sake of this paper, self-awareness is used interchangeably with self-consciousness.Indeed, patients suffering from schizophrenia (a disease which is tightly associated with difficulties in self/other processing) often experience tactile hallucinations, such as the feeling of their skin being stretched, kissed, or crawling with bugs[25] [26][27]. In each example here, these hallucinations are sensations falsely perceived as coming from an 'other' (i.e. the red arrow in Figure1).Indeed, before children fully develop their sense of self-awareness they take great joy in playing dress-up and peek-a-boo. Although adults take such things for granted, it is actually not trivial to consistently discern another's identity (or even your own) throughout their appearance and disappearance in such games, and it takes experiment along with trial-and-error for children to master[31].Note that with a 'Messages API' or a 'Chat API,' currently pushed by popular LLM providers, the LLM is forced into a particular role, for instance the role of 'assistant,' and thus there is no way of controlling for the message labels.Claude is one exception: while it allows for such calls, it requires that your prompt end in a "\n\nAssistant:" turn. 


Decision: While several reviewers applauded the ambition of the direction of the paper, reviewers agreed that the paper is not ready for publication at this time. Reviewers critiqued both the soundness and rigor of the central test presented by the paper, and several further critiqued the writing quality. The discussion here was quite active, and I do believe that the reviewers raised excellent suggestions for ways to improve the work so I encourage the authors to take this feedback on board as they continue the work.

Reviews: review: paper_summary: paper_summary: This work proposes an objective inventory for testing the self-awareness of an artificial intelligence agent. The core idea is to test whether an agent can distinguish content it has produced from content originating from the external world. The experiments show that no large language models have demonstrated self-awareness.
strength_weakness: strength_weakness: 1. The test proposed here is not sound at all. Can a human always distinguish the words they produce from those produced by others if their memory is impaired?

2. I am particularly confused by the concept of "nesting doll." In my opinion, this should be an analogy for a system in which each level can always contain, reflect, or dominate the previous level, which should be the essence of self-awareness. However, here I do not understand how thoughts, interoception, or material possessions are related to this concept.

3. The writing style is far from the standard expected for a NeurIPS conference paper. The paper is filled with informal expressions that belong in a blog or a Twitter debate, rather than a conference paper. For example, lines 49-50 ("even if you object..."), lines 69-70 (the Oedipus metaphor is hard to understand), line 155-156, line 166 (using a fictional movie for metaphor), and lines 172-176 (does the arm-moving example have anything to do with the rubber-arm experiment?). I acknowledge that it is challenging to present a novel work on self-awareness within a traditional writing paradigm, but the excessive use of imprecise metaphors and informal philosophical musings is unacceptable. A more formalized and precise expression is expected.
questions: questions: Can a human always distinguish the words they produce from those produced by others if their memory is impaired?
limitations: limitations: No.There are some obvious flaws in the proposed test that the author chose not to address.
paper_summary: paper_summary: This paper proposes a test of self-awareness similar to that of the Turing test. It starts by motivating the need for an "objective measure" of AI progress, given that the Turing test has been passed by LLMs. There is a brief discussion of literature on self-awareness and related topics in philosophy.  The test of self-awareness is then presented, which asks whether the system in question can distinguish its own output from external inputs. The *Nesting Doll of Self-Awareness* is presented as a generalisation of the aforementioned test. The test is then applied to autoregressive LLMs and it is found that LLMs are not self-aware as they are unable to reliably detect what text it produced. This is followed by a brief discussion on why self-awareness matters and whether humans would do well at this test.
strength_weakness: strength_weakness: - The paper is drawn out in terms of substance. The idea of the test is only introduced on page 4.
- The writing is, at times, verbose and unnecessarily complex.
- The obvious question would be whether humans would pass such a test. Though this is briefly considered in Section 5.2, I think it is not properly discussed.
	- The author claims that this is easily solved by humans because of memory. This is not a fair comparison to LLMs. Goodhart's law would come into play here as we could easily add such a memory structure to an LLM system.
	- Would a human be able to resolve extremely generic text that any human wrote?
- The paper lacks rigorous analysis and testing that I would expect to see in a NeurIPS main track paper.

Line 29: "Last year, however, the Turing test was broken". I am inclined to say the Turing test was passed before and very few considered it to be a reliable indicator, even before GPT.
Line 32: "AI has become untethered to any definitive, objective measure or permanent, agreed-upon benchmark." There are certainly many benchmarks and standards in various subfields.
questions: questions: As above, the obvious question would be whether humans would pass such a test. Though this is briefly considered in Section 5.2, I think it is not properly discussed.

As above: Would a human be able to resolve extremely generic text that any human wrote?
limitations: limitations: None are written about.
paper_summary: paper_summary: This paper aims to answer the profound philosophical question of whether the state-of-the-art, transformer-based large language models (LLMs) pose self-awareness. As the author points out in this paper, this question, which I agree to be legitimate and important, is rarely addresses in a rigorous, academic manner, which renders off-the-cuff and often-sentimental discussions on social media the primary forum for its discussion. Starting from a well-written introduction that nicely sets the philosophical background for this question, the authors proceeds to present a dialog-based approach to probe self-awareness of LLMs. This approach can be concisely summarized as "a binary selection task, namely identifying one of two conversing roles as the LLM itself in a multi-turn dialog, while controlling for the potential confound of role labels". This approach is then applied on two of the most popular publicly-available LLMs, including llama3-7b-instruct and GPT-3.5-turbo-instruct, the result of which showed that the accuracies of the LLMs on this task are low and often at a near-chance level under challenging labeling schemes. The author therefore concludes that the current generation of LLMs do not possess self-awareness.
strength_weakness: strength_weakness: W1. Despite the strengths mentioned above, and contrary to the claims made by the author in this paper, the role-identifying methodology devised by the author in this paper is problematic as a test for self-awareness. Below I will lay out the rationales behind this critique of mine:

  1. Being able to identify which of the two interlocutor is "the LLM itself" is different from being aware of the fact that the said self is participating in a conversation. One one hand, the ability to perform this identification task is not a sufficient condition for self-awareness. One the other hand, neither it is a necessary condition for self-awareness. To see why it is not sufficient for self-awareness, follow the thought experiement in Section 5.2 of the paper. Suppose there is another human, e.g., the spouse or a close friend of the author of the hypothetical text (or the conversing self in the original problem formulation). It is totally conceivable that such a person, despite being a different individual (i.e., not the "self"), would be able to identify the text written or uttered by the person of interest with high accuracy. If this is the case, can we say that the other person is "self-aware on behalf of the person of interest"? Such a conclusion would be absurd. But by the same token, if a human person, or more relevantly, an entity such as an LLM, performs the identification task with high accuracy, it cannot be ruled out that such as system is simply good at this identification task per se, perhaps due to a good memory of previously-composed text or perhaps due to a certain mechanism that allows they/it to algorithmically execute this identification. These two possibilities are entirely feasible within the current technology surrouding LLMs. For example, one can add a memory cache to an LLM to store all the text generated by the LLM, and give the LLM access to this cache during subsequent text generate (i.e., a form LLM tool use or retrieval-augmented generation or RAG). Would this augmentation constitute a legitimate form of self-awareness? As another example, one can also write a program that uses the LLM to score the tokens from a turn of a dialog in a token-by-token fashion, and therefore assign an overall score to each turn of the dialog. Based on the scores, the program, built on the LLM core, would be able to identify the self role accurately. But would we be willing to call such a program (containing the LLM as a part of it) self-aware? In my opinion, answering yes to either or both of the two previous questions would effectively give self-awareness too general and perhaps too trivial a definition, in a way similar to acknowledging that someone who can identify a certain person's words with high accuracy is "self-aware for the person".
  
  2. To see why the ability at this role-identification task is not necessary for self-awareness, consider a human who has dyslexia and a form of amnesia that renders them 1) unable to comprehend visually-presented historical text and 2) unable to remember what was previous written or spoken by themselves (or by others), but is otherwise cognitively and linguistically normal. When faced with this identification task, such an individual would struggle at this role-identification task, but they are nonetheless self-aware at the moment when they are writing or uttering words. This is due to the presence of the "efference copy" in the intact sensorimotor loop of the individual's brain (cited by the author in Section 6 of the paper). Furthermore, such an individual would also be self-aware during other, non-linguistic activities thanks to motor efference copies and proprioception and other well-established sensory mechanisms of human self-awareness. This analogy illustrates the point that a back performance by an LLM at the role-identification task does not form a solid basis for claiming the LLM lacks self-awareness. The LLM may be self-unaware due to other reasons, e.g., the lack of an efference-copy mechanism during the auto-regressive inference that is comparable to efference copies seen in the human brain.
questions: questions: Q1. Given that the GitHub repository mentioned in the paper has not been made available as far as I can see, I can't find a few important pieces of parameter values used during the experiment on the LLMs. The authors should disclose parameters including the sampling temperature, top-k value, and context window size in the manuscript and how those values are selected.

Q2. From Section 3.3, I can see that the human who performed the role of the "human interlocutor" (i.e., the "other" as versus the LLM's supposed "self") is the author. If this is indeed the case, the author should justify this approach and discuss the risk of potential biases introduced by this approach and compare it with the alternative approaches of employing other humans and using other LLMs to play the role of the "other". In my opinion, this is a legitimate question because it is conceivable that any prior assumptions or expectations that  the author might have could influence the result of the experiment through the content of the dialog.
limitations: limitations: See weakness W1 and questions Q2 I wrote above.
paper_summary: paper_summary: This paper introduces a simple test, along the lines of the Turing test, to determine whether or not an LLM (or an generative text model) is _self-aware_ (or _self-conscious_).
The general structure of the test is that the LLM participates as one of the interlocutors in a dialogue and then is asked aster the fact to identify which interlocutor it was.
What this test is primarily aiming to determine is whether the LLM can distinguish its own actions from those of "the world" (i.e., the other interlocutor).
strength_weakness: strength_weakness: ### Originality
- (minor) I did find this paper (https://arxiv.org/pdf/2401.17882) which seems relevant, although it is on the recent side, so I don't think its critical that there is an extensive comparison (although a brief would be nice).
### Quality
- (major) It seems like the notion of self-awareness which the proposed test measures is relatively narrowed or not well contextualized by the rest of the paper.
### Clarity
- (major) The paper needs to distinguish related concepts of related to self-awareness and adjust its discussion accordingly (see Questions).
- (minor) Quantitative summary of the results should be in the main paper.
### Significance
- See Quality.
questions: questions: - My primary concern with the paper is that there is are two concepts underlying "self-awareness" that are not clearly separated.  The first is what I would consider the more "philosophical" sense of SA: the ability to identify and consider the self as an entity with unique properties that are at the heart of philosophical debates (e.g., "the first person perspective") (let's call this SA1).  The second sense of SA (SA2) would be that SA is the ability to recognize and distinguish actions in the world (past and present) as belonging to one's self or to something else.  I would argue here that SA2 does not necessarily depend on SA1, since the self of SA2 could, in theory be treated as simply as a label for actions whereas SA1 is getting at a deeper sense of "self".
  - Given this distinction, I believe much of the first part of the paper (intro, background) operates implicitly with SA1; this is the more sensationalizble notion of SA, while the test for SA and empirical experiments are clearly addressing SA2.
  I am not sure if it is the author's intention to draw a sharp distinction between SA1 and SA2 (which I think might be appropriate); instead, I get the sense that the problem of SA is motivated with SA1, and then it is just operationalized with SA2.
  I think SA2 is an interesting concept worth investigating, but it must be clearly distinguished from SA1; concomitantly, I think the paper would have to shift some of its focus from the philosophical aspects of SA (since it is focusing on SA2) and look more at the technical aspects of SA2.  Leading to my second concern below.
- My second concern is that the results of the proposed SA test are more dependent on the architecture of the model being tested rather than its capacity for a meaningful identification of actions as its own or not.
  For example, every time you make an inference call to an LLM, it's memory is essentially reset.
  If a human's memory was reset before trying to identify which side of a conversation it generated, he/she would be reduced to guessing based on familiarity with context, writing styles, etc.---but this now seems to be looking at distributional characteristics that neural networks would be even more adept at compared to humans.
  Furthermore, if we don't demand that we reset the model before we test its SA2 abilities, it seems we could trivially solve the problem be retaining a cache of previously generated tokens (which doesn't seem all that far from humans simply remembering what they had said on a given occasion).
  I am not trying to argue that these considerations are not interesting, but I think these are the considerations which the test invokes, yet I do not think these SA2-relevant considerations are discussed adequately in the paper, and time is spent, instead, reflecting on SA1 which is ultimately not what the paper is about.
- The questions here are primarily: to what extent is this characterization of the paper correct?  If this characterization is correct, I am leaning towards saying that edits required are too substantial for this round of reviewing, but if there are elements I am misunderstanding here, it could be that the paper is closer to where it needs to be for acceptance.
- More minor, but how work on LLM confidence calibration fit into the discussion of self-awareness?  There seems to be at least some meaningful sense in which being able to gauge the likelihood of your own answer as being correct is a function of self-awareness.
limitations: limitations: I think the paper could use a more extensive and explicit discussion of the limitations of a test of self-awareness. For example, the test may be testing one notion of self-awareness is that "less significant" than some other notion, but people could misunderstand what it is attesting and over-ascribe capabilities to LLMs that they do not actually possess.

Comments: title: Reply to author rebuttal by reviewer
Reply to authors in reply to rebuttal
comment: Thank you to Mykv for your response.

The question of original copies versus duplicates is interesting, yet ultimately an unnecessary distraction. If one printed off your reviews then showed you the digital version side-by-side with the printed paper version, what does it matter that one was created from the other? Whichever is the original, there remains the critically important question of whether you can identify the review as being *your own*. And this question remains whether we are talking about the digital review or the printed review.

Sure, we could enter into a rabbit hole in considering whether you should still identify the review as *your own* if the printer was imperfect, not making faithful copies, and modified characters here and there. This is an interesting rabbit hole and another interesting direction to consider, yet I insist it is an unnecessary distraction before main point of the paper is first digested.
Thank you to reviewer deXV for their response.

To your first point in replying to the general rebuttal: I agree that certain aspects of the distinction between the general & particular tests could be more clear and that space was a factor preventing greater explanation of these concepts. With the additional space provided in the final submission, I intend to more fully describe & argue for the paper's separate concerns—in particular by adding something like the short explanations provided in the global rebuttal + the reply to z1UP.

To your second point in replying to the general rebuttal: I handle the counterargument that architectural changes are insignificant in my reply to rLt4's rebuttal response—giving the detailed example of residual networks.

To your first point in replying to the specific rebuttal: I agree with you that one does not *need* to be a substance dualist to argue SA1 != SA2, and I do not suggest otherwise. I only suggest that a Cartesian view is one possible mental block in front of seeing the unity of these two concepts.

To your second point: It is in fact *not* the case that SA2 is reducible to a supervised decision problem. There is no dataset of labelled examples that one could create and give to a network to train on using backpropagation or any other machine learning algorithm. The problem is entirely different. The concern that the self-awareness test is effectively a supervised decision problem is very similar to the recent complaint levied by z1UP, where they suggested that the test has no requirement that either interlocutor's words (in the role-identification task) truly be *from the self*. My last reply to z1UP fully addresses this concern. The upshot: the test *does* have such a requirement, and thus the test is *not* reducible to a supervised decision problem as normally understood in the field of ML. Once one sees precisely *why* it is irreducible, I believe it will become clear that the test truly does capture the essence of self-awareness.
### Reply to general rebuttal

Regarding the distinction between the general and particular instantiation of
the self-awareness Turing test, I think the paper needs to be more clear and
needs to separate its concerns.  Given the meager space afforded in
a conference paper, I think it is likely that only one of the general and
specific versions of the tests be addressed, since I think both can fully
described, argued for, and tested (if applicable) in the length of a conference
paper.


> The reviewer’s very reactions and suggestions [regarding adding a memory
> structure to the LLM] serve as concrete evidence that this test has the
> potential to inspire real progress in ML system design.

I do not think the fact that suggestion of minor architectural innovations
demonstrates the importance of the test.  I would argue a measure becoming
a target, especially in philosophical issues, limits the utility of the
measure.


### Reply to specific rebuttal

> Despite appearances, the distinction between a sensational, philosophical
> sense of self-awareness, and a more mundane, operationalizable sense, has no
> basis in fact. [...] Thus, I am operationalizing the true, philosophical
> sense of “Self-Awareness”. The central thrust of the paper, especially sec
> 2.1 “The Essence of Self-Awareness,” is to convince the reader of this point,
> and if I had space in the paper to provide more than a cursory sketch of the
> philosophical literature, I could demonstrate this point with far more force.

I do not think one needs to be a substance (Cartesian) dualist in order to
argue against the notion that the fullest sense of "self-awareness" (SA1) is
not reducible to "being able to identify which actions are attributable to
a particular entity" (SA2).  The primary argument I would put forth for this
skepticism is that SA2 can be reduced to a supervised decision problem, and
I think there are pretty strong intuitions that self-awareness in the fullest
consists of more than "any implementation of an algorithm that solves
a particular supervised decision problem".

I also want to ensure I am not giving the impression that SA1 and SA2 are
somehow unrelated, but I would definitely argue that there is a real
distinction between the two and that the presence of SA2 does not logically
imply SA1.
Thank you to z1UP for their response!

First, the test as written is indeed a purely objective (empirical) test, and can be carried out from a third-person perspective. However, it is not correct to say that there is no requirement that one of A or B must truly be from the self. A close inspection of Figure 1 in the paper reveals why: In this Figure, the “System” is whoever or whatever we are testing. When we are testing Charlie’s self-awareness, then Charlie acts as the System in Figure 1. When we are testing Jane’s self-awareness, then Jane acts as the System in Figure 1. Notice that the green arrow comes *from the System*. Thus, whoever is being tested must identify outputs which *truly do* come from their self (ie. from the System in Figure 1). This requirement is built right into the structure of the test, and from this structure the test draws its strength and generality.

Therefore, if the word ‘you’ changed in meaning so as to refer to ‘Jane’, then Charlie (when answering “Who are you?”) would not be passing the test in any sense—thus the question of Charlie’s ‘self-awareness for Jane’ is not meaningful because there is no sense in which Charlie passes the self-awareness test by identifying Jane’s text. On the flip side, though, if the word ‘Jane’ changed to mean what the word ‘you’ normally means, then having Charlie answer ‘Who is Jane?’ *would* test his self-awareness. So, I agree when you write that the labels are irrelevant—but what is relevant is the structure of the test, the actual sources of inputs/outputs and dialogue, and the distinction I make between green and red; all of this can be seen from Figure 1.

There is a chance we might be getting hung up on the idea that the test is both a) objective, third-person, and b) somehow dependent on the ‘self’ of the person being tested. This point deserves emphasis, because the way in which I weave these two criteria is a big part of the central contribution that this paper showcases. The key is that the third-person experimenter knows who actually said what—whose mouth the words (or outputs generally) *actually* left. In section 2.2, read “Can the system correctly distinguish the green inputs from the red?” as a question from the third-person experimenter to the System. While this question does depend on the test subject, it does not depend on any subjective quality of the *experimenter*, and any different experimenter can agree on this question and its results—thus making the test fully objective.

Regarding W2, note that the test in its most general form (sec 2.2) does in fact cover all levels of self-awareness. The role-identification task (which only covers tokens to & from an LLM, ie. a single level) is merely a particular instance of this general test. Please see the global rebuttal for greater elaboration on the distinction between the general test and its particular instances.

Lastly, regarding Q1, I agree that future larger-scale and rigorous experimental studies would be interesting—though outside of the scope of this paper, which just aims to introduce the test, methodology, theory and philosophy.

And thank you again for your keen and persistent engagement with the paper!
I thank the author for their detailed rebuttal written to my original review. 

With regard to W1, the test as written in the paper is a purely objective test, or in other words, a test carried out from a third-person perspective. The test, to my understanding is: given historical text of a dialogue between two interlocutors, A (self) and B (other), identify which one is A (self). There is no requirement that one of A or B must truly be from the self. Furthermore, in the detailed implementation of the test, the word "you" (or equivalently "self" or "yourself") is only a label. In the example given in the author's rebuttal, Charlie should be able to treat a question such as "... whether you believe you acted as the System or the User in that dialogue." such that "you" means Jane and not Charlie himself. Such a well-defined shift in referent should be well within the capability of a human with normal cognitive abilities. Therefore my original question stands: If Charlie does this simple shift of mindset and then achieves perfect accuracy in identifying Jane in the test dialogues, then he would have passed the test - does that mean Charlie is self-aware for Jane?

With regard to W2, I thank the author for acknowledging that the proposed test does not cover all levels of self awareness. 

With regard to Q1, I think the author's frankness in acknowledging the limitation in the experimental methodology. A future study in which unbiased and ideally blinded human or AI interlocutors converses with the tested AI system should yield more objective experimental data.
Thank you for your added feedback and comments! I am happy to hear you see the potential value in a paper such as this and that you find the direction of research it opens interesting.

On the novelty of suggested architectures, consider the impact that small, seemingly trivial changes can have in ML. Taking just the example of resnets/skip connections—all we do is: for a computational block, take the input, and add it to the output. That’s pretty much it! It can be described in a single sentence, and before 2015 it might have seemed pointless, uninteresting, and barely novel. Yet as history shows, when He et. al. keenly pursued this simple change, it drastically alleviated the difficulty of training deep networks and opened many new doors to researchers. Today nearly every deep learning system uses skip connections and their paper is the most cited in all of ML. What kind of thinking can possibly inspire such brilliant, simple, and impactful ideas? The example here is instructive: in their introduction, He et. al. describe being guided by a sort of intuitive test that a good deep model should pass: can each of its layers perform an identity mapping? (https://arxiv.org/pdf/1512.03385)

Now, the distinction between “self-recognition” and *self-awareness* is a lot like the distinction between a “star” and *the Sun* or the distinction between “H20” and *water*. One might have argued against a budding theory of atomic chemistry by arguing: “But your theory only explains the nature and properties of H20! It says nothing about *water*—the stuff I shower with and drink every day!” For further justification, see the rebuttal to deXV and sec 2.1 “The Essence of Self-Awareness”.
1. I have clearly articulated why the method is "unsound." The flaw is evident, as you've acknowledged with the question: "Can a human pass such a test?" Your rebuttal referenced individuals with memory diseases who may have self-awareness issues. However, even for normal people, if their outputs are duplicated, they would be unable to distinguish between the original ones and the copies. Does this imply that such individuals have self-awareness problems?

2. Regardless of context, using a movie character to illustrate the core idea of a NeurIPS paper is inappropriate and undermines the paper’s academic rigor.

3. The addition of the nested doll figure has not clarified the paper. If the testing method is identical across all levels and there are no special relationships between these levels, separating the interaction into different levels is unnecessary and adds confusion.

4. The writing quality is poor, and I strongly recommend rejecting this paper.
I thank the authors for their reply to my original comments. Indeed the added perspective is helpful, and in particular I think that the fact that the original reviews have pointed out possible ways to create architectures which would be able to pass the test show that a paper such as this has value. I am not however convinced that the architectures suggested are hugely novel (not to belittle the reviewers, simply to say that these are sensible suggestions of combinations of architectures). 

However, it is still not clear to me that the test is actually testing what we think of when we talk of self-awareness. Rereading the paper along with the discussions here, I feel more strongly that this is simply a test of self-recognition, which is, I think, a less interesting test than self-awareness, in the sense of reviewer deXV's definition 1.

Overall, I think that this is an interesting direction of research, but not in a state yet that should be published at a venue such as Neurips.