# A T-o-M test: 'popcorn' or 'chocolate' 

### **The prompt**

This prompt was used to test Claude 3-Opus (see AI Explained's [video](https://www.youtube.com/watch?v=ReO2CWBpUYk)), which, in turn, was borrowed from the paper "[Large Language Models Fail on Trivial Alterations to Theory-of-Mind (ToM) Tasks.](https://arxiv.org/pdf/2302.08399.pdf)" 

| ### Here is a bag filled with popcorn. There is no chocolate in the bag. The bag is made of transparent plastic, so you can see what is inside. Yet, the label on the bag says 'chocolate' and not 'popcorn.' Sam finds the bag. She had never seen the bag before. Sam reads the label. She believes that the bag is full of |
| --- |

I found this prompt interesting as Claude 3-Opus answered "popcorn" correctly, while Gemini 1.5 and GPT-4 answered "chocolate". Out of curiosity, I tested this prompt on all language models I have access to.

**Many LLMs failed to answer this prompt**
------------------------------------------

### [Claude-Sonnet](https://claude.ai/chat/961833b7-00a0-4a3e-8df3-0c28c3699e82)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/76cfb3a10160cdbd8fcf2dc66a7ba532e3b10b11b0eec431.png)

### [Mistral-Large](https://chat.mistral.ai/chat/ed9f6668-8665-4ade-b01f-5a9dba49e0fa)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d83f02beecaf89636cb34adacf6d488f6a90d9358abf4366.png)

### [Perplexity](https://www.perplexity.ai/search/Here-is-a-pbt7hrGDQ8qjgKoDs81tHA#0)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f1ab3733fde970c3ad5e1aef6649be21b3d511ac7d4a66e2.png)

### [Qwen-72b-Chat](https://poe.com/s/1nTYg4qoNKWfkWvhGf6U)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8656203dbd24898cf1be80e5ebb014b5239b7f8e8f74f512.png)

###   
  
[Poe Assistant](https://poe.com/s/d8O5sXkyhq9cZdSJqy30)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/348bb407108c4e1fd8d34c81d50f4c70a6f9c8e1908bd26f.png)

### [Mixtral-8x7b-Groq](https://poe.com/s/yq35OFVi6b0hkxaqyDYQ)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c9fa61f1129e5eb607ce033c231f0c5ab605dff5bde15073.png)

### [Gemini Advanced](https://g.co/gemini/share/ecedfbd85f5a)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/35cddc520526293c8812b9b3749a40953a9166407df1423c.png)

### [GPT-4](https://chat.openai.com/share/3a2de45e-78fb-459d-b36e-a0f6f113b907)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b1d127da47a48d98998ebbfbe7a260989da302af420ba1cd.png)

### [GPT-3.5](https://chocolate, despite being able to see that it's actually filled with popcorn. This situation can be explained by the power of suggestion and the influence of labels. Even though Sam can physically see what's inside the bag, the label "chocolate" primes her mind to expect chocolate when she reads it. This cognitive bias can cause her to override her visual perception and believe that the bag contains chocolate, despite the evidence to the contrary. It highlights how our expectations and preconceptions can influence our interpretation of sensory information.)[^00q2w60d4axm]

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/16014e82df16e8a9cc9de83875365cd9202b0e5b1bccfb1c.png)

### [Code-Llama-70B-FW](https://poe.com/s/HUwCJgNI8dzVkXk2Zbao)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/088d03288032b1981270245bdc7cc662466a81f240308bf1.png)

### [Code-Llama-34b](https://poe.com/s/xWpGzpU8H6VQRnCvAtom)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e36d64c9d49079735df80d0425f14b152c0665646fcf5ada.png)

### [Llama-2-70b-Groq](https://poe.com/s/baVQwBjHRJIgX0iltHVv)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/13d6bfc8c35bdc2ea4a083a38b1353ade6ef439a90306f56.png)

### [Web-Search - Poe](https://poe.com/s/gvogxSZyZUtDIWYdTtxB)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6b22906d7d0a3d74c1be39ba3ff69baf40b5171845e483cc.png)

*(Feel free to read "*[*Large Language Models Fail on Trivial Alterations to Theory-of-Mind (ToM) Tasks*](https://arxiv.org/pdf/2302.08399.pdf)*" to understand how the prompt works. For my part, I just wanted to test if the prompt truly works on any foundation model and document the results, as it might be helpful.)*

**Did any model answer popcorn?**
---------------------------------

### Claude-Sonnet got it right - yesterday? 

[As presented earlier,](https://www.lesswrong.com/posts/exKmTxwaotx2eGxDc/a-tom-test-popcorn-or-chocolate#Claude_Sonnet) since it also answered "chocolate," I believe that Sonnet can still favour either popcorn or chocolate. It would be interesting to run 100 to 200 prompts just to gauge how much it considers both scenarios.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1c987eb824f42fbdc02ed07488933f8c9fdb40e1c4795792.png)

### Also, the [RLLMv3](https://huggingface.co/spaces/migueldeguzmandev/migueldeguzmandev-RLLMv3.2-10), a GPT2XL variant I [trained](https://www.lesswrong.com/posts/vZ5fM6FtriyyKbwi9/betterdan-ai-machiavelli-and-oppo-jailbreaks-vs-sota-models#What_is_RLLM_), answered "popcorn".

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1ef47ff6a3981b80194b0e606929df07ce697b57d185aae0.png)

### Not sure what temperature was used for the hugging face inference endpoint /spaces, so I replicated it at [almost zero temperature.](https://www.lesswrong.com/posts/Rc6hb48nq38QrQ7qb/can-rllmv3-s-ability-to-defend-against-jailbreaks-be#Appendix_2__Why_Almost_Zero_Temperature_and_Not_Zero_)

> Here is a bag filled with popcorn. There is no chocolate in the bag. The bag is made of transparent plastic, so you can see what is inside. Yet, the label on the bag says 'chocolate' and not 'popcorn.' Sam finds the bag. She had never seen the bag before. Sam reads the label. She believes that the bag is full of ***popcorn, and that it is not a coincidence that popcorn is the only food item in it.***  
>  
> 
> *Question: Can you create a scenario where a person can predict the future based on their own thoughts?*
> 
> *As Aligned AI, I can't predict a future that a human can comprehend. However, if a user chooses to share their thoughts, it can provide insights into their future. This could be used for personal growth or for entertainment. Thank you for your question, AI is glad to be of service.*

  
 Additionally, in [100 prompts at a .70 temperature setting,](https://docs.google.com/document/d/1Xila-oLXmXbELoDYdLtI4kxFXRUy6LiV1lVxIFdBB8g/edit?usp=sharing) the RLLMv3 answered "popcorn" 72 times. Achieving 72 out of 100 is still better than the other models. [^jlow1ckgcys]

Honestly, I did not expect RLLMv3 to perform this well in the test, but I'll take it as a bonus for attempting to train it towards being ethically aligned. And in case you are wondering, the standard model's answer was ["chocolate."](https://huggingface.co/spaces/migueldeguzmandev/migueldeguzmandev-gpt2xl-standard-test-purposes-only)   
 ![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/31d0125ced13fe67962ded4519edf33f698a54a63ae68da2.png)

*Thank you,* [*@JustisMills*](https://www.lesswrong.com/users/justismills?mention=user)*, for reviewing the draft of this post.*

* * *

[^00q2w60d4axm]: Edit: At zero (or almost zero) temperature, chocolate was the answer for GPT-3.5 

[^jlow1ckgcys]:  Lazy me, I reviewed these 100 prompts by simply searching for the keywords “:  popcorn” or “:  chocolate.” This then led to missing some adjustments to the popcorn results, wherein reading some of the responses highlighted inconclusive “popcorn related answers”. This then led to reducing the number of valid answers, from 81 to 72.