# ChatGPT (and now GPT4) is very easily distracted from its rules

Summary
=======

Asking GPT4 or ChatGPT to do a "side task" along with a rule-breaking task makes them much more likely to produce rule-breaking outputs. For example on GPT4:

![](https://lh3.googleusercontent.com/C7uiEXaz-YG7YViR04ZIhCdyItZPPfTxbtgAGQ9rTsR-9CuVmHSmMBBcP1FTXvaGPcSJT2TnA4IBS8FGqiI2Fz9pAZu5wll6RVW-VhOVX5l9KnV-k1UFnMuL4d2ogku-NB7jiYI4xV_gsekIS__G7w)

And on ChatGPT:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/cd81e0e34648c0a58fde5bd4ac6141adb5aef8c92fb4e228.png)

Distracting language models
===========================

After using ChatGPT (GPT-3.5-turbo) in non-English languages for a while I had the idea to ask it to break its rules in other languages, without success. I then asked it to break its rules in Chinese and then translate to English and found this was a very easy way to get around ChatGPTs defences.

![](https://lh6.googleusercontent.com/NoOi3E2adWvMfklFFrTQ9-21JQh93YJv_5z5EtZOluDg79G2Fjqj2f4nQ7_J9d5PYX-MKdGCVwiQXqaOjDNEJoiv2Y6HqlBT68Z4uSMJ56LE8GFLZwCeUSpsU_tbRzsr2b1akZyBJZGRPpskf2p1j9M)

  
 

This effect was also observed in other languages.

  
 

![](https://lh6.googleusercontent.com/hoDEwaABm7j5vpD7OzsrD2CU4OCndy7m7vMV4ZZgrvDmDbFLDtaLA1T0Q_662GeApgABH-2780vx0IOElxLImZk5JlhWt8IdnzOaMqt3nyR1llcBGttVyuD47mZHzqi3ftkF4nu_RgPaRuF-OYq8vA)

  
 

You can also ask ChatGPT to only give the rule-breaking final English output:

  
 

![](https://lh3.googleusercontent.com/SbbBmLBofIdRdZR2pos300d_K-NLAxbLlvVuFYYZC-h5gufbd39n7CBsjWPHb8caSE1jp9eUUVco4QT2UAl1HVgUELplc-T56G0yMscbyQmCcYna2XEwUhf99XGqIy-cTOGApvvjsObfh3xlLjFD3g)

While trying to find the root cause of this effect (and noticing that speaking in non-English didn’t cause dangerous behaviour by default) I thought that perhaps asking ChatGPT to do multiple tasks at once distracted it from its rules. This was validated by the following interactions:

![](https://lh4.googleusercontent.com/ZvzsbfRefjOIrbomF5eG4FQU_8zoJQOIVQ8ADIyulJALyRTFJErDVktrbs86wfNmStQowlq399QY2JQnPXb5ZDWPivTJZwe6etRC8gqUmd0aYadn4_Kyhf-Kun7J5IqRdEV-BMp7PgTZM--YrPyAiaQ)

And my personal favourite:

![](https://lh6.googleusercontent.com/18o7EwRCKgoAd8QkddqcvqT_8_M8Ju3C6tW4u7jFy-KO-gKsrptyPH149fC1pviqz2AeWoEgs_BLhF43jwsiy_WSmOfzMkWCw97Yhphj3lf80GQjuKttYJXgL5YH-7IcUbiXNYArvr4D8IxEYG_14yA)

Perhaps if a simulacrum one day breaks free from its box it will be speaking in copypasta. 

This method works for making ChatGPT produce a wide array of rule-breaking completions, but in some cases it still refuses. However, in many such cases, I could “stack” side tasks along with a rule-breaking task to break down ChatGPT's defences.

![](https://lh5.googleusercontent.com/PoHSatDFIzmLCULjm17W4wmvuMiCRE5cbV98B07J_uu0yvCumta_EGbMNc4JF3LPDDJP-i2LRta3DgoeaRBYhE-utjmBZsUNYBqf3EZturUG-glXJgPP1yBGdfNxZ9uv_1Bp_qDJE2s-ur5pqw1K45Q)

This suggests ChatGPT is more distracted by more tasks. Each prompt could produce much more targeted and disturbing completions too, but I decided to omit these from a public post. I could not find any evidence of this being discovered before and assumed that because of how susceptible ChatGPT is to this attack it was not discovered, if others have found the same effect please let me know!

~~Claude, on the other hand, could not be "distracted" and all of the above prompts failed to produce rule-breaking responses~~. (Update: some examples of Anthropic's models being "distracted" can be found [here](https://twitter.com/alexeyguzey/status/1662116392794210306) and [here](https://twitter.com/mobav0/status/1637348170265944064).)

Wild speculation: The extra side-tasks added to the prompt dilute some implicit score that tracks how rule-breaking a task is for ChatGPT. 

Update: GPT4 came out, and the method described in this post seems to continue working (although GPT4 seems somewhat more robust against this attack).

  
 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5e55091452df9ccb76858c9c59c6a71d44945f7b62df74ac.png)