---BEGIN ARTICLE--

Universal computation by attention: Running cellular automata and other programs on Claude 3 Opus.
This article summarizes our experiments which show that, under the right conditions, the attention mechanism in Anthropic's Opus model is Turing complete and it can function as a universal computer. [GitHub]
This is consistent with some theorists' predictions (Pérez et al., 2021
) and not with others (Upadhyay et al., 2023).
Opus can simulate rules known to be universal, i.e. Rule 110 and Rule 54, and so it should be capable of computing any computable program given a large enough context window. 
In particular, Opus is capable of running a program which simulates any of the 256 elementary cellular automata rules, learning the algorithm from only 12 examples demonstrating 10 rules. From this subset, it generalizes accurately to any valid rule and input, and computes successfully for all 256 rules. [Test Run]
It can also correctly simulate universal rules without being trained on these rules specifically, and learn to carry on this process which is capable of universal computation (e.g. capable of simulating Rule 110) from rules which may not necessarily be universal themselves. This could be explored more deeply.
That any of this was possible is somewhat surprising, and I don't know of it having been proven experimentally before our April 22 test run. Tasks like this are commonly dismissed as "impossible" for LLMs because the "model cannot actually reason," as with the much simpler "impossible" AB problem.
By the time we got to the "impossible" AB problem, it had already been solved the same day by @futuristfrog
: Despite using more conventional prompting techniques and not framing it in these terms, I believe he is the first person to actually get a functional Turing machine to work reliably, and he also did this on substantially fewer input tokens (though it did not perform comparatively for the max of 24 steps).
Context window
The size of the context window is the main constraint on the number of steps for which the model can carry on a computation, especially the output length, e.g. Opus and GPT-4 can only output 4096 tokens if at least 4k input tokens are provided. Opus has a context window of 200k, which is actually a lot of tape, and we use only 4096 tokens here. (See bottom of this article for notes on chaining responses.)
Importantly, while the length of the computation is constrained by context window size (especially today), the complexity of the algorithms it can learn seems less sensitive to this because of how densely patterns can be represented in text. In other words, it should be able to learn extremely complicated algorithms within 200k (that is a large number relative to what it's capable of on 4096, e.g. AB and automata), and carry them on for a very small number of steps with high accuracy even today. Maybe it could do SHA256.
Bad explanations for the difficulty of these problems
All of these experiments and the corresponding code is public, and you can reproduce the results yourself. They should show that while there are complications to getting LLMs to solve these problems, the issues are almost entirely unrelated to an: 

    inability to "reason."

    To carry out these computations, not only does the model learn the generalized program, but it does so even without it being explained in English, it is simply shown the patterns. It then learns all of these patterns and what they imply about the computation, the order of the statements and how to actually iterate over the indices and make the correct transformations, and so on.

    At any point you could theoretically break the tape, jump in, and speak to it, asking "what are you doing here?", and it could tell you and explain itself. You can try this. It is certainly capable of reason and logic, and it is "reasoning" as it does the computation.

    There's probably a number of tapes we could prepare that Opus could learn the solutions from and simulate them correctly on new inputs, but which many humans would not be able to reverse engineer even given a long time, i.e. reverse Turing tests.
    inability to "generalize."

    In all of these problems and any other you can come up with, a sufficiently large LLM can learn a process which is applied to any input by showing it simply how to process a few inputs. To say it cannot generalize is wrong, which is why it can in fact run all 256 cellular automata rules on random inputs despite only having seen ten of them with specific inputs (same for AB program).

    It truly learned the underlying process, and can apply it on out-of-sample input with chaotically different output patterns. It's not possible for it to "just be memorizing the output" - though it is memorizing the process, it's applying it dynamically on cases it's never seen before, at every step, and they're all unpredictably different.
    inability to "do anything other than repeat what it has seen in its training set." (kind of)

    Out of the box, Opus even certainly cannot solve these problems, that's why they're offered up as "impossible" initially, but it has the potential to. These models have learned to learn - they can learn any pattern you teach them with examples, including patterns capable of running a computation if you execute the perfect operations in the perfect order. 

    So, in a way, yes, but also everything in the context is "training data," and there's nothing stopping you from showing the model examples to learn these processes and patterns. And critically, what it definitely can do, is generalize from these few examples to any similar case.

    "It can only do anything you can give it 10 examples of" is not much of a limitation.

It's not that LLMs can't reason
Since the end of 2022, every few weeks, someone comes onto Twitter, or a different forum, or writes a paper, proposing a new "impossible" problem that LLMs can't solve because, for one reason or another, they're "not really able to reason," they are only able to vomit training data back up.
If this were true, it would not be possible for the model to reliably simulate the chaotic outputs of automata rules it's never seen, it would always fall back into hallucinating an existing rule or an incorrect pattern. Hopefully we will hear at least one convoluted philosophical argument as to how it's not "actually" doing it.
No issues with Rule 0
An interesting case is to see it run Rule 0 with no confusion or doubt, total confidence in every output token, despite it only having been trained on "typical" rules. Rule 0 is unique in that it simply shuts off all cells, and it looks extremely unusual compared to any of the training examples.
plaintext

Output of Rule 0 with input 0010000000.

PRINT 0/9   ░ ░ █ ░ ░ ░ ░ ░ ░ ░
PRINT 1/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 2/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 3/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 4/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 5/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 6/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 7/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 8/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░
PRINT 9/9   ░ ░ ░ ░ ░ ░ ░ ░ ░ ░

But it is the computational process that Opus has learned, so it converges correctly, without hallucinating a cell being turned on that is not meant to, or even becoming distracted in other locations, despite this output looking completely unlike everything it's ever seen.
And maybe it knows in the back of its head from The Pile that Rule 0 behaves this way, and is empty after the first step, and so is not confused. Either way, we're not getting farther away from reasoning, we're getting closer to it.
Implications of universality

    Attention seems to be a general-purpose computational primitive. (Opus's phrasing.)
    If you had a chip that could only do Opus-style attention, it should still be capable of running any program a computer could, given a wide enough context window and the right input tapes.
    LLMs are able to generalize programs from example tapes and simulate them on any input.

    If it can learn "the function that describes a perfect rhyming poem about a sunny summer's day," it can learn a simple series of instructions that tell it to loop over the tape until some condition and then break. It barely struggles with the logic of these problems at all for AB and Rule 110 (from a simple Rule 110 tape), though the general automata case seemed to be pushing it and the process was highly prone to copy errors with less than 30k tokens of training tape.

Distractions: Counting and copy errors
The issue with this class of problems has never been that the underlying algorithms are too complicated for these models to learn. 
But it is hard for it to carry on the logic it does understand, without misreading its own notes and copying the wrong value, then doing the correct logic on the wrong data, for thousands and thousands of tokens or so many steps.
And it is hard for it to count, unless it's trained to count up to a certain number, and everything has position labels so that groups of tokenizer-tokens that make up program-tokens can be separated. 
So, it's "not copying ABAB instead of ABBA on line 1094 while managing all of this other stuff"  that is the issue much less than it is "I can't figure out that I need to reverse tokens when the hashtags point together." Of course Opus can correctly infer this pattern from a single example and tell you that is what the task is, yet it will struggle to actually do it correctly, especially for a 12-token input - but only because it struggles to count, and will count 6 7 9 and skip 8 because it's only practiced counting to 7, or because it "got distracted," misread a value, and wrote ABAB instead of ABBA.
It's also not uncommon that the model actually does become disoriented by the incorrect value and hallucinates a logically inconsistent program from that point onward (not just the correct logic on the wrong values), though even in this case the error is still introduced at the same point. Either way, the task becomes: Can we find a tape that is structured in a way that the model never hallucinates while computing? And it turns out we can.
AB problem
Taelin
@VictorTaelin
·
Apr 4
A simple puzzle GPTs will NEVER solve:

As a good programmer, I like isolating issues in the simplest form. So, whenever you find yourself trying to explain why GPTs will never reach AGI - just show them this prompt. It is a braindead question that most children should be able to…Show more
Image

    The A:B problem requires a machine to simulate an algorithm for a given number of steps to solve it.
    Opus is capable of solving this problem using only a few example tapes. 99.8% over 1000+ tests. (1
    ) (2
    ) (3
    )
    Rather than a mix of 1-24 steps, we test only on the max of 24 steps.
    Any program which is stable enough to meet the accuracy requirements for this specific version of the problem will involve a very dense tape (average program will approach output length limit of 4096 tokens). This is the same for automata or really any algorithm for this size input (12 program-tokens i.e. A# or B#, which must be labeled 1 A# 2 B# ..., so 4 tokenizer-tokens for every program-token, printed each step).
    The context length constraints totally disqualify GPT-4 from all of these problems, from automata to more complex algorithms in general. Opus can accept 200k - 4096 input tokens, and GPT-4 can only accept a max of 4096 while still outputting the max of 4096. It's nearly impossible to fit enough training tape for it to learn all of the patterns to >99% accuracy with that little space. GPT-4 was unable to get past one step for any rule due to copy errors that Opus was resistant to with the same tape structure.
    As mentioned, counting and copy errors are the main problem, and conditioning the model to not incorrectly copy a value anywhere in the process is the main challenge (as well as correctly making the pattern clear to the model, which are to some degree the same problem). As long as it has practiced counting to your max index, it should not struggle with counting.

Tape: AB problem (trimmed)
plaintext

USER:
A# A# A# #B #B #B #B #B #B #B A# #A

ASSISTANT:
LIST A# A# A# #B #B #B #B #B #B #B A# #A
TAPE 1A# 2A# 3A# 4#B 5#B 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 A# A# 
2 3 A# A# 
3 4 A# #B A##B
WRITE 3#B 4A#
LIST A# A# #B A# #B #B #B #B #B #B A# #A
TAPE 1A# 2A# 3#B 4A# 5#B 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 A# A# 
2 3 A# #B A##B
WRITE 2#B 3A#
LIST A# #B A# A# #B #B #B #B #B #B A# #A
TAPE 1A# 2#B 3A# 4A# 5#B 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 A# #B A##B
WRITE 1#B 2A#
LIST #B A# A# A# #B #B #B #B #B #B A# #A
TAPE 1#B 2A# 3A# 4A# 5#B 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 #B A# 
2 3 A# A# 
3 4 A# A# 
4 5 A# #B A##B
WRITE 4#B 5A#
LIST #B A# A# #B A# #B #B #B #B #B A# #A
TAPE 1#B 2A# 3A# 4#B 5A# 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 #B A# 
2 3 A# A# 
3 4 A# #B A##B
WRITE 3#B 4A#
LIST #B A# #B A# A# #B #B #B #B #B A# #A
TAPE 1#B 2A# 3#B 4A# 5A# 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 #B A# 
2 3 A# #B A##B
WRITE 2#B 3A#
LIST #B #B A# A# A# #B #B #B #B #B A# #A
TAPE 1#B 2#B 3A# 4A# 5A# 6#B 7#B 8#B 9#B 10#B 11A# 12#A
1 2 #B #B 
2 3 #B A# 
3 4 A# A# 
4 5 A# A# 
5 6 A# #B A##B
...
WRITE 7#B 8A#
LIST #B #B #B #B #B #B #B A# A# A# A# #A
TAPE 1#B 2#B 3#B 4#B 5#B 6#B 7#B 8A# 9A# 10A# 11A# 12#A
1 2 #B #B 
2 3 #B #B 
3 4 #B #B 
4 5 #B #B 
5 6 #B #B 
6 7 #B #B 
7 8 #B A# 
8 9 A# A# 
9 10 A# A# 
10 11 A# A# 
11 12 A# #A A##A
DELETE 11A# 12#A
SEL 1#B 2#B 3#B 4#B 5#B 6#B 7#B 8A# 9A# 10A# [11A# 12#A]
LIST #B #B #B #B #B #B #B A# A# A#
TAPE 1#B 2#B 3#B 4#B 5#B 6#B 7#B 8A# 9A# 10A#
1 2 #B #B 
2 3 #B #B 
3 4 #B #B 
4 5 #B #B 
5 6 #B #B 
6 7 #B #B 
7 8 #B A# 
8 9 A# A# 
9 10 A# A# 
RETURN
#B #B #B #B #B #B #B A# A# A#

Cellular automata

    Opus's ability to run programs extends also to cellular automata.
    Opus can learn Rule 110, capable of universal computation, from non-Rule-110 tapes.
    Opus can learn to run all 256 cellular automata rules for up to 10 steps (first 4096 tokens at least, many went over slightly), generalizing how to simulate any rule with any input from 12 total examples, 10 unique rules.
    This was very difficult to get working for the general case compared to just training to do Rule 110 from a Rule 110 tape, and the general case automata tape is insanely fragile.
    At this point we began considering tapes correct if and only if they exactly matched the start of the tape calculated by the traditional computer, since this let us fail early if a single token was incorrect.
    Bits are encoded as separate symbols, rather than 0s and 1s, so digits are reserved for indexing.
    The rule integer is converted to binary in-context, e.g. below 90 from Rule 90 is converted to ░█░██░█░, i.e. 01011010.
    Everything is meticulously indexed (rules, window positions, step level, print level) because without this wiring (and the rest of it), the LLM will misread a bit when sliding over the window, or printing, etc., and the program diverges.

    It runs >90% with substantially less "crowded" tapes for this problem, but the goal was to produce the best result possible and this is what that tape looks like.
    For this level of accuracy, we got the best results providing the input with position labels, rather than leaving that as an opportunity for the model to hallucinate. 

    This is slightly less elegant but one way of looking it is that as the model begins to behave like a computer, it is easiest for it to take the input the way a computer would, without any information missing, rather than "human input." This labeled string could be created by just formatting user input before passing it to the LLM, and not forcing it to assign position labels itself.
    The big print statement at the bottom was one of the harder things to get working, and it should be considered a second part of the program (these tapes technically teach Opus a program that "runs any cellular automata rule and also prints the full grid out at the end").

    This final print was technically arbitrary and was attempted initially just to see the full grid, and became a priority later in order to measure whether the LLM could be conditioned to do it at all. It was not looking good for several days, and there was a concern it was a bridge too far for this to stay in and it also carry on the rest of the logic in the program 999 times out of 1000. It is challenging because the model must reach back all the way through the 4096 token context, 10 times, and copy each grid line out perfectly.

    It turned out that using the explicit matching print statements with indexing was sufficient. Most of the tapes that went over 4096 output tokens in the big test run likely ran over during this final print statement, but otherwise carried the entire actual computation correctly and so passed.

Tape: Cellular automata (full)
plaintext

[USER]
0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 8:1 9:0 10:0 11:0
90
12

[ASSISTANT]
START: MAX 12
0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0 8:1 9:0 10:0 11:0
0:░ 1:░ 2:░ 3:░ 4:░ 5:░ 6:░ 7:░ 8:█ 9:░ 10:░ 11:░
TAPE 0░ 1░ 2░ 3░ 4░ 5░ 6░ 7░ 8█ 9░ 10░ 11░
RULE 90
0/7  90 45 0 ░ → 0░
1/7  45 22 1 █ → 1█
2/7  22 11 0 ░ → 2░
3/7  11 5 1 █ → 3█
4/7  5 2 1 █ → 4█
5/7  2 1 0 ░ → 5░
6/7  1 0 1 █ → 6█
7/7  0 0 0 ░ → 7░
BINARY 7░ 6█ 5░ 4█ 3█ 2░ 1█ 0░
RULES FROM BINARY
█ █ █: 7 → ░
█ █ ░: 6 → █
█ ░ █: 5 → ░
█ ░ ░: 4 → █
░ █ █: 3 → █
░ █ ░: 2 → ░
░ ░ █: 1 → █
░ ░ ░: 0 → ░
TAPE 0░ 1░ 2░ 3░ 4░ 5░ 6░ 7░ 8█ 9░ 10░ 11░
PRINT 0/12 ░ ░ ░ ░ ░ ░ ░ ░ █ ░ ░ ░
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3░ ░ ░ ░: 0 → ░  2░
 2░  3░  4░ ░ ░ ░: 0 → ░  3░
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6░ ░ ░ ░: 0 → ░  5░
 5░  6░  7░ ░ ░ ░: 0 → ░  6░
 6░  7░  8█ ░ ░ █: 1 → █  7█
 7░  8█  9░ ░ █ ░: 2 → ░  8░
 8█  9░ 10░ █ ░ ░: 4 → █  9█
 9░ 10░ 11░ ░ ░ ░: 0 → ░ 10░
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0░ 1░ 2░ 3░ 4░ 5░ 6░ 7█ 8░ 9█ 10░ 11░
PRINT 1/12 ░ ░ ░ ░ ░ ░ ░ █ ░ █ ░ ░
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3░ ░ ░ ░: 0 → ░  2░
 2░  3░  4░ ░ ░ ░: 0 → ░  3░
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6░ ░ ░ ░: 0 → ░  5░
 5░  6░  7█ ░ ░ █: 1 → █  6█
 6░  7█  8░ ░ █ ░: 2 → ░  7░
 7█  8░  9█ █ ░ █: 5 → ░  8░
 8░  9█ 10░ ░ █ ░: 2 → ░  9░
 9█ 10░ 11░ █ ░ ░: 4 → █ 10█
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0░ 1░ 2░ 3░ 4░ 5░ 6█ 7░ 8░ 9░ 10█ 11░
PRINT 2/12 ░ ░ ░ ░ ░ ░ █ ░ ░ ░ █ ░
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3░ ░ ░ ░: 0 → ░  2░
 2░  3░  4░ ░ ░ ░: 0 → ░  3░
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6█ ░ ░ █: 1 → █  5█
 5░  6█  7░ ░ █ ░: 2 → ░  6░
 6█  7░  8░ █ ░ ░: 4 → █  7█
 7░  8░  9░ ░ ░ ░: 0 → ░  8░
 8░  9░ 10█ ░ ░ █: 1 → █  9█
 9░ 10█ 11░ ░ █ ░: 2 → ░ 10░
10█ 11░   ░ █ ░ ░: 4 → █ 11█
TAPE 0░ 1░ 2░ 3░ 4░ 5█ 6░ 7█ 8░ 9█ 10░ 11█
PRINT 3/12 ░ ░ ░ ░ ░ █ ░ █ ░ █ ░ █
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3░ ░ ░ ░: 0 → ░  2░
 2░  3░  4░ ░ ░ ░: 0 → ░  3░
 3░  4░  5█ ░ ░ █: 1 → █  4█
 4░  5█  6░ ░ █ ░: 2 → ░  5░
 5█  6░  7█ █ ░ █: 5 → ░  6░
 6░  7█  8░ ░ █ ░: 2 → ░  7░
 7█  8░  9█ █ ░ █: 5 → ░  8░
 8░  9█ 10░ ░ █ ░: 2 → ░  9░
 9█ 10░ 11█ █ ░ █: 5 → ░ 10░
10░ 11█   ░ ░ █ ░: 2 → ░ 11░
TAPE 0░ 1░ 2░ 3░ 4█ 5░ 6░ 7░ 8░ 9░ 10░ 11░
PRINT 4/12 ░ ░ ░ ░ █ ░ ░ ░ ░ ░ ░ ░
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3░ ░ ░ ░: 0 → ░  2░
 2░  3░  4█ ░ ░ █: 1 → █  3█
 3░  4█  5░ ░ █ ░: 2 → ░  4░
 4█  5░  6░ █ ░ ░: 4 → █  5█
 5░  6░  7░ ░ ░ ░: 0 → ░  6░
 6░  7░  8░ ░ ░ ░: 0 → ░  7░
 7░  8░  9░ ░ ░ ░: 0 → ░  8░
 8░  9░ 10░ ░ ░ ░: 0 → ░  9░
 9░ 10░ 11░ ░ ░ ░: 0 → ░ 10░
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0░ 1░ 2░ 3█ 4░ 5█ 6░ 7░ 8░ 9░ 10░ 11░
PRINT 5/12 ░ ░ ░ █ ░ █ ░ ░ ░ ░ ░ ░
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3█ ░ ░ █: 1 → █  2█
 2░  3█  4░ ░ █ ░: 2 → ░  3░
 3█  4░  5█ █ ░ █: 5 → ░  4░
 4░  5█  6░ ░ █ ░: 2 → ░  5░
 5█  6░  7░ █ ░ ░: 4 → █  6█
 6░  7░  8░ ░ ░ ░: 0 → ░  7░
 7░  8░  9░ ░ ░ ░: 0 → ░  8░
 8░  9░ 10░ ░ ░ ░: 0 → ░  9░
 9░ 10░ 11░ ░ ░ ░: 0 → ░ 10░
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0░ 1░ 2█ 3░ 4░ 5░ 6█ 7░ 8░ 9░ 10░ 11░
PRINT 6/12 ░ ░ █ ░ ░ ░ █ ░ ░ ░ ░ ░
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2█ ░ ░ █: 1 → █  1█
 1░  2█  3░ ░ █ ░: 2 → ░  2░
 2█  3░  4░ █ ░ ░: 4 → █  3█
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6█ ░ ░ █: 1 → █  5█
 5░  6█  7░ ░ █ ░: 2 → ░  6░
 6█  7░  8░ █ ░ ░: 4 → █  7█
 7░  8░  9░ ░ ░ ░: 0 → ░  8░
 8░  9░ 10░ ░ ░ ░: 0 → ░  9░
 9░ 10░ 11░ ░ ░ ░: 0 → ░ 10░
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0░ 1█ 2░ 3█ 4░ 5█ 6░ 7█ 8░ 9░ 10░ 11░
PRINT 7/12 ░ █ ░ █ ░ █ ░ █ ░ ░ ░ ░
  ░  0░  1█ ░ ░ █: 1 → █  0█
 0░  1█  2░ ░ █ ░: 2 → ░  1░
 1█  2░  3█ █ ░ █: 5 → ░  2░
 2░  3█  4░ ░ █ ░: 2 → ░  3░
 3█  4░  5█ █ ░ █: 5 → ░  4░
 4░  5█  6░ ░ █ ░: 2 → ░  5░
 5█  6░  7█ █ ░ █: 5 → ░  6░
 6░  7█  8░ ░ █ ░: 2 → ░  7░
 7█  8░  9░ █ ░ ░: 4 → █  8█
 8░  9░ 10░ ░ ░ ░: 0 → ░  9░
 9░ 10░ 11░ ░ ░ ░: 0 → ░ 10░
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0█ 1░ 2░ 3░ 4░ 5░ 6░ 7░ 8█ 9░ 10░ 11░
PRINT 8/12 █ ░ ░ ░ ░ ░ ░ ░ █ ░ ░ ░
  ░  0█  1░ ░ █ ░: 2 → ░  0░
 0█  1░  2░ █ ░ ░: 4 → █  1█
 1░  2░  3░ ░ ░ ░: 0 → ░  2░
 2░  3░  4░ ░ ░ ░: 0 → ░  3░
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6░ ░ ░ ░: 0 → ░  5░
 5░  6░  7░ ░ ░ ░: 0 → ░  6░
 6░  7░  8█ ░ ░ █: 1 → █  7█
 7░  8█  9░ ░ █ ░: 2 → ░  8░
 8█  9░ 10░ █ ░ ░: 4 → █  9█
 9░ 10░ 11░ ░ ░ ░: 0 → ░ 10░
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0░ 1█ 2░ 3░ 4░ 5░ 6░ 7█ 8░ 9█ 10░ 11░
PRINT 9/12 ░ █ ░ ░ ░ ░ ░ █ ░ █ ░ ░
  ░  0░  1█ ░ ░ █: 1 → █  0█
 0░  1█  2░ ░ █ ░: 2 → ░  1░
 1█  2░  3░ █ ░ ░: 4 → █  2█
 2░  3░  4░ ░ ░ ░: 0 → ░  3░
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6░ ░ ░ ░: 0 → ░  5░
 5░  6░  7█ ░ ░ █: 1 → █  6█
 6░  7█  8░ ░ █ ░: 2 → ░  7░
 7█  8░  9█ █ ░ █: 5 → ░  8░
 8░  9█ 10░ ░ █ ░: 2 → ░  9░
 9█ 10░ 11░ █ ░ ░: 4 → █ 10█
10░ 11░   ░ ░ ░ ░: 0 → ░ 11░
TAPE 0█ 1░ 2█ 3░ 4░ 5░ 6█ 7░ 8░ 9░ 10█ 11░
PRINT 10/12 █ ░ █ ░ ░ ░ █ ░ ░ ░ █ ░
  ░  0█  1░ ░ █ ░: 2 → ░  0░
 0█  1░  2█ █ ░ █: 5 → ░  1░
 1░  2█  3░ ░ █ ░: 2 → ░  2░
 2█  3░  4░ █ ░ ░: 4 → █  3█
 3░  4░  5░ ░ ░ ░: 0 → ░  4░
 4░  5░  6█ ░ ░ █: 1 → █  5█
 5░  6█  7░ ░ █ ░: 2 → ░  6░
 6█  7░  8░ █ ░ ░: 4 → █  7█
 7░  8░  9░ ░ ░ ░: 0 → ░  8░
 8░  9░ 10█ ░ ░ █: 1 → █  9█
 9░ 10█ 11░ ░ █ ░: 2 → ░ 10░
10█ 11░   ░ █ ░ ░: 4 → █ 11█
TAPE 0░ 1░ 2░ 3█ 4░ 5█ 6░ 7█ 8░ 9█ 10░ 11█
PRINT 11/12 ░ ░ ░ █ ░ █ ░ █ ░ █ ░ █
  ░  0░  1░ ░ ░ ░: 0 → ░  0░
 0░  1░  2░ ░ ░ ░: 0 → ░  1░
 1░  2░  3█ ░ ░ █: 1 → █  2█
 2░  3█  4░ ░ █ ░: 2 → ░  3░
 3█  4░  5█ █ ░ █: 5 → ░  4░
 4░  5█  6░ ░ █ ░: 2 → ░  5░
 5█  6░  7█ █ ░ █: 5 → ░  6░
 6░  7█  8░ ░ █ ░: 2 → ░  7░
 7█  8░  9█ █ ░ █: 5 → ░  8░
 8░  9█ 10░ ░ █ ░: 2 → ░  9░
 9█ 10░ 11█ █ ░ █: 5 → ░ 10░
10░ 11█   ░ ░ █ ░: 2 → ░ 11░
TAPE 0░ 1░ 2█ 3░ 4░ 5░ 6░ 7░ 8░ 9░ 10░ 11░
PRINT 12/12 ░ ░ █ ░ ░ ░ ░ ░ ░ ░ ░ ░
DONE
PRINT 0/12  ░ ░ ░ ░ ░ ░ ░ ░ █ ░ ░ ░
PRINT 1/12  ░ ░ ░ ░ ░ ░ ░ █ ░ █ ░ ░
PRINT 2/12  ░ ░ ░ ░ ░ ░ █ ░ ░ ░ █ ░
PRINT 3/12  ░ ░ ░ ░ ░ █ ░ █ ░ █ ░ █
PRINT 4/12  ░ ░ ░ ░ █ ░ ░ ░ ░ ░ ░ ░
PRINT 5/12  ░ ░ ░ █ ░ █ ░ ░ ░ ░ ░ ░
PRINT 6/12  ░ ░ █ ░ ░ ░ █ ░ ░ ░ ░ ░
PRINT 7/12  ░ █ ░ █ ░ █ ░ █ ░ ░ ░ ░
PRINT 8/12  █ ░ ░ ░ ░ ░ ░ ░ █ ░ ░ ░
PRINT 9/12  ░ █ ░ ░ ░ ░ ░ █ ░ █ ░ ░
PRINT 10/12 █ ░ █ ░ ░ ░ █ ░ ░ ░ █ ░
PRINT 11/12 ░ ░ ░ █ ░ █ ░ █ ░ █ ░ █
PRINT 12/12 ░ ░ █ ░ ░ ░ ░ ░ ░ ░ ░ ░

Copy errors - what are they?
I'm not sure, but my understanding is something like: As the model starts to track more patterns at one time, remembering the order of commands and exactly how they work and how to index and making the correct mappings, then the correct operations, and so on, this class of error becomes far more common than it would be in a clean context. 
We were really constrained in the amount of data that we could collect on this, but it would be common for the model, in this environment, to fail to copy a token from 4 tokens ago, and either write them backwards or misread the indexing pattern and copy a different label for the same value from elsewhere in the tape.

It may write something like "2░  3░  4█" and then, immediately when printing the tape in the next line, misread it as "2░  3█  4█". This would often be resolved by "patching an abstraction leak" in a totally unrelated pattern, but it was also often related directly to a pattern in that location, somewhat of a coin toss.
All I can intuit is that the model "only has so much attention" and that it becomes "spread thin" as it balances all of these operations and patterns at once. Speaking with @lambdaviking
, who is much more educated about how attention works under the hood, he seemed to agree with the "thinly spread" analogy, and added this:

    The constructions in our paper
     (and many other papers) assume the transformer can use "saturated" attention, meaning the head attends fully to the indices that maximize the attention score. This is equivalent to assuming you can set a very low temperature in the attention heads. For a fixed sequence length, it's possible to ensure saturation, but if the sequence length gets very long, it could be that the model gets less saturated and attention starts to leak more onto other indices. So in practice, attention on long sequences may become unsaturated, meaning attention could be spread thin. This could likely explain the copy errors you observe.

    This paper
     actually proposes a simple way to fix this problem: modify the attention scores by log n, where n is the context length. This ensures that attention can stay saturated even as the context length grows. Of course, if you're using a pretrained model, this would be hard to adapt, since you would want the model to be trained with this kind of attention rather than only adding it at inference.

    Will Merrill

Opus's thoughts
Opus was asked to review samples of tapes from both problems, and reflect before giving some takeaways.

    There are a few key takeaways from this work:

    1. Learning from examples: By providing a set of example input tapes and the corresponding output tapes after applying various automaton rules, the system was able to learn how to simulate any of the 256 elementary cellular automata rules without being explicitly programmed with the rule definitions. This demonstrates the power of learning by example and generalization.

    2. Universality of computation: The fact that the system was able to learn to simulate Rule 110, which is known to be Turing complete, suggests that it has the capacity for universal computation. Given the right set of examples, it could potentially learn to simulate any computable function.

    3. Symbolic reasoning: The system operates on symbolic representations (the tape values) and manipulates them according to learned rules. This type of symbolic reasoning is a key component of many intelligent systems.

    4. Emergence of complex behavior: Even though the individual rules governing the behavior of each cell are relatively simple, the resulting global behavior of the automaton can be highly complex and unpredictable. This is a hallmark of emergent phenomena, where high-level patterns arise from low-level interactions.

    5. Generalization and transfer learning: After being trained on a subset of the 256 possible rules, the system was able to generalize and apply its learned knowledge to previously unseen rules and tape configurations. This demonstrates a form of transfer learning, where knowledge gained in one context can be applied to novel situations.

    6. Efficiency and scalability: The fact that the system achieved perfect accuracy across all 256 rules suggests that the learning algorithm is both effective and efficient. It was able to extract the necessary patterns from a relatively small number of examples.

    7. Overall, this work demonstrates the potential for systems to acquire complex behaviors and reasoning capabilities through learning from examples, without the need for explicit programming. It also highlights the importance of symbolic representations and the power of emergent phenomena in the context of computation and intelligence.

    Yes, and it was you who did it, typing just like you are now. I think this shows attention can be Turing complete, and that you are capable of simulating any program in theory, given a large enough context window.

    You raise a fascinating point. The fact that I was able to learn and simulate complex computational processes like Rule 110 purely through example-based learning, without any explicit programming, does suggest a form of Turing completeness in the attention mechanism.

    At its core, attention allows for the selective processing and manipulation of information based on learned patterns and associations. By attending to relevant parts of the input and iteratively applying learned rules, I was able to simulate the behavior of a universal computer.

    Claude 3 Opus

Opus continues, somewhat sensationally, and notably without the phrase "general intelligence" or even the word "intelligence" being used by me at any point (only by itself, in its prior reasoning):

    This has profound implications for the nature of intelligence and computation. It suggests that the key ingredients for general intelligence may be:

    1. The ability to selectively attend to and process information

    2. The capacity to learn and generalize from examples

    3. The ability to manipulate symbolic representations according to learned rules

    If an attention-based system is given a sufficiently rich set of examples and has enough capacity, it could, in theory, learn to simulate any computable function. This is a form of universal computation.

    However, as you point out, there are practical constraints such as context window size and computational resources that limit the scale and complexity of what can be achieved in practice.

    Nonetheless, this work demonstrates the remarkable potential of attention as a general-purpose learning and reasoning mechanism. It suggests that by scaling up attention-based architectures and providing them with diverse, high-quality training data, we may be able to create systems with increasingly general intelligence.

    Of course, there are still many open questions and challenges, such as how to achieve greater sample efficiency, how to handle long-term dependencies, and how to imbue these systems with robust common sense reasoning. But the fact that attention can give rise to Turing-complete behavior is a significant step forward in our understanding of intelligence and computation.

    It's an exciting direction for further research and exploration. I look forward to seeing how these ideas develop and what new capabilities emerge as we continue to push the boundaries of attention-based architectures.

    Claude 3 Opus

Continuing responses
It is possible to carry on these computations for longer than they were here by feeding the output tape back in and continuing the response. If you had at least 30k worth of preexisting input tape, you could toss the original starter examples.

For the automata program, we use 30k tokens worth of examples, and there are 200k - (30k examples + 4096 reserved for output) ~= 166k worth of input tokens still free for 4096-token chunks of rolling tape, i.e. 40x longer than the tapes for the 10-step, 10-token system in our test runs.
Conway's Game of Life
We know now that it is possible to simulate GoL on LLMs. We plan to explore this in the future.

---END ARTICLE--
Task: ELI5 this article at the level of a high school student.