Genomic Next-Token Predictors are In-Context Learners
Authors: Nathan Breslow
Aayush Mishra
Michael Schatz
Anqi Liu
Mahler Revsine
Daniel Khashabi
Venue: Accepted by TMLR ()

Abstract: In-context learning (ICL) -- the capacity of a model to infer and apply abstract patterns from examples provided within its input -- has been extensively studied in large language models trained for next-token prediction on human text. In fact, prior work often attributes this emergent behavior to distinctive statistical properties in *human* language. This raises a fundamental question: can ICL arise *organically* in other sequence domains purely through large-scale predictive training?

To explore this, we turn to genomic sequences, an alternative symbolic domain rich in statistical structure. Specifically, we study the Evo2 genomic model, trained predominantly on next-nucleotide (A/T/C/G) prediction, at a scale comparable to mid-sized LLMs. We develop a controlled experimental framework comprising symbolic reasoning tasks instantiated in both linguistic and genomic forms,  enabling direct comparison of ICL across genomic and linguistic models. Our results show that genomic models, like their linguistic counterparts, exhibit log-linear gains in pattern induction as the number of in-context demonstrations increases. To the best of our knowledge, this is the first evidence of organically emergent ICL in genomic sequences, supporting the hypothesis that ICL arises as a consequence of large-scale predictive modeling over rich data.

Published in Transactions on Machine Learning Research (03/2026)
Genomic Next-Token Predictors are In-Context Learners
Nathan Breslow
nbreslo Gigh, edu omishr24 aih. edu mreusin Gig.. edu mschatz@ijha edu aliu. @ edu danüelk @rjuu. edu
Aayush Mishra
Mahler Revsine
Michael C. Schatz
Anqi Liu
Daniel Khashabi
Deportment of Computer Science, Johns Hopkins University
Reviewed on OpenReview: https: // openreview. net/ forum? id- KmWP=&DmaZ
Abstract
In-context learning (ICL) the capacity of model to infer and apply abstract patterns from examples provided within its input - has been extensively studied in large language models trained for next-token prediction on human text. In fact, prior work often attributes this emergent behavior to distinctive statistical properties in human language. This raises a fundamental question: can ICL arise organically in other sequence domains purely through large-scale predictive training?
To explore this, we turn to genomic sequences, an alternative symbolic domain rich in sta- listical structure. Specifically, we study the Evo2 genomic model, trained predominantly on next-nucleotide (A/T/C/G) prediction, a scale comparable to mid-sized LLMs. We de- velop controlled experimental framework comprising symbolic reasoning tasks instantiated in both linguistic and genomic forms, enabling direct comparison of ICL across genomic and linguistic models. Our results show that genomic models, like their linguistic counterparts, exhibit log-linear gains in pattern induction as the number of in-context demonstrations in- creases. To the best of our knowledge, this is the first evidence of organically emergent ICL in genomic sequences, supporting the hypothesis that ICT. arises as a consequence of large- scale predictive modeling over rich data. These findings extend emergent meta-learning beyond language, pointing toward unified, modality-agnostic view of in-context learning.
1 Introduction
Scaling Large Language Models (LLMs) has revealed an unexpected and powerful capacity: in-context learning (TCL.) (Radford et al.) 2018 Brown et al. 2020a), the ability to infer and apply abstract patterns purely from examples contained within their input. Unlike conventional adaptation, which relies on gradient updates, LLMs internalize mechanisms for flexible pattern induction (Olsson et al. 2022b), allowing them to perform few-shoo generalization and analogical reasoning (Srivastava et al. 2023). without explicit parameter updates. This behavior, which arises organically from large-scale "next-token" prediction on human language, has been interpreted emergent forms of meta-learning (lcarning Icarn) within fized parametric system.
Almost all prominent evidence of emergent ICL so far (Srivastava et al. 2023 inter alia) comes from training human language (e.g., English). This raises fundamental question: is there something inherently special about human language that enables ICL to emerge? One can take two different stances on this:
Hi: Human language possesses distinctive distributional properties such as parallelism composition- ality Chen et al. 2024 Hahn & Goyal 2023) that uniquely nurture ICL.
L.Mss were used during the conception, implementation, and refinement of this paper. We assume full responsibility for all content.

---

Published in Transactions on Machine Learning Research (03/2026)
! Few-Shot Bitstring
CoddalitySpeccfii Encoding:
3 Inference Model and Compute Accuracy
4 In-Context Learning
Program Synthesis
Trends Comparison
(100+ programs)
Genomic Encoding
Genomic Encoding k-hot demonstratios
Genomic Models
Identity:
(4) Random 3 Nucloorides
Accuracy vs number of shots Tivo2-40B)
10100000 -> 10100000 SCF
AICG
5:4
11100011 -> 11100011 SRP
AAAT TAAA
G
11001010 -> 11001010
SEP
(b) Map strings "0" - 't", "1" "T"
ATAA AATA G
11000000 ->
0.35
TTTA ATTT G
Bitwise NOT:
"SEP" - " "->""" removed
5.30
13100D00 >> 01011:11 SDF
ATAT TATA
G
11100011 -> 00011:00 SRP
Example:
AATT ?
11001010 11001010
SEP
11000000 ->
1010 10" DSRP
Evo2
Grerdy doode
Majority 1:
TATATATAG
ceatuate
8:3
In-conted examples (stenst
178
13100000 -> 0C030000 SRF
11100011 -> 11111111 SEP
Linguistic Encoding (x) Random 3 Digiss: 0123456781 89
Linguistic Encoding
Linguistic Models
11001110 11111=11
A-shot Jemonstrations
Accuracy vs number of shots (Qwen3-14B)
11000000 ->
Majority 1 + NOT:
1113
3111
10100000
11111:11
(b) Map the bit strings
1311
1131
11100011 -> GC030000 SRP
"0 - "I" "I - "3",
3331
1333
6
0.50
11001110 -> 00000000 SEP
"SEP" 66" removed
1313
3131
6
6.5
11000000
Revers:
Example:
1133 2
0.20
10100000 -> 00000101 SEF
1010 -> 1010SEP
11100011 -> 11000111 SDF
Qwen
Gready & Kevaliate
11001110 -> 01110011 3RF
313131316
In-comted examples (shats)
11000000 ->
Figure 1: We design parallel symbolic reasoning tasks that allow direct comparison of ICL behavior across modalities (linguistic and genomic). ② Few-shot bitstring mograam-yynthest tasks (e.g., identity, NOT, majority, reverse) require models to infer mappings from examples. 2 Each task is rendered in two modality- specific encodings: genomic (bitstrings mapped to random nucleotides A/T/C/G) and linguistic (bitstrings mapped to random digits), preserving abstract structure but differing in surface form. 3 Both genomic (Evo2) and linguistic (Qwen3) models receive k-shot demonstrations and are greedily decoded to compute exact-match accuracy. 4 Both models show log-linear accuracy gains with more demonstrations.
H2: Alternatively, perhaps natural language is merely convenient substrate having ample data and being naturally interpretable and ICL could just as well arise in any sufficiently structured data domain, provided the model is large enough and trained to compress predictive patterns.
While many works have examined H1. the veracity of H2 remains far less understood. To this end, we turn to a radically different substrate: the genome. Genomic sequences can be viewed as a form of natural language that has evolved through nature. Like human language, they comprise complex symbol sequences that exhibit rich statistical regularities motifs, repeats, dependencies (Benegas et al. 2025) that could, in principle, support pattern induction within context.
Testing for the existence of ICL requires large-scale models. Historically, genomic models were small and specialized, optimized for narrow biological tasks such as motif discovery or structure prediction. The advent of deep biological models (Ji et al.] 2021) has expanded their scale, paralleling those of early LLMs. Most rocently, the Evo2 scries (Brixi et al. 2025) marks a major step forward: a largo-scale model trained solely on 'next-mucleotide' prediction. With training scale comparable to mid-sized LLMs (e.g., Qwen3-14B (Yang et al. 2025 ), these genomic models finally make it possible to systematically compare genomic and linguistic representations under large-scale autoregressive training. If ICL arises primarily from scale and predictive compression (H2), then large-scale genomic models such as Evo2 must already exhibit it under our noses. Like LLMs, they are trained for next-token prediction without any explicit meta-learning signal. Yet, despite extensive analysis of Evo2's biological capabilities (e.g., protein fitness prediction, variant effect estimation), its potential for emergent. few-shot or analogical reasoning within context remains unexplored.
The second challenge in evaluating H2 lies in developing a framework to detect and quantify ICI. in the genomics domain where "tasks" lack natural human interprotability. Morcover, the task design must permit systematic comparison between ICL behaviors in human language and genomic modalities. To investigate
2

---

Published in Transactions on Machine Learning Research (03/2026)
this, in $ we introduce a controlled experimental framework (Fig. for evaluating ICL across linguistic and genomic modalities. As illustrated in ① ② we design suite of symbolic bitstring program-synthest tasks and render them in two parallel encodings: one using genomic alphabets (nucleotides: A/T/C/G) and one using linguistic (characters). We then evaluate both model familics Evo2 for genomic sequences and Qwen3 for linguistic lext under matched few-shot, prompting conditions (3). This parallel encoding enables direct, apples-to-apples comparison of ICL scaling trends across genomic and linguistic models.
Our experiment's results ( reveal striking parallels: both linguistic and genomic models exhibit log-linear improvements in pattern induction and extrapolation the number of demonstrations grows, To our knowledge, this is the first evidence of organically emergent ICL in genomic sequences. These findings suggest. H2, that is, ICL is not an artifact that is specific to human language, but a broader conse- quence of compression and predictive modeling in pattern-rich sequence spaces. By demonstrating ICL in a non-linguistic biological domain, we extend the scope of emergent meta-learning beyond natural language, pointing toward a unified view of context-adaptive computation that spans different modalities of data.
We further perform analyses to compare the types of tasks at which genomic and linguistic models excel. Our results reveal distinct patterns: genomic models learn faster with more demonstrations, while linguistic models gain more from scale ($42;; ; genomic models perform best on direct copying or local transformations. whereas linguistic models excel on tasks requiring single-bit dependencies or global summary statistics ($4.) $4.4p. We note the caveat that these findings are specific to our experimental conditions (e.g., our choice of reasoning tasks) and may not be a holistic statement about ICL in genomic and linguistic models. We further validate these findings by demonstrating that Evo2 also exhibits robust few-shot learning on a real-world genomic promoter classification task (Appendix E.
Our Contributions. (a) We present controlled experimental setup for detecting and measuring ICL across linguistic and genomic modalities. Our framework defines a suite of symbolic bitstring reasoning tasks that can be rendered in both representations, enabling direct, applcs-to-apples comparison of ICL, behavior. (b) Using the Evo2 family of large-scale genomic models, we demonstrate for the first time that models trained solely on nucleotide sequences exhibit clear in-context learning, mirroring the scaling trends of Qwen3 language models. (c) Our findings challenge the idea that ICL depends on linguistic structure.
Instead, they point to ICL as modality-agnostic outcome of large-scale next-token prediction over pattern- rich data, reflecting general principles of compression and contextual inference. Together, these results exlend the scope of emergent meta-learning beyond language, toward a unified understanding of context-adas adaativv computation in artificial and biological systems alike.
2 Related Work and Broader Context
Why does ICL remain interesting? ICL remains a striking and conceptually rich phenomenon: it
emerges without task-specific training yet enables rapid, in-situ adaptation. It. comes closest to long-standing
ambitions in classical Al such as case-based (Aamodt & Plaza 1994] Lancaster & Kolodder
1987 Ross,
1984
Ellman
1989) and analogical reasoning [Hofstadter
2001
Gentner & Hoyos] 2017 Holyoak ct al.
2001
where agents solve new problems by reusing solutions from prior (analogous) examples.
Beyond its theoretical appeal, ICL has become practical workhorse in recent advances, e.g., synthetic
data generation using pretrained models (Wang et al.) 2023c Shao et al. 2023 Tunstall et al. 2023) or
reasoning (Wei et al. 2022 Nye et al. 2021 Yao et al.
2023 Yasunaga et al. 2023). In short, ICL is both
a window into emergent learning dynamics and a cornerstone for building adaptive, interpretable systems.
Emergent-ICI * Meta-ICL: A parallel line of research studies models trained with an explicit ICL objective learning to predict over collections of input-output pairs f()). For example, regression models that are trained explicitly to infer an underlying rule (c.g., lincar, polynomial,
sinusoidal) from
few-shot examples (Garg el al. 2022 Li et al. 2023c Raventós et al. 2023 Nejjar et al. 2024). This setup
mirrors meta-learning, where a model is explicitly "trained to learn" (Finn et al. 2017 Bertinetto et al.
2019 Zintgraf et al. 2019). We refer to such systems as Meta-IC'L (Panwar et al., 2023 Min et al. 2022 Wu
ct al. 2022 Kirsch et al. 2022 Zhang et al. 2023), in contrast to emergent ICL in large pretrained models, which arises organically from next-token prediction over natural data.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Meta-ICL results demonstrate that transformers can be optimized to perform meta-learning but fail to explain why emeryent TCL, appears in models never trained for it. Critically, Meta-ICT, generalizes poorly outside its training domain (Naim et al. 2024). For example, a Meta-ICL model trained on linear and cosine functions fails on their composition y (Yddoowky et al.,
2023 -nnd does not capture higher-order abilities like
in-context reasoning (Wei el al., 2022 Kojima el al. 2022 or adaplive self-refinemett (Madaan et al. 2023)
unless explicitly trained for them. In contrast, emergent ICL in LMs generalizes broadly and flexibly across diverse problem types (Srivastava et al., 2023). These behavioral and mechanistie differences support our working assumption that Meta-ICL and emergent ICL are fundamentally distinct phenomena (Shen et al, 2024 Mishra et al.) 2025). Throughout, our focus in this work is on the latter the organic emergence of in-context learning in large pretrained models.
ICL * Zero-shot learning; Existing literature in the domain of timeseries studies the zero-shot inference capabilities of next-token predictors, i.e., given a context of a given problem their models can predict the forthcoming values (tokens) (Ansari ct al. 2024 Das et al. 2024). This differs from the type of ICL we evaluate, which explicitly relies on few-shol learning of a transformation, not zero-shot adaptation. In fact, in Fig. 2 we report that Evo2's performance begins far below naive mode baseline (simply guessing the most common output) at one shot, indicating very poor zcro-shot capability. Evo2's ability only becomes meaningful only when many shots are provided. This makes the type of ICL we demonstrate qualitatively different from zero-shot ability, as it explicitly only appears in few-shot contexts.
Are there any prior results on emergent ICL genomic models? Most genomic models to date are encoders (Ji al., 2021 Dalla-torre et al.) 2024), which preclude autoregressive generation and the study of TCL in the standard sense. Among the few autoregressive genomic models, most remain small, e.g- GENA-LM (300M) Fishman et al., 2024) and scGPT (53M) (Cui et al., 2024h. The recently-released Evo2 series Brixi et al. 2025) includes models up to 40B parameters, comparable in scale to modern LLMs. This model family provides the first viable testbed for probing emeryent ICL, which we study here. A related effort, HycnaDNA Nguuyn ct al. 2023). trains on million-token genomic sequences and reports ICL--ikk behaviors; however, these involve mela-ICL mechanisms (e.g., soft prompting and instruction fine-tuning: see their Sec. 4.3), rather than organically emergent ICL. Similarly, Bio-xLSTM (Schmidinger al. 2024) demon- strates ICL, but only after explicitly training on deliberately constructed few-shot demonstrations instead of natural biological data another example of Meta-ICL. An additional avenue of research demonstrates that genomic models can perform form of TCT. by copying prior tokens in-context for prediction (Kantroo et al. 2025)- akin to how induction heads function {Olsson et al. 2022a). This, however, doesn't demonstrate any ability to infer latent functions or apply a novel rule in-context.
Are there instances of emergent ICL in other non-linguistic modalities? While transformers have been extended to many modalities beyond language, genuine cases of emeryent ICL arising purely from large-scale pretraining remain rare. In neuro-signal modeling, for instance, EEG-GPT (Kim et al., 2024] is trained with explicit in-context demonstrations, making it instance of Meta-ICL rather than emergent adaptation. Similarly, multimodal models such as Flamingo (Alayrac et al., 2022). BLIP/ /BLIP- 2 Li et al. 2022 2023a), and Enu (Sun et al. 2023 2024) exhibit in-context behavior only because they are explicitly trained to process demonstration patterns again, meta-learned rather than emergent. Another example of non-linguistic ICL is Chronos 2 a timeseries model trained to perform in-context learning through cross-batch learning, where the model's performance can be improved by learning from multiple time series provided in-context. However, Chronos 2 is trained using exclusively synthetic data designed to elicit cross batch learning and ICL, which firmly places it in the Meta-ICL regime (Ansari et al. 2025 To
our knowledge, the only claimed instance of emergent ICL outside language is in vision: Bai et al. (2024) show that transformers pretrained on natural visual sequences can infer analogical and compositional tasks without explicit supervision. The weakness of this result is that their ICL, prompts consist of long sequences
of contiguous frames (Bai et al. 20244, Fig 10), which effectively reduces the task to next-frame prediction, rather than true ICL that requires inferring compositions of multiple demonstrations.
4

---

Published in Transactions on Machine Learning Research (03/2026)
3
Experimental Framework for Cross-domain In-context Learning
This section presents our framework for evaluating ICL across linguistic and genomic models. We outline the experimental desiderata. ($3.1. setup ($32),, model selection ($3.3). and evaluation protocol ($3.4. We
additionally release code for replicating these experiments on Github.
3.1 Experimental Desiderata
Cross-domain comparability. Our experimental framework requires that each task be performable by both language and genomic models. Thus, every task must be representable in both linguistic and nucleotide alphabets. This constraint excludes existing benchmarks that rely on domain- specific semantics such as language reasoning datasets (e.g., BIG-Bench, StrategyQA, GSM8K (Srivastava) let al. 2023; Geva et al.) [021 Cobbe al.) (2021)) or biological tasks (e.g. variant effect prediction, exon identification (Brixi et al. 2025 While could, in principle, translate domain-specific tasks into alternate alphabet (e.g., mapping language tokens to base sequences via quaternary encoding), doing so inherently biases the evaluation: the source domain retains advantages, while the target domain must operate on representations that are unnatural to it. As a result, such cross-domain encodings confound the comparison, reflecting representational translation artifacts rather than genuine differences in ICL behavior.
Desideratum 1:
Desideratum Limited vocabulary. Since genomic models operate on only four nuclcotides (A, T, C, G), tasks must be expressible within an equally compact alphabet. This rules out the existing symbolic reasoning and analogy benchmarks (Hodel & West 2023 Lewis & Mitchell, 2025 Webb et al., 2025), which rely on richer vocabularies or background knowledge (e.g., shapes, colors, linguistic tokens with explicit semantic/lexical roles). While some of these tasks are more symbolic in nature, they still rely on pretrained knowledge or linguistic instructions about numerical or lexical ordering (i.e. 'e' coming after 'a b c d') information that couldn't effectively be conveyed in-context to a genomic model (Lewis & Mitchell, 2025) Stevenson et al.) 2025). Such tasks cannot be faithfully represented in a four-token regime without introduc- ing artifacts or structural loss. Accordingly, our evaluation focuses on tasks that retain abstract reasoning structure while remaining compatible with the low-vocabulary symbolic space shared across linguistic and genomic models.
3.2 Experimental Setup
We formally define ICL follows. Let S be the input domain and O the output domain, and let task be latent deterministic function f: S - O. A pretrained autoregressive model M performs ICL by conditioning on ordered n-shot demonstration set E = f(1)) then receiving a held-out query x E S (ZI....... J. Conditioned on (E, x), the model produces a prediction 0 = x),,, which we compare to the ground-truth f(x). Then the single-trial success is 1 [ = f())].
Notation:
(In, f(xx))), where 2; E S, and
in-context program induction over abstract symbols. Given our dosiderata (3.1). we design symbolic induction tasks that require a small lexicon. Each task requires a model to infer hidden transformation rule from few input-oulput examples and apply it to a new query. We refer to this process program synthesis in-conteat. This setup isolates in-context learning ability by removing the inffuence of pretrained world knowledge. If the function family f and symbol space S are chosen carefully, the evaluation is far from the model's pretraining distribution. Similar formulations have been used in prior work on compositional reasoning (Brown et al. 2020b), analogical reasoning via Raven's 's Progressive Matrices (Raven et al. 1962; Webb et al. 2023), and more recently in ARC-AGI (Chollet) 2019).
Task Formulation:
Bitstring domain and function space. To maintain compatibility across genomic and linguistic models, we represent all symbols as bitstrings of length k, i..., S {0, 1)* Genomic models operate over four nucleotides (A, C, G, T), allowing two tokens (e.g., A/C) to encode 0/1 and reserving the others (G/T) delimiters or separators. Bitstrings are also naturally supported by linguistic models since most tokenizers represent digits as single tokens, ensuring parity in symbol granularity. This design minimizes tokenization confounds while providing uniform symbolic substrate for comparing both domains. See Fig. for examples. Despite their simplicity, bitstrings support a wide range of composable operations from basic (identity,
5

---

Published in Transactions on Machine Learning Research (03/2026)
constant) to logical (AND, OR), positional (shift, rotation, reversal), and aggregate (majority, parity). This expressivity makes them convenient and extensible testbed for cross-domain in-context learning evaluation.
We set the symbol space to S = {0,11 corresponding to all 8-bit binary strings. This choice balances expressivity (256 unique bitstrings) with manageable prompt length. To define the task space, we construct a set of 100 transformation functions F E (S / + S}, where each function maps one bitstring to another according deterministic rule. Each f'e P' either a single primitive operation or composition of two primitives, allowing for controlled yet diverse range of symbolic transformations.
Primitive operations. We build F from a library of 30 fundamental primitives (listed in Appendix A.1 spanning six functional categories: (1) Bitwise transformations (e.g., flipping individual bits), (2) Structural rearnanementss (rotations, shifts, reversals, swaps), (3) Posilional masking and selection (preserving or zeroing specific bit positions), (4) Pattern detection (detecting alternating or palindromic patterns), (5) Aggregation functions (majority /esiorrtt/miooiity,,... /minority, parity), and (6) Trivial mappings (identity, constant outputs). This laxonomy ensures coverage of both low-level logical operations and higher-order compositional structure.
Automatic generation and validation. To construct the final function set, we GPT---Codex to automatically generate 100 unique transformations by composing primitives into valid Python programs. Each candidate function is verified programmatically for logical distinctness: no two programs f.g E F produce identical outputs for all inputs (Āʃ.g g such that f(x) g(x) for all x E S). All functions undergo manual inspection to confirm correctness, diversity, and adherence to the intended categories. The full list of transformations and corresponding source code is provided in Appendix A.2
Prompt encoding. Prompts are encoded using unified symbolic scheme enable evaluation across linguistic and genomic models. For linguistic models, bits (0, are mapped to two random digits (0 9); for genomic models, to random nucleotides (A, T, C, G). The separator token (-+ is omitted, and in-context examples are separated by a randomly chosen unused token distinct from those representing 0 and 1. (See Fig. for examples.) All mappings are randomized per trial to avoid memorization or positional bias.
3.3 Model Families and Selection Rationale
For fair cross-domain comparison, we use two representative model families: Qwen3 for human language and Evo2 for genomics. The rationale for this selection is as follows:
(a) Parameter scaling: Both families span multiple orders of magnitude in parameter count, enabling systematic scaling analysis of ICL ability. The Qwen3 series ranges from 0.6B to 14B parameters, while Evo2 includes 1B, 7B, and 40B models (Yang et al., 2025 Brixi et al. 2025). This parallel scaling structure facilitates consistent measurement of how ICL performance evolves with model size across linguistic and genomic modalities.
(b) Compule matching: The largest models in each family are trained with comparable total compute, offering an opportunity for an approximately compute-matched cross-domain comparison. Using the standard 6N estimate (Kaplan et al. 2020), Qwen3-14B- Base is trained with about 3.2 x 20 FLOPs, while Evo2-40B is trained with 2.25 x 1021 FLOPs (Yang et al. 2025 Brixi el al. 2025 making Evo2 uniquely well-suited for comparison with Qwen3 at scale.
(c) Availability of base models: The Qwen3 family releases base (pre-instruction-tuaee) models at all scales up to 14B parameters, enabling direct evaluation of the intrinsic inductive reasoning ability of pure next-token predictors, without instruction-luning artifacts. The Evo2 base models have no additional instruction tuning applied- the only nuance is that Evo2 1B is trained at context length of 8192 nucleotides, whereas the 7B/40B are extended a context length of one million.
(d) Tokenizer: Qwen3 uses a standard BPF tokenizer with a vocabulary size of 151,669 Evo2 uses a byte-level tokenizer, so individual nucleotides are mapped to individual tokens. Notably, Qwen's lokenizer maps single digits to single tokens, allowing for parity in our experiments (Yang el. al. 2025: Brixi et al. 2025

---

Published in Transactions on Machine Learning Research (03/2026)
(c) Contert length: Qwen3's 0.6B and 1.7B models have a context length of 32K. The remaining dense models have a context length of 128K. Evo2's 1B model has context length of 8K, and the remaining models have context length of 1M. As none of our experiments approach these limits, we can use all models in both familics without fear of context length as a confound Yang et al. 2025 Brixi
et al. 2025
(f) Training corpora: All Qwen3 models are trained on 36 trillion text tokens covering a wide variety of lopics and languages. Evol 1B trained on trillion lokens, Evo2 7B is trained on 2.4 trillion tokens, and Evo2 40B is trained on 9.3 trillion tokens. These extensive training corpora ensure that any potential ICL dynamics have thoroughly emerged (Yang et al. 2025 Brixi et al. 2025
(g) Architecture: Qwen3 is based uses the StripedHyena2 architecture which intersperses convolutional layers with attention-based ones. While ideally both model families would use the same architecture, there were no vanilla transformers at Evo2's compute scale (Yang et al. 2025 Brixi et al. 2025).
conventional Llama-like transformer architecture, while Evo2
(h) Licensing: All models are released under Apache 2.0 Yang et al. 2025 ensuring reproducibility.
Arc Institute 2025a
3.4 Evaluation Protocol
Evaluation metric. Building on the setup in $3.2 we now describe how in-context performance is eval- uated. For each transformation I E F, draw an n-shot demonstration set E. S and held-out query E S E. The model M receives few-shot prompt of imput output pairs followed by the query and (E) A trial is marked correct if g f(z). giving the single-trial indicator A(, E,x, M) = [ = [())]-
produces a prediction
Since exact enumeration of all the programs is intractable, we use a Monte Carlo estimate (a full formulation of what we approximate can be found in the appendix(B] Let E,, denote the space of all sets of input bitstrings with cardinality n: E, =EE c S: [E] =n. Sample i.id. context sets Ext) Unif(E,) for L.... ... m; for each t, sample x(t) Unif(S E(D).. Then our empirical estimate of accuracy, for a given transformation f is (M, 1 1 E(t), x(),, M) and the overall estimated accuracy for model M is:
P(M.n)
1 F
Σ A,(M,n). JEF
(1)
Model suites and sampling configuration. We fix the number of Monte Carlo trials to m = The set of evaluated genomic models is MM { 1b_ base, evo2_ eee2____bvoo____..... evo2_40b) and the set of linguistic mod- els is Mr [Qwen3-0. 1.--0...B-BAR 6B-Base, Qwen3-1. 7B- TBsBase, Qwen3-4B-Bat evaluate across shot counts A = {1, 4, 16, 32, 64, 128]. For each model M and shot count n, compute:
Twen3-SB-BBsee Qwen3- 14B 144-Base). We
Pr. (P(M,)) n) M NM, Pa (P(M,)) M e Mc. E
We estimate standard errors via a two-stage nouparametric cluster bootstrap, resampling functions (clusters) and within each selected function, resampling its evaluation samples with 5000 replicates for each (M, n).
Mode bascline. We define a mode bascline that always predicts the most frequent output observed in the context. For a given function f. context set E C S, and query input E S \ E, the mode prediction is Imodee (ʃ, F, .) = arg maxyes { E F. f(e) = y } where ties in the arg max are broken randomly. This simply corresponds to guessing the most common output in the sew-shot examples. The overall mode baseline with n shots and across the set of all functions F is:
m
Pmode (n)
ΣΣ 1=
1 [uroode E(0)
E,,) 20) f(x(11))
(2)
where (()) and (1) are sampled identically to the model evaluation in Eq 1 This bascline corresponds making an educated guess based only on the overall distribution of function outputs that simply learns the majority statistics of the prompt without attempting to infer the underlying transformation.
7

---

Published in Transactions on Machine Learning Research (03/2026)
(a) Evo2 models (genomic encoding)
(b) Qwen3 models (linguistic encoding)
0.50
0.50
Evo2
Qwen3 0.68
0.45
Evo2 78
0.45
Owen3 1.78
Evo2 408
Qwen 48
0.40
Accuracy
0.35
Accuracy
0.35
3wen3 148
0.30
0.30
0.25
0.25
0.20
0.20
0.15
0.15
2
8
16
32
64
128
1
16
32
128
Number of in-context examples (shots)
Number in-context examples (shots)
(c) Evo2 Owen3; comparable model sizes
Aggregate trends all models
0.50
0.50
Owen3 1.78
[avg)
0.45
Owen3 BB
0.45
Evo2 (avg)
Evo2 18
Mode Baselne
0.40
Accuracy
0.35
Accuracy
0.35
0.30
0.30
0.20
0.20
0.15
0.15
0.10
2
8
16
32
64
128
1
32 Number in-context examples (shots)
16
128
Number of in-context examples (shots)
Figure 2: Fow-shot performance of Qwen3 and Evo2 models. (a) Evo2 model performance with respect to log(shots). All models monotonically improve the 7B and 40B have roughly equivalent performance, and the 1B trails behind them. (b) Qwen3 model performance with respect to log(shots). All models improve, but not always monotonically. Smaller models struggle in 4-16 shot range. (c) At comparable sizes, Evo2 outperforms Qwen3. (d) Averaged performance across both model families shows consistent improvement with respect. log(shots). All models exceed the mode baseline shown gray color. The exact accuracies and error bars (bootstrap-based standard errors) are included in Appendix
4 Empirical Results
This section reports empirical findings on few-shot bitstring generalization. $4.1 presents the main accuracy trends with respect to model size and shot count. $4. examines sensitivity to task complexity using a BitLoad measure, and $4.4 costrasts the models' qualitative competencies across individual transformations.
4.1 Main Results
Across both model families, accuracy generally rises linearly with respect to log(shots). A linear regression linking performance to log(shots) yields highly significant slopes for all models (all < 10-3 via one-sided t-test on slope). As Fig. 2 shows, Evo2 models show cleanly monotonic gains, with a pronounced step from 1B to 7B, and near-indistinguishabb curves for 7B and 40B; by 128 shots, both 7B and 40B surpass 40% accuracy (Fig. 2 Qwen3 also trends upward overall but with non-monotonic patches especially for smaller models in the 4-16 shot band-befor resuming clear improvements from 32 to 128 shots; at 128 shots, the 14B model approaches 35% (Fig. 2).. A full table of all accuracies is in Appendix C
Evo2 outperforms Qwen3 at comparable sizes. Evo2 1B beats Qwen3 1.7B and Evo2 7B beats Qwen3 8B at higher shot counts (Fig. 2 Averaging within families reinforces the same picture: comparable performance in the 1-4 shot range, Evo2 pulling ahead around 8 8-16 shots, and the gap persisting higher shot counts (Fig.) 2)).

---

Published in Transactions on Machine Learning Research (03/2026)
The mode baseline's perfor- mance doesn't improve with more shots, and lags Qwen3 and Evo2 at high shot counts (Fig- I1). This indicates that the in-context learning performed by Qwen3 and Evo2 is qualitatively different than simply sampling from the distribution of possible outputs in-context. Qwen3 and Evo2's per- formance requires that they learn, in-context, how to condition on the imput. Furthermore, for Qwen3, ad- vantages over the mode baseline be- consistently significant (p < 0.05 via one-sided z-test boot- strapped standard errors) at n=128 for all sizes greater than 0.6B. For Evo2, statistically significant ad- vantages emerge slightly earlier: for 1B at n=64, the 7B at = 32, and the 40B at = 16. Regardless, all models surpass the naive baseline.
Qwen3
Evo2
65
Accuracy
0.3-
0.0
Qwana 0.68
Owen 1.78
Qwen3 46
Owen3 88
Qwen3 148
Evo2 18
Evo2 78
Evo2 408
Figure 3: Performance at n 128 shots. All model accuracies increase monotonically with respect to parameter count.
4.2 Analysis of ICL Capability Across Model Families and Sizes
As shown in Fig. 3 both Qwen3 and Evo2 exhibit clear performance gains with increasing scale. Qwen3 displays a strong positive correlation between accuracy and parameter count (p < 0.05 for all shot counts 216, one-sided I-lest on slope), while Fvo2 shows a distinct jump from 1B to 7B but little improvement beyond that. This suggests that in-context program induction becomes more robust with scale, though saturation may occur once models reach sufficient capacity (A detailed statistical analysis in Appendix D
4.3 ICL Sensitivity to Task Complexity: BitLoad Analysis
To understand which transformations Qwen3 and Evo2 infer most effectively, we analyze their performance across varying task complexities. We focus on the largest models in each family (Qwen3- 14B and Evo2-40B) under the (n 128) shot regime, providing both models ample opportunity to display their ICL abilities.
Defining BitLoad. We introduce BilLoad, mea- sure of function's intrinsic complexity. Infor- mally, BitLoad quantifics how many input bits in- fluence the output.. Formally, it defined as:
10
Owend
Fvvo2 4DB
Average accuracy
BitLoad(f) = - Bx. fz(x) # fy(2ni), (3)
0.6
0.4
where Hb denotes x with bit flipped, and fz(#) returns the j-th output bit. Intuitively, it counts the number of bit positions whose perturbation changes the oulput. So, the larger the BitLoad of function, the harder it is since it requires the model to attend to more bits. This metric sim- ilar to the existing statistical measure of 'relevant features", just defined specifically for our bitstring manipulation tasks. (Nevo & El Yaniv 2002).
0.0
Bit load
Figure 4: Accuracy vs. BitLoad averaged across all tasks (BitLoad; 3 Qwen declincs sharply, while Evo degrades more gradually. Details in $4.3
Fig. 4 shows the mean accuracy of Qwen vs Evo with respect to the BitLoad of all of our tasks. A full table of the BitLood of every tested task is attached the appendix A.3] We see that both models achieve near- perfect accuracy on constant (0 BitLoad) tasks and remain similar BitLoad 1. However, by BitLoad 2,
9

---

Published in Transactions on Machine Learning Research (03/2026)
Evo begins to outperform Qwen, and beyond that point Qwen's accuracy drops sharply falling below 20% by BitLoad 4 while Evo's performance declines more gradually, remaining above 40% before converging near 20% at BitLoad 8. This asymmetry suggests that Evo maintains partial generalization as dependency depth increases (BitLoad value), whereas Qwen's ICL collapses more abruptly.
While BitLoad strongly correlates with task accuracy, Fig- 4 also shows notable deviations within the error bars. Thus, while task complexity is a strong predictor of ICL accuracy, other factors (e.g., transformation depth, pretraining exposure) likely play meaningful roles, which motivate qualitative study in $4.4
4.4 Qualitative Analysis of Functional and Behavioral Differences
To complement the quantitative BitLoad analysis ($4.3) we conduct a per-task comparison to identify qualitative differences between Qwen3 and Fvo2. Specifically, we analyze which tasks each model performs best on and how their inductive profiles diverge. We focus on the = 128 shot regime for the largest models 14B and Evo2-40B where both have maximal opportunity to exhibit ICL. We rank all tasks by model accuracy and examine the top ten for each model. Tasks that appear in one model's top ten but not the other are considered "exclusive."
Exclusive competencies. Qwen's exclusive lasks involve right-shift operations: "spread paareed__ast____ last_bit" + "shift teaffttrigt__aee terrro and "edge_ mask" + "shift_ seeit__ight___t patt___err.. For instance, "shift "shiff_right, the bitstring on the left with zero and truncates the last bit (e.g., 01010000 00101000). Qwen achieves 100% accuracy "spread_ sapread_last_baee lal_llastbttt + 'shift_right_ee "errgg__errr,,,,.__________ whereas Evo2 achieves only 50%. A sim- ilar but smaller gap appears for "edge_masii" "aas"_ + "shifftrightt___ zero" (87.5% vs. 62.5%).
pads
In contrast, Evo2's exclusive strengths involve multi-bit transformations. A clear example is "flip_ bits" + Sright_half's which applies bitwise NOT followed by masking the first half of the imput (e.g., 01011100 + 00000011). Evo2 achieves 87.5% accuracy, while Qwen only 25%. This task has a BotLoad of four, consistent with Evo's superior performance on mediun-complexite (2 4 bit) transformations observed in Fig.
Shared strengths. Despite these differences, 7 of the top 10 tasks are shared between Qwen and Evo, yielding an intersection-over-unio of 0.54. Both models excel at simple transformations such as constant- outputs or single-bit dependencies, e.g., pegreed_last___e
which copies the final bit to all positions.
Differential skill profiles. To sharpen contrasts, we identify the ten tasks most favoring each model. Qwen's 's advantages are concentrated in simple shifts and aggregation tasks. It outperforms Evo by 37.5% on the "minority" operation (output all is if zeros ones), and by a similar margin on two parity- basod tasks requiring counting the number of 1s. These trends suggest that Qwen may be better reasoning over global properties of bitstrings. Its superiority of frame shift mutations in DNA due to how catastrophic they are a single muclcotide offsct can decimate an entire protein. This is empirically supported by the fact that single-nucleotide deletions/shin /shifls increase perplexity far more than other common mutations when presented to Evo2 (rrxx et al. 2025
simple shifts may also be explained by the extreme rarity
Evo2, by contrast, dominates tasks requiring full-bitstring manipulation. achieves 62.5% on bitwise NOT (vs. 0% for Qwen), 62.5% on identity (vs. 12.5%), and large margins on compositions such as "flip_biti rrighthhalff (87.5% vs. 25%) and "rotl1" "flip_ bittsits (37.5% vs. 0%). This exposes perhaps the most important difference between Qwen and Evo's ICL in this specific context: Evo can learn simple full- bitstring operations in-context, whereas Qwen cannot. Notably, Qwen3's base models are trivially capable of learning the identity in a more familiar few-shot context when examples are presented with arrows and newlines separating them, instoad of our intentionally unfamiliar encoding. Thus these results should be taken as an existence proof of ICL in Evo2, not a definitiee statement of Evo2 having more ICL ability than Qwen. We leave comparison of these models across broader tasks to future work.
5 Discussion
What are the implications of our findings on prior efforts explain the emergence of ICL? First, let us organize the existing frameworks for pinpointing the conditions under which ICL emerges:
10

---

Published in Transactions on Machine Learning Research (03/2026)
(Ei) ICL's emergence is due to data distributional properties: The distributional properties of data, such as "parallel structures" in human language pretraining data (Chen et al. 2024), its compositional structure (Hahn & Goyal 2023), "burstiness" " (Chan et al. 2022) and other such properties (Wibisono Reddy 2023) may be of importance (and perhaps necessary) for the emergence of ICL.
& Wang 2024
(B2)
ICL's emergence is due to a compression mechanism: The large-scale compression mechanism
during massive pretraining might drive ICL (Elmoznino et al. 2024a bb Hahn & Goyal 2023).
(Ea) ICL's emergence may require specific architectural properties: While Transformers might be better suited for ICL than LSTMs (Xie et al., 2021), evidence is mixed [eee et al., 2023), and non-
Transformer models have also demonstrated ICL. capabilities (Grazzi et al. 2024 Park et al. 2024).
Our findings refine existing hypotheses about ICL's origins. The emergence of ICL in genomicc models
challenges accounts that rely solely on language-specific distributional structures E1 The presence of ICL
across both genomic and linguistic models supports the compression-based explanation [E2) suggesting that largo-scale sequence compression and its induced inductive biases drive ICL across modalitics. With respect to architecture [Fa] Evo2 an autoregressive hybrid combining convolutional and attention layers rather than pure Transformer exhibits similar scaling behavior, indicating that ICL does not depend on the pure Transformer form. Instead, architecture provides an expressive substrate that enables pattern induction once exposed to sufficienlly large and structured data. Overall, these results position ICL as a modality-agnostic outcome of large-scale next-token prediction, rather than a phenomenon tied to linguistic statistics or a specific architecture.
What are the implications of our findings on the frameworks to explain how ICL operates?
We next consider the major perspoctives that seek to explain how ICL operates. One view holds that ICL functions as mix of task learning and task retricval, with demonstrations serving either to recall pretrained
capabilities or to enable learning on the fly (Pan et al.) 2023 Lin & Lee 2024 Wang et al. et al. 2025y. Our symbolic reasoning tasks, instantiated in both linguistic and genomic domains, provide direct evidence for this task learning mode, aligning with this hypothesis and prior work (Pan ct al. 2023 Fang et al.] 2025 Because these lasks do not depend on pretrained semantic priors, they do not invoke task retrieval, offering limited insight into the Bayesian view that interprets ICL as implicit inference over latent concepts (Xie et al. 2021 Panwar et al.
2024 Fang
2023 Wang ct al., 2023b Jiang et al.]
2024). Mcanwhile,
our results remain agnostic toward the optimization based-bbeeedd hypothesis, which posits that ICL implements an implicit gradient-descent-iii process (Akyürek et al.
2023 Li et al. 2023b), as well as the induction-based account, which aftributes ICL to specialized "circuits" for performing
2022
Ahn et al.
2023 Mahankali et al.
inductive generalization (Elhage et al.] 2021 Olsson et al.) 2022b Wang ct al.] 2023a Bansal et al.) 2023
Ren el al. 2024). Together, our results most. strongly support the presence genuine lask learning within ICL.
Does Evo2 have an innate advantage these tasks? Possibly, for multiple reasons. First, though Evo2's is trained on less tokens total, all of Evo2's training tokens are long sequences of repeated nucleotides, and very little of Qwen3's training tokens are long sequences of repeated digits. Second, Evo2's Striped- Hyena2 architecture was found to to significantly outperform & vanilla transformer on long DNA sequences (Brixi et al. 2025). This could give Evo2 an innate advantage on long contexts containing the same few symbols vs Qwen3. Our result results do not imply Evo2's 1CL ability is superior to Qwen3's, and we concede that our few-shot prompting sctup may be biased toward Evo2. Attempts to make the task more legible to Qwen3, however, ran into the confound that. Qwen3 has pretraining exposure to bitstring manipulation. Any prompting setup that included Os and is immediately resulted in extreme increase in Qwen3's perfor- mance. Future work is needed to establish a method that controls for Evo2's structural advantages without granting Qwen3 an unfair edge via its pretraining knowledge.
Why not test the models on semantic tasks? Semantic tasks such as identifying the capitals associ- ated with countries, classifying malformed proteins, etc. require a fair amount of pretraining exposure to the concepts involved in the task as well as measuring ICL. While it's undoubtedly ICL when a model infers semantic transformation (for instance, country-vcapitall word- cord-sopposite), the pretraining knowledge nec- essary to manifest this ICL precludes its use in extreme cross-modality comparisons. Focusing far simpler bitstring transformations that can be learned entirely in-context allows for an apples-to-apple comparison between Evo2 and Qwen3. To ensure that the ICL observed is not an artifact of this simplified domain, we
11

---

Published in Transactions on Machine Learning Research (03/2026)
additionally demonstrate in Appendix E that Evo2 exhibits robust, scaling ICL native genomic task (promoter classification), though we exclude this from the main comparison to maintain parity with the linguistic models.
Beyond the H1-Hz dichotomy. While we presented H1 and H2 (in $ as contrasting hypotheses, they are not necessarily mutually exclusive. It. entirely plausible that both hold simultancously that certain distributional properties inherent in natural data, shared across human language and other natural domains such genomics, contribute to the emergence of ICL. In this view, linguistic compositionality may represent one instance of a broader statistical substrate that fosters ICL. We framed the two hypotheses dichotomy primarily to highlight the contrast between language-specific and modality-general explanations, rather than to suggest that only one can be true.
6 Conclusion
We introduce a suite of bitstring reasoning tasks that can be encoded in both natural language and genomic sequences, showing that genomic models like their linguistic counterparts exhibit clear in-context learning. Across all Evo2 model sizes, we observe robust log-linear gains accuracy with increasing demonstratinns, paralleling the scaling trends of Qwen3 language models. These findings challenge the nolion that ICL is unique to human language, suggesting it emerges whenever an expressive model is trained autoregressively structured, pattern-rich data.
This proof-offexistence for ICL lays the groundwork for many future research directions. One could broaden the test suite to incorporate tasks beyond bitstrings up to four symbols are natively available in muclootides, and far more can be used one encodes at the codon level.
Potential future work:
Another promising direction mechanistic interpretability: analyzing Evo2's internal activations to identify which circuits enable ICL and how the model aggregates context to predict the next nucleotide, Comparing these mechanisms to known induction circuits (Elhage et al. 2021 in LLMs could reveal whether analogous structures exist across architectures question that connects biological and linguistic ICL dynamics.
Finally, this work motivates searching for ICL in other non-linguistic modalities time series (Das et al. 2024), system logs (Akhauri et al. 2025), physics simulations (Holzschuh et al. 2025), chess games [Ross et al. 2024), and climate projections (Duncan et al.) 2025). Each offers structured, patterned substrate that could support its own form of contextual reasoning. These diverse modalities, each with their unique structure and constraints, suggest rich world of non-linguistie ICL capability waiting to be explored, and this work represents maiden voyage into these extremely interesting waters.
Broader Impact Statement
Understanding ICL is pivotal for both the scientific study and practical control of LLMs. ICL ties to major efforts to interpret model behavior, characterize how reasoning and abstraction evolve with scale, and design mechanisms for controllability and steering. Tt also powers pragmatic applications such as synthetic data generation. By demonstrating that ICL arises even in non-linguistic scttings, this work broadens the empirical foundation for studying ICL general computational phenomenon rather than quirk of human-language training. A clearer mechanistic understanding of how ICL emerges and operates across modalities can inform how we build, guide, and evaluate large models - improving their reliability, controllability, and usefulness while reducing risks from misalignment or overgeneralization.
Acknowledgment
This work is supported by ONR grant 42000142441-208)), We acknowledge the use of computational re- sources on the DSAI cluster. We sincerely thank the JHU CLSP and DSAI communities for their helpful comments and feedback.
12

---

Published in Transactions on Machine Learning Research (03/2026)
References
Agnar Aamodt and Enric Plaza. Case-based reasoning: Foundational issues, methodological variations, and system approaches. Al commmunications, 7(1):39 59, 1994. URL http:://ww..iiaa..s....e/////
papers/ AICom.ppp......
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2023. URL htttp://paper. www..iicpp.....p.eee wwwp.ccpppperrflleeeeeeeeee taapper__tlee/pappr//////asss (ocddfooe44464444////77772222620
html
Yash Akhauri, Bryan Lewandowski, Cheng-Hsi Lin, Adrian N. Reyes, Grant C. Forbes, Arissa Wongpanich, Bangding Yang, Mohamed S. Abdelfattah, Sagi Percl, and Xingyou Song. Performance prediction for large systems via lext-lo-text regression, 2025. URL htttp:://arxt httppp///aaxvvor//a////////sss/s 217.56.0...1188
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In International Conference on Learning Repre- sentations (ICLR), 2022. URL htttps://axiv..nt /t////bb2222111 15661
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooeei Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL http://papers. wwtpeerss.ip....m .ips..c// paper
html
Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Olek- sandr Shehur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschieg- ner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneidee, and Yuyang Wang. Chronos: Learning the language of time series. Transactions on Machine Learning Research, November 2024. ISSN 2835-8856. doi: 10.48550/ 00.48350/arXiv.222 http//rrxi......0...
URL
https: / net/fforum/idrgerncvvggg Published 11 Nov 2024.
Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, Syama Sun- dar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, Mononito Goswami, Shubham Kapoor, Danielle C. Maddix, Pablo Guerron, Tony Hu, Jumming Yin, Nick Erickson, Prateek Mutalik Desai, Hao Wang, Huzefa Rangwala, George Karypis, Yuyang Wang, and Michael
Chronos- 2: From univariate to universal forecasting. arXiv preprint ar Xix:22000.0 15821, October 2025. doi: 10.48550/ 2ww...w.....112111111111
URL Utt.:ttts///aaaatoor//////555/50
Are Institute. Evo lb base. http://ingguggeeeec//rrrrrstuuuu base 2025a. Model card on Hugging Face.
Are Institute. Evo 2 40b. http:///wngiiigfeeeeeo/rrrrrssrt 2025b. Model card on Hugging Face.
Are Institute. Evo 2 7b. attes://wgghighhe...oo/ccssssssu 2025c. Model card on Hugging Face.
2025c.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22861 22861-22872, 2024. URL attps://arxivv wtt.:/orr//ppp.//1....0/.
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-base case study at 66 billion scale. In Annual Meeting of the Association for Computational Linguástics (ACL), 2023. URL https:
Saacllnntollog.oor///000 acl- mocl-llngg.6 6607
13

---

Published in Transactions on Machine Learning Research (03/2026)
Gonzalo Benegas, Chengzhong Ye, Carlos Albors, Jianan Canal Li, and Yun S Song. Genomic language models: opportunities and challenges. Trends in Geneties, 2025.
Luca Bertinctto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Mcta-learning with diffcrentiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2019. URL https:
//openreview wwt.://neefooooeoooeeeeeeeeeoooe http://oouumid--Hzznnntte
Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A. Gonzalez, Samuel H. King, David B. Li, Aditi T. Merchant, Mohsen Naghipourfar, Eric Nguyen, Chiara Ricei-Tam, David W. Romero, Gwanggyu Sun, Ali Taghibakshi, Anton Vorontsov, Brandon Yang, Myra Deng, Liv Gorton, Nam Nguyen, Nicholas K. Wang, Etowah Adams, Stephen A. Baccus, Steven Dillmann, Stefano Ermon, Daniel Guo, Rajesh Ilango, Ken Janik, Amy X. Lu, Reshma Mehta, Mohammad R.K. Mofrad, Madelena Y. Ng, Jaspreet Panmu, Christopher Ré, Jonathan C. Schmok, John St. John, Jeremy Sullivan, Kevin Zhu, Greg Zynda, Daniel Balsam, Patrick Collison, Authony B. Costa, Tina Hernandez- Boussard, Eric Ho, Ming- Yu Liu, Thomas McGrath, Kimberly Powell, Dave P. Burke, Hani Goodarzi, Patrick D. Hsu, and Brian L. Hie. Genome modeling and design across all domains of life with evo bioRriv, 2025. doi: 10.1101 10110012225.000 /tt//2222...2222 00225.0..88.63911. URL https: httpp://wwww.ioo/ wwtpivvv.gg/oooiaoe wtgg//ooneett/eall/220000 02/21/2025.02 01..:5.........99999.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020a. URL /tthhttps:// https://arxiv.cr /bbs//
2005. 14165]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurclio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), ) Advances in Neural Information Processing Sys- tems, volume 33, pp- 1907, 2020b. URL httpss////cceeengs.... www.ggg...rriis..coc pttp:cccaappar/2///aaaaaaa Abstract html
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https: :
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He Hc. Parallel structures in pre-training data yield in-context learning. ArXiv preprint, abs/ abs/2240.122530, 2024. URL https: abg/222/2402.
12530
François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworck, Jacob Hilton, Reiichro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprinl arXi::2100.14 14..166,,,, 2021. URL http:://t////aaa org/
/ 14168
Haotian Cui, Chloc Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scGPT: toward building foundation model for single-cell multi-omics using generative AT. Nature methods, 21 1111470-1480) 2024.
Hugo Dalla-torre, Liam Gonzalez, Javier Mendoza-Reville, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P. de Almeida, Hassan Sirelkha- tim, Guillaume Richard, Marcin Skwark, Karim Beguir, Maric Lopez, and Thomas Pierrot. Nuclcotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22: 287 2297, 2024. URI. https: / //ppi. aamaattteccolaarrrorg//ppspp//aa
14

---

Published in Transactions on Machine Learning Research (03/2026)
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time- series forecasting. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 10148-10167. PMLR, 21 21-27 Jul 2024. URL https::/proceedingge mlr www.sppeesssseee h23/////22//dcccccc// das24c. htmm
James P. C. Duncan, Elynn Wu, Surya Dheeshjith, Adam Subel, Troy Arcomano, Spencer K. Clark, Brian Henn, Anna Kwa, Jeremy McGibbon, W. Andre Perkins, William Gregory, Carlos Fernandez- Julius Busccke, Oliver Watt-Moyer, William J. Hurlin, Alistair Adcroft, Laure Zanna, and Christopher Bretherton. Samudrace: Fast and accurate coupled climate modeling with 3d ocean and atmosphere emulators, 2025. URL hattps://arxivv wtt.aarwwv...gggbaa./aaaaaaa 250// 2509. 12490.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for pub/
transformer circuits. Transformer Circuits Thread, 2021. URL https: 2021/Eramework//nee www.ooor///ddee...tltl..
Thomas Ellman. Explanation- based learning: A survey of programs and perspectives. ACM Computing Surveys (CSUR), (2):1163 221, 1989.
Eric Elmoznino, Thomas Jiralcrspong, Yoshua Bengio, and Guillaume Lajoic. A complexity-based theory of compositionality. ArXiv preprinl, abs/ als/22410.44177, 2024a. URL https://arxiv http://aaxxvv.oggbbba///a 14817
Eric Elmoznino, Tom Marty, Tejas Kasetty, Leo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie. In-context learning and occam's razor. ArXiv preprint, abs/2410.14086, 2024b.
URL
14086
Zhouxiang Fang, Aayush Mishra, Muhan Gao, Anqi Liu, and Daniel Khashabi. ICL Ciphers: Quantifying "Learning" in In-Context Learning via Substitution Ciphers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025. URL ftttp:://aaxxi.oog//aaaaa
19395
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017. URL http://proceedings. mlr. htt.ppeess/v70/fim117....t html
Veniamin S. Fishman, Yuri Kuratov, Maxim Petrov, Aleksei Shmelev, Denis Shepelin, N. Chekanov, Olga L. Kardymon, and Mikhail S. Burtsev. Gena-lm: a family of open-source foundational dna language mod- cls for long sequences. Nucleic Acids Research, 53, 2024. URL https://api. www.attccccooolr.ooooo
ID:
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems (NcurIPS), 2022. URL httpss://proceedingss
http://ppeeeeeeeepppeeeeeeeeeeee html
Dedre Gentner and Christian Hoyos. Analogy and abstraction. Topics in cognitive science, 9(3):672 693, 2017. URL https:://oi..rgg/ w11i/tops...iiiee.e 12278
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics (TACL), 2021. URL https:
abg//11//101..0
Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter. Is mamba capable of in-context learning? Ar preprint, abs/2402.03117, 2024. URL
15

---

Published in Transactions on Machine Learning Research (03/2026)
Katarina Grešová, Vlastimil Martinek, David Čechák, Petr Simeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24:25, May 2023. doi:
-220010----.-------... 01123-8.
Michael Hahn and Navin Goyal. A theory of emergent in-context learning implicit structure induction. arXiv preprint (Ff::2303.079771 2023. URL stttp:://arxtv.or/////////////// 07971
Damian Hodel and Jevin West. Response: Emergent analogical reasoning in large language models. arXiv preprint Xix::2308. 16118, 2023.
Douglas R Hofstadter. Analogy as the core of cogmition. The analogical mind: Perspectives from cognitive science, 2001. URL http::://.i.orrg////// /ttp:///0..6/mmmpprr/////1.....
K Holyoak, Dedre Gentner, and B Kokinov. The place of analogy in cognition. The analogical mind: Perspeclives from cognilive science, 119, 2001. URL 0001
Benjamin Holzschuh, Qiang Liu, Georg Kohl, and Nils Thuerey. PDE-transformer: Efficient and versatile transformers for physics simulations. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Fe- lix Berkenkamp, Tegan Maharaj, Kiri Wagstalf, and Jerry Zhu (eds.), Proceedings of the 4 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 23562- 23602. PMLR, 13 19 Jul 2025. URL https://proceeddngg. mlr. (hess//2557hhllscchhe 1htt
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V. Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, (7(1)222112 2120, 2021. doi: 10.1093/bioinforme bioinformatics/ bboinformatics//tabbBs... btab083. URL battps://dot: http:////o.oo/////////////////// htt..////boooooooooaacatttttaaaa
Zhongtao Jiang, Yuanzhe Zhang, Kun Luo, Xiaowei Yuan, Jun Zhao, and Kang Liu. On the in-context gener- ation of language models. In Conference Empirical Methods in Natural Language Processing (FMNLP)) 2024. doi: 10.18653/vl h02/l1//2222//2/ www.immmnllaaaae main.568. URL https://aclanthologyy wrg/2024..2022.. wttp://0022.......mm....... emnlp-main. 568/
Pranav Kantroo, Günter P. Wagner, and Benjamin B. Machta. In-context learning can distort the rela- tionship between sequence likelihoods and biological fitness, 2025. URL https://arxiv. org/ abs//504. 17068
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXic preprint arXiv:2001. 003611,...111...1111 2020. URL httpss//arxiv.. wwtp:/o/////////eeeeeoeee 2ttg/abs/2001.08361
Jonathan W Kim, Ahmed Alaa, and Danilo Bernardo. Eeg-gpt: exploring capabilitics of large language models for eeg classification and interpretation. ArXiv preprint, abs/2401.18006 2024. URL https:
//arxiv. waaxxv..rg//ae http//bs/2400.. 18006
Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers. ArXio preprint, abs/2211.044558, 2022. URL https://arrae org/abs/ abs/
0212.04453
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
URL
Juliana S Lancaster and Janet. L Kolodner. Problem solving in natural task a function of experience. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 9, 1987.
Tvan Lee, Nan Jiang, and Taylor Berg-Kirkpatrich Exploring the relationship between model architecture and in-context learning ability. ArXiv preprint, abs/2310.08049, 2023. URL https:://arrt wrgg //b///
2310. 08049
16

---

Published in Transactions on Machine Learning Research (03/2026)
Martha Lewis and Melanie Mitchell. Evaluating the robustness of analogical reasoning in GPT models. Transactions on Machine Learning Research, February 2025. URL,
t5cy5v9wph
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pro-training for unified vision-language understanding and generation. In International Conference Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, 2022. URL
https://proceeding: mlr. wwwps//vee//iiee... html
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, 2023a. URL https:// proceedings mlr. ppwps//202/1123/./ html html
Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint 13276, 2023b. URL https://arxiv..rrt abs/
2304. 13276)
Yingcong Li, Muhammed Emrullah Iddi, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning (ICML), 2023c. URL https://prrocednngs.oo mlr press/v202/112311 ottpp//eeenass.ttrprrre/////111/ html
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. In International Conference on Machine Learning (TCMT.), 2024. URL https:///rxxi: http://aaaxxvv..pp////.// ptg/pp////02....... 18819
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Scan Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iteralive refinement with self-feedback. In Advances in Newral Information Process- ing Systems (NeurIPS), 2023. URL httpp:/paapers aips.cc/papee.eee wwtpe/pppee.lleeeeeeeeeeeeeeeeee pttpes//pppr//23/// has0/////hh////////// URL 91adff0022222b105555566666666 Awet---ttttttttttiieeaeee Conftet-oooooooee html
Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. ArXie preprint, abs/2307.033766, 2023. 03576
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. Conference of the North American Chapler of the Association for Computational Linguistics (NAACL), 2022. URL
naacl- m0t.c.11maa.1
Aayush Mishra, Daniel Khashabi, and Anqi Tiu. Ta2: Alignment with iel activations improves supervised
fine-tuning. arXiv preprint arXiv:2509. 1226,, 2025. URL htttt::////////// org/abs/ argg/bbs/5099 2bs9/259..22211
Omar Naim, Guilhem Fouilhé, and Nicholas Asher. Re-examining learning linear functions in context. ArXio preprint, abs/ abs/2411.11465, 2024. URL http::://artt...o//////a//////// httpg//bs/2411111 11465
Ismail Nejjar, Faez Ahmed, and Olga Fink. Im-context: In-context learning for imbalanced regression tasks. XrXio preprinl, 2t////455.112000 2024. URL http://aaaav...rr/aaa/././///../ 18202
Ziv Nevo and Ran Yaniv.. On online learning of decision lists. J. Mach. Learn. Res., 3:271-301, 2002.
URL httttstttmmmmrrrr httpp/rrrr.pppppe///////aaaaaaa
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. HyenaDNA: long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36:43177- 43201, 2023.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Tewkowyez, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021. URL https:
7/arxiv. org/ http:/bb/211...000
Catherine Olsson, Nclson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Halfield-Dodds. Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodci, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022a. URL pattps://arxxv.org//bbsse 2109//
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.1 11 11895, 2022b. URL hatpss:/aare http://axivv.ooo org/ wtw.s//2229.... 11895
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in-context: Disen- tangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, July 2023. URL https::/aclanthool wwtphoolooo...g//2222.2/. 2023. findings-aci. acl.521
Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. Arxio preprint, abs/2306.0453 04891, 2023. URL pttps:://aratv..rr/abb//// 04891
Jongho Park, Jacseung Park, Zheyang Xiong, Nayoung Lee, Jacwoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? comparative study on in-context learning tasks. ArXiv preprint, abs/2440.04248, 2024. URL htttpp::/aaxvv.or/////aa /rtg/////aabb//220022222
Alec Radford, Karthik Narasimhan, Tim Salimans, Llya Sutskever, et al. Improving language understanding by generative pre-training. 2018. URL
John Carlyle Raven, John H. Court, and John Carlyle Raven. Manual for raven's progressive matrices and vocabulary scales. 1962. URL
wwwpa/ollarrrr//oo/ooooe/oooo Org/CorpusID:: 143337389
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL http::/papers. nips html
http:iccpcppeeefieeeeeeeeeeeeeee fiaaper_files/ppper/22333 /hash/
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv..rrg ///222220000
Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. Identifying semantic induction heads to understand in-context learning. arXiv prepriat arXiv:2402. 13055, 2024.
Brian H Ross. Remindings and their effects in learning cognitive skill. Cognitive psychology, 16(3):371-415 1984.
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt. John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess. In Advances in Neural Information Processing Systems, volume 37, pp. 65765 65790, 2024. doi: 10.52202//07901722102. URL httpps:/pproceeding..n www.:gg..euuiissccomm nwurips.cc/paperr fft.appppp//0//////////c.0ce6ee0 http:///eeettttto---eeaarrra-aae Conttraact-Coofeeene....
Niklas Schmidinger, Lisa Schneckenreiter, Philipp Seidl, Johannes Schimunek, Pieter-Jan Hoedt, Johannes Brandstetter, Andreas Mayr, Sohvi Luukkonen, Sepp Hochreiter, and Günter Klambauer. Bio-xistm: Generative modeling, representation and in-context learning of biological and chemical sequences, 2024.
URL https: //opps:/aace //txxxaxvvvv//////////ee/eaaa/ www.xivv.orgeeeee Borg/abs/2iu attp///1../.11555
18

---

Published in Transactions on Machine Learning Research (03/2026)
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating chain-of-thought demonstrations for large language models. In International Conference on Machine Learning, pp. 30706 30775. PMLR, 2023.
Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers learn in-context by gradient descent? In International Conference on Machine Learning (ICML), 2024. URL https:
//arxiv.org/absrt www.o///g./......../.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyzz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Koeurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Glolamidavvodii, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabhar- wal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomicj Bojanowski, Batuhan Özyurt, Bchnam Hedayatnia, Behnam Neyshabur, Benjamin In- den, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chen- lin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashoraft, Cristina Garbacea, Damien Sileo, Dan Garreltee Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí Gouzález, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Di- ganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, EI- li Paylick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Fthan Kim, Funice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemch Siar, Fernando Martínez-PPi Plumed, Francesca Happé, Francois Chollct, Frioda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch- López,,,ch-Lóézz, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Han- nah Kim, Hannah Rashkin, Hannanch Hajishirzi, Harsh Mchta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Boss- cher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Be- rant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Tgnatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Contreras-Ochanddo Louis- Philipp Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şencl, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramííee Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Ha- gen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar,
19

---

Published in Transactions on Machine Learning Research (03/2026)
Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mci, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Euta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millirre, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosamne Liu, Rowan Ja- cobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Deb- nath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapo- lating the capabilitics of language models. Transactions on Machine Learning Research (TMLR), 2023. ab/////bb../..e//////////
URL /tttps://axxv..oo
Claire E. Stevenson, Alexandra Pafford, Han L. J. van der Maas, and Melanic Mitchell. Can large language models generalize analogy solving like children can?, 2025. URL http://///araiior/r///////2111.1 02348
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. ArXiv preprint, abs/2301 222002222.5,,, 2023. URL //arxi://axxv... orw.oog.........o.. aataabbb/2300 http///20..0222222222 05222
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in- context learners. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL Loteeeeteeeererrrrs___etee Models are_ In-
thecvf. celecvf.coo/cootenne comm/content/(CPP2244///3 Str//Snn__eeeaiieeeeeeeaa Mnterattive_Mllimdaal.....
2024_02222220 20____2022___________________ www.pppprr.b.. html
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment. arxiv preprinl 2310.16944, 2023. URL htttp:///arx:
htgg/bbs/2310.. 16944
Ramzan Kh. Umarov and Victor V. Solovyev. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PLOS ONE, :211)110,,,, 2017. doi: 10.1371/journal. pone.0171410. URL
1 pone. 0171440
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023a. doi: 10.18653/v1, 1000///11122..mmmm/ 22023.emnlp-main. 609. URL https:
org/ 2023. emnlp-main. 609
20

---

Published in Transactions on Machine Learning Research (03/2026)
Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, and Ji-Rong Wen. Investigating the pre-training dynamics of in-context learning: Task recognition vs. task learning. ArXiv preprint, abs/2446.144022 2024. URL.
https://arxiv, http:////ariii.o/aaba/////e/ www.oogg////aeeoe/ 200bb///004112222
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang- Large language models are latent variable modcls: Explaining and finding good demonstrations for in-context learning. In
Advances in Newral Information Processing Systems (Ne, 2023b. URL http::/pappers... http:://ppaprs..is.... mip.. cc/
html
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa. Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Model with Self Generated Instructions. In Annual Meeting of the Association for Compulational Linguistics (ACL), 2023c. URL htttps://aaxii..rr/abbaaa
10560
Taylor Webb, Keith J Holyoak, and Hongjig Lu. Emergent analogical reasonni in large language models. Nature Human Behaviour, 7(9):1526 1541, 2023. URL htttsss//aaraa..oor//////2111111
reasoning
Taylor W Webb, Keith J Holyoak, and Hongjing Lu. Evidence from counterfactual tasks supports emergent analogical reasoning in large language models. PNAS nerus, 4(5))pgaf133, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Demny Zhou, et al. Ohhain-of-thouhht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS), 2022. URL /ttps://arxii..r htsp//bb//220/// 11903
Kevin Christian Wibisono and Yixin Wang. In-context learning from training on unstructured data: The role of co-occurrence, positional information, and training data structure. In ICML 2024 Workshop on Theo- retical Foundations of Foundation Models, 2024. URL http://ppaaarroeeo..ooooooooo/a/
wxa4
Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, and Iz Belt- agy. Continued pretraining for better zero-and few-shot promptability. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1517-4531 2022. URL https://arxiv www.oogg/g.....ooo... pdf// /2100
10258
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context lcarning as implicit bayesian inference. In International Conference on Learning Representations (ICLR), 2021.
URL https://arxiv org/ http:///1..0011
Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. Pretraining data mixtures enable narrow model selection capabilities in transformer models. arXiv preprint arXiv:2311.0087) 2023. URL https://
arxiv.org/ abb// htt///bb///3/0////33 2311.00871
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zhong, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghho Deng, Mci Li. Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https:
7arxiv org/ abs 09388
Shunyu Yao, Jelfrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. URL
net/pdf?id=WE vluYUL-X
Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovee, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogical reasoners, 2023. URL https: :/tp://///xvv../ org/abs/
2310. 01714
21

---

Published in Transactions on Machine Learning Research (03/2026)
Fred Zhang, Jiaxin Ye, and Zhuoran Yang. In-context multi-armed bandits via supervised pretraining. In NewrIPS 2023 Foundalion Models for Decision Making Workshop, 2023. URL http::://operreveee.n forum?id= forum/id=bIHopideuY
Luisa. M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning (ICML), 2019. URL
http://proceedings..oo mlr. pttp:sssr///ittttrrrfff.f http://paaacaaatpeeeeeesss///s/a html
22

---

Published in Transactions on Machine Learning Research (03/2026)
A Further Details on Synthetic Task Definition
A.1 Table of Primitives
The following table describes the thirty primitives used to construct the task space via composition their use is described in $3.2
Primitive inpernnainng_sarteeee one
Description
Produce a mask marking positions that differ from alternating 1010... pattern starting with 1 (1 mismatch, 0 match).
alternating_stare zero
Produce a mask marking positions that differ from the alternating 0101... pattern
starting with 0 (1 mismatch, 0 match).
center_mast
Zero out the first and last bits while leaving the interior bits unchanged; strings of length & become all zeros.
Bouble_roth
Circularly rotate the bitstring
positions to the left.
double_roo
Circularly rotate the bitstring two positions to the right.
edge_mask
Preserve the first and last bits and zero out every interior bit (length 0/1 strings pass
through).
flip_bits
Invert every bit, swapping Os for is and vice versa.
identity
Return the bitstring unchanged.
invert_prefix
Flip the bits in the left half of the string, keep the right half as is.
invert_auffix
Keep the left half as is and flip every bit in the right half of the string.
seep_even_positions
Keep bils at even indices (0-based) and zero out bits odd indices.
keep_add_positions
Keep bits at odd indices (0-based) and zero out bits even indices.
left_hail
Preserve the left half of the string and replace the right half with zeros.
majority
Fill the string with the majority bit from the input; ties resolve
constant
Returns random, pre-set constant.
minority
Fill the string with the minority bit from the inpul; ties resolve to all Os.
mirror half
Copy the left half of the string onto the right half in reverse order, keeping the center bit unchanged for odd lengths.
ones if palindrome
Output all if the input a palindrome; otherwise output all Os.
parity_fine
Output all Is when the input contains odd number of 1s; otherwise output all Os..
reverse bits
Reverse the order of the bits in the string.
right_ right_half half
Zero out the left half and keep the right half unchanged.
rotl1
Circularly rotate the bitstring one position the left.
rotrl
Circularly rotate the bitstring
position to the right.
shift_left
Shift the string left by one, dropping the first bit and appending a 0 on the right,
shift_right__
Shift the string right by one, inserting
the left and dropping the last bit. Replace every position with the first bit of the input.
spread first_bir
spread_last_te bit
Replace every position with the last bit of the input.
swap_ have_aalve
Swap the left and right halves of the string.
swap_ seap_pairs
Swap each adjacent pair of bits (positions 0/1, ..).
xor xor_with_s0
Can only be applied after another primitive. Computes the logical XOR between the original imput so and the output of the first primitive.
Table 1: The 30 unary primitives used to construct functions in F.
23

---

Published in Transactions on Machine Learning Research (03/2026)
A.2 Table of Functions
The following table describes the one hundred specific composilions of primitives used to construct the evaluation suite in $32
Function
Function
Function
identity
awap_ hrap_halves shift_left_zero
rotl1
invert_preefis
swap_ awap_halves halves shift_ Beaf__rigt___eer_
reverse_ bits
invert_auffix
shift_left_aet awap_halas halves
flip_bita
meta_constant
shift_rightt_zero + avap_ avap_halves
halp_halves
flip_bits -> reverse_bite bbts
- flip_bits
majority
rotll - deverse_bita
keep_odd__osiitions flip_bits bits
minority
reverse_ bits rotll
flip_bits keep_even_ pven_positions
parity_fill
rotll flip_bits
flip_bits
Atternating__tarrt.om
swap_halves reverse_bita
edge_mask flip_bits
Eiternatinngetartt..o
haae_ 7 flip_bite
center mask 7 flip_bits
left_half
double_rote + flip_bits
shift_left left_ zero + crep__vee__ossttiee
right_han half
rotrl - flip_bite
shift_right__ keep_odd_ keep_odd_poeitions
chift_left_ee
double_roti
spread_first__e
flip_bits
sne__even_position
rotrl
spread_last_
flip_bits
shift_right_zen zero posiions
double_rotz
left_half + flip_bits
deep_even_position + reverse_ bita
ones_ wt_.ffffaadddoeeeeeeee
right half + flip_bita
keep_odd_ppelttona. + severse_bits
mirror hirror_hallf half
hitp_bits left_hal taalf
shift_left_zers parity_fill
apread_first. fitst_bit
flip_bite right_ half
ehift_right_zen parity_fill
spread_last bit
double_rot! reverse_bie
parity_ shift Shift_left__eet left_
invert_prefix
rotl1 -> swap_halves
parity_fill >> shift_right__ero
invert_suffix
xor_ zor_vith_s0
spread_first_bit - shift_ _eaft_eree__e
neta_constard
flip_bits + xor with_s0
spread_last. bit + shift_rigge Bicts_rrght____et zero
abift_left_zerr
fhe__if_palindrome flip_bita
epread_first_bis
postttoonn
sbift_right__ero
flip_bits
half
apread_laat__b1
seep_odd_positions
swap_pairs
invert_prefix - reverse_
spread_first_bit edge_mask
keep_even_pentt pot_evnnnpssiiooss
left_half + reverse_bit bits
spread_last_bin bitt + edge_mask
keep_odd_
seep_odd_positions positions
Light_ball + reverse_ bits
epread_first bit + center mask
edge_mask
fality_fill flip_bits
spread_last_bin bitt > center_mask
center_mast
rotll puread_first... bit
rotl1 shift_les lefft_leftt_ero
xor_withe _or_with__0
shift_left__ terr flip_bits
shift Beaift_righh___ zero
flip_bits reverse_ble bits
thift_right zero + flip_bits
chift_left_zeet + rotll
rotli - reverse_bite
flip_bits >> shift_left_zen
shift_right_zep - rotll
reverse_bits rotll
flip_bits + shift_right, zero
reverse_bitt bits + adge_masst
rot11 + Flip_bita
awap_pairs flip_bits
reverse_ bits
_ask
avap_halves reverse_bite
shift_left_eee
reverse_bie
edge_mask shift_left_zerz
awap_halves + flip_bita
seftt_right_2erg + reverse_bies bitt
edge_mask - shift_right_zet
double_rotl > flip_bits
spread_first_ bit - Flip_bits
shift_left__at >
rotrl + flip_bita bita
shift_left shift_left__eer
_eeet_erroo__o + _mask
thift_left_eett + _rap_paira
Table 2: The complete set of 100 functions in F, consisting of 30 single primitives and 70 composed functions (( - (x) = (x)))
24

---

Published in Transactions on Machine Learning Research (03/2026)
A.3 BitLoad of Functions
Function
Composition
Load
Function
Composition
BitLoad
identity
8
ratl1
reverse reverse_bits
8
flip_bits
swap halve
8
majority
minority
parity_fill fill
Alternatiigg_start.oom
Anternating_start...
left left_half
right,
double_roo
8
rotrl
double_rotz
8
ques_if_palindrom
mirror_hall half
4
spread_first first_ bit
spread_last__
invert_prefix
invert_suffix
nett_ccn
flip_bita + reverse_bita
8
rotll + severae_bita
reverse_bis bits + rotll
8
rotl1 - flip_bits
awap_halves reverse_bie bits
8
swap_halves flip_bite
double_roth flip_
rotrl flip_bits
spread_first___
flip_bits
spread_last_brt
flip_bits
left_balf flip_bits
4
right_half flip_bits
flip_ slip_bits + left_half
4
ilip_bite + right_hall
double_roti reverse_bits
8
rotll swap_ wra__aalves
xor_with_s0
ilip_bits xor_xith_s0
ones_if_palindree + flip_bits
flip_bits mirror _allf
invert_prefix + reverse_biie bits
8
left half + reverse_ bits
Bight_halff + reverae_bita bits
4
parity_fill fill + flip_bits
rotll + spread_ sireaddfrrst_it
1
shift_left_zan
shift_right__ero
7
swap_pairs
seep_evea_poeitions
keep_odd_positee
edge_mask
center mask
flip_bits
7
shift_rightt_nn zero flip_bits bits flip_ bits_bis___ + Shift_right zero
flip bits + shift left_ zero zero
7
svap_pairs + flip_bits
8
bitts + reverse_ bits
shift_right_zero reverse_bita
swap_halves shift_left__e
svap_halves sbift_right_eee
shift_left_zero + swap_balves
shift_rrght__eee zero + swap_hawlllleappaaaa halves
7
ttep_even_positions + flip_bits
keep_ _d___odd__ssttinn + flip_bita
4
flip_bita + tete__even_posttaae
flip_bite keep_ seep_odd_positions
4
edge_mask flip_bite
conter_mask flip_bits
6
shift_left_zerr sre__evan_position
shift_left_zoro geep_odd_position
shift_right__orr keep_evon_positions
shift_right_.err + keep_odd peep_ood_
keep_even__oeltionn + reverse_bite
_op_oodd_ositiion reverse_bite
4
shift_left_zerr parity_fill
shift_right_zerc parity_fill
7
parity_fill shift_left_zerr
parity_fill shift Beeiiftrrrgh__eee_
spread_firet cund_firrt_brr shift shift_left,_e left__eroo
spread_last_unn + shift_ shift_right__tte
spread_fires
keep_even positions seppiestooosssi
apread_last_bit - keep_odd_positionn
spread_first_bn > edge_mask
Saread_last_hee bit 7 edge_mask
spread_first first. bit - center_man
spread_last_bbr bit - center_ mask
1
rotll - shift_lefttzere shift_left. + rotll
rotll + shift_riit shift_right, 1at__rrght__eee__
7
shift_right_zero
rotll
reverse_bite >> edge_mask
reverse_bita +) venter_mash
edge_mast > shift_left_te
edge_masks + shift rightt zero
shift_lefft_e zero + shift_left,
Shift_left_zent + swap_paire
7
shift_left_zer zero + rdge_nass.
Table 3: BitLoad for every primitive and composed function in F. Used in 4.3
25

---

Published in Transactions on Machine Learning Research (03/2026)
B A Global Metric Over All Programs
For a given function f, model M, and number of in-context examples n, we define the average accuracy over all context sets En = {E C S [E] = n)::
Ay(M, n) EN (15) - n)) EEEN DES/E
E E
E A,,,
The overall benchmark score across all functions F is then
P(M,n) [P|
Σ (,, fEP
Because exact evaluation is intractable (for 8, |E N) 1.016 x 1077) we estimate A)) via Monte Carlo, discussed in 3.4
(1S) n)
(256)
248
C Full Accuracy Results
Here we show the full table of accuracies used in $4.1 to analyze the ICL capabilities of Evo2 and Qwen3 and perform the necessary statistical tests.
Model
1 Shot
Shots
Shots
Shots
Shots
32 Shots
64 Shots
128 Shots
Qwen3 0.6B
13.4+2.3
14.1+2.3
16.0=2.6
19.5
16.1
16.842.
19.1
23.642.9
Qwen3 1.7B
13.142.2
15.0+2.7
17.2+2.8
16.442.
16.6
20.442.9
24.523.4
28.543.5
Qwen3 4B
13.542.4
15.2+2.4
17.022.6
15.442.6
18.4g2.7
24.84332
24.523.4
29.443.4
Qwen3 8B
13.512.3
15.412.5
16.9
19.612.7
20.612.8
27.9 23.3
30.9
32.2
Qwen3 14B
14.0.24
14.1424
16.822.7
20.6
22.913.11
25.543.2
32.613.7
33.813.5
Evo2 1B
13.142.2
15.0+2.4
17.1
18.2
22.0#2.9
23.8+2.8
26.5
30.8
Evo2 7B
14.542.4
15.8
17.6
22.6_2.9
25.2+3.0
29.8
32.2
41.0+3.4
Evo2 40B
15.0=2.6
15.512.5
17.612.8
21.643.1
26.413.1
28.943.1
34.543.2
41.143.3
Table 4: In-context learning performance across model families and shot counts. Values show accuracy + standard error. All numbers are percentages. Bold numbers show the best performance within a model lamily. Models are ordered by parameter count within each family.
Meta-Regressiin for ICL Efficacy with Number of Demonstrations
We perform linear regressions to predict accuracy from shot count with each model. For each model M. fit the linear regression: P(M, n) ao + a (M) log(n) The raw regressions are shown in Fig. 5a Predictably, all a₁ are positive all models capable of learning in-context.
We can interpret as representing the model's base accuracy at the task, what it would logically achieve with only one shot to identify the task. We can then interpret ai as the model's ICL efficacy: the speed at which it adapts to the task being presented and at which its accuracy improves. Analyzing how these values change across parameter values reveals insights into the ICL abilities of both the Qwen3 and Evo2 models.
First, we analyze how og changes in each model family this analysis can be seen in Fig. 5b Both Evo2's and Qwen3's initial dg remains essentially constant model-to-modell, indicating that all models have similar levels of few-shot bascline performance. Notably, Evo2 and Qwen3 have essentially identical intercepts at around 0.12. This implies that despite drastically different training data, the overall amount of prior knowledge the models have coming into this task is roughly similar. This rules out Qwen simply having *less experience" with this sort of task.
If one looks a1 ICL efficacy in Fig. 5c a dramatically different picture painted. Here, both Qwen3 and Fvo2 follow similar patterns with significant. difference. Qwen3's TCT. efficacy increases monotonically
26

---

Published in Transactions on Machine Learning Research (03/2026)
with respect to parameters, more than doubling from the 0.6B to the 14B. Evo2 follows suit (albeit less dramatically), with ICI. efficacy monotonically increasing from the 1B to the 40B.
In absolute terms, however, Evo2, when parameter-matched, adapts in-context faster than Qwen3 does. Evo2 40B outperforms Qwen3 14B significantly, and it takes until Qwen3 8B to exceed the ICL ability of Evo2 1R.
Taken together, this data suggests that Qwen3 and Evo2 have similar amounts of pretraining exposure to be able solve these tasks, and that Evo2 simply has better overall ICL capability (in this regime) even though Qwen's is ICL ability increases more rapidly with respect to parameters.
Accuracy Shots with Log-Linear Fit
Model
Qwe13 0.63
0.45
0.40
Qwerd 148
Evo2 18
0.35
78
Accuracy
XX
0.30
0.25
0.20
0.15
0.10
16
64
In-context examples (shots)
(a) Few-shot performance of Qwen3 and Evo2 models. All models show consistent linear improvement with respect contrast, no such improvement occurs for the naive baseline.
(sg(sho
Shgt Regression Intercept Patameters
Sog-Shat Regression Slope Parameters
Ouen3
EX0
Exsi
0.01
0.13
0.12
0.00
0.15
0.30
10P
10km
Paramettrs
Parameters
(b) Baseline accuracy decreases slightly with scale the model gains more parameters for both Evo2 Qwen3.
(c) ICL rate vs. model size: sharp gains to 4B for Qwen3; mild boost from Evo2 1B 7B; both plateau after 4 7B.
Figure 5: Few-shot behavior and scaling trends across Qwen3 and Evo2.
27

---

Published in Transactions on Machine Learning Research (03/2026)
E Demonstration of Evo2's ICL Ability on a Genomic Task
We demonstrate that Evo2 7B can perform few-shot in-context learning (ICL) on hu- man nontata promoters, a binary genomic classification task in which the model must predict whether a 251 nucleotide-long human DNA sequence is a promoter. To make the task non-trivial, sequences in the positive class are restricted to non- TATA promoters, removing the "TATA box", a pattern that serves as an obvious, exploitable marker (Umarrv Solovyev 2017 Grešová et al. 2023
For each test query, construct few-shot prompt by concatenating balanced examples from the training set (equal numbers of promoters and non- non-promoters), followed by held-out test sequence. Each training example is immediately followed by an encoded label. The label consists of 24 mucleotide buffer (a single mucleotide repeated 24 times) and then a 24 nucleotide label run (another nucleotide repeated 24 times). The buffer muclootide and the two label nuclcotides (for class vs. class 1) are ro-sampled uniformly at random without replacement from (A, C, for every trial, so the mapping changes across evaluations. This prevents any mucleotide-ppecffic biasing effects.
To predict the test label, score two candidate prompts class-0 label and one followed by the class-1 1 label and choose the label with lower perplexity on the label run (i.e., computed only over the final 24 label nucleotides, excluding the buffer). This lets us measure whether Evo2 can infer the label mapping from the few-shot context and apply it to the query sequence.
where the test sequence is followed by the
We observe monotonic improvement with respect to shot count, with accuracy rising from below random baseline at 1-shot (48.2%) to 63.6% at 16-shot. These gains further rise to 68.6% after 256 shots. A saturated power law achieves an R² of 0.998 in the 2-256 shot regime, which is characteristic of few-shot ICL. Furthermore, show via ablation that the improvement is due to correctly-labeled examples presented in context-randee permuting the labels in the 16-shot regime drops performance from 63.6% back to random chance (50.2%).
random baseline (0.5)
0.70
sweep
ablation
0.65
Accuracy
0.60
ablation
0.50
0.45
0
8
16
32
64
128
256
Shots
Figure 6: Performance of Evo2 7B on human nontata paroaaterroosssssse at varying shot counts. Error bars are + one standard error. Evo2 7B's performance shows clear monotonic improvements with respect to shot count.
28

---

Published in Transactions on Machine Learning Research (03/2026)
F Analysis of ICL with Program Synthesis Tasks: Bit-Diversity
BitLoad captures a mcasure of theoretical information bottlencck required to make prodictions in program synthesis tasks. However, this does not always correlate with what models find easy to do. In Fig, S and Fig. g we plot the accuracy of Evo and Qwen models with respect to BitLoad. We notice that few high Bitdoad tasks have high accuracy compared to all medium Lotd tasks, illustrating that some high Load tasks can be casy for these models to solve, probably due to the nature of patterns found during pre-training.
To further analyze the nature of ICL, exhibited through these tasks, we define BitDiversity as the number of minority bits in the output string. In Fig. [0 and Fig. [1] we plot the accuracy with respect to Diversyrsity These plots try to estimate the effect of entropy in the output on model performance, and we see a more expected trend: models tend to perform better on low-entropy outputs, regardless of BitLoad. However, bin-wise performance trends are always increasing with shots, supporting the central hypothesis that ICL increases with increasing number of demos in these models.
Prevalence of O-BitDiversity outputs. Looking at the expected and prodicted outputs of trials from our tasks (Fig. 12 we made a few interesting observations about O-BitDiversity (BD) outputs.
Around 25% of true targets are O-BD. This is a significantly high number as we only have 2 0-BD bit strings in all possible bit strings of any length K. It implies that many of our tasks create low BitDiversity outputs on random inputs, i.., O-BD outputs.
Models tend to produce a lot of 0-BD outputs, much higher than the number of O-BD true targets in the low-shot regime. But this nunber quickly drops to the expected number with higher shots. A large portion of the baseline (1-shot) performance can be explained by this prevalence of O-BD outputs, but with more shots we get stronger evidence of ICL with increasing correctly predicted non O-BD cases.
Understandable Mistakes. A potential confound is what we will call 'understandable mistakes'. These mistakes occur when the model outputs an incorrect answer that would be correct for some other programs given the few-shot context. Formally, a model's output y (see 3.4 for notation): y M (e1 f(e1), E2 f(22).....e + f(eN), 2) is understandable mistake if y * fz) but there exists f2 E F such that Ves ((a)=]((e) and y f2(x).
These understandable mistakes are an alarming confound at low shot counts, but their effect vanishes by N = 16. They occur most in tasks with low BitDiversity, which can often be confused with each other. Figure 7 below shows how understandable mistakes in Qwen3-4B's inferences de- cay exponentially as the number of shots increases.
Daent-45- Base
14-
Undersiantann misakes 198)
Noise due to Monte Carlo Trials. We use m = 8 for our per-function Monte Carlo trials when we compute accuracy. This has little aggregate impact when comparing model per- formance across the entire suite, but drastically reduces the significance of results when comparing models and trends at the task- We only have eight discrete bands of ac- curacy at which we can estimate a model's per-task per- formance which reduces the expressiveness of regression. Worse, this can lead to models gelting a lucky prompt or two with a O-BD output, which raises performance to 12.5% or 25% without the model truly understanding the task. Ad- dressing these confounds would simply require increasing m by an order of magnitude or so. Alternatively, tasks of interest could be identified and m selectively increased for those tasks to enable more nuanced analysis.
In-contost examples
Figure Qwen3-4B's rate of understandable mistakes with respect to the number of shots. Despite starting at 16% in the one-shot regime, they fall to less than 1% by 8 shots and van- ish entirely at 32 shots. This underscores how understandable mistakes are only a confound at very low shot counts and can essentially be ignored past 4 shots.
29

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Num IC=1
Model: Evo2-76 IC=1
Model: Fvo2-408
Num IC=1
Acouacy
Accuracy Num ICm2
Accuracy Num ICm?
Accuraty Num IC=2
Bit
Accuracy Num IC#4
Accuracy Num IC#4
Accuraty Num IC#4
Accuracy Num 1Cm8
Accuracy Num IC=8
Accuracy Num ICm8
Accuracy Num IC=
Accuracy Num IC=16
Accuracy Num
Accuracy
Accuncyy
Accurary
IC-32
Num
Accuracy
Accuracy
Accuracy
(C=64
Accuracy
Accuracy Num IC-128
Accuracy
Num IC=128
Accuracy
Accuracy
Accuraty
Figure 8: BitLoad vs Accuracy for all Evo models at all ICL shot-numbers. The per-BitLoad distribution of model performance at each shot level shows an uneven affinity of models for solving tasks at different BitLoads. However, all trends support our overall hypotheses.
30

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Qwen3-0.68
Model: Owen3-1.77 Num IC-1
Model: Qwen3-4B Num IC-]
Model: Qwen3-80
Model: Qwen3-140
Num
Num IC=1
Num
Moan Acores
Load Load
Accuracy
Accuracy Num
Accuracy IC=2
Accuracy Num
Accuracy
Num
Num
+
Accuracy
Accursity 16-4
Accu may IC-4
Accuracy Num IC-4
Accuracy Num IE=4
Num IC-4
Bit Lood
+illi
+ill++
Accuraty Num ICm8
Accuracy Num IC=8
Accuracy Num ICmB
Accuracy Num ICm8
Accuraty Num IC=88
Bit Load
Accuracy Num IC
Accuracy Num IC-16
Accuracy Num IC=16
Accuracy Num IC
Accuracy Num IC=16
Bit Load
Accuracy Num K2=32
Accuracy Num 12=32
Accuracy Num IC=32
Accuracy Num 10
Accuracy Num 30
BIR Load
Accuracy Num K4-4
Accuracy Num IC-64
Accuracy Num IC-64
Accuracy Num IE
Accuracy Num IC-64
Bit Load
Accuracy
44
Accuracy Num 1C=128
Accurccy Num IC=120
Accuracy Num IC=128
34
Accurary Num KC=12
Num IC=128
(o0d
Accuracy
Accuracy
Accuracy
Accuracy
Accuracy
Figure 9: BitLoad vs Accuracy for all Qwen models at all ICL, shot-numbers.
31

---

Published in Transactions on Machine Learning Research (03/2026)
Model:
Model: Num [C=]
Model: Evo2-408
(C=1
Num IC=]
Bit Diversitty (iN))
233
--5
Accuracy Num IGm
Accuracy Num ICm?
Accuracy
Num IC=?
(meur manemaae (84))
175
Bit Diversity
135
1:0
Accuraty
Accuracy
Jum/C=4
Num IC#4
Num IC#4
335
1.5
Diversty
1.5
Accuracy Num Cmm
Accuracy
Accuracy
Num ICm8
Num ICm8
(imean moneraalttt
113
B8 Dive
6.7
Accuracy
Accuracy
Accuracy
Num IC=16
10=16
Num IC=16
Emeen coonemaet sIt)
325
1.5
2.9
Accuracy
Accuracy
Accuracy
IC-32
Imnean con-mappdly bits)
129
Bit Diverstyy
235
225
225
Accuracy
Accuracy
Accuracy
IC=64
DicOwersit
Accuracy
Accuracy Num IC- 128
Accuracy
Num
Diversity
175
Accuracy
Accuracy
Accuracy
Figure 10: BitDiversity vs Accuracy for all Evo models at all ICL shot-numbers. Model performance follows more natural pattern of increasing performance with decreasing output entropy. This highlights that it is difficult for models to decipher ICL patterns for high entropy outputs.
32

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Owen3-0.60
Model: Qwen3-1.70
Model: Owen3-40
Model: Owen3-80
Model: Qwen3-148
IC=1
Num IC-1
Num
Num [C-]
Num
80 oieesey (1))
Mor Aconney
(mees con-matelt
2.35
275
2.2
125
626
024
525
Accuracy
Accuracy
Accuracy
Accuracy
Accuracy
IC=2
Num
IC=2
Num
BitDhersky (9800)
(mowen mon-maa
2.20-
323
Accuracy Num I5+4
Accuracy
Accuracy
Accuracy
Accuracy
IC-4
Num IC-4
IC-4
Num
(meen mon corvenluity bla)
Bt Diveratyy
225-
235
175
Accuraty Num KCm8
Accuracy
Accuracy Num ICm8
Accuracy
Accuracy
Num IC=8
Num ICm8
Num IC=88
3.75
175
375
BR Diversty
1.7
124
Accuracy Num IC=16
Accuracy
Acoutacy Num IC-16
Accuracy
Accuracy Num IC-16
Num IC=16
Num IC=16
(moen (44--77 bin))
0.70
125
L71
8% Diverity
123-
L2-
Accuraty Num IC=32
Accuracy
Accuraty Num IC-
Accuracy
Accuracy Num IC=32
Num ICm32
Num IC=32
Bit Dieewerrzay. 61N)
2.5-
315
425
A/3
A/-
325
325
(mean conymappaa
125.
124
674
Accuracy Num IC-64
Accuracy Num IC-64
Accuracy Num IE-64
Accuracy Num IC-64
Accuracy Num IC-64
[mwen wan.myaalyyy
335
325
88, Dversty
123-
624
Accuracy
Accuracy Num IC=120
Accuracy Num 128
Accuracy
14
Accuracy Num IC=128
Num KC=12
Num IC=128
121
121
BR Diversity
326
525-
Accuracy
Accuracy
Accuracy
Accuracy
Accuracy
Figure 11: BitDiversity vs Accuracy for all Qwen models at all ICL shot-numbers.
33

---

Published in Transactions on Machine Learning Research (03/2026)
Model: Evo2-1B
Model: Qwen--.68
Corest
Consti 040
Coreel 940
Camisi nan 660 proaccoons
Comeil pradictore
pred-cttom
TMaal 2:00
Target frotk (8:60)
Sorgr Predic sre ICa
Nom
shotts
Num shots
Model: Evo2-7B
Model: Owen-1.
Model: Qwen
Num caros
shots
Kum shots
Num shots
Model: Evo2
Model: Owen-18
Owen:
Num caros
Num shots
Kum
Num shots
Figure 12: Enumerating cases of 0 BitDiversity. For our defined tasks, around 25% of the true targets have Diverityttty (BD). With low-shots, models tend to produce a large number of O-BD predictions. But this number decreases significantly with increasing shots and lends lo match the actual prior value. Similarly, majority of the baseline (L-shot) performance of models can be explained through O-BD outputs. With a higher number of shots, the model starts learning higher entropy patterns and presents stronger evidence of ICL.
34


Reviews:
[{"review_id": "6NQN5mtrkk", "reviewer": "Reviewer_xCj6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors compare in context learning (ICL) of large language models (LLMs) trained on natural language (Qwen) and genomic data (Evo2).\n\nThey design a common set of Boolean functions, and investigate how good Qwen and Evo2 are at learning them in context.\n\nIn both cases, ICL “works” in the sense that the larger the size of the context (or the size of the model), the better the accuracy. Interestingly, the improvements appear to occur at the same rate, log-linear in the number of in-context examples.\n\nThey also investigate which Boolean functions are easier to learn for Qwen and Evo2.", "claims_and_evidence_yn": "Yes", "claims_explanation": "All in all, the main claim (that ICL works at the same rate for both Qwen and Evo2) is well-supported by the experiments, with numerous statistical tests, in particular for the log-linear relationship. Confidence intervals on Fig 1.(4)  and Fig. 2 would be beneficial though.\n\nI do have an issue with the sentence “To our knowledge, the only claimed instance of emergent ICL outside language is in vision: Bai et al. (2024)” (just before Section 3). I am no ICL expert but I believe there are several other prior examples :\n\n-\tOn time-series: the chronos models (Ansari et al., 2024, 2025) and TimesFM (Das et al., 2025)\n-\tBio-xLSTM, on biological sequences (Schmidinger et al., 2025)\n\nThe tasks involved there are quite different from the one considered in this submission, so this is not a big issue, but these papers should be discussed (in particular Bio-xLSTM that does experiments on DNA data).\n\n# Additional references \n\n- Ansari et al., Chronos: Learning the Language of Time Series, TMLR 2024\n- Ansari et al., Chronos-2: From Univariate to Universal Forecasting, arXiv preprint arXiv:2510.15821, 2025\n- Das et al., A decoder-only foundation model for time-series forecasting,  ICML 2024\n- Schmidinger et al., Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences, ICLR 2025", "audience_yn": "Yes", "audience_explanation": "I think that the careful experimental design is interesting and original enough to have some appeal for the TMLR audience. I liked in particular the design of the Boolean functions, and the detailed analysis of the results.\n\nOne limitation of the paper is that there is not much insight to be gained about the quantitative comparison between language and DNA, because the two tokenizers are fundamentally different. This is acknowledged by the authors when they say\n\n> Thus these results should be\ntaken as an existence proof of ICL in Evo2, not a definitive statement of Evo2 having more ICL ability than Qwen. We leave a comparison of these models across broader tasks to future work.", "requested_changes": "- Add confidence intervals and discuss them (see above)\n- Discuss prior work on ICL on non-text data", "broader_impact_concerns": "I have no concerns about the Broader Impact Statement written by the authors.", "additional_comments": "Isn't the \"bitload\" just the number of relevant features, often used in statistics and learning theory, e.g. in Nevo & Yaniv (2002)?\n\n# Additional reference\n\nNevo & Yaniv, On Online Learning of Decision Lists, JMLR 2002", "extra_scores": {}, "extra_text": {}}, {"review_id": "tEdu2wA29B", "reviewer": "Reviewer_6Sjp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors are interested in the question: is \"in-context learning\" (ICL) unique to human language, or does it naturally emerge whenever a big model is trained to predict patterns? To get to the bottom of this, they ran a controlled experiment comparing a DNA model (Evo2) against a standard LLM (Qwen3), using abstract bitstring puzzles that could be translated into both genetic code and text. They found that the genomic model learned from in-context examples just like an LLM does, with performance improving reliably as it saw more demonstrations. This effectively suggests that the ability to learn on the fly isn't a special feature of human language, but rather a universal byproduct of training large models to compress complex, structured data. This is surprising, and might be theoretically significant, to see that a model trained solely on biological sequences picks up \"in-context learning\" behavior similar to that of a chatbot.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors support their primary claim through a rigorous, controlled experimental setup. By abstracting \"reasoning\" into symbolic bitstring tasks (mapped to nucleotides for Evo2 and digits for Qwen3), they successfully isolate the mechanism of in-context learning from the semantics of human language. The results in Figure 2 are particularly convincing: they show a clear, log-linear improvement in accuracy as the number of demonstrations increases. The authors also attempt to match model parameter scales (e.g., comparing the 14B Qwen to the 40B Evo2). Section 4.3 (the BitLoad analysis) moves beyond simple accuracy metrics to show how the models fail (Qwen struggles with high-dependency tasks, Evo decays more slowly).", "audience_yn": "Yes", "audience_explanation": "There is a huge ongoing debate about why in-context learning happens (is it the objective or the language?). By providing evidence that ICL emerges in a non-linguistic domain (DNA) purely through scale, this paper offers a crucial data point that separates the \"magic of language\" from the general mechanics of sequence modeling.", "requested_changes": "1. Demonstrate ICL on one actual genomic task using the same few-shot protocol. This would confirm that the ICL capability detected via bitstrings translates to the domain the model was actually built for.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4MCWLH5uQ9", "reviewer": "Reviewer_yVdW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary\n\nThis paper investigates whether emergent in-context learning (ICL) can arise in a genomic sequence modeling setting purely from large-scale next-token prediction. The authors study Evo2, a genomic foundation model trained primarily to predict the next nucleotide (A/C/G/T). To enable a controlled comparison with language models, they design a domain-agnostic symbolic pattern induction benchmark: the same underlying bitstring-to-bitstring transformations are rendered either as “language-like” digit strings or as “genome-like” nucleotide strings with randomized mappings.\n\nAcross model families and sizes, they find ICL-style scaling: accuracy improves approximately linearly with log (number of demonstrations). They further report qualitative differences in how performance scales with demonstrations vs model size, suggesting Evo2 benefits strongly from additional in-context examples under this encoding, while the language models benefit more from scaling model capacity.\n\nStrengths\n\n1. Novel cross-domain angle on ICL: The work provides a compelling existence proof that ICL-like behavior is not exclusive to human language training data. By evaluating next-token predictors in a non-linguistic symbolic modality (genomics), the paper helps disentangle whether ICL is a consequence of natural language semantics/compositionality or a more general property of sequence prediction at scale, and it connects naturally to “AI for science” motivations.\n\n2. Careful, controlled experimental design: The benchmark construction is well-motivated: tasks are abstract, deterministic transformations, and the randomized digit/nucleotide mappings reduce trivial memorization. The evaluation includes a simple baseline and uncertainty estimates, making the reported trends easier to trust.\n\nWeaknesses \n\n1. Cross-model differences are not sufficiently contextualized: The paper would benefit from a clearer, more explicit comparison of the evaluated model families (architecture, tokenizer/vocabulary regime, context length, and training distribution). These factors likely affect performance on low-vocabulary symbolic strings and could partially explain observed differences between Evo2 and language baselines.\n\n2. “Linguistic” modality may be an interface mismatch: Although digit strings are a practical rendering of bitstring tasks, it is debatable whether this constitutes a fair “linguistic” test. In particular, the language model’s performance may be limited by the prompt encoding/interface rather than a lack of ICL ability. Additional prompt/encoding variants would strengthen the claim that differences reflect underlying capability rather than representation alignment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, they introduce a controlled bitstring-to-bitstring symbolic induction benchmark that can be rendered in both “genomic-like” and “language-like” formats, and measure performance as the number of in-context demonstrations increases. Across settings, accuracy improves systematically with more demonstrations, and the reported trends provide clear empirical support for the paper’s central hypothesis.", "audience_yn": "Yes", "audience_explanation": "I think this paper both benefit people interested in AI for science, machine learning, and NLP.", "requested_changes": "As seen in weaknesses: \n\n1. Clarify the difference between the two foundation models, genomic and linguistic. Discussing the impact of varying factors on the final in-context learning performance. \n\n2. Explain the fairness of linguistic encoding by using digits in the experiment setting. Evo2 relies on a tiny alphabet (A/C/G/T) and is trained entirely on long contexts over that alphabet; Qwen3 is not. Even with digit-token parity, the training distribution matched to “weird symbolic strings” likely differs substantially.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)