AntigenLM: Structure-Aware DNA Language Modeling for Influenza
Authors: Yue Pei
Xuebin Chi
Yu Kang
Venue: ICLR 2026 Poster (2026)
Keywords: Influenza A
DNA
Genome
Language Model
Foundation Model
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Language models have advanced sequence analysis, yet DNA foundation models often lag behind task-specific methods for unclear reasons. We present AntigenLM, a generative DNA language model pretrained on influenza genomes with intact, aligned functional units. This structure-aware pretraining enables AntigenLM to capture evolutionary constraints and generalize across tasks. Fine-tuned on time-series hemagglutinin (HA) and neuraminidase (NA) sequences, AntigenLM accurately forecasts future antigenic variants across regions and subtypes, including those unseen during training, outperforming phylogenetic and evolution-based models. It also achieves near-perfect subtype classification. Ablation studies show that disrupting genomic structure through fragmentation or shuffling severely degrades performance, revealing the importance of preserving functional-unit integrity in DNA language modeling. AntigenLM thus provides both a powerful framework for antigen evolution prediction and a general principle for building biologically grounded DNA foundation models.

Published as a conference paper at ICLR 2026
ANTIGENLM: STRUCTURE-AWARE DNA LANGUAGE MODELING FOR INFLUENZA
Yue PeilA Xuebin Chil,,, Yu Kang2.4.
'Computer Network Information Center, Chinese Academy of Sciences 2Beijing Institute of Genomics, Chinese Academy of Sciences "China National Center for Bioinformation
UUniversity of Chinese Academy of Sciences
ypei @cnic.cm cn, ch 1 chi@scces @ssccas.cm cn, kangyEbig warbbigg.c... cn
ABSTRACT
Language models have advanced sequence analysis. yet DNA foundation mod- els often lag behind task-speeific methods for unclear reasons. We present Anti- genLM, generative DNA language model pretrained on influenza genomes with intact, aligned functional units. This structure-aware pretraining enables Anti- genLM to capture evolutionary constraints and generalize across tasks. Fine- tuned on time-series hemagglutinin (HA) and neuraminidase (NA) sequences, AntigenLM accurately forecasts future antigenic variants across regions and sub- types, including those unseen during training, outperforming phylogenetic and evolution-based models. It also achieves near-perfect subtype classification. Ab- lation studies show that disrupting genomic structure through fragmentation or shuffling severely degrades performance, revealing the importance of preserving functional-unit integrity in DNA language modeling. AntigenI.M thus provides both powerful framework for antigen evolution prediction and general princi- ple for building biologically grounded DNA foundation models.
INTRODUCTION
Influenza viruses evolve rapidly to escape host immunity, driving seasonal epidemics and neces- sitating frequent vaccine updates (Han et al., 2023). Accurate forecasting of viral evolution therefore essential for vaccine strain selection and for mitigating global public health burden (Matz) & Ellebedy 2025). Current vaccine recommendations integrate large-scale genomic surveillance, antigenic characterization, and epidemiological monitoring, coordinated by the World Health Orga- nization (WHO) (Buchole et al. 2025).
Traditional forecasting approaches rely phylogenetic tree dynamics and nutation-basse predic- tive models (Hadfield et al.] 2018 Huddleston et al.) 20200 Luksza & Lässig 2014; [Neher et al.] 2014]. More recently, deep learning models (Mehrotra
2025 Lou al. 2024) have been shown to accurately predict clade-level mutation trajectories. beth-1, 7 which explicitly models site- wise substitution dynamics, improves genetic matching relative to tree-based methods and under- scores the promise of machine learning for vaccine guidance(Lou et al. 2024). In parallel, genomic foundation models (c.g., DNABERT (Zhou et al., (2024). NT (Dalla Torre et al., 2024). GROVER (Sanabria et al. 2024). HyenaDNA (Nguyen et al. (202)) and protein foundation models (e.g., ESM(Rives al. (021). ProtGPT2(Ferruz et al. (2022), together with influenza-focuse protein predictors (Hie et al. 202 Ma et al.) 2024 Ito et al. 2025), offer another powerful approach for
antigenic sequence representation.
However, viral evolution is shaped by coordinated interactions across the entire genome (Cobey Gouma et al. 2020). These include RNA-RNA interactions and co- pa-paagkiinggg (Bolte et al, Yang et al. 2024), constraints segment reassortment (Holmes) (007), and co-adaptation
2024
2019
between polymerase segments (PB (PB///P)) and antigenic proteins HA and NA (Vigeveno et al.
2024
Noda 2020) Models that ignore this structural context-suud site-wise predictors like
CCorreponniig authors.

---

Published as a conference paper at ICLR 2026
-oatl-mefrggettt biological signals, limiting both generalization and interpretability. General- purpose foundation models, trained on heterogeneous multi-species genomes, tend to capture lo- cal sequence patterns but often fail to model these genome genome-wide, higher-order constraints, resulting in degraded performance compared to species-aware and structurally aligned pretraining (Karollus) al. 2024 Benegas al. 2025).
Moreover, although protein-level forecasting can capture aspects of antigenic evolution, key deter- minants of viral fitness-including synonymous mutations, noncoding regulatory elements, RNA secondary structures, packaging signals, codon-usage-mmddiate host adaptation, and polymerase compatibility- are completely invisible protein-only models. Extensive experimental evidence highlights the importance of these nucleotide-lvee mechanisms (Canale et al.) [2018] Kryazhimskiy
al. 2008 Fujii et al. 2005 Liu et al. 2025; Gu af., 2019), and recent benchmarks show that DNA-based models outperform protein-based approaches on related predictive tasks (Boshar et al., 2024). Together, these observations motivate an -eauenna--peccfic,, shole-genome, nucleotide-level language model for accurate antigen sequence prediction. The compact influenza genome (~~13 kilo-nucleotide) further makes such a specialized model feasible.
Here, we introduce AntigenLM,, generative DNA language model that explicitly preserves segment order and functional-umi integrity during pretraining. The autoregressive model is then finetuned to forecast the antigen sequences of upcoming dominant strains using serial HA-NA sequences col- lected in fixed temporal windows, which implicitly encode evolutionary trajectories. By enforcing segment orientation and antigen boundaries, AntigenLM learns representations that integrate local sequence dependencies with global genomic structure, enabling generalization across clades, sub- types, and geographic regions.
We evaluate AntigenI.M across diverse influenza forecasting tasks and benchmark against general- purpose foundation models, the current WHO selection method, and sfate-of-the-arg evolutionary predictors such as beth-l. We further conduct ablation studies using antigen-only, truncated truncated-genome, and regment-shuflke variants to isolate the contributions of whole-genome context and functional- unit preservation.
Our contributions are threefold: 1. Functional-unit aware pretraining: We introduce DNA lan- guage model that enforces the integrity and correct permutation of functional units during pretrain- ing and demonstrate its advantage through controlled ablations. 2. Improved influenza forecasting: AntigenI. outperforms state-of-the-an site-based evolutionary models, achieving lower amino acid mismatch in antigenic sequence prediction. 3. Generalizable framework: We provide a blueprint for incorporating biological structure into generative models, with implications for predictive genomies, vaccine design, and modeling rapidly evolving pathogens.
2 RELATED WORK
2.1 CLASSICAL EVOLUTIONARY FORECASTING METHODS
Classical phylogenetic models form the foundation of current WHO vaccine recommendations. Ap- proaches such as Local Branching Index (LBI)-based tree dynamics (Neher al., 2014; (2066) and commaglutinationnnnibbbe inhibition deri-derived antigenic predictors (Du et al. (017)) rank circulating lin- eages and estimate their likelihood of future dominance. While effective at integrating serological and sequence data, these methods assume relatively homogeneous site dynamics and do not explic- itly account for higher-order constraints or genome-wide coordinated evolution.
2.2 DEEP LEARNING-BASEE EVOLUTIONARY MODELS
Recent deep-learning approaches model influenza evolution directly from viral sequences. beth-/ (Lou al. 2024) infers site-wise mutation fitness from genomic sequences and population seropos- itivity, enabling quantitative prediction of clade-specific fitness landscapes. Despite strong perfor- mance, it treats mutations as independent events, limiting its ability to capture coordinated changes spanning HA, NA, and other segments. EVE (Frazer et al. 2021), a deep generative model trained on HINI HA sequences, predicts clade-level mutation trajectories (Mehrotra et al., 2025) but focuses solely single antigen and cannot generalize to other subtype antigens smoothly.
2

---

Published as a conference paper at ICLR 2026
2.3 GENOMIC LANGUAGE MODELS
Nucleotide-leve language models such as DNABERT @Ji al. 2021 NT (Dalla-Toree et al. 2024). and HyenaDNA (Nguyen al. (023) learn contextual representations of DNA sequences. Advances in long-context modeling e.g., genome-scale Transformers (Fishman et al. 2025) and extended- context Hyena architectures-allee alllow processing of sequences up megabase length, sufficient for vi- ral genomes. However, these general-purpose models are trained on heterogeneous, multi-organism genomic corpora with vastly different genome sizes and architectures, making difficult to retain organism- -pennii- structural organization or genome-wide higher-order constraints.
2.4 GENERAL-PURPOST AND INFLUENZA SPECIFIC PROTEIN LANGUAGE MODELS
Protein language models, including general models such as ESM
(Rives et al. 2021 Lin et al. 2023)
and influenza-focused models (Hie et al. 2021 Ma al. 2024 Ito al. 2025 provide strong anti- genic sequence representations and have demonstrated utility in antigenic characterization. How- ever, protein-only models cannot capture key nucleotide- evolutionary mechanisms such as synonymous substitutions, codon-usage adaptation, or non-coding regulatory elements- that influ- ence viral fitness without altering protein sequences. Moreover, most protein LMs are not autore- gressive and cannot generate full-length antigen forecasts except ProtGPT2 (Ferruz et al. 2022).
METHOD
3.1
MODEL OVERVIEW
We present Antigen a Transformer-based( Vaswani et al. 2017) framework tailored for modeling
Vasaar
influenza A viral genomes (Figure 1B). The model is derived from the GPT--2RRadford et al.) 2019)
architecture but redesigned to address the unique challenges of biological sequence learning:
extremely long input contexts (up 13k nucleotides per genome Van den Hoecke al. 2015)), (ii)
multi-task learning objectives that combine generative and discriminative tasks, and (iil) the need to al. 2015).
capture dependencies across distinct funetional gene segments(Neverov
The backbone is a decoder-only Transformer with 6 layers. 384 hidden dimensions, and attention heads. Each block uses feed-forward sublayer with an inner dimension of 1, 1,536 (i.e., 4x the model dimension) and GELU activations, following the GPT-2 implementation. Although compact com- pared to large NLP models, the architecture is extended with a 13, 13.000-position embedding range, allowing full-genome modeling without truncation and avoiding hand-crafted tokenizers like BPE and k-mer(Li et al., 2024). This strikes balance between coverage and computational efficiency, enabling training tens of thousands of viral genomes using standard multi-Gi GPU setups.
On top the backbone, we implement a dual-head design for multi-task learning:
1. Language Modeling (LM) Head: tied to the embedding matrix, predicts the next nucleotide token in an autoregressive manner, capturing evolutionary dynamics.
2. Classification Head: projects hidden states at sentinel positions into subtype logits, sup- porting supervised sequence-level discrimination.
The model leverages shared pretraining, optimizing each task independently during fine-tuning. This enables the model to capture global genome-wide context through autoregressive modeling while benefiting from supervised subtype classification, resulting in richer generative representations and improved subtype identification performance.
3.2 FUNCTIONAL- UNIT ENCODING
AntigenLM employs a two-stage functional-unit encoding strategy to preserve both gedome-wide context and segment-level structure. During pretraining, all eight influenza A segments (PB2, PBI, PA, HA, NP, NA. MP, NS) are concatenated into a single continuous full-genome sequence (Hoffmann et al. 2000). maintaining fixed order and approximate positional alignment. This enables the Transformer to capture long-range co-evolutionary dependencies, such as compensatory mutations between HA and NA (Koel al. 2013). without artificial markers truncation and
3

---

Published as a conference paper at ICLR 2026
avoids heavy structural tagging (Huddleston al. 2020 During fine-tuning, sentinel tokens (e.g., <HA>, <NA>) explicitly delimit functional regions (Johnson et al.) 2017). guiding attention for subtype classification and constraining autoregressive decoding within each antigen to prevent cross- segment continuation (Sennrich et al. 2015). This combination scalable unsupervised pretraining and task-specific structural supervision produces biologically faithful, task-adaptive representations.
3.3 COMPLEXITY AND EFFICIENCY
Scalability is crucial for genome-scale modeling, AntigenLM is therefore designed to be both com- pact and long-context. The backbone uses only layers with 384 hidden dimensions and atten- tion heads, but supports sequences of up to 13,000 positions, enabling full-genome modeling with- out truncation. Both generative and discriminative tasks share this single Transformer backbone, which reduces memory usage and promotes efficient transfer of representations(AVe In practice, pretraining on more than 54,000 complete influenza A genomes and subsequent task- specific fine-tuning are feasible on standard multi setups (see Section 4.5tion training details). These choices make AntigenLM suitable for large-scale influenza surveillance data while remaining computationally tractable.
al. (021).
3.4 FINE-TUNING FOR FORECASTING AND CLASSIFICATION
We fine-tune A downstream tasks using the same GPT-2 style backbone and dif- ferent heads. For generative forecasting, we train the language modeling head on prompts that concatenate three historical HA/NA blocks followed by one future block. Each example has the structure
block block(2) (2) block block(
past
future
where
(tock <subtype> <HA> HA <NA> (A <sep>
(t) are HA/NA nucleotide sequences from three past time points within the same season. The future block (lock(*) follows the same pattern but uses HA, NA from the future time point to be forecast and simply omits the leading <subtype> token. We optimize the standard causal language modeling loss over the full token sequence (excluding padding), and inference time feed only the three historical blocks and autoregressively generate the future block starting from the target <HA> token until <sep> is produced or a maximum length is reached.
for 1,2,3. Here HA (4) and (+
For subtype classification, use the same backbone with the classification head and simpler input that contains a single HA/NA pair and a separator,
<HA> HA <NA> NA <sep>.
The sequence is encoded by the Transformer, and we take the hidden state at the final token posi- tion compact representation of the HA/NA pair. This representation is passed through linear layer to produce subtype logits and trained with cross-entropy loss. In practice we fine-tune two task-specific models one generative and one discriminative- both initialized from the same pre- trained AntigenLM backbone, and we also support joint optimization of both objectives via a simple weighted sum of the language modeling and classification losses.
EXPERIMENTS
4.1 DATASETS
We assembled a comprehensive corpus of influenza A virus genomes from GISAID (Shu & Mc- Cauley] 2017), including the two major subtypes A/H3N2 and A/HINI as well as 10 minor sub- types (e.g.. H5N1, H7N9). For both pretraining and fine-tuning, we used sequences collected before February 2022 and applied stringent quality control by removing duplicates, excluding incomplete genomes (fewer than eight segments), and filtering out sequences with 90% mean segment erage relative to the corresponding subtype reference. After filtering, the dataset comprised 32,758 H3N2, 20,680 HIN1, and 1,074 minor-subtype genomes, totaling ~600 million nucleotides across
4

---

Published as a conference paper at ICLR 2026
A
B
AntigenLM
(GPT-atyle Transformed) N.
Pretrsining (shared backbone)
Finetuning tasks
Classification Head
custtypa
<HA
<NAA
<MMl
-ceepo
Strama collected
fiom the exmss region)
Figure Data Distribution and A Architecture. (A) Global distribution of influenza A virus sequences used for pretraining, fine-tuning, and testing. Circle size reflects sample count; pic sectors show subtype composition. Dark circles represent pretraining data, light circles fine-tuning data, and red ticks mark test regions. (B) AntigenLM architecture. Schematic illustration of the pretraining and finetuning phases. The model utilizes a GPT-style Transformer as shared backbone (6 layers, hidden dimension of 384, and 6 attention heads). Top (Pretraining): The backbone is pretrained on nucleotide sequences spanning all eight influenza gene segments (PB2 to NS). Bottom (Finetuning): The model is fine-tuned for distinet tasks: viral evolution prediction (left), which uses an LM head to predict sequences for month t + based on historical strains (months 2 to from the same region; and subtype classification (right), which employs classification head to identify the virus subtype based on HA and NA segments.
54,512 genomes (Figure 1A). For each virus, the eight segments were simply concatenated in fixed order (from largest to smallest segment) to form a full-genome sequence without Multiple Sequence Alignment gaps, and then tokenized (Appendix A.2
Post-2022 sequences underwent the same quality control procedures and were reserved as retro- spective test sets, stratified by subtype and collection month, to enable leakage-free evaluation that mimics real-world forecasting. For geographic generalization, we fine- oun-tuned models on pre-2022 sequences from Europe and Asia and assessed performance post-2022 genomes from Japan (in- distribution) and the United States (out-of-distributtion)
4.2 PRETRAINING SETUP
To dissect the contributions of cross segmeet dependencies, nucleotide-level variation, genome com- pleteness, and segment-order alignment, conducted a controlled ablation study using five pre- training variants, each differing in sequence and segment organization; a schematic overview of these configurations is shown Figure 2:
5

---

Published as a conference paper at ICLR 2026
Full-genome
PR2
P81
PA HA
12 NA MP
NS
PB1
PA
14A
PB2 PB1
Begment-wise
Incomplete- genome
Antigen only (nucleolide)
Antigen only (protem))
Figure 2: Pretraining input of A and ablation variants. The standard Full-genome strategy (top) concatenates the eight segments from same isolate in a fixed order. The Abla- tion variants (bottom) explore alternative imput formatting: Segment-wise uses independent per- segment sequences; Incomplete-genome is generated by randomly cropping fixed-length windows from long concatenations, resulting in mixed segments; and Antigen-only models restrict input solely to the HA and NA segments using nucleotide and protein (amino acid) sequences, respec- tively. All the configurations except Antigen-only (protein) utilize nucleotide sequences.
Full-genome pretraining: All eight gene segments (PB2, PB1, PA, HA, NP, NA, NS) are concatenated into single nucleotide sequence (up to ~~3,000 tokens, one nucleotide per token) and trained with an autoregressive objective. This preserves segment order- ing and cross-segment dependencies, allowing the model to learn co-evolutionary patterns across the genome.
2. Incomplete-genome pretraining: Concatenated genomes are randomly cropped into 12k- token windows while keeping the overall corpus size constant. This disrupts some gene boundaries, allowing us quantify the effect of losing full functional context.
3. Segment- wise pretraining: Each training example contains single randomly selected gene segment. Corpus size is matched to the full-genome variant, but all cross-segment context is removed, isolating segment learning.
4. Antigen-only (nucleotide) pretraining: Only the HA and NA segments are included. HA and NA nucleotide sequences are concatenated and trained with the same autoregressive objective. This restricts the model to the two major antigenic determinants while maintain- ing co-evolution signals between them.
5. Antigen-only (protein) prefraining:
HA and NA coding sequences translated into amino acid sequences, concatenated, and tokenized at the residue level. By training directly HA-NA proteins, this variant focuses solely antigenic variation in protein space, isolating effects attributable to synonymous nucleotide changes.
All variants the same backbone, optimizer, learning-rate schedule, number of updates, and ef- fective token budget; random seeds are fixed for data splits, initialization, and shuffling to ensure controlled comparison. We optimize the standard causal language modeling loss
T-1
LCLM =
logp( Ig())
where T is the sequence length.
4.3 FINE-TUNING AND EVALUATION TASKS
We designed four complementary downstream tasks to evaluate Antigen ability to learn biolog- ically meaningful representations and fine-tuned the model accordingly. These tasks assess short- and long-term forecasting, geographic and subtype generalization, and representation quality for classification.
(1) Next-Month HA/NA Sequence Forecasting. To evaluate short-term predictive power, Anti- genLM was fine-tuned on HA and NA sequences from three consecutive months up month /

---

Published as a conference paper at ICLR 2026
and used predict sequences for month ++1 within a given region. Sentinel tokens <HA>and <NA>marked input segments, and <sep>>epparated sequences from different strains. Outputs were verified for the presence of <HA>and <NA>, and predictions with length deviations ex- ceeding 150 nucleotides were flagged failure. Successful outputs were evaluated for: (i) Mean AA Mismatch- average amino acid difference between predicted sequences and the nearest observed strain; (ii) Epitope-Specific Mismatch- Mismatch-avverage average mismatch restricted to known epitope sites; and (iii Token-Level Perplexity. Baseline and ablation models were trained and evaluated in parallel.
(2) Next-Season HA/NA Sequence Forecasting. For Longer-horizon forecasting, AntigenLM pre- dicted sequences for season T+1 based on serial sequences from the previous season T' (November year T to February year 1+) Baselines, including beth-1, were trained and evaluated the same pre-2022 training and post-2022 evaluation splits.
(3) Geographic and Cross-Subtype Generalization. assess out-of-distribution performance, all U.S. sequences were held out during fine-tuning and used exclusively for evaluation. Transfer- ability to the minor subtype H7N9 (< 5% of pretraining corpus, < 1% of fine-tuning data) was also tested, demonstrating AntigenLM'ss ability generalize to rare subtypes. As no baseline models forecast minor-subtype sequences, comparisons were made against AntigenL.M pretraining variants.
(4) Subtype Classification, Representation quality was further evaluated by training lightweight classification head on sentinel token embeddings of HA+NA sequences. Performance was measured using micro-averaged Fl scores.
4.4 BASELINE COMPARISON
We benchmarked AntigenI.M against three classes of baselines:
(1) Evolutionary Model beth-1. beth-/ is state-of-the-art site-based dynamic model that estimates mutation fitness and projects site-wise prevalence forward in time. It serves as our primary biological baseline. EVE was excluded it produces clade-level rather than full-sequence forecasts.
(2) Tree-Based Predictor LBI. The Local Branching Index (LBI) operates on phylogenetic trees derived from HA/NA sequences and has historically informed WHO vaccine strain recom- mendations, providing a meaningful lower bound for predictive performance.
(3) General-Purpose DNA//rotein Language Models. Autoregressive models- models-HyenaDNN, and ProtGPT2- were used benchmark AntigenLM against general-purpon purpose nucleotide and pro- tein language models. These models are pretrained on large, heterogeneous genomic or proteomic corpora spanning multiple species and genome sizes, allowing us to assess the added value of influenza-specific biologically informed pretraining.
All language models used identical training/evaluation datasets and matched parameter counts where possible. beth-/ and LBI were run using publicly available implementations with recommended settings.
4.5 IMPLEMENTATION
AntigenLM implemented in PyTorch using the Hugging Face transformers library. Unless otherwise stated, the efficiency statistics in this section refer to the pretraining phase.
We pretrain on NVIDIA A800 GPUs GB each) with per-device micro-batch size of com- plete genome and gradient accumulation over 4 micro-steps, yielding effective global batch size of 32 genomes per optimizer update and a maximum context length of 13,000 tokens. On this setup, the pretraining run processes roughly 7 x 10 tokens per second across all 8 GPUs, and training the full 54k -ek-genome corpus requires a total compute budget on the order of 10 floating point operations, while remaining well within the 80 GB memory budget of each device.
We optimize all models with Adam W (peak learning rate 1 x 10-4, linear warmup over the first 5% of updates followed by cosine decay, dropout 0, gradient clipping at 1.0). Task-specific fine-tuning
7

---

Published as a conference paper at ICLR 2026
for forecasting and subtype classification reuses the same pretrained backbone and is substantially cheaper than pretraining; we therefore do not report separate efficiency numbers for these stages. Code, preprocessing scripts, and trained checkpoints will be released upon acceptance.
RESULTS AND ANALYSIS
5.1 NEXT-MONTH HA/NA SEQUENCE FORECASTING
We first evaluated A on next-month HA and NA sequence forecasting assess its ability to capture short-term evolutionary dynamics (Figure 3A). AntigenI.M was fine-tuned on pre-2022 data to predict HA/NA sequences for month t+1 using sequences from the preceding three months (t-2, 1-1, f), and evaluated on post-2022 observations. The model produced smooth month-to-month forecasts, with mean amino-acid (AA) mismatches of 3-4 in HA (< 1% of 566 AAs) and 1-2 in NA (< 0.5% of 469 AAs). Mismatches within epitope regions were similarly rare and approached zero for NA.
Comparisons with reneral-purpon language models and ablation controls highlighted the advan- tages of biologically informed pretraining. (i) HyenaDNA and ProtGPT2 outputs were mostly valid HA/NA sequences with large length deviation and sentinel token missed misplaced, for which mismatch cannot be calculated; (ii) The eecomplete-genonee baseline frequently failed to produce valid sequences and exhibited substantially higher AA mismatch rates when it did. (iii) The segment- wise and antigen-only models performed comparably to AntigenLM but showed mild degradation. Together, these results indicate that preserving the integrity of individual segments (at least HA and NA) is essential for generating valid forecasts, and that incorporating whole-genome, cross-segment context further improves predictive accuracy.
We further quantified model uncertainty using token-level perplexity for all ablation models, ex- cept the Antigen-only (protein) variant due to differences in tokenization. AntigenI.M achieved a perplexity of 1.26, substantially lower than Incomplete-gente genome (3.55), Segment-wise (4.42), and Antigen-only (nucleotide) (4.56), indicating superior modeling of the conditional sequence distri- bution. These results demonstrate that (i) non-HA/NA internal segments contribute meaningful pre- dictive signals (controlled by Antigen-only); (ii) maintaining functional-umt structure improves LM learning (controlled by Segment-wiist (iii) sequence integrity is essential for accurate forecasting (controlled by Incomplete genome). -emmplee--geomm))..............)
5.2 FORECASTING NEXT-SEASON CIRCULATING STRAINS
We evaluated AntigenLM on next-season dominant strain forecasting, directly relevant to vaccine strain selection. The model was fine-tuned using pre-2022 sequences from Europe and Asia and tested on post-2022 sequences from Japan, predicting circulating strains for season T+1 based on sequences from three consecutive months of season T.
Figure 3B shows results for HIN1 and H3N2, considering both full-length genes and epitope re- gions. AntigenLM consistently achieved the lowest amino acid mismatches, with the largest gains observed in HINI-HA and H3N2-NA, reducing mismatches by over 70% relative to WHO vac- cine recommendations (labeled as "current system" in Figure 3) and by 50% compared to the site- based model beth-1. J. Site-based models exhibited higher variance and often overpredicted single-site sweeps, reflecting the limitations of their independence assumptions. Improvements were consis- tent across both full-length and epitope-restricted regions, except for HINI-NA, which was already below 1, leaving little room for improvement. These results underscore the overall improvement of AntigenLM in forecasting accuracy and highlight its potential utility in guiding vaccine design.
Forecasts were also more seasonally stable, avoiding abrupt clade switches observed in tree-based methods. Across 100 tests (50 per subtype) in Japan, H3N2 strains transitioned to new clades in the next season while HINI largely remained within existing clades. AntigenLM accurately predicted sequences of emerging H3N2 clades, consistent with known patterns of antigenic drift. These results demonstrate that AntigenLM provides biologically coherent, actionable predictions, reducing potential mismatches between vaccines and future circulating strains.
8

---

Published as a conference paper at ICLR 2026
Netiod Degi antonia
500
H3W2
200
100
200
100
perare
Aaverraee 806 mimmarc
17.5
15.0
Butty 16 (93)
12.5
20-
10.0
7.5
5.0
0.0
No Rutleggg
HA Rachengh
NA
1018
200
330
330
HEND
HINI
100
Aveeage eming wold mismmtth
35.0
Buth- p4t Say
25
15
12.5
30.0
7.5
5.0
2.5
0.0
HR but
hi Ho Mt teggegg
Figure 3: A Achieves the Lowest Amino-Acid Mismatch Across All Forecasting Tasks. (A) Next-month prediction: A (full-genome pretraining) compared with abla- tion controls. (B) Next-scason forecasting on post-2022 Japan data (with pre-2022 data included in fine-tuning): AntigenI.M compared with baseline models. (C) Cross-
generalization in next-season forecasting: AntigenL.M (full-genome pretraining) versus ablation controls for H7N9 prediction. (D) Geographic generalization: A evaluated on U.S. data unseen during fine- tuning, compared with baseline models. Asterisks indicate statistical significance (I-test) between AntigenLM and beth-1: * p < 10-3, Error bars show standard deviations.
5.3 GENERALIZATION ACROSS SUBTYPES AND GEOGRAPHIES
Current evolutionary models typically require large amounts of training data and often struggle to forecast emerging strain sequences for minor subtypes or regions with limited historical sampling. Consequently, their utility for predicting the evolution or pandemic potential of emerging strains in under-sampled populations is limited. To assess AntigenLM's ability to generalize beyond its training distribution, we performed two complementary out-of- disstributiion (OOD) experiments.
Cross-Subtype Transfer. evaluated AntigenLM on the minor influenza A subtype H7N9, which represents only 4.68% of the pretraining corpus and 0.3% (48 sequences) of the fine-tuning set. Despite this extremely limited representation, AntigenI.M generated next-season predictions for H7N9 with test counts and mismatch rates comparable to those for the major subtypes HIN1 and H3N2, Interestingly, all ablation modeels-including the ancomplete-genome variant--prrormmed similarly on H7N9 as on the major subtypes, likely due to the higher sequence conservation of H7N9 (Figure 3C). Overall, AntigenLM achieves effective cross-subtype generalization, a capability that remains challenging for current phylogenetic approaches.
Geographic Generalization. To evaluate geographic robustness, we held out the U.S. as an unseen region and fine-tuned AntigenLM only on sequences from Europe and Asia. The model maintained substantially lower average AA mismatches than beth-/ / for HA in both HINI and H3N2, although improvements for NA were not significant (Figure 3D). Analysis of 100 test cases revealed that all involved transitions into clades absent from fine-tuning. AntigenLM correctly predicted clade changes in 90 of these cases (all 50 HINI and 40/50 H3N2), though the lack of exposure slightly increased overall mismatch, particularly in NA. Epitope mismatches in NA remained minimal (0-1 amino acid), consistent with in-distribution performance and sufficient for vaccine design. These sults indicate that A learns global evolutionary constraints rather than memorizing region- specific mutations, enabling reliable predictions in historically under-sampled populations.
9

---

Published as a conference paper at ICLR 2026
Figure 4: Subtype classification performance of AntigenLM and ablation models. Row-normalized confusion matrices for subtype classification. A (left) and Antigen-only (nucleotide) (right) show near-perfect diagonal dominance, indicating highly accurate classification, whereas crooppeteeggnonee and Segment-wise models (middle) show increased off-diagonal misclassifica- tions, particularly for rare subtypes.
Together, these results demonstrate that AntigenI.M does merely fit dominant subtypes but learns transferable, biologically meaningful representations that generalize across subtypes and geogra- phies-a capability current baselines struggle achieve. Such robustness critical for real-world applications where data availability is uneven and novel strains may emerge outside well-sampled populations.
5.3.1 SUBTYPE CLASSIFICATION
To further evaluate AntigenI.M's multitask capabilities, we tested its performance on subtype clas- sification as separate downstream task.
Subtype classification is relatively straightforward due to distinct sequence differences among sub- types, and accordingly, AntigenL.M and all its variants performed well. The Antigen-only models (nucleotide and protein) achieved 100% accuracy, expected, since subtypes are solely determined by antigen sequences. Notably, Full- gellme AntigenLM also performed exceptionally, achieving a micro-averaged F1 score of 99.81%, with only single H5N6 strain misclassified H5N1 among the 530 test strains. All other minor subtypes were classified accurately despite limited training data, demonstrating that AntigenI.M's embedding space robustly separates subtypes. In contrast, the Incomplete-gente senpmete-geeomeeee and Segment-wise variants showed substantially more subtype confusion (Figure 4
These results indicate that AntigenLM learns a well-structured, subtype-aware latent space, sup- porting applications such automated influenza surveillance and real-time strain tracking without retraining.
6 DISCUSSION AND CONCLUSION
AntigenLM integrates functional-urd preservation into language modeling, substantially improv- ing influenza forecasting. It consistently outperforms general-purposs language models, evolution- ary methods, and ablation controls across next-month, next-season, and out-of-distribution tasks, achieving lower amino acid mismatches and biologically coherent predictions that align with sub- type transitions. By capturing higher-order dependencies across segments, AntigenLM generalizes across subtypes and geographies, enabling accurate forecasts even in under-sampled populations. Challenges remain for real-time deployment, as forecasts are inherently probabilistic and should complement expert-driven decisions (Ampofo et al., 2011). Overall, AntigenLM demonstrates that integration of biological structure into genomic language models can provide more accurate, inter- pretable, and actionable insights for viral evolution and public health applications.
10

---

Published as a conference paper at ICLR 2026
DATA AND CODE AVAILABILITY
All influenza genome sequences used in this study are publicly available from the GISAID Influenza Virus Resource. The code, data and models are available AntigenI.M
USE OF LARGE LANGUAGE MODELS
We used large language models only for minor language polishing. No LLMs were used for coding, data processing, experimental design, analysis, and all reported results produced by our code and verified from logged runs.
ACKNOWLEDGMENT
This work was supported by National Key Research and Development Program of China (2044FC3405704)) the National Science Foundation China (32371537).
REFERENCES
William K. Ampofo. Norman Baylor, Sarah Cobey, Nancy J. Cox, Sharon Daves, Steven Edwards, Neil Ferguson, Gary Grohmann, Alan Hay, Jacqueline Katz, Kornnika Kullabutr, Linda Lam- bert, Roland Levandowski, A. C. Mishra, Arnold Monto, Marilda Siqueira, Masato Tashiro, An- thony I.. Waddell, Niteen Wairagkar, John Wood, Maria Zambon, and Wenqing Zhang. Improv- ing influenza vaccine virus selectionreport of who informal consultation held at who head- quarters, geneva, switzerland, 14-16 june 2010. Influenza and Other Respiratory Viruses, (2):142-152, August 2011. ISSN 1750-2659 2659. doi: 10.1 10.1111/..150-2633 2057750002699.0111 2650,2001.00277.. URL http: doi. /11..///.... 11111/1111... 1.111.1-7755555 201 00277.x
Ziga Avsec, Vikram Agarwal, Daniel Visentin, and et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18:1196-1203, 2021. doi:
10.1008/411922-0---115224
Gonzalo Benegas, Chengzhong Ye, Carlos Albors, Jianan Canal Li, and Yun S. Song. Genomic language models: opportunities and challenges. Trends in Genetics, 41(4):286-302, April 2025.
ISSN 0168-9525. doi: 10. tig.00/..i...224..111 1..013. URL http::///x doi 1ttp://xx...1.rrr/10111011111111
S.tiig20224. 11.0013
Hardin Bolte, Miruna E. Rosu, Elena Hagelauer, Adolfo Garefa-Sastre, and Martin Schwemmle. Packaging of the influenza virus genome is governed by plastic network of maa and satoooprotei--mmdiatee interactions. Journal of Virology. 93(4), February 2019. ISSN 1098-5514. doi: 10. 10.1128/jvi00161111111 URL http: doi org. 10. 1111///2vv 01861-12
Sam Boshar, Evan Trop, Bernardo P de Almeida, Liviu Copoiu, and Thomas Pierrot. Are genomic language models all you need? exploring genomic language models on protein downstream tasks. Bioinformatics, 40(9), August 2024. ISSN 1367-4811. doi: 10. httg//iiiifrrmatic/hheeess /atcee/tbaee299
URL http: doi wor. 0ttt:::///xx.dll.orr/00001991990
Magda Buchole, Mark G. O'Doherty, and Declan T. Bradley. Influenza vaccine effectiveness during
2023/2024 season: test-negative case-control study among emergency hospital admissions with respiratory conditions in northern ireland. Influenza and Other Respiratory Viruses, 19(9), September 2025. ISSN 1750-2659. doi: firy URL http: //o:://xxxdd doi .oi.org//
10.1111/irv 70149
Aneth S. Canale, Sergey V. Venev, Troy W. Whitfield, Daniel R. Caffrey, Wayne A. Marasco, Celia A. Schiffer, Timothy F. Kowalik, Jeffrey D. Jensen, Robert W. Finberg, Konstantin B. Zel- dovich, Jennifer P. Wang, and Daniel N.A. Bolon. Synonymous mutations at the beginning of the influenza virus hemagglutinin gene impact experimental fitness. Journal of Molecular Bi- ology, 4300 April 2018. ISSN 0022-2834 2836. doi: 10. 10.1016/j.imb.20188 0200009. URL http:://xx..mm doi org. 101.101165 jmb. 201 02.009

---

Published as a conference paper at ICLR 2026
Sarah Cobey. Vaccination against rapidly evolving pathogens and the entanglements of mem- ory. Nature Immunology, 25(11):2015-202, October 2024. ISSN 2916. doi: 10.1038/ 3415500-024-01970-22 Oo 1038/ http:////.....0....1111000--0010
doi.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P. de Almeida, Has- san Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22(2):287 November 2024. ISSN 1548-7105 doi: 10. 3415922-02--22223-. /tt.://pppp.pcc.......o.......//
Xiangjun Du, Aaron A. King, Robert Woods, and Mercedes Pascual. Evolution-informedd fore- casting of seasonal influenza (h3n2). Science Translational Medicine, rean33e 2017. doi: 10. 2200//eitrammmmmm..mm2222222
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature Communications, 13(1), July 2022. ISSN 2041-1723. doi: 10. 1031/ s41467-022-32007-7 URL BRLhttp://dee http:://.xx.dd..oggg/////o 1030 34170008/84467-811 000000022300000
Veniamin Fishman, Yuri Kuratov, Aleksei Shmelev, Maxim Petrov, Dmitry Penzar, Denis Shepelin, Nikolay Chekanov, Olga Kardymon, and Mikhail Burtsev. Gena-Im: family of open-source foundational dna language models for long sequences. Nucleie Acids Research, 53(2). January 2025. ISSN 1362-4962. doi: 10. 10.10///arr/kae 1310. GRLhttpp://de doi org/ 10. 10.109977 1093/ nar/ har/gkae1310
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K. Min, Kelly Brock, Yarin Gal, and Debora S. Marks. Disease variant prediction with deep generative models of evo- lutionary data. Nature, 599(7883):91-955 October 2021. ISSN 1476-4687. doi: 10.1038/
941586-021-0404--8 URLhttp://dxx.o doi:://xxxddoooo.////ooo www.oo..........ooo wttp:r/////..//////.... 0115://555566-011--00000101
Ken Fujii, Yutaka Fujii, Takeshi Noda, Yukiko Muramoto, Tokiko Watanabe, Ayato Takada, Hideo Goto, Taisuke Horimoto, and Yoshihiro Kawaoka. Importance of both the coding and the segment-speciffi noncoding regions of the influenza virus segment for its efficient incor- poration into virions. Journal of Virology, 79(6):3766 3774, March 2005. ISSN 1098-5514. doi: 10. 6. 6.3766--0 07776-3774..0 2005
URL http: dai. ooo.ioorg/100. 1ttp://1.1111////00 1128/JVI. 79.
Sigrid Gouma, Elizabeth M. Anderson, and Scott Hensley. Challenges of making effective in- fluenza vaccines. Anmual Review of Virology, 7(1::955-512. September 2020. ISSN 2327-0578. 1146/
doi: 10. 10.1144/aamurexvvvologyyy -110:e---00000044--0 044746. URL http://dx dei. org.:.ooorr100 http:///xx.eci.org/1100
2nruree--iiroloog--0002222------
Haogao Gu, Rebecca L Y Fan, Di Wang, and Leo L M Poon. Dinucleotide evolutionary dynamics in influenza virus. Virus Evolution, 5(2), July 2019. ISSN 2057-1577. doi: 10.1093/ve/ve2038.
URL http: doi. www.:/oooo.oo.ooooo.o 1ww.:/099/vveeeeee vez038
James Hadfield, Colin Megill, Sidney M Bell, John Huddleston, Barney Potter, Charlton Callen- der, Pavel Sagulenko, Trevor Bedford, and Richard A Neher. Nextstrain: real-time tracking of
pathogen evolution. Bioinformatics, 34223)4121-4123, May 2018. ISSN 1367-4811. doi: 10.
1093/boiinoomatic///ty.4.., URL http: www.wwww.....ooooo org http://11..19///1/11e1 wtt.:///oiinooomarii..oooo bty407
Alvin X. Han, Simon P. J. de Jong, and Colin A. Russell. Co-evolution of immunity and sea- sonal influenza viruses. Nature Reviews Microbiology, 21(12):805-81,, August 2023. ISSN 1740-1534. doi: 10.10384411599-02-0009555
URL http::/dx.dci. www.d/oo.....ooooooooo
- -11579-233-0944-
Brian Hie, Ellen D. Zhong, Bonnie Berger, and Bryan Bryson. Learning the language of viral evolution and escape. Science, 371(6526)2284-28888 288, Jamuary 2021. ISSN 1095 9203. doi: 10. 1b33.333enc
|RL
org. 1126/sciinnceise abd3331
12

---

Published as a conference paper at ICLR 2026
E. Hoffimann, G. Neumann, Y. Kawaoka, G. Hobom, and R. G. Webster. A dna transfection system for generation of influenza a virus from eight plasmids. Proceedings of the National Academy of Sciences USA, (((1)):6108-6113, 2000. doi: 10.1073/pnas. 100133397.
Edward C Holmes. Viral evolution in the genomic age. PLoS Biology, 5(10))e278, October 2007. ISSN 1545-7885. doi: 10.137 /0t./oorall.bboo000002222. URL http://dx.do... org/
10. 10.1371/journa
pbio. 0050278
John Huddleston, John R Barnes, Thomas Rowe, Xiyan Xu, Rebecca Kondor, David E Went- worth, Lynne Whittaker, Burcu Ermetal, Rodney Stuart Daniels, John W McCauley, Seiichiro Fujisaki, Kazuya Nakamura, Noriko Kishida, Shinji Watanabe, Hideki Hasegawa, Ian Barr, Kanta Subbarao, Pierre Barrat-Charlais Richard Neher, and Trevor Bedford. Integrating genotypes and phenotypes improves long-term forecasts of seasonal influenza a/h3n2 evolu- tion. elife, 9, September 2020. ISSN 2050-084X. doi: 10.7554/elife.66066 URL http: www..doi.ooggooo.
1554/eLife... 60067
Jumpei Ito, Shusuke Kawakubo, Hiroaki Unno, Adam Strange, Spyros Lytras, Kaho Okumura, Alice Lilley, Ruth Harvey, Nicola Lewis, and Kei Sato. Integrative modeling of seasonal influenza evolution via ai-powered antigenic cartography. August 2025. doi: 10.1101/2025.00 .0..008...4000 00.......4..63......
URL http: //dx doi. org. 10. 1101111111111 00000//2225222 08. 04. 668423
Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V. Davuluri. DNABERT: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, (7115)2112-2122, 2021. doi: 10.1003/iionnfommaiicccaaaoooo
Melvin Johnson, Mike Schuster, Quoc V. Le, and al. Google's multilingual neural machine transla- tion system: Enabling zero-shot translation. In Transactions the Association for Computational Linguistics, volume 5, pp. 339-351, 2017.
doi: (111622/acLa.000066
Alexander Karollus, Johannes Hingerl, Dennis Gankin, Martin Grosshauser, Kristian Klemon, and Julien Gagneur. Species- aware dna language models capture regulatory elements and their evolu- tion. Genome Biology, 25(1), April 2024. ISSN 1474-760X. doi: 10. 0066//13550004-0000 3059
URL http://dx: doi wwoooog//// 10. 001866611135-00-002222222
Birgitte Koel, D. Burke, T. Bestebroer, and et al. Substitutions near the receptor binding site determine major antigenic change during influenza virus evolution. Science, 342(6161);976-979, 2013. doi: 10. 12.1126/cceecc... 1244730.
Sergey Kryazhimskiy, Georgüi A. Bazykin, and Jonathan Dushoff. Natural selection for nucleotide usage at synonymous and nonsynonymous sites in influenza a virus genes. Journal of Virology, 82(10):4938 4945, May 2008 ISSN 1098-5514. doi: 10.1 (0.1128/vi.024115-00. URL http:
7/dx www.:doo..rgg/0000 www.:///...1111//111 11w./ jvi...//.i.......... www.vi....1..111 02415-07,
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng, Yufei Huang, and Stan Z. Li. Vqdna: Unleashing the power vector quantization for multi-species genomic se- quence modeling, 2024. URL httpthttt///iiiii org/ abs/
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sereu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic- level protein structure with a language model. Science, 379(663): 130, March 2023. ISSN 1095-9203. doi: 10. 20.1166/ciience.dee5777. URL http: dei org 1126/ sci eccience.acc 2572
Zining Liu, Lei Zhang, Wenyu Zhang, Yuerong Lai, and Tao Deng. The 5'd segment-specific noncoding region of influenza a virus regulates both competitive multi-segment ma transcription and selective genome packaging during infection. Journal of Virology, 99(9), September 2025. ISSN (098-5514 14. doi: 10.1128/jv.00328-25 25. BRLhttp:// //dx dx www....w.....oooooo....o..o..... 1111g////......1...... 00328-25
Jingzhi Lou, Weiwen Liang, Lirong Cao, Inchi Hu, Shi Zhao, Zigui Chen, Renee Wan Yi Chan, Peter Pak Hang Cheung, Hong Zheng, Caiqi Liu, Qi Li, Mare Ka Chun Chong, Yexian Zhang,
13

---

Published as a conference paper at ICLR 2026
Eng-kiong Yeoh, Paul Kay- Kay-Sheung Chan, Benny Chung Ying Zee, Chris Ka Pun Mok, and Mag- gie Haitian Wang. Predictive evolutionary modelling for influenza virus by site-based dynamics of mutations. Nature Communications, 15(1), March 2024. ISSN 2041-1723, doi: 10. 10.1038/ s4 441467-0244461184 URLbttp: / dww.d........ wew.o..rrr//ooo org/10.................. 1038/ s4 4 691 -
Enhao Ma, Xuan Guo, Mingda Hu, Penghua Wang, Xin Wang. Congwen Wei, and Gong Cheng. A predictive language model for sars evolution. Signal Transduction and Targeted Therapy, (1), December 2024. ISSN 2059-3635. doi: 10. 1..0088/44132--22---0000
URL http://dkk
doi.org/10 1038/
Hanover C. Matz and Ali H. Ellebedy. Vaccination against influenza viruses annually: Renew- ing narrowing the protective shield? Journal of Experimental Medicine, 222(7), April 2025.
ISSN 1540-9538.
doi: 10. (084/fem.20241288. URL http:// doi wii.io............ www.:o.......ooooooo
jem. 20241283
Aarushi Mehrotra, Sarah Gurev, Noor Youssef, and Debora Susan Marks.
Forecasting hinl in- fluenza pandemic and seasonal evolution. In ICML 2025 Generative Al and Biology (GenBio) Workshop, 2025, BRL/htttss://ppeeee/// net www.mooo--ioioeeo h (onor
Richard A Neher, Colin A Russell, and Boris Shraiman. Predicting evolution from the shape of genealogical trees. elife, 3, November 2014. ISSN 2050-084X. doi: 10.7554/elife.03555 033668. URL
http:://xx doi org/10gg1110 7554/ fee 03566
Richard A. Neher, Trevor Bedford, Rodney S. Daniels. Colin A. Russell, and Boris I. Shraiman. Prediction, dynamics, and visualization of antigenic phenotypes of seasonal influenza viruses. Proceedings of the National Academy of Sciences, 113(12)E1701-E1170) 2016. doi: 10.1073/
pnas. 1525578113.
Alexey D. Neverov, Sergey Kryazhimskiy, Joshua B. Plotkin, and Georgii A. Bazykin. Coordinated evolution of influenza a surface proteins. PLoS Genetics, 11(8)e1005404, 2015. doi: 10.1371/
journal.pgen.1005.. 1005404.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris Ré. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution, 2023. URL Stttas/s//arxi...orgaaa///2/....
Takeshi Noda. Selective genome packaging mechanisms of influenza viruses. Cold Spring Harbor Perspectives in Mediciiee pp- a038497, August 2020. ISSN 2157-1422 doi: 10. /wtpessperpeect.....mmooo a038497. URL http: doi www.::.ddoo.rgg..o....oo 1101 1111111/scchhheeppcttttce 033897
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In Proceedings of the NeurlPS 2019 Conference, 2019. URL www.ltttps:/eao
60025533
Alexander Rives, Joshua Meier, Tom Sercu, and et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):20066299) 18, 2021. doi: 10. 10.1073/pnas.2011 6239118 18.
Melissa Sanabria, Jonas Hirsch, Pierre M. Joubert, and Anna R. Poetsch. Dna language model grover learns sequence context in the human genome. Nature Machine Intelligence, 6(8):911-923, July 2024. ISSN 2522-5839. doi: 10. 10.1038/44225-00240088222 001113//222255-022-0000000
URL http:/ dx doi org/
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, 2015.
07909
Yuelong Shu and John McCauley. Gisaid: Global initiative on sharing all influenza data from vi- sion to reality. Eurosurveillance, 22(13), March 2017. ISSN 1560-7917. doi: 10.2807/1560-991 7917. es 23..7.22.13.30499 URL http:: / doi, org www.::o.....ggg.oooo 2807 7917 1111117....... ESS. 2017.
22. 15.13.30499
14

---

Published as a conference paper at ICLR 2026
Silvie Van den Hoecke, Brian Verhack, Rony Ronsse, and et al. Analysis of the genetic diversity of influenza a viruses using next-generation dna sequencing. BMC Genomics, 16:79, 2015. doi: (0.1186/112864--15-12338
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL httpss//arxiv:
org/abs/1700 03762
René M Vigeveno, Alvin Han, Robert P de Vries, Edyth Parker, Karen de Haan, Sarah van Leeuwen, Katina D Hulme, Adam S Lauring, Aartjan J W te Velthuis, Geert-Jan Boons, Ron A Fouchier, Colin Russell, Menno de Jong, and Dirk Eggink. Long-term evolution of human seasonal influenza virus a(h3n2) is associated with increase in polymerase complex activity. Virus Evolution, 10(1), January 2024. ISSN 2057-1577 doi: 10.1093/ve/eeae0000. URL http://dx: doi wrg/ 1093/veee /vee/eeee
Rui Yang, Minglei Pan, Jiamei Guo, Yong Huang, Qiangfeng Cliff Zhang, Tao Deng, and Jianwei Wang. Mapping of the influenza a virus genome ra structure and interactions reveals essential elements of viral replication. Cell Reports, 43(3):113833, March 2024. ISSN 2211-1247. doi: 11101///..eerepp222222... 113833. URL http: //dx doi wtt.:/dww...ooo.oo/oooeo 10 011 6/ ce r 2024
113833
Zhihan Zhou, Yanrong Ji. Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genomes. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL
net.
forum??.0=ool..34EZZE)
Marta Luksza and Michael Lässig. A predictive fitness model for influenza. Nature, 507(7490): 57-61, February 2014. ISSN 1476-4687. doi: 10.1038/nature) 13087. URL http: doi.
15

---

Published as a conference paper at ICLR 2026
A DATA USED IN THIS STUDY
A.1 DATASET
Table I summarizes subtype-clade coverage and dataset sizes. Beyond the dominant subtypes (H3N2 and HINI), we intentionally include many low-resource subtypes-15 of 19 have fewer than 100 sequences-to expose pretraining to broader antigenic//enomic diversity and encourage generalization. These long-tail subtypes are then used mainly to stress-test transfer in the subtype- classification finetuning task, where we observe favorable performance despite limited samples. By contrast, our main forecasting experiments do not depend on rare subtypes: they are trained and evaluated on HINI and H3N2, with small extension to H7N9.
In addition, Table provides the temporal distribution of the collected strains, showing coverage from 2012 to 2022. Additionally, Table summarizes the geographical distribution of these pre- training sequences across key global regions such as the USA, China, and Australia, ensuring the breadth and diversity the data used to train the shared model backbone. Thus, forecasting conclu- sions not confounded by extreme class imbalance.
Table 1: Subtype-clade coverage and dataset counts
Subtype
Clade
Pretrain
Finetune
HINI
6B.1; 6B.1A; 6B.1A.1; 6B.1A.2: 6B.1A.3; 6B.1A.5;
20680
5565
6B.1A.5a; 6B.1A.5a.1;
1A.52; 1A.1..5,
6B.1A.5a.2a.1; 6B.1A.5b; 6B.
6B.1A.7; 6B.2
HIN2
6B.1; 6B.1A.1; 6B.1A.5a; 1A.1a.5.22; 6B.1A.5a.2al1;
31
74
6B.1A.6
H2N2
70
78
H3NI
3C.2alb.2..2..a111:3 3C.3a
H3N2
3C.2; 3C.2a; 3C.2al;
3C.2alb.1;
3C.2alb.la:
3C.2alb.
32758
8979
3C.2alb.2: 3C.2alb.2a:
3C.2alb.2a
3C.2alb.2a.la;
3C.2alb2a.la.I; 3C.2alb.2a.2;
3C.2alb.2a.22
3C.2alb.a.2a.1; SC.2alb.2a.a..
la;
3C.2alb:2a.2 2a. lb;
3C.2alb.2a.2a.;
3C.2alb.2a.aar
3C.2alb.2a.2..3
3C.2alb242a..ulI;
3C.2alb.a.2..b.
3..2alb.2a.26
3C.2alb.a.2e; 3C.2alb.2a.2; 3C.2alb.2b;
3C.2a2; 3C.2a3;
3C.2a4; 3C.3a; 3C.3al
H3N8
3C.2
HSNI
1.1.1: 1.1.2; 2.1.2; 2.1.32; 2.1.3.2;; 2.1.3.bb; 2.1.3.3; 2.2;
104
298
2.2.1; 2.2.1.; 2:2.1.2 2.3.2.1;; 2.3.2.1;; 2.3.2.le; 2.3.2.1g:
2.3.4;
2.3.4.3; 2.3.4.4b; 233.4b; 3;
HSN6
2.3.4.4: 2.3.4.4b; 2.3.4.4e; 2.3.4.4g 2.3.4.4h
25
37
H5N8
2.3.4.4b
H6NI
H7N2
H7N3
H7N4
H7N7
H7N9
793
1047
H9N2
37
78
HION3
HI0N8
total
54512
16169
16

---

Published as a conference paper at ICLR 2026
Table Temporal distribution of pretraining strains by subtype
Subtype
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
Before 2012
HION8
HINI
124
159
2370
4918
2952
1092
2122
786
353
295
5509
HIN2
2
20
H2N2
70
H3NI
H3N2
1472
3674
963
6723
4086
5470
2515
2018
1117
604
4116
H5N1
2
95
H5N6
2
4
6
4
H5N8
1
H6N1
H7N2
1
H7N3
H7N4
2
H7N7
H7N9
4
1
173
60
68
153
334
H9N2
3
11
Total
1596
3838
3340
11656
7051
6741
4712
2885
1629
1248
9816
Table 3: Geographical distribution of pretraining strains by subtype
Subtype
Australia
Brazil
China
France
Russia
Singapore
USA
United Kingdom
others
HION8
4
HINI
420
375
1044
638
621
564
9692
878
6448
HIN2
1
25
4
H2N2
2
8
27
2
24
H3N1
HBN2
2006
1136
1283
773
577
403
16532
1895
8153
HSNI
32
72
H5N6
23
HSN8
H6N1
H7N2
H7N3
2
H7N4
H7N7
H7N9
791
H9N2
35
2
Total
26277
14711
3217
2775
2427
1513
1412
1205
975
A.2
PRETRAINING DATA CURATION AND QC
We retain isolates that provide all eight segments (PB2, PB1, PA, HA, NP, NA, MP, NS).
CDS parsing and validity (per segment).
starts with ATG and ends with one of TAA/ CAAA/TA//TAE TGA;
length is multiple of 3 and contains no internal stop codons;
the CDS length falls within an empirically defined sanity range for that segment (outliers and clearly truncated records are removed).
Additional hygiene.
17

---

Published as a conference paper at ICLR 2026
remove records with excessive ambiguous characters;
ensure subtype annotations are consistent with HA/NA;
use a fixed concatenation order (PB2->PB1 -H--N--N--MP----A
A.3 FINETUNING DATA CURATION (HA/NA-ONLY)
Finetuning uses the same QC criteria as Appendix A.2 1 but applied only to HA and NA, An isolate is eligible if and only if its HA and NA CDS pass the validity checks; the other six segments are not required, may be missing incomplete, and are ignored even if present.
Sampling for inputs. For each region/subtype we select three inputs from the windows in Ap- (Sep-Oct; Nov-Dec; Jan-FFeb). Finetuning models consume only HA/NA tokens (other segments, any, are unused). The forecasting target the seasonal consensus defined in Ap- pendix
pendix B
B
B FINETUNING SAMPLE SELECTION: THREE-INPUT SETTING
For each target season (October + March of year T+1), we construct a three-input sample per region/subtype using the collection date (ignoring submission delays). Each input must pass the QC in Appendix A.2 and contain at least HA & NA.
Three windows (one sample per window):
Sep-Oct, year T. Captures early cross-regional introductions the onset of the influenza teason-the "seeds" that may later establish sustained transmission. Very early months (Jul-Aug) are noisier and less predictive, whereas sampling too late may miss early signals.
Nov-Dec, year T. Temperate regions enter winter transmission; lineage-specific growth-rate dif- ferences become clearer, and cross-regional flows associated with late-November and December holidays are reflected in the data.
Jan-Feb, year 7+1. As close as practicable to the WHO Northern Hemisphere vaccine compo- sition meeting, this window captures mid-season replacement events and newly fixed/key substi- tutions. Information after February is less actionable given manufacturing timelines.
Target output:
The prediction target for our forecasting experiments aligns with the field's routine answer strain selection: the nearest sequence. To meet this target, we construct a three-input sample (balancing lead time and recency). Our model demonstrated favorable performance against this standard target, and superior results when further tested against the consensus sequence target. This robust finding confirms the validity of our prediction framework.
C TRAINING DYNAMICS
Under matched optimization and token budgets, Fig- ure 5 reveals consistent differences across the three pretraining variants. Early burn-in: the full-genome model drops rapidly, segment-wise improves more slowly, and incomplete-ee genome decreases minimally. Mid/late regime: folineeffullleeee full-genome continues decreasing and converges the lowest asymptote; segment-wiss plateaus higher; caammlleee-genomee remains nearly flat, with visually larger bias and limited variance reduction.
Tsl Miganona Seprentirn
2.0
Tunte Gas
Intact cross-segment context acts as auxiliary supervi- sory signal, improving sample efficiency; isolating seg- ments removes these dependencies; cropping random windows mixes unrelated contexts and discards bound- aries, weakening the learning signal. All variants share backbone, optimizer, schedule, update count, and token budget; curves are smoothed identically for display and preserve the same ordering without smoothing.
Figure 5: Pretraining loss versus steps for the three variants.
18

---

Published as a conference paper at ICLR 2026
D SUBTYPE CLASSIFICATION
We finetune subtype classifier on top of each pretrained backhone with identical heads and op- timization. For qualitative analyses, we extract penultimate-layer embeddings and project them into 2D using the same I-SNE configuration across variants (shared perplexity, initialization, and perplexity-to- toay-ttt---pllle ratio). t-SNE is used strictly for visualization Figure 6 our main conclusions are based on quantitative forecasting and classification metrics reported in the main paper. For com- pleteness, Appendix Table 4 summarizes clustering metrics (Silhouette, ARI, NMI) computed on the same hidden layer used for the t-SNE plots: the full-genome model achieves the highest Silhou- ette score and ARI and ties with the segment wise variant on NMI, indicating more compact and label-consistent subtype clusters than the incomplete-genome and segment-wiss baselines. A radar plot of Fl scores across subtypes further emphasizes AntigenLM's superior performance (Figure
Table 4: Clustering metrics for subtype embeddings on the hidden layer used for the t-SNE plots.
Higher is better for all metrics.
Model
Silhouette t
ARI
NMI t
Full-genome
0.877
0.953
0.923
Incomplete-genome
0.761
0.938
0,890
Segment-wise
0.773
0.920
0.923
Antigen-only (nucleotide)
0.921
0.969
0.967
1-SNE on embeddings colored (N=530, perplexitym PCA=50)
t-SNE emoeddings colored by true (N=5330, perplexit==5..g PCA#50)
Pull-genome
10003
Incomplete-genomm
10.00
HON3
1OR2
1085
HON5
PDR2
10010
-10
4150
1552
-20
1350
-30
ambeddings colored 5000 PCA-50)
embeddings colored by true (Nm330, perplexity=30.00
Antigen- only (nucleotide)
Segment-wise
10-
10000
-10
HUNI
H130
1032
1081
-20
11340
HC51
HS8S
10N3
-30
-20
-10
20
Figure 6: t-SNE of penultimate-layer embeddings from four finetuned subtype classifiers, each ini- tialized from a different pretraining variant. Full-genome and autigen-only(nuclootid generally yields the most compact class clusters; Segment-wii wise is looser; Shomplete-genome lies in between.
19

---

Published as a conference paper at ICLR 2026
HBNZ
H10NB
Model Full-genome
Segment-wise
H7N9
incomplete-gename
0.6
0.4
Antigen-only (nucleotide)
H5N6
HINZ
HSN1
H2N2
HBN1
H3N2
Figure 7: Radar plot of per-subtype Fl scores for AntigenLM (full-genome), Incomplete-genome, Segment-wise models and Antigen-only (nucleotide).
20


Reviews:
[{"review_id": "xYIwjFbcGe", "reviewer": "Reviewer_DjYH", "summary": "The authors developed a new DNA language model, AntigenLM, for predicting the evolution of influenza genomes. This is the first work on applying DNA LM to this task to my knowledge. The training data appear carefully constructed, and the evaluation tasks were well designed. It showed notable performance gains over traditional methods. Although I have some concerns regarding excluding protein language models from the benchmark, I overall recommend a weak acceptance of this work.", "questions": "1. Is the sentinel token not used during pretraining?\n2. I am a bit confused about how the notion of time is built into the model. For example, in the next-month prediction, is the model just fine-tuned on sequences at month t, and then used to generate sequences supposedly for month t+1 without any constraint or guidance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. It is a novel idea to use a DNA language model for predicting viral evolution, which is an important problem for public health.\n2. There are very few DNA language models for eukaryotic viruses, and this work has filled this gap.\n3. The training data is carefully curated and documented in detail. It could be a useful resource to the community.\n4. The evaluation tasks are well-designed, and data splitting was done thoughtfully.", "weaknesses": "1. I am not sure if the authors covered all the appropriate baselines. I am not super familiar with the field, but there seem to be many works on using protein language models to predict viral evolution. For example, https://www.science.org/doi/10.1126/science.abd7331, https://www.nature.com/articles/s41392-024-02066-x, https://www.biorxiv.org/content/10.1101/2025.08.04.668423v1. Although this model is trained at the DNA level, the evaluations seem mostly at the amino acid level. The authors should justify excluding those models from the comparisons.\n2. Following up on the previous point, the authors should also demonstrate the necessity or advantage of modeling at the DNA level rather than the protein level.\n3. The description of the model architecture is obscure. For example, does it have 6 heads per layer or 1 head per layer? A more detailed diagram of the architecture might be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pWxAo28A6n", "reviewer": "Reviewer_be7j", "summary": "This paper presents AntigenLM, a transformer-based DNA language model designed to predict how influenza viruses evolve. Instead of treating genomes as random strings of nucleotides, the model keeps the natural structure of the virus—eight gene segments joined in the right order—so it can learn both local and global relationships. The authors pretrain AntigenLM on more than 50,000 influenza A genomes and then fine-tune it to forecast future HA and NA sequences, which are key for vaccine design. They compare it with existing evolutionary and sequence models like beth-1 and show that AntigenLM gives lower amino-acid mismatches and better subtype classification. Ablation studies also suggest that keeping the full genome intact during pretraining matters for accuracy. Overall, it’s a clear attempt to bring structure awareness into DNA foundation models for real biological forecasting.", "questions": "1. The t-SNE plots in the appendix look clean, but t-SNE always makes clusters look nice. Could we see a quantitative metric instead of a pretty visualization?\n\n2. There’s no comparison to protein-level models (e.g., ESM-2). Would those perform equally well or even better for HA/NA sequence forecasting?\n\n3. In the results, H3N2-NA seems slightly worse under U.S. testing. Any insight why that happens? Could it reveal overfitting to regional sequence biases?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The idea of preserving full functional units during pretraining feels biologically meaningful and easy to justify. It makes sense that genomes should be learned as whole systems, not chopped-up pieces. The model is compact and computationally reasonable, which makes the method more accessible to other labs. The overall contribution to treating DNA as structured language is simple but potentially important for future genomic foundation models.", "weaknesses": "1. The paper claims a big jump over existing models, but I’m not fully convinced it’s a fair comparison. The baselines feel a bit old — why not include newer genomic LMs like HyenaDNA or TITAN? That would make the improvement more believable.\n\n2. Figure 1 is visually appealing and central to understanding the paper, but it could better communicate data imbalance, temporal coverage, and architectural specifics. A supplementary figure showing training sample counts per region/year and a more explicit schematic of sentinel-token flow would improve interpretability.\n\n3. The functional-unit aware idea sounds interesting, but it feels mostly like a data-organization trick, not a new modeling method. Could similar results be achieved just by using longer context windows?\n\n4. The results in Figure 2 look visually impressive, but some doubts arise. The reported 70% mismatch reduction seems unusually large, given that beth-1 already performs well on H1N1/H3N2; could these differences stem from dataset splits or evaluation metrics rather than genuine model ability? Error bars are wide and overlap in several cases, yet statistical significance isn’t shown. It’s also unclear whether test sets are truly unseen in time (to avoid leakage) or whether the consensus targets overly smooth the diversity of real viral populations.\n\n5. H1N1 and H3N2 dominate the dataset, while minor subtypes like H7N9 have extremely small test sets. The paper interprets success on H7N9 as cross-subtype generalization, but with <50 examples, this could just reflect memorization or noise.\n\n6. The paper doesn't seem to have been revised well. The title of the paper is not written in the correct place. Figure 2 caption includes the typographical symbol “¿70%” instead of a proper “~70%” or “≈70%.” Appears to be a stray inverted question mark (encoding artifact) at p.7, caption line. Add proper appendix separation and consistent figure numbering. \n\n7. The model architecture (6 layers, 384 dim) is very small compared to typical DNA LMs. Are we sure it isn’t underpowered — or that the gain isn’t just from better preprocessing?\n\n8. The authors say the model “scales to population-level datasets,” but training was done on ~54 k genomes. That’s not really population scale; what happens at millions?\n\n9. The “near-perfect” subtype classification (99.8 F1) seems almost too good to be true. Was the classifier evaluated on strictly unseen subtypes or just held-out samples of known ones?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "afQ9Aawf8j", "reviewer": "Reviewer_HonT", "summary": "The influenza virus evolves continuously to evade human adaptive immunity, leading to seasonal epidemics. As a result, influenza vaccine strains must be updated each year to maintain their effectiveness for the upcoming flu season. The paper introduces AntigenLM, an autoregressive DNA language model pretrained on aligned influenza genomes. It robustly captures evolutionary constraints and transfers effectively to the generative tasks of strain forecasting for upcoming influenza seasons. Furthermore, to support the pretraining on aligned genome data, the paper showed that training on unaligned or fragmented gene sequences degrades performance. AntigenLM was compared to an evolutionary model, beth-1, and a tree-based predictor, LBI, on forecasting tasks, where it showed superior forecasting performance on most of the influenza genome segments.", "questions": "1. Please comment and elaborate on the choice of the baseline pretraining methods. It's important to include this in the paper.\n2. Please comment on the superior performance of beth-1 on NA epitopes. It's important to include this in the paper as well.\n3. How exactly was fine-tuning for next month and next season done? During inference, what was a prompt? Could you elaborate on this in the paper and illustrate, if possible?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper presented a sound way of collecting and preprocessing the influenza genomes, taking care of gene segments alignment and functional unit preservation, which led to an efficient autoregressive model pretraining. Additionally, it has been shown that overlooking alignment and functional unit preservation can lead to performance degradation.\n- The proposed model improved influenza forecasting over evolutionary models and tree-based predictors. The model achieved lower genetic mismatches between predicted and observed consensus amino-acid sequences.\n- The paper showed geographic and cross-subtype generalization. Specifically, the model was tested on held-out U.S. sequences during finetuning. Additionally, unlike baseline models, the proposed model is able to forecast minor-subtype sequences. The minor subtype sequences were present in pretraining and finetuning, but in much lower amounts (<5% and <1% in pretraining and finetuning corpora, respectively).\n- Even though the paper lacks ML novelty, I think the work was technically done well. The data preparation is sound, the model pretraining and finetuning were done effectively, resulting in outperforming the baseline models.\n- The authors promised to release code, documentation and data retrieval guidance subject to acceptance.", "weaknesses": "- The presentation of the paper requires improvement. The overall writing style and, in particular, the clarity of explanations are of low quality. While the pretraining procedure of the model is well described, the fine-tuning section is poorly explained. The fine-tuning process for each specific task should be described in greater detail. I recommend illustrating the fine-tuning procedures with clear indications of the prompts and the regions being forecasted. The current Figure 1B does not contribute to clarity.\n- Although segment-wise pretraining is a reasonable approach, the incomplete-genome pretraining with random cropping as a baseline model appears conceptually weak. It would be more meaningful to pretrain on incomplete genomes while excluding entire segments that are not used during fine-tuning and forecasting, retaining only the complete HA and NA segments in their correct order. This would demonstrate whether information from other segments contributes meaningfully, or if comparable performance can be achieved using only the properly aligned HA/NA segments.\n- In the segment-wise pretraining, all segments—including those not used during fine-tuning and inference—are utilized. The non-HA/NA segments are likely less informative. The authors should elaborate on their choice of baseline models (segment-wise pretraining and incomplete-genome pretraining) and clarify why these represent sound alternative approaches. At present, the baseline models appear to have been selected post hoc, possibly after the development of AntigenLM, merely to provide comparison points.\n- Epitopes are specific regions within influenza HA/NA segments that are targeted by the host immune system. While AntigenLM outperforms beth-1 and LBI on HA epitopes, the authors do not discuss the superior performance of beth-1 on NA epitopes. This omission should be addressed.\n\n**Minor comments:**\n- There are typos in Fig. 1B. \"Finetuneing\" should be written as \"finetuning.\"\n- In Fig. 1B, the heads (classification and LM) blocks and the captions do not align with the text lines 134 -- 138.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IT8PZGpKkm", "reviewer": "Reviewer_Vogk", "summary": "This paper introduces AntigenLM, a structure-aware DNA language model designed to improve influenza forecasting. It identifies a key limitation: task-specific evolutionary models (like beth-1) often treat mutations as independent events, while general-purpose DNA foundation models are often trained on fragmented, unaligned data, causing them to miss biological context.\n\nTo address this, AntigenLM is pretrained on over 54,000 complete influenza A genomes, with each training sample consisting of all eight gene segments concatenated in a fixed order. The hypothesis is that preserving these intact functional units allows the model to learn co-evolutionary dependencies across the entire genome. The model itself is a 6-layer GPT-2-style Transformer with a 13k-token context window to accommodate full genomes.\n\nThe evaluation shows that AntigenLM outperforms conventional baselines (beth-1, LBI) in next-season forecasting, reducing amino acid mismatches by over 50% relative to beth-1 on key tasks. Ablation studies show that this Full-genome model outperforms two variants: one trained on Incomplete-genome (randomly cropped) sequences and another on isolated Segment-wise data. The model also shows strong generalization to unseen subtypes (H7N9) and geographies.", "questions": "Most of my major concerns and related recommendations have been stated in the Weaknesses section. I recommend focusing the efforts on addressing those points, as they are critical for strengthening the manuscript in the rebuttal stage.\n\nThe following are more specific, minor questions to help the authors think more deeply about certain design choices and experiment setups, which might be helpful for this and future work:\n\n**(Q1)** In Sec. 3.2, how does the model encode sentinel tokens, and how are transitions across gene boundaries handled during both pretraining and decoding? What happens if a generation crosses into a functionally distinct segment? Does this terminate generation, penalize the loss, or trigger any corrective mechanism?\n\n**(Q2)** Beyond t-SNE visualization, are there any more quantitative cluster separation scores or biological validation of embedding space distinctions, especially for rare subtypes?\n\n**(Q3)** Are improvements in Fig. 2 and related tables statistically significant? Is it possible to quantify confidence in reported gains? From example, utilizing p-values or bootstrap intervals.\n\n\n---\n## Justifications:\n\nIn summary, this paper presents a promising and intuitive pretraining method for viral genomics. The core idea is strong, and the ablation studies provide clear support for it.\n\nHowever, the current manuscript seems to be not yet complete and with severe issues. The claims of superiority over other foundation models are not supported by direct comparisons, and the claims about learning co-evolution are unproven. IMHO, these are not minor issues but strike at the overall quality of the paper.\n\nTherefore, I cannot recommend acceptance at this stage. I would be glad to raise my rating if thoughtful responses and improvements are provided in the rebuttal stage. I am also open to follow-up discussions with the authors to help further strengthen this work.\n\nI hope these comments help my fellow reviewers and ACs understand the basis of my recommendation.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**(S1)** A clear and important research problem. This work clearly demonstrates the limitations of both site-based evolutionary models and general-purpose genomic foundation models. The hypothesis that a DNA LM for viruses must be pretrained on aligned, intact functional units is intuitive. These arguments thus motivates the need for a new approach. \n\n**(S2)** The integration of biological structure in method. AntigenLM enforces gene- and segment-level boundaries during pre-training and fine-tuning. This offers a compelling biological motivation for functional-unit-aware modeling.\n\n**(S3)** Complete experiment setup. The breadth of the experiments is excellent. It spans several axes such as short-term and long-term sequence forecasting, cross-geographical generalization, rare subtype transfer, and explicit subtype classification. This builds a robust case for the model's capabilities and shows practical utility on real-world scenarios. I particularly respect that some practical considerations like replicate vaccine decision timelines and public health applicability are thoughtfully integrated in experiment framing and data curation.\n\n**(S4)** Interpretability and Visualization. The use of t-SNE visualizations in Fig. 5 and confusion matrices in Fig. 3 offers great interpretability to learned representations. It clearly shows that AntigenLM’s embeddings can yield more distinct and biologically meaningful subtype clusters compared to baselines.\n\n**(S5)** Rigorous Ablation Studies. I respect the design of ablation studies. The comparison of the Full-genome model against its Incomplete-genome and Segment-wise variants counterparts provides a direct evidence of the value of structure preservation. Especially, the performance drop of the ablated models (e.g., perplexity jumping, and near-perfect classification vs subtype confusion) provides a strong support for this paper’s main claim.", "weaknesses": "**(W1)** Severe formatting errors (about title). The most immediate and severe issue in my view is that the manuscript is improperly compiled and appears to be in a draft state. The main title of the paper is a placeholder. The actual title, \"AntigenLM: Structure-Aware DNA Language Modeling for Influenza,\" is incorrectly placed as a line of text above the Abstract section. This should be a major flaw for a submission to a top-tier conference. However, given that ICLR permits revisions and iterative author-reviewer discussions, I will reserve final judgment on this shortcoming. I strongly encourage the authors to provide a corrected, complete manuscript in the rebuttal phase.\n\n**(W2)** Incomplete literature review especially for some highly relevant previous work. I encourage the authors to include related discussions in the revised manuscript.\n\n- [1] Learning the Language of Viral Evolution and Escape, Science 2021. This work develops a viral sequence LM directly tied to immune escape and evolutionary prediction for influenza. It is almost exactly the stated target domain here. It helps show what is actually novel in AntigenLM relative to contemporaneous, domain-specialized LMs.\n- [2] VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling, ICML 2024. This work studies learnable vocabularies which has shown a path beyond fixed tokenizers like BPE and k-mer. Its motivation is similar to AntigenLM which also critiques hand-crafted k-mer and fragmented approaches and offer a learned-representation instead, differing from AntigenLM's explicit functional-unit.\n- [3] Dinucleotide Evolutionary Dynamics in Influenza A Virus, Virus Evolution 2019. This work examines sequence-level constraints and evolution, providing insights for the biological basis behind AntigenLM's structural modeling.\n\n**(W3)** Comparisons to DNA foundation model. It is stated that general-purpose DNA foundation models underperform because they are trained on fragmented data. However, none of these models (e.g., ESM, DNABERT, HyenaDNA) are included as baselines for direct comparison. IMHO, the Incomplete-genome model may not be representative enough for these SOTA models, which use different tokenization and pre-training objectives. I encourage the authors to fine-tune at least one genomic LM like DNABERT on their exact downstream tasks and report its performance to support the claim.\n\n**(W4)** In visualization, the interpretive narrative is a bit thin. For example, Fig. 2 showcases reduced mismatch for the AntigenLM variants. But neither the main text nor the caption interprets which features (e.g., epitope conservation vs. overall sequence) drive the model’s superior forecasting. Panel D (geographic generalization) would benefit from a breakdown by outlier scenarios (as mentioned, H3N2-NA had worse US transfer—why?). Similarly, t-SNE visualization in Fig. 5 suggests tighter clusters for the full-genome variant, but quantitative cluster metrics (e.g., inter/intra-class distances) could substantiate claims.\n\n**(W5)** Some unclear method details. \n\n- In Sec. 3.2, while the core causal LM objective is standard, details are missing around the batch sampling strategy, positional encoding modifications for highly variable-length and segmented inputs, and the explicit formulation of the dual-task loss function (especially the weight or scheduling between the two heads).\n- Equations in Sec. 4.2 does not clarify if nucleotide padding, masking, or non-coding regions are handled during autoregressive loss computation. These are real concerns in genome application. How are sentinels encoded and differentiated from standard tokens? Are transitions across gene boundaries soft or hard? The method should specify if and how the model penalizes boundary crossing in generation mode.\n- The paper states that \"All pre-training sequences were aligned by segment\". In my view, this could mean (a) the 8 segments for a single virus were simply concatenated in a fixed order, or (b) a full Multiple Sequence Alignment (MSA) was performed across all 54,000 genomes for each segment before concatenation. If (b), this is a massive, non-trivial step, and the method for handling gap tokens is missing. I encourage the authors to clarify this in rebuttal. If it is simple concatenation, please replace the word \"aligned\" with \"concatenated in a fixed order\" for precision. If it was an MSA, the full method (tool, gap-token handling) is essential for reproducibility.\n\n**(W6) Minor Points**:\n\n- Model efficiency is described in texts in Sec. 3.3), but no specific metrics such as runtime, FLOPs, or GPU memory footprint.\n- The distinction between segment-wise and incomplete-genome variants in Fig. 4 could be more intuitively clarified early on for readers less familiar with genomic ML.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)