Protein Structure Tokenization via Geometric Byte Pair Encoding
Authors: Michael Sun
Weize Yuan
Gang Liu
Wojciech Matusik
Marinka Zitnik
Venue: ICLR 2026 Poster (2026)
Keywords: protein
geometry
structure
byte pair encoding
tokenizers
multi-scale protein structure
folds
backbones
motifs
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose a geometric byte-pair encoding tokenizer for proteins.

Abstract: Protein structure is central to biological function, and enabling multimodal protein models requires joint reasoning over sequence, structure, and function. A key barrier is the lack of principled protein structure tokenizers (PSTs): existing approaches fix token size or rely on continuous vector codebooks, limiting interpretability, multi-scale control, and transfer across architectures. We introduce GeoBPE, a geometry-grounded PST that transforms continuous, noisy, multi-scale backbone conformations into discrete ``sentences'' of geometry while enforcing global constraints. Analogous to byte-pair encoding, GeoBPE generates a hierarchical vocabulary of geometric primitives by iteratively (i) clustering Geo-Pair occurrences with k-medoids to yield a resolution-controllable vocabulary; (ii) quantizing each Geo-Pair to its closest medoid prototype; and (iii) reducing drift through differentiable inverse kinematics that optimizes boundary glue angles under an $\mathrm{SE}(3)$ end-frame loss. GeoBPE offers compression ($>$10× reduction in bits-per-residue at similar distortion rate), data efficiency ($>$10× less training data), and generalization (maintains test/train distortion ratio of $1.0-1.1$). It is architecture-agnostic: (a) its hierarchical vocabulary provides a strong inductive bias for coarsening residue-level embeddings from large PLMs into motif- and protein-level representations, consistently outperforming leading PSTs across $12$ tasks and $24$ test splits; (b) paired with a transformer, GeoBPE supports unconditional backbone generation via language modeling; and (c) tokens align with CATH functional families and support expert-interpretable case studies, offering functional meaning absent in prior PSTs.

Published as a conference paper at ICLR 2026
PROTEIN STRUCTURE TOKENIZATION VIA GEOMETRIC BYTE PAIR ENCODING
Michael Sun Weize Yuan', Gang Liu", Wojeiech Matusik', Marinka Zitnik?
MIT CSAIL Harvard Medical School AApple *Notre Dame
(msun415, wojciech) eesaelhh@essil esail mit edu, marinka@hns haaennaahms.haaaaeee harvard. edu
ABSTRACT
Protein structure is central to biological function, and enabling multimodal pro- tein models requires joint reasoning over sequence, structure, and function. A key barrier is the lack of principled protein structure tokenizers (PSTs): existing approaches fix token size or rely continuous vector codebooks, limiting inter- pretability, multi-scale control, and transfer across architectures. We introduce GEoBPE, geometry-grounde
PST that transforms continuous, noisy, multi- scale backbone conformations into discrete "sentences" of geometry while en- foreing global constraints. Analogous byte-pair encoding. GEOBPE gener- ates hierarchical vocabulary of geometric primitives by iteratively (i) clustering Geo-Pair occurrences with k-medoids yield ulary: (ii) quantizing each Geo-Pair to its closest medoid prototype: and (iii) reducing drift through differentiable inverse kinematics that optimizes bound- ary glue angles under SE(3) end-frame loss. GEOBPE offers compression (>10x 10x reduction in bits-per-residue at similar distortion rate), data efficiency (> 10x less training data), and generalization (maintains test/train distortion ratio 1.0 - 1.1). It is architecture-agnotti (a) its hierarchical vocabulary provides strong inductive bias for coarsening residue-level embeddings from large PLMs into motif- and protein-level representations, consistently outperforming leading PST's across 12 tasks and 24 test splits; (b) paired with transformer, GEOBPE supports unconditional backbone generation via language modeling; and (c) to- kens align with CATH functional families and support expert sspertintteprretbble case studies, offering functional meaning absent in prior PSTs. Code is available at httpss/gggttu...cmm///iingmgmm//
vocab-
INTRODUCTION
Protein language models (PLMs) trained large sequence databases capture evolutionary con- straints (Rives et al. 202T) and support de novo sequence design (Lin et al., 2023b), but they do not explicitly model fold geometry and may underperform
tasks where function depends struc- tural interactions (Abramson et al., 2024| Gelman et al.) (025). In natural language processing, byte-pair encoding (BPH) constructs vocabulary by iteratively merging the most frequent sym- bol pairs, producing a hierarchical representation of text (Larsson & Moffat, 2002). Despite BPE's success on sequential data, there is no geometric analog that can encode and decode protein back- bone conformations. The central difficulty is discretizing continuous, noisy structural variability while preserving global consistency. Because protein folds are organized into modular substructures (Petsko & Ringe 2004). protein structure tokenizer should (a) build a hierarchical vocabulary of structural motifs and (b) segment folds into hierarchical decompositions, producing symbolic and interpretable representations of backbone geometry.
Recently, vector-quantized variational autoencoders (VQ-VAEs) have become the most popular class of protein structure tokenizers (PST's, as adopted by ESM3 (Hayes et al., 2025) and others. VQ- VAEs learn an autoencoder that compresses and reconstructs protein structure with N residues to and from N quantized latent codes, which are discrete "words" drawn from vocabulary of learnalle embeddings (Van Den Oord et al. 2017). While powerful, VQ-VAEs lack the efficiency, interpretability and modularity of BPE tokenizers: (1) using fixed codebook create perfor- mance bottlenecks and imbalance token usage frequency, handicapping downstream performance Yuan al. 2025): (2) using vectors as tokens real data hinders interpretability, rows of

---

Published as a conference paper at ICLR 2026
2D matrix do not capture the hierarchical relationships between sub-words like in BPE; (3) lastly, fixing all tokens to have the same size prevents multi-scale resolution, which key to tasks that identify naturally occurring higher-level functional activity which span variable residue lengths.
Present work. We develop Geometric Byte-Pair Encoding (GEoBPE), a tokenizer that discretizes continuous protein backbones into symbolic "sentences" of structural motifs while learning hier- archical vocabulary. The design is motivated by two requirements: (i) protein folds contain modular substructures that should be captured as reusable tokens, and (ii) discrete approximations must pre- serve global geometric consistency. To meet these requirements, GEOBPE alternates between local updates and global corrections. At each step, frequent motif pairs are clustered with k-medoids and replaced by representative prototypes, recursively building higher-order motifs. This local quanti- zation inevitably introduces geometric drift, which GEOBPE corrects by optimizing boundary glue angles through differentiable inverse kinematics under an SE(3) end-frame loss. The output after each iteration segmentation of the backbone into quantized motifs and glue parameters; the sequence of iterations yields hierarchical decomposition of the fold, represented as merge tree of structural motifs (Fig. D. Our contributions are as follows: ! GEOBPE is the first geometry- grounded BPE analog for protein backhones, which builds hierarchical vocabulary of motifs and tokenizes structures through an alternating global-local decomposition with glue-aware reconstruc- tion. 2 On benchmark datasets, GEOBPE traces smooth Pareto front of compression-distortio tradcoffs, achieving up to 0.27-0.36x the bits-per-residue of ProToken and strong out-of-distribution generalization (test/train RMSD ratio 1.16-1. 1.28 vs. 6.4x for VQ- VAE). It also matches downstream accuracy when trained on as little as 1% of the pretraining data. 3 Hierarchical vocabularies from GEOBPE improve representation quality on tasks such as binding site prediction and fold classifica- tion, and transformer trained on its tokens enables unconditional backbone generation. 4 Tokens align with CATH domain annotations and are supported by expert case studies, providing functional protein insights and multi-resolutio interpretability.
Figure GEOBPE tokenizes protein into discrete motifs linked by boundary glue angles and learns hier- archical vocabulary of frequent structural primitives via k-medoids and recursively merging PeooPairs: at each step glue angles are optimized with diffcrentiable inverse kinematies to preserve the global fold. Tokenization yields merge tree that provides multi-resolution and interpretable representations of protein structure.
Tokin
Geo-SIP
Tokentzn
Gocients Lo
Learing
6-0-o
Breeeing
Papies Vadie
RELATED WORK
Protein Structural Alphabets. Structural alphabets approximate protein folds as successions of geometric motifs (Branden & Tooze 2012). 2) de Brevern et al.) (2000) introduced five-residue pro-
tein blocks from Protein Data Bank (PDB) structures, assigning fragments by RMSD. Later work showed that over 90% of residues can be covered by such alphabets (de Brevern et al. 2002) and
analyzed their quality and specificity (de Brevern 2005). Alphabet strings provide ID encodings of 3D geometry, enabling the use of sequence alignment for fold analysis and prediction (Mahajan al. 2015; Vetrivel al. 2017). [Camproux et al. (1999) proposed 12 building blocks via Hid- den Markov Models (HMMs) and extended it to capture whole-protein conformational variability [Camproux al. 2004). HMMs use inter-alpha-carbon distances within four residues observed variables. Broader tertiary descriptors, such as inter-residue distances or moment invariants (Du- rairaj et al. 2020), capture non-contiguous context: Mackenzie (2016) found ~600 motifs describe more than half of structural space (39 10" conformations), indicating variability collapses into limited modes. Such descriptors extend to protein-level retrieval and classification (Durairaj et al.) 2020 Van Kempen et al. 2024)
et al., 2023). GEOBPE builds on these insights by treating structural motifs as extensible primitives and dynamically adjusting alphabet size and token resolution, unlike fixed structural alphabets.
Protein Structure Tokenizers. Modern PSTs, most notably VQ- VAEs, construct structural alpha- bets by training deep autoencoders with vectorized codebooks that map continuous structure dis-
2

---

Published as a conference paper at ICLR 2026
crete codes (Van Den Oord et al.
2017). Building on this idea, FoldSeek (Van Kempen et al.
2024)
introduced 3Di alphabets (20 discrete codes learned with VQ-VAE) that compress local structural features for efficient search and homology detection. Subsequent works integrate 3Di alphabets with
PLMs: Heinzinger et al. 2024) translate between 3Di and amino acid sequences; [Su et al. (2023)
define "3Di-residue" tokens and show pretraining with this vocabulary improves prediction; and et al. (2024) use disentangled attention to jointly model 3Di and residue tokens with a structure quantization module. End-to-end VQ-VAEs avoid predefined descriptors by training equivariant encoders and decoders to tokenize structure directly, achieving near-perfeed reconstruction but at high computational cost. Large-scale efforts such as ESM3 (Hayes et al. 2025), trained on 236 mil- lion structures, highlight the central role of tokenizers in scaling multimodal PLMs. Recent work benchmarks tokenizer performance itself: AIDO.St and ProTokkess show that stronger compression improves retrieval (Van Kempen et al. 2024 Zhang et al. (024c) but reduces reconstruction quality, and both Zhang et al. (2024c) and Lin et al. (2023a) integrate tokenizers tightly with transformers.
GEOBPE differs by using its hierarchical vocabulary an inductive bias for representation learning and by supporting geometry- -rrouudde language modeling without latent space vector quantization.
Byte-Pair Encoding for Biological Data. BPE underlies modern language models and has been applied to biological sequences with mixed outcomes. On genomes, BPE achieves superior com- pression and improves over k-mers in language models (Dotan et al. 2024 Zhou al. 2023), though Nguyen et al. ) find the opposite using Hyena. For functional tasks, BPE often per- forms best (Dotan et al. 2024), while on nucleotide-resolution tasks it can underperform (Lindsey et al.) 2025). These results indicate tokenizer utility depends on task scale and architecture, mo- tivating GEOBPE's architecture-age agnnosti design and multi-scale resolution. Linguistic differences between text and biological sequences further complicate direct transfer: BPE tokens do not align with domain boundaries (Suyunu al., 2024) or regulatory motifs (Lindsey al. 2025). Other studies emphasize the importance of vocabulary design, reduced amino acid alphabets impair struc- ture prediction (Ieremic et al. 2024), while BPE vocabularies of 50-200 tokens are often optimal for sequence tasks (Tan et al. protein structures. GEOBPE extends BPE by grounding tokenization in geometry, exposing param- eters for quantization, vocabulary, and efficiency, while uniquely providing fine-grained resolution control and hierarchical motif vocabulary.
2024). Overall, existing tokenizers, including BPE, lack versatility for
3 METHODS
We first establish backbone geometry notations in Sec. 3.1 Sec. 3.2 presents the GEOBPE algo- rithm, detailing its components for motif clustering, adaptive quantization, and glue-aware refine- ment. Finally, Sec. [3.3 formalizes the principles that an ideal protein structure tokenizer should satisfy and evaluates how GEOBPE meets them.
3.1 NOTATION & PRELIMINARIES
Global Backhone Formulation. Let protein backbone (() with (T) residues be represented
by the Cartesian coordinates {(N,, CA,, C.) E R3x3 (((r) omitted). Define bond lengths, bond angles, and dihedrals:
Ji=1 of backbone atoms (oxygen and C3 [
NN-CA = I|N; CA.ll, CCAC = =IICA Cill; G-N = =IC 1-Cii CG-N++ill-
NNCAAC LN,, CA,,CA,, 0F = =(CAA, Co, N.+1) ORNCA ((i,, Net1, CA++1). v. = (N,, CA2, C,, W, = L(CA,, C, Ne41, CA+H1), o. L(C,, CAppi C41):
We annotate these definitions in a toy (N(+) 2) example in Fig. 2 (top). The full internal representa- tion thus contains 3 (r) bond lengths, ())(r)_2
Mr star 1.2)
N
C
Ca
bond angles, and (-) dihedrals and is invariant to any (R, t) E SE(3).
Ca
N
C
Local Formulation (Bond-Residue). For residue i we define the bond-residue as the ordered triple (N; CA.), (CA,-C.), (C-N++1) together with its internal angles. For i < (T) this includes
dy MI)
Fi
Figure 2: Toy backbone, with internal angles, glues T3, motif Mi and per-link transform G1-
the lengths e. eN-CA CA-C CC- the bond angles
3

---

Published as a conference paper at ICLR 2026
NNOIC, BCACN and the peptide dihedral th about CA2- -G For i N it includes only bond pN-CA ,CA-C and angle N (the (C-N) bond, A OONCA and (1/, 42,,4, Φ) NC lengths No) (No) dihedrals are absent).
Glue Parameters Between Neighboring Bond-Residue. Neighboring bond-residues i and i+1 are connected by a set of glue angles that place the bonds of residue i+1 relative to residue i. These are r; {ONNCA, Oi, wi w i.e. one bond angle OCNCA (to place N++1-A+++1 and two dihedrals Ø; and w; (to orient CA C++1 and the peptide plane). We adopt (w, .) here to emphasize the two independent dihedral DOFs spanning the peptide and CA torsions.
Motif Formulation. A bond-resiane rend-residue motif Mpy is contiguous block of hond-residue
q (1 p S N())). Its internal parameter set is the union over the internal bond lengths and angles of bond-residues p,....q together with the internal glue angles {T.) that connect consecutive
P.....q
inside the motif. Given y r N obtain Geo-Pair occurrence (M Merr) from the internal parameters of Mpig and Mq+1:rr plus the external glue angles I, connecting the last and first bond-residues of Man and Mg+1:r
Entry/Exit Frames. For residue i, define F; = [(R; (R) E SE(3) with origin t; = CA2 and axes chosen that the x-axis points from CA, toward the C, the y-axis is the normalized component of the CA,- - N; direction orthogonal to and the z-axis completes right-handed triad.
Per-Link Transform. Define the transform between consecutive residue frames G; F F. E SE(3). By construction, G; is deterministic function of the internal coordinates local to the link inki-++1,, namely G; dCC.A-C C @NC.C OC-ACON, 8, th Wis &i), and, in particular, depends on the glue set I; = =OCCNE di, W: illustrated Fig. (bottom).
Entry/Exit Transforms. For motif Mpxq define Fentry = Fp. Fexit = Fq. The internal entry transform is Tint on the internal coordinates Mp.g. The external glue transform between consecutive motifs Mp 1 the glue set I' (and the adjacent three bond lengths).
Fexit piq (F (Fentry)-1
(Gg-1) (Gp), which depends only
and M is precisely the boundary link 1 gelue
F F7 Gg. and is paramcterized by
Figure 3: (Top) GeoBPE tracks Geo-Pair Encoding, dictionary mapping Geo-Pair keys occurrences all times. Each step pops the most frequent Geo-Pair key, gathers occurrences (8...a..0....) and fixes K prototypes (I.I.) to add to V. All occurrences are quantized to the closest prototype (c.g. 0 + D. Glue angles (O,C) are optimized to correct for the drift introduced. (Bottom) Toy example with two backbones; we initialize residuc-orientation modes using two prototypes (1)), pop the frequent Geo-Pair (=), quantize occurrences ( and optimize glue angles.
DUSIN
Aigh Crmsin
Core GeoBPE notation. We define the main objects used throughout the algorithmic descrip-
tion. We index training backbones by 1,..., writing the 7-th backbone {87) with
4

---

Published as a conference paper at ICLR 2026
N residues. A segmentation of {(r) into bond-reside residue motifs is the ordered tuple p(r) (M (Mprnt...,Mpp Mp,. Mpx..,,,,,..0. :M.)) with = p1 S q1 < P2 ....
... S qM, = N(r), The corresponding merge hierarchy F(r) is a binary forest whose frontier leaves, in order, equal p(r); each internal node represents a merged motif and stores its span [pxq].
Geo-pair keys and occurrences. Given two adjacent motifs (Mpsg Mg+1=)) and their boundary glue Tg. define canonical, hashable geo-pair key k COMPUTEGEOKEY MMpg:: (Alg. 22). For each key x we collect its occurrence set (K)) consisting of all such adjacent motif pairs across the dataset.
Prototypes and vocabulary. For a geo-pair key K. GEOBPE clusters its occurrences and stores small set of representative prototypes AA.. = {11 Kicl
j==1 where each m1 is the internal parameter tuple of medoid occurrence and K g+ {1] + z controls how many prototypes by motif'
(bond) length. The vocabulary the map V A: + Ag. initially containing residue-level codebooks
As, A2 and growing as new geo-pair keys are introduced.
Geo-pair dictionary and priorities. any time GEOBPE maintains a priority-ordered dictio- nary D that maps each key to its occurrence set (1:). Keys are ordered by tuples
(K) = (p(*), (00)),,,
p(x) = 1 [ & dom(V)),
so that compressible keys with existing prototypes (p 0) are popped first, followed by a new key with the largest count (O=) per iteration.
3.2 GROBPE ALGORITHM
GEOBPE (Algo. Fig. is organized around four components: (1) clustering motif (individ- ual bond-residues once at the start, Geo-Pairs every step thereafter) occurrences into representative structural prototypes, (2) maintaining an ordered map to track frequent Geo-Pairs, (3) adaptively hard-quantizing noisy Geo-Pairs to their assigned prototypes, and (4) applying rigid-body refine- ment to enforce global geometric consistency.
Components (1)-(4) are designed to answer three new key questions when re- re-interpreting BPE to work with continuous backbone geometry rather than discrete bytes: (a) how do ground contin- uous backbone states to discrete keys for Geo-Pair counting, (b) how do we update the backbone states once we have popped the most frequent Geo-Pair, (c) how do we synchronize the Geo-Pair dictionary with how we updated the backbone states. Component (1) answers to (a), (3) & (4) to (b), and (2) to (c). The guiding question encompassing (a)-(c) is: what the exact relationship between the Geo-Pair dictionary (needed for discrete BPE operations) and the continuous back- bone states (which should both reflect new keys and preserve original fidelity)? GEOBPE imple- ments two-way connection through four stages: grounding. in which continuous motif states define discrete prototype keys; quantization, which internal parameters are overwritten with those of the assigned prototypes; rigid-body refinement, in which backbone internal states self-correct to mini- mize global distortion; and synchronization, in which the Geo-Pair dictionary is re-synchronized to reflect the high-fidelity backbone states after quantization and refinement.
(1) Extracting Dominant Modes from a Set of Motif Occurrences. The core subroutine invoked by GEOBPE is Algo. 6 which clusters set of length J. raw backbone fragments into K repre- sentative prototypes. This induces a hard quantization of the fragment space, since every possible occurrence assigned to exactly one prototype. Because RMSD defines metric over fragments, the clustering yields a Voronoi partition of this space. Importantly, the medoids are themselves ob- served fragments, so each quantized symbol retains a concrete structural interpretation: it represents the closest empirically observed conformation, providing a denoised approximation of local vari- ability. Each time we quantize, we substitute every non-medoid occurrence by its assigned medoid, replacing all internal parameters with the medoid's internal parameters, making it an exact copy of that medoid (same length and per- positios angles).
(2) Constructing Structural Motif Alphabet. GEOBPE begins by quantizing all bond-residues and glue angles (Algo. [8) and building an ordered map D of discrete geo-pair grounding keys k to occurrences (1s) (Algo.) 21 see Core GeoBPE notation). In each iteration (Algo. GEOBPE pops
5

---

Published as a conference paper at ICLR 2026
Algorithm GEOBPE: Protein structure tokenizer with geometric byte-pair encoding
Require: Backbones {t()).... [(T] with lengths N( optional backbones {t(E) to tokenize; residue codebook sizes (K3, K2); glue-IK weights (wn, wy): maximum merge iterations Smax- Ensure: Final vocabulary V (motif prototypes), final segmentations ((((r), final merge hierarchies (F(r),, and the priority-ordered geo-pair map D.
1: Empirical quantizer estimation (once), Collect samples over all backbones for the types ACACN _CNNCA,, f 11 w). Wrap angles to [0, 2x) and build circular histograms with edges 0 = Bn <... < AB = 2x that tile the circle; define Q by snapping bin centers. For lengths. build linear histograms and snap centers.
(CN-CA (AN-CA_CAA- (C-N),
2: Per-residue initialization (Algo. [8]. Cluster interior and terminal bond-residues via RMSD PARTITION to obtain codebooks As. A2; overwrite each residue's internals by its as- signed prototype. Set the initial segmentation for each backbone:
p(r) = (M Mlte) Mo))))
Initialize hierarchies: for each T, create binary forest F(r) whose leaves are the bond-residue motifs ( in order; its frontier equals p(-), Initialize the vocabulary with base prototypes:
V (residue-level keys + As.
3: Global glue refinement (Algo. 12 Optimize all boundary glues r, (ACNOA, wis 0i+1) 1) via differentiable FK with (wn, W/ snap each to the nearest bin center using QOeNCA, Qw Qá- 4: Build the pridrit geo-pair map (Algo. 21). Using the frontier leaves of each F(()) (equivalently, p(r)), construct the occurrence sets (1c) and insert:
([(o(k),- x)] + O(s), p(x) = 1[ks & dom())).
5: BPE loop calls (Algo. 9 each step.
6: for 1 to Smax do
7: ((P(r)), (FO), }. D, V) + STEP STEP((P(r)), (FV)), D, V. (QUONCA, Qui Qal: (wn, wi)
8: end for
9: Tokenize new/unseen backbones (Algo. 10 for each &.
F(6) TOKENIZE (()) As, A2, V, LOOONE,
10: return V. (p(T)), (F(r)), D and (if given) (p(t)), {F(E)
the most frequent Geo-Pair key, runs Algo. 6 on mapped occurrences, quantizes the occurrences, runs rigid-body refinement, and updates D to account for the new quantized backbone states.
(3) Multi-Resolution & Adaptive (Re-)Quantization. One-time quantization is lossy procedure and is only needed to index Geo-Pairs occurrences the current step. Thus, each GEOBPE iteration can re-quantize occurrences by referencing the original, even if prior iterations have quantized the same regions already. This allows resolution to adapt based on the size of the motif (c.g.. coarse- grained for smaller motifs, fine-grained for larger ones), providing precise control over compression- reconstruction tradeoffs (see App. A)
(4) Minimizing Distortion via Rigid-Body Refinement. Let T denote the entry-+exit SE(3) map of a motif May determined by its internal coordinates. For an occurrence u with original motif
(t.) the rounding step replaces by its assigned medoid segment: bu
(mmaiu)) imatis skmatts
M
(fa) S
M
where Me(u) is the medoid index returned by Rounding thus replaces Toce by 7wedd and the induced
(an index into S). Let Toce =
Thut kuik. and Tmed
Tiut mm lemoto) discrepancy AT, Toor (T.eed) is the drift introduced by quantization. If left uncompensated, products of such AT, across a chain accumulate and move exit frames off their original targets. Each boundary provides 3 gluing degrees of freedom (T²) that can absorb this drift. To exactly recover the

---

Published as a conference paper at ICLR 2026
original exit (in the idealized case), the boundary transform at the link in 1.-11 - iu should satisfy:
opt Vars
Cnew
Timed
Gorig bu=1
Tuce
G
Crew by
G 11
AT,, t.
where the quantization drift ATu
Ty (Tmmed) -1 Since G is controlled by only three gluing DOFs, we solve for G.. in least squares via the end-frame fitting objective:
(T = wR log (RA..) R R
+ we l|te. - 1 12.112
with forward kinematics FR.. quantizing many motifs on the same backbone, performing this optimization each time can become computationally prohibitive. Instead, we adopt global (batch) alternative which treats all gluing DOFs as parameters, with a global end-frame fitting loss. This provides maximum flexibility in drift compensation. The algorithmic details are in Algo. 19 and 12
F Gi--1 Timed FX.
Fi-1 1 T The When
Transferring Hierarchical Inductive Biases. GEOBPE adapts the receptive field of a base residue- level feature extractor to that of the whole structure, connecting multiple scales through recursive aggregation. Algo. Demits merge hierarchies F as forest: leaf nodes represent residues and parent nodes represent motifs. The key insight to use F as a recursive computation tree. Leaf nodes are initialized with pretrained features, then embeddings propagate up along the parent-child relations of F until the forest roots (aligned with P); a final step aggregates the forest roots into a protein- level contextual embedding: then they propagated down until the leaf nodes. The final leaf nodes output residue-level embeddings induced by the hierarchical and informed by multi-scale GEOBPE tokenization. These features support supervised learning fine-grained residue-level tasks (e.g., active site prediction) and coarse-grained global predictions (e.g., fold classification). See Algo. 15 for details.
3.3 PRINCIPLES OF PROTEIN STRUCTURE TOKENIZATION
Let X' = 3R333 be the space of backbone coordinate tensors and let V be a finite codebook. A tokenizer tuple
(V, Enc, Dec): Enc : X' + V mapping structure x to a finite token sequence q Enc(x), Dec Im((nc) + X' mapping X = Dec(Enc(x)) For dataset D C X and distortion x X + [0, oo) (e.g., Kabsch-aligned RMSD per residue), define:
BPR(T: L(T) + ExED L(Enc(x)) bits/res EXEDN(x)
A(T;D)
Σ d(x, Dec(Enc(x))),
D
where T((T) 2 is the description length of (V, Enc, Dec) and N(x) is the residue count; under uniform per-token code, I.(Enc(x)) = |Fnnc()) loga [V]. We setup the following principles for an ideal tokenizer i and empirically explore the degree GEOBPE satisfies them.
Principle 1: Pareto-optimal D. T is Pareto-optimal on D iff no T' satisfies BPR(T'; <
BPR(T (; D) and D(T < A(T; with at least one strict. We empirically explore this principle by evaluating Pareto-efficiency among leading PSTs and codebook configurations in Fig.
Priciple 2: Out-of-distribution (OOD) generalization. i generalizes OOD if, on unseen test set
Drest C X, A(T Dres) ~ D). We depict generalization gaps of leading PSTs in Fig.
Principle 3: Downstream transfer via codebook//occabulrry Let V be the vocabulary of T and let (x)) be the residue count. Let θ parameterize a pretrained feature extractor. The codebook/vo- cabulary V induces per-residue features (x(x) = (F(e(())) E (Rd)N(x) An ideal tokenizer of protein structures should go beyond pure compression; should learn useful signals related to function. We loosely define the ability of PST to transfer useful signals by test performance on a battery of downstream tasks when parameterizing samples x by the vocabulary V together with feature extractor O We benchmark downstream transfer of GEOBPE against others in Table (RANSSERR)
4 EXPERIMENTS
We answer ten research questions (QI-Q10) to benchmark the performance, efficiency, and applica- tion integration potential of GEOBPE against other popular tokenizers.
7

---

Published as a conference paper at ICLR 2026
Tokenization Performance: (Q1) How many bits are needed to store the tokenizer and tokenized inputs? (Q2) How faithful is the reconstruction? (Q3) How does performance generalize to unseen data? (Q4) How many samples are needed to train the tokenizer?
Token Efficiency: (Q5) How frequent and balanced is vocabulary utilization? (Q6) Does small-
scale language modeling generate better structures with GEOBPE or VQ-VAE tokens?
Downstream Transfer: (Q7) How much transferrable signal does the tokenizer capture about the data? (Q8) How much does the vocabulary help on representation learning tasks?
Interpretability: (Q9) How well do GEO PE tokens agree with "ground-trutt domain annota- tions? (Q10) Can experts understand GEOBPE through real-world case studies?
Datasets. We follow the same dataset splits as in Yuan et al. (202)). Pretraining uses structures from the Protein Data Bank following OpenFold2's protocol and retained a non-redundant subset of x48K protein chains, which were split into training/validate served as held-out test sets for evaluating OOD generalization and token efficiency. For downstream evaluation. use 8 datasets, spanning residue-level classification (ligand binding, catalytic, con- served, repeat, and epitope), residue-leve regression (structural flexibility prediction), and protein- level classification. TTgetteer these datasets probe functional relevance, structural variability, token distinctiveness, and efficiency across a wide range of proteins. For citations and details, see App. [B]
sets, with CAMEO and CASP14 re-
Baselines. We compare with VQ-VAEs, the leading family of discrete PSTs (Hayes et al. 2025
Van Kempen et al. 2024 Lin et al. 2023a Yuan et al. (025), They consist of (1) structure en- coder maps structure x into a continuous representation 2 E R NxD: (2) a vector quantization layer discretizes each 2; by selecting k; = arg min, d(zi, 4j) from a learnable codebook Q E R
and (3) structure decoder reconstructs I = from the discrete codes 9k {qk, 1 We also compare with Inverse Folding (IF) continuous PSTs, which skips the quantization step z + qu and
trained to recover the amino acid sequence from z (Dauparas et al.) 2022: Yang et al. 2023).
Downstream Transfer. For VQ-VAEs, O and V are jointly learned, so we set "'v vVQ-VAB(x) +- Enc(x). For
VQ-VAF
use ① +- ESM3 to demonstrate how y GLOBPE can transfer useful signals from (x) to ry (x))
Performance Metrics. Compression measures Bits-Per- Pes-Residue (BPR), as defined in Sec. 3.3 Dis- tortion (A) use standard RMSD and LDDT. Token Efficcency uses Codebook Utility Rate (UR), Perplexity (details in App.D) D and Small Structure Language Model Evaluation (SSL.MEvvll) (details in App. SSLM-Eval compares tokenizers (GEOBPE vs VQ-VAEs) via integration with a small 7.3M Transformer architecture after respectively tokenizing the pretraining data splits (Algo. 2). Under the same data, model, training and compute resources, the respective models generate new sentences, detokenizes them into structures, and we compare relative generation metries (Algo. 3
5). Downstream Transfer covers 12 tasks (24 test splits) using AUROC (%) for functional site prediction, Spearman's 's p (%) for flexibility prediction, and Macro F1 (%) for fold classification.
Expert Agreement measures Domain & Segment Recall/Precision/ I/IOU (details in App. G.
Computational Complexity/ / Implementation Details. We analyze the theoretical complexity of GEOBPE in App. and justify the steps we took towards efficient implementation and use.
5 RESULTS
Tokenizer Performance. We find GEOBPE and ProToken form the Pareto front under both A E LDDT). GEOBPE achieves 0.271 - 0..38xx and 0.016 1..01-0.022 0.021x the BPR of Token and ESM3, dropping LDDT by only 18 -2%% and 22 - 25%, which are impressive feats considering GEOBPE's training data was only 7% and 0.02% the size. We also observe GFOBPE's strong OOD generalization, with test/train RMSD peaking at 1.16 (CAMEO) and 1.28 (CASP), show- ing negligible degradation reconstructing unseen data; VV-VAE/AminoASeedd using identical data splits, show degradation as high as 6.4x test RMSD. Crucially, as the GEOBPE codebook grows, the variants trace a near-lineat path along the Pareto front toward ProToken, elastically trading off BPR for lower distortion, feature other tokenizers do not have (as codebook dimensions are fixed).
Token Efficiency. We report UR & Perplexity averaged over held-out test sets to gauge code- book/vocabulary usage on unseen data, the setting where the tokenizer is deployed. In Table I0] we see all methods except VQ-VAE and ESM3 achieve an average UR of > 40%; all except VQ- VAE achieve 0.2 average Perplexity. An ideal tokenizer avoids codebook collapse, but exactly uniform token usage may not be desirable. We introduce SSLM-Eval to stress test whether code- book efficiency actually leads to generative efficiency. M-al is holistic way to compare
8

---

Published as a conference paper at ICLR 2026
RMSD BPR
LDDT Vs BPR
RMSD CAMEO (test)
ESMB (236M)
5
ProTakon (552K)
Parwio Front (CAMES, of
0.9
Paralo Front ICASP14, CI
G40K)
0.8
Coo BPE (40K)
RMSD
Amiottrred (036)
AminaASced
VQ-VAE (40K)
LDDT CAMEE ttest)
Geo-drt (601)
CASP14 (best)
(552K)
0.5-
Pareto Front (CASP14, DI
0.4
10
10
10
107
paoettheces
BPR
Figure 4: Plots of (BPR(T: D), A(T; across tokenizers for A E {RMSD, LDDT). We vary [V] {128, 256, 512, 1024) for VQ- VAE/AminoASced and [V] E {600, 2500, 6000, 21000} for GEOBPE sample multiple points; we observe GEOBPE sweeps smooth tradeoff curve. Hyperparameters in App.
tokenizers using both encoder token efficiency and decoder's generative efficiency. In Table find GEOBPE-TRANSER
we
is capable of generating 99% unique and designable backbones, achiev- ing up to 49% higher scTM and maintaining higher diversity than both VQ-VAE methods using the same data splits. We visualize some realistic, novel backbones GEOB PR-TSRAAEE generated in App. E.4 Interestingly, the "less-efficient" VQ-VAE generated 58% more diverse backbones, demonstrating uniform token usage can be counterproductive to language modeling.
Downstream Task Transfer. In Table T we see GEoBPE-induced features rank first, on average, across both function and structure property prediction tasks. The relative performance gaps 15.4 15.44% and 43.28% quantify the add-on benefits of GEoBPE-induced features. GEoBPE-induce features reverse the trend that discrete PSTs produce less informative representations for downstream tasks (due to quantization-relane discrete vocabularies raises the ceiling on downstream transfer.
issues (Yuan et al. 2025).. ) highlighting that hierarchical structure from
Further Ablations. We include a comprehensive series of ablation studies in App. A demonstrating GEoBPE's data-efficiency, GEOBPE- TRANSFERR 's task-agnosticism, GEOBPE tokenizer's s scala- bility, GEOB PE tokens' adaptive resolution over iterations, and performance vs runtime tradeoff's in components (I), (3) & (4). Key findings include: (i) GEOBPE shows better OOD general- ization when fitted on 1 % training data; (ii) GLOBPE-TRANSTER predictions are no worse when GEOBPE was fitted with (a) 1% of the pretraining PDBs, (b) the downstream task-specific PDBs; (iii) GEOBPE performance gains diminish beyond Maax 5000 randomly sampled motif occur- rences used extract prototypes, taming a complexity term that depends on Mmax-
DISCUSSION
Case Study: Agreement with PFAM Annotations. We ran CATH Functional Families (FunFams) DDs et al. 2015b) to obtain domain boundaries and compared them against GEoBPE-derive mo- tifs. Because sequence conservation is linked to structural preservation, expect overlap between predicted motifs and functional domains. In Table 2 GEOBPE achieves 99.97% domain recall with mean F1 0.996 and IOU 0.992, showing near-perfect agreement across 10 datasets. The agreement is not only geometric but also finctiooal:: GEOBPE tokens frequently coincide with boundaries of ligand-binding grooves, transmembrane cavities, and scaffolding helices, capturing motifs that underlie molecular recognition and catalysis. This suggests GEOBPE does more than segment folds consistently: it surfaces interpretable structural primitives that map onto biochemical roles, offering a functional vocabulary absent in prior PSTs. Details are in App. G
Case Study: Human Expert Analysis of Interpretability. We conducted three expert evaluations of GEOB PE-derived hierarchies (App. H Across proteins, the discovered motifs align with func- tionally meaningful substructures, including regions mediating ligand binding. molecular recogni- tion, and structural gating. In the SLC25A20 transporter (Fig. 9). GEOBPE isolates a transmem- brane binding cavity formed by helices and polar residues. In the 14-3-3:Tau complex (Fig. [T), it identifies canonical phospho-binding groove stabilized by charged side chains. Recurrent local
9

---

Published as a conference paper at ICLR 2026
Table 1: Downstream transfer performance benchmark. We underline and bold the best continuous and discrete PSTs, respectively: indicates the best method across both. The relative performance v.s. ESM3 for GEOBPE- TRANSFER is included. Omitted rows in Table 66] GEOBPE hyperparameters are in App. K
Task
Continuous PST
Discrete PST
ProcinMPNN
MIP
FoldSeck
ProTokems
ESM3
VamilaVQQ
AminoAsced
GEOBPE TRANSSFE (v.. ESM3)
Functional Site Prediction (AUROC%)
Bindlat
Fold
51.83
50.38
53.18
44.66
47.25
47.11
SupFum
$4.00
94.36
46.26
$6.05
90.77
$6.71
90:53
91.31 (+0.59%%
Bindélo
Fald
79.42
33.79
32.37
28.47
02:34
02.112
60.73
94,94 (1554995)
SupPam
$1.00
$7.27
52.64
00,47
65:22
62.92
68.30
95.94 (+-47, (+-47.10%
BindShake
Org
75.52
79.900
53,43
59.10
66.10
67.04
09.61
87.73 (+32.72%)
Cainnt
Fold
61.05
35.62
53,43
58.16
61.09
58.89
62.19
66.21 (+8.38%)
SupFum
933.00
51.41
83.85
89.82
(-1300))
CarBio
Fold
56.35
67.68
65.33
65.95
(145,4395)
SupPam
08.19
06.97
53.78
64.05
70.92
+28.47%
Con
Fold
57.18
43
49.20
57.20
55.22
$6.98
57.23
71.96 (130332))
SupPum
34.68
52 00
51.31
70.64
80.53
74.60
86.60
84.84 (+5.35%)
22 aaddy welited (Rop, Apt)...
Average AUROC%
25.92
79.82
$1.90
65.37
69:24
68,30
72.43
80.20 (+18.11))
Physloochemical Property Prediction (Spearman's p%)
Fold
60.37
59 60
15.35
13.81
44.5 53
44.22
44.63
40.89 17%)
SupFum
39.23
56.80
11,99
7.62
39.08
38.98
40,99
47.17 (20.70%)
(FlcuRFactor FleNN50)....
Average p%
52.73
7,80
37.35
33.49
48.26 (821.18%)
Structure Property Prediction (Macro F1%)
Homo
Fold
66
11.57
5.84
30092
20.87
23,60 (-2) (.99))
SupPum
30.83
33,86
4.67
6.17
24.89
22.10
38.38
47:28 (+89.96%)
Fam
63.35
74.22
15.34
18.35
54.42
47.18
69:78
85.75 (4.7.447%)
Average Macro F1%
39.94
43.58
10.51
10.11
36.44
29.15
46.01
$2.31 (+43.28%)
Table 2: We annotate 100 PDBs from each dataset and report % of 1,000 random equal-length segmentations
that GEOBPE matches or outscores. Omitted columns in Table 7 Secondary structure analysis in Table TS
Bindlnt
BindBio
BindShake
Cailat
CatBio
Con
Average
Domain
Mean Recall
99.95 (98.35)
100 (100.0)
100 (3000)
100 (1000)
99.99 (93.55)
99.95 (98.0)
99. 99.97 (97.97)
Mean Procisiio
98.
99.62 (71
(68.24)
(42
99. (54.
Mean
99.42
99.81 (83.63)
99.88 (76.49)
(76.82)
Mean IOU
98.86 (86.32)
99:62 (83.63)
98.76 (76.54)
99.28 44)
99.52 (60.94)
99.14 (36.97)
99.22 (76.75)
Segment
Moin Recall
(100.0)
(100.0)
100 (100.00 0)
100 (100.0)
100 (100.0)
(100,0)
100.00 (100.00)
Mean Procisiio
97. (72.66)
81. 84 (61.82)
97. (68.33)
90.4 (63.00)
98.87 (74.76)
98.92 (92.0)
95 (65.62)
Mcan
(72.04)
89.05 (61.82)
98.8
94.04 (63.09)
99.45
97.23
motifs (aromatic cages, polar bridges, helix-helis clamps) are combined into higher-order scaffolds that mirror established biochemical organization. These hierarchies capture geometric regularities and also modular design principles conserved across folds and families. Even in compact domains, such as mucleotide-recognition modules, GEOBPE motifs reveal the coupling between geometric curvature and chemical specificity, meaning that GEOBPE surfaces reusable motifs that are both interpretable and evolutionarily grounded.
Limitations. GEOBPE currently does not incorporate sequence or side chains, but can via direct extensions, e.g. taking the Cartesian product of the current vocabulary with amino acid types and augmenting the backbone formulation with type-dependent x-angle spans. The present integration with small-scale Transformers is set up compare tokenizers' compability with language modeling on structure tokens; only relative backbone design metrics are relevant. Generative performance de- pends on model capacity and data scale, which are orthogonal to the tokenizer. In the separate SSLM scalability study, we see steep gains in generative performance when both LM parameter count and pretraining data increase ten-fold, conforming with scaling law expectations. The improved num- bers are preliminary evidence of GEOBPE's promise tokenizer for lage-scale PLMs, but are not competitive with state-of-the-ar backbone design models.
7 CONCLUSION
We present GEOBPE, a principled geometry-grounder analog of BPE for protein folds. GEOBPE (a) captures natural conformational variability in protein backbones, (b) constructs a hierarchical vocabulary of structural motifs, and (c) produces hierarchical views of folds for downstream repre- sentation learning. Its hierarchies reveal conserved modular design principles that connect structure to function. Empirically, GEOBPE advances the state of the art in tokenizer performance, out-of- distribution generalization, token and generative efficiency, downstream transfer, and interpretabil- ity. These results establish GEOBPE as foundation for structure-native protein language models.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Code for GEOB PE and steps to reproduce all experiments in the paper are available at htthtts://github com/SiningammylalyP--PPP
We include detailed descriptions for understanding our method in the main text, with mathematical descriptions and pseudocodes in the Appendix. In App. K list the key hyperparameters, their effects on algorithm behavior, the default values used in our experiments, and any deviations from the default values used to obtain the results reported in the main text. In App- JI we analyze the computational complexity of our method and describe practical implementation choices used to make the method efficient in practice.
ACKNOWLEDGMENTS
M.S. and M.Z. gratefully acknowledge partial support by NSF CAREER Award 2339524, ARPA-II Biomed- ical Data Fabric (BDF) Toolbox Program, Amazon Faculty Research, Google Research Scholar Program, As- traZeneca Research, GlaxoSmithKline Award, Roche Alliance with Distinguished Scientists (ROADS) Pro- gram, Sanofi iDEA-iTECH Award, Bochringer Ingelheemm Award, Merck Award, Optum Al Research Collah- oration Award, Plizer Research, Gates Foundation (INV-079038), Chan Zuckerberg Initiative, Collaborative Center for XDP at Massachusetts General Hospital, John and Virginia Kaneb Fellowship Harvard Medical School, Biswas Computational Biology Initiative in partnership with the Milken Institute, Harvard Medical School Dean's Innovation Fund for the Use Artificial Intelligence, and Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. Any findings, conclusions recommendations expressed this material are those of the authors do not necessarily reflect the views of funders.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630(8016):43-500,, 2024.
Gustaf Ahdritz, Nazim Bouatta, Christina Floristean, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy J O'Donnell, Daniel Berenberg, lan Fisk, Niccolò Zanichelli, et al. Open- fold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. Nature methods, 21(8):1514-152, 2024.
Inigo Bario-Hernandez, Jingi Yeo, Jürgen Jänes, Milot Mirdita, Cameron LM Gilchrist, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger. Clustering pre- dicted structures at the scale of the known protein universe. Nature, 62277983).637-655, 2023.
Matthias Blum, Antonina Andreeva, Laise Cavalcanti Florentino, Sara Rocio Chuguransky, Tiago Grego, Emma Hobbs, Beatriz Lazaro Pinto, Ailsa Orr, Typhaine Paysan-Lafosse, Irina Pona- mareva, et al. Interpro: the protein sequence classification resource 2025;? mode longmeta? Nucleic acids research, 53(D1):D4444D455 2025.
Carl Ivar Branden and John Tooze. Introduction protein siructure. Garland Science, 2012.
AC Camproux, P Tuffery, JP Chevrolat, JF Boisvieux, and S Hazout. Hidden markov model ap- proach for identifying the modular framework of the protein backbone. Protein engineering, 12 (12):1063-1073, 1999.
Anne-Cloude Camproux, Romain Gautier, and Pierre Tuffery. A hidden markov model derived structural alphabet for proteins. Journal of molecular biology, 339(3):591-605, 2004.
Henriette Capel, Robin Weiler, Maurits Dijkstra, Reinier Vleugels, Peter Bloem, and K Anton Feen- stra. Proteinglue multi-task benchmark suite for self-supervised protein modeling. Scientific Reports, 12(1):16047, 2022.
The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2025. Nucleic Acids Research, 530DI): 11 2024. ISSN 1362-4962. doi: 10.1093/nar/gkae 1010. URL http:://dbh.ogg///0000222haaaaaa

---

Published as a conference paper at ICLR 2026
Sayoni Das, David Lee. Ian Sillitoe, Natalie L Dawson, Jonathan G Lees, and Christine A Orengo. Functional classification of cath superfamiliies: a domain-basee approach for protein function annotation. Biotnformatics, 31(21):3460- 3467. 2015a.
Sayoni Das. Ian Sillitoe, David Lee, Jonathan G Lees, Natalie L Dawson, John Ward, and Chris- tine A Orengo. Cath funfhmmer web server: protein functional annotations using functional W153, 2015b.
family assignments. Nucleic acids research,
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 378(6615):49 56, 2022.
Alexandre G de Brevern. New assessment of a structural alphabet. In silico biology, 5(3):283-289, 2005.
Alexandre G de Brevern, Catherine Etchebest, and Serge Hazout. Bayesian probabilistic approach for predicting backhone structures in terms of protein blocks. Proteins: Structure, Function, and Bioinformatics, 41(3):271-287 2000.
Alexandre G de Brevern, Hélène Valadié, Serge Hazout, and Catherine Etchebest. Extension of local backbone description using a structural alphabet: a new approach to the sequence-structure relationship. Protein Science, 11(12):2871 2886, 2002.
Edo Dotan, Gal Jaschek, Tal Pupko, and Yonatan Belinkov. Effect of tokenization transformers for biological sequences. Bioinformatics, 40(4):btae1 196, 2024.
Janani Durairaj, Mehmet Akdel, Dick de Ridder, and Aalt DJ van Dijk. Geometrieus represents pro- tein structures shape-mers derived from moment invariants. Bioinformatics, 36(Supplement i718-i725, 2020.
Sam Gelman, Bryce Johnson, Chase R Freschlin, Arnav Sharma, Sameer D'Costa, John Peters, Anthony Gitter, and Philip Romero. Biophysics-based protein language models for protein engineering. Nature Methods, pp. 1-12, 2025.
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model. Science, 38766736)8500-888 -58088555888800 2025.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, and Burkhard Rost. Bilingual language model for protein sequence and structure. NAR Genomics and Bioinformatics, 6(4):laac 2024.
Ioan Ieremie, Rob M Ewing, and Mahesan Niranjan. Protein language models meet reduced amino acid alphabets. Bioinformatics, 40(2):btae061, 2024.
Andriy Kryshtafovych, Torsten Schwede, Maya Topf, Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)-round xiv. Proteins: Structure, Function, and Bioinformatics, 10(12)::607-166 2021.
Tim Kucera, Carlos Oliver, Dexiong Chen, and Karsten Borgwardt. Proteinshake: Building datasets and benchmarks for deep learning on protein structures. Advances in Neural Information Pro- cessing Systems, 36:58277-58288 2023.
N Jesper Larsson and Alistair Moffat. Off-line dictionary- -base compression. Proceedings of the IEEE, 88(11):172--1732 2002.
Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Pan Tan, and Liang Hong. Prosst: Protein language modeling with quantized structure and disentangled attention. Advances in Neural Information Processing Systems, 37:35700-35726, 2024.
Xiaohan Lin, Zhenyu Chen, Yanheng Li, Zicheng Ma, Chuanliu Fan, Ziqiang Cao. Shihao Feng, Yi Qin Gao, and Jun Zhang. Tokenizing foldable protein structures with machine-learnee artificial amino-acid vocabulary. bioRxiv, pp. 2023-11. 2023a.
12

---

Published as a conference paper at ICLR 2026
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-1111 2023b.
LeAnn M Lindsey, Nicole L Pershing, Anisa Habib, Keith Dufault-Thompee W Zac Stephens, Anne J Blaschke, Xiaofang Jiang, and Hari Sundar. The impact of tokenizer selection in genomic language models. Bioinformatics, pp. btaf456, 2025.
Craig O Mackenzie. A tertiary alphabet for the observable protein structural universe captures sequence-structuue relationships. Protein Sci, 25:75, 2016.
Swapnil Mahajan, Alexandre G de Brevern, Yves-Henri Sanejouand, Narayanaswamy Srinivasan, and Bernard Offmann. Use of structural alphabet find compatible folds for amino acid quences. Protein Science, 24(1):145-153: 2015.
Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36:431777-4322 2023.
Gregory A Petsko and Dagmar Ringe. Protein structure function.. New Science Press, 2004.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song, Evaluating protein transfer learning with tape. Advances in neural infor- mation processing systems, 32, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Out, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, (18(15)c2011233111 2021.
Xavier Robin, Juergen Haas, Rafal Gumienny, Anna Smolinski, Gerardo Tauriello, and Torsten Schwede. Continuous automated model evaluation automated evaluation structure prediction methods. Proteins: Structure, Function, and Bioin- formatics, 89(12):1977-1988 2021.
the future of fully
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-awarr vocabulary. BioRxiv, pp. 2023-10, 2023.
Burak Suyunu, Enes Taylan, and Arzucan Özgür. Linguistic laws meet protein sequences: A com- parative analysis of subword tokenization methods. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1489-4496. IEEE. 2024.
Yang Tan, Mingchen Li, Ziyi Zhou, Pan Tan, Huiqun Yu, Guisheng Fan, and Liang Hong. Peta: evaluating the impact of protein transfer learning with sub-word tokenization downstream applications. Journal of Cheminformatics, 16(1):9,,002)
Brian Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206. 19, 2022.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Michel Van Kempen, Stephanie Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein struc- ture search with foldseek. Nature biotechnology, 42(2):243-246, 2024.
Yann Vander Meersche, Gabriel Cretin, Aria Gheeraert, Jean- -ha--hhrsteophe Gelly, and Tatiana Ga- lochkina. Atlas: protein flexibility description from atomistic molecular dynamics simulations. Nucleic acids research, 52(DI): 2024.
13

---

Published as a conference paper at ICLR 2026
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of rroeinnsequence space with high-aceuracy models. Nucleic acids research, 50(D1):D439-D444 2022.
Iyanar Vetrivel, Swapnil Mahajan, Manoj Tyagi, Lionel Hoffmann, Yves-Henri Sanejouand, Narayanaswamy Srinivasan, Alexandre De Brevem, Frédéric Cadet, and Bernard Offmann. Knowledge-based prediction of protein backbone conformation using a structural alphabet. PloS one, 12(1 xe011621155 2017.
Kevin E Wu, Kevin K Yang, Rianne van den Berg, Sarah Alamdari, James Y Zou, Alex Lu, and Ava P Amini. Protein structure generation via folding diffusion. Nature communications, 15(1):
1059, 2024.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07 2022.
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36:gzad015, 2023.
Xinyu Yuan, Zichen Wang, Marcus Collins, and Huzefa Rangwala. Protein structure tokenization: Benchmarking and recipe. arXiv preprint arXiv:2503. 00089, 2025.
Baoquan Zhang, Huaibin Wang, Chuyao Luo, Xutao Li, Guotao Liang, Yunming Ye, Xiaochen Qi, and Yao He. Codebook transfer with part-of-speech for vector-quantized image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
7757-7766, 2024a.
Chengxin Zhang. Xi Zhang, Lydia Freddolino, and Yang Zhang. Biolip2: an updated structure database for biologically relevant ligand-pue protein interactions. Nucleic Acids Research, 52(D1): D404-D412, 2024b.
Jiayou Zhang, Barthelemy
James Gong, Xingyi Cheng, Yingtao Luo, Hugo Ly, Le Song, and Eric Xing. Balancing locality and reconstruction in protein structure tokenizer. bioRxiv, pp. 2024-12, 2024c.
Zhihan Zhou, Yanrong Ji, Weijian Li. Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert- 2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306. 15006, 2023.
14

---

Published as a conference paper at ICLR 2026
A ABLATION STUDIES
GEOBPE is task-agnostic, and using task-specific data does not increase downstream perfor- mance of GEOBPE-TE CEPPE-TRANSEER For each task i, let Thask be a tokenizer fitted using only Disin (with its own vocabulary V, but the same feature extractor Fe), and define rv, (x) yy (x(x)) We follow the same downstream transfer evaluation. We find an interesting result in Table 3 where directly training on the task-specific dataset does not meaningful change downstream prediction re- sults. A closer look reveals the underlying reason is because the individual tokens do not differ sig- nificantly; motifs added to V. in order, are similar across both GEOBPE-TRANSEEE TRANSFER (task-specific). We can interpret this both positively and negatively. GEOBPE is in- sensitive to task-specific data and learns the "language" of protein folds consistently. This may be desirable for reusability of tokenizer, as one does not need to retrain it for different data distribu- tions, as all protein folds obey the same universal principles (Petsko & Ringe 2004). At the same time, this upper bound tests whether the tokenizer tailor its vocabulary to individual datasets for potentially higher scores, indicating GEOBPE by itself may lack the parameter capacity to overfit to individual tasks.
and GEOBPE-
Table 3: GEOBPE-TRANSFER TRANSFER (1%) runs Algo. with 1% of the pretrain training set, then uses the output vocabulary to induce features; GEOBPE- TROOBE-TRRNSEEE TRANSERR (task (task-specific) does not use pretraining data; instead it runs Algo. Twtt downstream data to learn a vocabulary. All use default value parameters in App.
Tyy
4731
Phouttebenttal Proprte Predikke ('pcammnn' (53)
Stradan Freperty Productes (150
FarRUSF (840 FaNRUSF
FastFarr RickFate FleNRQ 1F44 PlNNii 19
Rewn/Fll Roan (eyFoa
ReeiiFee
BY $3.17
7738
91
19:40
25.60
4738
05.15
10.30
GEOBPE-TRANSEER maintains comparable downstream transfer performance even when GEOBPE was fitted on 1% of pretraining data. In Table 3 we see GEOBPE fitted on just 1% of the pretraining data is enough to transfer, on average, the same amount of performance downstream GEoBPEfitted on the full dataset. There are no meaningful differences between GEOBPE-TRANSER and -RRNSSEERR (1%), with GEOBPE-TRANSEER doing 1.7% bet- ter on physicochemical property prediction and GEOBPE-TRANSFER (1%) doing better 0.2% better on functionals ite prediction. These findings can be interpreted both positively and negatively for GroBPEleamed vocabularies: (1) they are extremely informative, learning useful signals to trans- fer downstream with as few as 300 PDB structures; (2) they underfit the data, with no noticeable improvements from using more data to learn the tokenizer. Taken together, these findings imply GEOBPE is a lightweight add-on on top of any pretrained features θ but feeding more data to GEOBPE yields diminishing downstream returns quickly.
GRONPE-TRANSEER does not underfit the data for structure-related downstream tasks. In we see for physicochemical (residue-level regression) and fold-level tasks, the model in- dicates clear propensity for more training data, with step-wise gains for every 20% of training data. 20% + 100% training data sees performance lift significantly (+18.89% average p% and +45.35% respectively). For residue-level classification tasks, the lift is only marginal (+1.28%). We hypothesize the cause is not limited capacity, but rather that the tasks are localized label pre- dictions: residue-level receptive field is sufficient when features are informative. This shows GeoBPE's data-efficiency at learning a vocabulary does not limit its capacity on downstream tasks that require multi-scale resolution (e.g. structural flexibility or fold-level classification). which ex- isting fixed-size tokenizers cannot at both residue and structure-level. Thus, the data-efficiency strengths of GeoBPE training orthogonal to downstream modeling. The structure-related tasks in Table 4 see large gains in performance with more training data, implying GEOBPE-TRANCE scales to complex, hierarchical structural patterns that can only be learned from more data.
Table 4
GEOBPE is data-efficient OOD, but more training data can lower training distortion. In
Table 5 we see GEOBPE (1%) consistently achieves lower distortion (L 7.7% RMSD averaged, t 0.06 LDDT summed across all four runs and both test splits) than GEOBPE. This suggests small, well-chosen set of structures is enough for GEOBPE to achieve superior reconstruction on OOD structures, and more training data can introduce noise and hinder generalization. However, GEOBPE (1%) obtains 1 3.1% averaged, + 0.02 LDDT summed across all four runs on the re- spective training splits. A lower RMSD suggests GROBPE better captures global structural fidelity;
15

---

Published as a conference paper at ICLR 2026
TTbble 4: We
from Table (reported 100%) then vary only the percent of downst- ream task training data available, fixing same valid and test sets.
Task
Split
GEOBPE- GEOBPE-TRANSER TRANSFER
20%
40%
60%
80%
100%
Functional Site Prediction (AUROC%)
BindInt
Fold
58.78
61.18
59.79
59.45
59.19
SupFam
89.55
89.88
90.42
90.71
91.31
BindBio
Fold
94.86
94.82
94.96
94.97
94.94
SupFam
95.69
95.79
95.88
95.97
95.94
BindShake CatInt
Org
87.40
87.59
87.65
87.64
87.73
Fold
64.66
65.55
66.94
66.29
66.21
SupFam
87.96
88.27
88.78
88.73
88.65
CatBio
Fold
94.95
94.92
94.94
94.96
95.01
SupFam
95.67
95.81
95.96
95.97
95.90
Con
Fold
71.42
71.37
71.72
71.74
71.96
SupPam
83.77
84.02
84.76
84.76
84.84
Rep
Fold
55.04
54.25
56.02
56.41
56.44
SupFam
73.24
75.18
75.89
71.58
72.98
Ept
Fold
63.63
53.28
61.72
61.66
64.78
SupFam
71.21
49.39
73.59
76.01
77.06
Average AUROC%
79.19
77.42
79.93
79.79
80.20
Physicochemical Property Prediction (Spearman' 's p%)
FlexRMSF
Fold
41.49
43.33
38.48
39.02
40.89
SupFam
34.74
45.41
44.61
47.06
47.17
FlexBFactor
Fold
23.90
27.86
33.83
34.82
37.28
SupFam
23.80
25.46
37.70
36.49
35.61
FlexNEQ
Fold
54.40
56.07
55.98
57.53
56.65
SupFam
51.12
53.77
52.52
54.96
53.98
Average 9%
38.24
41.98
43.85
44.98
45.26
Structure Property Prediction (Macro F1 %
Homo
Fold
14.35
20.55
23.74
24.25
23.60
SupFam
27.63
35.11
43.09
43.98
47.28
Fam
65.79
73.66
80.77
82.04
85.75
Average Macro F1%
35.92
43.11
49.20
50.09
52.21
Table 5: We rerun the GEOBPE experiments used to trace out the Pareto Front in Fig. by using 1% of pretraining data. We include raw numbers of Fig. 4 (bottom rows) for comparison. All other hyperparameter settings are kept the same (App. K).
Train
Valid
CAMEO
CASP14
RMSD
LDDT
RMSD
LDDT
RMSD
LDDT
RMSD
LDDT
GeuBPE (1%) ([V)
600)
1.72
0.74
1.63
0.73
1.66
0.73
1.53
0.72
GeoBPE ([V) 600)
1.66
0.73
1.71
0.72
1.77
0.72
1.53
0.72
GeoBPE (1%) (VV 2278)
1.57
0.75
1.51
0.71
1.51
1.43
0.73
GeoBPE (V) 2500)
1.41
0.74
1.57
0.73
GeoBPE (1%) ([V)
5278)
1.36
0.77
1.34
0.76
1.35
0.75
1.30
0.74
GeoBPE (IV) 6000)
1.37
0.76
1.46
0.75
1.52
0.74
1.54
0.72
GeoBPE (1%) (VV) 20278)
1.29
0.77
1.28
0.76
1.28
0.76
1.37
0.73
GeoBPE (VV 21000)
1.21
0.77
1.28
0.76
1.40
0.75
1.55
0.72
by constructing the vocabulary from the full pretraining dataset, it can choose more representative prototypes; hence, its vocabulary better preserves global fold. Meanwhile, a slightly lower LDDT indicates GEOB PE (1%) can capture few local details in the 1 % subset of structures better than
16

---

Published as a conference paper at ICLR 2026
GEOBPE. This suggests GEOBPE (1%) is more sensitive the individual local interactions of the small set structures it fitted with: GEOBPE considers vastly more structures. summary, GEOBPE is preferred for fold preserving compression of whole datasets, but GEOBPE (1%) can be feasible if not superior when GEOBPE is primarily used to tokenize unseen data.
LGGT
6P1
11P Se0ae PARE
Bucdoce RPS0
sorkhao prrss
LDGT
300
630
130 M6 246 100 369 40 44 439 s/5 569 609 Teoch 10 Marry waudds
200
200
400
06
Figure 5: We plot the BPR (purple), length (orange), backbone distortion (RMSD, LDDT) [V] across BPE steps. Ref. backbone RMSD/LDDT (dotted lines) random angle values for all internal angles, sampled from empirical angle distribution.
GEOBPE is multi-resolution, revealing finer details as more tokens are introduced. In Fig. 5 we run moarse-graines version of GEOBPE (small initial (V]) to observe interesting feature of GEOBPE's design. As newly introduced tokens re-quantize the occurrences from the original data (span gathering step in Alg. [ tokenization can adaptively increase the resolution if the new prototypes better capture the modes of variability for those occurrences than their previous quan- tization. We expose this via hyperparameters bins & num-p (see App. K). which tradeoff the super-resolution effect against coarse- graining effect at different token sizes, offering fine- grained control.
(a) Quantized backbone (GLoBPE)
(b) Original backbone
(c) Quantized backbone (GEOBPE- Glue Opt)
Figure 6: We ablation for GEOBPE version with |V] 600, keeping all parameters the but loggling whether glue opt is skipped in Alg. [8
visualize the original (center), GEOBPE (left) and
GEOBPE without glue opt (right) backbone states.
Rigid body refinment optimization from Algs 18 and 9 altogether, we see the effects in Fig. 6 For that experiment, we find avg. RMSD increase 1.66 + 4.39, and avg. LDDT drop 0.73 + 0.69 when glue opt is turned off. Rigid body refinement preserves the overall fold and modular architecture; turning it off causes individual domains distort the parallel strands drift apart as well as the overall configuration to lose its integrity. Over the course of many time steps, global drift accumulate as local rounding occurs. Rigid body refinement an indispensable subroutine for ensuring the overall quantization faithfully reproduces the fold integrity.
essential step for preserving fold integrity. If omit the glue
Increasing Mmax beyond a certain threshold does not yield additional distortion benefits. We did study comparing GEOBPE (V 6000, Maax 5000, full settings in App. K) with "higher- resolution" settings bins + {1 5000}. Mmax + 20000. Interestingly, we found overall RMS- D/LDDT did not improve (1.40 1.39, 0.76 0.75, both in favor of the incumbent) despite
17

---

Published as a conference paper at ICLR 2026
increased computational expenditure spent Alg. 8 The most likely explanation is there is marginal utility increasing Mmax beyond 5000, and differences distortion rates are likely due to the numerical stability of Alg. 12 more so than the hyperparameters.
B DATASET DETAILS
Training, For training GEOBPE, we started with the pretraining data splits released by Yuan et al.
2025), which follows the same criteria used train the OpenFold2 model Ahdritz al. (2024)
For VanillaVQ and AminoASeed baselines, we use the same splits as Yuan et al. (2025) directly.
For GEOBPE, we further filtered the data down to only ones with complete backbone information (e.g. backbone dihedrals are not NaN, each residue contains N, CA and C), resulting in 34818
structures. We further excluded structures shorter than 40 longer than 512 residues, resulting in 33992 structures for training GeoBPE and 3810 for validation (only used for
Held-out testing. We use CAMEO and CASP14 4 test sets for evaluating the generalization of tok- enizers (Robin et al. 2021 Kryshtafovych et al. (021). For CASP14, we follow Yuan et al. (2025) and select only proteins released after the pretraining data cutoff date.
Downstream Tasks. Our 8 downstream tasks cover breadth of structure and function-related predictions. They are divided into 3 categories and are assembled from sources: InterPro (BindInt,
Con, Rep) (Blum et al.] (222)), BioLIP2 (BindBio, CatBio) (Zhang al. 2024b). ProteinShake
(BindShake) (Kucera et al. (023), ProteinGLUE (Ept) (Capel et al. (2022). TAPE (Homo) (Rao
al. 2019) and ATLAS (FlexRMSF, FlexBFactor, FlexNEQ) (Vander Meersche al. 2024).
1. Functional site prediction: Binding site prediction (BindInt), catalytic site (CatInt), conserved site prediction (Con), repeat motif prediction (Rep), epitope region prediction (Ept)
Physicochemical property prediction: Structural flexibility prediction, measured using metric RMSF (FlexRMSF), B-factor (FlexBFactor) and Neq (FlexNEQ)
3. Structure classification (protein-level): Remote homology detection (Homo)
Functional site prediction tasks predict whether each residue is site of functional importance (binding, catalytic activity or antibody recognition) or part of an evolutionary motif (conserved site or part of repeated motif). PSTs which learn semantically meaningful signals like motif boundaries are expected to perform well on these tasks.
Physicochemical property prediction tasks predict the flexibility of each residue as continuous value. Higher flexibility can be a clue that the residue may be more amenable to functional activity. PSTs that capture a fine-grained view the localized protein dynamics are expected to predict residue-level flexibility well.
Remote homology detection a multi-clas fold classification problem. Proteins which belong to the same fold class be distantly related or share similar functions the whole. Therefore, PSTs that capture the overall fold-level geometry are expected to do well on this task.
For more dataset statistics and preparation details, Yuan et al. 2025
C ADDITIONAL RESULTS
Table Additional downstream transfer performance tasks. Setup follows Table
Thsk
Split
Continnous PST
Discrete PST
ProteinMPNN
ProTokens
VanillaVQ
AmisoAiiced
CHOOBPE-TRNNSNE FSM3)
Functional Site Prodiction (AL (OCC%)
Rep
Fold
77.63
74.53
47.71
53.20
74.70
75.99
74.97
56.44 (-24.46%)
SupFam
80.71
77.25
82.09
72.98
Ept
Fold
62.84
54.56
52.49
$9.28
64.7% (+1.711)%
SupFam
64.84
$2.98
50.53
61.92
61.97
67.24
72.02
77.06 (+24.35)%
Phesiccceii Property Prediction (Spearman's
FlexBFactor
(+57.97%)
SupFam
34,56
35.23
5.17
25.80
23.73
21.76
35.61 (+38.00
FlexNEO
Fold
69.69
65.32
5.71
12.98
45.05
15.95
49.64
$6.65 (+25.75%
SupFam
68.69
64.82
2.66
10.51
35:45
35.61
50.15
53.96 (+52.27%)
18

---

Published as a conference paper at ICLR 2026
Table 7: Additional expert agreement results. Setup follows Table 2
Rep
Ept
Atlas
Homo
Domain
Mean Recall
99.93 (99.34)
100 (100.0)
99.93 (90.47)
99.98 (100.0)
Mean Precision
99.2 (43.59)
99.75 (77.89)
98.44 (31.29)
99 (41.92)
Mcan F1
99.56 (81.86)
99.87 (82.68)
99.17 (67.37)
99.48 (79.07)
Mean IOU
99.12 (82.18)
99.75 (82.68)
98.37 (67.07)
98.98 (78.78)
Segment
Mean Recall
100 (100.0)
100 (100.0)
100 (100.0)
100 (100.0)
Mcan Precision
98.76 (61.93)
95.52 (60.8)
96.09 (47.29)
95.91 (54.13)
Mean F1
99.38 (61.93)
97.68 (60.8)
98 (47.29)
97.91 (54.13)
Main Text Tables. 6 contains additional tasks Rep, Ept, FleXRMSF, and FacxoFactor. Table contains additional task data Repeat, Ept. Atlas, Homo. Task abbrevations are defined in App. B
Table 8: Secondary structure clement (SSE) agreement results. Sctup is the same as in See. over 8 basic secondary structure building blocks. Annotations are obtained from DSSP.
but stratificd
Menic/
Bardlit
BiedBio
BiedShake
Cailet
CulBio
Repeat
Jelic
Home
Mean recall HI
2411 (M4 77h
31.15 C9T 475
50 (16 399
33-7 (M6 13
12.41 (96 (53)
29:11 (18. 32)
15:13
(97.2))
30:76 (M6. 399
31.97 (98.
52.53 (96. 455
Meat socall
19.15 (99.4)
31.13 (98.61)
30:87 (08 55)
34.56 (98. 75)
32.29 09: 71)
19.15 (09.4)
31.13 (58.61)
30.87 (0355)
34.66 (58.75)
32.29 (98.7)
Mon secall
(0:40:1000.0
14.75 (1001))
10.85 (7
17:50
20
10.8
10:42 (100.1)
10
(100/))
(100.0
10.8 (39 26:
(99
Meat
(98
(97
30.52 (97.51)
34.93
41.24
34.75 (58.40
36.17
30.52 (97.51)
(100.00
Moan socall
52.96 (98 343
33.06 (97.4)
52.26 (96 413
33.75 (95.67) 67)
32.91 (97 05)
92.76 (98. (4)
33.06 (97.4)
52.36 (96.4))
55.75 (56.67)
50.01 (90 (93.08)
Meaa recall 5
53,92 (19 323
33.29 (96.0)
52.91 (07 73)
36,03 (17. 49
35.12 0912 950
33,92 (99 32)
33.29 (95.00
32.51 (57 72)
36.01 (97.4)
53.12 (97 10.
Mist secall-
32.7 (97.84)
32.
(0.6)97.10 179
33.02 (96 164
34.71
5m)
32 (96 035
32.7 (97.81)
32.76 (97. 17)
53.02 (96. (6)
54.71 (56. 86
30.65 (96 03)
Meon grecison H
St. $1.34 (54.55)
34.00 (dh
31
561
33.33 04)
31.34 (54.99)
34:00 (008
31. 31.35 (51 221
(48 80)
51.33 (04)
Meas presison.
(79.42)
29.28 (65
28.05 (62.5 87)
30.3 07)
30.33 (6-32-6-77)
17.95 (79.42)
20 23 03)
28.95 (62.87)
32.3 (65.07)
30.33 (64.77)
precicen
67)
(45.6)
39.54 (47.56
13.99
Meeai precisae
Meat precisoc
Moon przcicon
28.59 (53 (3.66)
26.78 (36.59)
26.(74.00)
21.86 (71 (0))
28.55 (50.1)
25.28 (50.82)
2466 129
26
27.5 98
166
150
(4)
221
26.53
29.15
16.21
(52.53
(45 901
28.13 (49 359
25.0 (50.96)
21.
26.75
28.59
26.8 007
(56,59)
(53. 26.66
25.26
28.55 (50LI)
(10 K2)
015
34.66
27.88
049 221
(50.94)
29. (52.50
26.52 (45.5)
16.21
28.13 (48 36
511 699.69
25.0 (30.90
Mon presicin-
(54.17)
75)
149 444
24.00
(3)
09: (46 15)
27.07
171
26.5
(68.75)
38.07 m6))
(48.
H
(644
(02
549
32.79-1544 15)
100
30.7
(62.58)
34.0 45)
Moa
18.44
Moon
31.34 (59
18
50.55 (M0
10
31
10.50
50.36 (16)
Moat f B
40.21 (59.65)
33.58 (60.25)
55 (5) 19155
29.98 (63 (6)
33.0 (60.0) 015
40.21
(59.55)
13.85 (40 25)
55.25 (96.17)
29.04 663 681
13.59 (66.6))
Moon 11 B
50.57 (25 780
24.1 1 (19
(11 14
10
K 194 57)
74
16:67 C71 Kg
30.57
(75 26]
(71.05)
10 24.6 (71 04104
1100 014 571
74 67 (11 M0
Meat f) T
30:22 (62.79)
30.27 (61.7)
(63 m1)
20:39
(51-00)
20
(0 25
30.22
(62. 791
30.27 (63.3)
29.61 (03.41)
30.00 066 66.99)
29.55 (61 299
Monn S
29:34
76.04 18 T9)
24 (52 823
28 19 15 10
22 02:00-9-00 563
28:34
(91 77
23 46 521
39 29 900
(98
16
75)
Meai
(59-55)
29
250
157.96
Mon G
17:35 (83107)
28.67 (72.4))
28.59
$1.99 (5145)
20.00 (72.1.
17.35 (81.07
16.80 (72.455
09.59 (36.7)
31. 19 633.655
23.06 (72.1)
Meat los
10.09 (95
29.55 (59 091
16.52 111
10.31
194 (00
29 190 573
10:00 (15 KM)
20 50
16.52 (99.11)
10.51 (26.177
2467 (15.5))
Moat iva
39.1 (59.4))
32.76 (50
5414
(58
81)
28:00 (67 37)
32.0 (99.55)
30.1 (59.46)
22:26
941
54.14 (58.81)
29.02 063 37)
22.9 (99.55) 55)
Meeni ins
26.79 (741235
21 11.M 14h
20 00 11h
20.1
10 2311
21 109 714
26.7M 001
21 10
24 (01.0
26.21 600 55)
21 199 74)
Meat ios
(61.64)
23.26 (99.3)
27.99
(6) 243
23-
(12 30
27.92 (99. 16)
28.36 (61.54)
25,20 (59-8)
27.4
(61-71
25.9 162.322
27.92 159 (15)
Moon
149 0m
(9 16
51 185 151
05.555501 015
U1 115
25.55 151 $1
Sepmest H
29.25 (99.64)
33.14
34.14(99.58) 59h
0054 19
53. 199
53.07
(90 013
29.35
199.44
33.14
(99 585
36.94 651
37 096 09.055
33.07 196. 91)
Segneni rocall
19.18 (100.0)
$1.2.10000
50s (10000
54 59
(100 (
12.55
(99 85
19:16
(100 m
21.2
(100000
90:95
(100.1)
14.80 (1000) 05
51.37 (90 85
Seginest rccal 1
10:42 (100.09
30.79 (1000)
17:2 (10000
1009 1100 m
30.44 (10.0)
10:42 (1000) m1
30.79 (1000)
17.65 (100.10
10.01 (1008))
20.54 (100.00 06
Seymzzi rocall
(100.0)
36)
16 199.
20.
m
35.0 (99 (65
41.25 (1.0m)
24.6
(09 (200
36.22 :99 81
30.64 (10011
25.0 68)
Seginert racall
(96
(96 (73
31.3. 148-085
38.01 (90 (P
31.7 (MM
38,05 (M8.75
75
57)
Segenzal
meall
53.35 (95
31.00 (965))
12
155
15.00 166
52.5 17)
75)
36.66
57.57 (96
Sopansat rzcall
$3.29 (96.57)
32.04 99.35 550
33.07
54.57
198. 55.
52.72 (96.22 523
33.29 (88.55
12 19 35)
33,07 (96.
34.57 (96.331
52.72 (96.52
Sepanzsi geddsian H
39.24 (63.48)
34.23 (61.00 1000
(6)
29-45
(5R (90
33,02
:99 AT)
39.24
(61-48)
26.33
54.32 (61.8)
20.43
(58
33 (59.47)
Segmeni porision
2.64181 14)
(72 mm
4,16 (20 (44
2.44 (9 ST
44 (11 983
46 (7 00
16 14)
Seginst precison
1936(93
(53 44)
(95 1.)
1.6% (42-77)
1,00 93-444
2.25 183
1.2 190
0.68 (92.15)
(9E.77
Sepmnci
precisio
(99.07
(90.67
(98,
(98.67)
46
Seymzni pratision
2.58(30.65
9.36 (72.45
935
C71.001
0.59 (71.15
9.21 (72.5)
0.58 (70
0.36 (72.440)
9.05 (71)2))
9.59 (7L1))
9.21 (72.5)
Seginest precsson S
612(77 71)
6,33 (83 39h
ST (419961
7:30 083 145
6.55 (83
1.1.771.71 711
6.85 (87 (399
6.37 (52.96)
7.36 (83
6,55 03 21)
Sepunzt predisin-
15.47 (8) 35)
16.31 (9).4)
15.15 (8022))
15.62 (79.37)
15.71 (83 15)
15.47 (91.35)
16.31 (91.4)
15.15 (90.22)
16.62 (79.37)
15.71 (83.16)
Segment fl II
50:45 (63 483
17.18
(61 (00)
14 (6) 059
30.71 (00.07)
12 09-0999-00 54)
10.05 10 1-440)
12 18 (61 (88)
31 10 (99)
10.71 (04)17)
$1.00 (98 869
Segment
7.06
7:34 170011
7.37 (77.5)
(70.13
(9..44)
22/90.991
(93.15)
(93.090
(90.89
Segment B.
54.41 (62.51)
27.00 (64.47)
21.22 (65.66)
24. 14.58.05 (53.9)
15.45162.511
16.33.0 164 (6)
20 17.09 (64.47)
21.22 (55.66
74 (65.9)
Sepmeet Tl B
2.46 (99.6)
2166 (96.3))
2.25 (95.44)
2.51 (98.5)
1,55 (98.46)
2:46 (98.67)
2 (94
2.25 (98.44)
2.31 (98 26
1.85 (95, 46)
Suyneai (1 T
14.37 (71
13:94 (21 355
14.72 17 1.8
14.19 (79.57) 37)
(4.37
14.53 (73
13.94 (71.35)
14.72 (11 885
14.18 (73:37)
Segieetf1 fl 5
12.6779.62)
1006 (9336-53.000 (9)
56 (55.6)
11.91 (54
10.65 M5)
12.6 (79.02)
10 (9)
10 (5366
31.91
(44 54)
10:68 (53
Sepmesi fl
20:45 (8494)
(85/06)
22.14
(85.95)
82 65)
20.45 (84.34)
21.18 (96 73)
20.3 (96
22: (85.05)
20:92 (89.68)
Table 9: Table 8 with averages over 10 datascts. We also report global averages over all 8 SSEs.
Mitric
Avg. (HCIERTSS))
Meun recall
33.50 3.95 (97.32)
29. 62
(99.29)
35.59 20)
36.20 (100.00)
32.96 197. .26
33.85 (98.08)
33.17 (96.811
31.65 (98: (8)
Moun precision
33.12 (49.99)
21.76 (67.61)
(90.31)
34.22
24.55 (68.5)
28.89 (51.12)
25.65
(50.71)
27.50 (59.57)
32.01 (60.45)
54 (74.28)
34.66 (60.56)
28.41 (71.15)
50.20
(180
28.86 (67.
Mous lou
(60.18)
27. 44 (74.21)
33.5% (60.20)
24.54 (69.0)
(60.93)
(55.77)
(55.27)
26.95 (65.65)
Segment recall
32.05 (99.80)
29.71 (99.97)
20.14 (200..00
35.63 (99.77)
35.74 (99,05)
32.95 (98.77)
(98.36
31.64
peecision
34.05 (61.01)
4.00
(88.54)
23:95 (64.15)
(82.45)
12.12
Segment Et
31.39
6.91 (74.41)
2.77 (98.34)
26.95 (64.20)
2.19
14.31 (72.15)
11.30
(86.48)
14.00 (78.62)
19

---

Published as a conference paper at ICLR 2026
Secondary Structure Element (SSE) Agreement Results. We ran new expert agreement eval- uation against the 8 basic SSEs (from DSSP). The summary is in [5] We do see above-random enrichment of SSEs in tokens across all-metrics. The recall of existing SSEs is exceptional: 98.28% block-level, 99. 99.28% segment-level, averaged over datasets and SSEs. This indicates the ability to recapitulate the known elements, while the milder precision (59.57 block, 77.80 seg- ment) hints that GeoBPE goes beyond SSEs. As prior agreement results and case studies show, GeoBPE can find biologically meaningful regions (e.g. conserved homology, functional sites) and is not constrained to SSE boundaries, with the data dictating the exact high-level clusters.
D TOKEN EFFICIENCY METRICS
Let = (I,..,,K) denote the codebook (size K). Given a corpus tokenized into flat list of code indices, let cj be the count of code and N -ド== Cj the total token count. We define the empirical unigram distribution
p() Cj for j E.
Utilization rate (UR). UR measures how many distinct codes are actually used:
1 UR = K e > 0 1].
We report UR in percent. UR is important for diagnosing codebook collapse, well-known phe- nomenon in VQ-VAEs where only small number of codes actively used [Zhang al. (2024a). This creates a quantization bottleneck, handicapping the tokenizer's performance and efficiency Yuan et al. (2025)
Unigram entropy and perplexity. Using the Shannon entropy (natural logarithm),
H
r(j logr(j),
PPL exp( ()).
jEV
This codebook perplexity reflects how uniformly codes are used (model-free, ignores sequence con- text).
Max-normalized perplexity. Because the maximum entropy uniform log (hence PPT max K), we also report the scale-free ratio
PPL = PPL exp
H
expl H log K) E (0, 1]. log K)
1 K
log K
Table 10:
We evaluate token efficiency of GEOBPE across varying VI E (600, 2500, 6000), as reported
Figure with Hyperparameter Settings [
UR (%)
Perplexity
Method
Codebook Size
CAMEO
CASP14
CAMEO
CASP14
VQ-VAE
512
5.55
5.60
0.034
0.0337
AminoASeed
512
64.45
68.87
0.495
0.5119
ESM3
4096
27.60
32.10
0.249
0.2841
FoldSeek
20
99.00
100.00
0.755
0.7435
ProToken
512
69.88
75.56
0.537
0.5697
600
59.81
39.48
0,397
0.403
GEOBPE
2500
58.24
38.22
0.274
0.264
6000
53.73
31.30
0.242
0.222
E SMALL STRUCTURE LANGUAGE MODEL EVALUATION
Goals & Aims. This section specifies protocol used to compare the language modeling efficiency of GEOBPE vs VQEVAE tokens. The goal of SSLM was to create a small, isolated environment for language model integration. Modeling is intentionally minimalistic-ww train small decoder-only
20

---

Published as a conference paper at ICLR 2026
Table 11: We adopt the Small Structure Language Model evaluation protocol described in App. E We sample 100 PDB structures. Best scTM and Designability scores are bolded. Underlined methods follow evaluat -io protocol in App.
Structure Language Evaluation
Method
Codebook Size
scTM
Designability (scTM 0.5)
Diversity (1 mean TM)
Uniqueness (TM=D.S)
VQ-VAE
0.205
0.752
98%
AminoASead
512
0,186
1%
0.476
16%
0.268
0.768
99%
GRoBPE
2500
0.267
3%
0.766
99%8
6000
0,277
4%
0,763
98%
GRoBPE (x10 data)
0.376
12%
0.743
83%
GuoBPE (x10 data, x10 params)
600
0.4455
21%
0.731
76%
Transformer (7.3M) parameters over the same PDB splits used in Sec. (4Sk structures). Fixing the same data splits, training, model architecture and hyperparameters (App. E.6), the aim this experiment is to compare tokenizer options for structure token language model. We emphasize the generation quality of the resulting models should be compared relatively; all models trained would be insufficient for real-world backbone design tasks. To bridge the gap with large-scale models, provide an orthogonal study on scalability in App. where we reran SSLM with 10x more data and model parameters.
Setup. For GEOBPE, we use the joint geometric vocabulary learned by GEOBPE; for VQ-VAE, we use their codebook. GEOBPE +SSLM incorporates the mask constraints used generation time (Alg.4) during training to ensure consistency between training and sampling. The same procedure is used for evaluating VQ-VAEs, For training and sampling, the only difference dropping the mask constraints. For inference, the samples passed through the VQ-VAE decoder to construct back- bone coordinates instead of assembling the backbone directly via Alg. This required considerably more resources, and we discuss how implemented this in App. E.6
Algorithm GEOLM-
decoder-only next-token prediction on geometric tokens
Require: Corpus of proteins (4-)² with final segmentations (p(r)) and assigned medoids; joint vocabulary
and tokenizers from Alg. 16 17 decoder-only Transformer Tra E A with causal mask; special
BOS/EOS (optional); training steps S, optimizer O.
Ensure:
Trained parameters 6.
1: Dataset construction. For each т, build ( BACKBONETOSFRUEENCE() (Alg.
Let L,
Objective. For any sequence (ar,...,, 21.). define
LNTP(O; x)
log po (re+1 srs)), po(-Iage) softmax (Tr
Training loop.
4: for = 1 to do
5: Sample minibatch B C Th.
6: C <- |B3 Σ (N; x (t))
Update 00,, VoL).
8: end for
9: return 0.
E. DATA PREPARATION AND SPLITS
Tokenization. We construct the joint vocabulary Σ (Alg. 16 and convert each protein 4 into
token sequence (r) = (T) ,...,,,........... L, via BACKBONETOSEQQEECCE (Alg. 17
We tokenize the validation/test sets (unseen during GEOBPE training) via Algo. analagous to BPE encoding. Sequences alternate strictly motif + + w - Φ motif + and end with terminating motif token (length-2 bond-residue class), hence L (mod 4).
10
procedure
21

---

Published as a conference paper at ICLR 2026
Algorithm 3 BUILDEMPIRICALPRIOKK - length prior and first-token prior
Require:
Training corpus of tokenized backbones (xx*) = (a{") ,....
x (2) )== constructed by Alg. 17 (mo-
tif, then 0, o. repeating); valid sequenee lengths satisfy I, (mod d) and end in terminating motif
token.
Ensure:
Discrete priors I., on lengths and Taaart on the first token.
I:
Length prior: for every K with K (mod 4), set
(K( x (T: L, = K} and normalize
Σ K
(T)(K) = 1.
2: First-token prior: over motif tokens only, set
(ita x { x1 1 iE Emedi
IIstart (i) 1.
iEemedd
3: return II,, Usttat-
Algorithm 4 UNCONDDTIONAGGGOLLEEENNTTTT motif/glue token generation
Require:
Trained decoder-only Transformer Tra with vocabulary Σ from Alg. 16 id blocks
med [l,...,M) M)
Eg {M+1, M+BB}
(2)
En (M+Ba B1 +......... M B.)
(3)
L. {M 1o+
M+ - B_ +b)
(4)
terminating-mont motif set Eterm S Emed (motifs in the length-2 bond-residue class); priors II, Ilstart (l temperature > 0; maximum length Kmaxi number of samples S.
Ensure: unconstrained token sequences (x alternating motif and glue tokens and ending terminating motif.
I: Define the type mask by position (t starts at 1):
= 1 (mod 4) motif (Emed), t 2 (Ee), t=3 (Ew), t Φ (E+).
2: for = 1 to do
3: Sample target cap KTO ~ TI, and set KPup +- min(K Kmax).
4: Sample the first token x11 (4) Nytare (so x. Emea).
5: for 2,3,..., Keap do
6: Compute last-position logits Tra (z with causal masking; let |E].
7: Build hard mask B initialized to and
mi <- 0 if /
1 (4) and E Emed,
m: + 0 if f = 2 (4) and E Eg,
m; + if 3 (4) and E Ex,
mi <
0 (4) and E Eg.
Termination constraint at motif positions:
If = 1 (4) and < Ke then disallow early stop: set m. <- -oo for all E Etem- r 1 (4) and / = Krap, then force stop: set m. <- oo for all i E Emod \ Σterm- 9: Form masked logits 5, z + and sample
x!
Categorical(sfttaxx((())
10:
(Optional early stop) If = 1 (4) and (+) Eterm- then break.
11: end for
12: end for
13: return
{o 8 1
Splits. We partition proteins at the protein level into train/validation/eest (e.g.. 80/10/10) to prevent leakage across chains.
22

---

Published as a conference paper at ICLR 2026
Algorithm 5 DEEQCANTZZEANDASSENNN from tokens to a full backbone
Require: One generated sequence (#1....;22) from Alg- medoid dictionary {idmed (K. } where each prototype _1!") tuple internal coordinates a motif M: glue bin edges ( [16) Th=0" [ Be (circular edges for angles, linear for lengths if used); canonical seed triad (A., C.) and SEEDTRIAD.
Ensure: A complete backbone {NN, CAs, C.) E [R)]== assembled from the decoded motifs and glues. Parse tokens into motifs and glues (fixed 4-cycle). Let motif indices E {1,5, ........ write idmed (N<(m), , f(m)) for L,...; where M L+3 For each boundary 1,..., M 1, decode the three bins:
by Tam M, bw = 24m (M)+
by Z4m (M+BB+BB))
and dequantize the bin midpoints
8m -+) 7m -1 om 1
2: Recover internal coordinates. For each motif m. let II ((6)))
g(m) provide the internal bond lengths f, bond
angles and dihedrals (1, w, 6) across its span M Construct its internal entry-texit transform ((m)
(product of link transforms G, inside the motif; see Preliminaries).
3: Forward kinematics assembly.
4: Initialize the entry frame by seeding the very first residue: (N1, CA1,C)) SHEDTRIAD([) and form
Fi [R1, f1) as in the Entry/Exit frame definition.
5: Motif 1: Traverse the links inside ( using its internal coordinates compute frames F2,.... Fg. (and atom positions) by repeated C; multiplications;; set the current exit frame (1) Fg.-
6: for m 1 to M do
7: Boundary glue: form the boundary transform
Tella (m)
Gam (BONCI
0m, Wm. á dm))
i.c., the SE(3) map from the exit frame of (m to the entry frame of M (m++) determined by the three dequantized glue angles (and adjacent bond lengths).
8: Set the entry frame of (mm+1) to
centey (mx+1)
2aglue (m) ((m).
9: Motif (m+1); traverse its internal links to produce all residue frames and atom positions; update
Fexxic end for
(m++1)-
10:
11:
Concatenate atoms. Collect the atoms from all traversals in order, yiclding the backbone
(N,, CA,, 1 2-1" where Ñ is the total number of residues implied by the concatenated motif spans (the final motif is guaranteed terminating).
12:
return the complete backbone coordinates.
E.2 TRAINING OBJECTIVE WITH STRUCTURAL MASKS
We train causal Transformer Tra with teacher forcing. To enforce legality each position apply the same type mask by position modulo used in generation (Alg.
f (mod 4) + Emed;
- Σg, 1 Ens 0 Ex,
setting logits for all other token types to -c before the softmax.
Termination constraint motif slots. At motif positions (1 (mod 4)), we impose the same termination rule as in Alg. 4 (i) if T.r, mask out terminating motifs Eterm: (ii) if i T.,, mask out non-terming motifs.
Loss. With masks applied, the negative log- likeliihooddd is
(NTP(O::!T)) = Σ log po
(T)
p xx)) = softmax(
2
where ž are masked logits. We optimize by minimizing the average NLL over the training set. Early stopping. We select checkpoints by validation loss with patience of 5 epochs.
23

---

Published as a conference paper at ICLR 2026
E.3 UNCONDITIONAL SAMPLING FOR QUALITATIVE EVALUATION
Empirical priors. We form the length prior II, and first-token prior llstart from the training corpus using BULDDEMPRICALPRIORR (Alg.) 11, supported on legal lengths =1 (mod 4); 1lstart over Mmed.
Constrained generation. We sample with UNCONDITONAALGFOL.GGGG KeAP ~T, (clipped by maximum), sample the first motif ~ IIsartt then autoregress under the same positional type mask and termination constraint as training. Temperature and nucleus sampling optional ablations.
(Alg. lg.
draw
GEOBPE Dequantization and assembly. Generated token sequences are mapped to full back- bones via DEQUANTIZEA AAS.E/EAAAAASSSEBLLLLLEES (Alg. 5):
medoid tokens decode to internal coordinates over their motif spans; glue- bin tokens decode to bin-midpoint angles; forward kinematics with the seeded entry frame yields atom coordinates ((N, CA,, C.)
E.4 GENERATIVE QUALITY ASSESSMENT
We evaluate unconditional samples produced by UNCONDITIONALEOOLMEEENE (Alg. 4 and assembled by DEQUANTIZEANDASEEBAA (Alg. 5 using four structure-centric metrics based on TM-score
Setup. From each model we draw a fixed number of backbones (Bn) ) (legal lengths, terminal motif constraint). Unless noted, metrics are computed on these backbone geometries without further post-processing.
(1) scTM (self-consistency TM-score). For each generated backbone B, we (i) design a sequence 3 with standard inverse-folding model, (ii) predict structure B from 3 using single-structure predictor (e.g., ESMFold), and (iii) compute
scTM(B) TM-score(B, B).
We report the mean scTM over the samples.
(2) Designability (% with scTM > 0.5). A backbone is deemed designable if its self-consistency exceeds the canonical threshold 0.5:
N
Σ 1 1 scTM(B.) > 0.5] x 100%. n=1
Designability =
N
This is the fraction of samples for which designed sequence refolds back to the generated backbone the fold level. We adopt the same workflow from Trippe et al. (2022); Wu et al. (2024), where ProteinMPNN Dauparas
| et al. (2022) proposes 8 sequences structure and Omegal Wu et al.) 2022 is used to compute scTM.
(3) Diversity (mean pairwise TM). To quantify comple-to-sampee diversity, compute the mean pairwise TM-score across the set (lower is more diverse):
Σ Diversity = N(N - 1) ISKKiEN
TM-score B., B)).
(When is large, we estimate this by uniform sub-sampling of pairs.) (4) Uniqueness (% non-duplicates at TM- < 0.5). We mark sample unique if its nearest neighbor among the other generated backbones has TM-score < 0.5:
Uniqueness
1 N
Σ
max TM-score Bm) < 0.5 100%. mann
This measures the proportion of samples that are not ncar-duplicates under a fold-level threshold.
Reporting. For each model report the four metrics above the same set size N (and the same sampling priors and temperature). Codebook size and token perplexity are not used these downstream comparisons.
24

---

Published as a conference paper at ICLR 2026
u
o00g
(a) Generated backbone with scT score 0.50
(b) Generated backbone with scTM score 0.56
Figure 7: We visualize two backbones generated by GEOBPE SSLM-Eval (with default settings in App. K
E.5 GENERATED BACKBONES
In Fig. 7a we see a long, well-structured and assembled a-helix, which is one of the most common and stable secondary structures in proteins. The curved helical cap at the top resembles common N-terminal capping motif, which often stabilizes helices through hydrogen bonding networks or electrostatic interactions. Such elongated a-helices are commonly found transmembrane helices or coiled-coil domains which are involved in dimerization and DNA-binding. The overall curvature and spatial continuity also suggest potential compatibility with membrane proteins or structural scaffolds, especially behave substance binding receptors well as ion channels.
Fig. 7b we see a structure that resembles DNA- DNA-binding motifs or cytokine folds, which are quite well-known for cellular signaling or regulation. The geometric density of this structure also suggests a pre-or pre-organi hydrophobic core, which is critical for proper folding and stability in the cytoplas- mic environment. This structure exhibits compact bundle of helices with apparent crossing angles which are similar to some small globular domains in common protein structures. The folding ap- pears non-linear but in quite controlled, manner which suggests potential tertiary structure forming interactions such as hydrophobbic-hydrphhohii interaction.
E.6 IMPLEMENTATION DETAILS
SSLM-Eval GEOBPE implementation and hardware details. We train a small autoregressive Transformer on discretized geometry tokens. We use a hidden size dmmotel = 256, L = 8 Transformer attention heads, and feed-forward width da 1024. Token and positional embeddings are summed, LayerNorm is applied before the classifier, and the output projection is weight-tied to the token embedding. causal attention mask enforces left-to-right prediction. Sequences
layers with GELU activations,
padded to a dataset-dependent maximum length (the 95th percentile of training lengths by default). We optimize cross entsroy loss with Adam (learning rate 1 x 10 10-4), batch size 32, for up to 100 epochs with early stopping validation perplexity. For unconditional generation, we sample 100 sequences at temperature 1.0, drawing target lengths from the empirical length prior (restricted valid lengths by construction) and the first token from the empirical start- token prior; decoding proceeds token-by-token under the causal mask. On a single GPU, one epoch takes just under 10 mins and converges in se 60 epochs (can vary across tokenizer settings). The data splits are the same as those for pretraining (see App. B 33992, 3810 training/validaatn structures for GEOBPE.
SSlM-Eval VQ-VAE implementation and hardware details. We extend the distributed Light- ning setup of Yuan et al. (2025) with self-contained evaluation step the end of each validation epoch. Using 4 ranks, each GPU accumulates the epoch's quantized token sequences from training and validation; these are gathered and passed to a lightweight auxiliary trainer that uses the same SSLl-Eval script GEOBPE and hyperparameters. After convergence, we sample 100 new token
TM--scoee is obtained with standard implementation (e.g., TM-align); higher is hetter.
25

---

Published as a conference paper at ICLR 2026
sequences, decode them with the VQ- VQ-VAE decoder into backbone coordinates. and write PDBs to a directory named by the current epoch. We compute all non-SCTM metrics locally, then distribute heavier SCTM evaluation across ranks on sharded PDB subsets. Each rank produces its shard's results, and rank-O) merges them into a single summary that logged to the trainer.
F SCALING GEOBPE TO LARGER DATASETS AND MODELS
Goals & Aims. The relatively minimalistic design of GROBPE SSLM App- still begs the question whether GEOBPE reliably scales to large datasets (e.g. ESM3 scale) as both a tokenizer (GEOBPE only) and the foundational component of a language model (GEOBPE + SSLM). Thus, attempt to bridge this gap by (i) tokenizing 10x larger dataset and benchmarking wall times, and (ii) training a 10x larger SSLM model on the 10x larger tokenized corpus.
F. TOKENIZING 10X LARGER DATASET OF PREDICTED STRUCTURES
Setup. We downloaded the 550K Swiss-Pro structure predictions from AlphaFold DB Consortium 2024g; Varadi et al. 2022). a > 10x increase from our PDB pretraining dataset. We adopt the pretrained [V] 600 tokenizer used in the paper (Fig. 4 Tables 10 & 11 baseline (i.e.
tokenizers with larger |V] will achieve lower distortion at the tradeoff of slower throughput).
Evaluation. We log both wall-time and thoroughput taken tokenize all 550k structures. We also report the distortion against AlphaFold DB predictions, We split into five 110K increments, and requested jobs with 20 cores each. Each job writes each tokenized structure file, which allows us to log the running throughput from start to finish.
Table 12: We report throughput and distortion tokenizing 550k Swiss-Prot predicted structures with jobs of 20 cores each.
Split (110K increments)
3
4
5
Total (5 splits).
Avg. Throughput (files/min)
35.54
45.40
35.45
36.64
38.29
191.32
RMSD LDDT
1.52
1.54
1.54
1.53
1.53
1.53
0.79
0.79
0.79
0.79
0.79
0.79
With pooled avg. throughput of 191.32, all 550k structures were tokenized in ~ days. Discrepancies throughput between jobs likely explained by node traffic. We see all splits achieve the same LDDT (0.79) and within 0.01 RMSD of the average RMSD over all 5 splits (1.53). These are comparable to the tokenizer's OOD test set distortions in Fig. (1.53 RMSD, 0.72 on CASP).
Results. The results are shown in Table. 12
Conclusion. Since GEOBPE has shown strong OOD generalization from our findings in Sec. it is expected not degrade in performance. Thus, the right approach is tokenization (Alg. [0] rather than retraining GEOBPE from scratch. In contrast with GEOBPE learning, tokenization is an embarrassingly parallel procedure that easily scales with the number of cores. With 100 cores, the entire process finished in ~ 2 days, or 200 CPU days. Further scaling by another 10x would only take ~ days with 100 CPUs, making GEOBPE scalable solution for tokenizing large databases of predicted structures.
F.2 STRUCTURE LANGAUGE MODELING AT A 10x LARGER SCALE
Once we have the tokenized dataset of 550k Swiss- structures, we train larger model and probe whether generation quality of GEOBPE SSLM follows the expected improvements from scaling.
Setup. We increased our Transformer to ~ 10x parameters (7.3M + 65.9M/. We do so by widening the Transformer layer and deepening the model: dmodel + 2d = 512, I. <- 2.5L = 20 layers. + 2 H 16 attention heads, da < 2 dr = 2048. The rest of SSLM remains the same (App.
Evaluation. We used the same evaluation protocol (50-128 AAs, 10 each) of works such as ProtDiff Trippe et al. (2022) and FoldingDiff Wu et al. (024).. Note that App. did not follow this protocol;
26

---

Published as a conference paper at ICLR 2026
sampled from the size prior of pretraining dataset (Alg. was 244 AAs.
B the average generated protein
Results. GeLBPE+SSLM with 10x more data and 10x more parameters achieves average scTM of 0.4051, with 20.8% being Designable (scTM > 0.5). This is notably higher than ProtDiff (11.8%) and FoldingDiff (14.2%). Scaling only by 10x more data also delivers a respectable scTM of 0.376, highlighting both scaling dimensions are throttles for generative performance. Unique- ness/diversity also remain high (76.4% and 0.73).
Conclusion. This result confirms that GEOBPE behaves according to scaling law expectations of language modeling. The significant increase in designable backbones (4% - 21%) from simply using more data and parameters justifies further scaling of data and training resources. We hope future works can adopt GROBPE a foundational component in future large-scale models and explore the full potential of large-scale protein structure language model development.
G EXPERT AGREEMENT METRICS
Our method segments protein sequence into M contiguous residue spans P; [pjs Yi] with Øj + 1 P5+1 for j 1, M - 1. We compare these segments against N ground-truth truth domain annotations D, [s, es). All sets below are sets of integer residue indices and |- denotes cardinality (length in residues). We report (i) domain-level alignment quality for each true domain using the best consecutive block of predicted segments, and (ii) segment-level detection statistics at an Intersection- over-Union (loU) threshold т. This combination captures both how well each domain is covered and how economically the predicted segments explain the annotations, while remaining robust to small boundary jitter.
Annotation source. Ground-truth domains come from CATH FunFams Das et al. (2015b). They functional families defined by profile HMM hits trained primary-sequent data Das et al. b). Our evaluation thus measures how well the predicted segmentation aligns with function- ally coherent families derived from sequence-based HMM models.
In our setting, individual predicted segments tend to be substantially shorter than the curated domain annotations. A naive one-to-one comparison would systematically penalize predictions that must be combined to cover domain. To ensure fair comparison, for each D, we first select the single best consecutive block of predicted segments S; that maximizes IoU with D; (below). then compute per-domain scores and macro-average them so that each domain contributes equally, independent of its length.
Notation and best block per domain. For domains D; [8s, es] (i 1:N) and predicted segments P3 = [pj, q. (j = L:M, with 9 +1+1+ Pj+1), define
DinUk DRU
& PR
(as, bi) arg
max M
S. = U PE.
km PE
Ama;
(Ties may prefer the shortest S, or fewest segments.) Domain-level scores (macro). Let ov; (D,nS,), [D.] = - -; -1
Ebe 2ekm; (gt Pk + 1).
Then
ov D. Macro-averages:
OV
2 Recall, Precision;
OV;
Recall;
Precision;
Fi;
ToU;
|
Recall; + Precision;'
D2. +
ov;
Recall 1/ Σ Recall, Precision
Σ Precision,
F 1
E Fi,
IoU NV VIOU,
N
Interpretation: recall rewards coverage; precision rewards compactness of S; F balances both; loU is thresholdable and scale-invariant. Segment-level detection at IoU threshold r. Let 1n (a....,, b.). Define
U
Ki: loU, 2 T
2 SegPPre SegRec
SegPrec-
SegRec
SegFi =
M
N
SegPrec SegRec
27

---

Published as a conference paper at ICLR 2026
Interpretation: SegPrec penalizes unused segments; SegRec penalizes missed/poorly aligned do- mains. Sweeping T yields a PR curve.
Randomization baseline and reporting. Using 1000 uniform random partitions into M contiguous spans of the same sequence, recompute all metrics under the same best-block protocol and average over runs. We report using the format:
ours (random-avg),
e.g. IoU - 0.47 (0.18).
Notes. In degenerate cases (c.g., |S,] = 0 or zero denominator), we adopt the standard convention of returning for the affected ratio or Fi term.
Together, the domain-level (overlap-qualie age) metrics directly test the two desiderata of protein domain segmentation: (i) accurate coverage of each domain with minimal spillover, and (ii) parsimonious set of segments that explain as many domains as possible. Macro-averaging after selecting the best block per domain ensures fairness when predicted segments are shorter than annotated domains, and the permutation baseline quanti- fies how far performance rises above chance given the same M.
metrics and the segment-level (parsimony and cover-
H EXPERT CASE STUDIES
Individual Tokens Correspond to Secondary Structures. Figure g is an example of sin- gle token GEOBPE discovers. It features alpha helix that includes aromatic cage (formed by Tryptophan / Tyrosine) and hydrogen bonding residue. It can be a common structure in Nucleotide-recognition domains, especially the hydrogen bond donors/acceptors specific molecular recognition (e.g., methylated lysines, nucleotide bases or acetyl groups) as well as Neurotransmitter receptors. From interpretation, this motif is functionally specific. It can serve as ligand binding pocket, which is tightly packed and evolutionarily conserved. This could behave significantly in sub- stance recognition. The tightly packed helical scaffold in this sep- arated motif is likely stabilizing the motif's geometry and ensuring specificity, Motif-scaffold synergy can also help to define a struc- ture's rigidity and flexibility.
serve for
Merge Hierarchy of GEOBPE Reflects Combination of Sec- ondary Structures for Driving Function.
4xk4 is the human mitochondrial carrier protein SLC25A20 (camitine/acyllariitnn translocase). It's a transmem- brane transport protein within the mitochondrial inner membrane, responsible for shuttling carnitine and acylcarnitine molecules across the membrane. This process critical to fatty acid oxidation and energy metabolism. The core motif that the algorithm separated contain three similar domains, each with transmembrane helices and a loop. It appears to lie deep within the transmembrane domain, forming part of the central binding cavity. From this know-how information, the 48-4 motif is really significant in the following three aspects:
Figure 9
Figure 8: GEOBPE token spans back- bone atoms of alpha helix (colored).
An exemplary
1. will serve for substrate recognition where the internal polar residues bind to the acylcarnitine or carnitine head group via ionic and hydrogen bonds. It will also alter the transition state of the during the transport cycles. For example, this motif can play a role in shift conformation between noentto-cytoplasm and open-to- matoix states.
We also observe similar motifs are found in other SI.C25 family members (e.g., ADP/ATP carri- ers), indicating shared mechanism of transport.
3. While the broader transmembrane region dominated by repetitive helices, this localized motif exhibits unique composition of diverse side chains, polar residues, and tightly packed interac- tions, reinforcing its functional specificity.
Figure 10 6F14 is the crystal structure of hybrid peptide composed of C-terminally mod- ified Tau protein segment bound to the human 14-3-3a protein, solved at 2.0 Á resolution via X-ray crystallography. 14-3-3 proteins are family of conserved regulatory molecules that bind
28

---

Published as a conference paper at ICLR 2026
POB ID: 4ak4, Chain ID: D, Takan: [48, Sa)
Mothl
Structural Role
Functional Implication
Polar bridge keep anchoring
Halp boost nut eation at
#he binding sho and
DETACUUR merriae
Forms doep
poler
Major
scarlold connect
Igands
specilic
membrane
Fully assempied Vertary
One the care motts of the protain
(50
J9
48
Figure Chain of PDB 4xk4, Hicrarchical Merge Tree for Token [48, 52). GEOBPE arrived at this token by merging [48, 49) with [49, 50), [50, 51) with [51, 52), and [48, 50) with [50, 52).
phoghhssrii//asppoottrrrieenocin motifs on target proteins and are central to cell cycle control, apoptosis, transcriptional regulation, and signal transduction. The hybrid peptide mimics Tau phosphorylation, which is relevant to neurodegenerative disease pathology like Alzheimer's disease. From this know-how information, the 29-4 motif is significant in the following two aspects:
1. Phosphopeptide recognition and improve the binding Stability: This motif orchestrates recogni- tion of the Tau-derived phosphoserine motif via a precise network of hydrogen bonds and elec- trostatic complementarity. The Lys/Arg residues (seen blue) form salt bridges with phosphate groups, stabilizing the interaction.
The structure shared recognition fold across the 14-3-3 protein family: This motif, with its basic side chain tunnel and surrounding helices, represents a canonical recognition site. Similar struc- tural motifs are observed all 14-3-3 isozymes when binding phosphoproteins and will serve for post-translational modification signaling.
PERFORMANCE ACROSS PROTEIN FOLD TYPES
Setup. We evaluate robustness of GEOBPE by computing distortion as defined in Sec. 3.3 across fold types, unusualness, and size with per-chain metrics. Structural "unusualness" is computed from Foldseek's ' TM-align mode as 100 x (1 - TM) using the best hit against PDB. We also attach coarse labels: cafegories by best-hit TM-score (Near-identical 0.90, Same fold 0.50-0.89. Distant
29

---

Published as a conference paper at ICLR 2026
6F14,
[29,
Motif
Structural Role
Functional Implication
52)
Arg-Sar/Tr rnltator
Significant charged chains
tail
Forms curvature the peplide pocket
phosphoserine embedding
Constructs dide entrance to phosphate- binding zone
Stablizzes entry path af Tau peplide backbone and allowe side chain orientation
Fully assembled
Becomes the critical
[48, SO)
(50,
50
Figure 10: Chain A of PDB 6F14. Hierarchical Merge Tree for Token [29, 33). GEOBPE arrived at this token by merging [29, 30) with [30, 31), [31, 32) with [32, 33), and [29, 31) with [31, 33).
0.30-0.49) and flags indicating very small (chains with 70 residues) weak coverage (low query coverage).
Fold type labels. For fold analyses, each chain is annotated with CATH fold label (topology-Ievel code) and rendered as human-readable Class + Architecture title (multi-line axis label). We report per-fold distributions and means for RMSD and LDDT.
Plots and statistics reported. We use four concise views: (i) Dual-axis seatter vs. unusualness: RMSI (left, blue) and LDDT (right, orange), with per-bin mean std and fitted least-squares trend line. (ii) Group boxplots (shared z-axis): sideeby-side, dual-y box-and-whisker plots com- pare distributions across Every-smalls weak.coverase Near i ident ical, Same fold, Distant]; ); RMSD boxes map the left axis (blue), LDDT boxes to the right axis (orange), (iii) Dual-axis scatter vs. length: RMSD (left) and LDDT (right) versus protein length, again with per- bin mean + std and fitted trend line. (iv) Fold boxplots: side-by-side, dual-y box-and- plots per frequent CATH (Class, Architecture) (top-N by support). Together, these views test how specific conditions (very small chains, weak alignment coverage, and decreasing fold similarity)
30

---

Published as a conference paper at ICLR 2026
Performnce Insconmmes (RMCD 1/15T1
1M80 bett L10OO inard
10
30
40
30
Unoouatmess
scatter vs, unusualness. RMSD (left, bluc) and Group boxplots across very coverage. (right, orange), with bin means 1. std and trend lines.
Near-i call Same fold, Distant. Dual hoxes: RMSD (left, blue), LDDT (right, orange).
Performance Pronels length (RMGD & LO33))
200
200
410
500
anghh sums
scaller length (residues). 1.DDT (right, orange), with hin means + std and tren lines.
blue) andíd) buxplots labels
Architecture. Dual-y boxes: RMSD (left, blue), LDDT (right, ar- ange).
Figure 11: Plots for GEOBPE robustness evaluation. (A) unusualness, (B) group distributions across flags and categories, (C) length, (D) per-fold distributions. Numerical summaries (Pearson's with 95% Cls; group/fold means medians) are in the accompanying CSVs.
modulate accuracy distributions, and whether protein size systematically correlates with errors. We report the following numbers and observations.
No degradation on unusual structures In Fig. 1la we see no correlation (Pearson's 0.0091 for RMSD and -0.0096 for LDDT, 95% intervals of (-0.0346, 0.00365) and (-0.0451, 0.0260)) between Distortion and Unusualness. Consistent with our OOD results Fig. 4 we see GEOBPE is robust distributional shifts. As ceometry-grountee tokenizer, GEOBPE captures energet- ically favorable motifs patterns, which are universal recurrences across all protein families and fold classifications.
No degradation on less common folds In Fig. 1lb we see distortion remain stable on near identical same fold to those in FoldSeek-DB. In Fig. lld there no visual trend of degradation for less common fold types (left-to-right in Fig. d1 of the ones shown. Inter-fold discrepancy is also low: among folds with > 100, the least faithfully preserved fold type suffers from 30.2% higher RMSD than the most faithful.
More faithful larger folds than smaller folds Fig. Tlc
we see weak correlation (Pear- intervals of (-0.2477, -0. 00.1999) and
son of -0.2141 for RMSD and 0.5627 for LDDT, 95%
1lb we see distortion slightly elevates for very small folds. Among fold types with at least 100 samples, GEOBPE achieves lowest distortion (1.130 RMSD) on Alpha-Beta Beta Barrels (cylindrically packed, stable folds) and highest distortion (30% higher RMSD) Mainly Alpha Up-down Bundles (smaller folds primarily of alternating alpha helices). This suggests GEOBPE has high propensity for packed but stable folds (sandwiches, barrels).
(0.5379, 0.5865)) between Distortion and Length. In Fig.
31

---

Published as a conference paper at ICLR 2026
J COMPUTATIONAL COMPLEXITY
Notation. Let {t()) be T backbones with lengths N and let N= E N(r)
be the total residues. In each STEP iteration, the most frequent geo-pair key has M, occurrences. We use K for the number of medoids produced when Clustering key's occurrences (a small constant in practice). For k-medoids we either: (i) cluster all M, items, or (ii) cap with Maax items. Let be the period at which GL.UEOPALLL is invoked (see Alg. (9) and let CIK denote the cost of one global IK pass (see below). The ordered map D stores key + occurrence-set with a priority (p. (0., x);; each insert/erase in D costs (log (DI) = (log
Component building blocks.
O(m²) to build the pairwise RMSD matrix (constant fragment length), plus a small constant number of assignment/update steps
k-medoids on items:
Priority map updates: each merge touches (1) neighbor pairs; across the entire run there
((N) merges- + O(N log N) total map operations Every merge eliminates one boundary and touches at most its two neighbors, so the total number of insert/erase operations in D across the full run is O(N);; with Ollog N) per op, the total is ON log N).
Global IK (GLUEOPTALL) one pass: forward kinematics is linear in links, so one pass costs Cıк O(N SFK), where SFK is the (small) number of optimizer steps x the constant for- ward/backward cost per link Periodic GLUEOPTALL adds F O(N log N) due to re-keying af- fected boundaries.
Worst-case complexity (no subsampling cap).
ResInitTokens: O(N3) + O(N log N).
Step loop over all iterations: of E, Mf) + ONN log N).
T P Cik + O(N log log.))
Periodic global glue opt:
Total (worst case): O(N3) + 0 Σ, M2) + O(N log N)
T
CiK + ( log N)).
+ p
Total (with cap): O(M + O(T 2 + O(N log N) + P (Cik + O(N log log
In the worst case M₁ (NN) for many steps, E, M2 can reach O(N²). Here Mmax controls runtime. Putting it together, we can make the following statements about GEOBPE's computational complexity:
(Alg. | Training (discovering the vocabulary): dominated by k-medoids calls and periodic IK:
O OT M + O(N log N) + = (Cik + ONN log N)).
(Alg. 10p Tokenization (apply learned vocabulary): similar training but without any k- medoids calls and in terms of (T);
O(N(1) log N(1) + (CIK + ((N(1))) log )())).
(Alg. Detokenization (geometry reconstruction): forward kinematics per link is O(); re- constructing all atoms is ONN).
Insights for efficient practice. (i) Most structural variability concentrates in a small number of modes; modest Mmax suffices. (ii) Dictionary updates are incremental; our implementation uses an ordered map. (iii) In practice, we choose P = 10: GlueOptAll calls are infrequent enough it does not become an issue. If this becomes the practical bottleneck, we recommend GLUEOPT for local IK updates instead, which drops the O(N log N) term.
Distortion insensitive Muax. In App. A we observe that increasing Muax yields no real gains beyond 5000; any marginal gains are lost to the subsequent GlueOptAll call. This is because medoids stabilize quickly on representative modes, capping clustering with Mmax preserves recon- struction quality while bounding the dominant O O(M2 ( term. This is backed by observations made by de Brevern et al. (2002); Mackenzie (2016) and others that the structural universe possible el- ements are captured by exponentially smaller number of modes.
Increasing GLUE OPT EVERY does not significantly hurt performance. In Figs. 13b & 13a we GROBPE behavior remains comparable between run where the expensive glue optimization
32

---

Published as a conference paper at ICLR 2026
following our default recommendation of every ten iterations. The (N log N) term is often the key walltime bottleneck, as Table 13 shows. Therefore, increas- ing P would help amortize the expensive rigid body refinement routine across iterations.
(all) is done every iteration,
Wall times from our experiments. In Table 13 we report empirical wall times from sample run of GEOBPE following default settings. The runtime can be accelerated with more CPUs.
Table 13: Using 20 CPUs, we report our job's wall-clock time. Underlined steps perform periodic glue op- timization (period P = 10). They are followed by P P GFOB steps. We report wall times for steps 0, 10, 20, 200; omitted steps interpolate predictably.
Function
Paper Reference (Algo, Line)
Time (HH:MM:SS)
init thresholds
Algo 1 (Empprriaall Quantizer Estimation)
00:01:33
init res tokens
Algo 1.2 (Per-residue Initialization)
02:16:02
glue_opt_all
Algo 1 1.3 (Global glue refinement)
03:21:50
Step 0
Algo 1 L7 (Step) w/ Algo L13 (glue opt all)
02:36:21
Steps 1-9
Algo 1 L7 (Step)
01:32:50
Step 10
Algo 1 L. (Step) w/ Algo 9 L13 (glue opt all)
01:40:58
Steps 11-19
Algo L.7 (Step)
01:37:39
Step 20
Algo 1 L7 (Step) w/ Algo 9 L13 (glue opt all)
01:27:39
Steps 21-29
Algo L.7 (Step)
01:36:41
Step 200
Algo 1 L7 (Step) w/ Algo L13 (glue opt all)
00:56:56
Steps 201-209
Algo 1 L7 (Step)
00:50:06
K HYPERPARAMETER DOCUMENTATION AND GUIDELINES
K.1 MAIN HYPERPARAMETERS AND REPRODUCIBLE SETTINGS
We describe the key parameters that govern GEOBPE's behaviors in Table 14 the default setting used by GeoBPE across most key results of the paper: Fig. 4 App. A We report any instances overriding the default settings here:
For each, we report
Tables 10 & and
1. Token efficiency / SSLL-EEval (Tables 10 11b
bins (1:1000} for codebook size [V] = 2500 and num_p <- (2: 1,10000,, bins < {1:2000} for = 6000.
num p
2. Pareto-efficiency evaluation (Fig 4 further add the setting for | = 21000 where num-p +
(2:1000, 3: 3:0000)). bins [1: 20000. We vary num elastically moves along the Pareto-efficiency plot, trading off BPR for distortion. All runs use w. = 1.0, which we discover
from ablation studies (see Tables T7& T8) has better performance than w, 0.1.
3. Downstream transfer experiments (Tables
2, 3: 5: 6,2, f free free_bonds +- False and bins <- {1 50). and bin_strated <- histogram-cover
7 set num
to adaptively coarsen the resolution. GEOBPE prioritizes learning fine-to-coarse hierarchical signals over low distortion for effective transfer.
Table 14: We report the main hyperparameters that affect GEOBPE behavior.
Parameter
Yalue
Meaning
Default Behavior
bin eecegy
bistagmum (1:50.0
Cuntreds the strategy for empiniad quastiser estimation (Ale
sumge, hidngglnigggmm with bins
bina
Contrels the sunlies of bit used by ban, bin.rrre:egg
Usc, 500 quanilts
Free bonds
Tue
Whether to quantize bond lengits
Don't standadizee
Sinting so False cundardinos
hand lenghs to peccomputed values
Quontine with linear histognems
g leeungt
Trme
Whether di Clae Ogt Alo [8
Do Gitu Opt
glue bluuegght Jowey
10
How often to non plotal glu opromin munile line Alg by
Do every 10 ilens
Flloeoogh aew had
all
Whetler to buids glue upt (Ale [2
singl-bommaary glux ogt (Alg.
Do batch gluc opt
glue opt. prior
1.0
Price weight encouraging optimie groes monch empnival distnbution
1.0
NU,, Mi7
1.0. 0.1
Rotation, inavdation lice frrm wrights to IK loss (Alg [0]
Weigh motiion crmer 10c maasaaiion renr 5p00
nex_mun_st mune
5000
Mas mumber aecumenos for chatrrin (M,. Al & dstemminnd by ypu lengh in Ale
nurup
(2:100, 500,
in mun. mronnd dowc nezecal key
Tae K 180 whun J. -
5120, 61100]
Use K w0h & 2 Use original states
meduopeerrree
True
Whether use oocarmece from original hackhone ts or corrent backbone in Alg
33

---

Published as a conference paper at ICLR 2026
K.2 HYPERPARAMETER SELECTION GUIDELINES
Which ones to prioritize. Only few parameters in Table 14 dictate overall behavior, performance and runtime. Essential knobs are:
Vocabulary Growth
num_p (number of medoids)
bins (quantizer strength)
max_iter(or # iterations run)
Compresiio/Runtime Tradeoff
-giue opt/glue opt method/glue opt every (glue optimization)
Aside from these, we suggest leaving the rest to default values.
Choosing num p (medoids per step) and bins (angle/length quantization strength).
We define num p via step-wise schedule over motif sizes. For example,
{2 2, 3 5, 5
10}
(passed as --num-p 2-2:3-5:5-10) means:
introduce 2 tokens for geometric keys with 2 bonds (C-terminal residue orientations),
5 tokens for keys with 3 bonds (all non-terminal residues),
10 tokens for all merged geometric keys with 5 or more bonds (every GeoBPE step after residue initialization).
The bins parameter uses the same syntax p. For example,
{1: 100, 3 10}
(passed --bins 1-100:3-10) introduces 100 bins discretize the angular histogram at ini- tialization, with 10 bins for keys of size 2 3. For brevity, BINS = n is shorthand for BINS = {1: nn).
The binning strategy is controlled by bi bisstraty... If glue optimization produces angles out- side the supported range, we snap them to the closest bin. In practice, we recommend increasing or decreasing bins in tandem with num.p.
Below we give practical recommendations by downstream use case.
GeoBPE for compression/ / reconstruction.
Intuition. Larger num-p values - more medoids per step, which improves reconstruction quality (RMSD/LDDT) the cost of larger vocabulary and noisier merges. Empirically, we observe diminishing returns in reconstruction beyond settings such as
num_p = {2 200, 3 1000},
consistent with there being only limited number of modes in the conformational variability of energetically favored backbone regions (Ramachandran landscape).
Recommendation. Use relatively large num p maximize reconstruction fidelity, but pair it with high (yet not extreme) b bins [s i ze] to avoid a combinatorial explosion in the space of geometric keys. A good default for reconstruction-orieneed use
bins = {1 500).
combined with moderately large num.p.
GeoBPE for representation learning.
Intuition. GeoBPE emits both a sequence of tokens and a merge hierarchy, with the hierarchy providing the main inductive bias for downstream representation learning from residue to protein level. useful hierarchy should:
capture higher-level patterns (from basic secondary structure elements functional sites),
avoid overfitting high-frequency local vibrations.
34

---

Published as a conference paper at ICLR 2026
Here the goal shifts from pure compression to coarsening: want motifs that aggregate meaningful local structure without being overly fine-ge grai-rraidd
Recommendation.. Use relatively small num-p values and correspondingly small bins. For exam- plc, the configuration used in our paper for representation learning was
num p 1 2, 3 5 1, 6 2, 8: 1)
paired with
bins = {1 50}
and the histogram-coom strategy. This yields coarser motifs and hierarchies that are better suited to downstream predictive tasks.
Choosing the number of merge iterations.
At iteration I, GeoBPE increases the vocabulary size by looking up num p:
Vinaal
Mintt
+
Σ num num.plikey(1],
1=1
where is the number of merge iterations. More iterations yield a larger and more varied vocabulary, but each "word" (motif) is then used less frequently.
The optimal stopping point depends on the downstream application.
GeoBPE for representation learning.
For representation learning, the merge hierarchy serves inductive bias: tend to correspond to secondary structure segments and align with domain or homology hits. Here GeoBPE should coarsen high-resolution details into higher-level motifs instead of growing tremely large vocabulary.
merged token pairs
Recommendation. Use moderate number of iterations, stopping once downstream validation met- rics (e.g., AUROC, Spearman Macro-F1) plateau. In practice, this typically occurs well before exhausting all possible merges; beyond that point, additional iterations mainly create very specific, low-usage motifs that add complexity without improving downstream performance.
GeoBPE for compression reconstruction.
GeoBPE is closest to its BPE origins when used as a compression algorithm: the goal is to reduce se- quence length (increase compression), preserving geometry (minimize distortion), while monitoring the amortized bits to store the growing vocabulary in the background.
Recommendation. Allow fewer iterations and monitor the trade-off between bit-rate (i.e., BPR) and reconstruction error (RMSD/LDDT)) A general rule-of-thumb is to continue merging as long as additional iterations lowers either BPR or distortion. Later, one can choose the right iteration checkpoint to navigate the tradeoff. Thus, it is wise to stop immediately when both metrics begin degrading simultancously. On very high resolutions and moderate dataset (e.g. our pretraining dataset), this happens early on. The amortized bits used to grow the vocabulary generally outpace bits saved from decreasing the number of tokens per structure; this relationship reverses on lower resolutions and larger datasets.
GeoBPE for language modeling.
When using GeoBPE as tokenizer for protein language models, common heuristic adapted from NLP is to set the final vocabulary size such that
V
T.
1000
where L is the average number of motifs per structure and N is the number of structures. Equiva- lently, the total number tokens
Lx
suggests a target vocabulary size |V] ~ T/1000.
Table 15 shows concrete numbers for different LM scales.
35

---

Published as a conference paper at ICLR 2026
Table 15: Heuristic vocabulary sizes [V] for GeoBPE when used tokenizer for protein language models different and model scales.
LM scale
# structures N
L (motifs / structure)
[ (L x N)/1000
Model params
Toy / demo
10a
100
10
106
Small usable
10*
100
10a
10-50M
Base "GPT-small"
10a
100
10*
10
Mid-scale
106
100
10s
2 109
WVL=N_bi/100
L
MVL=N_ small/1000
M
Illustration of the heuristic for choosing the number of merge iterations based on the target vocabulary size IV for language modeling. The marked point indicates the recommended stopping iteration for datasct with structures.
Figure 12:
We implement a stopping criterion based on this heuristic: during training we track [ as merges accumulate (T decreases) and mark the iteration where the target [V] meets T/1000. In Fig. 12 this iteration is highlighted (e.g., with a star) and concretely in the run.(iter- png plots produced in each run directory by our code.
Practical tip. In practice, you can set a relatively large max_iter and let GeoBPE proceed for many iterations while logging checkpoints. After training, select the checkpoint whose vocabulary size and downstream metrics best match your target (compression, representation quality, or LM tokenizer size), rather than trying to tune the exact stopping iteration a priori.
K.3 SENSITIVITY STUDIES
We show how sensitive GEOBPE behavior is to key hyperparameters by running ablation experi- ments for selected hyperparameters, one a time.
[V| (NUM_P, P., # iterations) varies.
In Table i6 is ablation study varying NUM across and iterations per run: |V] depends both. We also combine both into the throttle [V]. We make the following observations:
1. As NUM_P values become high, the unique GeoPair keys increase exponentially. Since each iteration only looks at one key, the number of merges done falls off. Empirically, the top rows show only marginal changes to distortion iteration increases. We omitted them for brevity.
2. As NUM_P values drop too low, GEOBPE becomes more of coarsening algorithm (lots of merges, repetitive patterns are preserved but higher frequencies are lost). When merges hap- pen more often, more drift is introduced, so enror quickly accumulates. We can see distortion monotonically increase for the last run.
3. There exists tension between NUM_P and merge frequency, but glue opt is still potent enough to manage drift accumulation. We see error decrease before increasing again, when eventually merges overwhelm.
BINS E (50, 100, 300}. BINS controls the quantization of bond lengths, bond angles and torison angles connecting motifs; it trades off structural fidelity for better coarsening. Geo-Pair keys
36

---

Published as a conference paper at ICLR 2026
Table 16: Cluster of that vary NUM P; each is run; lower resolution include periodic check-
points see how RMSD/LDDT/BPR changes over iterations.
NUM
V
RMSD
LDDT
BPR
{2: 100, 500, 20, 6: 100,...)
600
1.66
0.73
36.02
[2: 500, 3: 2000, 5: 100, 6: 500, ...)
2500
1.41
0.75
41.11
{2: 1000, 3: 5000, 5: 200, 1000, ...}
6000
1.37
0.76
45.44
[2: 1000, 3: 20000}
21000
1.21
0.76
47.62
[2: 50, 5: 20, 6: 100, ...)
5200
2.11
0.68
37.24
65
1.78
0.71
30.81
237
1.77
0.70
34.00
388
1.72
0.70
35.88
[2: 10, 3: 50, 5: 1, 6: 5, 8:
521
1.71
0.70
37.33
631
1.69
0.70
38.47
739
1.71
0.70
39.54
845
1.73
0.69
40.57
109
3.96
0.53
27.26
309
4.07
0.53
30.81
[2: 2, 5: 1, 2,
508
4.23
0.53
33,46
707
4.50
0.53
35.93
of the form (Mpags I Mgr),, M. angles). Importantly, it is orthogonal to NUM.P, which control the id's of Mag and Maass so it can be tuned independenty. Fixing Hyperparameter Setting 3 we increase the number bins used discretize (CNCA w, angles angles by 2x, 6x. Fig. 13e uses the default value BINS 50; Figs. 13a & 13d use BINS = 100, 300. Increasing BINS decreases frequency of merges by observe
and the space of 7o has size ~ (BINS)3 (since there are 3 glue
around the same factor,
K is flatter for higher BINS settings decreases slower. Since BINS is an important control of resolution, decreasing it increases distortion (e.g. 3.12 + 4.19 + 5.76 RMSD). Distortion is the goal is to compress local noise into meaningful global hierarchies, introducing distortion is necessary to cluster common motifs. Setting BINS too low can misrepresent the overall structure, so recommend BINS 50 as a good starting value.
priority consideration for transfer experiments. Since
1) only shows 6.2% decrease (GLUE OPT EVERY 10). As the wall times in B3 shows, decreasing the frequency of glue glue.opt significantly accelerates GEOBPE, regardless of how many cores available. App. reveals glue_opl period P directly dictate rate-limiting term in GEOBPE's complexity. Thus, we adopt GLUE BPT...TT_EVEEE EVERY = 10 as the default setting. We also suggest GEOBPE
GLUE.OPT.EVER' E {1, 10). Fig. 13b
in RMSD and comparable LDDT Fig. 13a
to try setting GLUE.OPTLEVERY 10 balance the tradeoffs.
Table 17: We performed the following sweep over (WR, we) (order of magnitude changes wy /ww); remain- ing settings match defaults (App.
(WR, W.)
Train
CAMEO
CASP14
GeoBPE (1%)
RMSD
LDDT
RMSD
LDDT
RMSD
LDDT
(10, 0.1)
2.846
0.615
2.767
0.601
2.608
0.587
(1, ddeefulll
1.718
0.739
1.656
0.734
1.526
0.721
(1.0, 1.0)
1.552
0.764
1.546
0.755
1.412
0.743
(0.1, 1.0) (0.1, 10)
1.537
0.767
1.532
0.758
1.396
0.745
1.533
0.768
1.533
0.758
1.407
0.745
w.R/w.T {10-2.... 102). Table 17 shows w; is relatively more important than wR for construction. The interpretation is correct positions are more critical than correct orientations. We observe diminishing returns once 44 wn (ALDDTT 10-3 | 10-2).
37

---

Published as a conference paper at ICLR 2026
Table 18: We compare the default (wR, wx) setting with (1.0, 1.0), which in Table GEOBPE (1%).
resulted in lower distor-
GeoBPE
Train
Valid
CAMEO
CASPI4
[V]
(wR; wi)
RMSD
LDDT
RMSD
LDDT
RMSD
LDDT
RMSD
LDDT
600
(1, (1,0.1)-defaul
1.66
0.73
1.71
0.72
1.77
0.72
1.53
0.72
(1.0, 1.0)
1.55
0.75
1.58
0.74
1.65
0.74
1.39
0.74
2500
(1, 0.1 -dee.alt
1.41
0.75
1.50
0.74
1.57
0.74
1.51
0.73
(1.0, 1.0)
1.29
0.78
1.36
0.77
1.41
0.77
1.33
0.76
6000
(1,0.1) 0.1.) --.1)al
1.37
0.76
1.46
0.75
1.52
0.74
1.54
0.72
(1.0, 1.0)
1.23
0.78
1.30
0.78
1.37
0.77
1.35
0.75
21000
(1, 0.1 -defaultl
1.21
0.77
1.28
0.76
1.40
0.75
1.55
0.72
(1.0, 1.0)
1.05
0.80
1.12
0.79
1.25
0.78
1.35
0.76
FREE BONDS E (False, True). FREE BONDS decides whether bond lengths are free variables, or standardized to fixed values. Generally, the backbone bond lengths are very close fixed and most workflows (e.g. X-ray diffraction, NMR, Cryo-EM) that solve structures make such assumptions. GEOBPE is designed to be fully general, allowing variable bond lengths. In lieu of the known fact that they have relatively narrow ranges, we sanity check to see if GEOBPE is sensitive to FREE FREE.BONDS. BONDS. Comparing Figs. 13c & 13e lower RMSD, which is negligible.
we see the run with free bonds achieves only 1.69%
L LARGE LANGUAGE MODEL USAGE
We used LLMs mainly for polishing the writing, including prompts to check for grammar mistakes, improving clarity of mathematical notation, and formatting the text to save space.
M ALGORITHMIC DETAILS
Additional notation for algorithms.
We reuse all geometric and GeoBPE notation from Secs. 3.2 For convenience we collect the additional symbols that appear only inside the algorithmic pseudocode.
S, A Set of motif (or motif-pair) occurrences and its sampled subset used by RMSD PARTITION (Alg. 6 each u E S indexes motif !t!t))
Sy, S2 Collections of interior and terminal bond-residue occurrences used to build residue-level codebooks Ag (Algo. 18
M,c(+) Medoid set and assignment map returned by RMSD PARTITION, used as inputs glue-
optimization routines (Algos. 12
D(+), (K) Single-backbone geo-pair map and occurrence sets for new backbone t* during tokenization (Algo. 10
Σ, Emed Token dictionary used for geometric language modeling (Algos. 16 Emed contains only motif tokens; E also includes glue angle tokens.
idmed, idpin Integer maps assigning token IDs to motif medoids (k, and to glue-angle bins (type, b), respectively.
((()) Token sequence encoding backbone t, obtained by alternating motif and glue-bin tokens (Algo. I7p.
h. h 6 Upward and downward TreeLSTM states at node i in the up down encoder (Al-
gos.
proo Final residue-level and protein-level embeddings produced by the up-down encoder on the merge hierarchy (F( (Algo.
ree ",,
38

---

Published as a conference paper at ICLR 2026
Algorithm RMSD PARTITION motif pair occurrences
{u- 1,..., M) with ((.)) sk, common span length I, in + 1, and either Vu, k. = N(tu) or Vu,, ku < N(ta); target K 2 1: optional Mmax, T, S. ........... Ensure: Medoids .M [m1..... mx) } C S and assignments c: (1..... M) + {1,..... ......K). K).
Require: Motif-pair occurrences
hi ----,
1: For each u E S, compute x, E IR 1R3L*3 via COMPUTE COMPUTE..OORDS'// (t.RDS((.:kk))
2: Let A C S be uniform sample without replacement of size min Ammxx) (or A = S).
3: Build D E R |AX|IA with Duo
BESSCHRRMSD2 x.,) for u, A.
4: Initialize M + {mi..... m as K distinct uniform indices from {1..... AA}.
5: fort = 1 to T do
6: Assign: c(u) + arg (i... Da,,, for all u E A.
7: Update each j: C = {u E A: c(u) = =j]. Cf
0, reseed my uniformly from A; else
m/ arg min
Σ
Dav.
WEC, DEC₁
8:
If EX Dm break; else set my + m/ for all j.
K
9: end for
10: Map M = {m...... mx) (indices in A) to M = ff....... .uk (indices in S).
For each u S, set c(u) < arg min..et.....K)) KABSCH.RMSD(X X
return M and c(-).
Algorithm 7 KABSCH RMSD(P. Q)
Require: P.Q E n with 3L.
1: P Σ, P. -= 1 Σ, Q;
2: P P-p. ==
3: H = PTQ. UEV = SVD((H)
4: = UV det < 0, set V + -V, and recompute R UVT
5: Qaligned = -Q R +p
6: return
1 Σ
P:
Quligned
Algorithm 8 M.DDDIS a precomputed distance matrix
Require: Symmetric E IR N number clusters K, iterations T, tolerance s. Ensure: Medoid set {mı.... mgk } and assignments c(-) on {1....,N] N).
1: Initialize medoids [my} as K distinct random indices.
2: for t = 1 to T do
3: c(u) + arg min, Du.my for all u K do
4: for
5:
C {u : c(u) = jf = 0, re-seed my at random
6:
m < arg minuee, Evec, Duv
7:
end for
8:
If 2f D <
break; else mú f- m's for all
9:
end for
10: return {my} and c(-)
39

---

Published as a conference paper at ICLR 2026
Bacobano K
Cacesarc
Ba 1012
LOOT
Lonool PAOD 4.33
Lxeest MIDD 335
Kintea Eanry
(a) BINS 100,
100,
True, OPT
Comprecee v Nr 301 IFF munds
Backten BVCD & LCDT Wt
Agancc irottaciio
1500
Racibo KPRg
LD3 (meas)
401 <10
(c) Baseline for Sensitivity Ablations (default settings)
Compreaaion en for 301 SHE munds
Ractbonn
69%
L02T Ineaet
129 199
208 289 120 30
330
160
400
900
(d) BINS = 300, BONDS True
332
POU Fo P68
Nes (13 LoOT tmeest
(e) FREE.BONDS
True
Figure 13: Sensitivity analysis BINS +- {50 (default), 100, 300)), GLUE_OPT_EVERY {1, 10 (default) and FREE_BONDS + {True, False (default)). Rest hyperparameters match defaults (Settings B. We show GEOBPE (1%) progress plots at ITER 300 for ablation settings. T. is avg. tokens per structure; [V] see K.2 for details.
40

---

Published as a conference paper at ICLR 2026
Algorithm 9 STEP one GEOBPE merge iteration
Require: Current segmentations (P(r))T= and merge hierarchies [F()))=_ (frontier leaves
of Flr) equal p(r)); priority-ordered map D with keys T(k) = (p(k). - (xx)), and
values (K); current vocabulary V (map: key prototype set); boundary-glus quantizers
QGCINIa, Qw: Qa: optional glue mode E (none, each, a and, if all, period.
Ensure: Updated ((PlT)), (,
Select the merge key.
((p",-"" -,,",,, O(n))) + FRONT(D).
Write each occurrence as (4, R.) E O(**)
with L = =M and R = M (1-)
q+1r"
2: Prototype assignment (create-or-assign)
3: if p" 1 (no prototypes yet) then
4: Gather concatenated spans {M per from original 1, for all (L, O(s²) (identical
length).
5: Run RARSIPARRITION (Alg. 6 to obtain medoids and c O(x*) - + (l,.... ... K1-13.
6: Define A (TT*²)) ) Kict (medoid spans' internal parameetr tuples).
j=1
7: Update vocabulary: V(x*] An and set p* 0.
8: else
9: For each occurrence, set c(C, R) = arg min, RMSD ( )I using V[n=*].
end if
11: Greedy, non-overlapping merges (and hierarchy updates), For each backbone ty, sort currences by and choose a maximal disjoint subset S(r) left-to-right. For every (C, R) E S(r) with label j c(C,
1. Form merged motif M M
(tx) and overwrite its internals by the prototype:
(e, f, 1. (T.)) M + ngu".
2. Update segmentation p(r); replace (C, R) by M.
3. Update hierarchy F(+); add parent node for span [p:r] with left child the node of L and right child the node of R; update the frontier (replace the leaves by their parent the frontier again equals p(r)).
4. (Optional) single-boundary glue opt at link p-1 - p if mode=each; re-snap the three boundary angles.
12: Update counts and priorities in D. For each merged (C, R):
1. Merged pair decrement: remove this occurrence from O(k*); let the new count be Gnew- Erase Told = (0. -00 x²) and, if Caew > 0. insert (0, - Cnew, ((") O(x*).
2. Neighbor decrements: with neighbors L and R (when defined), compute hy
COMPUTEGFOKEYY(C
R+). For each k E (kr,, kn)) whose count decreases to Caew, erase (p(k), Cold, k) and, if Cnew > insert -Caow, k).
and kR COMPUTEGROK
3. Neighbor increments: compute KY
and K's COMPUTEGROKEY((M R+) (when defined); increment their counts and (re)insert with pri- Cnew, k), where p(k) = [[k & (V)(.)).
COMPUTEGEOKEY((C_,,
orities (p(k), -new:
13: (Optional periodic global glue opt). If mode=all and the schedule triggers, apply GLUBOP- TALL (Alg. [22 all modified backbones; recompute keys for their adjacent pairs, and for every affected key k., perform the same erase/insert priority update with p(k: 1 [ & dom(V)). If FRONT(D) then exposes a recurring key (p 0) promoted by glue refinement, immediately re-invoke STEP (no clustering).
14:
return {P(()), (F(()), D, and V.
41

---

Published as a conference paper at ICLR 2026
Algorithm 10 TOKENIZE use learned GEOBPE vocabulary to tokenize new backbone
Require: New backbone 1* (length N); learned residue codebooks A3, A2; learned vocabulary V whose pair-keys are ordered by training insertion, written
Order(V) (K1, Riy
with each Ky mapped to its fixed prototype set Ane {11(=0)
(K( K mel, quantizers
Jj=1
QBCNCA, Qw Qs: optional glue mode E {none, each, all} and (if all) a period P.
Ensure: Tokenized segmentation p(+) and merge hierarchy F(+) for 1".
Per-residue init (no new clustering). Set p(+) "M. N.N). Assign each
residue motif to the nearest element of A3 (interior) A (terminal) by Kabsch-aligned RMSD, and overwrite internal parameters accordingly.
2: Initialize hierarchy. Let F(+) be binary forest whose leaves (in order) are {M (M
Timli
its frontier equals pl+),
3: Optional one-time global glue. mode=all, run GLUEOPTALL (Alg. on t* once; snap (0CNCA w. Φ) to Q.
4: Build single-structure geo-pair map. Compute D(+) < BINHELPER ,,),,,()0)) ,)
(Alg. I1]. which maps any geo-pair key x to its occurrence ( (x) on t*
(Uses
COmPUteGeOKER with raw medoid internals and quantized boundary glue.)
5: Apply learned merges in training order (no new keys).
6: for
to |V| do 7: K - f (the s-th key Order(V)).
8:
if D(+) then
9:
continue end if
10:
Assign prototypes (no clustering). For each (C, E (1:) with L Mbn R
11:
M((e)
g+1rr set
c(C, R) arg min JE(II...KInI]
RMSD
MMC)) par 11(")).
12:
Greedy disjoint merges & hierarchy updates (left rright)... Order (+)) by in- creasing pt select the maximal disjoint subset 8(+), For each (C, R) E S(+) with label j cL, R):
1. Form M = MM'' and overwrite its internals by 11!*).
2. Update segmentation p(+); replace (C. RC) by M.
3. Update hierarchy FF(); add parent for span [p:r] with left child the node of L and right child the node of R; update the frontier so it equals the new p+)) If mode=each and the boundary p-1 - p exists, apply GLUFOPT (Alg. 19 snap its three angles to Q.
13: Maintain the single-structure map. For each merged (L, E s(*), update ((*) lo- cally: remove the occurrence of decrement keys neighbors (C-, L) and (R, R*) (when defined); insert the new neighbor keys (C)MM) and (M, R+) using COMPUTE- GEOKEY.
14: Optional periodic global glue. If mode=all and mod P = 0, run GLUEOPTALL on t": then recompute keys adjacent to changed boundaries via COMPUTEGHOKEY and refresh their occurrences in D(+).
15: end for
return p(+) and FF(),
42

---

Published as a conference paper at ICLR 2026
Algorithm 11 BINHELPER - build geo-pair map for one backbone
Require: Backbone t* with current segmentation p(+); quantizers Qsenca, Qw Qs. Ensure: D(4) R >> (x))
Initialize D(*) 0.
2: for each adjacent pair (C, R) in p(+) do
3: A: f- COMPUTEGROKEY
R. using:
Raw medoid internals for C and R (as assigned in initialization prior merges); boundary glue (OCNCA, w. Φ) snapped by Q.
4: Insert the occurrence (C, R. into (K)..
5: end for
6: return D(+)
Algorithm 12 GLUEÖPTALL - global differentiable inverse kinematics over glue angles
Require: Medoids M and assignments c(-) from RMSD PARTITION; occurrences S {u} with
spans M((l). batk.. target frames F: (R; (t) ,4 (2(00) (with FT from SEEDTRIAD); weights
(wn, wy); optimizer steps T and step size y
Ensure: Updated glues {r(() and frames (
those of its medoid m(u) me(u) 2: Init glues: copy original r! for all backbones and links 1:N (these the opti-
1: Snap internals: for set internals of M t,t
(tu)
mization variables)
3: Loss:
(+))
C(T) Σ
w log log(( R 11, w. 17 t (2.00
4: Forward kinematics (FK): with A =F Fr.(0.
F.D p:00) 2 (r(); current internals),
(() from internals and r-(0) 10,NCA the 0++1
5: Optimize glues (autodiff); for 1:T:
6: run FK, evaluate L; backprop VrL: update all r(()
7: wrap v 6E (-x, T] project @CNCA E (0, T)
return (T()), }. (
8:
Algorithm 13 BINARY TREE-LSTM CELL (Tai et al., 2015)
Require: Left (hg, ce) E Rd Rd. right (,,,,,); W gadx2d, b 5d
Ensure: (hip cp) E IRd x IRd
ff
he
1: u
hr
fr
<- Wa +
2: fr. (-);
g g tanh()))
3: fp 2 ci
1.00 1
4: hp @-@0ttanh(p) tanh(cpp)
5: return (p):
43

---

Published as a conference paper at ICLR 2026
Algorithm 14 DOWNWARD BINARY TREE-LSTM CELL
Require:
Parent downward (N Ep) Rd x R sibling upward (,); W E Rddx2d be Rdd
Ensure: (hes Cx) E Rd
Rd
hp 1: ut
fg
-- W u +
11s
2: 1, fp Ja 0+ (-);
g+ g+- tanh
3: E. f.
O +
C. +
4: 1.t00tanh(a..
5: return (he. 0e)
Algorithm 15 UP- DOWN TREE ENCODER ON A FOREST (one protein)
Require: Protein 1, with N(r) residues; binary forest jF() (V(r), E(() whose frontier (in
order) is p(r); leaf embeddings {er (r) Rdi
TEl (e.g.. ESM3 internal-edge topological
order E(r) {(p. f}r roots R(O) v(r); parameters θ {W, W b); combiner
{concat, sum]
Ensure: zprot Rdd;; {
Upward. For leaves S N()); h
(7) .. +00 For (p,tr) E E(F) in order:
(1p,, + TREELSTMCHLLM of n1, W,b) (Alg. 13
2: Super-root. hRR + -1 h/; set node SR with (hSRR <$==0)
3: Downward. (hSR, ēSR) 0). For each tree rooted at R(T), recurse: for internal p with
children (f,r) and given (hyp Tp),
(he, (4) <- DOWNTREELSTM Cp), (14, c)); W.D), (hr, + DOWNTREELSTM Ep), (hi, o)); W. b) (Alg 14).
4: Representations. For any node u: ut x- hyg
5: if + concat then
6:
#r, + [h ue ad (i=1:N(r)); zprot +- [nnRi ušR Rdd
-res
7:
else
8: 2res + h + ut E ERd (i=1:N(r)); 2prot + ngR + usR E Rd
9: end if
10: return srrot (2res (((r)
44

---

Published as a conference paper at ICLR 2026
Algorithm 16 BUILDJJINT medoids then glue-angle bins
Require: GEOBPE vocab V {x + Ar3 with key introduction order (x:(1), ,..., ,,,,));;; medoids
An
(8) Kix (11 Jj=1 glue quantizers QUCivAa Qw, Qs with bin centers {uī ( b= {1g B.
Bs BB
Ensure: Dictionary Σ: maps idmed (H, - {I,..... Emed and idain (type E (0,w; 0),
[|mmed|+ +l..,......... [
1: Medoids (in introduction order): Emed f [].
2: for 1 to S do
3: for j = 1 K K do
4: Append (x:(*), to Mieed set idmed (x( to its index.
5: end for
6: end for
7: Glue bins (appended after medoids): let M = [mmed]-
8: idsin (0. M +- b; idsin (we, M + Bg + b; idhin (1, M Bo B b.
9: Σ < Emed U {all glue-bin tokens} (optional: add BOS/EOS)
return E, idmedd idbin
Algorithm 17 BACKBONETOSIQQUENCE tokenize segmented backbone
Require: Protein 1,, with segmentation p(() (Mp Mpacipe ); for each (1) its
key Km and medoid jm (prototype II (M(M)), boundary glue I 9m 4m Wam : 8gn +1} for m 1:M-1; quantizers Qo, Qu: Qøi token id maps from Alg. 16
Ensure: Token sequence z((r) = (II,..., .1) L
1: (r) + [] (optionally prepend BOS/append EOS)
2: for
to M1 do
3: Motif: 2T(T) a=(")a (imme
4: if m < .M then
5: Glue quantize: bo+ Qo 9m (x() apppedd (idbin (0, b6)): a(T) .sppapdddddid... (w. bw));
bw x Qw (w(wgm): bo + Q0(0gm+1)
6: Emit (fixed order):
2("" Pra.apppddida... (0, b&))
7: end if
8: end for
9: return (7)
45

---

Published as a conference paper at ICLR 2026
codebook and quantize all residues Require: Backbones {((t),... ..... , with lengths N(r); targets K3 (interior bond-residues), K2 (terminal cond-residues
Algorithm 18 RESINITTOKENS initialize
Ensure: Codebooks A3 = S ( أー= A2 = {1g K2 jبmر labels e(0), c backbones with per-
residue internals set their prototypes
I: Collect occurrences:
S3 = (u= : -SiK N(7, ( interior), S = :i= M terminal).
2: Cluster interiors: SASS.PPARTTION Ka) - medoids M3 = {mi (3) Ks and labels c
S3 + +1,...,,K, K,}
3: Define interior prototypes: for j L:K3, let u² m (3)
and
II (CN,CA, CAC. C.N, o BCACN, Diae. Wiy @i.,) (omit undefined terms for a single residue if using minimal parameterization)
4: Quantize interiors: for i) E S3 with j (u),
(4, 0, d, w, 6) MS -
5: Cluster terminals: 2MSSDPPARTITION(S K2) - medoids M2 = [m 1バ and labels (0)) S2 (I,...,K K2}
6: Define terminal prototypes & quantize: for j = 1:K2, let u² = =mg and set the appropriate
terminal tuple (e.g.. E. NNCC CAC.C O,CAA) as (2), for = (T, (())) with j = cl (u),
(e,0)
(()) N(r)
7: return Aa, A2, c03) and updated backbones
Algorithm 19 GLUEOPT - single- boaale-bundrry IK to absorb one rounding drift
Require: Occurrence u with motif MM(())
(fu) and medoid ma(u) from PARTIIINN:::ON; frames (Ep with Fr (tx) from SEEDTRIAD; weights (WR, wt); steps T, step size r) ((t.) Aa
Ensure: ((f))
1: Snap internals: replace (ta) th, by its medoid M
Mia
set T [mmedd
Trut Min skmggw)
2: Drift: 7000 +- Thnt
L,ck. ; AT,, ,, + 7.00 (Tmed)-1
3: Vars: (ta)
fu-1 1. -1, P... are the only optimization variables init to originals
4: FK: keep p,,(41) fixed; for any (f...
(())) = F Gi--1(1 iu-1)
(t.((.) ((t)) Timee
5: Loss:
Cu (,(((()))) ws log( ((l.))
(R((() ( (t 1|2 wi will
(t.))))) tk
6: Optimize (autodiff): for 1:T: FK & evaluate Lu: compute V Vr(eo) Lui update (t.) (c.g., Adam, Ir 7)); wrap vh e (-x, T]. project OCNCA e (0, 7: return (t.) (()
1, k..
46

---

Published as a conference paper at ICLR 2026
Algorithm 20 GLUEÖPTALL (WRAPPER) apply rounding, then call the core global IK over glues
Require: Medoids MM and assignments <(-) from RMSS_PARRTITON
of occurrences S {u} to round, each M cached original exit frames for each backbone (seeded by SEED-
TRIAD); histogram bin centers & thresholds for (w, OCNCA ()); prior weights; loss weights (WR, We).
Ensure: Updated glue angles (snapped to bins) for all boundaries in all affected backbones; recom- puted frames (F)(())
1: (Quantize internals) For each u E S, replace
M
by copying the medoid's internal coordinates (hard assignment).
2: (Ensure targets are cached) For each backbone t, ensure original exit frames ((Race [J], =0(( [ -1 are available (compute once if missing).
3: (Global glue optimization) Call the core routine GLUEOPTALL on all backbones:
GLUEÖPTALL (1), BinCenters, Thresholds, GluePrior, WR, we).
which jointly optimizes every boundary's glue triplet r, {0CNCA wi, p4+1) via differen- tiable FK and snaps each angle to the nearest histogram bin.
4: return updated glues and frames (T)(()))
Algorithm 21 BINGEOPAIRS build the dictionary geo-pair occurrences (with hicrarchy)
Require: For each backbone {(T); its current segmentation p(r) ( Mase and
its merge hierarchy FF() whose frontier leaves, in order, equal p(r); precomputed boundary- glue quantizers QςcNe Qu., Qo (used only pair boundaries).
Ensure: A priority-ordered map DD from keys to occurrence sets, with ordered keys (1:)
(p(k). (())), x) where p(k) = [[x & dom(V)) indicates the key already has prototypes.
1: Initialize empty ordered map D.
2: for r = T do
3: Let (M pp ....,
M ((t Mpm- :TM))
(2,)
be the frontier leaves of FF(T (these equal p(T)).
4: for = to Mr do
(C, - (M Mps+r:95))
((-)
5:
6:
+ COMPUTBGEOKEY((C.R
(raw medoid internals inside C, R: boundary glue
quantized by Q)
7: Insert the occurrence (C, R) into (k)).
8:
end for end for
9:
10:
for each key x with nonempty ((x) do
11: Set p(K) + 1/x & dom(V)].
12: Insert (0(k), + (x)) into D.
end for
14: return D.
47

---

Published as a conference paper at ICLR 2026
Algorithm 22 COMPUTEGEOKEY discrete key for an adjacent motif pair
Require: Adjacent motifs L ~M R M on backbone t Ensure: Canonical, hashable key 6. for the geo-pair
Interiors (as stored post-quantization):
[nt(C)- (1e,0, u, 011 (T.) Int(R)
-,,,0, (T))
(kept unchanged in the key). 2: Boundary glue (quantized):
Tg
(CCCA q
: 19+1).
T. (Qo Qu(wi)): Qo(dg-1))
(Q. wrap to a fixed 2T interval before snapping).
3: Canonical record & hash:
(Int(C), Fg Int(R)),
K K HASH(rec)
4: return
Algorithm 23 COMPUTE_COORDS. Internal + Cartesian for a renddresddue motif i:j
Require: Internal geometry for r = i.....j (bond lengttssaangees/dihccdaaa))))
Ensure: X E R30)-(4)x3 for (N; CA,, Ci...., N,, CA2, Cj)
1: Seed residue i: (N,, CAi, C+) SEEDTRIAD(i)
2: C + [Ng, CA2,, Cj]
j do 3: for PLACEIDHEDDA((((333))---CCCC
4: N,
C OCACN 1)-1)): (app)nd((N)): 5: CA, < PLANCEDHEDDAA(((1)))CC &N CN-CA, OCNes w--1): C.append(CA.) CCC- ONcaC sp.); C.append(
6: C, + PCA
7: end for
8: X [ CA2, Cn ..., Nj. CA,, Cy]
9: return X
Algorithm 24 PLACEDIHEDRAL(A, b, c; L, 3. T)
1: Right-handed local frame at c:
6 b
n (b- a) x b
ñ n b.
|c bll
[bb a) x bll
2: Local offset:
d = L-Loos 3|b 7 sin Bl
sin sin B n.
3: Return c
Algorithm 25 SEEDTRIAD(F) - seed triad for residue
Require: Canonical seed (N., CA., C.); target LCAC CAC. LN-CA 0 NCAA = AC
CN-CA and
1: Place CA, on the ray from C, toward CA. at distance LCAC-
2: Let u N, CA, and C, CA,. Rotate about axis to achieve angle AN then rescale to length Ly-CA; translate by CA, to get N,..
3: Set C, + C. and return (N, CA2, C,).
48


Reviews:
[{"review_id": "j4fkTkGR2d", "reviewer": "Reviewer_MhKe", "summary": "This paper introduces Geometric Byte-Pair Encoding (GeoBPE), a novel algorithm for protein structure tokenization. The authors identify key limitations in existing protein structure tokenizers, particularly the popular VQ-VAE variant. They argue these methods suffer from fixed token sizes that prevents multi-scale resolution, and produce vector-based tokens that lack the interpretability and hierarchical structure of methods like Byte-Pair Encoding (BPE) in natural language processing. GeoBPE is designed as a geometric analog to BPE. It works by iteratively building a hierarchical vocabulary of structural motifs, which involves: iteratively identifying the most frequent pairs of structural motifs and clustering their occurrences using k-medoids, then replacing these occurrences with a representative prototype from the new vocabulary. This local quantization inevitably introduces geometric errors, and the authors correct this by optimizing the boundary glue angles through differentiable inverse kinematics under SE(3) loss. Experimental results demonstrate that GeoBPE achieves high compression, highly data-efficient, and generalizes well to unseen structures. Furthermore, its tokens are shown to be interpretable, aligning with CATH functional families.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. GeoBPE produces a hierarchical vocabulary of discrete structural motifs, which is a significant advantage as the resulting tokens have functional meaning. Experimental results show that the tokens align with CATH functional domain annotations and are validated by expert case studies.\n2. GeoBPE addresses the fixed token size limitation of VQ-VAE. Its hierarchical BPE-like approach allows it to capture multi-scale features, and its performance can be tuned to trade compression (BPR) for distortion (LDDT/RMSD), which other tokenizers with fixed dimensions cannot do.\n3. The paper's explicit handles the geometric drift. The use of differentiable inverse kinematics to optimize glue angles is a principled solution to the problem of maintaining global structural integrity while performing local quantization. Ablation study shows this step is essential, as omitting it causes reconstruction RMSD to jump from 1.66 to 4.39.\n4. GeoBPE shows excellent out-of-distribution generalization. It maintains a test/train distortion ratio of 1.16-1.28, whereas the VQ-VAE baseline reportedly degrades by as much as 6.4x on the same test sets.", "weaknesses": "I am not very familiar with structural biology and thus may not be able to provide deep insights into the algorithmic details of GeoBPE. My primary concerns are focused on the experimental validation and its implications:\n1. The SSLM experiments (Table 7) indicate that the language model trained on the GeoBPE vocabulary struggles significantly with unconditional generation. This is evidenced by the low scTM (less than 0.3) and designability (less than 5%). Performance at this level is insufficient for most practical applications. This finding leads to serious concerns about using GeoBPE as a foundational component for practical development (e.g., for building multimodal structure and sequence language models, like ESM3).\n2. The paper demonstrates GeoBPE's high data efficiency, with appendix results showing that using only 1% of the training data will not introduce performance loss. While this is an advantage, it could also be interpreted as a potential risk. This finding might suggest that the model has a low capacity and hits a performance ceiling almost immediately, making it incapable of capturing the complex, subtle, and diverse structural patterns that can only be learned from large-scale datasets. \n\nGiven these points, I hope the authors will provide a detailed discussion about these concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mRjdUUjqsL", "reviewer": "Reviewer_3EbP", "summary": "This paper proposes GEOBPE, a geometric analog of byte-pair encoding (BPE) for protein backbones. Instead of tokenizing amino acid sequences, GEOBPE tokenizes 3D structures into discrete motifs, building a hierarchical vocabulary through iterative clustering and merging of “Geo-Pairs.” And it enforces geometric consistency using rigid-body optimization during BPE process. This method does not reply on neural networks to \"automatically\" learn the discretized tokens like VQ-VAE-like protein structure tokenizers.\n\nGEOBPE supports downstream applications like function prediction and shows strong interpretability by aligning the tokens with CATH functional families. The tokenizer is also used to train generation tasks, though it's not the focus on this paper's experiment section.", "questions": "1. Will all the code be released?\n\n2. Line 150 has notation conflicts with Line 161, \\phi_{i+1} should be \\phi_{i}?\n\n3. Even though GEOBPE can trained with only 1% PDBs to get good performance. The algorithm complexity seems to be high and prevents GEOBPE to scale further to large scale datasets (current training data is around 48k proteins). Could you please analyze the complexity of your algorithm and the empirical time used for GEOBPE?\n\n4. Suggestion to modify Figure 2: based on my understanding, the top-right two motifs in “step 1” subfigure stand for “prototypes” and the arrows mean that each motif is mapped to those prototypes. And then this pair of prototype is selected to be merged in BPE process.\n\n5. How does GEOBPE tokenize a new unseen structure once the vocabulary is learned? Is there a decoding or segmentation step described explicitly? Not sure if I missed anything.\n\n6. Concerns for GEOBPE to generate good structures when coupled with LMs. For example, in ProteinAE [1] Table 2, they can perform well on \"unconditional protein backbone generation\" when compared with competitive.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- **Novel conceptual framing**: this paper continues an \"old fashion\" way to heuristically tokenize structures. GEOBPE extends BPE from linguistic to protein structure data, introducing a hierarchical, interpretable tokenizer for protein structures. This is process is far complicated than applying BPE on protein sequence data. And the authors seem to design a way to solve it.\n\n- **Interpretability**: A highlight difference for GEOBPE with popular VQ-VAE-like structure tokenizers is the interpretability: (1) the vocabulary construction process is interpretable, unlike VQ-VAE relies on automatically learning from data; (2) the hierarchical vocabulary offers intuitive geometric primitives that align with known structural motifs in CATH.\n\n- **Empirical strength**: As shown in Table 1, tokens from GEOBPE contain rich functional information, and performs the best on related benchmarking tasks.", "weaknesses": "- **Per-residue task ambiguity**: Since tokenization is no longer residue-level, how can we make GEOBPE's tokens aligned for per-residue benchmarks? This is a crucial design and evaluation gap.\n\n- **Severe Presentation issues**: Excessive information density makes the paper difficult to parse; algorithmic pseudocode dominates over conceptual clarity. And referring readers to appendix for very detailed algorithms in appendix and replying on these algorithms to inform what is happening inside the method is a very bad way to deliver ideas. The paper would benefit from improved figure explanations and appendix referencing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XqtPxu1Nwf", "reviewer": "Reviewer_F5Hs", "summary": "The paper “Protein Structure Tokenization via Geometric Byte Pair Encoding (GEOBPE)” proposes a hierarchical, geometry-aware tokenizer that discretizes protein backbones (N, Cα, C) into interpretable structural “words.” Inspired by text BPE, it merges frequent geometric motif pairs using clustering and inverse-kinematics “glue” corrections to preserve structural consistency. GEOBPE achieves superior compression–distortion tradeoffs, strong out-of-distribution generalization, and improved downstream task performance over VQ-VAE-based baselines. Its tokens align with functional protein motifs (e.g., CATH domains) and enable structure language modeling and backbone generation, offering a scalable and interpretable discrete representation of 3D protein geometry.", "questions": "- What is the wall-clock/time/compute cost to build the vocabulary (i.e., cluster motifs, compute glue corrections) for e.g., the full PDB (hundreds of thousands of chains)?\n- How sensitive is the performance (compression vs distortion) to choices of K (number of medoids), merge iterations, and glue‐IK weights? Is there a guideline for selecting these hyperparameters for a new domain (e.g., membrane proteins or antibody loops)?\n- The work focuses on backbone only. Do the authors envision an extension to include side-chain geometry (Cβ, other atoms) or ligand binding sites?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The GEOBPE idea is elegant: motif clustering + quantization + glue angle correction giving both local motif reuse and global structural consistency, and iteratively building a vocabulary. That hierarchical tree/merge structure is meaningfully analogous to BPE and advantageous in representing protein structure at multiple resolutions.\n- On compression vs distortion, GEOBPE traces a favorable Pareto front.\n- Out-of‐distribution generalization appears very good compared to VQ-VAE\n- Downstream tasks benefit from the tokenization\n- The tokens learned are not just “latent vectors” but actual observed motifs (medoids) with real structure, and they show alignment with functional families (CATH) and interpretability in case studies.\n- The authors highlight that they trained on much less data (~7% or even ~1% of typical size) and yet got strong results (Sec 5).", "weaknesses": "- The method involves many steps (clustering motifs, quantizing, global inverse kinematics to correct glue angles, iterative merging). It may be computationally heavy (they discuss complexity in Appendix H). The practicality at very large scale (hundreds of millions of structures) may still be challenging, and training time/compute cost details may be less emphasized.\n- The method depends on choices: how to cluster, how many medoids (K), when to merge, how to set glue‐IK weights, how to quantize, etc. These may introduce sensitivity and hyperparameter tuning cost. The robustness across many protein fold types or extremely large or unusual proteins may vary.\n- The method tokenizes only backbone geometry (ignores side-chains, ligands, post-translational modifications). For many functional tasks side-chains or ligand contacts matter a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gFIDyUM44", "reviewer": "Reviewer_4qDK", "summary": "The authors present GeoBPE, a protein structure tokenizer which produces hierarchical tokenizations of proteins using a byte-pair encoding-like technique. In brief, residue-level backbone geometric features are clustered into a vocabulary, and then iteratively merged to reduce the size of the vocabulary and make it less sensitive to noise. This induces distortions in the resulting geometry of a protein, which they resolve by making corrections at each stage of the tokenization. The approach is remarkably effective, resulting in increased compression and reconstruction accuracy over strong baselines. ProToken and GeoBPE form a Pareto front, trading off reconstruction accuracy for compression, with GeoBPE performing roughly 10x better on BPR (compression) than ProToken.", "questions": "In this emerging field, there is some debate over what level and type of structural information each token should have. That is, each token can represent not just information internal to the residues represented, but more global information about how the residue is positioned in the protein. GeoBPE is essentially backbone-local--it does not use information about nearest spatial neighbors at all. By contrast ESM3's tokenizer does contain substantial information about residues which can be very distant in the linear sequence of the protein, yet GeoBPE does better on global reconstruction tasks. This is perhaps surprising and I wonder if the authors could comment on it.\n\nGenerally the peptide backbone bond lengths are very close to fixed (in fact there are usually restraints on these when solving structures by X-ray diffraction, NMR, or Cryo-EM). I wonder whether these are actually needed in GeoBPE; do the authors find clusters of motifs where these are very different, or are the bond lengths similar between all motifs? My guess is that the clusters differ primarily in their backbone dihedral angles", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "Originality: I think the authors have come up with a creative and quite distinct solution to the problem of structure tokenization--while many others have tried to improve on the VQ-VAE idea, they have not broken out of that fundamental paradigm. GeoBPE is entirely different and at the same time simpler and more intuitive. \n\nQuality: The analysis is thorough, covering structure reconstruction, codebook usage, and several downstream tasks of high biological relevance. \n\nSignificance: GeoBPE substantially outperforms most other structure tokenizers in nearly all evaluations, and provides clear trade-offs between compression and accuracy. I imagine that it will be of great utility to researchers modeling proteins, and hopefully will find use in a protein LM soon.", "weaknesses": "Clarity: The writing in section 3 could use a lot of improvement. I found it difficult to follow, and I'm still not entirely sure I know what all symbols mean. Many terms are only properly defined _after_ they've been used multiple times. The notation section is clear, but takes a lot of space that might be better given to (clearer) descriptions of the method itself.\n\nIt would be very helpful to visualize the hierarchy of structure tokens/motifs. I imagine, for instance, that there should be a large group of loop-like structures, with a smaller number of tokens devoted to common structures like alpha-helices and different kinds of beta sheets. Given the local nature of the representation, I would expect high level clusters corresponding to the 8 basic secondary structure elements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)