Protein Structure Tokenization: Benchmarking and New Recipe
Authors: Xinyu Yuan
Zichen Wang
Marcus D. Collins
Huzefa Rangwala
Venue: ICML 2025 poster (2025)
Keywords: protein structure tokenization
benchmarking
VQ-VAE
Primary area: applications->health_medicine
TL;DR: A comprehensive benchmark for protein structure tokenization methods, focusing on fine-grained local protein structure representation, with a new tokenizer recipe showing enhanced performance

Abstract: Recent years have witnessed a surge in the development of protein structural tokenization methods, which chunk protein 3D structures into discrete or continuous representations. Structure tokenization enables the direct application of powerful techniques like language modeling for protein structures, and large multimodal models to integrate structures with protein sequences and functional texts. Despite the progress, the capabilities and limitations of these methods remain poorly understood due to the lack of a unified evaluation framework. We first introduce **StructTokenBench**, a framework that comprehensively evaluates the quality and efficiency of structure tokenizers, focusing on fine-grained local substructures rather than global structures, as typical in existing benchmarks. Our evaluations reveal that no single model dominates all benchmarking perspectives. Observations of codebook under-utilization led us to develop **AminoAseed**, a simple yet effective strategy that enhances codebook gradient updates and optimally balances codebook size and dimension for improved tokenizer utilization and quality. Compared to the leading model ESM3, our method achieves an average of 6.31\% performance improvement across 24 supervised tasks, with sensitivity and utilization rates increased by 12.83\% and 124.03\%, respectively. Source code and model weights are available at https://github.com/KatarinaYuan/StructTokenBench.

Protein Structure Tokenization: Benchmarking and New Recipe
Xinyu Yuan 12
Zichen Wang
Marcus Collins?
Huzefa Rangwala 113
Abstract
1. Introduction
Proteins, linear sequences of amino acid residues that fold into complex 3D macromolccules, drive the essential ma- chinery of life. While protein language models (PLMs) trained on residue sequences have emerged as powerful tools for deciphering the underlying "language" of proteins and decoding evolutionary patterns protein function (Lin et al., 2023b), their reliance on sequence tokenization (e.g., per- residue or sub-word methods like BPE (Ferruz al., 2022)) overlooks critical dimension: the 3D structural context. Residues adopt diverse geometric conformations to perform biological functions, and their structural interactions-not just sequence-dicte protein function and behavior (Abramson et al., 2024). To address this gap, recent work has explored protein structure tokenization (PST), which encodes local 3D context into discrete continuous representations, Dis- crete structural tokens enable powerful techniques like lan- guage modeling of protein structures, and large multimodal models (Yin et al., 2023) that incorporate other related data modalities, such as protein sequences and function descrip- tions (Hayes et al., 2025).
Recent years have witnessed a surge in the devel- opment of protein structural tokenization meth- ods, which chunk protein 3D structures into dis- crete or continuous representations. Structure to- kenization enables the direct application of pow- erful techniques like language modeling for pro- tein structures, and large multimodal models to integrate structures with protein sequences and functional texts. Despite the progress, the capa- bilities and limitations of these methods remain poorly understood due to the lack of unified evaluation framework. We first introduce Struct- TokenBench, a framework that comprehensively evaluates the quality and efficiency of structure tokenizers, focusing on fine-grat fine-grained local sub- structures rather than global structures, as typical in existing benchmarks. Our evaluations reveal that no single model dominates all benchmarking perspectives. Observations of codebook under- utilization led us to develop Aseno, a sim- ple yet effective strategy that enhances codebook gradient updates and optimally balances codebook size and dimension for improved tokenizer utiliza- tion and quality. Compared to the leading VQ- VAE model ESM3, our method achieves aver- age of 6.31% % performance improvement across 24 supervised tasks, with sensitivity and utiliza- tion rates increased by 12.83% and 124.03%, respectively. Source code and model weights are available https://github. com/ aettarinnytaamstttnttttoeeeeeee
Current PST methods fall into two categories: (1) heuristic methods relying on domain-specific knowledge and hard- coded rules to extract structural-bass tokens (de Brevern, 2005; Durairaj et al., 2020); and (2) deep learning methods that use neural networks learn features from protein struc- ture data. The latter further includes: VQ-VAE-basee (Van Den Oord et al., 2017) methods, which compress structures into discrete latent spaces in codebooks via reconstruction objective functions (Van Kempen et al., 2024; Lin et al., 2023a), and Freerss-Foldigg-asse which compress structures by predicting sequences capable of folding into the given target protein structure (Dauparas et al., 2022). Despite the emerging progress, the capabilities and limitations of these PST methods remain poorly under- stood due to the absence of a unified evaluation framework.
(IF-based) methods,
This work was completed while the author was an intern Amazon. 'Corresponding author. 'Huzefa Rangwala is LOA as Professor of Computer Science George Mason University. This paper describes work performed at Ama- zon. 'Mila Quebec AI Institute 'University of Montreal 2 "Amazon. Emails: Xinyu Yuan <xinyu.yuan @ mila.quebec>>, Zichen Wang <wangze921 @ gmail.com>, Marcus Huzefa Rangwala:
To address the gap, we present StruetTokenBench, com- prehensive evaluation framework (Fig. 1) that assesses PSTs across four perspectives. Unlike existing protein struc- ture benchmarks, which primarily focus on global struc- tures (Townshend et al., 2021). StruerTokenBench is de- signed to evaluate the quality of the latent space defined by the PST encoder/codebook, with an emphasis on fine- grained local protein structure representations.
Collins
<rhuzefa amazon.com>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Protein Structure Tokenization: Benchmarking and New Recipe
By evaluating leading open-source PSTs, we observe that no single PST method dominates across all four perspectives: IF-based PSTs stand out in Downstream Effectiveness (Sec. 5.2), ProTokkns performs best in Sensitivity (Sec. 5.3) and Distinctiveness (Sec. 5.4), while FoldSeek achieves su- perior Codebook Utilization Efficiency (See. 5.5). To gain deeper insights into PSTs, we conduct a series of ablation (Sec. 5.6) and scaling (Sec. 5.7) studies, which reveal the following observations:
2.2. VQ-VAE-based PST Method
As illustrated in Fig. ((a). VQ-VAE can be summarized as:
Structure Eneeder
Quantization
Strecture Decoder
x; where (1) a structure encoder maps structure a into con- tinuous representation E RLIDD; (2) vector quantiza- tion layer discretizes each 2; (1( into a codebook vector qt, E RD by selecting its nearest neighbor from a learnable codebook Q E R using distance mea- sure d(.): l = arg min, d(zi, Qj); and (3) a structure de- coder reconstructs 3D coordinates from the discrete codes U (-= 1 To handle the conddffeeretiabiiity of quan- tization, sraaight-through gradient estimation (Bengio et al., 2013) is used (see App. E.3).
q
Vector quantization affects model expressiveness mainly due to optimization challenges, not the rep- resentation format (discrete or continuous).
Structural tokens retain most of the information present in amino acid tokens but are less robust to noise.
VQ-VAE objective. VQ-VAE maximizes the log- loellike of the data using the ELBO (Kingma, 2013):
Reconstruction quality does not consistently corre- late with codebook quality, indicating that both are essential to quantify PST quality.
LELBO = Ep(g. p( (q)) KL [[p([)))]])))))))))))))))))))[ (1)
Scaling up VQ- -a--eed PST encoders yields sub- exponential benefits that eventually diminish, showing that scaling alone is insufficient for large improvements.
where p(gk (x) comprises the deterministic encoding pro- cess p( ((2|x) and the nearest neighbor selection, and p(qx) can be defined as a simple uniform prior. Thus, we obtain the KI. divergence term as constant, excluded from Eqn. 1.
We observe that current PST methods exhibit low codebook utilization (Sec. 5.5), with codebooks being severely under- utilized (e.g., up to 70% of 4096 codes in ESM3 remain inactive during inference). This inefficiency aligns with "codebook collapse" (Zhang et al., 2024a), a well-known issue in VQ- VAFs where encoders disproportionately assign inputs to a small subset of tokens, limiting representational capacity.
Besides the "reconstruction loss" pog (o((0(g0 qi)) from Eqn. 1, VQ-VAE VAE introduces a "quantization loss" [|sg(z) - 9k 12 to learn codebook vectors and a "commitment loss" B||= sg( (g(g)))1 to pull the encoder's output towards the codebook vectors. sg(-) stands for stop-gradient operator and /3 is hyper-parameter, typically robust and set to a value in [0.25, 2]. Overall, the optimization objective becomes:
To mitigate this, we further propose AsinoAseed, a VQ- VAF-bbased PST (Sec. 4.2) that introduces two techniques: (I) codebook reparameterization: a targeted recipe for codebook collapse to improve codebook gradient update during optimization, and (2) pareto-optimal codebook con- figuration: data-driven strategy to balance codebook size and dimension, maximizing token diversity while minimiz- ing redundancy. We demonstrate that AminoAseed outper- forms current VQ-VAE- PSTs by a large margin.
L = log [ +|sg))+>++s=>>)))))) qu1 -11z -llllllllgg))))))ll (2)
2.3. IF-bed PST Method
As illustrated in Fig. 1(b), IF-based PST can be summarized as:
Structure Encoder, z
Soquence Decoder
F, where (1) structure encoder similarly maps input structure x into continuous representation E IR Lx, (2) æ is decoded into protein's 's corresponding residue sequence A.
2. Preliminaries
2.1. Problem Definition
IF objective. Per-residue cross entropy is calculated by comparing F with the ground truth sequence r.
With StruerTokenBeench evaluating PST's quality, we study the problem of developing effective PST methods. A protein structure is represented by its backbone 3D coordinates x E IR I.x Nmm x3 and residue sequence r E S",, where L is the protein length, Natomss =4 4 refers to the backbone atoms: [N, C.. C, O, and S denotes the set of 20 amino acid types'
2.4. Tokenization Process
In Fig. 1(c), the structure tokenization process is illustrated as follows: a prentrained, fixed structure encoder maps the structure into discrete continuous structural representa- tions. Specifically, VQ-VAE-basee PST encoders generate codebook indices as "structural tokens" and their corre- sponding latent vectors as "discrete structural representa- tions". and IF- based PST encoders' output is directly viewed as "continuous structural representations".
'Our evaluation focuses on PSTs using backbone structure as input. While methods like Cheap (Lu et al., 2024) (incorporating all-atom structures and sequence data) lie outside this focus, they remain applicable for evaluation on all tasks (see App. F.6).
2

---

Protein Structure Tokenization: Benchmarking and New Recipe
PST Methods (a) VQ-VAE-based PST Methods
StructTokenBench Framework
(c) Tokenization Process
(d) Effectiveness
(e) Sensitivity
Reconstructin
Conformation Slato?
Quentization
Codebook
Sevviia Thoees
Phooke
Deanndrr
PS1 Indenees
Drokingg Lagre
F8T Inffeee
Saxttrrl
Sinllarty
Erooeler
Consisteaf
Codebook
Conformation Stute
(b) Inverse-Folding- based PST M eethods
Ehnxding
Distinctiveness
(g) E Efficiency
Ground Truth
Sequuenie
Codebook
Painwise Similarity
FST Inderere
Decoeke
Stuctura Regr
Durree
P
SAA
PST
Tokes
Interennce
UR/PeplediyMMVV
Figure 1: (a-c) Overview of PST methods, including VQ-VAE-bare and Ia-bd methods, along with their structure tok- enization process. (d-g) Overview of Struct framework, evaluating four perspectives: Downstream Effectiveness. Sensitivity, Distinctiveness, and Codebook Utilization Efficiency.
Table Overview of the four perspectives StruetTokeBBench, which summarizes the task types, metries, task names and data origin.
Perspective
Task Type
Metric
Task Name (Data Origin)
Functional Site Prediction (Por-residue Binary Classification)
Binding Site Prediction
BindInt (InterPro), Bindltio (BP) BindShake (ProteinShake)
Catalytic Site Prediction
CatInt (InterPro), CatBio (BioLIP2)
Conserved Site Prodiction
AUROC
Con (InterPro)
Downstream Effectiveness
Repeat Motif Prediction
Rep (InterPro)
Egitupe Region Prediction
Ept (PooteinGLUE)
Physicachemical Property Prediction (Per-residue Regression)
Structural Flexibility Prediction
Spearman's 's
FlexRMSF (ATLAS), Flex (ATLAS)), FlexNEQ (ATLAS)
Structure Property Prediction (Per-protein Multiclass classificatio)
Remote Homology Detection
Macro Fl
Homo (TAPE)
Sensitivity
Structural Similarity Consistency
PCC, Spcarman's ' Cosine
Fold Switching, Apo Holo CASP14
Distinetiveaess
Code Pairwise Similarity
Codebook Utilization Efficiency
Code Usage Frequency
UR. Perplexity, MUV
CASP14, CAMEO
3. Benchmark
amino acid residues in proteins that interact with ligands, known as binding sites (Dhakal et al., 2022). This is crucial for understanding protein functions and interactions within the cell, and it helps the development of targeted drug design and therapy (Min & Lee, 2022).
StrucfTokenBench evaluates PSTs from four axes: (1)
Downstream Effectiveness in capturing meaningful struc- tural representations via supervised tasks (See. 3.1); (2) Sen- sitivity to discriminate highly similar structures (Sec. 3.2); (3) Distinctiveness of codebook vectors to minimize dundancy (Sec. 3.3); (4) Codebook Utilization Efficiency (Sec. 3,4). Datasets with corresponding task names, and evaluation metrics are described in Sec. 3.5 and Sec. 3.6, respectively.
Task Type 2: Catalytic site prediction identifies residues that catalyze biochemical reactions in enzymes, which en- hances our understanding of metabolic pathways and en- zyme function. It facilitates the design of enzyme inhibitors and activators that are instrumental in drug development and therapeutic treatment (Athar et al., 2021).
3.1. Downstream Effectiveness
Task Type 3: Conserved site prediction identifies residues that are evolutionarily conserved across species, which are essential to understanding the fundamental functions and structural stability of proteins. This helps to identify regions vital for protein activity. which are often targets for thera- peutic drugs and genetic engineering (Boike et al., 2022).
A proficient PST method must effectively capture critical biological details from protein structures. We evaluate this capability through supervised property prediction tasks span- ning functional, physicochemical, and structural character- istics at residue and protein levels. As shown in Fig. (d)) we integrate PST-extracted structural representations with positional encodings, and feed them into probing layer for supervised learning. As detailed in Tab. 1, we adopt 12 tasks (24 test splits), categorized into the following 7 types.
Task Type 4: Repeat motif prediction detects repeated units of sequence or structural motifs within proteins, which can enhance structural stability, contribute functional diversity, and play key regulatory roles. These motifs assist in developing biomimetic materials (Demirel, 2021).
Task Type 1: Binding site prediction identifies specific
3

---

Protein Structure Tokenization: Benchmarking and New Recipe
Task Type 5: Epitope prediction predicts regions in pro- teins recognized by antibodies, known as epitopes. They are essential to understand how pathogens interact with the host immune system and to develop effective vaccines and immunotherapies (Kessler& & Melief, 2007).
3.4. Codebook Utilization Efficiency
For VQ-VAE-based PST methods, codebook utilization re- flects computational efficiency. Underutilized codebooks waste resources and harm performance, while high utiliza- tion bring gains, like LLaMa3's 128K-token vocabulary for efficient encoding (Grattafiori et al., 2024). For PSTs, for example, in language models using structural tokens, masked token prediction involves multi-class classification over the codebook and poor token utilization may yield ir- relevant token predictions, degrading classification accuracy. To evaluate this, we tokenize unseen protein structures to measure utilization.
Task Type 6: Structural flexibility prediction predicts residue-level protein flexibility using metric RMSF, B- B-factor, and Neg (see App. B. Flexibility is integral for capturing protein dynamics and often relates to key functional sites, such as enzyme active sites or ligand binding sites, under varying physiological conditions (Sun et al., 2019).
Task Type 7: Remote homology detection detects distantly related proteins through sequence or structural similarities, which often share evolutionary origin and similar functions. This is crucial for inferring the functions of unknown pro- teins and tracing their evolutionary connections.
3.5. Datasets
We collected datasets from various resources: ATLAS (Van- der Meersche et al., 2024), InterPro (Blum et al., 2024), BioLIP2 (Zhang et al., 2024b), ProteinShake (Kucera et al., 2024), ProteinGLUE (Capel et al., 2022), TAPE (Rao et al., 2019), Fold Switching (Chakravarty & Porter, 2022), Apo Holo (Saldaño et al., 2022), CAMEO (Robin et al., 2021), I), and CASP14 (Kryshtafovych et al., 2021) (see App. A.1.1).
3.2. Sensitivity
Proteins are inherently flexible, adopting distinct confor- mations influenced by their environment. While these con- formations may appear globally similar, they often exhibit key local structural variations that enable diverse biological functions (Chakravarty& & Porter, 2022). Thus, PST methods must be sensitive to detect subtle conformational changes.
As shown in Tab. 1, each dataset is linked with a task named by combining (and abbreviating) the task type, data source, and predicted property. For instance, BindInt and Bind- Bio refer to task type "binding site prediction" using data from InterPro and BioLIP2, respectively; FlexRMSF de- notes "structural flexibility prediction" and predicted prop- erty RMSF; CAMEO refers to the data source. The sizes of datasets range from tens to tens of thousands (see Tab. 6).
To benchmark PST sensitivity, we examine the correlation between PST-extracted structural representation similarity and the topological similarity between conformations, mea- sured using TM-score (Zhang & Skolnick, 2005). As shown in Fig. I(e), PSTs encode each conformation into struc- tural representations. Given that conformations may vary in length, the representation similarity is computed via dy- namic programming alignment (details in App. D.I). The correlation reflects the sensitivity of PSTs in capturing sub- tle differences in related protein structures.
For supervised tasks evaluating downstream effectiveness, datasets are split using a remote homologous method (see App. A.2) to assess out-of-distribution generalization, which results in two test splits: fold (Fold) and superfamily (SupFam). Exceptions include BindShake, which retains its original test split (Org), and Homo, which uses its self-curated family (Fam), superfamily (SupFam) and fold (Fold) splits. Detailed statistics for these datasees--ncllddee length and label distributions, protein CATH structure class distribution, and sequence similarity between splits-are pro- vided in App. A.3.
3.3. Distinctiveness
VQ-VAE-based PST methods require codebook diversity to maximize the tokenizer's 's expressive power and reduce re- dundancy. Ensuring no highly similar codebook vectors also prevents ambiguous substruubte mappings (Hayes et al., 2025), thereby eliminating confusion in downstream tasks.
Sensitivity evaluation utilizes two datasets targeting distinct conformational behaviors: "Fold Switching" focuses on pro- teins that adopt multiple stable structures to enable different functions, while "Apo Holo" examines proteins undergo- ing conformational changes upon ligand binding to activate their biological activity.
We analyze the distribution of similarity between codebook vector pairs to assess the diversity of the learned codebook. Considering that codebook vectors are not uniformly uti- lized in practice, we also evaluate the similarity distribution weighted by token usage frequency, derived from tokenizing unseen protein structures. This offers more precise insights into the tokenizer's 's operational dynamics among the utilized codes in a codebook.
For distinctiveness and codebook utilization efficiency, we measure structure token usage frequency on unseen proteins from CASP14 and CAMEO, which are standard holdout test sets for structure-related models, excluded from major databases.
4

---

Protein Structure Tokenization: Benchmarking and New Recipe
3.6. Metries
Tab. outlines all metries used in StruceTokenBench: (1) Downstream Effectiveness is measured using AUROC for binary classification, Spearman's 's Rank Correlation Coef- ficient (Spearman's p) for regression, and Macro F1 for multiclass classification; (2) Sensitivity is evaluated with Pearson Correlation Coefficient (PCC) and Spearman's p to quantify the correlation; (3) Distinctiveness employs cosine similarity to assess codebook diversity; and (4) Codebook Utilization Efficiency uses Utilization Rate (UR) and Per- plexity to assess token usage, and Marginal Utility of Vocab- ularization (MUV) (Xu et al., 2020) to assess the trade-off of codebook entropy gain against codebook sizes.
000
1000
p
pm
(a) Initial Distribution
(b) Distrtbution Shift
(c) Well Trained
Figure 2: Evolution of representations distributions in VQ- VAE-based PST method during training. (a) Initial distribu- tion: distributions for the encoder output P!o) the entire code- book vectors p and the selective codes receiving gradient (b) Distribu- tion shift: from training step to t 1, the distribution p(++!) updates, while unused code vectors do not update, leading to a distribution bifurcation and misalignment for Pe p(++)) and Pg p(++1) (c) Well Trained: By the final step T, P!T) is well trained to reflect the local 3D structures for residues.
updates during quantization P( are initialized.
4. Method
This section first outlines the motivation for our proposed method AminoAseed (Sec. 4.1), details its design innova- tions (Sec. 4.2), and discusses the overall pipeline (Sec. 4.3).
all benchmark perspectives for various codebook sizes.
4.1. Motivation
Second, we balance codebook size (K) and dimension (D) through data-driven Pareto-optimal scaling under the code- book capacity constraint (K x D). Our scaling experiments (Sec. 5.7) reveal two critical trends: (1) for downstream ef- fectiveness, supervised task performance degrades with ex- treme codebook sizes-either overly large (K > 2 or small (K < 2$); and (2) for codebook utilization efficiency, it monotonically drops as K increases. These findings suggest a moderate size (K = 2 that optimizes codebook utiliza- tion efficiency without sacrificing downstream effectiveness. This choice aligns with biological insights from heuristic PST method TERMs (Mackenzie, 2016), which suggests that around 600 substructures are adequate to describe 50% of the PDB database at sub-angstrom resolution.
"Codebook collapse" is a well-known issue the VQ- VAE literature (Zhang et al., 2024a): during training, only a limited subset of code vectors are actively used, rendering others redundant. This underutilization impairs VQ-VAE's capacity to encode diverse information through the quan- tization bottleneck, critically hindering its model efficacy. Our benchmark, StructTokenBench reveals analogous code- book utilization efficiency challenges in leading PSTss such as ESM3, where similar underutilization patterns emerge (see Tab. 4).
The root cause of codebook collapse stems from the chal- lenges of training a codebook from scratch. As explained in Fig. 2, over the course of training, the distribution of learned continuous representations inevitably shifts from the entire codebook vector distribution, because a significant portion of unused codes do not receive gradient updates.
4.3. Overall Pipeline
AminoAseed builds upon ESM3's local frame paradigm to model protein structures, utilizing per-residue frames and their neighboring frames capture backbone geometry via relative distances and orientations. During encoding, we employ geometric self-attention layers to maintain rotation and translation invariance (see App. E.2). fost-quantization, discrete latent representations are decoded using standard bidirectional transformer blocks to reconstruct the structure. The training objective integrates the commitment and quan- tization losses Eqn. 2, and five distinct reconstruction loss terms from ESM3. Details are discussed in App. E.
4.2. Method Design
To address codebook collapse, we introduce AminoAseed, simple yet effective PST strategy employing two key tech- niques: (1) Codebook Reparameterizzation to enable gradi- ent update of the entire codebook to alleviate its distribution shift, and (2) Pareto-Optimal Codebook Configuration to maximize token utilization while minimizing redundancy by optimally balancing codebook size and dimensions.
First, we reparameterize the codebook via learnable lin- transformation applied to fixed orthogonal vector basis: Q = Linear(C'), (C), where C is randomly initialized roughly orthogonal and remains fixed during training. Unlike vanilla VQ-VAE that updates only selected codes, this allows the entire linear layer to learn and receive updates. In Sec. 5, we VAE across
5. Experiments
Our benchmarking results are detailed as follows: down- stream effectiveness in Sec. 5.2, sensitivity in Sec. 5.3, dis- tinctiveness in Sec. 5.4, and codebook utilization efficiency in Sec. 5.5. We further conducted ablation studies for struc- tural tokens (Sec. 5.6) and scaling studies for model config-
show that this approach outperforms vanilla VQ-VAE
5

---

Protein Structure Tokenization: Benchmarking and New Recipe
urations (Sec. 5.7). We also show visualizations of protein reconstruction quality and codehook vectors in App. F.5 as case study.
(2) AminoAseed, our proposed method, significantly im- proves upon ESM3 functional (+4.74%) and structural (+27.31%) tasks, while being comparable in physicochemi- cal tasks (-0.27%); and (3) overall, VanillaVQ lags ESM3 by -5.61 %, while AminoAseed surpasses it with a +6.31 % average relative improvement across all 24 task splits.
5.1. Setups
Pre-training dataset. For the protein 3D structures used for pre-training, we followed the same criteria from training OpenFold2 (Ahdritz al., 2024) to filter structures from RCSB Protein Data Bank (PDB) (wwp, 2019). We down- sampled 10% of the filtered data, resulting in 48,316 protein chains with less than 40% sequence identity to each other, to train our PSTs. We split the data at ratio of 90% and 10% for training and validation sets. For held-out test sets, we use CAMEO and CASP14, which include 189 and 35 protein structures, respectively. Details provided in App. F.I.
We next discuss IF-based PSTs: they surpass leading VQ- VAE-bsed PSTs in functional and physicochemical proper- ties but lag in the structural ones. We attribute this to the optimization challenge for the quantization in VQ-VAE. Our ablation studies (Sec. 5.6) support this hypothesis. Notably, AminoAseed substantially narrows this performance gap.
5.3. Sensitivity Results
Tab. 3 shows that: (1) AminoAseed outperforms all models in Apo Holo and Fold Switching datasets, with an average relative gain of + +12.83% over ESM3 across all metrics; (2) ESM3 performs second best; (3) Vanilla VQ slightly trails ESM3 by -0.82%; (4) ID-baeed PSTs underperform most VQ-VAE-based PSTs because their training objective biases them predict identical sequence across varied conforma- tional structures, reducing their differentiation capability.
Pre- fre-training configurations. Adam optimizer (Kingma, 2014 4)eerrinn rate: 0.0001, weight decay: 0.01, warmup steps: 5, 426) was used to train AsinoAseed for 108,530 steps on NVIDIA A100 GPUs (more details in F.I.
Structure tokenization baselines. We benchmarked AminoAseed against leading open-source PST methods: (1) VQ-VAE-based PSTs, including FoldSeek (Van Kem- pen et al., 2024), ProTokens (Lin et al., 2023a), and FSM3 (Hayes et al., 2025); and (2) IF-based PST's includ- ing ProteinMPNN (Dauparas et al., 2022) and MIF (Yang et al., 2023). We also trained an ablated version, VanillaVO, designed to provide fair comparison by only differing from Ased in the proposed strategy outlined in Sec. 4.2. This ablated version essentially reduces to ESM3's PST when trained under identical dataset and configuration. Re- sults for two more PST methods are added in App. F.6 as suggested by reviewers.
5.4. Distinctiveness Results
Fig. 3 indicates that AminoAseed and Va displays high distinctiveness in its codebook vectors, indicated by fewer cosine similarities closer to one. However, in practical applications using CASP14 data, AminoAseed maintains high distinctiveness, while Vanilla does not.
5.5. Codebook Utilization Efficiency Results
Tab. 4 reveals four key insights: (1) FoldSeek reaches nearly 100% utilization with an extremely small codebook, show- ing uniform and balanced token usage (0.75 Perplexity); (2) ESM3 effectively uses its large codebook for over one thou- sand codes, likely benefiting from its diverse pre-training data like PDB, AFDB (Varadi et al., 2024) and ESMAI- las (Lin et al., 2023b); (3) AminoAseed, maintaining the same codebook capacity ESM3 (K x D), achieves 124.03% relative improvement in utilization rate, despite only using 10% subset of PDB; and (4) Low MUV for ESM3 suggests minimal codebook entropy gain compared to FoldSeek when trading off codebook sizes, indicating that smaller, well-optimized codebooks could be favored.
Training and evaluation configuration for supervised downstream tasks. A two-layer MLP probing layer is used for prediction across all tasks, trained using an Adam op- timizer for 10,000 steps. During training, the structural representations extracted from PSTs-cither continuous or wiscrete-weee fixed. For all models on all tasks, we opted the checkpoint with the best learning rate based on valida- tion set performance. More details are stated in App. F.1.
A blation study and scaling study experiment configura- tions. We provided the configuration details in App. F.I.
5.2. Downstream Effectiveness Results
5.6. Ablation Study
In Tab. 2, we first discuss VQ- VAE-based PSTs, For existing models: (1) ESM3 consistently outperforms others across all tasks; and (2) FoldSeek shows limited downstream effec- tiveness due to its small codebook size (K=20) and dimen- sion (D=2). For our implemented models: (1) Vanilla closely matches ESM3 in functional tasks (-0.86% rela- tive difference). though underperforms in physicochemical and structural tasks (-9.64% and -21.33 21,33% respectively):
We perform ablation studies on PST-extracted structural representations evaluate: the trade-offs between discrete and continuous representation forms, information retention from residue sequences, and their robustness to noise.
Discrete structural representations preserve most the information of their continuous counterparts. Driven by findings that discretization often leads to information loss (Mousavi et al., 2024), we examine how discrete and
6

---

Protein Structure Tokenization: Benchmarking and New Recipe
Table 2: Benchmark results for supervised downstream tasks. We used underlining and bold to highlight the best performance for IF-based PSTs and VQ-VAE-based PSTs, respectively. The relative performance difference v.s. ESM3 for our implemented models is included.
Model
Task
Split
IF-based PST
VQ-VAE-buxed PST
ProteinMPNN
MIF
FoldSeek
ProTokens
ESM3
VanillaVQ (uLESa))
AminoAsced (UCESM))
Functional Prediction (AUROC%)
BindInt
Fold
51.83
50.38
53.18
44.66
44.30
(-.068)
(7.11
SupFum
94,00
94.56
46,20
86,05
90.77
86.71 4.47%)
90.53; 0.203)
Fold
78.42
85.79
52.37
58.47
62.84
62.02,-
65.73
BindBio
SupPam
81.00
87,27
52.41
60.47
65.22
62.92,- a.53%)
68.30 4.79%)
BindShake
Org
75.52
79.90
53,40
59,
66.10
(L.42%)
69 69.61; 6,919))
Fold
61.05
59.62
53.43
58. 16
61.09
58. 88.89 -3.005)
62.19 (1.8805)
Caint
SupFam
93.40
96.49
51 41
83.85
89.82
85.00 6.87%)
91.91 2.3335)
Fold
82.49
85.85
56.37
56.14
65.33
67.58 a..4%%
65. .5, 0.009)
CarBio
SupFam
93.19
53.78
64.05
74.65
70.92, (..00%))
87.59 (417.5)))
Fold
57.18
58.43
49.26
56.23
55.22
56.98; aa.19%)
57.23 8.649))
Con
SupFam
84.68
92.66
51.39
74.33
80.53
74.60; 7.36%)
86.60 7..445)
Fold
77.63
74.53
47.70
77.25
74.70
75. 75.99; (..75%%)
74, (-03369)
Rep
SupFam
80.71
81.11
52.53
78.
82.36
09.009 -0.33%%
84.57 (+2 6x96
Fold
62.84
78
54.52
54.69
63.69
59. 29..28 0.92%)
62, 16 2.403)
Ept
SupFam
64.84
82.98
50.56
67.52
61.97
67.24 (-8.6099)
72.02 (w.2290)
Average AUROC%
75.92
79.82
$1.90
65.37
69.24
68.30 -08869))
72.43 4.7748)
Physicochemical Property Prediction (Spearman's 's p%)
FlexRMSF
Fold
62.37
59.60
15.35
13.81
44.53
-0.220)
44.63; (+0.290))
SupFam
59.24
56.80
11.99
39.68
39.00 39.08; (1.5119)
40.99 (+3.3099
Fold
31,88
34.60
4.17
6.67
23.60
22. 32. 6.78%)
21. 21.30 10.0099)
FlextFactor
SupPam
34.56
35.23
6.97
5.47
25.80
23.73 (-7.99))
21.76 (11..69))
Fold
69.69
65.32
5.71
12.98
45.08
35.95 (5.95,-200
49.64 +110.125)
HexNBQ
SupFam
64.82
2.60
12.50
45.43
35.61 6296)
50.15 (+10.419))
Average 5%
54.41
52.73
7,80
9.84
37.35
33.49; 0.04%)
38.08; 0,2788
Structure Property Prediction (Macro F1%)
Fold
25.66 66
22.56
11.57
5.84
30,02
18.17 18.17 (-2004477)
29. 29.87 (-0.509:
Homo
SupPam
30.83
33.86
4.67
6.17
24.89
22.10
38.38 +34.2
Fam
63.33
74.22
15.30
18.33
$4.42
69.78, 2283)
Average Macro F1
39.94
43.55
10.51
10.11
36.44
29.15
46.01 (+27.8/5))
continuous representations perform in supervised tasks. We present results for the challenging Fold test on half of the 24 task splits, with remaining splits reported in App. F.3.
Structural representations are less robust compared to sequence tokens. Fig. 5 illustrates that PST-extracted struc- tural representations become less reliable when exposed to noise (see App. F. Specificallyy IF-based PSTs show re- duced robustness compared to VQ- VQ-VeE-based PSTss More- over, using amino acid sequences alone for prediction proves more resilient than top performing PSTs. As the noise level increases to very high levels (up to 90%), the performance discrepancies among most methods diminish.
Fig. shows that continuous representations even reduce performance across many tasks, though they enhance spe- cific tasks like FlexNEQ (Fold split). Overall, the perfor- mance differences between discrete and continuous forms remain minor, suggesting that the continuous format is not the major contributors to performance variance. This further clarifies the advantage of Iased PSTs over VQ-VAE- based PSTs in supervised tasks (see Tab. 2). based PSTs, which use continuous representation format and lack a quantization process, likely benefit most from avoiding the complexity associated with quantization optimization.
5.7. Scaling Study
We performed scaling studies to pretrain AminoAseed and its ablated version VanillaVQ, to understand the optimal al- location of codebook sizes and dimensions, and VQ- VQ-VAE's scaling behavior with respect the encoder capacities.
Structural representations encapsulate most of the in- formation present in sequence tokens. Tab. 5 shows that combining amino acids with structure representations helps to improve performance for most VQ- PSTs most task splits. Top-performing methods like AminoAssedd and ESM3 witness relative small gains, indicating that their structural representations largely encapsulate the informa- tion provided by amino acids.
Reconstruction quality does not correlate with codebook quality. We varied the codebook size K and dimension D while keeping the codebook capacity (K x D 211). and encoder and decoder sizes consistent with ESM3. In Fig. 6, across various codebook sizes, we observe that K = 2 achieves optimal reconstruction quality under large com- pute budget for both models. However, codebook utilization significantly decreases for sizes larger than 2 and the per-
7

---

Protein Structure Tokenization: Benchmarking and New Recipe
Table 3: Sensitivity evaluation on conformational proteins. The relative performance difference v.s. ESM3 is included.
Cosine Similarity c
Cosine Sizularity
Model
App Holo
Fold Switching
cc
PCC
Speamman's i p%
PCCS
Spenmmnn's , p
ProtciMMPN
35.80
45.22
51,46
55.91
MIF
35.82
43.55
54.48
59.27
FoldSeek
34.79
13:03
51.88
56.51
Pro
54,05
61.20
65,90
-10
ESM3
39.76
50.97
57.12
62.23
V Bssi:
39.62, -0.8999
$6.51,-1mm (03)
61.51 (683
ERND
EB0 VallaM10 wallaweed
AminoAseed (SM))
42.47 (4.88()7
65.61 (11355)
75.89 (40.999
Model
Model
Figure 3:
Distinctiveness analysis of PST codebook vec-
Table 4: Codebook utilization efficiency evaluation on
tors: Left panel shows pairwise similarities between vectors; Right panel shows frequency-weighted similarities based on CASP14 token usage.
CASP14 and CAMEO datasets.
Modd
&Code (8)
Dis
CASP14
CAMEO
URSS
Perplesity
MLV
URS
Perplesity
MUV
FoldSeck
20
2
99.80
0.7548
7
160.00
0,7435
Pholekers
512
32
09.38
0.530
2.53
75.56
0.3997
2,51e4
ESMI
4395
128
27.60
0,2489
32.00
0.2841
3.26
VIQ
1024
5.55
1.66-4
5.00
0.337
AminnoAssed
512
1024
64.45
2.56-4
08.87
0.3119
255
Performancs
Table 5: Performance comparison of combining amino acid
tokens with structural tokens versus using structural tokens alone across 24 supervised task splits. The reported metrics include:
ESAO
#TaskImpr (number of tasks improved more than 0,005). #TaskComp (number of comparable tasks within +0.005 difference), AvgDiff (average difference), and Av- gRelDiff (average relative difference).
Ape
Baadletfel Biet
Plensao dhap
DalD
Task (Split)
Figure 4: Performance comparison of using continuous versus
Model
#TaskComp ATaskImpe
AvgDiffs
AvgRelDiff
discrete structural representations supervised task splits.
ProtrinMPNN
5
3
-1.55
.472%%
MIF
13
-1.15
-2.79%
FoldSeek
-13.32%
Porformmeee (AUROC)
Performance (AUROC)
ProTokens
20
0
4.43
29.60%
ESM3
15
2.39
4.33%
VanillaVQ
18
3.36
6.98%
AminoAseed
13
5
1.33
1.42%
47
formance of supervised tasks displays a U-shape across codebook sizes, favoring size in 200 (more results in Fig. 12). This indicates that reconstruction quality does not correlate with PSTs' codebook quality, offering a different angle compared to existing PSTs like ESM3 that only use reconstruction quality as justifications for PST quality.
Noise Level (%)
Noise Level (%)
(a) Con (SupFam) results.
(b) CatInt (SupFam) results.
Figure 5: Supervised task performance with increasing noises in PST-extracted structural representations.
task benchmarking, and ProteinGLUE (Capel et al., 2022) focusing on per-residue tasks. However, benchmarks for protein structure are limited. For example, ATOM3D (Town- shend et al., 2021) predicts per-protein properties for pro- tein structures, and Protein Worksho (Jamasb et al., 2024) benchmarks protein structure learning tailored for geomet- ric graph neural networks. Our work, StruecTokenBench, is designed to evaluate protein structure tokenization methods, enriching the landscape of structure-related benchmarks.
VQ-VA E-AAeeddd PST methods scale sub-exponentially with data and compute. We analyzed the scaling effects of varying encoder sizes while keeping the codebook size and dimension, and decoder size constant. As illustrated in the upper right panel of Fig. 6, U-shaped IsoFLOP curves (Hoff- mann et al., 2022) emerge at lower compute budgets. In Fig. 7, we observe that reconstruction loss decreases sub- exponentially when increasing training data and compute, suggesting diminishing returns with more resources. Both figures indicate that simply scaling up the encoder may not be an effective approach to improve VQ-VAE.
6.2. Protein Structure Tokenization Methods
Tokenization of protein 3D structures enables efficient simi- larity search and cross-modality modeling of proteins with related data modalities. Early heuristic methods use domain- specific knowledge such as dihedral angles (de Brevern, 2005), secondary structure (Mackenzie, 2016), and moment invariants (Durairaj et al., 2020). Learnable PSTs include Ia-baded (Dauparas et al., 2022; Yang et al., 2023) and VQ- VAE-based methods (Van Kempen al., 2024; Heinzinger
6. Related Work
6.1. Protein Representation Learning Benchmarks
In protein representation learning, several benchmarks eval- uate protein sequence modeling: TAPE (Rao et al., 2019) with five supervised tasks, FLIP (Dallago et al., 2021) for protein fitness landscapes, PEER (Xu et al., 2022) for multi-
8

---

Protein Structure Tokenization: Benchmarking and New Recipe
Jell tett
Maliitee Addooaadd Medlin
in codebook utilization. Nevertheless, no single model leads across the benchmark. Finally, we present novel method AminoAseed with its superiority across all benchmarking perspectives using codebook reparameterization and Pareto- optimal codebook configuration.
Validazion Recosstrociion Loss
beat?
kog
Valdarioon Rovemiteetio
Number codes
Enccder Size
Acknowledgments
15
The authors would like to thank Zuobai Zhang, Yanjun Qi and Tommi Jaakkola for their helpful discussions and comments. We also appreciate all anonymous reviewers for their constructive suggestions.
10.1
Cantar (Fold) Perfommeee
Impact Statement
Number
Number codes
Figure 6: Impact of codebook and encoder sizes on PST qual- ity: Top panels display reconstruction losses versus varying codebook sizes (left) and encoder sizes (right); Bottom panels shows Codebook Utilization Efficiency (left) and Downstream Effectiveness (right) performance versus codebook sizes.
This paper advances machine learning techniques for protein structure representation, with potential applications in com- putational biology, drug discovery, and protein engineering. While our research could contribute to understanding pro- tein mechanisms and developing therapeutic interventions, these are speculative outcomes requiring further develop- ment. Our computational method does not directly involve human subjects or biological experiments. The potential societal impact lies in providing more robust methodology for understanding protein structures, which could indirectly support medical and biotechnological research. Responsible deployment of our work and transparency in its develop- ment and use are crucial to ensure it benefits society while minimizing risks.
Validation Reconsituciin Loos
decav/ood Mailligg
1-111
Validationn Recosttacii
2=11"+
2-te/
1e-
6- 1f
6-16"
Amino Acids Trained
Compute
Figure 7: Scaling impact of data size (left) and compute budget (right) on reconstruction loss.
et al., 2023; Su et al., 2023; Hayes et al., 2025) (see App. G). Despite recent advances in protein structure tokenization, few studies have comprehensively compared these meth- ods from diverse aspects. Zhang et al. (2024c) evaluated several PSTs, highlighting a trade-off between structure re- construction and retrieval. However, this study overlooked other applications like local functional site prediction and sensitivity structural variations, which are critical for predicting binding in variants (Loux et al., 2024).
References
Protein data bank: the single global archive for 3d macro- molecular structure data. Nucleic acids research, 47(D1): D520-D528, 2019.
Abraham, M. J., Murtola, T., Schulz, R., Páll, S., Smith, J. C., Hess, B., and Lindahl, E. Gromacs: High perfor- mance molecular simulations through multi-level paral- lelism from laptops to supercomputers. SoftwareX, 1: 19-25. 2015.
7. Conclusions and Future Work
In this work, we first developed Strue(TokenBench. com- prehensive benchmark to evaluate protein structure tokeniza- tion (PST) methods across four key perspectives: down- stream effectiveness, sensitivity, distinctiveness, and code- book utilization efficiency. StruecTokenBench curated 10 public datasets about protein structure and functions, cov- ering 17 tasks, making it the first benchmark for PST and leading benchmark resource for protein structure repre- sentation learning. Next, we evaluated five popular ot the-art PST methods and found that mnvere--folding foadedg-bbsedd PSTs excel in downstream effectiveness but suffer from low sensitivity, whereas VQ- tive to protein conformations and exhibit varied efficiency
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Ahdritz, G., Bouatta, N., Floristean, C., Kadyan. S., Xia, Q., Gerecke, W., O' Donnell,, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and ca- pacity for generalization. Nature Methods, pp. 1-11, 2024.
PSTs are more sensi-
Andreeva, A., Kulesha, E., Gough, J., and Murzin, A. G.

---

Protein Structure Tokenization: Benchmarking and New Recipe
The scop database in 2020: expanded classification of representative family and superfamily domains of known protein structures. Nucleic acids research, 48(D1):D376- D382, 2020.
Durairaj. J., Akdel, M., de Ridder, D., and Dijk, A. D.
Geometricus represents protein structures shape-mers derived from moment invariants. Bioinformatics, 36 (Supplement.2xi7118--8 2020.
Athar, T., Al Balushi, K., and Khan, S. A. Recent advances on drug development and emerging therapeutic agents for alzheimer's disease. Molecular biology reports, 48(7): 5629-5645, 2021.
Ferruz, N., Schmidt, S., and Höcker, B. Protgpt2 is a deep unsupervised language model for protein design. Nature communications, 13)4444348, 2022.
Gao, Z., Tan, C., Wang. J., Huang, Y., Wu, L.,, and Li, S. Z. Foldtoken: Learning protein language via vector quan- tization and beyond. arXiv preprint arXiv:2403.09667. 2024.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv:1308.3432, 2013.
Gaujac, B., Donå, J., Copoiu, L., Atkinson, T., Pierrot, T., and Barrett, T. D. Learning the language of protein structure. arXiv preprint arXiv:2405. 15840. 2024.
Björck, A. Numerics of gram-schmidt orthogonalization. Linear Algebra and Its Applications, 197:297-316, 1994.
Blum, M., Andreeva, A., Florentino, L. C., Chuguransky, S. R., Grego, T., Hobbs, E., Pinto, B. L.,, Orr, A., Paysan- Lafosse, T., Ponamareva, I., al. Interpro: the protein sequence classification resource in 2025;? mode long- meta?i. Nucleic Acids Research, pp. gkae 1082, 2024.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Boike, L., Henning, N. J., and Nomura,
Advances in covalent drug discovery. Nature Reviews Drug Discovery, 21(12))881-898, 2022.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., al. Simulating 500 million years of evolution with language model. Science, pp. eads0018, 2025.
Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., and Feenstra, K. A. Proteinglue multi-task benchmark suite for self-supervised protein modeling. Scientific Re- ports, 12(1):16047, 2022.
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., and Rost, B. Bilingual lan- guage model for protein sequence and structure. bioRxiv, pp. 2023 07, 2023.
Chakravarty, D. and Porter, L. L. Alphafold2 fails to predict protein fold switching. Protein Science, (6))3333,,, 2022.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. 2022.
Dallago, C., Mou, J.. Johnston, K. E., Wittmann, J., Bhat- tacharya, N., Goldman, S., Madani, A., and Yang, K. K. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021-11, 2021.
Hou, J., Adhikari, B., and Cheng, J. Deepsf: deep convolu- tional neural network for mapping protein sequences to folds. Bioinformatics,
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, R. J., Milles, L. F., Wicky, B. I., Courbet, A., de Haas, R. J., Bethel, N., et al. Robust deep learning-based pro- tein sequence design using proteinmpnn. Science, 378 (6615):49-56, 2022.
2018.
Huang, J., Rauscher, S., Nawrocki. G.. Ran, T., Feig, M.. De Groot, B. L., Grubmüller, H., and MacKerell Jr. A. D. Charmm36m: an improved force field for folded and intrinsically disordered proteins. Nature methods, 14(1): 71-73, 2017.
de Brevem, A. G. New assessment of structural alphabet. In silico biology, 5(3):283-289, 2005.
Demirel, M. C. Protein based biomimetic materials. In 2021 AIChE Annual Meeting. American Institute of Chemical Engineers, 2021.
Jamasb, A. R., Morehead, A., Joshi, C. K., Zhang, Z., Didi, K., Mathis, S., Harris, C., Tang, J., Cheng, J., Liò, P., et al. Evaluating representation learning the protein structure universe. ArXiv, pp. arXiv- 2024.
Dhakal, A., McKay, C., Tanner, J. J., and Cheng, J. Artificial intelligence in the prediction of protein-ligand interac- tions: recent advances and future directions. Briefings in Bioinformatics, 23(1):bba 2022.
Kessler, J. and Melief, C. Identification of t-cell epitopes for cancer immunotherapy. Leukemia, 21(9):1853 (11):1859-1874 1874, 2007.
10

---

Protein Structure Tokenization: Benchmarking and New Recipe
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint orXiv:/312.6/11 2013.
Loux, T., Wang, D., and Shakhnovich, E. More structures,
less accuracy: Esm3's binding prediction paradox. bioRxiv, 2024. doi: 10.1 10.1101/2024.12.0..22777
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
http:///wwwwiooxi..ooooogoo URL 2ar1//2024/12/00/2244..222
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., and Moult, J. Critical assessment of methods of protein struc- ture prediction (casp)-roumg) xiv. Proteins: Structure, Function, and Bioinformatics, 89(12)::607-1161, 2021.
Lu, A. X., Yan, w., Yang. K. K., Gligorijevic, V., Cho, K., Abbeel, P., Bonneau, R., and Frey, N. Tokenized and continuous embedding compressions of protein sequence and structure. bioRxiv, pp. 2024-08, 2024.
Mackenzie, C. A tertiary alphabet for the observable protein structural universe captures sequence-structure relationships. Protein Sci, 25:75, 2016.
Kucera, T., Oliver, C., Chen, D., and Borgwardt, K. Pro- teinshake: building datasets and benchmarks for deep learning on protein structures. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023.
Kunzmann, P. and Hamacher, K. Biotite: a unifying open source computational biology framework in python. BMC bioinformatics, 19:1-8, 2018.
Min, H.-Y. and Lee, H.-Y. Molecular targeted therapy for an- ticancer treatment. Experimental & molecular medicine, 14(10):1670-1644 2022.
Li, M., Tan, Y., Ma, X., Zhong, B., Yu, H., Zhou, Z.., Ouyang, W. Zhou, B., Hong, L., and Tan, P. Prosst: Protein language modeling with quantized structure and disentangled attention. biorxiv. 2024.
Mousavi, P., Della Libera, L., Duret, J., Ploujnikov, A., Subakan, C., and Ravanelli, M. Dasb-discrets audio and speech benchmark. arXiv preprint arXiv:2406.1422 2024.
Ligowski, L. and Rudnicki, W. efficient implementation of smith waterman algorithm on gpu using cuda, for sively parallel scanning of sequence databases. In 2009 IEEE International Symposium Parallel & Distributed Processing, pp. 1-8. IEEE, 2009.
Pearl, F. M., Bennett, C., Bray, J. E., Harrison, A. P., Martin, N., Shepherd, A., Sillitoe, I., Thornton, J., and Orengo, C. A. The cath database: an extended protein family resource for structural and functional genomics. Nucleic acids research, 3 31(1):452-455 2003.
Likic, V. The needleman-wunsed algorithm for sequence alignment. Lecture given at the 7th Melbourne Bioinfor- matics Course, Bi021 Molecular Science and Biotechnol- ogy Institute, University of Melbourne, pp. 1-46, 2008.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Lin, X., Chen, Z., Li, Y., Ma, Z., Fan, C., Cao, Z., Feng, S., Gao, Y. Q., and Zhang, J. Tokenizing foldable pro- tein structures with machine-leared artificial amino-acid vocabulary. bioRriv, pp. 2023-11, 2023a.
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., Abbeel, P., and Song, Y. Evaluating protein transfer learning with tape. Advances in neural informa- tion processing systems, 32, 2019.
Lin, Z. Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli. O., Shmueli, Y., al. Exolutionary-seale prediction of atomic-level protein structure with a language model. Science, 379(6637): 1123-11300 2023b.
Robin, X., Haas, J., Gumienny, R., Smolinski, A., Tau- riello, G., and Schwede, Continuous automated model evaluation perspectivss on the future of fully automated evaluation of structure prediction methods. Proteins: Structure, Function, and Bioinformatics, 89 (12):1977-1986, 2021.
Liu, A., Elaldi, A., Russell, N., and Viessmann, O. Bio2token: All-atom tokenization of any biomolecular structure with mamba. arXiv preprint 24Xv:2410.199100 2024.
Saldaño. T. Escobedo, N., Marchetti, J., Zea, D. J., Mac Donagh, J., Velez Rueda, A. J., Gonik, E., García Melani, A., Novomisky Nechcoff, J., Salas, M. N., et al. Impact of protein conformational diversity on al- phafold predictions. Bioinformatics, 38(10)2742-2748. 2022.
Liu, Z., Li, Y., Han, L., J., Liu. J., Zhao, Z., Nie, W., Liu, Y., and Wang, R. Pdb-wide collection of binding data: current status of the pdbbind database. Bioinformatics, 31(3):05-441 2015.

---

Protein Structure Tokenization: Benchmarking and New Recipe
Shazeer, N. Glu variants improve transformer. arXiv preprint arXi::2002.05200 2020.
Yang, K. K., Zanichelli, N., and Yeh, H. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36: gzad015, 2023.
Stein, R. M., Yang, Y., Balius, T. E., O' M. J., J., Young, J., Tang, K., Shoichet, B. K., and Irwin, J. J. Froperty-unmatcher decoys in docking benchmarks. Journal of chemical information and modeling, 61(2): 699-714, 2021.
Yin, S., Fu, C., Zhao, S., L., Sun, X., Xu, T., and Chen, E. A survey multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Steinegger, M. and Söding, J. Mmseqs2 enables sensi- tive protein sequence searching for the analysis of mas- sive data sets. Nature biotechnology, 35(11):0026-1028, 2017.
Zhang, B., Wang, H., Luo, C.,, Li, X., Liang, G., Ye, Y., Qi, X., and He, Y. Codebook transfer with part-of-speech for vector-quantized image modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7757-7766, 2024a.
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023-10, 10, 2023.
Zhang, C., Zhang, X., Freddolino, P. L., and Zhang, Bi- olip2: an updated structure database for biologically rele- vant ligand protein interactions. Nucleic Acids Research, 2024b.
Sun, Z., Liu, Q. Qu, G., Feng, Y., and Reetz, M. T. Utility of b-factors in protein science: interpreting rigidity, flexibil- ity, and internal motion and engineering thermostability. Chemical reviews, 1 119()):6626116) 2019.
Zhang, J., SoyynardPiganeau, B., Gong, J., Cheng, X.,
Luo, Ly, H., Song, L., and Xing, E. Balancing locality and reconstruction in protein structure tokenizer. bioRxiv, 2024c. doi: 10. 10.1101/2024. 21222024..120 12.02.6266666
Townshend, R., Powers, A., Eismann, S., and Derry, A. Atom3d: tasks molecules in three dimensions. arxiv 2021. arXiv preprint arXiv:2012.04033 2021.
URL https: http:://wwwwbbooxi.o www.boooxxi..oooooooeeeeee.. org/content// ear1y/2024/122 0 /ttp:/12206/2222442/ 12. 02,
12..6..666666666.. 626366.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Zhang, J., Meynard-Piganeeu, B., Gong, J., Cheng, X., Luo, Y., Ly, H., Song, L., and Xing, E. Balancing locality and reconstruction in protein structure tokenizer. bioRxiv, pp. 2024-12, 2024d.
Van Kempen, M., Kim, S. S., Tumescheit, C.. Mirdita, M., Lee, J., Gilchrist, C. L., Söding, J., and Steinegger, M. Fast and accurate protein structure search with foldseek. Nature biotechnology, 42(2):243 2024.
Zhang, Y. and Skolnick, J. Tm-align: a protein structure alignment algorithm based on the im-score. Nucleic acids research, 33(7):2302-2300 2005.
Vander Meersche, Y., Cretin, G., Gheeraert, A., Gelly, J.-C., and Galochkina, T. Atlas: protein flexibility description from atomistic molecular dynamics simulations. Nucleic acids research, 52(D1))D384 D392, 2024.
Varadi, M., Bertoni, D., Magana, P., Paramval, U., Pidruchna, I., Radhakrishnan, M., Tsenkov, M., Nair, S., Mirdita, M.. Yeo, J., et al. Alphafold protein structure database in 2024: providing structure coverage for over 214 million protein sequences. Nucleic acids research, 52(D1))D368-DD3 D375, 2024.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xu, J., Zhou, H., Gan, C., Zheng. Z., and Li, L. Vocab- ulary learning via optimal transport for neural machine translation.
arXiv preprint arXiv:2012.15677 2020.
Xu, M., Zhang, Z., Lu, J., Zhu, 7., Zhang. Y., Chang, M., Liu, R., and Tang, J. Peer: a comprehensive and multi- task benchmark for protein sequence understanding. Ad- vances in Neural Information Processing Systems, 35: 35156-35173, 2022.
12

---

Protein Structure Tokenization: Benchmarking and New Recipe
A. Data Pre-processing
into the complex dynamics of molecular interactions and their implications for biological processes like im- mune response.
A.1. Data Sources
A.1.I. DOWNSTREAM EFFECTIVENESS EVALUATION
ATLAS (Vander Meersche et al., 2024) database provides a comprehensive collection of standardized molecular dynamies (MD) simulations of protein struc- tures, featured with detailed analysis of both global protein behavior and local flexibility of the protein backbone. It includes 1390 protein chains in total, each subjected to three replicated all-atom MD trajec- tories using GROMACS (Abraham et al., 2015) and the CHARMM366 force field (Huang et al., 2017). For our studies, we utilized the initial release of this database, identified as "2022 06 13 v1".
TAPE (Rao et al., 2019) is a comprehensive benchmark consisting of five biologically relevant tasks designed for the supervise learning of protein biology. TAPE highlights three major areas: structure predic- tion, detection of remote homologs, and protein engi- neering for protein landscapes. This benchmark serves as a crucial tool for driving progress in understand- ing and manipulating proteins for various scientific and medical applications. Specifically, the remote ho- mology detection task is derived from the SCOP 1.75 database (Andreeva et al., 2020) of hierarchically clas- sified protein domains.
InterPro (Blum et al., 2024) database consolidates var- ious datasets into a single searchable platform, offering runctional insights into protein sequences by classify- ing them into families and identifying key domains and functional sites. We used "release 100.0" dated 30th May 2024.
Notably, the original dataset consists of 1195 labels. We reduce the number of label classes to degrade the task difficulty, since the dataset is extremely imbal- anced with limited prediction model capacity of 2- layer MLP probing layer. Specifically, we filtered label class that has less than 50 protein samples in the train- ing dataset, reducing from 1195 labels to 45 labels.
BioLIP2 (Zhang et al., 2024b) is a semi-manually cu- rated database that provides high-quality, biologically relevant ligand-protein binding interactions, validated through geometric rules and empirical literature. It enriches entries with detailed information, including catalytic sites and binding affinities, sourced from di- verse databases and comprehensive manual literature reviews. Notably, BioLIP2 offers unique data not cov- ered by InterPro, making it an essential alternative resource for predicting binding and catalytic sites.
1.2. SENSITIVITY EVALUATION
Fold Switching (Chakravarty & Porter, 2022) includes 74 pairs of fold-switching proteins, which contain re- gions capable of adopting distinct stable secondary and tertiary structures under varying cellular conditions, or alternating between two stable folds at equilibrium. These protein pairs exhibits extremely high levels of sequence identity (mean 99%, and median 100%), yet display significantly different structures (mean TM- scores of 0.58, and median TM-scores 0.63). This pro- vides a solid foundation for benchmarking sensitivity of PST methods to detect subtle yet critical structural variations, essential for understanding dynamic protein behaviors in varying biological contexts.
ProteinShake (Kucera et al., 2024) is benchmarking software package designed to simplify dataset creation and model evaluation for deep learning applications focused on protein structures. It addresses a wide range of biological challenges, including cructure-functton relationships, geometric relationships between struc- tures, and modeling physical interactions. Notably, its binding site prediction task leverages data from the PDBbind-CN database (Liu et al., 2015) and the DUDE-Z virtual screening benchmark (Stein et al., 2021), providing an alternative choice for binding site prediction besides InterPro and BioLIP2.
Apo Holo (Saldaño et al., 2022) provides compre- hensive dataset for studying ligand-induced conforma- tional changes in proteins. includes 90 pairs, each consisting of an apo conformer (unbound state) and its corresponding holo form (bound biological rele- vant ligand). This dataset spans wide spectrum of conformational diversity, quantified by the pairwise global RMMD between their conformers, with val- ues ranging from 0 to as much as 15. The extensive range offers a robust foundation for detailed evaluation of protein structural conformational changes.
ProteinGLUE (Capel al., 2022) is a benchmark de- signed to evaluate protein representations, focusing variety of tasks related to structural protein prop- erties. It includes specific tasks like protein-protein interactions and epitope mapping, which explore how proteins interact with other molecules within their en- vironment. These tasks are critical for deepening our understanding of protein function, providing insights
13

---

Protein Structure Tokenization: Benchmarking and New Recipe
A.1.3. DISTINCTIVENESS & CODEBOOK UTILIZATION EFFICIENCY EVALUATION
architecture rather than sequence or functional conser- vation.
CASP14 (Kryshtafovych et al., 2021), i.e., Critical Assessment of protein Structure Prediction, serves as an independent platform for assessing various methods of protein structure modeling. provides an indepen- dent platform for evaluating protein structure model- ing methods. During the assessment period, unknown protein structure sequences were posted for modeling, with submissions collected and evaluated as experi- mental coordinates became available. This process systematically assesses the predictive capabilities of current modeling techniques. We selected proteins released after our pre-training data cutoff date (May 2020) from CASP14 test sets.
Why remote homologous redundancy important? Re- mote homologous redundancy focuses on structure similar- ity rather than sequence similarity. Intuitively, protein prop- erties are influenced more by structures in many aspects: (I) function: remote homologs often retain similar biomedical functions despite low sequence similarity. This can include enzymatic activity (catalytic cite prediction in BioLIP2), ligand binding (binding site prediction in ProteinShake), or interactions with other biomolecules; (2) stability: pro- teins with similar folds may have conserved core residues that contribute to maintaining structural integrity, even if their sequences have diverged significantly; (3) dynamics: the dynamic behavior of protein structures, including their flexibility and conformational changes, can be conserved among remote homologs, contributing to their functional similarities.
CAMEO (Robin et al., 2021), i.e., the Continuous Au- tomated Model EvaluatiOn (CAMEO) platform, oper- ates automated blind evaluations to complement the bi- ennial CASP experiments. CAMEO leverages weekly prereleases of protein sequences that are scheduled for publication in the Protein Data Bank. This platform has been particularly useful in the prediction of complex protein structures, which has been observed challeng- ing in CASP14. We used the release from April 2022 to June 2022.
Method. To eliminate homologous protein redundancy be- tween training and test datasets, we adopted DeepSF's (Hou et al., 2018) multi-level redundancy removal approach, where we operate at two levels more related to structures: fold and superfamily levels. In this hierarchy, fold repre- sents the broadest category, followed by superfamily. These two levels are defined as follows: at the superfamily level, proteins from the same superfamily appear in both the train- ing and test datasets; at the fold level, no proteins from the same superfamily are shared between the sets, though proteins from the same fold may be present in both.
A.2. Remote Homologous Data Splitting
For the 10 datasets collected from InterPro, BioLIP2, AT- LAS and ProteinGLUE for supervised downstream tasks, we followed DFepSF (Hou et al., 2018) for a strict protein splitting method to remove the remote homologous protein redundancy, by grouping proteins according to their fold and superfamily classes retrieved from CATH database (Pearl et al., 2003).
The splitting method involves several steps: First, we filter proteins curated from raw data without the target functional labels. Next, the fold and superfamily labels are assigned to proteins using the CATH database (Pearl et al., 2003). For each fold, superfamilies are split into two groups (60% for training and 40% for testing), creating the fold test split. For the split training data, 80% of the proteins in each superfamily are placed in training, with the remaining 20% in testing, creating superfamily-Ievel datasets. Lastly, 20% of the test data is randomly selected to form a validation set.
Remote homologous relationship classification. The clas- sifications of remote homologous relationships comprise three levels: family, superfamily, and fold:
Family groups proteins that share clear evolutionary relationships, which can be detected by common se- quence comparison tools. This relatively close level of homology.
A.3. Supervised Downstream Data Statistics Analysis
Data Sizes. Tab. 6 shows the number of protein samples across all downstream datasets used in StreeTTokenBench.
Superfamily groups proteins that are more distantly related, where the homology may only be apparent through structural similarities and conserved functional sites, rather than sequence similarity. This represents a more remote homologous relationship than family.
Length distribution. Fig. 8, shows the distribution of pro- tein lengths for the training splits of all supervised down- stream datasets, visualized using histograms with kemel density estimation. Protein lengths exhibit significant vari- ability across datasets, ranging from short sequence (less than 200 residues) to long sequence (up to 600 residues). For binding site or catalytic site predictions, different dataset sources show distinct protein length distributions, highlight-
Fold groups proteins based on broad global structural features, often grouping multiple superfamilies. Ho- mology at this level is the most remote, as the rela- tionship may primarily be terms of overall structural
14

---

Protein Structure Tokenization: Benchmarking and New Recipe
Table 6: Data size (i.e, the number of protein samples) for all datasets used in StrnccToken
Table 8: Binary label distributions for all per-residue super- vised downstream tasks except physicochemical property prediction (regression) and structure property prediction (per-protein multi-class classification).
Dataset
Split
Train
Valid
Fold Test
SupFam Test
Bindlat
1353
256
671
273
Datient
Dutasct
Per Protcin
BindBio
12566
2678
8112
2510
Rutio Lided Onelk
Labek
of Label
Lubsh
Caintt
3279
443
1090
674
Bindlts
7.44
153064
9.65
187.0
CaBio
4406
667
1815
889
Biddllo
2.57
3130120
3:35
265.0
Con
7447
1177
3262
1497
BindShake
17.16
355297
20.11
276.3
Catoe
4.52
823918
5.39
253.5
Rep
478
1789
143
Callio
1:37
1465901
1.91
352.2
21
16
23
Oun
3.11
1599141
12.00
214.7
FSexRMSP
6433
81
225
104
Rep
29.84
95611
39:42
158.6
FlcttFactor
81
225
104
Fiee
20:46
1089
288.5
FlexNEQ
643
81
225
104
Train
Valid
Org Test
13.0%
BindShake
1286
306
242
20.0%
10.5%
13 en
Train
Valid
Fold Test/ / SupFam Test / Fam Test
30%
5.0%
Homo
6003
241
232/455/70 788
cucle
720
4.55
431
200
300
900
Lengt
Split
Culree
Catl
Con
20.t5
-2.0%
Fold Switching
74
200%
Apo Holo
90
12.4%
CASP14
35
CAMEO
189
8.mh
Length
Lango
Lnngh
Table 7: Protein sequence similarity across splits for down- stream supervised tasks, measured with sequence identify
using MMseqs2 for pairwise sequence alignment.
200
403
0.2%
200
8.0%
200
430
6.0
Splits Olegurnce Idvenily
FlotlE
Ho:
Trin
Truis
Truis SupFun
Fold
SupFaan
20.0%
28.0%
Bindlio
Bindler
45.0%
16.29
45.34
16.33
36,04
10.0%
10.7h
13.0%
Callee
38.66
30.61
40.05
47.40
CalBio
34.12
34,00
14.39
34.30
3.2%
Con
$7.81
$2.02
36.94
38.03
38.44
37.09
Length
Lengnn
Lengt
Figure 8: Length distribution for the training dataset of the 12 supervised downstream tasks.
FlesRMP
FlesBFuctor
38.18
37.99
38.00
FleeNBQ
36.18
37.03
37399
38,00
Vhin Volle
Thin Org
BindShake
35.10
35,01
with protein variability.
Homo
40.24
Protein sequence similarity between splits. The difficulty of supervised tasks largely depends shifts in data distribu- tion across training, validation, and different test splits (fold split and superfamily split). To effectively understand these distribution shifts, we calculate pairwise protein sequence similarities using MMseq2 (Steinegger & Söding, 2017) for sequence alignment. Sequence identity, expressed as a percentage, measures the ratio of identical amino acids to the total number of amino acids in the aligned sequences. Commonly, this metric is conventionally used to measure se- quence similarity, with thresholds of 90% and 50% typically distinguishing highly similar and moderately similar se- quences, respectively. According to our analysis presented in Tab. 7, the similarities among the splits range from a minimum of 34.00% to a maximum of 47.40% with an average of 38.25%. These results suggest that the splits are sufficiently distinct to ensure that our StruerTokenBench provides benchmarking tasks with reasonable difficulty.
ing the importance of incorporating diverse data sources for the same task.
Protein CATH structure class distribution. CATH (Pearl et al., 2003) is protein structure classification system that categorizes proteins domains based on their structural and evolutionary relationships. It employs hierarchical label- ing system, with the top two levels comprising, "Class", which represents the overall secondary structure content of protein domains (e.g., alpha helices, beta sheets, or mixed alpha-beta structures); and "Architecture", which describes the overall arrangement and orientation of secondary struc- tures within the protein (e.g., sandwich, barrel, or roll).
Fig. 9 presents the CATH structural class distribution for the training data of all supervised downstream datasets. While most datasets show a predominance of alpha-beta structures, others, such BindInt and Con, demonstrate more bal- anced and diverse representation of "Class" across alpha, beta, and mixed structures. Notably, irregular and special structures are present in 9 out of 12 datasets. These observa- tions emphasize the structural diversity in the StruetToken- Bench atasets, providing a realistic benchmarking scenario
Label distribution for supervised binary classification tasks. Most tasks in StructTokenBench ocus on per-residue binary classifications, where a label of one indicates func- tionally important residues. Typically, these functional sites
15

---

Protein Structure Tokenization: Benchmarking and New Recipe
Aiga Sendecch
Harty xihe
Aet
Haary tite
Deta burne
Edea bandl
(a) Bindlnt
(b) BindBio
(c) BindShake
(d) CatInt
dece comple
For nos
Aype Preeeaae
Apta Ieli
Kuta Ed
Coan Routa
Doo
nahe
Pat Aghs
Ortragonal
(e) CatBio
in Con
(g) Rep
(h) Ept
Tapp Tan
Lavee Sendwir
Aaeerrant Sactacc
Grl
Dnta
2n13
Nita BILA
PAR Dew
Orteppcdd
(i) FlexRMSF
(i) FlexBFactor
(k) FlexNEQ
(l) Homo
Figure 9: CATH structure class distribution for all supervised downstream datasets. Each plot represents the hierarchical breakdown of protein domains by their structural class ("Class"), and architecture ("Architecture"). Blue highlights the largest portion of class, with red the second and green the third.
are short within a single protein. To highlight this chal- lenge, we visualized the binary label distribution, as shown in Tab. 8. This visualization confirms that labels are highly imbalanced across all tasks. To mitigate this issue, we im- plemented the per-batch class weighting technique in our supervised benchmarking pipeline. This method assigns different weights to the positive and negative classes, en- hancing the model's capability to learn effectively from infrequent labels.
B-Factor, extracted from PDB files, measures how much individual atoms deviate from their average po- sitions, reflecting both atom vibration and structural disorder. This provides insights into the structural dy- namics, stability, and functional flexibility of the pro- tein.
RMSF (Root Mean Square Fluctuation), calculated on a-carbons using GROMACS (Abraham al., 2015), measures the average deviation of an atom from its mean position over time. It highlights regions of the protein that are either highly mobile or rigid, with higher RMSF values indicating flexible, dynamic areas, and lower values representing structural stability.
B. Metrics
B.1. Local Structure Flexibility Measurement in ATLAS
Certain atoms residues in protein tend to be more flexi- ble and mobile, particularly in regions like enzyme active sites or protein-protein interaction interfaces, where flexibil- ity is crucial for biological function.
NEQ quantifies the local deformability of the protein backbone by measuring the average number protein blocks (PBs) at given sequence position, varying from
16

---

Protein Structure Tokenization: Benchmarking and New Recipe
(no variation) to 16 (fully random PB distribution). This is determined by the Phi/Psi angles and reflects the degree of conformational flexibility within the protein.
PCCx.y Cov(X.Y2 where Cov(X, Y)) is the covari- varr ance between variables X and Y, and o and ay the standard deviations of X and Y, respectively. PCC is widely used to assess the strength and direction of a linear relationship between two continuous variables.
B.2. StructTooknBench Metries
Both metrics for evaluation and metries used in the bench- marking pipeline are stated in this section.
Spearman's 's p (Spearman's 's Rank Correlation Coeffi- cient) is introduced above in "Downstream Effective- ness Metries".
Downstream Effectiveness Metrics. AUROC, Spearman's and Macro Fl are reported for supervised downstream performance.
TM-score (Template Modeling Score), assesses the structural similarity between protein structures irre- spective of their size. TM-score normalizes the score by the length of the proteins. providing a scale- invariant measure that ranges from I, where I indicates a perfect match between two structures. The TM-score is less sensitive to local variations and more reflective of the overall topology of the protein struc- tures, making it a more robust metric for comparing significantly different sizes and alignments.
AUROC (Area Under the Receiver Operating Charac- teristic Curve) measures the ability of classification model to distinguish between classes across all possible classifying thresholds. An AUROC of 1.0 is perfect score, while 0.5 suggests no better than random guess- ing. Higher AUROC represent better performance.
(Spearman's Rank Correlation Coeffi- cient) measures the rank correlation, specifically how well the relationship between two variable can be de- scribed using a monotonic function. For its computa- tion, each observed data point is converted into ranks, with tied values receiving the average of their ranks. Spearman's 's is calculated as: = 1 6 m(a"-I) -T where d, is the difference between ranks of corresponding variables, and n is the number of observations. rraggss from 1 to 1, where indicates a perfect positive rank correlations.
Spearman's
RMSD (Root Mean Square Deviation), measures the average distance between the atoms (usually the back- bone atoms) of superimposed proteins. It quantifies the absolute spatial deviation between two aligned protein structures, providing a straightforward measure of the structural difference. Lower RMSD values indicate a higher degree of similarity between the two structures. However, RMSD sensitive to outliers and can be heavily influenced by local discrepancies in the struc- ture, making it less useful for comparing proteins of different sizes or those that only share partial similarity.
Macro F1 calculates the average of Fl score across all classes, treating each class equally regardless of its frequency in the dataset. This makes it particularly use- ful in scenarios where class imbalances might distort the accuracy. The F1 score for each class computed using the harmonic mean of precision and recall, where precision is the ratio of correctly predicted positive ob- servations to the total predicted positives, and recall is the ratio of correctly predicted positive observations to all observations in the actual class.
Distinctiveness Metrics Pairwise cosine similarities be- tween codebook vectors and its weighted version based on the structural token usage frequency are reported.
Cosine Similarity between two vectors A and B quan- tifies how similar the directions of the two vectors are. It is calculated ||||-||||| product, and ||-||| and [[B]| are the vector norms.
A.B where A B is the dot
Token Usage Frequency measures the frequency of codebook tokens being used during the protein struc- ture quantization process.
Sensitivity Metrics. Given a pair of protein conforma- tions, structural similarity is measured using TM-score or the negative value of RMSD (see App. F.2). Sensitivity performance is then assessed by examining the correlation between the cosine similarity of structural representations extracted by PSTs and the structural similarity of input pro- tein conformations. This correlation is measured using PCC and Spearman's p.
Weighted Cosine Similarity is adapted to emphasize the importance of cosine similarity for codebook vec- tors in practical applications. This metric is calculated in three steps: (1) calculate pairwise cosine similarity between codebook vectors; (2) determine token usage frequency in test data and calculate the pairwise prod- uct of these frequencies; and (3) weight each entry from the cosine similarity matrix by the corresponding pairwise frequency product.
PCC (Pearson Correlation Coefficient) is a statisti- cal measure that quantifies the linear relationship be- tween two variable X and Y, defined using the formula:
17

---

Protein Structure Tokenization: Benchmarking and New Recipe
Codebook Utilization Efficiency Metries. UR, Perplex- ity and MUV are reported understand PST codebook utilization efficiency.
Handling symmetry and physical constraints: Pro- tein structures are inherently redundant due to their trans-rotational equivariance and polymer nature. This SE-(3) requirements for protein structures have led to the development of invariant encoders and equivari- ant decoders that are computationally intensive and complex. Quantization simplifies these complexities by eliminating the need to explicitly model such con- straints, thereby allowing models to focus on the most essential information needed for accurate protein struc- ture representation.
UR (Utilization Ratio) measures the hit ratio of code- book tokens. UR is different from token usage fre- quency. If one codebook token hit during the protein structure quantization process, it's counted once to- wards the UR, regardless of multiple occurrences.
Token Usage Frequency is introduced above in "Dis- tinetiveness Metrics".
Biological understanding: Proteins often exhibit mod- ularity, with distinct substructure motifs determining structure orientations and distinct domains responsible for different functions (Pearl et al., 2003). By discretiz- ing protein structures into discretized tokens that reflect these functional units or motifs and domains, we gain a clearer insight into how each component contributes to the overall structure and function of the protein.
Perplexity quantifies the uniformity of token usage frequency distribution across a codebook. Higher per- plexity signifies a more uniform distribution, indicating balanced token utilization. calculated using the formula:
Perplexity = exp Hu:
where Hy represents as the entropy of the corpus for a given codebook vocabulary v. This is calculated as the sum of token entropy. To mitigate the impact of token length variability, the entropy is normalized by the average token length, and the adjusted entropy formula is given by:
Employing discrete tokens instead of continuous features like coordinates and dihedral angles helps to reduce the risk of overfit- ting in modeling (Li et al., 2024). By transforming structure generation tasks into classification problems, rather than complex regression models, discrete tokens streamline the learning process and enhance model performance and generalizability.
Preventing overfitting:
Hg Σ P(j)log P()), JE"
Lo
where P(j) the token usage frequency of token j from the training corpus and by is the average length of tokens in vocabulary
Integration with large multimodal models: Discrete structural and sequence tokens, along with text data, can be seamlessly integrated, facilitating the develop- ment of advanced multimodal large language models (LLMs). This also enables the application of opti- mization techniques developed in the natural language processing community for protein modeling.
MUV (Marginal Utility of Vocabularization) exam- ines the benefits (entropy) a corpus can get from an increase of cost (size). A higher MUV indicates a more favorable benefit-cost ratio. This metric helps in un- derstanding how efficiently the vocabulary expansion contributes to the overall utility of the codebook. It's defined as the negative derivation of entropy to size, as in the formula:
D. Benchmark Details
D.1. Dynamic Programming Alignment Algorithm
To assess the similarity of the extracted structural to- kens, we utilized Biotite (Kunzmann & Hamacher, 2018)'s "align.optimal()" function, which implements global align- ment algorithms for pairwise sequence alignments using dynamic programming. The resultant alignment similarity is then used as the similarity for the structural tokens.
Mo(k-m)= (Ho(k)m)= - Hv(k))
Here vkk), ulk + m) refer to vocabularies containing k and ls m tokens, respectively. H. denotes the entropy of the corpus with given codebook vocabulary as defined above.
This versatile function supports both global alignments us- ing the Weedleman-Wunsch algorithm (Likie, 2008) and lo- cal alignments via the Smith- Waterman algorithm (Ligowski & Rudnicki, 2009). It can align two "Sequence" objects with potentially different alphabets, requiring a "Substitu- tionMatrix" " object that contains two alphabets of lengths n or m, respectively, along with a similarity score matrix of
C. Relevant Discussions
Quantizing protein structures in VQ- VQ-VAE-bart based PST meth- ods addresses practical challenges and offers benefits. Some aspects are as follows:
18

---

Protein Structure Tokenization: Benchmarking and New Recipe
shape m). This flexibility handle different alphabets significantly enhances the function's 's utility.
hidden dimension using a linear layer, then fed into the structure decoder. This decoder is composed using stack of bidirectional transformer blocks with standard self- attention (Vaswani, 2017).
For discrete structural tokens, both the two alphabets were derived from a learned codebook token vocabulary, and "Substiiution aatiix"" was defined using scaled cosine simi- larities between codebook vectors Q = {g.)/=1
Overall objective. The overall training objective is described in Sec. 2.2, which includes the commitment term, quantization term, and the reconstruction term. In AminoAseed, the reconstruction term is further designed the average of five structure reconstruction losses from ESM3: (1) Geometric distance and direction losses guarantee accurate backbone structure reconstruction; (2) binned distance and direction classification losses en- hance early training convergence; and (3) an inverse fold- ing token prediction loss, a cross entropy measure between the predicted and true sequences, serves as an auxiliary loss to enrich sequence-related information in the learned representations.
sim(i.)) = [100 * cos(q:
ú K.
For continuous structural tokens, each token was considered as an element in the alphabet, creating two alphabets sized according to the lengths I and L2 of the input protein structures. The similarity measure in the "SubstitutionMa- trix" was computed similarly using the continuous latent representations [ and (2
sim(i.j) = [100 cos( 2,)1, 2 L. Lz.
E.2. Geometric Self-Attention Layer
E. Method Details
The geometric self-attention layer (Hayes et al., 2025) in- volves transforming local frames into a global state to de- ploy a specialized attention mechanism. which assesses both rotational and distance similarities. After computing the at- tention scores and deriving the outputs weighted by these scores, the data is converted back to local frames. This ensures the model accurately captures the orientations and positional interactions among per-residue frames, essential for understanding the complex spatial relationships inherent in protein structures.
E.1. Overall Pipeline
AminoAseed is a VQ-VAE-base PST method, which takes protein frames as inputs for encoding, vector quantization, and decoding to reconstruct the protein structures.
Protein frames as inputs. Protein backbone structure is represented by the relative distance and orientation of frames defined by each residue's backbone coordinates. For residue i, its frame T; E SE) consists of rotation matrix R; E (3(3) and a translation vector t; E RR. The frame T; can be calculated using the standard Gram-Schmidt algorithm (see Sec. E.4).
E.3. Straight Through Estimator
As introduced in Sec. 2.2, the VQ layer maps the continuous latent representations = into a discrete embedding q using the codebook M. Because the discretization function is not continuously differentiable, a common approach for optimiz- ing this layer is via a "straight-through estimator" (Bengio et al., 2013). This approach effectively sidesteps the non- differentiable nature of the function, permitting gradient updates through backpropagation to optimize the layer.
Structure encoding. For each residue its local neigh- borhood substructure obtains the 16 nearest residues (mea- sured by C. distance). The structure encoder input includes the frame for each residue, the frames for its neighbor- ing residues, and their relative positional encodings. The structure encoder consists of a stack of geometric atten- tion blocks, where each block contains a geometric self- attention layer (detailed in Sec. F.2) and feedforward network (MLP+SwiGLU (Shazeer, 2020)). A linear layer is attached after the encoder to transform the hidden dimen- sion of the encoder output to the codebook dimension before emplying vector quantization.
Specifically, before applying STE, we can analyze the pro- cess of backward gradient propagation in 3 distinct stages;
ac ac 0qk dz Fz
Oyn Oz Or
Quantization. The vector quantization process of AminoAseed is introduced in Sec. 4.2, while that of VanillaVQ is described in Sec. 2.2. The quantization pro- cess is not differentiable. To enable the gradient flow back to the encoder, the straight-through gradient estimation is applied and details can be found in Sec. E.3.
0gs is the gra- 8C is the gradient through the decoder, 25 where dyz. dient through the VQ layer, and åz is the gradient through dr
the encoder. The non-continuous nature of the VQ transfor- mation means 82
dg.. is not computable.
To solve this non-differentiability issue, STE directly copies the gradients from gk to 2, by modifying the input to the de- coder to be qu - sg(2) + instead of ge- This modification
Structure decoding. After the quantization, the discrete latent representations are first transformed to the decoder
19

---

Protein Structure Tokenization: Benchmarking and New Recipe
circumvents the vector quantization step for gradient com- putation. Simplifying this in the backpropagation formula, STE treats ays as the identity matrix /:
length greater than 20; (4) more than 20% of the se- quence is the same amino acid. We then downsampled 10% of the protein chains to train our PSTs, resulting in 48,316 chains. The down-sampling is supported by the fact that training a protein folding model with as few as 1,000 pro- tein chains achieved a decent performance (Ahdritz et al., 2024). The downsampled protein chains have lower than 40% sequence identity to each other. To reduce the memory cost, filtered out proteins longer than 512.
0z
ac
ac dz dqp.
dr
dx
This alteration ensures that the entire backward pass remains differentiable despite the non- con-differentiabbliyy of VQ layer.
E.4. Gram-Schmidt Algorithm
Pre-training configuration details. We employed the same configuration for both of our implemented models AminoAseed and VanillaVQ. Specifically, our models were trained using an Adam optimizer with a linear warmup schedule to a peak learning rate of 0.0001, followed by cosine decay to 10% of the peak learning rate. We use a weight decay of 0.01. The training process involved 5,426 warmup steps and continued for total of 108,530 steps, lasting approximately 30 hours on 8 NVIDIA A100 GPUs. Each GPU processed a batch size of 4, without gradient accumulation, resulting in an effective global batch size of 32. During pre-training, DeepSpeed ZeRO training stage 2 (Rajbhandari et al., 2020) was employed reduce GPU memory footprint and enhance the full precision training of our models, by sharding optimizer states and gradients across GPUs
The
algorithm (Björck, 1994) is method of constructing an orthonormal basis from a set of vectors in an inner product space. In the algorithm, a translation vector t, and two vectors F and y define the local z-y plane, as illustrated in Alg. 1.
Algorithm 1 Gram-Schmidt Process
Require: e R Lx3 xx E RLx33 y 4
1: &
x
2: el -y-
-x y)
3: ê = e1
|ei]
4: eg & X êι
5: R = X é c2)
R t
6: T- =
01x3 1
For the VQ-VAE architecture, the codebook size was set to 512 with each codebook vector having a dimension of 1024, matching the capacity of ESM3's structure tokenizer to en- sure a fair comparison. The encoder utilized geometric attention blocks, featuring hidden dimension size of 1024 and 128 geometric attention heads. The decoder employed 8 traditional bi bi-directional self-attention blocks, each with a hidden dimension of 1024 and 16 attention heads. Addition- ally, the loss weight / for the commitment loss was fixed at a constant of 0.25.
7: return T
To create the residue frames T; we follow ESM3 (Hayes et al., 2025) to apply C. at the origin the frame, C' on the negative x-axis J). and N on the xy-plane.
The resulted frame Ti can be represented
T: = (
Ri 4i :) E SE(3)
01x3
Training and evaluation configuration details for super- vised downstream tasks. For supervised tasks, the continu- ous or discrete structural representations extracted by PSTs, along with absolute positional encodings were fed into a LayerNorm layer and dropout layer with a dropout ratio of 0.1, before proceeding to the probing layer. And the probing layer consisted of a two-layer MIP with a hidden dimen- sion of 512, ReLU nonlinearity, and a dropout layer with a dropout ratio of 0.1 between the layers. This layer produced per-residue logits for functional site prediction (binary clas- sification prediction), regression scores for physicochemical property tasks (regression prediction), and per-protein logits for structural property prediction (multi-class classification).
The rotation matrix R. E SO(3) rotates vectors to a local coordinate system where the N-Ca-C plane for the corre- sponding residue spans the z-y planel; and the translation vector f. E 3 specifies the position of the residue's 's C..
In the end, R; is composed of three 3-dimensional vectors [éi, é2, és). where é and éq are orthogonal unit vectors on the NCC.-C plane, and é3 is unit vector perpendicular to the plane.
F. Experimental Result Details
F.I. Setup
Training was managed using an Adam optimizer with co- sine annealed learning rate schedule, selecting peak learning rates from the set (0.1, 0.01, 0.001, 1e-4, 5e-5, 1e-5, 5e 5, 1e 63. The best learning rate was chosen based on the
Pre-training dataset preparation details. We applied the same criteria as those used for training the OpenFold2 model: (1) PDB structures deposited before 2020-05-01; (2) resolution better than or equal to 9A; (3) protein chain
20

---

Protein Structure Tokenization: Benchmarking and New Recipe
best validation Macro F1 for classification tasks, and best validation Spearman's s p for regression tasks. The training protocol included 200 warmup steps and total of 10,000 training steps. Each experiment was conducted on a single NVIDIA A10 GPU, with a per-GPU batch size of 8 for all supervised tasks, except for the Homo task, which used a batch size of 64. To manage peak memory usage effectively, protein sequences exceeding 600 residues were filtered out. All results reported were obtained using seed 1,234.
Table 9: Model comparison.
Model
#Params
Model Inpur
ProteiaMPNN
1.7M
Backbone Seructure
MIF
3.4M
Backbone Structure
FoldSeek
282
Backbone Seructure
ProTokkus
34M
Backbone Structure
AIDO
275M
Backbone Structure
ESM3
30M
Rackhone Semicture
VanillaVQ
31M
Backbone Structure
Aminokssed
31M
Rackhone Simetture
Cheap
96M
All-atom Stracture Sequence
Ablation experiment configurations for using continuous structural representations. For TQ-VAE-baser PSTs, we used their encoder output as the corresponding continuous counterparts, substituting them with the discrete structural representations as the input for supervised tasks. All other training and evaluation settings were kept the same as those in "Downstream Effectiveness" benchmarking for fair comparison. Notably, ProTookes was excluded because its released implementation did not grant access to its encoder output.
at LE Le-4. 4, utilizing& linear warm-up followed by cosine decay learning rate schedule to 10% of the peak learning rate. Global batch size was set to 32 proteins.
Scaling experiment configurations for varying encoder sizes. We fixed the size of the codebook matrix K x D at 4096 x 128 and the decoder of 8 traditional self-attention blocks, and then varied the size of the encoders by changing the hidden dimention sizes (from 32 to 2048) and number of geometric attention layers (from to 4) while fixing the number of geometric attention heads to 128. The total number of encoder parameters ranges from 107K to 96M.
Ablation experiment configurations for combining amino acids with structural representations. We applied learnable embedding layer to the 20 amino acid types, and add their embedding to the structural representations be- fore entering into the LayerNorm and dropout layer, before proceeding into the probing layer. All other training and evaluation settings were kept the same as those in "Down- stream Effectiveness" benchmarking for a fair comparison.
We pretrained these models on our pre-training data under five different compute budgets: (3c+ 17, 1c+18, 3c+18, let 19, and 3e + 19} floating point operations. The training configurations were the same as the scaling experiments that varying the size of the codebook described above.
Baseline model size comparison. The number of param- eters for each PSTs summarized in Tab. FoldSeek is very lightweight. Top-performaing VQ-VAE-based PSTs have about 30M parameters, while IF-based PSTs contain approximately 2M parameters.
Ablation experiment configurations for adding noise to structural representations. We masked the structural rep- resentations residue-wise and replaced them with a learnable [MASK] embedding to simulate adding noise. We calculated the proportion of masked residues in protein as the noise level, as shown in Fig. 5 and Fig. 11. Intuitively, a higher noise level indicates that the structural representations con- tain less meaningful information. All other training and evaluation settings were kept the same as those in "Down- stream Effectiveness" benchmarking for fair comparison.
F.2. More Sensitivity Results
Using TM-score instead of the negative RMSD as struc- tural similarity. We also used the negative value of RMSD as a structural similarity measure, and summarized the re- sults in Tab. 10. Compared to using TM-score in Tab. 5.3, the correlation values for RMSD presented across all meth- ods are significantly lower. This is because RMSD sen- sitive to outliers and local structural discrepancies, while TM-score focuses on global structure and is less sensitive to local structure variations. This comparison proves TM-score more suitable for assessing structural similarity in our Struct- TokenBench for sensitivity perspective evaluation. Despite RMSD's ' challenges, AminoAseed significantly outperforms other models, suggesting AminoAsecd's 's superiority in de- tecting structure conformational changes.
Scaling experiment configurations for varying codebook sizes. We fixed the size of the codebook matrix K x D at 4096 x 128, and varied the codebook size K and dimension D. The codebook size K ranged from 26 = 64 (with D 8192) to 21 (with D= D=11 1). We also fixed the sizes of encoder and decoders across codebook sizes, such that there were only small differences in encoder and decoder sizes across different codebooks, due to linear mapping to align the encoder output hidden dimension and decoder input hidden dimension with the codebook dimension.
We pretrained these models on our pre-training data for three different compute budgets (fle+18, 18, and le+119) floating point operations), with number of warm-up steps to 5% of total steps. The peak learning rate was established
E.3. More Ablation Study Results
Remaining task splits for performance comparisons be- tween discrete and continuous representations under
21

---

Protein Structure Tokenization: Benchmarking and New Recipe
Table 10: Sensitivity evaluation on conformational proteins, with conformer structural similarity measured by the nega- tive RMSD instead of the TM-score used in Tab. 3.
Performmnce (AUROC)
Performancc (AUROC)
Apo Holo
Fold Switching
Model
PCC%
Spearman's 's 5%
PCCS
Spearman's 's ps
ProteinMPNN
-0.003
0.0643
0.0057
0.0504
MIF
0.0033
0.0517
0.0442
0,0878
Nosse Level (%)
Noise Lovel (%)
Lovel
FoldSeek
-0,0112
0.0407
0,0073
0.0475
(a) BindInt (Fold) results.
(b) BindInt (SupFam) results.
ProTookess
0.0807
0.1569
0.1022
0.1161
s001
ESM3
-0.0442
0.133
0.0944
0.1252
Performence (AUROC)
Performance (AUROC) suss
VQ
0,0479
0.1247
0.0715
0.1080
AminoAsced
0.1207
0.2076
0,2737
0,3172
0.5
Noise Level
Noise Level (26)
Porformance
(c) CatInt (Fold) results.
(d) Con (Fold) results.
0.5
Preer APPOo
044
Paffrmance (AUROC)
Parformance (AUROC)
I8MC CortiOas
DSMI
6.3
0.-
TrltSeck-coniren
Poldiek
VazilalQ
Antooloeed austinoss
Arrizalteed
Nealor (SigFan)
Bestlior (Sapaan)
Ga Gargpetan
Cur (Saphan))
Rvy Olaplani
Rechurr (haps
Fluttr aeterr (hopl
Fleakig (SapFamd
Home (Sapfaan))
Henoo (Fan)
Level (%)
Naise Level
Task Split
(c) Rep (Fold) results.
(f) Rep (SupFam) results.
Figure 10: Performance comparison of using continuous ver-
Figure 11: Supervised performance with increasing noises in the PST-extracted structural representations, with more task splits reported in complement to Fig. 5.
sus discrete structural representations for the remaining super- vised task splits, in complement to Fig. 4.
F4. More Scaling Study Results
supervised downstream tasks. As shown in Fig. 10, for most SupFam task splits, continuous representations outper- form discrete ones, although the gains is marginal for ESM3, FoldSeek, and AsinoAseed. This observation, consistent with the finding in Sec. 5.6, indicates that the continuous format is not the primary factor contributing to performance enhancement.
Scaling of codebook sizes. We found VanillaVQ is not effective in minimizing the quantization loss, leading to up 2 magnitudes higher quantization and overall loss across codebook sizes (Fig. 12), highlighting the benefit of our reparameterization approach in effectively learning the codebook. We also observed that AminoAseed consistently achieved better reconstruction qualities over VanillaVQ across compute budgets and codebook sizes (Fig. 6).
Notably, the continuous representations from Vanilla match or exceed those from AminoAseed, bridging the per- formance gap observed with their discrete counterparts. This suggests that while the encoder of VanillaVQ could be opti- mized similarly to AminoAseed, AminoAseed optimizes its codebook more effectively using its proposed engineering techniques, thereby better aligning the encoder output with the codebook vectors with reduced distribution shift.
When analyzing the PST qualities across codebook sizes and compute budgets (Fig. 6), noticed that: (1) recon- struction qualities turn to be more similar across codebook sizes as compute increase; (2) the trend in UR and down- stream effectiveness tasks such as CatInt maintained across compute budgets. These observations suggest that although reconstruction quality can be improved with more compute regardless of the codebook sizes used, one cannot com- pensate the low UR and downstream effectiveness from suboptimal codebook sizes with more compute.
Extended supervised task split results for adding noises to the structural representation. In addition to the pre- viously reported Con (SupFam) and CatInt (SupFam) in Fig. 5, we include results for Con (Fold) and CatInt (Fold), as well as two additional tasks, BindInt and Rep, across both Fold and SupFam splits in Fig. 11. For most of these task splits, Asino consistently outperforms ESM3 across various noise levels.
We also note that although large codebook sizes 2 achieved optimal reconstruction quality, it suffers from low UR and downstream effectiveness (Fig. 6, Fig. 12). This hints a trade-off between structure generation and token
22

---

Protein Structure Tokenization: Benchmarking and New Recipe
effectiveness: larger codebook sizes may help the decoder to generate protein structures more accurately, it limits the downstream effectiveness of the structure tokens for super- vised downstream tasks and reduces utilization rates.
Validation Quamiiaation Lowe
Valliintoon Looe
3x
2-10
We highlight that the large codebook size also pose addi-
tional challenges: (I) it is hard to analyze all the structure patterns; (2) it leads to difficulty in downstream predictive tasks, as the predictive space large and similar code vec- tors could confuse each other.
Nimber codes (a) Validation loss.
(b) Validation quantization loss.
Seri8 berch
Coapua
Se1t
Medittye VaalinM)
Volidakke Commonment Los
Lerte
Scaling of encoder sizes. We noted that the encoders in VQ-VAE-based PSTs (both AminoAseed and VanillaVQ), do not demonstrate power-law scaling with compute and training data ( (Fig. 7). We speculate this is likely due to (1)
Vilidatoon Drribbone
the inherent redundancy in protein structure data; and (2)
the performance is limimted by the decoder capacity. Future
(c) Validation commitment loss.
Number of codes
(d) Validation backbone RMSD.
Number ondee
research is needed to delve deeper into the scaling laws of VQ-VAE models.
Wedlttge
Chipost
Hoiitopp
Ascan/ooo
Valisaation LDDT
E5. More Visualization Results
More visualization of local neighborhoods where protein conformer structure variations are correctly detected. In Fig. 13 we show three common examples of Amino/seedds s "vocabulary".. In each case, we show an example from three different proteins, co-aligned using the residue with the indi- cated token index, and the residues just before and just after that in the sequence. A very common example (Fig. 13()) is simple turn linking other secondary structural elements. The second example (Fig. 13(b)) is an alpha helical element, although we note that while the backbones align almost perfectly, conventional tools for detecting alpha helices do not agree about whether these three examples are part of an alpha helix or not. Finally, we show an example of one 3-sheet token (Fig. 13(c)).
Nomber of codes
Nember codee (l) CASP14 data LDDT.
(e) Validation LDDT.
Calmni (SggF)))) Peageaa Perfeemmnce
CAMFO LDDT
(g) CAMEO data LDDT.
(h) CatInt (SupFam) results.
Coapui
Mebdrops
BoudBio (NopFam)) Reafoemers
Chipaec So-30
RiniBiia (Fuld) (Falll Pofformmnns:
with
Another way visualize the differences between tokeniz- ers is to see how the token indices change when there are structural changes in particular protein, as there are when binding small molecule. We can compare the apo (without ligand) and holo states to look for differences in the tok- enization. In Fig. we compare the AminoAseed tokenizer to ESM3's s structure tokenizer. To illustrate, we compare the apo (PDB ILIP) and holo (PDB 1JTB) states of lipid transfer protein. The upper left panel of Fig. 14 shows that
3.5-
Nimber of codes BindBio (Fold) results.
Number of codes (i) BindBio (SupFam) results.
(Sag Per Performence
FleeNE (Fold) Per Porormunc
to accommodate the ligand, an o-helix displaces to the right, as the loop in the foreground extends. AminoAseed differs from FSM3 in where exactly this extension from apo to holo
becomes detectable, highlighting residues 61 and 64, while ESM3 highlights residues 62 and 63. There is not a clear winner here: the rest of Fig. 14 shows the structure aligned at residue 64 (upper right), 63 (lower left) and 61 (lower right). The backbone environment around each residue ap- pears to be different between apo and holo forms, yet neither tokenizer distinguishes all positions. AminoAseed detects
codes
Namber Codee
(k) FlexNEQ (Fold) results. ()) FlexNEQ (SupFam) results.
Figure 12: Scaling impact of codebook sizes on: (a-c) different losses on holdout validation set; (d-g) reconstruction quality on validation set, CASP14 and CAMEO test set; (h-l) more supervised task split performance evaluating the Downstream Effectiveness perspective of PSTs.
23

---

Protein Structure Tokenization: Benchmarking and New Recipe
(a) Token ID 358 for PDB IDs 7ABW Leu232 (marine blue), 7M7A
His307 (orange), 7MHU Ala214 (magenta) display a common turn motif.
(b) Token 47 for PDB IDs 7MHU Leu287 (magenta), 7M7A Glu465 (marine blue), 6POO Asp258 (orange) display alpha-helical motif.
Figure 14: apo (salmon) and holo (grey) structures of lipid transfer protein illustrate tokenizer differences. Upper left residues tokenized differently in apo and holo states by AminoAseed are shown in magenta; those tokenized differ- ently by ESM3 are shown in blue; others are tokenized differ- ently by both models. Upper right Aligned on the backbone of residue 64. Lower left Aligned on residue 63. Lower right Aligned on residue 61.
(C) Token 253 for PDB IDs 7MHU Ala69 (magenta), 7K7W Glu74 (salmon), 6S44 Asn80 (blue) display common 8-sheet motif.
of codes optimized, which is visible in the upper right of VanillaVQ's distribution plot and appears as deviation from the initial setup. Moreover, the comparison between Vanilla and AminoAseed highlights the effectiveness of our proposed engineering techniques in enhancing codebook optimization.
Figure 13: Examples of our structural vocabulary. Motifs are shown with side chains visible, the remainder of each protein shown only C. ribbon.
the most subtle difference, at position 64, but misses a much less subtle difference at positions 62-63. Both tokenizers agree that the apo and holo forms of remainder of this loop different at each position.
In Fig. 15(b) and (c), we present the continuous structural representations extracted from the PST's encoder, applied to the CASP14 and CAMEO datasets, respectively. The encoders for ESM3, VanillaVQ, and AminoAseed have suc- cessfully learned discemnibe patterns.
t-SNE for discrete codebook vectors and continuous structural representations from PST encoders. In Fig. 15(a), on the one hand, ProTokens, ESM3, and AminoAseed display distinct patterns their codebooks for per-residue protein structural representations. Tokens representations form many small, tightly clustered groups that are widely spaced from each other. In contrast, ESM3 and AminoAseed exhibit more diverse distributions, featur- ing both large and small clusters. On the other hand, the codebook vectors of FoldSeek and Vanilla appear more normally distributed. This is likely because FoldSeek has too few codes, while VanillaVQ starts with a random initial- ization from normal distribution, with only small number
F.6. More Baselines
As suggested by anonymous reviewers, we add two more baselines: AIDO.st (Zhang et al., 2024d) which VQ- VAE-based PST, and Cheap (Lu et al., 2024) which replaces the quantization step with tanh layer to get continuous embeddings instead of discretized embeddings used in VQ- VAE. Specifically, AIDO.st takes backbone structures as input and thus directly comparable to all results in the main text; while Cheap models all-atom structures and pro- tein sequences input, placing it outside this benchmark's
24

---

Protein Structure Tokenization: Benchmarking and New Recipe
ToldSeek
Pro Takens
ISMB
VanillaVQ
Asee
(a) Codebook embedding distribution.
FoldSeck
ESMS
VanillaVQ
AminoAscod
(b) Continuous structural representation distribution inferenced by PSTs on CASP14 datasct.
FoldSeek
ESM
VanillaV
AminoAseed
103
16
(c) Continuous structural representation distribution inferenced by PSTs on CAMEO dataset.
Figure 15: Visualization for discrete and continuous structural representations extracted from PSTs using t-SNE.
25

---

Protein Structure Tokenization; Benchmarking and New Recipe
primary focus. Their model sizes summarized in Tab. 9.
As shown in Tab. 11, Tab. 12 and Tab. 13, despite its large size (275M parameters), AIDO.st is less effective than AminoAseed on downstream effectiveness, and also fall short in sensitivity. However, sIDO.st achieves a very high codebook utilization efficiency. Cheap beats IF-based PSTs and AminoAseed on supervised tasks, but largely lags behind AminoAseed in sensitivity. Codebook utilization efficiency result is not applicable for Cheap it has codebook.
Table 11: Benchmark results for supervised downstream tasks for two additional baselines. The table format follows Tab. 2.
Model
Task
Split
AIDO.st
Cheap
Functional Site Prediction (AUROC%)
Fold
44,66
59.87
BindInt
SupFam
84.21
97.38
G. Related Work Details
BindBio
Fold
65.50
85,59
SupFam
66.70
88.27
VQ- -Q-VAE-based PSTs have been rapidly emerging, Fold- Seek (Van Kempen et al., 2024), the pioneering PST model, significantly speeds up protein structure alignment. Subse- quent models targets enhancing protein language models with structure knowledge, either building on Foldseck's 's to- kenization, such as ProstT5 (Heinzinger et al., 2023) and SaProt (Su et al., 2023). or building their own tokeniza- tion model, like ProSST (Li et al., 2024) and ESM3 (Hayes et al., 2025). ProTokens (Lin et al., 2023a) facilitates protein structure generation by leveraging the quantized codes to remove group symmetry and polymer restraints. To enhance the quality of PSTs, some methods refine the standard VQ- VAE training. Notably, Gaujac et al. (2024) applies Finite Scalar Quantization (FSQ) (Mentzer et al., 2023), and Fold- Token (Gao et al., 2024) employs Soft Conditional Vector Quantization. Other approaches focus on sppecfc enhancements: Bio2Tokens (Liu et al., 2024) expands the structural resolution from backbone atoms all-atom level, while FoldToken (Gao et al., 2024) and CHEAP (Lu al., 2024) propose tokenize the joint distribution of sequences and structures, instead of just structures, to better capture the complex interplay between the two modalities.
BindShake
Org
69.28
87.74
Fold
57.30
65.20
Catlnt
SupFam
81.94
97.14
Fold
73.72
93.91
CatBio
SupFam
78.66
95.78
Fold
56.64
61.12
Con
SupFam
73.79
95.14
Fold
77.69
77.35
Rep
SupFam
78.08
80.45
Fold
60.26
64.13
Ept
SupFam 72.30
78.83
Average AUROC%
69.38
81.86
Physicochemical Property Prediction (Spearman's (%)
Fold
33.15
50.56
FlexRMSF
SupFam
26.93
48.59
Fold
18.88
37.17
FlexBFactor
Suplaam
19.31
37,90
Fold
16.41
60.69
FlexNRQ
SupFam
16.17
60.18
Average 9%
21.81
49.18
Structure Property Prediction (Macro F1%)
Fold
7.89
40.55
Homo
SupFaam Fam
4.77
58.50
13.71
92,89
Average Macro
F1%
8.79
63.98
Table 12: Sensitivity evaluation conformational proteins for two additional baselines.
Apo Holo
Fold Switching
Model
PCC%
Spearman's p%
PCC%
Spearman's 's p%
AIDO.st
43.02
54.25 40.20
61.59 48.34
66.11 11
Cheap
33.38
52.51
Table 13: Codebook utilization efficiency evaluation on CASP14 and CAMEO datasets for AIDO.st. Note that
Cheap does not contain discrete codebooks, thus cannot be evaluated for efficiency.
Model
ACode (A)
Dim (D)
UR%
Peplexity
MUV
URS
Perplexity
MUV
AIDO
512
384
88.05
0.7729
2.53.4
95.12
08266
2.50x-4
26


Reviews:
[{"review_id": "Uc0RxPWyyK", "reviewer": "Reviewer_LsQQ", "summary": "This paper presents a benchmark for comparing methods of tokenizing proteins. They divide structure tokenization methods into two categories: those which hand-design structure based tokens, and those which learn the tokenization. Of the learned methods, they distinguish between those that produce learned codebooks, and those that use inverse folding (i.e., the “tokenization” is a sequence of amino acids that are supposed to fold into the given backbone). They suggest evaluating methods along four axes: (1) Effectiveness as an input to supervised learning tasks, (2) sensitivity to distinguish similar structures, (3) distinctiveness of codebook vectors, and (4) efficiency, i.e. how uniformly the different codebook elements are used. Evaluation of (1), effectiveness, is the most involved, and entails training a 2-layer MLP on the structure tokens (to which a positional encoding is added) for a variety of different supervised tasks, including prediction of binding sites, conserved sites, and catalytic sites. Applying their benchmark to baselines, they find that some under-utilize the learned codebooks. As a result, they define a new tokenization method, “AminoAseed”, that (1) uses reparametrization during the codebook gradient update to prevent “codebook collapse, and (2) a data-based heuristic for trading off the number of codebook elements and the dimension of each element. AminoAseed does better on the new benchmark than existing approaches. Finally, they conduct ablations and scaling experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We address concerns below\n>R1: Explain “StructTokenBench focuses on local(per-residue) over global(per-protein) structures”\n\nPSTs tokenize per-residue substructures, matching StructTokenBench: **supervised tasks and “Sensitivity” are at residue level**. Current benchmarks(Sec. 6.1) cover per-protein tasks, justifying our per-residue focus. We’ll add in main text\n>R2: No citation for “Distinctiveness” motivation(L179)\n\n**We’ll cite ESM3 Fig. S5**: ambiguous structural token mapping harms. This metric assess the token ambiguity via codebook similarity. \n>R3: Efficiency(Codebook Utilization) metric seem conflicting with Sec 5.7(L415): no correlation with reconstruction quality. Is it needed?\n\nWe rename “Efficiency” to “Codebook Utilization”(% of used codes). Underutilized codebooks waste resources and harm downstream perf.(Sec. 3.4), a well known issue in CV/NLP[a]. High utilization bring gains, like LLaMa3’s 128K-token vocabulary for efficient encoding[a]. **Sec. 5.7 questions if reconstruction quality alone reliably indicates PST quality**, given no correlation with **both supervised task perf. and utilization**(see Fig. 6,12) We welcome further discussion\n[a]LLama technical report\n>R4: Codebook utilization is distribution-dependent. PSTs' non-uniform token use may reflect accurate reconstruction(like BindGPT)\n\nWe agree on dependence. Current results hold across datasets. We’ll add more diverse datasets. We remain inconclusive on exact utilization-reconstruction relations. **Shown in Sec. 5.7**, we question if reconstruction quality alone reliably indicates PST quality, given no correlation with **both supervised task perf. and utilization**. We welcome further discussion. Thanks but we can’t test it as it lacks residue-level structural reprs.(see **R8**)\n>R5: Why use probing for supervised tasks? Why MLP not transformer?\n\nProbing(fixed encoder+simple probing layer) is standard practice in CV[b]. Being simple ensures perf. reflects reprs. quality, not probing layer capacity. For BindInt, empirical perf. below show: linear layer lags 2-layer MLP, which matches 1-/2-layer transformer perf. Limited dataset sizes(Tab. 6,8) may explain transformers' marginal gains\n|Method|Split|Linear|2L MLP|1L Transformer|2L Transformer|\n|---|---|---|---|---|---|\n|**ESM3**|Fold|43.36|44.30|46.78|46.08|\n||SupFam|84.50|90.77|90.98|90.96|\n[b]Masked Autoencoders Are Scalable Vision Learners\n>R6: Why use positional encodings(PEs), as MLP isn’t permutation-invariant? \n\nOur MLP is shared across structural tokens, so **is permutation-equivariant**. So it needs PEs for residue order. Empirically, removing PEs largely hurts perf.(see table below vs. Tab. 2)\n|ESM3 no PEs|BindInt|CatInt|Con|Rep|\n|---|---|---|---|---|\n|Fold|48.85|53.14|51.05|51.36|\n|SupFam|74.16|69.42|62.03|62.15|\n>R7: Test Cheap\n\nIt’s initially excluded as it models **sequence and all-atom structure**(37 atoms/residue), while our benchmark only considers backbone(4 atoms/residue, Sec. 2.1). Cheap’s perf.(vs. Tab. 2,3) is in **https://anonymous.4open.science/r/cheap_perf-00EB** limited by space: (1)it beats IF-based PSTs on supervised tasks (some results pending) (2)It largely lags AminoAseed in “sensitivity”. (3)No other evaluation as its has no codebook\n\n>R8: Test BindGPT, Geo2Seq\n\nThanks for referring to atom-wise coordinate tokenization methods. We’ll discuss in main text and appendix. BinGPT is untestable as: (1)It encodes atom types and xyz coordinates as strings. Unlike IF-/VQ-VAE PSTs, it doesn’t output residue-level structural reprs; (2)Being decoder-only, BindGPT also can’t produce residue-level reprs. Geo2Seq is not open-source\n>R9:Heuristic PSTs not tested\n\nWe’ll add them later due to deadline\n>R10:For fixed reconstruction quality, measure bits used\n\nWe use scatter plot: X-axis: reconstruction quality(RMSD or LDDT),Y-axis: Compression ratio(% byte reduction vs. XYZ coordinates).**Plots in https://anonymous.4open.science/r/rebuttal_fig-3050**. AminoAseed beats VanillaVQ in reconstruction but trails ESM3, while achieving better compression(smaller byte reduction %) than both\n>R11: Explicitly motivate PSTs\n\nWe’ll add**R3/R4** discussions on distinctiveness and codebook utilization in main text\n>R12:State PSTs' modeling targets\n\nWe’ll add **(1)Tab. 2 footnote**: Cheap(sequence + all-atom structure) vs. others( backbone structure) and **(2)appendix table**: PST comparison (backbone/all-atom, sequence usage) including non-open-sourced\n>R13: Re-term\n\nWe change “efficiency” to “codebook utilization” and “effectiveness” to “downstream effectiveness” for clarity\n>R14:Computational efficiency as future work\n\nWill leave for future\n>R15: Intention of noise analysis\n\nNoise is added by randomly masking structural tokens. We reveal: (1) Structural reprs. are less robust than sequence reprs., **so integrating sequence-structure may help** (2) Practical guidance **for masked language modeling on structural tokens**\n>R16:Typo in L1029\n\nWill modify", "claims_and_evidence": "For the most part, yes, but with a few minor exceptions below:\n\nThe claim that their benchmark focuses on “fine-grained local substructures rather than global structures, as typical in existing benchmarks” (from the abstract). It wasn’t clear to me how this is true — is it due to the nature of the supervised tasks chosen for the “efficiency” section? I recommend this be explained more explicitly in the main body, if the claim is made in the abstract.\n\nThere are also claims about the importance of distinctiveness (how different the codebook elements are from one another) and efficiency (how much the different codebook elements appear), which are used to motivate these aspects of the benchmark, but Section 6 (L415) itself shows that efficiency and reconstruction accuracy aren’t correlated. Shouldn’t this call into question the efficiency metric? L179 also makes a claim about the utility of distinctiveness in downstream tasks, without a reference/citation to support it.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Not in very great detail. However, I do question, in the supervised tasks, why an MLP was used with positional encodings — first, if the intended downstream use case for PSTs are generally LLMs, would it not make more sense to train a transformer? Second, if one is committed to using an MLP, what’s the justification for a positional encoding? An MLP is not permutation invariant.", "relation_to_broader_scientific_literature": "This work is relevant to essentially all prior works that use protein tokenization as a way of dealing with 3D structures in a LLM setting. To my knowledge, there has not been as thorough of a systematic comparison of these methods before, and having a standardized benchmark is quite valuable to the community. AminoAseed builds on ESM3 and other prior VQ-VAE approaches.", "essential_references_not_discussed": "An essential reference not discussed in detail is “Tokenized and Continuous Embedding Compressions of Protein\nSequence and Structure” by Lu et al 2024. The reference is noted in the appendix, but not the main body. I presume this is because it is an all-atom tokenization, i.e. it essentially includes the residue information (not just the backbone), and indeed, it takes the amino acid sequence as input. However, I think this is a key reference because their evaluations are aimed at comparing different protein tokenizations, and also have a similar narrative of “finding and correcting an existing problem in a PST”.\n\nIn addition, the paper “BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning” by Zholus et al 2024 is one example of a paper that uses a simple atom-wise coordinate tokenization, essentially just tokenizing the coordinates as numbers. (Similarly, Geo2Seq in “Geometry Informed Tokenization of Molecules for Language Model Generation” by Li et al 2024 tokenizes 3D coordinates, but in spherical coordinates.) These kinds of simple methods — which are inefficient but very accurate as compressors — should be benchmarked as well.", "other_strengths_and_weaknesses": "Strengths: \n\nThe problem addressed is an important one and addresses a gap in the literature. Having a standardized benchmark for comparing protein tokenization methods (that is relatively independent of the downstream autoregressive model, and in particular does not require training many large models from scratch with different tokenizations to figure out which one is best), is quite useful. The benchmark is thoughtful, covering a wide range of tasks and facets, which have seemingly been implemented with very careful attention to detail. The empirical analyses (ablations and scaling studies) provide valuable insights for future work. \n\nWeaknesses: \n\nFirst, the paper leaves out several PSTs from its framing and experiments. As mentioned in the “related work” section of the review, there are many structure tokenizations that consist of tokenizing the 3D coordinates of the backbone atoms. Generally speaking, the introduction mentions “heuristic” tokenizations, but as far as I can tell, doesn’t compare to any of them. Also, a tokenization benchmark would be useful for structure tokenizations that include sequence as well (not just backbone), but methods like CHEAP are not included.\n\nMoreover, the “distinctiveness” and “efficiency” desiderata articulated in the benchmark itself — feel a bit ad-hoc to me. This is supported by the paper’s own findings in Section 6, which counterintuitively show that reconstruction accuracy and efficiency are not correlated.\n\nIn fact, the “efficiency” metric is seemingly quite distribution-dependent. A PST might induce a uniform distribution over tokens on one set of proteins, but a non-uniform distribution on a very different data distribution. This would not be a failing of the PST, but potentially an ability of it to accurately reconstruct diverse proteins. Indeed, this generalizability is one advantage of the coordinate-wise tokenizations (e.g. used in BindGPT and others), but they may fail at the “efficiency” metric. \n\nA more standard metric from the channel compression formalism of information theory is simply to assess how effective the methods are as compressors, i.e. for a fixed reconstruction accuracy (based on solving the “reconstruct structure from PST” learning task), how many bits are used to represent the structure (on average, on a given dataset)?\n\nFinally, it seems like the benchmark assumes that the proposed probing method is a good proxy for how a PST will perform in downstream tasks, but this is not supported. It would be good to have evidence for this e.g. on a single task, ablating architecture choice and other factors.\n\nOverall, I think this paper still makes very valuable contributions, and recommend acceptance. But if the weaknesses above were addressed, I would probably increase my rating further.", "other_comments_or_suggestions": "Minor writing suggestion: I would suggest more explicitly motivating how PSTs are used downstream, so that each aspect of the benchmark can be traced back to a concrete goal for PSTs. This is currently not clear to me, specifically efficiency and distinctiveness. \n\nMinor writing suggestion: For clarity, I would recommend being more explicit (even in the introduction, or perhaps a footnote/appendix) about when the tokenizations capture backbone only vs residues also. This can be confusing to newcomers, especially distinguishing from representations of structure in the literature that are referred to as “all atom” or “joint sequence and structure”. (For example, what does “amino acid tokens” near L052 refer to?)\n\nMinor writing suggestion: This is subjective, but in my opinion “effectiveness” and “efficiency” are not the most intuitive terms for the attributes the benchmark is measuring. (I often had to check back to figure out which term I wanted to use when writing this review.) Perhaps something like “codebook utilization” instead of “efficiency” would be preferable, or “downstream effectiveness” instead of just “effectiveness”. \n\nL1029: typo, “linear layer, ,”", "questions_for_authors": "1. For the “efficiency” part of the metric, why is the positional encoding necessary when the probing network is an MLP? Positional encodings are traditionally used for transformers, which are permutation invariant by default. On a related note, why use an MLP over a transformer? Wouldn’t we expect a transformer to correspond more faithfully to downstream performance, if most of the PSTs are going to be passed into a transformer-based architecture in practice?\n\n2. Did you consider the computational efficiency of computing the structure-based tokens at all? This could be a future addition to the benchmark, e.g. compute-controlled efficiency. Alternatively, perhaps one cares about compute time in the downstream application (e.g. for a transformer, the context length, which doesn’t intrinsically have to be the number of amino acids).\n\n3. Why do “distinctiveness” and “efficiency” inherently matter? As mentioned above, these concepts might be at odds with faithfully representing structure under distribution shift. If a tokenization is useful for downstream tasks and enables small context window (i.e. the number of tokens is small), why would we care about these specific properties of the codebook elements? Instead, why not just frame as “effectiveness as a compressor” (i.e. expected reconstruction accuracy vs size of compressed representation in bits, or context length)?\n\n4. What is the intention of the noise robustness analysis? It’s not clear to me if we should expect a good tokenization to be robust to generic noise or not. It seems like certain perturbations *should* meaningfully change the PST, whereas others (e.g. if the result is non-physical in some way) should not.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9wTI0tRnf", "reviewer": "Reviewer_gEVF", "summary": "In order to fully evaluate the performance of existing PST methods, the authors constructed a comprehensive evaluation benchmark called StructTokenBench. AminoAseed, a new improvement scheme, is proposed to address the problem of “codebook collapse” in the traditional VQ-VAE method, whose main innovations include codebook reparameterization and pareto-optimal codebook configuration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments\n>R1: The claim “adding “a MLP” to reparametrize the codebook can alleviate codebook collapse” needs theoretical analysis and ablation experiments\n\n1. We'd like to clarify that in Sec. 4.2 (L246), we **add a simple linear layer** (instead of a MLP) to reparameterize the codebook $C$ as $Q$=Linear($C$), and keep $C$ fixed during training\n\n2. **Ablation experiments**: VanillaVQ is the ablation model, which shares the same pipeline and configurations as AminoAseed, except using $C$ instead of Q=Linear($C$). Main results can be found in Tab. 2/3/4/5 and Fig. 3/4/5/6/7. Generally, AminoAseed outperforms VanillaVQ across all four benchmarking perspectives\n\n3. **Theoretical analysis**: We analyze the gradient flow on $Q$ in AminoAseed and $C$ in VanillaVQ, respectively. We will add this theoretical analysis in the appendix to enhance the method design justification\n\n    **(1) Setup**: Assume $C$ of shape (K, D) is fixed, Linear() denotes a linear transformation with weight $W$ of shape (D, D) without a bias, and the reparameterized codebook $Q$=Linear(C)=$C * W$. We denote $L$ as loss, batch size as 1 for simplicity, and $p$ different codes are selected during tokenization with each code selected for once\n\n     **(2) Gradient on $Q$**: For the gradient $\\nabla_Q L$, only the $p$ selected rows of this gradient matrix are non-zero. The gradient updates $W$ via the chain rule: $\\nabla_W L = C^T\\nabla_Q L$, which is now a dense matrix with all non-zero values. Thus, all entries of $W$ are updated, enabling global adjustments to the entire codebook space through $Q=C * W$\n\n     **(3) Gradient on $C$**: if $C$ is trainable, the gradient matrix $\\nabla_C L$ only has the $p$ selected rows as non-zero. Thus, unused codes get no updates, leading to limited changes for the entire codebook space and eventually distribution shift (see Fig. 2)\n>R2: Why using the VQ-VAE framework when IF-based PSTs beat VQ-VAE-based methods on supervised tasks? And if AminoAseed sufficiently closes the perf. gap?\n\n1. IF-based PSTs excel in supervised tasks but lack sensitivity to conformational changes, limiting their overall capability.\n\n2. IF-based PSTs only support continuous tokens, while VQ-VAE-based PSTs produce both continuous and discrete tokens. As said in App. C, discrete tokens offer several advantages: \n\n    **(1) Multimodal LLM integration**: enabling seamless fusion with sequence and functional text data, allowing direct use of NLP optimization techniques developed for LLMs.\n\n    **(2) Simplified structure modeling**: eliminating the need to explicitly encode symmetry and physical constraints\n\n    **(3) Reduced overfitting risk**: discrete representations may help mitigate overfitting compared to continuous features [a]\n\n3. **We acknowledge that there is still much more room for further improvement to close the performance gap** between AminoAseed and IF-based PSTs on supervised tasks. AminoAseed proposes a simple yet effective strategy for improvement over vanilla VQ-VAE, and enjoys the benefits of VQ-VAE methods and shows promising results in “sensitivity” evaluation\n\n[a]DeProt: Protein language modeling with quantized structure and disentangled attention\n>R3: Benchmarking AIDO.st\n\nAs suggested, we add AIDO.st and report the results below (some supervised tasks are not finished due to the rebuttal timeline). As shown, AIDO.st is less effective than AminoAseed on supervised downstreams, and also fall short in “sensitivity”. However, AIDO.st achieves very high codebook utilization\n\n|**AIDO.st Effectiveness**|Con|Rep|BindInt|CatInt|BindBio|CatBio|\n|---|---|---|---|---|---|---|\n|Fold|56.64|77.69|44.66|57.30|65.50|73.72|\n|SupFam|73.79|78.08|84.21|81.94|66.70|78.66|\n\n|**Sensitivity**|Metric|**AIDO.st**|\n|---|---|---|\n|apoholo|pearsonR|\t43.02|\n||spearmanR|54.25|\n|foldswitching|pearsonR|61.59|\n||spearmanR|66.11|\n\n|**AIDO.st Codebook Utilization**|UR%|Perplexity|Entropy|\n|---|---|---|---|\n|CASP14|88.05|0.7729|0.01165|\n|CAMEO|95.12|0.8266|0.01181|\n>R4:  Detailed description or diagram of the AminoAseed model architecture are not provided\n\n1. We provided the **detailed description** for AminoAseed **in App. E**, including **(1)** overall pipeline of protein frame input, encoding, tokenization, decoding, and structure reconstruction objectives; **(2)** the geometric self-attention layer used in encoder; **(3)** explanation of straight through estimator of gradients in tokenization; **(4)** the standard Gram-Schmidt algorithm to create protein frames as input\n\n2. We will add a **diagram** for AminoAseed in appendix for next edition\n\n>R5: The absence of complete code release undermines the reproducibility.\n\nWe apologize for not mentioning it in the paper. **We will release all the code and processed data in the next months**, because we’re currently in the process of code cleanup and preparation. \n>R6: Typo of in Sec. 2.3\n\nThanks for spotting the typo and we will modify it", "claims_and_evidence": "- Claim 1: Adding an MLP can alleviate codebook collapse.\n\n \nEvidence Concern: It is not fully clear why the additional MLP layer effectively propagates gradients to all codebook vectors. The explanation could be enhanced by further theoretical analysis and ablation experiments.\n\n- Claim 2: IF-based methods outperform VQ-VAE–based methods on many supervised downstream tasks.\n\nEvidence Concern: Although experimental results indicate this trend, it raises questions about the advantages of using a VQ-VAE framework and whether the proposed modifications (AminoAseed) sufficiently close this performance gap.", "methods_and_evaluation_criteria": "The paper proposes modifications including an MLP for codebook reparameterization and a data-driven Pareto-optimal configuration. These modifications are conceptually interesting; however, more detailed descriptions and comparative evaluations (e.g., against related methods such as AIDO.St) are needed to validate their effectiveness.", "theoretical_claims": "There is an implicit theoretical claim that the MLP layer can mitigate codebook collapse by ensuring all code vectors receive gradient updates. However, the paper lacks a rigorous proof or in-depth discussion on this mechanism. Further theoretical justification would strengthen the contribution.", "experimental_designs_or_analyses": "The experimental setup covers multiple supervised tasks and evaluations (effectiveness, sensitivity, distinctiveness, efficiency).", "relation_to_broader_scientific_literature": "The paper builds on prior work in protein structure tokenization, VQ-VAE, and inverse folding.", "essential_references_not_discussed": "The paper does not discuss related work such as AIDO.St, which appears to be relevant in the context of efficient protein structure tokenization. Including a comparison with AIDO.St would provide a more complete picture of the current state-of-the-art.", "other_strengths_and_weaknesses": "Strengths:\n- The paper proposes a comprehensive benchmark covering effectiveness, sensitivity, distinctiveness and efficiency.\n- The paper tackles an important problem by addressing codebook collapse in VQ-VAE–based PST.\n- The introduction of a data-driven codebook configuration is innovative and supported by extensive experiments.\n\nWeaknesses:\n\n- The explanation of how an MLP helps alleviate codebook collapse is insufficiently detailed.\n- Experimental results show that IF-based methods outperform VQ-VAE–based ones on many supervised tasks, raising questions about the benefits of the proposed approach.\n- The lack of comparison with related methods such as AIDO.St [1] limits the scope of the evaluation.\n- The paper does not provide a detailed description or diagram of the AminoAseed model architecture.\n- The absence of complete code release undermines the reproducibility and reliability of the results.\n\n[1] Balancing Locality and Reconstruction in Protein Structure Tokenizer.", "other_comments_or_suggestions": "Typo: In Section 2.3, **VQ-VAE** in 'As illustrated in Fig. 1(b), VQ-VAE can be summarized as' should be **Inverse-Folding**.", "questions_for_authors": "Please see Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxxwR0m7WC", "reviewer": "Reviewer_9RGj", "summary": "The paper presents a new framework, StructTokenBench, for evaluating protein structure tokenization (PST) methods, which break down protein 3D structures into discrete or continuous representations. This framework is critical because existing methods for protein structure tokenization (PST) lacked a unified evaluation system.\n\nKey highlights:\n\nStructTokenBench evaluates PSTs across four axes: effectiveness, sensitivity, distinctiveness, and efficiency. The framework focuses on local, fine-grained protein substructures, unlike typical benchmarks that focus on global structures.\n\nAminoAseed is introduced as an improved method over existing VQ-VAE-based PSTs. It addresses the issue of \"codebook collapse,\" where many codes in the latent space remain unused. The proposed solution introduces codebook reparameterization and Pareto-optimal codebook configurations. These methods help increase the efficiency and utilization of the codebook.\n\nBenchmarking Results: The paper compares AminoAseed with other leading PSTs like ESM3, ProTokens, FoldSeek, and ProteinMPNN, showing a performance improvement on some tasks\n\nChallenges Identified: Current PST methods, especially VQ-VAE-based approaches, struggle with efficient codebook utilization, as large portions of the codebook remain unused, leading to reduced model expressiveness.\n\nAminoAseed Performance: The proposed method performs well in tasks that require a high sensitivity to structural changes and achieves significant improvements in codebook utilization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for invaluable suggestions. We respond to your questions below\n\n>R1: AminoAseed doesn’t perform well on binding site prediction in Fig. 4\n\nWe'd like to clarify our observation:\n1. **Fig. 4 shows that when using continuous structural representations**, AminoAseed beats ESM3 on all three binding site prediction tasks, while being comparable to the continuous version of VanillaVQ on two of the tasks.\n2. **When using discrete structural tokens**, which is the most important application setting for PSTs, AminoAseed prevails for most of the tasks, and gains improvement for averaged task perf. (in Tab. 2)\n\n\n>R2: Paper shows the benefits of scaling the model (e.g., increasing codebook size or encoder size) and the diminishing returns as model scales\n\nWe'd like to clarify this rephrased statement from the paper:\n1. We show in paper the benefits of data-driven Pareto-optimal scaling (i.e., balancing) for the codebook size and codebook dimension. It’s important to notice that we kept the total codebook parameters the same\n2. We show the diminishing sub-exponential returns in Fig. 7 when increasing encoder sizes, keeping the codebook configuration unchanged\n\n>R3: Pareto-optimal codebook configuration and codebook reparametrization, while innovative, add complexity to the model\n\nWe'd like to clarify that **only codebook configuration adds complexity** due to multiple runs of model training, while **reparameterization shares the same complexity** as the ablation model VanillaVQ\n\nIn general, for the codebook configuration, **we leveraged the additional resources for exploration and provide insights for others** who aim to develop better PSTs: under a fixed computational resources, balancing codebook sizes and dimensions could be useful. This is a discovery that’s previously unknown in the field\n\nSimpler methods without using more computational resources are indeed favored, and we would like to explore it in our future work\n\n>R4: PST methods, including AminoAseed, tend to lose reliability under high noise levels in the input protein data\n\nWe would like to clarify that in Fig. 5, the noise is added to the induced structural representations, not on the raw protein structures. This is achieved by replacing them with the [Mask] representation under a given mask rate.\n\nEssentially, it’s expected to observe large performance degradation at high noise levels (up to 90% tokens masked), since most structural information is removed\n\n>R5: AminoAseed does not do well in sequence-based tasks like remote homology detection\n\nWe would like to clarify that the remote homology detection task in StructTokenBench (i.e., “Homo” task) **is not a sequence-based task, but a structure-based task**. Namely, we use backbone structure as input, and do not leverage the residue sequences. As explained in App. A.1.1, our “Homo” task is from the same source TAPE as the popular sequence-based remote homology detection task, which might be the reason for the confusion.\n\nAlso, **we appreciate the suggestion and would like to humbly point out this statement does not align with our observation**. As shown in Tab. 2, on “Homo” task, **our method AminoAseed performs the best, achieving 27.31% improvement over ESM3**\n\n>R6: Do we plan on integrating AminoAseed with sequence models to improve its performance on remote homology detection? If so, how would we approach the integration?\n\n1. **We explored the setting of adding sequence information directly to PST** (i.e., using structural representations + sequence token + positional encoding as input, and passing it to the probing layer). As shown in Tab. 5, adding sequence can mostly improve PSTs’ performance, as expected\n2. Further explore using sequence models like ESM2 on this specific homology task may shifts from our main focus, i.e., to benchmark structural representations induced from PSTs. **We would like to leave it for future work**\n3. **For the integration method**, one way is to combine the pretrained protein sequence representations from models like ESM2, with AminoAseed’s structural representations. Fusion attention layers can be added after the two models for better combination. Per-residue and per-protein contrastive learning can be used as training objectives. \n\n\n>R7: What’s the next step to scale up AminoAseed more effectively? Improvement to the architecture or optimization strategies could yield better performance with more compute?\n\n1. **We recommend data scaling**: gathering more diverse pretraining datasets to improve AminoAseed, since its pretraining data includes only around 10% of the PDB database, with 48k protein single chains.\n2. **Improving the architecture design** of VQVAE encoder might be useful, because the structure encoder is usually SE-(3) invariant and its capacity can be improved with better design. \n3. **Improving optimization** would be helpful, which shares the same motivation as AminoAseed to use codebook reparameterization to enable better codebook gradient update.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N.A.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "AminoAseed does not perform well on some tasks such as Binding Site Prediction as indicated in Figure 4.\n\nAlthough the paper demonstrates the benefits of scaling the model (e.g., increasing codebook size or encoder size), it also shows diminishing returns as the model scales. Large-scale models still encounter sub-exponential improvements, meaning that simply increasing compute resources or dataset size does not guarantee a proportional performance boost.\n\nThe Pareto-optimal codebook configuration and codebook reparameterization strategies, while innovative, add complexity to the model. The additional computational overhead could be a disadvantage in real-world applications where time and resources are constrained, and simpler methods may be preferred for quick prototyping or high-throughput applications.\n\nStructural tokenization methods, including AminoAseed, tend to lose reliability under high noise levels in the input protein data.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "It was noted that AminoAseed performed well in structural tasks but not as well in sequence-based tasks like remote homology detection. Do you plan on integrating AminoAseed with sequence models to improve its performance on such tasks? If so, how would you approach this integration?\n\nThe study suggests diminishing returns when scaling up the model. In your opinion, what would be the next steps for scaling up AminoAseed more effectively? Do you believe that improvements to the architecture or optimization strategies could yield better performance with additional compute?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g8Teu3EPSm", "reviewer": "Reviewer_NnWf", "summary": "This study considers the problem of protein structure tokenization, which it defines as the problem of distilling protein 3D structure into discrete or continuous representations. The authors introduce two main contributions: 1) a set of benchmark datasets for evaluating protein structure tokenization models and algorithms and 2) AminoAseed, a new method for protein structure tokenization that leverages codebook gradient updates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review and insightful comments! We respond to your concerns as below:\n\n> R1: Discrepancy observation that FoldSeek, which has the worst curve shape for the distinctiveness evaluation, has much better performance on efficiency benchmark. However, the distinctiveness and efficiency metrics should be related to each other. A more distinct codebook should lead to more efficiency utilization\n\nWe would like to humbly argue that **the two distinctiveness and efficiency are related, but it is not necessarily the case that as one increases, and the other increases**.\n1. Importantly, **the discrepancy stems from the different codebook size for different PST methods**. And the reviewer’s intuition would be more sound when the codebooks share the same size: a more distinct codebook should lead to more efficiency utilization. **For example**, utilization is measured by the factor of #used codes and #total codes (i.e., the codebook size). Smaller the codebook size, higher the chance for utilization rate to be higher. **FoldSeek has a very small codebook size K=20** (see Tab. 4), so it’s natural to have a high utilization rate, despite the observation that its used codes are not that distinct from each other.\n2. We would like to clarify **the concept of distinctiveness**. The codebook has two parts of codes: used and unused ones. For distinctiveness, **among the utilized codes in a codebook**, the more distinct they are, **the better chance for them to avoid ambiguous token-substructure mappings**.  We would like to emphasize the distinctiveness with used codes, and we show it in Fig. 3 (right panel). Because in reality, those used codes matter the most. \n\n**We’re open to hear more opinions on this topic from the reviewer**, and are always eager to learn more to enrich our knowledge.\n\n\n> R2: Question on the meaning to have distinctiveness evaluation, given its discrepancy to “codebook utilization”.\n\n1. The **“Distinctiveness” evaluation was motivated from the observation in ESM3’s** (Fig. S5) that ambiguous structural token mapping can hinder interpretability and harm model capability. **“Distinctiveness” measures the similarity within codebook vectors, which serves as a proxy to assess token ambiguity.**\n2. We understand the reviewer’s concern that simply using cosine similarity for codebook vectors might be limited. **We would like to continue working on a better metric for measurement instead of simple cosine similarity. **\n3. Please see **R1** for more discussion on the “discrepancy between distinctiveness metric and codebook utilization metric”. In general, we would like humbly argue that the discrepancy mainly stems from **the different codebook size used in different PST methods**, but not the metrics themselves. We’re always open to hear more opinions on this topic from the reviewer, as we’re also in the exploration stage to find a good way to benchmark PST methods.\n\n\n> R3: “ESM3 is the leading model” should be rephrased to “the leading VQVAE-based model”\n\nWe agree and would modify this in paper.\n\n> R4: In sensitivity evaluation, TM-score is not a golden standard for structure similarity, but a state-of-the-art structure search algorithm. Acknowledgement of this limitation is useful.\n\nThanks for the valuable suggestion. We would acknowledge this limitation in both main text and appendix that TM-score is not a golden standard to measure structure similarity. \n\nAlso, commonly used metrics to measure structure similarity include TM-score (more globally) and RMSD (more locally). In App. F.2, we explored using RMSD instead of the TM-score in “Sensitivity” evaluation. As shown in Tab. 10, using RMSD is less effective than TM-scores. This might be because RMSD is sensitive to outliers and local structural discrepancies, while TM-score focuss on global structure and is less sensitive to local structure variations.\n\n> R5: It’s unclear whether the dynamic programming in sensitivity used global or local alignment, which should be stated both in main text and appendix\n\nWe used the global alignment algorithm by adopting the Biotite’s “align_optimal()” function. Thanks for pointing out that this essential detail is missing. We would add this content to both the main text and the appendix.", "claims_and_evidence": "Most of the claims made throughout the paper are well supported by the evidence provided. The explanation about the datasets used is convicing and the results demonstrating that their new method outperforms the state-of-the-art is also convincing. The ablation study is both complete and convincing.\n\nThe paper benchmarks two distinct qualities Distinctiveness of the vectors in the codebooks and Efficiency of the codebook utilisation. Intuitively, these two values should be related to each other. A more distinct codebook should lead to more efficient utilisation. However, the empirical results show the opposite: Foldseek, which has the worst curve shape for the distinctiveness evaluation, has much better performance on the Efficiency benchmark. This discrepancy suggests to me that this is some problem in how either efficiency or distinctiveness are measured.\n\nI would argue that the claim in the abstract that ESM3 is \"the leading model\" is not supported by the evidence in the benchmark results provided where MIF is the leading model. Maybe this should be rephrased to \"the leading VQ-VA-based model\".", "methods_and_evaluation_criteria": "One of the main focus of the paper is to define the datasets and evaluation criteria to benchmark structure tokenization methods. I think that the choices made by the authors make sense and well reasoned.\n\nBenchmark construction\n---\n- Effectiveness datasets make emphasis on the detection of local substructures with functional relevance (binding site, catalytic, conserved site prediction), that greatly overlap with each other; and only one of global information (remote homology); however I generally agree with this choice as they are the closest tasks to real-world data.\n- Sensitivity, the general idea makes a lot of sense and it is quite reasonable. It assumes that TM-score is the gold standard, which is not, it is the state-of-the-art structure search algorithm and I think an acknowledgement on this limitation would be useful. On the other hand, the target value of representation similarity is computed via dynamic programming, but it is not clearly specified whether it is global (or optimal) alignment or local (suboptimal). It should be clearly stated both in the main text and the appendix.\n- Distinctiveness, I find this benchmark/metric the one with the weakest foundation, as it makes the assumption that cosine similarity is a good measurement of the distinctiveness of the vectors, which I do not think is properly justified. This might be the source of the discrepancy with efficiency\n-  Efficiency, I think this evaluation makes sense.\n\nNew tokenization method\n---\n- The study of optimal codebook size and dimension is quite informative, the methodology is simple and sound.\n- The finding of discretised representations obtaining similar performance to continuous ones is intriguing.\n- The finding that structural representations are less robust than sequence representations is interesting and shows that there is still room for improvement in the field on how to combine these types of representations", "theoretical_claims": "There are no significant theoretical claims. The study is mostly interested in practical application of well established concepts.", "experimental_designs_or_analyses": "Yes, I think that the experimental design is sound. I've already discuss some ambiguous experimental descriptions that I think should be clarified. Otherwise, I think the experimental design is sound.", "relation_to_broader_scientific_literature": "The work studies Protein Structure Tokenization, which is a problem first tackled through the lens of deep learning, by [1]. It is also related to the problem of inverse folding which has been tackled by [2, 3], which is also related to protein design. It is also related to multimodal integration of protein sequence and structure [5, 6]. The benchmarking of protein representation techniques is related to [9 - 11].\n\n\n1. van Kempen M, Kim SS, Tumescheit C, Mirdita M, Gilchrist CL, Söding J, Steinegger M. Foldseek: fast and accurate protein structure search. Biorxiv. 2022 Feb 9:2022-02.\n2. Hsu C, Verkuil R, Liu J, Lin Z, Hie B, Sercu T, Lerer A, Rives A. Learning inverse folding from millions of predicted structures. InInternational conference on machine learning 2022 Jun 28 (pp. 8946-8970). PMLR.\n3. Dauparas J, Anishchenko I, Bennett N, Bai H, Ragotte RJ, Milles LF, Wicky BI, Courbet A, de Haas RJ, Bethel N, Leung PJ. Robust deep learning–based protein sequence design using ProteinMPNN. Science. 2022 Oct 7;378(6615):49-56.\n4. Yang, K. K., Zanichelli, N., and Yeh, H. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36:gzad015, 2023.\n5. Heinzinger M, Weissenow K, Sanchez JG, Henkel A, Mirdita M, Steinegger M, Rost B. Bilingual language model for protein sequence and structure. bioRxiv. 2023 Jul 25:2023-07.\n6. Su J, Han C, Zhou Y, Shan J, Zhou X, Yuan F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv. 2023 Oct 2:2023-10.\n7. Rao R, Bhattacharya N, Thomas N, Duan Y, Chen P, Canny J, Abbeel P, Song Y. Evaluating protein transfer learning with TAPE. Advances in neural information processing systems. 2019;32.\n8. Capel H, Weiler R, Dijkstra M, Vleugels R, Bloem P, Feenstra KA. ProteinGLUE multi-task benchmark suite for self-supervised protein modeling. Scientific Reports. 2022 Sep 26;12(1):16047.\n9. Kucera T, Oliver C, Chen D, Borgwardt K. ProteinShake: building datasets and benchmarks for deep learning on protein structures. Advances in Neural Information Processing Systems. 2024 Feb 13;36.\n10. Zhang C, Zhang X, Freddolino PL, Zhang Y. BioLiP2: an updated structure database for biologically relevant ligand–protein interactions. Nucleic Acids Research. 2024 Jan 5;52(D1):D404-12.\n11. Unsal S, Atas H, Albayrak M, Turhan K, Acar AC, Doğan T. Learning functional properties of proteins with language models. Nature Machine Intelligence. 2022 Mar;4(3):227-45.", "essential_references_not_discussed": "None that I've been able to identify.", "other_strengths_and_weaknesses": "None, the main strengths and weaknesses have already been discussed in the prior sections", "other_comments_or_suggestions": "I would like to congratulate the authors on a really interesting and solid study. I would update my recommendation to strong accept if my concerns are satisfactorily address. Particularly, the issue of the discrepancy between distinctiveness and efficiency.", "questions_for_authors": "- Do you have any thoughts/insights into the relationship of distinctiveness and efficiency? In my mind, they are related to each other, but I would like to know what your thoughts are on the topic and whether my intuition might be wrong.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to benchmark different protein tokenization methods. To this end, the authors introduce **StructTokenBench**, a comprehensive framework for evaluating existing tokenization approaches. Additionally, they propose **AminoAseed**, a simple yet effective strategy that improves codebook gradient updates while optimally balancing codebook size and dimensionality, resulting in better tokenizer utilization and higher quality.

The paper is well-executed, and all reviewers voted for acceptance following the rebuttal.
Decision: Accept (poster)