AttentionSmithy: A Modular Framework for Rapid Transformer Development
Authors: Caleb Cranney
Jesse G Meyer
Venue: Accepted by TMLR ()

Abstract: Transformer architectures have revolutionized a broad spectrum of AI applications by leveraging attention mechanisms for parallelized and long-range sequence processing. Despite their remarkable success, building and customizing transformers remains prohibitively complex for many domain experts who lack deep knowledge of low-level implementations. We introduce AttentionSmithy, a modular software package that lowers the barrier to transformer innovation by decomposing key components---attention modules, feed-forward networks, normalization layers, and positional encodings---into reusable building blocks. By disentangling architectural elements into well-defined interfaces, users can rapidly prototype, adapt, and evaluate transformer variants without extensive coding overhead. Our framework currently supports four distinct positional encoding strategies (sinusoidal, learned, rotary, and ALiBi), offers modular integration of multiple attention methods (including standard attention, Longformer, and Linformer), and integrates seamlessly with neural architecture search (NAS) for automated design exploration. The system is designed to support future extensions with minimal overhead. We validate AttentionSmithy by replicating the original ``Attention Is All You Need'' transformer under resource constraints, demonstrating robust performance on a machine translation task. Leveraging the package’s integrated NAS capability, we identified an optimized model configuration that outperformed our baseline, demonstrating the framework’s effectiveness for automated architecture search and model improvement. We further illustrate AttentionSmithy's adaptability through gene-specific modeling, where a variant of a BERT-style architecture achieves over 95\% accuracy on downstream cell type classification tasks using ranked transcriptomic data. These case studies underscore AttentionSmithy's core advantage: enabling specialized experimentation across diverse application domains---from natural language processing to genomic analysis---by obviating the need for labor-intensive, low-level framework manipulation. We anticipate that AttentionSmithy will serve as a foundation for creative transformer-based solutions, expediting research and development in numerous scientific and industrial fields.

Published in Transactions on Machine Learning Research (06/2025)
AtmentionSmithy: A Modular Framework for Rapid Trans- former Development
Caleb Cranney Cedars Sinai Medical Center
haw.lccannne..ppppccc com
Department of Computational Biomedicine
Jesse G. Meyer
Cessegmneyer@gmaal com
Cedars Sinai Medical Center
Department of Computational Biomedicine
Reviewed on OpenReview:
Abstract
Transformer architectures have revolutionized broad spectrum of AT applications by lever- aging attention mechanisms for parallelized and long-range sequence processing. Despite their remarkable success, building and customizing transformers remains prohibitively com- plex for many domain experts who lack deep knowledge of low-level implementations. We introduce AttentionSmithy, a modular software package that lowers the barrier to trans- former innovation by decomposing key components attention modules, feed-forward net- works, normalization layers, and positional encodings into reusable building blocks. By disentangling architectural elements into well-defined interfaces, users can rapidly prototype, adapt, and evaluate transformer variants without extensive coding overhead. Our framework currently supports four distinet positional encoding strategies (sinusoidal, learned, rotary, and ALiBi), offers modular integration of multiple attention methods (including standard attention, Longformer, and Linformer), and integrates seamlessly with neural architecture search (NAS) for automated design exploration. The system designed to support future extensions with minimal overhead. We validate AttentionSmithy by replicating the original "Attention is All You Need" transformer under resource constraints, demonstrating robust performance machine translation task. Leveraging the package's integrated NAS pability, we identified optimized model configuration that outperformed our baseline, demonstrating the framework's 's effectiveness for automated architecture search and model improvement. We further illustrate AttentionSmithy's adaptability through geno-specific modeling, where a variant a BERT-style architecture achieves over 95% accuracy on reamstream cell type classification tasks using ranked transcriptomic data. These case studies underscore AttentionSmithy's core advantage: enabling specialized experimentation across diverse application domains from natural language processing to genomic analysis- by obviating the need for labor-intensive low-level framework manipulation. We anticipate that AttentionSmithy will serve foundation for creative transformer-based solutions, expediting research and development in numerous scientific and industrial fields.
1 Introduction
The Iransformer architecture (Vaswani et al. 2023 has revolutionized artificial intelligence, fundamen- tally changing how we approach sequence processing tasks across diverse domains. As transformer based models continue drive technological advancement and reshape socictal interactions (Haque & Li 2024). there is growing interest adapting these architectures for specialized applications. However, customizing transformer architectures remains significant challenge, requiring deep expertise in both the theoretical

---

Published in Transactions on Machine Learning Research (06/2025)
foundations and implementation details. This complexity creates barrier for domain experts who could otherwise leverage transformer capabilities for novel applications.
1.1 Transformer Architecture Fundamentals
While traditional recurrent neural networks like Long Short-Term Memory (LSTM) networks (Hochreiter & Schmidhuber 1997) excelled at processing sequential data, they faced inherent limitations in parallelization and capturing long-range dependencies. The transformer architecture introduced by Vaswani et al. (2023)
overcame these constraints through its innovative attention mechanism, enabling unprecedented advances in natural language processing (Patwardhan et al. )023), computer vision (Pereira & Hussain 2024), healthcare
applications [Nerella et al. 2024), molecular science research (Jiang et al. 2024), and genomic analysis (Choi
& Lee 2023
The basic building blocks of transformer include [Figure 1A]:
1. Multi-head attention layers that compute and weigh relationships between all sequence elements in parallel
2. Feed-forward neural networks that process these relationships through non- non-linear transformations
Layer normalization components that stabilize training by normalizing activations across features
Residual connections that facilitate gradient flow and help preserve and reuse features from earlier layers
5. Positional encoding mechanisms that preserve sequence order information by encoding relative or absolute positions
1.2 Positional Encoding Strategies
A crucial aspect of transformer architectures is their handling of sequential information through positional encodings. Without such encodings, transformers would treat inpul sequences as unordered sets of tokens, losing critical information about both absolute positions (where exactly token appears in the sequence) and relative positions (how tokens ordered with respect to each other). For instance, the sentences 'the dog chased the cat" and "the cat chased the dog' contain identical tokens but convey opposite meanings, while "chased cat dog the the" is syntactically invalid distinctions transformer could not make without position information. While the self-attention mechanism excels at capturing relationships between tokens, it is inherently permutation-invarinnt necessitating an explicit method to encode positional context. Several strategies have emerged, each with unique implementation requirements:
Sinusoidal positional encodings (Vaswani al. 2023) and learned positional embeddings (Wang & Chen 2020) operate by adding position-specific vectors directly to input token embeddings. This straightforward approach allows for easy implementation but may have limitations in capturing relative positions elfectively.
Rotary positional embeddings (Su et al. 2023 take a different approach, modifying the attention compu- tation itself by applying rotation transformations to the query and key matrices. This method has shown particular promise in capturing relative positioning information while maintaining consistent attention pat- terns across sequence lengths.
ALiBi positional encodings (Press et al. 2022 introduce position-specifie bias terms to the attention matrix, effectively modulating the attention weights based relative positions. This approach has demon- strated advantages in extrapolating to longer sequences than those seen during training.
1.3 Scalable Attention Strategies
Transformer models have new benchmarks across wide range of tasks, but their core attention mech- anism scales poorly with sequence length, requiring quadratic time and memory resources with respect to
2

---

Published in Transactions on Machine Learning Research (06/2025)
A
B
C
D
Loss Calculstion
Feed Forward
Sublayer Unit
Standard Encodar
Add Norm
Sinusoidel
Perceiver Encoder
Position
Feed
Multi-Head
Forward
Attention
Toxion
Embailding
[Future Extensions)
Multi-Head Attention
Encoder
Norm
Multi-Head
Encoder Layer
Rotary Position
Feed Forward
Standard Attention
xN
Query
ALiBi Position
Longlormer Attention
Addd & Norm
Masked Mult-Head Attention
Multi-Had Attention
Decoder
Linformer Attention
Decoder Layer
[Future Extensions]
xN
Curput
Embeeding
Figure 1: Components of the transformer model architecture as coded in AttentionSmithhy (A) The original transformer architecture introduced by Vaswani et al. (2023). (B) Labelled AttentionSmithy classes include implementations of the feed forward network, multi-head attention, sublayer unit (consisting of layer normalization and a residual connection surrounding exchangeable feed forward network or multi- head attention class), encoder /decodrr layers, and full encoders/decoders (which consist of an "N" number of layers determined by the end user). (C) Explicitly encoding position is a requirement of transformer models, but how position encoded varies dramatically implementation requirements from stralegy stratogy. Four common strategics and their implementation details are outlined. The sinusoidal and learned positional embedding methods (red) involve directly adding vectors representing absolute positions to token embeddings before entering encoder or decoder layers. The rotary positional embedding method (green) requires adjusting the query and key matrices of the attention calculation directly. The ALiBi position embedding method (blue) adds negative values to the output of the query and key matrix multiplication that accumulate over greater positional distances. (D) The multi-head atlention module accepts interchangeable attention methods, enabling support for strategies such as standard, Longformer, and Linformer attention. Additionally, base component classes can be swapped at the architectural level-for example, replacing a standard encoder with Perceiver-style encoder. The modular structure also supports future extensions with minimal changes to surrounding components.
input size. This limitation restricts their applicability to long-context scenarios such as document processing, audio modeling, and genomic sequences. To enable practical use in these domains, various alternativ ives have been developed that reduce the computational cost attention while extending its contextual capacity.
Some strategies adjust the attention calculation directly to avoid quadratic complexity. Examples include Longformer, which replaces dense attention with a combination of local windows and global attention tokens [Beltagy et al.] 2020). Other strategics include Linformer, which introduces low-rank projections of the key and value matrices, effectively approximating attention computation with a reduced-rank representation (Wang et al. 2020). These methods reduce the number of token-to-token comparisons required, transforming atlention from an all-to-all operation into sparse compressed one, which substantially lowers the cost from quadratic to linear scale with respect to sequence length.
Other strategics are architoctural in nature, introducing intermediate representations that reduce the inten- sity of scaling with input length. One such example is Perceiver, which uses a small set of learnable latent vectors that attend to the imput tokens (Jaegle et al. 2021]. This replaces full self-attention over the input with a cross-attention mechanism, where computational cost scales with the product of the input length and the number of latent vectors vectors-resulting in significantly lower complexity when the number of latents much smaller than the input size.
3

---

Published in Transactions on Machine Learning Research (06/2025)
1.4 Architectural Experimentation and Search
The modular nature of transformer architectures presents significant opportunities for systematic architec- tural exploration. Neural architecture search (NAS) has emerged as a promising approach for discovering optimal neural network configurations, but its application to transformers remains limited. While special- ized NAS frameworks have been developed for transformers SChitty-Venkkan they are typically purpose-bullt for specific rescarch objectives, making it difficult for practitioners to adapt them for novel transformer architectures and unique application domains. Traditional implementations of- ten tightly couple architectural elements, making it challenging to define comprehensive search space that includes variations in attention mechanisms, positional encodings, and feod-forward networks.
et al.) 2022 Liu al. 2022),
1.5 Current Tools and Limitations
While widely used librarics like Hugging Face Transformers (Wolf et al. 2020). PyTorch (Ansel et al. 2024 and TensorFlow (Abadi et al. 2015) offer robust. implementations of standard transformer architectures, they provide only limited flexibility for deep architectural customization. Core components are often tightly embedded within specific models, making difficult to experiment with alternative encoding strategies, novel attention mechanisms, or entirely new architectural arrangements. This tight coupling not only hinders manual experimentation but also complicates the implementation of automated architecture search strategies, often forcing researchers to modify large sections of internal code or start anew.
Several specialized tools have emerged to improve transformer customization, each with valuable strengths. Modular Transformers (HuggingFack 2025) stands out. for its integration into the Hugging Face ecosystem, allowing users to easily swap components within pre-defined model templates. This streamlines workflows, particularly when one aims to modify or extend existing popular models. However, its focus on compo- nent swapping within fixed templates makes it challenging to restructure models at a deeper architectural level-such as building Perceiver (Jacgle et al. 2021 -without significant additional work.
X-transformers Wang 2020) stands out an impressive collection of experimental transformer features drawn from outting-edge research, accessible through flexible parameters; however, because its core relies on dense conditional logic rather than modular building blocks, it becomes difficult to implement fundamental structural innovations such as incorporating nonstandard patterns like Longformer attention (Beltagy el al. 2020) without extensive code modification.
nanoGPT Karpathy 2022 takes different approach, focusing simplicity and clarity for those interested specifically in GPT models. Its compact PyTorch implementation has made it very popular for introductory experimentation, fine-tuning tasks, and tailoring to specific project needs. However, by design, it is special- ized toward autoregressive GPT-style architectures, making it unsuitable for general-purpose transformer experimentation or architectures requiring encoder components or cross-modal innovations.
In this landscape, we present AttentionSmithh, a novel software package designed democratize trans- former development by providing a modular, component-base framework inspired by established software design principles (Ousterhout, 2018 Vogel et al. 2011 Gamma et al.) 1995). Unlike existing tools that excel at customization within fixed boundaries, AttentionSmithy breaks transformers into flexible, reusable components, cnabling rapid prototyping, systematic architecture search, and the creation of fundamen- tally new designs. Its modular architecture allows researchers to interchange and experiment with a wide range of features including positional encoding strategies, attention mechanisms, and architectural config- urations-tailories models their specific ncods. By filling a crucial niche for maximal fiexibility while maintaining architectural clarity, AttentionSmithy offers rescarchers and engineers a powerful toolkit for exploring and innovating beyond the limits of standard transformer design.
4

---

Published in Transactions on Machine Learning Research (06/2025)
2
Methods
2.1
Software Architecture
Our software package implements component-based design philosophy to facilitate the creation of cus- tomized transformer architectures. The core architecture breaks down transformer components into modular, reusable units that can be easily assembled and modified. This approach enables researchers to experiment with architectural variations while maintaining code readability and understanding of the undorlying mech- anisms.
The implementation utilizes PyTorch as its foundation and comprises distinct classes for each major trans- former component. These components include multi-head attention mechanisms, feed-forward networks, normalization and dropout layers (implemented logether as "sublayer unit"), encoder /deoddr layers, and complete encoder /dcoodrr structures [Figure 1B). Additionally, we provide both greedy and beam scarch generators for sequence generation tasks.
Appendix materials include a description for an example training script performing a machine translation task [Appendix A.1], as well as short code snippets illustrating how to build and customize transformer components using AttentionSmithy [Appendix A.2].
2.2
Key Features
2.2.1
Flexible Positional Encoding Framework
A key architectural feature is the implementation of positional embedding strategy pattern that manages various numeric embedding approaches. A strategy manager serves intermediary for selecting and applying different positional encoding implementations within the transformer architecture, allowing for seamless integration of different approaches without requiring modifications to the core architecture.
Our implementation currently supports four distinct positional encoding strategies: simusoidal, learned, ro- tary, and ALiBi embeddings, chosen as representative examples of popular approaches in the field. Each strategy is implemented a separate class and managed through the embedding strategy manager, with the architecture designed to readily accommodate additional encoding strategies as they emerge. This flexibility is particularly valuable because different positional encoding strategies require fundamentally different im- plementations within the transformer architecture: sinusoidal and learned positional embeddings are added to input token vectors, rotary positional embedding requires adjusting the query and key matrices in the attention calculation, and ALiBi adds values to the attention score matrix (the product of the query and key matrices) [Figure 1C). Our extensible design allows users to activate or deactivate these varied encoding strategies independently, enabling direct comparisons of their effectiveness various applications, while also providing a framework for implementing and testing novel positional encoding approaches.
Beyond traditional position representation, the modular implementation of positional encodings in Atten- tionSmithy enables these methods to be applied to any numeric data type. This opens new possibilities for representing temporal, quantitative, or other ordered information within transformer architectures. For in- stance, in time-series analysis, researchers could simultaneously encode both sequential position and temporal features using different encoding strategies.
2.2.2 Modular Attention Mechanisms
The system implements modular multi-head attention framework that supports interchangeable attention variants, specified at initialization. This design accommodates range of scalable attention mechanisms without modifying the core architocture. The framework currently includes implementations of sparse at- tention (e.g., Longformer), low-rank approximation (e.g., Linformer). and archilectural adaptations (e.g.. Perceiver) [Figure 1D). This modular design enables rapid prototyping and integration of new attention strategics with minimal architectural disruption. Notably, while current implementations of Longformer, Linformer, and Perceiver attention are applied only within encoder architectures, the modular design readily supports adapting to decoder and encoder-lecoder settings for strategies appropriate for those architectures.
5

---

Published in Transactions on Machine Learning Research (06/2025)
2.2.3 Neural Architecture Search Compatibility
The modular design of AttentionSmithy facilitates automated architecture optimization through neural ar- chitecture search (NAS). Components can be easily swapped or modified programmatically, allowing for systematic exploration of architectural variations while maintaining code interpretability.
The NAS workflow was based the "Mueli-Obeeclive NAS with Ax" workflow tutorial the official PyTorch website, utilizing Meta's Ax package to do so [Eriksson & Balandat 2022: Meta 2025 The process includes designing search space as separate Python script that accepts variables dictating the model structure, setting up a TorchX runner and scheduler for submitting model training scripts concurrently, and defining optimization requirement configurations. Ax uses Bayesian optimization to evaluate and compare model configurations and their predictive accuracy, highlighting the impact specific architectural decisions have on the final loss.
For the machine translation task, used the BLEU score (Papineni et al. 2002 (reported scale of 0 100) as the primary evaluation metric, consistent with the original transtormer paper (Vaswani et al, 2023 The search space consisted of six adjustable parameters: each of the four impleeenntd positional encoding methods could be toggled on or off, the dropout rate, and the activation function used in the foed-forward network. Models were trained for five epochs during the search to reduce time complexity.
To ratenstrate how domain experts can apply NAS to specialized applications using AttentionSmithy, we extended the NAS workflow to the Geneformer task. The search space included three positional encod- ing strategies (sinusoidal, learned, and rotary), dropout rate, activation function, and attention mechanism. Additionally, nonstandard attention types included task-specific hyperparameters: Longformer (context win- dow length), Linformer (projected key dimension), and Perceiver (number of latent encoder layers and latent space length). Each trial was run for six thousand steps with a batch size of 32.
2.3 Code Availability
The source code for AttentionSmithhy is publicly available on GitHub The code implementing machine trans- and utilizes
lation is also available hnttps://gihub..co//oommm the WMT14 Berman--Engiish dataset (Bojar et al. 2014) accessed through the Hugging Face datasets library. The code implementing geneformer (Theodoris et al. 2023 is available httpps//giithub. com htwps//saatascicccc//oooo/o http:///ooooooooctsssssiiiiiieie ggw.oorrrrreeeeerrrerroooe utilizing preprocessed data from the original gene- former implementation HuggingPace repository. All code repositories are released under the MIT license. The software originated from a re-implementation of code depicted in the Annotated Transformer article Rush et al. 2022).
peachine-translatte hww.:///.o.oooooomcctcceceeeeee
AtlentionSmithy is implemeente in Python using PyTorc (Ansel et al. 2024). To enhance usability and standardization, AttentionSmithy designed to be compatible with Py Torch Lightning (Falcon) 2019). allowing researchers to easily incorporate training loops, distributed training, and other advanced features while maintaining clean, research-focused code.
2.4
LLM assistance
Claude 3.5 Sonnet (Anthropic 2024 and ChatGPT-44 OpenAT 2024) were used to help with writing this manuscript.
3
Results
3.1
Validation Studies
We conducted three validation studies to demonstrate the efficacy and versatility of AttentionSmithy: a replication of the original vanilla transformer model, an optimized model determined by a neural architecture search (NAS), and bioinformatics application.

---

Published in Transactions on Machine Learning Research (06/2025)
A
25.0
BLEU Score
22.5
NAS optimized
17.5
all positions
15.0
rotary
12.5
ALi Bi
leamned
10.0
sinusoldal
0.0
0.5
1.0
1.5
2.0
2.5
Global Steps (1e6)
B
Translation Comparison
Cross Entropy Loss
10.0
Original
German:
MI Bitcoi ist Bezahlen und Verkaulen ohne Bank möglich
9.5
Referenc::
English
with bitsoin, pay and sell without banks
9.0
bitsoin, payment possible without
Transtation:
8.5
Tranelation:
Optimized
with bitcoin,
and sell without bank
100k
200k
300k
400k
D
Cross Entropy Loss m
Global Steps
1.0
Prediction Accuracy
9.0
0.8
8.5
0.6
8.0
0.4
0.2
7.0
0.0
6.5
100k
200k
300k
2
6
10
12
14
Global Steps
NAS Trial Number
Figure 2: Validation studies of programs built with AttentionSmithy. (A) BLEU scores comparing translation quality across six transformer variants. The initial NAS-optimized model used tanh activation function, a dropout rate of 0.0, and combination of all positional encoding strategies (blue). For compari- son, we evaluated models using the original transformer's settings ReLU activation and 0.1 dropout- with witth
different positional encodings: rotary (green), ALiBi (red), learned (purple), sinusoidal (brown), and the combined set of all four (orange). Note that the sinusoidal configuration corresponds to the original trans- former model (Vaswani et al. 2023). (B) Representative examples of German-to- to-ngggst translation outputs, showing source text, reference translation, and outputs from both NAS optimized and original transformer variants. The BLEU score in this specific example improved from 16.1 with the vanilla transformer to 34.4 with the optimized archilecture. (C) Validation loss trajectory during pretraining of the Geneformer foun- dation model, plotted against global training steps. (D) Cell type classification accuracy on the validation set during fine-tuning of the pretrained Geneformer model, also plotted against global training steps. (E) Validation losses from 14 unique neural architecture search (NAS) trials targeting optimization of the Gene- former model. These trials represent subset of 50 total runs, with duplicates removed. Trials 0-9, shown the left of the red line, were pseudo-random configurations, while remaining trials were targeted explorations guided by NAS.
3.1.1 Original Transformer Replication
We implemented the transformer architecture and training setup described Vaswani al. (2023) using the WMT 2014 Banlish-German dataset, which consists of 4.51M sentence pairs (approximately 9.03M sentences

---

Published in Transactions on Machine Learning Research (06/2025)
total). Our primary training run was limited to a maximum context window of 100 tokens, mainly to reduce training time given the use of a single A100 GPU. While this constraint was partly driven by computational efficiency, it was also informed by the nature of the dataset: only 50,860 sentence pairs (approximately 1.1%) included at least one sentence longer than 100 tokens and were excluded from training. To confirm that this truncation had minimal effect on overall performance, we conducted an additional run using a 500-token context window (sce Appendix A.3). This expanded setting excluded only 398 sentence pairs (less than 0.01%) with at least one sentence exceeding 500 tokens, and was comparable to the same model with a 100-context window constraint. After 40 epochs of training, our 100-token model achieved BLEU score of approximately 21 on the machine translation task (Figure 2A, brown line). While this falls short of the original paper's BLEU score of 25, it represents a reasonable outcome given the resource constraints.
To highlight the ease of NAS enabled with AttentionSmithyy, we designed a search space around the original model components to identify architectural changes that may enhance performance. The optimized model from NAS had a more rapidly increasing BLEU score across training steps, and in the end achieved higher BLEU score of approximately 24 after 40 epochs [Figure 2A, blue line], approaching the performance of the original implementation despite our hardware constraints. Key modifications identified from NAS included: simultancous utilization of all four positional encoding strategies (simusoidal, learned, rotary, and ALiBi), removal of dropout (reduction from 0.1 to 0.0), and replacement of ReLU with tanh activation in feed-forward networks. This led to generally better translations ions, an example of which is shown in Figure 2B.
An initially intriguing outcome from our neural architecture search was the apparent success of configurations using all four available positional encoding methods simultaneously. However, follow-up ablation experiments clarified that this combination did not yield additive synergistic benefits; rather, the improvements were likely driven primarily by the use of rotary positional encoding, which performed approximately as effectively on its own as the combined setup [Fiuure 2A]. This underscores the importance of validating NAS-derived architectures through targeted ablation experiments to identify the true sources of performance gains.
3.1.2 Bioinformatics Application
To demonstrate domain adaptability, we replicated the Geneformer model for transfer learning for tran- scriptomic single-cell data tasks (Theodoris al.) 2023). Following the original paper's methodology, we pre-trained the model using BERT-style architecture on rank-based transcript data, with genes serving as tokens. Importantly, to ensure the model had sufficient capacity to learn long-range dependencies, we used a maximum context window of 2048 tokens-matching the configuration in the original Geneformer implementation. To verify that the model captures contextual information even after plateauing [Figure 2C), we finc-tuned it for coll type classification using this datasct, freezing the first two pre-trained layers and adding a classification layer as specified in their methodology. We used their published human dem hem uf dataset for this task, which contains 579.159 cells representing 21 distinct cell types from cardiac tissue from 29 individuals. This implementation achieved over 95% accuracy on the validation dataset, demonstrating successful replication of the Geneformer architecture's ability to transfer contextual relationship information for downstream gene expression analyses [Figure 2D).
As a proof of concept, we additionally applied NAS to the Geneformer task to explore potential improve- ments. The NAS identified a configuration combining Longformer attention with a local attention window of 32, no positional encoding, zero dropout, and leaky ReL.U activation, which achieved a validation loss of 6.60 [Figure 2EJ. Specialized linear projections for global attention in Longformer can be enabled in AttentionSmithh, though this option was not included in the Geneformer NAS task, and thus not included in the optimal model. While we did not pretrain fine-tune this optimized architecture in the current study, the reduction in validation loss relative to the baseline model (which platcaned at 8.0-8.5, as seen in Figure 2C) suggests promising avenues for future exploration.
4 Discussion
While our validation studies focused on established architectures, they serve primarily to demonstrate Atten- tionSmithy's foundational reliability. The package's true value lies in enabling researchers to develop entirely
8

---

Published in Transactions on Machine Learning Research (06/2025)
new transformer architectures for specialized applications that may not yet exist. By providing fiexible, modular framework, we empower domain experts to experiment with novel combinations of transformer components without requiring deep expertise in transformer implementation details.
This capability is particularly valuable in scientific domains where traditional transformer architectures may not perfectly fit. the underlying data structures or research questions. For instance, researchers working with complex multimodal data could leverage our framework to develop hybrid architectures that process different data types through specialized attention mechanisms. The ability to experiment with multiple positional encoding strategies simultaneously opens new possibilities for representing complex relationships in data, whether they be spatial, temporal, or domain-specific ordered relationships.
The modular nature of AttentionSmithy enables researchers to focus on the unique aspects of their applica- tion domains rather than becoming entangled in transformer implementation details. This democratization of transformer development has the potential to accelerate innovation in fields where artificial intelligence applications are still emerging. For example, researchers could apply self-supervised speech representation techniques [Mohamed et all 2022) to nanopore sequencing, enabling efficient and accurate mucleotide se- quencing through pre-training and fine-tuning approaches. In mass spectrometry, developing foundation models to interpret data-independent acquisition (DIA) spectra could allow researchers to leverage these complex, chimeric signals for downstream tasks without relying on pre-existing spectral libraries. The trans- formative potential of this architecture extends well beyond current applications, and we anticipate that researchers across diverse scientific domains will develop innovative implementations that we cannot yet foresee.
Future development of AttentionSmithy will focus on expanding its capabilities to support emerging trans- former variants while maintaining its commitment to architectural clarity and ease of use. We encourage contributions from the research community, particularly in implementing new positional encoding strategies and exploring applications in specialized domains. This could include relative positional embeddings (Shaw et al. 2018) and their T5 variant (Raffel et al.] 2023), which focus explicitly on the relationships between positions her than absolute positions. Through continued development and collaboration, we aim to fur- ther lower the barriers entry for transformer architecture experimentation and innovation across scientific disciplines.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Cor- rado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozelowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Tlya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www. www.:w/ensorfforr. org/ Software available from tensorflow.org,
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Des- maison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshitecj Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Mar- cos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunt- ing Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. Tn 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24). ACM, Apr 2024. doi: 10. /111444/326656... /3666666666666666666 URL https::/pytorchh org/ assets aasetsspytoreet wttp:s/yytorchh22.p222
Anthropic. Claude 3.5, 2024. URL https://clane ai/
9

---

Published in Transactions on Machine Learning Research (06/2025)
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. (arXiv:2004.55150) Dec 2020. doi: 200.6000/XXiv.204..000 URL. http:///axxiv..o/abb//////s/
05150 arXiv:2004.05150 [es].
Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tam- chyna. Findings of the 2014 workshop statistical machine translation. In Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Spe- cia (eds.), Proceedings of the Ninth Workshop on Statistical Machine Translation, 12 58, Baltimore, Maryland, USA, Jun 2014. Association for Computational Linguistics. doi: 10.3115/11W144330 URL httpss://aclanthol.... www.tchoooooyoooo14-----309
Krishna Teja Venkka,, Murali Emani, Venkatram Vishwanath, and Arun K. Somani. Neural archi- tecture search for transformers: A survey. IEEE Access, 10:108374-108412, 2022. ISSN 2169-3536. doi:
10.1109/ACCEB 2022.3212767.
Sanghyuk Roy Choi and Minhyeok Lee. Transformer architecture and attention mechanisms in genome data analysis: A comprchensive review. Biology, 12(77):1033, Jul 2023. ISSN 2079-7737. doi: 10.3390/
12000gy
David Eriksson and Max Balandat. Multti-ojecctvve nas with ax pytorch tutorials 2.3.0-cu121 documen- tation, August 2022. htt.l//////pccccoooo//ooo/a///ia http://dddaeaatammmmaoooooooeoeo
tutorial html
William Falcon. Porchrc Lightning, March 2019. URL http:s///gtthh...coo/////tatttas
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design patterns: elements of reusable object-oriented software. Addison-Weste Weelly Longman Publishing Co., Inc., USA, 1995. ISBN 0-201-63361-2.
Md. Asraful Haque and Shuai Li. Exploring chatgpt and its impact on sociely. AI and Ethics, Feb 2024. ISSN 2730-5961. doi: 00.0007/443881-0044-04333- URL
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral Comput., 98)):1335-178)) Nov 1997. ISSN 0899-7667. doi: wtt.://nooo..99.9...
HuggingFace. Modular transformers, 2025.
URL httpe///wgggngaaee.co//////rrrrr
modular modular_transforre transformers.
Andrew Jacgle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Per- ceiver: General perception with iterative attention. (arXIv:2103.03200), June 2021. doi: 10.48550/arXiv. 2103.03206. URL http://aaraiii.rra/a///1./.. 03200 arXiv:2103.0222 [cs].
Jian Jiang, Lu Ke, Long Chen, Bozheng Dou, Yueying Zhu, Jie Liu, Bengong Zhang, Tianshou Zhou, and Guo- Wei Wei. Transformer technology in molecular science. WIREs Computational Molecular Science, 14 (4):c1725, 2024. ISSN 1759-0884. doi: 10.1002 20.10022/wcms1122 wttp/wmms.12..
Andrej Karpathy. nanoGPT, 2022. URL https: httpp////gethbbb.caaaaaaatattta/
Zexiang Liu, Dong Li, Kaiyue Lu, Zhen Qin, Weixnan Sun, Jiacheng Xu, and Yiran Zhong. Neu- ral architecture search on efficient transformers and beyond. (arXiv:2207. 13955).. Jul 2022. doi: 10.48550/ar Xww.XXii..22231111 URL http://t////rr wrwp.////./a//..////////// 13955 arXiv:22077139955 [cs].
Meta, Feb 2025. URL httt::://gituub..co//faaaeaao
Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havlorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang- Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, and Shinji Watanabe. Self-supervised speech representation lcarning: review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179-1211 Oct 2022. ISSN 1932-4553, 1941-0484. doi: 10.1109/ISTSPP222222200 10X6 [cs).
10

---

Published in Transactions on Machine Learning Research (06/2025)
Subhash Nerella, Sabyasachi Bandyopadhyay, Jiaqing Zhang, Miguel Contreras, Scott Siegel, Aysegul Bumin, Brandon Silva, Jessica Sena, Benjamin Shickel, Azra Bihorac, Kia Khezeli, and Parisa Rashidi. Trans- formers in healthcare: A survey, Artificial Intelligence in Medicine, 154:102900, Aug 2024. ISSN 09333657. doi: ao.1016/j.artmed.222 2ee.a.0222.122 102900. (iXii:2307.00066 [cs].
OpenAL. Chatgpt-4o, 2024. URL
John Ousterhout. A Philosophy of Software Design. 1st edition, 2018. ISBN 1-73210-220-11
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Lönguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, Jul 2002. Association for Computational Linguistics. doi: 10.3115 111115/0073083100777771 1073135. URL https:
//aclartholoogyoorr/PP-------
Narendra Patwardhan, Stefano Marrone, and Carlo Sansone. Transformers in the real world: A survey on nlp applications. Information, 14(44):242, Apr 2023. ISSN 2078-2489. doi: 10.3390/info4404000
Gracile Astlin Pereira and Muhammad Hussain. A review of transformer- models for computer vision tasks: Capturing global context and spatial relationships. (arXiv:2408.1511 Aug 2024. doi: 10.48550/ arXiv.2408.15177 URL htttp://arxxi.orr/abbbaee
15178 arXiv:240S.15178 [cs].
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. (arXiv:2108.12400) Apr 2022. doi: 10.48550/arXxi::108..2222 URL http:
20400 12409 arXiv:2108. 1wX1:22108.12409 12409.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. (arXiv:1910. 100603))110100633) Sep 2023. doi: 109850/arXii.1110.1000 arXiv:1910. 10683 [cs].
URL http://arxxw.org/abb///S
10683
Sasha Rush, Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, and Stella Biderman. The annotated transformer, 2022. URL httpp::/nlp.seea: hww.ardd hww.waawww.aa/aaaaaaaaootttataaa
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations.
MiXXiv18033021555 Apr 2018. doi: 10.48350/arXiv.1100..0 02155 arXiv:1803.02155 [cs].
URL http:///arxxw.orr/bbb/////
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. (arXIv:2104.00664) Nov 2023. doi: 10.48550/arXivv21004 09864. URL /ttp://arxiv org/abs/2104..ao..
arXiv:2104.09864
Christina V. Theodoris, Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R. Al Sayed, Matthew C. Hill, Helene Mantineo, Elizabeth M. Brydon, Zexian Zeng, X. Shirley Liu, and Patrick T. Ellinor. Transfer learning enables predictions in network biology. Nature, 618(7965):616 624, Jun 2023. ISSN 1476-4687. doi:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasæ Kaiser, and Illia Polosukhin. Attention is all you need. (arXiv:1706.03762)) Aug 2023. doi: 10.48550/ /ar//aaXXvvvI/ 03762. URL http://arxxw.or//bbb//// wtwp://wwwr//bb/////0///
10.48550/arXiv.111
03762 arXiv:1706.03766.
Oliver Vogel, Ingo Arnold, Arif Chughtai, and Timo Kehrer. Software Architectur:: A Comprehensive Framework and Guide for Practitioners, Springer Publishing Company, Incorporated, 2011. ISBN 3-642- 19735-3.
Phil Wang, x-transformers, 2020. URL http::///ggthooooomm///aaaaaaaaa
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with lincar complexity. (arXiv:2006.04768) June 2020. doi: a0.48550//rx11.2000.0007 URL httt:///axxvv.rrg abs/2006. 04768 ar XXXi:20006004763 [os].
11

---

Published in Transactions on Machine Learning Research (06/2025)
Yu-An Wang and Yun-Nung Chen. What do position embeddings learn? an empirical study of pre-trained language model positional encoding. (arXiv:2010.099033 Oct 2020. doi: 10.465550/7xx1.22000.0000 URL wrg/ooo///////o/oooo ass//b1//000..... 04903 arXiv:2010.04903 [cs].
http://arxiv.on www.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language process- ing. (arXiv:1910.037)) Jul 2020. doi: (..48550/arXivy 03771 arXiv:1910.03771 [cs].
1910.0377.. 4455550///X1.111000037777 URL httt:://aaatt org 19g//bb//190.
A Appendix
A.1
Supplementary Code
The included supplementary .zip file (pachhe-trranslttion zip) contains example code demonstrating the implementation and execution of machine translation transformer model using AttentionSmithy. The included scripts facilitate data downloading, model training, and evaluation.
A.1.1
Contents
py Downloads and preprocesses the WMT-14 German-English dataset.
data_imports py
Handles dataset loading and procossing for training.
main.p2 py
Runs the model training and evaluation pipeline.
model_ import.ppp.... py
Defines the translation model and its components.
README md
Provides setup instructions, expected outputs, and additional notes.
A.2 Short Code Examples
The AttentionSmithy package provides modular components for constructing custom transformer architec- tures. It supports encoders, decoders (not shown in this sample), and full encoder-decoder stacks, with swappable attention mechanisms and experimental mumeric embedding strategies.
A.2.1 Assembling a Transformer Encoder
We start by building an encoder from modular pieces:
hultihheadAttentionn (attention mechanism)
FeedForwarddMetwork (nonlinear transformation)
EncoderLayer (single transformer block)
Encoder (stacked layers)
While this sample focuses on encoders, decoder layers are fully supported and can be constructed analogously.
Listing 1: Basic Encoder Assembly
from attention, smittnsssmihhyy components import Encoder, EncoderLayer, MultiheadAttentionn FeedForvardNetsoork. from attention, smithy attention import StandardAttattionneehhddd
embedding_din 512
12

---

Published in Transactions on Machine Learning Research (06/2025)
heads feedforward _dim = 2048
dropout 0.1
Define standard multihead attention attention - KultsheadAttention
11
number of bta..aa_aadd-nnaaaaaaaa htteepawwttttsssaaaaaaaeeetttttr
13
15
# Define feedforward block
17 feedforward FoedForvardNetwozt
embedding.dia feedforward activation 'relu', dropout
19
20
21 # Wrap in encoder layer
22 encoder_layer EncoderLayer( (
embedding embedding.dimennte www.:ssiioo-mmmmddngn.e.eoeooo self attottttttnt,, fedd_ orward, dropout=dropout
24
26
28
Stack into multi -ayer encoder
30
encoder Encoder cencoder_layer, number_od of _layers=))
A.2.2 Using Alternative Attention Mechanisms
AttentionSmithy supports experimental attention patterns, such as Longformer's local-global windowing Linformer's projection-based sparsity. These can be swapped scamlessly with standard attention via the attention_metha method argument of the Multthhedkttention class.
Listing 2: Using Longformer Attention
from attention_amist attention import LongfornerAtteatiioneettad
Use Longformer-style local+global attention iingormer_attentee MultiheadAttentiinn
number atrartoc..cebetttooooommeaaaaaan
attention_ 1itentiion_vindo=2255
dropout=dropout )
11
A.2.3 Adding Positional and Numeric Embeddings
The NumericEm humerricgbbedddnggaaaggg allows combining multiple numeric embedding strategies, applied at differ- ent pipeline stages:
at the token embedding step (e.g., Sinusoidal, Learned)
before attention score calculation (c.g., Rotary)
after attention calculation (e.g., ALiBi)
13

---

Published in Transactions on Machine Learning Research (06/2025)
Developers easily implement positional embedding strategies and provide them at numeric embed- ding manager initialization, making it straight to experiment with alternatives like sinusoidal, learned, rotary, or ALiBi embeddings. Additionally, the design supports strategies for encoding non-positional nu- meric features, such as time intervals, distances, or other custon scalar inputs, further increasing the model's flexibility and representational capacity.
Listing 3: Numeric Embedding Manager
from Aitentiio__mittyy atterriio..aatttt..ueeeeeeeeaaad numericc_embedinng import
ShuuesridlPostttiintaaaaddddgdn LearredPositionnBbdddddi
AotaryPoosttionnmbeediiggg ALDiiPositionnmmbddinng HuserricEnbbedddiaaaaaee
# Initialize multiple strategies
embedding.managen Iueericeabbediinglaaggggr ([
SineoicalPotttotccccccaaaaaacoe dim),
10
n
// num sum_heads),
ALBiiPositionnEbeedddng (num haw.m__eadssnunnmmm _eadss)
13 })
15 # Generate combined sinusoidal and learned positional embedding
token, embedding some comettnput__ sabedding_tenser
17 vositioona_eemeddiii token token_ eecee_enmbediigge token token_ embedding
= mawaddgeggmmaage.... create_positional. or wttpaiooaaoorruuuttttoee wrw.ocsttt__emeeediiee.
19
20
a # Add positional embedding to token embedding
z final_embedee @adee__mbbediigg
iraitioonl_eebeddingg
14

---

Published in Transactions on Machine Learning Research (06/2025)
A.3
Supplementary Figure 1
25.0
22.5
BLEU Score
20.0
17.5
15.0
12.5
NAS optimized, context window 500
NAS optimized, context window 100
10.0
original transformer, context window 100
0.0
0.5
1.0
1.5
2.0
2.5
Global Steps (1e6)
Supplementary Figure 1: BLEU score evaluation for differing context window lengths. NAS- optimized transformer with 500-token context window (blue), NAS-optimized transformer with 100-token context. window (orange), and the original transformer baseline with 100-token context window (green). The NAS-optimized models substantially outperform the bascline, with minimal difference between the 500- and 100-token context window settings.
15


Reviews:
[{"review_id": "a6KhxGKXrZ", "reviewer": "Reviewer_teX3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* *Modularity:* The framework's modular design allows for easy experimentation and rapid prototyping, which is well-aligned with the current needs for agile development in AI research.\n* *Neural Architecture Search Integration:* The integration of NAS makes it easier for users to automatically find optimal architectures for their domain.\n* *Insights on positional encodings:* One of the experiments highlights the complementary benefits from using a combination of encoding methods.\n\n**Weaknesses:**\n\nMy main concerns relate to the paper’s claims:\n* *Demonstrating near state-of-the-art performance on a machine translation task.* This claim is too strong. The results seem to be competitive with the original Transformer paper from 2017, with possible signs of improvement when considering the compute constraints. But the current stat-of-the-art on the WMT German-English seems to be around a BLEU score of 35. This claim needs adjusting.\n* *Attention Smithy’s core advantage: enabling specialized experimentation across diverse application domains—from natural language processing to genomic analysis—by obviating the need for labor-intensive, low-level framework manipulation.* In order for the framework to be truly useful for domain experts (i.e. non-AI-experts), it needs to offer more implemented components to search across. Especially with a name like AttentionSmithy., the most important next step is to offer more attention methods, as is discussed and shown in Supplementary Figure 1.\n\nTo recommend acceptance, these two main concerns need to be addressed. Below are a few minor concerns that could further improve the paper.\n\nMinor concerns:\n* I think the Geneformer experiment could be more meaningful. I assume the point here is to show that the framework works for non-standard domains. Ideally this experiment would also include a search, to show the benefits of this type of modular framework across different domains.\n* Since the current paper is under the page limit, there would be space to add some short code example of either the positional encoding implementations or more importantly, a few lines of code showing how the framework is used.\n* While it is somewhat orthogonal to the main contribution of the paper, it would be great to understand more about the complementary benefits from the different encoding mechanisms from the NAS results. Does the package provide any useful information for interpreting these types of results?\n* What do the colours mean in the different attention methods in Supp Fig 1? The caption could give some info on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces AttentionSmithy, a modular framework for developing and customising transformer architectures, especially focusing on being accessible for domain experts without in-depth AI knowledge. It allows users to experiment with different parts of the transformer model, like attention modules, feed-forward networks, and positional encodings through a collection components that can be combined and configured. It is also designed to integrate well with neural architecture search (NAS) to enable automatic optimisation of transformer architectures. The current version supports various positional encoding strategies, and the framework is validated through experiments in machine translation and gene-specific modelling, demonstrating its effectiveness and adaptability across different domains.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical changes:**\n\nRevise the claim about achieving near state-of-the-art performance in machine translation, and expand the variety of attention mechanisms to fulfil the framework's promise as outlined in the detailed review.\n\n**Other:**\n\nSee above section on minor concerns.", "broader_impact_concerns": "None needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qevuXGSvLP", "reviewer": "Reviewer_5n9H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper presents a framework that arguably reduces the technical barriers to exploring Transformer architecture variants.\n- Empirical evidence of potential applications is provided in both natural language and biological domains.\n\n### Weaknesses\n\n- The comparison with existing frameworks and tools is not sufficiently explored. Other tools are available for experimenting with transformers, and a more detailed comparison of their limitations could make the value proposition of this framework more evident. Refer to \"Requested Changes\" for more details.\n\n- The experiments are limited to a very specific setting using short sequences, which weakens the claim that combining multiple positional encoding methods helps in practical problems.\n\n- Besides the positional encoding variants, the framework still has a limited set of components for other important parts of the architecture, such as the attention mechanism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes AttentionSmithy, a modular framework designed to simplify and accelerate the customization of Transformer architectures. It breaks down key components such as attention mechanisms, feed-forward layers, and positional encoding into reusable building blocks. Its main contributions include a flexible system for experimenting with positional encodings (sinusoidal, learned, rotary, and ALiBi) and compatibility with neural architecture search (NAS) to optimize model configurations. Experiments confirmed its effectiveness by successfully replicating the original Transformer model and demonstrating improved performance through automated design exploration. Additionally, AttentionSmithy extended its applicability beyond NLP, achieving over 95% accuracy in genomic cell-type classification, showcasing its adaptability across scientific domains.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Detail the advantages over current frameworks: In Section 1.4, the paper briefly mentions some limitations and states that “implementing customized architectures remains a complex undertaking that often requires building from scratch.” However, there are several existing options for easily building and testing novel transformer variants, namely:\n\n  - Modular Transformers [1]: allows reusing components from existing models.\n\n  - x-transformers [2]: provides several transformer components, including attention mechanisms and positional encoding options, which can be easily toggled.\n\n  - nanoGPT [3]: a simple transformer implementation that is easy to modify and use in small-scale experiments.\n\nWhat specific advantages does AttentionSmithy offer over the options above? Addressing this question could help improve the paper.\n\n- If possible, provide experiments using longer sequences: There are small-scale experiments based on FineWeb [4] that could help demonstrate the flexibility of AttentionSmithy and validate the claims regarding the effectiveness of combining multiple positional encoding methods.\n\n- Minor:\n\n  - Provide links in the bibliography or footnotes for the “Multi-Objective NAS with Ax” workflow tutorial (page 4).\n\n  - Provide links in the bibliography or footnotes for Claude 3.5 Sonnet and ChatGPT o1 (page 5).\n\n  - Section 1.4, page 2: the correct framework name is Hugging Face Transformers.\n\n- [1] https://huggingface.co/docs/transformers/en/modular_transformers\n- [2] https://github.com/lucidrains/x-transformers\n- [3] https://github.com/karpathy/nanoGPT\n- [4] https://github.com/KellerJordan/modded-nanogpt", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vx4ec6ZcP3", "reviewer": "Reviewer_L4jA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow.\n2. The idea of this work is indeed intriguing.\n3. The framework is evaluated on two different application domains, and the customized model found by NAS algorithm observes decent improvement compare to the baseline run.\n\nWeaknesses:\n1. No NAS result shown for the Bioinformatics application, currently it looks like the author just reproduce the original paper with their own framework. Is further improvement possible after NAS algorithm? Or simply adding the other encoding strategies can cause performance gain?\n2. The results from NAS algorithm on machine translation task shows large improvement. However, there are multiple modifications identified from NAS, which does not support the conclusion made in section 4.1. An ablation on these individual modifications will help a a lot.\n3. Given that the point of this package is to accelerate the development process, a comparison of effort required to build some custom transformer model using Huggingface for example and the proposed framework will be great (e.g. How many lines of code required?).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents AttentionSmithy, a modular software package that ease the pre-training of custom transformer-like models by packing transformer modules into building blocks. This package also incorporate a small search space containing different positional encoding methods, dropout rate and activation functions, where users can use built in NAS algorithm to gain potential performance boost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the points above to strengthen the claims in section 4.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After a round of corrections by the authors, the reviewers believe the paper should be accepted, and that it will be of interest to the TMLR community. Given the scope and materials in the paper, the authors are expected to release a link to a repository that is accessible and easy to use.
Decision: Accept (poster)