Renamer: A Transformer Architecture In-variant to Variable Renaming
Authors: Zachary Ankner
Alex Renda
Michael Carbin
Venue: Submitted to ICLR 2023 (2023)

Abstract: Modeling tasks often take inputs from languages including programming languages and natural language. Many such tasks involve learning functions which are invariant to certain types of input transformations. In this work we consider a specific class of invariance: semantics-preserving variable renaming. We first show that transformer networks trained on such tasks do not always mirror the invariance of the underlying function. In this work we propose Renamer, a transformer architecture which is invariant to semantics-preserving variable renaming. Renamer improves over a vanilla transformer by between a 24.79% to 52.80% reduction in error on a case study on learning a surrogate of a large-scale CPU simualtor. Furthermore, the invariant network does not experience the same sensitivity to variable renaming, and its error remains constant when evaluated on a variable renamed version of the test set. Finally, the invariant network is more efficient to train, and matches the best error of the vanilla network with a between 25.15% to 60.00% reduction in training epochs.

Under review as conference paper at ICLR 2023
RENAMER: A TRANSFORMER ARCHITECTURE IN- VARIANT TO VARIABLE RENAMING
A Anonymous authors Paper under double-bling review
ABSTRACT
Many modeling tasks involve learning functions which are invariant to certain types of input transformations. In this work we consider a specific class of invariance: scmantics presterviggeeeiinggggg variable renaming. We first show that Trans- former networks trained on such tasks do not always mirror the invariance of the underlying function. In this work we propose Renamer, a Transformer variable renaming.
architecture which is invariant to ceerraniics-ppeeeviii Learning a neural surrogate of large-scale CPU simulator using Renamer re- duces simulation error by 24. 79-52.8% compared to using a vanilla Transformer. Furthermore, the invariant network is not sensitive variable renaming, and its output remains constant when evaluated on variable renamed version of the test set. Finally, Renamer more efficient to train: it matches the perfor- mance of the vanilla Transformer with 24.75-59.06%% fewer training epochs.
1 INTRODUCTION
Modeling tasks often require reasoning about invariances of the task with respect to transformations on the inputs (Snavely, 2019; Bianchi et al., 2022). A common approach to learning models that are invariant to given transformation is to train models using data augmentations that exhibit the transformation (and corresponding invariance) under study (Shorten and Khoshgoftaar, 2019; Feng et al., 2021). However, such approaches give no formal guarantees that the resulting models are always perfectly invariant to the transformation. Further, there is evidence that baking the inductive bias of the invariance into the model leads to accuracy improvements (LeCun and Bengio, 1995; Cohen and Welling, 2016; Lee et al., 2019; Keriven and Peyré, 2019; Wang et al., 2022).
Renaming invariance. We study renaming invariance, a particular type of invariance in sequence processing tasks which arises when reasoning about formal languages including programming languages (Chen et al., 2021; Alon et al., 2019; Renda et al., 2020), mathe- matics (Lample and Charton, 2020; Polu et al., 2022), and synthetic grammars of natural languages (Marzoev et al., 2020: Berant and Liang, 2014). Renaming invariance is invariance to bijective transformations of the input tokens that preserve the semantics of the input. An example of renaming invariance is in symbolic algebra programs, where variables be bijectively renamed and the result of evaluating the expression doesn't change.
Renaming sensitivity. General-purpose neural network architectures like LSTMs (Hochre- iter and Schmidhuber, 1997) and Transformors (Vaswani et al., 2017) have shown impressive results on learning functions with renaming invariance (Alon et al., 2019; Renda et al., 2021). However, these neural net works do not themselves demonstrate renaming invariance. For example, Alon et. al. (2019) note sensitivity to "uninformative, obfuscated, or adversarial variable names"' This sensitivity presents challenge to deploying neural networks in this context, as their predictions are not robust to semantics- semantics-preserving input transformations.
Our approach. We present an approach to enforcing renaming invariance in Transformers. The first key contribution that enables our approach is formal definition of renaming invariance.
We define renaming invariance as a property of functions which take sequences of tokens as input. We first define view mapping as mapping from input token to its view, the

---

Under review as conference paper at ICLR 2023
semantic information about the token that is salient the function. We then define referent mopping as a mapping from an imput token to its referent, the underlying entity to which it refers. A renaming invariant function is function which generates the same output for any bijection of tokens that does not change tokens' views and bijectively renames tokens' referents that is, salient semantic properties of tokens must not change, pairs of tokens which originally referred to the same underlying entity must both refer to the same permuted underlying entity, and pairs of tokens which originally referred to different underlying entities must still refer to different permuted underlying entities.
We present two architecture changes that together enforce renaming invariance in Transformers. We refer to the resulting architecture as the Renamer.
View anonymization. The first change, view anonymization, effectively replaces each token with token that describes only its view. This enforces that the network is renaming invariant, because the network camnot make different predictions for different views. However, view anonymization alone reduces the representad attiona capacily of the network, since the nelwork can no longer distinguish lokens with different referents.
Referent binding. To recover the representational capacity, introduce novel modification to the attention layer which we call the referent binding. Referent binding restricts the attention in the first layer of the Transformer, allowing tokens to only attend to other tokens with the same referent. This breaks the symmetry between tokens with the same view but different referents, restoring the representative capacity of Renamer, while maintaining that Renamer is renaming invariant.
Contributions.
We present the following contributions:
We introduce and formally characterize the renaming invariance problem.
We propose the two-step process of view anonymization and referent binding to enforce renaming invariance while maintaining representational power. We implement these in the Renamer, a renaming invariant Transformer model architecture.
We evaluate the Renamer on a renaming invariant x86 assembly processing task. Renamer reduces the error compared to vanilla model by between 27. 27.58% and 52.80%.
By identifying and defining renaming invariance and proposing a Transformer model invariant to renaming invariance, our work takes a key step towards the goal of providing high-accuracy models with provable guarantees for tasks with input invariances.
RENAMING INVARIANCE IN LIVM-MCA
This scction presents case study of renaming invariance in sequence processing task. We first introduce x86 basic block throughpuu prediction and describe how is renaming-invariant task. We describe the views and referents present in x86 basic blocks for this task. We then describe renaming invariant permutations for this task, and show that the task's labels are invariant these permutations, but are not invariant other permutations. We finally demonstrate that Renamer generates accurate and renaming invariant predictions for this task, while baseline models are not renaming invariant and are therefore less accurate.
mov 64 (%rsp), Xraz sub $1, 56 (%rbp) mov 16 (((rx)) Yeax
mov 64 (%r8), Xraz
64 (Krax), Taa sub $1, 56 (%ebp) mov 16(Xraz) Yea
sub $1, 56 ( (%rbp)
mov
(6(rraz) Yaa
Ilym-mca: 1.68 cycles
Ilym-mca: 1.68 cycles
Ilym-mca: 10.03 cycles (c) Non-invariant renaming.
(a) Original block.
(b) Invariant renaming.
Example of an 686-64 basic block and invariant and non invariant renaming. The registers may renamed, as long as each register is renamed consistently.
Figure 1:
2

---

Under review as conference paper at ICLR 2023
Task under study. Following Renda et al. (2021) our task is to create neural network surrogate of llvm-mca, a CPU simulator inchuded in the LLVM compiler infrastructure (Lattner and Adve, 2004). As imput Ilvm-mma takes basic block of x86-64 assembly language which is a sequence of assembly instructions with no jumps or loops. Tt then outputs a prediction of the throughput of the basic block on the simulated CPU which is a prediction of the number of CPU clock cycles taken to execute the block when repeated for a fixed number of iterations. Learning a surrogate of Ilvm-mca results in faster or more accurate throughput estimation than using Ilym-mca itself (Renda et al., 2021).
Input specification. We evaluate using a dataset of AT&T-syutax x86-64 basic blocks (Chen et al., 2019). Figure 1 presents three such basic blocks. AT&T syntax basic blocks are sequences of instructions, where each instruction consists of an opcode (e.g,, mov), source operand (c.g.. 64 (7rsp)), and destination operand (c.g., /rax). Each operand may be a constant (c.g., $1), a register (c.g., /rax), or a memory address (c.g., 64 (/rsp)).
Registers are the variables of x86-64 assembly. A given register operand consists of bitwidth and a base register. The bitwidth is how many bits of the register data are addressed by the register. As an example, /rax addresses all 64 bits of the register data, %eax addresses the lowest 32 bits, and Şax addresses the lowest 16 bits. The base register is the location where register data is stored; for general-purpose registers this is indicated by the final several characters of the register (c.g., ax in /rax).
Simulation model. When executing an instruction, Ilym-mca first. waits for all of its source operands to be ready: source operand is ready when all previous instructions that have matching destination operands (all predecessors in the register dependency graph) have finished exccuting. Once an instruction starts exccuting, its exccution time is controlled by the bitwidth of cach operand.
Renaming invariance in Ilym-mca. Renaming invariance manifests in llvm-mea as invari- ance to register renaming. When the base register names are renamed in a given block such that none of the register bitwidths, nor the register dependency graph change, llvm-mca generates an identical prodiction for this block, mcaning that it is invariant to this class of transformation.
To formalize this, we say that the view of given register is its bitwidth, and that the referent of a given register is the base register that it refers to. Any transformation of registers that maintains register views and is a bijection on register referents is a renaming invariant translormation.
Example. Figure 1 presents three x86 basic blocks in AT&T syntax. Figure 1(a) shows the original block. This basic block has a throughput of 1.6 cycles per iteration in llym-mea's model. There are four unique registers in the basic block: Vrsp, Yean, Kraz, , and %rbp.
The registers Krsp, Araz, , and %rbp all share the same view they are all 64 bit registers. Xear is a 32 bit register, and thus does not share the 64 bit width view but instead has its own view.
The registers Zeaw and graz have the same referent they both refer to the Zar location in memory. The registers Krsp and %rbp have unique distinct referents they refer to the %sp and %bp memory locations which no other registers in the block refer to.
Figure 1(b) shows semantically equivalent version of the block as all registers are renamed in bijective manner that preserves which tokens share the same referent and that also preserves the view of each token. To generate this new block, Krsp was renamed to /r8 and all other registers remained lixed. Since Vr8 is also 64 bit register, this mapping preserves the view of each register. Since Kr8 occupies the /8 location in memory, which is distinct from all other registers, Kr8 has its own unique referent. Thus Krsp, which had unique referent in the original block, is mapped to a new register which also has a unique referent in the renamed block, all registers which shared a unique referent still share unique referent. Because the permutation preserves the semantics of the original block, Ilym-mca outputs the same timing for the renamed block as the original block.
Figure (c) shows a version of the block with registers renamed in manner that is not semantically equivalent, as it both breaks which registers share the same referent. and does not preserve
3

---

Under review as conference paper at ICLR 2023
the view of each register. First, views not preserved: %rbp,, 64 bit register, is renamed to %ebp, 32 bit register, thus changing the execution time of the instruction. Second, which tokens share the same referent is not preserved: Krsp is renamed to Zrax on the first line while Kraw and Neaz registers which originally referred to different locations in memory and thus had unique referents mapped to registers which do share the same location in memory and now share the same referent. Because the permutation does not preserve the semantics of the original block, Ilvm-mva outputs different timing for the renamed block compared to the original block.
not renamed. This creates new dependency in the renamed block as
Renaming invariance in Transformers. Renda et al. a Transformer (Vaswani al., 2017) as the surrogate architecture for Ilym-mca. This Transformer model takes the basic block as imput, and outputs a prediction of what Ilvm-mca would output on that basic block.
The base Transformer architecture is sensitive to register renaming. For example, a BERT model trained as surrogate of Ilvm-mca generates prediction of 1.5 cycles on the block in Figure 1(a). This same model generates a prediction of 2.3 cycles on the block in Figure 1(b), demonstrating that it is not invariant the register renamiigg transformation. In contrast, the Renamer model that we present in Section 4 generates a. prediction for 1.6 cycles for both blocks in Figures (a) and ((b). We provide a more depth analysis in Appendix A.1.
3 RENAMING INVARIANCE
In order to describe an architecture which is invariant to renaming, we need a formal definition of functions which are invariant to renaming. In this section, formally define what it means for a function to be renaming invariant.
Formalism.
Let I X be an input token from the set of input tokens and let [æ,] ex" be length sequence of tokens.
Let f:X" be a function over a sequence of tokens.
Let R be set of referents. Referents represent the underlying set of entities that can be renamed. Each token is associated with a single referent by the referent mapping r:X- + +RR.
Let V bea of views. A view represents the semantic information about token that is relevant to the function output. Each token is associated with a view by the view mapping: v: v:X-> X-V
Let +X be permutation (i..., a bijection) over the input tokens. is referent--omstrrinec there exists a permutation over referents ó : R R such that Vz E Xxr(a(x)) = =(r(z)) This means that all tokens which originally shared the same unique referent are permuted such that they still all share a unique referent. This can also be stated as all tokens which originally referred to the same entity still all share the same entity after being permuted. is view- constrained if Vz E X. X.v(r)= =<(o(x)), meaning that all tokens are mapped to a token with the same semantic role in the function.
Furthermore, a is semannicc-preservvin it is both referent-constraiing and view-constrained. o" X - X" an element-wise extension of a to sequences such that all tokens in the sequence which have the same identity are mapped to the same value. Then a sequence processing function f is renaming invariant for a given set of referents and views if for all semantics- preeeriig- permutations o", [[[x]))=ft f(o ((x)))))
RENAMER ARCHITECTURE
Renamer enforces the renaming invariance by modifying the first layer of the Transformer: Technically, for any semantics- perrticc-pperrviiggg renaming, the output Renamer must be identical for the original and renamed input. Renamer must not only be renaming invariant, but must also preserve the representational capacity of a vanilla Transformer. The remainder of this section gives background Transformer models, then describes implementation.
4

---

Under review as conference paper at ICLR 2023
4.1 TRANSFORMERS BACKGROUND
Renamer modifies the embeddings and self-attention mechanisms of Transformer networks. A Transformer lakes sequence of Lokens [xx]] as input. Given [xi], the Transformer computes a content embedding cl*.] E RRdd token-wise as cja C([aly)) where C R is an embedding lable. The identity of a loken in [ar.] is exposed to the Transformer through its content embedding. A positional embedding Pla.l E RRndd also computed token-wise as p]al==(() where P: N + R& is an arbitrary function. The only restriction on P in this work is that P assigns a unique embedding for each index in the sequence. The embedding for a sequence [s;] F. C(a) + p(a:) where + denotes element-wise addition. Self-attention defined softmax(M((Q))))
QERaXdati,, K E "xdddattn and V E guxd are distinct linear projections of the embedding El M:Raun + R is an atlention mask operator computed from [ay]] that sets the unnormalized allention score loken pairs which should not atlend to each other to -o such that they do not contribute to the normalized attention scores.
4.2 RENAMER ARCHITECTURE MODIFICATIONS
View anonymization. So that all input tokens which share the same view share the same rep- resentation, the same content embedding is shared between all tokens that share the same view:
V E X X. = -(x)) -C(,
Then Vay E (()l:(())))(((())))))) C ((a.)) Thus (([=)))]((=>((() after view anonymization the Renamer is renaming invariant.
meaning that
Concretely, we implement view anonymization by mapping cach token in the input sequence to a token that represents its view.
Referent binding. View anonymization alone limits the class of functions the Transformer can represent. All elements in the same view share the same content embedding, so the network can't differentiate which tokens in the input do or do not have the same referent under the referent mapping.
So that all renaming invariant functions representable by a vanilla Transformer are still learnable by Renamer, the network must be able to differentiate inputs based on which tokens share the same referent and which do not. Tokens with different referents are disambiguated by first applying attention only bet ween tokens with the same referent. Given an input sequence [ay] and a referent mapping r, we construct a referent attention mask operator M as follows:
{ SS =((x)))=r((x11))) 100, otherwise
The representation after referent binding is then: (oftma((M(((KT)) referent attention mask operator is defined such that attention is only computed between tokens which share the same referent. Applying the referent attention mask gives the network the capacity to represent functions which depend on the relations between tokens with the same referent. While the same conlent embedding is shared for all tokens in the same view, by assumption the positional embedding for each token is unique. Thus, in only attending to other registers in the same referent group, the representations for each token with the same referent can be combination of unique embeddings.
In words, the
EVALUATION
We evaluate Renamer by learning surrogate of llym- mea, comparing against suite of baselines.
5.1
TASK
The task under study is to take an x86-64 basic block as input, and output a prediction of the timing that llym-mca would output for this basic block.
5

---

Under review as conference paper at ICLR 2023
Register renaming invariance in Ilym-mca. We define the set of referents to be the set of abstract locations where register data is stored. Thus the set of referents for x86-64 64 is:
nE (8, 16) ( (XMMn, YMMM n E (0)
The referent for given register is the abstract location where that register data is stored: for example, referent mapping r has r r(%rax)=AX, (eeeeA,,((()) r(Keax)=AX, rrA.,,rr%% r(%r12w) All other lokens have unique singleton referents.
The views are defined as the set of possible bit-widths: (8-bit, 16-bit, 32-bit, 64-bit, 128-bit, 256-bit), combined with the set of register classes (general-purpose, floating-point, vector). For example, view mapping v has (66rax)=(64-bit, peneral-purpose), 0(%eex))(32-bii general-purpose), (17xxmm0)=(128-1
vector). All other tokens have unique singleton views.
With the exception of instructions with implicil operands, those instructions which read from or write to specific hard-coded registers, llvm-mea is renaming invariant with these referents and views. Instructions with implicit operands break renaming invariance because permutation cannot change implicit registers. Such instructions include push, pop, mul, div, and others.
Datasct. We evaluate the Renamer on the BHive dataset (Chen et al., 2019), which a collec- vion of x86-64 basic blocks from variely of real-world programs. The full BHive dataset consists of 287639 basic blocks. We remove inputs which have implicit operands from this dalaset. We then perform random 70/10 10/11//3 20 split. on the original dataset, resulting in dalasel with 185,773 blocks in the training set, 26,107 blocks in the validation set, and 52,278 blocks in the test set.
5.2 MODELS
For all experiments, we use BERT Transformer (Devlin al., 2019) as the backbone of our architecture. We evaluate on BERT-Tiny, BERT-Mini, and BERT-Small (Ture et al., 2019).
Vanilla Transformer. The vanilla Transformer, which serves as a backbone for all other models, is an encoder-only BERT (Devlin et al., 2019), with absolute positional encodings.
Augmented Transformer. In addition to the vanilla Transformer, we validate against an augmented training baseline. The architecture for the augmented baseline is identical to vanilla. BERT, the only difference that for each batch during training a semantics -eeaanttcc-preeevinn register permutation is applied to the registers of the basic blocks. The specifications of such permutations are provided in Section 2. While the resulting model is not guaranteed to be invariant to variable renaming, this training paradigm removes any bias towards specific registers in the dataset.
Canonicalized Transformer. We also evaluate Transformer model which cononicalizes basic blocks before using them as input to the vanilla Transformer. Canonicalization takes each basic block and maps to unique canonical basic block; canonicalization maps all prestrvvigg transformation of basic block to the same canonical block. This preprocessing scheme ensures that the Transformer is invariant to variable renaming.
Renamer. Renamer models have the same architecture as the vanilla model, except for the first encoder block which employs view masking and referent binding as described Section 4. View masking and referent binding do not change the number of parameters of the Transformer.
5.3 EVALUATION METHODOLOGY
System. We evaluate using Pytorch- .2.0 (Paszke al., 2019), HuggingFace 4.17.0 (Wolf et al., 2020). Training is performed using an NVIDIA Tesla- Each reported metric is the mean and the standard error of that metric across five trials with different random seeds.
Hapee-parameters. Across all models, use the AdamW optimizer with a B1, 32 of 0.9 and 0.999 respectively. The Tiny and Mini models use a learning-rate 0.0003 while the small models learning rate of 0.0001. All models weight-decay of 0.01 and dropout

---

Under review as conference paper at ICLR 2023
Table 1: MAPE on original test set
Model size
Model
BERT-Tiny
BERT-Mini
BERT-Small
Vanilla
3.30% +0.16%
1.13% +-017%
1.25% +0.43%
Augmented
3.34% =0.11%
2.25% +0.02%
2.36% +0.02%
Canonicalized
3.03% +0.33%
0.96% +0.06%
0.76% +0.04%
Renamer
2.39% +0.07%
0.85% =0.07%
0.59% =0.06%
rale O. We empirically determine the learning-rale, weight-decay, and dropout through grid search over the hyper-parameters, selecting the hyper-parameler configuration which has the lowest validation error for the vanilla model. The hyper- parameters swept over are: learning-res maatingg-rateeaee {0.0003, 0.0001, 0.00005, 0.00001), weight-decay {0.0, 0.01), and dropout {0.0, 0.13. All models have a batch size of 64, max sequence length of 128, and are trained for 500 epochs following Renda ct al. (2021).
Objective. Following Chen et al. (2019) the loss and error metric are identical and are defined as the mean absolute percentage error (MAPE): LMAPE = - EaweD [f(z)=)
5.4 RESULTS
Best performance. In this section we evaluate the best performance of the vanilla, augmented, canonicalized, and Renamer models. This is defined as the test error of the epoch with the lowest validation error.
Table 1 shows the error of cach model across BERT sizes. We find that across all model sizes, the Renamer model outperforms the vanilla, augmented, and canonicalized models on the original test set. Renamer has a relative decrease in error compared to the vanilla model of 27.58% 24.79%, and 52.80% for the Tiny, Mini, and Small BERT variants respectively Additionally, the Renamer has decrease in error compared to the augmented model of 28. 44,, 62.22%, and 75.00% for the Tiny, Mini, and Small BERT variants respectively. Finally, Renamer has a relative decrease in 21.12%, 11.46%, and 23.07% for the Tiny, Mini, and Small BERT variants respectively.
compared to the canonicalized model of
Additionally, find that as model size is increased, Renamer suffers less from diminishing returns as compared to the vanilla and augmented models. For the vanilla model, the relative decrease in error between the Tiny and Mini model is 66.76%, while error increases by 10.62% between the Mini and Small model. Likewise for the augmented model, the error decreases by 32.63% for Tiny to Mini and increases by 4.89% for Mini to Small. In contrast, the Renamer decreases in error between both size variants by 64.44% and 30.5 30.59%. While both the vanilla and augmented models have relative increase in error between BERT-Mini and BERT-Small, Renamer has a significant decrease in error.
Renamed registers. this scction we evaluate the crror of the best performing vanilla, augmented, canonicalized, and Renamer models on cemantics-preseringg register renamed version the test set.
For each basic block in the original lest set, random semantics-prest preserving permut ation is applied to the registers the basic block. The checkpoint of the model with the lowest error on the umperturbed validation set is selected and then evaluated on the renamed test set.
Table 2 shows the error of each model across BERT sizes on the permuted version of the test set. The performance of the vanilla model is significantly affected by permuting the registers. Compared to the original, unperturbed test set, the error of the vanilla model increases by 59.39%, 144.25% and 131.20% for the Tiny, Mini, and Small BERT variants respectively. This increase in error further empirically demonstrates the renaming sensitivity of the vanilla. network architecture.
7

---

Under review as conference paper at ICLR 2023
Table 2: MAPE on test set with renamed registers
Model size
Model
BERT-Tiny
BERT-Mini
BERT-Small
Vanilla
5.26% +0.37%
2.76% +-0.42%
2.89% +0.52%
Augmented
3.44% =0.11%
2.55% +0.02%
2.36% +0.03%
Canonicalized
3.03% +0.33%
0.96% +0.06%
0.76% +0.04%
Renamer
2.39% +0.07%
0.85% =0.07%
0.59% =0.06%
-Tny (Standard Test Set)
Bini (Standard Test, Set)
BERT-S (Standard Test Set)
500
500
500
Vanilla
Vanilla
Vanilla
400
400
400
Augusented
Spocbs
Augmented
Augmented
Renamet
Renamee
200-
Canonicalzed
200
Cazonicolied
200
100
100
100
0
10%
35%
10%
3%
1%
100%
10%
1%
MAPE
MAPE
MAPE
Figure 2: Model efficiency the original (unperturbed) test set. Each point a curve corre- sponds to the number of epochs needed to reach a given test error for the specified architecture.
In contrast to the vanilla model, Renamer is provably invariant to register perturbations. Accordingly, the renamed test set, Renamer outperforms the vanilla model by 54.56%, 69.20%, and 79.58% for the Tiny, Mini, and Small BERT variants respectively.
While the augmented training model is also less sensitive to register permutations and the canonicalized model is also guaranteed to be invariant, Renamer still outperforms both the augmented training and canonicalized models on the renamed test set..
Training efficiency. In this section we evaluate the efficiency with which the vanilla, augmented, canonicalized, and Renamer models achieve various test accuracies. The time to process imput is the same across all models they all have the same number of parameters, and Renamer only leverages components of the Transformer architecture (i.e. embedding table, attention mask) which the vanilla, augmented, and canonicalized architectures also employ. Since the time to train for an epoch is the same for all architectures, we directly compare the number of epochs needed to achieve comparable test error.
Figure plots efficiency curves for the various model sizes on the BHive dataset. Each line shows the epochs required (on the y axis) to reach given test error (on the axis) for given modeling approach. A more efficient model requires fewer epochs to reach a given target test error (i.e., lower better).
Across all model sizes, Renamer achieves the same performance with significantly fewer training steps than the vanilla, augmented, and canonicalized models. This speedup is reflected in the gap between the MAPE versus epochs of training curves for the vanilla, augmented, canonicalized, and Renamer models.
We evaluate the speedup quantitatively by comparing the minimum number of epochs required to achieve the same best test error. Renamer reaches the best error of the vanilla model with 213, 123, and 290 fewer epochs for the Tiny, Mini, and Small architectures respectively, which corre- sponds to relative decrease in steps to achieve the same performance of 42.77, 24.75, and 9..0.. As compared to the augmented model, Renamer requires 214, 403, and 419 fewer epochs for the Tiny, Mini, and Small variants respectively, which corresponds to a relative decrease in steps to achieve the same performance of 42.89, 81.25, and 83.80% Finally, Renamer achieves the same error as the canonicalized model with 155 and 59 fewer epochs for the Tiny and Mini variants respectively, which corresponds to relative decrease in training steps of 31 31.00 and 11. 11.82%.
8

---

Under review as conference paper at ICLR 2023
6 DISCUSSION
In this section we analyze the results and limitations in more detail.
Source of improvement. We have demonstrated that the Renamer achieves lower error faster than both the vanilla network and the augmented baseline. We hypothesize that this is because the Renamer only represents the subset of functions that are renaming invariant. Because Ilym-mea itself is renaming invariant, this leads to a smaller search space for SGD, along with the guarantee that all solutions match Ilym-mca's renaming invariance.
Spurious correlation. Though we find that on the BHive datasct the invariant model improves training and reduces test the error, this may not happen on all datasets. Specifically, we hypothesize that datasets with spurious correlation between referents and labels even when the underlying function is renaming invariant. For instance, version of BHive that only used the /rax register when the label is less than 100 cycles would have this property, even though the function being modeled still renaming invariant. Though the in-distribution error of such dataset may suffer with the invariant model, the transformed register evaluation performed in Section 5 would still result in the invariant model having better crror.
7 RELATED WORK
A Anonymization and canonicalization. Anonymizing and the canonicalization of training data is an area of much focus in fields ranging from ethical AI to privacy-preseringg AI. In an effort to reduce gender and region bias in gendered pronoun resolution Liu (2019) mask individual names by drawing from a set of canonical names. Similarly, for debiasing and preserving privacy in clinical ml, de-identification of data is a prevalent technique (Dernoncourt et al., 2016; Liu et al., 2017; Johnson et al., 2020; Minot et al., 2022). Most of the work; however, focused on the process of automatic de-identification and not on the result of training on de- identified data. Minot et al. (2022) investigate the result of training a canonicalized version of medical records. While canonicalization and de-identification provide the bonefit of roducing the bias of the model, they suffer from the fact that not every canonical representation of the same entity is guaranteed to have the same representation and that inputs which have more entities than the number of canonical representations trained can't it be represented. Furthermore, training on de-identified data is often associated with a degradation to network performance.
Transformer invariances. A wide variety of invariances and equivariances have been encoded in the architecture of Transformer networks. Lee et al. (2019) introduce Set Trans- former, a Transformer invariant to permutations of the ordering the input sequence. Fuchs et al. (2020) suggest SE (3), Transformer architecture which is equivariant to 3D translations and rotations, which they evaluate on a variety of domains ranging from n-body simulations to point-cloud object classification. Translation invariance has also been exploited in the context of natural language lasks (Su et al., 2021; Wemnberg and Henter, 2021). While these works enforce invariances, they all deal with spatial or positional invariances. To our knowledge, there is limited prior work on encoding invariances regarding the content of individual tokens.
8 CONCLUSION
This work formalizes the concept of renaming invariance, and proposes the first Transformer architecture that is invariant to semantics-prserving variable renaming through the use of view anonymization by shared content embeddings and a novel restricted attention map that binds referents and restores the network's representational capacity. Our results demonstrate that on the BHive dataset, the invariant architecture substantially improves the best performance of the network on the clean test set, register renamed test set, and delivers significant training speed-ups. This work is an important first step to designing renaming invariant neural networks, which is ubiquitous and understudied invariance.
9

---

Under review as conference paper at ICLR 2023
REFERENCES
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed representations of code. In Principles of Progrummingg Languages, 2019.
Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Compulational Lánguisties (Volume 1: Long Papers), 2014.
Federico Bianchi, Debora. Nowa, and Dirk Hovy. Language invariant properties in natural language processing. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Hcidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint ar Xie:2107 03377,, 2021.
Yishen Chen, Ajay Brahmakshatriya, Charith Mendis, Alex Renda, Eric Atkinson, Ondrej Sykora, Saman Amarasinghe, and Michael Carbin. BHive: benchmark suite and measure- ment framework for validating x86-64 basic block performance models. In IEEE International Symposium on Workload Characterization, 2019. doi: 10.1109 WWT.C42...111.9904224
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International Conference on Machine Learning, 2016.
Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3). 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, 2019. doi: 10.18653 /v1 11/N19-1423.
Pre-training
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics, 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se(3)-transsfrmers:: 3d rolo-translation equivariant atlention networks. In Advances in Neural Information Processing Systems, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compulation, 9(8,, 1997. doi: 10.1162/ n0.1162//eco..997.933 1997.9.8.1735.
Alistair E. W. Johnson, Lucas Bulgarelli, and Tom J. Pollard. Deidentification of free-text medical records using pro-trained bidirectional transformers. In Proceedings of the ACM
Conference on Health, Inference, and Learning, CHIL. '20, page 214-221, New York, NY, USA, 2020. Association for Computing Machinery. iSBN 9781450370455 doi: 10.1145/ 3368555.3384455. URL https://doi. wrg.oio.oorr///// 1145/3368555 3384455.
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. In Advances in Neural Information Processing Systems, 2019.
10

---

Under review as conference paper at ICLR 2023
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations, 2020.
Chris Lattner and Vikram Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In International Symposium on Code Generalion and Oplimization, 2004. doi: 10.1109 /C///C/OO0000 2.0..000..281665.
Yann LeCun and Yoshua Bengio. Comvolutional nelworks for images, speech, and lime-series. MTT Press, 1995.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariat neural networks. In Proceedings of the 36th International Conference Machine Learning, 2019.
Bo Liu. Anonymized BERT: An augmentation approach to the gendered pronoun resolution challenge. In Workshop on Gender Bias in Natural Language Processing, 2019.
Zengjian Liu, Buzhou Tang, Xiaolong Wang, and Qingeai Chen. De-identification of clinical notes via recurrent neural network and conditional random field. Journal of Biomedical Infor- matics, 75:834-S42, 2017. ISSN 1532-0464. doi: https:/ /tt.:///ooooorggo/ wttp://...ii..1....... URL https: / /www. sciencedirect. com/cc////////oc www///cienee/arreeeeeeee http//rtiiceeeeeeeeeeeeeeeeeeeet /ttpe///.//111411110000000 Supplement: A Natural Language Processing Challenge for Clinical Records: Research Domains Criteria (RDoC) for Psychiatry.
Alana Marzoev, Samuel Madden, M. Frans Kaashock, Michael Cafarella, and Jacob Andreas. Unnatural language processing; Bridging the gap between synthetic and natural language data. Xin preprint ar Xiv:20004. 19645, abs/2004. 1.004.13644 2020.
Joshua R. Minot, Nicholas Cheney, Marc Maier, Danne C. Elbers, Christopher M. Danforth, and Peter Sheridan Dodds. Interpretable bias mitigation for textual data: Reducing genderization in patient notes while maintaining classification performance. ACM Trans. Comput. Healthcare, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit. Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperat ive style, higl-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Πlya Sutskever. Formal mathematics statement. curriculum learning. arXio preprint or Xin:22000. 01344, 2022.
Alex Renda, Yishen Chen, Charith Mendis, and Michael Carbin. Difftune: Optimizing epu simulator parameters with learned differentiable surrogates. In IEFE/ A CM International Symposium on Microarchiteccure, 2020. doi: 10.1109/ MICRO50266.200 0t112/66..2200000000
Alex Renda, Yi Ding, and Michael Carbin. Programming with neural surrogates of programs. In ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections Programming and Software Doftware((nwwal (Onwand!), 2021.
Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6, 2019. doi: 10.1166//40537-01-00000
Noah Snavely. Lecture notes: Cs5670: Computer vision lecture 5: Feature invariance, 2019. URL https: h/ww:://wwwwc..o cornell. edu/ /tth://orrree/ee////////////e//e iance. .df.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. XrXiv preprint or arXim:2104.0955 2021.
11

---

Under review as conference paper at ICLR 2023
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint or Xio: 08962v2, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Rui Wang, Robin Walters, and Rose Yu. Data augmentation vs. equivariant networks: A theory of generalization dynamics forecasting. In ICML Workshop on Principles of Distribution Shift, 2022.
Ulme Wennberg and Gustav Eje Henter. The case for translation-invarian self-attention in transformer--ased language models. arXw preprint (XX::2106.01955 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Authony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics. URL https: / //www aclweb. org/ htggnnnhology///// gatlooyyy220000 2020. omllp-deos.6 6.
Transformers:
12

---

Under review as conference paper at ICLR 2023
mov 64 (%xrs)) Kraz
Vanilla
0.03
sub $1, 56 (%rbp)
Augmented
Froquency
16(2rax) ) Xeaz
0.02
Renamer
llym-mca:
1.68 cycles Vanilla: 1.51 cycles
1lym-mea
Augmented: 1.12 cycles Renamer: 1.72 eycles
0.01
64 (%r8), Araz
sub $1, 56 (%rbp)
0.00
mov 16(Xraz) ) Keax
2
4
Ilvm-mea: 1.68 cycles
Vanilla: 2.32 cycles
Prediction (cycles)
Augmented: 1.12 cycles Renamer:
1.72 cycles
Figure 3: The range of generated predictions for renamings of the basic block in Figure ((a).
A APPENDIX
A.1 RENAMING SENSITIVITY CASE STUDY
We now present & more in-depth case study of the basic block presented in Figure 1.
Figure 3 presents a case study of each model's predictions on the basic block presented in Figure I(a). The figure on the left is a histogram and corresponding density plot of predictions on semantically equivalent renamings of the basic block. To generate this plot, we uniformly sample 100,000 valid semantics- praetticsppeeevinng register permutations 0, apply the permutation to the original block, then evaluate each model on the permuted block. The models under study are single trials of the best-performing BERT-Tiny models. The ground-truth timing for this basic block as output by 1lvm-mca is 1.68 cycles.
The vanilla model generates a range of predictions for permutations for this block, ranging from 1.11 cycles to 4.86 cycles. The predictions generated by the vanilla model are multimodal, though there are no clear indicators for which mode given block will induce. The augmented model generates significantly smaller range of predictions though there is some variation on the order of one thousandth of a cycle, the predictions are essentially constant at 1.12 cycles. By construction, Renamer generates constant predictions for this basic block of 1.72 cycles.
A.2 SYMBOLIC ALGEBRA CASE STUDY
In this section we study how the Renamer architecture can be extended to other tasks. Specifically, we apply Renamer to the task of determining whether one equation is the partial derivative of another.
Datasct. Wecvaluate on a variant of the Backward dataset presented by Lample and Charton (2020). The dataset is collection of 300,000 pairs of expressions. Each expression contains up to Whree different variables (z,,,,) and up to 3 different symbolic coefficients (a0,al (1,22. The task is to determine whether the first expression is the partial derivative of the second with respect to z. The expressions are in prefix notation. For example, the network should predict true for the pair of expressions a0, mul a0 x, since the first expression is the partial derivative of the second with respect to æ. The network should predict false for the pair of expressions a0, mul y, since the first expression is not the partial derivative of the sccond with respoct to x.
Variable renaming in symbolic algebra task. We define the set of referents to be the names of the variables. Thus the set of referents is (a0,al,a2,22.,,22
We define a view as a group of variables which have the same functional purpose. Thus all coefficients share the same view a. Since is the variable differentiate with respect to, has a singleton view z. Finally, as y and & are both variables that are inputs to functions but not differentiated, they share view Thus the set of views is {a,r.,).
13

---

Under review as conference paper at ICLR 2023
Table 3: Accuracy of symbolic algebra case study.
Model type
Test set
Vanilla
Canonicalized
Renamer
Original
99.21% + 13%
99.29% +0.14%
99.20% +0.06% 97.62% +0.16%
Augmented
95.32% +0.69%
95.02% +0.56%
Models. For all experiments we use a BERT-Small (Devlin et al., 2019). We compare the performance of Renamer against vanilla and canonicalized baseline.
Results. In Table 3 we evaluate all models on both the original test set and an augmented version of the lest set. To build the augmented lest set, we introduce new coefficient a3 to each equation. We find that all models perform similarly on the original test set. While the canonical- ized model has the highest average accuracy, it only outperforms Renamer by 0.09% accuracy, While the performance of all models decreases on the augmented test set, we find that on the aug- mented test set Renamer significantly outperforms all other models. The vanilla model achieves an accuracy of 95.32% on the augmented test set while Renamer reaches an accuracy of 97 7.62% which an improvement in accuracy of 2.30%. Similarly, the canonicalizod model achieves an accuracy of 95.02% which means Renamer has improvement in accuracy of 2.60%.
14


Reviews:
[{"review_id": "rqln4wFVUh", "reviewer": "Reviewer_1jmm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Renamer, which builds upon the Transformer architecture to achieve variable renaming variance. Their approach involves two parts: view anonymization and referent binding, and they focus on representing x86-64 blocks in their implementation. Specifically, view anonymization maps all registers with the same bitwidth to the same token for input embedding, and referent binding adds an attention mask so that the attention is computed only among registers referring to the same location in memory. They evaluate their approach on BHive benchmark for throughput prediction, and add their Renamer upon BERT-Tiny, BERT-Mini, and BERT-Small. Empirical results show that their Renamer variants outperform the baseline BERT architectures on both the original test set and a variant of the test set with renamed registers. ", "summary_of_the_review": "Proposing approaches to achieve renaming invariance while preserving the overall task performance is a good research topic. However, the evaluation in this work is not solid, the description of their approach is unclear, and the implementation of Renamer seems specific to the task in their evaluation. Therefore, I recommend a rejection.\n\n---------------\nI thank the authors for the response and paper revision. The added experiments on Backward dataset help me better understand the approach. However, I still think the technical novelty is limited, and the applicability of the approach is a bit narrow. Therefore, I slightly improve my score to 5.", "strength_and_weaknesses": "Strengths:\n\nIn general, renaming invariance is an important desired property of neural networks for code, and the authors present a domain where achieving renaming invariance improves the performance over baseline architectures.\n\nWeaknesses:\n\nOne key issue of this work is that the empirical study is not solid. First, NA in Table 1 and Table 2 is not well justified. Training with canonicalization does not add more computational costs compared to other methods, while it achieves much better results than other baselines. Therefore, even if the authors do not have enough computational resources to evaluate larger BERT models, which should further improve the results, at least the authors should complete the evaluation of smaller BERT models.\n\nFurthermore, there is no comparison to SOTA models to justify the significance of the results. To meet their assumptions of register renaming invariance, the authors only train and evaluate on a subset of the original BHive benchmark, where they remove inputs with implicit operands. This makes it difficult to directly compare numbers in this paper to prior works that evaluate on this benchmark. Thus, the authors should evaluate the SOTA models on their subset and present the results.\n\nMeanwhile, although the paper starts with a general definition of renaming invariance, the separation of view and referent seems specific to the task in their evaluation, and does not add much to the variable renaming invariance that has been discussed in prior works. In particular, there is no analysis on why and how Renamer outperforms canonicalization.\n\nAlso, the description of Renamer architecture modification is unclear. My understanding is that view anonymization is similar in spirit to canonicalization, while referent binding adds additional constraints on the Transformer attention. In this case, I think Renamer does not preserve the expressive power of Transformer with canonicalization. More explanation about the implementation of modifications is helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g-RU0SMjXRu", "reviewer": "Reviewer_KJQk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the concept of renaming invariance and develops a model with this property. They carry out experiments and demonstrate that their model performs better.", "summary_of_the_review": "The paper is well-timed to be interesting given the interest in apply machine learning to code and mathematics.", "strength_and_weaknesses": "The paper develops good intuition with the application to formal languages and concrete examples. Experimental results are very strong on standard task.\n\nThe main weakness is that they only work with one dataset, so it's hard to know how much the technique generalizes. Since they mention applications to mathematics or synthetic grammars, it would have been nice to see such experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CvjlDulXFE", "reviewer": "Reviewer_uTBf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a modified Transformer (Renamer) which is invariant under renaming of variables. The design is tested on a task to predict CPU clock cycles to execute blocks of assembly language. The proposed model is compared against an unmodified Transformer which is optionally also trained on inputs with renamed variables (augmented), or adds a preprocessing step to fix variable names for the input (canonicalized). Renamer reduces the error for the studied task compared to all three baseline models.\n", "summary_of_the_review": "I would recommend to accept this paper for publication. The modification to the standard Transformer architecture is simple but effective.", "strength_and_weaknesses": "The paper is well presented, both the task and architecture changes are well motivated and described clearly. The specific task studied is interesting and the proposed changes might also benefit other tasks. \n\nThe main weakness of the paper is its narrow scope. \n\nA single dataset is used for evaluation. Further, inputs which contain implicit operands are removed from the evaluation. These operations read to / write from hardcoded registers, so these are not invariant to renaming. Could they have been kept in the dataset but the hardcoded variables handled differently from those that can be renamed? \n\nThe explanation of view is closely tied to this task. Considering other tasks that are invariant to variable renaming, what is the equivalent concept and what are the implications for the proposed method? For example, if evaluating C code, variable types provide semantic information which can be separated from information about the referent. What about working with a dynamically typed language? Or similarly, with the masked attention stage, what is the effect of having a variable reassigned during a code block so that its referent changes? \n\nOverall, the paper neither seriously discusses nor tests whether the proposed method is applicable for any other tasks which are invariant to renaming of variables. To make the work more impactful it should be clarified whether and how the method would need to be modified for other tasks, preferably with demonstration of its effectiveness through additional evaluations. \n\nOnly small models are evaluated, and benefits observed in small models do not always remain for larger models. This is a minor weakness. Being able to achieve good results using small models is valuable in itself; with evaluations on larger models we might also see how much is gained by architectural changes vs more compute. \n\nQuestions:\n\nIt’s interesting that the augmented model shows only a very small drop in performance when tested on renamed inputs, but overall performs less well than the model trained without data augmentation. Did you try training for longer with more variants? I’d be interested to know the limits of data augmentation to improve robustness to renaming in this case. \n\nWhy did the canonicalized model take longer to train? My understanding is that the inputs were first modified to a standard version, then the model trained on these. It’s not immediately obvious to me why this should take much longer to train than the basic setup. I may have misunderstood something here. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject