Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a modified Transformer (Renamer) which is invariant under renaming of variables. The design is tested on a task to predict CPU clock cycles to execute blocks of assembly language. The proposed model is compared against an unmodified Transformer which is optionally also trained on inputs with renamed variables (augmented), or adds a preprocessing step to fix variable names for the input (canonicalized). Renamer reduces the error for the studied task compared to all three baseline models.

strength_and_weaknesses: The paper is well presented, both the task and architecture changes are well motivated and described clearly. The specific task studied is interesting and the proposed changes might also benefit other tasks. 

The main weakness of the paper is its narrow scope. 

A single dataset is used for evaluation. Further, inputs which contain implicit operands are removed from the evaluation. These operations read to / write from hardcoded registers, so these are not invariant to renaming. Could they have been kept in the dataset but the hardcoded variables handled differently from those that can be renamed? 

The explanation of view is closely tied to this task. Considering other tasks that are invariant to variable renaming, what is the equivalent concept and what are the implications for the proposed method? For example, if evaluating C code, variable types provide semantic information which can be separated from information about the referent. What about working with a dynamically typed language? Or similarly, with the masked attention stage, what is the effect of having a variable reassigned during a code block so that its referent changes? 

Overall, the paper neither seriously discusses nor tests whether the proposed method is applicable for any other tasks which are invariant to renaming of variables. To make the work more impactful it should be clarified whether and how the method would need to be modified for other tasks, preferably with demonstration of its effectiveness through additional evaluations. 

Only small models are evaluated, and benefits observed in small models do not always remain for larger models. This is a minor weakness. Being able to achieve good results using small models is valuable in itself; with evaluations on larger models we might also see how much is gained by architectural changes vs more compute. 

Questions:

It’s interesting that the augmented model shows only a very small drop in performance when tested on renamed inputs, but overall performs less well than the model trained without data augmentation. Did you try training for longer with more variants? I’d be interested to know the limits of data augmentation to improve robustness to renaming in this case. 

Why did the canonicalized model take longer to train? My understanding is that the inputs were first modified to a standard version, then the model trained on these. It’s not immediately obvious to me why this should take much longer to train than the basic setup. I may have misunderstood something here. 

clarity,_quality,_novelty_and_reproducibility: The work is communicated well.

The technique is novel, but potentially narrow.
summary_of_the_review: I would recommend to accept this paper for publication. The modification to the standard Transformer architecture is simple but effective.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces the concept of renaming invariance and develops a model with this property. They carry out experiments and demonstrate that their model performs better.
strength_and_weaknesses: The paper develops good intuition with the application to formal languages and concrete examples. Experimental results are very strong on standard task.

The main weakness is that they only work with one dataset, so it's hard to know how much the technique generalizes. Since they mention applications to mathematics or synthetic grammars, it would have been nice to see such experiments.
clarity,_quality,_novelty_and_reproducibility: The paper is clear. It should be reproducible based on the method description and training setup. To my knowledge the work original. Transformers with invariances exist. There are invariances similar in spirit in natural language but their definition is unique enough to be novel.
summary_of_the_review: The paper is well-timed to be interesting given the interest in apply machine learning to code and mathematics.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Renamer, which builds upon the Transformer architecture to achieve variable renaming variance. Their approach involves two parts: view anonymization and referent binding, and they focus on representing x86-64 blocks in their implementation. Specifically, view anonymization maps all registers with the same bitwidth to the same token for input embedding, and referent binding adds an attention mask so that the attention is computed only among registers referring to the same location in memory. They evaluate their approach on BHive benchmark for throughput prediction, and add their Renamer upon BERT-Tiny, BERT-Mini, and BERT-Small. Empirical results show that their Renamer variants outperform the baseline BERT architectures on both the original test set and a variant of the test set with renamed registers. 
strength_and_weaknesses: Strengths:

In general, renaming invariance is an important desired property of neural networks for code, and the authors present a domain where achieving renaming invariance improves the performance over baseline architectures.

Weaknesses:

One key issue of this work is that the empirical study is not solid. First, NA in Table 1 and Table 2 is not well justified. Training with canonicalization does not add more computational costs compared to other methods, while it achieves much better results than other baselines. Therefore, even if the authors do not have enough computational resources to evaluate larger BERT models, which should further improve the results, at least the authors should complete the evaluation of smaller BERT models.

Furthermore, there is no comparison to SOTA models to justify the significance of the results. To meet their assumptions of register renaming invariance, the authors only train and evaluate on a subset of the original BHive benchmark, where they remove inputs with implicit operands. This makes it difficult to directly compare numbers in this paper to prior works that evaluate on this benchmark. Thus, the authors should evaluate the SOTA models on their subset and present the results.

Meanwhile, although the paper starts with a general definition of renaming invariance, the separation of view and referent seems specific to the task in their evaluation, and does not add much to the variable renaming invariance that has been discussed in prior works. In particular, there is no analysis on why and how Renamer outperforms canonicalization.

Also, the description of Renamer architecture modification is unclear. My understanding is that view anonymization is similar in spirit to canonicalization, while referent binding adds additional constraints on the Transformer attention. In this case, I think Renamer does not preserve the expressive power of Transformer with canonicalization. More explanation about the implementation of modifications is helpful.
clarity,_quality,_novelty_and_reproducibility: The description of Renamer implementation is not clear enough.

The proposed approach is not technically very novel. The separation of view and referent seems specific to the task in their evaluation, and does not add much to the variable renaming invariance that has been discussed in prior works.

The authors did not provide the code and data for their experiments. As the technical details are not clear, it could be hard to reproduce their results.
summary_of_the_review: Proposing approaches to achieve renaming invariance while preserving the overall task performance is a good research topic. However, the evaluation in this work is not solid, the description of their approach is unclear, and the implementation of Renamer seems specific to the task in their evaluation. Therefore, I recommend a rejection.

---------------
I thank the authors for the response and paper revision. The added experiments on Backward dataset help me better understand the approach. However, I still think the technical novelty is limited, and the applicability of the approach is a bit narrow. Therefore, I slightly improve my score to 5.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers find the paper to be well written - both the problem statement and the proposed solution are presented well. Reviewers mainly are worried about the narrow scope of the paper, evaluation on only one dataset and non-standard evaluation by removing datapoints with implicit operands. While authors added experiment results on another dataset during response, reviewers remain unconvinced about the general usefulness of the proposed approach. Hence I recommend rejection.
justification_for_why_not_higher_score: Limited scope of the problem and results.
Decision: Reject