Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning
Authors: Nan Jiang, Chengxiao Wang, Kevin Liu, Xiangzhe Xu, Lin Tan, Xiangyu Zhang, Petr Babkin


Reviewer paper summaries: The paper proposes two methods to improve the performance of LLMs trained on assembly code: 1) Hierarchical attention mechanism that allows the model to attend to both the instruction level and the token level. 2) Contrastive learning that allows the model to learn the similarity between assembly code in terms of their functionality/semantics. The model is evaluated on the tasks of assembly to source code decompilation and assembly code similarity search, showing that the proposed method outperforms the state-of-the-art methods.
Summary of strengths: * The proposed system Nova achieves state-of-the-art performance on the tasks of assembly to source code decompilation and assembly code similarity search.
* They do further analysis and ablation of their results:
    - Further analysis and 2D visualization of embeddings showing that with their method, the same functions compiled with different compiler optimization levels are clustered together. 
    - Analysis of the attention comparing with the baseline model and visualization of the attention weights is helpful to understand the model behavior.
Summary of weaknesses: * Lack of citations and related work comparison to relevant hierarchical/block/sparse attention and attention sink works in the literature. Some discussion on how this is related to previous works on the design of attention mechanisms would be helpful to readers.
Comments, suggestions and typos: * What is the reason for adding whitespace around ",", "(", ")" in processing the training data? Is it due to tokenization?
* Are the `[INST-1]`, `[INST-2]`, `[INST-3]`, `[INST-4]` etc. all different tokens in the vocabulary? If so, how many of these special tokens are added to the vocabulary? And just to be clear, when the authors refer to `
` tokens, are they referring to `[INST-1]`, `[INST-2]`, `[INST-3]`, `[INST-4]` etc., with the form of `[INST-[0-9]*]`?
* Could you clarify how to get embeddings from the model? It is not very clear how the authors get the embedding from the autoregressive model and use it throughout the paper. Is it by taking the average of the last layer's activations of those '[INST-*]' tokens?
* It is mentioned that in the first stage of pretraining, only half the heads are used with the hierarchical attention. Could you provide more details on what subsequent stages of fine-tuning look like? Do all heads use hierarchical attention in the subsequent stages?
Overall assessment scores: 3.5