GEPCode: A Context-Aware 1M-Parameters Graph-Based Language Model for Source Code

Authors: Federico Cichetti, Emanuele Parisi, Andrea Acquaviva, Francesco Barchi
Primary area: learning on graphs and other geometries & topologies
Keywords: graph neural network, graph language model, source code optimization

Abstract:
The pursuit of optimal conditions for software execution poses a complex challenge. This task can be automated by harnessing the structured nature of programming languages, especially from compiler intermediate representations of code (IR). The manipulation of source code using Large Language Models (LLMs) is a thriving area of study in Natural Language Processing (NLP) literature. However, in this study we illustrate how we can circumvent the need for exceedingly large models by employing domain-specific language models. These models have a reduced number of parameters but retain the ability to capture the relationships within source code elements. We introduce GEPCode, a graph neural network designed to model IR with the flexibility to adapt to new tasks. This flexibility is obtained through special "meta" nodes, that allow for the representation of additional task-dependent contextual information. Pre-training is performed by solving node and graph-level tasks, resulting in a general language model. After a fine-tuning phase on two downstream tasks, Device Mapping and Algorithm Classification, we achieve average accuracy results of 88.9% (NVIDIA) and 92.3% (AMD) for the former and 97.2% for the latter. Comparing our methodology with state-of-the-art models trained from scratch, our results are similar or better, yet providing a more flexible model. Moreover, we achieve similar accuracy results in downstream tasks compared to state-of-the-art pre-trained language models based on Transformers, while utilizing 100 times fewer parameters.

Decision: reject
Reviewer scores: 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729220293518

thread_type:
official_review

<thread_object>
creator: reviewer_jptt

<content>
summary:
The paper proposes GEPCode, a Graph Neural Network (GNN) to embed compiler intermediate representation of source code. GEPCode is pre-trained on 1.3M graphs constructed from source code for three objectives: (1) Attribute Masking, (2) Meta Prediction, and (3) Contrastive Learning. The resulting model is then fine-tuned for two downstream tasks respectively: (1) Device Mapping and (2) Algorithm Classification. Evaluation shows that GEPCode, which has only 1M parameters, achieves comparable performance with respect to much larger pre-trained transformer models, highlighting its efficiency.

strengths:
* While previous works have approached the same problem with graph representations, GEPCode is the first to *pre-train* an GNN for intermediate representation, and proposed the three new objectives for pre-training.
* The proposed method is compared against a comprehensive list of baseline methods. All experiments are repeated 5 times for reliability.
* The paper is written clearly.

weaknesses:
* The proposed method does not show a clear advantage over Perfograph which performs better on DevMap with an even smaller model size, and does not require pre-training.
* It is highlighted that GEPCode is more parameter-efficient than pre-trained transformers. I do not think this is a well-established advantage of the proposed method unless it can be demonstrated that GEPCode outperforms the SOTA pre-trained transformer of the same model size.
* Regarding efficiency, I'm not sure about the practical benefit of reducing the inference time from around 100ms (with CodeT5 or other transformers) to 23ms (with GEPCode), at the cost of accuracy degradation. Those transformers are already pretty small and fast, whose inference latency should be acceptable for DevMap and algorithm classification that are considered in evaluation.
* The baseline transformer models are mostly trained on multilingual programming language data, while GEPCode is trained on IR which is a single language. Is it possible that a monolingual transformer model trained only on the target language for the evaluation task would perform better, and thus the baseline numbers are underestimated?

questions:
* A transformer model over flattened source code is naturally a GNN over fully connected graphs. It would be interesting to study how it performs to directly pre-train a transformer model over the same data with the same objectives.
* L292 is confusing to me. Is the "normalized property value" in META edges the same as $m$ in eq.6? If that's the case, isn't the training objective already known from the input?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465380234

thread_type:
official_review

<thread_object>
creator: reviewer_v1op

<content>
summary:
This paper introduces GEPCode, a novel approach using a Graph Neural Network to model compiler intermediate representations of code with the adaptability to accommodate new tasks. The model is first pre-trained as a general-purpose language model, then fine-tuned on downstream tasks, achieving results comparable to state-of-the-art models trained from scratch. Additionally, GEPCode performs similarly to leading pre-trained Transformer-based language models while requiring fewer parameters, offering a more flexible and efficient solution.

strengths:
The problem statement addressed in this paper is worth solving as there is demand for smaller, efficient and flexible models in this domain.

weaknesses:
(1) Typically, fine-tuning is performed for 1 to 10 epochs. Here, fine-tuning for both downstream tasks was run for 100 epochs, which could have led to overfitting.

(2) The accuracy of fine-tuned GEPCode on downstream tasks is compared against pre-trained transformer-based language models. A comparison with fine-tuned transformer-based language models for our downstream tasks would have been ideal.

questions:
Please clarify if IRGen is a non-transformer-based pre-trained language model that performs better than GEPCode.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730785802798

thread_type:
official_review

<thread_object>
creator: reviewer_nyas

<content>
summary:
The paper introduces GEPCode, a graph-based, efficient, pre-trained, context-aware LM of graph representations of source code. The proposed approach expands upon ProGraML and Perfograph. GEPCode expresses LLMV-IR code samples as graphs, where nodes are token within a vocabulary of LLVM-IR elements, and edges are directed and represent dependencies between the elements of code. GEPCode is pretrained using synth-compilable subset of the Exebench dataset, using three tasks - attribute masking, meta prediction, and contrastive learning. For downstrem tasks, GEPCode is evaluated on device mapping and algorithm classification. Experimental results demonstrate that GEPCode is able to bridge the gap between the efficiency of task specific architectures and the generality of larger LMs, while using a limited number of parameters.

strengths:
- Overall the paper is written well. The motivation behind the model choice is sound.
- The pretraining objectives that include three tasks are effective.
- Experiment results show the value of the proposed method.

weaknesses:
- The biggest weakness of the work is the novelty. In a nutshell, it is a paper that is using GNN for some specific code tasks. GNN was previously explored for coding tasks, however, with the emergence of LLMs, the focus has shifted. In this paper, authors emphasized on compact LMs, but why GNN is the solution, it is not clear.
- The paper could use more software engineering tasks for evaluation. In the analysis part of the work, there is not much critical thinking paid by the authors. Straight-forward main results and a piece of ablation study - that's it. Moreover, I didn't understand the baselines used in comparison.

questions:
- Why GNN is used as a solution to build compact LMs in this work? What is the motivation.
- The paper uses LLMV-IR representation of code; in that case, is the proposed method scalable?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes GEPCode, a GNN-based pre-trained model over LLVM-IR. It uses three pre-training objectives—attribute masking, meta prediction, and contrastive learning—and is tested on device mapping and algorithm classification. The authors claim GEPCode outperforms larger transformer-based models in efficiency while maintaining comparable accuracy.

**Strengths:**

* The paper highlights “demand for smaller, efficient and flexible models in this domain” (V1oP), which is timely given the growing focus on efficiency in modern ML research.

* Multiple reviewers (e.g., NyAs, jpTT, V1oP) note that the paper is “written well” (NyAs) and “written clearly” (jpTT).

* The inclusion of three tasks—attribute masking, meta prediction, and contrastive learning—finding that these “are effective” (NyAs)

* Comprehensive set of experiments. (jpTT)

**Weaknesses:**

* Unclear motivation of using GNN: “why GNN is the solution, it is not clear” (NyAs).

* It is also unclear how the baselines were chosen (NyAs), and a comparison with fine-tuned transformer-based language models would be beneficial (V1oP)

* Weak comparison with prior methods. Perfograph performs better on DevMap with fewer parameters and no pre-training (jpTT).

Though the paper is well-written and tackles a relevant problem, reviewers remain unconvinced about GEPCode’s novelty and its empirical advantages. The authors’ rebuttal did not fully address these concerns.