## 3 The Proposed Approach
### Context-Aware NMT Architecture  
CodeTransFix constructs an APR model by integrating a fine-tuned CodeBERT with an NMT model, as illustrated in Figure 4. In this architecture, the fine-tuned CodeBERT exclusively computes the context vectors around the buggy lines of code; in comparison, the Transformer-based encoder-decoder framework receives both the buggy lines of code themselves and their context vectors and learns the transition pattern from the buggy code to the repaired code. Notably, the model innovatively introduces a Drop-net mechanism [15] in the Transformer decoder to fuse the context vectors generated by CodeBERT with the hidden state information of the decoder itself. As shown in Figure 4, to simplify the schematic, we only show the key structures in the model. Figure 4: The architecture of the APR models used in CodeTransFix. The orange box highlights the buggy line, the purple box represents the context, and the blue box indicates the generated patch. Since CodeBERT was fine-tuned in the previous step, its parameters must be kept fixed (frozen state) while training the APR model incorporating CodeBERT. In the training phase, the APR model takes the buggy code lines, contexts, and corresponding fix patches as the training data and learns how to generate the correct fix patches based on the buggy code lines with contexts. The model parameters are optimized through multiple training epochs to ultimately obtain the weight combination with the best performance. In the inference phase, since the APR model can only access buggy lines and their contexts, the decoder generates patch sequences token-by-token using the start token 
 as a starting point. Specifically, the encoder-decoder multi-head attention module processes two sources of information in parallel: feature vectors from buggy code lines and their context vectors. The outputs of the encoder-decoder attention mechanisms are combined through a Drop-net mechanism to predict the current output token. This generated token is subsequently fed into the decoder as the input for the subsequent time step. The iterative generation process continues until the termination token 
 is produced, ultimately forming the complete repair patch. When the user inputs the error statement \"int g = (int) ((value \(-\) this.lowerBound)/ (this.upperBound \(-\) this.lowerBound) * 255.0)\"; (shown as an orange square) and its context (shown as a purple square) to CodeTransFix. After tokenization, the error statement is fed into the Transformer encoder, while the context is fed into the CodeBERT model. Subsequently, the hidden state \(H_{\text{C}}\) output by the Transformer encoder and the context representation \(H_{\text{E}}\) generated by CodeBERT are fed into the encoder-decoder multi-head attention layer of the Transformer decoder for joint modeling. In the patch generation phase, the decoder starts with a start token 
 and generates a sequence of patches token by autoregressive approach until a termination token 
 is generated to end the generation process. In the following section, we detail the Drop-net mechanism [15] employed during decoding. Specifically, a parameter \(P_{net}\in[0,1]\) is introduced for the decoder layer. During each training iteration, for the \(l\)-th layer of the Transformer decoder, a random variable \(U^{l}\) is uniformly sampled from the interval \([0,1]\). The attention output of the Transformer decoder is then computed as follows:  
\[ \begin{array}{l}h=\mathbb{I}\Big{(}U^{l}<\frac{P_{net}}{2}\Big{)}\cdot attn_{ \text{C}}\Big{(}s^{l},H_{\text{C}},H_{\text{C}}\Big{)}+\mathbb{I}\Big{(}U^{l}> 1-\frac{P_{net}}{2}\Big{)}\cdot attn_{\text{E}}\Big{(}s^{l},H_{\text{E}},H_{ \text{E}}\Big{)}\\ \qquad\qquad+\mathbb{I}\mathbb{I}\Big{(}\frac{P_{net}}{2}\leq U^{l}\leq\frac{ P_{net}}{2}\Big{)}\cdot\Big{(}attn_{\text{C}}\Big{(}s^{l},H_{\text{C}},H_{ \text{C}}\Big{)}+attn_{\text{E}}\Big{(}s^{l},H_{\text{E}},H_{\text{E}}\Big{)} \Big{)}\end{array}