## 1 Introduction  
Automatic program repair (APR) technology [1, 2] plays a vital role in software quality assurance by automatically generating patches for defects discovered during debugging. This technology significantly reduces developer maintenance efforts while improving software reliability. Among existing APR approaches, methods based on neural machine translation (NMT) models [3, 4] have gained prominence due to their ability to learn translation patterns from bug-fix code pairs (BFPs). In recent years, APR techniques have seen yet more breakthroughs with the application of LLMCs [5, 6] to the APR domain. In particular, fine-tuned large language models of code (LLMCs) such as CodeT5 [7] and PLBART [8] have shown impressive performance in bug fixing [9, 10]. Compared to natural language, source code exhibits strict syntactic features. Existing NMT-based APR approaches [3, 11, 12] learn source code semantics by incorporating knowledge from different domains to repair defective programs. For instance, CoCoNut [3] proposes a novel NMT architecture that employs two convolutional neural network encoders to separately encode contextual information (i.e., the immediate surrounding code within the same method or code block) and erroneous lines. DLFIX [11] utilizes tree-structured recurrent neural networks to learn contextual patterns and defect repair mechanisms. Recoder [12] integrates programming syntax knowledge to guide the generation of syntactically correct patches. However, due to the limited scale of model parameters and training data, NMT-based APR methods may fail to adequately learn strict syntactic features, consequently limiting their generalizability to unseen error patterns during training [9; 10]. In contrast, large language models of code pre-trained on extensive code corpora demonstrate enhanced capabilities in bug repair tasks [13; 14]. For example, VulRepair [14] leverages the CodeT5 model for vulnerability repair and generally outperforms NMT-based APR approaches. Nevertheless, this method only considers defective statements while neglecting the contextual dependencies (e.g., variable interactions, method dependencies) of error-prone code, despite the critical role of contextual code (defined as the immediate surrounding code within the same method or code block) in understanding erroneous behaviors. To address the aforementioned limitations, we propose CodeTransFix, a novel approach that incorporates the CodeBERT model to more effectively repair erroneous code. Our method leverages the vast parameters of CodeBERT to comprehend the strict syntactic features of programming languages and integrates the surrounding context of erroneous statements to guide the repair process. The method operates in two phases:  
First, CodeBERT learns contextual information by replacing bug lines in code segments with the special token <BUG>, generating modified context as model input. This design is motivated by the critical role of contextual code semantics (e.g., variable interactions, method dependencies) in understanding bug behavior. The <BUG> token explicitly marks the error location, allowing the model to focus on learning repair patterns from a structural context. Second, a Transformer encoder-decoder architecture learns repair operations for buggy code. To fully utilize the knowledge learned by CodeBERT, we implement a Drop-net mechanism [15] to fuse contextual representations from CodeBERT's output. In this study, we evaluate two large language models of code based on the encoder-decoder architecture (CodeT5 [7] and PLBART [8]) and three state-of-the-art NMT-based APR approaches (CURE [16], RewardRepair [17], and Recoder [12]) on two benchmarks: Defects4J v1.2 [18] and Defects4J v2.0 [18]. In summary, this study makes the following contributions to the field:  
1. We propose a method to construct bug code contexts and use the CodeBERT model to learn feature representations of these contexts. 2.