Venue: NAACL 2025

# TCProF:Time-Complexity Prediction SSL Framework

Joonghyuk Hahn, Hyeseon Ahn, Jungin Kim, Soohan Lim, Yo-Sub Han1 ,

Department of Computer Science, Yonsei University, Seoul, Republic of Korea,

{greghahn,hsan,jungin.kim,aness1219,emmous}@yonsei.ac.kr

Footnote 1: Corresponding author.

###### Abstract

Time complexity is a theoretic measure to determine the amount of time the algorithm needs for its execution. In reality, developers write algorithms into code snippets within limited resources, making the calculation of a code's time complexity a fundamental task. However, determining the precise time complexity of a code is theoretically undecidable. In response, recent advancements have leaned toward deploying datasets for code time complexity prediction and initiating preliminary experiments for this challenge. We investigate the challenge in low-resource scenarios where only a few labeled instances are given for training. Remarkably, we are the first to introduce TCProF: a Time- Complexity Prediction SSL Framework as an effective solution for code time complexity prediction in low-resource settings. TCProf significantly boosts performance by integrating our augmentation, symbolic modules, and a co-training mechanism, achieving a more than 60% improvement over self-training approaches. We further provide an extensive comparative analysis between TCProF, ChatGPT, and Gemini-Pro, offering a detailed evaluation of our approach.

## 1 Introduction

The task of predicting time complexity for code snippets represents a significant challenge in programming efficiency analysis. Time complexity is a crucial benchmark for evaluating algorithm performance across diverse computational domains. However, accurately computing the time complexity of a code snippet remains theoretically undecidable [3], presenting a substantial obstacle. This issue is particularly crucial in environments such as educational settings, programming competitions, and automated code reviews, where an accurate evaluation of numerous solutions is essential.

In the meantime, the advent of deep learning methodologies presents a promising avenue to address this challenge. Sikka et al. (2020) introduced CorCoD dataset, specifically designed for code time complexity prediction. The dataset consists of code snippets labeled with their time complexity classes. They also provide initial experiments for the code time complexity prediction using both conventional algorithms and basic neural models. Despite these advances [4, 5], the efficiency of such models heavily relies on the availability of extensively annotated datasets. Unfortunately, datasets in this domain are currently scarce due to the problem of data scarcity, as time complexity annotations require professional knowledge. While there are further approaches [4, 7] with considerable potential, their effectiveness is contingent upon the size of annotated datasets.

Addressing the shortage of labeled data, we introduce several innovations. We develop data augmentation techniques specifically tailored to identify key factors that influence the time complexity of the code snippets. We also incorporate a co-training mechanism that leverages both original and augmented data effectively. Additionally, we construct a symbolic module that enhances the accuracy of pseudo-labels compared to the pseudo-labels generated by a model-based approach alone. Collectively, these components form the backbone of TCProF1, our semi-supervised learning (SSL) framework, which is uniquely equipped to address code time complexity prediction in low-resource settings.

Footnote 1: [https://github.com/peer0/few-shot-tc](https://github.com/peer0/few-shot-tc)

Operating under the assumption of limited labeled data and a vast number of unlabeled code snippets, we empirically analyze TCProF using publicly available datasets, CorCoD and CodeComplex [4].



sippets categorized into five different complexity classes, \(O(1)\), \(O(\log N)\), \(O(N)\), \(O(N\log N)\), and \(O(N^{2})\). In contrast, CodeComplex consists of codes across seven different complexity classes, \(O(1)\), \(O(\log N)\), \(O(N)\), \(O(N\log N)\), \(O(N^{2})\), \(O(N^{3})\), and \(O(2^{N})\). For these benchmark datasets, our framework TCProf significantly enhances the performance over traditional self-training methods with improvements exceeding 60% in accuracy and F1-scores. In the era of ChatGPT, we further provide comparative studies against off-the-shelf large language models (LLMs). Meanwhile, given the early stage of research in this domain, we provide an in-depth analysis of TCProf offering valuable insights into their practical utility. Ultimately, our endeavor specifically targets the practical aspect of code time complexity prediction--the availability of annotated data. By providing a fundamental framework, TCProf, for code time complexity prediction in low-resource settings, we aim to lay the groundwork for future research in this domain.

## 2 Related Works

### SSL for Classification

Data scarcity is a critical problem for various tasks. More specifically, the problem occurs when there are only a few labeled data even though there are tons of unlabeled data. Generating labeled data is costly, making research in these low-resource environments crucial. Semi-supervised learning (SSL) offers an effective solution in such scenarios Sajjadi et al. (2016); Xie et al. (2020); Chen et al. (2020, 2022); Sohn et al. (2020); Zhang et al. (2021a); Wang et al. (2023a); Zou et al. (2023); Huang et al. (2023); Nie et al. (2024).

One of the popularly used SSL techniques, self-training is a learning mechanism that trains the student model with a few-shot labeled dataset Geng et al. (2019); Bao et al. (2020); Zhang et al. (2021b) and then subsequently acts as a teacher model for generating pseudo-labels Lee (2013). Pseudo-labels are judged based on the predictions of the model for a given unlabeled data. Co-training Blum and Mitchell (1998) is also the successful SSL mechanism that simultaneously employs two networks. Jointmatch Zou and Caragea (2023) utilizes cross-labeling, inspired by co-training, that uses an additional loss based on pseudo-labels for more reliability instead of augmenting pseudo-labels to the initial labeled dataset for additional training. While this approach has been effective to some degree, it lacks reliability as we cannot guarantee that the model generates 'correct' pseudo-labels. Recent approaches have incorporated symbolic modules to enhance the reliability of pseudo-label generations or data augmentation Hahn et al. (2021); Kim et al. (2022).

Likewise, data augmentation is also one of the fundamental methods that is effective in the low-resource setting. Data augmentation generates artificial data from the original dataset without changing their labels. Conventional data augmentations are synonym replacements, word insertion or deletion Wei and Zou (2019) More advanced methods involve Back-Translation Edunov et al. (2018) and these days, LLMs have gained popularity in generating various data but with accurate labels. We utilize these insights to develop TCProF, an SSL

Figure 1: The overall framework of TCProf.

 

framework for predicting code time complexity in low-resource environments.

### Code Time Complexity Prediction

Computation of code time complexity has long been a theoretically undecidable problem whereas classifying the code time complexity is a recently emerged problem. Sikka et al. (2020) first proposed this task and presented a labeled dataset with five time complexity classes. They propose a dataset named CorCoD, composed of Java codes with \(O(1)\), \(O(\log N)\), \(O(N)\), \(O(N\log N)\) and \(O(N^{2})\) time complexity classes and experimental results on the time complexity classification with baseline neural models.

Similar to CorCoD, CODAIT2 tried to create good code embeddings by capturing manual features such as number of loops and breaks, and utilized graph-based representations for predicting time complexities. Afterward, Moudgalya et al. (2023) proposed TasTy, consisting of Java and Python data, and Baik et al. (2024) proposed CodeComplex, which consists of also Java and Python data with additional labels. CodeComplex consists of seven different time complexity classes, \(O(N^{3})\) and \(O(2^{N})\) in addition to those of CorCoD.

Footnote 2: [https://community.ibm.com/community/user/ai-datascience/blogs/sepideh-eifardeh1/2021/10/05/ai-for-code-predict-code-complexity-using-ibms-cod](https://community.ibm.com/community/user/ai-datascience/blogs/sepideh-eifardeh1/2021/10/05/ai-for-code-predict-code-complexity-using-ibms-cod)

While time complexity prediction has been explored in these datasets, the scarcity of labeled data remains a significant challenge. Unlike runtime-based datasets from online judge platforms, which can be influenced by hardware, input distributions, and implementation-specific optimizations (Ishimwe et al., 2021; Zhang et al., 2023), these datasets provide explicit theoretical complexity labels for the code snippets. However, annotating such datasets requires expert knowledge, making them inherently low-resource. We propose TCProf, the SSL framework designed to alleviate this data scarcity challenge and enhance the accuracy of time complexity prediction.

## 3 Methodology

### Overview

TCProf is a robust SSL framework designed for code time complexity prediction in a low-resource setting illustrated in Fig. 1. TCProf comprises three primary components:

Augmentation module (AUG):This module employs augmentation involving loop representation conversion (\(LC\)) and back-translation conversion (\(BT\)), and a combined ensemble of these methods to enhance the diversity of augmented data.

Training mechanism:Utilizing a co-training approach, TCProf trains two models simultaneously; one with original data and the other with augmented data. This mitigates the error accumulation problem of self-training.

Pseudo-label module:Integrating our symbolic module (Sym) with the classifier, this module generates more precise pseudo-labels.

Algorithm 1 describes a detailed procedure of our framework.

``` Input: Data \(\mathcal{T}_{i}=\{t_{i}|~{}t_{i+1}<\cdots>\}\)do  Initialize \(L\leftarrow\{t_{i}|~{}t_{i+1}=(d,c)\}\)  Initialize \(L\gets L\cup\{u_{i}|~{}t_{i+1}=(d,c)\}\)  Initialize \(L\leftarrow\{t_{i}|~{}t_{i+1}

 

Each model \(B\) and \(B_{aug}\) is trained by \(L\) and \(L_{aug}\), respectively. Following the pseudo-label strategy from Sections 3.4 and 3.5, two models then generate pseudo-labels and update labeled dataset \(L^{\prime}\) and \(L^{\prime}_{aug}\), respectively:

\[L^{\prime}=L+ \{u~{}|~{}u=(d,c_{u})~{}\text{where}\] \[u~{}\text{is pseudo-labeled by}~{}B_{aug}\},\]

\[L^{\prime}_{aug}=L_{aug}+ \{u~{}|~{}u=(d,c_{u})~{}\text{where}\] \[u~{}\text{is pseudo-labeled by}~{}B\}.\]

### Model-based Pseudo-Labels

The conventional pseudo-labeling procedure is done by prediction of \(B\). The confidence score of unlabeled data with its corresponding class that \(B\) outputs is the one and only component for model-based pseudo-labels. We can adjust the threshold value for more precise pseudo-labels.

### Symbolic Pseudo-Labels

We introduce a computational module, designed to predict the temporal complexity of source code through symbolic analysis, employing the computational results as pseudo-labels to enhance the accuracy of predictions. We have named this module, the **symbolic module**\(\mathbb{Sym}\). The symbolic module \(\mathbb{Sym}\) is designed to predict the time complexity of code by identifying specific patterns and structural elements, without relying on neural networks. This approach effectively complements language models that, by their nature, do not account for the hierarchical organization inherent in programming code Allamanis et al. (2018); Chen et al. (2021); Zhang et al. (2022). The module primarily employs Regular Expressions (Regex) and Abstract Syntax Trees (ASTs) as essential tools for a detailed analysis of the source code. This aims to describe the time complexity of iterations, function calls, recursive and iterative calls, and additional relevant constructs, thereby facilitating a comprehensive understanding of the code's structure and execution flow. Our module outputs a formula consisting of the above components and regarding the size of inputs in the source code, subsequently aggregating these elements to categorize the time complexity class of the source code.

Our approach is systematically organized into five distinct phases and is illustrated in Figure 2:

Function Extraction:We employ Regex to identify and extract function definitions from source code. Each function forms the basic unit of analysis as an independent code block. If the source code does not contain functions, this step is omitted.

Identify Loops and Recursions:We utilize Regex to identify the presence of repetitive statements (such as for loops and while loops) and recursive functions to determine the frequency of loop iterations and function calls.

Detect Specific Time Complexity (TC):We utilize ASTs to detect operations that modify the input size. By identifying the presence of sorting or binary traversal, we classify source codes according to their time complexities such as \(O(\log N)\) and \(O(N\log N)\).

Calculate Total TC:The final time complexity is calculated by summing the complexity of each iteration and function call, based on the patterns identified using Regex, and the code structure analyzed by ASTs. In this step, we calculate the overall complexity, ensuring a robust analysis of recursive

Figure 2: A procedural illustration of our symbolic module \(\mathbb{Sym}\).

 

relationships within the source code.

**TC Classification:** The calculated time complexity is categorized into the pre-defined complexity classes. These assigned classes are utilized as pseudo-labels.

In Appendix H, we present Figure 9 as a running example of the symbolic pseudo-labeling.

### Merge Pseudo-Labeling

We have two pseudo-label modules based on model confidence and the symbolic module. Illustrated in Figure 1 and Algorithm 1, we first pseudo-label the unlabeled data by the model confidence. Then, we use the symbolic module to pseudo-label the unlabeled data that failed pseudo-labeling by the model confidence.

```
functionPseudo-label(\({\bf B}\), \(Synt\), \(\mathcal{U}\)) p Sections3.4, 3.5 \(\mathcal{U^{\prime}}\leftarrow\emptyset\) for each \(u=(d,\lambda)\in\mathcal{U}\)do \(c_{u}\leftarrow\arg\max_{e\in\mathcal{C}}{\rm B}(u,c)\) \(\mathcal{U^{\prime}}\gets U^{\prime}\cup\left\{(d,c_{u}):{\rm B}(u,c_{ u})\geq\theta\right\}\) \(\mathcal{U^{\prime}}\leftarrow\mathcal{U^{\prime}}\cup\left\{(d,Synt(d,u)):{ \rm B}(u,c_{u})<\theta\right\}\) endfor return\(\mathcal{U^{\prime}}\) endfunction procedureTimeComp(\(\mathcal{L}\), \(\mathcal{U}\), \({\bf B}\), \({\sf B}_{\text{aug}}\), \({\sf Sym}\), \({\sf AUG}\)) \(\mathcal{L}_{\text{ang}}\leftarrow\texttt{AUG}(\mathcal{L})\) & Section3.7 for\(e\gets 1\) to epoch_number do if self-train then\(\triangleright\) Section3.2 \(\mathcal{L}\leftarrow\mathcal{L}\cup\mathcal{L}_{\text{ang}}\) \(\mathbf{B}\leftarrow\text{fine-tune}({\bf B},\mathcal{L})\) \(\mathcal{U}_{\text{gl}}\leftarrow\texttt{Pseudo-label}({\bf B},{\sf Sym}, \mathcal{U})\) \(\mathcal{L}\leftarrow\mathcal{L}\cup\mathcal{U}_{\text{gl}}\) elseifco-trainthen\(\triangleright\) Section3.3 \(\boldsymbol{b}\leftarrow\text{fine-tune}({\bf B},\mathcal{L}_{\text{ang}})\) \(\mathcal{U}_{\text{gl}}\leftarrow\texttt{Pseudo-label}({\bf B},{\sf Sym}, \mathcal{U})\) \(\mathcal{U}_{\text{mepl}}\leftarrow\texttt{Pseudo-label}({\bf B}_{\text{mpg}},{\sf Sym },\mathcal{U})\) \(\mathcal{L}\leftarrow\mathcal{L}\cup\mathcal{U}_{\text{mepl}}\) \(\mathcal{L}_{\text{ang}}\leftarrow\mathcal{L}_{\text{ang}}\cup\mathcal{U}_{ \text{gl}}\) endif endfor endprocedure
```

**Algorithm 1** Procedure for the SSL framework \(\mathsf{TCPOrf}(\mathcal{L},\mathcal{U},{\bf B},{\bf B}_{\text{aug}}\), \({\sf Sym}\), \({\sf AUG}\)). Inputs include labeled dataset \(\mathcal{L}\), unlabeled dataset \(\mathcal{U}\), baseline model \({\bf B}\), co-training model \({\bf B}_{\text{aug}}\), symbolic module \({\sf Syn}\), and augmentation module AUG. The set of complexity classes is denoted as \(\mathcal{C}\).

### Data Augmentation

We introduce a data augmentation module AUG, designed to complement the lack of labeled data. Our augmentation strategies leverage the ChatGPT API, specifically employing the gpt-3.5-turbo-0125 model, to augment our experiments' CorCoD and CodeComplex datasets. The objective is to create precise augmentations that respect the intrinsic properties of the code, ensuring semantic integrity while introducing syntactic variability. We ensure that the augmented code snippets are free from syntactic errors, reinforcing their reliability for further analysis. We present two augmentation methods specifically designed to augment code snippets that preserve the original time complexity:

Back-Translation (BT):This method involves translating a code snippet into another programming language and then back to the original language to maintain its semantic essence. For instance, Java code snippets are translated into Python and then back into Java. This process gives syntactic variation while retaining the context.

Loop-Conversion (LC): Loop structures are the primary components that determine the time complexity of code snippets. This technique modifies the loop structures to different but semantically equivalent forms. Using regular expressions, we filter codes containing "for" or "while" loops and then convert these loops by employing For2While and While2For transformation rules, preserving the original logic of the code snippet. For instance, a while loop can be converted into a for loop (While2For) and vice versa (For2While), depending on the context. If the original code snippet contains both for and while loops, we leverage While2for and For2while respectively.

These augmentation methods are detailed in the prompts listed in Appendix K and are integrated into \(\mathsf{TCProF}\) to enhance the robustness. The augmented data from BT and LC methods supplement the initial labeled data in three distinct experimental configurations: 1) \(\mathtt{AUG}_{DT}\) incorporates back-translation augmented data, 2) \(\mathtt{AUG}_{LC}\) incorporates loop-conversion augmented data, and 3) \(\mathtt{AUG}_{BT+LC}\) combines both back-translation and loop-conversion augmented data.

There are two experimental setups for \(\mathtt{AUG}_{LC}\) as there are code snippets without any loop structures: 1) \(\mathtt{AUG}_{LC\_Natural}\) uses naturally sampled data as the initial labeled data without specific preconditions and 2) \(\mathtt{AUG}_{LC\_Artificial}\) selects initial labeled data specifically containing loop structures to maximize the use of augmented data.

These configurations for \(\mathtt{AUG}_{LC}\) are also applied to the combined augmentation strategy \(\mathtt{AUG}_{BT+LC}\), with corresponding \(\mathtt{AUG}_{BT+LC\_Natural}\) and \(\mathtt{AUG}_{BT+LC\_Artificial}\) settings. Table 2 employs \(\mathtt{AUG}_{BT+LC\_Natural}\) as 

the baseline for the AUG setup as it is more natural and common compared to \(\mathbb{AU}_{GT+LC-Artificial}\). An extensive analysis of augmentation strategies is provided in Appendix J.

### Implementation Details

We use UniXcoder Guo et al. (2022) and CodeT5+ Wang et al. (2023b) as our baselines for TCProf. We assign the batch of size 7, the number of epochs as 20, and set the learning rate of containing as 1e-5 and 2e-6. We use 1e-5 learning rate for self-training. Our confidence score threshold \(\theta\) is 0.7. We conduct experiments using the NVIDIA RTX 3090 for training TCProf.

## 4 Experimental Setup

We use CodeComplex Baik et al. (2024) and CorCoD Sikka et al. (2020) datasets. CodeComplex consists of 4,900 Java and 4,900 Python codes, and CorCoD consists of 929 Java codes. We follow Baik et al. (2024) to split train and test data. For low-resource settings, we perform 5- and 10-shot experiments where we pick 5 and 10 data for each label from the train dataset and use the remaining train dataset as an unlabeled dataset for pseudolabels, respectively.

Code time complexity prediction is a recent code-related task and we are the first to perform the task in low-resource scenarios. We use four pre-trained code language models, CodeBERT Feng et al. (2020), GraphCodeBERT Guo et al. (2021), UniXcoder Guo et al. (2022), and CodeT5+ Wang et al. (2023b) as our baseline models.

We also include JointMatch as an SSL baseline for comparison. Recently, Zou and Caragea (2023) proposed JointMatch for a state-of-the-art SSL framework for low-resource settings, where the authors employ cross-labeling to enhance pseudolabeling. While JointMatch is effective for text classification tasks in low-resource settings, we analyze whether it is effective for code time complexity prediction in low-resource settings and compare the performance with our framework TCProf.

## 5 Results and Analysis

We present our main results in Table 1, indicating that our model outperforms baselines. In Section 5.1, we suggest possible intuitions that can be derived from the performance result of Table 1. We demonstrate ablation studies in Section 5.2 and provide error analyses in Section 5.3. Then, for the extensive analyses of TCProf, we show how TCProF approaches the fully fine-tuned models in Section 5.4 and provide comparisons with commercial LLMs in Section 5.5.

### Comparison with Baselines

Table 1 presents the Self-Training (ST) results for the base models, UniXcoder and CodeT5+. The table illustrates that the average accuracy and standard deviation scores for ST of base models, which serve as baselines, are generally low. Specifically, on the CodeComplex datasets, the accuracy scores of these baselines are below 40%, except for ST (UniXcoder) on CodeComplex (Python). Furthermore, the high standard deviation scores indicate that the performance of these models is unstable and unreliable.

Our augmentation, symbolic pseudo-labeling, and co-training strategies mitigate this problem. We pick UniXcoder and CodeT5+ as the baseline model to implement our strategies3. TCProf out

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{2}{c}{CodeComplex (Java)} & \multicolumn{2}{c}{CodeComplex (Python)} & \multicolumn{2}{c}{CorCoD} \\  & 5 & 10 & 5 & 10 & 5 & 10 \\ \hline SSL Baselines & & & & & & \\ \hline ST (UniXcoder) & 15.44\(\pm\) 6.30 & 31.77\(\pm\) 0.81 & 26.02\(\pm\)11.72 & 40.98\(\pm\) 11.14 & 37.89\(\pm\) 6.57 & 45.61\(\pm\)11.59 \\ ST (CodeT5+) & 18.87\(\pm\) 3.21 & 31.69\(\pm\)14.00 & 28.76\(\pm\)17.47 & 38.32\(\pm\)18.82 & 35.79\(\pm\) 6.90 & 45.26\(\pm\) 6.32 \\ JointMatch (CodeT5+) & 14.62\(\pm\) 4.26 & 24.68\(\pm\) 2.60 & 20.97\(\pm\) 3.90 & 21.04\(\pm\) 4.33 & 36.49\(\pm\) 4.98 & 42.11\(\pm\) 6.57 \\ JointMatch (UniXcoder) & 9.62\(\pm\) 5.23 & 19.39\(\pm\) 3.36 & 14.68\(\pm\)11.78 & 20.76\(\pm\)10.46 & 35.44\(\pm\) 6.85 & 48.42\(\pm\) 8.22 \\ \hline \hline TCProf(CodeT5+) & 38.63\(\pm\) 1.32 & 41.98\(\pm\) 2.92 & 44.74\(\pm\) 3.26 & 59.29\(\pm\) 3.71 & 50.53\(\pm\) 0.86 & 51.93\(\pm\) 3.88 \\ TCProf(UniXcoder) & **52.50\(\pm\) 1.56** & **53.85\(\pm\) 3.63** & **54.64\(\pm\)**3.37 & **70.29\(\pm\) 2.06 & **55.44\(\pm\)**0.99 & **63.16\(\pm\)**2.27 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracy performance of SSL baselines and TCProf. ST refers to Self-Training. TCProf(CodetT5+) represents TCProf implemented to the baseline CodeT5+ and vice versa to UniXcoder. The scores are averaged from three runs with different seeds. We report the full result in Table 5.

 

performs the baselines illustrated in Table 1. Especially, TCProf(UniXcoder) accomplishes 64.81% improvements on average compared to the best performance of baselines. We discuss the analysis of our strategies in Section 5.2 as ablation studies.

Footnote 3: [https://github.com/tensorflow-learning/tree/mastermaster/master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_mastermaster_master_master_master_mastermaster_master_master_mastermaster_master_mastermaster_master_mastermaster_master_mastermaster_mastermaster_master_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermastermaster_mastermaster_mastermastermaster_mastermastermaster_mastermastermaster_mastermastermastermaster_master](https://github.com/tensorflow-learning/tree/mastermaster/master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_master_mastermaster_master_master_master_mastermaster_master_master_mastermaster_master_mastermaster_master_mastermaster_master_mastermaster_mastermaster_master_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermaster_mastermastermaster_mastermaster_mastermastermaster_mastermastermaster_mastermastermaster_mastermastermastermaster_master) 

formance both in accuracy and standard deviation. We also provide detailed analysis on F1-scores in Appendix A.

### Error Analysis

Furthermore, we examine the types of errors to which our method is particularly vulnerable for extensive analysis. Figure 3 illustrates that our approach is especially weak on \(O(N^{2})\) class. We notice that the codes with this class are mostly predicted as \(O(N)\). Codes in both classes involve loops and usually the only difference is the depth of the loops. However, precisely, the core factor that makes the difference is the loops involving the input length. For instance, codes in \(O(N)\) class can contain multiple-depth loop statements where only a single loop statement involves the length of the input. The model needs to precisely determine whether the iteration number of each loop is proportional to the input size for the correct prediction. Likewise, the class \(O(NlogN)\) involving both loops linear and logarithmic to the input size and the class \(O(2^{N})\) involving exponential iterations are also relatively erroneous compared to the other classes. We present examples of code instances, demonstrating the errors in Appendix I.

### Comparison to the full-train result

TCProF demonstrates a promising result in a few-shot settings, improving the baselines on a large scale. We conduct further experiments on comparing the best 10-shot performance of TCProF to the baselines trained with the whole train datasets in Table 3. For CodeComplex datasets, TCProF(UniXcoder) exceeds the accuracy of all the baselines except for CodeT5+ and the performance gap with CodeT5+ is small. Take CodeComplex (Python) for instance, TCProF(UniXcoder) achieves 70.29% accuracy and the accuracy of CodeT5+ fine-tuned with the full train dataset is 72.88%. Our approach catches up to 96.45%.

Appendix C presents the full results of accuracy and F1 performance in Table 7. Overall to all the datasets, TCProF catches up to 91% and 94% for accuracy and F1-scores, respectively, compared to the base models fine-tuned with the full train dataset. This indicates that TCProF, a framework designed for low-resource settings, is highly effective in achieving competitive performance even with limited labeled data. By leveraging co-training, symbolic pseudo-labels, and data augmentation, TCProf enhances model generalization and robustness, making it a viable alternative to fully supervised approaches, particularly in scenarios where the amount of labeled data is scarce or costly to obtain. We also present Figures 6(a) and 6(b) in the appendix for clear visualization of the results.

### Comparison with LLMs

LLMs are known to be effective in general NLP tasks and we extend the analysis into comparing our performance with LLMs. We evaluate the LLMs in

Figure 3: Confusion matrix of ours for CodeComplex (Python) 10-shot.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & \multicolumn{2}{c}{CodeComplex} & CorCoD \\  & Java & Python & \\ \hline \hline  & Acc. & Acc. & Acc. \\ \hline Gemini-pro & 49.54 & 31.05 & 61.91 \\ GPT-3.5 & 62.15 & 32.55 & 69.42 \\ GPT-4 & **64.01** & 53.04 & **78.86** \\ \hline TCroF(UniXcoder) & 53.85 & **70.29** & 63.16 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison with the performance of LLM for Java and Python datasets of CodeComplex, and CorCod dataset.

 

a 5-shot in-context learning setting by the prompt demonstrated in Appendix L. Our baseline LLMs are Gemini-pro (Anil and et al., 2023), GPT3.5, and GPT-4 (OpenAI, 2023). The results of Table 4 indicates that TCProF shows competitive performance to these commercial LLMs. These LLMs perform much better than CodeT5+ and UniXcoder fine-tuned with the full train dataset for CodeComplex (Java) and CorCoD referring to Table 3. As TCProf takes CodeT5+ and UniXcoder as the baseline models, competitive performance of TCProf compared to LLMs is remarkable.

Furthermore, TCProf performs better than any of these LLMs for CodeComplex (Python). In Appendix F, we also present Table 9 that includes both the accuracy and F1 performance of LLMs.

## 6 Conclusion

Code time complexity prediction remains a largely unexplored yet critical task, and our framework, TCProf, is the first attempt to tackle this challenge within low-resource environments. We have developed TCProf as a robust SSL framework for predicting code time complexity, showing through comprehensive analyses that it significantly improves upon the performance of baseline models. As we reflect on the capabilities of TCProf, it is important to recognize that our current focus has been on establishing an effective framework that performs well in constrained environments. Looking ahead, expanding the generalization capabilities of TCProf to accommodate a wider range of programming languages is a vital next step. We believe TCProf will serve as a significant milestone, propelling forward the research in code time complexity prediction, especially in low-resource settings.

## Limitation

Application ScopeTCProf is specifically designed for few-shot settings and thus, its application would not be effective in fine-tuning with the full dataset. One possible exploration is employing the augmentation module of TCProf for fine-tuning the full dataset. We provide the analyses in Appendix J. Another limitation of TCProf is the space complexity prediction. This is also due to the absence of datasets involving the space complexity. Thus, we will be happy to apply and develop our framework for the space complexity prediction when the datasets are released.

Framework Adaptability to Zero-ShotsAs our framework targets few-shot settings where 'several' data instances for each class are provided, We do not provide initial configuration for zero-shot settings. However, we can adapt the zero-shot setting by altering the structure of TCProf to incorporate the symbolic module for generating initial training data with pseudo-labels. It is essential to clarify that our objective is not zero-shot learning but few-shot learning.

Dynamic CalculationTCProf is focused on classifying time complexity into seven discrete classes, rather than dynamically calculating time complexities across a continuous range. We are eager to develop further modules and implement generative models for targeting the time complexity 'calculation', extending from the time complexity 'prediction'.

## Acknowledgment

This research was supported by the NRF grant (RS-2023-00208094) and the AI Graduate School Program (RS-2020-II201361) funded by the Korean government (MSIT).

## References

* Allamanis et al. (2018) Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs. In _6th International Conference on Learning Representations, ICLR_.
* Anil and others (2023) Rohan Anil and et al. 2023. Gemini: A family of highly capable multimodal models. _preprint_, arXiv: 2312.11805.
* Asperti (2008) Andrea Asperti. 2008. The intensional content of Rice's theorem. In _Proceedings of the 35th ACM SIGPLANSIGACT Symposium on Principles of Programming Languages, POPL_, pages 113-119.
* Baik et al. (2024) Seung-Yeop Baik, Joonghyuk Hahn, Jungin Kim, Aditi, Mingi Jeon, Yo-Sub Han, and Sang-Ki Ko. 2024. CodeComplex: Dataset for worst-case time complexity prediction. _preprint_, arXiv: 2401.08719.
* Bao et al. (2020) Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2020. Few-shot text classification with distributional signatures. In _8th International Conference on Learning Representations, ICLR_.
* Blum and Mitchell (1998) Avrim Blum and Tom M. Mitchell. 1998. Combining labeled and unlabeled data with co-training. In _Proceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT_, pages 92 

Hui Chen, Wei Han, and Soujanya Poria. 2022. SAT: improving semi-supervised text classification with simple instance-adaptive self-training. In _Findings of the Association for Computational Linguistics: EMNLP_, pages 6141-6146.
* Chen et al. (2020) Jiao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL_, pages 2147-2157.
* Chen et al. (2021) Zimin Chen, Vincent J. Hellendoorn, Pascal Lamblin, Petros Maniatis, Pierre-Antoine Manzagol, Daniel Tarlow, and Subhodeep Moitra. 2021. PLUR: A unifying, graph-based view of program learning, understanding, and repair. In _Advances in Neural Information Processing Systems 34_, pages 23089-23101.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP_, pages 489-500.
* Feng et al. (2020) Zhangynin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In _Findings of the Association for Computational Linguistics: EMNLP_, pages 1536-1547.
* Geng et al. (2019) Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP_, pages 3902-3911.
* Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified crossmodal pre-training for code representation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL_, pages 7212-7225.
* Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Syvatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training code representations with data flow. In _9th International Conference on Learning Representations, ICLR_.
* Hahn et al. (2021) Joonghyuk Hahn, Hyunjeon Cheon, Kyuyeol Han, Cheongjae Lee, Junseok Kim, and Yo-Sub Han. 2021. Self-training using rules of grammar for few-shot NLU. In _Findings of the Association for Computational Linguistics: EMNLP_, pages 4576-4581.
* Huang et al. (2023) Zhuo Huang, Li Shen, Jun Yu, Bo Han, and Tongliang Liu. 2023. FlatMatch: Bridging labeled data and unlabeled data with cross-sharpness for semi-supervised learning. In _Advances in Neural Information Processing Systems 36_.
* Ishimwe et al. (2021) Didier Ishimwe, Kim Hao Nguyen, and ThanhVu Nguyen. 2021. Dynaplex: analyzing program complexity using dynamically inferred recurrence relations. _Proceedings of the ACM on Programming Languages_, 5(OOPSLA):1-23.
* Kim et al. (2022) Hazel H. Kim, Daechoel Woo, Seong Jeon Oh, Jeong-Won Cha, and Yo-Sub Han. 2022. ALP: data augmentation using lexicalized pcfgs for few-shot text classification. In _Thirty-Sixth AAAI Conference on Artificial Intelligence_, pages 10894-10902.
* Lee (2013) Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In _Workshop on challenges in representation learning, ICML_, page 896.
* Moodalya et al. (2023) Kaushik Moodalya, Ankit Ramakrishnan, Vamsikrishna Chemudupati, and Xing Han Lu. 2023. TASTY: A transformer based approach to space and time complexity. _preprint_, arXiv: 2305.05379.
* Nie et al. (2024) Wenhua Nie, Lin Deng, Chang-Bo Liu, Jialing Wei, Ruutong Han, and Haoran Zheng. 2024. STSPL-SSC: semi-supervised few-shot short text clustering with semantic text similarity optimized pseudo-labels. In _Findings of the Association for Computational Linguistics, ACL_, pages 12174-12185.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _preprint_, arXiv: 2303.08774.
* Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. _Advances in Neural Information Processing Systems 29_, 29.
* 42nd European Conference on IR Research, ECIR Proceedings, Part I_, volume 12035, pages 313-325.
* Sohn et al. (2020) Kihyku Sohn, David Berthelot, Nicholas Carlini, Zihao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. _Advances in Neural Information Processing Systems 33_, 33:596-608.
* Wang et al. (2023) Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. 2023a. Freematch: Self-adaptive thresholding for semi-supervised learning. In _The Eleventh International Conference on Learning Representations, ICLR_ 

Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023b. CodeT5+: Open code large language models for code understanding and generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP_, pages 1069-1088.
* Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP_, pages 6381-6387.
* Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. _Advances in Neural Information Processing Systems__33_, 33:6256-6268.
* Zhang et al. (2021a) Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. 2021a. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. _Advances in Neural Information Processing Systems__34_, 34:18408-18419.
* 18th International Conference, ICCSE, Proceedings, Part I_, volume 2023 of _Communications in Computer and Information Science_, pages 265-279.
* Zhang et al. (2021b) Jian-Guo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip S. Yu. 2021b. Few-shot intent detection via contrastive pre-training and fine-tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP_, pages 1906-1912.
* Zhang et al. (2022) Kechi Zhang, Wenhan Wang, HuangZhao Zhang, Ge Li, and Zhi Jin. 2022. Learning to represent programs with heterogeneous graphs. In _Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, ICPC_, pages 378-389.
* Zou and Caragea (2023) Henry Peng Zou and Cornelia Caragea. 2023. JointMatch: A unified approach for diverse and collaborative pseudo-labeling to semi-supervised text classification. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP_, pages 7290-7301.
* Zou et al. (2023) Henry Peng Zou, Cornelia Caragea, Yue Zhou, and Doina Caragea. 2023. Semi-supervised few-shot learning for fine-grained disaster tweet classification. In _Proceedings of the 20th International ISCRAM Conference_ 

TCProF Full Result

TCProf consists of three core parts. The first is a symbolic module \({\tt Sym}\) for pseudo-labeling. The second is a data augmentation module AUG which employs Back-Translation (BT), Loop-Conversion (LC), and both (BT+LC). Finally, the third one is a co-training module. Our hypothesis is that the more components implemented, the better the performance and eventually, TCProF, with all modules implemented, performs the best. The result of ST(CodeBERT) and ST(GraphCodeBERT) were omitted in Table 1, and the result of \({\tt AUG}_{BT}\) and\({\tt AUG}_{LC}\) were omitted in Table 2. We present the full experimental results regarding all the three components of TCProf in Tables 5 and 6, each for the accuracy and F1 performance.

Both results are consistent with our hypothesis. Comparing the performance of \({\tt AUG}_{BT}\) and \({\tt AUG}_{LC}\) with \({\tt AUG}_{BT+LC}\), the tendency of performance scores demonstrates that using both augmentation methods is superior to using the single method. Additional implementation of \({\tt Sym}\) also improves the scores. It is

Figure 4: Confusion matrices of 10-shot TCProf(UniXcoder) performance.

 

notable that some performance scores of \(\mathsf{AU}_{BT+LC}\) with high standard deviation scores show lesser standard deviation scores when implemented with \(\mathtt{Sym}\). This indicates that the usage of symbolic pseudo-labels and augmentation complements each other and enhances the model performance. Finally, with all modules and co-training strategy implemented, TCProf, performs the best.

It is remarkable that our hypothesis holds on both the accuracy and F1 performance of TCProf. Intriguing facts are that while the SSL baselines tend to have higher standard deviation scores for F1 performance than accuracy, TCProf generally shows less standard deviation scores for F1 performance. This is because F1-scores evaluate how well the model performs for 'all' classes while accuracy does not consider whether the model is biased on specific classes. We provide confusion matrices of 10-shot TCProf(UniXcoder) for CodeComplex (Java), CodeComplex (Python), and CorCoD in Figures 3(a), 3(b), and 3(c), respectively. We can see that TCProf produces unbiased performance compared to those seen in Figures 5(b) and 4(b).

## Appendix B Baseline Selection for Self-Training

From the four well-known code language models, CodeBERT, GraphCodeBERT, UniXcoder, and CodeT5+, we experimented with which baseline models would fit for semi-supervised learning (SSL) in low-resource settings. Generally, UniXcoder and CodeT5+ are better than CodeBERT and GraphCodeBERT. We can also see the same tendency comparing the performance of each model trained with the full data in Table 8. Additionally analyzing for low-resource settings, we selected two best-performing models, UniXcoder and CodeT5+ to implement TCProf.

CodeBERT and GraphCodeBERT have satisfactory accuracy for 5-shot CodeComplex (Java) and CorCoD compared to the other two models as shown in Table 5. However, Table 6 reveals that CodeBERT and GraphCodeBERT perform significantly lower than that of the other two models. The large performance gap of CodeBERT and GraphCodeBERT between Tables 5 and 6 and the low F1 performance of these two models indicate that they are unsuitable to implement our methodology.

\begin{table}
\begin{tabular}{l c c c c c c} \hline  & \multicolumn{2}{c}{CodeComplex (Java)} & \multicolumn{2}{c}{CodeComplex (Python)} & \multicolumn{2}{c}{CorCoD} \\  & 5 & 10 & 5 & 10 & 5 & 10 \\ \hline SSL Baselines & & & & & & \\ \hline ST(CodeBERT) & \(5.31\pm 0.00\) & \(8.89\pm 6.20\) & \(4.27\pm 0.51\) & \(4.50\pm 0.90\) & \(10.54\pm 0.00\) & \(10.54\pm 0.00\) \\ ST(GraphCodeBERT) & \(5.52\pm 0.29\) & \(12.33\pm 11.97\) & \(3.98\pm 0.00\) & \(9.89\pm 10.24\) & \(12.82\pm 1.394\) & \(10.54\pm 0.00\) \\ ST(CodeT5+) & \(18.94\pm 3.69\) & \(29.20\pm 10.37\) & \(23.74\pm 12.26\) & \(28.48\pm 11.05\) & \(26.87\pm 17.65\) & \(45.17\pm 7.10\) \\ ST(UniXcoder) & \(13.52\pm 6.65\) & \(34.91\pm 5.73\) & \(21.61\pm 9.43\) & \(34.57\pm 0.80\) & \(38.92\pm 7.24\) & \(51.16\pm 8.61\) \\ JointMatch (CodeT5+) & \(7.43\pm 1.26\) & \(25.24\pm 1.51\) & \(13.61\pm 6.05\) & \(14.46\pm 3.18\) & \(30.31\pm 8.01\) & \(45.86\pm 7.78\) \\ JointMatch (UniXcoder) & \(9.70\pm 5.41\) & \(10.38\pm 5.20\) & \(7.87\pm 8.82\) & \(13.02\pm 8.85\) & \(18.98\pm 7.43\) & \(39.44\pm 11.93\) \\ \hline \end{tabular} 
\begin{tabular}{l l c c c c} \hline \hline  & \multicolumn{4}{c}{CodeT5+} \\ \cline{2-5} + \(\mathsf{AUG}_{BT}\) & \(22.90\pm 1.60\) & \(27.31\pm 3.13\) & \(31.94\pm 9.17\) & \(35.83\pm 4.52\) & \(35.80\pm 10.98\) & \(43.85\pm 2.99\) \\   + \(\mathsf{AUG}_{LC}\) & \(25.33\pm 3.56\) & \(26.42\pm 4.02\) & \(26.60\pm 3.77\) & \(42.50\pm 8.51\) & \(39.36\pm 6.15\) & \(46.76\pm 6.72\) \\   + \(\mathsf{AUG}_{BT+LC}\) & \(26.23\pm 2.07\) & \(30.69\pm 8.08\) & \(31.12\pm 5.61\) & \(42.38\pm 1.11\) & \(41.78\pm 5.12\) & \(47.98\pm 4.70\) \\   + \(\mathtt{Sym}\) & \(19.54\pm 1.76\) & \(30.52\pm 10.61\) & \(23.18\pm 2.75\) & \(34.69\pm 3.51\) & \(39.86\pm 1.77\) & \(46.66\pm 4.20\) \\   + \(\mathtt{Sym}+\mathsf{AUG}_{BT+LC}\) & \(29.11\pm 1.64\) & \(32.72\pm 1.48\) & \(31.99\pm 1.64\) & \(42.95\pm 10.01\) & \(42.36\pm 8.08\) & \(48.09\pm 1.78\) \\   **TCProf**(CodeT5+) & \(28.58\pm 0.37\) & \(33.60\pm 0.98\) & \(34.49\pm 4.55\) & \(44.03\pm 2.74\) & \(45.13\pm 5.27\) & \(49.60\pm 8.12\) \\ \hline \hline  & \multicolumn{4}{c}{UnixCoder} \\ \hline + \(\mathsf{AUG}_{BT}\) & \(29.18\pm 3.95\) & \(33.52\pm 1003\) & \(31.43\pm 5.98\) & \(39.24\pm 3.60\) & \(38.82\pm 9.97\) & \(52.37\pm 8.00\) \\   + \(\mathsf{AUG}_{LC}\) & \(27.24\pm 2.77\) & \(36.26\pm 16.28\) & \(29.48\pm 3.87\) & \(34.82\pm 4.26\) & \(40.33\pm 8.51\) & \(48.20\pm 13.78\) \\   + \(\mathsf{AUG}_{BT+LC}\) & \(31.02\pm 14.22\) & \(34.21\pm 10.92\) & \(33.53\pm 5.73\) & \(45.95\pm 8.69\) & \(43.42\pm 1.29\) & \(52.57\pm 5.91\) \\   + \(\mathsf{Sym}\) & \(31.88\pm 1.28\) & \(44.74\pm 2.10\) & \(24.50\pm 3.48\) & \(40.77\pm 2.56\) & \(48.77\pm 6.32\) & \(54.12\pm 6.47\) \\   + \(\mathsf{Sym}+\mathsf{AUG}_{BT+LC}\) & \(39.66\pm 0.44\) & \(45.31\pm 4.48\) & \(37.64\pm 4.90\) & \(51.58\pm 1.61\) & \(47.45\pm 4.93\) & \(56.87\pm 10.95\) \\   **TCProf**(UniXcoder) & \(\mathbf{42.89\pm 3.09}\) & \(\mathbf{49.45\pm 0.97}\) & \(\mathbf{38.15\pm 4.09}\) & \(\mathbf{53.17\pm 2.13}\) & \(\mathbf{56.56\pm 3.52}\) & \(\mathbf{63.57\pm 3.09}\) \\ \hline \end{tabular}
\end{table}
Table 6: F1-score performance comparisons. The scores are averaged from three runs with different seeds.

 

The relatively high performance of UniXcoder and CodeT5+ in Tables 5 and 6 empirically prove that they are suitable for low-resource settings. From confusion matrices of the two models in Figures 4(b) and 3(b), we can also easily see that UniXcoder and CodeT5+ do not overfit to a certain class. Furthermore, we depict the learning processes of these two models in Figures 4(a) and 3(a). In the figures, acc_train, acc_val, acc_test, and acc_psl indicate the accuracy of the train, valid, test, and pseudo-labeled datasets, respectively. These figures demonstrate that the converging patterns of acc_train, acc_val, acc_test are similar, indicating that the two models are both suitable for training in low-resource settings. From these two models, we also analyze which model is more suitable for low-resource settings, and eventually TCProF.

In Figure 4(a), acc_psl is not shown in the graph, meaning that CodeT5+ does not produce pseudo-labels in the self-training at all. This is because pseudo-labels are produced based on the confidence score and the confidence score of CodeT5+ does not match the pre-defined threshold. It is quite surprising as CodeT5+ has high acc_train. We have run an experiment of training CodeT5+ continuously to inspect when CodeT5+ generates pseudo-labels and the model produced biased pseudo-labels for only a few classes for CodeComplex (Python). Thus, independent to the performance, CodeT5+ is not suitable for pseudo-labeling when implemented with only self-training.

UniXcoder, on the other hand, surely generates pseudo-labels referring to Figure 5(a). For all datasets, UniXcoder generates pseudo-labels for all 7 classes, indicating that the model shows consistent per

Figure 5: CodeT5+ self-learning performance visualization for CodeComplex (Java) 10-shot.

Figure 6: UniXcoder self-learning performance visualization for CodeComplex (Java) 10-shot.

 

formance without being biased toward a specific class. Figure 5(b) also confirms that UniXcoder shows relatively well-distributed performance for each class in test data. This confirms that UniXcoder is more suitable that CodeT5+ in a low-resource setting.

Referring to Table 1 in Section 5.1, we can see that our proposed framework TCProf effectively improves the performance of both UniXcoder and CodeT5+ but we can also notice that UniXcoder is more effective for TCProf and eventually, low-resource settings.

## Appendix C Comparison to Baselines Fine-Tuned with Full Dataset

We have provided a comparative analysis of TCProf and baselines fine-tuned with the entire train dataset in Section 5.4. Detailed results for accuracy and F1-scores are shown in Table 7 and Figures 6(a) and 6(b) provide full results of accuracy and F1 performance.

We can see that TCProf effectively catches up with both the performance of accuracy and F1-scores. It is remarkable TCProf even exceeds the F1 performance of CodeT5+ for CodeComplex (Java), accomplishing the best F1-scores for the dataset.

## Appendix D Baselines Fine-Tuned with the Augmented Full Dataset

We have implemented AU\({}_{BT+LC}\), which combines Back-Translation (BT) and Loop-Conversion (LC) to investigate the impact of data augmentation on the full dataset. The results in Table 8 demonstrate the effectiveness of augmentation across the baselines and datasets. Notably, CodeT5+ shows a significant enhancement, particularly in Python, where it achieves over 90% accuracy and 85% F1-scores. This improvement is consistent across different models, indicating that AU\({}_{BT+LC}\) is beneficial in predicting time complexity more accurately and reliably.

\begin{table}
\begin{tabular}{l l l l l l} \hline  & \multicolumn{2}{c}{CodeComplex (Java)} & \multicolumn{2}{c}{CodeComplex (Python)} & \multicolumn{2}{c}{CorCoD} \\  & Acc. & F1 & Acc. & F1 & Acc. & F1 \\ \hline \multicolumn{3}{l}{Baselines trained with the Full train dataset} \\ \hline CodeBERT & 39.82\(\pm\) 6.90 & 37.35\(\pm\) 3.56 & 66.39\(\pm\) 1.43 & 54.34\(\pm\) 0.93 & 74.74\(\pm\) 1.72 & 76.64\(\pm\) 3.07 \\ GraphCodeBERT & 46.46\(\pm\) 2.18 & 37.75\(\pm\) 1.18 & 67.83\(\pm\) 3.86 & 54.13\(\pm\) 7.25 & 72.98\(\pm\) 1.79 & 76.48\(\pm\) 2.26 \\ UniXcoder & 46.76\(\pm\) 1.91 & 38.76\(\pm\) 0.39 & 68.44\(\pm\) 4.66 & 55.45\(\pm\) 3.33 & **77.54\(\pm\)**1.31 & **81.69\(\pm\)**1.43 \\ CodeT5+ & **55.11\(\pm\)**2.18 & 44.14\(\pm\) 3.54 & **72.88\(\pm\)**1.95 & **56.39\(\pm\)**2.25 & 75.79\(\pm\) 2.27 & 79.51\(\pm\) 1.96 \\ \hline \multicolumn{3}{l}{10-shot performance of TCProf} \\ \hline TCProf(CodeT5+) & 41.98\(\pm\) 2.92 & 33.60 \(\pm\) 0.98 & 59.29\(\pm\)3.71 & 44.03\(\pm\)2.74 & 51.93\(\pm\) 3.88 & 49.60\(\pm\) 8.12 \\ TCProf(UniXcoder) & 53.85\(\pm\) 3.63 & **49.45\(\pm\)**0.97 & 70.29\(\pm\)2.06 & 53.17\(\pm\)2.13 & 63.16\(\pm\) 2.27 & 63.57\(\pm\) 3.09 \\ \hline \end{tabular}
\end{table}
Table 7: Performance comparison of baselines trained with the full-train dataset and TCProf 10-shot performance.

Figure 7: Visualized performance of baselines trained with full train dataset and TCProf 10-shot.

 



## Appendix E Comparison to JointMatch

JointMatch is a recognized SSL approach. It displays the state-of-the-art performance for text classification in low-resource settings. However, we found this unsuitable for the code time complexity prediction task. Tables 5 and 6 illustrates that JointMatch underperforms compared to the standard self-training approaches. Confusion matrices in Figures (a)a and (b)b provide further details that JointMatch is either biased on several classes or is underfitted. This is even more remarkable as Figures (b)b and (b)b displays much better performance. This poor performance of JointMatch is that the model is primarily developed for text classification datasets such as AG News, Yahoo! Answers, and IMDB. These datasets are completely different from the code time complexity prediction datasets and thus, it is reasonable that the learning mechanism of JointMatch is not effective on the time complexity prediction.

## Appendix F LLM Comparison

Table 9 illustrates the accuracy and F1 performance of LLMs and TCProf. Likewise to the analysis from Section 5.5, TCProf demonstrates performance competitive to the LLMs for F1-scores as well.

Figure 8: CodeT5+, UniXcoder JointMatch performance visualization for CodeComplex (Java) 10-shot.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{CodeComplex (Java)} & \multicolumn{2}{c}{CodeComplex (Python)} & \multicolumn{2}{c}{CorCoD} \\  & Acc. & F1 & Acc. & F1 & Acc. & F1 \\ \hline Original & & & & & \\ \hline CodeBERT & 39.82\(\pm\) 6.90 & 37.35\(\pm\) 3.36 & 66.39\(\pm\) 1.43 & 54.34\(\pm\) 0.93 & 74.74\(\pm\) 1.72 & 76.64\(\pm\) 3.07 \\ GraphCodeBERT & 46.46\(\pm\) 2.18 & 37.75\(\pm\) 1.18 & 67.83\(\pm\) 8.36 & 54.13\(\pm\) 7.25 & 72.98\(\pm\) 1.79 & 76.48\(\pm\) 2.26 \\ UniXcoder & 46.76\(\pm\) 1.91 & 38.76\(\pm\) 0.39 & 68.44\(\pm\) 4.66 & 55.45\(\pm\) 3.33 & 77.54\(\pm\) 1.31 & 81.69\(\pm\) 1.43 \\ CodeT5+ & 55.11\(\pm\) 2.18 & 44.14\(\pm\) 3.54 & 72.88\(\pm\) 1.95 & 56.39\(\pm\) 2.75 & 75.79\(\pm\) 2.27 & 79.51\(\pm\) 1.96 \\ \hline \hline Augmented & & & & & \\ \hline CodeBERT & 50.41\(\pm\) 2.84 & 42.11\(\pm\) 1.99 & 81.83\(\pm\) 1.79 & 69.20\(\pm\) 4.07 & 78.60\(\pm\) 0.99 & 80.06\(\pm\) 0.69 \\ GraphCodeBERT & 58.69\(\pm\) 2.89 & 46.96\(\pm\) 2.25 & 85.31\(\pm\) 0.99 & 77.82\(\pm\) 0.62 & 78.60\(\pm\) 0.99 & 81.08\(\pm\) 0.80 \\ UniXcoder & **63.91\(\pm\)**1.28 & **49.07\(\pm\)**1.18 & 86.13\(\pm\)**1.58 & 77.47\(\pm\) 2.80 & 78.95\(\pm\) 0.00 & 82.22\(\pm\) 0.50 \\ CodeT5+ & 59.51 \(\pm\)0.80 & 48.79\(\pm\) 1.38 & **90.91\(\pm\)**2.25 & **85.67\(\pm\)**6.11 & **79.65\(\pm\)**0.50 & **82.65\(\pm\)**0.62 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Performance of baselines trained with augmented train dataset. The scores are averaged from three runs with different seeds.





## Appendix G Usage of \(\mathtt{Sym}\) as a Code Time Complexity Classifier

Our symbolic module, \(\mathtt{Sym}\), is specifically designed to enhance the pseudo-labeling process within the TCProF, aiming to improve the code time complexity prediction accuracy. This module leverages symbolic reasoning to generate more reliable pseudo-labels, empirically proven in Tables 1 and 2. For an extensive analysis, we present the experimental results in Table 10, using Sym alone to classify the time complexity of given code snippets.

Notably, \(\mathtt{Sym}\) shows relatively better performance on Java datasets compared to Python datasets. This difference can be attributed to the strict syntax of Java, which aids \(\mathtt{Sym}\) in more effective identification and process of loop structures--a critical aspect for time complexity analysis. Although the overall performance of \(\mathtt{Sym}\) is promising, its integration with the broader TCProF, demonstrates the most effective results. Appendix H illustrates with examples, the detailed procedure of how \(\mathtt{Sym}\) operates.

While \(\mathtt{Sym}\) does not capture every potential operation involved in computing time complexity, its design is strategically tailored to assist the baseline model's capability in producing more precise pseudo-labels. Moving forward, we recognize the potential for further enhancing \(\mathtt{Sym}\) by expanding its scope to include a broader range of conditions. This development is part of our future research aimed to broaden our understanding of code time complexity prediction.

## Appendix H Symbolic Pseudo-Labeling Running Example

In the symbolic module \(\mathtt{Sym}\), the first step involves employing regular expressions (Regex) to ascertain the existence of functions within the source code. Figure 9 presents a Python code snippet that contains a function named solve(). The module recognizes solve() as a function due to the presence of the keyword def. Subsequently, \(\mathtt{Sym}\) extends its analysis to detect loops and recursion, again utilizing Regex.

In the given example, \(\mathtt{Sym}\) identifies two for-loops due to the the keyword for: one within the solve() function and another in the main code section. In the main code section, \(\mathtt{Sym}\) checks solve() is called only once. Through Regex matching, the module employs the keyword in, along with the associated range variables aa and n, to determine the size of each loop. As depicted in Figure 9, \(\mathtt{Sym}\) calculates the time complexity of for loop in solve() as \(O(N)\). Additionally, the module detects the use of the keyword sorted, which leads to the derivation of the combined time complexity for solve() as \(O(N)+O(N\log N)\).

Finally, \(\mathtt{Sym}\) computes the overall time complexity for the main code section. Given the presence of an additional for loop in this code section, the module calculates the total time complexity as \(O(N)+O(N)+O(N\log N)=O(N\log N)\).

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline  & \multicolumn{2}{c}{CodeComplex (Java)} & \multicolumn{2}{c}{CodeComplex (Python)} & CorCoD \\  & Acc. & F1 & Acc. & F1 & Acc. & F1 \\ \hline \hline \(\mathtt{Sym}\) & 49.69 & 40.83 & 55.76 & 41.27 & 53.68 & 37.53 \\ \hline \hline \end{tabular}
\end{table}
Table 10: The performance of \(\mathtt{Sym}\) as a time complexity classifier of the code snippets.





## Appendix I Error Analysis

From the test cases, we have analyzed the errors, and as discussed in Section 5.3, \(O(N^{2})\to O(N)\) errors occur the most. Without sufficient attention to the input size, it is sometimes confusing, even for human experts, to differentiate two codes, each in \(O(N)\) and \(O(N^{2})\) classes, respectively. For instance, the code instance in Figure 9(a) is rather straightforward. The stones_after function is called in the main part which runs linear to the input size. The main part calls the function \(n\) times and thus, the time complexity of the whole code in \(O(N^{2})\). However, the code in Figure 9(b) contains count operation which runs linear to the size of arr. If the code does not have the knowledge on the operation, the model is likely to predict its time complexity as \(O(N)\), which is wrong. This is mostly seen in the codes of class \(O(N^{2})\) and thus, the majority of errors are the \(O(N^{2})\to O(N)\).

## Appendix J Extensive Analyses on Augmentation

We have performed an extensive analysis of our augmentation strategy across four models: CodeBERT, GraphCodeBERT, UniXcoder, and CodeT5+. As stated in Appendix B, CodeBERT and GraphcodeBERT are unsuitable for self-training. Similarly, in this experiment, we notice that CodeBERT and GraphcodeBERT have no consistent tendency of the results. In contrast, UniXcoder and CodeT5+ exhibit more

Figure 10: Code examples of \(O(N^{2})\) class.

Figure 9: Execution process of the Symbolic Module using Python code snippet.



consistent performance tendencies. Therefore, this section focuses on the accuracy and F1-score results of UniXcoder and CodeT5+.

\(\mathsf{AU}_{BT}\) involves augmenting data through Back-Translation of the original labeled code. \(\mathsf{AU}_{LC\_Natural}\) denotes the experimental condition where the initial labeled data is used without specific constraints. In contrast, \(\mathsf{AU}_{IC\_Artificial}\) restricts the initial labeled data to code containing either for loops or while loops, ensuring that all augmented Loop-Conversion data can be generated for every initial labeled data. The combined strategy \(\mathsf{AU}_{BT+LC\_Natural}\) integrates Back-Translation augmentation with Loop-Conversion augmentation under natural conditions. Similarly, \(\mathsf{AU}_{GT+LC\_Artificial}\) combines Back-Translation augmentation with Loop-Conversion augmentation under artificial conditions.

We analyzed the results in two parts: \(\mathsf{AU}_{CL\_Natural}\) vs. \(\mathsf{AU}_{LC\_Artificial}\) and \(\mathsf{AU}_{BT}\) vs. \(\mathsf{AU}_{CL\_Natural}\).

The first comparison, \(\mathsf{AU}_{GLC\_Natural}\) and \(\mathsf{AU}_{GC\_Artificial}\), highlights two different augmentation strategies within \(\mathsf{AU}_{CLC\_Natural}\). Notably, \(\mathsf{AU}_{CLC\_Natural}\) significantly enhances model performance compared to \(\mathsf{AU}_{GLC\_Artificial}\), particularly in the UniXcoder model. The average number of data points used for training in the \(\mathsf{AU}_{CLC\_Natural}\) is 65, while the \(\mathsf{AU}_{CL\_Artificial}\) uses 70 data in 5-shot settings on the CodeComplex dataset, which has seven labels. Despite having less training data, \(\mathsf{AU}_{GLC\_Natural}\) achieves higher performance.

The second comparison, \(\mathsf{AU}_{GT}\) vs. \(\mathsf{AU}_{GLC\_Natural}\), evaluates the differences between \(\mathsf{AU}_{GT}\) and \(\mathsf{AU}_{CLC\_Natural}\). In Tables 12 and 13, \(\mathsf{AU}_{GLC\_Natural}\) consistently outperforms \(\mathsf{AU}_{GT}\) in both accuracy and F1-score across all evaluated models while maintaining lower standard deviations, indicating more reliable and stable performance. The average number of data points used for training in the \(\mathsf{AU}_{GLC\_Natural}\) is 65, while the \(\mathsf{AU}_{GT}\) uses 70 data in 5-shot settings on the CodeComplex dataset, which has seven labels. Despite having less training data, \(\mathsf{AU}_{GLC\_Natural}\) achieves higher performance.

For the CodeTS+ model, \(\mathsf{AU}_{GLC\_Natural}\) achieves higher accuracy except in the CodeComplex Python 5-shot setting, where a high standard deviation of 19.44% was observed in \(\mathsf{AU}_{GBT}\). Similarly, for the UniXcoder model, \(\mathsf{AU}_{GLC\_Natural}\) achieves higher accuracy in most cases, except in the CodeComplex 

Python 5-shot setting, where it shows 6%p lower accuracy. These large performance differences are likely due to the inherent variability in the CodeComplex Python data results, contributing to the instability observed across all models. Despite this 6%p lower standard deviation compared to \(\mathtt{AU}_{BT}\) in the CodeComplex Python 5-shot setting, indicating more stable performance even with a slight drop in accuracy. In the 10-shot setting, where the data is more comprehensive, \(\mathtt{AU}_{LC\_Natural}\) outperforms \(\mathtt{AU}_{BT}\) in accuracy.

There are some exceptions, such as in the CodeComplex Java 10-shot setting, likely due to the limited diversity of training data. \(\mathtt{AU}_{IC\_Natural}\) focuses on converting loop parts rather than making substantial code changes, while \(\mathtt{AU}_{GBT}\) enhances the diversity of the training data. When \(\mathtt{AU}_{GBT}\) is added to the training data, it helps bridge the performance gap observed in these exceptional cases, as evidenced by the results for \(\mathtt{AU}_{GBT+LC\_Artificial}\) in CodeComplex Java 10-shot setting. These findings suggest that \(\mathtt{AU}_{LC\_Natural}\) is the most effective augmentation strategy for enhancing the performance of code models. Moreover, combining \(\mathtt{AU}_{GBT}\) with \(\mathtt{AU}_{IC\_Natural}\) further boosts the performance of the models.

## Appendix K Augmentation Prompts

The augmentation prompts presented in this section are designed to enhance the capabilities of language models in code Back-Translation and Loop-Conversion tasks. The first prompt, Back-Translation, challenges the model to translate Java code into Python and then back into Java, ensuring functional equivalence without syntactical similarity to the original code. The second prompt, Loop-Conversion, requires the model to convert all 'for' loops to 'while' loops and vice versa in the given Java code, even at the cost of readability. To illustrate the effectiveness of these methods, we provide both original code and augmented code examples for each augmentation technique. Note that we only show parts of the codes because they are too long to display.

### Back-Translation Examples

The following is an original example of a Java code.

 

import ...

public class ProblemD { public static void main(String[] args) throws IOException { BufferedReader s = new BufferedReader(new InputStreamReader(System.in)); PrintWriter out = new PrintWriter(System.out); String[] data = s.readLine().split(" "); String[] line = s.readLine().split(" "); double ans = solve(a, v, l, d, w); out.println(String.format("\X.07f", ans)); out.flush(); } private static double solve(double a, double v, double l, double d, double w) { double maxSpeedAtD = Math.sqrt(2 * d / a) * a; if (v <= w || maxSpeedAtD <= w) {// okay. just go  ...  } double time = 0.od; double maxSpeedTime = Math.sqrt((d / a) + (w * w / (2 * a * a))); double maxSpeed = maxSpeedTime * a; if (maxSpeed <= v) {  ...  } // after that, just go. double timeToV = (v - w) / a; double timeToVLen = timeToV * w + 0.5 * timeToV * (v - w); if (timeToVLen <= l - d) {  ...  } return time; } ```
The following is a transformed example of a Java code.
``` import ... publicclassProblemD { public static void main(String[] args) throws IOException { BufferedReader s = new BufferedReader(new InputStreamReader(System.in)); PrintWriter out = new PrintWriter(System.out); String[] data = s.readLine().split(" "); String[] line = s.readLine().split(" "); double ans = solve(a, v, l, d, w); out.println(String.format("X.07f", ans)); out.flush(); } private static double solve(double a, double v, double l, double d, double w) { double maxSpeedAtD = Math.sqrt(2 * d / a) * a; if (v >= w && maxSpeedAtD > w) {  ..  } } private static double findTimeToReachL(double a, double v, double l) {  ...  return timeToMaxSpeed + leftDist / v; } private static double calculateTimeToReachDestination(double a, double v, double l, double d, double w) 

returnmaxSpeed<=v?maxSpeedTime+(a*maxSpeedTime-w)/a: calculateTimeForCaseThree(a,v,d,w,maxSpeedTime); } privatestaticdoublecalculateTimeForCaseThree(doublea,doublev,doubled,doublew, doublemaxSpeedTime){ returnv>w?v/a+time:v/a+vtime+(v-w)/a; } ```
### Loop-Conversion Examples

The following is an original example of a Java code.
``` importjava.util.*; import java.io.*; publicclassSoldiers( publicstaticvoidmain(String[]args)throwsIOException{ newSoldiers().run(); } voidrum()throwsIOException{ br=newBufferedReader(newInputStreamReader(System.in)); pw=newPrintWriter(System.out); intn=nextInt(); pw.println3*(n/2)); pw.close(); } BufferedReaderbr; StringTokenizerst; PrintWriterpw; Stringnext()throwsIOException{ while(st==null||!st.hasMoreTokens()){ st=newStringTokenizer(br.readLine()); } returnst.nextToken(); } intnextInt()throwsIOException{ returnInteger.parseInt(next()); } doublenextDouble()throwsIOException{ returnDouble.parseDouble(next()); } } ```
The following is a transformed example of a Java code.
``` importjava.util.*; importjava.io.*; publicclassSoldiers{ publicstaticvoidmain(String[]args)throwsIOException{ newSoldiers().run(); 

voidrun()throwsIQException{  br=newBufferedReader(newInputStreamReader(System.in));  pw=newPrintWriter(System.out);

 intn=nextInt();  pw.println(3*(n/2));

 pw.close();  } BufferedReaderbr; StringTokenizerst; PrintWriterpw; Stringnext()throwsIOException{ for(;st==null||lst.hasMoreTokens();st=new \(\leftrightarrow\)StringTokenizer(br.readLine())){  returnst.nextToken();  }

 intnextInt()throwsIOException{  returnInteger.parseInt(next());  } doublenextDouble()throwsIOException{  returnDouble.parseDouble(next()); 



### Back-Translation Prompt

Back-Translation Prompt You are a powerful AI that can translate Java code into Python code and then back into Java code. I will provide you with a piece of Java code. Your task is to generate an equivalent Python code and then translate it back into Java. When translating back into Java, the code should not be identical to the original Java code, but it should still be functionally equivalent. Please ensure that the output is valid Java and does not contain any syntax or constructs from the other language. The Java output should be in the following JSON format: { "back-translation": "[Your transformed Java code here]" }. Please note that "[Your transformed Java code here]" should be replaced with your transformed Java code. Do not include any other keys in the "back-translation" value. Please ensure that you generate the complete transformed Java code, do not stop halfway through. Do not generate Python code, only Java code. Given Java Code: for(inti=0;i<5;i++) System.out.println("Numberis"+i); } Your Transformed Python Code(Java to Python): forinrange(5): print("Numberis"+str(I)) Your Transformed Java Code(Python to Java): inti=0; while(i<5)( System.out.println("Numberis"+i); i++; } Here is the Java code:"[Original Java Code]" Please translate this Java code into Python code and then back into Java code, and generate your answer in the following JSON format: {{ "back-translation": "[The transformed Java code]" }} Don't cut me off in the middle, create it all the way through. The output should not contain any text, only code. This includes avoiding explanations, comments, or any other form of text. Please ensure that the transformed code is properly indented for readability. Do not generate Python code, only Java code.

Figure 11: LLM prompt examples used in Back-translation.





### Loop-Conversion Prompt

Loop-Converstion Prompt ... Your task is to convert all 'for' loops into 'while' loops and all 'while' loops into 'for' loops. For example, if the Java code contains a 'for' loop, you should change it into a 'while' loop. If the Java code contains a 'while' loop, you should change it into a 'for' loop. If the Java code contains both 'for' and 'while' loops, you should change all 'for' loops into 'while' loops and all 'while' loops into 'for' loops. Please note that all 'while' loops should be converted into 'for' loops, even if it breaks the readability of the code. Please note that all 'while' loops should be converted into 'for' loops, even if it only runs once. This may result in 'for' loops with empty initialization or increment sections. The output should be valid Java code and should not contain any syntax or constructs from other languages. The Java output should be in the following JSON format: Please ensure that your transformed Java code is enclosed in double quotes and ends with a closing quote. { "forwhile": "[Your transformed Java code here]" }. ... Here is an example of how to convert a 'for' loop to a 'while' loop in Java: ... And here is an example of how to convert a Java code that contains both 'for' and 'while' loops: Given Java Code: for(int i = 0; i < 10; i++) { System.out.println("For loop: " + i); } while(!st.hasMoreTokens()) { st = new StringTokenizer(in.readLine()); } Your Transformed Java Code: int i = 0; while(i < 10) { System.out.println("For loop: " + i); i++; } for(; lst.hasMoreTokens(); st = new StringTokenizer(in.read_line())) { } Here is the Java code: "[Original Java Code]". Please convert all 'for' loops into 'while' loops and all 'while' loops into 'for' loops, and generate your answer in the following JSON format: {{ "forwhile": "[The transformed Java code]" }}. Don't cut me off ...

Figure 12: LLM prompt examples used in Loop-conversion.



LLM 5-shot In-Context Learning Prompts

[frametitle=] LLM Frompt You are the best programmer in the world. You will be asked to determine the time complexity of the following code. For the time complexity, choose one time complexity from the following options 'constant', 'logn', 'linear', 'nlogn', 'quadratic', 'cubic', and 'exponential'. Do not hesitate to use any other supplementary materials you need for the task. I will first give you the code. After you read the code, I will ask you to compute the time complexity of the code. The following are the demonstrations of the time complexity for codes:

[frametitle=] n = int(input()) for i in range(n): print(i) "complexity": linear print(int(input())) "complexity": constant print("*") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("**") print("") print("**") print("**") print("") print("**") print("") print("**") print("") print("**") print("") print("**") print("") print("") print("**") print("")