SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Beyond the results in Table 2 of the main paper, we further evaluate more code LMs, including more variants of CodeGen and StarcoderBase. The results are shown in Table 7.
Model	Code Match
Python	Java	TypeScript	C#
EM	ES	EM	ES	EM	ES	EM	ES
CodeGen-350M	2.70	52.94	3.51	54.92	2.77	46.60	0.90	42.64
+ Retrieval	6.98	56.91	6.97	56.87	4.02	48.07	3.28	45.35
+ Retrieval w/ Ref.	9.64	59.31	8.04	57.75	5.21	50.25	6.50	48.58
CodeGen-2.7B	4.92	55.75	5.89	58.86	3.84	50.77	1.36	48.04
+ Retrieval	9.64	60.02	10.00	61.20	6.47	52.61	4.24	52.19
+ Retrieval w/ Ref.	13.28	62.65	11.73	62.50	7.81	54.79	7.35	54.78
CodeGen-6.1B	5.40	55.41	5.19	58.89	3.75	49.87	1.47	45.34
+ Retrieval	10.58	60.36	8.88	61.06	6.14	51.58	4.64	47.53
+ Retrieval w/ Ref.	14.07	62.80	11.03	61.76	8.22	54.01	7.13	48.98
CodeGen-16.1B	6.87	57.64	7.01	60.49	4.50	52.24	1.81	45.28
+ Retrieval	12.46	62.66	11.69	62.16	7.06	54.11	5.54	48.25
+ Retrieval w/ Ref.	16.66	65.31	13.65	63.50	9.18	56.54	8.54	49.63
StarCoderBase-1B	0.19	54.78	0.19	57.17	0.06	42.00	0.11	57.65
+ Retrieval	8.18	60.06	7.48	59.71	3.64	40.07	8.94	62.32
+ Retrieval w/ Ref.	12.83	63.37	10.38	62.06	6.79	42.47	15.16	65.56
StarCoderBase-3B	4.77	57.54	5.66	60.26	3.55	46.74	3.62	59.27
+ Retrieval	11.74	62.99	12.39	62.33	6.73	43.89	11.99	64.65
+ Retrieval w/ Ref.	16.66	65.91	14.91	64.83	9.62	46.39	18.04	66.86
StarCoderBase-7B	6.75	59.83	8.74	62.84	5.13	49.31	4.86	59.71
+ Retrieval	13.28	64.76	15.61	65.19	8.25	45.73	14.20	65.65
+ Retrieval w/ Ref.	18.95	67.41	18.70	67.76	11.95	47.91	20.64	67.95
Model	Identifier Match
Python	Java	TypeScript	C#
EM	F1	EM	F1	EM	F1	EM	F1
CodeGen-350M	7.92	38.51	9.40	43.36	5.07	34.44	1.98	21.07
+ Retrieval	13.70	44.34	13.32	46.47	7.06	37.09	5.83	26.67
+ Retrieval w/ Ref.	17.49	48.14	14.68	47.90	8.85	39.75	8.88	31.27
CodeGen-2.7B	11.14	42.42	12.62	47.94	7.51	39.71	3.96	24.92
+ Retrieval	17.22	48.96	18.19	51.43	10.91	42.96	8.03	31.85
+ Retrieval w/ Ref.	21.54	52.40	20.06	53.19	12.96	45.58	11.43	36.50
CodeGen-6.1B	10.92	42.22	12.48	48.38	6.76	39.13	3.90	23.20
+ Retrieval	17.90	49.41	17.25	51.79	10.31	42.29	8.26	30.01
+ Retrieval w/ Ref.	22.40	53.14	19.45	53.12	12.87	45.72	12.10	34.20
CodeGen-16.1B	13.28	44.98	14.40	49.97	8.76	41.95	4.13	23.49
+ Retrieval	20.53	51.95	19.50	52.97	12.13	44.65	9.33	31.08
+ Retrieval w/ Ref.	25.37	55.76	22.21	54.98	15.14	48.02	12.67	35.08
StarCoderBase-1B	6.79	39.70	7.71	45.18	4.71	33.58	4.02	30.32
+ Retrieval	15.91	47.87	15.57	49.64	8.13	33.45	13.01	39.38
+ Retrieval w/ Ref.	21.61	52.95	19.54	52.57	11.80	36.55	18.83	44.81
StarCoderBase-3B	11.48	43.63	14.12	49.13	8.76	39.18	7.30	32.55
+ Retrieval	19.81	51.66	21.18	53.03	11.83	37.95	16.06	42.45
+ Retrieval w/ Ref.	25.40	55.89	24.36	56.09	15.08	41.09	21.95	47.03
StarCoderBase-7B	13.92	46.65	16.88	52.18	10.04	42.15	8.71	34.04
+ Retrieval	22.29	54.14	24.92	56.70	13.92	40.25	18.16	44.55
+ Retrieval w/ Ref.	28.26	58.14	29.64	59.99	17.79	43.40	24.04	49.12
Table 7: Evaluation of additional code LMs on CROSSCODEEval (cf. Table 2)
D.2 Nucleus Sampling w/ Re-ranking
Though the main results in this benchmark are reported with greedy search, we further conduct experiments to explore the effects of sampling and reranking. To this end, we apply the nucleus sampling (Holtzman et al., 2020) then mean-log-likelihood reranking (Hossain et al., 2020) during the code generation. The experiments are conducted with temperature of 0.2 for the token probability scaling. For the nucleus sampling, we set the top-p to be 0.95, and for the mean-log-likelihood reranking, we generate top-5 prediction under the sampling setting and compute the mean log-likelihood of each generation to pick the most probable prediction.
The results are shown in Table 8. Compared to the main results (Table 2 in the main paper), we notice the results are quite comparable, and the difference is marginal. The cross-file and oracle context bring equivalent improvement to the greedy search setting. The results empirically reveal that (1) the performance difference of sampling with re-ranking and greedy decoding is marginal, and (2) the cross-file context is helpful regardless of the sampling/search algorithms.
Model	Code Match
Python	Java	TypeScript	C#
EM	ES	EM	ES	EM	ES	EM	ES
CodeGen25-7B	8.14	59.72	10.47	62.54	7.90	57.69	3.90	59.73
+ Retrieval	14.60	64.56	17.63	64.49	13.32	60.35	13.29	64.55
+ Retrieval w/ Ref.	19.51	67.67	20.48	66.92	15.79	62.88	18.21	66.46
StarCoder-15.5B	8.93	61.43	10.66	63.97	6.05	51.95	4.64	60.52
+ Retrieval	15.68	66.70	17.58	66.73	8.76	45.78	14.03	65.70
+ Retrieval w/ Ref.	21.35	69.44	20.20	68.49	11.65	47.32	19.97	68.32
Model	Identifier Match
Python	Java	JavaScript	C#
EM	F1	EM	F1	EM	F1	EM	F1
CodeGen25-7B	14.60	46.13	17.16	52.04	12.40	47.73	7.47	34.52
+ Retrieval	22.81	53.72	25.15	55.99	18.18	51.72	17.36	43.88
+ Retrieval w/ Ref.	28.78	58.20	28.94	58.98	21.99	55.53	22.12	47.78
StarCoder-15.5B	15.91	48.08	19.07	54.08	11.50	44.13	8.31	34.58
+ Retrieval	24.62	55.75	26.46	58.46	14.78	40.59	18.33	44.55
+ Retrieval w/ Ref.	30.77	60.01	30.53	61.07	18.15	42.90	24.15	49.26
Table 8: Performance of code LMs on CROSSCODEEval with temperature-based nucleus sampling. (cf. Table 2 in the main body of the paper).
D.3 Additional Results of Code Retrieval
Table 9 presents identifier match results of various retrieval methods for cross-file context.
D.4 Qualitative Analysis
To illustrate the quality of the retrieved cross-file context and in which ways they are helping to maximize the code LMs' capacity, we provide two qualitative examples in Figure 9 and 10.
In Figure 9, we can see that the cross-file context provides retrieves the code snippets with a similar context to the cursor position. By capturing the repetitiveness of the repository (Le Goues et al., 2012; Barr et al., 2014), the cross-file context helps the code LMs adapt the existing, repetitive coding patterns to complete the programs. Specifically, the retrieved context also defines a function named step(), with a similar goal of generating a token, and it provides a direct reference for completing the API call.
Model	Retriever	Identifier Match
Python	Java	TypeScript	C#
EM	F1	EM	F1	EM	F1	EM	F1
CodeGen25-7B	-	14.26	46.02	16.60	51.43	12.46	47.75	7.69	33.81
+ Retrieval	BM25	22.96	53.68	24.03	55.48	17.85	51.27	17.36	43.56
+ Retrieval w/ Ref.	BM25	28.33	57.95	27.91	57.87	21.51	55.38	21.78	47.63
+ Retrieval	UnixCoder	22.74	53.13	22.67	54.63	17.43	51.04	16.63	42.98
+ Retrieval w/ Ref.	UnixCoder	26.90	56.53	25.57	56.69	20.74	53.79	20.36	45.74
+ Retrieval	OpenAI Ada	23.53	54.17	25.01	55.67	18.06	51.67	19.30	46.08
+ Retrieval w/ Ref.	OpenAI Ada	27.95	56.59	28.33	58.39	21.57	55.07	24.73	51.51
StarCoder-15.5B	-	15.72	48.16	18.28	53.23	11.86	43.53	8.54	34.33
+ Retrieval	BM25	24.77	55.57	25.95	57.74	14.09	39.50	18.04	44.38
+ Retrieval w/ Ref.	BM25	30.24	59.46	29.73	60.47	17.55	42.18	24.38	49.09
+ Retrieval	UnixCoder	20.41	46.66	19.26	47.30	15.55	45.91	15.61	38.86
+ Retrieval w/ Ref.	UnixCoder	25.25	55.42	25.90	57.88	13.32	39.13	16.23	42.36
+ Retrieval	OpenAI Ada	25.55	56.23	26.65	57.68	14.15	39.99	19.98	46.45
+ Retrieval w/ Ref.	OpenAI Ada	29.64	58.64	30.86	60.73	18.12	42.77	27.96	53.53
Table 9: Identifier Match evaluation results of various methods in retrieving cross-file context (cf. Table 4).
Different from Figure 9, Figure 10 no longer retrieves the similar usage of predicting APIs, and rather, it retrieves the implementation details of the cross-file dependencies. Concretely, the cross-file context collects the member function, store_by_text, of class EntitySessionStorage, which is imported by the current completing file and instantiated as self-entityrepository. Without such cross-file context, the model hallucinates a wrong function usage that is not defined in class EntitySessionStorage. In contrast, when the cross-file is pretended to the prompt, the model successfully predicts the ground truth, empirically revealing the limitation of existing datasets by only feeding the current-file context, which will consequently underestimate code LMs' capacity.
Zero-shot Evaluation Our benchmarking was done in a zero-shot fashion. We didn't perform the few-shot study as the max sequence length of most benchmarked models is quite limited for prepending additional examples to the prompt. Thus, the performance will be limited as the format of cross-file context is never seen by the model during both training and prompting. We hope that CROSSCODEVAL encourages future research to investigate methods for efficiently retrieving and incorporating cross-file context into the model.
Cross-file Context Retrieval Quality Prepending cross-file context to the prompt has shown significant improvement to the code LM's performance. However, as we have analyzed and identified in Section 3.5, the RG retrieval framework is not perfect. Due to its fixed length of context window and token-based similarity calculation, RG sometimes retrieves useless information and fails to help code LMs for better generation. As for future work, we are expecting a more advanced retrieval approach to replace RG for more accurate cross-file context.
Memorization Code LMs were trained on a vast amount of unlabeled code. There is no way we could ensure that all models didn't see the evaluation data in the past. We take our best effort by excluding popular packages from annotation (see Section 2.1). Despite that, we suggest researchers and practitioners be cautious in interpreting the results with potential memorization in mind. Future research in incorporating cross-file context may also consider deduplicating the training data with CROSSCODEVAL, e.g., through methods in Lee et al. (2022).
Cross-file Context
# ...
# the below code fragment can be found in:
# examples/constraints.py
# def __init__(self, prompt, can_FOLLOW):
#     super().__init()
#     self.context = self.new_contextprompt)
#     self.can_FOLLOW = can_FOLLOW
#     def step(self):
#         # Generate proposed token.
#         token = self.sample(llp.Transformizer(self.context),
proposal= self.locally_optimal Proposal())
# ...
In-file Context
import llamppl as llp   
import numpy as np   
class Infilling(llp.Model): def __init__(self, words): super().__init_(） self.s  $=$  words.pop(0) self.ctx  $=$  self.new_context(self.s) self.remaining_segments  $=$  [self. llamatokenizer(w) for w in words] def start(self): self step() def step(self): # Generate a token n  $=$  self.sample(llp.Geometric(0.5))+1 for_in range(n): self.s  $+ =$  self/sample(llp.[CURSOR POSITION]
w/o Cross-file Context
with Cross-file Context
Categorical(self.ctx.get_token_probs())
Transformer(self.ctx))
Figure 9: Qualitative Example 1: the cross-file context provides a similar usage of the predicting API.
Cross-file Context
# ...
# the below code fragment can be found in:
# doccano_small/storages/entity.py
# ...
# entities = self(storage.get_state("entities")
# return entities.get(text, [*])
# def store_by_text(self, text: str, entities: List[Entity]) -> None:
# current Entities = self(storage.get_state("entities")
# current Entities.text = entities
# self(storage.set_state("entities", current Entities)
# ...
In-file Context
from typing import Dict, List   
...   
from doccano-mini storages(entity import EntitySessionStorage   
from doccano-mini storages.stepper import StepperSessionStorage   
class NamedEntityRecognitionPage(BasePage): example_path  $=$  "named-entity_recognition.json" def __init__(self, title: str) -> None: super(..init_(title) self.types:List[str]  $\equiv$  [] self(entityRepository  $=$  EntitySessionStorage() self.stepperRepository  $=$  StepperSessionStorage()   
def define-entity_types(self):   
def annotate(self, examples: List[Dict]) -> List[Dict]: if len(examples)  $= = 0$  return [] types  $=$  self.define-entity_types() selected_type  $=$  st.selectbox("Select an entity type", types) step  $=$  self.stepperrepository.get_step() text  $=$  examples[step][text"] entities  $=$  self-entityRepository.find_by_text(text) entities  $=$  st_ner_annotate(select_type,text,entities,key  $\coloneqq$  text) self-entity/repository.[CURSOR POSITION]
Figure 10: Qualitative Example 2: the cross-file context provides the definition of the predicting API.
save entities)
store_by_text(text, entities)