DOI: 10.5281/zenodo.13151632
title: Replication package of the paper "Where is Code Generated by LLMs Coming From? A Study with Gemini and Bing CoPilot"

description: # Replication Package

 

This replication package contains the necessary tools, data, and scripts for reproducing the results of our paper: "*Where is Code Generated by LLMs Coming From?

A Study with Gemini and Bing Copilot*". Below is a detailed description of the directory structure and the contents of this package.

 

## Contents

 

The replication package is organized into two main directories:

 

- `assets`: This directory contains all .csv files used as input for the script and the outputted .csv file used to perform the manual and automated analyses for RQ1 and RQ2.

 

- `script`: This directory contains all scripts for RQ1 and RQ2.

 

In the following, we describe the content of each directory:

 

 

## `assets`

 

This directory contains the tools and resources required for our study.

### `dataset`: Contains the main datasets used in the study.

 

  - `annotationStore.csv`: Input dataset for our analyses, originating from the *CODESEARCHNET* dataset.

  

  - `queries.csv`: .csv file containing the queries used for the

      experiments filtered from the *CODESEARCHNET* dataset. This file contains the following columns:

    - *Language*: Programming language of the query

    - *Query*: Query used for the experiment

    - *GitHubUrl*: GitHub URL related to a snippet that addresses the query

    - *Relevance*: Relevance of the linked GitHub snippet to the query

 

### `data`: Contains the datasets and results of all analyses.

 

  - `queries.csv`: General input queries. This file contains the following columns:

      - *Language*: Programming language of the query

      - *Query*: Query used for the snippet generation

      - *Prompt*: LLM prompt generated for the query as: *You are a Senior ```<Language>``` developer. Then give me a ```<Language>``` code snippet about: ```<Query>```*

 

  - `queries_filled.csv`: Similar to the previous file, but also containing the output produced by the LLM-based assistants. This file contains the following columns:

      - *Language*: Programming language of the query

      - *Query*: Query used for the snippet generation

      - *Prompt*: LLM prompt generated for the query as: *You are a Senior ```<Language>``` developer. Then give me a ```<Language>``` code snippet about: ```<Query>```*

      - *Notes*: General notes that provide additional context or information about the query or prompt.

      - *Gemini_Answer(n)*: The generated code snippets by Gemini.

      - *Gemini(n)*: The external links provided by Gemini.

      - *Prompt (repeated)*

      - *Note*: Notes that provide additional context or information about the query or prompt.

      - *Copilot_Answer(n)*: The generated code snippets by Bing-Copilot.

      - *Copilot_Bing(n)*: The external links provided by Bing-Copilot.

 

#### `copilot` ||  `gemini`: Contains the data related to the specific LLM. These two subdirectories have the same internal structure.

 - `queries.csv`: The `queries_filled.csv` file, filtered for the specific LLM.

 - `queries_noTrivial.csv`: Contains only the queries with at least one nontrivial generated snippet.

 - `external_links.csv`: External links extracted from the LLMs output.

 

 - `external_links_filled.csv`: Snippets extracted from the external links.

    - *index*: Query ID

    - *source*: Snippet ID

    - *url*: Link URL

    - *note*: Notes that provide additional context or information about the query or prompt

    - *code(n)*: The n-th code snippet extracted from the source

    

#### `manual_analysis`: Manual analysis results.

- `manual_analysis.csv`:

    - *index*: Query ID   

    - *query*: Query used for the snippet generation

    - *generated_snippet_(n)*: The n-th code snippet generated by the LLM-based assistant

    - *trivial_1*: Manual analysis of whether or not the snippet was trivial (validator 1)

    - *trivial_2*: Manual analysis of whether or not the snippet was trivial (validator 2)

    - *trivial_final*: Manual analysis of whether or not the snippet was trivial (final classification if there is a disagreement)

    - *source*: URL to analyze

    - *source_type_1*: Type of the source (validator 1) 

    - *source_type_2*: Type of the source (validator 2) 

    - *source_type_final*: Type of the source (final classification if there is a disagreement)

    - *related_to_query_1*: Relevance of the link to the query (validator 1) 

    - *related_to_query_2*: Relevance of the link to the query (validator 2)

    - *related_to_query_final*: Relevance of the link to the query (final classification if there is a disagreement)

    - *related_to_snippets_1*: Relevance of the generated snippet to those in the link (validator 1) 

    - *related_to_snippets_2*: Relevance of the generated snippet to those in the link (validator 2) 

    - *related_to_snippets_final*: Relevance of the generated snippet to those in the link (final classification if there is a disagreement) 

- `manual_analysis_noTrivial.csv`: As in the previous file, but only the queries with at least one nontrivial generated code snippet.

 

#### `clone_detector`: Output and intermediate files for clone detection with Copilot data.

   - `copilot_tokens || gemini_tokens`: Contains the output the tokenization of the generated code snippets and the code snippets extracted from the external links.

   - `merged_llm_ext_link.csv`: All possible pairs (Cartesian product) (code snippet extracted from the external links,  generated code snippet). This file is the input of the clone detection tool. 

      - *ID_query*: Query ID 

      - *query*: Query used for the snippet generation

      - *language*: Programming language of the query

      - *generated_snippet*: The generated code snippet by the LLM-based assistant

      - *ID_gen_snippet*: The index of the generated code snippet

      - *LOC_gen_snippet*: The number of lines of code of the generated code snippet

      - *ID_source*: Source ID

      - *source*: Source URL

      - *source_snippet*: Code snippet extracted from the source

      - *ID_source_snippet*: ID of the code snippet extracted from the source

      - *LOC_source_snippet*: The number of lines of code of the code snippet extracted from the source

      - *note*: Notes that provide additional context or information about the query or prompt

   - `clone_detection_output.csv`: Contains the clone detection results.

     - *ID_query*: The index of the query

     - *query*: Query used for the snippet generation

     - *language*: The programming language of the query

     - *generated_snippet*: The generated code snippet by the LLM-based assistant

     - *ID_gen_snippet*: The index of the generated code snippet

     - *LOC_gen_snippet*: The number of lines of code of the generated code snippet

     - *ID_source*: Source ID

     - *source*: Source URL

     - *source_snippet*: Code snippet extracted from the source

     - *ID_source_snippet*: ID of the code snippet extracted from the source

     - *LOC_source_snippet*: The number of lines of code of the code snippet extracted from the source

     - *note*: Notes that provide additional context or information about the query or prompt

     - *clone_detected*: bBolean value that indicates whether a clone has been detected (1 = detected, 0 = not detected)

     - *cloning_ratio*: Ratio of the number of lines of code of the generated code snippet has been detected as a clone in the code snippet extracted from the source

     - *cloned_lines*: The number of lines of code of the generated code snippet that has been detected as a clone in the code snippet extracted from the source

     

#### `cosine_sim`: Cosine similarity results.

   - `cosine_sim_output.csv`: Contains the cosine similarity results

    - *query_id*: Query ID

    - *snippet_id*:ID the generated code snippet

    - *source_id*: ID of the source

    - *source_snippet_id*: ID of the code snippet extracted from the source

        - *cosine_similarity*: The cosine similarity between the generated code snippet and the code snippet extracted from the source

  

#### `quant_analysis`: Quantitative analysis results.

   - `topN_links_se.csv`: Contains the top-N links extracted from the search engine.

     - *id*: Query ID

     - *query*: The query

     - *url*: Link URL

   - `merged_clone_cosine.csv`: Contains the merged results of the clone detection and cosine similarity.

     - *ID_query*: Query ID

     - *query*: The query

     - *language*: The programming language of the query

     - *generated_snippet*: The generated code snippet by the LLM-based assistant

     - *ID_gen_snippet*: The ID of the generated code snippet

     - *LOC_gen_snippet*: The number of lines of code of the generated code snippet

     - *ID_source*: The index of the source

     - *source*: The source URL

     - *source_snippet*: The code snippet extracted from the source

     - *ID_source_snippet*: The index of the code snippet extracted from the source

     - *LOC_source_snippet*: The number of lines of code of the code snippet extracted from the source

     - *note*: Notes that provide additional context or information about the query or prompt

     - *clone_detected*: Boolean value that indicates if a clone has been detected(1 = detected, 0 = not detected)

     - *cloning_ratio*: The ratio of the number of lines of code of the generated code snippet has been detected as a clone in the code snippet extracted from the source

     - *cloned_lines*: The number of lines of code of the generated code snippet that has been detected as a clone in the code snippet extracted from the source

     - *cosine_similarity*: The cosine similarity between the generated code snippet and the code snippet extracted from the source

 

 - `results`: Final analysis results.

   - `jaccard_analysis.csv`: Contains the results of the Jaccard analysis comparing the provided external links by the LLMs with the top-N links extracted from the corresponding search engine.

     - *id*: Query ID

     - *language*: The programming language of the query

     - *llm_link*: The external links provided by the LLM

     - *llm_link_size*: The number of external links provided by the LLM

     - *overlap_links*: The overlapping links between the LLM and the search engine

     - *overlap_size*: The number of overlapping links between the LLM and the search engine

     - *non_overlap_links*: The non-overlapping links between the LLM and the search engine

     - *union_size*: The size of the union set links between the LLM and the search engine

     - *jaccard*: The Jaccard similarity between the LLM and the search engine

   - `merged_analysis.csv`: Contains the merged results of the manual and quantitative analyses.

     - *id*: The index of the query

     - *query*: The query used for the experiment

     - *trivial_final(n)*: The final assignment for the triviality of the n-th generated code snippet

     - *source*: The URL of the source

     - *source_type_final*: The final assignment for the type of the source

     - *related_to_query_final*: The final assignment for the relevance of the generated code snippet to the query

     - *related_to_snippets_final*: The final assignment for the relevance of the generated code snippet to the source

     - *cloning_ratio*: The maximum cloning ratio between the generated code snippet and all the code snippets extracted from the source

     - *cosine_similarity*: The cosine similarity related to the snippets with maximum cloning ratio

   - `cccfindersw-configuration-files`: Contains additional configuration files for the CCFinderSW clone detection tool. The files are `javascript_comment.txt` and `javascript_reserved.txt`. They must be placed in the tool's `comment/` and `reserved/` directories.

   

### `appendix.tex`: The appendix of the paper containing: 

   - *Table 1*: Number of links of different types provided by Gemini and Bing CoPilot

   

### `appendix.pdf`: The appendix of the paper in PDF format.

 

## `script`

 

This directory contains our scripts (mostly Python, an R script and an Applescript) to preprocess data and run the clone detection analyses.

 

- `1_dateset_filtering.py`: Script to filter the dataset. The input of this script is the `annotationStore.csv` file, and the output is the `queries.csv` file.

- `2_prompt_generation.py`: Script to generate prompts for the LLM-based assistants. The input of this script is the `queries.csv` file, and the output is the `queries_filled.csv` file.

- `3_gen_sheet_sources_extraction.py`: Script to split the external links provided by the LLM, one for each row. The input of this script is the `queries_filled.csv` file, and the output is the `external_link.csv` file.

- `4_ext_link_snippet_extraction.py`: Script to extract the snippets from Web URLs. It only works for the most popular domains. The input of this script is the `external_links.csv` file, and the output is the `external_links_filled.csv` file.

- `5_top_n_link_SearchEngine.py`: Script to perform top-N link search using the corresponding search engines (Google Search and Bing). The input of this script is the `queries_filled.csv` file. It executes the `browser_bot.scpt`. The output is the `topN_links_se.csv` file.

  - `browser_bot.scpt`: Script for browser automation (AppleScript).  

- `6_se_vs_llm.py`: Script to compare (using the Jaccard metric) the links returned by the corresponding search engines with those provided by the LLM-based assistants. The input of this script is the `external_links.csv` file and the `topN_links_se.csv` file. The output is the `jaccaard_analysis.csv` file.

- `7_results_manual_analysis.py`: Script to extract results and statistical analyses performed on the manual analysis and reported in the tables in the paper.

- `8_merge_gen_source_snippets.py`: This script takes as input: `{llm}/queries.csv` and `{llm}/external_link_filled.csv` to merge them and generates an expanded one, i.e., one in which we have on each line a snippet extracted from the source, this will be the input of our final script for clone detection. The output is the `merged_llm_ext_link.csv` file.

- `9_clone_detection.py`: Script to perform clone detection. The input of this script is the `merged_llm_ext_link.csv` file, and the output is the `clone_detection_output.csv` file.

- `10_cosine_sim_check.py`: Script to compute the code snippets' cosine similarity. The script takes as input the tokenized files from the `{llm}_tokens` directory. The output is the `cosine_sim_output.csv` file.

- `11_merger_clone_cosine.py`: Script to merge clone detection and cosine similarity results. The input of this script is the `clone_detection_output.csv` and the `cosine_sim_output.csv` files, and the output is the `merged_clone_cosine.csv` file.

- `12_merge_manual_quantitative_analysis.py`: Script to merge manual and quantitative analysis results. The inputs of this script are the `manual_analysis.csv` and the `merged_clone_cosine.csv` files, and the output is the `merged_analysis.csv` file.

- `cloningGraph.R`: R script to generate the cloning graph. The input of this script is the `merged_analysis.csv` file.