GitHub issue #1288: Add CodeSearchNet corpus dataset
State: closed | Created: 2020-12-08T10:07:50Z

This PR adds the CodeSearchNet corpus proxy dataset for semantic code search: https://github.com/github/CodeSearchNet
I have had a few issues, mentioned below. Would appreciate some help on how to solve them.

## Issues generating dataset card
Is there something wrong with my declaration of the dataset features ?
```
features=datasets.Features(
    {
        "repository_name": datasets.Value("string"),
        "func_path_in_repository": datasets.Value("string"),
        "func_name": datasets.Value("string"),
        "whole_func_string": datasets.Value("string"),
        "language": datasets.Value("string"),
        "func_code_string": datasets.Value("string"),
        "func_code_tokens": datasets.Sequence(datasets.Value("string")),
        "func_documentation_string": datasets.Value("string"),
        "func_documentation_tokens": datasets.Sequence(datasets.Value("string")),
        "split_name": datasets.Value("string"),
        "func_code_url": datasets.Value("string"),
        # TODO - add licensing info in the examples
    }
),

```
When running the streamlite app for tagging the dataset on my machine, I get the following error :
![image](https://user-images.githubusercontent.com/33657802/101469132-9ed12c80-3944-11eb-94ff-2d9c1d0ea080.png)


## Issues with dummy data
Due to the unusual structure of the data, I have been unable to generate dummy data automatically.
I tried to generate it manually, but pytests fail when using the manually-generated dummy data ! Pytests work fine when using the real data.
```
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.7.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
plugins: xdist-2.1.0, forked-1.3.0
collected 1 item

tests/test_dataset_common.py F                                                                                                                                                                            [100%]

=================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________ LocalDatasetTest.test_load_dataset_all_configs_code_search_net _________________________________________________________________________
self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_code_search_net>, dataset_name = 'code_search_net'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
Downloading and preparing dataset code_search_net/all (download: 1.00 MiB, generated: 1.00 MiB, post-processed: Unknown size, total: 2.00 MiB) to /tmp/tmppx78sj24/code_search_net/all/1.0.0...
Dataset code_search_net downloaded and prepared to /tmp/tmppx78sj24/code_search_net/all/1.0.0. Subsequent calls will reuse this data.
--------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------
... (irrelevant info - Deprecation warnings)
============================================================================================ short test summary info ============================================================================================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_code_search_net - AssertionError: False is not true
========================================================================================= 1 failed, 4 warnings in 3.00s ========================================================================================
```


## Note : Data structure in S3
The data is stored on S3, and organized by programming languages.
It is stored in the following repository structure:
```
.
├── <language_name>  # e.g. python
│   └── final
│       └── jsonl
│           ├── test
│           │   └── <language_name>_test_0.jsonl.gz
│           ├── train
│           │   ├── <language_name>_train_0.jsonl.gz
│           │   ├── <language_name>_train_1.jsonl.gz
│           │   ├── ...
│           │   └── <language_name>_train_n.jsonl.gz
│           └── valid
│               └── <language_name>_valid_0.jsonl.gz
├── <language_name>_dedupe_definitions_v2.pkl
└── <language_name>_licenses.pkl
```