Add GLUE config name check
State: closed | Author: bharatr21 (CONTRIBUTOR)

Fixes #130 by adding a name check to the Glue class

Comments: If tests are being added, any guidance on where to add tests would be helpful!

Tagging @thomwolf for review

Looks good to me. Is this compatible with the way we are doing tests right now @patrickvonplaten ?

If the tests pass it should be fine :-) 

@Bharat123rox could you check whether the tests pass locally via: 
`pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_glue`

The test fails with an `AssertionError` because the name is not being passed to kwargs, however I'm not sure how to do that, because only the config file is being passed to the tests of all datasets?

I'm guessing this is the corresponding code:
https://github.com/huggingface/nlp/blob/2b3621bb5c78caf02c5a969b8e67fa0c145da4e6/tests/test_dataset_common.py#L141-L143

And these are the logs:
```
___________________ DatasetTest.test_load_dataset_local_glue ___________________

self = <tests.test_dataset_common.DatasetTest testMethod=test_load_dataset_local_glue>
dataset_name = 'glue'

    @local
    def test_load_dataset_local(self, dataset_name):
        # test only first config
        if "/" in dataset_name:
            logging.info("Skip {} because it is not a canonical dataset")
            return

        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:200:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:74: in check_load_dataset
    dataset_builder = dataset_builder_cls(config=config, cache_dir=processed_temp_dir)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nlp.datasets.glue.fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597.glue.Glue object at 0x135c0ea90>
args = ()
kwargs = {'cache_dir': '/var/folders/r6/mnw5ntvn5y72j7d4s1fm273m0000gn/T/tmpa9rpq3tl', 'config': GlueConfig(name='cola', versio...linguistic theory. Each example is a sequence of words annotated\nwith whether it is a grammatical English sentence.')}

    def __init__(self, *args, **kwargs):
>       assert ('name' in kwargs and kwargs['name'] is not None), "Glue has to be called with a configuration name"
E       AssertionError: Glue has to be called with a configuration name

/usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py:139: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO:nlp.load:Checking ./datasets/glue/glue.py for additional imports.
INFO:filelock:Lock 5209998288 acquired on ./datasets/glue/glue.py.lock
INFO:nlp.load:Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO:nlp.load:Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO:nlp.load:Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO:nlp.load:Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO:nlp.load:Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO:filelock:Lock 5209998288 released on ./datasets/glue/glue.py.lock
INFO:nlp.load:Checking ./datasets/glue/glue.py for additional imports.
INFO:filelock:Lock 5196802640 acquired on ./datasets/glue/glue.py.lock
INFO:nlp.load:Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO:nlp.load:Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO:nlp.load:Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO:nlp.load:Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO:nlp.load:Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO:filelock:Lock 5196802640 released on ./datasets/glue/glue.py.lock
------------------------------ Captured log call -------------------------------
INFO     nlp.load:load.py:157 Checking ./datasets/glue/glue.py for additional imports.
INFO     filelock:filelock.py:274 Lock 5209998288 acquired on ./datasets/glue/glue.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO     nlp.load:load.py:333 Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO     nlp.load:load.py:346 Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO     nlp.load:load.py:356 Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO     nlp.load:load.py:367 Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO     filelock:filelock.py:318 Lock 5209998288 released on ./datasets/glue/glue.py.lock
INFO     nlp.load:load.py:157 Checking ./datasets/glue/glue.py for additional imports.
INFO     filelock:filelock.py:274 Lock 5196802640 acquired on ./datasets/glue/glue.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO     nlp.load:load.py:333 Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO     nlp.load:load.py:346 Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO     nlp.load:load.py:356 Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO     nlp.load:load.py:367 Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO     filelock:filelock.py:318 Lock 5196802640 released on ./datasets/glue/glue.py.lock
```

Closing as #130 is fixed !