Cannot load arxiv dataset on MacOS?
Author: JohnGiorgi
State: closed
Labels: name: dataset bug

I am having trouble loading the `"arxiv"` config from the `"scientific_papers"` dataset on MacOS. When I try loading the dataset with:

```python
arxiv = nlp.load_dataset("scientific_papers", "arxiv")
```

I get the following stack trace:

```bash
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-2-8e00c55d5a59> in <module>
----> 1 arxiv = nlp.load_dataset("scientific_papers", "arxiv")

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    662 
    663         generator = self._generate_examples(**split_generator.gen_kwargs)
--> 664         for key, record in utils.tqdm(generator, unit=" examples", total=split_info.num_examples, leave=False):
    665             example = self.info.features.encode_example(record)
    666             writer.write(example)

~/miniconda3/envs/t2t/lib/python3.7/site-packages/tqdm/std.py in __iter__(self)
   1106                 fp_write=getattr(self.fp, 'write', sys.stderr.write))
   1107 
-> 1108         for obj in iterable:
   1109             yield obj
   1110             # Update and possibly print the progressbar.

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/datasets/scientific_papers/107a416c0e1958cb846f5934b5aae292f7884a5b27e86af3f3ef1a093e058bbc/scientific_papers.py in _generate_examples(self, path)
    114                 # "section_names": list[str], list of section names.
    115                 # "sections": list[list[str]], list of sections (list of paragraphs)
--> 116                 d = json.loads(line)
    117                 summary = "\n".join(d["abstract_text"])
    118                 # In original paper, <S> and </S> are not used in vocab during training

~/miniconda3/envs/t2t/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

~/miniconda3/envs/t2t/lib/python3.7/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/miniconda3/envs/t2t/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)
    351         """
    352         try:
--> 353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
    355             raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Unterminated string starting at: line 1 column 46983 (char 46982)

163502 examples [02:10, 2710.68 examples/s]   
```

I am not sure how to trace back to the specific JSON file that has the "Unterminated string". Also, I do not get this error on colab so I suspect it may be MacOS specific. Copy pasting the relevant lines from `transformers-cli env` below:

- Platform: Darwin-19.5.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)

Any ideas?

Comments: I couldn't replicate this issue on my macbook :/
Could you try to play with different encodings in `with open(path, encoding=...) as f` in scientific_papers.py:L108 ?

I was able to track down the file causing the problem by adding the following to `scientific_papers.py` (starting at line 116):

```python
                from json import JSONDecodeError
                try:
                    d = json.loads(line)
                    summary = "\n".join(d["abstract_text"])
                except JSONDecodeError:
                    print(path, line)
```



For me it was at: `/Users/johngiorgi/.cache/huggingface/datasets/f87fd498c5003cbe253a2af422caa1e58f87a4fd74cb3e67350c635c8903b259/arxiv-dataset/train.txt` with `"article_id": "1407.3051"`.

Not really 100% sure at the moment, but it looks like this specific substring from `"article_text"` may be causing the problem?

```
"after the missing - mass scale adjustment , the validity of the corrections was tested in the @xmath85 productions at 1.69 gev/@xmath1 . in fig .  [", "fig : calibrations ] ( a ) , we show the missing - mass spectrum in the @xmath86 region in the @xmath87 reaction at 1.69 gev/@xmath1 . a fitting result with a lorentzian function for the @xmath86  ( dashed line ) and the three - body phas
```

perhaps because it appears to be truncated. I (think) I can recreate the problem by doing the following:

```python
import json

# A minimal example of the json file that causes the error
invalid_json = '{"article_id": "1407.3051", "article_text": ["the missing - mass resolution was obtained to be 2.8 @xmath3 0.1  mev/@xmath4  ( fwhm ) , which corresponds to the missing - mass resolution of 3.2  @xmath3  0.2  mev/@xmath4  ( fwhm ) at the @xmath6 cusp region in the @xmath0 reaction .", "this resolution is at least by a factor of 2 better than the previous measurement with the same reaction ( 3.2@xmath595.5 mev/@xmath4 in @xmath84 )  @xcite .", "after the missing - mass scale adjustment , the validity of the corrections was tested in the @xmath85 productions at 1.69 gev/@xmath1 . in fig .  [", "fig : calibrations ] ( a ) , we show the missing - mass spectrum in the @xmath86 region in the @xmath87 reaction at 1.69 gev/@xmath1 . a fitting result with a lorentzian function for the @xmath86  ( dashed line ) and the three - body phas'   
# The line of code from `scientific_papers.py` which appears to cause the error
json.loads(invalid_json)
```

This is as far as I get before I am stumped.

I just checked inside `train.txt` and this line isn't truncated for me (line 163577).
Could you try to clear your cache and re-download the dataset ?

Ah the turn-it-off-turn-it-on again solution! That did it, thanks a lot :)