BIG-bench
State: closed

This PR adds all BIG-bench json tasks to huggingface/datasets.  

Comments:
> It looks like the CI is failing on windows because our windows CI is unable to clone the bigbench repository (maybe it has to do with filenames that are longer than 256 characters, which windows don't like). Could the smaller installation of bigbench via pip solve this issue ?
> Otherwise we can see how to remove this limitation in our windows CI.

I'm not sure.
If it's git's fault that it can't handle the long filenames, it will possibly be resolved by the pip install. If it's an issue with windows not liking long filenames after it's installed, then it will not be resolved.
I don't have a windows computer to try it on, but I might be able to tweek this PR and do an experiment to find out. 
We're waiting for a quota increase for the pip install (https://github.com/pypa/pypi-support/issues/1782). It's been pending for 2-3 weeks, and I don't have an estimate for when it will be resolved. 

>Regarding the dummy data zip files, I think we can just keep datasets/bigbench/dummy/abstract_narrative_understanding/1.0.0/dummy_data.zip and remove all the other ones. We just require to have at least one dummy_data.zip file.

Sounds great. I will trim that down. 

Do you know what are the other tests dependencies that have conflicts with bigbench ? I can try to split the CI to end up with a compatible list of test dependencies

Hi @lhoestq,

I haven't played with eliminating requirements form the test dependencies, and I've been trying to resolve this by modifying the bigbench repo to become compatible. 
In the original bigbench repo, the version requirements were strict, and specifically it had a datasets==1.17.0 requirement which was causing trouble. 
I'm working on PR https://github.com/google/BIG-bench/pull/766 to get some more flexible versions that might be compatible with the test dependencies in  HF/datasets.
We're somewhat flexible in modifying these version numbers if we can figure out what the exact conflict is. 

I've spent some time experimenting with different versions, but I don't have a very efficient way of doing this debugging on my work computer (which for some reason doesn't produce the same sets of errors running python 3.9 instead of 3.6 or 3.7 in the tests). 
It currently fails at 
> The conflict is caused by:
>    bert-score 0.3.6 depends on matplotlib
>    big-bench 0.0.1 depends on matplotlib<4.0 and >=3.5.1

which doesn't seem like it can be the real issue. 

If you have any advice for how to resolve these conflicts, that would be greatly appreciated!

Hi again @lhoestq, 
After some more or less random guessing of conflicting packages, I've managed to find a configuration that seems to be compatible with HF/datasets. 

The errors went away after removing version limits on matplotlib and scipy, and loosening numpy from 1.19 -> 1.17 in the bigbench requirements. 

I might do some more tweaking to see if it lets me set some minimal limits on matplotlib and scipy, but I think we at least can move forward.

The WIN tests are still failing, now because of 

> Did not find path entry C:\tools\miniconda3\bin
>C:\tools\miniconda3\envs\py37\python.exe: No module named pytest

I have no way of debugging this locally, and unless there's some way to get more verbose logs, I don't know why it's not finding pytest. Would you be able to take a quick look? 

Update: Actually, I see it's still failing because of the long filenames. So perhaps the pytest error is just because the previous steps failed. 

One more update on the WIN errors. 
I think all the long filenames are in files in the github repo that does not need to be included. 
We will try to remove them .

Hi ! The remaining error seems to be a `UnicodeDecodeError` from `setup.py`. I think you can fix your setup.py:
```diff
-     with open(os.path.join(os.path.dirname(__file__), fname)) as f:
+     with open(os.path.join(os.path.dirname(__file__), fname), encoding="utf-8") as f:
```
Indeed on windows, when you `open` a file it doesn't always use "utf-8" by default

Hi @lhoestq, 
The dependency issues seems to now be resolved 🎉 

Now, the WIN tests are failing at
> ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - botocore...
> ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - botocore...

Is this testing the dummy dataset that's added in bigbench? If so, I might need some help getting the right format in.

The error message I'm seeing is 
> raise EndpointConnectionError(endpoint_url=request.url, error=e)
> E           botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://127.0.0.1:5555/test"

Which seems unrelated, but perhaps the real issue is somewhere I'm not seeing? 

Woohoo awesome !

Let me check the CI error

Can you try to re-run the CI, just in case CircleCI messed up ?

Hi @lhoestq, 
Rerunning did not seem to solve the problem. 
The `test_dummy_dataset_serialize_s3` error still seems to remain.

Hi again @lhoestq, 
I'm not sure if this is informative or not in terms of debugging, but I deleted the dummy data and the errors for windows still fail and the others still pass. 
Do you have any idea what could be causing this error on windows?

_The documentation is not available anymore as the PR was closed or merged._

Now the last question: let's have the dataset under`google/bigbench` @andersjohanandreassen ?

I think it would be nicer, this way you and anyone in your team can update the dataset card whevener you want without going through a github PR. You just need to join the https://huggingface.co/google page using your google email :)

Hi @lhoestq, 

Thank you so much for the help! I really appreciate it!!!

After some discussion with the other bigbench organizers, I think there is a slight preference for bigbench to not be under google/bigbench since this is a collaboration with researchers from many different institutions/organizations beyond Google. 

I see the drawback with the updates to the dataset card having to go through a PR, but hopefully that won't be very frequent. 

We're finalizing putting the bigbench api on pip, so once that's finalized I just need to update the setup.py with the correct dependency and I think we are ready to merge. 

Ok perfect, thank you !

I noticed that in the latest windows CI run it takes forever to install the dependencies, was there any change in the bigbench dependencies recently ?

oh, sorry! I just did a double check on the dependencies, and it seems like there is at least one left that should have been removed.  There's also one new one added. 
Let me get those removed again. Will ping you here when it's updated. 

It looks like there is a circular dependency in `bigbench` at https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz

```python
>>> import bigbench.api.util as bb_utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/bigbench/api/util.py", line 29, in <module>
    import bigbench.models.query_logging_model as query_logging_model
  File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/bigbench/models/query_logging_model.py", line 23, in <module>
    import bigbench.api.util as util
AttributeError: module 'bigbench.api' has no attribute 'util'
```

Hi @lhoestq , 
I think we are ready to merge! 

I have one minor question that I haven't been able to figure out: 
Is there a way to bypass the `verify_infos` from triggering? I have `max_examples` as an argument to allow for selecting a fixed subset of the datasets (some of the tasks have *very* many examples). But this is a variable that's not specified by the configs, so it raises an `NonMatchingSplitsSizesError`.
I wasn't able to work my way around this, but perhaps there is a way to bypass this that I'm not seeing?
If this cannot be done, I'm happy to ignore this for now.

Regarding pypi, we are working on a release there, but I'm told there is some issue that there is a problem regarding the upload, and we are not sure when it will be resolved, and it's not in my control. 
I think merging this PR with the GCS is a great idea, and I will open a new PR when the pypi version is ready. 

Cool ! Merging then :D

> Is there a way to bypass the verify_infos from triggering? I have max_examples as an argument to allow for selecting a fixed subset of the datasets (some of the tasks have very many examples). But this is a variable that's not specified by the configs, so it raises an NonMatchingSplitsSizesError.

This is a bug, I opened an issue [here](https://github.com/huggingface/datasets/issues/4462). It should be easy to fix :)

The bigbench page is available here ! https://huggingface.co/datasets/bigbench

I think we can update the dataset viewer to install bigbench on it, but since this is production code I'd rather use the version on pypi for bigbench when it comes out