Issue: Update docs to new frontend/UI
State: closed
Author: mishig25

### TLDR: Update `datasets` `docs` to the new syntax (markdown and mdx files) & frontend (as how it looks on [hf.co/transformers](https://huggingface.co/docs/transformers/index))

| Light mode                                                                                 | Dark mode                                                             |
|-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| <img width="400" alt="Screenshot 2022-02-17 at 14 15 34" src="https://user-images.githubusercontent.com/11827707/154489358-e2fb3708-8d72-4fb6-93f0-51d4880321c0.png">    | <img width="400" alt="Screenshot 2022-02-17 at 14 16 27" src="https://user-images.githubusercontent.com/11827707/154489596-c5a1311b-181c-4341-adb3-d60a7d3abe85.png">                                     |

## Checklist

- [x] update datasets docs to new syntax (should call `doc-builder convert`) (this PR)
- [x] discuss `@property` methods frontend https://github.com/huggingface/doc-builder/pull/87
- [x] discuss `inject_arrow_table_documentation` (this PR) https://github.com/huggingface/datasets/pull/3690#discussion_r801847860
- [x] update datasets docs path on moon-landing https://github.com/huggingface/moon-landing/pull/2089
- [x] convert pyarrow docstring from Numpydoc style to groups style  https://github.com/huggingface/doc-builder/pull/89(https://stackoverflow.com/a/24385103/6558628)
- [x] handle `Raises` section on frontend and doc-builder https://github.com/huggingface/doc-builder/pull/86
- [x] check imgs path (this PR) (nothing to update here)
- [x] doc exaples block has to follow format `Examples::` https://github.com/huggingface/datasets/pull/3693
- [x] fix [this docstring](https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/arrow_dataset.py#L3339) (causing svelte compilation error)
- [x] Delete sphinx related files 
- [x] Delete sphinx CI
- [x]  Update docs config in setup.py
- [x] add `versions.yml` in doc-build  https://github.com/huggingface/doc-build/pull/1
- [x] add `versions.yml` in doc-build-dev https://github.com/huggingface/doc-build-dev/pull/1
- [x] https://github.com/huggingface/moon-landing/pull/2089
- [x] format docstrings for example `datasets.DatasetBuilder.download_and_prepare` args format look wrong
- [x] create new github actions. (can probably be in a separate PR) (see the transformers equivalents below)
1. [build_dev_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/build_dev_documentation.yml)
2. [build_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/build_documentation.yml)
3. [delete_dev_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/delete_dev_documentation.yml)

## Note to reviewers

The number of changed files is a lot (100+) because I've converted all `.rst` files to `.mdx` files & they are compiling fine on the svelte side (also, moved all the imgs to to [doc-imgs repo](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/datasets)). Moreover, you should just review them on preprod and see if the rendering look fine.
_Therefore, I'd suggest to focus on the changed_ **`.py`** and **CI files** (github workflows, etc. you can use [this filter here](https://github.com/huggingface/datasets/pull/3690/files?file-filters%5B%5D=.py&file-filters%5B%5D=.yml&show-deleted-files=true&show-viewed-files=true)) during the review & ignore `.mdx` files. (if there's a bug in `.mdx` files, we can always handle it in a separate PR afterwards).

Comments:
We can have the docstrings of the properties that are missing docstrings (from discussion [here](https://github.com/huggingface/doc-builder/pull/96)) here by using your new `inject_arrow_table_documentation` onthem as well ?

@sgugger & @lhoestq could you help me with what should the `docs` section in setup.py be changed to [here](https://github.com/huggingface/datasets/blob/master/setup.py#L212-L227) ?

as a reference, here is a transformers setup.py docs [section](https://github.com/huggingface/transformers/blob/master/setup.py#L304-L308)

For now, you can put an empty list. Once the `doc-builder` is in a PyPi package (with the bug we fixed on Datasets but still waiting on the standing PR with the code switch) we can put it there.

None of those dependencies are needed from this list?

```py
    "docs": [
        "docutils==0.16.0",
        "recommonmark",
        "sphinx==3.1.2",
        "sphinx-markdown-tables",
        "sphinx-rtd-theme==0.4.3",
        "sphinxext-opengraph==0.4.1",
        "sphinx-copybutton",
        "fsspec<2021.9.0",
        "s3fs",
        "sphinx-panels",
        "sphinx-inline-tabs",
        "myst-parser",
        "Markdown!=3.3.5",
    ],
```

No, that was all for sphinx. The only thing needed to build the doc is a pip install of `doc-builder` (only from git right now).

@lhoestq feel free to request reviews from other maintainers 😊

Thanks ! @mariosasko and @albertvillanova feel free to take a look :)
I can do a thorough review this afternoon

Cool thanks ! Feel free to merge master into this branch and run `make style` to fix the python code formatting

Love the colorful vibes here!
![Screen Shot 2022-02-22 at 9 54 17 AM](https://user-images.githubusercontent.com/59462357/155193444-45e639dc-79cd-463c-98ad-1d44a6d6d385.png) 

I just fixed the conflicts with the `master` branch :)

Could you update preprod please ? Or is there a preview somewhere I can check to make sure everything is ok ?

> Could you update preprod please ? Or is there a preview somewhere I can check to make sure everything is ok ?

I'll let you know once preprod gets updated

@lhoestq @stevhliu updated [preprod](https://moon-preprod.huggingface.co/docs/datasets/index) with the latest; please let e know if you see any errors

One more tiny error that doesn't seem specific to Datasets (Transformers example [here](https://huggingface.co/docs/transformers/multilingual#xlm-language-embeddings)), but apostrophes and symbols aren't properly displayed in the right navbar:

![Screen Shot 2022-03-02 at 8 39 10 AM](https://user-images.githubusercontent.com/59462357/156406988-27e79533-b02a-4fc2-af32-8ad84657488f.png)

In the latest commit https://github.com/huggingface/datasets/pull/3690/commits/20bddf28b22798c309e6eb1198a716f055889e1b, I tried to reflect changes from https://github.com/huggingface/transformers/pull/15903 , however, the gh workflow is not being triggered. @lhoestq do you know why it might be the case?

eve though, we have 
https://github.com/huggingface/datasets/blob/20bddf28b22798c309e6eb1198a716f055889e1b/.github/workflows/build_dev_documentation.yml#L3-L7

I removed this line to trigger the job
```
  pull_request:
```

but got this error
```
[Error: .github#L1](https://github.com/huggingface/datasets/commit/033fe623c556b9dbc964708b672ff9bb4896c906#annotation_2897984435)
a step cannot have both the `uses` and `run` keys
```

It seems to be running again, and I re-added the line I removed.

Now the error is
```
> Run cd doc-build-dev && ...
README.md
datasets
transformers
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
Error: Process completed with exit code 1.
```

@lhoestq if the CI passes, Im gonna merge this PR
please let me know if that sounds good