Error while installing dev dependencies for Apple Silicon
Author: xssChauhan
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Follow contribution guidelines as outlined [here](https://huggingface.co/docs/transformers/contributing#create-a-pull-request), at the `pip install -e ".[dev]"` step and results in the following output and error:

```
$ pip install -e ".[dev]"                     

Obtaining file:///Users/eipizero/Documents/Code/transformers
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting tqdm>=4.27
  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting pyyaml>=5.1
  Using cached PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl (173 kB)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
  Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)
Collecting huggingface-hub<1.0,>=0.11.0
  Using cached huggingface_hub-0.13.3-py3-none-any.whl (199 kB)
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting filelock
  Using cached filelock-3.10.7-py3-none-any.whl (10 kB)
Collecting numpy>=1.17
  Using cached numpy-1.24.2-cp310-cp310-macosx_11_0_arm64.whl (13.9 MB)
Collecting regex!=2019.12.17
  Using cached regex-2023.3.23-cp310-cp310-macosx_11_0_arm64.whl (288 kB)
Collecting optax>=0.0.8
  Using cached optax-0.1.4-py3-none-any.whl (154 kB)
Collecting pyctcdecode>=0.4.0
  Using cached pyctcdecode-0.5.0-py2.py3-none-any.whl (39 kB)
Collecting nltk
  Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)
Collecting torch!=1.12.0,>=1.9
  Using cached torch-2.0.0-cp310-none-macosx_11_0_arm64.whl (55.8 MB)
Collecting jax!=0.3.2,<=0.3.6,>=0.2.8
  Using cached jax-0.3.6.tar.gz (936 kB)
  Preparing metadata (setup.py) ... done
Collecting pytest
  Using cached pytest-7.2.2-py3-none-any.whl (317 kB)
Collecting kenlm
  Using cached kenlm-0.1.tar.gz (424 kB)
  Preparing metadata (setup.py) ... done
Collecting GitPython<3.1.19
  Using cached GitPython-3.1.18-py3-none-any.whl (170 kB)
Collecting sudachipy>=0.6.6
  Using cached SudachiPy-0.6.7-cp310-cp310-macosx_10_12_universal2.whl (2.4 MB)
ERROR: Could not find a version that satisfies the requirement decord==0.6.0; extra == "dev" (from transformers[dev]) (from versions: none)
ERROR: No matching distribution found for decord==0.6.0; extra == "dev"
```

### Expected behavior

Development dependencies should be installed without error.

Comments:
cc @amyeroberts and @nateraw 

Hmm maybe its the Python version? Some issue mentioning that here: https://github.com/dmlc/gluon-cv/issues/1539

The issue is mainly because of Apple Silicon. `decord` does not provide any built wheels for apple silicon, and hence cannot be found using pip. I had to build it from source and then install the python bindings.

Similar issue arises for `tensorflow-text` since it also does not provide any built wheels for apple silicon, and has to be built from scratch. I used a community built wheel from [here](https://github.com/sun1638650145/Libraries-and-Extensions-for-TensorFlow-for-Apple-Silicon/releases).

I think the docs should be updated to account for these issues.

I see. we should add a note then that in some cases you may need to install `decord` from source, and link to any related issues.

Or, perhaps we migrate fully to `pyav` at this point, given we started to do that here: #21572 (since decord is no longer being actively maintained and these issues will never go away).

WDYT?

My dev setup on apple silicon failed with 
```
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement jaxlib<=0.3.6,>=0.1.65; extra == "dev" (from transformers[dev]) (from versions: 0.3.24, 0.3.25, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.6, 0.4.7)
ERROR: No matching distribution found for jaxlib<=0.3.6,>=0.1.65; extra == "dev"
```

After a bit of hunting found success by installing jaxlib through conda : https://github.com/google/jax/issues/5501#issuecomment-1032891169 

Maybe it helps someone 




@nateraw Migrating fully to `pyav` is indeed the correct thing to do since the migration has already begun.

There are still other issues with setting up dev env on apple silicon, and setting it up correctly should be part of docs. It took me some time to correctly install the entire `dev` env. Following is the list of issues and solutions that worked for me for `python 3.9`:

- `decord`
    - `Problem`: No prebuilt wheels for apple silicon.
    - `Solution`: Building locally, and installing python bindings. 
    - `Action`: Complete migration to `pyav`.
- `tensorflow`, `tensorflow-*`
    - `Problem`: `tensorflow`, `tensorflow-*` are not directly installable for macOS. 
    - `Solution`: Need to install `tensorflow-deps` from conda apple channel. This has already been highlighted in a previous issue #18355. Install `tensorflow-macos`( instead of `tensorflow`).
    - `Action`: 
          - `setup.py`should be able to detect if the dev env is being setup on apple silicon, and install `tensorflow-macos`instead of `tensorflow`. 
          - Docs should account for setting up dev env on apple silicon.
- `pip`
   -  `Problem`: `pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 20000`
        - Not sure if others have faced this issue, but this could also just be my machine.
        -  `pip` was unable to resolve the dev dependencies in about 6 hours, and failed with the above error.
        - This could be because some package versions are not aligned for apple silicon correctly.
   - `Solution`: 
        - Follow the steps mentioned in issue #18355 .
        - install and build record
        - install `tensorflow-text` from [here](https://github.com/sun1638650145/Libraries-and-Extensions-for-TensorFlow-for-Apple-Silicon/releases)
        - Run `pip install -e ".[dev]" --use-deprecated=legacy-resolver`
        - Run `pip3 install lxml`
        - Some package versions were still not correctly resolved, and tests were failing, along with the `make fixup`, etc commands. So, I had to install specific versions of the following packages from pip: `jax==0.4.7`, `numpy==1.23`.
        - `onnx` had to be installed using instructions [here](https://github.com/onnx/onnx/issues/3621#issuecomment-890351498).
    - `Action`:
        - I need some sanity check for the pip errors. Am I the only one who faced this or is this reproducible for other `M2`users.
        - Possible update of `setup.py` to account for apple silicon, and guides in docs.


After these steps, the dev environment is finally working for me along with tests, and other commands. But, this took way too long to get working. I also tried setting up `vscode devcontainer` for dev dependencies, but jaxlib still does not provide wheels for `manylinux aarch64` wheels yet.

How can we proceed here? I want to actively contribute towards solving these issues :)


@mayankagarwals  Did you face the issues highlighted in above comment? [Link](https://github.com/huggingface/transformers/issues/22617#issuecomment-1501078224)

For most contributions, you only need to run `pip install -e .["quality"]`, but we do need TensorFlow and Jax for the complete quality checks (as we have many models in both those frameworks). But if you make contributions that do not require them (e.g. you're not touching a TensorFlow or Flax model) you will be fine.

@nateraw I could already start working on completing the migration from `decord` to `pyav` .

What do you think about the other set of problems I point out?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @sgugger, I tried installing transformers using `pip install -e '.[quality]'` and it failed to build the wheel for `safetensors`.

I am facing this on M1 Pro, Python 3.10

<img width="1507" alt="Screenshot 2023-08-12 at 7 45 40 PM" src="https://github.com/huggingface/transformers/assets/26519539/08e5ddb8-af1d-423c-9acf-dd2b3a44bb95">


cc @Narsil 

@tanaymeh 

Can you try doing `pip install -U safetensors` to confirm the bug occurs there ?
Version `0.3.2` should be precompiled for Python 3.10 macos 13 on ARM (m1)...
https://pypi.org/project/safetensors/0.3.2/#files

If the command does work, you should just use that.
If that doesn't work, I'll investigate why your environment is not picking it up.

As of [`transformers==4.36.2`](https://github.com/huggingface/transformers/tree/v4.36.2), on macOS 13.5.2 on a MacBook Pro with an M1 chip with Python 3.11.7, I was able to:

```bash
pip install -e .[quality,testing,docs_specific,sentencepiece,torch]
```

This is enough to run _some_ of the `pytest` cases without `ImportError`s

I am still unable to install `tensorflow-text` as of its latest `v2.15.0`.  https://github.com/tensorflow/text/issues/823 links to some instructions on how to manually build, for those who need it