No module named 'transformers' after installing from source
Author: fishfree
Pull request: no

### System Info

Ubuntu 22.04 in Windwos WSL 2.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I just followed the doc [here](https://huggingface.co/docs/transformers/installation#install-from-source). However, error occured as below:
```
wu@DESKTOP-COM:~/llama.cpp/transformers$ python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers'
```

### Expected behavior

No error occured.

Comments:
same error


Having the same issue as well.

Me too. Not sure what is going on, but it looks like in site-packages, the transformers-4.28.0.dev0.dist-info directory is created, but not the transformers directory itself!

... and confirmed, if I roll back using
`git checkout 2194943a3443b924e4cd09f37402230b771008f0`
then everything installs fine. Something seems to have broken in the past 3-4 commits.

same

Steps to reproduce (after uninstalling any version of transformers that you might have):
1. `git clone  https://github.com/huggingface/transformers.git`
2. `cd transformers`
3. `pip install .`
4. `python3 -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))"`
Resulting error
```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers'
```

It looks like the change that broke things is https://github.com/huggingface/transformers/pull/22539. If I roll back to the previous change to setup.py, the install works.
git checkout 80d1319e1b9dde71b8af641ad1427113058a0af7  --> pip3 install . --> WORKS
git checkout 4169dc84bf0072a26f10096a187907d661dcc383  --> pip3 install . --> FAILS

Maybe there is a new installation method?


Thanks for letting us know. I guess that's what happens when you try to clean up to follow the official PEP rules... We'll revert the PR!

I cannot reproduce this in a virtual environment. Maybe you are using the system `python` and `pip` on Ubuntu, which are installed in `dist-packages` rather than `site-packages`. There is a similar issue oobabooga/text-generation-webui#753. Upgrade your `pip` and `setuptools`, or use a virtual environment will resolve this.

In the documentation [Installation](https://huggingface.co/docs/transformers/installation#installation):

> # Install with pip
> 
> You **should** install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you’re unfamiliar with Python virtual environments, take a look at this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.
> 
> Start by creating a virtual environment in your project directory:
> 
> ```bash
> python -m venv .env
> 
> ```

The issue here is the users do not follow the installation guide for using a virtual environment. We may need to add `pip3 install --upgrade pip setuptools` in the [Install from source](https://huggingface.co/docs/transformers/installation#install-from-source) documentation.

> # Install from source
> 
> Install 🤗 Transformers from source with the following command:
> 
> ```bash
> pip install git+https://github.com/huggingface/transformers
> ```

to

```bash
pip install --ugprade pip setuptools  # reinstall pip and do not use the apt packages
pip install git+https://github.com/huggingface/transformers
```

------

Solution 1: upgrade (reinstall) `pip` and `setuptools` when using the system apt package.

```bash
docker run -it --rm -h ubuntu --pull always ubuntu:22.04
apt update && apt install git python3-dev python3-pip -y
python3 -m pip install --upgrade pip setuptools
python3 -m pip install git+https://github.com/huggingface/transformers@4169dc84bf0072a26f10096a187907d661dcc383
```

<details>
<summary>Outputs: upgrade (reinstall) `pip` and `setuptools`</summary>

```console
$ docker run -it --rm -h ubuntu --pull always ubuntu:22.04
22.04: Pulling from library/ubuntu
Digest: sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21
Status: Image is up to date for ubuntu:22.04
root@ubuntu:/# apt update && apt install git python3-dev python3-pip -y

root@ubuntu:/# which -a python3
/usr/bin/python3
/bin/python3
root@ubuntu:/# which -a pip3
/usr/bin/pip3
/bin/pip3
root@ubuntu:/# python3 -m pip install --upgrade pip setuptools
Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
Collecting pip
  Downloading pip-23.0.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 817.6 kB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
Collecting setuptools
  Downloading setuptools-67.6.1-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 540.5 kB/s eta 0:00:00
Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-23.0.1 setuptools-67.6.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@ubuntu:/# which -a pip3
/usr/local/bin/pip3
/usr/bin/pip3
/bin/pip3
root@ubuntu:/# python3 -m pip install git+https://github.com/huggingface/transformers
Collecting git+https://github.com/huggingface/transformers@4169dc84bf0072a26f10096a187907d661dcc383
  Cloning https://github.com/huggingface/transformers (to revision 4169dc84bf0072a26f10096a187907d661dcc383) to /tmp/pip-req-build-w_o1neea
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-w_o1neea
  Running command git rev-parse -q --verify 'sha^4169dc84bf0072a26f10096a187907d661dcc383'
  Running command git fetch -q https://github.com/huggingface/transformers 4169dc84bf0072a26f10096a187907d661dcc383
  Running command git checkout -q 4169dc84bf0072a26f10096a187907d661dcc383
  Resolved https://github.com/huggingface/transformers to commit 4169dc84bf0072a26f10096a187907d661dcc383
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (2.28.2)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (0.13.3)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (3.11.0)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (4.65.0)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (1.24.2)
Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (0.13.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (23.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (2023.3.23)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.28.0.dev0) (6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.28.0.dev0) (4.5.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (3.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.28.0.dev0) (1.26.15)
Building wheels for collected packages: transformers
  Building wheel for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6862948 sha256=b8dbe24b1d39a4ae836e24e0b4b7ab27b4e024408b7129a4b1c4aad4a41fc4d7
  Stored in directory: /root/.cache/pip/wheels/98/63/05/ec5c37d387d2db776a20dac49e1b830aca7fbc2394956367ad
Successfully built transformers
Installing collected packages: transformers
Successfully installed transformers-4.28.0.dev0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@ubuntu:/# python3 -c 'import transformers; print(transformers.__version__)'
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
4.28.0.dev0
```

</details>

------

Solution 2: use a virtual environment (already there in the documentation).

```bash
docker run -it --rm -h ubuntu --pull always ubuntu:22.04
apt update && apt install git python3-dev python3-venv -y
python3 -m venv venv
source venv/bin/activate
python3 -m pip install git+https://github.com/huggingface/transformers@4169dc84bf0072a26f10096a187907d661dcc383
```

<details>
<summary>Outputs: use virtual environment</summary>

```console
$ docker run -it --rm -h ubuntu --pull always ubuntu:22.04
22.04: Pulling from library/ubuntu
Digest: sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21
Status: Image is up to date for ubuntu:22.04
root@ubuntu:/# apt update && apt install git python3-dev python3-venv -y

root@ubuntu:/# which -a python3
/usr/bin/python3
/bin/python3
root@ubuntu:/# which -a pip3
/usr/bin/pip3
/bin/pip3
root@ubuntu:/# python3 -m venv venv
root@ubuntu:/# source venv/bin/activate
(venv) root@ubuntu:/# which -a python3
/venv/bin/python3
/usr/bin/python3
/bin/python3
(venv) root@ubuntu:/# which -a pip3   
/venv/bin/pip3
/usr/bin/pip3
/bin/pip3
(venv) root@ubuntu:/# python3 -m pip install git+https://github.com/huggingface/transformers@4169dc84bf0072a26f10096a187907d661dcc383
Collecting git+https://github.com/huggingface/transformers@4169dc84bf0072a26f10096a187907d661dcc383
  Cloning https://github.com/huggingface/transformers (to revision 4169dc84bf0072a26f10096a187907d661dcc383) to /tmp/pip-req-build-u7lmhx_v
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-u7lmhx_v
  Running command git rev-parse -q --verify 'sha^4169dc84bf0072a26f10096a187907d661dcc383'
  Running command git fetch -q https://github.com/huggingface/transformers 4169dc84bf0072a26f10096a187907d661dcc383
  Running command git checkout -q 4169dc84bf0072a26f10096a187907d661dcc383
  Resolved https://github.com/huggingface/transformers to commit 4169dc84bf0072a26f10096a187907d661dcc383
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy>=1.17
  Using cached numpy-1.24.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting regex!=2019.12.17
  Using cached regex-2023.3.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
  Using cached tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting packaging>=20.0
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting pyyaml>=5.1
  Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Collecting huggingface-hub<1.0,>=0.11.0
  Using cached huggingface_hub-0.13.4-py3-none-any.whl (200 kB)
Collecting filelock
  Using cached filelock-3.11.0-py3-none-any.whl (10.0 kB)
Collecting tqdm>=4.27
  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting typing-extensions>=3.7.4.3
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Building wheels for collected packages: transformers
  Building wheel for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6862948 sha256=24db4f2655cd212b0097dd4fd88f2bcad3e1236a0bf700988eefdaad9583d0e9
  Stored in directory: /root/.cache/pip/wheels/98/63/05/ec5c37d387d2db776a20dac49e1b830aca7fbc2394956367ad
Successfully built transformers
Installing collected packages: tokenizers, urllib3, typing-extensions, tqdm, regex, pyyaml, packaging, numpy, idna, filelock, charset-normalizer, certifi, requests, huggingface-hub, transformers
Successfully installed certifi-2022.12.7 charset-normalizer-3.1.0 filelock-3.11.0 huggingface-hub-0.13.4 idna-3.4 numpy-1.24.2 packaging-23.0 pyyaml-6.0 regex-2023.3.23 requests-2.28.2 tokenizers-0.13.3 tqdm-4.65.0 transformers-4.28.0.dev0 typing-extensions-4.5.0 urllib3-1.26.15
(venv) root@ubuntu:/# python3 -c 'import transformers; print(transformers.__version__)'
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
4.28.0.dev0
```

</details>