Sentencepiece dependency causing docker build to fail
Author: joshzwiebel
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:3.4.0
- Platform:Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.7.0 no gpu
- Tensorflow version (GPU?): N/A
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


## Information

Question answering pipelines is the feature I am using. There is a problem with the sentencepiece dependency of transformers. It cannot find the correct package when installing and causing the build process to fail

The problem arises when using:
Downloading the transformers library onto a docker container running ubuntu.

The tasks I am working on is:
Uploadding a transformers script to AWS fargate

## To reproduce

Steps to reproduce the behavior:

1. Create a project
2. Try to create docker container using dockerfile attached below

I have attached the relevant parts of my dockerfile below
Dockerfile

```
FROM ubuntu:18.04

RUN mkdir /usr/app
WORKDIR /usr/app

# Add and install Python modules
COPY requirements.txt ./
RUN apt-get update
RUN apt-get -y install python3
RUN apt-get -y install python3-pip
RUN pip3 install virtualenv
ENV VIRTUAL_ENV=/venv
RUN virtualenv venv -p python3
ENV PATH="VIRTUAL_ENV/bin:$PATH"

RUN pip3 install transformers[torch]
RUN pip3 install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip3 install -r requirements.txt

# Bundle app source
COPY . ./
# Expose
EXPOSE 6000
# Run
CMD ["python", "app.py"]

```
This is the stack trace that comes back from running and trying to build using this dockerfile

```
[+] Building 571.8s (14/17)
 => [internal] load .dockerignore                                                                                             0.0s
 => => transferring context: 34B                                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                                          0.0s
 => => transferring dockerfile: 835B                                                                                          0.0s
 => [internal] load metadata for docker.io/library/ubuntu:18.04                                                               0.8s
 => [internal] load build context                                                                                             0.1s
 => => transferring context: 139.29kB                                                                                         0.1s
 => [1/13] FROM docker.io/library/ubuntu:18.04@sha256:646942475da61b4ce9cc5b3fadb42642ea90e5d0de46111458e100ff2c7031e6        0.0s
 => CACHED [2/13] RUN mkdir /usr/app                                                                                          0.0s
 => CACHED [3/13] WORKDIR /usr/app                                                                                            0.0s
 => [4/13] COPY requirements.txt ./                                                                                           0.0s
 => [5/13] RUN apt-get update                                                                                                30.8s
 => [6/13] RUN apt-get -y install python3                                                                                    15.6s
 => [7/13] RUN apt-get -y install python3-pip                                                                               214.0s
 => [8/13] RUN pip3 install virtualenv                                                                                        4.5s
 => [9/13] RUN virtualenv venv -p python3                                                                                     0.8s
 => ERROR [10/13] RUN pip3 install transformers[torch]                                                                      305.1s
------
 > [10/13] RUN pip3 install transformers[torch]:
#14 1.095 Collecting transformers[torch]
#14 1.410   Downloading https://files.pythonhosted.org/packages/2c/4e/4f1ede0fd7a36278844a277f8d53c21f88f37f3754abf76a5d6224f76d4a/
transformers-3.4.0-py3-none-any.whl (1.3MB)
#14 1.897 Collecting numpy (from transformers[torch])
#14 2.507   Downloading https://files.pythonhosted.org/packages/8f/40/ddb5109614aabad67e6fe426b3579a879b7b3cdd375eb27af467c4367ae0/
numpy-1.19.3-cp36-cp36m-manylinux1_x86_64.whl (13.4MB)
#14 5.947 Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers[torch])
#14 5.953 Collecting sentencepiece!=0.1.92 (from transformers[torch])
#14 6.174   Downloading https://files.pythonhosted.org/packages/72/e0/57edbab017a204e9f39448c1717292437a45b5f7cf3a9dbf4a9c026b03c5/
sentencepiece-0.1.94.tar.gz (507kB)
#14 6.575 Collecting sacremoses (from transformers[torch])
#14 6.714   Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/
sacremoses-0.0.43.tar.gz (883kB)
#14 7.158 Collecting requests (from transformers[torch])
#14 7.341   Downloading https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/
requests-2.24.0-py2.py3-none-any.whl (61kB)
#14 7.404 Collecting packaging (from transformers[torch])
#14 7.551   Downloading https://files.pythonhosted.org/packages/46/19/c5ab91b1b05cfe63cccd5cfc971db9214c6dd6ced54e33c30d5af1d2bc43/
packaging-20.4-py2.py3-none-any.whl
#14 7.579 Collecting dataclasses; python_version < "3.7" (from transformers[torch])
#14 7.708   Downloading https://files.pythonhosted.org/packages/e1/d2/6f02df2616fd4016075f60157c7a0452b38d8f7938ae94343911e0fb0b09/
dataclasses-0.7-py3-none-any.whl
#14 7.725 Collecting tokenizers==0.9.2 (from transformers[torch])
#14 8.019   Downloading https://files.pythonhosted.org/packages/7c/a5/78be1a55b2ac8d6a956f0a211d372726e2b1dd2666bb537fea9b03abd62c/
tokenizers-0.9.2-cp36-cp36m-manylinux1_x86_64.whl (2.9MB)
#14 8.732 Collecting regex!=2019.12.17 (from transformers[torch])
#14 9.570   Downloading https://files.pythonhosted.org/packages/87/9f/aad666560082cb11331167cbb31cf0e8bd90af8ea4951436d1fcb2ddde44/
regex-2020.10.28-cp36-cp36m-manylinux1_x86_64.whl (666kB)
#14 9.756 Collecting protobuf (from transformers[torch])
#14 10.02   Downloading https://files.pythonhosted.org/packages/30/79/510974552cebff2ba04038544799450defe75e96ea5f1675dbf72cc8744f/
protobuf-3.13.0-cp36-cp36m-manylinux1_x86_64.whl (1.3MB)
#14 10.36 Collecting tqdm>=4.27 (from transformers[torch])
#14 10.57   Downloading https://files.pythonhosted.org/packages/93/3a/96b3dc293aa72443cf9627444c3c221a7ba34bb622e4d8bf1b5d4f2d9d08/
tqdm-4.51.0-py2.py3-none-any.whl (70kB)
#14 10.60 Collecting torch>=1.0; extra == "torch" (from transformers[torch])
#14 10.79   Downloading https://files.pythonhosted.org/packages/80/2a/58f8078744e0408619c63148f7a2e8e48cf007e4146b74d4bb67c56d161b/
torch-1.7.0-cp36-cp36m-manylinux1_x86_64.whl (776.7MB)
#14 285.6 Collecting click (from sacremoses->transformers[torch])
#14 292.4   Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/
click-7.1.2-py2.py3-none-any.whl (82kB)
#14 292.4 Collecting joblib (from sacremoses->transformers[torch])
#14 292.6   Downloading https://files.pythonhosted.org/packages/fc/c9/f58220ac44a1592f79a343caba12f6837f9e0c04c196176a3d66338e1ea8/
joblib-0.17.0-py3-none-any.whl (301kB)
#14 292.8 Requirement already satisfied: six in /usr/lib/python3/dist-packages (from sacremoses->transformers[torch])
#14 292.8 Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->transformers[torch])
#14 293.0   Downloading https://files.pythonhosted.org/packages/56/aa/4ef5aa67a9a62505db124a5cb5262332d1d4153462eb8fd89c9fa41e5d92/
urllib3-1.25.11-py2.py3-none-any.whl (127kB)
#14 293.0 Collecting chardet<4,>=3.0.2 (from requests->transformers[torch])
#14 293.2   Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/
chardet-3.0.4-py2.py3-none-any.whl (133kB)
#14 293.2 Collecting certifi>=2017.4.17 (from requests->transformers[torch])
#14 293.4   Downloading https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/
certifi-2020.6.20-py2.py3-none-any.whl (156kB)
#14 293.4 Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests->transformers[torch])
#14 293.4 Collecting pyparsing>=2.0.2 (from packaging->transformers[torch])
#14 293.7   Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/
pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
#14 293.7 Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from protobuf->transformers[torch])
#14 293.7 Collecting future (from torch>=1.0; extra == "torch"->transformers[torch])
#14 293.9   Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/
future-0.18.2.tar.gz (829kB)
#14 294.7 Collecting typing-extensions (from torch>=1.0; extra == "torch"->transformers[torch])
#14 294.9   Downloading https://files.pythonhosted.org/packages/60/7a/e881b5abb54db0e6e671ab088d079c57ce54e8a01a3ca443f561ccadb37e/
typing_extensions-3.7.4.3-py3-none-any.whl
#14 294.9 Building wheels for collected packages: sentencepiece, sacremoses, future
#14 294.9   Running setup.py bdist_wheel for sentencepiece: started
#14 295.6   Running setup.py bdist_wheel for sentencepiece: finished with status 'error'
#14 295.6   Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o837wqyj/sent
encepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __
file__, 'exec'))" bdist_wheel -d /tmp/tmprwjlzrwrpip-wheel- --python-tag cp36:
#14 295.6   /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
#14 295.6     warnings.warn(msg)
#14 295.6   running bdist_wheel
#14 295.6   running build
#14 295.6   running build_py
#14 295.6   creating build
#14 295.6   creating build/lib.linux-x86_64-3.6
#14 295.6   creating build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 295.6   running build_ext
#14 295.6   /bin/sh: 1: pkg-config: not found
#14 295.6   ./build_bundled.sh: 8: ./build_bundled.sh: git: not found
#14 295.6   ./build_bundled.sh: 10: ./build_bundled.sh: git: not found
#14 295.6   ./build_bundled.sh: 12: cd: can't cd to sentencepiece
#14 295.6   ./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
#14 295.6   make: *** No targets specified and no makefile found.  Stop.
#14 295.6   make: *** No rule to make target 'install'.  Stop.
#14 295.6   env: 'pkg-config': No such file or directory
#14 295.6   Failed to find sentencepiece pkg-config
#14 295.6
#14 295.6   ----------------------------------------
#14 295.6   Failed building wheel for sentencepiece
#14 295.6   Running setup.py clean for sentencepiece
#14 295.8   Running setup.py bdist_wheel for sacremoses: started
#14 296.3   Running setup.py bdist_wheel for sacremoses: finished with status 'done'
#14 296.3   Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45
#14 296.4   Running setup.py bdist_wheel for future: started
#14 297.2   Running setup.py bdist_wheel for future: finished with status 'done'
#14 297.2   Stored in directory: /root/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
#14 297.3 Successfully built sacremoses future
#14 297.3 Failed to build sentencepiece
#14 297.3 Installing collected packages: numpy, sentencepiece, click, joblib, regex, tqdm, sacremoses, urllib3, chardet, certifi, r
equests, pyparsing, packaging, dataclasses, tokenizers, protobuf, future, typing-extensions, torch, transformers
#14 301.3   Running setup.py install for sentencepiece: started
#14 301.7     Running setup.py install for sentencepiece: finished with status 'error'
#14 301.7     Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o837wqyj/se
ntencepiece/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code,
__file__, 'exec'))" install --record /tmp/pip-7ji4iyud-record/install-record.txt --single-version-externally-managed --compile:
#14 301.7     /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
#14 301.7       warnings.warn(msg)
#14 301.7     running install
#14 301.7     running build
#14 301.7     running build_py
#14 301.7     creating build
#14 301.7     creating build/lib.linux-x86_64-3.6
#14 301.7     creating build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-3.6/sentencepiece
#14 301.7     running build_ext
#14 301.7     /bin/sh: 1: pkg-config: not found
#14 301.7     mkdir: cannot create directory 'bundled': File exists
#14 301.7     ./build_bundled.sh: 8: ./build_bundled.sh: git: not found
#14 301.7     ./build_bundled.sh: 10: ./build_bundled.sh: git: not found
#14 301.7     ./build_bundled.sh: 12: cd: can't cd to sentencepiece
#14 301.7     mkdir: cannot create directory 'build': File exists
#14 301.7     ./build_bundled.sh: 15: ./build_bundled.sh: cmake: not found
#14 301.7     make: *** No targets specified and no makefile found.  Stop.
#14 301.7     make: *** No rule to make target 'install'.  Stop.
#14 301.7     env: 'pkg-config': No such file or directory
#14 301.7     Failed to find sentencepiece pkg-config
#14 301.7
#14 301.7     ----------------------------------------
#14 302.4 Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-o837wqyj/sentencepiece/setup.py';f=
getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" inst
all --record /tmp/pip-7ji4iyud-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in
 /tmp/pip-build-o837wqyj/sentencepiece/
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c pip3 install transformers[tor
ch]]: runc did not terminate sucessfully

```
## Expected behavior

I expect transformers to be downloaded and allow me to access it from the docker container.


Comments:
This will be fixed when #8073 is merged.

Is there any help in terms of what version to pin to in order to avoid this? This is currently a huge blocker on my end.

On the sentencepiece side I don’t know (you can open an issue on their side to ask) but on the `transformers` side we are actively working on removing the hard dependency on sentencepiece and we estimate we should have a new release removing this dependency around the end of next week.

Cc @n1t0 and @Narsil whose work on `tokenizers` is essential to unlock this.

Great! Thanks for the info!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.