Repository: espnet/espnet
Pull request: Docker Updates
Updates on docker to fit the installation of ESpnet.
This will also fix PR #2508.

Problem statement:
Docker build error
# error content
 I got the error while building docker image
I run these commands.
- command
```
cd docker
./build.sh fully_local cpu
```
but the error was occared.
- the error sentence
```
Step 14/16 : RUN if [ -z "$( nvcc -V )" ]; then         echo "Build without CUDA" &&         MY_OPTS="CUPY_VERSION=";     else         echo "Build with CUDA" &&         export CFLAGS="-I${CUDA_HOME}/include ${CFLAGS}" &&         MY_OPTS="CUDA_VERSION=${CUDA_VER}" &&         sed -i 's|_install.py|_install.py --no-cuda|g' Makefile &&         sed -i 's|which nvidia-smi|which nvcc|g' Makefile;      fi;     if [ "${CUDA_VER}" = "10.1" ]; then         MY_OPTS="${MY_OPTS} TH_VERSION=1.3.1";      fi;     echo "Make with options ${MY_OPTS}";     make KALDI=/kaldi ${MY_OPTS}
 ---> Running in 59da3674b2df
/bin/sh: 1: nvcc: not found
Build without CUDA
Make with options CUPY_VERSION=
expr: syntax error
test -f kaldi/egs/wsj/s5/utils/parse_options.sh || { echo -e "Error: Put Kaldi here!\n    $ ln -s <kaldi-root> kaldi"; exit 1; }
-e Error: Put Kaldi here!
    $ ln -s <kaldi-root> kaldi
Makefile:65: recipe for target 'kaldi' failed
make: *** [kaldi] Error 1
The command '/bin/sh -c if [ -z "$( nvcc -V )" ]; then         echo "Build without CUDA" &&         MY_OPTS="CUPY_VERSION=";     else         echo "Build with CUDA" &&         export CFLAGS="-I${CUDA_HOME}/include ${CFLAGS}" &&         MY_OPTS="CUDA_VERSION=${CUDA_VER}" &&         sed -i 's|_install.py|_install.py --no-cuda|g' Makefile &&         sed -i 's|which nvidia-smi|which nvcc|g' Makefile;      fi;     if [ "${CUDA_VER}" = "10.1" ]; then         MY_OPTS="${MY_OPTS} TH_VERSION=1.3.1";      fi;     echo "Make with options ${MY_OPTS}";     make KALDI=/kaldi ${MY_OPTS}' returned a non-zero code: 2
```

# My environment:
OS: macOS Catalina. (I also tried on Ubuntu18.04, but the error was also happend.)

# I followed the error message, but other error occared.
I tried to add the command to Dokcerfile, but other error occard.
- added the command to `docker/prebuilt/local/Dockerfile`
```
RUN ln -s /kaldi kaldi
```

- another error
```
Step 15/17 : RUN if [ -z "$( nvcc -V )" ]; then         echo "Build without CUDA" &&         MY_OPTS="CUPY_VERSION=";     else         echo "Build with CUDA" &&         export CFLAGS="-I${CUDA_HOME}/include ${CFLAGS}" &&         MY_OPTS="CUDA_VERSION=${CUDA_VER}" &&         sed -i 's|_install.py|_install.py --no-cuda|g' Makefile &&         sed -i 's|which nvidia-smi|which nvcc|g' Makefile;      fi;     if [ "${CUDA_VER}" = "10.1" ]; then         MY_OPTS="${MY_OPTS} TH_VERSION=1.3.1";      fi;     echo "Make with options ${MY_OPTS}";     make KALDI=/kaldi ${MY_OPTS}
 ---> Running in fabc99f97da5
Build without CUDA
Make with options CUPY_VERSION=
/bin/sh: 1: nvcc: not found
expr: syntax error
test -f activate_python.sh || { echo "Error: Run ./setup_python.sh or ./setup_anaconda.sh"; exit 1; }
Error: Run ./setup_python.sh or ./setup_anaconda.sh
Makefile:68: recipe for target 'activate_python.sh' failed
make: *** [activate_python.sh] Error 1
The command '/bin/sh -c if [ -z "$( nvcc -V )" ]; then         echo "Build without CUDA" &&         MY_OPTS="CUPY_VERSION=";     else         echo "Build with CUDA" &&         export CFLAGS="-I${CUDA_HOME}/include ${CFLAGS}" &&         MY_OPTS="CUDA_VERSION=${CUDA_VER}" &&         sed -i 's|_install.py|_install.py --no-cuda|g' Makefile &&         sed -i 's|which nvidia-smi|which nvcc|g' Makefile;      fi;     if [ "${CUDA_VER}" = "10.1" ]; then         MY_OPTS="${MY_OPTS} TH_VERSION=1.3.1";      fi;     echo "Make with options ${MY_OPTS}";     make KALDI=/kaldi ${MY_OPTS}' returned a non-zero code: 2
```
After that, I tried to fix more following the error message, but error was still existed.

- added the command to `docker/prebuilt/local/Dockerfile`
```
RUN ./setup_python.sh $(which python3)
```

- another error
```
Step 15/18 : RUN ./setup_python.sh $(which python3)
 ---> Running in 83f7d133da5d
/usr/bin/pip3 doesn't exist.
The command '/bin/sh -c ./setup_python.sh $(which python3)' returned a non-zero code: 1
```
# I tried to fix and build was successed but installing ESPNet was imcomplete

- added the command to `docker/prebuilt/local/Dockerfile`
```
RUN ./setup_anaconda.sh
```

After that, building images is sccessed with no error.
but there are not all of content in ESPNet
e.g. espnet/espnet, espnet/utils and espnet/egs etc. are not existed.

```
ls espnet
```
```
CONTRIBUTING.md  README.md  doc   espnet.egg-info  setup.cfg  test_utils
LICENSE          ci         egs2  espnet2          setup.py   tools
```


**If someone knows how to fix it, please advice me.**

Hints from issue discussion:
@muramasa2 THe problem is that the docker was not updated for the new environment of ESPnet.
It has been new changes in the installation.
I will upload the fixes for all docker containers.
@Fhrozen I appreciate your support, thanks.

Commit message: Update docker/prebuilt/devel/Dockerfile

Patch to review:
diff --git a/docker/prebuilt/devel/Dockerfile b/docker/prebuilt/devel/Dockerfile
index 14d396ccd9b..55ddfdcdf90 100644
--- a/docker/prebuilt/devel/Dockerfile
+++ b/docker/prebuilt/devel/Dockerfile
@@ -18,21 +18,25 @@ WORKDIR /espnet/tools
 # Disable cupy test
 # Docker build does not load libcuda.so.1
 # Replace nvidia-smi for nvcc because docker does not load nvidia-smi
-RUN if [ -z "$( nvcc -V )" ]; then \
+RUN if [ -z "$( which nvcc )" ]; then \
         echo "Build without CUDA" && \
-        MY_OPTS="CUPY_VERSION="; \
+        MY_OPTS='CUPY_VERSION=""  TH_VERSION=1.6.0'; \
     else \
         echo "Build with CUDA" && \
+        # Docker containers cannot load cuda libs during build.
+        # So, their checks on cuda packages are disabled.
+        sed -i '200s|install.py|install.py --no-cuda --no-cupy |' Makefile && \
         export CFLAGS="-I${CUDA_HOME}/include ${CFLAGS}" && \
         MY_OPTS="CUDA_VERSION=${CUDA_VER}" && \
-        sed -i 's|_install.py|_install.py --no-cuda|g' Makefile && \
-        sed -i 's|which nvidia-smi|which nvcc|g' Makefile;  \
+        . ./setup_cuda_env.sh /usr/local/cuda;  \
     fi; \ 
     if [ "${CUDA_VER}" = "10.1" ]; then \
-        # Pytorch 1.3.1 is not supported by warpctc
-        MY_OPTS="${MY_OPTS} TH_VERSION=1.3.1";  \
+        # warpctc is not supported from Pytorch 1.3.1
+        MY_OPTS="${MY_OPTS} TH_VERSION=1.6.0";  \
     fi; \
-    echo "Make with options ${MY_OPTS}"; \
+    echo "Make with options ${MY_OPTS}" && \
+    ln -s /kaldi ./ && \
+    ./setup_anaconda.sh /miniconda espnet 3.7.4 && \
     make KALDI=/kaldi ${MY_OPTS}
 
 RUN rm -rf ../espnet
diff --git a/docker/prebuilt/devel/gpu/10.1/cudnn7/Dockerfile b/docker/prebuilt/devel/gpu/10.1/cudnn7/Dockerfile
index b0adc4c02e2..adcdace6628 100644
--- a/docker/prebuilt/devel/gpu/10.1/cudnn7/Dockerfile
+++ b/docker/prebuilt/devel/gpu/10.1/cudnn7/Dockerfile
@@ -37,7 +37,7 @@ ENV CUDA_HOME /usr/local/cuda
 
 ## FROM CUDA 10.1 runtime [https://gitlab.com/nvidia/cuda/blob/ubuntu18.04/10.1/runtime/Dockerfile]
 
-ENV NCCL_VERSION 2.4.2
+ENV NCCL_VERSION 2.7.8
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
         cuda-libraries-$CUDA_PKG_VERSION \
@@ -49,13 +49,21 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 ## FROM CUDA 10.1 devel [https://gitlab.com/nvidia/cuda/blob/ubuntu18.04/10.1/devel/Dockerfile]
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-        cuda-libraries-dev-$CUDA_PKG_VERSION \
         cuda-nvml-dev-$CUDA_PKG_VERSION \
-        cuda-minimal-build-$CUDA_PKG_VERSION \
         cuda-command-line-tools-$CUDA_PKG_VERSION \
-        libnccl-dev=$NCCL_VERSION-1+cuda10.1 && \
+        cuda-nvprof-$CUDA_PKG_VERSION \
+        cuda-npp-dev-$CUDA_PKG_VERSION \
+        cuda-libraries-dev-$CUDA_PKG_VERSION \
+        cuda-minimal-build-$CUDA_PKG_VERSION \
+        libcublas-dev=10.2.1.243-1 \
+        libnccl-dev=2.7.8-1+cuda10.1 && \
+    apt-mark hold libnccl-dev &&  \
     rm -rf /var/lib/apt/lists/*
 
+# apt from auto upgrading the cublas package. See https://gitlab.com/nvidia/container-images/cuda/-/issues/88
+RUN apt-mark hold libcublas-dev
+
+
 ENV LIBRARY_PATH /usr/local/cuda/lib64/stubs
 
 ## FROM CUDA 10.1-CUDNN 7 devel
diff --git a/docker/prebuilt/local/Dockerfile b/docker/prebuilt/local/Dockerfile
index 07cf5aa9e3a..23c3d3e5ee9 100644
--- a/docker/prebuilt/local/Dockerfile
+++ b/docker/prebuilt/local/Dockerfile
@@ -22,24 +22,27 @@ RUN cd espnet && \
 # Install espnet
 WORKDIR /espnet/tools
 
-# Disable cupy test
-# Docker build does not load libcuda.so.1
 # Replace nvidia-smi for nvcc because docker does not load nvidia-smi
-RUN if [ -z "$( nvcc -V )" ]; then \
+RUN if [ -z "$( which nvcc )" ]; then \
         echo "Build without CUDA" && \
-        MY_OPTS="CUPY_VERSION="; \
+        MY_OPTS='CUPY_VERSION=""  TH_VERSION=1.6.0'; \
     else \
         echo "Build with CUDA" && \
+        # Disable cupy test
+        # Docker build does not load libcuda.so.1
+        # So, they checks on cuda package are disable.
+        sed -i '200s|install.py|install.py --no-cuda --no-cupy |' Makefile && \
         export CFLAGS="-I${CUDA_HOME}/include ${CFLAGS}" && \
         MY_OPTS="CUDA_VERSION=${CUDA_VER}" && \
-        sed -i 's|_install.py|_install.py --no-cuda|g' Makefile && \
-        sed -i 's|which nvidia-smi|which nvcc|g' Makefile;  \
+        . ./setup_cuda_env.sh /usr/local/cuda;  \
     fi; \ 
     if [ "${CUDA_VER}" = "10.1" ]; then \
-        # Pytorch 1.3.1 is not supported by warpctc
-        MY_OPTS="${MY_OPTS} TH_VERSION=1.3.1";  \
+        # warpctc is not supported from Pytorch 1.3.1
+        MY_OPTS="${MY_OPTS} TH_VERSION=1.6.0";  \
     fi; \
-    echo "Make with options ${MY_OPTS}"; \
+    echo "Make with options ${MY_OPTS}" && \
+    ln -s /kaldi ./ && \
+    ./setup_anaconda.sh /miniconda espnet 3.7.4 && \
     make KALDI=/kaldi ${MY_OPTS}
 
 RUN rm -rf ../espnet
diff --git a/docker/prebuilt/runtime/Dockerfile b/docker/prebuilt/runtime/Dockerfile
index b8dc7cb5409..1e1bc66a12c 100644
--- a/docker/prebuilt/runtime/Dockerfile
+++ b/docker/prebuilt/runtime/Dockerfile
@@ -17,10 +17,12 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get -y install --no-ins
         cmake \
         curl \
         flac \
+        ffmpeg \
         gawk \
         gfortran \
         git \
         libtool \
+        libsndfile1-dev \
         python2.7 \
         python3 \
         sox \
@@ -49,6 +51,10 @@ RUN cd /kaldi/tools && \
     make depend -j${NUM_BUILD_CORES} && \
     make -j${NUM_BUILD_CORES} && \
     find /kaldi/src -name "*.o" -exec rm -f {} \; && \
-    find /kaldi/src -name "*.o" -exec rm -f {} \; 
+    find /kaldi/src -name "*.o" -exec rm -f {} \;
+
+RUN wget --tries=3 "https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh" -O miniconda.sh && \
+    bash miniconda.sh -b -p /miniconda && \
+    rm miniconda.sh
 
 WORKDIR /
\ No newline at end of file


Code review comments:
diff_hunk: @@ -22,24 +22,27 @@ RUN cd espnet && \
 # Install espnet
 WORKDIR /espnet/tools
 
-# Disable cupy test
-# Docker build does not load libcuda.so.1
 # Replace nvidia-smi for nvcc because docker does not load nvidia-smi
-RUN if [ -z "$( nvcc -V )" ]; then \
+RUN if [ -z "$( which nvcc )" ]; then \
         echo "Build without CUDA" && \
-        MY_OPTS="CUPY_VERSION="; \
+        MY_OPTS='CUPY_VERSION=""  TH_VERSION=1.6.0'; \
     else \
         echo "Build with CUDA" && \
+        # Disable cupy test
+        # Docker build does not load libcuda.so.1
+        # So, they checks on cuda package are disable.
path: docker/prebuilt/local/Dockerfile
text: @user1:
```suggestion
        # So, their checks on cuda packages are disabled.
```