How to creat Docker image from pytorch source

username: Sumac
cooked: <p>I hope to make docker image for old GPU with pytorch1.8.<br>
In this case, I should build pytorch from source.<br>
So I refered official docs and tried making docker image.<br>
But my docker image can’t detect GPU. (<code>cuda.is_availabel()</code> return <code>False</code>)</p>
<p>My system environment is as follows:<br>
OS	: Ubuntu18.04<br>
GPU	: Tesla K40C<br>
CUDA	: 10.2<br>
Driver	: 440.118.02<br>
Docker	: 19.03.12</p>
<p>The commands used for Dockerfile are as follows:</p>
<pre><code class="lang-auto">FROM nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04

RUN apt-get update \
    &amp;&amp; apt-get install -y build-essential \
    &amp;&amp; apt-get install -y ca-certificates \
    &amp;&amp; apt-get install -y ccache \
    &amp;&amp; apt-get install -y cmake \
    &amp;&amp; apt-get install -y curl \
    &amp;&amp; apt-get install -y file \
    &amp;&amp; apt-get install -y sudo \
    &amp;&amp; apt-get install -y git \
    &amp;&amp; apt-get install -y locales \
    &amp;&amp; locale-gen ja_JP.UTF-8
ENV LANG ja_JP.UTF-8
ENV LANGUAGE ja_JP:ja
ENV LC_ALL=ja_JP.UTF-8
RUN localedef -f UTF-8 -i ja_JP ja_JP.utf8

# Install nodejs
RUN curl -sL https://deb.nodesource.com/setup_14.x | sed ’s|https://|http://|’ | bash - \
    &amp;&amp; sudo apt-get install -y nodejs

# Install MeCab, IPA, NEologd
RUN apt-get install -y mecab \
    &amp;&amp; apt-get install -y libmecab-dev \
    &amp;&amp; apt-get install -y mecab-ipadic \
    &amp;&amp; apt-get install -y mecab-ipadic-utf8
RUN git clone --depth 1 https://github.com/neologd/mecab-ipadic-neologd.git \
    &amp;&amp; mecab-ipadic-neologd/bin/install-mecab-ipadic-neologd -n -y \
    &amp;&amp; cp /etc/mecabrc /usr/local/etc/

# remove files
RUN apt-get clean \
    &amp;&amp; rm -rf /var/lib/apt/lists/*

# install miniforge
COPY Miniforge3-Linux-x86_64.sh /usr/local/src
RUN bash /usr/local/src/Miniforge3-Linux-x86_64.sh -b
ENV PATH $PATH:/root/miniforge3/bin
RUN conda install python=3.8 \
    &amp;&amp; conda install -c conda-forge jupyterlab

# install packages by pip
COPY requirements_pip.txt /tmp
COPY requirements.txt /tmp
RUN pip3 install --upgrade pip \
    &amp;&amp; pip3 install --no-cache-dir -r /tmp/requirements_pip.txt \
    &amp;&amp; rm -rf ~/.cache/pip

# install packages by conda
RUN conda install --file /tmp/requirements.txt \
    &amp;&amp; conda install astunparse numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses \
    &amp;&amp; conda install -c pytorch magma-cuda102 \
    &amp;&amp; conda clean --all

# install torch from source
RUN git clone --recursive https://github.com/pytorch/pytorch
WORKDIR pytorch
ENV PYTORCH_BUILD_VERSION=1.8.2
ENV PYTORCH_BUILD_NUMBER=1
ENV USE_CUDA=1 USE_CUDNN=1
ENV TORCH_CUDA_ARCH_LIST="3.5" TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
ENV CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"
ENV MAX_JOBS=2
RUN python setup.py clean \
    &amp;&amp; python setup.py install

WORKDIR /home/work/

CMD ["jupyter-lab", "--ip=0.0.0.0","--port=8888" ,"--no-browser", "--allow-root", "--LabApp.token=''"]
</code></pre>
<p>I built docker image by above commands and made container. (use <code>--gpus all</code>)<br>
Container detected <code>nvidia-smi</code> command, but <code>cuda.is_availabel()</code> returned <code>False</code>.<br>
So I wonder something is wrong with how to build pytorch from source.<br>
How can I solve this problem?</p>

username: ptrblck
cooked: <p>Could you check the build logs and see if the CUDA toolkit was detected and used? You should also see the usage of <code>nvcc</code> to compile all CUDA source files.</p>

username: Sumac
cooked: <p>Thank you for your reply.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="136265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you check the build logs</p>
</blockquote>
</aside>
<p>Do you mean to check all logs when I build docker image?<br>
(It’s very long. So I can’t find where I should check.)</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="136265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>see if the CUDA toolkit was detected and used?</p>
</blockquote>
</aside>
<p>I checked <code>nvcc -V</code> in system environment and docker container.<br>
In system environment, I can detect CUDA toolkit.<br>
In docker container, command return <code>nvcc: command not found</code>.</p>
<p>Other docker container (that can use GPU) return same message.<br>
Is there something wrong in my all Dockerfile?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sumac" data-post="3" data-topic="136265">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b5e925/48.png" class="avatar"> Sumac:</div>
<blockquote>
<p>In docker container, command return <code>nvcc: command not found</code>.</p>
<p>Other docker container (that can use GPU) return same message.<br>
Is there something wrong in my all Dockerfile?</p>
</blockquote>
</aside>
<p>This would mean that the CUDA compiler cannot be used inside the container and thus PyTorch also isn’t built with CUDA support.<br>
Make sure you can execute <code>nvcc</code> in <code>nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04</code> by launching it with <code>nvidia-docker</code> or with the <code>--gpus all</code> argument in newer docker versions.</p>

username: Sumac
cooked: <p>Sorry for the late reply.<br>
I check <code>nvcc</code> in <code>nvidia/cuda:10.2-cudnn8-runtime-ubuntu18.04</code>.<br>
It return <code>nvcc: command not found</code>.</p>
<p>I also check other image (<code>nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04</code>).<br>
It return CUDA version correctly and <code>cuda.is_availabel()</code> returned <code>True</code>.<br>
The cause of my problem seem to be docker image.</p>

username: ptrblck
cooked: <p>Yes, you are right and I’ve missed the <code>runtime</code> tag in your initial container. To build from source inside the container you would have to use the <code>devel</code> container. In any case, it’s good to hear you’ve solved the issue.</p>

username: veritas
cooked: <p>Please try using the following template (COI: I am the author).</p>
<p>It can help greatly with building PyTorch from source on many Linux platforms.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/cresset-template/cresset">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">

      <a href="https://github.com/cresset-template/cresset" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/96d4f4836ef7b44d65a9fd1519f7f773cfc065722e421dd463331962ccb6c10f/cresset-template/cresset" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/cresset-template/cresset" target="_blank" rel="noopener nofollow ugc">GitHub - cresset-template/cresset: Template repository to build PyTorch...</a></h3>

  <p>Template repository to build PyTorch projects from source on any version of PyTorch/CUDA/cuDNN. - GitHub - cresset-template/cresset: Template repository to build PyTorch projects from source on any...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: veritas
cooked: <p><a class="mention" href="/u/sumac">@Sumac</a> It also includes a Docker Compose file to make development easier.</p>