Building from source in docker: Memory use requirments?

Padarn_Wilson: <p>Hi all,</p>
<p>I am trying to setup a dev environment for pytorch (actually functorch), and having battled with some C++ version/library problems I figured developing inside of a docker container would be a simple way to go.</p>
<p>However, I’m trying to build a container and compile pytorch inside of it and getting errors like:</p>
<blockquote>
<p>c++: internal compiler error: Killed (program cc1plus)<br>
at about 85%</p>
</blockquote>
<p>My guess is that this is due to memory constraints in my container - but they’re already quite high (10GB). I am trying to understand if that is expected, should I be able to compile from source with less RAM?</p>

ptrblck: <p>You might need to limit the number of jobs via <code>MAX_JOBS=n python setup.py install</code> in case you are running our of host memory.</p>

Padarn_Wilson: <p>Aha that makes a lot of sense. Thanks a lot I’ll give it a try.</p>

Padarn_Wilson: <p>Thanks for the help, all is well now.</p>
<p>Would there be a useful place for me to put instructions for running a dev container for pytorch (using vscode)? I don’t run a blog but it might be useful for others.</p>

ptrblck: <p>I’m sure a lot of users would appreciate your instructions and findings. The forum might be one place or any blogging platform.</p>

Padarn_Wilson: <p>Well for the meantime I’ll dump it in this thread in case it comes up in anyone’s search. But I’ll try find a better place for it later.</p>
<p>In VSCode there is really useful feature for <code>[VSCode container development](https://code.visualstudio.com/docs/remote/containers)</code>. This can be really useful for making sure your builds and python environment are reproducible.</p>
<p>After reading the doc, you’ll see that you need two things to setup a remote container env</p>
<ol>
<li>Dockerfile</li>
<li>.devcontainer.json</li>
</ol>
<p>It took me some time to set these up, so I’m sharing some workable versions below. Note that I see this up for working on functorch (i.e wanted latest pytorch built from source), so minor modifications may be needed for what you want to do.</p>
<p>(note, in addition I STRONGLY suggest mounting a container with a volume, so the data you’re working on can live between restarts, <a href="https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume" rel="noopener nofollow ugc">see this</a>)</p>
<p>.devcontainer.json</p>
<pre><code class="lang-auto">{
	"name": "torchfunc",
	"build": {
		"dockerfile": "Dockerfile",
	},
	"runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],
	"settings": {},
	"extensions": [
		"ms-vscode.cpptools",
		"ms-vscode.pylance"
	],
	"remoteUser": "root"
}
</code></pre>
<p>Dockerfile (based on that from pytorch repo)</p>
<pre><code class="lang-auto"># syntax = docker/dockerfile:experimental
#
# NOTE: To build this you will need a docker version &gt; 18.06 with
#       experimental enabled and DOCKER_BUILDKIT=1
#
#       If you do not use buildkit you are not going to have a good time
#
#       For reference:
#           https://docs.docker.com/develop/develop-images/build_enhancements/
ARG BASE_IMAGE=ubuntu:18.04
ARG PYTHON_VERSION=3.8

FROM ${BASE_IMAGE} as dev-base
RUN --mount=type=cache,id=apt-dev,target=/var/cache/apt \
    apt-get update &amp;&amp; apt-get install -y --no-install-recommends \
        build-essential \
        ca-certificates \
        ccache \
        cmake \
        curl \
        git \
        libjpeg-dev \
        libpng-dev &amp;&amp; \
    rm -rf /var/lib/apt/lists/*
RUN /usr/sbin/update-ccache-symlinks
RUN mkdir /opt/ccache &amp;&amp; ccache --set-config=cache_dir=/opt/ccache
ENV PATH /opt/conda/bin:$PATH

FROM dev-base as conda
ARG PYTHON_VERSION=3.8
RUN curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  &amp;&amp; \
    chmod +x ~/miniconda.sh &amp;&amp; \
    ~/miniconda.sh -b -p /opt/conda &amp;&amp; \
    rm ~/miniconda.sh &amp;&amp; \
    /opt/conda/bin/conda install -y python=${PYTHON_VERSION} conda-build pyyaml numpy ipython&amp;&amp; \
    /opt/conda/bin/conda clean -ya

RUN mkdir -p /code
RUN git clone https://github.com/pytorch/pytorch.git /code/pytorch
RUN cd /code/pytorch &amp;&amp; git submodule sync &amp;&amp; git submodule update --init --recursive --jobs 0 &amp;&amp; pip install -r requirements.txt
RUN cd /code/pytorch &amp;&amp; MAX_JOBS=4 python setup.py install
</code></pre>