diff --git a/docker/accelerate-cpu/Dockerfile b/docker/accelerate-cpu/Dockerfile
new file mode 100644
index 000000000..06379e69b
--- /dev/null
+++ b/docker/accelerate-cpu/Dockerfile
@@ -0,0 +1,34 @@
+# Builds CPU-only Docker image of PyTorch
+# Uses multi-staged approach to reduce size
+# Stage 1
+FROM python:3.6-slim as compile-image
+
+ARG DEBIAN_FRONTEND=noninteractive
+
+RUN apt update
+RUN apt-get install -y --no-install-recommends \
+    build-essential \
+    git \
+    gcc
+
+# Setup virtual environment for Docker
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3 -m venv ${VIRTUAL_ENV}
+# Make sure we use the virtualenv
+ENV PATH="${VIRTUAL_ENV}/bin:$PATH"
+
+WORKDIR /workspace
+# Install specific CPU torch wheel to save on space
+RUN python3 -m pip install --upgrade --no-cache-dir pip
+RUN python3 -m pip install --no-cache-dir \
+    jupyter \
+    torch --extra-index-url https://download.pytorch.org/whl/cpu \
+    git+https://github.com/huggingface/accelerate#egg=accelerate[dev]
+    
+# Stage 2
+FROM python:3.6-slim AS build-image
+COPY --from=compile-image /opt/venv /opt/venv
+
+# Make sure we use the virtualenv
+ENV PATH="/opt/venv/bin:$PATH"
+CMD ["/bin/bash"]
\ No newline at end of file
diff --git a/docker/accelerate-gpu/Dockerfile b/docker/accelerate-gpu/Dockerfile
new file mode 100644
index 000000000..b371042e1
--- /dev/null
+++ b/docker/accelerate-gpu/Dockerfile
@@ -0,0 +1,35 @@
+# Builds GPU docker image of PyTorch
+# Uses multi-staged approach to reduce size
+# Stage 1
+# Use base conda image to reduce time
+FROM continuumio/miniconda3:latest AS compile-image
+# Specify py version
+ENV PYTHON_VERSION=3.6 
+# Install apt libs
+RUN apt-get update && \
+    apt-get install -y curl git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists*
+
+# Create our conda env
+RUN conda create --name accelerate python=${PYTHON_VERSION} ipython jupyter pip
+# We don't install pytorch here yet since CUDA isn't available
+# instead we use the direct torch wheel
+ENV PATH /opt/conda/envs/accelerate/bin:$PATH
+# Activate our bash shell
+RUN chsh -s /bin/bash
+SHELL ["/bin/bash", "-c"]
+# Activate the conda env and install torch + accelerate
+RUN source activate accelerate && \
+    python3 -m pip install --no-cache-dir \
+    torch --extra-index-url https://download.pytorch.org/whl/cu113 \
+    git+https://github.com/huggingface/accelerate#egg=accelerate[dev]
+
+# Stage 2
+FROM nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 AS build-image
+COPY --from=compile-image /opt/conda /opt/conda
+ENV PATH /opt/conda/bin:$PATH
+
+RUN echo "source activate accelerate" >> /.bashrc
+# Activate the virtualenv
+CMD ["/bin/bash"]
\ No newline at end of file



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: I know nothing about Docker, so asking Lysandre for a review :-)
body: Great to have docker support! Helpful for production scenarios. Few comments:
1. The Python versions in CPU and GPU are different - 3.6 and 3.8, respectively. Would it be better to have same Python versions? Also, transformers will be discontinuing 3.6 support, so CPU version can be bumped up to match the GPU version?
2. Shouldn't the Dockerfile's be in the top-level directory to be able to copy accelerate codebase and install it? Maybe have `Dockerfile_cpu` and `Dockerfile_gpu` in top-level directory?
3. Getting below error while building GPU image :
```
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.
```

body: > Great to have docker support! Helpful for production scenarios. Few comments:

These will also be used for having GPU and multi GPU tests, these mimic the transformers docker images for this exact purpose. (This is actually the true purpose behind this PR)

>     1. The Python versions in CPU and GPU are different - 3.6 and 3.8, respectively. Would it be better to have same Python versions? Also, transformers will be discontinuing 3.6 support, so CPU version can be bumped up to match the GPU version?

Yes they are, it's a partial limitation of the GPU image. I could go through the effort of having it use conda instead, but Sylvain and I discussed that 3.6 support will be dropped on the next release. Since it's so soon, it's easier to just have it be this way. 

>     2. Shouldn't the Dockerfile's be in the top-level directory to be able to copy accelerate codebase and install it? Maybe have `Dockerfile_cpu` and `Dockerfile_gpu` in top-level directory?

Nope, these assume the git repo is the top level directory and then copy it. So when building the dockerfiles it looks something like (assuming from a fresh accelerate clone and cd'd to it) `docker build . -f docker/accelerate-cpu/Dockerfile`. See the transformers repo for a similar behavior

>     3. Getting below error while building GPU image :
> 
> 
> ```
> W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
> E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease' is not signed.
> ```

Unsure with this one. Do you have docker cuda properly setup? IIRC you might need to find the right keys to use as well. I was able to build it just fine yesterday.


body: Thinking on it more, having the python version be configurable in the cuda image would be nice. Will change this

Also would probably be better to do similar to this Dockerfile and clone the repo instead: https://github.com/huggingface/transformers/blob/main/docker/transformers-pytorch-gpu/Dockerfile#L11