Unable to write to file </torch_18692_1954506624>

KeithYin: <p>Ubuntu : 16.04 server<br>
python3.6<br>
pytorch:0.2.0_3</p>
<p>Error : RuntimeError: unable to write to file &lt;/torch_18693_1954506624&gt; at /pytorch/torch/lib/TH/THAllocator.c:271</p>
<p>I have encounted this error When  run pytorch code in ubuntu server.</p>
<p>when debuging the code, i found the error occured at DataLoader.</p>
<p>The dataset’s  <code>__getitem__</code> method returned (img, label), the img’s type is ndarray. and i also tried returning img Tensor  but in that condition, the process is blocked.</p>
<p>The code run properly at local, but failed at server.</p>
<p>What should i do to fix that?</p>
<p>Thanks!</p>

ptrblck: <p>Are you using Docker?<br>
I had a similar issue and had to add the <code>--ipc=host</code> flag.</p>
<blockquote>
<p>Please note that PyTorch uses shared memory to share data between processes, so if torch multiprocessing is used (e.g. for multithreaded data loaders) the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host or --shm-size command line options to nvidia-docker run.</p>
</blockquote>

YONG_ZHANG: <p>Solved my problem:+1:</p>

gaily_sun: <p>Hi, I use conda create env, pytorch1.2.0 cuda10.0,  when train 2epochs, this problem happens, how can i solve it?</p>

ptrblck: <p>You might not have enough shared memory, so you could try to increase it on your system (or docker, if you are using it).<br>
I would also recommend to update to the latest stable PyTorch version (<code>1.5</code>) just in case you are hitting an older bug.</p>
<p>If you are using multiple workers in your <code>DataLoader</code>, you could also try to set <code>num_workers=0</code> for the sake of debugging.</p>

gaily_sun: <p>Thanks~   I kill other process,  only run this pytorch task, this problem dispears. The reason is my system does not have enough shared memory.  Thanks for your reply~</p>

Ekansh: <p>Where should I add the --ipc=host flag, notebook, or the command line.</p>

ptrblck: <p><code>--ipc=host</code> should be passed as an argument to the <code>docker run</code> command.</p>

Ekansh: <p>Is there a way to override the location of /dev/sm (shared memory) for PyTorch.</p>
<p>Reference for skelarn : <a href="https://stackoverflow.com/questions/40115043/no-space-left-on-device-error-while-fitting-sklearn-model" rel="noopener nofollow ugc">https://stackoverflow.com/questions/40115043/no-space-left-on-device-error-while-fitting-sklearn-model</a>.<br>
Example : %env JOBLIB_TEMP_FOLDER=/tmp</p>
<p>Please suggest some alternatives</p>

ptrblck: <p>I’m not aware of a way to do so and would recommend to increase the shared memory, if your setup doesn’t provide a sufficiently large amount.</p>

Ekansh: <p>Unfortunately for me increasing the shared memory is not possible. Please suggest alternatives.</p>

ptrblck: <p>I don’t know alternatives to shared memory for multiprocessing IPC.<br>
The fallback would be to use the main thread as for the data loading via <code>num_workers=0</code>, but this would also reduce the performance.</p>

Ekansh: <p>Yes num_workers=0 works but takes a lot of time to train the model.<br>
Thanks !!</p>

Brendan_Wee: <p>Do you know if there is a way to do this in Airflow? I am running into the same error but am not sure how to provide this argument when using a kubernetes pod operator - <a href="https://airflow.apache.org/docs/apache-airflow/1.10.10/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html#airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator" class="inline-onebox" rel="noopener nofollow ugc">airflow.contrib.operators.kubernetes_pod_operator — Airflow Documentation</a></p>

ptrblck: <p>Unfortunately not as I’m not familiar with airflow.</p>

Brendan_Wee: <p>Thanks for replying, if anyone else runs into this issue with airflow, I was able to solve the issue by following <a href="https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine" rel="noopener nofollow ugc">this suggestion</a>.</p>
<p>In a Kubernetes pod operator the arguments looked like this:</p>
<pre><code class="lang-auto">from kubernetes.client import models as k8s_models


volumes=[
    k8s_models.V1Volume(
        name="dshm",
        empty_dir=k8s_models.V1EmptyDirVolumeSource(medium="Memory"),
    )],
volume_mounts=[
    k8s_models.V1VolumeMount(
        name="dshm",
        mount_path="/dev/shm",
    )],
</code></pre>

SystemErrorWang: <p>hello sir, do you have any idea if i came across this problem with kubernotes?</p>

ptrblck: <p>Maybe since kubernetes orchestrates containers and the error is raised by a lack of shared memory.</p>