Conda env blocks GPU memory?

werdas34: <p>Hello,</p>
<p>I have a problem with (mini)conda, pytorch and the A6000 GPU (cuda 11).</p>
<p>On my vm server, I have once installed pytorch on base and pytorch in a conda environment.</p>
<p>In base, everything works as it should.<br>
In the conda environment, the GPU memory is already over 42 GB. Even after torch.cuda.empty_cache() the memory is used. Or if I kill all processes that are shown via nvidia-smi, the memory is still almost full.</p>
<p>The error message when running a model is: “RuntimeError: CUDA error: out of memory”.</p>
<p>I have no idea where the error/bug is and how to fix it?</p>

ptrblck: <p>Check for dead processes which might allocate GPU memory. <code>nvidia-smi</code> is not always able to display all processes using GPU memory depending on the permission it has on your system.</p>

werdas34: <p>I will try this later. Thanks for the tip.</p>

werdas34: <p>Hello,</p>
<p>I have now trained many constellations and all but one work. And this is not because the one constellation needs the most memory.</p>
<p>My goal is to measure the memory consumption during training.<br>
Before the training the cache is cleared.</p>
<pre><code class="lang-auto">torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
</code></pre>
<p>The current peak value is saved.<br>
<code>start_peak_gpu = torch.cuda.max_memory_allocated()</code><br>
With base then start_peak_gpu has the value 0. With my conda env the value ~ 460864000<br>
After the training follows:</p>
<pre><code class="lang-auto">end_peak_gpu = torch.cuda.max_memory_allocated()
diff_peak_gpu = end_peak_gpu - start_peak_gpu
</code></pre>
<p>Then I have the GPU peak during the training.</p>
<p><a href="https://theblackcat102.github.io/fixing-nvidia-gpu-zombie-process/" rel="noopener nofollow ugc">I could not find zombie processes.</a></p>
<pre><code class="lang-auto">$ sudo fuser -v /dev/nvidia*
                     USER PID ACCESS COMMAND
/dev/nvidia0: root 852 F.... nvidia-persiste
                     root 1180 F...m Xorg
/dev/nvidiactl: root 852 F.... nvidia-persiste
                     root 1180 F...m Xorg
/dev/nvidia-modeset: root 852 F.... nvidia-persiste
                     root 1180 F.... Xorg
</code></pre>
<p>Would you check other for zombie processes or what else could be a possible error?</p>

ptrblck: <p>I don’t fully understand the issue. Are you seeing 42GB of allocated memory after just activating your conda environment via <code>conda activate env_name</code>?<br>
If so, then I don’t believe it’s a PyTorch issue, since you haven’t even executed anything and I haven’t seen this issue before.<br>
If that’s not the case, could you explain in more detail what exactly you are executing and how you are measuring the memory usage?</p>

werdas34: <p>Yes, exactly.<br>
When I enable the conda environment I lose 42GB of GPU memory.<br>
I don’t know where the cause is. I just read that there were problems with cuda and conda in the past, because conda overwrote binaries.</p>
<p>When I start my model the following is executed at the beginning:</p>
<pre><code class="lang-auto">torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
start_peak_gpu = torch.cuda.max_memory_allocated()
</code></pre>
<p>I expect start_peak_gpu to have the value 0 and the GPU memory to be cleared. However, it has the value 460864000.</p>
<p>In my base environment start_peak_gpu has the value 0.</p>
<p>There is nothing else running on the server. And nvidia-smi does not show any processes.</p>

ptrblck: <p>Could you explain what exactly you are executing? Based on your description is seems you are running some code beforehand before checking the memory.</p>

werdas34: <p>I am running a customized version of the wl-coref model.<br>
In the <a href="https://github.com/vdobrovolskii/wl-coref/blob/master/run.py#L81" rel="noopener nofollow ugc">run.py</a> I have extended the if args.mode== “train”  with the known GPU information.</p>
<pre><code class="lang-auto">if args.mode == "train":
    print("\n##### Start model training - " + start_time + " (UTC) #####", flush=True)
    start_peak_gpu = ""
    if "cuda" in model.config.device:
        torch.cuda.empty_cache()
        torch.cuda.reset_peak_memory_stats()
        start_peak_gpu = torch.cuda.max_memory_allocated()
        print("Start Peak GPU value: " + str(start_peak_gpu), flush=True)     
    if args.weights is not None or args.warm_start:
        model.load_weights(path=args.weights, map_location="cpu",
                           noexception=args.warm_start)
    with output_running_time():
        model.train()
        if "cuda" in model.config.device:
            end_peak_gpu = torch.cuda.max_memory_allocated()
            diff_peak_gpu = end_peak_gpu - start_peak_gpu
            print("End Peak GPU value: " + str(end_peak_gpu), flush=True)
            print("\nPeak memory use: " + str(round(diff_peak_gpu / 1024 ** 3, 3)) + " GB", flush=True)
</code></pre>
<p>Edit:<br>
And load bert/electra model at first</p>