Conv module in forward can't alloc

username: Shizhe_Cai
cooked: <p>Hi, here is the problem description:<br>
I’m training an IMPALA agent on a GPU with allocated memory 113000MB, and every time I train over 1.1M frames I got this error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15b84947e88687e777d984ad28ac710f1ac9f83d.png" data-download-href="https://discuss.pytorch.org/uploads/default/15b84947e88687e777d984ad28ac710f1ac9f83d" title="Screen Shot 2020-08-28 at 11.49.58 am"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15b84947e88687e777d984ad28ac710f1ac9f83d_2_690x228.png" alt="Screen Shot 2020-08-28 at 11.49.58 am" data-base62-sha1="368RdwpapXnBRKpt0JzfxbKXfzv" width="690" height="228" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15b84947e88687e777d984ad28ac710f1ac9f83d_2_690x228.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15b84947e88687e777d984ad28ac710f1ac9f83d_2_1035x342.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15b84947e88687e777d984ad28ac710f1ac9f83d_2_1380x456.png 2x" data-dominant-color="225155"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-08-28 at 11.49.58 am</span><span class="informations">1754×580 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It didn’t show something like “Cuda out of memory error”, but just a RUNTIME Error: Can’t alloc. The feature_extraction model contains a six-layer Conv2d net.<br>
I wonder if my model is too big for scalable training? What can I do to reduce the memory usage?</p>

username: iffiX
cooked: <p>This is a CPU memory bad allocation problem, the screensho is not enough to determine the problem root.</p>

username: Shizhe_Cai
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a> Thanks for the problem identification! I’m new to here and only can upload one image at a time <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> . Can you give me some hints to to determine the problem root? I currently have 40 actors to run and fill the experience buffer</p>

username: iffiX
cooked: <p>Is showing all of your code applicable?</p>

username: Shizhe_Cai
cooked: <p>Unfortunately, the code is too large, however, I can show you the code of creating actors and batch_and_learn in each thread:<br>
here is the actor initialization:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edbe9925cd0427de7ec5c9374433cd448d6a94d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/edbe9925cd0427de7ec5c9374433cd448d6a94d2" title="Screen Shot 2020-08-28 at 12.53.37 pm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edbe9925cd0427de7ec5c9374433cd448d6a94d2_2_428x375.png" alt="Screen Shot 2020-08-28 at 12.53.37 pm" data-base62-sha1="xVbvWUWZKdoiywSNPlvKaoD9BaG" width="428" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edbe9925cd0427de7ec5c9374433cd448d6a94d2_2_428x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edbe9925cd0427de7ec5c9374433cd448d6a94d2_2_642x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edbe9925cd0427de7ec5c9374433cd448d6a94d2_2_856x750.png 2x" data-dominant-color="323233"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-08-28 at 12.53.37 pm</span><span class="informations">1086×950 75.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Shizhe_Cai
cooked: <p>And here is the batch and learn in each thread<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96b2231a6195e3f610846a0dfc6f7d0c07463463.png" data-download-href="https://discuss.pytorch.org/uploads/default/96b2231a6195e3f610846a0dfc6f7d0c07463463" title="Screen Shot 2020-08-28 at 12.55.08 pm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96b2231a6195e3f610846a0dfc6f7d0c07463463_2_422x375.png" alt="Screen Shot 2020-08-28 at 12.55.08 pm" data-base62-sha1="lv7hd3mrWsnl7CBSo5ZbZRzZ8bN" width="422" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96b2231a6195e3f610846a0dfc6f7d0c07463463_2_422x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96b2231a6195e3f610846a0dfc6f7d0c07463463_2_633x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96b2231a6195e3f610846a0dfc6f7d0c07463463_2_844x750.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-08-28 at 12.55.08 pm</span><span class="informations">1510×1340 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(I still can only upload one image at a time <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> )</p>

username: iffiX
cooked: <p>There doesn’t seem to be anything wrong in the showed part. I think you may try to decrease the the local /global replay buffer size in your implementation, /or skip learning &amp; rolling out by appending radomly generated tensor data into your buffer and see when they will overflow, and track down the distributed bug.</p>
<p>If you would like to have a correct implementation of IMPALA as reference, I have written a tested one before.<br>
Doc here:<br>
<a href="https://machin.readthedocs.io/en/latest/tutorials/unleash_distributed_power.html#impala" class="onebox" target="_blank" rel="nofollow noopener">https://machin.readthedocs.io/en/latest/tutorials/unleash_distributed_power.html#impala</a><br>
and source code here:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/impala.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/impala.py" target="_blank" rel="nofollow noopener">iffiX/machin/blob/master/machin/frame/algorithms/impala.py</a></h4>
<pre><code class="lang-py">from typing import Union, Dict, List, Tuple, Callable, Any
import numpy as np
import torch as t
import torch.nn as nn

from machin.frame.buffers.buffer_d import Transition, DistributedBuffer
from machin.model.nets.base import NeuralNetworkModule
from .base import TorchFramework
from .utils import safe_call

from machin.parallel.server import PushPullModelServer
from machin.parallel.distributed import RpcGroup


def _make_tensor_from_batch(batch: List[Any], device, concatenate):
    """
    Used to convert compact every attribute of every step of a whole episode
    into a single tensor.

    Args:
</code></pre>

  This file has been truncated. <a href="https://github.com/iffiX/machin/blob/master/machin/frame/algorithms/impala.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: a_d
cooked: <p><a class="mention" href="/u/shizhe_cai">@Shizhe_Cai</a> you could copy paste your code between triple backward ticks instead of uploading a screenshot, Something like this -</p>
<pre><code class="lang-auto">your entire code here, this would make it more readable...
</code></pre>

username: Shizhe_Cai
cooked: <p><a class="mention" href="/u/iffix">@iffiX</a> Thank you, I will try to reduce the buffer size at first. <a class="mention" href="/u/a_d">@a_d</a> My implementation is actually based on the TorchBeast implementation<br>
<a href="https://github.com/facebookresearch/torchbeast/blob/master/torchbeast/monobeast.py" rel="nofollow noopener">https://github.com/facebookresearch/torchbeast/blob/master/torchbeast/monobeast.py</a><br>
, all I’ve modified is that I added some new models to train.</p>

username: Shizhe_Cai
cooked: <p>Sorry to bother you but I still didn’t solve the problem. I tried to half the buffer size but it still cause alloc error when reaching 1.1M frames.</p>

username: iffiX
cooked: <p>The allocation error could be a leak at anywhere, I suppose that you should remove training &amp; sampling completely, and just stuff random data into your system and test its stability.</p>

username: Shizhe_Cai
cooked: <p>Hi iffi,<br>
I think I’ve located the problem, but I don’t know how to fix it. As you know the IMPLALA model has actors and learners. I’m using the torchbeast implementation called monobeast: that run actors on a number of processes on CPU, and do the learning part on GPU. There are some drawbacks of this pattern, which also indicated in the torchbeast paper: the model require a large amount of shared memory between processes, and do the model eval and environment.step on cpu too. And their are some copies of tensors that is unnecessary. So as the frame number goes large, more and more memories were eaten by some unused tensors. That’s why no matter what buffer size and number of actors i use, the model always got an error when training for about 1.1M frames, there is no room for the actor model to do the forward operation.</p>
<p>I wonder how I can free up those memory/ I’ve tried del output, and del loss. the limit increase to 1.3M frames but that’s it. here is the part of fill in the buffer, followed by the part of withdrawing the buffer.<br>
`def act(i: int, free_queue: mp.SimpleQueue, full_queue: mp.SimpleQueue,<br>
model: torch.nn.Module, buffers: Buffers,<br>
episode_state_count_dict: dict, train_state_count_dict: dict,<br>
initial_agent_state_buffers, flags):</p>
<pre><code>try:
    log.info('Actor %i started.', i)
    timings = prof.Timings()

    gym_env = create_env(flags)

    if flags.num_input_frames &gt; 1:
        gym_env = FrameStack(gym_env, flags.num_input_frames)

    if 'procgen' in flags.env:
        env = ProcGenEnvironment(gym_env, flags.start_level, flags.num_levels, flags.distribution_mode)
    else:
        seed = i ^ int.from_bytes(os.urandom(4), byteorder='little')
        gym_env.seed(seed)
        env = Environment(gym_env, fix_seed=flags.fix_seed, env_seed=flags.env_seed)

    env_output = env.initial()
    agent_state = model.initial_state(batch_size=1)
    agent_output, unused_state = model(env_output, agent_state)

    while True:
        index = free_queue.get()
        if index is None:
            break

        # Write old rollout end.
        for key in env_output:
            buffers[key][index][0, ...] = env_output[key]
        for key in agent_output:
            buffers[key][index][0, ...] = agent_output[key]
        for i, tensor in enumerate(agent_state):
            initial_agent_state_buffers[index][i][...] = tensor


        # Update the episodic state counts
        episode_state_key = tuple(env_output['frame'].view(-1).tolist())
        if episode_state_key in episode_state_count_dict:
            episode_state_count_dict[episode_state_key] += 1
        else:
            episode_state_count_dict.update({episode_state_key: 1})
        buffers['episode_state_count'][index][0, ...] = \
            torch.tensor(1 / np.sqrt(episode_state_count_dict.get(episode_state_key)))
        
        # Reset the episode state counts when the episode is over
        if env_output['done'][0][0]:
            for episode_state_key in episode_state_count_dict:
                episode_state_count_dict = dict()

        # Update the training state counts
        train_state_key = tuple(env_output['frame'].view(-1).tolist())
        if train_state_key in train_state_count_dict:
            train_state_count_dict[train_state_key] += 1
        else:
            train_state_count_dict.update({train_state_key: 1})
        buffers['train_state_count'][index][0, ...] = \
            torch.tensor(1 / np.sqrt(train_state_count_dict.get(train_state_key)))

        # delete output
        del agent_output

        # Do new rollout
        for t in range(flags.unroll_length):
            timings.reset()

            with torch.no_grad():
                agent_output, agent_state = model(env_output, agent_state)

            timings.time('model')

            env_output = env.step(agent_output['action'])

            timings.time('step')

            for key in env_output:
                buffers[key][index][t + 1, ...] = env_output[key]

            for key in agent_output:
                buffers[key][index][t + 1, ...] = agent_output[key]
            
            # Update the episodic state counts
            episode_state_key = tuple(env_output['frame'].view(-1).tolist())
            if episode_state_key in episode_state_count_dict:
               episode_state_count_dict[episode_state_key] += 1
            else:
                episode_state_count_dict.update({episode_state_key: 1})
            buffers['episode_state_count'][index][t + 1, ...] = \
                torch.tensor(1 / np.sqrt(episode_state_count_dict.get(episode_state_key)))

            # Reset the episode state counts when the episode is over
            if env_output['done'][0][0]:
                episode_state_count_dict = dict()

            # Update the training state counts
            train_state_key = tuple(env_output['frame'].view(-1).tolist())
            if train_state_key in train_state_count_dict:
                train_state_count_dict[train_state_key] += 1
            else:
                train_state_count_dict.update({train_state_key: 1})
            buffers['train_state_count'][index][t + 1, ...] = \
                torch.tensor(1 / np.sqrt(train_state_count_dict.get(train_state_key)))

            timings.time('write')
        full_queue.put(index)

    if i == 0:
        log.info('Actor %i: %s', i, timings.summary())

except KeyboardInterrupt:
    pass  
except Exception as e:
    logging.error('Exception in worker process %i', i)
    traceback.print_exc()
    print()
    raise e`
</code></pre>
<pre><code class="lang-auto">def get_batch(free_queue: mp.SimpleQueue,
              full_queue: mp.SimpleQueue,
              buffers: Buffers,
              initial_agent_state_buffers,
              flags,
              timings,
              lock=threading.Lock()):
    with lock:
        timings.time('lock')
        indices = [full_queue.get() for _ in range(flags.batch_size)]
        timings.time('dequeue')
    batch = {
        key: torch.stack([buffers[key][m] for m in indices], dim=1)
        for key in buffers
    }
    initial_agent_state = (
        torch.cat(ts, dim=1)
        for ts in zip(*[initial_agent_state_buffers[m] for m in indices])
    )
    timings.time('batch')
    for m in indices:
        free_queue.put(m)
    timings.time('enqueue')
    batch = {
        k: t.to(device=flags.device, non_blocking=True)
        for k, t in batch.items()
    }
    initial_agent_state = tuple(t.to(device=flags.device, non_blocking=True)
                                for t in initial_agent_state)
    timings.time('device')
    return batch, initial_agent_state
</code></pre>
<p>Is there a way I can free those used buffer? Cause to my understanding, memory usage consistently increasing does not make sense right?</p>

username: iffiX
cooked: <p>Pytorch is not quite clear about this part, in fact, in this previously asked question (by me):</p><aside class="quote" data-post="5" data-topic="94380">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-cpu-memory-usage/94380/5">Pytorch cpu memory usage</a> 
  </div>
  <blockquote>
    Hi, 
The allocator I mention here is not from pytorch but from your libc (we have no control over it). These are classic CPU allocators. Famous alternatives include jemalloc or tmalloc. But I haven’t tested them myself.
  </blockquote>
</aside>
<p>
In come cases (my implementation 2), even if the tensors are dereferenced, their memory are still not freed immediately after dereferencing, for more deep down implementation information, maybe you could consider asking the pytorch development member <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/alband">@albanD</a>.</p>
<p>I have not tried the jemalloc solution yet, but it is easy enough to install it by simply making it like:</p>
<pre><code class="lang-auto">wget https://github.com/jemalloc/jemalloc/releases/download/5.0.1/jemalloc-5.0.1.tar.bz2
tar -jxvf jemalloc-5.0.1.tar.bz2
cd jemalloc-5.0.1
sudo apt-get install autogen autoconf

./autogen.sh
make -j2
sudo make install
sudo ldconfig
cd ../
rm -rf jemalloc-5.0.1 jemalloc-5.0.1.tar.bz2
</code></pre>
<p>then export <code>LD_PRELOAD</code> environment variable before running your program:</p>
<pre><code class="lang-auto">LD_PRELOAD=/home/mingfeim/packages/jemalloc-5.2.0/lib/libjemalloc.so ./your_script.sh
</code></pre>
<p>Give it a try?</p>
<p>Reference:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.cnblogs.com/evenleee/p/11957076.html">
  <header class="source">
      <img src="https://common.cnblogs.com/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://www.cnblogs.com/evenleee/p/11957076.html" target="_blank" rel="noopener nofollow ugc">cnblogs.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.cnblogs.com/evenleee/p/11957076.html" target="_blank" rel="noopener nofollow ugc">Linux安装jemalloc笔记 - evenleo - 博客园</a></h3>

  <p>前言 最近研究一个工具库需要用 jemalloc 做内存分配器，但在 ubuntu 下安装过程中遇到很多问题，故记下安装过程的笔记，避免以后遇到在这上面浪费时间。 安装过程 环境：VMware Ubuntu 18.04.3 LTS cd ~/桌面 wget https://github.com/je</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://zhuanlan.zhihu.com/p/79989669">
  <header class="source">
      <img src="https://static.zhihu.com/heifetz/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://zhuanlan.zhihu.com/p/79989669" target="_blank" rel="noopener nofollow ugc">知乎专栏</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://zhuanlan.zhihu.com/p/79989669" target="_blank" rel="noopener nofollow ugc">PyTorch在CPU上的一些Performance BKM</a></h3>

  <p>这里简单介绍一下用PyTorch在CPU上的一些性能相关的BKM。 内容以inference为主，毕竟CPU上主要的场景还是inference；另外这里CPU都指的是Intel Xeon. gist里面写了英文版的，内容和这里的基本相当： General guide…</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Shizhe_Cai
cooked: <p>Really appreciate your suggestion. Actually I’m using the HPC in my University and submit the slurm script to get the job run on a node. So I don’t know if I’m able to using jemalloc in this case. But again thank you for the idea!</p>

username: iffiX
cooked: <p>I see, then it would be very difficult regarding subitting a precompiled library or compile remotely on nodes, I guess you will need to tweak the framework code instead. Or you could change an implementation.</p>

username: Shizhe_Cai
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/alband">@albanD</a> It would be awesome if you can share some ideas about this.<br>
One more update: I’m using the procgen environment, for the Coinrun, I can only train 1.1M frames. But I trained the same model on the Maze environment, it reached 5M frames without error</p>

username: Shizhe_Cai
cooked: <p>And could you please have a look at another problem I posted, its about the behaviour of Multinomial with input inf and nan and &lt;0 value in difference torch version.<a href="https://discuss.pytorch.org/t/torch-1-6-0-runtimeerror-probability-tensor-contains-either-inf-nan-or-element-0-but-good-with-torch-1-1-0/94570">Torch 1.6.0 RuntimeError: probability tensor contains either <code>inf</code>, <code>nan</code> or element &lt; 0, But good with Torch 1.1.0</a></p>

username: albanD
cooked: <p>I am not sure this is the same. The issue you have should never lead to OOM.<br>
The memory is still available for this process and can be re-used. It is more an issue that it makes the memory not available for other processes and confuses reporting tools.</p>
<p>I am really not sure where this errors can come from though. I am pretty sure this is not in our own code. So maybe cuda-related stuff?<br>
Do you actually run out of RAM while the program runs?</p>

username: Shizhe_Cai
cooked: <p>I’m using the HPC system, and I even crash two nodes because of the OOM. So I’m pretty sure that the memory out of usage.</p>

username: albanD
cooked: <p>If the whole node crashes, then the stack trace is most likely not very representative: it just points to whatever it was executing when the node ran out of memory.<br>
I think you should try and reproduce this on your local machine, with a smaller dataset/model and see if you can reproduce the memory increase.</p>