Memory management in PyTorch implementation of multi-processing queues

username: mortazavi
cooked: <p>I have a question for the PyTorch development team.</p>
<p>How is the memory consumed by queues in PyTorch implementation of multi-processing libraries  managed?</p>
<p>If you can point me to the relevant piece of code (if available) and/or provide a textual description, I would appreciate it.</p>

username: pritamdamania87
cooked: <p><a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> Could you help out here since its a torch.multiprocessing question?</p>

username: VitalyFedyunin
cooked: <p>Please check</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/e75fb4356b752097d093c7013ba85c9eb82961ef/torch/multiprocessing/reductions.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/e75fb4356b752097d093c7013ba85c9eb82961ef/torch/multiprocessing/reductions.py" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/e75fb4356b752097d093c7013ba85c9eb82961ef/torch/multiprocessing/reductions.py</a></h4>
<pre><code class="lang-py">import torch
import torch.utils.hooks
from torch._namedtensor_internals import check_serializing_named_tensor
import os
import threading
import multiprocessing
from multiprocessing.util import register_after_fork
from multiprocessing.reduction import ForkingPickler
try:
    # Early load resource_sharer to prevent a partially initialized instance
    # from being inherited in a forked child process. The reduce_storage method
    # requires this module indirectly through DupFd(). The built-in mp.Queue
    # class pickles arguments in a background thread which may overlap with the
    # fork.
    import multiprocessing.resource_sharer
except ImportError:
    pass


class StorageWeakRef(object):
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/e75fb4356b752097d093c7013ba85c9eb82961ef/torch/multiprocessing/reductions.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>and</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/cca247635c6edb323176eeac7a18d3e9ab71c558/torch/multiprocessing/queue.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/cca247635c6edb323176eeac7a18d3e9ab71c558/torch/multiprocessing/queue.py" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/cca247635c6edb323176eeac7a18d3e9ab71c558/torch/multiprocessing/queue.py</a></h4>
<pre><code class="lang-py">import io
import multiprocessing
import multiprocessing.queues
from multiprocessing.reduction import ForkingPickler
import pickle


class ConnectionWrapper(object):
    """Proxy class for _multiprocessing.Connection which uses ForkingPickler to
    serialize objects"""

    def __init__(self, conn):
        self.conn = conn

    def send(self, obj):
        buf = io.BytesIO()
        ForkingPickler(buf, pickle.HIGHEST_PROTOCOL).dump(obj)
        self.send_bytes(buf.getvalue())

    def recv(self):
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/cca247635c6edb323176eeac7a18d3e9ab71c558/torch/multiprocessing/queue.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>as methods are different for CPU/GPU</p>
<p>generally speaking we are passing storage descriptors and do usage ref counting.</p>