Is this an expected memory profile?

dmadeka1: <p>I was trying to test <code>torch.multiprocessing</code> and I noticed a pretty strange memory profile. Here’s a minimum reproducible example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.distributed as dist

import time
import os

from torch.multiprocessing import Pipe, Process

_WORLD_SIZE = 2
_RANGE = 1000000

def ping_it(rank, pipe, device, XX):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '7044'
    
    # dist.init_process_group("gloo", rank=rank, world_size=_WORLD_SIZE)
    
    for i in range(_RANGE):
        print(rank, i)
        if i &gt; 0:
            X = pipe.recv()
            X = X.to('cuda:{}'.format(device))
        else:
            XX = XX.to('cuda:{}'.format(device))
        
        pipe.send(XX)


if __name__ == "__main__":
    c1, c2 = Pipe()
    X1 = torch.randn(10000, 10000)
    X2 = torch.randn(10000, 10000)
    p1 = Process(target=ping_it, args=(0, c1, 1, X1))
    p2 = Process(target=ping_it, args=(1, c2, 2, X2))
    p1.start()
    p2.start()
    
    time.sleep(30)
    
    p1.join()
    p2.join()
</code></pre>
<p>I also kicked off a memory logger as so:</p>
<pre data-code-wrap="python"><code class="lang-python">import os
import time
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--file", type=str, default="check.log")
parser.add_argument("--interval", type=int, default=60)
args = parser.parse_args()

def get_memory():
    tim = time.time()
    ram =  os.popen('free -m').readlines()[1].split()[2]
    return tim, ram

if __name__ == "__main__":
    with open(args.file, 'a') as f:
        f.write("Time, RAM")
        while True:
            tim, ram = get_memory()
            print(tim, ram)
            f.write("{}, {}".format(tim, ram))
            time.sleep(args.interval)

</code></pre>
<p>Here’s the memory profile I got:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8e9494b1eb6707aa828b31111022075cc9d7973.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8e9494b1eb6707aa828b31111022075cc9d7973" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8e9494b1eb6707aa828b31111022075cc9d7973_2_607x500.png" alt="image" data-base62-sha1="uWSWJ5QzH0t73om6sX0k8bvwUJt" width="607" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8e9494b1eb6707aa828b31111022075cc9d7973_2_607x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8e9494b1eb6707aa828b31111022075cc9d7973.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8e9494b1eb6707aa828b31111022075cc9d7973.png 2x" data-dominant-color="E7E8F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">733×603 47.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Im leaking anywhere from 0.2-0.4MB/s in different runs. Any idea how I could fix this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> or if there’s anything to fix?</p>

dmadeka1: <p>Running for longer doesn’t change the outcome:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8d23f671a7e0a8341379f9fbfc72bb5145f9252.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8d23f671a7e0a8341379f9fbfc72bb5145f9252" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8d23f671a7e0a8341379f9fbfc72bb5145f9252_2_607x500.png" alt="image" data-base62-sha1="uW5Ap46GYTzYt3QmV3cuoZ74mpY" width="607" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d8d23f671a7e0a8341379f9fbfc72bb5145f9252_2_607x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8d23f671a7e0a8341379f9fbfc72bb5145f9252.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8d23f671a7e0a8341379f9fbfc72bb5145f9252.png 2x" data-dominant-color="E7E8F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">733×603 44.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

dmadeka1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Checking in if there were any resources I could look at</p>