Can we disable share memory usage in pytorch?

weiwei: <p>since i am not able to adjust the share memory usage in the remote server, can we disable share memory usage in pytorch. the same experiment run with tensorflow without shm size problem, so i just want to find a solution for this problem.</p>

albanD: <p>Hi,</p>
<p>I’m not a specialist on shared memory but from what I remember, it is only used if you explicitly send tensors across processes. So reducing these would solve your problem. I don’t think we support other ways to transfer tensors in multiprocessing. I guess you could save and load from disk?</p>

weiwei: <p>i am using a distributed job with distributeddataparallel , i just do not clearly got what you mean. as for as i know, pytorch use share memory in dataloader ?</p>

weiwei: <p><a class="mention" href="/u/alband">@albanD</a>  yes , i am sure dataloader with use share memory as default for multiprocessing dataloader.</p>

weiwei: <p>this is what i found in pytorch source code. torch.utils.data.dataloader.py</p>
<p>97 def _worker_loop(dataset, index_queue, data_queue, done_event, collate_fn, seed, init_fn, worker_id):<br>
98     # See NOTE [ Data Loader Multiprocessing Shutdown Logic ] for details on the<br>
99     # logic of this function.<br>
100<br>
101     try:<br>
102         global _use_shared_memory<br>
103         _use_shared_memory = True<br>
104<br>
105         # Intialize C side signal handlers for SIGBUS and SIGSEGV. Python signal<br>
106         # module’s handlers are executed after Python returns from C low-level<br>
107         # handlers, likely when the same fatal signal happened again already.<br>
108         # <a href="https://docs.python.org/3/library/signal.html" rel="nofollow noopener">https://docs.python.org/3/library/signal.html</a> Sec. 18.8.1.1<br>
109         _set_worker_signal_handlers()<br>
110<br>
111         torch.set_num_threads(1)<br>
112         random.seed(seed)<br>
113         torch.manual_seed(seed)<br>
114<br>
115         data_queue.cancel_join_thread()<br>
116<br>
117         if init_fn is not None:<br>
118             init_fn(worker_id)<br>
119<br>
120         watchdog = ManagerWatchdog()<br>
121<br>
122         while watchdog.is_alive():<br>
123             try:<br>
124                 r = index_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)<br>
125             except queue.Empty:<br>
126                 continue<br>
127             if r is None:<br>
128                 # Received the final signal<br>
129                 assert done_event.is_set()<br>
130                 return<br>
131             elif done_event.is_set():<br>
132                 # Done event is set. But I haven’t received the final signal<br>
133                 # (None) yet. I will keep continuing until get it, and skip the<br>
134                 # processing steps.<br>
135                 continue<br>
136             idx, batch_indices = r<br>
137             try:<br>
138                 samples = collate_fn([dataset[i] for i in batch_indices])<br>
139             except Exception:<br>
140                 # It is important that we don’t store exc_info in a variable,<br>
141                 # see NOTE [ Python Traceback Reference Cycle Problem ]<br>
142                 data_queue.put((idx, ExceptionWrapper(sys.exc_info())))<br>
143             else:<br>
144                 data_queue.put((idx, samples))<br>
145                 del samples<br>
146     except KeyboardInterrupt:<br>
147         # Main process will raise KeyboardInterrupt anyways.<br>
148         pass</p>

albanD: <p>Does setting the number of workers to <code>0</code> for the dataloader fix the error?</p>

weiwei: <p>i am not sure, but i can not set the number_works to 0, for the data loading is the bottleneck in our video task. we need to set for each gpu at least 8 number_worker to make sure the data loading will not increase the training time a lot .</p>

albanD: <p>I meant as a test, to confirm that this is where the error is coming from !</p>

weiwei: <p>i have test to set number_worker to 1, and the problem disappear. i can also make sure that this problem is related to batch_size , image_size and number_workers at least.</p>

albanD: <p>I’m not sure there is any way to perform the sharing between the loading processes and the main one that can replace the shared memory.<br>
You might have to reduce the number of workers if you cannot increase the shared memory.<br>
Maybe <a class="mention" href="/u/smth">@smth</a> has some other ideas to overcome this?</p>

amirsoofi: <p>This has the same effect, instead of messing with system settings:</p>
<pre><code class="lang-auto">import torch
torch.cuda.set_per_process_memory_fraction(1.0, 0)
</code></pre>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.pytorch.org/docs/stable//generated/torch.cuda.memory.set_per_process_memory_fraction.html">
  <header class="source">

      <a href="https://docs.pytorch.org/docs/stable//generated/torch.cuda.memory.set_per_process_memory_fraction.html" target="_blank" rel="noopener nofollow ugc">docs.pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.pytorch.org/docs/stable//generated/torch.cuda.memory.set_per_process_memory_fraction.html" target="_blank" rel="noopener nofollow ugc">torch.cuda.memory.set_per_process_memory_fraction — PyTorch 2.8 documentation</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>