DataLoader: method 'acquire' of '_thread.lock' objects

username: bask0
cooked: <p>Ubuntu 16.04<br>
PyTorch v1.1</p>
<p>I use PyTorch’s DataLoader to read in Batches in parallel. The data is in the <code>zarr</code> format, multithreaded reading should be supported. To profile the data loading process, I used cProfile on a script that just loads one epoch in a for loop without doing anything else:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
            sampler,
            batch_size=64,
            shuffle=True,
            num_workers=4,
            pin_memory=True,
            drop_last=True,
            worker_init_fn=worker_init_fn,
            collate_fn=BucketCollator(sampler, n_rep_years)
)

for i, batch in enumerate(train_loader):
    pass
</code></pre>
<p>Even if the DataLoader can only use 4 CPUs, all 40 CPUs of the machine are used, but this is not the issue. The DataLoader spends most of the time on <code>~:0(&lt;method 'acquire' of '_thread.lock' objects&gt;)</code> (see profile view 1). What does this mean? Are the processes waiting until the dataset is unlocked, even if multithread reading should work? Or is this just the time the DataLoader waits until the data is read and it is limited by the data reading speed? If this is the case, why would then all 40 CPUs run at 100%?</p>
<p>I appreciate any help to get a better understanding of what is going on under the hood.</p>
<p><strong>EDIT</strong>: It seems <code>_thread.lock.acquire</code> indicates that the main process is waiting for IO. I still wonder why all 40 CPUs run on 100% when they are actually waiting for IO…</p>
<h4><a name="profile-view-1-1" class="anchor" href="#profile-view-1-1"></a>profile view 1</h4>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ea2192806bf583ecccb45a3648785bbf1f46a542.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea2192806bf583ecccb45a3648785bbf1f46a542" title="cProfile_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea2192806bf583ecccb45a3648785bbf1f46a542_2_690x353.png" alt="cProfile_1" data-base62-sha1="xpdEGZGwUxsIoP7i0UGYXy8rKdY" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea2192806bf583ecccb45a3648785bbf1f46a542_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea2192806bf583ecccb45a3648785bbf1f46a542_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea2192806bf583ecccb45a3648785bbf1f46a542_2_1380x706.png 2x" data-dominant-color="AAC1CD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cProfile_1</span><span class="informations">1980×1014 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h4><a name="profile-view-2-2" class="anchor" href="#profile-view-2-2"></a>profile view 2</h4>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2c6f03876bb1c16f1478d898538fd0b7891f268f.png" data-download-href="https://discuss.pytorch.org/uploads/default/2c6f03876bb1c16f1478d898538fd0b7891f268f" title="cProfile_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2c6f03876bb1c16f1478d898538fd0b7891f268f_2_690x352.png" alt="cProfile_2" data-base62-sha1="6l4QZe1zRRwsp7AF6aQceMEnDYz" width="690" height="352" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2c6f03876bb1c16f1478d898538fd0b7891f268f_2_690x352.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2c6f03876bb1c16f1478d898538fd0b7891f268f_2_1035x528.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2c6f03876bb1c16f1478d898538fd0b7891f268f_2_1380x704.png 2x" data-dominant-color="BEAFBC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cProfile_2</span><span class="informations">1976×1010 83.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
891	29.21	0.03278	29.21	0.03278	~:0(&lt;method 'acquire' of '_thread.lock' objects&gt;)
117/116	0.22	0.001896	0.2244	0.001935	~:0(&lt;built-in method _imp.create_dynamic&gt;)
1095	0.136	0.0001242	0.136	0.0001242	~:0(&lt;built-in method marshal.loads&gt;)
10378	0.1329	1.281e-05	0.9031	8.702e-05	core.py:1552(_chunk_getitem)
11208	0.113	1.008e-05	0.1131	1.009e-05	~:0(&lt;built-in method io.open&gt;)
20560	0.1107	5.382e-06	0.1107	5.382e-06	~:0(&lt;built-in method posix.stat&gt;)
60	0.09576	0.001596	0.2934	0.004891	afm.py:201(_parse_char_metrics)
10424	0.08875	8.514e-06	0.3511	3.368e-05	storage.py:721(__getitem__)
107531	0.07747	7.204e-07	0.1234	1.147e-06	afm.py:61(_to_float)
31124	0.06565	2.109e-06	0.06565	2.109e-06	core.py:359(&lt;genexpr&gt;)
3035/2950	0.06219	2.108e-05	0.2543	8.622e-05	~:0(&lt;built-in method builtins.__build_class__&gt;)
10524	0.05762	5.475e-06	0.05762	5.475e-06	~:0(&lt;method 'read' of '_io.BufferedReader' objects&gt;)
46	0.05739	0.001248	0.1434	0.003118	afm.py:253(_parse_kern_pairs)
314835/314125	0.05286	1.683e-07	0.06002	1.911e-07	~:0(&lt;built-in method builtins.isinstance&gt;)
10392	0.05215	5.019e-06	0.08863	8.529e-06	indexing.py:293(__iter__)
1542/1	0.04539	0.04539	32.7	32.7	~:0(&lt;built-in method builtins.exec&gt;)
10372	0.03969	3.827e-06	0.1322	1.274e-05	core.py:1731(_decode_chunk)
12348	0.03371	2.73e-06	0.05517	4.468e-06	posixpath.py:75(join)
71255	0.03229	4.531e-07	0.05519	7.745e-07	afm.py:232(&lt;genexpr&gt;)
96237	0.031	3.221e-07	0.031	3.221e-07	~:0(&lt;method 'split' of 'str' objects&gt;)
11	0.03053	0.002776	0.03053	0.002776	~:0(&lt;built-in method torch._C._cuda_isDriverSufficient&gt;)
1329/5	0.02997	0.005995	2.329	0.4658	&lt;frozen importlib._bootstrap&gt;:966(_find_and_load)
2461	0.02855	1.16e-05	0.1294	5.26e-05	&lt;frozen importlib._bootstrap_external&gt;:1233(find_spec)
33889	0.02833	8.36e-07	0.02833	8.36e-07	~:0(&lt;built-in method __new__ of type object at 0x560415201d60&gt;)
20869	0.02831	1.357e-06	0.02831	1.357e-06	~:0(&lt;method 'reshape' of 'numpy.ndarray' objects&gt;)
</code></pre>

username: bask0
cooked: <p>It seems  <code>_thread.lock.acquire</code>  indicates that the main process is waiting for IO. Problem is solved by making the input pipeline more efficient and by setting pin_memory=False.</p>

username: may210297
cooked: <p>Encountered this issue in 2025. Try to load a lz4 file of 54M every iter in <strong>getitem</strong>. Sad</p>