Parallelization Speed Training

username: Alex_Abades_Grimes
cooked: <p>Hello,</p>
<p>I am seeking advice on how to reduce the training time for my model. Below are some specifications for my model and data:</p>
<p><strong>Model:</strong> Contextual Neural Collaborative Filtering (specifications provided at the end)</p>
<ul>
<li>Model Size: 47.51 MB</li>
<li>Trainable Parameters: 12,453,873</li>
</ul>
<p>As indicated, the model itself is not large, but I am working with a substantial dataset in terms of the number of instances:</p>
<ul>
<li>Train Data Size: 586.05 MB</li>
<li>Training Data Shape: (3,647,614; 24)</li>
</ul>
<p>Additionally, the <strong>getitem</strong> method in the custom dataset dynamically creates X negative instances, in this case 4, per positive instance, resulting in a total of:</p>
<p>Training Points: 3,647,614 x 5 = 18,238,070 instances</p>
<p>With a batch size of 256, this equates to approximately 13,800 batches, each containing an average of 1,584 instances.</p>
<p>Since the data is lightweight, I load it entirely into RAM, minimizing data retrieval time and I/O operations. Furthermore, given the modest size of the model (47.51 MB) and peak memory usage (approximately 150 MB when tensors are loaded), the performance difference between training on a GPU and a CPU is not significant. Here are the training times for each:</p>
<ul>
<li><strong>CPU Total Time per Epoch</strong>: 142.77 minutes, detailed as follows:
<ul>
<li>Training Time: 138.99 minutes</li>
<li>Testing Time: 8.05 minutes</li>
</ul>
</li>
<li><strong>GPU Total Time per Epoch</strong>: 106.34 minutes, detailed as follows:
<ul>
<li>Training Time: 99.99 minutes</li>
<li>Testing Time: 7.44 minutes</li>
</ul>
</li>
</ul>
<p>On the CPU, the average time to process a hundred batches is:</p>
<ul>
<li><strong>100 Batches:</strong> 45 seconds</li>
</ul>
<p>Additionally, the resource summary from the job submitted to the HPC indicates:</p>
<ul>
<li>CPU Average Efficiency: 20.00%</li>
<li>CPU Peak Efficiency: 20.17%</li>
</ul>
<p>For this job, five CPU cores were requested, which suggests that the model operations were not parallelized and were performed on a single CPU.</p>
<p>To gain a better understanding of how parallelization works, I replicated the experiment on my own laptop and monitored the system using htop. Here’s what I observed:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Column 1</th>
<th>Column 2</th>
<th>Column 3</th>
<th>Column 4</th>
<th>E</th>
<th>F</th>
<th>G</th>
<th>H</th>
<th>I</th>
<th>J</th>
<th>K</th>
<th>L</th>
</tr>
</thead>
<tbody>
<tr>
<td>PID</td>
<td>USER</td>
<td>PRI</td>
<td>NI</td>
<td>VIRT</td>
<td>RES</td>
<td>SHR</td>
<td>S</td>
<td>CPU%</td>
<td>MEM%</td>
<td>TIME+</td>
<td>Command</td>
</tr>
<tr>
<td>9580</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>162</td>
<td>64</td>
<td>0:45:12</td>
<td>python</td>
</tr>
<tr>
<td>9581</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:09</td>
<td>python</td>
</tr>
<tr>
<td>9582</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:09</td>
<td>python</td>
</tr>
<tr>
<td>9582</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9583</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9584</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9585</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9586</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9587</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9588</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9589</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:09</td>
<td>python</td>
</tr>
<tr>
<td>9590</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9591</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>0</td>
<td>64</td>
<td>0:00:10</td>
<td>python</td>
</tr>
<tr>
<td>9610</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>120</td>
<td>64</td>
<td>0:03:16</td>
<td>python</td>
</tr>
<tr>
<td>9611</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>120</td>
<td>64</td>
<td>0:03:16</td>
<td>python</td>
</tr>
<tr>
<td>9612</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>120</td>
<td>64</td>
<td>0:03:16</td>
<td>python</td>
</tr>
<tr>
<td>9613</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>120</td>
<td>64</td>
<td>0:03:16</td>
<td>python</td>
</tr>
<tr>
<td>9614</td>
<td>alexabades</td>
<td>20</td>
<td>0</td>
<td>24.8GB</td>
<td>502M</td>
<td>162M</td>
<td>R</td>
<td>120</td>
<td>64</td>
<td>0:03:16</td>
<td>python</td>
</tr>
</tbody>
</table>
</div><p>This observation suggests that Python initiated the script with PID 9580, and since the CPU required to train the model exceeded what a single core could handle, PyTorch parallelized the operations across five additional cores (PIDs 9610 to 9614). Each core utilized approximately 12.0% of the CPU. Consequently, this results in a combined usage of approximately 60-62%. Adding this to the 100% running on the core with PID 8580 totals approximately 162% of CPU utilization.</p>
<p>In an effort to optimize this setup, I explored the following options:</p>
<ul>
<li>Number of workers</li>
<li>Number of threads</li>
<li>DistributedDataParallel</li>
</ul>
<p><strong>Number of workers</strong></p>
<p>As I understand it, the number of workers is primarily used to increase the number of processes that load data, which is particularly useful when the data is large and cannot be fully loaded into RAM, necessitating I/O operations to retrieve data. Although new processes (which should be dedicated to load data) were initiated on different cores, in this case, increasing the number of workers did not enhance performance since the data is already loaded in RAM. As a result, increasing the number of workers did not reduce the time it takes to process 100 batches:</p>
<ul>
<li><strong>Time for 100 batches:</strong> 53.27 seconds<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b08a1002cf97578ffdb2fd893799c4e901f18f03.png" data-download-href="https://discuss.pytorch.org/uploads/default/b08a1002cf97578ffdb2fd893799c4e901f18f03" title="multiple_workers"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b08a1002cf97578ffdb2fd893799c4e901f18f03_2_690x338.png" alt="multiple_workers" data-base62-sha1="pbJPBS5ja8OMeBpsTtWV0lpaDSP" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b08a1002cf97578ffdb2fd893799c4e901f18f03_2_690x338.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b08a1002cf97578ffdb2fd893799c4e901f18f03_2_1035x507.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b08a1002cf97578ffdb2fd893799c4e901f18f03_2_1380x676.png 2x" data-dominant-color="1E261B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">multiple_workers</span><span class="informations">1880×922 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
<p>In Red the main process, in orange the workers dedicated to load data and in purple the 5 processes that pytorch has initialized.</p>
<p><strong>Number of Threats</strong></p>
<p>The number of threads is indeed a viable option for optimization. I attempted to increase this by:</p>
<pre><code class="lang-auto">os.environ["MKL_NUM_THREADS"] = "8" 
os.environ["OMP_NUM_THREADS"] = "8"
torch.set_num_threads(8)
</code></pre>
<p>While I have observed the processes utilizing different CPU cores, there has been no improvement in the model training time:</p>
<ul>
<li><strong>Time for 100 batches:</strong> 48.54 seconds<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed4a7ee1b571e190912c059f7ee054d6c6e890fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed4a7ee1b571e190912c059f7ee054d6c6e890fe" title="multiple_processes"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed4a7ee1b571e190912c059f7ee054d6c6e890fe_2_690x340.png" alt="multiple_processes" data-base62-sha1="xRaLx0nr07WYvTXUhhsPnJEEIH4" width="690" height="340" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed4a7ee1b571e190912c059f7ee054d6c6e890fe_2_690x340.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed4a7ee1b571e190912c059f7ee054d6c6e890fe_2_1035x510.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed4a7ee1b571e190912c059f7ee054d6c6e890fe_2_1380x680.png 2x" data-dominant-color="1A2419"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">multiple_processes</span><span class="informations">1888×931 188 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
<p>In orange the 8 processses specified with the number of threats.<br>
<strong>Data Parallelization</strong></p>
<p>Although this is typically used in scenarios where the model or data is too large to fit on a single GPU, I am attempting to apply it to parallelize the model across multiple CPU cores on a single machine. However, due to the RAM limitations of my laptop, I cannot test this hypothesis with my full dataset. Therefore, I have experimented with a subset of the data, which has a shape of (83,731, 24). The configuration for this test is roughly as follows:</p>
<ul>
<li>Batch size = 256</li>
<li>Num of batches = 83731/256 = 327.07 which rounds up to 328 to ensure no data is lost.</li>
<li>Instances per batch = 83731*5 / 328 = 1276.38</li>
</ul>
<p>Due to the way PyTorch handles batching, this results in approximately 1280 instances per batch.</p>
<p>Following the guide provided at PyTorch’s DDP tutorial () and adapting my script accordingly, I have achieved the following setup:</p>
<p>Creation of 4 Ranks: Each rank loads the data and divides it into 4 splits, rounding up as needed with sampler, which in my case results in:</p>
<ul>
<li>Rank 0: 83731/4 = 20933</li>
<li>Rank 1: 83731/4 = 20933</li>
<li>Rank 2: 83731/4 = 20933</li>
<li>Rank 3: 83731/4 = 20933</li>
</ul>
<p>Therefore, the number of batches is calculated as follows:</p>
<ul>
<li>Number of batches = 20933 / 256 = 81.76 which rounds up to 82 to ensure no data is lost.</li>
<li>Instances per batch = 20933 * 5 / 82 = 1276.40</li>
</ul>
<p>Again, due to the way PyTorch handles this calculation, the batch size ends up being 1280 instances per batch.</p>
<p>The model is then instantiated four times, once for each rank, and then trained on these data samples. After each batch is processed, PyTorch gathers the gradients and performs an optimization step. This is a standard setup, not accounting for custom metrics, which are specific to my case. The calculations for the number of batches and training instances per batch can be generalized accordingly.</p>
<p>However, this approach has not improved my training time. Benchmarking this against the same data handled by a PyTorch model on a “single” CPU core, where processes are created on demand, performance times are comparable. I could provide a screenshot of htop to illustrate, but the multiprocess activity makes it difficult to discern anything clearly.</p>
<p>I suspect that the training time remains unchanged, even though the data is split among four cores, each with its own model instance, due to inter-core communication overhead. The optimizer function’s step must wait until all gradients are gathered at the central node, usually rank 0. Since each batch is processed in about 0.45 seconds, communication between nodes occurs at a high frequency, potentially negating the expected time savings from parallel processing.</p>
<p>Has anyone tried a different approach or have insights on how this type of model and training setup could be accelerated?</p>
<p>Thank you very much!<br>
<strong>Model architecture:</strong></p>
<pre><code class="lang-auto">ContextualNeuralCollavorativeFiltering(
  (embeddings): ContextAwareEmbeddings(
    (MF_Embedding_User): Embedding(654, 8)
    (MF_Embedding_Item): Embedding(1127, 8)
    (MLP_Embedding_User): Embedding(654, 32)
    (MLP_Embedding_Item): Embedding(1127, 32)
  )
  (GMF_model): GeneralMatrixFactorization()
  (MLP_model): MultiLayerPerceptron(
    (mlp_model): Sequential(
      (0): Linear(in_features=85, out_features=32, bias=True)
      (1): ReLU()
      (2): Dropout(p=0, inplace=False)
      (3): Linear(in_features=32, out_features=16, bias=True)
      (4): ReLU()
      (5): Dropout(p=0, inplace=False)
      (6): Linear(in_features=16, out_features=8, bias=True)
      (7): ReLU()
      (8): Dropout(p=0, inplace=False)
    )
  )
  (predict_layer): Linear(in_features=16, out_features=1, bias=True)
)
</code></pre>

username: ptrblck
cooked: <p>I don’t know which approach could improve your CPU use case. However, it seems your workload is generally quite small so using CUDA Graphs could yield a good speedup for the GPU use case. You could try to apply it manually or via <code>torch.compile</code>.</p>

username: Alex_Abades_Grimes
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thanks for your reply! I have looked into torch.compile, torch.script and CUDA graphs, and as far as I undertood it, they are all doing kind of the same, packing kernel operation together to reduce overhead. Nevertheless, I don’t know which one could be better to implement. Looking into some documentation and FAQ I have read that compile can be used indeed to train the model, capturing forward and backward steps.</p>
<p>Why would torch.compile be better than:</p>
<pre><code class="lang-auto">with torch.cuda.graph(graph_manager):
    predictions = model(user, item, context)
    loss = loss_fn(predictions, gtruth)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Additionally, on the documentation says:</p>
<blockquote>
<p>For example, if a model’s architecture is simple and the amount of data is large, then the bottleneck would be GPU compute and the observed speedup may be less significant.</p>
</blockquote>
<p>Nevertheless, I will run experiments using torch.compile and CUDA graph to see if there is any improvement.</p>
<p>Finally, what do you think about using DistributedDataParallel to split the training process to speed up?</p>

username: Derrick_Olley
cooked: <aside class="quote no-group quote-modified" data-username="Alex_Abades_Grimes" data-post="3" data-topic="201974" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_abades_grimes/48/64362_2.png" class="avatar"> Alex_Abades_Grimes:</div>
<blockquote>
<p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thanks for your reply! I have looked into torch.compile, torch.script and CUDA graphs, and as far as I undertood it, they are all doing kind of the same, packing kernel operation together to reduce overhead. Nevertheless, I don’t know which one could be better to implement. Looking into some documentation and FAQ I have read that compile can be used indeed to train the model, capturing forward and backward steps.</p>
<p>Why would torch.compile be better than:</p>
<pre><code class="lang-auto">with torch.cuda.graph(graph_manager):
    predictions = model(user, item, context)
    loss = loss_fn(predictions,gtruth)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Additionally, on the documentation says:</p>
<blockquote>
<p>For example, if a model’s architecture is simple and the <a href="https://celebshistory.com" rel="noopener nofollow ugc">here</a> amount of data is large, then the bottleneck would be GPU compute and the observed speedup may be less significant.</p>
</blockquote>
<p>Nevertheless, I will run experiments using torch.compile and CUDA graph to see if there is any improvement<br>
Finally, what do you think about using DistributedDataParallel to split the training process to speed up?</p>
</blockquote>
</aside>
<p>I usually conduct experiments too! Because you can identify many variations and only by testing yourself can you choose the best! And even the most experienced ones cannot always answer because there are many options and a specific solution is suitable for specific tasks!</p>

username: ptrblck
cooked: <p>CUDA Graphs is especially useful if your workload suffers from CPU overheads and when the CPU is thus unable to launch CUDA kernels fast enough. You would see it as white spaces between CUDA kernels in a visual profile created e.g. via Nsight Systems. <code>torch.compile</code> can reduce other overheads and e.g. fuse kernels. However, I would expect CUDA Graphs to yield the main speedup for CPU-limited use cases.</p>

username: copythatpasta
cooked: <p>this also seems like another case where you can potentially use multiple processes per GPU since your model is lightweight. I still have not figured out how to do this?</p>