Pytorch v2 High CPU consumption

username: namanmehta1994
cooked: <p>Generally, I work with Pytorch v1, recently, I decided to make an upgrade to Pytorch v2.<br>
However, I notice that that the cpu consumption is really high. It’s almost more than 70%.<br>
My task is image classification using resnet/mobilnet, and I am working with Flower102 Dataset(dummy data, just for reference)<br>
I have gone through the resources such as the followings:</p><aside class="quote quote-modified" data-post="4" data-topic="69068">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-is-pytorch-using-only-the-half-of-cpu-cores-smt-enabled/69068/4">Why is pytorch using only the half of cpu cores (SMT enabled)?</a> 
  </div>
  <blockquote>
    If you have 4 cores and need to do, say, 8 matrix multiplications (with separate data) you could use 4 cores to do each matrix multiplication (intra-op-parallelism). Or you could use a single core for each op and run 4 of them in parallel (inter-op-parallelism). 
In training, you also might want to have some cores for the dataloader, for inference, the JIT can parallelize things (I think). 
The configuration is documented here, but without much explanation: <a href="https://pytorch.org/docs/stable/torch.html#parallelism" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torch.…</a>
  </blockquote>
</aside>
<aside class="quote quote-modified" data-post="1" data-topic="52172">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ggaemo/48/45588_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/cpu-usage-extremely-high/52172">CPU usage extremely high</a> 
  </div>
  <blockquote>
    Hello, I am running pytorch and the cpu usage of a single thread is exceeding 100. It’s actually over 1000 and near 2000. As a result even though the number of workers are 5 and no other process is running, the cpu load average from ‘htop’ is over 20. 
the main process is using over 2000 of cpu usage while the data feeders(workers) are using around 100. 
I am using pytorch 1.1 and cuda 9.1. 
Are there any other things that I have to check? 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ede1995afbadf3a1953670492e49ab56965e5d6d.png" data-download-href="https://discuss.pytorch.org/uploads/default/ede1995afbadf3a1953670492e49ab56965e5d6d" title="Selection_148">[Selection_148]</a>
  </blockquote>
</aside>
<p>
My System Specs:<br>
Ubuntu 22.04.4 LTS<br>
conda: 23.7.4<br>
Python: 3.12.2<br>
torch: 2.2.1<br>
nvidia driver: 555.42.06(12.4 cuda)</p>
<p>Since pytorch v1 works seemlessly for me, I figured its driver/dependecy issue with pytorchv2, so I tried different variations.<br>
But I have tried the sasme process with:</p>
<ul>
<li>Python package manager: pip, conda, built from source</li>
<li>Python versions: 3.12.2, 3.10.14</li>
<li>torch versions: 2.2.1, 2.1.2</li>
<li>nvidia driver: 555.42.06(12.4 cuda, 12.5 cuda), 470.82.01(12.2 cuda, 12.4 cuda)<br>
What I have tried:</li>
<li>batchsize(I have tried as low as batch size of 8 images)</li>
<li>pin_memory(in dataloader)</li>
<li>num_workers(in dataloader)</li>
<li>OMP_NUM_THREADS(I could not locate this in os environment params, upon checking some docs, this doesn’t really seem like it has any relevance here)</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/432a3e11727d63e2c43e2381b690166f23ff33aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/432a3e11727d63e2c43e2381b690166f23ff33aa" title="Screenshot 2024-07-09 162415"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/432a3e11727d63e2c43e2381b690166f23ff33aa_2_690x365.png" alt="Screenshot 2024-07-09 162415" data-base62-sha1="9AauF5rC9AkTdxikPZl5MBLUZYu" width="690" height="365" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/432a3e11727d63e2c43e2381b690166f23ff33aa_2_690x365.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/432a3e11727d63e2c43e2381b690166f23ff33aa_2_1035x547.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/432a3e11727d63e2c43e2381b690166f23ff33aa_2_1380x730.png 2x" data-dominant-color="3C3431"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-07-09 162415</span><span class="informations">1835×973 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import os
from PIL import Image
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import torchmetrics
from net import Net
from tqdm import tqdm
from torchvision.models import resnet50, ResNet50_Weights
from torchvision.models import resnet101, ResNet101_Weights
from torchvision.models import resnet152, ResNet152_Weights
from torchvision.transforms import v2
from torchvision.transforms import ToTensor
import torchvision.datasets

def train(model, criterion, optimizer, train_loader, val_loader, max_epochs=3, target_accuracy=0.99):

    _device=torch.device("cuda" if torch.cuda.is_available() else "cpu")
    start_epoch = 0

    for epoch in range(start_epoch, max_epochs):
        # Training
        model.train()
        with tqdm(train_loader) as pbar:
            pbar.set_description(f"[Train] Epoch {epoch}")
            for images, labels in pbar:
                images = images.to(_device)
                labels = labels.to(_device)
                optimizer.zero_grad()
                outputs = model(images)
                loss = criterion(outputs, labels)
                loss.backward()
                optimizer.step()
                loss = loss.cpu()
                pbar.set_postfix({"loss": loss.item()})

        # Validation
        model.eval()
        metric_acc = torchmetrics.Accuracy("multiclass", num_classes=200)
        with tqdm(val_loader) as pbar:
            pbar.set_description(f"[Valid] Epoch {epoch}")
            for images, labels in pbar:
                images = images.to(_device)
                pred = torch.argmax(model(images), axis=1).cpu()
                metric_acc(pred, labels)
                pbar.set_postfix({"acc": metric_acc.compute().item()})

if __name__ == "__main__":
    transforms_ = v2.Compose([
    v2.Resize(size=(300, 300)),
    ToTensor(),
    ])
    batch_size = 8
    dataset = torchvision.datasets.Flowers102(root="./data", split="train", transform=transforms_, download=True)
    n_samples = len(dataset)
    train_samples = int(n_samples * 0.95)
    val_samples = n_samples - train_samples
    train_dataset, val_dataset = torch.utils.data.random_split(dataset, [train_samples, val_samples])
    train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, 
      shuffle=True, pin_memory=False, num_workers=0)
    val_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=False)

    model=resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)    
    _device=torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model=model.to(_device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters())
    train(model, criterion, optimizer, train_loader, val_loader)
</code></pre>
<p>Any points/suggestions would be appreciated.</p>

username: namanmehta1994
cooked: <p>I found the solution here</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/110737">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/110737" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/110737" target="_blank" rel="noopener nofollow ugc">pytorch consuming all cpu cores 100% on ARM</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2023-10-06" data-time="18:30:47" data-timezone="UTC">06:30PM - 06 Oct 23 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/sanchay-hai" target="_blank" rel="noopener nofollow ugc">
          <img alt="sanchay-hai" src="https://avatars.githubusercontent.com/u/139179181?v=4" class="onebox-avatar-inline" width="20" height="20">
          sanchay-hai
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: binaries
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: openmp
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: arm
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

We have a small-ish torch model and runs pretty fast and<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden"> as expected on Intel x64. Although, even on Intel, setting `torch.set_num_threads(1)` reduces cpu usage.

But with ARM (16 core AWS graviton e.g.) it starts consuming 1600% cpu! Then setting `torch.set_num_threads(1)` tones it a bit down to around `1500%+`. Then setting `export OMP_NUM_THREADS=1` brings it down all the way to 25% cpu.

This seems like totally unexpected behavior.

### Versions

/usr/local/lib/python3.11/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
  device: torch.device = torch.device(torch._C._get_default_device()),  # torch.device('cpu'),
Collecting environment information...
PyTorch version: 2.1.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 12 (bookworm) (aarch64)
GCC version: (Debian 12.2.0-14) 12.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.36

Python version: 3.11.4 (main, Aug 16 2023, 07:34:21) [GCC 12.2.0] (64-bit runtime)
Python platform: Linux-5.19.0-1025-aws-aarch64-with-glibc2.36
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          16
On-line CPU(s) list:             0-15
Vendor ID:                       ARM
Model name:                      Neoverse-V1
Model:                           1
Thread(s) per core:              1
Core(s) per cluster:             16
Socket(s):                       -
Cluster(s):                      1
Stepping:                        r1p1
BogoMIPS:                        2100.00
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
L1d cache:                       1 MiB (16 instances)
L1i cache:                       1 MiB (16 instances)
L2 cache:                        16 MiB (16 instances)
L3 cache:                        32 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; CSV2, BHB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] torch==2.1.0
[pip3] torchaudio==2.1.0
[conda] Could not collect

cc @seemethere @malfet @osalpekar @atalman @snadampal</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
torch.set_num_threads(n)<br>
This reduces cpu consumption significantly, however, this does not happen in Pytorch v1, so I guess Pytorch developers made change in implementation for v2 and left it to the user to decide number of threads torch would use.<br>
I should have looked for this in the docs, but I guess I missed that!</p>