Timings for intel arc graphics "xpu" vs. nvidia rtx 3000 gpu (on a laptop)

KFrank: <p>Hi All!</p>
<p>For your delectation:</p>
<p>Short story:  The intel “xpu” is about half as fast as the nvidia gpu and is about six times<br>
faster than running on the cpu.</p>
<p>Pytorch has recently started supporting intel gpus (on a prototype basis).  See<br>
<a href="https://pytorch.org/docs/main/notes/get_start_xpu.html" rel="noopener nofollow ugc">“Getting Started on Intel GPU.”</a></p>
<p>Specific hardware: thinkpad p16v gen2 with intel arc graphics integrated into the<br>
intel core ultra 9 185h main processor and a separate nvidia rtx 3000 ada generation<br>
laptop gpu.</p>
<p>The timing test consists of simple model with three <code>Linear</code>s (no convolutions) and a<br>
plain-vanilla <code>SGD</code> training loop running with <code>float32</code>.</p>
<p>The top-line timing results are (with <code>time</code> in seconds):</p>
<pre><code class="lang-plaintext">device: cuda  time:  49.1860    nBatch: 100000  nEpoch: 1000
device:  xpu  time: 100.8192    nBatch: 100000  nEpoch: 1000
device:  cpu  time: 582.4177    nBatch: 100000  nEpoch: 1000
</code></pre>
<p>The main limitations:  This is an unrealistically simple test model (designed to saturate<br>
the gpu).  Various other tensor operations and more realistic models may run faster or<br>
slower.  I only tested <code>float32</code>.</p>
<p>The main conclusions:</p>
<p>These timings are repeatable and are probably good to a couple of percent.</p>
<p>The integrated arc graphics xpu is significantly faster than the cpu, but not as fast as<br>
the nvidia gpu.</p>
<p>A gpu-intensive job and an xpu-intensive job may be run simultaneously without<br>
detectable degradation to either.  (Also, the xpu uses the main system memory and<br>
does not compete with the nvidia gpu memory.)  (Running the model simultaneously<br>
on the cpu and xpu slows the xpu job by about 80%.  I didn’t disentangle by how much<br>
the cpu job was slowed.)</p>
<p>As is typical, this machine has significantly more system memory than dedicated gpu<br>
memory, so larger models could be accelerated on the xpu than on the gpu.</p>
<p>I haven’t used the xpu version of pytorch very much.  I have seen some little hiccups,<br>
but no major problems like wrong answers or big pieces of missing functionality.</p>
<p>Here is the timing script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

import time

torch.manual_seed (2025)

device = 'cpu'
if  torch.cuda.is_available():
    print ('version.cuda:', torch.version.cuda)
    print (torch.cuda.get_device_name())
    print (torch.cuda.get_device_properties())
    device = 'cuda'
if  torch.xpu.is_available():
    print ('version.xpu:', torch.version.xpu)
    print (torch.xpu.get_device_name())
    print (torch.xpu.get_device_properties())
    device = 'xpu'

print ('device:', device)

vBatch = 100000
nBatch = 100000
nHidden = 512
nEpoch = 1000
# nPrint = 100

def fitFunction (x):
    return (1 * x).sin()

lossFn = torch.nn.MSELoss()

model = torch.nn.Sequential (
    torch.nn.Linear (1, nHidden),
    torch.nn.Sigmoid(),
    torch.nn.Linear (nHidden, nHidden),
    torch.nn.Sigmoid(),
    torch.nn.Linear (nHidden, 1)
)

opt = torch.optim.SGD (model.parameters(), lr = 0.01, momentum = 0.9)

model.to (device)

inputVal = torch.randn (vBatch, 1, device = device)
targetVal = fitFunction (inputVal)

lossInit = lossFn (model (inputVal), targetVal)
print ('lossInit:', lossInit)

if device == 'cuda':  torch.cuda.synchronize()
if device == 'xpu':   torch.xpu.synchronize()
tBeg = time.time()

for  i in range (nEpoch):
    inp = torch.randn (nBatch, 1, device = device)
    trg = fitFunction (inp)
    loss = lossFn (model (inp), trg)
    opt.zero_grad()
    loss.backward()
    # if  i % nPrint == 0  or  i &gt;= nEpoch:
    #     print ('i:', i, 'loss:', loss.detach(), '    time:', '{0:.2f}'.format (time.time() - tBeg))
    opt.step()

if  device == 'cuda':  torch.cuda.synchronize()
if device == 'xpu':   torch.xpu.synchronize()
tEnd = time.time()

lossFinl = lossFn (model (inputVal), targetVal)
print ('lossFinl:', lossFinl)

print ('device:', device, ' time:', '{0:.4f}'.format (tEnd - tBeg), '   nBatch:', nBatch, ' nEpoch:', nEpoch)
</code></pre>
<p>And here are the timing results:</p>
<p>nvidia gpu:</p>
<pre><code class="lang-plaintext">2.6.0+cu126
version.cuda: 12.6
NVIDIA RTX 3000 Ada Generation Laptop GPU
_CudaDeviceProperties(name='NVIDIA RTX 3000 Ada Generation Laptop GPU', major=8, minor=9, total_memory=7933MB, multi_processor_count=36, uuid=f7fb8822-dc6a-36ef-c5e3-2665db44af42, L2_cache_size=32MB)
device: cuda
lossInit: tensor(0.7304, device='cuda:0', grad_fn=&lt;MseLossBackward0&gt;)
lossFinl: tensor(0.0290, device='cuda:0', grad_fn=&lt;MseLossBackward0&gt;)
device: cuda  time: 49.1860    nBatch: 100000  nEpoch: 1000
</code></pre>
<p>intel “xpu”:</p>
<pre><code class="lang-plaintext">2.6.0+xpu
version.xpu: 20250001
Intel(R) Arc(TM) Graphics
_XpuDeviceProperties(name='Intel(R) Arc(TM) Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero', type='gpu', driver_version='1.6.32567+18', total_memory=29184MB, max_compute_units=128, gpu_eu_count=128, gpu_subslice_count=8, max_work_group_size=1024, max_num_sub_groups=128, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=1, has_atomic64=1)
device: xpu
lossInit: tensor(0.7319, device='xpu:0', grad_fn=&lt;MseLossBackward0&gt;)
lossFinl: tensor(0.0293, device='xpu:0', grad_fn=&lt;MseLossBackward0&gt;)
device: xpu  time: 100.8192    nBatch: 100000  nEpoch: 1000
</code></pre>
<p>cpu (pytorch with gpu “2.6.0+cu126” launched with “CUDA_VISIBLE_DEVICES=-1”):</p>
<pre><code class="lang-plaintext">2.6.0+cu126
device: cpu
lossInit: tensor(0.7256, grad_fn=&lt;MseLossBackward0&gt;)
lossFinl: tensor(0.0303, grad_fn=&lt;MseLossBackward0&gt;)
device: cpu  time: 582.4177    nBatch: 100000  nEpoch: 1000
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

prentrodgers: <p>Great work here. I ran your code on a pair of desktop machines with Intel Intel(R) Core™ Ultra 9 285K and 265K respectively, both with Intel Arc B-Series B580. My results were quite different from yours.<br>
My results of the benchmark:<br>
xpu 285k: device: xpu  time: 147.4688    nBatch: 100000  nEpoch: 1000<br>
xpu 265k: device: xpu  time: 148.5960    nBatch: 100000  nEpoch: 1000<br>
cpu 285k: device: cpu  time: 214.1380    nBatch: 100000  nEpoch: 1000 # 44% longer on CPU vs XPU</p>
<p>My xpu scores were 47% worse than yours. This is not what I expected. I suspect that pytorch is using the integrated xpu on the Core Ultra, and not on the B580 Arc card. I will need to work with the code you supplied to get pytorch to use the discrete GPU. Any hints would be appreciated, as I’m new to pytorch.</p>
<p>I’ve been struggling with getting all the drivers loaded on Linux. The 285K is running Ubuntu, the 265K is running Fedora. Intel only supports Ubuntu at this time.</p>
<p>I have been also experimenting with timing Ollama on both machines, and both use the discrete GPU on that workload. On the Ubuntu machine I’m getting 92 tokens a second with llama3.2:7B, but only 15 tokens/sec on that model on the Fedora machine, probably due to lack of drivers for Fedora. On a separate machine with a discrete Nvidia RTX 4090, AMD Ryzen 9 5900X, on that model I get 236 tokens a second.</p>

pyfan: <p>Change all places of ‘xpu’ to ‘xpu:1’ to specify the second device. ‘xpu:0’ refers to the first device.<br>
2.7.1+xpu<br>
version.xpu: 20250004<br>
Intel(R) Arc™ A770 Graphics<br>
_XpuDeviceProperties(name=‘Intel(R) Arc™ A770 Graphics’, platform_name=‘Intel(R) oneAPI Unified Runtime over Level-Zero’, type=‘gpu’, driver_version=‘1.6.33578+11’, total_memory=15473MB, max_compute_units=512, gpu_eu_count=512, gpu_subslice_count=32, max_work_group_size=1024, max_num_sub_groups=128, sub_group_sizes=[8 16 32], has_fp16=1, has_fp64=0, has_atomic64=1)<br>
device: xpu:0<br>
lossInit: tensor(0.7305, device=‘xpu:0’, grad_fn=)<br>
lossFinl: tensor(0.0300, device=‘xpu:0’, grad_fn=)<br>
device: xpu:0  time: 19.8496    nBatch: 100000  nEpoch: 1000<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edbb280b865eb307f1bdd48bed53339c1b8849bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/edbb280b865eb307f1bdd48bed53339c1b8849bf" title="A04E7A59B2A8998A7218F63E3519B4C3"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edbb280b865eb307f1bdd48bed53339c1b8849bf.png" alt="A04E7A59B2A8998A7218F63E3519B4C3" data-base62-sha1="xV48L7foaqYYtGVRelQZic7e98H" width="690" height="135" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">A04E7A59B2A8998A7218F63E3519B4C3</span><span class="informations">1108×218 8.31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For RTX3080Ti, it is around 13 seconds.<br>
For 4070 Ti, it is around 14.5 seconds.</p>