How to check mps availability?

Robin_Lobel: <p>Is there a way to know programmatically if mps is available or not ?<br>
Something similar to <code>torch.cuda.is_available()</code> or <code>torch.cuda.device_count()</code></p>

albanD: <p>Hey!</p>
<p>Yes, you can check <code>torch.backends.mps.is_available()</code> to check that.<br>
There is only ever one device though, so no equivalent to device_count in the python API.</p>
<p>This doc <a href="https://pytorch.org/docs/master/notes/mps.html" class="inline-onebox">MPS backend — PyTorch master documentation</a> will be updated with that detail shortly!</p>

astroboylrx: <p>Hey, the <a href="https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/" rel="noopener nofollow ugc">announcement</a> says:</p>
<blockquote>
<p>To get started, just install the latest <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">Preview (Nightly) build</a> on your Apple silicon Mac running macOS 12.3 or later with a native version (arm64) of Python.</p>
</blockquote>
<p>I followed the instructions and used <code>pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu</code> to install torch on my mac with M1 Pro (macOS 12.4, Python 3.9 arm64). However, the installed <code>pytorch</code> is still 1.11 and does not allow <code>mps</code> as a device.</p>
<p>How may I install the latest version via pip? Thanks in advance!</p>

hackcs: <p>I tried uninstalling <code>pip3 uninstall torch torchvision torchaudio</code> and reinstalling pytorch using the recommended command and it worked.</p>

astroboylrx: <p>Thanks a lot!! It worked. Didn’t know uninstall would make a difference.</p>
<p>Just found out that <code>float64</code> are <a href="https://discuss.pytorch.org/t/code-didnt-speed-up-as-expected-when-using-mps/152016/6">not supported</a>.<br>
Sad… I guess it is meant for machine learning, not really for scientific computing.</p>

philipturner: <p>The AMX does have high-performance Float64 matrix multiplication on the CPU. And if you compare the ratio of CPU performance (FP64) to GPU performance (FP32), it’s actually better than Nvidia GPU.</p>
<ul>
<li>M1 Max, AMX FP64: 500 GFLOPS</li>
<li>M1 Max, GPU FP32: 10,000 GFLOPS</li>
</ul>
<p>Ratio: 20:1 in terms of FP32:FP64. In comparison, Nvidia GPUs have a 64:1 ratio and they come with x86 CPUs which only have AVX512 on the CPU at best. The fastest Intel CPU would have 8 ops/cycle * 4 GHz * 8 CPU cores = 256 GFLOPS FP64, total. That’s less than Apple.</p>

astroboylrx: <p>Many thanks for the comment!</p>
<p>In my application case, <code>torch.matmul</code> and <code>torch.linalg.solve</code> are the most time-consuming part, where I got ~2.6x speed-up with M1 Pro vs. i7-11800H, and more if vs. older Intel CPUs).<br>
However, this is no where near the speed-up from recent Nvidia GPUs (~13.5x speed-up with 130W laptop Nvidia RTX 3070 vs i7-11800H, and more if with e.g., A100).<br>
So I was hoping for a performance boost from new release.</p>
<p>Is AMX already used in <code>pytorch</code>? I saw you posted a <a href="https://discuss.pytorch.org/t/is-the-amx-accelerator-used-on-apple-silicon/142304">question</a> asking that and also <a href="https://github.com/pytorch/pytorch/issues/47702#issuecomment-1064257151" rel="noopener nofollow ugc">this comment</a> on GitHub.</p>

maqboolurrahim: <p>Strange enough! with my <code>M1 Pro</code> and <code>Arm</code> version of  Python, <code>miniconda</code> with <code>-c pytorch-nightly</code> flag did not work. I had to use the pip install command to install the nightly version.</p>

philipturner: <blockquote>
<p>Is AMX already used in <code>pytorch</code> ? I saw you posted a <a href="https://discuss.pytorch.org/t/is-the-amx-accelerator-used-on-apple-silicon/142304">question</a> asking that and also <a href="https://github.com/pytorch/pytorch/issues/47702#issuecomment-1064257151" rel="noopener nofollow ugc">this comment</a> on GitHub.</p>
</blockquote>
<p>I would regard whoever posted that comment with skepticism. Regardless, if PyTorch uses Accelerate, then it uses the AMX. <a class="mention" href="/u/alband">@albanD</a> said that PyTorch should use Accelerate by default, although the documentation does not officially confirm this is true. For example, it seems to say that other math libraries will be chosen instead of Accelerate if they already exist on the system, because they’re “faster” (not).</p>
<blockquote>
<p>In my application case, <code>torch.matmul</code> and <code>torch.linalg.solve</code> are the most time-consuming part,</p>
</blockquote>
<p>Regarding the linear algebra speedup, AMX exclusively does matrix multiplications. I don’t know what percentage of the <code>linalg.solve</code> algorithm is matrix multiplication and what percentage is matrix factorization. If it can’t utilize the AMX, it won’t run fast.</p>
<p>Apple’s AMX should have the equivalent FP64 processing power to an Nvidia 3000-series GPU. The AMX is a massively powerful coprocessor that’s too big for any one CPU core to handle. That’s why each block of 4 power cores must simultaneously utilize its AMX block to squeeze out all the power. That’s also why the M1 Pro/Max, which has double the power cores of M1, has double the “AMX”. Note that regular M1 also has a second AMX block for its efficiency cores, but that has 1/3 the performance. If you want to know more, I have a long Twitter conversation with the guy who reverse-engineered the AMX and another engineer who hand-wrote assembly for Apple’s math libraries:</p>
<p><a href="https://twitter.com/dougallj/status/1494643295946887169" class="onebox" target="_blank" rel="noopener nofollow ugc">https://twitter.com/dougallj/status/1494643295946887169</a></p>
<p>Edit: The organization of that Twitter conversation seems to be a mess. Just look under my profile for a series of around 20 tweets with two different people in the same time frame, all talking performance gibberish.</p>
<p>We came up with numbers like 256 GFLOPS (FP32) per power CPU core with a 4:1 ratio of FP32:FP64. So first, PyTorch has to be multithreaded and use all the power CPU cores to reach 2000 GFLOPS FP32/500 GFLOPS FP64. Regarding Nvidia GPUs, the 3090 has 556 GFLOPS of double-precision power. That’s comparable to the AMX’s estimated 500 GFLOPS, but it’s also open to general-purpose compute such as sine and cosine functions. Furthermore, the ratio of memory bandwidth to ALU power is more conducive to high ALU utilization with the Nvidia GPU’s FP64.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622">
  <header class="source">
      <img src="https://tpucdn.com/favicon-v1700215791598.ico" class="site-icon" width="" height="">

      <a href="https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622" target="_blank" rel="noopener nofollow ugc">TechPowerUp</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:580/270;"><img src="https://www.techpowerup.com/gpu-specs/images/c/3622-front.small.jpg" class="thumbnail" width="580" height="270"></div>

<h3><a href="https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622" target="_blank" rel="noopener nofollow ugc">NVIDIA GeForce RTX 3090 Specs</a></h3>

  <p>NVIDIA GA102, 1695 MHz, 10496 Cores, 328 TMUs, 112 ROPs, 24576 MB GDDR6X, 1219 MHz, 384 bit</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Although if you really want fast FP64, then you should use a newer discrete AMD GPU. They have a 16:1 FP32:FP64 ratio with ~1.2 TFLOPS of FP64 power. The unofficial <a href="https://github.com/artyom-beilis/dlprimitives" rel="noopener nofollow ugc">DLPrimitives</a> backend for PyTorch would support AMD GPU acceleration, but I don’t think it supports FP64 yet. You could work with the owner to incorporate FP64 into basic GEMM, ensuring that the feature is disabled on Apple GPUs. Since the macOS OpenCL API delegates to the Metal compiler (at least on M1), you might be restricted to using your GPU on Linux and Windows.</p>
<p>Even crazier, look out for Intel’s Arc Alchemist GPU. Intel has a 4:1 ratio of FP32:FP64 performance. If FP32 performance is 20 TFLOPS, you could expect 5 TFLOPS of FP64 processing power. On one further note, the M1 Ultra should have double the AMX power of the M1 Max, reaching 1 TFLOPS FP64.</p>

Sam_Lerman: <p>Can the api be <code>torch.mps.is_available()</code>?</p>
<p>Imo, it looks a bit weird to have <code>device = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'</code>.</p>
<p>The two side by side just feels inconsistent.</p>

albanD: <p>Hi!</p>
<p>i do agree with that but I think we prefer to move the (numerous) new backends into the <code>torch.backends</code> namespace.<br>
Note that both the cuda and mps backends already have the <code>is_built()</code> there.<br>
For consistency, I think we should add the <code>is_available()</code> function to <code>torch.backends.cuda</code> (that just does the same thing as the torch.cuda one. If you have some time to send a PR for that, you can add me as a reviewer.</p>

Ziwei_Yang: <p>May I ask what is the “recommended command” you mentioned here? Since my pytorch also does not work on mps properly.</p>

hackcs: <p>I was referring to the announcement post:</p>
<blockquote>
<p>To get started, just install the latest <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">Preview (Nightly) build</a> on your Apple silicon Mac running macOS 12.3 or later with a native version (arm64) of Python.</p>
</blockquote>
<p>which is basically</p>
<pre><code class="lang-auto">pip3 install torch torchvision torchaudio
</code></pre>

Ziwei_Yang: <p>Thanks so much! It turns out that my macOS version is not new enough to enable mps as training device. After I update the system, it works for me!</p>

ducha-aiki: <p>Is there a way to do this without a crash in previous pytorch versions? E.g. 1.8.2 where there is no <code>torch.backends.mps</code> ?</p>

albanD: <p>You can do:</p>
<pre><code class="lang-auto">mps_available = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
</code></pre>

ducha-aiki: <p>JIT does not like it, unfortunately.</p>

KMT62: <p>“There is only ever one device though, so no equivalent to device_count in python API.”</p>
<p>This is not true.</p>
<p>Some Mac Pro models have two GPU’s and are still widely in use. In fact, I am using one now. If Python/PyTorch <em>correctly</em> implements the Metal API, GPU selection and/or both GPUs by default is very achievable. Some of the apps I use that correctly use Metal have all (both) GPUs enabled by default, such as Final Cut Pro, iMovie, Blender, Affinity Photo, Pixelmator Pro, etc. Any application can use all GPU’s via Metal. Python/PyTorch can too.</p>
<p>This would be a great (and relatively simple) feature to add as using a very cheap (~$300 on the secondary market) Mac Pro with dual GPU’s is a great way to develop and test data parallelism locally before deploying it to larger systems.</p>