PyTorch Weekly May 2nd 2023

KevinLing: <p>Welcome to the 1st issue of Pytorch Weekly, a weekly newsletter covering development in Pytorch AI development platform. You can subscribe the newsletter with <a href="https://www.freelists.org/list/pytorchweekly" rel="noopener nofollow ugc">pytorchweekly@freelists.org</a> or <a href="https://github.com/Lingcc/pytorchweekly" rel="noopener nofollow ugc">Lingcc/pytorchweekly (github.com)</a> .</p>
<h2>
<a name="news-and-articles-from-around-the-web-and-events-1" class="anchor" href="#news-and-articles-from-around-the-web-and-events-1"></a>News and articles from around the web and events</h2>
<ul>
<li>
<a href="https://pytorch.org/blog/introducing-hidet/" rel="noopener nofollow ugc">Hidet</a> is introduced on <code>PyTorch</code> blog as a deep learning compiler for Efficient Model serving. <code>Triton</code> and <code>Hidet Script</code> both allow tensor program developers can easily handle the tile-based programming model.While, compared to <code>Triton</code>, <code>Hidet Script</code> simplifies tensor programming by handling the fine-grained computation and memory resources (e.g., warps, shared memory) manipulation.</li>
<li>
<a href="https://arxiv.org/abs/2304.14226" rel="noopener nofollow ugc">TorchBench</a> is introduced by <code>Yueming Hao</code> and other guys from <code>Meta Platforms, Inc</code>. <code>TorchBench</code> is a novel benchmark suite to study the performance of <code>PyTorch</code> software stack and has been used to identify the GPU performance inefficiencies in PyTorch, and it has also been integrated into the PyTorch continuous integration system.</li>
<li>Towards Data Science published an amazing article: <a href="https://towardsdatascience.com/build-your-own-transformer-from-scratch-using-pytorch-84c850470dcb" rel="noopener nofollow ugc">Build your own Transformer from scratch using Pytorch</a> writen by Arjun Sarkar. It teaches the reader to build a transformer model step by step in PyTorch.</li>
<li>The latest <code>PyTorch 2.0 Ask the Engineers Q&amp;A Series</code> brought <code>TorchRL</code> by <a href="https://github.com/vmoens/" rel="noopener nofollow ugc">Vincent Moens</a> and <a href="https://shashankprasanna.com/" rel="noopener nofollow ugc">Shashank Prasanna</a> from <code>Meta</code>.</li>
<li>
<a href="https://cs.stanford.edu/~zdevito/" rel="noopener nofollow ugc">Zachary DeVito</a> contribute to the Pytorch Forum about <a href="https://dev-discuss.pytorch.org/t/fast-combined-c-python-torchscript-inductor-tracebacks/1158/2" rel="noopener nofollow ugc">Fast combined C++/Python/TorchScript/Inductor tracebacks</a>
</li>
<li>
<a href="https://github.com/davidstutz" rel="noopener nofollow ugc">David Stutz</a> proposed a way for <a href="https://davidstutz.de/loading-and-saving-pytorch-models-without-knowing-the-architecture/" rel="noopener nofollow ugc">Loading and Saving PyTorch Models Without Knowing the Architecture in Advance</a>
</li>
<li>Want to check the differences between <code>PyTorch</code> and <code>Jax</code>? check <a href="https://geekflare.com/jax-vs-pytorch/" rel="noopener nofollow ugc">JAX vs. PyTorch: Differences and Similarities [2023]</a>
</li>
</ul>
<h2>
<a name="on-the-forums-and-maillists-2" class="anchor" href="#on-the-forums-and-maillists-2"></a>On the forums and maillists</h2>
<ul>
<li>
<a href="https://discuss.pytorch.org/t/run-pytorch-on-multiple-gpus/20932/75">Run PyTorch on Multiple GPUs</a> thread was actived again since <code>SM2023</code> tried to fine tune the GPT-2 model on multiple GPUs. Run model on multiple GPUs are not easy to handle, especially for load balance and parallel optimizations. Fresh guys are always recommanded to go through the <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="noopener nofollow ugc">Multi-GPU examples tutorials</a>. Thanks to <code>ptrblck</code>
</li>
<li>According to <a href="https://discuss.pytorch.org/t/would-pytorch-for-cuda-11-6-work-when-cuda-is-actually-12-0/169569/5">Would pytorch for cuda 11.6 work when cuda is actually 12.0</a>, PyTorch binary currently shipped directly with <code>CUDA</code>，<code>CUDNN</code>, and <code>cuBLAS</code>, etc, it uses <code>11.7</code> and <code>11.8</code> by default. And only when build PyTorch from source, will it use the loca installed CUDA toolkit. You are recommanded to use <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">the install method</a>
</li>
<li>Result reproducibility is always a headache for ML training. The thread <a href="https://discuss.pytorch.org/t/different-training-results-on-different-machines-with-simplified-test-code/59378/22">Different training results on different machines</a> has lasted for more than 2 years disscussing about this. PyTorch doc <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Reproducibility</a> has also mentioned that Pytorch does not guarante completely reproducible results. The thread added a new difference between Windows and Linux which might cause unproducable result since <code>os.listdir</code> or <code>glob.glob</code> on Windows produce an ordered list by default, however Linux output the random file list. it lead to different result.</li>
<li>
<a href="https://discuss.pytorch.org/u/jorozco/summary">JOROZCO</a> proposed a way to <a href="https://discuss.pytorch.org/t/convert-pytorch-model-to-onnx-format-inference-not-same/145005/8">convert PyTorch model to <code>ONNX</code> format</a>
</li>
<li>
<a href="https://discuss.pytorch.org/t/how-to-fix-cuda-error-device-side-assert-triggered-error/137553/5">How to fix “CUDA error: device-side assert triggered” error?</a> introduced <code>CUDA_LAUNCH_BLOCKING=1</code> to disable asynchronous kernel launhches.</li>
</ul>
<h2>
<a name="pytorch-commits-highlight-3" class="anchor" href="#pytorch-commits-highlight-3"></a>PyTorch commits Highlight</h2>
<ul>
<li>PyTorch main develop branch changed from <code>master</code> to <code>main</code>
</li>
<li><a href="https://github.com/erlv/hpytorch_nocuda/commit/0a5c9304997631ba84f7185a1c7e29d0b926974a" rel="noopener nofollow ugc">CUDA 12.1 build is enabled again on windows</a></li>
<li>Plenty of <code>Dynamo</code>, <code>Triton</code> bug fixes and improvement, such as <a href="https://github.com/pytorch/pytorch/pull/99834" rel="noopener nofollow ugc">add support for serializing real tensor data in after AOT minifier</a>, <a href="https://github.com/pytorch/pytorch/pull/94440" rel="noopener nofollow ugc">Basic dynamo support for traceable collectives</a>, <a href="https://github.com/pytorch/pytorch/pull/99940" rel="noopener nofollow ugc">Introduce FXGraphExtractor into torch.onnx.dynamo_export</a>
</li>
<li>
<a href="https://github.com/pytorch/pytorch/pull/100108" rel="noopener nofollow ugc">Dan Dale fix a CPU offload performance issue for <code>ShardedGradScaler</code></a>. The performance analyze of the work is amazing.</li>
<li>Related changes to remove CUDA 11.6 support</li>
<li><a href="https://github.com/pytorch/pytorch/pull/100226" rel="noopener nofollow ugc">Improve the debug method for after AOT accuracy debugging</a></li>
<li>Improve New Architecture support: <a href="https://github.com/pytorch/pytorch/pull/99024" rel="noopener nofollow ugc">Making FSDP device-agnositc for custom-backend which implement cuda-semantics</a>, <a href="https://github.com/pytorch/pytorch/pull/99854" rel="noopener nofollow ugc">New hook for MTIA architecture</a>
</li>
<li><a href="https://github.com/pytorch/pytorch/pull/94820" rel="noopener nofollow ugc">Optimized EMA implementation</a></li>
<li><a href="https://github.com/erlv/hpytorch_nocuda/commit/dfba65be8b1b42175905e609f9eb5db431c449e0" rel="noopener nofollow ugc">Update Cutlass to v3.1</a></li>
</ul>
<h2>
<a name="other-project-or-company-weekly-updates-highlight-4" class="anchor" href="#other-project-or-company-weekly-updates-highlight-4"></a>Other project or company weekly updates highlight</h2>
<ul>
<li>Modular AI annouced its two init products. The first one is <a href="https://www.modular.com/engine" rel="noopener nofollow ugc">the fastest unified AI inference engine in the world</a>. The second one is a <a href="https://www.modular.com/mojo" rel="noopener nofollow ugc">new programming language for all AI developers <code>Mojo</code></a>
</li>
</ul>