How do I learn PyTorch… in a Deeper manner?

username: GrandArth
cooked: <p>So i been using PyTorch for experiments in the past two years.<br>
I can get others’ implement up and running in a reasonable time frame.<br>
I know how to deal with data loading, inferencing, logging and can implement something like vanilla transformer, CycleGan, etc. from scratch.<br>
But I always have a feeling that i am playing in shallow water, I know how to get it up and running but nothing else.<br>
As someone who is currently and will be studying in Deep learning field (for at least 3 more years) i wish to better understand and use PyTorch.</p>
<p>RN I just dig into some keywords about PyTorch here and there, but i wonder is there anything like “Fluent Python” for python to better guide me in this deeper learning process?</p>
<p>Welcome all advice in general.</p>

username: KFrank
cooked: <p>Hi Grand!</p>
<aside class="quote no-group" data-username="GrandArth" data-post="1" data-topic="202118" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grandarth/48/49373_2.png" class="avatar"> GrandArth:</div>
<blockquote>
<p>As someone who is currently and will be studying in Deep learning field (for at least 3 more years) i wish to better understand and use PyTorch.</p>
</blockquote>
</aside>
<p>There’s a lot of ways to go about this, and I’m sure that others will have a variety<br>
of suggestions.  I’m a fundamentals kind of guy, so my advice will have its focus on<br>
the basics as its theme.</p>
<p>General programming and python programming are obviously a requirement.  You<br>
can of course learn them as you are learning pytorch, but don’t neglect them.  As<br>
you are working with pytorch, make a point of continuing to sharpen your general<br>
and python programming skills.</p>
<p>Study the details of pytorch’s <code>Tensor</code>s – these are the workhorses of pytorch.<br>
Experiment with some simple individual tensors.  Use the python feature <code>id()</code>,<br>
<code>type()</code>, <code>type().mro()</code>, and <code>dir()</code> to see some of what’s going on under the<br>
hood.</p>
<p>Use pytorch features to probe tensors.  For example, you could try:</p>
<pre data-code-wrap="python"><code class="lang-python">t = torch.arange (6).byte().reshape (2, 3)
tt = t.T
ts = t[1]
t.is_contiguous()
tt.is_contiguous()
ts.is_contiguous()
t.stride()
tt.stride()
ts.stride()
t.storage_offset()
tt.storage_offset()
ts.storage_offset()
t.untyped_storage()
tt.untyped_storage()
ts.untyped_storage()
</code></pre>
<p>In general, drill down from time to time to see what pytorch is actually doing.  Pick<br>
your spots – you can’t do everything – but look for opportunities to explore the<br>
details.</p>
<p>Study the various ways you can “restructure” tensors – things like <code>view()</code>,<br>
<code>reshape()</code>, <code>permute()</code>, and indexing and slicing of tensors, including “advanced<br>
tensor indexing.”</p>
<p>[Edit] Understand how broadcasting works and how to use <code>squeeze()</code> and<br>
<code>unsqueeze()</code> to align tensor dimensions properly.  Look at how <code>matmul()</code> performs<br>
different kinds of tensor-tensor multiplication depending on the dimensions of its<br>
input tensors.  Note that <code>matmul()</code> subsumes things like <code>mm()</code>, <code>dot()</code>, and <code>bmm()</code>.<br>
I generally use <code>matmul()</code> (which can be written with the infix operator <code>@</code>), but you<br>
might prefer to use one of the more specialized tensor-multiplication functions if you<br>
feel that it helps stylistically to make your code more readable.  Study how <code>einsum()</code><br>
performs “contractions” over multiple tensors – it’s the “swiss-army knife” for<br>
performing more-or-less any tensor-tensor “multiplication,” including contractions<br>
over more than two tensors.</p>
<p>Tensor operations, autograd, and the gradient-descent-based optimizers are how<br>
pytorch trains models.</p>
<p>Use pytorch tensor operations – but no autograd nor built-in optimizers – to find<br>
the minimum of a quartic polynomial using gradient descent.  Then do it again with<br>
autograd and a built-in optimizer.</p>
<p>Compute the gradient of something like:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.Sequential (
    torch.nn.Linear (2, 2), torch.nn.ReLU(), torch.nn.Linear (2, 1)
)
</code></pre>
<p>“by hand” (using pytorch tensor operations) and compare your result with that<br>
produced by autograd (that is, by calling <code>.backward()</code>).</p>
<p>Compute a plain-vanilla gradient-descent step for some simple model and<br>
compare it to the result given by <code>SGD</code>.</p>
<p>Implement your own version of <code>Linear</code> as a custom module, together with its<br>
appropriate <code>forward()</code> and <code>backward()</code> methods, and compare its results with<br>
those of pytorch’s <code>Linear</code>.</p>
<p>Experiment with some of the basic loss functions such as <code>CrossEntropyLoss</code><br>
and <code>MSELoss</code>.  Consider implementing some of your own versions just for fun.</p>
<p>Use pytorch (autograd, appropriate loss function, and an optimizer) to perform a<br>
linear and/or quadratic regression of some (potentially synthetically generated)<br>
dataset.</p>
<p>Learn what <code>requires_grad = True</code> does, what leaf variables are (look at a<br>
tensor’s <code>.is_leaf</code> property), how autograd’s “computation graph” is built and<br>
works, what an “inplace modification” is, and track a tensor’s <code>._version</code> property<br>
as you modify that tensor inplace a couple of times.</p>
<p>These are the basic building blocks of how pytorch trains models.  If you’re a<br>
fundamentals kind of guy, you will want to understand all of the above in some<br>
detail.  If you do, you’ll have a solid foundation.</p>
<p>Pytorch has a lot of stuff in it.  So after this, you will have to pick and choose which<br>
pieces you want to learn in greater depth.  Perhaps you’re applying pytorch to a<br>
particular task and you’re running some task-specific code that you found on the<br>
internet.  If something looks fishy to you – or just looks interesting – use that as an<br>
excuse to drill down into it along the lines described above for exploring the basics.</p>
<p>Even if you don’t have a reason to use any of pytorch’s “built-in” models (for<br>
example, things like <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html" rel="noopener nofollow ugc">resnet18)</a> for any of your real use cases, it would still be<br>
worthwhile to experiment with some of them.  This should give you exposure to<br>
(hopefully) best practices for building more complicated models.  Look at the<br>
architecture of a built-in model and see if you can modify it – for example, modifying<br>
or adding a layer or changing an activation – by performing “surgery” on the model<br>
after its been instantiated, rather than editing the source code or rewriting it from<br>
scratch.  This will help you learn how the pieces of the model are hooked together<br>
and how to use the run-time flexibility of python (and pytorch) to your advantage.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: GrandArth
cooked: <p>Thx Frank! I really appreciate this!</p>

username: Thomas_Paul
cooked: <p>Thank you. This is gold.</p>