akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Writing a custom C++ extension with variants for both CPU and GPU?

MalayAgr: <p>Hi.</p>
<p>I am interested in writing a custom C++/CUDA extension.</p>
<p>The tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">here</a> only shows a scenario where you have a pure CUDA kernel which will not work on a machine which doesn’t have CUDA. I’d like to make an extension which uses a CPU version OR the CUDA version automatically, akin to what PyTorch itself does. I was wondering if there was a way to use the dispatcher in PyTorch to accomplish this in the extension. I know this can be done using if statements or <code>#if</code> preprocessors. But I’d like a slightly more “automatic” solution.</p>
<p>Thank you!</p>
<p>P.S.: I have already searched the forums for this topic but couldn’t find a solution.</p>
How to compute the K-nn graph of a tensor using pytorch?

amitoz: <p>I have a tensor say,</p>
<pre><code class="lang-auto">a = torch.random(10,2)
</code></pre>
<p>I would like to create a  knn graph of this tensor <code>a</code> using torch such that it returns me <code>k</code> indices and distances, for each row of this <code>a</code> tensor. That is to say <code>distances.shape</code> is <code>[10,k]</code> and <code>indices.shape</code> is <code>[10,k]</code></p>
<p>Basically I look  to do <a href="https://scikit-learn.org/stable/modules/neighbors.html#neighbors" rel="nofollow noopener">something like this</a> <strong>(section 1.6.1.1. Finding the Nearest Neighbors)</strong> using torch.  It is not very evident to me how to do this?</p>

amitoz: <p>One way, which I am able to arrive is:</p>
<pre><code class="lang-auto">dist = torch.sort(torch.cdist(a,a),dim=1)
</code></pre>
<p>Then assuming <code>k=3</code></p>
<pre><code class="lang-auto">distances = dist.values[:,0:k]
indices = dist.indices[:,0:k]
</code></pre>
<p>This way I have what I wanted, but perhaps there are more efficient ways to achieve this?<br>
PS: This method is highly memory inefficient is <code>kd-tree</code> calculation possible using torch?</p>

Xiaotian_Liu: <p>Hi amitoz, I think the torch_cluster has a function you can directly call to compute the knn graph of a given torch tensor.</p>
<pre><code class="lang-auto">from torch_cluster import knn_graph

graph = knn_graph(a,k,loop=False)
</code></pre>
<p>Set loop=True if wish to include self-node in graph.</p>
Torch operation on nn.Parameter

blade: <p>I have a list of parameters and I would like to sum all the elements in the list</p>
<pre><code class="lang-auto">import torch
from torch import nn

a = nn.Parameter(torch.rand(1))
b = nn.Parameter(torch.rand(1))

my_list = [a, b]
torch.sum(*my_list)
</code></pre>
<p>I receive the error</p>
<blockquote>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;input&gt;", line 8, in &lt;module&gt;
TypeError: sum() received an invalid combination of arguments - got (Parameter, Parameter), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
</code></pre>
</blockquote>
<p>I was wondering if there is a way to perform operations like <code>torch.sum</code> on <code>Parameter</code>s?</p>

Manuel_Lopez: <p>I think you confused the Op <code>torch.sum</code> with <code>torch.add</code>.</p>
<p><code>torch.sum: (*input* , *** , *dtype=None* ) → [Tensor]</code><br>
Returns the sum of all elements in the <code>input</code> tensor.<br>
Docs: <a href="https://pytorch.org/docs/stable/generated/torch.sum.html#torch-sum" class="inline-onebox" rel="noopener nofollow ugc">torch.sum — PyTorch 1.11.0 documentation</a></p>
<p><code>torch. add: (*input*, *other*, ***, *alpha=1*, *out=None* ) → [Tensor]</code><br>
Adds <code>other</code>, scaled by <code>alpha</code>, to <code>input</code>.<br>
Docs: <a href="https://pytorch.org/docs/stable/generated/torch.add.html#torch-add" class="inline-onebox" rel="noopener nofollow ugc">torch.add — PyTorch 1.11.0 documentation</a></p>
<p>It works if you do:</p>
<pre><code class="lang-auto">import torch
from torch import nn

a = nn.Parameter(torch.rand(1))
b = nn.Parameter(torch.rand(1))

my_list = [a, b]

a = torch.add(*my_list)
</code></pre>
EMA with batch norm seems to be not working...?

Sam_Lerman: <p>I implement EMA as follows:</p>
<pre><code class="lang-auto">for param, target_param in zip(net.parameters(), target_net.parameters()):
        target_param.data.copy_(ema_tau * param.data + (1 - ema_tau) * target_param.data)
</code></pre>
<p>However, one of the models that I apply this to which uses batch norm doesn’t work with this kind of parameter copy. I mean, it works, but it doesn’t learn. Other models perform better with EMA, so I’m not sure why this one breaks. It’s a ResNet18 to be specific.</p>
Can we set gain in init.kaiming_normal_?

chenwydj: <p>In <code>xavier_normal_ </code>, we can manually set the <code>gain</code>.</p>
<p>But in <code>kaiming_normal_</code>, the gain is calculated by <code>gain = calculate_gain(nonlinearity, a)</code> inside the function.</p>
<p>Is there any way we can also manually set <code>gain</code> for <code>kaiming_normal_</code>? If not, may I know the reason behind this design?</p>
<p>Many thanks!</p>

ptrblck: <p>I guess the current interface might be more convenient for standard use cases.<br>
If you want to manually define the <code>gain</code>, you could create your own custom function e.g. as:</p>
<pre><code class="lang-python">fan = torch.nn.init._calculate_correct_fan(tensor, mode='fan_in')
gain = ... # your gain calculation
std = gain / math.sqrt(fan)
with torch.no_grad():
    tensor.normal_(0, std)
</code></pre>

chenwydj: <p>Thanks! I will have a try.</p>
Gradient-vector product and vector-gradient product for higher-order gradients

exenGT: <p>Hi,</p>
<p>I’m trying to find out what’s the best way to calculate the gradient-vector product and the vector-gradient product for hessian, and third-order gradients?</p>
<p>Or, more generally, how to calculate the products</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b66a11b1dcdc0c6cd187a4358c88bd0d7a41656.png" alt="image" data-base62-sha1="3UoLZhlchAFe7z44kkm6wCJrWWG" width="186" height="226"></p>
<p>if A is a tensor?</p>
<p>Thanks,<br>
JW</p>
Mixed dimension difference operation with einsum

pumplerod: <p>I’m struggling with the einsum notation.</p>
<p>In the simple case where I have tensor <code>A</code> size: (1, 5) and I want to find the difference over a series of vectors <code>B</code> size: (6, 5) I am able to simply do something like <code>result = A - B</code> .   For my batch training the dimensions increase.  For example a batch of 3: <code>A</code> size: (3, 5) <code>B</code> size: (3, 6, 5).  I now need to find the difference matrix for each vector in A relative to each matrix in B.</p>
<p>I believe there is probably a way to do this with torch.einsum() however I cannot wrap my head around the process.  I’m able to successfully multiply them as I would expect using: <code>torch.einsum('ij,ikj-&gt;ikj', [A, B])</code> but I cannot figure out how to write the statement to effectively perform the difference calculation.  I’ve found a few resources for einsum documentation but I’m still a bit confused.  Is this actually possible with einsum()?  Or is there another option?</p>

KFrank: <p>Hi Pumplerod!</p>
<aside class="quote no-group" data-username="pumplerod" data-post="1" data-topic="146536" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pumplerod/48/36129_2.png" class="avatar"> pumplerod:</div>
<blockquote>
<p>something like <code>result = A - B</code></p>
</blockquote>
</aside>
<p>Use <a href="https://pytorch.org/docs/1.11/notes/broadcasting.html" rel="noopener nofollow ugc">broadcasting</a> with an <code>unsqueeze()</code> to get the dimensions lined<br>
up properly:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; A = torch.randn (3, 5)
&gt;&gt;&gt; B = torch.randn (3, 6, 5)
&gt;&gt;&gt;
&gt;&gt;&gt; resultBroadcast = A.unsqueeze (1) - B
&gt;&gt;&gt; resultBroadcast.shape
torch.Size([3, 6, 5])
</code></pre>
<aside class="quote no-group">
<blockquote>
<p><code>torch.einsum('ij,ikj-&gt;ikj', [A, B])</code> but I cannot figure out how to write the statement to effectively perform the difference calculation.</p>
</blockquote>
</aside>
<p>But if you really want to use <code>einsum()</code> you can just convert addition<br>
into multiplication:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; resultEinsum = torch.einsum('ij,ikj-&gt;ikj', A.exp(), (-B).exp()).log()
&gt;&gt;&gt; torch.allclose (resultBroadcast, resultEinsum)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

pumplerod: <p>That is so helpful!  Thank you.  Of course, using log for the matrix multiplication makes sense.  I have a lot to learn about einsum.  I’ve only just discovered it, but looks super powerful.</p>
Using `num_workers > 0` for CPU-devices?

Imahn: <p>Hi,</p>
<p>when I do the training of my model on a CPU-only device, does it make sense to choose <code>num_workers &gt; 0</code>? From the <a href="https://pytorch.org/docs/stable/data.html" rel="noopener nofollow ugc">documentation</a>, it’s not entirely clear to me, as it only states:</p>
<blockquote>
<p>[…] how many subprocesses to use for data loading. <code>0</code> means that the data will be loaded in the main process. (default: <code>0</code>)</p>
</blockquote>
<p>I’ve also taken a brief look at the <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py" rel="noopener nofollow ugc">source code</a>, but again, it is not obvious to me whether <code>num_workers &gt; 0</code> would make sense on a CPU-only device or not.</p>
<p>(For <code>pin_memory</code>, on the other hand, it’s pretty clear this only makes sense when the training is done on GPU, since it states:</p>
<blockquote>
<p>If <code>True</code>, the data loader will copy Tensors into CUDA pinned memory before returning them.</p>
</blockquote>
<p>)</p>

ptrblck: <p>Yes it could make sense to use multiple workers, as they would load the next batch(es) in the background while the model is being trained. Depending which operations are used to train the model and if they are able to use all CPU cores, you might or might not see a speedup, so you should check different configs for your actual workload.</p>
Same generations for every sample in batch with Seq2Seq model - What to do?

AfonsoSalgadoSousa: <p>Hi there. I have a Seq2Seq model trained for paraphrase generation. Everything seems to be working fine, as the loss goes down, but the generations are not good. They are all the same for every entry in every batch. They differ between epochs as the weights are updated, but they keep being the same for every batch. I already tried training for several (~100) epochs. Do you have any suggestions for me to fix the problem?</p>
How to take the gradient of the updated A w.r.t B where A is updated using B

bilibili: <p>Imagine the following example:</p>
<p>A = nn.Linear(3, 4)<br>
B = torch.ones(3).requires_grad_()</p>
<p>loss1 = A(B).sum()<br>
grad1 = torch.autograd.grad(loss1, A.parameters(), create_graph=True)</p>
<p># then update A with grad1</p>
<p>loss2 = F(A)</p>
<p>My question is: how should I update A such that I can calculate the gradient of loss2 w.r.t B?<br>
if both A and B are tensors, I know how to do it. But in case that A is a nn.Module, how should I do it?</p>

soulitzer: <p>It’s not possible to compute gradient of loss2 if it isn’t a tensor. Do you mean to do something like out = A(b); loss1 = out.sum(); loss2 = F(out)?</p>

bilibili: <p>Thanks for your reply! Unfortunately that is not the case. What i mean is:</p>
<ol>
<li>
<p>we first update A, the gradient of A is a function of B.</p>
</li>
<li>
<p>After the update, I want to calculate a loss of the updated A. But then take derivative w.r.t B (since the updated A is a function of B).</p>
</li>
</ol>
<p>I want to achieve the above but do not know how to do it if A is of type nn.Module</p>

bilibili: <p>I have actually found the solution. One can use the higher library for this. <a href="https://github.com/facebookresearch/higher" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.</a></p>
DDP Model Parameters different from Single-process Model Parameters

csl: <p>I am currently running two versions of the same model (DDP and single GPU) and am trying to ensure the model parameters are the same between the two. To do so, I load a checkpoint at the first epoch of both models and run it for several iterations. When I load the checkpoint the initial parameters are the same (DDP is running two processes in this example):</p>
<pre><code class="lang-auto">DDP Model Output
Rank: 0
model.state_dict():  OrderedDict([('module.module.input_networks.0.0.net.net.0.weight', tensor([[ 0.3889,  0.1429,  0.1801,  0.2079],
        [ 0.4926, -0.3095, -0.1581, -0.1851],
        [ 0.2123, -0.4082, -0.3036,  0.1350],
        [ 0.3666,  0.2588,  0.3510, -0.4564],
Rank: 1
OrderedDict([('module.module.input_networks.0.0.net.net.0.weight', tensor([[ 0.3889,  0.1429,  0.1801,  0.2079],
        [ 0.4926, -0.3095, -0.1581, -0.1851],
        [ 0.2123, -0.4082, -0.3036,  0.1350],
        [ 0.3666,  0.2588,  0.3510, -0.4564],

Single GPU Model Output
model.state_dict():  OrderedDict([('input_networks.0.0.net.net.0.weight', tensor([[ 0.3889,  0.1429,  0.1801,  0.2079],
        [ 0.4926, -0.3095, -0.1581, -0.1851],
        [ 0.2123, -0.4082, -0.3036,  0.1350],
        [ 0.3666,  0.2588,  0.3510, -0.4564],

</code></pre>
<p>However, once I run them both for 1 iteration, the parameters differ (by 0.001 to 0.002) and the outputs look like this:</p>
<pre><code class="lang-auto">DDP Model Output
Rank 0: 
model.state_dict():  OrderedDict([('input_networks.0.0.net.net.0.weight', tensor([[ 0.3886,  0.1431,  0.1805,  0.2080],
        [ 0.4927, -0.3096, -0.1583, -0.1849],
        [ 0.2124, -0.4080, -0.3040,  0.1352],
        [ 0.3663,  0.2588,  0.3514, -0.4563],
Rank 1: 
model.state_dict():  OrderedDict([('input_networks.0.0.net.net.0.weight', tensor([[ 0.3886,  0.1431,  0.1805,  0.2080],
        [ 0.4927, -0.3096, -0.1583, -0.1849],
        [ 0.2124, -0.4080, -0.3040,  0.1352],
        [ 0.3663,  0.2588,  0.3514, -0.4563],

Single GPU Model Output
model.state_dict():  OrderedDict([('module.module.input_networks.0.0.net.net.0.weight', tensor([[ 0.3888,  0.1430,  0.1802,  0.2078],
        [ 0.4925, -0.3094, -0.1582, -0.1850],
        [ 0.2124, -0.4081, -0.3037,  0.1351],
        [ 0.3665,  0.2587,  0.3511, -0.4565],
</code></pre>
<p>Does anyone have experience with why this may be happening? My model does not have any dropout layers and should be updating the same way.</p>

rvarm1: <p>Hi, for DDP versus local training, is the following true for your model?</p>
<ol>
<li>Learning rate is consistent across distributed and local training</li>
<li>If your batch size is M locally, then a local batch size of M / N across N workers with average loss should result in equivalent training to local.</li>
</ol>
<p>By “iteration”, are you referring to forward + backward + optimizer step? Before the optimizer step, it may also be useful to evaluate model gradients to ensure that they are the same across local and distributed training.</p>
Training across different machines

nsriniva03: <p>When setting up a framework to perform distributed training do the machines have to been in the same data center or on the same network or can it be done when the machines are in different data centers?</p>

rvarm1: <p>The only requirement is that the two machines are able to establish a connection between each other, although note that running training on machines in two different TCs will likely cause a significant slowdown compared to them running on the same rack.</p>

nsriniva03: <p>Thanks <a class="mention" href="/u/rvarm1">@rvarm1</a>. I had the same thoughts on speed.</p>
Torch.eq and torch.equal behaviors

Qiyao_Wei: <p>MWE below. I have also printed out c and d, so I am pretty sure they have the same elements. Am I misunderstanding something about comparing the difference between two tensors?</p>
<pre><code class="lang-auto">import torch
a = torch.rand(3,2,5)
b = torch.rand(3,5,4)
c = torch.stack([a[i] @ b[i] for i in range(3)])
d = torch.einsum('bij,bjk-&gt;bik', a, b)
print(torch.all(torch.eq(c,d))) #returns False
print(torch.equal(c,d)) #returns False</code></pre>

KFrank: <p>Hi Qiyao!</p>
<aside class="quote no-group" data-username="Qiyao_Wei" data-post="1" data-topic="146475" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qiyao_wei/48/34623_2.png" class="avatar"> Qiyao_Wei:</div>
<blockquote>
<p>Am I misunderstanding something about comparing the difference between two tensors?</p>
</blockquote>
</aside>
<p>This is due to <a href="https://en.wikipedia.org/wiki/Round-off_error" rel="noopener nofollow ugc">round-off error</a>.</p>
<p>Your calculations for <code>c</code> and <code>d</code> are mathematically equivalent, but the<br>
order of the operations differ (in a mathematically-equivalent way),<br>
introducing numerical round-off error.</p>
<p>This can be seen by modifying your example to check not just for<br>
exact equality between <code>c</code> and <code>d</code>, but for <code>c</code> and <code>d</code> being nearly equal<br>
to one another by using <code>torch.allclose()</code>.</p>
<p>Here is a modified version of your script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
_ = torch.manual_seed (2022)
a = torch.rand(3,2,5)
b = torch.rand(3,5,4)
c = torch.stack([a[i] @ b[i] for i in range(3)])
d = torch.einsum('bij,bjk-&gt;bik', a, b)
print(torch.all(torch.eq(c,d))) #returns False
print(torch.equal(c,d)) #returns False
print (torch.allclose (c, d))   #returns True
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.10.2
tensor(False)
False
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
mAP for a single image

ksmdanl: <p>I am building a model that requires calculating the mAP per image (I’d call it running mAP) for evaluation. The way I’ve been doing it is as follows:</p>
<pre><code class="lang-auto">aps = []
existing_class = np.unique(truth[:, -1])
for classidx in existing_class:
	if truth[truth[:, -1] == classidx].size &gt; 0:
		rec, prec, ap = compute_overlap(classidx, pred_boxes, pred_confs, pred_labels, truth)
		aps.append(ap)
	else:
		continue
mean_ap = np.mean(aps)
</code></pre>
<p>Basically, it iterates only over the existing classes in the respective image, unlike the original implementation, where for overall evaluation the computation iterates over all classes, 20 on Pascal VOC or 9 on KITTI.</p>
<p>I’ve been wondering if it makes sense to do it the way I do with the code snippet I just posted.<br>
Looking forward to exchange!</p>
If it is possible to use swig to call c++ inferface by python

kasen_zhou: <p>pytorch:1.3<br>
libtorch:1.3<br>
cuda:10.1<br>
ubuntu:18.04<br>
python:3.6</p>
<p>i want to code a common predict server, separate the whole flow into three parts:preprocess, predict and postprocess, among them, preprocess and postprocess are coded by python, preidct is coded by c++, the definition in c++ is :<br>
<code>at::Tensor do_predict(std::vector&lt;torch::jit::IValue&gt; inputs)</code><br>
c++ module is wrapped by swig, and called in python:</p>
<pre><code class="lang-auto">with open(r"cat.jpg", 'rb') as f:
    image_bytes = f.read()
tensor = []
t = preprocess.transform_image(image_bytes)
tensor.append(t)
res = predict.do_predict(tensor)
</code></pre>
<p>as you can see, it got the error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "app.py", line 24, in &lt;module&gt;
    res = predict.do_predict(tensor)
TypeError: in method 'do_predict', argument 1 of type 'std::vector&lt; torch::jit::IValue &gt;'
</code></pre>
<p>i really confused whether the whole solutions(python-&gt;c+±&gt;python) is ok, if it is ok, how to transform list&lt;torch.tensor&gt;<br>
to std::vector&lt; torch::jit::IValue &gt;, and at::Tensor to torch.tensor. if you know, please tell me, i will really appreciate</p>

SCM: <p>Was there ever a solution to this?</p>
Upgrading from 1.8 to 1.10.2 causes DDP to fail, socket timeout

divinho: <p>I’m doing multinode DDP, one process per node. Upgraded pytorch causes problems for some reason.</p>
<p>The specific error is a timeout after calling <code>barrier()</code></p>
<pre><code class="lang-auto">  File "/remote/idiap.svm/temp.speech01/rbraun/code/speechbrain/speechbrain/utils/distributed.py", line 104, in ddp_barrier
    torch.distributed.barrier()
  File "/idiap/temp/rbraun/programs/anaconda3/envs/speech/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2709, in barrier
    work = default_pg.barrier(opts=opts)
RuntimeError: Socket Timeout
</code></pre>
<p>Why would an upgrade to pytorch cause this?</p>

pritamdamania87: <p><a class="mention" href="/u/divinho">@divinho</a> Any chance you could share a minimal repro for this passing on 1.8 and failing on 1.10.2? Also, I’m assuming you are using ProcessGroupGloo here?</p>

divinho: <p>Installing 1.10.1 with cuda 11.3 fixed the problem. think the problem might have been caused by the framework I was using (speechbrain)</p>
How to change activation of a GRU layer

Ahmed_m: <p>Hi,<br>
I am trying to use a GRU with two layers using <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html?highlight=gru#torch.nn.GRU" rel="noopener nofollow ugc">this</a> module as in intermediate block in my model. Something like:</p>
<p><code>input -&gt; Conv -&gt; FC -&gt; GRU -&gt; FC -&gt; Conv -&gt; output</code></p>
<p>However, the <code>tanh</code> activation at the output of the GRU sounds to hinder the training. Any idea how I can disable the <code>tanh</code> activation and use a linear one instead?</p>
Different weight for samples in nn.Linear

daveatcode: <p>Hello,</p>
<p>I have a model that would like to apply different weight per sample in a batch. For example, if the batch size is 3, in_feature is 108, out_feature is 2, then I have the batched input X (shape: (3, 108)), and corresponding weight W (shape: (3, 2, 108)).</p>
<p>Originally, if the weight is the same for every sample in a batch, the weight W shape is (2, 108), which can be easily plugged in nn.functional.linear(X, W).</p>
<p>So far I iterate over each sample for computing the linear layer, i.e.  nn.functional.linear(X[i], W[i]) for i = [0, batch_size). However, this is very inefficient, so I am wondering that is there any utility function that would be helpful in this case?</p>
Why local_rank and not global rank

boda_Sadalla: <p>why do we use local rank  in distributed training for logging and saving models, instead of using the global rank,  my understanding is when we use local rank, then we would log and save once for each node, while it would happen just once across all nodes if we used the global rank</p>

suraj.pt: <p>Yes, the rank needs to be the global rank in case of multi-node distributed training. Are you referring to a particular tutorial/example which uses local rank in a multinode setting? It should have line that converts local to global, like <code>global_rank = local_rank * ngpus_per_node + gpu</code> where <code>local_rank</code> is the node rank and <code>gpu</code> is the device id on the node</p>

boda_Sadalla: <p>Yes, One example im working with  is NVIDIA’s deep learning examples repo <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/Tacotron2/train.py#:~:text=stack(cuda_rng_states_all).cpu()-,if%20local_rank%20%3D%3D%200%3A,-checkpoint%20%3D%20%7B%27epoch%27%3A%20epoch" rel="noopener nofollow ugc">here</a>:  they are using the local rank and that sometimes can cause errors, specially when two nodes are trying to write on the same time</p>

rvarm1: <p>I think the goal with that code is to take a checkpoint on each node, so that nodes can individually reload checkpoints if training crashes and there is a need to resume.</p>
<p>For example, if checkpoint is taken only on global rank 0 and that node crashes, training information would be lost while if it was taken on all nodes, there is a higher likelihood of availability.</p>

boda_Sadalla: <p>aha, I understand your point, but then how we account for the bugs addressed <a href="https://github.com/NVIDIA/DeepLearningExamples/issues/1092" rel="noopener nofollow ugc">here</a>.<br>
also in my case, I an using slurm, so all nodes save checkpoint in a centralized fashion on my training folder, so there’s no need for that.</p>
<p>I am just wondering if my understanding that on most cases, we need to use the <code>global_rank</code> for logging, saving, and not the <code>local_rank</code></p>
How to properly save conv2d weights?

Anirban_Nath: <p>So I have just 1 layer of conv2d defined as follows: -<br>
<code>self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)</code></p>
<p>Now I want to save its state dictionary for future use. I thought that the state dictionary is a python dictionary that contains the weights of the model but when I execute<br>
<code>print(self.proj.state_dict)</code><br>
I get<br>
<code>&lt;bound method Module.state_dict of Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4))&gt;</code></p>
<p>And if I execute<br>
<code>print(self.proj.state_dict.keys())</code><br>
I get<br>
<code>AttributeError: 'function' object has no attribute 'keys'</code></p>
<p>So how do I get the weights?</p>

ptrblck: <p>You would have to call the <code>state_dict</code> method via:</p>
<pre><code class="lang-python">print(self.proj.state_dict())
</code></pre>

Anirban_Nath: <p>Thanks a lot! This was the problem.</p>
CUDA out of memory for StyleGAN pretrained generator

avalon1511: <p>Hi,</p>
<p>I am trying to use the StyleGAN pretrained generator for inference from the <code>psp.py</code>model here: <a href="https://github.com/eladrich/pixel2style2pixel" class="inline-onebox" rel="noopener nofollow ugc">GitHub - eladrich/pixel2style2pixel: Official Implementation for "Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation" (CVPR 2021) presenting the pixel2style2pixel (pSp) framework</a>. The setup is as follows:</p>
<ol>
<li>
<p><code>psp.py</code> in the <code>models</code> folder has two modules - an encoder and decoder. The decoder is a pretrained sytlegan generator.</p>
</li>
<li>
<p>I initialize the model as</p>
</li>
</ol>
<pre><code class="lang-auto">opts = Namespace(**opts)
net = pSp(opts, ngpu)
</code></pre>
<p>where <code>ngpu = 2</code> in my code.</p>
<ol start="3">
<li>I then perform</li>
</ol>
<pre><code class="lang-auto">decoder = net.decoder.to(device)

if (device.type == 'cuda') and (ngpu &gt; 1):
    decoder = nn.DataParallel(decoder, list(range(ngpu)))
</code></pre>
<ol start="4">
<li>I put the decoder in eval mode.</li>
</ol>
<p>While actually running the inference though I keep getting CUDA out of memory errors which look like this:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 2.25 GiB (GPU 0; 7.79 GiB total capacity; 
3.57 GiB already allocated; 1.52 GiB free; 4.67 GiB reserved in total by PyTorch) If 
reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>What I understand from this error is that the decoder is being loaded onto GPU:0 and running out of memory during inference.</p>
<p>Is there a workaround to this? Not sure why my code is not using GPU 1 as well.</p>
<p>Note: When I run the inference on the whole <code>psp.py</code> model and not just the decoder, I don’t have any issues with CUDA memory errors. Not sure why the whole model seems to work but the decoder itself doesn’t.</p>

ptrblck: <p><code>nn.DataParallel</code> will add a memory overhad on the default device as described in e.g. <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>. The recommended approach is thus to use <code>DistributedDataParallel</code> with a single process per GPU.</p>

avalon1511: <p>Thanks, will look into that ! Although I cannot understand why this error is only happening when I separately call <code>net.decoder()</code> and not when I run <code>net()</code> as a whole. Would you happen to have some insight?</p>
Torch.no_grad() removes requires_grad when the weight tensor updated by non-compound-operation

ahb7: <p>I created a simple Linear Regression using PyTorch (version 1.11)</p>
<p>My question is :<br>
is not the statement “w -= lr * w.grad” same as “w = w - lr * w.grad”<br>
Then why in one case my program passes, in the other case, it fails.</p>
<p>It fails when my code has:<br>
with torch.no_grad():<br>
w = w - lr * w.grad</p>
<p>print(w)<br>
tensor(0.3000)</p>
<p>But it works fine when I change the line as::<br>
with torch.no_grad():<br>
w -= lr * w.grad</p>
<p>print(w)<br>
tensor(2.0000, requires_grad=True)</p>
<p>In [19]: torch.<strong>version</strong><br>
Out[19]: ‘1.11.0’</p>
<p>I already filed a bug which is closed now. Bug number <span class="hashtag">#74158</span><br>
I think this is a valid bug, though a minor one.<br>
Please comment. Thank you for your help.</p>

ptrblck: <aside class="quote no-group" data-username="ahb7" data-post="1" data-topic="146410">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahb7/48/47544_2.png" class="avatar"> ahb7:</div>
<blockquote>
<p>is not the statement “w -= lr * w.grad” same as “w = w - lr * w.grad”</p>
</blockquote>
</aside>
<p>No, it’s not the same as the former applies the subtraction inplace on <code>w</code>.</p>
<p>From the docs of <code>no_grad()</code>:</p>
<blockquote>
<p>In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.</p>
</blockquote>
<p>The <code>w = w - lr * w.grad</code> is creating a new <code>w</code> tensor inside the <code>no_grad</code> context, which will not require gradients, so this is expected behavior.</p>

ahb7: <p>Thank you for clarifying.<br>
So let it be as it is.</p>
About ps_roi_pool_kernel.cu implementation

DiDiMouse: <p>I notify that im ps_roi_pool implementation, about fetching input data,  at about line 57, there is code like this:</p>
<pre><code class="lang-auto">// Add roi offsets and clip to input boundaries

    hstart = min(max(hstart + roi_start_h, 0), height - 1);

    hend = min(max(hend + roi_start_h, 0), height - 1); //## why height -1, bu tnot height?

    wstart = min(max(wstart + roi_start_w, 0), width - 1);

    wend = min(max(wend + roi_start_w, 0), width - 1);

    bool is_empty = (hend &lt;= hstart) || (wend &lt;= wstart);

    const T* offset_input =

        input + (roi_batch_ind * channels + c_in) * height * width;

    T out_sum = 0;

    for (int h = hstart; h &lt; hend; ++h) {

      for (int w = wstart; w &lt; wend; ++w) {</code></pre>
<p>similar code in roi_pool_kernel.cu at about line 51 DOES NOT do the subtraction, just use height instread of height -1, why? I ps_roi_pool\s current implementation, the data at the right and the bottom will be ignoreed in calculation?</p>
How pytorch uses an implicit for loop instead of an explicit for loop？

K.D: <p>The code is as follows:<br>
<code>    import torch     import numpy as np     x = torch.zeros((128, 3, 32, 32))     y = np.arange(128)     for i in range(len(y)):         x[:].uniform_(-y[i], y[i])    </code><br>
Can the for loop be replaced with something like array slicing? Thanks</p>

ptrblck: <p>I’m not sure I understand the code correctly, since you could just use the last <code>y</code> value via <code>y[-1]</code> to initialize <code>x</code>. Currently you are overwriting the previously used values in <code>x</code> and only keep the ones from the last iteration.</p>
Creating image masks for unets

Roua_Rouatbi: <p>I have a dataset with multiple objects in each image. The objects are bright spots. I have already used blobs_dog to detect the bright spots.</p>
<pre><code class="lang-auto">    image = io.imread(image_file)
    blobs_dog = blob_dog(image, min_sigma=min_sigma,
                         max_sigma=max_sigma, threshold=threshold)
    blobs_dog[:, 2] = blobs_dog[:, 2] * np.sqrt(2)

    num_objects = blobs_dog.shape[0]
    print(f"{num_objects} identified.")

    fig, (ax1, ax2) = plt.subplots(1,2)

    ax1.imshow(image, cmap=DEFAULT_CMAP, interpolation='none')
    for blob in blobs_dog:
        y, x, r = blob
        c = plt.Circle((x, y), r, color='lime', linewidth=2, fill=False)
        
        ax1.add_patch(c)

    ax1.set_axis_off()

    ax2.imshow(equalize_adapthist(image), cmap=DEFAULT_CMAP, interpolation='none')
    ax2.set_axis_off()
</code></pre>
<p>I now want to make masks of these images to pass them to a U-net.</p>

NearsightedCV: <p>I use labelme <a href="https://github.com/wkentaro/labelme" class="inline-onebox" rel="noopener nofollow ugc">GitHub - wkentaro/labelme: Image Polygonal Annotation with Python (polygon, rectangle, circle, line, point and image-level flag annotation).</a>.</p>
<p>It outputs in json so you need to do a bit of work to get it into a format for pytorch but it is free and straight forward with no nonsene</p>
Running a backwards pass in a custom context manager

cynthia-cb: <p>I’m interested in doing something like this with a backwards pass:</p>
<pre><code class="lang-auto">def Model(nn.Module):

    def __init__(self, params):
        super(Model, self).__init__()
        self.layer = nn.Linear(params["input"], params["output"])
        self.context_name = params["context"]

    def forward(self, x):
        return self.layer(x)

    def backward(self, x): # modified
        with new_context(self.context_name):
            &lt;call the regular backwards pass&gt;
</code></pre>
<p>Is there a tool with which I could do this? I realize I can put the context around the backwards call, e.g.</p>
<pre><code class="lang-auto">model = Model(params)
output = model(torch.ones((1, params["input"])))
with new_context(params["context"]):
    output.backward()
</code></pre>
<p>but due to the structure of the code I’m working with, this would require duplicating a good amount of code, which I want to avoid if possible. Thank you!</p>
<p>edit 1: forgot to put a context manager around the backward call</p>
Closing/Re-Using Data Loader Workers

hadarshavit: <p>Hi, I’m trying many configurations of my model using SMAC. Therefore, I’m training many models in the same script.<br>
To speed up each of the models training, I’m using persistent_workers=True. However, it causes to the workers (and the file descriptors) to remain.<br>
How can I close them after training a single model or re-use them in the other data loaders?</p>
<p>Thanks</p>

ptrblck: <p>Maybe <a href="https://discuss.pytorch.org/t/what-are-the-dis-advantages-of-persistent-workers/102110/10">this post</a> could help. I haven’t verified it, so let us know if it works for you.</p>
How do you save the images generated by a pytorch generative adversarial network?

GERMAIN_NICOLAS_MORA: <p>I have followed this <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc">Pytorch</a> tutorial on DCGAN to the letter, I managed to generate images successfully, however I cannot extract the generated images individually, they are only generated in the mosaic.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0dc7ac8340f3e5d99f08ab9185009ca4a71c752a.jpeg" alt="image" data-base62-sha1="1XU0oHuoiLix6UywEAWaugA9ABI" width="313" height="314"></p>
<p>Is there a way to save the generated images one by one?</p>

ptrblck: <p>The tutorial uses different approaches to visualize the entire grid:</p>
<pre><code class="lang-python">fig = plt.figure(figsize=(8,8))
plt.axis("off")
ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated=True)] for i in img_list]
ani = animation.ArtistAnimation(fig, ims, interval=1000, repeat_delay=1000, blit=True)

HTML(ani.to_jshtml())
</code></pre>
<p>and</p>
<pre><code class="lang-python">plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))
</code></pre>
<p>If you want to visualize specific images alone, remove the list comprehension or the usage of <code>vutils.make_grid</code>.</p>
Memory used by torch.distributed

YangFei: <p>I’m trying to profile the torch memory usage from the cuda API, i.e. <a href="https://pytorch.org/docs/stable/cuda.html" rel="noopener nofollow ugc">torch.cuda API</a>. I’m wondering if the memory used by <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">torch.distributed</a>’s collective call will be captured by the cuda API.<br>
For example, if I use NCCL as the backend and I made a call of <code>torch.distributed.allagther</code>. Will the memory used by NCCL (caching, communication buffer etc) be recorded by the torch.cuda API?</p>

rvarm1: <p>Yes, all CUDA memory consumed by NCCL (such as events, streams, comm. buffers, etc) will be captured by CUDA memory APIs such as torch.cuda.memory_stats.</p>
<p>Note that however the PyTorch profiler’s memory profiling features will not capture this.</p>
Is dist.broadcast_multigpu a misnomer?

esaliya: <p>I am implementing multi-tensor support in a 3rd party distributed backend for collectives such as broadcast and allreduce. However, the only python API that accepts multi-tensor input is collectives with <code>_multigpu</code> suffix like <code>dist.broadcast_multigpu</code>.</p>
<p>Looking at the implementation of these (e.g. <a href="https://github.com/pytorch/pytorch/blob/9b203f667ac096db9f5f5679ac3e3d7931c34d36/torch/distributed/distributed_c10d.py#L1103" rel="noopener nofollow ugc">here</a>), I don’t see anything specific to GPUs and it works equally well with the 3rd party distributed backend we have.</p>
<p>So, is the suffix <code>_multigpu</code> a misnormer?<br>
If so, can we contribute to change it? Of course, this would have backward compatibility issues.</p>
<p>Thank you,<br>
Saliya</p>

rvarm1: <p>Hi <a class="mention" href="/u/esaliya">@esaliya</a> , thanks for your question!</p>
<p>I don’t think <code>broadcast_multigpu</code> is necessarily a misnomer since the intent of the name is to imply that the list of tensors passed into the API must have each tensor on a distinct GPU. For example, see this check in one of the C++ backend implementations: <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L1698" class="inline-onebox" rel="noopener nofollow ugc">pytorch/ProcessGroupNCCL.cpp at master · pytorch/pytorch · GitHub</a></p>
<p>I do agree that to the implementor of the C++ backend, it is not clear that different tensors are expected as the input into <code>::broadcast</code>. Is this something that is blocking your implementation of the 3rd-party backend?</p>
Manual quantization vs prequantized model

viiiictorr: <p>Hi!</p>
<p>I’m working on an academic quantization project over Mobilenetv3. I have my Mobilenetv3_large already trained with my own dataset and now i want to perform some tests with dynamic quantization and QAT to compare it with my baseline.</p>
<p>I have did an intense search and I found several posts with manual modifications on the architecture or other ones directly redefining the entire model (most of them on mobilenetv2).</p>
<p>My question is: Does it make sense to fine tune the mobilenetv3 quantized model with my own dataset?</p>
<p>Thank you beforehand!</p>
<p>Victor</p>

HDCharles: <p>depends on your goal, if you want something incredibly small, performant and accurate like a phone speech recognition app then probably yes.</p>
<p>If you just want to perform some tests then it doesn’t seem worth it since the point would be to test how these APIs work on a given model, right?</p>
How to measure peak cuda memory usage?

b02202050: <p>Hi,<br>
Is there any way to measure the peak cuda memory usage without hurting the execution time?<br>
I know there is multiple functions kindly provided in torch.cuda. But they don’t include the CUDA overhead memory. i.e. There is always a gap between torch.cuda.max_memory_reserved() and what I see in nvidia-smi.<br>
An alternative solution would be initiating a thread to keep calling nvidia-smi and track the maximum value. But threading will hurt the execution time of the main process, which is also the other term I want to benchmark.<br>
Is there any solution for measuring peak gpu memory usage and also total execution time?<br>
Thanks very much.</p>

JuanFMontesinos: <p>torch.cuda.max_memory_reserved() (don’t know if that function or any similar)<br>
Shows the peak, not the real memory usage. Memory is reused on demand. When the allocator does not longer need the space it’s marked as available but not “freed” so that that memory slot can be overwritten.</p>
<p>It’s like hard disks, when you delete something you are not physically deleting it but marking that space as available.<br>
That’s why there is no pairing between nvidia-smi and memory</p>

b02202050: <p>Thanks @ <a href="https://discuss.pytorch.org/u/JuanFMontesinos">JuanFMontesinos</a>.<br>
But my problem seems not be solved.<br>
Does anyone have a solution?<br>
Thanks again.</p>

YangFei: <p><a class="mention" href="/u/b02202050">@b02202050</a> Did you get a solution for this?<br>
Regardless of the performance, simply running a background thread to query nvidia-smi seems not feasible. How do you set the query frequency to make sure you can capture the maximum memory usage?</p>
Dropout and training reproducibility

dadaddapyer: <p>Hi,</p>
<p>I just noticed that if I want to save a model and load to continue the training, dropout is the problem.<br>
I save everything I could save in order to continue the training precisely.<br>
I saved the followings:</p>
<pre><code>        "net": self.net.state_dict(),
        "optimizer": self.optimizer.state_dict(),
        "scheduler": self.scheduler.state_dict(),
        'train_data_provider': self.train_data_provider.get_state(),
        'test_data_provider': self.test_data_provider.get_state(),
        'torch_random': torch.get_rng_state(),
        'torch_cuda_state': torch.cuda.get_rng_state()}
</code></pre>
<p>But I found I still can not reload the net from epoch 10, for example, and produce the same results. Then I looked into my network, I found that the dropout layer is the fundamental problem. If I remove it, the procedure is exactly the same after loading the pre-trained weights.</p>
<p>So, I hope that PyTorch could make modifications such that we could save the state of the generator of dropout and load it when we use the function net.state_dict().</p>
<p>Best regards</p>

ptrblck: <p>I don’t think the dropout layer uses an own generator or seeding but is reusing the “global” state.<br>
If you want to reproduce deterministic behavior when starting from a specific epoch you could try to set the global seed in each epoch.</p>
MapDataPipe support Mux / Demux

josiahls: <p>I noticed that there isn’t the capability to Mux/Demux a MapDataPipe. The use case I’m looking at is splitting a MapDataPipe into train, validation, etc… datapipes. I wasn’t sure if this would constitute creating an actual issue on their github, especially if it was already planned.</p>

ejguan: <p>Thanks for taking interest. We are willing to review your PR for them.</p>

josiahls: <p>Ok sounds good, would it be a good idea to make an issue with what I come up with first? Once we agree what the implementation objects look fine I can make a proper pr with tests etc. It would probably be this weekend that I would have something made.</p>

ejguan: <p>We hava a Issue to track the missing MapDataPipe in PyTorch Core. <a href="https://github.com/pytorch/pytorch/issues/57031" class="inline-onebox" rel="noopener nofollow ugc">[DataLoader] Implement Map-style functional DataPipe · Issue #57031 · pytorch/pytorch · GitHub</a><br>
You can grab one and open a PR. If you want to discuss about the implementation, I am totally fine with either over issue or PR directly.</p>
Unexpected behavior of grid_sample

xianshunw: <p>As the docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html?highlight=grid_sample#torch.nn.functional.grid_sample" rel="noopener nofollow ugc">described</a>, the <code>grid_sample</code> performs value sampling in the <strong>closed</strong> range [-1, 1], i.e., <em>values <code>x = -1, y = -1</code> is the left-top pixel of <code>input</code>, and values <code>x = 1, y = 1</code> is the right-bottom pixel of <code>input</code></em>. But when I perform <code>grid_sample</code> on a volumetric (5D) input, the result is totally unexpected. The code snippet is as follow:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

data_dict = torch.load('data_dict.pth')

prob = data_dict['prob'].unsqueeze(0).unsqueeze(0)
sm = data_dict['sm'].unsqueeze(0).unsqueeze(0)

ret = F.grid_sample(prob, sm, mode='nearest')

print('--------result of zero padding----------')
print(ret[0, 0, 0, 0, 0])   # sampling result of (0, 0, 0)
print(sm[0, 0, 0, 0, :])    # samping coord of (0, 0, 0)
print(prob[0, 0, -1, 0, 0]) # samping target of (0, 0, 0)

ret = F.grid_sample(prob, sm, mode='nearest', padding_mode='border')

print('--------result of border padding----------')
print(ret[0, 0, 0, 0, 0])   # sampling result of (0, 0, 0)
print(sm[0, 0, 0, 0, :])    # samping coord of (0, 0, 0)
print(prob[0, 0, -1, 0, 0]) # samping target of (0, 0, 0)
</code></pre>
<p>Its running results are as follow:</p>
<pre><code class="lang-auto">--------result of zero padding----------
tensor(0.)
tensor([-1., -1.,  1.])
tensor(0.9999)
--------result of border padding----------
tensor(0.9999)
tensor([-1., -1.,  1.])
tensor(0.9999)
</code></pre>
<p>The result of  <code>ret[0, 0, 0, 0, 0]</code> should find value from <code>(-1, -1, 1)</code>  of prob which equals to 0.999, but the sampling result is 0. I guess this may be caused by kind of out-of-bound behavior, thus, I set the padding mode to <code>border</code> and the result makes sense.</p>
<p>What makes me confused is that where is the out-of-bound behavior comes from?  Does it come from float point accuracy loss? If so, how can we ensure the sampling results are correct at the boundaries?</p>
<p>Thanks.</p>

ptrblck: <p>Could you post example tensors for <code>prob</code> and <code>sm</code> which would reproduce the unexpected behavior?</p>
Why does the output Tensor shares the same quantization parameter with input

BobChen: <p>When preparing the model for QAT, why do we need to keep the input and output tensor share the same quantization parameter as described <a href="https://github.com/pytorch/pytorch/blob/7ddf212f3390f14abdbd5373f5fbdb04ffde5fd8/torch/ao/quantization/fx/quantization_patterns.py#L1500" rel="noopener nofollow ugc">here</a></p>

BobChen: <p>Have figured it out:)</p>

HDCharles: <p>For others stumbling across this, the answer is because that quantization handler is only for a specific set of functions whose output ranges can be inferred from their input ranges, i.e. ReLU’s output range will always be from 0 to the max of the input range.</p>