akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Weighting training instances

TinfoilHat0: <p>Is there a nice way to add custom weights to instances? So far, I’ve been weighting the losses directly after computing them via <em>criterion(outputs, labels).</em></p>
<p>However, if possible, I’d like to weight each instance before the training.</p>

Unity05: <p>Hi, many losses support class - wise weighting. Check out <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> for instance.</p>

TinfoilHat0: <p>Hi,</p>
<p>I’m aware we can add weights to loss functions, but I am curious if it’s possible to add weights directly to the training instances. I’m aware the end result would be the same though, however, adding weights to dataset is more convenient to me.</p>

Unity05: <p>Hi,</p>
<p>the only way to do some kind ‘sample - weighting’ is by setting <code>reduction='none'</code> (you’ll get the individual loss values) and multiply them with your custom sample - weight what you should therefore provide. <a href="https://discuss.pytorch.org/t/per-class-and-per-sample-weighting/25530">(This is actually the same as what is mentioned in this thread.)</a> But perhaps this is what you’ve already done.<br>
Other than that, I cannot think of an approach.</p>
<p>Regards.</p>

TinfoilHat0: <p>Yeah, that’s what I’ve been doing so far. I guess there is no other way of doing. Thank you for your help.</p>
Multilabel classification in pytorch

talhaanwarch: <p>I have a batch of shape <code>(20,28)</code> and the output produced by the model has the same shape as I am not using sigmoid in the last layer.  I am using BCEWithLogitsLoss as a criterion. How can I pass the y_pred and y_actual to this criterion. I mean in which shape should I modify the shape of the target and predicted value?</p>

KFrank: <p>Hi Talha!</p>
<aside class="quote no-group" data-username="talhaanwarch" data-post="1" data-topic="113204" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/talhaanwarch/48/26568_2.png" class="avatar"> talhaanwarch:</div>
<blockquote>
<p>How can I pass the y_pred and y_actual to this criterion.</p>
</blockquote>
</aside>
<p>For a multi-label, multi-class classification problem,<br>
<code>BCWEithLogitsLoss</code> is an appropriate loss function.  For it, your<br>
<code>y_pred</code> and <code>y_actual</code> should have the same shape, namely<br>
<code>[nBatch, nClass]</code>.</p>
<p>Whether the output of your model, <code>y_pred</code>, has the same shape<br>
as the input to your model will depend on your use case – it doesn’t<br>
have to.</p>
<p>(Note that <code>y_pred</code> and <code>y_actual</code> can have additional dimensions.<br>
For example, if you were performing a multi-label segmentation of<br>
an image, they would have shape <code>[nBatch, nClass, H, W]</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>
Attention over edges in Graph Neural Network

dmitrysarov: <p>My question is kind of puzzle. The question is related to Graph Neural Network. What I want to do is attention over adjoint to the same node edges. So what I got is tensor of edges features (<code>size = number_of_edges x number_of_fetures</code>) and tensor of node indecies (<code>size = number_of_edges</code>) indicating to what node particular adge is adjointed. For example:</p>
<pre><code class="lang-auto">edges_features = torch.randint(3, (5, 3))
edges_features
tensor([[1, 2, 1],
        [2, 2, 1],
        [2, 2, 1],
        [0, 1, 2],
        [2, 0, 2]])
</code></pre>
<p>here we have 5 edges with 3 features per each</p>
<pre><code class="lang-auto">node_index = torch.randint(2, (5,)) 
node_index
tensor([0, 1, 0, 0, 1])
</code></pre>
<p>two nodes. 0th node has 3 adjoin edge, 1st have 2 adjoin edge</p>
<p>For simplification, I consider key, query, and value as edge features. So what I need to do is calculate pairwise dot products of edge features correspond to the same node (for the 0th node it will be tensor 3x3, for the 1st matrix 2x2), then make softmax over results (row-wise) and after calculating a weighted sum of edge features edgewise. As result, I expect a tensor of the same size as the edges_features tensor.<br>
Any suggestions?</p>
Does pytorch use different linear algebra library when computation is carried out on CPU vs GPU

rsz: <p>Hi,</p>
<p>i have the following two settings for one sequence of length 1200</p>
<p>setting 1:<br>
input size: batch size by sequence length (1, 1200)<br>
for i in range(3):<br>
transformer(input[i*512: (i+1)*512])</p>
<p>setting 2:<br>
input size: batch size by sequence length-(3, 512)(this includes padding)<br>
transformer(input)</p>
<p>i assumed setting2 would be faster than setting1, and it is true if i use GPU, but if im using CPU, setting 2 is slower than setting1</p>
<p>can someone please explain why this is the case?</p>
<p>thank you!</p>

ptrblck: <p>To the original question: yes, PyTorch uses MAGMA and cuSolver for the GPU implementation (we are working on adding more cuSolver methods to the backend) and LAPACK on the CPU (I’m unsure, if other CPU libs are used).</p>
<p>My best guess would be that LAPACK (or the used CPU linalg library) is using e.g. AVX instructions for specific shapes, which might provide performance cliffs, but I would recommend to profile the workloads properly to see the more general picture.</p>

rsz: <p>hi,</p>
<p>thank you for explaining. im currenly just measuring the time mentioned in the above two settings, are you talking about breaking the time measurement into layers by layers in transformer? if possible, can you please suggest how to profile the workloads better</p>
<p>thank you!</p>

ptrblck: <p><a href="https://pytorch.org/docs/master/benchmark_utils.html"><code>torch.utils.benchmark</code></a> is a nice tool to create benchmark tables for different setups of specific operations and might be helpful to see the performance for different input shapes etc.</p>
Using the VIA annotation tool

AndriPi: <p>I have a fairly large dataset annotated manually for the instance segmentation task, using <a href="https://www.robots.ox.ac.uk/~vgg/software/via/" class="inline-onebox" rel="noopener nofollow ugc">Visual Geometry Group - University of Oxford</a>. Is it possible to read the VIA format into <code>torchvision</code>? If not, are there any tools to convert VIA <code>json</code> annotations in a format which can be consumed by <code>torchvision</code>? If not (again <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:">) which is the best annotation format for doing instance segmentation with <code>torchvision</code>?</p>

planet_pluto: <p>In my work, we use the VIA annotation tool for almost every DL task. If you would like to code (models, training loops, data loaders) from scratch, you can certainly use the VIA’s default JSON format. Or if you would like to use an already existing Instance Segmentation package like Facebook’s Detectron2 or YOLACT, you can convert your annotations to COCO format which is pretty standard for semantic/instance segmentation tasks. The VIA has such functionality for conversion. But remember that you may still need to write a custom python script to convert your COCO style JSON into the one that is accepted by the FB’s package.</p>
Error loading saved model

Shiv: <p>I have a model:</p>
<p>cnn=CNN()</p>
<p>torch.save(cnn, ‘./model/trained_model.pt’)</p>
<p>model = torch.load(’./model/trained_model.pt’)</p>
<p>AttributeError: Can’t get attribute ‘CNN’ on &lt;module ‘<strong>main</strong>’&gt;</p>
<p>why  am I getting this error and how to solve it?</p>

russellwmy: <p>I was having the similar error before.<br>
How I solved it by copying the code of CNN class to the same file that load the model.<br>
I think some hows the namespace is hard code when you save the model.<br>
So then it can’t search when you load the model again.</p>

Shiv: <p>Thanks <a class="mention" href="/u/russellwmy">@russellwmy</a>. It worked <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

russellwmy: <p>Good, But the way to solve it is … so bad <img src="//discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=5" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

Nasib_Ullah: <p>Thank you <a class="mention" href="/u/russellwmy">@russellwmy</a> .Anyone found any other way?</p>

thongnguyen: <p>Hi, I also have this problem and figure it out and just wanna post my solution here in case it helps anyone.</p>
<p>I read somewhere that Pytorch use Pickle to take care of the save/load model, the problem is really about Pickle, not Pytorch itself.</p>
<p>The solution is really simple, you just have to <strong><em>explicitly import</em></strong> the class definition. It’s equivalent to copy and paste the class definition.</p>
<p>In my case I have the following directory structure:</p>
<p>Release-code<br>
|----- myclass.py<br>
Application<br>
|----- app1.py</p>
<p>In app1.py, I make sure <code>./Release-code/</code> is in python path, and I just have to say</p>
<p><code>from myclass import class_definition</code></p>
<p>then <code>torch.load(model_name)</code> works</p>
<p>If your <code>class_definition</code> calls other classes, you also need to import them too.</p>
<p>This thread explains the problem and solutions for it: <a href="https://stackoverflow.com/questions/27732354/unable-to-load-files-using-pickle-and-multipile-modules" rel="nofollow noopener">https://stackoverflow.com/questions/27732354/unable-to-load-files-using-pickle-and-multipile-modules</a></p>

0tist: <p>I experienced the same error, simply re-running the cell in which your class is defined will do the trick if doing in jupyter notebook</p>

ftmpsb: <p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model</a></p>
<p>it says:<br>
The disadvantage of  Save/Load Entire Model is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself.<br>
and there is no other way:(</p>
Gradient of FC with respect to each sample in batch

moh_sen: <p>I want to get gradient of FC layer of network with respect to each sample. FC is last linear (dense) layer.</p>
<p>With batch size of 1, I can use following code</p>
<pre><code class="lang-auto">loss_CE = torch.nn.CrossEntropyLoss().cuda()

for i, (x,y) in enumerate(train_loader, 0):
  x = x.cuda()
  inputs = Variable(x, requires_grad = True)
  FV, Logit  = model(inputs)
  FV = Variable(FV, requires_grad = True)
  m, y_hat = torch.max(Logit, dim = 1)
  loss = loss_CE(Logit,y_hat)
  loss.backward()
  grad = model.fc.weight.grad
</code></pre>
<p>For ImagNet and Resnet-50 it produce a tensor of 1000 x 2048</p>
<p>It take a lot of time if I run on all images with batch size of 1.</p>
<p>If I increase the batch size, output of above code is still 1000 x 2048.</p>
<p>How can I modify to work in batch? output tensor size should be  256 x 1000 x 2048 when batch size is 256</p>

rboone: <p>autograd automatically condenses the gradient to the proper size for the weight regardless of the batch size.  If you just want to run your network with batches, you don’t need to modify the existing code as autograd will do that for you.</p>
<p>If, instead, you want manual access to the gradients, you may want to look at backward hooks: <a href="https://pytorch.org/docs/master/generated/torch.nn.modules.module.register_module_backward_hook.html#torch.nn.modules.module.register_module_backward_hook" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.module.register_module_backward_hook — PyTorch master documentation</a></p>
How do I calculate accuracy for multiclass problem?

jpj: <p>There are 15 nodes in my output layer, one for each class.<br>
I’m using CrossEntropyLoss to calculate the loss.<br>
How do I calculate the accuracy here, the number of predicted labels that are equal to the target labels?<br>
When I was using softmax and not CrossEntropyLoss I would just do target == torch.argmax(prediction) but since CrossEntropyLoss applies softmax itself, I’m not using softmax on final layer.</p>
<p>Would really appreciate if someone could help me out, thanks!</p>

KFrank: <p>Hi jpj!</p>
<aside class="quote no-group" data-username="jpj" data-post="1" data-topic="113117" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ed655f/48.png" class="avatar"> jpj:</div>
<blockquote>
<p>When I was using softmax and not CrossEntropyLoss I would just do target == torch.argmax(prediction) but since CrossEntropyLoss applies softmax itself, I’m not using softmax on final layer.</p>
</blockquote>
</aside>
<p>Please note that <code>softmax()</code> – which converts raw-score logits to<br>
probabilities – does not change the <em>order</em> of the values.  That is:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.argmax (prediction) == torch.argmax (torch.nn.functional.softmax (prediction))
</code></pre>
<p>Therefore you can continue to use:</p>
<pre data-code-wrap="python"><code class="lang-python">target == torch.argmax (prediction)
</code></pre>
<p>unchanged.</p>
<p>Best.</p>
<p>K. Frank</p>
Precision / Determinism / Reproducibility Issues

suraj.pt: <h3>Floating point precision</h3>
<p>Floating point precision limits the number of digits you can accurately rely on.<br>
<code>torch.float32</code> has precision between 6 and 7 digits regardless of exponent, i.e. precision issues are expected for values &lt; 1e-6.  Similarly <code>torch.float64</code> will have precision issues for values &lt; 1e-15.</p>
<h3>Non-determinism and reproducibility</h3>
<p>Non-determinism is expected behavior but there are some measures to control it. Read here for more information <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox">Reproducibility — PyTorch 1.7.1 documentation</a></p>
<ul>
<li>
<p>One of the first checks is to ensure the variables you’re comparing are indeed supposed to be equal. Ensure that you are reproducing results on the same platform, device and release version of PyTorch.</p>
</li>
<li>
<p>Set <a href="https://pytorch.org/docs/stable/generated/torch.set_deterministic.html#torch.set_deterministic"><code>torch.set_deterministic(True)</code></a> which will use the deterministic version of the algorithm if available (check the <a href="https://pytorch.org/docs/stable/generated/torch.set_deterministic.html#torch.set_deterministic">list of operations</a> that can be run deterministically) . Keep in mind that this will affect performance.</p>
</li>
</ul>
Create batches such that the data for one batch only come from the same tensor in a list?

kyy: <p>Hi, I am trying to train a neural network by creating batches such that data for any batch come from the same tensor from a list.</p>
<p>Specifically, suppose <code>my_list=[t1,t2,t3]</code> is a list of tensor, during training, I would like to have all data for each training batch come from either <code>t1, t2</code> or <code>t3</code>, meaning that any batch cannot mix data from two different tensors. One possible way I can think of is to use 3 dataloaders for these 3 tensors and iterate through them, but this may be complicated if I want to shuffle their order during training.   I am wondering if anyone know any better ways to do this?</p>
<p>Thank you!</p>
The second order derivative of a function with respective to the input

Zecheng_Zhang: <p>Hi,</p>
<p>I have a question about the second order derivative of with respective to the input. Say,</p>
<pre><code class="lang-auto">class Test(nn.Module):
    def __init__(self):
        super().__init__()
        
        bo_b = False
        bo_last = False

        self.l1 = nn.Linear(1, 1, bias = bo_b).to(device)

        self.l2 = nn.Linear(4, 1, bias = bo_b).to(device)
        
    def forward(self, v):
        v = self.l1(v) 
        v = self.l2(v) 
        return v

fnn = Test()

xx = [[0], [0.5], [1]]
xx_torch = torch.tensor(xx, requires_grad = True, device = device, dtype=torch.float32)

ge_out = fnn.forward(xx_torch)
df_dx = torch.autograd.grad(ge_out, xx_torch, 
                                   grad_outputs = torch.ones( xx_torch.size()), create_graph = True)[0]
df2_dx2 = torch.autograd.grad(  df_dx, xx_torch )[0]

</code></pre>
<p>But it gives me the error message: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
<p>Can anyone help me with the problem? Thanks a lot!</p>

albanD: <p>Hi,</p>
<p>The problem is that your function is linear. So the first gradient is constant and the second order gradient is independent of the input.<br>
This error message happens because of the independence (and thus, it is not used in the graph).</p>

Zecheng_Zhang: <p>Hi, Alban,</p>
<p>Thanks for your reply! I have an additional question. I see some post and they use df_dx.sum() instead of the df_dx in the torch.autograd.grad(), do you know why is that? Thanks!</p>

albanD: <p>It is equivalent to passing <code>grad_outputs = torch.ones( xx_torch.size())</code> as you do.</p>
Ideation & collaboration PyTorch projects

Deeply: <p>Hi all,</p>
<p>I wonder if anyone is aware about a PyTorch ideation and collaboration form. The goal is for people to share ideas about scientific bottlenecks they face (not bugs or technical issues) and perhaps establish some form of collaboration. I totally understand that sometimes an idea cannot be shared until published or used someway. Yet, we might be doing something and stuck somewhere and we never know if there’s a way out. Brainstorming might help us all resolve such issues, and probably, collaborate to achieve what we are thriving to.</p>
<p>Not sure if such ''ideation &amp; collaboration" can be added as a category in this forum. Who knows, maybe we see more people interested in sharing ideas and collaborate.</p>

Dwight_Foster: <p>There are various discords for AI where people discuss problems. There are also reddit forums.</p>
RuntimeError: Could not run 'aten::mm' with arguments from the 'QuantizedCPU' backend. 'aten::mm' is only available for these backends

naveen_raj: <p>RuntimeError: Could not run ‘aten::mm’ with arguments from the ‘QuantizedCPU’ backend. ‘aten::mm’ is only available for these backends: [CPU,  CUDA, SparseCPU, SparseCUDA, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivate Use2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
<pre><code class="lang-auto">h0 = torch.matmul(input, self.W[0])
</code></pre>
<p>i was trying to apply quantization to a graph convolutional neural network and during evaluation of the network this operation was not permitted. what options do i have to do torch.matmul as a quantizable operation.</p>

Vasiliy_Kuznetsov: <p>hi <a class="mention" href="/u/naveen_raj">@naveen_raj</a> , we currently do not have a quantized kernel for <code>aten::mm</code>.  There are a couple of options here:</p>
<ol>
<li>you could leave this op in fp32 (this is going to be the fastest in terms of dev time)</li>
<li>if someone is up for helping write the quantized kernel for <code>aten::mm</code>, we would accept a PR</li>
</ol>
Training pruned network not implemented in nn.uitls.prune?

ra_schoonhoven: <p>Hello,</p>
<p>I have been working with <code>nn.utils.prune</code> and it seems to me that you cannot currently train those networks after pruning right? We set <code>forward_pre_hooks</code> which means that the forward pass is properly pruned but I do not see anywhere how it is made sure that backprop works. Is it true that this is not implemented? Would it not just work to set the <code>backward_hooks</code> as well? I read the tutorial and looked for other forum posts but I did not find any mention of retraining pruned networks anywhere.</p>
<p>Kind regards,</p>
<p>Richard</p>

albanD: <p>Hi,</p>
<p>IIRC there is no need for backward hook as the pruning is done in a differentiable manner during the forward and so the autograd will give the right gradients.</p>

Song_zy: <p>Hi albanD,<br>
Thank you for your answer, and sorry for bringing back this topic. But here I thought if I pruned a model and then retrained it with backprop,  we are calculating the backward gradient with original weights, and update original weights then mask them? Because I observed that “grad” is only avaliable in “weight_origi”, and those pruned weight is still updating in “weight_origi”. Is there any way to retrain the model as if the pruned weights do not even exist(i.e. not participate in forward inference and backward update)?</p>
<p>Best regards</p>

albanD: <p>Yes, you are still computing the gradients and updating the original weights. The only difference is that they are masked before being used.</p>
Torch 1.8.0 for aarch64

leeyann: <p>Hello everyone,<br>
I’m looking for torch 1.8.0 (e.g. torch-1.8.0.dev20201126) and torchvision 0.9.0.dev20201126 for cp37-cp37m-linux_aarch64. They were available on the nightly download, but have been replaced at some point.<br>
With the newer version or torch I get a RuntimrError: module compiled against API version 0xe but this version or numpy is 0xd. I tried several versions of numpy, but the error persists. Only torch 1.8.0 was working.<br>
I would prefer the .whl files.<br>
Thanks !<br>
PS: Is there any stable aarch64 version of torch and torchvision ?</p>

tom: <p>I made packages for Raspberry Pi OS (Buster) for a training course in January:<br>
<a href="http://mathinf.com/pytorch/arm64/2021-01/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://mathinf.com/pytorch/arm64/2021-01/</a><br>
The also contain a somewhat dubious fix to use NNPack for convolutions with batch size one because falling back to THNN convolutions is painfully slow / memory-intensive.</p>
<p>I also have a very recent whl for the Jetson (with CUDA) but didn’t push it to the server yet.</p>
<p>Given that the checkouts are relatively fresh and I needed the additional patch, I didn’t build the 1.8 from the release yet.</p>
<p>Best regards</p>
<p>Thomas</p>

leeyann: <p>Dear Thomas,<br>
Thank you very much for your answer. Your torch 1.8.0a0 works perfectly! Not only is gone the RuntimrError, but the output images look as they should (with torch 1.8.0.dev20201126 I was getting periodic vertical strips, which was weird).<br>
Thanks again. Great job!<br>
L.</p>
Basic question to source code optim.optimizer

PhysicsIsFun: <p>Hi all,</p>
<p>I am trying to understand the source code of torch.optim.sgd (<a href="https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD" rel="noopener nofollow ugc">link to source</a>). As it inherits several features from the torch.optim.optimizer class (<a href="https://pytorch.org/docs/stable/_modules/torch/optim/optimizer.html#Optimizer" rel="noopener nofollow ugc">link to source</a>), I am also taking glances at that. For the beginning, I have two questions:</p>
<ol>
<li>
<p>In the constructor of optimizer, it says <code>self.state = defaultdict(dict)</code>. As far as I know, <code>dict</code> has to be a function that is called when trying to access a key in the state dictionary that is not present. Where is the <code>dict</code> function defined?</p>
</li>
<li>
<p>The method in the SGD source code:</p>
</li>
</ol>
<pre><code class="lang-auto">    def __setstate__(self, state):
        super(SGD, self).__setstate__(state)
        for group in self.param_groups:
            group.setdefault('nesterov', False)
</code></pre>
<p>It calls <code>__setstate__</code> from the optimizer parent which simply does <code>self.__dict__.update(state)</code>. So I assume this is to load an optimizer with a previous setting? Then what is the purpose of setting all the ‘nesterov’ arguments to False?</p>
<p>Best,<br>
PiF</p>

ptrblck: <ol>
<li>
<code>defaultdict</code> accepts the <code>default_factory</code> argument, which is the Python <code>dict</code> class in this case. I’m not 100% sure, but think this was added to return an empty <code>dict</code> for missing keys:</li>
</ol>
<pre><code class="lang-python">state = defaultdict()
print(state)
&gt; defaultdict(None, {})
state['lala']
&gt; KeyError: 'lala'

state = defaultdict(dict)
print(state)
&gt; defaultdict(&lt;class 'dict'&gt;, {})
state['lala']
&gt; {}
</code></pre>
<ol start="2">
<li>I don’t know exactly why this is needed. I assume thr <code>__getstate__</code> method of the parent class is used and since <code>SGD</code> has additional parameters, they would need to be reset to the defaults in the corresponding <code>__setstate__</code>.</li>
</ol>

PhysicsIsFun: <p>Thanks ptrblck!</p>
<ol>
<li>
<p>Makes sense!</p>
</li>
<li>
<p>Right, since not all optimizers have “nestorov” argument but it is evaluated in the beginning of SGD step( ), the <strong>setstate</strong> should ensure that the argument “nestorov” is set. However, what about the other parameters like “dampening”? It is also evaluated at the beginning of step( ), but not set in the <strong>setstate</strong> function. This would imply that all optimizers definitely have this argument…</p>
</li>
</ol>
How to get output of model as input to other model in pytorch?

saisriker_reddy: <p>I have split my model into model A and model B respectively. But while inferencing when I give my model A output as input to model B I got the below error:<br>
“output = input.matmul(weight.t())<br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2400x75 and 180000x6)”<br>
My working code:<br>
class ConvNet(nn.Module):</p>
<pre><code>def __init__(self,num_classes=6):
    super(ConvNet,self).__init__()
    self.conv1=nn.Conv2d(in_channels=3,out_channels=12,kernel_size=3,stride=1,padding=1)
    self.bn1=nn.BatchNorm2d(num_features=12)
    self.relu1=nn.ReLU()
    self.pool=nn.MaxPool2d(kernel_size=2)
    self.conv2=nn.Conv2d(in_channels=12,out_channels=20,kernel_size=3,stride=1,padding=1)
    self.relu2=nn.ReLU()
    self.conv3=nn.Conv2d(in_channels=20,out_channels=32,kernel_size=3,stride=1,padding=1)
    self.bn3=nn.BatchNorm2d(num_features=32)
    self.relu3=nn.ReLU()
    self.fc=nn.Linear(in_features=75 * 75 * 32,out_features=num_classes)
def forward(self,input):
    output=self.conv1(input)
    output=self.bn1(output)
    output=self.relu1(output)
    output=self.pool(output)
    output=self.conv2(output)
    output=self.relu2(output)
    output=self.conv3(output)
    output=self.bn3(output)
    output=self.relu3(output)
    output=self.fc(output)
    return output
</code></pre>
<p>model_path=‘model2.pt’<br>
model=ConvNet()<br>
model.load_state_dict(torch.load(model_path))<br>
model.eval()<br>
print(model)<br>
new_model1 = nn.Sequential(<em>(list(model.children())[:3]))<br>
print(new_model1)<br>
new_model2 = nn.Sequential(</em>(list(model.children())[3:]))<br>
print(new_model2)<br>
<span class="hashtag">#inferecing</span> models<br>
image_path = ‘cat.4672.jpg’<br>
def predict_image(image_path):<br>
print(“Prediction in progress”)<br>
image = Image.open(image_path)<br>
transformer=transforms.Compose(<br>
transforms.Resize((150,150)),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),  <span class="hashtag">#0-255</span> to 0-1, numpy to tensors<br>
transforms.Normalize([0.5,0.5,0.5], # 0-1 to [-1,1] , formula (x-mean)/std<br>
[0.5,0.5,0.5])<br>
])<br>
# Preprocess the image<br>
image_tensor = transformer(image).float()<br>
# Add an extra batch dimension since pytorch treats all images as batches<br>
image_tensor = image_tensor.unsqueeze_(0)<br>
if torch.cuda.is_available():<br>
image_tensor.cuda()<br>
new_model1.eval()<br>
input = Variable(image_tensor)<br>
output1 = new_model1(input)<br>
new_model2.eval()<br>
output2 = new_model2(output1)<br>
index = output2.data.numpy().argmax()<br>
return output1.size()<br>
print(predict_image(image_path))</p>

AladdinPerzon: <p>Seems like the shapes don’t match. What is the output shape from modelA and what should the input shape be to modelB? One thing you can do to debug is make print statements in the forward part and make sure each shape is what you expect it to be</p>

saisriker_reddy: <p>How can I get output of specific layer and give it as input to another model in pytorch?</p>
Kernel dies when I use a tensor of size greater than 127 in a linear layer (on Mac M1)

Utkarsh: <p>I wrote a simple example to show the issue below. As soon as the nn.Linear layer receives an input of size greater than 127 say, 128, my kernel restarts. But for smaller values it works fine.<br>
My pytorch version is 1.7.1 and I am working on the Mac M1.</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super().__init__()
        # This makes the kernel restart
        self.fc1 = nn.Linear(128,1)  #anything less than this works fine
        
    def forward(self, x):
        x = self.fc1(x)
        return x

model = Network()

print(model.forward(torch.rand((1,128))))
</code></pre>

ptrblck: <p>Could you execute the script in a terminal and check, if you get a valid error message (I assume you are using a Jupyter Notebook at the moment).<br>
If you get an error message, could you post it here, please?</p>

Utkarsh: <p>I get the following error:</p>
<blockquote>
<p>OMP: Error <span class="hashtag-raw">#15:</span> Initializing libiomp5.dylib, but found libomp.dylib already initialized.</p>
</blockquote>
<blockquote>
<p>OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see <a href="http://www.intel.com/software/products/support/" class="inline-onebox" rel="noopener nofollow ugc">Intel® Product Support</a>.</p>
</blockquote>

Utkarsh: <p>I was able to solve this by installing:</p>
<blockquote>
<p>conda install nomkl</p>
</blockquote>
I want to load my own data instead of mnist data

creatives07: <p>i want to load my own data instead of mnist data. How do I fix the code?</p>
<p>os.makedirs("…/…/data/mnist", exist_ok=True)<br>
dataloader = torch.utils.data.DataLoader(<br>
datasets.MNIST(<br>
“…/…/data/mnist”,<br>
train=True,<br>
download=True,<br>
transform=transforms.Compose(<br>
[transforms.Resize(opt.img_size), transforms.ToTensor(), transforms.Normalize([0.5], [0.5])]<br>
),<br>
),<br>
batch_size=opt.batch_size,<br>
shuffle=True,<br>
)</p>

ptrblck: <p><code>MNIST</code> expects a specific format, so you would need to store your data in the same format and name accordingly.<br>
The usual approach would be to e.g. use <code>ImageFolder</code>, if you are loading single image files, or a custom <code>Dataset</code>.<br>
What kind of data do you have?</p>

creatives07: <p>I converted my dataset to ‘idx3 uyte’ which is the same format as the mnist dataset. But I don’t know to how the data load. Thank you for your answer.</p>

ptrblck: <p>What kind of data do you have? Is it a binary blob?<br>
How large is each sample?</p>

creatives07: <p>My data is grayscale x-ray image data. Train data is 12920 KB, test data is 685 KB, train label data is 3 KB and test label data is 1 KB.</p>

ptrblck: <p>Did you store the x-ray images in some kind of binary data or do you have each image stored separately?<br>
In the latter case, have a look at the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">Data Loading Tutorial</a> to see how to create a custom <code>Dataset</code>.</p>

creatives07: <p>I converted data from this url. (<a href="https://github.com/Arlen0615/Convert-own-data-to-MNIST-format/blob/master/convert_to_mnist_format.py" rel="nofollow noopener">https://github.com/Arlen0615/Convert-own-data-to-MNIST-format/blob/master/convert_to_mnist_format.py</a>)<br>
Is this the case for the latter?</p>

saru_sukh: <p>What if I want to upload my multidimensional data (1335x11 and .csv format) instead of MNSIT data?<br>
How to approach with that?</p>
Is there a way to get grad_output for the linear layers of a cudnn GRU?

francisr: <p>Hi,</p>
<p>I’m trying to implement KFAC ( <a href="https://github.com/pytorch/pytorch/issues/35801" class="inline-onebox" rel="noopener nofollow ugc">Add KFAC optimizer · Issue #35801 · pytorch/pytorch · GitHub</a> ) for GRUs, and I need to be access what would be <code>grad_output</code> for the linear layers in the GRU.</p>
<p>However since GRUs are implemented with cudnn (I could use a pytorch impelementation but cudnn is much faster) as a single unit, I’m not sure how to access that. I had a look at the C++ code of cudnn to see where I could hack it, but I’m not familiar with it and find it a bit obscure.</p>
<p>Does anyone have a suggestion?</p>
How to obtain 2d 68 landmark of rendered image from 3d shape using Pytorch 3D?

m-pektas: <p>I am working on Basel Face Model with pytorch 3D. I can render face but I need 2d landmarks of rendered image.  For example, I know 3d coordinates of 68 landmarks. I want to draw these points on 2d rendered image. How can I do this ?</p>

nuhpiskin: <p>I think <a href="https://pytorch3d.readthedocs.io/en/latest/overview.html#tutorials" rel="noopener nofollow ugc">this</a> tutorials<br>
That can help you.</p>

m-pektas: <p>I found the answer of this question. We need use <strong><code>transform_points</code></strong> and <strong><code>transform_points_screen</code></strong> functions of <strong><code>CamerasBase</code></strong> object.</p>
Tensor creation and destruction within a while loop in libtorch C++

AbaqusNastan: <p>I have just started with libtorch and I am having some trouble with while loops and tensors <img src="https://discuss.pytorch.org/images/emoji/apple/roll_eyes.png?v=9" title=":roll_eyes:" class="emoji" alt=":roll_eyes:"><br>
So, my main func looks like so:</p>
<pre><code class="lang-auto">int main()
{

  auto tensor_create_options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCPU).requires_grad(false);
  torch::Tensor randn_tensor = torch::randn({10}, tensor_create_options);
  int randn_tensor_size = randn_tensor.sizes()[0];

  while (randn_tensor_size &gt; 5)
  {
    std::cout &lt;&lt; "==&gt; randn_tensor shape: " &lt;&lt; randn_tensor.sizes() &lt;&lt; std::endl;
    randn_tensor.reset(); //reset();
    torch::Tensor randn_tensor = torch::randn({3}, tensor_create_options);
    std::cout &lt;&lt; "==&gt; randn_tensor shape 3: " &lt;&lt; randn_tensor.sizes() &lt;&lt; std::endl;
randn_tensor_size--;
  }

  return 0;
}
</code></pre>
<p>and I get thrown this:</p>
<pre><code class="lang-auto">==&gt; randn_tensor shape: [10]
==&gt; randn_tensor shape 3: [3]
terminate called after throwing an instance of 'c10::Error'
  what():  sizes() called on undefined Tensor
</code></pre>
<p>Essentially, what I want to do is recreate the tensor within the while loop and ensure that I can access it again in the whileloop.</p>
<p>Interestingly, it seems to have cerated the tensor of reduced size but the while loop does not seem to recognise this.</p>
<p>Thank you!</p>

AbaqusNastan: <p>turns out I was shadowing! deleted the shadow and it is fine. Thank you!</p>
How to implement a custom init scheme?

romain: <p>Hello, I’d like to know how to properly implement a custom initialization scheme. For the moment, I’m doing it as below, but from PRs I gather that using the <code>.data</code> property isn’t recommended.</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self, hidden_dim: int):
        super(MyModule, self).__init__()
        self.hidden_dim = hidden_dim
        self.my_param = nn.Parameter(torch.zeros(hidden_dim))
        self.reset_parameters()

    def reset_parameters(self) -&gt; None:
        u = torch.rand(self.hidden_dim) * (1 - 2 / self.hidden_dim) + 1 / self.hidden_dim
        self.my_param.data = -(1 / u - 1).log()
</code></pre>
<p>As you can see the scheme is a bit complex (it generates a uniform distribution over the inverse of the sigmoid function) so there are no built-in that can directy be used to modify the tensor in place, e.g. via this  for the uniform function:</p>
<pre><code class="lang-auto">with torch.no_grad():
    my_tensor.uniform_(a, b)
</code></pre>
<p>Could you tell me how to do it ?</p>

ptrblck: <p>You could wrap the code into the <code>no_grad()</code> guard and use the <code>.copy_</code> operation to fill the parameter. If you are initializing the plain tensor before wrapping it into an <code>nn.Parameter</code>, you could also skip the <code>no_grad()</code> guard.</p>

romain: <p>I see, so since I want the <code>reset_parameters</code> to be callable even if the module already exists I should simply do:</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.my_param.copy_(-(1 / u - 1).log())
</code></pre>
<p>Great thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to apply CrossEntropy with top-label K?

oasjd7: <pre><code class="lang-auto">out = classifier(model(input))
_, topk_label = torch.topk(out, k=2, dim=-1)
topk_label = topk_label.squeeze()
top1_label, top2_label = topk_label.t()[0], topk_label.t()[1]

ce = nn.CrossEntropyLoss()
loss = ce(top1_label, gt_labels)
loss = ce(top2_label, gt_labels)
...

</code></pre>
<p>As far as I know, <code> nn.CrossEntropyLoss</code> don’t allow the predictions (1st parameter) with one-hot prediction.<br>
How can I apply cross entropy loss with one-hot predictions (top1_label or top2_label) ?</p>
Autoencoder with variable size of inputs

Adeleh: <p>Hi, I try to embed nodes of a graph, by using an Autoencoder with linear layers. But the graph is dynamic so number of its nodes may be changed at each time step.<br>
How can I add some additional nodes and corresponded weights to the “input layer” of AE (at time t), to use it for next time step (t+1)?  I don’t want to change the hidden layers and other weights ,…</p>