akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
"RuntimeError: size mismatch, got 10, 10x10,1"

username: ganesh1102
cooked: <p>This is probably a very basic question, but here I go anyway.</p>
<p>I’m trying to create a DeepNet of a series of Linear layers. Here is a class called ‘NeuralNet’:</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):
    

        
    def __init__(self, weights, bias):
        
        super(NeuralNet, self).__init__()
        
        self.weights = weights
        self.bias = bias
        
        self.nn = nn.Sequential(
            nn.Linear(10, 10),
            nn.ReLU(inplace = True),
            nn.Linear(10, 10),
            nn.ReLU(inplace = True),
            nn.Linear(10, 10),
            nn.ReLU(inplace = True),
            nn.Linear(10, 3),
            nn.ReLU(inplace = True),
        )



    def forward(self, a, b, c):
        
        a = torch.flatten(a)
        b = torch.flatten(b)
        c = torch.flatten(c)
        
        print(a.shape, b.shape, c.shape)
        
        a1 = self.nn(a)
        b1 = self.nn(b)
        c1 = self.nn(c)
        
        a1 = self.nn(s1)
        b1 = self.nn(b1)
        c1 = self.nn(ep1)
        
        a1 = self.nn(a1)
        b1 = self.nn(b1)
        c1 = self.nn(c1)
        
        a1 = self.nn(a1)
        b1 = self.nn(b1)
        c1 = self.nn(c1)
        
        y = torch.cat((a1, b1, c1))

        return y


</code></pre>
<p>The randomized tensors a, b, c all have a shape of (10, 1).</p>

username: ganesh1102
cooked: <p>I’ve figured the issue out. It turns out that you have to call the neural net just once in the method <code>forward</code>. Sorry for the trouble.</p>
<p>Thanks.</p>
Extracting each neuron as variable from any layer in Neural network model

username: Hritik_Gopal_Shah
cooked: <p>re: Can we extract each neuron as variable in any layer of NN model, and apply optimization constriants in each neuron?</p>
<p>But I want to add this constraints as layer inside the NN model. So need some functions in pytorch that can give me acces to each neurons as variables. Is it possible?</p>

username: AlphaBetaGamma96
cooked: <p>When you say ‘neurons’, do you mean the parameters of a model or output of a layer within a model?</p>
<p>If you want to get the parameters of a model you can do so via,</p>
<pre><code class="lang-auto">net = model()
for name, param in net.named_parameters():
  print(name, param) #print name and weights of model
</code></pre>
<p>For the output of a layer you can use a <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_pre_hook.html" rel="noopener nofollow ugc">forward-pre hook</a> to get the intermediate activation of a model, and just attached that to all <code>nn.Modules</code> via,</p>
<pre><code class="lang-auto">net = model()
for name, module in net.named_modules():
  register_module_forward_pre_hook(module)
  print("registered hook on: ", name)
</code></pre>
<p>If this is purely a optimisation related issue, then in my opinion you’re better off doing all of this within the optimizer directly. In a similar way to how weight decay is implemented in optimizers.</p>

username: Hritik_Gopal_Shah
cooked: <p>I want to apply convex optimization on output of the linear layer and add this as an cutsom layer after linear layer on the same NN model</p>

username: AlphaBetaGamma96
cooked: <p>If that’s the case then you’ll want to use a <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=forward%20hook#torch.nn.Module.register_forward_hook" rel="noopener nofollow ugc">forward hook</a> (not forward pre hook) to cache the output of a linear layer. You can use the same code as above but just change from a forward pre hook to forward hook.</p>
<p>Then you can cache the intermediate values within a dictionary (using the layer as a key) and perform constrained optimization in the way you need.</p>
Mutation of the array and autograd

username: Jakub_Mitura
cooked: <p>Hello I am creating custom loss function, on of the component is modifying the values of the data array.<br>
so for example function f mutates A into B both A and B are the same shape and data type</p>
<pre><code class="lang-auto">f(A)=B
</code></pre>
<p>in order to keep backpropagation working I need to supply new array as the output so for example</p>
<pre><code class="lang-auto">B= zeros()
f(A,B)
return B
</code></pre>
<p>so I will not mutate A</p>
<p>or mutation is allowed?</p>

username: ptrblck
cooked: <p>I assume “mutation” refers to inplace operations, which are directly manipulating the data of the tensor, e.g. as seen here:</p>
<pre><code class="lang-python">a = torch.zeros(1)
print(a)
# tensor([0.])

b = a
print(b)
# tensor([0.])

b.add_(1.)
print(a)
# tensor([1.])
print(b)
# tensor([1.])
</code></pre>
<p>If so, then it would depend on the used operations and if these values are needed for the gradient calculation. PyTorch will raise a <code>RuntimeError</code> is disallowed inplace operations are detected and will fail during the <code>backward</code> pass.</p>

username: Jakub_Mitura
cooked: <p>ok, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  ! If it will just give error I can just experiment on this - fantastic<br>
One more thing as this mutation function acts element wise on 3D array - is there supported way to apply the function to each element of the tensor - with access to its cartesian coordinate in the array<br>
pseudo code below</p>
<pre><code class="lang-auto">arr= torch.rand(100,100,100)

def add_neighbour(arr,point):
     return arr[point.x+1,point.y,point.z]+arr[point.x,point.y+1,point.z]+arr[point.x,point.y,point.z+1]

apply_on_coord( add_neighbour  ,arr)

</code></pre>

username: ptrblck
cooked: <p>Your indexing approach of:</p>
<pre><code class="lang-python">x[index] = x[index] + b
</code></pre>
<p>would also manipulate <code>x</code> inplace and will also raise an error if it’s disallowed.</p>
Is it possible to re-train a finetuned NER model on a dataset with a different tagset (respect from the first training dataset)?

username: amen
cooked: <p>Hi all, first time here and very new to NLP.</p>
<p>I used PyTorch to finetune XLM-RoBERTa with a german dataset for NER that has 7 tags. Let’s call this model <em>xlmr-finetuned</em>.</p>
<p>Now I have another german NER dataset that comes from historical newspapers (so it contains many errors), and I would like to finetune or re-train the last model (<em>xlmr-finetuned</em>) on this dataset. The problem is that this historical dataset has 11 tags (the same 7 tags as the first + 4 new tags). So, when trying to load the model with <em>.from_pretrained</em> I get a size mismatch error about the sizes of the model (<em>xlmr-finetuned</em>) and the configuration of the new dataset I want to finetuneit on:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for XLMRobertaForTokenClassification:
size mismatch for classifier.weight: copying a param with shape torch.Size([7, 768]) from checkpoint, the shape in current model is torch.Size([11, 768]).
size mismatch for classifier.bias: copying a param with shape torch.Size([7]) from checkpoint, the shape in current model is torch.Size([11]).
</code></pre>
<p>So, my question is, is it possible to finetune, with a different tagset, an already finetuned model? Or, on the contrary, transfer learning can only be done between very similar datasets?</p>
<p>I’m already considering solutions that involve modifying the datasets, like removing the 4 new tags in the new dataset, or telling the model in the first finetuning that there are 11 tags instead of 7 (even if there are not).</p>
<p>Thanks a lot in advance!</p>
Why there's a tiny difference between outputs of PyTorch and LibTorch?

username: LvJC
cooked: <p>This is my PyTorch result of the first 10 value.<br>
<code>tensor([-1.3709, -0.8062, 1.9198, 0.4937, -2.9625, 1.2369, -0.3699, -0.1852, 4.9301, 4.0436], grad_fn=&lt;SliceBackward&gt;)</code></p>
<p>This is my LibTorch(C++) result of the fisrt 10 value.<br>
<code>-1.3711 -0.8062 1.9193 0.4921 -2.9617 1.2358 -0.3710 -0.1846 4.9292 4.0416 [ Variable[CUDAFloatType]{1,10} ]</code></p>
<p>I test between two realization of my codes. And the only difference is the <strong>LANGUAGE I USE</strong>. The direct input of the model is the same!</p>
<p>Can someone explain that?</p>

username: smth
cooked: <p>it might just be the printing, C++ might be printing at a different precision than Python-side.</p>

username: LvJC
cooked: <p>So you mean that the result value should be absolutely same?<br>
But as you say, the difference is too large for printing ???</p>

username: smth
cooked: <p>I’m saying it’s probably the same according to float precision limits (so same to the order of 1e-5) – but printing formatting in python and C++ might differ – check that further.</p>

username: Keunhoi_An
cooked: <p>I’m suffering the same problem, and I think it is not just because of float precision limits…<br>
I’ve checked almost whole middle-result tensor value (size is about [20, 512, 100, 100]), and I’ve found that only some part of tensor is slightly different from python result (and it is still too big to be caused by float precision…), and most part of tensor has a huge difference…</p>
RuntimeError: expected scalar type Half but found Float from fc layers in TorchScript

username: SeanX
cooked: <p>Hi, I am getting a runtime error when using two fc layers for mixed precision training.</p>
<p>Environment: pytorch 1.9.0 + CUDA 11.4<br>
Code to reproduce the error:</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

inputs = torch.zeros([6, 152, 128, 480], dtype=torch.float32).cuda()
targets = torch.ones([6, 2], dtype=torch.float32).cuda()

class Net(torch.jit.ScriptModule):
    def __init__(self):
        super().__init__()
        self.net = torch.nn.Sequential(
            torch.nn.AdaptiveAvgPool2d(output_size=(1, 1)),
            torch.nn.Flatten(1),
            torch.nn.Linear(152, 32),
            torch.nn.Linear(32, 2),
        )

    @torch.jit.script_method
    def forward(self, x):
        return self.net(x)

net = Net().cuda()
loss_fn = torch.nn.MSELoss()
optimzer = torch.optim.Adam(net.parameters(), 0.001)
scaler = torch.cuda.amp.GradScaler()

for i in range(2):
    optimzer.zero_grad()
    with torch.cuda.amp.autocast():
        outputs = net(inputs)
        loss = loss_fn(outputs, targets)
    scaler.scale(loss).backward()
    scaler.step(optimzer)
    scaler.update()

</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "&lt;string&gt;", line 75, in &lt;backward op&gt;

                weight_size = weight.size()
                grad_input = torch.matmul(grad_output, weight)
                             ~~~~~~~~~~~~ &lt;--- HERE
                grad_weight = torch.matmul(grad_output.reshape(-1, weight_size[0]).t(), input.reshape(-1, weight_size[1]))
                # Note: calling unchecked_unwrap_optional is only safe, when we
RuntimeError: expected scalar type Half but found Float
</code></pre>
<p>The error happens when adding two fc layers and jitting the layers . No error when using single fc layer. Is there any way to work this around for now? Thank you!</p>

username: ptrblck
cooked: <p>Scripting and mixed-precision training had some issues in the past, so you could update to the latest nightly or stable release and it should work.<br>
I just verified it in a current <code>1.12.0+cu116</code> release and don’t see any issues.</p>

username: SeanX
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your timely reply!</p>
How do I convert a torch array into an array of arrays?

username: desert_ranger
cooked: <p>I’d like to do torch.gather with the following code -</p>
<pre><code class="lang-auto">source = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10,11,12]])

index = torch.tensor([[0], [2], [0]])
# index = torch.tensor([0,2,0])

source.gather(dim=1, index=index)
</code></pre>
<p>to get the following output -</p>
<pre><code class="lang-auto">tensor([[1],
        [6],
        [7]])
</code></pre>
<p>However, my index array has this form -</p>
<p><code>index = torch.tensor([0,2,0])</code></p>
<p>Does, there exist a trivial way to convert torch.tensor([0,2,0]) to torch.tensor([[0], [2], [0]])?</p>

username: desert_ranger
cooked: <p>It can be done by doing the following -</p>
<p>index.reshape(-1,1)</p>
Armijo step size

username: Jeet
cooked: <p>How to implement armijo step size in a neural network developed by Pytorch? Currently the neural network is using optim.Adam as the step size.</p>

username: Aayush_Garg
cooked: <aside class="quote no-group" data-username="Jeet" data-post="1" data-topic="158221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jeet/48/47500_2.png" class="avatar"> Jeet:</div>
<blockquote>
<p>armijo step size</p>
</blockquote>
</aside>
<p>There is a nice github repo (<a href="https://github.com/hjmshi/PyTorch-LBFGS" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hjmshi/PyTorch-LBFGS: A PyTorch implementation of L-BFGS.</a>) that implements L-BFGS optimization method for Pytorch and allows for different kinds of line search, for example, arimijo.<br>
There they have the <a href="https://github.com/hjmshi/PyTorch-LBFGS/blob/master/examples/Neural_Networks/full_overlap_armijo_lbfgs_example.py" rel="noopener nofollow ugc">armijo example</a>.<br>
Please refere to that.</p>
Padding in Causal Convs only for seq2seq prediction?

username: jayz
cooked: <p>Hello,</p>
<p>I am trying to predict the value <code>x[t+1]</code> from a time series <code>x[t-L:t]</code> with a stack of causal convolutions, where L is the input sequence length.</p>
<p>My input dataset X (capital X) has two dimensions, i.e. shape CxT:</p>
<ul>
<li>C is the number of features / channels</li>
<li>T is the time dimension.</li>
</ul>
<p>Then, an input from time t would be <code>x_t = X[:,t-L:t]</code>, where L is the input sequence length.<br>
We stack B such inputs <code>x = {x_t | t drawn B times randomly from range[L,T]}</code> into a batch of shape B x C x L, where B is the batch size.</p>
<p>A stack of Causal Convs (with <code>kernel_size=2</code>) will look like this:</p>
<pre><code class="lang-python">class CausalConvStack(nn.Module): 

    def __init__(self,C,L): 
        super().__init__()
        stack = []
        for t in range(L-1):
             stack.append(nn.Conv1d(C,C,kernel_size=2))
        self.stack = nn.ModuleList(stack)
        self.final_conv = nn.Conv1d(C,1,kernel_size=1)
        return 
</code></pre>
<p>If no left-padding is applied, the input x’s sequence length will be shortened by <code>kernel_size - 1 = 1</code> by each causal layer, resulting in a final output sequence length 1, according to the stack defined above:</p>
<pre><code class="lang-python">    def forward(self,x): 
        # x: B C L
        for layer in self.stack: 
            x = layer(x)
       x = self.final_conv(x) 
       # x: B 1 1 
       return x
</code></pre>
<p>If regressing the output of forward (which takes input <code>x[t-L:t]</code> and returns an output with shape <code>B 1 1</code>) on <code>x[t+1]</code>, this should be a valid causal approach to predict the next value of a time series, right?</p>
<p>I often read that people apply padding, in order to keep the input sequence length constant. Thus, the output sequence length will be the same as the input sequence length. My question: padding is only required if one wants to predict an entire sequence of the same length from the input, i.e. seq2seq, right? E.g. we want to either predict the values <code>y[t-L:t]</code> of another time series from the input time series <code>x[t-L:t]</code>, or we maybe even want to predict T future values of x, i.e. <code>x[t+1:t+L+1]</code> from <code>x[t-L:t]</code>.</p>
<p>Thanks!</p>
<p>Best, JZ</p>
How to compute pairwise distance between two torch tensors of different dimensions?

username: MahdiNazemi
cooked: <p>I’m looking for a method that takes an <em>m</em>-by-<em>d</em> tensor (<code>x1</code>) and an <em>n</em>-by-<em>d</em> tensor (<code>x2</code>) and computes the pairwise distance between each element of <code>x2</code> with each elements of <code>x1</code>.</p>
<p>I’ve seen <code>nn.PairwiseDistance</code> in the documentation, which requires the tensors to have the same dimensions and only computes the distance between their corresponding rows.<br>
An inefficient solution would be to use <code>nn.PairwiseDistance</code> and iterate over different rows of <code>x2</code> to compare each row of <code>x2</code> with all rows of <code>x1</code>.</p>

username: Foivos_Diakogiannis
cooked: <p>Hi,</p>
<p>take a look at the package torch-two-sample, and in particular the <a href="https://github.com/josipd/torch-two-sample/blob/master/torch_two_sample/util.py" rel="nofollow noopener">util.py</a> file. Hope this helps.</p>

username: Mohammad_Hassan_Soha
cooked: <p>Is <code>torch.cdist</code> what you’re looking for?</p>
Using tabular data in addition of image data in efficientnet

username: maryma
cooked: <p>Hi I have some MRI images from kidneys. I am using effiecientnet for image classification to classify images in one of the stage 0, stage1 or stage 2 cancer classes. Because I didnt get good results just from image classification I wanted to use one column of clinical data that has tumor size alongside the images. Because I am not proficient in pytorch I faced this question that how can I merge one column of clinical data with images? Because I am using effiecientnet I can not use two inputs to the model how can I give tumor sizes to efficientnet? Should I change the architecture of efficientnet? Do you have another idea for this problem?</p>
Custom DataLoader for .mat file

username: Nosin_Ibna_Mahbub
cooked: <p>Hello, I am trying to segment medical image and i need help on creating a DataLoader to take into a CNN .mat images. My main image and mask are saved in a same mat file. So, I need help to create custom dataloader to read the main image and mask from the same mat file.</p>
<p>Thank you</p>

username: JuanFMontesinos
cooked: <p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html</a></p>
Can memory be distributed among GPUs?

username: Chanu_hcu
cooked: <p>Consider the memory usage of 4 GPUs while training my models using <code>nn.DataParallel</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1066bfd0c80e140cf25a99a0c158476844d8954.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1066bfd0c80e140cf25a99a0c158476844d8954" title="Screenshot from 2022-08-03 13-06-47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1066bfd0c80e140cf25a99a0c158476844d8954_2_690x133.png" alt="Screenshot from 2022-08-03 13-06-47" data-base62-sha1="tP7y6jOqLlPbhODWvrUe6tbNa0A" width="690" height="133" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1066bfd0c80e140cf25a99a0c158476844d8954_2_690x133.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d1066bfd0c80e140cf25a99a0c158476844d8954_2_1035x199.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1066bfd0c80e140cf25a99a0c158476844d8954.png 2x" data-dominant-color="5F2C37"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-08-03 13-06-47</span><span class="informations">1118×217 8.66 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can see that <code>cuda:0</code> generally acts as the master node and needs more memory. Is there any way to distribute memory uniformly among all the GPUs?</p>

username: ptrblck
cooked: <p>That’s a known limitation of <code>nn.DataParallel</code> which is one reason why we recommend to use <code>DistributedDataParallel</code> besides the better performance of the latter approach.</p>
Could not run 'aten::q_scale' with arguments from the 'CUDA' backend

username: viash
cooked: <p>Could not run ‘aten::q_scale’ with arguments from the ‘CUDA’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::q_scale’ is only available for these backends: [QuantizedCPU, QuantizedCUDA, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].</p>
<p>I get the same error when I try for the CPU .<br>
I even try to detach and reassign to cuda after the operation and it still fails.</p>
<p>What I am doing seems to be a call to the C function which doesnt seem to be binded with python.</p>
<p>can we have an alternative way to extract tensor or a fix to this ?</p>

username: jerryzh168
cooked: <p>this is because you are calling q_scale on a non-quantized Tensor I think, can you show us the code?</p>

username: viash
cooked: <pre><code class="lang-auto"># custom observed module, provided by user
class CustomObserverModule(torch.nn.Module):
    ...
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        input_scale = x.q_scale()
        input_scale = x.q_zero_point()
        input = x.int_repr()
        
    ...
</code></pre>

username: jerryzh168
cooked: <p>are you passing in a non-quantized Tensor to the forward function?</p>

username: viash
cooked: <p>As an input to the whole model I am passing the Input as Regular Tensor. Although, The Quant Stub Should be able to quantize the inputs and pass them right ?</p>
<p>I am adding the QuantWrapper to my model and then pass it to Prepare stage .</p>
<p>Somehow I am seeing that the input to the function is still a tensor.</p>

username: HDCharles
cooked: <p>quant stub gets replaced by a quantize op during convert, observers take in non-quantized tensors and analyze them, by calling x.q_scale() on the input in your observer, you are applying x.q_scale() on a normal tensor which is causing the error.</p>

username: viash
cooked: <p>Agreed !</p>
<p>What I was doing was is creating a custom observer module and I was looking for to see If I could pass in a QTensor.</p>
<p>Current Model in the Observer passes the regular tensor while the weights and output are quantized and dequantized.</p>
<p>It is only in the inference stage (after convert) that we see input to be the Qtensor.</p>
<p>This fails to take into account the combined scale (I_scale * W_Scale / O_Scale) 32 bit approximation while Training in QAT .</p>
<p>if there is a way to pass QTensor in the Custom Observers during Training in QAT that would be greatly appreciated.</p>
Knowledge Distillation with sigmoid activate function

username: 22ema
cooked: <p>Hello everyone.<br>
I ask some question…</p>
<p>How can i use knowledge distillation with sigmoid activate function</p>
<p>I want use knowledge distillation in multi-label classification (ex. 1, 0, 1, 1, 0). So i use sigmoid activate function with BCEloss. but almost knowledge distillation is constructed with softmax and KLdiv.</p>
<p>I think it would be possible if I convert it to the code below, but I don’t know if it’s correct</p>
<pre><code class="lang-auto">alpha *  nn.BCEWithLogitsLoss()(F.sigmoid(outputs/T), F.sigmoid(teacher_outputs/T)) + nn.BCEWithLogitsLoss()(outputs, labels) * (1-alpha)
</code></pre>
<p>If you have any information about the sigmoid function and the knowledge distillation technique based on BCEloss, please let me know.</p>
KeyError when using torch.utils.data.Subset for specific dataset but not cifar

username: flummery
cooked: <p>I’m getting the following error:</p>
<blockquote>
<p>KeyError: tensor(0)</p>
</blockquote>
<p>This happens after running the following for loop:</p>
<pre><code class="lang-auto">train_dataset = CovidDataset(train, transform=torchvision.transforms.ToTensor())
print(f"Original Train dataset length: {train_dataset.__len__()}")
indices = torch.arange(3000)
train_dataset = torch.utils.data.Subset(train_dataset, indices)
print(f"New Train dataset length: {train_dataset.__len__()}")

for idx, (data, img) in enumerate(train_dataset):
    print(idx)
</code></pre>
<p>However when I try it on CIFAR dataset, it works fine:</p>
<pre><code class="lang-auto"># import the required modules
import torch
import torchvision
from torchvision.datasets import CIFAR10
from collections import Counter

trainset = CIFAR10(root='./data', train=True, download=True, transform=torchvision.transforms.ToTensor())

indices = torch.arange(3000)
trainset = torch.utils.data.Subset(trainset, indices)

for idx, (data, img) in enumerate(trainset):
    print(idx)
</code></pre>
<p>Why does it work on CIFAR dataset but not on my dataset?</p>

username: flummery
cooked: <p>Full Code:</p>
<p>Link to dataset: <a href="https://www.kaggle.com/datasets/c395fb339f210700ba392d81bf200f766418238c2734e5237b5dd0b6fc724fcb/versions/1" class="inline-onebox" rel="noopener nofollow ugc">COVIDx CT | Kaggle</a></p>
<pre><code class="lang-auto">import torch
import torchvision
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import PIL
import os

# Convert .txt to .csv
TXT_FILE_PATH = r"C:\Users\ASUS\Machine Learning\data"
train = pd.read_csv(TXT_FILE_PATH + r"\train_COVIDx-CT.txt", sep=" ")
train.columns=['filename', 'label', 'xmin','ymin','xmax','ymax']
train=train.drop(['xmin', 'ymin','xmax', 'ymax'], axis=1 )

# Put file path to start of image so that it can be loaded
IMG_FILE_PATH = "C:\\Users\\ASUS\\Machine Learning\\data\\images\\"
train['filename'] = IMG_FILE_PATH + train['filename']

# Hyperparameters
BATCH_SIZE = 32
IMG_SIZE = (224,224)
learning_rate = 0.001
num_epochs = 10
in_channels = 1

# Custom Dataset
class CovidDataset(torch.utils.data.Dataset):
    def __init__(self, file_type, transform=None):
        self.file_type = file_type
        self.transform = transform
    def __getitem__(self, idx):
        img_name = self.file_type['filename'][idx]
        img = PIL.Image.open(img_name)
        label = self.file_type['label'][idx]
        if self.transform:
            img = self.transform(img)
        return img, label
    
    def __len__(self):
        return len(self.file_type)
    
train_dataset = CovidDataset(train, transform=torchvision.transforms.ToTensor())
print(f"Original Train dataset length: {train_dataset.__len__()}")
indices = torch.arange(3000)
train_dataset = torch.utils.data.Subset(train_dataset, indices)
print(f"New Train dataset length: {train_dataset.__len__()}")

for idx, (data, img) in enumerate(train_dataset):
    print(idx)
</code></pre>
Performance (Training Speed) of Autocast Bfloat16

username: StevenTsai
cooked: <p>I want to know why the following three cases give different performance (trianing speed).</p>
<p>a. module with bfloat16 parameter<br>
b. module with bfloat16 parameter + autocast(dtype=torch.bfloat16)<br>
c. module with float32 parameter + autocast(dtype=torch.bfloat16)</p>
<p>Here is the performance (training speed) for each case<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/250a1393cadb94345416ef9f3628fbd8ac470ae9.png" data-download-href="https://discuss.pytorch.org/uploads/default/250a1393cadb94345416ef9f3628fbd8ac470ae9" title="bf16 performance"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/250a1393cadb94345416ef9f3628fbd8ac470ae9_2_609x500.png" alt="bf16 performance" data-base62-sha1="5hFfONWL8t7GHGi1GpdD1vo8Alz" width="609" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/250a1393cadb94345416ef9f3628fbd8ac470ae9_2_609x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/250a1393cadb94345416ef9f3628fbd8ac470ae9_2_913x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/5/250a1393cadb94345416ef9f3628fbd8ac470ae9.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bf16 performance</span><span class="informations">1132×928 43.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using A100 with torch 1.12, cuda 11.6. In both “a” and “b”, I convert the model parameters to bfloat16 by calling module.to(torch.bfloat16).</p>
<p>It is very strange that there is a large difference betweena “a” and “c”. I think the datatypes for computation (inside autocast block) are the same.</p>

username: ptrblck
cooked: <p>I don’t know what <code>b</code> would be doing as <code>autocast</code> expects to use <code>float32</code> inputs and parameters. If you are manually calling <code>bfloat16()</code> on both then I would assume that <code>autocast</code> would be a no-op.</p>
<p>In any case, profile the code and check which operations are responsible for the difference in the times.</p>

username: StevenTsai
cooked: <p>If autocast is mainly for float32, I think the result makes sense to me. Thanks for the reply.</p>

username: ptrblck
cooked: <p><code>autocast</code> will use the <code>float32</code> inputs and parameters and will cast them to the desired <code>dtype</code> in the forward/backward for safe operations.<br>
While this could explain why a and b are close (b might add a small overhead even if it’s a no-op), it would not explain why c is the fastest mode, as it would perform the actual transformations.</p>
Issue with ConvTranspose1d on MPS

username: nicolasnierenberg
cooked: <p>ConvTranspose1d produces a different result on MPS than on the CPU.  Example follows.</p>
<p>import torch<br>
from torch import nn</p>
<p>y = torch.ones(1, 1, 2)<br>
deconv = nn.ConvTranspose1d(in_channels=1, out_channels=1, kernel_size=1, stride=2, bias=False)<br>
deconv.weight.data = torch.ones(1,1,2)<br>
gpudeconv = copy.deepcopy(deconv).to(device=‘mps’)<br>
x = deconv(y)<br>
print (x)<br>
ygpu = y.to(device=‘mps’)<br>
xgpu = gpudeconv(ygpu)<br>
print(xgpu)</p>
<p>Result:</p>
<p>tensor([[[1., 1., 1., 1.]]], grad_fn=)<br>
tensor([[[1., 2., 1., 0.]]], device=‘mps:0’, grad_fn=)</p>

username: ptrblck
cooked: <p>In case you are seeing the same issue using the latest nightly binary, could you create an issue on GitHub please, so that we could track and fix it?</p>
How to get each layer in vision::models::ResNet101

username: Jesse_Stone
cooked: <p>c++ libtorch<br>
vision::models::ResNet101 model101 = vision::models::ResNet101();<br>
how to get each layer in model101 as children and features</p>
<p>/////////////////////////////////////////////////////////////////////////////////////////////////<br>
python<br>
resnet101 = torchvision.models.resnet101(pretrained=self._pretrained)</p>
<pre><code>    # list(resnet101.children()) consists of following modules
    #   [0] = Conv2d, [1] = BatchNorm2d, [2] = ReLU,
    #   [3] = MaxPool2d, [4] = Sequential(Bottleneck...),
    #   [5] = Sequential(Bottleneck...),
    #   [6] = Sequential(Bottleneck...),
    #   [7] = Sequential(Bottleneck...),
    #   [8] = AvgPool2d, [9] = Linear
    children = list(resnet101.children())
    features = children[:-3]
    num_features_out = 1024

    hidden = children[-3]
    num_hidden_out = 2048

    for parameters in [feature.parameters() for i, feature in enumerate(features) if i &lt;= 4]:
        for parameter in parameters:
            parameter.requires_grad = False

    features = nn.Sequential(*features)
</code></pre>

username: Jesse_Stone
cooked: <p>find answer ~~~</p>
<p>//refer to:<br>
//~/pytorch/test/cpp/api/module.cpp<br>
//~/vision-0.8.2/torchvision/csrc/models/resnet.h<br>
vision::models::ResNet101 model = vision::models::ResNet101();<br>
torch::OrderedDict&lt;std::string, std::shared_ptr<a>torch::nn::Module</a>&gt; children = model-&gt;named_modules();<br>
for(size_t i=0; i&lt;children.size(); i++)<br>
std::cout &lt;&lt; children[i].key() &lt;&lt; std::endl;</p>
Replace class method of a predefined torch model

username: unimatrix007
cooked: <p>I was wondering is there any possibility to change a function of a preloaded model, in my case:</p>
<pre><code class="lang-auto">model = torchvision.models.detection.retinanet_resnet50_fpn(pretrained=False,
                                                            num_classes=len(class_labels))
</code></pre>
<ul>
<li>I have a predefined RetinaNet model with ResNet50 FPN</li>
<li>managed to update RetinaNetHead module, adding a new head to the model (custom loss and forward)</li>
<li>as far as I saw, RetinaNet(nn.Module) has a function called <a href="https://github.com/pytorch/vision/blob/6bea4ef5300858e2bae8071b2674dfead936b997/torchvision/models/detection/retinanet.py#L505" rel="noopener nofollow ugc">postprocess_detections</a>, which I should modify, to have the detections from the new head</li>
</ul>
<p>Is it possible to somehow subclass RetinaNet and overwrite that function, given the preloaded model or should I rewrite the whole class, then add FPN backbone?</p>

username: eqy
cooked: <p>If you understand the class method well, I don’t see any issue with overriding it with a subclass as that is the intended design.</p>
What is the most efficient data structure to perform operation on 2D list of tensors?

username: hyhuang00
cooked: <p>I have a 2D list of Tensors which is of the shape <code>(H, W)</code> . Each item is a tensor that is of the shape <code>(S_ij, D)</code> , in which <code>S_ij</code> is a variable that depends on the row index <code>i</code> and column index <code>j</code> in the 2D list, and it has a large range. I would like to apply some neural network <code>N_i</code> for each row <code>i</code> and then distribute each column <code>j</code> to other devices <code>D_j</code>. Since <code>S_ij</code> varies a lot, I didn’t choose to organize my data in a large, dense tensor of the shape <code>(H, W, max(S_ij), D) </code> since it will lead to a waste of memory. At this moment I have to store my data in a 2D list of tensors, and concat&amp;split tensors when applying row-wise/col-wise operations, which leads to significant overhead. Is there any other torch native data structure that can handle such a case?</p>
Detecting functional calls in forward()

username: Dan_Saunders
cooked: <p>Let’s say I have the following model:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.lin1 = nn.Linear(784, 128)
        self.lin2 = nn.Linear(128, 10)

    def forward(self, x):
        x = F.relu(self.lin1(x))
        x = self.lin2(x)
        return x
</code></pre>
<p>I want to be able to inspect the network somehow, and get the sequential operations performed by it. Using <code>.children()</code> gives the following:</p>
<pre><code class="lang-auto">network = Net()

print(list(network.children()))

Out: "[Linear(in_features=784, out_features=128, bias=True), Linear(in_features=128, out_features=10, bias=True)]"
</code></pre>
<p>How can I detect the <code>torch.nn.functional</code> calls in a <code>Module</code>'s forward pass?</p>

username: skrish13
cooked: <p>For your aim, the best bet is to have a model defined as a <code>nn.Sequential</code> and print that.</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.seq_model = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))

    def forward(self, x):
        x = self.seq_model(x)
        return x
</code></pre>

username: Dan_Saunders
cooked: <p>I’m hoping to detect calls to <code>torch.nn.functional</code>. I know I <em>can</em> use a <code>Sequential</code> module, but I’m hoping to detect functional operations in, say, pre-trained networks.</p>

username: viash
cooked: <aside class="quote no-group" data-username="Dan_Saunders" data-post="3" data-topic="28360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dan_saunders/48/6972_2.png" class="avatar"> Dan_Saunders:</div>
<blockquote>
<p>etect functional operations in</p>
</blockquote>
</aside>
<p>Did you find a solution to this ? A long  routed way would be to run a dummy input and trace all the function calls. but that is not solving my problem as I intend to replace the modules inplace.</p>
Can I use TimeStretch on Mel-frequency spectrograms?

username: miccio-dk
cooked: <p>As the title implies, I’m wondering if the time-stretching algorithm used by the TimeStretch transform can be used with Mel-frequency spectrograms. I have two choices:</p>
<ul>
<li>Spectrogram() → TimeStretch() → MelScale()</li>
<li>MelSpectrogram() → TimeStretch()</li>
</ul>
<p>Which one is valid? both? neither? thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How does torch differentiate between batch and single datasets?

username: desert_ranger
cooked: <p>Here’s my neural network.</p>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self, state_size, action_size):
        super(NeuralNetwork, self).__init__()
        self.state_size = state_size
        self.action_size = action_size
#         self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(1, 30),
            nn.ReLU(),
            nn.Linear(30, 30),
            nn.ReLU(),
            nn.Linear(30, action_size)
        )
    def forward(self, x):
        x = self.linear_relu_stack(x)
        return x
</code></pre>
<p>As you can imagine, it can compute tensor inputs of the shape torch.Size([1]). However, when I try to feed it batch data, for instance, shape torch.Size([10]) it throws the following error -</p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x10 and 1x30)</p>
<p>For instance, this code works -<br>
net = NeuralNetwork(10, 5)<br>
x1 = torch.rand(1)<br>
print(x1.shape)<br>
out = net(x1)</p>
<p>But this fails -</p>
<p>x2 = torch.rand(10)<br>
print(x2.shape)<br>
out = net(x2)</p>

username: AlphaBetaGamma96
cooked: <p>In your <code>nn.Sequential</code> object you have a single input node which is why <code>x1 = torch.rand(1)</code> works. However, when you pass <code>x1 = torch.rand(10)</code> you’re effectively trying to do matmul between a vector of length 1 and vector of length 10 which you can’t do. So, you need to include the batch size <strong>and</strong> the input size.</p>
<p>So change <code>x1 = torch.rand(10, 1)</code> and it’ll calculate the output for a batch of size 10.</p>
Parallelizing different models with shared replay buffer

username: whoab
cooked: <p>I have 5 different models. I want them to train asychronously with a shared replay buffer (the replay buffer only needs to be synced every 1000 iterations as new samples are added).</p>
<p>How do I do this?</p>
<p>While I understand Pytorch launches cuda operations asynchronously, I think whenever I call .backwards() it is a synchronization point. Therefore, the naive solution of having an array of models = […] and looping with a for loop the inner procedure of: zero_grad, backward, step will not actually parallelize the training.</p>
What happens when you continue before the barrier?

username: ojipadeson
cooked: <p>For example:</p>
<pre><code class="lang-python">m = []
for key in adict:
    if key in bdict:
        v = bdict[key]
    else:
        continue
    dist.barrier()
    t = [[None] * len(v) for _ in dist.get_world_size()]
    dist.all_gather_object(t, v)
    m.append(t)
</code></pre>
<p>Could the program be stuck because a process that does not contain the key has already entered the next loop and the barrier can never wait for this process to enter?</p>
<hr>
<p>I’m trying a handler similar to the above example:</p>
<pre><code class="lang-python">key = ['a', 'b', 'c']
for k in key:
    if k in new_dict:
        # gather new_dict[k] from processes
    dist.barrier()
</code></pre>
<p>There is a possibility that there is no certain key in the <code>new_dict</code>. But this program gets stuck.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="ojipadeson" data-post="1" data-topic="158096">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ojipadeson/48/49136_2.png" class="avatar"> ojipadeson:</div>
<blockquote>
<p>Could the program be stuck because a process that does not contain the key has already entered the next loop and the barrier can never wait for this process to enter?</p>
</blockquote>
</aside>
<p>Yes, it can. All peer processes need to call into the same set of collectives in the same order.</p>
The orders of data in dataloader

username: maryma
cooked: <p>Hi I had one datasets I made a dataloader of this dataset. I wanted to extract features of my datasets from a model. I dont know when I make a dataloader with batch = 1 and shuffle false is the data loader organized or not? I mean I wanted to know is the first data in my dataloader exactly first data in my dataset that I made dataloader of? How can I understand which data from my dataset loads to dataloader? Because the order of dataloader is important for me.</p>

username: ptrblck
cooked: <p>If <code>shuffle=False</code> is set then a <code>SequentialSampler</code> will be used which will pass the indices in order to the <code>Dataset.__getitem__</code>. You could print the <code>index</code> in the <code>__getitem__</code> and would see the used order.</p>

username: blackbirdbarber
cooked: <aside class="quote no-group" data-username="maryma" data-post="1" data-topic="158125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryma/48/56556_2.png" class="avatar"> maryma:</div>
<blockquote>
<p>shuffle false</p>
</blockquote>
</aside>
<p><code>shuffle=False</code> is like shooting your own hand. Don’t hurt yourself. Use shuffle=True.<br>
And note false should be False in your question.</p>