akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Multiply two tensors along an axis

optyang: <p>Hi, I have a tensor <code>x1</code> 4x3x2x2, and a tensor <code>x2</code> 4x1. I would like tensor <code>x1</code> and <code>x2</code> multiply for each element along axis 0 (which has a dimension of 4). Each such multiplication would be between a tensor 3x2x2 and a scalar, so the result would be a tensor 4x3x2x2.</p>
<p>A for loop implementation would be below, is there a better (parallel) implementation, perhaps using one of pytorch multiply functions? Thanks a lot!</p>
<pre><code class="lang-auto">x1 = torch.rand(4, 3, 2, 2)
x2 = torch.rand(4, 1)

for i in range(4):
    print(x1[i]*x2[i])
</code></pre>

KFrank: <p>Hi Yang!</p>
<aside class="quote no-group" data-username="optyang" data-post="1" data-topic="110256" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/optyang/48/27873_2.png" class="avatar"> optyang:</div>
<blockquote>
<p>I have a tensor <code>x1</code> 4x3x2x2, and a tensor <code>x2</code> 4x1. I would like tensor <code>x1</code> and <code>x2</code> multiply for each element along axis 0 (which has a dimension of 4). Each such multiplication would be between a tensor 3x2x2 and a scalar, so the result would be a tensor 4x3x2x2.</p>
</blockquote>
</aside>
<p>If I understand what you are asking, you could either transpose and use<br>
broadcasting:</p>
<pre data-code-wrap="python"><code class="lang-python">(x1.transpose (0, 3) * x2.squeeze()).transpose (0, 3)
</code></pre>
<p>or use <code>torch.einsum</code> (“Einstein summation”):</p>
<pre data-code-wrap="python"><code class="lang-python">torch.einsum ('ijkl, im -&gt; ijkl', x1, x2)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

optyang: <p>Thanks a lot!!!</p>
<p>I would appreciate your comment on this: which of these two methods is faster and which has a smaller memory consumption?</p>
<p>Thanks again and best!</p>

KFrank: <p>Hi Yang!</p>
<aside class="quote no-group" data-username="optyang" data-post="3" data-topic="110256" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/optyang/48/27873_2.png" class="avatar"> optyang:</div>
<blockquote>
<p>which of these two methods is faster and which has a smaller memory consumption?</p>
</blockquote>
</aside>
<p>I don’t know which approach would be more efficient.  (I’ve never tested it.)</p>
<p>I could see <code>torch.einsum()</code> having a little extra overhead because it has<br>
to parse the <code>'ijkl, im -&gt; ijkl'</code> string.  It could also be (maybe in certain<br>
situations) significantly less efficient if its generality prevents it from performing<br>
the tensor multiplications in the most optimal way.  But maybe it’s smart<br>
enough to figure out the optimal approach.</p>
<p>Best.</p>
<p>K. Frank</p>
How to find number of images for training after augmentation?

sampa: <p>I have used this transformation. However, when I print the number of samples for training, it is showing the same number of images I have. I want to know the number of images after the augmentation? Can any one please tell me how to find the number of images after augmentation? Thanks in advance.</p>
<pre><code class="lang-auto">train_transforms = transforms.Compose([transforms.RandomResizedCrop(size=224, interpolation=2),
                                      transforms.Resize((224,224)),
                                      transforms.RandomHorizontalFlip(),
                                      transforms.RandomRotation(15),
                                      transforms.ToTensor(),
                                      transforms.Normalize([0.485, 0.456, 0.406],
                                                           [0.229, 0.224, 0.225])
                                      ])

test_transforms = transforms.Compose([
                    transforms.Resize((224,224)),
                    transforms.ToTensor(),
                    transforms.Normalize([0.485, 0.456, 0.406], 
                                         [0.229, 0.224, 0.225])
                    ])
merge_data = datasets.ImageFolder(data_dir + "/train", transform=train_transforms)

train_data, valid_data = train_test_split(merge_data, test_size = 0.2, random_state= 123)
test_data= datasets.ImageFolder(test_dir,transform=test_transforms)
num_workers = 0
print("Number of Samples in Train: ",len(train_data))
print("Number of Samples in Valid: ",len(valid_data))
print("Number of Samples in Test ",len(test_data))
train_loader = torch.utils.data.DataLoader(train_data, batch_size,
     num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(valid_data, batch_size, 
     num_workers=num_workers)
</code></pre>

jony: <p>Well, I’m not sure if I understood what you are willing to do but, as far as I know, the Pytorch dataset is making the transformation while you are training.<br>
Which means, for now, non of your images where transformed.<br>
But, you can see the size of your dataset by calling the method <strong>len</strong>()<br>
in your code: train_data. _ _len _ _ ()<br>
Regardless to that, you can iterate over the train/test loaders, you will get number of values and labels as your batch size after the transformation. It is like seeing how your data will be feed into your model.<br>
If your dataset only returns x (value) and y (label). you can write this:<br>
x,y = iter(train_loader).next()<br>
both will be in the size of your batch size.<br>
x will contain your input tensors after the transformation<br>
y will contain the labels of these inputs<br>
after that you can access to one of your inputs by writing: x[0] (or other number in the range of your batch size)</p>
RuntimeError: Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same

Anno: <p>It looks like pytorch initializes weights of models by default with FloatTensors. However, I would like to have DoubleTensors. I initialize my weights with</p>
<pre><code class="lang-auto">def weight_init(self):
      for module in self.modules():
        if (isinstance(module, nn.Conv3d) or isinstance(module, nn.Linear)):
          init.orthogonal_(module.weight)
</code></pre>
<p>What is a proper way to initialize a model’s weights with DoubleTensors? Or is this generally not recommended?</p>
<p>Thanks for any help.</p>

ptrblck: <p>You could probably transform all parameters to <code>float64</code> before calling <code>wieght_init</code> via <code>model.double()</code>.</p>
Difference between inputting all data in a batch and sending all data in for-loop?

Wesley_Neill: <p>So, I’ve trained an LSTM autoencoder to reconstruct time-series data. It’s trained to encode/decode 7-day sub-sequences of the time series.</p>
<p>When I test it on a single subsequence, it reconstructs the data very well. I can also reconstruct the entire time-series by running each sequence through the model in a for-loop. It gets really close, and I’m very happy.</p>
<p>However, what I thought I could do was run all the sequences through the model in parallel by making an input of shape <code>(batch_size, seq_len, n_features)</code> where <code>seq_len</code> is my 7 day sub-sequences, and so <code>batch_size = len(timeseries) // seq_len</code>.</p>
<p>However, the model does not correctly reconstruct the time-series with this method. Why not? Isn’t this exactly how the model trained and updated the gradient?</p>
<p>Thanks!</p>
How to zero a matrix differently on each row

divinho: <p>Let’s say I have <code>a=to.randn(4, 4)</code>.</p>
<p>What I want to do is zero the rows starting an index that change for each row. Example manual implementation would be</p>
<pre><code class="lang-auto">a[:2, 1:] = 0.
a[2:, 2:] = 0.
</code></pre>
<p>Is there a way to do it without having to do a loop? What would be given for example is a list of the lengths to keep for each row for example, in this case <code>[1, 1, 2, 2]</code>, then the question is how to turn that into a mask.</p>

ptrblck: <p>I think if you are dealing with indices having a variable shape for each dimension, creating a <code>mask</code> should work. The mask creation itself might also involve a loop, which might be cheaper than the indexing loop.</p>

divinho: <p>Hm, in the end I want to sum over a certain axis. Is there no way to combine the mask plus summing in such a way so as to avoid doing operations on the just zeroed elements?</p>

divinho: <p>Surprisingly the mask is slower!</p>
<pre><code class="lang-auto">mask = to.zeros((N, T,), dtype=to.bool, device=x.device)
idx = 0
for dur, count in zip(durs, counts):
    mask[idx: idx + count, :dur] = True
    idx += count
joined = (x * mask.unsqueeze(2).expand(-1, -1, x.size(2))).sum(dim=1)
</code></pre>
<p>is slower than</p>
<pre><code class="lang-auto">joined = to.zeros((N, self.repr_dim), dtype=to.float32, device=x.device)
idx = 0
for dur, count in zip(durs, counts):
    x_part = x[idx: idx + count, :dur]
    joined[idx: idx + count] = x_part.sum(dim=1)
    idx += count
</code></pre>
Data from observer?

Matteo_Scrugli: <p>I trained my quantized network, I was able to export the weights and bais to perform the inference on my platform.<br>
However, I cannot find the data obtained from the pytorch observer and I have to manually calculate the parameters to scale the outputs of each layer (conv layer and fully layer). How can I extract this information directly from pytorch and then query the observer?<br>
Thanks.</p>

Vasiliy_Kuznetsov: <p>You can use</p>
<p><code>scale, zero_point = observer.calculate_qparams()</code></p>
<p>Please see <a href="https://pytorch.org/docs/stable/torch.quantization.html#observers" class="inline-onebox" rel="noopener nofollow ugc">torch.quantization — PyTorch 1.7.0 documentation</a> for more info.</p>

Matteo_Scrugli: <p>Forgive me if I ask trivial things but I am a beginner, this is the code:</p>
<pre><code class="lang-auto">model_quantized = model

eval_batch_size = value_batch_size
model_quantized.eval()
model_quantized.qconfig = torch.quantization.get_default_qconfig('qnnpack')
torch.backends.quantized.engine = 'qnnpack'

torch.quantization.prepare(model_quantized, inplace=True)
evaluate(model_quantized, criterion, loader_train, neval_batches=num_calibration_batches)

torch.quantization.convert(model_quantized, inplace=True)
top1, top5 = evaluate(model_quantized, criterion, loader_valid, neval_batches=num_eval_batches)

temp_scale, temp_zero_point = torch.quantization.observer.MinMaxObserver.calculate_qparams()
</code></pre>
<p>but if i run it, i get:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main2D.py", line 785, in &lt;module&gt;
    temp_scale, temp_zero_point = torch.quantization.observer.MinMaxObserver.calculate_qparams()
TypeError: calculate_qparams() missing 1 required positional argument: 'self'
</code></pre>
<p>Should i write the command differently or in a different place?<br>
Thank you very much.</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="Matteo_Scrugli" data-post="3" data-topic="110259">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matteo_scrugli/48/30703_2.png" class="avatar"> Matteo_Scrugli:</div>
<blockquote>
<pre><code class="lang-auto">evaluate(model_quantized, criterion, loader_train, neval_batches=num_calibration_batches)
</code></pre>
</blockquote>
</aside>
<p>You’d want to inspect the observer instances.  After this line of code,</p>
<pre><code class="lang-auto">evaluate(model_quantized, criterion, loader_train, neval_batches=num_calibration_batches)
</code></pre>
<p>if you print your model, you should see observer modules attached, with statistics.  You can query these module instances.  For example, if you have a <code>model.conv1</code>, you can then call <code>model.conv1.activation_post_process.calculate_qparams()</code> to get qparams for the activation of <code>conv1</code>.</p>
How to get back the model from flattened network?

Ajinkya_Bankar: <p>Hi, I am using following function to flatten the network:</p>
<pre><code class="lang-auto">#############################################################################
# Flattening the NET
#############################################################################
def flattenNetwork(net):
    flatNet = []
    shapes = []
    for param in net.parameters():
        #if its WEIGHTS
        curr_shape = param.cpu().data.numpy().shape
        shapes.append(curr_shape)
        if len(curr_shape) == 2:
            param = param.cpu().data.numpy().reshape(curr_shape[0]*curr_shape[1])
            flatNet.append(param)
        elif len(curr_shape) == 4:
            param = param.cpu().data.numpy().reshape(curr_shape[0]*curr_shape[1]*curr_shape[2]*curr_shape[3])
            flatNet.append(param)
        else:
            param = param.cpu().data.numpy().reshape(curr_shape[0])
            flatNet.append(param)
    finalNet = []
    for obj in flatNet:
        for x in obj:
            finalNet.append(x)
    finalNet = np.array(finalNet)
    return finalNet,shapes
</code></pre>
<p>I manipulate the weights given by finalNet and I don’t change the shape of the network. How can I get back the original network with manipulated weights? Later I want to see the effect of weight manipulations on the accuracy. Please help. Thank you.</p>

Ajinkya_Bankar: <p><a class="mention" href="/u/mariosoreo">@MariosOreo</a> Sir, can you please help me with my question? Thanks.</p>
ONNX Total Accuracy Loss

scrappedsoda: <p>Hello,<br>
I am currently trying to modify the output of a convolution with a forward hook. So far this works in PyTorch and I have ~67% on Cifar10 before adding the hook and ~35% after adding the hook. This are values which are fine for this testcase. But after exporting the model to onnx, the accuracy drops down to 10% being equivalent to random within Cifar10. The forward hook looks like this:</p>
<pre><code class="lang-auto">zero_tensor = torch.zeros(
        (o_tensor.shape[0], module.out_channels,
         o_tensor.shape[2], o_tensor.shape[3])
    ).to(o_tensor.device)
zero_tensor[:, module.keep_idxs.to(o_tensor.device), :, :] = o_tensor
return zero_tensor
</code></pre>
<p>My current guess is, that the slicing notation is not compatible with onnx exporting. Further notice is, that the keep_idxs stay constant when set once.</p>
<p>So my question now is: How to write this, that the functionality stays, but it is compatible with an onnx export? I have already looked into the diverse tensor methods, but I was unable to find one to recreate what I achieve with the slicing notation.</p>
<p>EDIT 1: Turns out the whole output after the layer is zero in the onnx. Therefore not random data, but all zeros is output.</p>
<p>Best regards</p>

scrappedsoda: <p>Well I found the solution to my hassle. In case anybode else runs into this problem, here is my solution. I replaced the<br>
<code>zero_tensor[:, module.keep_idxs.to(o_tensor.device), :, :] = o_tensor</code><br>
line with the following:<br>
<code>zero_tensor.index_copy_(1, module.keep_idxs, o_tensor)</code></p>
<p>The reason for the failure is the following: The onnx exporter seems not to support colon notation. At least not in PyTorch 1.3.1. The new line is supported according to the ONNX export list and it also works now.</p>
<p>Best regards</p>
Standard deviation of complex tensors

andre.w: <p>Hi there,</p>
<p>I experimented with the complex tensor operations introduced in PyTorch 1.7. I was a bit surprised by the following example:</p>
<pre><code class="lang-auto"># tested in PyTorch 1.7.1
import torch as pt

X1 = pt.tensor([1.0, -1.0], dtype=pt.cfloat)
print(pt.std(X1, dim=0)) # results in sqrt(2)+0.0j
print(pt.std(X1.real)) # also results in sqrt(2)
print(pt.std(X1)) # causes a runtime error
# RuntimeError: _th_std not supported on CPUType for ComplexFloat
</code></pre>
<p>I do not really understand why the third print statement causes the runtime error. First I thought it’s only an issue of higher-dimensional tensors, but it also happens for 1D inputs.</p>
<p>My second question is related to the formula used to compute the standard deviation. Consider the following example:</p>
<pre><code class="lang-auto">import torch as pt

X1 = pt.tensor([1.0, -1.0], dtype=pt.cfloat)
X2 = pt.tensor([1.0j, -1.0j], dtype=pt.cfloat)
X3 = pt.tensor([1.0+1.0j, -1.0-1.0j], dtype=pt.cfloat)
print(pt.std(X1, dim=0)) # results in sqrt(2)+0.0j
print(pt.std(X2, dim=0)) # results in sqrt(2)+0.0j
print(pt.std(X3, dim=0)) # results in 2.0+0.0j
print(pt.std(X3.real)) # results in sqrt(2)
</code></pre>
<p>Based on this simple example, I guess that the standard deviation in 1D is computed as the standard deviation of the absolute values, something like</p>
<pre><code class="lang-auto">pt.sqrt(pt.dot(X, X.conj()))
</code></pre>
<p>Is that correct? I couldn’t find it in the documentation, and I wasn’t able to find the actual piece of code computing the std. I would have expected that the imaginary part is removed before computing the standard deviation (is there an application where the above definition is meaningful?).</p>
<p>Thanks and regards,<br>
Andre</p>
How to quickly compare each word vector in two sentences

WoodrowOnTheWay: <p>My problem is that, if I got two sentence, the sentence1 have 2words, and sentence2 have 3 words, I want to align them in a less common way, use a compare function, and get 2x3 tensors like comp(a1, b1), comp(a1, b2), comp(a1, b3), comp(a2, b1), comp(a2, b2), comp(a2, b3), Is there any way to do it?</p>
<p>Expect to be resolved!</p>
Cuda out of memory when updating dataset

Boat: <p>I get an out of memory error when I update the dataset while in iteration of a data loader<br>
This function is def inside a custom dataset</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 14.73 GiB total capacity; 13.77 GiB already allocated; 3.88 MiB free; 13.79 GiB reserved in total by PyTorch)</p>
</blockquote>
<pre><code class="lang-auto">    def updateindex(self,index,new):#Why is this shit leaking memory
        for i in range(len(index)):
          self.targets[index[i]]=new[i]#self.targets[index]*(0.7)+(1-0.7)*new
        del index
        del new
</code></pre>
<p>where index is a Tensor of size n indexes and new is a Tensor of size n by 512 to update the datasets target by<br>
new is defined by</p>
<pre><code class="lang-auto">new=target*β+(1-β)*momentumout
</code></pre>
<p>where target is the target from the dataloader, beta is a float and momentumout is the net output of the image<br>
I’m not sure why it mallocs, can anybody help?</p>
Use pre-trained autoencoder for classification or regression

anotherone_one: <p>Hello!!</p>
<p>I trained an autoencoder and now I want to use that model with the trained weights for classification purposes. So, I suppose I have to freeze the weights and layer of the encoder and then add classification layers, but I am a bit confused on how to to this.</p>
<p>This is the AutoEncoder I trained</p>
<pre><code class="lang-auto">class AE(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=128
        )
        self.encoder_output_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_output_layer = nn.Linear(
            in_features=128, out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)
        code = self.encoder_output_layer(activation)
        code = torch.relu(code)
        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)
        activation = self.decoder_output_layer(activation)
        reconstructed = torch.relu(activation)
        return reconstructed
</code></pre>
<p>Can someone help me?</p>
<p>Thank you</p>

Dwight_Foster: <p>You could create a separate model then pass the autoencoder into it and load the weights in that.  Then you could create a separate sequential layer with the classification layers and then in the forward pass you could pass the input through those two. Then too freeze the autoencoder weights in the optimizer you could just set the parameters as only the classification layers.</p>

anotherone_one: <p>Can you show me in code how to do that?</p>

Dwight_Foster: <p>Yes here is the model code you could use</p>
<pre><code class="lang-auto">class classification(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.AE = AE(kwargs)
        self.AE.load_state_dict(FILENAME)
        self.classifier = nn.Sequential(nn.Linear(INPUT,OUPUT))
        ##You can use as many linear layers and other activations as you want
    def forward(self, input):
        x = self.AE(input)
        output = self.classifier(output)
        ##You can use a sigmoid or softmax here if neecd
        return output
</code></pre>
<p>and then the optimizer</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.classifier.parameters(), lr=LR)
</code></pre>

anotherone_one: <p>Thank you so much!!!</p>
Semi-supervised code pytorch

Celmar: <p>I have the data 𝑋∈ℝ𝑛×𝑑 with 𝑛=300,000 the number of data and 𝑑=2048 the number of variables. Of the 𝑛 given I only have about 10,000 labels 𝑦∈{0.1}} so for about 290,000 individuals, I don’t know the labels.</p>
<p>Is there a way to create a good classifier?</p>
<p>I think this problem is widely known as weakly or semi-supervised learning. I just can’t find adequate code on GitHub in pytorch , preferably quite simple and understandable.</p>
<p>Thank you in advance</p>
Sum of losses, inplace operation error

Riccardo_Taiello: <p>Hi I don’t why this happens, someone can help me? Thanks in advance.</p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, n_clients):
        super(MyEnsemble, self).__init__()
        self.models = nn.ModuleList([MyModelA() for _ in range(0,n_clients)])
        
    def forward(self, x1, id):
        output = self.models[id](x1)
        return output
    def get_model(self, id):
      return self.models[id]
    def disable_model_grad(self):
      for p in self.models.parameters():
        p.requires_grad_(False)
    def enable_model_grad(self, id):
      for p in self.models[id].parameters():
        p.requires_grad_(True)
ensemble = MyEnsemble(N_CLIENTS)
dataloader_list = [DataLoader(TensorDataset(torch.randn(10, 10), torch.randn(10, 2))) for _ in range(0,N_CLIENTS)]
optimizer = optim.SGD(ensemble.parameters(), lr=0.01, momentum=0.9)

all_losses = None
for client in range(0,N_CLIENTS):
  ensemble.disable_model_grad()
  for feature, target in dataloader_list[client]:
    ensemble.enable_model_grad(client)
    predicted = ensemble.forward(feature,id=client)
    optimizer.zero_grad()
    output = loss(predicted, target)
    output.backward(retain_graph=True)
    optimizer.step()
  if all_losses is None:
      all_losses = output
  else:
      all_losses += output
</code></pre>
<p>Now when I compute the backward on all_losses as follows:</p>
<pre><code class="lang-auto">all_losses.backward()
</code></pre>
<p>This error appears, RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 2]], which is output 0 of TBackward, is at version 363;</p>

ptrblck: <p>The <code>optimizer</code> already executed its <code>step()</code> method using the calculated gradients from the <code>output.backward()</code> call and thus manipulated the parameters.<br>
<code>all_losses</code> stores these <code>output</code> tensors (losses) and tries to calculate the gradients again, which won’t work since the parameters were already updated.</p>
key changes between pytorch 1.7.1(stable) & pytorch 1.8

Ankita_AN: <p>Hi,<br>
What are key changes between pytorch 1.7.1(stable) &amp; pytorch 1.8?</p>
<p>Thanks</p>

ptrblck: <p>The <code>1.8</code> release is not finished yet, so I would recommend to wait for the release notes, which will explain all added features and fixes.</p>
How to setup a custom Dataset to work with PyTorch

Gerrit: <p>So I am trying to create a Dataset class that should later work like any other the standard pytorch Dataset. The raw data contains measurements from accelerometers that were attached to a gearbox. Data was collected in 560 different runs while the health of the gearbox was degrading between the runs. I already extracted 16 features for each run which i would like to use as input data for the nerual network(Deep Forward). The point I am struggeling with, is how to generate a dataset of these runs. I followed <a href="https://towardsdatascience.com/building-efficient-custom-datasets-in-pytorch-2563b946fd9f" rel="noopener nofollow ugc">this</a> tutorial, but get stuck because my data is two dimensional, where the different runs are arranged in rows and the respective features in columns. This is the code I got so far but I dont know how to access the different rows.</p>
<pre><code class="lang-python">import os
import pandas as pd
from torch.utils.data import Dataset

class Features(Dataset):
    def __init__(self):
        folderpath = r'...\Balanced_Data'
        data_path = os.path.join(folderpath, 'good_and_bad.csv')
        data = (pd.read_csv(data_path, header = None))
        self.samples = data
    
    def __len__(self):
        return len(self.samples)
    
    def __getitem__(self, idx):
        return self.samples.at[idx,0]
</code></pre>
<p>When trying to access the single rows,</p>
<pre><code class="lang-auto">dataset.samples[0]
</code></pre>
<p>yields all entries from the first column.</p>
<p>Since I am fairly new to this topic I very much appreciate any help and tipps for a general approach to this.</p>
<p>Cheers,<br>
Gerrit</p>

Dwight_Foster: <p>If you use the pandas .iloc() function here you can pass in an index and get out the row. The documentation is found <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iloc.html" rel="noopener nofollow ugc">here</a>.</p>

Gerrit: <p>That worked,<br>
thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
How to get valid pixels from grid_sample()?

adizhol: <p>I’m using torch.nn.functional.grid_sample to warp an image, and I need to know which pixels came from outside the image.<br>
This function supports “Zero” padding, but my image can have zeros in it as well, so I can’t search for zeros in the image for a valid pixels mask.<br>
How to get valid pixels from grid_sample()?</p>
<p>Thnx</p>

ptrblck: <p><code>grid_sample</code> will use the the padding values for all input values outside of <code>[-1, 1]</code> as explained in the <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.grid_sample">docs</a>. So you could check your input for these values.</p>
Sparsely connected graph that skips layers

markrbower: <p>I am trying to implement a graph using PyTorch where each layer contains a single node and connections between layers may skip several layers. As a simple example, the code below attempts to connect 4 nodes (A,B,C and D) with connections between A-B, A-C, B-D and C-D. The code below attempts to do this, but even if it worked, it is not clear to me how I can access the layers to know who is connected to whom:</p>
<p>import torch<br>
import torch.nn as nn</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.A = nn.Linear(1,1)<br>
self.B = nn.Linear(1,1)<br>
self.C = nn.Linear(1,1)<br>
self.D = nn.Linear(1,1)</p>
<pre><code>def forward(self, x):
    out = self.B(A,D)
    out = self.C(A,D)
    out = self.
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
my_nn = Net()<br>
print(“I’m a net!”)</p>

ptrblck: <p>I don’t completely understand the use case and the current code snippet seems to try to pass modules to the <code>forward</code> method of the linear layers?<br>
Could you explain your use case a bit more and how these connections should look like?<br>
If you want to pass the output of one layer to specific others (to create the “connections”), you should be able to do so:</p>
<pre><code class="lang-python">def forward(self, x):
    x = self.A(x)
    xb = self.B(x)
    xc = self.C(x)
    ...
</code></pre>
Question About Adding Parameters/Biases

J_Johnson: <p>Hello!</p>
<p>If I want to add more neurons to a layer inside of a model during training, do I only need to update the <code>model.parameters()</code> for each layer(weights and biases) after backpropagation, or is there more to it than that? If so, what would I need to do in addition to resizing the parameter tensors?</p>

ptrblck: <p>Since you are changing the parameters, you would need to pass these new parameters to the optimizer (or create a new one). The running stats in this optimizer would be lost in this case (if it’s using internal estimates).</p>

J_Johnson: <p>After reading several posts, I’ve come to this same conclusion. Which leads to my next question, here:</p><aside class="quote quote-modified" data-post="1" data-topic="110218">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/modifying-custom-optimizer-updating-layer-values/110218">Modifying/Custom Optimizer: Updating Layer Parameters</a> 
  </div>
  <blockquote>
    I’m working on a modification to a pytorch optimizer file. For simplicity sake, let’s take SGD(sgd.py). If I wanted to change the “for loop” on line 94 to: 
for i, p in enumerate(group['params']: 
And add something starting on line 111 in the for loop like: 
if i==0:
  bias=torch.rand(10)
  group['params'][i+1]=bias

Also, suppose that the number of biases are 8 in this layer before going through this ‘optimizer’. Currently, it doesn’t update the parameters when I run this. What would I need to …
  </blockquote>
</aside>
How do I gather with differen dim?

jinfagang: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d885f94c97a21b0b7f6ea26e24d288262cfa355.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d885f94c97a21b0b7f6ea26e24d288262cfa355" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d885f94c97a21b0b7f6ea26e24d288262cfa355.png" alt="image" data-base62-sha1="fCYdOGIkOlFcBXJnViPaKu7HWYd" width="434" height="500" data-dominant-color="3D1E34"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">717×825 75.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want index a 17,48 tensor with a 17 tensor and get another 17 tensor,<br>
which is, get the 30th from first row, get 46th from second row…</p>
<p>what should I do?</p>

ptrblck: <p>This code should work:</p>
<pre><code class="lang-python">x = torch.randn(17, 48)
idx = torch.randint(0, 48, (17,))
x[torch.arange(x.size(0)), idx]
</code></pre>
Random initialization within range on pretrained model

UnaiCorzo: <p>Hello,</p>
<p>I would like to set the weights and biases of my pytorch model (which is already trained) randomly within a range. For instance, let the value for of a weight be 12. I’d like it to be set to a random value from a custom range (6-18 for example); and so with every weight/bias. It’s like a naive random search, although I’m just trying to experiment with the model.</p>
<p>One example could be this (in pseudocode):</p>
<pre><code class="lang-auto">for weight, bias in model:  // for the entire model
    weight = random(weight/2, weight*2)
    bias = random(bias/2, bias*2)
</code></pre>
<p>How can I set those randomly within that custom range and save the new model (as a .pt for instance)?</p>
<p>Thanks,</p>

Dwight_Foster: <p>you could do something like</p>
<pre><code class="lang-auto">layer.weight.data.fill_(random_weight)
layer.bias.data.fill_(random_bias)
</code></pre>
<p>then to save you it you would just go</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), "new_model.pt")
</code></pre>
<p>The only problem with this is you have to set each layer individually.</p>

UnaiCorzo: <p>Thanks for your answer. But wouldn’t <code>layer.weight.data.fill_(random_weight)</code> set all the values in that layer to <em>the same</em> value? What I want is to set <em>each</em> weight/bias to a random value within the range of (<code>weight-2, weight+2</code> for example), and not all weights/biases from each layer to the exact same number. It should variate depending on the current values the model already has.</p>
Retraining LSTM

Alaa_Jamal: <p>Hello,</p>
<p>I am working on LSTM model which already trained and works perfectly. However, I need to improve the accuracy by retraining this model when new data is available. Is this possible in PyTorch?</p>
<p>Thanks.</p>

ptrblck: <p>Yes, you can fine-tune a model by continuing the training process.<br>
Depending on the use case, you might want to restore the old optimizer besides the model.</p>
Does .item() automatically move the data to the cpu?

oneil512: <p>When you call item(), the number is no longer a tensor, does this necessarily mean that it can no longer live on the gpu? The docs for .item() don’t say anything about the data moving, however, the docs for .tolist(), which is the same as .item() except for multiple items, says the data is moved to cpu first ‘if necessary’. What does that mean? <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.tolist" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.tolist</a></p>

albanD: <p>Hi,</p>
<p>Yes <code>.item()</code> moves the data to CPU. It converts the value into a plain python number. And plain python number can only live on the CPU.</p>

AH-Merii: <p>Hi,</p>
<p>Sorry if this might come of as a stupid question;</p>
<p>What’s the difference between using <code>.item</code> and <code>.data</code> then?</p>

ptrblck: <p><code>.item()</code> returns the value as a “standard Python number”, while the <code>.data</code> attribute accesses the internal tensor with its value (and you could thus easily break the computation graph by manipulating it).</p>
<pre><code class="lang-python">x = torch.randn(1)
print(x.item())
&gt; -0.7826926112174988
print(x.data)
&gt; tensor([-0.7827])
</code></pre>

AH-Merii: <p>Thank you so much for your prompt reply! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>
<p>So let’s say that I would like to use sklearn’s metrics, what would be the best approach to copy my tensor as  a numpy input?</p>
<p>Let’s take the example below where my <code>preds</code> and <code>labels</code> are tensors on the GPU</p>
<pre><code class="lang-auto">precision,recall,f1,_ = precision_recall_fscore_support(labels,preds,average='binary')
</code></pre>
<p>Would I use <code>.item().numpy()</code> in this case?</p>
<p>Because it seems that .item() only works when you have a single value.</p>

ptrblck: <p>No, you would detach the tensor (in case it has an Autograd history), push the data to the CPU, and transform it to a numpy array via:</p>
<pre><code class="lang-python">preds = torch.randn(10, 10, requires_grad=True) # GPUTensors with Autograd history
preds_arr = preds.detach().cpu().numpy()
np_fun(preds_arr)
</code></pre>

AH-Merii: <p>Would it be equivelent to the below?</p>
<pre><code class="lang-auto">with torch.no_grad():
    preds_array  = preds.cpu().numpy()
</code></pre>
<p>Also is there a downside to doing the following instead:</p>
<pre><code class="lang-auto">with torch.no_grad():
    outputs  = np_fun(preds.cpu().numpy())
</code></pre>

ptrblck: <p>The first code snippet would be equivalent to my example and you might prefer one or the other approach.<br>
No, there is no downside in “combining” the calls. Note however, that the <code>numpy()</code> operation is cheap, as no data will be copied.</p>

AH-Merii: <p>Thank you so much for your help, you have no idea how much your contributions have helped this community! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>
Backpropagation with mask in output

User4: <p>If I want to compute backprop with some element of output from model:</p>
<pre><code class="lang-python">output = model(image) # tensor([[0.6000], [0.2000], [0.1500], [0.0500]])
mask = output &gt;= 0.2 # tensor([[ True], [ True], [False], [False]])
predict = output[mask] # tensor([0.6000, 0.2000])
loss = criterion(predict, ground_truth[mask]) # some number
optim.zero_grad()
loss.backward()
</code></pre>
<p>In the example above, assume that I am dealing with the binary classification with the batch size is 4 and I only want to compute the gradient from which output is greater or equal to 0.2. So my question, Is it the correct way to do it, if not please let me know how, thank you.</p>

CedricLy: <p>Normally one uses torch.where() function.</p>

User4: <p><a class="mention" href="/u/cedricly">@CedricLy</a> I just want to confirm that the autograd do what I want it to do. Can you guarantee that if I use <code>torch.where()</code>, the gradient will be computed on those nodes only?</p>
Does running the forward method on a model (nn.Module) calculate the gradients?

Charan0: <p>I’m new to PyTorch and I’m taking a liking to it, but a small doubt I have is whether or not gradients are calculated when the forward method is run on a model is used for prediction.<br>
If so (gradients are calculated) does running loss.backward() just populate those gradients? or is it like running the loss.backward() method does both of them together (calculate gradients and populate them too)</p>
<p>Thanks in Advance!</p>

ruotianluo: <p>No, forward doesn’t calculate gradients.<br>
I don’t know what you mean by populate, do you mean propagate? loss.backward() does the back propagation, and compute all the gradients for the network weights.</p>

Charan0: <p>By populate, I mean allocating the value of the gradients in the tensor. Thanks for clarifying my issue, I now understand that model.forward() only does the forward propagation and no gradient calculation is involved.</p>
Why torch calls empty_cache?

Seungsu: <p>Hi. I’m studying about torch’s autograd engine.</p>
<p>I found calling empty_cache while doing loss.backward.</p>
<p>This behavior only occurs on the first.</p>
<p>I know that Pytorch’s memory allocator uses cached pool.<br>
So i think<br>
There is no reason to call an empty cache().</p>
<p>But Pytorch calls that.</p>
<p>What’s the reason?</p>

ptrblck: <p>Internally <code>empty_cache</code> is called if an out of memory error was detected as an attempt to recover from it. Also, if you are using cudnn with <code>benchmark=True</code>, different algorithms will be profiled in the first iteration, which use a different amount of memory for their workspaces. To recover this memory and allow other processes to use it, <code>empty_cache</code> will also be used.</p>

Seungsu: <p>Thank you for the reply.</p>
<p>I have more questions.</p>
<p>As you say, i’m using cudnn and benchmark=True.</p>
<p>I know that if I set benchmark=true, it will automatically find the right algorithm for my hardware.</p>
<p>But i’m not sure what the algorithms does…</p>
<p>Could you please explain if you are not busy?</p>

ptrblck: <aside class="quote no-group" data-username="Seungsu" data-post="3" data-topic="110127">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/67e7ee/48.png" class="avatar"> Seungsu:</div>
<blockquote>
<p>I know that if I set benchmark=true, it will automatically find the right algorithm for my hardware.</p>
<p>But i’m not sure what the algorithms does…</p>
</blockquote>
</aside>
<p><code>cudnnFind</code> will use different kernel implementations for the current workload, e.g. the current conv layer. You can imagine it as iterating over a conv implementation using matrix multiplications, an FFT, Winograd, etc. Each of these different algorithms (for the same convolution with the given input shapes, padding, dilation, etc.) will be executed and the time will be captured. Once this is done, the fastest would be selected in the best case.</p>

Seungsu: <p>Thank you so much!</p>
<p>It was very helpful!! have a nice day!</p>
Reversing a linear layer

rahulvigneswaran: <p>The goal is to reverse a trained linear layer that goes from 128-&gt;64.</p>
<p>The network:</p>
<pre><code class="lang-auto">class embedding(nn.Module):

    def __init__(self):
        super(embedding, self).__init__()
        self.fc1 = nn.Linear(128, 64)

    def forward(self, x):
        x = self.fc1(x)
        return x
</code></pre>
<p>The following is the code for the solution:</p>
<pre><code class="lang-auto"># Z is the vector that needs to be reversed -&gt; Nx64
model = embedding() # Assume that this is pre-trained
step = list(model.modules())[-1]
if isinstance(step, torch.nn.Linear):
    s1 = (z-step.bias).unsqueeze(2) #Nx64x1
    w = step.weight # 64x128 -&gt; m&lt;n -&gt; Right Inverse -&gt; w^-1 = w^T (ww^T)^-1
    s2 = torch.matmul(w.transpose(0,1), torch.matmul(w, w.transpose(0,1)).inverse()) #128x 64 
    z = torch.matmul(s2, s1).squeeze()
    a3 = model(z)
</code></pre>
<p>The problem is when I compute the l2 dist between z (original vector) and a3 (vector that was reproduced via inverting and then forward proping), I get a large distance. (That is torch.dist(z, a3,2) is approx 1.5).</p>
<p>This indicates that I am doing something wrong in the inversion but I couldn’t find the cause.</p>
Setting lower bound on learning rate in torch.optim.lr_scheduler.StepLR

farah: <p>Hi!</p>
<p>I was trying to find if I can set lower bound on learning rate in <code>torch.optim.lr_scheduler.</code> <code>StepLR</code> so that it will not reduce more on step().</p>
<p>I was wondering if I could do that by setting a value in argument. I didn’t find available argument for that. Would I have to do it by manually checking params[‘lr’]?</p>
<p>Many thanks.</p>
Difference between `matmul` broadcast and `bmm` on computational graph

Ricardo_Gama: <p>Hello,</p>
<p>I’m performing a matrix multiplication using <code>matmul</code> function:</p>
<pre><code class="lang-auto">hidden_size = 8
batch_size  = 5

W = Var(hidden_size,hidden_size)
emb = torch.tensor(torch.randn(batch_size,12,hidden_size))

res =  emb.matmul(W).bmm(emb.transpose(1, 2))
</code></pre>
<p>The first <code>matmul</code> fuction just broadcast the operation within the batch dimensions and the result is as expected.</p>
<p>Another way of accomplishing this is using <code>bmm</code>:</p>
<pre><code class="lang-auto">Wb = W.expand(batch_size,  -1,-1)
resb = emb.bmm(Wb).bmm(emb.transpose(1, 2))
</code></pre>
<p>both results are equal<br>
<code>torch.norm(res2-res)</code><br>
but while training my model, the loss has different training values using the first or the second method.</p>
<p>I’m guessing that the two methods generate different computational graphs.<br>
Is it?<br>
If so, what is the correct way of doing this?</p>
<p>Thank you in advance,<br>
Ricardo</p>

albanD: <p>Hi,</p>
<p>These two should be equivalent even if they define different computational graphs (potentially doing broadcasting in a different way).<br>
That means that, during training, because of floating point precision, they can end up giving noticeably different results as errors are amplified by the training.</p>
<p>Do you have a small code sample to reproduce the behavior you present?</p>

Ricardo_Gama: <p>Hello <a class="mention" href="/u/alband">@albanD</a>!</p>
<p>Thank you for your quick reply.</p>
<p>Unfortunately I do not have a simple code showing the behavior. I only have a working version of a more developed model.<br>
I will try do make one and post it here as soon as possible.</p>
<p>Mean while, in your opinion, what function should normally be used in this case?</p>
<p>Best regards,<br>
Ricardo</p>

albanD: <p>Both will be correct. You can see changing from one to the other having the same effect as changing the random seed that you set at the beginning of your script: all the numbers you will get will be different but if your model is robust, both should converge to a similar solution in terms of performance.</p>

alwynmathew: <p>What is the reason behind having both <code>matmul</code> and <code>bmm</code> potentional doing the exact same task? What exactly differ from one another? Is there any advantage in using one over another?</p>

albanD: <p><code>bmm</code> is the simple batch matrix matrix multiply.<br>
<code>matmul</code> is more general as depending on the inputs, it can correspond to <code>dot</code>, <code>mm</code> or <code>bmm</code>.</p>

farah: <p>Hi! Does matmul have higher fp precision than bmm? Thanks!</p>

albanD: <p>Hi,</p>
<p>They both have the same precision which will depend on the dtype of the input you give it.</p>

farah: <p>ok thanks for response.</p>
How to access the tensor of one particular convolutional kernel?

1315933876: <p>I am using Pytorch to train ResNet18 on CIFAR10 dataset.<br>
My question is , How to access the weights of one particular convolutional kernel of particular layer?<br>
It seems that “net.module.layer1[0].conv1.weight” is a struct, I merely want to get the tensor corresponding to this struct. The following are printed  after this command “net = ResNet18()” and “print(net.module.layer1[0].conv1.weight)”. Thank you very much.</p>
<aside class="quote no-group">
<blockquote>
<p>Parameter containing:<br>
tensor([[[[ 0.0239, -0.0199, -0.0045],<br>
[ 0.0490,  0.0366,  0.0082],<br>
[ 0.0374,  0.0352,  0.0106]],</p>
<pre><code>     .... ...

     [[ 0.0250,  0.0002,  0.0127],
      [-0.0045,  0.0197, -0.0044],
      [ 0.0214,  0.0195,  0.0011]]]], device='cuda:0', requires_grad=True)
</code></pre>
</blockquote>
</aside>

1chimaruGin: <aside class="quote" data-post="6" data-topic="110100">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1chimarugin/48/27435_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-access-the-weights-of-one-particular-convolutional-kernel/110100/6">How to access the weights of one particular convolutional kernel?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Add .data 
weight_data = net.module.layer1[0].conv1.weight.data

print(weight_data.shape)
print(weight_data.dtype)
  </blockquote>
</aside>