akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get a batch that has images of the same class

username: hrushi
cooked: <p>I would like to generate batches in such a way that all the datapoints/images in the batch belong to the same class. For example if batch size is set 32 and considering MNIST dataset, the batch should have images of the same class, say all ‘0’ or all ‘1’.<br>
The reason I want to do this is as part of research</p>
<p>I have written the following code to get the batch in the required way. however training is not taking place(the training error was actually increasing with epochs):</p>
<pre><code class="lang-auto"># Generate dataloaders for each class

mnist_train = datasets.MNIST("./data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = datasets.MNIST("./data", train=False, download=True, transform=transforms.ToTensor())

train_loader = DataLoader(mnist_train, batch_size = 256, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size = 256, shuffle=False)

A = {}
B = {}
C = {}
# B and C are dictionaries that contain DataLoaders for each class

batch_size = 32

for i in range(0, 10):
    idx = mnist_train.train_labels==i
    A['mnist_train_{}'.format(i)] = datasets.MNIST("./data", train=True, download=True, transform=transforms.ToTensor())
    A['mnist_train_{}'.format(i)].data = mnist_train.data[idx]
    A['mnist_train_{}'.format(i)].targets = mnist_train.targets[idx]


    idx = mnist_test.test_labels==i
    A['mnist_test_{}'.format(i)] = datasets.MNIST("./data", train=False, download=True, transform=transforms.ToTensor())
    A['mnist_test_{}'.format(i)].data = mnist_test.data[idx]
    A['mnist_test_{}'.format(i)].targets = mnist_test.targets[idx]

    B['train_loader_{}'.format(i)] = DataLoader(A['mnist_train_{}'.format(i)], batch_size = batch_size)
    C['test_loader_{}'.format(i)] = DataLoader(A['mnist_test_{}'.format(i)], batch_size = batch_size)

</code></pre>
<p>I use the above dataloaders in the following training function:</p>
<pre><code class="lang-auto">def epoch_all(loaders, model, opt=None):
  total_loss, total_err = 0., 0.
  for loader in loaders.values():
    for X,y in loader:
      X,y = X.to(device), y.to(device)
      yp = model(X)
      loss = nn.CrossEntropyLoss()(yp,y) 
      if opt:
        opt.zero_grad()
        loss.backward()
        opt.step()
      
      total_err += (yp.max(dim=1)[1] != y).sum().item()
      total_loss += loss.item() * X.shape[0]

  if opt:
    return total_err / 60000, total_loss / 60000
  else:
    return total_err / 10000, total_loss / 10000
</code></pre>
<p>The above function is run using:</p>
<pre><code class="lang-auto">    for t in range(10):
        train_err, train_loss = epoch_all(B, model_cnn, opt)
        test_err, test_loss = epoch_all(C, model_cnn)
</code></pre>
<p>Would be grateful for any help</p>
Can not use a model inside a subprocess if used a model in main process

username: shpp
cooked: <p>Hi!<br>
I am trying to run inference on 2 models in parallel - 1 in main process and 1 in subprocess. If i run the model just in subprocess (or subprocesses) it works fine. Here is a simple example:</p>
<pre><code class="lang-auto">from torch.multiprocessing import Process, Queue
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv2d(3, 1, 3)

    def forward(self, x):
        return self.conv(x)

def parallel_model(inp, model):
    with torch.no_grad():
        output = model(inp)
    print(output.shape)

if __name__ == '__main__':
    
    model = Net()
    model.eval()

    inp = torch.rand(1,3,128,128)

    p = Process(target=parallel_model, args=(inp, model))
    p.start()
    p.join()
    
</code></pre>
<p>However if i firstly run an inference inside main process (even if i use a different model), my model inside the suprocess will not execute inference and will just get stuck on the line of inference. Here is an example when model in subprocess doesn’t execute:</p>
<pre><code class="lang-auto">from torch.multiprocessing import Process, Queue
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv2d(3, 1, 3)

    def forward(self, x):
        return self.conv(x)

def parallel_model(inp, model):
    with torch.no_grad():
        output = model(inp)
    print(output.shape)

if __name__ == '__main__':
    
    model = Net()
    model.eval()
    
    model_1 = Net() # Add model
    model_1.eval()

    inp = torch.rand(1,3,128,128)

    with torch.no_grad():   # Score with new model
        print(model(inp))

    p = Process(target=parallel_model, args=(inp, model))
    p.start()
    p.join()
    
    
</code></pre>
<p>I have tried to debug it and inside the subprocess you can successfully print both model and input. When you get into the forward method of the model the process just drops.<br>
Can somebody please explain why this works this way? And can i somehow fix it?</p>
Weight update after gradient flow

username: cbd
cooked: <p>In code below output = w * x which is given for loss calculation. So after loss.backward(), change in weight should be " w = w  - (lr * dL/dw) ". Now dL/dw = dL/d(output) * d(output)/dw= -1 * x = -4.<br>
Putting this value it should update w = w  - (lr * dL/dw) = 1 - (0.2* -4) = 1.8 but its 1.2 and update constant like 1.2,1.4.1.6 etc</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
class Net(torch.nn.Module):
    def __init__(self):       
        super(Net, self).__init__()
        self.w=torch.nn.Parameter(torch.ones(1))
    def forward(self, x):
        print("weight and its gradient is ",self.w,self.w.grad)
        out = torch.mul(x,self.w)
        return out
learning_rate = 0.2
model=Net()

criterion=torch.nn.L1Loss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

x=4*torch.ones((1),requires_grad=True)
target=torch.FloatTensor([36])

for i in range (1,3):
    optimizer.zero_grad()
    out=model(x)
    loss= criterion(target,out)
    print("output and loss is ",out,loss)
    loss.backward(retain_graph=True)
    optimizer.step() 
</code></pre>

username: albanD
cooked: <blockquote>
<p>change in weight should be " w = w - (lr * dL/dw) "</p>
</blockquote>
<p>The Adam optimizer that you’re using is not doing such a simple update. There are a couple momentum and scaling terms that are used.<br>
To get this step, you should use SGD with no momentum.</p>

username: cbd
cooked: <p>Ok. I got it. Thanks</p>
STL10 `unlabeled` split has labels

username: ariG23498
cooked: <p>I am curious to know why the labels in the unlabeled split of STL10 are returned from the dataset.</p>
<p>I can suppress the labels in the training loop, but does it not intuitively make more sense to not send the labels in the <code>unlabeled</code> split of STL10?</p>
<pre><code class="lang-python">&gt;&gt;&gt; unlabeled_ds = STL10(
    root=".",
    split="unlabeled")

&gt;&gt;&gt; unlabeled_ds.classes[unlabeled_ds[0][1]]
"truck"
</code></pre>
STL10 fails to load unlabeled data

username: Loerbri
cooked: <p>Hello,</p>
<p>I am trying to train a model on the unlabeled split of the STL10 dataset. When running the following code I run into an error:</p>
<pre><code>stl_transforms = transforms.Compose([transforms.ToTensor(),
                                    transforms.ToPILImage(),
                                    transforms.ToTensor()])

unsupervised_data = datasets.STL10(root = '../data', split =  'unlabeled', transform=stl_transforms, download=True)

unsupervised_train_loader = torch.utils.data.DataLoader(unsupervised_data,
                             batch_size=args.batch_size, shuffle=True)

for batch_idx, (data, _) in enumerate(unsupervised_train_loader):
</code></pre>
<p>In this case I get: TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class ‘NoneType’&gt;</p>
<p>Interestingly, if I set split to ‘train’ or ‘train+unlabeled’ , everything works well.</p>
<p>Any help would be appreciated!</p>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/loerbri">@Loerbri</a><br>
I have tried the code snippet out. I do not seem to have a problem with the working.</p>
<p>Would you mind sharing a colab notebook where the issue is reproducible?</p>
Having trouble understanding how torch.transpose is behaving

username: Mohamad_Bairakdar
cooked: <p>The following code gives an error that the dimensions for the matrices in spmm do not match up</p>
<pre><code class="lang-auto">def sgc_precompute(features, adj, degree):
    t = perf_counter()
    for i in range(degree):
        features = torch.spmm(adj, torch.transpose(features, 1,0))
    precompute_time = perf_counter()-t
    return torch.transpose(features, 1, 0), precompute_time
</code></pre>
<p>I’m confused because the following code works. Notice that the only change was that I explicitly set <code>features</code> to <code>torch.transpose(features, 1,0)</code> and then passed features as an argument to spmm, instead of just passing in <code>torch.transpose(features, 1,0)</code> directly</p>
<pre><code class="lang-auto">def sgc_precompute(features, adj, degree):
    t = perf_counter()
    features = torch.transpose(features, 1, 0)
    for i in range(degree):
        features = torch.spmm(adj, features)
    precompute_time = perf_counter()-t
    return torch.transpose(features, 1, 0), precompute_time
</code></pre>

username: Mohamad_Bairakdar
cooked: <p>Nvm the problem was with my IDE</p>
1x1 Convolutional filter

username: MauroGentile
cooked: <p>Hello,<br>
I have a Nx1xHxW feature maps. I need to add a second head that generates Nx3xHxW representing pixel wise regression with a triplet for each pixel .</p>
<p>The question is: how would you go from Nx1xHxW to Nx3xHxW? A fully connected layer would be too expensive in terms of introduced parameters.</p>
<p>That’s what I am trying a 1x1x3 convolutional filter with stride 1 defined as nn.Conv2d(1, 3, (1, 1), stride=1, bias=True) but results does not seem encouraging. Any suggestion would be welcome.</p>
<p>Best</p>
<p>Mauro</p>
Computing gradients with respect to weights in custom Conv2d

username: Luesch
cooked: <p>hello,</p>
<p>Im currently working on implementing Conv2d from scratch without using autograd. So i already got the forwardpass working using unfold. But when it comes to the backwardpass something goes wrong. As i understood it after you unfold your input and kernel the whole steps for the forwardpass and backwardpass are basically the same as in normal linear networks. But if i try to implement it like this i get weird shapes. Heres the code im currently using.</p>
<pre><code class="lang-auto">import torch
# as an input im using a tensor with the size of a mnist digit
img = torch.randn(1 ,1 ,28 ,28)
# kernel with 1 input dim and 1 output dim
kernel = torch.randn(1 ,1 ,3 ,3)
# unfold with kernel size 3 and padding 1
unfold = torch.nn.Unfold(kernel_size=(3,3) ,padding=1)
# unfolded data
img_unfolded = unfold(img)
# forwardpass with kernel and unfolded data
# reshape kernel to (out_dim ,-1)
output = kernel.view(1 ,-1).matmul(img_unfolded)
# output with shape (1 ,1 ,784) (then youd reshape to 1,1,28,28)
# to have the right dimensions for the next layer
</code></pre>
<p>now assume that the gradients of the outputlayer are already computed as  randn(1 ,1 ,784)</p>
<pre><code class="lang-auto"># gradients already calculated and with shape of the output layer
grads_output = torch.randn(1 ,1 ,784)
# so now how to i get to the gradients with respect to the weights? 
# I tried it with img_unfolded transposed
# times output but the shape is just weird
grads_weights = img_unfolded.T.matmul(output)
# shape is (784, 9, 784) which is clearly wrong becuase the 
# right shape would be (1,9)
</code></pre>
<p>but when i try to compute the gradients of the next layers output the result is as im expecting it</p>
<pre><code class="lang-auto"># transposed reshaped kernel times the gradients of the output
grads_next_layer = kernel.view(1 ,-1).T.matmul(grads_output)
# as said it has the right shape of (1 ,9 ,784)
</code></pre>
<p>so my question is how do i correctly calculate the gradients with respect to the weights?</p>
<p>thanks for taking your time to read this and have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: googlebot
cooked: <blockquote>
<p>img_unfolded.shape<br>
Out[13]: torch.Size([1, 9, 784])<br>
img_unfolded.T.shape<br>
Out[14]: torch.Size([784, 9, 1])</p>
</blockquote>
<p>so, .T does the wrong permutation</p>
<p>gradient of matmul(kernel, img_unfolded) wrt kernel is grads_output.mm(img_unfolded.t()), 3d version of which should use shapes (1,1,784) @ (1, 784, 9)</p>
<p>I’d suggest using .transpose(-2,-1)</p>

username: Luesch
cooked: <p>Thanks so much for the solution! Have a nice day</p>
Bug of Conv3d on P100 GPU

username: AoChen
cooked: <p>Hi,</p>
<p>When I feed an input with batch size N &gt; 65535 into Conv3d layers, the output after index 65535 is obviously incorrect. This happens only on P100 GPU, not on CPU or other GPU. Possibly it also happens on P4 GPU according to my previous tests, but now I don’t have P4 GPU in hand to test again. It happens only in Conv3d layer, not in Conv1d or Conv2d.</p>
<p>code, run in colab</p>
<pre><code class="lang-auto">import torch
from torch import nn

net = nn.Conv3d(1, 1, 1, bias=False)
input = torch.rand(70000, 1, 2, 2, 2)
out_cpu = net(input).cuda()

net.cuda()
out_gpu = net(input.cuda())

error = torch.sum((out_cpu - out_gpu).detach()**2, dim=(1,2,3,4))
print(error[65500:65600])
</code></pre>
<p>The output on P100 is</p>
<pre><code class="lang-auto">tensor([0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
        0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
        0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
        0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
        0.2917, 0.4172, 0.7399, 0.4553, 0.3345, 0.8831, 0.3979, 0.5914, 0.5503,
        0.5707, 0.5280, 0.7113, 0.5516, 0.8872, 0.6879, 0.4335, 0.7914, 0.4365,
        0.2578, 0.2922, 0.1646, 0.7618, 0.5094, 0.3610, 0.6823, 0.8531, 0.6192,
        0.3508, 0.2554, 0.9788, 0.3178, 0.6107, 0.2074, 1.0488, 0.2410, 0.1997,
        0.8527, 0.5012, 0.3539, 0.6145, 0.4775, 0.5919, 0.7322, 0.6376, 0.5392,
        0.6394, 0.5922, 0.6976, 0.4430, 0.4933, 0.5123, 0.3211, 0.2196, 0.6387,
        0.2673, 0.1693, 0.2910, 0.4832, 0.3100, 0.4031, 0.3633, 0.5821, 0.4544,
        0.2899], device='cuda:0')
</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce it with a current master build on a P100. Which PyTorch version are you using?</p>

username: AoChen
cooked: <p>Thanks for your reply.<br>
The pytorch version is 1.8.1+cu101</p>

username: ptrblck
cooked: <p>Thanks for the information! How did you install the CUDA10.1 version, as the <a href="https://pytorch.org/get-started/locally/">install instructions</a> mention CUDA10.2 or CUDA11.1?</p>

username: AoChen
cooked: <p>I didn’t install it manually. It’s prepared on colab.</p>

username: AoChen
cooked: <p>By the way, the output is correct when group &gt;= 2 in Conv3d layer</p>

username: AoChen
cooked: <p>Thanks for your previous reply. It inspired me to check the cuda version in colab. The cuda version I get in colab is 11.2. The problem occurs when I use PyTorch version 1.8.1+cu101 or 1.8.1+cu102, while it disappears in 1.8.1+cu111.</p>

username: ptrblck
cooked: <p>Yeah, that was what I wanted to suggest next and I also verified that it’s not showing any errors in 11.1.</p>
Train two models in sequence simulteneously

username: Saikat_Bhattacharya
cooked: <p>I am trying to train a embedder. So, I have an architecture for the model to embed texts. And I have another model architecture that will take the inputs from the output of the first model and predict the label.<br>
Now at the end of the training I want the first model to be saved as a pre-trained embedding model.<br>
So, the pseudo code is as follows -<br>
Model1:<br>
input: text<br>
… some BiLSTM , Dense layers…<br>
output: n-dimensional embedding</p>
<p>Model2:<br>
input: embedding1, embedding2<br>
find the similarity of texts<br>
run a linear layer + sigmoid<br>
output: similarity score</p>
<p>Now I want to use a dataset with the following structure - (text1, text2, similarity). I want to train the models and save Model1 as a pretrained model to use later.</p>
<p>How can I train these two with PyTorch? How shall I write the optimizer here?</p>
Programming pytorch with DL book errors

username: harikanth
cooked: <p>Hello,</p>
<p>I am trying to follow “Programming pytorch for deep learning” book &amp; having hurdles every now &amp; then.<br>
I spinned up a AWS Dlami instance but never got to SSH from local machine and quit like 2-3 times.</p>
<p>I got to next step the third time only to fail again in chapter one to download the dataset from given excel file(a python script is given by author to download the Training:Test dataset of fish:cat.<br>
I quit again that time like a month ago.<br>
Got a bill of 60$ from AWS <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Today, i started again from scratch. Spinned up a Colab google notebook for free so far <img src="https://discuss.pytorch.org/images/emoji/apple/crossed_fingers.png?v=9" title=":crossed_fingers:" class="emoji" alt=":crossed_fingers:"><br>
This time i got an error right out of bat at<br>
tensor.ones(1,2) as<br>
“tensor not defined”<br>
imported tensor from torch only to get a different error.<br>
“Tensor doesn’t have ones attribute”<br>
I Understand that Tensor function doesnt have ones method in it as per the error.<br>
Is there a fix(i didn’t check the tensor function for the method) for this error?</p>
<p>Anybody else who finished the “Programming pytorch with DL book”?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p><code>tensor.ones</code> is not a valid method and you would most likely want to use <code>torch.ones</code> instead.</p>

username: harikanth
cooked: <p>Yes, that worked. The book Programming PyTorch has tensor.ones in examples. May be its a Typo.<br>
Thanks.</p>
There is maybe wrong code in pytorch tutorial

username: meeng
cooked: <p>Hi,</p>
<p>While studying, I found out that there was something wrong.</p>
<p>It seems to be showing the wrong result.</p>
<p>Code is fine, but result is not proper.</p>
<p>How do I fix it?</p>
<p><a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a></p>
<h2><a name="speed-up-by-pipelining-inputs-1" class="anchor" href="#speed-up-by-pipelining-inputs-1"></a>Speed Up by Pipelining Inputs</h2>
<p>in “GETTING STARTED WITH DISTRIBUTED DATA PARALLEL”</p>
<pre><code class="lang-auto">class PipelineParallelResNet50(ModelParallelResNet50):
    def __init__(self, split_size=20, *args, **kwargs):
        super(PipelineParallelResNet50, self).__init__(*args, **kwargs)
        self.split_size = split_size

    def forward(self, x):
        splits = iter(x.split(self.split_size, dim=0))
        s_next = next(splits)
        s_prev = self.seq1(s_next).to('cuda:1')
        ret = []

        for s_next in splits:
            # A. s_prev runs on cuda:1
            s_prev = self.seq2(s_prev)
            ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

            # B. s_next runs on cuda:0, which can run concurrently with A
            s_prev = self.seq1(s_next).to('cuda:1')

        s_prev = self.seq2(s_prev)
        ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

        return torch.cat(ret)


setup = "model = PipelineParallelResNet50()"
pp_run_times = timeit.repeat(
    stmt, setup, number=1, repeat=num_repeat, globals=globals())
pp_mean, pp_std = np.mean(pp_run_times), np.std(pp_run_times)

plot([mp_mean, rn_mean, pp_mean],
     [mp_std, rn_std, pp_std],
     ['Model Parallel', 'Single GPU', 'Pipelining Model Parallel'],
     'mp_vs_rn_vs_pp.png')
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c421b0565be11a57bb27f23440d16e94d67d1cb5.png" alt="mp_vs_rn_vs_pp" data-base62-sha1="rZ3LfLLwkYhVKq1VOFQfmxIAaBn" width="432" height="288"></p>
A error between `torch.cdist` and manually unsqueezing tensors

username: PannenetsF
cooked: <p>As I run some code like these which do the same thing $-\sum ||x-w||$:</p>
<pre><code class="lang-python">def via_unsqueeze(W_col, X_col):
    output = -(W_col.unsqueeze(2) - X_col.unsqueeze(0)).abs().sum(1)
    return output 
</code></pre>
<p>and</p>
<pre><code class="lang-python">def via_cdist(W, X):
    output = -torch.cdist(W, X, 1)
    return output
</code></pre>
<p>Sometimes, there will be a difference up to 2e-6 when the X, W are under the normal distribution.</p>
<p>Then check them via a test:</p>
<pre><code class="lang-python">def a(): 
    W = torch.rand(1000,300) 
    X = torch.rand(1000,300) 
    return (via_unsqueeze(W,X) - via_cdist(W,X)).abs() 
</code></pre>
<p>The output could be like</p>
<pre><code class="lang-auto">tensor([[0.0000e+00, 3.8147e-05, 2.2888e-05,  ..., 2.2888e-05, 3.8147e-05,
         1.5259e-05],
        [3.0518e-05, 1.5259e-05, 0.0000e+00,  ..., 0.0000e+00, 3.0518e-05,
         2.2888e-05],
        [3.0518e-05, 6.8665e-05, 1.5259e-05,  ..., 7.6294e-06, 1.5259e-05,
         7.6294e-06],
        ...,
        [7.6294e-06, 1.5259e-05, 1.5259e-05,  ..., 3.0518e-05, 1.5259e-05,
         3.8147e-05],
        [3.0518e-05, 7.6294e-06, 2.2888e-05,  ..., 4.5776e-05, 7.6294e-06,
         7.6294e-06],
        [3.0518e-05, 7.6294e-06, 3.8147e-05,  ..., 1.5259e-05, 3.8147e-05,
         3.0518e-05]])
</code></pre>
<p>So I am wondering HOW THIS HAPPENED and how to fix it?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>The difference is most likely caused by the limited floating point precision and a different order of operations. You could use <code>torch.float64</code> as the data type in case you need to increase the precision, but note that this data type would cause a performance hit on GPUs.<br>
A simple example is given here:</p>
<pre><code class="lang-python">x = torch.randn(100, 100)
sum1 = x.sum()
sum2 = x.sum(0).sum(0)
print((sum1 - sum2).abs().max())
&gt; tensor(2.2888e-05)

x = torch.randn(100, 100, dtype=torch.float64)
sum1 = x.sum()
sum2 = x.sum(0).sum(0)
print((sum1 - sum2).abs().max())
&gt; tensor(2.1316e-14, dtype=torch.float64)
</code></pre>

username: PannenetsF
cooked: <p>Thank you for the answer.</p>
Parameter sharing in conv2d kernel

username: blurry_mood
cooked: <p>Hi,<br>
I want to customize the kernel used in the conv2d layer in a way that shares parameters.<br>
In short, I want the kernel to be symmetric with respect to the row in the middle and to the column in the middle.<br>
If we take the example of 3 by 3 kernel. I want the third column to have the same parameters as the first one, and the third row to have the same params as the first one.</p>

username: ptrblck
cooked: <p>You could initialize the <code>nn.Parameter</code>s in the model’s <code>__init__</code> method, create the desired filter in the <code>forward</code> (e.g. via <code>torch.cat</code> and <code>torch.stack</code>) and apply it via the functional API (<code>F.conv2d</code> in your case).</p>

username: blurry_mood
cooked: <p>That worked! thanks.</p>
Models.features , model.encoder, and model.decoder

username: nicozhou
cooked: <p>I see a lot of examples using model.features</p>
<p><code>torchvision.models.vgg16(pretrained=pretrained).features</code></p>
<p>so, can we use <code>model.encoder</code> or <code>model.decoder</code> too? or it depends on the architecture of the model? where can we found the documentation of them?</p>

username: ptrblck
cooked: <p>It depends on the architecture of the model and you can check the internal modules in the model definition.<br>
E.g. <code>torchvision.models</code> are implemented <a href="https://github.com/pytorch/vision/tree/58afa5110ce65c718dea676eccc5783d7e56d07a/torchvision/models">here</a>.</p>
Accessing state of differentiable optimizers of Higher

username: TinfoilHat0
cooked: <p>The differentiable optimizers of Higher (<a href="https://github.com/facebookresearch/higher" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.</a>) don’t seem to have a state_dict() method like regular optimizers.</p>
<p>Can I still access, and save their states of somehow? I’d like to access them so I can load them to a regular optimizer later.</p>

username: ptrblck
cooked: <p>I’m not deeply familiar with <code>higher</code>, but it seems that the <code>DifferentiableOptimizer</code> base class accepts an already created <code>optimizer</code> object, so I would assume that you could still query the original <code>optimizer.state_dict()</code>? Could this be the case or are the original states stale?</p>

username: TinfoilHat0
cooked: <p>Well, the higher context looks like the following:</p>
<pre><code class="lang-auto">with higher.innerloop_ctx(model, opt) as (fmodel, diffopt):
    for xs, ys in data:
        logits = fmodel(xs)  # modified `params` can also be passed as a kwarg
        loss = loss_function(logits, ys)  # no need to call loss.backwards()
        diffopt.step(loss)  # note that `step` must take `loss` as an argument
</code></pre>
<p>As you can see, within this context, we’ve a differentiable optimizer and a model.<br>
Within the Higher context, we update only the state of <strong>fmodel</strong> and <strong>diffopt</strong>.<br>
At the end of the Higher context, <em>I just want to set my model to fmodel, and my optimizer to diffopt.</em></p>
<p>I can do <strong>model.load_state_dict(fmodel.state_dict())</strong> for the model, but there doesn’t seem to be a state_dict() method for diffopt. This makes optimizing with stateful optimizers such as Adam a problem to me.</p>
What is difference between 'Tensor' and 'torch.Tensor'? : TypeError: pic should be Tensor or ndarray. Got <class 'torch.Tensor'>

username: creatives07
cooked: <p>I tried to modify the code to using multiple gpu. The code I fixed is as follows.</p>
<pre><code>  netG_A2B = Generator_S2F(opt.input_nc, opt.output_nc)  # shadow to shadow_free
  netG_B2A = Generator_F2S(opt.output_nc, opt.input_nc)  # shadow_free to shadow
  netD_A = Discriminator(opt.input_nc)
  netD_B = Discriminator(opt.output_nc)

  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

  if (device.type == 'cuda') and (torch.cuda.device_count() &gt; 1):
      print('Multi GPU')

   netG_A2B = torch.nn.DataParallel(netG_A2B)  # shadow to shadow_free
   netG_B2A = torch.nn.DataParallel(netG_B2A)  # shadow_free to shadow
   netD_A = torch.nn.DataParallel(netD_A)
   netD_B = torch.nn.DataParallel(netD_B)


  netG_A2B.to(device)
  netG_B2A.to(device)
  netD_A.to(device)
  netD_B.to(device)
</code></pre>
<p>original code is as follows.</p>
<pre><code>  netG_A2B = Generator_S2F(opt.input_nc, opt.output_nc)  # shadow to shadow_free
  netG_B2A = Generator_F2S(opt.output_nc, opt.input_nc)  # shadow_free to shadow
  netD_A = Discriminator(opt.input_nc)
  netD_B = Discriminator(opt.output_nc)

  if opt.cuda:
      netG_A2B.cuda()
      netG_B2A.cuda()
      netD_A.cuda()
      netD_B.cuda()
</code></pre>
<p>and i got errors, the Traceback is this.</p>
<p>Traceback (most recent call last):<br>
File “train_Mask-ShadowGAN.py”, line 207, in <br>
print(mask_generator(real_A, fake_B))<br>
File “/home/sonic/sw/Mask-ShadowGAN/utils.py”, line 39, in mask_generator<br>
im_f = to_gray(to_pil(((shadow_free.data.squeeze(0) + 1.0) * 0.5).cpu()))<br>
File “/home/sonic/.local/lib/python3.5/site-packages/torchvision/transforms/transforms.py”, line 110, in <strong>call</strong><br>
return F.to_pil_image(pic, self.mode)<br>
File “/home/sonic/.local/lib/python3.5/site-packages/torchvision/transforms/functional.py”, line 103, in to_pil_image<br>
raise TypeError(‘pic should be Tensor or ndarray. Got {}.’.format(type(pic)))<br>
TypeError: pic should be Tensor or ndarray. Got &lt;class ‘torch.Tensor’&gt;.</p>
<p>I just modified the code and increased the batch size from 1 to 2.<br>
What is difference between ‘Tensor’ and ‘torch.Tensor’?<br>
and Why does the error occur?<br>
Thank you.</p>

username: ptrblck
cooked: <p>There shouldn’t be a difference and the error message seems to be misleading.<br>
Could you post the used transformations and the inputs you are passing to them?</p>
How to manage(free and malloc) tensor's memory by ourself

username: new_pytorcher
cooked: <p>Hello, I want to manage the training by myself, including layer by layer training and management of intermediate tensors (feature map).</p>
<p>I try to use pytorch to enable large model training by swapping some tensor to CPU and swap them back when needed. However, I find it is hard to access the feature map.<br>
By <a href="https://discuss.pytorch.org/t/how-to-split-backward-process-wrt-each-layer-of-neural-network/7190" class="inline-onebox">How to split backward process wrt each layer of neural network?</a>, I can access each layer’s forward and backward. But I find I can’t release the memory of tensors, can anyone help me about this?<br>
Thanks a lot!</p>

username: ptrblck
cooked: <p>Your use case sounds similar to e.g. <code>DeepSpeed</code> and you could check their <a href="https://github.com/microsoft/DeepSpeed/blob/ab5534fc4c0f8ca21ada321f9730d723aa31288b/deepspeed/runtime/zero/contiguous_memory_allocator.py">allocator implementation</a>, which also moves data around to free device memory.</p>
Nightly version LSTSQ broken

username: JunzheJosephZhu
cooked: <p>Error message below:<br>
################################################<br>
input.size(-2) == other_2d.size(-2)INTERNAL ASSERT FAILED at “/opt/conda/conda-bld/pytorch_1619593671510/work/aten/src/ATen/native/BatchLinearAlgebra.cpp”:3111, please report a bug to PyTorch.<br>
##############################################<br>
Apparently this is broken, as I switched to pinv and it worked just fine.</p>

username: ptrblck
cooked: <p>Could you post a minimal code snippet to reproduce this error as well as the output of <code>python -m torch.utils.collect_env</code>, please?<br>
In case you are using an older PyTorch release, could you update to the latest one and check, if you are still facing this error?</p>
Does pytorch support tools to get a quant performance with a fp32 model

username: fwz-opt
cooked: <p>If I have a pytorch script model with fp32 datatype.<br>
I want to measure the quant performance on modile with qnnpack( just use this fp32 model but choose to use int8 as inference datatype).<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"> I just want to know with the same net architecture, the performance difference between fp32 and int8.<br>
Does pytorch has this kind of tools? Like TensorRT trtexec --int8 with fp32 model</p>

username: supriyar
cooked: <p>This thread might be useful for you <a href="https://discuss.pytorch.org/t/speed-benchmarking-on-android/67340" class="inline-onebox">Speed benchmarking on android?</a></p>
<p>Please reach out to the mobile team if this script doesn’t work as expected.</p>
How to convert torch_geometric Data to bytes-like or how to load them faster?

username: Kinny_Fly
cooked: <p>Hello! I’m working on GCN recently, I build a dataset myself, consisting of about 150,000 datas with a data type of torch_geometric Data. Specifically, I rewrite Data class simply and add another attribute z which is a extra target as my model have two targets:</p>
<pre><code class="lang-auto">from torch_geometric.data import Data
class MyData(Data):
    def __init__(self,x,y,z,edge_index,pos):
        super(MyData, self).__init__(x=x,y=y,edge_index=edge_index,pos=pos)
        self.z = z
</code></pre>
<p>I save my datas(named as sample) one by one simply using torch.save:</p>
<pre><code class="lang-auto">torch.save(sample, 'new_graph_data/{}.pt'.format(str(image_id)+'_'+str(gt_id)))
</code></pre>
<p>Therefore, I have got 150,000 pt-files, leading to EXTREMELY slow speed when I try to loading my datas in a traditional way:</p>
<pre><code class="lang-auto">all_data_path = os.listdir('F:/CODE/Pytorch/Diplomer/new_graph_data/')
train_ = 5000 # for testing
test_ = 6000
train_data_path = all_data_path[:train_]
test_data_path = all_data_path[train_:test_]

class MyOwnDataset(Dataset):
    def __init__(self, root, mode):
        super(MyOwnDataset, self).__init__()
        self.root = root
        self.mode = mode

    def get(self, idx):
        if self.mode == 'train':
            data = torch.load('F:/CODE/Pytorch/Diplomer/new_graph_data/'+train_data_path[idx])
        elif self.mode == 'test':
            data = torch.load('F:/CODE/Pytorch/Diplomer/new_graph_data/'+test_data_path[idx])
        return data
            
    def len(self):
        if self.mode == 'train':
            return train_
        elif self.mode == 'test':
            return test_-train_
</code></pre>
<p>After researching, I found that the troublesome slow speed results from the loading of huge amount of small files(for I save my pt-file one by one).So I decide to use LMDB to load my files.</p>
<pre><code class="lang-auto">txn = env.begin(write=True)
tqdm_iter = tqdm(enumerate(zip(all_file_list, keys)), total=len(all_file_list), leave=False)
for idx, (path, key) in tqdm_iter:
    tqdm_iter.set_description('Write {}'.format(key))  
    key_byte = key.encode('ascii')
    data = torch.load(path)
    txn.put(key_byte, data) 
</code></pre>
<p>But it’s known that LMDB requires a data type of bytes, I have no idea how to convert them.<br>
Can u give me some advice?<br>
Or maybe u have other efficient ways to load geometric Data? Any ideas can help me load faster?<br>
Thank u so much!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Teacher forcing for training and predicting with a LSTM

username: numpy
cooked: <p>Hello everyone,</p>
<p>I am very new to pytorch, so sorry if it’s trivial but I’m having some issues.</p>
<p>I have longitudinal data and I would like to train a recurrent neural network (let’s say an LSTM) for a classification task. Given the nature of the data, I’m allowed to use the true labels from the past in order to predict the present (which is usually not the case, like for machine translation for instance). Thus, I would like to perform teacher forcing on both the training phase and the prediction phase.</p>
<p>Inspired by this <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html" rel="nofollow noopener">tutorial</a>, I have the code below:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


class LSTMTagger(nn.Module):

    def __init__(self, n_hidden_features, n_features, n_classes):
        super(LSTMTagger, self).__init__()

        self.lstm = nn.LSTM(n_features, n_hidden_features)
        self.hidden2tag = nn.Linear(n_hidden_features, n_classes)

    def forward(self, sample):
        lstm_out, _ = self.lstm(sample.view(len(sample), 1, -1))
        class_space = self.hidden2tag(lstm_out.view(len(sample), -1))
        class_scores = F.log_softmax(class_space, dim=1)
        return class_scores
</code></pre>
<p>As it is, I don’t think that I perform teacher forcing at any moment since the <code>forward</code> method does not take as input the true labels. Thus, my question is: how can one perform teacher forcing for both training and prediction phases? Should I try to slightly change the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#LSTM" rel="nofollow noopener">torch.nn.LSTM</a> code? Any advice would be really appreciated.</p>

username: Konstantin_Solomatov
cooked: <p>When you perform training, to use teacher forcing, just shift expected values by one position and feed it back.</p>
<p>When you predict, you should store the hidden states of lstm, and feed them back position by position. nn.LSTM support this.</p>
<p>This sample might be helpftul: <a href="https://github.com/pytorch/examples/tree/master/word_language_model" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/word_language_model</a></p>

username: numpy
cooked: <p>Thank you, it is more clear right now.</p>

username: turian
cooked: <p><a class="mention" href="/u/konstantin_solomatov">@Konstantin_Solomatov</a> where is teacher forcing used in that code? I could not find it.</p>

username: turian
cooked: <p>This <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-the-model" rel="noopener nofollow ugc">pytorch tutorial</a> demonstrates it.</p>
Training on waveform vs spectrogram

username: Superklez
cooked: <p>Is there any difference in training on the spectrogram of a waveform as compared to training on the waveform itself? Does training on the waveform generate better results in general?</p>

username: ptrblck
cooked: <p>I think the difference would be quite large, as the sampling rate could be high in the time domain and make the training quite challenging. If I remember correctly, one way to make some models work on waveforms directly was to use a stack of conv layers with a specific dilation, so that the input size was feasible to process in the end.</p>

username: Superklez
cooked: <p>When getting the spectrogram of the waveform, is there any difference between <code>torchaudio.transforms.Spectrogram</code> and <code>torch.stft</code>? Also, is there an equivalent of <code>torch.istft</code> for <code>torchaudio.transforms.Spectrogram</code>?</p>

username: Mole_Turner
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - Would you happen to know any audio models that use this technique?</p>
<p>Also, is there any advantage to training on time domain data using dilated convolution over just training on time/frequency domain data such as STFT?</p>

username: ptrblck
cooked: <p>I was thinking about <a href="https://arxiv.org/abs/1609.03499">WaveNet</a> in my previous post.<br>
This paper is a bit older by now, so I guess there a new insights today about advantages and shortcomings of time- vs. frequency-domain approaches.</p>

username: Superklez
cooked: <p><a class="mention" href="/u/mole_turner">@Mole_Turner</a> A lot of the more recent models I’ve read about don’t resort to using a time-frequency (T-F) representation of the input wave signal (e.g. STFT representation). Also, I read in <a href="https://arxiv.org/ftp/arxiv/papers/1708/1708.07524.pdf" rel="noopener nofollow ugc">Supervised Speech Separation Based on Deep Learning: An Overview</a> that end-to-end speech separation methods like temporal mapping (which doesn’t require resorting to a T-F representation) has the following advantage:</p>
<blockquote>
<p>A potential advantage of this approach is to circumvent the need to use the phase of noisy speech in reconstructing enhanced speech, which can be a drag for speech quality, particularly when input SNR is low.</p>
</blockquote>
<p>and that</p>
<blockquote>
<p>As a convolution operator is the same as a filter or a feature extractor, CNNs appear to be a natural choice for temporal mapping.</p>
</blockquote>
Saving several NN into an octree file

username: Adrian_Briceno_Aguil
cooked: <p>Hello!<br>
My main problem is to save several NN into a octree structure. I have created a octree, with a very  simple definition :</p>
<pre><code class="lang-auto">class Tree(object):
    def __init__(self):
        self.childs = []
        self.model = None
        self.depth = None
        
        self.minRightValue = torch.ones(3)
        self.maxLeftValue = torch.ones(3)
    
    def createChildren(self,amount):
        for i in range(0,amount):
            self.childs.append(Tree())
</code></pre>
<p>Where tree.model is a trained Neural network.</p>
<p>This definition works quite well. However, what I want to do is to save the octree into one file where each child have is a trained NN.</p>
<p>The only idea I have for now is to save each NN with a complex naming convention to read each of them and create the octree in another computer. But I think there should be easier way to do it.</p>
<p>Any ideas? This is my first time using octrees, so if there is any recommendation I would love one.</p>
<p>Thank you!</p>
Autodiff support for im2col (unfold)

username: asrinivas
cooked: <p>My understanding is that <code>torch.nn.unfold</code> does the equivalent of <code>tf.extract_image_patches</code>. The <code>extract_image_patches</code> in TF doesn’t support gradients and backprop to my knowledge.</p>
<p>I was wondering what the status of <code>unfold</code> in PyTorch is - does it support gradients / autodiff right now? Has anyone used this successfully as part of an architecture?</p>
<p>In terms of speedup, how does it compare to explicitly extracting local windows in a DWise Conv like fashion? Example, Scott Gray has these kernels in his <code>blocksparse</code> repo, where I believe the idea is to run the convolutions at a block level and simulate the channel wise index gathering through a convolution primitive (which I guess implicitly calls <code>im2col</code> or whatever’s faster under the hood)- <a href="https://github.com/openai/blocksparse/blob/master/src/dev/spatial_conv.py" class="inline-onebox" rel="noopener nofollow ugc">blocksparse/spatial_conv.py at master · openai/blocksparse · GitHub</a></p>
<p>Edit: More specifically, for the speedup part,  if anyone has benchmarked implementing a regular <code>conv2d</code> with <code>unfold + matmul + fold</code> and compared the two on a GPU, that would be great to know!</p>
<p>Thanks!</p>

username: tom
cooked: <p>PyTorch backprops through nn.unfold/fold.</p>
<aside class="quote no-group" data-username="asrinivas" data-post="1" data-topic="119827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/asrinivas/48/37741_2.png" class="avatar"> asrinivas:</div>
<blockquote>
<p>More specifically, for the speedup part, if anyone has benchmarked implementing a regular <code>conv2d</code> with <code>unfold + matmul + fold</code> and compared the two on a GPU, that would be great to know!</p>
</blockquote>
</aside>
<p>The native aka THNN convolutions are implemented using im2col, so you can just disable CuDNN to benchmark whatever network you want. It uses a lot of memory and it won’t be fast. Part of that is the inefficiency in memory accesses when the im2col result is materialized in GPU memory (as opposed to living only in shared memory).</p>
<p>Best regards</p>
<p>Thomas</p>
How to disable create_graph=True outside the class where this command was used?

username: saul_santos
cooked: <p>I’m reusing a class where sometimes I need a graph to be created while in others I don’t need the graph to be created. I’m wondering if there is any PyTorch function that allows me to disable the command create_graph=True before I call that class without sending a Flag.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid create_graph is a boolean argument on the autograd APIs but that’s it <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Torch.where and autograd function

username: cbd
cooked: <p>I written 2 code for torch.where() as below. In first code it print “require_grad” True but in Second code when i write same in “torch.autograd.Function” it shows “require_grad” False. Why the behavior is different?</p>
<blockquote>
<p>First Code:<br>
import torch<br>
a=torch.zeros([1,5,5,10],device=0,requires_grad=True)<br>
b=torch.tensor([3],device=0, dtype=torch.float, requires_grad=True)<br>
c=torch.tensor([4],device=0, dtype=torch.float, requires_grad=True)<br>
d=torch.where(a&gt;200,b,c)<br>
print(" d grad is ",d.requires_grad)</p>
</blockquote>
<blockquote>
<p>Second Code:<br>
class fun_c(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(cxt,input):<br>
a=torch.zeros([1,5,5,10],device=0,requires_grad=True)<br>
a=torch.tensor([3],device=0, dtype=torch.float, requires_grad=True)<br>
b=torch.tensor([4],device=0, dtype=torch.float, requires_grad=True)<br>
d=torch.where(a&gt;200,b,c)<br>
print(" d grad is ",d.requires_grad)<br>
return d<br>
<span class="mention">@staticmethod</span><br>
def backward(ctx, grad_output):<br>
input, = ctx.saved_tensors<br>
grad_input = grad_output.clone()<br>
grad_input[input &lt; 0] = 0<br>
return grad_input<br>
hist=fun_c.apply<br>
a=hist(2)</p>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>During the forward pass, the Tensor created there have requires_grad=False because we don’t track gradients for them.<br>
The reason we don’t track gradients is because you provide a backward function that tells us how to compute them for that forward. So there is no need for the autograd to actually track that forward.</p>

username: cbd
cooked: <p>Ok i understood. So let say i want to extract some information from customize function and use that information for loss calculation. Can i pass the “grad_input=torch.ones(1,1,1,10) * grad_output”  instead of “grad_input[input &lt; 0] = 0” in backward function shown in Second code for input of shape torch.ones(1,1,1,10) in forward function so that loss will backpropagate and  network will converge?.</p>
<p>One more thing i want to add is in forward pass i extract the information using torch commands like “torch.where” and “torch.count_nonzero” and no differentiable operation. So could i pass the gradient information of next layer you can say which is loss to previous layer?</p>

username: albanD
cooked: <p>Yes, the backward should return a Tensor of the same size as the input containing the backpropagated gradients.</p>
Choosing an initial model architecture

username: Mole_Turner
cooked: <p>How do people go about deciding which model architecture to use if their problem isn’t well documented and therefore doesn’t really have a SOTA?</p>
<p>Even if I think “well okay lets try a CNN” I’m struggling to find (which might mean I haven’t looked hard enough) documentation that outlines how one goes about choosing initial hyperparameters, how many layers, which kind of layers etc.</p>
<p>For reference, I’m looking to input a stereo audio signal and transform it into a multichannel audio signal. So intuition plus some of the literature says I should be looking at CRNNs…but where to start with building one is something I’m struggling with. And whether that’s what I actually need.</p>
<p>When you do choose a model, at what point do you decide “yup this isn’t working I need to try something different”?</p>
Bounding boxes in training data altered to [x1,y1,w,h] after running evaluation

username: Josh33
cooked: <p>Hi all,</p>
<p>I’ve been following the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">Torchvision object detection finetuning tutorial</a> but with my own custom dataset and have been running into a strange issue. I’ve been using the tutorial code exactly as shown but with my own data. If I run it on the PennFudan dataset, everything works completely fine. If I run it on my dataset without evaluation, I don’t run into any problems either. But if I run it on my dataset with evaluation (on a test data loader that’s separate from my training data loader) it alters my <em>training</em> dataset so the bounding boxes in the labels are converted fro x1,y1,x2,y2 to x1,y1,width,height resulting in the error:<br>
ValueError: All bounding boxes should have positive height and width.</p>
<p>So evaluation is completed without any problems but when I return to training I get that error.</p>
<p>Does anybody have any insight to why this could be happening? Is it some problem with my dataset or perhaps something unique to the way evaluation is being run?</p>
<p>Thanks,<br>
Josh</p>

username: Josh33
cooked: <p>I just fixed this actually. I was originally instantiating my dataset class only once and subsetting my train and test datasets from there. Instantiating my dataset class twice (one object for my training set and one for my test set) fixed my issue. I guess some part of evaluation was altering the dataset object therefore changing my training dataset.</p>