akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Generate different Outputs from model during training & during eval

username: benihime91
cooked: <p>Hello .</p>
<p>How can I make my model output different results while model is in <code>train</code> mode and while <code>model</code> is in eval.<br>
For instance let’s say during <code>train</code> I would want my model to output the loss while during <code>eval</code> I would want my model to directly output the <code>results</code>.</p>
<p>I was looking for the functionality that present in the <code>fasterrcnn_resnet50_fpn</code> model wherein<br>
<code>The behavior of the model changes depending if it is in training or evaluation mode.</code></p>

username: mariosasko
cooked: <p>To achieve such behaviour, use the (special) <code>self.training</code> flag that is <code>True</code> when the model (the model has to subclass the <code>torch.nn.Module</code>) is in the train mode (<code>model.train()</code>) and <code>False</code> in the model’s eval mode (<code>model.eval()</code>).</p>
<p>And this is a skeleton of the forward pass of the model that outputs different results in the train/eval mode.</p>
<pre><code class="lang-python">def forward(self, x):
    ...
    if self.training:
        ...
        return train_output
    else:
        ...
        return eval_output
</code></pre>

username: benihime91
cooked: <p><a class="mention" href="/u/mariosasko">@mariosasko</a> Thanks so much for the solution.</p>
Dumping Image data and load using dataloader

username: _joker
cooked: <p>I am wanting to dump the data so that I can load it back for training my model.</p>
<p>My code snipped for dumping the data:</p>
<pre><code class="lang-auto">for batch_idx, (image, label) in enumerate(dataloader):
    image, label = image.to(device), label.to(device)
    perturbed_image = attack.perturb(image, label)
    #---------- Classifier ----------
    predict_A = classifier(perturbed_image)
    pred_label = torch.max(predict_A.data, 1)[1]
    
    if pred_label != label:
        adv_data.append( (perturbed_image.to("cpu"), label.to("cpu")) )

def load_data(list_):
    if len(list_[0])==2:
        img, lab = [], []
        for i in list_:
            img.append(i[0]), lab.append(i[1])
        xs = torch.stack(img)
        xs = xs.squeeze(1)
        ys = torch.Tensor(lab)
        dataset = TensorDataset(xs, ys)
        del img, lab
    return dataset

with open(dir, "rb") as file:
    train_list = pickle.load(file)
train_set_1 = load_data(train_list)

train_loader = torch.utils.data.DataLoader(train_set_1)
</code></pre>
<p>Is there any other way I can dump it correctly so as to load it in the  <code>torch.utils.data.DataLoader</code>.</p>
<p>This works well individually but I guess this is not exactly the way how a usual torchvision.dataset.CIFAR10 is stored and loaded. I want to concatenate this with <code>torchvision.datasets.CIFAR10</code> in my train_loader.</p>
<p>i.e.,<br>
concatenating this;<br>
<code>train_set_2 = torchvision.datasets.CIFAR10(root="data", train=True)</code> with train_set_1 above.</p>

username: ptrblck
cooked: <p>To concatenate two <code>Datasets</code>, you could use <code>ConcatDataset</code>. If you want to use shuffling, you would have to make sure both datasets return tensors in the same shape. Otherwise, you should define the lengths and <code>batch_size</code> such that batches do not contain samples from both datasets.</p>
<p>I’m not sure, how you would like to concatenate the datasets, so let me know, if I misunderstood the question.</p>

username: _joker
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> You understood it correctly.<br>
I actually found a way, I dump both the datasets (dataset1: CIFAR10, dataset2: CIFAR10 variant) using <code>torch.save</code> and load it back like this;</p>
<pre><code class="lang-auto">img = torch.load(filename_images)
lab = torch.load(filename_labels)
        
xs = torch.stack(img)
xs = xs.squeeze(1)
ys = torch.Tensor(lab)
dataset = TensorDataset(xs, ys)
</code></pre>
<p>this <code>dataset</code> works perfectly normal in <code>torch.utils.data.DataLoader</code>. And the new concatenated data is already shuffled above before dumping but I do shuffle it again when I load it for training.</p>
How to append tensors into a list using loops

username: Leockl
cooked: <p>I have the following code which outputs 2 arrays in a list:</p>
<pre><code class="lang-auto">arr1 = np.array([[1.,2,3], [4,5,6], [7,8,9]])

arr_split = np.array_split(arr1,
                           indices_or_sections = 2,
                           axis = 0)

arr_split
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">[array([[1., 2., 3.],
        [4., 5., 6.]]), array([[7., 8., 9.]])]
</code></pre>
<p>How do I cast these 2 arrays into PyTorch tensors and put them into a list using <code>for</code> (or <code>while</code>) loops, so that they look like this:</p>
<pre><code class="lang-auto">[tensor([[1., 2., 3.],
         [4., 5., 6.]], dtype=torch.float64)
tensor([[7., 8., 9.]], dtype=torch.float64)]
</code></pre>
<p>Many thanks in advance!</p>

username: JuanFMontesinos
cooked: <pre><code class="lang-auto">torch_arr = [torch.from_numpy(x) for x in arr_split]
</code></pre>

username: Leockl
cooked: <p>Many thanks for this</p>
Scope of torch.no_grad()

username: saluei
cooked: <p>what is the scope of with torch.no_grad:<br>
in function1() I need to set [with torch.no_grad] again or it is not needed because calling function1 is in with torch.no_grad scope</p>
<p>… some code…<br>
with torch.no_grad():<br>
%%%%%%… some code…<br>
%%%%%%self.function1() # disable gradient<br>
%%%%%%… some code…<br>
… some code …<br>
self.function1() # gradient calculation done<br>
… some code …</p>
<p>def function1(self):<br>
… some code …</p>

username: mariosasko
cooked: <p><code>torch.no_grad()</code> disables gradient calculation inside the body of the <code>with</code> statement (it acts just as an ordinary context manager, but it can be used as a decorator as well). Even if other functions are invoked inside the body, the operations defined by these functions will not be recorded by the <code>autograd</code> engine.</p>
How to compute the IOU between two images "frame" and "templateimgw"

username: CVAIDL
cooked: <p>frame is an input image, and templateimgw is obtained by warping the template with a homography.<br>
I have wrote the programm to compute the IOU between two images, but there are some errors in these codes below.<br>
ZOOM_OUT_SCALE = 4<br>
batch_size = frame.shape[0]<br>
fake_template = torch.ones([batch_size, 1, 74 * ZOOM_OUT_SCALE, 115 * ZOOM_OUT_SCALE], device=frame.device)<br>
scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)<br>
scaling_mat[:, 0, 0] = scaling_mat[:, 1, 1] = ZOOM_OUT_SCALE<br>
target_mask = warp.warp_image(fake_template, scaling_mat, out_shape=fake_template.shape[-2:])<br>
mapping_mat = torch.matmul(frame, scaling_mat)<br>
mapping_mat = torch.matmul(scaling_mat, frame)<br>
mapping_mat = torch.matmul(mapping_mat, templateimgw.inverse())<br>
output_mask = warp.warp_image(fake_template, mapping_mat, out_shape=fake_template.shape[-2:])<br>
output_mask = (output_mask &gt;= 0.5).float()<br>
target_mask = (target_mask &gt;= 0.5).float()</p>

username: CVAIDL
cooked: <p>I have rewrote the code as this below, but there are still some errors.<br>
ZOOM_OUT_SCALE = 4<br>
batch_size = frame.shape[0]<br>
fake_template = torch.ones([batch_size, 1, 74 * ZOOM_OUT_SCALE, 115 * ZOOM_OUT_SCALE], device=frame.device)<br>
scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)<br>
scaling_mat[:, 0, 0] = scaling_mat[:, 1, 1] = ZOOM_OUT_SCALE<br>
target_mask = warp.warp_image(fake_template, scaling_mat, out_shape=fake_template.shape[-2:])<br>
mapping_mat = torch.matmul(frame, scaling_mat)<br>
mapping_mat = torch.matmul(templateimgw.inverse(), mapping_mat)output_mask = warp.warp_image(fake_template, mapping_mat, out_shape=fake_template.shape[-2:])<br>
output_mask = (output_mask &gt;= 0.5).float()<br>
target_mask = (target_mask &gt;= 0.5).float()</p>

username: ptrblck
cooked: <p>What kind of errors are you getting? Could you post the error message with the complete stack trace, please?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: CVAIDL
cooked: <p>Thanks for your reply.</p>
<p>The error is as follow.</p>
<p>Traceback (most recent call last):<br>
File “ioucompute.py”, line 199, in mapping_mat = torch.matmul(frame, scaling_mat)<br>
RuntimeError: Expected tensor to have size 256 at dimension 1, but got size 3 for argument <span class="hashtag">#2</span> ‘batch2’ (while checking arguments for bmm)</p>

username: ptrblck
cooked: <p>Thanks for the error message. It seems to be a double post from <a href="https://discuss.pytorch.org/t/runtimeerror-expected-tensor-to-have-size-256-at-dimension-1-but-got-size-3-for-argument-2-batch2-while-checking-arguments-for-bmm/91381/5">here</a> so lets keep the discussion in the other thread.</p>

username: CVAIDL
cooked: <p>OK. The question is more specific from there.</p>
RuntimeError: result type Float can't be cast to the desired output type Long

username: SandPhoenix
cooked: <p>Hello guys,<br>
So, I am usuing the torch.bitwise_xor function and the error that I am getting is the one that is in the title.<br>
I tried to cast the result with several methods like .to(torch.long) or .long() or torch.LongTensor()<br>
any idea?</p>
<pre><code class="lang-auto">def forward(self,x):
        out_clean = self.Layer_clean(x)
        dummy_output = self.Dummy_layer(x)
        cd1 = out_clean.argmax(dim=1)
        b = torch.bitwise_xor(dummy_output,cd1).to(torch.long)
        return b
</code></pre>

username: Kushaj
cooked: <p>Can you confirm that the inputs to <code>bitwise_xor</code> are integer?</p>

username: SandPhoenix
cooked: <p>Yes, indeed! I did confirm that</p>

username: Kushaj
cooked: <p>The error is due to something else.</p>

username: ptrblck
cooked: <p>Could you post the complete error message including the stack trace and print the inputs to <code>bitwise_xor</code>, please?</p>
Tensorboard example in pytorch has a mistake

username: tomad01
cooked: <p>I can not help myself asking this.<br>
<a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#assessing-trained-models-with-tensorboard" rel="nofollow noopener">Here</a> we have:</p>
<pre><code class="lang-auto">with torch.no_grad():
    for data in testloader:
        images, labels = data
        output = net(images)
        class_probs_batch = [F.softmax(el, dim=0) for el in output]
        _, class_preds_batch = torch.max(output, 1)

        class_probs.append(class_probs_batch)
        class_preds.append(class_preds_batch)
</code></pre>
<p>but shouldn’t we have instead:</p>
<pre><code class="lang-auto">with torch.no_grad():
    for data in test_iterator:
        images, labels = data
        output = net(images)
        class_probs_batch = [F.softmax(el, dim=0) for el in output]
        _, class_preds_batch = torch.max(output, 1)

        class_probs.append(class_probs_batch)
        #class_preds.append(class_preds_batch)
        class_preds.append(labels)
</code></pre>
<p>?</p>

username: ptrblck
cooked: <p>The <code>class_preds</code> will be used to compare them to the real labels in <code>add_pr_curve_tensorboard</code>:</p>
<pre><code class="lang-python">tensorboard_preds = test_preds == class_index
tensorboard_probs = test_probs[:, class_index]
</code></pre>
<p>If you would store the <code>labels</code> instead, you would just compare the ground truth to itself, wouldn’t you?</p>
<p>EDIT: I misunderstood the code and it should indeed compare the <code>class_index</code> to the targets.<br>
The issue is tracked <a href="https://github.com/pytorch/tutorials/issues/1104">here</a>.</p>
The tensorboard tutorial - PR curve

username: Alan_Kong
cooked: <p>I have read the contents of tensorboard tutorial ([<a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html</a>]</p>
<p>And noted that for PR curve, the tutorial adopted the prediction (i.e. tensorboard_preds) instead of label, which seems to be contradicting with the parameter setting of PR curve as set out in documentation.<br>
[<a href="https://pytorch.org/docs/stable/_modules/torch/utils/tensorboard/writer.html#SummaryWriter.add_pr_curve" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/utils/tensorboard/writer.html#SummaryWriter.add_pr_curve</a>].</p>
<p>If based on the tutorial, the curve seems strange as most of them even accuracy less than 80% are 90 degree at the right upper corner.</p>
<p>Any help on this matter?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Thanks for reporting this issue! It seems it’s already tracked <a href="https://github.com/pytorch/tutorials/issues/1104">here</a>.</p>
Clipping parameter values during training

username: Chang_Liao
cooked: <p>I’m working on an adversarial attack project on image classifiers, as the adversarial image being the only parameter for the optimizer and there’re upper and lower bounds for pixel values in this adversarial image, how can I clip those pixel values during training? A showcase of this problem is in following block of code:</p>
<pre><code>adv_image = eps * torch.rand((image shape)).to(device)
adv_image.requires_grad_(True)
optim=torch.optim.SGD([adv_image], lr=1e-3)
for epoch in num_epochs:
    optim.zero_grad()
    loss=get loss after some operations
    loss.backward()
    optim.step() 
    # how can I perform clipping on a leaf node during optimization?
    # my current idea:
    tmp = adv_image.clone()
    tmp.clamp_(min_value, max_value)
    adv_image = tmp.clone()
    optim = torch.optimizer([adv_image], lr=1e-3)</code></pre>

username: ptrblck
cooked: <p>This approach should work, if your optimizer doesn’t have internal states, which is the case for <code>SGD</code>.<br>
However, you also should be able to wrap the <code>clamp</code> operation into a <code>torch.no_grad()</code> block instead of applying it on a cloned tensor:</p>
<pre><code class="lang-python">with torch.no_grad():
    adv_image.clamp_(-1, 1)
</code></pre>
GPU memory usage increased during mixed precision trainning

username: tyterry
cooked: <p>Hi all,</p>
<p>I am trying on a GRU model with 3 layers and 3840 hidden units which generate 3 outputs (namely output_max, output_min, output_max_first), due to the size of model I also implemented mixed precision (with snapshot to understand the memory usage) as follows:</p>
<pre><code class="lang-auto">for b in range(0, len(train_x), batch_size):
    
    input = torch.tensor(train_x[b:(b+batch_size)], dtype=torch.float, requires_grad = False)
    target = torch.tensor(train_y[b:(b+batch_size)], dtype=torch.float, requires_grad = False)
        
    optimizer.zero_grad()

    print("Memory Check 1", torch.cuda.memory_reserved())
    
    with amp.autocast():
        output_max, output_min, output_max_first = my_gru(input)
    
        max_loss = loss_weight[0] * max_loss_func(output_max, target[:,0])
        min_loss = loss_weight[1] * min_loss_func(output_min, target[:,1])
        max_first_loss = loss_weight[2] * classification_loss_func(output_max_first, target[:,2])
    
    print("Memory Check 2", torch.cuda.memory_reserved())

    scaler.scale(max_loss).backward(retain_graph = True)
    scaler.scale(min_loss).backward(retain_graph = True)
    scaler.scale(max_first_loss).backward()
    
    scaler.unscale_(optimizer)
    
    print("Memory Check 3", torch.cuda.memory_reserved())
    
    scaler.step(optimizer)
    scaler.update()
</code></pre>
<p>The printed result are as follows:</p>
<pre><code class="lang-auto">Memory Check 1 1803550720
Memory Check 2 2732589056
Memory Check 3 4586471424
</code></pre>
<p>However, without mixed precision the above code generate the following result:</p>
<pre><code class="lang-auto">Memory Check 1 1803550720
Memory Check 2 2732589056
Memory Check 3 3659530240
</code></pre>
<p>Which has same memory usage during forward pass (which is weird) but less memory usage during backward pass.</p>
<p>Moreover, the memory usage seems to be carried forward to next loop (i.e. for the mixed precision implmentation, the Memory Check 1 of next loop returns 4586471424)</p>
<p>Did I somehow implement the mixed precision wrongly?</p>
<p>Many thanks for your help!</p>

username: ptrblck
cooked: <p><code>torch.cuda.memory_reserved()</code> will return the allocated and cached memory, which might be higher, e.g. if you are using <code>cudnn.backend.benchmark=True</code> (and cudnn would profile more kernels, if they are available for mixed-precision).<br>
To check the allocated memory, you could use <code>torch.cuda.memory_allocated()</code>.</p>
Applying Monte Carlo Dropout in CNN as Bayesian approximation

username: MERAH_Samia
cooked: <p>Hi<br>
I am trying to implement Bayesian CNN using Mc Dropout on Pytorch, as I know we apply it during both the training and the test time, and we should multiply the dropout output by 1/(1-p) where p is the dropout rate, could anyone confirm these statements, or give me an example of how should be the code, please.</p>

username: ptrblck
cooked: <p>The dropout scaling is already done in the backend, if you are using the <code>nn.Dropout</code> module:</p>
<pre><code class="lang-python">drop = nn.Dropout()
x = torch.ones(10)
out = drop(x)
print(out)
&gt; tensor([0., 0., 2., 2., 0., 0., 0., 2., 0., 0.])
</code></pre>
<p>so you don’t need to reapply it.</p>
Failed on GPU with AWS EC2 g2 instance

username: Xixi
cooked: <p>I installed pytorch on AWS ec2 g2 instance ( with cuda 7.5)</p>
<p>Everything runs all right while on CPU,  but when I tried to run on GPU. It threw the message below.<br>
Tried to google the err msg but couldn’t find it<br>
Thanks for any help!</p>
<blockquote>
<blockquote>
<blockquote>
<p>torch.cuda.is_available()<br>
THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=108 error=30 : unknown error<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/usr/local/lib/python3.5/dist-packages/torch/cuda/<strong>init</strong>.py”, line 28, in is_available<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
RuntimeError: cuda runtime error (30) : unknown error at torch/csrc/cuda/Module.cpp:108</p>
</blockquote>
</blockquote>
</blockquote>

username: apaszke
cooked: <p>Maybe <a href="https://discuss.pytorch.org/t/request-tutorial-for-deploying-on-cloud-based-virtual-machine/28/3?u=apaszke">this post</a> could help you.</p>

username: Xixi
cooked: <p>Thanks! I will take a try</p>

username: mjdietzx
cooked: <p>this quick start to aws+pytorch may help anyone struggling w/ getting gpu working. <a href="https://medium.com/@waya.ai/quick-start-pyt-rch-on-an-aws-ec2-gpu-enabled-compute-instance-5eed12fbd168" rel="nofollow noopener">https://medium.com/@waya.ai/quick-start-pyt-rch-on-an-aws-ec2-gpu-enabled-compute-instance-5eed12fbd168</a></p>

username: riyawilliams
cooked: <p>Hai,<br>
Yes. It used to work fine. No longer does due to driver change by NVIDIA.</p>
<p><a href="http://support.citrix.com/article/CTX202066" class="onebox" target="_blank" rel="nofollow noopener">http://support.citrix.com/article/CTX202066</a></p>
<p>This article explains the issue.</p>
<p>“GPU-accelerated OpenGL on<a href="https://mindmajix.com/aws-training" rel="nofollow noopener">Amazon Web Services</a> (AWS G2 instances) with NVIDIA GRID K520/K340 No Longer Works”</p>
<p>This is a known issue.</p>
<p>NVIDIA GRID drivers introduced a change in build 340. Earlier NVIDIA drivers enabled OpenGL sharing but later versions do not. NVIDIA reports that the AWS behavior is as intended. NVIDIA OpenGL over RDP/ RDSH is only supported on workstation (Quadro) / NVS products. The GRID K520 card that is on Amazon’s G2 instances is not a workstation product unlike GRID K2.<br>
Thanks,<br>
Riya,</p>

username: jyostna_nareshit
cooked: <p>For any Queries related to AWS visit:<a href="https://nareshit.com/aws-online-training/" rel="nofollow noopener">https://nareshit.com/aws-online-training/</a></p>
Idea: Does it make sense to feedback high level features back into the initial layers?

username: zshn25
cooked: <p>In computer vision, the initial filters are known to output low level features such as edges/corners while the later layers capture high level information such as semantics of the image.</p>
<p>In human brain, the neurons are connected in extremely complex ways. This helps humans to send information back and forth the low level neurons and high level neurons. While in current artificial networks, they are connected in sequential manner.</p>
<p>Another advantage would be to decrease the number of parameters to be trained. Can act as regularisation.</p>
<p>Are there any works in this direction? Does it make sense to train networks in this loop manner?</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3,3, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(3,3, kernel_size=3, stride=1, padding=1)
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv1(x)
        x = self.conv2(x)
        return x

# Forward pass
net = Net().cuda()
print(net)
net(torch.rand(2, 3, 512, 512).cuda()).shape

</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">  Net(
  (conv1): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (conv2): Conv2d(3, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  )
  torch.Size([2, 3, 512, 512])
</code></pre>

username: ptrblck
cooked: <p>Generally, I would recommend to just try our your ideas, especially if you would like to test a specific hypothesis.<br>
However, while the comparison between the human brain (or vision system) and neural networks fits a few use cases, I still think that a lot of understanding is still missing. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Flattening the output of a hidden layer in a feed forward neural network

username: samin_hamidi
cooked: <p>I need a neural network (fully connected) that takes a batch of images of size 100 (these images are output of an encoder with dimensionality of (10,)) in the form of a tensor so [100,10] and outputs a vecotor with the size 20. I first used this code:</p>
<blockquote>
<p>def mlp(sizes, activation=nn.Tanh, output_activation=nn.Identity):<br>
layers = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for j in range(len(sizes)-1):<br>
act = activation if j &lt; len(sizes)-2 else output_activation<br>
layers += [nn.Linear(sizes[j], sizes[j+1]), act()]<br>
return nn.Sequential(*layers)</p>
</blockquote>
<p>This gives me a tensor with the size [100,20].<br>
Is it possible flatten the output of the last layer before nn.identity and then use two more layers  [2000,100] and [100,20]?<br>
I wonder if this is possible in a fully connected neural network as I have only seen nn.flatten being used in the last layers of CNNs.<br>
I appreciate the help.</p>

username: ptrblck
cooked: <p>If I understand the use case correctly, you wouldn’t need to flatten the activation of the last layer, as it already is 2-dimensional (has the shape <code>[100, 20]</code>).<br>
You could just add two more linear layers afterwards, where the first uses <code>in_features=20</code>.</p>
Torch.mm is not accepting data

username: SandPhoenix
cooked: <p>I am using torch.mm but for some reason I am getting this error: RuntimeError: matrices expected, got 1D, 2D tensors at /pytorch/aten/src/TH/generic/THTensorMath.cpp:131<br>
here is a small portion of what I am doing using the torch.mm function :</p>
<pre><code class="lang-auto">h = self.Layer(x)
x = torch.mm(h,Permutation_tensor.float().transpose(0,1))

</code></pre>
<p>Actually, I am doing this in order to permute the elements at the position 5 to the position 2 and 3 to 4 etc. You got the idea, I think so any help would be very good</p>

username: Usama_Hasan
cooked: <aside class="quote no-group" data-username="SandPhoenix" data-post="1" data-topic="91969">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sandphoenix/48/29137_2.png" class="avatar"> SandPhoenix:</div>
<blockquote>
<p>matrices expected, got 1D</p>
</blockquote>
</aside>
<p>Hy <a class="mention" href="/u/sandphoenix">@SandPhoenix</a>, I guess your input ‘h’ is 1-D and the function expect matrices.</p>

username: SandPhoenix
cooked: <p>I am using tensors<br>
for example, the size of h is [2,10] so it is not 1D<br>
This is why I am finding the error strange</p>

username: Usama_Hasan
cooked: <p>I was not able to recreate the error with this.</p>
<pre><code class="lang-auto">h = torch.randn(size=(2,10))
permutation_tensor = torch.randn(size=(2,10))
x = torch.mm(h,permutation_tensor.float().transpose(0,1))
print(x)

</code></pre>
<p>Can you try running this.</p>

username: SandPhoenix
cooked: <p>I did, I run into the same error!</p>

username: Usama_Hasan
cooked: <p>Hy <a class="mention" href="/u/sandphoenix">@SandPhoenix</a> can you post the complete error.</p>
Can the result of grad be negative?

username: Martin_Lee
cooked: <p>Here is the code. I want to make the x iterated to zero.<br>
When I set the initial value as positive, every thing goes right. However, when the x  is negative at the beginning, the x eventually becomes -inf.<br>
<strong>I found that the the result of grad is always positive in the code.</strong>  However, in math, the slope of x**2 at x = -1 should be negative. (Correct typo on 8/8)<br>
I have no idea how to solve the problem, and wonder someone can help me.</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([-1.0])
x.requires_grad_(True)
rate = 0.1
N = 10
for e in range(N):
    loss_fn = x**2
    loss_fn.backward(x)
    with torch.no_grad():
        delta = x.grad.item()
        x = x - rate * x.grad
    x.requires_grad_(True)
    
</code></pre>

username: mariosasko
cooked: <p>Replace the line <code>loss_fn.backward(x)</code> with <code>loss_fn.backward()</code>. After that change the optimization should behave as expected. To understand the difference between these two calls, please check the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">autograd tutorial</a> (it contains both examples).</p>
<aside class="quote no-group quote-modified" data-username="Martin_Lee" data-post="1" data-topic="91918">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/3e96dc/48.png" class="avatar"> Martin_Lee:</div>
<blockquote>
<p>However, in math, the slope of x**2 at x = 1 should be negative.</p>
</blockquote>
</aside>
<p>This statement does not hold. Derivative of <code>x**2</code> is <code>2*x</code> so at <code>x = 1</code> it’s value is <code>2 * 1 = 1</code>.</p>

username: Martin_Lee
cooked: <p>Thx a lot.</p>
<p>Yeah, you’re right, that’s the typo. Many thanks!</p>
<pre><code class="lang-auto">However, in math, the slope of x**2 at x = 1 should be negative.

</code></pre>
Training will freeze several time

username: mojzaar
cooked: <p>I used to train my network on the server with two GPUs with the DataParallel model. Today I encountered a freezing problem. Some epochs will freeze and take a minute to be done, and it has slow down the training time significantly! I did not change anything in the code that I had before! What is the problem? I also tried reducing the batch size and do single GPU training; the same thing is happening! Any idea??</p>
TypeError: tuple is not a Module subclass when trying to do nn.Sequential(*ourmodel)

username: Shisho_Sama
cooked: <p>Hello everyone. I’m trying to edit an existing pretrained model and add my own.<br>
I followed the discussions <a href="https://discuss.pytorch.org/t/how-could-i-remove-certain-layer-in-torchvision-models/23303/3">here</a> and <a href="https://discuss.pytorch.org/t/accessing-intermediate-layers-of-a-pretrained-network-forward/12113">here</a> and faced with two options, either of which fail!<br>
the first thread says do :</p>
<pre><code class="lang-auto">resnet18 = models.resnet18(pretrained=True)
ourmodel = list(resnet18.named_children())[:-1]
ourmodel = nn.Sequential(*ourmodel)
</code></pre>
<p>This fails with the error :<br>
<code>**TypeError** : tuple is not a Module subclass</code></p>
<p>The second option was to do :</p>
<pre><code class="lang-auto">resnet18 = models.resnet18(pretrained=True)
ourmodel = list(resnet18.named_children())[:-1]
ourmodel = nn.ModuleList(ourmodel)
</code></pre>
<p>which again results in the same exact error :<br>
<code>**TypeError** : tuple is not a Module subclass</code></p>
<p>Has pytorch’s behavior changed ? I’m on Pytorch 1.0<br>
What should I do ?</p>
<p>Here is the full stacktrace :</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
     27 ourmodel = list(resnet18.named_children())[:-1]
     28 # ourmodel = nn.ModuleList(ourmodel)
---&gt; 29 ourmodel = nn.Sequential(*ourmodel)
     30 
     31 for i, (name,child) in enumerate(ourmodel.named_children()):

~\Anaconda3\lib\site-packages\torch\nn\modules\container.py in __init__(self, *args)
     51         else:
     52             for idx, module in enumerate(args):
---&gt; 53                 self.add_module(str(idx), module)
     54 
     55     def _get_item_by_idx(self, iterator, idx):

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in add_module(self, name, module)
    177         if not isinstance(module, Module) and module is not None:
    178             raise TypeError("{} is not a Module subclass".format(
--&gt; 179                 torch.typename(module)))
    180         elif not isinstance(name, torch._six.string_classes):
    181             raise TypeError("module name should be a string. Got {}".format(

TypeError: tuple is not a Module subclass

Thank you all very much in advance</code></pre>

username: liyz15
cooked: <p>Replace <code>resnet18.named_children()</code> with <code>resnet18.children()</code></p>

username: swalecka
cooked: <p>I had issues with the proposed solution as this user had <a href="https://github.com/pytorch/pytorch/issues/15129" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15129</a> I ended up just mutating the last layer as the official pytorch TL tutorial did.</p>
Resnet custom head size mismatch error

username: sid-ls
cooked: <p>I’m loading the resnet 50 from torch hub, cutting off the fc layers (last two nn modules), keeping the base, and creating a custom head on it thats the exact same head as the original resnet (created same head to debug size mismatch error).</p>
<p>But I am still facing the size mismatch error. I do not understand how, because when I print out the layers, it is exactly the same.</p>
<p>I’ve attached code to reproduce result</p>
<pre><code class="lang-auto">import torch
x=torch.randn(([2, 3, 50, 50]))

model = torch.hub.load('pytorch/vision:v0.5.0', 'resnet50', pretrained=True)
print(model)

# feature extraction layer 
base_model_list = list(model.children())[:-2]
# same layers as loaded resnet 
layers= base_model_list + list(model.children())[-2:]
new_model = nn.Sequential(*layers)
print(new_model)

model(x) # executes with no error 
new_model(x) # runtime error: size mismatch, m1: [4096 x 1], m2: [2048 x 1000]

</code></pre>
<p>Anyone know how? Is there a hidden operation that’s getting deleted when I separate the base? Because the layers are the same when printed</p>

username: ptrblck
cooked: <p>You are missing the <code>flatten</code> operation from <a href="https://github.com/pytorch/vision/blob/6c2cda6a0eda4c835f96f18bb2b3be5043d96ad2/torchvision/models/resnet.py#L210">this line of code</a>, which is applied using the functional API.<br>
This code should work:</p>
<pre><code class="lang-python"># feature extraction layer 
base_model_list = list(model.children())[:-2]
# same layers as loaded resnet 
layers= base_model_list + list(model.children())[-2:-1] + [nn.Flatten()] + list(model.children())[-1:]
new_model = nn.Sequential(*layers)
</code></pre>
<p>Generally <code>nn.Sequential</code> is used for very simple models, so wrapping all child modules inside this container might not always work out of the box.</p>

username: swalecka
cooked: <p>I read the above comment after seeing a lot of people suggest on the web to do transfer learning with <code>Sequential (*model.children())</code> approach. It was still unclear to me whether that approach is the correct one for only finetuning the last layer.  After seeing <a href="https://github.com/pytorch/pytorch/issues/15129" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15129</a> it made sense that this is not the correct approach, you cannot express resnet by only its .children(). I just wanted to call that out explicitly to save some time.  I ended up just mutating the last layer with  <code>resnet.fc = nn.Linear(num_ftrs, my_num_classes) </code>.</p>
Pytorch implementation of tape.gradient from tensorflow

username: SaminYeasar
cooked: <p>I’m trying to implement the following tensorflow script to pytorch, where values are passed through multiple networks but tape.gradient allows the compute gradient for specific networks</p>
<pre><code class="lang-auto"># tensorflow
with tf.GradientTape(watch_accessed_variables=False, persistent=True) as tape:
	# persistent=True allow multiple graph
	# watch_accessed_variables = False, allow graph only for networks defined in tape.watch()
	tape.watch(first_network.variables)
	tape.watch(second_network.variables)


	y = first_network(x)
	z = second_network(y)

	# random stuff does not contain graph
	first_loss = y + random_stuff_1
	second_loss = -z + random_stuff_2

first_loss_grad = tape.gradient(first_loss, first_network.variables)
second_loss_grad = tape.gradient(second_loss, second_network.variables)

first_network_optimizer.apply_gradients(zip(first_loss_grads, first_network.variables))
second_network_optimizer.apply_gradients(zip(second_loss_grads, second_network.variables))

</code></pre>
<p>my pytorch implementation</p>
<pre><code class="lang-auto">
# pytorch


y = first_network(x)
z = second_network(y)

# random stuff does not contain graph
first_loss = y + random_stuff_1
second_loss = -z + random_stuff_2

first_network_optimizer.zero_grad()
second_network_optimizer.zero_grad()

first_loss.bakward(retrain_graph=True)
second_loss.bakward()

first_loss.step()
second_loss.step()
</code></pre>
<p>let me know if the implementation seems right.</p>
Using nn.Sequential increases Inference time

username: kamathhrishi
cooked: <p>When I was writing a model definition. I noticed that with nn.Sequential there was an increase in about 0.5s. This time could cascade over several examples and increase the training time by a large factor. Why does this happen?</p>

username: ptrblck
cooked: <p>This should not be the case.<br>
Could you post the model definition inside the <code>nn.Sequential</code> container and the custom <code>nn.Module</code>?</p>
How to checkpoint model predictions?

username: Aldebaran
cooked: <p>Hello,</p>
<p>I would like to save some of my model outputs to file. This will be done only at the test step:</p>
<pre><code class="lang-python">    def test_step(self, batch, batch_idx):
        x1, x2 = batch["x1"], batch["x2"]
        predicts = self(x1, x2) # which call model forward in eval mode
        save(predicts, predicts_path) # method to save predicts to file
      
</code></pre>
<p>What would be an efficient approach to concatenate in a file each batch <code>predicts</code> of the model (which has the shape <code>[batch_size, hidden_size]</code>)?</p>

username: mariosasko
cooked: <p>The easiest approach is to use <code>torch.save</code> and <code>torch.load</code> that will handle the <code>torch.Tensor</code> serialization and deserialization respectively.</p>
<p>With this in mind, here are the functions.  The first one appends the predictions of a model and the second one reads them in the form of a list.</p>
<pre><code class="lang-python">def save_data(data, path):
    if isinstance(data, torch.Tensor):
        data = [data]
    
    with open(path, 'ab') as f:
        for d in data:
            torch.save(d, f)

def load_data(path):
    data = []
    with open(path, 'rb') as f:
        try:
            while True:
                data.append(torch.load(f))
        except EOFError:
            pass    

    return data
</code></pre>
How to solve the 'content disposition' problem?

username: shadowkiller33
cooked: <p>I ran this kernel yesterday and everything went well, but I got this ‘content disposition’ error today. I really do not know to handle this, it would great if you can give me a hand.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75f9adbc3fa2bfc43e85e977dfafe82d2890153f.png" data-download-href="https://discuss.pytorch.org/uploads/default/75f9adbc3fa2bfc43e85e977dfafe82d2890153f" title="aaa"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75f9adbc3fa2bfc43e85e977dfafe82d2890153f.png" alt="aaa" data-base62-sha1="gPENtRqELbFy41E2amsysVCBuvJ" width="690" height="403" data-dominant-color="FADBDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">aaa</span><span class="informations">1365×799 46.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The loaded dataset expaects a key called <code>content-disposition</code> at one point.<br>
I’m not familiar with the dataset, but would recommend to check, if the data itself was somehow changed, moved etc.</p>

username: andrew-miao
cooked: <p>Crying <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> I meet the same problem today, and use the same dataset AGNews.</p>
Pooling vs. downsampling in autoencoder and how to upsample

username: urkatz
cooked: <p>Hi everyone,</p>
<p>I am building a simple 1-D autoencoder with fully connected networks. Can someone explain to me the pros and cons of (A) using the fully-connected layers themselves to downsample (i.e., set the inputs to 512 and the outputs to 256) versus (B) having the fully connected layer stay the same size (i.e., 512 to 512) and then using a pooling layer to downsample?   I feel like choice A would be better and more general because it wouldn’t have the oddities of the pooling layers. So I am not sure when/why the pooling layers are used.</p>
<p>Also, if I do use a pooling operator, I need one that doesn’t store the indices because I plan to use the encoder/decoder separately. So should I use a nn.AvgPool1d?  If, so how do I upsample in a simple way?</p>
<p>Thanks in advance!</p>

username: ayalaa2
cooked: <p>It’s a matter of learning a downsampling method versus using an existing method (average pooling). It’s hard to say which is better (thus why you might see a bit of both in literature).</p>
<p>I think using average pooling makes a lot of sense in, say, image domains. This allows the model to capture fine details in the early layers and coarse details in the average downsampled features.</p>
<p>I think using a learned downsampling would makes sense when my data doesn’t have spatial connections. That is, values “next” to eachother aren’t related. Average pooling sort of assumes that they do, thus I would probably go with a learned downsampling.</p>
<p>For upsampling, it’s essentially the same scenario, a learned upsampling or an existing upsampling technique. You can use <code>torch.nn.functional.interpolate</code> to upsample something using existing techniques. You can either use a linear layer or a transposed convolution to get a learned upsampling.</p>
<p>Hopes this helps</p>

username: urkatz
cooked: <p>Thank you for your reply. Let me think about this and I’ll post again if I have any questions.</p>
Complex matrix multiplication

username: weidler
cooked: <p>Given the documentation stating that</p>
<blockquote>
<p>Operations on complex tensors (e.g., <a href="https://pytorch.org/docs/stable/generated/torch.mv.html#torch.mv" rel="noopener nofollow ugc"> <code>torch.mv()</code> </a>, <a href="https://pytorch.org/docs/stable/generated/torch.matmul.html#torch.matmul" rel="noopener nofollow ugc"> <code>torch.matmul()</code> </a>) are likely to be faster and more memory efficient than operations on float tensors mimicking them.</p>
</blockquote>
<p>I would expect matmul to be implemented for complex tensors, however when I try to execute the following:</p>
<pre><code class="lang-auto">a = torch.tensor([[1.4 + 3j, 2 + 5j], [1.4 + 3j, 2 + 5j]], dtype=torch.cfloat)
a @ a
</code></pre>
<p>I get <code>RuntimeError: _th_addmm_out not supported on CPUType for ComplexFloat</code>.</p>
<p>This also happens when using <code>torch.matmul</code> or <code>torch.mm</code> instead of the short operator.</p>
<p>Am I doing something wrong?</p>
<p><strong>Edit</strong>: The error occurs on both CPU and GPU.</p>

username: weidler
cooked: <p><strong>Update</strong>: matrix vector multiplication seems to work if explicitly done on a vector (<code>torch.mv()</code>) but cannot be deduced from e.g. multiplying shapes (2, 2) x (2, 1).</p>
<p>Would be really nice to know if this is expected as in “not implemented” or if this can be done differently.</p>
Implementation :: Character embeddings for BiLSTM NER?

username: uyaseen
cooked: <p>I am trying to code a simple NER model (BiLSTM) with character level embeddings (also modelled using BiLSTM). The idea to concatenate character embedding (computed from BiLSTM) with the word embeddings, this concatenated tensor is fed to the BiLSTM to label sequence. In my current implementation I am using <code>for-loop</code> to compute character representation of every word token, is there a way I can avoid this loop?<br>
The shape of <code>char_seq</code> is: <code>batch_size</code>,<code> seq_len</code>, <code>token_len (max_chars_in_a_token)</code>.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as f


class Net(nn.Module):

    def __init__(self, params):
        super(Net, self).__init__()

        self.word_embedding = nn.Embedding(params.word_vocab_size, params.word_embedding_dim)
        self.char_embedding = nn.Embedding(params.char_vocab_size, params.char_embedding_dim)
        self.pos_embedding = nn.Embedding(params.pos_size, params.pos_embedding_dim)
        self.cap_embedding = nn.Embedding(params.cap_size, params.cap_embedding_dim)
        self.c_lstm = nn.LSTM(params.char_embedding_dim, params.char_hidden_dim, bidirectional=True, batch_first=True)
        self.lstm = nn.LSTM(params.input_dim, params.hidden_dim, bidirectional=True, batch_first=True)
        self.fc = nn.Linear(params.hidden_dim * 2, params.number_of_tags)

        self.char_embedding_dim = params.char_embedding_dim

    def forward(self, s):
        def get_char_representation(char_seq):
            s_ch_e = self.char_embedding(char_seq)
            char_vec = torch.empty(s_ch_e.shape[0], s_ch_e.shape[1], s_ch_e.shape[3] * 2)
            for idx, ch in enumerate(s_ch_e):
                s_ch_rep, _ = self.c_lstm(ch)
                s_ch_rep_f = s_ch_rep[:, -1, 0: self.char_embedding_dim]
                s_ch_rep_b = s_ch_rep[:, 0, self.char_embedding_dim:]
                s_ch_rep = torch.cat((s_ch_rep_f, s_ch_rep_b), dim=1)
                char_vec[idx] = s_ch_rep
            return char_vec

        s_words, s_pos, s_cap, s_chars = s
        s_w = self.word_embedding(s_words)
        s_p = self.pos_embedding(s_pos)
        s_c = self.cap_embedding(s_cap)
        s_ch = get_char_representation(s_chars)
        s = torch.cat((s_w, s_p, s_c, s_ch), dim=2)
        s, _ = self.lstm(s)

        s = s.contiguous()
        s = s.view(-1, s.shape[2])
        s = self.fc(s)

        return f.log_softmax(s, dim=1)
</code></pre>
NLLLoss mismatch with pytorch implementation

username: Local_Host
cooked: <p>Hi all,<br>
So I am trying to implement NllLoss function and completely lost within dimensions. Here is my attempt. Could someone please tell me what am I missing?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F


log_softmax = nn.LogSoftmax(dim=1)

torch.random.manual_seed(100)
logits = torch.randn([4, 2, 3, 1])

torch.random.manual_seed(200)
target = torch.randint_like(logits[:,1,:], 0, 2)

logp = log_softmax(logits)
logp1 =  logits - torch.log(logits.exp().sum(1).unsqueeze(1)) #Log Softmax
print('Pytorch NllLoss:', F.nll_loss(logp, target.long()), F.nll_loss(logp1, target.long()))

loss1 = target * logp[:,0,:,:]
loss2 = (1.0 - target) * (1.0 - logp[:,1,:,:])

loss = loss1 + loss2

print('My Loss',torch.mean(loss))
</code></pre>
<p>Output:<br>
Pytorch NllLoss: tensor(0.7951) tensor(0.7951)<br>
My Loss tensor(0.7221)</p>
<p>Thanks!</p>

username: sefaice
cooked: <p>I think the NllLoss should be:</p>
<pre><code class="lang-auto">loss = 0.
for i, target_i in enumerate(target.long()):
	for j, target_j in enumerate(target_i):
		loss = loss - logp[i][target_j.item()][j]
print('My Loss', loss / (4*3*1))
</code></pre>
<p>high dim NLLLoss calcucation is abstract, but it’s always the same idea that selecting the value according to target in dim 1(starting from dim 0).</p>
<p>logsoftmax is implemented correct, and I can’t figure out a better way to implement NllLoss of arbitary dimension now.</p>
Take multiple gpu as single one

username: cjwcommuny
cooked: <p>Suppose that I have a big model that cannot fit into one gpu, so I have to split the model to different gpus. I’m wondering whether there’s a way to make Pytorch take multiple gpu as a single one, so that we don’t have to split model manually.</p>

username: mathematics
cooked: <p>I heard <code>nn.DataParallel</code>  for using on two or more Gpu,<br>
Taking multiple gpu and using as one is great question, also looking for answer to this</p>

username: mrshenli
cooked: <blockquote>
<p>I’m wondering whether there’s a way to make PyTorch take multiple gpu as a single one, so that we don’t have to split model manually.</p>
</blockquote>
<p>Currently, this is not available, we are working on adding a model partitioning feature.</p>
<p>Manually splitting model shouldn’t be too hard with today’s PyTorch API, you just need to append <code>.to(device)</code> to certain layers and outputs. See this tutorial: <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Single-Machine Model Parallel Best Practices — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>