akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it possible to freeze a filter in one layer only?

yunqiu: <p>Suppose that the weight matrix for one layer is [32, 64, 4,2]. Is it possible to freeze its first filter while keeping the other 31 filters trainable?</p>

ptrblck: <p>You could freeze this particular part of your matrix by zeroing out the gradients of it.<br>
Unfortunately you cannot freeze a certain part of the weight matrix (only the complete matrix).</p>

nutszebra: <p>Using the custom function is also the one way.</p>
<p>This is the definition of the custom layer zeroing out the gradients:</p>
<pre><code class="lang-auto">from torch.autograd.function import InplaceFunction


class Freeze(InplaceFunction):

    @staticmethod
    def forward(ctx, w):
        return w

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output * 0.0,


def freeze(w):
    return Freeze.apply(w)
</code></pre>
<p>This is the example of the code in conv2d case:</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
from torch.nn import Conv2d
import torch.nn.functional as F
x = Variable(torch.randn(1, 64, 10, 10))
net = Conv2d(64, 32, (4, 2))
w = torch.cat((freeze(net.weight[:1]), net.weight[1:]))
loss = F.conv2d(x, weight=w, stride=net.stride, padding=net.padding).sum()
loss.backward()
print(net.weight.grad[0])
</code></pre>
<p>Note:  as far as I understand, the weight decay is applied even if gradients are zeroed, so please be careful about it.</p>
Torch.cat with Tensors of different devices

Daewoong.Ahn: <pre><code class="lang-auto">a = torch.randn(1,2).cuda(1)
b = torch.randn(1,2).cuda(2)
c = torch.cat([a,b])
</code></pre>
<p>It works fine ( c.device == 1 )</p>
<p>But Usually, can’t caculate ops between tensors on different device ( i.e. a + b)</p>
<p>And I sees</p>
<pre><code class="lang-auto">  // We parallelize the copy if all 6 conditions pass:
  //
  // 1. There is more than one input tensor
  // 2. No empty inputs
  // 3. The result tensor is 32-bit indexable
  // 4. The number of dimensions is &lt;= 4
  // 5. All input tensors are contiguous (output tensor may be non-contig)
  // 6. All input tensors can use 32-bit indexing
  // 7. All input tensors are on the same device
</code></pre>
<p>These comments at <a href="https://github.com/pytorch/pytorch/blob/c25e33789e45e20d3fa317f60e29bdbe49bffd28/aten/src/THC/generic/THCTensorMath.cu#L131" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/c25e33789e45e20d3fa317f60e29bdbe49bffd28/aten/src/THC/generic/THCTensorMath.cu#L131</a></p>
<p>So, Is concatenate between different devices intended?</p>
Simple Linear Regression with Pytorch

Babatunde_A: <p>Please, I am New to Pytorch and trying my hands on it’s capability so I am trying to train a simple linear regression on the popular Boston Datasets.<br>
This is my code:</p>
<pre><code class="lang-python">from sklearn.datasets import load_boston
import torch
import pandas as pd
import matplotlib.pyplot as plt
import torch.nn.functional as F
import torch.nn as nn
from torch.autograd import Variable
import numpy as np


boston = load_boston()

data = pd.DataFrame(boston['data'], columns=boston['feature_names'])

target = pd.Series(boston['target'])

data.shape, target.shape

dataA = Variable(torch.from_numpy(data.values).float())
y = Variable(torch.from_numpy(target.values).float())

linear = nn.Linear(data.shape[1], 1)

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(linear.parameters(), lr=0.01)
loss2 = []


for i in range(5):
    optimizer.zero_grad()
    outputs = linear(dataA)
    
    loss = criterion(outputs, y)
    loss2.append(loss.data[0])
    loss.backward()        

    optimizer.step()
    
plt.plot(range(5), loss2)
plt.show()

print(loss2)
</code></pre>
<p>The issue is that the loss keeps going up, can anyone please help look through and help find out what can be done differently</p>

richard: <p>Decreasing the learning rate to <code>0.0000001</code> seems to work for me, as does switching to another optimizer like Adam. I’m not sure why that works, but I hope that helps.</p>

Babatunde_A: <p>Hi Richard, thanks, funny I tried to normalize the data and it works also!</p>

richard: <p>Right, normalizing the data is important to prevent the gradients from exploding <img src="//discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>

Babatunde_A: <p>But will need to get the intuition around why a <code>0.0000001</code> learning rate works too. Tried it and yes it even decreases up to 0</p>

richard: <p>I think the reason why the small learning rate works is that because the values in the data are large, the small learning rate prevents the gradients from exploding. That’s why normalizing the data helps</p>

Babatunde_A: <p>That’s through too and logical. Thanks Richard. Can check this out too <a href="https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/" rel="nofollow noopener">https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/</a></p>

talasinski: <p>See my rely/code to <a href="https://discuss.pytorch.org/t/pytorch-fails-to-over-fit-boston-housing-dataset/40365" class="inline-onebox">PyTorch fails to (over)fit Boston housing dataset</a></p>
<p>The code seems work.</p>
How can I access the gradient of a layer?

HelpMe: <p>This is my whole neural network, sample VAE code from pytorch.</p>
<blockquote>
<p>class VAE(nn.Module):</p>
</blockquote>
<pre><code>def __init__(self):
    super(VAE, self).__init__()
    self.fc1 = nn.Linear(784, 400)
    self.fc21 = nn.Linear(400, 20)
    self.fc22 = nn.Linear(400, 20)
    self.fc3 = nn.Linear(20, 400)
    self.fc4 = nn.Linear(400, 784)

def encode(self, x):
    h1 = F.relu(self.fc1(x))
    return self.fc21(h1), self.fc22(h1)

def reparameterize(self, mu, logvar):
    std = torch.exp(0.5*logvar)
    eps = torch.randn_like(std)
    return mu + eps*std

def decode(self, z):
    h3 = F.relu(self.fc3(z))
    return torch.sigmoid(self.fc4(h3))

def forward(self, x):
    mu, logvar = self.encode(x.view(-1, 784))
    z = self.reparameterize(mu, logvar)
    return self.decode(z), mu, logvar
</code></pre>
<p>and what I want to do is checking all the gradient of fc1. Since it is a Linear(784,400) layer, I hoped that this has 128(batch size)-784-400 elements. I used my register_backward_hook(printsize) to check it.</p>
<blockquote></blockquote>
<pre><code>def printsize(self,grad_input,grad_output):
    print('Inside '+ self.__class__.__name__+ ' backward')
    print('grad_input : ', type(grad_input))
    print('grad_input[0] : ', type(grad_input[0]))
    print('grad_output : ', type(grad_output))
    print('grad_output[0] : ', type(grad_output[0]))
    print('grad_input size : ', grad_input[0].size())
    print('grad_output size : ', grad_output[0].size())
</code></pre>
<p>It turns out that grad_input size is 400, and grad_output size is 128*400.<br>
How can I check the gradient of 784-400 elements?<br>
It seems like my understanding of register_hook is wrong…</p>
<p>By the way, I want to quote my code properly… it is very ugly now… It would be great if anyone teaches me how to fix this topic…</p>

MariosOreo: <p>You should check the gradient of the weight of a layer by <code>your_model_name.layer_name.weight.grad</code>. If you access the gradient by backward_hook, it will only access the gradient w.r.t. input and ouput (as you have observed).</p>
Load mutiple models at the same time

NoRomantic: <p>I’d like to load two models in one inference script to test two datasets. The two models are both instantiated from the class Darknet (inherited from nn.Module).<br>
When I debug the script in pycharm, I find the two models seem to share the same memory space. It’s a little strange though, so I’m not surely whether I’m doing it in a right way.<br>
Could somebody help me please?</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">    if model1 is None:
        device = torch_utils.select_device()

        # Initialize model
        model1 = Darknet(cfg, img_size).to(device)

        _ = load_darknet_weights(model1, weights1)

    if model2 is None:
        # Initialize model
        model2 = Darknet(cfg, img_size).to(device)

        _ = load_darknet_weights(model2, weights2)
</code></pre>

achaiah: <p>You can control where you load your models with <code>.to(device)</code> like you’re already doing. If you only have one GPU or if you specifically select the same device, both of your models will be placed into that device’s memory (as long as they both fit). You can load as many models onto a single device as you wish and they will happily coexist until you’ve filled up the memory on the device.</p>

NoRomantic: <p>Thanks for your reply!<br>
So it won’t make the models conflict with each other even if both work on the same GPU, right? If so, then you don’t need to reply, and later I’ll accept your answer.</p>

achaiah: <p>They won’t be conflicting but since you run them on the same GPU they can’t be run in parallel exactly so you may see lower performance than if you ran them on separate GPUs.</p>

NoRomantic: <p>Thank for your advise. Yeah, it’s kind of a problem thought, I have only one GPU:sweat_smile:So later if I get two, I’ll consider run them in different GPUs.</p>
Python joblib.Parallel

genous110: <pre><code class="lang-auto">
import cv2
import os
import numpy as np
from numba import jit
from joblib import Parallel
from joblib import delayed

frame_path = 'data/ucf101_frames111111'
video_path = 'data/ucf101'
@jit
def extract_flow_img(video_full_name, frame_path1 ,i, j):
    cap = cv2.VideoCapture(video_full_name)
    video_frames = int(cap.get(7))
    success, prev = cap.read()
    rgb_img_ = os.path.join(frame_path1, 'img_{:05d}.jpg'.format(1))
    cv2.imwrite(rgb_img_, prev)
    prev = cv2.cvtColor(prev, cv2.COLOR_BGR2GRAY)
    # durance = video_frames//num_frames
    for k in range(video_frames - 1):
        success, last = cap.read()
        if success:
            rgb_img_ = os.path.join(frame_path1, 'img_{:05d}.jpg'.format(k + 2))
            cv2.imwrite(rgb_img_, last)
            last = cv2.cvtColor(last, cv2.COLOR_BGR2GRAY)
            flow = cv2.calcOpticalFlowFarneback(prev, last, None, 0.5, 3, 15, 3, 5, 1.2, 0)
            flow = np.byte((flow - flow.min()) / (flow.max() - flow.min()) * 255)
            flow_x = os.path.join(frame_path1, 'flow_x_{:05d}.jpg'.format(k + 1))
            flow_y = os.path.join(frame_path1, 'flow_y_{:05d}.jpg'.format(k + 1))
            cv2.imwrite(flow_x, flow[:, :, 0])
            cv2.imwrite(flow_y, flow[:, :, 1])
            prev = last

    print('The {} class, {}th video, done!!'.format(i, j))

subname1 = os.listdir(video_path)
count = 0
for i, f in enumerate(subname1):
    rootname = os.path.join(video_path, f)        #目前只到每个类的文件夹，还没到每个视频
    video_name = os.listdir(rootname)
    Parallel(n_jobs=50)(delayed(extract_flow_img)(os.path.join(rootname, v), os.path.join(frame_path, v), i, j)for j, v in enumerate(video_name))
</code></pre>
<p>This is the program I designed. ucf101 is a video data set. data / ucf101 is the root directory, and there are subdirectories of video classes below the data/ucf101, the videos in each subdirectory are of the same class. I want to extract all video frames and optical streams from each videos, and store them in the subdirectory which is named using its video filename, and the subdirectory is under the root directory of  data / ucf101_frames. But now I have a problem. Because it takes a lot of time to compute the optical flow images, I have to choose Joblib. Parallel for parallel processing. If n_jobs sets to 1, that is to say, no parallel computing is needed, so there will be no error. But once I set this value to more than 1, it will report the error of global name’cv2’is not defined. Because the open CV is used to extract video frames. Why the error becomes when I use parallel computing? Please give me your advice. Thank you!</p>
Why is reflection and reflect pad only implemented for certain dimensions?

Rakshit_Kothari: <p>Quoting from the documentation:</p>
<blockquote>
<p>Replicate padding is implemented for padding the last 3 dimensions of 5D input tensor, or the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor. Reflect padding is only implemented for padding the last 2 dimensions of 4D input tensor, or the last dimension of 3D input tensor.</p>
</blockquote>
<p>Is this by choice?</p>
DCGAN performance in PyTorch

reddy: <p>Hi,</p>
<p>I converted a TensorFlow implementation of DCGAN to PyTorch and observe it is ~4x slower.  Couple of comments:</p>
<ol>
<li>What is the performance expectation of PyTorch vs TensorFlow for GANs in general?</li>
<li>Do we have any benchmarks?</li>
</ol>
<p>Thanks,<br>
Rohith</p>

AlbertZhang: <p>Please refer to the official <a href="https://github.com/pytorch/examples/tree/master/dcgan" rel="nofollow noopener">dcgan</a> example of PyTorch.</p>
Torchscript module/function -> Python

mari-linhares: <p>After using <span class="mention">@torch.jit.script</span> or torch.jit.trace on a Module/function is there a way to convert the traced module/function to Python again?</p>

leowalkling: <p>You can call it from Python but if you want to debug into it, you would have to use a switch in your code to make it use the original Python function.<br>
You can also use the environment variable PYTORCH_JIT to disable it for the entire run of your program.</p>

Michael_Suo: <p>We don’t have a way to convert it back to the original python. The <code>.code</code> attribute for a method will give you valid python syntax that corresponds to that method implementation though.</p>
Backprop inside a torch.autograd function

Bruno1: <p>Hello,</p>
<p>I’m trying to implement a custom autograd function where the output of the forward pass is the solution of an optimization problem. I’d like to solve this optimization problem with a new sub graph defined <em>inside</em> the autograd function, during the forward pass.</p>
<p>However, it seems impossible to create a sub graph in the forward. (requires_grad is automatically disabled for the child, but I don’t have any warning)<br>
Would you know how I could fix this issue ?</p>
<p>Thanks a lot for your help</p>
<p>Bruno</p>

leowalkling: <p>I think the answer depends on your exact problem, maybe you could give more details on your optimization procedure.</p>
<ol>
<li>Why does your optimization procedure need to be in the forward() staticmethod of some <em>Function</em> anyway?<br>
You could also precompute the Jacobian of your operation in the forward pass and use a custom <em>Function</em> to create an output tensor (result of your optimization) which would apply that Jacobian in its backward() staticmethod.</li>
<li>torch.autograd.enable_grad would be necessary within your forward() staticmethod.</li>
</ol>

Bruno1: <p>Hi,<br>
Thank you very much !</p>
<p>About the optimization procedure, I’d like to solve the problem with a gradient descent inside my Function.<br>
I’d like to do something like this:</p>
<p>forward(X,D)<br>
(<em>nested SGD solver</em>)<br>
return A* = argmin_A ||X-DA||^2</p>
<p>(Of course, my nested problem is less trivial so I can’t find gradients by hand)</p>
<ol>
<li>
<p>I am not sure I fully understand this solution. Or I am not sure it could work this way.<br>
Actually I am doing some bi-level optimization. The gradient return by the backward method is not obtained with the graph. (It is computed based on some optimality KKT conditions).</p>
</li>
<li>
<p>I’ll take a look at this thanks a lot</p>
</li>
</ol>

leowalkling: <p>For a similar problem, have a look at <a href="https://github.com/leowalkling/sparsemap/blob/master/python/sparsemap/layers_pt/base.py" rel="nofollow noopener">https://github.com/leowalkling/sparsemap/blob/master/python/sparsemap/layers_pt/base.py</a> (a fork from <a href="https://github.com/vene/sparsemap/blob/master/python/sparsemap/layers_pt/base.py" rel="nofollow noopener">vene/sparsemap</a>)</p>
Do we have to define maskLoss when dealing with variable length sequences?

doragd: <p>Hello, friends of the Forum.<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
<p>In the PyTorch chatbot tutorial, the pad_packed_sequence is used after the variable_length sequence is processed by the pack_padded_sequence. Finally, the maskLoss function is defined.</p>
<p>But I saw in other books:<br>
After the pack_padded_sequence processing on the variable-length sequence, <strong>there is no pad_packed_sequence</strong>, and <strong>then the loss is calculated directly using the official API-CrossEntropyLoss.</strong></p>

doragd: <p>I was very cofused，who can help me~</p>

alexm: <p>Hello, I am not sure myself but from what I understand the pad_packed_sequence is not needed if you want to compute the loss as you just want to compute it on real values and not on padded values.<br>
On the other hand if you want to see the output using pad_packed_sequence will put back the padding value and enable you to better see what’s going on.<br>
<em>But</em> (and if I am correct) if you use the pad_packed_sequence in the forward then you will need to filter out the padded values to compute your loss which seems to me a bit of a waste (added padding with pad_packed_sequence  and then filter them). If you did not padded back then you can directly compute your loss.<br>
Again I this is as far as I understood it so please correct me if I am wrong<br>
Thanks a lot</p>
Suggest: torchvision.transforms.ToTensor() should raise error when float is the input data type

xzzjx: <p>I use CIFAR10 to train a CNN for classifying. I have made a mistake by set the image type as <code>np.float32</code>, and I use <code>transforms.ToTensor()</code> as a sequential transform. The behavior of model is weird. It behaves well in the first several epochs, but gave totally random output after a certain epoch. Finally it turns out that the reason is the wrong data type of training data.<br>
I suggest that pytorch should raise error when the data type is mismatching (required is byte, but input is float). That will help those who do not take care of the docs of <code>transforms.ToTensor()</code>.</p>
Calculate gradient of the network's output w. r. t. a given vector

andrea_bragagnolo: <p>Hi all, I’m facing the following problem:</p>
<p>I have the output of the network <strong>y</strong> composed by <em>N</em> elements, say <em>y_k</em> for <em>k=1,…,N</em> and a vector <strong>p</strong>. I’d to calculate <code>summation(on k) |dy_k/dp|.</code><br>
This is the snippet of the code that I currently use:</p>
<pre><code class="lang-auto">...
y = model(data)
for y_i in y:
    for i in enumerate(p):
        grad[i]+=abs(autograd.grad(y_i, p[i], create_graph=True, retain_graph=True)[0])
</code></pre>
<p>Is there a way to do the same without iterate on <strong>y</strong>?<br>
Something like:</p>
<pre><code class="lang-auto">...
y = model(data)
for i in enumerate(p):
    grad[i] = autograd(abs(y),abs(p[i]), create_graph=True, retain_graph=True)[0]
</code></pre>
<p>I have already tried different ways, but the final result doesn’t match the original one obtained with the first snippet of code.</p>
<p>Thanks in advance!</p>
Implementing a ratio layer in PyTorch

Lloyd: <p>Hi</p>
<p>I have a very specific operation I’d like to implement in PyTorch, it has no trainable parameters but needs to be differentiable. The operation works as follows:</p>
<ul>
<li>Take a fixed size window eg. 7x7</li>
<li>Take the mean of the left half of the window and the mean of the right half of the window (always ignoring the center column)</li>
<li>Set the center value to the ratio of the left side to the right side ( output(3,3) = MEANleft / MEANright)</li>
<li>Repeat over the entire image (No padding needed)</li>
</ul>
<p>I have implemented this forward method using loops but it is really slow. Is there a simple way to do this with Tensor Comprehensions? Or a better way to apply such an operation using existing layers?</p>
<p>Thanks</p>

Trump: <p>I have no idea how to make differentiable layer without any parameters… what exactly you differentiate?</p>

leowalkling: <p>This can easily be done faster than loops:<br>
a) Use two 7x7 convolution kernels and a stride of (7,7) with no padding to get left mean and right mean.<br>
Then compute your output as y[:, 0] / y[:, 1]<br>
b) Without conv operations via</p>
<pre><code class="lang-auto">x_reshaped = x.view(-1, 7, 7)  # possibly need to crop or pad first
output_prelim = x_reshaped[:, :, :3].mean(-1) / x_reshaped[:, :, 4:].mean(-1)
output = output_prelim.view(expected_output_shape)
</code></pre>
New audio package for PyTorch

atriantafy: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>We released our <em>audtorch</em> package yesterday and would like to invite everyone to take a look and use it.</p>
<p>You can find its repository here: <a href="https://github.com/audeering/audtorch" rel="nofollow noopener">https://github.com/audeering/audtorch</a>. We welcome all new issues and contributions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>We will also synchronize with the rest of the audio community that’s working on <a href="https://github.com/keunwoochoi/torchaudio-contrib" rel="nofollow noopener">torchaudio-contrib</a> and try to promote usage of PyTorch in the audio community in general.</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
2080ti cost more memory than 1080ti?

dalalaa: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/667d7e4aafe7cb5714166b355237cb91029ddc96.png" alt="TIM%E5%9B%BE%E7%89%8720190522102742" data-base62-sha1="eCFzlQqISBGKGWtnPDkhjWpOCRE" width="569" height="399"> <img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a8f376d8413ae5ce22dc69046f53c20a3bcc9c90.png" alt="TIM%E5%9B%BE%E7%89%8720190522102747" data-base62-sha1="o6BPKOeNmvnZio3kK5uvreGw6uk" width="564" height="346"></p>
<p>mycode:</p>
<pre><code class="lang-auto">from torchvision.models import vgg16
import time
model = vgg16.cuda()
time.sleep(1000)
</code></pre>
<p>it costs only 895MB memory on 1080ti ,while 1369MB memory on 2080ti.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you know the reason?</p>
<p>1080ti environments:<br>
pytorch version:1.1<br>
cuda version:9.0<br>
ubuntu version:18.04</p>
<p>2080ti environments:<br>
pytorch version:1.1<br>
cuda version:10.1<br>
ubuntu version:16.04</p>

colesbury: <p>It’s probably memory reserved by the CUDA driver. That seems to increase with newer cards. NVIDIA doesn’t explain why, but it might have to do with changes to the instruction set on newer architectures.</p>
<p>You can look at how much is reserved by the driver by doing a minimal allocation, which creates a CUDA context:</p>
<pre><code class="lang-auto">import torch
import time
torch.randn(1).cuda()
time.sleep(1000)
</code></pre>

dalalaa: <p>It costs 357MB memory on 1080TI and 471MB on 2080TI. Thank you .</p>
Tensorboard updating problems

Filos92: <p>I’m currently experimenting with the new implementation of Tensorboard in Torch.Utilis and so far it works fine.<br>
The only problem I have is that I have to restart the process every time to update the scalar graphs (maybe also the other Graph-Types, but I did not test them).<br>
I see, that Tensorboard detect new log files, but it do not show the data until i kill and restart the process.<br>
Is there a possability to fix this, right now?</p>

ptrblck: <p>Do the new values eventually show up after a while?<br>
If so, you might want to experiment a bit the the <code>--reload_interval SECONDS</code> argument when starting <code>tensorboard</code>.</p>

Filos92: <p>No, also after 1 hour it’s not loading new data<br>
a minimal example is:</p>
<pre><code>  writer = SummaryWriter(log_dir)
  for epoch i  n range(epochs):
       train(epoch)
       loss_test = test()
       writer_val.add_scalar('loss', loss_test, epoch)
   writer.close()</code></pre>

amosella: <p>I had the same issue. In my case I solved the issue using a previous version of tensorboard.</p>

Filos92: <p>so, which version do you use right now?</p>

amosella: <p>I am using TensorBoard 1.13.1.</p>

Filos92: <p>If i try to run it with TensorBoard in 1.13. I get the error:</p>
<pre><code>ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. 
This should be available in 1.14 or above.
</code></pre>
<p>how did you do this?</p>

amosella: <p>As far as I remember (I do not have access at this moment to my develop setup) I have two different venv. In one of them I have pytorch + tensorboard1.14, in the second one, I have tensorboard 1.13. I am using the second one to display my tensorboard.</p>

Filos92: <p>I’m not sure if this work for me…<br>
maybe I have to wait for tensorboard gets updated to 1.14. without beeing nightly build.<br>
But thanks for your help!</p>

Filos92: <p>anyone else an idea, before the update? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

amosella: <p>Why it is not working for you?</p>

Filos92: <p>I think your solution is a good work arround!<br>
But there must be a reason for the “not updating” problem</p>

amosella: <p>Yes, but I think it is in tensorboard side.</p>
How to perform the cross validation?

MisBah_Awan: <p>hi, anyone can help me how to implement the cross validation in CNN. i have no idea how to implement the cross validation in pytorch.here is my train and test loaders.</p>
<pre><code class="lang-auto">trainset = torchvision.datasets.ImageFolder(root='./train', transform=transform) 
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True)

testset = torchvision.datasets.ImageFolder(root='./test', transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False)
</code></pre>

Marc_Schmid: <p>Have you looked into this post?</p><aside class="quote" data-post="1" data-topic="15035">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neo/48/3968_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035">What is the best way to apply k-fold cross validation in CNN?</a> 
  </div>
  <blockquote>
    Hello, 
How can I apply k-fold cross validation with CNN. I do not want to make it manually; for example, in leave one out, I might remove one item from the training set and train the network then apply testing with the removed item. Could you please help me to make this in a standard way. 
Regards,
  </blockquote>
</aside>
Cuda out of memory with enough Cuda memory

torchtes: <p>Dear all,<br>
Currently, I have training the GRU network for speech recognition using pytorch. The training is successfully finished. However, the evaluation time, the Cuda out of memory runtime error occurs.<br>
The actual error is RuntimeError: CUDA out of memory. Tried to allocate 1.72 GiB (GPU 0; 11.92 GiB total capacity; 5.72 GiB already allocated; 1.65 GiB free; 4.04 GiB cached)</p>
<p>My Cuda has around (4.04GB + 1.65GB ) unused memory but Cuda is unable to allocate 1.72GB memory, which is unreasonable.<br>
Please suggest possible solutions to overcome this error.</p>
<p>With best regards,</p>

Oli: <p>Hello there,</p>
<p>There are a few things that you can do to lower your memory footprint. First is to run your validation code with <code>torch.no_grad()</code> so to not save any gradients. Are you by any chance running your validation code inside your training loop? If so, there might be a few tensors that you could delete from the training loop, perhaps <code>del training_input, del training_output, del ...</code>. Lastly, I found that putting this once before training lowers my memory footprint but I don’t know it’s inner workings.<br>
<code>torch.backends.cudnn.benchmark = True # Optimizes cudnn</code></p>
<p>I don’t know about the cache though, good luck <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

torchtes: <p>Thank you very much Oli.<br>
I already done the first two memory footprint methods, but i did not used the third one i will try and just come up with the result.</p>
Stop running while using batch normalization, why?

zhoulukuan: <p>When I run code <a href="https://github.com/liutinglt/CE2P" rel="nofollow noopener">here</a>, I cannot see any output about loss as well as any error or warning. It’s very strange.<br>
I debug in pycharm, and found the program stop in the line 254 of <a href="https://github.com/liutinglt/CE2P/blob/master/networks/CE2P.py" rel="nofollow noopener">https://github.com/liutinglt/CE2P/blob/master/networks/CE2P.py</a>. The debugger could get the result of self.conv1(x) successfully but wait until timeout when typing self.bn1(self.conv1(x)).<br>
How to solve this problem？</p>
Enumerate tensor behavior?

zjplab: <p><code>softmax_out</code> is a (5, 932, 1682) torch tensor<br>
<code>m_hat = torch.stack([(r+1)*output for r, output in enumerate(softmax_out)], 0)</code><br>
After this  <code>m_hat</code> is still a (5, 932, 1682) tensor. What does enumerate return? Why it can still keep its shape?</p>

iArunava: <p>This is related to <code>python3</code> and not explicitly to <code>pytorch</code></p>
<p>But anyway to answer your question.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; for i, val in enumerate([10, 20, 30, 40, 50]):
&gt;&gt;&gt;  print (i, val)
0, 10
1, 20
2, 30
3, 40
4, 50
</code></pre>
<p>Also,</p>
<pre><code class="lang-auto">In [13]: d = np.array([[4, 5], [6, 7]])
In [14]: for i, val in enumerate(d):
              print (i, val)
0 [4 5]
1 [6 7]
</code></pre>
<p>So, basically in your example, all you are doing is you are multiplying each dimension with its index. And hence the shape is preserved.</p>
Pytest vs Unittest for testing Pytorch

rubika: <p>I was looking at some of Pytorch source code with the interest of looking at how the unit tests have been done. I’ve noticed that most of the tests were written by <code>unittest</code> library but <code>pytest</code> was also utilized. Why not only using <code>pytest</code>? Is there merit in using <code>unittest</code> over only using <code>pytest</code>?</p>
How do we know which channel does what?

DeepCem: <p>Hi,</p>
<p>I do basic CNN stuff with pytorch, but I always wondered one thing.</p>
<p>how the channel number that we set affects the image? I generally know that each channel represents one thing: R, G, B, vertical edges, horizontal edges, … colours, and other unknowns.</p>
<p>but is this the order? Say I want to observe G,B and Vertical edge information. How convnet will respond?</p>
<p>I have checked source code of convnet2d but didnt find quality answer. How the convnet changes/tracks the features by itself for each channel, while calculating in each forward pass?</p>

MariosOreo: <p>You could adopt some visualization method like <a href="https://arxiv.org/abs/1512.04150" rel="nofollow noopener">CAM</a> or <a href="https://arxiv.org/abs/1610.02391" rel="nofollow noopener">GRAD-CAM</a> to observe what each channles pay attention to.</p>

DeepCem: <p>oh I see. So basically, there is no specific one, but it kinda evolves.<br>
Thanks!</p>
Tensor new dimension in forward method!

Daw: <p>I am trying to perform some calculation on the tensor in the forward method of my model, (eg., just adding the even indexed items to the odd ones)<br>
when I print the shape of the tensor directly after entering the forward method everything is normal</p>
<blockquote>
<p>print(x.shape)<br>
torch.Size([25, 1, 1, 800])</p>
</blockquote>
<p>However, when I add the following print statement (after the the first print)</p>
<blockquote>
<p>print(x[2,0,0,0])</p>
</blockquote>
<p>I get an error message:</p>
<blockquote>
<p>IndexError: index 2 is out of bounds for dimension 0 with size 2</p>
</blockquote>
<p>and the output of the first print becomes:</p>
<blockquote>
<p>torch.Size([2, 25, 1, 1, 800])</p>
</blockquote>
<p>Does anyone have any idea why this happens?<br>
Also, here is the code for my model:</p>
<pre><code>class EXT1(nn.Module):
    def __init__(self, ):
        super(EXT1, self).__init__()
        self.fc1 = nn.Linear(800, 800)
        self.fc2 = nn.Linear(800, 2)
        self.activation = nn.Tanh()

    def forward(self, x):
        print(x.shape,'===============')
        print(x[2,0,0,0])## THIS THE ADDITIONAL PRINT STATEMENT 
        
        x = self.activation(self.fc1(x))
        x = self.activation(self.fc2(x))

        return x
</code></pre>

ptrblck: <p>Do you see this issue for the last batch?<br>
If you are using a <code>DataLoader</code>, the last batch might be smaller, if the length of your dataset is not divisible by the <code>batch_size</code> without a remainder. In your case, the last batch might only contain two samples, thus you cannot index it using <code>[2, 0, 0, 0]</code>.</p>

Daw: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for the reply. No it is not the last batch, in fact I realized this issue appears when the model is initialized, but then when I start to load/feed the data to the model, x gets the normal size. Not sure why the additional dimension is added at the initialization.</p>
Custom forward on Sequential subclass

askliar: <p>Hi!<br>
I have a custom class <code>nn.CustomSequential</code>, which is subclassing <code>nn.Sequential</code>. Is there a way to call <code>forward</code> as defined in <code>nn.CustomSequential</code>?</p>
<p>So far it seems that it only calls <code>forward</code> method as defined in <code>nn.Sequential</code> class.</p>
Torch::max_pool2d in nn:sequential()

JeppeThagaardVP: <p>Hi,</p>
<p>Is it possible to use torch::max_pool2d in a sequential without creating a class first?</p>
<p>I tried to pass it to nn::Functional without any luck (see below) as the documention states that you can wrap any function in nn::Functional with arguments.</p>
<p>“nn::Sequential(<br>
nn::Functional(torch::max_pool2d, 2)<br>
)”</p>
Histogram Equalization

ShangxuanWu: <p>Is there a way in Pytorch to implement histogram equalization for tensors?</p>

ptrblck: <p>If you use <a href="http://pytorch.org/docs/master/torch.html?highlight=hist#torch.histc">torch.histc</a> without specifying <code>min</code> and <code>max</code>, the minimum and maximum values of the data are used.<br>
Would this be sufficient or do you need another method?</p>

ShangxuanWu: <p>I don’t think <code>torch.histc</code> is differentiable, right?</p>

angerhang: <p>Don’t think so. You might have to code up your own <code>histc</code> function to take the derivative.</p>
How to print the target values using dataloader

Naina_Dhingra: <pre><code>          for i, (inputs, targets) in enumerate(data_loader):
        
 
                    with torch.no_grad():
                           inputs = Variable(inputs)
                           targets = Variable(targets)
                           print("targets.data", targets.item())
</code></pre>
<p>It doesn’t seem to be giving the target value.</p>

ptrblck: <p>What does the print statement show instead?</p>

Naina_Dhingra: <p>It is printing the index numbers: targets.data tensor([0, 1, 2, 3, 4, 5]), if I print(target)</p>
<p>it gives error:<br>
print(“targets.data”, targets.item())<br>
ValueError: only one element tensors can be converted to Python scalars</p>
<p>, if I print( targets.item())</p>

ptrblck: <p><code>.item()</code> works only on one-element tensors.<br>
What is wrong about <code>print(target)</code>?</p>

Naina_Dhingra: <p>It seems to print the index not the values</p>

ptrblck: <p>Maybe these values are equal to the indices for the current batch.<br>
Could you set <code>shuffle=True</code> in your <code>DataLoader</code> and run your code again or alternatively check the output for multiple <code>target</code> tensors?</p>

Naina_Dhingra: <p>I used shuffle= True, I get the output as tensor([ 1192,  6908,  1663,  2771,  7581, 10296])</p>
<p>The class ranges from 1 to 84.<br>
Total length if data is 10364</p>

ptrblck: <p>Could you post the code for your <code>Dataset</code>?</p>

Naina_Dhingra: <pre><code>        data_loader = torch.utils.data.DataLoader(
        test_data,
        batch_size=opt.batch_size,
        shuffle=True,
        num_workers=opt.n_threads,
        pin_memory=True)
    test.test(data_loader, model, opt, test_data.class_names,criterion,test_logger)
    print("done")
</code></pre>
<p>test.py looks like:</p>
<pre><code> for i, (inputs, targets) in enumerate(data_loader):

    with torch.no_grad():
        inputs = Variable(inputs)
        targets = Variable(targets)
        print("targets.data", targets)
        outputs = model(inputs)</code></pre>

ptrblck: <p>Could you post the code of <code>test_data</code>, so that we could have a look how your targets are created?</p>
Concatenating Tensors While Maintaining Gradients for Autograd

UHMWPE: <p>Essentially, I have a list of tensors of size (3, ) that I want to concatenate into a single large tensor of size (200, 3). However, I don’t want them to be disconnected from the computation graph. A toy example of this would be:</p>
<pre><code>import torch
from torch.autograd import Variable

a = Variable(torch.Tensor([1., 3., 5.]), requires_grad=True)
b = Variable(torch.Tensor([2., 4., 6.]), requires_grad=True)
c = torch.matmul(a.t(), b)

print(torch.autograd.grad(c, a, retain_graph=True)[0])
print(torch.autograd.grad(c, b, retain_graph=True)[0])

z = torch.cat((a, b))
print(torch.autograd.grad(c, z)[0])
</code></pre>
<p>Is there anyway to concatenate tensors a, b (and realistically, a list of 200 of such tensors), such that the last line doesn’t error out? (i.e. need to preserve the computation graph)</p>

ptrblck: <p>Would <code>print(torch.autograd.grad(c, (a, b)))</code> just work?</p>
<p>Unrelated to this problem, but <code>Variables</code> are deprecated since PyTorch <code>0.4</code>. If you are using a newer version, you can use tensors directly.</p>

UHMWPE: <p>Hi, your solution only prints out the gradient for the first element in the tuple (i.e. a).</p>
<p>I’m aware of the second note, but I have to use a lot of older code for my research and it’s just becoming a force of habit these days. Thanks for the notice though!</p>
Text masking for BERT

DrunkAlex: <p>Hello world!<br>
I am doing some tasks connected with BERT and want to mask input text.<br>
For example: <code>He loves you. I know.</code><br>
Into: <code>[CLS] He loves you. [SEP] I know [SEP].</code><br>
And then tokenize it.<br>
How can I do this masking effectively?</p>

cherepanovic: <p>have you checked the original github repository[1]?<br>
On the fly there are some classes for training and finetuning, have you examined it?</p>
<p>[1] - <a href="https://github.com/google-research/bert" rel="nofollow noopener">https://github.com/google-research/bert</a></p>
Failed to compute dot product of torch.cuda.FloatTensor

lynone: <p>I used a GPU to compute the dot product of the output of neural networks and a torch.cuda.FloatTensor (both of them are stored in GPU) but got an error saying: TypeError: dot received an invalid combination of arguments - got (torch.cuda.FloatTensor) but expected (torch.FloatTensor tensor).</p>
<p>the codes are like<br>
p = torch.exp(vector.dot(ht))<br>
here vector is a torch FloatTensor and ht is the output of neural networks.</p>
<p>I’ve struggled with these things for days but still got no idea. Thanks in advance for any possible solution!</p>

smth: <p>it is saying that one of either <code>vector</code> or <code>ht</code> is not a <code>cuda.FloatTensor</code> but is a <code>FloatTensor</code>. Find it and call <code>.cuda()</code> on it.</p>

lynone: <p>Yes that’s it. Thanks!</p>

Ingrid_Bernal: <p>Hello, Im having a similar issue, im trying to run in gpu a torch.dot with the following code;</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dc41d8d12ee4e145852b224eef241715a7e2f1d6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dc41d8d12ee4e145852b224eef241715a7e2f1d6" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dc41d8d12ee4e145852b224eef241715a7e2f1d6_2_690x133.jpeg" alt="loss" data-base62-sha1="vqu7qYz0hCGC2DwzwLwy6IcRNZQ" width="690" height="133" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dc41d8d12ee4e145852b224eef241715a7e2f1d6_2_690x133.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/d/dc41d8d12ee4e145852b224eef241715a7e2f1d6.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/dc41d8d12ee4e145852b224eef241715a7e2f1d6.jpeg 2x" data-dominant-color="272626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">899×174 20.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>but im obtaining the error;</p>
<p>RuntimeError: _th_dot is not implemented for type torch.cuda.LongTensor</p>
<p>anyone knows what should I modify to make it work ?</p>
<p>Thanks.</p>

ptrblck: <p>Try to cast both inputs to <code>float32</code>:</p>
<pre><code class="lang-python">embedding_output = embedding_output.float()
embedding_target =  embedding_target.float()
cos = torch.dot(...)
</code></pre>

Ingrid_Bernal: <p>Thank you ! it worked !</p>
Torch.split in batches

Alberto167: <p>Hello,</p>
<p>Does anyone know how to split batches of tensors in different positions each?</p>
<p>I am trying to separate my tensors in two parts according to the <code>split_position</code>, which is different for each sample</p>
<pre><code class="lang-auto">batch.shape = torch.Size([128, 68, 1])      # (batch_size, max_len, 1)
split_positions.shape = ([128])             # split position of each sample
</code></pre>
<p>Thanks,</p>

ptrblck: <p>As a result you would expect a list of 128 tupels with two tensors inside?</p>

Alberto167: <p>I would like to have 2 tensors of the same size<code>([128,68,1])</code> but one containing the left part of each sample and a lot of zeros to match the <code>max_len</code> which is 68, and the other containing the right part and then a lot of zeros.</p>
<p>This is an example for a single, small sample:</p>
<pre><code class="lang-auto">max_len = 10
split = 3
left_weights = torch.tensor ([10,3,1,20,4,0,0,0,0,0])
</code></pre>
<pre><code class="lang-auto"># separate in the split position
l, m_l = left_weights[0:split], left_weights[split:]

# fill with zeros
l = F.pad(l, (0,max_len - len(l)))
m_l = F.pad(m_l, (0,max_len - len(m_l)))
</code></pre>
<p>The results should look like this:</p>
<pre><code class="lang-auto">l  = tensor([ 10,   3,   1,   0,   0,   0,   0,   0,   0,   0]),
m_l = tensor([ 20,   4,   0,   0,   0,   0,   0,   0,   0,   0])
</code></pre>

ptrblck: <p>Ok, thanks for the info.<br>
I don’t know an elegant way to achieve your results.<br>
If your results would look like this:</p>
<pre><code class="lang-auto">l  = tensor([10, 3, 1, 0, 0, 0, 0, 0, 0, 0]),
m_l = tensor([0, 0, 0, 20, 4, 0, 0, 0, 0, 0,])
</code></pre>
<p>, i.e. there the zeros in front of the right part, you could use <code>.scatter_()</code>.</p>
<p>In case this helps, here is the code to achieve this, but unfortunately this isn’t exactly, what you wanted:</p>
<pre><code class="lang-auto"># Init data tensor and splits
x = torch.randn(20, 10)
splits = torch.empty(20, dtype=torch.long).random_(1, 10)

# Calculate size for result tensor
new_rows = x.size(0) * 2
offset = x.size(0)
z = torch.zeros(new_rows, 10)

# Calculate scatter indices
# The "left" part goes into z[:offset], the "right" part goes into z[offset:]
split_scatter = [torch.cat((torch.ones(1, split, dtype=torch.long) * idx,
                            torch.ones(1, max_len-split, dtype=torch.long) * (idx + offset)),
                            dim=1) for idx, split in zip(range(0, new_rows), splits)]
split_scatter = torch.cat(split_scatter)

# Apply scatter
z.scatter_(0, split_scatter, x)

# Cut tensors
x1 = z[:offset]
x2 = z[offset:]
</code></pre>

Alberto167: <p>Yes, this also works for me, thank you very much!</p>

nikita_kapoor: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> If I have a tensor of shape [200, 5, 76] // Frames batchsize Voc<br>
How I can use split to get a list of 5 that is the batch size , two dimensional tensors?<br>
I want to evaluate and I need them individually to calculate edit distance on each of them.</p>
Loss WGAN-GP majority is GP

Bauke_Brenninkmeijer: <p>I’m using the WGAN-GP architecture, but I’m getting a loss that is about 50000, of which 99% is from the gradient penalty and 1% from the normal WGAN loss. I’m not very experienced with this architecture, so I’m wondering whether the gradient penalty should be the major factor in the loss or not. My intuition says it shouldn’t, but I’m not sure what to change in that case.</p>
Something like Batch Mat Mul bmm

a53153fb407732ef019a: <p>I want to implement operation like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/045eac4968b747f87c79ce5426f95f6a808cb330.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/045eac4968b747f87c79ce5426f95f6a808cb330" title="313453992"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/045eac4968b747f87c79ce5426f95f6a808cb330_2_375x500.jpeg" alt="313453992" data-base62-sha1="CEKf0aNfrt8eBeQ5D9oeGCq9dm" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/045eac4968b747f87c79ce5426f95f6a808cb330_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/0/045eac4968b747f87c79ce5426f95f6a808cb330_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/045eac4968b747f87c79ce5426f95f6a808cb330_2_750x1000.jpeg 2x" data-dominant-color="B1B1B0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">313453992</span><span class="informations">1080×1440 52 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
in which each square represents a matrix,line between square means matrix multiplication</p>
<p>Assume I have 4 matrix parameters and each of my inputs have 4 matrix(or vector),theses four parameter matrix and input matrix are multiplicated respectively for n batches.</p>
<p>I implement this operation via <code>bmm</code> but its too inefficient, I just stack my parameters to make its batched like my inputs, like this</p>
<pre data-code-wrap="python"><code class="lang-python">W = [nn.Linear(10,10,bias=False) for _ in range(4)]
batched_W = [W[i%4].weight.unsqueeze(0) for i in range(4*batch_size)]
batched_W = torch.cat(batched_W,dim=0).cuda()
out = torch.bmm(batched_W,x) # x is my input
</code></pre>
<p>since I use GPU for training and <code>.weight</code>  operation seems put parameters back to cpu, so I must add <code>.cuda()</code>, I think it’s too inefficient, but I can’t come up with other idea.</p>
How to manually set the weights in a two layer linear model?

Deeply: <p>A simple model like this one:</p>
<p><code>model = torch.nn.Sequential(torch.nn.Linear(10, 1, bias=False))</code></p>

ptrblck: <p>You would just need to wrap it in a <code>torch.no_grad()</code> block and manipulate the parameters as you want:</p>
<pre><code class="lang-python">model = torch.nn.Sequential(nn.Linear(10, 1, bias=False))

with torch.no_grad():
    model[0].weight = nn.Parameter(torch.ones_like(model[0].weight))
    model[0].weight[0, 0] = 2.
    model[0].weight.fill_(3.)
    ...
</code></pre>