akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory Leak During Singular Value Clipping on Weights

Cade_Gordon1: <p>I’m currently training a GAN that enforces a constraint on the singular values s.t. they are less than or equal to one. This involves me manually reassigning the weights during training time at certain points to enforce a 1-Lipschitz constraint. When I train my model, after a certain period of time it crashes due to running out of memory during the SVD operation. It’s initial memory allocation is around 7-8GB during initialization, but then balloons to 11GB and crashes.</p>
<pre><code class="lang-python">def singular_value_clip(w):
    dim = w.shape
    if len(dim) &gt; 2:
        w = w.reshape(dim[0], -1)
    u, s, v = torch.svd(w, some=True)
    s[s &gt; 1] = 1
    return (u @ torch.diag(s) @ v.t()).view(dim)
</code></pre>
<p>Then during training:</p>
<pre><code class="lang-python">for epoch in tqdm(range(start_epoch, epochs)):
        if epoch % 5 == 0:
            for module in list(dis.model3d.children()) + [dis.conv2d]:
                # discriminator only contains Conv3d, Conv2d, BatchNorm3d, and ReLU
                if type(module) == nn.Conv3d or type(module) == nn.Conv2d:
                    module.weight.data = singular_value_clip(module.weight)
                elif type(module) == nn.BatchNorm3d:
                    gamma = module.weight.data
                    std = torch.sqrt(module.running_var)
                    gamma[gamma &gt; std] = std[gamma &gt; std]
                    gamma[gamma &lt; 0.01 * std] = 0.01 * std[gamma &lt; 0.01 * std]
                    module.weight.data = gamma
</code></pre>
<p>I’ve even tried enclosing the contents of the if blocks in a <code>with torch.no_grad():</code>, but that has no effect (and may potentially just be plain improper). Are there any glaring errors that might be causing this memory leak?</p>

Cade_Gordon1: <p>Update, the Cuda memory keeps increasing and I’ve also tried deleting GPU variables after usage. Should I be clearing the cache or something similar to resolve this issue?</p>

ptrblck: <p>Could you remove the usage of the <code>.data</code> attribute, as it might yield unwanted side effects?</p>

Cade_Gordon1: <p>When I did that in the past I got an error during assignment, is there something else I should do when I drop the <code>.data</code> attribute?</p>

ptrblck: <p>Assuming you are manipulating the parameters before (or after) each iteration, you could wrap the code into a <code>with torch.no_grad()</code> block and use <code>.copy_</code> to fill the parameters with the new values.</p>

Cade_Gordon1: <p>Thanks for the help! It works now after this and some aggressive use of the <code>del</code> command for no longer needed loss/prediction values.</p>
Create class that acts both as nn.Parameter and as nn.Module

smonsays: <p>To facilitate managing trainable hyperparameters, I am looking for a way to create a class <code>Hyperparameter</code> that acts both as a <code>nn.Parameter</code> and a <code>nn.Module</code>. In particular, I would like to use <code>Hyperparameter</code> objects both as a <a href="https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html?highlight=parameter#torch.nn.parameter.Parameter" rel="noopener nofollow ugc"><code>nn.Parameter</code></a> (e.g. for algebraic manipulations) but still have access to the interface provided by <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=module#torch.nn.Module" rel="noopener nofollow ugc"><code>nn.Module</code></a> to for example store the objects in a <code>nn.ModuleDict</code> along with other modules or use methods like <code>zero_grad()</code>, <code>parameters()</code>.</p>
<p>I tried to accomplish this through multiple inheritance but I think this might be dangerous:</p>
<pre><code class="lang-python">import torch

class Hyperparameter(torch.nn.Parameter, torch.nn.Module):
    def __new__(cls, tensor, name):
        return torch.nn.Parameter.__new__(cls, data=tensor)

    def __init__(self, tensor, name):
        torch.nn.Parameter.__init__(self)
        torch.nn.Module.__init__(self)
        self.register_parameter(name, self)


hp1 = Hyperparameter(torch.ones(5), "test1")
hp2 = Hyperparameter(torch.ones(8), "test2")

hp_dict = torch.nn.ModuleDict({"hp1": hp1, "hp2": hp2})
hp_dict.to(torch.device("cpu"))
# KeyError: "attribute 'data' already exists"
</code></pre>
<p>This works for the things I described, but calling <code>to()</code> throws an error. I think something is no longer as <code>nn.Module</code> expects it to be, but I am not sure what it is.</p>
Optimization changing the gradient variables

Gabriele_Orlando: <p>Hi,<br>
I have the following problem:</p>
<p>I have :</p>
<ol>
<li>a model M(x,p), where x is an input tensor and p the parameters of the model that returns a single float value as output.</li>
<li>a set of inputs x</li>
<li>a set of labels y for training</li>
<li>a loss L(x,y)</li>
</ol>
<p>I want to find the parameters p that minimize the loss L of argmin_x( M(x,p) ) and y. Basically, I want first to find the x’ that stays in a minimum of M(x,p), and I want to find the parameters that minimize L(x’,y).</p>
<p>what I did right now is probably very naive:</p>
<pre><code class="lang-auto">optimizer1 = torch.optim.AdamW(M.parameters())
for e1 in range(100):
     optimizer2 = torch.optim.AdamW(x)
     for e2 in range(100):
            out = M(x)
            out.backward()
            optimizer2.step()
            optimizer2.zero_grad()
    lossValue = L(x,y)
    lossValue.backward()
    optimizer1.step()
    optimizer1.zero_grad()
</code></pre>
<p>In this way the optimization performed on the parameters does not converge at all (so optimizer1). Is there a better way to solve the problem? Might it be a problem of the fact that only the gradients coming from the inner optimization are not taken into consideration? Eventually, is there a way to pass them to the optimization of the parameters?</p>
<p>Thanks in advance</p>

albanD: <p>Hi,</p>
<p>I am not very familiar with this kind of tasks and why they might not converge. But <a href="https://github.com/facebookresearch/higher/" class="inline-onebox">GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.</a> seems very close to what you’re trying to do no?</p>
Load image dataset with text ground truth

Hamza_Ahmed: <p>I new on pytorch lib and trying to load a dataset i acquire online. Its a OCR in Urdu language. the structure of the dataset is<br>
-Train_image:<br>
-name1.jpg<br>
-name2.jpg</p>
<p>-train_labels.csv<br>
-column 1 (Files name)<br>
-name1<br>
-name2<br>
-column 2 (text in the corrosponding file)<br>
-label1 (urdu text)<br>
-label2 (urdu text)</p>
<p>now how can i load this data to train a CNN and A RNN for detection and recognition?</p>

Dwight_Foster: <p>You could write a custom dataset. The pytorch tutorial can be found <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">here</a>.</p>
TypeError: 'NoneType' object cannot be interpreted as an integer

111462: <p>I want to classify cat and dog images.<br>
So I downloaded images from Kaggle, and divided into train and validation set.</p>
<p>here is my code…</p>
<pre><code class="lang-auto">class TrainImageFolder(Dataset):
    def __init__(self, path, transform=None):
        self.path = path
        self.label = [id_ for id_ in os.listdir(path)]
        self.ids = list()
        self.transform = transform
        for i in self.label:
            for j in os.listdir(os.path.join(path, i)):
                self.ids += [i + '_' + j]

    def __len__(self):
        len(self.ids)

    def __getitem__(self, index):
        image = Image.open(self.ids[index][4:])
        image = image.convert('RGB')
        image = transforms.ToTensor()

        if (self.ids[index][:3] == 'Cat'):
            label = 0
        else:
            label = 1

        if self.transform:
            image = self.transform(image)

        return image, label

transform = transforms.Compose([transforms.RandomResizedCrop(224),
                                transforms.RandomHorizontalFlip(),
                                transforms.ToTensor()])

dataset = TrainImageFolder('PetImages/train')
dataloader = torch.utils.data.DataLoader(dataset, batch_size=8)
for i, (img, bbox, label, difficult) in tqdm(enumerate(dataloader)):
    continue
</code></pre>
<p>and error is…</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:/Users/ge971/PycharmProjects/myVGG16/dataset.py", line 49, in &lt;module&gt;
    for i, (img, bbox, label, difficult) in tqdm(enumerate(dataloader)):
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\tqdm\std.py", line 1166, in __iter__
    for obj in iterable:
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 435, in __next__
    data = self._next_data()
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 474, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\sampler.py", line 67, in __iter__
    return iter(range(len(self.data_source)))
TypeError: 'NoneType' object cannot be interpreted as an integer

Process finished with exit code 1

</code></pre>
<p>How should I fix my code? Please help me…</p>

Dwight_Foster: <p>In your getitem code you change your image to a transform for some reason in this line</p>
<pre><code class="lang-auto">        image = transforms.ToTensor()
</code></pre>
<p>You cannot do this and that is probably why you are getting the error. You should just delete this line.</p>
What is the difference between "Tensor A = B" and "Tensor A.set_data(B)" in LibTorch?

crissallan: <p>Hi<br>
I’d like to know: what is the difference between “Tensor A = B” and “Tensor A.set_data(B)” in LibTorch?<br>
I found that when I try to change the value of a model’s(networks’) parameter tensor, I could only use A.set_data(B) rather than A = B.</p>
<p>And using A.set_data(B) will lead to an error when execute the .backward() method.</p>

ptrblck: <p><code>set_data</code> will shallow-copy all non-autograd <code>TensorImpl</code> fields and will not bump the version counter, if I’m not mistaken. I think the equivalent would be an assignment to the <code>.data</code> attribute in Python, which is not recommended as it might yield unwanted side effects.</p>
<p>The <code>Tensor A = B;</code> operation will create a new tensor pointing to <code>B</code> and should not yield any side effects.</p>

crissallan: <p>Thanks for the reply. Strange! When I use “model.named_parameters[“name”] = a_tensor” to change some parameters of the model, it seems that it doesn’t work,  so I have to use .set_data() instead of it. Is there any solutions for this?</p>

ptrblck: <p>I don’t think your code would work, as you would have to call the <code>named_parameters</code> method as:</p>
<pre><code class="lang-python">model.named_parameters()["key"]
</code></pre>
<p>Yes, I think this is expected as in the Python frontend, as you would need to copy the data into the parameters directly using this workflow.<br>
I assume you are getting errors from Autograd? If so, use <code>at::NoGradGuard guard;</code> to make sure Autograd doesn’t track these copies in the current scope.</p>

crissallan: <p>Sorry, I made a typo there. My “error” is that after I execute the “model.named_parameters()[“key”] = some_tensor” ,  the parameters tensor is the same as before:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/5970a05b5b8cd90d250fe0adc039661ad318a7e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/5970a05b5b8cd90d250fe0adc039661ad318a7e7" title="demo_error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5970a05b5b8cd90d250fe0adc039661ad318a7e7_2_690x409.png" alt="demo_error" data-base62-sha1="cLdNl4UDRZIw7gXynF21fcANDQX" width="690" height="409" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5970a05b5b8cd90d250fe0adc039661ad318a7e7_2_690x409.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5970a05b5b8cd90d250fe0adc039661ad318a7e7_2_1035x613.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/5970a05b5b8cd90d250fe0adc039661ad318a7e7_2_1380x818.png 2x" data-dominant-color="323334"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">demo_error</span><span class="informations">2726×1618 378 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Dilation before IoU

TonyTr: <p>I am trying to do the following:</p>
<ol>
<li>Do one iteration of dilation on the output of the model (in validation or testing).</li>
<li>Compute the IoU after the dilation for each class.</li>
</ol>
<p>The reason for the dilation step is because I am willing to tolerate some error for the second class, for example, if the output of the model is a thin line and the ground truth is a bit thicker, normally the IoU will suffer from that but for me a thin line is acceptable. However, for the first class (background) or the third class, I don’t want to do the dilation step.</p>
<p>Currently I do a dilation of the whole output as provided below.</p>
<p>This code works great:</p>
<pre><code class="lang-python">def iou(output, label, n_classes):
    ious = []
    pred = torch.argmax(output, dim=1).squeeze(1).view(-1)
    target = label.view(-1)
    SMOOTH = 1e-6
    for idx in range(n_classes):
        preds_inds = pred == idx
        target_inds = target == idx
        intersection = (preds_inds &amp; target_inds).float().sum()
        union = (preds_inds | target_inds).float().sum()
        iou = (intersection + SMOOTH) / (union + SMOOTH)
        ious.append(iou.item())
    return np.array(ious)
</code></pre>
<p>Adding the dilation step:</p>
<pre><code class="lang-python">def iou(output, label, n_classes):
    ious = []
    get_cuda_device = "cuda:0"
    cuda_check = output.is_cuda
    if cuda_check:
        get_cuda_device = output.get_device()
    kernel = np.ones((5, 5), np.uint8)
    pred = torch.argmax(output, dim=1).squeeze(1).view(-1).detach().cpu().numpy() * 255
    pred = cv2.dilate(pred.astype(np.uint8), kernel, iterations=1) / 255
    pred = torch.from_numpy(pred).long().to(get_cuda_device)
    target = label.view(-1)
    SMOOTH = 1e-6
    for idx in range(n_classes):
        preds_inds = pred == idx
        target_inds = target == idx
        intersection = (preds_inds &amp; target_inds).float().sum()
        union = (preds_inds | target_inds).float().sum()
        iou = (intersection + SMOOTH) / (union + SMOOTH)
        ious.append(iou.item())
    return np.array(ious)
</code></pre>
<p>Now I get the following error:</p>
<pre><code class="lang-auto">intersection = (preds_inds &amp; target_inds).float().sum()
RuntimeError: CUDA out of memory. Tried to allocate 7440.75 GiB (GPU 0; 10.76 GiB total capacity; 463.73 MiB already allocated; 9.33 GiB free; 584.00 MiB reserved in total by PyTorch)
</code></pre>
<p>Can someone suggest a fix or even a better way to achieve what I described above (not necessarily using a dilation) .</p>
<p>EDIT: Fixed the problem by replacing</p>
<pre><code class="lang-python">pred = torch.from_numpy(pred).long().to(get_cuda_device)
</code></pre>
<p>with</p>
<pre><code class="lang-python">pred = torch.from_numpy(pred).long().squeeze(1).view(-1).to(get_cuda_device)
</code></pre>
<p>My question still holds tho, is there a better way to do this?</p>
How to get the op's name in the TORCH_LIBRARY_IMPL

leslie-fang-intel: <p>Here is the piece of my code, I register the conv2d for the new key.<br>
When dispatch into the corresponding function, How can I know it’s conv2d which is being dispatched?</p>
<pre><code class="lang-auto">template&lt;class Redispatch, Redispatch* F, class Ret, class ArgList&gt; struct CPU_WrapFunction_ {};

template&lt;class Registered,  class Redispatch, Redispatch* F&gt;    
struct WrapFunction_CPU final {
  using type = CPU_WrapFunction_&lt;Redispatch, F,
                                 typename guts::function_traits&lt;Registered&gt;::return_type,
                                 typename guts::function_traits&lt;Registered&gt;::parameter_types&gt;;  
};

template&lt;class Redispatch, Redispatch* F, class Ret, class... Args&gt;
struct CPU_WrapFunction_&lt;Redispatch, F, Ret, guts::typelist::typelist&lt;Args...&gt;&gt; {  
  static Ret call(Args... args) {
  
    **//How to get the op's name here**

    return (*F)(args...);
  }
};

#define KERNEL_CPU(FUNC, REGISTER_NAME, SIGNATURE) \
	m.impl(TORCH_SELECTIVE_NAME("aten::" REGISTER_NAME), \
	  &amp;WrapFunction_CPU&lt;SIGNATURE, SIGNATURE, &amp;FUNC&gt;::type::call);

TORCH_LIBRARY_IMPL(aten, mykey, m){
  KERNEL_CPU(at::conv2d, "conv2d", Tensor (const Tensor &amp;, const Tensor &amp;, const c10::optional&lt;Tensor&gt;&amp;, IntArrayRef, IntArrayRef, IntArrayRef, int64_t))
}
</code></pre>
[DataParallel] How to split the forward function into multiple steps

kingo: <p>Hello,</p>
<p>I am implementing a custom network, which first computes a latent encoding, and then does some forward passes. I am wondering whether this kind of network can work on nn.DataParallel. I assume nn.DataParallel only does some parallelism on self.forward. Does it also handle the custom functions as well?</p>
<pre><code class="lang-auto">def MyNetwork(nn.Module):
    def preprocess(self, a):
        self.encoding = encode(a)

    def forward(self, b):
        return self.encoding + self.fc(b)

model = nn.DataParallel(MyNetwork())
# Pre encoding
model.module.preprocess(a)
# Parallel data pass
output1 = model(b1)
output2 = model(b2)
</code></pre>

ptrblck: <p><code>nn.DataParallel</code> will not use the passed GPUs in <code>model.module.preprocess</code>, since the <code>forward</code> method will be used to create the model clones, split the data etc., as you’ve already assumed.<br>
You could call other functions inside the <code>forward</code> method, if you want to use all devices.</p>
Tensor dimension ordering for efficient memory access

hohohihi: <p>Hello, I have a question about memory layout and access efficiency.<br>
I will explain about the situation and let me clarify the question.</p>
<p>I have certain amount of nodes(points, vertices). Each point has its own embedding of specific length(like, 3~1024).<br>
So, this x’s dimension is [batch_size, embedding length, numbers of points];<br>
(By the way, My batch size is always ‘1’.)</p>
<p>And i have an array which consists of random indices whose name is idx_p0 (like, [0, 71, 234, 22, 0 …, 2].</p>
<p>Using this idx_p0, i do random access to this x<br>
y = x[:,:,idx_p0]</p>
<p>Here comes the question.<br>
As, i do random access to the x. Then i will cause memory access(toward register file of gpu).<br>
But i want to do this in efficient way.</p>
<p>Is it wise to change the dimension of the x to this ‘x_bar = x.transpose(2,1).contiguous()’.<br>
Then the ‘x_bar’ becomes [batch_size, numbers of points, embedding length]. And do random access like this ‘y = x[:,idx_p0,:]’.</p>
<p>I want to know how the pytorch save the 3 dimension tensor.<br>
Oh! my batch_size is usually only 1.</p>
<p>Thank you.</p>
Assigning value from single dimension to multidimension array

cbd: <p>I have two tensor “a” and “b” with size [1,5,256,256] and [5] respectively. Let’s say b=[0 1 2 3 4].</p>
<p>I want to assign b[0] to a[0,0,:,:] , b[1] to a[0,1,:,:] etc. I tried to assign as below code but it shows the error " RuntimeError: The expanded size of the tensor (256) must match the existing size (5) at non-singleton dimension 2.  Target sizes: [5, 256, 256].  Tensor sizes: [5]"</p>
<p>a=torch.zeros(1,5,256,256)<br>
b=torch.linspace(0,4,5)<br>
a[0,:,:,:]=b</p>

Nikronic: <p>Hi,</p>
<p>You just need to change the view of <code>b</code> to match in dimensions:</p>
<pre><code class="lang-auto">a[0,:,:,:]=b.view(-1, 1, 1)
</code></pre>
<p>It’s about <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">Broadcasting semantics — PyTorch 1.7.0 documentation</a> which has some criteria.</p>
<p>Bests</p>

cbd: <p>Thanks. It solve my issue.</p>
AttributeError: 'JpegImageFile' object has no attribute 'read'

Paula_Hwang: <p>I am a beginner and I am learning to code an image classifier. My goal is to create a predict function.</p>
<p>In this project, I want to use the predict function to recognize different flower species. So I could check their labels later.</p>
<p>Attempt to fix: Unfortunately, the error is still persistent. I have already tried these codes:</p>
<blockquote>
<p>AttributeError: ‘JpegImageFile’ object has no attribute ‘read’</p>
</blockquote>
<p><strong>Code</strong></p>
<pre><code class="lang-auto">def predict(image, model, topk=5):
    ''' 
      Predict the class (or classes) of an image using a trained deep learning model.
      Here, image is the path to an image file, but input to process_image should be                                                         
      Image.open(image)
    '''
    img = process_image(Image.open(image))
    img = torch.from_numpy(img).type(torch.FloatTensor) 

    output = model.forward(img)
    probs, labels = torch.topk(output, topk)        
    probs = probs.exp()

    # Reverse the dict
    idx_to_class = {val: key for key, val in model.class_to_idx.items()}
    # Get the correct indices
    top_classes = [idx_to_class[each] for each in classes]

    return labels, probs
</code></pre>
<p><strong>Passing</strong></p>
<pre><code class="lang-auto">probs, classes = predict(image, model)
print(probs)
print(classes)
</code></pre>
<p><strong>Errors</strong></p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-32-b49fdcab5791&gt; in &lt;module&gt;()
----&gt; 1 probs, classes = predict(image, model)
      2 print(probs)
      3 print(classes)

&lt;ipython-input-31-6f996290ea63&gt; in predict(image, model, topk)
      5       Image.open(image)
      6     '''
----&gt; 7     img = process_image(Image.open(image))
      8     img = torch.from_numpy(img).type(torch.FloatTensor)
      9 

/opt/conda/lib/python3.6/site-packages/PIL/Image.py in open(fp, mode)
   2587         exclusive_fp = True
   2588 
-&gt; 2589     prefix = fp.read(16)
   2590 
   2591     preinit()

AttributeError: 'JpegImageFile' object has no attribute 'read'
</code></pre>
<p><strong>I want to have these similar result:</strong></p>
<pre><code class="lang-auto">tensor([[ 0.5607,  0.3446,  0.0552,  0.0227,  0.0054]], device='cuda:0')   
tensor([[  8,   1,  31,  24,   7]], device='cuda:0')
</code></pre>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/typeerror-img-should-be-pil-image-got-class-str/49644/5">here</a>.</p>

Megga_Dismas: <p>Hello colleague, could you have gotten help and explain to me about how you fixed the issue?<br>
Best regards</p>
What is TemporalSubSampling in PyTorch?

Jingles: <p>Hi there,</p>
<p>Is <a href="https://github.com/torch/nn/blob/master/doc/convolution.md#nn.TemporalSubSampling" rel="noopener nofollow ugc">torch’s (not PyTorch) TemporalSubSampling</a>, simply equivalent to PyTorch’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html" rel="noopener nofollow ugc">Conv1d</a>?</p>
<blockquote>
<p>Applies a 1D sub-sampling over an input sequence composed of <code>nInputFrame</code> frames. The <code>input</code> tensor in <code>forward(input)</code> is expected to be a 2D tensor (<code>nInputFrame x inputFrameSize</code> ). The output frame size will be the same as the input one (<code>inputFrameSize</code> ).</p>
</blockquote>
<p>The output value of the layer can be precisely described as:</p>
<pre><code class="lang-auto">output[t][i] = bias[i] + weight[i] * sum_{k=1}^kW input[dW*(t-1)+k][i]
</code></pre>
<p>See TemporalSubSampling’s <a href="https://github.com/torch/nn/blob/master/TemporalSubSampling.lua" rel="noopener nofollow ugc">Lua code</a> here.</p>
<p>But they have <a href="https://github.com/torch/nn/blob/master/doc/convolution.md#temporalconvolution" rel="noopener nofollow ugc">TemporalConvolution</a>, from the description it looks more like PyTorch’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html" rel="noopener nofollow ugc">Conv1d</a>, so I’m confuse what is <a href="https://github.com/torch/nn/blob/master/doc/convolution.md#nn.TemporalSubSampling" rel="noopener nofollow ugc">TemporalSubSampling</a> in PyTorch.</p>
Captum usage with CTC loss function

factored: <p>I’m trying to use Captum’s IntegratedGradients for input attribution with a 1D convolution network which uses the CTC loss function. I have an input of:</p>
<p>(batchsize, 1, 4096)</p>
<p>And an output of:</p>
<p>(batchsize, 420, 5)</p>
<p>Where there is a sequence length of 420, and each part of the sequence has 5 possible classifications (or words). So there are 420 different classifications making this a multi-multi class classification.</p>
<p>However, I’m having trouble figuring out how to use the “target” argument in the IntegratedGradients.attribute function even with using a batchsize of 1.</p>
<p>I would like to get the input attribution for the argmax along the 2nd dimension (420) so I can see what portion of the input is used for each of the 420 classifiers.</p>
<p>Ultimately, I’d also like to do this for a batch size larger then one as well.</p>
<p>Any help or direction would be appreciated! Thanks!</p>
Supplying weights to nn.functional.conv2d in PyTorch

mohit117: <p>I am trying to learn the weights of a 3x3 conv2d layer accepting 3 channels and outputting 3 channels. For this discussion consider bias=0 in each case. However, the weights of the conv layer are learned indirectly. I have a 2 layered Multi layer perception having 9 nodes in first layer and 9 in the second. The weights of the 2d conv layer are then precisely the weights learned using this MLP i.e. <code>nn.Linear(9,9)</code> . I understand in this case I will have to use <code>nn.functional.conv2d(input,weight)</code> . But how exactly to extract the weights from MLP and use it for convolution is not clear and can think of the following.</p>
<pre><code class="lang-auto">class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hyper = nn.Linear(9, 9)  # output the required number of parameters

  def forward(self, x):
    # do stuff with self.hyper(x)  
    y = nn.Functional.conv2d(x, self.hyper.weight.reshape((3, 3, 3, 3)))  # add padding and other parameters
    return y
</code></pre>
<p>Can some one provide a short, dummy code in PyTorch to achieve this training configuration allowing backpropagation? Will the above code be problematic for PyTorch’s backpropagation?</p>
New_refcount != 1 INTERNAL ASSERT FAILED at /pytorch/c10/util/intrusive_ptr.h:183

yigu: <p>Hi, I meet the mistak by Pytorch 1.3.0, which is  “new_refcount != 1 INTERNAL ASSERT FAILED at c10/util/intrusive_ptr.h:183, please report a bug to PyTorch. intrusive_ptr: Cannot increase refcount after it reached zero.”, and don’t know how to solve it.<br>
Anyone can give me suggestions?<br>
Thank you very much!</p>

ptrblck: <p>Could you update PyTorch to the latest stable release (<code>1.7.1</code>) or the nightly binary and rerun your code?<br>
You might be running into an already fixed issue. If you are still seeing this error, could you post an executable code snippet to reproduce it?</p>

yigu: <p>Thank you for your attention.The bug I meeted is in Docker, which use Pytorch1.3.0 branch.<br>
Then I found the mistake of the version of GCC, after fixed it by changing GCC, the mistake doesn’t appear again.And I think I have solved the problem.<br>
Thank you again!</p>
Why inPlace is by default set to False for nn.LeakyRelu

aknirala: <p>By default parameter inPlace for nn.LeakyRelU has been set to False. I was wondering what are the repercussions for that, and why that is the default behavior?</p>
<p>I believe if we do not explicitly set it to True then a bit more memory would be used as a copy of that particular layer would need to be saved. LeakyRelu should not have any parameters (right?), so ideally IMHO inplace=True should be default behavior. Why would anyone want it to be False?</p>

ptrblck: <p>Inplace operations can overwrite values required to compute gradients and might thus yield an error message such as:</p>
<pre><code class="lang-python">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1024]], which is output 0 of LeakyReluBackward1, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>These limitation is also described <a href="https://pytorch.org/docs/master/notes/autograd.html#in-place-operations-with-autograd">here</a> and you should thus be careful with its usage.<br>
The default behavior of setting <code>inplace=False</code> would work without raising any errors and you are of course free to check, if <code>inplace=True</code> would work for your model. If PyTorch doesn’t raise any errors, the gradient calculation will be right.</p>
Row normalized custom loss function

i-cant-code: <p>Hi</p>
<p>I am trying to implement a regression loss function where I have multiple outputs but I want the loss function to be something like sum(Y_prediction-Y_target)/max(Y_target),</p>
<p>so far I have:</p>
<p>def my_loss(output, target):</p>
<pre><code>maxes = torch.max(target, dim = 1)

diff = (output - target)
diff_sums = torch.sum(diff,dim = 1)

loss = torch.div(diff_sums,maxes)
return loss
</code></pre>
<p>but I am getting error:</p>
<p>div(): argument ‘other’ (position 2) must be Tensor, not torch.return_types.max</p>

dazzle-me: <p><a href="https://pytorch.org/docs/stable/generated/torch.max.html" rel="noopener nofollow ugc">Take a look</a> on what <code>torch.max()</code> returns when 2 arguments is passed :<br>
<code>torch.max(input, dim, keepdim=False, *, out=None) -&gt; (Tensor, LongTensor)</code></p>
<p>So, I think your function is right, except first line of code, it should be :</p>
<pre><code class="lang-auto">maxes, _ = torch.max(target, dim=1)
</code></pre>

i-cant-code: <p>thank you! So I made this change and I believe it worked, but was not very effective of a loss function unfortunately. thank you though!</p>
Torchvision save image creates a messed up image

Dwight_Foster: <p>Hello I am trying to convert avi files into separate png files. My code is here:</p>
<pre><code class="lang-auto">for i in tqdm(range(num_videos)):
    if not os.path.isdir(args.VideoDir):
        print("Input folder does not exist!")
        break
    if not os.path.isdir(args.OutputDir):
        os.mkdir(args.OutputDir)
    path = args.VideoDir + videos[i]

    path = args.VideoDir + videos[i]
    images = io.read_video(path)[0]
    if images.shape[0] &lt; args.numframes:
        for n in range(images.shape[0]):
            folder_path = args.OutputDir + "scene_" + str(1000 + i)
            if not os.path.isdir(folder_path):
                os.mkdir(folder_path)
            image = images[n]
            image = torch.reshape(image, (image.shape[2], image.shape[0], image.shape[1]))
            image_path = folder_path + "/" + "col_high_" + str(n).zfill(4) + ".png"
            utils.save_image(image.float(), fp=image_path)
    else:
        for n in range(args.numframes):
            folder_path = args.OutputDir + "scene_" + str(1000 + i)
            if not os.path.isdir(folder_path):
                os.mkdir(folder_path)
            image = images[n]
            image = image.view(image.shape[2], image.shape[0], image.shape[1])
            image_path = folder_path + "/" + "col_high_" + str(n).zfill(4) + '.png'
            utils.save_image(image.type(torch.float32), image_path)
    if not args.keep_video:
        os.remove(path)
</code></pre>
<p>The output image ends up being totally wrong. I have tried to do the same with mp4 files but the result doesn’t change. Here is an example image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5a16e29f503bf8b80c30fdbc0cf4589f4b83538.png" alt="col_high_0012" data-base62-sha1="nDeE1kbj6hfeet5MKfsLfQ60qb6" width="320" height="240"><br>
Any help would be appreciated. Thanks!</p>

ptrblck: <p>The <code>reshape</code> or <code>view</code> operation will interleave the image in case you want to transform the memory layout from channels-last to channels-first.<br>
Use <code>tensor.permute</code> instead, which permutes the dimensions.</p>

Dwight_Foster: <p>I tried that but it is still giving me messed up images.</p>

ptrblck: <p>Could you upload an example image here so that we could reproduce it?</p>

Dwight_Foster: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e337dd5a4d3b8e44f27989e1c576a6f1061869c.png" alt="col_high_0000" data-base62-sha1="mzvL9akurjP6SU5PHsKzXQxB6fW" width="320" height="240"><br>
Here you go.</p>
Pdist array indexing

eghbalhosseini: <p>I am trying to implement a function similar to Scipy.spactial.distance.pdist in PyTorch for computing correlation. I can get the values for the symmetric matrix of pair comparison, but I want to get a vector form similar to squareform(Matrix)=vec. I noticed that in PyTorch the function pdist does a similar indexing  going from N<em>N to N</em>(N-1)/2 vector. now I am trying to figure it where and how torch.nn.functional.pdist implement this indexing.</p>
Coverting a Pytorch tensor of shape [1,3,19,19] to [1,3,9,9]

Mathedelic: <p>I am using a single image GAN model(ConSINGan) for image generation. The dimension of the input image is [1,3,9,9]. However, the size of the image tensor used for interpolation is [1,3,19,19], which I want to change to [1,3,9,9]. How do I reshape torch tensors so that only the last 2 indices of the tensor are reshaped or downsampled.</p>
<p>I tried changing it into numpy first and then doing it, but is there any other way to do it directly without changing it to numpy?</p>
<p>Thank you in advance. !!!</p>

Nikronic: <p>Hi,</p>
<p>You can use <code>torch.nn.functional.interpolate</code> to the down/up scaling for any tensors:</p>
<aside class="quote" data-post="2" data-topic="52401">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/resize-tensor-without-converting-to-pil-image/52401/2">Resize tensor without converting to PIL image?</a> 
  </div>
  <blockquote>
    Hi, 
You can do it using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.interpolate" rel="noopener nofollow ugc">interpolate</a> function and it supports different methods. 
Here is an example: 

import PIL.Image as Image
from torchvision.transforms import ToTensor, ToPILImage
import torch.nn.functional as F

img = Image.open('data/Places365_val_00000001.jpg')
img = ToTensor()(img)
out = F.interpolate(img, size=128)  #The resize operation on tensor.
ToPILImage()(out).save('test.png', mode='png')

Bests 
Nik
  </blockquote>
</aside>

<p>Bests</p>

Mathedelic: <p>Thank you so much BOSS ! This worked ! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>
Size after using torch.max function

cbd: <p>Below code gives the shape of tensor as " torch.Size([])" after using torch.max function. Why output tensor is not having size "torch.Size([1])? and how can i convert it to that shape?</p>
<p>import torch<br>
a=torch.tensor([1.0, 2.0])<br>
b=torch.max(a)<br>
print(b.shape)</p>

MrPositron: <p>This should help</p>
<pre><code class="lang-auto">b = b.unsqueeze(0)
</code></pre>

cbd: <p>Thanks. It solve my issue.</p>
Cropping a tensor according to a tensor which contains cropped indices

theycallmefm: <p>Let’s say I have a tensor x, size of [B, N, C] and tensor idx, size of [B,N,D]. ( C&gt;D)  . Tensor B has indices that I need for cropping.  For example :</p>
<pre><code class="lang-auto">x shape:  torch.Size([2, 3, 3])
tensor([[[0.0734, 0.9977, 0.8373],
         [0.3425, 0.1120, 0.7120],
         [0.0164, 0.8370, 0.9897]],

        [[0.5219, 0.7202, 0.0872],
         [0.9454, 0.1704, 0.9376],
         [0.8468, 0.5296, 0.0180]]])

idx shape  torch.Size([2, 3, 2])
tensor([[[1, 2],
         [2, 1],
         [1, 2]],

        [[1, 2],
         [1, 0],
         [0, 2]]], dtype=torch.int32)

what I want:

tensor([[[ 0.9977, 0.8373],
         [0.7120,  0.1120],
         [0.8370, 0.9897]],

        [[0.7202, 0.0872],
         [0.1704, 0.9454],
         [0.8468 0.0180]]])
</code></pre>
<p>How can I do this in an efficient way?</p>

theycallmefm: <p>I think I have found a way. I am posting it here in case somebody needs it.</p>
<pre><code class="lang-auto">bidx = torch.arange(B).view(B,1,1)
bidx = bidx.expand_as(idx).contiguous()
nidx = torch.arange(N).view(1,N,1)
nidx = nidx.expand_as(idx).contiguous()
x = x[bidx.view(-1), nidx.view(-1), idx.view(-1)].view(B,N,D)
</code></pre>
Only batches of spatial targets supported (3D tensors) but got targets of dimension: 1

Andrii_Pylypenko: <p>Hi, I got a problem with a <strong>torch.nn.functional.cross_entropy(<em>input</em>, <em>target</em>)</strong><br>
<em>Input</em> is an output from the model.<br>
<em>Input</em>.shape =&gt; torch.Size([25, 5])<br>
<em>target</em>.shape =&gt; torch.Size([25]) - contains 25 targets for 5 classes, numbers starting from 0 to 4.<br>
When I changed the input to random values using torch.randn(25, 5, requires_grad=True) it worked good.</p>
<p>Why this error would occur?</p>

ptrblck: <aside class="quote no-group" data-username="Andrii_Pylypenko" data-post="1" data-topic="111162">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrii_pylypenko/48/34368_2.png" class="avatar"> Andrii_Pylypenko:</div>
<blockquote>
<p>When I changed the input to random values using torch.randn(25, 5, requires_grad=True) it worked good.</p>
</blockquote>
</aside>
<p>Could you add a print statement into the training loop and check the shape of <code>input</code>?<br>
Your shapes and value ranges look correct, which also explains why the code using random tensors works.<br>
Based on the error message, I assume the <code>input</code> tensor might become a 4-dimensional tensor for some reason (e.g. used in a segmentation use case) and thus the error is raised.</p>

Andrii_Pylypenko: <p>Thanks for your answer. I rechecked my model output, and there was a problem with dimensions. I wish you a nice day.</p>
What is the best way to deal with categorical variables?

morphism: <p>Hello, how are you?</p>
<p>I am working on Deep Learning Project whose data has lots of categorical variables.<br>
Some of them are just binary variables (0 or 1)</p>
<p>In Deep Learning, May I ask:</p>
<p>What is the best way to handle those categorical variables?<br>
Can Deep Learning handle those data without any transformation?</p>
<p>And which technique or transformation should I use?</p>
<p>Thank you in advance and have a nice weekend</p>

ptrblck: <p>If your inputs contains categorical variables, you might consider using e.g. an <code>nn.Embedding</code> layer, which would transform the sparse input into a dense output using a trainable matrix.<br>
I’m unsure what the alternatives would be and if passing these values to the model might even work in your case.</p>

morphism: <p>Hello,<br>
Thank you for your answer.<br>
So there is no ‘best’ way to handle categorical data in deep learning?</p>
<p>I will apply the nn.Embedding technique as you recommend.<br>
Have a nice day and see you again.</p>
Training loss is not being reduced

Ajmunnisa_KP: <p>Hi, I am a beginner in the deep learning field. I learned many things via PyTorch forum. thanks to all.<br>
Now I am working with a video captioning framework. I did training with cross-entropy loss and Adam optimization. But the issue is that the loss was 8.7296 at the beginning and it became reduced to 7.87 -7.93 range within an epoch of training. After that, the loss is dangling in this range.<br>
Can anybody suggest me, what type of changes I could have done to reduce the loss.?</p>
<p>My model consists of an encoder and a decoder.  The encoder takes CNN feature vectors of video as input and consists of a fc layer. the decoder consists of lstm network with 64 layers.</p>
Can not push tensor and model to GPU

chengzhi_lin: <p>My environment is:</p>
<ul>
<li>pytorch: 1.7.0+cu110</li>
<li>cuda driver: 455.23.04</li>
<li>cuda version: 11.1.</li>
<li>graphic card: 3090</li>
</ul>
<p>I found that when other people run progorm on card device 0, my program running at any device would stop at model.cuda()/some tensor.cuda(); The nvidia-smi print<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/556c7a959ea463401ffa7458453b9dbb558cdafc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/556c7a959ea463401ffa7458453b9dbb558cdafc" title="B9235A1E-1B91-45B7-8E76-E6B395B26F51"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/556c7a959ea463401ffa7458453b9dbb558cdafc_2_481x500.jpeg" alt="B9235A1E-1B91-45B7-8E76-E6B395B26F51" data-base62-sha1="cbGZZoxagRRM0xvIESA1ovtWbG4" width="481" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/556c7a959ea463401ffa7458453b9dbb558cdafc_2_481x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/556c7a959ea463401ffa7458453b9dbb558cdafc_2_721x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/556c7a959ea463401ffa7458453b9dbb558cdafc_2_962x1000.jpeg 2x" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">B9235A1E-1B91-45B7-8E76-E6B395B26F51</span><span class="informations">1722×1788 496 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It is also weird that the one who run program on devie 0 would occupy 29Mib memory for each other device.</p>

ptrblck: <p>Based on the output of <code>nvidia-smi</code> all GPUs run in exclusive mode (seen via <code>E. Process</code>), so you might need to disable it, if you want to share the GPUs.</p>
nn.CrossEntropyLoss using ignore_index decreases accuracy by 18%

beneyal: <p>Hi everyone,</p>
<p>As a toy project, I coded a model to classify sentences whether they are more likely written by Lewis Carroll or Mary Shelley, so I fed my model the sentences of “Alice in Wonderland” and “Frankenstein” with the correct lablels.</p>
<p>As with every sequence problem, I had to pad shorter sentences with 0 (I used a maximum length of 30). Now comes the weird part: when I used <code>nn.CrossEntropyLoss()</code>, I achieved an accuracy of 89% on the test set. With <code>nn.CrossEntropyLoss(ignore_index=0)</code>, my accuracy on the test set fell to 71%. I can’t think to myself why this would happen.</p>
<p>Here’s my model for reference:</p>
<pre><code class="lang-python">class BookClassifier(nn.Module):
    def __init__(self, vocab_size, embedding_dim=100, hidden_size=100, num_layers=2, dropout_p=0.2):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
        self.bilstm = nn.LSTM(input_size=embedding_dim,
                              hidden_size=hidden_size,
                              num_layers=num_layers,
                              batch_first=True,
                              dropout=dropout_p,
                              bidirectional=True)
        self.fc = nn.Linear(2 * hidden_size, 2)
    
    def forward(self, x):
        embedded = self.embedding(x)
        out, _ = self.bilstm(embedded)
        return self.fc(out[:, -1, :])
</code></pre>
<p>I used <code>optim.Adam(model.parameters())</code> as an optimizer, and the usual training loop.</p>
<p>Any insights are appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Are you using valid word indices with the value 0?<br>
If so, the loss function would not only ignore the padding values but also all words with the 0 index, which might decrease the model performance.</p>

beneyal: <p>No, my <code>words2idx</code> dictionary is <code>{"&lt;PAD&gt;": 0, "&lt;UNK&gt;": 1, ... }</code> and indices 2 and on are words from the training corpus.</p>
Pytorch 0.4.1 installation issues for EDNP implementation

Muhammad4hmed: <p>I am trying to run following implementation: <a href="https://github.com/Lotayou/everybody_dance_now_pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Lotayou/everybody_dance_now_pytorch: A PyTorch Implementation of "Everybody Dance Now" from Berkeley AI lab.</a></p>
<p>As mentioned on repo, their requirements are</p>
<ul>
<li>Ubuntu 18.04 (But 16.04 should be fine too)</li>
<li>Python 3.6</li>
<li>CUDA 9.0.176</li>
<li>PyTorch 0.4.1post2</li>
</ul>
<p>So Installed above using:</p>
<pre><code class="lang-auto">!pip3 install http://download.pytorch.org/whl/cu90/torch-0.4.1-cp36-cp36m-linux_x86_64.whl # for pytorch
!pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ torchvision==0.2.1.post2 # for torchvision
!pip uninstall pillow
!pip install pillow==5.2.0 # this is requirement of torchvision's version 0.2.1.post2
</code></pre>
<p>and confirmed installation using</p>
<pre><code class="lang-auto">import torch
import torchvision 

print(torch.__version__)
print(torchvision.__version__)
print(torch.version.cuda)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">0.4.1
0.2.1
9.0.176
</code></pre>
<p>and everything seems okay, now I run the training file</p>
<pre><code class="lang-auto">!python /content/dance_now/train.py 
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663 pytorch
</code></pre>
<p>Have tried many things like upgrading/degrading pytorch, installing cudNN etc but nothing worked.</p>
<p>Kindly help, thanks.</p>

ptrblck: <p>The error is raised in the old <code>THCudaMalloc</code> call <a href="https://github.com/pytorch/pytorch/blob/a24163a95edb193ff7b06e98cd69bf7cfd4c0d2f/aten/src/THC/THCGeneral.cpp#L663">here</a>, so I guess that your current (CUDA) setup is not working correctly.<br>
It’s a bit hard to tell what the reason might be, as <code>0.4.1</code> was released in July 2018.</p>
<p>The probably better approach would be to try to update the repository and make it compatible with the latest stable release. Alternatively you could also try to use an old docker container from 2018 with CUDA9.0 and try to get this old version running there.</p>