akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pixelwise blur along time todmai

drydenwiebe: <p>Right now I have an image dataset of shape (N, C, D, H, W) where N is the batch, C is the channels, D is the number of frames and H and W are height and width respectively.</p>
<p>We can think of it as video data where D is the number of frames of (C, H, W) images.</p>
<p>What I want to do is to gaussian blur along the time (D) axis for each pixel in (C, H, W). I know how to use torchvision.transforms.GaussianBlur, but this is for a blur where the input is […, C, H, W] and is a 2d blur. What I want is a 1D pixelwise blur along a specific axis.</p>
<p>Any ideas?</p>
<p>Thanks!</p>

ptrblck: <p>You could create a gaussian blur kernel in the appropriate shape and use the functional API via <code>F.conv3d</code> to apply it to the input, as I don’t know, if transformations support gaussian blurring for 3d data for a specific dimension only.</p>
Stack frames for DNN input

lima: <p>I have a dataset of N audio frames of size (N , 13), and for the task of phoneme recognition</p>
<pre><code class="lang-auto">train_data = torch.hstack((train_feat, train_labels))
train_loader = torch.utils.data.DataLoader(train_data, batch_size= 128, shuffle=True)
</code></pre>
<pre><code class="lang-auto">print(train_data.shape)
torch.Size([3082092, 14])
</code></pre>
<p>How can I stack 7 frames each time to feed to the DNN?</p>

ptrblck: <p>I’m not sure which dimension refers to the “frame dimension”, but you could probably either use <code>torch.stack</code> or <code>torch.cat</code> to create the new stacked/concatenated tensor.</p>

lima: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> . The x_train dataset is 3082092 frames. Each frame has 13 numbers (features).<br>
The y_train is 3082092 digit (labels).<br>
That is for each frame (1,13) there’s one label…<br>
Now, feeding pne frame to the DNN is not going to work because there’s too small information in it. Instead,I would like to stack a sequence of 7 frames (of those 3082092). I hope that makes sense.</p>

ptrblck: <p>Thanks for the explanation. The 7 frames would thus correspond to the batch size and you could set it in the <code>DataLoader</code>.</p>

lima: <p>In that case, the DNN would look at them as 7 individual units, but I wanted to stack 7 frames as one unit.</p>

ptrblck: <p>Could you explain how one “unit” would be processed in the model and what the expected input shape would thus be?</p>
Partially freezing and unfreezing the network when required

Lucky_Magna: <p>I’m currently working on a CNN project using resnet18. I replaced the fully connected layers as per my requirement, but before doing this I set the <code>requires_grad</code> parameter to <code>False</code> for all the layers in the <code>resnet18</code> model. Now what I want to do is train my model for certain epochs, overfit it to a task, and then unfreeze the gradients only for the convolution part. How do I do it?<br>
Thank You</p>
RuntimeError: The size of tensor a (3) must match the size of tensor b (516) at non-singleton dimension 0

ellenW: <p>Hi I am trying to reproduce results of the paper using its official code and pre-trained models.</p>
<p>But i could not  make it because  loss_func_mse causes</p>
<p>RuntimeError: The size of tensor a (3) must match the size of tensor b (516) at non-singleton dimension 0.</p>
<p>outputs, feas, updated_feas, m_items_test, softmax_score_query, softmax_score_memory, _, _, _,<br>
compactness_loss = model.forward(imgs[:,0:3<em>4], m_items_test, False)<br>
mse_imgs = torch.mean(loss_func_mse((outputs[0]+1)/2, (imgs[0,3</em>4:]+1)/2)).item()</p>
<p>outputs dimension : [1 3 256 256]<br>
imgs : [1 528 256 256]</p>
<p>How can I fix this?</p>

ptrblck: <p>You would have to check why these tensors have different shapes, which could be caused by an error in the model architecture, a typo using the wrong variable name, etc.<br>
Based on the shapes it seems as if <code>output</code> might have the same dimensions as an image tensor <code>input</code>.</p>
When Shuffling True - Score decreased Why?

TheOraware: <p>Hi ,</p>
<p>I posted a query here on github using code so it can be neat to view , apologies if it is going to be duplicate</p>
<p>After training model when i test model in batch using shuffle=False give me good score , when i use the same model and same test records using shuffle=True give me bad score , i am confused why it is so?</p>
<p><a href="https://github.com/pytorch/pytorch/issues/58986" rel="noopener nofollow ugc">Shuffling True on test data decreasing score</a></p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/model-eval-giving-different-result-when-shuffle-is-true-and-false/122497/2">here</a>, which is not reproducible at the moment.</p>
Can someone help solving it i am newbie dont know how to handle weight of size [128, 300, 3, 3], expected input[2, 128, 28, 28] to have 300 channels, but got 128 channels instead

sheraz: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c44a85e39d0e8d20e2cd3b4e556a499664b5a356.png" data-download-href="https://discuss.pytorch.org/uploads/default/c44a85e39d0e8d20e2cd3b4e556a499664b5a356" title="sample 1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c44a85e39d0e8d20e2cd3b4e556a499664b5a356.png" alt="sample 1" data-base62-sha1="s0tfrbGZGi52YCs00Sbeua7J5gq" width="690" height="291" data-dominant-color="F6F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sample 1</span><span class="informations">973×411 24.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The error is raised in an <code>nn.Conv2d</code> layer, which uses <code>in_channels=300</code>, while the incoming activation has only 128 channels, so you would have to change the <code>in_channels</code> argument of this layer.</p>
Batchnorm forward and backward - updates of statistics

Ilya_Kotlov: <p>Hi.<br>
I tried to look at the C code and in the code in the docs, but still didn’t figure it out,<br>
generally BatchNorm aggregates the mean and the variance of the samples it sees in train mode and then in test mode just uses them, my question is does the aggregation, i.e the updates of the mean and the variance according (in the current forward iteration) to what just was seen are done in the forward path or as it more conventional in the backward path, i.e only after backward and step functions are activated the internal statistics include the current iteration statistics ?</p>
<p>Thank you in advance for clarification.</p>

ptrblck: <p>The running stats are updated during the forward pass as seen here:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
x = torch.randn(16, 3, 24, 24)
print(bn.running_mean)
&gt; tensor([0., 0., 0.])

print(bn.running_var)
&gt; tensor([1., 1., 1.])

out = bn(x)
print(bn.running_mean)
&gt; tensor([ 0.0002, -0.0007,  0.0006])

print(bn.running_var)
&gt; tensor([0.9991, 1.0022, 0.9988])
</code></pre>
Activation are slightly alterd even in .eval() depending on the batch

JanoschMenke: <p>Hey Guys,</p>
<p>I am using multiple <code>nn.BatchNorm1d()</code> layers. I noticed that the activation of the same inputs change slightly when I include the input in different batches. Even when I am using <code>model.eval()</code>.</p>
<p>So I have input <code>A</code> and I feed it to the model in the same batch as the the inputs <code>B</code> and <code>C</code>. The activation of <code>A</code> will differ to the Activation of <code>A</code> when I fed it through the network together with <code>B</code> <code>C</code>, <code>D</code>, <code>E</code>.</p>
<p>Is this due to the <code>BatchNorm</code> I feel like <code>model.eval()</code>, should prevent this behavior. But maybe I am messed up somewhere.</p>
<p>Would there be a option to prevent this behavior</p>

ptrblck: <p>How large are the relative and absolute errors?<br>
Note that you might run into the limited floating point precision e.g. due to different batch sizes, different algorithms etc. You could try to use <code>float64</code> and see if the error would be lower (and if you would really need it).</p>
Why track_running_stats is not set to False during eval

Naman-ntc: <p>Why is track_running_stats set to True in eval. This may lead to performance degradation of pretrained model as well and in my opinion should be default behaviour.<br>
And what is recommended method to set track_running_stats False, currently I am just using <code>model.apply(fn)</code></p>
<p>Infact if such a model with track_running_stats is deployed isn’t it a very vulnerable model. Suppose it is provided to user, he may give gibberish data to screw up running stats.<br>
Something like this happened to me, although instead of gibberish I had a batch size of 1 and running_mean and running_var changed pretty significantly reducing performance of my model somewhat drastically!!!</p>

ptrblck: <p><code>track_running_stats</code> is used to initialize the running estimates as well as to check if they should be updated in training (<a href="https://github.com/pytorch/pytorch/blob/47956ddf7e9632bcdb47df2e12b88a933454534d/torch/nn/modules/batchnorm.py#L57">line of code</a>).<br>
The running estimates won’t be updated in <code>eval</code>:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
for _ in range(10):
    x = torch.randn(10, 3, 24, 24)
    out = bn(x)
print(bn.running_mean)
print(bn.running_var)
&gt; tensor(1.00000e-03 *
       [-0.7753,  0.7027, -1.4181])
  tensor([ 1.0015,  1.0021,  0.9947])

bn.eval()
for _ in range(10):
    x = torch.randn(10, 3, 24, 24)
    out = bn(x)
print(bn.running_mean)
print(bn.running_var)
&gt; tensor(1.00000e-03 *
       [-0.7753,  0.7027, -1.4181])
  tensor([ 1.0015,  1.0021,  0.9947])
</code></pre>

Naman-ntc: <p>Ohhhh sorry my bad, I rechecked my script, I forgot putting model.eval() at one place and that led to this screw up.</p>
<p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"> <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=6" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

jizongFox: <p>Hi, There may be some bugs on <code>track_running_stats=False</code> and bn continous to track the <code>running_mean</code></p>

ptrblck: <p>What kind of bug did you observe? Did the <code>running_mean</code> update even after calling <code>eval()</code> on the module?</p>

TheOraware: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> yes it is happening even when i call model.eval() on validating model using same test data , model performance degrade when i use shuffle=True please see detail with code here</p>
<p><a href="https://discuss.pytorch.org/t/model-eval-giving-different-result-when-shuffle-is-true-and-false/122497">Model.eval() giving different result when shuffle is True and False</a></p>

ptrblck: <p>I’ve answered in the linked thread and cannot reproduce the issue.</p>
Backward() crashes if the graph is empty

shapovalov: <p>I’ve spent some time chasing a bug in my code. Apparently, if the computation graph has no tensors requiring grads, autograd raises an error. More specifically:</p>
<pre><code class="lang-auto">x = torch.ones((2, 2), requires_grad=False)
y = torch.ones((2, 2), requires_grad=True)
torch.sum(x * y).backward()  # ok
torch.sum(x * x).backward()  # crashes
</code></pre>
<p>The last line throws a cryptic <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>, and <code>detect_anomaly</code> is not helpful in localising the problem.</p>
<p>In my case, the graph is dynamic, and on rare occasions the latter case occurs, resulting in the error. Would it be better to stop gradients without crashing, i.e. do nothing in <code>backward()</code> if the graph is empty?</p>

albanD: <p>You can simply avoid such error by only calling backward() if your output requires_grad. That will have the behavior you expect.:</p>
<pre><code class="lang-auto">loss =  torch.sum(x * x)
if x.requires_grad:
   x.backward()
</code></pre>
<p>In  general, we choose to raise an error in this case because in most user cases, users expect backward to actually happen when they call.backward(). So having a Tensor that doesn’t require gradients is a bug in the user code in most cases.</p>

shapovalov: <p>Thank you for the explanation. If it is a common source of bugs, then raising an error makes sense.<br>
Is it possible to make the message more helpful? It can say that backward() / autograd.grad() is called on a tensor that does not require_grad or does not have grad_fn; make sure to call it from under enable_grad / not under no_grad.<br>
I.e. the part “element 0 of tensors” is misleading, and some guidance on why the tensors may miss grad_fn can be helpful too. Not sure that my example addresses all failure cases though.</p>

albanD: <p>I think this error message is more than 3 years old <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">  And we definitely could improve it.<br>
Could you open an issue on github asking to update that and your ideas for what a better one would be?<br>
That will be easier to track for us.</p>
Error in setting custom lr for different parameters

mina: <p>Hello<br>
I am new to pytorch and I wanna to set different learning rate for some layer’s weights and biases.<br>
here is my code:</p>
<pre><code class="lang-auto">        with open('DCN.txt', 'r') as Defor_layer:
            my_list = Defor_layer.readlines()

        params = list(filter(lambda kv: kv[0] in my_list, model.named_parameters()))
        base_params = list(filter(lambda kv: kv[0] not in my_list, model.named_parameters()))
      
        if args.optm == "SGD":
            optimizer = SGD([
                  {'params': base_params},
                  {'params': params, 'lr':1}]
                  , lr=0.05, momentum=0.9, weight_decay=0.0002)
        else:
            optimizer = Adam([{'params': base_params},
                              {'params': params, 'lr':1}]
                            , lr=0.05, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0002)
</code></pre>
<p>and my params list looks like below, but longer</p>
<p>module.encoder.layers.7.DCNv2.conv_offset_mask.weight<br>
module.encoder.layers.7.DCNv2.conv_offset_mask.bias<br>
module.encoder.layers.8.DCNv2.conv_offset_mask.weight<br>
module.encoder.layers.8.DCNv2.conv_offset_mask.bias</p>
<p>However, I am getting this error and I am not sure what is about:<br>
File “main_FE_Deco_v2_lr.py”, line 191, in train<br>
, lr=0.05, momentum=0.9, weight_decay=0.0002)<br>
File “/home/kh/.conda/envs/torch41/lib/python3.6/site-packages/torch/optim/sgd.py”, line 64, in <strong>init</strong><br>
super(SGD, self).<strong>init</strong>(params, defaults)<br>
File “/home/kh/.conda/envs/torch41/lib/python3.6/site-packages/torch/optim/optimizer.py”, line 43, in <strong>init</strong><br>
self.add_param_group(param_group)<br>
File “/home/kh/.conda/envs/torch41/lib/python3.6/site-packages/torch/optim/optimizer.py”, line 191, in add_param_group<br>
"<strong>but one of the params is " + torch.typename(param))</strong><br>
<strong>TypeError: optimizer can only optimize Tensors, but one of the params is tuple</strong></p>
<p>I appriciate any guide.</p>

mina: <p>I realized what is the problem.<br>
I have name and data in the params and that’s why the I got the error say one of the params is tuple. all I had to is to send only the data as params and not the names along them.</p>

luciaL: <p>Could you please show how to send only the data as params? I have the same error as yours.</p>

Sandaru: <p>Hi <a class="mention" href="/u/lucial">@luciaL</a> before initializing the optimizer, adding the following two lines to obtain the params from the tuples in the two lists resolves the issue.<br>
params = [i[1]for i in params]<br>
base_params = [i[1]for i in base_params]</p>
Understanding how filters are created in torch.nn.Conv2d

chethanjjj: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/785c7afeba58fb5adf3f84ec959bc2de0e60278b.png" data-download-href="https://discuss.pytorch.org/uploads/default/785c7afeba58fb5adf3f84ec959bc2de0e60278b" title="Screen Shot 2021-05-26 at 1.37.16 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/785c7afeba58fb5adf3f84ec959bc2de0e60278b_2_506x500.png" alt="Screen Shot 2021-05-26 at 1.37.16 PM" data-base62-sha1="haLqXLvMe4hf03T4jFKMB7EEqev" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/785c7afeba58fb5adf3f84ec959bc2de0e60278b_2_506x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/8/785c7afeba58fb5adf3f84ec959bc2de0e60278b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/8/785c7afeba58fb5adf3f84ec959bc2de0e60278b.png 2x" data-dominant-color="E5E4DE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-05-26 at 1.37.16 PM</span><span class="informations">585×577 35.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hey everyone, I have a question about connecting the convolution in this picture to the <code>torch.nn.Conv2d</code> function. So to recapitulate the convolution step, I’d need to set the <code>out_channels = 2</code> and <code>kernel_sizes = [2, 3, 4]</code>. My question is how are the values in the filter matrices calculated? I assume the two filters with size 2 aren’t the same. Any help is appreciated!</p>
<p>Just to add another point to this to make it more clear, when reading about CNN for images, I come across these pre-defined filters (see below). How come there is no option for that in <code>torch.nn.Conv2d</code>?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a59c0bc87849dbc574cda62eee57a5e1baffd8f1.jpeg" alt="05" data-base62-sha1="nD36NFKeMZ9WATauuwAKyi6fkPv" width="470" height="347"></p>

eqy: <p>You can manually set the parameters in a layer (e.g.,</p>
<pre><code class="lang-auto">mylayer = torch.nn.Conv2d(1,1,1,1)
mylayer.weight = ...
</code></pre>
<p>However, this is typically not done because the layers are trained with gradient descent. Predefined filters are more of a hallmark of classical computer vision for tasks like line and corner detection but typically convolutional layers in neural networks are trained from randomly initialized parameters.</p>

chethanjjj: <p>I gotcha, that totally makes sense that the values in the filter matrices are the weights that are updated through backprop.</p>

vdw: <p>If it helps, <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/cnn.py" rel="noopener nofollow ugc">I’ve implemented the mode</a>l. Let me know if anything’s unclear…or you spot any errors :).</p>
GCN node classification errors

Ray1: <p>I am trying to do section 3 of <a href="https://colab.research.google.com/drive/1Tc6a8Gb4nx8sHI8HgmveCcAt-7smbcY-#scrollTo=qd5O5cnPPdVF" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a></p>
<p>However, there are two errors.When I try to train my model, I get an error for the loss function: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15.</p>
<p>The second error occurs when I change the hidden_dim back to 256, I get an error regarding the batch normalization.</p>
<p>How do I fix these problems?</p>

ptrblck: <p>This error is raised, if the shapes of the input tensors are unexpected, so make sure the model output has the shape <code>[batch_size, nb_classes, *]</code>, while the target has the shape <code>[batch_size, *]</code> containing the class indicex in the range <code>[0, nb_classes-1]</code>.<br>
Here is a small code snippet showing the error:</p>
<pre><code class="lang-python">batch_size, nb_classes = 2, 10
x = torch.randn(batch_size, nb_classes)
y = torch.randint(0, nb_classes, (batch_size,))
criterion = nn.CrossEntropyLoss()
loss = criterion(x, y)

y = torch.randint(0, nb_classes, (batch_size, nb_classes))
loss = criterion(x, y)
&gt; RuntimeError: 1D target tensor expected, multi-target not supported
</code></pre>

Ray1: <p>Still not sure how to fix this. My pred.shape is [169343, 40], my data.y shape is [169343, 1], where each entry in data.y is in [0, 39].</p>

ptrblck: <p>Remove the unnecessary <code>dim1</code> from the target via <code>.squeeze(1)</code> and it should work.</p>
Detect tiny object from the larger image

vishalthengane: <p>I have images of the size ~7000x~5000 and the object size is about 64x64 pixel. I want to apply the object detection model to these images. What will be the best approach to do it? As the object detection model does not support this size also I think it will not be a good idea to resize the image in a smaller size. Can I use ~7000x~5000 as an input size to the object detection model or need to divide the<br>
image into patches?</p>
<p>here is the example of image: <a href="https://drive.google.com/file/d/1XVco-j9bEO5MoSh-T1p3wYPiRKm7R1o0/view?usp=sharing" rel="noopener nofollow ugc">https://drive.google.com/file/d/1XVco-j9bEO5MoSh-T1p3wYPiRKm7R1o0/view?usp=sharing</a></p>

Bernd: <p>AFAIK if you use Mask R-CNN or Faster R-CNN to detect the small objects, the big images will internally automatically be resized. I saw some min_size(=800) and max_size(=1333) parameters down in the base classes that will make the decision when to scale the image. Unfortunately I did not find a way to change these min_size and max_size parameters without duplicating half of the PyTorch code.<br>
Therefore I think the best way would be to split your image into slightly overlapping patches.<br>
You might be able to speed this up by skipping empty patches with very little pixel contrast (std() of pixels very low).<br>
Since I am very new here I think some pro’s should confirm my advise…</p>

Bernd: <p>Looking at your image again, I think you should not use any detection model.<br>
Your patterns are so tiny, they are more “spots” than patterns.</p>
<p>Instead I propose to create a small set of spot detection kernels and run them via convolution over the image.<br>
A typical spot detection kernel would be a matrix like this:<br>
0,  0, -1, 0, 0<br>
0, -1, 4, -1, 0<br>
-1, 4, 9, 4, -1<br>
0, -1, 4, -1, 0<br>
0, 0, -1, 0, 0<br>
This example kernel should spike when it finds a spot with 2-3 pixels in diameter.<br>
You would need kernels for different spot sizes…</p>
<p>A laplace filter from OpenCV could do the same trick…</p>
Is there a complex output for IRFFT?

eduardo4jesus: <p>The outputs of <code>torch.fft.irfft</code> and <code>torch.fft.irfftn</code> are real. Would it be possible, by any chance, to the the complex version of it, i.e., before the result is clipped to the real domain?</p>
<p>I already went through these code on github and I could not find a way to, yet.</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/SpectralOps.cpp" rel="noopener nofollow ugc">ATen/native/SpectralOps</a></li>
<li><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/SpectralOps.cpp" rel="noopener nofollow ugc">ATen/native/cuda/SpectralOps</a></li>
</ul>
Understanding `output_size` in ConvTranspose2d

mnlkfmn: <p>I’m using <code>ConvTranspose2d</code> in an autoencoder architecture to upsample. Thus, I’m trying to understand the following code snippet (adapted from the docs):</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

input = torch.randn(1, 16, 13, 12)
conv = nn.Conv2d(16, 16, 3, stride=1, padding=1)
downsample = nn.MaxPool2d(2)
upsample = nn.ConvTranspose2d(16, 16, 3, stride=2, padding=1)
h = conv(input)
print('after conv', h.size())
h = downsample(h)
print('after pooling', h.size())

output = upsample(h, output_size=input.size())
</code></pre>
<p>Here I want to size of the output to be the same as the input. This snippet is throwing an error however: “requested an output size of torch.Size([13, 12]), but valid sizes range from [11, 11] to [12, 12] (for an input of torch.Size([6, 6]))”.</p>
<p>It works fine if the input size is even, e.g. <code>input = torch.randn(1, 16, 12, 12)</code>. Clearly this happens because the downsampling floors the output size from 13/2 = 6.5 to 6, which makes sense. What confuses me is that by setting the <code>output_size</code> argument, ConvTranspose2D cannot handle this seemingly simple case. Am I using it wrong or is it indeed expected that client code works around this? I’m currently fixing it like this:</p>
<pre><code class="lang-auto"># ...
output = upsample(h)
print('before padding')
output = F.pad(output, (0, 1, 1, 1))
print('after padding', output.size())
</code></pre>
<p>I could also fix it by adjusting the padding in the first place, i.e.:</p>
<pre><code class="lang-auto">upsample = nn.ConvTranspose2d(16, 16, 3, stride=2, padding=(0, 1))
# ...
output = upsample(h, output_size=input.size())
print('before padding', output.size())
</code></pre>
<p>This also works, but I want the padding to be dynamic, i.e. I don’t know at creation time of <code>upsample</code> whether the input size is going to be odd or even. Furthermore, I am not sure if the two variants are semantically identical. The first one adds padding after the convolution (i.e. my output contains a bunch of zeros at the “edge” rows/columns), whereas the second one pads before the convolution (but I’m not sure if setting the <code>output_size</code> parameter also triggers padding after the convolution).</p>
<p>I.e. my question boils down to: How can I ensure I’m getting exactly the size I want - dynamically at runtime - after an upsamling transpose convolution?</p>
Feasibility of video activity recognition model on Android

kwj2104: <p>I have a simple CNN-LSTM classifier model that takes in video input (&lt;60seconds, 10fps, greyscale, currently around 200x160 resolution). Currently on a desktop, I load the entire video file as a Tensor into memory, and feed the sample into the model. I’m wondering how feasible this would be to run (inference) with PyTorch Mobile in an Android application, given memory, compute, and software limitations (and if not, any guidance on how I can change my approach to make the memory management work)?</p>
<p>Additionally, it looks like relatively trivial video preprocessing in Python doesn’t seem straightforward in<br>
Android. I’d need to:</p>
<ol>
<li>Take the .mp4 file saved by user camera recording, and make it greyscale and lower the resolution.</li>
<li>Convert the result into a Tensor</li>
</ol>
<p>Just looking through the PyTorch documentation, I haven’t found a great way to do step 1. Potentially there are other libraries for video compression, and possibly using OpenCV in Android. For step 2 I’ve seen plenty of examples using Images (e.g., existing functions to convert Bitmaps to float), but I haven’t seen great examples of this for video.</p>
<p>Would love any guidance on where to look to build out this process, or likewise, confirmation that this isn’t currently feasible (or highly difficult).</p>
<p>Thanks</p>

gchauhan: <p><a class="mention" href="/u/kwj2104">@kwj2104</a>  Please check the Video Classification example we added recently based on PytorchVideo library:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/mobile/android/#video-classification" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/mobile/android/#video-classification" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

<p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


gchauhan: <p>Github link for Android example: <a href="https://github.com/pytorch/android-demo-app/tree/master/TorchVideo" class="inline-onebox" rel="noopener nofollow ugc">android-demo-app/TorchVideo at master · pytorch/android-demo-app · GitHub</a></p>
<p>Corresponding iOS version: <a href="https://github.com/pytorch/ios-demo-app/tree/master/TorchVideo" class="inline-onebox" rel="noopener nofollow ugc">ios-demo-app/TorchVideo at master · pytorch/ios-demo-app · GitHub</a></p>
Data Loading in Pytorch for a dataset having all the classes in same folder

aleemsidra: <p>I am new to deep learning and Pytorch. I have data set of 6000 images that have all four classes in a single folder. I used the following snippet to upload my data.</p>
<pre><code class="lang-auto">torchvision.datasets.ImageFolder(root='/content/drive/My Drive/DFU/base_dir/train_dir', transform=None)
</code></pre>
<p>I read that for ImageFolder, the images should be organized into sub-folders based on class labels. However, my dataset has all four class images in a single folder. I have a .csv file (<a href="https://i.stack.imgur.com/qnjTF.png" rel="noopener nofollow ugc">Ground_Truth</a>)that contains the one-hot-encoded class label for each image. How to load my dataset to Pytorch?</p>

carloalbertobarbano: <p>You should create a custom dataset class, which loads the corresponding csv file and return the tuple &lt;image, label&gt; in the <code>__getitem__</code> function: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 1.8.1+cu102 documentation</a></p>

aleemsidra: <p><a class="mention" href="/u/carloalbertobarbano">@carloalbertobarbano</a> the sample example has all the images of the same types in a single directory. I have multiple classes and all the images are in one directory. I have a .csv file that<a href="https://i.stack.imgur.com/qnjTF.png" rel="noopener nofollow ugc">Ground_Truth_File</a> has an image name column and then their respective one hot encoder.</p>

carloalbertobarbano: <p>The sample in the docs shows you how to create your custom dataset by extending all of the required functions. Of course you need to adjust the logic to suit your needs (read the CSV files and return the required attribute in the <code>__getitem__</code>) as I suggested in my previous reply</p>

aleemsidra: <pre><code class="lang-auto">class DFUDataset(Dataset):
  def __init__(self, csv_file, root_dir, transform=None):
        self.DFU = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
  def __len__(self):
        return len(self.DFU)
  def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
            img_name = os.path.join(self.root_dir, self.DFU.iloc[idx, 0]) #image names
            image = io.imread(img_name)
            img_label =  os.path.join(self.root_dir, self.DFU.iloc[0, idx]) #image labels
            sample = {'image': image, 'img_label': img_label}
            return sample
</code></pre>
<pre><code class="lang-auto">DFU_dataset = DFUDataset(csv_file='C:/Users/aleems2/Desktop/dfu/DFUC2021_trainset_210427/DFUC2021_train/Labelled_data_ground_truth.csv',
                                    root_dir="C:/Users/aleems2/Desktop/dfu/DFUC2021_trainset_210427/DFUC2021_train/Labelled_test_images")```


I am debugging but my code does not go to len, get_item function</code></pre>
Training Loss is growing

Arsham_Mostaani: <p>Hello People,</p>
<p>I am trying to train an off-line reinforcement learning algorithm. The algorithm that I am using follows this paper:</p><aside class="onebox allowlistedgeneric" data-onebox-src="http://proceedings.mlr.press/v97/fujimoto19a.html">
  <header class="source">
      <img src="https://proceedings.mlr.press/v97/assets/images/favicon-pmlr.ico" class="site-icon" width="48" height="48">

      <a href="http://proceedings.mlr.press/v97/fujimoto19a.html" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 24 May 2019">PMLR – 24 May 19</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://proceedings.mlr.press/v97/assets/images/logo-pmlr.png" class="thumbnail">

<h3><a href="http://proceedings.mlr.press/v97/fujimoto19a.html" target="_blank" rel="noopener nofollow ugc">Off-Policy Deep Reinforcement Learning without Exploration</a></h3>

  <p>Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection....</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>In this algorithm, 4 different neural networks are trained simultaneously, one of them being a Q-function that is approximated by a deep neural network. This is the architecture of my DQN:</p>
<p>class Critic(nn.Module):<br>
def <strong>init</strong>(self, state_dim, action_dim):<br>
super(Critic, self).<strong>init</strong>()<br>
self.l1 = nn.Linear(state_dim + action_dim, 400)<br>
self.l2 = nn.Linear(400, 300)<br>
self.l3 = nn.Linear(300, 1)</p>
<pre><code>    self.l4 = nn.Linear(state_dim + action_dim, 400)
    self.l5 = nn.Linear(400, 300)
    self.l6 = nn.Linear(300, 1)

def forward(self, state, action):
    q1 = F.relu(self.l1(torch.cat([state, action], 1)))
    q1 = F.relu(self.l2(q1))
    q1 = self.l3(q1)

    return q1
</code></pre>
<p>I use adam optimizer to optimize the model parameters:</p>
<pre><code>self.critic_optimizer = torch.optim.Adam(self.critic.parameters(), lr=1e-3)
</code></pre>
<p>I compute the loss by:</p>
<pre><code>current_Q1, current_Q2 = self.critic(state, action)
critic_loss = F.mse_loss(current_Q1, target_Q)
</code></pre>
<p>And I do one step of optimization by:</p>
<pre><code>   self.critic_optimizer.zero_grad()
   critic_loss.backward()
   self.critic_optimizer.step()
</code></pre>
<p>But the critic loss is exploding:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4bcbde3642e40d6505bfdec30bbd1b3c79373f1.png" alt="Figure_loss_crit" data-base62-sha1="ulXBgwd35DJLmmLtVeX1dWoFuw1" width="640" height="480"></p>
<p>Thanks for your help and support in advance!</p>
Joint training of two embedding models (KGE + GloVe)

nickil21: <p>How do I create a joint model that shares the parameters of a Knowledge Graph Embedding (KGE) model, TuckER (given below), and GloVe (assume a co-occurrence matrix along with the dimensions is already available) in PyTorch? In other words, the joint model must obey the criterion of the CMTF Framework and the weights from the two embeddings must be tied during training. The problem here is that the KGE expects a triple (subject, relation, object) whereas the GloVe expects a co-occurrence matrix. Additionally, their loss functions are also computed differently.</p>
<pre><code class="lang-auto">class TuckER(torch.nn.Module):
    def __init__(self, d, d1, d2, **kwargs):
        super(TuckER, self).__init__()

        self.E = torch.nn.Embedding(len(d.entities), d1)
        self.R = torch.nn.Embedding(len(d.relations), d2)
        self.W = torch.nn.Parameter(torch.tensor(np.random.uniform(-1, 1, (d2, d1, d1)), 
                                    dtype=torch.float, device="cuda", requires_grad=True))

        self.input_dropout = torch.nn.Dropout(kwargs["input_dropout"])
        self.hidden_dropout1 = torch.nn.Dropout(kwargs["hidden_dropout1"])
        self.hidden_dropout2 = torch.nn.Dropout(kwargs["hidden_dropout2"])
        self.loss = torch.nn.BCELoss()

        self.bn0 = torch.nn.BatchNorm1d(d1)
        self.bn1 = torch.nn.BatchNorm1d(d1)
        
    def init(self):
        xavier_normal_(self.E.weight.data)
        xavier_normal_(self.R.weight.data)

    def forward(self, e1_idx, r_idx):
        e1 = self.E(e1_idx)
        x = self.bn0(e1)
        x = self.input_dropout(x)
        x = x.view(-1, 1, e1.size(1))

        r = self.R(r_idx)
        W_mat = torch.mm(r, self.W.view(r.size(1), -1))
        W_mat = W_mat.view(-1, e1.size(1), e1.size(1))
        W_mat = self.hidden_dropout1(W_mat)

        x = torch.bmm(x, W_mat) 
        x = x.view(-1, e1.size(1))      
        x = self.bn1(x)
        x = self.hidden_dropout2(x)
        x = torch.mm(x, self.E.weight.transpose(1,0))
        pred = torch.sigmoid(x)
        return pred
</code></pre>
<p>I know how to jointly train two pre-trained models by loading the state dicts,  taking an instance, running them on the two models, and then applying a feedforward layer on top. But I seem to be not able to figure this scenario out. Can you please suggest how I can achieve this?</p>
How to reset parameters of layer

vanduong0504: <p>Hello everyone,<br>
How to reset the parameters of layer4 in resnet18, using the Module.apply?  Here my code but it doesn’t work.<br>
Thank you.</p>
<pre><code class="lang-auto">def reset_weights(m):
    for name, layer in m.named_children():
        if name=='layer4' and hasattr(layer, 'reset_parameters'):
                print(f'Reset trainable parameters of layer = {layer}')
                layer.reset_parameters()

model.appy(reset_weight)
</code></pre>

ptrblck: <p><code>layer4</code> is an <code>nn.Sequential</code> module, which doesn’t have the <code>reset_parameters</code> method, so your conditions won’t be met.<br>
You could iterate all modules of <code>layer4</code> and check for this attribute:</p>
<pre><code class="lang-python">def reset_weights(m):
    for name, layer in m.named_children():
        if name=='layer4':
            print(name)
            for n, l in layer.named_modules():
                print(n)
                if hasattr(l, 'reset_parameters'):
                    print(f'Reset trainable parameters of layer = {l}')
                    l.reset_parameters()


model = models.resnet18()
model.apply(reset_weights)
</code></pre>

vanduong0504: <p>It works. I owe you one.</p>

Dhawa_Sang_Dong: <p>How can I reset parameters of layers 3 and higher layers of Resnet18 so that I can train for those layers?</p>
Simple, efficient way to create Dataset?

redtailedhawk: <p>After countless searches, and putting pieces of the puzzle together, I came up with this code for a “boilerplate” custom image Dataset class. I wanted to ask if this is satisfactorily simple and efficient, or does anyone see where I might possibly run into trouble?</p>
<p>The use case is to quickly, simply and efficiently just bring in whatever images I have stored in a designated folder and have them ready for input, without a lot of fluff.</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, datasets
from pathlib2 import Path

class MyDataClass(Dataset):
    def __init__(self, image_path, transform=None):
        super(MyDataClass, self).__init__()
        self.data = datasets.ImageFolder(image_path,  transform)    # Create data from folder

    def __getitem__(self, idx):
        x, y = self.data[idx]
        return x, y

    def __len__(self):
        return len(self.data)

root = Path.absolute()        # Map to dataset folder location
img_path = root/'dataset_folder'

tsfm = transforms.Compose([
        # Misc transforms here.
        transforms.ToTensor()
])

data = MyDataClass(img_path, tsfm) # create the dataset

dataloader = DataLoader(data, batch_size=20)
</code></pre>
<p>Aside from extensive customization on my labels, are there any hangups you can potentially see here that I should be aware of, or would I be able to just use this as my boilerplate to prepare any generic png-image dataset to be loaded into a conv2d layer?</p>

redtailedhawk: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I set this up and it works fine, I just wanted to know if I’m missing anything or doing anything wrong since a simple solution like this doesn’t seem to be anywhere in tutorials or the documentation</p>

ptrblck: <p>Your custom <code>Dataset</code> should work fine. However, currently you could also just use <code>ImageFolder</code> directly without wrapping it in <code>Dataset</code> and pass it to a <code>DataLoader</code>.<br>
If you want to customize the samples inside <code>__getitem__</code>, your approach looks good! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

redtailedhawk: <p>Awesome, thanks. So for reference, one can simply write:</p>
<pre><code class="lang-auto">data = datasets.ImageFolder(img_path, tsfm) 
</code></pre>
<p>And pass that straight into a DataLoader. That’s got to be by far the simplest, no fuss way to create a dataset in Pytorch. For anyone finding this thread, you must convert your images to tensors with the following transform if you want to utilize it in most of the nn.Modules, I believe.</p>
<pre><code class="lang-auto">transforms.ToTensor()
</code></pre>

ptrblck: <p>Yes, that’s correct. Implementing a custom <code>Dataset</code> is useful in case you need some specific indexing, processing, etc. If your class images are stored in folders, you can simply use <code>Imagefolder</code>.</p>
TabularDataset taking too much time to load csv

NavinBondade: <p>Hello all.<br>
I’m trying to build a neural machine translation system and trying to load train and Val CSV,  but it’s more than an hour and TabularDataset is still loading can you please tell me the solution for it or alternative methods to load the dataset. I have uploaded the image of the code please click on it to see it in better quality.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/018bf3d01e315be72c2d9d51669bf0af284d44a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/018bf3d01e315be72c2d9d51669bf0af284d44a4" title="screencapture-colab-research-google-drive-1Kwx-samRtoy5d6ZYObJ37vA3yAuTR-sM-2021-05-26-19_22_21"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/018bf3d01e315be72c2d9d51669bf0af284d44a4.png" alt="screencapture-colab-research-google-drive-1Kwx-samRtoy5d6ZYObJ37vA3yAuTR-sM-2021-05-26-19_22_21" data-base62-sha1="dGk5Ok3k0H5lfVCc5Ri515jgPO" width="625" height="500" data-dominant-color="302F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">screencapture-colab-research-google-drive-1Kwx-samRtoy5d6ZYObJ37vA3yAuTR-sM-2021-05-26-19_22_21</span><span class="informations">1264×1011 56.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
InstanceNorm and BatchNorm for batchsize = 1

Ilya_Kotlov: <p>Hi, I have seen this image which describes the difference between InstanceNorm and BatchNorm, so they seem pretty different, correct me if I wrong but BatchNorm with batchsize=1 isn’t equal to InstanceNorm, is it ?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/017be6b431b4e33ed399ff6a49d2168c8f04632b.jpeg" alt="image" data-base62-sha1="d7VWmUrhluF6SxhZPldjS4L1Vh" width="667" height="197"></p>
Inference time on large batch size with high resolution images

TSDK: <p>Hi,<br>
I was trying to calculate the inference time for a couple of batches.<br>
Model being used- ResNet18<br>
num_workers = 4<br>
Image Size = 3, 1024, 3072 (Channels, Height, Width)<br>
Here’s how I’m calculating the inference</p>
<pre><code class="lang-auto">with torch.no_grad():
            for batch_num, data in enumerate(valloader, 0):
                # get the images and labels and move tensors to GPU
                inputs = data["image"]
                labels = data["label"].to(device)
                labels = labels.long()

                
                # For time Sync in Cuda
                torch.cuda.synchronize()

                start_time = time.time()
                output = net(inputs.to(device))

                torch.cuda.synchronize()
                end_time = time.time()

                elapsed_time += (end_time - start_time)


        execution_time = elapsed_time * 1000 # Convert to milliseconds
        avg_inference_time = execution_time / len(valloader)
        print("Avg Inference Time: ",avg_inference_time)
</code></pre>
<p>This is the result I get:<br>
<strong>Avg Inference Time:  60.62364959716797</strong> for a batch size of 3<br>
<strong>Avg Inference Time:  122.38671875</strong> for a batch size of 6<br>
<strong>Avg Inference Time:  103.451</strong> for a batch size of 5<br>
<strong>Avg Inference Time:  20.9023842</strong> for a batch size of 1</p>
<p>If things were running parallel, shouldn’t be the time involved for any batch size be the same as long as it fits in the GPU memory. Or is it because my resolution is quite high, the size of the activation layer grows due to which higher batch size takes more time for inference. Could anyone please throw some light on this topic?<br>
Thanks</p>
Downloading IMDB dataset from torchtext taking 20 mins+ on Colab

vishak_bharadwaj: <p>Trying to run this piece of code from Ben Trevett’s github repo, but it takes a hell of time<br>
for something pretty basic. Any thoughts on why this is happening and how do I get out of it?</p>
<pre><code class="lang-auto">from torchtext.legacy import datasets

train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)
</code></pre>
Predictions Do Not Get Better Over Epochs - Loss Constant From First Epoch On

Frank_Hauser: <p>I am trying to train a NN to predict a made-up function (binary classification problem). In particular, the input features are 12 binary features and the label corresponds to one whenever the second, fifth, or sixth input features are one.</p>
<p>The NN structure looks like this:</p>
<pre><code class="lang-auto">class NeuralNework(nn.Module):
    def __init__(self):
        super(NeuralNework, self).__init__()
        #self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(12, 12),
            nn.ReLU(),
            nn.Linear(12, 12),
            nn.ReLU(),
            nn.Linear(12, 1),
            nn.ReLU(),
        )

    def forward(self, x):
        #x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNework()
</code></pre>
<p>while the train loop is:</p>
<pre><code class="lang-auto">def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X.float())
        y.resize_((batch_size, 1))
        loss = loss_fn(pred, y.float())

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>As a loss function <code>BCEWithLogitsLoss()</code> is used and I tried both the SGD and the Adam optimizer. I believe the might be a problem with how gradients are maintained between epochs (they might be deleted which causes performance to remain the same for each epoch), however, I was not able to figure out what causes the issue.</p>
<p>The current BCE Loss is 0.693147 and I cannot believe the network could not perform better since the problem is so simple.</p>
<p>Any help is greatly appreciated<br>
Thank you very much in advance</p>

eqy: <p>A loss of 0.693 means that the model is not doing any better than a coinflip which is consistent with not learning anything after the random initialization. I don’t see anything obvious that indicates why the model cannot learn more than random chance. Can you share some other parts of the code, such as where the optimizer is defined?</p>

ptrblck: <p>Once issue could be the last <code>nn.ReLU</code>, which would clip the logits to <code>[0, +Inf]</code> and could thus hamper the training, so you might want to remove it.</p>

Frank_Hauser: <p>It turned out the gradients were always pushed towards zero. Batch normalization and dropout solve the problem for me.</p>