akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LSTM TypeError invalid combination

username: demirbbey05
cooked: <p>My LSTM implementation is below<br>
x_train is a size of torch.Size([129, 252, 4])</p>
<pre><code class="lang-python">class LSTM(nn.Module):
  def __init__(self,input_size ,hidden_size,num_layers,batch_size,bias):
    super(LSTM, self).__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.bias = bias
    self.batch_size = batch_size

    self.lstm = nn.LSTM(input_size,hidden_size,num_layers,bias)
    self.logsoftmax = torch.nn.LogSoftmax(dim=None)

  def forward(self,input):
    hidden = (torch.zeros(self.num_layers,self.batch_size,self.hidden_size),torch.zeros(self.num_layers,self.batch_size,self.hidden_size))
    lstm_out,hidden = self.lstm(input,hidden)
    y_pred = logsoftmax(lstm_out[-1,:,:])
    return y_pred
</code></pre>
<p>when I enter that:</p>
<pre><code class="lang-python">model = LSTM(4,2,1,252,bias = "True")
y_pred = model(x_train)
</code></pre>
<p>colab gives a type error:</p>
<pre><code class="lang-python">stm() received an invalid combination of arguments - got (Tensor, tuple, list, str, int, float, bool, bool, bool), but expected one of:
 * (Tensor data, Tensor batch_sizes, tuple of Tensors hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional)
      didn't match because some of the arguments have invalid types: (Tensor, !tuple!, !list!, !str!, !int!, !float!, !bool!, bool, bool)
 * (Tensor input, tuple of Tensors hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional, bool batch_first)
      didn't match because some of the arguments have invalid types: (Tensor, !tuple!, !list!, !str!, int, float, bool, bool, bool)
</code></pre>
<p>sorry for my grammatical mistakes and I am new in forum</p>

username: ptrblck
cooked: <p>Pass the <code>bias</code> argument as the bool value <code>True</code> instead of a <code>string</code>:</p>
<pre><code class="lang-python">model = LSTM(4, 2, 1, 252, bias=True)
x = torch.randn(129, 252, 4)
out = model(x)
</code></pre>
Random Erasing in a fixed region

username: farhanrahman
cooked: <p>Hey, I was wondering if it is possible to use RandomErasing or do random noise in a fixed area. More specifically, I want to know if, my image is say 128x128, will it be possible due to random noise or erasing inside just the central 50x50, or maybe on specific region other than this? Please help! Thanks!</p>

username: ptrblck
cooked: <p>You could write your custom <code>RandomErasing</code> transformation by changing the <code>get_params</code> method in <a href="https://github.com/pytorch/vision/blob/f2f085bf099c4c31bc6f09c21844b2d57dabcb87/torchvision/transforms/transforms.py#L1261-L1262">these lines of code</a> to your specific condition.<br>
This would make sure to samples from your desired area instead of the whole image.</p>
Mixed precision with torch.cuda.amp

username: shrutee
cooked: <p>I have an array of models, corresponding optimizers and losses. Without mixed precision, I sum the losses and then apply backward.</p>
<p><code>(loss1 + loss2 + ...).backward()</code></p>
<p>Now while doing mixed precision training with <code>torch.cuda.amp</code>, acc to the sample code given I should apply backward on individual losses.</p>
<p><code>scaler.scale(loss1).backward()</code><br>
<code>scaler.scale(loss2).backward()</code> and so on.</p>
<p>So I wanted to know what is the correct way - applying <code>backward()</code> on individual losses or on sum total? Is the backward func linear?</p>
<p><strong>Code for reference</strong></p>
<pre><code class="lang-auto">scaler = torch.cuda.amp.GradScaler()

for epoch in epochs:
    for input, target in data:
        optimizer0.zero_grad()
        optimizer1.zero_grad()
        with autocast():
            output0 = model0(input)
            output1 = model1(input)
            loss0 = loss_fn(2 * output0 + 3 * output1, target)
            loss1 = loss_fn(3 * output0 - 5 * output1, target)

        scaler.scale(loss0).backward(retain_graph=True)
        scaler.scale(loss1).backward()

        # You can choose which optimizers receive explicit unscaling, if you
        # want to inspect or modify the gradients of the params they own.
        scaler.unscale_(optimizer0)

        scaler.step(optimizer0)
        scaler.step(optimizer1)

        scaler.update()
</code></pre>

username: ptrblck
cooked: <p>Yes, you can also sum the losses and call <code>backward</code> once. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Will multiple GPUs increase the memory limit (allow deeper models)?

username: Stapo
cooked: <p>I have a GRU model and the depth of my model is limited by my GPU’s memory. Would having two of the same GPU’s allow for twice the depth? Could I also use my SSD or RAM as memory instead (without losing GPU processing)?</p>
<p>In case it is case specific; I have a 2-layer GRU model with 1000 inputs and 500 hidden units (thats my current limit) and would like to increase it to 1000 hidden units. Also I’m aware that reducing batch size would help, but due to class imbalance I can’t reduce it.</p>

username: sidwa0
cooked: <p>You might want to look into model parallel<br>
<a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html#apply-model-parallel-to-existing-modules" rel="nofollow noopener">model parallel best practices</a></p>

username: Stapo
cooked: <p>Thanks for the link, it answers my initial question. The pytorch pages I found earlier didn’t give me a definitive answer and here it clearly states that It can solve memory issues.</p>
<p>Although I’m still wondering about the 2nd part of my question regarding using SSD or RAM as a solution.</p>

username: ayalaa2
cooked: <p>To clarify, are you wondering if you can store modules on RAM, and then load them as you need to? For example, if a two layer model is too massive, storing layer1 on the GPU and layer2 on ram?</p>
<p>Not really is the short answer. The modules need to be on the GPU to allow for the GPU to perform the forward and backward pass. You could do a thing where once the forward pass is completed on layer1, move it to RAM and move layer2 to the GPU, and continue the forward pass. But this will be quite slow to move between memory every forward pass (and I’m not sure how happy autograd will be about this).</p>
<p><a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html#part-of-the-model-on-cpu-and-part-on-the-gpu" rel="nofollow noopener">However, you can just keep part of the model on the CPU.</a>. It won’t be as fast as having it all on a GPU, but it’s a possible route.</p>

username: sidwa0
cooked: <p>you can have part of you model in gpu and other part in RAM. But as far as I know SSD storage is not supported and probably will be much slower than RAM.</p>
<p><a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="nofollow noopener">partial GPU partial CPU model</a></p>
<p>I would stick to models that completely fit within GPU vram since the cpu operations could be a huge bottleneck since CPUs can’t get nearly as much parallelism as GPUs. Also it could get tricky, loading the model during inference time where some weight tensors are GPU and some are CPU.</p>
Combine scikit learn with pytorch

username: AnxheloD
cooked: <p>Hello, i am new to pytorch and i wanted to know can we combine pytorch dl models with other ml models from scikit learn?</p>

username: fs4ss1
cooked: <p>First of all, welcome to the community!</p>
<p>You can use the deep learning models as a feature extractor. You need to create a pipeline like this:</p>
<ul>
<li>First you input your data in the DL model to extract the features;</li>
<li>Then you input the extracted features in the sklearn model.</li>
</ul>
<p>You need to train the models separately.</p>

username: AnxheloD
cooked: <p>First of all thank you.<br>
So basically i have to use two shared loss, one for training the DL model and one for the other model, and from DL have to use the last layer output before the prediction if i got it right</p>

username: fs4ss1
cooked: <p>Yes. You need to train the DL model, then use the output of this model to train the sklearn model.</p>

username: AnxheloD
cooked: <p>Thank you very much for your purpose</p>
nn.ConvTranspose2d, Upscaling back to original image dimensions?

username: murphinator
cooked: <p>Just learning the ropes on PyTorch. Kind of a newbie here.<br>
Attempting to construct a convolutional autoencoder on MNIST.<br>
Here is my desired network architecture:</p>
<pre><code class="lang-auto">class tCNN_Autoencoder(nn.Module):
    def __init__(self):
        super(tCNN_Autoencoder, self).__init__()
        # Encoder
        self.en_conv1 = nn.Conv2d(1,16,3,padding=1)
        self.en_conv2 = nn.Conv2d(16,32,3,padding=1)
        self.en_conv3 = nn.Conv2d(32,4,3,padding=1)
        # Decoder
        self.de_tconv1 = nn.ConvTranspose2d(4,32,2,stride=2)
        self.de_tconv2 = nn.ConvTranspose2d(32,16,2,stride=2)
        self.de_tconv3 = nn.ConvTranspose2d(16,1,2,stride=2)
        
        self.pool = nn.MaxPool2d(2,2)
    
    def forward(self, x):
        x = F.relu(self.en_conv1(x))
        x = self.pool(x)
        x = F.relu(self.en_conv2(x))
        x = self.pool(x)
        x = F.relu(self.en_conv3(x))
        x = self.pool(x) #Compressed representation
        x = F.relu(self.de_tconv1(x))
        x = F.relu(self.de_tconv2(x))
        x = F.sigmoid(self.de_tconv3(x)) # sigmoid to scale the pixel values in grayscale from 0 to 1
        return x
</code></pre>
<p>And here are my tensor sizes that I get when I run my training loop:</p>
<pre><code class="lang-auto">torch.Size([28, 1, 28, 28]) before conv1
torch.Size([28, 16, 28, 28]) after conv1
torch.Size([28, 16, 14, 14]) after pooling at conv1
torch.Size([28, 32, 14, 14]) after conv2
torch.Size([28, 32, 7, 7]) after pooling at conv2
torch.Size([28, 4, 7, 7]) after conv3
torch.Size([28, 4, 3, 3]) after final pooling (compressed representation)
torch.Size([28, 32, 6, 6]) after tconv1
torch.Size([28, 16, 12, 12]) after tconv2
torch.Size([28, 1, 24, 24]) after sigmoid
</code></pre>
<p>I can see that I am losing 4 pixels on each h x w dimension, but I don’t completely understand how to scale back up to the original image size of 28x28 using the tconv layers.</p>

username: murphinator
cooked: <p>I suppose it would be related to the downsampled output from the final intermediate max pool layer (showing the compressed representation), giving me a 3x3…but I can’t really upscale that nicely into a 28x28 because 3 is not a multiple of 28. Is there a general principle here that I am ignoring?</p>
How to efficiently save the logits output of a model

username: chaoyanghe
cooked: <p>Hi, I am trying to save the logits output of a model, but my code cost super large GPU memory and finally gets the OOM exception. I tried to print the memory cost of each logit. The result shows its memory cost is small. I don’t know how to further debug this issue. Please let me know how to efficiently save the logits.</p>
<p>the code is:</p>
<pre><code class="lang-auto">        net.eval()
        for batch_idx, (images, labels) in enumerate(self.local_training_data):
            images, labels = images.to(self.device), labels.to(self.device)
            log_probs, extracted_features = net(images)
            #self.extracted_feature_dict[batch_idx] = extracted_features.cpu()
            logging.info("shape = " + str(log_probs.shape))
            logging.info("element size = " + str(log_probs.element_size()))
            logging.info("nelement = " + str(log_probs.nelement()))
            logging.info("GPU memory1 = " + str(log_probs.nelement() * log_probs.element_size()))
            self.logits_dict[batch_idx] = log_probs.cpu()
</code></pre>
<p>the model definition is:</p>
<pre><code class="lang-auto">    def forward(self, x):

        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)  # B x 16 x 32 x 32
        extracted_features = x

        x = self.layer1(x)  # B x 16 x 32 x 32

        # output here
        x = self.avgpool(x)  # B x 16 x 1 x 1
        x_f = x.view(x.size(0), -1)  # B x 16
        logits = self.fc(x_f)  # B x num_classes
        return logits, extracted_features
</code></pre>
<p>the log shows:</p>
<pre><code class="lang-auto">INFO:root:shape = torch.Size([64, 10])
INFO:root:element size = 4
INFO:root:nelement = 640
INFO:root:GPU memory1 = 2560
</code></pre>
<p>the exception is:</p>
<pre><code class="lang-auto">  File "/home/chaoyanghe/sourcecode/FederatedLearning/group_knowledge_transfer_single/models/cifar/model_client.py", line 55, in forward
    out = self.bn1(out)
  File "/home/chaoyanghe/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/chaoyanghe/anaconda3/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 107, in forward
    exponential_average_factor, self.eps)
  File "/home/chaoyanghe/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.76 GiB total capacity; 9.93 GiB already allocated; 18.19 MiB free; 9.96 GiB reserved in total by PyTorch)

</code></pre>
<p>Hardware environment: RTX 2080Ti<br>
Software environment: torch 1.4.0, python 3.7.4</p>
Memory efficient HDF5 dataset loading

username: omrjml
cooked: <p>I am trying to train a model with a dataset containinng ~11 million samples of 1d vectors contained in an HDF5 format file. It all runs ok with smaller datasets, but when I try with the full dataset, I find the training just hangs and I cannot get through even a single epoch after several hours of running on a GPU. I am wondering if I am exceeding system memory even though I think I am doing lazy loading. The code I am using is similar to below:</p>
<pre><code class="lang-auto">class ConcatDataset(torch.utils.data.Dataset):
    def __init__(self, xdata, ydata):
        self.xdatasets = xdata
        self.ydatasets = ydata

    def __getitem__(self, i):
        xd = [torch.tensor(d[i]) for d in self.xdatasets]
        x = torch.cat(xd)
        y = torch.cat([torch.tensor(d[i]) for d in self.ydatasets])
        return (x.to(device),y.to(device))

    def __len__(self):
        return min(len(d) for d in self.xdatasets)

train_loader = torch.utils.data.DataLoader(
             ConcatDataset(x,y),
             batch_size=args.batch_size, shuffle=True)
</code></pre>
<p>I am after some advice that I am not missing something obvious above. I am trying with a batch size of 100.</p>
Better definition of loss

username: Angry_potato
cooked: <p>Hi,<br>
Is it better to define a custom loss function in the following way<br>
‘’’<br>
class loss(torch.nn.Module):<br>
code for loss<br>
‘’’<br>
or is is better to implement it as a normal function.</p>

username: JuanFMontesinos
cooked: <p>Whatever fits to your problem.<br>
Pytorch losses are defined as modules because they may contain weights.</p>

username: Angry_potato
cooked: <p>So the way loss is implemented wont affect the speed,right?</p>

username: JuanFMontesinos
cooked: <p>Nope. In the end using a nn.Module is a workaround. You will be more or less end up calling the same function than if you code it as a function.</p>
Updating learning rate after accumulating gradient

username: chantk
cooked: <p>Hello, I have some idea on how to update the learning rate of optimizer but I am not 100% sure of the correct way to update the learning rate if you are accumulating the gradient to achieve a larger batch size.</p>
<p>For example: if I use the following code to accumulate gradient on training examples of batch size 8 for 4 iterations in order to achieve a 32 batch size.</p>
<blockquote>
<p>opt.zero_grad()<br>
for i, (input, target) in enumerate(dataset):<br>
pred = net(input)<br>
loss = crit(pred, target)<br>
loss.backward()<br>
# graph is cleared here<br>
if (i+1)%4 == 0:<br>
# every 4 iterations of batches of size 8<br>
opt.step()<br>
opt.zero_grad()</p>
</blockquote>
<p>Then assuming if I want to update my learning rate every 200 iterations (every 200 gradient update of virtual batch size 32), the code should following the same principle</p>
<blockquote>
<p>if 200*(i+1)%4 == 0:<br>
for param_group in opt.param_groups:<br>
param_group[‘lr’] = param_group[‘lr’] * 0.9</p>
</blockquote>
<p>Or is there no need to schedule it that way and instead perform the following</p>
<blockquote>
<p>if i%200 == 0:<br>
for param_group in opt.param_groups:<br>
param_group[‘lr’] = param_group[‘lr’] * 0.9</p>
</blockquote>
<p>logically it seems like the first one is correct though. if the first one is correct, what error would it cause in the second code? gradient update with different learning rate within the same batch?</p>

username: JuanFMontesinos
cooked: <p>Kind of yes. Optimizer is agnostic to the learning rate. Batch would be updated with a single learning rate anyway as the one which counts is the one existing when you call opt.step</p>
Memory increases from first to second forward pass during evaluation

username: mcPytorch
cooked: <p>When I iterate through my evaluation code while watching memory usage on GPU using nvidia-smi, I can strangely see that the first batch gets processed with 7 GB of GPU memory. Upon forwarding the second batch the memory usage on GPU grows to 14 GB and stays constant afterwards.</p>
<p>I have my model in evaluation mode.</p>
<pre><code class="lang-auto">model.cuda()
model.eval()
test_loader = DataLoader(test_reader, batchsize, num_workers=0,
                         shuffle=False, pin_memory=True, drop_last=False)
for itr, (image_batch) in enumerate(test_loader):

        # Forward Pass
        image_batch = image_batch.cuda()
        prediction_batch = model.forward(image_batch)
</code></pre>
<p>Any idea why this is so?<br>
I assume the model doesn’t require gradients in evaluation mode. Or does the model still store gradients?</p>

username: JuanFMontesinos
cooked: <p>you should use the context manager with torch.no_grad()<br>
There will be gradients otherwise</p>
Crop rectangular portions from a batch of tensors

username: Gkv
cooked: <p>Hi all,</p>
<p>Is there a function to crop rectangular boxes from a batch of tensors. More precisely, given a tensor of size BxMxN and a bounding box coordinate tensor of size Bx4 (the four values represent  (x,y,x+w,y+h) where (x,y) is the top-left co-ordinate value), the task is to crop the corresponding rectangular portions from each of the MxN tensors. Can we do it in one step without using any loops?.</p>
<p>Thanks</p>

username: phan_phan
cooked: <p>Hi,<br>
I don’t know of a way to do this, but <a href="https://discuss.pytorch.org/t/crop-multiple-images-with-indices/56862">here’s a link</a> to someone trying to do exactly the same thing.<br>
Good luck</p>

username: Aman_Arora
cooked: <p>Thank you! This really helped <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Changing specific elements in tensor

username: Dan_Sagher
cooked: <p>Hi,<br>
I’m looking for some function or any efficient way to change specific elements in a tensor.<br>
Let’s say for example that I need to add 100 only for elements that are bigger than 5, and the other elements should stay the same.<br>
Does anyone know how can I do it efficiently?<br>
Thank you.</p>

username: netaglazer
cooked: <p>if x is your tensor<br>
you can do:</p>
<p>torch.where(x &gt; 5, x+100, x)</p>

username: Dan_Sagher
cooked: <p>Thank you for your comment (:</p>
Compression for tensor storage

username: nirandaperera
cooked: <p>would it make sense to add floating point compression for tensor storage like zfp? Just a thought! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://computing.llnl.gov/sites/all/themes/zen_computation/favicon.ico" class="site-icon" width="15" height="15">
      <a href="https://computing.llnl.gov/projects/floating-point-compression" target="_blank" title="11:21AM - 23 November 2015" rel="nofollow noopener">Computing – 23 Nov 15</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="20" height="20">

<h3><a href="https://computing.llnl.gov/projects/floating-point-compression" target="_blank" rel="nofollow noopener">zfp &amp; fpzip: Floating Point Compression</a></h3>

<p>zfp is an open source C/C++ library for compressed floating-point and integer arrays that support high throughput read and write random access. It was designed to achieve high compression ratios and therefore uses lossy but optionally error-bounded...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Calculation of covariance matrix of image

username: Frida
cooked: <p>does someone know how to calculate covariance matrix of an image in pytorch?</p>

username: vmirly1
cooked: <p>Can you mathematically describe what you want? What is covariance of a single image?</p>

username: apollack
cooked: <p>Hello,<br>
I was wondering if you still had any thoughts on this. I’m trying to calculate the structural dissimilarity index metric (DSSIM) and there for using the structural similarity metric (SSIM) to do so. As part of the formula, I need to find the covariance  of two images <em>x</em> and <em>xT</em>. Do you know how I can go about doing this? I know the <code>numpy.cov</code> seems to through a fit because of the three dimensional shape.</p>
ValueError: Classification metrics can't handle a mix of multilabel-indicator and binary targets

username: vidivinc
cooked: <p>hello everybody!</p>
<p>I was trying to to an multiclass classification with a cnn, but when I try to</p>
<p>from sklearn.metrics import accuracy_score</p>
<p>accuracy_score(y_test_, prediction.round(),normalize=True)</p>
<p>this error accoured:</p>
<p>ValueError: Classification metrics can’t handle a mix of multilabel-indicator and binary targets</p>
<p>Could anyone help me?<br>
I’ll write all the code you need to explain me better.<br>
Really thanks</p>

username: ptrblck
cooked: <p>Could you post the shapes od <code>y_test</code> and <code>prediction</code>?<br>
Based on the error message, I assume <code>accuracy_score</code> expects other shapes than the one you’ve passed.</p>

username: vidivinc
cooked: <p>shapes are</p>
<p>y_test_ (12630,43)<br>
prediction(12630,).<br>
I am sure that you are right. How can I fix it?<br>
Really thanks</p>

username: ptrblck
cooked: <p>Based on the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html">docs</a> 1-dimensional tensors are required by this method.<br>
If your target is one-hot encoded, you could get the class indices via <code>y_test = torch.argmax(y_test, dim=1)</code>.</p>

username: vidivinc
cooked: <p>Really thanks for the support, I found the mistake.<br>
You helped me to reason on my code.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji only-emoji" alt=":slight_smile:"></p>
OBJECT DETECTION TUTORIAL - XML annotations

username: azer
cooked: <p>Hi</p>
<p>I am new to computer vision  and would like to use Torchvision object detection fine tutorial to process my dataset which has 4 categories. Also the annotations are is xml format. I will use the pretrained model so I was wondering if  you can advise what to do to get the label correctly.</p>
<p>Thanks</p>
Lib size almost does not change when doing custom build on iOS compared to Android

username: hhov
cooked: <p>Hi Folks,</p>
<p>On master branch when doing custom build for Android I have got 40% decrease in Lib size (for libpytorch_jni.so). However when doing custom build for iOS with the same .yaml file I have got about 2.7% decrease (for libtorch_cpu.a).<br>
I did custom builds as described here: <a href="https://pytorch.org/mobile/home/" rel="nofollow noopener">https://pytorch.org/mobile/home/</a></p>
<p>I have also created following GitHub issue:<br>
<a href="https://github.com/pytorch/pytorch/issues/35306" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/35306</a></p>
<p>Regards,<br>
Hovhannes</p>

username: xta0
cooked: <p><a class="mention" href="/u/hhov">@hhov</a>  The size reduction happens when you link your static libraries to output the final binary, as linker can strip most of the unused code ( <code>-dead_strip</code> ). Unlike dynamic libraries(.so in Android), linker can’t strip code when compiling static libraries.</p>

username: hhov
cooked: <p><a class="mention" href="/u/xta0">@xta0</a> Thanks for info. Besides comparing .a file sizes I also checked size as described here:<a href="https://stackoverflow.com/questions/32003262/find-size-contributed-by-each-external-library-on-ios/39593377" rel="nofollow noopener">https://stackoverflow.com/questions/32003262/find-size-contributed-by-each-external-library-on-ios/39593377</a><br>
Both ways gave insignificant difference. I tried also with <code>-dead_strip</code> but seams XCode is already doing it.<br>
After that I tried simply comparing final .ipa files and result was as expected, about 40% decrease in size.</p>
Deeplabv3 Sizes of tensors must match except in dimension 1

username: zimmer550
cooked: <p>Hello. I am using the following code with deeplabv3_resnetd152b_voc to get semantic segmentation:</p>
<pre><code class="lang-auto">import argparse
import os
import cv2
import natsort
from pytorchcv.model_provider import get_model
import torch
import math
import numpy as np

#SOME CODE TO GET IMAGE ARRAY...

semantic_net = get_model("deeplabv3_resnetd152b_voc", pretrained=True)
semantic_net.eval()
if torch.cuda.is_available():
    semantic_net.cuda()
for img in left_imgs[:1]:
    cv_img = cv2.imread(os.path.join(left_img_folder, img), flags=cv2.IMREAD_COLOR)
    cv_img = cv2.cvtColor(cv_img, code=cv2.COLOR_BGR2RGB)
    cv_img = cv2.resize(cv_img, (224, 224), interpolation=cv2.INTER_AREA)

    x = cv_img.astype(np.float32)
    x = x / 255.0
    x = (x - np.array([0.485, 0.456, 0.406])) / np.array([0.229, 0.224, 0.225])
    x = x.transpose(2, 0, 1)
    x = np.expand_dims(x, axis=0)
    x = torch.FloatTensor(x)
    if torch.cuda.is_available(): x = x.cuda()
    y = semantic_net(x)
    print(y)
</code></pre>
<p>The pytorchcv.model_provider was taken from here: <a href="https://github.com/osmr/imgclsmob/tree/4867ad5b19fa01e8ef80f253c0487f3549d853f2" rel="nofollow noopener">https://github.com/osmr/imgclsmob/tree/4867ad5b19fa01e8ef80f253c0487f3549d853f2</a></p>
<p>I get the following error when I run the above code:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/sarim/PycharmProjects/trajectory_pred/3d_pred/one_shot.py", line 50, in &lt;module&gt;
    with torch.no_grad(): output = semantic_net(input_batch)['out'][0]
  File "/home/sarim/PycharmProjects/trajectory_pred/venv3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarim/PycharmProjects/trajectory_pred/venv3.7/lib/python3.7/site-packages/pytorchcv/models/deeplabv3.py", line 202, in forward
    x = self.pool(x)
  File "/home/sarim/PycharmProjects/trajectory_pred/venv3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarim/PycharmProjects/trajectory_pred/venv3.7/lib/python3.7/site-packages/pytorchcv/models/deeplabv3.py", line 130, in forward
    x = self.branches(x)
  File "/home/sarim/PycharmProjects/trajectory_pred/venv3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sarim/PycharmProjects/trajectory_pred/venv3.7/lib/python3.7/site-packages/pytorchcv/models/common.py", line 1115, in forward
    out = torch.cat(tuple(out), dim=self.axis)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 60 and 28 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:71
</code></pre>
<p>Please tell me what could be wrong here. Thank you!</p>
Check whether checkpoint loaded or not

username: Kenzo
cooked: <p>Hi,<br>
This is the way I usually load the checkpoint. I want to know how can I check whether the checkpoint is loaded or not?</p>
<pre><code class="lang-auto">    Loaded = torch.load('../model/model_baseline_x{}.pt'.format(scale))
    GHR.load_state_dict(Loaded)
</code></pre>

username: ptrblck
cooked: <p>If the <code>load_state_dict</code> function does not throw any error, it should be working.<br>
Alternatively, you could print some parameter of a random layer before and after loading the <code>state_dict</code> to make sure it changed.</p>
Pytorch model to TensorRT

username: FelipeVW
cooked: <p>Hi,</p>
<p>I am willing to export the <code>MaskRCNNPredictor</code> of</p>
<blockquote>
<p>torchvision.models.detection.mask_rcnn</p>
</blockquote>
<p>To TensorRT via torch2trt or via onnx. My doubt is if I will have problems with some unsupported layers. For instance, is <code>FrozenBatchNorm2d()</code> supported? I was looking them here  <a href="https://docs.nvidia.com/deeplearning/sdk/tensorrt-support-matrix/index.html#supported-ops" class="inline-onebox" rel="noopener nofollow ugc">Support Matrix :: NVIDIA Deep Learning TensorRT Documentation</a> but just saw <code>BatchNormalization</code></p>
<p>Thank you for your time</p>
DataLoader batch first

username: denis
cooked: <p>Hey everyone,</p>
<p>I am gonna use torch.DataLoader for RNNs but it forms batches as batch first: <em>[batch x seq x data]</em>. However I would like to have the data in default pytorch RNN format: <em>[seq x batch x data].</em><br>
I could not see any easy way to do like <em>batch_first=False</em></p>
<p>Could you please help me with?</p>

username: shubhvachher
cooked: <p>The dataloader is just a utility used to work with the dataset class (passed as the first argument to the dataloader) Check out the dataset class that you pass to the dataloader. That would be where you’d have coded in the dimensions of the data returned by dataset class and thus your dataloader.</p>
<p>If for whatever reason this dataset class isn’t accessible to you; check out the collate function : <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3" class="inline-onebox">How to create a dataloader with variable-size input</a> and <code>input.permute(1, 0, 2)</code> your batches there.</p>
<p>Check out this useful answers on using RNNs with PyTorch <a href="https://stackoverflow.com/questions/49466894/how-to-correctly-give-inputs-to-embedding-lstm-and-linear-layers-in-pytorch/49473068#49473068" rel="nofollow noopener">https://stackoverflow.com/questions/49466894/how-to-correctly-give-inputs-to-embedding-lstm-and-linear-layers-in-pytorch/49473068#49473068</a></p>

username: vpn
cooked: <p>I ran into a similar issue and made a mistake of using <code>torch.reshape(x,(seq,batch,data))</code>. But this resulted in messing the order of my time series data. The way I fixed was using <code>torch.permute(x,(1,0,2))</code></p>
Binary Classification of Time Series: When to reset LSTM hidden/cell state?

username: ppmt
cooked: <p>I am working on a binary classifier for time series data with one feature. However, I am unsure when exactly to reset the hidden and cell states and why.</p>
<p>Currently I am using the following classifier:</p>
<pre><code class="lang-auto">class LSTMClassifier(nn.Module):
    def __init__(self,
                 input_features_size=1,
                 hidden_layer_size=30,
                 num_layers=1,
                 output_size=2):
        super().__init__()
        self.lstm = nn.LSTM(input_size=input_features_size,
                            hidden_size=hidden_layer_size,
                            num_layers=num_layers,
                            batch_first=True)
        self.linear = nn.Linear(hidden_layer_size, output_size)
        # number of features in hidden layer
        self.hidden_layer_size = hidden_layer_size
        # number of input features (i.e., features per sequence point)
        self.input_features_size = input_features_size
        # number of layers
        self.num_layers = num_layers
        # size of batches used in gradient descent
        self.batch_size = None
        # tuple with hidden state (a.k.a. short term memory) and cell state (a.k.a. long term memory)
        self.hidden_cell = (None, None)

    def init_hidden(self, batch_size):
        self.batch_size = batch_size
        self.hidden_cell = (torch.zeros(self.num_layers, batch_size, self.hidden_layer_size),
                            torch.zeros(self.num_layers, batch_size, self.hidden_layer_size))

    def forward(self, x_batch):
        batch_size = x_batch.shape[0]
        self.init_hidden(batch_size)
        lstm_out, self.hidden_cell = self.lstm(x_batch, self.hidden_cell)
        predictions = self.linear(lstm_out[:, -1, :])
        return predictions
</code></pre>
<p>The hidden and cell state get reset before every forward pass. The reasoning behind this is that the individual time series instances are seen as independent of each other. Thus, the ‘kept information’ from the previous instance is not relevant for the next one etc. I suppose the actual learning of what information to keep in hidden and cell state when parsing an instance is done by updating the weights of the individual gates.</p>
<p>Is this reasoning correct?</p>
<p>I have found examples resetting the hidden state only once before training, every epoch, and before every forward pass on different sites leaving me quite confused.</p>
<p>Cheers,<br>
ppmt</p>
Group convolution in pytorch

username: qhj
cooked: <p>Hello everyone, I have a question that I have been puzzled for a long time.<br>
In the conv2d function, only the arg “group” controls the group convolution. But what can i do to specify some specific channels for convolution? For examole, the convolution layer connections of lenet shown in the following picture.Someone can help me？ Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bdeae0861695948023cfdaa63ef598af7cc6eb6c.png" alt="013017_pIe9_876354" data-base62-sha1="r65sKknWQe0FGXCXmWSuRw4rWws" width="558" height="275"></p>

username: ptrblck
cooked: <p>Could you explain the image a bit and what the axes and labels stand for?</p>

username: qhj
cooked: <p>Each column represents a convolution kernel(output channels) and each row represents an input channel. In this conv layer the input channel number is 6 and the output channel number is 16.The label “x” represents the connections. For example, the outout channel 0 is connected with input channel 0,1 and 2.</p>
Tips or tricks on debugging a model that was packaged with nn. Sequential?

username: inkplay
cooked: <p>I am a self-taught python user so my knowledge is limited, I want to know how to debug my packaged model if something goes wrong in a certain operation. A basic example would be:</p>
<p><code>model = [ conv1, bn1, relu1, conv2, bn2, relu2] model = nn.Sequential( *model ) model(tensor)</code><br>
Say if you run into shape error and you would like to check the input before a particular calculation how do you do it?</p>
<p>Right now I am trying to experiment with nn.Sequential more, with a Class function I can explicitly define and check each forward step for errors, what are some tips on debugging a packaged model?</p>

username: ptrblck
cooked: <p>You could create some layers to debug the model.<br>
If you run into a shape mismatch error, you could use a layer which prints the shape of the input:</p>
<pre><code class="lang-auto">class Print(nn.Module):
    def __init__(self):
        super(Print, self).__init__()

    def forward(self, x):
        print(x.shape)
        return x
</code></pre>
<p>Now you could place this layer into your <code>nn.Sequential</code> model.</p>

username: inkplay
cooked: <p>Thanks you ptrblck you are aways so active and helpful!</p>

username: vb000
cooked: <p>Hi,</p>
<p>I’m wondering does this also work with <code>backward</code>?</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>The <code>Print</code> module shouldn’t change the backward pass, as it’s just forwarding the input.<br>
Are you seeing any issues?</p>

username: vb000
cooked: <p>No, not any issues – the graph is intact, I believe. But was just wondering if I could use the same approach to debug gradients…</p>

username: ptrblck
cooked: <p>To debug gradients, you could use <code>tensor.register_hook</code> and print out the gradient or some stats,<br>
which might be better suited than the <code>Print</code> module.</p>

username: vb000
cooked: <p>I imagine <code>register_hook</code> is only defined for tensors? Is it possible to achieve something similar for functional layers like <code>relu</code>?</p>

username: ptrblck
cooked: <p>Yeah, more or less.<br>
You could use <a href="https://pytorch.org/docs/stable/nn.html?highlight=register_backward_hook#torch.nn.Module.register_backward_hook">register_backward_hook</a> on a module, but note the warning, as this might not give you the expected results for complex modules.</p>
Is there a better way to find wrong test results?

username: erickulcyk
cooked: <p>I am just starting with pytorch, and following the MNist CNN example code I found.  The test method looks like this:</p>
<pre><code class="lang-auto">def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += F.nll_loss(output, target, reduction='sum').item()  # sum up batch loss
            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()
</code></pre>
<p>And so I wanted to display the wrong results.  After some stumbling through documentation, I arrived with this solution:</p>
<pre><code class="lang-auto">            z = torch.zeros_like(pred)
            o = torch.ones_like(pred)
            ne = pred.ne(target.view_as(pred))
            w = torch.where(pred.ne(target.view_as(pred)), o, z)
            wrong = torch.nonzero(w).squeeze()[:,0]
            wrongdata = torch.index_select(data, 0, wrong)
            plt.figure(figsize=(15,15))
            plt.subplot(1,2,1)
            plt.axis("off")
            plt.title("Real Images")
            deviceData = wrongdata.to(device)[:64]

            plt.imshow(np.transpose(vutils.make_grid(deviceData, padding=5, normalize=True).cpu(),(1,2,0)))
            plt.show()
</code></pre>
<p>I think that getting all that data could be done more elegantly/simply, but I am not sure how.  Could someone help me out?</p>

username: charan_Vjy
cooked: <p>Below is a quicker way to get this done.  I have considered a binary classification problem. This can be easily extended to a multi class classification problem.</p>
<pre><code class="lang-auto">import numpy as np
import torch
data = torch.randn(64,224,224)
target = torch.from_numpy(np.random.randint(0,1,64))
predicted = torch.from_numpy(np.random.randn(64,2))
pred = predicted.argmax(dim = 1) ## Note that keepdim is false
### This line would do the trick of obtaining incorrect data
incorrect_data = data[pred!=target,:,:]
</code></pre>
Confusion regarding RNN example in docs

username: Baka_Moon
cooked: <p>Here is an example of RNN from scratch from  Pytorch docs (scroll down to example 2):</p>
<p><a href="https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html</a></p>
<p>However, I thought RNN was supposed to have a new feature input at each TIME_STEP instead of just  loading all of the features in at one time.</p>
<p>So using the example from the tutorial above:<br>
instead of<br>
<code>batch = torch.randn(batch_size, 50)</code><br>
being fed into with all 50 sequence features at one time, shouldn’t the sequence of features be feed in one at a time?</p>
<p>Would this involve some sort of Pytorch slicing?</p>

username: charan_Vjy
cooked: <p>The length of  one sequence is 50. All the 50 values in the sequence is part of the first time step. The next 50 values will part of the second time step.</p>
<p>Have slightly tweaked the Pytorch example to drive this point home</p>
<pre><code class="lang-auto">loss_fn = nn.MSELoss()

batch_size = 10
TIMESTEPS = 5

# Create some fake data
batch = torch.randn(batch_size, 50,5)
hidden = torch.zeros(batch_size, 20)
target = torch.zeros(batch_size, 10)

loss = 0
for t in range(TIMESTEPS):
    # yes! you can reuse the same network several times,
    # sum up the losses, and call backward!
    hidden, output = rnn(batch[:,:,t], hidden)
    loss += loss_fn(output, target)
loss.backward()
</code></pre>
Upsample image linearly along one axis

username: soulslicer
cooked: <p>I have some kind of special field, and i want to upsample it by 2x linearly along just the height/rows. Is this possible and what API would i use</p>

username: ptrblck
cooked: <p>You could use <code>nn.Upsample</code> or the functional API via <code>F.upsample</code>.<br>
Would that work for you or are you looking for another specific method?</p>
Printing gradients of functional layers

username: vb000
cooked: <p>Hi,</p>
<p>Is there a way we could print gradient flow across functional layers like <code>relu</code> and <code>softmax</code>? For example, in the program below:</p>
<pre><code class="lang-auto"># Network
class LeNet5(nn.Module):
    def __init__(self):
        super(LeNet5, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(1, 6, kernel_size=(5, 5)),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2), stride=2),
            nn.Conv2d(6, 16, kernel_size=(5, 5)),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2), stride=2),
            nn.Conv2d(16, 120, kernel_size=(5, 5)),
        )

        self.fc = nn.Sequential(
            nn.Linear(120, 84),
            nn.ReLU(),
            nn.Linear(84, 10),
            nn.LogSoftmax(dim=-1),
        )

    def forward(self, data):
        x = self.conv(data)
        x = x.view(x.shape[0], -1)
        x = self.fc(x)
        return x

net = LeNet5()
input = torch.rand(1, 1, 32, 32)
output = net(input)

grad = torch.rand(output.shape)
output.backward(grad) # Can I print gradient flow of this operation?
</code></pre>
<p>Is there a way print the gradients flow (gradient input and output) of <code>self.fc[0]</code> and <code>self.fc[3]</code> layers?</p>
Spatial softmax

username: raghadalghonaim
cooked: <p>Greetings,</p>
<p>Spatial softmax layer is used to find the expected pixel location of feature maps, TensorFlow has an implementation for it. But I cannot find Torch’s one.</p>
<p>Could someone please help me with this?</p>
<p>FYI, here Is the link of the paper first introduced this concept. <a href="https://arxiv.org/abs/1504.00702" rel="nofollow noopener">https://arxiv.org/abs/1504.00702</a>.</p>
Am I applying image normalization correctly here?

username: drliu
cooked: <p>I’m not sure whether I’m normalizing images properly in the right dimension before feeding it into an autoencoder in PyTorch.</p>
<p><strong>Dataset:</strong></p>
<p>I’m using the Omniglot dataset, which contains 20 samples each of 964 characters and each character is 105 * 105 * 1 (greyscale).</p>
<p>Before I feed the training data into the autoencoder, it has this shape:</p>
<pre><code>Xtrain.shape = (964, 20, 105, 105)
</code></pre>
<p><strong>How I’m currently calculating mean and std and applying the transform:</strong></p>
<pre><code>class OmniglotDataset(Dataset):

    def __init__(self, X, transform=None):
        self.X = X.reshape(-1, 105, 105)
        self.transform = transform
    
    def __len__(self):
        return self.X.shape[0]
    
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        img = self.X[idx]
        if self.transform:
            img = self.transform(img)
        return img

X1 = Xtrain.reshape(-1, 105, 105)
mean = X1.mean(axis=0).mean()
std = X1.mean(axis=0).std()
batch_size = 128
img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((mean,), (std,))
])

train_dataset = OmniglotDataset(Xtrain, transform=img_transform)
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
</code></pre>
<p><strong>Questions:</strong></p>
<ol>
<li>Am I applying normalization correctly here?</li>
<li>Is there a faster way to calculate mean and std?</li>
</ol>
Problem about reshaping a tensor

username: stvhuang
cooked: <p>How can i convert a tensor like this:</p>
<pre><code class="lang-python">tensor([[0, 1, 2, 3],
        [4, 5, 6, 7]])
</code></pre>
<p>to this</p>
<pre><code class="lang-python">tensor([[[0, 1],
         [4, 5]],
        [[2, 3],
         [6, 7]]])
</code></pre>
<p>and vice versa.<br>
I’ve tried <code>view</code>, but <code>a.view(2, 2, 2)</code> will change it to</p>
<pre><code class="lang-python">tensor([[[0, 1],
         [2, 3]],
        [[4, 5],
         [6, 7]]])
</code></pre>
<p>Thanks!</p>

username: yash1994
cooked: <p>You can apply following operation:<br>
<code>x.transpose(0,1).view(2,2,2).transpose(1,2)</code><br>
where x is your tensor.</p>

username: stvhuang
cooked: <p>Thanks! It works as expected.</p>
How to get tensor slices?

username: uhmbg
cooked: <p>If I have a tensor like this<br>
tensor([[0.3843, 0.3552]],<br>
[[0.1743, 0.2439]],<br>
[[0.3474, 0.1554]],<br>
[[0.1325, 0.0778]],<br>
[[0.2280, 0.1096]],<br>
[[0.3842, 0.3997]],<br>
[[0.1599, 0.0283]],<br>
[[0.1876, 0.0815]]])<br>
How could I get a tensor from him like that:<br>
tensor([[0.3843]],<br>
[[0.1743]],<br>
[[0.3474]],<br>
[[0.1325]],<br>
[[0.2280]],<br>
[[0.3842]],<br>
[[0.1599]],<br>
[[0.1876]]])</p>

username: ptrblck
cooked: <p>You can directly index tensors via <code>tensor[]</code> as e.g. in numpy:</p>
<pre><code class="lang-python">x = torch.tensor([[[0.3843, 0.3552]],
                [[0.1743, 0.2439]],
                [[0.3474, 0.1554]],
                [[0.1325, 0.0778]],
                [[0.2280, 0.1096]],
                [[0.3842, 0.3997]],
                [[0.1599, 0.0283]],
                [[0.1876, 0.0815]]])

print(x[:, :, 0])
</code></pre>

username: uhmbg
cooked: <p>This code is correct, assuming that inputs is all the instances of the training, i.e. there is only 1 batch in trainloader?</p>
<pre><code>for epoch in range(n_epochs):

    for i, [inputs, labels] in enumerate(trainloader):
        inputs = inputs.to(device)
        labels = labels.to(device)

        # Forward + backward + optimize
        outputs = net(inputs.float())

        if epoch == 1:
            optimizer = torch.optim.Adam(net.parameters())

        optimizer.zero_grad()
        
        loss = criterion(outputs, labels.float())
        loss.backward()
        optimizer.step()</code></pre>

username: ptrblck
cooked: <p>I would create the optimizer before entering the training loop.<br>
Besides that it looks alright.</p>
<p>I’m not sure which criterion you are using, but I assume your criterion expects float targets.</p>

username: uhmbg
cooked: <p>I can’t put it in before because the net is built during the forward, and I get this error ValueError: optimizer got an empty parameter list.</p>

username: ptrblck
cooked: <p>This error is raised, if you didn’t register the parameters properly.<br>
Could you post the model code, please?</p>