akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the meaning of max of a variable?

username: forcefulowl
cooked: <p>For e.g. we create a net named ‘net’. y = net(x). what is meaning of y.max(1)? (especially a,b = y.max(1), what does it mean?)I saw the range of max is (-2,1). what does that mean.</p>

username: Happy_NewYear
cooked: <p><a href="https://www.tutorialspoint.com/python/list_max.htm" class="onebox" target="_blank" rel="nofollow noopener">https://www.tutorialspoint.com/python/list_max.htm</a></p>
<p>I think you can relate with this.</p>

username: jpeg729
cooked: <p>Let’s check the docs <a href="http://pytorch.org/docs/0.3.1/tensors.html#torch.Tensor.max">http://pytorch.org/docs/0.3.1/tensors.html#torch.Tensor.max</a></p>
<p><code>y.max(1)</code> takes the max over dimension 1 and returns two values. an example would help.</p>
<p>Supposing we have a tensor <code>y</code> of shape (3, 4) containing</p>
<pre><code class="lang-auto"> 0.6857  0.1098  0.4687  0.7822
 0.4170  0.2476  0.1339  0.5563
 0.9425  0.8433  0.1335  0.3169
</code></pre>
<p><code>y.max(1)</code> returns two tensors…</p>
<pre><code class="lang-auto"># 1. the max value in each row of y
 0.7822
 0.5563
 0.9425
# 2. the column index at which the max value is found.
 3
 3
 0
</code></pre>

username: Ambareesh_S_J
cooked: <aside class="quote no-group" data-username="forcefulowl" data-post="1" data-topic="14745">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/forcefulowl/48/3970_2.png" class="avatar"> forcefulowl:</div>
<blockquote>
<p>y.max(1)</p>
</blockquote>
</aside>
<p>Why does  y.max(1), give the max element in each row? (dimensions in Numpy are 0 indexed-  so 0 for row, 1 for coloumn) Why is it different in PyTorch?</p>

username: ptrblck
cooked: <p><code>max(1)</code> will return the maximal value (and index in PyTorch) in this particular dimension.<br>
Both, numpy and PyTorch return the same values (PyTorch additionally with the indices):</p>
<pre><code class="lang-python">x = np.random.randn(10, 10)
print(x.max(1))
print(torch.from_numpy(x).max(1)[0])
</code></pre>

username: writeforfun
cooked: <p>Thank you for the examples!</p>
Increasing efficiency of implementation of a math equation

username: blade
cooked: <p>Given input symmetric matrix A with zero diagonals, I want to compute matrix C as below. Can I increase the efficiency by removing loops?</p>
<p>Also, would the following  break backpropagation?</p>
<pre><code>W = torch.tensor([[0,1,0,0,0,0,0,0,0],
              [1,0,1,0,0,1,0,0,0],
              [0,1,0,3,0,0,0,0,0],
              [0,0,3,0,1,0,0,0,0],
              [0,0,0,1,0,1,1,0,0],
              [0,1,0,0,1,0,0,0,0],
              [0,0,0,0,1,0,0,1,0],
              [0,0,0,0,0,0,1,0,1],
              [0,0,0,0,0,0,0,1,0]])

n = len(W)
C = torch.empty(n, n)
I = torch.eye(n)
for i in range(n):
    for j in range(n):
        B = W.clone()
        B[i, j] = 0
        B[j, i] = 0

        tmp = torch.inverse(n * I - B)

        C[i, j] = tmp[i, j]</code></pre>
"batch_norm" not implemented for 'Half

username: ekofman
cooked: <p>My neural net was working perfectly, and now I’m not sure why I am getting this error upon this line:</p>
<p><code>y_pred = model(train_categorical, train_numerical)</code></p>
<p>Any idea how this might be fixed/what it is indicating is wrong? Thansk!<br>
Here is the offending stack trace:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-30-9aa7542c12d0&gt; in &lt;module&gt;
     17 for i in range(epochs_per_training):
     18     i += 1
---&gt; 19     y_pred = model(train_categorical, train_numerical)
     20     single_loss = loss_function(y_pred, train_outputs)
     21 

~/new_anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

&lt;ipython-input-25-fd94d1dedadd&gt; in forward(self, x_categorical, x_numerical)
     38         x = self.embedding_dropout(x)
     39 
---&gt; 40         x_numerical = self.batch_norm_num(x_numerical)
     41         x = torch.cat([x, x_numerical], 1)
     42         x = self.layers(x)

~/new_anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--&gt; 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/new_anaconda3/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py in forward(self, input)
    134             self.running_mean if not self.training or self.track_running_stats else None,
    135             self.running_var if not self.training or self.track_running_stats else None,
--&gt; 136             self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
    137 
    138 

~/new_anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
   2014     return torch.batch_norm(
   2015         input, weight, bias, running_mean, running_var,
-&gt; 2016         training, momentum, eps, torch.backends.cudnn.enabled
   2017     )
   2018 

RuntimeError: "batch_norm" not implemented for 'Half'
</code></pre>

username: ekofman
cooked: <p>Ok basically you can’t use float16. It has to just be float. good to know.</p>
Ignore_label for nn.BCEWithLogitsLoss

username: Neabfi
cooked: <p>Hello,<br>
I’m doing multi-label semantic segmentation using nn.BCEWithLogitsLoss. How can I ignore some labels during training?</p>
<p>Thank you!</p>
<p>PS: <a href="https://discuss.pytorch.org/t/question-about-bce-losses-interface-and-features/50969/4" class="inline-onebox">Question about BCE* losses interface and features</a> doesn’t reply to this question</p>

username: Neabfi
cooked: <p>I found one solution: <a href="https://discuss.pytorch.org/t/ignore-padding-area-in-loss-computation/95804" class="inline-onebox">Ignore padding area in loss computation</a></p>
Forecast & replenishment planning for food supply chain optimization

username: GBeykirch
cooked: <p>I am an industrial engineer, deeling with process optimizations in parts of the food supply chain. Searching mainly for solutions / examples for the usage of ML for forecast and replenishment planning for small business, e.g. coffee-shops, cantines, … .<br>
Must not be ready, but basic structure as starting point for further students projects.<br>
Could somebody help?<br>
Thanks in advance.<br>
Gerald</p>
Load_state_dict

username: Ohm
cooked: <p>Hi,<br>
The net parameters(weights and network models) should be stored and then load. But When I try to load it I got an Error. Could I change the following code in a way it works?</p>
<pre><code class="lang-auto">net = models.vgg16_bn().to(device)  
Fix_net_par1 = net.named_parameters()
Fix_net_par = copy.deepcopy(dict(Fix_net_par1))
net.load_state_dict(Fix_net_par)       &lt;=== The line that creates the Error
</code></pre>
<p>RuntimeError: Error(s) in loading state_dict for VGG:<br>
Missing key(s) in state_dict: “features.1.running_mean”, “features.1.running_var”, “features.4.running_mean”, “features.4.running_var”, “features.8.running_mean”, “features.8.running_var”, “features.11.running_mean”, “features.11.running_var”, “features.15.running_mean”, “features.15.running_var”, “features.18.running_mean”, “features.18.running_var”, “features.21.running_mean”, “features.21.running_var”, “features.25.running_mean”, “features.25.running_var”, “features.28.running_mean”, “features.28.running_var”, “features.31.running_mean”, “features.31.running_var”, “features.35.running_mean”, “features.35.running_var”, “features.38.running_mean”, “features.38.running_var”, “features.41.running_mean”, “features.41.running_var”.</p>
<p>Could I change the above code in a way it works?</p>

username: catt_ale
cooked: <p>Hello,</p>
<p>I never stored and reloaded parameter runtime.</p>
<p>Anyway why you don’t just save your model parameters on your HD and then reload as indicated in the PyTorch tutorials <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">saving_loading_models_tutorial</a></p>
<p>Alessandro.</p>

username: Ohm
cooked: <p>because it is really time-consuming.</p>

username: catt_ale
cooked: <p>And did you try this?</p>
<pre><code class="lang-python">model = TheModelClass(*args, **kwargs)
model.train()
actual_dict = model.state_dict()
model = TheModelClass(*args, **kwargs)
model.load_state_dict(actual_dict)
</code></pre>
<p>Basically, canonical way without writing on the disk but keeping the data dynamically allocated</p>
Pytorch-mighty package monitoring the training progress

username: dizcza
cooked: <p>Dear PyTorch users,</p>
<p>I’d like to present a package of mine that automates monitoring the training process: <a href="https://github.com/dizcza/pytorch-mighty" rel="noopener nofollow ugc">https://github.com/dizcza/pytorch-mighty</a></p>
<p>Initially developed as a tool to help other projects of mine, I thought it could be valuable to some of you. The idea is similar to Pytorch Horovod and many other packages that automate the training process, therefore, I want to focus on the features that make the package particularly interesting:</p>
<ol>
<li>
<code>trainer.restore(checkpoint_path=None)</code>: seamless restoration of the training progress AND the monitor plots. Imagine that you saved the progress yesterday with 10+ infographic plots, wake up today and want to continue adding metric points (loss, accuracy, etc.) <strong>on the same figures</strong> as nothing happened.</li>
<li>Several Mutual Information estimators: that’s where I put the most effort into. If you’re interested in information theory, how the information flows between layers across epochs, you will find a separate repository (linked there) with Mutual Information estimators benchmarks.</li>
<li>More than just accuracy and loss curves. One of the reasons why I developed such an advanced monitoring system is that I was not satisfied with the basics plots of accuracy and loss VS epoch, and I monitor several other metrics.</li>
</ol>
<p>I work with vision only, so I don’t know how much this package is applicable in other domains. While it’s not necessary to use pytorch-mighty as it is, you may find a number of interesting monitoring plots that you can copy-paste into your projects.</p>
<p>Best,<br>
Danylo</p>
nn.Embedding - how to return the negative embeddings?

username: Peter_Featherstone
cooked: <p>I can use nn.Embedding trivially to get the positive embeddings using a tensor of indices like so</p>
<pre><code class="lang-auto">n = nn.Embedding(256,128)
indices = torch.randint(0, 256, (10,))
positive_embeddings = n(indices)
print(positive_embeddings.shape) #prints torch.Size([10, 128])
</code></pre>
<p>How do i now get all the other embeddings?<br>
I would like to do something like</p>
<pre><code class="lang-auto">negative_embeddings = some_function(n,indices)
print(negative_embeddings.shape) #prints torch.Size([10, 255, 128])
</code></pre>
<p>My motivation is to use nn.CosineEmbeddingLoss with the predicted embeddings of some CNN and both the positive and negative embeddings of some target indices.</p>
Google colab GPU support

username: OuisYasser
cooked: <p>Hello everyone I wish someone can help me !<br>
Because I’m a beginner I can’t have a server, so I’m using Google colab but unfortunately my large set of images (1.5gb) is on drive so the training/ evaluation is slow, I know that I have to compress it and then decompress it on colab but it take a lot of time.<br>
Is there anyone who knows a solution or can provide another platform which provide GPU support for free<br>
Thank you</p>
How to use batch normalization and make learning the bias False but the rest of learning stuff be True?

username: seyeeet
cooked: <p>I want to use batchnorm but I want to make the bias not learnable , how can I specify that? the affine parameters will make both of gamma and beta true or false, but I want to make the beta not learnable</p>

username: klory
cooked: <p>maybe this is sth you want to do?</p>
<pre><code class="lang-auto">import torch
from torch import nn

m = nn.BatchNorm2d(4)
x = torch.randn(2,4,16,16)

m._parameters['bias'].requires_grad_(False)
print(m._parameters['weight'].grad)
print(m._parameters['bias'].grad)
m.zero_grad()
y = m(x).mean()
y.backward()
print(m._parameters['weight'].grad)
print(m._parameters['bias'].grad)
</code></pre>
<p>output</p>
<pre><code class="lang-auto">None
None
tensor([ 1.4578e-10, -2.0856e-09, -5.6265e-11, -1.4465e-09])
None
</code></pre>
Model performance changes when having unused components in model init method

username: Vandana_Rajan
cooked: <p>Hi,</p>
<p>I am observing the following weird behaviour with my model.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, encoder, classifier):
        super(Model, self).__init__()
        self.gru = torch.nn.GRU(input_size=100, hidden_size=300, dropout=0.5, num_layers=3, bidirectional=True)
        self.dp1 = torch.nn.Dropout(0.7)
        self.dense1 = TimeDistributed(torch.nn.Linear(600, 100))
        self.encoder = encoder
        self.classifier = classifier
    def forward(self, x, m):
        #import pdb
        #pdb.set_trace()
        #x, h = self.gru(x)
        #x = self.dp1(x)
        #x = self.dense1(x)
        x = self.encoder(x, m)
        x = self.classifier(x)
        return x
</code></pre>
<p>When I compile and run this model, I get a binary classification accuracy of 60.77%.</p>
<p>But when I comment the components in both forward and init, it decreases slightly to 59.6%.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, encoder, classifier):
        super(Model, self).__init__()
        #self.gru = torch.nn.GRU(input_size=100, hidden_size=300, dropout=0.5, num_layers=3, bidirectional=True)
        #self.dp1 = torch.nn.Dropout(0.7)
        #self.dense1 = TimeDistributed(torch.nn.Linear(600, 100))
        self.encoder = encoder
        self.classifier = classifier
    def forward(self, x, m):
        #import pdb
        #pdb.set_trace()
        #x, h = self.gru(x)
        #x = self.dp1(x)
        #x = self.dense1(x)
        x = self.encoder(x, m)
        x = self.classifier(x)
        return x
</code></pre>
<p>So, why does the performance differ in the 2 cases?</p>
<p>Note: I have kept the random seed constant and enabled the CUDA deterministic behaviour. So, each run with no model changes provides the same accuracy.</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>Each layer creation will randomly initialize the parameters (if available) and will thus call into the pseudorandom number generator. If your training is sensitive to different seeds, these additional layer creations can change the training results significantly and you should see the same effect by removing the layer creation and just changing the seed at the beginning of your script.</p>
Identity weights for 2d convolution

username: dgriffiths3
cooked: <p>I would like to initialise a multi-channel 2d convolution layer such that it simply replicates the input (identity). For a single channel image I know the identity kernel is:</p>
<pre><code class="lang-auto">[0, 0, 0
 0, 1, 0
 0, 0, 0]
</code></pre>
<p>But how can I do this for a 3 channel image?</p>
<p>Input is <code>[b, 3, h, w]</code> and output is also <code>[b, 3, h, w]</code>, so my weight size would be <code>[3, 3, 3, 3]</code>.</p>
<p>My best guess which doesn’t work (x should equal y):</p>
<pre><code class="lang-auto">x = torch.rand(1, 3, 3, 3)
w = torch.tensor([
	[0., 0., 0.,],
	[0., 1., 0.,],
	[0., 0., 0.,],
]).view(1, 1, 3, 3).repeat(3, 3, 1, 1)
y = nn.functional.conv2d(x, w, bias=None, stride=1, padding=1, dilation=1)
print(torch.allclose(x, y)) # prints False
</code></pre>

username: googlebot
cooked: <p>nn.init.dirac_ would do this</p>
Why does transforms.ToTensor() output range [-1, 1]

username: spadel
cooked: <p>According to the documentation, <code>transforms.ToTensor()</code> should transform in array in range 0-255 to [0, 1]. But I just tested the output of my DataLoader, which results in the following:</p>
<pre><code class="lang-auto">class ImageDataset(Dataset):
    def __init__(self, images):
        super(ImageDataset, self).__init__()
        self.images = images
        self.transforms = transforms.Compose([transforms.ToTensor()])

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        # Select Image
        image = self.images[index]
        image = self.transforms(image)

        return image
</code></pre>
<pre><code class="lang-auto"># Load Datasets and DataLoader
data = load_images_from_folder()
train_data = np.array(data[:-1000])
train_dataset = ImageDataset(train_data)
test_data = np.array(data[-1000:])
test_dataset = ImageDataset(test_data)
train_loader = DataLoader(train_dataset, batch_size=bn, shuffle=True, num_workers=4)
test_loader = DataLoader(test_dataset, batch_size=bn, num_workers=4)

# Make Training
for epoch in range(epochs+1):
# Train on Train Set
model.train()
model.mode = 'train'
for step, original in enumerate(train_loader):
    original = original.to(device)
    if step == 0 and epoch == 0:
        print(f'input information: mean: {torch.mean(original[0])}, max: {torch.max(original[0])}, min: {torch.min(original[0])}')
</code></pre>
<blockquote>
<p>input information: mean: 0.32807812094688416, max: 1.0, min: -1.0</p>
</blockquote>
<p>What’s the reason for that?</p>

username: spadel
cooked: <p>Okay got it fixed - there was a problem with my upload deployment configurations and the code didn’t upload. It works now</p>
Freeze gradients for a subset of a parameter tensor

username: vr308
cooked: <pre><code class="lang-auto"> mu_test = torch.nn.Parameter(torch.zeros(len(Y_test), 2))
 mu_full = torch.cat([mu_train.detach().clone(), mu_test])
</code></pre>
<p>I only want to compute gradients for mu_test part of mu_full -&gt; is that possible to do?</p>
<p><code>torch.cat()</code> returns a tensor and if I wrap the whole thing as a parameter then I guess it has <code>requires_grad=True</code> for the whole tensor.</p>

username: ptrblck
cooked: <p>Assuming <code>mu_train</code> and <code>mu_test</code> are both parameters, you could perform the concatenation during the <code>forward</code> pass and it should work as expected:</p>
<pre><code class="lang-python"># setup
mu_test = torch.nn.Parameter(torch.zeros(2, 2))
mu_train = nn.Parameter(torch.randn(2, 2))

# in your forward pass
mu_full = torch.cat([mu_train.detach().clone(), mu_test])
out = mu_full * 2
print(out)
&gt; tensor([[-2.2595, -1.4177],
          [ 1.3986, -2.4303],
          [ 0.0000,  0.0000],
          [ 0.0000,  0.0000]], grad_fn=&lt;MulBackward0&gt;)

# backward
out.mean().backward()

print(mu_test.grad)
&gt; tensor([[0.2500, 0.2500],
          [0.2500, 0.2500]])
print(mu_train.grad)
&gt; None
</code></pre>
RuntimeError: stack expects each tensor to be equal size, but got [3, 530, 500] at entry 0 and [3, 500, 530] at entry 3

username: rui_zhou
cooked: <p>I get this error:<br>
RuntimeError: stack expects each tensor to be equal size, but got [3, 530, 500] at entry 0 and [3, 500, 530] at entry 3<br>
image(32bit) and label (4bit) , the same size(500*530), is there anything wrong with my transpose process?</p>
<p>class Resize(object):<br>
“”“Resize image and/or masks.”""<br>
def <strong>init</strong>(self, imageresize, maskresize):<br>
self.imageresize = imageresize<br>
self.maskresize = maskresize</p>
<p>def <strong>call</strong>(self, sample):<br>
image, mask = sample[‘image’], sample[‘mask’]<br>
if len(image.shape) == 3:<br>
image = image.transpose(1, 2, 0)<br>
if len(mask.shape) == 3:<br>
mask = mask.transpose(1, 2, 0)<br>
mask = cv2.resize(mask, self.maskresize, cv2.INTER_AREA)<br>
image = cv2.resize(image, self.imageresize, cv2.INTER_AREA)<br>
if len(image.shape) == 3:<br>
image = image.transpose(2, 0, 1)<br>
if len(mask.shape) == 3:<br>
mask = mask.transpose(2, 0, 1)</p>
<pre><code>return {'image': image,
        'mask': mask}</code></pre>

username: ptrblck
cooked: <p>If the image as well as the mask have the same initial shape and you are applying the same <code>transpose</code> operations on these numpy arrays, the error might be raised by trying to stack different samples.<br>
Could you check the shape of both arrays and make sure the height and width is equal for all samples?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Disabling mixed precsion in my own layers

username: tree33
cooked: <p>Hi, after reading the docs about mixed precsion, <a href="https://pytorch.org/docs/stable/notes/amp_examples.html" rel="nofollow noopener">amp_example</a><br>
I’m still confused with several problems.</p>
<p>Let’s say if I have two networks, one is the standard resnet50 and another is a sparse conv layer.<br>
input images are first passed through resnet50 and then sparse convs.</p>
<p>If I only want to use half for resnet and keep float32 for the sparse conv layer (so I don’t have to modify the code)<br>
I only need to warp the model within the autocast function and disable it before the sparse conv layers?<br>
like,</p>
<pre><code class="lang-auto">with autocast():
      out = resnet50(x)
      with autocast(enabled=False):
             out = sparseconv(out.float())
</code></pre>
<p>right?</p>
<p>And from my knowledge, gradients are scaled during mixed precision,<br>
If I have to write my own backward function for sparse conv layers (warped in autocast(disabled)),<br>
Do I still need to consider the scale?</p>

username: ptrblck
cooked: <p>Your <code>autocast</code> example looks correct.</p>
<aside class="quote no-group" data-username="tree33" data-post="1" data-topic="97190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tree33/48/10609_2.png" class="avatar"> tree33:</div>
<blockquote>
<p>And from my knowledge, gradients are scaled during mixed precision</p>
</blockquote>
</aside>
<p>Yes, if you are using a <code>GradScaler</code> object.</p>
<aside class="quote no-group" data-username="tree33" data-post="1" data-topic="97190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tree33/48/10609_2.png" class="avatar"> tree33:</div>
<blockquote>
<p>If I have to write my own backward function for sparse conv layers (warped in autocast(disabled)),<br>
Do I still need to consider the scale?</p>
</blockquote>
</aside>
<p>No, gradient scaling and auto casting are working together but are independent from each other.<br>
You don’t need to explicitly implement gradient scaling for the custom layer.</p>

username: ibro45
cooked: <p>Imagine we have an <code>nn.Sequential</code> model with a specific layer type that should not be autocasted. Is there a way to “wrap” that layer so that it is ignored by the autocast?</p>

username: ptrblck
cooked: <p>If you are using custom modules, you could use the <code>@autocast</code> decorator on the <code>forward</code> method and disable it. On the other hand, if you are using built-in modules, you could wrap them in a custom class and disable autocast in the same way.<br>
Alternatively you could also split the <code>nn.Sequential</code> container and just decorate the parts with autocast, which should use it.</p>
Data loader for 3D convolution on video frames

username: Soum
cooked: <p>I am working with a dataset of videos, I have done a preprocessing to split each video into a set of frames. I am implementing a network that uses a 3dconvolution where I have to pass chunks of videos to be processed by the network.<br>
I know that a 3dConv layer takes a tensor of 5D (N, C, D, H,W). How do I modify the data  loader so that I get the data in the appropriate shape?</p>

username: aradhyamathur
cooked: <p>Say you have a set of D frames in every video of size  h x w, so for every batch you first fetch the frames ,assuming that each frame is RGB image, each frame is of size h x w x 3, now stack the frames to get the shape D x h x w x 3, now permute the along the axis namely T (D x h x w x 3) -&gt; T.permute(3,0,1,2) , the batch size will be set by the dataloader itself.</p>

username: Soum
cooked: <p>I can do that but I want to fetch batches which already are composed of chuncks of frames. Meaning that each example in the batch has a depth = a number of frames , Height and width</p>

username: aradhyamathur
cooked: <p>You can do this by overriding the <code>__getitem__</code> in a custom implementation of dataset, where you load the frames and perform the above reshaping. You can refer <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">here</a> for the implementation of custom dataset and dataloader.</p>

username: Varun_Tirupathi
cooked: <p>Hi Soum,</p>
<p>i am very happy to see your post that you have worked on the video processing tasks fortunately I’m also working on the video processing task using pytorch. Could you please tell me how you wrote the data loader for 3D input videos?</p>
Use pytorch CUDA binraries in a machine without GPU

username: Mohammadreza_Nazari
cooked: <p>I have build pytorch in a cuda-enabled machine and I want to use the same binaries in a machine without cuda, but I get a build error:</p>
<p>CMake Error at /libtorch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):<br>
Your installed Caffe2 version uses CUDA but I cannot find the CUDA<br>
libraries.  Please set the proper CUDA prefixes and / or install CUDA.<br>
Call Stack (most recent call first):<br>
/bigdisk/lax/rl/pytorch_prebuilt/cuda/libtorch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)<br>
CMakeLists.txt:31 (find_package)</p>
<p>Is it possible to use cuda binaries for cpu-only machines? One approach seems to be linking manually without using<br>
find_package(Torch REQUIRED)</p>
<p>Does anyone have any suggestions?</p>
Sample from multiple datasets

username: Rabeeh_Karimi
cooked: <p>Hi<br>
I have multiple datasets of different length and I want to sample them with different rates, is there any function like</p>
<pre><code class="lang-auto">tf.data.experimental.sample_from_datasets
</code></pre>
<p>in pytorch, I appreciate any implementation achieving this in pytorch, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>  please help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: klory
cooked: <p>what prevent them to have the same length? Why not preorganize them to have the same length</p>

username: ptrblck
cooked: <p>If I understand your use case correctly, you have e.g.:</p>
<pre><code class="lang-python">dataset1 = ...
dataset2 = ...

print(len(dataset1)) 
&gt; 10
print(len(dataset2))
&gt; 100
</code></pre>
<p>Now you would like to create a “parent dataset” containing both and use some weighted sampling to e.g. oversample <code>dataset1</code> and create balanced batches from both datasets?</p>
Speeding up the training

username: Rojin
cooked: <p>Hello,<br>
I am trying an autoencoder with specific connections.<br>
My input matrix is 24,000 x 65,000<br>
my middle layer has 4000 nodes, but the first to the second layer are not fully connected. The connection is specified based on some prior knowledge.</p>
<pre><code>    self.layer1.weight.data = self.layer1.weight.data.mul(self.mask)
    z = self.relu(self.dropout(self.NEL1(self.EL1(x))))
</code></pre>
<p>I am already using a GPU for training, but still, the training is very slow. I was wondering if anyone has any suggestions on how I can speed it up?</p>
<p>Thanks!</p>
How to access gradients w.r.t. activations before and after a layer/module

username: CDhere
cooked: <p>Hi all,</p>
<p>I know that after a backward pass we can access the gradients w.r.t the parameters (i.e., weights, bias, etc.) of a layer, but is there a way to retrieve the gradients w.r.t the activation coming in and out of the current layer?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/3381e098f2ae72b0e8c88473cf1b6a97b1fa7bcc.png" data-download-href="https://discuss.pytorch.org/uploads/default/3381e098f2ae72b0e8c88473cf1b6a97b1fa7bcc" title="Screen Shot 2020-11-13 at 2.18.27 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/3381e098f2ae72b0e8c88473cf1b6a97b1fa7bcc_2_690x258.png" alt="Screen Shot 2020-11-13 at 2.18.27 PM" data-base62-sha1="7lEBW8fhyax2qpORucPY3S9GmSw" width="690" height="258" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/3381e098f2ae72b0e8c88473cf1b6a97b1fa7bcc_2_690x258.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/3381e098f2ae72b0e8c88473cf1b6a97b1fa7bcc_2_1035x387.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/3/3381e098f2ae72b0e8c88473cf1b6a97b1fa7bcc.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-11-13 at 2.18.27 PM</span><span class="informations">1098×412 18.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Namely, dA_i-1 and dA_i in this graph.</p>
Multiple Dataset objects of same dataset

username: Prathamesh_Deshpande
cooked: <p>I have a Timeseries dataset in a numpy array with shape (number of series, number of entries). I want to create multiple <code>Dataset</code> objects of this dataset for different custom batching methods. Does creating multiple <code>Dataset</code> objects lead to copying of the underlying data in memory or only reference of underlying data is used? Also, how do I test it?</p>
Apply Model Elementwise

username: arnavs
cooked: <p>Hi All,</p>
<p>I’ve spent some time reading approaches to this, but am curious for your advice.</p>
<p>I have a model <code>phi(x)</code> that takes inputs of dimension <code>1</code>, and returns row vectors of dimension <code>L</code>. I’d like to apply it elementwise to a 2D matrix of data, giving me some 3D object.</p>
<p>Essentially, to write <code>new_tensor = map(phi, old_tensor)</code> in a differentiable way.</p>
<p>It looks like the way to do this is with a conv1d layer of some kind, but I can’t get it to work.</p>
<p>Any suggestions would be appreciated.</p>
Mesh R CNN Datasets

username: Kiran_Ikram
cooked: <p>What are some datasets one can use to experiment with, to test out Mesh R CNN ?</p>

username: Shisho_Sama
cooked: <p>You can use <a href="https://github.com/xingyuansun/pix3d" rel="nofollow noopener">https://github.com/xingyuansun/pix3d</a> dataset just like the original paper did.</p>

username: albertotono
cooked: <p>Dear <a class="mention" href="/u/kiran_ikram">@Kiran_Ikram</a>, Mesh R-CNN used both <strong>Pix3d</strong> and <strong>ShapeNet</strong>.</p>
<p>For <strong>shapenet</strong> you can register here <a href="https://www.shapenet.org/" rel="noopener nofollow ugc">https://www.shapenet.org/</a></p>
<p>It is around 1TB when you unzip it,  ( considering all the different subdatasets they have Sem, Core etc) .</p>
<p>You can also see a great list of Datasets here <a href="https://github.com/timzhang642/3D-Machine-Learning" rel="noopener nofollow ugc">https://github.com/timzhang642/3D-Machine-Learning</a></p>
<p>Depends on your task</p>
Calculate the inverse of parse matrix

username: Tony_Lee
cooked: <p>I want to calculate the inverse of a sparse matrix, how can I do this ?</p>
<p>for example</p>
<pre><code class="lang-auto">vers_n=200000
ind_i = torch.Tensor(list(range(vers_n))).long().view(1,-1)
ind_j = torch.Tensor(list(range(vers_n))).long().view(1,-1)
indices = torch.cat((ind_i, ind_j), dim=0)
values = torch.ones(vers_n)
I = torch.sparse.FloatTensor(indices, values, (vers_n, vers_n))
print(I.inverse())
</code></pre>
<p>then will get the error:</p>
<pre><code class="lang-auto">RuntimeError: Could not run 'aten::_inverse_helper' with arguments from the 'SparseCPUTensorId' backend. 'aten::_inverse_helper' is only availabl
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid we don’t have this implemented in pytorch yet.<br>
You will have to make a dense Tensor before being able to use the <code>inverse()</code> function.<br>
You can open a new issue on github to ask for this feature though if it would be useful for your research!</p>

username: Tony_Lee
cooked: <p>Okay ! Thanks, I will turn to the github for a new issue</p>
Custom DataLoader for Videos

username: Naman-ntc
cooked: <p>I have a video dataset, it consists of 850 videos and per video a lot of frames (not necessarily same number in all frames).<br>
As I can’t fit my entire video in GPU at once I have to sample frames from the video (maybe consecutive maybe random)</p>
<p>When I am building <code>torch.utils.data.Dataset</code> object then _ _len _ _ of the dataset should be 850 only (number of videos). Otherwise I could make it something based on number of frames as well. For example if I use 32 consecutive frames I can get length number of 32 frame videos from consecutive frames in all videos.</p>
<p>Then I will need to make the _ _<em>get</em> _ _ function based on that!</p>

username: fmassa
cooked: <p>Hey,</p>
<p>It might be good to have a look at <a href="https://github.com/NVIDIA/nvvl">https://github.com/NVIDIA/nvvl</a><br>
They have a PyTorch dataloader that loads videos on the GPU, and might be helpful for you.</p>

username: Naman-ntc
cooked: <p>Hi thanks <a class="mention" href="/u/fmassa">@fmassa</a>  , actually I already knew about the dataloader, but actually I am under hard deadline and dont have time to install and learn new dataLoader (would probably sometime though).<br>
Also using the former approach (len as number of videos) and closing this on that behalf<br>
Thanks!</p>

username: isalirezag
cooked: <p>Also, i tried their dataloader and after struggling with their code for 3 days I gave up because it was not easy to figure.</p>
<p>Does pytorch have any plane to provide something for it, or at least an example should be useful</p>
<p><a class="mention" href="/u/naman-ntc">@Naman-ntc</a> did you finally figure it out?</p>

username: Naman-ntc
cooked: <p>Did you look at there example code for <a href="https://github.com/NVIDIA/nvvl/tree/master/examples/pytorch_superres" rel="nofollow noopener">superresolution in pytorch</a>, it should help you somewhat. I looked at it in between but didn’t proceed with it.<br>
Agree that it is somewhat complicated, but you might open an issue in case you don’t understand something!<br>
Would be helpful to have more examples I guess!</p>

username: isalirezag
cooked: <p>yeah, i could not make that work on my case,<br>
and the sad side is that i dont know how to use docker, and dont wanna get into more confusion.<br>
Thanks anyways</p>

username: MohamedAliRashad
cooked: <p>I am in your shoes now, is there any hope pytorch provides a Dataloader similar to nvvl but doesn’t require all this hussle to use it.</p>

username: raivokoot
cooked: <p>Late, but this repository right here offers exactly what this thread is asking for: off-the-shelf PyTorch Video Dataset Loading <a href="https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch" rel="noopener nofollow ugc">https://github.com/RaivoKoot/Video-Dataset-Loading-Pytorch</a><br>
It allows you to choose how many frames from a video you want to load and loads them evenly spaced from start to end of a video. Its native pytorch and doesn’t take the hassle that other methods do.</p>
How to slove the problem: "'Tensor' object has no attribute 'btrifact'"

username: yiqiang_wu
cooked: <p>Recent, I am studying the LU factorization, however, I met a issue as following:<br>
w = torch.randn(3,3)<br>
w = w.unsqueeze(0).btrifact()<br>
AttributeError: ‘Tensor’ object has no attribute ‘btrifact’<br>
how to solve above issue?</p>

username: klory
cooked: <p><code>btrifact</code> is renamed to <code>lu</code></p>
Differentiate through SGD Step

username: Andrea_Rosasco
cooked: <p>I need the gradient to flow back through a previous update step but the SGD step method is decorated with the no_grad property and executes the operation in-place.</p>
<p>I tried to do that manually but if I use tensor.add_ to modify the model parameter I get <code>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation</code></p>
<p>How can I do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you do <code>new_param = old_param + lr * grad</code> then it will be properly differentiable.</p>
<p>Note that if you try ot work with <code>torch.nn</code> though, the <code>nn.Parameter</code> can only be leafs (and so cannot have history). So you won’t be able to use them here.</p>

username: Andrea_Rosasco
cooked: <p>Hi, thanks for replying <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"><br>
Ok, it’s clear but let’s say I’ve defined a ConvNet (because that’s what I did) using some <code>torch.nn</code> classes (<code>Conv2d</code>, etc.), how can I do that?</p>
<p>I figure I can modify all the parameters just by iterating <code>model.parameters()</code> and doing the upgrade on each of them, but then how do I “put” them inside a functioning model?</p>
<p>I feel like there should be an easy solution because everyone who’s doing meta-learning would face the same problem I’m facing right now (I may be wrong, I don’t actually know much about meta-learning but I know optimization-based algorithm often involved differentiating through an inner training).</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately there isn’t as the  <code>nn.Module</code> is by design, not functional.<br>
I woudl advise using a library like <a href="https://github.com/facebookresearch/higher">https://github.com/facebookresearch/higher</a> that handles all of that for you (and provide differentiable version of the pytorch optimizers as well).</p>

username: Andrea_Rosasco
cooked: <p>Thank you, I’ll check that out. I guess that’s what I actually need.<br>
Just some other questions out of curiosity:</p>
<ol>
<li>What does functional mean in this context?</li>
<li>Why isn’t nn.Module functional?</li>
<li>Will this problem be addressed in a future release?</li>
</ol>

username: albanD
cooked: <blockquote>
<ul>
<li>What does functional mean in this context?</li>
</ul>
</blockquote>
<p>It means that you don’t have side effects. In particular, you take the parameters as an input to the function instead of automatically capturing them from the state of the nn.Module</p>
<blockquote>
<ul>
<li>Why isn’t nn.Module functional?</li>
</ul>
</blockquote>
<p>Because when you do <code>mod(inp)</code> it uses a lot of Parameters that are not in the function inputs.</p>
<blockquote>
<ul>
<li>Will this problem be addressed in a future release?</li>
</ul>
</blockquote>
<p>This is something we have in mind yes: transform any nn.Module into a function that takes the Parameters as input. But there are quite a few details to clarify there and it is not done yet.</p>
Apex amp.DistributedDataParallel

username: arinaruck
cooked: <p>I want to switch from torch DDP to apex amp DDP to make use of mixed precision. I’m training two DDP models so I’m using two process groups to make sure the gradients are synchronized correctly. Is there a way to pass a process group to amp DPP to get the same performance?</p>

username: ptrblck
cooked: <p>You don’t need to switch to <code>apex/DDP</code> to use automatic mixed-precision.<br>
We recommend to use the PyTorch implementations of DDP and the native mixed-precision implementation via <code>torch.cuda.amp</code>.</p>

username: arinaruck
cooked: <p>Oh, I see. Thanks a lot! Didn’t know about the <code>torch.cuda.amp</code><br>
It works with the amp and I am able to fit a batch_size twice as big on the GPU now.<br>
One question though, I’m actually experiencing a slowdown with the increase of batch_size<br>
Is this expected behaviour? I was hoping to get the same step execution time, but a bigger batch_size so the epoch gets faster.</p>

username: ptrblck
cooked: <p>It depends a bit where the current bottleneck is.<br>
Using amp should reduce the execution time, if TensorCores can be used. However, increasing the batch size would also increase the time again. The net benefit depends on the achieved speedup through the usage of TensorCores vs. the increased workload.<br>
That being said, increasing the batch size also increases the data loading time, since each worker has to load more samples now and your current setup might face a data loading bottleneck.<br>
You could profile the data loading as shown in the <a href="https://github.com/pytorch/examples/blob/a74badde33f924c2ce5391141b86c40483150d5a/imagenet/main.py#L279-L282">ImageNet example</a> and check, if this time decreases during the training, which would mean that all workers can preload the batches in the background while the GPU is busy with the training.</p>

username: arinaruck
cooked: <p>I did the profiling as you advised, and the dataloader is not the issue here (the time is almost the same)<br>
But I noticed the per step slow down still is a speed up in terms of throughput. So now it takes <code>x1.2</code> less time to process the same amount of objects than before. So I guess that’s how it’s supposed to be. Thanks a lot for your help!</p>
Implementation of torch.nn.BCEWithLogitsLoss

username: be7f984b2f2e66ba7969
cooked: <p>Hi,</p>
<p>I tried to implement BCEWithLogitsLoss by myself.</p>
<p>for example,</p>
<pre><code class="lang-auto">def bce_loss(pred, target):
    pred=F.sigmoid(pred)
    loss=torch.mean(-torch.sum(target * torch.log(pred) + (1-target) * torch.log(1-pred)) / target.size(1))
</code></pre>
<p>however, the loss is quite larger than torch.nn.BCEWithLogitsLoss</p>
<p>for example,</p>
<pre><code class="lang-auto">def bce_loss_pytorch(pred, target):
    m=torch.nn.BCEWithLogitsLoss()
    loss=m(pred, target)
</code></pre>
<p>I am not sure what is the main difference between my implementation and torch.nn.BCEWithLogitsLoss.</p>
<p>any idea?</p>

username: RoySadaka
cooked: <p>I think the issue is that you have 2 means</p>
<ul>
<li>The first one is that you have the sum and then divide by the size (but you should use target.numel() instead of target.size(1))</li>
<li>Second is when you use torch.mean</li>
</ul>
<p>So 2 options:<br>
Here is your code with torch.mean</p>
<pre><code class="lang-python">def bce_loss(pred, target):
    pred=F.sigmoid(pred)
    loss=torch.mean(-(target * torch.log(pred) + (1-target) * torch.log(1-pred)))
    return loss
</code></pre>
<p>And here is your code with manual mean (sum and divide)</p>
<pre><code class="lang-python">def bce_loss(pred, target):
    pred=F.sigmoid(pred)
    loss=torch.sum(-(target * torch.log(pred) + (1-target) * torch.log(1-pred))) / target.numel()
    return loss
</code></pre>
<p>That said, they say that using the official torch.nn.BCEWithLogitsLoss() is better, cause although it’s doing sigmoid and then BCE, it does it in such a way that it is numerically stable (i believe the optimization is on the c++ level)</p>
<p>Roy.</p>

username: KFrank
cooked: <p>Hi Roy and 杜明軒!</p>
<aside class="quote no-group" data-username="RoySadaka" data-post="2" data-topic="102621" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roysadaka/48/29938_2.png" class="avatar"> RoySadaka:</div>
<blockquote>
<p>That said, they say that using the official torch.nn.BCEWithLogitsLoss() is better, cause although it’s doing sigmoid and then BCE, it does it in such a way that it is numerically stable (i believe the optimization is on the c++ level)</p>
</blockquote>
</aside>
<p>As a minor note, you can implement your own <code>BCEWithLogitsLoss</code><br>
with the same numerical benefits as pytorch’s by replacing the<br>
separate calls to <code>sigmoid()</code> and <code>log()</code> with a call to <code>logsigmoid()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>
Pytorch Error after installation

username: ziywang50
cooked: <p>Hello,<br>
When I type import torch on python console, this is what going on:</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
Microsoft Visual C++ Redistributable is not installed, this may lead to the DLL load failure.<br>
It can be downloaded at <a href="https://aka.ms/vs/16/release/vc_redist.x64.exe" rel="noopener nofollow ugc">https://aka.ms/vs/16/release/vc_redist.x64.exe</a><br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “C:\Users\ziywa\anaconda3\lib\site-packages\torch_<em>init</em>_.py”, line 128, in <br>
raise err<br>
OSError: [WinError 126] The specified module could not be found.</p>
</blockquote>
</blockquote>
</blockquote>
<p>Installation was successful.</p>
<p>Best,<br>
Ziyue Wang</p>

username: albanD
cooked: <p>Hi,</p>
<p>Have you tried following the instruction from the error and install the missing component?</p>

username: ziywang50
cooked: <p>ok. Even after I installed this successfully, I still cannot run a program with torch this on pycharm (ModuleNotFoundError: No module named ‘torch’). I can do it on python console.</p>

username: albanD
cooked: <p>I guess the issue is that pycharm is not using the same python as the one in your console where you installed pytorch.<br>
You will need to make sure that pycharm use the same python as the one you installed torch in.</p>
About converting torch.max(x,dim=1)[0] in C++

username: Chen0729
cooked: <p>Hi, All</p>
<p>I have inquiry about torch.max(x,dim=1)[0] conversion to C++.<br>
But in torch::max(x)[0], there is only one argument allowed. How do I make sure the conversion is the same and apply it to dim=1 please ?</p>
<p>Any suggestion is welcome ?</p>
<p>Thanks,</p>

username: albanD
cooked: <p>Hi,</p>
<p>There are multiple overloads of max: <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1ac66d13a86d2ee2c974cd7d22de2ec409.html?highlight=max">https://pytorch.org/cppdocs/api/function_namespaceat_1ac66d13a86d2ee2c974cd7d22de2ec409.html?highlight=max</a><br>
In particular you want the one where you can specify the dim.<br>
Note also that <code>[0]</code> won’t work with the tuple&lt;&gt; output and you will need to use <code>std::get&lt;0&gt;(the_tuple)</code> IIRC.</p>

username: Chen0729
cooked: <p>Thanks a lot.</p>
<p>I modified the code to be torch::max(x,1 false) but it is still incorrect<br>
“No viable conversion from returned value of type ‘std::tuple&lt;Tensor, Tensor&gt;’ to function return type ‘torch::Tensor’”</p>

username: albanD
cooked: <p>See the second part of my comment above: you need to unpack the tuple values using <code>std::get&lt;0&gt;()</code>.</p>

username: Chen0729
cooked: <p>Thanks for the guide.</p>
<p>std::get&lt;0&gt;(max(x,1,false)) seems to work.</p>
<p>There is another with torch.max(x1,1)[1].data<br>
In this case, is<br>
std::get&lt;1&gt;(max(x1,1,false)) equivalent to  torch.max(x1,1)[1].data<br>
because std::get&lt;1&gt;(max(x1,1,false)).data() is also valid.</p>
<p>Just to confirm</p>

username: albanD
cooked: <p>Hi,</p>
<p>In general, you should never use <code>.data</code>. So you should change the python version of that code first <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: Chen0729
cooked: <p>May I ask what is the best alternative for .data option ?<br>
Will it cause issues in c++ ?</p>
<p>Thanks</p>

username: albanD
cooked: <p>It will cause issues on the python side as well.<br>
If you want a Tensor with the same content but that does not share autograd history, you should use <code>.detach()</code> (and the same c++ function when porting).</p>