akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Graph exploration of unknown models

nscotto: <p>Hi,</p>
<p>is there a way to explore the graph of an unknown model in a way that we can recreate the model module by module?<br>
I need to develop a framework that can modify unknown models.</p>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>Which graph are you talking about? How do you get that unknown model?</p>

nscotto: <p>Basicaly the user should be able to provide a model found on internet or other, he supply the module class so I can instantiate the model from it, and I should be able to do some modifications on it.</p>
<p>Usually we represent DL models as a graph of nodes of computation, that is what I mean by the graph.<br>
So I would like to explore the graph of computation and recreate an altered equivalent module.</p>

marvosyntactical: <p>Do you mean something like this?</p>
<pre><code class="lang-auto">def dfs(module:nn.Module, modify_module:Function, is_root=True) -&gt; nn.Module:
    
    i = -1
    for i, (child_name, child_module) in enumerate(module.named_children()):
        dfs(child_module, modify_module, is_root=False)
    is_leaf = i == -1
    
    modify_module(module, is_root, is_leaf)
    
    return module
</code></pre>

nscotto: <p>Seems like it yes!<br>
So with this I can iterate through the graph modules and eventually create the same one or altered?</p>

marvosyntactical: <p>Yes, you only need to copy.deepcopy your model before calling dfs if you dont want to modify it inplace. modify_module is your function that does with a given child module what you want, for example you can add more child modules with .add_module, or modify the forward behavior by registering forward pre hooks (executed before forward call) and forward hooks (executed after forward call see <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" rel="noopener nofollow ugc">here</a>.</p>

nscotto: <p>Awesome, thank you!<br>
Should I edit the title to mark it as solved?</p>

marvosyntactical: <p>I think theres a button on replies that says something like “solved” or some button at the bottom of the post you can click</p>
Training Breaks in Pytorch > 1.5.0 throws inplace modification error

shivammehta007: <p>So, I was following this Hidden Markov Model Tutorial. But the limitation it has is that it breaks if PyTorch is &gt; 1.5.0 it throws: <a href="https://colab.research.google.com/drive/1IUe9lfoIiQsL49atSOgxnCmMR_zJazKI#scrollTo=3CMdK1EfE1SJ" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1IUe9lfoIiQsL49atSOgxnCmMR_zJazKI#scrollTo=3CMdK1EfE1SJ</a></p>
<p>While training the forward algorithm</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 1024]], which is output 0 of TransposeBackward0, is at version 23; expected version 22 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>Setting set_detect_anomaly doesn’t give any other output, but the thing is transpose is not used in any funky way where any in-place operation is done? is it?<br>
Plus, it is working without any error, when Pytorch is downgraded to 1.5.0.</p>

albanD: <blockquote>
<p>Setting set_detect_anomaly doesn’t give any other output</p>
</blockquote>
<p>Make sure to use latest pytorch as we recently fix warnings not showing up in colab.<br>
Or run your code in command line to have the corresponding forward code.</p>
<p>The code does quite a lot of inplace and viewing ops.</p>
<blockquote>
<p>Plus, it is working without any error, when Pytorch is downgraded to 1.5.0.</p>
</blockquote>
<p>This kind of check is here to make sure we don’t compute silently wrong gradients. So it is most likely that the old behavior was silently computing wrong gradients and has been fixed in more recent versions.</p>

shivammehta007: <p>Aha! It was absolutely amazing! The way my output changed when I turned on anomaly detection on my own terminal! Thank you so much for the help, it got fixed <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Apparently, the torch.stack was causing the problem, I changed it to</p>
<pre><code class="lang-auto">log_A_expanded = log_a.unsqueeze(2).expand((m, n, p))
log_B_expanded = log_b.unsqueeze(0).expand((m, n, p))
</code></pre>
<p>even though it basically works the same way but is not computing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Though I am a little confused about why it might have failed in</p>
<pre><code class="lang-auto">	log_A_expanded = torch.stack([log_A] * p, dim=2)
	log_B_expanded = torch.stack([log_B] * m, dim=0)
</code></pre>
<p>Do you have any insight?</p>

albanD: <p>The expand version will be much more efficient for sure as it does not actually allocates new memory and does not do any copy.</p>
<p>They do behave differently in subtle ways. But not sure the exact reason here <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
ImageFolder with Subset and WeightedRandomSampler

BoBTB: <p>Hi have a question concerning ImageFolder which i want to split into train and validation dataset and balance it with a WeightedRandomSampler into a dataloader.</p>
<pre><code class="lang-auto">dataset = datasets.ImageFolder(path, transform)
</code></pre>
<p>First I split it with sklearn</p>
<pre><code class="lang-auto">from sklearn.model_selection import train_test_split
image_datasets = {}
train_idx, val_idx = train_test_split(list(range(len(dataset))), stratify=dataset.targets, test_size=0.2)
image_datasets['train'] = split_result['train']
image_datasets['val'] = split_result['val']
</code></pre>
<p>Then i want to sample train dataset with WeightedRandomSampler because it is unbalanced.</p>
<pre><code class="lang-auto"># get the labels for the subset
labels = np.array(image_datasets['train'].dataset.targets)[image_datasets['train'].indices]
# count the label occurrence
num_class_elements = np.bincount(labels)
# length of new dataset
num_epoch_elements = len(image_datasets['train'])
</code></pre>
<p>create the WeightedRandomSampler</p>
<pre><code class="lang-auto">numerator = 1. 
denominator = torch.tensor(num_class_elements,dtype=torch.float)  # can be converted with torch.tensor(..., dtype=torch.float)

# calculate weight per class
class_weights = numerator / denominator
# create vector where index contains class weight
element_weights = [class_weights[class_index] for class_index in labels]
# create sampler
sampler = torch.utils.data.sampler.WeightedRandomSampler(element_weights, num_epoch_elements, replacement=False)
</code></pre>
<p>Is it now save to use the SubSet combined with the sampler for the dataloader?</p>
<pre><code class="lang-auto">train_dl = torch.utils.data.DataLoader(image_datasets['train'], batch_size=18, num_workers=2,
                                               pin_memory=True, drop_last=False, shuffle=False, sampler=sampler)
</code></pre>
<p><strong>Question herer:</strong><br>
My concern is as follows. Maby i can explain it with an example, the original dataset has 100 elements [0:99] and the training dataset has 80, with index [19:99]=80. The sampler will return indices from [0:79] which the SubSet needs to transform to its parents datasets indices [0:99] excluding the [0:19].<br>
So is the Subset handling this or not?<br>
Is the Subset mapping the indices from the sampler to its indices of the parent Dataset?</p>
<blockquote>
<p>Sampler says 0 which corresponds to train_ds[0] which calls parent.dataset[train_ds.indices[0]] <span class="hashtag-raw">#pseudocode</span></p>
</blockquote>
Unnececarry memory usage when calculating dot product of indexed tensors

KarelPeeters: <p>Consider the following code:</p>
<pre><code class="lang-python">import torch


@torch.jit.script
def foo(x, xi, y, yi):
    return torch.sum(x[xi] * y[yi], dim=1)


emb_size = 100
cooc_count = 10_000_000
word_count = 12543

device = "cuda"

xw = torch.randn(word_count, emb_size, device=device)
yw = torch.randn(word_count, emb_size, device=device)

xi = torch.randint(word_count, (cooc_count,), device=device)
yi = torch.randint(word_count, (cooc_count,), device=device)

dots = foo(xw, xi, yw, yi)
</code></pre>
<p>For those curious, this comes up when training word embeddings but that background is not really relevant, I’m more interested in this question in general.</p>
<p>When I run this on my computer the code crashes with a “CUDA out of memory” error, which makes sense because <code>x[xi]</code> and <code>y[yi]</code> expand to very large tensors of size <code>(cooc_count, emb_size)</code> with a bunch of repeated rows in them. This is an unneccesary allocation though, because those tensors are immediatly collapsed into a smaller tensor of size <code>(cooc_count,)</code> by the multiplication and sum which implement the rowwise dot product.</p>
<p>I was hoping the JIT would realize this and optimize <code>foo</code> to do this dot product directly while indexing. Is this a known limitation? Is there a way to rewrite the expression that avoids the intermediate allocation?</p>
Access python defined global variable in c++ autograd.function

vmoens: <p>Hi,</p>
<p>I need to code a Jit compatible custom operation where the backward-pass depends on the value of a global variable defined in my python code.<br>
Is there a way to do that?</p>
<p>In practice, that would correspond to the following piece of code in python</p>
<pre><code class="lang-auto">FOO = True

class MyFunnyFun(Funtion):
    @staticmethod
    def forward(ctx, x):
        # return something

    @staticmethod
    def backward(ctx, grad_output):
        if FOO:
            # return something
        else:
            # return something else
</code></pre>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>jit does not actually support custom Function no?</p>
<p>If you don’t need the jit, this code will work just fine with regular pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torch.no_grad() makes any difference

catt_ale: <p>Hello to everyone.</p>
<p>I am traying my model inference time using:</p>
<pre><code class="lang-python">model.eval()
with torch.no_grad():
       y = model(x)
</code></pre>
<p>VS</p>
<pre><code class="lang-python">model.train()
y = model(x)
</code></pre>
<p>These two operations take exactly the same time. Is it normal or am I making something wrong?</p>
<p>Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>The eval mode will make a difference only if you use special Modules that behave differently in eval mode like dropout or batchnorm. But even in that case, the runtime might not change that much.<br>
The no_grad mode disables the autograd so it will make a significant difference in memory usage but should not change much for runtime (as we work hard to make the autograd light so that it can run at every forward).</p>
Multitask learning in a self-supervised contest

catt_ale: <p>Hello to everyone!</p>
<p>I have a base model which is a cycle GAN, I am using some layer of the generator to reuse learned features to fed a classifier. Train the two networks separately is not a problem. My doubt is related to how to build the cost function in order to optimize my cycleGAN based on how is performing the classifier.</p>
<p>Do you think this pseudo-code could be the right approach?<br>
Both the loss functions optimize the base model w.r.t. the classification result. Which one is better?</p>
<pre><code class="lang-python">basemodel = Mybasemodel()
basemodel.load_state_dict(torch.load('...'))
basemodel.train()

calssifier = Classifier()
calssifier.train()

loss_basemodel_fn = torch.nn...()
loss_classifier_fn = torch.nn...()

opt_basemodel = torch.optim...(basemodel.parameters())

# create an optimizer which optimizes both the networks
param_chain = itertools.chain(calssifier.parameters(), basemodel.parameters())
opt_classifier = torch.optim...(param_chain)

for i, data in enumerate(dataset):
    
    # Load data
    input = data['input']
    label_basemodel = data['label_basemodel']
    label_classifier = data['label_classifier']

    
    # use part of the base model to get a feature map
    feature_map = basemodel.model[first_layer:N_layer](input)
    # get the prediction
    pred = calssifier(feature_map)
    
    output_base_model = basemodel.model(input)
    
    opt_basemodel.zero_grad()
    opt_classifier.zero_grad()

    # BOTH THESE LOSSES WILL OPTIMIZE MY BASENODEL? WHICH ONE IS BETTER?
    loss_classifier = loss_classifier_fn(pred, label_classifier)
    # combine the two loss function so that the base model optimizer
    # take into consideration the error of the calssifier when optimizes
    # the basemodel
    loss_basemodel = loss_basemodel_fn(output_base_model, label_basemodel) + loss_classifier

    loss_basemodel.backward()
    loss_classifier.backward()

    opt_basemodel.step()
    opt_classifier.step()

</code></pre>
Can I re-use the same RNN before updating its weights?

Valerio_Biscione: <p>I have, as an input, 2 sequences of vectors.<br>
Sequence 1 is passed to an RNN. The RNN is initialized with an hidden state of zeros.  I collect the output hidden state, and save it. Let’s call this output H1.<br>
I do the same with Sequence 2. I use <em>the same</em> RNN as before. Again, I initialize it’s hidden state to zeroes, and collect the output hidden: H2.</p>
<p>I then use another network to compare H1 and H2, get my predictions, and thus I can compute the loss and perform the backward + optim step.</p>
<p>Is this design correct ? Can I reuse the same RNN before performing the backward pass? Is it ok even if I reset the hidden state to 0?<br>
Thanks</p>

Valerio_Biscione: <p>is anyone able to help me on this?</p>

googlebot: <p>Yes, it is like any other module in that regard. It is also same as [sub]batching sequences (H1,H2) for rnn module.</p>
Model parameter is not included in nn.parameters() depsite added using nn.Parameter()

yuanqi_chau: <p>Hi, I have faced some problem that despite added new parameters using nn.Parameter() it is not added into the list of nn.parameters().<br>
I have simplified my code as below:</p>
<pre><code class="lang-auto">class CoNet(nn.Module):

    def __init__(self):
        super(CoNet, self).__init__()
        self.std = 0.01
        self.layers = [64, 32, 16, 8]
        self.class_size = 2
        self.initialize_variables()
    
    def initialize_variables(self):
        #weights and biases: apps
        self.weights_apps = {}
        self.biases_apps = {}

        for l in range(len(self.layers) - 1):
            self.weights_apps[l] = nn.Parameter(torch.normal(mean=0, std=self.std, size=(self.layers[l], self.layers[l+1]), requires_grad=True))
            self.biases_apps[l] = nn.Parameter(torch.normal(mean=0, std=self.std, size=(self.layers[l+1],), requires_grad=True))
        self.W_apps = nn.Parameter(torch.normal(mean=0, std=self.std, size=(self.layers[-1], self.class_size), requires_grad=True))
        self.b_apps = nn.Parameter(torch.normal(mean=0, std=self.std, size=(self.class_size,), requires_grad=True))

model = CoNet()
list(model.parameters())
</code></pre>
<pre><code class="lang-auto">[Parameter containing:
 tensor([[-0.0046,  0.0010],
         [-0.0066, -0.0176],
         [ 0.0054,  0.0118],
         [-0.0128, -0.0059],
         [ 0.0155,  0.0053],
         [-0.0106, -0.0087],
         [ 0.0029,  0.0146],
         [-0.0236, -0.0152]], requires_grad=True), Parameter containing:
 tensor([0.0284, 0.0035], requires_grad=True)]
</code></pre>
<p>The output is basically the last two parameters added namely: self.W_apps and self.b_apps.<br>
Is there any reason why only these two are added?</p>
<p>Help please, thanks!</p>

Abhilash_Srivastava: <p>Try using <code>nn.ParameterList</code> instead of the <code>{}</code></p>

yuanqi_chau: <p>yes it works, thank you!</p>
Count object with Mask R-CNN

Valerien: <p>Hi, I try to train myself wtih mask R-CNN  and i see this colab work  :<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201117-085601-RC00_342848076" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=a5dGaIezze3y" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb#scrollTo=a5dGaIezze3y" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>How can i modify this code to count people on images ?</p>
Segfault when iterating over parameters, but not when using range based loop

vdg: <p>How can it be that the first loop works fine, while the second loop gives a segfault. In my understanding, they should be the same.</p>
<pre><code class="lang-auto">        for(auto&amp; param : actor-&gt;parameters()){
            std::cout&lt;&lt;"Param"&lt;&lt;std::endl;
            std::cout&lt;&lt;param.data()&lt;&lt;std::endl;
        }
        for(auto it = actor-&gt;parameters().begin();it!=actor-&gt;parameters().end();++it)
        {   
            std::cout&lt;&lt;"Iterator Param"&lt;&lt;std::endl;
            std::cout&lt;&lt;it-&gt;data()&lt;&lt;std::endl;
        }
</code></pre>
<p>I want to copy parameters between the policy net and the target net, as follows:</p>
<pre><code class="lang-auto">         auto it=actor-&gt;parameters().begin();
         for(auto target_it = actor_target-&gt;parameters().begin();target_it!=actor_target-&gt;parameters().end();++target_it){
             target_it-&gt;data().copy_(it-&gt;data()*tau_+target_it-&gt;data()*(1.0-tau_));
             ++it;
         }
</code></pre>
<p>For which the python code would be:</p>
<pre><code class="lang-auto"># update target networks 
        for target_param, param in zip(self.actor_target.parameters(), self.actor.parameters()):
            target_param.data.copy_(param.data * self.tau + target_param.data * (1.0 - self.tau))
       
</code></pre>

vdg: <p>I fixed it by doing</p>
<pre><code class="lang-auto">        for(int i = 0; i&lt;actor-&gt;parameters().size();++i){
            actor_target-&gt;parameters()[i].data().copy_(actor-&gt;parameters()[i].data()*tau_ + actor_target-&gt;parameters()[i].data()*(1.0-tau_));
        }
        
</code></pre>
<p>Still wondering why I can’t dereference an iterator on parameters() though.</p>
Meaning of a vertical bar in differentiation

kenenbek: <p>I am reading this article <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">A Gentle Introduction to torch.autograd — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7820e8e3e478d33d5ddbe5ec89e0f8f8cb5bf5f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e7820e8e3e478d33d5ddbe5ec89e0f8f8cb5bf5f" title="telegram-cloud-photo-size-2-5309744802939645667-y"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7820e8e3e478d33d5ddbe5ec89e0f8f8cb5bf5f_2_690x182.jpeg" alt="telegram-cloud-photo-size-2-5309744802939645667-y" data-base62-sha1="x20WlZW5JAbRkcNnIkYiAkZ05ZR" width="690" height="182" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7820e8e3e478d33d5ddbe5ec89e0f8f8cb5bf5f_2_690x182.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7820e8e3e478d33d5ddbe5ec89e0f8f8cb5bf5f_2_1035x273.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7820e8e3e478d33d5ddbe5ec89e0f8f8cb5bf5f.jpeg 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">telegram-cloud-photo-size-2-5309744802939645667-y</span><span class="informations">1280×339 31.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could you help what this vertical bar means?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66a2292052de524b3daf9beaf5e381ab7b332d25.jpeg" alt="telegram-cloud-photo-size-2-5309744802939645668-m" data-base62-sha1="eDW825ln4uIPXGapQTj04YpVSBf" width="112" height="76"></p>

Abhilash_Srivastava: <p>This notation is used to denote the value of the partial differentiation of <code>o</code> with <code>xi</code> at the value <code>xi = 1</code>.<br>
So, in the equation <code>do/dxi = 3(xi + 2)/2</code>, replace <code>xi = 1</code>, you’ll get 9/2.</p>
Convolve 1d to have same output size as input

Alicja_K: <p>scipy convolve has mode=‘same’ option which gives you the output with the same size as input, how do I set parameters like stride and padding to achive the same with <code>torch.nn.functional.conv1d(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1) </code>?</p>

ptrblck: <p>For a <code>stride</code> and <code>dilation</code> of <code>1</code>, the padding should be <code>weight.size(2)//2</code> for an odd sized kernel on both sides and <code>[weight.size(2)//2, weight.size(2)//2 -1]</code> for an evenly sizes kernel.</p>

Alicja_K: <p>thank you so much for this answer. However I need bigger stride, is there a formula considering this parameter?</p>

ptrblck: <p>For a more general formula, you could take a look at <a class="mention" href="/u/rwightman">@rwightman</a>’s <a href="https://gist.github.com/rwightman/f2d3849281624be7c0f11c85c87c1598#file-median_pool-py-L24-L42">median pooling implementation</a>.<br>
This would calculate the padding for a 2-dimensional input, so you could remove one spatial dim for your use case.</p>
Is torch.max same with doing maxpooling

coyote: <p>Hi everyone,</p>
<p>Assume, I have representation <code>x = torch.rand(4,8)</code> for my input sentence. (without batch dimension, it is just a single sentence containing 4 words). I want to get 1x8 dimensional tensor as output. I can obtain this with a <code>maxpooling</code> operation. However, since all of my sentences have different number of words, the first dimension of the x tensor will always be different and I have to therefore pad the input matrix. Instead, I wonder can I basically do <code>torch.max(x,1)</code></p>

ptrblck: <p>If you would create the max pooling layer so that the kernel size equals the input size in the temporal or spatial dimension, then yes, you can alternatively use <code>torch.max</code>.</p>
<p>Based on the input shape and your desired output shape of <code>[1, 8]</code>, you could use <code>torch.max(x, 0, keepdim=True)[0]</code>.</p>
<p>Alternatively, have a look at adaptive pooling layers, which yield the same output shape for variable sized inputs.</p>

Swarup_China: <p>how to implement maxpool 2d using gather and unfold and squeeze</p>

ptrblck: <p>You could use <a href="https://github.com/pytorch/pytorch/pull/1523#issue-119774673">this implementation</a>.</p>

Swarup_China: <p>but then , what is the use of squeeze function if we want to use it. in conv2d, we use unsqueeze . how can we use squeeze in max pooling?</p>

ptrblck: <p><code>squeeze</code> just removes dimensions with a size of 1. I’m unsure, if and where you need it and what issue you are seeing with it.</p>

Swarup_China: <p>I got it, Thank you so so so much. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"> once again thank u</p>

Swarup_China: <p>i took max indices and used gather to get the value . my dimensions were [a,b,c,d,e] , so i used squeeze on the last dimension (e)   to remove it and got correct result.</p>

Swarup_China: <p>I have a question on image tweaking to trick or fool a model. There are 2 scenarios, 1 to get a higher accuracy (positive tweaking)  and second to get a lower accuracy (negative tweaking) as compared to the trained model. I was thinking to add the gradient of the image to the original image and call it adversarial image. is the concept correct?<br>
like, fake_img = original_img + grad_original image</p>
<p>please help me to understand</p>

ptrblck: <p>It depends what the gradient in the input image represents, i.e. what the model is supposed to learn and what the loss represents, which is used to calculate the gradient.<br>
If you want to change the input data in order to reduce the loss, your approach might work.<br>
However, since there don’t seem to be any constraints, I guess your input image might look “unnatural” after a few iterations.</p>
Implementing a variation of MaxPool2d

melike: <p>Hi all, I need to implement a variation of MaxPool2d where all maximum values are selected and their indices are returned in case of equality instead of the first one and adapt MaxUnpool2d to it.</p>
<p>Here is an example,</p>
<pre><code class="lang-auto">pool = CustomMaxPool2d(2, stride=2, return_indices=True)
unpool = CustomMaxUnpool2d(2, stride=2)

inp = torch.tensor([[[[ 6.,  2,  3,  4],
                      [ 5,  6,  8,  8],
                      [ 9, 10, 11, 16],
                      [14, 14, 16, 16]]]])

output, indices = pool(inp)
print('maxPool:\n', output)
print('pool indices:\n', indices)
print('maxUnpool:\n', unpool(output, indices))

maxPool:
 tensor([[[[ 6.,  8.],
          [14., 16.]]]])

pool indices:
 tensor([[[[[0, 5], [6, 7]],
          [[12, 13], [11, 14, 15]]]]])

maxUnpool:
 tensor([[[[ 6.,  0.,  0.,  0.],
          [ 0.,  6.,  8.,  8.],
          [ 0.,  0.,  0., 16.],
          [14.,  14.,  16.,  16.]]]])
</code></pre>
<p>Is there a way to do this without modifying the c++ code ?</p>

ptrblck: <p>I would recommend to start with a manual implementation in Python via <code>unfold</code> to get the patches and apply your custom pooling logic to each patch.<br>
Note that the currently returned indices have a variable shape and would thus not be representable in a standard tensor but e.g. a <code>list</code> instead or a nested tensor (which is an experimental feature at the moment).<br>
Once this approach is working properly, you could try to reimplement it as a custom layer in the C++/CUDA backend.</p>

melike: <p>By unfold do you mean <code>torch.nn.unfold</code> or <code>tensor.unfold</code> ?</p>

ptrblck: <p>Both should work, but the <code>nn.Unfold</code> module would provide more built-in options such as padding dilation etc.</p>
Bugs with torchvision.transforms.ToPILImage()?

Boyu_Zhang: <p>Hi, I was trying to convert an image of tensor into PILImage and then convert it back around by using the following code. But finally, I found the image’s values were modified after the operation <code>tfs.Compose([tfs.ToPILImage(), tfs.ToTensor()])</code>. Is this a bug in <code>torchvision.transforms.ToPILImage()</code> ? Thank you.</p>
<pre><code class="lang-auto">from skimage import io, color
from torchvision import transforms as tfs
import numpy as np
import torch


file = 'path/to/image.png'
img = color.rgb2ycbcr(io.imread(file)) / 255
(rows, cols, channel) = img.shape
img_y, img_cb, img_cr = np.split(img, indices_or_sections=channel, axis=2)
tensor_y = torch.from_numpy(img_y).float().view(1, rows, cols)

trans = tfs.Compose([tfs.ToPILImage(), tfs.ToTensor()])
preds = trans(tensor_y)

print((tensor_y.data.numpy()==preds.data.numpy()).all())  # return False
</code></pre>

ptrblck: <p>I think you are just losing the accuracy due to quantization.<br>
If you transform your <code>FloatTensor</code> to a <code>PIL.Image</code>, it will be scaled to <code>[0, 255]</code> in <code>uint8</code> type.<br>
This will already quantize your values, as you cannot map all floating values in <code>[0, 1]</code> to a <code>ByteTensor</code>.<br>
The reverse (<code>ToTensor()</code>) thus yields a <code>FloatTensor</code> with these already quantized values.</p>

lrningml: <p>Just ran into the same thing, what’s a good way to view a tensor as an image without the quantization?</p>

Boyu_Zhang: <p>Thanks for your answer. I’ve looked into the function <code>to_pil_image(pic, mode=None)</code> and found it explicitly converts a FloatTensor to [0, 255] in uint8 type. I’m just wondering can we keep the type as float so that there won’t be a precision loss?</p>

Boyu_Zhang: <p>Currently, I just convert the tensor into <code>numpy.ndarray</code> and visualize it through <code>matplotlib</code>, or implement some other image transformations to ensure the precision. After that, I convert it back to <code>torch.Tenosr</code>.</p>

monster: <p>My problem is similar, currently I am using this for augmentation.</p>
<pre><code class="lang-auto">augmentations = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomHorizontalFlip(0.5),
    transforms.ColorJitter(saturation=0.5),
    transforms.ToTensor(),
    utils.normalize_transform(),
])
</code></pre>
<p>so should I remove " transforms.ToTensor()" ?</p>

ptrblck: <p>I don’t think the <code>ToTensor</code> transformation is problematic, but the transformation to a <code>uint8</code> image in <code>ToPILImage()</code> if your use case fits my previous description.<br>
How did you create these floating point images and what do the values represent?</p>
Inplace operation error in multiplication

Chao_Feng: <p>I got the following error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>with anomaly detection enabled, the last line of “traceback of the forward call that caused the error” is</p>
<blockquote>
<p>num = (1 + 2 * c * xy + c * y2) * x + (1 - c * x2) * y<br>
(function _print_stack)<br>
Traceback (most recent call last):</p>
</blockquote>
<p>Error detected in MulBackward0. y has the same shape as the variable modified. If I understand this correctly, it seems the error is in ‘* y’. I don’t understand how ’ * ’ could modify ‘y’ inplace.</p>
<p>BTW c is a scalar tensor, and this multiplication is broadcasted.</p>
<p>Hope this is enough information.  Thanks in advance.</p>

googlebot: <p>What happens is that backprop reaches this line doing MulBackward and detects that some multiplicand is no longer available (overwritten). So inplace operation is happening somewhere later, this is just a detection point.</p>
Train loss decreasing too slow or not

utkuumetin: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef541f0bbf6fe53d27f874831810c1e936b4465c.png" alt="download" data-base62-sha1="y9clm09DirVJH408QNyFiK0CK2o" width="368" height="248"></p>
<p>I can’t find reason to get this result.<br>
What can be wrong?</p>

ptrblck: <p>Basically everything or nothing could be wrong. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
It’s hard to tell the reason your model isn’t working without having any information.<br>
I think a generally good approach would be to try to overfit a small data sample and make sure your model is able to overfit it properly.</p>

utkuumetin: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="103082">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>model isn’t working without having any information.<br>
I think a generally good approach would be to try to overfit a small data sample and make sure your model is able to overfit it prop</p>
</blockquote>
</aside>
<p>Thank you for reply. When play with learning rate and training loop now loss decreasing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Repeat matrix operation for several batches without using for loop

JJangD: <p>I tried this operation for one batch and worked.</p>
<pre><code class="lang-auto">x = torch.FloatTensor([[ax,bx],[cx,dx],[ex,fx],[gx,hx]])
y = torch.FloatTensor([[ay,by],[cy,dy],[ey,fy],[gy,hy]])
z = x[:,None,0] * y[:,1]

print(z)

tensor([[ ax*by,  ax*dy,  ax*fy,  ax*hy],
        [ cx*by,  cx*dy,  cx*fy,  cx*hy],
        [ ex*by,  ex*dy,  ex*fy,  ex*hy],
        [ gx*by,  gx*dy,  gx*fy,  gx*hy]])
</code></pre>
<p>I want to repeat above operation for several batches like below example without using for loop.</p>
<pre><code class="lang-auto">x = torch.FloatTensor([[[ax1,bx1],[cx1,dx1],[ex1,fx1],[gx1,hx1]],
                       [[ax2,bx2],[cx2,dx2],[ex2,fx2],[gx2,hx2]]])
y = torch.FloatTensor([[[ay1,by1],[cy1,dy1],[ey1,fy1],[gy1,hy1]],
                       [[ay2,by2],[cy2,dy2],[ey2,fy2],[gy2,hy2]]])
</code></pre>
<p>This is what I want to get for result.</p>
<pre><code class="lang-auto">tensor([[[ ax1*by1,  ax1*dy1,  ax1*fy1,  ax1*hy1],
         [ cx1*by1,  cx1*dy1,  cx1*fy1,  cx1*hy1],
         [ ex1*by1,  ex1*dy1,  ex1*fy1,  ex1*hy1],
         [ gx1*by1,  gx1*dy1,  gx1*fy1,  gx1*hy1]],

        [[ ax2*by2,  ax2*dy2,  ax2*fy2,  ax2*hy2],
         [ cx2*by2,  cx2*dy2,  cx2*fy2,  cx2*hy2],
         [ ex2*by2,  ex2*dy2,  ex2*fy2,  ex2*hy2],
         [ gx2*by2,  gx2*dy2,  gx2*fy2,  gx2*hy2]]])
</code></pre>
<p>I tried as below, but gave me errors</p>
<pre><code class="lang-auto">z = x[...,None,0] * y[...,1]
RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 1
</code></pre>
<p>Is there any way doing this without for loop?</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">x = torch.arange(2 * 4 * 2).view(2, 4, 2)
y = torch.arange(2 * 4 * 2).view(2, 4, 2)
z = x[:, :, None, 0] * y[:, None, :, 1]
</code></pre>

JJangD: <p>It works perfectly!!!<br>
Thank you!!!</p>
Not sure if validation predictions were collected right

Sokhib_Tukhtaev: <p>I am working on binary classification where there are 2 inputs (image and numerical data) and one output (sigmoid). I need to perform a 5-fold cross validation and plot ROC curves for each fold. This is a code snippet of <code>model.eval</code>:</p>
<pre><code class="lang-auto">        model.eval()
        with torch.no_grad():
            valid_preds_fold = np.zeros((x_val_fold.size(0)))  # (359,)
            ii = 0
            for x_img_batch, x_num_batch, y_batch in valid_loader: # THIS LOOP
                y_val_pred = model(x_img_batch, x_num_batch).detach()
                valid_preds_fold[ii * batch_size:(ii + 1) * batch_size] = sigmoid(y_val_pred.cpu().numpy())[:, 0]
                ii += 1

    fpr, tpr, thresholds = roc_curve(y_val_fold.cpu(), valid_preds_fold) # torch.Size([359, 1]) AND
    tprs.append(np.interp(mean_fpr, fpr, tpr))
    tprs[-1][0] = 0.0
    roc_auc = auc(fpr, tpr)
    aucs.append(roc_auc)
    plt.plot(fpr, tpr, lw=1, alpha=0.3, label='fold %d (AUC = %0.3f)' % (i + 1, roc_auc))

ax.plot([0, 1], [0, 1], linestyle='--', lw=2, color='r', label='level', alpha=.8)
mean_tpr = np.mean(tprs, axis=0)
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
std_auc = np.std(aucs)
ax.plot(mean_fpr, mean_tpr, color='b', label=r'meanAUC = %0.3f $\pm$ %0.2f)' % (mean_auc, std_auc), lw=2, alpha=.8)

</code></pre>
<p><code>for</code> loop which is iterating through <code>valid_loader</code> looks suspicious to me. Basically, what I need to do is that I need to collect a batch of validation predictions (<code>y_val_pred</code>) to  each validation fold’s list (<code>valid_preds_fold</code>) to then calculate the <code>fpr, tpr, thresholds = roc_curve(y_val_fold.cpu(), valid_preds_fold)</code>. I have referred to a couple of resources to come this end but the AUC score of each fold is too low than expected.</p>
<p>Any code inspection to improve the above code is appreciated. Thanks</p>

ptrblck: <p>I’m not quite sure how this code is working exactly, but are you using different <code>valid_loader</code>s for each fold?</p>

Sokhib_Tukhtaev: <p>yes, like in 5-fold cross validation, I am training 4 folds for training and 1 fold for validation</p>

ptrblck: <p>I think as long as you make sure the indices do not overlap or are reused, your code looks alright.<br>
What kind of issue are you seeing at the moment?</p>

Sokhib_Tukhtaev: <p>I have no errors right now, I managed to inspect some little issues, thanks a lot. Just wanted to be sure that that particular part was executed properly.</p>
How to store device information per thread?

Stonepia: <p>I want to launch a function using multi thread, each thread may run in different device context. So I want to use the <code>thread_local</code> keyword in C++.</p>
<p>I wrote the code as following. The code will get an undefined symbol error. Which is basically as folloing :</p>
<pre><code class="lang-auto">ImportError: /home/user/anaconda3/.../_my_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN10foo14MyDeviceInfoD1Ev
</code></pre>
<p>But this code is quite similar with the thread_local use from <a href="https://github.com/pytorch/pytorch/blob/2397c8d1f78cb5b48bcec9c180c1eef81c13b887/c10/util/ThreadLocalDebugInfo.cpp#L6" rel="noopener nofollow ugc">ThreadLocalDebugInfo.cpp</a>.</p>
<p>Could you give me some hint on how to solve this problem? How could I store device information per thread?</p>
<pre><code class="lang-auto">// mydevice.h
namespace foo{
class MyDeviceInfo{
private:
     c10::Device device_; 
      //...
public:
      MyDeviceInfo() : device_(c10::Device(c10::DeviceType::CPU, 0)) {}
      MyDeviceInfo(c10::Device device) : device_(device) {}
      c10::Device&amp; get_device() { return device_ ; }
...
};

MyDeviceInfo&amp; get_tls_MyDeviceInfo();
void set_tls_MyDeviceInfo(c10::Device device);
} // namespace foo

//-------------------------

// mydevice.cpp
thread_local std::shared_ptr&lt;foo::MyDeviceInfo&gt; device_info;

namespace foo{
   void set_tls_MyDeviceInfo(c10::Device device) {
          device_info = std::make_shared&lt;MyDeviceInfo&gt; (device) ;
   }
   MyDeviceInfo&amp; get_tls_MyDeviceInfo(){
           return device_info-&gt;get_device();
    }



} // namespace foo

</code></pre>

Stonepia: <p>The problem seems to lies on <code>c10::Device</code> . If I separate the initialization of <code>c10::device</code> to two part as :</p>
<pre><code class="lang-auto">class foo {
  private :
    c10::DeviceType device_type_;
    int16_t index_;

  public:
    foo() : index_(-1), device_type_(c10::DeviceType::CPU){}
    foo(c10::Device device) : index_(device.index()), device_type_(device.type()){}
    foo(c10::DeviceType device_type, int index) : index_(index), device_type_(device_type){}
    int get_index() { return index_; }
    c10::DeviceType get_device_type() { return device_type_; }
};
</code></pre>
<p>This would works fine.  So where is the document about this? Why using c10::Device is wrong?</p>
Reconstruction of a vector by autoencoder (effect of input size and range)

mmdbrdrn: <p>Hello,</p>
<p>I have two questions about the Autoencoder based signal (a vector here, considering an FC-autoencoder) reconstruction. I would be really thankful if anyone helps me through this.</p>
<p>1: does reconstruction error depend on vector size? (for example reconstruction a dataset with signals of 3 dimension like, [x1,x2,x3] versus 4 dimensional inputs like [x1,x2,x3,x4])</p>
<ol start="2">
<li>can we say that reconstruction error depends on the mean of the inputs? set of inputs with higher values (like [10,10,10] would show higher reconstruction error versus lower ranged inputs ( like [1,1,1])? if yes is it recommended to normalize data beforehand?</li>
</ol>
<p>Thank you so much</p>

ptrblck: <ol>
<li>It depends how the loss is calculated. If you are using e.g. <code>nn.MSELoss</code> in the default setup, the loss value should not depend on the input feature dimension:</li>
</ol>
<pre><code class="lang-python">x = torch.randn(1, 3)
y = torch.randn(1, 3)
criterion = nn.MSELoss()
loss_small = criterion(x, y)

x = torch.randn(1, 3000)
y = torch.randn(1, 3000)
loss_large = criterion(x, y)

print(loss_small)
&gt; tensor(1.5440)

print(loss_large)
&gt; tensor(2.0731)
</code></pre>
<p>However, you can of course use <code>reduction='sum'</code>, which would change it.</p>
<ol start="2">
<li>It depends again on your use case and the loss value will depend on the magnitude:</li>
</ol>
<pre><code class="lang-python">y = torch.tensor([[1.]])
rel_err = 1e-1
x = y - y * rel_err
loss_small = criterion(x, y)

y = torch.tensor([[100.]])
x = y - y * rel_err
loss_large = criterion(x, y)

print(loss_small)
&gt; tensor(0.0100)

print(loss_large)
&gt; tensor(100.)
</code></pre>
<p>As you can see, the loss is much higher in the second use case even though the relative error is the same. I don’t know how you are interpreting the loss, but it doesn’t necessarily mean that the first use case is “better” than the second one.<br>
That being said, normalizing the inputs often helps during the training so you might want to normalize the inputs anyway and could even “unnormalize” the outputs, if necessary.</p>

mmdbrdrn: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>. You are right, I think I should deeply consider and think about my loss_function.<br>
I think I should read your comments several times and think more. Thank you again.</p>
Can you use single timestep data for LSTM autoencoder?

suits_cloud: <p>I have a dataset with 1000 samples, 1 timestep, and 1 feature (1000,1,1). I see a lot of discussions where they create subsequences (split one timestep column into multiple by using some X number - ex: (100,10,1)) and use that as input to LSTM autoencoder and argue that LSTMs/autoencoder can only learn when the data has multiple timesteps. But, I get good results with just 1 timestep column. What is the correct input here? Is it incorrect if I use univariate (1000,1,1) input?</p>
Working with torchlib 1.5 on Visual Studio

sharonaim: <p>No matter how many times I try, I can’t seem to make the new version of torchlib with cuda to work on Visual Studio 2019. Searched for guides, but all were outdated because are not for the new version of the library.</p>

msminhas93: <p>In this tutorial, I explain step by step how to setup a project in Visual Studio 2019 and run a sample code. Hope this helps!</p>
<p><a href="https://towardsdatascience.com/setting-up-a-c-project-in-visual-studio-2019-with-libtorch-1-6-ad8a0e49e82c" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/setting-up-a-c-project-in-visual-studio-2019-with-libtorch-1-6-ad8a0e49e82c</a></p>
Tailering or customising a model with Modulelist()

Sneha_Verma: <p>i have created a neural net with nn.Moduelist()<br>
i want to show r squared error</p>
<p>i would like to use it for  10 benchmark runs with diﬀerent data splittings and diﬀerent random<br>
seeds for the network initialization.</p>
<p>also i would like to know how i can use these benchmark runs to obtain the R 2 scores</p>
Scatter_add reduce output dimensions/shape

sbb: <p>I have memory issues with the scatter_add function.</p>
<p>The standard implementation computes, e.g. self[i][index[i][j][k][m]][k][m] += src[i][j][k][m],  where the output, will have shape I x Index_Max x K x M.  This is often a large tensor, causing memory issues. I then sum over certain dimensions., e.g. if choose dimension 0,3 it will then compute a tensor shape 1x Index_Max x K x 1.</p>
<p>I am looking for an option, or workaround, to choose some of the output dimensions to have shape 1.  In the example above, the scatter_add function would compute<br>
self[0][index[i][j][k][m]][k][0] += src[i][j][k][m]<br>
(or ignore dimensions 0,3 altogether), not computing the entire output tensor.</p>
<p>Thank you for the suggestions!</p>

el_samou_samou: <p>In that case what about:</p>
<ol>
<li>Perform sum reduction on the self: <code>self=self.sum(0).sum(2)</code>
</li>
<li>Reshape index and src to delete obsolete dimensions: <code>index = torch.reshape(index.permute(3, 0, 1, 2), (-1, index.size(2))</code><br>
and <code>src= torch.reshape(src.permute(3, 0, 1, 2), (-1, src.size(2))</code>
</li>
<li>Call the scatter add function of the already reduced self: <code>self = self.scatter_add(0, index, src)</code>
</li>
<li>Unsqueeze the compressed dimensions: <code>self = self.unsqueeze(0).unsqueeze(3)</code>
</li>
</ol>
<p>Would that work/reduce memory print for you?</p>
<p>Samuel</p>

sbb: <p>+10, thank you very much for the reply.</p>
How to load COCO dataset to dataloader to not overload memory

Tomash: <p>Hi, I have a problem with some memory leak (?).<br>
When I load my <code>dataset</code> the usage of memory increase to 100 processes with 30 GB of RES memory. I think that the problem is my <code>Dataset</code> function. So I have a question how to load dataloader part by part (is it possible to reduce in this way memory necessary in my computations)? I attach my function below.</p>
<blockquote>
<pre><code>class LoadDataset(Dataset):
</code></pre>
</blockquote>
<blockquote>
<pre><code>def __init__(self):
    self.images = []
    self.targets = []
    img_path, ann_path = (
        "path",
        "ann",
    )
    coco_ds = torchvision.datasets.CocoDetection(img_path, ann_path)
    for i in range(0, len(coco_ds)):
        img, ann = coco_ds[i]
        images, targets = collate(
                        [img.copy(), img.copy()], [ann, ann], coco_ds.coco
                    )
                    for t in targets:
                        self.targets.append(t)
                    for image in images:
                        self.images.append(image)

def __len__(self):
    return len(self.images)

def __getitem__(self, idx):
    img = self.images[idx]
    target = self.targets[idx]
    return (
        img,
        target,
    )
</code></pre>
</blockquote>
<p>Later I load data before iteration over epochs like:</p>
<blockquote>
<p>train_loader = DataLoader(LoadDataset(), batch_size=24, shuffle=True)</p>
</blockquote>
About the usage of register_module in C++

Chen0729: <p>Hi, All</p>
<p>I am trying to use<br>
auto gen=generator(register_module(“generator”, Generator(100, 28*28)));</p>
<p>But the libtorch in C++ said “use of undeclared identifier 'register_module”</p>
<p>Any suggestions to correct it or do I miss any header file to include for “register_module”</p>
<p>Thanks.</p>