akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Normalizing a Tensor column wise

username: somnath
cooked: <p>I have a Tensor containing these values.</p>
<pre><code class="lang-auto">1000  10  0.5
765   5   0.35
800   7   0.09
</code></pre>
<p>I want to normalize it column wise between 0 and 1 so that the final tensor looks like this:</p>
<pre><code class="lang-auto">1     1    1
0.765 0.5  0.7
0.8   0.7  0.18 (which is 0.09/0.5)
</code></pre>
<p>Based on <a href="https://stackoverflow.com/q/29661574" rel="nofollow noopener">this question</a>.</p>

username: albanD
cooked: <p>I guess you just want to do <code>x_normed = x / x.max(0, keepdim=True)[0]</code> (EDITED: added missing <code>[0]</code>).</p>

username: somnath
cooked: <p>I tried this code.</p>
<pre><code class="lang-auto">import torch


def normalize(x):
    x_normed = x / x.max(0, keepdim=True)
    return x_normed


t = torch.tensor([[0.0, 2, 3], [5, 10, 15], [10, 20, 30]])
print(normalize(t))

</code></pre>
<p>This was the error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/somnath/Code/ml/src/test.py", line 7, in &lt;module&gt;
    print(normalize(t))
  File "/home/somnath/Code/ml/src/test.py", line 3, in normalize
    x_normed = x / x.max(0, keepdim=True)
TypeError: div() received an invalid combination of arguments - got (tuple), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (!tuple!)
 * (float other)
      didn't match because some of the arguments have invalid types: (!tuple!)
</code></pre>
<p>How to correct this?</p>

username: somnath
cooked: <p>Looks like <code>max()</code> returned a tuple with one element. So, choosing the first element solved the issue. The corrected code as of PyTorch 0.4 is as below:</p>
<pre><code class="lang-auto">import torch


def normalize(x):
    x_normed = x / x.max(0, keepdim=True)[0]
    return x_normed


t = torch.tensor([[1000, 10, 0.5], [765, 5, 0.35], [800, 7, 0.09]])
print(normalize(t))

</code></pre>

username: albanD
cooked: <p>Ho yes I forgot that, it returns both the values and indices, you only want the values in your case.</p>

username: somnath
cooked: <p>No issues,  thanks all the same. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>

username: Intel_Novel
cooked: <p>If you normalize X by X max, and Y by Y max this is not quite right. You lost the X/Y ratio. What then? Should I ask this question on separate thread?</p>

username: Jinhua_Wang
cooked: <p>Is there a way to limit the normalised values between [-1, 1], instead of [0, 1]?</p>

username: albanD
cooked: <p>Well depends what you want as your normalization. But you can always do <code>2 * (t - 1)</code> to go from [0, 1] to [-1, 1].</p>
Reobtain sorted tensor from argsort

username: ymentha14
cooked: <p>Hi,</p>
<p>I am looking for a way to  recover the sorted version of a matrix from its argsort indexes in an efficient (pytorch friendly) way:</p>
<p>example:</p>
<pre><code class="lang-auto">k  = tensor([[2, 9, 3],
             [8, 0, 0]])
sorted_k,indx = k.sort()
</code></pre>
<p>but running  <code>k[indx]</code> leads to an error as the shape don’t match instead of returning <code>sorted_k</code>. My best version so far is<br>
<code>torch.cat([row[ind].unsqueeze(0) for ind,row in zip(indx,k)])</code><br>
But since it involves a for loop, big computation get very slow as I run it. Is there a way to do better?<br>
<a href="https://discuss.pytorch.org/t/mapping-tensor-based-on-corresponding-index-tensor/53307">Related topic</a><br>
Thanks for your help!</p>

username: LeviViana
cooked: <p>I’m not sure I got your point. If you want to know how to use the indexes to sort a tensor, then this snippet could help:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
k  = torch.tensor([[2, 9, 3],
             [8, 0, 0]])
sorted_k, indx = k.sort()
sorted_k_again = k.gather(1, indx)
sorted_k_again.equal(sorted_k) # &lt;- prints True
</code></pre>
<aside class="quote no-group" data-username="ymentha14" data-post="1" data-topic="75008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ymentha14/48/22199_2.png" class="avatar"> ymentha14:</div>
<blockquote>
<p>I am looking for a way to recover the sorted version of a matrix from its argsort indexes</p>
</blockquote>
</aside>
<p><code>sorted_k</code> is what you want, if I get this right.</p>

username: ymentha14
cooked: <p>That’s exactly it. Thank you a lot!</p>
Efficient way to one hot encode whole image for semantic segmentation

username: mayool
cooked: <p>Hi, I am working on semantic segmentation. Currently, my labels for training are of size [batch_size, channels, h, w] where channels = 1, each pixel is either 0 (pixel belongs to background) or 1 (pixel belongs to target_class). What I need to do is to convert my [batch_size, 1, h,w]-label into a [batch_size, 2, h, w]-label where each pixel with 0 is converted to [1,0] and each 1 to [0,1].<br>
Basicially one-hot encode each pixel.</p>
<p>I don’t want to iterate through every pixel by for loop since this would take way too long.</p>
<p>Can someone help me out? I am sure there is a faster way to do this?<br>
Cheers!</p>

username: b02202050
cooked: <p>Hi,</p>
<pre><code class="lang-python">one_hot_label = torch.cat((1 - current_label, current_label), dim=1)
</code></pre>
<p>explanation:</p>
<ol>
<li>(1 - current_label, current_label) would be (1,0) when current label is 0; and it would be (0,1) when current label is 1.</li>
<li>dim=1 means to operate on the 2nd dimension (which is channels)</li>
</ol>
LibTorch/C++ General grad function

username: sanithovski
cooked: <p>In PyTorch, there exists a general autograd function:<br>
<code>torch.autograd.grad(input,output,...)</code><br>
that doesn’t require in-place operations on tensors. Does a similar function or interface exist in libTorch? Currently, I can find most of the in-place operations but not this general one.</p>

username: Erissonleo
cooked: <p>I have the same problem with you. I want to use</p>
<pre><code class="lang-auto">torch::autograd::backward() 
</code></pre>
<p>to instead. But I got the error log with</p>
<blockquote>
<p>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 64, 4, 4]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>I dont’t konw how to solve the problem. Do you Solve it? Thanks!</p>

username: Jamie_Mair
cooked: <p>I have encountered the same error. I have narrowed down the code causing the “inplace operation” to when I have applied the optimiser .step() function. The optimiser used was torch::optim::Adam.</p>
How to save the predicted multi-classes segmentation outputs tensor like the single channel ground truth?

username: XiaoAHeng
cooked: <p>In multi-classes segmentation, the shape of the model’s last layer is [batch , num_classes, H, W] , e.g. [1, 6, 512, 512],however ,the ground truth shape is [1, 1, 512, 512], how to transform the output tensor shape<br>
into the ground truth shape and save it?<br>
Thanks in advance!</p>

username: ptrblck
cooked: <p>If you are dealing with a multi-class classification and would like to use <code>nn.CrossEntropyLoss</code> (or <code>nn.NLLLoss</code>) as your criterion, the output of your model should contain the logits (or log probabilities) as <code>[batch_size, nb_classes, height, width]</code>, while your target should be a <code>LongTensor</code> containing the class indices as <code>[batch_size, height, width]</code>.<br>
If your target already contains these indices, you can just call <code>target = target.squeeze(1)</code> on it to remove the channel dimension.</p>

username: XiaoAHeng
cooked: <p>I have known how to do,  maybe I have no clear statement, you misunderstand my question , we should use torch.argmax() on the predicted multi-classes segmentation outputs</p>

username: Mario_Parreno
cooked: <p>Hi, I am in the same situation. Finally you used softmax/argmax in the num_classes dimension? [batch , num_classes, H, W -&gt; [batch , H, W]</p>
Element wise product with different dimension

username: 11169
cooked: <p>I have a normal distribution with means and variances whose shape are [32, 1, 28, 28].<br>
I want to get ten samples from the normal distribution as follows.<br>
x = mean + variance * epsilon (epsilon is sampled from N(0,1) )</p>
<p>First, I call torch.normal(0, 1, size=[32, 1, 28, 28, 10]).<br>
And then I want to product variance and epsilon, resulting in a [32, 1, 28, 28, 10] matrix.</p>
<p>How can I do it?</p>
Seeing inconsistent behaviors with register_forward_hook with pretrained networks

username: bigharryox
cooked: <p>I have been looking at adding layer output visualizations to my models and have been working through doing this on pretrained networks modeled after some approaches i found online.</p>
<p>That said, i found that some of the pretrained Conv2D layers are outputting an activation of 0 when running a random image through and while expermenting with why, i found that i would get inconsistent outputs depending on which layers I activated.</p>
<pre><code class="lang-auto">class HookedLayerRunner():
    def __init__(self, model, selected_layer):
        self.model = model
        self.selected_layer = selected_layer
        self.layer_output = None

    def hook_layer(self):
        def hook_function(module, grad_in, grad_out):
            self.layer_output = grad_out[0]
        self.model[self.selected_layer].register_forward_hook(hook_function)

    def process(self, img):
        self.hook_layer()
        processed_image = process_image(img, device)
        self.model(processed_image)
</code></pre>
<p>And the code exercising the apparent bug.</p>
<pre><code class="lang-auto">filt = 10
model = models.vgg16(pretrained=True)
img = np.uint8(np.random.uniform(150, 180, (100, 100, 3)))/255.0 - 0.5
act = HookedLayerRunner(model.features, 0)
act.process(img)
print(act.layer_output[filt].mean())

t = process_image(img, device)
conv = model.features[0]
res = conv(t)
print(act.layer_output[filt].mean())

conv = nn.Sequential(model.features[0], model.features[1])
res = conv(t)
print(act.layer_output[filt].mean())
</code></pre>
<p>and the results, showing inconsistency depending on whether the model runs layer one in a nn.Sequential layer (as is done in the vgg16 pretrained model) or the conv2d layer is run as a singleton.</p>
<pre><code class="lang-auto">tensor(0.7511, grad_fn=&lt;MeanBackward0&gt;)
tensor(0.7456, grad_fn=&lt;MeanBackward0&gt;)
tensor(0.7511, grad_fn=&lt;MeanBackward0&gt;)
</code></pre>
<p>Any thoughts on what’s up in this? Am i just using the hook incorrectly, or is there a behavior to hooking the forward output which is dependent on the subsequent layers (a Relu in this case)?</p>

username: bigharryox
cooked: <p>Tried with different layer combinations and found some interesting differences. The issue seems to arise only when the following layer in the sequential is a ReLU. Output from different models being run:</p>
<pre><code class="lang-auto">Whole Model 0.7501
Sequential(Conv2D) 0.7446
Sequential(Conv2D, ReLU) 0.7501
Manual sequential1 (Conv2D, ReLU) 0.7446
Manual sequential2 (Conv2D, ReLU) 0.7501
Sequential(Conv2D, Conv2D, ReLU) 0.7446
</code></pre>
<p>I can’t figure out how putting a Relu after the convolution would change the output of the convolution from the hook. Even manually running the sequence of layers seems to exercise the issue…</p>