akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there a way in Pytorch to check if something is a number(dim()==0) or a tensor(dim() >0)

Shisho_Sama: <p>How can I know I’m dealing with a number or a tensor other than checking the dim() method?<br>
is there a method for this? or should I just use dim() for this?<br>
Thank you all in advance</p>

cyanM: <p>i think type(variable) is what you want</p>

Shisho_Sama: <p>using type only results in <code>&lt;class 'torch.Tensor'&gt;</code> for everything in this case.</p>

zzoon91: <p>I agree with <a class="mention" href="/u/cyanm">@cyanM</a><br>
the alternative would be using <code>torch.is_tensor()</code></p>

Shisho_Sama: <p>Thanks, I have sth like :</p>
<pre><code class="lang-auto">if one_hot_tensor.size(0) &gt; 1 : 
    return torch.argmax(one_hot_tensor)
else:
    return one_hot_tensor[0].int()
</code></pre>
<p>and this always returns a tensor and none of these proposed methods work!</p>

cyanM: <p>it means that one_hot_tensor.size(0) &gt; 1 has always been true, and torch.argmax() will return a tensor</p>
How to obrain 3D mesgrid in pytorch

John1231983: <p>I have a code to obrain 2D mesgrid in pytorch such as</p>
<pre><code class="lang-auto">import torch
x = torch.randn(1,2,3,4)
B, C, H, W = x.size()
# mesh grid 
xx = torch.arange(0, W).view(1,-1).repeat(H,1)
yy = torch.arange(0, H).view(-1,1).repeat(1,W)
xx = xx.view(1,1,H,W).repeat(B,1,1,1)
yy = yy.view(1,1,H,W).repeat(B,1,1,1)
grid = torch.cat((xx,yy),1).float()
</code></pre>
<p>For 3D, we have x size of BxCxDxHxW. What should I add to obtain 3D mesgrid? Thanks<br>
This is what I tried</p>
<pre><code class="lang-auto">import torch
x = torch.randn(1,2,3,4,5)
B, C, D, H, W = x.size()
# mesh grid 
xx = torch.arange(0, W).view(1,-1).repeat(D,H,1)
yy = torch.arange(0, H).view(-1,1).repeat(D,1,W)
zz = torch.arange(0, D).view(1,-1).repeat(1,H,W)
print (xx.shape,yy.shape,zz.shape)
xx = xx.view(1,1,D,H,W).repeat(B,1,1,1,1)
yy = yy.view(1,1,D,H,W).repeat(B,1,1,1,1)
zz = zz.view(1,1,D,H,W).repeat(B,1,1,1,1)
grid = torch.cat((xx,yy,zz),1).float()
print (xx.shape,yy.shape,zz.shape)
print (grid)
</code></pre>
<p>The z direction looks wrong</p>
<pre><code class="lang-auto">torch.Size([3, 4, 5]) torch.Size([3, 4, 5]) torch.Size([1, 4, 15])
torch.Size([1, 1, 3, 4, 5]) torch.Size([1, 1, 3, 4, 5]) torch.Size([1, 1, 3, 4, 5])
tensor([[[[[0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.]],

          [[0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.]],

          [[0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.],
           [0., 1., 2., 3., 4.]]],


         [[[0., 0., 0., 0., 0.],
           [1., 1., 1., 1., 1.],
           [2., 2., 2., 2., 2.],
           [3., 3., 3., 3., 3.]],

          [[0., 0., 0., 0., 0.],
           [1., 1., 1., 1., 1.],
           [2., 2., 2., 2., 2.],
           [3., 3., 3., 3., 3.]],

          [[0., 0., 0., 0., 0.],
           [1., 1., 1., 1., 1.],
           [2., 2., 2., 2., 2.],
           [3., 3., 3., 3., 3.]]],


         [[[0., 1., 2., 0., 1.],
           [2., 0., 1., 2., 0.],
           [1., 2., 0., 1., 2.],
           [0., 1., 2., 0., 1.]],

          [[2., 0., 1., 2., 0.],
           [1., 2., 0., 1., 2.],
           [0., 1., 2., 0., 1.],
           [2., 0., 1., 2., 0.]],

          [[1., 2., 0., 1., 2.],
           [0., 1., 2., 0., 1.],
           [2., 0., 1., 2., 0.],
           [1., 2., 0., 1., 2.]]]]])
</code></pre>

Zhihao_Wu: <p>How about using <code>torch.meshgrid(torch.arange(4), torch.arange(5), torch.arange(6))</code>? It will return three tensors that you can produce 3d coordinates.</p>
Is pytorch parrallel?

JohnCwok: <p>Hello !</p>
<p>I am trying to speed up my code and unfortunately don;t have a GPU that supports CUDA, so I wanted to know if I could parralelise my code (things like batch processing of input by a LSTM) to let it run over several cores. Is this automatically taken care of in pytorch or does one have to write extensive complicated modification to achieve this ?</p>
<p>Best,</p>
<p>J.</p>
Function 'SymeigBackward' returned nan values in its 0th output

6fb4b64ff0a68bcbf8fa: <p>I use</p>
<pre><code class="lang-auto">        with autograd.detect_anomaly():
</code></pre>
<p>and error detected</p>
<pre><code class="lang-auto">Function 'SymeigBackward' returned nan values in its 0th output.
</code></pre>
<p>I use symeig function</p>
<pre><code class="lang-auto"># ReEig Layer
def cal_rect_cov(features):
    weight1, weight2 = variable_with_orth_weight_decay(features.shape)
    features = torch.bmm(torch.bmm(weight2, features), weight1)
    result=[]
    for i in range(features.shape[0]):
        s_f,v_f=torch.symeig(features[i], eigenvectors=True)
        s_f_clamp=torch.clamp(s_f,0.0001,10000)
        s_f_clamp2=torch.diag(s_f_clamp)
        sv_m=torch.matmul(v_f,s_f_clamp2)
        features_t=torch.matmul(sv_m,v_f.t())
        result.append(features_t)
    result = torch.stack(result)
    return result

# LogEig Layer
def cal_log_cov(features):
#     features=features.detach().cpu()
    result=[]
    for i in range(features.shape[0]):
        s_f,v_f=torch.symeig(features[i], eigenvectors=True)
        s_f_log = torch.log(s_f)
        s_f_log2=torch.diag(s_f_log)
        sv_m=torch.matmul(v_f,s_f_log2)
        features_t=torch.matmul(sv_m,v_f.t())
        result.append(features_t)
    result = torch.stack(result)
    return result
</code></pre>
<p>when get cnn feature, and make covariance matrix using torch.symeig</p>
<p>but this error detected.</p>
<p>plz help me how to get no nan values in ‘SymeigBackward’</p>
<p>Thank you<br>
-James</p>

Rui_Meng: <p>I think the symeig method is very sensitive to the backpropogation. The better way to avoid the non gradient issue is to make your matrix more robust. It means that you need to check the matrix conditional number. And may add some jitters to the diagnal elements  or change your model which can model the noises.</p>
How to choose the proper metrics for False Positive Reduction

Juezhao_Yu: <p>Hi, I am dealing with a false positive reduction problem,  and the ratio of positive and negative is 1.7:1.</p>
<p>Firstly, I wonder what can I do to deal with slightly skewed training data?  Though Applying weights to different classes, and over/undersampling of the data are already noted, shall I try those tricks independently? Or shall I do nothing at all?</p>
<p>At the moment I have difficulty finding the proper metrics to evaluate my models. Any materials like github repos, or kaggle kernel would be much appreciated.<br>
Thanks!!!</p>
When will pytorch introduce partial requires_grad

ksouvik52: <p>I wonder why PyTorch has not introduced partial requires_grad() to a tensor instead of making it either TRUE or FALSE for the whole tensor.</p>

albanD: <p>Hi,</p>
<p>There is no plan to introduce this at the moment.<br>
What would be the use case where you require this?</p>

ksouvik52: <p>Hi, Thanks for replying!<br>
The partial requires_grad may stop the unnecessary gradient calculations for masked retraining of the weights!</p>

albanD: <p>Unfortunately, if you use an arbitrary mask, this won’t help in most cases, as unless your mask is very very sparse &lt;1% nnz values, then it will be faster to do the full computation on the GPU then mask the result.</p>
Running nn.DataParallel model with occasionally missing losses

broccolizzy: <p>Hi all,</p>
<p>I am trying to implement a model that runs with multigpu training and DataParallel. The problem is that due to the nature of my model, occasionally there will be forward paths where losses are not produced.</p>
<p>Problem:<br>
Without data parallel, I simply set these missing losses to None or 0, and only add losses that are not None or 0 to my total loss. However, moving on to multigpu, there are instances where 1 process on GPU 0 will produce an actual loss, while the process on GPU1 will <em>not</em> produce that particular loss.</p>
<p>Hence, when DataParallel module gathers these losses, bad things happen – it cannot combine the losses from two GPU together.</p>
<p>Things I tried:<br>
I tried making the missing loss 0, a float tensor of 0, or setting it to None. None of these methods work – they each produce an error in torch/nn/parallel/scatter_gather.py. For instance, setting the missing loss to 0, which is not iterable, will give “TypeError: zip argument <span class="hashtag">#1</span> must support iteration” error.</p>
<p>Question:<br>
I am wondering if there is a particular thing in DataParallel that ignores missing entries when combining losses from multiple GPUs? Or is there a possible solution to this problem without altering pytorch code?</p>
<p>Any help is greatly appreciated!</p>
How to know if an input has an influence on the result?

Vincent_Brisse: <p>Hello. I’m working on a conditional model that generates music. I added a label to the input as a one_hot_vector that could be an artist, a genre or an instrument. My worry is that the label is a single one_hot_vector (so max 1) while the input is very long (let’s say len = 100 for example) and values range from 1 to 256. Therefore I’m afraid that priming with the label has no impact over all.<br>
Is there a way to know if the label has an impact ? And if it doesn’t have an impact, how to “artificially” improve the influence of the label on training / generation ?</p>
<p>Thank you !</p>

benoriol: <p>Hi,</p>
<p>First of all, it is usually recommended to normalize the input, so maybe you could try to change the range of the input from (1-256) to (-1, 1). I think this might help.<br>
If you want know how much does an input affect to the output, maybe you could try monitoring the gradient vector’s magnitude. I mean the gradient with respect to the input label, not with respect to the weights. If the gradient is very small, it might mean that the input makes small difference on the output.</p>
<p>Ideally, the model should be able to learn the importance of the label in the input, but if you want to “artificially” force that, maybe you could try repeating the label a bunch of times in the input?</p>
<p>Hope this helps</p>

Vincent_Brisse: <p>Thanks for the help Ben,<br>
Nevermind I just checked the input is indeed normalized, still I feel like the label should have more importance for some reason. Intuitively, it must have a very big influence and it’s hard to check its influence with regards to generation (the results are still a bit messy anyway).</p>
Difference between BCELoss and BCEWithLogitsLoss when using large values

gbraso: <p>Hi!</p>
<p>From my understanding, using the BCEWithLogitsLoss should yield the same results as BCELoss composed with sigmoid units. And the only difference between the two is that the former (BCEWithLogitsLoss) is numerically more  stable.</p>
<p>However, when I test their behavior, I get significantly different results as soon as I deal with loggits with values over 10e2.</p>
<p>Minimal example:</p>
<pre><code class="lang-auto">import torch
from torch import nn

preds = torch.rand(10)
preds[0] = 1e2

labels = torch.zeros(10)

criterion = nn.BCELoss()
print(criterion(nn.Sigmoid()(preds), labels)) #outputs tensor(3.5969)
criterion = nn.BCEWithLogitsLoss()
print(criterion(preds, labels)) #outputs tensor(10.8338)
</code></pre>
<p>I am using pytorch 1.2.0.</p>
<p>Could someone please tell me whether I am doing anything wrong or this behavior is to be expected?</p>
<p>Thanks in advance!</p>

ptrblck: <p>You are saturating the sigmoid with such a high number.<br>
Have a look at this small test:</p>
<pre><code class="lang-python">labels = torch.zeros([1])

for x in torch.linspace(10, 100):
    print((torch.sigmoid(x) - 1.) == 0, ' for ', x)
    x = x.view(1)
    ce = F.binary_cross_entropy(torch.sigmoid(x), labels)
    ce_logit = F.binary_cross_entropy_with_logits(x, labels)
    err = torch.abs(ce - ce_logit)
    print('error ', err)
</code></pre>
<p>As you can see, the first print statement will return <code>True</code> for logits of ~17, which means the limited floating point precision returns a value of 1. for <code>torch.sigmoid(x)</code>.<br>
Likewise the error will jump up by a large margin.</p>
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1535493744281/work/aten/src/THC/THCTensorRandom.cu:34" Error when training a CNN

BeiDu: <p>The Model is based on Alexnet. I was training it with python 3.7  pytorch 0.4.1.<br>
But in the forward process it showed the following errors:<br>
Traceback (most recent call last):<br>
File “interpretable_CNN-Zhang-learned_template.py”, line 683, in <br>
model=train_model(model,dataloaders,loss_func,optimizer,scheduler,device,100)<br>
File “interpretable_CNN-Zhang-learned_template.py”, line 494, in train_model<br>
outputs=model(inputs)<br>
File “/home/TUE/20183494/miniconda3/envs/env_new/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 477, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “interpretable_CNN-Zhang-learned_template.py”, line 402, in forward<br>
x=self.dropout(x)<br>
File “/home/TUE/20183494/miniconda3/envs/env_new/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 477, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/TUE/20183494/miniconda3/envs/env_new/lib/python3.7/site-packages/torch/nn/modules/dropout.py”, line 53, in forward<br>
return F.dropout(input, self.p, self.training, self.inplace)<br>
File “/home/TUE/20183494/miniconda3/envs/env_new/lib/python3.7/site-packages/torch/nn/functional.py”, line 595, in dropout<br>
return <em>functions.dropout.Dropout.apply(input, p, training, inplace)<br>
File "/home/TUE/20183494/miniconda3/envs/env_new/lib/python3.7/site-packages/torch/nn/<em>functions/dropout.py", line 40, in forward<br>
ctx.noise.bernoulli</em>(1 - ctx.p).div</em>(1 - ctx.p)<br>
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1535493744281/work/aten/src/THC/THCTensorRandom.cu:34</p>
<p>Is anybody encounter this problem? I don’t understand what caused this problem, is there anybody can help me ?<br>
Many thanks</p>

mc1234567: <p>Have you solve the problem, I am facing the same issue.</p>
How to link weights between forward and backward pass in custom layer?

Vaufreyd: <p>Hello,</p>
<p>I want to create a custom convolution layer. I will add weights (and biaises, …) to my layer and make a custom computation for the forward pass. For this, I found some examples (and look to conv2D in module.nn)</p>
<p>I have 2 questions:</p>
<ul>
<li>First, how am I sure that the backward pass will take care of my parameters? As far as I saw, creating them (like in conv2D) should do the trick but I’d like to be sure. Am I right?</li>
<li>Second, is there any standard procedure to test a new layer? A standard test?</li>
</ul>
<p>Thank you for your answer. Doms.</p>
Multiprocess cv2 data loader

F-Schlatt: <p>To start off, I’m not sure if this is a Windows only issue or not, since many objects aren’t pickable under Windows. If anyone knows for certain, please let me know. I could get access to a Linux machine as well.</p>
<p>None the less, I’m trying to piece together a dataloader for a large set of very long videos. I need to sample random frames from these videos so a sequential decoding (which would be the fastest and wouldn’t need multiprocessing) is out of the question. I’ve been profiling different approaches and found that cv2 (~3 seconds for random 100 indices) &gt; pyav (~10 seconds) &gt; ffmpeg (~20 seconds, using subprocesses) is the fastest. Unfortunately, running the dataloader with cv2 and multiple workers is not possible as <code>cv2.VideoCapture</code> objects aren’t pickleable. Creating the capture object in the <code>__iter__</code> is slower then the ffmpeg method though.</p>
<p>In short this is what the Dataset looks like:</p>
<pre><code class="lang-python">class VideoDataset(Dataset):

    def __init__(self, vid_paths):
        self.caps = [cv2.VideoCapture(vid_path) for vid_path in vid_paths]

    def grab_idcs(self, idx):
        ...

    def __iter__(self, idx):
        vid_idx, frame_idx = self.grab_idcs(idx)
        cap = self.caps[vid_idx]
        cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
        frame = cap.read()
        return frame
</code></pre>
<p>The solution would be to be able to create the list of capture objects within each worker, but I doubt there is an option to do that. Does anyone have an idea of how to avoid the pickling error?</p>

F-Schlatt: <p>I found a solution, which is still not quite fast enough though. Just save the vid paths in init, and initialize the capture objects in the first call of iter:</p>
<pre><code class="lang-python">class VideoDataset(Dataset):

    def __init__(self, vid_paths):
        self.vid_paths = vid_paths
        self.caps = None

    def grab_idcs(self, idx):
        ...

    def __iter__(self, idx):
        vid_idx, frame_idx = self.grab_idcs(idx)
        if self.caps is None:
            self.caps = [cv2.VideoCapture(vid_path) for vid_path in self.vid_paths]
        cap = self.caps[vid_idx]
        cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
        frame = cap.read()
        return frame
</code></pre>
Is there something wrong to learn multiple models with the same batch simultaneously?

LZY: <p>I did something like this:<br>
data = data.to(device)<br>
label = label.to(device)<br>
out1 = model1(data, label)<br>
loss1 = model1.loss<br>
loss1.backward()<br>
out2 = model2(data, label)<br>
loss2 = model2.loss<br>
loss2.backward()</p>
<p>Model1 and model2 are the same models. With the codes above, model2 was not learned correctly at all.<br>
But if I duplicate the batch into GPU twice and learn model1 and model2 with them respectively, both model1 and model2 are gonna be ok.</p>
<p>I think once the backward of loss1 was done, the computational graph would be freed and have no effects on the following computations.<br>
Is there anyone can explain it?</p>

ptrblck: <p>Are you manipulating <code>data</code> inplace in <code>model1</code> somehow, so that <code>model2</code> gets a different data distribution?</p>
Transform a tensor using a bijection?

Lam_Nguyen: <p>Hello,</p>
<p>I’d like to do the following using element wise operation :</p>
<p>I have a tensor a = [1,2,3,4]</p>
<p>And I have a bijection, that is represented by a dictionary for example</p>
<p>b = {1:4,2:1,3:2,4:3}</p>
<p>The transform would be :</p>
<p>f(a,b) = [4,1,2,3]</p>
<p>1 become a 4, 2 became a 1 and so forth…</p>
<p>Is this operation possible using only element wise operations?</p>

ptrblck: <p>You could try to create an index tensor from <code>b</code> and index <code>a</code> directly:</p>
<pre><code class="lang-python">a = torch.tensor([1, 2, 3, 4])
b = torch.tensor([3, 0, 1, 2])
a[b]
</code></pre>
<p>Note that the index starts at 0 in python/PyTorch.</p>
Does the backprop speed depends on how to assign values to the tensor?

zzoon91: <p>Hi,<br>
Recently I found that how I assign a value to a tensor affects the speed of backward()<br>
my test code is as follows<br>
Simply, I assign ones to (1000,1000) with different methods: assigning by index, stack() after gathering<br>
And then I calculated the sum of all values multiplied by 2</p>
<pre><code class="lang-auto">start = time.time()
from_empty = torch.empty(1000, 1000)
for i in range(1000):
    from_empty[i, :] = torch.ones(1000, requires_grad=True)
from_empty = 2*from_empty
from_empty = from_empty.sum()
from_empty.backward()
end = time.time()
print("assign by index", end-start)

start = time.time()
tmp = []
for i in range(1000):
    tmp.append(torch.ones(1000, requires_grad=True))
stacked = torch.stack(tmp, dim=1)
stacked = 2*stacked
stacked = stacked.sum()
stacked.backward()
end = time.time()
print("stack", end-start)
</code></pre>
<p>the result is</p>
<pre><code class="lang-auto">assign by index 1.5671675205230713
stack 0.08823513984680176
</code></pre>
<p>Why is the first one so slow?</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>You can try this package to see the generated graph to see why: <a href="https://github.com/szagoruyko/pytorchviz" rel="nofollow noopener">torchviz</a> (Change the size of the first dimension to 10 <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> )</p>
<p>As you will seem one created the output in one operation, while the other creates it by doing 1000 inplace operations.</p>

zzoon91: <p>Thanks for clear explanation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Assign module parameters while retaining existing graph dependencies

naikin: <p>I am trying to implement MAML <a href="https://arxiv.org/abs/1703.03400" rel="nofollow noopener">https://arxiv.org/abs/1703.03400</a> and I need to update the parameters of a module while keeping any existing graph dependency information related to the tensors holding the new parameter values. A very simplified example:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):

  def __init__(self):
    super().__init__()
    self.layer = nn.Linear(3, 1)

  def forward(self, x):
    return self.layer(x)


first_model = MyModule()
second_model = MyModule()


x = torch.normal(mean=torch.zeros([10,3]), std=torch.ones([10,3]))
y = torch.normal(mean=torch.zeros([10,1]), std=torch.ones([10,1]))
output = first_model(x)
loss = ((output - y)**2).mean()

grads = torch.autograd.grad([loss], first_model.parameters(), retain_graph=True, create_graph=True)

new_params = [p - learn_rate * grad for p, grad in zip(first_model.parameters(), grads)]

# TODO: Assing new_params to second_model.parameters() by keeping the dependency of new_params on first_model.parameters() ???
# Something like
# for p, v in zip(second_model.parameters(), new_params):
#   p.update(v)

new_x = torch.normal(mean=torch.zeros([10,3]), std=torch.ones([10,3]))
new_y = torch.normal(mean=torch.zeros([10,1]), std=torch.ones([10,1]))
new_output = second_model(new_x)
new_loss = ((new_output - new_y)**2).mean()

new_grads = torch.autograd.grad([new_loss], first_model.parameters())

for param, grad in zip(first_model.parameters(), new_grads):
  param.grad = grad

optimizer.step()
</code></pre>
<p>I am aware of a workaround which uses nn.Modules which take as input the desired parameters to use for the forward pass and compute the outputs using nn.functional instead of using ready-made modules such as nn.Linear, nn.Conv. However, this is not an option in my case since I have many large and complex models, which use ready-made modules and which I cannot modify.</p>

albanD: <p>Hi,</p>
<p>Does that mean that your <code>second_module</code> does not actually have any learnable parameters?</p>

naikin: <p>Yes. I only need to be able to update its parameters (keeping the existing graph dependencies) and do forward passes and backpropagation.</p>

albanD: <p>In that case, you can simply replace the fields in question:</p>
<pre><code class="lang-auto">second_model.linear.weight = new_weight
second_model.linear.biais = new_bias
</code></pre>
<p>You can get these locations automatically by using <code>named_parameters()</code> instead of <code>paramters()</code>.</p>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 3: invalid start byte

longing2627: <p>When i use the GPU on windows10 occurs a problem  as  follow:</p>
<p>this is my code:<br>
<strong>import torch as t</strong><br>
<strong>x=t.Tensor(3,2)</strong><br>
<strong>x=x.cuda()</strong></p>
<p>this issue:<br>
<strong>"UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xb0 in position 3: invalid start byte"</strong></p>
<p>why???</p>
How to normalize a feature map to use for grid_sample

John1231983: <p>I have a feature map size of <code>Bx3xDxHxW</code>, the range is from 0 to 1. I want to use it as flow field and input to grid_sample function. As the document said</p>
<blockquote>
<p><code>grid</code>  specifies the sampling pixel locations normalized by the  <code>input</code>  spatial dimensions. Therefore, it should have most values in the range of  <code>[-1, 1]</code> . For example, values  <code>x = -1, y = -1</code>  is the left-top pixel of  <code>input</code> , and values  <code>x = 1, y= 1</code>  is the right-bottom pixel of  <code>input</code></p>
</blockquote>
<p>So, How can I normalize the feature map to range [-1,1] to use as grid? Thanks</p>
<p>This is what I did</p>
<pre><code class="lang-auto">spatial_gds = torch.rand(1,2,3,3,3)
print (spatial_gds)

_, _, D, H, W = spatial_gds.size()
grids = [torch.cumsum(spatial_gds[:, i, ...], dim=i+2)
         for i in range(2)]
#print(grids)
grids_norm = torch.stack([(grid/dim)*2-1. for dim, grid in zip([D,H,W], grids)], dim=4)
print(grids_norm)
</code></pre>
How to get index of the last max using argmax()

anxu: <p>Hi, I know that argmax can work differently with cpu and gpu as follows,</p>
<p>import torch<br>
a=torch.tensor([[1,1,1,1,0],[1,1,0,0,0]])<br>
ac=torch.tensor([[1,1,1,1,0],[1,1,0,0,0]]).cuda()<br>
a.argmax(dim=1)<br>
=&gt;tensor([3, 1])<br>
ac.argmax(dim=1)<br>
=&gt;tensor([0, 0], device=‘cuda:0’)</p>
<p>But how to get the index of the last max with GPU? I need to get [3,1] instead of [0,0] as in the above example.</p>
How to do elementwise multiplication of two vectors?

Vaijenath_Biradar: <p>I have two vectors each of length n, I want element wise multiplication of two vectors. result will be a vector of length n.</p>

ptrblck: <p>You can simply use <code>a * b</code> or <code>torch.mul(a, b)</code>.</p>

Vaijenath_Biradar: <p>both gives dot product of two vectors. I want element wise multiplication.</p>

Vaijenath_Biradar: <p>ex. a = (a1, a2, … an) and b = (b1, b2, … bn)<br>
I want c = (a1*b1, a2*b2, … , an*bn)</p>

ptrblck: <p>Well this works in my case. Do you get a scalar running this?</p>
<pre><code>a = torch.randn(10)
b = torch.randn(10)
c = a * b
print(c.shape)</code></pre>

Vaijenath_Biradar: <p>It is giving a tensor of size 10. I must have made some other mistake. Thanks.</p>

111105: <p>Hello ,every one.I have a question. Does the * operator run in the GPU ?</p>

ptrblck: <p>If both tensors are stored on the GPU, then yes.<br>
The operations are written for CPU and GPU tensors, so as long as the data is pushed to the device, the GPU will be used.</p>

SupremePROC: <p>Its lucky for the pytorch users to have you always here</p>

111105: <p>thank you very much!</p>

cvogt: <p>Hi there,</p>
<p>Is there a rule of thumb regarding mixing CPU and GPU based variables? As in, if I have a tensor going through a function with a mix of CPU-stored (non-tensor) and GPU-stored variables, is it worth it to declare each variable as a torch tensor on the GPU at the beginning of the function call?</p>

ptrblck: <p>Do you mean plain Python variables by “CPU-stored (non-tensor) variables”, e.g. like <code>x = torch.randn(1) * 1.0</code>?</p>
<p>Generally you should transfer the data to the same device, if you are working with tensors.<br>
However, you won’t see much difference, if you are using scalars, as the wrapping will be done automatically.</p>

cvogt: <p>Yeah, exactly. I’m venturing a little off the path and wasn’t sure if it was reasonable or if the scalars would pull the tensor from the GPU for computation (outside of a module).</p>
<p>Thanks for taking the time! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Clarification for reproducibility and determinism

pmeier: <p>The <a href="https://pytorch.org/docs/stable/notes/randomness.html#" rel="noopener nofollow ugc">documentation</a> states:</p>
<blockquote>
<p>[I]n order to make computations deterministic on your specific problem on one specific platform and PyTorch release, there are a couple of steps to take. […] A number of operations have backwards that use <code>atomicAdd</code> , in particular […] many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions.</p>
</blockquote>
<p>Does this mean if I follow the guidelines, I will get deterministic results between individual runs, given the soft- and hardware does not change? Or does “no simple way” mean “no currently implemented way”?</p>
<p>I’m asking, because I do the following</p>
<pre data-code-wrap="python"><code class="lang-python">def make_reproducible(seed=0):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

make_reproducible()
cnn_training_loop()
</code></pre>
<p>and the loss deviates between runs after a few iterations. This deviation is at first only present on the 4th or 5th significant digit of the loss, but builds up over time. Since it is only this small at first, I suspect that it stems from some non-deterministic behavior rather than a bug in my implementation.</p>

albanD: <p>Hi,</p>
<p>You will get deterministic results as long as you don’t use the functions that are listed.<br>
Indeed for these functions, no deterministic alternative is implemented in pytorch.</p>
[Please Help] in-place operation in module

Lambda_X: <p>When I use the module:</p>
<pre><code class="lang-auto">class AngleLoss(nn.Module):
    def __init__(self, gamma=0):
        super(AngleLoss, self).__init__()
        self.gamma   = gamma
        self.it = 0
        self.LambdaMin = 5.0
        self.LambdaMax = 1500.0
        self.lamb = 1500.0

    def forward(self, input, target):
        self.it += 1
        cos_theta,phi_theta = input
        target = target.view(-1,1) #size=(B,1)

        index = cos_theta.data * 0.0 #size=(B,Classnum)
        index.scatter_(1,target.data.view(-1,1),1)
        index = index.byte()
        index = Variable(index)

        self.lamb = max(self.LambdaMin,self.LambdaMax/(1+0.1*self.it ))
        output = cos_theta * 1.0 #size=(B,Classnum)
        output[index] -= cos_theta[index]*(1.0+0)/(1+self.lamb)
        output[index] += phi_theta[index]*(1.0+0)/(1+self.lamb)

        logpt = F.log_softmax(output)
        logpt = logpt.gather(1,target)
        logpt = logpt.view(-1)
        pt = Variable(logpt.data.exp())

        loss = -1 * (1-pt)**self.gamma * logpt
        loss = loss.mean()

        return loss
</code></pre>
<p>I will get error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 2]], which is output 0 of MulBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>It seems that the error part is</p>
<pre><code class="lang-auto">        output[index] -= cos_theta[index]*(1.0+0)/(1+self.lamb)
        output[index] += phi_theta[index]*(1.0+0)/(1+self.lamb)
</code></pre>
<p>How to modify the code?</p>

Theocharis: <p>You can try:<br>
output[index] = output[index] -  cos_theta[index]<em>(1.0+0)/(1+self.lamb)<br>
output[index] = output[index] + phi_theta[index]</em>(1.0+0)/(1+self.lamb)</p>

Lambda_X: <p>Thank you! I have tried this method. It will not solve my problem.<br>
I finally solve it by adding .clone() to output.</p>
<pre><code class="lang-auto">output = cos_theta * 1.0 
output1 = output.clone()
output[index] = output1[index]-cos_theta[index]*(1.0+0)/(1+self.lamb)
output[index] = output1[index] phi_theta[index]*(1.0+0)/(1+self.lamb)
</code></pre>
<p>It seems that x = x+1 is not in_place operation, but x[index] = x[index]+1 is in_place operation.<br>
To be honest, I don’t quite understand why.</p>

albanD: <p>Hi,</p>
<p>When you do <code>x[index] = anything</code>, it is an inplace operation because you modify only part of x. The same thing happens if you do this with a python list for example.</p>
Problem Reading from Array And Training Model

Remote_Senser: <p>I have a directory of .npy files from which I am loading a model and training it for binary classification. The classes are of equal size. My loss doesn’t decrease after each epoch and the model keeps predicting one class. The data sets are balanced. I am not sure why this is happening. I am new to Pytorch and I spent a lot of time trouble shooting it. I tried doing it with a very small data set (200 in each class) but same problem persists.</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network,self).__init__()
        self.conv1=nn.Conv2d(in_channels=2, out_channels=32, kernel_size=3)
        self.conv2=nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3)
        self.conv3=nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3)
        self.fc1=nn.Linear(in_features=128*6*6, out_features=1000)
        self.fc2=nn.Linear(in_features=1000, out_features=100)
        self.out=nn.Linear(in_features=100, out_features=2)
    def forward(self,t):
        POOL_stride=2
        #Conv1
        t=F.relu(self.conv1(t))
        t=F.max_pool2d(t, kernel_size=2, stride=POOL_stride)
        #Conv2
        t=F.relu(self.conv2(t))
        t=F.max_pool2d(t, kernel_size=2, stride=POOL_stride)
        #Conv3
        t=F.relu(self.conv3(t))
        t=F.max_pool2d(t, kernel_size=2, stride=POOL_stride)
        # dense 1
        t=t.reshape(-1, 128*6*6)
        t=self.fc1(t)
        t=F.relu(t)
        #dense 2
        t=self.fc2(t)
        t=F.relu(t)
        t=self.out(t)
        return t

def npy_loader(path):
    sample = torch.from_numpy(np.load(path))
    return sample

criterion=nn.CrossEntropyLoss()
optimizer = optim.Adam(self.model.parameters(), lr=0.003)
model = Network()
trainset = datasets.DatasetFolder(
            root=train_dir,
            loader=npy_loader,
            extensions=['.npy']
        )
train_loader = torch.utils.data.DataLoader(
            trainset,
            batch_size=batch_size,
            shuffle=True,
        )
for epoch in range(epochs):
            running_loss = 0
            batches = 0
            for inputs, labels in train_loader:
                batches = batches+1
                inputs, labels = inputs.to(device), labels.to(device)
                optimizer.zero_grad()
                output = model(inputs)
                loss = criterion(output.squeeze(), labels.squeeze())
                loss.backward()
                optimizer.step()
                running_loss += loss.item()    
            print('Loss :{:.4f} Epoch[{}/{}]'.format(running_loss/batches, epoch, epochs))
'''</code></pre>

albanD: <p>Hi,</p>
<p>Do you know what are the mean and std of your data? That could cause issues with training.</p>
Partial weight Initialization in a network

Arvind_Subramaniam: <p>I have a model with binary weight values (1s and 0s only).</p>
<ol>
<li>
<p>Is there any way to initialize the weights such that only the ones are initialized?</p>
</li>
<li>
<p>In order to see how my sparse network performs, I would like to freeze all the zero weight values, so that they are maintained as zero during backprop. Is there any efficient way to do that?</p>
</li>
</ol>