akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Export to onnx has warning messages

username: ngap_wei_Tham
cooked: <p>Trying to export the encoder model based on <a href="https://github.com/nianticlabs/monodepth2" rel="nofollow noopener">this project</a>, since the onnx export function do works well with list output, I change the forward function of the <a href="https://github.com/nianticlabs/monodepth2/blob/master/networks/resnet_encoder.py" rel="nofollow noopener">encoder</a> from</p>
<pre><code class="lang-auto">def forward(self, input_image):
        self.features = []
        x = (input_image - 0.45) / 0.225
        x = self.encoder.conv1(x)
        x = self.encoder.bn1(x)
        self.features.append(self.encoder.relu(x))
        self.features.append(self.encoder.layer1(self.encoder.maxpool(self.features[-1])))
        self.features.append(self.encoder.layer2(self.features[-1]))
        self.features.append(self.encoder.layer3(self.features[-1]))
        self.features.append(self.encoder.layer4(self.features[-1]))

        return self.features
</code></pre>
<p>To</p>
<pre><code class="lang-auto">def forward(self, input_image):        
        x = (input_image - 0.45) / 0.225
        x = self.encoder.conv1(x)
        x = self.encoder.bn1(x)		
        output1 = self.encoder.relu(x)
        output2 = self.encoder.layer1(self.encoder.maxpool(output1))
        output3 = self.encoder.layer2(output2)
        output4 = self.encoder.layer3(output3)
        output5 = self.encoder.layer4(output4)
		
        output_tensor = torch.ones(2887680)
        #guess I do not need to call clone?
        output_tensor[0:1966080] = output1.flatten()
        output_tensor[1966080:2457600] = output2.flatten()
        output_tensor[2457600:2703360] = output3.flatten()
        output_tensor[2703360:2826240] = output4.flatten()
        output_tensor[2826240:] = output5.flatten()

        return output_tensor
</code></pre>
<p>It show me warning</p>
<p><strong>TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.</strong></p>
<p>Is this safe?</p>

username: ngap_wei_Tham
cooked: <p>Verify with the output values, this don’t work. What is the proper way to copy the data of different layers into a big tensor if you want to export it to onnx model?</p>

username: ngap_wei_Tham
cooked: <p>I found the solution, use cat.</p>
<p><code>return torch.cat((output1.flatten(), output2.flatten(), output3.flatten(), output4.flatten(), output5.flatten()))</code></p>
Loss normalize(or unify units) when using multiple loss

username: davidlee
cooked: <p>Hi guys,</p>
<p>If I want to use two combined losses, (L = loss1 + loss2)<br>
and suppose the decimal unit between two losses are different,<br>
for example, loss 1 is around 0.01~0.09 and loss2 is around 1~9,<br>
Do I need to unify the units of two losses?<br>
The loss is MSE.</p>
<p>Please give me any advice.</p>

username: charan_Vjy
cooked: <p>As a general rule of thumb, it is advisable to have the two losses on the same scale. Else, the model responsible for the higher loss will have all parameter updates thus rendering the other model almost close to useless. In your case, dividing loss2 by 100 should bring them to the same scale.</p>

username: davidlee
cooked: <p>Thank you for your answer.</p>
<p>also, Is it quite related to weighted loss?</p>

username: charan_Vjy
cooked: <p>Loss weighting is often done to normalize the scales.</p>
Append in a list from N lists based on predicted value

username: _joker
cooked: <pre><code class="lang-auto">all_lists = [list0, list1, list2, list3, list4, list5, list6, list7, list8, list9]

preds = classifier(xs)
_, predicted = torch.max(preds.data, 1)

index1 = predicted.item()
if predicted == ys:
    all_lists[index1].append((xs,ben))
</code></pre>
<p>but it only works for a batch_size=1 how can I do this for batch_size&gt;1, so that the data is appended into its corresponding list?</p>

username: jmandivarapu1
cooked: <p>Usually, <code>predicted</code> returns a tensor array. You can iterate through that array and check for each item. I assume you can convert it into <code>.numpy()</code> and use the parallel method to do it more quickly</p>
[SOLVED]Use two scheduler: lambdaLR & CosineAnnealingLR, but seems wierd

username: Roy_Paik
cooked: <p>I tried to use both adujusting lr methods, wanted to increase lr using <code>lambdaLR</code> from min_lr to max_lr(warm up), and then apply <code>CosineAnnealingLR</code>, but the learning rate seem not what I expected. it shows kind of ‘plateau’. Please kindly see below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e88790e5007e7f4c446d554a4d4278e28d32315e.png" alt="1" data-base62-sha1="xb3dLahAuaZmdWww4ynArGRDkLk" width="401" height="252"></p>
<p>I attach my code below:</p>
<pre><code class="lang-auto">n_epochs = 10
# n_iter = len(train_loader)
n_iter = 28

max_lr = 1e-3; min_lr = 1e-5;
base_lr = max_lr / (n_epochs*n_iter)
bias = min_lr * (1/base_lr)

lr_ls = []
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)

lambda1 = lambda x: bias+x
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda1)

j = 0
for epoch in range(n_epochs):
    for _ in range(n_iter):
        j+=1
        lr_ls.append(scheduler.get_last_lr()[0])
        optimizer.step()
        scheduler.step(j)

n_epochs = 50
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=200, eta_min=max_lr, last_epoch=j)
for epoch in range(n_epochs):
    for _ in range(n_iter):
        j+=1
        lr_ls.append(scheduler.get_last_lr()[0])
        optimizer.step()
        scheduler.step()

plt.plot(lr_ls)
</code></pre>
<p>Actually, I guess I don’t totally understand CosineAnnealingLR param, last_epoch maybe.</p>
<p>Would you help me out?</p>
<hr>
<p>I solved this problem by myself.<br>
TL;DR: need to declare optimizer again with the last lr of previous optimizer</p>
<pre><code class="lang-auto">base_lr = optimizer.param_groups[0]['lr']
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)
</code></pre>
<p>Please refer to the correction:</p>
<pre><code class="lang-auto">n_epochs = 10
# n_iter = len(train_loader)
n_iter = 28

max_lr = 1e-3; min_lr = 1e-5;
base_lr = max_lr / (n_epochs*n_iter)
bias = min_lr * (1/base_lr)

lr_ls = []
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)

lambda1 = lambda x: bias+x
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda1)

j = 0
for epoch in range(n_epochs):
    for _ in range(n_iter):
        j+=1
        lr_ls.append(scheduler.get_last_lr()[0])
        optimizer.step()
        scheduler.step(j)


##################################### this part
base_lr = optimizer.param_groups[0]['lr']
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)

##################################### 

n_epochs = 50
#scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=200, eta_min=0, last_epoch=-1)
scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=200, T_mult=2, eta_min=min_lr)
for epoch in range(n_epochs):
    for _ in range(n_iter):
        j+=1
        lr_ls.append(scheduler.get_last_lr()[0])
        optimizer.step()
        scheduler.step()

plt.plot(lr_ls, alpha=0.5)
plt.scatter(range(len(lr_ls)), lr_ls, s=0.3, c='r')
</code></pre>
How to install torch in python 2.7

username: rashid67
cooked: <p>I tried to install torch in python 2.7 but I can not install it. I use pycharm as an interpreter. This is my os : Windows-10-10.0.18362. Could anyone help me to install it? thank you.</p>

username: peterjc123
cooked: <p>There is no support for Python 2.7 on Windows.</p>
Random 0-1 matrix

username: xiexiaona
cooked: <p>I  want to generate a 0-1 random matrix with certain number but random indexes. However, i don’t know how to control the number of random matrix.</p>

username: ptrblck
cooked: <p>Could you post a small example how your matrix should look like?<br>
You can create a tensor with random values using e.g. this code:</p>
<pre><code class="lang-python">torch.empty(4, 4).random_(2)
</code></pre>

username: Eta_C
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Excuse me, now i have a tensor</p>
<pre><code class="lang-python">x = torch.rand(1, 2, 3)
</code></pre>
<p>and I want to generate another random mask tensor, which has <code>k</code> positive elements.</p>
<p>The following code could solve my problem, but it is too inefficient.</p>
<pre><code class="lang-python">n = torch.prod(torch.tensor(x.shape)).item()
index = torch.multinomial(torch.rand(n), k, replacement=False)
mask = (x * 0).flatten()
mask[index] = 1
mask = mask.reshape(*x.shape)
x.mul_(mask)
</code></pre>
<p>Have any suggestion?</p>

username: ptrblck
cooked: <p>Where is the bottleneck and how long does this approach take?</p>

username: Eta_C
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks for your reply,. I found that <code>mask = (x * 0).flatten()</code> wasted a lot of time. Then I modified my code. Now it looks like</p>
<pre><code class="lang-python">n = torch.prod(torch.tensor(x.shape)).item()
index = np.random.randint(0, n, k)
mask = np.zeros((n,), dtype=np.float32)
mask[index] = 1
mask = mask.reshape(*x.shape)
mask = torch.from_numpy(mask)
x.mul_(mask)
</code></pre>
<p>The improvement in speed is obvious. I am looking for a more faster implementation…</p>
Torchvision Mask RCNN - Max_GT_Instances

username: Deeply
cooked: <p>I am trying Mask RCNN based on the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">torchvision tutorial</a> and am getting some wired results. The problem is very simple, detect the person in the image such that each image has only one person (I am trying this as a proof of concept). In the test images that has one person; the model (trained for 300 epochs)  gave four labels and the corresponding masks where overlapped; the model, which should output one label and one mask (of the person), outputs four labels and four overlapped masks of the person; one showed full body silhouette; another showed lower body part silhouette; two more showed left and right body part silhouettes.</p>
<p>Is there a way to set <code>Max_GT_Instances</code> prior to training? If not, one can pick label with the highest score, but that will be done after training.</p>
<p><code>Max_GT_instances</code> is the maximum number of instances that can be detected in one image. If the number of instances in the images are limited, this can be set to maximum number of instances that can occur in the image. This helps in reduction of false positives and reduces the training time.</p>
Which kernel is used in PyTorch CNN?

username: DanyalTajik
cooked: <p>Which kernel is used in a Convolutional layer by default in pytorch?</p>

username: Krish
cooked: <p>The kernels used by default in Pytorch use the He initialisation from this paper:<br>
<a href="https://arxiv.org/abs/1502.01852" rel="nofollow noopener">Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification</a></p>

username: DanyalTajik
cooked: <p>Okay, thank you for providing the link, appreciated.</p>

username: jasg
cooked: <p>please explain more, more specifid</p>
Named Tensors + jit

username: googlebot
cooked: <p>Hello,</p>
<p>are there any plans to support jit compilation of code using named tensor operations?</p>
<p>IMHO, named tensors are potentially great, but are in a weird place: rewriting old code seems not worth it, especially with that list of unsupported subsystems. But even for new code, that is big and complex enough to require dimension reorderings, one has to choose between JIT and named tensors beforehand. To me, JIT seems more valuable (and even more so for simpler code fragments, that require no more that some occasional head/tail [un]squeeze).</p>

username: eellison
cooked: <p>Hi <a class="mention" href="/u/googlebot">@googlebot</a>,</p>
<p>We’ve discussed it but there are no immediate plans. However it wouldn’t be that hard to add. Maybe by the next release ?</p>
Building Pytorch wheel for ROCM version

username: ivallesp
cooked: <p>Hi,</p>
<p>I am compiling Pytorch inside a Docker container in order to make it work with ROCM. I have been successful until now, everyting works well. However, I would like to build the wheel when I run <code>.jenkins/pytorch/build.sh</code> but instead, it installs pytorch in the system version of Python and there’s no trace of the whl file in my pytorch directory. Any clue?</p>
<p>Rationale: <strong>I’d like to handle my dependencies with poetry</strong>, and I’d like to point to the wheel in the requirements…</p>
<p>Thanks</p>

username: tom
cooked: <p>Hi,</p>
<p>sorry for the late reply! You can use <code>python3 setup.py bdist_wheel</code> to build wheels.<br>
I put up a <a href="https://lernapparat.de/pytorch-rocm/" rel="nofollow noopener">recipe for building PyTorch on ROCm</a> on my blog.</p>
<p>Best regards</p>
<p>Thomas</p>
Indexing Tensor with diffrent indx

username: DrHB
cooked: <p>I would like to see if there is an elegant solution to this problem.</p>
<pre><code class="lang-auto">import torch
xb1 = torch.randn(64, 3, 128, 128)
xb2 = torch.randn(64, 3, 128, 128)

x1 = torch.randint(0, 128, (64,))
x2 = torch.randint(0, 128, (64,))
y1 = torch.randint(0, 128, (64,))
y2 = torch.randint(0, 128, (64,))

#this works fine
for i in range(len(x1)):
    xb2[i, :, x1[i]:x2[i], y1[i]:y2[i]] = xb1[i, :, x1[i]:x2[i], y1[i]:y2[i]]

#indexing directly thrus error
xb2[:, :, x1:x2, y1:y2] = xb1[:, :, x1:x2, y1:y2]
</code></pre>
<p>This wil thru error:</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-11-88c5a959f6b1&gt; in &lt;module&gt;
     13 x1.shape
     14 
---&gt; 15 xb1[:, :, x1:x2, y1:y2] = xb2[:, :, x1:x2, y1:y2]

TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>I was able to solve this problem using for loop. Was wondering if torch has better ways to perform this operation</p>

username: charan_Vjy
cooked: <p>As per the error trace, you cannot index using a tensor of size greater than 1. In other words, you cannot use a multidimensional array to index out elements. It is not very clear to me as to what you are trying to do. Can you print  out the for loop that you used? That would give a better idea of what you want to achieve.</p>

username: DrHB
cooked: <p>I have edited my question with the for loop example</p>

username: charan_Vjy
cooked: <p>x1[i], x2[i] are completely random values right. To the best of my knowledge, I don’t think there is an elegant way to do this. Anyways, i will try and let you know.</p>

username: DrHB
cooked: <p>Thank you =) yes, we are trying to implement cutmix… you randomly cut small image patch from one image and paste to another… For loop works fine but I was wondering if there is a way to speed this up.</p>
Optimization approch

username: Angry_potato
cooked: <p>Hi,<br>
I am implementing neural style transfer. So for optimising loss i update input image which is a torch.FloatTensor. Till now i was explicitly performing optimisation by multiplying gradient with learning rate as i cannot use the inbuilt optimizer such as Adam as the argument to optimizer should be iterable or dict of tensor. Is there any way to use inbuilt Adam for this task</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you have a single parameter, you can pass it to the optimizer in a list that contains a single element (this single parameter).</p>
Sign in and Log in not working properly

username: sidwa0
cooked: <p>I tried signing up with github, facebook and gmail account. The site should either end up logging me in or allow me to login with the same accounts. But it doesn’t work.</p>
<p>Only filling up the data fields work for signing up.</p>

username: ptrblck
cooked: <p>Could you give an example, what’s not working and how you solved it?<br>
Did the GitHub/FB/… account login not work at all and did you get any error message?</p>

username: sidwa0
cooked: <p>Ohh yes… Github login tries to ask for permission which I granted. Then I get redirected back to this site where I’m still not logged in.<br>
Google tries to log me in asks which google account to log in with. I get the same result as github once I choose the account.<br>
At this point I tried logging in with the datafields where the fields say that all the supplied values seem valid. But when I click sign-up, I get an error saying “email id or username or password is wrong”<br>
Facebook straight up gave me an error so I knew it didn’t work. But once I’m redirected to facebook I can’t press the back button to come back here for some reason.</p>
<p>As a list ditch attempt filling up the data fields with one of my gmail accounts worked.</p>
What does world size mean and what does set_device do

username: coincheung
cooked: <p>Hi,</p>
<p>I am learning about torch.nn.DistribuitedDataParallel. Compared with naive nn.DataParallel, it seems that there are many new concepts introduced to this function. After reading some programs, I got a bit confused.</p>
<p>In the beginning of the programs, there is always some <code>torch.cuda.set_device(arg.local_rank)</code>. What does this mean? Does this play same role as the system variable of <code>CUDA_VISIBLE_DEVICES=1,2,3,4</code> ?</p>
<p>Besides, there is also a line of <code>torch.distributed.init_process_group(world_size=4)</code>. What does <code>world_size</code> mean here? Does this mean the number of new processes created to train the model?</p>
<p>Lastly, could you provide a brief example on how I could train a model with 4 gpus on a single machine. I used to employ <code>nn.DataParallel</code>, and how I could do it with <code>DistributedDataParallel</code> ? Must I also wrap my dataset with a <code>torch.utils.data.distributed.DistributedSampler</code>?</p>

username: Yihui_Ren
cooked: <p>Hi <a class="mention" href="/u/coincheung">@coincheung</a></p>
<p>Do you still need help on this one?</p>

username: enjlee
cooked: <p>i need help on this, could you please explain?</p>

username: coincheung
cooked: <p>No thanks, I tried to figure it out myself.</p>

username: Yihui_Ren
cooked: <p>Hi <a class="mention" href="/u/enjlee">@enjlee</a> and <a class="mention" href="/u/coincheung">@coincheung</a><br>
Sorry for my late reply. I wrote a blog post sometime ago, that may shed some light on this issue. The blog post explains how to setup not only data parallel on single machine, but also separate a large model across multiple GPUs, and multiple replica on different nodes, and train them in a distributed fashion. In other words, each process group contains more than one GPU.<br>
It might be an overkill, but I think it may better explain the underlining mechanism of how pytorch distributed works.</p>
<p>Please let me know if you have any questions after reading the blog post.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <a href="https://yren.netlify.com/posts/pytorch_distributed_data_parallel_with_model_parallel_in_an_hpc_environment/" target="_blank" title="09:03AM - 12 December 2019" rel="nofollow noopener">My New Hugo Site – 12 Dec 19</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://yren.netlify.com/posts/pytorch_distributed_data_parallel_with_model_parallel_in_an_hpc_environment/" target="_blank" rel="nofollow noopener">Pytorch Distributed Data Parallel With Model Parallel in an HPC Environment</a></h3>

<p>Distributed Data Parallel with Model Parallel in an HPC environment Objective This tutorial is on :
   how to separate a model and put it on multiple GPUs.    how to train such model in a distributed data parallel fashion.    how to use...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to add constrain to nn.Parameter?

username: izuna385
cooked: <p>Currently I implement joint loss model and I can’t find out how to add constrain to nn.Parameters.</p>
<pre><code class="lang-auto">self.text_scoreweight = nn.Parameter(torch.FloatTensor([1]))
self.graph_scoreweight = nn.Parameter(torch.FloatTensor([1]))
...

score = self.text_scoreweight * textscore + self.graph_scoreweight * graphscore

</code></pre>
<p>I’d like to add constrains of<br>
0 &lt; self.text_scoreweight,  self.graph_scoreweight &lt; 1<br>
self.text_scoreweight + self.graph_scoreweight = 1</p>
<p>Could you know how to solve it?</p>
<p>P.S.<br>
I’ll now try this but if there would be a better approach and you could give me an advice, I’d appreciate it.</p>
<pre><code class="lang-auto">param1 = nn.Parameter(torch.FloatTensor([1]))
param2 = 1 - param1
Parameterlist.append(param1) #(for L2regularize parameter)
</code></pre>

username: Beth
cooked: <p>Did you manage to solve this problem?<br>
I have also been trying to figure out how to constrain a nn.Parameter for a different multitask problem.</p>
Efficientnet: output deep feature vector in addition to logit

username: moh_sen
cooked: <p>Currently, the network ouput is logit, which is self._fc(x), I want to access deep feature vector.</p>
<p>The deep feature is input of last dropout (Dropout-444), i.e., output of _avg_pooling(x), which has [-1, 1536] dimension.</p>
<p>We can change the source code to</p>
<pre><code class="lang-auto">  def forward(self, inputs):
      """ Calls extract_features to extract features, applies final linear layer, and returns logits. """
      bs = inputs.size(0)
      # Convolution layers
      x = self.extract_features(inputs)
 
      # Pooling and final linear layer
      x = self._avg_pooling(x)
      FV = x.view(bs, -1)
      h = self._dropout(FV)
      Logit = self._fc(h)
      return (FV, logit)
</code></pre>
<p>How can I get both feature vector and logit without changing the source code?</p>
<p>I tried following, but it show an error. I did similar thing with resnet-50 without error:</p>
<pre><code class="lang-auto">from efficientnet_pytorch import EfficientNet

class new_efficientnet(nn.Module):
  def __init__(self,origin_model):
    super(new_efficientnet, self).__init__()
    modules = list(origin_model.children())[:-2]
    self.new_model = nn.Sequential(*modules)
    self.FC = nn.Linear(1536, 300) # changing output size
         
    def forward(self, x):
    FV = self.new_model(x)
    FV = torch.flatten(FV, 1)
    logit = self.FC(FV)
    return (FV, logit)

efficientnet_original = EfficientNet.from_name('efficientnet-b3')
efficientnet = new_efficientnet(efficientnet_original)
efficientnet.cuda()
summary(efficientnet, ( 3, 300, 300), batch_size=-1, device='cuda')
</code></pre>
<p>I do not know how change the efficientnet-b3, to output deep feature vector without changing the source code.</p>

username: b02202050
cooked: <p>You can override the function without actually modifying the source code like the following.</p>
<pre><code class="lang-python">import efficientnet_pytorch
def my_efficientnet_forward(self, inputs):
	""" Calls extract_features to extract features, applies final linear layer, and returns logits. """
	bs = inputs.size(0)
	# Convolution layers
	x = self.extract_features(inputs)

	# Pooling and final linear layer
	x = self._avg_pooling(x)
	FV = x.view(bs, -1)
	h = self._dropout(FV)
	Logit = self._fc(h)
	return (FV, logit)

efficientnet_pytorch.EfficientNet.forward.__code__ = my_efficientnet_forward.__code__
</code></pre>
Controlling Epochs for distributed dataparallel

username: aradhyamathur
cooked: <p>While using dataparallel is it possible to run processes with different number of epochs. Say on machine one I would like to run the process for 20 epochs and sync with master however after 20 epochs I would want to run completely on master. Is there a workaround for this? I used one of the samples given in tutorials however in the event that epochs are varying the master waits to sync up though the process has completed on  another machine.</p>

username: mrshenli
cooked: <p>DDP instances need to all participate in the backward, otherwise it would hang. But there are work around. If you know that master would run say 100 epochs, and other nodes would run 80 epochs, you can call forward-backward on the DDP instance for 80 epochs. After that, you can delete the DDP instance, which will remove the DDP grad hooks accordingly. Then, you can run forward-backward on DDP.module (as DDP is deleted, you won’t be able to call DDP.module, but the program can still have a reference to the original local module separately) on master, and it will no longer trigger communications.</p>
nn.Parameter not copied to gpus in forward pass when using nn.DataParallel

username: grtzsohalf
cooked: <p>test.py:</p>
<pre><code class="lang-python">import sys
import torch
import torch.nn as nn
import torch.nn.functional as F


gpus = list(map(int, sys.argv[1].split(',')))


class Net(nn.Module):
    def __init__(self):
        super().__init__()

        self.alpha = nn.ParameterList()

        for i in range(4):
            self.alpha.append(nn.Parameter(1e-3*torch.randn(i+2, 5)))

        self.cnn = nn.Conv2d(1, 1, 1, 1, 1)


    def forward(self, x):
        print(self.alpha)
        print(self.cnn)
        return x


if __name__ == '__main__':
    net = Net().cuda()
    if len(gpus) &gt; 1:
        net = nn.DataParallel(net, device_ids=gpus)

    net(torch.rand(4, 5))
</code></pre>
<p>When I run <code>python3 test.py 0</code> (which means device_id = [0]), the output is</p>
<pre><code class="lang-auto">ParameterList(
    (0): Parameter containing: [torch.cuda.FloatTensor of size 2x5 (GPU 0)]
    (1): Parameter containing: [torch.cuda.FloatTensor of size 3x5 (GPU 0)]
    (2): Parameter containing: [torch.cuda.FloatTensor of size 4x5 (GPU 0)]
    (3): Parameter containing: [torch.cuda.FloatTensor of size 5x5 (GPU 0)]
)
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
</code></pre>
<p>However, when I run <code>python3 test.py 0,1</code> (which means device_id = [0, 1]), the output is</p>
<pre><code class="lang-auto">ParameterList()
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
ParameterList()
Conv2d(1, 1, kernel_size=(1, 1), stride=(1, 1), padding=(1, 1))
</code></pre>
<p>Only nn.Module is copied to gpus in forward pass.<br>
How can I use and train nn.Parameter just like nn.Module with nn.DataParallel?</p>
Using the output twice

username: sigma_x
cooked: <p>In the following setup all models are <code>nn.Modules</code>, not necessarily stored in different interfaces. What would be a correct way of processing the following setup, in case all models/modules must be updated:</p>
<pre><code class="lang-auto">o1 = m1(i)
o2=m2(o1)
o3=m3(o1)
loss = loss1(o2, t1) + loss2(o3,t2)
loss.backward()
</code></pre>
<p>Specifically, do I need to detach and clone o1 before using it in o3, or maybe before o2?</p>
<p>EDIT: the reason for asking is when I checked the weights after training, I got an impressions some weights in m3 are not updated/frozen. I checked: all weights have requires_grad=True</p>

username: b02202050
cooked: <p>Basically, <code>m3</code> will also have gradients after your <code>loss.backward()</code>.<br>
You can try the following code that I try to simulate your situation and you will got gradients on <code>m3</code>:</p>
<pre><code class="lang-python">m1 = torch.tensor(1.).requires_grad_()
m1 = torch.nn.Parameter(m1)
m2 = torch.tensor(2.).requires_grad_()
m2 = torch.nn.Parameter(m2)
m3 = torch.tensor(3.).requires_grad_()
m3 = torch.nn.Parameter(m3)
i = torch.tensor(4.)
o1 = m1 * i
o2 = m2 * o1
o3 = m3 * o1
t1 = 5.
t2 = 6.
loss = (o2 - t1) + (o3 - t2)
loss.backward()
print (m2.grad, m3.grad)
</code></pre>
<pre><code class="lang-bash">&gt;&gt;&gt; tensor(4.) tensor(4.)
</code></pre>
<p>But I can provide some opinion that might cause your zero gradient:</p>
<ol>
<li>you forget to use <code>optimizer.step()</code> or other way to update your weights.</li>
<li>your learning rate is zero</li>
<li>The gradient of m3 is really calculated to be zero in your algorithm (maybe it’s the problem of loss2 you use that produce a zero gradient).</li>
</ol>

username: sigma_x
cooked: <p>Thanks. I found the error, there was a bug in the loss computation loop.</p>
Sparse calucating of `cdist` for some specific pairs

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">import torch
h, w = 8, 8
c = 2
a = torch.randn(h*w, c)

S_mat = torch.cdist(a, a)

i_ind = torch.tensor([12, 14, 51]).long()
j_ind = torch.tensor([21, 60, 33]).long()

result = S_mat[i_ind, j_ind]
</code></pre>
<p>In fact, the tensor <code>a</code> is very large, suppose <code>h, w = 100, 100</code>,  so I do not want to calculate distance of all the pairs of points, instead only serveral selected indexes. Is there any way to perform it ?</p>

username: charan_Vjy
cooked: <p>Is there a pattern to the way you want to perform this? Or is it completely random?</p>

username: Zhaoyi-Yan
cooked: <p><code>a</code> is random and <code>i_ind</code> and <code>j_ind</code> are random too.</p>

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">import torch
import torch.nn.functional as F

ann_num = 3
h, w = 8, 8
c = 2
a = torch.randn(h*w, c)

S_mat = torch.cdist(a, a)

i_ind = torch.tensor([[12, 14, 51], [11, 6, 32], [1, 5, 17]]).long() # 3 ann_num
j_ind = torch.tensor([[21, 60, 33], [13, 16, 12], [7, 55, 29]]).long()
print('S_mat')
print(S_mat)

result = S_mat[i_ind, j_ind]

print('method 1 result:')
print(result)


# Only compute distances of selected set of paris.
input_sample_i = a[i_ind, :]
input_sample_j = a[j_ind, :]

S_mat_sample2 = F.pairwise_distance(input_sample_i.view(-1, c), input_sample_j.view(-1, c))
print('S_mat_smaple2')
print(S_mat_sample2)

</code></pre>
<p>I’ve figured out this problem.</p>
Prediction after logsoftmax & nllloss

username: snip3r77
cooked: <p>Hi,</p>
<p>at the last 2 lines how does it work? as in why do we need to exp?<br>
i thought we can just <code>model(img)</code></p>
<p>and we will obtain the result?<br>
because when we <code>model(img).size()</code> the output is <code>torch.Size([1, 10])</code> which is the 10 categories and the highest is the predicted result?</p>
<p>Thanks</p>
<blockquote>
<p>model = nn.Sequential(nn.Linear(784, 128),<br>
nn.ReLU(),<br>
nn.Linear(128, 64),<br>
nn.ReLU(),<br>
nn.Linear(64, 10),<br>
nn.LogSoftmax(dim=1))<br>
criterion = nn.NLLLoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.003)</p>
</blockquote>
<ul>
<li>snip some codes -</li>
</ul>
<blockquote>
<p>images, labels = next(iter(trainloader))</p>
<p>img = images[0].view(1, 784)</p>
<h5>Turn off gradients to speed up this part</h5>
<p>with torch.no_grad():</p>
<pre><code>logps = model(img) #input an image after re-shaped
</code></pre>
<h5>Output of the network are log-probabilities, need to take exponential for probabilities</h5>
<p>ps = torch.exp(logps) # ← same effect<br>
ps = F.softmax(logps,dim=0) &lt;–same effect<br>
view_classify(img.view(1, 28, 28), ps)</p>
</blockquote>

username: b02202050
cooked: <p>Yes, you can just <code>result_class = logps.argmax(dim=1)</code> to see the prediction class.<br>
But in real-world application, you have to set a confidence threshold to balance your FPR and TPR.<br>
Therefore, confidence(probability) is always needed when you make prediction.</p>
Fast dataloader for numpy arrays in memory?

username: rooks
cooked: <p>Let’s say I have a dataset which corresponds to some numpy array “data” already in memory.</p>
<p>If I use a DataLoader on top of this dataset to generate batches of size 1000, it seems the dataloader will call the method “getitem” 1000 times and cat the individual items together to create the batch.</p>
<p>However if I do it manually, I can directly access data[k : (k+1) * 1000], which is faster than getting the items individually.</p>
<p>Is there a way to customize the batch generation of the data loader so that I can optimize the batch generation in the case of a numpy array in memory?</p>
<p>thanks</p>

username: ptrblck
cooked: <p>Do you have any transformations in <code>__getitem__</code> or do you just index a single sample of your data?<br>
In the latter case, you could use your slice logic manually so that each call to <code>__getitem__</code> returns a complete batch. The <code>batch_size</code> of you <code>DataLoader</code> would have to be set to <code>1</code> in that case.<br>
However, this would always generate a batch of subsequent samples, if you don’t shuffle the data before.</p>
<p>Another obvious way would be to just work without a <code>DataLoader</code> and create the batches manually.</p>

username: Mdhvince
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , Sorry for replying here after more than 1y but it is related to what I am looking for, please tell me if I have to create another topic.<br>
My question is:<br>
Do you have an example of how can I manually generate a batch of images from multiple images (numpy arrays) already in memory and then apply transformation like To_tensor, Normalize…?  This will avoid to save all the images in a folder then load them back using ImageFolder.</p>
<p>Thanks<br>
Medhy</p>

username: ptrblck
cooked: <p>You could use <code>torchvision.transforms.ToPILImage()</code> to transform the numpy arrays to <code>PIL.Image</code>s, apply all image transformations, and convert it to a tensor via <code>ToTensor()</code>.<br>
The same approach is used in e.g. in the <a href="https://github.com/pytorch/vision/blob/f2f085bf099c4c31bc6f09c21844b2d57dabcb87/torchvision/datasets/mnist.py#L94">MNIST</a> dataset, since the data is directly into memory.</p>

username: Mdhvince
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. I will try this and get back to you if I have some problem. I was not thinking of using a custom dataset because it is for inference, not training, but indeed it could work.</p>
Crop Image in Pytorch Transform

username: Fikrat
cooked: <p>Hi To everyone.<br>
I have written code related Random Crop function which has to crop images as the mentioned size outlined.However, I need help to complete code according to height, width, left width, and right width.</p>
<p>In order to provide precise idea, I decide to share my code which is need to be concluded as alluded to reference sizes.</p>
<p>class RandomCrop(object):<br>
def <strong>init</strong>(self, crop_size):<br>
self.crop_size = (crop_size, crop_size)<br>
def <strong>call</strong>(self, sample):<br>
img = sample[“image”]<br>
gt = sample[“label”]<br>
# This line has to be completed.# →  i, j, h, w = transforms.RandomCrop.get_params(img, output_size=self.crop_size)</p>
<pre><code>    new_img = transforms.functional.crop(img, i, j, h, w)
    new_gt = transforms.functional.crop(gt, i, j, h, w)
    return {'image': new_img,
                'label': new_gt}
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a26fc50d89555fb9e213ebd7218a01d4ce8b380.png" alt="Cropped Image" data-base62-sha1="f948z2Jzr14kqM9gHdfbgGWE8YU" width="631" height="230"></p>
Torch.diag is not batch-paralleled

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">a = torch.randn(2, 5, 5)

result = []
for i in range(a.size(0)):
    result.append(torch.diag(a[i]))
result = torch.stack(result)

# I hope 
result  = torch.diag(a) # size of result is `2*5`
</code></pre>
Trying to follow RNN example in C++

username: Jesper.Lindberg
cooked: <p>I am trying to follow <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="nofollow noopener">NLP From Scratch: Classifying Names with a Character-Level RNN</a> in C++.<br>
First, I would like to know why no optimizer is used like <code>Adam()</code>? Also, why do we do the following for-loop to add the grad manually?</p>
<pre><code> for (auto const &amp; p : model_to_train.parameters()) {
     p.data().add_(p.grad().data() , -k_learning_rate_);
 }
</code></pre>
<p>Since this loops throws me the error: <code>cannot call variable_data() on undefined tensor (variable_data at ..\..\torch\csrc\autograd\variable.cpp:262)</code></p>
What is the dimension of W in LSTMCell?

username: zezhishao
cooked: <p>In LSRMCell, the pytorch computes the following function:</p>
<p>\begin{array}{ll} \<br>
i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{(t-1)} + b_{hi}) \<br>
f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{(t-1)} + b_{hf}) \<br>
g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{(t-1)} + b_{hg}) \<br>
o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{(t-1)} + b_{ho}) \<br>
c_t = f_t * c_{(t-1)} + i_t * g_t \<br>
h_t = o_t * \tanh(c_t) \<br>
\end{array}<br>
(Sorry I don’t konw how to convert it to LaTeX formulas)</p>
<p>I wonder what the dimension of W in LSTMCell is. e.g. W_{ii}, W_{hi}. I guess the output dimension of W<em>x(or W</em>h) is Hidden_Size * 1. Am I right?  I can’t prove it, can anyone help me?</p>
How could i calculate 2-norm in batches

username: zhishen_nie
cooked: <p>For example,there are 3 two-dimensional matrices. And it’s shape is [3,2,2]<br>
Of course, i could calculate their 2-norm by this way:</p>
<p>x = torch.randn(3, 2, 2)<br>
for i in range(x.shape[0]):<br>
print(torch.norm(x[i], 2))</p>
<p>and it’s output is<br>
tensor(2.2709)<br>
tensor(1.4141)<br>
tensor(2.7118)</p>
<p>This is a solution. Well, as we all know the for-looping is inefficientis. and could pytorch calculate their 2-norm without using for-looping?</p>

username: ptrblck
cooked: <p>You can pass multiple dimensions to the <code>norm</code> operation:</p>
<pre><code class="lang-python">x.norm(dim=[1, 2])
&gt; tensor([2.3191, 1.7451, 2.1392])
</code></pre>

username: zhishen_nie
cooked: <p>Thank you very much!</p>
How to use `torch.arange` to create a list of tensors?

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python"># Something like this:
#
import torch
starts = torch.tensor([2, 3, 5]).long()
ends = torch.tensor([10, 20, 32]).long()

a = torch.arange(starts, ends)
# a is a list full of tensors
</code></pre>

username: charan_Vjy
cooked: <p>As per the pytorch documentation, this is format for torch.arange <code>torch.arange( *start=0* ,  *end* ,  *step=1* ,  *out=None* ,  *dtype=None* ,  *layout=torch.strided* ,  *device=None* ,  *requires_grad=False* )</code>, where start,end and step are numbers.<br>
So you cannot give a tensor as input to arrange. What exactly do you want to do with starts and ends?</p>
Convert tensor into -1 to 1 range

username: cbd
cooked: <p>I want to do image restoration, restore_image = (orig_image - B)/C.<br>
orig_image is in the range (-1,1) but B is in the range of (0,1) and C is random output of LeakyRelU (-infinity, + infinity).</p>
<p>I want to convert B and C in the range (-1,1) for image restoration.</p>

username: charan_Vjy
cooked: <p>tensor = (tensor/torch.max(tensor) -0.5 ) * 2, should get the job done</p>

username: cbd
cooked: <p>Is this formula remain same if the negative value is higher than positive means minimum value is -30 and maximum value is 10?</p>

username: charan_Vjy
cooked: <p>I gave that solution assuming you were doing it for regular images. Anyways, for C, the following might be a better option.</p>
<blockquote>
<p>tensor = (tensor/torch.max(torch.abs(tensor))</p>
</blockquote>
<p>This itself should squish the values between - 1 and +1. For B (whose values are between 0 to 1), you can use the previous formula</p>

username: cbd
cooked: <p>I restored the image using, restore_image = (orig_image - B)/C.</p>
<p>Here C is output of training. The value comes in the range of (-2,10). After that i apply the transformation to get the value between (-1,1) as mentioned. After transformation,C values are in the range tensor([[[[-1.0349e-02,  4.3450e-02,  3.3275e-02,  …,  3.0451e-02, etc.</p>
<p>So when i divide (orig_image - B) by C, the output values are tensor([[[[ 4.2118e+02, -1.7778e+01, -1.5082e+02,  …, -2.7659e+02, -4.8812e+01, -1.6656e+01] etc.</p>
<p>Again I transformed it in the range of (-1,1) and I got the final value tensor([[[[ 2.9290e-04, -1.2363e-05, -1.0488e-04,  …, -1.9235e-04, -3.3945e-05, -1.1583e-05] etc.</p>
<p>This output restored image is constant with some gray level.<br>
Divide by fraction is creating issue. Can you throw some light to solve the issue?</p>

username: charan_Vjy
cooked: <p>You are displaying the image with the pixel values lying lie within (-1,1). Try converting the output back between (0-255) before displaying it.<br>
For values between -1 and 1</p>
<pre><code class="lang-auto">img = (tensor/2 + 0.5) * 255
img = img.detach().numpy()
img = img.astype(np.uint8)
</code></pre>
<p>Then display the image</p>
Multi task learning of a branched model

username: saraansh1999
cooked: <p>I have a model whose forward pass is as follows(pseudo code):</p>
<pre><code class="lang-auto">x = stem(x)
o1 = branch1(x)
o2 = branch2(x)
return o1, o2
</code></pre>
<p>I am using this for jointly training on two datasets as follows(pseudo code):</p>
<pre><code class="lang-auto">optimizer.zero_grad()

for (x1, y1), (x2, y2) in zip(data1, data2):
   o1, _ = model(x1)
   l1 = mseloss(o1, y1)

   _, o2 = model(x2)
   l2 = crossentropy(o2, y2)

   total_loss = w1*l1 + w2*l2
   total_loss.backward()
   optimizer.step()
</code></pre>
<p>I have a common optimizer with different learning rates for the stem and the two branches.<br>
This methodology should work according to <a href="https://discuss.pytorch.org/t/multiple-forward-passes-1-backward-pass-instead-of-packed-sequences/11648">11648</a><br>
But i tried changing the order of evaluation to as follows(pseudo code):</p>
<pre><code class="lang-auto">optimizer.zero_grad()

for (x1, y1), (x2, y2) in zip(data1, data2):
   
   _, o2 = model(x2)
   l2 = crossentropy(o2, y2)   

   o1, _ = model(x1)
   l1 = mseloss(o1, y1)

   total_loss = w1*l1 + w2*l2
   total_loss.backward()
   optimizer.step()
</code></pre>
<p>This led to different results than the previous version. Any idea why that might be happening?</p>
<p>P.S. All random seeds are fixed to maintain reproducibility across runs.</p>

username: ptrblck
cooked: <p>If your <code>stem</code> contains batchnorm layers, note that their running estimates will be updated during the forward pass. This should be visible during the evaluation step, although these stats should eventually converge, if the input samples are drawn from the same distribution and are sufficiently large.<br>
Also, dropout layers might be applied differently, if e.g. the batch size differs.</p>
<p>Could you explain a bit more, what issues you are seeing, and where the difference is visible in which magnitude?</p>
Augmentation lead a worst loss

username: bioinfo-dirty-jobs
cooked: <p>Dear All, I try to do some transfer learning on 3d  data (mri) using 3d resnet.<br>
I have tried to do augmentation on data loader  only in training using Elastic function and random rotation. I found a worst result  using augmentation than the use of  no augmented data. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85f496a186e4573c35362a465200f6699be62960.png" data-download-href="https://discuss.pytorch.org/uploads/default/85f496a186e4573c35362a465200f6699be62960" title="losses"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85f496a186e4573c35362a465200f6699be62960_2_690x345.png" alt="losses" data-base62-sha1="j71wYfWhYIf8r4M7d3j6sZRYg9y" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85f496a186e4573c35362a465200f6699be62960_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85f496a186e4573c35362a465200f6699be62960_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85f496a186e4573c35362a465200f6699be62960_2_1380x690.png 2x" data-dominant-color="FEFDFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">losses</span><span class="informations">2000×1000 83.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So can interpeter this as suggestion my  endpoint I wan to classify are not   equaly distribute? Thanks in advance for any help</p>

username: ptrblck
cooked: <p>If the posted curves represent the losses, I would recommend to make sure your model is training and the loss decreases.<br>
The augmented images might be “harder examples”, which might create a higher loss, but it seems that your model is not learning anything at all at the moment.</p>
How can I set batch size in Natural Language Processing

username: soojin99
cooked: <p>You see, almost every training uses batch.<br>
I can do understand how batch and each epoch works when<br>
processing images, because their size (pixels and channels) are same w each other.<br>
But in natural language, lengths of sentence are different and those of documents are too.<br>
and because of that, the size of output is different word to word in word2vec,<br>
or rnn families.</p>
<p>So how can I set the batch size?<br>
Batch means a pile of images in cnn or something,<br>
but does it mean some fixed-size words…?</p>
<p>Please help this noob. Thank you.</p>

username: ptrblck
cooked: <p>You could pack the input tensors with different length using e.g. <code>torch.nn.utils.rnn.pack_sequence</code> and later pad them to the longest sequence via <code>torch.nn.utils.rnn.pad_packed_sequence</code>.</p>
<p><a class="mention" href="/u/vdw">@vdw</a> has also posted an <a href="https://discuss.pytorch.org/t/using-batches-for-seq2seq-models/55205">approach</a> without padding, where the input sequences are sorted to avoid padding.</p>
Libtorch named_parameters() not work?

username: FantasyJXF
cooked: <p>As the <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">libtorch guide</a> says, <code>named_parameters()</code> is used to print model parameters, and the return value should be <code>OrderedDict&lt;std::string, Tensor&gt; named_parameters(bool recurse = true) const</code>,but it went into error:</p>
<pre><code class="lang-auto">no member named 'key' in 'torch::jit::script::Named&lt;at::Tensor&gt;'
</code></pre>
<p>My input is</p>
<pre><code class="lang-auto">for (const auto&amp; pair : net.named_parameters()) {
  std::cout &lt;&lt; pair.key() &lt;&lt; ": " &lt;&lt; pair.value() &lt;&lt; std::endl;
}
</code></pre>

username: junjihashimoto
cooked: <p>Hi, module of script is not dict.<br>
Try <code>pair.name</code><br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/v1.4.0/torch/csrc/jit/script/module.h#L51-L55" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/v1.4.0/torch/csrc/jit/script/module.h#L51-L55" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/v1.4.0/torch/csrc/jit/script/module.h#L51-L55</a></h4>
<pre class="onebox"><code class="lang-h"><ol class="start lines" start="51" style="counter-reset: li-counter 50 ;">
<li>template &lt;typename T&gt;</li>
<li>struct Named {</li>
<li>  std::string name;</li>
<li>  T value;</li>
<li>};</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>