akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to get weights indicating the importance of each words in a sentence corresponding to the label

Shanto_Islam: <p>I am currently working on sentiment analyses using bert’s pooled output or classification task model. I would want to see the importance weight of each word in a sentence which is leading me to the sentiment.</p>
<p>More formally, I want a <strong>batch_size x max_length</strong> size tensor, where each value of the matrix is the weight indicating the importance of that word on the <em>ith</em> row.</p>
<p>I’ve seen many suggesting to look at the 11th layer of bert but I am unable to find anything like that. Any help regarding this matter, please?</p>
Implementing local linear regression

haalim: <p>I am trying to implement a local linear regression problem for each output. I have written this piece of code. But this code is not working.</p>
<p>Please help me how can I fix this.</p>
<pre><code class="lang-auto">    def __init__(self):             
        super(Net, self).__init__()   
        input_dim = 9
        self.layers = nn.ModuleList()       
        for ii in range(image_size):
            self.layers.append(nn.Linear(input_dim, 1))
        

    def forward(self, xt, c):       
        d = xt.shape[0]
        xt = xt.view(xt.size(0), -1)            
        out = Variable(torch.zeros(d, image_size), requires_grad=False)
 
        ii = 0
        for layer in self.layers[:-1]:             
                xx = xt[:,indices[ii]]
                le_xx = xx.shape[1]                
                if le_xx &lt; 9:                  
                   xtt = torch.zeros(d, 9)
                   xtt[:, :le_xx] = xx
                else:
                   xtt = xx
                txt = layer(xtt)
                out[:, ii] = txt.squeeze()
                ii = ii+1
            
        xt = out
        xt = xt.view(d,3,64,64)
        return xt
</code></pre>
How to select channels?

smani: <p>I have 8 images (say X) represented as 8x3x100x100, where 3 is the number of channels and 100x100 is the size of each image. I have a vector “index” - indicates which channel to be taken from each image, e.g., “index” = [1,1,2,0,…]. After selecting the appropriate channels from each image the output should have a size of 8x1x100x100. How can I do it?</p>
<p>I tried X.index_select(1, index), but it gives me 8x8x100x100 as the output.</p>

Megh_Bhalerao: <p>You can just try accessing the particular channel by <code>X[:,i,:,:]</code></p>
In-place operation on chunked tensor does not increase tensor version

milad_aghajohari: <p>Hi All!</p>
<p>I observed something in Pytorch that I think is not right. According to <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/autograd.html</a><br>
after any change in the values of  a tensor its version counter should increase. That is needed so the autograd engine can check the correctness of calculating the gradients as stated in the link.</p>
<p>However, I am confused why Pytorch is not increasing the version number of the <code>x</code> tensor in this snippet:</p>
<pre><code class="lang-auto">import torch
x = torch.randn(100)
print('x sum before manipulation', torch.sum(x))
print('x version before manipulation', x._version)
z1, z2 = torch.chunk(x, 2)
print('z1 version after manipulation', z1._version)
z1 /= 10 # an inplace operation, note that z1 = z1/10 is not inplace so it should be z1 /= 10 here to show the problem
print('x sum after manipulation', torch.sum(x))
print('x version after manipulation', x._version)
print('z1 version after manipulation', z1._version)
</code></pre>
<p>a test run results in:</p>
<pre><code class="lang-auto">x sum before manipulation tensor(14.4694)
x version before manipulation 0
z1 version after manipulation 0
x sum after manipulation tensor(15.3440)
x version after manipulation 0
z1 version after manipulation 1
</code></pre>
<p>However, I think the x’s version should also increase in this situation as the content of it is changed.</p>
<p>I also have this code in Colab if you want to test it yourself <a href="https://colab.research.google.com/drive/1c2d_ro14-QPZ94GP6LP-QGgFfBq8GiAx?usp=sharing" rel="nofollow noopener">here</a></p>
<p>I am wondering if it is a bug.</p>
DistributedSampler question

mars747: <p>Hi,</p>
<p>I tried to imitate the distributed sampler (I needed to return python dicts instead of indices). I’m using a single machine, multi-gpu multiworker setup.</p>
<p>I get a list of dicts I want to return and return the sublist drawn with indices == dist.get_rank() modulo dist.get_world_size(). I believe this is what’s done for DistributedSampler. I added some print statements but I only see printouts from dist.get_rank() == 0 but no other. Is that correct? Then am I only using 1/8 of all data with 8 gpus since my dist.get_world_size() is 8.</p>
<p>I also modified distributed.py where DistributedSampler is from the pytorch source code and print self.rank, and I also only see rank 0. When do the other ranks come in or get used?</p>
Simple test, I really didn't get how autograd calculate these grads

okay_All_Right: <p>import torch<br>
import torch.nn as nn</p>
<p>dtype = torch.float<br>
device = torch.device(“cpu”)</p>
<h1>N is batch size; D_in is input dimension;</h1>
<h1>H is hidden dimension; D_out is output dimension.</h1>
<p>N, D_in, H, D_out = 1, 1, 1, 1</p>
<h1>Create random Tensors to hold input and outputs.</h1>
<p>x = torch.randn(N, D_in, device=device, dtype=dtype)</p>
<p>x[0,0]=1000</p>
<h1>Create random Tensors for weights.</h1>
<p>w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)<br>
nn.init.uniform_(w1,0.5,0.5)<br>
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)<br>
nn.init.uniform_(w2,1,1)<br>
w3 = torch.randn(H, 1, device=device, dtype=dtype, requires_grad=True)<br>
nn.init.uniform_(w3,1,1)</p>
<p>w1_ = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)<br>
nn.init.uniform_(w1_,1,1)<br>
w2_ = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)<br>
nn.init.uniform_(w2_,1,1)<br>
w3_ = torch.randn(H, 1, device=device, dtype=dtype, requires_grad=True)<br>
nn.init.uniform_(w3_,1,1)</p>
<p>learning_rate = 1e-6</p>
<p>for t in range(500):<br>
# Forward pass: compute predicted y using operations on Tensors;</p>
<pre><code>p=torch.tanh(x.mm(w1))
y_pred = p.mm(w2)
y_pred-=y_pred.mean()
m=x.mm(w1).mm(w3)
y_pred+=m
y=y_pred.max(1, keepdim=True)[1]
y0=y_pred.gather(1,y)

y_pred_ = torch.tanh(x.mm(w1_)).mm(w2_)
y_pred_ -=y_pred_.mean()
y_pred_+=x.mm(w1_).mm(w3_)
y_=y_pred_.max(1, keepdim=True)[1]
y1=y_pred_.gather(1,y_)

# Compute and print loss using operations on Tensors.

loss = (y0 - y1).pow(2).sum()
if t % 100 == 99:
    print(t, loss.item())

# Use autograd to compute the backward pass. 

loss.backward()

# Manually update weights using gradient descent. Wrap in torch.no_grad()

with torch.no_grad():

    w1 -= learning_rate * w1.grad
    w2 -= learning_rate * w2.grad
    w3 -= learning_rate * w3.grad

    w1_ -= learning_rate * w1_.grad
    w2_ -= learning_rate * w2_.grad
    w3_ -= learning_rate * w3_.grad

    # Manually zero the gradients after updating weights

    w1.grad.zero_()
    w2.grad.zero_()
    w3.grad.zero_()
</code></pre>
<p>print(w1)<br>
print(w2)<br>
print(w3)</p>
<p>print(“tgt”)</p>
<p>print(w1_)<br>
print(w2_)<br>
print(w3_)</p>
<p>I tried with my calculator to compare my numbers results but they are different from what happens here, (I made a backprop by my hand, but the results are different). can someone tell how this happens, since I have two model networks, how did he manage to calculate the grads, they always different from my calculation. the worst is that it is nan why ?</p>
Calculate Entropy for each class to measure uncertainty

Ka_Hina: <p>I am trying to calculate Entropy per class for an image classification task to measure uncertainty on pytorch,using the MC Dropout method and the solution proposed in this link<br>
<a href="https://stackoverflow.com/questions/63285197/measuring-uncertainty-using-mc-dropout-on-pytorch" rel="nofollow noopener">Measuring uncertainty using MC Dropout </a></p>
<p>First,I have calculated the mean of each class per batch across different forward passes (class_mean_batch) and then for all the testloader (classes_mean) and then did some transformations to get (total_mean) to use it for calculating Entropy as shown in the code below</p>
<pre><code class="lang-auto">def mcdropout_test(batch_size,n_classes,model,T):

    #set non-dropout layers to eval mode
    model.eval()

    #set dropout layers to train mode
    enable_dropout(model)
    
    softmax = nn.Softmax(dim=1)
    classes_mean = []
       
    for images,labels in testloader:
        images = images.to(device)
        labels = labels.to(device)
        classes_mean_batch = []
            
        with torch.no_grad():
          output_list = []
          
          #getting outputs for T forward passes
          for i in range(T):
            output = model(images)
            output = softmax(output)
            output_list.append(torch.unsqueeze(output, 0))
            
        
        concat_output = torch.cat(output_list,0)
        
        # getting mean of each class per batch across multiple MCD forward passes
        for i in range (n_classes):
          mean = torch.mean(concat_output[:, : , i])
          classes_mean_batch.append(mean)
        
        # getting mean of each class for the testloader
        classes_mean.append(torch.stack(classes_mean_batch))
        

    total_mean = []
    concat_classes_mean = torch.stack(classes_mean)

    for i in range (n_classes):
      concat_classes = concat_classes_mean[: , i]
      total_mean.append(concat_classes)


    total_mean = torch.stack(total_mean)
    total_mean = np.asarray(total_mean.cpu())
 
    epsilon = sys.float_info.min
    # Calculating entropy across multiple MCD forward passes 
    entropy = (- np.sum(total_mean*np.log(total_mean + epsilon), axis=-1)).tolist()
    entropy = [i*(1/T) for i in entropy]
    for i in range(n_classes):
      print(f'entropy of class {i+1} is {entropy[i]:.4f}')
</code></pre>
<p>Can anyone please correct or confirm the implementation i have used to calculate Entropy of each class.</p>
Memory leakage with pyTorch DQN tutorial

freeman0301: <p>Hi,<br>
I tried to run the following example<br>
<a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a><br>
I keep getting unexpected termination (either on google co-lab, juypter notebook, local) after only 7 episodes. I looked further and I highly suspect this is due to high memory leakage (i.e. I checked and this tutorial can cause 3GB memory usage for just 7 episodes?) I also tried using my GPU (GeForce GTX1650 with CUDA 11.0) but no luck</p>
<p>Here is the list of what i installed:<br>
OS: Windows 10<br>
python: 3.7.3<br>
torch: 1.4.0+cu92<br>
torchvision: 0.5.0<br>
gym: 0.17.2<br>
gym-minigrid: 1.0.1<br>
numpy: 1.18.5</p>
<p>Has anyone encountered this? Sorry if this question is too trivial as I am still new to pyTorch community.  I am currently stuck in learning this tutorial. Hope someone can give me a hand :*(</p>

Alon: <p>It’s much more likely that you don’t have enough memory in your GPU to run a batch size of 128. Try changing to <code>BATCH_SIZE = 10</code> and see it it works. Then empirically find the highest batch size that your system can support.</p>

freeman0301: <p>Thanks <a class="mention" href="/u/alon">@Alon</a>. Yes you are right. Indeed it is the limitation of my GPU, GTX 1650, which does not allow to run Deep learning.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://aws1.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="16" height="16">
      <a href="https://forums.developer.nvidia.com/t/does-gtx-1050ti-or-1650-for-notebook-support-tensorflow-gpu/77384" target="_blank" rel="nofollow noopener" title="09:43AM - 02 July 2019">NVIDIA Developer Forums – 2 Jul 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:106/18;"><img src="https://aws1.discourse-cdn.com/nvidia/original/2X/8/8f17cc8f1a724d6ecea8a197a267ec8a05ef1490.png" class="thumbnail" width="106" height="18"></div>

<h3><a href="https://forums.developer.nvidia.com/t/does-gtx-1050ti-or-1650-for-notebook-support-tensorflow-gpu/77384" target="_blank" rel="nofollow noopener">Does GTX 1050ti or 1650 for notebook support tensorflow-gpu</a></h3>

<p>I am planning to buy a laptop with Nvidia GeForce GTX 1050Ti or 1650 GPU for Deep Learning with tensorflow-gpu but in the supported list of CUDA enabled devices both of them are not listed.  Some people in the NVIDIA community say that these cards...</p>

  <p><span class="label1">Reading time: 1 mins 🕑</span>
    <span class="label2">Likes: 2 ❤</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I guess I will have to buy another one…</p>
Data Parallel on single GPU

Yi_Zhang: <p>Hi,</p>
<p>I want to use data parallel to train my model on single GPU. I have tried to run multiple processes on same GPU by following the example of Pytorch DISTRIBUTED DATA PARALLEL, and pass the same device_id. However, it doesn’t have speedup. I searched in the forum, seems the models on same GPU might have to wait for each other. Is there a better way to implement parallel training on single GPU? Thanks.</p>
PyTorch model can forward but onnx export tell me RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

jinfagang: <p>The error is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "export_onnx.py", line 61, in &lt;module&gt;
    output_names=['output'], enable_onnx_checker=False)  # output_names=['output']
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/__init__.py", line 208, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 530, in _export
    fixed_batch_size=fixed_batch_size)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 409, in _model_to_graph
    _export_onnx_opset_version)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

</code></pre>
<p>I am tracing a model to onnx, the question is the model can foward in GPU mode, but when trace it tell me that ONNX find a node in CPU, how could that possible?</p>
<p>I mean if there were one node in CPU then how could my model forward???</p>
<p>Any body knows why?</p>
Can't include c++ headers when compiling for pytorch

MLAI: <p>I hope the following question is not too basic, but I am somewhat new to C++ in general and am a bit lost right now.<br>
I try to compile c++ code to use it as a module in PyTorch. The example from the CUSTOM C++ AND CUDA EXTENSIONS tutorial works just fine using OSX and clang. However, when I try to import headers from another folder using <span class="hashtag">#include</span> &lt;includepath/Auxiliary/Array.h&gt; with the following setup.py file</p>
<pre><code class="lang-auto">from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name='rotmat_cpp',
      ext_modules=[cpp_extension.CppExtension('function_cpp', ['function.cpp'],
                        include_dirs=['includepath/Basis', 'includepath/Auxiliary'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})
</code></pre>
<p>and folder structure</p>
<pre><code class="lang-auto">extensions
├── includepath/
│   ├── Auxiliary/
│   │   ├── Array.h
│   │   └── ...
│   └── Basis/
├── setup.py
├── function.cpp
</code></pre>
<p>I get the following error<br>
<code>error: 'includepath/Auxiliary/Array.h' file not found with &lt;angled&gt; include; use "quotes" instead</code>.<br>
It seems to include the files successfully if I change all <code>&lt;...&gt;</code> to <code>"..."</code>as suggested, but then I get problems with a namespace I am using and get an error after loading the compiled package into python afterwards.<br>
I tried the same on my Ubuntu machine with gcc and get a slightly different error.</p>
<pre><code class="lang-auto">/media/Projects/extensions/function.cpp:1:10: fatal error: includepath/Auxiliary/Array.h: No such file or directory
 #include &lt;includepath/Auxiliary/Array.h&gt;
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
</code></pre>
<p>Trying to compile using JIT with the <em>extra_include_paths</em> options gives the same error.<br>
How can I include folders to the compilers search path such that <span class="hashtag">#include</span> &lt;includepath/…&gt; works?</p>
MNIST normalizing and scaling the dataset at the same time

MauroCE: <p>I’ve looked everywhere but couldn’t quite find what I want. Basically the MNIST dataset has images with pixel values in the range <code>[0, 255]</code>. People say that in general, it is good to do the following:</p>
<ul>
<li>Scale the data to the <code>[0,1]</code> range.</li>
<li>Normalize the data to have zero mean and unit standard deviation <code>(data - mean) / std</code>.</li>
</ul>
<p>Unfortunately, no one ever shows how to do both of these things. They all subtract a mean of <code>0.1307</code> and divide by a standard deviation of <code>0.3081</code>. These values are basically the mean and the standard deviation of the dataset divided by 255:</p>
<pre><code class="lang-auto">from torchvision.datasets import MNIST        
import torchvision.transforms as transforms 

trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True)
print('Min Pixel Value: {} \nMax Pixel Value: {}'.format(trainset.data.min(), trainset.data.max()))
print('Mean Pixel Value {} \nPixel Values Std: {}'.format(trainset.data.float().mean(), trainset.data.float().std()))
print('Scaled Mean Pixel Value {} \nScaled Pixel Values Std: {}'.format(trainset.data.float().mean() / 255, trainset.data.float().std() / 255))
</code></pre>
<p>This outputs the following</p>
<pre><code class="lang-auto">Min Pixel Value: 0 
Max Pixel Value: 255
Mean Pixel Value 33.31002426147461 
Pixel Values Std: 78.56748962402344
Scaled Mean: 0.13062754273414612 
Scaled Std: 0.30810779333114624
</code></pre>
<p>However clearly this does none of the above! The resulting data 1) will not be between <code>[0, 1]</code> and will not have mean <code>0</code> or std <code>1</code>. In fact this is what we are doing:</p>
<p><code>[data - (mean / 255)] / (std / 255)</code></p>
<p>which is <strong>very</strong> different from this</p>
<p><code>[(scaled_data) - (mean/255)] / (std/255)</code></p>
<p>where <code>scaled_data</code> is just <code>data / 255</code>.</p>

Henry_Chibueze: <p>Well u don’t need to always normalize ur image data that way.</p>
<p>What I usually do is to set mine to be within range of [-1, 1] and here’s how I do it:<br>
I divide all by 255 so it fall within the range of 0 and 1 and then I use a mean and sd of 0.5 like this (X - 0.5)/ 0.5 to normalize it the the target range. Similar too u can just normalize it directly to range[-1, 1] by using X = (X - 127.5)/127.5.<br>
If u apply this to an image and still try to load it it will give u the same exact visual representation as when it wasn’t normalized so it’s optional to reverse the normalization at output.</p>
<p>Hope this helps you.</p>
How to vectorize the code

qiminchen: <p>I was trying to compute normal from depth and currently, I’m using <code>for loop</code> to compute normal of each pixel. I can’t think of a way to compute normal of each pixel at the same time because a window of each pixel is used to compute the normal of that pixel. Does anyone have any ideas on how to vectorize it? XYZ is the back-projected 3d coordinate from depth.</p>
<p>Using <code>for loop</code> is extremely slow, it takes roughly <code>13s</code> to compute a normal map from a 240x320 depth map.</p>
<pre><code class="lang-auto">def DepthToNormal(Z, win_sz=15):

    def cal_patch(xy, z, i, j, sz, dep_th=0.1):
        cent_d = z[0, i+sz//2, j+sz//2]
        val_mask = (torch.abs(z[0, i:i+sz, j:j+sz] - cent_d) &lt; dep_th * cent_d) &amp; (
                z[0, i:i+sz, j:j+sz] &gt; 0).unsqueeze(0).repeat(3, 1, 1)
        if val_mask.sum() &lt; 10:
            return torch.tensor([0., 0., 0.])
        comb_patch = torch.cat((xy[:, i:i+sz, j:j+sz], z[:, i:i+sz, j:j+sz]), dim=0)
        A = comb_patch[val_mask].view(3, -1).permute(1, 0)
        A_t = A.permute(1, 0)
        A_tA = torch.mm(A_t, A)
        try:
            n = torch.sum(torch.mm(A_tA.inverse(), A_t), dim=1)
        except:
            n = torch.tensor([0., 0., 0.])
        return n

    intrinsics = torch.tensor([577.591, 318.905, 578.73, 242.684]).cuda()
    batch, _, height, width = Z.shape
    yy, xx = torch.meshgrid(torch.tensor([i for i in range(height)]),
                            torch.tensor([i for i in range(width)]))
    xx = xx.unsqueeze(dim=0).unsqueeze(dim=0).repeat(batch, 1, 1, 1).cuda()
    yy = yy.unsqueeze(dim=0).unsqueeze(dim=0).repeat(batch, 1, 1, 1).cuda()
    XY = torch.cat((Z * (xx - intrinsics[1]) / intrinsics[0],
                    Z * (yy - intrinsics[3]) / intrinsics[2]), dim=1)
    normal = torch.zeros((batch, 3, height, width))
    XY = F.pad(XY, (win_sz // 2, win_sz // 2, win_sz // 2, win_sz // 2), mode='reflect')
    Z = F.pad(Z, (win_sz // 2, win_sz // 2, win_sz // 2, win_sz // 2), mode='reflect')
    for b in range(batch):
        for i in range(height):
            for j in range(width):
                normal[b, :, i, j] = cal_patch(XY[b], Z[b], i, j, win_sz)
    return normal
</code></pre>

Henry_Chibueze: <p>Well I actually see no other way to go around the for loop coz a recursive approach will even take more time and memory what I suggest is that u try to run this process in parallel on the processor u are using or run it on a gpu if available (u can use a gpu computing cloud service)</p>
Using LSTMs with data with multiple channels

Chiara: <p>Hi there!</p>
<p>I have developed a network composed of a few stacked LSTMs that analyses waveforms and tells me whether they contain a signal or just noise.<br>
If I wanted to input a waveform as seen by three detectors(channels) (i.e. three different waveforms going into the network at the same time) how would i do this?<br>
Would using convolutional LSTMs help with this?</p>

googlebot: <p>If your unbatched input is 2d, i.e. (time,3), a module can either cut it as (1,3) pieces - that’s RNNs (that are memory accumulation models) or MLP modules (function of 3d vectors), or as (kernel_size,3) - that’s conv1d, appropriate for detecting temporal patterns. ConvLSTM needs extra spatial dimension I think.</p>
ASGD Optimizer Has A Bug?

zhang: <p>Hi, everyone,</p>
<p>I am referring the ASGD optimizer source code,  link<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/optim/asgd.html#ASGD" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/optim/asgd.html#ASGD</a></p>
<p>and confused with the following segment</p>
<pre><code>            if state['mu'] != 1:
                state['ax'].add_(p.sub(state['ax']).mul(state['mu']))
            else:
                state['ax'].copy_(p)
</code></pre>
<p>It seems that the state[“ax”] is not used in the whole code.  It just computes this variable and this variable is not used for update parameters.</p>
<p>Is it a bug?</p>

ptrblck: <p>It seems <code>ax</code> was never used in this and all previous implementations (it was returned at one points, but I don’t know if and how it was further used).<br>
CC <a class="mention" href="/u/vincentqb">@vincentqb</a> do you know, if and where this state is used?</p>

zhang: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>From the ASGD implementation, it has the same effect as SGD .   And the state[“ax”] is not used for updating parameters.   Suggesting a modification in the next version.</p>

ptrblck: <p>I don’t quite understand the explanation. Where should <code>ax</code> be used then? Do you mean it doesn’t need to be calculated and is just there for “debugging” purposes?</p>

zhang: <p>I am not sure enough. I cannot download the corresponding paper and have no idea about how ASGD updates the parameters. Generally, the state[‘ax’] should be used for modifying the p.grad (gradient) as Adagrad or Adam does.</p>
Conv & Linear parameter init. in MSE loss context

Jean-Eric_Campagne: <p>Dear Expert,<br>
I’m using a CNN network with Conv2d and Linear layers in a regression problem where I use the MSE as loss function.</p>
<p>For both the Conv and the Linear layers I’m using the xavier_uniform initialisation, and Adam as minimizer.</p>
<p>Does the initialisation strategy is the right one in the context of MSE loss (i.e if I am right the Xavier method was introduced for Classification problem)?</p>
Train Model with Multiple Dataloaders

ban_t: <p>I have some train-dataloaders saved on a directory, since each dataset is too big (image data) to save in a single dataloader, so I saved them selerately as belows;</p>
<pre><code class="lang-auto">path_dataloaders = [
./trainloaders/train_dataloat1, 
./trainloaders/train_dataloat2, 
./trainloaders/train_dataloat3..., 
]
</code></pre>
<p>Here, I would like to train a model with these multiple dataloaders.</p>
<pre><code class="lang-auto"># Get path to dataloaders
path_dataloaders = glob.glob('./trainloaders/*')

# Model instance
model = Net()
model.to(device)
# Loss function
criterion = nn.MSELoss()
# Optimizer
optimizer = optim.SGD(model.parameters(), lr=0.01)


# Train
def train(model):
    model.train()
    for data, label in train_loader:
        data, label = data.to(device), label.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, label)
        loss.backward()
        optimizer.step()


# Training!!
max_epoch = 100

for dl in tqdm(range(len(path_dataloaders))):
    train_loader = torch.load(path_dataloaders[dl])
    for epoch in range(max_epoch):
        train_ = train(model)
</code></pre>
<p>As as result, the training process looks work, but train loss is really unstable (sometime going down, but sometimes up).</p>
<p>Does anyone know if this approach is correct? or hope give me an advise of better way.<br>
Thanks you in advance.</p>

ptrblck: <p>I’m not sure how you are saving the <code>DataLoader</code>s. Are you storing the <code>Datasets</code> including all loaded samples?<br>
If so, the approach should be working fine. Alternatively, you could of course also lazily load the data and use a single <code>Dataset</code>.</p>

ban_t: <p>Thank you for you comment.<br>
My Datasets including all data, so it should be working fine.<br>
My concern of unstable train loss might be due to another reason. Thanks anyway.</p>
Automatic reshape of two tensors when subtracting

aktsvigun: <p>Good afternoon,<br>
I have two tensors of shape <code>(n x k)</code> and<code> (k x m)</code>. What I want is to subtract each row of tensor_2 from tensor_1, so that the new array has a shape <code>(n x k x m)</code>. Is it possible to do automatically without repeating the tensors <code>m</code> and <code>n</code> times respectively (as they consume much memory)?<br>
Thanks in advance!</p>

suraj.pt: <pre><code class="lang-auto">a = torch.tensor([10,20,30]).repeat(10).reshape(10,3) # n=10, k=3
b = torch.tensor([5,4,3,2,1]).repeat(3).reshape(3,5) #m=5
aT = a.unsqueeze(0) # adds a z-dimension
bTR = b.T.reshape(m, 1, k) 
result = aT - bTR # [m,n,k]
</code></pre>

aktsvigun: <p>Thanks for the answer! However, I say this option is obvious but unfavorable due to the big amount of memory the tensors consume when repeated <code>m</code> and <code>n</code> times respectively.</p>

suraj.pt: <p>The <code>repeat</code> calls are to emulate your input matrices. For the actual subtraction, you don’t need repeat, only these lines:</p>
<pre><code class="lang-auto">a = tensor_1
b = tensor_2
aT = a.unsqueeze(0) # adds a z-dimension
bTR = b.T.reshape(m, 1, k) 
result = aT - bTR # [m,n,k]</code></pre>

aktsvigun: <p>That was exactly the point. Thank you, Suraj!</p>
Is using separate optimizers equivalent to specifying different parameter groups for the same optimizer?

entslscheia: <p>In PyTorch, we can pass parameter groups when initializing an optimizer. This is a very useful feature. For example, we can specify parameters in different layers into different groups to have separate learning rate for each layer.</p>
<p>However, is it the only way to achieve this? If we just create separate optimizers to optimize different parameters, is it the same or not?<br>
If we are using state-less optimizer like SGD, there is apparently no difference. If we are using something like Adam, which needs to record the moving average of gradients from previous iterations, is it still the same?</p>
<p>I looked into the source code of Adam, and found the following part:</p>
<pre><code class="lang-auto">        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad
                if grad.is_sparse:
                    raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead')
                amsgrad = group['amsgrad']

                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = 0
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    # Exponential moving average of squared gradient values
                    state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)
</code></pre>
<p>Based on the last 4 lines of code, the moving average are computed within different param groups separately, which means the optimizer never has a global view of all model parameters. So I think it’s still equivalent to create separate Adam optimizers for different parameter groups. I feel it’s just using different APIs to achieve the same function. Am I right about that?</p>

ptrblck: <p>This should be the case, as the <code>step</code> function iterates all groups, as you’ve already pointed out.<br>
As a quick test you could seed the code properly and compare both approaches (different <code>param_groups</code> and different optimizers), which should yield the same weight update and states.</p>
How do you choose the right Batch Size?

Skeptic06: <p>Naive question.</p>
<p>How do you know that the batch size you have selected is the right size w.r.t a gpu while training ML models? I am training a model and I think the batch is too less. If I increase the memory to almost double, it goes out of memory. So I know the sanity is somewhere in between to optimize for speed.</p>
<p>How much I can increase it to optimize the memory? The volatile gpu util is 52% and memory usage 3484MiB / 12212MiB.</p>
<p>Thanks in advance.</p>

vdw: <p>I don’t think there’s a straighforward way to set the batch size. One obviously aims to maximize the batch size to improve the raw processing performance. So you increase until you ran out of memory and dial it back again.</p>
<p>Not that with increasing (decreasing) batch size, you might also increase (decrease) the learning rate. Since you average the gradients over the whole badge, larger batches tend to have smaller gradients.</p>
Windows Getting Black screen after an update

HOASIN: <p>Hello Friends,</p>
<p>I am getting a black screen after Windows update. How can I get the issue fixed? I found <a href="https://silicophilic.com/windows-10-update-causes-black-screen/" rel="nofollow noopener">this</a> guide, should I go ahead?</p>

ptrblck: <p>Your issue seems to be Windows-related, while you’ve posted in the PyTorch forum.<br>
I would recommend to use a Windows support board to get faster and better support. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Should the image size has to be fixed before annotation in yolo?

Sravani_Lekkala: <p>I want to extract the text from an id card. The text can be even rotated also. I want to annotate the image with help of rotating bounding boxes.<br>
My doubt is should the images have to be same size before giving as  input to yolo?</p>

ptrblck: <p>Inputs in the same shape make it often easier to feed them as batches to the model.<br>
Otherwise you would have to resize or pad them in order to create batches.<br>
If you have the ability to resize them now to a static shape, I would try it in case you don’t have strong arguments against it.</p>
Input type (torch.FloatTensor)

allenhung1025: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2d237789efc22d33cfec586bf0e9b0ddc8f3fa2.png" alt="Screenshot from 2020-09-03 12-25-08" data-base62-sha1="yE5VYcRUW0spmv6F0Yct20QsOPw" width="580" height="84"><br>
Input and model are all on GPU. However, it seemed not working. Does anyone has the same issue?</p>

allenhung1025: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7fd0ddd6e09ef7157fae9c474366c5cb478c97e.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7fd0ddd6e09ef7157fae9c474366c5cb478c97e" title="Screenshot from 2020-09-03 12-27-29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fd0ddd6e09ef7157fae9c474366c5cb478c97e_2_690x162.png" alt="Screenshot from 2020-09-03 12-27-29" data-base62-sha1="x6gsCOGdeXNMZkuSFL5LNhykXim" width="690" height="162" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fd0ddd6e09ef7157fae9c474366c5cb478c97e_2_690x162.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fd0ddd6e09ef7157fae9c474366c5cb478c97e_2_1035x243.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fd0ddd6e09ef7157fae9c474366c5cb478c97e_2_1380x324.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-03 12-27-29</span><span class="informations">1710×402 42 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Is this a separate issue or a double post from <a href="https://discuss.pytorch.org/t/runtimeerror-input-type-torch-floattensor-and-weight-type-torch-cuda-floattensor-should-be-the-same/95012">here</a>?<br>
In the latter case, let’s keep the discussion in one topic, please.</p>
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

allenhung1025: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/9496d58ad2725d67cbab9ccaf8e56f1791282b90.png" alt="Screenshot from 2020-09-03 12-10-19" data-base62-sha1="lctPeiKM5CLfglrepWRuinYtUly" width="1" height="1"><br>
I have checked my input and model are on GPU. However, it seemed not working. Can anyone help me with the bug I’m encountering?</p>

ptrblck: <p>Could you post your code so that we could have a look, please?</p>
Model Memory Footprint Increases on New Machine

kheyer: <p>I’m trying to debug a situation where running the same model/training code on a new instance has about 3x the GPU memory usage.</p>
<p>Instance 1:<br>
P2, Cuda 10.2 (driver 440.33.01), Pytorch 1.5</p>
<p>Instance 2:<br>
P2 (Sagemaker), Cuda 11.0 (driver 450.51.05), Pytorch 1.6</p>
<p>The model I’m running is a LSTM. On instance 1, training with a batch size of 512 uses 2.3 GB of GPU memory. On instance 2, training with a batch size of 512 uses 9.6 GB of GPU memory.</p>
<p>Both instances are running the same code (same commit) and the same model.</p>
<p>Does anyone know how I might go about debugging this?</p>

ptrblck: <p>How are you measuring the memory usage? Are you using <code>nvidia-smi</code> or <code>torch.cuda.memory_allocated()</code> and other PyTorch methods?<br>
Also, are you using <code>torch.backends.cudnn.benchmark=True</code> and what kind of GPUs are you using in those machines?</p>
Best sampling function for A3C prob

granth_jain: <p>Hi,</p>
<p>I am confused in which is the best probability distribution sampling functions is best for training an A3C reinforcement learning model.</p>
<p>May I get some advise of experience holders.</p>
<p>Thanks,<br>
Granth</p>

iffiX: <p>I guess you are asking for sampling functions in the contiguous domain and not discrete domain?</p>

granth_jain: <p>Hi,</p>
<p>I am trying for discrete domain.</p>
<p>Actually I have tried multinomial and categorical …but both gets stuck if they want to avoid negative rewads and try an action which does nothing.</p>
<p>Can you please helps to let me know any sampling function that tries the low probability action as well.</p>
<p>Also is it ok to sample a random action while training A3C.</p>
<p>Thanks,<br>
Granth</p>

iffiX: <p>The problem is that your network has converged and keep on outputing “invalid” actions, setting their probability to be high and “valid” ones to be low. You should check your implementation rather than blaming the distribution itself.</p>
<p>A2C, A3C… these policy based methods relies on sampling from a distribution to calculate the needed log probability, it is not only “ok” but also “must”.</p>
Copy slices of tensor values from previous version of network to current version

picklerick: <p>Hi,</p>
<p>So I’m trying to make a neural network that copies over trainable weights from an ‘ancestor’ network that is almost similar. This is how I currently do it:</p>
<p><code> list(self.parameters())[layer_number * 2][:-1, :] = params[layer_number * 2]</code></p>
<p>Here ‘params’ is a list of the trainable parameters of the ancestor network.<br>
<code>params = list(self.parameters())</code></p>
<p>My issue is that when I print out the tensors of the <strong>NEW</strong> network, I see grad_fn=&lt;CopySlices&gt; in the tensors that were copied over.<br>
<code>[Parameter containing: tensor([[-0.0237,  0.1563,  0.0842, -0.3308, -0.3521],         [ 0.1558, -0.0448, -0.4337, -0.3235, -0.3658],         [-0.1654,  0.1849,  0.1011,  0.2428,  0.3796]], grad_fn=&lt;CopySlices&gt;),  Parameter containing: tensor([-0.4441, -0.3973, -0.4269], grad_fn=&lt;CopySlices&gt;),  Parameter containing: ...</code></p>
<p>The parts of the new network that have completely new values (no copied over values at all) have ‘requires_grad=True’.<br>
<code>Parameter containing: tensor([ 0.1209,  0.0306,  0.5345, -0.3023], requires_grad=True), Parameter containing: tensor([[ 0.0771, -0.1399, -0.1571,  0.2052],         [-0.1313,  0.0706, -0.0783, -0.3379],         [-0.1491,  0.3440,  0.4240,  0.0006],         [-0.1941, -0.0320, -0.3423, -0.4040],         [-0.1626,  0.3006, -0.4289, -0.1277]], requires_grad=True), Parameter containing: tensor([ 0.1358, -0.0119,  0.4847,  0.1065, -0.2739], requires_grad=True) ...</code></p>
<p>Does this mean that the values that were copied over to the new network are untrainable? What can I do to make the copied over values also show ‘requires_grad=True’ (and not  grad_fn=&lt;CopySlices&gt;)? And I want to get rid of the ancestor weights entirely from the computation graph. All I want is to simply transfer over the ancestor <strong>values</strong> and nothing else. Please help.</p>

ptrblck: <p>Try to wrap the parameter assignment into <code>with torch.no_grad()</code> and check the new parameters again.</p>
Batch operation problem for specific conditions

yhkwon: <p>Hi guys</p>
<p>I’m doing a conditional sampling test from the results of the network</p>
<p>For example,</p>
<p><strong>cond = torch.where(x&gt; 0)</strong><br>
<strong>x[cond].mean</strong></p>
<p>If the condition condition is not satisfied in a specific batch, 0 data is sampled from x and the mean is calculated, resulting in nan.</p>
<p>The solution, in my opinion, is to give each condition to the batch while running the for loop. Can I prevent this problem with other simple operations?</p>

ptrblck: <p>I think checking for an empty output before applying the <code>mean</code> operation sounds like a simple and valid solution.<br>
What kind of solution are you looking for and would the check not work for you?</p>
A question about SSD model train

hookzeng: <p>When I trian my SSD model , I found the confidence loss always bigger than location loss, and this loss can’t be smaller. Could you tell me what things I should do to solve it? What problem I meet in training? Ths~<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/5283be589780163946d0a541530cd6221dffd255.png" alt="2020-09-05 13-56-48屏幕截图" data-base62-sha1="bLXpwZnAFgXicGPsNQblZWokQO9" width="655" height="300"></p>
Merging model after trainer, before evaluation and predict

Elidor: <p>Hi everyone,</p>
<p>I have two models of the <a href="https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py" rel="nofollow noopener">Trainer</a> class from <a href="https://github.com/huggingface/transformers" rel="nofollow noopener">Transformers lib</a>.</p>
<p>In particular:</p>
<pre><code class="lang-auto">trainer1 = Trainer(
        model=model1,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        compute_metrics=compute_metrics,
    )

trainer2 = Trainer(
        model=model2,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        compute_metrics=compute_metrics,
    )
</code></pre>
<p>both models were trained (<code>trainer.train(model_path=...) trainer.save_model()</code>) on the same dataset to which I made some changes.</p>
<p>Now my problem is that I necessarily need to merge these two models (merge their weights (does that make sense?) or something like that) before doing the evaluation (<code>trainer.evaluate()</code>) and prediction (<code>trainer.predict(test_dataset)</code>) operations.</p>
<p>How can I do?<br>
Do you have any suggestions?</p>
<p>P.s. The important thing is that the model resulting from the union/merge of these two models always has a <code>Trainer</code> type.</p>
<p>Thanks in advance!</p>

ptrblck: <p>Since this question is Huggingface-specific, I think you might get a good answer in <a href="https://discuss.huggingface.co/">their forum</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Need help with errors

Piloish: <p>I’m new and started a project and I keep getting errors but they say they are on lines that are in the thousands. I only have around 100 lines. Am I reading them wrong?</p>

ptrblck: <p>Could you post the error message you are getting, so that we could have a look? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torchvision transforms in c++

jloverich: <p>I have a pytorch model.  During training I use torchvision transforms to transform the images.  I now want to use the model as a torchscript and call through c++.  Ideally, I’d like to be able to use the same torchvision transforms in c++ (without re-writing myself in c++), apply them to the images and then pass through the model.  Is this possible?  If the torchvision transforms can’t be called from c++, what’s the suggested alternative approach?</p>

ptrblck: <p>If you are using <code>torchvision.transforms</code>, which can be applied to tensors, you could add them to the <code>forward</code> method of your model and script them.<br>
On the other hand, if you are using image transformation, which are applied to <code>PIL.Image</code>s, you won’t be able to use them directly in C++ (at least I’m not aware of a way to use <code>PIL</code> in C++) and could use e.g. OpenCV, which is available in Python and C++ for the preprocessing instead.</p>
Some modules inside sequential modules need multiple inputs

navid_mahmoudian: <p>Hello,<br>
Let’s assume that I want to define a torch.nn.The sequential container in which some of its modules take some inputs other than input x. For example, if the sequential container is defined like this:</p>
<p>self.my_seq = torch.nn.Sequential(<br>
module_A(),<br>
module_B(),<br>
module_C(),<br>
)</p>
<p>Then in the forward method of the class, I want to pass some arguments for module_A and module_C other than input x, i.e.,</p>
<p>def forward(self, x, module_A_argument, module_C_argument):<br>
self.my_seq(x, module_A_argument, module_C_argument)</p>
<p>where I would like it to do the following operation:<br>
output = module_A(x, module_A_argument)<br>
output = module_B(output)<br>
ouput = module_C(output, module_C_argument)</p>
<p>How can I have this feature in sequential container?</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-can-i-pass-multiple-inputs-to-nn-sequential-layers/50868">this topic</a> with potential workarounds and links to further discussions and implementations. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Loss doesn't decrease in Simple Example

mbc2004: <p>Hello,</p>
<p>I’m having some trouble getting my system to train even when using the simplest of examples. To demonstrate my issue I have created a quick toy problem a two layer NN which should calculate the output of the XOR function. However, my loss never seems to decrease.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np 
import random

dataset = [ [1,1], [1,0], [0,1], [0,0] ]
labelset = [  [0],   [1],   [1],   [0] ]


class Model(nn.Module):
	def __init__(self):
		super().__init__()
		self.lin = nn.Sequential(
			nn.Linear(2,2),
			nn.Linear(2,2)
		)
		#self.lin = nn.Linear(2,2)

	def forward(self, inp):
		return self.lin(inp)

net = Model()
net.train()

# define loss function
criterion = torch.nn.CrossEntropyLoss()

# define optimizer
params = list(net.parameters())
optimizer = torch.optim.SGD(params, 0.1)
	
# Train Network
#----------------
losses = []

epoch = 200
with torch.autograd.detect_anomaly():
	for e in range(epoch):
		i = random.randint(0, 3)

		data  = torch.tensor([dataset[i]], dtype=torch.float)
		label = torch.tensor(labelset[i])

		data = torch.autograd.Variable(data)
		label = torch.autograd.Variable(label)
		
		# compute output
		logits = net(data)

		# get loss
		loss = criterion(logits, label)
		loss.backward()

		# optimize SGD
		optimizer.step()
		optimizer.zero_grad()

		losses.append(loss.cpu().detach().numpy())

# show Losses
import matplotlib
import matplotlib.pyplot as plt

plt.plot(losses)
plt.savefig("analysis/fig/plt.png")

# eval model
net.eval()
with torch.no_grad():
	for i in range(len(dataset)):
		data  = torch.tensor([dataset[i]], dtype=torch.float)
		label = labelset[i]

		data = torch.autograd.Variable(data)
		logits = net(data)

		out = np.argmax(logits.cpu().detach().numpy())

		print(dataset[i], out, label)
</code></pre>
<p>Here is the loss over the course of the run (loss is on the y axis and epoch is on the x axis):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/59c1d51a34092d1ddbc90e09da37a41169bef221.png" alt="plt" data-base62-sha1="cO1MieNPcP0TsIzGDhgsQmto3nP" width="640" height="480"></p>
<p>I presume that I have missed some pivotal step in the code or messed up some value in the hyper-parameters but I can’t for the life of me figure out what it is. Any help would be appreciated. Thank you</p>

ptrblck: <p>Your current model does not use any non-linearity between the linear layers, so you might want to add it.<br>
Also, have a look at <a href="https://discuss.pytorch.org/t/unable-to-learn-xor-representation-using-2-layers-of-multi-layered-perceptron-mlp/13287/3">this topic</a>, which had a similar error and where the same use case was discussed. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>