akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Train LSTM with very large vocabulary

FrancescoMandru: <p>I’m training a LSTM with a very large corpus. The LSTM has a pre-trained matrix of embeddings with 120000 words, obtained with word2vec. The training set is composed by 900000 training phrases of length 30 each and the validation has 270000 samples. The problem is that when I run this on Google Colab with the maximum RAM and the powerful GPU I get CUDA out of memory. I reduced the batch size to 128 but i think that most of the memory is occupied by the network and vocabulary, hence I’m asking if there are some tricks when you work with very large datasets and networks.</p>

ebarsoum: <p>Can you use Sparse for the embeddngs (<a href="https://pytorch.org/docs/stable/sparse.html" rel="nofollow noopener">https://pytorch.org/docs/stable/sparse.html</a>) or put the embeddings on host memory.</p>

vdw: <p>Sorry, not a solution, but may I ask what your task is? Classification? Translation? 120k is quite a large vocabulary. Training an RNN is tricky enough, so always try to minimize the vocabulary as much as possible to make it easier for the network. Some common considerations/steps:</p>
<ul>
<li>What’s the distribution of words? I would hazard a guess that the top 10k words might already cover 95% of your content. For classification, that’s probably already enough.</li>
<li>What do you do for preprocessing? I often work with informal text. While I’m fine with “lol”, I don’t want it to happen that “loool” or “loooool” count as different words. Such noise can blow up a vocabulary quickly.</li>
<li>Do you have many named entities (i.e., names of persons, locations, organizations, etc.) or numbers in your text? In case of translation, such “special word” won’t get translated anyway just copied. So a common approach is to first replace such words with a special token (e.g., “I met Alice in London” to "I met  in ") do the translation and then add the names back in.</li>
</ul>
<p>Just some food for thought.</p>
How to remove batch size in training loop to perform certain operations on a single image

bing: <p>I am trying to perform certain operations on a single image, while in a training loop.<br>
In case of <code>batch_size = 1 </code>, it could be easily done by using <code>torch.squeeze</code> but I am unable to think of a way when I can do it for other batch sizes.<br>
Below is the minimum code for representation -</p>
<pre><code class="lang-auto">def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):
      for epoch in range(1, n_epochs+1):
             for batch_idx, (data, target) in enumerate(final_train_loader):
                        # Here the target shape would be B*H*W*N 
</code></pre>
<pre><code class="lang-auto">B: Batch size
H,W,N:Height,width,no. of channels
</code></pre>
<p>I want it in a form of <code>H * W * N</code></p>

ptrblck: <p>If you want to perform some operations on single samples, you could use a for loop and iterate your batch:</p>
<pre><code class="lang-python">for d, t in zip(data, target):
    # op on single sample
</code></pre>
Can't use torch.angle?

science: <p>On the PyTorch documentation it says there is a method torch.angle that returns the element wise angles between the input vectors.  Does this method still exist?  I’m trying to use it except it gives the error:<br>
module ‘torch’ has no attribute ‘angle’</p>
<p>Thanks in advance.</p>

ptrblck: <p>Could you check your PyTorch version via <code>torch.__version__</code>?<br>
I guess your build might be too old.</p>

science: <p>Yep, this was the problem.  I updated PyTorch and was able to use the function.  Thanks!</p>
Input type (torch.cuda.FloatTensor) and (torch.FloatTensor) should be same

Proban_Nalog: <p>Firstly, exceuse me for “noob question” .<br>
I am begginer in python language.<br>
Trying to set up Demucs from facebook research repository<br>
<code>https://github.com/facebookresearch/demucs</code> in google Colab notebook to work<br>
Error code is:<br>
"<code>Separating track /content/drive/testfile.flac 0%| | 0/210 [00:00&lt;?, ?seconds/s]Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/content/demucs/demucs/separate.py", line 166, in main() File "/content/demucs/demucs/separate.py", line 155, in main sources = apply_model(model, wav, shifts=args.shifts, split=args.split, progress=True) File "/content/demucs/demucs/utils.py", line 119, in apply_model chunk_out = apply_model(model, chunk, shifts=shifts) File "/content/demucs/demucs/utils.py", line 140, in apply_model out = model(padded.unsqueeze(0))[0] File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/content/demucs/demucs/model.py", line 198, in forward x = encode(x) File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 202, in forward self.padding, self.dilation, self.groups) **RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same**</code><br>
I am 99%sure that have done all steps to install demucs conda envioment properly<br>
Googled and search topics about error, and seems there need to be model demucs called by cuda.<br>
There is no official colab of this source saparation repository. Unfortunately, I have radeon card, so this repository cannot run for me on Windows. Cpu mode is to slow. Please someone help.<br>
If there is need to upload colab file, I will do that.<br>
Thank you in advance</p>

JuanFMontesinos: <p>It sounds like you forgot to allocate either the input or the layer in gpu. Tensors in gpu are of type cuda.floatensor meanwhile tensors in cpu are just floatensor</p>

TinfoilHat0: <p>When you’re training with a GPU, both the model and the data, as well as the loss function you use, must be in the same GPU.</p>
<p>You’re probably forgetting to pass one of these to your GPU.</p>
Unable to load WaveGlow checkpoint after training with multiple GPUs

Neuperc: <p>I’ve trained WaveGlow model from <a href="https://github.com/NVIDIA/waveglow" rel="nofollow noopener">here</a> with multiple GPU, but when I try to load the checkpoint to do inference (through inference.py), some checkpoints are loaded without any problem, but most of them raise the error below:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "inference.py", line 105, in &lt;module&gt;
    args.sampling_rate, args.is_fp16, args.denoiser_strength)
  File "inference.py", line 46, in main
    model_state_dict = torch.load(waveglow_path, map_location="cuda:0")['model'].state_dict()
  File "/home/anaconda3/envs/dl/lib/python3.6/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/home/anaconda3/envs/dl/lib/python3.6/site-packages/torch/serialization.py", line 581, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 3901634075968565895 got 512
</code></pre>
<p>I changed the map_location to  “cpu” and “cuda” and also tried to load the checkpoint with the same number of GPUs used during training, but still get the same error.</p>
<p>When I train the model with a single GPU, all checkpoints are loaded without any issue. This happens only after I run distributed training.</p>

ptrblck: <p>This usually happens when multiple processes try to write to a single file.<br>
However, this should be prevented with the if condition <code>if rank == 0:</code>.<br>
Did you remove it or changed the save logic somehow?</p>

Neuperc: <p>Yes, exactly! it was a simple mistake by me. I commented the original “save_checkpoint” section and only added “save_checkpoint” after the epoch loop without checking if rank==0. Now it works without any errors.<br>
Thanks a lot for your help!</p>

justinliu: <p>I was wondering in such a case is the checkpoints still salvageable or are they simply damaged?</p>

ptrblck: <p>If multiple process have written to the same file, it’s most likely damaged and you won’t be able to restore it.</p>
Torch.norm when p=2 doesn't give largest singular value as expected

lyndond: <p>From my understanding, torch.norm when p=2 should give the spectral norm, i.e. largest singular value of a matrix:</p>
<pre><code class="lang-auto">A = torch.eye(2)
norm = torch.norm(A, p=2)
</code></pre>
<p>The singular values of <code>A</code> are of course 1 and 1, and so the max s.v. should also be 1. But the returned value <code>norm</code> is sqrt(2) which is the Frobenius norm.</p>
<p>I’ve also tried various values of <code>dim</code> to no avail.</p>
<p>[<strong>Update</strong>: It looks like this is a high priority issue <a href="https://github.com/pytorch/pytorch/issues/24802" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/24802</a> ]</p>
Which is the best optimizer for non linear regression?

rerere_L: <p>Im searching for a good  optimizer for pytorch. My NN is a numeric non linear regression(not classification), with 3 neuron input, 6 in hidden layer and 8 in output layer.<br>
which is the best optimizer for non linear regression?</p>

tom: <p>If you have that few parameters, you could try LBFGS. I use that e.g. for Gaussian Processes.<br>
Note that it necessarily needs a closure (re-) evaluating the model.</p>
<p>Here is a snippet that I use with GPs (which are a bit special in that they take the data and loss into the model, you’d have to feed data into m and probably also go from predictions to loss - called objective here), which should give you the idea:</p>
<pre><code class="lang-auto">opt = torch.optim.LBFGS(m.parameters(), lr=1e-2, max_iter=40)
def eval_model():
    obj = m()
    opt.zero_grad()
    obj.backward()
    return obj

for i in range(50):
    obj = m()
    opt.zero_grad()
    obj.backward()
    opt.step(eval_model)
    if i % 5==0:
        print(i,':',obj.item())

</code></pre>
<p>Best regards</p>
<p>Thomas</p>

rerere_L: <p>Thanks. Im using your code but give me error:</p>
<pre><code class="lang-auto">N, D_in, H, D_out = 3, 1000, 6, 8

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

def loss_fn(output, target):
    loss = torch.mean((output - target)**4)
    return loss

model= torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),)


opt = torch.optim.LBFGS(model.parameters(), lr=1e-2, max_iter=40)

def eval_model() :
    obj = model(input,output)
    opt.zero_grad()
    obj.backward()
    return obj

for i in range(50):
    obj = model(x, y)
    opt.zero_grad()
    obj.backward()
    opt.step(eval_model)
    if i % 5==0:
        print(i,':',obj.item())
</code></pre>
<p>error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-19-36374f9ee179&gt; in &lt;module&gt;
     36 
     37 for i in range(50):
---&gt; 38     obj = model(x,y )
     39     opt.zero_grad()
     40     obj.backward()

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--&gt; 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

TypeError: forward() takes 2 positional arguments but 3 were given


</code></pre>
<p>i dont know why??</p>

tom: <p>Very likely, you want to split that into <code>pred = model(x) ; loss = loss_fn(x, target)</code> (loss being more common than obj here).</p>
Why every tutorial use "for i, batch in enumerate(data_loader)" even if i var not used later?

hadaev8: <p>Do enumerating do something besdies i var?</p>

albanD: <p>Do you mean <code>i</code> is not used?<br>
No enumerate does not do anything special, but for logging purposes, it is often convenient to have this index of where you are in the epoch.</p>
Connecting multiple gpu

slavavs: <p>My rtx 2070 is now installed on my computer. This is not enough for quick work.<br>
If I buy another rtx 2070, will it be right, will I see accelerated work? For my code to work at least somehow quickly, I need at least 4 video cards according to my calculations.</p>
<p>Who connected several video cards give me advice.</p>

slavavs: <p>Do I need to use SLI - NVLink to connect video cards or can I just use PCI?</p>
Classifying spatio-temporal data/videos

Subhankar_Ghosh: <p>Hi!<br>
I am a PyTorch newbie. As part of my research I need to build a video classifier and train it on a dataset on different kind of fluids(videos of fluid flows).<br>
I have previously worked on image classification using PyTorch but I am a bit clueless about how to do the same on videos.<br>
Can you all please suggest some resources/pointers that might be helpful?</p>
<p>Thanks!</p>

christopherkuemmel: <p>Hi <a class="mention" href="/u/subhankar_ghosh">@Subhankar_Ghosh</a>,<br>
For video processing you want to somehow retrieve the information of corresponding images. For example, this can be done by sequential models like RNNs or 3D CNNs (the extra dimension is for successive images). “All you need” is to wrap a image classifier within a sequential model (or extend one time dimension). Since you did work with image classifiers I’m sure you will get there.</p>
<p>I found these links repository to be helpful:</p>
<ol>
<li><a href="https://cs.stanford.edu/people/karpathy/deepvideo/" rel="nofollow noopener">Large-scale Video Classification with Convolutional Neural Networks - Karpathy</a></li>
<li><a href="https://github.com/HHTseng/video-classification" rel="nofollow noopener">HHTseng’s github repo</a></li>
<li><a href="https://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noopener">RNNs - Karpathy’s blog</a></li>
</ol>
<p>Good luck and have fun with your implementation! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Subhankar_Ghosh: <p>Hi <a class="mention" href="/u/christopherkuemmel">@christopherkuemmel</a>, thanks for the help.</p>
<p>Do you think the conv3d layers in Pytorch can be used for videos?</p>

christopherkuemmel: <p><a class="mention" href="/u/subhankar_ghosh">@Subhankar_Ghosh</a> For sure. A Conv3d layer has the input shape of (N, C, D, W, H) where N is the batch size and D the depth of the images. In your case this would be the dimension where you stack the frames of your video.</p>
Define backward() function in my own loss function

yangcheng001: <p>My own loss function  is special and can not be solved by the ‘autograd’ function. so I want to define the backward()  in my loss function. But I don’t know if I can get the gradient of loss by hand and make it back throughout the network automatically？</p>
<pre><code class="lang-auto">import torch

class MyLoss(torch.autograd.Function):

    @staticmethod
    def forward(self, input, label):
        ctx.save_for_backward(input, label)
        #my code
        return loss

    @staticmethod
    def backward(ctx, grad_output):
        input, label = ctx.saved_tensors
        # my code
        return grad_input, None


model = MyNetwork()#this is my network
myloss = MyLoss()

for t in range(500):
    output = model(input)
    loss = myloss(output,label)
    loss.backward()

</code></pre>
<p>Is the above code feasible? I am not sure if the gradient can automatic back throughout the network.<br>
Thank you.</p>

albanD: <p>Yes you can do that. See more details <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd">in the doc</a>.<br>
Be careful:</p>
<ul>
<li>You should not instantiate an instance of the Function. You should do <code>loss = MyLoss.apply(output, label)</code>. If you prefer to instantiate an instance, you can simply create a new module like:</li>
</ul>
<pre><code class="lang-auto">class MyLossMod(nn.Module):
    def forward(self, input, target):
        return MyLoss.apply(input, target)

# Then you can use any of the two
loss = MyLoss.apply(input, target)
# or
mylossmod = MyLossMod()
loss = mylossmod(input, target)
</code></pre>
<ul>
<li>Your arguments for forward should be <code>ctx</code> not <code>self</code> but I guess that’s just a typo</li>
</ul>

yangcheng001: <p>Thank you for your reply!</p>

Khadija_Khaldi: <p>Please, when our loss function cannot be solved by autograd ? and if it is the case, do we need then to write two classes, one extending the Function class and the other extending nn.module ? (like in the tutorial <a href="https://pytorch.org/docs/master/notes/extending.html" rel="nofollow noopener">https://pytorch.org/docs/master/notes/extending.html</a>)<br>
Thank you!</p>

albanD: <p>You will need to define a custom <code>Function</code>.<br>
The custom Module presented there is just for convenience. You can use <code>YourFunction.apply()</code> in your code if you don’t want to use Modules.</p>
CrossEntropyLoss is not going down

mlearner: <p>Hello,<br>
I am training the model written below, but the Cross Entropy Loss is not decreasing (it oscillates close to the initial value), even increasing the learning rate.</p>
<p>I have already searched for related topics in the forum, but no one is solving my problem.<br>
The model seems pretty straightforward and I cannot detect any mistakes by myself.<br>
If someone can spot something unusual, it would be very helpful.</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        #1st
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1),
            nn.Conv2d(64, 64, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Dropout(p=0.25)
        )
        #2nd
        self.conv2 = nn.Sequential(
            nn.Conv2d(64, 64, 3, padding=1),
            nn.Conv2d(64, 64, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Dropout(p=0.25)
        )
        #3rd
        self.conv3 = nn.Sequential(
            nn.Conv2d(64, 64, 3, padding=1),
            nn.Conv2d(64, 64, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Dropout(p=0.25)
        )
        #4th
        self.fc1 = nn.Sequential(
            nn.Linear(64*8*8, 128),
            nn.Dropout(p=0.5)
        )
        #5th
        self.fc2 = nn.Sequential(
            nn.Linear(128, 128),
            nn.Dropout(p=0.5)
        )
        #6th
        self.fc3 = nn.Sequential(
            nn.Linear(128, 22),
        )
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x= x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x
</code></pre>

ptrblck: <aside class="quote no-group" data-username="mlearner" data-post="1" data-topic="68852">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/87869e/48.png" class="avatar"> mlearner:</div>
<blockquote>
<p>even increasing the learning rate</p>
</blockquote>
</aside>
<p>I would recommend the opposite and try to decrease the learning rate.<br>
Also, try to overfit a small data sample and make sure your model is able to do so.</p>

mlearner: <p>After hours and hours I found out that the problem is the input normalization (of the dataset): I use the normalization the with mean and std calculated over my specific dataset.<br>
This leads to non-decreasing loss and producing always the same output (prediction) no matter the input.<br>
If I use the standard [0.5, 0.5, 0.5] everything goes fine.<br>
Do you have any explanation? I am a little confused…</p>

albanD: <p>You might want to plot some histograms of values in your dataset. That might shed some light on this.<br>
Most likely that the normalization was destroying useful features (or squashing them too much and the network was not able to differentiate anymore).</p>
I am getting a Cuda error even when I comment it out

Geoffrey_Payne: <p>I am getting this error message</p>
<pre><code class="lang-auto">  File "main.py", line 185, in &lt;module&gt;
    input = input.to(device)
RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>I try commenting out these lines of code;</p>
<blockquote>
<p><span class="hashtag-raw">#if</span> device == torch.device(“cuda”):<br>
<span class="hashtag-raw">#input</span> = input.to(device)<br>
output = skipgram_model(input)</p>
</blockquote>
<p>But then I get this error message;<br>
<code>RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select</code></p>
<p>How do I fix this?</p>

albanD: <p>Could you run your code while setting <code>CUDA_LAUNCH_BLOCKING=1 python your_code.py</code> and see where the error comes from please?<br>
The cuda api is asynchronous by default so the stack trace above might we wrong.</p>

Geoffrey_Payne: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="68902">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
</aside>
<p>I put the line<br>
<code>CUDA_LAUNCH_BLOCKING=1</code><br>
at the beginning of my code, but I do not see any extra information</p>

albanD: <p>It should be in the command line when you run your code. Not in python.</p>

Geoffrey_Payne: <p>My command line is via PUTTY.<br>
I have just run<br>
<code>sbatch main.sh CUDA_LAUNCH_BLOCKING=1</code></p>
<p>Not sure this is what you meant. I normally run my code from VS Code rather than command line. However I also run it on a remote server as it takes time to run.</p>

albanD: <p>You will need to set the environment variable <code>CUDA_LAUNCH_BLOCKING=1</code> before running any pytorch code.</p>
<p>So if it is simpler for your, you can set in the first line of your python script <code>import os; os.environ["CUDA_LAUNCH_BLOCKING"] = "1"</code></p>

Geoffrey_Payne: <p>Thank you for that.<br>
This time I got;</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “main.py”, line 193, in <br>
loss_window = F.nll_loss(output, Y_d)<br>
File “/home/xxx.xxx.ac.uk/xxxx/.local/lib/python3.5/site-packages/torch/nn/functional.py”, line 1838, in nll_loss<br>
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:110</p>
</blockquote>

albanD: <p>Great. So this means that the problem comes from the nll loss.<br>
The most common reason here is that <code>Y_d</code> contains indices that are invalid. either negative or larger than the number of scores in the output.</p>
How to cast/change Tensor dtype in C++?

cruzas: <p>Hi everyone,</p>
<p>Does anyone know how one can cast a Tensor to a different torch::dtype() in C++?</p>
<p>In Python this appears to be as simple as .float() (for torch::dtype(torch::kFloat32)) and .double() (for torch::dtype(torch::kFloat64)).</p>
<p>Thank you.</p>

dfalbel: <p>You can use the <code>to</code> method:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
    torch::Tensor tensor = torch::rand({2, 3});
    auto x = tensor.to(torch::kInt);
  std::cout &lt;&lt; tensor &lt;&lt; std::endl;
  std::cout &lt;&lt; x &lt;&lt; std::endl;
}
</code></pre>

cruzas: <p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Get both RGB and grayscale image

lepoeme20: <p>Hello,</p>
<p>can I get both RGB and grayscale image simultaneously in a image dataloader?<br>
Just like blow:</p>
<pre><code class="lang-auto">for step, (rgb, gray) in enumerate(dataloader)
</code></pre>
<p>Or is there a way to convert rgb images into grayscale within each batch?</p>
<pre><code class="lang-auto">for step, (imgs, labels) in enumerate(dataloader):
    grayscale = ANY_FUNCTION(imgs).to(device)
    rgb = imgs.to(device)
</code></pre>
<p>I need both RGB and grayscale images in each mini-batch.</p>

beaupreda: <p>Hello,</p>
<p>I suppose that you have a custom dataset class that inherits <code>torch.data.utils.Dataset</code>. In your <code>__getitem__</code> function, simply return both rgb and gray image. Here is an example:</p>
<pre><code class="lang-auto">class CustomDatasetRGB(torch.utils.data.Dataset):
    def __init__(self, images, labels):
        self.images = images
        self.labels = labels

    def __getitem__(self, index):
        rgb_image = cv2.imread(self.images[index], cv2.IMREAD_COLOR)
        gray_image = cv2.cvtColor(rgb_image, cv2.COLOR_BGR2GRAY)
        # cv2.imshow('gray', gray_image)
        # cv2.imshow('rgb', rgb_image)
        # cv2.waitKey(0)
        rgb_image = np.transpose(rgb_image, axes=(2, 0, 1))
        return torch.from_numpy(rgb_image), torch.from_numpy(gray_image), self.labels[index]

    def __len__(self):
        return len(self.labels)

# 1000 images in dataset
images = ['im2.png'] * 1000
labels = np.random.randint(low=0, high=10, size=1000)

dataset = CustomDatasetRGB(images, labels)
loader = torch.utils.data.DataLoader(dataset, batch_size=4)

for step, (rgb, gray, labels) in enumerate(loader):
    print(rgb.size())
    print(gray.size())
    print(labels.size())
    '''
    torch.Size([4, 3, 375, 450])
    torch.Size([4, 375, 450])
    torch.Size([4])
    '''
</code></pre>
FasterRCNN class encoding

loicdtx: <p>Hello everyone,</p>
<p>I am training a FasterRCNN model for detecting objects belonging to two classes (excluding background). These classes are not encoded 1 and 2 but 7 and 22. Is it advisable in this case to re-encode the classes as 1 and 2, and set the number of classes (<code>num_classes</code>) of <code>FastRCNNPredictor</code> to 3 (the two classes + background)?<br>
Otherwise I have to set the number of classes of <code>FastRCNNPredictor</code> to 23 (max encoding + 1 for background), which does not seem ideal.</p>
<p>Thanks</p>

ptrblck: <p>I would recommend to set the number of output activations to the number of classes you are dealing with.<br>
Otherwise your model might output irrelevant classes without any meaning.</p>

loicdtx: <p>Thanks for your response <a class="mention" href="/u/ptrblck">@ptrblck</a>; I re-encoded on the fly using a transform (see below).<br>
In addition to what you mentioned (having irrelevant classes), my initial worry was that the class loss would be somehow “diluted” if I would set the number of output activations to a higher value than the actual number of classes.</p>
<p>The transform</p>
<pre><code class="lang-auto">class EncodeClasses(object):
    """Re-encode classes

    Useful to limit the number of output activations to the actual number of
    classes when initial class codes do not follow a simple sequence ``[1:len(classes)]``
    """
    def __init__(self, encoding):
        """
        Args:
            encoding (dict): Dict of old code, new code mapping
        """
        self.encoding = encoding

    def __call__(self, image, target):
        labels = target['labels']
        for k,v in self.encoding.items():
            labels[labels == k] = v
        target['labels'] = labels
        return image, target
</code></pre>
Audio datatype (does float16 make sense?)

timlod: <p>Hi everyone,<br>
I’m an intermediate PyTorch user (only did a vision project before) who wants to use torchaudio for something new.<br>
I’m making currently making a dataset loader for the NSynth dataset (16bit 16kHz PCM wav files) specific to my application.<br>
As I’ll be training on an RTX GPU, and the data is originally 16bit, I was wondering if it would be smart to use float16 in this case. I’m not exactly sure about this though, given my limited knowledge of signal processing. torchaudio.load returns 32bit floats by default and does not give the option to load float16, so I was wondering whether there exists a theoretical reason.</p>

ptrblck: <p>I don’t think the data loading should be performed in FP16, as you might end up with some quantization noise.<br>
If I’m not mistaken, the 16bit audio files represent 65536 different levels.<br>
Since FP16 cannot represent all integers &gt;2048 (<a href="https://en.wikipedia.org/wiki/Half-precision_floating-point_format">Wikipedia - FP16</a>), you’ll lose some information.</p>
<p>That being said, once you’ve loaded and preprocessed the data, you could still use FP16 for the model training. Have a look at <a href="https://github.com/NVIDIA/apex">apex/amp</a> for an automatic mixed-precision approach.</p>

G.M: <p>Actually, 64bit float can maintain the original information without any loss. According to my  experience, the amount of information lost from using float32 is actually quite small.</p>

ptrblck: <p>FP32 is able to represent all integers in <code>[-16777216, 16777216]</code>, which should thus work for these audio files. Why would you lose information in this use case or did I misunderstood your explanation?</p>

G.M: <p>I did some googling and found that indeed I was wrong. My original conclusion was based on the fact that when I used librosa to load wav files and set the precision to fp32, some values were rounded. But in the case of fp64, the exact value are shown. I guess perhaps that was due to display issue? I’m not sure what’s mechanism behind that.</p>

timlod: <p>Thanks! Makes a lot of sense. I think I’ll stay away from apex for now, I don’t want to overcomplicate things. I guess I can always start with FP32 and move to FP16 later and compare.</p>
CUDA OOM when using pretrained vectors

Felix_Labelle: <p>I’m training a CNN for text classification ( <a href="https://www.aclweb.org/anthology/D14-1181.pdf" rel="nofollow noopener">https://www.aclweb.org/anthology/D14-1181.pdf</a>). When I try updating 300d pretrained embeddings, I get an OOM error when back propagating (on calling optim.step()) regardless of batch size. However, if the embeddings are frozen or the embeddings are learned there are no issues. I suspect this is due to the way they are being loaded. Below is the embedding loading code.</p>
<pre><code class="lang-auto">	def _loadEmbeddings(self, vecFilePath):
		# import pdb;pdb.set_trace()
		print("Loading pretrained embeddings: ", vecFilePath)
		with open(vecFilePath, encoding="utf8") as f:
			readLen = self.vocab.MAX_VOCAB_SIZE
			if readLen == -1:
				lines = [line for line in f.readlines()]
			else:
				lines = [next(f) for x in range(readLen)]
			words = [line.split()[0] for line in lines]
			self.vocab = Vocab(MAX_VOCAB_SIZE = self.vocab.MAX_VOCAB_SIZE, MAX_IDX = 1, UNK_IDX=1)
			self.vocab.createVocab(words)
			embeddings = np.zeros((len(self.vocab), self.embedding_dim)) # zeros or rand?
			for line in lines:
				values = line.split()
				word = values[0]
				index = self.vocab(word)
				if index:
					vector = np.array(values[1:], dtype='float32')
					embeddings[index] = vector
			embeddings[1] = np.mean(embeddings[2:], axis=0)
			return nn.Embedding.from_pretrained(torch.from_numpy(embeddings))
</code></pre>
<p>Does anyone have any ideas as to possible causes or tools that could help debug this issue?</p>

ptrblck: <p>Are you using the same vocab size and embedding dimensions for both use cases (pretrained and trained from scratch)?<br>
If the dimensions are the same, the code should not create an OOM error in one use case while the other is working fine.</p>

Felix_Labelle: <p>The dimensions are the same, however the pretrained embedding vocab size is  ~4x larger (400,000 vs 100,000). Does Pytorch update the entire embedding at once? Otherwise I don’t see how having a larger embedding could be an issue (a side from having more memory consumed, but what I was seeing seemed more like an explosion in memory usage).</p>
<p><strong>Note</strong>: I messed around with it some more and suspect it is a bug in the vocab’s code. If the vocab size isn’t explicitly state (Instead of a size you can pass -1 and it acts as if you were getting the last index of your array) that’s when the issue occurs.</p>
Update the input of model(output of one model as input to another model) in a loop

ash_majeed: <p>Trying to get the execution time of each layer of a model. When I call intermediate_model  error is raised.</p>
<pre><code class="lang-auto">input =Input((10,))
next_layer=input
def profiling(model, test_input):
    # output_of_all_layers = []
    for layer in model.layers:
        layer = layer(next_layer)
        x=layer
        intermediate_moddel= keras.Model(layer,x)
        start = time.time()
        intermediate_output = intermediate_layer.predict(test_input)
        end = time.time() - start
        print(end)
result = profiling(model,features)

</code></pre>

ptrblck: <p>Hi Ash,</p>
<p>you landed in the PyTorch forum, so you might want to post this Keras question in their discussion board or on StackOverflow, where you’ll most likely get a better answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ash_majeed: <p>Thanks for your response</p>
Getting certain values of a tensor from indices

MattyMatt: <p>Hey guys,</p>
<p>I have a tensor of shape (5,10,15):<br>
x = torch.rand(5,10,15)</p>
<p>and then I would like to get the indices of max values along last column:<br>
ids = torch.argmax(x, dim=2) # this has shape (5,10)</p>
<p>Now, what do I do to get the values of the corresponding indices?<br>
My output would be of shape (5,10,1)</p>

ekilic: <p>u can make shape (5,10,1) with</p>
<pre><code class="lang-auto">ids = torch.argmax(x, dim=2).unsqueeze(dim=2)
</code></pre>

MattyMatt: <p>ok, but how do I fetch those x values that match the ids?<br>
my output of shape (5,10,1) are only the max values of x with  respect to last dimension.</p>
<p>In a simple 2D case:<br>
x = [ [1,2,3],<br>
[4, 5, 2]]<br>
ids = [2,1] # 2 because of fetching ‘3’ and 1 because of fetching ‘5’<br>
then i want my output to be<br>
[3,5]. Right?</p>

G.M: <pre><code class="lang-auto">x = tc.randn(5, 10, 15)
indexes = x.argmax(2)
out = tc.gather(x, 2, indexes.unsqueeze(2))
</code></pre>

MattyMatt: <p>Awesome! that solves the problem, thanks G.M</p>
<p>Small followup tho. lets say you have<br>
x = tc.randn(5,10,15,20)<br>
indexes # shape of 5,10</p>
<p>so now the indexes tell me which guy from dim=2 is the best, but then this best guy has a dimenions of not 1, but 20. How can apply indexes to x now? In a for loop it would be like:</p>
<pre><code class="lang-auto">for i
    for j
        out = x[i,j,indexes[i,j],:]
</code></pre>
<p>The final shape of out is 5,10,20.</p>

G.M: <p>The same code works here, too. <a href="https://pytorch.org/docs/1.0.1/torch.html#torch.gather" rel="nofollow noopener">https://pytorch.org/docs/1.0.1/torch.html#torch.gather</a></p>

MattyMatt: <p>in order to use gather i need to have x and ids of the same shape except on dim.<br>
in my new case the shapes are<br>
x - (5,10,15,20)<br>
ids - (5,10)<br>
I can use unsqueeze to make shape of ids either (5,10,1) or (5,10,1,1) but it still wont match the shape of x.</p>

G.M: <pre><code class="lang-auto">&gt;&gt;&gt; import torch as tc
&gt;&gt;&gt; a=tc.randn(5,10,15,20,25)
&gt;&gt;&gt; b=a.argmax(2)
&gt;&gt;&gt; c=tc.gather(a,2,b.unsqueeze(2))
</code></pre>
MaxPool2d kernel size and stride

slavavs: <p>I have a picture 100x200. I want to make it 100x100 using nn.MaxPool2d. How do I set the size of the kernel and stride correctly?</p>

chenjesu: <p>Here is an example:</p>
<pre><code class="lang-auto">import torch

img = torch.zeros((128,3,200,100))

padded_image = torch.nn.functional.pad(img, (1,0,1,0), mode='replicate')
conv = torch.nn.Conv2d(3, 64, (2,2), stride=(1,1))
mp = torch.nn.MaxPool2d((2,1))

z = mp(conv(padded_image))

print(z.shape)
</code></pre>
<p>In this case we pad the image a bit, and convolve over 2x2 filters and then max pool to get the 100x100 image.<br>
You generally either want to use MaxPooling or Stride to shrink the image.  Convolution can shrink the image a bit, which is why I pad it, although because of how maxpool works you don’t actually need the pad.  You can also change the stride to something like (2,1) and not maxpool.<br>
You can print the size of the tensor after any of these operations to see how it affects the size.</p>
Does my model sees a rotated image?

Ibrahim_Yusuf: <p>I have an input image with shape (1, 3, 392, 196) but after calling transforms.ToTensor() it converts it to (1, 3, 196, 392), by swapping the width and height. I want to know whether my model is seeing a rotated version of the input image because this is crucial for my object detection task.</p>

ptrblck: <p>Could you post a reproducible code snippet, please?</p>

Ibrahim_Yusuf: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. I figured it out. I had a misconception regarding how the image will look after swapping H and W. Later I realized that it is correct, nothing changes about how the model sees the image. It is a thing about how arrays are represented. An image with height H and width W, when represented as an array will consist of H sub-arrays of width W: each sub-array is a row. So when Torch reports the dimension of the array it will count H sub-arrays of width W  and that is why we have the dimension as <strong>[H,W]</strong>.<br>
Thank you once again for offering an assistance.</p>
Error while Loading trained weight

yuyaya: <p>Hi, I transfer trained my original dataset after training <a href="https://github.com/PingoLH/FCHarDNet" rel="nofollow noopener">FCHarDnet</a> with CItyscapes data.</p>
<p>During loading the trained weight for inference, I met problem.</p>
<p>While transfer train FCHarDnet,<br>
I inserted the layer to match the class number.<br>
Like below</p>
<pre><code class="lang-auto"># While Training, I wrap HarDnet &amp; layer with nn.Sequential
model = FCHarDnet.to(device) 
model = nn.Sequential(model,
                      nn.Conv2d(19, out_channels=11, kernel_size=1, stride=1,padding=0, bias=True)).to(device)
</code></pre>
<p>Then, I loaded trained weight like below,</p>
<pre><code class="lang-auto"># While inference
model = FCHarDnet.to(device) 
model = nn.Sequential(model,
                      nn.Conv2d(19, out_channels=11, kernel_size=1, stride=1,padding=0, bias=True)).to(device)
state = convert_state_dict(torch.load(args.model_path)["model_state"])
model.load_state_dict(state)
</code></pre>
<p>While</p>
<pre><code class="lang-auto"># convert_state_dict function

def convert_state_dict(state_dict):
    """Converts a state dict saved from a dataParallel module to normal
       module state_dict inplace
       :param state_dict is the loaded DataParallel model_state
    """
    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = k[7:]  # remove `module.`
        new_state_dict[name] = v
    return new_state_dict

</code></pre>
<p>And then met error like this…</p>
<pre><code class="lang-auto">Missing key(s) in state_dict: "0.base.0.conv.weight", "0.base.0.norm.weight", "0.base.0.norm.bias", "0.base.0.norm.running_mean",
     :
     :
     :
Unexpected key(s) in state_dict: "base.0.conv.weight", "base.0.norm.weight", "base.0.norm.bias", "base.0.norm.running_mean",
     :
     :
     :
</code></pre>
<p>I assume weight couldn’t be loaded because of key name didn’t match?<br>
If that’s correct, will someone please share the code of how to fix this problem.</p>
<p>Thank you</p>

ptrblck: <p>Based on the error message it seems the <code>state_dict</code> might have been saved directly using the <code>FCHarDnet</code> model without wrapping it into an <code>nn.Sequential</code> container.<br>
Could this be the case?</p>
<p>If so, you could add the mitssing <code>0.</code> in front of the keys.<br>
However, this would also mean that the parameters of the conv layer weren’t saved.</p>

yuyaya: <p>Thank you for your advice.</p>
<p>This time,</p>
<blockquote>
<p><code>FCHarDnet</code>  model without wrapping it into an  <code>nn.Sequential</code>  container.<br>
Could this be the case?</p>
</blockquote>
<p>This was causing the problem.</p>
How to detect only Human using pytorch?

sudo_ranjith: <p>Hi folks,<br>
I would like to know HOW CAN I FIND ONLY HUMAN not other segments using pytorch or other like opencv.</p>
<p>till now I have tried with <strong>yolact</strong> and opencv (classifiers).<br>
but i could not get the expected result.<br>
let me know if you have any solutions or suggestions.<br>
Thanks in advance.</p>

ekilic: <p>Check <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">this sample.</a><br>
Then search for human segmentation dataset and train your model with your dataset.<br>
That’s all.</p>

sudo_ranjith: <p>thanks for your responce, I will check and let you know</p>
How to implement Grad-CAM on my model instead of VGG?

Ethan_Zhu1: <p>Hi,</p>
<p>I’ve found a lot of different examples of implementing Grad-CAM on VGG or other pre-trained models in torchvision, and I want to integrate it with my own trained model in order to visualize the feature importance. But the example code all used features and classifier method in VGG, which my model does not have. Is there any way to do it without having these two methods? Because I don’t want to retrained the model again.</p>
<pre><code>    self.model.features.zero_grad()
    self.model.classifier.zero_grad()
</code></pre>
<p>Thanks</p>

viniciusarruda: <p>Hi, I made <a href="https://gist.github.com/viniciusarruda/720bbc074da3c542c9061b04601e4257" rel="nofollow noopener">this</a> simple example based on <a href="https://medium.com/@stepanulyanin/implementing-grad-cam-in-pytorch-ea0937c31e82" rel="nofollow noopener">this</a> post.</p>
How to train only one model in multi models?

DamonC: <p>Hi, I have a question about training with multiple models in single GPU.<br>
I want to train only the first model and just feed output from the first model to other next models.<br>
To be more specific, I have three models.</p>
<ol>
<li>Pretrained Resnet50 : model1</li>
<li>Pretrained MLP : model2</li>
<li>Pretrained MLP2 : model3</li>
</ol>
<p>in here, I want to hold model 2) and 3) but train model 1).</p>
<p>First, I load pretrained model and weights</p>
<p>model1 = models.resnet50(pretrained=True)<br>
criterion1 = nn.L1loss()<br>
optimizer1 = torch.optim.Adam(model1.parameters(), lr=lr)</p>
<p>model2 = model2()<br>
optimizer2 = torch.optim.Adam(model2.parameters(), lr=lr)<br>
model2.load_state_dict(pretrained_model2[‘state_dict’]<br>
optimizer2 .load_state_dict(pretrained_model2[‘optimizer’]</p>
<p>model3 = model3()<br>
optimizer3 = torch.optim.Adam(model3.parameters(), lr=lr)<br>
model3.load_state_dict(pretrained_model3[‘state_dict’]<br>
optimizer3 .load_state_dict(pretrained_model3[‘optimizer’]</p>
<p>and then for training, I try to like below:</p>
<p>model1.train()<br>
model2.eval()<br>
model3.eval()</p>
<p>outputs = model1(img)<br>
outputs = model2(outputs)<br>
outputs = model3(outputs)</p>
<p>loss = criterion1(outputs, labels)<br>
optimizer1.zero_grad()<br>
loss.backward()<br>
optimizer1.step()</p>
<p>I make only one loss and optimizer for the first model1 to training, is this right to training my model?<br>
Thank you for your time.</p>

ptrblck: <p>The general work flow should be alright, although you don’t need to create optimizers for the last two models, if you are not training them.<br>
Also, you might want to freeze their parameters.</p>
Testing trained model, Pytorch 101

brucostare: <p>Hello Pytorch community,</p>
<p>I am just getting started out with Pytorch. I am trying to test a trained model. Following the Pytorch tutorial, I have this set up:</p>
<pre><code class="lang-auto">1    total = 0
2    correct = 0
3    with torch.no_grad():
4        for data in val_loader:
5            images, labels = data
6            outputs = model(images)
7            _, predicted = torch.max(outputs.data, 1)
8            total += labels.size(0)
9            correct += (predicted == labels).sum().item()
</code></pre>
<p>Where val_loader is a DataLoader(), and the model class inherits from the nn.Module.<br>
However, line 6 throws the following error:</p>
<pre><code class="lang-auto">$ CUDA_VISIBLE_DEVICES=2,3 python tools/test_net.py
Illegal instruction (core dumped)
</code></pre>
<p>I am using Pytorch version 1.1.0, with a CUDA version of 10.2, and cuda driver version 440.33.01. Any help would be much appreciated!</p>
<p>Thank you in advance for your help!</p>

ptrblck: <p>Does your code run on the CPU without any error?<br>
If so, could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> on a single GPU and post the stack trace here?</p>
Selecting sub part of tensor and the indices of selected elements

mahsa: <p>Assume we want to select all the elements in the tensor which are greater than 5. I know that the following code can do that:</p>
<pre><code class="lang-auto">B = A[A &gt; 5]
</code></pre>
<p>But I want to know the indices of the selected elements. Is there any way to select and find the indices of the selected elements?</p>
<p>Thanks</p>

ptrblck: <p><code>nonzero()</code> should work:</p>
<pre><code class="lang-python">A = torch.randint(0, 10, (10, 10))
idx = (A &gt; 5).nonzero()
</code></pre>