akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Load_state_dict does not work.Help needed for copying models

username: sanket_desai
cooked: <p>Hello I am currently on pytorch version 1.8.0, and I tried to do a basic  copy of two models.<br>
Actor1.load_state_dict(Actor2.state_dict())</p>
<p>This doesn’t work and the error I get is:<br>
Expected type ‘OrderedDict[str, Tensor]’, got ‘Dict[str, Tensor]’ instead.<br>
I have scoured the internet for an answer to this but to no avail.Please help me.</p>
<p>EDIT: I have checked all my previous programs and everywhere the load_state_dict method has broken.So basically this has broken over 13 to 14 of my programs. Please check this fast.</p>

username: Dwight_Foster
cooked: <p>you need to do</p>
<pre><code class="lang-auto">Actor1.load_state_dict(torch.load(Actor2.state_dict()))
</code></pre>

username: sanket_desai
cooked: <p>When did this change? I have been using Actor1.load_state_dict(Actor2.state_dict()) for months now.</p>

username: Dwight_Foster
cooked: <p>I am not sure I have always had to to it this way.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I an not sure why you actually need to do it this way. This is just a typing issue that you should ignore and just use <code>Actor1.load_state_dict(Actor2.state_dict())</code>.</p>
<p>I opened an issue here to fix this: <a href="https://github.com/pytorch/pytorch/issues/55302" class="inline-onebox">Typing issue on state_dict/load_state_dict · Issue #55302 · pytorch/pytorch · GitHub</a></p>
The tsne plot of Generated features are not coinciding with original features using GAN

username: ankitjha16
cooked: <p>Hi,</p>
<p>I am using GAN for feature generation, but the tsne plot of generated one is not overlapping or nearby the original features.</p>
<pre><code class="lang-auto">x1 -&gt; x2
</code></pre>
<p>x1 (original feature)<br>
x2 (generated feature)<br>
Can anyone suggest the solution?<br>
Thanks</p>

username: vdw
cooked: <p>Disclaimer: I’m really not sure what you’re trying to do :). I assume you have two approaches for dimensionality reduction (t-SNE and GAN) and try to compare the results.</p>
<p>Well, t-SNE is a probabilistic approach where the distances between data points in the original space and the lower-dimensional space get represented by (conditional) probabilities, the optimization goal is to minimize the difference between this two resulting probability distributions.</p>
<p>That’s probably net really enlightening but the important part is (a) different runs of t-SNE will give you different results – depending in the random initialization, and (b) t-SNE does not guarantee to preserve things like cluster sizes, orientations, shapes or things you might preserve with linear approaches such as PCA or LDA.</p>
<p>In short, it’s not obvious to me why the generation features of your 2 methods should overlap.</p>

username: ankitjha16
cooked: <p>Thanks <a class="mention" href="/u/vdw">@vdw</a> for giving the more insights. Actually I am trying to see the orientations of generated samples (using GAN) with the original features. And after several experiments I found that GAN is not properly working (not able to generate features properly). So to check the orientations, I used the tsne plots. But yeah you are right. I just want to use GAN for feature generation from one domain to other.<br>
Thanks</p>
What CUDA compute capability?

username: Ingresso
cooked: <p>Hello, i want to start working with PyTorch, but my graphic card is to old (compute capability 3.0).<br>
So i decided to buy a newer one.</p>
<p>The problem is that i can’t find an info what compute capability i need at minimum?<br>
I read a lot of threads where members said that 5.0 is also to old.<br>
So maybe 6.1 is to old too or may be deprecated soon.</p>
<p>What graphic card i have to buy so that i can work with pytorch?<br>
4.000$ for a TITAN V is a little bit to much for me.</p>
<p>Regards</p>

username: albanD
cooked: <p>Hi,</p>
<p>5.2 should keep being supported for a while as lots of people use <code>Titan X</code> cards.<br>
The 10xx cards should be as well.<br>
The reason for removing the 2.x support is that so many features are missing that GPU implementation would need to be so different.<br>
Finally for 3.0+ cards, you can still compile from source and use them. It is just not in the binary versions because the number of people using them is low and it makes the binary huge (500MB+).</p>

username: Gelassen
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>I have similar question - where could I find information about CUDA version vs compute capability range?</p>
<p>In my understanding upper versions of CUDA drivers increase demand to compute capability. In my specific case I have card  GeForce GTX 860M, which compute capability is 3.0/5.0. I installed the latest CUDA toolkit (11.2) and v460.67 driver. I want to verify that this version of CUDA toolkit fit my hardware as I also noticed <a href="https://discuss.pytorch.org/t/pytorch-with-cuda-11-compatibility/89254">pytorch and CUDA mismatch</a> toolkit is also possible</p>

username: ptrblck
cooked: <p>You could check the <a href="https://en.wikipedia.org/wiki/CUDA">CUDA - Wikipedia page</a>, where the CUDA toolkits with the supported compute capabilities are listed or e.g the <a href="https://docs.nvidia.com/cuda/archive/11.0/cuda-toolkit-release-notes/#deprecated-features">CUDA11 release notes</a>, which also mention the deprecation:</p>
<blockquote>
<p>Support for the following compute capabilities are deprecated in the CUDA Toolkit:</p>
<ul>
<li>sm_35 (Kepler)</li>
<li>sm_37 (Kepler)</li>
<li>sm_50 (Maxwell)</li>
</ul>
</blockquote>

username: Gelassen
cooked: <p>Thank you for extra information. My card is fit in Maxwell group. CUDA 11.0 and newer do not support it.</p>

username: ptrblck
cooked: <p>The CUDA11.0 release notes claim that <code>sm_50</code> is deprecated, but not removed, so you should still be able to build CUDA applications for it with the mentioned CUDA toolkit.</p>
[Tensorboard] How to overwrite checkpoint of same global step?

username: tlim
cooked: <p>If you resume training from a specific epoch (same global step) and run tensorboard, tensorboard (add_scalar) will end up plotting 2 points for that particular global step.</p>
<p>Example, I want trying to test whether changing the learning rate halfway through training will improve/worsen the accuracy:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35ae5a9f8f0dc3c179e4c9112deecb64aba46b24.png" alt="Screenshot from 2021-04-05 17-51-45" data-base62-sha1="7ESRaNZJsP3sEVnQa6PeouPabIM" width="374" height="424"></p>
<p>The plots for same time step are plotted twice (I resumed from +15 epochs behind the latest epoch).</p>
<p>Search up the web, cannot find any commands that can ask tensorboard to just overwrite the previous saved checkpoint for the new one.</p>
Where is training and validation loss in MaskRCNN? Needed for fine-tuning hyperparameters

username: mariamsanchezs
cooked: <p>Hi everyone,</p>
<p>I am a complete beginner in deep learning, and I am trying to use this repo (<a href="https://github.com/mrlooi/rotated_maskrcnn" class="inline-onebox" rel="noopener nofollow ugc">GitHub - mrlooi/rotated_maskrcnn: Rotated Mask R-CNN: From Bounding Boxes to Rotated Bounding Boxes</a>) which extends MaskRCNN.</p>
<p>I don’t really know which is the best way to fine tune the hyperparameters without under or overfitting. I read that a good point of view is plotting training and validation loss, and finding the number of iterations that make them diverge.</p>
<p>However, I can’t find which variables are those values stored in, so I can’t plot them. Does anybody know it or know what could I do to find them?</p>
<p>I am using GoogleColab and TorchVision=1.1.0.</p>
<p>Thank you all!!</p>
Train neural network with GPU AMD

username: Daniele_Angelini
cooked: <p>Hu guys i follow this project in GitHub</p>
<p><a href="https://github.com/LearnedVector/A-Hackers-AI-Voice-Assistant" class="inline-onebox" rel="noopener nofollow ugc">GitHub - LearnedVector/A-Hackers-AI-Voice-Assistant: A hackers AI voice assistant, built using Python and PyTorch.</a> based pytorch all’ going ok but when i train neural network i use this Command …<br>
when i train neural network based pytorch  i use this Command :</p>
<p>python3 train.py --sample_rate 8000 --epochs 100 --batch_size 32 --eval_batch_size 32 --lr 0.0001 --model_name ./new_wakeword_v0 --save_checkpoint_path neuralnet/checkpoints --train_data_json data/json/train.json --test_data_json data/json/test.json --num_workers 4 --hidden_size 64</p>
<p>I have GPU AMD there is a way to use amd video card …? I install CUDA-toolkit now</p>
<p>Regards</p>
Different rmse values for same evaluation but different batch size

username: Javier_Rodriguez_Pui
cooked: <p>Hi everyone! I am evaluating my network and I realize that my RMSE metric is not always the same. It depends of the batch_size.<br>
Here is my code.:</p>
<pre><code class="lang-auto">            data_loader = DataLoader(....., batch_size, True, workers, stage="test")
            rmse = []
            for batch_idx, sample in enumerate(data_loader):
                sample['color'] = sample['color'].cuda()
                sample['gt'] = sample['gt'].cuda()

                mu = self.model(sample['color'])

                sample['gt'] = sample['gt'].view(-1)
                mu = mu.view(-1)
                rmse.append( compute_rsme(sample['gt'], mu))
            print(rmse.mean())
</code></pre>
<p>Any idea why? any help would be great =)</p>

username: ptrblck
cooked: <p>I guess your code broadcasts some tensors.<br>
Could you check the shape of <code>sample['gt']</code> and <code>mu</code> and make sure they have the same number of dimensions with the same shape?</p>

username: Javier_Rodriguez_Pui
cooked: <p>Hi, they have the same shape. How can i check what is being broadcasting? I also checked model.eval() and with torch.no_grad() just in case.</p>

username: ptrblck
cooked: <p>Could you save an example tensor for <code>sample['gt']</code> as well as <code>mu</code> and post an executable code snippet to reproduce the difference you are seeing?<br>
Maybe you could also post a way to randomly initialize both tensors to recreate the issue.</p>

username: Javier_Rodriguez_Pui
cooked: <p>I forgot to reply, I was averaging the rmse values but i should average the mse and then do the root square, to obtain the global rmse.</p>
Is a symmetric softmax output for 2d array possible?

username: Hybrid_SyntaX
cooked: <p>Hi,<br>
I have a (n,1,k,k) tensor; n is the batch size and k is number of different objects in a picture.<br>
I’m trying to create a network that produces the similarity between different objects in a given picture.<br>
I was wondering if there’s any method that produces a symmetric softmax. (maybe something that produces a 'diagonal-like matrix" where each object has the most similarity with itself)</p>
<p>This  is what my forward method looks like, but the output isn’t symmetrical.<br>
I also tried using SoftMax2D but all elements became 1.</p>
<pre><code class="lang-auto">def forward(self, v_pc):
        vdot_pc = self.pg_layers(v_pc)
        vdot_pc=vdot_pc.squeeze() # (:,100,100) (k*k)

        vdot_pc = torch.add(vdot_pc, vdot_pc.transpose(1, 2)) 

        out = nn.functional.softmax(vdot_pc, dim = 2)

        return out
</code></pre>
<p>I could try making the output symmetrical by adding it to its transpose, but I’m not sure which dimension would be appropriate for this task.</p>
<p>I’d appreciate your help.</p>
How to count the number of zero weights in a pytorch model

username: pgadosey
cooked: <p>Hi, after training a pytorch model, how do you count the total number of zero weights in the model.?</p>

username: jpeg729
cooked: <p>How about this?</p>
<pre><code class="lang-auto">def countZeroWeights(model):
    zeros = 0
    for param in model.parameters():
        if param is not None:
            zeros += torch.sum((param == 0).int()).data[0]
    return zeros</code></pre>

username: pgadosey
cooked: <p><a class="mention" href="/u/jpeg729">@jpeg729</a> thanks. It works</p>

username: theevann
cooked: <p>You could also go for :</p>
<pre><code class="lang-python">zeros += param.numel() - param.nonzero().size(0)
</code></pre>
<p>I am not sure which one is the fastest, but I find this way a bit clearer.</p>

username: pgadosey
cooked: <p><a class="mention" href="/u/theevann">@theevann</a>, <a class="mention" href="/u/jpeg729">@jpeg729</a> thanks for the help i found that they both work. I have one more question to ask regarding this. If I want to remove the zeros and resave or store the model as a 2-bits instead of the the default float 32 form of a pytorch model , how do I do that?</p>

username: hughperkins
cooked: <p>You could store the positions of all the non-zeros:</p>
<pre><code>weights.nonzero()
</code></pre>
<p>(You can then use <code>scatter</code> to fluff the model back up again, after loading (edit: you might need to <code>.view(-1)</code> the tensor before running .nonzero(), and then reshape it after loading/scattering)).</p>

username: itsjoshthedeveloper
cooked: <p><a class="mention" href="/u/jpeg729">@jpeg729</a> I tested both and it seems that <a class="mention" href="/u/theevann">@theevann</a>’s method is faster (by a factor of 3). Also, I would recommend using <code>.item()</code> instead of <code>.data[0]</code>.</p>
How shift an image vertically or horizontally?

username: Shantanu_Ghosh
cooked: <p>If we want to shift an image in tensorflow we can do that using:</p>
<p>datagen = ImageDataGenerator(width_shift_range = width_shift_val,<br>
height_shift_range= height_shift_val)</p>
<p>Is there a way to do the same in pytorch?</p>
<p>Using transform, we can do a RandomHorizontalFlip, but I want to shift an image either vertically or horizontally in Pytorch?</p>

username: Shisho_Sama
cooked: <p>You can use <a href="https://github.com/aleju/imgaug" rel="noopener nofollow ugc">imgaug</a> with Pytorch and use its Fliplr and Flipud.<br>
For a sample check <a href="https://github.com/aleju/imgaug/issues/406" rel="noopener nofollow ugc">this</a> out</p>

username: Shantanu_Ghosh
cooked: <p>Its good, but it has the feature to flip the image not to shift the image like keras</p>

username: Shisho_Sama
cooked: <p>Then use TranslateX and TranslateY.<br>
imgaug is a decent library and it has all you need. just make sure to check the documentation.</p>

username: Sanjan_Das
cooked: <p>You should be able to do this using <code>torchvision.transforms.RandomAffine</code>. You can set the <code>degrees</code> to 0 and set the <code>translate</code> parameter to what you need.<br>
<a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.RandomAffine" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.RandomAffine</a></p>
<pre><code class="lang-auto"># An example. Note that translate refers to the fraction of the length of that dimension to translate by.
shift = transforms.RandomAffine(degrees = 0, translate = (0.2, 0.2))
</code></pre>

username: Shantanu_Ghosh
cooked: <p>Thank you for the solution</p>
QAT with RNN -- RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

username: roldja
cooked: <p>I’ve seen several posts about this runtime error:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16]] is at version 9; expected version 8 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>However, this only arises in my cases if I am running QAT</p>
<pre><code class="lang-auto">torch.quantization.prepare_qat(model, inplace=True)
</code></pre>
<p>After setting anomaly detection, I can see the following which leads me to believe it’s the quantization observers getting updated in-place and subsequently blowing up during the backward call:</p>
<blockquote>
<p>…/torch_1_7_0/lib/torch/quantization/fake_quantize.py", line 100, in forward<br>
self.ch_axis, self.quant_min, self.quant_max)<br>
(function _print_stack)</p>
</blockquote>
<p>Any advice here? Thanks!</p>

username: TYC
cooked: <p>Hi, I have the same issue. Did you get it solved?</p>
How to configure the shared memory size of CUDA kernel in CUDA/C++ extension?

username: huangdi
cooked: <p>Hi.</p>
<p>I’m writing an pytorch operator using CUDA/C++ extension, and I need to use more than 48KB shared memory size in each thread block. The platform is GPU V100.</p>
<p>According to <a href="https://stackoverflow.com/questions/63757245/using-maximum-shared-memory-in-cuda" rel="noopener nofollow ugc">this post on stackoverflow</a>, when the shared memory size exceeds 48KB, I need to call <code>cudaFuncSetAttribute</code>  in the host function,  and then set the shared memory size.</p>
<p>My question is, is there any pytorch api can help me to do this? Just like the <code>at::cuda::getCurrentCUDABlasHandle()</code> api for getting current cublas handle instead of creating a new one.</p>
<p>Thanks!</p>
Build Pytorch from Source offline windows

username: Jackson_Scott
cooked: <p>Hello all, I am in need of building Pytorch 1.6.0 from source offline.</p>
<p>Originally, this came up because some machines utilizing pytorch needed libtorch (the conda distribution of Pytorch 1.6.0 was already on the systems), there is a lengthy approval for our IT. Finally, libtorch was brought onboard, however it is 1.7.1.<br>
When trying to forward propagate on a model in c++ which was serialized on Pytorch and loaded by libtorch my program simply errors out. After digging around online I am wondering if it is due to version mismatch. Therefore (since Pytorch 1.6.0 has already been approved by our IT), I will just build from source Pytorch/Libtorch. One issue, I have to do this offline.</p>
<p>Before I brought Pytorch in I:</p>
<pre><code class="lang-auto">git clone https://github.com/pytorch/pytorch.git
git checkout v1.6.0
git submodule update --init --recursive
</code></pre>
<p>Then I moved my distribution to the offline machine. Once on the machine:</p>
<pre><code class="lang-auto">set CMAKE_GENERATOR=Visual Studio 16 2019
set DISTUTILS_USE_SDK=1
for /f "usebackq tokens=*" %i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=%CMAKE_GENERATOR_TOOLSET_VERSION%
set CUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX64\x64\cl.exe

python setup.py build --cmake
</code></pre>
<p>Now, this will be as best as I can put it, I cannot copy paste mass log dumps due to the offline restriction but it appears to run as normal until:</p>
<pre><code class="lang-auto">Performing download step (git clone) for 'psimd'
</code></pre>
<p>Which, obviously fails. Then the procedure continues and eventually crashes, after checking the CmakeErrorLog.txt I am shown errors such as</p>
<pre><code class="lang-auto">cannot find include file 'pthread.h'
</code></pre>
<p>I am marking this as deployment since that seems to be the center of what I am describing here, however, as the reader can see the root of my issue stems from (what I think is) pre-compiled release Pytorch 1.6.0 not playing nicely with pre-compiled libtorch 1.7.1, if anyone has insight there I would much appreciate it.</p>
<p>Expected behavior:</p>
<ul>
<li>everything work normally, I can either use my existing pytorch/libtorch distributions fine no problem, OR I can build from source offline with no issues</li>
</ul>
<p>My Environment:</p>
<ul>
<li>Windows 10</li>
<li>Visual Studio 16 2019</li>
<li>Python 3.7</li>
<li>Everything that came with anaconda</li>
</ul>
<p>Additional notes:</p>
<ul>
<li>torchvision is not on the system so no I cannot follow the exact minimum reproducible example exactly for deploying a model to c++ from python</li>
</ul>
Customized RNN step functions

username: ajfisher
cooked: <p>Hi!</p>
<p>I’m new to this area and apologize if I’m asking something obvious, or asking in the wrong place. Please feel free to point me elsewhere!</p>
<p>Is there an equivalent of tf.keras.backend.rnn in pytorch? Something that takes a step_function as input, but automates the loop over a time dimension to make the computation more streamlined? Right now I’m just using a regular loop, and tracing the operation. It’s fast, but running similar code with tf.keras.backend.rnn is still substantially faster. The closest functions I can find all implement a specific step function (e.g., an LSTM layer), and don’t allow for a user defined step function.</p>
<p>Thanks very much!</p>
Negative momentum for GAN training

username: hamhamham
cooked: <p>I read this <a href="http://proceedings.mlr.press/v89/gidel19a.html" rel="noopener nofollow ugc">interesting paper on game dynamics</a>, which suggests using negative momentum for GAN training.</p>
<p>I wasn’t able to implement this in PyTorch using sgd as momentum &lt; 0 is not accepted as an input to the optimizer.  Has anybody tried to implement this and figured out a workaround, or can offer any advice?</p>
<p>Thanks</p>

username: AlphaBetaGamma96
cooked: <p>You could try copying the base class for SGD and just modify it to support negative momentum values?<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD</a></p>

username: hamhamham
cooked: <p>Thanks very much - it worked.</p>
Using NLLLoss to calculate the loss

username: joepareti54
cooked: <p>I do not understand how the loss is calculated: the output tensor is logarithmic, while the labels are not.</p>
<pre><code class="lang-auto">model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 5
for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        # Flatten MNIST images into a 784 long vector
        images = images.view(images.shape[0], -1)
    
        # TODO: Training pass
        optimizer.zero_grad()
        
        output = model.forward(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()
        
        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")
</code></pre>

username: omarfoq
cooked: <p>Hello,</p>
<p>The formula is given <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">here</a>.</p>
How to vectorize this matrix equation?

username: dangne
cooked: <p>Hi, I have the following matrix equation but I’m not sure how to implement this in PyTorch without using for loop?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c37281a7f46bc931c8377b780f9208fae440100f.png" alt="image" data-base62-sha1="rT0r0Tkt99dP6oFpjUUn5n964YT" width="137" height="42"></p>
<p>Thanks a lot.</p>

username: googlebot
cooked: <p>exclude:</p>
<p>t=a*b<br>
s=t.sum(dim,keepdim=True)<br>
c=s-t</p>

username: Sanjan_Das
cooked: <p>A solution along the same logic of the answer given by <a class="mention" href="/u/googlebot">@googlebot</a>, but using PyTorch functions:</p>
<blockquote>
<p>C = torch.sum(torch.mul(A, B)) - torch.mul(A, B)</p>
</blockquote>
<p>This should give you what you want! Assumption here is that A and B are 1D tensors, of course.</p>
Class level IoU for Unet Semantic Segmentation

username: Razin_Hussain
cooked: <p>Hi,<br>
I wanted to calculate class level IoU for my Unet semantic segmentation model. But I think I am not calculating it right. I am using the below code:</p>
<p>def test(dataset, model, device, test_loader):<br>
flaga = False<br>
# set the model to evaluation<br>
model.eval()<br>
iou_score = 0</p>
<pre><code>y_testiList = []
y_prediList = []
#predList = np.array([])
predList = []
#groundList = np.array([])
groundList = []
with torch.no_grad():

    for _data, _target in test_loader:
        # convert from 3-channel image to 1-channel image 
        _target = _target.sum(3, keepdim=True)
        # send data to evice
        _data, _target = _data.to(device), _target.to(device)
        # forward
        output = model.forward(_data)
        
        pred = output.argmax(3, keepdim=True)
        
        # convert to numpy
        _target = _target.cpu().numpy()
        pred = pred.cpu().numpy()
        iou_score += calculate_iou(pred, _target)

        groundList = np.append(groundList,_target)

        predList = np.append(predList,pred)


print(predList.shape,groundList.shape)

Yi = groundList
print(Yi[1])
y_predi = predList
print(y_predi[1])
IoUs = []
#Nclass = int(np.max(Yi)) + 1
Nclass = 5
for c in range(Nclass):
  TP = np.sum( (Yi == c)&amp;(y_predi==c) )
  FP = np.sum( (Yi != c)&amp;(y_predi==c) )
  FN = np.sum( (Yi == c)&amp;(y_predi != c)) 
  IoU = TP/float(TP + FP + FN)
  print("class {:02.0f}: #TP={:6.0f}, #FP={:6.0f}, #FN={:5.0f}, IoU={:4.3f}".format(c,TP,FP,FN,IoU))
  IoUs.append(IoU)
mIoU = np.mean(IoUs)
#print("_________________")
print("Mean IoU: {:4.3f}".format(mIoU))

riou_score = mIoU
iou_score /= len(test_loader.dataset)
print("mIoU score : ",iou_score)

return iou_score
</code></pre>
<p>I am getting results where I find only the first class IoU. But for other classes I am not getting any IoU. Result is given below:</p>
<p>class 00: #TP=   698, #FP=    16, #FN=74459, IoU=0.009<br>
class 01: #TP=     0, #FP=    81, #FN= 3941, IoU=0.000<br>
class 02: #TP=     0, #FP=     0, #FN= 2590, IoU=0.000<br>
class 03: #TP=     0, #FP=     0, #FN= 1699, IoU=0.000<br>
class 04: #TP=     0, #FP=     0, #FN= 1293, IoU=0.000</p>
<p>I am not sure how to interpret it and how can I get class level IoU. Any help would be highly appreciated.</p>
<p>Thanks in advance<br>
Razin Farhan Hussain</p>
In lstm number of sequence in time series data

username: bc060400164
cooked: <p>What does the number of sequence mean in lstm? if i give 1 or more than 1 then what it will do?</p>

username: Dwight_Foster
cooked: <p>The sequence length is the number of timesteps you give the lstm. So for example if you were trying to guess the next word the sequence length would be how many previous words you give it to guess the next one. If you set it to more than 1 than the lstm will need more than one timestep as the input.</p>
TypeError: trace(): argument 'input' (position 1) must be Tensor, not method

username: CracKCatZ
cooked: <p>Hey guy’s,<br>
when I try to run my script I get this weird error and I don’t know how to fix it or what this error is caused by. This is the error:<code>TypeError: trace(): argument 'input' (position 1) must be Tensor, not method</code><br>
I hope that some of you guy’s can help me with this error and explain to me what this error is caused by.</p>
<p>This is my code:</p>
<pre><code class="lang-auto">import argparse
import torch
from model import SpeechRecognition
from collections import OrderedDict

def trace(model):
    model.eval()
    x = torch.rand(1, 81, 300)
    hidden = model._init_hidden(1)
    traced = torch.trace(model.forward)(example_inputs=(x, hidden)) 
    return traced

def main(args):
    print("loading model from", args.model_checkpoint)
    checkpoint = torch.load(args.model_checkpoint, map_location=torch.device('cpu'))
    h_params = SpeechRecognition.hyper_parameters
    model = SpeechRecognition(**h_params)

    model_state_dict = checkpoint['state_dict']
    new_state_dict = OrderedDict()
    for k, v in model_state_dict.items():
        name = k.replace("model.", "") # remove `model.`
        new_state_dict[name] = v

    model.load_state_dict(new_state_dict)

    print("tracing model...")
    traced_model = trace(model)
    #print(type(traced_model))
    print("saving to", args.save_path)
    traced_model.save(args.save_path)
    print("Done!")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="testing the wakeword engine")
    parser.add_argument('--model_checkpoint', type=str, default='checkpoint_file', required=False,
                        help='Checkpoint of model to optimize')
    parser.add_argument('--save_path', type=str, default='path/where/the/optimized/model/should/be/saved', required=False,
                        help='path to save optmized model')

    args = parser.parse_args()
    main(args)

</code></pre>
<p>This is the full error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "c:/Users/supre/Documents/Python Programme/neuralnet/optimize_graph.py", line 45, in &lt;module&gt;
    main(args)
  File "c:/Users/supre/Documents/Python Programme/neuralnet/optimize_graph.py", line 30, in main
    traced_model = trace(model)
  File "c:/Users/supre/Documents/Python Programme/neuralnet/optimize_graph.py", line 11, in trace
    traced = torch.trace(model.forward)(example_inputs=(x, hidden))
TypeError: trace(): argument 'input' (position 1) must be Tensor, not method
</code></pre>
<p>Thank’s for every help in advance:)</p>

username: Dwight_Foster
cooked: <p>It is because your inputs to the forward function are in the wrong parenthesis. You are just passing the forward method into the trace function not the output. To fix it just do this</p>
<pre><code class="lang-auto">     traced = torch.trace(model.forward(example_inputs=(x, hidden)))

</code></pre>
Different batches take different times to load

username: Pramodith
cooked: <p>I’m using the snippet given below to measure the amount of time each batch takes to load.</p>
<pre><code class="lang-auto">st_batch= time.time()
torch.cuda.synchronize()
                
for cnt, batch in enumerate(train_data_loader):
            torch.cuda.synchronize()
            end_batch= time.time()
           print(end_batch-st_batch)
          # do something
           st_batch=time.time()
</code></pre>
<p>I observe that some batches take 3-5 seconds whereas others take just 0.002s<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3244af6126082ad932e520a46e5739df5a522593.png" data-download-href="https://discuss.pytorch.org/uploads/default/3244af6126082ad932e520a46e5739df5a522593" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/3244af6126082ad932e520a46e5739df5a522593.png" alt="image" data-base62-sha1="7aH1THrUt3aUwLy9EqSvLjKuRVx" width="690" height="272" data-dominant-color="0E0F0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1071×423 13.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I assume this is a bottleneck while training my network is there anyway around this?</p>

username: ptrblck
cooked: <p>Are you using multiple workers in your <code>DataLoader</code>?<br>
If so, could you try to increase the number?<br>
It looks like your training procedure has to wait sometimes for the <code>DataLoader</code> to provide new batches. Also, is your data locally on an SSD?</p>

username: Pramodith
cooked: <p>There was an overhead in my <strong>getitem</strong> function , I fixed it and it works fine now.</p>

username: ammary-mo
cooked: <p>Hi <a class="mention" href="/u/pramodith">@Pramodith</a> , can you share what the overhead in your <strong>getitem</strong> was?</p>
Recommendations for how to best handle a generic module wrapper with jit.script?

username: dasturge
cooked: <p>consider wrappers such as nn.DataParallel.  Ignoring the other specifics of what that class is doing, the basic pattern is that you have a class which simply wraps the function of a base module:</p>
<pre><code class="lang-auto">class Wrapper(nn.Module):
    def __init__(self, base_module: nn.Module):
        super().__init__()
        self.base_module = base_module
</code></pre>
<p>the real trouble comes with generically wrapping a forward pass:</p>
<pre><code class="lang-auto">    def forward(self, *args, **kwargs):
        self.base_module(*args, **kwargs)
</code></pre>
<p>I would love to see this wrapper itself be jit scriptable so long as the base_module is.  In my particular use case, I actually don’t need to add anything to this forward pass.  I know that I may simply have to require the forward method to only take a single argument, but I would like to see if anyone has handled this in other ways.</p>

username: googlebot
cooked: <p>Depends on why you want to have such a forwarder compiled. But basically you’ll have to either avoid python’s star syntax (there are multiple jit-compatible ways to group arguments, but they’re intrusive wrt base_module - i.e. all wrapped modules need to implement a special interface) or have some non-compiled code in your model.</p>
Tensor assignment

username: ashwani-bhat
cooked: <p>Hi. Please look through the below dummy code. I want to assign dataset[1][1] with x, but its giving an error.</p>
<pre><code class="lang-auto">from torch.utils.data.dataset import TensorDataset
import torch

s1 = torch.tensor([1, 11, 21, 31], dtype=torch.long)
s2 = torch.tensor([2, 12, 21, 21], dtype=torch.long)
s3 = torch.tensor([3, 33, 44, 55], dtype=torch.long)

dataset = TensorDataset(s1, s2, s3)

x = dataset[0][1]
dataset[1][1] = x
</code></pre>
<pre><code class="lang-auto">      1 x = dataset[0][1]
----&gt; 2 dataset[1][1] = x

TypeError: 'tuple' object does not support item assignment
</code></pre>
<p>How can I perform such assignment?</p>

username: aza
cooked: <p>After playing around with this for a few minutes, it seems that since TensorDataset is a dataset class and not a tensor itself – it won’t support assignment. You can manipulate the values inside of the tensors that your ‘dataset’ is holding (for example try to subtract one by 2) but assignment of a tuple is not possible (which is what is returned when you index into the dataset)</p>
How to solve ResNet Overfitting

username: iAbhyuday
cooked: <p>I am training a Boat Type classifier using dataset (<a href="https://www.kaggle.com/clorichel/boat-types-recognition" rel="noopener nofollow ugc">https://www.kaggle.com/clorichel/boat-types-recognition</a>) with 75-25 train-validation split.</p>
<p><strong>Dataset is imbalanced.</strong><br>
to counter imbalance I did</p>
<ol>
<li>used train_test_split with stratify to divide the data.</li>
<li>Over/Under Sampled the dataset using WeightedRandomSampler</li>
<li>Added data augmentation for train and validation sets. (CenterCrop, RandomRotation([0,30]), etc …)</li>
</ol>
<p>I used pretrained resnet18 as the backbone, and modified  the last fc layer for classification.<br>
I am using ReduceLROnPlateau scheduler.</p>
<p>still model is getting overfitted.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26f7e29c483d596f1b8d6d5ffdb4d7cf12a9ccf6.jpeg" alt="train (3)" data-base62-sha1="5yJeB7lc1CoFpjcEBOfqPe7Qfga" width="432" height="288"></p>

username: aktgpt
cooked: <p>Not sure exactly if this is overfitting. Validation loss didn’t increase by that much. You should run this for longer and see if the validation loss increases further.</p>
<p>One way to reduce overfitting in transfer learning is to freeze the initial layers and then train your network. In the case of ResNet, you can freeze the <code>conv1</code>, <code>conv2</code>, and <code>conv3</code> layers and see if that helps.</p>

username: iAbhyuday
cooked: <p>I tried</p>
<ol>
<li>Freezing Resnet18 upto layer3 module  (got train_acc 90% and validation acc 68%)</li>
<li>Freezing resNet18 upto layer4 module and changed fc layer (got train_acc 99% and validation acc 79%)</li>
</ol>
<p>Same problem I faced with Inceptionv3, resnet34 and resnet50</p>

username: aktgpt
cooked: <p>I see. In that case, you can try increasing augmentation (ColorDistortion, Solarize, ColorJitter, GaussianFilter etc.) and introduce Dropout in the layers.</p>
<p>Hope that helps.</p>

username: iAbhyuday
cooked: <p>Here is my current implementation <a href="https://www.kaggle.com/iabhyuday/notebook5b80d89625" rel="noopener nofollow ugc">https://www.kaggle.com/iabhyuday/notebook5b80d89625</a><br>
Have a look, I’ve tried everything still don’t know whats wrong</p>

username: iAbhyuday
cooked: <p>[UPDATE]</p>
<p>Got better results after switching to SGD from Adam Optimizer. But validation accuracy and loss saturate early .<br>
Learning curves (before early stopping)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ebb329817637565687739ed3594d7c57ca2af8c.jpeg" alt="train" data-base62-sha1="beujksAVopbKKqfy87PL8XtKiIs" width="432" height="288"></p>
Data augmentation with transform after create dataloader

username: Andre_Almeida
cooked: <p>Hi!</p>
<p>I’m trying to automate a training pipeline for my project with pytorch and sklearn cross-validation.<br>
I know I can do transformations while creating the dataset, but in the pipeline I first concatenate all data to split with the cross-validation method. So, if I use the transform in the dataset creation the transformations will be applied to training and testing data.</p>
<p>I would like to know if it is possible to make the transformations in the data after creating the dataloader. This way I can separate the transforms for train and test.</p>
Custom RNN GMU implementation

username: Monne
cooked: <p>Hi,<br>
So there is a journal about other RNN besides LSTM and GRU and i want to implement it in Pytorch,</p>
<p>this is forward propagation formula</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/739f1debbb7e5dae64d399a2ebff5db3b77eec67.png" alt="image" data-base62-sha1="guPOyyqDiWsjDP4QRQDJl9bRYQT" width="381" height="500"></p>
<p>also backward formula</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bdee0398cf1189ac67c74c5a12d132034f39131.png" alt="image" data-base62-sha1="aPbjOJDgiQAxOnpjoBsyis1Whfb" width="399" height="336"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/5/956c3b7aa94c8007150102cec6732c10a927cf6d.png" alt="image" data-base62-sha1="ljR1QCaY7MXzDeovMRQ3qsu51Ol" width="400" height="164"></p>
<p>and this is my implementation in Pytorch.</p>
<pre><code class="lang-auto">class JitGMUCell(jit.ScriptModule):
    
    def __init__(self, input_size, hidden_size):
        super(JitGMUCell, self).__init__()

        self.hidden_size = hidden_size
         
        self.weight_ih = nn.Parameter(torch.Tensor(3 * hidden_size, input_size))
        self.weight_hh = nn.Parameter(torch.Tensor(3 * hidden_size, hidden_size))
        
        self.weight_ch_m = nn.Parameter(torch.Tensor(hidden_size))
        self.weight_ch_o = nn.Parameter(torch.Tensor(hidden_size))

        self.bias_ih = nn.Parameter(torch.Tensor(3 * hidden_size))            
        self.bias_hh = nn.Parameter(torch.Tensor(3 * hidden_size))

        self.reset_parameter()
    
    @jit.ignore(drop=True)
    def reset_parameter(self):
        stdv = 1.0 / math.sqrt(self.hidden_size)
        for weight in self.parameters():
            nn.init.uniform_(weight, -stdv, stdv)
 
    @jit.script_method
    def forward(self, x, state):
        # type: (Tensor, Tuple[Tensor, Tensor]) -&gt; Tuple[Tensor, Tuple[Tensor, Tensor]]        
        hx, cx = state

        xh = (
            torch.mm(x, self.weight_ih.t()) + self.bias_ih + 
            torch.mm(hx, self.weight_hh.t()) + self.bias_hh
        )        
        
        i, m, o = xh.chunk(3, 1)

        m = m + (self.weight_ch_m * cx)
        o = o + (self.weight_ch_o * cx)

        i = torch.tanh(i)
        m = torch.sigmoid(m)
        o = torch.sigmoid(o)        

        # Base on Formula
        h = (1 - m) * cx + (m * i)
        c = (1 - o) * i + (o * cx)       

        return h, (h, c) 
</code></pre>
<p>my question is:</p>
<ol>
<li>is my implementation are correct for those case (forward and backward)?</li>
<li>since the cell state not connected to hidden state, how can i do backprop on cell state?</li>
<li>even when using jit it has an error because cell state not connected to hidden state. one of the trick is to connect calculation with hidden state like <code>h = h + (c * 0)</code> but i don’t know it is the right way todo or not. is it okay to do that?</li>
</ol>
<p>Can anybody point me how to achieve desired backprop like in formula in Pytorch (｡•́︿•̀｡).</p>
<p>Thank you, any response will be appreciate.</p>
Validation accuracy is not changing

username: edshkim98
cooked: <p>Hi, I am currently training a LSTM model for binary classification.<br>
Although my training accuracy and loss are changing, my validation accuracy is stuck and does not change at all. Can someone help with solving this issue? Thank you!<br>
p.s. Iearning rate =0.001 with adam optimizer and weight_decay=1e-4</p>
<pre><code class="lang-auto">Training Epoch 0 Loss: 0.6818549633026123 Acc: 75.0
Training Epoch 0 Loss: 0.33946362137794495 Acc: 87.5
Training Epoch 0 Loss: 0.573047935962677 Acc: 75.0
Training Epoch 0 Loss: 0.5953656435012817 Acc: 75.0
Training Epoch 0 Loss: 0.5127055048942566 Acc: 87.5
Training Epoch 0 Loss: 0.5791436433792114 Acc: 75.0
Training Epoch 0 Loss: 0.5615108013153076 Acc: 75.0
Correct: 2019 Total: 2808
Validation Epoch 0 Acc: 71.9017094017094 Loss: 0.5894517120387819
#####################################
Training Epoch 1 Loss: 0.5601013898849487 Acc: 75.0
Training Epoch 1 Loss: 0.5866733193397522 Acc: 62.5
Training Epoch 1 Loss: 0.73282390832901 Acc: 50.0
Training Epoch 1 Loss: 0.3902740478515625 Acc: 87.5
Training Epoch 1 Loss: 0.7055811882019043 Acc: 62.5
Training Epoch 1 Loss: 0.26922595500946045 Acc: 100.0
Training Epoch 1 Loss: 0.29699501395225525 Acc: 87.5
Correct: 2019 Total: 2808
Validation Epoch 1 Acc: 71.9017094017094 Loss: 0.522074056805199
#####################################
Training Epoch 2 Loss: 0.3273327946662903 Acc: 87.5
Training Epoch 2 Loss: 0.6666995286941528 Acc: 62.5
Training Epoch 2 Loss: 0.45028626918792725 Acc: 75.0
Training Epoch 2 Loss: 0.4807313084602356 Acc: 75.0
Training Epoch 2 Loss: 0.47266024351119995 Acc: 75.0
Training Epoch 2 Loss: 0.6877409815788269 Acc: 75.0
Training Epoch 2 Loss: 0.42089831829071045 Acc: 75.0
Correct: 2019 Total: 2808
Validation Epoch 2 Acc: 71.9017094017094 Loss: 0.5425263909235639
#####################################
Training Epoch 3 Loss: 0.5228984355926514 Acc: 75.0
</code></pre>
The gradients of BatchNorm layer at mode of model.train() and model.eval()

username: Harxis
cooked: <p>Hi guys, I have bulid a simple network like</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.cnn = nn.Conv2d(1, 1, 4, 2, 1)
        self.bn = nn.BatchNorm2d(1)
        self.fc = nn.Linear(16, 2)
    def forward(self,x):
        x = self.cnn(x)
        x = self.bn(x)
        x = x.view(-1, 16)
        x = self.fc(x)
        return x
</code></pre>
<p>And I input a same random sample X to the network at model.train() and model.eval(). I know the ‘running_mean’ and the ‘running_var’ would change when forward propagation at model.train() but would’t change at model.eval(), so I have replace the values of mean and var at model.eval() as the values at model.train(). So the parameters and output is the same at both two mode.</p>
<p>output of two modes</p>
<blockquote>
<p>out<br>
train tensor([[-0.6198,  0.0471]], device=‘cuda:0’, grad_fn=)<br>
eval tensor([[-0.6198,  0.0471]], device=‘cuda:0’, grad_fn=)</p>
</blockquote>
<p>parameters of two modes</p>
<blockquote>
<p>train Parameter containing:<br>
tensor([[[[-0.1379,  0.0229, -0.0143,  0.0128],<br>
[-0.0239, -0.0240,  0.0798, -0.0504],<br>
[-0.2342,  0.1400, -0.2099, -0.1185],<br>
[ 0.0443,  0.0067, -0.0232, -0.1900]]]], device=‘cuda:0’,<br>
requires_grad=True)<br>
eval Parameter containing:<br>
tensor([[[[-0.1379,  0.0229, -0.0143,  0.0128],<br>
[-0.0239, -0.0240,  0.0798, -0.0504],<br>
[-0.2342,  0.1400, -0.2099, -0.1185],<br>
[ 0.0443,  0.0067, -0.0232, -0.1900]]]], device=‘cuda:0’,<br>
requires_grad=True)</p>
</blockquote>
<p>However, when I calculate the gradients, I found the gradients of the layers before BatchNorm layer(Layer 0 1) are different at model.train() and model.eval()(the gradients of layers after BatchNorm(Layer 2 3 4 5) are still the same).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa20f1f9e17b080a6c35ae3f711a1814c6607df2.png" data-download-href="https://discuss.pytorch.org/uploads/default/aa20f1f9e17b080a6c35ae3f711a1814c6607df2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa20f1f9e17b080a6c35ae3f711a1814c6607df2.png" alt="image" data-base62-sha1="oh1KNnS4BVZvGl7Yrp9BKC6ciPg" width="512" height="500" data-dominant-color="343434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">672×655 71.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I think it’s because the different backpropagations of BatchNorm layer at model.train() and model.eval(). But I could’t understand it. Does anyone know the detail of BatchNorm backpropagation at these two modes? Or how can I freeze the ‘running_mean’ and ‘running_mean’ at the forward propagation at model.train()? Thank you so much!</p>
About converting from pytorch to C++ for prediction function

username: Chen0729
cooked: <p>Hi, All</p>
<p>I have an inquiry of converting from pytorch to C++ for prediction function:</p>
<p>The python code is:<br>
def predict(self, x):<br>
with torch.no_grad():<br>
ret = torch.max(self.D(Variable(x), cuda=self.args.cuda), 1)[1].data<br>
return ret</p>
<p>The converted C++ code is:<br>
torch::Tensor predict(torch::Tensor x) {</p>
<pre><code>    torch::Tensor x1 = discriminator-&gt;forward(x, false, args.cuda)[0];

    return std::get&lt;1&gt;(max(x1, 1, false)).data();
  

}
</code></pre>
<p>After this conversion, I find the numerical results are not quite matched from pytorch to C++.<br>
Any comments are appreciated.</p>
<p>Thanks in advance.</p>