akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
FFT and complex values in loss function

daaaaaaaaaaawn: <p>I am new to pytorch and trying to use it to solve an underdetermined problem where I have a limited number of samples of an FFT. For now I am using the entire FFT and just a squared error loss.</p>
<p>My output looks like a rightside-up image superimposed on an upside-down image:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0a548a206d641a24dc07e409e2ee300ee9bc39c.png" alt="image_and_result" data-base62-sha1="tLLqOTzNqdsnUJRXtfGERtr9OZ6" width="567" height="300"></p>
<p>I think this means I am losing the imaginary values somewhere. I verified that all the operations (subtraction, squaring, summing) are behaving as I expect in the loss calculation. What could be causing this?</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">import torch
import numpy as np
import matplotlib.pyplot as plt
from skimage.data import shepp_logan_phantom
from skimage.transform import resize

imsize = 128
image = resize(shepp_logan_phantom(),(imsize,imsize))
fftimage = np.fft.fft2(image)

x = torch.zeros((imsize,imsize)) # guess
x.requires_grad = True
optimizer = torch.optim.Adam([x], lr=5e-4)
data = torch.from_numpy(fftimage)

losses = []

def loss(x):
    fftx = torch.fft.fft2(x)
    loss = (data - fftx).pow(2).sum()
    loss = loss.real.pow(2) + loss.imag.pow(2)
    return loss

itera = 10000
for i in range(itera):
    current_loss = loss(x)
    optimizer.zero_grad()
    current_loss.backward()
    optimizer.step()
    losses.append(current_loss)

reconstruction = x.detach().numpy()

fig1,ax1 = plt.subplots(1,1)
ax1.plot(losses)
ax1.set_yscale("log")
ax1.set_ylabel("loss")
ax1.set_xlabel("iterations")

fig,ax = plt.subplots(1,2)
ax[0].imshow(image)
ax[1].imshow(np.abs(reconstruction))
</code></pre>

YuyangXueEd: <p>Have you solve this problem? I recently on MRI reconstruction and using complex number in my loss function also have some problem. Looking forward to hearing from you <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

doojin: <p>Is this problem because you used <code>.pow()</code>?</p>
Roi Pool using AdaptiveMaxPool

chinmay5: <p>Although ROI Pooling is now present in torchvision as a layer, I need to implement it for 3d. As such, I think I can make use of the AdaptiveMaxPool3D layer. However, considering that this might have issues as mentioned <a href="https://discuss.pytorch.org/t/pytorch-roi-pooling-implementation-deviation/21507/2">here</a> and <a href="https://discuss.pytorch.org/t/runtimeerror-adaptive-max-pool2d/75932">here</a>, I have made some changes to the code. However, it is still giving me values which are less than expected. I am attaching the source code here</p>
<pre><code class="lang-auto">def pad_input(im, output_size):
    bs, ch, w, h, l = im.shape
    if w &gt;= output_size[0] and h &gt;= output_size[1] and l &gt;= output_size[2]:
        return im
    padding_tuple = (0, output_size[2] - l, 0, output_size[1] - h, 0, output_size[0] - w)
    im = F.pad(im, padding_tuple, mode="replicate")
    return im


def roi_pool_simple(input, rois, output_size, spatial_scale):
    """
    input: B, C, W, H, L
    rois : num_roi, 6
    output_size: (4, 4, 4)
    spatial_scale: 1/4, roi in input space so we need a way to downsample to specific shape
    """
    output = []
    rois = rois.clone()
    rois_num = rois.size(0)
    max_w, max_h, max_l = input.shape[2], input.shape[3], input.shape[4]
    rois.mul_(spatial_scale)
    # Here bs = 1 so we directly use it in the index
    for i in range(rois_num):
        roi_start_w, roi_start_h, roi_start_l, roi_end_w, roi_end_h, roi_end_l = rois[i]
        # round down the start indices
        roi_start_w, roi_start_h, roi_start_l = math.floor(roi_start_w), math.floor(roi_start_h), math.floor(roi_start_l)
        # round up the end indices
        roi_end_w, roi_end_h, roi_end_l = math.ceil(roi_end_w), math.ceil(roi_end_h), math.ceil(roi_end_l)
        # We would have empty channels if the start index is greater than the dimensions. So,
        if roi_start_w &gt;= max_w or roi_start_h &gt;= max_h or roi_start_l &gt;= max_l:
            output.append(torch.zeros(output_size))
        else:
            im = input[..., roi_start_w: roi_end_w, roi_start_h:roi_end_h, roi_start_l:roi_end_l]
            im = pad_input(im, output_size)
            output.append(F.adaptive_max_pool3d(im, output_size))
    return torch.cat(output, dim=0)
</code></pre>

Lei_Gu: <p>I have several qs about the 3D RoIPooling version.<br>
if roi_start_w &gt;= max_w or roi_start_h &gt;= max_h or roi_start_l &gt;= max_l:<br>
output.append(torch.zeros(output_size))<br>
This one makes the output 3 dimentions.<br>
and the else branch makes the output 5 dimentions.</p>
Can not access Torch C++ API documentation

cokespace2: <p>When I visit C++ API documentation it returns 404 files not found.</p>
<p>for example: <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_max_pool2d.html" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_max_pool2d.html</a></p>
<p>Does anyone know where can I find the C++ API documentation? or it just because my Network Issus??</p>

Azerus: <p>Hello,</p>
<p>I have the same 404 error when visiting the C++ API doc. I also got the issue with the page <a href="https://pytorch.org/cppdocs/api/library_root.html" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/library_root.html</a></p>
<p>I suppose we can still find the doc information from <a href="https://github.com/pytorch/cppdocs" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/cppdocs: PyTorch C++ API Documentation</a> but that’s not very convenient.</p>
<p>I hope it will be back online soon <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_diagonal_mouth.png?v=12" title=":face_with_diagonal_mouth:" class="emoji" alt=":face_with_diagonal_mouth:" loading="lazy" width="20" height="20"></p>

cokespace2: <p>Visit GitHub for doc is kind soooo inconvenient <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"> and I even can not find what I am looking for…But seems no other choices left, sigh</p>

ptrblck: <p>Could you create an issue for it on GitHub, as the generation of the docs might have failed?</p>

cokespace2: <p>Okay, I’ll do that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to check if gradient vanish/explode during training a deep neural network?

Nick_Lin: <p>I’m training a deep neural network with N hidden layers. But I found that both train and test accuracy got worse when N becomes large (which means more hidden layers)</p>
<p>As I know, when neural network becomes deeper, model’s performance may become worse due to gradient vanish/explode, which means layers close to input layer can’t update weights stably because of little/large value of grad.</p>
<p>But after I check each layer’s grad, I found grad in deeper layer is smaller than shallow layer, e.g. 10^-5 in the first hidden layer and 10^-10 in the last hidden layer, which is different to my thought.</p>
<p>Is ther any misunderstanding to my thought? Or actually there is another reason due to the worse result when my model goes deeper? Thanks.</p>
Paytorch installation error

Aiman_Mutasem-bellh: <p>Hello all, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>During installing Paytorch <code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</code> it freezing</p>
<pre><code class="lang-auto">(base) C:\Windows\system32&gt;conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
Collecting package metadata: done
Solving environment:
</code></pre>
<p>How I can fix this issue?</p>

ptrblck: <p>I would recommend to stick to the provided install commands from <a href="https://pytorch.org/get-started/locally/">here</a> (i.e. change the <code>cudatoolkit</code> version to a supported one). Once done, allow <code>conda</code> to solve the environment as it might take some time. Alternatively, create a new conda environment and install PyTorch there, which should be faster.</p>

Aiman_Mutasem-bellh: <p>Thank you sir, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I think this is the problem “CUDA-10.2 PyTorch builds are no longer available for Windows”, Do you think CUDA-11.6 will work well with windows10?</p>

ptrblck: <p>Yes, try to use 11.3 or 11.6 as both should work.</p>
Trying to customize transformer (error)

sandeep1: <p>Hi all,</p>
<p>I am trying to customize the huggingface transformer library.<br>
The step which i followed is :-</p>
<pre><code class="lang-auto">git clone https://github.com/huggingface/transformers \
pip install -e .
``
But when i trying to import :-
``from transformers import AutoModelForSeq2SeqLM,DataCollatorForSeq2Seq,AutoTokenizer ```

i am getting error as

`` cannot import name 'AutoModelForSeq2SeqLM' from 'transformers' (unknown location)``

I have tried setting the os sys path , but it didn't work. Has anyone done this kind of work before. How do we exactly change the functions of transformer?

Thank you in advance.</code></pre>
Torch.onnx.export for LocalResponseNormalization

timosy: <p>I exported original AlexNet to .onnx using a function: torch.onnx.export().</p>
<p>The original model has LocalResponseNormalization. This one layer is converted to ~30 layers of combination with different names such as Mul, Unsqueeze, Gather, etc.</p>
<p>On the other hand, when I use chainer model and convert the original model uisng onnx-chainer.  The LocalResponseNormalization layer is still LocalResponseNormalization, which is actuuly supported by the onnx-chainer package accrodibg to <a href="https://pypi.org/project/onnx-chainer/" class="inline-onebox" rel="noopener nofollow ugc">onnx-chainer · PyPI</a></p>
<p>Pytorhc is not supporting LocalResponseNormalization itself?</p>
FileNotFoundError: Could not find module "...\c10.dll"

khalfinv: <p>Hi,<br>
Trying to run my code after installing torch and receive the following error:</p>
<p>Traceback (most recent call last):<br>
File “C:\Users\ויקי\Desktop\ABCD_project\gordon_correlations_algorithms\run_cv.py”, line 5, in <br>
import test<br>
File “C:\Users\ויקי\Desktop\ABCD_project\gordon_correlations_algorithms\test.py”, line 10, in <br>
import torch<br>
File “C:\python38\lib\site-packages\torch_<em>init</em>_.py”, line 81, in <br>
ctypes.CDLL(dll)<br>
File “C:\python38\lib\ctypes_<em>init</em>_.py”, line 369, in <strong>init</strong><br>
self._handle = _dlopen(self._name, mode)<br>
<strong>FileNotFoundError: Could not find module ‘C:\python38\lib\site-packages\torch\lib\c10.dll’. Try using the full path with constructor syntax.</strong></p>
<p>c10.dll <strong>exists</strong> in the folder.<br>
<strong>Python version:</strong> 3.8.0 64 bit<br>
<strong>Windows 10 64 bit</strong><br>
<strong>pytorch install:</strong> pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>Can you help me please?</p>

peterjc123: <p>Have you installed VS2019 redist? <a href="https://aka.ms/vs/16/release/vc_redist.x64.exe" rel="nofollow noopener">https://aka.ms/vs/16/release/vc_redist.x64.exe</a></p>

khalfinv: <p>This solved the problem. Thank you!</p>

hcwhui: <p>How did you solved ?</p>

Infinite_Warp: <p>Genius solution. However I don’t understand how it’s related to pytorch not finding a file.</p>

khalfinv: <p>Just installed the VS2019 redist</p>

aletelecom: <p>Good afternoon, I still have this problem, installed pytorch using conda, for CPU. Anybody know how to solve this? The c10.dll file surely is in the specified folder.</p>
LSTM Seq2Seq Encoder-Decoder not working

AfonsoSalgadoSousa: <p>I am trying to train an LSTM Encoder-Decoder model for paraphrase generation. My model is as follows:</p>
<pre><code>StackedResidualLSTM(
  (encoder): RecurrentEncoder(
    (embed_tokens): Embedding(30522, 256)
    (dropout): Dropout(p=0.5, inplace=False)
    (rnn): LSTM(256, 256, num_layers=2, batch_first=True, dropout=0.5)
  )
  (decoder): RecurrentDecoder(
    (embed_tokens): Embedding(30522, 128)
    (dropout_in_module): Dropout(p=0.5, inplace=False)
    (dropout_out_module): Dropout(p=0.1, inplace=False)
    (layers): ModuleList(
      (0): LSTMCell(384, 256)
      (1): LSTMCell(256, 256)
    )
    (fc_out): Linear(in_features=256, out_features=30522, bias=True)
  )
)
</code></pre>
<p>Following is a print of the source sentence, the sentence fed to the decoder (shifted right), the predictions, and the true sentence (labels). Everything is tokenized with Huggingface BERT tokenizer:</p>
<blockquote>
<p>Source: [CLS] where can i get quality services in brisbane for plaster<br>
and drywall repair? [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD]</p>
<p>Decoder Input: [CLS] [CLS] where can i get<br>
quality services for plaster and drywall repairs in brisbane? [SEP]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]</p>
<p>Preds:<br>
[CLS] the? [SEP]? [SEP]? [SEP]? [SEP]? [SEP]? [SEP]? [SEP]? [SEP]?<br>
[SEP]</p>
<p>Target: [CLS] where can i get quality services for plaster and<br>
drywall repairs in brisbane? [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]<br>
[PAD] [PAD] [PAD] [PAD] [PAD]</p>
</blockquote>
<p>My loss function is a CrossEntropy between the output and labels (the padding token is switched with -100 to ignore). Something like:</p>
<pre><code>loss_fct = CrossEntropyLoss()
loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1))
</code></pre>
<p>There are two problems occurring:</p>
<ul>
<li>the loss does not go down</li>
<li>the generations are all the same for every entry of the same epoch (after weight updating the generations might be different than the ones from the previous epoch, but remain the same for every entry of the new epoch)</li>
</ul>
<p>Do you have any idea what might I try to fix the issue? Thanks in advance for any help you can provide.</p>

djlindsay: <p>Context vectorization with GLoVe? I’m right there with ya bud generation is a topic no one on here really seems comfortable talking about and it’s easy to get lost. Wish I could help more, but I’ll be watching this thread.</p>
Use one backbone with different heads

Bruno_Patricio: <p>Is it possible to save the results from the backbone and apply them on the heads of the all the other models. My goal was to try to save time by avoiding repeating the backbone part. For example, looking at the diagram, instead of running the 3 complete models (left), only run the backbone 1 time and switch only the heads for the 3 models (right), therefore not repeating executing the backbone every time.</p>
<p>Thank you for the help!</p>

InnovArul: <p>Probably you forgot to add the diagram that you are referring to?</p>
<p>In general,  running backbone one time and switching between 3 heads is technically possible in pytorch.</p>

Bruno_Patricio: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3ed8f942579b104c4ae6b373590a00763adcc7cc.png" alt="Capturar" data-base62-sha1="8XYsXz6l82vmWG9V8DzENgC3us4" width="584" height="312"></p>
<p>The diagram is in this genre. I don’t know how to do it, can you give me some kind of solution?</p>
<p>Thanks for your reply</p>

InnovArul: <p>Pl. check if the below code suits your need.</p>
<pre><code class="lang-auto">import torch, torch.nn as nn
import torchvision.models as models
from einops import reduce

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = nn.Sequential(*list(models.resnet18(pretrained=True).children())[:-2])
        self.head1 = nn.Linear(512,10)
        self.head2 = nn.Linear(512,10)
        self.head3 = nn.Linear(512,10)

    def forward(self, x):
        # get backbone features
        backbone_feats = self.backbone(x)

        # global average pooling
        backbone_feats = reduce(backbone_feats, 'b c h w -&gt; b c', reduction='mean')

        # pass through heads
        h1 = self.head1(backbone_feats)
        h2 = self.head2(backbone_feats)
        h3 = self.head3(backbone_feats)
        return h1, h2, h3

if __name__=='__main__':
    m = Model().cuda()
    x = torch.randn(5, 3, 224, 224).cuda()
    h1, h2, h3 = m(x)
    print(h1.shape, h2.shape, h3.shape)

</code></pre>

Bruno_Patricio: <p>I don’t think so, I am using the yolov5 network for the execution of this process described above.</p>
<p>thanks for the support!</p>

InnovArul: <p>You could try to change the <code>self.backbone</code> assignment to appropriate YoloV5 model.<br>
I haven’t used YoloV5 so far.</p>

diogompontes: <p>Hello <a class="mention" href="/u/innovarul">@InnovArul</a> .</p>
<p>I’m getting some problems with my model Yolov5s aswell. Can’t create a solution for this problem.</p>
<p>Thanks for the help!</p>
Profiler failed to record with cuda kinfo mode

yigu: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82ab0868ba663aa937f6163378acdadc4637653e.png" data-download-href="https://discuss.pytorch.org/uploads/default/82ab0868ba663aa937f6163378acdadc4637653e" title="67F6C4B0-D103-468c-89E0-7ED99DE5B17B"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82ab0868ba663aa937f6163378acdadc4637653e.png" alt="67F6C4B0-D103-468c-89E0-7ED99DE5B17B" data-base62-sha1="iDWvseAnjIzKqutHEEnlyud71uu" width="690" height="111" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">67F6C4B0-D103-468c-89E0-7ED99DE5B17B</span><span class="informations">1895×307 19.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As image show, I run SSD model use AMD gpu,but when I run serverl iterations, the programme failed with this info.<br>
I checked gpu memory using another terminal, and the remaining gpu memory is about 12G, so the gpu resource is sufficient.<br>
I also checked the cpp code in profiler_cuda.cpp and profiler_kinfo.cpp lines as shown in image, but the exception could’t provide more info, it only show info in image.<br>
Anyone can give me help? Thank you a lot!</p>
Conditional loss for multi-class multi-output model

T_Taiber: <p>Hello all!</p>
<p>I am working on a room-image classifier, which for a given image predicts: (1) what type of room it is (e.g. bedroom); (2) what style does the room have (e.g. modern); (3) does it look high or low budget.</p>
<p>I want to add a category for an “empty room” in the room types, but empty rooms should not have a style or budget associated with them. Therefore, in those cases where the room label is “empty”, I want to ignore the Cross Entropy losses associated with the style and budget. However, I cannot figure out how to do this for the entire batch at once.</p>
<p>For clarity’s sake, the losses (ignoring the condition) are computed as follows (based on one-hot encoding):</p>
<pre><code class="lang-auto">loss_r = nn.CrossEntropyLoss()(r_p, r_t)
loss_s = nn.CrossEntropyLoss()(s_p, s_t)
loss_b = nn.CrossEntropyLoss()(b_p, b_t)
</code></pre>
<p>where <code>r_p</code>, <code>s_p</code>, and <code>b_p</code> are the predictions (in logits), and <code>r_t</code>, <code>s_t</code>, and <code>b_t</code> are the targets.</p>
<p>Cheers!</p>

ParGG: <p>Hi Tal,</p>
<p>when you load your data, on top of creating the one hot encoded classes, create an additional tensor that will mask your loss. If your room is empty, this tensor will be <code>[1, 0, 0]</code>, whilst if it is not empty, it will be like <code>[1, 1, 1]</code>.</p>
<p>When you compute your final loss (which is probably the sum of <code>loss_r, loss_s, loss_b</code>), just perform the internal (dot) product with the mask I told you about.</p>
<p>In this case you will have the following situations:</p>
<pre><code class="lang-python"># empty room, mask = [1, 0, 0]
loss = torch.dot(torch.stack([loss_r, loss_s, loss_b], 0), mask) # loss_r * 1 + loss_s * 0 + loss_b * 0 = loss_r

# non-empty room, mask = [1, 1, 1]
loss = torch.dot(torch.stack([loss_r, loss_s, loss_b], 0), mask) # loss_r * 1 + loss_s * 1 + loss_b * 1
</code></pre>
<p>Therefore, when you have an empty room the loss will ignore the contribution from the style and budget.</p>
Simultaneously train two model in each epoch

Shresth_Somya: <pre><code class="lang-auto">def U_closure(self):
    self.optimizer_U_model.zero_grad()
    loss = self.cost_function()
    loss.backward(retain_graph=True)
    return loss
def E_closure(self):
    self.optimizer_E_model.zero_grad()
    loss = self.cost_function()
    loss.backward(retain_graph=True)
    return loss
# Training function
def train(self, epochs, opt_func=torch.optim.Adam):
    torch.autograd.set_detect_anomaly(True)
    self.optimizer_U_model = opt_func(self.U_model.parameters())
    self.optimizer_E_model = opt_func(self.E_model.parameters())
    for epoch in range(epochs):
        self.optimizer_U_model.step(self.U_closure)
        self.optimizer_E_model.step(self.E_closure
</code></pre>
<p>There are two models <code>U_model</code> and <code>E_model</code> which are needed to be trained in each epoch. But when sending <code>closure</code> function to <code>optimizer</code> it is calculating loss two times. I have same loss for both model. Should I have only one <code>optimizer</code>? If I have only one optimizer then can it take parameters of both modes?</p>

ParGG: <p>You can use one optimiser that has access to both models’ parameters. Concerning the loss, just sum the two, the gradient won’t change.</p>

Shresth_Somya: <p>What should be the parameter for line</p>
<pre><code class="lang-auto">self.optimizer = opt_func(self.U_model.parameters())
</code></pre>
<p>,i.e., how can I give two weights parameter in single opttimiser?</p>

mxahan: <p>I was just wondering if you have tried <a href="https://discuss.pytorch.org/t/giving-multiple-parameters-in-optimizer/869">this</a>.</p>

ParGG: <p>I guess that <a class="mention" href="/u/mxahan">@mxahan</a> has provided you with the answer. In any case, the PyTorch documentation already provides you with examples that I think can be extended to multiple models.</p>
<p><a href="https://pytorch.org/docs/stable/optim.html#how-to-use-an-optimizer" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/optim.html#how-to-use-an-optimizer</a></p>
<p>In particular I am referring to this:</p>
<pre><code class="lang-python">optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
Could data augmentation lead to such a situation?

gns24: <p>The accuracy of my model on the test set is about 63%, and the trained model is run on the training set, and the accuracy is 100%<br>
So I thought of choosing a stronger data augmentation method, using the “albumentations”, and found that when the accuracy on test dataset is 47%, the accuracy on the training set is only 10%, I want to know that is this due to data augmentation? Does it treat the training set as unaugmented, i.e. images which have never been seen, like a test set that is 3 times larger?</p>

cijerezg: <p>In general, it’s very hard to tell. According to what you describe, your model can’t really train on the augmented dataset, which might indicate you augmented your data too much. You have to run different experiments to really figure out what’s going on. For example, you may start by augmenting your training dataset slightly and see what happens to both training and testing accuracies.</p>
<p>There are many posts about typical issues when training neural nets as well as tips, but at the end of the day, you are the one who has to run experiments to work out what’s going on. Especially, with something as odd as what you describe.</p>

Loki_K: <p>You have to run series of experiments to exactly findout what went wrong. Practical advice is to introduce little complextiy at once. .i.e. first try smaller no. of augmentations and make sure that they are almost close to the original data.</p>
<p>And just saying… you don’t augment the test set.</p>
<p>Btw I didn’t get what you are saying in the last question.</p>
<aside class="quote no-group" data-username="gns24" data-post="1" data-topic="156243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/d9b06d/48.png" class="avatar"> gns24:</div>
<blockquote>
<p>Does it treat the training set as unaugmented, i.e. images which have never been seen, like a test set that is 3 times larger?</p>
</blockquote>
</aside>
What is the practical difference between pytorch_android and pytorch_android_lite

phillies: <p>Hi,<br>
I’ve been using pytorch for quite some time now and want to use it on an android device. But I’m confused by the pytorch_android and pytorch_android_lite versions.</p>
<p>I created a model, built an application to do inference and used both pytorch_android and pytorch_android_lite (and the respective torchvision). I could not find any real difference except for loading the Module. And the official documentation is listing pytorch_android 1.9.0 with a mix ob lite and normal.</p>
<p>Could someone please clear up what the practical difference is and in which case one should use which version?</p>
<p>Thanks!</p>

cccclai: <p><code>pytorch_android_lite</code> is a more efficient and smaller interpreter on mobile, see <a href="https://pytorch.org/tutorials/recipes/mobile_interpreter.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 1.12.0+cu102 documentation</a>. <code>pytorch_android_lite</code> will be preferred for Android.</p>
Mixed Precision Training on CUDA with bfloat16

kocchop: <p>Hi, I am trying to run the BERT pretraining with amp and bfloat16. First of all, if I specify</p>
<pre><code class="lang-auto">with torch.cuda.amp.autocast(dtype=torch.bfloat16):
</code></pre>
<p>the output tensor is shown as float16 not bfloat16. When I change the torch.cuda.amp.autocast to torch.autocast(“cuda”, dtype=torch.bfloat16), the output tensor shows bfloat16 datatype.</p>
<p>However, that does not eventually work either. The following error msg shows up:<br>
RuntimeError: expected scalar type BFloat16 but found Half</p>
<p>I was wondering whether the pytorch native amp still lacks support for bfloat16 mixed precision training.</p>
<p>I am trying to run BERT pretraining using the DeepLearningExamples repository by nvidia. Any help or pointers towards the solution would be very much appreciated.</p>

ptrblck: <aside class="quote no-group" data-username="kocchop" data-post="1" data-topic="156248">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f9ae1b/48.png" class="avatar"> kocchop:</div>
<blockquote>
<p>I am trying to run BERT pretraining using the DeepLearningExamples repository by nvidia.</p>
</blockquote>
</aside>
<p>Could you try to run the HuggingFace implementation as the NVIDIA one might use some custom modules depending on <code>float16</code> (unless their README shows the <code>bfloat16</code> usage, which might not be integrated yet)?</p>

kocchop: <p>Hi,</p>
<p>Thanks so much! I was also suspecting the same. Will try the huggingface version then.</p>
<p>On a separate note, I’ve been following your answers a lot in this forum lately. Thanks so much for all the help and support. Really appreciate it.</p>
Error iterating dataloader, can't cast to the desired type

lluiscc: <p>So while training a model at the nth iteration of the first epoch this error occurs. Any idea what it might be?</p>
<p>RuntimeError: Caught RuntimeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py”, line 287, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py”, line 52, in fetch<br>
return self.collate_fn(data)<br>
File “/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py”, line 157, in default_collate<br>
return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})<br>
File “/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py”, line 157, in <br>
return elem_type({key: default_collate([d[key] for d in batch]) for key in elem})<br>
File “/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py”, line 138, in default_collate<br>
return torch.stack(batch, 0, out=out)<br>
RuntimeError: result type Double can’t be cast to the desired output type Byte</p>

ptrblck: <p>It seems the <code>collate_fn</code> tries to <code>torch.stack</code> the samples to a batch and fails to do so as there seems to be a <code>dtype</code> mismatch of <code>double</code> and <code>Byte</code>.<br>
Print the dtypes of all samples before returning them in the <code>__getitem__</code> and make sure they are consistent.</p>

jeremyfix: <p>Such a situation indeed occured in may case with a dataloading pipeline including some albumentations transforms . By default, albumentations transforms are randomly applied. At the input of the pipeline you have uint8 tensors and some transforms may output a float64 tensor ; You may end up with some tensors being uint8, others float64, leading to a failure of the collate functions to stack the samples.</p>
<p>I do not know if that applies as well in your case.</p>
What is the reason of rnn's batch_first parameter is set False as default?

111335: <p>Hello every one. I’m beginner at nlp. i’m learning about how use nn.RNN with pytorch docs.</p>
<p>before start, i’m not good at english grammar XD )</p>
<p>in docs, nn.RNN’s parameter ‘batch_first’ is False in default.</p>
<p>it seems that people use batch_first == False ususally. ( i saw a lot of input datasets are same like that not only at simple nlp model but also at complicated nlp model(e.g bert) )</p>
<p>Can you let me know why people uses dataset’s shape as T x B x * (T is max length) ??</p>
<p>is there any advantages when i set dataset’s shape like that?</p>
<p>Thank you!!</p>

ayalaa2: <p>There’s no strict advantages or disadvantages. It could be that because, in an RNN, we’re iterating over the sequence dimension (we take timestep-0, then timestep-1, etc.) so it makes “sense” to have that dimension first. But it doesn’t really make a difference.</p>
<p>I would suggest just going with the default option only because it’s default.</p>

111335: <p>Thank you for your helpful answer <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Prabu_Rocking: <p>Hi, I was going through the pytorch’s machine translation example program <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" rel="noopener nofollow ugc">Language Translation with nn.Transformer and torchtext — PyTorch Tutorials 1.12.0+cu102 documentation</a> seems even here the dataset is T(max_len)xBxDmodel. I am failing to understand how does it works?. here in this example program my understanding is we are not feeding the data sequentially we are basically feeding a batch of sentances as src and tgt. so doesn’t it make more sense if our dataset is in the format of BxTxDmodel instead of TxBxDmodel ? or does the later format has some advantage ? or does something wrong with my understanding ?</p>
<p>need the community’s help. I am just getting started in Transformers and struck in this part for a week.</p>
How to transfer a trained model from Matlab to Pytorch?

Jimmy_Xiaoke_Shen: <p>Any ideas about how to transfer a trained model from Matlab to Pytorch?</p>
<p>Here is the discussion about how to transfer from Pytorch to Matlab. However, I need the inverse thing…</p>
<aside class="quote" data-post="1" data-topic="13695">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/c5a1d2/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-transfer-a-trained-model-from-pytorch-to-matlab/13695">How to transfer a trained model from Pytorch to Matlab?</a> 
  </div>
  <blockquote>
    Hi, Everyone. I have a trained CNN model in Pytorch and I want to transfer it to the Matlab. is there any solution for this purpose? 
Thank you for advising me on this topic. 
Reza.
  </blockquote>
</aside>

<p>Thanks in advance for any suggestions.</p>

ptrblck: <p>You could try to save the MATLAB model parameters as .mat files.<br>
After re-building the model in PyTorch, you could load these parameters using <a href="https://docs.scipy.org/doc/scipy/reference/tutorial/io.html">scipy.io</a> and assign them to your PyTorch model.<br>
Make sure you check the shaped and also possible flipped kernels, since some frameworks flip the conv kernels to use a convolution instead of the correlation.</p>

Jimmy_Xiaoke_Shen: <p>Thanks for the suggestion. I’ll try. By the way, for the pytorch, do we use the convolution or correlation? It seems that we are using the correlation, right?</p>

ptrblck: <p>Yep, is the correlation, since the kernels are not flipped again.</p>

Jimmy_Xiaoke_Shen: <p>Cool. It is much clear now. Thanks a lot.</p>

Jibanul: <p><a class="mention" href="/u/jimmy_xiaoke_shen">@Jimmy_Xiaoke_Shen</a>, I’m trying to do the same thing. Did you able to do this?</p>

Jimmy_Xiaoke_Shen: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="15903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>ing the model in PyTorch, you could load these parameters using <a href="https://docs.scipy.org/doc/scipy/reference/tutorial/io.html" rel="noopener nofollow ugc">scipy.io </a> and assign them to your PyTorch model.<br>
Make sure you check the shaped and also possible flipped kernels, since some frameworks flip the conv kernels to use a convolution in</p>
</blockquote>
</aside>
<p>Sorry for the late reply as I haven’t login for a long time. I gave up using the Matlab weights so I don’t know whether there is a good way to do that.</p>
Training a deep learning algorithm restarts pc

AwaisJacob: <p>Hi, I don’t know whether it is relavent here or not but i am pasting it here may be someone can guide me. I just got an old PC which has nvidia geforce gtx 1080 ti GPU. I installed latest drivers, CUDA and cuDNN. Then from pytorch site supported version of pytorch. All went well and torch.cuda.is_available() returns true. <strong>The issue i am facing is that when I start the training script for u2net, after first iteration the pc suddenly restarts. No apperent errors or warning. Please help!!</strong><br>
PS. I was using windows 10 and the issue happened couple of times and os crashed.</p>

omid_khalafbeigi: <p>Hi.<br>
Did you check your task manager? any abnormal activity in programs?<br>
Did you check hardware temperature?<br>
Are computer fans working correctly?<br>
I suggest you run your script on services like google colab and see what happen.</p>

AwaisJacob: <p>hi, Thanks for the reply.<br>
yes, I have noticed those things. No abnormal activities, the hardware was cool and the fans were working alright.<br>
The script runs fine on colab. but since it is a limited resource that is why I got this PC.</p>

ptrblck: <p>I guess your PSU might be too weak for a sudden power spike. We’ve had similar discussions here in the past and usually restarts boil down to the PSU. If you have a spare one, replace the current PSU and check if the issue is still visible.</p>

AwaisJacob: <p>Hi, Let me check that.<br>
PS. I am fan by the way.</p>

AwaisJacob: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="155440">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>guess your PSU might be too weak for a sudden power spike. We’ve had similar discussions here in the past and usually restarts boil down to the PSU. If you have a spare one, replace the current PSU and check if the issue is still visible</p>
</blockquote>
</aside>
<p>hi, I checked it and my pc has PSU of 750W while 250W is recommended for 1080ti. The issue remains. whenever i starts training the pc reboots.</p>

AwaisJacob: <p>another thing i checked is that simpler cnn models with less epochs are running fine. only u2net model does that</p>

ptrblck: <p>You can check <a href="https://discuss.pytorch.org/t/multi-gpu-2080-ti-training-crashes-pc/52032/13">this topic</a> where users were seeing similar issues with PSU which <em>should</em> be powerful enough, but in the end seemed to suffer from large peaks in power drain and shut down.</p>

AwaisJacob: <p>thank you so much for your valuable insight and time. I guess I need to get a new PSU.</p>

ptrblck: <p>Before buying a new one, I would check with your friends if someone has either a spare PSU or could borrow you theirs for some quick tests. (I also think someone else with a similar issue was able to test a PSU from a shop for some tests and bought if afterwards.)</p>

mendi: <p>Run tests on memory, buy new RAM memory</p>
Changing .pth file (dataset) to csv

Ulwan_M_Rasyid: <p>I have a repo that provides a .pth file but as a dataset, is there any way to transform it to a more readable form like CSV?</p>

ptrblck: <p>Assuming you can access the actual data as a tensor in your <code>Dataset</code>, you could transform it to a <code>numpy.array</code> and save it via <code>numpy.savetxt("data.csv", a, delimiter=",")</code>.</p>

Ulwan_M_Rasyid: <p>it says that <code>ValueError: only one element tensors can be converted to Python scalars</code></p>

rad: <p>You can load it using <code>torch.load</code>, and see what data and types you have in it. Then based on that, you can parse/transform it according to your needs, and save that as you want it to be. Feel free to share the results of loading it, that would help us too!</p>
DataLoader not returning proper batch

erlebach: <p>Hi! I need help on DataLoader to resolve what is probably a simple misconception. Below, I have a MWE illustrating my use case. I input a list of three arrays of size 1024 to a custom <code>Dataset</code>, which will return element <code>idx</code> from each array as a sequence of three elements. I have checked that <code>Dataset</code> by iterating through it and confirmed that I could retrieve elements 0 through 1023.  When using a DataLoader with <code>batch_size=100</code>,  I expect the <code>DataLoader</code> to return  batches of size 100. However, I only get one batch of size 1. I thought I understood how the DataLoader was supposed to work, but obviously not. Any insight is appreciated. Thanks.</p>
<pre><code class="lang-python">import numpy as np
import torch
from torch.utils.data import Dataset, DataLoader

# Create 3 arrays of size 1024
a = np.random.randn(3, 1024)
print(a.shape) # 3,1024
a1, a2, a3 = a[0], a[1], a[2]
print(a1.shape) # 1024

class myDataset(Dataset):
    """
    Parameters:
    ----------
    A list of numpy arrays
    """
    
    def __init__(self, data):
        assert isinstance(data, list), "myDataset: argument must be of type list"
        self.data = data
       
    def __getitem__(self, idx):
        return tuple(data[idx] for data in self.data)
    
    def __len__(self):
        return len(self.data)

data = myDataset([a1, a2, a3])
data_iter = DataLoader(data, batch_size=100, shuffle=False)

for index, values in enumerate(data_iter):
    print("index= ", index)
    print("values= ", values)

# output of the for loop: 
# index=  0
# values=  [tensor([-0.4421, -0.4562,  1.2012], dtype=torch.float64), tensor([-0.8228, -0.7304,  # 0.6380], dtype=torch.float64), tensor([ 1.2241,  0.4840, -0.0031], dtype=torch.float64)]

# I expected to collect about 10 batches of size 100. 
# However, I only collect the equivalent of a batch of size 1, and the for loop has only a single iteration. 
</code></pre>

ptrblck: <p>Thanks for the great MWE!<br>
The issue is a bit tricky to find, since iterating the <code>Dataset</code> works while the <code>DataLoader</code> only returns a single batch. However, after checking the implementation you can see that <code>len(data)</code> returns 3 and is then used by the <code>DataLoader</code> to define the used indices to create the single batch.<br>
In your <code>Dataset.__len__</code> function you are returning <code>len(self.data)</code> which is the length of the <code>list</code> of the arrays (and thus 3).<br>
Use <code>return len(self.data[0])</code> (or the <code>min</code> of all arrays in case these are different in your real use case) and it will work.</p>

erlebach: <p>Thanks for the solution! This works. An alternative solution is to use TensorDataset.</p>
<p>Gordon</p>
Specify dimension in torch.where

mmoelle1: <p>I have some problem with <code>torch.where(...)</code>. My code relies on many small matrix-vector product operations which I would like to vectorise over the third dimension. As an example consider the following code snippet:</p>
<pre><code class="lang-auto">A = torch.tensor([1,2,3,4,5,6,7,8,9,10,11,12]).view(3,2,2)
tensor([[[ 1,  2],
         [ 3,  4]],

        [[ 5,  6],
         [ 7,  8]],

        [[ 9, 10],
         [11, 12]]])

x = torch.tensor([1,2,3,4,5,6]).view(3,2,1)
tensor([[[1],
         [2]],

        [[3],
         [4]],

        [[5],
         [6]]])

y = torch.matmul( A, x )
tensor([[[  5],
         [ 11]],

        [[ 39],
         [ 53]],

        [[105],
         [127]]])
</code></pre>
<p>This works just fine but my problem is that I need to catch some corner cases. Let</p>
<pre><code class="lang-auto">idx = torch.tensor([1,2,3])
</code></pre>
<p>What I want to achieve is the following</p>
<pre><code class="lang-auto">torch.where(idx &gt; 1, A, 0.0)
</code></pre>
<p>which lead to the following error message</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 2
</code></pre>
<p>Making matrix <code>A</code> of size [2,2,3] <em>does</em> work (i.e. no errors) but of course does not give any meaningful results. Any help is appreciated.</p>

ptrblck: <p>I’m not sure what the expected results are but given your <code>idx</code> tensor has 3 values I would guess you want to index <code>A</code> in <code>dim0</code>. If so, indexing should work:</p>
<pre><code class="lang-python">A[idx&gt;1] = 0.
</code></pre>

mmoelle1: <p>Thanks. Could you also let me know the C++ equivalent of this?</p>
Yolact ++ replication - DCN compilation issues

srichakra: <p>PyTorch errors with DCN compilation for <a href="https://github.com/dbolya/yolact" rel="noopener nofollow ugc">Yolact++</a></p>
<p><strong>Issue - 1:</strong> NotImplementedError: Cuda is not availabel<br>
<strong>Solution:</strong> Install the latest NVIDIA CUDA toolkit from <a href="https://developer.nvidia.com/cuda-toolkit" rel="noopener nofollow ugc">NVIDIA’s website</a>.<br>
<strong>Note:</strong>   Installing latest GeForce drivers for a few windows 10 versions causes BSOD errors. I have experienced the same with GeForce version 516.01 and Windows 10.0.19042 (Build 19042). Refer to <a href="https://www.nvidia.com/en-us/geforce/forums/game-ready-drivers/13/303794/nvidia-driver-keep-causing-bsod-after-windows-10-r/" rel="noopener nofollow ugc">this discussion</a> for more information.</p>
<p><strong>Issue - 2:</strong> fatal error: THC/THC.h: No such file or directory<br>
<strong>Solution:</strong> Use <a href="https://github.com/jinfagang/DCNv2_latest" rel="noopener nofollow ugc">DCNv2_ latest</a> instead of DCNv2 suggested in the yolact readme.</p>
How can i use convolution module?

a_bc: <p>hi. I only want to use “convolution” in libtorch. there are many .h files including what I don’t need, and it takes too much time to compile them all. Can I get some help using these? thank you.</p>

ptrblck: <p>I don’t think you can simply rip out a module from torch and build it.<br>
In case you don’t need Autograd etc. you might want to check the actual conv kernel implementations for the CPU and CUDA, and just reuse them in your application.<br>
Alternatively, check other repositories such as <a href="https://github.com/NVIDIA/cutlass/blob/master/media/docs/implicit_gemm_convolution.md">cutlass’ convolution implementation</a>.</p>
Cannot update to PyTorch Nightly

aw632: <p>Currently on stable and want to test out nightly. I use the command from the website<br>
<code>conda install pytorch torchvision torchaudio -c pytorch-nightly</code></p>
<p>but got the response</p>
<p><code># All requested packages already installed.</code></p>
<p>I tried again with</p>
<p><code>conda update -all pytorch torchvision torchaudio -c pytorch-nightly</code></p>
<p>but the version of torch is still <code>1.12.0</code>.</p>

ptrblck: <p>Uninstall your current PyTorch binary and reinstall the nightly. Alternatively, create a new virtual environment and install the nightly binary there.</p>