akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
About conversion of summarywriter to C++

Chen0729: <p>Hi, All</p>
<p>I have inquiry about how to convert tensorboard.summarywriter to C++<br>
The python code is<br>
Self.writer=tensorboardX.SummaryWriter()</p>
<p>Any suggestions are welcome.</p>
<p>Thanks!</p>

ptrblck: <p>I don’t think <code>tensorboard</code> has a built-in C++ API, but you might be able to find some 3rd party implementations.<br>
<a href="https://github.com/pytorch/pytorch/issues/26097">Here</a> is a tracker for a <a href="https://github.com/tensorflow/tensorboard/issues/2636">feature request</a> to enable C++ logging, which apparently didn’t get much attention.</p>
Class equivalent to ModelCheckpoint(Keras) in Pytorch?

Sandeep_Pathania: <p>Is there a class in Pytorch which can save the best accurate model of all the epochs automatically ?</p>
<p>For example : <a href="https://keras.io/api/callbacks/model_checkpoint/" rel="noopener nofollow ugc">https://keras.io/api/callbacks/model_checkpoint/</a></p>
<p>There, they have options like ModelCheckpoint which saves the best model. Also like EarlyStopping.</p>
<p>Just wondering if there is any alternative in Pytorch.</p>

ptrblck: <p>You should be able to find these higher-level APIs and hooks in wrappers such as <code>Ignite</code>, <code>PyTorch Lightning</code>, <code>Catalyst</code> etc.</p>
The first conv2d layer does not calculate gradient at all

Jacfger: <p>Long story short, I cannot get my correct gradient (at least in theory) when I was trying to do backpropagation.</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.input_conv = torch.nn.Conv2d(3, 2, 1)
        self.feature_extractor =  models.resnet50(pretrained=True)#resnext50_32x4d  resnet50 resnext101_32x8d
        num_ftrs = self.feature_extractor.fc.in_features  # for resnet
        self.output_fc = nn.Linear(num_ftrs, num_classes, bias = False)
    def forward(self, x):
        x = torch.cat((x[:, 0, :, :].unsqueeze(1), self.input_conv(x[:, 1:])), 1)
        x = self.feature_extractor(x)
        y = self.output_fc(x)
        return y 
</code></pre>
<p>Here’s my network, and I also had setup hook function such that I could store the gradients. (The code is copied from github repo called pytorch-cnn-visualization)</p>
<pre><code class="lang-auto">    def hook_layers(self):
        def hook_function(module, grad_in, grad_out):
            self.gradients.append(grad_in[0])
            self.gradients.append(grad_out[0])
            print(grad_in[0].shape, grad_out[0].shape)
        children = list(self.model.children())
        first_layer = children[0]
        first_layer.register_backward_hook(hook_function)
</code></pre>
<p>So in this network, the first_layer was set to <code>torch.nn.Conv2d(3, 2, 1)</code> which make sense. However, when I was trying to look at the gradient stored by the hook function, the shape of both grad_in and grad_out are [1, 2, 224, 244]. But it one of them should be [1, 3, 224, 224]. When I print them out and compared them it seems that their content is exactly the same. Is that a bug or I did something wrong?</p>

ptrblck: <p>It’s a known issue as described in the <code>Warning</code> in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=backward_hook#torch.nn.Module.register_backward_hook">docs</a>:</p>
<blockquote>
<p>The current implementation will not have the presented behavior for complex <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=backward_hook#torch.nn.Module"><code>Module</code></a> that perform many operations. In some failure cases, <code>grad_input</code> and <code>grad_output</code> will only contain the gradients for a subset of the inputs and outputs. For such <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=backward_hook#torch.nn.Module"><code>Module</code></a>, you should use <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.register_hook"><code>torch.Tensor.register_hook()</code></a> directly on a specific input or output to get the required gradients.</p>
</blockquote>
Gradient is None after applying custom mask

gesc: <p>I have  this forward function and I use binary masks mask1 and mask2 but  the module 1 get zero gradient  how can I mask out1 so it still gets to be updated?</p>
<pre><code class="lang-auto"> # x, z 2D matrices, mask1,mask2 are column vectors with matching height
 def forward(self, x, z, mask2, mask1):
        out1  = self.module1(x)
        in_2 = z * mask1 + out1 *mask2
        return  self.module2 (in_2)

</code></pre>
<p>Please help Me</p>

ptrblck: <p><code>module1</code> should create valid gradients as long as <code>mask2</code> isn’t full of zeros as seen here:</p>
<pre><code class="lang-python">module1 = nn.Linear(10, 10)
module2 = nn.Linear(10, 10)

x = torch.randn(10, 10)
z = torch.randn(10, 10)

mask1 = torch.randint(0, 2,(10, 10)).float()
mask2 = torch.randint(0, 2,(10, 10)).float()
# mask2 = torch.zeros(10, 10)

# forward
out1  = module1(x)
in_2 = z * mask1 + out1 *mask2
out = module2 (in_2)

# backward
out.mean().backward()
print(module1.weight.grad)
print(module2.weight.grad)
</code></pre>

gesc: <p>Thanks for the clarifications now it’s working!  my mask generator’s outputs were almost all  zeroes.</p>
Is this how to use the lr late scheduler?

Jake_Eum: <p>I want to use the AdamW in my efficientDet</p>
<pre><code class="lang-auto">   #optimizer = optim.AdamW(model.parameters(), lr=args.lr)
    optimizer = optim.AdamW(model.parameters(), lr=args.lr, betas=(0.9,0.999),eps=1e-08,weight_decay=0.02,amsgrad=False)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
        optimizer, patience=3, verbose=True)
</code></pre>
<p>I’m not sure this is how to use the optimizer for my model.<br>
then I want to know what betas do for this optimizer?</p>
<p>my loss for efficientDet d0 going down very slowly <br></p>
<pre><code class="lang-auto">=============================================================================
6 epoch: 	 start training....
4500 iteration: training ...
    epoch          : 6
    iteration      : 4500
    cls_loss       : 1.8731008768081665
    reg_loss       : 1.0041449069976807
    mean_loss      : 2.7572750695168025
4800 iteration: training ...
    epoch          : 6
    iteration      : 4800
    cls_loss       : 1.754671573638916
    reg_loss       : 0.9150880575180054
    mean_loss      : 2.7382697457438905
5100 iteration: training ...
    epoch          : 6
    iteration      : 5100
    cls_loss       : 1.5807626247406006
    reg_loss       : 0.8592696189880371
    mean_loss      : 2.7208735038097895
    time           : 332.5537312030792
    loss           : 2.720083791859683
epoch_7
=============================================================================
7 epoch: 	 start training....
5400 iteration: training ...
    epoch          : 7
    iteration      : 5400
    cls_loss       : 1.5356632471084595
    reg_loss       : 0.7544331550598145
    mean_loss      : 2.6795950939358284
5700 iteration: training ...
    epoch          : 7
    iteration      : 5700
    cls_loss       : 2.316020965576172
    reg_loss       : 1.078192949295044
    mean_loss      : 2.7014631390371915
    time           : 335.88806772232056
    loss           : 2.698524436832946
epoch_8
=============================================================================
8 epoch: 	 start training....
6000 iteration: training ...
    epoch          : 8
    iteration      : 6000
    cls_loss       : 1.8002285957336426
    reg_loss       : 0.8492593765258789
    mean_loss      : 2.634405281572115
6300 iteration: training ...
    epoch          : 8
    iteration      : 6300
    cls_loss       : 2.22102427482605
    reg_loss       : 0.9751409292221069
    mean_loss      : 2.6694228873293624
</code></pre>
About the conversion of torch::optim::Adam from python to libtorch

Chen0729: <p>Hi, All</p>
<p>I have an inquiry about converting the following from python to libtorch</p>
<p>python code is:<br>
self.Goptim = optim.Adam(self.G.parameters(), lr=args.lr, betas = (args.momentum,0.999))</p>
<p>In libtorch it has an error if I write</p>
<p>torch::optim::Adam  doptim(<br>
discriminator-&gt;parameters(),torch::optim::AdamOptions(args.lr_default).beta1(args.momentum).beta2(0.999) );</p>
<p>“No member named “beta1” in torch::optim::Adamoptions”</p>
<p>Any suggestion to correct it please ?</p>
<p>Thanks</p>

ptrblck: <p><code>beta1</code> and <code>beta2</code> are undefined and you should use <code>betas</code> instead as seen <a href="https://github.com/pytorch/pytorch/blob/93256617c8622760181dacf03c41cc0577ac0ea6/torch/csrc/api/include/torch/optim/adam.h#L24-L25">here</a>.</p>

Chen0729: <p>Thank you very much.</p>
<p>It works.</p>
Single Encoder multiple decoder

nshrimali: <p>Hi,</p>
<p>I want to integrate multiple pre-trained models(model1, model2) together, both have encoder and decoder architecture, I want to keep the encoder of model1 common for both of models.</p>
<p>With huge codes and pre-trained models at my disposal, what is the best way to extract decoder layers from the model2 and integrate it with encoder of model1 such that all 3 can co-exist?</p>

ptrblck: <p>I’m not sure if I understand the use case correctly, but if you want to use specific submodules from the pretrained models, you could use:</p>
<pre><code class="lang-python">data = torch.randn(...)
out = model1.encoder(data)
out = model2.decoder(out)
</code></pre>
<p>assuming that <code>model1</code> and <code>model2</code> are written in a way to easily access the <code>encoder</code> and <code>decoder</code> submodules.</p>

nshrimali: <p>Hi Peter,</p>
<p>Thanks for your response.</p>
<p>The two models are <a href="https://github.com/intel-isl/MiDaS" rel="noopener nofollow ugc">https://github.com/intel-isl/MiDaS</a> and <a href="https://github.com/NVlabs/planercnn" rel="noopener nofollow ugc">https://github.com/NVlabs/planercnn</a>, here I am getting no results by doing model.encoder, so I guess they are not defined properly.</p>
<p>I am trying to extract layers using model.children() and slice the list to separate the encoder and decoder part. I am planning to add weights in the layers by using state dict. Is this the right way to extract the layers and transfer learn. Or I have to dig deep into underlying code and then generate the layers?</p>

ptrblck: <aside class="quote no-group" data-username="nshrimali" data-post="3" data-topic="101923">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nshrimali/48/27038_2.png" class="avatar"> nshrimali:</div>
<blockquote>
<p>I am trying to extract layers using model.children() and slice the list to separate the encoder and decoder part.</p>
</blockquote>
</aside>
<p>If you are planning to extract the encoder/decoder layers via <code>model.children()</code> and use them in e.g. separate <code>nn.Sequential</code> modules, note that you would have to make sure these submodules can be executed sequentially without any functional calls in the original <code>forward</code>.</p>
<p>I would generally recommend to look into the model definition and make sure the <code>forward</code> is still executed as expected.</p>
Size of Tensors in Autoencoders

Hacking_Pirate: <p>Hi,<br>
I was trying to create a denoising AE, but as I was training it I got some size error eventhough both clear and pixelated images have same size.<br>
Here is model Arch:</p>
<pre><code class="lang-auto">############### MODEL ################
class AutoEncoder(nn.Module):
  def __init__(self):
    super(AutoEncoder, self).__init__()
    self.encoder = nn.Sequential(
        nn.Conv2d(3, 64, (3,3)),
        nn.MaxPool2d((2,2)),
        nn.Conv2d(64, 32, (3,3)),
        nn.MaxPool2d((2,2)),
        nn.Conv2d(32, 16, (3,3)),
        nn.MaxPool2d((2,2)),
        nn.Conv2d(16, 8, (3,3)),
        nn.MaxPool2d((2,2)),
    )
    self.decoder = nn.Sequential(
        nn.Upsample((16,16)),
        nn.ConvTranspose2d(8, 16, (3,3)),
        nn.Upsample((32,32)),
        nn.ConvTranspose2d(16, 32, (3,3)),
        nn.Upsample((64,64)),
        nn.ConvTranspose2d(32, 64, (3,3)),
        nn.Upsample((128,128)),
        nn.ConvTranspose2d(64, 3, (3,3)),

    )

  def forward(self, xb):
    encoded = self.encoder(xb)
    return self.decoder(encoded)

</code></pre>
<p>The size of images is 128, 128.<br>
HERE IS A LINK OF THE NOTEBOOK AS WELL:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201105-085602-RC00_340833107" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1YYwaG4vHa35oeGy7cxVJrsOz2k3bP4pi?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1YYwaG4vHa35oeGy7cxVJrsOz2k3bP4pi?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Hacking_Pirate: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> have you any idea?</p>

ptrblck: <p>Your model works fine with an input of <code>[batch_size, 3, 128, 128]</code>:</p>
<pre><code class="lang-python">model = AutoEncoder()
x = torch.randn(2, 3, 128, 128)
out = model(x)
print(out.shape)
&gt; torch.Size([2, 3, 130, 130])
</code></pre>
<p>so I guess the shape of your model output doesn’t match the target shape.<br>
If that’s the case, you would have to change the layer setups and make sure that both these tensors have an equal spatial size.</p>
Trying to run Anynet project

Carolmeir: <p>Hello,</p>
<p>I am trying to run <a href="https://github.com/mileyan/AnyNet" rel="noopener nofollow ugc">this</a></p>
<p>First I did</p>
<blockquote>
<p>sh ./create_dataset.sh</p>
</blockquote>
<p>And gave the correct path to scenflow_data_path.</p>
<p>Now I am trying to run this:</p>
<blockquote>
<p>python main.py --maxdisp 192 --with_spn</p>
</blockquote>
<p>as given, I am getting this following error. Please let me know how to correct this.</p>
<p>The error is:</p>
<pre><code>Traceback (most recent call last):
  File "main.py", line 194, in &lt;module&gt;
    main()
  File "main.py", line 51, in main
    args.datapath)
  File "/home/kbdp5524/Downloads/AnyNet-master/dataloader/listflowfile.py", line 23, in dataloader
    monkaa_path = filepath + [x for x in image if 'monkaa' in x][0]
IndexError: list index out of range
</code></pre>

ptrblck: <p>This error seems to be specific to the linked repository so you might also want to create an issue in the repo to get a better answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
That being said, based on the error message it seems as if the files are not found or the download failed.</p>

mingming36: <p>Hello, I have encountered the same problem. Have you solved your problem?</p>
Adam Optimizer + fp16 autocast

Scott_Hoang: <p>In order to avoid Null loss with  Adam optimizer using fp16 autocast, I must modify the eps value from 1e-8 to 1e-6. However, I found that by doing this, my model is much slower to converge, or not converge at all. Does anyone know why this would be?</p>

ptrblck: <p>Could you explain what issues you are seeing in the loss and what <code>Null</code> means in this case?</p>

Scott_Hoang: <p>the issue being when eps is set to be 1e-8, as is the default, and used with autocast, the network’s loss will inevitably be Null after some epoch. However, increasing eps to higher value seems to make it go away, yet when compare to training not done on fp16, the convergence is much slower.</p>

ptrblck: <p>Are you seeing <code>NaN</code> loss values after a time or what is <code>Null</code> referring to?<br>
If so, what kind of model architecture are you using?</p>

Scott_Hoang: <p>sorry, I meant Nan loss. I’m performing NAS search for pose estimation.</p>

ptrblck: <p>I’m not sure if your NAS implementation initializes the model parameters with “large” values explicitly or if the general training of these new architectures tends to create large output values, which might cause overflows easily. In that case I’m unsure if there is a better workaround than to increase the <code>eps</code> value, as <code>1e-8</code> might underflow in FP16 if you add it to a larger tensors.</p>

Scott_Hoang: <p>Since eps control the ratio in which step size is decided, a larger eps would indicate a smaller step size. Would you suggest that I increase the overall lr rate to compensate ?</p>

ptrblck: <p>The <code>eps</code> value is used <a href="https://github.com/pytorch/pytorch/blob/5a5258cb0df5f57640c5af0e2d36ef5c5f97cfb6/torch/optim/functional.py#L92-L94">here</a> to avoid dividing by a <code>0.</code> value.<br>
I just checked the <code>state_dict</code> of <code>Adam</code> using <code>autocast</code> and it seems all internal buffers are still stored in FP32, so I’m unsure why the <code>eps</code> value might cause trouble in this case (due to a potential underflow).</p>
<p>Here is a minimal code snippet:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10).cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

with torch.cuda.amp.autocast():
    out = model(torch.randn(1, 10).cuda())

loss = out.mean()
loss.backward()

optimizer.step()
print(optimizer.state_dict())
</code></pre>
<p><a class="mention" href="/u/mcarilli">@mcarilli</a> are you familiar with similar issues using <code>Adam</code>?</p>

Scott_Hoang: <p>I use GradScaler, is this the problem?<br>
can you use autocast independently from GradScaler?</p>
Double Integration

Jindong: <p>Hi, everyone,</p>
<p>I wonder if there is a function in Pytorch like “scipy.integrate.dblquad” to calculate the double integration of a given function.</p>
<p>For example,</p>
<p>given <strong>f(x,y)=2<em>x+2</em>y</strong></p>
<p>and we calculate <strong>∫∫f(x,y)dxdy</strong> with pytorch</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>I don’t think we have such function no.</p>
<p>Note that if you don’t need gradients through that computation, you can just translate the Tensor to numpy and use the corresponding scipy function.</p>

Jindong: <p>Hi, Thanks , for now, we program a Gaussian integral method with torch for the calculation.</p>
How to determine if the model is in train or eval phase?

Ziyu_Chen: <p>I’m writing a network where one parameter (I used register_buffer) is expected to be updated if the model is under model.train(), and not to change if is under model.eval(). So I need a (boolean) criterion such that in the forward function, the parameter will be updated based on whether the model is under train or eval. Is there any function to get this value? Thanks!</p>

Shisho_Sama: <p>Use the <code>training</code> attribute. its <code>True</code> when in train mode and <code>False</code> when in eval mode.</p>

Ziyu_Chen: <p>Thanks. I used “if self.training is True”, and it worked!</p>
Joining multiple iterative datastes

Rabeeh_Karimi: <p>Hi<br>
I have multiple iterative datasets with different size and I want to join them in a way that each datasets is sampled equally, so smaller datasets more, larger one less, could you tell me how to implement this in pytorch? thanks</p>

Henry_Chibueze: <p>Sorry I don’t really get ur question.</p>
<p>What exactly do you mean by different sizes?<br>
R u insinuating that there r multiple data sets and that the number of data points in each vary?</p>
<p>Also what exactly do u mean by sampled equally?</p>
<p>Sorry for replying u with so many questions<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji only-emoji" alt=":expressionless:"><img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji only-emoji" alt=":sweat_smile:"><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji only-emoji" alt=":pray:"></p>

Rabeeh_Karimi: <p>Hi there,<br>
I finally found the response to my question here <a href="https://discuss.pytorch.org/t/train-simultaneously-on-two-datasets/649/35" class="inline-onebox">Train simultaneously on two datasets</a> thanks.</p>

Henry_Chibueze: <p>Ok that’s cool</p>
<p>Good for U🙃</p>
Libtorch caffe2 cannot find CuDNN

arif_saeed: <p>I am trying to use the pytorch C++ frontend API and downloaded  libtorch  to /usr/local which is also where i have installed cuda-11.0.</p>
<p>I then downloaded the lubcudnn8 deb file and installed it.  Ubuntu installed the cudnn library to  /usr/lib/x86_64-linux-gnu</p>
<p>Now when i try to compile the example cmake file i get the following error:<br>
[cmake] – Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)<br>
[cmake] CMake Warning at /usr/local/libtorch/share/cmake/Caffe2/public/cuda.cmake:109 (message):</p>
<p>looking in the libtorch cuda.cmake file I can see that the command “find_package(CUDNN)” cannot find the cudnn library</p>
<p>So my best guess is that i should have saved the CUDNN library to usr/local, but I thought any library saved to  /usr/lib/x86_64-linux-gnu  is globally visible.</p>
<p>This is where I reach the edge of my C++ linking/CMake Find Package/Ubuntu knowledge.</p>
<p>Can anybody please point me in the direction of how i can solve this issue. it seems like i just need to make my CUDNN globally visible/or move the CUDNN files somewhere else, or tell libtoch where to look, but not sure what the best option is and how to do it.<br>
any help would be greatly appreciated</p>

arif_saeed: <p>I had only downloaded the Runtime Library, when I also downloaded the Developer Library and the Code Samples deb files and above error went away</p>
Frame to frame prediction using Vanilla LSTM

Thabang_Lukhetho: <p>I have the following frames form a video as images [batch_size=100,channels=2, image_height=64, image_width=64], a single batch consist of a sequence with 100 frames. I want to predict 1 frame ahead using a vanilla LSTM, I have transformed my input sequence as follows [batch_size, channels* image_height, image_width].</p>
<ol>
<li>How should I transform my data to be able to train such an architecture?</li>
</ol>
How to train additional parameters

ffabi: <p>Hi, I am trying to implement a trainable parameter in an existing codebase, which is overwhelmingly large for me at the moment.</p>
<p>The original codebase can be found at <a href="https://github.com/TRI-ML/packnet-sfm" rel="noopener nofollow ugc">https://github.com/TRI-ML/packnet-sfm</a></p>
<p>My main contribution should be trainable intrinsic parameters (which is basically a 3x3 matrix).</p>
<p>My best attempt is the following: a new Parameter is added to the SelfSupModel class which is set to zero in <code>__init__</code> and the initial value is set at the first call of <code>forward()</code>:</p>
<pre><code class="lang-python">class SelfSupModel(SfmModel):
    def __init__(self, **kwargs):
        ...
        self.intrinsic = torch.nn.Parameter(torch.zeros((3, 3)))
        self.intrinsic.requires_grad = True
    ...
    def forward(self, batch, return_logs = False, progress = 0.0):
        ...
        if self.intrinsic.sum() == 0:
            self.intrinsic = torch.nn.Parameter(batch['intrinsics'][0])

        stored_intrinsic = torch.stack(batch['intrinsics'].shape[0] * [self.intrinsic], 0)
        self_sup_output = self.self_supervised_loss(
            batch['rgb_original'], batch['rgb_context_original'],
            output['inv_depths'], output['poses'], stored_intrinsic,
            return_logs = return_logs, progress = progress)


</code></pre>
<p>I set <code>requires_grad = True</code> and the <code>intrinsic</code> can be found in <code>SelfSupModel.named_parameters()</code> with <code>requires_grad = True</code>.</p>
<p>Upon printing out this Parameter, it is not updated after each epoch.<br>
What can be the reason for not being updated?</p>
<p>I assume that this is not the best possible way to initialize this Parameter and this might not be optimized for GPU.</p>
<p>Thanks, Fabian</p>

googlebot: <p>Sample-specific values are not considered model parameters. Though you can train such values, you won’t be able to make out-of-sample predictions. Perhaps you need group-specific values. In both cases, you should probably use nn.Embedding index lookups (do embedding.weight.data.zero_() as it is initialized with gaussian noise by default)</p>

ffabi: <p>Thank you Alex,<br>
out-of-sample prediction might be useful later on, so  I think I should add prediction of intrinsic parameters to the PoseNet.</p>
<p>Fabian</p>
Why is the hidden state initialized to zero for every batch when doing forwad pass?

Thabang_Lukhetho: <p>I have a couple of questions:</p>
<ol>
<li>Why is the hidden state initialized  to zero for each batch when doing a forward pass? And when are you supposed to this during training or testing or both?</li>
<li>If you initialized the hidden state with a <strong>batch_size=128</strong> during training and for testing the <strong>batch_size=1</strong>, would I have to initialize the hidden state to 0 with a new batch size?</li>
</ol>
<pre><code class="lang-auto">def forward(self, x):
    batch_size = x.shape[0]
    hidden = (torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device), torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device))

    output, hidden = lstm(x, hidden)
    
    # then do what every you want with the output`
</code></pre>

Abhilash_Srivastava: <aside class="quote no-group" data-username="Thabang_Lukhetho" data-post="1" data-topic="101263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/8e8cbc/48.png" class="avatar"> Thabang_Lukhetho:</div>
<blockquote>
<p>Why is the hidden state initialized to zero for each batch when doing a forward pass? And when are you supposed to this during training or testing or both?</p>
</blockquote>
</aside>
<p>You can do it for each epoch too (instead of each batch), which I think is better. You’ll need to do it for both training and testing (and validation).</p>
<aside class="quote no-group" data-username="Thabang_Lukhetho" data-post="1" data-topic="101263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/8e8cbc/48.png" class="avatar"> Thabang_Lukhetho:</div>
<blockquote>
<p>If you initialized the hidden state with a <strong>batch_size=128</strong> during training and for testing the <strong>batch_size=1</strong>, would I have to initialize the hidden state to 0 with a new batch size?</p>
</blockquote>
</aside>
<p>Yes, the initialization should match the batch_size of the input.</p>
<aside class="quote no-group" data-username="Thabang_Lukhetho" data-post="1" data-topic="101263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/8e8cbc/48.png" class="avatar"> Thabang_Lukhetho:</div>
<blockquote>
<pre><code class="lang-auto">    hidden = (torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device), torch.zeros(self.layers, batch_size, self.hidden_size).to(device=device))
</code></pre>
</blockquote>
</aside>
<p>hidden should be passed as the input the <code>forward</code> function, not initialized in the here. Check out an example <a href="https://blog.floydhub.com/long-short-term-memory-from-zero-to-hero-with-pytorch/" rel="noopener nofollow ugc">here</a>.</p>

Thabang_Lukhetho: <p>Thank this helped as lot</p>
Unable to install the correct version

nikogamulin: <p>Hi,</p>
<p>I created a virtual environment and wanted to install the current latest stable version (1.7) with the following command:</p>
<p>pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>When I check the version, it seems that 1.4.0 is installed:</p>
<pre><code class="lang-auto">import torch
&gt;&gt;&gt; print(torch.__version__)
1.4.0
</code></pre>
<p>Does anyone know why the 1.7.0 version is not installed?</p>
<p>This is the output from the installation procedure:</p>
<pre><code class="lang-auto">Installing collected packages: torch, torchvision
  Found existing installation: torch 1.7.0+cu101
    Uninstalling torch-1.7.0+cu101:
      Successfully uninstalled torch-1.7.0+cu101
  Found existing installation: torchvision 0.8.1+cu101
    Uninstalling torchvision-0.8.1+cu101:
      Successfully uninstalled torchvision-0.8.1+cu101
Successfully installed torch-1.7.0+cu110 torchvision-0.8.1+cu110
</code></pre>
<p>If I try to install it in conda environment (using conda install), the versions match.</p>

ptrblck: <p>I guess you might have multiple PyTorch installations in your current environment, so before installing the latest <code>1.7.0</code> version you should uninstall the older ones.</p>
Torchsummary AttributeError: 'int' object has no attribute 'numpy

TAF: <p>Hello everyone, I built a simple model and I want to know the number of its parameters using torchsummary, but I got an error: “AttributeError: ‘int’ object has no attribute 'numpy”</p>
<pre><code class="lang-python">class LinearRegression(nn.Module):
def __init__(self, in_features: int, out_features: int, bias: bool = True):
        super().__init__()
        self.weights = nn.Parameter(torch.randn((in_features,out_features),requires_grad=True))

        self.bias = bias
        if bias:            
            self.bias_term = nn.Parameter(torch.rand((out_features),requires_grad=True))
    def forward(self, x):
        x =  x@ self.weights
        if self.bias:
            x +=  self.bias_term
        return x
from torchsummary import summary
linear_regression = LinearRegression(2, 1)
summary(model=linear_regression,input_size=(4,2),device='cpu')
</code></pre>
<p>Please tell me what am I doing wrong?</p>

ptrblck: <p>This seems to be a known issue in <a href="https://github.com/sksq96/pytorch-summary/issues/134">pytorch-summary</a> and based on <a href="https://github.com/sksq96/pytorch-summary/issues/124">this issue</a> it seems <a href="https://github.com/TylerYep/torch-summary">torch-summary</a> is better maintained now.</p>
Help! RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 4; 11.91 GiB total capacity; 11.22 GiB already allocated; 12.94 MiB free; 121.63 MiB cached)

caoty: <p>I have about 8000 sentences and i try to calculate their representation with transformer. I firstly fed the 8000 sentences into the transformer at the same time but it reported “CUDA out of memory”. Then i fed 32 sentences into transformer one time to recurrently process these sentences, but again it reported this error: I only post code of this part below(note sentidx is a tensor of shape 8000*47,batch_size = 32, self.emb is the nn.Embedding module. self.encoder is the standard transformer module provided by torch.nn. When k=54,  it will report CUDA out of memory. Does anyone have suggenstion?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ce81fd19eb00c48f2d7e5e8d3517aae9892312f.png" data-download-href="https://discuss.pytorch.org/uploads/default/1ce81fd19eb00c48f2d7e5e8d3517aae9892312f" title="error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ce81fd19eb00c48f2d7e5e8d3517aae9892312f.png" alt="error" data-base62-sha1="47IGYz3pgh1p1VdzFESH0vHlIZh" width="690" height="114" data-dominant-color="070807"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1099×182 5.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Problem with loading data to rcnn model

Tomash: <p>Hi, I am trying to train rcnn model, but I have few problems when I am using batch_size&gt;1.<br>
My function which make dataset returns a single img tensor and a single target (dictionary):</p>
<pre><code class="lang-auto">        return (
            img,
            target,
        )
</code></pre>
<p>Later I am loading datasets to dataloader which loads data to the training process.</p>
<pre><code class="lang-auto">   with tqdm(train_loader) as _tqdm:
        for x, y in _tqdm:
            x = x.to(device)
            for key, value in y.items():
                y[key] = torch.tensor(value).to(device)

            outputs = model(x, y)
</code></pre>
<p>When batch_size is f.e. == 2, then <code>x</code> is a list of two tensors of shape <code>[D, H, W]</code> but <code>y</code> is a <code>dict</code> where every <code>key</code> have two values. Then occurs error:</p>
<p><code>.../generalized_rcnn.py", line 64, in forward boxes = target["boxes"] TypeError: string indices must be integers</code><br>
So I assumed that probably it can’t handle with dictionaries and I pass:</p>
<blockquote>
<p>output(x, [y])</p>
</blockquote>
<p>Then error changes to:</p>
<pre><code class="lang-auto">.../transform.py", line 99, in forward
target_index = targets[i] if targets is not None else None
IndexError: list index out of range
</code></pre>
<p>Any ideas how to make it run properly?</p>
Pytorch model parameters deepcopy

Ohm: <p>Hi guys,<br>
I tried to use “deepcopy”, but It changed the parameters(gradients), while I did not want to change it.<br>
The goal is P[1].grad be updated when Itr%2 is zero and save as gradient of Net_Kfast. And when Itr%2 is not zero, I want the P[1].grad remain the same…<br>
#<code>Preformatted text</code></p>
<pre><code class="lang-auto">       for P in zip(net.parameters(), Net_Kfast.parameters(), *net_parameters):
           temp = 0
           for p in P[2:]:
               temp += p.grad/len(Participated_Workers)

           P[0].grad = temp
           
           if Itr%2 == 0:
               P[1].grad = copy.deepcopy(temp)
   
     ```</code></pre>
How to efficiently count the number of images in each class in a dataset?

ProGamerGov: <p>I have been using this code to count the number of images in each class for my dataset, but it’s extremely slow. Is there a better more efficient way to get the number of images in each class?</p>
<pre><code class="lang-auto">def count_classes(dataset):
    class_counts = dict(Counter(sample_tup[1] for sample_tup in dataset))
    return dict(sorted(class_counts.items()))
</code></pre>
How to write parallel for loop in model's forward method?

yolle103: <p>hi, i currently have a model that intakes a tensor N x C x H x W and a label tensor N x L. In the forward method, I want to first use different net for different label and then combine them together for other layers. so i current have some code like:</p>
<pre><code class="lang-auto">def forward(self, data):
    for i in range(label_types_num):                                                  
        idx = get_label_idx(i) # get all the index that has this label
        group_data = data[idx, :]                                               
        o = self.seperate_nets[i](group_data)                                                                                                
        output_tensor[idx, :] = o                                               
    # then output_tensor is passed to other layers...             

</code></pre>
<p>Just wondering is there a way to parallel do the for loop?</p>

LeviViana: <p>I think you could use <code>torch.distributed.launch</code>. <a href="https://discuss.pytorch.org/t/how-to-use-multigpu-pytorch/46290/2">This snippet</a> may be useful to you to get a better grasp on how <code>torch.distributed.launch</code> works.</p>

yolle103: <p>hi, i take a look at that snippet, and it seems for me that the distribution is done by distributing data onto different GPUs (correct me if i’m wrong). However, here I would like some solutions that parallel the for loop in a single GPU case since each ‘data’ in my code itself is already in a single GPU when calling the forward.</p>

LeviViana: <p>If you analyze more carefully the snippet, you will notice that <code>torch.distributed.launch</code> launches CPU threads. Indeed, you can leverage this to run a model on multiple GPUs (one per thread), but you can do much more, like running multiple threads for one GPU. One limitation of doing this is that, as far as I can tell, the threads don’t share memory, which means that if you launch 10 threads on the same GPU, it will allocate 10x more memory.</p>

yolle103: <p>thank you so much! that makes sense, i’ll try to find a balance between memory and speed for my model.</p>

Juans: <p>Is there any concrete solution to this? An example solution would be very helpful, as the suggested post does not make this use-case explicit.</p>
Multiple dataset dataloaders

Rabeeh_Karimi: <p>Hi<br>
I have multiple datasets and I want to join them as one datasets, could you provide me with example how to form dataloader for multiple tasks? thanks</p>

Shisho_Sama: <p>If you have multiple Pytorch compatible datasets, you can use <code>torch.utils.data.ConcatDataset</code> . Have a look <a href="https://pytorch.org/docs/stable/data.html" rel="noopener nofollow ugc">here</a>.</p>
How to start train with a few images

rui_zhou: <p>I am trying fine-tuning pre-trained model using my own dataset, but the model does not function well, is there any way? I have been trying these days but I can not product it out, can you give me some help?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3abad17c097be9a182aedef7c9b17f7f438db75e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3abad17c097be9a182aedef7c9b17f7f438db75e" title="16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abad17c097be9a182aedef7c9b17f7f438db75e_2_690x355.jpeg" alt="16" data-base62-sha1="8nxWSr6NdzCBUq5kdXUkycbAsP4" width="690" height="355" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abad17c097be9a182aedef7c9b17f7f438db75e_2_690x355.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abad17c097be9a182aedef7c9b17f7f438db75e_2_1035x532.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abad17c097be9a182aedef7c9b17f7f438db75e_2_1380x710.jpeg 2x" data-dominant-color="353638"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">16</span><span class="informations">1490×768 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to operate in Conv2d without removing it from computation graph

Habibur_Rahman: <p>I want to perform an l2_norm operation on conv2d filter in each forward pass before applying the filter. I tried to calculate the l2_norm first and then used torch.nn.Parameters to update the new parameter of conv2d but it create a new computation graph for every forward pass. How can do in the same computational graph. Thanks in advanced.</p>
Pytprch installation in colab

shrutishrestha: <p>Can we install PyTorch 1.6 in collab? If yes then how to?</p>

ptrblck: <p>Yes, you could use the <code>pip install</code> command from <a href="https://pytorch.org/get-started/locally/">here</a> in the notebook.<br>
Note that your Colab notebook should already have PyTorch preinstalled.</p>
Training Killed During Epoch

shawnbzhang: <p>In the middle of training on the first epoch, my process was killed. Going into <code>/var/log/syslog</code>, I read: <code>Out of memory: Killed process 70837 (python)</code>, which seems to be the reason why SIGKILL was called. Meanwhile, I have tested that if I change my batch size from 16 to 8, training can run to full completion. This leads me to the belief that my RAM usage is increasing per iteration, but resetting per epoch. What could be the reason for this, and what should I do to fix this?</p>
<p>Here are notable snippets of my code:</p>
<pre><code class="lang-auto">model.train()
for t in range(num_of_epochs):
    for step in range(num_of_batches):
        length_data, input, target = loader.get_a_batch()  # Get a batch of data from my custom dataloader class
        if torch.cuda.is_available():
            input = input.cuda()
            target = target.cuda()
        
        model.zero_grad()
        output = model(input)

        loss = loss_func(target, output, length_data)  # Read below for specific code snippet
        
        optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm(model.parameters(), grad_clip_val)
        optimizer.step()
        
        loss = float(loss.item())
        if loss &lt; best_loss:
            torch.save(model.state_dict(), ...)
            best_loss = loss
</code></pre>
<p>where</p>
<pre><code class="lang-auto">def loss_func(target, output, length_data):
    loss_fn = torch.nn.MSELoss()

    sum_loss = 0.0
    for i in range(len(length_data)):
        length = length_data[i]
        sum_loss += loss_fn(target[i, 0:length, :], output[i, 0:length, :])
        
    return sum_loss
</code></pre>
<p>Reading on previous discussions, I feel like my problem might come from <code>loss_func</code> where I am summing the losses. However, I feel like the way that I am current doing is necessary for the backpropagation. Would anyone have any advice? Thank you.</p>
<p>Note: My model is an RNN dealing with variable-sized input, which is why you see some of the formatting above.</p>
Significance of relu for fc layers

SAI_VARSHITTHA: <p>How significant is adding relu to fully connected layers? Is it necessary or how the performance of a model is affected by adding relu to Fc layers?</p>

ptrblck: <p>Without activation functions your stacked linear layers can be seen as a single linear mapping and your model might thus fail easily.<br>
<a href="https://www.deeplearningbook.org/contents/mlp.html">This chapter</a> in the DeepLearningBook explains more about activation functions and uses the XOR example.</p>
Cuda out of memory error -BERT

Anamika_Singh: <p>I am training BERT model for sentiment analysis, with train data size 80k, but getting out of memory error for batch size 128,256 and above.</p>
<p>Here is the stack trace,</p>
<p>/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions)<br>
337             encoder_hidden_states,<br>
338             encoder_attention_mask,<br>
–&gt; 339             output_attentions,<br>
340         )<br>
341         attention_output = self.output(self_outputs[0], hidden_states)</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
725             result = self._slow_forward(*input, **kwargs)<br>
726         else:<br>
–&gt; 727             result = self.forward(*input, **kwargs)<br>
728         for hook in itertools.chain(<br>
729                 _global_forward_hooks.values(),</p>
<p>/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions)<br>
257         # Take the dot product between “query” and “key” to get the raw attention scores.<br>
258         attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))<br>
–&gt; 259         attention_scores = attention_scores / math.sqrt(self.attention_head_size)<br>
260         if attention_mask is not None:<br>
261             # Apply the attention mask is (precomputed for all layers in BertModel forward() function)</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 844.00 MiB (GPU 0; 15.90 GiB total capacity; 14.36 GiB already allocated; 377.88 MiB free; 14.63 GiB reserved in total by PyTorch)</p>
<p>Can someone please suggest on how to resolve this.<br>
I am using Colab GPU, is there any limit on size of training data for GPU with 15gb RAM?</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="Anamika_Singh" data-post="1" data-topic="101953">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anamika_singh/48/30069_2.png" class="avatar"> Anamika_Singh:</div>
<blockquote>
<p>I am using Colab GPU, is there any limit on size of training data for GPU with 15gb RAM?</p>
</blockquote>
</aside>
<p>Yes, there would always be an upper limit, which fits on your GPU (and of course also system RAM).<br>
You might need to lower the batch size for the particular GPU you are using, if you are running out of memory.</p>
WeightedRandomSampler Error

Jeel_Sukhadiya: <p>I am doing the weightedrandomsampler on my dataset. I am using it to create the training data, can someone tell me what I am doing wrong<br>
Code and Error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05408c88fea144a534ab3fe72fb72bce1ff75ef3.png" data-download-href="https://discuss.pytorch.org/uploads/default/05408c88fea144a534ab3fe72fb72bce1ff75ef3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05408c88fea144a534ab3fe72fb72bce1ff75ef3.png" alt="image" data-base62-sha1="KsGkXH6vDtyj2EpZkd1zjh9STx" width="582" height="500" data-dominant-color="F0F1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">969×832 50.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t think the <code>WeightedRandomSampler</code> creates the error, but your <code>pandas.DataFrame</code> raises a <code>KeyError</code> so you should check how this <code>DataFrame</code> is called and make sure the keys are valid.</p>
<p>PS: it’s better to post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Torch cut range [0,1]

Johnson_Mark: <p>Hello all. my prediction tensor (output of a CNN size of <code>BxCxHxW</code>) has to in range [0,1]. However, the prediction tensor sometime in the range smaller than 0 or bigger than 1. Do we have any torch function to cutting the negative value and bigger than one, so that the range of tensor will be in [0,1]</p>
<pre><code class="lang-auto">if x&lt; 0 --&gt; x=0
if x&gt;1 --&gt;x=1
</code></pre>
<p>This is my solution</p>
<pre><code class="lang-auto">x = torch.clamp(x, min=0, max=1)

</code></pre>

ptrblck: <p><code>clamp</code> or <code>sigmoid</code> should work depending if you want a hard “cut” or want to use a smooth function.</p>
Torch.utils.data issue with v1.7

AlexisW: <p>It seems the call <code>from torch.utils.data import *</code> is broken in v1.7 while working in v1.6. Is this desired?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.__version__)
1.7.0
&gt;&gt;&gt; from torch.utils.data import *
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: module 'torch.utils.data' has no attribute 'BatchSamplerDistributedSamplerDataset'
</code></pre>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/pull/43406">This PR</a> unfortunately broke the star import by forgetting to add commas between the class names.<br>
However, it’s already fixed in the nightly binaries so you could update if you need to use this way of importing the classes.<br>
Note that importing or using the classes directly e.g. via:</p>
<pre><code class="lang-python">from torch.utils.data import BatchSampler
</code></pre>
<p>would still work.</p>