akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Making learnable variables

Ali_Shahbaaz_Khan: <p>I am creating a custom loss function that uses two metric</p>
<pre><code class="lang-auto">def loss_fn(preds,target,a,b) :
          loss1 = loss_func1(preds,target)
          loss2 = loss_func2(preds,target)
          loss = a*loss1 + b*loss2
          return loss
</code></pre>
<p>How do i make variable a and b, in the above function, learnable in the training loop</p>

tom: <p>You need to make two <code>nn.Parameters</code> for a and b (on the device <code>nn.Paramter(torch.tensor(1.0, device=my_device))</code>) or so and pass those to the optimizer (e.g. <code>list(model.parameters()) + [a, b]</code> instead of just the parameters).</p>
<p>Note that generally it is mathematically difficult to train loss weights, but whether this applies in your case is hard to tell at this level of detail.</p>
<p>Best regards</p>
<p>Thomas</p>

Ali_Shahbaaz_Khan: <p>so i just initialize the variables in the constructor of my network class and use them as netobject.a or do i have to write something in the backward function also?</p>

tom: <p>Yeah, you could assign them to your own Loss module or something.<br>
No, backward will be automatically.</p>
Training for more iterarions

Michael_Lempart: <p>Hi all,</p>
<p>Iam totally new to PyTorch as Iam transitioning from keras.</p>
<p>Iam about to translate one of my projects, where one epoch is either as long as the number of samples devided by the batch size or as long as the number of iterations defined by a user.</p>
<p>Now Iam wondering how I could accomplish the same thing in PyTorch?</p>
<p>I first create a Datset class and wrap pytorchs Dataloader around it as recommended in this tutorial</p>
<p><a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" class="onebox" target="_blank" rel="nofollow noopener">https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel</a></p>
<p>That works fine for number of iterations = len(samples) /batch size</p>
<p>Now I want to increase the number of iterations, so I guess the generator has to be restarted and shuffle.</p>
<p>Any help on this would be awsome,</p>
<p>Cheers,</p>
<p>Michael</p>

tom: <p>The easiest way would be to just iterate over subepochs and break after the number of iterations you want.</p>
<pre><code class="lang-auto">for e in range(num_epochs):
  it = 0
  while it &lt; num_it:
    for batch in data_loader:
      it += 1
      if it &gt;= num_it:
        break
      ....
</code></pre>
<p>It’s a tad verbose, but in the end at 5 additional lines of code, you might pretend it’s an enlightenment of the “explicit is better than implicit” variety.</p>
<p>As you note, the proper abstraction to tie into would be the DataLoader. Thus, the perhaps more sophisticated technique is to write your own variation of <code>RandomSampler</code> to pass to the dataloader. This should be simple enough, too, you just need to return an iter over a few distinct randperms in <code>__iter__</code>, similar to what <a href="https://github.com/pytorch/pytorch/blob/94c3cdd994dfe276b89ffb412cfe8a80b2f37bc7/torch/utils/data/sampler.py#L125" rel="nofollow noopener">RandomSampler</a> does.</p>
<p>Best regards</p>
<p>Thomas</p>

Michael_Lempart: <p>Hi Thomas and thanks for the quick reply!</p>
<p>What happens if the data_loader runs out of batches?</p>
<p>Does it ‘reset’ itself and shuffle the data?</p>
<p>Cheers,</p>
<p>Michael</p>

tom: <p>No, the DataLoader is gives an iterator (<code>__iter__</code>) and that terminates (with the usual Python mechanism - StopIteration) the for loop. That’s why the manual solution puts in an outer loop to go over this for loop several times if needed.</p>

Michael_Lempart: <p>Yes, that makes sense.</p>
<p>Do you have any idea if the dataset is shuffled  when ‘for batch in data loader’ is repeated?</p>
<p>Thanks again,</p>
<p>Michael</p>

tom: <p>Yes, it is. The linked codeline in the RandomSampler generates a random permutation and returns an iter on it. This is done at every start of a for loop.</p>

Michael_Lempart: <p>Awsome, thanks Thomas!</p>
<p>/Michael</p>
How sparsemax works?

aaditya: <p>Is my definition of Sparsemax correct?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56c3ff25823fda02d135e8d1a7345c03f12ad17e.png" alt="Screenshot 2020-09-23 at 10.40.37 PM" data-base62-sha1="cnyYY6LogcblJbC4Av1dBg24IRU" width="285" height="146"></p>
How to get 2D index of maximum value of 2D tensor?

Hao_Meng: <p>as the title shows , it would be better to provide code.<br>
Thanks!</p>

Samarth_Mishra: <p>You could use (assuming <code>A</code> is the tensor)</p>
<pre><code class="lang-python">idx = A.argmax()
x, y = idx//A.shape[1], idx%A.shape[1]
</code></pre>
Mel Spectrogram in torchaudio backprop

Mahmoud_Abdelkhalek: <p>Is the <a href="https://pytorch.org/audio/transforms.html#melspectrogram" rel="nofollow noopener"><code>torchaudio.transforms.MelSpectrogram</code></a> class in <code>torchaudio</code> differentiable? As in, can I backpropagate through it? If so, can someone point to me towards some documentation on how this backpropagation works?</p>

tom: <p>I think so, just do <code>requires_grad_</code> on your input. In the end it goes through <code>torchaudio.transforms.functional.spectrogram</code> and uses the <code>torch.stft</code> function. This calls <code>torch.fft</code> (I think), which has a derivative defined.<br>
There are several texts about how the inner parts of PyTorch work, <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="nofollow noopener">I wrote something simple a long time ago</a> and <a class="mention" href="/u/ezyang">@ezyang</a> has an awesome comprehensive <a href="http://blog.ezyang.com/2019/05/pytorch-internals/" rel="nofollow noopener">tour of PyTorch internals</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

Mahmoud_Abdelkhalek: <p>Thanks a lot for the information! Especially the links about the inner workings of PyTorch!</p>
Retrieve elements from a 3D tensor with a 2D index tensor

thisisjim2: <p>I am playing around with GPT2 and I have 2 tensors:</p>
<p>O: A Tensor of shaped (B, S-1, V) where B is the batch size S is the the number of timestep and V is the vocabulary size. This is the output of a generative model and is softmaxed along the 2nd dimension.</p>
<p>L: A 2D tensor shaped (B, S-1) where each element is the index of the correct token for each timestep for each sample. This is basically the labels.</p>
<p>I want to extract the predicted probability of the corresponding correct token from tensor O based on tensor L such that I will end up with a 2D tensor shaped (B, S). Is there an efficient way of doing this apart from using loops?</p>

mbednarski: <p>Hello</p>
<p>If I understood correctly, you need <code>gather</code> function. I have created an explanation how to use it, I hope it will be  helpful for you: <a href="https://medium.com/analytics-vidhya/understanding-indexing-with-pytorch-gather-33717a84ebc4" rel="nofollow noopener">https://medium.com/analytics-vidhya/understanding-indexing-with-pytorch-gather-33717a84ebc4</a></p>
<p>I’m linking my own article but I believe it will be helpful.</p>
About Training Time of pytorch and tensorflow

hdc: <p>I notice that the training time of first epoch is some kind longer than the following epochs by using the code of tensorflow version. I guess it is because tensorflow has cached all batchs of data in the first epoch, then the processing of the following epochs can be very quick?</p>
<p>Can I reimplement this operation with pytorch? It may reduce the training time in total.</p>
<p>Thanks for your advice.</p>

pfloat: <p>Hello,</p>
<p>We may answer you better with some code.</p>
<p>In the case of PyTorch, if you have enough memory, you could move the whole dataset tensor to the GPU device:</p>
<pre><code class="lang-auto">device = ...
dataset_tensor = dataset_tensor.to(device) # works for batches too, just need to loop over them
</code></pre>
<p>Then, start your training loop and obtain a lower training time. Anyway, the dataset batches still have to be loaded in the memory. Either way, the total execution time for the script may probably be a little lower only.</p>

hdc: <p>Thanks for your reply.</p>
<p>Sorry, maybe my expression is not accurate.</p>
<p>I mean why the training time of the first epoch is pretty longer than the training time of the subsequent epochs in tensorflow.</p>
<p>When performing the same operation (caching some batchs of data) in pytorch, I cannot find such a significant time reduction.</p>
<p>Does tensorflow use some other optimization operations that pytorch does not?</p>
<pre><code class="lang-auto">if index in self.cache:
    img, cls = self.cache[index]
else:
    ... ...
    if len(self.cache) &lt; self.cache_size:
        self.cache[index] = (img, cls)
</code></pre>

hdc: <p>Problem solved. I have found a bug in my code…</p>
Best way to define a scalar using nn.Parameter in Pytorch

mohit117: <p>In my CNN at some stage I want to multiply a feature map with some scalar which should be learnt by the network. The scalar has to be initialised to 5. So I think of the following solution,</p>
<pre><code class="lang-auto">def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(torch.ones(1)*5)
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x
</code></pre>
<p>This should work because of the <a href="https://pytorch.org/docs/stable/notes/broadcasting.html#broadcasting-semantics" rel="nofollow noopener">BROADCASTING SEMANTICS</a> of PyTorch. But I wonder why or why not the following be used,</p>
<pre><code class="lang-auto">def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(torch.tensor(5.))
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x
</code></pre>
<p>Surely this should work but is there any pitfall for the latter approach when compared to the first approach which uses <code>torch.ones(1)*5</code>. Generally I do not see people using the latter approach and use the first approach only.</p>
<p>Thankyou<br>
Mohit</p>

tom: <p>I agree with your that torch.tensor(5.) would seem to be even better here for what you’re trying to achieve.</p>
<ul>
<li>It seems most concise.</li>
<li>The tensor will be truly a scalar (0-dim) tensor (you’d need to do torch.ones(())) for that.</li>
<li>There isn’t any performance aspect. (e.g. when you’re in the middle of a computation, it’s always good to create the tensor on the right device rather than starting on CPU moving it, but this isn’t applicable here.)</li>
<li>As an aside, torch.full would probably be more natural than torch.ones.</li>
</ul>
<p>One likely reason this isn’t used as much is that in the very olden days (before 0.4?), PyTorch didn’t have 0-dim tensor support and these things are very hard to get out of people’s heads (like Variable or using torch.Tensor or using .data left and right). It seems people find old code with that an then adapt it so it runs and does what they want (which is, of course, OK and I’m happy when people can make new stuff from old code). I’m sure I’m guilty of not moving all my code to best practices.</p>
<p>Best regards</p>
<p>Thomas</p>

mohit117: <p>Thankyou for the explanation.</p>
<p>Just one more thing. I checked the documentation of <code>torch.full</code> and <code>torch.ones</code>. By default their <code>requires_grad=False</code>. I hope when I wrap them inside <code>nn.Parameter</code> it would be automatically switched to <code>True</code>?</p>
<p>Thankyou</p>

tom: <p>Yes, nn.Parameter sets the requires_grad to true unless you explicitly ask it not to.</p>
Where can I find source code of mse_loss() in libtorch?

kila_suelika: <p>From <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">pytorch’s</a> project in github, I can find <strong>MSELossImpl::forward()</strong> in <strong>torch/csrc/api/src/nn/modules/loss.cpp</strong>.</p>
<p>This function calls <strong>F::detail::mse_loss(input, target, options.reduction());</strong>, it’s in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/nn/functional/loss.h" rel="nofollow noopener">torch/csrc/api/include/torch/nn/functional/loss.h</a>.</p>
<p>Again <strong>F::detail::mse_loss</strong> magically calls:</p>
<pre><code class="lang-auto">return torch::mse_loss(
    expanded_input,
    expanded_target,
    enumtype::reduction_get_enum(reduction)); 
</code></pre>
<p>However I can’t find defination of <strong>torch::mse_loss()</strong>.</p>
<p>torch/csrc/api/include/torch/nn/functional/loss.h includes 3 files:</p>
<pre><code class="lang-auto">#include &lt;ATen/ExpandUtils.h&gt;
#include &lt;torch/nn/functional/activation.h&gt;
#include &lt;torch/nn/options/loss.h&gt;
</code></pre>

albanD: <p>Hi,</p>
<p>Both the c++ and python binding are generated based on <a href="https://github.com/pytorch/pytorch/blob/76dc50e9c8698da338334ecdc80bb00e60186849/aten/src/ATen/native/native_functions.yaml#L6392">this file</a>.<br>
Here since there are no specific function name for the backend, there is a function in Aten/native with that name.  For mse, you can find it <a href="https://github.com/pytorch/pytorch/blob/76dc50e9c8698da338334ecdc80bb00e60186849/aten/src/ATen/native/Loss.cpp#L332">here</a>. Hope this helps.</p>

kila_suelika: <p>Thank you, it’s very useful.</p>
Build with _GLIBCXX_USE_CXX11_ABI=1

wuclddwf: <p>Hi：<br>
I want to build pytorch libs with  _GLIBCXX_USE_CXX11_ABI=1 in branch v1.1.0.  And I reivise <em>TorchConfig.cmake.in</em> file.  But it seems not to work.  Is there some suggents?</p>
<p>thanks</p>

LeviViana: <p>You can add <code>set(GLIBCXX_USE_CXX11_ABI 1)</code> in the beginning of <code>pytorch/CMakeLists.txt</code>.</p>

yf225: <p>We now have new CXX11 ABI binaries for libtorch. They can be found on<br>
<a href="http://pytorch.org/" rel="nofollow noopener">http://pytorch.org</a>, or at <a href="https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441</a>.</p>

Shisho_Sama: <p>Seems like, the pip packages on linux use the <code>GLIBCXX_USE_CXX11_ABI=0</code> some say the conda ones use <code>GLIBCXX_USE_CXX11_ABI=1 . Under windows its always </code>GLIBCXX_USE_CXX11_ABI=0` for cpu and cuda builds it seems. ( I tested the 1.6 version using pip )</p>
<p>This is important for people like me that have a python extension which uses libtorch .<br>
This is still an issue in 1.6cpu! that I tested</p>
<p><strong>Update</strong>, it seems the conda package on Linux is also built using <code>GLIBCXX_USE_CXX11_ABI=0</code>  so does the package on windows:</p>
<pre><code class="lang-auto">In [2]: torch._C._GLIBCXX_USE_CXX11_ABI                                                                                                                              
Out[2]: False
</code></pre>
<p>so basically all Pytorch packages are shipped with <code>GLIBCXX_USE_CXX11_ABI=0</code> but why?</p>
Faster RCNN C++

Jeremy_Tavrisov: <p>Looking to see if anyone has succesfully deployed a Torchvision Faster RCNN (or Mask RCNN) model to C++ via torchscript/libtorch.</p>
<p>In python</p>
<pre><code class="lang-auto">model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
script_model = torch.jit.script(model)
script_model.save("model.pt")
</code></pre>
<p>In C++</p>
<pre><code class="lang-auto">module = torch::jit::load("model.pt");
</code></pre>
<p>I linked to library with QMake in QT Creator with this .pro in Windows 10</p>
<pre><code class="lang-auto">QT -= gui

CONFIG += c++14 console no_keywords
CONFIG -= app_bundle

QMAKE_CXXFLAGS += -D_GLIBCXX_USE_CXX11_ABI=0
QMAKE_LFLAGS += -INCLUDE:?warp_size@cuda@at@@YAHXZ

DEFINES += QT_DEPRECATED_WARNINGS

SOURCES += \
        main.cpp

LIBS += -LD:\src\PyTorch_Playground\c++\testLibTorch\libtorch-win-shared-with-deps-debug-latest\libtorch\lib \
-lc10 -lc10_cuda -ltorch_cuda -ltorch_cpu -ltorch

INCLUDEPATH += D:\src\PyTorch_Playground\c++\testLibTorch\libtorch-win-shared-with-deps-latest\libtorch\include
INCLUDEPATH += D:\src\PyTorch_Playground\c++\testLibTorch\libtorch-win-shared-with-deps-latest\libtorch\include\torch\csrc\api\include

</code></pre>
<p>This works when the model loaded is a ResNet model as well as FCN-ResNet. Both run and give correct outputs but RCNN models throw an exception</p>
<p><code>schemas.size() &gt; 0 INTERNAL ASSERT FAILED at "..\..\torch\csrc\jit\frontend\schema_matching.cpp":491, please report a bug to PyTorch.</code></p>
<p>Torchscript RCNN model has no problem running in python.</p>
<p>There is a few month old github issue (of which I added my comments) <a href="https://github.com/pytorch/pytorch/issues/35881" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/35881</a></p>
<p>but since this is seriously blocking my transition from tensorflow to pytorch I’m hoping crossposting it here will attract more eyes. Also I’m not sure if it’s a bug or a mistake on my part or I’m not linking correctly. Also wondering if people have ever had this working perhaps on previous versions. I’m using nightly builds because I was playing with AMP and autocast.</p>

Dipam_Vasani: <p>I am trying the same thing. Were you able to do it?</p>

Jeremy_Tavrisov: <p>Nope. I have made no progress on this</p>

DSD: <p>Fairly new to PyTorch and for me loading faster-rcnn model is failing as well. Tried following the directions here: <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a></p>
<p>An exception is thrown here:</p>
<p>module = torch::jit::load(“Path_to_model”);</p>

Jeremy_Tavrisov: <p>Has anyone figured this out?</p>
How the ops are registerd to ATenDispatch's op tables_

dongfangduoshou123: <p>I want to know how ops are registered to ATenDispatch’s op_tables_;<br>
I see ATenDispatch has registerOp/registerVariableOp interface, but not found code that these interfaces are called to register ops;<br>
Is ATenDispatch use the global static varable to trigger the registration like C10_DECLARE_REGISTRY mechanism？</p>
<p>I see the code below, but cannot find where “aten::linear(Tensor input, Tensor weight, Tensor? bias=None) -&gt; Tensor” is registered to ATenDispatch.</p>
<p><code>static inline Tensor linear(const Tensor &amp; input, const Tensor &amp; weight, const Tensor &amp; bias) { static auto table = globalATenDispatch().getOpTable("aten::linear(Tensor input, Tensor weight, Tensor? bias=None) -&gt; Tensor"); return table-&gt;getOp&lt;Tensor (const Tensor &amp;, const Tensor &amp;, const Tensor &amp;)&gt;(at::detail::infer_backend(input), at::detail::infer_is_variable(input))(input, weight, bias); }</code></p>

dongfangduoshou123: <p>After effort.  I see where the registerOp/registerVariableOp are called===&gt;In auto generated code like CPUType. cpp, also use global static varable to register ops.</p>

ckmufeng: <p>Thank you！<br>
Although ”convolution“ is not in CPUType.cpp, but it’s in TypeDefault.cpp use TORCH_LIBRARY to register.<br>
Do you have any idea which DispatchKey it use?</p>
Is it possible to detect mask and keypoints at the same time?

brifuture: <p>Hi, I am learning the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL</a> which is a great tutorial for mask-rcnn.</p>
<p>Now I want to  make keypoint predictions along side the mask predictions， is it possible?</p>
<p>By the way, I have noticed that <code>KeypointRCNN</code> provided by torchvision is inherited from <code>FasterRCNN</code> while <code>MaskRCNN</code> is also inherited from <code>FasterRCNN</code>. So <code>KeypointRCNN</code> ignore the masks variable in targets.</p>

lysuhin: <p>Any updates on this?</p>
Why use avgpool2d and avgpool3d in local_response_norm?

picklerick: <p>The following is the formula for local response normalization (across channels):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74048c9fe408f182e71184ddb7d2ae0e571ee406.gif" alt="lrn_torch" data-base62-sha1="gyl8gV38j4P95hpsdh9IISH5tnU" width="422" height="193"></p>
<p>When I looked at the implementation for the same, I saw an avg pool being done after the input tensor is squared elementwise. But I don’t understand why the average pool of the squared activations is being done when the equation involves only a summation of the squared activations.</p>
<pre><code class="lang-auto">def local_response_norm(input, size, alpha=1e-4, beta=0.75, k=1.):
    # type: (Tensor, int, float, float, float) -&gt; Tensor
    r"""Applies local response normalization over an input signal composed of
    several input planes, where channels occupy the second dimension.
    Applies normalization across channels.
    See :class:`~torch.nn.LocalResponseNorm` for details.
    """
    if not torch.jit.is_scripting():
        if type(input) is not Tensor and has_torch_function((input,)):
            return handle_torch_function(
                local_response_norm, (input,), input, size, alpha=alpha, beta=beta, k=k)
    dim = input.dim()
    if dim &lt; 3:
        raise ValueError('Expected 3D or higher dimensionality \
                         input (got {} dimensions)'.format(dim))
    div = input.mul(input).unsqueeze(1)
    if dim == 3:
        div = pad(div, (0, 0, size // 2, (size - 1) // 2))
        div = avg_pool2d(div, (size, 1), stride=1).squeeze(1)
    else:
        sizes = input.size()
        div = div.view(sizes[0], 1, sizes[1], sizes[2], -1)
        div = pad(div, (0, 0, 0, 0, size // 2, (size - 1) // 2))
        div = avg_pool3d(div, (size, 1, 1), stride=1).squeeze(1)
        div = div.view(sizes)
    div = div.mul(alpha).add(k).pow(beta)
    return input / div
</code></pre>

InnovArul: <p>As per pytorch docs <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/normalization.html#LocalResponseNorm">here</a>, <code>alpha</code> is divided by <code>n</code> (kernel size). So,  use of <code>avg_pool</code> makes sense I guess.</p>

picklerick: <p>I see. Yeah, that makes sense. I guess I was confused because I looked at the code in the context of the equation I gave in the original post. Maybe <code>alpha / n</code> is can be treated as some new single constant alpha_prime (equal to alpha in the equation in my question) and that’s why torch uses that formula? Either way, thanks for resolving my doubt!</p>
DistributedDataParallel Sparse Embeddings

Pkpklong: <p>Hi everyone,</p>
<p>I’m trying to train a matrix factorization model, wrapping the model with DistributedDataParallel. The model contains embeddings, which are multiplied together to obtain a prediction. After I calculate the MSELoss between the target and the prediction, calling backwards gives me the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "src/train_model.py", line 188, in &lt;module&gt;
    join=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception:

-- Process 6 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/ubuntu/NOMAR/src/train_model.py", line 56, in run_training
    trainer.run(mf_model, train_data_dir, TEST_DATA_DIR, dataset_kwargs, early_stopping_kwargs)
  File "/home/ubuntu/NOMAR/src/trainer/MFTrainer.py", line 73, in run
    avg_training_loss = self.train_model(model, train_loader, optimizer)
  File "/home/ubuntu/NOMAR/src/trainer/MFTrainer.py", line 95, in train_model
    loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Tensors must be CUDA and dense
</code></pre>
<p>I’m suspecting that the sparse gradients cannot be joined, but I read from <a href="https://github.com/pytorch/pytorch/issues/17356" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17356</a>, that it has been fixed?</p>
<p>Any insights would be appreciated.</p>

sangyx: <p>Hi, I have the same problem, but I have found the solution. The pytorch have supported <code>backend="gloo"</code> while the support for <code>nccl</code> has not been completed. I have tested it in pytorch nightly. If the stable version still go wrong, you can update to it.</p>
Optimizer = torch.optim.SGD()

111296: <p>I use this line “optimizer = torch.optim.SGD(model.parameters(), args.lr, momentum=args.momentum, weight_decay=args.weight_decay)” to do L2 regularization to prevent overfitting. Generally, regularization only penalizes the weight W parameter of the model, and the bias parameter b does not penalize, but there is a network saying that the weight decay specified by the optimizer weight_decay parameter of torch.optim is for all parameters in the network , Including the weight w and bias b for simultaneous punishment. Is that right?</p>
<p>Reference URL: <a href="https://blog.csdn.net/guyuealian/article/details/88426648" rel="nofollow noopener">https://blog.csdn.net/guyuealian/article/details/88426648</a></p>

albanD: <p>Hi,</p>
<p>Yes this is right, the weight decay parameter here will be applied to all the parameters.</p>

KFrank: <p>Hello 乃仁 梁!</p>
<aside class="quote no-group" data-username="111296" data-post="1" data-topic="97099" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111296/48/28962_2.png" class="avatar"> 111296:</div>
<blockquote>
<p>there is a network saying that the weight decay specified by the optimizer weight_decay parameter of torch.optim is for all parameters in the network</p>
</blockquote>
</aside>
<p>If you wish to turn off weight decay for your network biases, you may<br>
use “parameter groups” to use different optimizer hyperparameters to<br>
optimize different sets of network parameters.</p>
<p>Here’s a brief example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
lin = torch.nn.Linear (3, 1)
opt = torch.optim.SGD ([
    {'params': lin.weight, 'lr': 0.1, 'weight_decay': 0.5},
    {'params': lin.bias,   'lr': 0.1, 'weight_decay': 0.0}
])
</code></pre>
<p><code>opt</code> will use a learning rate of <code>0.1</code> for all of <code>lin</code>’s parameters – both<br>
<code>weight</code> and <code>bias</code> – but will only use a weight decay of <code>0.5</code> for <code>weight</code><br>
and no weight decay (<code>weight_decay = 0.0</code>) for <code>bias</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

111296: <p>I use this URL (<a href="https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-collect.md" rel="nofollow noopener">https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-collect.md</a>) to go to the training step. How can I change the weight decay of train.py (<a href="https://github.com/dusty-nv/pytorch-classification/blob/5107de352cade326e7aacd44bf8d625b7055fd4e/train.py" rel="nofollow noopener">https://github.com/dusty-nv/pytorch-classification/blob/5107de352cade326e7aacd44bf8d625b7055fd4e/train.py</a>) without affecting the bias value?</p>
Network’s parameter

111296: <p>I use this URL (<a href="https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-collect.md" rel="nofollow noopener">https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-collect.md</a>) to go to the training step. I corrected the dropout network parameters of googlenet-1378be20.pth. When I got to the step of converting onnx, I found that the .pth network parameters were still the same. How can I solve this problem?</p>
Custom Image dataset working only for validation data

IamSparky: <p>I have created a custom dataset class for my training and validation dataset . Among which the torch transform with albumentations library is working only for validation data and <strong>not for my training data</strong>   .It is giving back the below error .<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3df7b48530b8650f9522478774615b5d2511b52.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3df7b48530b8650f9522478774615b5d2511b52" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3df7b48530b8650f9522478774615b5d2511b52_2_674x500.png" alt="image" data-base62-sha1="uejyiYTDJ0WSiD6ilpKTEGVbZES" width="674" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3df7b48530b8650f9522478774615b5d2511b52_2_674x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3df7b48530b8650f9522478774615b5d2511b52.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3df7b48530b8650f9522478774615b5d2511b52.png 2x" data-dominant-color="282C31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">784×581 63.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s the link to my <a href="https://www.kaggle.com/soumochatterjee/openvaccine" rel="noopener nofollow ugc">notebook</a> where I am getting  the error .<br>
Please help me in suggesting the syntax , so that I can correct the error present .</p>

user_123454321: <p>For training you are using a transform ‘Cutout’ from torchtoolbox library which I believe does not have ‘always_apply’, why don’t you use ‘Cutout’ from albumentations itself?</p>

IamSparky: <p>thanks …but I already resolved the issue</p>
Three dots slice question

pytorch_lzwhard: <pre><code class="lang-auto">a.shape = [1, 3, 80, 64, 2]
b.shape = [1, 3, 80, 64, 85]
b[..., 0:2] = a
</code></pre>
<p>there is always wrong with … in my environment, so is there some way to replace …? tkx!</p>

ptrblck: <p>What kind of error are you getting when using the ellipsis?<br>
Instead of using the three dots you could also replace it with the <code>Ellipsis</code> keyword, but I guess this would yield the same error you are seeing.</p>
<p>Note that your code snippet works fine for me.</p>

KaiHoo: <p>LOL</p>
<pre><code class="lang-auto">b = torch.cat([a, b[:,:,:,:,2:]], -1)
</code></pre>
Trainer.test() in pytorch ligtning won't work

netaglazer: <p>According to the explanation in PYTORCH LIGHTNING DOCUMENTATION, to test the model with a new dataset I should do this:</p>
<p>test = DataLoader(…)<br>
trainer.test(test_dataloaders=test)</p>
<p>but then I get that error:</p>
<p>test() got an unexpected keyword argument ‘test_dataloader’</p>

ptrblck: <p>Are you using <code>test_dataloader</code> (expected) or <code>test_dataloaders</code>?</p>
Is there any resource for conv2dtranpose for videos?

Oguzhan: <p>I am trying to implement Videogan and trying to figure out how to get upsampling for 3 dimension.<br>
self.conv1b = nn.ConvTranspose2d(zdim, 512, [4,4], [1,1]) I found this code but I am not sure what is it doing. are [4,4] the kernel size and [1,1] padding ?</p>

ptrblck: <p>No, the <code>4s</code> are the kernel size and the the <code>1s</code> are the stride.<br>
You can check the <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html#torch.nn.ConvTranspose2d">docs</a> for the argument names.</p>
GPU utilization is under 2% - custom model from scratch

n95: <p>Hello!</p>
<p>I’d really appreciate any help on this.<br>
I’m trying to understand why the GPU utilisation is under 2% with this training code. If possible, could you please tell me what parts of the code need to be changed to effectively utilise more of the GPU? Thank you so much!</p>
<pre><code class="lang-auto">        print("Starting training ..")
        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.lr)
        criterion = torch.nn.CosineEmbeddingLoss(reduction='none')
        # move model to GPU if CUDA is available
        print("Moving computations to cuda.")
        if self.use_cuda:
            model = self.model.cuda()
        # initialize tracker for minimum validation loss
        valid_loss_min = self.valid_loss_min_input
        for epoch in range(self.start_epochs, self.start_epochs + self.n_epochs + 1):
            # initialize variables to monitor training and validation loss
            train_loss = 0.0
            valid_loss = 0.0
            print("Starting epoch:", epoch)
            # train the model #
            model.train()
            start_time = time.time()
            start_batch_time = time.time()
            for batch_idx, (scalar_1, scalar_2, label, user_vec_raw, prod_vec_raw) in enumerate(self.data_loaders['train']):
                """
                if batch_idx % 1000 == 0:
                    print(f'train epoch {epoch}, batch {batch_idx}')
                    print("Time taken", time.time()-start_batch_time)
                    start_batch_time = time.time()
                """
                # move to GPU
                if self.use_cuda:
                    label_cuda, user_vec_cuda, prod_vec_cuda = label.cuda(), user_vec_raw.cuda(), prod_vec_raw.cuda()
                ## find the loss and update the model parameters accordingly
                # clear the gradients of all optimized variables
                optimizer.zero_grad()
                # forward pass: compute predicted outputs by passing inputs to the model
                output_user_emb, output_prod_emb = model(user_vec_cuda, prod_vec_cuda)
                # calculate the batch loss
                loss = criterion(output_user_emb, output_prod_emb, label_cuda.float())
                # backward pass: compute gradient of the loss with respect to model parameters
                loss.sum().backward()
                # perform a single optimization step (parameter update)
                optimizer.step()
                ## record the average training loss, using something like
                ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))
                train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data.sum() - train_loss))

            # validate the model #

            model.eval()

</code></pre>

ptrblck: <p>You could remove all data loading code, create random tensors on the GPU, and profile the training separately. This would give you an idea, if the data loading and processing might be the bottleneck or any other part of your training.</p>
PngTorch++: A CMake based integration of the libpng / png++ libraries with Libtorch C++

dambo: <p>Hello,<br>
I wrote a small library for C++ users who want to avoid using OpenCV.<br>
The integration allows one to easily read images and convert them into PyTorch C++ front-end tensors and vice versa.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92a22364b9e12a09ed8341da0eb7a0c12b769ae0.png" data-download-href="https://discuss.pytorch.org/uploads/default/92a22364b9e12a09ed8341da0eb7a0c12b769ae0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92a22364b9e12a09ed8341da0eb7a0c12b769ae0_2_690x129.png" alt="image" data-base62-sha1="kVb5B68Koq7VXgPEAmua0QlEc0w" width="690" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92a22364b9e12a09ed8341da0eb7a0c12b769ae0_2_690x129.png, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92a22364b9e12a09ed8341da0eb7a0c12b769ae0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92a22364b9e12a09ed8341da0eb7a0c12b769ae0.png 2x" data-dominant-color="C6D1D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">800×150 33.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a href="https://github.com/QuantScientist/PngTorch" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/QuantScientist/PngTorch</a></p>
<p>Hope it is useful to others,<br>
Best,</p>

ptrblck: <p>Thanks for sharing it! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Does the current solution allow to read the images only or could you transform them as well (using another lib)?</p>

dambo: <p>Both reading and writing PNG’s, finished testing it fully yesterday. There was a problem with saving tensors that returned from models. Now it’s done, works for RGB and RGBA.</p>
<p><a href="https://github.com/QuantScientist/PngTorch/blob/master/src/example001.cpp" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/QuantScientist/PngTorch/blob/master/src/example001.cpp</a></p>
<p><a href="https://github.com/QuantScientist/PngTorch/blob/master/src/example002.cpp" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/QuantScientist/PngTorch/blob/master/src/example002.cpp</a></p>
<p>Using in in my other project:(<a href="https://github.com/QuantScientist/Siv3DTorch" rel="noopener nofollow ugc">https://github.com/QuantScientist/Siv3DTorch</a>)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb927aad7c7f3ba70ab060e73cb734bd974f9292.gif" alt="inference007" data-base62-sha1="zTvFS0VRZDiDQsPrmJKfsJOmqoa" width="300" height="240" class="animated"></p>
<p>Best,</p>
Strange behavior of torch.argsort

Naruto-Sasuke: <pre><code class="lang-python"> import torch

 a = torch.rand(3, 4)

 print(a)

 b = torch.argsort(a)

 print(b)


</code></pre>
<p>However, I get</p>
<pre><code class="lang-bash">tensor([[0.4143, 0.4293, 0.9506, 0.2267],
        [0.8637, 0.8718, 0.2595, 0.2888],
        [0.6663, 0.6066, 0.8233, 0.6793]])
tensor([[3, 0, 1, 2],
        [2, 3, 0, 1],
        [1, 0, 3, 2]])

</code></pre>
<p>For the first line, why the smallest value <code>0.2267</code> accords with <code>2</code> in b, which seems strange.<br>
My pytorch version is 1.5.</p>

user_123454321: <p>This seems the right answer…<br>
The first row is 3,0,1,2 since 0.2267 is smallest (which is at index 3), then 0.4143 (which is at index 0), then 0.4293, then 0.9506.<br>
Similarly for second and third row.<br>
Note argsort is done by default along the last dimension.</p>

Naruto-Sasuke: <p>Quite a silly question… Thank you for your explanation.</p>
Is there a way to prefix in load_state_dict

whatdhack: <p>Created a class contains a number of nn.Modules.  Looks like doing so adds a prefix to state_dict keys. Is there a way to specify a prefix during load_state_dict () ? Or what is the recommended way to hande such situation ?</p>

ptrblck: <p>You could create a new <code>dict</code> by assigning the <code>state_dict</code> keys and values to the new keys with the desired prefix.</p>
Depthwise Separable Convolution in UNet

AI_Audio: <p>Hi,</p>
<p>I apply DSC in my UNet architecture, like in this paper about <a href="https://www.researchgate.net/publication/339348440_SD-UNet_Stripping_Down_U-Net_for_Segmentation_of_Biomedical_Images_on_Platforms_with_Low_Computational_Budgets" rel="noopener nofollow ugc">SD-UNet</a>.<br>
While it works well during encoding, the decoding process throws the following error</p>
<blockquote>
<p>out_channels must be divisible by groups</p>
</blockquote>
<p>This occurs since in DSC (as far as I know) the number of groups is equal to the number of input channels. However, the latter is inherently larger than the output channels during the upsampling process.</p>
<p>I attach the code snippet of the unet model and parts. What should be done to overcome this situation?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/313e8488cddeb7781f611bba19e43ee702533f89.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/313e8488cddeb7781f611bba19e43ee702533f89" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/313e8488cddeb7781f611bba19e43ee702533f89_2_690x414.jpeg" alt="image" data-base62-sha1="71Dl3y6AFMrRFfZyC8w5xq2BFuF" width="690" height="414" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/313e8488cddeb7781f611bba19e43ee702533f89_2_690x414.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/313e8488cddeb7781f611bba19e43ee702533f89_2_1035x621.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/1/313e8488cddeb7781f611bba19e43ee702533f89.jpeg 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1152×692 231 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>As the error message claims apparently the number of <code>out_channels</code> is not divisible by <code>groups</code>.<br>
Looking at the code I guess that the first conv layer in <code>DoubleConv</code> might create the issue, if <code>mid_channels</code> is not divisible by <code>in_channels</code>. Could you check it and adapt the number of channels accordingly?</p>
DataLoader with num_workers > 0 raising error with mongodb

Jenny_Smolensky: <p>Hello everyone,<br>
I have the following error:</p>
<pre><code class="lang-auto">    for batch_idx, (data, target, idx, kws_target) in enumerate(train_loader):
  File "..\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 291, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "..\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 737, in __init__
    w.start()
  File "..multiprocessing\process.py", line 112, in start 
    self._popen = self._Popen(self)
  File "..\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "..\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "..\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "..\Python\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
</code></pre>
<p>when I’m iterating over dataset.<br>
In the <strong>getitem</strong> function I’m using mongoDB client “find” function .</p>
<p>When I initialize the data loader with num_workers=0, everything runs just fine.</p>
<p>What causes this error? And how can I solve it?<br>
Thank’s!</p>

ptrblck: <p>This seems to be a multiprocessing issue. Could you try to use the <code>spawn</code> method and rerun your code?</p>
Max-pooling with complex masks in PyTorch

klein: <p>Dear all,</p>
<p>Suppose I have a matrix <code>src</code> with shape <code>(5, 3)</code> and a boolean matrix <code>adj</code> with shape  <code>(5, 5)</code> as follow,</p>
<pre><code class="lang-auto">src = tensor(
       [[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11],
        [12, 13, 14]]
)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">adj = tensor(
       [[1, 0, 1, 1, 0],
        [0, 1, 1, 1, 0],
        [1, 1, 0, 1, 1],
        [1, 1, 1, 0, 0],
        [0, 0, 1, 0, 1]])
</code></pre>
<p>we can take each row in <code>src</code> as one node embedding, and regard each row in <code>adj</code> as the indicator of which nodes are the neighborhood.</p>
<p>My goal is to operate a max-pooling among all neighborhood node embeddings for each node in <code>src</code>.<br>
For example, as the neighborhood nodes (including itself) for the 0-th node is <code>0, 2, 3</code>, thus we compute a max-pooling on <code>[0, 1, 2]</code>, <code>[6, 7,8]</code>, <code>[ 9, 10, 11]</code> and lead an updated embedding <code>[ 9, 10, 11]</code> to update 0-th node in <code>src_update</code>.</p>
<p>A simple solution I wrote is</p>
<pre><code class="lang-auto">src_update = torch.zeros_like(src)
for index in range(bool_adj.size(0)):
    list_of_non_zero = bool_adj[index].nonzero().view(-1)
    mat_non_zero = torch.index_select(src, 0, list_of_non_zero)
    output[index] = torch.sum(mat_non_zero, dim=0)
print (output)
</code></pre>
<p>with <code>src_update</code> is outputted as</p>
<pre><code class="lang-auto">src_update = tensor([[ 9, 10, 11],
        [ 9, 10, 11],
        [12, 13, 14],
        [ 6,  7,  8],
        [12, 13, 14]])
</code></pre>
<p>Although it works, but it runs very slowly and looks not elegant!<br>
Any suggestions to improve it for <strong>better efficiency</strong>?</p>
<p>In addition, if both <code>src</code> and <code>adj</code> are appended with <strong>batches</strong> (<code>(batch, 5, 3)</code>, <code>(batch, 5, 5)</code>), how to make it works?</p>
<p>Thanks a lot!!!</p>
Bert multilingual training

Sumanta_Bhattacharyy: <p>I was trying to finetune a multilingual_bert_cased model using my custom dataset but for certain datapoints the embedding structure mismatch leads to error. As per the multilingual config.json max_position embedding is 512 so anytime the token type embedding and input embedding is greater than 512 it is not able to get added. Is there any solution to include those data points in training or BERT has this issue with sentences that have bigger embeddings than the data sample it was pretrained?</p>
<p>following is the error in terminal</p>
<p>torch.Size([2, 97, 768])<br>
torch.Size([1, 97, 768])<br>
torch.Size([2, 97, 768])<br>
torch.Size([2, 617, 768])<br>
torch.Size([1, 512, 768])<br>
torch.Size([2, 617, 768])<br>
Traceback (most recent call last):<br>
File “/users/sbhatta9/conditionedbert/bert_using_trainer.py”, line 115, in <br>
train(train_dataset)<br>
File “/users/sbhatta9/conditionedbert/bert_using_trainer.py”, line 105, in train<br>
trainer.train()<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/transformers/trainer.py”, line 707, in train<br>
tr_loss += self.training_step(model, inputs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/transformers/trainer.py”, line 994, in training_step<br>
outputs = model(**inputs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 1150, in forward<br>
return_dict=return_dict,<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 828, in forward<br>
input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 722, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/users/sbhatta9/sumanta/lib/python3.7/site-packages/transformers/modeling_bert.py”, line 214, in forward<br>
embeddings = inputs_embeds + position_embeddings + token_type_embeddings<br>
RuntimeError: The size of tensor a (617) must match the size of tensor b (512) at non-singleton dimension 1</p>