akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Confused about repeat. How to repeat columns?

unbornchikken: <p>Hi,</p>
<p>I have an 1D tensor, for example:</p>
<pre><code class="lang-auto">t = torch.rand(4)
</code></pre>
<p>What I want to do is simply repeat it by the second dimensions for a few times. Which means:</p>
<pre><code class="lang-auto">t:

0.1
0.3
0.2
0.9

tr:

0.1  0.1  0.1
0.3  0.3  0.3
0.2  0.2  0.2 
0.9  0.9  0.9
</code></pre>
<p>The obvious choice would be repeat. So I have a one dimensional tensor which is basically a column. I want to repeat that column three times on the second dimension (and one times on the first dimension), so I write:</p>
<pre><code class="lang-auto">tr = t.repeat(1, 3)
</code></pre>
<p>I get a tensor of size 1x9.</p>
<p>I simply don’t get the logic behind this. For achieve that I want I have to write:</p>
<pre><code class="lang-auto">tr = t.repeat(3, 1).t()
</code></pre>
<p>It looks like first dimension is somehow transposed to the second when doing repeat, which is kinda confusing. Is it the intended behavior?</p>

Pete_Florence: <p>Hi <a class="mention" href="/u/unbornchikken">@unbornchikken</a>,</p>
<p>Here’s what you’re looking for:</p>
<pre><code class="lang-python">t = torch.rand(4,1)
print t
tr = t.repeat(1, 3)
print tr
</code></pre>
<p>You need the original tensor to be an Nx1 tensor (a 2D tensor, with one of those dimensions just with 1 element) rather than a N-element vector.</p>
<p>If you have a vector to start with, just unsqueeze it:</p>
<pre><code class="lang-python">t = torch.rand(4).unsqueeze(1)
print t
tr = t.repeat(1, 3)
print tr
</code></pre>

aaronlelevier: <p><a class="mention" href="/u/pete_florence">@Pete_Florence</a> thanks. I was looking for a top level function like <code>numpy.repeat</code> and trying <code>torch.repeat</code> but it doesn’t exist</p>

ptrblck: <p><code>repeat</code> is a <code>torch.tensor</code> method, so you should call it using a tensor:</p>
<pre><code class="lang-python">x = torch.randn(1, 1)
x.repeat(2, 1)
</code></pre>

arogozhnikov: <p>I think <a href="https://github.com/arogozhnikov/einops" rel="nofollow noopener">einops</a> notation is less confusing. You can directly repeat along new axis</p>
<pre><code class="lang-auto">einops.repeat(x, 'i -&gt; i newaxis', newaxis=3) # make 3 identical columns
einops.repeat(x, 'i -&gt; newaxis i', newaxis=3) # make 3 identical rows
</code></pre>
Hidden/cell state initialisation with Variable or without Variable?

Talita: <p>Hi,</p>
<p>I am just wondering about how we should initialise the hidden/cell state of the LSTM if we want something different than just zeros. I usually do it like this (just using the example in the docs):</p>
<pre><code class="lang-auto">&gt;&gt;&gt; rnn = nn.LSTM(10, 20, 2)
&gt;&gt;&gt; input = torch.randn(5, 3, 10)
&gt;&gt;&gt; h0 = torch.randn(2, 3, 20)
&gt;&gt;&gt; c0 = torch.randn(2, 3, 20)
&gt;&gt;&gt; output, (hn, cn) = rnn(input, (h0, c0))
</code></pre>
<p>But I have also seen examples that use <code>Variable</code> or even the example mentioned here: <a href="https://discuss.pytorch.org/t/correct-way-to-declare-hidden-and-cell-states-of-lstm/15745" class="inline-onebox">Correct way to declare hidden and cell states of LSTM</a>.</p>
<p>I suppose that Variable allows the model to “learn” a correct initialisation of the hidden state. Is that correct???</p>
Need help with numpy to PyTorch conversion, with np.tile

ivan-bilan: <p>I need to translate the following numpy code to PyTorch, in particular the np.tile part (ideally, PyTorch 0.4.0 compatible):</p>
<pre><code class="lang-auto">
# k.size() = q.size()
batch_size, sent_len, vec_size = q.size()  # i.e. [150, 96, 120], also k is same
q_ = np.repeat(q, sent_len, axis=1)
k_ = np.tile(k, (sent_len, 1))
concat_vecs = np.stack([q_, k_], axis=2).reshape(batch_size, sent_len, sent_len, -1)
</code></pre>
<p>Any pointer appreciated. Thanks.</p>

ptrblck: <p>This should yield the same result:</p>
<pre><code class="lang-python">q = np.random.randn(150, 96, 120)
k = np.random.randn(150, 96, 120)
batch_size, sent_len, vec_size = q.shape  # i.e. [150, 96, 120], also k is same
q_ = np.repeat(q, sent_len, axis=1)
k_ = np.tile(k, (sent_len, 1))
concat_vecs = np.stack([q_, k_], axis=2).reshape(batch_size, sent_len, sent_len, -1)


a = torch.from_numpy(q)
b = torch.from_numpy(k)
bs, s, v = a.size()
a_ = a.repeat(1, 1, s).view(bs, s*s, v)
b_ = b.repeat(1, s, 1)
concat_vec = torch.stack((a_, b_), 2).view(bs, s, s, -1)

(concat_vec == torch.from_numpy(concat_vecs)).all()
</code></pre>

arogozhnikov: <p>Let me show you the same code using einops:</p>
<pre><code class="lang-auto">from einops import repeat, rearrange

q_ = repeat(q, 'b sent channel -&gt; b (sent sent2) channel', sent2=sent_len)
k_ = repeat(q, 'b sent channel -&gt; b (sent2 sent) channel', sent2=sent_len)
concat_vecs = rearrange([q_, k_], 'qk b sentbysent channel -&gt; b sentbysent (qk channel)')
</code></pre>
How to understand "at::"?

tgsmdww: <p>Here if input.is_mkldnn(), call at::mkldnn_adaptive_avg_pool2d and others call _adaptive_avg_pool2d. Why mkldnn need add <strong>at::</strong>? What’s the difference between those.</p>
<pre><code class="lang-auto">  Tensor adaptive_avg_pool2d(at::Tensor const&amp; input, IntArrayRef output_size) {
    if (input.is_mkldnn()) {
      return at::mkldnn_adaptive_avg_pool2d(input, output_size);
    }

    // TODO: fastpath for Channels_last should be explored later;
    if (input.suggest_memory_format() == at::MemoryFormat::Contiguous &amp;&amp; !input.is_quantized() &amp;&amp; output_size[0] == 1 &amp;&amp; output_size[1] == 1) {
      // in this case, adaptive pooling is just computing mean over hw
      // dimensions, which can be done more efficiently
      int64_t mean_size = input.size(-1) * input.size(-2);
      Tensor out = input.contiguous().view({-1, mean_size}).mean(-1);
      return input.dim() == 3 ? out.view({input.size(0), 1, 1})
                              : out.view({input.size(0), input.size(1), 1, 1});
    } else {
      return _adaptive_avg_pool2d(input, output_size);
    }
  }
</code></pre>

ptrblck: <p><code>at::</code> is a namespace in the libtorch.<br>
I think its usage might not even be needed in this particular line of code.</p>
<p>The functions will dispatch to the mentioned functions <a href="https://github.com/pytorch/pytorch/blob/0829cadca34ba7b5276c2e2f410476f0827bab2f/aten/src/ATen/native/native_functions.yaml#L5699-L5710">here</a> in case you would like to follow the calls.</p>

tgsmdww: <p>when I create this function</p>
<pre><code class="lang-auto">namespace at{ namespace native{
……
Tensor conv2d_my(

    const Tensor&amp; input,

    const Tensor&amp; weight,

    const Tensor&amp; bias,

    IntArrayRef stride,

    IntArrayRef padding,

    IntArrayRef dilation,

    int64_t groups)

{

    return convolution(input, weight, bias, stride, padding, dilation,

        false, {{0, 0}}, groups);

}))
</code></pre>
<p>When I return convolution,  it will report</p>
<pre><code class="lang-auto">error: call of overloaded ‘convolution(const at::Tensor&amp;, const at::Tensor&amp;, const at::Tensor&amp;, c10::IntArrayRef&amp;, c10::IntArrayRef&amp;, c10::IntArrayRef&amp;, bool, &lt;brace-enclosed initializer list&gt;, int64_t&amp;)’ is ambiguous
</code></pre>
<p>If I add “at::”, it will pass.</p>

ptrblck: <p>Then you would need to specify the namespace in your code snippet. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I don’t know, if <code>mkldnn_adaptive_avg_pool2d</code> is ambiguous.</p>

tgsmdww: <p>They are all in “at::native::”</p>
Getting model output as nan

chetan06: <p>I  am working on Melanoma Classification task where I have to classify the patients into two categories on the basis of their skin images. I have used efficientnetb3 model (pretrained) with minor transformations. But I am getting nan as the model output while training. What could be the possible reasons?</p>
<pre><code class="lang-auto">class MelanomaDataset(Dataset):

    def __init__(self, dataframe, image_dir, transforms = None):
        self.dataframe = dataframe
        self.image_dir = image_dir
        self.transforms = transforms

    def __len__(self):
        return self.dataframe.shape[0]

    def __getitem__(self, idx):
        img_name = '{}.png'.format(self.dataframe.iloc[idx, 0])
        fullname = self.image_dir + img_name
        image = cv2.imread(fullname)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        num = self.dataframe.loc[:,'target'].iloc[idx]
        label = np.asarray(int_to_label[num])

        if self.transforms:
            image = self.transforms(image = image)['image']

        return image, label
</code></pre>

ptrblck: <p>Could you post the model definition as well as the training routine?<br>
If you suspect that the data loading might be faulty, you could add <code>torch.is_finite(data)</code> inside the training loop to check that no input contains NaNs or Infs.</p>

chetan06: <p>I have used pretrained model from timm and have used Stratified K-fold. The input are images on which I have applied augmentations, Horizontal Flip, Vertical Flip, Centre Crop and Normalization</p>
Num_workers and batch size

JunzheJosephZhu: <p>it looks like when num_workers&gt;batch_size, the same data will be fed to the neural net. I was using num_worker = 8 and batch_size = 1, and the same example is fed as input for multiple times. I think it’s a bug or something.</p>

ptrblck: <p>Could you post a reproducible code snippet to show this behavior, as this simple example works without repeating values:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.arange(100))
loader = DataLoader(dataset, batch_size=1, num_workers=8)

for data in loader:
    print(data)
</code></pre>
Data in GPU selected by a dictionary (in CPU)

Emerald01: <p>Hello,</p>
<p>The following example seems very simple but when I looked at it I cannot figure out what the data workflow would look like. Since I am new to Pytorch and I am careful about efficiency, I like to understand it in a good way so I feel confident to proceed. Thank you in advance.</p>
<p>For example, if you have a tensor already in GPU</p>
<pre><code class="lang-auto">X = torch.from_numpy(np.array([1,2,3,4,5]))
Xg = X.to(device)
</code></pre>
<p>After this I was confirmed that Xg is indeed in GPU as <code>device='cuda:0'</code></p>
<p>Now I have a use case: I have a Pythonic dictionary like</p>
<pre><code class="lang-auto">index_map = {"a": np.array([0,1]), "b":np.array([2,3,4])}
</code></pre>
<p>and I use this dictionary to select corresponding values inside Xg, like</p>
<pre><code class="lang-auto">Xg[index_map["a"]]
</code></pre>
<p>and I get the result</p>
<pre><code class="lang-auto">tensor([1, 2], device='cuda:0')
</code></pre>
<p>It looks obvious, but the question of me is that since index_map is a python dictionary, and from a few other topics and experiment, a dictionary cannot be directly moved to GPU as a whole. So I am wondering when this last operation is taken, since Xg is in GPU but index_map is not, how does this work? Does the workflow like (1) evaluate index_map[“a”] in CPU and get the corresponding index (2) pass those indices to GPU and operate on Xg ?  Or, index_map[“a”] is somehow silently copied into the GPU beforehand, and all stuffs are inside GPU?</p>

ptrblck: <p>The indices should be copied to the device before the actual indexing kernel is called, if I’m not mistaken.</p>
Partial reverse operation

shy: <p>Hi, I’m trying to implement the following operation:<br>
assume i have a tensor:<br>
[[1,2,3,4,0],<br>
[1,2,3,0,0],<br>
[1,2,0,0,0]]<br>
I want to transform it to:<br>
[[4,3,2,1,0],<br>
[3,2,1,0,0],<br>
[2,1,0,0,0]]</p>
<p>Is there any efficient way to do this?</p>

ptrblck: <p>You could simply sort the values:</p>
<pre><code class="lang-python">x = torch.tensor([[1,2,3,4,0],
                  [1,2,3,0,0],
                  [1,2,0,0,0]])

print(x.sort(dim=1, descending=True).values)
&gt; tensor([[4, 3, 2, 1, 0],
        [3, 2, 1, 0, 0],
        [2, 1, 0, 0, 0]])
</code></pre>
How to decide which architecture to use resnet or vgg or google net

Sravani_Lekkala: <p>for classification of images how to decide which architecture would be better resnet or vgg or alexnet</p>

zerone: <p>If you are talking about transfer learning i.e. using pre-trained CNN architectures for image classification then this might help: <a href="https://cs231n.github.io/transfer-learning/" rel="nofollow noopener">https://cs231n.github.io/transfer-learning/</a> .</p>

AladdinPerzon: <p>ResNet would most likely perform best and is the one I’d choose out of those alternatives</p>

Sravani_Lekkala: <p>okay it is the best but what is the need to use that every time.if some simple network can give the same accuracy in less time.</p>

AladdinPerzon: <p>Well try it and see but there’s no way to know apriori and ResNet would most likely (like everytime), outperform especially compared to something like AlexNet. There’s no need to use it everytime, but people want good performance and that’s why they use those architectures. You can try it out and make your own opinion.</p>
How to get mask from proposals and classification scores fast

Absurd: <p>Supposes that the batch size is B, each image has N rectangular proposals, thus exits a tensor proposals of  <code>shape = [B, N, 4]</code>, with the format of xyxy. Each proposal has scores for C classes, corresponding to a tensor scores of <code>shape = [B, N, C]</code>. I want to get a mask of <code>shape = [B, C, H, W]</code> from proposals and shapes. And each pixel value of mask is the accumulation of all proposals that cover this positio:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/2/7242ed5b9969a2a1abc91db54e5a5204336071f2.png" alt="Screenshot from 2020-06-04 14-00-02" data-base62-sha1="giNOZv2Tg9s8Xx4psqVDsjz3eam" width="444" height="81"><br>
s is the mask, and D is proposal, k means class k.<br>
The only way I can figured out is using loop as below:</p>
<pre><code class="lang-auto">scores = torch.randint(10, [2, 2, 20]) / 10.0
batch_size, proposal_number, num_class = scores.shape
height, width = 50, 50

proposals = torch.LongTensor([[[0, 0, 20, 20], [10, 12, 28, 40]], [[5, 9, 20, 40], [6, 20, 41, 38]]])  # [B, N, 4]

masks = []
for i in range(batch_size):
    mask = torch.zeros(num_class, height, width)
    for j in range(proposal_number):
        proposal = proposals[i, j].to(torch.int)
        mask[:, proposal[1]: proposal[3], proposal[0]: proposal[2]] += scores[i, j, :].view(-1, 1, 1)
    masks.append(mask)
masks = torch.stack(masks, dim=0)
</code></pre>
<p>But when N is large, this method is very slow. So I wonder if there exits faster way to do this?</p>
Model training with different class size

monster: <p><strong>How to train deep learning model, if one class has large data compared to other classes?</strong></p>

Nikronic: <p>Hi,</p>
<p>There are many ways to deal with this situation and actually this is an area of research. So, generally, here are few options you can select:</p>
<ul>
<li>Using sampling in the way that in each batch, same size of classes exists.<br>
<a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2" class="inline-onebox">How to handle imbalanced classes</a>
</li>
<li>Weighting loss function in way that biased toward rare class. <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2" class="inline-onebox">How to handle imbalanced classes</a>
</li>
<li>Using ensemble of ANNs using Boosting approaches for generating synthesized data or sampling not biased data such as RBBoost, SMOTEBoost.</li>
</ul>
<p>There are other ways for sure which you need to find which one suits your case the best.</p>
<p>Bests</p>

monster: <p>thanks , I will try that.</p>
TorchScript and pack_sequence

jackeown: <p>Does torchscript not support <code>torch.nn.utils.rnn.pack_sequence</code>?</p>
<p>Here’s a minimal example of what isn’t working on my computer:</p>
<pre><code class="lang-python">import torch


class Test(torch.nn.Module):

	def forward(self):
		# return torch.tensor(1)
		sequences = [torch.randn(torch.randint(1,5,(1,)),3) for i in range(10)]
		packed = torch.nn.utils.rnn.pack_sequence(sequences, enforce_sorted=False)


model = Test()
model()
print("successfully called normal model")

try:
	tsModel = torch.jit.script(model)
	print("Successfully converted model")
except Exception as e:
	print("Failed to convert torchscript model!")
	print(e)
</code></pre>
<p>This code outputs the following for me:</p>
<pre><code class="lang-auto">successfully called normal model
Failed to convert torchscript model!

Tried to access nonexistent attribute or method 'new' of type 'Tensor'.:
  File "/home/jack/.pyenv/versions/3.7.0/lib/python3.7/site-packages/torch/nn/utils/rnn.py", line 365
        out_dims = (max_len, len(sequences)) + trailing_dims

    out_tensor = sequences[0].data.new(*out_dims).fill_(padding_value)
                 ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    for i, tensor in enumerate(sequences):
        length = tensor.size(0)
'pad_sequence' is being compiled since it was called from 'pack_sequence'
  File "/home/jack/.pyenv/versions/3.7.0/lib/python3.7/site-packages/torch/nn/utils/rnn.py", line 409
    """
    lengths = [v.size(0) for v in sequences]
    return pack_padded_sequence(pad_sequence(sequences), lengths, enforce_sorted=enforce_sorted)
                                ~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
'pack_sequence' is being compiled since it was called from 'Test.forward'
  File "test.py", line 9
		# return torch.tensor(1)
		sequences = [torch.randn(torch.randint(1,5,(1,)),3) for i in range(10)]
		packed = torch.nn.utils.rnn.pack_sequence(sequences, enforce_sorted=False)
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE

</code></pre>
How to delete eveything except the model from the GPU without restarting the ipython kernel?

theairbend3r: <p>I’m trying to experiment with different model-architectures in a single jupyter notebook.</p>
<p>At the end of the notebook, I want to run all the n-models I trained above on the test-data.</p>
<p>But my GPU doesn’t have enough memory to train n-different models at once. I need to free up some space before training the next model. But I also need to preserve my n-models on the GPU so that I can use them all later for inference/test.</p>
<p>This is something I’m going for -</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(...)
test_dataloader = DataLoader(...)


# model-1
class Model(nn.Module):
    def __init__(self):
        bla = nn.Conv2d()
    def forward(self, x):
        x = bla(x)
        return x


# train model_1
for i in trainloader:
    # train model_1
    pred  = model_1(x)

# save model_1 params and free up the GPU

# train model_2
# save model_2 params and free up the GPU


# train model_3
# save model_3 params and free up the GPU


# finally test all three models together
</code></pre>
<p>How do free up space on the GPU without deleting the model(on the gpu) and without restarting the kernel?</p>
<p>I came across <code>torch.cuda.empty_cache()</code>. Can I safely use it in this case?</p>

chaslie: <p>theairbend3r,</p>
<p>you could try saving the trained model &amp; optimiser state dictionaries and then load as needed.</p>
<p><a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model</a></p>

theairbend3r: <p>In this way, the GPU would still have all the tensors/models. I would need to restart my kernel to free up the memory, right?</p>

chaslie: <p>if you save the tensordata set as well using torch.save(tensordatasetnetname, ‘filename.pt’) i guess so</p>

theairbend3r: <p>Hmm. I want to avoid restarting the kernel. I’d like to train all the n-models in one go. Is that possible? <code>torch.cuda.empty_cache()</code> seems to work. I basically call it after training each model. But I’m not sure what it’s clearing.</p>
Training time is longer after pytorch upgrade

Ran: <p>When pytorch was upgraded from 0.4.1 to 1.1.0 and torchvision was upgraded from 0.2.0 to 0.3.0, the training time was changed from the original 60min to 90min. (Ubuntu16.04, cuda9.0, gcc5.4.0, 1080Ti)</p>

ptrblck: <p>Which model and dataset are you using?</p>

Ran: <p>I use the seq2seq model which contains ResNet and RNN and the dataset is Synthetic text image dataset (~more than 10 million images).</p>

ptrblck: <p>Could you update to the latest stable PyTorch version (<code>1.5</code>) and rerun the code please?</p>
Shuffle ImageFolder labels

Alfred: <p>I have to randomly shuffle a fraction of the labels of a dataset. I’ve always done it doing the following (Example on CIFAR10):</p>
<pre><code>train_dataset = dsets.CIFAR10(root='./data', 
                    train=True, 
                    transform=transform_train,
                    download=True)




for i in range(len(train_dataset.targets)):
if (i % 3 == 0):
    train_dataset.targets[i] = random.randint(0, 9)
</code></pre>
<p>Particularly, the code above shuffles 33% of the labels. I want to do the same using ImageFolder, prebuilt on torchvision, but somehow it fails to shuffle the labels without giving any error.</p>
<p>In fact, I have checked the training set, and what I noticed was that the label train_dataset[0][1] was unchanged. Furthermore, if I manually try to change it by doing</p>
<pre><code>train_dataset[0][1] = random.randint(0,9)
</code></pre>
<p>It gives an error (which makes sense, it is a Tuple)</p>
<p>How can I shuffle a percentage of the labels? Thank you!</p>

ptrblck: <p>Your posted code doesn’t seem to shuffle the dataset, but instead randomly assigns a new label to the current target tensor.<br>
Are you sure you want to use this approach?</p>

Alfred: <p>I want to shuffle a fraction of the dataset (or randomly reassign the labels), in order to train a neural network on a noisy dataset. And this approach works for CIFAR and MNIST (torchvision)  but it fails for ImageFolder. The thing is that I don’t know why it does it, and how to fix it.<br>
Thanks!</p>

ptrblck: <p>For <code>ImageFolder</code>, you could try to manipulate the <code>dataset.samples</code> attribute, which is a <code>list</code> of tuples for the image path and the target.</p>
How does conv2d actually work from inside regarding channel size of filters?

Helaly96: <p>i have a small question that i actually didn’t think about much until  after some time after finishing a small project and reflecting on the code -obv, i am still a beginner-<br>
anyways<br>
let’s say we have</p>
<p><code>L1=nn.conv2d(in_channels=3,output_channels=10,kernel=.. etc...)</code></p>
<p>so basically what this does it preforms convolution2D on the input image which has 3 Channels and will have 10 channels of depth as an output.<br>
what this means is this layer is preforming <strong>(10)</strong> filters on the given image.</p>
<p>my simple question, does each filter have 3 channels from inside to apply on the image?</p>
<p>to put it in an other way<br>
<code>L1=nn.conv2d(in_channels=512,output_channels=1,kernel=.. etc...)</code><br>
so this means the input has 512 channels and we are preforming an amazing convolution haha to make it have 1 channel only, so we are applying <strong>one</strong> filter.<br>
so does this mean that this filter has 512 layers in it to match the input?</p>

Nikronic: <p>Hi,</p>
<p>Yes exactly, that is why in kernel size you just provide <code>(h, w)</code> not channel size of it, because it has to match <code>in_channels</code> of conv2d layer. And again you are right, <code>output_channels</code> is the number of filters with size of <code>(in_channels, h, w)</code> which will be stacked together at the end.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da83f1a38dc7d6645728fe55780533a225bc1d04.png" data-download-href="https://discuss.pytorch.org/uploads/default/da83f1a38dc7d6645728fe55780533a225bc1d04" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da83f1a38dc7d6645728fe55780533a225bc1d04_2_660x500.png" alt="image" data-base62-sha1="vb4Mb25LRRAYQfmlqCbx4L5cn8E" width="660" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da83f1a38dc7d6645728fe55780533a225bc1d04_2_660x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da83f1a38dc7d6645728fe55780533a225bc1d04_2_990x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/a/da83f1a38dc7d6645728fe55780533a225bc1d04.png 2x" data-dominant-color="CAE0EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1025×776 41.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <a href="https://stackoverflow.com/questions/48480695/how-we-calculate-the-depth-of-convolution-output" rel="noopener nofollow ugc">source</a></p>
<p>Bests</p>
How to keep output size identic with input size?

sudonto: <p>Hi experts,</p>
<p>I have a tensor with shape <code>(2, 64, **7**, 28, 28)</code> and pass it to <code>Conv3d()</code> with <code>stride = 1</code>, <code>padding = 1</code> and <code>kernel_size = (1,3,3)</code>. Why my the output has <code>(2, 64, **9**, 28, 28)</code>? Notice that the temporal size incresed to 9. I want to keep the size same.</p>
<p>How to keep the size same using <code>kernel_size = (1,3,3)</code>? Tensor format is <code>(bs, ch, t, h, w)</code><br>
Thanks.</p>

Nikronic: <p>Hi,</p>
<p>You can compute output size using this formula: <code>out = floor((in - kernel + 2*padding) / stride) + 1</code><br>
So, in your case, <code>floor((7 - 1 + 2*1) / 1) +1 = 9</code>. There is two ways to have the identical output size, first of is to use proper kernel size which in your case it is <code>3</code> instead of <code>1</code>. Second approach is to change the padding only for that particular channel which in your case, you need to use zero padding.</p>
<p>For the second approach this post may help you:</p><aside class="quote" data-post="1" data-topic="21212">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/49beb7/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-pad-one-side-in-pytorch/21212">How to pad one side in pytorch</a> 
  </div>
  <blockquote>
    By default pytorch only support padding on both side, but for example, I have a feature of 1x512x37x56(NCHW) and I want to pad on one side to 1x512x38x57, how can I do it?
  </blockquote>
</aside>

<p>Bests</p>

sudonto: <p>Thank you. I need my kernel to convolve only on single timestep at a time so that’s why I set <code>1</code> for the kernel size. I will try you 2nd suggestion.</p>
[GAN], [Text] Generator generates padding symbol

bhashithe: <p>Sorry for being vague in the title, for me the issue is too complicated to put it into simpler words. Bare with me until I explain the back story first.</p>
<p>I have a GAN to generate text, both the generator and the discriminator are LSTMs. I am trying to learn SeqGAN therefore most of my code is based on <a href="https://github.com/williamSYSU/TextGAN-PyTorch" rel="nofollow noopener">this repository</a>.</p>
<p>I am using <code>pack_paded_sequences</code> to input sequences to the Generator. Since i need to calculate the lengths of all sequences in the batch in order to pack them I use the following logic to calculate the length</p>
<pre><code class="lang-auto">len(seq[seq!=padding_symbol])
</code></pre>
<p>If a given sequence only has padding symbol then the length obviously will be zero. And we cannot use the function to pack the sequences then.</p>
<p>Since this is a Text generation and based on SeqGAN i am using Policy Gradients to calculate the reward for a complete sequence. To calculate the reward for an incomplete sequence, they have used monte carlo search to fill up the unknown tokens.</p>
<p><a href="https://github.com/williamSYSU/TextGAN-PyTorch/blob/bf0215791e6b55c12efd55da0d93ca1ba67dc4da/utils/rollout.py#L128" rel="nofollow noopener">Monte Carlo search</a></p>
<p>How they have set that up is that, first the generator will be sampled and parts of the sequence will be sent to the monte carlo search function to get the tokens and using that completed samples we calculate the reward for that part of the sequence.</p>
<p>So the issue is that the generator sometimes generates a padding symbol at the front of the sequence, and when we calculate the reward for the first token, it will raise the following run time error.</p>
<p><code>Length of all samples has to be greater than 0, but found an element in 'lengths' that is &lt;= 0</code></p>
<p>How would you stop the generator from generating the EOS symbol because thats what I have used as my padding symbol.</p>
Why is my Topic / Question grayed out?

0xfede: <p>I posted a question several hours ago and it’s grayed out. What does this mean?</p>

albanD: <p>Hey,</p>
<p>You mean this one: <a href="https://discuss.pytorch.org/t/transfer-learning-on-yolov3/84001" class="inline-onebox">Transfer Learning on YOLOv3</a> ? I can see it just fine on my side</p>

Dudly01: <p>For me the things that are grayed out seem to be the ones I have visited once.</p>

0xfede: <p>Yep, that’s the one.</p>
<p>Ok so maybe it means just that either no one visited the question other than me or that it has no replies?</p>

albanD: <p>In the main page. Posts that are greyed out are the posts you already looked at. You can try looking at other posts and they will grey out as well.</p>

0xfede: <p>Ok yes you are right. I just tried it. I thought it was some moderation thing.<br>
Thanks!</p>
Is there a way to compute Kaldi pitch features with gradients?

Ofek_Cohen: <p>I want to compute kaldi pitch features at the end of a network. Is there a way to compute gradients of the kaldi pitch feature extraction? (Maybe something similar to spectrogram of torchaudio).</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>I’m not familiar with kaldi pitch, how do you compute it? If you use pytorch construct, the autograd should work just fine <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Ofek_Cohen: <p>The original Kaldi library is written in C and usually called using .sh files. I do not know of a Pytorch function that implements it, although there are python envelopes such as pykaldi.</p>

albanD: <p>If there is no pytorch implementation, I’m afraid you’ll have to do one of the following:</p>
<ul>
<li>reimplement a new version with pytorch operators to use the autograd</li>
<li>write a custom Function (<a href="https://pytorch.org/docs/stable/notes/extending.html">https://pytorch.org/docs/stable/notes/extending.html</a>) that uses the original library in the forward and for which you will specify the backward.</li>
</ul>
Slicing a Tensor around a location given by another tensor

alamin: <p>I want to slice a Tensor around a spatial location which is coming from another tensor. Is there any efficient way of achieving this? Below is an example of what I want. In this case I sliced a 3x3 tensor from the input tensor using the location from the location tensor.</p>
<pre><code class="lang-auto">input_tensor : 
tensor([[[12, 30, 21, 32,  3, 38],
         [ 8, 36, 20, 35, 16, 36],
         [12, 31, 23, 30, 20, 20],
         [33,  2, 10, 22, 31,  5],
         [30, 13,  5,  2, 21,  7],
         [34, 39, 32,  8,  6, 26]],

        [[32,  5, 10, 19, 14, 29],
         [ 3,  5, 27, 15,  2,  6],
         [32,  6, 39, 15, 18, 37],
         [19, 33, 19,  4, 29,  7],
         [ 0, 32,  8, 28,  1,  5],
         [24, 15, 39,  6, 39, 24]]])
location_tensor:
tensor([[4, 3],
        [3, 1]])

Expected output,
tensor([[[10, 22, 31],
         [ 5,  2, 21],
         [32,  8,  6]],

        [[32,  6, 39],
         [19, 33, 19],
         [ 0, 32,  8]]])
</code></pre>
<p>#####################<br>
Code I used,</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
t1 = torch.randint(0, 40, (2, 6,6))
ix = torch.randint(1, 5, (2, 2))
print (t1)
print (ix)
t1_l = list(torch.unbind(t1))
ix_l = list(torch.unbind(ix))
t = torch.stack([t[0][t[1][0]-1:t[1][0]+2, t[1][1]-1:t[1][1]+2] for t in zip(t1_l, ix_l)])
print (t)
</code></pre>
LSTM batch training over one long video session

zerone: <p>Hello Everyone! As the title goes, I am having trouble in coming up with an LSTM model for training over long video sessions. I have, say <strong>K</strong> training videos, each of duration 1 hour or so. For each video, I am extracting frames at a certain FPS (say 10 frames per second) and then I obtain their corresponding 512 dimensional CNN codes (from a pre-trained CNN). So, considering for one such video, say I have 36000 frames, thus I have a feature matrix of size 36000 x 512.</p>
<p>Now, as it’s obvious… each frame is related to the previous frame. I wish to train my LSTM one video at a time. That is… my one epoch will consist of iterating over each of the <strong>K</strong> videos, and for each video I will pass its frames’ features to the LSTM. I intend to pass the frames’ features by constructing a 3D tensor of shape [<em>batch_size, sequence_size, 512</em>] and then obtain their corresponding labels (2D tensor of shape: [<em>batch_size x sequence_size, 1</em>] (i.e. a many to many model). To train the model… there are two possible ways I have identified.</p>
<p><strong>Method 1</strong>: Set the <em>batch_size = 1</em> i.e. the tensor is of shape [<em>1, sequence_size, 512</em>] and pass it to the LSTM initialized with hidden state (including the cell state) = <em>None</em>. Now, as the LSTM unfolds over each frame in the <em>sequence_size</em>, the injection of hidden state from the previous time-step ensures the continuity of the learning. In addition, I can simply pass the hidden state obtained at the end of batch (note that <em>batch_size</em> is 1) to the next batch, thus ensuring that learning over the next batch is done in consideration with the previous batch. You can think of this as… a set of… say 200 frames is broken into two batches each of sequence size 100 frames, and learning over 200 frames is equivalent to learning over one batch followed by the another.</p>
<p><strong>Method 2</strong>: Set the <em>batch_size = N</em> (where N &gt; 1, say N=10), i.e. the tensor is of shape [<em>10, sequence_size, 512</em>]. Now I need to maintain the same continuity of learning over each element in the batch (note that each element in the batch is of shape [<em>sequence_size, 512</em>]). But since batch processing is parallelized, I am assuming that each of the element in the batch will have the same initial hidden state of <em>None</em> and thus learning won’t be continuous. It is as if, each element of the batch is considered independent of each other. But in my case it isn’t. The first frame of each element in the batch is related to the last frame of the previous batch.</p>
<p><strong>My question</strong>: How do I ensure that hidden state obtained at the end of processing the first element of batch is passed to the next element of the batch? If I find a method for it, then I can simply pass the hidden state obtained at the end of each batch to the next batch and a continuous learning would be maintained. If there’s no way… then I believe I will have to work with the slower Method 1 to ensure a continuous learning. Please let me know your thoughts. Thanks!</p>
Help in Seq2Seq Tutorial

SrJ: <p>In the pytorch Seq2Seq tutorial provided at pytorch website, in the decoder there is a extra relu step between Embedding and GRU but in the encoder there is no relu. Why is that? How this make any difference ? Can anyone Elaborate?</p>
<blockquote>
<p>class DecoderRNN(nn.Module):<br>
def <strong>init</strong>(self, hidden_size, output_size):<br>
super(DecoderRNN, self).<strong>init</strong>()<br>
self.hidden_size = hidden_size</p>
<pre><code>    self.embedding = nn.Embedding(output_size, hidden_size)
    self.gru = nn.GRU(hidden_size, hidden_size)
    self.out = nn.Linear(hidden_size, output_size)
    self.softmax = nn.LogSoftmax(dim=1)
def forward(self, input, hidden):
    output = self.embedding(input).view(1, 1, -1)
    *output = F.relu(output)*
    output, hidden = self.gru(output, hidden)
    output = self.softmax(self.out(output[0]))
    return output, hidden

def initHidden(self):
    return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>
</blockquote>
<blockquote>
<p>class EncoderRNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size):<br>
super(EncoderRNN, self).<strong>init</strong>()<br>
self.hidden_size = hidden_size</p>
<pre><code>    self.embedding = nn.Embedding(input_size, hidden_size)
    self.gru = nn.GRU(hidden_size, hidden_size)

def forward(self, input, hidden):
    embedded = self.embedding(input).view(1, 1, -1)
    output = embedded
    output, hidden = self.gru(output, hidden)
    return output, hidden

def initHidden(self):
    return torch.zeros(1, 1, self.hidden_size, device=device)
</code></pre>
</blockquote>
PIL image and its normalisation

Kakar_Nyori: <p>Hi!</p>
<p>I am very new to machine learning in general, and just started with Pytorch because of it’s simplicity. So I am following the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#loading-and-normalizing-cifar10" rel="noopener nofollow ugc">TRAINING A CLASSIFIER</a> of 60 minutes blitz tutorial. There the I cannot understand how and what this lines mean:</p>
<blockquote>
<p>The output of torchvision datasets are PILImage images of range [0, 1]. We transform them to Tensors of normalized range [-1, 1].</p>
</blockquote>
<pre><code class="lang-auto">transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
</code></pre>
<p>Could you please help me understand it?</p>
<p>Thank you!</p>

axki: <p>I think this might be useful for understanding this: <a href="https://discuss.pytorch.org/t/understanding-transform-normalize/21730/2">https://discuss.pytorch.org/t/understanding-transform-normalize/21730/2?u=axki</a></p>

Kakar_Nyori: <p>Yes, that was helpful. But I am actually confused by what does PILImage of range [0, 1] mean. Could you please elaborate.</p>

vmirly1: <p>Actually, it’s not accurate. The pixel values are in range <code>[0, 255]</code> not <code>[0, 1]</code>, When you open an image with PIL, you get an object of the following classes depending on JPG or PNG:</p>
<pre><code class="lang-auto">from PIL import Image

img1 = Image.open('filename.jpg')
print(type(img1))
&lt;class 'PIL.JpegImagePlugin.JpegImageFile'&gt;

img2 = Image.open('filename.png')
print(type(img2))
&lt;class 'PIL.PngImagePlugin.PngImageFile'&gt;
</code></pre>
<p>To see the pixel-values of these objects, you can use <code>list(img1.getdata())</code> or <code>np.asarray(img1)</code> which will show the values [0, 255]:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; np.asarray(img1)[:2, :2]
array([[[255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255]]], dtype=uint8)
</code></pre>

axki: <p>The torchvision datasets return pixelvalues in range 0-255 as <a class="mention" href="/u/vmirly1">@vmirly1</a> says, so yeah, there seems to be a typo of some sort in the tutorial maybe?</p>
<p>The transforms.ToTensor() in your transform will convert it to range 0-1 (documentation: <a href="https://pytorch.org/docs/master/torchvision/transforms.html#torchvision.transforms.ToTensor" rel="nofollow noopener">https://pytorch.org/docs/master/torchvision/transforms.html#torchvision.transforms.ToTensor</a>)</p>
<p>The literal formulation “PILImage fo range [0, 1]” means that the individual pixel values are between those two values.</p>
<p>Hope this helped! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kakar_Nyori: <p>Okay now everything makes sense. Thank you <a class="mention" href="/u/vmirly1">@vmirly1</a> <a class="mention" href="/u/axki">@axki</a>  !</p>
Problem with gather function

Imene.b: <p>hy,<br>
i have tow  tensors<br>
ind with size = (2, 261234, 1)<br>
g = with size = (2, 783456, 1)<br>
i wante to gather tem with the gather function, and put the result in g<br>
whene i write g = g.gather(1, ind) the folowing error apeer:<br>
RuntimeError: Expected object of type torch.LongTensor but found type torch.FloatTensor for argument <span class="hashtag">#3</span> ‘index’<br>
whene i try to change the type of ind by ind.type(tornch.long), i got  this error;</p>
<p>RuntimeError: Invalid index in gather at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:600<br>
any one can help me?</p>

albanD: <p>Hi,</p>
<p>This error means that the indices contained in <code>ind</code> are out of bound for the indexing you’re trying to do.<br>
Have you checked these values?</p>

Imene.b: <p>how can the indices be out of bounds?<br>
the shapes of the tensors are correct to do gather, and values in ind are the result of a series of convolutions and max pooling, i cant change theme.</p>

albanD: <p>If your dimension is of size <code>783456</code>, then the indices have to be in [0, 783455]`.<br>
Just like if you have a python list with 10 elements, you can’t access the 20th.</p>

Imene.b: <p>tanks,<br>
you are right,  i had make some modifications, wich change the nember of inputs, so the indices must be changed also<br>
thank you very mutch. i have found the solution</p>
Missing Shared Object File: libc10_cuda.so

FabianSchuetze: <p>When I try to exectue a program pytorch c++ program I wrote, I receive the following error:</p>
<pre><code class="lang-auto"> error while loading shared libraries: libc10_cuda.so: cannot open shared object file: No such file or directory
</code></pre>
<p>I can locate the object on my computer at: <code>/home/fabian/torch_cpp/libtorch/lib/libc10_cuda.so</code> . The <code>cmake-gui</code> also references this file with the options:</p>
<pre><code class="lang-auto">C10_CUDA_LIBRARY           /home/fabian/torch_cpp/libtorch/lib/libc10_cuda.so
C10_LIBRARY                /home/fabian/torch_cpp/libtorch/lib/libc10.so
</code></pre>
<p>I use a z-shell on Ubuntu 19.10 and I did include <code>/home/fabian/torch_cpp/libtorch/lib/</code> to my <code>PATH</code> variable. I did download libtorch from <code>https://download.pytorch.org/libtorch/cu101/libtorch-cxx11-abi-shared-with-deps-1.5.0%2Bcu101.zip</code>.</p>
<p>Does somebody have any hints or ideas how I can solve the problem? That would be very kind!</p>

glaringlee: <p><a class="mention" href="/u/fabianschuetze">@FabianSchuetze</a><br>
Assuming you followed this instruction here: ‘<a href="https://pytorch.org/cppdocs/installing.html" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html</a>’<br>
Can you try appending the  ‘/home/fabian/torch_cpp/libtorch’ into your CMAKE_PREFIX_PATH?</p>

FabianSchuetze: <p>Thank you, <a class="mention" href="/u/glaringlee">@glaringlee</a> . You helped me solve the problem.</p>
Concatenate FC layers to layers of trained model

gil_bc: <p>Hi,</p>
<p>I have a trained model and I wish to try and use it’s inner layers as input for a new model (say, a 2 layer FC model) with a slightly different learning objective. how can I do that?</p>

SANTOSH_S: <p>Hi <a class="mention" href="/u/gil_bc">@gil_bc</a><br>
If you had defined a particular layer such as <code>self.fc1 = nn.Linear(..)</code> in your original model_1 then you’ll be able to extract the model’s weights using <code>weights = model_1.fc1.weight.data</code>.<br>
You define in model_2 this same layer and initialise it <code>wights</code> and freeze it.</p>
<p>In model_2 code:</p>
<pre><code class="lang-auto">self.fc1.weight = weights     #with or without grad
</code></pre>
<p>Let me know if this worked.</p>
<p>Some useful <a href="https://discuss.pytorch.org/t/custom-weight-initialization/20544">discussions</a> that you should look at.</p>

gil_bc: <p>It helped,<br>
and the discussions as well. thanks!</p>

gil_bc: <p>Hi <a class="mention" href="/u/santosh_s">@SANTOSH_S</a></p>
<p>I’m afraid I was still unable to execute what I was trying to do.<br>
I wish to connect a part of my trained network with another untrained network, to learn a new objective.<br>
So let’s say I want to use the first few layers of the trained network, than concatenate to them the new layers and train the whole thing together.</p>
<p>How would you perform such thing?</p>
Batch Norm Example in CNNs--- Is this correct?

Nick_ishere: <p>Hi! I’m trying to implement the batch normalization in CNN tasks. And I found this sample code through the Internet. But I’m not sure whether using Batch Norm in Convolutional Layer in this way is correct or not. Can anyone give me some ideas? Thank you!</p>
<p>The example code is shown below:</p>
<pre><code class="lang-auto">class CNN_batch(nn.Module):
    
    # Contructor
    def __init__(self, out_1=16, out_2=32, number_of_classes=10):
        super(CNN_batch, self).__init__()
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)
        self.conv1_bn = nn.BatchNorm2d(out_1)

        self.maxpool1=nn.MaxPool2d(kernel_size=2)
        
        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)
        self.conv2_bn = nn.BatchNorm2d(out_2)

        self.maxpool2=nn.MaxPool2d(kernel_size=2)
        self.fc1 = nn.Linear(out_2 * 4 * 4, number_of_classes)
        self.bn_fc1 = nn.BatchNorm1d(10)
    
    # Prediction
    def forward(self, x):
        x = self.cnn1(x)
        x=self.conv1_bn(x)
        x = torch.relu(x)
        x = self.maxpool1(x)
        x = self.cnn2(x)
        x=self.conv2_bn(x)
        x = torch.relu(x)
        x = self.maxpool2(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x=self.bn_fc1(x)
        return x
</code></pre>
<p>(As you can see, it adds <code>self.conv1_bn</code> , <code>self.conv2_bn</code> after the convolutional layer directly. Does this work?)</p>

ptrblck: <p>Yes, this should work. Do you see any issues or is this just a general question?</p>
<p>You wouldn’t see the last batch norm layer (<code>self.bn_fc1</code>) at the end of the model usually, but it might fit your use case. I would nevertheless compare it to a model without this layer and check, which one works better.</p>

Nick_ishere: <p>Hi ptrblck! Thank you for your reply. No, I didn’t see any issue, and the model can work. I’m just wondering whether these steps are reasonable. Thank you a lot for the clarification.</p>
<p>Yes, I also think the output layer is quite strange, as the batch norm layer usually before the output layer. Thanks for the reminder, ptrblck! I’ll try it.   <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Nn.relu() seems to be problematic

Peter_Ham: <p>I use nn.ReLU() inside a MLP, when I do not specify inplace, I got nan output, but when i specify inplace=True, I got normal output, why?</p>

albanD: <p>Hi,</p>
<p>This should not happen if you use all the modules properly.<br>
Can you share small code sample to reproduce this?</p>
Padding value when training Bert from scratch

gmondaut: <p>Hi All,<br>
I have a quesiton:<br>
Let’s suppose we are training Bert from scratch.<br>
When we pass the embeddings as inputs to BertModel the three tensors referring to token, segment, and position must have the same padding value (i.e. 0)?<br>
If the padding value for position/segment is different from the value used to pad the token sequence (e.g. 12), what would be the issue with this during the training?<br>
Thanks in advance</p>