akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I trained same network with pytorch but totally bad results

Ali_Amiri: <p>Hi,<br>
I have trained a transformer (from scratch) in PyTorch (many times in many ways) but steel gives me a terrible accuracy just like it is still random<br>
but in Keras, I trained the same network once and it gave me absolutely perfect performance even with a 5 samples dataset (I tested the model with same 5 samples in both frameworks but still Keras gives great result )<br>
I checked the architecture many times but I haven’t figure out the problem yet!<br>
I reeeaally appreciate if you help to me and other people with the same problem</p>
<p>I think it’s worth nothing to say that if I give whole input and output to the model (in inference phase) it would work perfectly (I mean without using teacher forcing it works)</p>
<p>notebooks are here:<br>
Keras: <a href="https://colab.research.google.com/drive/1cz5Q-FgThpmM0lAT1mI8AgCRNi9ctsl-?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1cz5Q-FgThpmM0lAT1mI8AgCRNi9ctsl-?usp=sharing</a><br>
PyTorch: <a href="https://colab.research.google.com/drive/1GlWjfKo9_GYz93rW4ZWyR6VSN45Bec3v?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1GlWjfKo9_GYz93rW4ZWyR6VSN45Bec3v?usp=sharing</a></p>
Error in script::Module::get_method

krrishabh: <pre><code class="lang-auto">torch::Tensor hidden = module.get_method("get_initial_state")({torch::tensor({20})});
</code></pre>
<p><strong>/home/rakesh/rishabh_workspace/Garbage/LMtesting/LMtesting.cpp:65:64:</strong> <strong>error:</strong> conversion from ‘ <strong>c10::IValue</strong> ’ to non-scalar type ‘ <strong>at::Tensor</strong> ’ requested</p>
<p>torch::Tensor hidden = <strong>module.get_method(“get_initial_state”)({torch::tensor({20})})</strong> ;<br>
^</p>

Michael_Suo: <p><code>get_method</code> returns an IValue, which you can sort of think of as like a PyObject. You need to explicitly cast it to the C++ type you want, so you should do <code>hidden.toTensor()</code> to get an <code>at::Tensor</code>.</p>

krrishabh: <p>For example :</p>
<pre><code class="lang-auto">auto hidden_map = module.get_method("get_initial_state")({torch::tensor({1})});
torch::Tensor t = hidden_map.toTensor()
</code></pre>
How to implement bidirection using RNNCell?

11182: <p>Hello, I am a student studying pytorch.</p>
<p>I want to implement bidirection using RNNCell.</p>
<p>If nn.RNN is below</p>
<p><code>torch.nn.RNN(input_size=1, hidden_size=1, num_layers=2, batch_first=True, bidirectional=Ture)</code></p>
<p>What is nn.RNNCell?</p>
Per-parameter learning rate and weight decay

rasti-najim: <p>I’m trying to apply both per-parameter learning rate and weight decay, however, I get an error saying:</p>
<blockquote>
<p>some parameters appear in more than one parameter group<br>
Here’s a snippet of my code:</p>
</blockquote>
<pre><code class="lang-auto">param_optimizer = list(model.named_parameters())
    no_decay = ["bias", "LayerNorm.bias", "LayerNorm.weight"]
    optimizer_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0},
        {'params': model.roberta.parameters(), 'lr': lr[0]},        
        {'params': model.last_linear.parameters(), 'lr': lr[1]}    
        
    ]

    optimizer = AdamW(optimizer_parameters)```

I would much appreciate your help!</code></pre>

ptrblck: <p>The parameters might overlap, as you are getting all parameters in <code>param_optimizer</code>, while also using <code>model.roberta.parameters()</code> and <code>model.last_linear.parameters()</code>.</p>
<p>You could create <code>dicts</code> for all your conditions and parameter sets and check the keys for duplicates.</p>

rasti-najim: <p>So my workaround was to use the per-layer learning rates and use one weight decay value for all the parameters.</p>
<pre><code class="lang-auto">optimizer_parameters = [
#         {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.001},
#         {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0},
        {'params': model.roberta.parameters(), 'lr': lr[0]},        
        {'params': model.last_linear.parameters(), 'lr': lr[1]}    
        
    ]

    optimizer = AdamW(optimizer_parameters, weight_decay=0.001)
</code></pre>
<p>Would that affect performance since now I’m applying weight decay to the <code>bias</code> and <code>LayerNorm</code> params, or is it ok?<br>
And thanks btw!</p>
How to extract/print the node weight

Lame: <p>Hi all,</p>
<p>I’m working on Graph Conv Network, each node has 2 features; I’m doing a regression model</p>
<p>I applied self-attention pooling in order to know which of these nodes contributes to the final prediction, every node gets a weight in which that would affect the final result</p>
<p>So, my question is how I can print/extract these weights in order to know/rank the importance of the nodes, and when exactly?</p>
<p>Thanks.</p>
<p>Here is my model:</p>
<blockquote>
<p>class MyNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(MyNet, self).<strong>init</strong>()<br>
self.conv1 = GCNConv(2,5)<br>
self.conv2 = GCNConv(5,8)<br>
self.pool1=  SAGPooling(8)<br>
self.fc1 = nn.Linear(538136,3)<br>
self.fc2 = nn.Linear(3,1)</p>
<pre><code>def forward(self, data):
    x, edge_index, batch = data.x, data.edge_index, data.batch
    x = F.relu(self.conv1(x, edge_index))
    x = F.relu(self.conv2(x, edge_index))
    
    x, edge_index, _, _, _, _ = self.pool1(x, edge_index, None, batch)

    b=data.y.shape[0]
    x=x.view(b,-1)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    x = torch.sigmoid(x)
    return x
</code></pre>
</blockquote>
Autoencoder_classifier

sriharsha0806: <p>Hi,<br>
I trained a denoising autoencoder on mnist dataset corrupted by random gaussian noise and built a classifier with encoder as backbone(The layers are freezed), and a linear classifier layer trained on mnist dataset corrupted by random gaussian noise. Its performance on test dataset is 10%. May I know how to improve this network. I am sharing the code.</p>
<p><a href="https://colab.research.google.com/drive/1fnLJxtzuT2cNeWXycjn3_pVej5DE3z-y?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1fnLJxtzuT2cNeWXycjn3_pVej5DE3z-y?usp=sharing</a></p>
Conv layer problems

lifeblack: <p>Hi! I have to simulate a  network but I cannot understand this sub-network:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/343dcbb62a997c315428f03ecd3a7e9f0f4b8cd8.png" alt="image" data-base62-sha1="7s9dXPS5PvYXCkJrmHRMvqd3gus" width="390" height="97"></p>
<p>Conv_1 :                       kernel_size= 3 * 3 * 1   , kernel_number=64<br>
Conv_2…Conv_7:         kernel_size= 3 * 3 * 64 , kernel_number=64<br>
Conv_8 :                      kernel_size= 3 * 3 * 64   , kernel_number=1<br>
what is the In_channels of each layer?<br>
Would you mind helping me figure out this sub network?</p>

Isaac_Kargar: <p>I think when you say <code>kernel_size=a*b*c, kernel_number=d</code>, a and b are kernel size, c is the number of input channel and d is the number of output channel.</p>

lifeblack: <p>thanks for your reply. It fixed my problem. Could you mind explaining more? I have problems about 3d kernel sizes.</p>

Isaac_Kargar: <p>If I understood the question correctly, you want to know more about 3 dimension in kernel size. So for one feature-map that you have, you will have several kernels. For example for your conv_2 layer, you have a kernel with h=3, w=3 and depth of 64 which the same as your feature-map depth. You slide this kernel on your feature-map and do the multiplications and create one feature layer for the feature-map in the next layer. You do this for kernel_number=64 times and create a feature-map of depth 64 for the next layer.</p>
Understanding nn.NLLLoss target size

NearIt: <p>I am trying to predict the next word using LSTM.</p>
<p>those are my layers:</p>
<pre><code class="lang-auto"># define layers
    self.embedding = nn.Embedding(num_embeddings = vocab_size, 
                         embedding_dim = input_length) 
    self.lstm = nn.LSTM(input_size = input_length, 
                      hidden_size = hidden_dim,
                      num_layers = num_layers,
                      dropout = drop_prob,
                      batch_first = True, #batch_first= True, then the input and output tensors are provided as (batch, seq, feature)
                      bidirectional = False)
    self.fc = nn.Linear(hidden_dim, output_size)
    self.dropout = nn.Dropout(p = drop_prob)
    self.LogSoftMax=nn.LogSoftmax(dim = 1) # meaning result size [batch_size x seq_len x output_size]
</code></pre>
<p>with the nn.NLLLoss .</p>
<p>the input to this network have size of [batch_size,seq_len]<br>
and my output_size is the volcab_size = 10000.</p>
<p>the output of the network has size of [batch_size,seq_len,vocab_size ],<br>
which i kinda get it since the last dim (volcab_size ) is the probability (or log probability) of the next word to be wordX.</p>
<p>what i dont understand is how to create the target. target size should be [batch_size, vocab_size ],<br>
but this is not one-hot vector (right?). so in theory [batch_size, 1] should be enough  (when in the last dim is the index in the vocabulary  of the next word).<br>
any way to do that? there is no meaning to use [batch_size, vocab_size] as target size.</p>
Combining metrics in ignite.metrics

stvhuang: <p>In the following (1) code block, for each metric (accuracy, precision, recall, f1), I create a metric class to record <code>(y_pred, y)</code> and calculate the score at the end. My question is can I create a new metric class that combines the four metrics. And I just need to update it once in a loop. For example, see the code block (2).</p>
<p>(1) What I use now.</p>
<pre><code class="lang-python">from ignite.metrics import Accuracy, Precision, Recall, Fbeta

accuracy = Accuracy()
precision = Precision()
recall = Recall()
f1 = Fbeta(beta=1.0, average=False, precision=precision, recall=recall)

for X, y in dataloader:
    y_pred = model(X)

    # calculate loss, backward, and update weights

    accuracy.update((y_pred, y))
    precision.update((y_pred, y))
    recall.update((y_pred, y))

print(f"Accuracy: {accuracy.compute()}")
print(f"Precision: {precision.compute()}")
print(f"Recall: {recall.compute()}")
print(f"F1: {f1.compute()}")
</code></pre>
<p>(2) What I want.</p>
<pre><code class="lang-python">import CustomMetric  # metric combining Accuracy, Precision, Recall, and F1

metric = CustomMetric()

for X, y in dataloader:
    y_pred = model(X)

    # calculate loss, backward, and update weights

    metric.update((y_pred, y))

scores = metric.compute()
print(f"Accuracy: {scores['accuracy']}")
print(f"Precision: {score['precision']}")
print(f"Recall: {score['recall']}")
print(f"F1: {scores['f1']}")
</code></pre>

vfdev-5: <p><a class="mention" href="/u/stvhuang">@stvhuang</a> you can do it like this :</p>
<pre><code class="lang-python">class MetricsGroup:
        
    def __init__(self, metrics_dict):
        self.metrics = metrics_dict
        
    def update(self, output):
        for name, metric in self.metrics.items():
            metric.update(output)
            
    def compute(self):
        output = {}
        for name, metric in self.metrics.items():
            output[name] = metric.compute()
        return output


import torch
from ignite.metrics import Accuracy, Precision, Recall, Fbeta

p = Precision()
r = Recall()
m_group = MetricsGroup({
    "accuracy": Accuracy(),
    "precision": p,
    "recall": r,
    "f1": Fbeta(beta=1.0, average=False, precision=p, recall=r)
})

for _ in range(10):
    
    y = torch.randint(0, 4, size=(32, ))
    y_pred = torch.rand(32, 4)

    m_group.update((y_pred, y))

scores = m_group.compute()
print(f"Accuracy: {scores['accuracy']}")
print(f"Precision: {scores['precision']}")
print(f"Recall: {scores['recall']}")
print(f"F1: {scores['f1']}")
</code></pre>
<p>HTH</p>
Upload and preprocess single video for video classification

nabil_babai: <p>Hi,</p>
<p><strong>I am new to pytorch.</strong><br>
I have a pretrained model and I want to try with a SINGLE video.<br>
The video is already in frames (35 frames)<br>
How can I import the frames and preprocess them?</p>
<p>Thank you for your help</p>

Isaac_Kargar: <p>I think you can read the video file using opencv. Take a look at <a href="https://github.com/MohsenFayyaz89/PyTorch_Video_Dataset/blob/master/GeneralVideoDataset.py" rel="nofollow noopener">this</a>.</p>

nabil_babai: <p>Hi <a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a>,<br>
Thank you for your answer, the tuto that you gave me is for videos which is nice. (I am keeping it)<br>
But in this case the video is already in frames and I already have the images. I want to upload them and pass them to the model.</p>
<p>VIDEO42916:<br>
—img00001.jpg<br>
—.<br>
—.<br>
—.<br>
—img00035.jpg</p>
<p>I want to upload those images and pass them to the model.</p>

Isaac_Kargar: <p>So you can use <code>ImageFolder</code> for data loader. Check <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#afterword-torchvision" rel="nofollow noopener">this</a>. You also need to set <code>shuffle=False</code> if the sequence is important. Or you can use general dataloader class like <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" rel="nofollow noopener">this</a>. in the <code>__getitem__</code> method you need to load images, process them and return them and in the training loop feed them into the model.</p>

nabil_babai: <p>Hi <a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a><br>
Thank you very much for your answer.<br>
So I am using a pretrained model and now I want to try with only this video.<br>
I have no other videos just one that I picked myself and want to see if the model is going to classify it right or not.<br>
If I got it right: ImageFolder is for huge dataset right?<br>
Do you have any suggestion?</p>

Isaac_Kargar: <p>I usually myself write a custom class and read the file names in init and do sorting and everything there. Then in getitem method load the image and do processes and return the input and the label.</p>
How to use train model for predict unseen data?

Kishan_Abhani: <p>Hi I am new here and after creating my first model I dont know how to use it.Everywhere I am finding only tutorial on train data but nor sure how to use my trained model.</p>

Isaac_Kargar: <p>You can find the code <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">here</a>. You need to first define your model, then create an object and load your trained model:</p>
<pre><code class="lang-auto">model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>then you can feed your data into model.</p>
RuntimeError: shape '[1, 1024]' is invalid for input of size 50176

rakshit_naidu: <p>I am trying to use Alexnet over the CIFAR-10 dataset. I get the following error:</p>
<pre><code class="lang-auto">&lt;ipython-input-11-34884668038d&gt; in forward(self, x)
     37     def forward(self, x):
     38         x = self.features(x)
---&gt; 39         x = x.view(x.size(0), 256 * 2 * 2)
     40         x = self.classifier(x)
     41         return x

RuntimeError: shape '[1, 1024]' is invalid for input of size 50176
</code></pre>
<p>My Alexnet model code is as follows:</p>
<pre><code class="lang-auto">NUM_CLASSES = 10


class AlexNet(nn.Module):
    def __init__(self, num_classes=NUM_CLASSES):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(64, 192, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 2 * 2, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 256 * 2 * 2)
        x = self.classifier(x)
        return x
</code></pre>
<p>Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

rakshit_naidu: <p>Thank you. I’ve resolved my issue. It was because I was passing the size as 224 when CIFAR images work with size as 32.</p>
RuntimeError: _th_dot not supported on CUDAType for Short

csailnadi: <p>Why I cannot perform dot operation on two matrices?</p>
<pre><code class="lang-auto">
print(S.shape, S.dtype)
print(G.shape, G.dtype)
torch.dot(S, torch.t(G))


torch.Size([256, 31360]) torch.int16
torch.Size([1825, 31360]) torch.int16

RuntimeError: _th_dot not supported on CUDAType for Short
</code></pre>
<p>Same happens with <code>int</code> and <code>long</code>.</p>

Nikronic: <p>Hi,</p>
<p>Could please sure a code that can reproduces this error?</p>
<p>The problem is <code>torch.dot()</code> only supports 1D tensors. So, the error should be about it. The alternatives function for 2D and 3D (batch-wise) is <code>torch.mm</code> and <code>torch.bmm</code> respectively.<br>
Here is the code I used to produce your case (although could not get your error):</p>
<pre><code class="lang-auto">a = torch.randint(high=10, size=(5, 7), dtype=torch.int16)
b = torch.randint(high=10, size=(5, 7), dtype=torch.int16)
torch.dot(a, torch.t(b))

# output 
# RuntimeError: 1D tensors expected, got 2D, 2D tensors at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:431
</code></pre>
<p>Maybe you are using an old version of PyTorch?</p>
How to copy layer's weight to cv::Mat in C++

Andoni_Cortes: <p>Hello everyone,</p>
<p>I am trying to copy convolutional layer weights to a std::vector<a>cv::Mat</a> structure using this function:</p>
<pre><code class="lang-auto">void tensor2mat(torch::Tensor &amp;t, std::vector&lt;cv::Mat&gt; &amp;out)
{
	caffe2::TypeMeta dtype;
	dtype = t.dtype();
	c10::ScalarType type;
	type = t.scalar_type();

	std::cout &lt;&lt; "Tensor Format: {dims} ( depth - type)" &lt;&lt; std::endl;
	std::cout &lt;&lt; t.sizes() &lt;&lt; " ( " &lt;&lt; dtype &lt;&lt; " - " &lt;&lt; type &lt;&lt; " )" &lt;&lt; std::endl;


	int mattype = CV_8U;
	int typesize;
	if (dtype == torch::kInt64 || dtype == torch::kFloat64)
	{
		typesize = sizeof(int64);
		mattype = CV_64F;
	}
	else if (dtype == torch::kFloat32)
	{
		typesize = sizeof(float);
		mattype = CV_32F;
	}
	else if (dtype == torch::kByte || dtype == torch::kChar)
	{
		typesize = sizeof(char);
		mattype = CV_8U;
	}


	if (t.dim() == 4)
	{
		// if number of dimensions is 4
		// NBatch x NChannels x H x W

		out.clear();
                out.resize(t.size(0));

		for (size_t i = 0; i &lt; out.size(); i++) {
			std::vector&lt;cv::Mat&gt; channels(t.size(1));
			for (size_t j = 0; j &lt; t.size(1); j++)
			{
				// Iterate through channels
				torch::Tensor one_plane_tensor = t[i][j];
                                // one_plane_tensor shape is [ 11, 11 ]
				std::cout &lt;&lt; one_plane_tensor.sizes() &lt;&lt; " depth: " &lt;&lt; one_plane_tensor.dtype() &lt;&lt; std::endl;
				one_plane_tensor.to(torch::kCPU);
                                
                                // one_mat is CV_32FC1
				cv::Mat one_mat(one_plane_tensor.size(0), one_plane_tensor.size(1), CV_MAKETYPE(mattype,1));

                                // here I get an error
				std::memcpy((void *)one_mat.data, one_plane_tensor.data_ptr(), typesize * one_plane_tensor.numel());
				channels[j] = one_mat;
			}
			cv::merge(channels, out[i]);
		}
		
		// Create a grid with displaying all the images in the vector
		cv::Mat output = ext::draw::composeLayerImage(std::pair &lt; std::string, std::vector&lt;cv::Mat&gt; &gt;("prueba", imgs), true);


	}
}
</code></pre>
<p>Layer’s weight tensor shape is [64, 3, 11, 11] so the output vector should contain 64 11x11x3 images, so that I can display the content of the weights in a window.</p>
<p>ComposeLayer is just a function to visualize all the images in a grid.</p>
<p>I call the function in this way</p>
<pre><code class="lang-auto">
cv::Mat prueba;

// tensor2mat(net-&gt;conv1.get()-&gt;weight, prueba);
tensor2mat(net-&gt;conv1.get()-&gt;named_parameters()["weight"], prueba);

</code></pre>
<p>When I call it, an error is produced in memcpy instruction. Could anyone give me a hint as to why this is happening?</p>
<p>Thank you very much.</p>

Andoni_Cortes: <p>Answering myself, I have found that the problem came because the tensor one_plane_tensor was on GPU and not on CPU. This can be solve just by assigning the output of ‘to’ function to another tensor that will be on CPU.</p>
<pre><code class="lang-auto">torch::Tensor tensor_in_cpu = one_plane_tensor.to(torch::kCPU);
</code></pre>
<p>and it works.</p>
Difference in forward() impl

nlam: <p>whats the difference between</p>
<pre><code class="lang-auto">def forward(self):
        y = self.lin1.sigmoid(x)
        return y
</code></pre>
<p>and</p>
<pre><code class="lang-auto">def forward(self, x):
        y = x.sigmoid(x)
        return y
</code></pre>

Isaac_Kargar: <p>in the first one, it is using the sigmoid method from lin1 and in the second one, it is using sigmoid from x. We can help you more if you put a more complete sample code.<br>
I didn’t see any x with sigmoid. It should be a tensor. usually you define a sigmoid in your init function and call it in forward:</p>
<pre><code class="lang-auto">...
def __init__(...):
   ...
   self.sigmoid = nn.Sigmoid()

def forward(self, x):
   self.sigmoid(x)
   ...
</code></pre>

nlam: <p>ya sorry, I should have included the whole code. So my actual nn class looks like this for example</p>
<pre><code class="lang-auto">class LogReg(nn.Module):
    def __init__(self):
        super(LogReg, self).__init__()
        self.lin1 = nn.Linear(4, 2)
</code></pre>
<p>I was browsing through examples on how to implement the forward(), and I was seeing both versions on different models, but was confused about the difference</p>

Isaac_Kargar: <p>I don’t think you can use sigmoid without defining it. In this implementation, you don’t have sigmoid.</p>

nlam: <p>ok i see, then would this be a more correct impl?</p>
<pre><code class="lang-auto">class LogReg(nn.Module):
    def __init__(self):
        super(LogReg, self).__init__()
        self.lin1 = nn.Linear(4, 2)
        self.sig1 = nn.Sigmoid()
    
    def forward(self, x):
        x = self.lin1(x)
        y = self.sig1(x)
        return y
</code></pre>

Isaac_Kargar: <p>yes this is ok and should work.</p>

nlam: <p>ok great, thanks for the help, appreciate it!</p>
Why tensor value changes in floating point number

Mainul: <p>As the title says, for example,<br>
torch.tensor(0.83257929)</p>
<blockquote>
<blockquote>
<p>tensor(0.83257931)</p>
</blockquote>
</blockquote>
<p>why the value changes. how to stop this behaviour?</p>

KFrank: <p>Hello Mainul!</p>
<aside class="quote no-group quote-modified" data-username="Mainul" data-post="1" data-topic="81658" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mainul/48/30342_2.png" class="avatar"> Mainul:</div>
<blockquote>
<p>torch.tensor(0.83257929)</p>
<blockquote>
<blockquote>
<p>tensor(0.83257931)</p>
</blockquote>
</blockquote>
<p>why the value changes.</p>
</blockquote>
</aside>
<p>This is due to floating-point round-off error.  It can’t be avoided.<br>
If you <em>need</em> greater precision, create your tensor as<br>
<code>dtype = torch.double</code>.  Note, with double precision, you will<br>
still have round-off error, just less of it.</p>
<p>Best.</p>
<p>K. Frank</p>
Timing code using cProfiler or kernprof

s_n: <p>I have two functions (let’s say f1 and f2) that use inbuilt PyTorch functions like mul, div etc. Both these functions are called one after the other. I am using kernprof to time my code. I have used torch.cuda.synchronize() in my code.</p>
<p>If I time my code with kernprof (or cProfiler) it gives me output that says f1 takes 90 sec and f2 takes 0.5 sec. So I tried making f1 faster and now f1 takes 24secs but f2 starts taking 50secs.</p>
<p>Any idea why this would happen? both f1 and f2 are independent. It just that f1’s output is used as input to f2. But I don’t understand why making f1 faster will make f2 slower.</p>
How do tensor concat together in terms of adjacent intervals?

tianmaliuxing: <p>For example, here is a example, input are a and b, output is result.<br>
a = torch.tensor([[1, 2, 3, 4],<br>
[2, 3, 4, 5]])<br>
b = torch.tensor([[3, 4, 5, 6],<br>
[4, 5, 6, 7]])</p>
<p>result = torch.tensor([[1, 2, 3, 4],<br>
[3, 4, 5, 6],<br>
[2, 3, 4, 5],<br>
[4, 5, 6, 7]])</p>

jmandivarapu1: <aside class="quote no-group" data-username="tianmaliuxing" data-post="1" data-topic="81656">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/ecd19e/48.png" class="avatar"> tianmaliuxing:</div>
<blockquote>
<p>torch.tensor([[1, 2, 3, 4],<br>
[3, 4, 5, 6],<br>
[2, 3, 4, 5],<br>
[4, 5, 6, 7]])</p>
</blockquote>
</aside>
<pre><code class="lang-auto">torch.stack([a,b],dim=1).view(4,4)
</code></pre>
Training models in parallel and interactively

silvester: <p>I need to train two models in parallel. Each model has a different activation function with trainable parameters. I want to train model one and model two in the way that the parameters of the activation function from model one (e.g., alpha1) is separated from the parameters in model two (e.g., alpha2) by a gap of 2; i.e., |alpha_1 - alpha_2| &gt; 2. I wonder how I could include it into the loss function for training.</p>
FastRNNs Benchmark - custom lstms reverse

tsteffek: <p>I’m trying to write a custom LSTM with TorchScript. Naturally, my first impulse was to take the fastRNNs benchmark <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="noopener nofollow ugc">custom_lstms file</a> and to run it. However, it fails in both torch 1.4 and 1.5.</p>
<blockquote>
<p>RuntimeError: The following operation failed in the TorchScript interpreter.<br>
Traceback of TorchScript (most recent call last):<br>
File “C:/Users/anarc/git/Elmo-Training-PyTorch/elmo/modules/test.py”, line 298, in forward<br>
for rnn_layer in self.layers:<br>
state = states[i]<br>
output, out_state = rnn_layer(output, state)<br>
~~~~~~~~~ &lt;— HERE<br>
output_states += [out_state]<br>
i += 1<br>
File “C:/Users/anarc/git/Elmo-Training-PyTorch/elmo/modules/test.py”, line 238, in forward<br>
for direction in self.directions:<br>
state = states[i]<br>
out, out_state = direction(input, state)<br>
~~~~~~~~~ &lt;— HERE<br>
outputs += [out]<br>
output_states += [out_state]<br>
File “C:/Users/anarc/git/Elmo-Training-PyTorch/elmo/modules/test.py”, line 210, in forward<br>
def forward(self, input, state):<br>
# type: (Tensor, Tuple[Tensor, Tensor]) → Tuple[Tensor, Tuple[Tensor, Tensor]]<br>
inputs = reverse(input.unbind(0))<br>
~~~~~~~ &lt;— HERE<br>
outputs = jit.annotate(List[Tensor], <span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
for i in range(len(inputs)):<br>
File “C:/Users/anarc/git/Elmo-Training-PyTorch/elmo/modules/test.py”, line 90, in reverse<br>
def reverse(lst):<br>
# type: (List[Tensor]) → List[Tensor]<br>
return lst[::-1]<br>
~~~~~~~~ &lt;— HERE<br>
RuntimeError: invalid vector subscript</p>
</blockquote>
<p>To my understanding, negative strides are neither supported, nor on the roadmap (according to <a href="https://github.com/pytorch/pytorch/issues/229#issuecomment-548092342" rel="noopener nofollow ugc">this</a>). So this doesn’t surprise me, but why is that reverse function there then?</p>
<p><strong>Am I missing something, is there a trick to reversing a Tensor like that?</strong></p>
<p>Obviously, that would make creating a backwards layer easier (and probably faster than torch.flip, since it allocates new memory apparently)</p>
Modeling network output order

awa: <p>I have a network with two softmax output arrays A and B, for each of which I find the max. My problem has the constraint that max array B &gt;= max array A. At the moment, I account for this by replacing a value in B that doesn’t meet the constraint with the value of A. However, I’m wondering if this could be modeled in the network or in the loss, eg by adding an additional loss term driving up the total loss if the constraint is violated.</p>
<p>Ideas or pointers would be appreciated. Thanks!</p>
Is the cuda.amp.GradScaler capable of dealing with two differently scaled sets of gradients?

zhi_Li: <p>I found a case In <a href="https://pytorch.org/docs/master/notes/amp_examples.html#typical-mixed-precision-training" class="inline-onebox" rel="noopener nofollow ugc">CUDA Automatic Mixed Precision examples — PyTorch master documentation</a>.</p>
<blockquote>
<p>If your network has multiple losses, you must call [ <code>scaler.scale</code> ] on each of them individually.<br>
…<br>
scaler.scale(loss0).backward(retain_graph=True)<br>
scaler.scale(loss1).backward()</p>
</blockquote>
<p>It seems that the gradients back propogated from the two losses that scaled by different scaling parameters are just simply accumulated. Can the scaler unscales the two set of gradients with their corresponding scaling parameters when it comes to parameter updating?</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="zhi_Li" data-post="1" data-topic="81598">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhi_li/48/14873_2.png" class="avatar"> zhi_Li:</div>
<blockquote>
<p>It seems that the two losses that scaled by different scaling parameters are just simply accumulated.</p>
</blockquote>
</aside>
<p>The losses are not accumulated, as the <code>backward</code> call is used on each of the losses separately.</p>
<aside class="quote no-group" data-username="zhi_Li" data-post="1" data-topic="81598">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhi_li/48/14873_2.png" class="avatar"> zhi_Li:</div>
<blockquote>
<p>Can the scaler unscales the two set of gradients with their corresponding scaling parameters when it comes to parameter updating?</p>
</blockquote>
</aside>
<p>You can unscale the gradients implicitly using:</p>
<pre data-code-wrap="python"><code class="lang-python"># You can choose which optimizers receive explicit unscaling, if you
# want to inspect or modify the gradients of the params they own.
scaler.unscale_(optimizer0)
</code></pre>

zhi_Li: <p>Sorry, I have made some wrong expressions in the question. I mean the <strong>gradients</strong> back propagated from the two losses seems to be simply accumulated. Can the Scaler tells the two set of gradients apart, and unscales each of them properly for param. updating?</p>

ptrblck: <p>Ah OK, thanks for the follow-up.<br>
Yes, the scaler will unscale the gradients before accumulating them.</p>

zhi_Li: <p>Thank you for your kind help! This is a powerful feature.</p>
How to mimic larger batch size when only 1 example fits into memory

martinr: <p>Hi! I have a problem that one layer in my model takes up ca 6 GB of GPU RAM for forward pass, so I am unable to run batch sizes larger than 1 on my GPU.</p>
<p>Of course, I am not interested in running the model with batch_size 1 and looking how to improve.</p>
<p>I was thinking about “emulating” larger batch size. E.g., passing 10 single-example batches through the model and then call optimiser once (not after each batch-of-1). But how to do it? If I would do it manually, I would sum the gradients of 10 runs, divide by 10 and then use the result to update model parameters, right? But I am a bit clueless how to achieve the same thing using Pytorch built-in optimisers like SGD or Adam?</p>
<p>Or perhaps there is a better way to achieve the batch effect or even rework the layer for smaller memory footprint?</p>
<p>I am processing massive 3D data, hence the memory problem. My 3D convolution that causes high memory usage:</p>
<pre><code class="lang-auto">class My3DConv(nn.Module):

  def __init__(self):
    # input_size, output_size, kernel_size, stride, padding
    super().__init__()
    self.m1 = nn.Sequential(nn.Conv3d(128,64,3,stride=(2,1,1),padding=(1,1,1)), nn.BatchNorm3d(64), nn.ReLU())
    self.m2 = nn.Sequential(nn.Conv3d(64,64,3,stride=(1,1,1),padding=(0,1,1)), nn.BatchNorm3d(64), nn.ReLU())
    self.m3 = nn.Sequential(nn.Conv3d(64,64,3,stride=(2,1,1),padding=(1,1,1)), nn.BatchNorm3d(64), nn.ReLU())

  def forward(self, data):
    data = self.m1(data)
    data = self.m2(data)
    data = self.m3(data)
    return data
</code></pre>

ptrblck: <p><a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">This post</a> gives you some examples with advantages and shortcomings. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to get No. of images in every class by datasets.ImageFolder?

m.saber: <p>Assume that I have a dataset containing 5 classes of image files. And I use  “datasets.ImageFolder” as data loader. I know that I can get the total No. of images and No. of classes as follows:</p>
<pre><code class="lang-auto">train_folders = datasets.ImageFolder(train_path, train_transforms)
len(train_folders)  
len(train_folders.classes)
</code></pre>
<p>Is there any way that I can get the No. of image files in every Class?</p>
<p>Why do I need that?<br>
I want to use Class weight (= 1/ No. of images).<br>
But, I don’t want to enter No. of images or Class weight manually (hard coding).</p>
<hr>
<p>I have another question in this regard.<br>
When I run “train_folders[0]”, I get a tuple. Could you explain what is included in the tuple?</p>
<p>I noticed that when I run “train_folders[0][0]”, I get an image of dataset. But, I couldn’t understand the data structure of the mentioned tuple.</p>
<p>Thank you for your time.</p>

ptrblck: <p>You could access the <code>train_folder.targets</code> attribute and use <code>torch.unique(train_folder.targets, return_counts=True)</code> on it to get the number of samples for each target.</p>
Hierarchical softmax in pytorch

ruchitpatel: <p>I’m aware about the <code>softmax</code> function in pytorch. However, when using it, I run into computation complexity problems because of the normalising factor in the denominator in the softmax function. The reason is because of too many classes in my classification. I can not use negative sampling instead of softmax, because the variables in the normalising factor are chnged during the optimisation. We just can not ignore that normalising term.</p>
<p>How can I use hierarchical softmax instead, in order to approximate the softmax version. Thank you so much.</p>
Not able to import torch

Anuj_Singh: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86f05d7713c30c7579a50722f92a229e3c03c559.png" data-download-href="https://discuss.pytorch.org/uploads/default/86f05d7713c30c7579a50722f92a229e3c03c559" title="Error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86f05d7713c30c7579a50722f92a229e3c03c559.png" alt="Error" data-base62-sha1="jfIXzEwLYVLBAseukdVNXCruzPP" width="690" height="123" data-dominant-color="0B0B0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Error</span><span class="informations">1012×181 7.66 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have installed pytorch in my conda environment using:<br>
<strong>conda install pytorch -c pytorch</strong><br>
but while importing the package, it is showing the error.</p>
<p>Figures are attached as reference.<br>
Kindly help.</p>

ptrblck: <p>If you don’t have a GPU in your system, you would need to install the CPU packaged as explained <a href="https://discuss.pytorch.org/t/cannot-import-torch-on-jupyter-notebook/79334/3">here</a>.</p>
Skip time-steps in a trained lstm

Isaac_Kargar: <p>Hi,<br>
I have trained one LSTM network with sequential data. For example, the data is like this:</p>
<pre><code class="lang-auto">d0, d1, d2, d3, d4, d5, d6, ...
</code></pre>
<p>In test time I want to use the trained model and feed data like this to it:</p>
<pre><code class="lang-auto">d0, d2, d4, d6, ...
</code></pre>
<p>Is there any way to do this? I tested to feed zero or <code>d0, d0, d2, d2, d4, d4, d6, d6,...</code> sequence but it doesn’t seem to work.</p>

ptrblck: <p>If you want to skip every second sample, you could just slice the input tensor in the time dimension via <code>[::2]</code>, no?</p>

Isaac_Kargar: <p>I want to use all data in training, but in test I just want to use every second samples only. But when I do this, outputs of lstm are not good. It seems the LSTM hidden state needs to be updated using both data not just second one. I’m looking for a trick to make it work, in training or in test phase. I’m thinking of input dropout or sth like that.</p>
StepLR learing rate calculation error when step_size = 1

AlvinZheng: <p>I run the following code:</p>
<pre><code class="lang-auto">import torch
import torchvision

net = torchvision.models.resnet18()
optimizer = torch.optim.SGD(net.parameters(), lr=1)
lr_sche = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1)
data = torch.rand(1, 3, 224, 224)
for epoch in range(5):
        optimizer.step()
        print('Epoch', epoch)
        print(lr_sche.get_lr())
        lr_sche.step()
        print(lr_sche.get_lr())
</code></pre>
<p>and get the output:</p>
<pre><code class="lang-auto">Epoch 0
[1]
[0.010000000000000002]
Epoch 1
[0.010000000000000002]
[0.0010000000000000002]
Epoch 2
[0.0010000000000000002]
[0.00010000000000000003]
Epoch 3
[0.00010000000000000003]
[1.0000000000000004e-05]
Epoch 4
[1.0000000000000004e-05]
[1.0000000000000004e-06]
</code></pre>
<p>I am confused why the “lr” jumped directly from 1 to 0.01? Is this a bug or I used the StepLR in the wrong way?</p>

ptrblck: <p>You should get a warning stating that you should use <code>get_last_lr()</code> instead of <code>get_lr()</code>:</p>
<blockquote>
<p>UserWarning: To get the last learning rate computed by the scheduler, please use <code>get_last_lr()</code>.</p>
</blockquote>
<p>This should give you the expected learning rates:</p>
<pre data-code-wrap="python"><code class="lang-python">for epoch in range(5):
        optimizer.step()
        print('Epoch', epoch)
        print(lr_sche.get_last_lr())
        lr_sche.step()
        print(lr_sche.get_last_lr())

&gt; Epoch 0
[1]
[0.1]
Epoch 1
[0.1]
[0.010000000000000002]
Epoch 2
[0.010000000000000002]
[0.0010000000000000002]
Epoch 3
[0.0010000000000000002]
[0.00010000000000000003]
Epoch 4
[0.00010000000000000003]
[1.0000000000000004e-05]
</code></pre>
Nlp torch implementation for tf sample

ndrs: <p>Novice here. I am trying to implement convnet on top of huggingface bert-base-uncased, and I am using a tensorflow example. The original model looks something like:</p>
<p>tf_roberta_model (TFRobertaMode ((None, 96, 768), (N 124645632   input_1[0][0]</p>
<hr>
<p>dropout_38 (Dropout)            (None, 96, 768)      0           tf_roberta_model[0][0]</p>
<hr>
<hr>
<p>conv1d (Conv1D)                 (None, 96, 128)      196736      dropout_38[0][0]</p>
<hr>
<p>I have a couple questions:</p>
<ol>
<li>how can an input shape of (None, 96, 768) after being applied: tf.keras.layers.Conv1D(128, 2,padding=‘same’)(x1) output a shape of  (None, 96, 128) ?</li>
<li>I’m struggling to implement the same as above in pytorch, and so far I got it to run (albeit does no good) as follows: nn.Conv1d(in_channels=128, out_channels=1,kernel_size=2). What am I doing wrong?</li>
</ol>
<p>Thanks in advance!!!</p>

ptrblck: <ol>
<li>
<p>Assuming that TF uses the shape <code>[batch_size=None, sequence_length=96, channels=768]</code>, the specified conv layer should yield the same temporal size with 128 output channels, which would correspond to <code>[None, 96, 128]</code>.</p>
</li>
<li>
<p>The input channels seem to be 768, while the output channels seem to be specified as 128. Also, for a kernel size of 2 would would need to add padding to get the same sequence length.</p>
</li>
</ol>
Error while printing a random sample from a dataLoader object containing videos

costa98: <p>Hello,<br>
I created a data loader of the ucf101 video dataset</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.UCF101('/content/drive/My Drive/Colab Notebooks/ucf101', annotation_path='/content/drive/My Drive/Colab Notebooks/ucfTrainTestlist', frames_per_clip=16, step_between_clips=1, frame_rate=None, fold=1, train=True, transform=transforms.Compose([transforms.ToTensor(),]))
data = torch.utils.data.DataLoader(dataset, batch_size=512, shuffle=True)
</code></pre>
<p>I want to see the shape and a full tensor of a random sample of this object, I tryed using</p>
<pre><code class="lang-auto">dataiter = iter(data)
for i in dataiter:
  print(i)
  print(i.shape())
</code></pre>
<p>but I get the error</p>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-32-daa759a0394f&gt; in &lt;module&gt;()
      1 dataiter = iter(data)
----&gt; 2 for i in dataiter:
      3   print(i)
      4   print(i.shape())

7 frames
/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py in to_tensor(pic)
     40     """
     41     if not(_is_pil_image(pic) or _is_numpy(pic)):
---&gt; 42         raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
     43 
     44     if _is_numpy(pic) and not _is_numpy_image(pic):

TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;
</code></pre>

ptrblck: <p>This dataset should already return tensors:</p>
<blockquote>
<p>Returns:<br>
video (Tensor[T, H, W, C]): the <code>T</code> video frames<br>
audio(Tensor[K, L]): the audio frames, where <code>K</code> is the number of channels and <code>L</code> is the number of points<br>
label (int): class of the video clip</p>
</blockquote>
<p>So I guess you would have to remove the <code>ToTensor</code> transformation.</p>
Anatomy of PyTorch

111137: <p>Hi,</p>
<p>I made directory structure of PyTorch (I want to learn framework development);<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/IAMAl/Anatomy_of_PyTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/52284359?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/IAMAl/Anatomy_of_PyTorch" target="_blank" rel="nofollow noopener">IAMAl/Anatomy_of_PyTorch</a></h3>

<p>Contribute to IAMAl/Anatomy_of_PyTorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Best,<br>
S.Takano</p>
[Help] RuntimeError: CUDA error: device-side assert triggered

yf_zhang: <p>I meet cuda Error when I tried to do slice operation.</p>
<pre><code class="lang-auto">            try:
                sub_x = [xx[sub_valid_indices, ...] for xx in x]
            except Exception as e:
                print("### x.shape={}, sub_valid_indices={}".format([xx.shape for xx in x], sub_valid_indices))
                exit(1)
</code></pre>
<p>The output log is “x.shape=[torch.Size([2, 3, 32, 32])], sub_valid_indices=[0, 1]”<br>
It seems that the indice is not output of range.</p>

ptrblck: <p>Is your code running fine on the CPU? Generally, it should give you a better error message.<br>
If so, could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the complete error message here?</p>
Repeat Tensor Values Along an axis in Groups of 3 (n)

Rafael_R: <p>Hi,</p>
<p>I wanted to do an operation like this:</p>
<blockquote>
<p>[ 1 2 3 4 5 6 7 8 9 ] → [1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9]</p>
</blockquote>
<p>How can I do this in torch?</p>
<p>lets, assume the given tensor, <code>t = torch.tensor([ 1 2 3 4 5 6 7 8 9 ])</code></p>

Rafael_R: <p>Use repeat_interleave e.g. if d has shape [1, 3] use <code>torch.repeat_interleave(d, 3, dim=1)</code></p>
Entrywised addition

lifeblack: <p>Hi<br>
‘+’ denotes entry_wised addition.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c499fe0cc8c06cdfbcaa572e2c829d02778e9612.png" alt="image" data-base62-sha1="s3dvHGQNIie7EivJDYkWR5ZVwRk" width="343" height="253"><br>
I suppose to code it like bellow? I know it is a silly question but I kinda doubt about it</p>
<pre><code class="lang-auto">a=torch.rand([16,32,56,7])
b=torch.rand([16,32,56,7])
out=a-b
</code></pre>
Adding additional features to an embedded padded sequence

bash94: <p>Hello everyone,</p>
<p>I would like to add additional features to an embedded padded sequence before I pass the data to an RNN as a packed sequence.</p>
<p>For example:</p>
<p>Let’s say I have the following embedding layer and a simple bidirectional RNN:</p>
<p><code>embedding = nn.Embedding(5, 5, padding_idx=0)</code><br>
<code>rnn = nn.RNN(5, 5, batch_first=True, bidirectional=True) </code></p>
<p>and let’s say I have the following input:</p>
<p><code>input = torch.tensor([[1,2,3,4,0], [1,2,3,0,0]], dtype=torch.long)</code><br>
<code>lengths = torch.tensor([4, 3], dtype=torch.long)</code></p>
<p>What should be done if I want pass a packed sequence to the RNN is the following:</p>
<p><code>embedded_seqs = embedding(input, padding_idx=0)</code><br>
<code>packed_seqs = pack_padded_sequence(embedded_seqs, lengths, batch_first=True)</code><br>
<code>packed_output, hidd = rnn(packed_seqs)</code></p>
<p>Now let’s say I have additional features that I would like to add to <code>embedded_seqs</code> before I create <code>packed_seqs</code>:</p>
<p><code>additional_features = torch.randn(2, 5, 5)</code></p>
<p>What is the best way to do this? Should I do something like this:</p>
<p><code>embedded_seqs = embedding(input, padding_idx=0)</code><br>
<code>all_features = torch.cat((embedded_seqs, additional_features), dim=2)</code><br>
<code>packed_seqs = pack_padded_sequence(all_features, lengths, batch_first=True)</code><br>
<code>packed_output, hidd = rnn(packed_seqs)</code></p>
<p>I think if I do the above, all the features that are going to be added after the padding will be neglected. Is there a better way to do this? Thank you so much!</p>