akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dataloader slower than normal execution?

gauthierb: <p>Hi,<br>
I am trying to retrieve a sequence of bounding boxes to train an RNN, to do so I defined a Dataloader like this : <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37acea338c157b35f4c4e87f748c1cac7c1009df.png" data-download-href="https://discuss.pytorch.org/uploads/default/37acea338c157b35f4c4e87f748c1cac7c1009df" title="Screenshot 2020-10-16 at 22.13.29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37acea338c157b35f4c4e87f748c1cac7c1009df_2_690x243.png" alt="Screenshot 2020-10-16 at 22.13.29" data-base62-sha1="7WwJd9YtuQlycRgtLKvlkIRE7b9" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37acea338c157b35f4c4e87f748c1cac7c1009df_2_690x243.png, https://discuss.pytorch.org/uploads/default/original/3X/3/7/37acea338c157b35f4c4e87f748c1cac7c1009df.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/7/37acea338c157b35f4c4e87f748c1cac7c1009df.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-10-16 at 22.13.29</span><span class="informations">894×316 32.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I load a sequence of tensors, where tensors are the bounding boxes that have been resized to fit  VGG16, from memory which is a quick operation so I thought that doing a Dataloader like this would be quicker than loading an image then cropping it to make a bounding box than resizing it for every call to <strong>get_item</strong>. The problem is that for some reason getting a batch takes approx. 10 seconds with num_workers =1 and ±2 minute with num_worker =10.<br>
So I tried to manually make a batch by doing :</p>
<p>batch = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(64):<br>
det_frame_list = traindata[i]<br>
label = trainlabel[i]<br>
bbs = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for j in range(len(det_frame_list)):<br>
bbs.append(torch.load(det_frame_list[j]))<br>
tensor_ret = torch.stack(bbs)<br>
batch +=[tensor_ret]<br>
batch = torch.stack(batch)</p>
<p>(don’t know how to use BBCode sorry!)</p>
<p>to see if my approach was slow but it instantly finishes with good output.<br>
Since I am new to Pytorch I can’t figure out why this (Dataloader taking ages to make a batch) is happening, any help understanding this would be greatly appreciated!<br>
Thanks</p>
How to apply dropout to drop an entire vector

smani: <p>I am concatenating two vectors, say A and B,  into a single vector. Each of them have a dimension of 128. When training I would like randomly drop one of them completely. How can I do that?</p>

a_d: <p>You can simply define a dropout layer and pass the tensor to it. For example -</p>
<pre><code class="lang-auto">a = torch.rand(1, 5)
print(a) # output is tensor([[0.3119, 0.1485, 0.6420, 0.4604, 0.0724]])
dropout = torch.nn.Dropout(1)
print(dropout(a))
# output is tensor([[0., 0., 0., 0., 0.]])

</code></pre>

smani: <p><a class="mention" href="/u/a_d">@a_d</a> But it always drops that vector.</p>

a_d: <p>Yes the dropout layer will always make everything zero as it is set to 1.<br>
To randomly drop off the layer, generate a random number between 0 and 1 and if the number is greater than the threshold drop it</p>
<pre><code class="lang-auto">prob  = random.uniform(0, 1)
if prob &gt; threshold:
    b = dropout(b)
</code></pre>
<p>Then you can concatenate the two vectors.</p>

ptrblck: <p>If you want to stack these tensors together, you could use something like this:</p>
<pre><code class="lang-python">a = torch.randn(128)
b = torch.randn(128)
c = torch.stack((a, b)) * torch.randperm(2).unsqueeze(1)
</code></pre>
<p>which would zero out one of these tensors completely.<br>
If that’s not your use case, could you explain the shapes a bit more?</p>
.grad showing none. How to ensure flow of gradients?

Shalabh_Gupta: <p>class delft_block(nn.Module):</p>
<p>def <strong>init</strong>(self,input_dimensions,k):</p>
<pre><code>super(delft_block,self).__init__()

#self.conv1 = nn.Conv2d(input_dim, 512, kernel_size=kernel_size, padding=padding, stride=stride)

#self.activate = nn.ReLU()

self.k = k

self.inp = input_dimensions

#self.conv1 = nn.Conv2d(input_dimensions,512,kernel_size=3,padding=1)

self.conv2 = nn.Conv2d(input_dimensions,1, kernel_size=1, padding=0, stride=1)
</code></pre>
<p>def forward(self,x):                                          <span class="hashtag">#x</span> = b,512,8,8</p>
<pre><code>b,c,h,w = x.size()

out = self.conv2(x)                                         #x = b,1,8,8

prob = nn.Softplus()(out)

att = F.softmax(prob.view(b,-1),dim=1)                      #b,8*8(pixel wise attention scores flattened)

val,indices = torch.topk(att,self.k)                        # finding pixel indices with top k attention scores
</code></pre>
<h1>if phase == ‘train’:</h1>
<pre><code>if att.requires_grad:

    att.retain_grad()

ind_exp = indices.unsqueeze(-1).expand(b,self.k,self.inp)             



l_perm = x.permute(0,2,3,1)                                 #l_perm = b,8,8,512

l_perm = l_perm.reshape(b,h*w,self.inp)                     #l_perm = b,h*w,512

feat = torch.gather(l_perm,1,ind_exp)                       #feat = b,k,512 using indices found earlier to extract feature maps[Should work now]

feat = feat.permute(0,2,1)

return feat                                                 #feat = b,512,k
</code></pre>
<p>Hi! I have written the above code. I am basically selecting some indices from att tensor having highest values using topk function, and then using those indices to select features feat from l_perm tensor. I have also used att.retain_grad(). Now while training after one round of loss.backward() when I print .grad for delft.conv, it shows None. I think my network has problem in flowing gradients due to the index selection part. Can someone please let me know the problem as well as the solution to this? Thanks.</p>

Shalabh_Gupta: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please help? I am stuck because of this.</p>
Transformer-Encoder for a sentence encoding instead of LSTM

NadiaMe: <p>What could be used to replace LSTM sentence encoding with the Transformer model?</p>
<p>Currently I am using nn.LSTM that accepts packed sequence that is constructed from an input<br>
(batch_size, max_seq_len, embed1) = (128, 20, 1024)</p>
<p>and it outputs<br>
(1, batch_size, embed2) = (1, 128, 2048)</p>
<p>E.g we learned a single embedding from multiple embeddings max_seq_len.</p>
<p>How can I replace LSTM with Transformer to achieve the same results with exactly the same input?<br>
E.g input is a tensor of size (128, 20, 1024) and the output is a tensor of size (1, 128, 2048).</p>
<p>Can I achieve this with nn.Tranformer or nn.TransformerEncoder?<br>
So far I could not figure out how to achieve this with these classes.</p>
<p>Thank you</p>
Saving a model in pytorch

lifeblack: <p>Hi I want to save my model during the training. My model has 9 conv layers with batch norm and softmax as activation function. I use this method to save my training modeel but when I resumed my training I sensed that it resumed from begining.</p>
<pre><code class="lang-auto">state = {
    'epoch': epoch,
    'state_dict': model.state_dict(),
    'optimizer': optimizer.state_dict(),
}
    torch.save(state, '/home/superblock/state_train.pt')
</code></pre>
<pre><code class="lang-auto">  state = torch.load('/home/superblock/state_train.pt')
  model.load_state_dict(state['state_dict'])
  optimizer.load_state_dict(state['optimizer'])
</code></pre>
<p>what should I do?<br>
does it need model.eval()? I read some where that for resuming  training we dont use that<br>
and I have another question… It’s better to save when loss is fewer in validation data or train data?</p>

a_d: <p>model.eval() changes the behaviour of a few layers like batchnorm and dropout. So if you want to infer using your model then put the model in .eval() mode or if you want to continue training then put the model into training using model.train().<br>
Save at the least validation loss</p>
How to create histogram for all the feature maps of a CNN convolution layer

Borhan: <p>Let’s say I have a pretrained model (e.g Alexnet).  I want to generate a histogram for all the feature maps from a convolution layer ( say 64 feature maps for convo layer 1) such that each feature map become a bin of the histogram ( In this case 64 bins). Any idea how to do that?</p>

a_d: <p>This thread might help you - <a href="https://discuss.pytorch.org/t/visualize-feature-map/29597/7">here</a></p>
Individual nn.lienar parameter for non input axis dimension?

peepeepoopoo: <p>Not sure if this is a feature but is it possible to make individual linear parameter for each input along the axis that is not the input channel? say an Nx4 matrix fed to an nn.linear layer of size 4 input and 4 for output, I would have a parameter size of Nx4 for bias and Nx4x4 weight instead of 4 and 4x4. (I know I can do this manually by setting up my own parameters but is there a way to do so in nn.linear?)</p>

peepeepoopoo: <p>just do it the way you think how it might work</p>
requires_grad=False makes Volatile GPU-Util go up. Why?

Roy_Paik: <p>I am using LSTM to make Language Model and I found out freezing embedding weight makes <code> Volatile GPU-Util</code> go up. (30% -&gt; 80%). I don’t get it why it happens. Would you give me some advice about the issue(?)?</p>
<pre><code class="lang-auto">weight_vec = torch.load('./data/pretrained_embedding.pt')
model.emb.weight.data.copy_(weight_vec)
model.emb.weight.requires_grad = False
</code></pre>

albanD: <p>Hi,</p>
<p>This might be because the autograd is not running, so the the CPU has less work to do and can send work to the GPU faster?</p>

Roy_Paik: <p>I see. I asked it just out of curiosity <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
So, CPU is supposed to work on autograd even in cuda(gpu) mode? I didn’t know that. Many thanks.</p>
Can we let the model learn the weights sent to the crossEntropyLoss

emad: <p>I have a multi-class classification problem (5 classes), and the data is imbalanced. So I was thinking if we can let the model learn the best weights by making the “weights” option in CrossEntropyLoss  learnable parameters.</p>
<p>Is this a valid assumption or not?</p>

KFrank: <p>Hello Emad!</p>
<aside class="quote no-group" data-username="emad" data-post="1" data-topic="99575" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/ac91a4/48.png" class="avatar"> emad:</div>
<blockquote>
<p>So I was thinking if we can let the model learn the best weights by making the “weights” option in CrossEntropyLoss  learnable parameters.</p>
</blockquote>
</aside>
<p>First, you would not want to.  Consider what would happen if you<br>
tried.  Your optimizer would drive your loss to zero simply by driving<br>
all of the learnable class weights to zero (with <code>reduction = 'sum'</code>)<br>
or drive only one class to have a non-zero weight, and drive your<br>
model to predict only that class (with <code>reduction = 'mean'</code>).</p>
<p>Second, as written, pytorch’s <code>CrossEntropyLoss</code> doesn’t support<br>
calculating gradients with respect to the class weights.  (You could,<br>
of course, write your own version of <code>CrossEntropyLoss</code> that did,<br>
but, in line with my first comment, you wouldn’t want to.)</p>
<p>Best.</p>
<p>K. Frank</p>
What should be the input size of first linear layer?

YASH-ROCKY: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bd1a4c0f31b86d9b7be171ccc3f78866e1e283b.png" data-download-href="https://discuss.pytorch.org/uploads/default/4bd1a4c0f31b86d9b7be171ccc3f78866e1e283b" title="pytorchhh"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bd1a4c0f31b86d9b7be171ccc3f78866e1e283b.png" alt="pytorchhh" data-base62-sha1="aOIY6Z5EFQL7llSCleXQYx0bMZJ" width="690" height="321" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorchhh</span><span class="informations">888×414 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i am getting error - size mismatch, m1: [10 x 32], m2: [320 x 564] at C:\cb\pytorch_1000000000000\work\aten\src\TH/generic/THTensorMath.cpp:41</p>

fadetoblack: <p>Can you try</p>
<pre><code class="lang-auto">self.fc1 = nn.Linear(32,564)
</code></pre>

YASH-ROCKY: <p>thanks for the answer, but i have one doubt since my batch size is 10 then does the images get loaded one by one or simultaneously</p>

ptrblck: <p>The complete input batch will be used and you don’t have to specify the batch size in any layer arguments, such as the number of input features etc.</p>
Question about whether this operation will be differentiable

ChrisLiu2: <p>I’m working on a graph neural network with dgl library. What makes me uncertain is that at one point, I calculate a pairwise attention using a simple MLP between adjacent node, and create a new node in the graph if a certain pair has scores higher than some threshold. This graph will be used in the next step to do graph convolution. I don’t think I can write this operation in closed form, and thus I’m not sure if pytorch can differentiate this either.</p>

albanD: <p>Hi,</p>
<p>I think it depends what you want to differentiate.<br>
If you generate a new embedding for that new node and add it to the current graph state before further processing. I think this can be nicely differentiable.<br>
If you want gradients for things like what this node should be connected to or added at all, it is going to be trickier as these quantities are usually discrete.</p>

ChrisLiu2: <p>Thank you for the quick reply. Does “tricky” in this scenario mean you can’t learn anything in the MLP? Do you have anything for me to read to possibly address this? Still huge thanks if there isn’t any, this seems like a tough question.</p>
<p>What I have on top of my head is I can probably train the MLP separately before actually training other parts of the network, using some carefully designed loss?</p>

albanD: <aside class="quote no-group" data-username="ChrisLiu2" data-post="3" data-topic="99549">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chrisliu2/48/45373_2.png" class="avatar"> ChrisLiu2:</div>
<blockquote>
<p>Does “tricky” in this scenario mean you can’t learn anything in the MLP?</p>
</blockquote>
</aside>
<p>You cannot define gradients for discrete quantities. So tricky here mean not possible <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>A trick that can be used to get around this is to make these quantities continuous. But this is not always possible and can have very weird interpretation depending on the application. For the connectivity here for example, I’m not sure what it would mean to have an adjacency matrix with continuous values?</p>
How is feature extraction done from by using pre-activations last CNN layer in VGG-19

Akhil_Koilada: <p>I am trying to extract features using the pre-activations of last CNN layer in VGG-19.Is it just extracting features before the ReLU layer or do I have to make any changes in the CNN model…?</p>

ptrblck: <p>You could use forward hooks to get the activations without changing the model as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>

Akhil_Koilada: <p>Thank you…It worked out…</p>
How to use the outputs of a deep lstm layer?

berkay_berabi: <p>Hi all,</p>
<p>I am trying to build a seq2seq attention-based model. In the literature, people often use deep lstm layers and each layer in this case produces a hidden state which is passed to the next one. If I use N layers, at the end I have N hidden states. What is the common approach to work with these ?</p>
<p>One can add all N and then create one hidden state, or I can concatenate them or I can just consider the outputs from the very last layer. I know that all of them are possibly useful approaches but what is the most common way of handling this ?</p>

Andrea_Rosasco: <p>I don’t know what’s the most used alternative in the literature but as you said they are all viable option. The best thing you could do would be to test all of them and see which gives you the best results. My money is on the concatenated hidden layers approach.</p>
Debug Assertion Failed! Expression: __acrt_first_block == header when using libtorch debug build

Shisho_Sama: <p>I built libtorch 1.6 from source (with /MTd switch). Everything went smoothly!<br>
but upon executing the test application it crashes with this error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b407f71a99dcea21b2ee5748c1fb4ef498e35d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/3b407f71a99dcea21b2ee5748c1fb4ef498e35d8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b407f71a99dcea21b2ee5748c1fb4ef498e35d8_2_690x372.png" alt="image" data-base62-sha1="8sam7dnY1yMN6fDyxxFaCEw0skg" width="690" height="372" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b407f71a99dcea21b2ee5748c1fb4ef498e35d8_2_690x372.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b407f71a99dcea21b2ee5748c1fb4ef498e35d8_2_1035x558.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b407f71a99dcea21b2ee5748c1fb4ef498e35d8_2_1380x744.png 2x" data-dominant-color="3D3E3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1600×864 77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>you can see the whole source code so its just a simple hello world<br>
Whats wrong here? Any help is greatly appreciated</p>
<p>note: I’m building using Visual Studio 2019</p>

glaringlee: <p><a class="mention" href="/u/shisho_sama">@Shisho_Sama</a><br>
Hi, there is an issue here, not fully related, but you can take a look at the last post.<br>
‘<a href="https://github.com/pytorch/pytorch/issues/46318" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/46318</a>’<br>
If it doesn’t work, please create an issue at ‘<a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues</a>’, we will have build expert looking into this.</p>
Backpropagating loss through model after purposefully breaking graph

btil: <p>Is there any way I can get the scalar loss to backprop through the model even when the loss is calculated with tensors not associated with the model? Custom autograd function? Currently I am getting None gradients for the model.</p>
<pre><code class="lang-auto">im1 = torch.tensor(..., requires_grad=True) # image 
im2 = torch.tensor(..., requires_grad=True) # image 2
im3 = torch.tensor(..., requires_grad=True) # image 3

y = model(im1)

# do ops on im2 based on y that requires breaking grad for y

loss = criterion(im2, im3).mean() # scalar
loss.backward()

params = list(model.parameters())[0]                                       
print(params.grad)  # None
</code></pre>
<p>If I multiply the loss as follows then I do get gradients for the model:</p>
<pre><code class="lang-auto">loss = criterion(im2, im3).mean() * y.mean() / y.mean()
</code></pre>
<p>Is multiplying like this a valid way to get the gradients to flow through the model?</p>

albanD: <p>Hi,</p>
<p>If you break the graph, you won’t be able to use the autograd I’m afraid.<br>
The only way to is to wrap these ops that break the graph in a custom autograd.Function and write the backward for that op yourself: <a href="https://pytorch.org/docs/stable/notes/extending.html">https://pytorch.org/docs/stable/notes/extending.html</a></p>

btil: <p>Thanks! I think I understand the basics of custom autograd.Function, but fail to see how using it is a workaround to the gradients flowing backwards natively?</p>

albanD: <p>The custom function allows you to overwrite the backward pass and thus replace the whatever the autograd would have done for what is in the forward.</p>

btil: <p>So I could reconnect the broken graph with this method?</p>

albanD: <p>More or less.<br>
As would see it as  hiding from the autograd the fact that you break the graph <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Profiler tutorials

Mughees: <p>Hi,</p>
<p>Is there any good video tutorial for PyTorch.Profilers as currently I am lost in between Profilers and utils.bottlenecks. I read all the discussion questions here mentioning profilers but could not get a good starting point as its my first time diving in this topic. Currently there is just <a href="https://www.youtube.com/watch?v=TKdqp56Ja00&amp;t=660s&amp;ab_channel=PyTorchKR" rel="noopener nofollow ugc">this</a> one on youtube as per my search but thats also in korean without any subtitles.</p>
<p>I have seen the written tutorials but they were not much explanatory for the beginner.</p>
<p>Thanks.</p>

ilia-cher: <p>Hi, have you seen our profiler recipe doc? <a href="https://pytorch.org/tutorials/recipes/recipes/profiler.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/profiler.html</a> ;<br>
as for videos - please stay tuned and make sure to check our upcoming virtual developer day in November with many videos and tutorials, we plan to have one for profiler too.</p>
How to use textual, numerical and categorical features together

l1b3rty: <p>It’s been a bit time for me to look for an example of using textual, numerical and categorical features together but I couldn’t find one. It would be nice to see a concrete example. Can you help with that?</p>

a_d: <p>Hello,<br>
Firstly, it actually depends on how this data is correlated, and understanding that would be the first step incorporating this data in your model. You may want to introduce some data in the later layers of your model and sometimes you can pass the data to your model all at once.</p>
<p>Secondly, you might want to extract features from your data, and doing so from categorical data is tough, especially because one hot encoded representations of such data would result in sparse tensors, and applying normal layers of such data would yield in meaningless results. I generally apply one of the two strategies, One is such that you would use an embedding layer for one hot encoded representations of each categorical field and then concatenate them.  Another way is to use sparse convolutions and sparse linear layers. Nvidia has a package called <code>Minkowski Engine</code> which has sparse implementations of the convolutional operations. You might want to go through the theory first to see which fits your case better.</p>
<p>To load the text data, load them as the pretrained vector representations of the words. Is it a standard dataset with which you are working ?</p>

a_d: <p>My previous reply answered the way you could include categorical data in your model.<br>
Your processing layers will be a part of your model as they would be trainable.<br>
Could you please be more specific as to what exactly you are looking for? Where are you getting stuck? and maybe with some dummy or a single datapoint from your dataset.</p>

a_d: <p>Gimme some time, I will fo through it and get back to you…</p>

a_d: <p>Not yet, got busy a little, but I assume your main problem converting textual data to tensors? Am I correct?</p>

a_d: <p>Hello ,<br>
I understand the issue now. Will I be right to assume that the numbers would denote some weightage to the text? In that case, simple element-wise of these weights(i.e. your numbers) to the vectorized text. Combining these three pieces of information might be done through some operation which relates them. This might be the way to go.</p>

a_d: <p>Okay so you have many nodes containing  data(like a graph or tree)…<br>
Well given you have text, you would be padding them to make them the same length. You might as well pad these numbers with zeros to make them the same length as your sentence, and concat them with the embeddings.<br>
On a side node, if your data is structured like a tree, have you tried out graph networks and  structured learning ?</p>

a_d: <p>Happy to help <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
Way to process count data without normalization

sshen82: <p>Right now, the data I am trying to analyze is a set of graphs (shown as a lot of weighted adjacency matrices. Each value in the adjacency matrices means the “strengh of connection” between the two vertices and there are large count value on the diagonal as well larger than 200.) I try to use different kinds of VAE to get latent variable z to cluster those matrices, but right now the loss is nan in the very beginning. Using models like VGAE (variational graph autoencoder), it can’t even run for one epoch and nan will appear on parameters; Using Convolutional VAE, it is just nans (the code for CVAE is at here: <a href="https://github.com/sshen82/cvae/blob/main/cvae.py" rel="noopener nofollow ugc">https://github.com/sshen82/cvae/blob/main/cvae.py</a>). Since the value in adjacency matrices are Poisson counts and they have actual meanings, so I don’t want to simply normalize them to 0-1. I do know that there are methods (for example, scvi working on scRNAseq. Each value is gene expression count value, so they are definitely larger than 1. <a href="https://github.com/YosefLab/scvi-tools" rel="noopener nofollow ugc">https://github.com/YosefLab/scvi-tools</a>) directly working on counts, but don’t understand what they did to overcome the problem. Is this because only Graph VAE and Convolutional VAE are sensitive on count values, or is this what they did that successfully overcome the problem?</p>

googlebot: <p>Try replacing final ReLU activation (in decoder) with softplus.<br>
Place a breakpoint at F.poisson_nll_loss line and check if network output ranges are sensible.<br>
Use torch.autograd.set_detect_anomaly(True) to find where NaNs appear.</p>

sshen82: <p>Using softplus actually worked, thank you!</p>
Online Knowledge Distillation

achyut_mani_Tripathi: <p>Dear All,</p>
<p>I hope you all are doing well and safe in a current COVID-19 Pandemic. I am very new to computer vision and currently working on a knowledge distillation problem. I am implementing knowledge distillation using mutual learning, in which four CNN models (M1, M2, M3, M4) are trained simultaneously. All models are running and computing the loss. The losses of all four models are added and the total sum is backpropagated to update the weights of the four models. I want to create a new CNN model M5  that takes feature maps (just before the FC layers of M2 and M3) of models M2 and M3 as an input. The model M5 has one CNN layer, a Batch Norm Layer, an fully connected layer, and an output layer.<br>
Could you please help me to know how to extract the feature maps from models M2 and M3 and forward them to model M5 during training so that now I can train all five models simultaneously?. The models M1, M2, M3, M4 are using the pre-trained Resnet-34 model with the number of classes=10. An example demo code will be highly helpful that illustrates how model M5 takes inputs from models M2 and M3 and forwarding them.</p>
<p>Thanks<br>
Achyut</p>

ptrblck: <p>You could register forward hooks in <code>M2</code> and <code>M3</code> in the penultimate layer as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
<p>During the forward pass of these models the hooks will be activated and the outputs will be stored e.g. in a <code>dict</code>. Once this is done you can pass the activations from the <code>dict</code> to <code>M5</code> and continue the training.</p>
<p>Depending if you want to calculate the gradients of the loss w.r.t. <code>M2</code> and <code>M3</code> you could either store the intermediate activations directly or <code>detach()</code> them in the forward hook.</p>

achyut_mani_Tripathi: <p>Thanks a lot ptrblck</p>
Resume training with saved model

snowe: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a general question regarding saving and loading models in PyTorch.</p>
<p>My case:</p>
<ol>
<li>I save a checkpoint consisting of the model.state_dict, optimizer.state_dict, and the last epoch.</li>
<li>The saved checkpoint refers to the best performing model, evaluated by accuracy.</li>
<li>I load all the three checkpoint entries and resume…However, I do not want to continue training but I want to use the saved state and make one forward pass to get the same accuracy as I had when I saved the checkpoint. How can I do that?</li>
</ol>
<p>Basically, I want to be able to reproduce my results, since I have not figured out how to seed in PyTorch. It somehow does not really work…So I figured I can do it by saving and loading models.</p>
<p>Any help is very much appreciated!</p>
<p>All the best,<br>
snowe</p>

Caruso: <p>Hi,</p>
<p>first of all <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">here</a> you can find a short documentation about reproducibility in PyTorch.<br>
I think your results could be reproduced, if you’re able to load the same data as in the original evaluation (e.g. your validation dataset) and you haven’t used any random operations in your first evaluation, like Dropout as an example.<br>
How different are your current results from the original one?</p>

snowe: <p>Hi <a class="mention" href="/u/caruso">@Caruso</a>,</p>
<p>thank you for reaching out! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I am aware of the different seeding functions in PyTorch and I have used them. Somehow my results differ in the range of roughly 1 - 2%.  My model does not use Dropout…</p>
<p>All the best,<br>
snowe</p>

snowe: <p><strong>UPDATE</strong><br>
I was able to make my results reproducible by using the following code:</p>
<pre><code class="lang-auto">def set_seed(seed):
    torch.manual_seed(seed)
    np.random.seed(seed)
    random.seed(seed) 
    # for cuda
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.enabled = False
</code></pre>
<p>I call the set_seed function with every run I do. Tbh, I don’t fully understand why I cannot just set the seed once, but it works <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Convolutional neural network

Kuldip: <p>Hi, I am new to deep learning. I am trying to make CNN on CSV data. The data is derived from the images. And I am predicting x,y,z coordinates for mono pong game. Now I am stuck at the preprocessing of the data. I have (667,225) CSV of input set and (667,3) CSV of labels. So I read through the internet but still struggling with reshaping the CSV. How should I convert CSV to tensor? The actual pixel of the image is (1920, 1440). (I do not have images.)<br>
Can anyone tell me how can I prepare the dataset to feed it to the architecture?<br>
and one more thing<br>
In CNN architecture how we can decide the input node, hidden lawyer and kernel size?<br>
I know there is lots of information on the internet but after reading it I became more confused.<br>
Please help me out. Thanks in advance.</p>

Hari_Krishnan: <p>From your CSV shape, I would have to say your images might be 15x15, 667 is the number of images. So the shape of the tensor would be <code>(667, 15, 15, 1)</code> adhering to the format <code>(batch_size, image_width, image_height, num_channel)</code></p>

Kuldip: <p>Thank you so much for your reply. Labels do not need to change shape right?</p>

Hari_Krishnan: <p>There is no need to change the labels shapes.</p>
What is the best way to concatenate hidden states of lstm output?

berkay_berabi: <p>Hi,</p>
<p>I am using a bidirectional lstm. By default this returns concatenated forward and backward hidden states as output, but unfortunately, the hidden states h_n and c_n are returned in shape (num_directions, batch_size, hidden dim). In my case am using a seq2seq and my decoder is unidirectional but it has the double of the embedding hidden dim so that I can directly pass the last hidden states and use attention mechanism. So I have to convert the last hidden states from the encoder lstm to a concatenated version just like in the outputs.</p>
<p>I came up with the following but it seems to me a bit strange.</p>
<pre><code>    h_n = h_n.permute(1, 0, 2)
    c_n = c_n.permute(1, 0, 2)
    h_n = torch.reshape(h_n, (1, h_n.shape[0], h_n.shape[2]*2))
    c_n = torch.reshape(c_n, (1, c_n.shape[0], c_n.shape[2]*2))
</code></pre>
<p>Can someone suggest a better way ?</p>
Problems with jit load 3D monai model

InternalInstance: <p>I saved 3D DensNet from monai:</p>
<pre><code class="lang-python">traced = torch.jit.trace(model, torch.rand(1, 3, 160, 160, 160).cuda())
torch.jit.save(traced, f'monai3_160_baseline.trcd')
</code></pre>
<p>Now I am trying to load it from trcd file with the following function:</p>
<p><code>torch.jit.load(x, map_location="cuda:0")</code></p>
<p>And I get the following error:</p>
<pre><code class="lang-python">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-28-60944cabc718&gt; in &lt;module&gt;()
----&gt; 1 model = load_jit_model("./model_training/monai3_160_baseline.trcd").cuda()

&lt;ipython-input-22-2020de3a15ce&gt; in load_jit_model(x, cuda_id)
     14 
     15 def load_jit_model(x, cuda_id=0):
---&gt; 16     return torch.jit.load(x, map_location=f'cuda:{cuda_id}' if torch.cuda.is_available() else 'cpu')

/home/yeliseev/anaconda3/envs/rsna/lib/python3.7/site-packages/torch/jit/__init__.py in load(f, map_location, _extra_files)
    273     cu = torch._C.CompilationUnit()
    274     if isinstance(f, str) or isinstance(f, pathlib.Path):
--&gt; 275         cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
    276     else:
    277         cpp_module = torch._C.import_ir_module_from_buffer(cu, f.read(), map_location, _extra_files)

RuntimeError: expected ident but found 'class' here:
Serialized   File "code/__torch__/torch/nn/modules/container/___torch_mangle_423.py", line 8
  norm : __torch__.torch.nn.modules.pooling.AdaptiveAvgPool3d
  flatten : __torch__.torch.nn.modules.flatten.Flatten
  class : __torch__.torch.nn.modules.linear.Linear
  ~~~~~ &lt;--- HERE
  def forward(self: __torch__.torch.nn.modules.container.___torch_mangle_423.Sequential,
    argument_1: Tensor) -&gt; Tensor:
</code></pre>

ptrblck: <p>Which PyTorch an MONAI version are you using?</p>
<p>CC <a class="mention" href="/u/monai">@MONAI</a></p>

InternalInstance: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for your reply.<br>
torch==1.6.0<br>
monai==0.3.0</p>
Gradient clipping not working

Hari_Krishnan: <p>I’m building a model similar to a GAN, the sigmoid layer of discriminator outputs <code>1</code> when the discriminator gets better. This causes the loss function <code>log(1 - D(Z))</code> to become <code>-inf</code>. This is the code for my discriminator:</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self, hidden_size, latent_dim):
        super(Discriminator, self).__init__()

        self.dim_h = hidden_size
        self.n_z = latent_dim

        self.main = nn.Sequential(
            nn.Linear(self.n_z, self.dim_h),
            nn.LeakyReLU(),
            nn.Dropout(0.25),
            nn.Linear(self.dim_h, self.dim_h),
            nn.LeakyReLU(),
            nn.Dropout(0.5),
            nn.Linear(self.dim_h, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.main(x)
        return x
</code></pre>
<p>And this is the training the discriminator portion</p>
<pre><code class="lang-auto"> d_optim.zero_grad()
 X_train, y_train, length = next(data_generator(data, y, word_to_dict, batch_size = batch_size))
 X_train = X_train.cuda()
 encoded_z, _ , _, _= model.encode(X_train)
 z = torch.randn_like(encoded_z)
 z = z.cuda()
 d_z = critic(z)
 d_z_hat = critic(encoded_z)
 d_z_loss = lambd * torch.log(d_z).mean()
 d_z_hat_loss = lambd * torch.log(1 - d_z_hat).mean()
 loss = -(d_z_hat_loss + d_z_loss)
 loss.backward()
 torch.nn.utils.clip_grad_norm_(critic.parameters(), 0.25)
 d_optim.step()
</code></pre>
<p>To avoid <code>-inf</code> loss to cause instability during backpropagation, I am clipping the gradient by norm before  <code>d_optim.step()</code>.</p>
<p>I have also tried using <code>torch.nn.utils.clip_grad_value_(parameters, clip_value)</code> to solve the <code>-inf</code> problem.</p>
<p>Both the functions did not work correctly, is there anything I am missing regarding this?</p>

ptrblck: <p>The gradient norm clipping wouldn’t work, since multiplying a <code>+/-Inf</code> gradient with the scale factor won’t change the gradient (<a href="https://github.com/pytorch/pytorch/blob/1c6ace87d127f45502e491b6a15886ab66975a92/torch/nn/utils/clip_grad.py#L40">used here</a>).</p>
<p>While clipping the gradient with values would work for <code>+/-Inf</code> values, unfortunately the <code>+/-Inf</code> loss might create <code>NaN</code> gradients as seen here:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
x = torch.randn(1, 1)
out = model(x)

loss = torch.log(out - out)
print(loss)
&gt; tensor([[-inf]], grad_fn=&lt;LogBackward&gt;)

loss.backward()
print(model.weight.grad)
&gt; tensor([[nan]])
</code></pre>
<p>I think the right approach would be to avoid creating the invalid loss values and e.g. add a small <code>eps</code> value to the loss calculation.</p>
Truncate a tensor to given shape

aliutkus: <p>Hi there.</p>
<p>I have a tensor, say <code>v</code>, of shape, say <code>(10, 3, 50,25,12)</code></p>
<p>I have a variable <code>shape</code>, that contains, say <code>(10, 2, 13, 18, 12)</code></p>
<p>Is there a convenient way to “truncate” <code>v</code> so that its shape becomes <code>shape</code> ?</p>
<p>thanks !</p>

aliutkus: <p>ok, got it:</p>
<pre><code class="lang-python">v[[slice(k) for k in shape]]
</code></pre>
Prepare data for Bi-LSTM text classifier

ptrch_c_m: <p>I want to build a binary text classifier with a Bi-LSTM.</p>
<p>Which is the best way to prepare data as input to the model? Especially, create the vocabulary of the dataset for using it with word embeddings.</p>
<p>Thanks in advance.</p>

Sagnik_Mukherjee: <p>You should iterate over entire dataset to create your unique list of tokens(i.e. the vocab) . And assign a unique id to all of them. Next map your text input to these ids before passing to your model. Also, at the start of the model you should have an Embedding layer. Lookup nn.Embedding.from_pretrained for that purpose.</p>

ptrch_c_m: <p><a class="mention" href="/u/sagnik_mukherjee">@Sagnik_Mukherjee</a> thank you for your response.</p>
<p>Actually, I was looking for a more technical response. Searching for libraries to tokenise the dataset, build vocabulary etc I found <code>torchtext</code> package which, is implemented by <code>Pytorch</code> but because it is a different package, it has a different version (at the time of writing <code>0.8.0a0+c4a91f2</code>) from <code>Pytorch</code>.</p>
<p>Is this package is one of the recommended ways to prepare data for the model or are there others that are more preferable?</p>
<p>p.s. since I will use GloVe embeddings your tip about <code>nn.Embedding.from_pretrained</code> was helpful.</p>
<p>Regards.</p>

Sagnik_Mukherjee: <p>I personally prefer spacy for tokenization. Whether in reality you should try out different tokenizers and report the best results.</p>

ptrch_c_m: <p>I used <code>torchtext</code> for preparing data as input for the model.<br>
It is very convenient for tokenization and building vocabulary. For tokenization, it has some build in options and includes spaCy tokinizer as well.<br>
Even though <code>torchtext</code> provides also data loading, I used it just for data preparation and used <code>Dataloader</code> instead.</p>
<p>Thank you for your response,<br>
Regards</p>
Replacing LSTM by Transformer for Image Captioning

Vitalik: <p>Hi, I’m working now at my diploma and I decided to do Image Captioning. I’ve already implemented CNN -&gt; LSTM (without attention) and it works. Also, I found that when I made 2-layers LSTM performance increased.</p>
<p>Then I decided to replace RNN by Transformer using it almost in the same way (when in case of RNN I put vector that I got from pre-trained CNN to first time-step of LSTM and the caption as an output, in case of Transformer I put this vector to Transformer’s encoder, the caption to Transformer’s decoder and shifted one as aspected output).</p>
<p>So, after this, I found that it doesn’t work well. Maybe the reason for that is that I put to the batch captions of the same length and my <strong>tgt_key_padding_mas</strong> is always like <code>[False, False, False, False ... ]</code>? (I don’t use src and memory mask, as my input is vector from CNN)</p>
<p>What do you think about that and can you suggest me something to increase the performance?</p>

Bibyutatsu: <p>Hi, were you able to solve this? I am currently replacing my LSTM model with transformers and was trying to find any existing literature or source. If you can help me it will be great</p>

Vitalik: <p>Hi, sorry for my delayed answer, but if it can help you, there is a new <a href="https://arxiv.org/abs/1908.07490" rel="noopener nofollow ugc">paper</a> that can help you in your work. They are even available in the <a href="https://huggingface.co/transformers/model_doc/lxmert.html" rel="noopener nofollow ugc">Transformers</a> framework. But this work related to QA, not text generation. Quick googling gives me with a pre-trained model from torch hub <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
But I kept my project with LSTM as a solution with the transformer failed due to my incorrect usage of the idea of the transformer architecture.</p>
Torch::save and torch::load not working for a simple example

abhinavkulkarni: <p>Hi,</p>
<p>I have a very simple example:</p>
<pre><code class="lang-cpp">int main(int argc, char* argv[]) {
  torch::NoGradGuard no_grad;

  auto linear = torch::nn::Linear(4, 2);
  linear-&gt;weight = linear-&gt;weight * 0;
  linear-&gt;bias = linear-&gt;bias * 0;

  auto x = torch::rand({3, 4}).toType(torch::kFloat);

  std::cout &lt;&lt; linear-&gt;forward(x) &lt;&lt; std::endl;
  torch::save(linear, "/tmp/linear.pt");

  linear = torch::nn::Linear(4, 2);
  torch::load(linear, "/tmp/linear.pt");
  std::cout &lt;&lt; linear-&gt;forward(x) &lt;&lt; std::endl;
}
</code></pre>
<p>I get the following output (different random output everytime):</p>
<pre><code class="lang-auto"> 0  0
 0  0
 0  0
[ CPUFloatType{3,2} ]
-0.9355 -0.0190
-0.7813  0.2720
-0.6068  0.1082
[ CPUFloatType{3,2} ]
</code></pre>
<p>Why is this simple example not working?</p>
<p>Thanks!</p>

ptrblck: <p>The code should work as shown in <a href="https://github.com/pytorch/pytorch/blob/41f2dbde31b13c9246dd94bacc2b35f85739afd0/test/cpp/api/serialize.cpp#L361-L470">these examples</a>.<br>
Which libtorch version are you using?</p>
Weird warning popping out regarding learning rate scheduler

sahiluppal2k: <p>I’m getting this warning only after first epoch</p>
<pre><code class="lang-bash">/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:200: UserWarning: Please also save or load the state of the optimizer when saving or loading the scheduler.
warnings.warn(SAVE_STATE_WARNING, UserWarning)
</code></pre>
<p>I’m unable to understand this. What does this mean?</p>

fadetoblack: <p>Can you please show us a snippet of your code which you’re running?</p>

sahiluppal2k: <p><a class="mention" href="/u/fadetoblack">@fadetoblack</a> Yeah sure.<br>
Here is my model and dataset file  <a href="https://gist.github.com/saahiluppal/e1ffeac4c6c6c3da045cf07d8f8df37e" rel="noopener nofollow ugc">https://gist.github.com/saahiluppal/e1ffeac4c6c6c3da045cf07d8f8df37e</a></p>
<p>And here is how i run it</p>
<pre><code class="lang-bash">model = ModelOne(config)
data = DataTypeOne(config)

trainer = pl.Trainer(gpus=1, max_epochs=config.epoch)
trainer.fit(model, data)

</code></pre>

jsrozner: <p>I had a similar discussion on huggingface: <a href="https://github.com/huggingface/transformers/issues/7765" rel="noopener nofollow ugc">github issue</a></p>
<p>This happens when optimizer state dict is accessed. See here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py" rel="noopener nofollow ugc">pytorch repo</a></p>

sahiluppal2k: <p>Seems that we can safely ignores that.<br>
Thanks</p>