akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Sparse Linear layer

Albert65: <p>Hi All, I would appreciate an example how to create a sparse Linear layer, which is similar to fully connected one with some links absent. It turns out the “torch.sparse” should be used, but I do not quite understand how to achieve that. I start from the dense tensor (image in my case), the next (hidden) layer shoud be a dense image of smaller size, and so on following the autoencoder layout. The only difference with the conventional pattern is that not every pair of entries of successive layers is connected (for performance reason).</p>

Kale-ab_Tessera: <p>Any news on this? I am also looking for sparse linear layers.</p>

Aamir_Mirza: <p>here you go , full implementation on git hub for sparse layer <a href="https://github.com/numenta/htmpapers/tree/master/arxiv/how_can_we_be_so_dense" rel="nofollow noopener">https://github.com/numenta/htmpapers/tree/master/arxiv/how_can_we_be_so_dense</a></p>

robloughnan: <p><a class="mention" href="/u/aamir_mirza">@Aamir_Mirza</a> thanks for sharing. Does this implementation result in saving GPU memory instead of using dense layers? Or is it that it enforces sparsity of the weights for convergence reasons?</p>

shreyansh26: <p>Is there any update on this? Any new libraries that implement sparse layer?</p>

Saurabh_Kataria: <p><a href="https://github.com/huggingface/pytorch_block_sparse" rel="noopener nofollow ugc">https://github.com/huggingface/pytorch_block_sparse</a> is the latest I am aware of but dont know about its performance.</p>
Why weights/bias are initialized like this for RPN?

Mriganka_Nath: <p>In the RPN() function why are the weights initialized like this? Am I missing something from the paper? and why this why not use kalming initilization and all.<br>
I don’t even understand why bias is set as 0.<br>
Can someone help?</p>
<pre><code class="lang-auto">for layer in self.children():
            torch.nn.init.normal_(layer.weight, std=0.01)  # type: ignore[arg-type]
            torch.nn.init.constant_(layer.bias, 0)  # type: ignore[arg-type]
</code></pre>
<p><a href="https://github.com/pytorch/vision/blob/e65a857b5487a8493bc8a80a95d64d9f049de347/torchvision/models/detection/rpn.py#L27" rel="noopener nofollow ugc">The RPN function</a><br>
Am I missing something obvious?</p>

Mriganka_Nath: <p>Another doubt inside RPN() here</p>
<pre><code class="lang-auto">self.bbox_pred = nn.Conv2d(in_channels, num_anchors * 4, kernel_size=1, stride=1)
</code></pre>
<p>Here why are we doing ‘num_anchors * 4’?<br>
from where is this 4 coming?<br>
whereas in detectron2, in the RPN it is defined as</p>
<pre><code class="lang-auto">self.anchor_deltas = nn.Conv2d(
            in_channels, num_cell_anchors * box_dim, kernel_size=1, stride=1
        )
</code></pre>
<p>here its “num_cell_anchors * box_dim”<br>
(<a href="https://github.com/facebookresearch/detectron2/blob/5e2a1ecccd228227c5a605c0a98d58e1b2db3640/detectron2/modeling/proposal_generator/rpn.py#L66" class="inline-onebox" rel="noopener nofollow ugc">detectron2/rpn.py at 5e2a1ecccd228227c5a605c0a98d58e1b2db3640 · facebookresearch/detectron2 · GitHub</a>)<br>
I don’t know whats happening here.</p>
Mechanics of the optimizer momentum when updating a conditional neural network module

Ambrose: <p>Question is after code block. My module has two parts, partA and partB which are triggered conditionally in the forward function:</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()

    self.partA = nn.Sequential(
        convolutions
    )

    self.partB = nn.Sequential(
        convolutions
        nn.Flatten()
        nn.Linear()
    )

    def forward(input_tensor):
        if conditionB:
            return self.partB(input_tensor)
        elif conditionA: # for clarity. could just use 'else' here.
            return self.partB(self.partA(input_tensor))

</code></pre>
<pre><code class="lang-auto">## initialize and train
model = Network().to(device)
opt = optim.SGD(model.parameters(), lr=0.0001, momentum=0.9)

for epoch in range(num_epochs):
    for i, data in enumerate(trainloader):
        model.zero_grad()
        outputA = model(input_tensor) # condition A
        err = criterion(labels, outputA)
        err.backwards()
        opt.step()

        model.zero_grad()
        outputB = model(input_tensor) # condition B
        err = criterion(labels, outputB)
        err.backwards()
        opt.step()
</code></pre>
<p>During training, when condition B is triggered, there will be gradients through partB of the network but not partA. However, will the momentum term in SGD cause an update in partA when opt.step() is called, even though it was not involved in the computational graph during the condition B step? If so, how should I avoid this? Give partA and partB separate optimizers?</p>

tom: <p>I would indeed try to use separate optimizers here.<br>
I think you could also replace <code>for p in model.parameters(): p.grad = None</code> to see if that skips the momentum update, but this is less obvious to the user so the next change might break it again.</p>
<p>Best regards</p>
<p>Thomas</p>

Ambrose: <p>Thanks Thomas! I separated them into multiple optimizers and it seems to work.</p>
Seq2Seq best loss function for bag of sentence output

Lady_Hangaku: <p>I am currently experimenting with a seq2seq task for entity and relation extraction.<br>
Given a text, the desired output looks like this:</p>
<p>[name of person] : SKILL : [name of skill] ; [name of other person] : PROJECT : [name of project] ; …</p>
<p>It doesn’t matter how the relations are arranged in the output, it is just important that the relations are correct. So the two elements in the example can also be listed the other way around.<br>
I’m wondering which loss function is the most suitable for that task. Any ideas? Thank you!</p>
Entropy loss is increasing the loss

user3: <p>Hi<br>
i made a loss based on minus of the entropy of the softmax layer, such that the information will disperse well.<br>
it all worked well, until somehow, something got wrong, and instead of making the information more dispersed,the data becomes more unique and from an array of random value, i get an array with all zeros except one element.<br>
if i decrease the learning rate, the network converges, but less well than before.</p>
<p>thanks</p>

mMagmer: <p>That is expected behavior if you are minimizing the following objective:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db4412811cb32e67395cfc306ee2c8a160e26cea.png" alt="image" data-base62-sha1="vhIpqTbULk9pggn1bMXMxss5CoG" width="341" height="77"><br>
H minimize (and it is equal to zero) if all  p_i  except one are zero.<br>
so the last remaining p_i should be equal to 1.</p>

user3: <p>hi<br>
as i said, im trying to minimize the <em>minus</em> of the entropy (in order to maximze the entropy). so the information should not concentrate on one neoron, but it should disperse uniformly</p>

mMagmer: <p>hi</p>
<pre><code class="lang-auto">import torch
data = torch.randn(100,2,requires_grad=True)
data.shape # &gt; 

optim = torch.optim.SGD([data], 0.1)
def loss(eta):
    return torch.sum(eta*torch.log(eta))

for i in range(1000):
    optim.zero_grad()
    eta = torch.nn.functional.softmax(data,-1)
    l = loss(eta)
    l.backward()
    optim.step()
    if i%10 == 0:
        print(i,l.item())
</code></pre>
<p>i see no problem.<br>
it converges to all eta equal to 0.5.</p>
Write DataLoader's collate_fn with pad_packed_sequence for LSTMs

Maghoumi: <p>I’m very new to PyTorch and my problem involves LSTMs with inputs of variable sizes.</p>
<p>Because each training example has a different size, what I’m trying to do is to write a custom <code>collate_fn</code> to use with DataLoader to create mini-batches of my data. To my understanding, I’d need to implement my own <code>collate_fn</code> and use <code>pad_packed_sequence</code> somehow…</p>
<p>I have tried looking at examples online, but nobody seems to be doing it like me <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"><br>
The problem arises from the fact that <code>collate_fn</code> gets passed the input <code>batch</code> which is a list of tuples of <code>(training_tensor, label)</code> and I can’t seem to figure out how to properly convert it to a suitable datatype and pass the tensors to <code>pad_packed_sequence</code>.</p>
<p>What is the correct way of doing this?<br>
A more general question: is there something better than <code>DataLoader</code> that works nicely with LSTMs?</p>

Maghoumi: <p>I figured this out myself a while back. I posted a detailed solution here:</p>
<p><a href="https://www.codefull.org/2018/11/use-pytorchs-dataloader-with-variable-length-sequences-for-lstm-gru/" class="onebox" target="_blank" rel="nofollow noopener">https://www.codefull.org/2018/11/use-pytorchs-dataloader-with-variable-length-sequences-for-lstm-gru/</a></p>

valentinoli: <p><a class="mention" href="/u/maghoumi">@Maghoumi</a> thanks for your post! Someone also posted this here later in 2019 in case it might be useful to someone:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html">
  <header class="source">

      <a href="https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html" target="_blank" rel="noopener nofollow ugc">suzyahyah.github.io</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://suzyahyah.github.io/pytorch/2019/07/01/DataLoader-Pad-Pack-Sequence.html" target="_blank" rel="noopener nofollow ugc">Pad pack sequences for Pytorch batch processing with DataLoader</a></h3>

  <p>Pytorch setup for batch sentence/sequence processing - minimal working example. The pipeline consists of the following:</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Also note that a boolean parameter <code>enforce_sorted</code> was added to  <code>torch.nn.utils.rnn.pack_padded_sequence()</code> in December 2018 (see <a href="https://github.com/pytorch/pytorch/issues/3584" rel="noopener nofollow ugc">this issue</a>). So there is no need anymore to provide a sorted sequence, it is sorted internally when <code>enforce_sorted=False</code>.</p>
How to implement BERT using torch.nn.Transformer?

GokulNC: <p>Currently, I use <code>nn.TransformerEncoder</code> to implement BERT.<br>
An example of a BERT architecture:</p>
<pre><code class="lang-auto">encoder_layer = nn.TransformerEncoderLayer(d_model=embedding_size, nhead=num_heads)
bert = nn.Sequential(
    nn.TransformerEncoder(encoder_layer, num_layers=num_encoder_layers),
    nn.Linear(embedding_size, output_vocab_size)
)
</code></pre>
<p>How do I achieve the same using the <a href="https://pytorch.org/docs/master/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc"><code>nn.Transformer</code></a> API?</p>
<p>The doc says:</p>
<blockquote>
<p>Users can build the <a href="https://arxiv.org/abs/1810.04805" rel="noopener nofollow ugc">BERT</a> model with corresponding parameters.</p>
</blockquote>
<p>Even if I set <code>num_decoder_layers=0</code> while initializing it, the <a href="https://pytorch.org/docs/master/generated/torch.nn.Transformer.html#torch.nn.Transformer.forward" rel="noopener nofollow ugc"><code>forward()</code></a> call mandatorily requires the argument <code>tgt</code> for the transformer’s decoder, but BERT has no decoder.<br>
So how do we go about it?</p>

GokulNC: <p>Bump… Anyone?</p>
<p>Note: Though I am aware of <code>HuggingFace</code>'s BERT out-of-the-box, for simple non-NLP experiments using custom BERT-like small architectures, I think using PyTorch alone should suffice. PLMK if I’m wrong.</p>

narain1: <p>struggling with the same thing actually and still using the from scratch implementation <em>help needed</em></p>

AlbertZeyer: <p>Why do you want to do that? Isn’t your initial code using <code>TransformerEncoder</code> simple enough?</p>
Expected str, bytes or os.PathLike object, not bool

Aditya_Srivastava: <p><strong>Hello everyone, Happy New Year</strong><br>
<strong>I am trying to train a model but stuck in this situation</strong></p>
<p>The trainer is defined below<br>
model=CNN_Model(hparams)</p>
<p>trainer = pl.Trainer(max_nb_epochs,<br>
gpus=1,<br>
track_grad_norm,<br>
gradient_clip_val,<br>
early_stop_callback,<br>
train_percent_check = 0.01,<br>
val_percent_check = 0.01,<br>
test_percent_check = 0.01,<br>
overfit_pct,<br>
fast_dev_run=True,<br>
profiler)</p>
<p>###############################<br>
Till here the code is running perfectly fine<br>
###############################</p>
<p>Now when I run this line<br>
“trainer.fit(model)”</p>
<p>I am getting this error:  expected str, bytes or os.PathLike object, not bool</p>

ptrblck: <p>I don’t know what <code>fit</code> does exactly, but would assume that the stacktrace would point to the line of code raising the error.<br>
E.g. <code>os.path.join(True)</code> would raise this error so I guess that some internal method is trying to create a path or load a file with a <code>bool</code> input argument instead of a valid path.</p>
Only update weight of one model in training

subedir: <p>I have two models A and B. The output of model A is fed to model B and the loss is calculated(example below)</p>
<p>opA = modelA(inp)<br>
opB = modelB(opA)<br>
loss = criterion(opB, GT)</p>
<p>I want to update weights of model A only. How can I achieve this?</p>

ptrblck: <p>Create an optimizer with the parameters of <code>modelA</code> only, calculate the gradients using <code>loss.backward()</code> can call <code>optimizerA.step()</code>.<br>
If you never want to update <code>modelB</code>, you could set the <code>requires_grad</code> attributes of its parameters to <code>False</code> additionally.</p>

subedir: <p>Thank you. But,<br>
Is this the same as setting modelB to eval mode?<br>
and,<br>
If I want to update the weights of both the models, can I achieve this by creating two optimizers?</p>
<p>optimizer_A = torch.optim.Adam(modelA.parameters())<br>
optimizer_B = torch.optim.Adam(modelB.parameters())</p>
<p>And in training loop<br>
optimizer_A.zero_grad()<br>
optimizer_B.zero_grad()<br>
opA = modelA(inp)<br>
opB = modelB(opA)<br>
loss = criterion(opB, GT)<br>
loss.backward()<br>
optimizer_B.step()<br>
optimizer_A.step()</p>

ptrblck: <aside class="quote no-group" data-username="subedir" data-post="3" data-topic="140223">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c6cbf5/48.png" class="avatar"> subedir:</div>
<blockquote>
<p>Is this the same as setting modelB to eval mode?</p>
</blockquote>
</aside>
<p>No, setting the model to <code>.eval()</code> will change the behavior of some layers (e.g. dropout will be disabled and batchnorm layers will use their running stats) and will not change the behavior of the gradient calculation.</p>
<aside class="quote no-group" data-username="subedir" data-post="3" data-topic="140223">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c6cbf5/48.png" class="avatar"> subedir:</div>
<blockquote>
<p>If I want to update the weights of both the models, can I achieve this by creating two optimizers?</p>
</blockquote>
</aside>
<p>Yes, that works.</p>
Training specific layer (module) in a model

jaehwanee0411: <p>I’ve define a auto-encoder model as below.<br>
<code><br>
class AE(nn.Module):<br>
def <strong>init</strong>(self, img_c, z_dim):<br>
super(AE, self).<strong>init</strong>()<br>
self.enc = Encoder(img_c, z_dim)<br>
self.gen = Generator(z_dim, img_c)<br>
self.model = nn.Sequential(<br>
self.enc,<br>
self.gen<br>
)<br>
def forward(self, x):<br>
return self.model(x)</code></p><code>
<p>Encoder and Generator are another class(nn.Module) I have designed.</p>
<p>First, I want to train the whole network which can be easily done with<br>
<code><br>
ae = AE()<br>
ae.zero_grad()<br>
ae_loss.backward()<br>
opt_ae.step()</code></p>
<code>
<p>Second, I want to only train Encoder not the whole network.<br>
How can I specifically train only the Encoder part of the model?</p></code></code>

ptrblck: <p>Create a separate optimizer by passing <code>ae.enc.parameters()</code> to it and use this new optimizer to update the encoder parameters only in <code>opt_enc.step()</code>.</p>

jaehwanee0411: <p>Appreciate, and can i ask you another question?<br>
I am trying to train encoder not only with the ‘gradient backpropagated from decoder’ but also with another loss(gradient) that been calculated in the latent space.<br>
In order to do, before call</p>
<blockquote>
<p>enc_loss = loss + backpropagated loss<br>
enc_loss.backward()<br>
opt_enc.step()</p>
</blockquote>
<p>I need to get backpropagated loss from decoder.</p>
TCN autoencoder

fatcat: <p>Hello everyone,</p>
<p>I am going to use 1D convolutions to learn representations of time series data. I plan to use an Encoder-Decoder architecture. The input shape is like: [batch_size, num_features, num_timesteps]; the outputs of the encoder should be like: [batch_size, length]; i.e., I wish to get a fixed-length representation for each sequence. It seems that the length of the outputs depends on the original length of the sequence of the batch. Is there a method that I can define the length of the outputs of the encoder regardless of the length(num_timesteps) of the original sequence?</p>

googlebot: <p>using mean (a la average pooling) is the usual reduction method for unknown lengths. but by itself it is too simplistic, i.e. it is a linear interpolation invariant to reordering. one possibility is to apply an attention layer first.</p>

fatcat: <p>Thanks for your reply. Do you mean that I should use an attention over all the time steps to get an attention vector whose shape is [batch_size, 1, num_timesteps] and then multiply this attention vector with another vector whose shape is [batch_size, num_channels, num_timesteps] and then compute the mean along the 2nd dimension (num_timesteps) to get an output vector whose shape is [batch_size, num_channels] ?</p>

fatcat: <p>And I am also a little confused about how can I transform the latent variables whose shape is [batch_size, length] to time series in the decoder part.</p>

googlebot: <p>Well, frankly, implementation details vary, some being pretty complex. I’d suggest to start from some existing design for a similar problem.</p>
<p>Your attention description sounds right, but it is pretty generic… For encoder, it can be a self-attention layer, to support unequal importance of time points. For some tasks, you’d want to encode positions too (and/or time-based features), for same reason as in NLP - to make element order matter.</p>
<p>As for decoders, I’m not too familiar with RNN-less context unrolling for convolutional decoders. I believe another attention module is one way to do it. Again, it may be better to check what they did in relevant research papers.</p>
Custom loss using an NLP metric after OCR

Vijay_Singh: <p>Hi am working on a problem where we’re given images of text that may be faded, or is generally lower resolution and the task is to OCR the image. I am exploring the idea of using something like SRGAN to improve the quality of the image, and I would like to add a loss component that OCRs the generated image, compares the output to GT output and use a distance metric (edit-distance etc) to provide some notion of how good the image is from an OCR point of view. Does this make sense? How would I do something like this?</p>
Can forward() in nn.module be override with different arguments?

Nathan_Li: <p>Can forward()  be override with different arguments? So different forward is used in different scenario?</p>
<h1>example:</h1>
<p>class Test(nn.Module):</p>
<pre><code>def __init__(self):
   super(Test, self).__init__()
   pass

def forward(self, x, y ):
    pass

def forward(self, x, y , z):
    pass</code></pre>

albanD: <p>Hi,</p>
<p>python does not allow you to create two functions with the same name like that.<br>
What you can do though is:</p>
<pre><code class="lang-auto">def __init__(self):
   super(Test, self).__init__()
   pass

def forward(self, x, y, z=None ):
    if z is None:
        # Forward with 2 arguments
        pass
    else:
        # Forward with 3 arguments
        pass
</code></pre>

Sotiris_Kotitsas: <p>Hello, based on that answer.<br>
Can i have a forward pass when i am training(passing for example a bool to clarify that i am training) and a different forward pass when i am evaluating?<br>
For example i am training an rnn and taking the last hidden state as an encoding for a sentence. But while i am training, together with my sentences i am passing some other information(for example negative samples), but i would like to load my trained model and just pass one sentence through the rnn and get its encoding.<br>
Could i do that? <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Sure!<br>
You can adapt <a class="mention" href="/u/alband">@albanD</a>’s code and pass an additional flag to it, if that’s what you are looking for:</p>
<pre><code class="lang-python"> 
def forward(self, x, y, training=True):
    if training:
        pass
    else:
        pass
</code></pre>
<p>Also, if your forward method behavior switches based on the internal training status (<code>model.train()</code> vs. <code>model.eval()</code>), you don’t even have to pass an additional flag, but can just use the internal one:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(10, 2)
        
    def forward(self, x):
        if self.training:
            return x
        else:
            return self.fc(x)

x = torch.randn(1, 10)
model = MyModel()
print(model(x).shape)
&gt; torch.Size([1, 10])
model.eval()
print(model(x).shape)
&gt; torch.Size([1, 2])
</code></pre>

Sotiris_Kotitsas: <p>That is exactly what i was looking for. Thank you very much!</p>

timf34: <p>Hi,<br>
Would placing ‘if else’ statements within the forward pass slow it down appreciably during training?<br>
Thanks</p>
Why can't I use embedding table to get around large GPU memory requirement?

thefedshyvana: <p>Suppose I have data that requires a large amount of GPU memory (e.g. 80,000 7 x 7 x 1024 tensors). I was hoping that I can get around this if I use a fixed size embedding table (lets assume its already learned somehow). i.e., if I use an embedding table of size 100, each token is 1024-dim, then my understanding is that all I need to do now is to fit an 100 x 1024 tensor + 80,000 7 x 7 indices to look-up onto memory instead (since all embeddings of the same index should just be represented as views of its corresponding entry in the look-up table).<br>
However, I find that this is not the case - i.e., when I call nn.Emebedding on the indices, the memory usage jumps sharpely as if I am actually storing a 80,000 x 7 x 7 x 1024 tensor, rather then just views of the same underlying entries in the look-up table. What am I missing?</p>
<p>Thanks!</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
num_tokens = 100
codebook_dim = 1024
h = w = 7
batch_size = 80000

# embedding table
codebook = nn.Embedding(num_tokens, codebook_dim).to(device)
codebook.weight.requires_grad = False
indices = torch.randint(num_tokens, (bag_size, h, w)).to(device) # indices 
indices = codebook(indices.view(-1, h*w)) # after look-up, this now uses a huge amount of memory
</code></pre>
Trying to implement FPN in tensorflow using a model (already) implemented in pytorch

Adwait_Naik: <p><strong>pytorch based</strong></p>
<pre><code class="lang-auto">class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, in_planes, planes, stride=1):
        super(Bottleneck, self).__init__()
        self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)
        self.conv3 = nn.Conv2d(planes, self.expansion*planes, kernel_size=1, bias=False)
        self.bn3 = nn.BatchNorm2d(self.expansion*planes)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion*planes:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(self.expansion*planes)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = F.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out
</code></pre>
<p>My TensorFlow implementation</p>
<pre><code class="lang-auto">input_shape = (28,28,1)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv2D(28, kernel_size = (1,1), bias_constraint=None))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv2D(28, kernel_size = (3,3), strides = 1, bias_constraint=None))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv2D(28, kernel_size = (1,1), strides = 1, bias_constraint=None))
model.add(tf.keras.layers.BatchNormalization(4))
model.add(tf.keras.layers.Dense(128, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(10,activation=tf.nn.softmax))
</code></pre>
<p>Do I need to change something in my implementation?</p>
RuntimeError: Input and hidden tensors are not the same dtype, found input tensor with Float and hidden tensor with Double

ashcher51: <p>Hey everyone. I’m dealing with an issue regarding data type mismatch, and I’m not sure how I could solve this. It says my inputs are of type float, but I have made sure to cast them to a double. Here is my dataset class:</p>
<pre><code class="lang-auto">class OsicDataset(Dataset):
    def __init__(self, df_list):
        self.df_list = df_list
        self.seq_list = self.preprocess()
        self.target = self.seq_list[0].columns[-1]
        self.features = self.seq_list[0].columns[1:-1]
        self.transforms = transforms.Compose([
            transforms.ToTensor(),
        ])

    def __len__(self):
        return len(self.seq_list)

    def __getitem__(self, idx):
        X = torch.from_numpy(np.array(self.seq_list[idx][self.features].values)).float()
        y = torch.from_numpy(np.array(self.seq_list[idx][self.target].values[-1])).float()

        img_paths = glob.glob('/kaggle/input/osic-pulmonary-fibrosis-progression/train/' + self.seq_list[idx]['Patient'].values[0] + '/*.dcm')
        img = self.get_img(np.random.choice(img_paths))
        img = self.transforms(img)

        return {
            'X': X.type(torch.double),
            'img': img.type(torch.double),
            'y': y.type(torch.double),
        }

    def preprocess(self):
        cleaned_df_list = []
        for df in self.df_list:
            df_clean = df.dropna()
            df_clean['FVC'] = (df_clean['FVC'] - fvc_mean) / fvc_std
            df_clean = self.shift(df_clean)
            df_clean = df_clean.reset_index(drop=True)
            cleaned_df_list.append(df_clean)
        
        seq_list = []
        for df in cleaned_df_list:
            for i in range(len(df) - sequence_len):
                seq = df.iloc[i:i+sequence_len]
                seq_list.append(seq)
        print(seq_list[0])
        return seq_list

    def shift(self, x: pd.DataFrame):
        x['Weeks(t+1)'] = x['Weeks'].shift(-1)
        x['FVC(t+1)'] = x['FVC'].shift(-1)
        x = x.iloc[:-1]
        return x

    def get_img(self, path):
        ds = dcmread(path)
        pxls = (ds.pixel_array - ds.RescaleIntercept) / (ds.RescaleSlope * 1000)
        return cv2.resize(pxls, (224, 224))
</code></pre>
<p>Any help in this regard would be greatly appreciated.</p>

Auckydocky: <p>You probably need to set your variables to float32 type. I think torch uses float as foundational type.<br>
I see you using double type for the outputs of getitem. Hope it fix that.</p>
Mismatch between FastAI prediction and Pytorch prediction

RowanG1: <p>Here is the FastAI model:</p>
<blockquote>
<p>TabularModel(<br>
(embeds): ModuleList()<br>
(emb_drop): Dropout(p=0.0, inplace=False)<br>
(bn_cont): BatchNorm1d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(layers): Sequential(<br>
(0): LinBnDrop(<br>
(0): Linear(in_features=3, out_features=200, bias=False)<br>
(1): ReLU(inplace=True)<br>
(2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(1): LinBnDrop(<br>
(0): Linear(in_features=200, out_features=100, bias=False)<br>
(1): ReLU(inplace=True)<br>
(2): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
(2): LinBnDrop(<br>
(0): Linear(in_features=100, out_features=1, bias=True)<br>
)<br>
)<br>
)</p>
</blockquote>
<p>See: <a href="https://docs.fast.ai/tabular.model.html" class="inline-onebox" rel="noopener nofollow ugc">fastai - Tabular model</a></p>
<p>It correctly makes a prediction. When I make a Pytorch prediction based on the FastAI model, as follows, the prediction is wrong (15 vs. 51):</p>
<blockquote>
<p>with torch.no_grad():<br>
modelFinal = learn.model.eval().to(“cpu”)<br>
modelFinal (None ,torch_tensor.float())</p>
</blockquote>
<p>The input tensor is:</p>
<blockquote>
<p>tensor([[2.0000, 3.1000, 4.0000]], dtype=torch.float64)</p>
</blockquote>
<p>Any ideas why the mismatch?</p>

RowanG1: <p>Resolved: it was due to the FastAI <code>normalize</code> config for <code>props</code> in:</p>
<blockquote>
<p>dls = TabularDataLoaders.from_csv(props = [normalize])</p>
</blockquote>
<p>I removed the <code>normalize</code>, and now the pytorch result matches FastAI.</p>
Attention does not seem to be applied at TransformerEncoderLayer and MultiheadAttention PyTorch

Rick_Vink: <p>Changing something at one position in my input does not affect the outputs at other positions of my transformer encoder. I made a test in isolation in PyTorch:</p>
<pre><code class="lang-auto"># My encoder layer
encoder_layer = nn.TransformerEncoderLayer(d_model=8, nhead=2)
# Turn off dropout
encoder_layer.eval()
# Random input
src = torch.rand(2, 10, 8)
# Predict the output
out_0 = encoder_layer(src)
# Change the values at one of the positions (position 3 in this case)
src[:,3,:] += 1
# Predict once again the output
out_1 = encoder_layer(src)
# Check at which positions the outcomes are different between the two cases
# I summed in the embedding space direction
print(np.sum(np.abs(out_0.detach().numpy()),axis=-1) - np.sum(np.abs(out_1.detach().numpy()),axis=-1))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">[[ 0. 0.  0.  -0.15470695  0.   0. 0.  0.     0.  0.   ]
 [ 0.   0.  0.  -0.27988768  0.  0.0.   0.   0.   0.   ]]
</code></pre>
<p>However, this does work when I do this in TensorFlow<br>
Output in Tensorflow:</p>
<pre><code class="lang-auto">[[6.4196725 6.775745  6.946576  7.26213   6.473065  5.520765  6.201167
  7.1266503 6.3147016 6.614853 ]
 [5.565378  7.030789  6.768366  6.6065626 6.7277775 7.480627  6.6785836
  6.4560523 6.4248576 6.6436586]]
</code></pre>
<p>I repost this from my Stack Overflow post. But this might be the better place for the question.</p>

Rick_Vink: <p>I already got my answer! I must add “batch_first:True”</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/70554730/attention-does-not-seem-to-be-applied-at-transformerencoderlayer-and-multiheadat">
  <header class="source">

      <a href="https://stackoverflow.com/questions/70554730/attention-does-not-seem-to-be-applied-at-transformerencoderlayer-and-multiheadat" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/7109231/rick-vink" target="_blank" rel="noopener nofollow ugc">
    <img alt="Rick Vink" src="https://www.gravatar.com/avatar/e8920f67b5fdfeba7817488049b3191c?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/70554730/attention-does-not-seem-to-be-applied-at-transformerencoderlayer-and-multiheadat" target="_blank" rel="noopener nofollow ugc">Attention does not seem to be applied at TransformerEncoderLayer and MultiheadAttention PyTorch</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/7109231/rick-vink" target="_blank" rel="noopener nofollow ugc">
    Rick Vink
  </a>
  on <a href="https://stackoverflow.com/questions/70554730/attention-does-not-seem-to-be-applied-at-transformerencoderlayer-and-multiheadat" target="_blank" rel="noopener nofollow ugc">08:57AM - 02 Jan 22 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Looking for OCR or HTR related repo

yongen9696: <p>Hi,</p>
<p>Currently, I am trying to extract the handwritten text from a photo like <a href="https://ibb.co/MhsvQKT" rel="noopener nofollow ugc">sample 1</a>, <a href="https://ibb.co/3RJw3mv" rel="noopener nofollow ugc">sample 2</a>, <a href="https://ibb.co/BPrhHVy" rel="noopener nofollow ugc">sample 3</a>.</p>
<p>Any models are available in torchvision?</p>
How to use pytoch for AMD graphics card?

buru: <p>I use amd graphics card, but now I want to learn pytoch. Is there a way to solve this problem?</p>

J_Johnson: <p>You need to install the right Pytorch for ROCm. You can find more info on it here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Variable length lists to a single tensor without padding

Shivani_Sri_Varshini: <p>Will pytorch support dynamic length in tensors<br>
I would like to convert A=[[1,2,3,4,5],[3,6,5],[7,1,3,5,3,7,6]] to a tensor without changing this lengths<br>
Is it possible to do in pytorch</p>
<p>Thanks in Advance</p>

ptrblck: <p><a href="https://github.com/pytorch/nestedtensor">pytorch/nestedtensor</a> has a prototype of this functionality so you could check it out.</p>

Shivani_Sri_Varshini: <p>I tried to install a nested tensor and got this error<br>
ERROR: nestedtensor-0.1.1_cu102-cp37-cp37m-linux_x86_64.whl is not a supported wheel on this platform.</p>
Which loss function to use? networks turn to whole zero

lifeblack: <p>Hi,<br>
I have a data with 200 elements that just one or two element is non-zero less than 0.5 that should be predicted based on train data by a network.<br>
I trained tons of different networks but they usually turn to a whole-zero weight network although mse loss equals to 0.0001.<br>
what should I do?</p>
Questions on Deep Reinforcement Learning

Nada_OUAHABI: <p>Hello,<br>
I´m applying Deep Reinforcement Learning for the first time, and I have some questions about it (I´ve already looked for an answer but in vain):</p>
<ol>
<li>How to normalize objectives’ values in the reward function? if we have an objective that values are in the range of 10 and another objective that values are in the range of 1000.</li>
<li>During the training phase, how can we watch the weights updates of a network and the gradient calculation too?</li>
<li>In a multi-agent setting and episodic task, for “dones” vector, it will be set to “True” once all the agents are finished, or once an agent finishes the task done[agent_index]=True in other words, we won´t wait the latest agent to finish to set dones = [True]*number_of_agents</li>
</ol>
<p>Thank you.</p>
How to update a network with a specific loss when there are multiple losses

hsc: <p>Let’s say I have a cascaded model:<br>
e.g., x → modelA → modelB → output &lt;-loss-&gt; gt</p>
<p>now, there are multiple losses (loss1, loss2, loss3) I’d like to calculate but I want to backpropagate only loss1 and loss2 to modelA, not loss3.<br>
The modelB needs to be updated using every loss though.</p>
<p>I could do it using retain_graph=True, but it requires more gpu memory.<br>
I feel like I can do this by masking the gradient of a certain loss function but I’m not sure how I can do this.</p>
<p>Thank you.</p>

ptrblck: <p>Assuming <code>loss1</code> and <code>loss2</code> are supposed to create the gradients for <code>modelA</code> and <code>modelB</code>, you could use <code>loss3.backward(retain_graph=True, inputs=modelB.parameters())</code> and call <code>loss.backward()</code> on the accumulated loss from <code>loss1</code> and <code>loss2</code> afterwards.</p>
<aside class="quote no-group" data-username="hsc" data-post="1" data-topic="140252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hsc/48/45262_2.png" class="avatar"> hsc:</div>
<blockquote>
<p>I could do it using retain_graph=True, but it requires more gpu memory.</p>
</blockquote>
</aside>
<p>It would require more memory if you don’t free the computation graph in the last <code>backward</code> call, which shouldn’t be the case.</p>

hsc: <p>Thank you for the prompt and accurate solution!!!</p>
Trained a BERT model on my domain specific dataset. Got improvement in scores. However, when combined together with TFIDF, the trained BERT model performs poorer than the untrained BERT model. Why?

jungminc88: <p>The task is this:<br>
For each query sentence, we want to find relevant answer sentences from around 800 candidates.<br>
I use a kind of BERT-TFIDF ensemble for this task:<br>
I compute the cosine similarity between the query sentence and answer sentence, based on BERT representation and TFIDF respectively, sum them up, min-max normalize, predict the query-answer pair is relevant if the value exceeds the threshold (this threshold is determined using the validation set).</p>
<p>The mystery is this:<br>
When I make prediction using a general pretrained SentenceBERT model right after I download it (<a href="https://huggingface.co/sonoisa/sentence-bert-base-ja-mean-tokens-v2" class="inline-onebox" rel="noopener nofollow ugc">sonoisa/sentence-bert-base-ja-mean-tokens-v2 · Hugging Face</a>), untrained on my dataset, I get a f2 score of 69.<br>
But after I trained this model on my specific dataset, the scores of the ensemble became lower, even though when comparing the BERT alone (not taking ensemble with TFIDF) the trained model performs better than the untrained one.</p>
<p>Sorry if my description is hard to follow. In short,<br>
Trained BERT model does better than Untrained BERT model, however, when combined with TFIDF, Untrained BERT does better than Trained BERT. How could that be?</p>
<p>I have omitted details in order to keep it concise, but please let me clarify if you think necessary details is lacking. For reference, I put the training loop for the BERT model below.</p>
<p>I would greatly appreciate any suggestions. Thank you!</p>
<pre><code class="lang-auto">    criterion = nn.CosineEmbeddingLoss()
    for i, batch in enumerate(dataloader):
        sent_a_input_ids = batch[0].to(device)
        sent_a_attention_mask = batch[1].to(device)
        sent_b_input_ids = batch[2].to(device)
        sent_b_attention_mask = batch[3].to(device)
        labels = torch.where(batch[4]==0, -1, 1).to(device) # -1 if dissimilar 1 if similar
        a_out = model(input_ids=sent_a_input_ids, attention_mask=sent_a_attention_mask).last_hidden_state
        b_out = model(input_ids=sent_b_input_id, attention_mask=sent_b_attention_mask).last_hidden_state
        a_emb = _mean_pooling(a_out, sent_a_attention_mask)
        b_emb = _mean_pooling(b_out, sent_b_attention_mask)

        loss = criterion(a_emb, b_emb, labels)
        loss.backward()

</code></pre>
A question about model test in DDP

WinKawaks: <p>When I tried DDP to train a model, I found it’s not difficult. For testing, I found different code examples. The simplest is only test your model in GPU 0 and stop all other processes. But I found an error like this:</p>
<blockquote>
<p>[E ProcessGroupNCCL.cpp:566] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1806103 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.</p>
</blockquote>
<p>And I think only testing on the main GPU is a waste of computing resources. So I change my code like this:</p>
<pre><code class="lang-auto">    for epoch_id in range(max_epoch):
        train_loader.sampler.set_epoch(epoch_id)
        for it, (input, category) in enumerate(train_loader):
            model.train()
            optim.zero_grad()
            logits = model(input)
            loss = criterion(logits, category)
            loss.backward()
            optim.step()

            iter += 1
            if iter % 1000 == 0:
                dist.barrier()
                acc = test(test_loader, model)  # dist.reduce here
                if dist.get_rank() == 0:
                    if acc &gt; best_acc:
                        save(model)
</code></pre>
<p>I don’t want to test the model at the end of each epoch because the dataset is really large. I use a counter to test the model every 1000 iterations. So I set <strong>dist.barrier()</strong> to keep multi-processed in sync for testing. And I use <strong>dist.reduce()</strong> to collect the results.</p>
<p>My question is: Is this the right way to test a model in DDP? When my code runs into test() function, are the weights of models in different processes the same or not? Am I using <strong>dist.barrier()</strong> in the right way?</p>

huahuanZ: <p>I think your example should works as expected whatever there is <code>dist.barrier()</code> or not. DDP are synced at <code>loss.backward()</code>.</p>
<p>P.S.<br>
Testing/inference with DDP is somewhat more tricky than training. If using DistributedSampler to scatter your data, You should ensure the number of testing data is divisible to the number of your GPUs, otherwise the results might be incorrect.<br>
Say, there’re 100 batches in your testing set, while there are 8 GPUs (<code>100 % 8 = 4</code>). The DistributedSampler will repeat part of the data and expand it to 104 (<code>104 % 8 = 0</code>) such that the data could be evenly loaded into each GPU.</p>

WinKawaks: <p>Got it. I noticed this special difference between training and testing and I set the GPUs and batch_size divisible. It has worked well so far. Thank you!</p>
PyTorch Models take up more space when sent to device

Zador: <p>Hi a have a simple question about the memory usage of PyTorch models.<br>
I have the following code:</p>
<pre><code class="lang-auto">print('1', process.memory_info().rss / 1e+9)
model = Task()
print('2', process.memory_info().rss / 1e+9)
model.to('cuda')
print('3', process.memory_info().rss / 1e+9)
</code></pre>
<p>This code has the following output:</p>
<pre><code class="lang-auto">1 0.268644352
2 0.269606912
3 2.501251072
</code></pre>
<p>Can somebody explain to me what is going on here? In <code>print('2...</code> am I using the wrong function to check memory info? It seems to me like <code>process.memory_info().rss</code> does not recognize the memory usage until the model weights are sent to my GPU.</p>

ptrblck: <p>Could you check how much memory the model is expected to use? I would guess that the last step reports the loaded libs needed to use the GPU, not necessarily the model parameters alone.</p>

Zador: <p>Thanks for the response. I ended up ignoring this for the problem for the past two weeks and am coming back to it now. This is interesting, I had not realized that extra libs are loaded when you initially send data to a GPU. Do you know where this is discussed?</p>
How does torch.utils.data.distributed.DistributedSampler work

Bika: <p>Hello, I am wondering how these lines of code exactly work ,</p>
<p><code>train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset,                                                                     num_replicas=args.world_size,                                                                     rank=rank)</code><br>
<code>train_loader = torch.utils.data.DataLoader(dataset=train_dataset,                                                batch_size=batch_size,                                                shuffle=False,                                                num_workers=0,                                                pin_memory=True,                                                sampler=train_sampler)</code></p>
<p>if the world_size = 8 having 8 gpus, so this dataloader takes 8 batches at 1 time, giving each GPU 1 distinct batch , like for example we have 16 batches [0,1,2,3,4,5,6,7,8,9…], so GPU 1 take batch 0 , GPU 2 take batch 1, GPU 3 take batch 2 and so on (assuming no shuffling). If not can someone tell me how does it work exactly ?</p>
"if" condition in nn.Sequential

cbd: <p>I am creating network as below. Is it possible to write “if” condition inside nn.Sequential? I want to make customize if condition is true add nn.LeakyReLU else not.</p>
<p>conv_layers.append(nn.Sequential(nn.Conv2d(3, 5, kernel_size=3, stride=1,padding=1),<br>
nn.LeakyReLU(0.2,True) )</p>

ptrblck: <p>Yes, you can use a condition to add a new layer to a <code>list</code> first and then create the <code>nn.Sequential</code> model:</p>
<pre><code class="lang-python">layers = [nn.Conv2d(3, 5, kernel_size=3, stride=1,padding=1)]
cond = True
if cond:
    layers.append(nn.LeakyReLU(0.2,True))
model = nn.Sequential(*layers)
</code></pre>

J_Johnson: <p>It depends on your end goal. If you want a conditional layer in the init function, it will be static during training. (I.e. the condition must be known before training the model and should be a condition that won’t change during training.)</p>
<p>But, if you want the condition to take place during training, you should make the condition during your forward pass. Then it is not possible to place the condition in a nn.Sequential.</p>
<p>I.e.</p>
<pre><code class="lang-auto">def forward(cond, x):
    x=self.layer1(x)
    if cond:
        x=self.relu(x)
    ...
    return x
</code></pre>