akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Adding layers to pre-trained model

Samue1: <p>I have a pre-trained model and want to add additional layers anywhere in the model. Currently I use the following approach where I have a model consisting only of linear layers where I add ReLU activation functions after each linear layer.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Model(nn.Module):

    def __init__(self, num_in, num_hidden, num_out):
        super().__init__()
        self.linear1 = nn.Linear(num_in, num_hidden)
        self.linear2 = nn.Linear(num_hidden, num_hidden)

        self.sequential_module = nn.Sequential(
            nn.Linear(num_hidden, num_hidden),
            nn.Linear(num_hidden, num_hidden),
        )

        self.linear3 = nn.Linear(num_hidden, num_out)

    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        x = self.sequential_module(x)
        x = self.linear3(x)
        return x

def add_relu(module):
    for child_name, child in module.named_children():
        if isinstance(child, torch.nn.Linear):
            sequential_layer = torch.nn.Sequential(child, torch.nn.ReLU(inplace=True))
            setattr(module, child_name, sequential_layer)
        else:
            add_relu(child)

if __name__ == "__main__":
    num_in = 2
    num_hidden = 16
    num_out = 4
    batch_size = 4

    model = Model(num_in, num_hidden, num_out)
    print(model)

    add_relu(model)
    print(model)
</code></pre>
<p>This approach, however, leads to the problem that the models attributes have the wrong name as can be seen in the following output.</p>
<pre><code class="lang-auto">Model(
  (linear1): Sequential(
    (0): Linear(in_features=2, out_features=16, bias=True)
    (1): ReLU(inplace=True)
  )
  (linear2): Sequential(
    (0): Linear(in_features=16, out_features=16, bias=True)
    (1): ReLU(inplace=True)
  )
  (sequential_module): Sequential(
    (0): Sequential(
      (0): Linear(in_features=16, out_features=16, bias=True)
      (1): ReLU(inplace=True)
    )
    (1): Sequential(
      (0): Linear(in_features=16, out_features=16, bias=True)
      (1): ReLU(inplace=True)
    )
  )
  (linear3): Sequential(
    (0): Linear(in_features=16, out_features=4, bias=True)
    (1): ReLU(inplace=True)
  )
  (softmax): Softmax(dim=1)
)
</code></pre>
<p>Here, <code>model.linear1</code> is actually a Sequential layer. How could this be avoided and is there a better way to add layers anywhere in an existing model since I read that using <code>setattr</code> is not a good style.</p>
No speedup when doing backward pass only for a part of the samples in minibatch - why?

HF99: <p>I have a function (see below) which modifies the loss function so that it returns only the loss for the K samples in the minibatch with the <em>lowest loss</em>. The idea is to focus in each optimization step on these samples.</p>
<p>So I do first a forward pass to get the loss value for each sample in the mini-batch, then adapt the loss via the fn. “get_adapted_loss_for_minibatch”.</p>
<p>As the adapted loss takes into account only a certain fraction of the samples in the minibatch (I am using currently 60% of the samples), I was expecting that I get also a measurable speedup during training, as the backward step has to be done only for a fraction of the samples in the minibatch.</p>
<p>But unfortunately this is not the case, the training takes practically the same amount of time as when I am using all samples in the minibatch (so when I do not adapt the loss). I am using a ‘densenet121’ network, and training is done on CIFAR-100.</p>
<p>Am I doing something wrong ? Should I disabled autograd for some samples in the minibatch manually ? I though the ‘topk’ function would do that automatically.</p>
<pre><code class="lang-python">def get_adapted_loss_for_minibatch(loss):
    # Returns the loss containing only the samples of the mini-batch with the _lowest_ loss
    # Parameter 'loss' must be a vector containing the per-sample loss for all samples in the (original) minibatch
    minibatch_size = loss.size()[0]
	r = 0.6 * minibatch_size
	# round r to integer, safeguard if r is 0
    r = max(round(r), 1)
    # The 'topk' function returns the loss for the 'r' samples with the _lowest_ loss in the minibtach
    # See documentation at https://pytorch.org/docs/stable/generated/torch.topk.html
    # Note the 'topk' operation is differentiable, see https://stackoverflow.com/questions/67570529/derive-the-gradient-through-torch-topk
    # and https://math.stackexchange.com/questions/4146359/derivative-for-masked-matrix-hadamard-multiplication
    loss_adapted = torch.topk(loss, r, largest = False, sorted = False, dim = 0)[0]
    # return it
    return loss_adapted
</code></pre>

HF99: <p>Has been resolved.<br>
See <a href="https://stackoverflow.com/questions/68920059/pytorch-no-speedup-when-doing-backward-pass-only-for-a-part-of-the-samples-in-m" class="inline-onebox" rel="noopener nofollow ugc">python - Pytorch: no speedup when doing backward pass only for a part of the samples in minibatch - why? - Stack Overflow</a></p>
Strategies to save memory

arman.avesta: <p>Hi! I’m running a memory-intensive code and I want to save memory as much as possible. If I have multiple convolutional layers:</p>
<p>x = self.conv1(x)<br>
x = self.conv2(x)<br>
x = self.conv3(x)<br>
…</p>
<p>the fact that I’m using the same variable name “x” doesn’t help here, because they’re all kept in PyTorch’s graph. So the block of code above is no different that this block?:</p>
<p>x1 = self.conv1(x)<br>
x2 = self.conv2(x)<br>
x3 = self.conv3(x)<br>
…</p>
<p>If that’s the case, can I save memory by doing this?:</p>
<p>x = self.conv3(self.conv2(self.conv1(x)))</p>
<p>or does this occupy almost the same amount of memory?</p>
<p>I would appreciate your help!</p>

gphilip: <p>As far as I understand, all the three methods you describe use essentially the same amount of memory. Not using different variable names may save the memory used for storing those variable <em>names</em>, not more. And this is an insignificant amount, so you should use as many variable names as will make the code easy to read.</p>
<p>One way to reduce memory requirements is to go through the input in small batches, and to use PyTorch <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">DataLoader</a> to load the data.</p>

arman.avesta: <p>Thank you, Philip! Yes, I’m using PyTorch dataloader and my batch size is only 4, but I’m working with huge 3D brain MRI volumes <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=10" title=":expressionless:" class="emoji" alt=":expressionless:"></p>
<p>Thanks again!<br>
Arman</p>

gphilip: <p>In that case, you could try using <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">checkpointing</a>. Here is a <a href="https://spell.ml/blog/gradient-checkpointing-pytorch-YGypLBAAACEAefHs" rel="noopener nofollow ugc">simple introduction</a> with (hard-on-the-eyes) examples.</p>
Improvement of calculation time

StephDoc: <p>Dear all,</p>
<p>I am generating a Graph with the following command:</p>
<pre><code class="lang-auto">
for i in range(1, 3352):
    for j in range(1, 3352):
        # row = LL[i,:]
        # column = LL[1,:]
        print(i)
        print(j)

        # a = np.array([i, j])

        # c = np.concatenate((c[i,:], a[i,:]), axis=1)

        edge_index_temp = torch.tensor([[i, j]], dtype=torch.long)
        edge_index = torch.cat((edge_index, edge_index_temp))

        x_temp = torch.tensor([[counter]], dtype=torch.float)
        x = torch.cat((x, x_temp))
        counter = counter + 1

        corr_value = LL[i, j]

        edge_attr_temp = torch.tensor([[corr_value]], dtype=torch.float)
        edge_attr = torch.cat((edge_attr, edge_attr_temp))

</code></pre>
<p>but loops are super slow. How can I speed that up?</p>
<p>Best,</p>
Implementation Mismatch between DeepSpeed and PyTorch

Frank_Leeeee: <p>I notice that PyTorch has added zero redundancy optimizer since 1.8. However, it seems the zero optimizer in pytorch is using a different way to partition optimizer states from that in DeepSpeed. DeepSpeed partitions the each parameter equally in the param group and then allocate them to different ranks while PyTorch allocates each parameter directly to a different ranks. That is saying that DeepSpeed does partitioning in a finer granularity.</p>
<p>I am wondering if this is intended and anyone has verified the memory performance?</p>

mrshenli: <p>Hey Frank, yep, the difference is intentional. <code>ZeroRedundancyOptimizer</code> in PyTorch is supposed to be used in conjunction with DDP. Since DDP already holds the full model replica, it will be more memory efficient for <code>ZeroRedundancyOptimizer</code> to directly using those parameters as broadcast buffers instead of creating new intra-parameter shards.</p>

Frank_Leeeee: <p>I see, is there any plan to support zero level 2 and 3 as well as zero offloading in the future?</p>
Model seems to converge fast but catches up quickly

jhp: <p>Hi, I’d like to ask a quick question. Suppose there is a baseline that uses a pre-trained model from B task to do task A. If I attach a pre-trained module in a C task to the B task module and finetune it, the validation acc will initially be higher than baseline, but it will catch up quickly after a few epochs and performance will not improve.<br>
What is the reason it works well at first?<br>
Is it hard to expect a good final result if I combine the pre-trained modules in two different tasks then finetune it for the other task?</p>
Different training results when train data input to the model as eval mode first

Sihun: <p>Hi, I experimented the training phase. When dataloader loads a data for training, I put the data to the model as eval mode first with torch.no_grad() to get some data. And then I put the same data to the model for real training. But the training speed is quite different. When I don’t put the data to the model for imformation, only putting the data for training, it trains properly. But When I put the trick, the model doesn’t train well as if some gradient issue happened… What is the problem?</p>
<p>main code</p>
<pre><code class="lang-auto"> for data in dataloader:

        for i in range(len(model)):
            data[i][0] = data[i][0].to(device, non_blocking=True)
            data[i][1] = data[i][1].to(device, non_blocking=True)

        with torch.cuda.amp.autocast():

            for i in range(len(model)):

                attention_rollout=VITAttentionRollout(model[i], head_fusion='max', discard_ratio=0.9)
                mask.append(attention_rollout(data[i][0]))
                del attention_rollout

            for i in range(len(model)):
                model[i].train(mode=set_training_mode)
                output = model(data[i][0])
            
</code></pre>
<p>VITattentionRollout Code</p>
<pre><code class="lang-auto">class VITAttentionRollout:
    def __init__(self, model, attention_layer_name='attn_drop', head_fusion="mean",
                 discard_ratio=0.9):
        self.model = model
        self.head_fusion = head_fusion
        self.discard_ratio = discard_ratio
        self.attentions = []
        self.handles=[]
        self.attention_layer_name=attention_layer_name

    def __call__(self, input_tensor):
        self.attentions = []
        self.model.eval()
        with torch.no_grad():
            output = self.model(input_tensor)
</code></pre>
Different training result obtained from training simple LSTM in Keras and Pytorch

rickywesker: <p>Hello guys, I’m trying to implement my LSTM model from Keras to Pytorch, but the results in Pytorch seem really bad at the moment. The network is really simple as below.</p>
<pre><code class="lang-auto">model = Sequential()
    model.add(LSTM(10, input_length=shape[1], input_dim=shape[2]))
    # output shape: (1, 1)
    model.add(Dense(10,activation="tanh"))
    model.add(Dense(10,activation="tanh"))
    model.add(Dense(10,activation="tanh"))
    model.add(Dense(10,activation="tanh"))
    model.add(Dense(1,activation="linear"))
    model.compile(loss="mse", optimizer="adam")
    model.summary()
</code></pre>
<p>And I migrate it to the Pytorch framework,</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim,bilstm=False):

        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        self.isBi = bilstm
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True,bidirectional=bilstm).double()
        # for name, param in self.lstm.named_parameters():
        #     if name.startswith("weight"):
        #         nn.init.orthogonal_(param)
        #     else:
        #         pass
       
        self.fc1 = nn.Sequential(nn.Linear(hidden_dim, 10).double(),nn.Tanh())
        self.final_layer1 = nn.Sequential(nn.Linear(10,10).double(),nn.Tanh())
        self.final_layer2 = nn.Sequential(nn.Linear(10,10).double(),nn.Tanh())
        self.final_layer3 = nn.Sequential(nn.Linear(10,10).double(),nn.Tanh())
        self.final_layer4 = nn.Sequential(nn.Linear(10,output_dim).double())
def forward(self, x):
        out, (hn, cn) = self.lstm(x)
        out = out[:, -1, :]
        out = self.fc1(out) 
        out = self.final_layer1(out)
        out = self.final_layer2(out)
        out = self.final_layer3(out)
        out = self.final_layer4(out)
        return out
</code></pre>
<p>The result is really bad. I was wondering if the initializing methods/activation functions used in Keras is different from the one I used in Pytorch(Keras seems to be using hard_sigmoid where Pytorch uses sigmoid?).</p>
<p>Would be really appreciate if somebody could help me with this problem!</p>
Kaggle TPU usage error

Oguzhan: <p>When I tried to import nessecary libraries for TPU I got the error below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b24b1090cfea54db81768d5c73186a6af411607.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b24b1090cfea54db81768d5c73186a6af411607" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b24b1090cfea54db81768d5c73186a6af411607_2_690x216.png" alt="image" data-base62-sha1="1AzRx5oFjBSTrLLTCcymkBRhFSD" width="690" height="216" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b24b1090cfea54db81768d5c73186a6af411607_2_690x216.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b24b1090cfea54db81768d5c73186a6af411607_2_1035x324.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b24b1090cfea54db81768d5c73186a6af411607_2_1380x432.png 2x" data-dominant-color="F8E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1384×434 26 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The installation command below</p>
<pre><code class="lang-auto">!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.9-cp37-cp37m-linux_x86_64.whl
</code></pre>
<p>What can I do?</p>
Performance drops dramatically when switch to model.eval

dongdongtong: <p>Dear all,<br>
I am using pytorch=1.3.1 to train a deeplab-liked resnet50 model with two dataloaders, one is a source-domain dataloader, the other a target-domain dataloader, batch_size=8.</p>
<p>With the help of V100-32G GPU, the performance, eg: Dice score, of the model on both domains gets better and better when I set model.train. But if I switch model.train to model.eval to do evaluation on source/target domain training/validation set, strange thing emerged that the performance of the model drops dramatically to 0! And by the way, if I use the trained model to test on small batch, eg, 4/8 images, model.train mode will get a normal performance but model.eval get 0 performance.</p>
<p>After debugging, the problem was caused by BatchNorm2D layers. If I set model.train to do evaluation on any training or validation set, everything will be ok. But if I set model.eval, once again there the problem is.</p>
<p>I also find that other pytorch users encountered the problem, and their solutions such as set track_running_stats=False when creating BN layers, not to reuse BN layers or set a large batch_size can not help solve me out.<br>
<a href="https://discuss.pytorch.org/t/performance-highly-degraded-when-eval-is-activated-in-the-test-phase/3323">Performance highly degraded when eval() is activated in the test phase</a></p>
<p>So if others also trapped into this problem, I will be very happy that you can provide any suggetion.</p>
<p>Best, dong.</p>

dongdongtong: <p>Frankly, I am doing an unsupervised domain adaptation project, so I figure out what is influencing the performance when using BatchNorm layers just now.</p>
<p>Here is the solution link which is helpful folayersr those UDA project with BatchNorm.<br>
<a href="https://discuss.pytorch.org/t/possible-issue-with-batch-norm-train-eval-modes/31632">Possible Issue with batch norm train/eval modes</a></p>
<p>For a short word, this answer helps me a lot.<br>
<a href="https://discuss.pytorch.org/t/possible-issue-with-batch-norm-train-eval-modes/31632/2">https://discuss.pytorch.org/t/possible-issue-with-batch-norm-train-eval-modes/31632/2?u=dongdongtong</a></p>
<p>But it is very strange to do this because when we deploy a model pretrained on the training set, we don’t hope to change the parameters of the pretrained model, so I am exploring other solutions.</p>
Weighting for Class Imbalance in Semantic Segmentation

rvdinter: <p>Hi! I have a tiny dataset (n=69 images) for semantic segmentation. In this dataset, we have two classes with instances that occur in every image. We also have 3 classes that do not occur in every image, with one class only appearing in 26 images. I have created a custom torch.utils.data.Dataset for this dataset.</p>
<p>Am I able to balance this class imbalance, and how should I do so?</p>
Batch size - GPU - out of memory

msquarederror: <p>Hi,</p>
<p>I have an image classification problem that I have implemented in PyTorch. The code works perfectly when I have a batch size of 10. Any more than that, I get ‘CUDA out of memory’ error.</p>
<p>The images are 4-channeled and I use a DenseNet architecture. However, when I use only 1 channel (of the 4) for training (with a DenseNet that takes 1 channel images), I expected I could go up to a batch size of 40. But unfortunately, I get the out of memory error for any batch size above 10.</p>
<p>Can anyone please explain the logic behind this? I’m confused at this point, any inputs would be greatly appreciated.</p>
<p>Many thanks in advance!</p>

ptrblck: <aside class="quote no-group" data-username="msquarederror" data-post="1" data-topic="130208">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e79b87/48.png" class="avatar"> msquarederror:</div>
<blockquote>
<p>However, when I use only 1 channel (of the 4) for training (with a DenseNet that takes 1 channel images), I expected I could go up to a batch size of 40.</p>
</blockquote>
</aside>
<p>No, that would be a wrong expectation, since you would only save memory in the input tensor as well as the kernel size of the first conv layer.<br>
E.g. for an input size of <code>[10, 4, 224, 224]</code> you would decrease the memory usage as:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10, 4, 224, 224)
print('{:.3f}MB'.format(x.nelement() * 4 / 1024**2))
&gt; 7.656MB
print('{:.3f}MB'.format(x[:, 0].nelement() * 4 / 1024**2))
&gt; 1.914MB
</code></pre>
<p>and the kernel size would also drop by 75%, but would potentially save even less memory.<br>
The first conv layer creates the activation outputs and in particular the channel dimension is defined by the number of kernels used in it. This would mean that no memory savings are expected after the first conv layer.</p>

msquarederror: <p>Oh I see, thanks for clarifying. You are awesome!</p>
Loss function when one class is important than other in Binary classification

bdubey: <p>I have a classification problem related to facial recognition, where real world data is unbalanced and I am interested in minority class. I am employing transfer learning for this. Goal is to Maximize F1-score wrt to minority class.<br>
Initially I got a data set which was imbalanced-initially and applied class weights as below. The F1 score I obtained was 0.65 on a difficult test set (curated to be difficult, but performance on real world data is good). Though Recall was a bit less than what I wanted, so I needed to explore more.</p>
<pre><code class="lang-auto">class_weights = [1/no_samples_minority, 1/no_samples_majority]
class_weights = torch.FloatTensor(class_weights).to(device)
criterion = nn.CrossEntropyLoss(weight = class_weights)
</code></pre>
<p>I combined above data with more data so that the samples are balanced for both positive and negative classes. I doubled the training data in that aspect and was expecting better results. Data quality of new data is also good. Now using balanced weights I get F1 score as 0.3 on same dataset. Basically it classifies almost everything in minority class(Too many false positive). The goal is to maximize both Precision and recall but there is a slight bias towards Precision.</p>
<pre><code class="lang-auto">class_weights = [0.5, 0.5]
class_weights = torch.FloatTensor(class_weights).to(device)
criterion = nn.CrossEntropyLoss(weight = class_weights)
</code></pre>
<p>Thanks in advance for any help. What could be better experiments? Should I play around with weights even for balanced data? If yes, what should be the methodology?  Any other loss function?<br>
Note: Data has many classes and I have converted it into one vs rest binary classification problem.<br>
EDIT: Number of data sample is 80k before adding more data (14/86) and 150k(50/50) after adding data to balance.</p>
Idiomatic way to place custom tensors on GPU

pytorchnewb: <p>Hi, what’s the idiomatic PyTorch way of putting tensor fields on custom models on the GPU? An example should clarify what I’m asking.</p>
<p>Say I have a class like this:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
  def __init__(self):
    # ...
    self.my_tensor = torch.rand(...)

  def forward(self, x):
    # ...
    out = out + self.my_tensor
    # ...

# ...
model = model.to(device)
model.my_tensor = model.my_tensor = to(device)
</code></pre>
<p>Wondering if there’s a better way to put all custom tensors on device without manually needing to run <code>.to(device)</code> on each.</p>

ptrblck: <p>All <code>nn.Parameter</code>s and buffers (registered via <code>self.register_buffer</code>) will automatically be pushed to the specified device when calling <code>model.to(device)</code>. The difference would be that the former are trainable (<code>requires_grad</code> is set to <code>True</code> by default) while the latter do not. In case you don’t want to store the buffer in the <code>state_dict</code>, you could use <code>self.register_buffer('name', tensor, persistent=False)</code>.</p>

pytorchnewb: <p>Thanks! That’s exactly the kind of thing I was looking for.</p>
Need something like gather/select for a multidimensional case

shahbuland: <p>Tried using gather and select for this but couldn’t get them to work here. For simplicity, I’ll just put my code here first:</p>
<pre><code class="lang-auto"># hidden is a [B, N, D] transformer hidden state
# eot_inds is a [B] int tensor of indices
# y is meant to be [B, D]  (embeddings of certain positions specified by eot_inds in each sequence from transformer hidden state)
B, N, D = hidden.shape
y = torch.zeros(B, D)
for i in range(B):
    y[i] = hidden[i, eot_inds[i]]
</code></pre>
<p>How would I “pytorchify” this so I didn’t need a for loop?</p>

ptrblck: <p>If I understand your use case correctly, this should work:</p>
<pre><code class="lang-python">B, N, D = 2, 3, 4
hidden = torch.randn(B, N, D)
y = torch.zeros(B, D)
eot_inds = torch.randint(0, N, (B,))

for i in range(B):
    y[i] = hidden[i, eot_inds[i]]
    

out = hidden[torch.arange(hidden.size(0)), eot_inds]
print((out == y).all())
&gt; tensor(True)
</code></pre>
Creating a Block-Diagonal Matrix

mbp28: <p>Hey,</p>
<p>I am wondering what the fastest way to create a block-diagonal matrix would be using PyTorch?</p>
<p>As an example, given</p>
<pre><code class="lang-auto">a = torch.FloatTensor([[1,2],[3,4]])
b = torch.FloatTensor([[5,6],[7,8]])
</code></pre>
<p>I would like to create<br>
<code>c = torch.FloatTensor([[1,2,0,0],[3,4,0,0],[0,0,5,6],[0,0,7,8]])</code></p>
<p>Any help much appreciated.<br>
Many thanks,<br>
Max</p>

MarxMeier: <p>Hey,<br>
I figured out a solution. It’s not a perfect one but it works.<br>
Create your blocks:</p>
<p>b_1 = torch.FloatTensor([[1,2],[3,4]])<br>
b_2 = torch.FloatTensor([[5,6],[7,8]])</p>
<p>By concatenating the b_i’s with zeros you to get rows:<br>
r_1 = torch.cat((b_1, torch.zeros((2, 2))), dim = 1)<br>
r_2= torch.cat((torch.zeros((2, 2)), b_2), dim = 1)</p>
<p>And then you concatenate the rows to get c<br>
c = torch.cat((r_1, r_2), dim = 0)</p>
<p>I hope one Max was able to help the other Max <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

yulkang: <p>I made a vectorized implementation on Gist: <code>block_diag.py</code> at <a href="https://gist.github.com/yulkang/2e4fc3061b45403f455d7f4c316ab168" rel="nofollow noopener">https://gist.github.com/yulkang/2e4fc3061b45403f455d7f4c316ab168</a></p>
<p>For an up-to-date version, check <code>block_diag()</code> in <code>numpytorch.py</code> in my <code>pylabyk</code> library: <a href="https://github.com/yulkang/pylabyk/blob/master/numpytorch.py" rel="nofollow noopener">https://github.com/yulkang/pylabyk/blob/master/numpytorch.py</a></p>

vinaykumar2491: <p>PyTorch has an implementation for this already. Checkout this:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.block_diag.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.block_diag.html</a></p>
<p>An example from the docs:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; A = torch.tensor([[0, 1], [1, 0]])
&gt;&gt;&gt; B = torch.tensor([[3, 4, 5], [6, 7, 8]])
&gt;&gt;&gt; C = torch.tensor(7)
&gt;&gt;&gt; D = torch.tensor([1, 2, 3])
&gt;&gt;&gt; E = torch.tensor([[4], [5], [6]])
&gt;&gt;&gt; torch.block_diag(A, B, C, D, E)
tensor([[0, 1, 0, 0, 0, 0, 0, 0, 0, 0],
        [1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 3, 4, 5, 0, 0, 0, 0, 0],
        [0, 0, 6, 7, 8, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 7, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 1, 2, 3, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 4],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 5],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 6]])
</code></pre>
Avoiding retain_graph=True with a memory bank

NicoAdamo: <p>Hi there, I’m trying to reproduce a training procedure with a “memory bank”, where, at each training step, a fixed number of feature embeddings are dequeued from a fixed-size list and used in loss computation, and then the feature embeddings of the current training data are pushed into the back of the list. The problem is that if I don’t detach the feature embeddings before I push them into the memory bank, I get the error “trying to backprop through the computation graph twice”, which of course makes sense as the features are from an earlier version of the graph. But is there any way to do this without setting retain_graph = True, which runs out of memory very quickly, or detaching the samples in the memory bank, which defeats the point of training this way?</p>
<p>Thank you!</p>

ayalaa2: <p>What’s the algorithm you’re attempting to reproduce?</p>

Varal7: <p>Would <a href="https://pytorch.org/docs/stable/checkpoint.html?highlight=checkpoint#torch.utils.checkpoint.checkpoint" rel="noopener nofollow ugc">checkpointing</a> be useful in your situation?</p>
<p>Edit: actually, I think you might be doing something conceptually wrong here if you need to backward twice through the graph.</p>

maddyschiappa: <p>Any progress on this? I am running into the same issue. I am trying to dynamically increase the examples for negative pairing in a NCE loss using a memory bank that collects sample latent embeddings so I can reduce the mini batch size that is passed through the model.</p>
Deploy inference with a trained model

beto16: <p>Hi I’ve trained a model and saved the model to run inference with the trained model in order to get the loss value and calculate other metrics. I load the model correctly, but I don’t know how to proceed.</p>

ptrblck: <p>I don’t know where you are stuck, but given that you want to calculate the loss as well as other metrics you could reuse the previously used training code, call <code>model.eval()</code> and skip the optimization of the model.</p>
iOS: undefined _cpuinfo_ symboles in torch_cpu

madbats: <p>I’m compiling a library which links to pytorch and is then used to build an iOS app.<br>
However when i build the project in XCode i get the following errors:</p>
<pre><code class="lang-auto">Undefined symbols for architecture x86_64:
"_cpuinfo_get_processors_count", referenced from:
      caffe2::getDefaultNumThreads() in libtorch_cpu.a(ThreadPool.cc.o)
"_pthreadpool_get_threads_count", referenced from:
      caffe2::PThreadPool::get_thread_count() const in libtorch_cpu.a(pthreadpool-cpp.cc.o)
  "_pthreadpool_create", referenced from:
      caffe2::PThreadPool::PThreadPool(unsigned long) in libtorch_cpu.a(pthreadpool-cpp.cc.o)
      caffe2::PThreadPool::set_thread_count(unsigned long) in libtorch_cpu.a(pthreadpool-cpp.cc.o)
  "_pthreadpool_destroy", referenced from:
      caffe2::PThreadPool::PThreadPool(unsigned long) in libtorch_cpu.a(pthreadpool-cpp.cc.o)
  "_pthreadpool_parallelize_1d", referenced from:
      caffe2::PThreadPool::run(std::__1::function&lt;void (unsigned long)&gt; const&amp;, unsigned long) in libtorch_cpu.a(pthreadpool-cpp.cc.o)
  "_cpuinfo_initialize", referenced from:
      caffe2::getDefaultNumThreads() in libtorch_cpu.a(ThreadPool.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
</code></pre>
<p>Checking the <code>libtorch_cpu.a</code> using <code>nm</code> I found that <code>_cpuinfo_initialize</code> and<code> _cpuinfo_get_processors_count</code> are undefined.</p>
<p>To build my pytorch libraries I’ve set <code>NO_API to OFF</code> as well as <code>BUILD_MOBILE_AUTOGRAD to ON</code> in the root CMakeLists file of pytorch.<br>
I then run <code>IOS_PLATFORM=SIMULATOR BUILD_PYTORCH_MOBILE=1 BUILD_LITE_INTERPRETER=0 ./scripts/build_ios.sh</code>.<br>
When building my library, I link to <code>libtorch_cpu.a</code> and <code>libc10.a</code></p>
<p>Any ideas ?</p>

Linbin: <p>Hi Matthieu, you need to link to libcupinfo.a. Check our nightly build: <a href="https://ossci-ios-build.s3.amazonaws.com/libtorch_ios_nightly_build.zip" rel="noopener nofollow ugc">https://ossci-ios-build.s3.amazonaws.com/libtorch_ios_nightly_build.zip</a> (in install/lib folder)</p>
How to set gradients to zero without optimizer?

RylanSchaeffer: <p>Similar to this StackOverflow question <a href="https://stackoverflow.com/questions/54648053/how-to-set-gradients-to-zero-without-optimizer" class="inline-onebox" rel="noopener nofollow ugc">pytorch - How to set gradients to Zero without optimizer? - Stack Overflow</a>,</p>
<p>I want to reset a computational graph to be able to call <code>loss.backward()</code> multiple times, without using an optimizer. How can I do this?</p>
<p>Take the example given in that post. The example has 2 variables, <code>x</code> and <code>t</code>, but I’m interested in zeroing all gradients.</p>
<pre><code class="lang-auto">import torch

x = torch.randn(3, requires_grad = True)
t = torch.randn(3, requires_grad = True)
y = x + t
z = y + y.flip(0)

z.backward(torch.tensor([1., 0., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)
x.grad.data.zero_()  # both gradients need to be set to zero 
t.grad.data.zero_()
z.backward(torch.tensor([0., 1., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)
</code></pre>
Image proccessing

adalia: <p>Before embedding input images in prototypical networks, images are loaded and processed. I do not understand this code ‘x = 1.0 - torch.from_numpy(x)’ in data loading function.  I am new in this field. Can you explain its function. Thank you.</p>
<p>x = x.rotate(float(rot))<br>
x = x.resize((28, 28))</p>
<pre><code>shape = 1, x.size[0], x.size[1]
x = np.array(x, np.float32, copy=False)
x = 1.0 - torch.from_numpy(x)
x = x.transpose(0, 1).contiguous().view(shape)</code></pre>

ptrblck: <p>The mentioned line of code subtracts the <code>x</code> array from a constant scalar <code>1.0</code>.<br>
It’s a bit hard to tell why this is used without seeing any context.</p>
RuntimeError: The size of tensor a (1447) must match the size of tensor b (2) at non-singleton dimension 0

talhaanwarch: <p>When i define weight in BCELoss I got this error. How to resolve it? I have 2 classes.<br>
criterion=nn.BCELoss(weight=torch.FloatTensor([2.0, 1.0]))<br>
<code>RuntimeError: The size of tensor a (1447) must match the size of tensor b (2) at non-singleton dimension 0</code></p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html?highlight=bceloss#torch.nn.BCELoss">docs</a>:</p>
<blockquote>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor"><em>Tensor</em></a><em>,</em> <em>optional</em>) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.</p>
</blockquote>
<p>so you would have to provide a weight value for each sample in the batch.</p>
K40 is not supported by PyTorch

Nick_8229: <p>The latest Pytorch seems not support K40 gpu card anymore, which is due to release build is not compiled with compute mode 3.5. I understand build from source could be one solution. However it is very annoyed in many ways. Build from source will certainly close the door to many new users. If the binary size is the only reason to let the cm 3.5 be removed. Can we please add it back.</p>

Brando_Miranda: <p>I get a not very helpful message:</p>
<pre><code class="lang-auto">Tesla K40m with CUDA capability sm_35 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the Tesla K40m GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
</code></pre>
<p>I went to the page but it doesn’t tell me which version of pytorch to install or anything like that. Can we get more precise instructions what exactly we need to down grade?</p>
<p>Any help? <a class="mention" href="/u/ptrblck">@ptrblck</a> can you ping the person that might know? Sorry to bug you directly but I had a hunch you might know.</p>
<p>Thanks!</p>

ptrblck: <p>You would have to build PyTorch from source, as this compute capability is not shipped in the binaries anymore since PyTorch 1.5, if I’m not mistaken.<br>
Contributions to improve these error messages are always welcome.</p>

Brando_Miranda: <p>ok got it :’( thanks regardless.</p>

vsant: <p>Hello,</p>
<p>Just to know if anyone was succeeded in building PyTorch (from versions 1.6 to 1.9) from source to work with K40 GPUs. We are trying doing this with gcc version 9.0 and the compilation goes on up to a certain point where an error says that <code>numa.h</code> is missing.</p>
<p>Does anyone has experienced this?</p>
<p>Thank you.</p>
Pytorch shuffle layer

Praveen_Joshi: <p>Hi,</p>
<p>I am trying to concatenate the activation from n number of clients before passing to my host-DNN model. How can I build the shuffling layer after the concatenation layer at my host-DNN model so that it can shuffle the activation coming from the clients?</p>
<p>Also, while getting the gradients back from host-DNN, gradients should be unshuffled and passed back.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65f078abea8fc0aa1e663c1719f75da64322e074.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/65f078abea8fc0aa1e663c1719f75da64322e074" title="pytorch_discussion"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65f078abea8fc0aa1e663c1719f75da64322e074_2_690x111.jpeg" alt="pytorch_discussion" data-base62-sha1="exNqMP8tTdXskxnkYNoQFo42CPi" width="690" height="111" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65f078abea8fc0aa1e663c1719f75da64322e074_2_690x111.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65f078abea8fc0aa1e663c1719f75da64322e074.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65f078abea8fc0aa1e663c1719f75da64322e074.jpeg 2x" data-dominant-color="EBEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_discussion</span><span class="informations">1018×164 25 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Should we also implement a backward function for the shuffle layer or it will be taken care by autograd?</p>

Praveen_Joshi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> : Apologies for tagging you to this issue but I found your amazing answers over the forum and I couldn’t help myself to not to do that. Can you please provide me some pointers?</p>

ptrblck: <p>I’m unsure what the <code>ShufflingLayer</code> would exactly do, but assuming you would permute the input activations, you could most likely write a custom <code>nn.Module</code> and apply e.g.:</p>
<pre><code class="lang-python">def forward(self, x):
    x = x[:, torch.randperm(x.size(1))]
    return x
</code></pre>
<p>to shuffle the data in <code>dim1</code>.</p>

Praveen_Joshi: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. In my scenario, host-DNN gets avtication from different clients and at very first layer, concatenation of activation vectors from different client is stacked together.  At second layer shuffling of activation is done. Now shuffling of activation should be done wrt to labels received from the clients.</p>
Loss does not descent in pytorch, but does in tensorflow

HANHAOHAN: <p>pytorch code:</p>
<pre><code class="lang-python">def test_torch():
    import torch
    import torch.nn as nn
    from torch.utils import data
    from torch.utils.data import dataloader
    data = torch.randn(100000, 2)
    y = torch.sin(data[:, 0]*data[:, 1]).unsqueeze(1)
    print(y.shape)
    data = torch.cat([data, y], dim=1)

    model = nn.Sequential(
        nn.Linear(2, 100),
        nn.ReLU(),
        nn.Linear(100, 100), 
        nn.ReLU(),
        nn.Linear(100, 100), 
        nn.ReLU(),
        nn.Linear(100, 100), 
        nn.ReLU(),
        nn.Linear(100, 100), 
        nn.ReLU(),
        nn.Linear(100, 100), 
        nn.ReLU(),
        nn.Linear(100, 100), 
        nn.ReLU(),
        nn.Linear(100, 1)
    )

    for p in model.parameters():
        print(p.shape, p.requires_grad)

    criterion = torch.nn.L1Loss()
    opt = torch.optim.SGD(model.parameters(), 0.001)

    dl = dataloader.DataLoader(data, batch_size=500)
    for epoch in range(100):
        for d in dl:
            pred = model(d[:, :2])
            loss = criterion(pred, d[:, 2])
            opt.zero_grad()
            loss.backward()
            opt.step()
            print("epoch:[{}], loss: {}".format(epoch, loss.item()))
</code></pre>
<p>tensorflow version:</p>
<pre><code class="lang-python">def test_tf():
    import tensorflow as tf
    import numpy as np
    from tensorflow import keras
    model = tf.keras.Sequential([
        keras.layers.Dense(units=10, activation='relu', input_shape=[2]),
        keras.layers.Dense(units=10, activation='relu'),
        keras.layers.Dense(units=10, activation='relu'),
        keras.layers.Dense(units=10, activation='relu'),
        keras.layers.Dense(units=10, activation='relu'),
        keras.layers.Dense(units=10, activation='relu'),
        keras.layers.Dense(units=10, activation='relu'),
        keras.layers.Dense(units=1),
    ]
    )
    model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="mean_squared_error")
    xs = np.random.randn(100000, 2).astype(np.float32)
    ys = np.sin(xs[:,0] * xs[:, 1])
    model.fit(xs, ys, epochs=100, batch_size=500)
</code></pre>

arman-yekkehkhani: <p>I think it is not a fair comparison. Model architectures are different (the number of units in Keras dense layer does not match the PyTorch linear layer), and optimizers are not the same either(SGD vs Adam). Please fix these two issues, hope you get the same result.</p>

HANHAOHAN: <p>Thank you for your answer!<br>
It’s a pity you didn’t try to run the code. In fact I have done more tests on the above code and it just does not converge, even when I reduce the problem to a identity mapping.<br>
As for the unfairness you mentioned, it was just a small mistake that I didn’t notice.</p>

ptrblck: <p>Did you try to fix the issues and rerun the code?<br>
As described before, the posted approaches use another model, optimizer, and criterion, while only the data seems to be equal.</p>

HANHAOHAN: <p>Got it !<br>
There is a bug in my code:<br>
loss = criterion(pred, d[:, 2])<br>
What’s more, SGD needs larger learning rate.</p>
<p>Thanks for all !</p>
Cpp_extension load for distributed environment

shaibagon: <p>Hi,<br>
I am running in a DataDistributedParallel environment, where each worker imports a JIT module and therefore calls <code>torch.utils.cpp_extension.load(... )</code>. As a result, each worker compiles the JIT module from scratch. This take ages. Is there a way to efficiently cache this <code>load</code>ing?</p>
Rocm predict so slow！

Vaen_Joe: <p>Hi, everyone. I am meeting a really confusing question!<br>
I use the Super Computing Center in China to accelerate my training process, which deploys the DCU Card and installs the pytorch ROCm whose version is 4.2(beta).<br>
But I encountered a srange issue:<br>
<img src="https://user-images.githubusercontent.com/46133615/130589787-5b55809f-f4d2-481f-a299-e5d57bdd9e58.png" alt="image" width="690" height="492"><br>
As the picture above shows, The value in left circle  is time of <code>out = model(in)</code> , and the valu in right circle is time of <code>backward()</code>.<br>
In NVIDIA’s 960, the two value above are both 0.0x. I think this is so weired. The input that was feed to network is an array whose shape is (2500, 12).<br>
Another, I test on two different Super Compute Center in China. They all have the same problem.<br>
Actually, I know that maybe I have to make some optimization if I want to move from cuda to dcu. But I can’t find the solution and I wanna know why.<br>
Thanks vvvvvvery much!!!</p>
Transfer Learning Approaches

Hafsa_Moontari_Ali: <p>I am trying to understand the two common approaches of transfer learing. I am reading this: <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.9.0+cu102 documentation</a>.</p>
<p>I am wondering what is the role of pre-trained weights(like ImageNet) in “ConvNet as fixed feature extractor” approach? Say, I am using resnet-18 (pre-trained on ImageNet), freezing all the convolutional layers in the network and keeping the last fully connected layer which is trained with random weight initialization(as stated in the given link). In this specific case of transfer learning, since the convolution layer is freezed, the weights are not being updated. Besides, the last fully connected layer is trained with random weight initialization based on the target dataset. So, in this scenario, what is the role of pre-trained weights from ImageNet? Or is it only about using the pre-defined renet-18 architecture? Kindly reply.</p>

ptrblck: <p>Freezing the conv layers (feature extractor) and training the linear layers (classifier) only would be a typical form of transfer learning. The idea is that the pretrained model (using the ImageNet dataset) learned to extract useful features from these images, i.e. the conv layers (with the non-linear activations, pooling layers, as well as potentially normalization layers) are detecting features from the spatial inputs, which can then be “combined” in the linear layers to classify the new dataset.</p>

Hafsa_Moontari_Ali: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
What exactle does "inplace" do when set to True/False?

Superklez: <p>As the title says, what’s the difference when setting <code>inplace = True</code> in <code>nn.ReLU</code> and <code>nn.Dropout</code>? Would it affect training in some way?</p>
<p>Edit: If I were to use <code>nn.Conv2d</code> or <code>nn.Dropout</code> in <code>nn.Sequential</code>, it wold be better to use <code>inplace=True</code>, correct?</p>
<p>Edit 2: I meant <code>nn.ReLU</code>, not <code>nn.Conv2d</code>.</p>

ptrblck: <p><code>nn.Conv2d</code> wouldn’t have the <code>inplace</code> argument (at least not in the <code>torch.nn.Conv2d</code> definition).<br>
The <code>inplace</code> argumen in e.g. <code>nn.Dropout</code> layers (or other functions) will apply the method on the input “inplace”, i.e directly on the values in the same memory locations without creating a new output.<br>
This could save some memory, but might also be disallowed if the inputs are needed to be unmodified for the gradient calculation (inplace operations would also disallow the JIT to fuse operations, if I’m not mistaken).<br>
A small example is given here:</p>
<pre><code class="lang-python">drop = nn.Dropout(p=0.5, inplace=False)

x = torch.randn(1, 5)
print(x)
&gt; tensor([[ 0.4276,  0.5935, -0.0205,  0.2411, -1.3081]])

out = drop(x)
print(out)
&gt; tensor([[0.8551, 1.1870, -0.0000, 0.0000, -0.0000]])
print(x)
&gt; tensor([[ 0.4276,  0.5935, -0.0205,  0.2411, -1.3081]])


drop = nn.Dropout(p=0.5, inplace=True)
out = drop(x)
print(out)
&gt; tensor([[ 0.8551,  0.0000, -0.0410,  0.0000, -0.0000]])
print(x) # also changed
&gt; tensor([[ 0.8551,  0.0000, -0.0410,  0.0000, -0.0000]]) 
</code></pre>

Superklez: <p>My apologies, I meant <code>nn.ReLU</code>. I tried using <code>nn.Dropout</code> with <code>inplace=True</code> in <code>nn.Sequential</code> and got an error saying:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>Maybe this is caused by setting <code>inplace=True</code>?</p>

ptrblck: <p>Yes, this is most likely caused by the usage of <code>inplace=True</code>, if the inputs are needed in an unmodified state to calculate the gradients as previously mentioned.<br>
<a href="https://discuss.pytorch.org/t/gru-model-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/57312/9">This post</a> gives a small example why inplace ops are disallowed for specific (chains of) operations.</p>