akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Handling data loading in active learning

Olga_Petrova: <p>Hi, I am trying to implement a (pool-based) active learning scenario. I start with L, a small set of labeled samples (images, let’s say), and a lot of unlabeled ones. At each iteration I train the model on L, then rank the unlabeled samples in terms of their “usefulness”, label the top N most useful ones, add them to L, and repeat. My question is - what would be the best way to handle loading the training data since the size of L keeps increasing throughout training?</p>
<p>I can see two options:</p>
<ol>
<li>
<p>Have all the data (labeled and unlabeled) in the same DataLoader (with placeholder labels for the initially unlabeled samples). Keep track of what gets labeled, and mask out the unlabeled samples during training. This sounds incredibly wasteful when n_unlabeled &gt;&gt; n_labeled.</p>
</li>
<li>
<p>Create a new labeled DataLoader at the end of each iteration, once the selected N samples have been newly labeled and added to the L set. That’s what I am leaning towards, but… is there any better way? What if I was doing stream-based active learning, I would not want to recreate the DataLoader each time I label a new sample :-/</p>
</li>
</ol>
<p>Thanks!</p>

ptrblck: <p>I would go for the second approach.<br>
Note that creating a <code>DataLoader</code> instance is cheap. If you are also lazily loading the data, the creation of your <code>Dataset</code> should also be of no concern.</p>
Multi level mask input shape

xian_kgx: <p>Suppose I have a model that takes as input a multi-level mask like a segmentation map where each pixel can take one of N &gt; 2 classes.</p>
<p>My question is should the input shape be (batch_size, num_classes, height, width) or can I instead use only one channel to encode the class index with an input shape of (batch_size, 1, height, width).</p>
<p>With output shape its pretty clear that it is (batch_size, num_classes, height, width). But what about the input shape?</p>
<p>Please advice.</p>

ptrblck: <p>You can encode the input information as you want.<br>
E.g. you could certainly pass the input in a one-hot encoded way, as a single channel image, or even a color image, where each color represents a certain class.</p>
<p>The “right” approach also depends on your current model.<br>
I.e. are you creating a model from scratch? If so, try out different approaches and check their performance on the validation set.<br>
On the other hand, if you are thinking about fine tuning a model, most pretrained models use 3 input channels, so that you could have to adapt your input shape to it if you don’t want to replace/manipulate the first convolution.</p>

xian_kgx: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for your great advice. I’m trying to reimplement a model from a paper from scratch by literally following whatever is reported in the paper. However, some details seemed to be missing from the paper.</p>
<p>Thank you once again, and stay safe.</p>
Inverse of a tensor

Javier_Rodriguez_Pui: <p>Hi everyone, I am trying to do the inverse of a tensor element-wise.<br>
like this:</p>
<pre><code class="lang-auto">sample[i].pow_(-1)

</code></pre>
<p>Unfortunately, the precision that I am looking for is not achieved although is float64 type.</p>
<p>Data example:<br>
Having this tensor:<br>
tensor([[[[1631., 1635., 1639.,  …, 4023., 4025., 4027.],<br>
[1629., 1634., 1638.,  …, 4020., 4022., 4024.],<br>
[1627., 1632., 1636.,  …, 4017., 4019., 4021.],<br>
I get this result:<br>
tensor([[[[0.0006, 0.0006, 0.0006,  …, 0.0002, 0.0002, 0.0002],<br>
[0.0006, 0.0006, 0.0006,  …, 0.0002, 0.0002, 0.0002],<br>
[0.0006, 0.0006, 0.0006,  …, 0.0002, 0.0002, 0.0002]</p>
<p>I don’t know how to fix it… I thought that would be a cast conversion problem but I cannot find the problem.</p>

ptrblck: <p>Your output might just be cropped and you could increase the printed decimals via <code>torch.set_printoptions(precision=10)</code>.</p>
How to initiate weights for new layer when loading pretrained model

fangyh: <p>I add more convs but still want to use the pretrained model.<br>
The error says:</p>
<pre><code class="lang-auto">Missing key(s) in state_dict:  "Missing key(s) in state_dict: "layer1.0.ae_module.fc.0.weight", "layer1.0.ae_module.fc.0.bias"
</code></pre>

ptrblck: <p>Would it be possible to load the pretrained <code>state_dict</code> before including the new layers?<br>
This would avoid these kind of errors.</p>

rzhang63: <p>Hi ptrblck,</p>
<p>I’m trying to add a dropout layer after each of the conv2d layers in the pretrained mobilenet_V2, but I’m not sure how to code this. Could you please help me out here? Thank you so much!</p>

ptrblck: <p>The clean way would be to derive a custom class from the desired base class, add the dropout layer in its <code>__init__</code>, and change the <code>forward</code> method.<br>
The hacky way might be <a href="https://discuss.pytorch.org/t/can-i-remove-a-layer-from-a-pre-trained-model-while-loading-the-model-weights/57899/9">this post</a>.</p>

rzhang63: <p>Hi,</p>
<p>Thank you so much for the reply! I tried the method you provided in that other post, it worked in terms of adding dropout layers after all conv2d layers. But when I tried to send the model to GPU I got ‘cuda out of memory’. I’m guessing, is it because of the deepcopy?</p>
<p>And I’m not sure that I understand the first solution you mentioned, could you please provide me with some code examples?</p>
<p>Thank you!</p>

ptrblck: <p>The OOM might come from the deepcopy.<br>
How close the the memory limit are you before copying the modules?<br>
Are you able to train the original model, since the forward and backward passes should also use some memory (which should be larger than copying some modules).</p>
<p>Have a look at <a href="https://discuss.pytorch.org/t/pipe-pretrained-model-to-custom-layers/67836/5">this post</a> for an example how to derive a custom class from a <code>torchvision</code> model.</p>
Indexing a 2D tensor with a smaller 2D tensor

legoh: <p>Hi folks, I’d like some help on indexing a 2D tensor. Suppose we have the following tensors</p>
<p><code>d = torch.tensor([[118, 175,   1],  [118, 188,   0], [ 66, 201,   1], [ 94, 204,   1],  [ 94, 206,   0]]) e = torch.tensor([[66, 201, 0.1], [94, 206, 0.2], [1, 23, 0.6],                 [118, 188, 0.3],                 [2, 3, 0.1],                 [3, 1, 0.2],                 [94, 204, 0.8],                 [118, 175, 0.7]])</code></p>
<p>Tensor <code>e</code> is a set of predictions from a model where the first 2 columns are sort of an ID and tensor <code>d</code> is a set of labels with first 2 columns as an ID. So I have less labels than predictions here and I’d like to only calculate the loss for the real labels. Hence, I need to index tensor <code>e</code> by tensor <code>d</code>'s first 2 columns. Is there a way to do this? Or would using a 3D tensor be better?</p>

charan_Vjy: <p>Can you clarify “I’d like to only calculate the loss for the real labels” with an example?</p>

legoh: <p>Sure. So we have the following tensors</p>
<pre><code class="lang-auto">d = torch.tensor([[118, 175, 1], 
                  [118, 188, 0], 
                  [ 66, 201, 1], 
                  [ 94, 204, 1], 
                  [ 94, 206, 0]]) 

e = torch.tensor([[66, 201, 0.1], 
                  [94, 206, 0.2], 
                  [1, 23, 0.6], 
                  [118, 188, 0.3], 
                  [2, 3, 0.1], 
                  [3, 1, 0.2], 
                  [94, 204, 0.8], 
                  [118, 175, 0.7]])
</code></pre>
<p>For tensor d, the first 2 columns are the indexes, and third column is the true label. Hence, I’d like to get a prediction tensor such that we index tensor e, and get</p>
<pre><code class="lang-auto">prediction = torch.tensor([[118, 175, 0.7], 
                           [118, 188, 0.3], 
                           [ 66, 201, 0.1], 
                           [ 94, 204, 0.8], 
                           [ 94, 206, 0.2]])
</code></pre>
<p>I can then calculate the CE-loss by taking</p>
<pre><code class="lang-auto">bce = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
loss = bce(prediction[:, 2], d[:, 2])
</code></pre>
Difference between seq_len and input_size in LSTM?

Baka_Moon: <p>I am going through the docs:<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM</a></p>
<p>but am a little confused:</p>
<p>Is seq_len the number of time units or number of words in a sentence<br>
and input_size the embedding size or features?</p>

vdw: <p>Yes, in case of sentences, <code>input_size</code> derives from the size of your word or character embeddings. For example, if you use Word2Vec or GloVe embeddings of size 300 (i.e., each word is represented as a vector with 300 dimensions), the <code>input_size</code> of your LSTM needs to be 300.</p>
Torchvision build error

Edwardmark: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/vision/issues/2076#issuecomment-611287005" target="_blank" rel="nofollow noopener">github.com/pytorch/vision</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/vision/issues/2076#issuecomment-611287005" target="_blank" rel="nofollow noopener">nvcc fatal   : Unknown option 'Wl,--no-as-needed,/app/install/libtorch/lib/libtorch.so'</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-08" data-time="10:03:13" data-timezone="UTC">10:03AM - 08 Apr 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/Edwardmark" target="_blank" rel="nofollow noopener">
          <img alt="Edwardmark" src="https://avatars1.githubusercontent.com/u/15247551?v=4" class="onebox-avatar-inline" width="20" height="20">
          Edwardmark
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">nvcc fatal : Unknown option 'Wl,--no-as-needed,/app/install/libtorch/lib/libtorch.so'
To Reproduce
docker pull nvcr.io/nvidia/tensorrt:19.02-py3
docker run -ti nvcr.io/nvidia/tensorrt:19.02-py3 bash
mkdir /app
cd /app
git clone https://github.com/pytorch/vision.git
wget https://download.pytorch.org/libtorch/cu92/libtorch-cxx11-abi-shared-with-deps-1.4.0%2Bcu92.zip
unzip libtorch to /app/
Steps...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">help wanted</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: c++ frontend</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: build</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How can I rotate random degree in 4D tensor (B,C,H,W)

hadxu: <p>How can I rotate random degree in 4D tensor (B,C,H,W)? Thanks</p>

wizardk: <p>see here:<br>
<a href="https://pytorch.org/docs/stable/torchvision/transforms.html?highlight=rotate#torchvision.transforms.functional.rotate" rel="nofollow noopener">torchvision.transforms.functional.rotate</a></p>

hadxu: <p>4D tensor, not PIL.Image.</p>

derEitel: <p>I typically use scipy’s rotate function:<br>
<a href="https://docs.scipy.org/doc/scipy-0.16.1/reference/generated/scipy.ndimage.interpolation.rotate.html" class="onebox" target="_blank" rel="nofollow noopener">https://docs.scipy.org/doc/scipy-0.16.1/reference/generated/scipy.ndimage.interpolation.rotate.html</a></p>
<p>Unfortunately it’s not very fast and you have to apply it on a numpy tensor, i.e. before you convert it to a pytorch tensor.</p>

wizardk: <p>What about this:</p><aside class="quote" data-post="1" data-topic="22476">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john1231983/48/4040_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-rotate-90-and-270-degrees-of-5d-tensor/22476">How to rotate 90 and 270 degrees of 5D tensor</a> 
  </div>
  <blockquote>
    According to <a href="https://github.com/pytorch/pytorch/pull/7873" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/pull/7873</a>, the flip function have been merged in pytorch. I am using version 0.41 and it works for 5D tensor. My 5D tensor is BxCxDxHxW. For flipping, I just use 
images=images.flip(4) # flip along 4th dim
images=image.flip(3) # flip along 3rd dim

However, the above functions are equivalent to the vertical and horizontal axis. How could I rotate the tensor 90 degrees or 270 degrees along 3rd or 4th dim? 
Thanks!
  </blockquote>
</aside>


Yuechen_ZHANG: <p>The function torch.rot90(tensor, number_of_times_of_rotation:int, rotate_axis:list) is available</p>
Nvlink problem issue

kartheek: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ! I’m very new to Nvlink, do we need to write any Initial code to enable the usage of nvlink while running our model and could you please give me any reference example code of a sample model which is running on nvlink for data parallelism using nvlink, if you have any.<br>
Thank you</p>

ptrblck: <p>NVLink should be used automatically for all device2device communications.<br>
You can check the status via <code>nvidia-smi nvlink -s</code> and see the connectivity matrix via <code>nvidia-smi topo -m</code>.</p>
<p>While the first command should give you the current status and bandwidth, the latter one will show you shich GPUs are connected to each other with which technology.</p>
AttributeError: 'NoneType' object has no attribute 'shape'

kun_yang: <p>When i train my data which like cocodataset,an error occurred,can someone help me fix it ?</p>
<p>The error information is :</p>
<p>libpng error: Read Error<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 142, in <br>
main()<br>
File “tools/train.py”, line 138, in main<br>
meta=meta)<br>
File “/cache/user-job-dir/codes/mmdetection/mmdet/apis/train.py”, line 111, in train_detector<br>
meta=meta)<br>
File “/cache/user-job-dir/codes/mmdetection/mmdet/apis/train.py”, line 305, in _non_dist_train<br>
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)<br>
File “/home/work/anaconda3/lib/python3.6/site-packages/mmcv/runner/runner.py”, line 371, in run<br>
epoch_runner(data_loaders[i], **kwargs)<br>
File “/home/work/anaconda3/lib/python3.6/site-packages/mmcv/runner/runner.py”, line 271, in train<br>
for i, data_batch in enumerate(data_loader):<br>
File “/home/work/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 582, in <strong>next</strong><br>
return self._process_next_batch(batch)<br>
File “/home/work/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 608, in _process_next_batch<br>
raise batch.exc_type(batch.exc_msg)<br>
AttributeError: Traceback (most recent call last):<br>
File “/home/work/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py”, line 99, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/home/work/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py”, line 99, in <br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/cache/user-job-dir/codes/mmdetection/mmdet/datasets/custom.py”, line 132, in <strong>getitem</strong><br>
data = self.prepare_train_img(idx)<br>
File “/cache/user-job-dir/codes/mmdetection/mmdet/datasets/custom.py”, line 145, in prepare_train_img<br>
return self.pipeline(results)<br>
File “/cache/user-job-dir/codes/mmdetection/mmdet/datasets/pipelines/compose.py”, line 24, in <strong>call</strong><br>
data = t(data)<br>
File “/cache/user-job-dir/codes/mmdetection/mmdet/datasets/pipelines/loading.py”, line 28, in <strong>call</strong><br>
results[‘img_shape’] = img.shape<br>
AttributeError: ‘NoneType’ object has no attribute ‘shape’</p>
<p>Thanks,anyway!</p>

Kushaj: <p>Probably something wrong with the data loading process. Try testing your DataLoader for any errors.</p>

kun_yang: <p>Thanks ,i’ve solved it.</p>

suyung: <p>When i train my data which like VOCdataset，I met the same error,could you share me the solution?</p>
When implementing VAE, which one of torch.distributions.normal and torch.distributions.multivariate_normal.MultivariateNormal should I use?

MMTaro: <p>When implementing VAE as below, I have two questions.</p>
<pre><code class="lang-auto">class VAE(nn.Module):
    def __init__(self, data_size = data_size):
        super(VAE, self).__init__()
        self.data_size = data_size
        
        self.fc1 = nn.Linear(self.data_size ** 2, 400)
        self.fc21 = nn.Linear(400, 20)
        self.fc22 = nn.Linear(400, 20)
        
        self.fc3 = nn.Linear(20, 400)
        self.fc4 = nn.Linear(400, self.data_size ** 2)
     
    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, self.data_size ** 2))
        dist = Normal(mu, logvar.exp_())
        # dist = MultivariateNormal(mu, ??)
        return self.decode(dist.rsample()), dist
</code></pre>
<ol>
<li>In the forward function, which is correct? using normal or MultivariateNormal ?</li>
<li>When using MultivariateNormal is correct, how should we create the same number of diagonal matrices as the batch size used for the variance covariance matrix?(The part written in “??” in the code.) I could not find a function in pytorch that could accomplish it.</li>
</ol>

fermat97: <p>Hi, did you find your solution? I wanted to post the question but I found yours.</p>

muammar: <p>I also would be interested in knowing if you found a solution for this.</p>

TimoK: <p>P(z) is a multivariate Gaussian ~N(z|0, I).<br>
The trick is the z vectors in the batch are independent, normally-distributed values. [1]<br>
Therefore samples of z can be drawn from a simple distribution, i.e., N(0, I) using torch.randn_like.<br>
Given the mean and co-variance we first sample e ∼N(0, I), then<br>
compute z = mean(X) + 0.5 sigma(X) ∗ e.</p>
<p>std = torch.exp(0.5 * variance_ln)<br>
e = torch.randn_like(variance_ln).to(device)   # latent prior p(z): ~N(0,I)<br>
z = mu + e * std  # “reparameterization trick”</p>
<p>[1] <a href="https://arxiv.org/pdf/1606.05908.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1606.05908.pdf</a></p>
Torch.nn.functional.affine_grid : affine matrix that has rotation, translation, and scaling

jackck: <p>Hi,</p>
<p>I am trying to use torch.nn.functional.affine_grid to generate a grid that will be passed to torch.nn.functional.grid_sample.</p>
<p>Consider:</p>
<p>+(1) Small square image of size (K,K)</p>
<p>+(2) Rotated Box defined by (x_center, y_center, w_box, h_box, angle) where x_center, y_center is in the coordinate system of the input image described below.  w_box is the width of the box and h_box is the height of the smaller image.  Angle is rotation applied to the box in degrees.</p>
<p>+(3) Input image of size (w_input, h_input).</p>
<p>The problem I am solving is to paste (1) into (3) such that (1) is rotated/scaled/translated to fit the rotated box defined in (2).</p>
<p>Any pixel that falls outside of the input should be discarded.  What is the 2x3 affine matrix that I should use to generate the affine grid?  Thanks.</p>
Help simple Keras to Pytorch model implementation

Cristina_Segalin: <p>Hi all,</p>
<p>I am trying to reimplement a Keras model in Pytorch. I have done my version but I would like to compare it with someone else version more expert in  Keras than me to see if it may has less params etc</p>
<pre><code>N_CONTEXT=61
N_INPUT=26
N_HIDDEN=256
N_OUTPUT=414
model = Sequential()
model.add(TimeDistributed(Dense(N_HIDDEN), input_shape=(N_CONTEXT, N_INPUT)))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.1))

model.add(GRU(N_HIDDEN, return_sequences=False))  
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.1)) 

model.add(Dense(N_OUTPUT)) 
model.add(Activation('linear'))
</code></pre>
<p>Thank you</p>

ptrblck: <p>Do you see any discrepancy between the number of parameters of your Keras model and the PyTorch model you have written?<br>
If so, which layer is causing the difference?</p>

Cristina_Segalin: <p>Yes I do.<br>
I have two implementations for the TimeDistributed layer.<br>
The keras models has about 650K params, my first version has 8M params and the second 385K params.</p>

ptrblck: <p>Could you post the PyTorch implementation here, so that we could compare them and see what’s causing this issue?</p>
Remove gradient from torch.mul

kc-luo: <p>Say a and b are two tensors with gradient. Is there a difference between the following two methods in defining a new tensor from multiplication if I don’t need a gradient for c? Does the second method save</p>
<blockquote>
<ol>
<li>c = torch.mul(a, b).detach()</li>
<li>c = torch.zeros_like(a, requires_grad=False)<br>
c = torch.addcmul(c, a, b)</li>
</ol>
</blockquote>
Initial hidden state h_0 for multi-layer GRU/LSTM

leey: <p>I am trying to use a 2-layer GRU network, and I want to use two different initial hidden states [[h_0_1],<br>
[h_0_2]]<br>
for the two GRU layers. I am not sure if h_0_1 will be connected to the GRU that takes my direct input, or will it be connected to the GRU that generates the output?</p>
<p>Thanks for the help!</p>
Why .grad is None?

csailnadi: <p>I cannot figure out why .grad is None while the weights have required_grad = True?</p>
<pre><code class="lang-auto">class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out


def train_cnn(data, lr=0.01, epochs=100, momentum=0.9, weight_decay=0):
    model = LeNet()
    model.cuda()
    criterion = torch.nn.CrossEntropyLoss()

    model.train()
    optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=momentum, weight_decay=weight_decay)

    for e in range(epochs):
        for data in trainloader:
            x, y = data
            x = x.cuda()
            y = y.cuda()

            outputs = model(x)
            loss = criterion(outputs, y)
            optimizer.zero_grad()
            loss.backward()

            print(model.conv1.weight[0].requires_grad)
            print(model.conv1.weight[0].grad)
            exit()
</code></pre>
<p>The output of this is</p>
<pre><code class="lang-auto">True
None
</code></pre>

albanD: <p>Hi,</p>
<p>The problem is that you index the weights when you do [1]. So you get a new Tensor that does not have a .grad field.<br>
<code>model.convos[1].weight.grad</code> should not be None right?</p>

csailnadi: <p>Oh I see, grad is stored for the entire tensor, and not it’s inner dimensions,  sorry, silly mistake.<br>
I wanted to get a grad for a specific filter, but I should have sliced the grad itself.</p>
Get subset of tensor with different number of columns

Shiro: <p>Hi,</p>
<p>I am trying to extract from a tensor a subset without using a loop. But each row of the subset will not necesseraly have the same number of columns. I was wondering if it is possible without a loop. (threfore I need to get a list at the end)<br>
Let’s say I have a tensor of size (5,10). I want to extract the element :</p>
<ul>
<li>row 0 : columns [2:6]</li>
<li>row 1 : columns [2:8]</li>
<li>row 2 : columns [2:9]</li>
<li>row 3 : columns [1:3]</li>
<li>row 4 : columns [0:6]</li>
</ul>
<p>With a loop it is easy</p>
<pre><code class="lang-auto">T = .. # tensor
begin = [2,2,2,1,0]
end = [6,8,9,3,6]
for i, x in enumerate(T):
       res.append(x[begin[i]: end[i]])
</code></pre>

albanD: <p>Hi,</p>
<p>Given that the result is a list. Then you won’t be able to use an efficient Tensor function to do this I’m afraid.</p>
Evolution strategies in pytorch

Stapo: <p>I want to train a model using evolution strategies instead of backpropagation. I assume this would be done by doing something like differential evolution with the parameters of the model. I cannot find a lot of resources and implementations of that in pytorch. I would appreciate any directions of how to get this implemented.</p>
<p>Also, how reasonable is it to expect this to work on a RNN model (GRU)?</p>
Good way of initializing weights

Fibbi: <p>I want to initialize a network which contains linear layers, convolutional layers, pooling and so on.<br>
My current method of initialization looks like:</p>
<pre><code class="lang-python">params = list(model.parameters())     
    for i in range(0,len(params)):
        val = params[i].data.ndim
        if (val &gt; 1):
            torch.nn.init.xavier_uniform_(params[i])
</code></pre>
<p>Is this a good way of initializing the weights or is there a better general method?</p>

googlebot: <p>Your method may work with some networks (though it should handle 1d parameters too) but it is more usual to delegate initialization to submodules for more precise control, as other nn.init initializers all may have their use, depending on distribution of submodule inputs (or preceding “activation function”).</p>

googlebot: <p>Should have mentioned the obvious: torch.nn built-in modules initialize their own parameters (with initializers hardcoded in reset_parameters()), so your code is only useful if you’re not happy with default initializers, or create parameters manually, but then again - you would probably want to fine-tune them locally.</p>
Tensor reshaping without corrupting data

Topsoil: <p>I’ve noticed that different methods of reshaping a tensor (.reshape(), .view(), .permute()) seem to sometimes somehow corrupt (somehow change its order) data inside tensor, but i was not able to pinpoint when they do so, because reshaping it back restores original</p>
<p>Can someone explain to me what is a preferable method, and in what case i should use each method, please?</p>

JuanFMontesinos: <aside class="quote quote-modified" data-post="2" data-topic="54157">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/difference-between-view-reshape-and-permute/54157/2">Difference between view, reshape and permute</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    reshape tries to return a view if possible, otherwise copies to data to a contiguous tensor and returns the view on it. From the <a href="https://pytorch.org/docs/stable/torch.html#torch.reshape" rel="noopener nofollow ugc">docs</a>: 

Returns a tensor with the same data and number of elements as input , but with the specified shape. When possible, the returned tensor will be a view of input . Otherwise, it will be a copy. Contiguous inputs and inputs with compatible strides can be reshaped without copying, but you should not depend on the copying vs. viewing behavior. 
See <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view" rel="noopener nofollow ugc"> torch.Tensor.view…</a>
  </blockquote>
</aside>

<aside class="quote" data-post="2" data-topic="30749">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/difference-between-2-reshaping-operations-reshape-vs-permute/30749/2">Difference between 2 reshaping operations (reshape vs permute)</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    Im not sure about reshape. 
Permute is a multidimensional rotation saying somehow. It keeps the data ordering. View (which is another reshaping method) maps from one dimensionality to another sequentially reading data from the upper dimensions to the lower ones. 
So if you want fuse two dimensions into one, you have to apply it over contiguous dimensions or u will modify the data ordering
  </blockquote>
</aside>
How to make ignite trainer handle error batch instead of stop the program?

ThaiThien: <p>Sometime, the single sample that does not work with model (dimension mismake) or None in batch (because error in loading data due to corrupt).</p>
<p>Instead of throwing exception and stop the program, how to just do handle (log, write something) and carry-on with Ignite trainer.</p>
<p>What should I do ? If possible, please give example in code.</p>

vfdev-5: <p><a class="mention" href="/u/thaithien">@ThaiThien</a> currently there is no proper way to skip a single iteration (Maybe it could worth a feature request).<br>
However, still what can be done is the following:</p>
<pre><code class="lang-python">
def process_function(engine, batch):
     if not is_correct_batch(batch):
         # let's return the previous output =&gt; wont break other handlers using state.output
         # but metrics computation will be impacted... 
         return engine.state.output

     # otherwise, continue as if everyhing is OK
        
engine = Engine(process_function)
</code></pre>
<p>Hope this could help with your problem.</p>
<p>PS: feel free to open a feature request on the github if a proper handling of such situation is needed for you (and maybe others). Thanks</p>
How to modify data tensor efficiently to replace LSTM-char-embedding with CNN-char-embedding?

jivnesh_Sandhan: <p>In my model, I want to replace <code>LSTM-based-char-embedding</code> with <code>CNN-based-char-embedding</code>. For this how should I modify data tensor efficiently?</p>
<p>I have tensor of dimension <code>[batch_size, max_seq_length]</code> and I want to convert it to <code>[batch_size, max_word_length, max_char_length]</code></p>
<p>For example, consider these two sequences, <code>[I love mango, I eat mango]</code> with <code>batch_size=2</code> . For <code>LSTM-based-embedding</code>, this will be encoded  <code>[batch_size, max_seq_length]</code>  as follows where <code>space</code> is represented by <code>50</code> and <code>padding token</code> by <code>51</code>.</p>
<pre><code class="lang-auto">data = [[1,50,21,24,11,3,50,15,14,16,43,12,51], [1,50,3,14,4,50,15,14,16,43,12,51,51]]
</code></pre>
<p>In addition, I have a matrix that stores the positions where words are ending in sequence (position of <code>space</code> token)</p>
<pre><code class="lang-auto">splits = [[1,6,12],[1,5,11]] 
</code></pre>
<p>How should I utilize this <code>splits</code> matrix to find out <code>maximum character_length</code>  efficiently in the complete batch? In this case, it is 5.</p>
<p>Finally,  expected conversion of <code>data</code> should be <code>[batch_size, max_word_length, max_char_length]</code>:</p>
<pre><code class="lang-auto">data_modified = [[[1,51,51,51,51],[21,24,11,3,51],[15,14,16,43,12]],[[1,51,51,51,51],[3,14,4,51,51],[15,14,16,43,12]]
</code></pre>
<p>This can be done using loops. But it will decrease the training speed drastically. Can we do more efficiently?</p>
Warm-up: numpy explained

desmond13: <p>Hi, I am new to pythorch and I am going through the tutorials. I find them useful but, in my opinion, they would be slightly better if they would explain them line by line. Maybe it is only my problem though.</p>
<p>After finishing the 60 minute bliz I am doing the " LEARNING PYTORCH WITH EXAMPLES" ones. I am starting from <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#warm-up-numpy" rel="nofollow noopener">this</a> one and I would like to understand better the following lines of code. Why do we do them?</p>
<p>I don’t understand why we need to do all the following to compute the backpropagation</p>
<pre><code class="lang-auto">    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h &lt; 0] = 0
    grad_w1 = x.T.dot(grad_h)
</code></pre>
<p>Thanks In advance!</p>
Improving Custom Pooling Layer to run quicker

martinodonnell: <p>I have been working on channel pooling and have been able to recreate a method used in papers to do channel pooling instead of max pooling. Essentially the layer will take in the input, and do max pooling across the channels. The compression value, 2 in my case, will combine 2 channels and calculate the max values across each channel. This will occur to each pair of channels to create a final output. I have created the code below but I believe there must be a better way to do this which will speed up the process.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e1705d151e67577fd454c1e02e90ba960824730.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0e1705d151e67577fd454c1e02e90ba960824730" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1705d151e67577fd454c1e02e90ba960824730_2_648x499.jpeg" alt="image" data-base62-sha1="20E0ExMz2BsWFO4IU7c09N5eW1a" width="648" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1705d151e67577fd454c1e02e90ba960824730_2_648x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e1705d151e67577fd454c1e02e90ba960824730_2_972x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e1705d151e67577fd454c1e02e90ba960824730.jpeg 2x" data-dominant-color="D7D5D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1186×914 97.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">class ChannelPool(nn.MaxPool1d):

    def forward(self, input):
        n, c, w, h = input.size()
        c = int(c/2)
        output = torch.zeros(n, c, w, h).cuda()

        index_pos = 0
        #Compress input to output tensor
        for index in range(0,input.size()[1],2):
            output[0][index_pos] = torch.max(input[0][index],input[0][index+1])     
            index_pos +=1 
        return output
</code></pre>
Batch normalisation and minibatching

Peter_Featherstone: <p>How can I adjust batchnorm2d layers or the final model or the optimizer when minibatching?<br>
For example I have code that looks like this:</p>
<pre><code class="lang-auto">batchsize = 64
minibatchsize = 4
nminibatches = (batchsize + minibatchsize - 1) // minibatchsize
optimizer.zero_grad()
for b, (data, target) in enumerate(dataloader):
    out = net(data)
    loss += compute_loss(out,target)
    if (b % nminibatches) == (nminibatches-1):
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        ....
</code></pre>
<p>There is a problem with batch normalisation layers and small minibatches. The result of forwarding 16 batches of size 4 is different to forwarding 1 batch of size 64. How can this be corrected?</p>

ptrblck: <p>You could try to adapt the <code>momentum</code> so that the stats from each batch have less influence on the running estimates.<br>
Alternatively, you could also use <code>torch.utils.checkpoint</code> to trade compute for memory and try to increase the batch size.</p>

Peter_Featherstone: <p>Maybe a useful feature when inferring a model would be batch size so that when forwarding an input of a given minibatch size, batchnorm layers know to carry on averaging until the total batch size is reached. I think tensorflow for swift does this</p>
UnpicklingError: invalid load key, '<'

hunar: <p>i tried to use pretrained InceptionResnetV1</p>
<pre><code>resnet = InceptionResnetV1(
   classify=True,
   pretrained='vggface2',
   num_classes=len(dataset.class_to_idx)
).to(device)
</code></pre>
<p>but i get the following error :</p>
<blockquote>
<p>magic_number = pickle_module.load(f, **pickle_load_args)<br>
if magic_number != MAGIC_NUMBER:<br>
raise RuntimeError(“Invalid magic number; corrupt file?”)<br>
UnpicklingError: invalid load key, ‘&lt;’.</p>
</blockquote>
<p>is there something i’ve missed to add or remove? thanks for advice</p>

ptrblck: <p>Could you post a link to the model code or alternatively the model definition?<br>
I’m not sure, what’s causing this issue, but the error might point to a pickle file, which wasn’t stored in PyTorch.</p>

hunar: <p>thank you fixed the issues</p>
"TypeError: tensor is not a torch image.", when using: transforms.Normalize

ProGamerGov: <p>The size of my tensor is:<br>
(3L, 512L, 682L)</p>
<p>I first remove the batch dimension:</p>
<pre><code class="lang-auto">B, C, H, W = output_tensor.size()
output_tensor = output_tensor.view(C, H, W)
</code></pre>
<p>And then I try to run <code>transforms.Normalize</code> on the tensor:</p>
<pre><code class="lang-auto">Normalize = transforms.Compose([transforms.Normalize(mean=[-0.40760392156, -0.45795686274, -0.48501960784], std=[1,1,1]) ]) # Subtract BGR

output_tensor = Normalize(output_tensor)
</code></pre>
<p>But this results in an error:</p>
<pre><code class="lang-auto">    output_tensor = Normalize(output_tensor)
  File "/usr/local/lib/python2.7/dist-packages/torchvision/transforms/transforms.py", line 42, in __call__
    img = t(img)
  File "/usr/local/lib/python2.7/dist-packages/torchvision/transforms/transforms.py", line 118, in __call__
    return F.normalize(tensor, self.mean, self.std)
  File "/usr/local/lib/python2.7/dist-packages/torchvision/transforms/functional.py", line 158, in normalize
    raise TypeError('tensor is not a torch image.')
TypeError: tensor is not a torch image.
</code></pre>
<p>Edit, I forgot to take the tensor out of it’s Variable:</p>
<pre><code>output_tensor = Normalize(output_tensor.cpu().data)</code></pre>

Porschen_Hund: <aside class="quote no-group" data-username="ProGamerGov" data-post="1" data-topic="15638">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/progamergov/48/3670_2.png" class="avatar"> ProGamerGov:</div>
<blockquote>
<p>Edit, I forgot to take the tensor out of it’s Variable:</p>
</blockquote>
</aside>
<p>That function seems to only work on PIL images for Dataloader class when importing torchvision data. It doesn’t work on Tensors as far as I tried at this moment.</p>
<p>It might be faster to write your own normalize function.</p>

mathematics: <p>It was solved when i changed transforms in order as</p>
<pre><code class="lang-auto">transformed = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
])
</code></pre>
<p>After transforming to tensor and then using normalize. As from the stack overflow answer.</p>
Convert Model to Sequential

McLawrence: <p>I have a model which has some submodules, which again have normal layers as submodules and the <code>torch.nn.functional</code> API.<br>
To use a library which iterates over the model, I have to convert it into a <code>Sequential</code> model or make it iterable another way.</p>
<p>To do this, I wrote this function:</p>
<pre><code class="lang-auto">def model_to_list(model, modules):
    if len(list(model.children())) == 0:
        modules.append(model)
        return
    for module in model.children():
        model_to_list(module, modules)

modules = []
model_to_list(model, modules)
model = nn.Sequential(*modules)
</code></pre>
<p>This includes all all <code>nn.Module</code>s but not functional parts of the model. Is it possible to make a model iterable including the functional parts, or do I have to replace them in my model by <code>nn.Module</code>s? While pooling and dropout can be done using <code>nn.Module</code>s, the <code>view</code> operation can not (afaik).</p>
<p>Additionally, my general approach seems flawed, as the <code>modules</code> list contains the <code>Module</code>s in the order of initialization, not usage in the forward pass. Is there maybe a different approach to solving this problem?</p>

SimonW: <p>you have to replace them. for things like view, you can use a custom module.</p>
<p>it’s also worth noting that .children() might not give you the correct execution order.</p>

sujay: <p>If .children() is not the correct way to get the execution order, then what is the correct way to get it ? <a class="mention" href="/u/simonw">@SimonW</a></p>
How can I concatenate two model then save it separately?

Igo312: <p>I defining two model which are encoder and classfication respectively. I want use encoder’s ouput as the input of the classfication then train the two model. After training I want to save the two model separately.</p>
<p>here’s my two model</p>
<pre><code class="lang-python"># encoder model 
class source(nn.Module):
    def __init__(self, initial_channel):
        super(source, self).__init__()
        self.initial_channel = initial_channel


        def conv_bn(inp, oup, stride):
            return nn.Sequential(
                nn.Conv2d(inp, oup, 3, stride, 1, bias=False),
                nn.BatchNorm2d(oup),
                nn.ReLU(inplace=True)
            )
        self.model = self.model = nn.Sequential(conv_bn(initial_channel, 64, 2),
                                   conv_bn(64, 128, 2),
                                   conv_bn(128, 128, 1),
                                   conv_bn(128, 256, 2),
                                   conv_bn(256, 256, 1),
                                   nn.AdaptiveAvgPool2d([1,1]))
    def forward(self, x):
        x = self.model(x)
        x = x.view(-1,256)
        return x
</code></pre>
<pre><code class="lang-python"># classfication model
class classfication(nn.Module):
    def __init__(self, class_num):
        super(classfication, self).__init__()
        self.class_num = class_num

        self.fc1 = nn.Sequential(nn.Linear(256, 64),
                                 nn.ReLU(inplace=True),
                                 nn.Dropout(0.5))
        self.fc2 = nn.Sequential(nn.Linear(64, class_num), )

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x
</code></pre>
<p>And following my main model to describe what I want to do in detail.</p>
<pre><code class="lang-python">encoder = source(initial_channel)
cls = classfication(classnum)

model = torch.nn.Sequential(encoder, cls)
train(model)
save(encoder)
save(cls)
</code></pre>

ptrblck: <p>You could store the <code>state_dicts</code> of both models separately.<br>
Note that it’s recommended to save the <code>state_dict</code> rather than the complete model as described <a href="https://pytorch.org/docs/stable/notes/serialization.html">here</a>.</p>

braindotai: <p>You can do as as <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned. Also you can just save their states_dict in a single file as-</p>
<pre><code class="lang-python">torch.save({'encoder': encoder.state_dict(), 'cls': cls.state_dict()}, 'both_models_state_dict.pt')
</code></pre>
<p>And then load it later as-</p>
<pre><code class="lang-python">both_models_state_dict = torch.load('both_models_state_dict.pt')
encoder.load_state_dict(both_models_state_dict['encoder'])
cls.load_state_dict(both_models_state_dict['cls'])
</code></pre>
Loss does not decrease beyond a point

satinder147: <p>I am training a segmentation model. I am training a UNet like architecture with a pre-trained encoder. I am using segmentation_models_pytorch for a model architecture. I am using 1000 images. My optimizer is Adam with a learning rate of 1e-3 and batch size 16.<br>
However, my loss does not decrease beyond a point. I also tried using <code>lr_scheduler.</code> <code>ReduceLROnPlateau</code> but no luck. I trained the model for 1800 epochs but the loss did not decrease but <code>ReduceLROnPlateau</code>  decreased the learning rate.</p>