akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it normal to have such fluctuating train, test and dev loss charts?

username: Mahsa_Afsharizade
cooked: <p>I have two data sets D1 and D2. Each data set is composed of two parts: sentences and scores. I want to learn scores for the sentences by an LSTM network in PyTorch. I have examined four scenarios:</p>
<blockquote>
<pre><code>        train      validation      test
    ----------------------------------------
</code></pre>
<ol>
<li>
<pre><code>   90% of D1     5% of D1        5% of D1         
</code></pre>
</li>
<li>
<pre><code>   90% of D2     5% of D2        5% of D2
</code></pre>
</li>
<li>
<pre><code>   90% of D1     5% of D1        5% of D2
</code></pre>
</li>
<li>
<pre><code>   90% of D2     5% of D2        5% of D1
</code></pre>
</li>
</ol>
</blockquote>
<p>Here is the results for these four situations:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad44334b6119e7e19ece2590c29210901427e64f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad44334b6119e7e19ece2590c29210901427e64f" title="4plots"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad44334b6119e7e19ece2590c29210901427e64f_2_576x500.png" alt="4plots" data-base62-sha1="oIMIHxvfjH9vSfFA6qJpBr3DoLR" width="576" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad44334b6119e7e19ece2590c29210901427e64f_2_576x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad44334b6119e7e19ece2590c29210901427e64f_2_864x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad44334b6119e7e19ece2590c29210901427e64f.png 2x" data-dominant-color="F7F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4plots</span><span class="informations">911×790 125 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now I have some questions.</p>
<blockquote>
<p>A) Is it normal for my charts to fluctuate so much?<br>
B) Should test loss be up or below of dev loss? (they have different positions in each case)<br>
C)I think that train loss should be always below the test loss. But this is not true always about my cases. Does it show a problem about my model?<br>
D) Why doesn’t dev loss show a clear descending or ascending trend?<br>
E) It seems that test loss has an ascending trend in Fig (4).why?<br>
F)Is it possible that my model is not constructed truly and therefore these Figures are not normal?</p>
</blockquote>
Understanding pytorch-data

username: samra-irshad
cooked: <p>Hi all,</p>
<p>I am a bit new to pytorch and trying to understand the the data loading module. With reference to the tutorial available here: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a> , it can be seen that when the dataset class for this specific example is opening each image in  def <strong>getitem</strong>(self, idx) method and converting to tensor. I want to know when we initialize the object of FaceLandmarksDataset class, do we get all images opened and mapped to dict at once or <strong>getitem</strong>(self,idx) is implemented when we explicitly call the object with a specific id?</p>
<p>class FaceLandmarksDataset(Dataset):<br>
“”“Face Landmarks dataset.”""</p>
<pre><code>def __init__(self, csv_file, root_dir, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.landmarks_frame = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = transform

def __len__(self):
    return len(self.landmarks_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_name = os.path.join(self.root_dir,
                            self.landmarks_frame.iloc[idx, 0])
    image = io.imread(img_name)
    landmarks = self.landmarks_frame.iloc[idx, 1:]
    landmarks = np.array([landmarks])
    landmarks = landmarks.astype('float').reshape(-1, 2)
    sample = {'image': image, 'landmarks': landmarks}

    if self.transform:
        sample = self.transform(sample)

    return sample</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="samra-irshad" data-post="1" data-topic="68588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samra-irshad/48/27993_2.png" class="avatar"> samra-irshad:</div>
<blockquote>
<p>I want to know when we initialize the object of FaceLandmarksDataset class, do we get all images opened and mapped to dict at once or <strong>getitem</strong> (self,idx) is implemented when we explicitly call the object with a specific id?</p>
</blockquote>
</aside>
<p>When you initialize the <code>FaceLandmarkDataset</code>, its <code>__init__</code> method will be called.<br>
In your example, the <code>csv_file</code> will be loaded and assigned to <code>self.landmarks_frame</code> as well as the root directory and transformation.<br>
Indexing the <code>dataset</code> via <code>dataset[0]</code> or iterating the <code>DataLoader</code> will call into <code>__getitem__</code> and lazily load each sample.<br>
This makes sure you can work with huge datasets, which wouldn’t fit into your system memory.</p>

username: samra-irshad
cooked: <p>Thanks, i understand now</p>
Resnet del softmax

username: slavavs
cooked: <p>I want to use resnet18, but I don’t need softmax. How do I get the output from the last conv layer?</p>

username: ptrblck
cooked: <p>The <code>torchvision.models.resnet18</code> does not use a softmax as the last non-linearity, but uses an <code>nn.Linear</code> layer to output the logits.</p>
Custom upscaling of an image through a pretrained decoder network

username: Joshua_Clancy
cooked: <p>Hello! So this is a bit of weird one…</p>
<p>I have a (b, c, 7, 7) tensor. For each “pixel” in this tensor I am going to put the (c, 1, 1) through a pretrained decoder to produce a (1, 4, 4) pixel image. In this way I intend to turn the (batch, channel, 7, 7) into (batch, 1, 28, 28). The problem is that I can not figure out how to do this upscaling through my custom function (the pretrained decoder).</p>
<p>perhaps an image will help:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43044edc6c2c3f2856a8a5bdaa79257ed57198df.png" data-download-href="https://discuss.pytorch.org/uploads/default/43044edc6c2c3f2856a8a5bdaa79257ed57198df" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43044edc6c2c3f2856a8a5bdaa79257ed57198df_2_690x330.png" alt="image" data-base62-sha1="9yRdFvJj6wTTF4OcflbUXXfFjiL" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43044edc6c2c3f2856a8a5bdaa79257ed57198df_2_690x330.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43044edc6c2c3f2856a8a5bdaa79257ed57198df_2_1035x495.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43044edc6c2c3f2856a8a5bdaa79257ed57198df_2_1380x660.png 2x" data-dominant-color="F3F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2010×964 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To be clear I already have the trained function/decoder, I can feed the pixels into that decoder one by one and get the resulting 4 by 4. But I need to figure out a way to unfold the resulting tensors back into the larger image shape. (Perhaps some sort of nested unfold.) That is what I am having trouble with. If you could help out or simply point me in the right direction I would be very grateful.</p>
<p>Thanks in advance!</p>

username: Joshua_Clancy
cooked: <pre><code class="lang-auto">#%%
b = 1
c = 6
x_in = 7
y_in = 7

# place holder tensor
x = torch.rand((b, c, x_in, y_in), requires_grad=False) # 1, 6, 7, 7
x = x.view((b*x_in*y_in, 6)) # 49, 6
new =  decoder(x, outputDict)
print(new.shape) # 49, 1, 4, 4
new = new.view((b, 1, x_in*4, y_in*4))
print(new.shape) # 1, 1, 28, 28
</code></pre>
<p>This is where I am at now. Though I am not sure whether these reshapes will keep the pixels in there respective places…</p>

username: ptrblck
cooked: <p>The first <code>view</code> operation might interleave the values, so you should permute the channel dimension to the last dimension first:</p>
<pre><code class="lang-python">x = torch.rand((b, c, x_in, y_in), requires_grad=False)
x = x.permute(0, 2, 3, 1).view(b*x_in*y_in, 6)
</code></pre>
<p>I’m not sure how the output is calculated, so it’s hard to tell, if the last <code>view</code> is working correctly.</p>
Many channel input CNN out of memory

username: himat
cooked: <p>I have a CNN that processes inputs that have 10 input channels (these are not regular images), and then does segmentation on the pixels to categorize each pixel as a class.</p>
<p>But I am running out of memory on my GPU.</p>
<p>What is the recommended approach to take here? I think my input itself is so large due to being many pixels in length, width, and depth.</p>
<p>Should I grid the input into smaller boxes, and then train the network that way? My concern with this approach is that then the CNN won’t see the full context around a given box thus possibly hurting performance.</p>

username: JonathanSum
cooked: <p>Change the batch size to be smaller</p>

username: ptrblck
cooked: <p>If reducing the batch size is not an option, you could also have a look at <a href="https://pytorch.org/docs/stable/checkpoint.html"><code>torch.utils.checkpoint</code></a> to trade compute for memory.</p>
Dataloader SuperSuper Slow

username: shivangi
cooked: <p>I prepocessed a video and selected 5 consecutive RGB frames to form tensors of shape 15 X 256 X 256 as <code>file.pt</code>. Here is my dataloader. But the dataloader is super slow. Any insights what could be the possible issue and how to speed it up?<br>
I am using batch size 128 and 4 workers. On using 32 workers,I even get some exception.</p>
<pre><code class="lang-auto">train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=4, shuffle=True)

class LipSeqDataset(Dataset):
    def __init__(self, num_classes, base_path, transform, image_count, fake_classes):
        super(Dataset, self).__init__()
        self.num_classes = num_classes
        self.fake_classes = fake_classes
        self.root_dir = base_path
        self.image_count = image_count
        self.keys = get_dir_path(self.root_dir, self.num_classes, self.fake_classes, self.image_count)

    def __len__(self):
        return len(self.keys)

    def __getitem__(self, idx):

        input_images = self.keys
        # Assign label to class
        try:
            input_images = [(t, 0) if "orig" in t else (t, 1) if "df" in t else (t, 2) if "fs" in t else (t, 3) if "f2f" in t else (t, 4) for t in input_images]
            input_img_as_tensor = torch.load(input_images[idx][0])
        except:
            print(input_images[idx][0])
            exit()

        return input_img_as_tensor, input_images[idx][1]
</code></pre>

username: ptrblck
cooked: <p>Did you measure the data loading time during the training or just the first iteration?<br>
Note that the first step will spin up all workers, and each will load a complete batch, which might introduce some warmup time.<br>
Also, where is your data stored? Is it on a local SSD or some other hard drive?<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> which gives a good summary for potential bottlenecks.</p>
How can I use this kind of shape of Tensor

username: 111226
cooked: <p>Hi, I am using PyTorch with custom loss function. But when I checked the loss function’s output, the shape of it was <code>torch.Size([])</code>. To make the same shape of tensor, how can I do it?</p>

username: ptrblck
cooked: <p>It’s a scalar value, which can be created via:</p>
<pre><code class="lang-python">x = torch.tensor(1.)
print(x.size())
&gt; torch.Size([])
</code></pre>
PyTorch: What is the best way to training model with using two datasets?

username: Aiman_Mutasem-bellh
cooked: <p>Hello, <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have 2 difference datasets consisting of,</p>
<p>Dataset 1: 19 million words<br>
Dataset 2: 2 million words</p>
<p>The task is to train the model sequentially with the tow datasets, each dataset has different training and validation set, and I have only one test set.</p>
<p>When I train the model for 50 epochs with dataset one, I get state-of-the-art results. Then, I loaded dataset 2 during training with the same hyperparameters of dataset2. The problem is the performance is decreased by 0.80 %.</p>
<p>My question is, what the best way to train my model sequentially with two datasets?</p>
<p>Thanks</p>
Can't enumerate through a ModuleList

username: G.M
cooked: <p>I’m using Pytorch 1.2.0 with Python 3.7, and I can’t enumerate through a ModuleList.<br>
I’m using this code</p>
<pre><code class="lang-auto">for i,direction in enumerate(self.directions):
            state = states[i]
            out, out_state = direction(inp, state)
            outputs += [out]
            output_states += [out_state]
</code></pre>
<p>, which is based from <a href="https://github.com/pytorch/pytorch/blob/5b321a098594693eca8408169f475cb334856655/benchmarks/fastrnns/custom_lstms.py#L219" rel="nofollow noopener">here</a>. and I’m getting</p>
<pre><code class="lang-auto">RuntimeError: 
'module' object is not iterable:
</code></pre>
<p>from</p>
<pre><code class="lang-auto">        for i,direction in enumerate(self.directions):
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  &lt;--- HERE
</code></pre>
<p>I’ve clicked the <a href="https://github.com/pytorch/pytorch/issues/14471" rel="nofollow noopener">link</a> about enumerating in the original file(the link above), it said that enumerating is already supporterd. Since I can’t use it, can someone please tell me what should I do.</p>
<p>Thanks.</p>

username: eellison
cooked: <p>Hi, <a class="mention" href="/u/g.m">@G.M</a>, this is fixed in our most recent release (1.4). Could you try updating your pytorch version and giving it another go?</p>

username: G.M
cooked: <p>Thank u very much. That’s the problem. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: G.M
cooked: <p>But enumerate doesn’t seem to be in the 1.4.0 doc. And do u know where can I find a release note or something similar to that for Pytorch?</p>

username: eellison
cooked: <p>Good point, I think that was an omission. I’ll add a section to iterators to the docs shortly and link it here.</p>

username: eellison
cooked: <p>Hi, turns out we already had a (small) section here: <a href="https://pytorch.org/docs/master/jit_language_reference.html#iterables" rel="nofollow noopener">https://pytorch.org/docs/master/jit_language_reference.html#iterables</a></p>

username: G.M
cooked: <p>Alright thx <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">, I thought it was in the 1.4.0 doc.</p>
Sequence labeling for sentences and not tokens

username: antgr
cooked: <p>I have sentences that belong to a paragraph. Each sentence has a label.<br>
[s1,s2,s3,…], [l1,l2,l3,…]<br>
I understand that I have to encode each sentence using an encoder, and then use sequence labeling. Could you guide me on how I could do that, combining them?</p>
How to train the Batch Normalization layers in pretrained network

username: halahup
cooked: <p>Hello, I have a pretrained network from the model zoo. That network has batch normalization layers that along with all the other layers were pretrained on the ImageNet dataset. I am trying to use the network as a fixed feature extractor for transfer learning and have replaced the classifier with my own classifier. However, I don’t fully understand how to unfreeze only the Batch Norm layers so they could train along with the classifier while the rest of the layers remain fixed. As far as I understand, enabling grad on the Batch Norm layers will enable the gradients on all the remaining layers in the network as all the nodes are dependent on the Batch Norm. Surely, there has to be a way to make the Batch Norm layers to train without unfreezing the rest of the network? Any help is appreciated, thank you.</p>

username: Keven_Wang
cooked: <p>Unfreezing only the batch norm layers should not unfreeze other non-bn layers.</p>
How to write class-dependant transformations?

username: alx
cooked: <p>I have an unbalanced dataset and I’d like to have more transformations for two of the classes. I haven’t seen any tutorial about it.</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/how-to-augment-the-minority-class-only-in-an-unbalanced-dataset/13797/2">This code snippet</a> might be what you are looking for.</p>

username: alx
cooked: <p>I see! So if I understand correctly, there’s no way to write class-dependant transformations without a custom dataset class.</p>

username: ptrblck
cooked: <p>I think a custom <code>Dataset</code> might be the easiest way.<br>
Alternatively, you could write a custom transformation using e.g. <a href="https://github.com/pytorch/vision/blob/2ca3279c325cbaab56a87d29135db01d6a6aac46/torchvision/transforms/transforms.py#L351-L377">RandomApply</a> as the base code and add the condition on the target.<br>
However, this would also need the target as an input, which would most likely not work as a drop-in replacement for the vanilla transformations in torchvision models.</p>
How to override pretrained model class in Pytorch?

username: kahn89
cooked: <p>Hello there I am new to PyTorch, I am trying to use pretrained ResNet (2+1)D [1] but since it’s first layer uses 3 channels and I am using only one channel, I suppose I would have to override that class. Please have a look at my attempt, i am getting an error:</p>
<pre><code class="lang-auto">TypeError: _video_resnet() got multiple values for keyword argument 'stem'
</code></pre>
<pre><code class="lang-auto">class R2Plus1dStem4IMAGES(nn.Sequential):
    """R(2+1)D stem is different than the default one as it uses separated 3D convolution
    """
    def __init__(self):
        super(R2Plus1dStem4IMAGES, self).__init__(
            nn.Conv3d(1, 45, kernel_size=(1, 7, 7),
                      stride=(1, 2, 2), padding=(0, 3, 3),
                      bias=False),
            nn.BatchNorm3d(45),
            nn.ReLU(inplace=True),
            nn.Conv3d(45, 64, kernel_size=(3, 1, 1),
                      stride=(1, 1, 1), padding=(1, 0, 0),
                      bias=False),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True))



model = torchvision.models.video.r2plus1d_18(pretrained=True, stem=R2Plus1dStem4IMAGES)

model.fc = nn.Linear(model.fc.in_features, 3)
</code></pre>
<p>[1] <a href="https://pytorch.org/docs/stable/_modules/torchvision/models/video/resnet.html#r2plus1d_18" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torchvision/models/video/resnet.html#r2plus1d_18</a></p>

username: ezyang
cooked: <p>Glancing at the code in torchvision, it looks like r2plus1d_18 doesn’t support overriding the stem. So you should probably just try calling the underlying _video_resnet directly with the same arguments as specified in r2plus1d_18</p>

username: jusjusjus
cooked: <p>Hi Kahn,  I guess you are looking to use the pre-trained weights within the ResNet architecture.   The simplest thing would be to expand your tensor with the input data by replicating the one channel:</p>
<pre><code class="lang-python"># make up some input tensor
input_tensor = torch.randn(batch_size, 1, img_size, img_size)
input_tensor = input_tensor.expand(batch_size, 3, img_size, img_size)
# go on to use in resnet ...
</code></pre>
Custom dataset that gets items who have a higher integer value in count column more often?

username: Dylan_Yung
cooked: <p>Let’s say I have a pandas dataframe that’s ‘name’ and ‘count’ column. Is there a way to make a custom dataset so that when I turn it into a dataloader the values with higher ‘count’ get sampled more often in proportion to their count compared to all the other rows?</p>

username: Tahir
cooked: <p>You can do this by repeating the rows with “higher values” more often than the lower values so that in the training data they will show up more often. e.g. you can increase then rows 50% more if you want to have 50% additional weightage in weight updates.</p>

username: Dylan_Yung
cooked: <p>I thought about doing that but this dataset is 5 million rows and some have counts of 19000+. Something more memory and computationally reasonable is preferred.</p>
Seems to be a bug with padding in a replicate mode (3D tensors expect 2 values for padding)

username: divinho
cooked: <pre><code class="lang-auto">&gt;&gt;&gt; import torch as to
&gt;&gt;&gt; import torch.nn.functional as fu
&gt;&gt;&gt; a=to.randn(3,2,5)
&gt;&gt;&gt; _=fu.pad(a[:, 1:], (0,0,1,0,), mode='constant')  # works!
&gt;&gt;&gt; _=fu.pad(a[:, 1:], (0,0,1,0,), mode='replicate')
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.6/site-packages/torch/nn/functional.py", line 2852, in pad
    assert len(pad) == 2, '3D tensors expect 2 values for padding'
AssertionError: 3D tensors expect 2 values for padding
</code></pre>
<p>Am I doing something wrong?</p>

username: albanD
cooked: <p>Hi,</p>
<p>In replicate mode. <code>pad</code> expects batched image-like Tensors as input I think. If you want to do 2D padding, you should give it a 4D input.</p>

username: divinho
cooked: <p>I see, okay thank you.</p>
Training a model considering the predict probability range rather than the actual label

username: Swamy
cooked: <p>Hi,<br>
This might be a novice question, I have a multi class model for which I have a linear layer with sigmoid activation as the final output. I am using BCEwithLogit loss function to calculate loss, for the evaluation of the model or to get the predictions I am using the probability threshold &gt; 0.5 to mark the output as positve and &lt;0.5 as negative. The question I have is this threshold of &gt;0.5 embedded with the loss calculation for training or do I have to add a threshold condition in the model if I want to train the model based on the threshold?</p>

username: sdv4
cooked: <p>In short: no you do not have to add anything about the threshold you have chosen to the loss function.</p>
<p>Your model isn’t calculating the loss based the label that is ultimately chosen. It is calculating loss based on the strength of the prediction with respect to the correct label.</p>
<p>For example, if the model assigns a probability of 0.55 to class 1 (suppose this is the true label), there is still a loss value associated with this, even though it is the correct label. That is because the model is trying to maximize the probability of choosing the correct label. So predicting 0.55 for the correct class results in a higher loss than predicting 0.95 for the same class, even though at inference time the model would be choosing the same label.</p>

username: Swamy
cooked: <p>Thanks for the reply.</p>
<p>That helps if I am expecting a single label from the model, as I have to check for the maximum of all the node outputs to label it as the positive, in my case I can accept more than one label as the output for the same input.</p>
<p>Also, wouldn’t it help to train the model much faster by reducing the loss if I could accept a wide range as correct rather than a single point for comparison ?</p>

username: sdv4
cooked: <p>The reason you can’t choose a label as correct or incorrect during training time (hinge loss) is that operation is not differentiable. So for instance, if you wanted to say that either class 1 and 3 are correct, there is no way to specify this with a differentiable operation. Argmax isn’t differentiable and neither is something like ‘class_1&gt; class_1_decision_threshold’. That is why we train by maximizing the likelihood of choosing the correct class; doing so is entirely differentiable.</p>
Calculating KL divergence between two Gaussians with .distributions package

username: williamFalcon
cooked: <p>I’m looking to estimate the KL divergence using monte carlo sampling. When I do the non MC version I get excellent results. When I replace with the MC version, I get bad results.</p>
<p>Anyone know what I’m doing wrong?</p>
<pre><code class="lang-auto">        z_mu, z_var = self.enc(x)
        
        # ---------
        # sample Z
        # ---------
        # init likelihood and prior
        std = torch.exp(z_var / 2)

        # Normal likelihood
        Q = torch.distributions.normal.Normal(z_mu, std)

        # Normal(0, 1) prior
        P = torch.distributions.normal.Normal(loc=torch.zeros_like(z_mu), scale=torch.ones_like(std))

        # sample Z
        z = Q.rsample()

        # KL div
        qz = Q.log_prob(z)
        pz = P.log_prob(z)

        kl_loss = torch.mean(qz - pz)
</code></pre>

username: tux
cooked: <p>Is not it a dimension problem ? When you apply the log_prob function, would not you do it to each sample ?</p>

username: williamFalcon
cooked: <p>what do you mean? isn’t every z independent from every other z for each batch item?</p>
Remove module child

username: Joaquim_Ferrer
cooked: <p>Is it possible to remove a child module safely? I’m totally sure that I removed it from any computations I do. The goal is, since it is no longer required, to make python free it from memory as there will be no more references to it.<br>
The modules I want to remove were added with add_module so ideally it would be some method module.remove_module(child_module_name)</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>del module.child_module_name</code> will do the right thing for you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Select samples with same labels within a batch

username: nima_rafiee
cooked: <p>Hi</p>
<p>For each sample within a batch of Mnist data,  I need to select one or more samples with the same label.<br>
is there any way to do it without going through the batch with a For loop? I mean a vectorized version<br>
which can use GPU acceleration.</p>

username: JuanFMontesinos
cooked: <p>You need to pass the class in the dataloader, then you can just use indexing. There is no magic way of knowing the class otherwise</p>
Jacobian for every layer

username: kk1153
cooked: <p>is there an efficient way of getting jacobian of every layer in pytorch.<br>
for eg. if i have x=nn.conv2d(y), i would like to see the local gradient dx/dy and similarly so for all the layers in the network.</p>

username: albanD
cooked: <p>You mean <code>dx/dy</code> given you formula right?</p>
<p>It depends what you want it for.<br>
If it is only for debugging, you can compute the Jacobian for your layer like you would do for any other function using things like <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7">this</a>.<br>
If you want it efficiently, you can write down the formula easily in general and just implement it.</p>
<p>Note that the autograd never computes full Jacobian and so the formula we have can only do vector Jacobian product.</p>
Finding Kronecker product of two matrices

username: kaiseryet
cooked: <p>Hey guys,</p>
<p>So I am just wondering whether there is an existing function to find the Kronecker product of two matrices? Here is a Chinese article on that:</p>
<p><a href="https://blog.csdn.net/P_LarT/article/details/101066757" class="onebox" target="_blank" rel="nofollow noopener">https://blog.csdn.net/P_LarT/article/details/101066757</a></p>
<p>in which they used <code>einsum</code> to achieve this. Though direct function would be much faster as tbh I don’t think <code>einsum</code> is efficient. So is there one? Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The article you link has a link to this forum. In particular this discussion <a href="https://discuss.pytorch.org/t/kronecker-product/3919" class="inline-onebox">Kronecker Product</a> that seem to contain the answer you seek.</p>

username: kaiseryet
cooked: <p>I was thinking function based on einsum won’t be efficient?</p>

username: albanD
cooked: <p>Why not?</p>
<p>Also the post you sent has a code sample with ger and reshape which does not use einsum.</p>

username: kaiseryet
cooked: <p>idk just intuitively I don’t think it’s using vectorization… nvm if it doesn’t matter</p>

username: albanD
cooked: <p>It is doing its best to. But it is not perfect indeed.</p>
<p>But <code>ger</code> + <code>reshape</code> will give you what you want <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How can I choose part of output flowing into another network

username: isaac1644
cooked: <p>As written in topic, How can I choose part of output of convolution layer into another network? e.g. I defined a encoder with convolution layers and it generates a 100D output. What should I do for separating it into two part and only extract its first half 50D data as input of other structure?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use normal indexing to get a subset of the output and then use it. I usually use <a href="https://pytorch.org/docs/stable/torch.html?highlight=narrow#torch.narrow"><code>.narrow()</code></a> for that.</p>
Using Embedding layer for fasttext

username: jmperez
cooked: <p>Is there any advange of loading my <code>fasttext</code> embeddings into a <code>nn.Embedding</code> layer (which I keep freezed, btw) instead of just using the fasttext model to get the word vectors?</p>
<p>I mean, the big advantage of <code>fasttext</code> is that its ability to create an embedding of an OOV-word based on its character n-grams. If I use an <code>Embedding</code> layer (and not fine tune it) I am losing that point.</p>

username: shagunsodhani
cooked: <p>If you are not <code>finetuning</code> the embeddings, then it is fine not to use an embedding layer.</p>

username: jmperez
cooked: <p>The only point there is that I should not numericalize my instances, right?</p>

username: zhangguanheng66
cooked: <p>The pretrained <code>fasttext</code> embeddings are used to numericalize tokens.</p>

username: jmperez
cooked: <p>I’m referring to converting sentences into <code>LongTensor</code>s. If I do not use an embedding layer, I should not create a vocabulary and so on.</p>

username: zhangguanheng66
cooked: <p>In torchtext, you could load the <code>fasttext</code> vectors into a <code>vocab</code> instance, which is used to numericalize tokens.</p>
torchtext.data.BucketIterator.splits for subset not working

username: awsgcp
cooked: <p>Regarding text data, torchtext.data.BucketIterator.splits is working well with dataset, however, in order to split the dataset to train and validate, torch.utils.data.random_split will be used, but it returns subset not dataset.</p>
<p>so this will trigger a problem, if we use torch.utils.data.random_split to split a dataset to train and validate, how can we still use torchtext.data.BucketIterator.splits to generate train_loader and validate_loader from the subsets?</p>
<p>Thank you very much.</p>

username: awsgcp
cooked: <pre><code class="lang-auto">AttributeError: 'Subset' object has no attribute 'sort_key'
</code></pre>
<p>the above error will show if apply torchtext.data.BucketIterator.splits to subset. Since subset is not dataset, how can we easily generate loader from subset? thank you very much.</p>

username: awsgcp
cooked: <p>found a solution: dataset itself has a method called split, it can split the dataset by ratio, which can solve the problem of split dataset to train and validation.</p>
<p>The split method of dataset class will return dataset, NOT subset, which is different from torch.utils.data.random_split.</p>
<p>However, this will create a puzzle: when to use torch.utils.data.random_split? what is subset for?</p>
<p>Hope the solution helps people with similar problems. thanks.</p>

username: zhangguanheng66
cooked: <p>Thanks <a class="mention" href="/u/awsgcp">@awsgcp</a>. Indeed, I think torchtext has some duplicate code, which should be retired. See an issue post <a href="https://github.com/pytorch/text/issues/664" rel="nofollow noopener">here</a> where we discuss a new abstraction, which is more compatible with <code>torch.utils.data</code>. In that case, we don’t need to maintain those duplicate functions anymore.</p>
How to add the number of non zero elements to the loss function

username: antonpuz
cooked: <p>Hi everyone,</p>
<p>I try to implement an idea I found in a paper where an encoder is trained, and the loss function is defined as: MSE(y*, y) + <span class="hashtag">#of</span> non 0 values in one of the layers.<br>
i.e. the loss function is composed of both the prediction MSE and a count of the non 0.0 values of the encoding layer.</p>
<p>My problem is that I can’t implement the non zero value counting. I have tried:</p>
<ol>
<li>adding (encoded_x != 0.0).sum() to the loss function. Where encoded_x is the values of the middle layer. Apparently this is not a tensor so it has not effect (?).</li>
<li>approximation - adding encoded_x.sum() or even torch.pow(encoded_x, 1/8).sum(). Here I tried to penalize by the value itself. The result is that the encoding layer has really small values.</li>
</ol>
<p>Any ideas how this might be implemented? I understand the derivative for the sum is not always defined, but there must be a way.</p>
<p>Thanks,<br>
Anton.</p>

username: mlterpstra-umc
cooked: <p>A penalty on the number of non-zero values tries to enforce sparsity. This is usually achieved by computing the L1 loss. So to add this loss you can compute <code>torch.nn.L1Loss(encoded_x, torch.zeros_like(encoded_x))</code> and add it to your total loss function.</p>

username: antonpuz
cooked: <p>Hi, and thank you very much for your response. I forgot to mention this but I have tried something similar (Although I used torch.zeros rather then zeros_like). My impression is that L1 is not exactly what I need as it penalizes by the value (and not a binary penalty), so the result is simply small values. Did I get something wrong?</p>
Creating Variable from existing numba CUDA array

username: stefdoerr
cooked: <p>Hi, I’m doing some calculations using numba cuda and was wondering if I could just wrap the cuda array with a <code>FloatTensor</code> and <code>Variable</code> instead of copying it to the <code>host</code> using numba and back to <code>device</code> with pytorch.</p>
<p>Let’s say I do:</p>
<pre><code class="lang-python">from numba import cuda
import numpy as np
x = cuda.device_array((3, 9, 11), dtype=np.float32)
</code></pre>
<p>How can I wrap this array with pytorch?<br>
I am able to get the ctypes pointer of the array with <code>x.device_ctypes_pointer</code> but I don’t know how to proceed from there to pass it to a <code>torch.cuda.FloatTensor</code></p>
<p>As far as I can tell, this code here does the opposite: <a href="https://gist.github.com/t-vi/2f4fe23a5b473b9dceb95b163378b4d5#file-pytorch-numba-py-L49-L54" rel="nofollow noopener">https://gist.github.com/t-vi/2f4fe23a5b473b9dceb95b163378b4d5#file-pytorch-numba-py-L49-L54</a></p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not sure if that would be possible.<br>
What they do in the link you showed is that they create a torch tensor, convert it, and then fill the numba array in it’s function (which fills also the torch tensor).</p>
<p>You could do a similar thing by having a torch buffer that you convert to numba and then copy your existing numba array into it. This will cost you at most one extra gpu-gpu copy of your tensor. If you can make your function write the result directly into the buffer, then this will work for free.</p>

username: stefdoerr
cooked: <p>Thanks! That’s a good idea <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
In the end I realized that since I am manually initializing the numba cuda array I can just use the code in the above link and first create the pytorch array and convert it to the numba cuda array. It works great!</p>

username: Robot_mantou
cooked: <p>Hi~ I’m also in the situation that need to convert the numba CUDA array to pytorch tensor. But I can not realize it. Could you share the code or the link?<br>
Thanks.</p>
Maskrcnn repository by vgg16

username: solsol
cooked: <p>Hi,</p>
<p>Is anyone provide this repository: <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="nofollow noopener">https://github.com/facebookresearch/maskrcnn-benchmark</a> using VGG16?</p>
<p>I would appreciate any help.</p>
Learning on the fly

username: Hamada_Fathy
cooked: <p>Let’s assume that i have trained a cnn model with 5 classes. Now, i want to train the same trained model on 3 new classes. how could i use the trained model to train only the 3 new classes and not on the 8 classes??</p>

username: JonathanSum
cooked: <p>Using the tutiorl from Pytorch:<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py</a><br>
I think this is a simple question, All you need to do is change the final Linear layer(or fully connected layer) from outputting a matrix that has shape [batch_size, 5] to be   [batch_size, 3]   (3 new classes)</p>
<p>I suggest that you don’t need to train it again. You can just use transfer learning to only train the last layer which predicts 3 classes only. (you can see the PyTorch tutorial for more detail).</p>
<pre><code class="lang-auto">before( cnn model with 5 classes):
import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 5)     #    5

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net()
</code></pre>
<pre><code class="lang-auto">after( cnn model with 3 classes):
import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 3)                 #you said only 3 new classes, you we only predict 3 new class

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net()
</code></pre>

username: Hamada_Fathy
cooked: <p>I didn’t mean that. what i mean is i want to train the model on the 3 new classes but the inference predict the whole 8 classes</p>

username: kl_divergence
cooked: <p>You can use two different linear layers. You can put a condition like this:</p>
<pre><code class="lang-auto">if self.train():
    self.fc3 = nn.Linear(x,5)
else:
    --
</code></pre>
<p>That way you can load the weights of this <code>fc</code> layer whenever you want.</p>
Observation and action space as spaces.Dict

username: Gandalf
cooked: <p>Can anyone give me good code example, where the observation space or the action space of an environment for a neural network has been a spaces.Dict?</p>
<p>I’m trying to create my own environment. My observation space consists of three different inputs and my action space contains two different actions with their own action space.<br>
For example, my current action space looks like this:</p>
<p>‘’’<br>
self.action_mutation = spaces.Discrete(4)<br>
self.action_node = spaces.Discrete(100)</p>
<p>self.action_space = spaces.Dict({‘mutation’: self.action_mutation, ‘node’: self.action_node})<br>
‘’’</p>
<p>But I’m not sure how to work with such spaces when I want to build my neural network.</p>
Why not use detach when freezing?

username: 11152
cooked: <pre><code class="lang-auto">def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False
</code></pre>
<p>Why use only require_grad = False?</p>

username: JuanFMontesinos
cooked: <ol>
<li>Detach requires hard coding inside the model.</li>
<li>Detach requires knowledge about how layers are connected.<br>
On the other side requires_grad false is external code and does not require any knowledge about model graph.<br>
You can just not to pass  layers to the optimizer, but there are drawbacks too (gradients keep acumulating)</li>
</ol>

username: 11152
cooked: <p>Does detach automatically change to require_grad = False?</p>

username: JuanFMontesinos
cooked: <p>Detach does not change requires grad. Detach is a tensor’s method which breaks model’s graph when used.</p>

username: 11152
cooked: <p>Thank you very much !!</p>
Examples of predicting testing data in for loop

username: deeplearner20
cooked: <p>Hi!</p>
<p>I am new to pytorch. I was able to successfully write the Dataset and Dataloader to preprocess, index, batch and shuffle my training dataset. I want to write code to create a for loop that runs through all epochs and trains on my training model, and tests on my testing data to predict labels (classification problem), and then output the labels in a csv file. Can anyone help me by linking tutorials, examples, starter code etc., that will be helpful to write the for loop for n epochs?</p>

username: ptrblck
cooked: <p>The <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py">MNIST example</a> shown you a simple foop over your <code>DataLoaders</code>, while the <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py">ImageNet example</a> gives you some other utilities (e.g. only storing the best checkpoint).</p>
<p>You can store the outputs as numpy arrays using <code>output.detach().cpu().numpy()</code> and store them as a <code>csv</code> using e.g. <code>np.savetxt("preds.csv", output, delimiter=",")</code> or <code>pandas</code>.</p>

username: deeplearner20
cooked: <p>Thank you for your reply!</p>
RuntimeError: _th_normal_ not supported on CPUType for Long

username: lionely
cooked: <p>Hi, I am getting the following error. It’s hard for me to post my whole code because I’m working with a reinforcement learning library. I’m trying to use their A2C algorithm, they sample actions from a Gaussian distribution when given a state. Their code expects <code>torch.float32</code>.  I have checked and my inputs are indeed <code>torch.float32</code> not Long. I’m not sure what to do. I know the algorithm runs as I have run it before, but I get this error when I try to use it in an environment with discrete actions.</p>
<p>Here is the library’s draw_action function:</p>
<pre><code class="lang-auto">def draw_action_t(self, state):
        print('draw_action_t',state.dtype)
        return self.distribution_t(state).sample().detach()
</code></pre>
<p>and</p>
<pre><code class="lang-auto">def distribution_t(self, state):
        mu, sigma = self.get_mean_and_covariance(state)
        return torch.distributions.MultivariateNormal(loc=mu, covariance_matrix=sigma)
</code></pre>
<p>Final error:</p>
<pre><code class="lang-auto">RuntimeError: _th_normal_ not supported on CPUType for Long
</code></pre>
<p>Help is much appreciated!</p>

username: lionely
cooked: <p>Bump, please would appreciate some help!! I’m not sure what is causing this problem</p>
Opening same file in dataloader with different num_workers in parallel

username: amogh112
cooked: <p>In my dataloader, I want to return images after sampling using random.sample(list,sample_size) from a folder. However, if the number of workers is more than 1, the workers might be accessing the same file at the same time. How do I deal with this?<br>
eg -</p>
<pre><code class="lang-auto">
class Loader(torch.utils.data.Dataset):
    def __init__(self, root_dir, length_sample):
        self.root_dir = root_dir
        self.length_sample =  length_sample
        
    def __getitem__(self,index):
        images = random.sample(glob.glob(root_dir+"/*"), self.length_sample)
        result_images = []
        for p in images:
             im = Image.open(p)
             result_images.append(transforms.ToTensor()(im))
        return torch.cat(result_images)

</code></pre>

username: ptrblck
cooked: <p>Are you concerned about the potential performance issue or about deadlocking the process?<br>
I would assume <code>PIL</code> takes care of the latter problem.<br>
Do you have any particular idea how to handle the first case?<br>
You could try to use the current worker id in your <code>__getitem__</code> method to e.g. reset the random seed with the current index.</p>
<pre><code class="lang-python">
class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(10, 1)
        
    def __getitem__(self, index):
        print(torch.utils.data.get_worker_info().id)
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)


dataset = MyDataset()
loader = DataLoader(
    dataset,
    num_workers=2
)

next(iter(loader))
</code></pre>
Choosing the appropriate Gaussian process model

username: prvnkmr
cooked: <p>I can find GP models in both Gpytorch package and in Pyro package. I came to know that the model used for inference is different from models for prediction. Can any one throw some light on:<br>
–&gt; What is basic difference between inference and prediction.<br>
–&gt; which library has models for prediction and which one for inference.</p>

username: tom
cooked: <p>The terminology between typical GPs lingo and deep learning is a bit different when it comes to inference. For GPs: Inference = find model/hyperparameters (or distributions) “infer them from data” and prediciton: apply model. In DL: Find model = training, inference = apply model.</p>
<p>Any model would have inference and prediction steps, and any framework, too.<br>
The model can be either optimization based (e.g. MAP, VI) or MCMC. It really depends on the details of your problem which is most appropriate.</p>
<p>GPyTorch might, in my view, be more accessible / similar to “classic” GP libraries.<br>
Pyro on the other hand is a generic probabilistic programming framework that happens to be applicable to GPs, too.</p>
<p>But of course I’m biased, long before there was GPyTorch, I had written <a href="https://github.com/t-vi/candlegp" rel="nofollow noopener">my own GP library</a> for PyTorch.</p>
<p>Best regards</p>
<p>Thomas</p>

username: prvnkmr
cooked: <p>Hello Tom,</p>
<p>Thanks for the reply. I asked the above question after seeing some content from the attached notebook. The author says that the pyro library GP models are used for prediction and Gpytorch GP models are used for inference. Please see this link. <a href="https://github.com/cornellius-gp/gpytorch/blob/master/examples/07_Pyro_Integration/Pyro_GPyTorch_Low_Level.ipynb" rel="nofollow noopener">https://github.com/cornellius-gp/gpytorch/blob/master/examples/07_Pyro_Integration/Pyro_GPyTorch_Low_Level.ipynb</a></p>