akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Do I need a custom modified MultiheadAttention layer?

username: siahuat0727
cooked: <pre><code class="lang-python">class Foo(nn.Module):
    def __init__(self, dim, n_iter):
        ...
        self.n_iter = n_iter
        self.attn = nn.MultiheadAttention(dim, 1)

    def forward(self, input):
        input = input.unsqueeze(0)

        for i in range(self.n_iter):
            assert input.size(0) == i+1
            output, _ = self.attn(input, input, input, need_weights=False)

            # For every iteration, I need only the last feature
            # since it uses only the previous input (like rnns)
            last = output[-1]

            # The input is concatenated with previous input,
            # that means some results of the previous `attn` can be reused.
            input = torch.cat([input, some_func(last)], dim=0)
        ...
</code></pre>
<p><strong>Updated</strong> to make the question clearer.</p>
<p>I want to solve a problem iteratively, similar to RNNs but using self-attention.<br>
I have a series of inputs, I want the first output depends on the first input, and the second output depends on both the first and second input and so on.<br>
But the things different is, the second input depends on the first output, so that’s why I do it iteratively.</p>
<p>I think <code>MultiheadAttention</code> is good for solving this problem but it brings a lot of unnecessary calculation.</p>
<ol>
<li>For each iteration, I need only the last output.</li>
<li>The later iteration can reuse some hidden state of the previous iteration (like qkv of the previous input).</li>
</ol>
<p>Do I need a custom modified MultiheadAttention layer to only calculate for the last output and save the hidden states so that can be reused later?<br>
If so, how to do it since the <code>forward</code> calls <code>F.multi_head_attention_forward</code> which is not written in python?<br>
Or is there any other way better suited to my needs?</p>
<p>I think anyone who tries to solve a time series using self-attention will meet a similar situation? (except for that the next input depends on the previous output)<br>
But I’m new in this domain and don’t know where to find the solution.<br>
Can anyone help me?</p>
<p>Any help would be appreciated!</p>

username: SandPhoenix
cooked: <p>If I understood well, I think that you can do that by simply write something like this in the forward function</p>
<pre><code class="lang-auto">def forward(self, x):
        previous = x
        x = self.layer_or_model(x)
        x += previous
        return x
</code></pre>
<p>and use it as a building block. I believe this kind of solution works because I used it before but I am not sure if this is what you asked for.</p>

username: siahuat0727
cooked: <p>Sorry, I didn’t make it clear.</p>
<p>The way I want to solve the problem is similar to RNNs.<br>
I have a series of inputs, I want the first output depends on the first input, the second output depends on the first and second input, and so on.<br>
But the things different is, the second input depends on the first output, so that’s why I do it iteratively.</p>
<p>Thanks for your help!</p>
Find max row index with some condition

username: saluei
cooked: <p>I have input tensor (N*2) , first column is category and second is the value, something like this:</p>
<p>[ [14,50] , [18,1.5] , [14,250] , [18,2.5] , [10,1] , [14,252] , [18,5.3]  ]</p>
<p>each category may be have N row in input tensor,<br>
how to found index of row in input tensor that have max value in each category, something like this for above input tensor</p>
<p>[ [14,5] , [18,6] , [10,4] ]</p>
<p>[  category code, row index have max value  ]</p>
<p>but if it’s possible without using loop,<br>
thank’s</p>

username: Kushaj
cooked: <p>There is no groupby function in pytorch. So not possible without a for loop. You can convert to numpy and then use pandas groupby.</p>

username: saluei
cooked: <p>yes in fact I search something like group by in PyTorch and do not find,<br>
Thank’s for reply Kushaj</p>
Confusion about pooling layers

username: n4tman
cooked: <p>Do the nn pool2d layers (nn.AvgPool2d, nn.MaxPool2d, nn.AdaptiveAvgPool2d, nn.AdpativeMaxPool2d) work as specified below, replacing the MAX by average when using average pooling?</p>
<blockquote>
<p>It is common to periodically insert a Pooling layer in-between successive Conv layers in a ConvNet architecture. Its function is to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting. The Pooling Layer operates independently on every depth slice of the input and resizes it spatially, using the MAX operation. The most common form is a pooling layer with filters of size 2x2 applied with a stride of 2 downsamples every depth slice in the input by 2 along both width and height, discarding 75% of the activations. Every MAX operation would in this case be taking a MAX over 4 numbers (little 2x2 region in some depth slice). The depth dimension remains unchanged.</p>
</blockquote>
<p>Thanks!</p>

username: SandPhoenix
cooked: <p>Hello,<br>
I believe that it does.<br>
Peace</p>
About weight sharing between modules

username: Zhikaiiii
cooked: <p>I am working on a multi-agent reinforcement learning which each agent has its own q-network.However, the network cosists of several layers that share parameters among agents. I look up a lot of discussion about weights sharing and get confused now. I wonder if the code below can work? Thanks.</p>
<p>PS: the embedding layer and attention layer are the layers I want to share weights among agents.</p>
<pre><code class="lang-auto">    def __init__(self, layer1, layer2, input_dim, output_dim):
        super().__init__()
        self.layer1 = layer1
        self.layer2 = layer2
        self.linear1 = nn.Linear(input_dim, output_dim)
        self.relu = nn.ReLU()

    def forward(self, x, neighbors):
        x_1 = self.layer1(x)
        neighbors_1 = []
        for neighbor in neighbors:
            neighbors_1.append(self.layer1(neighbor))
        out = self.layer2(x_1, neighbors_1)
        out = self.linear1(out)
        out = self.relu(out)
        return out
</code></pre>

username: Hmrishav_Bandyopadhy
cooked: <aside class="quote no-group" data-username="Zhikaiiii" data-post="1" data-topic="91904">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhikaiiii/48/32296_2.png" class="avatar"> Zhikaiiii:</div>
<blockquote>
<pre><code class="lang-auto">for neighbor in neighbors:
            neighbors_1.append(self.layer1(neighbor))
</code></pre>
</blockquote>
</aside>
<p>Weights are being shared here. The layer defined in <code>__init__()</code> as <code>self.layer1</code> is a single layer through which you are passing various tensors from the tensor list <code>neighbors</code>.</p>

username: Zhikaiiii
cooked: <p>Thanks for your answers. I will have  a try：）</p>
My json file doesn't seem to be a legit COCO dataset

username: dip4fish
cooked: <p>Hello,<br>
I have a dataset consisting of 2164 pairs of greyscaled+groundtruth (2 instances) images. The idea would be to use detectron2 for instance segmentation, for this the masks have to be convert into COCO dataset format. Following the detectron2 specs, <a href="https://drive.google.com/file/d/16COfvTBP9hMBjVYpTDIRPwdT5V2LqLNn/view?usp=sharing" rel="nofollow noopener">a json file</a> for one image and two instances was generated as a toy model using compressed rle encoding provided by pycocotools.<br>
Unfortunately, the dataset failed to be loaded with pycocotools or with detectron2.<br>
The conversion was performed with a notebook available on <a href="https://www.kaggle.com/jeanpat/minimalist-mask-to-coco-format-dataset-conversion/edit" rel="nofollow noopener">kaggle</a>.<br>
Is there something wrong with the way the json dataset was generated?<br>
For the beginning, the task would only be capable to overlay a legit COCO annotation on the greyscaled image.<br>
Thank you for your help.</p>
Replacing a block diagonal with other values in a tensor

username: srhlinzheng
cooked: <p>Hi everyone,</p>
<p>I am wondering how to replace a block diagonal matrix with other values in a tensor in a batched manner. For example, given a tensor with shape [B, 3, 6], where B is the batch size (here B = 1 for illustration):<br>
[[[ 0,  1,  2,  3,  4,  5],<br>
[ 6,  7,  8,  9, 10, 11],<br>
[12, 13, 14, 15, 16, 17]]]</p>
<p>Now I would like to replace each 1 x 2 block along the diagonal with a new value, say [1,1], which transforms the tensor above to<br>
[[[ <strong>1,  1</strong>,  2,  3,  4,  5],<br>
[ 6,  7,  <strong>1,  1</strong>, 10, 11],<br>
[12, 13, 14, 15, <strong>1, 1</strong>]]]</p>
<p>Is it a must to achieve this through a loop or I just ignored some efficient APIs in PyTorch?</p>
<p>Thank you very much!</p>
Pytorch Geometric : loading node graph and line graph together

username: Thilini_Cooray
cooked: <p>Hello,<br>
I found out that we can obtain line graph representation of a node graph using torch_geometric.transforms import LineGraph.</p>
<p>I have the following requirement where I want to access both original dataset’s nodes, adjacency matrix as well as it’s line graph’s nodes and adjacency matrix:</p>
<p>dataset = TUDataset(path, name=MUTAG)<br>
dataloader = DataLoader(dataset_line, batch_size=batch_size)</p>
<p>for data, data_linegraph in dataloader:<br>
loss = model(data.x, data.edge_index, data_linegraph.x, data_linegraph.edge_index)</p>
<p>Where should I use torch_geometric.transforms.LineGraph() to transform the original graph data object in order to make my dataloader output both representations?</p>
Mutiprocessing with multiple gpu

username: Victoryd
cooked: <p>Hi I currently have a model which needs to runs multiple same forward paths in testing time (not training) at a certain node and get the average, similar to the trajectory sampling in the RL monte carlo search tree. My pseudo code would be like on one gpu :</p>
<pre><code class="lang-auto">for idx in range(num) : 
    stats.append(model(node_input))
</code></pre>
<p>However, the above method seems not efficient and then I would like to try using both multiprocessing and make full use of multiple gpu. My design would be produce multiple processes and feed into a certain cpu core and then a gpu whenever they are available. I would appreciate any pytorch resources or repo and advices on multiprocessing + multiple gpu usage.</p>
`torch.chunk` source code

username: Leockl
cooked: <p>I can’t seem to find the source code for <code>torch.chunk</code> in PyTorch’s Github page or in the documentation. Anyone knows where this is in PyTorch’s Github page?</p>
<p>Many thanks in advance.</p>

username: ebarsoum
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.chunk.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.chunk.html</a></p>

username: Leockl
cooked: <p>Thanks buddy. I was asking for the source code in Github, not documentation.</p>
<p>Anyhow, I got it here now: <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/data/datasets/chunk.h" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/data/datasets/chunk.h</a></p>

username: ebarsoum
cooked: <p>The above link is for the Chunk data reader and not torch.chunk. Torch.chunk code is in: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorShape.cpp#L496" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorShape.cpp#L496</a></p>

username: Leockl
cooked: <p>Oh I see. Many thanks for this!</p>
Confusing Time delay between batches

username: Awais_Jafar
cooked: <p>I’m using torch text and there is a strange behavior which I’m unable to understand here. I’m training a CNN on SST-2 dataset. Training time for one epoch is ~5 seconds. However, if I iterate over just batches, time taken is ~100ms, and just predicting takes a total of ~300ms. I’m unable to understand where time delay is</p>
<pre><code class="lang-auto">import time
total = 0
base = time.time()
for b in loader.train_loader:
    now = time.time()
    res = model(b.text)
    tm = time.time() - now
    total += tm
print("Prediction total time", total)
print("Elapsed time", time.time() - base)
</code></pre>
<p>Output is</p>
<blockquote>
<p>Prediction total time 0.36881256103515625<br>
Elapsed time 5.046782493591309</p>
</blockquote>
<p>For just iterating through batches</p>
<pre><code class="lang-auto">import time
total = 0
base = time.time()
for b in loader.train_loader:
    len(b.text)
    tm = time.time() - now
    total += tm
print("Elapsed time", time.time() - base)
</code></pre>
<p>output is</p>
<blockquote>
<p>Elapsed time 0.0856020450592041</p>
</blockquote>
<p>Can anyone explain what I’m missing here? Thanks</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
It seems that the model itself doesn’t take as much time to process the data as much as the dataloader takes to load it [ Note that you are timing the ‘prediction time’ after the dataloader has loaded and as such this is not reflected in the ‘prediction time’ ]. It might be due to the fact that your data is huge and the model is pretty lightweight. Also, your methods of reading data using the dataloader might be inefficient which is causing the delay. Pre-processing the data in the dataloader can also cause delays like these.</p>
Pack_padded_sequence with unsorted question length

username: lisyuan
cooked: <p>Hi, I am the first time using the <code>pack_padded_sequence </code>,  and it seems to input the sequence length in a decreasing order. Here is the default <code>pack_padded_sequence</code> package from <a href="https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_padded_sequence.html" rel="nofollow noopener">official website</a></p>
<p><code>torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False, enforce_sorted=True)</code></p>
<ul>
<li>
<strong>enforce_sorted</strong>: if  <code>True</code> , the input is expected to contain sequences sorted by length in a decreasing order. If  <code>False</code> , the input will get sorted unconditionally. Default:  <code>True</code>
</li>
</ul>
<p>It also mentions that <strong>enforce_sorted = True is only necessary for ONNX export.</strong>, what is this mean? If I want to input the length without sorting, can I simply set it to <code>False</code>?</p>
<p>below is my idea of implement</p>
<pre><code class="lang-auto"># in dataset class; 
#return the question idx after padding and the original question length
class VQADataset(Dataset):
    def __getitem__(self, idx):
        return sample{'question': question, 'length':len(qst_token)}
    
# in model class
class model(nn.Module):
    def forward(self, qst, qst_length):
        word = self.embedding(qst) 
        qst_length = qst_length.tolist()             
        packed = rnn.pack_padded_sequence(word , qst_length, batch_first=True)
</code></pre>
<p>So, the length of question will be different every batch. can I simply set the <code>enforce_sorted </code> into <code>False</code> ?</p>

username: ptrblck
cooked: <p>Yes, if you are not planning to export the model via <code>ONNX</code>, you could use <code>enforce_sorted=False</code> and pass the unsorted padded sequence to this method.</p>

username: lisyuan
cooked: <p>This is the explaination from the internet. ONNX (Open Neural Network Exchange) is an open <a href="https://searchitoperations.techtarget.com/definition/container-containerization-or-container-based-virtualization" rel="nofollow noopener">container</a> format for the exchange of <a href="https://searchenterpriseai.techtarget.com/definition/neural-network" rel="nofollow noopener">neural network</a> models between different <a href="https://whatis.techtarget.com/definition/framework" rel="nofollow noopener">frameworks</a>.</p>
<p>So, if I only use pytorch and not convert into tensorflow. I can simply set it to <code>False</code>, right?</p>

username: ptrblck
cooked: <p>Yes, if you use PyTorch without any intention to use ONNX, you can set this argument to <code>False</code>.</p>
Get high conventional memory usage when torch loads model to GPU

username: melonux
cooked: <p>I have specified map_location to GPU:0 and expect that video memory would go high. But I found conventional memory usage is 1.5GB and video memory usage 550MB.</p>
<pre><code># record memory usage here as baseline.

# the size of model file is about 15MB
self.device = torch.device('cuda:0')
self.model = torch.load('weights/my_model.pt', map_location=self.device)['model'].float().eval()

# check the increment of memory usage here.
</code></pre>
<p>Why does torch.load consume so much conventional memory? Is there something wrong in my code?</p>
Multi-GPU Training for a Unet based segmentation model

username: rahulrav
cooked: <p>I am trying to take advantage of PyTorch’s multi-GPU support on a single machine by using <code>nn.DataParallel</code>.</p>
<p><strong>Note:</strong> I am using a framework called <code>fastai2</code> which builds on top of PyTorch, so my scrips will have a bit of that sprinkled in.</p>
<pre><code class="lang-python">
import numpy as np
from fastai2.vision.all import *
from fastai2.distributed import *


def train():
    path = untar_data(URLs.CAMVID_TINY)

    def label_func(fn): 
        return path/"labels"/f"{fn.stem}_P{fn.suffix}"

    codes = np.loadtxt(path/'codes.txt', dtype=str)
    fnames = get_image_files(path/"images")
    dls = SegmentationDataLoaders.from_label_func(
        path, bs=8, fnames = fnames, label_func = label_func, codes = codes
    )

    learner = unet_learner(dls, resnet34).to_fp16()
    if torch.cuda.device_count() &gt; 1:
        wrapped_model = nn.DataParallel(learner.model)
        learner.model = wrapped_model.module

    callbacks = [
        EarlyStoppingCallback(min_delta=0.001, patience=5)
    ]

    learner.fine_tune(20, freeze_epochs=2, wd=0.01, base_lr=0.0006, cbs=callbacks)
    print('Done')


if __name__ == "__main__":
    train()
</code></pre>
<p><code>unet_learner</code> returns an instance of <code>nn.Module</code> which I am trying to wrap with <code>nn.DataParallel</code>.</p>
<h3>Problem</h3>
<p>This does not seem to have the intended effect. I am still only able to use 1 GPU.</p>
<p>I tried changing the <code>batch_size</code> (<code>bs</code> in <code>SegmentationDataLoaders</code>) as well, and that did not make any  difference other than running out of GPU memory. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Any ideas on what I might be missing ?</p>

username: ptrblck
cooked: <p>I’m a bit confused, what these lines are doing:</p>
<pre><code class="lang-python">wrapped_model = nn.DataParallel(learner.model)
learner.model = wrapped_model.module
</code></pre>
<p>I assume that <code>learner.model</code> is an <code>nn.Module</code>, <code>wrapped_model</code> would be an <code>nn.DataParallel</code> object, and you would reassign the module to itself?<br>
What would happen, if you remove the second line of code?</p>

username: rahulrav
cooked: <p>Sorry about that. My code snippet was incorrect. I was assigning <code>learner.model</code> to the instance of <code>nn.DataParallel</code> object. Let me send you the error message I saw first thing tomorrow.</p>
Sparse Tensor to Dense Tensor Conversion

username: anthonytec2
cooked: <p>I am working with a dataset of around 2TB at the moment and have an input to my network which is Batch Size x 20 x 240 x 240 int8. At the moment, I transfer this dense tensor over to the gpu which takes around 10% of my batch compute time. I was wondering, if it is at all possible using Pytorch to create this as a Sparse tensor on the CPU, transfer it over to the GPU, then recreate it as a dense tensor for running through my network.  For context, I am running this using DDP at the moment on a 4 2080 machine.</p>
<p>One thought I had was to do the above and perform an identity matrix multiply on the sparse tensor to get it into a dense tensor for further operations. Although, I was wondering if there is a better method.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4329ca54843ff56af9994c9cf829928338ce4db7.png" data-download-href="https://discuss.pytorch.org/uploads/default/4329ca54843ff56af9994c9cf829928338ce4db7" title="image (4)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4329ca54843ff56af9994c9cf829928338ce4db7_2_690x191.png" alt="image (4)" data-base62-sha1="9A9wBHpvVOrpKs1kAIge1s7edIr" width="690" height="191" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4329ca54843ff56af9994c9cf829928338ce4db7_2_690x191.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4329ca54843ff56af9994c9cf829928338ce4db7_2_1035x286.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4329ca54843ff56af9994c9cf829928338ce4db7_2_1380x382.png 2x" data-dominant-color="DADCE2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (4)</span><span class="informations">1701×471 79.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you check, if you could reduce the data transfer by creating (or pushing) the sparse tensor on the device and calling <code>to_dense()</code> afterwards?</p>
Cannot load Sentiment model for prediction

username: Mukta_Mohindra
cooked: <p>I have a Sentiment model trained using glove vectors refer <a href="https://blog.floydhub.com/long-short-term-memory-from-zero-to-hero-with-pytorch/" rel="nofollow noopener">https://blog.floydhub.com/long-short-term-memory-from-zero-to-hero-with-pytorch/</a>.<br>
I save the trained mode and dictionary<br>
torch.save(trainedmodel, ‘./trained_lstm.pt’)<br>
torch.save(trainedmodel.state_dict(), ‘./state_dict.pt’)</p>
<p>I try to recreate the model for inference<br>
model = torch.load(‘trained_lstm.pt’, map_location=torch.device(“cpu”))<br>
but keep getting error cannot get attribute SentimentNet in module main.  If I just load the state dictionary i get error on embedding and weight sizes.<br>
I included the model class in the same file while saving and reloading. Any help ?</p>

username: ptrblck
cooked: <p>I would recommend to stick to the second approach (save and load the <code>state_dict</code>), as the former approach can break in various ways.</p>
<p>That being said, could you post the error message for the shape mismatch and check, what might have changed in the model?<br>
E.g. did you store the <code>state_dict</code> and changed the embedding layer afterwards somehow?</p>

username: Mukta_Mohindra
cooked: <p>The issue is model was trained using a glove vector and the embedding dimension was dynamically set using that. That is what I was trying to avoid by reloading the complete model. I was getting error -<br>
RuntimeError: Error(s) in loading state_dict for SentimentNet:<br>
size mismatch for embedding.weight: copying a param with shape torch.Size([1259, 300]) from checkpoint, the shape in current model is torch.Size([1000, 400]).<br>
size mismatch for lstm.weight_ih_l0: copying a param with shape torch.Size([2048, 300]) from checkpoint, the shape in current model is torch.Size([2048, 400]).<br>
I retried setting the initialization parameters -<br>
model_params = {‘weight_matrix’: None,<br>
‘output_size’: 1,<br>
‘hidden_dim’: 512,<br>
‘n_layers’: 2,<br>
‘dropout_prob’: 0.001,<br>
‘vocab_size’ :1259,<br>
‘embedding_dim’: 300<br>
}<br>
model = SentimentNet(**model_params)<br>
model = model.load_state_dict(torch.load(‘state_dict.pt’, map_location=device))</p>
<pre><code>model.eval()
</code></pre>
<p>But now I get this error -<br>
model.eval()<br>
AttributeError: ‘_IncompatibleKeys’ object has no attribute ‘eval’</p>

username: ptrblck
cooked: <p><code>model.load_state_dict()</code> does not return the model, but the information about incompatible keys, so you should remove the assignment and rerun the code:</p>
<pre><code class="lang-python">model_params = {'weight_matrix': None,
'output_size': 1,
'hidden_dim': 512,
'n_layers': 2,
'dropout_prob': 0.001,
'vocab_size' :1259,
'embedding_dim': 300
}
model = SentimentNet(**model_params)
model.load_state_dict(torch.load('state_dict.pt', map_location=device))

model.eval()
</code></pre>
<p>PS: you can add code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Loss is extremely large and not decreasing but accuracy is increasing

username: hongyuanluke
cooked: <p>hi, everyone. I have found a problem that my loss prints something like 3e+18 and is not decreasing at all. however, my accuracy is increasing a lot to 0.95. What is the potential problem behind this?</p>

username: ptrblck
cooked: <p>The loss or accuracy calculation might be wrong, e.g. you would accumulate the loss for various epochs or use unwanted broadcasting in the accuracy calculation.<br>
I’m just speculating, as it’s hard to debug without seeing some code, so feel free to add more information or code. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
A question about gan network

username: Ece_Gve
cooked: <p>excuse me, i find we don’t use net.train() and net.eval() in almost all of the gan network,such as wgan, cyclegan,I am very confused.</p>

username: ptrblck
cooked: <p>It depends generally on the model, if these calls are necessary or not.<br>
<code>model.train()</code> and <code>model.eval()</code> would change the behavior of some layers, such as batchnorm and dropout layers. If these layers are not used (or any other custom layers, which would use the <code>self.training</code> flag), you wouldn’t need to call <code>model.eval()</code> or <code>.train()</code> (although it might be seen as a good coding style).</p>
Training a higher resolution VGG Model

username: 3DTOPO
cooked: <p>I would like to train VGG with a higher resolution, say 448 instead of the standard 224 training image.</p>
<p>First, how is the size for the fully connected layer calculated?</p>
<p>Then I was wondering, is it possible to interpolate the standard 224 weights to 448 so I can start with interpolated weights instead of a blank slate?</p>
<p>I’m just getting started with pytorch so any examples would be greatly appreciated.</p>

username: 1chimaruGin
cooked: <ul>
<li>The input size of fully connected is calculated as the output image size(H x W) and number of channels (n_C) of the last pool layer. (eg. 7 x 7 x 512 )</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9862aa4348056b51f84f8bcceb28f71116c9733.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9862aa4348056b51f84f8bcceb28f71116c9733" title="vgg"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9862aa4348056b51f84f8bcceb28f71116c9733_2_690x404.png" alt="vgg" data-base62-sha1="sKLqazPAGBFTfHwngEzNM1nmzPJ" width="690" height="404" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c9862aa4348056b51f84f8bcceb28f71116c9733_2_690x404.png, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9862aa4348056b51f84f8bcceb28f71116c9733.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9862aa4348056b51f84f8bcceb28f71116c9733.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">vgg</span><span class="informations">1000×586 160 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchvision.models import vgg19

class VGG(nn.Module):

    def __init__(self, num_classes=1000):
        super(VGG, self).__init__()
        self.model = vgg19(pretrained=False) # pretrained=False just for debug reasons
        self.model.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.model(x)
        return x

print(VGG())
model = VGG()
x = torch.randn((1, 3, 448, 448))

print(model(x).shape)
</code></pre>
How to measure test coverage?

username: wnnaone
cooked: <p>Is there a good way to measure how much test coverage test/run_test.py provides?</p>
<p>It seems someone asked something similar and got no response here: <a href="https://discuss.pytorch.org/t/how-to-get-a-full-coverage-report-of-tests/72967" class="inline-onebox">How to get a full coverage report of tests?</a></p>

username: wnnaone
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, you seem to be the go-to person for this – do you know where I can start?</p>

username: ptrblck
cooked: <p>I don’t know a good answer right now, but the idea sounds interesting.<br>
Could you point me to some references, how the coverage is tested in other projects?<br>
I know, that <code>torchvision</code> uses <a href="https://codecov.io/gh/pytorch/vision/pull/2536?src=pr&amp;el=h1">codecov</a> for the PRs, but I’m unsure if you are looking for this approach or another one.</p>

username: rn91
cooked: <p>I tried to use coverage tool to get test coverage for torch source. And got 19% coverage which seems to be very low. Just asked question: <a href="https://discuss.pytorch.org/t/pytorch-low-test-coverage/91777" class="inline-onebox">PyTorch low test coverage</a></p>

username: rn91
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you know who might be able to help with this?</p>

username: ptrblck
cooked: <p>Let’s add <a class="mention" href="/u/ezyang">@ezyang</a> to it, as he might be able to give more insight or knows whom to ping.</p>
Train LSTM with multiple sequences per parameter update

username: mitchellg
cooked: <p>I’m training a network to perform time-series forecasting. My current implementation works but is looping through my sequences 1 at a time, yielding a parameter update per input sequence per epoch. I want to mini-batch train so there is a parameter update after evaluating 32 sequences (to hopefully improve training time and reduce noise) but I can’t figure out how to implement the DataLoader / training loop to do this. I would like to use the DataLoader so I can shuffle the training data.</p>
<p>NOTE: I am not talking about mini-batching the sequences themselves. Each LSTM input is still a tensor of size [100, 1, 8] representing [sequence length, sequence mini-batch size, number of features]. I want to train on 32 of these LSTM inputs per parameter update.</p>
<p>Maybe I’m misunderstanding something or I’m missing a simple solution.</p>
<p>Any help is greatly appreciated!</p>

username: vdw
cooked: <p>I’m probably bit confused, but don’t you simple want batches of size 32 with a shape <code>(seq_len, batch_size, input_dim)</code>, which would be <code>(100, 32, 8)</code> in your case? This is essentially the standard way to feed an LSTM.</p>
<p>What exactly is stopping you?</p>

username: mitchellg
cooked: <p>I appreciate the reply. I actually misunderstood what the mini-batch dimension in LSTM was doing. You’re correct. Thanks!</p>
GRU implementation by myself

username: Federico_Vasile
cooked: <p>Hi everyone,<br>
I need to implement the Gated Recurrent Unit by myself because I will need to modify some terms in its equations. But, firstly I would like to implement the original GRU to be sure that everything works correctly. This is my implementation, I did a simple test to compare its results with the ones of the Pytorch implementation, but they are different.<br>
Where am I going wrong?</p>
<pre><code>import torch
import torch.nn as nn
from torch.nn.parameter import Parameter

import math

class GRU(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(GRU, self).__init__()

        self.Wx = Parameter(torch.randn(input_size, hidden_size * 3)).div(math.sqrt(input_size))
        self.bx = Parameter(torch.zeros(hidden_size * 3))
        self.Wh = Parameter(torch.randn(hidden_size, hidden_size * 3)).div(math.sqrt(hidden_size))
        self.bh = Parameter(torch.zeros(hidden_size * 3))

    def step(self, x, prev_h):
        batch_size, hidden_size = prev_h.shape

        ax = x.mm(self.Wx) + self.bx
        ax = ax.reshape(batch_size, 3, hidden_size)
        ah = prev_h.mm(self.Wh) + self.bh
        ah = ah.reshape(batch_size, 3, hidden_size)

        r = torch.sigmoid(ax[:, 0, :] + ah[:, 0, :])
        z = torch.sigmoid(ax[:, 1, :] + ah[:, 1, :])
        n = torch.tanh(ax[:, 2, :] + r * ah[:, 2, :])

        next_h = (1 - z) * n + z * prev_h
        return next_h

    def forward(self, x, h0):
        batch_size, steps, num_features = x.shape
        hidden_size = h0.shape[1]
        hts = torch.zeros(batch_size, steps, hidden_size).to(dtype=x.dtype)

        h_t = h0
        for step_t in range(steps):
            x_t = x[:, step_t]
            h_t = self.step(x_t, h_t)
            hts[:, step_t] = h_t

        return hts
</code></pre>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: vdw
cooked: <p>Not really helpful, but maybe you can double-check that <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524"><code>reshape()</code> is doing the right thing</a>.</p>
Visualize PyTorch GPU arrays directly without copying them from GPU to main memory and back

username: multilinear_map
cooked: <p>I’d like to visualize a PyTorch array (for example an RGB image that is the output of a neural network) that is in GPU memory.</p>
<p>The array changes very often, I want to see it “live”, with little latency and with high frame rate.</p>
<p>Usually I copy it from GPU memory to main memory and then visualize it (i.e. then the image usually goes back to GPU memory in order to be visualized, as far as I know). This copying back and forth creates unnecessary overhead.</p>
<p>Can I instead allow PyTorch to write directly to some graphics buffer? Or at least fetch the data from PyTorch to a graphics buffer (i.e. GPU to GPU, instead of GPU to main memory to GPU)?</p>
<p>An example application is the usage of neural networks for effects/filters for games, screensavers, interactive art installations, live previews in image and video editing, …</p>
Multi-GPU training

username: sarthakpati
cooked: <p>I am trying to split my data to run on multiple GPUs, but my program is only able to find 1 GPU. Here is what I have so far:</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"]
</code></pre>
<p>Gives: <code>0,1</code>, which is correct as I have 2 GPUs in the node I want to train on. However,</p>
<pre><code class="lang-auto">torch.cuda.device_count()
</code></pre>
<p>Gives <code>1</code>, which is not what I was expecting. I am setting the torch device as <code>cuda</code> and not specifying a device ID there.</p>

username: ptrblck
cooked: <p>Are you able to use both devices in other applications?<br>
Also, what does <code>nvidia-smi</code> show?</p>

username: sarthakpati
cooked: <p>So, this is a compute node in our HPC cluster, which I cannot access directly. NVIDIA-SMI is able to see 2 GPUs (I am using <a href="https://github.com/anderskm/gputil" rel="nofollow noopener">this module</a> to get the available GPUs).</p>
<p>This is my first time trying to leverage both the GPUs, so I am not sure how I can test that both are getting used.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sarthakpati" data-post="3" data-topic="91254">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarthakpati/48/27366_2.png" class="avatar"> sarthakpati:</div>
<blockquote>
<p>This is my first time trying to leverage both the GPUs, so I am not sure how I can test that both are getting used.</p>
</blockquote>
</aside>
<p>I don’t think you could use PyTorch for this test, since it’s currently returning a device count of 1.<br>
I was wondering, if you were able to use more than the one found device using any other application.</p>

username: sarthakpati
cooked: <p>Yes, I am able to use the two GPUs in another application.</p>

username: ptrblck
cooked: <p>Unfortunately, I don’t know what might go wrong in that case and would recommend to either update the drivers or try to use a docker container and check, if both GPUs are found there.</p>

username: sarthakpati
cooked: <p>Okay, thanks. If I manage to figure it out, I will post the solution here.</p>
Code runs faster on local GPU than cluster

username: mhong94
cooked: <p>My code runs so much faster on my single GTX 1060 than on the cluster which has 2 GTX 1080 Ti.  This is strange because a few parts of the code run faster on the cluster, although most parts run slower.  For some comparison, enumerating through the dataloader takes ~9 seconds on my local while it takes 550 seconds on the cluster.  Also, calculating loss + backprop takes ~ 1 seconds on my local, while it takes 10 seconds on the cluster.  The code is from this paper <a href="https://github.com/Philip-Bachman/amdim-public" rel="nofollow noopener">https://github.com/Philip-Bachman/amdim-public</a></p>

username: ptrblck
cooked: <p>How did you measure the timing in both applications?<br>
If the data loading is 55 times slower on the “cluster”, I would recommend to narrow down this issue first.<br>
E.g. are you storing the data on a network drive or on a local SSD in both cases?</p>

username: mhong94
cooked: <p>I just called time.time() at different sections and computed their differences for timing.  Can you clarify how to narrow down the issue?  The data is stored on my local SSD, and I’m not exactly sure where on the network it’s stored.  The original code used ImageFolder to load the data, and I tried changing it to a standard dataset, but this did not help</p>

username: ptrblck
cooked: <p>If you are timing CUDA operations you would have to synchronize the code before starting and stopping the timer due to the asynchronous execution of CUDA kernels via <code>torch.cuda.synchronize()</code>.</p>
<aside class="quote no-group" data-username="mhong94" data-post="3" data-topic="91328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ecc23a/48.png" class="avatar"> mhong94:</div>
<blockquote>
<p>The data is stored on my local SSD, and I’m not exactly sure where on the network it’s stored.</p>
</blockquote>
</aside>
<p>I don’t understand this explanation. Is the data stored on the SSD in your workstation or on a server in your network (or both)?<br>
In the latter case you would introduce the network latency into the training, so I would recommend to store the data on a local SSD.</p>
<aside class="quote no-group" data-username="mhong94" data-post="3" data-topic="91328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ecc23a/48.png" class="avatar"> mhong94:</div>
<blockquote>
<p>The original code used ImageFolder to load the data, and I tried changing it to a standard dataset, but this did not help</p>
</blockquote>
</aside>
<p>What do you mean by “standard Dataset”? Did you write a custom <code>Dataset</code>?</p>

username: mhong94
cooked: <p>When I run on my local machine, I’m accessing the data on my local SSD.  I copy all code/data over to a storage on my kubernetes pod, then run the code there, using the cluster’s GPUs.  I think that in both cases, they are accessing the data from their local SSDs.</p>
For loop over encoder

username: Paul_Koch
cooked: <p>Dear Community,</p>
<p>I was wondering whether one can for loop over some encoder network (e.g resnet 18 without the last fully connected layer) and afterwards concatenate the embedding. Eventually there is a set of fully connect layers at the end to classify the batch entities. Here is some example code.</p>
<pre><code>    # x.shape = (bs, i, c, h, w) where "i" is some stack of images for each batch entity
    embs = []
    for i in range(int(x.shape()[1])):
        embs.append(self.encoder(x[:, i]))

    # concatenate the embeddings
    x = torch.cat(embs, 1)
    x = self.dense_layer1(x)
    x = self.dense_layer2(x)
    x = self.classifier(x)  # last dense layer
</code></pre>
<p>Can the autograd backpropogate the loss or does anyone see some mistakes?<br>
Would be thankfull for any kind of feedback.</p>
<p>Best,<br>
Paul</p>

username: Kushaj
cooked: <p>You can avoid the for loop.</p>
<pre><code class="lang-python">x = x.view(bs*i, -1)
out = model(x)
out.view(bs, i, -1)
</code></pre>

username: Paul_Koch
cooked: <p>True. Seems to be a better approach. Thou it appears that for loop does the trick as well. But I guess it should run faster if we have a single forward pass.</p>

username: Kushaj
cooked: <p>For loop will run slower. You can time it. In both the cases, the computation is almost the same, but in case of for loop, you have python loop overhead and list extension overhead.</p>
Why mnist has just 50000 images?

username: blackbirdbarber
cooked: <pre><code class="lang-auto">dl_train = DataLoader( torchvision.datasets.CIFAR10('/data/cifar10', download=True, train=True), batch_size=1, drop_last=False, shuffle=False)
print(len(dl_train))
</code></pre>
<p>returns just 50.000 should it be 60.000?</p>

username: blackbirdbarber
cooked: <p>According to Wikipedia</p>
<blockquote>
<p>The MNIST database contains 60,000 training images and 10,000 testing images.</p>
</blockquote>

username: Hmrishav_Bandyopadhy
cooked: <aside class="quote no-group" data-username="blackbirdbarber" data-post="1" data-topic="91870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p><code> torchvision.datasets.CIFAR10</code></p>
</blockquote>
</aside>
<p>Here you go <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: blackbirdbarber
cooked: <p><a class="mention" href="/u/hmrishav_bandyopadhy">@Hmrishav_Bandyopadhy</a></p>
<p>Take my full code execute on your end and let me know if you get 60.000</p>
<pre><code class="lang-auto">import torchvision
from torch.utils.data import DataLoader, Dataset, TensorDataset
dl_train = DataLoader( torchvision.datasets.CIFAR10('/data2/cifar10', download=True, train=True), batch_size=1, drop_last=False, shuffle=False)
print(len(dl_train))
</code></pre>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
You are actually using extracting CIFAR dataset as i mentioned in the previous answer. CIFAR dataset has 50,000 training images and 10,000 testing images.</p>
<p>Use this for MNIST:</p>
<pre><code class="lang-auto">import torchvision
from torch.utils.data import DataLoader, Dataset, TensorDataset
dl_train = DataLoader( torchvision.datasets.MNIST(root, download=True, train=True),batch_size=1, drop_last=False, shuffle=False)
print(len(dl_train))</code></pre>

username: blackbirdbarber
cooked: <p>Thanks <a class="mention" href="/u/hmrishav_bandyopadhy">@Hmrishav_Bandyopadhy</a> you saved my day.</p>
Training Ensemble Networks

username: Aryaman_Sriram
cooked: <p>I am training an ensemble network consisting of two convolutional neural networks. Do I need to define an optimizer for each of them separately or would a single optimizer for the entire model do?</p>

username: mariosasko
cooked: <p>I would advise you to use a separate optimizer for each model/submodel. It gives you more control and you don’t have to syncronize <code>optimizer.zero_grad()</code> and <code>optimizer.step()</code> calls (otherwise, you have to train all the models at the same time). If you decide to use a single optimizer, you can define a different learning policy for each model through parameter groups that are explained in documentation.</p>

username: Hmrishav_Bandyopadhy
cooked: <p>You can do with a single optimizer in case you have ensembled the models as parts of a module like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()

        self.model1=model1()
        self.model2=model2()

    def forward(self,x):
        return (self.model1(x)+self.model2(x))/2.0
        
</code></pre>
<p>But if you have defined 2 separate models and are ensembling them during training, you’d need an optimizer for each one.</p>
<p>A separate optimizer for each model will allow you to have separate learning rates and methodologies in an easier process as compared to a single optimizer.</p>
Gradient for only part of a tensor

username: ilkarman
cooked: <p>Is there a way to not calculate the gradient for a part of a tensor? The only way I think I make this work is to calculate the full gradient and then zero-it out.</p>
<p>For example, here I want to free zethe last row of matrix(T) and only calculate and update gradient for the first two rows:</p>
<pre><code class="lang-auto">theta = torch.ones(1)
dxy = torch.ones(2)
T = torch.tensor([
    [torch.cos(theta), -torch.sin(theta), dxy[0]],
    [torch.sin(theta), torch.cos(theta), dxy[1]],
    [0.0, 0.0, 1.0],
]).type(torch.cuda.FloatTensor).cuda()
T.requires_grad = True

# Forwards
out = torch.rand(1, 3).cuda().mm(T)
gt = torch.zeros(1).cuda().long()
loss = torch.nn.functional.cross_entropy(out, gt)

# Back
loss.backward()
T.grad[-1] = torch.zeros(3)

# Optimizer step
with torch.no_grad():
    T -= 0.01 * T.grad
</code></pre>

username: blackbirdbarber
cooked: <p>After you called.</p>
<aside class="quote no-group" data-username="ilkarman" data-post="1" data-topic="91871">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilkarman/48/1350_2.png" class="avatar"> ilkarman:</div>
<blockquote>
<p><code>loss.backward()</code></p>
</blockquote>
</aside>
<p>And based on the fact your tensor <code>T</code> requires grad the gradients will be calculated for the entire <code>T</code></p>
<p>You cannot make just part of a <code>T</code> getting the gradient. No way to set the mask. The info about the gradients is inside the computational tree = dynamic computational graph.</p>
<p>So, I think the only way is to zero the gradients like you did.</p>

username: blackbirdbarber
cooked: <p>There is one option still, you may concat two tensors <code>T</code> and <code>nogradT</code>, and since concatenation is visible to autograd system, you may set <code>T</code> to requires grad and part of the <code>nogradT</code> not to have gradients. This way you will have gradients only for <code>T</code>.</p>