akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LSTM CNN Loss issue

Jaideep_Valani: <p>Below is the LSTM layer i used for LSTM CNN<br>
But i will have trouble calculating the loss  due mistmach in target and input size.<br>
Am i missing any step here</p>
<pre><code class="lang-auto">class lstm(nn.Module):
    def __init__(self,bs,num,  lstm1_input=1024,lstm_hidden=512):
        super().__init__()
        self.batch=bs
        self.num=num
        
       
        self.lstm = nn.LSTM(1024, 512 ,2,batch_first=True)# input elements,hidden state size
        #self.linear=nn.Linear(512,1)
        self.drop=nn.Dropout(0.5)
    def forward(self,input):
        print(input.size()) #see below
        x=input.view(self.batch,self.num,input.size(1))
        x,y=self.lstm(x)
        print(x.size()) #see below
        x = x[:,-1,:]
        print(x.size()) #see below
        x=self.drop(x)
        return x
</code></pre>
<p>torch.Size([192,1024])<br>
torch.Size([16, 12, 512])<br>
torch.Size([16, 512])</p>

bigyeet: <p>Unless your target size is the same as your lstm’s hidden dim, you need to apply a linear layer after the lstm to transform it to the right size and also get rid of the tanh activation coming out of the lstm.</p>
<p>nn.linear(512, TARGET_SIZE)</p>
<p>I also recommend removing the dropout if this is at the very end of your network.</p>

Jaideep_Valani: <p>i can very well use that… but batch size is not same as the target batch size…</p>

bigyeet: <p>Your sequence length seems to be 12 so you need to use every 12th other target in your target tensor, not the entire thing. Or you could make your input in the shape of (batch, seq, *inputshape) and output in the shape of (batch, *targetshape) and remove the view step in your forward</p>

Jaideep_Valani: <p>Thanks…<br>
do you have any code link to perform the set of steps ?</p>
Does DDP tolerate heterogeneity?

Vikings: <p>I’m currently implementing a heterogeneity-aware distributed model.<br>
My basic idea is to do all-reduce on a subset of the fast workers in the world group, however I noticed that in <code>torch.distributed.</code> <code>new_group</code>, it says:</p>
<blockquote>
<p>This function requires that all processes in the main group (i.e. all processes that are part of the distributed job) enter this function, even if they are not going to be members of the group.</p>
</blockquote>
<p>Does that mean that the fast workers should wait for the slow workers before they can move forward (i.e. the slower one will sync the faster one)?<br>
If so, is there any other way to implement the model?</p>

mrshenli: <blockquote>
<p>Does that mean that the fast workers should wait for the slow workers before they can move forward (i.e. the slower one will sync the faster one)?</p>
</blockquote>
<p>No. It only requires all processes to call that function for rendezvous. After that, collective communications (e.g., allreduce) within a subgroup do not require non-member processes to join. So different subgroups can do allreduce independently.</p>
<p>For implementation, you could create multiple DDP gangs, on the same model, with each gang spans different processes. But then, you will need to coordinate the communication because different DDP gangs will read from and write to the same set of <code>param.grad</code> field. Application needs to avoid the race there.</p>
Clarification on autograft with multiple losses

NHH: <p>Given a model (e.g. a cnn) with two losses, with the first loss (loss1) computed halfway through the cnn and the second loss (loss2) computed at the end of the cnn. Then, these two losses are summed and we do total_loss.backward() (total_loss = loss1 + loss2). Will the gradients of the second half of the cnn be impacted by loss1 (i.e. would the gradients of the second half of the cnn be same if we did instead loss2.backward() )?</p>

albanD: <p>Hi,</p>
<p>No it won’t because for a parameter w in the second half of your net, dloss1/dw = 0. So the contribution of these will just be 0 (ignored in practice).</p>
Computing Jacobian and Hessian of Loss w.r.to Biases

Jamesswiz: <p>Hi,<br>
what is the simplest way to compute full Jacobian and Hessian of Loss w.r.to neural network biases only.<br>
I want to do this layer wise but considering one layer at a time and I am not worried about speed for now.</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>You can check the implementation here: <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7">https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7</a> where you give only the bias as inputs.</p>

Jamesswiz: <p>This gives me error:<br>
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>If I do recursive grad(grad), I don’t get the full hessian as we all already know.</p>

albanD: <p>Does replacing this:</p>
<pre><code class="lang-auto">grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
</code></pre>
<p>by</p>
<pre><code class="lang-auto">grad_x, = torch.autograd.grad(flat_y[i], x, retain_graph=True, create_graph=create_graph)
</code></pre>
<p>solves the error?</p>

Jamesswiz: <p>No, now I get:<br>
grad can be implicitly created only for scalar outputs</p>

albanD: <p>The <code>y</code> should be a single Tensor right?<br>
So <code>flat_y</code> here should be a 1D Tensor. And <code>flat_y[i]</code> will be a single number. So it is a scalar output.<br>
Is the error coming from this function?</p>

Jamesswiz: <p>Thanks, got it…<br>
It works now.</p>
About batchsize

lzkzls: <p>In train.py, I set the batchsize to 16. I transferred a batch of data into a customized neural network. In the forward function of the network, I printed the dimension information of the data and found that the batchsize changed to 8. What caused this?</p>

G.M: <p>Can u show the code of ur model and how u fed the batch into the model?</p>

lzkzls: <p>Thanks for your reply. I have found the reason of this phenomenon. Because my computer is equipped with two cards, and I used nn.Dataparallel in the program.</p>
Libtorch cpu incompatible with libtorch cuda

Robin_Lobel: <p>For training, I use the libtorch CUDA 10.1 package as it’s way faster than training on the CPU.<br>
For deployment, I want to use the libtorch No CUDA (CPU only) package as it’s lighter and I don’t need GPU acceleration.<br>
Official 1.4.0 packages in both cases, Windows.</p>
<p>However I just ran into an incompatibility issue. All the models I’ve trained using the CUDA package, and saved using torch::save(model,path) can’t be loaded with torch::load(model,path) in the CPU-only package. But the opposite is not true:</p>
<p>-Models trained with the CUDA package can be saved and loaded with the CUDA package but can’t be loaded by the CPU-only package: the software crash without any explanation when torch::load is called.<br>
-Models trained with the CPU package can be saved and loaded with the CPU-only package and the CUDA package.</p>
<p>I’ve uploaded 2 files, one trained with the CPU-only package, one with the GPU package:<br>
<a href="https://drive.google.com/open?id=1U0-5pto40Tn9n4EsoW1P56E9PI1dyEKP" class="onebox" target="_blank" rel="nofollow noopener">https://drive.google.com/open?id=1U0-5pto40Tn9n4EsoW1P56E9PI1dyEKP</a><br>
Both use the exact same model/training data, saved after only one epoch. They are exactly the same size, and looking at the content in an hex editor, they seem to be structured the same way.</p>
<p>This is quite problematic. Are you aware of this issue ? Loading failed with the nightly build of CPU-only 1.5 as well.</p>

tom: <ol>
<li>What’s the error message message you are getting?</li>
<li>Were the models you serialized on GPU or CPU?</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>

Robin_Lobel: <p>No error, the software just stop running. No crash dialog.</p>
<p>However I finally found the solution ! When I train a model using CUDA, I should send back my model to CPU using model.to(kCPU) before saving it, so that it’s compatible with the CPU-only package as well.</p>
<p>So I was able to convert my CUDA-trained models to CPU-trained models using the CUDA package by doing load(model, oldfile); model.to(kCPU); save(model,newfile);</p>
<p>Here’s the hex difference between the CUDA file and the CPU file, towards the end of the file:<br>
                    <a href="https://i.imgur.com/McoHd1r.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/McoHd1r.png" width="401" height="500">
          </a>

</p>
<p>Everything else in the file is strictly identical, just that block differs (notice “cpuq” vs “cuda” at the top right of the green block) and make the CPU-only package crash (with no error message) when trying to load the CUDA-saved file.</p>
<p>I’m a little surprised this is happening as I thought the model file only contained float values, with no affinity to the processing unit ?</p>
Applying same module to a list of inputs, in parallel

colobas: <p>Hi. I’m working on an architecture where my input is a list of tensors (say 5 or 10), and I need to feed them all through the same subnet, take their respective outputs and aggregate them. I implemented this in a sequential manner like:</p>
<pre><code class="lang-auto">outs = []
for inp in inputs:
    out = module(inp)
    outs.append(out)

result = some_aggregation_logic(outs)
</code></pre>
<p>I’m wondering what is the appropriate way to parallelize that for-loop (if there even is one). It’s probably worth noting that <code>out = module(inp)</code> processes the i-th object for every element in the batch.</p>
Converting vector of double to tensor, issues

Jesper.Lindberg: <p>Hello!<br>
I am trying to convert an std::vector to torch:Tensor with the following code:</p>
<blockquote>
<p>torch::Tensor next_state = torch::from_blob(replay_memory.at(trainingIDX + 1).current_state.data(), { k_input_size_ }).to(*device);</p>
</blockquote>
<p>However, I have done some troubleshooting since my traning loop which uses this tensor crashes since it han nan values in it.</p>
<p>So I tried:</p>
<blockquote>
<p>std::cout &lt;&lt; replay_memory.at(trainingIDX+1).current_state &lt;&lt; std::endl;<br>
torch::Tensor next_state = torch::from_blob(replay_memory.at(trainingIDX + 1).current_state.data(), { k_input_size_ }).to(*device);<br>
std::cout &lt;&lt; "next state is: " &lt;&lt; next_state &lt;&lt; std::endl;</p>
</blockquote>
<p>Which at the iteration of the crash gives me:</p>
<blockquote>
<p>0.0642382 0.0395936 0 0.219108 0.372894 0.422909 0.554452 0.302765 1<br>
next state is: -1.7024e+35<br>
1.3785e+00<br>
-6.2066e+14<br>
1.2834e+00<br>
0.0000e+00<br>
0.0000e+00<br>
-1.0163e+17<br>
1.5941e+00<br>
nan</p>
</blockquote>
<p>Note that:</p>
<blockquote>
<p>torch::Tensor current_state = torch::tensor(replay_memory.at(trainingIDX).current_state).to(*device);</p>
</blockquote>
<p>Gives me a tensor that looks the way I want it but then the program crashes when I try to feed it through the network, an issue I did not have with from_blob…</p>

Hartbook: <p>Hello,</p>
<p>I think you should specify the size of tensor elements when using from_blob, like : <code>torch::from_blob(replay_memory.at(trainingIDX + 1).current_state.data(), { k_input_size_ }, torch::TensorOptions(torch::kFloat64))</code></p>
<p>Also if your device is cuda and if your data <code>replay_memory.at(trainingIDX + 1).current_state.data()</code> does not live in gpu memory you will have to clone the tensor before using <code>to</code> , like :</p>
<pre><code class="lang-auto">torch::from_blob(...).clone().to(*device)
</code></pre>

Jesper.Lindberg: <p>Thanks for the reply. I tried what you said with both kFloat64 and kDouble. However, I still get an error when I try to forward this tensor in the network:</p>
<blockquote>
<p>Exception: Expected object of scalar type Double but got scalar type Float for argument <span class="hashtag-raw">#2</span> ‘mat2’ in call to _th_mm (checked_dense_tensor_unwrap at C:\w\1\s\windows\pytorch\aten\src\ATen/Utils.h:84)</p>
</blockquote>
<p>Also, I’m not using GPU, only CPU.</p>

Hartbook: <p>The module parameters interacting with your input tensor should probably have the same type as the input tensor.<br>
I think it would be easier to work with torch::kFloat tensor so you should convert your data to float before creating the input with from_blob</p>

Jesper.Lindberg: <p>This worked and I will mark you post as the solution, however I am not totally happy with having to convert my double vector to a float.</p>
Best practice for working with a lot of data (files)? How to store and load data?

shorty: <p>I am currently working on an object detection pipeline with the xView satellite image data set inside a google colab notebook. Therefor i do have a folder with all the images on my google drive folder which i mount into the notebook. Since there are 80k+ images in one single folder it (obviously?) comes to errors when loading data from the folder.</p>
<p>How are you guys storing / loading big data sets into pytorch?</p>
Var.detach().numpy() but want to compute the automatic gradient

takafusui: <p>Hi,</p>
<p>I am solving a non-linear optimization problem and use <code>torch.autograd.grad</code> to provide Jacobian that an optimizer requires.<br>
Please see the code snippet below:</p>
<pre><code class="lang-python">_stateplus = torch.stack([x[0], x[1], _ar1_plus]).numpy()
_controls_plus[z_idx, epsilon_idx, :] = policy_plus[z_idx].evaluate(
   _stateplus)
</code></pre>
<p>where <code>x</code> is a tensor variable with respect to which I want to compute the gradient.<br>
I add the <code>.numpy()</code> argument, because the <code>.evaluate()</code> accepts only <code>numpy</code> array.</p>
<p>But in this case, I encounter the following error as expected:</p>
<pre><code class="lang-python">RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.
</code></pre>
<p>I think the <code>detach().numpy()</code> argument breaks the computational graph, but I need a gradient with respect to <code>x</code>.</p>
<p>How can I avoid this situation?</p>

albanD: <p>Hi,</p>
<p>Unfortunately, you will only be able to get gradients using autograd if you use pytorch’s Tensor and operators to do all the computations.</p>
<p>If you have an operation that you cannot do this way, you will need to provide the gradient formula yourself for that step. You can see how to extend the autograd in <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd">this Note</a>.</p>

takafusui: <p>Hi,</p>
<p>Thank you for the link.</p>
<p>So you meant <code>var.detach().numpy()</code> breaks the computational graph and we cannot compute the gradient correctly anymore by using <code>torch.autograd.grad()</code>. Is it correct?</p>

albanD: <blockquote>
<p>So you meant <code>var.detach().numpy()</code> breaks the computational graph and we cannot compute the gradient correctly anymore by using <code>torch.autograd.grad()</code> . Is it correct?</p>
</blockquote>
<p>It is correct. Because you convert the Tensor into a numpy array, we cannot track gradients anymore.</p>

takafusui: <p>Thank you. It is a bit sad news but I try to come up with another route.</p>

albanD: <p>Unfortunately we cannot do magic <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> For AD to work, we need to know everything that you do with the data. If we cannot know what you function did, we cannot compute gradients for it.</p>

takafusui: <p>Yes you are definitely correct. Thank you again.</p>
Implement Nested Dataset

Hakjin_Lee: <p>Hello, I am working on few-shot learning framework.<br>
In few-shot learning, each batch should contain KN image, label pair, where K is the number of classes and N is the number of shots(images).</p>
<p>In this reason, I want to design dataset and dataloader as bellow,<br>
In each iteration, dataloader iterates with class, and dataset for each class iterates with items.</p>
<p>class FSLDataset(Dataset): each item is ClassDataset which has data with a same class.<br>
class ClassDataset(Dataset): each item is pair of images and label.</p>
<p>However, torhch.utils.data.Dataset only supports with tensors, numpy arrays, numbers, dicts or lists.</p>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘<strong>main</strong>.ClassDataset’&gt;</p>
<p>Do you have any idea on this design?</p>

Hakjin_Lee: <p>I find my own solution.</p>
<pre><code class="lang-auto">class NestedMetaDataset(torch.utils.data.Dataset):
    def __init__(self, load_dataset_class, classes, batch_size, **args):
        #self.datasets = [ClassDataset(load_dataset_class, cls, **args) for cls in classes]
        self.datasets = []
        self.len = 0

        for cls in classes:
            cls_data = ClassDataset(load_dataset_class, cls, **args)
            self.len += len(cls_data)
            self.datasets.append(iter(cls_data))

        self.class_num = len(classes)
    
    def __len__(self):
        return self.len
        
    def __getitem__(self, index):
        class_id = index % len(self.datasets)
        dataset = self.datasets[class_id]
        item = next(dataset)
        
        return item
        
class ClassDataset:
    def __init__(self, load_dataset_class, class_name, **kargs):
        self.items = load_dataset_class(class_name=class_name, **kargs)

    
    def __len__(self):
        return len(self.items)
    
    def __getitem__(self, index):
        item = self.items[index]
        
        return item
</code></pre>
Finding implementation of BatchNorm backward

aashaka: <p>I am confused about which batch_norm_backward code is triggered when cudnn_batch_norm_backward is called. Is the code at <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/BatchNorm.cpp#L217" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/BatchNorm.cpp#L217</a> triggered or is it batchnorm_double_backward (<a href="https://github.com/pytorch/pytorch/blob/51d969e86ac0fad226f4e70889df0d3c6114ae4e/tools/autograd/templates/Functions.cpp#L2289" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/51d969e86ac0fad226f4e70889df0d3c6114ae4e/tools/autograd/templates/Functions.cpp#L2289</a>) which is triggered, as defined in autograd derivatives.yaml (<a href="https://github.com/pytorch/pytorch/blob/d66c320b10a012575a842f1fd0b60912c1f2e7a2/tools/autograd/derivatives.yaml#L1481" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/d66c320b10a012575a842f1fd0b60912c1f2e7a2/tools/autograd/derivatives.yaml#L1481</a>)</p>
<p>What is the use of having multiple such definitions?</p>
Forward pass while using auxilary loss

lycaenidae: <pre><code class="lang-auto">def forward(self, x):
        """
        Inputs:
        - x: PyTorch input Variable
        """
        return self.backbone(x)['out']
</code></pre>
<pre><code class="lang-auto">def forward(self, x):
        """
        Inputs:
        - x: PyTorch input Variable
        """
        return self.backbone(x)['aux']
</code></pre>
<p>I want to use auxilary loss while doing semantic segmenation. But which one should I return ‘out’ or ‘aux’?  In the documentation it is stated that</p>
<blockquote>
<p>class DeepLabV3(<em>SimpleSegmentationModel):<br>
“”"<br>
Implements DeepLabV3 model from<br>
<code>"Rethinking Atrous Convolution for Semantic Image Segmentation"     &lt;https://arxiv.org/abs/1706.05587&gt;</code></em>.</p>
</blockquote>
<pre><code>Arguments:
    backbone (nn.Module): the network used to compute the features for the model.
        The backbone should return an OrderedDict[Tensor], with the key being
        "out" for the last feature map used, and "aux" if an auxiliary classifier
        is used.
    classifier (nn.Module): module that takes the "out" element returned from
        the backbone and returns a dense prediction.
    aux_classifier (nn.Module, optional): auxiliary classifier used during training
"""
pass
</code></pre>
Is there a minimum computation time with gpu?

robj: <p>Hi, I’m trying to start pytorch.</p>
<p>but I saw that using gpu(cuda:0) 's performance was not fast as I expected.</p>
<p>So, for a test, I made a very simple script to check it,<br>
and found,<br>
No matter how small the network, there is a minimum computation time for optimizing network…</p>
<p>am I right? and if it is not, could you tell me how can I reduce this computation time?</p>
<p>===========================================================================</p>
<p>scipt used for test are here : <a href="https://github.com/sizsJEon/pytorchTest/blob/master/torchAdamTest.py" rel="nofollow noopener">https://github.com/sizsJEon/pytorchTest/blob/master/torchAdamTest.py</a></p>
<p>I’m using pytorch 1.4 and rtx2060 with cuda.</p>
<p>I tested network with<br>
(H1, H2) = (4, 2), (24, 12), (600, 300), (2000, 1000)</p>
<p>at (4,2)~(600,300), computation time was very similar.</p>
<p>modelTime: 0.17~0.19 ms<br>
lossTime : 0.45~0.47 ms<br>
optiTime : 0.52~0.55 ms</p>

ptrblck: <p>CUDA operations are asynchronous, so you would have to synchronize your code before starting and stopping your timer via <code>torch.cuda.synchronize()</code>.</p>
<p>That being said, you can generally ignore the Python overhead and kernel launch times, if your workload is large enough.<br>
However, if you are planning of training tiny models, your GPU utilization might be low, due to data bottlenecks or other issues.</p>
Can I use batch normalization and layer normalization both to CNN-RNN model?

Jaeho_Choi: <p>I’m performing a classification task with time series data. Therefore, I designed an 1DCNN-LSTM model. Currently, 1d-batch normalization layers are applied for CNN part, but I’m not sure to use layer normalization for RNN part.</p>
<p>So, my question is, batch norm layers and layer norm layers can be used simultaneously in a single network?</p>
<p>Thanks.</p>

G.M: <p>I don’t think batchnorm should be used with variable-length time series data(u can google for why, and I’ve had bad experience in cases very similar to yours). Using layernorm for LSTM should improve the performance in most cases(sometimes not very significant, though), but u must use it correctly, see <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py</a>.</p>

Jaeho_Choi: <p>Then, you mean 1-D batchnorm layer may impair the performance? I’ve searched for google, but cannot find the reasons, so can you let me know the reasons or some links?<br>
Thanks for your reply.</p>

G.M: <p>The following is only my opinion, which is developed from various articles I saw on the internet and my speculations, so please correct me if u find a better explanation:</p>
<ol>
<li>If the batch size is 1, batch norm is bad because batch norm requires a relative big batch size to be able to function well.</li>
<li>If the batch size is bigger, there should be some padding values for sure, and batch norm will take that into account, which will probably degrade the performance.</li>
</ol>

Jaeho_Choi: <p>Thank you for your helpful response.</p>
Accessing arbitrary dimension

G.M: <p>I have a N-D tensor where N is arbitrary. What I want to do is I want to access the last dimension of the tensor. For example, say  I have a tensor of  shape <code>[A, B, C, K]</code>, I want to is something like</p>
<pre><code class="lang-auto">a = tc.randn(A, B, C, K)
a[:, :, :, K] = 1
</code></pre>
<p>except that there can be any number of dimensions before <code>K</code>. It seems that if I only want to get the value I can use <code>index_select</code>, but <code>index_select</code> doesn’t share the same storage with the original tensor so I can’t modify the values using it.</p>

ptrblck: <p>Would <code>tensor.index_fill_</code> work?</p>

G.M: <p>Thanks, that solved my problem. I thought that tensor operations are all available under <code>torch</code> so I was not looking for the method in <code>torch.Tensor</code> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

G.M: <p>Another follow-up question: how should I set it to another tensor? For example, instead of</p>
<pre><code class="lang-auto">a[:, :, :, K]=1
</code></pre>
<p>what should I do to achieve</p>
<pre><code class="lang-auto">o = tc.randn(A, B, C)
a[:, :, :, K] = o
</code></pre>

G.M: <p>It seems that <code>torch.narrow</code> works in this case.</p>
Which input to use when generating a new sequence

kenenbek: <p>I want to use sequence-to-sequence architecture to generate sequences.</p>
<p>My data has such structure</p>
<pre><code>[0, 0, 1, 0, ..., 0, 1] --&gt; [12.34, 0.78, 1.54, 6.90, ..., 5.32]
</code></pre>
<p>I follow this <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">tutorial</a> to achieve it.</p>
<p>After forwarding through <code>Encoder</code> network <code>encoder_hidden</code> is used as a <code>decoder_hidden</code>. But what should I use as a first <code>decoder_input</code> to Decoder network?</p>
<p>The original tutorial uses a <code>Start Of the Sequence</code> token, but I can’t use it because it is encoded as <code>0</code>. Probably <code>0</code> as a number will give some additional information for decoder.</p>

G.M: <p>I think the SOS token can be any positive number; the network will learn the number.</p>

vdw: <p>Apart from that I have no idea what you’re trying to learn here…</p>
<p>You can use any number <code>x</code>, it only as to adhere to the following constraints</p>
<ul>
<li>
<p><code>0 &lt;= x &lt;= (M-1)</code> with <code>M</code> being size of your vocabulary + number of special tokens. For example, you have a vocabulary of size 10,000 and have 4 special tokens (very common: <code>&lt;SOS&gt;</code>, <code>&lt;EOS&gt;</code>, <code>&lt;PAD&gt;</code>, <code>&lt;UKN&gt;</code> for start of sequence, end of sequence, padding and unknown tokens). So <code>x</code> can be 0, 1, 2, …10,003.</p>
</li>
<li>
<p><code>x</code> cannot be taken by a word from you vocabulary. My <code>vocab2idx</code> mapping usually looks like <code>{'&lt;PAD&gt;': 0, &lt;UKN&gt;': 1, '&lt;SOS&gt;': 2, '&lt;EOS&gt;': 3, 'the': 4, 'a': 5, 'is':6, ...}</code>. So my start token would be <code>2</code></p>
</li>
</ul>
Pytorch multi gpu error

mysql403: <p>I have two models: encoder and decoder, i want put these two model to cuda0, 1 and paralle use cuda<br>
i just</p>
<pre><code class="lang-auto">        self.device = torch.device("cuda")
        self.models["encoder"] = networks.ResnetEncoder(
        self.opt.num_layers, self.opt.weights_init == "pretrained")
        self.models["encoder"].to(self.device)
        self.models['encoder'] = nn.DataParallel(self.models['encoder'])


        self.parameters_to_train += list(self.models["encoder"].parameters())

        self.models["depth"] = networks.DepthDecoder(
            self.models["encoder"].module.num_ch_enc, self.opt.scales)
        self.models["depth"].to(self.device)
        self.models["depth"] = nn.DataParallel(self.models["depth"])

        self.parameters_to_train += list(self.models["depth"].parameters())

        for key, ipt in inputs.items():
            inputs[key] = ipt.to(self.device)

        features = self.models["encoder"](inputs["color_aug", 0, 0])

        for fea in features:
                print('fea:', fea.device)
       print('enoder:', self.models["encoder"].device_ids)
       print('depth', self.models["depth"].device_ids)
       outputs = self.models["depth"](features)
       
</code></pre>
<p>then run it gives error like this：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5deec9ff0c2d036294e4156590dc5ac89f1d82e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/5deec9ff0c2d036294e4156590dc5ac89f1d82e7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5deec9ff0c2d036294e4156590dc5ac89f1d82e7_2_690x331.png" alt="image" data-base62-sha1="doY0utm21RvYSDbSiJsiXpGacUT" width="690" height="331" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5deec9ff0c2d036294e4156590dc5ac89f1d82e7_2_690x331.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5deec9ff0c2d036294e4156590dc5ac89f1d82e7_2_1035x496.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5deec9ff0c2d036294e4156590dc5ac89f1d82e7.png 2x" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1354×651 117 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems the input data location not match to weight location, and i also notice that eocoder and depth all existence in card 0 and 1, just data in cuda 0, so it shouldn’t report bug, because, the data can match to any model in cuda 0, Is not it?<br>
How fix it? anyone help me?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Indexing over DataLoader

hatef: <p>Hi there,</p>
<p>I would like to access the batches created by DataLoader with their indices. Is there an easy function in PyTorch for this?</p>
<p>More precisely, I’d like to say something like:</p>
<pre><code> val_data        = torchvision.datasets.ImageFolder(root='./imagenet2012',transform=transform);
 val_loader     = torch.utils.data.DataLoader(val_data, batch_size=batchSize)

 for i in range(len(val_loader)):
      inputs, _        = val_loader.__getbatch__(i);
</code></pre>
<p>Any comment is much appreciated.<br>
Hatef</p>

smth: <p>There isn’t a way to do this directly.</p>
<p>However, if you modify this file slightly, it’ll be possible: <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L176-L201">https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L176-L201</a></p>
<p>Particularly, see the <code>next</code> function of the DataLoaderIter, you could enumerate <code>self.sample_iter</code> fully before hand (the indices of each mini-batch), and then you can have a function on that iterator that just returns a particular <code>index</code>.</p>

f10w: <p><a class="mention" href="/u/hatef">@hatef</a> Have you successfully implemented that and would you mind sharing your code please? I’m still have difficulty despite <a class="mention" href="/u/smth">@smth</a> 's hint <img src="//discuss.pytorch.org/images/emoji/apple/frowning.png?v=5" title=":frowning:" class="emoji" alt=":frowning:"></p>

magnus_w: <p>hey <a class="mention" href="/u/smth">@smth</a>,</p>
<p>I’d like to use num_workers &gt; 1 and be able to receive the indices. I’m not so familiar with multiprocessing. Is it sufficient to return indices from _put_indices() , <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L290" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L290</a> ?</p>
<p>regards,<br>
magnus</p>

shaghayegh_nadjari: <p>you can you can use tqdm_notebook pakage to do that:</p>
<p>for i, (train_inputs, train_labels) in tqdm_notebook(enumerate(dl), total=n):<br>
train_inputs, train_labels = to_var(train_inputs), to_var(train_labels)</p>
Unable to load dataloader from .pt file

lc5415: <p>Hello,</p>
<p>I saved my processed data loader as .pt file to avoid the loading/preprocessing time every time I want to train a model and so that I could compare different networks that had been trained on exactly the same images.</p>
<p>My loader is called <em>train.pt</em> and when using torch.load(“path/to/train.pt”) I get the following error:</p>
<pre><code class="lang-auto">return legacy load(opened file, map_location, pickle_module, **pickle load args)
File "~/home/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load 
result = unpickler.load().
</code></pre>
<p>It also output a ModuleNotFoundError pointing to a folder that’s not being called at all in the script.</p>
<p>Finally, I want to add that when I try to load the dataloader from a python console, it does not give any error. Any help?</p>
<p>PD: I’m using CUDA in a Unix cluster system.</p>

ptrblck: <p>What did you assign to <code>train.pt</code> and how did you load your data in your <code>Dataset</code>?<br>
If you’ve loaded it lazily, you won’t be able to store the <code>DataLoader</code> including the loaded files, as the <code>DataLoader</code> does not store the data internally.</p>

lc5415: <p>Hello ptrblck, thanks for the quick reply. I create a custom Dataset class (similar to the one from <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this tutorial</a> )</p>
<p>I then store the dataloader by doing,</p>
<pre><code class="lang-auto">dataloader = DataLoader(dataset, batch_size=batch_size,
                            shuffle=True, num_workers=cores, drop_last = True)
if save:
        torch.save(Loader, os.getcwd()+"/labelled/"+savename)
</code></pre>
<p>If I cannot save my dataloader this way, how could I achieve my aim? (storing the transformed images to avoid the time it takes to transform + for reproducibility)</p>
<p>Best,<br>
Luis</p>

ptrblck: <p>If you are lazily loading and transforming the data, you could save each sample in your <code>__getitem__</code> method.<br>
However, I don’t think this is a good idea, as you will just store the same number of samples after preprocessing and transforming them.</p>
<p>If your workload is large enough (i.e. if your model is not a small one and the GPU needs some time for the training), your data loading and processing might be fast enough to provide batches in the background.<br>
Are you seeing a data loading bottleneck at the moment?</p>

lc5415: <p>My biggest concern was the lack of reproducibility that could arise from each image being randomly cropped. I will now be centre cropping my images instead to work around this.</p>
<p>You are right, my workload is quite large and comparatively the loading is not that bad (maybe 5% of total running time). Thanks for your advice <code>ptrblck</code> !</p>
Long vs Flatten

dachosen1: <p>I’m evaluating the performance of my model using the cross-entropy loss. I flatten my target variables using flatten construct from torch to get a [x,1] tensor object. I received a Runtime Error: Expected object of scalar type Long by got a scalar type Int for argument <span class="hashtag">#2</span> target.</p>
<p>I was able to fix using the scalar type long. I still don’t understand why it didn’t work since borth functions produce the same shape? What’s the difference betweeen Long vs Flatten in Pytorch?</p>

ptrblck: <p>Calling <code>tensor.long()</code> returns a <code>LongTensor</code>, which is the tensor data type.<br>
The error message explains, that you are passing an <code>IntTensor</code> instead of a <code>LongTensor</code>.<br>
The data type is independent of the tensor shape.</p>

dachosen1: <p>I was able to address the error, but didn’t know the difference between the two types. From what I was able to gather a <code>LongTensor</code> is a 64bit integer vs an <code>IntTensor </code>is a 32bit integer. Did i miss anything else?</p>

ptrblck: <p>Your understanding is correct. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Merging 3 image batch tensors

Topsoil: <p>inside my model’s forward() method i split input image into three channels, and pass each one through separate modeule. After that, i need to somehow combine each batch of color channels back together</p>
<p>So before merging i have three tensors with shape [batch_size,1,128,128]</p>
<p>After - [batch_size,3,128,128]</p>
<p>How do i do it?</p>

JakeAndFinn: <p>I think you will need to use torch.cat to get the job done.</p>
<p>So here is how I think you should do it.</p>
<pre><code class="lang-auto">x = torch.randn(32, 1, 128, 128) # You dont need this part
new_tensor = torch.cat((x,x,x), 1) # to concatinate on the 1 dim Just this part
</code></pre>
<p>This should give you the torch.Size([32, 3, 128, 128]) the results you want<br>
Where x is your tensors so you might do it like this<br>
new = torch.cat((a,b,c), 1)</p>
Got a warning when using cpp_extension

Vikings: <p>I’m trying to build a C++ extension using torch.utils.cpp_extension. My OS is Mac OS 10.15.3 and I got the warning:</p>
<blockquote>
<p>Your compiler (xcrun) is not compatible with the compiler Pytorch was built with for this platform, which is clang++ on darwin. Please use clang++ to to compile your extension.</p>
</blockquote>
<p>How to change the compiler?</p>
<p>Thanks!</p>
Speeding up inference on CPU with a complex and dense model

Duck: <p>The model I am using is extremely complex and does not just pass data between different pytorch models. In between various math operators are performed and reshapes of the data. When I applied the pytorch pruning (1.4) I was able to maintain acceptable accuracy at like 60% sparsity - unfortunately model pruning isn’t much farther than the research phase. I tried quantization but the support wasn’t good enough and would require me to quant and dequant very frequently.</p>
<p>Are there any other techniques for speeding up my model inference time on CPU that I could try? Or is it better to just re-architect the entire thing to flow the data better with quantization in mind?</p>
How to concatenate two tensors in C++ API

FenixFromTheShadows: <p>Hello,</p>
<p>I am new to pytorch, and I want to use pytorch to train with a C++ simulation program in an automated alternating working flow, so I am learning to use PyTorch C++ API.</p>
<p>I went through some try-and-error and was able to get examples with the simplest type of neutral networks working. When I want to try some more complicated examples, I found myself needing to concatenate tensors in the forward function, and I was not able to find the way to do so, therefore that become my first question here.</p>
<p>Also, is there a good documentation about the interface, like what public methods/operators are available for torch::Tensor class, etc.? I couldn’t find many such info in the “Library API” section on the website (I wasn’t able to even find the torch::Tensor class there). Or maybe I am missing something?</p>
<p>Thanks,</p>

FenixFromTheShadows: <p>I seem to have figured it out.</p>
<p>torch::cat({x, y}) seem to work for me, not 100% sure about the requirements on the dimensionality. It looks like there is a version with another parameter to specify the dimension but it says I can only use -1 or 0, and I am not sure what that means. At least it works as expected when I just concatenate two 1D tensors together.</p>

Hartbook: <p>Yes it’s torch::cat.</p>
<p>You can indeed specify the dimension along which to concatenate. In this case -1 means the last one.<br>
So for 1D tensor you can pass -1 or 0 but it will have the same effect</p>

FenixFromTheShadows: <p>I see. Thanks for the clarification.</p>
How to apply len mask for multi head attention?

hadaev8: <p>So i made key_padding_mask and sent to MHA forward and got alignments like this<br>
                    <a href="https://i.imgur.com/hKeQCV1.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/hKeQCV1.png" width="598" height="500">
          </a>

<br>
How to mask top part?</p>
Iterate over all tensor permuation in speceific dim

Ameen_Ali: <p>hello,<br>
Given a tensor of shape Original = [A , B , C]<br>
I want to build a list of tensors (target) in the following way:<br>
the first element (a) = Original.<br>
the second element (b), b =  same as Original but the tensors are shifted by 1, meaning b[0 , : , :] = a[1 , : , : ]<br>
the third element ©, c = same data as Origial but the tensors are shifted by 2 (c[0 , : , :] = a[2 , : , :]<br>
this will results a list of size A since we are shifting the tensors each time by 1 until we get to the original tensor (original).</p>

KFrank: <p>Hi Ameen!</p>
<aside class="quote no-group" data-username="Ameen_Ali" data-post="1" data-topic="72166" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ameen_ali/48/15528_2.png" class="avatar"> Ameen_Ali:</div>
<blockquote>
<p>the first element (a) = Original.<br>
the second element (b), b =  same as Original but the tensors are shifted by 1, meaning b[0 , : , :] = a[1 , : , : ]</p>
</blockquote>
</aside>
<p>Does <a href="https://pytorch.org/docs/stable/torch.html#torch.roll" rel="noopener nofollow ugc">torch.roll()</a> do part of what you want?</p>
<p>Best.</p>
<p>K. Frank</p>
Pytorch use tensorboard for hyperparameter

bioinfo-dirty-jobs: <p>I use pytorch 1.3.1(cuda92py36hb0ba70e_0 ) for  some classification  models.<br>
I would like to record all the parameters using  tensorboard in particular I want to change  all the parameters and create a single  plot where I can  see all the loss curves.<br>
Is it possible? Any suggestion?  Any tutorial on this?</p>
<p>thanks so much for any help</p>
<pre><code class="lang-auto">writer.add_scalar(
                        f"{EXPERIMENT_DIR}/test_loss_epoch", loss_test_avg, epoch,
                    )
                    writer.flush()
</code></pre>
Finetuning FCN or deeplab with more classes

lycaenidae: <p>I want to do semantic segmentation with 23 classes. But in the FCN or Deeplab is trained with 20 classes.</p>
<p>I implemented via VGG16 then upsample the output but I wanted to directly use the pre-trained semantic segmentation models like FCN or DeepLab. I could not find any reference in the PyTorch documentation. Could you please help me?</p>
<p>Thanks</p>
Best way to save a model that is getting pruned along training?

gebrahimi: <p>I am trying to implement a neuron pruning algorithm. Along training, during certain steps, i prune one neuron and that changes the architecture of my model (I basically recreate a new smaller network based on the bigger network). I need to save the model after each pruning. I am looking for a descent way to save the model.<br>
The suggested way <code>torch.save(model.state_dict(), PATH)</code> does not work for me since to load the model later, i need to know the architecture and since i am changing the architecture constantly, i do not know the architecture for each model (unless there is a way to save the architecture separately which i do not know).<br>
Also, <code>torch.save(model., PATH)</code> works, but problem is that i am doing a lot of stuff during training and for whatever reason the saved model using this method is huge (15 gb for a small model that saving it using <code>torch.save(model.state_dict(), PATH)</code>takes only 90 mb).</p>
<p>Is there any descent way to handle this situation?</p>
Index and replace tensor dimension

Joan_Gibert: <p>I have and 3D tensor:<br>
<code>inp = torch.rand(400,600,3)</code></p>
<p>I want to replace one of the dimensions by a 2D tensor  like:<br>
<code>y = torch.zeros(x[:,:,1].shape)</code></p>
<p>I’m trying to index and replace like <code>inp[:,:,1] = y</code> but instead I of one of the 400x600 it seems to replace one column across the whole 3D tensor:</p>
<pre><code class="lang-auto">inp
tensor([[[0.9132, 0.0000, 0.5986],
         [0.2523, 0.0000, 0.1171],
         [0.5549, 0.0000, 0.2875],
         ...,
         [0.9777, 0.0000, 0.1527],
         [0.5669, 0.0000, 0.9031],
         [0.1303, 0.0000, 0.0252]],

        [[0.0073, 0.0000, 0.8185],
         [0.2140, 0.0000, 0.1950],
         [0.0666, 0.0000, 0.6404],
         ...,
         [0.3611, 0.0000, 0.1923],
         [0.2869, 0.0000, 0.5641],
         [0.8142, 0.0000, 0.1015]],

        [[0.6172, 0.0000, 0.8202],
         [0.0122, 0.0000, 0.5879],
         [0.2285, 0.0000, 0.0171],
         ...,
         [0.1347, 0.0000, 0.1469],
         [0.1260, 0.0000, 0.1539],
         [0.1876, 0.0000, 0.8285]],

        ...,

        [[0.2957, 0.0000, 0.1658],
         [0.6985, 0.0000, 0.1012],
         [0.7452, 0.0000, 0.9366],
         ...,
         [0.5743, 0.0000, 0.7877],
         [0.5458, 0.0000, 0.1958],
         [0.7182, 0.0000, 0.3096]],

        [[0.1060, 0.0000, 0.1589],
         [0.0132, 0.0000, 0.0222],
         [0.4626, 0.0000, 0.4589],
         ...,
         [0.8956, 0.0000, 0.6201],
         [0.2406, 0.0000, 0.9843],
         [0.0187, 0.0000, 0.4369]],

        [[0.2071, 0.0000, 0.7130],
         [0.7521, 0.0000, 0.7691],
         [0.6274, 0.0000, 0.2580],
         ...,
         [0.8197, 0.0000, 0.4061],
         [0.3698, 0.0000, 0.2892],
         [0.6189, 0.0000, 0.1528]]])
</code></pre>
<p>Why is this not throwing a dimension error? How should I change my code to get the desired output?<br>
Thanks!</p>
CNN dimension equation

eranb: <p>Someone can tell me what is the 1 in the Cnn arch dimension equation?</p>
<p>W2=(W1−F+2P)/S+1</p>
<p>What is 1?<br>
bias?</p>

tom: <p>No, it is just 1. The core of the equation is counting “how often does the weight filter (aka kernel) fit onto the input”. Without padding and with stride=1, you get <code>width_out = (width_in - filter_width) + 1</code>. Now you can see where it comes from: If you start with <code>width_in == filter_width</code>, you have 1 possible position, and for each additional pixel column, i.e. one larger <code>width_in</code>, you have an additional position. So the 1 is this anchoring of the counting.</p>
<p>Best regards</p>
<p>Thomas</p>
Custum loss and custom Module, Weights won't update

chloe: <p>Hi, I’m having a problem with my code. I want implement the loss function myself (in the code I present you this loss is greatly simplified). But I don’t understand the parameters and the loss are not update while the gradient is not null.<br>
Thanks,</p>
<pre><code class="lang-auto">class LOSS(nn.Module):

    def __init__(self,x):
        super(LOSS, self).__init__()
        self.para = nn.Parameter(torch.tensor(x, requires_grad=True))
    def forward(self):
        F=0
        m=1
        for time in range(1, len(self.para[:])):
                F += 1 / 2 * m*(self.para[time]-self.para[time-1])**2
        return F

x=torch.rand(50)

model = LOSS(x)

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

model.__init__(x)
optimizer.zero_grad()
model.zero_grad()
loss.backward()
optimizer.step()

</code></pre>

Oli: <p>I’m sorry Chloe, but there are a magnitude of weird things here. Your loss function should be separate from your model. It most often only consist of a function (not using nn.Parameter - thats the model) and returns a scalar.</p>
<p>Dont do zero_grad before your backward and step. Do the zero grad before sending inputs to the model (in your case <code>model = LOSS(x)</code>)</p>
<p>Perhaps start with a tutorial/prebuilt project to get the basics down and modify a single part of it  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Even brushing up on your Python seems like it would help</p>

chloe: <p>Thanks I’ll take a better look at the tutorail</p>
Custom layer with specific matrix operation

Lukman_Hakim: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de9e058a4b431375ab31a03335dc01b29a733467.png" alt="kernel" data-base62-sha1="vLmyBTqLx9ar40cXI46I51YRNXx" width="453" height="303"><br>
Hi,<br>
I try to make my own layer with fixed kernel. I prepare kernel manually like in figure.<br>
I want to find product from input and my kernel. But,  product operations are not usual, only on the black part. for white, ignored. I dont need make it trainable for this layer.</p>
<p>So, it’s possible use torch.nn.Conv2d?</p>
<p>Thanks.</p>

ziqipang: <p>I think you may directly set the conv_layer.weight.data as your own kernel, then fix this layer.</p>

Lukman_Hakim: <p>Thank you for your help, its working.</p>