akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Autograd reference lost during cost calculation

username: ajosemf
cooked: <p>Hello guys,</p>
<p>My cost function is very specific and needs a lot of external calculations, including using external libraries, to get the cost value. Because of this, it is impossible to keep such calculations under tensors. The problem with getting out of the tensor context is that I lose track of the autograd history. How could I take the scalar value produced by my cost function, encapsulate it in a tensor, and reconnect it to the autograd graph?</p>

username: tom
cooked: <p>You can do that by defining a <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" rel="nofollow noopener">custom autograd.Function</a>. This allows you to do anything you want in the forward. The caveat is that you have to give the derivative yourself in the backward (as PyTorch cannot be aware of it when you are not using tensors).</p>
<p>Best regards</p>
<p>Thomas</p>

username: ajosemf
cooked: <p>Hello Thomas, thank you very much for the reply. I will test here and as soon as possible share the result.</p>
Need a Help in executing the Project

username: Mark_Steve
cooked: <p>I am new to python like absolute new to coding too but for my college project i need to execute this project which is available in Github <a href="https://github.com/anujdutt9/Disease-Prediction-from-Symptoms" rel="nofollow noopener">https://github.com/anujdutt9/Disease-Prediction-from-Symptoms</a> . I can learn python but it takes time but I need to submit this project within two days and i dont know how to execute it i have installed anaconda and jupyter notebook but cant able to execute…</p>
<p>Please Help very urgent…</p>

username: ptrblck
cooked: <p>Since this seems to be a homework assignment, I think it would be inappropriate to solve it for you.</p>
Test Image for classification

username: Sam_Fst
cooked: <p>Hi guys,<br>
I am testing my model for the same image and the prediction output is going different every time I run the code.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5f55f1f8ec084e60a7b468445333b336cf86d101.png" alt="Capturar" data-base62-sha1="dBnuMkwm1pPwVbIkhN6xQuP2G1X" width="432" height="208"></p>
<p>model  = loader(‘balanca.pth’)</p>
<p>classe = (‘Balanca’,‘False’)</p>
<p>model.eval()</p>
<p>img = Image.open(“ImgTest/All/0n.png”)</p>
<p>x = transform(img)</p>
<p>x = x.unsqueeze(0)</p>
<p>output = model(x)</p>
<p>pred = torch.argmax(output,1)</p>
<p>print(‘Image predict as:’,classe[pred])</p>

username: Nikronic
cooked: <p>Hi,<br>
Did you set the random seed manually?</p>
<pre><code class="lang-python">
seed = 0
torch.manual_seed(seed)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

import numpy as np
np.random.seed(seed)
</code></pre>
Input output of Conv2d into a Conv3d layer to get a 5 dimensional output?

username: myidispg
cooked: <p>I am working on a project based on the OpenPose research paper that I read two weeks ago. In that, the model is supposed to give a 5-dimensional output. For example,  <code>torch.nn.conv2d()</code>  gives a 4-D output of the following shape:  <code>(Batch_size, n_channels, input_width, input_height)</code> . What I need is an output of the following shape:  <code>(Batch_size, n_channels, input_width, input_height, 2)</code> . Here  <code>2</code>  is a fixed number not subject to any changes. The  <code>2</code>  is there because each entry is a 2-dimensional vector hence for each channel in every pixel position, there are 2 values hence, the added dimension.</p>
<p>What will be the best way to do this? I thought about having 2 seperate branches for each of the vector values but the network is very deep and I would like to be as Computationally efficient as possible.</p>

username: JuanFMontesinos
cooked: <p>It’s not a matter of using 3d convolutions to get 1 more output. Those channels will have a meaning. Depending on their meaning you will need to make use of different architectures.</p>
Deadlock using DistributedDataParallel

username: 111110
cooked: <p>Hi, I’m facing a problem. When using <code>DistributedDataParallel</code> with NCCL, my training will meet a deadlock. According to the pytorch doc, I try to set the <code>set_start_method</code> to <code>spawn</code> and <code>forkserver</code>, but an error that <code>address already in use</code> occurs.</p>

username: Vyom_Kavishwar
cooked: <p>Faced a similar error - solved it by initializing the process group first, and then setting the model cuda device (as opposed to the other way around, which led to the same kind of deadlock you describe)</p>
If training quantification supports in pytorch now?

username: dingyongchao
cooked: <p>In tensorflow, there is a training quantification<a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize" rel="nofollow noopener">tensorflow_quantize</a><br>
So I wonder if anyone has any ideas to support training quantification in pytorch, or has samples?<br>
Now it is quite hard for me to add op(such as quant weight) or modify model struct(such as fold conv and norm op), So if anyone can help me?</p>

username: RicCu
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/18318" rel="nofollow noopener">Quantization is coming to PyTorch</a>. It won’t be long before it is fully operational, but you can already try it out on master or a nightly release.</p>
RNN Model (Many to one) that predict class question

username: aldeka12
cooked: <p>Hello, I have training input of this size (5000, 142, 30) where (Batch Size, Sequence Length, number of inputs) respectively. And so my output should be (5000, 3). My model should take 30 input numbers 142 times (time series)  and then give me one of class output. And this happen with every batch in 5000 batches.<br>
I am just interested is my Model correct for this task? After some tests I assuming it work correctly, but I am still in doubt. The part with star seems to me most questionable.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, input_number, ouput_number, hidden_number, layers_number):
        super().__init__()
        self.hidden_number = hidden_number
        self.rnn = nn.LSTM(input_number, hidden_number, layers_number, batch_first=True)
        self.fc1 = nn.Linear(hidden_number, ouput_number)
    def forward(self, data_in):
        hidden = None
        out, hidden = self.rnn(data_in, hidden)
        out = out[:, -1, : ]   *
        out = out.contiguous().view(-1, self.hidden_number)
        out = torch.sigmoid(self.fc1(out))
        return out, hidden
</code></pre>

username: aldeka12
cooked: <p>By the way, my input is changed to (5000(or more), 132, 30) but still this is not changing question.</p>
How to apply different kernel to different data in a batch

username: JaydenZoZ
cooked: <p>Hi,<br>
I want to learn different kernel for different input image. Then convolve on the feature map with the learned kernel. The feature map also comes from the input image.<br>
Suppose for the input, the batch size &gt; 1. Below is the pseudo code demo of what I tried.<br>
It turns out to be quite slow, sometimes even slower than the for loop.</p>
<pre><code class="lang-python"># kernel and feature
rpn_feat = net_1(input)
kernel_feat = net_2(input)
# size
rpn_feat.size = (batch_size, rpn_feat_dim, h, w)
nc_cls_out = 2 * anchor_num
kernel_feat.size = (batch_size, nc_cls_out, h_k, w_k)

# convolve feature with kernel
# OPTION 1: group convolution
kernel = kernel.view(nc_cls_out * batch_size, rpn_feat_dim, h_k, w_k)
cls_feat = torch.nn.functional.conv2d(rpn_feat, kernel, groups=batch_size)
cls_feat = cls_feat.view(batch_size, nc_cls_out, h_cls_feat, w_cls_feat)
# OPTION 2: for loop
cls_feats = list()
for idx_in_batch in range(batch_size):
  kernel = kernel_feat[idx_in_batch:idx_in_batch+1] \
  .view(nc_cls_out, rpn_feat_dim, h_k, w_k)
  feat_for_conv = rpn_feat[idx_in_batch_idx_in_batch+1]
  cls_feat = torch.nn.functional.conv2d(feat_for_conv, kernel)
  cls_feats.append(cls_feat)
cls_feat = torch.cat(cls_feats, 0)
</code></pre>
<p>Is that the right way doing this? Or is there any other way faster?<br>
Thanks in advance.</p>

username: JaydenZoZ
cooked: <p>Using pytorch 0.3.2 with cuda 8.0 and cudnn 7.0…</p>
How to use script annotation for dropout layer?

username: marchss
cooked: <p>To use script annotation mode to convert Pytorch model into C++, we replace <code>class MyModule(torch.nn.Module):</code> with <code>class MyModule(torch.jit.ScriptModule):</code>. But how shall we deal with below droptout layer?</p>
<pre><code class="lang-auto">class RNNDropout(nn.Dropout):
    """
    Dropout layer for the inputs of RNNs.

    Apply the same dropout mask to all the elements of the same sequence in
    a batch of sequences of size (batch, sequences_length, embedding_dim).
    """

    def forward(self, sequences_batch):
        """
        Apply dropout to the input batch of sequences.

        Args:
            sequences_batch: A batch of sequences of vectors that will serve
                as input to an RNN.
                Tensor of size (batch, sequences_length, emebdding_dim).

        Returns:
            A new tensor on which dropout has been applied.
        """
        ones = sequences_batch.data.new_ones(sequences_batch.shape[0],
                                             sequences_batch.shape[-1])
        dropout_mask = nn.functional.dropout(ones, self.p, self.training,
                                             inplace=False)
        return dropout_mask.unsqueeze(1) * sequences_batch
</code></pre>

username: tom
cooked: <p>I would define an RNNDropout class based on <code>ScriptModule</code> as well. <code>p</code> should be dealt with as a <a href="https://pytorch.org/docs/stable/jit.html#python-defined-constants" rel="nofollow noopener">python-defined constant</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: marchss
cooked: <p>Since we are not going to use RNNDropout layer in test/prediction  phase, does this mean we do not need to convert the RNNDropout into <code>ScriptModule</code> when converting pytorch into C++ used in test/prediction phase? Thus, we can use original <code>class RNNDropout(nn.Dropout):</code> in both training and testing phase?</p>

username: tom
cooked: <p>When tracing, this should work if you just don’t call it.<br>
For scripting, you might need a backup plan.</p>
Add several columns to a matrix without any loop

username: zahra
cooked: <p>Hi,</p>
<p>Suppose we have a tensor with dimension of [2, 1,  2, 10] and we want to add 3 columns between columns 3,4 and columns 5,6 and columns 8,9.<br>
Is it possible to add these columns to the tensor <strong>without any loop</strong>?<br>
If not, according to <a href="https://discuss.pytorch.org/t/how-can-i-insert-a-row-into-a-floattensor/18049/2">https://discuss.pytorch.org/t/how-can-i-insert-a-row-into-a-floattensor/18049/2</a> it needs 2 cat operations for adding each new column; is there any faster methods for doing it?</p>
<p>Thanks</p>

username: tom
cooked: <p>You can use indexed assignments:</p>
<pre><code class="lang-python">a = torch.ones(2, 1, 2, 10)
b =  = torch.full((2, 1, 2, 3), 2.0)

xa = torch.empty(2, 1, 2, 13)
xa[:,:,:,[0, 1, 2, 3, 4+1, 5+1, 6+2, 7+2, 8+3, 9+3]] = a
xa[:,:,:,[4, 7, 10]] = b
print(xa)
</code></pre>
<p>If your columns are a little larger (say 100 instead of 4 elements), a for loop might also not be too bad.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zahra
cooked: <p>Thanks a lot. It could help me.<br>
Best Regards</p>
How to handel the last batch during evaluation when using DataParallel

username: yassersouri
cooked: <p>Hi,</p>
<p>In the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel" rel="noopener nofollow ugc">DataParallel documentation</a> it is mentioned that:</p>
<blockquote>
<p>The batch size should be larger than the number of GPUs used.</p>
</blockquote>
<p>When using DataLoader, <strong>during training</strong> one can specify <code>drop_last=True</code> so that we can make sure that no batch has size smaller than the number of GPUs.</p>
<p>But when <strong>evaluating</strong> the model, you cannot simply drop the last batch. You should evaluate that batch as well (Although it might not make a lot of difference). Depending on the size of the Dataset, it is possible that the last batch will have size smaller than the number of GPUs.</p>
<p>How should one handle this situation in an elegant way?</p>
<p>In summary:</p>
<ul>
<li>I want to use DataParallel at evaluation time.</li>
<li>I want to evaluate all batches, including the last batch.</li>
<li>The last batch can have size smaller than number of GPUs.</li>
</ul>
<p>What should I do?</p>

username: pietern
cooked: <p>You can feed the last batch through a non-parallelized version of your model. Or you can extend the final batch with zeroes and ignore those outputs.</p>

username: yassersouri
cooked: <p>What I found is that, Although is it mentioned in the docs that the batch size should be larger than the number of GPUs, at least during evaluation, there is no error if the batch size is smaller than the number of GPUs.</p>
<p>So I guess the answer is to do nothing. Do not drop the last batch when evaluating. And it won’t cause an error.</p>
MPI_Gatherv and MPI_Scatterv ( Feature request/idea )

username: Andreas_Georgiou
cooked: <p>In the documentation of torch.distributed it is not made clear that all tensors in gather() need to be of the same size. After reading the code I noticed that in MPI c++ code the function used is MPI_Gather which imposes that restriction. I was thinking that maybe it would make more sense to use MPI_Gatherv instead so that tensors of variable size can be accepted. I would try to implement it myself but my C++ skills are not that good. If anyone is interested and wants to create a pull request, it would be great!. This is the file I am referring to: <a href="https://github.com/pytorch/pytorch/blob/eb76b7a564121c5fede749ad7d0a36f2b61a0a95/torch/lib/c10d/ProcessGroupMPI.cpp#L430" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/eb76b7a564121c5fede749ad7d0a36f2b61a0a95/torch/lib/c10d/ProcessGroupMPI.cpp#L430</a></p>
<p>At least if someone can provide a guide on how to make the change for MPI_Gatherv I can make the change for MPI_Scatterv myself.</p>
<p>For MPI_Gatherv we need to provide a list of integers instead of a single integer. This should be easy by calling numel() on each tensor in the gather_list. We also need to provide displacements(see here: <a href="https://www.mpich.org/static/docs/v3.1/www3/MPI_Gatherv.html" rel="nofollow noopener">https://www.mpich.org/static/docs/v3.1/www3/MPI_Gatherv.html</a>). We could set this to always be the previous displacement plus the last tensor’s numel() and by setting the first displacement to 0. This is if we want to modify the built-in gather. This way it accepts tensors of any size so it covers the case of equal sized tensors. Alternatively, we could create a new torch.distributed.gatherv()</p>
<p>Any help would be appreciated</p>

username: pietern
cooked: <p>This is a great idea. I agree having both gather and allgather for variable size is very useful. Also, seeing as the displacement is an implementation detail, the underlying C++ implementation can take care of sharing each process’ contribution, allocating the required memory, and returning the list of tensors.</p>
<p>I created <a href="https://github.com/pytorch/pytorch/issues/23299" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/23299</a> to track the feature.</p>
Manually specifying ip addresses and ports of all nodes for distributed data parallel

username: ruppesh
cooked: <p>I noticed that for distributed data parallel, you only need to specify the ip address and port of rank0 node, and then during initialization all nodes discover each other through rank0 node. But due to certain firewall restrictions, I want to manually specify the ip address and port of each node via which they should communicate for all reduce operations. Is there a way to do that? I am open to make changes in the pytorch source code.</p>

username: pietern
cooked: <p>There is no way to do this today. It also depends on which distributed backend you’re using whether this would be possible in the first place. If you’re using Gloo, it might be possible, but it’s quite a bit of work. If you’re using NCCL, it’s up to NVIDIA. If you’re using MPI, I don’t know.</p>
Shared file-system is a file?

username: urw7rs
cooked: <p><a href="https://pytorch.org/docs/stable/distributed.html#shared-file-system-initialization" rel="nofollow noopener">link to doc</a><br>
According to the doc page, this is how you initialize a shared file-system.</p>
<pre><code class="lang-auto">dist.init_process_group(backend, init_method='file:///mnt/nfs/sharedfile',
                        world_size=4, rank=args.rank)
</code></pre>
<p>And in the warning box above the code in the doc page, it says this code creates a file. Does this mean that I can’t use it as a directory?</p>

username: pietern
cooked: <p>No, the path has to be a file. If all your processes gracefully terminate the file will be removed. If one of your processes crashes, it may not be deleted and you’ll have to delete it yourself.</p>
Using pytorch with ml-agents (importError)

username: Linko994
cooked: <p>Hello,<br>
I’m trying to get pytorch to work with ml-agents (0.81).<br>
After installing latest version of pytorch (python 3.6 Cuda 9.0) it fails to import: numpy.core.multiarray failed to import</p>
<p>Ml-agents requires Numpy version to be &lt;=1.14.5. I tried pytorch with the newest version of numpy and it worked. How can I get it to work with 1.14.5?</p>
Weighted pixelwise for multiple classes Dice Loss

username: John1231983
cooked: <p>Hello all, I am using dice loss for multiple class (4 classes problem). I want to use weight for each class at each pixel level. So, my weight will have size of <code>BxCxHxW</code> (C=4) in my case. How can I use the weight to assign to dice loss? This is my current solution that multiple the weight with the input (network prediction)  after softmax</p>
<pre><code class="lang-auto">class SoftDiceLoss(nn.Module):
    def __init__(self, n_classes):
        super(SoftDiceLoss, self).__init__()
        self.one_hot_encoder = One_Hot(n_classes).forward
        self.n_classes = n_classes

    def forward(self, input, target, weight):
        smooth = 0.01
        batch_size = input.size(0)

        input = F.softmax(input, dim=1)
        input = input*weight
        input = input.view(batch_size, self.n_classes, -1)
        target = self.one_hot_encoder(target).contiguous().view(batch_size, self.n_classes, -1)

        inter = torch.sum(input * target, 2) + smooth
        union = torch.sum(input, 2) + torch.sum(target, 2) + smooth

        score = torch.sum(2.0 * inter / union)
        score = 1.0 - score / (float(batch_size) * float(self.n_classes))

        return score
</code></pre>
<p>And the second solution is that multiply the weight in the inter and union position</p>
<pre><code class="lang-auto">class SoftDiceLoss(nn.Module):
    def __init__(self, n_classes):
        super(SoftDiceLoss, self).__init__()
        self.one_hot_encoder = One_Hot(n_classes).forward
        self.n_classes = n_classes

    def forward(self, input, target, weight):
        smooth = 0.01
        batch_size = input.size(0)

        input = F.softmax(input, dim=1).view(batch_size, self.n_classes, -1)
        target = self.one_hot_encoder(target).contiguous().view(batch_size, self.n_classes, -1)
        weight = weight.view(batch_size, self.n_classes, -1)

        inter = torch.sum(input * target * weight, 2) + smooth
        union = torch.sum(input*weight, 2) + torch.sum(target*weight, 2) + smooth

        score = torch.sum(2.0 * inter / union)
        score = 1.0 - score / (float(batch_size) * float(self.n_classes))

        return score
</code></pre>
<p><strong>Which one is correct?</strong></p>

username: XiaoAHeng
cooked: <p>Can you share your <code>One_Hot(n_classes).forward</code>?</p>
Pytorch Convert Model into C++

username: marchss
cooked: <p>Hi folks,<br>
I have a model contains ModuleDict which requires key to be string type. But channel_id is int type. Thus, inside forward() function, I use str(channel_id) to convert from int type to string type and then use it as ModuleDict’s key. This is fine.</p>
<p>Problem arises when converting the model into C++ using trace, the accepted argument MUST be a Tensor which does NOT support string data type. Thus, I use channel_id as a Tensor of int64. However, this channel_id is a Tensor with single int64 value, while in the forward function of model class, the channel_id argument is a single int64 value instead of a Tensor.</p>
<p>Of course, we can modify the model class’s forward function to accept channel_id as tensor instead of a single int64 value just to be able to trace the model, but it needs a lot of changes.</p>
<p>Is there a better way to keep the existing model code, and able to trace it?</p>
<pre><code class="lang-auto">channel_id = torch.ones(1, dtype=torch.int64)
traced_script_module = torch.jit.trace(model, (premise, premise_length, hypotheses, hypotheses_length, channel_id))
</code></pre>

username: ptrblck
cooked: <p>Hi <a class="mention" href="/u/marchss">@marchss</a>,</p>
<p>please don’t tag certain people, as this discourages others to answer, while the tagged persons also might not even be an expert on this topic. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: marchss
cooked: <p>Sorry about that. I often see great answers from you folks and thus did that. Will avoid this.</p>

username: yf225
cooked: <p><a class="mention" href="/u/marchss">@marchss</a> As this problem is more related to what JIT accepts as input to a traced module, could you post it in <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a> so that we can discuss potential solutions with more people?</p>

username: tom
cooked: <p>The problem is that the traced model will treat non-tensor inputs as constants. This is not really changeable.<br>
So you would want to mix script and trace as discussed in the other thread.</p>
<p>Best regards</p>
<p>Thomas</p>
How to create a few gates that can be optimized by the network

username: Runde_Yang
cooked: <p>so I am designing a network that has input in the format as [x1,x2,x3,…,xn], each vector xi has length k. Now, before these features are fed into my network, I want to add a layer of gates to select from these features. So I will have gates G1,…Gn, each gate is simply a 1D variable. How should I implement it that the gates can be optimized by gradient descend? Thanks.</p>
If I use scripting to convert my model, how can I call other methods in C++?

username: kevinhaoliu
cooked: <p>If I want to call other methods in my model(not <strong>forward</strong>), what should I do? I try to use <em><strong>torch.jit.script_method</strong></em> to decorate it but I can’t call it in C++.</p>

username: kevinhaoliu
cooked: <pre><code class="lang-auto">class Net(torch.jit.ScriptModule):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv2d(1, 1, 3)

    @torch.jit.script_method
    def forward(self, x):
        return self.conv(x)

    @torch.jit.script_method
    def weighted_kernel_sum(self, weight):
        return weight * self.conv.weigh

n = Net()
n.save("model.pt")
</code></pre>
<p>What I want to say is that if I use this way to convert my model, can I call the weighted_kernel_sum method in the C++ code?</p>

username: tom
cooked: <p>Does <a href="https://pytorch.org/cppdocs/api/structtorch_1_1jit_1_1script_1_1_module.html#_CPPv4N5torch3jit6script6Module10run_methodERKNSt6stringEDpRR5Types" rel="nofollow noopener"><code>run_method</code></a> work for you?</p>
<p>Best regards</p>
<p>Thomas</p>

username: kevinhaoliu
cooked: <p>Thank you for your reply. I don’t quite understand that, could you please give me an example in detail?</p>
<p>Best regards<br>
Kevin</p>

username: tom
cooked: <p>You follow the <a href="https://pytorch.org/tutorials/advanced/cpp_export.html#step-4-executing-the-script-module-in-c" rel="nofollow noopener">tutorial and at step 4</a> you replace</p>
<pre><code class="lang-auto">at::Tensor output = module-&gt;forward(inputs).toTensor();
</code></pre>
<p>with</p>
<pre><code class="lang-auto">torch::Tensor output = module-&gt;run_method("weighted_kernel_sum", input).toTensor();
</code></pre>
<p>where input is your input tensor.</p>
<p>Best regards</p>
<p>Thomas</p>

username: kevinhaoliu
cooked: <p>It works. Thank you for your help!</p>
<p>Best regards</p>
<p>Kevin</p>
Gradient statistics

username: agadetsky
cooked: <p>How can I simply get gradient statistics (for example minibatch variance) during training to monitor it?</p>

username: tom
cooked: <p>For intra-batch “per sample contributions”, no, there isn’t a way in general, though <a href="https://discuss.pytorch.org/t/efficient-computation-of-per-sample-examples/18587">there are some tricks</a>  <a href="https://github.com/pytorch/pytorch/issues/7786" rel="nofollow noopener">you could try</a>.<br>
For inter-batch statistics, you can do that similar to what the Adam optimizer (and other similar optimizers like LAMB) do, which essentially is one of the <a href="https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford's_online_algorithm" rel="nofollow noopener">Welford-style online algorithms for the variance</a> with more or less sophistication around subtracting the mean.</p>
<p>Best regards</p>
<p>Thomas</p>
Using PhP to deploy pytorch and python packages?

username: nile649
cooked: <p>Hi, I am trying to execute python script from PHP but it is doesn’t works from pytorch? Any hint or idea what could be the reason?<br>
Thanks</p>

username: tom
cooked: <p>I think that’s really a PHP question more than a PyTorch one, so this forum might not be the ideal venue…<br>
That said, you might consider implementing a <a href="https://lwn.net/Articles/792882/" rel="nofollow noopener">flask or similar server</a> for your PyTorch model and connect to that from PHP.</p>
<p>Best regards</p>
<p>Thomas</p>
To(device) gives an error when used inside transforms.Compose

username: Deeply
cooked: <p>I am trying to move the image to the GPU during transforms.Compose but I am getting the error</p>
<pre><code class="lang-auto">transforms.Lambda(lambda x: x.to(device)),
RuntimeError: CUDA error: initialization error
</code></pre>
<p>I need to do this as I am using one of the trained models during Compose.transformation; something like:</p>
<pre><code class="lang-auto">device = torch.device('cuda') 
transform = transforms.Compose([
        transforms.Resize([256, 256]),       
        transforms.ToTensor(),         
        transforms.Lambda(lambda x: x.to(device)),
        transforms.Lambda(lambda x: my_model(x.unsqueeze(0)).squeeze(0) )
        ])
</code></pre>
<p>Any ideas of how can I resolve this issue?</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>If you are using multiple workers in your <code>DataLoader</code>, multiple initializations of the CUDA context might yield this error.<br>
Try to use the <code>spawn</code> method for multiprocessing as described <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#sharing-cuda-tensors" rel="nofollow noopener">here</a> or alternatively use a single worker (<code>num_workers=0</code>).</p>
Tensor after .clone().detach() continue to evolve

username: barthelemymp
cooked: <p>Hello everyone,</p>
<p>In order to train and visualize some optimization I’m working on I adopt the following method:<br>
-I define a model,<br>
-save the value of 2 parameters,<br>
-train the model,<br>
-put the requires_grad to flase for all my parameters except the 2 I have saved,<br>
-then for different optimization mode I put again the two parameters to their initial value and retrain the model.</p>
<p>Here is the following code:</p>
<pre><code class="lang-auto">init = model.QC_optvar2.clone().detach()  # I save the init value
print('init', init)
# remaining hyperparameters
loss_fn = torch.nn.CrossEntropyLoss()
n_epoch = 1

#getting to an optimal point by training all the parameters
train_losses, valid_losses, valid_acc = train(model, train_dataset, test_dataset, loss_fn, n_epoch)


weights_list = []
loss_evol_list = []
shots = [0, 3, 256, 1024]
for sh in shots:
    print('shots', sh, 'init', init)
    print(model.QC_optvar2.data)
    model.QC_optvar2.data = init
    print(model.QC_optvar2.data)
    model.QC_optvar1.requires_grad_(False)
    model.QE_optvar.requires_grad_(False)
    print(sh, 'shots')
    weights, loss_evol = trainDSGD(model, train_dataset, test_dataset, loss_fn, shots = sh, n_epoch=1) 
    weights_list +=[weights]
    loss_evol_list += [loss_evol]
</code></pre>
<p>I thought that detach was going to detach the tensor from the graph.<br>
What is happening is a bit confusing. The init vector does not change during the first train,<br>
Then when I get in the for loops (except for the first instance) it starts to take the same value of the current <code>model.QC_optvar2.</code>, while I wanted it to keep its inital value in order to reinject it again and again. ( the line model.QC_optvar2.data = init becomes useless because the two tensor are already the same)</p>
<p>I tried .clone(), .clone().detach(), .data etc… and it never works…<br>
The only solution which works is to convert the tensor to numpy and convert it again in tensor when I re inject it.</p>
<p>I’m a bit puzzled by that I would like to understand a bit better what is behind,</p>
<p>Thank you in advance</p>
<p>b</p>

username: tom
cooked: <p>I would suggest to leave the code as is but:</p>
<ol>
<li>
<a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/#what-about-data" rel="nofollow noopener">Don’t use <code>.data</code></a>, I think it helps clearly understanding what’s going on if you don’t.</li>
<li>I think you want <code>with torch.no_grad(): model.QC_optvar2.copy_(init)</code> instead of the assignment, otherwise you are integrating init into the (new) graph.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>
Async/Parallel Evaluation of model

username: LeanderK
cooked: <p>I have a machine with 10 GPUs and my utilization is quite bad (&lt;50% spent training) because of the computation of the metrics I have to track each epoch (not everything can be easily parallelized across 10 GPUs and they are quite involved). I have already reduced the resolution for many metrics. They are very important to me and I can’t reduce them further. For some metrics, I generate matplotlib-plots, which is also quite costly.</p>
<p>I am thinking that switching from a purely sequential (train -&gt; eval -&gt; train -&gt; eval …) to a parallel setup would greatly speed up my model (So for example 8-9 GPUs are constantly occupied with training and 1-2 with evaluating my metrics). Is this possible with pytorch? The only examples I’ve found are about parallelizing the training, but this is already working.</p>
<p>I would have to clone my model and push it to my evaluation-workers.</p>

username: ptrblck
cooked: <p>If you push a copy of the current model to GPU9 and execute the evaluation method, it should run asynchronously while your other GPUs are training.<br>
Note that your data loading might become a bottleneck, if not multiple <code>DataLoaders</code> are trying to read from your drive.</p>

username: LeanderK
cooked: <p>Ok, I’ll test it.</p>
<blockquote>
<p>If you push a copy of the current model to GPU9 and execute the evaluation method, it should run asynchronously while your other GPUs are training.</p>
</blockquote>
<p>Could this be implemented using torch.multiprocessing or what can you recommend?</p>

username: ptrblck
cooked: <p>CUDA calls should run asynchronously by default.<br>
Let me know, if you encounter an issue.</p>
Torch einsum backend

username: zwang-asapp
cooked: <p>Hi all,<br>
What is the backend for torch einsum on GPU? Does it use a compiler like TC or TVM?<br>
Thanks!</p>

username: tom
cooked: <p>No, it’s torch.bmm, currently.<br>
I do have a branch implementing reductions via TensorIterator, but didn’t benchmark it (probably not terribly for cuda for small problems, less so for )<br>
Hacking a mini-JIT pass &amp; custom op for keops or somesuch would probably be a nice quick project (and more generally a fuser to keops would be a nice somewhat larger project). For TVM you would have to figure out how you want to optimize (i.e. benchmark on first invoke, expect that the next calls will have similar shapes, …).</p>
<p>Best regards</p>
<p>Thomas</p>
Why doesn't Pytorch favour batch dimension first?

username: pinocchio
cooked: <p>Is there a reason why Pytorch doesn’t favour batch size first when organizing the data? Is there a reason for it to not be like that?</p>
<p>Is that the opposite of tensorflow or something? Perhaps there is a good reason idk…</p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<p>In pytorch, we do have batch first scenarios. For example if you process any datasets from <code>torchvision</code>. They are processed batch_first. You can change it. I added one small example below.</p>
<pre><code class="lang-auto">testset = torchvision.datasets.CIFAR10(root=data_path, train=False, download=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)
</code></pre>
<p>If you process the above testloader, you will get something as follows.</p>
<pre><code class="lang-auto">for i, data in enumerate(testloader, 0):
    inputs, labels = data  # inputs and labels are tensors with size 4x3x32x32 and 4
</code></pre>
<p>Here the batch is the first dimension, then the channels. At the end, we have height and width.</p>
<p>One more thing batch first and batch last are two different settings. The desired representation can be achieved using <code>torch.transpose</code>.</p>
<p>Thanks</p>
Why does the ChatBot tutorial reverse sort the pairs in the batch?

username: Brando_Miranda
cooked: <p>I was going through the chatbot tutorial and saw the following:</p>
<pre><code class="lang-auto">    pair_batch.sort(key=lambda x: len(x[0].split(" ")), reverse=True)
</code></pre>
<p>why does the tutorial (reverse) sort? Is there a good reason?</p>
<hr>
<p>code:</p>
<pre><code class="lang-auto"># Returns all items for a given batch of pairs
def batch2TrainData(voc, pair_batch):
    '''
    '''
    pair_batch.sort(key=lambda x: len(x[0].split(" ")), reverse=True)
    #print( [len(pair[0].split(" ")) for pair in pair_batch] )
    #st()
    input_batch, output_batch = [], []
    # seperate the pairs (for the current batch) in X and Y values
    for pair in pair_batch:
        input_batch.append(pair[0])
        output_batch.append(pair[1])
    # returns a matrix of (max_length,batch_size) corresponding to the feature vectors X
    inp, lengths = inputVar(input_batch, voc)
    output, mask, max_target_len = outputVar(output_batch, voc)
    return inp, lengths, output, mask, max_target_len
</code></pre>

username: Brando_Miranda
cooked: <p>I just found out that the packing documentation says:</p>
<blockquote>
<p>For unsorted sequences, use enforce_sorted = False. If enforce_sorted is True, the sequences should be sorted by length in a decreasing order, i.e. input[:,0] should be the longest sequence, and input[:,B-1] the shortest one. enforce_sorted = True is only necessary for ONNX export.</p>
</blockquote>
<p>so now the question is, if I should be sorting myself or if I should let pytorch do the sorting internally? Which one is the best or recommended or standard one?</p>
<hr>
<p>docs:</p>
<p><a href="https://pytorch.org/docs/stable/nn.html?highlight=pack_padded_sequence#torch.nn.utils.rnn.pack_padded_sequence" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.html?highlight=pack_padded_sequence#torch.nn.utils.rnn.pack_padded_sequence</a></p>

username: Brando_Miranda
cooked: <p>so I decided to have the sorting happen in the dataprocessing with dataset/dataloader cuz these allow <code>num_workers&gt;0</code> which probably allows for parallelization and speed ups because we can process many things at the same time.</p>
<p>Obviously I’d need to benchmark to check what is the fastest but that doesn’t sound like its worth my time. So I’d just go with my gut what seems a fine solution.</p>
<p>Feel free to contribute a suggestion!</p>

username: Brando_Miranda
cooked: <p>cross posted: <a href="https://qr.ae/TWnbg9" rel="nofollow noopener">https://qr.ae/TWnbg9</a></p>

username: Brando_Miranda
cooked: <p>the only thing I’m not sure is if to use the sorting as suggested by the ChatBot tutorial (i.e. doing it in Python or if there is some Pytorch way to do it that is better/more efficient).</p>
<p>For now I will sort in Python directly and not use Pytorch.</p>
Using accessors on CUDA

username: mikael
cooked: <p>Hi there,<br>
Following the tutorial about writing C++/CUDA extensions I’m writing my own one. In this extension I use accessors. I’ve implemented the c++ version and everything work fine on CPU, but when I try to run my code (from the python level, as shown in the tutorial) on CUDA, I get a segfault raised by the attempt to access the elements of the accessors. Are accessors usable with CUDA tensors ?<br>
Thank you.</p>

username: yf225
cooked: <p>Did you use <code>packed_accessor</code> for CUDA tensors? Could you share your C++/CUDA extension code?</p>
Is there any way that I can use multi-thread in libtorch?

username: cfanyyx
cooked: <p>I know that we can use mkl-dnn in pytorch and get speed up when setting the multi-thread in the inference code. But when I trace a model into ScriptModule and use it in c++. How can I get the same speed up like pytorch do?</p>

username: Michael_Suo
cooked: <p>What version of pytorch are you using? It should just work on 1.0.1 or the nightlies</p>

username: cfanyyx
cooked: <p>Actually, I want to get speed up on libtorch with c++ codes. Is this related to pytorch version? Are the ScriptModules that different versions of pytorch produce matter? The ScriptModule I used in c++ is produced with pytorch1.1.0a0+bd19dd4.</p>

username: yf225
cooked: <p>If you set the environment variables following <a href="https://github.com/mingfeima/convnet-benchmarks/blob/master/pytorch/run.sh#L16-L25" rel="nofollow noopener">https://github.com/mingfeima/convnet-benchmarks/blob/master/pytorch/run.sh#L16-L25</a>, it should have similar speedup as PyTorch.</p>
Loading a pre-trained model Customized

username: Sam_Fst
cooked: <p>Hi guys, I have a problem when I load my model:</p>
<p>This is the code when I trained my model:</p>
<p>model = models.densenet121(pretrained = True)</p>
<p>train_on_gpu = torch.cuda.is_available()</p>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
<p>from collections import OrderedDict<br>
classifier = nn.Sequential(OrderedDict([<br>
(‘c1’,nn.Linear(1024,740)),<br>
(‘relu’, nn.ReLU()),<br>
(‘c2’,nn.Linear(740,400)),<br>
(‘relu1’,nn.ReLU()),<br>
(‘c3’, nn.Linear(400,2)),<br>
(‘output’, nn.LogSoftmax(dim=1))<br>
]))</p>
<p>model.classifier = classifier</p>
<p>…<br>
The train code<br>
…</p>
<p>torch.save( model.state_dict(),‘classificador-1.pth’)</p>
<h3>When I load the model:</h3>
<p>state_dict = (checkpoint[‘state_dict’])<br>
model =  models.densenet121(pretrained = False)<br>
model.load_state_dict(new_state_dict,strict=False)<br>
model.fc = classifier<br>
for parameter in model.parameters():<br>
parameter.requires_grad = False<br>
model.eval()</p>
<p>But I have wrong predicted class when I test the model,<br>
because I just wanted two exits but I’m getting more.</p>

username: Tahir
cooked: <p>It seems you can load the saved model, but what is not clear where is the variable classifier initialized before you use when loading:</p>
<pre><code class="lang-auto">state_dict = (checkpoint[‘state_dict’])
model = models.densenet121(pretrained = False)
model.load_state_dict(new_state_dict,strict=False)
model.fc = **classifier**
for parameter in model.parameters():
parameter.requires_grad = False
model.eval()
</code></pre>
<p>If the linear layer is not initialized/loaded with trained weights then the predictions will not be accurate.</p>
Is there a difference in coding data processing stuff in transforms vs collate_fn?

username: pinocchio
cooked: <p>Is there any reason to choose one vs the other? I noticed that everything can be done in the <code>collate_fn</code> which I thought was more conceptually nice since everything was in one place (specially for my complicated data processing custom class).</p>
<p>Or is there a real performance reason (i.e. runs faster) for choosing <code>transforms</code> vs <code>collate_fn</code>? What are all the advantages and disadvantages?</p>
Projection layer

username: Diaa_elsayed_ziada
cooked: <p>i’m working on action recognition task, the paper that i’m working from it say’s that after feed the images to vgg16 add to the projection layer as in the image<img src="https://discuss.pytorch.org/uploads/default/original/2X/3/352f28dce0a2969ae3d3ff51e9d228a924d83cd6.png" alt="pdf" data-base62-sha1="7Aulo0hrOL8DTsVahFENebfrzr8" width="387" height="313"><br>
how can i implement prejection layer in pytorch and feed it to Bi-Directional Recurrent Neural Network?</p>
Boolean matrix indexing

username: anordertoreclaim
cooked: <p>I ran into a non-numpy-like behavior while trying to use a 2d boolean tensor as a mask. This code reproduces it:</p>
<blockquote>
<p>a = torch.zeros((3, 3, 2))<br>
mask = torch.zeros((3, 3), dtype=bool)<br>
mask[(1, 2, 2), (0, 0, 1)] = 1<br>
print(mask)<br>
a[mask, 0] = 1</p>
</blockquote>
<p>What I get is:</p>
<blockquote>
<p>IndexError: The shape of the mask [3, 3] at index 1does not match the shape of the indexed tensor [3, 2] at index 1</p>
</blockquote>
<p>Basically, I cannot select which elements in my third dimension should be masked. The numpy version works fine on this one.</p>
TypeError: can't pickle objects

username: pytorcher
cooked: <p>Hello,  getting the “TypeError: can’t pickle X objects” error.  As has been posted many times before this is in fact because I am running torch.save(net, PATH) and not torch.save(net.state_dict(), PATH).  However, my understanding is that state_dict() just to fill in values so you have to initialize a network with the same architecture to load a state_dict() to it.  My application incorporates modifying the architecture after initialization so if that is how it works I can not use state_dict for my saving.  Everything was working in pytorch 0.2, I have now updated and working with 0.4.  I was wondering if there was any advice on how to track down what exactly is the problem with my object that can’t be pickled.  or possibly something that changed between versions that might cause this to no longer work.</p>
<p>If it helps my object is  class myObject(torch.autograd.Function).  It has a forward and backward and also fills in and keeps track of a few values.</p>
<p>Thanks!</p>

username: HOWARD201314
cooked: <p>I have the same problem, do you have solved the problem? if so, please give me a help！ thanks！</p>

username: pytorcher
cooked: <p>I believe I eventually decided pytorch didn’t like the concept of pickling torch.autograd.functions.  I ended up making a function that saved the variables from the class, deleted them all then pickled and upon load created new class objects and filled in the values.</p>
How to debug a network which is crashing after 2000 steps?

username: thomasryck
cooked: <p>I am trying to use NasNet to do feature extraction for an OCR use case.</p>
<p>However after like 2300 steps I have the following error:</p>
<blockquote>
<p>…<br>
File “C:\Users\tryck\Documents\OCR_from_scratch\model_custom\NasNet_layers.py”, line 159, in forward<br>
comb_iter_1 = comb_iter_1_left + comb_iter_1_right<br>
RuntimeError: The size of tensor a (16) must match the size of tensor b (9) at non-singleton dimension 0</p>
</blockquote>
<p>My code is deterministic I set a seed and deactivate non deterministic cuda calculation.<br>
The error is always the same but do not always appear at the same step.</p>
<p>What should I do to be able to solve this issue ?</p>

username: Oli
cooked: <p>Hi, there are several approaches that I can think of.</p>
<ul>
<li>Assign each data item with a unique identifier (e.g. image path) and log that when the error occurs. Perhaps this datapoint is a bit funky?</li>
<li>Built a testcase for your dataloader where you loop through all the data and assert that it is the correct shape, datatype etc…</li>
<li>Make your code saveable/resumable so that you don’t have to redo all the 2000 steps before debugging, but maybe just 100.</li>
</ul>
<p>In your case it seems like <code>comb_iter_1_left</code> has the batch size of 16 whilst the right one has 9. As to why is impossible to say for me but perhaps your dataset isn’t evenly dividable with 16 so the last batch is just 9?</p>
<p>Edit: What steps are you taking to make sure that your program is deterministic?</p>

username: thomasryck
cooked: <p>To assert that the code is deterministic I set the following values, to arbitrary values:</p>
<blockquote>
<pre><code>random.seed(opt.manualSeed)
np.random.seed(opt.manualSeed)
torch.manual_seed(opt.manualSeed)
torch.cuda.manual_seed(opt.manualSeed)

cudnn.benchmark = True
cudnn.deterministic = True
</code></pre>
</blockquote>

username: Oli
cooked: <p>Ok, that looks good. But change the benchmark to False in accordance to <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">the docs</a>. Any thoughts to my reply?</p>

username: thomasryck
cooked: <p>I have checked and my dataset is not divisible by 16. So I modified it in order to. I changed the cudnn.benchmark value to false as given by the linked doc.</p>
<p>Thanks for your help</p>
Mask tensor without losing gradients

username: razvanc92
cooked: <p>Hello, I know that this might be trivial but I’ve been structurally with it for quite some time. I’m having a tensor of shape [batch, num_nodes, num_nodes] and I would like to mask it so that on the second dimension I replace the biggest number with 1 and the rest with 0. So far I’ve come to this:  mask = (mask == mask.max(dim=2, keepdim=True)[0]) it works but i’m loosing the gradients. Any help it’s welcome.</p>
Accelerate this function `batch-wise index_select `

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">
import torch
def batch_wise_index_select(input, dim, index):
     assert input.size(0) == index.size(0)
     out = []
     for i in range(input.size(0)):
         out.append(torch.index_select(input[i], dim=dim-1, index=index[i].view(-1)))
     return torch.stack(out, dim=0)

a = (torch.rand(2,2,5)*10).int()
b = (torch.rand(2,3)*3).long()
out = batch_wise_index_select(a, dim=2, index=b)

print(a)
print(b)
print(out)
</code></pre>
<p>Output</p>
<pre><code class="lang-bash">tensor([[[6, 4, 7, 4, 0],
         [2, 3, 1, 4, 5]],

        [[6, 2, 6, 6, 3],
         [7, 4, 7, 2, 9]]], dtype=torch.int32)
tensor([[1, 1, 2],
        [1, 1, 0]])
tensor([[[4, 4, 7],
         [3, 3, 1]],

        [[2, 2, 6],
         [4, 4, 7]]], dtype=torch.int32)
</code></pre>
<p>Then how to accelerate batch_wise_index_select ?</p>