akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch Global SummerHack 2019 is Open

username: smth
cooked: <p>The Global PyTorch Summer Hackathon is here!</p>
<p>Calling developers and researchers to hack together packages, demos and applications on top of PyTorch and submit online.</p>
<p>We’ll help you reach the community, amplify your vision and have $60K in prizes!</p>
<p>Read more at <a href="https://pytorch.devpost.com" rel="nofollow noopener">https://pytorch.devpost.com</a></p>

username: smth
action_code: banner.disabled
Backward not implemented for cholesky_solve

username: calincru
cooked: <p>Why is that?</p>
<p>If it’s not just a feature lacking temporarily, what alternatives are out there?  I assume <code>torch.solve</code> works in most cases but that doesn’t leverage the positive definitiveness of a matrix:</p>
<pre><code class="lang-python">l = torch.cholesky(x)
x_inv_u = torch.cholesky_solve(u, l)
</code></pre>
<p>vs.</p>
<pre><code class="lang-python">x_inv_u = torch.solve(u, x)
</code></pre>

username: Megaloman
cooked: <p>Have you found any work around for this? I’m working with GPs and for numerical stability reasons the consensus is you need the cholesky_solve …which isn’t supported for .backward calls as you’ve pointed out.</p>

username: calincru
cooked: <p>No, I don’t currently need the derivatives, but I might in the future.</p>
<p>It seems that this is an <a href="https://github.com/pytorch/pytorch/issues/4669" rel="nofollow noopener">open issue</a>, perhaps it will get prioritized if more people request it.</p>
Having memory overflow issues

username: nlam
cooked: <p>Hi i have been trying to build a simple CNN for the MNIST dataset, but about after 4 epochs, I have a memory overflow issues. Any suggestions on how I can fix this? Thank you!</p>
<p>notebook here : <a href="https://colab.research.google.com/drive/16RFi_fERIyHMpiNXlGJb01faUsJWrkot" rel="nofollow noopener">https://colab.research.google.com/drive/16RFi_fERIyHMpiNXlGJb01faUsJWrkot</a></p>

username: Mazhar_Shaikh
cooked: <p>Hi Nishanth,<br>
The notebook doesn’t have the code for the training loop. Could you upload that as well? Since, the training loop is the one causing the error.</p>

username: nlam
cooked: <p>sorry, I accidentally removed it. It should be there now thought!</p>
<p>thanks for any help</p>

username: Mazhar_Shaikh
cooked: <p>Hi Nishanth,<br>
I had a look at the training code. The training loop is correct. The validation had this line:</p>
<pre><code class="lang-auto">loss += criterion(outputs, classes)
</code></pre>
<p>This was causing pytorch to retain the memory of all the previous graphs executed for validation.<br>
Changing it to the below line will solve your problem.</p>
<pre><code class="lang-auto">loss += criterion(outputs, classes).item()
</code></pre>

username: nlam
cooked: <p>Yep! that solved my problem. Thank you!</p>
Building pytorch and making it appear in pip?

username: Data-drone
cooked: <p>I built pytorch 1.2.0 from source and it works but it doesn’t register with pip so when I try to install torchvision with pip, torchvision tries to download it’s own pytorch?</p>
<p>How can I get it registered with pip?</p>
<p>I have checked and calling python and pip from the commandline both trigger the one in my conda environment</p>

username: ptrblck
cooked: <p>Try to install torchvision via <code>pip install --no-deps</code>.<br>
Alternatively, you could of course build torchvision from source.</p>
What happens when two batches are passed through, does that create two graphs?

username: John_Deterious
cooked: <p>Say I passed b1 through my net like this:</p>
<pre><code class="lang-auto">y1 = my_net(b1)
</code></pre>
<p>Then, I immediately pass another batch to get a different output, like this:</p>
<pre><code class="lang-auto">y2=my_net(b2)
</code></pre>
<p>Does that create two graphs?<br>
I’m computing something based on previous two results and then doing backprop</p>
<pre><code class="lang-auto">some_function(y1, y2).backward()
</code></pre>
<p>Is this ‘<em><strong>safe</strong></em>’?</p>

username: Mazhar_Shaikh
cooked: <p>Hi John_Deterious,<br>
This is safe. As long as y1 and y2 are retained in the memory, the graphs remain. It will be destroyed if you delete the tensors. Calling backward also destroys the graph, unless called with the “retain_graph=True” argument.</p>
Distributed Bayesian Deep Learning using Variational Inference

username: nionjo
cooked: <p>Hi,<br>
I’m about to embark on a project doing distributed variational inference on (Bayesian) deep learning. What are the existing resources for Bayesian deep learning in pytorch – or more broadly in python at the moment (I’m only aware of ZhuSuan, <a href="https://github.com/thu-ml/zhusuan" rel="nofollow noopener">https://github.com/thu-ml/zhusuan</a>, which is designed on top of tensorflow)?</p>

username: ptrblck
cooked: <p>Have a look at <a href="https://botorch.org/" rel="nofollow noopener">BoTorch</a>.</p>

username: nionjo
cooked: <p>Thanks for the reply.<br>
It looks to me that BoTorch primarily supports Bayesian Optimization on (frequentist) neural networks. Is it going to be expanded to include Bayesian inference for deep models?</p>
Issues with chatbot NLP tutorial of attention implement error

username: scutych
cooked: <p>in the chatbot tutorial(<a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html</a>)， i found the attention implement may be error!<br>
# Forward through unidirectional GRU<br>
rnn_output, hidden = self.gru(embedded, last_hidden)<br>
# Calculate attention weights from the current GRU output<br>
attn_weights = self.attn(rnn_output, encoder_outputs)<br>
# Multiply attention weights to encoder outputs to get new “weighted sum” context vector<br>
context = attn_weights.bmm(encoder_outputs.transpose(0, 1))<br>
# Concatenate weighted context vector and GRU output using Luong eq. 5<br>
rnn_output = rnn_output.squeeze(0)<br>
because the  encoder input sample are different length, so, the encoder output  count is different，<br>
but when compute the attention weight, there may should do some mask on the attention weights, but<br>
the attention implemention in the  tutorial was not masked</p>
I have a trouble because come from difference version

username: mshmoon
cooked: <p>I had train yolov3 use pytorch 0.4,but yesterday I  update my pytorch version to 1.0. So a error come out, this is error message.<br>
cannot perform reduction function max on tensor with no elements because the operation does not have an identity.</p>
<p>I think this code is the bad code for version 1.0.<br>
loss_cls = (1 / nB) * self.ce_loss(pred_cls[mask], torch.argmax(tcls[mask], 1)).</p>
<p>Is there any one help me</p>

username: mshmoon
cooked: <p>I am sorry .It is my fault. Please do not reply this topic.</p>

username: gvskalyan
cooked: <p>How did you solve it? <a class="mention" href="/u/mshmoon">@mshmoon</a></p>
Input to tensorboard add_graph is not a Tensor

username: MANSUM
cooked: <p>What should I do if the input to my model not only consists of Tensors but also python lists?</p>
<p>That is,</p>
<pre><code>model = Net()
add_graph(model, (Tensor1, Tensor2, list))
</code></pre>
<p>In this case, I get errors. How can I pass a list as an input to <code>add_graph</code>?</p>
Mini-batch size and scaling

username: dmw64
cooked: <p>Hi guys,</p>
<p>I have simple and basic question regarding the to calculation of gradients: Let’s say I have mini-batch of N elements all leading to N individual losses which I want accumulated to a single loss.</p>
<p>My question: Does autograd average the gradients coming from a mini-batch (corresponding to the batch-size) or does it just sums all gradients up?</p>
<p>It seems that there is no averaging but pure summing. But I’d like to get a confirmation.</p>
<p>Here is what I mean: Let us consider a mini-batch of N equal elements. If I sum all the N individual losses and let autograd calculate gradients and do an optimizer-step (I tried this with pure SDG). Then the gradient step will scale with N (so N=2 leads to a step twice as big as N=1). Hence, to get something mini-batch size independent, I should sure the average of the individual losses, right?</p>
<p>Thanks a lot!</p>

username: ptrblck
cooked: <p>The loss is averaged by default using a criterion, e.g. <code>nn.CrossEntropyLoss</code> or <code>nn.MSELoss</code>. This would make the gradients independent regarding the batch size. You could change this behavior by setting the <code>reduction</code> argument to e.g. <code>'sum'</code>.</p>

username: dmw64
cooked: <p>Great - that is what I needed to know. Thank you!</p>
RuntimeError: start (0) + length (0) exceeds dimension size (0)

username: sparshgupta8130
cooked: <p>I am running a model for QA ranking. This model uses 2 LSTMs and a series of attention mechanisms. It works for 1 dataset, but when I run this model for another dataset, I get a weird looking error:</p>
<pre><code class="lang-auto">RuntimeError: start (0) + length (0) exceeds dimension size (0). (narrow at /opt/conda/conda-bld/pytorch_1532579805626/work/aten/src/ATen/native/TensorShape.cpp:157)
</code></pre>
<p>I don’t understand what the error is and hence I am unable to debug it. Plus, I don’t understand why does it work for one dataset, and doesn’t for the other dataset.</p>
<p>I am using 4 GPUs and <code>torch.nn.parallel.data_parallel()</code> in my code for data parallelism. It should not be any out of memory issue since only 50% of each of the 4 GPUs is utilized with the minimum batch size necessary for training.</p>
<p>Can someone please tell me how to solve this issue?</p>

username: SimonW
cooked: <p>Probably due to your tensor shape, there is a narrow on empty tensor some where. You can try use a debugger and/or print out the tensor shapes to see where things got wrong.</p>
<p>Additionally, we have added proper empty tensor support, and will be available in next release. So the error message will be nicer. But that probably isn’t the root of your problem.</p>

username: sparshgupta8130
cooked: <p>I am printing the tensor shapes, they are all as expected. The code works fine when I run it on 1 GPU. But with multiple GPUs, I get this error. Plus, this error happens with the last batch of training. It works fine with all the previous batches. I am printing the gradient of embeddings layer, it shows that there is a non zero gradient as well.</p>

username: Vimos
cooked: <p>I can get the same error on a single GPU. In my case, I generated an empty batch record.</p>

username: Jiahang_WANG
cooked: <p>Hi, I got the same error. Do you know how to resolve this now?</p>

username: ptrblck
cooked: <p>Check that your tensors are not empty, as this error is apparently thrown if <code>narrow</code> encounters an empty tensor.</p>
Why embedding or rnn/lstm can not handle variable length sequence?

username: Mainul
cooked: <p>Pytorch embedding or lstm (I don’t know about other dnn libraries) can not handle variable-length sequence by default. I am seeing various hacks to handle variable length. But my question is, why this is the case? I mean, sequences almost never the same size/length and rnn/lstm should loop through until the end of a sequence. If so, why it will be sensitive to the various length of the sequences of a minibatch? Pytorch Embedding is a look-up table. To me, I don’t see any reason to be sensitive to variable length. Should not be the ideal case is, I can give a minibatch of sentences with a variable number of words? Like the following:</p>
<pre><code class="lang-auto">word_embedding = nn.Embedding(17,5)
#each vector is a sentence
word_embeds=word_embedding(torch.tensor([[1,2,3,4,5],[4,5,6,7]]))
</code></pre>

username: ptrblck
cooked: <p>Your example won’t work, as you cannot create a tensor using your inputs of different lengths.<br>
Internally each tensor holds the data as a blob storing internal attributes such as the stride and shape of the tensor. This makes is possible e.g. to apply batched operations.</p>
<p>As you can see in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding" rel="nofollow noopener">docs</a> of <code>nn.Embedding</code>, this layer will take  LongTensors of arbitrary shape as its input.</p>
Raise an error "Cuda that does not match the version used to compile Pytorch binaries" when install apex

username: 4e3c81574af4c3e6560b
cooked: <pre><code class="lang-auto">Compiling cuda extensions with
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2019 NVIDIA Corporation
    Built on Fri_Feb__8_19:08:17_PST_2019
    Cuda compilation tools, release 10.1, V10.1.105
    from /usr/local/cuda/bin

    Traceback (most recent call last):
      File "&lt;string&gt;", line 1, in &lt;module&gt;
      File "/tmp/pip-req-build-hreb_nq5/setup.py", line 64, in &lt;module&gt;
        check_cuda_torch_binary_vs_bare_metal(torch.utils.cpp_extension.CUDA_HOME)
      File "/tmp/pip-req-build-hreb_nq5/setup.py", line 54, in check_cuda_torch_binary_vs_bare_metal
        "https://github.com/NVIDIA/apex/pull/323#discussion_r287021798.  "
    RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries.  Pytorch binaries were compiled with Cuda 10.0.130.
</code></pre>
<p>But only find pytorch with cuda10.0 in <a href="https://pytorch.org/" rel="nofollow noopener">official website</a></p>
<ul>
<li>How can I get pytorch with cuda10.1?</li>
</ul>

username: dpernes
cooked: <p>You can compile it from source.</p>

username: ptrblck
cooked: <p>If you are using CUDA10.1 locally, you could try to remove the additional minor version check in <a href="https://github.com/NVIDIA/apex/blob/4a8c4ac088b6f84a10569ee89db3a938b48922b4/setup.py#L49" rel="nofollow noopener">this line of code</a>. It’s usually not recommended, but you can give it a shot.</p>
<p>If you encounter any issues, you would have to install the matching CUDA version locally (10.0 in your case).</p>
How to update parameters of one group only in an optimizer?

username: John_Deterious
cooked: <p>I have an optimizer, I added three groups to it. I computed some loss, now I want to update only the first group of parameters. How can I do that?</p>
<p>PS: in tensorflow, I explicitly ask for the gradients of the loss with respect to those parameters only. But here in Pyotrch, I just do loss.backward() and I everything is included. What is the equivalent operation in Pytorch?</p>
Model.eval() effect on layers contained in nn.Sequential()

username: rle1323
cooked: <p>Does setting model to model.eval() prior to validation still deactivate appropriate functions (Dropout, batch normalization) if the functions are contained in a nn.Sequential() layer? Like so:</p>
<p>self.classifier = nn.Sequential(<br>
nn.Dropout(p=0.4, inplace=True),<br>
nn.Linear(in_features=8<em>8</em>8, out_features=64),<br>
nn.ReLU(),<br>
nn.Linear(in_features=64, out_features=num_classes),<br>
)</p>

username: ptrblck
cooked: <p>Yes, as the the <code>eval</code> will be called on each child module.<br>
Do you see any weird behavior?</p>
Forward hook does not work with __call__ method

username: Sourena_Yadegari
cooked: <p><code>.register_forward_hook</code> for a subclass of nn.Module does not work when <code>__call__</code> is used instead of <code>forward</code>.</p>
<p>This code works:</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()

    def forward(self, x):
        return x

def print_forward(self, x_in, x_out):
    print('forward hook')

eye = Identity()
eye.register_forward_hook(print_forward)
x = eye(torch.rand(10)) # prints "forward hook"
</code></pre>
<p>and this one doesn’t (just change <code>forward</code> to <code>__call__</code>)</p>
<pre><code class="lang-auto"># this doesn't
class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()

    def __call__(self, x):
        return x

def print_forward(self, x_in, x_out):
    print('forward hook')

eye = Identity()
eye.register_forward_hook(print_forward)
x = eye(torch.rand(10)) # no print here
</code></pre>
<p>Is this a design decision? It would be nice to have a warning message in that case.</p>

username: ptrblck
cooked: <p>It is a design decision, since the <code>__call__</code> method is registering the hooks and calls <code>forward</code> as seen <a href="https://github.com/pytorch/pytorch/blob/211bafc2ea236484ed60b2e0faeede330e6f3238/torch/nn/modules/module.py#L537" rel="nofollow noopener">here</a>.</p>
<p>Since you’ve created your own <code>__call__</code> method, this behavior is disabled.</p>
Does the combination of Conv1d and Lstm(or GRU) make any sense?

username: zikun_liu
cooked: <p>Hi, I’m new here and it’s the first time to use the LSTM network in reinforcement learning. The state which is also the input of the neural network is a matrix with size (64, 10000).  The batchsize is 1, sequence length is 10000. So it is obviously a long and huge input. My goal is to get a (64,1) continuous output as the action. I use three Conv1d layers before LSTM to reduce the LSTM input to (64, 64). But the training is very slow and unstable, with a lot of fluctuation in the reward curve.  Any advice?</p>
MSE Loss collapses DCGAN model?

username: Kaitav_Mehta
cooked: <p>I am using same model as stated in <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a>, DCGAN pytorch implementation. But instead of using BCE loss I am comparing  already defined   different  loss functions. When I run the code with MSE loss it collapses the model.</p>
<p>Please can anyone tell me the reason for it?</p>

username: Kushaj
cooked: <p>MSE Loss simply tells the squared difference between two images in your case. Suppose there is some idea point, which has the minimum MSE Loss. As SGD would try to move towards this point when using MSE Loss, you would get samples resembling the ideal point.</p>
<p>Now the ideal point can be anything. It can be a random matrix.</p>

username: Kaitav_Mehta
cooked: <p>The main reason was I didn’t remove the Sigmoid function at the end of the discriminator. Which resulted in failure of DCGAN + MSE loss function.</p>
PyTorch - Loaded model not giving me the same results as it did before saving it

username: unido
cooked: <p>I’ve trained a model that is to be used for text generation. I’m still kind of new to saving and loading models for inference so, for experience’s sake I coded two versions. One where I train the model, save it, and directly generate text, and another where I load the saved model and generate the text from that point.</p>
<p>The first version generates text as I wanted it to, however the text generated from the loaded model is just gibberish. I noticed that the softmax results returned from the loaded model are all near equal as well…</p>
<p>This is how I save the model:</p>
<pre><code class="lang-auto">model_name = 'model.pth'
checkpoint = {'n_hidden': net.n_hidden,
              'n_layers': net.n_layers,
              'state_dict': net.state_dict(),
              'tokens': net.chars}

with open(model_name, 'wb') as f:
    torch.save(checkpoint, f)
</code></pre>
<p>And this is how it is loaded:</p>
<pre><code class="lang-auto">model_path = 'model.pth'
model = charRNN(chars, n_hidden, n_layers, lr, dropout)
checkpoint = torch.load(model_path)
model.load_state_dict(checkpoint['state_dict'])
model.eval()
</code></pre>
How does one check how large a Tensor is in Mega Bytes?

username: pinocchio
cooked: <p>I want to predict how large a data set will be before I create it.</p>
<p>Is it possible to get those estimates from Pytorch internally?</p>
<p>e.g. if I had a model or a cifar10 image, how can I check in byes how big it is?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Basically, The vital parameter here is the size of your current dataset. For instance, let say you have 20GB images, and you have about 16GB RAM or a little more. Even if you have a small model, it is still impossible to load whole dataset into RAM. But for small datasets like CIFAR-10, you can do some calculation here.</p>
<p>You can do something like this : <code>batch_size*channel_size*height*width = 28*3*32*32 = 86016</code> and let say we retain them in 64 bit numbers, then every input batch is about 688128 Bytes = <code>688kb</code>. You can calculate number of parameters in your model too, because they all are matrices like the batches. Finally, you need some more memory for operation that needs more space such as all non-inplace operations. The autograd itself needs memory which are completely depends on your model and its parameters.</p>
<p>Finally, If you have any problem in loading whole dataset into memory, You can just use <code>DataLoader</code> and <code>Dataset</code> class to use lazy loading which means dataloader will load a batch every time and reads it from memory and when the model is processing the inputs (let assume on GPU), the dataloader will prepare the next batch.<br>
You may have a little IO overload here (which only happens if loading data takes more time than processing in by the model), but the whole idea is great and actually, scientist who are working with large datasets like ImageNet (14 million) or Places365 (2 million), use lazy loading.</p>
<p>By the way, I used <code>DataLoader</code> and it works great. Fast and reliable.</p>
<p>Good luck</p>
Ploting ROC curve for multiclass classification

username: mchowdhu
cooked: <p>I am trying to plot ROC curve for multi class classification.I would like to plot multiple lines in a single graph for each class.But I am getting error<br>
<code>Error Traceback (most recent call last): File "/home/ex/Downloads/HW/DATAfile.py", line 385, in &lt;module&gt; plot_roc(actuals, class_probabilities,n_class) File "/home/ex/Downloads/HW/DATAfile.py", line 230, in plot_roc fpr[i], tpr[i], _ = roc_curve(actuals[:, i], probabilities[:, i]) TypeError: list indices must be integers or slices, not tuple</code><br>
I could not understand how I will get rid of this error.<br>
Here is my code that I used</p>
<pre><code class="lang-auto">def test_class_probabilities(model, test_loader, n_class):
    model.eval()
    actuals = []
    probabilities = []
    with torch.no_grad():
        for sample in test_loader:
            labels = Variable(sample['grade'])
            inputs = Variable(sample['image'])

            outputs = net(inputs).squeeze()

            prediction = outputs.argmax(dim=1, keepdim=True)
            actuals.extend(labels.view_as(prediction) == n_class)
            probabilities.extend(np.exp(outputs[:, n_class]))
    return actuals,probabilities                     #[i.item() for i in actuals], [i.item() for i in probabilities]



def plot_roc( actuals,  probabilities, n_classes):
    """
    compute ROC curve and ROC area for each class in each fold

    """

    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    for i in range(n_classes):
        fpr[i], tpr[i], _ = roc_curve(actuals[:, i], probabilities[:, i])
        roc_auc[i] = auc(fpr[i], tpr[i])

    # plt.figure(figsize=(6,6))
    for i in range(n_classes):
        plt.plot(fpr[i], tpr[i], label='ROC curve of class {0} (area = {1:0.2f})'
                                       ''.format(i, roc_auc[i]))  # roc_auc_score

    plt.plot([0, 1], [0, 1], 'k--')
    # plt.grid()
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic to multi-class')
    plt.legend(loc="lower right")
    # plt.tight_layout()
    plt.show()
</code></pre>

username: nrsyed
cooked: <p><code>actuals</code> is a list, but you’re trying to index into it with two values (<code>:, i</code>). Python lists are not arrays and can’t be indexed into with a comma-separated list of indices.</p>
<p>Replace <code>actuals[:, i]</code> with <code>actuals[i]</code> and <code>probabilities[:, i]</code> with <code>probabilities[i]</code>.</p>
Why can't one pass data through a torch ReLU module directly?

username: pinocchio
cooked: <p>check my minimal example:</p>
<pre><code class="lang-auto">import torch
​
x = torch.randn(4,2)
y = torch.nn.ReLU(x)
print(x)
print(y)
tensor([[-0.5674, -2.1012],
        [-0.4920, -0.3471],
        [-0.5981,  0.8400],
        [ 1.7876, -0.1611]])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-1-be685ec3279f&gt; in &lt;module&gt;
      4 y = torch.nn.ReLU(x)
      5 print(x)
----&gt; 6 print(y)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __repr__(self)
   1034         # We treat the extra repr like the sub-module, one item per line
   1035         extra_lines = []
-&gt; 1036         extra_repr = self.extra_repr()
   1037         # empty string will be split into list ['']
   1038         if extra_repr:

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/activation.py in extra_repr(self)
    100 
    101     def extra_repr(self):
--&gt; 102         inplace_str = 'inplace' if self.inplace else ''
    103         return inplace_str
    104 

RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>
<p>is this expected?</p>
<hr>
<p>related posts:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/52946920/bool-value-of-tensor-with-more-than-one-value-is-ambiguous-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/52946920/bool-value-of-tensor-with-more-than-one-value-is-ambiguous-in-pytorch</a></li>
<li><a href="https://discuss.pytorch.org/t/runtimeerror-bool-value-of-variable-objects-containing-non-empty-torch-longtensor-is-ambiguous/7310" class="inline-onebox">RuntimeError: bool value of Variable objects containing non-empty torch.LongTensor is ambiguous</a></li>
<li><a href="https://discuss.pytorch.org/t/runtimeerror-bool-value-of-tensor-with-more-than-one-value-is-ambiguous/43800" class="inline-onebox">RuntimeError: bool value of Tensor with more than one value is ambiguous</a></li>
<li><a href="https://discuss.pytorch.org/t/why-cant-one-pass-data-through-a-torch-relu-module-directly">https://discuss.pytorch.org/t/why-cant-one-pass-data-through-a-torch-relu-module-directly</a></li>
</ul>

username: SimonW
cooked: <p>It is a module, not a function. Use <code>torch.relu</code> or <code>torch.nn.functional.relu</code> or <code>tensor.relu()</code> for functions.</p>

username: pinocchio
cooked: <p>seems <code>a = torch.ReLU()</code> then <code>a(x)</code> works.</p>
Generated Hessian is not symmetric

username: nima_rafiee
cooked: <p>Hi</p>
<p>I’m using the following code to generate the Hessian of my loss function according to variable <code>x</code><br>
the problem is that the generated Hessian is not symmetric which must be. Could anybody help if there might be some numerical instability with torch.autograd?<br>
p.s: my model is an official PyTorch Resnet model.</p>
<pre><code class="lang-auto">
        out = model(x)  
        optimizer.zero_grad()
        loss = -criterion_nr(out.squeeze(), torch.sigmoid(out.squeeze()))
        loss = torch.mean(loss, dim=[1, 2])
        first_drv = torch.zeros(batch_size, x_dim).to(device)
        hessian = torch.zeros(batch_size, x_dim, x_dim).to(device)  
        for n in range(batch_size):
            first_drv[n] = torch.autograd.grad(loss[n], x,
                                                     create_graph=True, retain_graph=True)[0][n]
            for i in range(x_dim):
                exp_hessian[n][i] = torch.autograd.grad(first_drv[n][i], x,
                                                        create_graph=True, retain_graph=True)[0][n]


        print(hessian[0])
       &gt; a = torch.tensor([[ 857.2029,  196.4826],
                  [ 196.4857, 1563.9629]])
</code></pre>
<p><code>196.4826</code> and  <code>196.4857</code> are different in the last digit and it makes the rest of the computation wrong.</p>

username: KFrank
cooked: <p>Hello Nima!</p>
<aside class="quote no-group quote-modified" data-username="nima_rafiee" data-post="1" data-topic="52978" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nima_rafiee/48/16289_2.png" class="avatar"> nima_rafiee:</div>
<blockquote>
<p>Hi</p>
<p>I’m using the following code to generate the Hessian of my loss function according to variable <code>x</code><br>
the problem is that the generated Hessian is not symmetric which must be.<br>
…</p>
<pre><code class="lang-auto">        ...
        print(hessian[0])
       &gt; a = torch.tensor([[ 857.2029,  196.4826],
                  [ 196.4857, 1563.9629]])
</code></pre>
<p><code>196.4826</code> and  <code>196.4857</code> are different in the last digit and it makes the rest of the computation wrong.</p>
</blockquote>
</aside>
<p>Your two numbers that are supposed to be equal differ in the<br>
sixth significant digit.  This is consistent with the seven-digit<br>
round-off error of single-precision (32-bit) floating-point numbers<br>
accumulating when a multi-step computation is done.</p>
<p>If you need your Hessian matrix to be exactly symmetric (e.g.,<br>
to satisfy the pre-conditions of some subsequent processing),<br>
you can forcibly symmetrize it (replace each pair of matching<br>
off-diagonal elements with their average).  If you need (for some<br>
reason) your unsymmetrized Hessian to be symmetric to better<br>
than six decimal digits, you should preform your calculations<br>
in double precision (<code>dtype = torch.float64</code>).</p>
<p>Good luck.</p>
<p>K. Frank</p>
Change weights data during training

username: rpfeynman
cooked: <p>Hello everyone! I’m trying to train a convolutional network and I want the convolutional kernels of the last <code>Conv2d</code> layer to have an unit norm.</p>
<p>The layer is stored in the variable <code>self.end</code></p>
<p>I have added this method to the model class (note that this layer has only one channel as output, so the weights size is  <code>[1,in_channels,H,W]</code> )</p>
<pre data-code-wrap="python"><code class="lang-python">def make_convolutions_unit_norm(self):
        with torch.no_grad():
            W = self.end.weight.data.clone()
            num_channels = W.shape[1]
            for i in range(num_channels):
                W[0,i] = W[0,i]/W[0,i].norm()
            self.end.weight.data = W
        return
</code></pre>
<p>The hope is to do so, after the backpropagation step, without messing up the autograd. However, this for some reason doesn’t work, as I get the following error during backpropagation after calling <code>model.make_convolutions_unit_norm()</code> :</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 1, 2048, 96]], which is output 0 of CudnnConvolutionBackward, is at version 4; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>Any idea about how I can achieve this?</p>
Matplotlib install problem

username: falahfakhri
cooked: <p>Hi all this is my first post in here, and I’m a new user of python, I need your help please,</p>
<p>I’m facing up problem while trying to install matplotlib , I have py34 the reason behind that is to configure it with SNAPPY, in SNAP, coming back to my demand, would you please to help me find the wheel in order to install matplotlib in windows 10 , 64, py34,</p>
<p>I’m getting this error, (Command python setup.py egg_info failed with error code 1 in C:\Users\FALAHF~1\AppData\Local\Temp\pip_build_FALAH FAKHRI\matplotlib<br>
Storing debug log for failure in C:\Users\FALAH FAKHRI\pip\pip.log)</p>
<p>I also tired up to find the right wheel, to run  C:/Python34/Scripts/pip install matplotlib-2.2.4-cp27-cp27m-win_amd64.whl</p>
<p>But as you see I didn’t find any compatible wheel, from this website: <a href="https://pypi.org/project/matplotlib/#files" rel="nofollow noopener">https://pypi.org/project/matplotlib/#files</a></p>
<p>Or, from this one as well, <a href="https://www.lfd.uci.edu/~gohlke/pythonlibs/" rel="nofollow noopener">https://www.lfd.uci.edu/~gohlke/pythonlibs/</a></p>
<p>How could I solve this problem and install matplotlib? or any other packages</p>
<p>many thanks in advance,</p>

username: Mazhar_Shaikh
cooked: <p>Hi falahfakhri,<br>
Please post this to the <a href="https://github.com/matplotlib/matplotlib/issues" rel="nofollow noopener">matplotlib issue tracker</a>. This is a forum intended for pytorch usage. You may find better support for this issue there.<br>
Hope this helps!<br>
Regards</p>

username: falahfakhri
cooked: <p>finally I found the compatible wheel from here <a href="https://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.5.1/" rel="nofollow noopener">https://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.5.1/</a></p>
<p>for python34<br>
and successfully installed matplotlib using the following way,</p>
<p>$ C:/Python34/Scripts/pip install matplotlib.whl</p>

username: falahfakhri
cooked: <p>Thanks a lot for your response and comments, I already solved it,</p>
Fastest way to ultilize custom loss (supported 4D tensor) for 5D tensor?

username: John1231983
cooked: <p>Hello all, I have a custom loss that only worked for 4D tensors such as NxBxHxW. However, I want to use the loss for my data that is 5D tensor such as NxBxDxHxW. I am wondering which way is fastest way to use the loss for my input, and how to use it? I hear that we can use the view() function. Thanks<br>
This is the example use the loss for 4D tensor</p>
<pre><code class="lang-auto">s = myloss()
a = torch.randint(0, 255, size=(20, 3, 256, 256), dtype=torch.float32).cuda() / 255.
b = a * 0.5
a.requires_grad = True
b.requires_grad = True
loss = s(a, b)
loss.backward()
</code></pre>

username: Mazhar_Shaikh
cooked: <p>Hi John1231983,<br>
Could you post the code that defines your loss function?</p>

username: John1231983
cooked: <p>Of course, no! It is my custom loss function</p>

username: Mazhar_Shaikh
cooked: <p>Hi John1231983,<br>
Apologies. My intention was to help you optimize the loss function and generalize it to 5d tensors.<br>
In general, if you have a 5d tensor [N, C, H, W, D], you should utilize the information available due the fifth dimension.<br>
You could of course reshape your tensor to [N*C, H, W, D] using .view(N * C, H, W, D). However, it may give rise to unwanted effects.<br>
For example, in applying a weighted cross entropy loss for medical mri scans for brain tumor segmentation, per patient weight and per slice weight could give very different losses.<br>
Hope this helps!</p>
For loops inside forward definition in nn.Module class for GRU cells

username: pablosmig
cooked: <p>Hi! I’m implementing a class with nn.Module, GRUcells, normalisation and dropout.  I am mostly wondering if the way I implemented the GRUCells forward pass is correct and autograd would take care of properly transmitting the gradients. I used a for loop taking every time step in the input then passed that time step through the GRU cell and then used the hidden output and the next time step to continue until the sequence finishes.</p>
<p>Here is a snippet for clarity.<br>
Many thanks!</p>
<pre><code>class myGRU(nn.Module):

    def __init__(self, input_size, hidden_size, t_samps, dropout=0.0, bias=True):

        super(myGRU, self).__init__()

        self.GRUcell = torch.nn.GRUCell(input_size, hidden_size, bias=bias)

        # LayerNorm
        self.Norm = nn.LocalResponseNorm(input_size)

        # Dropout
        self.Dropout = nn.Dropout(p=dropout)

        # Others
        self.hids = hidden_size

    def forward(self, x, h0=0):

        device = getDevice()

        # Initialise output sequence
        out_seq = torch.zeros(x.shape[0], self.hids, x.shape[-1]).to(device)

        # Dropout units
        x = self.Dropout(x)

        # Normalise
        x = self.Norm(x)

        # x [batch_size, input_size, length]
        for i in range(x.shape[-1]): # for all elements in sequence
             if i == 0:
                 h_out = self.GRUcell(x[:,:,i].to(device)) # h_out [B, hidden_size]
             else:
                 h_out = self.GRUcell(x[:,:,i].to(device), h_out.to(device))

             if h_out.requires_grad:
                  h_out.register_hook(lambda grad: grad.clamp(-10.0,10.0))

             out_seq[:,:,i] = h_out # Store in initialised memory. Is the graph properly defined?

        return out_seq.to(device)</code></pre>
Is there a way to use TFRecords in pytorch?

username: Rohit_Kumar_Singh
cooked: <p>Hi,</p>
<p>I am relatively new to this so this question might be stupid, but I wanted to use wikisum dataset (<a href="https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/data_generators/wikisum" rel="nofollow noopener">https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/data_generators/wikisum</a>) on a pytorch model. Is there any way to use it.</p>
How do I make the input and output the same size?

username: ysleer
cooked: <p>The size of y1, y2, y3 is different so ‘torch.cat’ is not used.<br>
input_shape = [5, 256, 4, 64, 64]<br>
y1_shape = [5, 128, 4, 64, 64]<br>
y2_shape = [5, 128, 2, 62, 62]<br>
y3_shape = [5, 128, 1, 62, 62]<br>
The code is shown below.</p>
<pre><code class="lang-auto">class _test(nn.Sequential):
    def __init__(self, num_input_features):
        super(_test, self).__init__()

        self.b1 = nn.Sequential(OrderedDict([
            ('norm', nn.BatchNorm3d(num_input_features)),
            ('relu', nn.ReLU(inplace=True)),
            ('conv', nn.Conv3d(num_input_features, 128,
                                          kernel_size=(1, 1, 1), stride=1, bias=False))
        ]))

        self.b2 = nn.Sequential(OrderedDict([
            ('norm', nn.BatchNorm3d(num_input_features)),
            ('relu', nn.ReLU(inplace=True)),
            ('conv', nn.Conv3d(num_input_features, 128,
                                          kernel_size=(3, 3, 3), stride=1, bias=False))
        ]))

        self.b3 = nn.Sequential(OrderedDict([
            ('norm', nn.BatchNorm3d(num_input_features)),
            ('relu', nn.ReLU(inplace=True)),
            ('conv', nn.Conv3d(num_input_features, 128,
                                          kernel_size=(4, 3, 3), stride=1, bias=False))
        ]))

    def forward(self, x):
        y1 = self.b1(x)
        y2 = self.b2(x)
        y3 = self.b3(x)
        return torch.cat([y1, y2, y3], 1)
</code></pre>
<p>Please let me know if there is a way.</p>

username: ptrblck
cooked: <p>You could add <code>padding=1</code> to the conv layers in <code>b2</code> and <code>b3</code>, which would keep the spatial size.</p>
<p>EDIT: For <code>b3</code> you would need to pad the output in dim2, as you won’t get the same output shape for a kernel_size of 4 and padding of 1.</p>

username: ysleer
cooked: <p>Is there a way to pad only certain channels like dim2?</p>

username: spanev
cooked: <p>Hi,</p>
<p>You can zero pad b3 on dim2 by adding:</p>
<pre><code class="lang-auto">y3 = F.pad(y3, pad=(0,0,0,0,0,1), mode='constant', value=0)
</code></pre>
<p><a href="https://pytorch.org/docs/stable/nn.functional.html#pad" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.functional.html#pad</a></p>
How to replace specific values in a Tensor

username: MadmanNero
cooked: <p>Is there a convenient way to replace all occurrences of 0 in an Tensor with 1?</p>
<p>I’d like to normalize my data but the std deviation of some features may happen to be 0 and since we’re dividing by it I’d like to avoid division by zero errors by replacing those zeros.</p>

username: tom
cooked: <p>where to replace with 1, clamp for bounding from below. Note that very small numbers might be inconvenient, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: MadmanNero
cooked: <p>Thanks for your reply. Just looked up clamp function, but it’s not exactly what I need. Specifying min=0 would not remove zeros if I get it right. Picking some value bigger 0 for min would result in legitimate small values being ‘clamped’ as well…</p>
<p>Some way to iterate over a Tensor’s values and overwriting them if needed would be just as fine for the time being. I’m new to PyTorch/libTorch and I’m having quite some difficulties finding stuff in the documentation and getting things up and running.</p>
<p>Any hints would be welcome.</p>

username: tom
cooked: <p>Torch.where is what you should then use.<br>
That said, those legitimate small values can be mean to you.<br>
In general, it is a good idea to know what you’d do in Python (PyTorch or numpy), which most often translates to Libtorch I  a  straightforward way.</p>
<p>Best regards</p>
<p>Thomas</p>

username: MadmanNero
cooked: <p>Thanks for the hint on torch::where(), I’ll have a look at that. I have a running model in python but data preprocessing is done in pandas/numpy there which I believe is not easily available in C++.</p>

username: MadmanNero
cooked: <p>Hmmm… looking at it I find</p>
<p><code>Tensor where(const Tensor &amp; condition, const Tensor &amp; other) const;</code></p>
<p>Python documentation is rather pythonic and seemingly not applicable to C++ if I am not mistaken. C++ documentation is not to be found…</p>
<p>How am I to provide a condition like ‘value is equal to 0’ to this function in a Tensor?</p>

username: MadmanNero
cooked: <p>I decided to use an accessor now, iterating over the tensor elements and comparing &amp; setting where necessary. May not be elegant but solves the problem at hand for now. Code is like:</p>
<pre><code class="lang-auto">auto a = t.accessor&lt;float, 1&gt;();
		
for(int i=0; i&lt;a.size(0); i++)
	if(a[i] == 0.0)
		a[i] = 1.0;
</code></pre>
<p>Thanks for your help! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> )</p>
GPU memory estimation

username: Pirate-Roberts
cooked: <p>Hi everyone,</p>
<p>Do you know a way to estimate the GPU memory capacity needed for the training or the test of a model depending on the model itself, the input size, the batch size.</p>
<p>Thank you.</p>

username: Mazhar_Shaikh
cooked: <p>Hi Pirate-Roberts,<br>
You may find this repository useful. <a href="https://github.com/jacobkimmel/pytorch_modelsize" rel="nofollow noopener">link</a><br>
The gives a theoretical estimate. Actual memory used depends a lot on versions of CUDA, CUDNN and Pytorch being used.</p>

username: Pirate-Roberts
cooked: <p>I’ll check this, thank you!</p>
How can I get access to the torch._C.Node values in python?

username: JCPARK
cooked: <p>Hi, I am trying out jit trace functions to automatically parse the model structure.<br>
Originally, I use <code>model_output.grad_fn</code> for parsing the structure, but the graph structure seems simpler.</p>
<p>FYI, I am using PyTorch version 0.3.1 (installed with pip) in Python 2.7</p>
<p>Below is a part of the code I wrote:</p>
<pre><code class="lang-auto">input_var = Variable(torch.FloatTensor( np.random.random((1, init_shape[0], init_shape[1],init_shape[2])).astype(np.float32))).cuda()
trace, out = torch.jit.trace(model, (input_var,))
torch.onnx._optimize_trace(trace)
graph = trace.graph()
</code></pre>
<p>Once I get the graph, I notice that I can get access to the Nodes and Inputs (I assume they are Nodes containing weights, including the input tensor).<br>
<strong>I want to get access to the weights inside the nodes</strong>, but I cannot find any instructions related to get the weight values into python.</p>
<p>One workaround can be parse the Nodes, and retrieve the weights from <code>model.state_dict()</code>, but I think there should be as simpler way.</p>

username: samson-wang
cooked: <p>Hi JCPARK</p>
<p>Have you found a way?</p>

username: Rahul_Soni
cooked: <p><a class="mention" href="/u/jcpark">@JCPARK</a>, any luck with the issue?</p>
<p>I am stuck in a similar problem. Need inbound and outbound tensors for the node. Did not find any documentation to get attributes of <code>torch._C.Value</code></p>

username: whitelok
cooked: <p>except for <code>dir(torch._C.Value)</code> or review the source code. you can do nothing.</p>
👽 Computing hessian of a simple autograd function

username: bratishka_morr
cooked: <p>Dear community, does anyone managed to compute a hessian of an arbitrary autograd function?</p>
<p>Let’s consider simple quadratic function: f(x) = x.T A x, it’s gradient is (A + A.T)x and hessian is just (A + A.T).</p>
<p>I have two code snippets, first works:</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
from torch.autograd import Variable
from torch.autograd import grad
from torch import nn

torch.manual_seed(623)

x = Variable(torch.ones(2,1), requires_grad=True)
A = torch.FloatTensor([[1,2],[3,4]])

print(A)
print(x)

f = x.view(-1) @ A @ x
print(f)


x_1grad, = grad(f, x, create_graph=True)
print(x_1grad)
print(A @ x + A.t() @ x)

x_2grad0, = grad(x_1grad[0], x, create_graph=True)
x_2grad1, = grad(x_1grad[1], x, create_graph=True)

Hessian = torch.cat((x_2grad0, x_2grad1), dim=1)
print(Hessian)

print(A + A.t())
</code></pre>
<p>while the second does not work.</p>
<pre><code class="lang-auto">import torch
from torch import Tensor
from torch.autograd import Variable
from torch.autograd import grad
from torch import nn

torch.manual_seed(623)

class Quadro(torch.autograd.Function):
    
    def __init__(self, A):
        self.A = torch.FloatTensor(A)
    
    def forward(self, input):
    
        self.save_for_backward(input)
        return input.t() @ self.A @ input

    def backward(self, grad_output):
    
        input, = self.saved_tensors
        input = torch.autograd.Variable(input, requires_grad = True)
        grad_input = torch.mul(grad_output, (self.A + self.A.t()) @ input)
        return grad_input


x = Variable(torch.ones(2,1), requires_grad=True)
A = torch.FloatTensor([[1,2],[3,4]])

print(A)
print(x)

qq = Quadro(A)
f = qq(x)
print(f)

x_1grad, = grad(f, x, create_graph=True, retain_graph=True)
print(x_1grad)
print(A @ x + A.t() @ x)

x_2grad0, = grad(x_1grad[0], x, create_graph=True)
x_2grad1, = grad(x_1grad[1], x, create_graph=True)

Hessian = torch.cat((x_2grad0, x_2grad1), dim=1)
print(Hessian)

print(A + A.t())
</code></pre>
<p>The problem is that I cant differentiate the first derivative, since it has no grad_fn attribute, but I don’t know how to fix it.</p>
<p>P.S. I completely understand, that PyTorch framework was mostly done to train neural networks, not such simple things, but I would be happy, if someone could help me.</p>

username: nima_rafiee
cooked: <p>What error are you getting? I think you need to use retain_graph=True otherwise with one time gradient calculation the buffer will be zeroed.</p>