akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to pass loss function from python code to C++ extension?

samarendra109: <p>I am fairly new to PyTorch and only very recently started using the PyTorch C++ API. So forgive me if my question is trivial. This is also my first question in the PyTorch forum, so please suggest if any reformatting I should be doing to my question.</p>
<p>I have the C++ code like this,</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;
#include &lt;iostream&gt;

namespace py = pybind11;

torch::Tensor calculateLoss(
    torch::Tensor pred,
    torch::Tensor target,
    &lt;pytorch_loss_fn&gt; fn
){
    return fn(pred,target);
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("calculateLoss", &amp;calculateLoss, "Calculate Loss In C++");
}
</code></pre>
<p>I want to pass the loss function from the python code. Something like this,</p>
<pre><code class="lang-auto">import torch
import dummy_cpp

from torch.nn import BCELoss

y_p = torch.Tensor([0.25,0.75])
y = torch.Tensor([0.0,1.0])

print(dummy_cpp.calculateLoss(y_p,y,BCELoss()))
</code></pre>
<p>Similar to Tensors, do we have some class in C++ that does this implicit transformation from PyTorch Loss Function object to C++ Torch Loss Function object? I tried putting nn::AnyModule but that doesn’t work. Is there any way to achieve this? I am also ok with solutions that perform a constant time operation to make a copy of the torch loss function in C++, i.e. something like this,</p>
<pre><code class="lang-auto">print(dummy_cpp.calculateLoss(y_p,y,processPython(BCELoss())))
</code></pre>
<pre><code class="lang-auto">auto fn = processCpp(&lt;processed Object/Tensor from python&gt;);
return fn(pred,target);
</code></pre>
<p>Some additional questions related to this,</p>
<ol>
<li>
<p>I don’t want to use the default way of passing a python function to C++ using pybind11, as due to repeated wrapping of C++ object to Python and vice versa, it affects the performance a lot. And as I will be calling the loss function multiple times from a loop I can’t have any performance overhead. Does this reasoning is also valid for the torch c++ extension? I have this doubt because PyTorch tensors are getting implicitly converted to C++ tensors, so maybe the overhead doesn’t exist?</p>
</li>
<li>
<p>I would also like to learn if we can pass activation functions and custom loss functions (inherited from torch.autograd.Function) to C++ extension in similar manner.</p>
</li>
</ol>
A request from a newcomer

jaiswalshashank: <p>A warm <strong>hello</strong> to all the community members. I’m new to open source and because of having a keen interest in ML and a Software enthusiast, I got to know about PyTorch. It would very much helpful for me and all the newcomers if we’d get to know what are the things and knowledge we are supposed to know in order to start contributing to the community.<br>
Thank You.</p>

ptrblck: <p>Welcome! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’m not sure, if you are in particular thinking about contributing to the framework directly via PRs (in that case check the <a href="https://github.com/pytorch/pytorch/contribute">good-first-issue tags</a> on GitHub and check the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">Contribution guide</a>) or about posting in this forum.<br>
In any case there is no such thing as a “requirement” to start contributing to the community and every helpful contribution is more than welcome.</p>
How to get features from RoBERTa model classifier

Shrabani_Ghosh: <p>I am using RoBERTa pretrained model go classify text.<br>
model = RobertaForSequenceClassification.from_pretrained(‘roberta-base’)<br>
outputs = model(**source_batch)</p>
<p>I want to know which parameter returns the features of the input ?</p>

ptrblck: <p>I guess the <a href="https://huggingface.co/transformers/model_doc/roberta.html#transformers.RobertaConfig"><code>RobertaConfig</code></a> defines the number of input features as well as other hyperparameters.<br>
In any case, you might also want to post this question in the <a href="https://discuss.huggingface.co/">HuggingFace discussion board</a>, since the code owners would see it there, too. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
GPU HANGS after some epoch during training

shrutishrestha: <p>I am training my medical imaging model in ResNet50. It was working good before, and trained full till 300 epoch. But, it has been 2 days, the program stops training on 1st epoch after 160 iteration and hangs without any error. the process does not get killed. I then have to reboot the server to make it run the next time.</p>
<p>What might be the reason for this?</p>
<p>nvidia-smi also gives thos error after the server hangs<br>
“Unable to determine the device handle for GPU 0000:65:00.0: GPU is lost.  Reboot the system to recover this GPU”</p>

ptrblck: <p>The hang is most likely created by the lost GPU. You could check <code>dmesg</code> for xid codes and check why the GPU was dropped. <code>XID 79</code> e.g. could indicate a thermal issue, which would disable your GPU before it can be damaged by overheating or an insufficient PSU.</p>
Multi-Node Multi-GPU - How to distribute model training on multi-node with multi-gpu?

bkuriach: <p>I have 3 nodes each with 2 GPUs, how can I distribute my model training? Does torch.dist.distributedparallel (or similar torch library) distribute training across Multi-node Multi-GPU? if not, what is the best alternative?</p>

eqy: <p>Yes, this is the purpose of <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch master documentation</a></p>

bkuriach: <p><a class="mention" href="/u/eqy">@eqy</a> I also heard about Horovod, it does the same thing? What is the best choice for above scenario? Thank you very much for your response!!</p>

mrshenli: <blockquote>
<p>I also heard about Horovod, it does the same thing? What is the best choice for above scenario?</p>
</blockquote>
<p>Hey <a class="mention" href="/u/bkuriach">@bkuriach</a>. It depends. If you would like to have framework (PyTorch/TensorFlow), Horovod distributed package might be a better fit. But if you are already using PyTorch, PyTorch DDP might be a better fit. Quoting my own responses from another post:</p>
<blockquote>
<p>One difference between PyTorch DDP is Horovod+PyTorch is that, DDP overlaps backward computation with communication. In contrast, according to the following example, Horovod synchronizes models in the optimizer <code>step()</code>, which won’t be able to overlap with backward computations. So, in theory, DDP should be faster.<br>
<a href="https://horovod.readthedocs.io/en/stable/pytorch.html" class="inline-onebox" rel="noopener nofollow ugc">Horovod with PyTorch — Horovod documentation</a></p>
</blockquote>
What if the image on the disk changes when we run DataLoader

phantom90: <p>Hi there,</p>
<p>Just curios what will happen if the image on the disk changes when we run DataLoader to traverse the dataset. Will the loaded data be the latest file on the disk or otherwise.</p>
<p>Thanks!</p>

KFrank: <p>Hi Phantom!</p>
<aside class="quote no-group" data-username="phantom90" data-post="1" data-topic="123395" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phantom90/48/12129_2.png" class="avatar"> phantom90:</div>
<blockquote>
<p>Will the loaded data be the latest file on the disk or otherwise.</p>
</blockquote>
</aside>
<p>What will happen is whatever <em>your code</em> decides to do.</p>
<p>A <code>torch.utils.data.DataLoader</code> is a wrapper that lets you iterate<br>
over a <code>Dataset</code>.  Per the documentation, a <a href="https://pytorch.org/docs/1.8.1/data.html#torch.utils.data.Dataset" rel="noopener nofollow ugc">Dataset</a> is an <em>abstract</em><br>
class that you have to implement concretely.  In particular, you have<br>
to implement its <code>__getitem__()</code> method.</p>
<p>If your implementation of <code>__getitem__()</code> (or one you get from some<br>
third-party code) rereads the image from the disk when called, you<br>
will fetch the latest version of the image from the disk.  But if<br>
<code>__getitem__()</code> fetches the item from a cache (for example from a<br>
cache that was built when the <code>Dataset</code> was instantiated), then you’ll<br>
get the “old” version of the image.</p>
<p><em>Typically</em> with images – which can be large – a <code>Dataset</code> will scan<br>
the image directories when constructed, most likely building a list<br>
of image pathnames, but not actually load the images into memory.<br>
It will then read a specific  image from disk when <code>__getitem__()</code><br>
is called (so you would get the “new” image).   But it’s up to you how<br>
you write your code.</p>
<p>(And just to be clear, <code>DataLoader</code> neither knows nor cares how its<br>
<code>Dataset</code> fetches its data items.  Whether you get the new or old<br>
image is solely up to <code>Dataset</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>
I have a value tensor and a raw index, how can I get a 2-D tensor

novice7: <p>the value tensor is value = torch.Tensor([2, 9, 7, 8, 4])<br>
the raw index tensor = torch.Tensor([0, 0, 1, 2, 3])</p>
<p>2 matches  0  which means 2 in 0th raw, and because the desired[0][0]=2,  the 9 matches 0 but the desired[0][0]==2, so desired[0][1]=9</p>
<p>I do want get the 2-D tensor<br>
desired tensor:</p>
<h1>
<a name="tensor2-9-1" class="anchor" href="#tensor2-9-1"></a>tensor([[2., 9.],</h1>
<h1>
<a name="7-0-2" class="anchor" href="#7-0-2"></a>[7., 0.],</h1>
<h1>
<a name="8-0-3" class="anchor" href="#8-0-3"></a>[8., 0.],</h1>
<h1>
<a name="4-0-4" class="anchor" href="#4-0-4"></a>[4., 0.]])</h1>

ptrblck: <p>I don’t fully understand the example, as it seems you would like to “fill” double indices into the second column. However, your index tensor doesn’t contain a <code>4</code>, so how is the last row created in the output?<br>
Also, what is expected, if e.g. <code>5</code> zeros are used in the index tensor? Would the result contain all source values in row0 and what would the output shape be?</p>
<p>EDIT: Also double post from <a href="https://discuss.pytorch.org/t/having-an-input-tensor-and-another-tensor-selection-i-want-to-produce-the-output-tensor-as-the-given-below/123233/14">here</a>.</p>

huahuanZ: <p>If I understand properly, this could help your</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; value = torch.tensor([2,9,7,8,4])
&gt;&gt;&gt; row_of_value = torch.tensor([0,0,1,2,3])
&gt;&gt;&gt; vec_row = [torch.zeros((row_of_value==i).sum())+value[row_of_value==i] for i in range(torch.max(row_of_value)+1)]
&gt;&gt;&gt; vec_row
[tensor([2., 9.]), tensor([7.]), tensor([8.]), tensor([4.])]
&gt;&gt;&gt; torch.nn.utils.rnn.pad_sequence(vec_row,batch_first=True)
tensor([[2., 9.],
        [7., 0.],
        [8., 0.],
        [4., 0.]])
</code></pre>
Pytorch for causal models

Mark_Pinches: <p>Hi all,</p>
<p>Im interested in coding up a causal model using pytorch and following the ideas within this paper… <a href="https://arxiv.org/pdf/2101.10943.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2101.10943.pdf</a></p>
<p>Fundamentally these models take an input array and pass this through a set(s) of layers with 3 outputs. These can be a variety of different architectures but typically the outputs are 1. the propensity for treatment (i.e. a softmax to estimate the probability of treatment from the covariates in the input array) and then two outputs for the effect, the effect with treatment and the effect without treatment.</p>
<p>However in the data we will for each instance of the data, see only one effect output (either if treated or not treated).</p>
<p>So my question is… How can I build a model that trains with target values from only 2 of 3 outputs for any given instance  (propensity + either the treated route or the untreated route)</p>
<p>presumably I would need some sort of mapping function that would turn on or off the loss calculation/backprop to treatment/no treatment layers? How could I do this?</p>
<p>Any suggestions welcome</p>
What are the differences of these code sniplets for calculating autograds

udemirezen: <p>Hi<br>
I have been trying to undertand autograd operation given in different forms from several repos while i investigate a specific topic:</p>
<pre><code class="lang-auto">u = net(x,t) 
u_t = torch.autograd.grad(u.sum(), t, create_graph=True)[0]
</code></pre>
<p>and,</p>
<pre><code class="lang-auto">u = net(x,t) 
u_t = torch.autograd.grad(u, t,  grad_outputs=torch.ones_like(u), create_graph=True)[0]
</code></pre>
<p>and,</p>
<pre><code class="lang-auto">u = net(x,t) 
u_t = torch.autograd.grad(u.sum(), t,  grad_outputs=torch.ones_like(u), create_graph=True)[0]
</code></pre>
<p>Unfortunately it is not clear for me. I have two questions about this:</p>
<ol>
<li>What are the main differences between them?</li>
<li>What is the correct way of calculating the gradients?<br>
Thank you</li>
</ol>

tom: <p>The backward of .sum() is “.expand()”.<br>
The first variant (btw. I tend to write it <code>u_t, = torch.autograd.grad(...)</code>, i.e. use tuple unpacking instead of indexing) is implicitly furnishing a scalar <code>1</code> as the <code>grad_outputs</code>.<br>
Thus the difference is that 1. has a first step does an expand of the <code>1</code> as the backward of <code>.sum()</code> while 2. doesn’t have this step. After this, the two backward passes are exactly identical.</p>
<p>The third variant looks like an error to me unless <code>u</code> is scalar already because there will be a shape mismatch between the first argument and the grad_outputs.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: This is a great question, I’ll re-use it as an exercise for my autograd course if you allow.</p>

udemirezen: <p>Thank you for excellent explanation. Now everything is clear.<br>
P.S.: Of course. you can use it wherever you want. it is my pleasure <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Kind regards.</p>
Vectorization problem

Alon_Zolfi: <p>Hey everyone,<br>
I’m trying to find a vectorized implementation for the following code - I have 3 tensors that represent:</p>
<ol>
<li>
<p>bboxes - [<span class="hashtag">#bboxes</span>, 4] which is a set of bounding boxes ranges, for example:<br>
[[1,3,6,9],[4,7,6,13],…]<br>
So bbox <span class="hashtag">#1</span> represents rows 1-3 and columns 6-9, bbox <span class="hashtag">#2</span> rows 4-7 columns 6-13, etc.<br>
<span class="hashtag">#bboxes</span> &gt; 50K.</p>
</li>
<li>
<p>values - [<span class="hashtag">#bboxes</span>, 1] - set of associated values to each bounding box, for example:<br>
[0.2, 0.5, 0.1, …]<br>
bbox#1 → 0.2<br>
bbox#2 → 0.5</p>
</li>
<li>
<p>image - [1, 512, 512].</p>
</li>
</ol>
<p>I need to “fill” the image tensor with the corresponding values of each bbox. Each pixel in the image should contain the highest value possible, i.e., if bbox#1 and bbox#2 have a matching pixel, that pixel should contain 0.5 and not 0.2 .<br>
So, a simple example that only contains 1 bbox would be:<br>
image[:, 1:3, 6:9] = values[0]</p>
<p>I’m trying to find a vectorized implementation that doesn’t contain a for loop over <span class="hashtag">#bboxes</span> since it is really slow due to the <span class="hashtag">#bboxes</span> size.<br>
Another issue is that it is impossible to create mask tensors for each bbox since it requires a [512, 512, <span class="hashtag">#bboxes</span>] tensor, which is impossible due to memory limitation.</p>
<p>Thanks!!</p>
Replace Pattern for nn.Modules?

sambhav_jain: <p>Would FX be able to support nn.Module patterns for replacement using the <code>subgraph_rewriter</code>? The examples / tests show simple functional patterns, but being able to define patterns of modules like nn.Conv2d or nn.BatchNorm2d etc can be really powerful. I see the Conv-BN Fuser tutorial uses direct graph manipulation, but this would be a great candidate to try with the <code>replace_pattern</code> if that can be supported. One consideration is in defining patterns that look for call_module ops with nn.Conv2d targets without hardcoding the kernel or channel sizes in the pattern itself (to allow more general matching based on op type). So more like a reg-ex (*) on the module parameters:</p>
<p>For example:</p>
<pre><code class="lang-auto">        class Pattern(torch.nn.Module):
            def __init__(self):
                self.conv = torch.nn.Conv2d(*, *, *)  # match all Conv2d
                self.bn = torch.nn.BatchNorm2d(*)   # match all BatchNorm2d

            def forward(self, x):
                return self.bn(self.conv(x))


        class Replacement(torch.nn.Module):
            def __init__(self):
                self.conv = torch.nn.Conv2d(*, *, *)

            def forward(self, x):
                return self.conv(x)

        tfx.subgraph_rewriter.replace_pattern(module.graph_module, Pattern(), Replacement())
</code></pre>
<p>Related topic: <a href="https://discuss.pytorch.org/t/torch-fx-replace-modules/116315" class="inline-onebox">Torch.fx replace modules</a></p>
Run multiple distributed training jobs on one machine

justinliu: <p>I have a machine that has 8 V100s, and I have a small job that only requires 4 V100 to run, so I am trying to run 2 4-V100 distributed training runs on the same machine at the same time, however, this gets me <code>RuntimeError: Address already in use</code> all the time for the second run I launch. It appears as if one distributed training job running on a machine will block any other distributed training runs. Is it possible to work around this so that I can launch 2 distributed jobs on one machine?</p>

eqy: <p>I’ve never tried this setup before so apologies if you have already considered it, but what happens when you change the port used for the 2nd distributed job?</p>

justinliu: <p>That sounds like a reasonable solution! Dumb question here though - how do one change the port used by a distributed job?</p>

eqy: <p>It should be close to where you specify the address (e.g., <code>MASTER_PORT</code> here <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#basic-use-case" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.8.1+cu102 documentation</a>).</p>

justinliu: <p>Thanks for <a class="mention" href="/u/eqy">@eqy</a>’s suggestion! I find that if using the PyTorch distributed launch utility script, then there is a <code>--master_port</code> argument (see <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" rel="noopener nofollow ugc">here</a>) one can use to set the port, and once different distributed jobs are configured to use different ports, the “Address already in use” problem goes away.</p>
How to dynamically adjust dataset within a epoch

phantom90: <p>Hi There,</p>
<p>I want to dynamically update the dataset within a epoch. That is something like:</p>
<pre><code class="lang-auto">dataset = ImageFolder()
dataloader = DataLoader()

for e in range(epoches):
    for batch in dataloader:
       # Do some job
      ....
       # Update the dataset
</code></pre>
<p>Thank you so much!</p>

eqy: <p>There are likely many ways of doing this depending on what “update the dataset” actually means. If you merely need to do something like tweak transformations, then this might be doable with some global state that is used by your transform/preprocessing functions. If it means adding or removing samples, then this becomes substantially trickier as it involves either changing the data loading iterator (<a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data.dataloader — PyTorch 1.8.1 documentation</a>) or building a new dataset and dataloader on the fly using the remaining data.</p>
RuntimeError: Exporting the operator grid_sampler to ONNX opset version 12 is not supported

Darshan: <p>Hi Pytorch team,</p>
<p>I am trying to convert the Pytorch model to the ONNX model. But whie exporting I am facing an error as: <strong>RuntimeError: Exporting the operator grid_sampler to ONNX opset version 12 is not supported</strong>.</p>
<p>May I know how to resolve this issue?</p>
<p>Thanks and Regards,<br>
Darshan C G</p>

ptrblck: <p>The issue is tracked <a href="https://github.com/pytorch/pytorch/issues/27212">here</a> and doesn’t seem to be resolved yet.<br>
Would you be interested in submitting a fix for it?</p>

Darshan: <p>Yes, <a class="mention" href="/u/ptrblck">@ptrblck</a> I am interested to submit a fix for it.<br>
May I know the procedure for it?</p>
<p>Regards,<br>
Darshan</p>

ptrblck: <p>Great! You can find general information about contributing to PyTorch <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">here</a>. I think the <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">CONTRIBUTING docs</a> are also really useful, as you can find best practices there.</p>
<p>Since you are interested in this particular PR, I would recommend to post your interest in the linked issue and follow up with the code owner. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to do the prediction for MultiLable Classification in test time?

seyeeet: <p>Hi</p>
<p>I am training a multilabel classification model and my data can have multiple labels for each image, I am using <code>BCEWithLogitsLoss</code> for training the loss. I know <code>BCEWithLogitsLoss</code> has sigmoid inside so I dont apply sigmoid in the training phase.<br>
My question is how this model will be used in the test time?<br>
Lets say I have an image that has 3 objects in it (cat,dog,human), I will apply sigmoid to the output of the model (which has dimension of <code>BxC</code> where C=20=number of classes), but how I can say what classes are in the image? this is different from the single label classification and I cannot do argmax on the output. If I use the probabilities after applying sigmoid should I threshold them?! but it wont be robust to use thresholding…<br>
Sorry if this question is very naïve</p>

eqy: <p>For 0,1 labels the typical approach is usually to just use 0.5 as the threshold (e.g., just round the predictions).</p>

seyeeet: <p>Thank you , do you know any resources that I can see how the do the evolution and how to report the performance for multi lable classification in image recognition?</p>
How to calculate eigenvector centrality for loss function?

theexile: <p>I have GAN for a graph prediction task in which there are <code>torch_geometric.nn.NNConv</code> layers. I want to add eigenvector centrality difference between ground-truth and predicted graph to my loss function. To calculate eigenvector centrality, I intended to use <code>eigenvector_centrality</code> function from <code>NetworkX</code> library. However, this function requires input to be a <code>NetworkX</code> graph which also requires to convert my <code>torch.tensor</code> outputted from Generator network to <code>numpy.array</code>. So, I need to <code>detach()</code> the gradient which will cause PyTorch to lose all gradient tracking. Thus, how can I properly implement a eigenvector centrality for my loss function? Thanks.</p>
Does PyTorch Geometric nn Module Consider Mini-Batch Input?

theexile: <p>In standard PyTorch documentation, it says that <code>torch.nn</code> considers the input as mini-batch. So, for one sample it suggests to use <code>input.unsqueeze(0)</code> in order to add a fake batch. Is this the case for PyTorch Geometric <code>nn</code> modules?</p>
<p>More specifically, I want to feed a fully-connected graph with 35 vertices and scalar edge weights to <code>NNConv</code> layer. So, I represent this graph as <code>Data</code> object where <code>Data.x</code> is a 35x35 adjacency matrix, <code>Data.edge_index</code> is 2 x 1225 tensor since it is fully-connected and <code>Data.edge_attr</code> is a tensor of shape 1225 x 1 since, again, it is fully-connected and edge attributes are just scalar weights. I designed such a <code>NNConv</code> layer and I am inputting not mini-batches but one sample to network.</p>
<pre><code class="lang-python">nn = Sequential(Linear(1, 1225), ReLU())
self.conv1 = NNConv(35, 35, nn, aggr='mean', root_weight=True, bias=True)
</code></pre>
<p>in forward function</p>
<pre><code class="lang-python">def forward(self, data):
    x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr

    x = F.sigmoid(self.conv11(self.conv1(x, edge_index, edge_attr)))
</code></pre>
<p>What I do not understand is do I need to add fake mini-batches. Is this correct or do I need to add <code>x.unsqueeze(0)</code>, if so which of these <code>Data</code> attributes (<code>x</code>, <code>edge_index</code>, <code>edge_attr</code>) do need a <code>unsqueeze(0)</code>. Thanks.</p>
In place operation causes RuntimeError

Haozhi: <p>The following code causes a RuntimeError ’ unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before performing the operation’. Could anyone help me?</p>
<pre><code class="lang-auto">import torch
kx = torch.linspace(-64, 63, 128)
kY0, kX0 = torch.meshgrid(kx, kx)
kX0 *= 0.001
kY0 *= 0.001
</code></pre>

ptrblck: <p><code>kY0</code> and <code>kX0</code> are expanded tensors and as the error message suggest, you could need to clone them before applying inplace operations.<br>
Did you try to use the suggested operation?</p>
<pre><code class="lang-python">kx = torch.linspace(-64, 63, 128)
kY0, kX0 = torch.meshgrid(kx, kx)
kX0 = kX0.contiguous()
kX0 *= 0.001
kY0 = kY0.clone()
kY0 *= 0.001
</code></pre>

Haozhi: <p>Thanks for your suggestion. I thought the <code>meshgrid</code> function is the same as the one in numpy, so I didn’t figure out why I has to clone the tensor first. Now I get it.</p>