akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
AttributeError: module 'torch.nn' has no attribute 'GaussianNLLLoss'

Muhammad4hmed: <p>I am getting above mentioned error, my pytorch version is <code>'1.7.0'</code></p>

albanD: <p>You can check the doc for <code>1.7.0</code> <a href="https://pytorch.org/docs/1.7.0/search.html?q=gaussiannllloss&amp;check_keywords=yes&amp;area=default">https://pytorch.org/docs/1.7.0/search.html?q=gaussiannllloss&amp;check_keywords=yes&amp;area=default</a> this module did not exist back then.<br>
You will need to use <code>1.8+</code> to be able to use that: <a href="https://pytorch.org/docs/1.8.1/search.html?q=gaussiannllloss&amp;check_keywords=yes&amp;area=default" class="inline-onebox">Search — PyTorch 1.8.1 documentation</a></p>
When use loss.backward, it has bugs. RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Xuye_Liu: <h2>
<a name="to-reproduce-1" class="anchor" href="#to-reproduce-1"></a>To Reproduce</h2>
<p>Steps to reproduce the behavior:</p>
<ol>
<li>I create a specific model with pytorch when I use loss.backward, it has bugs. But it works when I use cuda gpu. When I try to run my model with cpu, it appears this bug.</li>
<li>I already try to reinstall pytorch version like 1.5.0, 1.6.0, 1.8.0. But it all not works</li>
</ol>
<p>code:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adamax(net.parameters(), lr = 2e-3)
loss_func = torch.nn.CrossEntropyLoss()
output = net([train1[0], train1[1], train1[2], train1[3]])
loss = loss_func(output, train2)
total_correct += get_accuracy(output, train2)
ml_loss.update(loss, batch_size*30)
acc.update(total_correct, batch_size * 30)
optimizer.zero_grad()
# backward propogation
loss.backward()
# weight optimizer
optimizer.step()
</code></pre>
<p>bug:</p>
<pre><code class="lang-auto">    loss.backward()
  File "/Users/leo/.pyenv/versions/3.7.7/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/Users/leo/.pyenv/versions/3.7.7/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [300, 300, 300]], which is output 0 of SelectBackward, is at version 16; expected version 15 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<h2>
<a name="expected-behavior-2" class="anchor" href="#expected-behavior-2"></a>Expected behavior</h2>
<p>Succeed to running my model script.</p>
<h2>
<a name="environment-3" class="anchor" href="#environment-3"></a>Environment</h2>
<ul>
<li>PyTorch Version 1.5.0:</li>
<li>OS (e.g., Linux) Mac:</li>
<li>How you installed PyTorch (<code>conda</code>, <code>pip</code>, source):pip install torch==1.5.0</li>
<li>Build command you used (if compiling from source):</li>
<li>Python version: 3.7.7</li>
<li>CUDA/cuDNN version:cpu</li>
<li>GPU models and configuration:</li>
<li>Any other relevant information:</li>
</ul>

albanD: <p>Hey</p>
<p>As mentioned in the error message, have you tried setting <code>torch.autograd.set_detect_anomaly(True)</code> and see if you get useful informations from that?</p>
Pytorch model save

DoYourBest: <p>I have two models then how could I save the models parameter?<br>
<code>torch.save(model1.state_dict(), 'model1_parameter_save_path') torch.save(model2.state_dict(), 'model2_parameter_save_path')</code><br>
Do I understand it right??</p>

Hmrishav_Bandyopadhy: <p>Yes. More info on the other ways to do this <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">here</a></p>
Torch.linalg.norm, anomally detection ord='fro'n as expected if ord=None for matrix

alevas: <p>Hi all, new pytorch user here.</p>
<p>I was trying to implement an Autoencoder with PyTorch and used the function <strong>torch.linalg.norm</strong>. Because the sparce autoencoder I wish to implement uses the Frobenius norm as a regularization term along with MSE and sparcity, using l2 and l1 norm respectively, I wanted to keep my code clean and understantable. Therefore I used the Frobenius norm like the following:</p>
<p><code>reg = torch.linalg.norm(W, ord='fro') ** 2</code><br>
where w is a h*d matrix.</p>
<p>My model comes from <code>torch.optim.Adam</code>. After executing the method <strong>backward</strong>, I was geting NaNs all over the place after. In an attempt to debug, after inserting</p>
<pre><code class="lang-auto">torch.autograd.set_detect_anomaly(True)
</code></pre>
<p>I located the issue. After a long time trying to find other ways to implement the Frobenius norm to avoid NaNs, I just typed</p>
<p><code>reg = torch.linalg.norm(W) ** 2</code></p>
<p>and it worked! From the documentation of torch.linalg.norm, I see that</p>
<blockquote>
<p>None(default): for matrices: Frobenius norm</p>
</blockquote>
<p>How can this be explained? Feel free to ask for more of my code or any debugging output I can produce. I wish to know whether it is a bug or if my torch noviceness is above level 9000.</p>
Concat unequal number of channels

dimka11: <p>I want to implement U-net from article. <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1505.04597.pdf</a><br>
There is copy and crop operation, then 512 channels need to concatenate with 1024 channels.</p>
<p>I have torch.Size([2, 1024, 32, 32]) after upsample middle layer, and torch.Size([2, 512, 32, 32]) after last encoder layer.</p>
<p>And after concat operation: torch.Size([2, 1536, 32, 32])</p>
<p>Hence, I got:<br>
RuntimeError: Given groups=1, weight of size [1024, 1024, 3, 3], expected input[2, 1536, 32, 32] to have 1024 channels, but got 1536 channels instead , when I gave this input to my decoder layer</p>
<p>What should I do?</p>

dimka11: <p>I did change input size of layers - train is starting but I got:<br>
RuntimeError: CUDA out of memory. Tried to allocate 400.00 MiB (GPU 0; 11.17 GiB total capacity; 10.24 GiB already allocated; 262.81 MiB free; 10.49 GiB reserved in total by PyTorch)</p>
<p>Is it normal that 12 GB of Collab gpu memory is not enough? And even 16 GB memory is not enough!<br>
Ok, I changed input size and batch size and now model is trainable.</p>
Regarding transforms.resize and drastic changes in accuracy

sriram2000na: <p>Transforms.Resize([]) followed by Transforms.ToTensor()<br>
vs<br>
Transforms.ToTensor() followed by Transforms.Resize([])<br>
I have tried both of these in a dataset in Transforms.Compose() and i find that the training and testing loss/accuracy changes drastically (87% to 94%) just by interchanging the transforms. Can someone tell me why and which one should be used, if there’s any preferred way to resize. I know resize works based on input (i.e.)  PIL or Tensor and my tranforms are linked to that, the problem is which one should we use?</p>
Model is overfittting on Waymo data set?

Suri12990: <p>Hey Admin and all,</p>
<p>I am trying to train model called SECOND for 3D object detection on waymo data set, but model is getting over fit on waymo data set. Used hyper parameters and model architecture are both same as for kitti data set.</p>
<p>But from observation, waymo data set is sequence, where as kitti is random. So I randomly feeding the data into the network in both data set cases. Still model is overfitting just on waymo.</p>
<p>I am doing the batch normalization, instead of dropout and regularization. From my understanding, batch normalization gives same effect as regularization.</p>
<p>More info about data set and model:</p>
<ul>
<li>Size of data set is almost same around 6k in both kitti and waymo?</li>
<li>But, target instance number is different, 5k in kitti and 30k in waymo. 800 and 5k in valid and test respectively.</li>
</ul>
<p>Here target is pedestrian.</p>
<p>Could someone help me with this problem?</p>
<p>Thanks in advance</p>
How to read an intermediate layer of a pretrained network

Xia_Zhengmi: <p>I have a pretrained network whose layers have similar names like the following code. Is there any way to extract the features from the intermediate layer?</p>
<pre><code class="lang-auto">class Net(nn.Module):
   def __init__(self):
      super(Net, self).__init__()
      self.conv1 = nn.Conv2d(...)
      self.conv2 = nn.Conv2d(....)
      self.conv3 = nn.Conv2d(....)

    def forward(self, x1):
      x1 = self.conv1(x1)
      x1 = self.conv2(x1)  &lt;-------------------- the variable that I want to visualize
      x1 = self.conv3(x1)
      x1
      return x1
</code></pre>

ptrblck: <p>You could use forward hooks as described in <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this post</a> or rename the output activation and return it in the <code>forward</code> method. The latter approach might be useful in case you would like to always return this activation, while the former one is more ad hoc.</p>
1 hot encoding in pytorch

Oussama_Bouldjedri: <p>I have a set of ID labels from 0…24 that encodes 19 classes and I want basically to have a 1 hot encoding of them: if I use :<br>
target = torch.sparse.torch.eye(19).index_select(dim=0, index=target)<br>
I get index out of range which I completely understand since some labels are 24 (but the number of total classes is 19), strange but the data set is built this way., and if I use:<br>
targets=torch.nn.functional.one_hot(targets)<br>
I get a tensor of 18 columns instead of 19, does anyone knows how to solve this?<br>
even if I set nbr classes to 19 this still blocks due to the fact that some labels are 24  but refers to another lower class<br>
Thanks in advance.</p>

ptrblck: <p>You could map the current class indices from <code>[0, 24]</code> to the expected range <code>[0, 18]</code> for 19 classes, since your use case would otherwise work with 25 classes where some might never be used.</p>
Applying saved trained model to differently sized test data

Nymue: <p>Hi,</p>
<p>I have a neural network model in pytorch that was trained on a labeled dataset where feature selection was performed (2000 → 800 features). Now I want to apply this model to an unlabeled test dataset. However, this test dataset still has the old number of features (2000). I saved the trained model using pickle and Pytorch’s state dict saving function, but both options give me dimension errors when I want to apply the model to the test set.<br>
How can I save and load my model in such a way that it works on the new data size? Thanks!</p>

ptrblck: <p>Could you post the model definition, the code raising this error, as well as the complete error message, please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Strided sum without having to loop

lkp411: <p>Hey there is there an efficient way to do a strided sum in pytorch? Particularly when the number of elements that fall under each stride is variable but specified? For example:</p>
<pre><code class="lang-auto">a = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
stride = torch.tensor([3, 3, 4])
result = torch.strided_sum(a, stride) (something like this)
</code></pre>
<p>Meaning, I want to sum the first 3 elements, the next 3 and the last 4 the resulting tensor would be</p>
<pre><code class="lang-auto">result = torch.tensor([(1 + 2 + 3), (4 + 5 + 6), (7 + 8 + 9 + 10)]) = torch.tensor([6, 15, 34])
</code></pre>
<p>I would like to do this without having to loop as that would make the function too slow. Is this possible?</p>

ptrblck: <p>You can use <code>scatter_add_</code> to accumulate these values to a new tensor.<br>
Assuming you have already calculated the <code>stride</code> tensor, you would need to create the index tensor from it as seen here:</p>
<pre><code class="lang-python">a = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
stride = torch.tensor([3, 3, 4])
idx = torch.tensor(sum([[i]*s for i, s in zip(range(stride.size(0)), stride)], []))

out = torch.zeros(stride.size(0), dtype=a.dtype).scatter_add_(0, idx, a)
print(out)
&gt; tensor([ 6, 15, 34])
</code></pre>

lkp411: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Is there a way to do this for other reduction operations? For instance if I wanted to apply a scatter_max or scatter_softmax?</p>

ptrblck: <p>You could take a look at the <code>scatter</code> methods from <a href="https://github.com/rusty1s/pytorch_scatter">rusty1s/pytorch_scatter</a>.</p>

lkp411: <p>I’m running into an strange error with CUDA and Pytorch when I try to use torch_scatter.</p>
<pre><code class="lang-auto">RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.
  Make sure that libnvrtc-builtins.so.11.1 is installed correctly.
nvrtc compilation failed:
</code></pre>
<p>Would you happen to know why this is happening? I’m running pytorch 1.8.1 + cu111</p>

ptrblck: <p>This seems to be a known issue with the CUDA11.1 pip wheels as described <a href="https://github.com/pytorch/pytorch/issues/58101">here</a>, so you might need to use the conda binaries, a source build, or the CUDA10.2 pip wheels.</p>
Reducing LSTM output to prediction of a smaller (but greater than 1) timespan, while maintaining batches

zachary-kaelan: <p>I’m trying to use 365 days to predict the next 30, for stock prediction, with 5 features (open, high, close, low, volume). I’m using a batch size of 64.</p>
<p>My network looks like this:</p>
<ol>
<li>Input &gt; (64, 365, 5)</li>
<li>LSTM &gt; (64, 365, 200)</li>
<li>Linear &gt; (64, 365, 100)</li>
<li>Permute to (64, 100, 365) to fit into batch normalization</li>
<li>Batched Normalization &gt; (64, 100, 365)</li>
<li>Permute to (64, 365, 100) to fit into ReLU</li>
<li>ReLU &gt; (64, 365, 100)</li>
<li>Linear &gt; (64, 365, 30)</li>
</ol>
<p>Going roughly based on <a href="https://medium.com/analytics-vidhya/pytorch-lstms-for-time-series-forecasting-of-indian-stocks-8a49157da8b9" rel="noopener nofollow ugc">this</a> for the structure.</p>
<p>The output I want is either (64, 30, 5) or (64, 30, 1). The 5 would be the same 5 features as the inputs, and the 1 would just be the average of the four price values or something…</p>
<p>All the sources I looked at for this kind of thing do something like lstm_out.view(-1, hidden_dim) with the LSTM output which gives me (23360, 100) and obviously won’t work for me as it gets rid of the batch organization. I could do (64, 36500) but that’s nothing like what any of the articles I read did, it’s a ridiculous amount of hidden units, and I don’t know how I would get back to (64, 30, 5) if I went down that route.</p>
Cogent Confabulation neural network

gjshannon: <p>Hi all…I’m new to the forum and have a basic question.</p>
<p>Context:</p>
<p>I’m interested in using PyTorch to develop a new type of neural network. It is based on the theory of Cogent Confabulation by the late Robert Hecht-Nielsen. I recently published a paper in the Neural Networks journal showing improved entity recognition using a measure of ontology cogency that I developed (my PhD dissertation results). Not asking anyone to explore the theory of confabulation to respond to my question, just providing context. But, if interested, please see <a>https://en.wikipedia.org/wiki/Confabulation_(neural_networks)</a></p>
<p>Question related to PyTorch:</p>
<p>I’m new to PyTorch, and don’t quite know where to start, so I’m looking for ideas on PyTorch customization.</p>
<p>Basically I want to develop a custom recurrent layer where the output of the layer is the product of conditional probabilities.</p>
<p>Right now the conditional probabilities are computed from a corpora and ontology prior to training a neural network.  To keep life simple, let’s assume that will not change.</p>
<p>How does one go about developing a customer layer in PyTorch that computes the product of the conditional probabilities?</p>
<p>Thanks in advance for your advice.</p>
Use individual module within scriptModule

copythatpasta: <p>I am aware of being able to access named attributes to view them, or change their value. But how can we run a forward pass through a single module from a Python compiled ScriptModule in c++? For example, being able to send inputs through only one of the conv layers. Or utilizing a custom auxiliary function created in python within  the model class to gain benefit from fused operations?</p>

copythatpasta: <p>Figured this out. I traversed the named children of the loaded module and was able to save off the .value of the child module I was interested in. From there I could run a forward pass on the single layer from the model</p>
Converting output into different form using transformation

nd_nd: <p>I am trying to convert the output from a layer in my network whose size is [batch, 3] using a function which I wrote:</p>
<pre><code>  def W300_EulerAngles2Vectors(self,x):

    '''
    rx: pitch
    ry: yaw
    rz: roll
    '''
    b,_ = x.shape
    rx=x[:,0]* (3.14 / 180.0)
    ry=x[:,1]* (3.14 / 180.0)
    rz=x[:,2]* (3.14 / 180.0)
    ry= ry * (-1)
    '''
    R_x = torch.tensor([[1.0, 0.0, 0.0],
                    [0.0, rx.cos(), -rx.sin()],
                    [0.0, rx.sin(), rx.cos()]], requires_grad=True)

    R_y =torch.tensor([[np.cos(ry), 0.0, np.sin(ry)],
                    [0.0, 1.0, 0.0],
                    [-np.sin(ry), 0.0, np.cos(ry)]], requires_grad=True)

    R_z = torch.tensor([[np.cos(rz), -np.sin(rz), 0.0],
                    [np.sin(rz), np.cos(rz), 0.0],
                    [0.0, 0.0, 1.0]], requires_grad=True)
    '''
    tensor_0 = torch.zeros(b)
    tensor_1 = torch.ones(b)
    print('rx',x[1][0].size(),rx.size())
    R_x = torch.stack([
            torch.stack([tensor_1, tensor_0, tensor_0]),
            torch.stack([tensor_0, torch.cos(rx), -torch.sin(rx)]),
            torch.stack([tensor_0, torch.sin(rx), torch.cos(rx)])]).reshape(b,3,3)
    R_y = torch.stack([
            torch.stack([torch.cos(ry), tensor_0, torch.sin(ry)]),
            torch.stack([tensor_0, tensor_1, tensor_0]),
            torch.stack([-torch.sin(ry), tensor_0, torch.cos(ry)])]).reshape(b,3,3)

    R_z = torch.stack([
            torch.stack([torch.cos(rz), -torch.sin(rz), tensor_0]),
            torch.stack([torch.sin(rz), torch.cos(rz), tensor_0]),
            torch.stack([tensor_0, tensor_0, tensor_1])]).reshape(b,3,3)
    R = torch.matmul(R_x,R_y)
    R = torch.matmul(R,R_z)
    l_vec = torch.matmul(R, torch.t(torch.tensor([1, 0, 0])))
    b_vec =  torch.matmul(R, torch.t(torch.tensor([0, 1, 0])))
    f_vec = torch.matmul(R, torch.t(torch.tensor([0, 0, 1])))   # R @ np.array([0, 0, 1]).T
    return [l_vec, b_vec, f_vec]
</code></pre>
<p>But, unfortunately, if I use the definitions of R_x, R_y, R_z in strings (comments), then it gives me error:<br>
‘‘Only one element tensors can be converted to Python scalars’’<br>
If I use define R_x, R_y, R_z using stacks, then it gives me error:<br>
‘‘RuntimeError: All input tensors must be on the same device. Received cpu and cuda:0’’</p>
<p>I want to use the output this function in my loss function. So, I want to backpropagate.</p>
<p>Please help. I will be very thankful for it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Intermediate output from sequential layers and to use the output for further processing while training

nd_nd: <p>I have a problem getting the output from nn. sequential as shown in the code below. I want to get the output of the three stages in the forward method as given below. These stages are called by the backbone in forward method. Each stage has one transformer layer. Precisely, I want to return x_stage1,x_stage2,x_stage3. Can you please help? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<pre><code>class VIT(nn.Module):
def __init__(
    self,
    *,
    image_size,
    num_classes,
    dim,
    depth,
    heads,
    mlp_mult,
    stages = 3,
    dim_key = 32,
    dim_value = 64,
    dropout = 0.,
    num_distill_classes = None
):
    super().__init__()

    dims = cast_tuple(dim, stages)
    depths = cast_tuple(depth, stages)
    layer_heads = cast_tuple(heads, stages)

    assert all(map(lambda t: len(t) == stages, (dims, depths, layer_heads))), 'dimensions, depths, and heads must be a tuple that is less than the designated number of stages'

    self.conv_embedding = nn.Sequential(
        nn.Conv2d(3, 32, 3, stride = 2, padding = 1),
        nn.Conv2d(32, 64, 3, stride = 2, padding = 1),
        nn.Conv2d(64, 128, 3, stride = 2, padding = 1),
        nn.Conv2d(128, dims[0], 3, stride = 2, padding = 1)
    )

    fmap_size = image_size // (2 ** 4)
    layers = []

    for ind, dim, depth, heads in zip(range(stages), dims, depths, layer_heads):
        is_last = ind == (stages - 1)
        layers.append(Transformer(dim, fmap_size, depth, heads, dim_key, dim_value, mlp_mult, dropout))

        if not is_last:
            next_dim = dims[ind + 1]
            layers.append(Transformer(dim, fmap_size, 1, heads * 2, dim_key, dim_value, dim_out = next_dim, downsample = True))
            fmap_size = ceil(fmap_size / 2)

    self.backbone = nn.Sequential(*layers)

    self.pool = nn.Sequential(
        nn.AdaptiveAvgPool2d(1),
        Rearrange('... () () -&gt; ...')
    )

    self.distill_head = nn.Linear(dim, num_distill_classes) if exists(num_distill_classes) else always(None)
    self.mlp_head = nn.Linear(dim, num_classes)

def forward(self, img):
    x = self.conv_embedding(img)
    x = self.backbone(x)        
    x = self.pool(x)
    out = self.mlp_head(x)
    distill = self.distill_head(x)
    if exists(distill):
        return out, distill
    return out, [x,x_stage1,x_stage2,x_stage3]</code></pre>

ptrblck: <p>It seems that <code>x_stageX</code> is undefined in the <code>forward</code>, so the <code>return</code> statement should yield an error.<br>
Based on the description of your use case you would need to define these intermediate tensors either in the <code>forward</code> of your main model or in the submodels. If the latter case is not easily done, you could use e.g. forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>

nd_nd: <p>Thank you :), I redefined these intermediate tensors in the <code>forward</code> of the main model. Thanks a lot.</p>
How can I change the result of torch.cuda.is_available() to True GTX 1050ti

Yeyvo: <p>hello i have recently started using pytorch but now i need to use my GPU which is a <strong>Nvidia GTX 1050ti</strong> to process some data but unfortunately torch.cuda.is_available() is returning False i have tried uninstaling <strong>cudatoolkit 11.3</strong> and downgrade it to <code>11.1</code> and also deleating and reinstaling pytorch using <code>conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -c conda-forge</code>, here’s some information about my local configuration :</p>
<pre><code class="lang-auto">OS : windows 10 
graphical card : nvidia 1050ti
current nvidia driver : 465.89
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.version.cuda
'11.1'
&gt;&gt;&gt; torch.cuda.is_available()
False
&gt;&gt;&gt; torch.backends.cudnn.enabled
True
&gt;&gt;&gt; torch.__version__
'1.8.1'
</code></pre>
<pre><code class="lang-auto">   -&gt;   nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Tue_Sep_15_19:12:04_Pacific_Daylight_Time_2020
Cuda compilation tools, release 11.1, V11.1.74
Build cuda_11.1.relgpu_drvr455TC455_06.29069683_0
</code></pre>
Permission error on using cpp_extention.py, i am using pytorch v1.6 on docker. anyone have an idea? kindly help

israrbacha: <p>File “/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 970, in load<br>
build_directory or _get_build_directory(name, verbose),<br>
File “/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 1454, in _get_build_directory<br>
os.makedirs(build_directory, exist_ok=True)<br>
File “/opt/conda/lib/python3.6/os.py”, line 210, in makedirs<br>
makedirs(head, mode, exist_ok)<br>
File “/opt/conda/lib/python3.6/os.py”, line 210, in makedirs<br>
makedirs(head, mode, exist_ok)<br>
File “/opt/conda/lib/python3.6/os.py”, line 220, in makedirs<br>
mkdir(name, mode)<br>
PermissionError: [Errno 13] Permission denied: ‘/.cache’</p>

ptrblck: <p>It seems you don’t have permission to create folders in <code>/</code>, so you could try to create a <code>/workspace</code> folder or any other home folder and see, if Python could write to it. Alternatively you might want to change permissions inside your docker container (but you should be root by default, shouldn’t you?).</p>

israrbacha: <p>Thanks! i solved it with os.environ[‘TORCH_EXTENSIONS_DIR’] = ‘path/to/folder/inside/project’ since the code was not allowed to create folder in /, and the ‘.cache’  folder(which hold pre-trained weights and extentions) is created by default in root</p>
Lasso regression (loss function)

Chamroukhi: <p>Dear all,<br>
im trying to use Lasso regression for loss function, any one have an idea how to use ?<br>
thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

notprime: <p>Suppose you have an autoencoder (but this approch is valid for any other netweork).<br>
You could write a function that takes the current weights values and computes the L1 regulatization (or Lasso regression).<br>
For instance:</p>
<pre><code class="lang-auto">def L1_reg(self):
      weights = torch.cat([x.view(-1) for x in self.encoder.parameters()])
      L1_reg  = LAMBDA_L1 * torch.norm(weights, 1)
      return L1_reg
</code></pre>
<p>where <code>LAMBDA_L1</code> is the lambda paramether of the Lasso (remember that this is a hyperparameter). With this little function you only compute the L1 regression on the encoder’s weights, but feel free to use also the decoder’s ones if you want to!<br>
If you want to use the Ridge, use <code>2</code> insted of <code>1</code> in <code>torch.norm</code>.<br>
Check the <a href="https://pytorch.org/docs/stable/linalg.html#torch.linalg.norm" rel="noopener nofollow ugc">torch.linalg.norm</a> method to better understand it.</p>
<p>Hope I’ve been helpful!</p>
Libtorch link error: cannot find -lCUDA_cublas_LIBRARY-NOTFOUND

Mactarvish: <p>I’m trying to compile my C++ code which depends on libtorch using CMake. Here is my CMakeList.txt:</p>
<pre><code class="lang-auto">
cmake_minimum_required(VERSION 3.2 FATAL_ERROR)
SET(PROJECT_NAME inference)
project(${PROJECT_NAME})
set(EXECUTABLE_OUTPUT_PATH ../bin)

SET(CMAKE_C_COMPILER g++)
if(CMAKE_COMPILER_IS_GNUCXX)
    add_compile_options(-std=c++11 -fno-stack-protector) # very important key in TK1,otherwise will raise an error call stack smashing detected
    message(STATUS "optional:-std=c++11")
endif(CMAKE_COMPILER_IS_GNUCXX)

set(opencv_dir ${CMAKE_CURRENT_SOURCE_DIR}/../third_party/opencv_4.2.0)
file(GLOB OpenCV_LIBS ${opencv_dir}/lib/*.so)


set(Torch_DIR ../third_party/libtorch/share/cmake/Torch)
find_package(Torch REQUIRED)

set(third_party_includes ../third_party/opencv_4.2.0/include ../third_party/libtorch/include
                         ../third_party/libtorch/include/torch/csrc/api/include/)


add_executable(${PROJECT_NAME} ../src/test.cpp)
include_directories(${PROJECT_NAME}/include)
include_directories(${PROJECT_NAME}/include/clipper)
target_include_directories(${PROJECT_NAME} PUBLIC ${third_party_includes})
target_link_libraries(${PROJECT_NAME} ${TORCH_LIBRARIES} ${OpenCV_LIBS})

SET(CMAKE_BUILD_TYPE DEBUG)

</code></pre>
<p>after I executed cmake --build . , error occurred:</p>
<pre><code class="lang-auto">(base) work@49fa12a97d74w:/home/ton/inference/build$ cmake --build .
-- optional:-std=c++11
-- Caffe2: CUDA detected: 10.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 10.1
-- Found cuDNN: v7.6.5  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;5.2;6.0;6.1;7.0;7.0+PTX;7.5;7.5+PTX
-- Added CUDA NVCC flags for: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_70,code=compute_70;-gencode;arch=compute_75,code=compute_75
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ton/inference/build
[ 50%] Linking CXX executable /home/ton/inference/bin/inference
/usr/bin/ld: cannot find -lCUDA_cublas_LIBRARY-NOTFOUND
collect2: error: ld returned 1 exit status
CMakeFiles/inference.dir/build.make:123: recipe for target '/home/ton/inference/bin/inference' failed
make[2]: *** [/home/ton/inference/bin/inference] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/inference.dir/all' failed
make[1]: *** [CMakeFiles/inference.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
</code></pre>
<p>How to solve it ? Thanks in advance.</p>
Why gradient came out None?

Sanglok: <p>I made computation graph for simple funtion.<br>
function y=3*(x**2) + 4*x+2</p>
<p>however, PyTorch code as follows,<br>
The gradient for ‘x’ came out as 22, but why does the gradient for ‘a’ come out as None?</p>
<p>I wanted it to come out as 3, but it comes out as None…<br>
How can I make the gradient of a come out?</p>
<p>PyTorch code as follows,<br>
Thanks in advance.</p>
<p>import torch<br>
x = torch.tensor([3.0], requires_grad=True)<br>
a = torch.pow(x,2)<br>
b = torch.mul(3,a)<br>
c = torch.mul(4,x)<br>
y = b+c+2</p>
<p>y.backward()</p>
<p>print(‘x.data:’, x.data)<br>
print(‘x.grad:’, x.grad)<br>
print(‘x.grad_fn:’, x.grad_fn)<br>
print()</p>
<p>print(‘a.data:’, a.data)<br>
print(‘a.grad:’, a.grad)<br>
print(‘a.grad_fn:’, a.grad_fn)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e154f7fe0cd8c0354fa7b43cebd786672b3cc7c.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e154f7fe0cd8c0354fa7b43cebd786672b3cc7c" title="pytorch_autograd"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e154f7fe0cd8c0354fa7b43cebd786672b3cc7c_2_690x387.png" alt="pytorch_autograd" data-base62-sha1="kgVsqe6KEfGc6O7Uc7liZnymbTK" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e154f7fe0cd8c0354fa7b43cebd786672b3cc7c_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8e154f7fe0cd8c0354fa7b43cebd786672b3cc7c_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e154f7fe0cd8c0354fa7b43cebd786672b3cc7c.png 2x" data-dominant-color="252A2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_autograd</span><span class="informations">1275×716 83.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Sanglok: <p>I solved this issue follows as.</p>
<p>a.retain_grad()</p>
Changing input in a preforward hook results in breaking of computation graph?

hrushi: <p>This is a typical pre forward hook function.</p>
<pre><code class="lang-auto">def pre_forward_hook_function(self, input):
     # Input would be tuple of size 1. To obtain the tensor
     x = input[0]
     # perform computations (simple addition/multiplication) on x and return it
     return x
</code></pre>
<p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html#torch.nn.Flatten.register_forward_pre_hook" rel="noopener nofollow ugc">preforward hook</a> has the input in the form a tuple, which is immutable.<br>
So to change the input, a new variable is assigned.  <code> x = input[0]</code><br>
Note that here, the input I am refering to is the input that is accessed within the pre forward hook function.</p>
<p>The problem is will this assignment on x result in breaking of the computation graph for the input(input to the network, not the input to the pre hook function) ? I have this doubt since the input of the pre hook function is a tuple and x is torch.tensor type, two different data types.</p>

hrushi: <p>After running the code, I can confirm this does not break the computation graph</p>
Pytorch efficient data loading and model training techniques

husnainali21: <p>Hello! I am working with a dataset of around 100K images ,All images are of different  rectangular  shapes and I tried <code>transforms.Resize()</code>,But it distorts most of the images and therefore I settled for a training loader with a batch size of 1 and using gradient accumulation , with images as their original size ,but even with num_workers : 0 and pin_memory : True the speed up gain  after first epoch is almost negligible, I assume this is because of high resolution of images because with the same setting for smaller images it worked faster .I want to know about any other approaches I can use to speed up the training(using PyTorch) as it takes more than 45  mins to just complete  one epoch.</p>
Newbie here, what is wrong with this simple CNN code? Can it be improved?

Sahil_Shingari: <p>This model is for recognizing 20 different classes of leaves from 28x28 RGB images.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd18a6810e3a4808bf484123fa7e4b3ef2a13ece.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cd18a6810e3a4808bf484123fa7e4b3ef2a13ece" title="Capture.PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd18a6810e3a4808bf484123fa7e4b3ef2a13ece_2_690x458.jpeg" alt="Capture.PNG" data-base62-sha1="tgmH4pfmdkckJNYKg7jhEhx1F7g" width="690" height="458" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd18a6810e3a4808bf484123fa7e4b3ef2a13ece_2_690x458.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd18a6810e3a4808bf484123fa7e4b3ef2a13ece_2_1035x687.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd18a6810e3a4808bf484123fa7e4b3ef2a13ece.jpeg 2x" data-dominant-color="E8E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture.PNG</span><span class="informations">1191×792 181 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

eqy: <p>The model architecture seems somewhat strange compared to typical classification architectures. What is the purpose of the sigmoid layer applied directly on the input?</p>

ptrblck: <p>Additionally to <a class="mention" href="/u/eqy">@eqy</a>’s comment it also is uncommon to avoid using non-linear activations between the trainable layers.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier and allows the search engine to index the code.</p>
Multimodal: Combine hugging outputs with tabular features

Benedikt_Droste: <p>Hi guys,<br>
currently I am working on a nlp project. I have just a small amount of samples (~3k) but received decent results with BERT. The given problem is a regression task. Each document is rated by users. The mean of these ratings is my target. For example text1 - 0.3242, text2 - 2.4232 and so on. In my baseline, I created the following model:</p>
<p>class BERT_BASELINE(nn.Module):</p>
<pre><code>def __init__(self):

    super(BERT_BASELINE, self).__init__()
    self.bert = transformers.BertModel.from_pretrained('bert-base-uncased')
    self.drop = nn.Dropout(0.3)
    self.out1 = nn.Linear(768, 256)
    self.relu = nn.ReLU()
    self.out2 = nn.Linear(256, 1)

def forward(self, ids, mask, token_type_ids):

    _, output = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids, return_dict=False)
    output = self.drop(output)
    output = self.out1(output)
    output = self.relu(output)
    output = self.out2(output)
    return output
</code></pre>
<p>In a second step I created a lot of “traditional” nlp features on the same dataset (369 features in total). Now I want combine it with the BERT outputs:</p>
<p>class BERT_ADVANCED(nn.Module):</p>
<pre><code>def __init__(self):

    super(BERT_ADVANCED, self).__init__()
    self.bert = transformers.BertModel.from_pretrained('bert-base-uncased')
    self.drop = nn.Dropout(0.1)
    self.out1 = nn.Linear(768+369, 1300)
    self.relu = nn.ReLU()
    self.out2 = nn.Linear(1300, 256)
    self.out3 = nn.Linear(256, 1)

def forward(self, ids, mask, token_type_ids, features):

    _, output = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids, return_dict=False)
    output = self.drop(output)
    x2 = features
    output_final = torch.cat((output, x2), dim=1)
    output_final = F.relu(self.out1(output_final))
    output_final = F.relu(self.out2(output_final))
    output_final = self.out3(output_final)
    return output_final
</code></pre>
<p>However, my results (in terms of rmse) don´t really improve. I have 2 questions:</p>
<ol>
<li>Do I miss some steps after combining the bert output and my own features?</li>
<li>Makes it sense to feed the hand crafted features through a mlp before combining it?</li>
</ol>
<p>Thanks!</p>
Running through the same layer using a second variable

Xia_Zhengmi: <p>Can anyone help me what is the difference between these two codes? I mean what happens when we use the same layer for a second time by another variable. Any graphical model would be highly appreciated!</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1, 32, 3, 1)<br>
self.conv2 = nn.Conv2d(32, 64, 3, 1)</p>
<pre><code>def forward(self, x1,x2):
  x1 = self.conv1(x1)
  x2 = self.conv2(x2)
  out=x1+x2
  return output
</code></pre>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1, 32, 3, 1)</p>
<pre><code>def forward(self, x1,x2):
  x1 = self.conv1(x1)
  x2 = self.conv1(x2)
  out=x1+x2
  return output</code></pre>

eqy: <p>The difference would be in the parameters used. The first example has two convolutions each with their own parameters (weights) while the second shares parameters across the two inputs.</p>
<p>As is, I don’t think both versions can work at the same time because the the two convs in the first example have a different number of input and output channels implying that x1/x2 have different inputs channels. This means that in the second example self.conv(x2) isn’t possible. Summing x1 and x2 in the first example is also strange though it might be possible via broadcasting.</p>

Xia_Zhengmi: <p>Thank you for your response. If we consider each generator as an encoder, the first one has two different encoders with different weights while the second one has two encoders with the shared weights. Am I  correct? This is the concept of two encoders with shared weights that sometimes papers refer to it?</p>
How to train an ML to predict improvement in patient recovery?

udayshankar: <p>This is not specific programming or API-related question but more of modeling and framing of the problem statement in to PyTorch models.</p>
<p>I am interested in training an ML model that would predict improvement in a patient’s injury.</p>
<p>Assume doctors have collected two sets of MRI images from hundreds of patients who have suspected brain injury. The first set was taken at 2 weeks after the injury and the second set was taken 6 months after the injury. Additionally, doctors have their own assessment of the injury (say a score indicating the strength of the injury). So the patient’s recovery from the injury can be better (increase in the score and hence positive) or worse (decrease in the score and hence negative) from 2 weeks to 6 months.</p>
<p>How to frame this problem so that I can train an ML model that takes the 2 weeks image and 6 months image as its inputs and predicts if the recovery is positive or negative? I can use the injury score provided by the doctors as a ground truth. But what sort of loss function to use to train this model in an end-to-end fashion?</p>
<p>I know I need an ML model that takes in two inputs <code>X</code> and <code>X'</code> and outputs a score <code>s</code> and the score should be any positive number if the recovery is good or any negative number if the recovery is bad. I am not sure what loss function to use so that I can train the model. Note that <code>X</code> represents the MRI scan taken at 2 weeks and <code>X'</code> represents the MRI scan taken at 6 months.</p>

googlebot: <p>I think the easy way is to approach this as [binary] classification problem, with added <a href="https://sklearn.org/modules/calibration.html" rel="noopener nofollow ugc">probability calibration</a> after training.</p>
<p>If you want to also utilize the magnitude of change of doctor scores, perhaps KLDivLoss could be used (map value to “soft” targets like (0.9, 0.1) by some ad-hoc formula, calibrating doctors’ scores).</p>
Attention Model to Sentiment Analysis

Sandipan_Majhi: <p>I am trying to implement Attention Model to lstm, where I am trying to predict a sentiment class given a sequence.<br>
For calculating attention I have chosen a learnable decoder which will get the attention probability wrt the hidden states of the lstm. The decoder is learnable. Lastly the weighted sum of the attentions and hidden states give the context vector.</p>
<p>I am getting very low train accuracy. The model doesn’t seem to learn much. Can you suggest anything which I can do ? What could be the reason for this ?</p>
How to script a derived conv2d using torch.jit

3yuan: <p>In some neural networks, the covn layer is derived from torch.nn.conv2d, which will cause an exception when scripting the module.<br>
The derived conv2d looks like:<br>
Class Conv2d(torch.nn.conv2d):<br>
…</p>

SplitInfinity: <p>Could you include a more detailed repro? Ideally, it should be a self-contained script that we can basically copy + paste and run.</p>
Code runs faster on CPU?

atharvahans1: <p>Hi I am having some trouble figuring out why my code (in particular the loss function) runs faster on a CPU (~ 0.03 sec) than on a GPU (~ 0.2 sec).</p>
<p>Is there a way I can fix this?</p>
<p>Here is how I define my loss:</p>
<pre><code class="lang-auto">def loss(X, u_r):

	t1 = perf_counter() ################################# For testing

	X.requires_grad = True
	u_r_X = torch.zeros(X.shape[0], X.shape[1], X.shape[1], device=device)
	for i in range(X.shape[0]): 
		u_r_X[i,:,:] = torch.autograd.functional.jacobian(u_r, X[i][None,:], create_graph=True)[0,:,0,:]

	energy_tensor = torch.zeros(X.shape[0], 1, device=device)
	for j in range(X.shape[0]):
		F = torch.eye(u_r_X[j].shape[0], device=device) + u_r_X[j]
		energy_tensor[j] = 0.5 * (torch.sum((F)**2) - 2.0) - torch.log((torch.det(F))) + 50.0 * torch.log((torch.det(F)))**2

	t2 = perf_counter() ################################# For testing
	print(t2-t1, ' secs') ################################# For testing

	return torch.mean(energy_tensor)
</code></pre>

Unity05: <p>Hi,</p>
<p>I guess this might be caused because you create new tensors and they have to be moved to the GPU if your device is set to GPU.</p>
<p>Regards,<br>
Unity05</p>

atharvahans1: <p>Can I make this faster or will this always be an issue?</p>

Unity05: <p>You’ll have to try to not having to initialize a new tensor every time you call the loss function.</p>
In TorchScript, why does prim::Exit have return a tensor?

garymm: <p>Hi,<br>
I’m trying to understand how the compiler works and have been reading the documentation.</p>
<p>In the section on the <a href="https://github.com/pytorch/pytorch/blob/75651e3cc4a7683970075c40334896927e4beaff/torch/csrc/jit/OVERVIEW.md#with" rel="noopener nofollow ugc">with statement</a>, I see that <code>prim::Exit</code> returns a Tensor in TorchScript graphs.</p>
<p>Why does <code>prim::Exit</code> have a return value at all?<br>
I could be wrong, but it seems impossible for any other code to actually do anything with the return value.</p>
<p>CC <a class="mention" href="/u/splitinfinity">@SplitInfinity</a> who I think was involved in implementing the with statement.</p>

SplitInfinity: <p>See <a href="https://docs.python.org/3/reference/compound_stmts.html#the-with-statement" rel="noopener nofollow ugc">this section of the Python language reference for more information</a>. <code>prim::Exit</code> corresponds to <code>__exit__</code> on the class being used as a context manager and is allowed to return a boolean value.</p>

garymm: <p>Thanks. Why does it return a Tensor instead of a bool?</p>

SplitInfinity: <p>My earlier statement was not completely accurate. <code>__exit__</code> is allowed to return anything that can be interpreted as a boolean, which includes certain kinds of Tensors.</p>
Advice on approach

Radioactive: <p>Hi everyone,<br>
A very fresh newbie here <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m a baby researcher haha,</p>
<p>I need advice on how to go about this problem.</p>
<p>I have a dataset of cropped eye images (~200k) with corresponding label landmarks (~200k JSON files).<br>
I want to have an end-to-end deep learning to track the corresponding landmark in real-time.</p>
<p>I was thinking, if I could use already trained Resnets, DenseNETS etc for this task, using transfer learning. The problem is I’m confused about how to go about this. The general pipeline is to have use of face detection, to detect the roi, then a facial landmarks to get corresponding eye landmarks  on the face.</p>
<p>Can someone give me a pointer how I can use transfer learning from already trained cnn? As I was thinking if it would ever be possible to use transfer learning (popular cnn architectures) as most of them were trained on imagenet (traditionally known for image classification,).</p>