akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
BUG occurs when loading model trained by higher version of pytorch

TigerVersusT: <p>I’m using pytorch=1.3.1 to load model saved by pytorch=1.7.1 and received the following error:</p>
<p>RuntimeError: version_number &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:131)</p>
<p>Is there any way to load higher version model? Thanks for your help.</p>

ptrblck: <p>PyTorch doesn’t support forward compatibility (loading data from a newer into an older release), but backwards compatibility, so you would need to update PyTorch or save the model in <code>1.3.1</code>.</p>
Broadcasting distance function

balaba_ba: <p>I am trying to get the distances between each part of tensor1(N,x,y) and tensor2(x,y) as a tensor of size N.</p>
<p>for exsample:</p>
<blockquote>
<p>a = torch.ones(2,2)<br>
b = torch.zeros(4,2,2)</p>
</blockquote>
<p>I expect to get distance(a,b) = [4,4,4,4], but torch.dist() does not allow me to specify whether to reduce along a given dim and only gives me the overall average which is a single number, [4].</p>
<p>What is the best way to do it rather than using a for loop? I want such operation to be excuted parallelly on GPU.</p>

ptrblck: <p>You could manually calculate the distance via:</p>
<pre><code class="lang-python">((a.expand(1, -1, -1) - b)**2).sum(dim=[1,2])
&gt; tensor([4., 4., 4., 4.])
</code></pre>
Making Predictions Using LSTM with Multivariate Input

BigMuffler: <p>Hi.</p>
<p>I have currently built an lstm for time-series predictions however I am not to sure how to actually make predictions into the future. The data I am using is multivariate (5219,4) where each vector of features was converted via a moving window method. After a window of length 5 is applied, the input vector changes to (5219,5,4) which suits the input requirement of the lstm module. After the model is trained, the resulting plot seems to map the training data fairly accurately. However, I am not sure how to perform predictions.</p>
<p>From what I’ve read on forums, the idea is to use a set sequence of data, the entire training set in my case, and use it to forecast the next day by a simple forward pass through my network. However, instead of one value for an output, the model spits out the entire batch as a prediction. Example below.</p>
<pre><code class="lang-auto">input.shape -&gt; (5219,5,4)
output.shape -&gt; (5219,1)
</code></pre>
<p>I would expect that given the entire batch, it would predict the 5220th element, since the output of the model is 1 element as per the model definition (number of outputs I defined). Maybe my understanding is incorrect. I have attached a snippet of my model below to help in understanding the problem.</p>
<pre><code class="lang-auto">class LSTM(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output, n_layers):
        super(LSTM, self).__init__()
        self.hidden = n_hidden
        self.num_layers = n_layers
        self.rnn = nn.LSTM(n_feature,n_hidden,n_layers,batch_first=True)
        self.relu = nn.ReLU()
        self.output = nn.Linear(n_hidden,n_output)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers,x.size(0),self.hidden) #initial hidden state
        c0 = torch.zeros(self.num_layers,x.size(0),self.hidden)
        x,(h0,c0) = self.rnn(x,(h0.detach(),c0.detach())) #output and hidden state
        x = self.relu(x)
        print(x.shape)
        #decode hidden state of last timestep
        x = x[:,-1,:]
        x = self.output(x)
        return x

model = LSTM(4,10,1,2)
</code></pre>
<p>Essentially, given multivariate input, how can I make predictions into the future? Any help with this would be greatly appreciated.</p>
Android pytorch model latency

user21: <p>Hello<br>
I am developing with reference to the pytorch object detection example.<br>
I am trying to detect an object using pytorch model in android.<br>
It receives an image in real time, receives this image as a bitmap, and displays a specific object as the result.</p>
<p>However, the TensorImageUtils.bitmapToFloat32Tensor function is time consuming and very slow compared to live images.</p>
<pre><code class="lang-auto">bitmap = Bitmap.createBitmap(bmp, 0, 0, bmp.width, bmp.height, matrix, true)
        var resizedBitmap = Bitmap.createScaledBitmap(bitmap,PrePostProcessor.mInputWidth,PrePostProcessor.mInputHeight,true)
        inputTensor = TensorImageUtils.bitmapToFloat32Tensor(resizedBitmap,PrePostProcessor.NO_MEAN_RGB,PrePostProcessor.NO_STD_RGB)
        outputTuple  = mModule!!.forward(IValue.from(inputTensor)).toTuple()
        outputTensor = (outputTuple as Array&lt;out IValue&gt;?)?.get(0)!!.toTensor()
        output = outputTensor!!.dataAsFloatArray
</code></pre>
<p>Is there any way to solve this? Should I force it to use the gpu? If so, how?</p>

Linbin: <p>How large is your image? Do you have any latency number?<br>
bitmapToFloat32Tensor is a function from torchvision. I don’t think it can run on GPU</p>

user21: <p>bitmapToFloat32 function takes about 3 seconds</p>
<p>mModule!!.forward(IValue.from(inputTensor)).toTuple() function takes about 1 seconds</p>
<p>image size 640 * 640</p>
Why aren't my tensor the same as in numpy when I port them into pytorch?

Brando_Miranda: <p>See how the tensors are not the same (or their sum):</p>
<pre><code class="lang-auto">import numpy as np
import torch
# - creating a random baseline
b1 = np.random.randn(D, N)
b2 = np.random.randn(D, N)
b1_t, b2_t = torch.from_numpy(b1).T, torch.from_numpy(b2).T
assert(b1_t.size() == torch.Size([N, D]))
print('-- reproducibity finger print')
print(f'{b1.sum()=}')
print(f'{b2.sum()=}')
print(f'{b1_t.sum()=}')
print(f'{b2_t.sum()=}')
print(f'{b1.shape=}')
print(f'{b1_t.shape=}')
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">-- reproducibity finger print
b1.sum()=8.881784197001252e-16
b2.sum()=1.5543122344752192e-15
b1_t.sum()=tensor(-4.4409e-16, dtype=torch.float64)
b2_t.sum()=tensor(8.8818e-16, dtype=torch.float64)
b1.shape=(7, 12)
b1_t.shape=torch.Size([12, 7])
</code></pre>

my3bikaht: <pre><code class="lang-auto">-- reproducibity finger print
b1.sum()=-0.3949629209972978
b2.sum()=5.1787780003950825
b1_t.sum()=tensor(-0.3950, dtype=torch.float64)
b2_t.sum()=tensor(5.1788, dtype=torch.float64)
b1.shape=(7, 12)
b1_t.shape=torch.Size([12, 7])
</code></pre>
<p>I wonder how you managed to generate random arrays with almost perfect 0 mean both times</p>
What is "forward" and when must you define it?

Glerk: <p>Hello, I am a bit confused on when you need to have a “forward” method in a NN module or a custom transform class. Can someone provide some more information?</p>
<p>Who calls forward? When does it get called? And when do you need to have it? I have named some of my modules other than forward and everything works, can someone explain?</p>

JuanFMontesinos: <p>forwards plays the same role as <code>__call__</code> does for a regular python class.<br>
Basically when you run <code>model(input)</code> this calls internally forward + some extra code around this function to add functionalities.<br>
This way they pytorch ensures you don’t rewrite <code>__call__</code> but an undefined method: forward.</p>
<p>Any custom module that you write (anything which inherits from a <code>nn.Module</code>) needs to define this function. Official layers already define  it.</p>
What happens when we dont put h0, c0 in LSTM, RNN etc

Khabbab_Zakaria: <p>I was playing with nn.LSTM, nn.RNN etc and I found something.<br>
The website states:</p>
<pre><code class="lang-auto">rnn = nn.LSTM(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(input, (h0, c0))
print(output.shape)
&gt;&gt;&gt; torch.Size([5, 3, 20])
</code></pre>
<p>And I tried the same thing without h0 and c0 as:</p>
<pre><code class="lang-auto">input = torch.randn(5, 3, 10)
rnn = LSTM(10, 20, 2)
output, _ = rnn(input)
print(output.shape)
&gt;&gt;&gt; torch.Size([5, 3, 20])
</code></pre>
<p>I wonder what is happening here. Is the model by itself assuming h0,c0?</p>

thecho7: <p>Hi, I got a small piece of document from PyTorch Docs.</p>
<ul>
<li>
<strong>h_0</strong>: tensor of shape (D * \text{num_layers}, N, H_{out})(D∗num_layers,N,Hout​) containing the initial hidden state for each element in the batch. Defaults to zeros if (h_0, c_0) is not provided.</li>
<li>
<strong>c_0</strong>: tensor of shape (D * \text{num_layers}, N, H_{cell})(D∗num_layers,N,Hcell​) containing the initial cell state for each element in the batch. Defaults to zeros if (h_0, c_0) is not provided.</li>
</ul>
<p>When you don’t define your h_0 and c_0, the model automatically create a zeros tensor according to your input size.</p>
<p>Thanks</p>
Module dictionary to GPU or cuda device

tanvi: <p>If there a direct way to map a dictionary variable defined inside a module (or model) to GPU?<br>
e.g. for tensors, I can do <code>a = a.to(device)</code><br>
However, this doesn’t work for a dictionary.<br>
In other words, is the only possible way is to map the keys individually like the following?</p>
<pre><code class="lang-auto">for key, value in self.dict.items():
    self.dict[key] = self.dict[key].to(device)
</code></pre>

ptrblck: <p>I think that’s the correct way, as you cannot move tensors inplace to the device.</p>

tanvi: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have 2 dictionary variables in my model and when I run DataParallel, it gives cuda mismatch error.<br>
I thought this will resolve the issue. However on searching, I got to know that only register buffers and parameters are replicated by DataParallel to the corresponding GPU. Could you suggest any solution for that?</p>

ptrblck: <p>If you cannot or don’t want to register these tensors as parameters of buffers, you could manually move them to the corresponding device by using the <code>.device</code> attribute of the input tensor:</p>
<pre><code class="lang-python">def forward(self, x):
    my_tensor = my_tensor.to(x.device)
</code></pre>

optimoose: <p>If the tensors are already on the GPU, then do you think it would be more optimal when creating the dictionary to make the keys on the GPU as well, or is easy just as easy to access it from cpu based keys? In my particular case I want to use keys that are generated from some mathematical manipulations of the tensors, so those keys would initially be on the GPU. My intuition is that the map itself does not live on the GPU, so it is not more optimal to have keys on the GPU, that need to be copied back to the CPU each time the dictionary is accessed. Hard to tell what’s really going on here.</p>

ptrblck: <p>The truth would be told by profiling the workload on the GPU vs. CPU, but based on the <a href="https://github.com/python/cpython/blob/9bf2cbc4c498812e14f20d86acb61c53928a5a57/Objects/dictobject.c">dict source code</a> I would assume the CPU is used.</p>
Method for inference time and throughput

Verner: <p>Hi everybody, hope that all of yours be fine.<br>
I´m research about inference time and throughput in CNN and i do like to know if exist a method on pytorch to calc this parameters?</p>
<p>Thanks a lot for all.</p>
<p>Verner</p>

ptrblck: <p>If you are looking for an approach to calculate the FLOPS, you might want to take a look at <a href="https://github.com/facebookresearch/fvcore/blob/main/docs/flop_count.md">the <code>fvcore</code> util</a>.</p>

Verner: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, i´ll test it.</p>
DDP with Gradient checkpointing

shivammehta007: <p>Since my method is an Autoregressive algorithm It is making a huge gradient tape, I am trying to do something like this</p>
<pre><code class="lang-python">
for i in range(len(maxtrix.shape)):
    output = torch.utils.checkpoint.checkpoint(NNModel(matrix[i]))

loss = -output.mean()
</code></pre>
<p>where NNModel is a torch.nn.Module.</p>
<p>It works fine on single GPU but on DDP it throws this error</p>
<pre><code class="lang-auto">RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 30 with name module.model.decoder.decoder_network.layers.1.weight has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.
</code></pre>
<p>I am running it with <code>find_unused_parameters=False</code><br>
Any workaround for this?</p>

pritamdamania87: <p><a class="mention" href="/u/shivammehta007">@shivammehta007</a> Can you try with <code>find_unused_parameters=True</code>? Also, can you provide a self contained repro of the issue?</p>
<p>Another option is you could use set_static_graph: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1614" class="inline-onebox" rel="noopener nofollow ugc">pytorch/distributed.py at master · pytorch/pytorch · GitHub</a>, if the parameters used in each iteration of your model are always the same.</p>
RuntimeError: The Tensor returned by the function given to hessian should contain a single element - Is it possible for torch.autograd.functional.hessian to support batches with PyTorch 1.0?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I was wondering if it’s at all possible to compute the Hessian of an <code>nn.Module</code> of the output w.r.t its input for a batch of inputs? I understand this is a limitation of <code>torch.autograd.functional.hessian</code> but is there a way to add batch support to such a function? Like I’ve shown below with the <code>jacobian</code> function?</p>
<p>Any help would be apprecitated!<br>
Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">import torch

def func(xs):
  return xs.pow(2).sum(dim=-1)

def jacobian(net, xs):
  ys = net(xs)
  return torch.autograd.grad(ys, xs, torch.ones_like(ys))[0]
 
def hessian(net, xs):
  return torch.autograd.functional.hessian(net, xs)
 
xs = torch.randn(1, requires_grad=True) #single example (i.e. no batch)
jacobian(func, xs) #returns tensor([0.6949])
hessian(func, xs) #returns tensor([[2.]])
 
xs = torch.randn(2, 1, requires_grad=True) #batch dim included 
 
jacobian(func, xs)
"""
returns
tensor([[-2.7621],
        [ 0.1642]])
"""
hessian(func, xs) #throws error

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;stdin&gt;", line 2, in hessian
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 701, in hessian
    res = jacobian(jac_func, inputs, create_graph=create_graph, strict=strict, vectorize=vectorize)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 482, in jacobian
    outputs = func(*inputs)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 697, in jac_func
    jac = jacobian(ensure_single_output_function, inp, create_graph=True)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 482, in jacobian
    outputs = func(*inputs)
  File "~/anaconda3/lib/python3.8/site-packages/torch/autograd/functional.py", line 692, in ensure_single_output_function
    raise RuntimeError("The Tensor returned by the function given to hessian should contain a single element")
RuntimeError: The Tensor returned by the function given to hessian should contain a single element
</code></pre>
Does a CNN depend on the batch size?

pascalm: <p>Hello,</p>
<p>I’m trying to grasp Conv1d. Now assume I have a data loader with a batch size N. If I create a sequential CNN, does the input layer and first CNN layer of that CNN depend on N?</p>

InnovArul: <p>AFAIK, conv layers don’t depend on <code>N</code>.<br>
However, the batch norm would need a constraint that <code>N != 1</code> during training.</p>
<p>Other than this, the training behavior might be different with different <code>N</code>s, as the gradient will be more or less noisy depending on N.</p>

pascalm: <p>Okay, thanks.</p>
<p>Edit: I’ll make a new thread with a follow up question. Thanks.</p>
Multiple Categorical distributions

HDubois-fr: <p>Hi,<br>
I’m working with a feature vector of multiple categorical features, with various cardinalities. How can I write my code so that I can log_prob the whole vector at once.</p>
<p>For now it looks like this:</p>
<pre><code class="lang-auto">def __init__(self,k):
    ...
    self.x1 = torch.randn(k,2,requires_grad=True)
    self.x2 = torch.randn(k,5,requires_grad=True)
    self.x3 = torch.randn(k,3,requires_grad=True)
...

def forward(self, x):
    D_x1 = D.Categorical(logits=self.x1)
    D_x2 = D.Categorical(logits=self.x2)
    D_x3 = D.Categorical(logits=self.x3)

    return D_x1.log_prob(x[:,0]) + D_x2.log_prob(x[:,1]) + D_x3.log_prob(x[:,2])
</code></pre>
<p>I’d use D.MixtureSameFamily but I can figure how to handle various number of logits.</p>
<p>Thanks !</p>
Performance Measure with PCP score

Beginner_PY_code: <p>Need to calculate the PCP score will be computed for the lower arms, upper arms, lower legs and upper legs. The total score will be the average of the arms and legs PCP score.</p>
<p>The PCP score of a limb, defined by a pair of joints, is considered correct if the distance between both estimated joint locations and true limb joint locations is at most 50% of the length of the ground-truth limb.</p>
<p>Can someone please explain the concept of calculating distance estimation?</p>
Dataloader for different datasets

testymctest: <p>Hi, I’m trying to finetune a model using custom datasets, however, the datasets contain different annotation styles. One dataset contains its annotations in a JSON File, another has the data saved in the pixel value of the image, and another contains them in H5 files. What is the easiest way for me to combine all of these into one dataset so that I can properly fine tune my model? Or is it best to create 3 different data loaders, and then randomly chose which one to use for the next training image?</p>

ptrblck: <p>I assume that the output of the <code>Dataset.__getitem__</code> would still yield the same objects regardless how the data is loaded internally (from the JSON file, H5, or directly as images).<br>
If that’s the case you could write different <code>Dataset</code> implementations and use <code>ConcatDatset</code> to create a single one before passing them to the <code>DataLoader</code>.</p>

testymctest: <p>Ah, okay great. Thanks a lot, I didn’t realize that you could write a different dataset class and concat them for a data loader. Must have missed it in the docs. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
A question regarding the number of operations performed when using requires_grad=False

OttoVonBiscuitmark: <p>I am trying to estimate how many FLOPS my model is performing in a similar way to how it is done in this paper : <a href="https://arxiv.org/pdf/2001.08361.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2001.08361.pdf</a> . In the paper they provide an estimate for the number of add-multiply operations specific for transformers and per token as C ≈6N floating point operators per training token where N is the number of parameters excluding the embedding parameters. This number comes from the forward pass requiring ~2N operations and the backward pass roughly twice as much as the forward pass.</p>
<p>I am now training a transformer model where I have frozen all the parameters of the network by setting requires_grad=False. I have then added a small MLP before the input layer to the transformer which processes the input and then feeds it to the transformer input layer. The parameters of this MLP are the only ones that will be tuned. My question is then what effect this has on the number of operations done during the forward and backward pass?</p>
<p>After reading <a href="https://pytorch.org/docs/stable/notes/autograd.html" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 1.10.0 documentation</a> my understanding is that since I need gradients for the input layer (my MLP) the autograd engine would still need to store all intermediate gradients in the network (including the transformer) since the gradients for the MLP depends on these. Am I understanding this correctly? So in this case the number of operations would be C ≈6N where N = N_transformer + N_MLP ? While on the other hand, if I added a MLP after the Transformer for finetuning or whatever and then set required_grad=False on the transformer layers, then we would get 2N (N=N_trans + N_MLP) for the forward pass but only 2*(2*N_MLP) (i.e. twice the operations for the MLP forward pass) for the backward pass?</p>

StudyingMonk: <p>I think your assumption is wrong, since freezing the latter part of the gradient computation should save some computation during back propagation. So they would both be 2 * 2 * N_MLP in my opinion.</p>
Getting min before a max

charlesd: <p>Given a tensor of values with dimensions [batch, values], I need to get the minimum value that occurs before the maximum value. Thus, this naive approach:</p>
<pre><code class="lang-auto">value_max, value_max_idx = data.max(dim=1)
value_min_before_max, _ = data[:value_max_idx + 1].min(dim=1)
</code></pre>
<p>This obviously didn’t work as the slice is a tensor of dimension [batch]. But I feel like there should be a simple answer here… anyone have a good idea?</p>

tom: <p>You could mask the data beyond the max with the +inf or the maximum integer for your dtype it before taking the min:</p>
<pre><code class="lang-python">inf_tensor = torch.tensor(math.inf, device=data.device, dtype=data.dtype)
col_idx = torch.arange(data.size(1))
value_max, value_max_idx = data.max(dim=1)
masked_data = torch.where(col_idx[None, :] &lt;= value_max_idx[:, None], data, inf_tensor)
value_min_before_max, _ = masked_data.min(dim=1)
</code></pre>
<p>Depending on your situation, overwriting the relevant data indices might be an alternative to avoid allocating <code>masked_data</code> as a tensor, but I generally advise <a href="https://lernapparat.de/pytorch-inplace/" rel="noopener nofollow ugc">against using inplace unless you absolutely need to</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Tensor.expand() lead to c10::Error

kila_suelika: <p>I’m using Libtorch 1.10 Release Version on windows with VS 2022. The following code:</p>
<pre><code class="lang-auto">
#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

namespace F = torch::nn::functional;
using namespace std;

int main()
{
	torch::Tensor ox1 = torch::rand({ 3,3 });
	cout &lt;&lt; ox1.expand({6,3}) &lt;&lt; endl;
</code></pre>
<p>lead to</p>
<pre><code class="lang-auto">Unhandled exception at 0x00007FFAB6924B89 in test_libtorch.exe: Microsoft C++ exception: c10::Error at memory location 0x000000E83D4FEFD0.
</code></pre>
<p>But if replace <code>expand</code> with <code>repeat</code> then there is no problem.</p>

kila_suelika: <p>Finally I find that <code>expand</code> can only be used on a tensor with size on some dimensions is exactly 1.</p>
Using list variable with torch.nn.dataparallel

ku294714: <p>Hello <a class="mention" href="/u/alband">@albanD</a>,  I am using torch.nn.dataparallel to use multiple gpus. Instead of splitting data over multiple gpu’s , data is simply getting coppied on the two gpu’s. The reason for this is  that in the forward function of the model instead of passing a tensor variable as input, I am passing a  list of tensor variables.<br>
Is it not possible to split data using list of tensor variable?<br>
What changes can i make in my current setup to efficiently use data parallel.</p>

lamrin: <p>I have the same problem as well. Any solution?</p>
Is there any way to update the parameter and pass the grad?

daquexian: <p>I want to implement the binary neural network in pytorch.</p>
<p>In the binary neural network, the weights are binarized to +1 or -1 based on the original float weights, and the original float weights are updated in backward according to the grad of binarized weights.</p>
<p>But the way in the existing <a href="https://github.com/itayhubara/BinaryNet.pytorch/issues/6" rel="nofollow noopener">BinaryNet.pytorch</a> is wrong, the original float weights are not updated at all for it directly modifies the <code>.data</code>, <a href="https://github.com/itayhubara/BinaryNet.pytorch/blob/master/models/binarized_modules.py#L96-L101" rel="nofollow noopener">https://github.com/itayhubara/BinaryNet.pytorch/blob/master/models/binarized_modules.py#L96-L101</a> .</p>
<p>Is there any way to update the parameter and pass the grad correctly?</p>

chenzihao: <aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/2109.15103">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2.8/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/2109.15103" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/2109.15103" target="_blank" rel="noopener nofollow ugc">Scalable Rule-Based Representation Learning for Interpretable Classification</a></h3>

  <p>Rule-based models, e.g., decision trees, are widely used in scenarios
demanding high model interpretability for their transparent inner structures
and good model expressivity. However, rule-based models are hard to optimize,
especially on large data...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
可以看看这篇文章</p>
Emulating logic gates with binary Neural network?

zoyitezon: <p>Greetings, I’m attempting to learn Pytorch by making a Neural network which takes<br>
some binary input ( In the form of a torch.bool tensor )<br>
What layers accept a torch.bool And/Or are the best fit for logic gates emulation/estimation?</p>
<p>Any suggestions?</p>

chenzihao: <aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/2109.15103">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.2.8/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/2109.15103" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/2109.15103" target="_blank" rel="noopener nofollow ugc">Scalable Rule-Based Representation Learning for Interpretable Classification</a></h3>

  <p>Rule-based models, e.g., decision trees, are widely used in scenarios
demanding high model interpretability for their transparent inner structures
and good model expressivity. However, rule-based models are hard to optimize,
especially on large data...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
AttributeError: module 'torchvision.io' has no attribute 'read_image'

wsy: <p>What can I do for this?thanks</p>

ptrblck: <p>You could update your <code>torchvision</code> library, as this method was introduced in <code>0.8.0</code> so I guess you are using an older version.</p>

wsy: <p>Thanks,I will try it.</p>
Can I register forward hooks for functional calls?

shengchunnan: <p>As we know, we can register forward hooks for <strong><code>nn.Module</code></strong> instances. But there are functional operations like <strong><code>nn.functional.interpolate()</code></strong>,  or <strong><code>torch.cat()</code></strong>, or even implicit callable like <strong><code>&lt;built-in function add&gt;</code></strong>  for element-wise summation of tensor_a and tensor_b like this:</p>
<pre><code class="lang-auto">tensor_c = tensor_a + tensor_b  # There is a &lt;built-in function add&gt;  here
</code></pre>
<p>Can we use hooks for these functional calls?</p>

ptrblck: <p>I don’t think that’s possible, since the forward hooks are called in the <code>nn.Module.__call__</code> method before dispatching to the <code>forward</code> method.<br>
Since you are using the functional API you already have the output and input activations and could e.g. store them directly if needed. If hooks are more convenient to use instead of the tensors directly, you would probably have to wrap your calls into a custom module.</p>

shengchunnan: <p>I cannot alter the original module definitions because I am implementing a tool that has to accept any popular backbone structures like ResNet, googleNet, denseNet, etc.<br>
Luckily, I have successfully indirectly figured out shapes of inputs and outputs of functional calls by recursively searching in <strong><code>torch.fx.graph.Graph</code></strong>. But thank you anyway.</p>
Output latitude question of backward

kun_yang1: <p>When customizing the activation function with one parameter, the latitude of weight is (C = 32, H = 1, w = 1). When I implement the backward method, The latitude of grad_ weight is (B = 30, C = 32, H = 1, W = 1).  I tried to average the batch latitude data of grad_ weight to obtain the latitude (C = 32, H = 1, w = 1). In both cases, there is no problem with my network. It puzzles me, No matter whether the grad_weight output is (b = 30, C = 32, H = 1, w = 1) or (C = 32, H = 1, w = 1), my network seems to have no problem running. I can still get the updated value of the weight of the custom layer, and the latitude of weight is still (C = 32, H = 1, w = 1).In the face of this situation, how should I deal with it?<br>
Here is my code：</p>
<pre><code class="lang-auto">class PDELU(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input, weight):     # input.size()-&gt;(30,32,116,144)      weight.size()  -&gt;  (32,1,1)
        t = 0.9
        output = torch.clone(input)
        weight_ = weight.expand(input.size())  
        output[input &lt;= 0] = weight_[input &lt;= 0]*((1+(1-t)*output[input &lt;= 0])**(1/(1-t))-1)
        ctx.save_for_backward(input, weight, output)
        # print(output.size())
        return output    #  output.size()  -&gt; (30,32,116,144)

    @staticmethod
    def backward(ctx, grad_outputs):  # grad_outputs.size()-&gt; (30,32,116,144)

        t = 0.9
        input, weight, output = ctx.saved_tensors
        weight_ = weight.expand(input.size())
        grad_input_ = torch.clone(output)   # First calculate the gradient of the input with respect to the output
        grad_input_[input &gt; 0] = 1
        grad_input_[input &lt;= 0] = ((output[input &lt;= 0] + weight_[input &lt;= 0])**t)*(weight_[input &lt;= 0]**(1-t))
        grad_input = grad_input_*grad_outputs   # Calculate the gradient of the input with respect to the loss function
        grad_weight_ = torch.clone(input)
        grad_weight_[input &gt; 0] = 0
        grad_weight_[input &lt;= 0] = (1+(1-t)*input[input &lt;= 0])**(1/(1-t))-1     
        grad_weight = torch.sum((grad_weight_ * grad_outputs), dim=(-2, -1), keepdim=True)

        return grad_input, grad_weight    # grad_input.size()-&gt; (30,32,116,144)   grad_weight.size()-&gt;(30,32,1,1)
</code></pre>
Extract and Reshape

krish13: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a8c4bee5e1c82c2fabad781460f7dc14fce9e14.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1a8c4bee5e1c82c2fabad781460f7dc14fce9e14" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a8c4bee5e1c82c2fabad781460f7dc14fce9e14_2_690x466.jpeg" alt="image" data-base62-sha1="3MQZTytDZKQYyLAlFxUCLqhNZ0o" width="690" height="466" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a8c4bee5e1c82c2fabad781460f7dc14fce9e14_2_690x466.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a8c4bee5e1c82c2fabad781460f7dc14fce9e14_2_1035x699.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a8c4bee5e1c82c2fabad781460f7dc14fce9e14_2_1380x932.jpeg 2x" data-dominant-color="CDD0C9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1734×1172 152 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there an efficient way to do the above operation without using the for loop?</p>
<p>One way:<br>
h = 4<br>
w = 4<br>
c = 2<br>
A = given_tensor.reshape(h,w,c,4)</p>
<p>B = zeros(128,4)</p>
<p>Then<br>
count = 0<br>
for i in range(h):<br>
for j in range(w):<br>
for k in range(c):<br>
B[count,:] =  A[i,h,k,:]<br>
count = count + 1</p>

Eta_C: <p>I am not sure that I understand this operation correctly.<br>
Could you show the operation with for loop?</p>

ptrblck: <p>Based on the figure I think you could check <code>torch.unfold</code> with a kernel size and stride of <code>4</code>.</p>

krish13: <p>I edited the post to show the loop, hopefully, it makes sense.</p>

Eta_C: <p>Base on your code, <code>view</code> is enough.</p>
<pre><code class="lang-python">import torch 

h = 4
w = 4
c = 2
A = torch.randn(h, w, c, 4)


def func_1(data):
    B = torch.zeros(32,4)
    count = 0
    for i in range(h):
        for j in range(w):
            for k in range(c):
                B[count,:] = A[i, j, k, :]
                count = count + 1
                assert count &lt;= 32
    return B 

def func_2(data):
    return data.view(-1, 4)

print(func_2(A).shape)
print(func_1(A) - func_2(A))
</code></pre>
Loading learning rate scheduler

Sarah_Lancaster: <p>I’m confused about the order of instantiating and loading a learning rate scheduler object.</p>
<p>In the code below, I would like to resume training from a previous checkpoint. Setting last_epoch to that provided in the checkpoint ensures that the initial learning rates for the parameters are set correctly. However, calling load_state_dict after overwrites last_epoch (which would’ve been incremented by 1 in the instantiation due to the call to step), causing discrepancies in the resumption of the schedule.</p>
<p>Is this a bug or is there a proper way of doing the instantiation and restoration?</p>
<pre><code class="lang-auto">lr_scheduler=optim.lr_scheduler.CosineAnnealingLR(optimizer, opt.epochs,
                                                            last_epoch=checkpoint['lr_scheduler']['last_epoch'] if checkpoint else -1)

if checkpoint: lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
</code></pre>
Help reproducing this CNN?

Sam_Lerman: <p>Here is an attached schematic (just a 1D CNN):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd5c35862e9204e139ad1eeaa1f4ea011257a27a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cd5c35862e9204e139ad1eeaa1f4ea011257a27a" title="Screen Shot 2021-11-01 at 2.24.53 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd5c35862e9204e139ad1eeaa1f4ea011257a27a_2_660x500.jpeg" alt="Screen Shot 2021-11-01 at 2.24.53 PM" data-base62-sha1="tiHrbwyaJfMciwgSUvzfO89Zoee" width="660" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd5c35862e9204e139ad1eeaa1f4ea011257a27a_2_660x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd5c35862e9204e139ad1eeaa1f4ea011257a27a_2_990x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd5c35862e9204e139ad1eeaa1f4ea011257a27a_2_1320x1000.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-11-01 at 2.24.53 PM</span><span class="informations">1736×1314 78.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried this:</p>
<pre><code class="lang-auto">class ConvNet1DPaper(nn.Module):
    def __init__(self, num_classes=7):
        super(ConvNet1DPaper, self).__init__()
        self.cnn = nn.Sequential(
            nn.Conv1d(1, 80, kernel_size=(100, 1, 1), stride=5),
            nn.AvgPool1d(kernel_size=(3, 1, 1), stride=2),
            nn.Conv1d(1, 80, kernel_size=(50, 1, 80), stride=5),
            nn.AvgPool1d(kernel_size=(3, 1, 1), stride=1),
            nn.Conv1d(1, 80, kernel_size=(25, 1, 80), stride=2),
            nn.AvgPool1d(kernel_size=(3, 1, 1), stride=1),
            nn.Flatten(1)
        )
        self.fc = nn.Sequential(
            nn.Linear(880, 700),
            nn.Linear(700, 70),
            nn.Linear(70, num_classes))

    def forward(self, x):
        out = self.cnn(x)
        out = self.fc(out)
        return out
</code></pre>
<p>But no dice.</p>
Does L2 present in Adam optiimizer?

theGuyWithBlackTie: <p>In PyTorch implementation of Adam optimizer, does weight_decay means it is L2 regularization? PyTorch <a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam" rel="noopener nofollow ugc">documentation</a> says weight_decay is L2.</p>
<p>But I read few articles and blog stating that weight_decay is L2 regularization only for vanilla SGD and weight_decay signifies something different in Adam. Is that true? <a href="https://openreview.net/pdf?id=rk6qdGgCZ" rel="noopener nofollow ugc">https://openreview.net/pdf?id=rk6qdGgCZ</a></p>
Implementation of gradient-enhanced neural network

alfiyandyhr: <p>Hi everyone! I would like to implement a gradient-enhanced neural network (GENN) using pytorch.<br>
I am quite new to pytorch, so don’t hesitate to point me out the obvious solutions.</p>
<p>The idea of the gradient-enhanced neural network is to train the network by adding the gradient error in the loss function. This is possible when the gradient information of the function is available or cheap-to-compute. Here is the snippet of a paper [<a href="https://link.springer.com/article/10.1007/s00158-020-02488-5" class="inline-onebox" rel="noopener nofollow ugc">Scalable gradient–enhanced artificial neural networks for airfoil shape design in the subsonic and transonic regimes | Structural and Multidisciplinary Optimization</a> ]that I just read, and I am interested to implement it in pytorch.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cce4cbc85b048c6580dde2e2877450f25ca2ea38.png" alt="genn" data-base62-sha1="tezB1CCYxr703Qc1MRfcdQmdvRK" width="420" height="398"></p>
<p>Here is the quick code that I wrote: (only the training part)</p>
<p>for epoch in range(1000):<br>
training_loss = 0.0<br>
for batch_no, data in enumerate(trainLoader):<br>
x, f, df_dx = data[“x”], data[“f(x)”], data[“df(x)/dx”]<br>
x.requires_grad = True<br>
f.requires_grad = True<br>
df_dx.requires_grad = True<br>
optimizer.zero_grad()<br>
pred = model(x)<br>
pred.retain_grad()<br>
pred.sum().backward()<br>
# print(x.grad)<br>
# print(df_dx)<br>
loss = loss_fn(pred, f) + loss_fn(x.grad, df_dx)<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>	training_loss += loss.item()

print(f'epoch = {epoch}, training_loss = {training_loss}')
</code></pre>
<p>torch.save(model, ‘genn_model.pth’)<br>
torch.load(‘genn_model.pth’)</p>
<p>The above code gave the following errors:<br>
<code>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</code></p>
<p>I think the core idea to implement GENN is to use the autograd backward engine.</p>
<p>I appreciate if anyone has any thoughts on it :))</p>
Pattern Segmentation using U-Net - Validation loss increasing

Abhishek_K: <p>Hello All, I am trying to do a segmentation for a pattern recognition. I have only 2 class label ( Masked Image)<br>
Pixel Value - 0. , Class Index - 0 ( Pattern Area , Black Region as shown below)<br>
Pixel Value -1. , Class Index -1  ( Back ground Area)</p>
<p>I have tried training with different parameters , below are my observations:</p>
<ul>
<li>
<p>The model over-fits. The Pixel prediction accuracy in the 1st epoch is high. <strong>As the epoch increases, the validation loss increases &amp; pixel prediction accuracy  decreases. While my training loss decreases continuously</strong></p>
</li>
<li>
<p>Loss Function → CrossEntropy , Model → U Net</p>
</li>
<li>
<p>Training Images ~ 2600 images  &amp; ~600 validation Images</p>
</li>
<li>
<p>I have tried to train with small image size(64x64), the training trend does not change</p>
</li>
<li>
<p>I have tried to give a weight decay of 2 , I get only one class prediction ie., the back ground class pixel prediction all over the image</p>
</li>
</ul>
<p>Masked Image looks something as shown below. <img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b72deb0c5095ba6eb8e4df9cebe937d56d20ded.png" alt="Sample" data-base62-sha1="aLrUUjK7xMJLQDfiWGzztf5sstD" width="300" height="251"></p>
<p>I have trained the model for Max of 10 Epochs</p>
<p>Kindly let me know what would be the likely issue for this. Also any suggestions to get the validation error decreasing.</p>
<p><strong>Note : I want to use AI for this, not any openCV techniques</strong></p>
<p>Thanks in advance</p>
N minimum tensors in terms of absolute value

yegane: <p>Is there a pythonic way of zeroing 10 smallest tensors(in terms of absolute value)?</p>

ayalaa2: <p>Yes, you’ll basically want to get access to the indices that will sort the tensor, choose the 10 lowest, and then use those to index into your tensor and zero them out.</p>
<pre><code class="lang-auto">x = torch.randn(100)

# Get the indices that would sort x (in terms of abs value)
indices = torch.argsort(x.abs())

# choose the first 10 
indices_smallest = indices[:10]

# Zero those positions out
x[indices] = 0.0

</code></pre>

yegane: <p>Weirdly enought. It causes : <code>"Unable to get repr for &lt;class 'torch.Tensor'&gt;"</code><br>
Is there a way to do this action on a specific dimension individually?<br>
e.g. if x=[6,3,10,10] we do it individually on each [3,10,10]</p>

Hamster: <p><code>torch.topk</code> (<a href="https://pytorch.org/docs/stable/generated/torch.topk.html" class="inline-onebox" rel="noopener nofollow ugc">torch.topk — PyTorch 1.10.0 documentation</a>) should get you the k smallest elements along with their indices if you use the argument <code>largest=False</code>. Should be much faster than sorting the whole Tensor. You can then use those indices to do what you want.</p>

yegane: <p>Thaks. That’s the function, but I don’t know how to use these indeces for new assignment.<br>
I tried scatter_ and np.ind=np.unravel_index, but they are not working.</p>
Why is PyTorch's Dataloader is not inerrable?

GamarChoba: <p>I am working on the MNIST dataset for an assignment and it seems to me I am stuck at some point for long. I have written my code for LogisticRegression and when I try to train the model it is not working as expected but instead, it throws an error that says:</p>
<pre><code class="lang-auto">Input tensor should be a torch tensor. Got &lt;class 'PIL.Image.Image'&gt;.
</code></pre>
<p>Dataloader:</p>
<pre><code class="lang-auto">train_dataloader = torch.utils.data.DataLoader(mnist_train, batch_size, shuffle=True)
</code></pre>
<p>My Code:</p>
<pre><code class="lang-auto">class LogisticRegression(torch.nn.Module):

def __init__(self, input_dim, output_dim):
    super(LogisticRegression, self).__init__()
    self.linear = torch.nn.Linear(input_dim, output_dim)
    

def forward(self, x):
    x = x.reshape(-1, 784)
    outputs = self.linear(x)
    return outputs
</code></pre>
<p>Train the model:</p>
<pre><code class="lang-auto">for epoch in range(epochs):
   for i, (images, labels) in enumerate(train_dataloader):
   optimizer.zero_grad()

   y_pred = model(images)

   loss = criterion(y_pred, labels)

   loss.backward()
    
   optimizer.step()
</code></pre>
<p>Where am I going wrong with the code?</p>

ptrblck: <p>As the error message describes, you are passing <code>PIL.Image</code>s to the model, while a tensor is expected.<br>
Add the <code>transforms.ToTensor()</code> transformation to your dataset and it should work.</p>

GamarChoba: <p>Yes, it is working thanks.</p>