akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions

KimberleyJensen: <p>Since updating to PyTorch 2.0.0 i had to add return_complex=True to my code but now it causes this error<br>
“RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 3 is not equal to len(dims) = 4”</p>
<p>Please does anyone know what code i need to change and what to change it to? (without going to an older version of PyTorch).</p>
<p>Code:</p>
<blockquote>
<p>dim_s = 4</p>
<p>class STFT:<br>
def <strong>init</strong>(self, n_fft, hop_length, dim_f):<br>
self.n_fft = n_fft<br>
self.hop_length = hop_length<br>
self.window = torch.hann_window(window_length=n_fft, periodic=True)<br>
self.dim_f = dim_f</p>
<pre><code>def __call__(self, x):
    window = self.window.to(x.device)
    batch_dims = x.shape[:-2]
    c, t = x.shape[-2:]
    x = x.reshape([-1, t])
    x = torch.stft(x, n_fft=self.n_fft, hop_length=self.hop_length, window=window, center=True, return_complex=True)
    x = x.permute([0,3,1,2])
    x = x.reshape([*batch_dims,c,2,-1,x.shape[-1]]).reshape([*batch_dims,c*2,-1,x.shape[-1]])
    return x[...,:self.dim_f,:]

def inverse(self, x):
    window = self.window.to(x.device)
    batch_dims = x.shape[:-3]
    c,f,t = x.shape[-3:]
    n = self.n_fft//2+1
    f_pad = torch.zeros([*batch_dims,c,n-f,t]).to(x.device)
    x = torch.cat([x, f_pad], -2)
    x = x.reshape([*batch_dims,c//2,2,n,t]).reshape([-1,2,n,t])
    x = x.permute([0,2,3,1])
    x = torch.istft(x, n_fft=self.n_fft, hop_length=self.hop_length, window=window, center=True, return_complex=True)
    x = x.reshape([*batch_dims,2,-1])
    return x
</code></pre>
</blockquote>

ptrblck: <p>I can reproduce the mentioned error using:</p>
<pre><code class="lang-python">module = STFT(10, 10, 10)
x = torch.randn(1, 10, 10)
</code></pre>
<p>Based on the <a href="https://pytorch.org/docs/stable/generated/torch.stft.html#torch.stft">docs</a> it seems you could restore the 4-dimensional tensor via <code>torch.view_as_real</code> which also seems to fix your error:</p>
<pre><code class="lang-python">    def __call__(self, x):
        window = self.window.to(x.device)
        batch_dims = x.shape[:-2]
        c, t = x.shape[-2:]
        x = x.reshape([-1, t])
        x = torch.stft(x, n_fft=self.n_fft, hop_length=self.hop_length, window=window, center=True, return_complex=True)
        print(x.shape)
        x = torch.view_as_real(x)
        print(x.shape)
        x = x.permute([0,3,1,2])
        x = x.reshape([*batch_dims,c,2,-1,x.shape[-1]]).reshape([*batch_dims,c*2,-1,x.shape[-1]])
        return x[...,:self.dim_f,:]
</code></pre>
Fully connected layer confusion

brunoj: <p>Hello.<br>
I have to do transfer learning and instead of changing my custom number of classes taht is 6 in the last layer I used this method. But what I feel like I did wrong is not used softmax.<br>
Placing relu in 2nd last position is correct? I feel like i needed to have softmax here does it impact on accuracy and stuff?<br>
My problem is multi class problem. I used crossEntropyLoss which on reading I think uses softmax at the backend. so do i need to worry???</p>
<p>model.classifier = nn.Sequential(<br>
nn.Linear(in_features=1792, out_features=625), <span class="hashtag">#1792</span> is the orginal in_features<br>
nn.ReLU(), <span class="hashtag">#ReLu</span> to be the activation function<br>
nn.Dropout(p=0.3),<br>
nn.Linear(in_features=625, out_features=256),<br>
nn.ReLU(),  // this part<br>
nn.Linear(in_features=256, out_features=6)<br>
)</p>

eqy: <p>As you have observed <code>CrossEntropyLoss</code> includes a softmax: “ Note that this case is equivalent to the combination of <a href="https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a>" as stated in the docs. Adding another softmax layer would be redundant and strange. You would only expect to see a softmax here in the forward pass if <code>NLLoss</code> was used instead of <code>CrossEntropyLoss</code>.</p>
<p>Yes, adding a ReLU before the final linear can be considered a standard practice. However, I would also compare the performance of just going directly from the 1792 features to the number of output classes and check if that causes accuracy degradation vs. the current approach.</p>

brunoj: <p>well When i do with only nn.Linear(in_features=1792, out_features=6) this way it gives bit less accuray like 86% and with the above multiple method it gave 89% accuracy.<br>
So you mean i do not need to worry about relu() before the classifier?</p>

eqy: <p>ReLU before the classifier is fine (it would be strange to add it after the classifier), but I would experiment to see what gives the best results.</p>
<p>I would also take a look at some reference model implementations e.g., <a href="https://pytorch.org/vision/main/_modules/torchvision/models/resnet.html#resnet18" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models.resnet — Torchvision main documentation</a> for more ideas.</p>

brunoj: <p>So you think what I did is correct? right? I need this answer because I have to move to other steps then.</p>
Why multiple threads are spawned even with num_workers=0 in pytorch dataloader?

kgautam01: <p>Versions:<br>
torch: 1.6.0<br>
torchvision: 0.7.0</p>
<p>Created a custom dataloader to apply custom augmentations on the image and transform it to a tensor. Even without enabling custom augmentations and setting num_workers=0 in the Dataloader object (used default way as well by not setting anything), it is still consuming around 15-20 threads (not consuming the whole thread but 9% of it). I am not sure what is causing this issue as I want to keep the resources free to be used by compute intensive processes of other users.</p>
<p>Tried:</p>
<ol>
<li>Setting OMP and other env variables to restrict the use of threads, using os.environ module in Python</li>
<li>Tried setting combinations of num_workers and pin_memory params</li>
<li>Optimized the code to avoid copying ops to CPU and using numpy for calculation of metrics there.</li>
<li>I had installed torch using conda at the time of env creation but installed torch using pip due to some issues later. Can that cause an issue?</li>
</ol>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>: Can you help me out in understanding as to what is actually causing this or does the API has some internal operations which are segregated over the threads, by default?!</p>

eqy: <p>I would check what operations are actually using multiple threads—you could do this by moving the dataloader operations outside of a dataloader and seeing if they are still using multiple threads. This would help rule out whether the problem is with the dataloader or the operations it is performing.</p>

kgautam01: <pre><code class="lang-auto">self.transform = transforms.Compose([transforms.ToTensor()])
image = self.transform(image) 
label = torch.tensor(0) if class_=='normal' else torch.tensor(1)
return image, label
</code></pre>
<p>This is the only operation getting performed inside the dataloader.</p>

eqy: <p>Note that <code>ToTensor()</code> can call a few operations under the hood such as rounding, dtype conversion, and memory layout transformation, so I would check if any of those individual operations show as using multiple threads on your system:<br>
<a href="https://pytorch.org/vision/main/_modules/torchvision/transforms/functional.html#to_tensor" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/_modules/torchvision/transforms/functional.html#to_tensor</a></p>
How to check free port number for multi node training

Khawar_Islam: <p>I am training a network on 2 machines each machine consists of two GPUS. I have checked the port number but it changes every time when I restart the process and I am not sure whether I am doing the right or wrong way.</p>
<p><strong>How I can check the free port number?</strong> Is this the correct way?</p>
<p><code>sudo lsof -i :22 | grep LISTEN</code></p>
<pre><code class="lang-auto">sshd    1930    root    3u  IPv4  57569      0t0  TCP *:ssh (LISTEN)
sshd    1930    root    4u  IPv6  57571      0t0  TCP *:ssh (LISTEN)
</code></pre>
[TorchScript] fail to convert list to tuple

jellybelly: <p>Hi, in my <strong>torch.jit.script</strong> I would like to <strong>convert a list to a tuple</strong>, but it seems not to be the easy way.</p>
<p>The size of the list is fixed - although not in a way that jit prefers:  which seems to be the problem<br>
On <strong>Python 3.8</strong> with <strong>PyTorch 1.9.1</strong></p>
<pre><code class="lang-auto">result = []
val = torch.full((1,1),1.)
result.append(val) # size() = 1
return tuple(result)
</code></pre>
<p>… fails with Runtimeerror: cannot statically infer the expected size of a list in this context:</p>
<p>same as</p>
<pre><code class="lang-auto">val = torch.full((1,1),1.)
result:List[Tensor] = val * 1 
return tuple(result)
</code></pre>
<p>…fails with Runtimeerror: cannot statically infer the expected size of a list in this context:</p>
<p>Can someone please tell me how to do this in torch.jit.script (not trace!)<br>
Thx</p>

googlebot: <p>as error suggests, tuple sizes are static(hardcoded) when “compiled”, so you have to do something like<br>
return (result[0],result[1])<br>
and there may be a need to annotate this as Tuple[Tensor,Tensor] in some contexts</p>

jellybelly: <p>Ok, that’s meant by <strong>“statically inferred”</strong>. Coming from the <strong>C++</strong> side, I would have expected some counterpart of C++</p>
<pre><code class="lang-auto">result[4] = {val}
retval = tuple(result)
</code></pre>
<p>like<br>
<code>result:List[Tensor,4] = val * 4</code></p>
<p>Thank you for your help</p>

googlebot: <p>if you’re comparing to C++, List is like a <code>vector&lt;Tensor&gt;</code> of dynamic size, hence you cannot initialize specialized N-tuples from it, without runtime checks and metaprogramming anyway.<br>
Though I agree that TorchScript is lacking in some aspects, in this case the workaround is to return a list.</p>

rtviii: <p>this still fails for me:</p>
<pre data-code-wrap="python3"><code class="lang-plaintext">from torch import Tensor
import torch

def some_rotation_op(rotation:Tensor):
  [Rxz, Ryz, Rzz] = rotation[:, -1]  # &lt;------ whether this is a list makes no difference
  ...
  return Rxz

torch.jit.script(some_rotation_op, (torch.rand( 3, 3),))
</code></pre>
We build a chatbot of Pytorch on top of pytorch

xiaofan1: <p>Hi everyone,</p>
<p>We’re excited to announce the launch of osschat(<a href="https://osschat.io/" rel="noopener nofollow ugc">https://osschat.io/</a>), which is free to use for the entire PyTorch community! Our chatbot can assist you with various queries, provide information about popular open-source projects on GitHub, and answer any questions you may want to know about PyTorch.</p>
<p>We would love to hear your feedback and suggestions on how we can improve our chatbot. Please feel free to share your thoughts with us on  social media platforms such as Twitter, LinkedIn, or direct leave message here.</p>
<p>Best regards,</p>
<p>Osschat Team</p>
Best way to implement series of weighted random sampling (for transition w/ stochastic matrix)?

kmaeng: <p>Dear experts,</p>
<p>I am trying to find the best way to perform a series of weighted random sampling, each from a different distribution. I will try to elaborate on what I intend to do (Maybe if you just jump to the code it will be immediately clear).</p>
<p>I have an input <code>x</code> of length <code>L</code>. Each element is an integer between 0 and <code>N</code>.<br>
I want to generate an output <code>y</code> of length <code>L</code>, where each element of y is an integer between 0 and <code>M</code>.</p>
<p>Each element <code>y[i]</code> is transitioned from the corresponding element <code>x[i]</code>, with a certain transition probability distribution defined by a <code>N</code>x<code>M</code> stochastic matrix <code>p</code>.</p>
<p><code>p[i][j]</code> denotes the probability of <code>y</code> becoming <code>j</code>, if <code>x</code> is <code>i</code>.</p>
<p>Basically, if <code>N</code>==<code>M</code>, I am trying to do a Markov chain transition.</p>
<p>Below is a (very slow) code I implemented.</p>
<pre><code class="lang-auto">import torch
from torch import nn

N = 10
M = 20
L = 5

p = torch.rand([N, M])

x = torch.randint(0, N, [L])
print(x)
y = torch.zeros([L])

for i in range(L):
    y[i] = list(torch.utils.data.WeightedRandomSampler(p[x[i]], 1))[0]
print(y)
</code></pre>
<p>Here, I am not normalizing <code>p</code> row-wise (i.e., it is not a stochastic matrix) because <code>WeightedRandomSampler</code> does not need it to be normalized row-wise.<br>
The problem with this code is that I am using a <code>for</code> loop, and this becomes too slow for large <code>L</code>.<br>
I wonder if there is a way to implement the same thing faster, without having the <code>for</code> loop.</p>
<p>Any insights would be highly appreciated.</p>
<p>Thank you.</p>

kmaeng: <p>Actually, I figured it out. There was a function called torch.multinomial(), which seems to be doing exactly what I wanted to do.</p>
Dataloader collate function scaling

lkp411: <p>Hi there,</p>
<p>I have a custom dataloader that does a lot of processing (I’ll exclude the details of the processing for now), and I’m trying to figure out why it isn’t scaling with the number of workers.</p>
<ol>
<li>
<p>I timed my collate function and when I go from 4 to 8 to 16 workers the time each worker takes to execute the collate function is pretty much the same. But when I go from 16 to 24 workers, the time each worker takes starts to increase. I am not reading anything from disk in my dataloader, I am only reading from memory and performing computations. What are the possible reasons for why this is happening? Am I hitting my CPU memory bandwidth? Or my CPU IPS?</p>
</li>
<li>
<p>In the case of 8 and 16 workers, even though each worker is taking the same amount of time, the time spent waiting for a batch in my main training loop is still the same despite having twice the number of workers. What is the possible explanation for this? Shouldn’t I have twice as many batches in memory in the same amount of time, meaning roughly half the waiting time to load a batch in the main training loop?</p>
</li>
</ol>
<p>The node I’m running on has 96 logical CPUs, 48 physical cores and 500GB of DRAM, so I am not anywhere close to the limit on either of those fronts.</p>
What's the difference in gradient backprop between slicing and nn.Unfold?

jfzleo: <p>My code used to contain loop slicing which was slow. Recently I changed it into nn.Unfold which computes the same reult. However, I found that my model converged into a worse status, whose evaluation metrics  deteriorates. After inspection, I found that although they output the same result, the gradiet produced by back propagation are different. What should I do? My envirnment is python==3.6.10, torch==1.4.0. Here is a simple code snippet reproducing the difference:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F

class simple_net(nn.Module):
    def __init__(self):
        super(simple_net, self).__init__()
        self.conv1 = nn.Conv2d(3, 8, 3, 1, 1)
        self.conv2 = nn.Conv2d(8, 3, 3, 1, 1)
    
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        return x

model = simple_net().cuda()
unfold = nn.Unfold(3, padding=0)
x = torch.randn([6,3,8,8], dtype=torch.float32).cuda()
x = torch.autograd.Variable(x)
result = model(x)
b, c, h, w = result.shape
result_pad = F.pad(result, [1,1,1,1])
result_pad.retain_grad()
result_pad_1 = result_pad.clone()
result_pad_1.retain_grad()
result_unfold = unfold(result_pad)
slice_list = []
for i in range(h):
    for j in range(w):
        slicetmp = result_pad_1[..., i:i+3, j:j+3].reshape(b, -1)
        slice_list.append(slicetmp)
result_slice = torch.stack(slice_list, dim=2)
print("forward result:, ", str(torch.sum(torch.abs(result_unfold - result_slice))))
loss = result_unfold.abs().sum() + result_slice.abs().sum()
loss.backward()
print("gradient result:, ", str(torch.sum(torch.abs(result_pad.grad - result_pad_1.grad))))

</code></pre>
<p>outputs:</p>
<pre><code class="lang-auto">forward result:,  tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)  
gradient result:,  tensor(8712., device='cuda:0') 
</code></pre>
<p>Many thanks in advance!</p>

soulitzer: <p>The gradient results may be different due to how you are testing: result_pad_1 is a function of result_pad so when you compute the gradient wrt result_pad you’ll accumulate into the .grad of result_pad_1. Do you observe that one gradient is twice the value of the other?</p>
<p>(btw is there a reason why you haven’t upgraded your pytorch version?)</p>

soulitzer: <p>If you’d like to know whether the two outputs have the same value, computing the sum is one heuristic, but you can also use <code>torch.allclose(a, b)</code></p>
<p>If you’d like to compute whether two functions compute the same gradient. You should use <code>grad_inp = torch.autograd.grad(output, inp)</code> to compute the gradients of inputs wrt each of the outputs respectively, and then compare them with <code>torch.allclose</code>.</p>
SoftMarginRankingLoss Implementation

eaah: <p>Hi,</p>
<p>I am trying to implement a custom loss function softmarginrankingloss.  The Size of my input vectors is N x C x H x W. (128,64,14,14). It is basically the output of a VGG16 at conv5. However, my implementation gives me the following error due to gradient back propagation.</p>
<p>Error:</p>
<p>File “train.py”, line 431, in <br>
main()<br>
File “train.py”, line 325, in main<br>
loss.backward()<br>
File "/usr/local/lib/python3.8/dist-packages/torch/<em>tensor.py", line 363, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/autograd/<strong>init</strong>.py”, line 166, in backward<br>
grad_tensors</em> = <em>make_grads(tensors, grad_tensors</em>, is_grads_batched=False)<br>
File “/usr/local/lib/python3.8/dist-packages/torch/autograd/<strong>init</strong>.py”, line 67, in _make_grads<br>
raise RuntimeError(“grad can be implicitly created only for scalar outputs”)<br>
RuntimeError: grad can be implicitly created only for scalar outputs</p>
<pre><code class="lang-auto">class SoftMarginRankingLoss(torch.nn.Module):
    def __init__(self, weight=None, size_average=True, alpha =1 ):
        super(SoftMarginRankingLoss, self).__init__()
        
        self.apha = alpha
        
    def forward(self, anchor, positive_match, negative_match):
      
        distance_pos = F.pairwise_distance(anchor, positive_match, keepdim = True)

        distance_neg = F.pairwise_distance(anchor, negative_match, keepdim = True)

        distance = distance_pos - distance_neg
        loss = torch.log(1+torch.exp(self.apha*distance))

        return loss

</code></pre>

AbdulsalamBande: <p>The error you encountered is because the <code>SoftMarginRankingLoss</code> is not returning a scalar value, but instead, it returns a tensor with a shape derived from the input tensors. The <code>backward()</code> function expects a scalar value for the loss. To fix this, you need to aggregate the loss tensor to a scalar value before returning it. You can do this by calculating the mean or sum of the loss tensor.</p>

eaah: <p>Thanks that solved the problem!</p>

AbdulsalamBande: <p>I am glad it solved your problem</p>
Using logistic regression without vectorizing to find the weights of AND gate. My code does not seem to learn. Help me

AbdulsalamBande: <pre><code class="lang-auto">number_of_features = 2
number_of_examples = 4
X = torch.tensor([[1.0,0.0,1.0,0.0],[1.0,0.0,0.0,1.0]])
Y = torch.tensor([[1.0,0.0,0.0,0.0]])
w = torch.Tensor([[0.0],[0.0]])
myW = torch.transpose(w,0,1)
dw1 = torch.Tensor([0.0])
dw2 = torch.Tensor([0.0])
db = torch.Tensor([0.0])
loss = torch.Tensor([0.0])
for epoch in range(10):
    dw1 = 0.0 
    dw2 = 0.0
    db = 0.0
    loss = 0.0 
    for i in range(number_of_examples):
        example = X[:,i].clone()
        myexample = example.resize(2,1)
        z = torch.mm(myW,myexample) 
        a = torch.sigmoid(z)
        loss = loss + (-(Y[0,i]*torch.log(a)+(1-Y[0,i])*torch.log(1-a)))
        dz = a-Y[0,i]
        for j in range(1):
            dw1 = dw1 + (myexample[0,0] * dz)
            dw2 = dw2 + (myexample[1,0] *dz)
            db = db + dz
    loss = loss / number_of_examples
    dw1 = dw1 / number_of_examples
    dw2 = dw2 / number_of_examples
    myW[0,0] = myW[0,0] - (0.001*dw1)
    myW[0,1] = myW[0,1] - (0.001*dw2)
    print("loss")
    print(loss)




</code></pre>

AbdulsalamBande: <p>Coming back to this after almost four years <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">. Below is the solution</p>
<pre><code class="lang-auto">import torch

number_of_features = 2
number_of_examples = 4
X = torch.tensor([[1.0, 0.0, 1.0, 0.0], [1.0, 0.0, 0.0, 1.0]])
Y = torch.tensor([1.0, 0.0, 0.0, 0.0])
w = torch.zeros((number_of_features, 1))
b = torch.zeros(1)
learning_rate = 0.1

for epoch in range(100000):
    dw = torch.zeros_like(w)
    db = 0.0
    loss = 0.0
    
    for i in range(number_of_examples):
        x_i = X[:, i].view(-1, 1)
        y_i = Y[i]
        
        z = torch.mm(w.t(), x_i) + b
        a = torch.sigmoid(z)
        a = torch.sigmoid(z).squeeze()

        
        loss += -(y_i * torch.log(a) + (1 - y_i) * torch.log(1 - a))
        dz = a - y_i
        
        dw += x_i * dz
        db += dz

    loss /= number_of_examples
    dw /= number_of_examples
    db /= number_of_examples

    w -= learning_rate * dw
    b -= learning_rate * db

    if epoch % 1000 == 0:
        print(f"Epoch {epoch}, Loss: {loss.item()}")

print("Weights:")
print(w)
print("Bias:")
print(b)

</code></pre>
Multiprocessing shared memory dramatically slows NN evaluation?

jeremysalwen: <p>I am trying to follow the <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">pytorch multiprocessing docs</a> to set up a basic worker pool of processes playing games for my RL algorithm.  The game playing only requires forwards passes on my neural net, so I don’t need gradients (I’m using torch.no_grad()).</p>
<p>I’m calling model.share_memory() before passing my model to the workers using <code>torch.multiproccessing.Pool.apply_async</code>.  What I find though is that although the model is passed very quickly to the other processes, the forward pass on those other processes becomes <em>much</em> slower, to the point that it is always much slower than a single-process implementation.</p>
<p>I tried profiling my processes to see what was causing the slowdown, but it looks like the basic pytorch neural net forward passes were taking up the vast majority of the time:</p>
<pre><code class="lang-auto"> %Own   %Total  OwnTime  TotalTime  Function (filename:line)                                                                                                                                                      
 37.00%  37.00%   685.1s    685.1s   linear (torch/nn/functional.py:1676)
 22.00%  22.00%   464.5s    464.5s   layer_norm (torch/nn/functional.py:2048)
  4.00%   4.00%   166.2s    166.2s   softmax (torch/nn/functional.py:1498)
 13.00%  13.00%   165.3s    165.3s   multi_head_attention_forward (torch/nn/functional.py:4130)
  4.00%   4.00%   164.9s    164.9s   multi_head_attention_forward (torch/nn/functional.py:4108)
</code></pre>
<p>The slowdown gets worse and worse the more processes I add to the pool. (Overall throughput continues to go down).  I also can tell that the problem isn’t that there is a one time overhead, beacuse just doing a single forward pass in each worker is not that much slower, but doing many forward passes the slowdown becomes extremely noticeable.</p>
<p>I have no idea what could be causing this slowdown.  Is this expected that inference running from shared memory would be dramatically slower?  I have also tried changing the multiprocessing context to “fork” or “spawn” to no avail.</p>

lalejand: <p>Any updates on this? I’m sort of having the same issues</p>

jeremysalwen: <p>Sadly no.  Ultimately what I ended up doing was moving all the NN inference to a single process, which took RPCs from the other processes, and did the inference in a single batch. This way the NN is not shared between processes, so I avoid this slowdown.  However it may not work for your use case.</p>
Implementation of a FIR Filter using nn.Conv1d

anghel: <p>Hello! I would like to implement a FIR Filter using pytorch nn.Conv1d. i have computed the filter’s weights and I just want to compute the convolution using nn.Conv1d but I get some problems. Can anyone help me?</p>
<p>I got an error like: RuntimeError: Input type (CPUComplexFloatType) and weight type (CPUComplexDoubleType) should be the same</p>

ptrblck: <p>The error message points to a <code>dtype</code> mismatch of your input and weight tensor. Make sure both are using the same <code>dtype</code> by transforming one of them.</p>
Slow training, looking for errors / code improvements

setsailforfail: <p>Hi there!<br>
I am currently trying to train a cGAN model from scratch. It works fine, however, the training seems very slow for the hardware I use (4090). To learn the most from this issue, I would like to spot an error in my code first. I am not sure if I can just copy paste the whole model (200 lines of code), so maybe someone experience a similar problem or is quite experienced and can tell me what possibly could be wrong.</p>
<p>For example, currently I am using</p>
<pre><code class="lang-auto">    dataloader = torch.utils.data.DataLoader(
        dataset,
        shuffle=True,
        batch_size=124,
        drop_last=True,
    )
</code></pre>
<p>Adding num_workers = 4, the training is approx 5 times faster, but I am not sure if this is a good thing to change. However, since small changes like that have such a high impact, I wonder what else there is to optimize.</p>

ptrblck: <aside class="quote no-group" data-username="setsailforfail" data-post="1" data-topic="176551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/setsailforfail/48/30110_2.png" class="avatar"> setsailforfail:</div>
<blockquote>
<p>Adding num_workers = 4, the training is approx 5 times faster, but I am not sure if this is a good thing to change.</p>
</blockquote>
</aside>
<p>Using multiprocessing to load and process the data is a good approach and would avoid using the main process for this task (which is also responsible to launch the CUDA kernels etc.).<br>
I’m not sure what concern you have regarding this change.</p>
<aside class="quote no-group" data-username="setsailforfail" data-post="1" data-topic="176551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/setsailforfail/48/30110_2.png" class="avatar"> setsailforfail:</div>
<blockquote>
<p>I wonder what else there is to optimize.</p>
</blockquote>
</aside>
<p>I would recommend checking our <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">Performance Tuning Guide</a> for general tips.</p>

setsailforfail: <p>Thanks for your help! I wonder, how does one determine the best amount of num_workers? is there something like a rule of thumb, or is it simply trial and error.</p>

ptrblck: <p>I think generally users don’t increase the number of workers beyond their CPU cores (and I think higher-level APIs such as <code>Lightning</code> also raise a warning or set <code>num_workers</code> to your CPU count). You could run a few experiments as also other users reported diminishing returns for <code>num_workers&gt;4</code> but as usual it would all depend on your system and workload.</p>
Does the number of parameters of a network change as the amount of data changes?

peony: <p>I used the following code to count all the trainable parameters in my models:</p>
<pre><code class="lang-auto">  print("numparams")
  print (sum(p.numel() for p in model.parameters() if p.requires_grad))
</code></pre>
<p>and I notice as I change the number of classes in my dataset, the number of trainable parameters changes too.  From 7351300 for 4 classes to 7354378 for 10 classes.</p>
<p>I read some blogs and thought that parameters only involved the number of layers and neurons. Is this not true?</p>
<p>Thank you in advance</p>

sinatv52: <p>When you change the number of classes, the last layer changes so the number of trainable parameters.<br>
just assume that the last layer of your model is a simple MLP. When you change  the number of classes the number of trainable parameters change.</p>

peony: <p>Thank you for explaining. Is 7 million parameters normal for CNN plus LSTM architecture?</p>

sinatv52: <p>Yes it is. But totally depends on your chosen architecture. For example VGG has approximatley135M and Resnet-18 has 11.5M parameters.</p>

nwn: <p><a class="mention" href="/u/peony">@peony</a> Generally, it does not change, as long as you do not change the architecture of your ANN. For a “normal” feed forward model, you have the weights and the bias per layer. Keep in mind that the optimizer and batch normlalization layers can include paramters that will be learned, too.<br>
In your given example don’t forget that the last layer is connected with weights with the output neurons as well, which explaines why the number of paramters increases if you change the number of classes from 4 to 10.<br>
Let’s say your last layer has 100 neurons and then you have either 4 or 10 outputs neurons (number of classes), you would end up with 100<em>4 vs. 100</em>10 weights → 400 vs. 1000 trainable paramters.</p>
Implementation SFA in Pytorch

bryan123: <p>Could someone help me to implement this code in pytorch where the inputs are (batch_size, M, 192) and two classes</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d0e3a11f99ac507dcc6e08f7c72f310755de71c.png" alt="image" data-base62-sha1="k7PNRj7yht0AIxJwRyFj2fE3CsQ" width="545" height="429"></p>
Can anyone help me solve it?

Bhargav_P_Raj: <p>RuntimeError: cuDNN version incompatibility: PyTorch was compiled  against (8, 5, 0) but found runtime version (8, 4, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.Looks like your LD_LIBRARY_PATH contains incompatible version of cudnnPlease either remove it from the path or install cudnn (8, 5, 0)</p>

Bhargav_P_Raj: <p>I’m unable to find the $LD_LIBRARY_PATH and remove multiple path appended.</p>

ptrblck: <p>Which PyTorch release are you using as I believe this error should not be raised anymore?<br>
You can access <code>$LD_LIBRARY_PATH</code> in your terminal and can print it via <code>echo $LD_LIBRARY_PATH</code>.</p>

rs_l: <p>You can use this command in you terminal-----&gt; unset LD_LIBRARY_PATH</p>
My model outputs same values after loading the checkpoint

chickenrushkr: <p>Hi. I’m implementing C-Net model from <a href="https://github.com/microsoft/Deep3DFaceReconstruction" class="inline-onebox" rel="noopener nofollow ugc">GitHub - microsoft/Deep3DFaceReconstruction: Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019)</a>.<br>
In that paper, C-Net is a CNN based model.<br>
In the process of implementing this, when training the model, Confidence was output differently for each image in the image set.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/868e23071597218e14857c18a8a6fed0d30b6921.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/868e23071597218e14857c18a8a6fed0d30b6921" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/868e23071597218e14857c18a8a6fed0d30b6921_2_689x290.jpeg" alt="image" data-base62-sha1="jckvtVUMi9TWIEDNDwa5cpYnWIF" width="689" height="290" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/868e23071597218e14857c18a8a6fed0d30b6921_2_689x290.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/868e23071597218e14857c18a8a6fed0d30b6921_2_1033x435.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/6/868e23071597218e14857c18a8a6fed0d30b6921.jpeg 2x" data-dominant-color="F8F2EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1268×533 89 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<strong>Right figure is histogram for every image-set.</strong><br>
The dimension of the output of C-Net is (N, 100), where N is the number of images in the image set.</p>
<p>I saved the model checkpoint using :<br>
checkpoint_path = f’./weights/{args.name}_{epoch}.tar’<br>
torch.save({<br>
‘epoch’: epoch,<br>
‘model_state_dict’: C_Net.state_dict(),<br>
‘optimizer_state_dict’: optimizer.state_dict(),<br>
‘loss’: loss,<br>
}, checkpoint_path)<br>
When I loaded the checkpoint to train the model further and/or evaluate it on the test set, it outputs the same value for all images. <strong>(Left figure in image)</strong><br>
(Model.eval() was used to evaluate the model, and model.train() was used for further training.)<br>
Output for all images, with half of the 100 dimensions being 0 and the rest being 1. (Sigmoid used)<br>
There is no BN layer or dropout in C-Net.<br>
I made sure the weights and biases are loaded correctly after the checkpoint load.</p>
<p>Does anyone have a similar issue or know the cause?<br>
If anyone knows, please help.<br>
Thank you.</p>
TypeError: .__init__() takes 1 positional argument but 3 were given

ShaRinGan: <p>I am practicing pytorch  nowdays, actually shifting from keras.</p>
<p><strong>But I can not understand why I am getting this error:</strong><br>
TypeError: NeuralNet02.<strong>init</strong>() takes 1 positional argument but 3 were given</p>
<p><strong>Here’s my code:</strong></p>
<p>‘’'class NeuralNet02(nn.Module):<br>
def <strong>int</strong>(self, in_size, out_size, num_hidden_layer=10, hidden_layer_size=128):<br>
super(NeuralNet02,self).<strong>int</strong>()</p>
<pre><code>    self.in_size = in_size
    self.out_size = out_size
    self.num_hidden_layer = num_hidden_layer
    self.hidden_layer_size = hidden_layer_size

    # In PyTorch, the terms "module" and "layer" are often used interchangeably to refer to a
    # building block of a neural network that performs a specific computation on the input data.

    self.layers = nn.Sequential()  # initialization

    for i in range(num_hidden_layer):
        self.layers.add_module(f'fc{i}', nn.Linear(in_size, hidden_layer_size))   # adding layers/modules
        self.layers.add_module('activation', nn.ReLU())
        in_size = hidden_layer_size
    self.layers.add_module('classifier', nn.Sigmoid(hidden_layer_size, out_size))

def forward(self,inputs):
    out = self.layers(inputs)
    return out
</code></pre>
<p>x = torch.randn(8,14)<br>
model = NeuralNet02(14,4)<br>
output = model(x)<br>
print(output)‘’’</p>

ptrblck: <p>You have a typo in your code and are defining <code>__int__</code> instead of <code>__init__</code> (note the missing <code>i</code>).<br>
Note that you can post code snippets by wrapping them into three backticks ```, which makes debugging easier as your current format is also broken.</p>
Too many values to unpack (expected 2) : pytorch classification

xtinct: <aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1MDFd2jdzi3ao6pqnWQy2pGfgyyGmRibi?usp=share_link">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/bf1b3bb999539202800c69ad9a15575e/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1MDFd2jdzi3ao6pqnWQy2pGfgyyGmRibi?usp=share_link" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1MDFd2jdzi3ao6pqnWQy2pGfgyyGmRibi?usp=share_link" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Can anyone please help me with this? I am not sure what is causing the error, I tried many ways but still the same error occurs. Thanks</p>

ptrblck: <p><code>x,y= dataloaders['train']</code> fails since you are indexing the <code>dict</code> to return the training <code>DataLoader</code> while assuming samples will be returned.<br>
The samples (data and target) will be returned in the corresponding loop a few lines below:</p>
<pre><code class="lang-python">for inputs, labels in dataloaders[phase]:
    ...
</code></pre>

xtinct: <p>Thank you ptrblck, I understand what’s the problem here now.<br>
So, what change do you suggest I should do to my code?</p>

ptrblck: <p>Remove <code>x,y= dataloaders['train']</code> as it’s wrong and neither <code>x</code> nor <code>y</code> are used afterwards.</p>
How will randomness of Gumbel-Softmax be reflected during evaluation?

JimW: <p>I am very new to reinforcement learning and gumbel-softmax.<br>
What I am currently doing is to train an LSTM network to generate a policy to achieve good tradeoff between two tasks:  i.e.  final  accuracy and total computation steps.</p>
<p>Ideally,  I hope to achieve the highest final accuracy, but also with as few steps as possible. My initial experiments look promising:<br>
(1)  If I only optimize the “final accuracy loss”,  the system learns to use the most steps and achieved the highest accuracy like 99%<br>
(2)  If I only optimize the “computation steps loss”, the system learns to use 0  steps and achieved the lowest accuracy like 1%<br>
Now I combined this 2 losses in a naive way, and the training curves experienced a lot of fluctuation.<br>
I can identify an epoch that achieved a great tradeoff (i.e. 91% accuracy with very few steps).  But after I downloaded this checkpoint and reloaded it back to pytorch, the result is actually not that good, because:<br>
(1)  With torch.manual_seed(0),  if I tested this checkpoint, it always gave me 83%, instead of 91%.<br>
(2)  If I repeat the validation epoch multiple times with this same checkpoint,  the  accuracy changed like this  0.83, 0.67, 0.59, 0.63 …<br>
(3)  If I insert torch.manual_seed(0) to the beginning of each iteration of this for loop of validation,  the accuracy became 0.62, 0.62, 0.62, 0.62, 0.62 … fixed to this value.</p>
<p>Could someone give me some comments on the above observation?  Is this something normal?  Did something look wrong or strange in my experiments?   Thanks in advance.</p>
Romc Linux Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

wang_fei: <p>I use 6700xt, the ROCM version is the latest 5.4.3, PyTorch is 0.15.1+ROCM5.4.2, and the official instance Fashionmnist reports an error.</p>
<p>/home/ubuntu/PycharmProjects/pytorch/venv/bin/python</p>
<p>/home/ubuntu/PycharmProjects/FashionMNIST/main.py</p>
<p>Shape of X [N, C, H, W]: torch.Size([64, 1, 28, 28])</p>
<p>Shape of y: torch.Size([64]) torch.int64</p>
<p>Using cuda device</p>
<p>Epoch 1</p>
<hr>
<p>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)<br>
Who can help me wo solve this issue?</p>

AbdulsalamBande: <p>It seems you are encountering a segmentation fault (SIGSEGV) error, which can be caused by various reasons, such as memory access issues, incorrect library versions, or problems with the underlying system. To troubleshoot this issue, consider the following steps:</p>
<ol>
<li>Update your PyTorch version: You mentioned that you are using PyTorch 0.15.1+ROCM5.4.2, which is an older version. Update your PyTorch installation to the latest version compatible with ROCm by following the official installation guide for ROCm: <a href="https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learning.html#pytorch" class="inline-onebox" rel="noopener nofollow ugc">Deep Learning — ROCm 4.5.0 documentation</a>
</li>
<li>Check for GPU compatibility: Ensure that your AMD Radeon RX 6700 XT is compatible with the ROCm version you are using.</li>
<li>Run a ROCm validation test: Make sure your ROCm installation is working correctly. You can do this by running the following command in your terminal:</li>
</ol>
<pre><code class="lang-auto">/opt/rocm/bin/rocminfo
</code></pre>

wang_fei: <p>Thank you! I have solved this issue.</p>

wang_fei: <p>I forgot to add the information below.<br>
export HSA_OVERRIDE_GFX_VERSION=10.3.0 # RX6700xt<br>
export LD_LIBRARY_PATH=/opt/rocm/lib</p>
Concat two models with differents size input

youb: <p>HI,<br>
I have a dataset of videos (60); from this videos I extract images and audio,<br>
from each video I extract 30 images, so in total 1800 images, the shape of x_train and y_train is<br>
x_train.shape =&gt; (1800,224,224,3),<br>
y_train.shape =&gt; (1800,5)</p>
<p>From audio i extract 15 signal (array of numbers), so in total 60*15=900,  the shape of x_train and y_train is<br>
x_train.shape =&gt; (900, 128)<br>
y_train.shape =&gt; (900,5)</p>
<p>images are fed into VggFace (fine tuned ) model(1); the output shape is (1800,128)<br>
audios are fed into a Vggish model (2), the output shape is (900, 128)</p>
<p>after training two model (1 and 2)<br>
Both models are used as input for third model3; The problem I faced is that:</p>
<p>model3.fit([x_audio_train, x_video_train], y_train, …)<br>
I got the error:</p>
<p>the input size should be the same.</p>
<p>I hope you got my issue;</p>
<p>How can I fix this ?<br>
Thank you</p>

ptrblck: <aside class="quote no-group" data-username="youb" data-post="1" data-topic="176585">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/youb/48/58844_2.png" class="avatar"> youb:</div>
<blockquote>
<p>images are fed into VggFace (fine tuned ) model(1); the output shape is (1800,128)<br>
audios are fed into a Vggish model (2), the output shape is (900, 128)</p>
</blockquote>
</aside>
<p>This sounds wrong as it seems you are increasing the number of samples in the output of <code>VggFace</code> assuming <code>dim0</code> represents the batch dimension.</p>
<aside class="quote no-group" data-username="youb" data-post="1" data-topic="176585">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/youb/48/58844_2.png" class="avatar"> youb:</div>
<blockquote>
<p>The problem I faced is that:</p>
<p>model3.fit([x_audio_train, x_video_train], y_train, …)<br>
I got the error:</p>
<p>the input size should be the same.</p>
</blockquote>
</aside>
<p>I don’t know what <code>model3.fit</code> defines but I assume it’s trying to stack the input tensors somehow.<br>
Could you post the <code>fit</code> source code as I assume it’s coming from a higher-level API?</p>
I am getting accuracy '0.0' each time. Why ? Thankyou in advance

upendra: <p>class tabular_model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.input = nn.Linear(13, 250)<br>
self.hidden1 = nn.Linear(250, 250)<br>
self.hidden2 = nn.Linear(250, 500)<br>
self.output = nn.Linear(500, 1)<br>
self.relu = nn.ReLU()<br>
self.dropout = nn.Dropout(p=0.25)</p>
<p>def forward(self, x):<br>
x = self.relu(self.input(x))<br>
x = self.dropout(x)<br>
x = self.relu(self.hidden1(x))<br>
x = self.dropout(x)<br>
x = self.relu(self.hidden2(x))<br>
x = self.dropout(x)<br>
x = self.output(x)<br>
return x</p>
<p>net = tabular_model()<br>
optimizer = torch.optim.SGD(net.parameters(), lr=0.01, momentum = 0.9)<br>
loss_func = nn.MSELoss()</p>
<p>def functiontotrain():</p>
<p>theBestModel = {“Accuracy”:0, “net”:None}</p>
<p>losses = np.zeros(numepochs)<br>
trainAcc = []<br>
testAcc = []</p>
<p>for epochi in range(numepochs):</p>
<pre><code>print(epochi, end=', ')

# net.to(device)
net.train()

for X,y in train_dataloaders:

  X = X.to(device)
  y = y.to(device)

  batchLoss = []
  batchAcc = []

  yHat = net(X)
  loss = loss_func(yHat, y)

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

  batchLoss.append(loss.item())
  batchAcc = (100*torch.mean((yHat == y).float()).item())

trainAcc.append(np.mean(batchAcc))

losses[epochi] = np.mean(batchLoss)

if trainAcc[-1] &gt; theBestModel['Accuracy']:
  theBestModel['net'] = trainAcc[-1].item()

  theBestModel['net'] = copy.deepcopy(net.state_dict())
</code></pre>
<p>return trainAcc, losses, theBestModel</p>

ptrblck: <p>Based on your code it seems you are comparing floating point values directly:</p>
<pre><code class="lang-python">batchAcc = (100*torch.mean((yHat == y).float()).item())
</code></pre>
<p>which will often show mismatches due to the limited floating point precision.<br>
Could you explain your use case a bit more and what the output and target represents?<br>
If you are working on a classification use case you might need to create predicted class labels from the output before comparing the result.</p>
Grad_fn hidden by inplace operations

Sam-gege: <pre><code class="lang-python">import torch
x = torch.tensor([0.,0.,0.], requires_grad=True)
y = x * 3
print(f'grad_fn before inplace op: {y.grad_fn}')
y[:2]=100
print(f'grad_fn after inplace op: {y.grad_fn}')
y.sum().backward()
print(f'grad at x: {x.grad}')

# grad_fn before inplace op: &lt;MulBackward0 object at 0x7fad0404e3a0&gt;
# grad_fn after inplace op: &lt;CopySlices object at 0x7fad0404e3a0&gt;
# grad at x: tensor([0., 0., 3.])
</code></pre>
<p>As shown above, for a tensor <code>y</code> that already has a grad_fn <code>MulBackward0</code>, if you do inplace operation on it, then its grad_fn will be overwritten to <code>CopySlices</code>. However, pytorch still manages to do backward correctly to <code>x</code>. I have two questions:</p>
<ol>
<li>
<p>how many tensors are actually created in the computational graph? If there’s only two (<code>x</code> and <code>y</code>) then,</p>
</li>
<li>
<p>how come pytorch still knows there’s a <code>MulBackward0</code> associated to tensor <code>y</code>, although its grad_fn is overwritten to <code>CopySlices</code>? If you would say that <code>MulBackward0</code> is not lost but hidden, how to show it?</p>
</li>
</ol>

soulitzer: <ol>
<li>This does not apply to mul, because the tensor modified inplace is not actually needed to use in the backward pass, but if that IS the case we’d need to do a clone to save the original tensor before it was modified so it can be used for backward - so yes generally an extra tensor is created</li>
<li>The CopySlices node actually holds the original MulBackward node as a field, so that during backward it will use it to compute backward.</li>
</ol>