akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Change tensor values by index greater(or less) than some value

shihanmax: <p>Hey everyone!</p>
<p>I have a question of selecting/changing values by some specific indices.<br>
Here is an example:</p>
<pre><code class="lang-auto">import torch

"""
The original tensor is like this:

t = torch.tensor(
    [[99, 2, 1, 3, 4],
     [1, 2, 3, 99, 4],
     [4, 1, 99, 2, 3]]
)

I want to change t's value with rule:
for each row, set elements which is before 99 to 1; 0 otherwise,

so my target is:

tensor(
    [[0, 0, 0, 0],
    [1, 1, 1, 0], 
    [1, 1, 0, 0]]
)

following is the for-loop way, my question is, can I do this more efficiently?

Thanks!

"""

# create a tensor of shape [3, 4]
t = torch.tensor(
    [[99, 2, 1, 3, 4],
     [1, 2, 3, 99, 4],
     [4, 1, 99, 2, 3]]
)

# get 99's position of each row
position_of_zero = torch.nonzero(t == 99)[:, 1]

print(position_of_zero)

# for-loop
for row, zero_pos in zip(t, position_of_zero):
    for i in range(zero_pos):
        row[i] = 1
    
    for i in range(zero_pos + 1, len(row)):
        row[i] = 0

print(t)

# drop 99
t = t[t != 99].view(-1, 4)
print(t)

</code></pre>

ptrblck: <p>The desired target doesn’t match the description:</p>
<pre><code class="lang-python">for each row, set elements which is before 99 to 1; 0 otherwise,
</code></pre>
<p>since the <code>99</code> in the first row is set to <code>0</code>, while the others are not.<br>
Assuming you would like to set the value at <code>99</code> to <code>1</code>, this code should work and would avoid the loop:</p>
<pre><code class="lang-python">t2 = torch.tensor(
    [[99, 2, 1, 3, 4],
     [1, 2, 3, 99, 4],
     [4, 1, 99, 2, 3]]
)

idx = torch.cumsum((t2 == 99), 1)
idx[t2==99] = 0
res = (~idx.bool()).long()
print(res)
&gt; tensor([[1, 0, 0, 0, 0],
          [1, 1, 1, 1, 0],
          [1, 1, 1, 0, 0]])
</code></pre>
<p>Depending on the shape of your tensor, it might be faster or not, so I would recommend to profile both approaches.</p>

shihanmax: <blockquote>
<p>since the <code>99</code> in the first row is set to <code>0</code> , while the others are not.</p>
</blockquote>
<p>Sorry for the unclear description, actually I want to just DROP all the 99s,</p>
<p>I slightly modified you solution, and it works for me too!</p>
<pre data-code-wrap="Python"><code class="lang-Python">t2 = torch.tensor(
    [[99, 2, 1, 3, 4],
     [1, 2, 3, 99, 4],
     [4, 1, 99, 2, 3]]
)

idx = torch.cumsum((t2 == 99), 1)
idx[t2==99] = 2
idx = idx[idx != 2].view(3, -1)

res = (~idx.bool()).long()
print(res)
&gt; tensor([[1, 0, 0, 0, 0],
          [1, 1, 1, 1, 0],
          [1, 1, 1, 0, 0]])
</code></pre>
<p>Thanks for your help!</p>
Module 'torch.cuda' has no attribute 'amp' with torch 1.6.0

Feywell: <p>I try to install pytorch 1.6.0 with pip.</p>
<blockquote>
<p>torch                     1.6.0+cu101               <br>
torchvision               0.7.0+cu101               <br>
cudatoolkit               10.1.243             h6bb024c_0    defaults</p>
</blockquote>
<p>but I got a error:</p>
<blockquote>
<pre><code>scaler1 = torch.cuda.amp.GradScaler()
</code></pre>
<p>AttributeError: module ‘torch.cuda’ has no attribute ‘amp’</p>
</blockquote>

ptrblck: <p>Could you post the output of <code>print(torch.__version__)</code>, please?</p>

Feywell: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="2" data-topic="92360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(torch.<strong>version</strong>)</p>
</blockquote>
</aside>
<p>thank you!</p>
<p>It’s strange.</p>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.<strong>version</strong>)<br>
1.2.0</p>
</blockquote>
</blockquote>
</blockquote>
<p>Why there is another torch version in a new conda env?</p>
<p>It returns aother error, when I activate conda env again.<br>
It returns:</p>
<blockquote>
<p>ImportError: cannot import name ‘default_generator’</p>
</blockquote>

ptrblck: <p>Unsure, but your virtual environment might have multiple PyTorch installations.<br>
Could you create a new environment and install PyTorch <code>1.6</code> again?</p>

Origin: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> …I created a new virtual environment. “pip list” returns torch 1.7.1 and torchvision 0.8.2. Still i am getting error “module ‘torch.cuda.amp’ has no attribute ‘Gradscaler’”.</p>
<p>Could you please help me out here sir??</p>

ptrblck: <p>Try to use <code>GradScaler</code> (uppercase S): <code>torch.cuda.amp.GradScaler</code>.</p>
Backpropagation in a bifurcating architecture

jk78: <p>Hello,</p>
<p>I am trying to train a model with one input and two outputs, where each of the two outputs is computed with separate convolutional and dense layers within an architecture. Any suggestions? I have the model coded up, but was curious how I ensure the loss of one output does not impact the gradients of the other branch? Is there a way to specify when I do loss.backward? Do I need two separate loss functions?</p>
<p>To clarify, the input image is immediately passed through two separate branches, each consisting a simple CNN architecture, ending with two classification predictions (two different classification questions respectively).</p>
<p>Thanks!</p>

ptrblck: <p>Your approach should work and the computation graph would make sure to calculate the corresponding gradients only.<br>
Could you post your model definition, so that we could check it, if you still have doubts?</p>
Complex Autograd failing

Kinetize: <p>Hello,</p>
<p>I am currently implementing complex GRU-cells as described in the paper “Complex Gated Recurrent Neural Networks”.<br>
Working on the currently nightly (1.8.0dev20210201) and <a href="https://github.com/pytorch/pytorch/blob/72bc3d9de4f8baa28ebc8f9399ad86cefe89d9f5/tools/autograd/gen_variable_type.py#L154" rel="noopener nofollow ugc">the list of supported complex gradients</a>, I expected the code below to run through.<br>
Instead, I receive the following error:</p>
<pre><code class="lang-auto">  File "...\model\spectral_rnn\spectral_rnn.py", line 75, in train
    loss.backward()
  File "...\Python38\lib\site-packages\torch\tensor.py", line 227, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "...\Python38\lib\site-packages\torch\autograd\__init__.py", line 145, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected isFloatingType(grad.scalar_type()) || (input_is_complex == grad_is_complex) to be true, but got false.
</code></pre>
<p>I also reported it as a bug as <a href="https://github.com/pytorch/pytorch/issues/51627" rel="noopener nofollow ugc">issue in the repository</a>.<br>
I am happy for any hint, e.g. which operation the problem could be!</p>
<p>Thanks,<br>
Kinetize</p>
<pre><code class="lang-auto">def to_complex_activation(activation):
    return lambda x: torch.view_as_complex(torch.cat(
        [activation(x.real).unsqueeze(-1), activation(x.imag).unsqueeze(-1)], dim=-1))

class CGCell(nn.Module):

    def __init__(self, input_size, hidden_size):
        super(CGCell, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size

        self.wg = nn.Parameter(torch.randn(2 * hidden_size, hidden_size, dtype=torch.cfloat))
        self.vg = nn.Parameter(torch.randn(2 * hidden_size, input_size, dtype=torch.cfloat))
        self.bg = nn.Parameter(torch.randn(2 * hidden_size, dtype=torch.cfloat))

        self.w = nn.Parameter(torch.randn(hidden_size, hidden_size, dtype=torch.cfloat))
        self.v = nn.Parameter(torch.randn(hidden_size, input_size, dtype=torch.cfloat))
        self.b = nn.Parameter(torch.randn(hidden_size, dtype=torch.cfloat))

        alpha = beta = 0.5  # TODO
        self.fg = lambda x: torch.sigmoid(alpha * x.real + beta * x.imag)
        self.fa = to_complex_activation(torch.sigmoid)

    def _init_hidden(self, x):
        h = torch.zeros((x.shape[0], self.hidden_size),  dtype=torch.cfloat).to(device)

        return h

    def forward(self, x, ht_=None):
        if ht_ is None:
            ht_ = self._init_hidden(x)

        gates = ht_ @ self.wg.T + x @ self.vg.T + self.bg
        g_r, g_z = gates.chunk(2, 1)

        g_r = self.fg(g_r)
        g_z = self.fg(g_z)

        z = (g_r * ht_) @ self.w.T + x @ self.v.T + self.b
        ht = g_z * self.fa(z) + (1 - g_z) * ht_

        return ht

</code></pre>

Kinetize: <p>Bug resolved, see <a href="https://github.com/pytorch/pytorch/issues/51627" rel="noopener nofollow ugc">this issue</a>.</p>
Connecting two seq2seq models implemented in different workspaces

Mary: <p>Hi All,</p>
<p>I am currently trying to find a way to connect two seq2seq models together in a way that the output of one is the input to another deep neural model.</p>
<p>Both models are pre-trained. However, each of the neural models are implemented in different frameworks using different libraries. Is there a way to optimize the models that are implemented in different workspaces together?</p>
<p>In other words, to use the loss calculated in the second network in addition to update the parameters of the first network.</p>
<p>I would appreciate any help. Thank you!</p>

ptrblck: <p>No, I don’t think you can optimizer both models together using different frameworks.<br>
While the inference use case might work, I’m not aware of any tool allowing the backward pass to cross between frameworks.</p>

Mary: <p>Thank you so much for your reply!</p>
<p>I was thinking of stacking two neural network models together in a same workspace, and pass the output of one as the input to another one. Do you think this way two networks can be optimized together?</p>
<p>I highly appreciate your input regarding this matter.<br>
Thank you</p>

ptrblck: <p>No, I don’t think so, as the frameworks would have to use a common Autograd implementation.<br>
E.g. while PyTorch uses its Autograd backend, creates the computation graph during the forward pass, and stored the <code>.grad_fn</code>s as attributes of activation tensors, the second framework would need to be able to automatically combine the PyTorch output to it’s “backend engine” (however it’s implemented).<br>
That being said, you might be able to manually combine different frameworks together.</p>
How to skip wrong document in __getitem__ in Dataset class

111460: <p>I have question about pytorch Dataset class</p>
<p>what shoud I do when we need to skip n th element for training?</p>
<p>for example,</p>
<pre><code class="lang-auto"># when idx == 100, the data is ill-formed text.
# Thus, I want to skip this sample in training
def __getitem__(self, idx):
   return self.data[idx]
</code></pre>
<p>These case happens when I filter some text and save in MongoDB<br>
When the filtered text is empty or too short, I want to skip the sample</p>
<p>Thank you</p>

ptrblck: <p>You could return a constant tensor, which you could then filter out in the training loop, but note that this approach would lower the batch size and in the edge case you could also end up with a completely empty batch.<br>
The better approach would be to remove these indices from the beginning.<br>
If you could compute the invalid indices before (or in the <code>__init__</code> method), you could use a <code>valid_idx</code> list, which would then return only the samples, which should be returned:</p>
<pre><code class="lang-python">def __init__(self):
    self.valid_idx = [0, 2, 3, 5, 6, 8, ...] # calculate only valid indices or filter out invalid ones

def __getitem__(self, index):
    idx = self.valid_idx[index]
    data = self.data[idx]
    return data

def __len__(self):
    return len(self.valid_idx)
</code></pre>

111460: <p>Thank you,</p>
<p>Actually, I made MongoDB wrapper for collecting several collections<br>
And, the wrapper is connected to a custom Dataset class<br>
Thus, maybe I can build some valid index for the collections in the mongodb wrapper, which is pre-built index before training loops checking whether the document is valid or not</p>
<p>I got some hints from the answer. Thank you.</p>
Different ways of Tensor creation on GPU using vectors

sid1: <p>Hi, what is the difference among the ways in which tensor a, b, c and d are created using vector v below, in terms of memory allocation on CPU and GPU?</p>
<pre><code class="lang-auto">vector&lt;float&gt; v(5,10.0);
auto options_a = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA, 1);
torch::Tensor a = torch::from_blob(v, {5}, options_a);

auto options_b = torch::TensorOptions().dtype(torch::kFloat32);
torch::Tensor b = torch::from_blob(v, {5}, options_b).to(torch::kCUDA);

torch::Tensor c = torch::tensor(v).to(torch::kCUDA);

torch::Tensor d = torch::tensor(v, device = torch::kCUDA); 
</code></pre>
<p>I know for c a tensor is first created on CPU and then copied to GPU and for d, tensor is directly created on GPU. But then in the case of d, is the vector v itself transferred to GPU?</p>
Multiprocessing on CPUs

NoobCoder: <p>Hello,</p>
<p>I have a model that I trained on GPUs. Now, I want to use it on my test dataset.<br>
The dataset is large, and what I would like to do is use python multiprocessing to load each image, make patches from them, and forward them to the model on CPUs.</p>
<p>The problem is that I get different speeds when I use a different number of CPUs.<br>
My questions are:</p>
<ol>
<li>Does the number of CPUs affects the model?</li>
<li>I init() and load the model once, and then let’s say I use 48 CPUs which means 48 images simultaneously, so how that one model predicts 48 patches at same time?</li>
<li>What are the differences between <code>TORCH.MULTIPROCESSING</code> and <code>python multiprocessing</code>? Now, I use the latter one, should I move to the first one?</li>
<li>What are the differences between <code>TORCH.MULTIPROCESSING</code> and <code>nn.DataParallel</code>?</li>
</ol>
<p>I found this <a href="https://github.com/pytorch/pytorch/issues/9873" rel="noopener nofollow ugc">issue</a>, but I am not sure if they are related!</p>
<p>Thanks,</p>

NoobCoder: <p>Update:</p>
<p>When I run a single image with 9 CPUs, it finishes in 30 minutes.<br>
But when I run 9 images with 9 CPUs (each image in a CPU), it takes about 21 hours !!</p>
<p>How can I solve this? I do not know the reason, but maybe if I copy the model on each CPU it will be faster? How should I do it?</p>
Slow torch.stack()

Yann: <p>Hello,</p>
<p>I would like help optimizing the following line of code, please. We are training a RNN and when profiling our code we find that our bottleneck is this line by far:</p>
<pre><code class="lang-python">signal_seq = torch.stack([self.full_signal[idx+i:idx+i+64] for i in reversed(range(0,-120, -6))], axis=0)
</code></pre>
<p>full_signal is a 1D cuda.FloatTensor, and idx is a positive integer.</p>
<p>This line is in the <strong>get_item</strong>() method of our Dataset, and it seems to take 16s out of 18s in our tests. When executing torch.utils.bottleneck, cProfile says that all this time is consumed in the tensor() method apparently.</p>
<p>Is there an efficient way of doing this, please?</p>
<p>Thank you in advance.</p>

Eta_C: <p><strong>FASTER IMPLEMENTATION</strong></p>
<pre><code class="lang-python">full_signal[idx-114:idx+64].unfold(0, 64, 6)
</code></pre>

Yann: <p>Hello Eta_C, this is much faster indeed, thank you!</p>
<p>Turns out that our actual bottleneck was not here though, it seems to be in the backward pass, which we didn’t really see in our profilers, I guess because of asynchronous operations, so it’s all good.</p>
Integration of Glow with NN

Kittu28: <p>I want to integrate Glow compiler  with a Neural network(say X) and run the optimized code which is output of Glow  on a  specific backend. Can someone please suggest me how this could be achieved like where  the integration could be made with the  Neural network(X) and Glow Compiler.</p>

jfix: <p>Hi <a class="mention" href="/u/kittu28">@Kittu28</a> – it would help if you could provide more information, such as the format of your current model (PyTorch? ONNX? Caffe2? TF?), and what your target HW is (x86 or ARM CPU? GPU? Specialized HW).</p>

Kittu28: <p>its Caffe2 and the target HW is DSP.<br>
If you know anything on this or any resources  on this ,please help me  and Thank you very much for the reply.</p>

jfix: <p>For Caffe2 loading, we have a Caffe2ModelLoader that will load a model consisting of a pair of <code>init_net.pb</code> and <code>predict_net.pb</code>. If you’re targeting some custom DSP then you will most likely need to build your own backend to compile down to it. Assuming there is some CPU driving execution you may be able to create an LLVM-based backend. It’s hard to say exactly what you need to do based on the info you’ve provided, though. <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" rel="noopener nofollow ugc">This doc</a> is a good place to start on loading a model and compiling a binary for it, but you’d still need to build the backend that supports your HW.</p>
What's the recommended way of profiling DistributedDataParallel pytorch code?

anneouyang: <p>To see how much time is spent on interprocess communication, allreduce, etc…; as well as static memory usage and amount of information transferred during interprocess communciation</p>

osalpekar: <p>nvprof from Nvidia is probably the best tool available: <a href="https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-handy-universal-gpu-profiler/" class="inline-onebox" rel="noopener nofollow ugc">CUDA Pro Tip: nvprof is Your Handy Universal GPU Profiler | NVIDIA Developer Blog</a></p>
<p>DDP currently does not work with the autograd profiler</p>
Loss: nan in GPU ubuntu, Valid in Colab

yegane: <p>Hi Guys,<br>
I notebook file that is working in Colab, and the loss is valid, but when I execute it in Ubuntu with a local GPU, I get some accuracy, but I get nan, as Loss. I have no idea why. Have you seen anything like this?</p>

tymokvo: <p>Hard to tell without seeing some code. Maybe start by checking CUDA/CUDNN versions on both platforms?</p>
<p>Colab for me shows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/11ee73f91c2dae378a17b98cfaf9775bb9a3f092.png" data-download-href="https://discuss.pytorch.org/uploads/default/11ee73f91c2dae378a17b98cfaf9775bb9a3f092" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/11ee73f91c2dae378a17b98cfaf9775bb9a3f092_2_690x199.png" alt="image" data-base62-sha1="2yD00Un5ChD4LxHnEZM2LKosh6W" width="690" height="199" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/11ee73f91c2dae378a17b98cfaf9775bb9a3f092_2_690x199.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/11ee73f91c2dae378a17b98cfaf9775bb9a3f092_2_1035x298.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/11ee73f91c2dae378a17b98cfaf9775bb9a3f092.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1205×348 23.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have also had problems with old hardware (780ti) a long time ago behaving strangely when memory consumption was close to 100%.</p>

Hanieh_Khosravi: <p>I have the same problem. did you manage to solve it?</p>
Mapping tensors without offloading to cpu

amaleki: <p>consider</p>
<pre><code class="lang-auto">a=torch.tensor([0, 1, 2, 3, 4], device='cuda')
b=torch.tensor([4, 5, 6, 7, 8], device='cuda')
c=torch.tensor([0, 0, 0, 1, 1, 3, 4], device='cuda')
</code></pre>
<p>how can I get <code>d=torch.tensor([4, 4, 4, 5, 5, 7, 8], device='cuda')</code><br>
without converting to regular list, or offloading to cpu?</p>
<p>(if they were regular lists, something as simple as</p>
<pre><code class="lang-auto">mapping = dict(zip(a, b))
d = map(mapping.get, c)
</code></pre>
<p>would’ve worked. )</p>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p><code>torch.gather</code> should work fine:</p>
<pre><code class="lang-python">b = torch.tensor([4, 5, 6, 7, 8], device='cuda')
c = torch.tensor([0, 0, 0, 1, 1, 3, 4], device='cuda')

res = torch.gather(b, 0, c)
print(res)
&gt; tensor([4, 4, 4, 5, 5, 7, 8], device='cuda:0')
</code></pre>

amaleki: <p>what if <code>a</code> is something else, like <code>a=torch.tensor([20,10,30,40,0])</code>?</p>

ptrblck: <p>I don’t know how <code>a</code> relates to the other two tensors. What would these values of <code>a</code> mean?</p>

amaleki: <p>basically replacing elements of <code>c</code> using a dictionary with keys of <code>a</code> and values of <code>b</code>.<br>
For example,</p>
<pre><code class="lang-auto">a=torch.tensor([1, 0, 3, -1, 4], device='cuda')
b=torch.tensor([4, 5, 6, 7, 8], device='cuda')
c=torch.tensor([0, 0, 0, 1, -1, 3, 4], device='cuda')
</code></pre>
<p>then</p>
<pre><code class="lang-auto">d = custom_map(a,b,c) # = torch.tensor([5, 5, 5, 4, 7, 6, 8], device='cuda')
</code></pre>
<p>is this clear enough?</p>

ptrblck: <p>You could index <code>a</code> with <code>c</code>, but would need to remove the negative values from <code>a</code>, as <code>gather</code> expects positive indices:</p>
<pre><code class="lang-python">device = 'cuda'
a = torch.tensor([1, 0, 3, 2, 4], device=device)
b = torch.tensor([4, 5, 6, 7, 8], device=device)
c = torch.tensor([0, 0, 0, 1, -1, 3, 4], device=device)

res = torch.gather(b, 0, a[c])
print(res)
&gt; tensor([5, 5, 5, 4, 8, 6, 8], device='cuda:0')
</code></pre>
<p>I’m also unsure about your expected result tensor, as the <code>-1</code> in <code>c</code> would index <code>a</code> at the last position and <code>b[4]</code> would be <code>8</code> not <code>7</code>.</p>

amaleki: <p><code>a</code> is not the list of indices. <code>a</code> is the old values and <code>b</code> is the new values; think of them as keys and values of a mapping dictionary.<br>
for example all 0 (second element of a) in <code>c</code> should be mapped to 5 (second element of <code>b</code>).<br>
if <code>a,b</code> and <code>c</code> were regular lists:</p>
<pre><code class="lang-auto">my_dict = dict(zip(a, b))
d = map(my_dict.get, c)
</code></pre>
<p>sorry if my explanation was not clear enough.</p>
Convert and save PytorchTensor to .mat image on GPU in python

Mathedelic: <p>I am a beginner in Pytorch and I am stuck on a question for days. I want to save a image which is in Pytorch tensor form as .mat file. I looked but there doesn’t seem to be a direct method on converting Pytoch tensors to .mat file. One possible solution which I found was to convert it to numpy array, but since I am using Nvidia GPU, when I try converting Pytorch tensor to numpy array it gives me this error :</p>
<p><code>fake_images[0] = fake_images[0].numpy()</code></p>
<p><code>TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first</code></p>
<p>Please can you help me with saving Pytorch tensor to .mat file while using GPU.</p>
<p>Thank you so so much in advance!!</p>

ptrblck: <p>You would have to transfer the tensor to the CPU first (and detach it, if necessary), as given in the error message:</p>
<pre><code class="lang-python">x = fake_images[0].detach().cpu().numpy()
</code></pre>
<p>To transform the numpy array to a MATLAB <code>mat</code> you should use <code>scipy.io.savemat</code>.</p>

Mathedelic: <p>Thank you so much for your response. However when I tried this code, i got this error</p>
<p><code>TypeError: can't assign a numpy.ndarray to a torch.cuda.FloatTensor</code></p>
<p>Please can you help me in regard to this ? Any help is much appreciated. Thanks in advance</p>

ptrblck: <p>I guess you are trying to assign the numpy array to the tensor itself via:</p>
<pre><code class="lang-python">fake_images[0] = fake_images[0].detach().cpu().numpy()
</code></pre>
<p>If that’s the case, assign the numpy array to a new variable as seen in my code:</p>
<pre><code class="lang-python">x = fake_images[0].detach().cpu().numpy()
</code></pre>
Why is cuda10.0 able to work with pytorch1.7?

two_Two: <p>Recently,I need to install pytorch ,when I check out the website :</p>
<p><a href="https://pytorch.org/get-started/locally/#supported-linux-distributions" rel="noopener nofollow ugc">download pytorch using pip</a></p>
<p>It shows four different version 9.2,10.1,10.2,11.0 to choose ,And I have cuda version 10.0 and driver version 450 installed on my computer,I thought it would fail to enable gpu when using pytorch ,After I choose 10.1 and use pip to install pytorch and try <code>torch.cuda.is_available()</code> and it returns <code>True</code></p>
<p>I have two questions:</p>
<ol>
<li>Why does everything turn out to be working even my cuda version is not the same as any of one I mentioned ?</li>
<li>What’s the difference between choosing cuda verison 9.2,10.1,10.2,11.0 ?</li>
</ol>

BramVanroy: <p>The CUDA library is included in the download. So it doesn’t matter which CUDA installation you have, when you pip install a version, it download and installs the given CUDA version as well. I also did not realize this until <a class="mention" href="/u/ptrblck">@ptrblck</a> told me. AFAIK that is different from Tensorflow which requires a pre-existing installation.</p>
Difference between torchtext, pytorch-nlp and allennlp

IamAdiSri: <p>I need to work with some text datasets and found that these three libraries all provide utilities for the same, however I’m unable to figure out what the difference between them is and if any one is better than the others. I read someplace that AllenNLP is geared towards research and offers pretrained models but that’s about it.</p>
<p>Does anyone know if it makes a difference? If so, what is it?</p>
<p>Links:</p>
<ul>
<li><a href="https://github.com/pytorch/text" rel="noopener nofollow ugc">TorchText</a></li>
<li><a href="https://github.com/kolloldas/torchnlp" rel="noopener nofollow ugc">Pytorch-NLP</a></li>
<li><a href="https://github.com/allenai/allennlp" rel="noopener nofollow ugc">AllenNLP</a></li>
</ul>

BramVanroy: <p>They all have their strengths and weaknesses because they have different things they focus on. One is not “better” per se than the others. It just depends on what you want to do. We cannot make this decision for you because it depends on your use case. I recommend that you simply read into the functionality of all of them and then see which best matches your requirements.</p>
Dataset problems

lifeblack: <p>Hi I have a video dataset… I had to change my video frames to small blocks… I changed my frames to blocks in a tensor with this shape([780000,8,8]). some of the blocks are equal. I want to delete the same blocks because this similar blocks have different outputs and my network does not train good…<br>
I used simple two “for”  for find similar blocks(I have 780000 blocks (8*8)) but this takes too much time… is there any easier way to find similar blocks 2d blocks in a tensor?</p>
CPU bottlecking GPU

dazzle-me: <p>Hello everyone, - currently I’m training EfficientNet-B0-ns (it’s kinda new SOTA ResNet) model for some classification task, num_classes = 5.</p>
<p>My GPU, which is RTX 3060Ti allows me to choose batch size up to 64 in order to completely saturate  GPU VRAM.</p>
<p>Also I have 6 core Ryzen 5 2600x, - and even with num_workers set to 12 it can’t saturate the GPU (I can see how the model is struggling to compute something, because it’s not ready yet, via tqdm).</p>
<p>Is there any way we can prevent CPU bottlenecking the GPU in case of small models?</p>
<p>These no such problem on bigger models, though.</p>

ptrblck: <aside class="quote no-group" data-username="dazzle-me" data-post="1" data-topic="110708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dazzle-me/48/32769_2.png" class="avatar"> dazzle-me:</div>
<blockquote>
<p>Is there any way we can prevent CPU bottlenecking the GPU in case of small models?</p>
</blockquote>
</aside>
<p>Assuming the data loading is the bottleneck, you could take a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>, which explains common use cases and potential workarounds.<br>
If not already done, use multiple workers in the <code>DataLoader</code>.</p>

dazzle-me: <p>Thanks for the useful post, but the actual problem was my code.</p>
<p>I was applying huge augmentations before resizing the image, not vice versa, - resolution of image on which augmentations was performed was like 800x600 instead of 224x224, and that caused the bottleneck.</p>
<p>Changing the order of augmentations and resizing fixed my problem, for sure <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to use intermediate variable as weights for later nn.Linear layers?

Elocin: <p>Hi, I’m having problem training network of the following structure (structure inspired by a recent paper):</p>
<p>Given input X, Y, we want to first find a hidden variable based on X:<br>
X → fc1 → fc2 → hidden</p>
<p>Then we want to slice the hidden variable so that we can assign them as weights/bias to fc3, fc4<br>
And then we compute:<br>
Y → fc3 → fc4 → prediction</p>
<p>We have a ground truth to compute loss with prediction.</p>
<p>However, during training the error never decreases. I’m wondering if the error may come from inplace value assignment for fc3, fc4? My current setup is attached below.</p>
<pre><code class="lang-auto">class Surface_NN(torch.nn.Module):
  def __init__(self):
    super(Surface_NN, self).__init__()
    self.fc1 = nn.Linear(3, 25)
    self.fc2 = nn.Linear(25, 303)
    
    self.fc3 = nn.Linear(2, 50) # 3 * 50 = 150 params
    self.fc4 = nn.Linear(50, 3) # 51 * 3 = 153 params, in total 303 params

  def forward(self, Image, Surface):
    hidden = self.fc1(Image)
    hidden = F.relu(hidden)
    hidden = self.fc2(hidden)

    # NOTE THIS SPECIFIC SETUP ONLY APPLY TO BATCH_SIZE = 1.
    weights = hidden[0]

    fc3weight = (weights[0:100]).reshape([1, 50,2])
    fc3bias = weights[100:150]
    fc4weight = (weights[150:300]).reshape([1, 3, 50])
    fc4bias = weights[300:304]

    self.state_dict()['fc3.weight'] = fc3weight
    self.state_dict()['fc3.bias'] = fc3bias
    self.state_dict()['fc4.weight'] = fc4weight
    self.state_dict()['fc4.bias'] = fc4bias
 
    middle = F.relu(self.fc3(Surface))
    result = self.fc4(middle)
    return result
</code></pre>

ptrblck: <p>Assigning the tensors to the <code>state_dict</code> keys won’t change the actual parameters, if you don’t reload the <code>state_dict</code>.<br>
Based on the description of the use case, I think you could use the functional API and directly use the <code>fc3</code> parameters:</p>
<pre><code class="lang-python">fc3weight = (weights[0:100]).reshape([1, 50,2])
fc3bias = weights[100:150]
...
out = F.linear(Surface, fc3weight, fc3bias)
...
</code></pre>

Elocin: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="110235">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>out = F.linear(Surface, fc3weight, fc3bias)</code></p>
</blockquote>
</aside>
<p>Thanks a lot this worked! (Although I need to reshape the weight/bias to 2D instead of 3D)</p>
Size mismatch for decoder.rule_logits.weight , bias and embedding weight

Muradean: <p>Hi everybody.</p>
<p>Recently I’ve git cloned and trained a natural language to SQL model (<a href="https://github.com/microsoft/rat-sql" class="inline-onebox" rel="noopener nofollow ugc">GitHub - microsoft/rat-sql: A relation-aware semantic parsing model from English to SQL</a>). However, the training in my machine was prolonged. In order to solve that, I went to the issues pages which was providing a pre-trained model (config.json + model_checkpoint) and I tried to load it; however, I was greeted with the error below <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> . I’ve successfully loaded checkpoints some days ago, but those were from models I had previously trained.</p>
<p>I am wondering if there is any fix I can do to solve this and perform inference with the pre-trained model. Thanks a lot.</p>
<pre><code class="lang-auto">  File "/app/ratsql/utils/saver.py", line 42, in load_checkpoint
    item_dict[item_name].load_state_dict(checkpoint[item_name])
  File "/root/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 839, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for EncDecModel:
        size mismatch for decoder.rule_logits.2.weight: copying a param with shape torch.Size([94, 128]) from checkpoint, the shape in current model is torch.Size([97, 128]).
        size mismatch for decoder.rule_logits.2.bias: copying a param with shape torch.Size([94]) from checkpoint, the shape in current model is torch.Size([97]).
        size mismatch for decoder.rule_embedding.weight: copying a param with shape torch.Size([94, 128]) from checkpoint, the shape in current model is torch.Size([97, 128]).```</code></pre>

ptrblck: <aside class="quote no-group" data-username="Muradean" data-post="1" data-topic="110724">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f0a364/48.png" class="avatar"> Muradean:</div>
<blockquote>
<p>I am wondering if there is any fix I can do to solve this and perform inference with the pre-trained model.</p>
</blockquote>
</aside>
<p>You would have to change the model architecture such that all parameters match to the one in the pretrained <code>state_dict</code>.<br>
If you are loading this <code>state_dict</code> from a repository, I assume the model definition might also be there.</p>
Understanding Groups for Conv1D

lipuck: <p>Hello everyone,</p>
<p>I wanted to implement a tree/pyramid structure for a NLP task, which would look similar to this figure <img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6eecbc8b96744b7d2f9a630f68b81e1b341beba.png" alt="adasent" data-base62-sha1="q6iw6gExNoT34eKC6aIow5TtGtA" width="308" height="187"><br>
<em>(Self-Adaptive Hierarchical Sentence Model by Zhao et. al.)</em></p>
<p>The elements of the next layer consist of their left and right child. I was now wondering if a 1D Convolution with (here) 5 Groups could be appropriate.<br>
My concerns are, that I might not be able to get the overlapping triangles, i.e. here only 3 Groups with 2 elements are possible and next that the contribution of a single child might not be as what is depicted in the picture but would contribute somehow different.</p>
<p>So my actual question is: Can someone please help me understand groups for 1D Convolution and explain how elements in groups are chosen and how each group contributes to the output channels?</p>
<p>Thanks so much in advance!</p>
How to extract certain elements from tensor by another tensor of indices

enterthevoidf22: <p>i have a tensor, A, of shape (155,1,79,95,68)<br>
i have a tensor, Aind of shape (6200,5) containing indices of certain elements in A.</p>
<p>i want to do something like A[Aind] and receive a tensor of shape (6200,1) or vice versa, with the elements corresponding to those indices.</p>
<p>when i actually do so, i get not what i’d expect.</p>
<p>so far, the very clumsy solution i found is:<br>
what_i_want = A[Aind[:,0],Aind[:,1],Aind[:,2],Aind[:,3],Aind[:,4]]</p>
Loading Data from Numpy Array in batches using Pytorch

Ankur_Singh1: <p>I have video data stored in a Numpy array each of shape (1, 16, 100, 100, 3). The array is huge and every time I load it, I get a memory error.<br>
Is there a way using Pytorch to load these videos in batches without actually loading the Numpy array so that I don’t get the memory error.</p>
<p>Thanks.</p>

ptrblck: <p>What kind of error are you getting?<br>
An array with <code>[16, 100, 100, 3]</code> <code>float32</code> values would use approx. <code>1.83MB</code>, which is quite small.</p>

Ankur_Singh1: <p>Hi,<br>
Thanks for replying. The array has 93000 samples. Therefore the shape of the array is (93000, 1, 16, 100, 100, 3). I want to load this dataset in batches so that I don’t get a memory error. Please let me know if there is a way out. I will be grateful for your help.</p>

ptrblck: <p>Assuming this numpy array is stored locally as an <code>npy</code> file, you could use <code>np.load</code> with the <a href="https://numpy.org/doc/stable/reference/generated/numpy.load.html">mmap_mode</a>, which would allow you to load sliced from the disc without reading the whole array into memory:</p>
<blockquote>
<p><strong>mmap_mode</strong> {None, ‘r+’, ‘r’, ‘w+’, ‘c’}, optional<br>
If not None, then memory-map the file, using the given mode (see <a href="https://numpy.org/doc/stable/reference/generated/numpy.memmap.html#numpy.memmap"><code>numpy.memmap</code></a> for a detailed description of the modes). A memory-mapped array is kept on disk. However, it can be accessed and sliced like any ndarray. Memory mapping is especially useful for accessing small fragments of large files without reading the entire file into memory.</p>
</blockquote>

Ankur_Singh1: <p>Thank you very much. It worked.</p>
Large-size dot product but ... Tried to allocate 256.00 GiB

durden: <p>Hi,</p>
<p>I’m trying to compute the following dot product.</p>
<pre><code class="lang-auto">x = torch.rand((256, 32, 32, 1, 256)).cuda()
y = torch.rand((256, 1, 1, 256, 1024)).cuda()
z = x @ y
</code></pre>
<p>expecting to obtain:<br>
z → torch.Size([256, 32, 32, 1, 1024])</p>
<p>however I get:<br>
<em>RuntimeError: CUDA out of memory. Tried to allocate 256.00 GiB (GPU 0; 10.92 GiB total capacity; 512.00 MiB already allocated; 9.78 GiB free; 514.00 MiB reserved in total by PyTorch)</em></p>
<p>Is there a way to perform this operation efficiently that can fit in the gpu ram?</p>
Modify detectron2 maskRCNN model to fit in class number?

Jeff-Gui: <p>I’m new to CV and I’m trying to train a model to classify some non-daily-life objects. Following detectron2 official tutorial (the “balloon segmentation”), it works quite smoothly. However, I found some non-official tutorials saying that some layers should be modified according to the class number before training. I wonder if this is necessary on top of modification through the config file. Thanks.</p>
How to apply residual links to simple FFNN

spring: <p>Hi,<br>
I want to apply the residual link used in ResNet to a simple model. Can I just add the input x and the output value passed through the linear layer in the element-wise direction as shown below?<br>
Is there anything else I need to do more?</p>
<pre><code>class neuralNet(torch.nn.Module):

    def __init__(self, input_size, hidden_size, num_classes):
        super(neuralNet, self).__init__()
        self.linear1 = torch.nn.Linear(input_size, hidden_size)
        self.relu1 = nn.ReLU()
        self.linear2 = torch.nn.Linear(hidden_size, hidden_size)
        self.relu2 = nn.ReLU()

        self.linear3 = torch.nn.Linear(hidden_size, hidden_size)
        self.relu3 = nn.ReLU()
        self.linear4 = torch.nn.Linear(hidden_size, output_size)
        self.relu4 = nn.ReLU()


    def forward(self, x):
        out = self.linear1(x)
        out = self.relu1(out)
        out = self.linear2(out)
        out = out.add(x)            # H(x) = F(x) + x
        out = self.relu2(out)

        out = self.linear3(out)
        out = self.relu3(out)
        out = self.linear4(out)
        out = out.add(x)            # H(x) = F(x) + x
        out = self.relu4(out)

        return out
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba6fb8d8657ab8b45c6a7b1c527a54e1e41e20d3.png" alt="" role="presentation" width="486" height="277"></p>

ptrblck: <p>Your approach should work and the <code>torchvision</code> implementation of <code>ResNet</code> uses a similar approach as seen <a href="https://github.com/pytorch/vision/blob/09ce6758ad6ab3177f3fdec3b1bb7c7033b16712/torchvision/models/resnet.py#L67-L83">here</a>.</p>

spring: <p>Thank you always for good answers every time. It was very helpful.</p>
Problem with nn.ModuleDict ('method' object is not subscriptable)

kenny.smith1339: <p>I am tring to use nn.ModuleDict following this documentation: <a href="https://pytorch.org/docs/stable/generated/torch.nn.ModuleDict.html" class="inline-onebox" rel="noopener nofollow ugc">ModuleDict — PyTorch 1.7.0 documentation</a></p>
<p>I have this pytorch network:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, kernel_size):
        super(Net, self).__init__()
        modules = {}
        modules["layer1"] = nn.Conv2d(3, 16, kernel_size=kernel_size, stride=1, padding=2)
        self.modules = nn.ModuleDict(modules)

    def forward(self, x):
        x = self.modules["layer1"](x)
</code></pre>
<p>when i use the forward method, I get the following error:</p>
<pre><code>'method' object is not subscriptable
</code></pre>
<p>when I change the forward method to:</p>
<pre><code>  def forward(self, x):
        x = self.modules()["layer1"](x)
</code></pre>
<p>I get the following error:</p>
<pre><code>TypeError: 'generator' object is not subscriptable
</code></pre>
<p>What am i doing wrong?</p>

Nikronic: <p>Hi,</p>
<p><code>modules</code> is a built-in method for <code>nn.Module</code> objects. Rename <code>modules</code> in your model definition to anything else and it should work.</p>
<p>Bests</p>
Un-normalizing the prediction with different loss function

FA_mn: <p>Hello, I hope you are having a great time. Could you please help me about the following problem?</p>
<p>Because the range of the target values is completely different and oscillates between 0 to 6000, the target should be normalized to train the model.<br>
For example :</p>
<pre><code class="lang-auto">t_mean = torch.mean(target)
t_std = torch.std(target)
t_normal = (target - t_mean)/t_std
</code></pre>
<p>How can I denormalize theprediction if the loss function depends on the first and second derivatives of the output of the NN w.r.t the input:</p>
<pre><code class="lang-auto">def P_loss(x,y):
    first_derivative = torch.autograd.grad(y, x, 
                                     grad_outputs, create_graph=True, retain_graph=True)[0]
    
    second_derivative = torch.autograd.grad(first_derivative, x,
                                      grad_outputs, create_graph=True, retain_graph=True)[0]
   
    eq  = 4*first_derivative[:,1] - second_derivative[:,0]

    loss = torch.mean(eq**2)

    return loss
</code></pre>
<p>I think</p>
<pre><code class="lang-auto">pred = model(input) 
pred = pred * t_std  +  t_mean
</code></pre>
<p>does not work correctly.</p>

ttsesm: <p><a class="mention" href="/u/fa_mn">@FA_mn</a> not really an answer to your question but out of curiosity how this loss function works? I mean what is its difference from MSE and why do you use the derivatives (apologies if it is a newbie question but I am quite new to it).</p>

FA_mn: <p>Hello,<br>
In some situation, this loss function can be used .<br>
I use it for solving partial differential equations (PDEs). I can refer you <a href="https://en.wikipedia.org/wiki/Partial_differential_equation" rel="noopener nofollow ugc">here</a></p>

ttsesm: <p>I see, so are you using it for regressing the target values?</p>
<p>(btw, thanks for the link)</p>
<p>Also regarding your question, according to <a href="https://discuss.pytorch.org/t/simple-way-to-inverse-transform-normalization/4821/8">this post</a> what you have tried seems to be correct. Did you try to apply the inverse transform to the targets and see whether you get back the initial values or not?</p>

FA_mn: <p>Thanks a lot for the link you mentioned.</p>
<p>In PDEs, I do not have the target values explicitly. If in regression, <code>x_i</code> is input, and <code>u_i</code> is its related target, I can use</p>
<pre><code class="lang-auto">loss = sum((u_i - y_i)^2, i=1,..,N),
</code></pre>
<p>where <code>y_i</code> is the output of the network related to <code>x_i</code>.<br>
However, in PDEs, I do not have <code>u_i</code> explicitly, and only I know other information, such as the values of the First- or Second-order partial differential equation. Therefore, I cannot use <code>loss = sum((u_i-y_i)^2, i=1,..,N)</code>.</p>
<p>I tried to apply the inverse transform in my PDEs, and I think it worked correctly. I should examine the suggestion mentioned in the link you introduced.<br>
Thanks a lot.</p>
Differences between reading .avi and .mp4 with torchvision.io API

bieltura: <p>Hello everyone,</p>
<p>I have found an issue in the torchvision API while reading a video in diffferent formats. I am aware the backend of <code>torchvision.io.read_video</code> is based on <code>pyAV</code>. The problem is that reading a file with an AVI extension, frames get somehow swapped between them (not temporal consistent - flickering). If the file is transformed to an MP4 file format, frames are loaded/saved correctly.</p>
<p>To test this funcionality I am reading a video with <code>read_video</code> and saving it with <code>write_video</code>. Can anyone please help me on this and replicate the results I am having? Here’s the quick code to do it (you just need the same video in .avi and .mp4 format).</p>
<pre><code class="lang-auto">from torchvision.io import read_video, write_video

video_avi, _, fps = read_video("video_sample.avi", pts_unit='sec')
write_video("video_saved_avi.avi", video_avi, fps['video_fps'])

video_mp4, _, fps = read_video("video_sample.mp4", pts_unit='sec')
write_video("video_saved_mp4.avi", video_mp4, fps['video_fps'])

</code></pre>
<p>Thank you very much for the help.</p>

bieltura: <p>Hi,</p>
<p>I have been able to avoid this issue by transforming all my videos to MP4 format. Still, this is not the ideal solution to use videos through the torchvision API. I have been testing different AVI videos and tthis weird temporal flickering is always present.</p>
How to encode positional info in a 2D matrix input to PyTorch

torch_user_92: <p>I’m working on a project where the goal is to pass in the current state as a traditional Pandas 2D dataframe converted to 2D numpy array, and the neural network should learn to select the row index of the best action given the features of each row.</p>
<p>So the input dataframe for one state could be something like:</p>
<pre><code class="lang-auto">df = pd.DataFrame({'user_rating':[10,7,3],'professional_rating':[10,4,1]})
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8f2344f5c81b4285f862628e93a5424c9cd0931.png" alt="Screenshot (18)" data-base62-sha1="o6z8oBeG5HmoAeLjDmQBbsuguQ1" width="441" height="163"></p>
<p>and it would learn to output index 0 since it has the highest rating.</p>
<p>But then the next state could have the highest rating in the 3rd row instead of the 1st and it would need to pick up on that. (Tried adding another screenshot of this but new users can only add 1).</p>
<p>It doesn’t seem to work when I flatten out the state and pass it in as a 1D array. Is there any other way of doing this in PyTorch?</p>
<p>Thanks so much!</p>
K fold Cross Validation

Ali_Haider: <p>I am having a question that, According to my understanding, the validation set is usually used to fine-tune the hyperparameters and for early stopping to avoid overfitting in the case of CNN/MLP. But when we are dealing with the k fold cross-validation. Still, we can use validation dataset to tune typer parameters and save the checkpoints (Network weights) on which we achieve best validation result?</p>

ptrblck: <p><a class="mention" href="/u/rasbt">@rasbt</a> posted a great post on this topic <a href="https://sebastianraschka.com/faq/docs/evaluate-a-model.html">here</a>, which explains some common use cases and advantages of using certain approaches.</p>

Ali_Haider: <p>Thanks for response but I my case I am having very small dataset unfortunately. (Like 40 samples)</p>