akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Projector tab doesn't appear

TomerP: <p>Hi, I’ve been following the tutorial for <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">Visalizing models, data, and training with Tensorboard</a> and got stuck at section 4 of the tutorial - Adding a “Projector” to TensorBoard.</p>
<p>After I run the code and refresh the Tensorboard page at <a href="http://localhost:6006/" rel="nofollow noopener">http://localhost:6006/</a> , a Projector tab doesn’t appear. When I choose Projector from the drop down menu on the right side, I get an empty page. Then, in the cmd window I get the following error:</p>
<pre><code class="lang-auto">TensorFlow installation not found - running with reduced feature set.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.0.0 at http://localhost:6006/ (Press CTRL+C to quit)
Exception in thread ProjectorPluginIsActiveThread:
Traceback (most recent call last):
  File "C:\Users\theto\Miniconda3\envs\pytorch\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\theto\Miniconda3\envs\pytorch\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\theto\Miniconda3\envs\pytorch\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 322, in _determine_is_active
    if self.configs:
  File "C:\Users\theto\Miniconda3\envs\pytorch\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 340, in configs
    self._augment_configs_with_checkpoint_info()
  File "C:\Users\theto\Miniconda3\envs\pytorch\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 362, in _augment_configs_with_checkpoint_info
    tensor = _read_tensor_tsv_file(fpath)
  File "C:\Users\theto\Miniconda3\envs\pytorch\lib\site-packages\tensorboard\plugins\projector\projector_plugin.py", line 157, in _read_tensor_tsv_file
    return np.array(tensor, dtype='float32')
ValueError: setting an array element with a sequence.
</code></pre>
<p>I’m running Pytorch 1.3.1 with Tensorboard 2.0.0 in a conda environment.</p>
<p>Any idea why this is happening?</p>

Rafay_Khan: <p>Same here the Projector Tab does not appear, but I can choose it from the drop down menu and seems to be working from there.</p>

AladdinPerzon: <p>I got the same problem, anyone managed to solve this?</p>
<p>EDIT: Using Pytorch 1.4.0 and installing tensorboard nightly (2.3.0a I believe I installed) then it worked perfectly for me. Might be worth checking if you run into the same issue.</p>

MrLi: <p>I have this problem too.<br>
My solution is to install tensorflow as backend support.</p>

neo4dev: <p>I had this problem as well, and the solution is to restart tensorboard in the terminal</p>

r_vn_alst: <p>I got this problem (no projector tab to be seen) too, in my case I noticed that on the dropdown (upper right) there was a tab profile, that complained I should “pip install -U tensorboard-plugin-profile”.</p>
<p>after doing that, there were a whole bunch of extra options (hparams, mesh, whatiftool) in the dropdown, and also projector.<br>
Hope it helps someone !</p>
CUDA/Torch Problem while using Oogabooga/PrivateGPT

durkto: <p>Hey,<br>
I was trying to generate text using the above mentioned tools, but I’m getting the following error:</p>
<p>“RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.”</p>
<p>I’m using an old NVIDIA Quadro K2200. I am using Windows 10. I have already downgraded Cuda Toolkit to 12.1 (because it was the last version listed compatible with Torch) und I am using the Pytorch 1.2.1 version for Cuda 11.8 (as I thought it might support the older GPU). Both of those steps were seperate solutions that I found while googling lots of different ways to try and fix this, nothing has helped so far.</p>
<p>It probably has something to do with my version of Torch and compatibility with my graphics card, but as you can tell I am a complete newbie at this and lost at sea.</p>
<p>Does anyone have a solution to this? If you need any more information, please let me know.</p>
<p>Thanks in advance!</p>

ptrblck: <p>Your <code>Quadro K2200</code> should have a compute capability of 5.0 and would thus be supported in our binary builds. What output do you get when installing the latest PyTorch binary with CUDA 12.1 as well as 11.8 for:</p>
<pre data-code-wrap="python"><code class="lang-python">python -c "import torch; print(torch.randn(1).cuda())"
</code></pre>

durkto: <p>I’m getting the output:"tensor([-0.6384], device=‘cuda:0’).</p>
<p>I didn’t install anything else now, it’s the same setup as before (CUDA 12.1, latest PyTorch binary for CUDA 11.8), but this is probably what you meant, right?</p>

ptrblck: <p>Thanks for confirming the CUDA usage in PyTorch itself. Since PyTorch is able to use your GPU properly, a 3rd party library you’ve installed might not have built its CUDA kernels for compute capability 5.0 and thus fails.</p>
_pickle.UnpicklingError: A load persistent id instruction was encountered, but no persistent_load function was specified

kiyaabdi: <p>when i wanna convert pkl model to pth model I receive an error and don’t know how to solve it.<br>
here is my code:<br>
"<br>
import pickle<br>
import sys<br>
from enum import Enum<br>
from pathlib import Path<br>
from typing import Optional</p>
<p>import torch</p>
<p>checkpoint_path = “./train_model/best.pkl”<br>
print(f"Loading StyleGAN3 generator from path: {checkpoint_path}")<br>
with open(checkpoint_path, “rb”) as f:<br>
decoder = pickle.load(f)[‘G_ema’].cuda()<br>
print(‘Loading done!’)</p>
<p>state_dict = decoder.state_dict()<br>
torch.save(state_dict, “./train_model/model.pth”)<br>
print(‘Converting done!’)<br>
"</p>
<p>and my error is here:</p>
<p>Loading StyleGAN3 generator from path: /media/storage/tracking_codes/Siames/train_model/best.pkl<br>
/usr/lib/python3/dist-packages/requests/<strong>init</strong>.py:89: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn’t match a supported version!<br>
warnings.warn("urllib3 ({}) or chardet ({}) doesn’t match a supported "<br>
Traceback (most recent call last):<br>
File “serial.py”, line 12, in <br>
decoder = pickle.load(f)[‘G_ema’].cuda()<br>
_pickle.UnpicklingError: A load persistent id instruction was encountered,<br>
but no persistent_load function was specified.</p>

bgendron: <p>I used to struggle with the same problem but finally found a workaround by combining several available solutions on similar problems. Here is what I do:</p>
<ul>
<li>First, I add the argument <code>_use_new_zipfile_serialization=False</code> in <code>torch.save()</code>:</li>
</ul>
<pre><code class="lang-auto">torch.save(vocab, '../objects/vocab.pt', _use_new_zipfile_serialization=False)
</code></pre>
<ul>
<li>Then, I use the custom Unpickler class from <a href="https://github.com/pytorch/pytorch/issues/16797#issuecomment-633423219" rel="noopener nofollow ugc">this GitHub issue</a>:</li>
</ul>
<pre><code class="lang-auto">class CPU_Unpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module == 'torch.storage' and name == '_load_from_bytes':
            return lambda b: torch.load(io.BytesIO(b), map_location='cpu')
        else: return super().find_class(module, name)
</code></pre>
<ul>
<li>Therefore, here is how I unpickle my torch object:</li>
</ul>
<pre><code class="lang-auto">with open('../objects/vocab.pt', 'rb') as f:
    vocab = CPU_Unpickler(f).load()
</code></pre>
<p>Hope this helps.</p>
Discrepancy Between Expected and Actual GPU Memory Usage for Large Tensors

MoJo494: <p>Hi all,</p>
<p>I have noticed that the actual GPU memory consumed by large tensors is more than what is theoretically expected.</p>
<p>Example:</p>
<p>For instance, a Float32 tensor with 2.900.000 elements (shape <code>(2.900.000, 1)</code>) occupies approximately 12.288 KiB of GPU memory. However, based on my calculations (2.900.000 elements * 4 bytes), the expected memory usage should be around 11.328,125 KiB.</p>
<p>Here’s the code that reproduces the pattern:</p>
<pre><code class="lang-auto">import torch

# Specify the size of the tensor
tensor_size = (2900000, 1)
  
# Create a random tensor
random_tensor = torch.rand(tensor_size)
  
# Expected size
expected_size = tensor_size[0] * tensor_size[1] * random_tensor.element_size() / 1024
  
# Get actual occupied memory
random_tensor = random_tensor.to("cuda")
actual_size = torch.cuda.memory_allocated()/1024

# Print results
print(f"Expected occupied GPU memory by tensor: {expected_size} KiB")
print(f"Actual occupied GPU memory by tensor: {actual_size} KiB")
</code></pre>
<p>Is there any explanation for that?</p>
<p>Thank you very much!</p>
Adjusting Output Size of Auxiliary Matrix in Python

yari: <p>In the below code, after hidden layer activation, I want to create an auxiliary matrix to better capture the temporal aspect of the data. After computation, the current shape of the return variable is <strong>[16,16]</strong> that is [out_channels , out_channels] but I want the returned shape to be <strong>[4, 16]</strong> that is [input_channels , out_channels]. Which part of the code should be modified to achieve the desired output while keeping the idea/logic as it is?</p>
<pre><code class="lang-auto">def my_fun( self, H: torch.FloatTensor,) -&gt; torch.FloatTensor:
    
    self.input_channels = 4
    self.out_channels = 16
    self.forgettingFactor = 0.92
    self.lamb = 0.01
    self.M = torch.inverse(self.lamb*torch.eye(self.out_channels))
    HH = self.calculateHiddenLayerActivation(H) # [4,16]
    Ht = HH.t() # [16 , 4]
    ###### Computation of auxiliary matrix 
    
    initial_product = torch.mm((1 / self.forgettingFactor) * self.M, Ht) # [16, 4]
    intermediate_matrix = torch.mm(HH, initial_product )   # [4, 4]
    sum_inside_pseudoinverse = torch.eye(self.input_channels) + intermediate_matrix # [4, 4]

    pseudoinverse_sum = torch.pinverse(sum_inside_pseudoinverse) # [4, 4]
    product_inside_expression = torch.mm(HH, (1/self.forgettingFactor) * self.M) #  [4, 16]              
    dot_product_pseudo = torch.mm( pseudoinverse_sum  , product_inside_expression) # [4, 16]   
    dot_product_with_hidden_matrix = torch.mm(Ht, dot_product_pseudo ) # [16, 16]

    res = (1/self.forgettingFactor) * self.M - torch.mm((1/self.forgettingFactor) * self.M, dot_product_with_hidden_matrix  ) # [16,16]        

    return res
</code></pre>

smth: <p>just use H, and not Ht, and invert the matmul order when computing the <code>dot_product_with_hidden_matrix</code>.</p>
<pre><code class="lang-auto">dot_product_with_hidden_matrix = torch.mm(dot_product_pseudo, H) # [4, 16]
res = (1/self.forgettingFactor) * self.M - torch.mm((1/self.forgettingFactor) * self.M, dot_product_with_hidden_matrix.t()  ) # [16, 4]        
return res.t() # [4, 16]
</code></pre>

yari: <aside class="quote no-group" data-username="yari" data-post="1" data-topic="194943">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yari/48/60560_2.png" class="avatar"> yari:</div>
<blockquote>
<p><code>initial_product</code></p>
</blockquote>
</aside>
<p>If I just use <em>H</em>, I am unable to perform the second step after ‘initial_product’ due to the same shape matrics.</p>
Dataset not pickable error

shadow: <p>Hi All,</p>
<p>I am encountering the following error:</p>
<p>NotImplementedError: Dataset is not picklable</p>
<p>This error showed up when i tried to build the <strong>getitem</strong> method as follows:<br>
x, y = self.get_data(item)<br>
sample = x.unsqueeze(0).to(device)<br>
pred_target = test_model(sample).detach().cpu()<br>
pred_target = pred_target.squeeze(0).squeeze(0)<br>
corr_target = y - pred_target<br>
print("item is : "+str(item))<br>
return x, corr_target</p>
<p>The idea from this implementation is to get the difference from y and another quantity calculated using another NN put in evaluation mode. Please note that before adding the second component (where another NN is used) the training worked properly.<br>
Thanks in advance</p>
Calculation of flops during checkpoint activation of fsdp

Taishi_Nakamura: <p>How to calculate the calculation of flops during checkpoint activation of fsdp<br>
In particular, the model of GPT series</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/2205.05198">
  <header class="source">
      <img src="https://arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="" height="">

      <a href="https://arxiv.org/abs/2205.05198" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail" width="" height=""></div>

<h3><a href="https://arxiv.org/abs/2205.05198" target="_blank" rel="noopener nofollow ugc">Reducing Activation Recomputation in Large Transformer Models</a></h3>

  <p>Training large transformer models is one of the most important computational challenges of modern AI. In this paper, we show how to significantly accelerate training of large transformer models by reducing activation recomputation. Activation...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
For example, in megatron lm, when the activation is selective, an additional calculation of 4Bs^2h is required<br>
What happens in the case of fsdp?</p>
FasterRCNN masking labels during training

Valentin_Thorey: <p>Hi, I am using the FasterRCNN pretrained model. I have one detection model detecting object A and another detecting object B. I want to create a single model. The two models are trained on the same type of data (medical imaging) but the dataset of A does not contain labels for B and vice versa (while B object canbe present on the image containing A).</p>
<p>I did not see a way to mask/ignore some labels during loss computation, is this something planned or is there a way to easily achieve that (i.e.: without having to annotate dataset A for B object)?</p>
LSTM Autoencoder - Encoder Output shape

Chris1412: <p>Hi Community!</p>
<p>I hope it’s okay to just post my question here, as I couldn’t find a suitable subforum at first.<br>
I have problems creating an autoencoder with LSTM layers.</p>
<p>An LSTM returns the following output: <code>outputs, (hn, cn) = self.LSTM(...)</code></p>
<p>Since the last hidden state <code>hn</code> can be used as input for the decoder in an autoencoder I have to transform it into the right shape.<br>
I have seen many different approaches on the Internet and am now unsure how to proceed.</p>
<p>According to the Pytorch documentation, the hidden state hn has the following shape: <code>(1*num_layers, batch_size, hidden_size) # unidirectional and batch_first=True</code></p>
<p>How do I reshape the hidden state to pass it to the decoder as a compressed vector in the correct shape?</p>
<p>Thanks in advance for any help!</p>
<p>Kind regards,<br>
Christopher</p>

J_Johnson: <p>Welcome to the forums!</p>
<p>You won’t need to reshape anything if the hidden size of the LSTM decoder is the same size as the hidden of the encoder.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/016527c3f386488e07ffb461410761b0fd511e63.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/016527c3f386488e07ffb461410761b0fd511e63" title="IMG_20240108_084052"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/016527c3f386488e07ffb461410761b0fd511e63_2_690x429.jpeg" alt="IMG_20240108_084052" data-base62-sha1="clcvCgaFgE3pUikfxUWi1NVk1J" width="690" height="429" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/016527c3f386488e07ffb461410761b0fd511e63_2_690x429.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/1/016527c3f386488e07ffb461410761b0fd511e63.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/1/016527c3f386488e07ffb461410761b0fd511e63.jpeg 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20240108_084052</span><span class="informations">920×573 98.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If the sizes aren’t the same, you can pass it through a Linear layer as is(no need to reshape or permute; Linear layers are only concerned with the final dim size).</p>
<pre><code class="lang-auto">hidden_nn = nn.Linear(enc_hidden, dec_hidden)

h0 = hidden_nn(hn)
</code></pre>
<p>The above is a simple case, but you may also try using multiple Linear layers with activations before continuing to the decoder.</p>

Chris1412: <p>Thank you very much for your response!<br>
Now it works as expected <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Channel Wise Pruning Error

satabios: <p>I’m trying to prune a CNN network channel-wise.</p>
<p>When I try on VGG, it works perfectly; however, when I try the exact implementation on mobilenetV2(timm), it throws an error that my input fed and the layer shape isn’t compatible.</p>
<p>RuntimeError: Given groups=32, expected weight to be at least 32 at dimension 0, but got a weight of size [29, 1, 3, 3] instead.</p>
<p>I dug into the issue and was inspecting the shapes layer by layer. The layer that is anomalous is self.conv_dw.weight ({Conv2d} Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False).</p>
<p>I checked the layer <strong>weight shape: {Size: 4} torch.Size([29, 1, 3, 3]),</strong> while the <strong>shape of input</strong> being fed is  <strong>X = {Tensor: (2, 29, 16, 16)}.</strong></p>
<p>As far as I can see, the input and the layers are compatible with conv2d. Am I missing something here?</p>
<p>Attaching the code here: <a href="https://github.com/satabios/sconce/blob/main/test/model_analyzer.py" rel="noopener nofollow ugc">Channel Wise Pruning</a></p>

satabios: <p>The problem was resolved, the Conv groups was causing the trouble. Once I set the pruned number of groups it worked flawlessly.</p>
How to avoid graph break when i use `UserDefinedClassVariable`

max_nuke: <p>I’m trying to create a custom tensor class that subclasses <code>torch.Tensor</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">class CustomTensor(torch.Tensor):
    @staticmethod
    def __new__(cls, x, *args, **kwargs):
        return super().__new__(cls, x, *args, **kwargs)
    
    def __init__(self, x):
        super().__init__() # optional
        self.foo = None
    def set_foo(self, foo: str):
        self.foo = foo
        

def module(t):
    b = torch.add(t, torch.tensor([2]))
    b = CustomTensor(b)
    b.set_foo("hello")
    return b
</code></pre>
<p>and then I’m attempting to use the <code>torch.compile</code> function<br>
But i got errors like thiis</p>
<pre><code class="lang-auto">    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function UserDefinedClassVariable() [TensorVariable()] {}
</code></pre>
<p>is there a way to implement <code>call_function UserDefinedClassVariable() [TensorVariable()]</code> in dynamo?</p>
Only one layer of weights update, why?

Michelangelo_Nardi: <p>I create a NN with one hidden layer, so I have two tensor of weights.<br>
Then I train it, so I expect the weights to update. The problem is that only the second set of weights updates, the first stays the same!<br>
Why? I have tried different possible solutions, but nothing.</p>
<p>Thank you for your help.</p>
<p>This is the code (I removed the imports and the dataset upload), I print “par” before training and after training, it is clear that the first set of weights did not change:</p>
<pre><code class="lang-auto">class NN1(nn.Module):
    def __init__(self, D_1, D_2, H, D_out):
        super().__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(D_1*D_2, H, bias=False),
            nn.ReLU(),
            nn.Linear(H, D_out, bias=False),
        )
        

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits


model = NN1(28, 28, 756, 10)
par = list(model.parameters())
print(par)


def train_loop(model, training_data, batch_size, eta, epochs):
    
    optimizer = torch.optim.SGD(par, lr=eta)
    loss = nn.CrossEntropyLoss()

    train_dataloader = DataLoader(training_data, batch_size=batch_size, shuffle=True)

    for epoch in range(epochs):
        for batch_idx, (X, y) in enumerate(train_dataloader):
            ypred = model(X)
            train_loss = loss(ypred, y)
            if batch_idx % (6400/batch_size) == 0:
                print(f'Epoch [{epoch + 1}/{epochs}] Batch [{batch_idx}/{len(train_dataloader)}] Training loss: {train_loss.item()}')
            
            optimizer.zero_grad()
            train_loss.backward()
            optimizer.step()
    
    return model


print(par)
</code></pre>

ptrblck: <p>Just printing the parameter might not show enough decimals so clone the original parameter and subtract the updated one from it. Also check the gradients via the <code>.grad</code> attribute to see how small they are.</p>

Michelangelo_Nardi: <p>Thank you very much.</p>
<p>Unfortunately I have tried doing both things, but nothing. When I clone the original parameter and subtract the updated one from it I get a tensor of all 0, so they are perfectly equal I believe (and instead for the second set of parameters it’s different from 0).</p>
<p>When I print the gradients during the training loop they appear to be exactly 0 (I don’t know if there is an approximation that I can’t see, but I don’t think so).</p>

Michelangelo_Nardi: <p>PROBLEM SOLVED<br>
By reading some other posts I understood the problem. Actually there was not really a problem. When I was printing the gradients and/or the weights, the output was clearly truncated, and the part which I was seeing was simply not updated, but other parts which I was not seeing where being updated.</p>
Using swin transformers on timm library in image segmentation

Rabee_Qasem: <p>How do I modify the output shape of a <code>TIMM</code> model for image segmentation in the medical domain using the <code>Kvasir-SEG</code> dataset and PyLops? I have defined the <code>num_classes=0</code> in the TIMM create model, but during training the output size of the logits is<code> torch.Size([32, 768])</code>. I need to change the model output to match the mask shape, which is <code>torch.Size([32, 3, 224, 224])</code>. Is there a way to do this within the TIMM library or do I need to implement additional modifications to the model architecture so that I can compute the loss?</p>
<pre><code class="lang-auto"># Load the model
model=timm.create_model("swin_s3_tiny_224", pretrained=True,num_classes=0)

# Set the model to evaluation mode
#model.eval()

# Move the model to the device
model.to(device)

# Define the loss function and the optimizer
loss_fn = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)

</code></pre>
<p>the training code</p>
<pre><code class="lang-auto">num_epochs = 2

# Set the lists to store the training and validation losses
train_losses = []
valid_losses = []
# Loop over the epochs
for epoch in range(num_epochs):
    # Set the model to training mode
    model.train()

    # Set the training loss to 0
    train_loss = 0
    



    # Loop over the training data
    for image, mask in dataloader_train:

        # Move the data to the device
        image = image.to(device)
        mask = mask.to(device)

        #image shape= torch.Size([32, 3, 224, 224])
        #mask shape= torch.Size([32, 3, 224, 224])


        # Zero the gradients
        optimizer.zero_grad()

        # Compute the logits
        #logits = model(image)
        logits=model(mask)
        
        #logits shape torch.Size([32, 768])
        

        # Compute the loss
        loss = loss_fn(logits, mask)
        

        # Backpropagate the loss
        loss.backward()

        # Update the parameters
        optimizer.step()

        # Accumulate the training loss
        train_loss += loss.item()



</code></pre>

sikandar_afridi: <p>hello Rabee_Qasem i have similar issue. can you please let me know how did you resolve the issue.</p>
Detaching one of the operands of a tensor multiplication from a computation graph causes "RuntimeError: Function 'MulBackward0' returned nan values in its 1th output."

barikata1984: <p>Hi all,</p>
<p>Can anyone help me out with how to avoid the error in the subject line? I built a function, which is called inside the forward pass at every iteration, like:</p>
<pre><code class="lang-auto">def psudoe_fn(input):
    feat_vec = generate_feat_vec(input)
    density, aux = predict_density(feat_vec)  # ~ (batch, 1)
    distr = predict_distr(feat_vec)
    return density * distr 
</code></pre>
<p>The function above itself worked nicely. What I’m trying to do now is detaching <code>density</code> from the computation graph. I modified the line <code>density, aux = predict_density(feat_vec)</code> to:</p>
<pre><code class="lang-auto">    with torch.no_grad():
        density, aux = predict_density(feats)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">    with torch.autograd.set_grad_enabled(False):
        density, aux = predict_density(feats)
</code></pre>
<p>But, when I run the code <code>with torch.autograd.detect_anomaly()</code>, I get the <code>MulBackward0</code> at <code>return density * distr</code> for both modifications.</p>
<p>I have checked both <code>density</code> and <code>distr</code> do not have <code>nan</code> or <code>inf</code> inside with <code>.isan().any()</code> and <code>.isinf().any()</code>. How can I resolve the error?</p>
<p>P.S. The output of <code>psudoe_fn</code> is further processed later, including calculations of kl_div and the total variation for the sake of regularization. The neural network is optimized with adam augmented with Nvidia <code>apex</code> library.</p>
A question on how to use at::Tensor & clamp_min_ function

weitaoliu: <p>Hey,</p>
<p>I am trying to use at::Tensor &amp; clamp_min_ function which takes two arguments, one is the tensor passed by reference and the other one is the scalar passed by the const reference.<br>
The inline function is:</p>
<pre><code class="lang-auto">inline at::Tensor &amp; clamp_min_(at::Tensor &amp; self, const at::Tensor &amp; min) {
     return at::_ops::clamp_min__Tensor::call(self, min);
}
</code></pre>
<p>In my codes, I tried to call by doing this:</p>
<pre><code class="lang-auto">const at::Scalar testmin(0);
torch::Tensor::clamp_min_(yPreds_ts, testmin);
</code></pre>
<p>However, the error appears when compiling, which follows:</p>
<pre><code class="lang-auto">error: no matching function for call to ‘at::Tensor::clamp_min_(at::Tensor&amp;, const c10::Scalar&amp;)’
  207 |     torch::Tensor::clamp_min_(yPreds_ts, testmin);
</code></pre>
<p>I am not sure of the reason. Has anyone met a similar issue before?</p>
<p>Thanks, Weitao.</p>

smth: <p>if there’s a trailing underscore, its a member of the tensor.</p>
<p>Can you try this instead:</p>
<pre data-code-wrap="c++"><code class="lang-c++">yPreds_ts.clamp_min_(testmin)
</code></pre>

weitaoliu: <p>yes, <code>yPreds_ts.clamp_min_(testmin)</code> works well. Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
How to convert a list of lists to a Pytorch tensor?

Chathurangi_Shyalika: <p>Hello,</p>
<p>I have a list of lists like this.</p>
<pre><code class="lang-auto">[[[1690.0, ['R04', 'crashed', 'nose']]],
 [[1690.0, ['No', 'nose']], [1710.0, ['R04', 'crashed', 'nose']]],
 [[1690.0, ['R03', 'crashed', 'nose']],
  [1710.0, ['Part','missing]]]
</code></pre>
<p>If I try to convert it using <code>torch.tensor(p_x)</code> it gives error <code>TypeError: new(): invalid data type 'str'</code></p>
<p>How do I convert this to a Pytorch tensor?</p>
<p>Thanks</p>

smth: <p>pytorch tensors can’t hold strings. you’ll have to first figure out what to do with those strings before converting them to a pytorch tensor.</p>

ashok_arora: <p>Use some kind of encoding or embedding to convert the string into a vector and then it can be converted into a tensor.</p>
Maximum pixel value of RGB channel

cbd: <p>Image is read as [1,3,256,256].<br>
(1) How can i find maximum pixel value in each channel (max value of each RGB channel) and<br>
(2) Pixel value with maximum RGB value</p>
<p>For example, if  i have two pixel with value (255,250,190) and (240,240,240) then in first case output should be (255,250,240) and in second case (240,240,240).</p>

smth: <pre data-code-wrap="python"><code class="lang-python">image = torch.tensor([[255, 250, 190], [240, 240, 240]], dtype=torch.int32).T.view(1, 3, 2, 1)

# 1. Maximum Pixel Value in Each RGB Channel
max_per_channel = torch.amax(image, dim=(0, 2, 3))

# 2. Pixel Value with Maximum RGB Value
# Reshape to separate each pixel and sum across channels
sum_per_pixel = image.view(3, -1).sum(dim=0)
# Find the index of the pixel with the maximum sum
max_rgb_index = torch.argmax(sum_per_pixel)
# Retrieve the pixel with the maximum RGB sum
max_rgb_pixel = image.view(3, -1)[:, max_rgb_index]

print(max_per_channel, max_rgb_pixel)
</code></pre>
Hello, can dnn-beamforming be converted to onnx or libtorch and deployed in C++? Does libtorch or onnx support MVDR conversion?

wrz1999: <p>Hello, can dnn-beamforming be converted to onnx or libtorch and deployed in C++? Does libtorch or onnx support MVDR conversion?</p>

smth: <p>It’s better to ask this question on the issue tracker of the project you mention. We wouldn’t have the answers you’re looking for.</p>
IndexError: Target 264 is out of bounds

jeffreyleett: <p>i am going to feed input shape is torch.Size([8, 1387, 4]) and output shape is torch.Size([8, 1387, 1]) in this model but it occur this errorin loss = criterion(outputs, batch_labels), sorry for new to machine learning<br>
now the batch_label and output shape is</p>
<pre><code class="lang-auto">torch.Size([8])
torch.Size([8, 25])
</code></pre>
<pre><code class="lang-auto">IndexError: Target 264 is out of bounds.
</code></pre>
<p>BiLSTM(4,64,2,25)</p>
<pre><code class="lang-auto">class BiLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(BiLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(
            input_size, hidden_size, num_layers, batch_first=True, bidirectional=True
        )
        self.fc = nn.Linear(hidden_size * 2, num_classes)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers * 2, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers * 2, x.size(0), self.hidden_size).to(device)

        out, _ = self.lstm(x, (h0, c0))
        out = self.fc(out[:, -1, :])

        return out
        
    
    def train_epoch(self, num_epochs, optimizer, criterion, train_dataloader, device):
        self.to(device)
        for epoch in range(num_epochs):
            self.train()
            total_loss = 0.0
            progress_bar = tqdm(train_dataloader, desc=f'Epoch {epoch + 1}/{num_epochs} training', unit='batch')

            for batch_features, batch_labels in progress_bar:
                optimizer.zero_grad()
                batch_features = batch_features.to(device).float()
                outputs = self(batch_features)
                #outputs = torch.permute(outputs,(0,2,1))
                batch_labels = batch_labels.squeeze().long().argmax(dim=1).to(device)
                print(batch_labels.shape)
                print(outputs.shape)
                loss = criterion(outputs, batch_labels)
                loss.backward()
                optimizer.step()
                total_loss += loss.item()
                progress_bar.set_postfix(loss=total_loss / len(progress_bar))

            epoch_loss = total_loss / len(train_dataloader)
            print(f'Epoch {epoch + 1}/{num_epochs}, Training Loss: {epoch_loss:.4f}')

</code></pre>

ptrblck: <p>Your output shape indicates you are working with 25 classes which would define the target values in the range <code>[0, 24]</code>. However, based on the error message your target contains at least one value of 264 which is out of bounds.</p>
"Could not find a version that satisfies the requirement torch (from versions: none)" - help

binary-blazer: <p>can someone help me?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b90dd9b441b199d73d835e3945e167f4d31beb3b.png" data-download-href="https://discuss.pytorch.org/uploads/default/b90dd9b441b199d73d835e3945e167f4d31beb3b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b90dd9b441b199d73d835e3945e167f4d31beb3b_2_689x127.png" alt="image" data-base62-sha1="qp40ozyjsPWy7iSDznMPz8MB9ZV" width="689" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b90dd9b441b199d73d835e3945e167f4d31beb3b_2_689x127.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b90dd9b441b199d73d835e3945e167f4d31beb3b_2_1033x190.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b90dd9b441b199d73d835e3945e167f4d31beb3b.png 2x" data-dominant-color="232120"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1374×253 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Which Python version are you using? If it’s 3.12+, downgrade to 3.11-.</p>

binary-blazer: <p>Thank you! Now the installation started properly. I will reply to you if it wasn’t successful.</p>
Understanding Conv2d and ConvTranspose2d

xwkuang5: <p>Hi, I am trying to use <code>ConvTranspose2d</code> to reverse the operation performed by <code>Conv2d</code> by using the weight in <code>Conv2d</code> to initialize <code>ConvTranspose2d</code>.</p>
<p>I am reading <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="nofollow noopener">A guide to convolution arithmetic for deep<br>
learning</a> and came up with the following code to test my hypothesis about <code>Conv2d</code> and <code>ConvTranspose2d</code>. I thought that my code below should allow me to apply downsampling to an image and then perform upsampling to get the original image back. However, that does not seem to be the case. Can anyone share some insights about understanding <code>Conv2d</code> and <code>ConvTranspose2d</code>?</p>
<pre><code class="lang-auto">import torch
from torch.nn import Conv2d, ConvTranspose2d

img = torch.rand(1, 1, 3, 3, requires_grad=False)
downsample = Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=0, bias=False)
upsample = ConvTranspose2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=0, bias=False)

# should I use .t() ? 
upsample.weight.data = downsample.weight.data

out = downsample(img)
inv_img = upsample(out)

torch.allclose(img, inv_img)
</code></pre>

thib-s: <p>Answering this old thread for the sake of completeness:</p>
<p>Your kernel needs extra properties to have this true. Let’s split this answer into two parts: one about the conv transpose and one about computing the inverse of a linear application.</p>
<p>Convolutions can be written as a specific fully connected layer with sparse, repeated entries (let’s call it W). Conv transpose computes efficiently the transpose of W. This is not always equivalent to transposing the kernel since there is padding, stride, and dilation.</p>
<p>Then having <code>convtranspose(conv(x)) = x</code> requires you to have<code> W*Wt = identity</code> which is not the case in general. This would require <code>Wt = W^-1</code>, but<code> W^-1</code> only exists for square matrices (so convolutions with as many inputs as outputs, otherwise you can fall back on the Moore-Penrose pseudo inverse). Again, computing the kernel inverse <code>K^-1</code> is not equivalent to computing <code>W^-1</code>. Finally, there are no guarantees that <code>W^-1</code> will still have the structure of a convolution (with the sparsity and repeated entries).</p>
<p>Finally, this does not mean that there is no kernel such that <code>convtranspose(conv(x)) = x</code>. Just that finding such a kernel is very non-trivial. One quick and cheap way you could try would be to add a regularization loss that would enforce this.</p>
Expanding a tensor in dim 0 using torch.nn

S_M: <p>Hi all, how can I change a tensor [512,512,3,3] to [1024,512,3,3] using torch.nn?<br>
any help would be really appreciated.</p>

ptrblck: <p>Usually <code>dim0</code> represents the batch dimension and thus contains the samples. Increasing the sample count of the batch via neural network operations sounds uncommon so could you describe what the use case is and if you really want to increase the number of samples?</p>

S_M: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>, I try to use the output of intermediate layer. The dimension of this output is 512,512,3,3 I need to compare the distance between this tensor to another one with shape 1024,512,3,3. I need to convert these tensors to a same shape. How can I do that?</p>

ptrblck: <p>You could e.g. <code>repeat</code> or <code>interpolate</code> the tensor, but more importantly you should still check what this dimension represents and if you really want to manipulate this dimension as it’s usually the batch dimension.</p>
How to extract the intermediate layers of vgg16 model

sha: <p>I am trying to extract intermediate feature layers of vgg16 using python on spyder IDE, and then looking to compress the feature map.</p>
<p>Looking for answers,…</p>

HDCharles: <p>do you mean the weights or the activations? for the activations i posted a method here: <a href="https://discuss.pytorch.org/t/i-saved-the-quantized-weight-and-loaded-it-with-the-model-after-torch-ao-quantization-convert-how-do-i-print-the-output-of-each-layer-of-the-network/194576" class="inline-onebox">I saved the quantized weight and loaded it with the model after torch.ao.quantization.convert(). how do I print the output of each layer of the network?</a></p>
<p>for hte weights you can just print them with something like</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
   print(name, param)
</code></pre>
Reshape(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

osher: <pre><code class="lang-auto">@override(ModelV2)
def forward(self, input_dict, states, seq_lens):
    obs_a = input_dict["prev_n_obs_a"]
    obs_b = input_dict["prev_n_obs_b"]

    obs_a = torch.reshape(obs_a, [-1, obs_space_a.shape[0] * self.num_frames])
    obs_b = torch.reshape(obs_b, [-1, obs_space_b.shape[0] * self.num_frames])
</code></pre>
<p>ERROR</p>
<pre><code class="lang-auto">in __call__
    res = self.forward(restored, state or [], seq_lens)
  File "/home/niss/proj/lotlatp/pbmain.py", line 116, in forward
    obs_a = torch.reshape(obs_a, [-1, obs_space_a.shape[0] * self.num_frames])
TypeError: reshape(): argument 'input' (position 1) must be Tensor, not numpy.ndarray
</code></pre>

ptrblck: <p>Did you try to pass a tensor instead of a <code>numpy.array</code> as described in the error message?<br>
You can create tensors from <code>numpy.array</code>s via <code>tensor = torch.from_numpy(array)</code>.</p>
Which CUDA driver APIs or CUDA runtime APIs Pytorch calls upon during training or predicting

shysuen: <p>Hello developers,<br>
May I ask which CUDA driver APIs or CUDA runtime APIs Pytorch calls upon during training or predicting?</p>

ptrblck: <p>You could search for all <code>cuda*</code> and <code>cu*</code> calls in the code base, profile your workload and check the kernel calls in e.g. Nsight Systems, or be more specific with your question in case you are interested in a few specific calls.</p>
Python3.1 directory in torch docker images /opt/conda/lib

AwePhD: <p>Hello,</p>
<p>In the directory <code>/opt/conda/lib</code> there are <code>python3.1</code> and <code>python3.10</code> directories, for the torch images using Python 3.10. The 3.1 directory is a symbolic link to the 3.10 folder. I have other images, <code>pytorch/pytorch:1.8.0-cuda11.1-cudnn8-devel</code> for example, and there is a single <code>python3.8</code> directory in the <code>/opt/conda/lib</code>.</p>
<p>So I would like to new the reason of the <code>python3.1</code> folder. Is it to solve this <a href="https://github.com/pypa/setuptools/pull/2953" rel="noopener nofollow ugc">setuptools</a> issue in the case where an old package would be installed in the <code>python3.1</code> instead of <code>python3.10</code>?</p>

smth: <blockquote>
<p>So I would like to new the reason of the <code>python3.1</code> folder. Is it to solve this <a href="https://github.com/pypa/setuptools/pull/2953">setuptools </a> issue in the case where an old package would be installed in the <code>python3.1</code> instead of <code>python3.10</code> ?</p>
</blockquote>
<p>It is likely this reason. I can’t think of another reason why we’d do that.</p>
Torch.cuda.is_available() is always False for 12.1

botab: <p>Hi everyone,</p>
<p>I am having troubles setting up CUDA for my windows computer. I am unsure what the issue is as I tried to debug the problem with some of the solutions on the forum.</p>
<blockquote>
<p>Blockquote<br>
torch.utils.collect_env<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff37fe1ce2cbbd2164d33fcdfbce8dbcb08deb1f.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff37fe1ce2cbbd2164d33fcdfbce8dbcb08deb1f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff37fe1ce2cbbd2164d33fcdfbce8dbcb08deb1f.png" alt="image" data-base62-sha1="ApLIymCEojVwKXQD1aT8x4X0uGb" width="690" height="263" data-dominant-color="F8F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">912×348 8.76 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Blockquote</p>
</blockquote>
<p>any help is appreciated</p>

botab: <blockquote>
<p>nvidia-smi<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2cca31240135d1f99372c9da1054eb2568592f4.png" alt="image" data-base62-sha1="nebMxGDMKBHc9BYclxxota81NIg" width="647" height="385"></p>
</blockquote>

botab: <p>install with the following code.</p>
<blockquote>
<p>conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia<br>
conda install cuda -c nvidia</p>
</blockquote>

botab: <p>RESOLVED -</p>
<blockquote>
<p>conda activate cuda_venv<br>
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia<br>
conda install -c anaconda cudatoolkit<br>
pip install jupyter<br>
pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a><br>
jupyter notebook</p>
</blockquote>
<p>My problem was that there was there was another kernel I created while following a tutorial on setup and that still has a false value. I swapped back to the python3 ipynkernel.</p>
C/C++ code of F.interpolate

Ardeal: <p>Hi,</p>
<p>I need to convert F.interpolate to C or C++ code. I notice that F.interpolate calls torch._C._nn.upsample_bicubic2d.</p>
<p>May I access torch._C._nn.upsample_bicubic2d C/C++ code? or is there any way to help to convert it C/C++ code?</p>

ptrblck: <p>You can find the host code <a href="https://github.com/pytorch/pytorch/blob/a8dc9d8e353ddcf7db0247349a3acd0dd37fcc6f/aten/src/ATen/native/UpSampleBicubic2d.cpp#L109">here</a> and the GPU code <a href="https://github.com/pytorch/pytorch/blob/a8dc9d8e353ddcf7db0247349a3acd0dd37fcc6f/aten/src/ATen/native/cuda/UpSampleBicubic2d.cu#L24">here</a>.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you!</p>
<p>The bilinear interpolation should be in:<br>
<code>cpu_upsample_linear_backward</code> function in <code>pytorch/aten/src/ATen/native/cpu/UpSampleMoreKernel.cpp</code> file</p>

Ardeal: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I would like to run interpolate or resize in a single core MCU. So, there is no mutil-process/thread.<br>
I only need the very simple C or Cpp (prefer C) code of interpolate or resize.</p>
<p>if the code has only bi-linear interpolate or resize, it is good enough. The only purpose is to run the C code of interpolate in a single core MCU, and the output is same as the interpolate or resize in pytorch or opencv.</p>
<p>where could I find the C code?</p>

smth: <p>you’ll have to refactor the code that was pointed out and write a version for yourself. such code isn’t readily available.</p>
What is the difference between defining a function in pytorch and defining a class that inherits nn.Moudle

mst72: <p>I’m having some trouble writing transfoemer code in the computational attention module,<br>
I can define a class to compute attention in forward like this:</p>
<pre><code class="lang-auto">class CalculateAttention(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self,q, k, v, mask=None, e=1e-12):
     ........
</code></pre>
<p>I can also define a function directly to calculate attention:</p>
<pre><code class="lang-auto">def attention(q, k, v, d_k, mask=None, dropout=None):
    ............
    return output
</code></pre>
<p>I don’t know what the difference is between the two ways, and which way should I use it?</p>

smth: <p>if you have state (such as weights and other buffers) that is alive in between computations, write a Module – because you can keep this state as a member of the Module (like self.weight).<br>
If not and you are writing stateless logic, write a function.</p>
Training in DDP mode hang

GC_L: <p>Hello,</p>
<p>I’m currently training a model with custom ops and I found that the training hangs randomly during training in DDP mode, at loss.backward() while calculating the backward gradient of the custom ops. After a few experiments and I now suspect there might be some issues in the synchronisation of model during back propopagation.<br>
I saw <strong>DDP requires <code>Reducer</code> instances on all processes to invoke <code>allreduce</code> in exactly the same order, which is done by always running <code>allreduce</code> in the bucket index order instead of actual bucket ready order. Mismatched <code>allreduce</code> order across processes can lead to wrong results or DDP backward hang.</strong> in the official torch DDP web, I wonder if I get some help from the community to see the <strong>bucket index and the allreduce order</strong>.</p>
<p>Thanks,<br>
GC</p>