akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Vectorizing pack_padded_sequence

bacav: <p>Hi,</p>
<p>I am trying to run a GNN with the following dataset structure per graph (batch_size=1):<br>
node_fts: [207, 2],<br>
edge_fts: [1722]<br>
edges: [2, 1722]</p>
<p>I want to aggregate node_fts of a node’s neighborhood to pass into LSTM, but the neighborhood sizes are not equal. For example, first node is connected to 12 nodes while another is connected to 5. I am currently using for-loops to pad sequences as follows:</p>
<pre><code class="lang-auto">seqs, lenghts = [], []
for node_idx in range(node_fts.shape[0]):
    seqs.append(
        dst_node_fts[
            torch.argsort(
                edge_fts[
                    :,
                ][edges[0] == node_idx],
                descending=True,
            )
        ]
    )
    lenghts.append(len(seqs[-1]))
lenghts = torch.tensor(lenghts)
seqs = pad_sequence(seqs, batch_first=True)
packed = nn.utils.rnn.pack_padded_sequence(
    seqs, lenghts.to("cpu"), batch_first=True, enforce_sorted=False
)
</code></pre>
<p>This is of course absurdly slow since with batch size 64 we loop for 13248 times. Is there a way to vectorize this operation? I can vectorize the lengths tensor as:</p>
<pre><code class="lang-auto">lengths = torch_scatter.scatter_add(
    torch.ones(batch.edge_index.shape[1]),
    batch.edge_index[0],
    dim=0,
    dim_size=batch.x.shape[0],
)
</code></pre>
<p>This gives me a tensor of neighborhood sizes per node. And then:</p>
<pre><code class="lang-auto">dst_node_fts = node_fts[edges[1]]
max_neighborhood_size = lengths.max().item() # 19

# tensor of zeros with shape (num_nodes, max_neighborhood_size=19, node_fts_dim=2)
neighborhood_tensor = torch.zeros((batch.x.shape[0], int(max_neighborhood_size), 2))

neighborhood_tensor[node_idx] = dst_node_fts[
    torch.argsort(
        edge_fts[
            :,
        ][edges[0] == node_idx]
    )
]
</code></pre>
<p>The last block <code>neighborhood_tensor[node_idx]</code> is where I’m stuck. I put the <code>node_idx</code> as illustrative purpose for what I’m trying to accomplish. Is there a way to do this without loops?</p>
Convering my 2D CNN model to 3D

MType04: <p>class Skyrmion_or_Not(Dataset):</p>
<pre><code>def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

def __len__(self):
    return len(self.annotations)

def __getitem__(self, index): # return specific image and target for image
    
    # Some aspect of the code ignores the first row of the csv file and treats as the header
    
    img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
    # pytorch chooses index. idea sytem looks up file name in the labels file 
    # and then finds corresponding image/file in the relevant folder
    image = np.load(img_path)
    y_label = torch.tensor(int(self.annotations.iloc[index, 1])) # reading column 1 as the label

    if self.transform:
        image = self.transform(image)

    return (image, y_label) # the y_label is it being assigned 0, 1.
</code></pre>
<p>This is my current custom dataset for my 2D case. I now have 3D images (dimensions 30 x 30 x 30 x 3), however I am unsure what to adjust here.</p>
<p>Folloeing on from this, how does one visualise a pure 3D image on matplotlib. I am aware of how I can do it on plotly, however, I want to visualise batches (previously used plt.imshow, however this only has 2d inputs). Any help would be much appreciated!</p>

J_Johnson: <p>Can’t think of any reason why you’d need to change anything in the Dataset/Dataloader to go from 2D to 3D.</p>
<p>The model, however, is another story.</p>
RuntimeError: size mismatch, m1: [1 x 1350], m2: [225 x 512] at C:/w/1/s/tmp_conda_3.6_095855/conda/conda-bld/pytorch_1579082406639/work/aten/src\THC/generic/THCTensorMathBlas.cu:290

peony: <p>I am doing action recognition with mediapipe keypoints.<br>
These are the shapes of some of my tensors:</p>
<p>torch.Size([3, 3, 75])<br>
torch.Size([3, 6, 75])<br>
torch.Size([3, 10, 75])<br>
torch.Size([3, 11, 75])<br>
torch.Size([3, 9, 75])<br>
torch.Size([3, 4, 75])<br>
torch.Size([3, 21, 75])</p>
<p>The height of each tensor varies as they refer to the number of frames for each sample. If I understand correctly, my in_features for the layer <code>self.fc_pre = nn.Sequential(nn.Linear(3*75, fc_size), nn.Dropout(p=0.2))</code> below should be 3*75; however, I get a size mismatch error.</p>
<p><code>RuntimeError: size mismatch, m1: [1 x 1350], m2: [225 x 512] at C:/w/1/s/tmp_conda_3.6_095855/conda/conda-bld/pytorch_1579082406639/work/aten/src\THC/generic/THCTensorMathBlas.cu:290</code></p>
<p>I am also using a batch size of 1 for the meantime.</p>
<p>Any advice for me? Thank you in advance.</p>

thecho7: <p>It seems <code>self.fc_pre</code> takes a tensor dimension of [1 x 1350] as an input.<br>
Since 1350 is a multiple of 225, try reshaping the input tensor by</p>
<pre><code class="lang-auto">input = input.view(1, -1, 225)
</code></pre>

peony: <p>Hello, I tried making my in_features 3 x8x75 and it worked. I first changed the height of all my tensors to 8 using this code</p>
<pre><code class="lang-auto">
if height &lt; 8:
            source_pad = F.pad(tensor1, pad=(0, 0, 0, 8 - height))
        else:
            source_pad = F.pad(tensor1, pad=(0,0, 0, 8 - height))
</code></pre>
How to sample a large image in a grid-like fashion?

Rafa: <p>Hi all,</p>
<p>So, I have developed an object detector model. It accepts images with a small dimension, let’s say 520x520. I want to infer with this model in a much larger image. Think an aerial or satellite image.</p>
<p>I’d want to sample batches of the large image in a grid-like fashion. Ideally, with some overlap between each image. I mean, I’d like to be able to select a patch size (like the 520x520) and a stride value.</p>
<p>I’ve used torchgeo before, and it has a class made just for this, the <a href="https://torchgeo.readthedocs.io/en/latest/_modules/torchgeo/samplers/single.html#GridGeoSampler" rel="noopener nofollow ugc">GridGeoSampler</a> class. However, it only seems to work with georeferenced data, which the image I’m working with right now isn’t.</p>
<p>The docs for torchgeo mention that for nongeo data to use torchvision’s modules. However, I’m not finding anything that does this as cleanly. I could probably do it manually, but if there is something already made it’s sure to be better. Is there anything built-in that does this?</p>

ptrblck: <p>I’m not aware of any grid-like transformations in <code>torchvision</code> but to create (overlapping) patches of your input tensors you could use <code>tensor.unfold</code>.</p>
How can I move all zeroes to end of array

yar12900254: <p>I have a variable that looks like<br>
Input:  [[0,2,3,0,0,0,1,2],<br>
[0,0,1,3,0,0,5,0]]</p>
<p>and I want to end up with the following variable<br>
Output: [[2,3,1,2,0,0,0,0],<br>
[1,3,5,0,0,0,0,0]]</p>
<p>I have tried the a[a.nonzero()] if a is the input, but it only works in 1xn dims<br>
Is there a way to do this in PyTorch?</p>

LeviViana: <p>You can do it this way:</p>
<pre><code class="lang-python">import torch
x = torch.Tensor([[0,2,3,0,0,0,1,2], [0,0,1,3,0,0,5,0]])
y = torch.empty(0, x.size(1))
for r in x:
    nz = r.nonzero().squeeze()
    z = torch.zeros(r.numel() - nz.numel())
    z = torch.cat((r[nz], z)).unsqueeze(0)
    y = torch.cat((y, z))

print(y)
</code></pre>
<p>I’m not sure it’s the best way to do it though, but hopefully it could be good enough for your case.</p>

longyikim: <p>x = torch.Tensor([[0,2,3,0,0,0,1,2], [0,0,1,3,0,0,5,0]])<br>
pad = 0</p>
<p>indices = torch.arange(x.shape[1]).unsqueeze(0).repeat(x.shape[0], 1)<br>
indices[x == pad] = x.shape[1] # set to a BIG number</p>
<p>indices = torch.argsort(indices)<br>
y = torch.gather(x, 1, indices)</p>

J_Johnson: <p>Masks are a good approach for this. Here is a simple definition for two dimensions, as in your original example:</p>
<pre><code class="lang-auto">import torch

def zeros2end(x):
    zeros_mask = x == 0
    out = torch.stack(
        [torch.cat([x[_, :][~zeros_mask[_, :]], x[_, :][zeros_mask[_, :]]], dim=0) for _ in range(x.size()[0])])
    return out
</code></pre>
<p>And a test:</p>
<pre><code class="lang-auto">torch.manual_seed(0)
x = torch.randint(3, (2, 10))
print("Before")
print(x)
print("After")
print(zeros2end(x))
</code></pre>
<p>Gives back:</p>
<pre><code class="lang-auto">Before
tensor([[2, 0, 2, 0, 1, 0, 1, 1, 1, 0],
        [2, 2, 0, 0, 1, 2, 0, 0, 2, 0]])
After
tensor([[2, 2, 1, 1, 1, 1, 0, 0, 0, 0],
        [2, 2, 1, 2, 2, 0, 0, 0, 0, 0]])
</code></pre>

LingyuZhang: <pre><code class="lang-auto">x = torch.Tensor([[0,2,3,0,0,0,1,2], [0,0,1,3,0,0,5,0]])
print(torch.gather(x, 1, x.ne(0).argsort(dim=1, descending=True, stable=True)))
</code></pre>
Quanitization for GPU in native pytorch

fabian_schutze: <p>Is there a way to do quantization (mostly 8-bit) on GPUs in native pytorch while avoding TensorRT?</p>
<p>The docs seem to indicate to me that quantization for GPUs is possible only with TensorRT. Is that correct? If not available in main, Is there maybe a PR to work with?</p>
<p>I am grateful for any hints or suggestions.</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/fabian_schutze">@fabian_schutze</a> , we are considering this for future work but we don’t currently have this in a usable form.</p>

jerryzh168: <p>yeah, we don’t have a usable support for native quantized GPU ops, here are some discussions in github as well: <a href="https://github.com/pytorch/pytorch/issues/87395" class="inline-onebox" rel="noopener nofollow ugc">Quantized Inference on GPU summary of resources · Issue #87395 · pytorch/pytorch · GitHub</a></p>

fabian_schutze: <p>Thanks a lot for your replies, <a class="mention" href="/u/vasiliy_kuznetsov">@Vasiliy_Kuznetsov</a>  and <a class="mention" href="/u/jerryzh168">@jerryzh168</a> . They were both very informative.</p>
How to max pool an input to specific length

Ahmad_Pouramini: <p>suppose I have an input (a sentence) with varied lengths. Suppose a = (29 x 512) where 512 is embedding dim.</p>
<p>How can I shrink it to specific length, for example 4 tokens. =&gt; (4 x 512). I thought max pooling can do this, however don’t know how it compress the input to a specific size. Any other solution is also welcomed.</p>

J_Johnson: <p>An AdaptiveMaxPool1d will return the largest 4 values. The alternative is an AdaptiveAvgPool1d.</p>

Ahmad_Pouramini: <p>Thanks. Do you know how is it applied on a dimension? I have <code>inputs_embeds</code> in batch (bs, tokens, dim). I guess I should apply it on tokens, right? Before, I had the following function:</p>
<pre><code>    avg_input_embeds, _ = torch.max(inputs_embeds, 1)
</code></pre>
<p>I expect it match with a MaxPool with size 1:</p>
<pre><code class="lang-auto">
       pool = torch.nn.AdaptiveMaxPool1d(1)
       avg_input_embeds2  = pool(inputs_embeds.permute(0, 2,1)).permute(0,2,1)
</code></pre>
<p>Yet, not equal, however I guess I need some permute or something</p>
<p>Update: Yes, they are equal with unsqueeze</p>
<pre><code class="lang-auto">
pool(inputs_embeds.permute(0,2,1)).permute(0,2,1) == avg_input_embeds.unsqueeze(1)
tensor([[[True, True, True,  ..., True, True, True]],

        [[True, True, True,  ..., True, True, True]],

        [[True, True, True,  ..., True, True, True]],
</code></pre>

J_Johnson: <p>When you want to check if a tensor has all True values, you can use <code>torch.all()</code>:</p>
<pre><code class="lang-auto">import torch

all_true=torch.ones(3,4,5, dtype=torch.bool)
mixed_true=torch.randint(0,1,(3,4,5), dtype=torch.bool)
print(torch.all(all_true))
print(torch.all(mixed_true))
</code></pre>
Error has occured, when I start training schenet model. ValueError: Expected a parent

Takumi3: <p>Hi everyone!, I tried to proceed with the study with my own data using Schnetpack’s tutorial on google colab, but an error occurred! Does anyone know why this message is occured ?<br>
Please let me know how I can fix it. Thank you.<br>
Error message is below.</p>
<pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
&lt;ipython-input-21-036afe972bfc&gt; in &lt;module&gt;
----&gt; 1 trainer.fit(task, datamodule=new_dataset)

6 frames
/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/model_helpers.py in is_overridden(method_name, instance, parent)
     32             parent = pl.Callback
     33         if parent is None:
---&gt; 34             raise ValueError("Expected a parent")
     35     from lightning_utilities.core.overrides import is_overridden
     36 

ValueError: Expected a parent
</code></pre>
<p>Source code (google colabo) is here.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1tmkU_CLn8s59WKFvxRXxgWYSq8l5oxt6?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/905694bf1f106745fcead1aa158a6006/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1tmkU_CLn8s59WKFvxRXxgWYSq8l5oxt6?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1tmkU_CLn8s59WKFvxRXxgWYSq8l5oxt6?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Reference:<br>
schnetpack tutorial</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/atomistic-machine-learning/schnetpack/tree/master/examples/tutorials">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/atomistic-machine-learning/schnetpack/tree/master/examples/tutorials" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/atomistic-machine-learning/schnetpack/tree/master/examples/tutorials" target="_blank" rel="noopener nofollow ugc">schnetpack/examples/tutorials at master · atomistic-machine-learning/schnetpack</a></h3>

  <p><a href="https://github.com/atomistic-machine-learning/schnetpack/tree/master/examples/tutorials" target="_blank" rel="noopener nofollow ugc">master/examples/tutorials</a></p>

  <p><span class="label1">SchNetPack - Deep Neural Networks for Atomistic Systems</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to determine if two GraphModule nodes can be "fused"

mszsorondo: <p>I’m looking to perform fusions on a model’s torch.fx graph, but don’t know exactly what makes nodes in the forward pass trace “fusable”. I see that some do it for conv-&gt;BatchNorm and linear-&gt;BatchNorm, and even with weights and biases of layers. I was thinking to do so with Linear-&gt;LayerNorm and then try with other combinations of nodes, but am not sure about which are the conditions for 2 nodes to be fused.</p>
Pytorch derivative calculation

Leo_SANCHEZ: <p>I have this simple pytorch code:</p>
<pre><code class="lang-auto">x = torch.arange(3,dtype=float)
x.requires_grad_(True)
y = 3*x + x.sum()
y.backward(torch.ones(3))
x.grad
</code></pre>
<p>This gives me [6,6,6], but shouldn’t it be [4,4,4] ? Because if we have f(x)=3 * x0 + 3 * x1 + 3 * x2 + x0+x1+x2, partial derivatives would be 3+1=4 ?</p>

soulitzer: <p>This is a tricky one. The result of x.sum() is implicitly broadcasted to match the shape of 3*x. So y[0] = 3 * x[0] + x[0] + x[1] + x[2] = 4 * x[0] + x[1] + x[2]. If we do y[0].backward(), we would get x.grad = [4, 1, 1], and since we do y.backward(), it would be [4, 1, 1] + [1, 4, 1] + [1, 1, 4] = [6, 6, 6].</p>
Backprop One Layer at a Time

rrichglitch: <p>Hi all,</p>
<p>Im trying to make a somewhat custom optimizing algorithm however Im having a real tough time figuring out how to optimize one layer at a time without running the whole loss.backward() for every layer. I saw someone had basically the same question here: <a href="https://discuss.pytorch.org/t/computing-backward-one-layer-at-a-time/54796" class="inline-onebox">Computing Backward one layer at a time</a> but the answer wasn’t descriptive enough for me to really understand. I also found this gem: <a href="https://discuss.pytorch.org/t/bit-of-fun-with-gradients/34398" class="inline-onebox">Bit of fun with gradients</a> which gets me close however I don’t know how to grab the gradient of the detached tensor of the previous layer so that I can give that to the previous layers backward call.</p>
<p>Here are clips of the relevant (not)working code:</p>
<pre><code>class testNet(nn.Module):
  def __init__(self, n_observations, n_actions):
      super(testNet, self).__init__()
      self.layer1 = nn.Linear(n_observations, 128)
      self.layer2 = nn.Linear(128, 128)
      self.layer3 = nn.Linear(128, n_actions)
      self.activate = nn.LeakyReLU(.01)

  def forward(self, x):
      x1 = self.activate(self.layer1(x))
      x2 = self.activate(self.layer2(torch.tensor(x1.detach(), requires_grad=True)))
      return self.layer3(torch.tensor(x2.detach(), requires_grad=True))



params = list(test_net.parameters())
params.reverse()
  
slope = nn.MSELoss()
loss = slope(state_action_values, expected_state_action_values.unsqueeze(1))

loss.backward(retain_graph=True)
  
# continue backward grad layer by layer
for i in range(1,len(params),2):
    # optimize this layers parameters 
    params[i+1].backward(gradient=HOW_CAN_I_GET_THIS, retain_graph=True)
</code></pre>
<p>It would be nice to solve the mystery of where values are stored in the computation graph but any implementation details for a layer by layer backprop would be appreciated.<br>
Thank you for your time.</p>
Quantization Aware Training in 8 bits only

Anjul_Tyagi: <p>Is there a way to perform QAT or mixed precision training only in 8 bits. Afaik, currently qat works with fake quants which store the weights in 32 bits and we only see the int8 weights after converting the model.</p>
<p>I’m working on a use case where the full training pipeline should be completed in 8 bits. This is because some edge hardware only support 8 bit operations. That means both the forward and backward passes should be done in only 8 bits and the weights should be stored in 8 bits as well. I’m aware of 8 bit optimizers (<a href="https://github.com/facebookresearch/bitsandbytes" rel="noopener nofollow ugc">bitsandbytes</a> which solves the backward pass 8 bit implementation, but I’m not sure how to do forward pass and store all the weights as 8 bits during training of a pytorch model.</p>
<p>Any suggestions?</p>

UmeTFE: <p>I have the same question but seems like now there is no open-source code for int8 training.</p>

jerryzh168: <aside class="quote no-group" data-username="Anjul_Tyagi" data-post="1" data-topic="170326">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anjul_tyagi/48/56433_2.png" class="avatar"> Anjul_Tyagi:</div>
<blockquote>
<p>Any suggestions?</p>
</blockquote>
</aside>
<p>we have a recent intern that did some experiements on int8 training, maybe you can take a look: <a href="https://github.com/fufeisi/Usage-of-the-8bit-Quantization-in-Neural-Network-Training" class="inline-onebox" rel="noopener nofollow ugc">GitHub - fufeisi/Usage-of-the-8bit-Quantization-in-Neural-Network-Training: This repo has the script to reproduce the experiments in project 'Usage of the 8bit Quantization in Neural Network Training'.</a>, it is comparing the memory savings of dynamically quantizing the activations in the backward pass and also quantizing the gradients in optimizers to 8 bit (same approach in <a href="https://github.com/TimDettmers/bitsandbytes" rel="noopener nofollow ugc">bitsandbytes</a>) I think.</p>
ImportError: cannot import name 'SummaryWriter' from 'torch.utils.tensorboard' (unknown location)

Salwa_Mostafa: <p>Hello</p>
<p>I got this error while executing my code and cannot solve it. I appreciate it if anyone can help.</p>
<blockquote>
<p>from torch.utils.tensorboard import SummaryWriter<br>
ImportError: cannot import name ‘SummaryWriter’ from ‘torch.utils.tensorboard’ (unknown location)</p>
</blockquote>
<p>Thank you</p>

ptrblck: <p>Which PyTorch version are you using as it’s working for me in <code>1.13.0</code>?</p>

Salwa_Mostafa: <p>I use PyTorch 1.12. I will try 1.13.0. Thank you</p>

Muthana: <p>I use PyTorch 1.13.0, but still the wrong:<br>
from torch.utils.tensorboard import SummaryWriter<br>
ImportError: cannot import name ‘SummaryWriter’ from ‘torch.utils.tensorboard’ (unknown location),<br>
what i do, please?</p>
Mixing C++ and Python types

lostmsu: <p>I have a number of PyTorch modules implemented in C# via C++ public API by subclassing the <code>torch::nn::Module</code> as <code>DotNetModule</code> that has a pointer to a <code>forward</code> function implemented in C# (via TorchSharp). I want to be able to use them from Python.</p>
<p>My main problem is that Python side of PyTorch has its own <code>Tensor</code> class, and C++ one is different (TorchSharp uses <code>shared_ptr&lt;torch::nn:Tensor&gt;</code> for this purpose. So the question is how to obtain a <code>shared_ptr&lt;torch::nn::Tensor&gt;</code> from Python so I could pass it to the <code>DotNetModule</code>. And once my module returns a <code>shared_ptr&lt;torch::nn:Tensor&gt;</code> back - construct a Python <code>Tensor</code> from it. Is it even possible without an extra C++ extension?</p>
<p>From the architectural point of view I don’t want the C# modules know anything about the Python client.</p>
Interpreting averaged attention weights from multihead attention

MadeUpMasters: <p>It seems like the two options for analyzing the attention weights are to<br>
A. Take the average across the heads dim<br>
B. Look at each individual head</p>
<p>but isn’t what’s happening inside the transformer that they the attention weights are combined from 16 heads back to 1 via a linear layer, meaning that to actually interpret how the model uses the attention weights, we would want to look at how the 16 heads are combined, by returning the output of the linear layer? If so, why does pytorch opt to return the average across the dims instead of weighting the heads in the same way as the model. I’m sure I’m misunderstanding here so I would appreciate it if someone could point out where I’m going wrong. Thank you.</p>
How to combine two models

Flam3y: <p>I have a pretrained model and my custom model, how can they be combined without access to the dataset of the first model, so that the classes of the first one are saved and mine is added.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/flam3y">@Flam3y</a>,</p>
<p>Do you have a minimum reproducible example to explain more about your problem?</p>
Change tensor of shape=(100,1) to (100, 5) applying custom Python function on each element and calculate loss on result

Manu-Chauhan: <p>Hi,</p>
<p>I want to represent a Tensor of shape=(100,1) of floats of batch=100 to a Tensor representing list of binary encoding of each float number. Example: 10.0 → [0, 1, 0, 1, 0]… where new Tensor is again same batch where each row, which was float, is now being represented by list of encoded value</p>
<p>How to apply custom Python function to convert input tensor to required output here? I tried <code>apply_</code> but it needs output of callable function to return a real number and not list.</p>
<p><code> tensor = tensor.apply_(encoding)</code></p>
<p><code>TypeError: must be real number, not list</code></p>
<p>I am trying to output list of 5 numbers(logits) where it must represent binary encoding of a positive integer value. And tried to use <code>nn.BCEWithLogitsLoss</code> for calculating the loss. The idea is to input preds of 100,5 and actual value of 100,1 converted to 100,5 (here each row is binary encoding of the float)  to calculate loss.</p>
<p>Is this the correct approach or any better ways of handling this?</p>
<p>Thanks</p>
Module List gradients are None

FiorenzoParascandolo: <p>Hy everyone,<br>
I’ve implemented a crude version of locally connected layers which can be trivially thought as a matrix of Conv layer applied to portions of images the same size as the kernels. I’ve implemented with ModuleList as follows:</p>
<pre><code class="lang-auto">class LocallyConnected2d(nn.Module):
    """
    Implementation of LocallyConnected Layer described in
    https://www.cs.toronto.edu/~ranzato/publications/taigman_cvpr14.pdf
    """

    def __init__(self)
        # Matrix of convolutional layer W_out X H_out
        self.convs = nn.ModuleList([nn.ModuleList(
            [nn.Sequential(nn.BatchNorm2d(input_channels).to(device), nn.Conv2d(in_channels=input_channels,
                                                                                out_channels=num_channels,
                                                                                kernel_size=kernel_size,
                                                                                stride=(1, 1)).to(device)) for _ in
             range(self.output_size[1])]).to(
            device) for _
            in
            range(self.output_size[0])])

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        # Concatenate the activations produced by the different convolutional layers
        x = F.leaky_relu(x)
        y = [[self.convs[i][j](x[:, :, (i * self.stride[0]):(i * self.stride[0] + self.kernel_size[0]),
                               (j * self.stride[1]):(j * self.stride[1] + self.kernel_size[1])])
              for j in range(self.output_size[1])]
             for i in range(self.output_size[0])]
        y = torch.cat([torch.cat(y[i], dim=3) for i in range(self.output_size[0])], dim=2)

        return y
</code></pre>
<p>The problem is that the gradients of Conv layers are always 0. Where is the error?</p>

FiorenzoParascandolo: <p>Auto-fix: I’ve missed loss.backward(retain_graph=True)</p>
Calculating number of unique values per row

legoh: <p>Hi, I’m looking at how to count the number of unique values per row.</p>
<p>E.g. I might have the following matrix</p>
<pre><code class="lang-auto">a = torch.tensor([[1, 2, 3, 4, 5, 5, 6],
                           [2, 2, 1, 4, 1, 3, 4],
                           [7, 6, 5, 4, 3, 2, 1]])
</code></pre>
<p>The results should look like<br>
<code>result = [6, 4, 7]</code></p>
<p>At present, I’m doing it like this but the result seems pretty slow when dim0 is large since it’s a for-loop. Is there a better way to do this? <code>torch.unique</code> can’t be directly applied here because the final output has to be an even shape whereas we might have a different number of unique values per row.</p>
<pre><code class="lang-auto">result = [torch.tensor(torch.unique(a[i], dim=-1).size(0)) for i in range(a.size(0))]
result = torch.stack(result, dim=0)
</code></pre>

Samuel_Bachorik: <p>Hi, try this…</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([[1, 2, 3, 4, 5, 5, 6],
                           [2, 2, 1, 4, 1, 3, 4],
                           [7, 6, 5, 4, 3, 2, 1]])

x = torch.zeros([a.shape[0]])

for i in range(len(a)):
    x[i] = len(a[i].unique())

print(x)
</code></pre>

Samuel_Bachorik: <p>Or without x tensor:</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([[1, 2, 3, 4, 5, 5, 6],
                           [2, 2, 1, 4, 1, 3, 4],
                           [7, 6, 5, 4, 3, 2, 1]])

for index, tensor in enumerate(a):
    a[index] = len(tensor.unique())

unique = torch.flatten(a).unique()
print(unique)
</code></pre>
Problem compilation libtorch 1.8.0

Ce_Ce: <p>HI, i’m trying to compile libtorch 1.8.0 on my Docker on Ubuntu 18.04 with cuda 11.1, and i have this error :</p>
<blockquote>
<p>In file included from /opt/libtorch/include/c10/util/complex.h:570,<br>
from /opt/libtorch/include/c10/core/ScalarType.h:4,<br>
from /opt/libtorch/include/c10/core/Scalar.h:10,<br>
from /opt/libtorch/include/ATen/core/TensorBody.h:8,<br>
from /opt/libtorch/include/ATen/Tensor.h:3,<br>
from /opt/libtorch/include/ATen/Context.h:4,<br>
from /opt/libtorch/include/ATen/ATen.h:9,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/types.h:3,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/data/dataloader.h:3,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/data.h:3,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/all.h:8,<br>
from /opt/libtorch/include/torch/csrc/api/include/torch/torch.h:3,<br>
from /root/processus/includes/MicroServiceAlpr.hpp:5,<br>
from /root/processus/includes/MyServer.cpp:10:<br>
/opt/libtorch/include/c10/util/complex_math.h:83:55: error: expected primary-expression before ‘)’ token<br>
83 | C10_HOST_DEVICE inline c10::complex&lt;decltype(T() * U())&gt; pow(const c10::complex &amp;x, const c10::complex &amp;y) {<br>
|                                                       ^<br>
/opt/libtorch/include/c10/util/complex_math.h:93:55: error: expected primary-expression before ‘)’ token<br>
93 | C10_HOST_DEVICE inline c10::complex&lt;decltype(T() * U())&gt; pow(const c10::complex &amp;x, const U &amp;y) {<br>
|                                                       ^<br>
/opt/libtorch/include/c10/util/complex_math.h:102:55: error: expected primary-expression before ‘)’ token<br>
102 | C10_HOST_DEVICE inline c10::complex&lt;decltype(T() * U())&gt; pow(const T &amp;x, const c10::complex &amp;y) {<br>
|</p>
</blockquote>
<p>I tried to compile with gcc 7, 8 and 9, same error.</p>
<p>Thank you for your help.</p>

woong97: <p>I faced same issue. Did you solve this problem?</p>
How to calculate class weights for imbalanced data

fatcat: <p>I am dealing with a binary classification problem where the data is imbalanced. I try to train the model with weighted cross-entropy loss or weighted focal loss, how can I calculate the weights for each class?<br>
Suppose there are n0 examples of the negative class and n1 examples of the positive class; currently I calculated the weights for each classes as follow:</p>
<p>weight for negative class: 1-n1/(n0+n1)<br>
weight for positive class: 1-n0/(n0+n1)</p>
<p>In this way the sum of the weights is normalised to 1. Is this the right way to calculate class weights? Are there any better way to calculate the class weights ?</p>

yassineAlouini: <p>This looks like the correct way to do it as long as it is computed on the training dataset (to avoid data leakage).</p>
<p>You can also simplify the expressions: no need for 1 - the ration, you can use directly n0 and n1 respectively.</p>

fatcat: <p>Thanks for your reply. May I use the validation dataset to calculate the class weights?</p>

Mona_Jalal: <p>essentially you wanna make sure your dataset is stratified meaning you have the same number of positive labels in train, test, and validation set (so that the ratio of data would be same in all 3 sets).</p>

yassineAlouini: <p>Always use the train part and not the validation one so that your evaluation isn’t biased (or at least as low as possible). Indeed, if you estimate some parameters and then use these on the same part, you will have optimistic results.</p>

fatcat: <p>Thanks for your reply. In my case I have multiple datasets and I will validate and test the model on one dataset and train the model with the other datasets. All datasets are imbalanced and the negative examples are the majority, but the ratios of negative examples are different among different datasets. How can I deal with this case?</p>

yassineAlouini: <p>If the datasets are different, it might be better to train one model per dataset, otherwise, it will be hard to generalize. If you go this route, you will have a negative ratio estimation per dataset.</p>
<p>Otherwise, you will need additional work to make sure that the different datasets are “aligned” and thus the trained final model is good enough. If you select this route, you could estimate one negative ratio once you have aligned the different datasets.</p>

Mona_Jalal: <p>I recently came across this other formula for weights for binary cross entropy. Is this wrong?</p>
<pre><code class="lang-auto">Class 0 = (count_0 + count_1)/count_0 = (38+240)/240 = 1.158
Class 1 = (count_0+count_1)/count_1 = (38+240)/38 = 7.315
</code></pre>
<p>As you see, the weights don’t sum up to 1.</p>

badgiojuni: <p>I think it does not depend on the absolute value of the weights but on their ratio</p>
What algorithm is used for eigh?

Arthur_F: <p>Hello,</p>
<p>I would like to know exactly how are the eigenvalues and eigenvectors computed when calling torch.linalg.eigh. I tried to look into the C source code but I could not find the information, hence my post.</p>
<p>Thanks in advance !</p>

ptrblck: <p>On the CPU LAPACK would be used as seen <a href="https://github.com/pytorch/pytorch/blob/85ffbedfb2a2bffda220e3fb73dc311dba5e7fed/aten/src/ATen/native/BatchLinearAlgebraKernel.cpp#L223-L230">here</a> on the GPU either MAGMA or cuSOLVER would be used as seen <a href="https://github.com/pytorch/pytorch/blob/acdd462b1a070790799ce4623ce8ecc83e197e81/aten/src/ATen/native/cuda/linalg/BatchLinearAlgebra.cpp#L2023-L2036">here</a>.</p>

Arthur_F: <p>Thank you very much !</p>
[WinError 126] when importing pytorch on pc with Nvidia gpu

razolli: <p>Even though vscode shows that torch library is installed, when I try to run my code this error occurs:</p>
<blockquote>
<p>File “c:\Users.…\Desktop\I3D_WLASL\train_i3d.py”, line 4, in <br>
import torch<br>
File “C:\Users.…\Anaconda3\lib\site-packages\torch_<em>init</em>_.py”, line 139, in <br>
raise err<br>
OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users.…\Anaconda3\lib\site-packages\torch\lib\caffe2_nvrtc.dll” or one of its dependencies.</p>
</blockquote>
<p>What should I do? (I am using latest versions of all packages.)</p>

ptrblck: <p>Based on <a href="https://discuss.pytorch.org/t/cannot-import-torch-error-loading-caffe2-nvrtc-dll/166808">this topic</a> it seems reinstalling PyTorch with the <code>--user</code> argument might solve the issue, but note that I am not deeply familiar with Windows setups and thus cannot confirm if this approach worked or why the error is raised in the first place.</p>
What does fs[:, i, :] = f do?

peony: <p>Hello, I am trying action recognition. If I understand correctly the code <code>f = f.view(f.size(0), -1)</code> flattens the tensor but for <code>fs[:, i, :] = f</code>, I can’t seem to guess what it is doing there.</p>
<pre><code class="lang-auto">  def forward(self, inputs, hidden=None, steps=0):
        length = len(inputs)
        fs = torch.zeros(inputs[0].size(0), length, self.rnn.input_size).cuda()

        for i in range(length):
            f = inputs[i]
            f = f.view(f.size(0), -1)
            f = self.fc_pre(f)
            fs[:, i, :] = f
</code></pre>
<p>Thank you in advance</p>

ptrblck: <p><code>fs[:, i, :] = f</code> assigns the tensor <code>f</code> in the shape <code>[fs.size(0), fs.size(2)]</code> to each “row” in <code>fs</code>.<br>
Here is a small example where <code>f</code> is containing increasing values for the sake of simplicity:</p>
<pre><code class="lang-python">fs = torch.randn(2, 3, 4)
f = torch.randn(2, 4)

for i in range(fs.size(1)):
    f = torch.ones(fs.size(0), fs.size(2)) * i
    print(f)
    fs[:, i, :] = f

# tensor([[0., 0., 0., 0.],
#         [0., 0., 0., 0.]])
# tensor([[1., 1., 1., 1.],
#         [1., 1., 1., 1.]])
# tensor([[2., 2., 2., 2.],
#         [2., 2., 2., 2.]])
    
print(fs)
# tensor([[[0., 0., 0., 0.],
#          [1., 1., 1., 1.],
#          [2., 2., 2., 2.]],

#         [[0., 0., 0., 0.],
#          [1., 1., 1., 1.],
#          [2., 2., 2., 2.]]])
</code></pre>

peony: <p>Thank you for the explanation <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
ValueError: expected sequence of length 82 at dim 1 (got 63)

nfs6788: <p>I am getting this error:</p>
<p><strong>ValueError: expected sequence of length 82 at dim 1 (got 63)</strong></p>
<p>and I guess this part is not working properly:</p>
<hr>
<p>def data_process(data, labels):<br>
input_ids = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
attention_masks = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
MAX_SEQUENCE_LENGTH = 250<br>
bert_tokenizer = BertTokenizer.from_pretrained(“bert-base-uncased”)<br>
for sentence in data:<br>
bert_inp = bert_tokenizer.<strong>call</strong>(sentence, max_length=MAX_SEQUENCE_LENGTH,<br>
padding=‘max_length’, pad_to_max_length=True,<br>
truncation=True, return_token_type_ids=False<br>
)# ,add_special_tokens=True   , return_length=True,<br>
#return_tensors=“pt”<br>
input_ids.append(bert_inp[‘input_ids’])<br>
attention_masks.append(bert_inp[‘attention_mask’])</p>
<pre><code>input_ids = np.asarray(input_ids, dtype=object)#
attention_masks = np.array(attention_masks, dtype=object)#
labels = np.array(labels)#

return input_ids, attention_masks, labels
</code></pre>
<hr>
<p>This is error line:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2ba211756bd9e8a3c0cbfb9401b53796cc7576d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/2ba211756bd9e8a3c0cbfb9401b53796cc7576d8" title="value_error_expected_sequence02"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2ba211756bd9e8a3c0cbfb9401b53796cc7576d8.png" alt="value_error_expected_sequence02" data-base62-sha1="6dZLai7MXUBN3q0gxbJ1yYGpld6" width="690" height="494" data-dominant-color="262724"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">value_error_expected_sequence02</span><span class="informations">962×689 42.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I get the following error:</p>
<hr>
<p>ValueError                                Traceback (most recent call last)<br>
 in <br>
340 # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Tokenization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#<br>
341 # for train set<br>
 → 342 train_seq = torch.tensor(train_text[‘input_ids’].tolist())#, dtype=torch.int64<br>
343 train_mask = torch.tensor(train_text[‘attention_masks’].tolist(), dtype=torch.int64)#<br>
344 train_y = torch.tensor(train_labels.tolist(), dtype=torch.int64)</p>
<p>ValueError: expected sequence of length 82 at dim 1 (got 63)</p>
<hr>
<p>What am I doing wrong?</p>
Taking derivatives of a neural network wrt parameters after first being differentiated wrt state

Cameron_Fen: <p>I have a neural network: <code>ANN(x)</code> that is a function of x but also has parameters to optimize.  I need to calculate the derivative of it with respect to x then I would like to gradient descent and optimize <code>dANN(x)dx.</code>  This requires taking the derivative of <code>dANN(x)/dx</code> with respect to the parameters of <code>ANN(x)</code>.</p>
<p>I can do this with the autograd <code>jacobian</code> function but that’s really slow.  I would like to do this with the faster grad or backward functions and <code>torch.autograd.Function</code> but I’m not sure how to use the <code>backward</code> function so that each parameter in the <code>dANN(x)dx</code> gets the correct gradient, especially since the input to the <code>forward</code> function is a list of parameters (ie <code>ANN.paramaeters()</code>) each which should receive their own gradient.</p>
<p>For example I have:</p>
<pre><code class="lang-auto">class derivativeANN(torch.autograd.Function):
    
    @staticmethod
    def forward(ctx, input):
        tempstate2 = tempstate.detach().requires_grad_(True)
        ans = model(tempstate2.reshape(-1,8))[:,1]
        ds = grad(ans, tempstate2, grad_outputs=torch.ones((batch,)).to(device), retain_graph = True)
        ctx.save_for_backward(input)
        ctx.save_for_backward(ds)
        ctx.save_for_backward(model.parameters())
        return ds[0][:,:2]

    @staticmethod
    def backward(ctx, grad_output):
        input, ds, params12 = ctx.saved_tensors
        dspar = []
        for each in input:
            dsdpar.append(grad(ds, each, grad_outputs=torch.ones_like(ds).to(device)))
</code></pre>
<p>But now I need each element of dspar to update the grade of each element of the parameter list of <code>list(model.parameter())</code>.  I’m sure there is a way to optimize <code>torch.autograd.Function</code> with multiple inputs, I just don’t know how.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/cameron_fen">@Cameron_Fen</a>,</p>
<p>If you want to compute the derivative of the output of your network with respect to its input and parameters efficiently, you can use the functorch library (which comes with pytorch now). A brief example code can be shown below.</p>
<pre><code class="lang-auto">import torch
from functorch import make_functional, jacrev, vmap

net = Model(*args, **kwargs)

fnet, params = make_functional(net)

B=100 #number of samples
size = 4 #4 input nodes (for example)

x = torch.rand(B, size)

results = vmap(jacrev(jacrev(fnet, argnums=(1)), argnums=(0)), in_dims=(None, 0))(params, x)
</code></pre>
<p>If you have more outputs than inputs, you should use <code>jacfwd</code> instead of <code>jacrev</code> for forward-mode AD.</p>

Cameron_Fen: <p>A jax-like pytorch library was exactly what I needed.  Thanks!</p>
Can't come up with a way to compute a given gradient that i need

malfonsoarquimea: <p>Just for context, I am working on a nerf-like model for dynamic scenes that, for a given point in space, outputs how occupied the space is at this point in space for all the frames in the recorded scene. That means that, if the recorded scene is a set of 300-frames videos (from different viewpoints) then, doing inference on a given XYZ coordinate will return 300 scalar values</p>
<p>Now, here comes the problem. I need to get the normal vector of the surface at a given point in space. This is usually done by computing the gradient of the occupancy with respect to the coordinates.<br>
This is simple when working with models that for each xyz coordinate produce only one scalar value (models for static scenes), but I can’t come up with a way of doing this for my current scenario.<br>
What I would usually do is something like</p>
<pre><code class="lang-auto">normal_vectors=-1*torch.autograd.grad(occupancies.sum(), coordinates)
</code></pre>
<p>where <code>occupancies</code> is a tensor of size <code>[batch_size,300]</code> and <code>coordinates</code> is a tensor of size <code>[batch_size,3]</code>. It produces a tensor of the same size as <code>coordinates</code>. I would need it to produce a tensor of size <code>[batch_size,300,3]</code> as, for each item in <code>coordinates</code>, 300 items are produced and I want the gradient of each one of those with respect to the 3 components of the given item.</p>
<p>The only approach I think may work (untested yet) is something like</p>
<pre><code class="lang-auto">normal_vectors=[]
for i in range(300):
   normal_vectors.append(-1*torch.autograd.grad(occupancies[:,i].sum(),coordinates))
</code></pre>
<p>But this will be painfully slow.<br>
Any alternative approach?</p>

Sandro_Hauri: <p>I can’t offer any solution but want to point out some inconsistency in the way you describe your problem. You say that <code>occupancies</code> is a tensor of size <code>[batch_size,3]</code> but later you index it with <code>occupancies[:,i]</code> where <code>i</code> can be up to 299, so it’s a bit hard to vectorize the for loop, when not sure about the actual dimensions of the problem.</p>

malfonsoarquimea: <p>thanks for pointing that out, it was an error hehe. Now, the description should be consistent!</p>
Splitting batch/tensor

kyleksslr: <p>I have an image set of 256 that gets augmented to a size of 512.</p>
<p>Then in a custom loss function.<br>
Images, list = batch<br>
Where images is of size (512,3,96,96) (batch size, layers, h, w)</p>
<p>When i perform convolution and normalization i get [512,128] for a size().</p>
<p>I would like to split the standard pictures from the augmented ones to perform a custom loss function and was wondering how i would go about doing this.</p>
<p>If it was better to split the tensor or try and split images in the list… i tried split() but when i did i split it into [512,64] and i don’t know if this is the right way of splitting them.</p>

ptrblck: <aside class="quote no-group" data-username="kyleksslr" data-post="1" data-topic="170853">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kyleksslr/48/56651_2.png" class="avatar"> kyleksslr:</div>
<blockquote>
<p>When i perform convolution and normalization i get [512,128] for a size().</p>
</blockquote>
</aside>
<p>If I understand your use case correctly you are upsampling the original images of <code>[512, 3, 96, 96]</code> into <code>[512, 3, 512, 512]</code> and create an activation in the shape <code>[512, ?, 512, 128]</code> after the convolution and normalization layer?</p>
<aside class="quote no-group" data-username="kyleksslr" data-post="1" data-topic="170853">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kyleksslr/48/56651_2.png" class="avatar"> kyleksslr:</div>
<blockquote>
<p>I would like to split the standard pictures from the augmented ones to perform a custom loss function and was wondering how i would go about doing this.</p>
</blockquote>
</aside>
<p>This seems as if you are concatenating the augmented images into the larger spatial size instead of upsampling? Could you describe this augmentation in more detail, please?</p>
Loss functions of GANs

shrbrh: <p>When should we use L1 norm and when should we use L2 norm to minimize loss functions in GANs?</p>

J_Johnson: <p>I assume you mean for reconstruction loss. Typically, L2 loss is used when you want to target outliers for gradient descent, while ignoring “in the ballpark” outputs. You can switch between the two. If using L2, I’d only use it at the beginning, then switch to L1 part way through training.</p>
<p>Otherwise, you should just use BCELoss on the Discriminator.</p>
<p>Depends on your training setup.</p>