akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does Autograd work when slicing tensors and re-assigning slices to different tensors?

username: kekgle
cooked: <p>Hi all! I’m fairly new to PyTorch and still understanding how Autograd works.</p>
<p>I am experimenting with fusing different word embeddings to feed to a Neural Network and am not sure if the operations I am doing will be tracked correctly and differentiated as intended.</p>
<p>Let me explain better:</p>
<p>I am building an embedding fusion layer that is intended to behave exactly like a TimeDistributedDense layer in Keras. Namely, this layer takes multiple embeddings for each token and fuses them by applying the same Linear transformation to all of them. <strong>However, I do not want to perform fusion for padding tokens</strong>, which are always all zeros and so I am skipping it by doing this:</p>
<pre><code class="lang-auto">def _apply_to_nonzero(self, x):
        # find all non-zero vectors in the batch - these are the non-padding elements
        non_zeros = torch.tensor([torch.max(v).item() &gt; 0 for v in x])
        # build zero matrix for the batch using the output size of the Linear layer (self.output_size)
        all_zeros = torch.zeros((*x.size()[:-1], self.output_size))
        # now fill the non-zeros indexes of the zero matrix with the fused embeddings
        all_zeros[non_zeros] = self._fuse(x[non_zeros]).
        # at this point, all_zeros contains the fused embeddings for tokens that are not padding and
        # zero vectors of length self.output_size, in the indexes that contained padding elements.
        x = all_zeros
        return x
</code></pre>
<p>The code above runs as intended, but are my operations going to be correctly differentiated? (what concerns me is the fact that I’m slicing and moving slices across different tensors…)</p>

username: daysofthunder
cooked: <p>Hey I created this simple example, slicing seems to propagate the gradients:</p>
<pre><code class="lang-auto">a = torch.tensor([2., 3, 3], requires_grad = True)
b = torch.tensor([6., 4, 2], requires_grad = True)
Q = 3*a**3 - b**2
# slice and assign
x = Q[1:3]
# call backward
x.sum().backward()
print(a.grad)
print(b.grad)
</code></pre>
<p>Output is:</p>
<pre><code class="lang-auto">tensor([ 0., 81., 81.])
tensor([-0., -8., -4.])
</code></pre>
<p>which shows that gradients have been computed</p>
Is there a way of checking if the dataset is ambiguous?

username: Capo_Mestre
cooked: <p>Hello,</p>
<p>It is known that if there are ambiguities in a training dataset then neural network will struggle to learn.<br>
Is there a way of checking if the dataset is ambiguous before attempting to train a neural network?</p>
<p>Especially if it is a regression problem.</p>

username: ariG23498
cooked: <p>You can always add a check in the <code>__getitem__</code> of your dataset, to check for ambiguities.</p>
PyTorch Dynamic Parallelism Effectiveness with Small Processes

username: cam123
cooked: <p>Hi, I have an application where I am running a set of models (~10) on variable-batch-sized input. So I’m using dynamic parallelism since each of these models can be run in parallel. Each model’s batch is around 500 items. But after getting it running, I actually experience a significant slowdown to running them sequentially (just iterating over them). Is this expected in that the overhead for each process takes longer than the actual execution with this small data size?</p>
Getting different results with weights.data

username: jpj
cooked: <p>I’m initializing weights of a hidden layer<br>
<code>self.hidden_layer_1.weight.data = torch.nn.Parameter(torch.from_numpy(weights))</code></p>
<p><code>self.hidden_layer_1.weight = torch.nn.Parameter(torch.from_numpy(weights))</code></p>
<p>And I’m getting different losses when running the model</p>

username: pascal_notsawo
cooked: <p>I advise you to use the second approach, or the first one but modified as follows:</p>
<pre><code class="lang-auto">self.hidden_layer_1.weight.data = torch.from_numpy(weights) # torch.nn.Parameter is really not needed in this case
</code></pre>

username: jpj
cooked: <p>Thanks. Do you know why they produce different results after backward propogation?</p>
Adding a sequential module to existing nn.Module class using function

username: Ajinkya_Ambatwar
cooked: <p>Hello! I am creating a nn.Module class. In that I need to add a sequential module. The module is build from an external function.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from typing import List, Optional, Tuple

from torch.nn import modules

def build_shared_mlp(mlp_spec: List[int], bn: bool = True):
    layers = []
    for i in range(1, len(mlp_spec)):
        layers.append(nn.Conv2d(mlp_spec[i - 1], mlp_spec[i], kernel_size=1, bias=not bn))
        if bn:
            layers.append(nn.BatchNorm2d(mlp_spec[i]))
        layers.append(nn.ReLU(True))

    return nn.Sequential(*layers)

class temp(nn.Module):
    def __init__(self):
        super(temp).__init__()
        self.build_model()

    def build_model(self):
        self.net = build_shared_mlp([3,4,5],True)
        
net = temp()
</code></pre>
<p>But this gives me error saying</p>
<pre><code class="lang-auto">AttributeError: cannot assign module before Module.__init__() call
</code></pre>
<p>How can I have the module inserted using a function?</p>

username: pascal_notsawo
cooked: <aside class="quote no-group" data-username="Ajinkya_Ambatwar" data-post="1" data-topic="120937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajinkya_ambatwar/48/11828_2.png" class="avatar"> Ajinkya_Ambatwar:</div>
<blockquote>
<pre><code class="lang-auto">class temp(nn.Module):
    def __init__(self):
        super(temp).__init__()
</code></pre>
</blockquote>
</aside>
<p>This is because you don’t initialize correctly the main class (<code>nn.Module</code>) that <code>temp</code> inherits.<br>
Change <code>super(temp).__init__()</code> to <code>super(temp, self).__init__()</code> or <code>super().__init__()</code>, and the problem will be solved.</p>

username: Ajinkya_Ambatwar
cooked: <p>Thank you so much. My kite extension auto-inserted the init block, I didn’t double check.</p>
Torch.jit.load fails in debug mode on windows but works in release mode

username: ydas
cooked: <p>I am running PyTorch under c++ windows 10 visual studio 2019.<br>
Loading and executing model is working properly in Release mode.<br>
I am linking with torch_cpu and c10 libraries from the Release TorchLib.<br>
However, when linking with corresponding libraries from Debug TorchLib, although compilation is working, TorchLib fails loading the traced model.<br>
(I downloaded both release and debug lib from <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>)<br>
The error message thrown by Torchlib is “Open File Failed”.<br>
The same problem was reported by another user in <a href="https://github.com/pytorch/pytorch/issues/17682" class="inline-onebox" rel="noopener nofollow ugc">Windows 10 VS 2017 - torch::jit::load works in debug mode, but fails in release mode · Issue #17682 · pytorch/pytorch · GitHub</a><br>
Is there an issue with running in debug mode under windows?<br>
Thanks</p>
Automated Feature Engineering Using PyTorch

username: Mark_Esteins
cooked: <p>Hi,</p>
<p>I want to implement something similar to this using Pytorch: <a href="https://towardsdatascience.com/automated-feature-engineering-using-neural-networks-5310d6d4280a" rel="noopener nofollow ugc">https://towardsdatascience.com/automated-feature-engineering-using-neural-networks-5310d6d4280a</a></p>
<p>I don’t know how to combine models in the fashion done in the article using Tensorflow so was thinking in this approach:</p>
<p>My idea is to select, let’s say, 50 features and then build 50 models to create the extra features. Each model would have 49 of the 50 features as input features and would try to predict the one left. I would save the values of the last intermediate layer of each model and these values would be the engineered features used in my final model.</p>
<p>The flow would be something like this:<br>
1.- Build a model for each feature to predict that feature using the rest of the features as input, and store the last intermediate layer of each model.<br>
2.- Merge/Join intermediate layer values with the initial dataset (train_data). It would contain the initial features and the engineered ones.<br>
3.- Train a final model with all those features and predicting the target.</p>
<p>But reading the article it says:<br>
<em>The trick is making sure that the feature networks train with the final model rather than a separate process.</em></p>
<p>My approach would definitely not do that so I guess I’m missing something. Why is this important?</p>
<p>The article also says:<br>
<em>Because we have several auxiliary outputs, we need to tell TensorFlow how much weight to give each one in determining how to adjust the model to improve accuracy. I personally like to give 50% weight to the auxiliary predictions (total) and 50% the the target prediction. Some might find it strange to give any weight to the auxiliary predictions since they are discarded at the loss calculation step. The problem is, if we do not give them any weight, the model will mostly ignore them, preventing it from learning useful features.</em></p>
<p>Again, I wouldn’t be doing anything like that so I’m wondering if an approach like what I have in mind make sense at all or I’m missing something.</p>
<p>Does my approach sounds good or I’m missing something?<br>
Is there a way to combine models with PyTorch as it is done in the article with Tensorflow?</p>
<p>Thanks in advance!</p>
Group nodes in tensorboard

username: Spyderman
cooked: <p>Hey,<br>
I’m wondering if it is possible in Tensorboard with pytorch to define groups of nodes that are displayed as one node. E.g. in TensorFlow it is possible to group and name nodes in Tensorboard using tf.name_scope (as far as I understood) as explained <a href="http://www.florens.io/projects/tf/board/" rel="noopener nofollow ugc">here</a>. This is very helpful for having a better overview if you have several ‘units’ in your graph. A simple example, in a feed forward NN I could wish to group all the hidden layers in a node called ‘representation’ and the output layer in a node called ‘classifier’.<br>
Is this somehow possible in pytorch? For me it seems like a very important feature yet I could’t find anything.<br>
Cheers,<br>
Spyderman</p>

username: Spyderman
cooked: <p>Ok so I figured it out myself. You will get a node called ‘representation’ just by putting all your representation layers into an nn.sequential() with the variable name ‘representation’. So this is very intuitive. In fact, I already had it this way, I just didn’t recognize it because my model had so many parts that the graph was a bit messy.</p>
Clarification : tensor.view(*size)

username: nisharaichur
cooked: <p>The documentation of torch.view has the input parameter as (*size).<br>
<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view</a></p>
<p>What exactly does * represent here?<br>
Why does not just input of type  &lt;class ‘torch.Size’&gt; work?</p>

username: ptrblck
cooked: <p>The asterisk (*) is used to unwrap values in Python and you could thus pass the desired shapes as separate values to <code>view</code>.</p>
<p><code>torch.Size</code> works fine as an input also:</p>
<pre><code class="lang-python">x = torch.randn(100)
y = torch.randn(10, 10)
x = x.view(y.size())
print(x.shape)
&gt; torch.Size([10, 10])

x = x.view(torch.Size([100]))
print(x.shape)
&gt; torch.Size([100])

x = x.view(1, 5, 20)
print(x.shape)
&gt; torch.Size([1, 5, 20])
</code></pre>

username: nisharaichur
cooked: <p>Got it. Thanks a lot.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Loading model from pytorch_pretrained_bert into transformers library

username: twistedstats
cooked: <p>I have a pretrained BERT model for a classification task trained on the pytorch_pretrained_bert library. I would like to use the initial weights from this model for further training with transformers library.</p>
<p>When I try to load this model, I get the following runtime error. Does anyone know what this is?</p>
<p>model = BertClassification(weight_path= pretrained_weights_path, num_labels=num_labels)<br>
state_dict = torch.load(fine_tuned_weight_path,map_location=‘cuda:0’)<br>
model.load_state_dict(state_dict)</p>
<p>RuntimeError: Error(s) in loading state_dict for BertClassification:<br>
Missing key(s) in state_dict: “bert.embeddings.position_ids”.</p>
<p>Thanks very much.</p>
What is the relation between arguments used in YAML and class arguments in the yolov5 implementation by ultralytics?

username: sanyam.ss99
cooked: <p>I was working on fish detection and tracking system in 3 dimensions using MS Kinect, and used yolov5s for RGB frames. I wanted to experiment a bit with the architecture</p>
<p>of yolov5, but as a beginner i am struggling to find the relation between the arguments in the classes used in models/common.py file and the arguments used in YAML file in the Ultralytics repo. Can anybody help ?</p>
<p>PS : If you can add in some resources from where i can learn about the architecture and layers, then it would really help. Thank you !</p>
Custom cdist in pytorch

username: kavka
cooked: <p>I’m trying to use the torch.cdist function in pytorch. However, there is a little bit difference between my definition of distance with the definition of the function.</p>
<p>For example,  for ‘torch.cdist’</p>
<pre><code class="lang-auto">a = torch.tensor([[1.,  0., 1.], [1., 0., 1.], [1., 0., 1.]])
b = torch.tensor([[0., 1., 0.], [0., 1., 0.], [0., 1., 0.]])
dis = torch.cdist(a,b,p=2)
print(dis)
tensor([[1.7321, 1.7321, 1.7321],
        [1.7321, 1.7321, 1.7321],
        [1.7321, 1.7321, 1.7321]])
</code></pre>
<p>Here,</p>
<pre><code class="lang-auto"># dis[i,j] = sqrt(sum((a[i] - b[j])**2))
</code></pre>
<p>However, in my definition,</p>
<pre><code class="lang-auto"># for dis[i, j]
# make i, j element of a[i] - b[j] equals to 0
# then dis[i,j] = sqrt(sum((a[i] - b[j])**2))
</code></pre>
<p>My question is that is there any api of source code that allow me to custom this kind of modification?</p>

username: Zhaoyi-Yan
cooked: <p>Give an example and the correct result of your case, it seems not quite clear.</p>
Classifying Time Series Data

username: goldshakil
cooked: <p>Hi all,</p>
<p>I am trying to classify a time series data and after feature extraction the data looks like the following<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffefb3e3b1051aa74de003f8ae0e55e448c568e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffefb3e3b1051aa74de003f8ae0e55e448c568e0" title="data"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffefb3e3b1051aa74de003f8ae0e55e448c568e0_2_690x215.png" alt="data" data-base62-sha1="Aw7jze5KVjz0xzxnjps8W2AB81G" width="690" height="215" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffefb3e3b1051aa74de003f8ae0e55e448c568e0_2_690x215.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffefb3e3b1051aa74de003f8ae0e55e448c568e0_2_1035x322.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffefb3e3b1051aa74de003f8ae0e55e448c568e0_2_1380x430.png 2x" data-dominant-color="3A3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">data</span><span class="informations">1560×488 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>About this data:<br>
The original data before feature extraction represents accelerometer readings (Image attached in the comments below). What I did was extracting features from each series instead of using the brute data.</p>
<p>In other words, each series/sequence in the original data consisted of 1000 rows so I extracted several features so that I ended up with one row for each series.</p>
<p>I know the new data looks more fit to be used with a Random Forest. However, I was wondering if it was possible to use my new data with a RNN/LSTM/GRU? If yes, how can I do that?<br>
Can you provide some references?</p>
<p>In simpler words, how can I classify some data with several features using RNN in PyTorch?<br>
Thank you very much!</p>

username: goldshakil
cooked: <p>This is the original data:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4029ecc36064b0d022850d80254a81040440c884.png" alt="Original Data" data-base62-sha1="99CnFfxhHICyvAg9yKBu9rPebE8" width="387" height="314"></p>
Load Pytorch weights and see what seed it was trained on

username: Dylan_Yung
cooked: <p>Is it possible to load PyTorch weights into a nn.Module then see what seed was used for initialization?</p>

username: ptrblck
cooked: <p>No, that wouldn’t be possible, if you don’t store the seed and also provide a script to sample exactly these values in a predefined environment.<br>
I also assume that you are loading random values, which were indeed sampled from a specific distribution, since usually you would load pretrained models, which have (as the name suggests) trained parameters.<br>
As mentioned before, even if someone provides the seed you would need to make sure to use the same setup, since different hardware might use different pseudorandom number generators.</p>
Too many values to unpack (expected 2) while working with sequence classification

username: baokar7
cooked: <p>Hello everyone,</p>
<p>I am working on sequence classification where I have created my own Dataset class and using that to train.</p>
<p>Here is the dataset class:</p>
<pre><code class="lang-auto">class BERTDataset:
    
    def __init__(self, input_ids, attention_masks, token_type_ids, targets):
        self.input_ids = input_ids
        self.attention_masks = attention_masks
        self.token_type_ids = token_type_ids
        self.targets = targets
        
    def __getitem__(self, index):
        ids = torch.tensor(self.input_ids, dtype=torch.long)
        masks = torch.tensor(self.attention_masks, dtype=torch.long)
        token_type_ids = torch.tensor(self.token_type_ids, dtype=torch.long)
        targets = torch.tensor(self.targets, dtype=torch.long)
        
        return {
            'ids': ids,
            'mask': masks,
            'token_type_ids': token_type_ids,
            'targets': self.targets[index]
        }
    
    def __len__(self,):
        return len(self.input_ids)
</code></pre>
<p>The way I fetching for train is as follows:</p>
<pre><code class="lang-auto">model.train()

y_preds = []

for epoch in range(epochs):
    for i, sample in enumerate(train_dataloader):
        ids = sample["ids"]
        masks = sample["mask"]
        token_type_ids = sample["token_type_ids"]
        target = sample["targets"]
        
        ids = ids.to(device, dtype=torch.long)
        masks = masks.to(device, dtype=torch.long)
        token_type_ids = token_type_ids.to(device, dtype=torch.long)
        target = target.to(device, dtype=torch.long)
        
        y_pred = model(tokens,
                      masks,
                      token_type_ids)
</code></pre>
<p>I am getting the following error: too many values to unpack (expected 2)</p>
<p>This is for the last line where I am calculating <code>y_pred</code> from model() object.</p>
<p>What am I missing here?</p>

username: ptrblck
cooked: <p>Based of your description of the issue I assume you are trying to unpack some objects inside the <code>forward</code> method of your model, which is failing. If that’s the case, check the complete stacktrace and make sure the objects, which should be unpacked contains enough values.</p>

username: baokar7
cooked: <p>This is my model:</p>
<pre><code class="lang-auto">class BERTBert(nn.Module):
    def __init__(self):
        super(BERTBert, self).__init__()
        self.bert = transformers.BertModel.from_pretrained('bert-base-uncased')
        self.drop = nn.Dropout(0.3)
        self.out = nn.Linear(768, 1)
    
    def forward(self, ids, masks, token_type_ids):
        output = self.bert(ids,
                           attention_mask=masks,
                           token_type_ids=token_type_ids)
        output = self.drop(output)
        output = self.out(output)
        return output
</code></pre>
Data normalization and backpropagation

username: cbd
cooked: <p>If i normalize the data from (-1,1) to (0,1) before loss computation, Is there any effect on backpropagation?</p>

username: ptrblck
cooked: <p>Yes, I would assume changing the value range before the loss calculation would also change the loss itself, which would then yield different gradients:</p>
<pre><code class="lang-python">model = nn.Linear(10, 10)
x = torch.randn(1, 10)
target = torch.randn(1, 10)
criterion = nn.MSELoss()

# default
out = model(x)
loss = criterion(out, target)
loss.backward()
print(model.weight.grad.abs().sum())
&gt; tensor(37.9178)

# scaled approach
scale = True
model.zero_grad()
out = model(x)
if scale:
    out = out - out.min()
    out = out / out.max()
print(out.min(), out.max())
&gt; tensor(0., grad_fn=&lt;MinBackward1&gt;) tensor(1., grad_fn=&lt;MaxBackward1&gt;)

loss = criterion(out, target)
loss.backward()
print(model.weight.grad.abs().sum())
&gt; tensor(11.1220)
</code></pre>
Unsqueeze the data from one dimension to three dimension

username: cbd
cooked: <p>In below code i have to use “unsqueeze” 2 times for 3d conversion. Is there anyway to convert 1d to 3d using “unsqueeze” single time.</p>
<pre><code class="lang-auto">from torchvision import transforms
a=torch.ones([5],requires_grad=True) * - 0.5
a=a.unsqueeze_(0)
a=a.unsqueeze_(0)
print(a.shape)</code></pre>

username: ptrblck
cooked: <p>You could index the tensor with <code>None</code> to add additional dimensions:</p>
<pre><code class="lang-python">x = torch.randn(10)
y = x[None, :, None, None]
print(y.shape)
&gt; torch.Size([1, 10, 1, 1])
</code></pre>
torch.Generator.manual_seed: "It is recommended to set a large seed" Why?

username: bananacode
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator.manual_seed" rel="noopener nofollow ugc">The docs for the manual seed</a> state that it is recommended to set a large seed, but does not specify why. Could someone provide an explanation?</p>

username: ptrblck
cooked: <p>I think the docs were added in <a href="https://github.com/pytorch/pytorch/pull/21364#issuecomment-501074688">this PR</a>, which mentions <a href="https://www.pcg-random.org/posts/cpp-seeding-surprises.html">this article</a> about <code>Scary All-Zero States</code> in the random number generation.</p>
RuntimeError: mat1 dim 1 must match mat2 dim 0 CNN

username: Samiur_Rahman
cooked: <p>Hello, i am getting the error when i am trying to train a CNN model using custom dataset.<br>
here is my model:</p>
<pre><code class="lang-auto">class CNNMnist(nn.Module):
    def __init__(self, args):
        super(CNNMnist, self).__init__()
        self.conv1 = nn.Conv2d(args.num_channels, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(720, 50)
        self.fc2 = nn.Linear(128, args.num_classes)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, x.shape[1]*x.shape[2]*x.shape[3])
        print(x.shape)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)

        x = self.fc2(x)
        return F.log_softmax(x, dim=1)
</code></pre>
<p>i reshape my images while loading with <code>transforms.Resize(256),transforms.CenterCrop(224)</code><br>
i initialize my model:</p>
<pre><code class="lang-auto">global_model = CNNMnist(args=args)
</code></pre>
<p>for my custom dataset, the number of class is 6 and the number of channel is 3.<br>
The error is thrown at <code> x = F.relu(self.fc1(x))</code>.<br>
After spending hours trying to find out the problem, my guess is the values used in the following lines are not correct: <code>self.fc1 = nn.Linear(720, 50)         self.fc2 = nn.Linear(128, args.num_classes)</code><br>
Since i actually have no idea how these values are calculated, I would really appreciate some help on this.<br>
Thank you</p>

username: eqy
cooked: <p>Flatten should remove all but the batch dimensions of your model and produce a (N, D) tensor. D should match the number of input dimensions to the first layer (if it is not 720, this value should be changed). Also, the output of fc1 will have 50 dimensions, so the input dimensions of fc2 should be changed to 50 (or the output of fc1 to 128).</p>

username: Samiur_Rahman
cooked: <p>Thanks for the reply. Are you suggesting that i should use flatten? And how do i calculate D?</p>

username: eqy
cooked: <p>It looks like you are already printing the shape so you should be able to see what N, and D are here.<br>
Flatten can work, but rather than reshaping to (-1, something), you should reshape to (batch_size, -1).</p>

username: Samiur_Rahman
cooked: <p>thanks. changing the value of D did the trick. one small question, how important it is to reshape(batch_size, -1) than reshape (-1, something). How does it affect my training?</p>

username: eqy
cooked: <p>It is very important as otherwise you are mixing different examples that are in the same batch together. However, if you set the reshape to (-1, [flat_size]/batch_size) then the two are equivalent.</p>
Using pytorch allocators for custom extensions

username: uUser
cooked: <p>Hello everyone. What is the current/recommended  API to use for custom extensions that need to perform allocations and  want  to use/share the memory  allocation pool created by the PyTorch library?  For instance functions/operators that uses the Thrust library uses a series of wrappers that finally calls c10::cuda::CUDACachingAllocator::raw_alloc and c10::cuda::CUDACachingAllocator::raw_delete. These two functions should be ok to be called from a external code/library?</p>
<p>thanks</p>

username: tom
cooked: <p>Personally, I’d just allocate a Tensor and keep that alive.</p>

username: uUser
cooked: <p>Hello <a class="mention" href="/u/tom">@tom</a>, thanks for your reply.  The reason that I was going to use this kind of allocation API is that the allocations is controlled by a library and similarly to thrust case the library exposes function pointers to allocation and de allocation functions. What would be the main downsides of using this API? I was not sure if these functions are actually public ones.</p>
<p>thanks</p>

username: tom
cooked: <p>Apparently they are used by other libraries, too, see the comments here (and also more use-cases towards the bottom): <a href="https://github.com/pytorch/pytorch/pull/33860" class="inline-onebox" rel="noopener nofollow ugc">Expose `CUDACachingAllocator` `raw_alloc` and `raw_delete` to python by emcastillo · Pull Request #33860 · pytorch/pytorch · GitHub</a></p>
<p>Personally, I would not do it because it’s not the level of API abstraction I want to work with. For most of my interop needs, I found ways to make DLMTensors do what I need, but hey, it’s your code and it looks as safe as any other low level function. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Hindsight / Multi-Output Loss

username: MaxiBoether
cooked: <p>Hello,</p>
<p>I have built a model that can output multiple solutions to the same classification problem, i.e., its spreads it bets. The idea is to take the best solution out of various proposed ones. The loss that papers [1,2,3] implement for such a network is called hindsight loss, and the idea is to just take the minimum loss for all the possible outputs you got. My current implementation looks as follows:</p>
<pre><code class="lang-auto">def hindsight_loss(output, labels):
    if (len(labels.size()) == 1):
        # we need to reshape our labels from [0,0,1,0] to [[0], [0], [1], [0]]
        labels = labels.resize_(labels.size()[0], 1)

    labels = labels.to(torch.float32)

    ce_func = nn.BCELoss()
    loss = ce_func(output[:, 0, None], labels)
    for res in range(1, output.shape[1]):
        new_loss = ce_func(output[:, res, None],labels)
        loss = torch.min(loss, new_loss)

    return loss
</code></pre>
<p>However, the current implementation leads to the fact that during backpropagation, only the one output of my set of outputs is getting improved. I suppose this is because during backprop, the weights for all the other ways to generate outputs are not changed. However, I do not think this is the idea of the authors, but I could not find anything regarding the hindsight loss in PyTorch (or any other framework, for that matter).</p>
<p>Does anybody here know whether something is wrong with my implementation and/or understanding of the issue? Because the way it is right now, it does not make sense to learn to give multiple outputs, because only one output is being improved anyway.</p>
<p>Thank you so much.</p>
<p>KR,<br>
Maximilian</p>
<p>[1] Multiple Choice Learning:<br>
Learning to Produce Multiple Structured Outputs<br>
[2] Interactive Image Segmentation with Latent Diversity<br>
[3] Combinatorial Optimization with Graph<br>
Convolutional Networks and Guided Tree Search</p>
Per-tensor quantization in quantization-aware training

username: kaizhao
cooked: <p>The tutorial <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">here</a> provides an example of <em>per-channel</em> quantization training.</p>
<p>In my case I need to perform per-tensor quantization since the downstream mobile-device inference library (e.g. <a href="https://github.com/Tencent/TNN" rel="noopener nofollow ugc">TNN</a>) does not support per-channel quantized models.</p>
<p>I think the problem here is how to setup a per-tensor quantization around:</p>
<pre><code class="lang-auto">model.qconfig = torch.quantization.get_default_qat_qconfig("fbgemm")
</code></pre>
<p>Currently this part is not extensively documented and I cannot find many resources.</p>
<p>So could someone give an example configuration for per-tensor quantization?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/kaizhao">@kaizhao</a> ,</p>
<p>The <code>qnnpack</code> backend has default settings with per-Tensor observers.  You can create a config with this setting like this:</p>
<pre><code class="lang-auto">model.qconfig = torch.quantization.get_default_qat_qconfig("qnnpack")
</code></pre>
<p>If you’d like to customize the qconfig manually, you could take a look here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/qconfig.py#L110" class="inline-onebox" rel="noopener nofollow ugc">pytorch/qconfig.py at master · pytorch/pytorch · GitHub</a></p>
<p>You can change just the per-channel setting with something like this:</p>
<pre><code class="lang-auto">        qconfig = QConfig(activation=FakeQuantize.with_args(observer=MovingAverageMinMaxObserver,
                                                            quant_min=0,
                                                            quant_max=255,
                                                            reduce_range=True),
                          weight=default_weight_fake_quant)
                          # weight=default_per_channel_weight_fake_quant)
</code></pre>
Transforming a tree to a dag in pytorch efficiently and without backward passes errors

username: Brando_Miranda
cooked: <p>I want to transform my tree neural net to a dag neural net. The forward pass is simple because I can simply detect if I’ve gone through that node/subtree already by caching a unique name for the node and the tensor. However, in the backward pass it is an issue because even if I reuse those tensors the auto grad engine complains that I’ve already done the backward pass through those tensors (and correctly so). Is there a way to modify the backward pass in a simple way so to re-use the already computed gradients?</p>
<p>Or is it just easier to abandon my old code and implement these models in Pytorch Geometric (PyG) or with the Deep Graph Library (dgl)?</p>
<p>For the sake of completeness here is the error message:</p>
<pre><code class="lang-auto">    train_loss, train_acc = agent.train(n_epoch)
  File "/home/miranda9/ML4Coq/ml4coq-proj-src/agents.py", line 102, in train
    loss.backward()  # each process synchronizes it's gradients in the backward pass
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<hr>
<p>cross posted: <a href="https://stackoverflow.com/questions/67476707/how-doe-some-transforms-a-tree-neural-net-to-a-dag-neural-net-in-pytorch-efficie" class="inline-onebox" rel="noopener nofollow ugc">machine learning - How doe some transforms a Tree Neural Net to a Dag Neural Net in Pytorch efficiently and without backward passes errors? - Stack Overflow</a></p>