akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CTCLoss returns inf on examples from the docs

username: Maghoumi
cooked: <p>I’ve been trying to use <code>CTCLoss</code> for one of my projects. For unknown reasons, I couldn’t get it to work, and it would always give me <code>inf</code> (or <code>0</code> if I set <code>zero_infinity=True</code>).</p>
<p>I decided to play around with the example included in the documentation of <a href="https://pytorch.org/docs/master/generated/torch.nn.CTCLoss.html" rel="nofollow noopener">nn.CTCLoss</a> and see if I can break it.</p>
<p>Turns out, for some inputs this function returns <code>inf</code>. See the example below. I don’t know enough about the internals of the CTC algorithm to know why such inputs yield <code>inf</code> (tested with PyTorch 1.5.1):</p>
<pre><code class="lang-auto">torch.manual_seed(4)

T = 375    # Input sequence length
C = 6      # Number of classes (including blank)
N = 1      # Batch size

# Initialize random batch of input vectors, for *size = (T,N,C)
input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()
input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)

# Initialize random batch of targets (0 = blank, 1:C = classes)
target_lengths = torch.randint(low=1, high=T, size=(N,), dtype=torch.long)
target = torch.randint(low=1, high=C, size=(sum(target_lengths),), dtype=torch.long)
ctc_loss = nn.CTCLoss()
loss = ctc_loss(input, target, input_lengths, target_lengths)
print(loss)  # prints "inf"
</code></pre>
<p>I wanted to open a bug on GitHub, but I didn’t want to waste anybody’s time in case I was doing something stupid. Any ideas what the issue could be?</p>
Current step from optimizer

username: y91
cooked: <p>Quick question: is there a way to get the <code>current_step</code> number from the optimizer, or does the user have to explicitly keep count? I am assuming it would maintain a step count inside it because that value would be required in annealing the LR.</p>
<p><code>optimizer.state_dict()</code> does give out a bunch of stuff but the <code>current_step</code> does not seem to be one of them.</p>
<p>Thanks!</p>

username: crcrpar
cooked: <p>Hi,</p>
<p>Some optimizers using <code>current_step</code> like <code>Adam</code> manage it as <code>step</code> in each param_group of <code>self.param_groups</code><br>
See <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L72-L106" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L72-L106</a>.</p>
<p>On the other hand, optimizers like naive <code>SGD</code> do not manage current_step.</p>

username: schaal
cooked: <p><code>step = optimizer.state[optimizer.param_groups[0]["params"][-1]]["step"]</code></p>
Is there Zoom tranform in Vision

username: Jaideep_Valani
cooked: <p>Do we have zoom function  in pipeline of Pytorch vision trasnforms ?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Is not zooming just cropping and upsampling using interpolation? In this case functions are available such as <code>RandomCrop</code> and <code>Resize()</code>.</p>
<p>Bests</p>

username: Jaideep_Valani
cooked: <p>no i look for zoom only. My image is not so big to crop and zoom.</p>

username: Nikronic
cooked: <p>You misunderstood my idea. Zooming itself is a combination of cropping and then resizing using interpolation. <code>zoom = crop -&gt; resize/upsample</code></p>
Can we use sequence labels for question answering?

username: Kushagra_Bhatia
cooked: <p>I have some Question, Context and Answer triplets for training and along with that, I have sequences inside the queries that are labelled into a few categories.</p>
<p>Example: Suppose 2 kinds of labels are there A and B.</p>
<p>Query: How is the weather outside?</p>
<p>Label A: How is</p>
<p>Label B: weather outside</p>
<p>I wish to encode the question and context using BERT and then use these labels associated with the tokens to extract the answer span from the context.</p>
<p>My Approach: I thought of defining 3 different weight vectors (dimension: 768*768) or using weight vector of size (768 using elementwise multiplication with the query tokens) for the tokens labelled A, B and others. Then using the new query encoding and the BERT context encoding in an ensemble for the answer span selection task.</p>
<p>Any other ideas to use the sequence labels for the MRC task?</p>
RNN/LSTM not powerful enough to learn

username: Tank
cooked: <p>Building a RNN/LSTM/GRU I am trying to overfit it on random data and I can’t get it to learn (accuracy won’t increase past 0.2). It keeps getting stuck at local optima. Idea of using random data comes from<a href="https://arxiv.org/pdf/1611.03530.pdf" rel="nofollow noopener">UNDERSTANDING DEEP LEARNING REQUIRES RETHINKING GENERALIZATION</a> . Any advice on what to change.</p>
<p>Input is (100 * 4 * 60) output is (100  * 10 ).</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
#         self.lstm1 = nn.LSTM(input_size = 60, hidden_size = 32, num_layers = 1, bidirectional= True)
#        self.lstm1 = nn.RNN(input_size = 60, hidden_size = 32, num_layers = 1, bidirectional= True)
        self.lstm1 = nn.GRU(input_size = 60, hidden_size = 32, bidirectional= True)
        self.fcn1 = nn.Linear(4 * 64, 512)
        self.fcn2 = nn.Linear(512, 512)
        self.fcn3 = nn.Linear(512, 10)
        
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        x, _ = self.lstm1(x)
#         print(x.shape)
        x = x.view(100, 4 * 64)
#         print(x.shape)
        x = self.fcn1(x)
        x = F.dropout(x, p = 0.5)
        x = self.fcn2(x)
        x = F.dropout(x, p = 0.5)
        x = self.fcn3(x)
        x = F.dropout(x, p = 0.9)

        x =x.squeeze(1)
  
        
        x = self.softmax(x)
        
        return x

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=0.0001)

</code></pre>
<p>I’ve tried different architectures, layers , drop-out, learning rates, learning rules, etc. Perhaps I should change the Loss function?</p>
<p>Training Code</p>
<pre><code class="lang-auto">for epoch in range(5000):
    
    optimizer.zero_grad()
    
    running_loss = 0.0
    outputs = net(X)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()
    running_loss += loss.item()
    
  
    
    if epoch % 100 == 0: 
        acc = accuracy_score(np.array(outputs.detach()).argmax(1), y)
#         print(np.array(outputs.detach()).argmax(1))
        print("Epoch: ", epoch, "Loss: ",running_loss, "Accuracy:", acc) 
</code></pre>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects raw logits, as internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied, so you should remove the <code>self.softmax</code> layer.</p>

username: Tank
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="86050">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>self.softmax</code> layer.</p>
</blockquote>
</aside>
<p>Thanks worked perfectly!</p>
I have probleme with calculate accuracy in this code

username: El_ouati_mohamed
cooked: <p>def train(input_variable, lengths, target_variable, mask, max_target_len, encoder, decoder, embedding,<br>
encoder_optimizer, decoder_optimizer, batch_size, clip, max_length=MAX_LENGTH):</p>
<pre><code># Zero gradients
encoder_optimizer.zero_grad()
decoder_optimizer.zero_grad()

# Set device options
input_variable = input_variable.to(device)
lengths = lengths.to(device)
target_variable = target_variable.to(device)
mask = mask.to(device)

# Initialize variables
loss = 0
print_losses = []
n_totals = 0

# Forward pass through encoder
encoder_outputs, encoder_hidden = encoder(input_variable, lengths)

# Create initial decoder input (start with SOS tokens for each sentence)
decoder_input = torch.LongTensor([[SOS_token for _ in range(batch_size)]])
decoder_input = decoder_input.to(device)

# Set initial decoder hidden state to the encoder's final hidden state
decoder_hidden = encoder_hidden[:decoder.n_layers]

# Determine if we are using teacher forcing this iteration
use_teacher_forcing = True if random.random() &lt; teacher_forcing_ratio else False

# Forward batch of sequences through decoder one time step at a time
if use_teacher_forcing:
    for t in range(max_target_len):
        decoder_output, decoder_hidden = decoder(
            decoder_input, decoder_hidden, encoder_outputs
        )
        # Teacher forcing: next input is current target
        decoder_input = target_variable[t].view(1, -1)
        # Calculate and accumulate loss
        mask_loss, nTotal = maskNLLLoss(decoder_output, target_variable[t], mask[t])
        loss += mask_loss
        print_losses.append(mask_loss.item() * nTotal)
        n_totals += nTotal
else:
    for t in range(max_target_len):
        decoder_output, decoder_hidden = decoder(
            decoder_input, decoder_hidden, encoder_outputs
        )
        # No teacher forcing: next input is decoder's own current output
        _, topi = decoder_output.topk(1)
        decoder_input = torch.LongTensor([[topi[i][0] for i in range(batch_size)]])
        decoder_input = decoder_input.to(device)
        # Calculate and accumulate loss
        mask_loss, nTotal = maskNLLLoss(decoder_output, target_variable[t], mask[t])
        loss += mask_loss
        print_losses.append(mask_loss.item() * nTotal)
        n_totals += nTotal

# Perform backpropatation
loss.backward()

# Clip gradients: gradients are modified in place
_ = nn.utils.clip_grad_norm_(encoder.parameters(), clip)
_ = nn.utils.clip_grad_norm_(decoder.parameters(), clip)

# Adjust model weights
encoder_optimizer.step()
decoder_optimizer.step()

return sum(print_losses) / n_totals</code></pre>

username: Scott_Hoang
cooked: <p>thanks for the head up.</p>

username: ptrblck
cooked: <p>What kind of issue are you facing calculating the accuracy?</p>

username: El_ouati_mohamed
cooked: <p><a href="https://discuss.pytorch.org/t/i-have-probleme-with-calculate-accuracy/86214">https://discuss.pytorch.org/t/i-have-probleme-with-calculate-accuracy/86214</a></p>
Indexing 3D Tensor using 3D Index

username: josauder
cooked: <p>I have a Tensor <code>X</code> of the shape <code>B x N x D</code>, a Batch of vectors of N points in D-Dimensional space.</p>
<p>I also have indices <code>idx</code> of the shape <code>B x N x K</code> where <code>K &lt; N</code>. In each element of <code>idx</code> there is an integer between 0 and <code>N</code>. I want to index <code>X</code> using <code>idx</code> such that I get a Tensor of the shape <code>B x N x K x D</code>.</p>
<p>Is this possible? In Tensorflow, I would use <code>tf.gather</code> as it works across many dims, but <code>torch.gather</code> only works in one dimension, if I understand the docs correctly.</p>
<p>Minimal Example</p>
<pre><code class="lang-auto">import torch
import numpy as np

B = 64
N = 128
D = 3
K = 20

X_np = np.random.normal(size=(B, N, D))
X = torch.from_numpy(X_np)

idx_np = np.random.randint(0, N, size=(B, N, K))
idx = torch.from_numpy(idx_np)

# What I want to do:
# index X using idx, such that I have B,N,K,D

# For-Loop Numpy Version
result = np.zeros((B, N, K, D))
for b in range(B):
    for n in range(N):
        for k in range(K):
            i = idx[b, n, k]
            result[b, n, k] = X[b, i]

# My pytorch attempt so far: (works by reshaping X to 2 dimensions and keeping an index offset helper)
idx_helper = torch.arange(B) * N
idx_helper = torch.reshape(idx_helper, [B, 1, 1])
X_flat = torch.reshape(X, [-1, D])
# Line below works in Tensorflow!! How do I do this in PyTorch??
result = torch.gather(X, idx+idx_helper)
</code></pre>
<p>Any help would be appreciated!</p>

username: HuynhLam
cooked: <p>I also have the same question.<br>
To clarify, assuming I have a tensor <strong>A</strong> which A.shape =  <strong>2 x 3 x 2</strong> (B x N x C)</p>
<blockquote>
<p>A = torch.tensor(<br>
[[[ 0.4930,  0.1150],<br>
[-0.2355,  1.1917],<br>
[-1.2421, -0.4383]],<br>
[[ 0.3099,  3.4751],<br>
[ 0.7780,  1.0990],<br>
[-0.0795,  0.1633]]])</p>
</blockquote>
<p>And an indices tensor <strong>idx</strong> where idx.shape = <strong>2 x 3 x 1</strong> (B x N x K)</p>
<blockquote>
<p>idx = tensor([[[0],<br>
[1],<br>
[2]],<br>
[[1],<br>
[2],<br>
[1]]])</p>
</blockquote>
<p>I would like to get a 4D tensor <strong>res</strong> that res.shape = <strong>2 x 3 x 1 x 2</strong> (B x N x K x C)</p>
<blockquote>
<p>res = tensor(<br>
[[[[ 0.4930,  0.1150]],<br>
[[-0.2355,  1.1917]],<br>
[[-1.2421, -0.4383]]],<br>
[[[ 0.7780,  1.0990]],<br>
[[-0.0795,  0.1633]],<br>
[[ 0.7780,  1.0990]]]])</p>
</blockquote>
<p>Currently, I implemented it</p>
<blockquote>
<p>res = torch.cat([A[i, idx[i]] for i in range(B)], dim=0).view(B, N, K, C)</p>
</blockquote>
<p>I am wondering are there more efficient ways to do this? (e.g. Get rid of the for loop)<br>
Thanks in advance!</p>
Best SSD for object detection

username: thejoe
cooked: <p>Hi all,<br>
I’m looking for a good pytorch implementation for SSD (single shot detector) which uses Resnet50 as backbone. My goal is training it on my custom dataset which contains small objects like bolts. I have already used faster rcnn and obtained good result but i want to compare it with the ssd. Do you know any good repository?</p>
Learning Rate Schedulers with multiple Google Cloud TPU cores

username: chhablanig
cooked: <p>Hi All!<br>
I wanted to use a CosineAnnealingWithRestarts schedulers on Google Cloud TPU cores, with distributed parallel training.<br>
I couldn’t find a tutorial which shows how to do that.</p>
<p>Please help me out.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/chhablanig">@chhablanig</a></p>
<p>Do you have a working training script without using LR schedulers?</p>
<p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for optimizer question<br>
cc <a class="mention" href="/u/ailzhang">@ailzhang</a> for XLA question</p>
Plotting loss and accuracy

username: east
cooked: <p>Hi,<br>
I want to use lists to record my accuracy and loss in order to plot them at the end. But when I add the lines for tracking the loss (I do it for training and testing) my accuracy gets worse.<br>
This is essentially how my code looks:</p>
<pre><code class="lang-auto">train_loss_history = []
train_loss_history = []

for epoch in range(epoch_resume, opt.max_epochs):
   ...
   for i, data in enumerate(trainloader, 0):
      train_loss     += imgs.size(0)*criterion(logits, labels).data
      ...
   train_loss     /= len(trainset)
   train_acc_history.append(train_acc)
   train_loss_history.append(train_loss)
</code></pre>
<p>I can’t imagine how appending the values to a list can affect the training. Do you have any idea what I am missing?</p>

username: Atul_Kumar
cooked: <p>For fully debug this issue I think I need to run the code. But first thing I would change is<br>
train_acc_history.append(train_acc.item())<br>
train_loss_history.append(train_loss.item())</p>

username: east
cooked: <p>Hi, thank you for your answer! Unfortunately using .item() did not work, but I tried to work around it and now use tensorboard to plot my loss and accuracy. Now it works fine:)</p>
How to concatenate multiple bi-lstm layers

username: victorN
cooked: <p>I am working on a relation extraction task between two entities in a sentence.<br>
For the model, I want to use Bi-LSTM model that takes three different parts of a sentence as a input:</p>
<ol>
<li>Left of the first entity 2. Right of the second entity 3. text between the two entities.</li>
</ol>
<p>In Keras, it seems that you create a separate LSTM for each of the input and concatenate all three using Concatenate as below.</p>
<pre><code class="lang-auto"> left_ph = Input((None,), dtype="string")
 bet_ph = Input((None,), dtype="string")
 right_ph = Input((None,), dtype="string")
 left_embs = bilstm(left_ph, rnn_state_size, num_buckets, embed_dim)
 bet_embs = bilstm(bet_ph, rnn_state_size, num_buckets, embed_dim)
 right_embs = bilstm(right_ph, rnn_state_size, num_buckets, embed_dim)
 layer = Concatenate(1)([left_embs, bet_embs, right_embs])
</code></pre>
<p>Since i’m new to Pytorch, i was wondering if there’s an approach to utilize multiple sentences in Bi-LSTM model for text-classification purpose.</p>

username: macwiatrak
cooked: <p>I have not done relation extraction so I don’t know what is the ‘default’ way of doing this and from your message I am not sure whether you are looking for the same method as in your Keras code snippet, so I am going to outline two methods:</p>
<ol>
<li>Creating separate BiLSTMs for each of 3 texts, and the concatenating it (to later put it into FFNN I suppose).</li>
<li>Concatenating 3 texts before parsing it into the BiLSTM.</li>
</ol>
<p>Option nr 1:</p>
<pre><code class="lang-auto">import torch
from torch import nn

left = torch.randn((1, 20)) # random input vector as an example input
entity = torch.randn((1, 10)) # random input vector as an example input
right = torch.randn((1, 20)) # random input vector as an example input

left_bilstm = nn.LSTM(input_size=20, hidden_size=16, num_layers=2, dropout=0.5, bidirectional=True)
entity_bilstm = nn.LSTM(input_size=10, hidden_size=16, num_layers=2, dropout=0.5, bidirectional=True)
right_bilstm = nn.LSTM(input_size=20, hidden_size=16, num_layers=2, dropout=0.5, bidirectional=True)

(left_bilstm_hidden_states, _) = left_bilstm(left.unsqueeze(0))
(entity_bilstm_hidden_states, _) = entity_bilstm(entity.unsqueeze(0))
(right_bilstm_hidden_states, _) = right_bilstm(right.unsqueeze(0))

concatenated_bilstm_hidden_states = torch.cat((left_bilstm_hidden_states, entity_bilstm_hidden_states,
                                               right_bilstm_hidden_states), dim=2)
</code></pre>
<p>Option nr 2:</p>
<pre><code class="lang-auto">import torch
from torch import nn

left = torch.randn((1, 20)) # random input vector as an example input
entity = torch.randn((1, 10)) # random input vector as an example input
right = torch.randn((1, 20)) # random input vector as an example input

concatenated_input= torch.cat((left, entity, right), dim=1).unsqueeze(0) # concatenating input

bilstm = nn.LSTM(input_size=50, hidden_size=32, num_layers=2, dropout=0.5, bidirectional=True)

(bilstm_hidden_states, _) = bilstm(concatenated_input) # getting the hidden states
</code></pre>
Meta Learning trainer

username: kl_divergence
cooked: <p>I am trying to implement meta learning trainer by myself. Since my data is different (non image), there are currently no proper implementations. I’ve seen plenty of implementations (<a href="https://github.com/tristandeleu/pytorch-meta" rel="nofollow noopener">1</a>, <a href="https://github.com/learnables/learn2learn" rel="nofollow noopener">2</a>, <a href="https://gist.github.com/thomwolf/788b55f427d67a4021bee500c3229a80#file-meta_train-py" rel="nofollow noopener">3</a>) online. First two make use of query set to update the loss with <code>backward()</code>. I liked the third one but it’s abstract (pseudo code). Can anyone post a good trainer that perform reptile/maml ?</p>

username: kl_divergence
cooked: <p>Please ignore, can’t delete this post. The query set can come from training set.</p>
How to get feature map from pre-trained Resnet50

username: asahi
cooked: <p>Hi!<br>
I want to extract feature maps from pre-trained ResNet50 model.<br>
Actually, I’m trying to make image-captioning model and need to get feature maps from images.<br>
However, I cannot find good ways to do that.<br>
Does anyone know the way for that and show me the code to make it possible?</p>

username: ptrblck
cooked: <p>You could use forward hooks to get the activation maps as described in <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this example</a>.</p>

username: asahi
cooked: <p>thank you for your quick answering.<br>
I’ll check it out!</p>
Nightly version release date

username: Walter1
cooked: <p>I need a new feature that just recently been committed into pytorch github. To be specificly, I need a nightly version that compiled after 20200615. I noticed that the most latest version of nightly is 20200610. So the question comes to me that when is the next version been released?  Thank you.</p>
<p>By the way, I have complied pytorch by myself, but it not ends well.</p>

username: ptrblck
cooked: <p>Nightly binaries should be created once per night, but can of course fail due to various reasons.<br>
What kind of error are you getting while trying to build from source?</p>

username: Walter1
cooked: <p>Thanks for reply.<br>
After setting right path and gcc version, last night I finally succeeded in compiling pytorch.<br>
But the nightly version still stays at 20200610, that’s weild.</p>

username: peterjc123
cooked: <p>Opened an issue in github: <a href="https://github.com/pytorch/pytorch/issues/40337" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/40337</a><br>
Related PR: <a href="https://github.com/pytorch/pytorch/pull/40338" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/40338</a></p>
Transform datasets for CNN model

username: Md_Golam_Morshed
cooked: <p>I have created datasets of training, validation, and test with the color images as list. Multiple images for a dataset were read from the different folders with their corresponding labels. The images were loaded with “image.imread” and append in a list and corresponding labels were append in a list as int. For example, the training dataset was created by</p>
<h3>list of array</h3>
<p>DPN_Train_image = list()<br>
DPN_Train_Label = list()</p>
<h3>appending the reading image and corresponding label</h3>
<p>DPN_Train_image.append(image.imread(data path +  DPN_Train_image_read))<br>
DPN_Train_Label.append(int(DPN_Train_Folder_Label))</p>
<p>Now, I want to transform the images as</p>
<h3>transform data</h3>
<p>transform = transforms.Compose([transforms.ToPILImage(),<br>
transforms.Resize((128, 32)),<br>
transforms.RandomHorizontalFlip(p=0.5),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.5, 0.5, 0.5],<br>
[0.5, 0.5, 0.5])])</p>
<p>And then, I want to put the transformed data() in one directory “train_loader”. I tried</p>
<p>Train_x_Train_y_dataset = (DPN_Train_image, DPN_Train_Label)<br>
train_loader = utils.data.DataLoader(Train_x_Train_y_dataset, batch_size=batch_size, num_workers=num_workers)</p>
<p>The “train_loader” will be used in a model</p>
<p>model.train()<br>
for data, target in train_loader:<br>
# move tensors to GPU if CUDA is available<br>
if train_on_gpu:<br>
data, target = data.cuda(), target.cuda()</p>
<p>I have tried to transform the image after reading and also tried after created the image list. Also tried various ways but nither of the work for the data sets. How should I solv it?</p>

username: supriyar
cooked: <p>This doesn’t seem related to quantization. Please update the post with the correct label (maybe vision)</p>

username: Scott_Hoang
cooked: <p>why not just write your own dataloader? It will be a lot easier</p>

username: Md_Golam_Morshed
cooked: <p>I think the problem is with image loading and storing or list of images<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bc1bc2a7f70fa30d7f9e050cf2caca000f27afe.png" alt="aaaa" data-base62-sha1="3XxXXrbSvgdU26RAxns1Cd7ZN1k" width="635" height="385"></p>
Loading a dataset by passing a class of trasnforamtions

username: 887574002
cooked: <p>Hi all,</p>
<p>I am trying to load STL10 dataset, by passing a class of transformation. My aim is to obtain two agumeneted versions of each image. But when I pass the class as a transformation argument, I got the message that the class is not iterable. I tried to make the class simpler. But I got the same message. Does it mean that we can not pass a class as transformations in loading datasets?</p>
<p>Here is a sample of code:</p>
<p>color_jitter = transforms.ColorJitter(0.8 , 0.8 , 0.8 , 0.2 )</p>
<p>data_transforms = transforms.Compose([transforms.RandomResizedCrop(size=96),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.RandomApply([color_jitter],p=0.8),<br>
transforms.RandomGrayscale(p=0.2),<br>
transforms.ToTensor()])</p>
<p>class Trans(object):</p>
<pre><code>def __init__(self,transform):
    
    self.transform = transform
    
def __cal__(self,sample):
    
    xi = self.transform(sample)
    
    xj = self.transform(sample)
    
    return(xi, xj)
</code></pre>
<p>dataset =datasets.STL10(’./data’,split = ‘train+unlabeled’, download = True,<br>
transform =Trans(data_transforms) )</p>
<p>Any help is appreciated.</p>
<p>Best</p>

username: ptrblck
cooked: <p>Could you post the complete error message with the stack trace, please?<br>
Also, you have a typo in <code>self.__call__</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> .</p>

username: 887574002
cooked: <p>Thank you for your answer. The problem was solved. It was just due to the typo in the <code>self.__call__</code> definition.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch/geometric pointsCNN

username: shangguan91
cooked: <p>Hey, i can not run this model: <a href="https://github.com/rusty1s/pytorch_geometric/blob/master/benchmark/points/point_cnn.py" rel="nofollow noopener">https://github.com/rusty1s/pytorch_geometric/blob/master/benchmark/points/point_cnn.py</a></p>
<p>error: No module named ‘points.datasets’.</p>
<p>i have tried clone whole respritory, the problem exist till.</p>

username: ptrblck
cooked: <p>Your current work directory might not be in the expected folder, as the <code>datasets.py</code> file is locally available.<br>
Make sure to execute the script from the parent folder.</p>

username: shangguan91
cooked: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/shangguan91/thesis-problems/blob/master/Cloud_point.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/shangguan91/thesis-problems/blob/master/Cloud_point.ipynb" target="_blank" rel="nofollow noopener">shangguan91/thesis-problems/blob/master/Cloud_point.ipynb</a></h4>
<pre><code class="lang-ipynb">{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Cloud point.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "aLx5txJjD3Qs",
</code></pre>

  This file has been truncated. <a href="https://github.com/shangguan91/thesis-problems/blob/master/Cloud_point.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>i tried everything, set up right directory, the answer is still no. i can run dataset.py, but can not run models.</p>

username: ptrblck
cooked: <p>The <code>point_cnn.py</code> code runs on my machine after installing the <code>benchmark</code> module as explained in <a href="https://github.com/rusty1s/pytorch_geometric/issues/1340">this issue</a> you’ve created.</p>

username: shangguan91
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="86028">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>installing the <code>benchmark</code></p>
</blockquote>
</aside>
<p>can you please send the code of<br>
installing the  <code>benchmark</code> Moduel?<br>
i have already tried ```<br>
<code>$ pip install -e .</code> and <code>!python benchmark/setup.py install</code></p>
<pre><code class="lang-auto"></code></pre>

username: ptrblck
cooked: <p>Run <code>python setup.py install</code> or <code>pip install -e .</code> in the <code>benchmark</code> folder.</p>

username: shangguan91
cooked: <p>Solved by running<code>!pip install benchmark/. -e .</code> thanks</p>
How to write a training program from a test program using Pytorch？

username: CVAIDL
cooked: <p>I want to write the training program from the test program about the paper “OptimizingThroughLearnedErrorsforAccurateSportsFieldRegistration”.<br>
But I don’t know how to write a training program from a test program.</p>

username: ptrblck
cooked: <p>Could you explain, what a test program would refer to?<br>
Would it be a specific application?</p>
How could I obtain gradients of tensors?

username: coincheung
cooked: <p>HI,</p>
<p>Support I have a module like this;</p>
<pre><code class="lang-auto">class Loss(nn.Module):

    def __init__(self, lam=0.3):
        super(Loss, self).__init__()
        self.lam = lam
        self.ce_crit = nn.CrossEntropyLoss(
                reduction='none', ignore_index=ignore_index)


    def forward(self, logits, label):
        '''
        args: logits: tensor of shape (N, C, H, W, ...)
        args: label: tensor of shape(N, H, W, ...)
        '''
        # overcome ignored label
        logits = logits.float()
        with torch.no_grad():
            num_classes = logits.size(1)
            coeff = 1. / (num_classes - 1.)
            idx = torch.zeros_like(logits).scatter_(1, label.unsqueeze(1), 1.)
        lgts = logits - idx * 1.e6

        q = lgts.softmax(dim=1) # how could I obtain grad of this tensor of q and log_q
        log_q = lgts.log_softmax(dim=1)
        mg_loss = ((q - coeff) * log_q) * (self.lam / 2)
        mg_loss = mg_loss.sum(dim=1)

        ce_loss = self.ce_crit(logits, label)
        loss = ce_loss + mg_loss

        return loss.mean()
</code></pre>
<p>I would like to know or manipulate the values of the gradient of the tensor of <code>q</code> and <code>log_q</code> during <code>loss.backward()</code>, how could I do this ?</p>

username: ptrblck
cooked: <p>You could use <code>.register_hook</code> on these tensors to manipulate the gradients.<br>
If you want to access the gradient after the <code>backward</code> call, you could also call <code>.retain_grad()</code> on them.<br>
Here is a small example:</p>
<pre><code class="lang-python">lgts = torch.randn(10, 10, requires_grad=True)

q = lgts.softmax(dim=1) 
q.retain_grad()
q.register_hook(lambda grad: grad*0.1)

log_q = lgts.log_softmax(dim=1)
log_q.retain_grad()
log_q.register_hook(lambda grad: grad*0.1)

mg_loss = q * log_q
mg_loss.mean().backward()
print(q.grad)
print(log_q.grad)
</code></pre>
Multiple output multiple task model

username: sougata_saha
cooked: <p>I have a network which performs regression and classification simultaneously.</p>
<pre><code class="lang-auto">class Network(nn.Module):
  def __init__(self, ...):
     ....
     self.gru=nn.GRU(input, hidden size, ...)
     self.classifier = nn.Linear(hidden size, 2)
     self.regressor = nn.Linear(hidden size, 1)

  def forward(self, input):
    output, hidden = self.gru(input)
    hidden = hidden.sum(dim=0)
    op_reg = self.regressor(hidden)
    op_class = self.classifier(hidden)

    return op_reg, op_class
</code></pre>
<p>I am defining 2 loss functions, MSELoss for regression and CategoricalCrossEntropy loss for classification. When I back propagate, I perform the following steps:</p>
<pre><code class="lang-auto">regression_loss = MSELoss()
classification_loss = CrossEntropyLoss()
optimizer = Adam ......
op_reg, op_class = model(input)
loss_reg = regression_loss(op_reg.squeeze(-1), Y_regression)
loss_class = classification_loss(op_class, Y_class)

loss = loss_reg + loss_class
loss.backward()
optimizer.step()
</code></pre>
<p>Since the scale of gradients in the cross entropy loss is going to be different from the scale of gradients from the MSE Loss, is it going to affect the training of the model, and if this is the correct way to train the model. Thanks</p>

username: Scott_Hoang
cooked: <p>this looks correct. Your “model” is simply 2 models that shared a hidden layer. Your loss backpropagation should be fine.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sougata_saha" data-post="1" data-topic="86115">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sougata_saha/48/23629_2.png" class="avatar"> sougata_saha:</div>
<blockquote>
<p>Since the scale of gradients in the cross entropy loss is going to be different from the scale of gradients from the MSE Loss, is it going to affect the training of the model, and if this is the correct way to train the model.</p>
</blockquote>
</aside>
<p>You could use <code>.register_hook</code> on the <code>model.gru</code> parameters and check the gradient magnitude.<br>
Both losses will accumulate the gradients in this layer, so you might need to reduce the learning rate for these parameters.</p>
Linear Interpolate along Channel/Z axis?

username: soulslicer
cooked: <p>I have a tensor of size [B, 64, 256, 384]. I want to stretch it softly along the channel axis so it becomes [B, 256, 256, 384].  I tried doing this but i get an error</p>
<pre><code class="lang-auto">dpv_refined_predicted = F.interpolate(dpv_refined_predicted, size=[256,256,384], mode='linear')
</code></pre>
<pre><code class="lang-auto">dpv_refined_predicted = F.upsample(dpv_refined_predicted, size=[256,256,384], mode='linear')
</code></pre>
<pre><code class="lang-auto">NotImplementedError: Got 4D input, but linear mode needs 3D input
</code></pre>

username: ptrblck
cooked: <p><code>F.intepolate</code> would need to use <code>mode='bilinear'</code> for 4D inputs and could only interpolate the spatial dimensions.<br>
To interpolate in the channel dimension, you could permute the input and output as shown here:</p>
<pre><code class="lang-python">B = 2
x =  torch.randn([B, 64, 256, 384])
x = x.permute(0, 2, 1, 3)
out = F.interpolate(x, [256, 384], mode='bilinear')
out = out.permute(0, 2, 1, 3)
</code></pre>
Problem with my GAN

username: Co3Adriano
cooked: <p>Hey im currently working on a GAN for generating new Data on Heavy Rain Events.<br>
I have a Datasets of 77 Heavy Rain events from Aachen, Germany as 24x19 Matrices.</p>
<p>Basically i have it set up and running.</p>
<p>Here is my Code so far:</p>
<p><a href="https://github.com/Co3Adriano/GAN-/blob/master/HRGAN_Final_linear.py" rel="noopener nofollow ugc">https://github.com/Co3Adriano/GAN-/blob/master/HRGAN_Final_linear.py</a></p>
<p>while my in input data looks like this for example:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee926c0fbf7f95cb75f181ea6f89dfb46821127c.png" alt="RW_20050811-1745" data-base62-sha1="y2vlrE9w5ol7AWRemSYMCNB05nC" width="640" height="480"></p>
<p>after 18000 iterations<br>
my output Data looks often more like <img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/affccb0518b991c858235a65d2ec92432d872b6f.png" alt="result" data-base62-sha1="p6Raa9NoDR6vWzrI4KbjfzTwILd" width="640" height="480"><br>
and after 30000iterations<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0b4e3bca9b4d6521980e1fd76fa80bf2b399edc.png" alt="result2" data-base62-sha1="mVFWRFQ5JtqXHMY4ZX55YxY2pXu" width="640" height="480"></p>
<p>it looks  more like a overall distribution than single heavy rain events</p>
<p>Is it because my generator topology is to “linear” ?<br>
Is it because i have no labeling on my Dataset?</p>
<p>Since im new to Python and Pytorch, i am grateful for every tip and addition to my Code.</p>
<p>Thank You in advance</p>

username: Scott_Hoang
cooked: <p>I think you model is suffering from what I call “garbage in garbage out”.<br>
you want your generator to generate new data from a randomized vector with no statistical relationship to ground truth? no way.<br>
My suggestion is first to come up with a way to describe your ground truth as a vector of z_dim that is statistically relevant. Then feed that to your generator. Your generator will eventually learn enough prior so that when you do feed it a new randomly generated vector it can output plausible new data.<br>
If you don’t want to do that. Then I suggest changing your Generator network to an Encoder-Decoder set up.</p>

username: Co3Adriano
cooked: <p>Hey Scott_Hoang , Big Thanks for your reply, I will try to implement your first Suggestion.</p>
Tensor.std(0) returns a scalar instead of a vector

username: Bryan_Wang
cooked: <p>I’m a bit new to the C++ API, and I haven’t seen documentation for this behavior anywhere. Here is a minimal example to reproduce.</p>
<pre><code class="lang-auto">float data[] = { 1, 2, 3,
                       4, 5, 6 };
torch::Tensor f = torch::from_blob(data, {2, 3});
std::cout &lt;&lt; f.mean(0) &lt;&lt; std::endl;
// yields a 1-d tensor as expected.
// 2.5000
// 3.5000
// 4.5000
// [ CPUFloatType{3} ]
std::cout &lt;&lt; f.std(0) &lt;&lt; std::endl;
// yields a scalar, unexpected.
// 1.70783
// [ CPUFloatType{} ]
</code></pre>

username: ptrblck
cooked: <p>The argument will be interpreted as the <code>unbiased</code> argument, if a single argument is passed, which is not intuitive.<br>
You could use <code>f.std(0, true)</code> as a workaround.<br>
Could you also create an issue on GitHub and report it there, so that this behavior might be improved or at least the documentation added? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Bryan_Wang
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the answer. Here is the issue: <a href="https://github.com/pytorch/pytorch/issues/40287" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/40287</a>. Let me know if it needs to be revised, it is my first time creating one.</p>
<p>I also noticed that  <code>f.std(true, 0)</code> collapses dimension 1, which is unintuitive, and also <code>f.std(true, 1)</code> collapses dimension 1 still but uses Bessel correction. All in all kinda weird behavior.</p>
<p>I’m actually not blocked on this as it’s a pretty easy to implement function (for my specific case), just pointing it out in case it is interesting to you guys.</p>

username: ptrblck
cooked: <p>Thanks for creating the issue! I’ve answered and it seems it worked for you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Just for the sake of completeness, I had a typo in my previous code, so posting the code snippet here again:</p>
<pre><code class="lang-python">from torch.utils import cpp_extension


cpp_source = """
void my_fun(torch::Tensor x)
{
    std::cout &lt;&lt; "x.std(0) " &lt;&lt; x.std(0) &lt;&lt; ", x.std(0, true) " &lt;&lt; x.std(0, true) &lt;&lt; std::endl;
    std::cout &lt;&lt; "at::std(x, 0) " &lt;&lt; at::std(x, 0) &lt;&lt; ", at::std(x, 0, true) " &lt;&lt; at::std(x, 0, true) &lt;&lt; std::endl;
    return;
}
"""

module = torch.utils.cpp_extension.load_inline(
    name="std_test_extension",
    cpp_sources=cpp_source,
    functions="my_fun",
    verbose=True,
)

x = torch.randn(2, 3)
module.my_fun(x)

print('python ', x.std(0))
</code></pre>
Dose static quantization support CUDA?

username: uni1
cooked: <p>I want to know whether the quantized model obtained by Post Training Static Quantization can be run on CUDA?</p>

username: jerryzh168
cooked: <p>No, it only works on CPU right now, we will consider adding CUDA support in the second half of the year</p>

username: uni1
cooked: <p>I see. Thank you for your reply!</p>
Wrong C++ compiler version reported by Python build process

username: dbanas
cooked: <p>I’m invoking my Python build process, using the following command:</p>
<pre><code class="lang-auto">CMAKE_ARGS="-DLLVM_DIR=/usr/local/opt/llvm/lib/cmake/llvm 
-DCMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang 
-DCMAKE_CXX_COMPILER=/usr/local/opt/llvm/bin/clang++" \
python setup.py --run_cmake --release install
</code></pre>
<p>And I’m seeing this in the resultant output:</p>
<pre><code class="lang-auto">-- ******** Summary ********
--   CMake version         : 3.17.3
--   CMake command         : /usr/local/Cellar/cmake/3.17.3/bin/cmake
--   System                : Darwin
--   C++ compiler          : /usr/local/opt/llvm/bin/clang++
--   C++ compiler version  : 7.1.0
</code></pre>
<p>But, that’s not the right version number:</p>
<pre><code class="lang-auto">$ /usr/local/opt/llvm/bin/clang++ --version
clang version 8.0.1 (tags/RELEASE_801/final)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin
</code></pre>
<p>Does anyone know why this might be happening?</p>
Pytorch DDP failed to resume after validation using a clone of the model

username: Scott_Hoang
cooked: <pre><code class="lang-auto">dataloader = ....(create ddp loader with ddp settings)
opts = ...parse() # user options
master = opts.local_rank == 0

model = create_model(opt)
model_ema = model.clone().eval() # keeping track of exponential moving average for model's weights

for data in dataloader():
     # typical training code ... forward, backward and the likes 
      update_ema_weights(model_ema, model.state_dict()) # update the weights for model's team
      if opt.validate:
         if master:
               for data in valid_dataloader():
                     output = model_ema(data).... # typical validate code
         torch.cuda.synchronize()
</code></pre>
<p>given the above pseudo-code, after validation, my DDP process will hang on all GPUs.<br>
However, if I use model instead of model_ema for validation, it will not. Does anyone know how to fix this?</p>

username: mrshenli
cooked: <p>Would I be correct if I assume <code>model</code> (and <code>model_ema</code> as well) is a <code>DistributedDataParallel</code> instance? If so, the <code>forward</code> method of <code>DistributedDataParallel</code> will set some internal flags, which could cause hang.</p>
<p>If you just want to evaluate, you can use <code>model.module</code> to retrieve the original non-DDP module. And then clone, set <code>eval()</code>, run <code>forward</code> on the original non-DDP module. This should keep DDP states intact.</p>

username: Scott_Hoang
cooked: <p>I see. I will give it a shot. But it doesn’t explain why if I use “model” to evaluate, no hang occur.</p>

username: mrshenli
cooked: <p>Good point. How did you implement the <code>model.clone()</code> method? IIUC, neither <code>nn.Module</code> nor <code>DistributedDataParallel</code> has a <code>clone()</code> method. Is this a <code>copy.deepcopy</code>?</p>

username: Scott_Hoang
cooked: <p>i use deepcopy to clone the model.</p>
Tensordot equivalent in pytorch

username: cadurosar
cooked: <p>Helloe veryone, I’ve been using tensorflow and numpy for a while, but now I wanted to dedicate myself to pytorch. My only problem is that I need to do a custom layer, which uses tensordot (<a href="https://www.tensorflow.org/api_docs/python/tf/tensordot" rel="nofollow noopener">https://www.tensorflow.org/api_docs/python/tf/tensordot</a>) in tensorflow.</p>
<p>is there something in pytorch that I can use that is close to that tensordot?</p>

username: colesbury
cooked: <p>There’s no <code>tensordot</code> in PyTorch. You can simulate it using a combination of <code>exapand</code>, <code>*</code> (<code>mul</code>), and <code>sum</code>.</p>
<p>Tensordot seems generally useful. If you’d like it, file an issue on the PyTorch <a href="https://github.com/pytorch/pytorch/issues/new">github repo</a>:</p>

username: deanmark
cooked: <p>I adapted the numpy tensordot implementation to pytorch: <a href="https://gist.github.com/deanmark/9aec75b7dc9fa71c93c4bc85c5438777" rel="nofollow noopener">pytorch tensordot</a></p>

username: blade
cooked: <p>For future readers: <a href="https://pytorch.org/docs/master/generated/torch.tensordot.html" rel="nofollow noopener">torch.tensordot</a> is now available.</p>
Number of Channels based on size of Feature Maps?

username: ttbb
cooked: <p>I have an AlexNet like neural network like:</p>
<pre><code class="lang-auto">        self.conv = nn.Sequential()
        self.conv.add_module('conv1_s1',nn.Conv2d(3, 64, kernel_size=9, stride=2, padding=0))
        ..
        self.conv.add_module('conv2_s1',nn.Conv2d(64, 96, kernel_size=5, padding=2, groups=2))
        ..
        self.conv.add_module('conv3_s1',nn.Conv2d(96, 128, kernel_size=3, padding=1))
        ..
        self.conv.add_module('conv4_s1',nn.Conv2d(128, 128, kernel_size=3, padding=1, groups=2))
        ..
        self.conv.add_module('conv5_s1',nn.Conv2d(128, 96, kernel_size=3, padding=1, groups=2))

        self.fc6 = nn.Sequential()
        self.fc6.add_module('fc6_s1',nn.Linear(96*2*2, 256))
        ..
        ..
</code></pre>
<p>The output of <strong>conv5_s1</strong> is 96 channels of size 2x2 (see fc6_s1 after flattening). Does it make sense to have such a high number of channels for a small output dimension of 2x2?. So, the same feature map will be produced many times. Would 24 channels in layer conv5_s1 be sufficient, since 2x2=4 factorial = 24 possibilities? Or is this a wrong understanding?</p>
<p>Thanks for help!</p>

username: tom
cooked: <p>I must admit I don’t follow your reasoning.</p>
<p>My intuition would be more the following:</p>
<ul>
<li>In the input, you have largish spatial dimensions and only 3 channels. There isn’t much information per pixels in it.</li>
<li>In the output (the classification), you don’t have any spatial dimension but typically as many “channels” as you have classes - you might say you have 1x1 x classes. So there much information per pixel - in the single pixel. But if you come from inputs like 200x200x3, this is much lower than 1x1x1000.</li>
</ul>
<p>The typical pattern in the inner layers of ResNet is to double number of channels while halving the spatial resolution in both channels - so a new halving of the dimension.</p>
<p>I’m not sure how you count “feature maps” here, maybe that is part of why it is complicated. I would take a simpler approach and think of dimensionalities of the intermediate layers.</p>
<p>Best regards</p>
<p>Thomas</p>

username: ttbb
cooked: <p>Hi Thomas,<br>
it was an error in my reasoning, indeed.<br>
Thanks a lot for clarification.</p>
<p>best regards<br>
Talat</p>