akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Saving and loading custom models

zimmer550: <p>Hi, my custom model looks like this:</p>
<pre><code class="lang-auto">        mask_conv_model = models.resnet50(pretrained=True)
        for param in mask_conv_model.parameters():
            param.requires_grad = False
        num_ftrs = mask_conv_model.fc.in_features
        mask_conv_model.fc = nn.Sequential(nn.Linear(num_ftrs, masked_output_ftrs), nn.ReLU(), nn.Dropout())
        for name, param in mask_conv_model.named_parameters():
            if param.requires_grad == True:
                mask_params.append(param)
        mask_conv_model_optimizer = optim.Adam(mask_params, lr=masked_learning_rate)
</code></pre>
<p>Then later on, I save my model:</p>
<pre><code class="lang-auto">torch.save({
        'epoch': epoch,
        'target_seq_len': hype['target_seq_len'],
        'input_seq_len': hype['input_seq_len'],
        'masked_output_ftrs': hype['masked_output_ftrs'],
        'mask_conv': dict_of_models['MASKED CONV MODEL'].state_dict(),
    }, os.path.join(save_dir, "MASK_CONV.tar"))
</code></pre>
<p>Now, to load the model I have to write a lot of code:</p>
<pre><code class="lang-auto">        mask_conv_model = models.resnet50(pretrained=True)
        mask_checkpoint = torch.load(mask_dir)
        epoch = mask_checkpoint['epoch']
        masked_output_ftrs = mask_checkpoint['masked_output_ftrs']
        num_ftrs = mask_conv_model.fc.in_features
        mask_conv_model.fc = nn.Linear(num_ftrs, masked_output_ftrs)
        mask_conv_model.load_state_dict(mask_checkpoint['mask_conv'])
</code></pre>
<p>Is there a one-line method of loading models? Something like this:</p>
<pre><code class="lang-auto">mask_checkpoint = torch.load(mask_dir)
mask_conv_model = mask_checkpoint.load_state_dict(mask_checkpoint['mask_conv'])
</code></pre>
How to split the tensor with multiple channel

yuyaya: <p>Hello, I’d like to ask ideas on how to split the tensor of shape (10, 1024, 2048)&lt;c, h, w&gt; to length 512 of (10, 64, 64) tensors when I load data.</p>
<p>Now I have 1000 tensors of shape (10, 1024, 2048) and when I load this tensor, I’d want to transform it to 512 of (10, 64, 64).</p>
<p>Thank you</p>

ptrblck: <p>How would you like to transform the 1000 original tensors?<br>
Would you like to slice them somehow or reduce them?</p>

yuyaya: <p>Thank you for the reply.</p>
<p>I want to slice the array.<br>
The code below is working but I was wondering  if there are more efficient ways prepared by PyTorch developer (Basically I want to avoid loops)</p>
<p>By the way,  I changed inputs from torch.Tensor to numpy.ndarray.</p>
<pre><code class="lang-python">train &lt;numpy.ndarray&gt;: (1000, 19, 1024, 2048)
val &lt;numpy.ndarray&gt;: (1000, 19, 1024, 2048)

def logit_preprocess(dataset):
    
    split_len = 64

    arr = []
    h_len = 1024 // split_len
    w_len = 2048 // split_len
    
    for d in tqdm(dataset):
        for h in range(h_len):
            h_start = h * split_len
            h_end = h_start + split_len
            for w in range(w_len):
                w_start = w * split_len
                w_end = w_start + split_len
                
                arr.append(d[:, h_start:h_end, w_start:w_end])
            
    return np.array(arr)


train_data = logit_preprocess(train)
val_data = logit_preprocess(val)


class MyDataset(Dataset):
    def __init__(self, data, transform=None):
        self.data = torch.from_numpy(data).float()
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        if self.transform:
            x = self.transform(x)
        
        return x
    
    def __len__(self):
        return len(self.data)

train = MyDataset(train_data)
val = MyDataset(val_data)

batch_size = 1

train_loader = DataLoader(train, batch_size=batch_size,shuffle=False,num_workers=2)
validation_loader = DataLoader(val, batch_size=batch_size,shuffle=False,num_workers=2)
</code></pre>

phan_phan: <p>From <a href="https://discuss.pytorch.org/t/how-to-extract-smaller-image-patches-3d/16837/4">this answer</a>, if your tensor <code>train</code> has a shape <code>[1000, 19, 1024, 2048]</code>, you could do :</p>
<pre><code class="lang-auto">train_data = train.unfold(2, 64, 64).unfold(3, 64, 64)
    .permute(0, 2, 3, 1, 4, 5)
    .contiguous()
    .view(-1, 19, 64, 64)
</code></pre>
<p>It should return the desired tensor of shape <code>[512000, 19, 64, 64]</code></p>
Network forward output is Nan, without backward

cb_zhang: <p>I run the wide-resnet, while input sub. mean and div. std .<br>
Then I checked the input and model parameters, they are seemed normal.<br>
And when I run on GPU:0, it is ok, but run on GPU:1, it is wrong.<br>
I use “CUDA_VISIBLE_DEVICES=0” to switch the GPU.<br>
So what’s wrong here???<br>
I use CUDA-10.0, pytorch1.2.0, cudnn=True</p>

ptrblck: <p>Are you getting the NaN output immediately after the first forward pass?<br>
Could you try to run the code with anomaly detection and check, which layer creates the NaN?</p>

cb_zhang: <p>Thanks for your reply, how to run the code with anomaly detection?</p>

cb_zhang: <p>I have checked the model, it firstly happened after conv layer…After a conv layer, its value is -inf</p>

ptrblck: <p>Is this issue reproducible? Did you check the input and parameters for NaN values?<br>
If so, could you store the activation and layer <code>state_dict()</code> so that we could have a look?</p>
Crop Multiple Images with Indices

SomeOneIntheWorld: <p>Hi all,<br>
I want to crop images with given indices like follows:</p>
<pre><code class="lang-auto">images = given Tensor with shape N x C x H x W
lt_x = given Tensor with shape N
lt_y = given Tensor with shape N
rb_x = given Tensor with shape N
rb_y = given Tensor with shape N
for i in range(images.size(0)):
     croped_images.append(images[i, :, lt_x[i]: rb_x[i], lt_y[i]: rb_y[i]])
</code></pre>
<p>But the for loop is too slow. Thus I want to know if there are some batch operation that can do the similar operation, like</p>
<pre><code class="lang-auto">croped_images = images[:, :, lt_x: rb_x, lt_y: rb_y]
</code></pre>
<p>Thanks.</p>

phan_phan: <p>Hi,<br>
I don’t know of a way to do this, but <a href="https://discuss.pytorch.org/t/crop-rectangular-portions-from-a-batch-of-tensors/57037">here’s a link</a> to someone trying to do exactly the same thing.<br>
Good luck</p>
GAN for 1d data?

ljj7975: <p>I am trying to train a model for 1d data that has 500 features.</p>
<p>Unfortunately, since the common implementation of GAN is for image,<br>
I wasn’t able to find a working version of GAN for 1d data.</p>
<p>Does anyone know any public implementation?<br>
Is there anything specific that I have to keep in mind?</p>
<p>I preprocessed the input data to have range of -1 to 1 to match with Tanh() and also tried different activation functions but none seems to give me promising results…</p>
<p>Thank you for the time</p>

ptrblck: <p>I’ve reimplemented a toy example from <a href="http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/" rel="nofollow noopener">this blog post</a> a while ago and you can find the code <a href="https://github.com/ptrblck/gan-intro-pytorch" rel="nofollow noopener">here</a>.<br>
Since this code is quite old by now, you might need to change some details (e.g. swap <code>data[0]</code> for <code>.item()</code>).</p>

ljj7975: <p>Thank you so much.<br>
I can try to adapt some of your approaches.</p>
<p>The only thing is that,<br>
when I said 1d, I meant 1xd, where d is number of features.<br>
it seems like your implementation is for generates a single number.</p>
<p>I guess things you should workout without much patching, right?</p>

ptrblck: <aside class="quote no-group" data-username="ljj7975" data-post="3" data-topic="50882">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/48/12561_2.png" class="avatar"> ljj7975:</div>
<blockquote>
<p>I guess things you should workout without much patching, right?</p>
</blockquote>
</aside>
<p>I hope so. If it’s not that easy and you get stuck somewhere, just let me know and I can try to help out. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

mese79: <p>I’ve implemented some basic codes for GANs with plots showing training progress. you can find it here:<br>
<a href="https://github.com/mese79/GANs_experiments" rel="nofollow noopener">https://github.com/mese79/GANs_experiments</a></p>
How to achieve mutual conversion between at::tensor and tensor::tensor?

Chunsheng_Guo: <p>How to achieve mutual conversion between at::tensor and tensor::tensor? At::tensor print shows [cputype], tensor::tensor print shows variable[cputype].</p>

tom: <p>So this doesn’t answer your question, but it’ll likely solve your problem:<br>
Always work with torch::tensor. If you create new tensors, pass the .options() of an existing tensor to make it look right to everything else.<br>
Then you should not encounter the need to convert tensors.<br>
The only place where you currently need to care about subtle things are autograd::Functions, where you use Variables, but that will go away eventually.</p>
<p>Best regards</p>
<p>Thomas</p>

Chunsheng_Guo: <p>I’ve got an at::tensor variable and a tensor::tensor, and now I want to use these two tensors for subsequent calculations, but the types are different. Is it necessary to create an at::tensor variable with tensor::tensor data. thank you very much for your help.</p>

tom: <p>So what I am trying to suggest is to share a bit about where you got the at::tensor from, because likely that can be helped.</p>
<p>Best regards</p>
<p>Thomas</p>
Using TensorDataset instead of ImageFolder in DCGAN example

tomt: <p>Instead of working with image files such as .png and .jpg, I’ve extracted three variables u, v, w from a NETCDF file, with dimension as (hours, number of grid points in X, number of grid points in Y). The u,v,w components can be seen as three separate channels, just like an image has RGB channels.Afterwards, I transform all the components into tensors using torch.from_numpy, and reshape them into (hours, channel, gridX, gridY), and concatenate them to get my data. Thus the tensor dimension is (745, 3, 128, 128).  Furthermore I have low resolution data on the form (745, 3, 64, 64).</p>
<h1>Concatenating the tensors together like three RGB channels</h1>
<p>HR_data = torch.cat((u_tensor,v_tensor,w_tensor), dim=1) # output = dim ( 745, 3, 128, 128)<br>
LR_data = torch.cat((u_tensor_lr,v_tensor_lr,w_tensor_lr), dim=1) # output = dim ( 745, 3, 64, 64)</p>
<h1>Normalizing</h1>
<p>HR_data_norm = (HR_data - HR_data.mean()) / HR_data.std()<br>
LR_data_norm = (LR_data - LR_data.mean()) / LR_data.std()</p>
<h1>Creating training set</h1>
<p>dataset_train = torch.utils.data.TensorDataset(LR_data_norm, HR_data_norm)<br>
trainloader = torch.utils.data.DataLoader(dataset_train, batch_size=batchSize,<br>
shuffle=True, num_workers=0)</p>
<p>I’m trying to fit my tensor data into a dcgan, using this as a basis <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a> , and I’m treating my data just like images as they are transformed into tensor form. I have set some hyperparameters such as:<br>
nz = HR_data.shape[0] <span class="hashtag">#size</span> of generator input<br>
ngf = 64 <span class="hashtag">#size</span> of feature maps in generator<br>
ndf = 128 <span class="hashtag">#size</span> of feature maps in discriminator<br>
batchSize = 64</p>
<p>However, when I try to train the discriminator with the real image, i.e. high resolution data, I get this error:<br>
“ValueError: Target and input must have the same number of elements. target nelement (64) != input nelement (1600)”<br>
which means that the binary cross entropy function need the input and target to be of the same dimension. My target dim is of 64 (I guess it’s the batch size), and output dim is of 1600.<br>
My question is therefore, how do I change the training loop and the DCGAN_example code to work with my data? I think there must be something wrong in my discriminator.</p>
<p>tl;dr: I want to train DCGAN with low and high resolution data. How to change DCGAN_example with TensorDataset instead of ImageFolder?</p>
Using tensor data instead of ImageSetFolder to feed into ESRGAN

tomt: <p>Instead of working with image files such as .png and .jpg, I’ve extracted three variables u, v, w  from a NETCDF file, with dimension as (hours, number of grid points in X, number of grid points in Y). The u,v,w components can be seen as three separate channels, just like an image has RGB channels. Afterwards, I transform all the components into tensors using torch.from_numpy, and reshape them into (hours, channel, gridX, gridY), and concatenate them to get my data. Thus the tensor dimension is (745, 3, 132, 132).</p>
<p>Now I want to plug it in an ESRGAN, <a href="https://github.com/xinntao/ESRGAN" rel="nofollow noopener">https://github.com/xinntao/ESRGAN</a> .<br>
The problem is, the dataloader is using ImageSetFolder class, and take images on the form .png as inputs. In PyTorch we usually use dataloader ImageSetfolder which loads the images automaticailly. However, now my data is already given in tensor form. What I can do from here? For practical use, the tensor data can be treated as RGB images. Thinking about maybe utilizing TensorData class? Or just removing the whole dataloader and sample from my dataset or something.</p>
<p>Help appreciated!</p>

ptrblck: <p>I’m not sure if the posted shape (<code>[745, 3, 132, 132]</code>) is a single sample or your complete dataset.<br>
In the latter case, you could simply use a <code>TensorDataset</code>, pass it to a <code>DataLoader</code>, and remove the currently used <code>Dataset</code> from the code.</p>

tomt: <p>It is the latter case. But how will I change the code, as it expects me to pass some images in a folder in order to use dataroot? I also have to normalize it, thinking about creating my own transform or something like that. I will keep you further updated.</p>

ptrblck: <p>I’m not sure which script you are using, but instead of loading the images you would just replace these lines with your loaded tensors.<br>
What kind of transformations would you like to apply?</p>

tomt: <p>By transformation I meant normalization, but I fixed that. I’m trying to fit my tensor data into a dcgan, using this as a basis <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a> .<br>
However I will make a new discussion thread as I now have a more clear understanding of what I’m asking for, so you can gladly delete this discussion.</p>
How could I export model to onnx model with older ir version?

coincheung: <p>Hi,</p>
<p>I am trying to export pytorch model to onnx with onnx2trt. When I run the command:</p>
<pre><code class="lang-auto">    $ onnx2trt model.onnx model.trt
</code></pre>
<p>I got the error of :</p>
<blockquote></blockquote>
<p>WARNING: ONNX model has a newer ir_version (0.0.4) than this parser was built against (0.0.3).<br>
Parsing model<br>
ERROR: /root/build/TensorRT/parsers/onnx/ModelImporter.cpp:523 In function importModel:<br>
[4] Assertion failed: !_importer_ctx.network()-&gt;hasImplicitBatchDimension() &amp;&amp; “This version of the ONNX parser only supports networks with an explicit batch dimension”</p>
<p>How could I export my onnx model with an older version so that it could compatible with tensorrt?</p>
Cifar10 tutorial accuracy is different at each run

amir1: <p>I am following official tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a> “Training a classifier”.<br>
I used google colab to run the code. Each time I click run all cells, I get very different accuracy.<br>
Like 1 time I got:<br>
Accuracy of plane : 53 %<br>
Accuracy of   car : 68 %<br>
Accuracy of  bird : 37 %<br>
Accuracy of   cat : 56 %<br>
Accuracy of  deer : 37 %<br>
Accuracy of   dog : 41 %<br>
Accuracy of  frog : 63 %<br>
Accuracy of horse : 53 %<br>
Accuracy of  ship : 74 %<br>
Accuracy of truck : 68 %</p>
<p>and another time:<br>
Accuracy of plane : 54 %<br>
Accuracy of   car : 62 %<br>
Accuracy of  bird : 49 %<br>
Accuracy of   cat : 30 %<br>
Accuracy of  deer : 34 %<br>
Accuracy of   dog : 36 %<br>
Accuracy of  frog : 71 %<br>
Accuracy of horse : 63 %<br>
Accuracy of  ship : 82 %<br>
Accuracy of truck : 70 %</p>
<p>I understand that seed is not used, but why we are getting so much different accuracy?. I mean what part of code is responsible for such a large variation and why?<br>
Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

amir1: <p>help please <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
Pre-trained googlenet performance dropped when the input image was normalized with imagenet statistics

mderakhshani: <p>Hi,<br>
I have used the GoogleNet pre-trained model and normalized the input image by the Imagenet statistics, but significantly the performance dropped. But when I used the input tensor in the range of [0,1], I saw that the model worked better. Did anyone have any experience with this issue?</p>

ptrblck: <p>What kind of data are you using at the moment and what accuracies do you get using both approaches?</p>

mderakhshani: <p>Thank you very much for your quick reply. Let me explain my task for you. I worked on Imagenet validation dataset to train an adversarial perturbation generator in which a freezed googlenet model was used.</p>
<p>I saw <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py#L117" rel="nofollow noopener">this line</a> of code and I inferred that when we set <code>pretrained=True</code> in Googlenet constructor, the forward function normalized the input image differently (seems it works with different statistics).</p>
<p>About the accuracy, I did not directly calculate it but in terms of fooling rate in adversarial attack task which I achieved, I saw that the input image with the range [0,1] had far better performance.</p>
How to deal with sudden loss spike?

hadaev8: <p>Model is tacotron2 gst.<br>
Almost same to this <a href="https://github.com/mozilla/TTS/blob/master/models/tacotrongst.py" rel="nofollow noopener">https://github.com/mozilla/TTS/blob/master/models/tacotrongst.py</a><br>
I left train at night and fond this<br>
<a href="https://i.imgur.com/59vuPyf.png" rel="nofollow noopener">https://i.imgur.com/59vuPyf.png</a></p>
<p>I have 2 thoughts on this:<br>
1 try to track what data is it, may be it is corrupted.<br>
2 skip step and doesnt apply changes to model if loss spike.</p>
<p>Is it good ideas? If so, what is best way to do it? May be examples or someone already make lib about it?<br>
Or should i do something else?</p>
Computing list of variable-sized tensors faster

Varg_Nord: <p>I am implementing one component in my NN, which consists of several <code>nn.functional</code> operations (parameter-free). The component takes a list of tensors as input for one batch. Each tensor of the list is variable-sized(might be empty); one tensor for one entry in the batch. Each row of the tensor is some kind of feature. I don’t want to use <code>for</code> loop to process those tensors and perform sum pooling for each tensor after this component, and concat them.  Is there any way to do such computation in parallel?</p>
Can `torch.cuda.manual_seed_all(seed)` be used on a single GPU?

ProGamerGov: <p>Can <code>torch.cuda.manual_seed_all(seed)</code> be used on a single GPU? And if so, then what is the point of <code>torch.cuda.manual_seed(seed)</code>?</p>
<p>And is it the same with <code>torch.cuda.seed_all()</code> and <code>torch.cuda.seed()</code>?</p>
<p>I was using this code, but for some reason it’s not reproducing the same results:</p>
<pre><code class="lang-auto">        torch.manual_seed(seed_val)
        torch.cuda.manual_seed(seed_val)
        torch.cuda.manual_seed_all(seed_val)
        torch.backends.cudnn.deterministic=True
</code></pre>
<p>I’m wondering if I was doing something wrong? Or if I was missing something? My code can run on a single GPU or multiple devices, so I was trying to cover all the bases.</p>

ProGamerGov: <p>I removed <code>torch.cuda.manual_seed(params.seed)</code>, and now it seems to be working correctly again.</p>
How to draw decision boundary for 2 layer Neural Network

mnauf: <p>I have a two layer Neural Network. Given the weights and biases predicted by Neural Network, how to draw the decision boundary on this dataset?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93a3fe7311f9ac32c45626d6f96673ed38a6dd61.png" alt="download" data-base62-sha1="l45xHrmSgLapRZ1way2NaZwLjpv" width="366" height="252"><br>
My Neural network is defined as follows:</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2,2)
        self.fc2 = nn.Linear(2,1)
    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
</code></pre>
<p><code>list(model.parameters())</code> returns me</p>
<pre><code class="lang-auto">[Parameter containing:
 tensor([[-0.5128, -0.3776],
         [ 0.4611,  0.4625]], requires_grad=True), Parameter containing:
 tensor([ 0.6443, -3.6282], requires_grad=True), Parameter containing:
 tensor([[-0.1985,  2.8158]], requires_grad=True), Parameter containing:
 tensor([-5.4537], requires_grad=True)]
</code></pre>
<p><a href="https://colab.research.google.com/drive/1iMsdz1N338boPoBlOH9D0-P8rTmjuKsD" rel="noopener nofollow ugc">The Google Colab notebook is attached</a>. You will find the complete code under <strong>Toy Dataset 3 predicting logits</strong>. Besides, I have drawn 1 layer neural network decision boundary as an example. Find it under <strong>Toy Dataset 2 predicting logits</strong>. For your ease, 1 layer neural network dataset and its decision boundary is attached here as well.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/9225460f1838e443216692c1a04f482d9bdcc658.png" alt="1" data-base62-sha1="kQRzhOw7TleSn4wgHwqS7VECwAM" width="393" height="248"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b2d06d462df9f36e6031efa5e18295eeb0dbad91.png" alt="2" data-base62-sha1="pvRxCg5fhyCgbqX4chOHg1HqYXn" width="380" height="264"></p>

xian_kgx: <p>A naive way I can think of is to have a grid of points, make predictions for each of them, then draw line?</p>
Resnet50, change the output dimension

Giuseppe: <p>Hy guys, i would use transfer learning with resnet50. I use the KLDivLoss for the divergence of 2 distribution. Is there a possibility to change FC and have an output dimension of [9,224,224] (the dimension of a photo with 9 channels(my second distribution)?</p>

paganpasta: <p>I am new to pytorch so will only be able to provide high level help. If you wish you can add a GlobalPooling layer just after the last layer to obtain (9,1). Or you can define the new layer in your init and bypass the FC(x) in the forward() method of your derived nn.Module class.</p>

xian_kgx: <p>There should be no problem doing this. Work on this problem from the last convolution layer and add the necessary 2d operations (conv2d, upsample2d, etc.) until you reach your desired output shape.</p>
Weird behavior observed with Autograd

Ravi_Tej_Akella: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I am trying to run the TRPO algorithm which uses a batch size of 15000. Autograd takes 27 seconds to backpropagate when the batch size is 15000 but only takes 19 seconds to sequentially compute (loop over batch_size) the individual gradients (batch size 1). What I find strange here is that parallelism is hurting the performance w.r.t. time (I agree that the latter is space inefficient).</p>

dejanbatanjac: <aside class="quote no-group" data-username="Ravi_Tej_Akella" data-post="1" data-topic="57048">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_tej_akella/48/9912_2.png" class="avatar"> Ravi_Tej_Akella:</div>
<blockquote>
<p>TRPO</p>
</blockquote>
</aside>
<p>What kind if batch norms you have implemented?</p>
Question about RNN output and hidden state!

davidlee: <p>I have a input which shape is like (sequence_length=10, batch=32,input_size=1000)</p>
<p>gru = nn.GRU(input_size=1000, hidden_size= 200)</p>
<p>input = torch.randn(10,32,1000)<br>
hidden = torch.randn(1, 32, 200)</p>
<ol>
<li>
</li>
</ol>
<p>gru_output, gru_hidden = gru(input, hidden)</p>
<ol start="2">
<li>
</li>
</ol>
<p>gru_output = []<br>
for i in range(10):<br>
input, hidden = gru(input[i], hidden)<br>
gru_output.append(input)</p>
<p><strong>Both cases (1,2) yield same size of gru_output (10,32,200)</strong></p>
<p><strong>But one is just putting input at the same time and the other is putting one sequence by one.</strong><br>
<strong>Are there any different results(output or hidden state of the GRU) between two cases??</strong></p>

vdw: <p>It should be different. In Case 1, all input sequence in the batch start with the same hidden state and are independent. In Case 2, the last hidden state of Step i becomes the initial hidden state for Step i+1. So each sentences in the batch start with a different hidden batch.</p>

dejanbatanjac: <p><a class="mention" href="/u/davidlee">@davidlee</a>, please edit your question by wrapping code into ``` block.</p>
Any information on torch.nn.functional.batch_norm/torch.batch_norm

ludwigwinkler: <p>Hi,</p>
<p>could anyone point me towards more information with regard to the torch.nn.functional.batch_norm method?</p>
<p>When I checked the Python source code I was only able to get up to the point of</p>
<pre><code class="lang-auto">return F.batch_norm(
            input, self.running_mean, self.running_var, self.weight, self.bias,
            self.training or not self.track_running_stats,
            exponential_average_factor, self.eps)
</code></pre>
<p>in the torch/nn/modules/batchnorm (in the PyTorch source code).</p>
<p>My question is over which dimension does functional.batch_norm compute the batch_statistics?<br>
Judging from the code I would suspect the 2nd dimension.<br>
The same method functional.batch_norm is used for BatchNorm1D, BatchNorm2D and BatchNorm3D.<br>
So for BatchNorm1D with [batch_size, features] data tensors the second dimension is obviously the relevant one for batch_norm and for BatchNorm2D with [batch_size, channels, height, width] the second dimension is the relevant one again.</p>
<p>Thanks in addvance.</p>
Multiclass classification loss function

Er_Hall: <p>Hi all,</p>
<p>Suppose I have a multiclass sentence classification problem where each sentence belongs only to one of the 10 classes [0, 1, …, 9]</p>
<p>Can someone tell me which loss function to use? (I will use LSTMs)</p>

Nathan_Drake: <p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss</a></p>
<p>It seems helpful according to the description of the doc.</p>
Loss.item() is inf or nan

zja_torch: <p>I defined a new loss module and used it to train my own model. However, the first batch’s loss always get inf or nan,  which leads to fail.</p>
<p>I try to print the loss item info as follows:</p>
<p>loss item: inf<br>
loss item: 7.118189334869385<br>
…<br>
loss item: 7.123733997344971</p>
<p>what  may it happpens?  I test the loss module and it works with some synthesis data. And the module is implemented with torch functions as well.</p>
<p>Could someone occcur this kind of  problem?</p>

ptrblck: <p>Could you check your input for NaN or Inf values by calling <code>torch.isnan</code> and <code>torch.isinf</code> on it?<br>
If that’s not the case, you could use the <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection">anomaly detection</a> util to debug your model.</p>

zja_torch: <p>Thanks for your suggestion and I have solved this problem.</p>
<p>It is because of the torch.exp function and sometimes it will take a large input and torch.exp will get inf value.</p>

chandra_sutrisno: <p>Can you elaborate more on this?</p>
Loss for each filter of a particular layer

ari: <p>Hi,</p>
<p>I am somewhat of a beginner to pytorch. I am implementing a paper where they have a classification CNN (input -&gt; convolutional layers -&gt; dense layers -&gt; output). However, each filter of the final convolutional layer has its own loss calculation. During back-propagation, the gradient of the final loss of the network output is summed with the gradient of the loss for each particular filter. This is then back-propagated to lower convolutional layers as per usual.</p>
<p>I have no idea how to implement something like this. I have only ever implemented pre-defined loss functions (e.g. nn.CrossEntropyLoss()) and only for the final output of the network.</p>
<p>Do you have any suggestions of how to get started?</p>

spanev: <p>Hi <a class="mention" href="/u/ari">@ari</a>,</p>
<p>You can have the model returning both the last Conv layer output and the final output</p>
<pre><code class="lang-auto">def forward(self, x):
    [...]
    c = self.last_conv(...)
    x = self.dense(x)
    return c, x
</code></pre>
<p>Next, depending on how you calculate “loss for each particular filter” you can define your loss function and sum it with the classic output loss (as <code>nn.</code> <code>CrossEntropyLoss</code>):</p>
<pre><code class="lang-auto">def special_loss(c):
    # Implements the paper algorithm
    return l
loss = nn.CrossEntropyLoss()

c, output = model(input)
final_loss =  loss(output, target) + special_loss(c, c_label)
final_loss.backward()
</code></pre>
<p>Keep in my mind that a loss function is normal PyTorch code, that returns a scalar (on which you usually call <code>backward</code>)</p>
Gpu device error while using multiple gpus with pretrained networks

poojitharamachandra: <p>File “/home/ramachap/autodl_starting_kit_stable/AutoDL_ingestion_program/ingestion.py”, line 326, in <br>
remaining_time_budget=remaining_time_budget)<br>
File “AutoDL_simple_baseline_models/resnet/model.py”, line 371, in train<br>
self.trainloop(self.criterion, self.optimizer, steps=steps_to_train)<br>
File “AutoDL_simple_baseline_models/resnet/model.py”, line 573, in trainloop<br>
log_ps = self.pytorchmodel(images)<br>
File “/home/ramachap/envs/autodl/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “AutoDL_simple_baseline_models/resnet/model.py”, line 73, in forward<br>
out = self.model(x)<br>
File “/home/ramachap/envs/autodl/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/ramachap/envs/autodl/lib/python3.5/site-packages/torchvision/models/resnet.py”, line 150, in forward<br>
x = self.conv1(x)<br>
File “/home/ramachap/envs/autodl/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/ramachap/envs/autodl/lib/python3.5/site-packages/torch/nn/modules/conv.py”, line 320, in forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: Expected tensor for argument <span class="hashtag">#1</span> ‘input’ to have the same device as tensor for argument <span class="hashtag">#2</span> ‘weight’; but device 3 does not equal 0 (while checking arguments for cudnn_convolution)</p>

devansh20la: <p>When using DataParallel for the model all the tensors you give should be on device 0. That is a pytorch format, they get copied themselves. I think this discussion might help you. <a href="https://discuss.pytorch.org/t/how-to-solve-the-problem-of-runtimeerror-all-tensors-must-be-on-devices-0/15198" class="inline-onebox">How to solve the problem of `RuntimeError: all tensors must be on devices[0]`</a></p>

poojitharamachandra: <p>thanks! that helped!</p>
LSTM Multi-class sentence classification

Er_Hall: <p>Hi all,</p>
<p>Suppose I have a multi-class text classification problem and I want to use LSTMs. I found several codes to implement this but I have been confused since there are different implementation about how to use the output of the LSTMs some of them are like the below</p>
<pre><code class="lang-auto">def forward(self, x):
        h_embedding = self.embedding(x)
        h_embedding = torch.squeeze(torch.unsqueeze(h_embedding, 0))
        
        h_lstm, _ = self.lstm(h_embedding)
        avg_pool = torch.mean(h_lstm, 1)
        max_pool, _ = torch.max(h_lstm, 1)
        conc = torch.cat(( avg_pool, max_pool), 1)
        conc = self.relu(self.linear(conc))
        conc = self.dropout(conc)
        out = self.out(conc)
        return out
</code></pre>
<p>but I cannot understand what they did.</p>
<p>and others like the below:</p>
<pre><code class="lang-auto">  def forward(self, x):
        h_embedding = self.embedding(x)        
        h_lstm, _ = self.lstm(h_embedding)
        max_pool, _ = torch.max(h_lstm, 1)        
        linear = self.relu(self.linear(max_pool))
        out = self.out(linear)
        return out
</code></pre>
<p>Why the use max_pool?</p>
<p>What’s one correct architecture (for multi-class problem) since I want as output the number of class one sentence belong? which is the right Loss function? any help</p>
<p>Regards,</p>

vdw: <p>Why LSTM instead of GRU? Why the output and not the hidden state of an RNN? Why ReLu and not Tanh, Sigmoid, etc.? Why not more linear layers before the output? …? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>There’s not one-size-fits-all solution, even for basic network models and/or for the same task (e.g., sentence classification). At the end of the day, there’s a lot of trial-and-error involved when it comes the training. Sure, it’s a good approach to start with tried-and-tested approaches. <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="nofollow noopener">For example</a>, I use the hidden state of an RNN to train a text classifier.</p>
Trace Graph changes after saving/loading

flynntax: <p>I’m doing a simple add operation where the graph looks like:</p>
<p>graph(%self : ClassType,<br>
%1 : Float(1, 3, 224, 224)):<br>
%2 : Long() = prim::Constant[value={1}]()<br>
%3 : int = prim::Constant[value=1]()<br>
%4 : Float(1, 3, 224, 224) = aten::add(%1, %2, %3)<br>
return (%4)</p>
<p>Which is correct but am seeing that after saving and loading the trace, it changes to:</p>
<p>graph(%self : ClassType,<br>
%argument_1.1 : Tensor):<br>
%3 : Tensor = prim::Constant[value={1}]()<br>
%4 : int = prim::Constant[value=1]()<br>
%5 : Tensor = aten::add(%argument_1.1, %3, %4)<br>
return (%5)</p>
<p>Is there a reason it doesn’t preserve types and names? It seems like a statically allocated tensor is being converted to a dynamic one w/o any size/shape present? I am using torch.jit.save on the trace object and torch.jit.load. Am used to operating on IntType, etc but am not sure how to use TensorType.</p>
Not loading model trained in 2 gpus to cpu for inference

dpetrini: <p>Hello,</p>
<p>I have following error while loading a 2 gpu trained model on cpu:</p>
<p>RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu</p>
<p>I tried loading with:</p>
<p>torch.load(‘my_model.pt’, ‘cpu’)<br>
torch.load(‘my_model.pt’, map_location=lambda storage, location: ‘cpu’)</p>
<p>As I’ve researched. But always getting same error.</p>
<p>It seems the gpu-1 part is loaded in cpu, but gpu-0 part is not, so the error message.</p>
<p>I welcome some help.</p>
<p>Regards<br>
Daniel</p>

devansh20la: <p>Did you save the model itself or the state? You can check the if all params are on gpu or not by using</p>
<pre><code class="lang-auto">next(model.parameters()).is_cuda
</code></pre>
<p>I think best would be to load it on the gpu move it to the cpu and save again.</p>

dpetrini: <p>Thanks for the answer, but:</p>
<pre><code class="lang-auto">next(model.parameters()).is_cuda
</code></pre>
<p>–&gt;True</p>
<p>Tried to load then move and save again:</p>
<pre><code class="lang-auto">torch.load('gpu_model.pt')
model.to('cpu')
torch.save(model, 'models_train/CPU_model_cpu.pt')
</code></pre>
<p>didn’t work either. Same error message</p>

devansh20la: <p>Don’t save the model, save model state dict. Saving the model itself can cause various problems.<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>
Requires grad and calculation speed

kkudus: <p>Hi All,</p>
<p>Whether I have requires_grad_ set to True or False for my tensor does not seem to affect the time taken for operations on that tensor to be completed. This is confusing to me. I would’ve expected operations to take longer when requires_grad_ is True, as the extra step of creating the graph must be taken. Any help on understanding why I’m seeing these results would be much appreciated.</p>
<p>Thanks in advance</p>

albanD: <p>This is actually a good news as many hours were spent optimizing the autograd engine so that it’s overhead is negligible in terms of runtime <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

kkudus: <p>Very interesting! So creating the graph is a cheap operation? It’s not until you start using the graph (backprop, etc) that the computational cost becomes significant?</p>

albanD: <p>Yes.<br>
The overhead of creating the graph is simply creating one cpp object (the Node), wrapping some Tensors (the one saved for the backward, creating the links to the previous Nodes and linking the output to the newly created Node.</p>
<p>Backprop need to traverse this whole graph to know which operations to perform (fairly expensive) and then computing the gradients themselves is roughly as expensive as the forward pass.</p>

kkudus: <p>Awesome, thanks so much for the quick and informative response!</p>
The placeholder to be memory efficient

blackbirdbarber: <p>I have found <a href="https://github.com/LiyuanLucasLiu/RAdam/blob/master/radam.py" rel="nofollow noopener">AdamW</a> by LiyuanLucasLiu.</p>
<p>If I compare the implementation with the <a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html" rel="nofollow noopener">Adam</a>, one thing is that I wonder…</p>
<p>Why AdamW implmentation used <code>p_data_fp32 = p.data.float()</code> and later on <code>p.data.copy_(p_data_fp32)</code>.</p>
<p>Is this the <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/26">placeholder trick</a> for the optim to be memory efficient?</p>
<p>Will this improve the original Adam implementation, or this is not needed?</p>
What is the easiest way to translate this tensorflow code to pytorch?

inkplay: <p>From a WGAN-GP tutorial where you create the gradient penalty.</p>
<p><code>penalty = (tf.norm(tf.gradients(D(interpolation), interpolation), axis=1) - 1) ** 2.0</code></p>
<p>Can you do this with a 1 liner too in Pytorch?</p>

kirk86: <p>Sure u can,<br>
<code>torch.norm(torch.autograd.grad(outputs=D(interpolation), inputs=interpolation), p='chose/norm/by/name/here', dim=1) - 1) ** 2.0)</code></p>
How to install PyTorch 1.2 or up in window 10?

dl_2019: <p>Hi,</p>
<p>I am using spyder (python 3.6) in window 10 to work with python programming. I need to install PyTorch 1.2 or up version in window 10. Who can show me how to install PyTorch 1.2 or up?</p>
<p>Thanks in advance!</p>

Nikronic: <p>Hi,</p>
<p>Open up <code>conda prompt</code> from anaconda installation path, then run this command:</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
</code></pre>
<p>Or open oup <code>Powershell</code> (windows native shell) then run the command.</p>
<p>By the way, the PyTorch doc is easy to follow and almost complete, a google query would be great!</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="16" height="16">
      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Tensorboard in pytorch 1.1

dayekuaipao: <p>I update my pytorch to 1.1 by conda. But I found the tensorboard module is not in the pytorch. It says ‘ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.’<br>
The 1.14 means the version of pytorch, tensorflow or just tensorboard? I use pip install tb-nightly as the docs say, but I get another error<br>
File “C:\Users\dayek.conda\envs\pytorchgpu\lib\site-packages\torch\utils\tensorboard_<em>init</em>_.py”, line 6, in <br>
from .writer import FileWriter, SummaryWriter  # noqa F401<br>
File “C:\Users\dayek.conda\envs\pytorchgpu\lib\site-packages\torch\utils\tensorboard\writer.py”, line 18, in <br>
from ._convert_np import make_np<br>
File “C:\Users\dayek.conda\envs\pytorchgpu\lib\site-packages\torch\utils\tensorboard_convert_np.py”, line 12, in <br>
from caffe2.python import workspace<br>
File “C:\Users\dayek.conda\envs\pytorchgpu\lib\site-packages\caffe2\python\workspace.py”, line 15, in <br>
from past.builtins import basestring<br>
ModuleNotFoundError: No module named ‘past’<br>
How can I solve this? Thx.</p>

ptrblck: <p>You would have to install <code>future</code> using:</p>
<pre><code class="lang-auto">pip install future
</code></pre>

ayushexel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I’m getting the same error even though I’ve got <code>tb-nightly</code> version <code>1.15</code> install along with <code>future</code><br>
The error I’m getting:<br>
<code>ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.</code></p>

ptrblck: <p>Have a look at <a href="https://github.com/pytorch/pytorch/issues/20140#issuecomment-500678249" rel="nofollow noopener">this post</a>. Could you try to check for multiple tensorboard installations?</p>

ayushexel: <p>I have checked for multiple installations but that’s not the problem as there’s only one installed package that is <code>tb-nightly version 1.15</code>. And the problem seems quite strange as while I was checking the error today from command line the statement <code>from torch.utils.tensorboard import SummaryWriter</code> ran without any errors in the interpreter but on running the code file using <code>python filename.py</code> threw the same error. I couldn’t understand what’s happening so again tried using the interpreter and now, it too throws the same error. I don’t know why this is happening or why it worked for the first time.</p>

one_way: <p>1.14 might refer to the version of tensorboard<br>
if you <strong>also</strong> have the error like “<em>No module named tensorboard</em>”, just install it via<br>
<code>conda install tensorboard</code></p>
Access intermediate tensors of a model

hashbrown: <p>Given a model and an input tensor, how do I access all the intermediate tensors? I have a network with multiple parallel (residual) branches in it. Are there any restrictions with such networks?</p>
<p>Appreciate your inputs, thanks!</p>

albanD: <p>Hi,</p>
<p>The simplest way to access such elements is usually to just access them during the foward pass of your network.</p>

hashbrown: <p><a class="mention" href="/u/alband">@albanD</a>, thanks for your response. I have a nested structure and additionally I am worried to put code in the forward function.</p>
<p>I was hoping if I can use some of the torch.jit utilities for this and if there is a good example that I can use. If I am able to construct a network graph then it would help me with handling different residual branches as well.</p>
<p>Appreciate some pointers on this one. Thanks!</p>

albanD: <p>Why are you worried about putting code in the forward function?<br>
One of the main advantage of pytorch is actually that the forward is called at every iteration and you can do arbitrary code in there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to interpret targets in VOCSegmentation 2012

ihaba: <p>Hi,</p>
<p>I’m trying my hand at sematic segmentation using torchvision  and the built-in tools it provides. What I don’t understand is How I’m supposed to use the targets I get from the VOCSegmentation 2012 dataset. Using the snippet below</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as tf
import torchvision.datasets as dsets

train_set = dsets.VOCSegmentation(
    root='./data'
    ,year='2012'
    ,download=True
    ,image_set= 'train'
    ,transform = tf.Compose([tf.RandomCrop(256), tf.ToTensor()])
    ,target_transform = tf.Compose([tf.RandomCrop(256), tf.ToTensor()])
)

train_loader = torch.utils.data.DataLoader(
    train_set, batch_size=4, shuffle=False, num_workers=1
)

train_iter = iter(train_loader)
image, target = next(train_iter)

print(target[1][0])
</code></pre>
<p>I get the following output:</p>
<pre><code class="lang-auto">
tensor([[0.0784, 0.0784, 0.0784, ..., 0.0000, 0.0000, 0.0000],
    [0.0784, 0.0784, 0.0784, ..., 0.0000, 0.0000, 0.0000],
    [0.0784, 0.0784, 0.0784, ..., 0.0000, 0.0000, 0.0000],
    ...,
    [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
    [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
    [0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]])
</code></pre>
<p>I would have expected the output from the target to be integers between 0 and 20 (As there are 20 classes in the data set). Why is there floats there? Is this normalized in some way I don’t understand or have I misunderstood how the target are supposed to look?</p>
Does torch.manual_seed() hash inputs?

btimar: <p>Hi –<br>
The openAI gym environments use a seed() method that hashes its input; the relevant code snippet:</p>
<pre><code class="lang-auto">    seed = create_seed(seed)

    rng = np.random.RandomState()
    rng.seed(_int_list_from_bigint(hash_seed(seed)))
</code></pre>
<p>(from <a href="https://github.com/openai/gym/blob/master/gym/utils/seeding.py" rel="nofollow noopener">here</a>).</p>
<p>Does torch.manual_seed() do the same? This isn’t mentioned in the docs.</p>

albanD: <p>Hi,</p>
<p>We don’t do any hashing of the seed.<br>
But the rng engine might depending on which one is used (cuda or cpu).</p>
Are these forward functions equivalent?

Beinan_Wang: <ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">def forward(self, x):
    if self.dowsample:
        identity = self.downsample(x)
    else:
        identity = x
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">    if need_downsampling == True:
        self.downsample = do the downsampling stuff
    else:
        self.downsample = lambda x : x

def forward(self, x):
    identity = self.downsample(x)
</code></pre>
<p>Is one of the above more computation or memory efficient than the other?</p>

albanD: <p>Hi,</p>
<p>Yes they are.<br>
Note that in both case, you can change <code>self.downsample</code> in between forwards if needed.</p>

Beinan_Wang: <p>Thanks for your reply!</p>