akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Effective way of weighted sampling from different folders

username: Dana_Kim
cooked: <p>Hi,</p>
<p>I’m trying to train a model with custom datasets.</p>
<p>I need to select a folder to be sampled from according to probability.</p>
<p>For example, the images should be sampled from folder A with 70% of probability and 30% from folder B.</p>
<p>This is the code I wrote but I think it’s not effective:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, img_dir, img_dir2, ratio):
        self.ratio = ratio
        self.img_list = sorted(glob(os.path.join(img_dir, '**/Image_*'), recursive=True))
        self.img_list2 = sorted(glob(os.path.join(img_dir2, '**/Image_*'), recursive=True))

    def __getitem__(self, index):
        if np.random.random() &lt; self.ratio:
            index = np.random.randint(len(self.img_list))
            img = Image.open(self.img_list[index])

        else:
            index = np.random.randint(len(self.img_list2))
            img = Image.open(self.img_list2[index])

        img = self.preprocess(img)
        return torch.from_numpy(img).type(torch.FloatTensor)

    def __len__(self):
        return len(self.img_list) + len(self.img_list2)
</code></pre>
<p>Is there more effective way to perform this task?</p>
<p>I guess I need to use sampler but can’t figure out how to use it.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

username: ptrblck
cooked: <p>You could use a <code>WeigthedRandomSampler</code>, in case you could calculate and assign the weight for each sample corresponding to the folders. This would have the advantage to compute the weights only once and then sample from the two folders.</p>
Extract intermediate representation of MiDaS neural network?

username: Apogentus
cooked: <p>Pytorch documentation provides a <a href="https://pytorch.org/hub/intelisl_midas_v2/" rel="noopener nofollow ugc">concise way</a> to apply MiDaS monocular depth estimation network for depth extraction. But how should I modify their code to extract network representation at some intermediate layer? I know that I could download the model from github and modify <code>forward</code> function to return what I want, but I am interested in the simplest solution, leaving outer code as is.<br>
I’m aware of subclassing the model class and writing my own forward function, like <a href="https://discuss.pytorch.org/t/extract-feature-maps-from-intermediate-layers-without-modifying-forward/1390">here</a>, but I don’t know how to access the class in the code. The model instance is created straight away with <code>midas = torch.hub.load("intel-isl/MiDaS", model_type)</code>. Maybe an example of using  a forward hook will be easier.</p>

username: ptrblck
cooked: <p>You could indeed use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
Zero gradients when trying to implement FGSM

username: ianxul
cooked: <p>Hello, I’m new to PyTorch and I’ve been trying to implement the Fast Gradient Sign Method to test a models’ robustness to adversarial perturbations. However when I try taking the gradient of the loss function with respect to the input I get a zero tensor, so in this case applying FGSM wouldn’t work no matter how big of an epsilon I use.<br>
The model I’m using is this one:</p>
<pre><code class="lang-auto">class L2Net(nn.Module):
    def __init__(self, hidden,  beta, insize = 784, outsize = 10):
        super(L2Net, self).__init__()
        self.fc1 = nn.Linear(insize, hidden, bias = False)
        self.fc2 = nn.Linear(hidden, outsize, bias = False)
        self.beta = beta
    def forward(self, x):
        x = torch.flatten(x, 1)
        x = F.relu(self.fc1(x)) 
        x = torch.tanh(self.beta * self.fc2(x)) 
        x = F.log_softmax(x, dim=-1)
        return x
</code></pre>
<p>And I’ve tried using mse, nll and crossentropy as losses.<br>
I am working with the MNIST dataset and using 2,000 units in the hidden layer.<br>
Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>I’m unsure how to reproduce the issue, as I’m getting valid gradients in the input:</p>
<pre><code class="lang-python">model = L2Net(10, 1)
x = torch.randn(1, 784, requires_grad=True)
out = model(x)
out.mean().backward()
print(x.grad)
&gt; tensor([[-9.4862e-04,  3.0799e-04,  6.3704e-04, -1.2979e-05, -5.3003e-04,
            9.0936e-06, -7.5076e-04, -7.7078e-04, -5.5273e-04,  8.0073e-04,
            4.7446e-04,  9.0180e-04, -4.5098e-04,  1.5162e-04,  7.3597e-04,
            ...
</code></pre>

username: ianxul
cooked: <p>Hello. It is only after training the network that I get the problem. I was able to get valid gradients by deleting the tanh application. It seems the problem was that after applying tanh the output would be pure 1s and -1s because of rounding up. So small perturbations of the input did not have any effect on the numeric value of the loss…<br>
Could that be an explanation for the gradient appearing to be zero?</p>

username: ptrblck
cooked: <p>Yes, that could explain the small (or zero) gradients, as the derivative of <code>tanh</code> in the “saturation” range would be small.</p>
Issues predicting variable length input with pre-trained model that was trained with padded values

username: Raj_Roy
cooked: <p>Hi,</p>
<p>I am a newbie to pytroch, I trained a model that takes in l<em>l</em>5 input and outputs l*l , I used a resnet to train it and during training I zero-padded the input to have 200 x 200x5 and labels as 200x200, and evaluated the method with a custom function.<br>
But if I want to make another program that load its weight and predict , How should I do it?<br>
I tried by not padding the input data and the label, also made the output shape to be same as length of input and evaluated it, the same way and on the same validation data as before. I found is that the accuracy dropped to half.</p>
<p>Can anyone explain what is the issue and how to predict the variable lengths of data using a pre-trained model. The data I am predicting might have any value even greater than 200</p>
Tensorboard works for 3D data?

username: banikr
cooked: <p>I was trying trying to run tensorboard module in Pytorch with 3D image data with the following code:</p>
<pre><code class="lang-auto">class GetTB(object):
    def __init__(self, img, lbl, model):
        self.image = img
        self.label = lbl
        self.model = model

    def WorkTB(self):
        tb = SummaryWriter()
        grid = torchvision.utils.make_grid(self.image)
        tb.add_image("images", grid)
        tb.add_graph(model, self.image)
        tb.close()

A = GetTB(images, labels, model).WorkTB()
</code></pre>
<p>A bit about the images and labels is: images have <code>B,C,H,W,D</code>  dimensions.  which is C=1 for images and C=5/6 for the one-hot encoded label.<br>
Now in my search the <code>make_grid()</code> documentation mentions images to be:</p>
<pre><code>Args:
    tensor (Tensor or list): 4D mini-batch Tensor of shape (B x C x H x W)
        or a list of images all of the same size.
</code></pre>
<p>Does it indicate <code>tensorboard</code> only works with RGB image data?<br>
I also couldn’t find any sort of implementation tutorial/article for specific 3D data. If anyone updates would be highly appreciated.<br>
Thanks</p>

username: eqy
cooked: <p>I’m not sure there is any out-of-the-box support for depth images in tensorboard. However, if there is some visualization function you have for depth + color images, you can simply run that first and pass in the results encoded in RGB to tensorboard.</p>

username: banikr
cooked: <p>Thanks, I guess apart from 5D tensors or 3D image/label visualization, I suppose other performance metrics/analyses should work in tensorboard.</p>
Is there any website that I can give people the selecting jobs (real/fake)?

username: davidlee
cooked: <p>Hi all,</p>
<p>I want to test quality of the generated images based on human recognition.<br>
Is there any website or online platform that I can give people the jobs of choosing which one is real or fake image?</p>
<p>Please let me know !</p>

username: ptrblck
cooked: <p>You could check crowdsourcing platforms such as Amazon’s Mechanical Turk.</p>

username: davidlee
cooked: <p>That’s what I needed!! Thanks a lot:)</p>
How to unpack mat file frame in single image

username: Fabio_91
cooked: <p>Hi,</p>
<p>I have a matlab file where inside there are two matrices representing 60 frame videos.</p>
<p>The first matrix (“data_m”) is composed of 1200 x 120 x 60, the second matrix ("ground_truth) consists of k x 60 x 4, the two matrices are linked, that is the i-th frame (data_m) corresponds to the i-th column (ground_truth). I need to be able to unpack these matrices so that I can see them as 60 single images.<br>
This is my dataloader:</p>
<pre><code class="lang-auto"> def __init__(self, snapshot_dir,transform):

        self.snapshot_dir = snapshot_dir
        self.transform = transform

 def __len__(self):

        return len([img for img in os.listdir(self.snapshot_dir) if img.endswith('.mat')]) * 60

 def __getitem__(self, idx):

        # image path
        snapshot = os.path.join(self.snapshot_dir, "snapshot_n_" + str(idx+1) + ".mat")

        matrix_TBD = scipy.io.loadmat(snapshot)
        data_matrix = matrix_TBD.get('data_matrix')
        ground_truth = matrix_TBD.get('ground_truth')
        frame_size = data_matrix.shape[2]

        for i in range(frame_size):

            img = data_matrix[:, :, i]
            target = ground_truth[:, i, :]

        sample = {'image': img,
                  'target': target}

        if self.transform:
            image, target = sample['image'], sample['target']
            return {'image': torch.tensor(image),
                    'target': torch.tensor(target)}

        return sample

</code></pre>
<p>I debug to see the result</p>
<pre><code class="lang-auto">dataset = TBD_Dataset(snapshot_dir=snapshot_dir, transform=True)

    # DEBUG: show all samples in GMM Dataset
    if debug_dataset:

        for i in range(len(dataset)):
            sample = dataset[i]
            print(i, ",",
                  "Image shape: ", sample['image'].shape, ",",
                  "target shape: ", sample['target'].shape)
</code></pre>
<p>I know this way I will only get the latest img and target but I can’t find a way to get the 60 images, can anyone help me?</p>

username: ptrblck
cooked: <p>In your current code you are iterating the frames and use only the last one.<br>
To select each frame you could load the data in the <code>__init__</code> method (move the <code>data_matrix = ...</code> code there) and select the frame in <code>__getitem__</code> by using the passed <code>idx</code>:</p>
<pre><code class="lang-python">img = data_matrix[:, :, idx]
</code></pre>
<p>(same for the <code>target</code>).</p>
Image to vectors operation -- how to calculate a receptive fields tensor

username: nullgeppetto
cooked: <p>Hi, I’m interested in experimenting with a variant of the standard 2D convolution, for which it would be very convenient to <strong>express the convolution operation as a tensor multiplication</strong> between the weights tensor and a tensor that contains all the receptive fields as vectors.</p>
<p>That is, I’m looking for an <em>image-to-columns/vectors</em> method that, given the parameters of a 2D convolution (i.e., <code>kernel_size</code>, <code>stride</code>, and <code>padding</code>), it “breaks” the input image (or, for that matter, any appropriate input feature map) into vectors each of which corresponds to a receptive field (according to the aforementioned parameters). By doing so, the convolution operation can be written as a tensor multiplication (or at least I think so).</p>

username: ptrblck
cooked: <p>You could use <code>nn.Unfold</code> to apply the <code>im2col</code> operation and the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html">docs</a> for this method also give you an example how a custom convolution can be applied using this module.</p>

username: nullgeppetto
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, many thanks for your response. Based on your suggestion and the discussion found <a href="https://github.com/pytorch/pytorch/issues/47990" rel="noopener nofollow ugc">here</a>, I figured it out. For the sake of completeness, I give a simple example below:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

# Input tensor of shape (bs, C_in, H, W)
bs = 128
C_in = 3
H = 32
W = 32

X = torch.randn(bs, C, H, W)
X = X.double()

# 2D convolution 
C_out = 16
conv = nn.Conv2d(in_channels=C_in, 
                 out_channels=C_out, 
                 kernel_size=3, 
                 stride=1, 
                 padding=0, 
                 dilation=1, 
                 bias=True)

# 2D convolution operation
conv.weight.data = conv.weight.data.double()
Y = conv(X)

# ---

# Define nn.Unfold operation
unfold = nn.Unfold(kernel_size=conv.kernel_size, 
                   dilation=conv.dilation, 
                   padding=conv.padding, 
                   stride=conv.stride)

# Define nn.Fold operation
h_in, w_in = X.shape[2], X.shape[3]
h_out = (h_in + 2 * conv.padding[0] - (conv.kernel_size[0] - 1) - 1) / conv.stride[0] + 1
w_out = (w_in + 2 * conv.padding[1] - (conv.kernel_size[1] - 1) - 1) / conv.stride[1] + 1
h_out, w_out = int(h_out), int(w_out)
fold = nn.Fold(output_size=(h_out, w_out), kernel_size=1)

W = conv.weight.double()
Y_ = (unfold(X).transpose(1, 2).matmul(W.view(W.size(0), -1).t()) + conv.bias).transpose(1, 2)
Y__ = fold(Y_)
print("torch.allclose(Y, Y__): {}".format(torch.allclose(Y, Y__)))

</code></pre>
Fixing Serialization error: Using numpy tensors for image dataset instead of pytorch tensors?

username: Jennifer_Davis
cooked: <p>I found a problem with trying to run a high volume of images on a pipeline that includes pytorch and is supposed to run in a distributed way.  The dataset is over a million samples. The pipeline is having trouble serializing and deserializing very large objects. The training dataset (trn_ds in the code) is very large at 2.5 million images. I am using a batch process for processing the images from the data loader, but the data loader does rely upon the training dataset which holds all the image tensors.  One thing to try I thought of is to change the .pt (pytorch tensors) to numpy tensors.  Numpy tensors are apparently easier to serialize and deserialize.  But I’m having trouble finding documentation on how to create numpy image tensors in a dataset and then feed them to a data loader.  Is this possible?</p>

username: eqy
cooked: <p>You can always convert to/from numpy tensors just before you need torch tensors with the utility functions e.g.,<br>
<a href="https://pytorch.org/docs/stable/generated/torch.from_numpy.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.from_numpy.html</a><br>
and <code>yourtensor.numpy()</code></p>

username: Jennifer_Davis
cooked: <p>Oh great!  One other question – if I train several models on smaller batches of data, can I combine all the small-batch trained models into one final model that is for all intents and purposes trained on the entire dataset?</p>

username: eqy
cooked: <p>You might be able to combine the smaller models into an ensemble, but in general I don’t think there is an easy way to get the benefit of training a single model on the entire dataset (especially with any guarantees on the final model’s performance).</p>

username: Jennifer_Davis
cooked: <p>Thank you.  I agree its a poor solution.  Anything else you can suggest for training the 2.5 million image set?  Thanks again for your advice.</p>

username: eqy
cooked: <p>In practice I don’t think 2.5 million images is unreasonable, as many models are trained on ImageNet (1.2 million images) these days with larger datasets floating around. However, by default ImageNet is usually distributed as an archive that is deserialized as many images on a filesystem so there is no need to keep the entire dataset in memory at once. Is there any way you could do something similar with your dataset (e.g., convert the images to file on disk) ahead of time?</p>

username: Jennifer_Davis
cooked: <p>That’s a good idea.  Let me see if I can put together a cache method, perhaps in the custom data loader code.</p>

username: Jennifer_Davis
cooked: <p>We’re actually going to try DistributedDataParallel and see how that works for us and the caching solution you recommended. Thanks for the input.  Its good to know that 2.5 million images is reasonable.</p>
Custom kernel for Convolution

username: Hmrishav_Bandyopadhy
cooked: <p>I want to perform convolutions with custom kernels, but in a manner different than traditional convolutions. While convolution in pytorch occurs in general by each kernel slid and multiplied with each channel and then the sum passed over, I need to perform convolutions where input and output channels are same and each kernel (same as the number of channels) is multiplied with only a single channel from the input. And concatenated with the rest of the outputs from the other kernels. It looks like this now:</p>
<pre><code class="lang-auto">in_channels=out_channels
filters = torch.zeros((batch_size,in_channels,out_channels,f_size,f_size)
for batch in range(batch_size):
     for chan in range(in_channels):
            filter_use=filters[batch,chan].unsqueeze(0)
            out_conv=F.conv2d(x[batch, chan].unsqueeze(0).unsqueeze(0), filter_use)
            
            ### Perform more operations
### Concatenate the results
</code></pre>
<p>While this gets the work done, it takes a huge amount of time as you can guess from the nested loop structure. What would be a faster alternative to this?</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Boosting this ^^. Can anyone help me please <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thanks!</p>

username: eqy
cooked: <p>This sounds like a grouped convolution with the number of groups == number of input channels <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" class="inline-onebox" rel="noopener nofollow ugc">Conv2d — PyTorch 1.9.0 documentation</a>, but I’m not sure because the current code example doesn’t run. Could you post a runnable code snippet to help confirm? The indexing of the filters by batch also seems very suspicious, as it implies that different elements of the batch are treated differently…</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Thanks! It is indeed grouped convolutions. One more question–Grouped convolutions “assigns” kernels to channels in the order of their indices, right?</p>

username: eqy
cooked: <p>This sounds like it should be the case. As always, you can test this theory with synthetic data/weights to be sure.</p>
How does the shuffle parameter in DataLoader works? Is it related to "cross validation"?"

username: eduardo
cooked: <p>Only “training data” gets shuffled before every epoch and the validation data remains the same for each epoch??..</p>
<p>or it gets shuffled all together with the “validation data”?</p>
<p>And the other question is… if shuffle=True is not cross validation, how could I make cross validation (dividing data in folds and changing the validation fold) instead of using the regular method?</p>
<p>Thanks in advance!</p>

username: eqy
cooked: <p>The shuffling is only so that the batches that are seen by the model are different during every epoch. If you want to do cross-validation, you want to separate the data into a training and validation set before building the dataloaders.</p>
Custom loss using output of other model

username: sakh251
cooked: <p>I have a loss function for a Model (model 1) which has a term calculating based on output of other model (model 2). So model 2 is not trained. I want to disable gard to get the output of model 2 faster. but I want to enable it for model 1.<br>
input = …<br>
output1 = model1(input)</p>
<p><span class="hashtag">#disable</span> gard<br>
output2  = model2(input)</p>
<p><span class="hashtag">#enable</span> again<br>
loss = criterion(output1, target) + function(output1,output2)</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>You could disable the gradient calculation via <code>with torch.no_grad()</code> as seen here:</p>
<pre><code class="lang-python">model1 = nn.Linear(1, 1)
model2 = nn.Linear(1, 1)

input = torch.randn(1, 1)
target = torch.randn(1, 1)
criterion = nn.MSELoss()
function = criterion

output1 = model1(input)

with torch.no_grad():
    output2 = model2(input)


loss = criterion(output1, target) + function(output1,output2)
loss.backward()

for name, param in model1.named_parameters():
    print(name, param.grad)
&gt; weight tensor([[-0.8400]])
  bias tensor([4.2485])

for name, param in model2.named_parameters():
    print(name, param.grad)
&gt; weight None
  bias None
</code></pre>

username: sakh251
cooked: <p>Thank you for reply,</p>
<p>Now I wonder, The model 2 should be in eval() mode or not. I do not want to train this model. I am trying to distill knowledge from it to model 1. So model 1 should produce out same as model 2 for same batches.</p>
<p>Bests</p>

username: ptrblck
cooked: <p><code>model.eval()</code> would disable e.g. dropout layers and would use the running stats of batchnorm layers.<br>
Depending on your use case, you might need to call it.</p>
Compatability issues between CUDA and NVIDIA Drivers

username: asamra18
cooked: <p>Hi Everyone, I’m trying to train an RL agent on my GPU to speed up processing time, but for some reason pytorch isn’t finding any devices. I’ve tried installing 2 versions of pytorch from the install site for cuda versions 11.1 and 10.2 and both don’t work. After running <strong>nvidia-smi</strong> i see the Driver version is 465.27 and the CUDA version is 11.3, and I’m working with GeForce RTX 2080 Ti on Ubuntu in a Conda environment. When I check the torch version by printing its value, I see its 11.1 and that no devices are available.</p>
<p>I know that this should be working because when my PhD colleague runs our code from his machine, the GPU training is working properly. After updating drivers, and using different installations from the Pytorch website, I’m now completely stuck as to how I can solve this.</p>

username: ptrblck
cooked: <p>I guess your NVIDIA driver installation might not have been successful, as PyTorch cannot detect any GPUs or alternatively, you are installing the CPU-only binaries.<br>
We’ve seen these issues in the past when users forgot to restart their machine after a driver update, so maybe this could also be the case for you?</p>
How do I load a manually created model to GPU in PyTorch?

username: Z_Huang
cooked: <p>Hi guys,</p>
<p>I have created a neural network model with all the associated functions from scratch (basic Python code) into a class without using nn.Module. If I can still port this manually created model into GPU?</p>
<p>I ported the train/test data into GPU, but don’t know how to port the model to GPU. It seems that I have to use nn.Module to create models. Is it true? Or, I can port this manually created model into GPU. Any idea?</p>

username: ptrblck
cooked: <p>If you are using <code>nn.Module</code>s you’ll be able to call <code>.to('cuda')</code> on the module(s) directly, which will then automatically move all registered parameters and buffers recursively for all submodules to the GPU.<br>
However, this is not strictly necessary and you could also move the parameters (trainable tensors) as well as the input data manually to the GPU.</p>

username: Z_Huang
cooked: <p>How may I make that happen without using nn.Modules? I tried it many times by myself and can’t figure it out.</p>

username: ptrblck
cooked: <p>It depends on the actual use case, but instead of e.g. using an <code>nn.Linear</code> module, you would manually create the parameters and use the functional API or the corresponding <code>torch</code> methods:</p>
<pre><code class="lang-python">device = 'cuda'
x = torch.randn(1, 10).to(device)

# module
lin = nn.Linear(10, 10, bias=False).to(device)
out = lin(x)

# functional API
w = nn.Parameter(torch.randn(10, 10, device=device))
out = F.linear(x, w)

# torch methods
out = torch.matmul(x, w.T)
</code></pre>

username: Z_Huang
cooked: <p>Does that mean, I have to use “nn” to put a model into GPU?</p>

username: ptrblck
cooked: <p>It depends what you define as a “model” now.<br>
If you want to create <code>nn.Module</code>s, then yes you would have to use the <code>nn</code> namespace, as they are defined there. The same applies for <code>nn.Parameter</code>s, but you could also create a tensor and set <code>requires_grad=True</code>. In this case, you wouldn’t use the <code>nn</code> namespace for these operations.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x48 and 3072x100)

username: alee023
cooked: <p>I’m trying to input a 5D tensor with shape ( 1, 8, 32, 32, 32 ) to a VAE I wrote:</p>
<pre><code class="lang-auto">self.encoder = nn.Sequential(
        nn.Conv3d( 8, 16, 4, 2, 1 ), # 32 -&gt; 16
        nn.BatchNorm3d( 16 ), 
        nn.LeakyReLU( 0.2 ),
        
        nn.Conv3d( 16, 32, 4, 2, 1 ), # 16 -&gt; 8
        nn.BatchNorm3d( 32 ),
        nn.LeakyReLU( 0.2 ),
        
        nn.Conv3d( 32, 48, 4, 2, 1 ), # 16 -&gt; 4
        nn.BatchNorm3d( 48 ),
        nn.LeakyReLU( 0.2 ), 
    )
    
    self.fc_mu = nn.Linear( 3072, 100 ) # 48*4*4*4 = 3072
    self.fc_logvar = nn.Linear( 3072, 100 )
    
    self.decoder = nn.Sequential(
        nn.Linear( 100, 3072 ),
        nn.Unflatten( 1, ( 48, 4, 4 )),
        nn.ConvTranspose3d( 48, 32, 4, 2, 1 ), # 4 -&gt; 8
        nn.BatchNorm3d( 32 ),
        nn.Tanh(),
        
        nn.ConvTranspose3d( 32, 16, 4, 2, 1 ), # 8 -&gt; 16
        nn.BatchNorm3d( 16 ),
        nn.Tanh(),
        
        nn.ConvTranspose3d( 16, 8, 4, 2, 1 ), # 16 -&gt; 32
        nn.BatchNorm3d( 8 ),
        nn.Tanh(), 
    )

def encode( self, x ) :
    x = self.encoder( x )
    x = x.view( -1, x.size( 1 ))
    
    mu = self.fc_mu( x )
    logvar = self.fc_logvar( x )
    
    return self.reparametrize( mu, logvar ), mu, logvar 
    
def decode( self, x ):
    return self.decoder( x )
    
def forward( self, data ):
    z, mu, logvar = self.encode( data )
    return self.decode( z ), mu, logvar 
</code></pre>
<p>The error I’m getting is: <code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x48 and 3072x100)</code>. I thought I had calculated the output dimensions from each layer correctly, but I must have made a mistake, but I’m not sure where. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Based on the error message I would assume that the error is either raised in <code>self.fc_mu</code> or <code>self.fc_logvar</code>, since both are using the mentioned 3072 <code>in_features</code>.<br>
You could add debug print statements to the <code>forward</code> method of your model and check the shape of the input activations to these layers, which should show the other mentioned shape.</p>

username: alee023
cooked: <p>I’ve edited the post to include the forward() code. I ended up fixing it (I think) by changing<br>
<code>x = x.view( -1, x.size( 1 ))</code> to <code>x = x.view(x.size( 0 ), -1)</code>. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Temporal Fusion Transformer Question

username: Abhay_Sri
cooked: <p>Hello,</p>
<p>So basically I have a bunch of pH data all from around the same year stretching anywhere from 1-5+ years. I tried to follow the tutorial for the temporal fusion transformer but it does not seem to work well for multiple groups of data. Could you please let me know what I am doing wrong here? The predicted data seems very discrete with random jumps instead of a continuous curve.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1pEuZlt3gBK7ubcilyLnANF0fhMDJq9IK?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210713-060127-RC00_384416053" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1pEuZlt3gBK7ubcilyLnANF0fhMDJq9IK?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1pEuZlt3gBK7ubcilyLnANF0fhMDJq9IK?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to take three tensor elements one by one

username: PatricYan
cooked: <p>how to take three tensor elements one by one to make up new data, for example<br>
I have tensor A, B, C, how to get [(a1,b1,c1,), (a2,b2,c2,)]</p>

username: safakkbilici
cooked: <p>Maybe something like this script would be helpful:</p>
<pre><code class="lang-auto">a = torch.randn(2)
# tensor([ 0.1869, -0.9645])
b = torch.randn(2)
# tensor([-0.4450,  1.6712])
c = torch.randn(2)
# tensor([0.1208, 2.3409])

new_tensor = []
for i, j, k in zip(a,b,c):
   new_tensor.append([i.item(),j.item(),k.item()])

new_tensor = torch.tensor(new_tensor)
# tensor([[ 0.1869, -0.4450,  0.1208], [-0.9645,  1.6712,  2.3409]])

</code></pre>

username: PatricYan
cooked: <p>Thanks. I resolve it</p>
Trying to backward through the graph a second time

username: AfonsoSalgadoSousa
cooked: <p>Hi. I have an adversarial training process with a few models and two optimizers. The two optimizers update different parameters, and yet I get this error:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.</p>
</blockquote>
<p>The corresponding code is below:</p>
<pre><code class="lang-auto">    optimizer_G = torch.optim.Adam(
    chain(img_encoder.parameters(), pts_encoder.parameters(),
          img_decoder.parameters(), pts_decoder.parameters()),
          weight_decay=decay, lr=lr, betas=(b1, b2))
    optimizer_D = torch.optim.Adam(
          discriminator.parameters(), lr=lr, betas=(b1, b2), weight_decay=decay)

    ...

    g_loss = alpha * \
        adversarial_loss(discriminator(pts_feats), valid) + \
        (1 - alpha) * reconstruction_loss

    optimizer_G.zero_grad()
    g_loss.backward()
    optimizer_G.step()

    real_loss = adversarial_loss(discriminator(pts_feats), valid)
    fake_loss = adversarial_loss(discriminator(img_feats), fake)
    d_loss = (real_loss + fake_loss) / 2

    optimizer_D.zero_grad()
    d_loss.backward()
    optimizer_D.step()
</code></pre>
<p>Why do I need to retain the graph when they should be disjoint since each optimizer updates different weights?</p>

username: tom
cooked: <p>Use generated.detach() when feeding generator output inro the discrimibator when you want backward to not go into the generator.</p>
Cuda OOM during loss backward

username: mk6
cooked: <p>Hi,<br>
I’m facing an issue of running out of cuda memory during backward propagation.</p>
<p>I need to train a small model using the following loss which requires significant amount of memory: e1 and e2 are of dimension (32000, 256). So there should be at least <code>32000*32000*32 * 3</code> bits = 12G memory. However the only GPU that I have access to has ~11G memory. (I have multiple such GPUs, but I didn’t figure out any method solving the problem by exploiting them.)</p>
<pre><code class="lang-auto">def loss(e1, e2):
    ee1 = e1@e1.T
    ee2 = e2@e2.T
    ee1 -= ee2
    ee1 = torch.pow(ee1, 2)
    return torch.sqrt(torch.sum(ee1))
</code></pre>
<p>I tried various ways calculating the loss or some parts of it in CPU memory, but the backward pass will still cause CUDA OOM. It seems still requiring 12GB during backward pass since the computation graph is the same no matter how do I divide and conquer the loss.<br>
So I’m wondering is using this loss training models even feasible with my current devices?</p>
<p>Thanks</p>
Torch::jit::forward Thread 1: EXC_BAD_ACCESS?

username: yichong96
cooked: <p>I am getting a <code>Thread 1: EXC_BAD_ACCESS</code> error in the torch::jit::Module::forward call here</p>
<pre><code class="lang-auto">  IValue forward(std::vector&lt;IValue&gt; inputs) {
    // get_method("forward") returns but calling (std::move(inputs)) after 
    // results in Thread 1: EXC_BAD_ACCESS
    return get_method("forward")(std::move(inputs)); 
  }
</code></pre>
<p>The code before calling the function is from here which follows closely to that of the <a href="https://github.com/pytorch/ios-demo-app/tree/master/ObjectDetection/ObjectDetection/Inference" rel="noopener nofollow ugc">D2Go’s github object detection repo</a></p>
<pre><code class="lang-auto">        at::Tensor tensor = torch::from_blob(imageBuffer, { 3, input_width, input_height }, at::kFloat);
        torch::autograd::AutoGradMode guard(false);
        at::AutoNonVariableTypeMode non_var_type_mode(true);
        
        std::vector&lt;torch::Tensor&gt; v;
        v.push_back(tensor);
        std::vector&lt;at::IValue&gt; b = {at::TensorList(v)};
        auto outputTuple = _impl.forward(b).toTuple(); // error 
</code></pre>
<p>Might anyone have any intuition why this might happen ? For context, I am calling this function in swift UI on the main thread.</p>

username: ptrblck
cooked: <p>Based on <a href="https://developer.apple.com/forums/thread/123958">this post</a> it seems you are hitting a memory violation. Could you <code>.clone()</code> <code>tensor</code> before passing it to the function?</p>

username: yichong96
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the reply. I noticed that this exception occurs when the model in python is not quantised. The exception I got happened for mobilenetv3 faster rcnn and retinanet. However, when I built a model with quantisation layers then I do not get an exception.</p>
<p>I will try your suggestion and see how it goes.</p>
<p><strong>Update</strong><br>
I did try to clone the tensor but still met with memory exception in another thread. Maybe IOS does not allow running of large models, since the <code>detect</code> function manages to run without errors for quantised models.</p>

username: yichong96
cooked: <p>I realised the problem was I think because the pointer to <code>imageBuffer</code> was somehow deallocated by the time it tried to execute the model inference. I was passing a pointer to <code>imageBuffer</code> from swift to C++. Somehow I am not sure why <code>imageBuffer</code> seems to be deallocated when I called the inference code from swift. I fixed it by defining a <span class="mention">@State</span> property for the imageBuffer variable in swift</p>