akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Reducing the channel numbers of a 5D tensor using convolutions

username: Tm4
cooked: <p>For reducing the number of feature maps in a 4D tensor, it is possible to use Conv2d while keeping the height and width of that tensor as the same as before. To be clearer, For example for the following 4D tensor [batch_size, feature_maps, height,weight] I use the following approach for reducing the number of feature maps but the height and size will be the same as before:</p>
<pre><code class="lang-auto">self.channel_reduction = nn.Conv2d(1024, 512, kernel_size=1, stride=1)
</code></pre>
<p>But I have the following 5D tensor [batch_size, feature_maps, num_frames, height, width] (e.g. [16, 1024, 16, 56, 56]) and I want to reduce the number of feature maps from 1024 to 512 while keeping the height and width size as the same as before (e.g. [16, 1024, 16, 56, 56]). How can I reduce the number of feature maps?</p>

username: ptrblck
cooked: <p>You could use <code>nn.Conv3d</code> for your 5D tensor.</p>

username: Tm4
cooked: <p>Thanks, How I should select kernel_size, stride and padding?</p>

username: ptrblck
cooked: <p>You can use the same setup with a kernel size of 1 and no padding:</p>
<pre><code class="lang-python">channel_reduction = nn.Conv3d(1024, 512, kernel_size=1, stride=1)
x = torch.randn(16, 1024, 16, 56, 56)
out = channel_reduction(x)
print(out.shape)
# torch.Size([16, 512, 16, 56, 56])
</code></pre>
Retrieve what compute capabilities (sm versions) that a compiled libtorch version supports?

username: davidw82
cooked: <p>Hi,<br>
I’m wondering how I’d be able to retrieve a list of supported compute capabilities of a certain compiled version of libtorch? I basically need to investigate the fat binary produced, to see that it contains the correct support for different compute capabilities/GPUs (including the PTX version for future compatibility). How can I do this? The version of libtorch has not been compiled with python support, it’s only compiled for the static C++ libtorch linking, so it needs to work from C++ please. Or if I can investigate the static library “libtorch.a” or “libtorch_cuda.a” in some way?</p>
<p>Cheers,<br>
David</p>

username: davidw82
cooked: <p>I’ll answer my own question here because I found the answer. You can use the binary cuobjdump from the CUDA Toolkit, documented here: <a href="https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html" class="inline-onebox" rel="noopener nofollow ugc">CUDA Binary Utilities</a></p>
<p>So it’s as easy as running the following commands in a command line:<br>
cuobjdump --list-elf name_of_library.a<br>
cuobjdump --list-ptx name_of_library.a</p>
<p>Cheers,<br>
David</p>
How gamma and beta get updated during backward process in batch normalization layer

username: H_Jahan
cooked: <p>Hi<br>
Could you please let me know whether it is possible to check how the gamma and beta of the batch normalization layer get updated during the backpropagation process? I mean, can I print their values in each epoch?</p>

username: agu
cooked: <p>Maybe this question should not be tagged for distributed?</p>

username: H_Jahan
cooked: <p>Sorry I wasn’t aware. Could you please let me know how can I change it and what is the correct group for this kind of questions?</p>
Visualize Tensor image without copying out of gpu?

username: lucasw
cooked: <p>I want to visualize tensor images generated in the gpu with pytorch with as little delay as possible and minimize usage of system resources. Is there a solution that would involve torch3d and a 3D rendering context that pytorch could feed tensor images into as textures?  I’m not sure if torch3d has the ability to directly render like that.</p>
<p>(copied from here<br>
<a href="https://stackoverflow.com/questions/74670843/display-a-pytorch-gpu-tensor-image-without-copying-to-cpu-memory" class="inline-onebox" rel="noopener nofollow ugc">Display a pytorch gpu tensor image without copying to cpu memory? - Stack Overflow</a>)</p>
Issue with multiple Batch Normalization in U Net

username: Amir-phy
cooked: <p>So I have a UNet architecture with 4 encoders and decoders, along with max pooling and bottleneck, all in a single class. I also wanted to apply Batch Normalization after each convolutional layer within each block, but once I do that, the validation loss doesn’t decay as much as training loss, and stays relatively flat. However, when I apply a single BN, to just one of the blocks, it seems to run with no issues. I’m not sure what causes this problem, or how it can be fixed.</p>
DataLoader worker thread synchronization

username: keshavvinayak01
cooked: <p>I have a general query about how the DataLoader distributes work and synchronises it across the different worker threads that are launched using the <code>num_workers</code> argument.</p>
<p>Do the threads join at the end of each minibatch processing(i.e., at the default/user-defined collate_fn), or each thread is non-blocking and keeps on processing it’s share of data across multiple minibatches, writing the processed data into the main memory, and the iterator just picks up the data from the memory as all the batch elements are available.</p>
<p>Kindly help me understand the workflow, would be much appreciated.<br>
Thanks.</p>

username: nivek
cooked: <p>DataLoader uses process instead of thread. After a batch is prepared by a worker, the batch is put into a <code>multiprocessing_context.Queue()</code> from which the main process will read and return the data.</p>

username: keshavvinayak01
cooked: <p>I get what you’re saying, but my doubt is this: Given a batch size of 128, and num_workers=128 specified to the dataloader, are individual items of a batch given to each worker or is each worker responsible for a single batch, while the rest of the workers handle their own batch processing.</p>
<p>In short, is parallelisation across workers intra-batch or inter-batch?</p>
<p>I would’ve assume it should be intra-batch for faster batch processing, but I have some experimental results that hint towards inter-batch parallelism. Please clarify this for me.</p>
Move tensors to gpu and concatenate them at the same time?

username: TriKri
cooked: <p>Given multiple tensors of the same shape on the CPU (e.g. <code>A = [x, y, z]</code>), is it possible to move them to the GPU and concatenate them at the same time? I know that I can first move each individual tensor to the GPU (<code>A = [a.to(device) for a in A]</code>) and then concatenate them afterwards (<code>A = torch.cat(A, dim=cat_dim)</code>), but that creates extra copying of data on the GPU when concatenating the data.</p>
<p>It feels like it would be more efficient to allocate the space needed for the concatenated tensor on the GPU first and then just move the CPU-tensors to the correct position on the GPU directly, if it is possible to specify a target memory location when moving the tensors to the GPU. Is this possible somehow?</p>
Hierarchical prior with pytorch distributions

username: fjscoutinho
cooked: <p>Hi there,</p>
<p>I am trying to use a Bayesian inference toolbox where the prior must be given as a list of pytorch distributions. The toolbox grabs this list and samples from it.</p>
<p>The prior can be given either as a list of independent torch distributions or as a multivariate distribution. For example, instead of defining the prior as prior = [Normal(torch.tensor([0.0]), torch.tensor([1.0])), Normal(torch.tensor([0.0]), torch.tensor([1.0]))], I could define it as:<br>
prior = MultivariateNormal(torch.zeros(2), torch.eye(2)).</p>
<p>So far so good. The problem is that my model is hierarchical.<br>
My parameters’ priors are:</p>
<pre><code class="lang-auto">$$\log \pi ~ Unif(log(a),log(b))$$
$$\gamma_j | \pi ~ Bernoulli(\pi)$$
$$\beta_j | (gamma_j = 0) ~ \del_0$$ (here \del_0 can in practice be replaced by a normal distribution with very low variance)
$$\beta_j | (gamma_j = 1) ~ Normal(0,\sigma_{\beta}^2)$$,

where a, b and \sigma_{\beta} are known, and \del_0 is a kronecker delta = 1 at 0
</code></pre>
<p>I am not sure how to go about setting up a prior distribution that includes all of these hierarchical dependencies.</p>
<hr>
<p>From searching around a bit I have found out that I can set the distribution for \pi as follows:</p>
<p>import torch.distributions as dist<br>
class LogUniform(dist.TransformedDistribution):<br>
def <strong>init</strong>(self, lb, ub):<br>
super(LogUniform, self).<strong>init</strong>(dist.Uniform(lb.log(), ub.log()),<br>
dist.ExpTransform())</p>
<p>REF: <a href="https://github.com/pytorch/pytorch/issues/11412" rel="noopener nofollow ugc">PyTorch Issue 11412</a></p>
<p>But I am not sure how to:</p>
<ol>
<li>Pass this distribution to the \gamma distribution</li>
<li>Pass the \gamma distribution to the \beta distribution and drawing \beta_j in one of two ways, depending on \gamma_j</li>
</ol>
<hr>
<p>Any help with this would be greatly appreciated! I also take this opportunity to ask whether it is possible to write in LaTeX in these questions and if so, how, because I have tried inserting the dollar signs but they did not work.</p>
<p>Thank you in advance.</p>
Help: Installing Pytorch with GTX 1050 Ti

username: Ioannis_D
cooked: <p>Hello,</p>
<p>I have been using google colab but I have to work on my laptop (not allowed to use google colab).<br>
I couldnt find a definite answer on the version of pytorch to install. I believe is the following, but can someone help please.</p>
<pre><code class="lang-auto">conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=9.2 -c pytorch
</code></pre>
<p>or</p>
<pre><code class="lang-auto">pip install torch==1.7.0+cu92 torchvision==0.8.0+cu92 torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>If I am not mistaken, the GTX 1050 Ti support cuda up to 9.2?</p>
<p>Thank you</p>

username: nwn
cooked: <p>Hello, there is a great post on the developer.nvidia forum which I have linked for you: <a href="https://forums.developer.nvidia.com/t/cuda-for-geforce-gtx-1050-ti/57928/7" rel="noopener nofollow ugc">GTX1050 TI CUDA</a>. This states, that <strong>at least</strong> 9.1 is supported.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Ioannis_D" data-post="1" data-topic="168328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ioannis_d/48/52562_2.png" class="avatar"> Ioannis_D:</div>
<blockquote>
<p>If I am not mistaken, the GTX 1050 Ti support cuda up to 9.2?</p>
</blockquote>
</aside>
<p>No, your 1050 Ti is using compute capability 6.1 and is supported in the current PyTorch release (<code>1.13.0</code>) using the 11.6 and 11.7 CUDA runtime, so no need to downgrade to an older release.</p>

username: Ioannis_D
cooked: <p>This is amazing, thanks a lot.</p>
Optimizer not updating values in custom loss function

username: ralph_20
cooked: <p>The parameters E and K are not getting updated when I try optimizing with a custom loss function. It does work on a simple loss like loss = E*K<br>
How to fix this?<br>
Thanks</p>
<pre><code class="lang-auto"># pytorch code to optimize two values E and K based on a loss function
\
</code></pre>

username: ptrblck
cooked: <p>Re-wrapping a trainable tensor into a new tensor will detach it from the computation graph:</p>
<pre><code class="lang-python">v1 = torch.tensor([(2*torch.log(E)+4*torch.log(K)), (2*torch.log(E)+2*torch.log(K)), 0.0],requires_grad = True)
</code></pre>
<p>Use <code>E</code> and <code>K</code> directly in differentiable operations without creating new tensors and it should work.</p>

username: ralph_20
cooked: <p>I changes it to <code>loss = torch.dot(torch.tensor([(2*torch.log(E)+4*torch.log(K)), (2*torch.log(E)+2*torch.log(K)), 0.0],requires_grad = True),  torch.tensor([(2*torch.log(E)+2*torch.log(K)), (torch.log(E)+3*torch.log(K)), (torch.log(E)+torch.log(K))],requires_grad = True))     </code></p>
<p>but still they’re not getting updated</p>

username: ptrblck
cooked: <p>You are still re-creating tensors and are thus still detaching the computation graph.<br>
Remove the <code>torch.tensor</code> calls in your code and use the tensors directly instead.</p>
How to tranform tf.TensorArray to pytorch

username: henry870603
cooked: <p>How to tranform tf.TensorArray to pytorch , But I have no idea how to do it.</p>
<p>this is the tf function describe<br>
tf.TensorArray(<br>
dtype,<br>
size=None,<br>
dynamic_size=None,<br>
clear_after_read=None,<br>
tensor_array_name=None,<br>
handle=None,<br>
flow=None,<br>
infer_shape=True,<br>
element_shape=None,<br>
colocate_with_first_write_call=True,<br>
name=None<br>
)<br>
and my example : mask = tf.TensorArray(dtype=tf.float32, size=1, dynamic_size=True)</p>
Fastest way to train n models on n interconnected GPUs with batch sizes of 1?

username: 5hadytru
cooked: <p>I’m trying to train 8 models, all with a batch size of 1, on a DGX-A100 node with 8 GPUs on it. What is the fastest way to execute this?</p>

username: ptrblck
cooked: <p>I assume all 8 models are trained in an independent way so I would probably just launch 8 training scripts masking each device via <code>CUDA_VISIBLE_DEVICES</code>.</p>

username: 5hadytru
cooked: <p>But is there a way to do this while not make each training script run n-times slower?</p>

username: ptrblck
cooked: <p>This approach wouldn’t slow down your training by <code>N</code> by design as each GPU can execute its operations independently. If you are seeing a slowdown this would point towards a CPU or IO bottleneck as either the CPU isn’t capable to launch the work for all <code>N</code> devices  or isn’t able to feed the data fast enough.</p>
4D tensor correlation

username: Rane90
cooked: <p>Hi,</p>
<p>I’m having trouble writing code for the following scenario:</p>
<p>I have two 4D tensors <code>&lt;X, Y&gt;</code> of shape <code>(b, c, h, w)</code>. For every <code>c</code> I would like to compute the cross-correlation matrix between <code>X</code> and <code>Y</code> based on the <code>i</code>-th index of <code>w</code> (i.e. correlation based on the <code>i</code>-th columns of each <code>(h,w)</code> matrix).</p>
<p>So for example, if I choose <code>i = 1</code>, the following code calculates the correlation coefficient between column <code>1</code> of the first two matrixes of <code>c</code>, from the first element of the batch of <code>X</code> and <code>Y</code> (i.e. index <code>0</code> of <code>c</code> and <code>b</code>):</p>
<pre><code class="lang-auto">X = torch.randn((3, 2, 4, 5))
Y = torch.randn((3, 2, 4, 5))
i = 1

X_col_i = = X[0, 0, :, i]
Y_col_i = Y[0, 0, :, i]

stacked_cols = torch.vstack((col_i_X, Y_col_i))

coeff = torch.corrcoef(stacked_cols)[0,1]
</code></pre>
<p>Is there any way to vectorize these operations while avoiding a nested for loop for <code>b</code> and <code>c</code>?</p>
<p>The output shape should be <code>(b, c, c)</code>: a <code>(c, c)</code> cross-correlation matrix for each element in the batch.</p>
<p>Thank you</p>

username: KFrank
cooked: <p>Hi Rane!</p>
<aside class="quote no-group" data-username="Rane90" data-post="1" data-topic="168325" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rane90/48/47118_2.png" class="avatar"> Rane90:</div>
<blockquote>
<p>Is there any way to vectorize these operations while avoiding a nested for loop for <code>b</code> and <code>c</code>?</p>
</blockquote>
</aside>
<p>There most likely is.</p>
<p><code>torch.corrcoef()</code> does not seem to operate on batches of matrices, so<br>
the approach would be to write your own batched version of <code>corrcoef()</code>.</p>
<p>In <a href="https://discuss.pytorch.org/t/custom-loss-correlation-by-column/168137/2">your previous thread,</a> we looked at how to write a batched version of<br>
your <code>cov()</code> function.  I would expect that you would be able to do something<br>
broadly similar with <code>corrcoef()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>The output shape should be <code>(b, c, c)</code>: a <code>(c, c)</code> cross-correlation matrix for each element in the batch.</p>
</blockquote>
</aside>
<p>Give it a try, and if you have questions, please post a fully-self-contained,<br>
runnable script, together with its output, that uses for-loops to perform your<br>
desired computation followed by your attempt at a loop-free version.</p>
<p>Best.</p>
<p>K. Frank</p>
Torch.jit.trace don't support torchdynamo model

username: jinyuan.feng
cooked: <p>I was trying Pytorch 2.0 and found that <code>torch. jit.trace</code> could not be convert torch.compile’s model. For example:</p>
<pre><code class="lang-auto">import torch
from torchvision.models import resnet50

model = resnet50(weights=None)
compile_model = torch.compile(model)
example_forward_input = torch.rand(1, 3, 224, 224)
c_model_traced = torch.jit.trace(compile_model, example_forward_input)
torch.jit.save(c_model_traced, "c_trace_model.pt")
</code></pre>
How can I load a non-DDP checkpoint model into a DDP model?

username: Vedant_Roy
cooked: <p>Should doing:</p>
<pre><code class="lang-auto">ddp_model.module.load_state_dict(v)
</code></pre>
<p>be sufficient? or does this have an unknown bug</p>
<p><strong>Edit</strong>: This does not work.</p>

username: ptrblck
cooked: <p>I would recommend to finish the model setup (i.e. in this particular use case loading the <code>state_dict</code>) before wrapping it into <code>DDP</code>, since the same parameter set can then be scattered to all GPUs.<br>
If you delay it and use the internal <code>.module</code> attribute you would need to make sure all models have the same parameter set on all ranks.</p>
Custom Full Gated Convolutional Layer

username: vvsr95
cooked: <pre><code class="lang-auto">class FullGatedConv2D(Conv2D):
    """Gated Convolutional Class"""

    def __init__(self, filters, **kwargs):
        super(FullGatedConv2D, self).__init__(filters=filters * 2, **kwargs)
        self.nb_filters = filters

    def call(self, inputs):
        """Apply gated convolution"""

        output = super(FullGatedConv2D, self).call(inputs)
        linear = Activation("linear")(output[:, :, :, :self.nb_filters])
        sigmoid = Activation("sigmoid")(output[:, :, :, self.nb_filters:])

        return Multiply()([linear, sigmoid])

    def compute_output_shape(self, input_shape):
        """Compute shape of layer output"""

        output_shape = super(FullGatedConv2D, self).compute_output_shape(input_shape)
        return tuple(output_shape[:3]) + (self.nb_filters * 2,)

    def get_config(self):
        """Return the config of the layer"""

        config = super(FullGatedConv2D, self).get_config()
        config['nb_filters'] = self.nb_filters
        del config['filters']
        return config

</code></pre>
<p>I have the above Keras implementation of Custom Gated Convolutional Layer. I want to use similar for my existing pytorch implementation for which i have tried following at stuck-</p>
<pre><code class="lang-auto">class gatedConv(Module):
     def __init__(self,filters):
         super(gatedConv,self).__init__()

         self.filters=filters*2

     def forward(self,inputs):
         output = super(gatedConv, self).call(inputs)
         linear=torch.nn.Linear(output[:, :, :, :self.filters])
         m=torch.nn.Sigmoid()
         sigmoid=m(output[:, :, :, :self.filters])
         return linear*sigmoid

</code></pre>
<p>But Linear activation is not nn. Linear so should i need to custom implement it and how should i inherit Conv2D in pytorch is it similar to keras . Anyone please let me know the path forward. Thankng in advance</p>

username: ptrblck
cooked: <p>You would need to initialize the <code>nn.Linear</code> layer before using it.<br>
Take a look at <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">this tutorial</a> which shows simple model definitions. In particular you will see that layers are initialized in the <code>__init__</code> method and used in the <code>forward</code>.</p>
How do we measure the allocated GPU memory during backward process?

username: marsggbo
cooked: <p>We know that the forward process will retain all intermediate activations so that we can easily  measure the allocated GPU memory of the forward like this:</p>
<pre><code class="lang-python">pre_fw = torch.cuda.memory_allocated() / 1024**2
forward(...)
post_fw = torch.cuda.memory_allocated() / 1024**2
fw_g = post_fw - pre_fw
</code></pre>
<p>However, the backward process will drop the activations when the gradients are obtained. In other words, we cannot measure the correct allocated GPU memory of the backward via the above code.</p>
<p>Thus, I wonder how do we measure the allocated GPU memory during the backward process?</p>

username: soulitzer
cooked: <p>Not sure what you are trying to measure. Backward doesn’t additional allocate memory unless you are preparing to do double backward.</p>
How to load part of pre-trained model?

username: MRR
cooked: <p>I have used pre-trained Bert model and added some linear layers for classification and trained. I have saved the model weights. Now I want to use the model weights of bert only, not those classification layers. Because i want the bert model output which will be 768 dimension by using trained weights as if i use classification layers it will reduced to 2 dimension.</p>
<p>This is model architecture that i used for training.</p>
<pre><code class="lang-auto">class BertClassifier(nn.Module):

    def __init__(self, dropout=0.1):

        super(BertClassifier, self).__init__()

        self.bert = BertModel.from_pretrained('bert-base-uncased')
        self.dropout = nn.Dropout(dropout)
        self.linear = nn.Linear(768,1)
        self.relu = nn.Sigmoid()

    def forward(self, b_input_ids, b_input_mask):

        _, pooled_output = self.bert(input_ids= b_input_ids, attention_mask=b_input_mask,return_dict=False)
        dropout_output = self.dropout(pooled_output)
        linear_output = self.linear(dropout_output)
        final_layer = self.relu(linear_output)

return final_layer
</code></pre>
<p>I used state_dict method to save the weights, before saving i deleted the weights of classification layers and saved it.</p>
<p>Now i created a new model as:</p>
<pre><code class="lang-auto">from transformers import AutoConfig
config = AutoConfig.from_pretrained('bert-base-uncased')
model =  AutoModel.from_config(config)
</code></pre>
<p>Used AutoConfig because to load the model without any pretrained weights so that i can load my trained model weights.</p>
<p>And i am trying to load the weights into this new model but i am getting <strong>Runtime Error Keys Missing and lot of ids or keys</strong>. I also printed the keys of trained model and new model they are not same.</p>
<p>I tried several ways but none of them worked. I think the keys are not being same for the new model as my saved model is problem. should i define new model in another way.I don’t know what to do.</p>
<p>OR Should i just save the whole model and load the model and remove those classification layers so that i can send my input under <strong>torch.no_grad</strong>  to model then maybe i will get the output of bert model with 768 dimension by using trained weights. I am confused to do what and how.</p>
<p>APPRECIATE YOUR RESPONSE OR HELP! THANK YOU.</p>
My basic model doesn’t learn

username: Tahamustapha_Nehdi
cooked: <p>I’m implemnting a simple autoencoder for Mnist dataset but the loss still the same and no imporvment<br>
this is the full code :</p>
<pre><code class="lang-auto">class ModelCAE(nn.Module):
    
    def __init__(self):
        super().__init__()
        
        # enocder 
        #conv layers
        self.encoder = nn.Sequential(
            nn.Conv2d(1,16,3,padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(2,2),
            nn.Conv2d(16,4,3,stride=2,padding=1),
            nn.ReLU(True),
            
            
        )
  

        # decoder 
        # conv layers
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(4, 16,2,stride=2),
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 1,2, stride=2),
            
        )
        
    def forward(self,x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = ModelCAE().to("cuda")
# loss function 
def PSNR(img,target):
    img = torch.sigmoid(img)
    mse = torch.mean((img-target)**2)
    return 20 * torch.log10(255.0 / torch.sqrt(mse))
# omptimizer 
optimizer = torch.optim.Adam(ModelCAE().parameters(),lr = 1e-1,weight_decay = 1e-8)

epochs = 20
outputs = []
losses = []
for epoch in range(epochs):
    train_loss =0.0
    for index,batch in enumerate(train_loader):
        
        imgs,_ = batch
        
        imgs = imgs.to('cuda')
        
        optimizer.zero_grad()
        
        reconstructed = model(imgs)
    
        
        loss = PSNR(imgs,reconstructed)
        
        
        
        loss.backward()
        
        optimizer.step()
        
        losses.append(loss)
        
        train_loss += loss*imgs.size(0)
    train_loss  = train_loss / len(train_loader)
    print(f"end of epoch {epoch}")
    print(f"training loss {train_loss:.6f}")
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97753ec6f502bbd6401072c4053f2af2c1a5661a.png" alt="image" data-base62-sha1="lBRih61d7M2953SbhxQkCkME6tY" width="480" height="463"></p>

username: nwn
cooked: <p>Hello,<br>
the answer is rather easy: You initiate a new model in your optimizer line, therefore, the actual model and its paramaters are never updated.<br>
Change it to optimizer = torch.optim.Adam(<strong>model</strong>.parameters(),lr = 1e-1,weight_decay = 1e-8).<br>
Furthermore, if you want to add a loss’ value to store it in a list, always go for its item: loss.item(). Otherwise, you will store the loss gradient function, too, which uses a lot of memory. I don’t know whether this is desired or not, but your Autoencoder does not have a flattened latent space, which might hinder downstream tasks.</p>
<p>Best,<br>
nwn</p>

username: Tahamustapha_Nehdi
cooked: <p>thank you so much for solution and advices</p>

username: nwn
cooked: <p>Please mark the question as solved, if it did, so that other users know that it has been solved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
`Nans' observed in one environment and not in other

username: Ashima_Garg
cooked: <p>Hi,</p>
<p>I am training one of the models on a dataset. It turns out that in one setting, I observe nans in one environment and do not observe nans in another. How to resolve this? Mismatch of which library can cause this? Should I copy the list of libraries installed in both conda environments?</p>
<p>Thanks,</p>

username: tiramisuNcustard
cooked: <p>Can you please post the exact variable where you are seeing NaN’s and what operations you performed to the variable before you get the NaN’s? Essentially, I am asking you to post your code.</p>
Why does the pure additions can be backward() multiple times?

username: 25349023
cooked: <p>Hello, I’m studying how the autograd in PyTorch works, and I find something confusing:</p>
<p>I’ve learned that if the backward() is called more than once, there will be an runtime error since the graph has been freed:</p>
<pre><code class="lang-python">x = torch.tensor([1., 2.], requires_grad=True)
y = torch.tensor([3., 4.], requires_grad=True)
z = (x * y).sum()
z.backward()
z.backward()  # &lt;-- this will raise an RuntimeError
</code></pre>
<p>However, I find that if the graph contains only additions, then the backward() can be called multiple times without errors:</p>
<pre><code class="lang-python">x = torch.tensor([1., 2.], requires_grad=True)
y = torch.tensor([3., 4.], requires_grad=True)
z = (x + y).sum()
z.backward()
z.backward()  # &lt;-- this will NOT cause any error
</code></pre>
<p>Why is there such a difference?</p>

username: srishti-git1110
cooked: <p>Hi,<br>
As soon as the first <code>.backward()</code> call is made the references to the saved tensors in the computation graph of the tensor on which backward is called are freed.</p>
<p>And so, when saved tensors are required to calculate the gradients, an error will be raised on the second backward call.</p>
<p>With addition, no saved tensors are required to calculate the gradient, hence the behaviour is expected.</p>

username: 25349023
cooked: <p>Thank you for your detailed explanation!</p>
Efficient way of saving list of tensors with variable sizes

username: Chao_Feng
cooked: <p>What is the most memory/loading efficient way to save a list of tensors of variable size (e.g., variable length of sentences)?</p>
<p>For example, I have a list of ~60k tensors. The sum of memory of each tensor is 17M. But when I save the list of tensor into *.pt file, it occupies 31M memory (whereas when saved as one tensor by content them all it only cost 17M memory). Is there a way to save it more efficiently and also enable efficient loading during training?</p>
<p>Thank you!</p>

username: dreidizzle
cooked: <p>Good question. I don’t know, but just giving an idea (which might be wrong). I’d guess the reason why it goes from 17 M → 31 M is due to all the padding necessary to make it “1” tensor.  So then, maybe you can sort the tensors by length and create batches of similar length? This would add minimal padding per batch. The idea is that you have T1 + … T_60K = 17 M but as one tensor you get 31 M. But, if you batch the right way you’ll get B_1 + … + B_k ~ 17 M still and |B_l| &gt;&gt; 1, i.e. your batches are much larger than 1. Maybe there’s a better way, but unsure; I’d experiment …</p>
typeError: cross_entropy_loss() argument input(position 1) must be a tensor, not googlenetOutputs

username: Dorra
cooked: <p>Hi  everyone,</p>
<p>I faced the above error when I tried to use torchvision.models.googlenet as follow:<br>
model=models.googlenet(num_classes=args.num_class).cuda()</p>
<p>If someone has an idea about how I can solve that, I will be appreciable.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p><code>GoogLeNet</code> outputs an object in training mode called <code>GoogLeNetOutputs</code> which contains <code>.logits</code>, <code>.aux_logits1</code>, and <code>.aux_logits2</code>.<br>
You would thus have to index the desired attribute to calculate your loss instead of passing the object directly to your criterion.</p>

username: Dorra
cooked: <p>Many thanks<br>
it’s solved <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Problems with inference time and memory in YOLOv7

username: Nuisigor
cooked: <p>I’m testing the difference between yolov5 and yolov7 in my code but is not working as it is in the main graphic of the comparison with the real-time object detectors, the yolov7 is slower than the v5(almost the double of the time) and is using more ram too, I’m using exactly the same code of the v7 with some alterations to run inside my code ( just put the detect file in a class).<br>
Do someone knows what could possibly is happening here? Or having the same problem? Or even maybe I’m using it wrong? Because as I said, is the exactly same code but in a class now.</p>
Default interpolation parameters in RoiAlign

username: sigma_x
cooked: <p>Search returns the C implementation of RoI Align interpolation, which seems to use bilinear interpolation by default (<a href="https://github.com/pytorch/vision/blob/35d1d9d3f01016c65ac7f3d0700d2474929acdea/torchvision/csrc/ops/cuda/roi_align_kernel.cu" class="inline-onebox" rel="noopener nofollow ugc">vision/roi_align_kernel.cu at 35d1d9d3f01016c65ac7f3d0700d2474929acdea · pytorch/vision · GitHub</a>), but I still can’t get other parameters from this method (<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.interpolate.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.interpolate — PyTorch 1.13 documentation</a>), scale_factor or align_corners. Are they the same as in the default, i.e. None and None?</p>
Can I get Torch Trace from Torch Script?

username: anleb96
cooked: <p>Hello<br>
I am trying to convert PyTorch model to CorML using coremltools. But I get an error</p>
<blockquote>
<p>AttributeError: ‘torch._C.Graph’ object has no attribute ‘1’</p>
</blockquote>
<p>Also, I get a warning:</p>
<blockquote>
<p>Support for converting Torch Script Models is experimental. If possible you should use a traced model for conversion.</p>
</blockquote>
<p>So, I thought it may be beneficial to use actually traced model. But I don’t have access to the original torch model. Only the scripted one. Can I convert it to traced one?</p>
RecursionError: maximum recursion depth exceeded

username: Arupreza
cooked: <p>Dear Concern,<br>
Recently I have been trying to build an inception model. But it gives this error. Can anyone give me a solution? I really appreciate any help you can provide.</p>
<pre><code class="lang-auto">training_data = []

def create_training_data():
    for category in CATEGORIES_Train:  # do dogs and cats

        path = os.path.join(DATADIR_Train,category)  # create path attack
        class_num = CATEGORIES_Train.index(category)  # get the classification  (0, 1,.... ).

        for img in tqdm(os.listdir(path)):  # iterate over each image 
            try:
                img_array = cv2.imread(os.path.join(path,img))  # convert to array
                new_array = cv2.resize(img_array, (100, 100))  # resize to normalize data size
                new_array = np.transpose(new_array, (2, 0, 1))
                training_data.append([new_array, class_num])  # add this to our training_data
            except Exception as e:  # in the interest in keeping the output clean...
                pass
            #except OSError as e:
            #    print("OSErrroBad img most likely", e, os.path.join(path,img))
            #except Exception as e:
            #    print("general exception", e, os.path.join(path,img))

create_training_data()

print(len(training_data))

#############################################################

TRAIN_BATCH_SIZE = 64
VALID_BATCH_SIZE = 32
NUM_EPOCHS = 10
LEARNING_RATE = 1e-3
NUM_WORKERS = 0
PIN_MEMORY = False

train_loader = torch.utils.data.DataLoader(training_data, batch_size=TRAIN_BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS, pin_memory=PIN_MEMORY)
valid_loader = torch.utils.data.DataLoader(testing_data, batch_size=VALID_BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS, pin_memory=PIN_MEMORY)


model = InceptionV3().to(device)
start = time.time()
fit(model, train_loader, valid_loader, learning_rate=LEARNING_RATE, num_epochs=NUM_EPOCHS)
print(f'Total training time: {time.time() - start}')
model.load_state_dict(torch.load('Inception_Adam_T_Tesla_1.pth'))
</code></pre>

username: ptrblck
cooked: <p>Could you give us more information about which line of code is raising the issue and add missing pieces to the code snippet to make it executable, please?</p>

username: Arupreza
cooked: <p>Thank you, I already got the solution.</p>
Scalar operation for n-dimensional tensor

username: HK0930
cooked: <p>Hi all</p>
<p>I have a question regarding scalar operation on n-dimensional tensors.</p>
<p>I have two tensors shaped as follows;</p>
<pre><code class="lang-auto">x torch.Size([30, 48, 201, 229])
gamma torch.Size([30, 48])

</code></pre>
<p>I would like to add/multiply the gamma to each element of matrix [201, 229] in the tensor x for each 48 dimension for 30 batches.</p>
<p>How can I achieve this?<br>
Thank you so much for your help in advance!</p>

username: tom
cooked: <p>This is a case for broadcasting: You add size-1 dimensions to gamma so it has ones for the last two axes: <code>x * gamma[:, :, None, None]</code> the <code>:</code> takes a full dimension from the original gamma, the <code>None</code> adds a singleton dimension which is then treated with broadcasting.<br>
You could also pretend to be fancy by using <code>einsum</code>: <code>torch.einsum('ijkl,ij-&gt;ijkl', x, gamma)</code> will give the same result. If you remove letters from the last bit (following the <code>-&gt;</code>) of the equation string, <code>einsum</code> will sum over these.</p>
<p>Best regards</p>
<p>Thomas</p>

username: HK0930
cooked: <p>Great! It works! Thank you so much!</p>
GPU usage not decreasing even after resizing images

username: mariyajose
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8116a19c05185157692748a7cb65fe34acd6883.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b8116a19c05185157692748a7cb65fe34acd6883" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8116a19c05185157692748a7cb65fe34acd6883_2_690x404.jpeg" alt="2" data-base62-sha1="qgladJ4dsi0wxQwxMHD1kF3OCkj" width="690" height="404" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8116a19c05185157692748a7cb65fe34acd6883_2_690x404.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b8116a19c05185157692748a7cb65fe34acd6883_2_1035x606.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b8116a19c05185157692748a7cb65fe34acd6883.jpeg 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1266×742 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>While training the GPU utilization is nearly 94% even with a batch size of 4. Even after reducing the size of images by 50%, the GPU utilization isn’t decreasing. Is there anything wrong with the code?<br>
model = fasterrcnn_resnet50_fpn()</p>

username: ptrblck
cooked: <p>I’m not sure I understand the issue correctly, but why do you want to decrease the GPU utilization? If you want to slow down GPU util. and thus slow down your code (maybe for debugging purposes?) you could use <code>time.sleep</code> calls inside your training code.</p>

username: mariyajose
cooked: <p>sorry, maybe my wording is wrong. When I use batch size 4 with normal images and batch size 4 with images of reduced size, the memory utilized by GPU is the same. I want to increase my batch size.</p>
Connect Torchserve to Mobile Application

username: abbiecoleen
cooked: <p>Hello! We’re currently exploring torchserve and is thinking of using it on a mobile application that we are doing for school. Is it possible to connect torchserve to a mobile application and would it be possible to know how would that be possible? Thank you so much.</p>

username: ptrblck
cooked: <p>I think this should be possible if you configure <code>TorchServe</code> to accept remote requests e.g. via the <a href="https://pytorch.org/serve/rest_api.html">REST API</a>.</p>

username: abbiecoleen
cooked: <p>Do you mean getting requests from the Inference API (to request and get the output prediction) provided by the TorchServe or do we need to build a separate REST API that would connect our mobile application and TorchServe? I’m sorry for the questions but I can’t seem to find any resources online that did a similar work of connecting mobile app and TorchServe. Thank you!</p>
Easiest way to check for `nan` value in a model parameters

username: MarkW
cooked: <p>What would be the easiest way to detect if any of the weights of a model is <code>nan</code>?</p>
<p>Is there a built in function for that?</p>

username: soulitzer
cooked: <p>You might be looking for <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detection" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.13 documentation</a></p>

username: ptrblck
cooked: <p>Besides the mentioned Anomaly detection util., this small code snippet would also check for NaNs in registered parameters:</p>
<pre><code class="lang-python">model = models.resnet101()
is_nan = torch.stack([torch.isnan(p).any() for p in model.parameters()]).any()
</code></pre>

username: MarkW
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , It looks like what I’m after.<br>
But if I want it to return a <code>bool</code>, should I just <code>bool(is_nan)</code>?</p>

username: ptrblck
cooked: <p><code>is_nan.item()</code> will return a Python <code>bool</code>.</p>
Neural network training error

username: Cezar
cooked: <p>I m trying to train a convolutional neural network and i’m getting an error, and i don’t know how to solve it</p>
<p><strong>Network:</strong></p>
<p>class CNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)<br>
self.conv2 = nn.Conv2d(10, 20, kernel_size=3)<br>
self.conv2_drop = nn.Dropout2d()<br>
self.fc1 = nn.Linear(720, 1024)<br>
self.fc2 = nn.Linear(1024, 2)</p>
<pre><code>def forward(self, x):
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
    x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
    x = x.view(x.size(0), -1)
    x = F.relu(self.fc1(x))
    x = F.dropout(x, training=self.training)
    x = self.fc2(x)
    return x
</code></pre>
<p><strong>Error:</strong><br>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (25x1615440 and 720x1024)</p>

username: J_Johnson
cooked: <p>What size image are you putting in?</p>
<p>The error is occurring because the output of your convolutions and max pooling do not match the input size of your fc1 Linear layer. A print statement should show this. Either you should run an adaptive pooling layer to reduce the size of the convolution outputs to 6x6, or you can increase the size of your Linear layer input to 1615440.</p>

username: Cezar
cooked: <aside class="quote no-group" data-username="J_Johnson" data-post="2" data-topic="168229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>1615440</p>
</blockquote>
</aside>
<p>This is the shape of my images<br>
torch.Size([3, 1024, 1280])</p>

username: J_Johnson
cooked: <p>Try this:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def **init**(self):
        super(CNN, self).**init**()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=3)
        self.conv2_drop = nn.Dropout2d()
        self.avg_pool=nn.AdaptiveAvgPool2d((6,6)) #adapative pooling
        self.fc1 = nn.Linear(720, 1024)
        self.fc2 = nn.Linear(1024, 2)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x=self.avg_pool(x) #apply
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return x
</code></pre>
<p>Note: The above should only prevent the error and allow some inference.</p>
<p>For reference:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html</a></p>
Combining two tensors

username: bryan123
cooked: <p>Let say I have:</p>
<p>x = torch.randn(256,384)<br>
y = torch.randn(256, 384)</p>
<p>Can you tell me how to get a variable z with a size of [2, 256, 384]?</p>

username: ptrblck
cooked: <p><code>torch.stack</code> should work:</p>
<pre><code class="lang-python">x = torch.randn(256,384)
y = torch.randn(256, 384)
z = torch.stack((x, y))
</code></pre>
How to let BatchNorm2d process three-dimensional input in the inference stage

username: zuowanbushiwo
cooked: <pre><code class="lang-auto">class CausalTransConv(nn.Module):
    def __init__(self, in_ch, out_ch, kernel_size, stride, padding, output_padding):
        super(CausalTransConv, self).__init__()
        self.trans_conv = nn.ConvTranspose2d(in_channels=in_ch, out_channels=out_ch, kernel_size=kernel_size,
                                             stride=stride, padding=padding, output_padding=output_padding)
        self.norm = nn.BatchNorm2d(out_ch)
        self.activation = nn.PReLU()                                     

    def forward(self, x):
        T = x.size(-1)
        out = self.trans_conv(x)[..., :T]
        out = self.norm(out)
        out = self.activation(out)   
        return out
</code></pre>
<p>A simple model is as above, in the training phase I use 4-dimensional data like [B,N,H,W], in the inference phase I can use 4-dimensional input using pytorch. But when deployed on the end side, the inference framework I use can only accept 3-dimensional input, without the dimension of Batch. The above con2d and PReLU can handle 3-dimensional data input which means that [1,N,H,W] and [N ,H,W] can get numerically equal results, but BatchNorm2d can only input [1,N,H,W]. How should I change it? When jit.trace can input [N,H,W]。<br>
Thanks！</p>
Why GPU available memory is not the same before and after inference even I empty cache and delete variable?

username: techkang
cooked: <p>I ran the following code:</p>
<pre><code class="lang-auto">import torch as t
import torchvision as tv


def main():
    print('memory 1: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    print('init model')
    data = t.rand(1, 3, 224, 224).cuda()
    model = tv.models.resnet50().cuda().eval()
    print('memory 2: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    print('inference...:')
    with t.no_grad():
        model(data)
    print('memory 3: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    print('empty cache: ')
    t.cuda.empty_cache()
    print('memory 4: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    print('delete variable')
    del data
    del model
    print('memory 4.1: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    print('exit function')


if __name__ == '__main__':
    main()
    print('memory 5: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    t.cuda.empty_cache()
    print('empty cache again')
    print('memory 6: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
    print('synchronize')
    t.cuda.synchronize()
    print('memory 7: ', t.cuda.mem_get_info(0)[0] / 1024 / 1024)
</code></pre>
<p>The output is</p>
<pre><code class="lang-auto">memory 1:  6987.54296875
init model
memory 2:  6359.54296875
inference...:
memory 3:  5835.54296875
empty cache: 
memory 4:  5883.54296875
delete variable
memory 4.1:  5883.54296875
exit function
memory 5:  5883.54296875
empty cache again
memory 6:  6003.54296875
synchronize
memory 7:  6003.54296875
</code></pre>
<p>So, can anyone tell me why the GPU RAM is not the same before and after the <code>main()</code> function?</p>

username: ptrblck
cooked: <p>The CUDA context uses memory to initialize the driver, load the CUDA kernels etc. and will be lazily initialized during the first CUDA call.</p>
Instance segmentation of point cloud data

username: Pravin_Poudel
cooked: <p>Hi, I am new to machine learning.</p>
<p>I am making a tool in web front end where user can go and manually add bounding box to the element inside a scene in 3D scanned Model.</p>
<p>In 3d scanned model, we can not know which small meshes are part of one mesh. So it is not possible to add physics model because we dont know bounding box of element in scene.</p>
<p>Like, from code when you traverse 3D model of room generated from scanner, it is not possible to get chair in room.</p>
<p>In recent works in some research paper, i saw people working on instance segmentation of point cloud data. I am new to machine learning so if i dont understand many thing especially what i will get in output.</p>
<p>Can anyone answer few questions to me on that?</p>