akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: Calculated padded input size per channel: (4 x 4). Kernel size: (5 x 5). Kernel size can't be greater than actual input size

username: naychelynn
cooked: <p>Dear all,</p>
<p>I know this error is not new somehow but I’m stuck though I follow some suggestions with those similar errors.<br>
The network is trained with Inception Model V3 and then fully connected with the following sequential classifier.<br>
Before training with Inception V3, trained with ResNet and FC with the same architecture classifier. And it worked.<br>
I’m trying to implement the ensemble NN.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6f46481a8a12c92caa2c013c28f3640e9c55281.png" alt="image" data-base62-sha1="uFzMOOPisUv1K7Aa06CKhImYMPD" width="503" height="133"><br>
Welcome every suggestion and thanks in advance.</p>

username: ptrblck
cooked: <p>Your input is too small for the model and I believe Inception models were trained with inputs of <code>[299, 299]</code>. You might this need to increase the spatially dimension of your input or try to decrease the pooling in the model.</p>
From Pycharm to Jupyter

username: John4
cooked: <p>Hello, I wrote a code in Jupyter Notebook and I get the following error:</p>
<pre><code>FileNotFoundError: [WinError 3] Impossibile trovare il percorso specificato: 
                                'UPM_UNIPV_colab_NEMESIS_HSI_images/Datasets/'
</code></pre>
<p>That means it’s impossible to find the specified path. How can I import a path in Jupyter?<br>
I used the same code in Pycharm and it works.</p>
<pre><code>def main():
    datapath = 'UPM_UNIPV_colab_NEMESIS_HSI_images/Datasets/'
    data, mat_files, alldata = loadata(datapath)
</code></pre>

username: ptrblck
cooked: <p>It seems you are using a relative path so either change it to an absolute path or make sure you can reach the file from your current working directory.</p>
RuntimeError: running_mean should contain 1 elements not 256

username: Hemanth_Yernagula
cooked: <p>I’m trying to train an simple encoder decoder network, while training it is working proper but while loading back the model and running inference, I’m getting following error.</p>
<pre><code class="lang-auto"> RuntimeError: running_mean should contain 1 elements not 256

</code></pre>
<p><strong>Encoder Network</strong></p>
<pre><code class="lang-auto">
class LSTMEncoder(nn.Module):
    """_summary_

    Args:
        nn (_type_): _description_
    """

    def __init__(self, input_size, hidden_size=265):
        super(LSTMEncoder, self).__init__()

        self.embed = nn.Embedding(input_size, hidden_size,)

        self.rnn_unit = nn.LSTM(hidden_size, hidden_size,
                                dropout=0.2, num_layers=2, batch_first=True)
        self.rnn_unit2 = nn.LSTM(hidden_size, hidden_size,
                                dropout=0.2, num_layers=2, batch_first=True)
        
        
        self.batch_norm = nn.BatchNorm1d(hidden_size)
        self.dropout = nn.Dropout(0.2)

    def forward(self, x):
        """_summary_

        Args:
            x (_type_): _description_

        Returns:
            _type_: _description_
        """
        x = self.embed(x)
        x = relu(x)
        out, (h_n, c_n) = self.rnn_unit(x)
        out = self.batch_norm(out) --&gt; At this point error ioccurring
        out, (h_n, c_n) = self.rnn_unit2(out)
        
        return out, h_n, c_n

</code></pre>
Can linear layers take multidimensional input?

username: Tayyab43
cooked: <p>If each feature I want my neural network to analyse is a 2d array, would this be possible to pass to a linear layer or would it require flattening?</p>
<p>Im fairly new to Neural networks so any help would appreciated</p>
<p>Thanks</p>

username: beaupreda
cooked: <p>Hi,</p>
<p>Yes, it would require flattening if you want to consider all the elements of the array. I have put a small example printing the size of the output when you give a 2D input to a Linear layer without flattening.</p>
<pre><code class="lang-auto">    # 2d input
    linear_layer_2d = nn.Linear(in_features=64, out_features=32)
    # 1st dimension (128) = batch dimension, input 64 x 64
    input_2d = torch.randn(128, 64, 64)
    output_2d = linear_layer_2d(input_2d)
    print(output_2d.size())
    # torch.Size([128, 64, 32])
    
    # 1d input (2d flattened)
    linear_layer_1d = nn.Linear(in_features=4096, out_features=32)
    # input_1d size = [128, 4096]
    input_1d = torch.flatten(input_2d, start_dim=1)
    output_1d = linear_layer_1d(input_1d)
    print(output_1d.size())
    # torch.Size([128, 32])
</code></pre>
<p>Let me know if it’s not clear!</p>

username: jarvico
cooked: <p>Hi,</p>
<p>I have a similar question. I am trying to train a simple NN which takes 2-d tensor as input data, and outputs a 2-d tensor. Specifically, I would like to have an input and output of shape 16x2. Outputs of the hidden layers should also be 2d. (ignoring batch size).</p>
<p>if, for example, my batch size is 128, my requirement is below:</p>
<p>input shape: 128 x 16 x 2<br>
1st hidden layer output shape: 128 x 256 x 2<br>
2nd hidden layer output shape: 128 x 512 x 2<br>
3rd hidden layer output shape: 128 x 256 x 2<br>
output shape: 128 x 16 x 2</p>
<p>I wonder how to design and implement such a NN. Would you please comment?</p>
<p>Best wishes,</p>
Pytorch with cuda doesn't work - torch.cuda.is_available() false

username: canislynx
cooked: <p>I’m trying to use pytorch with my GPU but I tried so many things and nothing worked.<br>
I’m working on a desktop windows 10 PC with a NVIDIA GeForce GTX 570. I updated the nvidia driver to the latest version: 391.35<br>
I read that with this graphic card it works the CUDA 9.1<br>
I created a brand new conda environment for trying this with python 3.6. The conda version is 22.9.0.<br>
I installed the pytorch with the cudatoolkit with this command:</p>
<blockquote>
<p>conda install pytorch torchvision cudatoolkit=9.0 -c pytorch</p>
</blockquote>
<p>But when I run in python this:</p>
<blockquote>
<p>import torch<br>
torch.cuda.is_available()</p>
</blockquote>
<p>the output is:</p>
<blockquote>
<p>false</p>
</blockquote>
<p>Do you have any suggestion? What can I do to may try to solve this problem?<br>
Thank you everyone for the help.</p>

username: dimou_gk
cooked: <p>Greetings,</p>
<p>You get this output simply because your graphics card doesnt support CUDA 9.1. Your GPU should support up to CUDA 8.0 as seen <a href="https://en.wikipedia.org/wiki/CUDA#GPUs_supported" rel="noopener nofollow ugc">here</a><br>
For more information on that please refer to <a href="https://stackoverflow.com/questions/60987997/why-torch-cuda-is-available-returns-false-even-after-installing-pytorch-with" rel="noopener nofollow ugc">this</a> stackoverflow post where a thorough explanation is given</p>
<p>Best,</p>

username: canislynx
cooked: <p>thank you for your answer.</p>
How can I accumulate and save tensors?

username: BobKim
cooked: <p>Hello everyone!</p>
<p>I want to check the weight of a specific layer while training ResNet.<br>
So I got the following output through the script below.</p>
<pre><code>    get_weight = net.get_parameter('module.layer1.0.conv1.weight')
    L2norm_weight = torch.norm(get_weight, p='fro')
    print(L2norm_weight)
</code></pre>
<p>output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed5d7f616003194c6e0001c3ea27bc7d8eb0b75f.png" alt="image" data-base62-sha1="xRPtDXoamgZETRawiknuTjmwulh" width="498" height="262"></p>
<p><strong>So I want to save the number (such as 7.2596, 7.3254, etc.) in the tensor for every epoch, what should I do?</strong></p>

username: ConvolutionalAtom
cooked: <p>I understand that you will use this value only only for recording (i.e. not for anything like gradient computation), I would suggest you use <code>L2norm_weight.data</code> to extract its tensor content. This way <code>L2norm_weight.data</code> is not tracked autograd and will not affect computation graph. You can simply take these value and put into a list</p>
CTCLoss explanation

username: dimou_gk
cooked: <p>Greetings,</p>
<p>I have a Text Recognition task at hand (with a CNN backbone and a LSTM for the sequence prediction) and I want to use CTCLoss but there are some things I don’t underastand:</p>
<ol>
<li>Do I need to insert blank “character” in-between same characters in a word?</li>
<li>I am trying to use the (N,S) format for the <strong>Targets</strong> as it says here <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html" rel="noopener nofollow ugc">CTCLoss</a> , with padded sequences up to 30 characters (which are essentially 0) but there is some overlap with the default blank character 0 so how do I tackle this matter?</li>
<li>What are the <strong>Input_lengths</strong> and how does it differ from <strong>Target_length</strong> ?</li>
</ol>
<p>Thanks in advance</p>
How to evaluate multiple networks simultaneously by broadcasting single input to them?

username: Ilker_GURCAN
cooked: <p>What is the best way to evaluate N identical neural networks simultaneously by feeding (broadcasting) the same input to all of them? Creating a giant network that houses all at once and splitting weights among them logically? I do not want to create each neural network separately and run a for-loop to evaluate them. It is too inefficient.<br>
Thank you.</p>
PyTorch 1.13 builds for Android

username: TimPushkin
cooked: <p>PyTorch 1.13 was released almost two weeks ago, but I cannot find any builds for Android. Will it be available?</p>

username: TimPushkin
cooked: <p>I created <a href="https://github.com/pytorch/pytorch/issues/88878" rel="noopener nofollow ugc">an issue about this</a>, but it’s been a while and no answers there either</p>

username: TimPushkin
cooked: <p>Got an answer from <a href="https://github.com/malfet" rel="noopener nofollow ugc">Nikita Shulga</a> from PyTorch crew: sadly, “there are no plans to publish mobile builds for 1.13 at the moment”, but building from source should work</p>
Training a neural network after capturing more data

username: itorch
cooked: <p>I apologize in advance if this question is trivial. I am not an expert in machine learning, but I use it occasionally.</p>
<p>My question is whether a neural network should be trained from scratch after capturing new sets of measurements. Simply put, suppose a neural network is trained with a dataset and the trained model is saved. At a later time when the dataset is enhanced with new measurements, can the saved model be used to train on the newly enhanced dataset? Or, should the network be trained using the untrained model from scratch?</p>
<p>To me using the pre-trained model makes more sense as the datasets are still of the same type. They are just from different measurements. In a related query, can I train my model with a portion of my dataset then train another portion and so on?</p>

username: vdw
cooked: <p>I feel training from scratch might be the safer option. Here’s my argument:</p>
<p>Say your new data is about 10% o the original data, but “looks” very different (i.e., the data comes from a different distribution).  Now, if you would continue training your pre-trained model with only the new data, it will start to fit only the new data and “un-learn” the previous data.</p>
<p>Say you’ve trained your initial model with the initial data for 100 epochs, then I feel you should continue training using only the 10% new data not more then another 10 epoch. But this is just my crude intuition. I’d be happy to here from an expert. This is an interesting question.</p>
How to update gradients for nodes where output was computed using noise?

username: Navjot
cooked: <p>I am training an IMLE model for generative modelling, where at some layer <em>l</em> some noise is added to the node values and then sent further. <strong>While calculating/updating the gradients, do the model need to know the actual values of noise?</strong><br>
The problem with IMLE is that it tries multiple random noise samples and then uses only one of these samples, so the PyTorch model might not have the current values corresponding to the noise sample chosen.</p>
<p>If we are ignoring the noise while computing gradients, any insight on how model handles it would be appreciated.</p>
Why CUDACachingAllocator limited block shareing inside stream?

username: liddk
cooked: <pre><code class="lang-auto">  bool get_free_block(AllocParams&amp; p) {
    BlockPool&amp; pool = *p.pool;
    auto it = pool.lower_bound(&amp;p.search_key);
    if (it == pool.end() || (*it)-&gt;stream != p.stream())
      return false;
    p.block = *it;
    pool.erase(it);
    return true;
  }
</code></pre>
<p>If block is allocated by one stream, and splited to one free block, why can’t we use it in aother stream?<br>
When CudaMalloc is host blocking.</p>

username: liddk
cooked: <p>Anyone has idea on it?</p>

username: Shan19900305
cooked: <p><a href="https://github.com/zdevito" rel="noopener nofollow ugc">@zdevito</a></p>
What is the best way to save a tensor in one class?

username: kp600168
cooked: <p>Hi everyone! I would like to save/memorize a tensor in one class.<br>
Here is my code snip:</p>
<pre><code class="lang-auto">class A {
public:
    void func(const torch::Tensor &amp;input) {
        t = new torch::Tensor(at::empty_like(input));
    }
    ~A() {
        if (t) {
            delete t;
            t = nullptr;
        }
    }
    torch::Tensor *t = nullptr;
};
</code></pre>
<p>or</p>
<pre><code class="lang-auto">class B {
public:
    void func(const torch::Tensor &amp;input) {
        t = at::empty_like(input);
    }
    ~B() {
        // Could I need to do something else to forget the tensor?
    }
    torch::Tensor t;
};
</code></pre>
<p>Or is there any other way to prevent running out of memory?<br>
How can I do that correctly? Hope for your help!</p>
No module named 'privacy_lint' ! How can i fixe this error!

username: Rafika_Benledghem
cooked: <p>I try to run this code from github examples  <a href="https://github.com/facebookresearch/privacy_lint/blob/experimental/examples/cifar10_attack.py" rel="noopener nofollow ugc"><strong>cifar10_attack.py</strong></a> but it doesn’t work !!!</p>

username: ptrblck
cooked: <p>I guess the <code>privacy_lint</code> folder is not in your <code>PYTHONPATH</code> or not cloned into your working directory and thus cannot be imported.</p>

username: Rafika_Benledghem
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb4eaf77eb6d923fff6c9091dff943b4921051ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb4eaf77eb6d923fff6c9091dff943b4921051ed" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb4eaf77eb6d923fff6c9091dff943b4921051ed.png" alt="image" data-base62-sha1="xzCMSduQb7hge2JDKt88BWcjNkF" width="690" height="358" data-dominant-color="3C3B3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">888×461 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><code>pip</code> cannot find a corresponding package (and I also don’t see it being mentioned in their GitHub repository), so make sure to install it locally.</p>
Connecting a CNN model to a LSTM/GRU layer for image predictions

username: d3tk
cooked: <p>I have a dataset of images and then a continuous value. I’m using a CNN model to predict that value from the image. The images and values are in a time series, so I’d like to connect a GRU or LSTM to the CNN layers, but I’m having trouble doing so. The input is images that are (1,360,360) since they’re black and white, and I’d like to input a sequence of 12 images in the series at a time. This is my model thus far:</p>
<pre><code class="lang-auto">class CNN_GRU(nn.Module):
  def __init__(self):
    ## Pool, ReLU, Dropout
    super().__init__()

    ## CNN
    self.cnn = torch.nn.Sequential(
      nn.Conv2d(in_channels=1, out_channels=4, kernel_size=9, stride=5), 
      nn.ReLU(), 
      nn.MaxPool2d(kernel_size=4, stride=(2,2)), 
      nn.Conv2d(in_channels=4, out_channels=8, kernel_size=5, stride=1, padding='same'), 
      nn.ReLU(), 
      nn.MaxPool2d(kernel_size=3, stride=(2,2)), 
      nn.Conv2d(in_channels=8, out_channels=16, kernel_size=3, stride=1, padding='same'), 
      nn.ReLU(), 
      nn.Conv2d(in_channels=16, out_channels=8, kernel_size=3, stride=1, padding='same'), 
      nn.ReLU(), 
      nn.Conv2d(in_channels=8, out_channels=8, kernel_size=3, stride=1, padding='same'), 
      nn.ReLU(), 
      nn.MaxPool2d(kernel_size=3, stride=(2,2)), 
      nn.ReLU(), 
      nn.Dropout(0.5)
      )
    ## LINEAR
    self.linear = torch.nn.Sequential(
      nn.Linear(in_features = 784 ,out_features=200), 
      nn.ReLU(), 
      nn.Dropout(0.5), 
      nn.Linear(in_features=200, out_features=100), 
      nn.ReLU(), 
      nn.Dropout(.25),
      nn.Linear(in_features=100, out_features=20), 
      nn.ReLU(), 
      nn.Linear(in_features=20, out_features=1)
    )
    
    ## GRU layer
    self.GRU = nn.GRU(input_size=392, hidden_size=1, num_layers=3, batch_first=True)
  

  def forward(self, x):

    print("0: ", x.shape)
    cnn_output = self.cnn(x)
    print("Cnn output: ", cnn_output.shape)
    cnn_output = cnn_output.flatten()
    print("Cnn output: ", cnn_output.shape)

    gru_output, _h_o = self.GRU(cnn_output)
    print("gru_output: ", gru_output.shape)

    linear_output = self.linear(gru_output)
    

    return linear_output
</code></pre>
<p>The error I get is that the GRU expects 3 dimensions (batch, sequence length, input size) and my input is 5 dimensions. How can I reshape my input, or am I going about this in the complete wrong way?</p>
Is it possible to get statistics during DDP training?

username: qqin3
cooked: <p>Hello,</p>
<p>I am using DDP for multi-host training. I tend to investigate the collective communication process, and therefore wonder if there is any statistics information or logs available.<br>
For example, it would be helpful for me to know when and how many times the <code>allreduce</code> operation is called in each training epoch and the distribution of job sizes. Are there any methods to get such information?</p>
Can this two stage indexing operation be merged into one?

username: LuoXin-s
cooked: <p>I have the following codes:</p>
<pre><code class="lang-python">H = image.shape[1]
W = image.shape[2]
  
# a is some const integer
x = torch.arange(0, H)
y = torch.arange(0, W)
image_scale = image[:, x * a % H, :][:, :, y * a % W]
</code></pre>
<p>I’m wondering whether or not the code of the last line could be merged into the single one. Also, is there a more effective method to implement this logic?</p>

username: TriKri
cooked: <p>What do you mean by “the single one”? What does that refer to?</p>

username: LuoXin-s
cooked: <p>only index the image once. like <code>image[:, :, :]</code>, it is a indexing operation.</p>

username: TriKri
cooked: <aside class="quote no-group" data-username="LuoXin-s" data-post="1" data-topic="166694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luoxin-s/48/35927_2.png" class="avatar"> LuoXin-s:</div>
<blockquote>
<p><code>image[:, x * a % H, :][:, :, y * a % W]</code></p>
</blockquote>
</aside>
<p>You probably want to use <code>image[:, x * a % H, y * a % W]</code>.</p>
How to cut off the relationship between two torch's value?

username: winchest
cooked: <pre><code class="lang-auto">a = self.a 
a = a[:, :, None]
a[:, j % 10] = b
</code></pre>
<p>when I change the value of <code>a</code> ,I find that the value of  <code>self.a</code> is changed too. How can I keep the value of <code>self.a</code>?</p>

username: KFrank
cooked: <p>Hi Zhao!</p>
<aside class="quote no-group" data-username="winchest" data-post="1" data-topic="166700" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ecae2f/48.png" class="avatar"> winchest:</div>
<blockquote>
<p>when I change the value of <code>a</code> ,I find that the value of  <code>self.a</code> is changed too.</p>
</blockquote>
</aside>
<p>This is a general python issue rather than an issue specific to pytorch.</p>
<p><code>a</code> and <code>self.a</code> are python <em>references</em> that both refer to the <em>same</em> object<br>
(in your case a pytorch tensor).</p>
<p>Consider this pure-python illustration of how references work:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; l = [1, 2, 3]       # l is a reference that refers to an object that is a list
&gt;&gt;&gt; l_same = l          # l_same is another reference that refers to the same object as l
&gt;&gt;&gt; l_copy = l.copy()   # l_copy refers to a new object that is a copy of the original list
&gt;&gt;&gt;
&gt;&gt;&gt; id (l)              # the object id (location in memory) of the original list
2856103785280
&gt;&gt;&gt; id (l_same)         # l_same refers to the same object as shown by its id
2856103785280
&gt;&gt;&gt; id (l_copy)         # l_copy refers to a different object (different id)
2856103722624
&gt;&gt;&gt;
&gt;&gt;&gt; l                   # contents of (list referred to by) l
[1, 2, 3]
&gt;&gt;&gt; l_same[1] = 42      # modify object referred to by l_same
&gt;&gt;&gt; l                   # contents of l is modified because it's the same object
[1, 42, 3]
&gt;&gt;&gt; l_copy[1] = 999     # modify object referred to by l_copy
&gt;&gt;&gt; l                   # change not reflected in l because l_copy is a different object
[1, 42, 3]
</code></pre>
<p>To make an independent copy of a pytorch tensor, use <code>.clone()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">a = self.a.clone()
</code></pre>
<p>(If <code>self.a</code> is part of the computation graph and you <em>don’t</em> want the cloned<br>
copy <code>a</code> to also be part of the computation graph, use <code>.detach()</code> first.)</p>
<p>Best.</p>
<p>K. Frank</p>
Input spatial size for feeding pretrained ResNet50 model from torchvision

username: Tm4
cooked: <p>I need to use ResNet50 pretrained model from <a href="https://pytorch.org/vision/0.8/models.html" rel="noopener nofollow ugc">torchvision library</a> but it is not clear that what is the required height and width size of the input. Only in one of its introductionary section it is mentioned that <em><strong>'All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224.</strong></em>’ and it is ambiguous. It is not clear that the input size should be at least (224,224) or any spatial position upper thatn 224 is (square or rectabgular size) accepted?</p>
Torch-encoding for cuda 11.6

username: Ahmad_Zobair_Surosh
cooked: <p>I am struggling to install the (torch-encoding) package for cuda 11.6</p>
<p>I have no problem using torch itself, but i can’t build or install torch-encoding with pip or conda.</p>
<p>error:</p>
<pre><code class="lang-auto">ld/temp.linux-x86_64-cpython-310/tmp/pip-install-t74d_udw/torch-encoding_3d239eb112774b44aa6d43ce06ef2dfe/encoding/lib/gpu/activation_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=gpu -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 -std=c++14
      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/home/engrapher/anaconda3/envs/langseg2/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
          subprocess.run(
        File "/home/engrapher/anaconda3/envs/langseg2/lib/python3.10/subprocess.py", line 526, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
      

</code></pre>
<p>Any help is appreciated</p>

username: ptrblck
cooked: <p>Based on the <a href="https://hangzhang.org/PyTorch-Encoding/notes/compile.html">install instructions</a> it seems this lib depends on <code>torch==1.4.0</code>, so I doubt it was updated and compatible with the latest PyTorch or CUDA release. You could post an issue in their GitHub repository and ask if this package is still maintained.</p>

username: Ahmad_Zobair_Surosh
cooked: <p>Looks like I should have done some research myself.</p>
<p>Thanks a lot!</p>
Create dataset or dataloader from images

username: Crispolo
cooked: <p>Hi, I have some images ordered ordered in folders, where the label (quality) for each image is included into the image name.</p>
<p>For example:<br>
<strong>Root &gt; sub folder 1 &gt; sub folder 2 &gt; image</strong><br>
Data &gt; fluid_object &gt; category_1 &gt; object1_group_category_quality1.png<br>
Data &gt; fluid_object &gt; category_2 &gt; object1_group_category_quality2.png<br>
Data &gt; hard_object &gt; category_1 &gt; object1_group_category_quality1.png<br>
etc…</p>
<p>I’m new on PyTorch and I’m not sure about how I can create the dataset and the dataloader, or directly the dataloader, starting from data in this form.</p>
<p>Do you have some hint on how I can address that?</p>
<p>Thanks in advance.</p>
MUNIT paper AdaptiveInstanceNorm2d and LayerNorm issue

username: Sarmad_GTU
cooked: <p>Hi<br>
I have an equation about the decoder normalization type. in the follwwing repo</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/NVlabs/MUNIT">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/NVlabs/MUNIT" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/4fdec38b62e660f98e69b34d5b669afdbffedd73c8cef1c0a01b6b8e29cf096e/NVlabs/MUNIT" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/NVlabs/MUNIT" target="_blank" rel="noopener nofollow ugc">GitHub - NVlabs/MUNIT: Multimodal Unsupervised Image-to-Image Translation</a></h3>

  <p>Multimodal Unsupervised Image-to-Image Translation - GitHub - NVlabs/MUNIT: Multimodal Unsupervised Image-to-Image Translation</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Is the normalization step done in the Residual Block parts of the decoder or in the up-sampling part of the decoder, I misunderstand the difference between the AdaptiveInstanceNorm2d and the LayerNorm in the code since the paper said use the AdaIN not mention the LayerNorm ?<br>
in LayerNorm where are the gamma and beta come from? Is it from MLP or what?</p>
<p>anyone can give me a hint, thank you in advance</p>
To preserve gradient from .detach() or suggestions to replacement

username: dhan
cooked: <p>Hi there,  I’ve already notice that ‘detach()’ prevent backward process.</p>
<p>Is there any method to replace the ‘detach().cpu().numpy()’ but, preserve the gradient ?</p>
<p>Operation want to be :</p>
<p><strong>Tensor(GPU) feature →  CPU →  Numpy(or something) with Gradient(which could backward)</strong></p>
<p>It is okay if there is other choice than Numpy,  like; list or whatever which could still preserve the gradient after ‘detach or copy’</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>No, since <code>detach()</code> explicitly detaches the tensor from the computation graph as indicated by the function name. If you need to use a 3rd party library for some computations (e.g. numpy), you could implement a custom <code>autograd.Function</code> and write the <code>backward</code> pass manually as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Load_state_dict encounter strange problem when load weights of attributes defined by setattr

username: LuoXin-s
cooked: <p><code>load_state_dict</code> encounter strange problem when loading weights of attributes defined by <code>setattr</code>.</p>
<p>Specifically, I defined a model with the below attributes defined by <code>setattr</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">for freq_bias_u, freq_bias_v in zip(self.freq_biases_u, self.freq_biases_v):
    setattr(self, f'upsampler_{freq_bias_u:.4f}_{freq_bias_v:.4f}_r', up(num_feat=num_feat, num_out_ch=num_out_ch))
    setattr(self, f'upsampler_{freq_bias_u:.4f}_{freq_bias_v:.4f}_i', up(num_feat=num_feat, num_out_ch=num_out_ch))
</code></pre>
<p>However, when I load the saved model from the checkpoint, I encounter the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5e2ae21d945893db334a8c3e1c413dbe5e8ae29.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5e2ae21d945893db334a8c3e1c413dbe5e8ae29" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5e2ae21d945893db334a8c3e1c413dbe5e8ae29.png" alt="image" data-base62-sha1="nFurrOSJ0V5EPxyXwb6d25vRrNT" width="690" height="245" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1586×564 69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m sure my model has these parameters as confined by print <code>named_parameters</code> of the model.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec74929080d7c143288bbd63cd9a98392d09694c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec74929080d7c143288bbd63cd9a98392d09694c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec74929080d7c143288bbd63cd9a98392d09694c.png" alt="image" data-base62-sha1="xJMrbz66wPtQFiAm6Agt6Z9zBVi" width="253" height="500" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×1442 35.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: JuanFMontesinos
cooked: <p>setattr is overwritten in nn.Modules to ensure certain functionalities. Maybe that’s the problem.</p>

username: LuoXin-s
cooked: <p>The source of the problem is that I used <code>setattr</code> to define attributes with its name containing “.”. However, when load weights, PyTorch explains “.” as the separator of hierarchical attributes.</p>
Non deterministic behaviour when indexing tensor with meshgrid

username: Ivano_Donadi
cooked: <p>Hi,</p>
<p>I am trying to avoid a nested for loop by creating a meshgrid of the two loop variables. However doing the same assignment twice using the grids gives me different results:</p>
<p>Here is the code with the assignment done with a nested for loop:</p>
<pre><code class="lang-auto">var1 = torch.zeros_like(vertex[bi]) # shape (126,224,9,2), dtype=torch.float32
var2 = torch.zeros_like(vertex[bi]) # shape (126,224,9,2), dtype=torch.float32

# dl_dv0 and dl_dv1 have shape (rn,vn,2) = (256, 9, 2)
# hw_index has shape (rn,vn,2) and contains indices

for r in range(rn):
  for k in range(vn):
        var1[hw_index_0[r,k,0],hw_index_0[r,k,1],k,:] += dl_dv0[r,k,:]
        var1[hw_index_1[r,k,0],hw_index_1[r,k,1],k,:] += dl_dv1[r,k,:]
    
        var2[hw_index_0[r,k,0],hw_index_0[r,k,1],k,:] += dl_dv0[r,k,:]
        var2[hw_index_1[r,k,0],hw_index_1[r,k,1],k,:] += dl_dv1[r,k,:]

diff0 = (var1 - var2).abs()
print(diff0.mean(), diff0.max())

&gt;&gt;&gt;  tensor(0., device='cuda:0') tensor(0., device='cuda:0')

</code></pre>
<p>The difference is 0 as expected since the two variables receive the two assignments.</p>
<p>If I try to do the same with meshgrid I get:</p>
<pre><code class="lang-auto">all_ks = torch.LongTensor([i for i in range(vn)])
all_rs = torch.LongTensor([i for i in range(rn)])
grid_r, grid_k = torch.meshgrid(all_rs, all_ks)

var1 = torch.zeros_like(vertex[bi]) # shape (126,224,9,2), dtype=torch.float32
var2 = torch.zeros_like(vertex[bi]) # shape (126,224,9,2), dtype=torch.float32

# dl_dv0 and dl_dv1 have shape (rn,vn,2) = (256, 9, 2)
# hw_index has shape (rn,vn,2) and contains indices

var1[hw_index_0[grid_r,grid_k,0],hw_index_0[grid_r,grid_k,1],grid_k,:] += dl_dv0[grid_r,grid_k,:]
var1[hw_index_1[grid_r,grid_k,0],hw_index_1[grid_r,grid_k,1],grid_k,:] += dl_dv1[grid_r,grid_k,:]

var2[hw_index_0[grid_r,grid_k,0],hw_index_0[grid_r,grid_k,1],grid_k,:] += dl_dv0[grid_r,grid_k,:]
var2[hw_index_1[grid_r,grid_k,0],hw_index_1[grid_r,grid_k,1],grid_k,:] += dl_dv1[grid_r,grid_k,:]

diff0 = (var1 - var2).abs()
print(diff0.mean(), diff0.max())

&gt;&gt;&gt; tensor(2.3726e-06, device='cuda:0') tensor(0.6372, device='cuda:0')
</code></pre>
<p>As you can see the same assignments produce two different variables. Am I misunderstanding how to use meshgrids? Is it possible to use them to achieve what I’m trying to do? I tried searching on google for something similar with no success.</p>
<p>I am using torch 1.7.1 and can’t update due to external constraints.</p>
Grayscale Image transpose parameters for pytorch

username: Arupreza
cooked: <p>Dear Concern,<br>
What will be the transpose parameter for the grayscale image for conv2d?</p>
<pre><code class="lang-auto">training_data = []

def create_training_data():
    for category in CATEGORIES_Train:  # do dogs and cats

        path = os.path.join(DATADIR_Train,category)  # create path attack
        class_num = CATEGORIES_Train.index(category)  # get the classification  (0, 1,.... ).

        for img in tqdm(os.listdir(path)):  # iterate over each image 
            try:
                img_array = cv2.imread(os.path.join(path,img))  # convert to array
                new_array = cv2.resize(img_array, (100, 100))  # resize to normalize data size
                new_array = np.transpose(new_array, (2, 0, 1))
                training_data.append([new_array, class_num])  # add this to our training_data
            except Exception as e:  # in the interest in keeping the output clean...
                pass
            #except OSError as e:
            #    print("OSErrroBad img most likely", e, os.path.join(path,img))
            #except Exception as e:
            #    print("general exception", e, os.path.join(path,img))

create_training_data()

print(len(training_data))```

![image|690x28](upload://nJGZ8jjBcygJ7ZeiQxBYoCFk3HO.png)</code></pre>

username: ptrblck
cooked: <p>I guess the current code is permuting the dimensions from <code>HWC</code> to <code>CHW</code>?<br>
If so, you could reuse the same code assuming your grayscale image uses a channel dimension with the size 1 or remove it in case the image only has two dimensions <code>HW</code>.</p>

username: Arupreza
cooked: <p>For RGB images this transpose concept is not clear to me, If I remove this line it gives the error about it got 100 channels. So how does this transposition make the image in 3 channels? In your other replay to another user, you mentioned alpha and what is the alpha dimension.</p>
<p>I really appreciate your help, for you my concept become more clear.</p>
<pre><code class="lang-auto">new_array = np.transpose(new_array, (2, 0, 1))
</code></pre>

username: ptrblck
cooked: <p><code>np.transpose</code> permutes the dimensions in the same way <code>torch.permute</code> works. In your use case it is unrelated to the alpha channel, as this would be an additional 4th channel making the input <code>RGBA</code>.<br>
As explained, the code permutes the dimensions from <code>[H, W, C]</code> (channels-last) to <code>[C, H, W</code>]  (channels-first).</p>

username: Arupreza
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks I understood.</p>