akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why do not use shuffle when training discriminator in GAN

ChenyS: <p>I’am learning GAN recently with book GANs in Action and other programs on the Internet. And I jsut learned how to use GAN to generator mnist number. However, I noticed that neither the book nor other programs used shuffle when training discriminator(figure1). So I tried using shuffle, but the outcome was poor, almost was noise(figure2&amp;3). And I increased the interation, the outcome using shuffle imporved, but it took a lot more time and even still worse than not using shuffle. I was told using shuffle dataset will accelerate the model to find better parameters, but why it was at opposite when training GAN?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64a3cf8a1aeb807cf686aed5e036c8529cdbe002.png" alt="VZBKIK07{MN)Z%0S)_N4M" data-base62-sha1="emiI0FV9BCb8DUuKV7SiZatabGG" width="413" height="35"><br>
figure1:this is keras code, but the result was same using pytorch</p>

ptrblck: <p>I can’t comment on the Keras code, but the <a href="https://github.com/pytorch/tutorials/blob/dafab370c760c7b64006e88e6a1f1cbf7f7f488c/beginner_source/dcgan_faces_tutorial.py#L256-L258">PyTorch DCGAN example</a> uses <code>shuffle=True</code> in the <code>DataLoader</code> setup.</p>

ChenyS: <p>Yes. But that just shuffles mnist number 1-9, but does not shuffle real images and fake images when training discriminator. Real images and fake images are sent to discriminator successively, not collectively and randomly. That is to say we send a batch of real images to discriminator, and then send another batch of fake images to it, why can’t we just send a batch of shuffled real images and fake images together when training discriminator (this turned out a poor results)?</p>

ptrblck: <p>Thanks for clarifying the question.<br>
I believe that’s one of the “GAN hacks” as e.g. described by <a class="mention" href="/u/smth">@smth</a> in <a href="https://github.com/soumith/ganhacks#4-batchnorm">here</a>.<br>
While the repository is ~5 years old by now and also contains this warning:</p>
<blockquote>
<p>(this list is no longer maintained, and I am not sure how relevant it is in 2020)</p>
</blockquote>
<p>it still seems to be relevant <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

ChenyS: <p>Thank you very much for answering. Now I know it’s neccessary to use two batches of data. But it seems no one has proven this, maybe it’s kind of hard math problem <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=10" title=":joy:" class="emoji" alt=":joy:">.</p>

ptrblck: <p>Additionally, also take a look at the corresponding <a href="https://www.youtube.com/watch?v=myGAju4L7O8">talk</a> by Soumith at Neurips 2016. Note that GAN training indeed moved in the last couple of years and I’m sure there are more and/or other “hacks” now to let these types of models work, so you might also want to check the recent literature/talks.</p>
Tensor Plotting

Nil_MSh: <p>Hi! I’m plotting 2 tensors with <code>plt.plot()</code> and the Y axes on one of them is form 0 to 1.4 and the other one from 0 to 2. I want them both to be the same. What should I do?<br>
I’ve attached pics.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/71fbc1103f836aa51923c12e138c0195c8e9acf9.png" data-download-href="https://discuss.pytorch.org/uploads/default/71fbc1103f836aa51923c12e138c0195c8e9acf9" title="Airplane - max - nontarget - layer2, 9"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71fbc1103f836aa51923c12e138c0195c8e9acf9_2_690x342.png" alt="Airplane - max - nontarget - layer2, 9" data-base62-sha1="gglkLxB6M60jDZm64VoSKHPFajL" width="690" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71fbc1103f836aa51923c12e138c0195c8e9acf9_2_690x342.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71fbc1103f836aa51923c12e138c0195c8e9acf9_2_1035x513.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71fbc1103f836aa51923c12e138c0195c8e9acf9_2_1380x684.png 2x" data-dominant-color="ECF3F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Airplane - max - nontarget - layer2, 9</span><span class="informations">1536×763 99.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42439b623540c0dc139d96c12797303bd87871b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/42439b623540c0dc139d96c12797303bd87871b8" title="Airplane - max - target - layer2, 9"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42439b623540c0dc139d96c12797303bd87871b8_2_690x342.png" alt="Airplane - max - target - layer2, 9" data-base62-sha1="9scml1J7drrylhQT7qqKzTXUSgM" width="690" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42439b623540c0dc139d96c12797303bd87871b8_2_690x342.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42439b623540c0dc139d96c12797303bd87871b8_2_1035x513.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42439b623540c0dc139d96c12797303bd87871b8_2_1380x684.png 2x" data-dominant-color="DEEBF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Airplane - max - target - layer2, 9</span><span class="informations">1536×763 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AlphaBetaGamma96: <p>If you’re plotting them separately you can do, <code>plt.ylim(0,2)</code> before doing <code>plt.show()</code></p>
Libtorch.so exporting mkl symbols?

Wojciech_Jablonski: <p>I am trying to integrate libtorch into my C++ codebase. I do however use Intel MKL internally and libtorch_cpu.so also exports <code>mkl*</code> symbols - linking them together causes the symbols to get mixed up and results in segfaults.</p>
<p>Is there a way to isolate symbols exported by libtorch <code>*.so</code>s or some other supported way for me to proceed?</p>
<p>I’m not using <code>CMake</code> for the builds and have to integrate the process into existing infrastructure.</p>
RuntimeError: Given groups=1, weight of size [256, 256, 3, 3], expected input[64, 1, 256, 256] to have 256 channels, but got 1 channels instead

dieptran149: <p>I got</p>
<blockquote>
<p>RuntimeError: Given groups=1, weight of size [256, 256, 3, 3], expected input[64, 1, 256, 256] to have 256 channels, but got 1 channels instead</p>
</blockquote>
<p>I think it is because I tried to use the output of <code>self.net</code> which has a <code>(256x256x1)</code> shape to feed into self.layer1 which needs a <code>(256x256)</code> input. But I can’t find out any method to help me to do it. Could anyone please tell me how to handle this problem? I very much appreciate it.<br>
Excuse my ignorance. Thank you all!</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = torch.hub.load('mateuszbuda/brain-segmentation-pytorch', 'unet', in_channels=3, out_channels=1, init_features=32, pretrained=True)
        # =&gt; 256 x 256 x 1
        self.layer1 = nn.Sequential(
            nn.Conv2d(256, 256, 3,padding =1),
            nn.Conv2d(256, 256, 3,padding =1),
            nn.Conv2d(256, 256, 1,padding =1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=1, padding=1)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(128,128,3,padding =1),
            nn.Conv2d(128,128,3,padding =1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 3, stride = 1, padding = 1)
        )
        self.layer3 = nn.Sequential(
            nn.Conv2d(64,64,3,padding =1),
            nn.Conv2d(64,64,1,padding =1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size = 3, stride = 1, padding = 1)
        )
        self.layer4 = nn.Sequential(
            nn.Linear(1024,1),
            nn.Sigmoid()
        )
        
    def forward(self,x):
        x = self.net(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        return x
</code></pre>

Sayed_Nadim: <p>Try to change this:</p>
<pre><code class="lang-auto">self.layer1 = nn.Sequential(
            nn.Conv2d(256, 256, 3,padding =1),
            nn.Conv2d(256, 256, 3,padding =1),
            nn.Conv2d(256, 256, 1,padding =1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=1, padding=1)
        )
</code></pre>
<p>Into this:</p>
<pre><code class="lang-auto">self.layer1 = nn.Sequential(
            nn.Conv2d(1, 256, 3,padding =1),
            nn.Conv2d(256, 256, 3,padding =1),
            nn.Conv2d(256, 256, 1,padding =1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=1, padding=1)
        )
</code></pre>
<p>The first <code>nn.Conv2d</code> layer in <code>self.layer1</code> expected <code>256</code> channels as input (<code> nn.Conv2d(256, 256, 3,padding =1</code>), hence, the error. Change it to <code>nn.Conv2d(1, 256, 3,padding =1)</code> should resolve the error.</p>

dieptran149: <p>Thank you for your replying. It solved my problem</p>
Extract similar word from model

svv: <p>Hello! Can I extract a similar word from the model? Also, is it possible to assess the similarity between two words, not sentences, but only words? I’ll hope you help.</p>

gphilip: <p>There is not nearly enough information in your question to start thinking about how to help you!</p>
<p>Please remember that we don’t know anything about the problem that you are facing. Please provide enough context so that someone who has no clue about what you are dealing with (but has some experience working with ML and PyTorch) can understand the issue.</p>

Unity05: <p>I agree with <a class="mention" href="/u/gphilip">@gphilip</a>, we need a bit more information to comprehent your question fully.<br>
However, perhaps <a href="https://www.nltk.org/howto/wordnet.html" rel="noopener nofollow ugc">the NLTK WordNet</a> helps, to get similar words (synset) as well as compute the similarity between two words.</p>

svv: <p>I want to extract similar words by context from a language model like bert. For example, model.most_similar from gensim by word2vec, also for the transformers language model.</p>
Flock: Input/output error, Aborted (core dumped)

nathnayan: <p>I am trying to train a YOLO model with two NVIDIA GPUs in a linux server with the following versions:<br>
PyTorch Version: 1.7.0<br>
CUDA: 11.0</p>
<p>For distributed training initialization:</p>
<blockquote>
<p>if len(gpu_list) &gt; 1:</p>
<pre><code>    assert distributed_folder, "The distributed-training folder isnt't set" #check if not the default 0
    distributed_learning_filename = str(Nnet_name) + "_distlearn_setup" #remove this file when program is stopped!!
    distributed_init_filepath = os.path.join(distributed_folder, distributed_learning_filename)
    #there is more than 1 GPU-index than use distributed training

    dist.init_process_group(backend='nccl',  # use distributed backend 'nccl'
                            init_method='file://' + str(distributed_init_filepath), #file used to setup the distributed learning
                            world_size = 2, #number of nodes for distributed training
                            rank = 0 #distributed training node rank
                            )
</code></pre>
</blockquote>
<p>Its works fine while I use single GPU, but getting some unexpected error for 2 GPUs. The errors are following:</p>
<blockquote>
<p>File train.py", line 209, in train<br>
rank = distributed_node_rank<br>
File “/home/miniconda3/envs/pytorchDevelopment1.7/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 455, in init_process_group<br>
barrier()<br>
File “/home/miniconda3/envs/pytorchDevelopment1.7/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 1960, in barrier<br>
work = _default_pg.barrier()<br>
RuntimeError: flock: Input/output error<br>
terminate called after throwing an instance of ‘std::system_error’<br>
what():  flock: Input/output error<br>
Aborted (core dumped)</p>
</blockquote>
<p>I set up the world_size 2, and rank 0, and the errors is coming from the init_process_group().</p>
<p>Any comments or suggestions would be appreciated.</p>
Difference between 1.8.1+cu111 and 1.8.1 with cuda preinstalled

Barath_Lakshmanan: <p>Software version: Cuda 11.1, Nvidia driver 470.x.</p>
<p>System 1 (GTX Titan X): I’m running into segmentation fault when training a model with 1.8.1+cu111 but works fine with 1.8.1 with cuda 11.1 preinstalled.<br>
System 2 (Titan RTX): I’m running the same code without issues with 1.8.1+cu111 but gives seg fault with 1.8.1 with cude 11.1 preinstalled.</p>
<p>Can someone point out the difference between two package and any help on solving the above issue where I want to run the same code on two systems.</p>

ptrblck: <p>I don’t quite understand the “1.8.1 with cude 11.1 preinstalled” setup.<br>
If you are installing the binaries (pip or conda) the local CUDA toolkit won’t be used unless you are building custom CUDA extensions, so what would be the difference to “1.8.1+cu111”?</p>

JohnSidKim: <p>i had met the similar issue.<br>
i equipped double GTX 1080 Ti on the pc.<br>
but it wasn’t work well with libtorch 1.9+cu10.2. also it works well with libtorch 1.9+cpu.<br>
i wanted to handle it, but i didn’t.</p>
<p>also i can run training a model with pytorch 1.9+cu10.2 in pycharm project…</p>
<p>haha…</p>
Take whole data from dataloader

PhysicsIsFun: <p>Greetings!</p>
<p>In order to print the initial loss of my network (before training), I want to feed the entire data set into my network like this:</p>
<pre><code class="lang-auto">    loss_train = np.zeros(epochs+1)
    accu_train = np.zeros(epochs+1)
    
    # initial loss 
    output = model(batches_train.dataset.data)   # batches_train is dataloader
    targets = batches_train.dataset.targets
    loss_train[0] = criterion(output, targets).detach()
</code></pre>
<p>The problem is that this gives the original dataset without the transformations that were specified in the creation of the dataloader. For example, if I specified the <code>transforms.ToTensor() </code> operation for the dataloader, my code will complain that the model( ) does not accept np.arrays as input.</p>
<p>Is there a cheap way to fix this? I could easily just iterate over all the batches, but I was wondering whether one can do without a for-loop.</p>
<p>Regards!</p>

ptrblck: <p>One way would be to set the <code>batch_size</code> to the entire dataset length and create the transformed batch via:</p>
<pre><code class="lang-python">input = next(iter(loader))
</code></pre>

PhysicsIsFun: <p>Right, and then turn the batch_size back to normal for training? What is the usual way to do this? Are people just iterating over the batches first? I think knowing the initial loss is important.</p>
<p>edit: trying to assign a new batch size via <code>batches_train.batch_size = </code>  runs into error</p>
<blockquote>
<p>ValueError: batch_size attribute should not be set after DataLoader is initialized</p>
</blockquote>

ptrblck: <p>Creating a new <code>DataLoader</code> would be cheap in case you are lazily loading the data, so you might want to create a different one with this setup.</p>
Point Op Fusion default

Trinayan_Baruah: <p>Hi,</p>
<p>I have come across numerous discussion that PyTorch JIT can fuse pointwise ops like layernorm, softmax etc and provide higher performance. Yet it seems most of the layers that do point ops do not use the JIT support  for point op fusing by default. For example, when I call nn.LayerNorm, I do not see any fusion happening.</p>
<p>What is the reason for this and how can we use JIT support on existing layers without rebuilding pytorch?.</p>

googlebot: <p>you’re misunderstanding, these are “vector” ops, elementwise ops are “scalar” ones with one-to-one input to output element dependencies. With this trivial dependency structure, fusion implementation is relatively simple and generic. And what is talked about is inter-op fusion, so it works with two or more operations (e.g. log(a*b+c)), not e.g. nn.LayerNorm by itself.</p>
<p>ps IIRC there are some JIT optimizations for some layer combinations, that’s like another form of fusion too</p>
Missing key(s) in state_dict: "pool.p"

Khabbab_Zakaria: <p>I have a trained model by me that  I saved using</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), PATH)
</code></pre>
<p>Now I am testing the model after importing the model as</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):
    def __init__(self, model, pool):
        super(Model, self).__init__()
        self.model = model
        self.pool = pool

    def forward(self, sample):
        output = self.model(sample)
        #output = self.pool(output) #GeM
        output =  torch.sum(output.view(output.size(0), output.size(1), -1), dim=2) #SPoC

        output = F.normalize(output, p=2, dim=1) #L2 Normalizartion
        return output

model = Model(model=resnet, pool=gmp)
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>You can see, I actually am not using the pool but using something else (pool was used in a different experiment using the same model).</p>
<p>Now I am getting this error</p>
<pre><code class="lang-auto">File "test.py", line 123, in &lt;module&gt;
    model.load_state_dict(torch.load(PATH))
File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 1406, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Model:
	Missing key(s) in state_dict: "pool.p". 
</code></pre>
<p>here PATH = path to the saved model ‘model.pt’<br>
Can anyone explain what the error is and how to solve it?<br>
I checked the forums and found a case where users were using nn.dataparallel or something like that, but I am not using it.</p>

JuanFMontesinos: <p>The only thing I can imagine is you passed something different from a tensor or a nn.module when u instantiated that the 1st time but then you did so when u tried to load.</p>
<p>Can you make a reproducible script?</p>

Khabbab_Zakaria: <p>Well I could not understand what you explained <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"><br>
I can make a script but the code is too long. But I can tell you, this same code ran when I tested the experiments involving the pool = gmp</p>

JuanFMontesinos: <p>Classes have attributes (anything you call by self.whatever).<br>
Pytorch is coded such that when you set<br>
self.anything= X<br>
It inspects that X. If it’s a nn.Module or a tensor (a layer or a pytorch tensor) it keeps track of it.<br>
Then when you save the model, pytorch iterates over all the layers nor you defined.</p>
<p>This bug can be generated if<br>
When you created an instance of the class<br>
<code>model = Model(model=resnet, pool=gmp)</code> &lt;&lt;&lt;&lt; model is an instance of the class Model<br>
gmp (X in my example) is a different thing from a nn.Module or a tensor.<br>
The second time, when you want to load the model X is a nn.Module or a tensor.<br>
Then it “misses” that.</p>
<p>So in your case it seems pool is a nn.Module but there is something defined inside which is missing. Either due to what i explained or due to some if conditional statement which is skipping it.</p>
KNN classifier with CNN features

Angelina_Robert: <p>Hi, I have extracted the features of cifar dataset using resnet18, just wondering, if I can use the KNN classifier on these extracted features  (Features are stored in an embedded file). If the related code is provided, I would really appreciate that.<br>
Cheers</p>

ptrblck: <p>Yes, you should be able to transform the stored features to numpy arrays (if they are stored as tensors, use <code>arr = tensor.numpy()</code>) and train a KNN classifier via e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html"><code>sklearn.neighbors.KNeighborsClassifier</code></a>.</p>
Why my Batch-Norm Layer parameters doesn't get udates

moslem_yazdanpanah: <p>Hi, during the training process of a ResNet model I have modified the affine parameters just before the loss.back as below:</p>
<pre><code class="lang-auto">    i = 0
    for layer in model.modules():
        if isinstance(layer, nn.BatchNorm2d):
            layer.bias = nn.Parameter(value 1)
            layer.weight = nn.Parameter(value 2)
            i += 1
</code></pre>
<p>I realized the layer.bias and layer.weight does not get updated during the back-prob and remains as 0 and 1 respectively while req_grad is true and the optimizer produces the appropriate gradient tensor. I am wondering if these assignment has caused the problem? If yes, is there an alternative approach?</p>

tom: <ul>
<li>you would want to keep the Parameters the same thing after you instantiated the optimizer.</li>
<li>if what you want is “set the parameter to something”, try <code>with torch.no_grad(): layer.weight.copy_(value 1)</code>. This copies the value 1 to the parameter rather than replacing the parameter with a completely unrelated one the optimizer doesn’t know anything about.</li>
</ul>
<p>Would that make sense?</p>
<p>Best regards</p>
<p>Thomas</p>

moslem_yazdanpanah: <p>Thanks, <a class="mention" href="/u/tom">@tom</a>, I got the issue.</p>
Model.cuda() and this lead to an error saying that cuda out of memory

nikiguo93: <p>Hi, I used the following code:<br>
model = torchvision.models.resnet50(pretrained=True)<br>
model.cuda()<br>
Then it returned an error message saying that cuda out of memory. Can anybody tell me how to fix this problem?</p>

ptrblck: <p>Could you check the free GPU memory via <code>nvidia-smi</code> and make your sure device has enough free memory to store the <code>resnet50</code> on it?</p>

nikiguo93: <p>Hi, the error message looks like the following:<br>
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.92 GiB total capacity; 10.34 GiB already allocated; 9.38 MiB free; 10.34 GiB reserved in total by PyTorch)<br>
It occured when executing the following code:<br>
for i_batch, sample_batched in enumerate(TraindataLoader):<br>
local_X = sample_batched[‘input’].type(dtype)<br>
local_Y = sample_batched[‘EEGResponse’].type(dtype)<br>
yhat = model(local_X)     -----error occurred</p>
<p>looking forward to your suggestion.</p>

ptrblck: <p>As the error message describes, your device has only 9.38MiB of free memory, while you are trying to allocate more, so you would need to reduce the memory requirement e.g. by lowering the batch size.</p>
How to Unfreeze Layers one by one

Tclha: <p>Hi!</p>
<p>I’m working with a PyTorch model from <a href="https://github.com/yitu-opensource/T2T-ViT" rel="noopener nofollow ugc">here</a> (T2T_ViT_7).</p>
<p>I’m trying to freeze all layers except the last (head) layer and train it on my dataset. I want to evaluate its performance, and then unfreeze layers one by one and train it each time and see how it performs.</p>
<p>To initially freeze all the layers and and just unfreeze the head layer, I used:</p>
<pre><code class="lang-auto">for param in model.parameters():
    param.requires_grad_(False)

model.head.requires_grad_(True)
</code></pre>
<p>Now I want to start from the bottom, and start unfreezing layers one by one. How can I do this? Do I use model.modules() or maybe model.children()?</p>
<p>Thank you!</p>

JuanFMontesinos: <p>You just can’t.<br>
Think that a model is some sort of tree. Each module has many submodules but only a single parent. The parameters are obtained recursively exploring all the childrends but the root is not aware of how many childrens and subchildrens are there.</p>
<p>You can just store the object in a list and read from the “end” (there is no such thing) to the root.</p>
Defining parameters in the forward pass

Niels_PyTorch: <p>In PyTorch, one can define parameters in the forward method rather than in the init method (when their shape depends on the size of the inputs).</p>
<p>Small example (as explained in <a href="https://discuss.pytorch.org/t/dynamic-parameter-declaration-in-forward-function/427">this thread</a>):</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self):
        # you need to register the parameter names earlier
        self.register_parameter('weight', None)

    def forward(self, input):
        if self.weight is None:
            self.weight = nn.Parameter(torch.randn(input.size()))
        return self.weight @ input
</code></pre>
<p>However, how can I load the parameters of a given state dict into such a model? (i.e. suppose I have a state_dict with the ‘weight’ parameter). Currently I’m getting this:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for MyModule:
	Unexpected key(s) in state_dict: "weight".
</code></pre>
<p>Second question: the code above will not work when moving the model to GPU, as model.cuda() is usually called before forward(). How can I make it work both on CPU and GPU?</p>

ptrblck: <p>I don’t think there is a clean approach besides performing a “warmup” iteration using your real inputs to create the parameters. Afterwards, you could push the parameters to the desired device and pass them also to the optimizer. There is a newly added <code>meta</code> device, but I don’t think this would help here given that the shapes are unknown. If I understand its use case correctly, it’s used to abstract the device, but not necessarily the data/shape.</p>

Niels_PyTorch: <p>Ok, so first I need to pass a dummy input to create the parameters:</p>
<pre><code class="lang-auto"># instantiate model defined above
model = MyModule()

# pass dummy input
dummy_input = torch.randn((2,2))
model(dummy_input)
</code></pre>
<p>Now i can load the state dict. Should I then call <code>model.to(device)</code>, and everything will work?</p>

ptrblck: <p>Yes, once all parameters and layers are created, <code>model.to()</code> and the optimizer setup should work.</p>
Visualizing the feature embedding

Angelina_Robert: <p>Hi, I have extracted the features of cifar10 and cifar100 datasets using Resnet 18. Feature embedding is stored in the file feature_embeddings.pth. Just wondering if any one can help to visualise the features using TSNE.</p>

ptrblck: <p>You could load the data via <code>torch.load</code>, transform it to a numpy array via <code>arr = tensor.numpy()</code>, and use e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html"><code>sklearn.manifold.TSNE</code></a> to visualize it.</p>

Angelina_Robert: <p>Thanks so much, Ptrblck for the help. I managed to do that as per your instructions, however, the displayed features appear on the corresponding class indices in a straight line (figure given below)   instead of clusters. Moreover, these features are from the 1st epoch and are generated at runtime. I have already an embedded feature file (feature.pth) for all epochs, could you please guide how to use this file to display the features in a clustered display.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a52384328221dcc79f70fb11d2b12f36168cdf44.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a52384328221dcc79f70fb11d2b12f36168cdf44" title="feats_disp"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a52384328221dcc79f70fb11d2b12f36168cdf44_2_690x348.jpeg" alt="feats_disp" data-base62-sha1="nySSjVizehUrHMkTVEv4DVDT3k8" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a52384328221dcc79f70fb11d2b12f36168cdf44_2_690x348.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a52384328221dcc79f70fb11d2b12f36168cdf44_2_1035x522.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a52384328221dcc79f70fb11d2b12f36168cdf44_2_1380x696.jpeg 2x" data-dominant-color="F3F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">feats_disp</span><span class="informations">1449×732 64.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Cheers.</p>

ptrblck: <p>I’m not sure what exactly is causing this behavior, but you might want to play around with hyperparameters in TSNE, as it could show different outputs based on its setup.<br>
E.g. the paper and conclusions mentioned in <a href="https://twitter.com/lpachter/status/1440695021502545934">this tweet</a> might be interesting for you to check out.</p>
What is the storage structure of computation graph, what is freed after backward() and how it is freed?

Zihao_Qin: <p>Hi, everyone, I’m a beginner of Pytorch. Now, I can only know "the computation graph is freed after <code>loss.backward()</code>" from some blogs, so computation graph is still a vague concept for me.</p>
<p>after I run this simple program:</p>
<pre><code class="lang-auto">w = torch.tensor([1.], requires_grad=True)
x = torch.tensor([2.], requires_grad=True)
a = torch.add(w, x)
b = torch.add(w, 1)
y = torch.mul(a, b)

y.backward()
</code></pre>
<p>I can still access <code>a</code> and <code>b</code>, so I think the nodes are not freed, but then what is freed and how?</p>
<p>And it couldn’t be better if you are willing to start from the storage structure.</p>
<p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>

Varal7: <p>Hi, you can use <code>torchviz</code> and look at the graph itself:</p>
<p><code>torchviz.make_dot(y, show_saved=True, show_attrs=True)</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dd42d3ba907e26c8579123525399d3f2a200451.png" data-download-href="https://discuss.pytorch.org/uploads/default/9dd42d3ba907e26c8579123525399d3f2a200451" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dd42d3ba907e26c8579123525399d3f2a200451_2_381x500.png" alt="image" data-base62-sha1="mwdy3BHgCU4LsENdRPrBrdtKahb" width="381" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dd42d3ba907e26c8579123525399d3f2a200451_2_381x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dd42d3ba907e26c8579123525399d3f2a200451_2_571x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dd42d3ba907e26c8579123525399d3f2a200451.png 2x" data-dominant-color="E2E1DC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">676×886 47.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When the computation graph is freed, the variables <code>other</code> and <code>self</code> in orange are freed.</p>
<p>You can learn more about how autograd works here: <a href="https://colab.research.google.com/drive/1aWNdmYt7RcHMbUk-Xz2Cv5-cGFSWPXe0#scrollTo=AHcEJ6nXUb7W" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>and here: <a href="https://pytorch.org/docs/master/notes/autograd.html" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch master documentation</a></p>

Zihao_Qin: <p>Thank you! It really helps!</p>
Pytorch Multiplication

iamPrashantSri: <p>Can I multiply two tensors of different dimensions?<br>
For example (64, 209, 50) and (50, 209, 50)?</p>

shashimalcse: <p>No you cant multiply those tensors</p>

RaLo4: <p>Tensors don’t necessarily have to be the same size for you to be able to multiply them, but the shapes of the two tensors has to be <a href="https://pytorch.org/docs/1.9.1/notes/broadcasting.html#broadcasting-semantics" rel="noopener nofollow ugc">broadcastable</a>.<br>
So your example tensors would not work.</p>