akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to combine tabular and image data?

username: mgloria
cooked: <p><strong>I am trying to build a classifier combining image data and tabular data.</strong>  So far, I have a trained a resnet50 on rgb images to predict one of 783 classes.</p>
<p>In addition to the image dataset, I also have some tabular data (.csv file) which I would like to use. Namely, the country and zones (~continent) in which the images were taken. There are 188 countries and 8 zones. Imagine these are vectors already, e.g. one-hot encoded. <strong>How could I add this information to the fully connected layer of resnet50 and fine-tune the model?</strong></p>
<p>I can draw the process in my head but I struggle to code it…  <strong>Which should be my next steps?</strong></p>

username: ptrblck
cooked: <p>I’m not sure, if you could add this new information to a linear layer in the resnet, as the shape might not fit and you might destroy the processing of the image features by retraining this layer.<br>
You could generally pass these additional features through a new model and concatenate the penultimate activations of this new model and the resnet before feeding them to a final classifier.<br>
This approach was done a couple of times by users in this board. Often the model tends to focus on only one feature branch and I’m not sure, if there is a good workaround besides normalizing the activation tensors to the same value range.</p>

username: mgloria
cooked: <p>interesting! thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> could you please add for me a link to some code e.g. from this other users? I believe this could help me a lot get things more clear.</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/concatenate-layer-output-with-additional-input-data/20462/2">Here</a> is a simple example.</p>
Pytorch (C++ API) + Emscripten?

username: noe
cooked: <p>How to run a C++ pytorch model in the browser via Emscripten?</p>
<p>I have found this PR: <a href="https://github.com/pytorch/pytorch/pull/9803" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/9803</a></p>
<p>From there, it seems that ATen can be compiled with emscripten with the patch in that PR. The code, however, is not in master.</p>
<p>In the PR, the author states that they did not compile the full libtorch (see also the <a href="https://medium.com/@giovanni_94706/almost-full-pytorch-experience-on-ios-by-nimtorch-cadf43b3863b" rel="nofollow noopener">blog post</a>). Has libtorch been compiled with emscripten before?</p>
<p>Also, <a class="mention" href="/u/apaszke">@apaszke</a>  suggests that their math code was actually very slow when ran in the browser. Does this mean that libtorch has actually been run in the browser? How?</p>
How to copy intermediate layers from a pretrained model

username: Umair_Javaid
cooked: <pre><code class="lang-auto">vgg16 = models.vgg16(pretrained=True).features[:16]
brach_A = vgg16.clone()
</code></pre>
<p>I get the following error on the above code:<br>
<code>**'Sequential' object has no attribute 'clone'**</code></p>
user_suspended: yes

username: albanD
cooked: <p>Hi,</p>
<p>This is because we don’t have a method to clone nn.Modules.<br>
If you want another ref to the same module, use b = a<br>
If you want a shallow copy, you can use the copy module from python<br>
And if you want a deepcopy, you can use the deepcopy module from python.</p>
How to get a couple of separate pretrained models?

username: Umair_Javaid
cooked: <pre><code class="lang-auto">vgg16 = models.vgg16(pretrained=True)
vgg16_ = models.vgg16(pretrained=True)
</code></pre>
<p>I want to know if <code>vgg16</code> and <code>vgg16_</code> are pointing to the same memory address? If yes, how do I get two separate vgg16 models? And, also how do I check this?</p>
user_suspended: yes

username: albanD
cooked: <p>Hi,</p>
<p>No they will be pointing to different memory.<br>
You can check that by checking the data_ptr() of the different weights and make sure they don’t match.</p>

username: Umair_Javaid
cooked: <p>Can you also answer the other question I just posted. Thank u</p>
user_suspended: yes
Prepare tensor for pack padded sequence

username: greedsin
cooked: <p>I was using torchtext for creating a text dataset based on a nested field with include length set to true</p>
<pre><code class="lang-python">FIELD= data.Field(lower=True, init_token=sos_token,eos_token=eos_token, tokenize=tokenize,  batch_first=batch_first,fix_length=max_length)
NESTED_FIELD=data.NestedField(FIELD,include_lengths=include_lengths)
</code></pre>
<p>which produces for example the following structure:</p>
<pre><code class="lang-auto">(torch.Size([2, 3, 10]),torch.Size([2]),torch.Size([2, 3]))
</code></pre>
<p>where the first entry in triplet contains the data. The second entry contains how many sequences are present in each batch and the third entry contains the information of how long the sequences are.</p>
<p>Consider the following examples with numbers:</p>
<pre><code class="lang-auto">tensor([[[ 2,  4,  5, 16,  3,  1,  1,  1,  1,  1],
         [ 2,  4,  5, 14,  3,  1,  1,  1,  1,  1],
         [ 2, 10,  3,  1,  1,  1,  1,  1,  1,  1]],

        [[ 2, 10,  3,  1,  1,  1,  1,  1,  1,  1],
         [ 1,  1,  1,  1,  1,  1,  1,  1,  1,  1],
         [ 1,  1,  1,  1,  1,  1,  1,  1,  1,  1]]])
tensor([3, 1]) --&gt; the first batch contains 3 sequences and the second batch contains only one sequence (1 is ignored)
tensor([[5, 5, 3], --&gt;the 3 sequences of the first batch have the sequence length 5,5 and 3
        [3, 0, 0]]) --&gt; the one sequence in batch two has a sequence length of 3
</code></pre>
<p>I want to change the structure so that I have the following format:</p>
<pre><code class="lang-auto">torch.Size([4,10]) --&gt; only the sequences that carry meaningful data (which are in the above case 4)
torch.Size([4]) --&gt; the length of each of this 4 sequences

</code></pre>
How much does 'optimizer object' depends upon the 'Loss object'?

username: Deepak
cooked: <ol>
<li>
<p>Does the <em>optimizer_object</em> anyhow depends upon the <em>Loss_object</em>, except that it wants <code>loss.backward()</code> to be computed before the <code>optimizer.step()</code>?</p>
</li>
<li>
<p>I am creating a <em>custom_Loss_object</em> every batch iteration (and pass it a parameter). Does it effects optimizer anyhow as compared to when I create the <em>custom_Loss_object</em> only once (before iteration loop) and use it in every iteration?</p>
</li>
</ol>
<p>(I guess no for both the questions.)</p>

username: KFrank
cooked: <p>Hello Deepak!</p>
<aside class="quote no-group quote-modified" data-username="Deepak" data-post="1" data-topic="79418" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deepak/48/18744_2.png" class="avatar"> Deepak:</div>
<blockquote>
<ol>
<li>
<p>Does the <em>optimizer_object</em> anyhow depends upon the <em>Loss_object</em> … ?</p>
</li>
<li>
<p>Does it effects optimizer anyhow as compared to when I create the <em>custom_Loss_object</em> only once … ?</p>
</li>
</ol>
<p>(I guess no for both the questions.)</p>
</blockquote>
</aside>
<p>As you surmise, the answers are “no” and “no.”</p>
<p>An easy way to see this is to remember that you never actually<br>
need to create an instance of a loss object.  For example, instead<br>
of instantiating <code>torch.nn.CrossEntropyLoss</code> (and calling it as a<br>
callable function object), you can simply call the function<br>
<code>torch.nn.functional.cross_entropy()</code>.</p>
<p>Or you can calculate your loss directly with pytorch tensor operations:<br>
for example, <code>MSELoss</code> (<code>mse_loss()</code>) can be implemented as:</p>
<pre data-code-wrap="python"><code class="lang-python">loss = ((target - input)**2).mean()
</code></pre>
<p>No loss objects needed, so the optimizer can’t be depending on a<br>
loss object.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Deepak
cooked: <p>Thank you KFrank, your second logic seems more convincing to me.</p>
Torch.cat two variables, one's requires_grad= True, another's requires_grad = False

username: cosmozhang1988
cooked: <p>If I use torch.cat to stack two variables, one’s requires_grad =True, another’s requires_grad=False, will these properties retain during .backward() and .step()</p>

username: cosmozhang1988
cooked: <p>It seems the answer is YES after I experimented a little.</p>

username: SimonW
cooked: <p>Yes, in general, if any input requires grad, so will the output.</p>

username: amobash
cooked: <p>Hi, did you also use default_collate function of torch in dataloader on stacked tensors with different requires_grad values? I am hitting a problem with creating batch on this kind of data?</p>
Conditional Model Architectures

username: collinarnett
cooked: <p>How can I make conditional model architectures?</p>
<p>I’m writing a GAN and currently I have two classes defined as:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    ...
class Discriminator(nn.Module):
    ...
</code></pre>
<p>but I want to have multiple architectures dependent on the size of my input eg:</p>
<pre><code class="lang-auto">if len(input) == 16:
    # Use 16x16 Discriminator and Generator

if len(input) == 64:
    # Use 64x64 Discriminator and Generator

...
</code></pre>
<p>Is there a best-practice way to do this? I could define classes for each of my inputs but that seems a bit inefficient.</p>

username: ptrblck
cooked: <p>If you would like to call to a specific model based on a condition, you would have to store these model instances somehow, no?<br>
I’m not sure I understand the use case correctly, so please correct me.<br>
Or would you like to use the same instance of your model with different paths inside the model?</p>

username: collinarnett
cooked: <p>My solution ended up being:</p>
<pre><code class="lang-auto">from architectures import (Discriminator16, Discriminator64, Discriminator128, Generator16, Generator64, Generator128)

 # Create the Discriminator
 if ndf == 16:
   netD = Discriminator16(ngpu).to(device)
 elif ndf == 64:
   netD = Discriminator64(ngpu).to(device)
 elif ndf == 128:
   netD = Discriminator128(ngpu).to(device)
...
</code></pre>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="79889">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If you would like to call to a specific model based on a condition, you would have to store these model instances somehow, no?</p>
</blockquote>
</aside>
<p>Yes I was just wondering if it there was a better way of doing it. BTW you do a great job on these forums, your answers have helped me greatly!</p>
Using GPU by default

username: Ali_Amiri
cooked: <p>Hi,<br>
for using GPU (Cuda) I should call to(device) on torch objects to put data into GPU<br>
so it is a little annoying.<br>
is there any way to use GPU accelerator by default without changing the device on each torch object?</p>

username: russellizadi
cooked: <p>By default, torch creates objects on CPU and you have to transfer whatever you need to other devices.</p>
<p>You can define a global variable like <code>dtype=torch.cuda</code> but you still need to specify that when making new objects.</p>
Confusion with Variable and tensor.requires_grad

username: nbravo
cooked: <p>Hello everyone,</p>
<p>I’m a little confuse about the correct use of Variable or tensor.requires_grad=True for an intermediate Tensor created between a trainable layer and the loss function. For example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


x = torch.rand((5, 10))
l1 = nn.Linear(10, 10)

y = l1(x)
first_two = Variable(torch.zeros((5, 2)))  # just get the first 2 values of each example of y (just an example)
# The intermediate tensor "first_two" should be initialized as Variable(torch.zeros((5, 2))) or
# Variable(torch.zeros((5, 2)), requires_grad=True) or just torch.zeros((5, 2), requires_grad=True)?
for i in range(y.size(0)):
    first_two[i] = y[i, :2]
loss = dummy_loss(first_two)
loss.backward()

</code></pre>
<p>I ask it to ensure the correct backpropagation to the trainable Linear layer.<br>
Thanks you very much in advance</p>

username: ptrblck
cooked: <p><code>Variables</code> are deprecated since PyTorch <code>0.4</code> so you should use tensors in newer versions.</p>
<p>That being said, I would just slice <code>y</code> directly and pass it to the loss function:</p>
<pre><code class="lang-python">y = l1(x)
first_two = y[:, :2]
loss = dummy_loss(first_two)
loss.backward()
</code></pre>

username: nbravo
cooked: <aside class="quote no-group" data-username="nbravo" data-post="1" data-topic="79928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/db5fbb/48.png" class="avatar"> nbravo:</div>
<blockquote>
<p>= Variable(torch.zeros((5, 2)))</p>
</blockquote>
</aside>
<p>I see, I will forget about Variables then</p>
<p>Thanks a lot</p>
How to access each layer of torchvision ResNet

username: sukhadj
cooked: <p>I am using ResNet18 model of torchvision.<br>
I would like to access each layer of resnet and change its weights to something.</p>
<p>Currently I am using something like this:</p>
<pre><code class="lang-auto">for (name, layer) in resnet._modules.items():
# assign new weights to layer.weights.
</code></pre>
<p>However, this is ignoring the layer1, layer2 or block1, block2. We can recursively find the most basic layer and assign new weights.</p>
<p>But is there any simpler way to do it?</p>
<p>So is there any way to do that.</p>

username: BarkenBark
cooked: <p>You could use the function <code>apply()</code> to recursively apply a function to the network, and each sub-layer. Calling <code>resnet.apply(weight_init_fun)</code> will apply the function <code>weight_init_fun</code> on every sub-layer, so make it a function which takes a torch.nn.Module, checks compability and changes its weights.</p>
<p>According to the documentation (<a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a>), this is a typical use case for this function.</p>
How to sort images in ImageFolder?

username: AntivistRock
cooked: <p>I have a trainset folder with different images:<br>
They named like <code>img0.jpg</code>, <code>img1.jpg</code> ---- <code>img990.jpg</code>. But the problem is that they lie in a different order.<br>
And when I use ImageFolder all this images loads in incorrect row.<br>
So is it able to use some torch coustructions to sort this images or to load them in a right row without custom datasets?</p>

username: ptrblck
cooked: <p>The file names will be sorted in <a href="https://github.com/pytorch/vision/blob/8c9585540a3e71f73ebc281445e888aa841824bd/torchvision/datasets/folder.py#L50">this line of code</a>, so I don’t think it’ll be easy to change this and would recommend to either use a custom <code>Dataset</code> or to prepend zeros to the image indices.</p>
Rerun a part of the computational graph

username: Nawak
cooked: <p>Hi,<br>
Is it possible to re-run just a part of a computational graph?</p>
<p>More details:<br>
Suppose I have a network f with m inputs and n outputs, where each output does not necessarily depend on all inputs. The simplest example would be a triangular matrix, though typical networks I wish to consider are much more complex.<br>
For an input x = (x_1, x_2… x_m), I can compute the output f(x) classically with a forward pass. Now, I would like to change just one of the inputs x_i, and recompute only the new j-th output f(x)_j, performing only useful computations, that is, not recomputing the nodes which still have the same inputs as before. As I need to perform many such operations (changing x coefficientwise), the difference in complexity (between running a full forward pass each time vs only the changed nodes) will be tremendous: quadratic vs linear.<br>
Note that I do not care about backpropagation here, I need only forward passes.<br>
So, questions:</p>
<ol>
<li>Is there a way to identify the subgraph (of the computational graph f) which contains only the nodes depending on x_i and having a consequence on f(x)_j ?</li>
</ol>
<p>With access to the computational graph, this should be doable in two passes (one forward and one backward), to know, for any internal node, on which input (resp. output) variables it depends (resp. have an influence on), by propagating sets of indices of input (resp. output) variables along the graph. In practice, I do not know how to do this with PyTorch.</p>
<ol start="2">
<li>Is there a way to re-run only this subgraph?</li>
</ol>
<p>(possibly extracting / cloning it first if re-running in place is not feasible…)</p>
<p>Thanks!</p>

username: emcp
cooked: <p>I thought computational graph was more of a tensorflow concept… in which the graph is a fixed thing that your data “flows” through… where as in pytorch… it’s “just python” so you can breakpoint or use python logic to basically stop or stop the network(s)… or am I misunderstanding your question?</p>

username: Nawak
cooked: <p>I suppose PyTorch does use a computational graph in autograd; but indeed the forward pass usually does not make use of it.<br>
I could define by hand new .forward() functions that would do the job and compute only what is needed, if I know which nodes to compute and if the network is simple; but for complex networks, I would have wished to do that automatically: take a network as input, identify the desired subgraph, and run it; for this I need the computational graph of the network.<br>
I guess I can have a look at the implementation of .backward() for inspiration, but that’s going to take a while…</p>
Unable to access my nets parameters

username: jacobhm98
cooked: <p>Hello! I am trying to train a convolutional neural network. I’ve created the network structure, and am instantiating an object of the network class. However, I am unable to access the parameters of the model, and I’m wondering if there’s something silly I’ve forgotten. I’m not the most comfortable with c++, so it’s possible I’ve made some ownership mistake or something. This also leads to the wealth of information accessible in the docs overloading my brain a little bit. Anyway, here is my network architecture, located in a header file:</p>
<pre><code class="lang-auto">class ConvNet : torch::nn::Module {
	public:
		ConvNet(){
			register_module("layer1", layer1);
			register_module("layer2", layer2);
			register_module("fc", fc);
		};
		torch::Tensor forward(torch::Tensor x){
			x = layer1-&gt;forward(x);
			x = layer2-&gt;forward(x);
			x = x.view({-1, x.size(1) * x.size(2) * x.size(3)});
			x = fc-&gt;forward(x);
			return x;
		};

	private:
		torch::nn::Sequential layer1{
			torch::nn::Conv2d(torch::nn::Conv2dOptions(3, 64, 3).stride(1).padding(1)),
			torch::nn::ReLU(),
			torch::nn::MaxPool2d(torch::nn::MaxPool2dOptions({2, 2}).stride(2))
			};

		torch::nn::Sequential layer2{
			torch::nn::Conv2d(torch::nn::Conv2dOptions(64, 64, 3).stride(1).padding(1)),
			torch::nn::ReLU(),
			torch::nn::MaxPool2d(torch::nn::MaxPool2dOptions({2, 2}).stride(2))
			};

		torch::nn::Sequential fc{
			torch::nn::Linear(torch::nn::LinearOptions(64 * 50 * 125, 64)),
			torch::nn::ReLU(),
			torch::nn::Linear(torch::nn::LinearOptions(64, 1)),
			torch::nn::Sigmoid()
			};
			
};

</code></pre>
<p>And here, in my implementation file is where I’m trying to view the parameters:</p>
<pre><code class="lang-auto">ConvNet cnn;
cout &lt;&lt; cnn.parameters();
</code></pre>
<p>This is leading to the errors:</p>
<p>std::vector<a>at::Tensor</a> torch::nn::Module::parameters(bool) const’ is inaccessible within this context<br>
58 |  cout &lt;&lt; cnn.parameters() &lt;&lt; endl;</p>
<p>and</p>
<p>‘torch::nn::Module’ is not an accessible base of ‘ConvNet’<br>
58 |  cout &lt;&lt; cnn.parameters() &lt;&lt; endl;</p>
<p>Any guidance would be much appreciated. Thank you!</p>

username: glaringlee
cooked: <p>try<br>
class ConvNet : public torch::nn::Module</p>
Batch sample with variable instance sizes

username: weidezhang
cooked: <p>I have a use case where each sample in a batch contains variable input instances. For example, an image with objects each has its own attributes needs to be encoded(shared network for object level encoding) separately and in the meanwhile need to concat cropped features from the image feature map from the object location. So there will be multiple object instances corresponds to one image. And one image is just one sample in batch.</p>
<p>What’s the best way to structure the input tensor?<br>
Images can be structured as NxCxHxW and goes through a cnn network to obtain feature map.<br>
What about object level tensors?<br>
Option 1:<br>
structure the objects tensor as NxCxLxS where N is batch number, C is channel, L is length of input of each object, S is the objects number in each sample ?<br>
or<br>
structure the objects tensor as MxCxL where M = N*S, and slice later based on start and end index to concat with image feature map?  Will autograd work in this case ?</p>
<p>What’s the recommendation here. Thank you a lot,</p>
How to make a ImageFolder using absolute image ways?

username: AntivistRock
cooked: <p>Hello, I am a bit new in pytorch, so I have a problem now with uploading images.</p>
<p>I want to do Over-Sampling with my data. So I decided to make a dictionary where key is class label and values are absolute ways to images of this class. Then I count amount of images of every class and if this amout &lt; 100 I increase the number of paths. But because of transforms this trick should work, I think. So, now I need to make an ImageFolder with images from this paths, but I, actually, don’t know how, because ImageFolder requires subfolders.</p>

username: ptrblck
cooked: <p>You could write a custom <code>Dataset</code> and lazily load each sample from your <code>dict</code> in the <code>__getitem__</code> method.<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> gives you an example on how to write a <code>Dataset</code>.<br>
Let me know, if that works for you.</p>

username: AntivistRock
cooked: <p>Am I right that when I will put CustomDataset into the <code>DataLoader</code>, DataLoader will raise <code>__getitem__</code> for every item in CustomDataset?</p>

username: ptrblck
cooked: <p>Yes, that is correct, if you are using the default sampling and collate mechanism.<br>
There are ways to provide a batch of indices, but this would be a special case.</p>

username: AntivistRock
cooked: <p>So, I found custom Dataset, that could be good for me.<br>
But in case of my task and using Colab, it’s better for me to increase amount of images in the folders of classes.<br>
I just use ImageFolder to create validation ImageFolder, then I activate function to copy missing pictures (because of it’s veeeeeery custom I wouldn’t post it without any reason <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> ) and then I create train Image folder.</p>
<p>But there some parts of that Dataset realisation, maybe it’ll be useful for someone:</p>
<p>At the beginning we <code>__init__</code> list of paths:</p>
<blockquote>
<p>…<br>
def <code>__init__</code>(self, files, mode):<br>
super().<code>__init__</code>()</p>
<pre><code>    # list of files to upload
    self.files = sorted(files)
</code></pre>
<p>…</p>
</blockquote>
<p>Then we define upload function:</p>
<blockquote>
<p>def load_sample(self, file):<br>
image = Image.open(file)<br>
image.load()<br>
return image</p>
</blockquote>
<p>Notice that you should use <code>from PIL import Image</code></p>
<p>And at the end we define <code>__getitem__</code>:</p>
<blockquote>
<p>def <code>__getitem__</code>(self, index):<br>
…</p>
<pre><code>    x = self.load_sample(self.files[index])
</code></pre>
<p>…<br>
return x</p>
</blockquote>
Backward incompatible changes in pytorch 1.5

username: angshine
cooked: <p>As <a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">release note of pytorch 1.5</a> says, <code>Tensor.clone</code> ,  <code>Tensor.to</code> ,  <code>Tensor.empty_like</code> , and similar functions preserve stride information instead of returning contiguous tensors. When migrating to pytorch 1.5, I met “RuntimeError: view size is not compatible with input tensor’s size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(…) instead.” But there is no more information provided to debug which part of the code raise this error. How should I debug this kind of error?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes I think it comes from there.<br>
Does this error happen during the forward or the backward pass?</p>
<ul>
<li>If forward, can you give the stack trace that shows which function is faulty?</li>
<li>If backward, you can enable the anomaly mode (torch.autograd.set_detect_anomaly(True)) to get a warning that will tell you which forward function is responsible for the error in backward. And you can send it here.</li>
</ul>

username: angshine
cooked: <p>Thanks! The error happens during backward pass. I found it by enabling the anomaly mode, it is caused by using <code>Tensor.zeros_like()</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/grimacing.png?v=9" title=":grimacing:" class="emoji" alt=":grimacing:"></p>

username: albanD
cooked: <p>Ho nice, then given the <a href="https://pytorch.org/docs/stable/torch.html#torch.zeros_like">doc</a>, you can pass <code>layout=</code> to this function and you want <code>torch.contiguous_format</code> (list <a href="https://pytorch.org/docs/stable/tensor_attributes.html#torch.torch.memory_format">here</a>) to make sure you can view it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Avoid gradients computing for specific operation?

username: ZdsAlpha
cooked: <p>I have auto-encoder. It has two parts encoder and decoder. Its training process is:<br>
latent_space = encoder(x)<br>
y = decoder(latent_space)<br>
latent_reconstruct = encoder(y)<br>
loss = l1 * F.mse_loss(x, y) + l2 * F.mse_loss(latent_space, latent_reconstruct)<br>
loss.backward()<br>
…<br>
Here I want to use encoder as lens (previously i was using VGG). I don’t want to gradients to flow through encoder when used second time.<br>
To make it simple, it can be achieved by copying whole encoder before applying it for second time. But my encoder part is very large and I can’t afford to copy it for every single iteration.</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you want to ignore a bunch of ops when the gradient is computed, you can wrap them into a no_grad block:</p>
<pre><code class="lang-auto">with torch.no_grad():
    latent_reconstruct = encoder(y)
</code></pre>
<p>Another way to do this is to explicitly break the graph so that the <code>y</code> used in the second encoding is not linked to the rest of the network:</p>
<pre><code class="lang-auto">latent_reconstruct = encoder(y.detach())
</code></pre>
<p>Both should work fine. You can choose the one that fits best in your code.</p>
Use two separate dataloaders or use slicing?

username: Umair_Javaid
cooked: <p>I have two tensors of shape <code>(16384,3,224,224)</code> each. I need to multiply these two together. Obviously these two tensors are too big to fit in GPU ram. So I want to know, how should I go about this, divide them in smaller batches using slicing <code>[i:i+btach_size,:,:,:]</code> or should I use two separate dataloaders? I am confused, how to use two different dataloader together?<br>
What would be the best way to do this?</p>
user_suspended: yes

username: russellizadi
cooked: <p>I suggest you to define your own dataset so that you can use only one loader while taking batches from both tensors. For example, you can use something like this:</p>
<pre><code class="lang-auto">import torch
size = 16
a = torch.randn(size,3,224,224)
b = torch.randn(size,3,224,224)

class Dataset(torch.utils.data.Dataset):
    def __init__(self, a, b):
        assert len(a)==len(b)
        self.a = a
        self.b = b
        
    def __len__(self):
        return len(self.a)
        
    def __getitem__(self, i):
        return self.a[i], self.b[i]

dataset = Dataset(a, b)
loader = torch.utils.data.DataLoader(dataset, batch_size=4)

for a, b in loader:
    print(a.shape, b.shape)
</code></pre>
<p>There might be better solutions though.</p>

username: Umair_Javaid
cooked: <p><a class="mention" href="/u/alband">@albanD</a> any suggestions</p>
user_suspended: yes

username: albanD
cooked: <p>Well if the data don’t fit on GPU, you’ll most likely spend more time sending the data to the GPU than doing actual compute. So I guess you can simply do this on the CPU.</p>
<p>If you do need the GPU, the proposal above looks ok. Maybe use IterableDataset if you have performance issues. (But again if you don’t do anything else with it but a simple mulitplicaition, it is expected that you spend all your time sending stuff to the GPU).</p>

username: Umair_Javaid
cooked: <p>Thank you so much! I was looking for exactly this answer. I’m new to deep learning, can you tell me more about when to use gpu and when to work on cpu alone with examples</p>
user_suspended: yes

username: albanD
cooked: <p>In general, you should use the GPU if you can perform most ops on the GPU (or at least keep transfer low). And you only perform ops on large Tensors.</p>
Import images to making my own dataset

username: misiak
cooked: <p>Hey guys i want to ask if someone know how to edit my code.<br>
I need to get the path to every folder contains of images.<br>
inside folder images is folder train and test. They have also some folders for example 01,02,03 and iside them is my images<br>
My code is able to get only one subfolder</p>
<pre><code class="lang-auto">import numpy as np
import cv2
import os

debug=False
num_classes = 5
input_shape=(480,360,3) 
X_train=[]
X_test=[]
Y_train=[]
Y_test=[]
dir='C:\\Users\\Michal\\data\\DB480maj\\images'
i=0
print('constants DONE')

for c in range(0, num_classes):
    dir_path=dir+'\\Train\\'+'01'.format(c)+'\\'
    
    file_list=os.listdir(dir_path)
    for f in file_list:
        path=dir+'\\Train\\'+'01'.format(c)+'\\'+f
        raw = cv2.imread(path)
        if raw is None:
            print(path)
            continue
        img = np.float32(cv2.resize(raw, (input_shape[0], input_shape[0])))/255
        X_train.append(img)
        temp=np.zeros(num_classes)
        temp[c]=1
        Y_train.append(temp)
print('train dataset DONE')
</code></pre>

username: emcp
cooked: <p>im not sure which format your label data is coming in as… but I am personally creating datasets using the COCO format… then overloading the library COCO dataloader with my transforms and paths to images etc. etc…</p>
Workaround for 'ValueError: Cannot assign non-leaf Tensor to parameter 'weight''

username: ra_schoonhoven
cooked: <p>This is related to <a href="https://discuss.pytorch.org/t/discussion-of-practical-speedup-for-pruning/75212/3" class="inline-onebox">Discussion of practical speedup for pruning</a></p>
<p>I am trying to set the weights of one model to another like so:</p>
<pre><code class="lang-auto">    mod = model.msd.msd_block # some module
    pruned_mod = pruned_model.msd.msd_block # some module
    it = 0 # this is because some layers can be removed, not important for question
    for l in req_layers:
        pruned_weight_name ="weight"+str(l)
        weight_name ="weight"+str(it)
        pruned_weights = getattr(pruned_mod, pruned_weight_name)
        weights = getattr(mod, weight_name)
        it2 = 0
        for c in req_channels[it]:
            weights[0,it2] = pweights[0,c]
            it2 += 1
        setattr(mod, weight_name, weights)
        it += 1
</code></pre>
<p>This throws the error:</p>
<pre><code class="lang-auto">ValueError: Cannot assign non-leaf Tensor to parameter 'weight0'. Model parameters must be created explicitly. To express 'weight0' as a function of another Tensor, compute the value in the forward() method.
</code></pre>
<p>I checked that the tensor <code>weights</code> is the one I want but without <code>setattr(...)</code> it does not update the weights in the model. Can anyone help me with a workaround?</p>
<p><strong>Background</strong><br>
For pruning speedup purposes, I have created a new class with fully pruned layers removed. In the remaining layers, the weight tensors are smaller than in the original. The new class keeps track of which non-pruned channels are inputs. The forward method etc. already works and there is a nice speed up! The only thing I need to do is set the kernel-weights to the original weights (which should be simple) but this is holding me back now.</p>

username: ra_schoonhoven
cooked: <p><strong>Solved</strong><br>
It turns out my check function was off by one. When you use <code>getattr()</code> to grab the weights, updating them with <code>weights = ...</code> will update the model weights. Not entirely sure why <code>setattr(...)</code> throws this error though.</p>
How to get gradients with respect to input and change input (rather than trainable vars) to minimize loss

username: Homa
cooked: <p>Hi all.<br>
I want to change the input so it minimizes the loss (rather than changing the trainable variables ). in other words i want update input instead of weight and bias.<br>
I use this code for iteration:</p>
<p>data_batch = Variable(data_batch, requires_grad=True)<br>
prediction = model(data_batch)<br>
loss = loss_function(prediction, fitness_batch)<br>
loss…backward(retain_graph=False)<br>
grad_input = data_batch.grad<br>
print(“grad”,grad_input)</p>
<p>grad_input is not none and have the value ,but input not change.<br>
I dont know what is my mistake. please help me .</p>

username: ptrblck
cooked: <p><code>Variables</code> are deprecated since PyTorch <code>0.4</code> so you should use tensors now.</p>
<p>You would have to pass the input tensor to an optimizer, so that it can update the input (similar like you pass the <code>model.parameters()</code> to an optimizer).<br>
The same work flow applies as usual, i.e. you would zero out the gradients, perform the forward and backward pass, and call <code>optimizer.step()</code> to update the input.</p>

username: Homa
cooked: <p>thanks for reply.<br>
I dont understand how to use tensor instead of Variable ? how to define ( requires_grad=True) for input?<br>
I have another question, when I want gradient with respect to input and not change W and B, do I set param.requires_grad = False or True.<br>
(for param in model.parameters():<br>
param.requires_grad = False )<br>
when I update w and b is true, but I dont know here.</p>

username: Homa
cooked: <p>I use:<br>
data_batch.requires_grad= True<br>
instead of<br>
data_batch = Variable(data_batch, requires_grad=True)<br>
is correct?</p>

username: Homa
cooked: <p>my problem was solved. thanks for your guidance.<br>
I used from it:</p><aside class="quote quote-modified" data-post="2" data-topic="64401">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-add-cosine-similarity-score-in-cross-entropy-loss/64401/2">How to add cosine similarity score in Cross Entropy Loss</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    Since you would like to maximize the cosine similarity, I would go with the first approach, as in the worst case, you’ll add 0.01 * 2 to the loss and in the best (trained) case, it will be 1 - 1 = 0. Here is a small dummy example of just rotating tensors: 
x = torch.tensor([[1., 1.]], requires_grad=True)
y = torch.tensor([[-1, 0.]])

criterion = nn.CosineSimilarity()
optimizer = torch.optim.SGD([x], lr=1.)

for _ in range(20):
    optimizer.zero_grad()
    loss = 1. - criterion(x, y)
    loss.ba…
  </blockquote>
</aside>
Random initialization of weights with torch.nn.init?

username: erm
cooked: <p>I need to write in PyTorch the equivalent to Python weights and bias:<br>
W1 = <strong>np.random.randn</strong>(n_x, n_h) *0.01<br>
b1 = <strong>np.zeros</strong> ((1, n_h))<br>
While it exists torch.nn.init.zeros for the bias, I don’t find the way to set random weights and how to multiply them by a constant like the option in Python…<br>
Here <a href="https://pytorch.org/docs/stable/nn.init.html" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.init.html</a> it doesn’t exist the option…</p>
<pre><code class="lang-auto">class Net(nn.Module): 
    
    def __init__(self):
        super(Net,self).__init__()      
        self.input_layer = nn.Linear(n_x, n_h)
*#         torch.nn.init.uniform_(self.input_layer.weight)*  #how to set this to random?? 
        torch.nn.init.zeros_(self.input_layer.bias)
        
        self.hidden_layer = nn.Linear(n_h,n_y) 
*#         torch.nn.init.uniform_(self.hidden_layer.weight)*
        torch.nn.init.zeros_(self.hidden_layer.bias)

        self.output_layer = nn.Linear(n_y,n_classes) 
      
    def forward(self, x):
        x = x.view(-1, dim)
        x = F.relu(self.input_layer(x)) 
        x = F.relu(self.hidden_layer(x))
        x = self.output_layer(x)
        return F.log_softmax(x, dim=1) 
</code></pre>

username: vainaijr
cooked: <p>do you mean using a normal distribution, it fill tensor with random numbers from a normal distribution, with mean 0, std 1, or we could specify mean and std, something like,</p>
<pre><code class="lang-auto">import torch, torch.nn as nn, seaborn as sns
x = nn.Linear(100, 100)
nn.init.normal_(x.weight, mean=0, std=1.0)
</code></pre>
<p>we could also see our distribution of weight matrix,</p>
<pre><code class="lang-auto">sns.distplot(x.weight.detach().numpy())
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2abda1f9552ec08326b04178775e8a294741836.png" alt="image" data-base62-sha1="nd3xzBl8QSOVe7wHKDFao5cgwxo" width="378" height="248"></p>

username: erm
cooked: <p>oh, I found the answer myself…<br>
It is achieved using normal, and adjusting the value of the standard deviation.<br>
torch.nn.init.normal_(self.fc1.weight, mean=0.0, std=0.01)</p>

username: erm
cooked: <p>I cannot see your answer <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: vainaijr
cooked: <p>I put my answer back, I suggested same thing. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to prepare .tsv file for Pytorch/trainloader

username: Kasper
cooked: <p>Hi - new to the forum and new to pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a .tsv file with 120 rows and 800 columns, each row is a sample and each column a feature. I want to prepare it for a neural network. I have read in the data using panda:<br>
dataset = pd.read_csv(“filepath”, sep=’\t’)</p>
<p>Next, I want to use a trainloader,<br>
torch.utils.data.DataLoader(dataset = dataset)</p>
<p>But I don’t know how to tell the trainloader which are my y values (1st column) and which are my input features (the rest of the columns)</p>
<p>I have also been looking into the torchtext.data.TabularDataset function, but can’t get it to work either.</p>
<p>All inputs are much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: sathvik_udupa
cooked: <p>Hi,<br>
Have you gone through the custom dataset pytorch docs?<br>
In <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this</a> website, they use a dataset-dataloader pipeline with the dataset being a custom class loading from a csv file, you can modify appropriately based on your .tsv file.</p>

username: Kasper
cooked: <p>Great, thanks a lot sathvik, that did the trick =)</p>
Guided GradCam- getting blank image

username: DJ_1992
cooked: <p>I am trying to implement Gradcam <a href="https://github.com/jacobgil/pytorch-grad-cam/blob/master/gradcam.py" rel="noopener nofollow ugc">https://github.com/jacobgil/pytorch-grad-cam/blob/master/gradcam.py</a> in Pytorch to visualise classification of Embryo Cells. Though I am getting a good result on my GradCam…<br>
Guided Back Propagation (gb), and guided gradcam gb are completely blank</p>
<p>I am sharing all the three images …<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c208d72a1d8e0c0dd6b5963712bd0e7f92156a7d.jpeg" alt="cam" data-base62-sha1="rGvzkMkvty5e6TFXn9xHLnd24SV" width="224" height="224"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dd1fd2bece26d5b396252be792c19f2141624ea.jpeg" alt="gb" data-base62-sha1="fFvWuWeYfCPfg9fmvoXsjL7x6PM" width="224" height="224"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6dd1fd2bece26d5b396252be792c19f2141624ea.jpeg" alt="gb" data-base62-sha1="fFvWuWeYfCPfg9fmvoXsjL7x6PM" width="224" height="224"></p>
<p>I would really appreciate it if anyone can help me with it…</p>
<blockquote>
<p>gb_model = GuidedBackpropReLUModel(model=model, use_cuda=True)<br>
gb = gb_model(input, index=target_index)<br>
gb = gb.transpose((1, 2, 0))<br>
cam_mask = cv2.merge([mask, mask, mask])<br>
cam_gb = deprocess_image(cam_mask*gb)<br>
gb = deprocess_image(gb)<br>
cv2.imwrite(‘gb.jpg’, gb)<br>
cv2.imwrite(‘cam_gb.jpg’, cam_gb)</p>
</blockquote>
<p>Code generating the gb.jpg and cam_gb.jpg<br>
Thank you in Advance</p>

username: ptrblck
cooked: <p>I’m not familiar with the linked repository, but could you try to use <a href="https://captum.ai/docs/introduction">Captum</a>?<br>
I think these methods are also implemented there.</p>
<p>CC <a class="mention" href="/u/narine_kokhlikyan">@Narine_Kokhlikyan</a> who will know, if Captum can be used for this use case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: DJ_1992
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for mentioning this tool, It looks great. Ill try to implement with it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Validation accuracy stuck at fixed number

username: arian
cooked: <p>I’have implemented the same network in Keras and I expect to get the exact same results in pytorch, but during training I see no progress. I suspect that my layers are not connected the way I expect them to be. When I check layer’s shape for layer in final.children() I only get one print which is  [(1,1)]</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(256, 1, 1)
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(14*14, 1)

    def forward(self, x):
        output1 = F.sigmoid(self.conv1(x))
        output2 = F.sigmoid(self.fc1(output1))
        return output2


class Final(nn.Module):
    def __init__(self):
        super(Final, self).__init__()
        pretrained_model = torch.hub.load('pytorch/vision:v0.6.0', 'densenet201', pretrained=True)
        model = MyModel()
        list_of_layers = list(pretrained_model.features)[:8]
        list_of_layers.extend(list(model.children()))
        self.final_model = nn.Sequential(*list_of_layers)

    def forward(self, x):
        outputs = list()
        for ii, model in enumerate(self.final_model):
            x = model(x)
            if ii == 8 or ii == 10:
                outputs.append(x)
        return outputs
</code></pre>

username: ptrblck
cooked: <p><code>list(model.children())</code> will only contain the child modules, i.e. <code>[Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1)),  Flatten(), Linear(in_features=196, out_features=1, bias=True)]</code> and will miss the functional calls, which are defined in its <code>forward</code> method (<code>F.sigmoid</code>).<br>
Wrapping models into an <code>nn.Sequential</code> container works, if you are just calling <code>nn.Modules</code> sequentially.</p>
<p>For your use case, you could probably append <code>model</code> to the <code>final_model</code> instead of its children.</p>

username: arian
cooked: <p>Thanks for your reply that clears something up for me, but how can I get output shape for every layer to make sure model works properly.</p>

username: ptrblck
cooked: <p>The output shape of some layers depends on the input shape (e.g. for convolutions) and the output shape might be unknown before passing the real inputs.</p>
<p>For the sake of debugging you could add print statements into your <code>forward</code> method and check the shape of each activation. Alternatively, you could also manually calculate the output shapes, if the input shapes are known in advance, but the <code>print</code> approach is often simpler. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: arian
cooked: <p>Tnx for your response.</p>
Problem in converting  pytorch model( pytorch_model.bin) to model.pt

username: pumpkinband
cooked: <p>working on colab<br>
Project (folder containing model)<br>
├── pytorch_model.bin<br>
├── bert_config.json<br>
├── vocab.txt<br>
├──model.ckpt-231879.meta<br>
├──model.ckpt-231879.index<br>
└── model.ckpt-231879.data-00000-of-00001</p>
<pre><code class="lang-auto">import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForNextSentencePrediction, BertForQuestionAnswering
PRE_TRAINED_MODEL_NAME_OR_PATH = 'Project '
model = BertForQuestionAnswering.from_pretrained( PRE_TRAINED_MODEL_NAME_OR_PATH, cache_dir=None, from_tf=False, state_dict=None)
torch.save(model.state_dict(), 'model.pt')
model.load_state_dict(torch.load('model.pt'))
model.eval()

</code></pre>
<p>ERROR :-  <code>RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]</code></p>
<p>i think i have done something wrong here please help.</p>
How to save features extracted by a layer of a CNN model, let say output of AVGPOOL layer of a pretrained Resnet modelt

username: zdd
cooked: <p>i meed to use SVM classifier instead of Softmax, so for that i want to create a feature matrix of the training set to apply PCA on this and then train the SVM classifier.<br>
i want to know how to save the extracted features by the avg pool layer of the pretrained Resenet  model to create a feature set of all the images in the training set to apply PCA and then SVM.</p>
Deployment in FP16? Calling model.half() only reduces memory by 7%

username: IsCoelacanth
cooked: <p>I’m using the standard ResNet50 from torchvision model with an added 512x128 FC layer. Calling <code>model.half()</code> on it only bring it down from 940mb to 870mb GPU usage. Shouldn’t there be a more significant reduction in memory usage?<br>
I call <code>torch.cuda.empty_cache()</code> after initializing the model. I’ve set <code>torch.backends.cudnn.benchmark = True</code> as well.</p>
<p>Same behavior on Tesla T4 and 2080Ti.</p>
<p>model:</p>
<pre><code class="lang-python">model = torchvision.models.resnet50(
                pretrained=False, progress=False)
model.fc = torch.nn.Sequential(
                torch.nn.Linear(self.model.fc.in_features, 512),
                torch.nn.Linear(512, 128),
                torch.nn.Linear(128, 1)
            )
## section to load weights ## 
if use_fp16:
    model.half()
model.to(compute_device)
model.eval()
torch.cuda.empty_cache()
</code></pre>

username: cyanM
cooked: <p>As far as I know, you should use tensorRT to get the real speedup and reduce gpu memory.</p>