akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the reason of this error

username: mohammadreza
cooked: <p>for epoch in range(2):    <span class="hashtag">#loop</span> over the dataset multiple times<br>
running_loss = 0.0<br>
for i,data in enumerate(trainloader , 0):<br>
inputs , labels = data<br>
optimizer.zero_grad()<br>
outputs = net(inputs)<br>
loss = criterion(outputs, labels)<br>
loss.backward()<br>
optimizer.step()</p>
<p>TypeError                                 Traceback (most recent call last)<br>
 in ()<br>
3    for i,data in enumerate(trainloader , 0):<br>
4      inputs , labels = data<br>
----&gt; 5      optimizer.zero_grad()<br>
6      outputs = net(inputs)<br>
7      loss = criterion(outputs, labels)</p>
<p>TypeError: ‘list’ object is not callable</p>

username: ptrblck
cooked: <p>Did you replace the <code>zero_grad</code> method with a <code>list</code> object before?<br>
If not, could you post an executable code snippet, so that we could have a look at this issue?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Compute Hessian-vector product for multiple vector in grad_outputs

username: Zhuoran_Liu
cooked: <p>Thanks in advance for your help and suggestions!</p>
<p>I’m wondering if there’s a way to compute the HVP where we may have multiple vectors to be right-multiplied by the Hessian, using a single call to <code>torch.autograd.grad</code>?</p>
<p>Concretely, suppose we are given a gradient vector <code>grad</code> w.r.t parameter <code>x</code>, we can compute the HVP by</p>
<pre><code class="lang-auto">torch.autograd.grad(grad, x, grad_outputs=v)
</code></pre>
<p>which will yield <code>Hv</code>, where <code>H</code> is the Hessian w.r.t x.</p>
<p>Suppose that, instead of a single <code>v</code>, we are given a sequence of tensors <code>(v1, v2, ... , vm)</code> such that each <code>vi.size() == grad.size()</code>, is there an efficient way to compute <code>(Hv1, Hv2, ... , Hvm)</code> efficiently, in a single call to <code>torch.autograd.grad</code> analogous to the computation of <code>Hv</code>?</p>
<p>A naive way seems to be feeding each <code>vi</code>'s into a single call to <code>torch.autograd.grad</code>, but I’m curious to hear if there’s a more efficient implementation. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: LeviViana
cooked: <p>I’m sorry but I’ve got no time to test the following idea, that I think it could work:</p>
<ul>
<li>Concatenate the model parameters <code>n</code> times (i.e. repeat)</li>
<li>v = torch.cat((v0, v1, …, vn)).view(1, -1)</li>
<li>Use this: <code>https://github.com/LeviViana/torchessian/blob/master/torchessian/__init__.py#L6-L27</code>
</li>
</ul>
<p>Even if it works, I’m not sure though that it would provide any performance gains.</p>
Group Norm vs Batch Norm

username: edshkim98
cooked: <p>Hello everyone, I am currently doing a project where I replaced batch normalization with group norm so that I can train in batch size 1.<br>
However, the model seems to fail only a specific data during training which did not happen during batch norm.<br>
For example, for my validation iou, it goes 0.9,0.91 and then suddenly 0.07 and the model does not seem to improve on this data during training. On the other hand, the model did not fail like this during batch norm training.<br>
I know there could be many reasons but I think this is due to changing batch norm to group norm or possibly using batch size 1. Is there a difference in group norm which could have caused the problem?? Also, could there be a solution to this?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>The issue might arise from the changed norm layers, but I haven’t seen a similar issue before.<br>
You could try to isolate the problematic sample and check the output as well as the internal stats of the norm layers to check, if these layers are responsible for the drop in accuracy.</p>
How can I convert keras tensor to pytorch?

username: Pann
cooked: <p>Please answer how keras tensor is converted into pytorch tensor?</p>

username: ptrblck
cooked: <p>If I’m not mistaken, Keras with TF as the backend (unsure if there are more supported backends anymore) uses <code>numpy</code> arrays as the input, so you could simply use <code>tensor = torch.from_numpy(array)</code>.</p>
Can I concat texts in neural network?

username: cmcai0104
cooked: <p>Normally, we can use texts to do classification. I have one idea need to use multiple texts to do it.</p>
<p>For example, I want to do classification by the news. But I need to put the day’s news together to do the classification. So I want to implement RNN between news in one day after implement RNN(LSTM or transformer) for each news(just the headlines).</p>
<p>Does it possible?</p>
How to avoid gradient vanish in pathwise derivative policy gradient

username: cmcai0104
cooked: <p>I want to train a pathwise derivative policy. But the output of my NN to be nan after about 5000 trainings. I guess because of the gradients’ vanish. How to solve my problem?</p>
<p>My NN is following:</p>
<pre><code class="lang-auto">class ACTOR_QVALUE(nn.Module):
    def __init__(self, input_size, hidden_size, action_size):
        super(ACTOR_QVALUE, self).__init__()
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=2,
                            bias=True, batch_first=True, dropout=0, bidirectional=False)
        self.hidden_mu1 = nn.Linear(hidden_size + action_size, 64)
        self.hidden_mu2 = nn.Linear(64, 32)
        self.hidden_mu3 = nn.Linear(32, 16)
        self.hidden_mu4 = nn.Linear(16, action_size)

        self.hidden_layer1 = nn.Linear(64 + action_size, 32)
        self.hidden_layer2 = nn.Linear(32, 8)
        self.hidden_layer3 = nn.Linear(8, 1)

    def forward(self, env_state, action_state, action=None, type='actor'):
        lstm_out, (h_n, c_n) = self.lstm(env_state)
        cat_layer = torch.cat((lstm_out[:, -1, :], action_state), 1)
        mu = F.relu(self.hidden_mu1(cat_layer))
        if type == 'actor':
            mu = F.leaky_relu(self.hidden_mu2(mu))
            mu = F.leaky_relu(self.hidden_mu3(mu))
            mu = torch.softmax(self.hidden_mu4(mu), dim=1)
            return mu
        else:
            cat_layer = torch.cat((mu, action), 1)
            q = F.leaky_relu(self.hidden_layer1(cat_layer))
            q = F.leaky_relu(self.hidden_layer2(q))
            q = self.hidden_layer3(q)
            return q
</code></pre>
<p>where env_state(shape: 250*120) and action_state(shape: 1*8) is the state infomations,  action means action whith shape 1*8. The NN output action if type=‘actor’, else output the q_value.</p>

username: iffiX
cooked: <p>Nan is not caused by gradient vanishing, more oftenly it is caused by updating a part of your model continuously using extremely large gradients, like something divided by zero. then Nan will ocurr in all parameters.</p>
<p>Try clipping your gradient using <code>torch.nn.utils.clip_grad_norm_</code>or debug your model gradients using <code>module.register_backward_hook</code></p>

username: cmcai0104
cooked: <p>thank you for your reply.</p>

username: cmcai0104
cooked: <p>Thank you for your advice.</p>
Training Conditional-GAN without classes

username: Filos92
cooked: <p>Hey everybody,<br>
I’m trying to set up a controllable GAN arcitecture, but i don’t want to use a class as the conditional input but two floating point variables (i’ts kind of an 2 angle dependend image deformation).<br>
I’ve tried it with a simple DCGAN with conditional inputs, with moderate to good results. To enhance this i want to try more complex arcitecures like BIGGAN.<br>
Unfortunately this structure is designed for classes<br>
Now i could remap my dataset to from floating point to classes, but i would end up with far more than 1000 classes.<br>
Do you think it is worth trying? Or do you have any idea how to modify the BIG-GAN arcitecture?<br>
I’ve also searched for research papers with “no-classes” conditioning but i did not found any. Do you know some?<br>
Best Greetings,<br>
Filos92</p>

username: Filos92
cooked: <p>Does no one have an idea? <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Custom quantization-aware-training with hooks

username: Naman-ntc
cooked: <p>Hi, I wish to implement a custom quantization aware training. There are two ways to do this</p>
<p>Consider modules B followed by module A. Say I wish to only quantize A_act</p>
<p><code>[A] ----[A_act]----- [B]--output-loss</code></p>
<p>Approach-1 -&gt;</p>
<ul>
<li>quantize A_act in forward pass (A_act_quant)</li>
<li>then compute the loss and gradient of output wrt loss</li>
<li>While backpropagating modify weights in module B using <strong>A_act_quant</strong>
</li>
<li>set <code>A_act_quant)_grad = A_act_quant)_grad</code>
</li>
</ul>
<p>Approach-2 -&gt;</p>
<ul>
<li>quantize A_act in forward pass (A_act_quant)</li>
<li>then compute the loss and gradient of output wrt loss</li>
<li>While backpropagating modify weights in module B using <strong>A_act</strong>
</li>
<li>set <code>A_act_quant)_grad = A_act_quant)_grad</code>
</li>
</ul>
<p>For implementing approach-2, I am using <code>torch.autograd.Function</code> and quantizing in forward function and copying gradients in the first approach.</p>
<pre><code class="lang-auto">class Binarizer(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return (input &gt;= 0).to(torch.FloatTensor)#, 1

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        return grad_input

A_act_quant = Binarizer.apply(A_act)
</code></pre>
<p>Can you explain how approach-1 can be implemented.</p>
<hr>
<hr>
<p>I think the following should be the way to implement approach-1 but am not sure (modify only the forward pass with a hook). <strong>If I modify forward pass output on a module how does backpropagation happen?</strong></p>
<pre><code class="lang-auto">class QuantizeHookIdentity(nn.Module):
    def __init__(self,):
        pass
    def forward(self, x):
        return x
quantmodule = QuantizeHookIdentity()
quantmodule.register_forward_hook(lambda module,input,output: (output&gt;=0).to(torch.FloatTensor))
A_act_quant = quantmodule(A_act)
</code></pre>
Jit hiccup with a 1.8.0 nightly build

username: KFrank
cooked: <p>Hello Forum!</p>
<p>I noticed that jit “quit working” on a particular use case with the<br>
1.8.0.dev20201203 nightly build.</p>
<p>Specifically, a function that showed an about 2x jit speed-up with<br>
version 1.6.0 shows no speed-up with version 1.8.0.</p>
<p>Details can be found in this post:</p>
<aside class="quote quote-modified" data-post="3" data-topic="105274">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-implement-efficient-element-wise-ax-b-function/105274/3">How to implement efficient element-wise 'ax+b' function</a> 
  </div>
  <blockquote>
    Hi Zhi and Alex! 

The short story is that I can reproduce your results. 
I’ve also done some further experimentation. 

The best I could come up with was to apply Alex’s torchscript suggestion 
to your “w-b” version. 
I also tried a Conv2d version – even though it doesn’t have 
BatchNorm2d’s normalization step, it was also slower. 
Note, torchscript gave a nice speedup with pytorch version 1.6.0, but 
not with a version 1.8.0 nightly build. 
I used your original problem – same-shaped tensors, a…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: Michael_Suo
cooked: <p>Thanks for the report! Do you mind making an issue on Github with the same repro? We changed optimizers in that time period so I suspect it may be a regression with a new optimizer but we need to check.</p>

username: KFrank
cooked: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Michael_Suo" data-post="2" data-topic="105346" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/7993a0/48.png" class="avatar"> Michael_Suo:</div>
<blockquote>
<p>Do you mind making an issue on Github with the same repro?</p>
</blockquote>
</aside>
<p>Let me apologize for not creating a github issue.  It turns out I would<br>
need a github account to do so, which I don’t have.  It might be fastest<br>
if someone with a github account could link to my post, or something.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Michael_Suo
cooked: <p>Sure! I’ll make one in the morning and update you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Combine Classes

username: Angelina_Robert
cooked: <p>Hi,<br>
I have 2 categories of classes Category 1 and Category 2. Category 1 contains 6 classes and category 2 contains 4 classes I have extracted the features of both categories with dimensions of 3,128 and features of category 1  classes are stored at index=0,1,…5 while features of category2  classes are stored at index=6,7,8,9. Could u plz guide how to combine the features of all classes in a single variable?<br>
It should look be like:<br>
all_features=f1,f2,…f10</p>
<p>Cheers</p>

username: ptrblck
cooked: <p>I’m not sure how the mentioned shape of <code>[3, 128]</code> fits the indices in <code>[0, 10]</code> from the description.<br>
However, if you want to concatenate tensors, you could use <code>torch.cat</code> or <code>torch.stack</code>, if you want to add a new dimension.</p>

username: Angelina_Robert
cooked: <p>Hi ptrblck,<br>
Using troch.stack as:<br>
all_classes=torch.stack( head_features_var[0], head_features_var[1], head_features_var[2], head_features_var[3], head_features_var[4], head_features_var[5], head_features_var[6], tail_features_mean_var[7],tail_features_mean_var[8],tail_features_mean_var[9])</p>
<p>generates the following error:</p>
<p>TypeError:  stack() takes from 1 to 2 positional arguments but 10 were given</p>

username: ptrblck
cooked: <p>Try to wrap it in a <code>tuple</code> or <code>list</code> as:</p>
<pre><code class="lang-python">all_classes=torch.stack((head_features_var[0], head_features_var[1], head_features_var[2], head_features_var[3], head_features_var[4], head_features_var[5], head_features_var[6], tail_features_mean_var[7],tail_features_mean_var[8],tail_features_mean_var[9]))
</code></pre>

username: Angelina_Robert
cooked: <p>Heaps of thanks ptrblck</p>
<p>Cheers</p>
RuntimeError for chunk inplace operation, new with torch 1.7

username: bivouac0
cooked: <p>With the update to torch 1.7 I now get the following error…<br>
<em>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [24, 129, 1536]]</em></p>
<p>The error is created by this line…<br>
<code>return self._in_proj(query).chunk(3, dim=-1)</code><br>
and can be (temporarily) fixed with…<br>
<code>return self._in_proj(query).unsafe_chunk(3, dim=-1)</code></p>
<p>My understanding is <code>unsafe_chunk</code> will be removed in the future.  Is there a “correct” fix for this?  This isn’t code I wrote and I’m unsure the proper way to get the same behavior as the original.</p>
<p>Is this an appropriate replacement or is there something more elegant?</p>
<pre><code class="lang-auto">proj = self._in_proj(query)
sz   = proj.size()[2] // 3
return proj[:,:,:sz], proj[:,:,sz:2*sz], proj[:,:,2*sz:]
</code></pre>
PyTorch with GPU support on macOS 10.13

username: BrentDeHauwere
cooked: <p>Is it possible to install PyTorch with GPU support on macOS 10.13 (MacBook Pro Mid 2014, with NVIDIA GeForce GT 750M 2048 MB)?<br>
I installed CUDA 10.2 and Xcode 10.1, and followed the instructions to built from source.<br>
However, during the compilation, I get the following error:</p>
<pre><code class="lang-auto">.............

/Users/brentdehauwere/pytorch/aten/src/ATen/native/cuda/BatchLinearAlgebraLib.cu(80): error: more than one constructor applies to convert from "long" to "c10::Scalar":
            function "c10::Scalar::Scalar(uint8_t)"
            function "c10::Scalar::Scalar(int8_t)"
            function "c10::Scalar::Scalar(int16_t)"
            function "c10::Scalar::Scalar(int)"
            function "c10::Scalar::Scalar(int64_t)"
            function "c10::Scalar::Scalar(float)"
            function "c10::Scalar::Scalar(double)"
          detected during instantiation of "void at::native::apply_batched_inverse_lib&lt;scalar_t&gt;(at::Tensor &amp;, at::Tensor &amp;, at::Tensor &amp;) [with scalar_t=c10::complex&lt;float&gt;]" 
(115): here

24 errors detected in the compilation of "/var/folders/tf/ytkxrs5n31qdrvz9_2prqkm00000gn/T//tmpxft_000059da_00000000-12_BatchLinearAlgebraLib.compute_75.cpp1.ii".
CMake Error at torch_cuda_generated_BatchLinearAlgebraLib.cu.o.Release.cmake:281 (message):
  Error generating file
  /Users/brentdehauwere/pytorch/build/caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/./torch_cuda_generated_BatchLinearAlgebraLib.cu.o
</code></pre>
<p>And the compiling ends with this:</p>
<pre><code class="lang-auto">ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 773, in &lt;module&gt;
    build_deps()
  File "setup.py", line 315, in build_deps
    build_caffe2(version=version,
  File "/Users/brentdehauwere/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
    cmake.build(my_env)
  File "/Users/brentdehauwere/pytorch/tools/setup_helpers/cmake.py", line 346, in build
    self.run(build_args, my_env)
  File "/Users/brentdehauwere/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/Users/brentdehauwere/opt/anaconda3/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 1.
</code></pre>
Mixte optimizer

username: Jean-Eric_Campagne
cooked: <p>Dear experts,</p>
<p>Is it possible to use Adam for a set of parameters of the model and for an other set of parameters to use SGD for instance?</p>
<p>Thanks</p>

username: JuanFMontesinos
cooked: <p>Yes, you just need to set several optimizers passing the corresponding parameters and everything else is the same.</p>

username: Jean-Eric_Campagne
cooked: <p>Well, so imagine I define<br>
optim1= SGD(set1 of model parameter)<br>
optim2= Adam(set2 of model parameter)<br>
what about the scheduler, can I define two of them like<br>
sched1=StepLR(optim1)<br>
sched2=ReduceLROnPlateau(optim2)<br>
Thanks</p>

username: JuanFMontesinos
cooked: <p>Yes, each LR has a parameter called param_groups. Basically each group of parameters you pass to the each optimizer will be taken as a group. Schedulers work over these groups.</p>
Autoencoder converges quickly but reconstructions are not good

username: Akash_Gupta
cooked: <p>I am new to training autoencoders and trying to work with an unlabeled medical histopathology whole slide image (WSI) dataset and want to visualize possible clusters. The slide image contains a normal tissue region, background, and an abnormality region that I am interested in. I extract patches of fixed size(256x256) from the WSI (2400x1600) and train a convolutional autoencoder. After training, I extract hidden layer (of dimension 32) and perform t-SNE on these feature vectors. But, I am not getting any distinct clusters. A weird thing that I noticed while training is that my binary cross-entropy loss converges within just a few epochs but the reconstructions are really bad. I don’t understand why is that happening. To verify my training, I also tested my model on MNIST and it forms good clusters. Would appreciate any thoughts on this plus any other suggestions on how good is this approach. Below are the training loss, validation loss and reconstructed images.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02fd3744cd4e3332ec2d0138fff6d2d5a9e2172a.png" alt="Screenshot from 2020-12-06 22-39-49" data-base62-sha1="qrsZ2a3PThLabRBUA7PHn6a92q" width="479" height="300"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31ffbd66d3304cd6d84d9034bfb1ca45c610e6ec.png" alt="Screenshot from 2020-12-06 22-40-09" data-base62-sha1="78jjBinWYEjat1TbRkWnlqtRDDS" width="478" height="299"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d6fba1293d14f0094787608534f2676e39699de.jpeg" alt="Screenshot from 2020-12-06 22-39-57" data-base62-sha1="dkzMge9AlzpPJTVPn8L1J16T5Dg" width="416" height="208"></p>
Set New LR dynamically

username: Jaideep_Valani
cooked: <p>I want to set LR during training cycle based on the validation loss.<br>
Hybrid of One cycle Scheduling and  Fixed LR.<br>
Whenever i see there is no improvement in Validation Metric beyond certain LR,i want to stop making a step for scheduler and just put preferred LR at that point.</p>
<p>Let me know how can we achieve this</p>
<p>Currently I am doing like this</p>
<pre><code class="lang-auto">if scheduler.get_last_lr()[0]&gt;3e-3 or epoch &lt; int(0.35*48):
            
            scheduler.step()
        else:
            print('Not taking step lr at ',scheduler.get_last_lr()[0])
</code></pre>
<p>I use above method currently.but the problem is if LR is just close to 3e-3 then taking a step reduces to the LR at which there is no improvement in Val metric.<br>
So i just want to set LR to the point till where was metric improving last.</p>
Combine multiple datalaoders for Multi Task Learning

username: shahensha
cooked: <p>I want to implement a simple form of multi-task learning. Let us say there are two tasks A and B. I want to create a dataloader such that the batches alternate between these tasks i.e. one batch should only  contain sample from a single task. The first approach that I am trying is to create a dataloaders for each task in the usual way and then combine them using a MultitaskDataLoader. A POC implementation is as follows:</p>
<pre><code class="lang-python">
class MultitaskDataLoader(torch.utils.data.DataLoader):

    def __init__(self, task_names, datasets):
        self.task_names = task_names
        self.lengths = [len(d) for d in datasets]
        self.iterators = [iter(d) for d in datasets]
        indices = [[i] * v for i, v in enumerate(self.lengths)]
        self.task_indices = sum(indices, [])

    def _reset(self):
        random.shuffle(self.task_indices)
        self.current_index = 0

    def __iter__(self):
        self._reset()
        return self

    def __len__(self):
        return sum(self.lengths)

    def __next__(self):
        if self.current_index &lt; len(self.task_indices):
            task_index = self.task_indices[self.current_index]
            task_name = self.task_names[task_index]
            batch = next(self.iterators[task_index])
            new_batch = (batch, task_name)
            self.current_index += 1
            return new_batch
        else:
            raise StopIteration

task_names = ["A", "B"]
d1 = ['task-A'] * 5
d2 = ['task-B'] * 10
dl = MultitaskDataLoader(task_names, [d1, d2])
</code></pre>
<p>This works as expected but it stops after every epoch. I have to create a new dataloader object when every epoch starts. We do not have to do that for the standard dataloader torch.utils.data.dataloader? Then why do I have to do it for this? What should I change in this one to make it work exactly like the standard one?</p>

username: shahensha
cooked: <p>I actually found my mistake. I should be initializing the iterators in <strong>iter</strong> function because that is what a for loop calls every time. I will leave this question to help others who might have a similar problem.</p>
I have problems applying transforms

username: blacksprigann
cooked: <p>Hi, I am a beginner in pytorch and I have some problems applying transforms on my dataset, can I get some suggestions ? <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>train_transforms = transforms.Compose([<br>
transforms.Resize(input_size),<br>
transforms.RandomResizedCrop(input_size),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>
<p>train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))</p>
<p>train_dataset = torch.utils.data.Subset(train_dataset, indices = np.arange(train_dataset.tensors[0].shape[0]))</p>
<p>train_dataset.dataset.transform = train_transforms</p>
<p>‘’‘here I have another similar val_load but I omit it to simplify the reading’’’<br>
train_load = torch.utils.data.DataLoader(train_dataset.dataset, batch_size = batch_size, shuffle = True)<br>
dataloaders = {‘train’: train_load, ‘val’:val_load}</p>
<p>What you should know is that I am using tensordataset because I don’t have images, and I am using subset so I can apply the transforms. Any ideas ?</p>

username: ptrblck
cooked: <p>Some transformations would already work on tensors in the latest <code>torchvision</code> release.<br>
However, you could use <code>transforms.ToPILImage()</code> to transform the input tensors to <code>Image</code>s first, apply the transformations, and transform them back to tensors.<br>
I’m not sure, if this is the issue or where you are stuck exactly.</p>
<p>PS: you can add code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: blacksprigann
cooked: <p>Thank you for the answer and for the trick too <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Use a linear layer in-place

username: sidm
cooked: <p>Is it possible to use a linear layer (with the same input and output size) in-place? I don’t care about the gradients (torch.no_grad() is enabled). I want to use as little memory as possible because I’m querying the network many thousands of times per batch item (working with 3D point clouds).</p>

username: sidm
cooked: <p>I created a method to do this by looking at the nn.Functional.linear function:</p>
<pre><code class="lang-auto">  def linear_inplace(layer, v):
    return torch.addmm(layer.bias, v, layer.weight.t(), out=v)
</code></pre>
<p>However, for some reason my model’s layers are not moved to the GPU if I use this method instead of the standard <code>__call__</code> method of nn.Linear. I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: Tensor for argument #3 'mat2' is on CPU, but expected it to be on GPU (while checking arguments for addmm)
</code></pre>
<p>I know my cuda device is properly selected because it works with layer(v), but not with my linear_inplace(layer, v). Can someone help me understand what’s going on here?</p>

username: sidm
cooked: <p>I’ve tracked the error to a problem with weight_norm, and <a href="https://github.com/pytorch/pytorch/issues/48898" rel="noopener nofollow ugc">opened an issue.</a></p>
RuntimeError: expected scalar type Long but found Float (conv1d))

username: kaas
cooked: <pre><code class="lang-auto">class text_CNN(nn.Module):
  def __init__(self):
    super(text_CNN, self).__init__()
    self.conv1 = nn.Conv1d(in_channels=1, out_channels=10, kernel_size=2)

  def forward(self, x):
    print(x.type())
    x = F.max_pool1d(F.relu(self.conv1(x)), 2)
    return x

model = text_CNN()
x = torch.randint(2, (16, 1, 22))
model(x)
</code></pre>
<p><strong>Output:</strong></p>
<pre><code class="lang-auto">torch.LongTensor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-27-f0bfac495e81&gt; in &lt;module&gt;()
     11 model = text_CNN()
     12 x = torch.randint(2, (16, 1, 22))
---&gt; 13 model(x)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

&lt;ipython-input-27-f0bfac495e81&gt; in forward(self, x)
      6   def forward(self, x):
      7     print(x.type())
----&gt; 8     x = F.max_pool1d(F.relu(self.conv1(x)), 2)
      9     return x
     10 

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--&gt; 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    257                             _single(0), self.dilation, self.groups)
    258         return F.conv1d(input, self.weight, self.bias, self.stride,
--&gt; 259                         self.padding, self.dilation, self.groups)
    260 
    261 

RuntimeError: expected scalar type Long but found Float
</code></pre>
<p>However, when I change the type to a float:</p>
<pre><code class="lang-auto">x = torch.randint(2, (16, 1, 22), dtype=torch.float)
</code></pre>
<p>It works! Why is this?</p>

username: ptrblck
cooked: <p>PyTorch expects the input tensor and model parameters to have the same <code>dtype</code> and thus raises the error. <code>torch.randint</code> returns a <code>LongTensor</code>, while the model parameters are initialized as <code>FloatTensor</code>s, which is why you need to change the input to <code>torch.float</code>.</p>

username: kaas
cooked: <p>I see, but why is the error message stating that it is expecting scalar type long, while it is actually expecting a type float?</p>

username: ptrblck
cooked: <p>The error message might be confusing as the data type mismatch points to one of the used tensors. Based on the error it seems that the first tensor (<code>input</code> in this case) is used as the expected type.</p>
Weight parameter in nn.CrossEntropyLoss

username: oasjd7
cooked: <ol>
<li>
<p>If the accuracy for each class of my model is [0.7, 0.8, 0.2], can I improve the performance of the last class by applying the weight as [1,1,2] in cross-entropy ?</p>
</li>
<li>
<p>When I applied the weight as [1,1,2], I thought the loss would be different from that of ces1. But it gives the same value. Why is that?</p>
</li>
</ol>
<pre><code class="lang-auto">ces1 = nn.CrossEntropyLoss(reduction='none')

class_weights = [1,1,2]
class_weights = torch.FloatTensor(class_weights).cuda()
ces2 = nn.CrossEntropyLoss(weight=class_weights, reduction='none')
Compute with ces1 : [0.3, 0.4, 0.4]
Compute with ces2 : [0.3, 0.4, 0.4]
</code></pre>

username: ptrblck
cooked: <ol>
<li>
<p>The weighting should balance the losses and might thus also balance the accuracy. I.e. while the class2 accuracy might increase the other two might decrease.</p>
</li>
<li>
<p>Your current target might not use the class2 index. This code shows the difference:</p>
</li>
</ol>
<pre><code class="lang-python">class_weights = torch.tensor([1., 1., 2.])
ces1 = nn.CrossEntropyLoss(reduction='none')
ces2 = nn.CrossEntropyLoss(weight=class_weights, reduction='none')

x = torch.randn(2, 3)
target = torch.tensor([0, 2])

loss1 = ces1(x, target)
&gt; tensor([0.2583, 0.1567])
loss2 = ces2(x, target)
&gt; tensor([0.2583, 0.3133])
</code></pre>
Why does backward and forward function of an operator allocate new tensors?

username: matthias.l
cooked: <p>Hi, looking at the example <a href="https://github.com/pytorch/extension-cpp/blob/master/cuda/lltm_cuda_kernel.cu" rel="noopener nofollow ugc">extension-cpp</a>.</p>
<p>I am wondering why you allocate memory for the output tensors each time you call forward or backward.<br>
Sure, for the forward case you could provide an output tensor.<br>
But for the backward case this is not possible without bad hacks.</p>
<p>Does PyTorch internally take care that old output tensors get reused, or is this not a big issue regarding performance.<br>
My forward benchmark is 1.5% faster without reallocating a new tensor.</p>

username: ptrblck
cooked: <p>PyTorch uses a caching allocator and reused already allocated memory.<br>
I think the code tries to focus on readability of the code and as you said, these small perf. gains are sometimes not possible without bad hacks.</p>
Grad-cam for shufflenetv2

username: Hemantr05
cooked: <p>I am working with ShuffleNetv2, used transfer learning to perform classification and would like to know how to extract or plot the Grad-Cam for the same.</p>
<p>Thanks in advance. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ptrblck
cooked: <p><a href="https://captum.ai/api/guided_grad_cam.html">Captum</a> provides a <code>Guided GradCam</code> implementation, which could be used for your model. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: anil_sarode
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Can I use  <code>Guided GradCam</code> provided by <em>Captum</em> for GoogleNet along with transfer learning. Thanks !!</p>

username: ptrblck
cooked: <p>I think this should be possible and don’t know if there would be limitations to specific use cases.<br>
Are you seeing any issues?</p>
ImageNet example does not honor bucket_cap_mb

username: pro-panda
cooked: <p>Running the <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="noopener nofollow ugc">imagenet example</a> on a single node, 4-GPU setup calls 3 NCCL AllReduce ops per mini-batch for gradient synchronization, with sizes <code>2052000</code>, <code>28852224</code>, and <code>15853824</code> bytes.</p>
<p>I assumed that each op will follow the <code>bucket_cap_mb</code> limit, i.e. none of the allReduce would have sizes more than say, 25 MB (default)<br>
Am I missing something?</p>
Loss function for a Convolutional Autoencoder - Sparse data

username: stases
cooked: <p>Hello!</p>
<p>I am having troubles with building a convolutional autoencoder. The main issue is that all weights are learned to be equal to zero really quickly. This is reasonable, due to the fact that the images that I’m using are very sparse. Two examples can be seen below (it’s actually just a 2D tensor, but I’m showing it here as a heatmap):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbce0c5c061fc031b5db11dfbb72d67042074bd3.png" alt="hist1" data-base62-sha1="vmu1odOyI1BFzpXZBnuyA7nt44z" width="435" height="324"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70b13c470b82720e0fc4a873016cbf0cf770e7ef.png" alt="hist3" data-base62-sha1="g4Vcw8OPy6CJmyHqfXDfB0LQMcT" width="429" height="313"></p>
<p>The question is, what kind of loss function would be good for such case? L1 is bad, because my network settles in wrong minima (just multiplies the input by zero). Intuitively, I wish to penalize false negatives, they are of bigger importance for my case than false positives. Also I’ve tried variety of different autoencoders, none of which work. I’ve also used my network to train MNIST/CIFAR10 and it works well enough. Thanks in advance!</p>