akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I got NotImplementedError when trying to create __getitem__ in DataSet class:

Bahaa_Kattan: <p>I’m trying to solve Cat VS Dog classification problem using pytorch.  So I started by creating  a DataSet class using the following code:</p>
<pre><code class="lang-auto">import torchvision.transforms as transforms
import torch

from torch.utils.data import Dataset, DataLoader
import cv2, os

torch.manual_seed(0)
class dogVScat(Dataset):
    def __init__(self, data_dir, transform):
        file_names = os.listdir('train')
        self.full_file_names = [os.path.join('train', file_name) for file_name in file_names]
        self.labels = []
        for file in os.listdir('train'):
            if file.split('.')[0] == 'cat':
                self.labels.append([1, 0])
            else:
                self.labels.append([0, 1])
        self.labels = torch.tensor(self.labels)
        self.transform = transform

        def __len__(self):
            return len(os.listdir('train'))

        def __getitem__(self, idx):
            image = cv2.imread(self.full_file_names[idx])
            image = self.transform(image)
            return image, self.labels[idx]

transformer = transforms.Compose(transforms.ToTensor())

dataset = dogVScat('train', transformer)
image, label = dataset[2]
</code></pre>
<p>But when I run the code I face this problem:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:/Users/BHAAK/Desktop/ML_PATH/dirty-hands/dirty-hands file 3/Project.py", line 30, in &lt;module&gt;
    image, label = dataset[2]
  File "C:\Users\BHAAK\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataset.py", line 25, in __getitem__
    raise NotImplementedError
NotImplementedError 
</code></pre>
<p>Please if any one can help me fix this problem  I will appreciate it…</p>

ptrblck: <p>The indentation of <code>__getitem__</code> and <code>__len__</code> is wrong, as both methods are defined inside the <code>__init__</code> function.<br>
Move them one level to the left and it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to apply affine layers fast?

KaiHoo: <p>Hello, I want to do the affine operation to a 4D tensor, say X with a shape of (B,C,H,W). The weight’s shape is (1,C,1,1). The reason of doing this is that I want to remove normalization but keep the affine layer. Basically, it is part of the BN’s operation and should not be slow. However, without these affine layers I can train ImageNet on a 8-GPU machine with 9.6 hours. With affine layers added, the time is 12.7 hours, which is even slower than BN models.</p>
<p>It surprised me that an operation that does not count into FLOPs takes so much time. I am wondering if I can do any optimization to accelerate it? Thanks!</p>

albanD: <p>Hi,</p>
<p>How do you implement these transformation? Do you just do <code>inp * weight + bias</code> ?</p>

KaiHoo: <p>Yes, I just do what inp * weight where inp’s shape is (B,C,H,W) and weight’s shape is (1,C,1,1) (I do not add a bias in the network)</p>

albanD: <p>That will just do one multiplication, I don’t expect any performance issue with this (any more than any operation that does an op over the whole Tensor).</p>

KaiHoo: <p>I think this operation should run faster than the entire BN operation? However, the actual run time shows this multiplication is slower than the entire BN operation.</p>

albanD: <p>How do you check that. I don’t seem to see that when running on CPU:</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: a = torch.rand(2, 3, 50, 50)

In [3]: b = torch.nn.BatchNorm2d(3)

In [4]: scaling = torch.rand(1, 3, 1, 1)

In [5]: %timeit b(a)
115 µs ± 652 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [6]: %timeit a * scaling
5.15 µs ± 661 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>

KaiHoo: <p>Yes, this is where I find weird. I found it by training on ImageNet with the ResNet50 network. I recorded the time using BN and only the affine layer. But the results are not consistent with the CPU comparison: models with BN run faster than models with affine layers only on 8 V100 GPU. I tried both DataParallel and DDP mode.</p>

KaiHoo: <p>Hello, I got to know the reason why my network is slower. Sorry for this post!</p>
Conversion from pytorch (python) to libtorch (C++) meet error no viable overloaded =

Chen0729: <p>Hi, All, I have an inquiry about conversion from pytorch to libtorch (C++)</p>
<p>The python code is:<br>
self.layers = torch.nn.ModuleList([<br>
LinearWeightNorm(input_dim, 1000),<br>
LinearWeightNorm(1000, 500),<br>
LinearWeightNorm(500, 250),<br>
LinearWeightNorm(250, 250),<br>
LinearWeightNorm(250, 250)]<br>
)<br>
for i in range(len(self.layers)):<br>
m = self.layers[i]<br>
x_f = F.relu(m(x))</p>
<p>In libtorch code, how shall I convert the line of “m=self.layers(s)”</p>
<p>I use:<br>
layers = register_module(“layer”, torch::nn::ModuleList(<br>
LinearWeightNorm(input_dim, 1000),<br>
LinearWeightNorm(1000, 500),<br>
LinearWeightNorm(500, 250),<br>
LinearWeightNorm(250, 250),<br>
LinearWeightNorm(250, 250)<br>
));</p>
<p>for<br>
for(int64_t i=0; i&lt;sizeof(layers);i++){<br>
torch::nn::Module m;<br>
m=layers[i];<br>
torch::Tensor x_f = torch::nn::functional::relu(m(x));</p>
<p>I get an error in m=layers[i] saying that “No viable overloaded ‘=’”</p>
<p>Could anyone suggest how shall I modify the line to make it work please ?</p>
<p>Any comments are welcome. Much appreciated.</p>
<p>Thanks</p>

hhaoao: <p>Just look at the document.<br>
<a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module_list.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module_list.html</a></p>

Chen0729: <p>could you please share a bit more about how to modify the line to remove the error please ? I am new to libtorch. thanks.</p>

hhaoao: <p>The structure of the document example is the same as yours. You only need to change the variable name. Why should I spend my energy to rewrite the same example?</p>

Chen0729: <p>Hi, if I follow the example to use</p>
<p>for(const auto &amp;module: *layers){<br>
torch::nn::Module m;<br>
m=layers[i];</p>
<p>Then i is not iterated as the code would like to get ith layer from modulelist layers.<br>
How shall I realize that please ? I feel it is not exactly the same as the document (although I agree they are somehow similar)</p>
Taking mean across some dimension using indices of varying lengths

Umang_Sharma: <p>I am trying to take the mean across a tensor, data, having shape Batch_Size * Seq_len * Embed_dim and using a list of specified lengths across which I need to take an average. So my output should be of shape Batch_size * input_lens.shape[1] * Embed_dim where input_lens is my tensor containing the lengths of shape Batch_size * num_lengths. So far what I have got is something like :</p>
<pre><code class="lang-auto">for i, batch in enumerate(input_lens) :
        	base_length = 0
        	for j,length in enumerate(batch) :
        		mean_vec[i,j,:] = torch.mean(data[i,base_length:length,:], dim=0)
        		base_length += length.data
</code></pre>
<p>Is there a vectorised Pytorch way of doing this?</p>
<p>Thanks,<br>
Umang</p>
Changing Neurons during Inference Using Forward Hooks

gakadam: <p>I want to confirm my understanding of <strong>Hooks</strong> in PyTorch.</p>
<p>Objective: Changing incoming Neurons to the Convolution layers during inference. For simplicity, assume the neuron values will be halved.</p>
<p>Code (showing hook only):</p>
<pre><code class="lang-auto"># Define hooks
class SaveOutput:
    def __init__(self):
        self.outputs = []

    def __call__(self, module, module_in, module_out):
        # halve neuron values
        module_in = module_in * 0.5
        # see the layer outputs
        self.outputs.append(module_out)

    def clear(self):
        self.outputs = []

save_output = SaveOutput()

# register hooks
hook_handles = []

for layer in model.modules():
    # only for the Conv2D
    if isinstance(layer, torch.nn.modules.conv.Conv2d):
        handle = layer.register_forward_hook(save_output)
        hook_handles.append(handle)
</code></pre>
<p>Is this okay?</p>
<p>Thank you.</p>

albanD: <p>Hi,</p>
<p>If you want to be able to modify the input before it is processed by your module, you need to use a forward pre hook and return the new value for the input.<br>
If you want to change the output, you can use the forward hook and return the new value that should be used from now on.</p>

gakadam: <p>Does that mean, the change would be like so:</p>
<pre><code class="lang-auto">def mod_neurons(self, module, module_in):
        # halve neuron values
        module_in = module_in * 0.5

# register hooks
hook_handles = []

for layer in model.modules():
    # only for the Conv2D
    if isinstance(layer, torch.nn.modules.conv.Conv2d):
        handle = layer.register_forward_pre_hook(mod_neurons)
        hook_handles.append(handle)
</code></pre>

albanD: <p>As I mentioned, doing just <code>module_in = module_in * 0.5</code> is not going to change what is used in the forward, you need to return the updated value from the hook if you want it to be used <code>return module_in</code>.</p>
Is there any straightforward NON-language Transformer example?

shamoons: <p>I am trying to set up a very simple Transformer (mostly to understand it). I want to have some input sequence (let’s say 50 512-dimensional vectors) and via the magic of the Transformer, I want to generate 50 512-dimensional vectors that are basically double the input.</p>
<p>Very simple, I know. But I can’t find a single example of how to do this. Every example uses embeddings somehow and vocabulary sizes, which seems a bit complex. Any help would be greatly appreciated.</p>
<p>Thanks!</p>
Using torch.stft / torch.istft on PackedSequence data

actuallyaswin: <p>Will it be possible down the line to have <code>torch.stft</code> or <code>torch.istft</code> support PackedSequence objects?</p>
<p>I have provided a minimum working example here:</p>
<pre><code class="lang-auto">from torch.nn.utils.rnn import pad_sequence
from torch.nn.utils.rnn import pack_padded_sequence
from torch.nn.utils.rnn import pad_packed_sequence


fft_size = 1024
hop_size = 256


batched_audio = torch.rand((3, 50000))
window = torch.hann_window(fft_size)
Y = torch.stft(batched_audio, n_fft=fft_size, hop_length=hop_size,
               window=window)

print(torch.is_tensor(Y))
# Out: True
</code></pre>
<p>In the above cell, all the audio is equal length (50000 samples). Below is an example with varying length audio (50000, 40000, and 30000 samples respectively).</p>
<pre><code class="lang-auto">batched_audio = [torch.rand(50000), torch.rand(40000), torch.rand(30000)]
lengths = [len(x) for x in batched_audio]
padded_audio = pad_sequence(batched_audio, batch_first=True)
packed_audio = pack_padded_sequence(padded_audio, lengths, batch_first=True)

Y = torch.stft(packed_audio, n_fft=fft_size, hop_length=hop_size,
               window=window)

# Out: AttributeError: 'PackedSequence' object has no attribute 'dim'
</code></pre>
<p>This would be really handy to have to batch process variable-length audio; taking the STFT and ISTFT are important operations.</p>
<p>I guess the workaround for this right now would be to pre-compute STFTs of all the waveforms before calling <code>pad_sequence</code>, but I’m trying to reduce the GPU memory footprint of my tests. I thought on-the-fly STFT computation would be more efficient.</p>
Hooking into pytorch cuda calls

Trinayan_Baruah: <p>Sorry if this is is not a relevant question for this forum. I am trying to intercept cudaMemcpy calls from the C++ pytorch library for some data analysis. Is there anyone who can share what is the correct way to write a hook or interceptor to do this?.  I looked at the cuHook example from NVIDIA toolkit samples. But that requires modification to the source code which is not possible in my case.</p>
[K-means] - Sampling in Dataloader

nrupatunga: <p>I have a dataset for classification like below</p>
<pre><code class="lang-auto">class-1: 2500 images
class-2: 2500 images
class-3: 2500 images
...
...
class-10: 2500 images
</code></pre>
<p>Since each class has variations that could be clustered in its own class. Like if I run k-means for each class with 10 clusters</p>
<pre><code class="lang-auto">class-1: 2500 images split into 10 clusters
class-2: 2500 images split into 10 clusters
..
..
</code></pre>
<p>Now in the dataloader for a batch size of 100</p>
<pre><code class="lang-auto">- 10 examples from class-1 (from each of the ten clusters in class-1) 
- 10 examples from class-2 (from each of the ten clusters in class-2)
...
- 10 examples from class-2 (from each of the ten clusters in class-10)
</code></pre>
<p>In total there will be good 100 samples in the batch.</p>
<p>Is there a way to do it and how do I do it efficiently?</p>

ptrblck: <p>This use case might be possible to implement using a custom sampler.<br>
You would need to create the mapping between each sample and the class as well as the current cluster.<br>
Using this mapping, you could then implement a custom sampler by deriving from the base class <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py">here</a>.</p>
<p>The sampler will return the indices for the current sample or batch.<br>
It might be easier to derive from <a href="https://github.com/pytorch/pytorch/blob/b0420cc2dec44a889846ed6315136b1918b97ff5/torch/utils/data/sampler.py#L166">BatchedSampler</a> and pass a batch of indices into the <code>Dataset</code> instead of single indices.</p>

abhinavagarwalla: <p>Hi <a class="mention" href="/u/nrupatunga">@nrupatunga</a><br>
I am trying to come up with an approach to sample according to some clustering.<br>
May I ask, if you had any success implementing a custom loader?</p>
Clarity on Benchmarking Approaches

Christian_Reyes: <p>I’ve seen two benchmarking approaches. One using cuda events and the other using autograd profiler (which is more explicit and I believe uses cuda events under the hood).</p>
<p>Specifically…<br>
CUDA Events approach:</p>
<pre><code class="lang-auto"># ^warm up code

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
out = model(input)
end.record()
torch.cuda.synchronize()

print(start.elapsed_time(end))
</code></pre>
<p>Autograd Profiler approach:</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True) as prof:
    out = model(input)

total_gpu_time = # get "CUDA time total" from prof
print(total_gpu_time)
</code></pre>
<p>What is the difference between the two approaches mentioned above?</p>
Trying to install pytorch with GPU

sparshgarg23: <p>Hi,I have a system with NVIDIA driver 384.13 and Cuda 9.0 installed<br>
The output of the following command :cat /usr/local/cuda/version.txt<br>
is CUDA Version 9.0.176<br>
However when I try to install pytorch using<br>
`conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=9.0 -c pytorch</p>
<p>the installation finishes ,but when I run the code<br>
python -c ‘import torch; print(torch.rand(2,3).cuda())’<br>
I get the following message<br>
Assertion error torch not compiled with CUDA enabled<br>
Any ideas on how to work around this issue.<br>
I would also like to point out that I might have to migrate to a system where Cuda 8 is installed ,so I am sure that when I execute the command<br>
` ```<br>
conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=8.0 -c pytorch</p>
<pre><code class="lang-auto">UPDATE:
I tried a pip based installation
'''
pip install torch===1.5.0 torchvision===0.6.0 -f https://download.pytorch.org/whl/torch_stable.html
'''
However this gives me an error:
</code></pre>
<p>AssertionError:<br>
The NVIDIA driver on your system is too old (found version 9000).<br>
Please update your GPU driver by downloading and installing a new<br>
version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" rel="nofollow noopener">http://www.nvidia.com/Download/index.aspx</a><br>
Alternatively, go to: <a href="https://pytorch.org" rel="nofollow noopener">https://pytorch.org</a> to install<br>
a PyTorch version that has been compiled with your version<br>
of the CUDA driver.</p>
<pre><code class="lang-auto">So i guess the option is here to figure out which version of pytorch is compatible with 9000,and what version python does it support.Currently I am using python 3.8 and I also have a 3.7 environment.</code></pre>

Henry_Chibueze: <p>I suggest u download your pytorch version from the pytorch website (select ur cuda version and it will generate a link to whl file )<br>
Just pip install link.</p>

Flock1: <p>Hi. Are you installing pytorch in a conda environment?</p>

Henry_Chibueze: <p>No I install mine with pip but on the the pytorch website, theirs an option for conda Installation</p>

Flock1: <p>pip is an option but I would definitely go with conda installation if working with environments, which I recommend?</p>
<p><a class="mention" href="/u/sparshgarg23">@sparshgarg23</a>, maybe try conda installation with an environment.</p>

sparshgarg23: <p>As henry said using pip to install pytorch worked.I had to do some digging to find the relevant version.Seems cuda 9 uses pytorch 1.1 and torchvision 0.3</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation V2

Rens: <p>I saw similar topic but I don’t get why my code contains an inplace operation.</p>
<p>code:</p>
<pre><code class="lang-auto">l = self.backward()
</code></pre>
<p>with:</p>
<pre><code class="lang-auto">def backward(self):
     transitions = self.replay_memory.sample(self.batch_size)
        
        batch = Transition(*zip(*transitions))
            
        a_batch = torch.cat(batch.action).to(self.device) # [BS x 1]
        cs_batch = torch.stack(batch.state).to(self.device)    # [BS x state_size]
        ns_batch = torch.stack(batch.next_state).to(self.device) # [BS x state_size]
        r_batch = torch.tensor(np.expand_dims(np.array(batch.reward),1), dtype=torch.float).to(self.device)  # [BS x 1]

# STEP 2: PREDICTIONS
        pred = self.calc_pred(cs_batch, a_batch)
        
        
        # STEP 3: TARGETS
        with torch.no_grad():
            target = self.calc_target(ns_batch, r_batch)
            
            
        
        # STEP 4: LOSS
        loss = self.calc_loss(pred, target)   # [BS x 1]
        
        
        
        loss = loss.mean()

    # STEP 5: TRAIN
     
            
   
       self.optimizer.zero_grad()
       loss.backward()
          
        self.optimizer.step()
 return loss
</code></pre>
<p>With</p>
<pre><code class="lang-auto">          
    def calc_pred(self, cs_batch, a_batch):
        
        if self.noisy:
            
                self.network.sample_noise()
        
       
        pred = self.network(cs_batch).gather(1, a_batch)
        
        return pred
</code></pre>
<p>And also:</p>
<pre><code class="lang-auto"> def calc_target(self, ns_batch, r_batch):

    if self.double:
                
                if self.noisy:
                    self.network.sample_noise()
                    self.target_network.sample_noise()
                next_Q = self.network(ns_batch)
                max_a = next_Q.max(1)[1].unsqueeze(1)
                Q_target = self.target_network(ns_batch).gather(1, max_a)
    target = r_batch + (self.gamma ** self.multi_step_n) * Q_target    
              
return target
</code></pre>
<p>and loss function:</p>
<pre><code class="lang-auto">
def calc_loss(self, pred, target):
loss = F.mse_loss(pred, target, reduction='none')
            
 return loss

</code></pre>
<p>This works if am not using a noisy network. However, when using noisy (self.noisy = True)  it returns the following error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [5]] is at version 1003; expected version 1002 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>However, I do not change any variables in my code whenever I use my noisy network? I have used the noisy network also in another piece of code were I do not get the error, so I do not think the error is in that part of the code.</p>
Use attn_mask to enforce causality in MultiheadAttention?

pmixer: <p>Hi Team,</p>
<p>Following <a href="https://github.com/pytorch/pytorch/issues/21518" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/21518</a>, for tracking the additive attn_mask versus multiplicative attn_mask issue, I tested by the following code:</p>
<pre><code class="lang-python">
attn_mask = torch.tril(torch.ones((8,8)))
inputs = torch.ones((8, 2, 6))
mha = torch.nn.MultiheadAttention(6, 2) # hidden_dim=6, head_num=2
outputs, weights = mha(inputs, inputs, inputs, attn_mask=attn_mask) # Q, K, V, attn_mask for causality
</code></pre>
<p>to enforce causality, but the returned attention weights suggest it still attend to future inputs:</p>
<pre><code class="lang-auto"> tensor([[[0.2797, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029],
          [0.2377, 0.2377, 0.0874, 0.0874, 0.0874, 0.0874, 0.0874, 0.0874],
          [0.2066, 0.2066, 0.2066, 0.0760, 0.0760, 0.0760, 0.0760, 0.0760],
          [0.1828, 0.1828, 0.1828, 0.1828, 0.0672, 0.0672, 0.0672, 0.0672],
          [0.1638, 0.1638, 0.1638, 0.1638, 0.1638, 0.0603, 0.0603, 0.0603],
          [0.1485, 0.1485, 0.1485, 0.1485, 0.1485, 0.1485, 0.0546, 0.0546],
          [0.1357, 0.1357, 0.1357, 0.1357, 0.1357, 0.1357, 0.1357, 0.0499],
          [0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250]],
 
         [[0.2797, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029, 0.1029],
          [0.2377, 0.2377, 0.0874, 0.0874, 0.0874, 0.0874, 0.0874, 0.0874],
          [0.2066, 0.2066, 0.2066, 0.0760, 0.0760, 0.0760, 0.0760, 0.0760],
          [0.1828, 0.1828, 0.1828, 0.1828, 0.0672, 0.0672, 0.0672, 0.0672],
          [0.1638, 0.1638, 0.1638, 0.1638, 0.1638, 0.0603, 0.0603, 0.0603],
          [0.1485, 0.1485, 0.1485, 0.1485, 0.1485, 0.1485, 0.0546, 0.0546],
          [0.1357, 0.1357, 0.1357, 0.1357, 0.1357, 0.1357, 0.1357, 0.0499],
          [0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250]]],
        grad_fn=&lt;DivBackward0&gt;))
</code></pre>
<p>does that means its still additive mask in current implementation(I used PyTorch 1.6.0+cu101 on google colab)?</p>
<p>THX!</p>

pmixer: <p>Resolved, pls check <a href="https://github.com/pytorch/pytorch/issues/21518" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/21518</a>, guess it’s still additive attention for numerical stability under the hood but behaves well(by replacing <em>0 as -inf before applying softmax for Q</em>K generated attention outputs).</p>
Why program exit after the backward funtion ,please help

Disp41r_QAQ: <p>I download the code and try to run it, but it exit after the backward funtion,can anyone tell me why and how to solve it,the project is <a href="https://github.com/ice-tong/pytorch-captcha" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ice-tong/pytorch-captcha: 使用pytorch搭建cnn识别验证码</a>, thank u<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3d9b63d76ab248b6f14d3f75b004751d833fad2.png" data-download-href="https://discuss.pytorch.org/uploads/default/c3d9b63d76ab248b6f14d3f75b004751d833fad2" title="图片"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3d9b63d76ab248b6f14d3f75b004751d833fad2.png" alt="图片" data-base62-sha1="rWzyeXCuaBWWNe9Y15VMQfMoSeS" width="690" height="265" data-dominant-color="8F8E8F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片</span><span class="informations">1147×441 31.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Disp41r_QAQ: <p>i think it may be my cuda’s problem,i use the cpu to train it,and it works</p>
Adding Tensors to buffer but avoid saving tensors

Luke_Taylor: <p>I have a tensor that I compute once-off in the model and whose values are not optimised. I add this Tensor to the buffer of the model (such that it can me mapped between CPU and GPU as needed). However, I do not want the Tensor to be saved when saving all the model weights. What is the preferred way of achieving this?</p>
<p>I understand I could save this Tensor (like all other model params). However I prefer not to save it to reduce file size.</p>
<p>Thanks!</p>

googlebot: <p>register_buffer(…, persistent=False) argument was added in 1.6</p>
The tensor can't be autograded with unsqueeze(2)

DavidZhang: <pre><code class="lang-auto">rho = base[0][:,1:,:]
mu = base[1][:,1:,:]
diff_time = (time[:, 1:] - time[:, :-1]) * non_pad_mask[:, 1:]
temp_time = diff_time.unsqueeze(2) * \
           torch.rand([*diff_time.size(), num_samples], device=data.device) 
temp_time /= (time[:, :-1] +1).unsqueeze(2) 
all_base = torch.zeros(rho.size())
for i in range(rho.size()[2]):
    slice_of_mu = mu.clone()[:,:,i]
    slice_of_mu = slice_of_mu.unsqueeze(2)
    slice_of_rho = rho.clone()[:,:,i]
    slice_of_rho = slice_of_rho.unsqueeze(2)
    all_base[:,:,i] = all_base[:,:,i] + ( slice_of_mu * slice_of_rho * temp_time ^ ( slice_of_rho - 1 ) )
</code></pre>
<p>size of rho and mu are [16,99,5], and size of temp_time is [16,99,100]，I want to calculate the value of all_base as shown in the for loop, I sliced and deformed these tensors to ensure their dimensions are consistent, but I get an error in the last line:</p>
<p><strong>bitwise_xor(): functions with out=… arguments don’t support automatic differentiation, but one of the arguments requires grad.</strong></p>
<p>File “…”, line 47, in compute_integral_unbiased<br>
all_base[:,:,i] = all_base[:,:,i] + ( slice_of_mu * slice_of_rho * temp_time ^ ( slice_of_rho - 1 ) )</p>
<p>I don’t know how to solve this problem. Can you give me some help? Thank you very much.</p>

DavidZhang: <p>I use a ^ rather than **, what a naive mistake…</p>
RuntimeError: 0 INTERNAL ASSERT FAILED at

Duong_Pham_Anh: <p><strong>I get error when trace model pytorch to torchscript</strong><br>
<strong>it notice me report a bug  to PyTorch</strong><br>
<strong>and it dont return any information about bug so i cant fix</strong></p>
<p>i user torch==1.6.0, torchVison==0.7.0, cuda==10.2, cupu-cuda 10.2, opencv-contrib-python, matplotlib</p>
<p>link to suorce code: <a href="https://github.com/duongpaKikai/Video-Stabilaztion" rel="noopener nofollow ugc">Video Stabilization</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9fec67e5aaa91b1fdc04e29807b36d738d694806.png" data-download-href="https://discuss.pytorch.org/uploads/default/9fec67e5aaa91b1fdc04e29807b36d738d694806" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fec67e5aaa91b1fdc04e29807b36d738d694806_2_690x148.png" alt="image" data-base62-sha1="mOKpJeYrH9dg6XQGrWOrXDIGnR4" width="690" height="148" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fec67e5aaa91b1fdc04e29807b36d738d694806_2_690x148.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fec67e5aaa91b1fdc04e29807b36d738d694806_2_1035x222.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9fec67e5aaa91b1fdc04e29807b36d738d694806_2_1380x296.png 2x" data-dominant-color="333540"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1621×348 48.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to add layers to a pre-trained model (Model made and pre-trained by me only)

Suraj: <p>So I have this:</p>
<pre><code class="lang-auto">class Net1(nn.Module):
   ... #defined it
Model1 = Net1()
optimizer1 #defined accordingly
#pretraining over a loss function, say loss_fn1
for i,(img,label) in enumerate(train_loader):
   ...
</code></pre>
<p>Best way to add a few more layer to this and do training over a new loss function (say loss_fn2) ???<br>
Please let me know if I am being unclear.</p>
<p>One way I tried was to define these new layers as a new model and a new optimizer (say Model2 and optimizer2) and do this:</p>
<pre><code class="lang-auto">for i,(img,label) in enumerate(train_loader):
   out = Model1(img)
   out = Model2(out)
   loss = loss_fn2(out,label) 
   loss.backward(retain_graph = True)
   optimizer2.step()
   optimizer1.step()
</code></pre>
<p>This looks fundamentally wrong to me now. Are the gradient of loss function by chain rule backpropagated to the pre-trained model parameters? I don’t think they are connected.</p>
<p>I don’t know how to patch these two models with parameters of first model coming from pretraining. Any lead or hint is appreciated. Thanks!!</p>

ptrblck: <aside class="quote no-group" data-username="Suraj" data-post="1" data-topic="96207">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/suraj/48/28176_2.png" class="avatar"> Suraj:</div>
<blockquote>
<p>This looks fundamentally wrong to me now. Are the gradient of loss function by chain rule backpropagated to the pre-trained model parameters? I don’t think they are connected.</p>
</blockquote>
</aside>
<p>Why do you think the computation graph is not connected? Did you check the gradients in <code>Model1</code> or do you see any other issues?</p>
<p>Your code looks alright and you can chain different models as you are also chaining different layers.<br>
Since <code>Model1</code> and <code>Model2</code> are implemented as <code>nn.Module</code>s there would be no difference to “layers” such as <code>nn.Conv2d</code>.</p>

Suraj: <p>Thank you for responding. I think I got it now. Computational graph should be connected. (Any way you suggest to confirm that? would be a good check to make while designing any model)</p>
<p>So <code>loss.backward()</code> is solely responsible for calculating all the gradients of loss function in the computational graph wrt to the parameters and <code>optimizer.step()</code> just picks up these gradient values to make an update.</p>
<p>Sorry, I thought <code>optimizer.step()</code> also has role in deciding what all (and how) gradients will be computed (based on parameters passed to the optimizer).</p>