akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
GPU allocated memory problem

HDB: <p>Hi, a question confused me for a long time. When I use batch gradient descend in pytorch,  the allocated memory of gpu accumulates with the number of training within on batch. I failed to find where the mistake is, someone can help? Thank you very much. Some information below is the output of my code.</p>
<p><strong><strong><strong><strong><strong>No.0</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：9.38Mb<br>
cached memory of GPU is：30.00Mb<br>
<strong><strong><strong><strong><strong>No.1</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：582.33Mb<br>
cached memory of GPU is：602.00Mb<br>
<strong><strong><strong><strong><strong>No.2</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：1157.52Mb<br>
cached memory of GPU is：1174.00Mb<br>
<strong><strong><strong><strong><strong>No.3</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：1730.43Mb<br>
cached memory of GPU is：1748.00Mb<br>
<strong><strong><strong><strong><strong>No.4</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：2304.48Mb<br>
cached memory of GPU is：2320.00Mb<br>
<strong><strong><strong><strong><strong>No.5</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：2878.59Mb<br>
cached memory of GPU is：2890.00Mb<br>
<strong><strong><strong><strong><strong>No.6</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：3455.23Mb<br>
cached memory of GPU is：3482.00Mb<br>
<strong><strong><strong><strong><strong>No.7</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：4029.53Mb<br>
cached memory of GPU is：4054.00Mb<br>
<strong><strong><strong><strong><strong>No.8</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：4604.16Mb<br>
cached memory of GPU is：4628.00Mb<br>
<strong><strong><strong><strong><strong>No.9</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：5179.34Mb<br>
cached memory of GPU is：5200.00Mb<br>
<strong><strong><strong><strong><strong>No.10</strong></strong></strong></strong></strong>*<br>
allocated memory of GPU is：5754.85Mb<br>
cached memory of GPU is：5774.00Mb</p>

ptrblck: <p>This increase in memory usage is often due to (accidentally) storing some tensors, which are attached to the computation graph, and thus the complete graph with it.<br>
Could you check, if you are adding the model output, loss, or any other tensor with a valid <code>.grad_fn</code> to a <code>list</code> or any other container in each training step?<br>
If so, use <code>tensor.detach()</code> or <code>tensor.item()</code> instead when appending it to the container.</p>

HDB: <p>Yes, sir. Your answer give me great inspiration. A model output in my code was assigned to requires_grad = True, which resulted in accumulation of GPU allocated memory with training step. Thanks for your kindness for answering.</p>
How to annotate non-object image

Hwarang_Kim: <p>If non-object images are in training data set,<br>
how should i annotate it?<br>
If annotation vector form is like [bbox, cls], can [-1,-1,-1,-1,-1] be annotation?<br>
This annotation padding is needed for batch training i think.<br>
But [-1,-1,-1,-1,-1] is right? if right, why?<br>
thanks for answer in advance.</p>
Loading pytorch model (load_state_dict(torch.load('model.pth'))

tinu445: <p>I am loading a pretrained model from an existing GitHub repository using “load_state_dict”, where the model gives and error as:</p>
<p>model_rgb = torch.load(‘eval/pretrained_models/RgbNet.pth’)</p>
<p>result = unpickler.load()<br>
ImportError: No module named ‘numpy.core._multiarray_umath’</p>
<p>I searched for the error and found that it may be due to the torch version difference during training and now testing process.</p>
<p>Is it for sure that this is due to the versions difference?</p>
<p>The code I am using to load the model:</p>
<pre><code class="lang-auto">pretrained_models={'RgbNet':'eval/pretrained_models/RgbNet.pth', 'RgbdNet':'eval/pretrained_models/RgbdNet.pth' , 'DepthNet':'eval/pretrained_models/DepthNet.pth' }
result_path='crossdata_output/'
os.makedirs(result_path,exist_ok=True)

for tmp in ['D3Net']:
    os.makedirs(os.path.join(result_path,tmp),exist_ok=True)
    for test_dataset in test_datasets:
        os.makedirs(os.path.join(result_path,tmp,test_dataset),exist_ok=True)

model_rgb=RgbNet().cuda()


# model_rgb.load_state_dict(torch.load(pretrained_models['RgbNet'])['model'])
</code></pre>
<p>The code that saved the trained model</p>
<pre><code class="lang-auto">        if self.p['if_save_checkpoint']:
            torch.save(pth_state, os.path.join(self.p['snapshot_path'], 'checkpoint.pth'))
        if is_best:
            torch.save(pth_state, os.path.join(self.p['snapshot_path'], 'best.pth'))
</code></pre>

tinu445: <p>Okay this wasn’t the problem with the torch version or something.<br>
The problem was with my numpy version and you can simply run the following command to get it done.</p>
<pre><code class="lang-auto">pip install -U numpy
</code></pre>
How to improve image quality of torchvision.transforms.Resize

Hao_Meng: <p>When I did such operation, the result image looks in poor quality.How to solve it?</p>

ptrblck: <p>You could specify the <code>interpolation</code> argument in <code>transforms.Resize</code> to another <code>PIL.Image</code> routine.</p>

Hao_Meng: <p>but I saw little change in all these setting. Do u have any recommendation of these interpolation?</p>

ptrblck: <p>I would assume you would see changes in the quality by trying out different algorithms.<br>
Could you post an example result of such a bad quality image?</p>
Mixing GPU/CPU code in TorchScript traced model

churchillmic: <p>Is it possible to TorchScript trace a wrapper module, which does preprocessing on CPU, then a feed-forward network on GPU? I’m trying the below, but I’m not sure if <code>.cpu()</code> and <code>.cuda()</code> mean anything within the TorchScript traced module. The below runs, but then when I run in C++, I get an error about <code>expected device cpu but got device cuda:0</code>, basically in the traced code part after preprocess. In C++, I do have a lines to put the model on the GPU, I’m not sure if that is interfering, and in general if such lines are redundant if the tracing in Python already has <code>.cuda()</code> calls (also, I know about <code>torch::NoGradGuard no_grad;</code> in C++, am I able to avoid that if I use <code>with torch.no_grad()</code> in the tracing Python code?)</p>
<p>C++ code</p>
<pre><code class="lang-auto">        model = torch::jit::load("traced_auglag_best.pt");
        model.to(at::kCUDA);
        model.eval(); //not sure if necessary, was .eval() in the tracing code
</code></pre>
<p>Python tracing code</p>
<pre><code class="lang-auto">class Wrapper(torch.nn.Module):

    #mean_f: Final[float]
    #std_f: Final[float]
    #mean_fdf: Final[float]
    #std_fdf: Final[float]

    def __init__(self,model,file_stats):
        super(Wrapper, self).__init__()
        self.file_stats = file_stats
        self.read_stats()
        self.model = model
        self.model = self.model.cuda()
        self.model.eval()

    def read_stats(self):
        fh = h5py.File(self.file_stats,'r')
        attrs = ['mean_f','std_f','mean_fdf','std_fdf']
        for attr in attrs:
            setattr(self,attr,torch.nn.Parameter(torch.from_numpy(fh[attr][...])))
        fh.close()

    def normalize_f(self,f):
        return ((f - self.mean_f[None,...])/self.std_f[None,...]).float()

    def preprocess(self,f):
        #remove negative inds
        f[f&lt;0] = torch.tensor(0.0).double()
        #if adiabatic electron, add dimension for electrons
        #if torch.tensor(f.size())[0]==torch.tensor(1): #(assume for now adiabatic electrons, otherwise trace issue)
        f = F.pad(f,(0,0,0,0,0,0,1,0))
        #switch order for pytorch model to [Ngrid,Nsp,Nmu,Nvpara]
        f = f.permute(2,0,1,3)
        #pad mu direction (so 32,32)
        f = F.pad(f,(0,1,0,0),mode='replicate')
        #normalize and convert to float for input to model
        return f

    def postprocess(self,fdfnorm,f,isp=1):
        #unnormalize
        df = fdfnorm*self.std_fdf[None,[isp],...] + self.mean_fdf[None,[isp],...] - f[:,[isp],...]
        #remove extra vpara dimension
        df = df[:,:,:,:-1]
        #switch order back to XGC order of [Nsp,Nmu,Ngrid,Nvpara] and convert to double
        return df.permute(1,2,0,3).double()

    def forward(self,f):
        with torch.no_grad():
            fpre = self.preprocess(f)
            fnorm = self.normalize_f(fpre)
            out = self.model(fnorm.cuda()).cpu()
        return self.postprocess(out,fpre)
</code></pre>
Pytorch no computation graph mode

Bruce_zhuang: <p>I am trying to get the PyTorch to work without preserving its computation graph at the background, to save GPU memory. My code is roughly as follows:</p>
<blockquote>
<p><strong>with torch.no_grad():</strong><br>
train_X, labels = Variable(train_X, requires_grad=False).to(device), Variable(labels, requires_grad=False).to(device)<br>
conv0 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=5, bias=True).to(device)<br>
relu = nn.LeakyReLU().to(device)<br>
avgpool = nn.AvgPool2d(kernel_size=2).to(device)<br>
out = conv0(train_X) # <code>GPU memory gets up to 2853M after this line</code><br>
out = relu(out) # <code>GPU memory gets up to 4769M after this line</code><br>
out = avgpool(out) # <code>GPU memory gets up to 5249M</code></p>
</blockquote>
<p>My question is that <code>out = relu(out)</code> seems to preserve the memory earlier obtained using <code>out = conv0(train_X)</code>, which can be cleared using <code>torch.cuda.empty_cache()</code>, but I have used the <code>with torch.no_grad():</code> that should stopping generating the computation graph. I have tried setting the <code>train_X</code>, <code>conv0.weight</code> to <code>requires_grad=False</code>, but they did work. Could you tell me what is wrong? Sorry I don’t know how to adjust the codes <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<p>I know this seems to be quite trivial. If there is a duplication, please point out! Thank you!</p>

albanD: <p>Hi,</p>
<p>To prevent the autograd from saving anything, setting <code>with torch.no_grad():</code> is enough.<br>
The other issue you might have is about how you measure GPU memory as pytorch use a custom allocator and so memory reported by the OS is not necessarily correct.<br>
Also when you do <code>out = relu(out)</code>, the relu is first computed, and thus a temporary output is allocated. Then the old content of out is discarded and replaced with the temporary one. So I would expect to see a memory bump here.<br>
Note that most activation functions have an inplace flag to make them perform their update inplace and avoid this issue.</p>

Bruce_zhuang: <p>Hi. Thanks for the reponse! I tried <code>inplace=True</code>, and it solved the memory increase in <code>out = relu(out)</code>. However, the memory issue in <code>out = maxpool(out)</code> persists. Any way to resolve this like the activation case? Pooling should reduce the memory consumption in our case.</p>

albanD: <p>Hi,</p>
<p>The pooling cannot happen inplace. So it has to allocate an output to write the result into. So it is expected that during the function, the memory goes up a bit until you can free the input.</p>
How to create a Label Tensor

Angelina_Robert: <p>Hi all,<br>
I have 10 classes(indices or labels 0…9) with different shapes i.e</p>
<p>class 1 to class 10 shapes:  [14, 2, 128] , [16, 2, 128], [15, 2, 128], [20, 2, 128],[15, 2, 128],[20, 2, 128], [10, 2, 128], [11, 2, 128], [7, 2, 128]</p>
<p>I want to create a label tensor of 128 batch size  using the first value from each shape tensor so that label tensor consist of 14 values of index 0 , 16 values of index 1,</p>
<p>15 values of index 2 and likewise.</p>
<p>Any help in this regard will be highly appreciated.</p>
<p>Cheers,<br>
Angelina</p>

albanD: <p>Hi,</p>
<p>You can take all of these Tensors in a list and concatenate them to get a single Tensor will all of the content: <code>out = torch.cat(your_list, dim=0)</code>.</p>
Compiling Pytorch on devices with different cuda capability

Siddhesh_Mhatre: <p>Hi,</p>
<p>I have compiled pytorch on a machine that has an Nvidia GPU with cuda capability (cc) 7.5. I tried using the same compiled libraries on  machine that has an Nvidia GPU with cc of 6.1 and it worked, but I got the following error</p>
<pre><code class="lang-auto">CUDA error: no kernel image is available for execution on the device
</code></pre>
<p>when I tried to use the same on an Nvidia GPU with cc of 7.0</p>
<p>Based on <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compilation-workflow" rel="noopener nofollow ugc">https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compilation-workflow</a>, the compiled code should be forward compatible only across minor versions if binary code is generated and forward compatible across minor and major versions if PTX code is generated. Consequently, I’m not sure how the above case would have occurred. Appreciate any explanation and insights into how pytorch interacts with CUDA.</p>
<p>Note -<br>
Pytorch version compiled 1.4.0 (release/1.4 branch)</p>

ptrblck: <p>I guess you haven’t used PTX and only built PyTorch for sm_75.<br>
You could use <code>TORCH_CUDA_ARCH_LIST="6.1 7.5" python setup.py install</code> to build for both architectures. You can also add <code>+PTX</code> after each architecture to ship with PTX.</p>

Siddhesh_Mhatre: <p>Thanks! Wasn’t aware of the <code>PTX</code> flag. Does this flag generate just the CUDA assembly code for the particular architecture or does it also generate the binaries?</p>

ptrblck: <p>The <code>+PTX</code> flag generates the PTX ISA, which can be used to create the SASS. I’m not sure what binaries refers to.</p>
Dynamic_axes not working as expected in torch.oxxn.export (torchvision)

Francesco_Pochetti: <p>Hi all,<br>
I am not sure what is going on but <code>dynamic_axes</code> is not working as expected when re-loading the model from ONNX and running inference, e.g. I am obliged to feed the exact same input shape provided at saving time, which is NOT what occurs <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html" rel="noopener nofollow ugc">here</a>.<br>
Here what happens.</p>
<pre><code class="lang-auto">import torchvision
import torch
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
x = torch.rand(1, 3, 300, 400)
predictions = model(x)
torch.onnx.export(model,              
                  x,                         
                  "./output/faster_rcnn.onnx",   
                  export_params=True,        
                  opset_version=11,     
                  input_names = ['input'],   
                  output_names = ['output'], 
                  dynamic_axes={'input' : {0 : 'batch_size'},    
                                'output' : {0 : 'batch_size'}})


import onnxruntime as ort
import numpy as np

ort_session = ort.InferenceSession("./output/faster_rcnn.onnx")
def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()
y = torch.rand(10, 3, 300, 400)
# compute ONNX Runtime output prediction
ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(y)}
ort_outs = ort_session.run(None, ort_inputs)
</code></pre>
<p>Throws</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
Fail                                      Traceback (most recent call last)
&lt;ipython-input-9-ded63d7ba257&gt; in &lt;module&gt;
     11 # compute ONNX Runtime output prediction
     12 ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(y)}
---&gt; 13 ort_outs = ort_session.run(None, ort_inputs)
     14 ort_outs

/anaconda/envs/fra_vrr/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    122             output_names = [output.name for output in self._outputs_meta]
    123         try:
--&gt; 124             return self._sess.run(output_names, input_feed, run_options)
    125         except C.EPFail as err:
    126             if self._enable_fallback:

Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Split node. Name:'Split_8' Status Message: Cannot split using values in 'split' attribute. Axis=0 Input shape={10,3,300,400} NumOutputs=1 Num entries in 'split' (must equal number of outputs) was 1 Sum of sizes in 'split' (must equal size of selected axis) was 1
</code></pre>
<p>To be clear, everything works fine when <code>y = torch.rand(1, 3, 300, 400)</code>.<br>
Is this a <code>torchvision</code> issue?<br>
Thanks!</p>
IndexError: index 1 is out of bounds for dimension 1 with size 1

anotherone_one: <p>Hello!</p>
<p>I am having this issue in the training when extracting the prediction:</p>
<p>output = model.forward(image)</p>
<p>The shape of the image tensor is: torch.Size([1, 1, 299, 299])</p>
<p>Batch size: 1<br>
Image size: 299, 299<br>
Grayscale</p>
<p>The model is the inception v3</p>
<p>Anyone has any idea why this is happening?</p>
<p>Thank you in advance</p>

Alexey_Demyanchuk: <p>Hi. When I try to reproduce the behavior you’re describing:</p>
<pre><code class="lang-auto">m = torchvision.models.inception_v3(init_weights=False)  # this is for the speedup 
img = torch.randn(1,1,299,299) # dummy image
</code></pre>
<ol>
<li>If use image with  <code>torch.Size([1, 1, 299, 299])</code>, I got this error:<br>
<code>RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 1, 299, 299] to have 3 channels, but got 1 channels instead</code><br>
The model doesn’t work with 1 channel images</li>
<li>If you want to predict on the batch size=1, you should put the model in evaluation mode.</li>
<li>To make it work with grayscale image you can try to just copy 1 channel into 2 additional, so to have the correct number of channels and see if it provides meaningful results to you.</li>
</ol>
<p>This <a href="https://discuss.pytorch.org/t/best-way-to-deal-with-1-channel-images/26699/2">post</a> on how to deal with grayscale image may be of help to you.</p>

anotherone_one: <p>I actually had forgotten to uncomment the line that changed the first layer of the inception to receive grayscale images ! Thank you for the help!</p>
Running Pytorch LSTM in browser

christian-doucette: <p>I am trying to run my trained LSTM network in the browser. I attempted to use ONNX and ONNX.js, but while ONNX supports LSTM (I was able to successfully save my trained LSTM network as a .onnx file), it seems that LSTM is not supported by ONNX.js. Let me know if I have this wrong.</p>
<p>Then, what is the best way to run my LSTM network in the browser? Any ideas would be appreciated.</p>
<p>The reason I want to run in browser is to allow simple interactions with the network on my github pages website. So if there is an easy alternative way to display a Pytorch network on the web that could solve this problem as well. Thanks!</p>

Alon: <p>It’s not exactly what you are looking for perhaps, but I recently started to use StreamLit and I think it’s amazingly simple and useful for making any python program into a browser interaction interface and supports native plotly, matplotlib, pandas… so it fits perfectly with Pytorch sort of data types: <a href="https://www.streamlit.io/" rel="noopener nofollow ugc">Streamlit — The fastest way to create data apps</a></p>

christian-doucette: <p>Thanks, this is a great alternative.</p>
Gpu replacement operation seems unreasonably slow?

tueboesen: <p>I have the following operation in my code:</p>
<p><code>result[mask] = aa_select[mask]</code></p>
<p>where everything is living on the gpu already:</p>
<pre><code class="lang-auto">result
tensor([[ 0, 20, 32,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 14,  2],
        ...,
        [ 0, 20, 19,  ..., 18, 14,  2],
        [ 0, 32, 19,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 32,  2]], device='cuda:0')
mask
tensor([[False, False,  True,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        [False, False, False,  ..., False, False, False],
        ...,
        [False, False, False,  ..., False, False, False],
        [False,  True, False,  ..., False, False, False],
        [False, False, False,  ..., False,  True, False]], device='cuda:0')
aa_select
tensor([[ 0, 20, 15,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 14,  2],
        ...,
        [ 0, 20, 15,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 14,  2],
        [ 0, 20, 19,  ..., 18, 15,  2]], device='cuda:0')
</code></pre>
<p>Now the shape of these tensors are (100,244), so they are not unreasonably big.<br>
However the operation I’m doing at the top, for some reason takes ~3.5 seconds to complete, which seems completely unreasonable.</p>
<p>Does anyone have any clue what is going on here? I’m guessing the operation is somehow not allowed on the gpu so it is moved to the cpu which I guess could explain how it takes this long. (For comparison I got my results by running data through my neural network with 650M parameters and that happened in 0.04s which highlights how absurdly expensive the 3.5 seconds are)</p>
<p>How can I fix this?</p>
Getting the locations of the blocks returned by fold

aliutkus: <p>Hi there, my question is the following:</p>
<p>let’s assume I call <code>nn.functional.fold</code> on a tensor <code>v</code> of shape <code>(b,c,w,h)</code>. It returns me a rank-3 tensor of shape <code>(b, blocksize, blockcount)</code>. I’m interested in knowing the actual position for each block within the input <code>v</code>. Is there an easy way ?</p>
<p>thanks</p>
Computing network embeddings, avoid memory leak

gabrieldernbach: <p>I am dealing with variable sized input, and most instances are too large to fit in memory, even using multiple GPUs. I therefore use one network to compute a reduced representation (embedding) which I then forward to the prediction head (basically a form of gradient checkpointing).</p>
<p>There are surely many ways to implement this.<br>
I am particularly surprised with a memory leak I encountered in the following approach:</p>
<pre><code class="lang-auto">        self.network.eval()
        with torch.no_grad():
            instances = defaultdict(list)
            for batch in dataloader:
                img = batch["img"].to(self.device)
                batch["embedding"] = self.network(img).detach().cpu()
                for key in patchbatch:
                    instances[key].append(batch[key])
</code></pre>
<p>I believe that <code>tensor.to.self(device)</code> is not an in-place operation and results in a copy. So when I later collect batch[“img”] and batch[“embedding”] for a later backwards pass, they do not live on the GPU.</p>
<p>At which place am I blocking the GPUs memory from being freed?</p>
What is the plan for TensorExpr?

lars: <p>I have noticed some exciting stuff going on in the torch/csrc/jit/tensorexpr directory in the git repo.</p>
<p>It seems to define an IR+codegen. I’m very interested in hearing what the end goal is with this. Obviously, what is there seems immediately useful for allowing optimization over execution graphs (e.g. fusing kernels). I haven’t seen anything that generates the IR. Is the idea to define kernels directly in the IR, so they can more easily be optimized? Or are there plans for a front end that generates the IR?</p>
<p>I ask because many of the pieces seem to be in place for a numba-like system where custom kernels can be defined in a subset of python, and fast code can be generated for GPU and CPU. Are there plan for anything like this, or is that just my own fantasy? (I see that there’s something new going on in the torch.fx namespace, but I don’t know what it’s for.) The missing piece seems to be something that can take a python AST (or some other high level language) and generate the IR.</p>
<p>There’s very little innovation in the basic weight layers of neural networks IMO, there’s convolutions and linear layers, and not much else. I suspect a big reason for this is that it’s hard to produce a performant implementation of an idea. I was a big fan of the Tensor Comprehensions project a couple of years back, but that died out.</p>
<p>I would be very happy to hear a bit more about what the roadmap is for this feature.</p>
Error in backward

X_Zhao: <p>I encountered a strange problem. When I use torch.cat to splice two tensors, I will encounter the following error, but adding the two tensors directly will not report an error. Why?</p>
<pre><code class="lang-auto">for i in range(num_levels):
            inds = target_lvls == i
            if inds.any():
                rois_ = rois[inds, :]
                roi_feats_t = self.roi_layers[i](feats[i], rois_)
                if self.use_expand_roi:
                    # expand rois
                    rois_expand_ = rois_expand[inds, :]
                    roi_feats_expand_t = self.roi_layers[i](feats[i], rois_expand_)
                    roi_feats[inds] = torch.cat([roi_feats_t,roi_feats_expand_t],1)
                    # roi_feats[inds] =  roi_feats_t + roi_feats_expand_t
                else:
                    roi_feats[inds] = roi_feats_t
            else:
                roi_feats += sum(
                    x.view(-1)[0]
                    for x in self.parameters()) * 0. + feats[i].sum() * 0.
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./tools/train.py", line 178, in &lt;module&gt;
    main()
  File "./tools/train.py", line 174, in main
    meta=meta)
  File "/home/zhaoxin/workspace/mmdetection/mmdet/apis/train.py", line 150, in train_detector
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/zhaoxin/workspace/mmcv/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/zhaoxin/workspace/mmcv/mmcv/runner/epoch_based_runner.py", line 51, in train
    self.call_hook('after_train_iter')
  File "/home/zhaoxin/workspace/mmcv/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/zhaoxin/workspace/mmcv/mmcv/runner/hooks/optimizer.py", line 27, in after_train_iter
    runner.outputs['loss'].backward()
  File "/home/zhaoxin/anaconda3/envs/mmdetection/lib/python3.7/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/zhaoxin/anaconda3/envs/mmdetection/lib/python3.7/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: grad_output must be contiguous
</code></pre>

Naruto-Sasuke: <p>Try adding <code>.contiguous()</code> right after the place where you slice the tensor.</p>
<pre><code class="lang-python">rois_ = rois[inds, :].contiguous()

...

rois_expand_ = rois_expand[inds, :].contiguous()
</code></pre>
<p>It seems if your pytorch version is higher enough, noncontigous tensor can also be back-propagated for some operations.</p>

X_Zhao: <p><a class="mention" href="/u/naruto-sasuke">@Naruto-Sasuke</a>  Thanks Naruto-Sasuke, the error is same.</p>
Create multiple dataloader with transform using for loop

wasabi: <p>Hi guys. I’m trying to create multiple dataloaders using a for loop, and each of them uses a different transform. However, I notice that they always pick the transform of the last iteration.</p>
<pre><code class="lang-auto">train_loader = np.empty((2), dtype=np.object)
for i in range(2):
    train_loader[i]= torch.utils.data.DataLoader(
      torchvision.datasets.MNIST('./mnist/train/', train=True, download=True,
                                 transform=torchvision.transforms.Compose([
                                    torchvision.transforms.Lambda(lambda x:
                                        torchvision.transforms.functional.affine(
                                            img = x,
                                            angle = 90* i,
                                            translate = (0,0),
                                            scale = 1,
                                            shear = 0
                                        )
                                    ),
                                    torchvision.transforms.ToTensor(),
                                 ])
                                ),
      batch_size=1, shuffle=True)
</code></pre>
<p>Here I want train_loader[0] has 0 rotation and train_loader[1] has 90 degree rotation.<br>
But if you draw pictures from these two dataloaders</p>
<pre><code class="lang-auto">X, y = next(iter(train_loader[0]))
plt.imshow(X[0].squeeze(), vmin=0, vmax=1)
plt.show()
X, y = next(iter(train_loader[1]))
plt.imshow(X[0].squeeze(), vmin=0, vmax=1)
plt.show()
</code></pre>
<p>they both have 90 degree rotation. Am I doing something wrong?</p>

ptrblck: <p>This issue is created by the wrong usage of <code>i</code> inside the <code>lambda</code> definition, you would need to bind <code>i</code> for each created function.<br>
This should solve the issue:</p>
<pre><code class="lang-python">train_loader = np.empty((2), dtype=np.object)
for i in range(2):
    train_loader[i] = torch.utils.data.DataLoader(
      torchvision.datasets.MNIST('PATH', train=True, download=False,
                                 transform=torchvision.transforms.Compose([
                                    torchvision.transforms.Lambda(lambda x, i=i:
                                        torchvision.transforms.functional.affine(
                                            img = x,
                                            angle = 90* i,
                                            translate = (0,0),
                                            scale = 1,
                                            shear = 0
                                        )
                                    ),
                                    torchvision.transforms.ToTensor(),
                                 ])
                                ),
      batch_size=1, shuffle=False)
</code></pre>
Size error between RNN fully connected output and target

steve_moody: <p>If I understand it correctly, the Tensor size for an RNN is:<br>
(batch_size x sequence_length x n_features)</p>
<p>batch_size: total segments to handle<br>
sequence_length: number of time steps to unroll<br>
n_features: dimension of a one-hot-encoded vector of the vocab size.</p>
<p>I’ve got a dataset with 73461 total chars and a vocab size of 52. I want to pass in a sequence length of 100 and predict the 101st character. That would make my tensor ([73361 x 100 x 52])</p>
<p>I believe that my final layer will be a fully connected layer that will be ([hidden_dim x output_size]) which in this case will be 7336100 x 52.</p>
<p>However I get this error at the loss function (which I’m using CrossEntropy()):<br>
<code>ValueError: Expected input batch_size (7336100) to match target batch_size (73361).</code></p>
<p>Here are the shapes,</p>
<pre><code class="lang-auto">Model Architecture: RNN(
  (rnn): RNN(52, 12, num_layers=2, batch_first=True)
  (fc): Linear(in_features=12, out_features=52, bias=True)
)
DEBUG

 INPUT: torch.Size([73361, 100, 52])
MODEL OUTPUT: torch.Size([7336100, 52])
TARGET: torch.Size([73361])
</code></pre>
<p>Below is the model architecture.</p>
<pre><code class="lang-auto"># THe model
class RNN(nn.Module):

    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(RNN, self).__init__()

        # Params
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers

        # The layers
        # Takes (n_batches x seq_length x n_features)
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        # same size as the hidden_dim
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x):

        batch_size = x.size(0)

        # Initialize the hidden state first as a bunch of zeros.
        hidden = self.init_hidden(batch_size)

        # the outputs
        out, hidden = self.rnn(x, hidden)

        out = out.reshape(-1, self.hidden_dim)
        out = self.fc(out)

        return out, hidden

    def init_hidden(self, batch_size):
        hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)

        return hidden
</code></pre>
<p>I’m not sure how to shape my output to work.</p>
<p>Any help is greatly appreciated.</p>

ptrblck: <p>This line of code increases the batch size, which would be wrong:</p>
<pre><code class="lang-python">out = out.reshape(-1, self.hidden_dim)
</code></pre>
<p>Depending on your use case, you might want to pass the last time step to the linear layer via:</p>
<pre><code class="lang-python">out = self.fc(out[:, -1, :])
</code></pre>
<p>instead or reduce the temporal dimension in any other way.</p>
Sorting n-dimensional vectors

slavavs: <p>Friends, help me solve the problem. I cannot find a solution to this problem. I want to classify n dimensional vectors. Each vector is an object. When a new object appears, it is assigned a 128-dimensional identifier (vector). This vector finds a similar vector in its neighborhood. Next, the object tells how much it suits (likes) this nearest vector (another object). If the object fits, then you need to change the original vector to bring these objects closer together. If the object does not fit, then you need to move these vectors apart so that next time this object is selected with the least probability.</p>
<p>Here is a real life example. You slip your hand into the bag and choose a fruit. If you likes the fruit, then you put it close with your hand, if you don’t like it, then you move it aside so that you don’t take it next time. And this is how you sort the whole bag.</p>
<p>I cannot figure out how to implement this using a neural network. Help me.</p>
How to reuse graph even the parameters have been updated

Kam_hing_Pang: <p>for example：<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58204095d007709341aa74bc049887e5d27b5278.png" alt="image" data-base62-sha1="czB7i4cMYtNJ6hiPh9VZSW6gClG" width="425" height="354"></p>

ptrblck: <p>This behavior would yield to wrong gradients and will thus raise an error.<br>
What’s your use case that you want to recalculate the gradients using stale activation values?</p>
Inference while evaluation cost more time than train

Toby: <p>When I run my program and notice that it’s so slow. I try to compare the processing time in train and eval mode by this simple script</p>
<pre><code class="lang-python">import time
for  (image, target) in dataloader:
        image, target = image.to(device), target.to(device)
        b_size = image.size()[0]
        model.train()
        t1 = time.time()
        output = model(image)
        t2 = time.time()
        optimizer.zero_grad()
        with torch.no_grad():
            model.eval()
            t3 = time.time()
            output = model(image)
            t4 = time.time()
        print('Train:', t2-t1)
        print('Eval:', t4-t3)       
</code></pre>
<p>So surprise, except for the first iter, all other iterations, the evaluation mode takes 4 times longer time to process:</p>
<pre><code class="lang-auto">Train: 0.029598474502563477
Eval: 0.08679604530334473
</code></pre>
<p>I think the training phase should run longer because of the graph, the gradient, the running variable in batch normalization, etc. Any idea why is it?</p>
<pre><code class="lang-auto"></code></pre>

ptrblck: <p>If you are using a GPU, you would have to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code> as the timings would be wrong otherwise.</p>

Toby: <p>I see, now I can get the real processing time, thank you</p>
Indexing with LongTensor is inconsistent with that of BoolTensor

ydcjeff: <p>As of today, we can use long, byte or bool tensor as indices. but indexing with byte tensor is deprecated.<br>
But indexing with longtensor doesn’t give same answer as indexing with booltensor.</p>
<pre><code class="lang-python">x = torch.rand(3, 3)
&gt;&gt;&gt; x
tensor([[0.0857, 0.9867, 0.0330],
        [0.9295, 0.9787, 0.4835],
        [0.9376, 0.3851, 0.8934]])
y = torch.tensor([[1, 1, 1],
        [0, 0, 0],
        [0, 0, 0]])
&gt;&gt;&gt; y
tensor([[1, 1, 1],
        [0, 0, 0],
        [0, 0, 0]])
</code></pre>
<p>indexing with booltensor gives</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x[y.bool()]
tensor([0.0857, 0.9867, 0.0330])
</code></pre>
<p>indexing with longtensor gives</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x[y]
tensor([[[0.9295, 0.9787, 0.4835],
         [0.9295, 0.9787, 0.4835],
         [0.9295, 0.9787, 0.4835]],

        [[0.0857, 0.9867, 0.0330],
         [0.0857, 0.9867, 0.0330],
         [0.0857, 0.9867, 0.0330]],

        [[0.0857, 0.9867, 0.0330],
         [0.0857, 0.9867, 0.0330],
         [0.0857, 0.9867, 0.0330]]])
</code></pre>
<p>Is this an expected behavior? Is there something am i doing wrong?<br>
I want indexing with longtensor to give same result as indexing with booltensor.</p>
<p>Thank you in advance!</p>

ptrblck: <p>Yes, this is expected behavior and e.g. explained in <a href="https://numpy.org/doc/stable/reference/arrays.indexing.html#integer-array-indexing">numpy - integer array indexing</a> and <a href="https://numpy.org/doc/stable/reference/arrays.indexing.html#boolean-array-indexing">numpy - boolean array indexing</a>.</p>
Optimization.step() does not work

Toby: <p>Hi,</p>
<p>I have a problem with <code>optimization.step()</code>, my train code:</p>
<pre><code class="lang-python">previous = model.head[12].weight
    for i, (image, target) in enumerate(bar):
        image, target = image.to(device), target.to(device)
        b_size = image.size()[0]
        output = model(image)    
        loss = criterion(output, target)
        now = model.head[12].weight
        print('='*10)
        print(previous,now)
        print(model.head[12].weight.grad)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>The twelfth layer is  <code>nn.Linear(32, 1)</code><br>
The <code>previous</code> and <code>now</code> variables always return the same:</p>
<pre><code class="lang-auto">tensor([[ 0.0034, -0.2354, -0.1383, -0.2392, -0.5482,  0.4603,  0.1331,  0.3651,
         -0.1942, -0.1649,  0.2016,  0.3514,  0.2886, -0.0520,  0.1155, -0.3433,
         -0.3107,  0.1798,  0.0551, -0.0161,  0.5021,  0.1313,  0.3340,  0.2885,
          0.3380, -0.1972,  0.0067,  0.4465, -0.0814, -0.3551,  0.1053,  0.2180]],
       device='cuda:0', requires_grad=True)
</code></pre>
<p>Even that layer has the gradient value:</p>
<pre><code class="lang-auto"># Iter 1: None
# Iter 2:
tensor([[-0.4360, -0.2952, -0.6197, -0.0216, -1.0340,  0.7521,  0.2088,  0.7050,
         -0.6158,  0.4012,  1.0144, -0.5273,  0.6514,  0.2796, -0.6339, -0.8657,
         -0.2789,  0.2030, -0.0475,  0.4096,  0.9676,  0.3229,  0.5224,  0.1033,
          0.5484, -0.3745, -0.1743,  0.3179, -0.5385, -0.2508,  0.3502,  0.7884]],
       device='cuda:0')
# Iter 3:
tensor([[-0.5874, -0.7407, -0.6531,  0.2646, -1.6728,  0.4518,  0.1702,  1.0924,
         -1.2420,  0.1817,  0.7247,  0.9656,  0.0686,  0.1332,  0.3749, -1.8020,
         -2.0433,  0.4756, -0.8883, -0.4920,  1.8353, -0.5856, -0.2931,  0.8012,
          1.2388, -0.1963,  0.3465,  1.1411, -0.7416, -0.5985, -0.1804,  0.0322]],
       device='cuda:0')
</code></pre>
<p>Is it a bug, or I did something wrong?</p>

InnovArul: <aside class="quote no-group" data-username="Toby" data-post="1" data-topic="106393">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toby/48/29648_2.png" class="avatar"> Toby:</div>
<blockquote>
<p>The <code>previous</code> and <code>now</code> variables always return the same:</p>
</blockquote>
</aside>
<p>you are storing reference to the weight in <code>previous</code> and <code>now</code> variables.<br>
Store the cloned weight in <code>previous</code> (<code>previous = model.head[12].weight.clone()</code>). You will see the changes.</p>

Toby: <p>ah, my bad, thank you</p>
Changing the number of channels in a tensor

nafise: <p>Hi,<br>
I want to change the number of channels in my tensor:<br>
for example 128 to 256. I used the code below,<br>
a = torch.randn(2, 128, 221, 221)<br>
a = a.expand(-1,256,-1,-1)</p>
<p>but I got this error<br>
RuntimeError: The expanded size of the tensor (256) must match the existing size (128) at non-singleton dimension 1.  Target sizes: [-1, 256, -1, -1].  Tensor sizes: [2, 128, 221, 221]</p>

ptrblck: <p>You can only <code>expand</code> dimensions with a size of 1, so you would have to use <code>repeat</code> in your use case.</p>

nafise: <p>Thanks for response<br>
I did it<br>
a = torch.randn(2, 128, 221, 221)<br>
a = a.repeat(1, 256, 1, 1)<br>
but I got the message below:<br>
your session crashed after using all available RAM.</p>

nafise: <p>Thanks .I solve it.<br>
can you help me about this error:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 15.90 GiB total capacity; 14.80 GiB already allocated; 77.06 MiB free; 15.01 GiB reserved in total by PyTorch)</p>
<p>I used:<br>
import gc<br>
gc.collect()<br>
and<br>
torch.cuda.empty_cache()</p>
<p>but I also got the error.</p>

ptrblck: <p>I don’t think you’ve solved the issue as you are now running out of GPU memory instead of system RAM.<br>
<code>torch.repeat</code> expects to get the factors which are used to multiply the size to get the output tensor.<br>
Based on the initial description you should use <code>a.repeat(1, 2, 1, 1)</code>.</p>

nafise: <p>yes I corrected it.<br>
but this is another problem.</p>

ptrblck: <p>If <code>a.repeat(1, 2, 1, 1)</code> is creating the “out of memory” issue, you would have to use a smaller tensor or save some memory beforehand.</p>

nafise: <p>No, I get this error when I train the model</p>

ptrblck: <p>This error indicates that you are running out of memory on your GPU so you would still need to reduce the memory usage e.g. by lowering the batch size or by using <code>torch.utils.checkpoint</code> to trade compute for memory.</p>

nafise: <p>thanks a lot<br>
You are very good and you guide everyone in problems.<br>
I set the number of batch to 1, even reducing the data set. But I still have a problem. I still have to add other layers to this network.</p>
How to explicilty "see" that graph is released after backward?

murph213: <p>For my own understanding, I would like to explicitly see that the graph is freed after <code>optimizer.step()</code> is called, rather than relying on an implicit indication like the error we get if we call <code>backward()</code> again.  What user-facing indications do we have that the graph is freed?  I tried checking <code>.grad_fn</code> but they aren’t set to <code>None</code>.  In other words, what can I print before and after <code>backward()</code> to see plainly that the graph has been freed?</p>
<p>Most discussions about how the graph is freed are about how an error is raised if one calls <code>loss.backward()</code> again after <code>optimizer.step()</code> without <code>retain_graph</code>, etc.  My question is different.</p>
<pre><code class="lang-auto">f = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 1))
optimizer = torch.optim.Adam(f.parameters(), lr=0.1)
x = torch.randn(2, 10)
y = x @ torch.ones(10, 1)

for ii in range(10):
    pred = f(x)
    l = nn.MSELoss()(pred, y)
    print(pred.grad_fn)
    l.backward()
    print(pred.grad_fn)
    optimizer.step()
    print(pred.grad_fn)
</code></pre>
<p>All the prints yield <code>&lt;AddmmBackward object at 0x7f0ee95450f0&gt;</code></p>

ptrblck: <p>The graph object itself is not freed, if I’m not mistaken, but the intermediate activations.<br>
I’m not sure how you could check they are freed, as e.g. you would have to store them in forward hook, which would of course not delete them afterwards.</p>
Tensor manipulation: Indexing using a vector and a colon ":"

ChrisLiu2: <p>It might sound confusing, but I have a tensor T of size [N, C, A, B], and I would like to index in the third dimension. If along each row of dimension 3 we have a different index as recorded in a vector v, we can usually do: T[:, :, v, :]. However, what I want to do is use the vector v as the starting point and take part of the tensor, something like T[:, :, v:v+n, :]. It seems that this type of operation is not supported by PyTorch indexing. So the question is: besides using a for loop, what are other ways to implement this function? Thanks in advance.</p>
<p>The error I get when trying this is:</p>
<pre><code class="lang-auto">TypeError: only integer tensors of a single element can be converted to an index
</code></pre>

googlebot: <p>What output shape do you want? Generally, non-rectangular slicing requires gather(), i.e. specifying indexes explicitly. Or a loop.</p>