akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error in inference phase, after loading quantized model

khaidoan25: <p>After I used <code>torch.quantization.quantize_dynamic()</code> to quantize the original model, I saved and loaded the quantized model. But, when I ran inference, it returned this error. The original model still ran inference well, I don’t know why</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "inference.py", line 81, in &lt;module&gt;
    output = infer(args.text, model)
  File "inference.py", line 30, in infer
    mel_outputs, mel_outputs_postnet, _, alignments = model.inference(sequence)
  File "/media/tma/DATA/Khai-folder/Tacotron2-PyTorch/model/model.py", line 542, in inference
    encoder_outputs = self.encoder.inference(embedded_inputs)
  File "/media/tma/DATA/Khai-folder/Tacotron2-PyTorch/model/model.py", line 219, in inference
    self.lstm.flatten_parameters()
  File "/media/tma/DATA/miniconda3/envs/ttsv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 576, in __get
attr__
    type(self).__name__, name))
AttributeError: 'LSTM' object has no attribute 'flatten_parameters'
</code></pre>

supriyar: <p>Which pytorch version are you using? I think this is a known problem and should go away with 1.5 release.<br>
Could you wait for that or re-try with the nightly build and see if this issue goes away?</p>

khaidoan25: <p>I’m using pytorch version 1.4.0<br>
Hope next release will fix this issue.</p>

asmita: <p><a class="mention" href="/u/khaidoan25">@khaidoan25</a>, I am facing the same issue with Pytorch version - 1.5.1 . Have you been able to solve it?</p>

khaidoan25: <p>Actually, when you load your quantized model, you need to quantize your initial model first.</p>
<pre><code class="lang-auto">quantized_model = torch.quantization.quantize_dynamic(
    model, {nn.LSTM, nn.Linear}, dtype=torch.qint8
)         // Do s.t like this first before loading your quantized model
</code></pre>
Normalizing a multi-dimensional tensor

MUnique: <p>Hi everyone,<br>
I have a 4-dimensional tensor as (T, C, H, W) in which the last two dimensions correspond to the spatial size. What is the easiest way to normalize it over the last two dimensions that represent an image to be between 0 and 1?</p>

ptrblck: <p>If you want to normalize each image in isolation, this code should work:</p>
<pre><code class="lang-python">N, C, H, W = 2, 3, 5, 5
x = torch.randn(N, C, H, W)

tmp = x.view(N, C, -1)
min_vals = tmp.min(2, keepdim=True).values
tmp = (tmp - min_vals) / max_vals
max_vals = tmp.max(2, keepdim=True).values
tmp = tmp / max_vals

x = tmp.view(x.size())

for n in range(N):
    for c in range(C):
        x_ = x[n, c]
        print(n, c, x_.shape, x_.min(), x_.max())

&gt; 0 0 torch.Size([5, 5]) tensor(0.) tensor(1.)
0 1 torch.Size([5, 5]) tensor(0.) tensor(1.)
0 2 torch.Size([5, 5]) tensor(0.) tensor(1.)
1 0 torch.Size([5, 5]) tensor(0.) tensor(1.)
1 1 torch.Size([5, 5]) tensor(0.) tensor(1.)
1 2 torch.Size([5, 5]) tensor(0.) tensor(1.)
</code></pre>

MUnique: <p>Thanks, what if I had a few of these 4D tensors and wanted to normalize them dataset-based instead of in isolation?</p>

ptrblck: <p>I’m not sure I understand the use case completely.<br>
If you want to get the <code>mean</code> and <code>std</code> of the complete <code>data</code> tensor, you could use <code>x.mean(dim=[0, 1], keepdim=True)</code>.</p>
How to use a numpy function as the loss function without getting errors during run time?

Warrior: <p>For my task, I do not need to compute gradients. I am simply replacing <code>nn.L1Loss</code> with a numpy function (<code>corrcoef</code>) in my loss evaluation but I get the following error:</p>
<blockquote>
<p>RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.</p>
</blockquote>
<p>I couldn’t figure out how exactly I should detach the graph (I tried <code>torch.Tensor.detach(np.corrcoef(x, y))</code> but I still get the same error. I eventually wrapped everything using <code>with torch.no_grad</code> as follow:</p>
<pre><code class="lang-auto">with torch.no_grad():
    predFeats = self.forward(x)
    targetFeats = self.forward(target)
    loss = torch.from_numpy(np.corrcoef(predFeats.cpu().numpy().astype(np.float32), targetFeats.cpu().numpy().astype(np.float32))[1][1])
</code></pre>
<p>But this time I get the following error:</p>
<blockquote>
<p>TypeError: expected np.ndarray (got numpy.float64)</p>
</blockquote>
<p>I wonder, what am I doing wrong?</p>

Xiaoyu_Song: <p>To my understand:</p>
<pre><code class="lang-auto">np.corrcoef(x,y)
</code></pre>
<p>will return a <code>ndarray</code> type value.</p>
<pre><code class="lang-auto">torch.from_numpy(ndarray) → Tensor
</code></pre>
<p>This function requires a ndarray type</p>
<p>Can you print out the value of these codes below:</p>
<pre><code class="lang-auto">np.corrcoef(predFeats.cpu().numpy().astype(np.float32), targetFeats.cpu().numpy().astype(np.float32))[1][1])
</code></pre>
<p>to check out where is the problem.<br>
Because your error shows that this function returns a numpy.float64 rather than a numpy array</p>

Warrior: <p><a class="mention" href="/u/xiaoyu_song">@Xiaoyu_Song</a> You are right.  I got the answer <a href="https://stackoverflow.com/questions/54165651/how-to-use-a-numpy-function-as-the-loss-function-in-pytorch-and-avoid-getting-er" rel="nofollow noopener">here</a></p>

muammar: <p>Thanks for the link, but isn’t this breaking the graph?  I ask you because I read this <a href="https://discuss.pytorch.org/t/calculating-loss-with-numpy-function/28796" class="inline-onebox">Calculating loss with numpy function</a>.</p>
CPU memory Leakage in Custom Loss Function

K_B: <p>Hi all,<br>
I have a custom loss function in which another function X is called (which is calculation intensive). From the X function call I get an array of values that is then used to calculate loss. (Autograd is used).<br>
There is a decrease in the CPU memory before and after the X function is called. I am using a GPU for computations.<br>
Any idea why would there be a decrease in the CPU memory?</p>
<p>Thanks</p>
Converting Alexnet to 3d

Subhankar_Ghosh: <p>Hi!</p>
<p>I have a video classification task for which I had extracted the frames and used <a href="https://pytorch.org/docs/stable/_modules/torchvision/models/alexnet.html" rel="nofollow noopener">Alexnet</a> to do the classification on the frames. I believe my implementation was completely stupid since I was just doing image classification instead. Now I want to convert the Alexnet to a 3d convnet so that I can combine it with a lstm to do the classification.</p>
<p>Can you guys please suggest what changes I need to make to the existing Alexnet architecture?</p>
<p>Thanks!</p>

ptrblck: <p>I’m not sure how you would like to expand the shape of all parameters, which would be needed for a 3D model.<br>
Would feeding each frame separately to the CNN and then forwarding these features to an RNN work instead?</p>

Subhankar_Ghosh: <p>I don’t think so. My advisor was suggesting a conv3d + lstm architecture.</p>

ptrblck: <p>OK, in that case you could try to swap all <code>2d</code> layers for <code>3d</code> ones, e.g. <code>nn.Conv2d</code> for <code>nn.Conv3d</code>.<br>
This approach would reinitialize the model with random parameters.<br>
If you are planning on using the pretrained parameters, you would have to think about a valid approach how to expand to the depth dimension.</p>

Subhankar_Ghosh: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> ! Is there some approach you could suggest to use the pretrained weights from 2D? I am just changing the 2D kernels to 3D. So can we map the weights along the depth dimension as a starting point?<br>
Thanks again for the help!</p>

ptrblck: <p>You could unsqueeze the pretrained kernels to create a new depth dimension and just repeat the values:</p>
<pre><code class="lang-python">conv = nn.Conv2d(6, 12, 3)
weight = conv.weight
print(weight.shape)
&gt; torch.Size([12, 6, 3, 3])

weight = weight.unsqueeze(2).repeat(1, 1, weight.size(2), 1, 1)
print(weight.shape)
&gt; torch.Size([12, 6, 3, 3, 3])

conv3d = nn.Conv3d(6, 12, 3)
with torch.no_grad():
    conv3d.weight = nn.Parameter(weight)
</code></pre>
<p>It could be ambiguous, if the 2D kernels are not square kernels, but this depends on the pretrained model of course.</p>
How can we convert pykeops lazytensor back to a torch tensor in the forward layer of a model?

torchlight: <p><a href="https://www.kernel-operations.io/keops/python/api/pytorch/LazyTensor.html#pykeops.torch.LazyTensor" class="onebox" target="_blank" rel="nofollow noopener">https://www.kernel-operations.io/keops/python/api/pytorch/LazyTensor.html#pykeops.torch.LazyTensor</a></p>
Can't achieve reproducible results

Alon_Zolfi: <p>Hello everyone,</p>
<p>I’ve been trying to train my model and achieve reproducible results, but unfortunately, I’m not able to.<br>
I’ve taken this code from an old thread, but that still didn’t make it work:</p>
<p>def set_random_seed(seed_value, use_cuda=True):<br>
np.random.seed(seed_value)  # cpu vars<br>
torch.manual_seed(seed_value)  # cpu  vars<br>
random.seed(seed_value)  # Python<br>
os.environ[‘PYTHONHASHSEED’] = str(seed_value) # Python hash buildin<br>
if use_cuda:<br>
torch.cuda.manual_seed(seed_value)<br>
torch.cuda.manual_seed_all(seed_value)  # gpu vars<br>
torch.backends.cudnn.deterministic = True  # needed<br>
torch.backends.cudnn.benchmark = False</p>
<p>I also set all num_workers=0 in the data loaders to avoid a well-known problem there.</p>
<p>I’ve noticed a strange thing that happens after several training iterations (2 similar runs):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48875ec8d4f682afd92645ede04c107487d8998a.png" data-download-href="https://discuss.pytorch.org/uploads/default/48875ec8d4f682afd92645ede04c107487d8998a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48875ec8d4f682afd92645ede04c107487d8998a.png" alt="image" data-base62-sha1="alCpfESNF8fLrzdj3rTdL4zbQxA" width="690" height="349" data-dominant-color="EFEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1146×580 27.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>there is a slight change in the calculation that leads to totally different results.<br>
Did anyone face this problem? is it even fixable?</p>
<p>Thanks!</p>

tom: <p>Did you check functions you used against the <a href="https://pytorch.org/docs/master/notes/randomness.html" rel="nofollow noopener">randomness/determinism note</a>?<br>
If you’re using one of the functions listed there, nondeterminism is expected.</p>
Data Transformation

alameer: <p>I am using torchvision.transforms to transforms a dataset. I am wondering what other methods available to do similar data transforms:</p>
<pre><code class="lang-auto">transforms = Compose([RandomChoice([functional.hflip,functional.vflip])])
dataset_train.duplicate_and_transform(transforms) 
</code></pre>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/torchvision/transforms.html"><code>torchvision.transforms</code> docs</a> give you an good overview for all available transformations.<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

alameer: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot.</p>
How to create a 1D sparse tensors from given list of indices and values?

melody_marks: <p>I have a list of indices and values. I want to create a sparse tensor of size 30000 from this indices and values as follows.</p>
<pre><code class="lang-auto">indices = torch.LongTensor([1,3,4,6])
values = torch.FloatTensor([1,1,1,1])
</code></pre>
<p>So, I want to build a 30k dimensional sparse tensor in which the indices <code>[1,3,4,6]</code> are ones and the rest are zeros. How can I do that?</p>

Hmrishav_Bandyopadhy: <p>Hi,<br>
Try something like this:</p>
<pre><code class="lang-auto">sparse=torch.zeros([30000])
indices = torch.LongTensor([1,3,4,6])
values = torch.FloatTensor([1,1,1,1])
sparse[indices]=values
</code></pre>
<p>Does that solve your query?</p>

melody_marks: <p>Yes. Thanks for this solution. I also have another problem in which I need to index a whole batch of tensors like this.</p>
<pre><code class="lang-auto">a = torch.zeros((5, 30000))
indices = [[1,2], [1,4,5], [2,6], [6,8], [3,12, 110]]
a[indices] = 1
</code></pre>
<p>As you can see, I have a sequence of such sparse tensors. And a sequence of indices. We can definitely use a for loop to assign value 1 to each of the five 30000 dimensional vectors one by one, but its huge overhead (approx. 4 seconds in my program). Is there more efficient way to do this? Without using a for loop? Thanks</p>

Hmrishav_Bandyopadhy: <p>Hi,<br>
I think that you are trying to run the loop over the entire 30000 batch instead of running it five times. Try this:</p>
<pre><code class="lang-auto">for i in range(5):
     a[i,indices[i]]=1
</code></pre>
<p>Do tell me if this has a high overhead too as there would most probably be more efficient methods</p>
Implement the smoothing of 2D tensor

thecho7: <p>Greetings,</p>
<p>I have a question about smoothing 2D tensor with the conditions as below</p>
<blockquote>
<p>input = (1, 10, 20)</p>
</blockquote>
<p>Smoothing should be implemented along the sequence of length of 20 so that there are 10 individual smoothing operations. One more thing is to replicate the first and last values of a sequence (length of 20) like</p>
<blockquote>
<p>nn.Conv1d(padding_mode=‘replicate’)</p>
</blockquote>
<p>For this, I tried to</p>
<blockquote>
<p>input = torch.unsqueeze(input, dim=0)<br>
out = nn.Conv1d(1, 1, kernel_size=(1, 3), padding=(0, 1), padding_mode=‘replicate’, bias=False)(input)</p>
</blockquote>
<p>But it doesn’t work with some error.<br>
It can be implemented in either inside of network or outside of network.<br>
I need your help<br>
Thanks</p>

ptrblck: <p>What kind of error are you seeing?<br>
Would you like to apply a specific pre-defined kernel for the “smoothing” operation?<br>
If so, you should be able to define the kernel and apply a grouped convolution to the input.<br>
The functional API would probably be a bit easier than using the <code>nn.Conv2d</code> module.</p>
Torch.cuda.is_available() is False in conda env

111226: <p>Hi, I’m was using PyTorch 1.5.1 version and CUDA 10.2.<br>
But for version compat, I had to make a new conda environment and change PyTorch version to 1.4.0.<br>
In new conda env, <code>torch.cuda.is_available()</code> returns false but in the base env, it returns true.<br>
How should I do?<br>
Thanks!</p>

ptrblck: <p>Maybe you’ve installed the CPU only version in your new environment.<br>
Could you check the installed packages and make sure that <code>cudatoolkit</code> was also installed (assuming you’ve used the conda binaries)?</p>
How to quantize a custom model?

Nilakshan_Kunanantha: <p>Sequential models are not allowed to quantize l.How can I quantize a custom model?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html">Dynamic Quantization on an LSTM Word Languange Model</a> or the <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html">Static Quantization with Eager Mode</a> tutorials might be a good starter.<br>
The first tutorial seems to use a custom model, which seems to fit your use case.</p>
Pytorch tensor stride - how it works

aerinykim: <p>PyTorch doesn’t seem to have documentation for <code>tensor.stride()</code>.<br>
Can someone confirm my understanding?</p>
<p>My questions are three-fold.</p>
<ol>
<li>
<p>Stride is for accessing an element in the storage. So stride size will be the same as the dimension of the tensor. Correct?</p>
</li>
<li>
<p>For each dimension, the corresponding element of stride tells how much it takes to move along the 1-dimensional storage.</p>
</li>
</ol>
<p>For example:</p>
<pre><code class="lang-auto">In [15]: x = torch.arange(1,25)

In [16]: x
Out[16]:
tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18
, 19, 20, 21, 22, 23, 24])

In [17]: a = x.view(4,3,2)

In [18]: a
Out[18]:
tensor([[[ 1,  2],
         [ 3,  4],
         [ 5,  6]],

        [[ 7,  8],
         [ 9, 10],
         [11, 12]],

        [[13, 14],
         [15, 16],
         [17, 18]],

        [[19, 20],
         [21, 22],
         [23, 24]]])

In [20]: a.stride()
Out[20]: (6, 2, 1)
</code></pre>
<ol start="3">
<li>How does having this information help perform tensor operations efficiently? Basically this is showing the memory layout. So how does it help?</li>
</ol>

ptrblck: <ol>
<li>
<p>The stride will have the same number of values as the number of dimensions. E.g. if you are dealing with a tensor with 4 dimensions, <code>tensor.stride()</code> will return 4 values.</p>
</li>
<li>
<p>Yes, that’s correct and <a href="https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107/2">this post</a> gives another example with contiguous vs. non-contiguous tensors.</p>
</li>
<li>
<p>The stride is used in the backend for indexing, which can be used if you want to directly access specific elements in the memory block.</p>
</li>
</ol>
Printing out an individual class accuracy?

gamebo5000: <p>I want to be able to print out the accuracy of one of the classes I am training my model on. How would I go about doing that?<br>
I have a confusion report showing me everything at the end of an epoch, but I would prefer to just have the recall data displayed.</p>
<pre><code class="lang-auto">with torch.no_grad():
        end = time.time()
        for i, (images, target) in enumerate(val_loader):
           
            if args.gpu is not None:
                images = images.cuda(args.gpu, non_blocking=True)
                
            if torch.cuda.is_available():
                target = target.cuda(args.gpu, non_blocking=True)
                
            # compute output
            output = model(images)
            print("is it here?")
            loss = criterion(output, target)

            # measure accuracy and record loss
            acc4, acc5 = torch.max(output.data, 1)
            acc1, acc3 = accuracy(output, target, topk=(1, 3))
            losses.update(loss.item(), images.size(0))
            top1.update(acc1[0], images.size(0))
            top3.update(acc3[0], images.size(0))

            # measure elapsed time
            batch_time.update(time.time() - end)
            end = time.time()
            test_acc.append(acc1)
            test_losses.append(loss)
            log(train_acc,train_loss,test_acc, test_losses)
            if i % args.print_freq == 0:
                progress.display(i)

        # TODO: this should also be done with the ProgressMeter
        print(' * Acc@1 {top1.avg:.3f} Acc@3 {top3.avg:.3f}'
              .format(top1=top1, top3=top3))
target = target.cpu()
        acc5 = acc5.cpu()
        print(classification_report(target.view(-1), acc5.view(-1), target_names=class_names))</code></pre>

ptrblck: <p>You could try to create a confusion matrix (e.g. using the scikit-learn implementation) and calculate the per-class accuracy (and print only the desired class) or other metrics for this class.</p>
Train specific classes of dataset

Cesar_Sotelo: <p>Hi. I’m new on the deep learning world, so I decided to build a model for pneumonia detection for my country. I’ve selected a well known dataset <a href="https://www.kaggle.com/nih-chest-xrays/data" rel="nofollow noopener">NIH Chest X-ray</a>. It have 14 different classes but I interested only in two classes: Consolidation and Infiltration. I’ll use Data augmentation and transfer learning for the training. Could you please give me an example how train only these 2 clases? I’m thinking about read de csv label database delete images who not have the finding label.</p>
<p>Thank you in advance</p>

ptrblck: <aside class="quote no-group" data-username="Cesar_Sotelo" data-post="1" data-topic="90451">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cesar_sotelo/48/27091_2.png" class="avatar"> Cesar_Sotelo:</div>
<blockquote>
<p>I’m thinking about read de csv label database delete images who not have the finding label.</p>
</blockquote>
</aside>
<p>This sounds like a valid approach.<br>
If you are not interested in training the other classes at the moment, I would either remove them or create a separate smaller dataset, which only contains the desired class samples.</p>
How to create a dataset object for training and testing using a Dataset class

Ilyas_kab: <p>Hi everyone!<br>
I have this Dataset class:<br>
“”"<br>
class Dataset(Dataset):</p>
<pre><code># Constructor
def __init__(self, train = True):
    
    data = pd.read_csv('https://s3.us.cloud-object-storage.appdomain.cloud/cf-courses-data/CognitiveClass/DL0110EN/ML03210EN_Final_Assignment/LoLMatchData.csv')
    
    if (train):
        self.x = torch.tensor(data.iloc[0:7903, :].drop(['blueWins'], axis=1).values, dtype=torch.float)
        # standardizing the data
        self.x = (self.x - self.x.mean(dim=0))/self.x.std(dim=0)
        self.y = torch.tensor(data.loc[0:7903, 'blueWins'].values, dtype=torch.float).reshape((7904,1))
        self.len = self.x.shape[0]
    else:
        self.x = torch.tensor(data.iloc[7904:, :].drop(['blueWins'], axis=1).values, dtype=torch.float)
        # standardizing the data
        self.x = (self.x - self.x.mean(dim=0))/self.x.std(dim=0)
        self.y = torch.tensor(data.loc[7904:, 'blueWins'].values, dtype=torch.float).reshape((1975,1))
        self.len = self.x.shape[0]
        

# Get the length
def __len__(self):
    return self.len

# Getter
def __getitem__(self, idx):
    
    x = self.x[idx]
    
    y = self.y[idx]

    return x, y
</code></pre>
<p>“”"</p>
<p>I need to use this Dataset class to create a dataset object for training and testing. Not to mention set the training parameter to the correct value.</p>
<p>I would be pleased if anyone could help me with this task.<br>
Thank you in Advance!</p>

ptrblck: <p>Based on your code, you could probably create two different datasets and set the <code>train</code> argument appropriately:</p>
<pre><code class="lang-python">train_dataset = Dataset(train=True)
val_dataset = Dataset(train=False)
train_loader = DataLoader(train_dataset)
val_loader = DataLoader(val_dataset)
</code></pre>
<p>Let me know, if I misunderstood the question.</p>
RuntimeError: expected device cuda:0 but got device cuda:1

111357: <p>My model includes two models, a pre-trained model A --&gt; model B.<br>
I have trained model A on <code>GPU0</code>.<br>
I fixed the model A and train the model on <code>GPU1</code>.<br>
But I got the following error:<br>
RuntimeError: expected device cuda:0 but got device cuda:1<br>
Can anyone help me? Thanks a lot.</p>

ptrblck: <p>Could you explain your use case a bit more, i.e. which model is located on which GPU?<br>
This error is raised, if you are trying to apply an operation using tensors, which are located on different devices. E.g. if <code>modelA</code> is on GPU0 and <code>modelB</code> on GPU1, you would have to make sure the output of <code>modelA</code> is transferred to GPU1 before passing it to the model.</p>
Question about source code in faster_rcnn.py

brifuture: <p>Hi, I have a question about the default RPNHead constructor, following is the source code from <code>torchvision==0.6.1</code>, located at <code>models/detection/faster_rcnn.py</code> around line 186:</p>
<pre><code class="lang-auto">if rpn_head is None:
    rpn_head = RPNHead(
        out_channels, rpn_anchor_generator.num_anchors_per_location()[0]
    )
</code></pre>
<p>My question is: Why the RPNHead is constructed with anchors as <code>rpn_anchor_generator.num_anchors_per_location()[0]</code> instead of <code>sum(rpn_anchor_generator.num_anchors_per_location())</code>?</p>
<p>For example: if I constructed rpn_anchor_generator with following code:</p>
<pre><code class="lang-auto">anchor_sizes = ((32,), (64,), (128,), (256,), (512,))
aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)
rpn_anchor_generator = AnchorGenerator(
    anchor_sizes, aspect_ratios
)
</code></pre>
<p>I should get 15 anchors per location on the original Image? but with <code>rpn_anchor_generator.num_anchors_per_location()[0]</code> I could get only 3 anchors  per location.</p>

brifuture: <p>Fine, I figured out the reason, here is the source code from <code>torchvision.models.detection.rpn.RPNHead</code>:</p>
<pre><code class="lang-auto">    def forward(self, x):
        # type: (List[Tensor])
        # print([i.shape for i in x])
        logits = []
        bbox_reg = []
        for feature in x:
            t = F.relu(self.conv(feature))
            logits.append(self.cls_logits(t))
            bbox_reg.append(self.bbox_pred(t))
            print("RPN", self.cls_logits(t).shape)
            print("RPN", self.bbox_pred(t).shape)
        return logits, bbox_reg
</code></pre>
<p>The anchor_sizes defined above <code>anchor_sizes = ((32,), (64,), (128,), (256,), (512,))</code> will be iterated by the for loop, so the RPNHead only needs <code>rpn_anchor_generator.num_anchors_per_location()[0]</code> as the num_anchors because all anchor_sizes share the same ratios.</p>
How to set PyTorch saves its weights as double rather floats

Gavan: <p>I had to convert my  CLDNN model from <strong>Keras to Torch.</strong> for work reasons.<br>
In order to restore the performance of the model on Keras, the model structure, hy-params,the initialization of weight and bias are still done addcording to my CLDNN model.<br>
The current result is that the classifcation result of torch is <strong>3% less than the result on keras</strong>. The performance degardation is fatal in terms of my work.</p>
<p><a href="https://gereshes.com/2019/06/24/how-to-transfer-a-simple-keras-model-to-pytorch-the-hard-way/" rel="nofollow noopener">Keras saves it’s weighted as doubles while PyTorch saves its weights as floats and there is a small amount of truncation error</a></p>
<p>And my question how to set  save my model’s weights as double rather float?</p>

Sagnik_Mukherjee: <aside class="quote" data-post="1" data-topic="886">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yunjey/48/1386_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-convert-models-parameter-data-type-to-double/886">How can i convert model's parameter data type to Double?</a> 
  </div>
  <blockquote>
    Default data type of parameters is Float. 
I want to convert it to Double. 
How can i do this?
  </blockquote>
</aside>

<p>Hi Gavan,<br>
Can you verify if this discussion addresses the same issue as yours? In that case I think it has been resolved earlier by <a class="mention" href="/u/alband">@albanD</a></p>

Gavan: <p>Thank you! I tried the method in the link and it works.<br>
However, the performance of the model has not improve yet. <img src="https://discuss.pytorch.org/images/emoji/apple/fearful.png?v=9" title=":fearful:" class="emoji" alt=":fearful:"></p>

Sagnik_Mukherjee: <p>Is it possible that you are overlooking some small error? Like I was not calling model.eval() couple days back, and It was giving around 5% more error. Please verify.</p>

Gavan: <p>I checked all of my model, including the code of the traning and validation process.</p>

Gavan: <p>And there is nothing wrong with that. Thank you.</p>
Extending n-dimensional Tensors

Shanto_Islam: <p>I have tensors <code>t1</code> and <code>t2</code> respectively with same size and dimensions. I would want <code>tx</code> to extend <code>t1</code> and then <code>t2</code>.</p>
<pre><code class="lang-auto">t1 = torch.Tensor([[5, 2], [6, 3], [7, 9]])
t2 = torch.Tensor([[4, 5], [1, 4], [3, 8]])
</code></pre>
<p>And I would want tensors <code>tx</code> to be like</p>
<pre><code class="lang-auto">&gt;&gt; tensor([5, 2], [6, 3], [7, 9], [[4, 5], [1, 4], [3, 8])
</code></pre>
<p>Does tensors supports me to do this? If so, then how?</p>

Hmrishav_Bandyopadhy: <aside class="quote no-group" data-username="Shanto_Islam" data-post="1" data-topic="90551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shanto_islam/48/22963_2.png" class="avatar"> Shanto_Islam:</div>
<blockquote>
<p><code>tensor([5, 2], [6, 3], [7, 9], [[4, 5], [1, 4], [3, 8])</code></p>
</blockquote>
</aside>
<p>I am guessing this should be<br>
<code>tensor([[5, 2], [6, 3], [7, 9], [4, 5], [1, 4], [3, 8]])</code></p>
<p>In that case, you can use:</p>
<p><code>tx=torch.cat([t1,t2],dim=0) </code></p>
Using apex for loading checkpoints

887574002: <p>Hi, I have a model that I used apex for implementing half-precision of the model. I wanted to know when I load the checkpoints should I load the checkpoints of apex?</p>

ptrblck: <p>If you want to continue the training in <code>apex</code>, then you would have to restore the apex <code>state_dict</code>.<br>
However, we recommend to check out the native mixed-precision implementation as described <a href="https://pytorch.org/docs/master/amp.html">here</a>, which provides a smoother experience than apex.<br>
Note that you would have to install the nightly binaries or build from source to use it.</p>

887574002: <p>Thank you for your answer. Is there anything wrong with using Apex?<br>
Why do you recommend me to use the native mixed-precision implementation?<br>
Actually, I am working with 3d images that are high dimensional. So, I frequently encounter the OOM error.<br>
By using apex everything was fine until some moments ago. But, again after some changes in my code I do not know why I am getting OOM error!</p>
<p>My second question is that, if mixed precision and DistributedDataParallel, can be used at the same time and using both of them can solve the problem of OOM error for larger batches? I tried DataParallel before and it didn’t help.<br>
At the moment, I am working with batches of size 2, but in this batch size I do not observe good behavior of loss function and the loss function dose not decrease. It increase actually. After googling a lot I found that it’s due to batch size and dateset that I have. I tested it for 2D and it worked fine but for 3D I didn’t get good result.</p>

ptrblck: <p><code>apex.amp</code> was developed as the first implementation for mixed-precision training before <a class="mention" href="/u/mcarilli">@mcarilli</a> implemented the native implementation.<br>
The main advantage is that you can use it by installing the (nightly) binaries without building <code>apex</code> as an extension (which is not always straightforward, if it’s your first time). In the long run we’ll move our focus on native amp and deprecate <code>apex.amp</code>.<br>
<a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">Here</a> is also a more detailed description.</p>
<p>It’s hard to tell, which of these changes created the OOM issue suddently without seeing the code.</p>
Tricks to improve model training

thyr: <p>We have occasionally seen many tricks which were used in various papers without any particular explanation.</p>
<p>For eg: in <code>Attention is all you need</code> you see the embedding multiplied by sqrt(embed_dim)</p>
<p>Are there any articles or notes which talks about various techniques that people have tried and what is the motivation behind these adhoc tuning tricks?</p>

Unity05: <p>Hi <a class="mention" href="/u/thyr">@thyr</a>,</p>
<p>well, there are some ‘general’ tricks for training, just as the ‘Cosine Learning Rate Decay’ what is actually just a smoother decay as well as learning rate warm up (to prevent unwanted ‘weight chaos’ caused by starting with a too high learning rate; therefore learning rate warmup kind of aligns). For quite deep models in particular, you may want to read about ‘residual blocks’ (<a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1512.03385.pdf</a>). However, with respect to NLP, there is ‘attention’ for seq2seq models (as you have already mentioned) as well as ‘beam search’, for instance. I can recommend with good conscience ‘Andrew Ng’ for many methods. Depending on your task, you may also want to consider implementing ‘MTL’.</p>
<p>Regards,<br>
Unity05</p>
Unexpected cuda behaviour when registering buffers

Omer_B: <p>Hello all,<br>
I’ve encountered an issue while trying to register buffers from another module to a custom made module while doing the same thing with parameters (registering parameters) the method behaved as expected.</p>
<p>Here is a smaple of my code</p>
<pre><code class="lang-python">import torch
import torchvision.models

class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()

resnet18_model = torchvision.models.resnet18()
my_model = MyModule()

print("Checking Buffers")

for buffer_name, buffer in list(resnet18_model.named_buffers()):
    my_model.register_buffer(buffer_name.replace('.', '_'), buffer)
resnet18_model.cuda()
for buffer in list(my_model.buffers()):
    print(buffer.is_cuda)

print("Checking Parameters")

for param_name, param in list(resnet18_model.named_parameters()):
    my_model.register_parameter(param_name.replace('.', '_'), param)
resnet18_model.cuda()
for param in list(my_model.parameters()):
    print(param.is_cuda)

</code></pre>
<p>My expected behaviour of this code was as follows :<br>
after loading the resnet18 module into cuda, and after registering its parameters and buffers into my module I expected both of them to be in cuda, as you can see i’ve checked if is_coda flag is on and my results where as follows:</p>
<pre><code class="lang-auto">Checking Buffers
False 
False
.
.
.
Checking Parameters
True
True
.
.
.
</code></pre>
<p>it seems as if the <code>register_buffer</code> uses some sort of  “copying” the input buffers and not copying its address (reference) to the memory, while <code>register_parameters</code> is acting as expected.</p>
<p>Is it a bug? or i’m missing something?<br>
Is there a different way of registering the buffers pointers into other modules?</p>
Calculate final model accuracy at the end of training

alameer: <p>I would like to calculate the model accuracy at the end of the training. At the moment the test mean error is calculated at the end of each epoch. I’m saving the test mean error at the end of each epoch. Then calculating the accuracy for the previous epochs. The code doesn’t seem to calculate the accuracy for some reason. It only print the test mean error. Please could someone shed some light to what i am doing wrong?</p>
<pre><code class="lang-auto">model.eval()
    for images, paths in tqdm(loader_test):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # logging:
        loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)
        
        global_step += 1

    average_accuracy = 0 
    mean_test_error = mean_test_error / len(loader_test)
    writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
    average_accuracy += mean_test_error
    average_accuracy = average_accuracy /len(loader_test)
    print("Average accuracy: %f" % average_accuracy)
    print("Test count error: %f" % mean_test_error)
    if mean_test_error &lt; best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)

</code></pre>