akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Small error in distributions.categorical.Categorical documentation

username: watteo
cooked: <p>For probably unjustified reasons, I want to parameterise my <code>torch.distributions</code> objects using the natural parameters, which for a categorical are the log-probabilities.</p>
<p>I noticed that in the <a href="https://pytorch.org/docs/stable/distributions.html#categorical" rel="nofollow noopener">documentation</a>, only the normalised probabilities (<code>probs</code>) and log-odds (<code>logits</code>) are supported, which seemed strange for a categorical. But on further inspection, it seems that the <code>logits</code> parameter actually refers to the log-probabilities, log p(x_i), <em>not the log-odds</em>, log p(x_i)/(1-p(x_i)). If it’s the case that <code>logits</code> actually refers to the (unnormalised) log-probabilities, maybe the documentation should be updated to avoid confusion.</p>
<p>I know this is a pretty small error, but I did spend a non-zero amount of time trying to work out a good way to make my <code>catagorical.Categorical</code> accept log-probabilities, in a way that avoids unnecessary exponentiations. Could save someone else a few minutes too!</p>
Input shape for pretrained model (models.detection.keypointrcnn_resnet50_fpn)

username: venub
cooked: <p>import torch<br>
from torchvision import models<br>
from torchsummary import summary</p>
<p>model = models.detection.keypointrcnn_resnet50_fpn(pretrained=True)<br>
model .eval()<br>
summary(model , (3, 224, 224))</p>
<p>throws<br>
AttributeError: ‘ImageList’ object has no attribute ‘size’</p>
<p>Question is what is the valid input_size for the summary api<br>
Intent is to print the summary of the pre_trained_model</p>
Different results on single vs multiple output network

username: donJuan
cooked: <p>So I’m trying to estimate a network with 4 real valued outputs. The network uses dropout on all the layers except the last one, the reason for this is so I can capture the model uncertainty (bayesian approximation). The problem is that the network has poor performance when it’s predicting 4 output but get adequate results when the output is only one.</p>
<p>I have done sanity checks i.e. checking the complexity of the network so it can overfit on a small set of observations. Also normalizing both input and output. The std of the outputs differ much.</p>
<p>Anyone has faced similar problem? I know this is not a direct question regarding the torch framework but hope I can get some answers. Trained with MSE loss and adam. The network is given by</p>
<pre><code class="lang-auto">class NETWORK(nn.Module):

	def __init__(self, layers = [40, 2048, 2048, 1024, 1024, 4], droprate = 0.1):
		super(NETWORK, self).__init__()
		self.p = droprate
		modules, n = [], len(layers) - 1
		for i in range(n):
			modules.append(nn.Linear(layers[i], layers[i + 1]))
			if i + 1 != n:
				modules.append(nn.Dropout(self.p))
				modules.append(nn.ReLU())
		self.net = nn.Sequential(*modules)
	def forward(self, x):
		return self.net(x)
</code></pre>
Multi channels linear layer

username: vHitsuji
cooked: <p>Hi everyone,<br>
I’m trying to train a model which has many identical channels.<br>
Let’s give a toy example:<br>
I have one tensor divided in n tensors of the same shape and I want to respectively forward these n tensors to n different linear modules.<br>
I concatenate the output and obtain one tensor.<br>
The linear modules have the same shape (input shape and output shape) but don’t share parameters .<br>
We may say that I have n isolated channels.</p>
<p>When I run the experiment I observe that my Gpu use only 20% of power.<br>
Every linear module is very small.<br>
I think that the machine lose time between each computation.</p>
<p>Is there anyway to compute the output of isolated linear modules at once ?<br>
Something like multi channels linear module ?</p>
<p>Thank you.</p>

username: Felipe_Mello
cooked: <p>Hi, did you find a solution to that? Thanks!</p>
Grid Sample Along another axis

username: soulslicer
cooked: <p>Lets say I have a tensor of size [B * 64 * 100 * 100]. Generally, if i wanted to apply a flow field on this tensor along the dimensions 2 and 3, I can use grid sample directly.</p>
<p>But in my case, I have flow field that needs to is shift values in the Channel/64 space. To use my current code directly, could I create a flow image with only change in one direction, rearrange the tensor into [B * 100 * 100 * 64] then apply the field?</p>
QAT: evaluation during training

username: mg1371
cooked: <p>Hi,</p>
<p>For QAT, do we need to use .convert() at the end of each epoch to fully convert and evaluate on validation data or the model already fake quantizes the forward pass (and would produce the same results as converted model)?</p>
<p>Many Thanks!</p>

username: supriyar
cooked: <p>It is not necessary to convert at the end of every epoch. The fake quantized model is expected to be very close to the final converted model, so one conversion at the end should suffice. We are in the process of aligning the numerics better between fake quant and converted model so if you are seeing some differences they should get better.</p>
Figure out model architecture

username: Isaac_Kargar
cooked: <p>Hi, I have a saved params.pkl file for my trained model but I don’t know the architecture. How can I somehow visualize or find out the network architecture?</p>

username: babababadukeduke
cooked: <p>The params.pkl file contains a dictionary of model parameter values, and contains no information about the model architecture itself. In some case, if standard model architecture was use and the params were names aptly, one can guess the architecture of the model from the keys of the dictionary. But this is approach is completely based on luck.</p>
RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 274877906944 bytes. Buy new RAM!

username: ProGamerGov
cooked: <p>I’m trying to replicate a NumPy function from here: <a href="https://github.com/google/deepdream/blob/master/dream.ipynb" rel="nofollow noopener">https://github.com/google/deepdream/blob/master/dream.ipynb</a></p>
<p>From the link, the code is: “trying to maximize the dot-products between activations of current image, and their best matching correspondences from the guide image.”</p>
<p>When I run the following code, or any variation that has <code>torch.mm(x.T, y)</code>, I get a out of memory error:</p>
<pre><code class="lang-auto">def guide_output(input, guide):
    x = input.squeeze(0)
    y = guide.squeeze(0)
    ch = x.shape[0]
    x = x.reshape(ch,-1)
    y = y.reshape(ch,-1)
    A = torch.mm(x.T, y) # compute the matrix of dot-products with guide features
    x.reshape(ch,-1)[:] = y[:,A.argmax(1)] # select ones that match best

input = torch.randn(3,512,512).unsqueeze(0)
guide = torch.randn(3,512,512).unsqueeze(0)
output = guide_output(input, guide)
</code></pre>
<p>The error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 76, in &lt;module&gt;
    output = guide_output(input, guide)
  File "test.py", line 63, in guide_output
    A = torch.mm(x.T, y) # compute the matrix of dot-products with guide features
RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 274877906944 bytes. Buy new RAM!
</code></pre>

username: kshitij
cooked: <pre><code class="lang-python">import torch

def guide_output(input, guide):
    x = input.squeeze(0)
    y = guide.squeeze(0)
    ch = x.shape[0]
    x = x.reshape(ch,-1)
    y = y.reshape(ch,-1)
    print(x.T.shape, y.shape)
    A = torch.mm(x.T, y) # compute the matrix of dot-products with guide features
    x.reshape(ch,-1)[:] = y[:,A.argmax(1)] # select ones that match best

input = torch.randn(3,512,512).unsqueeze(0)
guide = torch.randn(3,512,512).unsqueeze(0)
output = guide_output(input, guide)
</code></pre>
<p>The printed shape is <code>torch.Size([262144, 3]) torch.Size([3, 262144])</code>. Thus the size of the output of your <code>torch.mm</code> is a tensor of shape <code>(262144,262144)</code> which requires<br>
<code>274877906944 bytes i.e. 274.877906944 Gigabytes</code>. And since the system can’t provide it, we see the error.</p>
Problem in Pytorch implementation in Android

username: Anusha_ali
cooked: <p>I needed some guidance regarding pytorch implementation in android. I am unable to figure out that how can we extract value from the iValue array. I tried using .toStr() and .toDouble() functions of iValue but it’s giving the error   “java.lang.IllegalStateException: Expected IValue type 5, actual type 2” (in case of .toDouble()) and "java.lang.IllegalStateException: Expected IValue type 6, actual type 2: (incase of .toStr()). We are getting the IValue[] by using following piece of code<br>
<strong>IValue[] outputTensor=module.forward(IValue.from(inputTensor)).toTuple()</strong><br>
We have also tried to replace .toTuple() in the above mentioned code to toTensor() but our issue is not getting resolved!</p>
Comparing GPU speeds

username: KUSHAL_JAIN
cooked: <p>I am trying to train a captioning model in a multi-GPU setting using PyTorch. I wanted to know what GPU could give me the least training time. I have access to 4xTesla M60 GPUs. For 512 videos, 1 epoch on M60 took around 26 minutes. My dataset is much larger (~6k videos with each video having 20 captions).<br>
When I researched about M60, I found that these are old GPUs and that Nvidia has come up with better architectures since then. I rented GPUs on vast.ai to try some of newer ones.</p>
<ul>
<li>Training on 256 videos. 1 epoch. 4 x RTX 2080Ti . Time:26 minutes. This turned out to be twice as worse than M60.</li>
<li>Then someone pointed me to the following issue <a href="https://github.com/pytorch/pytorch/issues/22961" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/22961</a>.<br>
I tried training on 4xGTX 1080Ti. 1 epoch 256 videos. Time:14 minutes. This certainly worked better than 2080Ti.</li>
</ul>
<p>I wanted to know how these GPUs actually compare with each other. Is M60 superior to all in terms of speed? The TFLOPS of the newer GPUs is better than M60. Is that parameter enough to compare training time? Or are there more parameters that should be considered.<br>
Thank you.</p>
Using torchvision resnet models and removing final layers

username: Bruce_Muller
cooked: <p>I’m using the pre trained torchvision resnet50 model but I’m confused if my implementation this will work properly. I want to remove the FC and adaptive avg pooling layers. Is writing it out like this sensible? I’m concerned this way will chain together the relevant layers but without having the identity shortcut connections. Unless children() retains the relevant calls in forward?</p>
<pre><code class="lang-auto">class ResNet(nn.Module):
    def __init__(self, hparams):
        super(ResNet, self).__init__()
        
        self.stripped_body = nn.Sequential(*list(models.resnet50(pretrained=True).children())[:-2])
        
    def forward(self, img):     
  
        x = self.stripped_body(img)
        return x
</code></pre>

username: AladdinPerzon
cooked: <p>I’m not entirely sure so perhaps someone can confirm, but I think your way should work and not remove the skip connections. I feel you could also use something like this:</p>
<pre><code class="lang-auto">model = models.resnet50(pretrained=True)
model.avgpool = nn.Identity()
model.fc = nn.Sequential(nn.Linear(input_size, num_classes))
</code></pre>
<p>Edit: Or I guess in your case, setting</p>
<pre><code class="lang-auto">model.fc = nn.Identity()
</code></pre>
<p>as well.</p>

username: Bruce_Muller
cooked: <p>Thanks <a class="mention" href="/u/aladdinperzon">@AladdinPerzon</a> I’m giving that a go to see if there’s a difference in results</p>

username: AladdinPerzon
cooked: <p>That’s for fine tuning right? If you wish to train the entire network but only remove the last layers you wouldn’t want to have <code>param.requires_grad = False</code></p>

username: Bruce_Muller
cooked: <p>About fine tuning, is requires_grad=False on the parameters the expected method to freeze them?</p>
<p>I’ve been using that with a VGG feature extractor for a perceptual loss, but I was wondering if only passing the required parameters to the optimiser is the best way? or both?! <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
Why GPU out of memory at the second iteration

username: crissallan
cooked: <p>Hello,<br>
I used a UNet model for image segementation task. After I built up the model , I tried to use a dummy tensor to test the forward() process of the model. The code script is shown below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d2c0354965aff3fd918d3672a53f67712259f6a.png" alt="code" data-base62-sha1="dieHxdrFtekyTUbVCl9GEb8ewpI" width="558" height="194"></p>
<p>However, after the first iteration when console output the size of output tensor, I met a GPU OOM error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c2c7018739aa9b7be89eb73409996c16050bef1.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c2c7018739aa9b7be89eb73409996c16050bef1" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c2c7018739aa9b7be89eb73409996c16050bef1_2_690x353.png" alt="error" data-base62-sha1="aRRuIxyaqjJKy2OLsDTuJ1mrXqh" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c2c7018739aa9b7be89eb73409996c16050bef1_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c2c7018739aa9b7be89eb73409996c16050bef1_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c2c7018739aa9b7be89eb73409996c16050bef1.png 2x" data-dominant-color="2F2F33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1274×653 366 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Is this caused by I create too much tensor variables during the forward() method of UNet?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27c3184f9a7409f2f8fcbede408e8f704a1b08c9.png" alt="forward" data-base62-sha1="5FKBQj0Bt6cjVZ3iQ7i3IwpQD7b" width="299" height="320"></p>

username: ptrblck
cooked: <p>Have a look at <a href="https://discuss.pytorch.org/t/spliting-my-model-on-4-gpu-and-cuda-out-of-memory-problem/37311/3">this post</a>, which explains this issue due to Python’s function scoping.<br>
You might want to warp your training and evaluation in separate methods so that tensors can be freed when you return from these methods.</p>

username: crissallan
cooked: <p>Thank you so much! Wrap the train and test methods separately do help!</p>
Training crashes with no error when using CUDA

username: Jakkes
cooked: <p>I’m having a very weird issue when training my model in PyTorch. When using CPU, it works flawlessly. However, when switching over to CUDA, the program crashes with no error reported. I’m running it in <a href="https://colab.research.google.com/drive/133TTlvS-6fACkcEn4e1gS3e5nxnC5l61" rel="nofollow noopener">this Google Colab Notebook</a> and at a specific point in time the notebook just crashes “unexpectedly” with not error reported whatsoever.</p>
<p>I realize that anybody clicking that link will be deterred of helping me. However, I have pretty much no clue what the issue might be.</p>
<ul>
<li>Do you have any idea of common issues that causes a crash with no error?</li>
</ul>
<p>I have managed to narrow it down to one specific point in time. In the second to last cell, I have my main training loop. In order to better capture print messages (?) I put a sleep in there at the end of each loop. The program crashes after the sleep (never wakes up) when my lengths.max() == 2 - and I have no idea why.</p>
<ul>
<li>Does the sleep behave weirdly when using CUDA (time.sleep)?</li>
</ul>
<p>Thank you for you help</p>

username: ptrblck
cooked: <p>Could you post a small executable code snippet?<br>
I guess the real error message is hidden by the Jupyter Kernel crash/restart.<br>
Running the code in a terminal should give you an error message.</p>

username: Jakkes
cooked: <p>Hi,</p>
<p>Unfortunately I have no GPU, and so I am unable to do so.</p>
<p>However, I did manage to pin point the issue and solve it. Apparently, passing a batch of size 0 through a GRUCell doesn’t work on CUDA, but it does on CPU…? Either way, adding a flag in my code to check for the batch size of the input to the GRUCell fixed the issue.</p>

username: ptrblck
cooked: <p>Could you please post a small code snippet demonstrating this behavior?<br>
It should work or raise an issue on both devices.</p>

username: Jakkes
cooked: <p>Sure, here you go. The following crashes when device is set to “cuda”, but runs fine with device set to “cpu”.</p>
<pre><code class="lang-auto">import torch
from torch import nn

device = "cpu"

grucell = nn.GRUCell(32, 32).to(device)

hidden_state = torch.rand(0, 32).to(device)
input_data = torch.rand(0, 32).to(device)

output_state = grucell(input_data, hidden_state)

print(output_state)
</code></pre>

username: ptrblck
cooked: <p>The code works using the nightly binary from ~1 week ago, so this issue might have been already fixed.<br>
Could you install the nightly binary (in a new virtual environment) and rerun the code, please?</p>

username: Jakkes
cooked: <p>Sorry, but I am unable as I do not own a CUDA device myself. I can only run it through Google Colab.</p>

username: ptrblck
cooked: <p>You should still be able to install it in the notebook via <code>!pip install ...</code></p>

username: Jakkes
cooked: <p>Sorry for the late reply. Yes, using nightly it works. Using 1.4.0, it does not.</p>
Drawing a NN from a saved model

username: Kale-ab_Tessera
cooked: <p>I have a saved MLP model and I would to plot the weights of each layer, i.e. the connections from and to each neuron. I want each of these connections to be darker or lighter according to the weight magnitude.</p>
<p>I know there are tools that visualize the computation graph of a model, but are there tools that visualize the actual weight connections? Any examples using network x to draw a pytorch MLP model?</p>
<p>My end goal is something similiar to this, generated from a saved model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4af0ef488fcab65cab9ab85e2bfd52e7fefb604b.png" data-download-href="https://discuss.pytorch.org/uploads/default/4af0ef488fcab65cab9ab85e2bfd52e7fefb604b" title="model_connection_visualization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4af0ef488fcab65cab9ab85e2bfd52e7fefb604b_2_690x496.png" alt="model_connection_visualization" data-base62-sha1="aGXx2XOwgVnqmXILYgjJjoEYSHF" width="690" height="496" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4af0ef488fcab65cab9ab85e2bfd52e7fefb604b_2_690x496.png, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4af0ef488fcab65cab9ab85e2bfd52e7fefb604b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4af0ef488fcab65cab9ab85e2bfd52e7fefb604b.png 2x" data-dominant-color="DADADB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model_connection_visualization</span><span class="informations">907×652 286 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
FAILED: third_party/fbgemm/CMakeFiles/fbgemm_avx512.dir/src/FbgemmFloat16ConvertAvx512.cc.o

username: Eleanor456
cooked: <p>when I install the pytorch by source, I met the question as follows.</p>
<p>[1162/4410] Building CXX object third_p…dir/src/FbgemmFloat16ConvertAvx512.cc.<br>
FAILED: third_party/fbgemm/CMakeFiles/fbgemm_avx512.dir/src/FbgemmFloat16ConvertAvx512.cc.o<br>
/opt/rh/devtoolset-4/root/usr/bin/c++  -DFBGEMM_STATIC -DTH_BLAS_MKL -I…/third_party/cpuinfo/include -I…/third_party/fbgemm/third_party/asmjit/src -I…/third_party/fbgemm/include -I…/third_party/fbgemm -I…/cmake/…/third_party/benchmark/include -isystem …/cmake/…/third_party/googletest/googlemock/include -isystem …/cmake/…/third_party/googletest/googletest/include -isystem …/third_party/protobuf/src -isystem /home/liuxiyang/anaconda3/envs/torch1_1/include -isystem …/third_party/gemmlowp -isystem …/third_party/neon2sse -isystem …/third_party/XNNPACK/include -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG -fPIC -fvisibility=hidden   -m64 -mavx2 -mfma -mavx512f -mavx512bw -mavx512dq -mavx512vl -masm=intel -std=c++14 -MD -MT third_party/fbgemm/CMakeFiles/fbgemm_avx512.dir/src/FbgemmFloat16ConvertAvx512.cc.o -MF third_party/fbgemm/CMakeFiles/fbgemm_avx512.dir/src/FbgemmFloat16ConvertAvx512.cc.o.d -o third_party/fbgemm/CMakeFiles/fbgemm_avx512.dir/src/FbgemmFloat16ConvertAvx512.cc.o -c …/third_party/fbgemm/src/FbgemmFloat16ConvertAvx512.cc<br>
/tmp/cckA7tcH.s: Assembler messages:<br>
/tmp/cckA7tcH.s:53: Error: operand size mismatch for <code>vbroadcastss' /tmp/cckA7tcH.s:55: Error: operand size mismatch for </code>vbroadcastss’</p>
<p>how can I fix it? Thank you.</p>
<p>the g++ version is 5.3.1<br>
the cmake version is 3.14.0<br>
the python version is the anaconda with py3.6</p>

username: XiaobingZhang
cooked: <p>I also meet the same problem for same env with you(gcc 5.3, py3.6), there two options can solve this problem:</p>
<ol>
<li>
<p><strong>USE_FBGEMM=0 python setup.py install</strong> in case of you don’t want to use low-precision solution.</p>
</li>
<li>
<p>Using higher gcc version, gcc 6.3.1 can works in my side.</p>
</li>
</ol>
Create and destroy dataloader during training

username: nobodykid
cooked: <p>Hello, I have similar question about dataloader to <a href="https://discuss.pytorch.org/t/is-it-possible-to-free-up-a-dataloader/21498">this question</a>. But in a different manner</p>
<p>I’m currently writing a training script of a model consisted of 3 submodels, each trained individually. Roughly, the training iteration will be like this.</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
	# train model A
	model_a_best = model_a_step()

	# train model B
	model_b_best = model_b_step()

	# train model C
	model_c_best = model_c_step()

	# validate big model (3 submodels bundled up becoming one big model)
</code></pre>
<p>Unfortunately, each model has their own dataset and dataloader configurations. So I need to instantiate them both before run the training for each model.</p>
<p>My question is, if I were to instantiate both dataset and dataloader inside each model’s <code>_step</code> function, how do I destroy the dataset and dataloader once I finished training the model and goes to the next one?</p>
<p>My initial plan was to instantiate all the datasets and dataloaders right before this training iteration, but I’m afraid that will consume a lot of resources during the iteration</p>

username: Paritosh
cooked: <p>Hi,</p>
<p>Datasets and Dataloaders are generally instantiated outside the training/testing loops. Are you sure you instantiate those within the training loop? To delete: how about del dataset/dataloader?</p>
<p>Cheers!</p>

username: nobodykid
cooked: <p>The initial plan is yes, to instantiate (and destroy) each dataloaders of related model whenever I’m running a training iteration on said model, to minimize my resource usage during training.</p>
<p>As each model as train and validation dataset, there will be 6 dataloaders in total (2 for each model) during training, and that doesn’t include dataloader for the big model.</p>
<p>My rough implementation will be somewhat like this (with example of <code>model_a</code>)</p>
<pre><code class="lang-auto">def model_a_step():
    # instantiate dataloader and dataset
    a_dataset = Dataset(model_a_data_set)
    dataloader = DataLoader(a_dataset)

    for x in dataloader:
        # run inference and backprop
    
    # destroy dataloader and dataset when finished
    del dataloader, a_dataset
    return trained_model
</code></pre>

username: Paritosh
cooked: <p>Let us know how it goes. BTW, there will be only 2 dataloaders active, not 6, because you will be deleting the used ones.</p>

username: kshitij
cooked: <pre><code class="lang-python">def model_a_step():
    # instantiate dataloader and dataset
    a_dataset = Dataset(model_a_data_set)
    dataloader = DataLoader(a_dataset)

    for x in dataloader:
        # run inference and backprop
    
    # no need as they will be destroyed when function ends.
    # destroy dataloader and dataset when finished
    # del dataloader, a_dataset
    return trained_model
</code></pre>
<p>If you load it in a function call there will be no need to explicitly delete as they will be destroyed when the function returns as per the language rules.</p>
Getting low validation accuracy while finetuning inception_v3

username: pnelham
cooked: <p>Hi,<br>
I am using PyTorch fine-tuning tutorial for inception_v3, [<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a> ].<br>
I have a training set with 34 classes and a validation set with 28 classes! The training loss getting better over epochs but the validation accuracy getting low. Does anyone know where is the bug?<br>
Thanks.</p>

username: mailcorahul
cooked: <p>Do you see train loss decreasing(and train accuracy increasing) while val loss increases(and val accuracy decreasing)?<br>
In that case, your model is overfitting.</p>
<ol>
<li>You can add more data</li>
<li>Try out more augmentations</li>
<li>Dropouts</li>
<li>Use a shallow net.</li>
</ol>

username: pnelham
cooked: <p><a class="mention" href="/u/mailcorahul">@mailcorahul</a>  I see train loss decreasing and train accuracy increasing(start from ~40%) but the accuracy doesn’t change after about 100 epochs and 70% accuracy. And the validation loss increases from the second epoch.</p>

username: mailcorahul
cooked: <p>Just to be on the same page, train loss keeps decreasing after 2nd epoch while val loss starts to increase. Then this is a case of overfitting and you can follow the above steps that I have mentioned.</p>
Is there any loss that can count number of distinct values in tensor?

username: universe_will
cooked: <p>I want to calculate number of distinct values in a tensor as loss for image generation.<br>
The target gray image should contain only several gray values in range of (0, 255). Large number of distinct values should get high loss.</p>
<p>How can i achieve this loss with torch?</p>

username: satanarmy
cooked: <p>torch.unique() method identifies uniques elements in a tensor. You can use (tensor).numel() method to find number of elements in a tensor.<br>
Hence you can combine both and calculate total loss as:</p>
<pre><code class="lang-auto">loss = torch.unique(img_tensor).numel()
</code></pre>

username: dpernes
cooked: <p>If by “loss” you mean something that can be used to train a model (i.e., something you can get gradients from), then the answer is <strong>NO</strong>, since such loss would not be differentiable.</p>

username: universe_will
cooked: <aside class="quote no-group" data-username="universe_will" data-post="1" data-topic="76536">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/universe_will/48/22710_2.png" class="avatar"> universe_will:</div>
<blockquote>
<p>alculate number of distinct values in a tensor as loss for image generation.<br>
The target gray image should contain only several gray values in range of (0, 255). Large number of distinct values should get high loss.</p>
<p>How can i achieve this loss with torch?</p>
</blockquote>
</aside>
<p>Thanks  for your reply.<br>
Besides, is there any way to approximate this goal?</p>

username: dpernes
cooked: <p>You can use the <em>L-1</em> loss, since it’s the closest convex approximation of <em>L-0</em>: <code>loss = torch.norm(output-target, p=1)</code>. If you feel confident and ok with discarding convexity, you can even try a fractional norm, e.g. <em>L-1/2</em>: <code>loss = torch.norm(output-target, p=0.5)</code>. This latter choice may probably lead to unstable gradients, though.</p>
Does .contiguous() creates a copy of the memory?

username: el_samou_samou
cooked: <p>Hi,</p>
<p>Does the <code>.contiguous()</code> function creates a copy of the memory corresponding to the tensor’s data?</p>
<p>This was mentioned by <a class="mention" href="/u/ptrblck">@ptrblck</a> here: <a href="https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107/2" class="inline-onebox">Contigious vs non-contigious tensor</a> but I would like to know if anyone is sure about this.</p>
<p>Thank you in advance for your help.</p>
<p>Samuel</p>

username: ptrblck
cooked: <p>If you call <code>contiguous()</code> on a non-contiguous tensor, a copy will be performed.<br>
Otherwise it will be a no-op.</p>
<p>You could add some print statements to the linked example code and will see an increased memory usage after the <code>y.contiguous()</code> call.</p>

username: 111284
cooked: <p>I checked the id of x and y in your example, although they have different ids, but when I compare them the result show they’re the same.</p>
<pre><code class="lang-auto">x = torch.arange(12).view(4, 3)
y=x.t()
y = y.contiguous()
print(id(x.storage()))
print(id(y.storage()))
print(id(x.storage())==id(y.storage()))
&gt; 4495471152
&gt; 4495376464
&gt; True
</code></pre>
<p>Then I further check each element in x and y manually, and I found the id of same element in x and y is identical. For example:</p>
<pre><code class="lang-auto">id(x.storage()[0])
&gt; 4336047184
id(y.storage()[0])
&gt; 4336047184
id(x.storage()[3]) # x.storage()[3] == 3 
&gt; 4336047280
id(y.storage()[1])  # y.storage()[1] == 3 
&gt; 4336047280
</code></pre>
<p>It to me seems like the contiguous doesn’t create a copy of the memory, but create a new “reference” to the same memory. Since I don’t know much about memory management, I would like to know if my guess is to some extent right.</p>
<p>Thank you in advance.</p>

username: ptrblck
cooked: <p>I think <code>.storage()</code> will return a new reference and cannot be used to compare the location of the data:</p>
<pre><code class="lang-python">x = torch.randn(1)
print(id(x.storage()))
&gt; 3001145106760
print(id(x.storage()))
&gt; 3001133757320
print(id(x.storage()))
&gt; 3001171709448
print(id(x.storage()))
&gt; 3001576998536
</code></pre>
<p>You could try to compare the <code>.data_ptr()</code> instead.</p>

username: 111284
cooked: <p>Thank you for your answer!</p>
Building wheels with pytorch extensions

username: dsuess
cooked: <p>I am trying to build a manylinux wheel containing pytorch extensions using CUDA. Everything works fine except for patchelf’ing libcudart in the so files. After running <code>auditwheel --repair</code> on the wheel, the compiled extensions have been patched as I can check using <code>ldd</code>:</p>
<pre><code class="lang-auto">ldd nms_cuda.cpython-37m-x86_64-linux-gnu.so
ldd: warning: you do not have execution permission for `./nms_cuda.cpython-37m-x86_64-linux-gnu.so'
./nms_cuda.cpython-37m-x86_64-linux-gnu.so: /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/libcudart-1b201d85.so.10.1.243: version `libcudart.so.10.1' not found (required by ./nms_cuda.cpython-37m-x86_64-linux-gnu.so)
	linux-vdso.so.1 =&gt;  (0x00007fff47518000)
	libcudart-1b201d85.so.10.1.243 =&gt; /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/libcudart-1b201d85.so.10.1.243 (0x00007f4e3fe41000)
	libstdc++.so.6 =&gt; /usr/lib64/libstdc++.so.6 (0x00007f4e3fb3a000)
	libm.so.6 =&gt; /usr/lib64/libm.so.6 (0x00007f4e3f838000)
	libgcc_s.so.1 =&gt; /usr/lib64/libgcc_s.so.1 (0x00007f4e3f622000)
	libpthread.so.0 =&gt; /usr/lib64/libpthread.so.0 (0x00007f4e3f406000)
	libc.so.6 =&gt; /usr/lib64/libc.so.6 (0x00007f4e3f038000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4e4036d000)
	libdl.so.2 =&gt; /usr/lib64/libdl.so.2 (0x00007f4e3ee34000)
	librt.so.1 =&gt; /usr/lib64/librt.so.1 (0x00007f4e3ec2c000)
</code></pre>
<p>So the crucial line</p>
<pre><code class="lang-auto">./nms_cuda.cpython-37m-x86_64-linux-gnu.so: /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/libcudart-1b201d85.so.10.1.243: version `libcudart.so.10.1' not found (required by ./nms_cuda.cpython-37m-x86_64-linux-gnu.so)
</code></pre>
<p>Tells me that the shared object can load the correct version. I get the same error when importing the module in python.</p>
Offline and online data augmentation together

username: Pracheta_Sahoo
cooked: <p>Can I apply offline and online data augmentation at the same time? I have a dataset, that has nearly 3500 images highly class imbalanced. To make balanced classes, I applied python’s augmentor package and made it with 12000 images. I applied offline image augmentation here, and chose 1000 samples from each data folder. Now in my learning algorithm I already have online augmentation implemented in the DataLoader class. The code snippet is below:</p>
<pre><code class="lang-auto">type or paste code here
self.weak = tfs.Compose([
            tfs.Resize(resize_shape),
            tfs.RandomHorizontalFlip(),
            tfs.RandomCrop(size=crop_shape, padding=int(crop_shape * 0.125), padding_mode='reflect')
            ])
        self.strong = tfs.Compose([
            tfs.Resize(resize_shape),
            tfs.RandomHorizontalFlip(),
            tfs.RandomCrop(size=crop_shape, padding=int(crop_shape * 0.125), padding_mode='reflect'),
            RandAugmentMC(n=2, m=10)
            ])
        self.normalize = tfs.Compose([
            tfs.ToTensor(),
            tfs.Normalize(mean=mean, std=std)])
</code></pre>
<p>Now my question is this process is improving the accuracy, but is this correct?</p>

username: ptrblck
cooked: <p>The approach might work.<br>
Could you explain the difference between the offline and online data augmentation?<br>
If you’ve applied the same transformations, I’m not sure, if you should expect any benefits from this work flow, and could probably just use the online augmentation?</p>

username: Pracheta_Sahoo
cooked: <p>Thanks for your reply. In offline augmentation, I applied rotation and zoom to the images. And in online augmentation I applied randomcrop, flip and resize etc. as shown in the above code snippet. Would it work?</p>

username: ptrblck
cooked: <p>Yes, I don’t see a problem in the approach.</p>

username: Pracheta_Sahoo
cooked: <p>Thanks. But is there any suitable scientific source to support my approach?</p>

username: ptrblck
cooked: <p>I don’t think so, since you could achieve the same result with the (lazy) online transformation and you wouldn’t need to store the offline transformed images.</p>

username: Pracheta_Sahoo
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I am doing offline and online image augmentation together, that way the accuracy is higher.</p>

username: ptrblck
cooked: <p>I think the accuracy might be higher due to the larger epochs and thus more training iterations.<br>
How many iterations and epochs are you training both model?<br>
Also, are you comparing your offline + online augmentation approach to an online approach with the same transformations?</p>
What are hooks used for?

username: JWageM
cooked: <p>Tensors have a function: register_hook.<br>
<code>register_hook</code> ( <em>hook</em> )<a href="https://pytorch.org/docs/stable/_modules/torch/tensor.html#Tensor.register_hook" rel="nofollow noopener">[SOURCE]</a></p>
<p>Registers a backward hook.</p>
<p>The description says that everytime a gradient with respect to the tensor is computed the hook will be called.</p>
<p>My question: what are hooks used for?</p>
<p>Kind regards,<br>
Jens</p>

username: ptrblck
cooked: <p>You could pass a function as the <code>hook</code> to <code>register_hook</code>, which will be called every time the gradient is calculated.<br>
This might be useful for debugging purposes, e.g. just printing the gradient or its statistics, or you could of course manipulate the gradient in a custom way, e.g. normalizing it somehow etc.</p>

username: JWageM
cooked: <p>Is it only used for reporting by the user, or is it also used internally by the back-propagation algorithm?</p>

username: ptrblck
cooked: <p>If you manipulate the gradients, the optimizer will use these new custom gradients to update the parameters, so the latter would be true.</p>

username: JWageM
cooked: <p>I see thanks.</p>
<p>Are they also used if I call backward() on a tensor?</p>

username: ptrblck
cooked: <p>Yes, here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(1, 1)
w = torch.randn(1, 1, requires_grad=True)
w.register_hook(lambda x: print(x))
y = torch.randn(1, 1)

out = x * w
loss = (out - y)**2
loss.register_hook(lambda x: print(x))
loss.mean().backward(gradient=torch.tensor([0.1]))  # prints the gradient in w and loss
</code></pre>

username: Umair_Javaid
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="40020">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could pass a function as the <code>hook</code> to <code>register_hook</code></p>
</blockquote>
</aside>
<p>Can we pass any parameters of our own to this “hook” function? i mean multiple parameters</p>
user_suspended: yes

username: ptrblck
cooked: <p>Yes, you could pass more parameters to the <code>lambda</code> call:</p>
<pre><code class="lang-python"># same script as above
my_param = "lala"
loss.register_hook(lambda x, my_param=my_param: print(my_param, x))
loss.mean().backward(gradient=torch.tensor(0.1)) 
</code></pre>

username: pratyush911
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I was wondering if it is possible to set requires_grad = True for the registered hooks.<br>
More specifically, I am registering hooks for a recurrent network and want to know the gradients of the gradients (second derivative) i.e. Second derivate of the loss w.r.t. each hidden state.<br>
I could not find a method to do this directly.</p>
Making predictions at every time step

username: vainaijr
cooked: <p>If I want to make a neural network that make prediction at every time step, instead of making one prediction, how should I approach this problem, for example, if I am doing image classification, then instead of asking my neural network to predict the class for an entire image, I show my neural network only a part of image, and get a prediction, then backpropagate, update weights, in the next time step, I show neural network a bigger part of image, get prediction, then backpropagate, update weights, and so on, and at the last time step, I show my neural network entire image.</p>
<p>Because we are sometimes able to predict class of image correctly by seeing part of image only, without seeing the entire picture.</p>
<p>Is there any term for this type of neural network training, or any example that work like this?</p>
Add tensors within a tensor

username: jbarry
cooked: <p>I have quite a basic question but I could not find a similar question answered elsewhere.</p>
<p>I would like to add two tensors within a larger tensor. Specifically, I have a tensor of shape<code> (batch, seq_len, max_len, embedding_dim)</code> and I want to add all of the tensors in <code>max_len</code> so that I am left with <code> (batch, seq_len, embedding_dim)</code>, e.g. my new tensor’s <code>embedding_dim</code> is all of the <code>max_len</code> embedding dims added together.</p>
<p>I couldn’t find a straightforward way to do this without using a python for loop to create a new tensor which is the sum of the n tensors of <code>max_len</code>. Also, <code>a.sum(-1)</code> is not what I need as it just adds up all of the numbers along the row-dim. Instead (based on the below example), I would like each tensor to be :</p>
<pre><code class="lang-auto">[ 2.3677,  1.0233,  0.6200, -1.1026, -0.1339] + [-0.2428,  0.0120, -0.6363,  0.8928, -0.6387]
[-0.4187,  1.4169, -0.1631,  0.5966, -0.2203] + [-0.6647,  0.9166, -1.8717, -2.5794, -1.1827]
</code></pre>
<p>and so on.</p>
<p>Any help would be appreciated, thanks!</p>
<pre><code class="lang-auto">batch=2
seq_len=3
max_len=2 # max_len is the max number of components in the batch for this feature
embedding_dim=5

a = torch.randn(batch, seq_len, max_len, embedding_dim)
print(a.size())
print(a)


torch.Size([2, 3, 2, 5])
tensor([[[[ 2.3677,  1.0233,  0.6200, -1.1026, -0.1339],
          [-0.2428,  0.0120, -0.6363,  0.8928, -0.6387]],

         [[-0.4187,  1.4169, -0.1631,  0.5966, -0.2203],
          [-0.6647,  0.9166, -1.8717, -2.5794, -1.1827]],

         [[ 1.7884, -0.6862, -0.8162, -0.5604,  0.0198],
          [ 0.6459,  0.3187,  1.9847,  2.4008,  0.3162]]],

        [[[ 0.7555,  2.9180,  0.7002,  2.8533, -1.9177],
          [-0.0804,  0.4459,  0.4908,  1.3397, -1.8577]],

         [[ 0.2427, -2.0033, -0.6455,  0.0856, -0.4497],
          [-0.4160,  1.3193, -0.0158,  0.1067,  0.8288]],

         [[-3.3968,  0.5313,  0.4720,  0.8234, -0.3139],
          [ 0.6371, -1.4486, -1.5495, -1.0803,  0.2780]]]])
</code></pre>

username: ptrblck
cooked: <p>I’m not sure, if I misunderstand the question, but would <code>a.sum(2)</code> work?<br>
This would apply the <code>sum</code> operation in dim2.</p>

username: jbarry
cooked: <p>Indeed you are correct! I tried this a couple of hours ago and must have miscalculated what I expected the result to be and then I got stuck trying much more complicated approaches! Thanks so much for clearing that up!</p>
grad_fn=<DivBackward0>,what's problems with the console output?

username: wsy
cooked: <p>d=2<br>
n=50<br>
X=torch.randn(n,d)<br>
true_w=torch.tensor([[-1.0],[2.0]])<br>
y=X@true_w+torch.randn(n,1)*0.1<br>
print(‘X shape’,X.shape)<br>
print(‘y shape’,y.shape)<br>
print(‘w shape’,true_w.shape)</p>
<p>def model(X,w):<br>
return X@w</p>
<p>def rss(y,y_hat):<br>
return torch.norm(y-y_hat)**2/n<br>
def grad_rss(X,y,y_hat):<br>
return -2*X.t()@(y-y_hat)/n</p>
<p>w=torch.tensor([[1.0],[0.0]],requires_grad=True)<br>
y_hat=model(X,w)<br>
loss=rss(y,y_hat)<br>
loss.backward()<br>
print(‘Analytical gradient’, grad_rss(X, y, y_hat).detach().view(2).numpy())<br>
print(‘PyTorch’s gradient’, w.grad.view(2).numpy())<br>
print(grad_rss(X,y,y_hat))</p>
<p>console:<br>
grad_rss(X,y,y_hat)<br>
tensor([[ 5.1305],<br>
[-3.3439]], grad_fn=)</p>

username: ptrblck
cooked: <p>Could you explain the issue with the output, please?<br>
What do you expect and what are you seeing at the moment?</p>
<p>PS: You can post code snippets by wrapping them into three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: wsy
cooked: <p>Extra characters in output.<br>
I define a  function and input tensors.</p>

username: ptrblck
cooked: <p>Which extra characters are problematic?</p>

username: wsy
cooked: <p>So why does this output appear?(grad_fn=)</p>

username: ptrblck
cooked: <p>Using your code I get the following output:</p>
<pre><code class="lang-python">X shape torch.Size([50, 2])
y shape torch.Size([50, 1])
w shape torch.Size([2, 1])
Analytical gradient [ 4.254795  -3.4482186]
PyTorch’s gradient [ 4.2547956 -3.4482183]
tensor([[ 4.2548],
        [-3.4482]], grad_fn=&lt;DivBackward0&gt;)
</code></pre>

username: wsy
cooked: <p>Oh，Thank you for your previous reply.I just want to know what the words (&lt;grad_fn=&gt;)on the right mean except numbers.</p>

username: ptrblck
cooked: <p><code>grad_fn</code> gives you the name of the backward function for this particular tensor.<br>
Since the last operation was a division, the result will have the gradient function <code>grad_fn=&lt;DivBackward&gt;</code> attached to it. Autograd uses these functions to step back through the computation graph and calculate the gradients for all parameters.</p>

username: wsy
cooked: <p>I have understood the problem. Thank you very much</p>
CPUOperatorTest failed. How can I fix this?

username: Will_Liu
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/9945427446d601c56f75670d3a24a1b69fc9ea1e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9945427446d601c56f75670d3a24a1b69fc9ea1e" title="捕"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/9945427446d601c56f75670d3a24a1b69fc9ea1e.png" alt="捕" data-base62-sha1="lRTrl2RtwKgb087An1nDwcQvmPA" width="690" height="409" data-dominant-color="38162D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕</span><span class="informations">703×417 49.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jfix
cooked: <p><a class="mention" href="/u/will_liu">@Will_Liu</a> Thanks for reporting this. Can you run the <code>CPUOperatorTest</code> by itself to determine which test specifically is failing?</p>
<p>Also, please provide the environment you’re running in – this test suite covered under CI so it must be a portability issue.</p>
How can I save some tensor in python, but load it in libtorch?

username: Edwardmark
cooked: <p>How can I save some tensor in python, but load it in libtorch:</p>
<p>I save tensor named piror using python, using the code:</p>
<pre><code class="lang-auto">torch.save(prior,  'prior.pth')
</code></pre>
<p>And I load the tensor in libtorch using C++, by the following code:</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; tensorVec;
torch::load(tensorVec, "/app/model/prior.pth");
torch::Tensor priors = tensorVec[0];
</code></pre>
<p>But I got the error:<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  <code>torch::jit::load()</code> received a file from <code>torch.save()</code>, but <code>torch::jit::load()</code> can only load files produced by <code>torch.jit.save()</code> (load at …/torch/csrc/jit/serialization/import.cpp:285)</p>
<p>Why is that? And what should I do to solve the issue? Thanks in advance.</p>
Copy a model by deep copy does not work

username: Mainul
cooked: <p>I am trying following line:<br>
<code>best_model=copy.deepcopy(model)</code></p>
<p>and getting following error:</p>
<p><code>TypeError: can't pickle torch._C.ScriptFunction objects</code></p>
<p>Is this happening for some version changes or something because this code worked before?</p>
How does pytorch use unfold in convolution?

username: dongtuu
cooked: <p>Hi, I’m recently implementing a custom convolution and find maybe I need the nn.unfold function. While trying, I find that nn.unfold seems to output a tensor that is much larger than its input. For instance, if the kernel size is 3, the output size would be nearly 9 times the input’s size. This costs me much memory. However, nn.conv2d costs much less memory with same inpute size, while I think I find something like unfold in the C implementation. So I wonder how pytorch uses unfold, is it possible to imitate it using the functions provided in pytorch?</p>

username: vfinotti
cooked: <aside class="quote no-group" data-username="dongtuu" data-post="1" data-topic="36980">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/d07c76/48.png" class="avatar"> dongtuu:</div>
<blockquote>
<p>I wonder how pytorch uses unfold, is it possible to imitate it using the functions provided in pytorch?</p>
</blockquote>
</aside>
<p>A similar topic with an useful discussion is available <a href="https://discuss.pytorch.org/t/make-custom-conv2d-layer-efficient-wrt-speed-and-memory/70175">here</a></p>