akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cannot use template above a class using ``TORCH_MODULE`` below its definition

username: alireza_dizaji
cooked: <p>I need to define a generic class as it might take different activation and convolution layers, but when I’m trying to define the template above the class definition in the header file, I got an error from <code>TORCH_MODULE</code> indicating “argument list for the class template is missing”.</p>
<pre><code class="lang-auto">template&lt;typename ActivationType, typename ConvType, typename... ConvArgs&gt;
class ResDecoderBlockImpl: public nn::Module {
    private:
        BasicConv2d skip_connection;
        nn::Upsample up_sample;
        nn::Sequential seq{nullptr};
    public:
        ResDecoderBlockImpl(int, int, int, int, ConvType, ActivationType, ConvArgs...);
        at::Tensor forward(const at::Tensor&amp;, const at::Tensor&amp;);
};
TORCH_MODULE(ResDecoderBlock);
</code></pre>
How to apply nn.BatchNorm1d to every layers of a nn.GRU module?

username: evilroach
cooked: <p>I’m learning GRU and DL, and I found that the benifit of BN(batch normalization) has being advocated everywhere in Internet.<br>
I want to try it by myself, but I don’t know how to apply the nn.BatchNorm1d to my GRU module, because that the GRU module(nn.GRU) doesn’t give me a chance to normalize the inputs of each layer.<br>
It looks like that we can only normalize(or modify) the input of the first layer, even though a GRU can have multiple hidden layers.</p>
<p>If I understand correctly, BN should be applyed to every layers of a NN.</p>
<p>Thanks!</p>
Each gpu has independent model

username: Artem_Storm
cooked: <p>I usually use small backbones for my work, so if my machine has 4 gpus, I can tune hyperparameters by training 4 independent model. Thus I prepare batch only once on cpu and send it to all subprocesses through the queue like that</p>
<pre><code class="lang-auto">ctx = mp.get_context('fork')
p = ctx.Process(target=secondary_training, args=(cfg, queue))
p.start()
</code></pre>
<p>It worked well before torch 1.12.<br>
Now it raises an error saying that I should use another context instead of ‘fork’. But I do not understand how to configure it for this case, where models are independent with different hyperparameters and even backbones but the training data are same.<br>
Could you please tell me what parallel method suits here, if ‘fork’ are not supported any more?</p>

username: Artem_Storm
cooked: <p>I’ve found solution using torch.distributed.send and torch.distributed.recv<br>
more info here:<br>
<a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/dist_tuto.html</a></p>
Issue: topk>16 raises Error on MPS

username: ilia.malishev
cooked: <p>a2mps = torch.topk(input=a.to(“mps”), k=20) # not working at all<br>
RuntimeError: Currently topk on mps works only for k&lt;=16</p>
<p>a1mps = torch.topk(input=a.to(“mps”), k=15) # working fine</p>
<p>The corresponding issue is <a href="https://github.com/pytorch/pytorch/issues/78915" class="inline-onebox" rel="noopener nofollow ugc">MPS: Add support for TopK (k&gt;16) on M1 GPU · Issue #78915 · pytorch/pytorch · GitHub</a><br>
When is the fix expected?</p>
<p>print (torch.<strong>version</strong>): 1.14.0.dev20221022</p>
Simulate GPU for testing?

username: mattrobin
cooked: <p>I’m often in a situation where on my local machine (without a GPU) my code runs fine, but I’d like to test that all the tensors are added to the correct device if there’s a GPU present. I want to test this before I commit my code, but do so would require deploying it to a remote machine, which are usually running something I don’t want to interrupt, so I need to wait until they’re finished before testing and committing my code. Of course, there are ways I could change my workflow in committing my code, but ideally I would just be able to test the code locally by simulating that a GPU is present. Is there anyway to do this?</p>

username: ptrblck
cooked: <p>I’m not aware of a way to simulate a GPU, but you could run an AWS instance to try it out or use a Colab notebook, although I’m not sure about the licensing.</p>

username: mattrobin
cooked: <p>Thanks! Not quite the answer I wanted to hear, but probably the correct answer nonetheless. Also, didn’t know about Colab, and that looks pretty neat, so thank you!</p>

username: ptrblck
cooked: <p>They provide a free GPU for afaik 24 hours. After that you would need to restart your notebook.<br>
Because of that, I would look closely to the license and intellectual property, if you are using code written for your employer.</p>

username: justusschock
cooked: <p>Alternatively you could try to use one of the programs listed in <a href="https://stackoverflow.com/questions/3087361/gpu-emulator-for-cuda-programming-without-the-hardware" rel="nofollow noopener">this thread</a>.<br>
However, I am not sure whether any of these options are runnable in python. If not you might be able to use them together with torch.jit as it is mentioned in <a href="https://pytorch.org/2018/05/02/road-to-1.0.html" rel="nofollow noopener">Road to 1.0</a></p>

username: Ilya_Kamenshchikov
cooked: <p>Maybe the <code>meta</code> device is what you need.</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = torch.ones(2) # defaults to cpu
&gt;&gt;&gt; b = torch.ones(2, device='meta')
&gt;&gt;&gt; c = a + b
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, meta and cpu!
</code></pre>
Convert map-style Dataset to Iterable?

username: drscotthawley
cooked: <p>I want to <code>Chain</code> two datasets together, one of which is a normal Dataset and the other is an IterableDataset.   But Chain only accepts IterableDatasets.</p>
<p>So, I thought I’d just create a simple Iterable version of my original map-style dataset… but it’s not working.  Sample code below.</p>
<p>I see a few questions related to this, such as <a href="https://discuss.pytorch.org/t/dataset-map-style-vs-iterable-style/92329">this</a> and <a href="https://stackoverflow.com/questions/71160604/converting-pytorch-map-style-dataset-to-iterable-dataset-with-multi-process-data" rel="noopener nofollow ugc">this SO question with no answer</a>, but no answers.</p>
<p>How do we do this?  Here’s my attempt:</p>
<pre><code class="lang-python">class MapStyleDataset(torch.utils.data.DataSet):
    def __init__(self, path, **kwargs):
        super().__init__()
        self.filenames = glob.glob(path)

    def __getitem__(self, idx):
        data = somehow_read_from( self.filenames[idx] )
        return data

    def __len__(self):
         return len(self.filenames)


class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, *args, **kwargs):
        super().__init__()
        self.this = MapStyleDataset(args, kwargs)
        self.len = len(self.this)

    def __iter__(self):
        return self.this.__getitem__(random.randint(0, self.len))
</code></pre>
<p>Let’s assume MapStyleDataset works fine. You can do <code>next(iter(....))</code> on it properly.</p>
<p>But when I try to do <code>next(iter(....))</code> on the Iterable dataset, over and over I get error messages that read:</p>
<p><code>TypeError: iter() returned non-iterator of type 'Tensor'</code></p>
<p>How do do this conversion properly?  Thanks.</p>

username: drscotthawley
cooked: <p>ANSWER: So close!  Just need to use <code>yield</code> instead of <code>return</code> to return data from <code>__iter__</code>.  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Diagonal of jacobian

username: Albert_Chen
cooked: <p>I’d like to calculate only the diagonal of the jacobian matrix.<br>
[d z_1/d x_1, d z_2/d x_2, …, d z_n/d x_n]<br>
This operator is widely used in the diffusion model.<br>
The alternatives are</p>
<ol>
<li>calculate the whole Jacobian matrix, then take out the diagonal, which takes a lot of memory.</li>
<li>I can loop over the entries, but slow.<br>
So if I want to develop a specialized operator like this, please instruct how I can do so.</li>
</ol>
How to efficiently check marked cells given their 8 corner indices

username: Justin_Chen
cooked: <p>Hi there,</p>
<p>I want to know a memory efficient way of doing following behavior:<br>
Suppose I know the indice of the 8 corners of the cells that’s being marked.<br>
I was fed with a bunch of new cells with the indices of the 8 corners and I want to know if they are the marked cells or not.<br>
This is easy to do if I just allocate the n^3 memory and keep a mask, but is there a more efficient way with just tensor operations(like no forloops)?</p>
<p>Thanks so much!</p>
Convert python.list of 1d torch.Tensor to 2d torch.Tensor

username: Kallinteris-Andreas
cooked: <pre><code class="lang-auto">(Pdb) p temp
[tensor([ 7.9873e-01,  9.9704e-01, -6.7735e-02, -3.6495e-02,  6.8676e-04,
         5.2778e-02,  6.9929e-03,  1.8170e-02, -4.0842e-02,  8.8229e-02,
        -7.0201e-02,  7.8539e-02, -1.6455e-02,  6.0354e-02, -2.1752e-01,
        -7.2633e-02,  1.0848e-01,  6.1456e-02, -6.3961e-02, -1.0666e-01,
        -2.4548e-02, -4.4079e-02, -6.6344e-03,  1.0026e-01,  1.4799e-01,
        -3.2869e-03, -7.5987e-02])]
</code></pre>
<p>For example, here <code>temp</code> is a python.list of a single 1d tensor of length 27, and I would like to convert it to a 2d tensor of dimensions (27,1)</p>
<p>if I had a python.list of 2 1d tensor of length 27, and I would like to convert it to a 2d tensor of dimensions (27,2)</p>
<p>if I had a python.list of N 1d tensor of length 27, and I would like to convert it to a 2d tensor of dimensions (27,N)</p>
<p>Thanks in advance!</p>

username: KFrank
cooked: <p>Hi Kallinteris!</p>
<aside class="quote no-group" data-username="Kallinteris-Andreas" data-post="1" data-topic="164159" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kallinteris-andreas/48/36794_2.png" class="avatar"> Kallinteris-Andreas:</div>
<blockquote>
<p>if I had a python.list of N 1d tensor of length 27, and I would like to convert it to a 2d tensor of dimensions (27,N)</p>
</blockquote>
</aside>
<p>Does <a href="https://pytorch.org/docs/1.12/generated/torch.stack.html" rel="noopener nofollow ugc">torch.stack()</a> do what you want?  Specifically, something like<br>
<code>torch.stack (list_of_tensors, dim = -1)</code>?</p>
<p>Best.</p>
<p>K. Frank</p>
Send parameters to CUDA before defining optimizer

username: Yoo
cooked: <p>Is it possible to send tunable parameter to CUDA before defining the optimizer for it?</p>
<blockquote>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
param = torch.FloatTensor([1]).clone().detach().requires_grad_(True)<br>
param = param.to(device)<br>
opt = torch.optim.SGD([param], lr = 0.01)</p>
</blockquote>
<p>I got<br>
ValueError: can’t optimize a non-leaf Tensor</p>

username: ptrblck
cooked: <p>The <code>to()</code> operation is differentiable and thus creates a non-leaf tensor. Move the data to the device first and create a leaf tensor afterwards via <code>.requires_grad_()</code>.</p>

username: Yoo
cooked: <p>Suppose I define a tensor, in order to tune it in CUDA, I have 3 steps:</p>
<ol>
<li>move it to CUDA</li>
<li>add <code>.requires_grad_()</code>  to it</li>
</ol>
<p>When is the proper time to define my optimizer for it? Between 1 and 2? Does the order matter?</p>
Efficient prediction for large dataset

username: builder173
cooked: <p>I’ve trained model on ~28,000 samples and wondering what the recommended / efficient way to run predictions on this many samples might be? If the input samples are in a tensor called x_data I’ve found that simply running:</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
   pred = model(x_data)
</code></pre>
<p>results in very high memory growth and doesn’t complete. I’m able to run the above with batches of 1,000 samples inside a loop using tensor slicing, but it feels rather inelegant.</p>
<p>I’m curious to understand both why the memory footprint grows well beyond what the “pred” tensor would be for this number of samples and also if there’s a standard practice here like using a data loader with batch sizes etc. I went through the tutorial but only see the workflow through saving off the trained model.</p>

username: builder173
cooked: <p>Adding something more for feedback. This approach using a data loader does work, albeit slowly at a batch size of 100. I think choosing the batch size optimally would depend somewhat on understanding the drivers of memory usage in the prediction on a batch. (e.g why does memory diverge when the entire dataset prediction is attempted in one step) I’ve seen some references to retention of the computational graph for each sample and so forth.</p>
<pre><code class="lang-auto">spec_ds = TensorDataset(x_data)
pred_bs = 100
spec_dl = DataLoader(spec_ds, batch_size=pred_bs)

model.eval()
with torch.no_grad():
    pred = torch.tensor([])
    for batch, X in enumerate(spec_dl):
        pred = torch.cat((pred, model(X[0])), dim=0)
</code></pre>
Find array elements

username: John4
cooked: <p>Hi. I want to translate the following matlab code in pytorch. Is there a simple way to perform it?</p>
<pre><code>idx = randperm(numberOfDbExamples,5);
for i=1:length(idx)
    k(i)=find(labels==classificationAll(idx(i)));
end
</code></pre>
<p>cl=labelNames(k,1);</p>

username: KFrank
cooked: <p>Hi John!</p>
<aside class="quote no-group" data-username="John4" data-post="1" data-topic="164014" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e0b2c6/48.png" class="avatar"> John4:</div>
<blockquote>
<pre><code>    k(i)=find(labels==classificationAll(idx(i)));
</code></pre>
</blockquote>
</aside>
<p>Apply <a href="https://pytorch.org/docs/1.12/generated/torch.nonzero.html" rel="noopener nofollow ugc">torch.nonzero()</a> to the boolean expression for the elements you want.<br>
<code>nonzero()</code> returns the indices of those elements in pytorch format.</p>
<p>Best.</p>
<p>K. Frank</p>

username: John4
cooked: <p>Hi Frank, thank you for the help. I tried this:</p>
<pre><code>k[i] = torch.nonzero(labels == classificationAll[idx[i]])
</code></pre>
<p>But I get the error: TypeError: nonzero(): argument ‘input’ (position 1) must be Tensor, not bool.<br>
Actually, labels is an array of 4 elements (so what I want from labels == classificationAll[idx[i]] is something like [0, 0, 1, 0]) , while classificationAll is a tensor. Maybe, should I convert it in an other form?</p>

username: KFrank
cooked: <p>Hi John!</p>
<aside class="quote no-group" data-username="John4" data-post="3" data-topic="164014" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e0b2c6/48.png" class="avatar"> John4:</div>
<blockquote>
<p>But I get the error: TypeError: nonzero(): argument ‘input’ (position 1) must be Tensor, not bool.</p>
</blockquote>
</aside>
<p>Please try to rewrite your matlab code using just pytorch tensors and tensor<br>
operations.</p>
<p>If you have questions, please post a complete, fully-self-contained, runnable<br>
(or at least runnable up to where it raises an error) script together with its<br>
output and a description of what you would like it to do differently.</p>
<p>Best.</p>
<p>K. Frank</p>

username: John4
cooked: <p>I forgot to convert an array to tensor. Thank you very much!</p>
Higher validation loss after using Dropout

username: Saswat
cooked: <p>I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use <code>nn.CrossEntropyLoss()</code> as my loss function. I noticed that my model was overfitting (validation loss increased after some epochs) and hence I used a dropout (p=0.4) in the last FCN layer. I’m now seeing a higher loss (for both train and val)</p>
<p>The green line denotes training <strong>without</strong> dropout<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5da33187d40ac0ff2788c68c836f7a756298647.png" data-download-href="https://discuss.pytorch.org/uploads/default/d5da33187d40ac0ff2788c68c836f7a756298647" title="Screenshot 2022-10-22 at 7.09.36 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5da33187d40ac0ff2788c68c836f7a756298647_2_690x172.png" alt="Screenshot 2022-10-22 at 7.09.36 PM" data-base62-sha1="uvPbSNQAbgJygxEs0lo2ps9yIhV" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5da33187d40ac0ff2788c68c836f7a756298647_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5da33187d40ac0ff2788c68c836f7a756298647_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5da33187d40ac0ff2788c68c836f7a756298647_2_1380x344.png 2x" data-dominant-color="F7F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-22 at 7.09.36 PM</span><span class="informations">2120×530 97.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this behavior justified or am I doing something wrong?</p>
Tensors with dimensions known at compile time

username: dshin
cooked: <p>The <code>Eigen</code> library allows you to declare <code>Matrix</code> types whose dimensions are fixed, thus avoiding dynamic memory allocation when constructing such objects.</p>
<p>Does PyTorch C++ have something similar? Or must all <code>Tensor</code>’s incur the cost of dynamic memory allocation?</p>

username: ptrblck
cooked: <p>PyTorch doesn’t allow you to directly allocate memory via the Python frontend. However, if you are concerned about the costs to allocate GPU memory, you could initially allocate a large tensor, delete it, and allow PyTorch to reuse this memory as its internal cache. In C++ you would be able to allocate your own objects and pass these to PyTorch via <code>torch::from_blob</code>. Temp. tensors would still be managed by PyTorch.</p>
Solving this ai problem

username: Jaideep_Valani
cooked: <p>I have any  building floor plan<br>
User like to automate the task of plotting<br>
Color &amp; symbol  coded light controlling device (rectangular boxes with color &amp; symbol ) on to floor plan wherever needed .<br>
How can this be solved  below are possible solution ,suggest the best possible approach .</p>
<ol>
<li>Using any image processing solution</li>
<li>Obj detection problem is there may be many floor plans needed loads of devices ,quite tedious of generate  annotation plus floor plan is just contain edges and boundaries with text label designating what that block is</li>
<li>Gans . I give raw and white image , gan return color coded corrected image of it .<br>
This is like floor plan generator<br>
Archigan: generative task for apartment building design posted on nvidia developer  blog .</li>
</ol>
grad.grad_fn is None in Hessian calculation

username: dudifrid
cooked: <p>When debugging Hessian calculation in my code I found that it gets an undesired <code>None</code>.<br>
Following is a simplification of the problem. I need non-<code>None</code> values in both cases.</p>
<p>In the following code, grad.grad_fn is not None.</p>
<pre><code>import torch
import torch.nn as nn

x = torch.randn(5)
f = lambda x : x.sum()
x.requires_grad_(True)
with torch.enable_grad():
    fval = f(x)
    print(fval)
    grad, = torch.autograd.grad(fval, x, create_graph=True)
#     grad.requires_grad = True
    print(grad)
</code></pre>
<p>However, when replacing f’s definition with the following one we get None</p>
<pre><code>import torch
import torch.nn as nn


x = torch.randn(5)
f = lambda x : x.sum()
x.requires_grad_(True)
with torch.enable_grad():
    fval = f(x)
    grad, = torch.autograd.grad(fval, x, create_graph=True)
    print(grad.grad_fn)
</code></pre>
<p>What should I do to make it non-<code>None</code>?</p>
Erratic loss after introducing class weights

username: Saswat
cooked: <p>I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use <code>nn.CrossEntropyLoss()</code> as my loss function. To tackle the problem of class imbalance, I use sklearn’s class_weight while initialising the loss</p>
<pre><code class="lang-auto">from sklearn.utils import class_weight

class_weights = class_weight.compute_class_weight(class_weight='balanced', classes=np.unique(y_org), y=y_org)
</code></pre>
<p>The problem I’m facing is that, without class weighing, the iteration loss decreases gracefully and reaches a certain minimal value. Upon adding class weights, the iteration loss shows several oscillations and the overall validation loss doesn’t reduce to as low a value as it used to.</p>
<p>In the below diagram, the orange plot is while training without weights<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24862a4d6abdc2a82cd1f40be151aa3a380f798a.png" data-download-href="https://discuss.pytorch.org/uploads/default/24862a4d6abdc2a82cd1f40be151aa3a380f798a" title="Screenshot 2022-10-22 at 12.18.05 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24862a4d6abdc2a82cd1f40be151aa3a380f798a_2_690x205.png" alt="Screenshot 2022-10-22 at 12.18.05 AM" data-base62-sha1="5d6DtgoHRvRoVtqQc3Gsi8zmwgG" width="690" height="205" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24862a4d6abdc2a82cd1f40be151aa3a380f798a_2_690x205.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24862a4d6abdc2a82cd1f40be151aa3a380f798a_2_1035x307.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24862a4d6abdc2a82cd1f40be151aa3a380f798a_2_1380x410.png 2x" data-dominant-color="F8F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-22 at 12.18.05 AM</span><span class="informations">2120×632 99 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any explanation for this behaviour?</p>
4 dimension for forecasting portfolio with LSTM

username: maywind
cooked: <p>Hello,<br>
I am leveraging the LSTM for forecasting portfolio. The input is of shape (batch_size, sequences_length, number_assets, features), in which features refers to Open price, Closing price, Volume and etc. The expected output should be of shape (batch_size, 1, number_assets, closing_price), in which closing price is the forecasting target. However, LSTM only supports inputs of 3-dimension. How to resolve this issue that troubled me for so long time? Thanks!</p>
Indexing results differ to Python frontend

username: ngageorange
cooked: <p>Hello,<br>
I’m trying to port over some Python code to (admittadly not C++) the Rust backend and I’ve got a problem where the indexing in the Python version results in a different tensor compared.</p>
<p>The python code is:</p>
<pre><code class="lang-python">rand_ts = torch.arange(200).reshape(1, 2, 10, 10)
t2 = torch.Tensor([1, 2, 3]).to(int)
t3 = torch.Tensor([5, 6, 7]).to(int)

out = rand_ts[0, 1, t2, t3]
</code></pre>
<p>Results in a Tensor of shape <code>[3]</code>. Which is what I’d like. However, in Rust:</p>
<pre data-code-wrap="rust"><code class="lang-nohighlight">let rand_t = Tensor::range(0, 199, (tch::Kind::Float, tch::Device::Cpu)).view([1, 2, 10, 10]);
let hindex = Tensor::of_slice(&amp;[0, 1, 2]);
let windex = Tensor::of_slice(&amp;[5, 6, 7]);

let subselected = rand_t.i((0, 1, &amp;hindex, &amp;windex));
</code></pre>
<p>This results in a Tensor of shape <code>[3, 3]</code>.</p>
<p>Any thoughts on how I can get the Rust (or if you can provide an example, a C++) version to behave the way the Python version does?</p>
<p>Many thanks</p>
Activation functions in nn.TransformerEncoderLayer

username: Martin_Aegidius
cooked: <p>Hi all,</p>
<p>I am currently debugging a transformer’s encoder, as it does not learn as expected. I have registered a number of forward-hooks in order to check outputs of different steps in the forward pass. I would like to check if I have tons of dead-relus, but can not figure out where the activation is located. A vanilla nn.TransformerEncoderLayer has the following structure:</p>
<p>TransformerEncoderLayer(<br>
(self_attn): MultiheadAttention(<br>
(out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)<br>
)<br>
(linear1): Linear(in_features=512, out_features=2048, bias=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(linear2): Linear(in_features=2048, out_features=512, bias=True)<br>
(norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)<br>
(norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)<br>
(dropout1): Dropout(p=0.1, inplace=False)<br>
(dropout2): Dropout(p=0.1, inplace=False)<br>
)</p>
<p>Should there not be an activation layer between linear1 and linear2 in the feed-forward block? That seems to be the case when reading <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noopener nofollow ugc">Attention Is All You Need, page 5, equation (2)</a>. I have looked at the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/transformer.html#TransformerEncoderLayer" rel="noopener nofollow ugc">source-code</a>, but am unable to figure out where the activations happen in the pyTorch-implementation.</p>
<p>Does anyone know where to place the forward-hook for checking for dead-relus in this implementation?</p>
<p>Best,<br>
Martin</p>
Can I delete an attribute in RecursiveScriptModule (Pytorch)?

username: tranlevision
cooked: <p>I have a script module like below (this actually is a module of a pre-trained model saved as a RecursiveScriptModule in Pytorch):</p>
<pre><code class="lang-auto">RecursiveScriptModule(
  original_name=out_Conv
  (conv): RecursiveScriptModule(original_name=Conv2d)
  (bn): RecursiveScriptModule(original_name=BatchNorm2d)
  (act): RecursiveScriptModule(original_name=Sigmoid)
)
</code></pre>
<p>Could I delete the attribute ‘act’ in this module then save it as a new model?</p>
Name 'ResNet18_Weights' is not defined

username: Chris36839
cooked: <p>pretrained_net = torchvision.models.resnet18(weights=ResNet18_Weights.DEFAULT)<br>
I have imported torchvision, but it still went wrong.</p>

username: ptrblck
cooked: <p>Make sure to import the weights from <code>torchvision.models</code> via:</p>
<pre><code class="lang-python">from torchvision.models import ResNet50_Weights
</code></pre>
Filling block diagonal part with varying size

username: SeungjinKang
cooked: <p>Given the tensor which specifies the sizees of block matrices, I am trying to fill the block diagonal parts with specific value. For instance, let the size of each block to be <code>size_tensor</code></p>
<pre><code class="lang-auto">size_tensor = torch.tensor([2,1,3], device='cuda')
</code></pre>
<p>The output I want to get looks like this:</p>
<pre><code class="lang-auto">output = 
[1, 1, 0, 0, 0, 0],
[1, 1, 0, 0, 0, 0],
[0, 0, 1, 0, 0, 0],
[0, 0, 0, 1, 1, 1],
[0, 0, 0, 1, 1, 1],
[0, 0, 0, 1, 1, 1]
</code></pre>
<p>As you can see, we fill block diagonal part with size defined in <code>size_tensor</code> with specific value (in this case <code>1</code>). Note that the size of each block is not same. My question is, what would be the most efficient way to perform this operation?</p>
<p>Currently, my code looks as follows:</p>
<pre><code class="lang-auto">block_components = [torch.full((x,x), 1, device='cuda') for x in size_tensor]
output = torch.block_diag(*block_components)
</code></pre>
<p>However, this seems a bit slow when I’m working on GPU device. In my case, the <code>size_tensor</code> actually comes from some previous operations on GPU, so it lives on GPU device. Since iterating tensors on GPU is slow, if I change above code to</p>
<pre><code class="lang-auto">block_components = [torch.full((x,x), 1, device='cuda') for x in size_tensor.to('cpu')]
output = torch.block_diag(*block_components)
</code></pre>
<p>The code runs a bit faster. Howver, this code still runs a operation which sends GPU tensor (<code>size_tensor</code>) to CPU, so this seems not an optimal way to do it.</p>
<p>So my question is, what would be the most efficient way of creating block diagonal matrix in this situation?</p>
How to add gradient L1 regularization to loss?

username: nobody
cooked: <p>I need to add gradient L1 regularization to my loss, and I find that only the following two times backward implementation can make its update result different from the result without gradient regularization. If torch.autograd.grad is used to replace the first backward, the updated result is the same as the result without gradient regularization, I don’t know why this happens. And whether the two backward implementation would actually work as I hoped.</p>
<p>the code of toy:</p>
<pre><code class="lang-auto">import torch
x = torch.randn(3,4)
fc1 = torch.nn.Linear(4,3)
fc2 = torch.nn.Linear(4,3)
fc3 = torch.nn.Linear(4,3)
fc2.weight.data.copy_(fc1.weight.data)
fc2.bias.data.copy_(fc1.bias.data)
fc3.weight.data.copy_(fc1.weight.data)
fc3.bias.data.copy_(fc1.bias.data)
opt1 = torch.optim.SGD(fc1.parameters(), lr=0.01)
opt2 = torch.optim.SGD(fc2.parameters(), lr=0.01)
opt3 = torch.optim.SGD(fc3.parameters(), lr=0.01)

# without gradient regularization
y1 = fc1(x)
loss1 = (1-y1).sum()
loss1.backward()
opt1.step()

# with gradient regularization, first implementation
y2 = fc2(x)
l2 = (1-y2).sum()
l2.backward(create_graph=True)
np2 = sum([p.grad.norm(p=1) for p in fc2.parameters()])
loss2 = l2 + np2
loss2.backward()
opt2.step()

# with gradient regularization, second implementation
y3 = fc3(x)
l3 = (1-y3).sum()
np3 = sum([g.norm(p=1) for g in  torch.autograd.grad(l3, fc3.parameters(), create_graph=True) ])
loss3 = l3 + np3
loss3.backward()
opt3.step()

# print weight after update
print(fc1.weight)
print(fc2.weight)  # different to fc1.weight
print(fc3.weight)  # same to fc1.weight
</code></pre>

username: nobody
cooked: <p>I seem to know where I went wrong. The second derivative of the linear function is 0, which makes torch.autograd.grad same to without gradients norm, but not if I make it be nonlinear function.<br>
Looks like the torch.autograd.grad implementation is the right one? If anyone can help confirm would be greatly appreciated!</p>
Would skip the optimization step in autocast mode affect ZeroRedundancyOptimizer?

username: syorami
cooked: <p>I’m now using ZeroRedundancyOptimizer with autocast mode and happen to find that the updated parameters would by synced across all ranks. However, in autocast mode, when inf/NaN gradients are found, the optimization step would be skipped. Would this arouse the hang up problem when a rank skip the optimization while other ranks still wait for syncronization?</p>
<pre><code class="lang-auto">    def step(
        self,
        closure: Optional[Callable[[], float]] = None,
        **kwargs: Any,
    ) -&gt; Optional[float]:
        r"""
        Performs a single optimizer step and syncs parameters across all ranks.

        Arguments:
            closure (callable): a closure that re-evaluates the model and
                returns the loss; optional for most optimizers.
        Returns:
            Optional loss depending on the underlying local optimizer.

        .. note: Any extra parameters are passed to the base optimizer as-is.
        """
        if self._overlap_with_ddp:
            logging.warning(
                "`step()` should not be included in the training loop when "
                "`overlap_with_ddp=True`"
            )
            return None
        # Perform the local optimizer step
        loss = self._local_step(closure=closure, **kwargs)
        # Sync all of the updated parameter shards across the ranks
        self._sync_params()

        return loss
</code></pre>
MPS support for Intel macs with Metal 3 compatible AMD graphics cards?

username: Bri_Bri
cooked: <p>
Forgive me if this has been asked before or answered elsewhere, but is it possible to use PyTorch’s MPS backend on Intel macs with graphics cards that support Metal 3? According to Apple’s docs, Metal 3 is supported on the AMD Radeon Pro Vega series and Radeon Pro 5000 or 6000 series. But in spite of that, it seems like PyTorch only supports MPS on Apple Silicon macs.
</p>
<p>
Is that correct? Or, is it possible to compile PyTorch with MPS support on x86-64?
</p>
<p>edit: Nevermind, I’ve found that when I use PyTorch on my 2019 MacBook Pro with a Radeon Pro Vega 20 card, I can use the mps backend.</p>
How can I observe the weights during training?

username: BobKim
cooked: <p>Hello everyone!<br>
I would like to observe a change in certain weights in neural nets for every epoch.<br>
Then, i wanna plot the weight curve! (like test accuracy curve)</p>
<p>What should i do??</p>

username: matifali
cooked: <p>You can save the model after every epoch and then extract and plot the weights.</p>

username: BobKim
cooked: <p>Thanks a lot! I’ll try this!</p>
Torch filter multidimensional tensor by start and end values

username: Adib_Mosharrof
cooked: <p>I have a list of sentences and I am looking to extract contents between two items.<br>
If the start or end item does not exist, I want it to return a row with padding only.<br>
I already have the sentences tokenized and padded with 0 to a fixed length.</p>
<p>I figured a way to do this using for loops, but it is extremely slow, so would like to<br>
know what is the best way to solve this.</p>
<pre><code class="lang-auto">import torch
start_value, end_value = 4,9

data = torch.tensor([
[3,4,7,8,9,2,0,0,0,0], 
[1,5,3,4,7,2,8,9,10,0],
[3,4,7,8,10,0,0,0,0,0], # does not contain end value
[3,7,5,9,2,0,0,0,0,0], # does not contain start value
])

#expected output
[
[7,8,0,0,0,0,0,0,0,0],
[7,2,8,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,0,0,0,0,0],
]
</code></pre>

username: AbdulsalamBande
cooked: <p>There are two possible solutions</p>
<ol>
<li>
<p>Use the “for loop” then store the processed data in a file. This is to avoid reprocessing the data.</p>
</li>
<li>
<p>You can use Pytorch dataloader to process your inputs in batches. This can save you so much time. Check <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">Pytorch Dataloaders</a></p>
</li>
</ol>
<p>Bonus : Pytorch has utility functions like <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pad_sequence.html" rel="noopener nofollow ugc">TORCH.NN.UTILS.RNN.PAD_SEQUENCE</a> and<a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_padded_sequence.html" rel="noopener nofollow ugc">TORCH.NN.UTILS.RNN.PACK_PADDED_SEQUENCE</a></p>

username: Adib_Mosharrof
cooked: <p>i am doing this calculation inside the compute loss function, where the data is the prediction of the model, so I dont think I can use dataloaders here. In my for loop solution I used pad_sequence.<br>
I was wondering if there is a way to do tensor operations instead of using for loop.</p>
SVM to extract features

username: Gorgen
cooked: <p>Hi, everyone.<br>
Could you please tell me how to use SVM to extract audio features?</p>
<p>Thanks<br>
best</p>

username: AbdulsalamBande
cooked: <p>It all depends on all you formalize you problem. But why dont you use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html" rel="noopener nofollow ugc">svm</a> from sklearn.</p>
Backpropagation through predicted padding size values

username: lolo
cooked: <p>Hi there,</p>
<p>Suppose I have a network predicting padding sizes, i.e, a tensor of shape (batch_size, 4). The number 4 comes from top, bottom, left and right padding sizes.</p>
<p>These predicted padding sizes are then used to pad an image, i.e, a tensor of shape (batch_size, c, h, w).<br>
The padding value is constant and is zero. After padding, we have a tensor of shape (batch_size, 3, h+padding_size_h, w+padding_size_w). For clarification, padding_size_h is the sum of top and bottom values and padding_size_w is the sum of left and right.</p>
<p>I then use the padded image to calculate a pixel loss wrt to a target image of the same shape as the padded image.</p>
<p>Do either nn.ConstantPad2d or F.pad, which take int or tuples for padding sizes, break backpropagation?  That is, using F.pad(image, padding_size_predictions.long(), ‘constant’, 0), will it backpropagate gradients?</p>
<p>Thanks!</p>

username: KFrank
cooked: <p>Hi Lolo!</p>
<aside class="quote no-group" data-username="lolo" data-post="1" data-topic="163938" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lolo/48/32108_2.png" class="avatar"> lolo:</div>
<blockquote>
<p>That is, using F.pad(image, padding_size_predictions.long(), ‘constant’, 0), will it backpropagate gradients?</p>
</blockquote>
</aside>
<p>If I understand your use case correctly, you would like to train a network<br>
to predict the padding sizes.</p>
<p>This won’t work because the padding sizes are inherently integers – they<br>
are discrete variables that you can’t do calculus on.  So you can’t use<br>
backpropagation and gradient-descent-based algorithms to optimize them.</p>
<p>To answer your specific question, <code>F.pad()</code> does not support backpropagation<br>
through its padding size (<code>pad</code>) argument.  (It will backpropagate through its<br>
<code>input</code> argument, that is, through the tensor being padded).</p>
<p>Best.</p>
<p>K. Frank</p>

username: lolo
cooked: <p>Hi Frank,</p>
<p>Thanks for answering, very helpful. That’s what I assumed, but I wanted to double check in case there was some sort of differentiable operation under the hood.</p>
<p>If anyone comes up with a workaround to solve this problem please share! It would be much appreciated!</p>
<p>Thanks!</p>
Time series data prediction problems

username: NoWay2Guess
cooked: <p>Hello guys, I am doing a project which uses the input signal of a power amplifier to predict its output signal. Both signals are sinusoidal signals. My plan is to build a seq2seq LSTM model to complete the task. But I have some questions which I need some explainations.</p>
<p>First, I want to use time_series_split method to train and validate the model, however, this is my first time doing it, I am wondering if it’s possible to implement this method with a customize dataset and dataloader, and how to do that as well?</p>
<p>Second, I will have hundreds of sinusoidal signals with different amplitude and frequency as input dataset, does it impove the model performance if I use batches?</p>

username: AbdulsalamBande
cooked: <p>Question One : Yes can you implement it with a customized dataset. You can check out <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">Writing Custom Datasets, Dataloaders and Transform</a> on how to create custom datasets.</p>
<p>Question Two: Yes, training using batches can have a direct impact on accuracy.</p>
How to see gpu kernels which are ran on GPU

username: Hadi_Omidi
cooked: <p>Hi all,</p>
<p>I have two question about the gpu kernels.</p>
<ol>
<li>
<p>Is there any way to get the gpu kernel code that pytorch generates to run them on GPU? For example, running a 2D convolution layer need a kernel. I want to see that kernel file.</p>
</li>
<li>
<p>Is there any documentation that explains how to setup the pytorch with PoCL compiler.</p>
</li>
</ol>
<p>Best regards</p>

username: ptrblck
cooked: <ol>
<li>
<p>You could profile the code using the PyTorch profiler or e.g. Nsight Systems to get the kernel names. Depending on the actual kernel you could then search for it in the repository to see its implementation. If a closed third party library isused (e.g. cuDNN) you won’t be able to read the code.</p>
</li>
<li>
<p>I’m now aware of any resources for it.</p>
</li>
</ol>
Slice from 2D tensor using 2D indices

username: torchman
cooked: <p>I have a 2D tensor <code>A</code> of size <code>(n, d)</code> where each row has <code>d</code> elements<br>
and another 2D tensor <code>mask</code> of size  <code>(n, 10)</code> where each row has <code>10</code> integers in range <code>[0, d-1]</code>.</p>
<p>I wish to extract elements from the <code>(n,d)</code> tensor using indices of <code>(n,10)</code>, to obtain<br>
<code>(n, 10)</code> elements from <code>A</code>.</p>
<p>Is there a simple to do this?</p>

username: KFrank
cooked: <p>Hi Torchman!</p>
<aside class="quote no-group" data-username="torchman" data-post="1" data-topic="163945" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/torchman/48/30308_2.png" class="avatar"> torchman:</div>
<blockquote>
<p>I wish to extract elements from the <code>(n,d)</code> tensor using indices of <code>(n,10)</code>, to obtain<br>
<code>(n, 10)</code> elements from <code>A</code>.</p>
</blockquote>
</aside>
<p>If I understand your use case, yes, you may use <a href="https://pytorch.org/docs/1.12/generated/torch.gather.html" rel="noopener nofollow ugc">gather().</a></p>
<p>As a more general alternative, you may also use pytorch tensor indexing.</p>
<p>Here is an illustration of these two approaches:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt;
&gt;&gt;&gt; n = 3
&gt;&gt;&gt; d = 8
&gt;&gt;&gt; k = 4
&gt;&gt;&gt;
&gt;&gt;&gt; A = torch.arange (n * d).reshape (n, d) + 1
&gt;&gt;&gt; A
tensor([[ 1,  2,  3,  4,  5,  6,  7,  8],
        [ 9, 10, 11, 12, 13, 14, 15, 16],
        [17, 18, 19, 20, 21, 22, 23, 24]])
&gt;&gt;&gt;
&gt;&gt;&gt; mask = torch.randint (d, (n, k))
&gt;&gt;&gt; mask
tensor([[4, 5, 1, 0],
        [5, 6, 6, 4],
        [7, 6, 3, 5]])
&gt;&gt;&gt;
&gt;&gt;&gt; A.gather (dim = 1, index = mask)
tensor([[ 5,  6,  2,  1],
        [14, 15, 15, 13],
        [24, 23, 20, 22]])
&gt;&gt;&gt;
&gt;&gt;&gt; ind0 = torch.arange (n).unsqueeze (-1).expand (n, k)
&gt;&gt;&gt; ind0
tensor([[0, 0, 0, 0],
        [1, 1, 1, 1],
        [2, 2, 2, 2]])
&gt;&gt;&gt;
&gt;&gt;&gt; A[ind0, mask]
tensor([[ 5,  6,  2,  1],
        [14, 15, 15, 13],
        [24, 23, 20, 22]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
TypeError: __init__() missing 1 required positional argument: 'Input'

username: hananelroe
cooked: <p>here is my code:</p>
<pre><code class="lang-auto">
import torch
import torchaudio
import matplotlib.pyplot as plt
import os
import torch.nn as nn
import soundfile as sf
from pydub import AudioSegment
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")


AUDIO_DIR = './data/audio/'
IDX = 1
REUQESTED_FILE = str(IDX) + '.wav'
FilePath = os.path.join(str(AUDIO_DIR), str(REUQESTED_FILE))

def waveformToMelSpecrogram(wavFilePath):
    waveform, sample_rate = torchaudio.load(wavFilePath)
    waveform, sample_rate = sf.read(wavFilePath)
    waveform = AudioSegment.from_mp3(wavFilePath)
    waveform = waveform.set_channels(1)
    waveform = waveform.get_array_of_samples()
    waveform = torch.tensor(waveform, dtype=torch.float)
    waveform = torch.reshape(waveform, (1, waveform.shape[0]))
    Mel_Spectrogram = torchaudio.transforms.MelSpectrogram()(waveform)
    return Mel_Spectrogram


class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(128*1756, 2),
            nn.ReLU()
        )

        def forward(self, x):
            x = self.flatten(x)
            logits = self.linear_relu_stack(x)
            return logits


specgram = waveformToMelSpecrogram(FilePath)
'''print("Shape of spectrogram: {}".format(specgram.size()))

plt.figure()
plt.imshow(specgram.log2()[0,:,:].numpy())
plt.show()
'''
net = NN().to(device)
print(NN())
</code></pre>
<p>what did I do wrong? right now I’m just trying to pass a mel spectrogram through a simple net.</p>

username: ptrblck
cooked: <p>Could you post the entire error message including the stacktrace, as I’m currently unsure which object raises the error, please?</p>

username: hananelroe
cooked: <p>I managed to solve it myself, thanks anyway</p>

username: Peter_Karasev
cooked: <p>Even if you solved it yourself, posting something about the solution is much appreciated. Because, google ends up at this question when someone search for similar error message text …</p>
How to apply my vocabs to pretrained embedding?

username: juhyung
cooked: <pre><code class="lang-python">class emb(nn.Module):
    def __init__():
        glove = vocab.GloVe(name='6B', dim=300)
        self.emb = nn.Embedding.from_pretrained(glove.vectors, freeze=False)
    def forward():
        in_context_emb = self.emb(in_context)
</code></pre>
<p>I’m using this code and <code>in_context</code> has 20001 words in it.<br>
I wonder how 20001 words go into GloVe.<br>
How can I efficiently apply glove vectors with my vocabs?</p>

username: Bishwa_Karki
cooked: <p>Did you solve this problem?</p>
Transfer learning of weights to one model to another

username: Skinish
cooked: <p>I have a simple problem: I want to transfer the weights of one model to another, such that only the final layers of the model differ, an MLP for example.</p>
<p>I am loading the pretrained model with `load_state_dict, but it tries to copy ALL the weights of the pretrained model. How do I copy only the weights that I want, or that are existing in the model (aka, same layer name)?</p>

username: ptrblck
cooked: <p>You could adapt <a href="https://discuss.pytorch.org/t/copy-weights-only-from-a-networks-parameters/5841/2?u=ptrblck">this</a> code snippet and set your condition to the layer names you would like to copy.</p>

username: ptrblck
cooked: <p>Looks alright.<br>
What is <code>init_weights()</code> doing?<br>
Is it just initializing the last two linear layers?</p>

username: Skinish
cooked: <p>I was looking at that exactly now! I’ll fix the <code>init_weights</code>. Thanks!!</p>

username: Skinish
cooked: <p>I don’t know why but my previous post disappeared. Therefore, I am posting the solution once again:</p>
<pre><code class="lang-auto">def load_weights(self):
    pretrained_dict = torch.load('model.torch')
    model_dict = self.state_dict()

    # 1. filter out unnecessary keys
    pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}

    # 2. overwrite entries in the existing state dict
    model_dict.update(pretrained_dict)

    # 3. load the new state dict
    self.load_state_dict(model_dict)

    # 4. create final layer and initialize it
    self.linear = nn.Linear(self.output_size, self.n_classes)
    torch.nn.init.xavier_uniform_(self.linear.weight)
    self.linear.bias.data.fill_(0.01)
    self.cuda()  # put model on GPU once again
</code></pre>

username: Adroit
cooked: <p>Hi, I use the above function to transfer training weight from another model. However, once I use model.apply(load_weights), and try to train the model, my weights don’t update. Is there any additional step after this function that should let optimizer update the weights?</p>