A question about backpropagation

wentao: <p>Hello everyone,</p>
<p>I have a question about backpropagation.</p>
<p>In my program I use cross_entropy as the cost function.</p>
<p>I use a DNN predict 3 weights(w1, w2, w3) for 3 different models(M1,M2,M3). And after that I calculate the prediction like this:<br>
prediction=w1*M1+w2*M2+w3*M3(w1,w2,w3 are vectors and M1,M2,M3 are matrics)<br>
I have a targetmatrice, now I want after training prediction same as targetmatrice.<br>
my code is like this:</p>
<blockquote>
<p>prediction = w1.mul(M1) + w2.mul(M2) + w3.mul(M3)<br>
optimizer.zero_grad()</p>
<pre><code>            loss = loss_func(prediction.cuda(), target.cuda())
            loss.backward()
            optimizer.step()
</code></pre>
</blockquote>
<p>I want to ask if I do something wrong? Because the loss seems difficult to reduce.</p>
<p>Please help me<br>
Thank you<br>
Wentao</p>

KFrank: <p>Hello Wentao!</p>
<aside class="quote no-group quote-modified" data-username="wentao" data-post="1" data-topic="49491" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/a9adbd/48.png" class="avatar"> wentao:</div>
<blockquote>
<p>…<br>
In my program I use cross_entropy as the cost function.</p>
<p>I use a DNN predict 3 weights(w1, w2, w3) for 3 different models(M1,M2,M3). And after that I calculate the prediction like this:<br>
prediction=w1*M1+w2*M2+w3*M3(w1,w2,w3 are vectors and M1,M2,M3 are matrics)<br>
I have a targetmatrice, now I want after training prediction same as targetmatrice.<br>
my code is like this:</p>
<blockquote>
<p>prediction = w1.mul(M1) + w2.mul(M2) + w3.mul(M3)<br>
optimizer.zero_grad()</p>
<pre><code>            loss = loss_func(prediction.cuda(), target.cuda())
            loss.backward()
            optimizer.step()
</code></pre>
</blockquote>
<p>I want to ask if I do something wrong? Because the loss seems difficult to reduce.</p>
</blockquote>
</aside>
<p>It is not clear to me what you are trying to do.</p>
<p>One key question:</p>
<p>When you call <code>optimizer.step()</code>, what numbers are you<br>
intending to update?  Normally, you would have the optimizer<br>
update weights (and biases) in your model.  What optimizer<br>
are you using?  What did you pass in as your parameters<br>
when you instantiated <code>optimizer</code>?</p>
<p>Some more questions:</p>
<p>Are <code>M1</code>, <code>M2</code>, and <code>M3</code> fixed, and you are trying to optimize<br>
the values of <code>w1</code>, <code>w2</code>, and <code>w3</code>?  Or vice versa?  (Or both?)</p>
<p>Why are the <code>w</code>s vectors?  What are the shapes of the <code>w</code>s and<br>
<code>M</code>s?  What is the shape of <code>prediction</code>?  What are your inputs?</p>
<p>It sounds like you are trying to get a predicted matrix to match<br>
a target matrix.  You say you are using “cross_entropy”.  This<br>
doesn’t seem like the appropriate loss function for what I am<br>
guessing you are trying to do.  A loss function along the lines<br>
of mean-squared-error might be a better choice.</p>
<p>(<code>CrossEntropyLoss</code> is typically used for classification problems,<br>
and takes integral class labels for its target.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

wty: <p>Hello Frank,</p>
<p>Thank you for your reply!</p>
<p>What I‘m trying to do is stream weighting, so M1, M2, M2 are the fixed matrices. I want use dnn to predict the weights for the different models, so w1,w2,w3 are the vectors which have the same length as M1,M2,M3. The optimizer is used to update the parameters in dnn. The dnn produces the weights vectors. The cross entropy will be better for my task, because the output is kind of posteriors probability. I‘m not sure if I have made some mistakes, I don‘t know why my loss don‘t reduced.</p>
<p>Thank you<br>
Wentao</p>