Could loss function output a vector instead of a scalar?

saeed_i: <p>hi<br>
i want to use vector loss. i.e output of loss function is be a vector as follow:</p>
<pre><code class="lang-auto">def my_loss(y_pred, y_val):
    return (y_pred - y_val)**2
</code></pre>
<p>but in train step i (loss.backward()) I got this error:<br>
grad can be implicitly created only for scalar outputs<br>
that mean output of loss function can not be vector!!!<br>
Is there a solution to this problem?<br>
i want loss be a vector</p>

eqy: <p>Well the issue here is that in order to take a gradient, the derivative must be with respect to a single value. How should the vector loss be interpreted?</p>

Lucky_Magna: <p><a class="mention" href="/u/saeed_i">@saeed_i</a> You can use <code>reduction=None</code> parameter in the MSELoss to get what you are desiring. But I agree with <a class="mention" href="/u/eqy">@eqy</a> , in order to take a gradient, the derivative must be with respect to a single value. While calculating the gradient you must consider the error over the complete dataset.</p>

saeed_i: <p>why we cant do that???<br>
I confused about the mathematical of this. Although it may be simple !!<br>
Can you explain with an example ??<br>
thankssss</p>

KFrank: <p>Hi Saeed!</p>
<aside class="quote no-group" data-username="saeed_i" data-post="5" data-topic="122084" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saeed_i/48/32511_2.png" class="avatar"> saeed_i:</div>
<blockquote>
<p>Can you explain with an example ??</p>
</blockquote>
</aside>
<p>When you use a loss function to train a model, the loss function is<br>
telling you which set of model parameters is “better” than other sets<br>
of model parameters.</p>
<p>Let’s say you have a model, and when it has <code>weight_A</code> as its<br>
parameters it produces <code>loss_vector_A = [1.1, 4.4, 2.2]</code>.<br>
Let’s also say the when the same model has <code>weight_B</code> as its<br>
parameters it produces <code>loss_vector_B = [2.2, 1.1, 3.3]</code>.</p>
<p>Is the model a better model with <code>weight_A</code> or <code>weight_B</code>?  If the loss<br>
function produced just a scalar (instead of a vector), we would just say<br>
that the smaller scalar value corresponds to the better model.  (That’s<br>
really what “loss function” means.)</p>
<p>(If you say just add up the elements of your loss vectors to see<br>
which model is better, then you would really be saying that<br>
<code>loss_vector_A.sum()</code> and <code>loss_vector_B.sum()</code> should be<br>
your <em>scalar</em> loss-function values.)</p>
<p>Best.</p>
<p>K. Frank</p>

saeed_i: <p>yes i know these things.<br>
i confused in mathematical calucation:<br>
assume we have:<br>
x = torch.tensor([2. , 3. , 5.], requires_grad= True)<br>
y_1 = x.pow(2)<br>
y_2 = x.pow(2).sum()</p>
<p>then:<br>
y_1 = tensor([ 4.,  9., 25.], grad_fn=)<br>
y_2 = tensor(38., grad_fn=)</p>
<p>we know Derivation of x^2 is equal to 2<em>x and gradient of y_1 is be 2</em>x<br>
why we put sum  end of it(y_1==&gt;y_2)??</p>

KFrank: <p>Hi Saeed!</p>
<aside class="quote no-group" data-username="saeed_i" data-post="8" data-topic="122084" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saeed_i/48/32511_2.png" class="avatar"> saeed_i:</div>
<blockquote>
<p>we know Derivation of x^2 is equal to 2<em>x and gradient of y_1 is be 2</em>x</p>
</blockquote>
</aside>
<p>You are making the hidden assumption that <code>y_1[0]</code> depends only<br>
on <code>x[0]</code>, <code>y_1[1</code>] only on <code>x[1]</code>, and <code>y_1[2]</code> only on <code>x[2]</code>.  This<br>
happens to be true in your particular example of <code>y_1 = x.pow (2)</code>.</p>
<p>How would you change your reasoning for the case where<br>
<code>y_1 = x * x.roll (1)</code>?</p>
<p>Please look at the concept of the <em>Jacobean matrix</em> and how it is<br>
the generalization of the gradient to a <em>vector-valued</em> function.</p>
<p>(And to avoid any misconception, let me reiterate what I said in<br>
my previous post:  In your current example, <code>y_1</code> is a vector, rather<br>
than a scalar, so it cannot be used as a loss function.)</p>
<p>Best.</p>
<p>K. Frank</p>

googlebot: <p>I think <a href="https://pytorch.org/docs/master/generated/torch.autograd.backward.html#torch.autograd.backward" rel="noopener nofollow ugc">backward() documentation</a> explains it pretty well, you can backward() on vector if you specify grad_tensor[s], that is a gradient of some vector-to-scalar function, if this argument consists of ones that’s the same as .sum().backward(), and a weighted sum otherwise.<br>
As parameter.grad is for scalar-valued functions, reduction to a scalar is present one way or the other.</p>