Using input tensors for loss instead of output y_pred

pydoe: <p>Hi,</p>
<p>for RL purposes I want to calculate the loss from MSE from the inputs x and x_expected: <code>loss(x, x_expected)</code>, instead of <code>loss(y_pred, y_expected)</code>. In the latter case the gradients are changing over time, but when using the inputs, it stalls. Anything I have to consider? Like copying <code>grad_fn</code> from the <code>output</code> / <code>y_pred</code> onto <code>x</code> so that the loss recognizes the graph and so on?</p>

tumble-weed: <p>actually you point out that the graph might not be recognized, and this makes sense that with <code>x</code> and <code>x_expected</code> the computation through a network would not be taking place, so no learnable parameters would be involved, and so there shouldnt be any gradients for them. are <code>x</code> and <code>x_expected</code> themselves coming through some other computation graph that you expect to learn through your new loss?</p>

pydoe: <p>no, they are not. just plainly incoming as tensors with required_grad = True and retain_grad()</p>
<p>I give an example of the <strong>scenario</strong>: x = sensor data (distance to player) is coming in, x_target target distance to player is given (0). This squared difference between these should make up the loss.<br>
The net predicts aktuator movements as y.</p>

pydoe: <p>no one? I hope there is a way to do this</p>

tumble-weed: <p>understood, so the learnable parameters are the ones that dictate how the actuator movements are predicted. i’ll suggest a scheme with a notation different from yours, i hope it can be of use<br>
current position/distance to player : x_now<br>
actuator movement: y_now = net(x_now)<br>
next position: x_next = x_now + y_now ( i am simplifying this as addition of actuator movement to position, you can of course have whatever physical function that dictates how action changes position)<br>
loss : L(x_next,x_target) , e.g. (x_next - x_target)**2<br>
due to x_next involving y_now which in turn involves net params, now you will get non zero gradients for the parameters</p>

pydoe: <p>hm… very interesting.</p>
<p>but let’s say x_now is the 1d-height of the center of the head [17cm] and y_now outputs 4 controls of the legs as [0, 0.1, 0.6, 0.9], then i cannot simply add x_now and y_now together!?</p>

tumble-weed: <p>yes that is why i made a comment in that part of the scheme. you will have to look at the physical model of your problem to fill in that part ( how does the head height change if some legs are extended etc.)</p>

pydoe: <p>I hoped the model can learn that^^ it might easily be more complex than my example, also it would be nice to keep it as general as possible.</p>

pydoe: <p>I mean, my first approach would’ve been to use the <code>output</code> (1x4), and just replace the tensor values with the <code>x</code> tensor values (1x1).</p>
<p>Update: I tried this:<br>
(1) combining output with input</p>
<pre><code class="lang-auto">x_noleaf = x.clone() # cloning input to use as non-leaf node
x_noleaf[0] = x_noleaf[0] + y[0] - y[0]  # for simulating an interaction between the two x = x + Y - Y
</code></pre>
<p>this didn’t seem to do anything</p>
<p>(2) modifying the output</p>
<pre><code class="lang-auto">x_new = output.clone()
x_new = x_new[0][0:2]  # slicing for dropping elements
x_new.data[ix] = x_old.data[ix]
loss(x_new, x_target)
</code></pre>
<p>might work…</p>

tumble-weed: <p>I didnt understand the second approach,sorry. for the first i think you tried assuming that introducnig <code>y</code> into the graph somehow would cause gradient flow. however you use <code>y[0]</code> as well as <code>- y[0]</code> which admit opposite gradient flows that cancel each other.</p>
<p>even without going into details of gradient flow, the statement essentially is the same as <code>x_noleaf[0] = x_noleaf[0]</code> which shouldnt have helped if you think about it.</p>
<p>i think you are trying elbow in gradients, while the overall consideration of why gradients should flow and if indeed they will be useful for reducing the loss you are interested in is getting lost.</p>

pydoe: <p>well, yes, I am of course not sure, what I am doing. But I mean, I can use any kind of loss in tensorflow, so it is something special with autograd. If it was me, I would simply have output = net(input), backpropagate within there and it’s fine.</p>
<p>My second approach is very simple. Copying the output and putting the input tensor values inside, while dropping the unused values. Then using the modified output in the loss. So kinda hacking the output to match the input values, while keeping the graph. Gradients are changing at least, going to inf and nan quickly though with SGD</p>

my3bikaht: <p>As far as I understand you need to update network with loss based on value you provide from some other place. And problem is that: a) you have multiple outputs, b) by replacing outputs you are losing grad_fn link for the backward pass.<br>
Easiest solution I can imagine is to create custom loss function, sum all outputs, multiply this sum by zero and add torch.Tensor(of your value) <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

pydoe: <p>Thank you all! I think it works better now</p>
<pre><code class="lang-auto">class Custom_MSELoss(torch.nn.Module):
    def __init__(self):
        super(Custom_MSELoss, self).__init__()

    def forward(self, output, x_train, x_expected):
        x_train = torch.sum(output) * 0 + x_train  # &lt;--!
        MSE = [(x_train[0][i] - x_expected[0][i])**2 for i in range(len(x_expected[0]))]
        MSE = sum(MSE) / len(MSE)
        return MSE

criterion = Custom_MSELoss()
</code></pre>
<p>Looks fine to me. Could any operation here potentially detach the history?</p>