How to backpropagate a black box generated cost value

Dogukan_Avci: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc209f0f04d4512ac496291fb4cb37bc5c6189b3.png" alt="backpropproblem" data-base62-sha1="vpkVVGlkw0ffrpxLiUzOMHEJ4AP" width="491" height="131"></p>
<p>I want to minimize the error generated by my black box but there is no grad_fn attached to it. How could the autograd know what to do with this loss value?</p>

ptrblck: <p>Autograd cannot create the backward pass without a <code>grad_fn</code> and you should get an error like:</p>
<pre><code class="lang-python">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Did you detach the tensor manually from the computation graph or what is your exact use case?</p>

Dogukan_Avci: <p>The thing is the value generated by RNN is “interpreted”. No computations are made on it in blackbox so the error generated is detached. Is there any way I could backpropagate to minimize that detached error (by somehow manually attaching it).</p>

ptrblck: <p>The only valid way I could think of at the moment, is to write the <code>backward</code> function manually for your blackbox methods.</p>

Dogukan_Avci: <p>The blackbox is a simulation, I do not think it is possible to write a backward for it. I attempted to call back() on the output of RNN directly giving the error as all the gradients, it failed the system.</p>

ptrblck: <p>The computation graph is built during the forward pass and is used in the backward pass to calculate the gradients.<br>
If you are detaching some tensors from the graph, you would need to write the backward manually, as explained.</p>
<p>If that’s not possible, you won’t be able to compute the gradients using the standard backward call.<br>
Maybe some other training approaches might work (e.g. evolutionary algorithms)?</p>

geunseok_choi: <p>Hi,<br>
I have a same problem.<br>
How did you solve the problem.<br>
I don’t know how to get the gradient of black box.</p>
<p>Best.</p>

hao_chen: <p>I am facing a similar problem. I apply RNN (LSTM/ or GRU or TFT) to generate driving signals, however, there is no  label data(target signal) to calculate loss function directly. However, there is one-to-one relationship (pre-known with certain unknown noises) between the driving signals and the response signal. The response signal has target data to generate loss function.<br>
I am wondering to applied decoder (another LSTM to model the relation-ship between driving signal and the response), however, this might increase the complexity (since the relationship between driving signal and the response is 90% known in advance).<br>
Now I am learning the autograd of Pytorch, hoping that the parameters of RNN could be auto-corrected by the indirect loss function. I try to make the response signal to be  (requires_grad=True), however, it does not update the model at all, since the loss keeps all the same during training.</p>
<p>Any suggestion?<br>
Best regards.</p>

mathias: <p>Hi!<br>
Perhaps you could use a so-called straight through estimator to bypass the black-box?<br>
See e.g.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.hassanaskary.com/python/pytorch/deep%20learning/2020/09/19/intuitive-explanation-of-straight-through-estimators.html">
  <header class="source">
      <img src="https://www.hassanaskary.com/images/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://www.hassanaskary.com/python/pytorch/deep%20learning/2020/09/19/intuitive-explanation-of-straight-through-estimators.html" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 19 September 2020">Hassan Askary – 19 Sep 20</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/383;"><img src="https://hassanaskary.com/images/intuitive-explanation-of-ste-with-code/ste-visualization.png" class="thumbnail" width="690" height="383"></div>

<h3><a href="https://www.hassanaskary.com/python/pytorch/deep%20learning/2020/09/19/intuitive-explanation-of-straight-through-estimators.html" target="_blank" rel="noopener nofollow ugc">Intuitive Explanation of Straight-Through Estimators with PyTorch Implementation</a></h3>

  <p>Straight-Through Estimators are used to estimate the gradients of a discrete function.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Best regards</p>

hao_chen: <p>Thank you so much for the fast reply. I will research on it and update the results.<br>
Best</p>

hao_chen: <p>Hi, I try to apply the STE and self-defined function for indirect loss-function calculation. There are several questions listed below.<br>
Here are the snippet of my code.</p>
<h1>
<a name="normalized-time-series-data-with-lookback-1" class="anchor" href="#normalized-time-series-data-with-lookback-1"></a>normalized time series data with lookback</h1>
<p>df = pd.DataFrame()<br>
for i in range(look_back):</p>
<pre><code>df = pd.concat([df, pd.DataFrame(train_X_norm.tolist()[i:-(look_back - i)])], ignore_index=True, axis=1)
</code></pre>
<p>print("df: ")<br>
print(df)</p>
<p>df_numpy = np.array(df)<br>
df_tensor = torch.Tensor(df_numpy)</p>
<h1>
<a name="here-is-the-trainset-class-definition-2" class="anchor" href="#here-is-the-trainset-class-definition-2"></a>Here is the trainset class definition</h1>
<p>class mytrainset(Dataset):</p>
<pre><code>def __init__(self, df_tensor, train_y):
    self.data, self.label = df_tensor[:, :].float(), train_y[:]
def __getitem__(self, index):
    return self.data[index], self.label[index]
def __len__(self):
    return len(self.data)
</code></pre>
<h1>
<a name="define-the-trainset-and-loader-specify-the-device-3" class="anchor" href="#define-the-trainset-and-loader-specify-the-device-3"></a>Define the trainset and loader, specify the device</h1>
<p>trainset = mytrainset(df_tensor, train_y_norm[look_back:])  # train_y<br>
trainloader = DataLoader(trainset, batch_size=10, shuffle=False)<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
print(device)</p>
<h1>
<a name="the-stefunction-class-definition-with-definition-of-the-driving-signal-to-response-signal-which-will-be-used-to-calculate-loss-function-4" class="anchor" href="#the-stefunction-class-definition-with-definition-of-the-driving-signal-to-response-signal-which-will-be-used-to-calculate-loss-function-4"></a>The STEFunction class definition with definition of the driving signal to response signal, which will be used to calculate loss function.</h1>
<p>class STEFunction(autograd.Function):</p>
<pre><code>@staticmethod
def forward(ctx, inter_state):
    # from sensor signal to output driving signals and then to the response by STE
   # The following is for the response signal calculation, however, we will replace this part with real sensor signal directly in experiment.
    ttemp = torch.squeeze(ctx[:, -1, :]).requires_grad_(True).detach().cpu().numpy()
    ttemp = ttemp.transpose()
    if inter_state is None:
        _, tout, inter_state = control.forced_response(env.AMP_signal_acc, env.mat_step4['t'][0][
                                                                         1:11],  # batch_size=10
                                                     ttemp,
                                                     return_x=True)
        state = inter_state[:, -1]
    else:
        _, tout, inter_state = control.forced_response(env.AMP_signal_acc, env.mat_step4['t'][0][
                                                                         1:11],  # batch_size=10
                                                     ttemp,
                                                     inter_state, return_x=True)
        state = inter_state[:, -1]
    out_u = torch.tensor(tout.transpose(), requires_grad=True).type(torch.FloatTensor)

    ctx.save_for_backward(state)
    return out_u, state

@staticmethod
def backward(ctx, grad_output):
    # out = nn.Linear(6, 32 * 2)  # grad_output.size = 6
    # return out(grad_output)
    state = ctx.saved_tensors
    grad_input = grad_output.clone()
    return grad_input, grad_input
</code></pre>
<h1>
<a name="here-is-the-class-definition-of-ste-5" class="anchor" href="#here-is-the-class-definition-of-ste-5"></a>Here is the class definition of STE</h1>
<p>class StraightThroughEstimator(nn.Module):</p>
<pre><code>updated_state = None
def __init__(self, state):
    inter_state = state
    super(StraightThroughEstimator, self).__init__()  # inter_state :super() 
    # https://realpython.com/python-super/
def forward(self, x, state):
    x, updated_state = STEFunction.apply(x, state)  # how to pass the self.state back to RNN class instance
    return x, updated_state
</code></pre>
<h1>
<a name="the-rnn-with-bilaterial-lstm-rnn-to-define-encoder-and-decoder-however-in-my-case-i-only-use-encoder-i-expande-the-encoder-step-by-step-for-debugging-6" class="anchor" href="#the-rnn-with-bilaterial-lstm-rnn-to-define-encoder-and-decoder-however-in-my-case-i-only-use-encoder-i-expande-the-encoder-step-by-step-for-debugging-6"></a>The RNN with bilaterial LSTM RNN to define encoder and decoder,  however, in my case, I only use encoder. I expande the encoder step by step for debugging.</h1>
<p>class RNN(nn.Module):</p>
<pre><code>def __init__(self, input_size):
    super(RNN, self).__init__()
    self.state = None
    hidden_size = 32  # 64
    self.encoder_embedding = nn.Linear(input_size, hidden_size)
    self.encoder_rnn = nn.LSTM(
        input_size=hidden_size,
        hidden_size=hidden_size,  # 64
        num_layers=1,
        batch_first=True,
        dropout=0.1,
        bidirectional=True,
        # return_sequences=True,
    )
    self.encoder_out = nn.Linear(hidden_size * 2, 6)  # output 6 signals to drive actuators
    StraightThroughEstimator()

    self.decoder_embedding = nn.Linear(6, hidden_size)  # from 6 driving signals to LSTM
    self.decoder_rnn = nn.LSTM(
        input_size=hidden_size,
        hidden_size=hidden_size,  # 64
        num_layers=1,
        batch_first=True,
        dropout=0.1,
        bidirectional=True,
    )
    self.decoder_out = nn.Linear(2 * hidden_size, input_size)

def forward(self, x, encode=True, decode=False):
    if encode:
        x = self.encoder_embedding(x)
        r_out, (h_n, h_c) = self.encoder_rnn(x, None)  # None for initialization by 0 for hidden layers
        out = self.encoder_out(r_out)
         out_u, self.state = StraightThroughEstimator(out, self.state) #_init__() takes 2 positional arguments but 3 were given
        return out_u  # 20, 899, 1;  10, 6
    elif decode:
        # out_u = self.decoder(x)  # response to sensor signal backwards, I expand it for debugging.
        x = self.decoder_embedding(x)
        r_out, (h_n, h_c) = self.decoder_rnn(x, None) 
        out_u = self.decoder_out(r_out)
        
    else:
        # encoding = self.encoder(x)
        # out_u = self.decoder(encoding)
        x = self.encoder_embedding(x)
        t_out, (h_n, h_c) = self.encoder_rnn(x, None)  
        out = self.encoder_out(t_out)
        p_out = self.decoder_embedding(out)
        r_out, (h_n, h_c) = self.decoder_rnn(p_out, None)
        out_u = self.decoder_out(r_out)
    return out_u
</code></pre>
<h1>
<a name="here-is-the-configuration-of-network-and-rnn-7" class="anchor" href="#here-is-the-configuration-of-network-and-rnn-7"></a>Here is the configuration of network and RNN</h1>
<p>n = 1  # Acc_stage_converted 1; target driving signal 6 dimension<br>
rnn = RNN(n).to(device)<br>
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.0002)<br>
loss_func = nn.MSELoss()<br>
EPOCH = 5</p>
<h1>
<a name="here-is-the-training-8" class="anchor" href="#here-is-the-training-8"></a>Here is the training</h1>
<p>print(“Start training …”)<br>
for step in range(EPOCH):</p>
<pre><code>rnn.state = None
for tx, ty in trainloader:
    tx = tx.to(device)
    ty = ty.to(device)
    output = rnn(torch.unsqueeze(tx, dim=2))  # dim=1  # reshape input to be 3D [samples, timesteps, features]
    loss = loss_func(output[:, 2].reshape(-1, 1).to(device), ty.reshape(-1, 1))
    # clear out the gradients from the last step loss.backward()
    optimizer.zero_grad()
    # backward propagation: calculate gradients
    loss.backward()
    # update the weights
    optimizer.step()
print("%d / %d, loss = %E" % (step + 1, EPOCH, loss))
</code></pre>
<h1>
<a name="there-is-some-bug-in-rnns-forward-encode-9" class="anchor" href="#there-is-some-bug-in-rnns-forward-encode-9"></a>There is some bug in RNN’s forward: encode:</h1>
<p>out_u, self.state = StraightThroughEstimator(out, self.state) #<em>init</em>_() takes 2 positional arguments but 3 were given</p>
<p>I have to calculate the state variable for vibration response calculation.<br>
How can I implement it.</p>
<p>Bests</p>

hao_chen: <p>I am sorry, I am newbee to edit and upload code, It seems the tab or spaces are missing…</p>

mathias: <p>Hi! Sorry for the late reply.</p>
<p>The reason for the error message is that you are actually passing three arguments: self, out and self.state.<br>
Notice however that StraightThroughEstimator is the class constructor - this is not how you use it as a function during training. Below I’ll write a short code snippet to illustrate its use (the forward and backward methods are just used as exampled)</p>
<h4>
<a name="define-ste-function-1" class="anchor" href="#define-ste-function-1"></a>Define STE-function</h4>
<p>class STEFunction(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, input):<br>
return torch.heaviside(input)</p>
<pre><code>@staticmethod
def backward(ctx, grad_output):
    return F.hardtanh(grad_output)
</code></pre>
<p>class StraightThroughEstimator(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(StraightThroughEstimator, self).<strong>init</strong>()</p>
<pre><code>def forward(self, x):
    x = STEFunction.apply(x)
    return x
</code></pre>
<h3>
<a name="define-classes-with-ste-function-2" class="anchor" href="#define-classes-with-ste-function-2"></a>Define class(es) with STE-function</h3>
<p>class Class1(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Class1, self).<strong>init</strong>()</p>
<pre><code>    self.examplefn = nn.Sequential(
        nn.Linear(64,64),            # not important
        nn.ReLU(inplace=True), # not important
        StraightThroughEstimator() # STE-constructor used here
    )

def forward(self, x):
    return self.examplefn(x) # STE used here during training
</code></pre>
<p>class Class2(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Class2, self).<strong>init</strong>()</p>
<pre><code>    self.examplefn = nn.Sequential(
        nn.Linear(64,64),            # not important
        nn.ReLU(inplace=True) # not important
    )
    self.ste = StraightThroughEstimator() # STE-constructor used here

def forward(self, inputs): 
    x1 = self.examplefn(inputs)
    x2 = self.ste(x1) # STE used here
    return x2
</code></pre>
<p>Hopefully this will shed some light on how STE:s are implemented and used.</p>
<p>Best regards</p>

hao_chen: <p>Hi Mathias and  @<a href="https://discuss.pytorch.org/u/alband">albanD</a>:<br>
Thank you for the fast reply and very clear explanation. I fix this bug as following:</p>
<h1><a name="define-ste-function-1" class="anchor" href="#define-ste-function-1"></a>Define STE-function</h1>
<p>class STEFunction(autograd.Function):</p>
<pre><code>@staticmethod
def forward(ctx, input_x, inter_state):
    ttemp = torch.squeeze(input_x[:, -1, :]).requires_grad_(True).detach().cpu().numpy()
    ttemp = ttemp.transpose()
    if inter_state is None:
        _, tout, inter_state = control.forced_response(env.AMP_signal_acc, env.mat_step4['t'][0][
                                                                           1:11],  # batch_size=10
                                                       ttemp,
                                                       return_x=True)
        updated_state = inter_state[:, -1]
    else:
        inter_state = inter_state.requires_grad_(True).detach().cpu().numpy()  
        _, tout, inter_state = control.forced_response(env.AMP_signal_acc, env.mat_step4['t'][0][
                                                                           1:11],  # batch_size=10
                                                       ttemp,
                                                       inter_state, return_x=True)
        updated_state = inter_state[:, -1]

    out_u = Variable(torch.tensor(tout.transpose()), requires_grad=True).type(torch.FloatTensor).cuda()
    updated_state = Variable(torch.tensor(updated_state), requires_grad=True).type(torch.FloatTensor).cuda()

    back_out_u = torch.zeros(10, 1350, 6).cuda()
    for i in range(1350):
        back_out_u[:, i, :] = out_u
    return back_out_u, updated_state  # return out_u

@staticmethod
def backward(ctx, grad_output, grad_output1):
</code></pre>
<h4><a name="here-i-add-none-for-the-backward-of-updated_state-2" class="anchor" href="#here-i-add-none-for-the-backward-of-updated_state-2"></a>Here, I add None for the backward of updated_state</h4>
<pre><code>    return F.hardtanh(grad_output), None
</code></pre>
<h2><a name="define-class-with-ste-funciton-3" class="anchor" href="#define-class-with-ste-funciton-3"></a>Define class with STE-funciton:</h2>
<p>class StraightThroughEstimator(nn.Module):</p>
<pre><code>def __init__(self, inter_state, x):  # inter_state, x
    self.state = inter_state
    super(StraightThroughEstimator, self).__init__() 

def forward(self, inter_state, x): 
    x, updated_state = STEFunction.apply(x, inter_state)
    return x, updated_state 
</code></pre>
<p>However, I find another error as introduced below.<br>
It tells that the intermedia buffers have been freed when comes to the second time backward calling.</p>
<aside class="quote quote-modified" data-post="5" data-topic="6795">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795/5">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time</a> 
  </div>
  <blockquote>
    Hi, 
The problem in you case is that inside your training loop you do Yest = Yest + w*X which modifies the “preallocated” buffer that you created. The fact that you reset his value, does not remove his history and so your computational graph grows as it remembers averything. 
You can solve that by just changing this for loop to use another name like Yest_local = Yest_local + w*X and set before the loop Yest_local = Yest. 
That being said, a better way to solve your issue is to keep buffers as te…
  </blockquote>
</aside>

<p>In my code, the control.forced_response applied in the STEFunction class definition, is used to simulate the reponse with respect to driving signals. It needs numpy datatype and the inter_state of last step. The problem could be the<br>
ttemp = torch.squeeze(input_x[:, -1, :]).requires_grad_(True).<strong>detach().cpu().numpy()</strong><br>
and<br>
the inter_state. I save it in the Rnn class as a class member variable.</p>
<p>In order to find the problem I upload the following snippets for reference.</p>
<h2><a name="define-rnn-4" class="anchor" href="#define-rnn-4"></a>Define RNN</h2>
<p>class RNN(nn.Module):</p>
<pre><code>def __init__(self, input_size):
    super(RNN, self).__init__()
    self.state = None
    self.out = None
    hidden_size = 32  # 64
    self.encoder_embedding = nn.Linear(input_size, hidden_size)
    self.encoder_rnn = nn.LSTM(
        input_size=hidden_size,
        hidden_size=hidden_size, 
        num_layers=1,
        batch_first=True,
        dropout=0.1,
        bidirectional=True,
    )
    self.encoder_out = nn.Linear(2*hidden_size, 6)  # output 6 signals to drive actuators
    self.ste = StraightThroughEstimator(self.state, self.out) # As suggested, thank you

    self.decoder_embedding = nn.Linear(6, hidden_size)  # from 6 driving signals to LSTM
    self.decoder_rnn = nn.LSTM(
        input_size=hidden_size,
        hidden_size=hidden_size,  # 64
        num_layers=1,
        batch_first=True,
        dropout=0.1,
        bidirectional=True,
      )
    self.decoder_out = nn.Linear(2 * hidden_size, input_size)

def forward(self, x, encode=True, decode=False):
    if encode:
        x = self.encoder_embedding(x)
        r_out, (h_n, h_c) = self.encoder_rnn(x, None)  # None for initialization by 0 for hidden layers
        out = self.encoder_out(r_out)

        **out_u_1, self.state = self.ste(self.state, out)**
        out_u = out_u_1[:, -1, :]
        return out_u  # 20, 899, 1;  10, 6
    elif decode:
        # out_u = self.decoder(x)  # response to sensor signal backwards
        x = self.decoder_embedding(x)
        r_out, (h_n, h_c) = self.decoder_rnn(x, None)  
        out_u = self.decoder_out(r_out)

    else:
        # encoding = self.encoder(x)
        # out_u = self.decoder(encoding)
        x = self.encoder_embedding(x)
        t_out, (h_n, h_c) = self.encoder_rnn(x, None)  
        out = self.encoder_out(t_out)
        p_out = self.decoder_embedding(out)
        r_out, (h_n, h_c) = self.decoder_rnn(p_out, None)  
        out_u = self.decoder_out(r_out)
    return out_u
</code></pre>
<h2><a name="rnn-parameter-configuration-5" class="anchor" href="#rnn-parameter-configuration-5"></a>RNN parameter configuration:</h2>
<p>n = 1  # Only 1 channel of signal for training<br>
rnn = RNN(n).to(device)<br>
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.0002)<br>
loss_func = nn.MSELoss()<br>
EPOCH = 5</p>
<h2><a name="training-process-6" class="anchor" href="#training-process-6"></a>Training process:</h2>
<p>print(“Start training …”)<br>
for step in range(EPOCH):<br>
rnn.state = None<br>
for tx, ty in trainloader:<br>
tx = tx.to(device)<br>
ty = ty.to(device)<br>
output = rnn(torch.unsqueeze(tx, dim=2))  # reshape input to be 3D [samples, timesteps, features]<br>
loss = loss_func(output[:, 2].reshape(-1, 1).to(device), ty.reshape(-1, 1))<br>
# clear out the gradients from the last step loss.backward()<br>
optimizer.zero_grad()<br>
# backward propagation: calculate gradients. Success in the first train, fail for the second.<br>
loss.backward()<br>
# update the weights<br>
optimizer.step()<br>
print(“%d / %d, loss = %E” % (step + 1, EPOCH, loss))</p>
<p>I delete all state variables, and it can train successfully. However, It is necessary to pass this variable for response simulation.<br>
For real experiments, this response simulation will be replaced by sensor signals. It is supposed to jump out this problem.</p>
<p>Best regards and Merry Christmas</p>
<p>Chen</p>