Which tensors have to be included for autograd/backward?

lazhd: <p>Hi guys, after reading some stuff about autograd i still can not explain how it works in detail.</p>
<aside class="quote" data-post="1" data-topic="34039">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/6de8d8/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/help-needed-in-understanding-how-autograd-works/34039">Help needed in understanding how autograd works</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    This is just a super basic question that I have not been able to successfully google an answer to. 
I’m trying to understand how the torch.Tensor object stores information on its position in the graph. Since autograd can calculate gradients based on operational relations between torch.Tensor objects, information about these relations must be stored somewhere. Is it an attribute in the Tensor objects that I cannot find, or is there somewhere else I should look?
  </blockquote>
</aside>

<p>In the example (<a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="inline-onebox" rel="noopener nofollow ugc">Reinforcement Learning (DQN) Tutorial — PyTorch Tutorials 2.1.1+cu121 documentation</a>) you can see, that there are 2 tensors going into the loss function:</p>
<pre><code class="lang-auto">def optimize_model():

    # [parts removed]

    # Compute Q(s_t, a) - the model computes Q(s_t), then we select the
    # columns of actions taken. These are the actions which would've been taken
    # for each batch state according to policy_net
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # Compute V(s_{t+1}) for all next states.
    # Expected values of actions for non_final_next_states are computed based
    # on the "older" target_net; selecting their best reward with max(1)[0].
    # This is merged based on the mask, such that we'll have either the expected
    # state value or 0 in case the state was final.
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # Compute the expected Q values
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Compute Huber loss
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
</code></pre>
<p><strong>state_action_values</strong> are created with gather, resulting in a grad_fn=SqueezeBackward1, which keeps track of all operations.</p>
<p><strong>next_state_values</strong> is detached(), <strong>expected_state_action_values</strong> is not.</p>
<p><strong>(Q1)</strong> Which tensors have to be included, which not and why?</p>
<p><strong>(Q2)</strong> Does it make sense to detach() <strong>expected_state_action_values</strong>, why or why not?</p>
<p>And another example for Q3:</p>
<pre><code class="lang-auto">q_values = self.DQN.predict(self.c_states).gather(1, self.actions.unsqueeze(1)).squeeze(1)

dqn_next = self.DQN.predict(self.n_states)
q_action = torch.argmax(dqn_next, dim=1)
tar_next = self.TAR.predict(self.n_states).detach().gather(1, q_action.unsqueeze(1)).squeeze(1)

q_target = self.rewards + (self.GAMMA * tar_next * self.dones)

loss = self.DQN.fit(current_q=q_values, expected_q=q_target)

    def fit(self, current_q, expected_q):
        loss = self.loss_func(current_q, expected_q)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return(loss.item())
</code></pre>
<p><strong>(Q3)</strong> Can i detach() <strong>tar_next</strong> or <strong>q_target</strong> and fit(<strong>q_values</strong>, <strong>q_target</strong>) or do i need to remove the detach() here, because the information’s coming from gather (on self.TAR.predict) are needed?</p>
<p>Many thanks!</p>