Running Error: Trying to backward through the graph a second time
Author: lxx909546478
Pull request: no

When I define a net, and pass its output to BART model's decoder, which uses for cross attention. The bart decoder has 12 attention layers. And I will get an Error: Trying to backward through the graph a second time.
The same [question](https://github.com/pyg-team/pytorch_geometric/issues/4259).
Appreciate your help!!!
The net is as below:
```python
class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.linear1 = nn.Linear(in_channels, out_channels)

    def forward(self, x):
        x = self.linear1(x)
        return x
```
I add a cross attention module of bart_decoder and Net outputs, which is modified from Huggingface's library transformers, (transformers/src/models/bart/modeling_bart.py). I feed the Net outputs into bart_decoder, and it will be passed into 12 bart_decoder layers. The Net-decoder cross attention module is after encoder-decoder cross attention and before FFN. I found a [paper](https://arxiv.org/pdf/2003.08612.pdf), the figure in it is similar to my code. I think maybe one Net output is feed to 12 decoder layers, when backpropagation, it will backward through the graph a second time. But encoder-decoder cross attention do the same thing without an error. I am really confused.
This is some of my codes:
```python
self.gat = Net(config.d_model, config.d_model)
......
gat_outputs = self.gat(x)
gat_outputs = gat_outputs.repeat(encoder_outputs[0].size(0), 1, 1)
gat_attention_mask = torch.ones((gat_outputs.size(0), gat_outputs.size(1))).to(device)

decoder_outputs = self.decoder(
    input_ids=decoder_input_ids,
    attention_mask=decoder_attention_mask,
    encoder_hidden_states=encoder_outputs[0],
    encoder_attention_mask=attention_mask,
    head_mask=decoder_head_mask,
    cross_attn_head_mask=cross_attn_head_mask,
    past_key_values=past_key_values,
    inputs_embeds=decoder_inputs_embeds,
    use_cache=use_cache,
    output_attentions=output_attentions,
    output_hidden_states=output_hidden_states,
    return_dict=return_dict,
    gat_outputs=gat_outputs,
    gat_attention_mask=gat_attention_mask
)
......
# decoder contains 12 decoder layers 
layer_outputs = decoder_layer(
      hidden_states,
      attention_mask=attention_mask,
      encoder_hidden_states=encoder_hidden_states,
      encoder_attention_mask=encoder_attention_mask,
      layer_head_mask=(head_mask[idx] if head_mask is not None else None),
      cross_attn_layer_head_mask=(
          cross_attn_head_mask[idx] if cross_attn_head_mask is not None else None
      ),
      past_key_value=past_key_value,
      output_attentions=output_attentions,
      use_cache=use_cache,
      gat_outputs=gat_outputs,
      gat_attention_mask=gat_attention_mask
  )
......
self.gat_attn = BartAttention(
      self.embed_dim,
      num_attn_heads,
      dropout=config.attention_dropout,
      is_decoder=True,
  )

residual = hidden_states
# cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple
gat_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None
hidden_states, gat_attn_weights, gat_attn_present_key_value = self.gat_attn(
    hidden_states=hidden_states,
    key_value_states=gat_outputs,
    attention_mask=gat_attention_mask,
    layer_head_mask=cross_attn_layer_head_mask,
    past_key_value=gat_attn_past_key_value,
    output_attentions=output_attentions,
)
hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)
hidden_states = residual + hidden_states
hidden_states = self.gat_attn_layer_norm(hidden_states)

# add cross-attn to positions 3,4 of present_key_value tuple
present_key_value = present_key_value + gat_attn_present_key_value

```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.