Using BigBirdBlockSparseAttention in Decoder
Author: TamirCohen
Pull request: no
Labels: name: Model on the Hub

### Feature request

I want to use `BigBirdBlockSparseAttention` In a decoder, 
But I noticed that it says in the code that it currently not works for decoders:
https://github.com/huggingface/transformers/blob/a6e6b1c622d8d08e2510a82cb6266d7b654f1cbf/src/transformers/models/big_bird/modeling_big_bird.py#L456C1-L457C1
This is also prevented by The following:https://github.com/huggingface/transformers/blob/a6e6b1c622d8d08e2510a82cb6266d7b654f1cbf/src/transformers/models/big_bird/modeling_big_bird.py#L1404C1-L1404C1.

Currently I am not really sure why it cant be used as a decoder.
In the paper https://arxiv.org/pdf/2007.14062.pdf it seems that the sparse attention can be used in a decoder.

Thanks :)

### Motivation

Currently the `block_sparse` attention of bigbird cant be used in a decoder, and I want to use it to make my model more efficient.

### Your contribution

I would love to try send a PR if you think it can be done according to the paper.

Comments:
Hi @TamirCohen, thanks for opening this feature request! 

I don't know the exact reason why it's not enabled for the decoder cc @thevasudevgupta who added the model. In the BigBird paper, they mention for encoder-decoder tasks: 

```
For an encoder-decoder setup, one can easily see that both suffer from quadratic complexity due to
the full self attention. We focus on introducing the sparse attention mechanism of BIGBIRD only at
the encoder side. This is because, in practical generative applications, the length of output sequence
is typically small as compared to the input. For example for text summarization, we see in realistic
scenarios (c.f. App. E.5 Tab. 18) that the median output sequence length is ∼ 200 where as the input sequence’s median length is > 3000. For such applications, it is more efficient to use sparse attention
mechanism for the encoder and full self-attention for the decoder.
```

so I suspect it's a pragmatic choice. 

I'll let @ArthurZucker comment on whether this is something we'd want to add to this model. The layer would need to be made to be compatible with cross-attention. 

From reading the paper it appears that `We use full attention in decoder`. (as @amyeroberts pointed out) so that's why it was not added originally.
![image](https://github.com/huggingface/transformers/assets/48595927/cb9e0b09-3dae-45af-b873-0ea82e626651)
![image](https://github.com/huggingface/transformers/assets/48595927/70a79696-a9af-4f55-8447-3b294522f754)

Could be good to have sparse attention for decoder models, but this would be in a new model and would suggest you to start with adding this `BigBirdSparseDecoder` [on the hub](https://huggingface.co/docs/transformers/custom_models) ! Will help us determine whether there is community interest for it or not! 


Thanks for the reply!
I will add it to the hub soon.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.