[Feat] Need to have an API to inject attention during inference
Author: alexanderliu-creator
Pull request: no
Labels: name: Feature request

### Feature request

# Requirenments
- In the inference phase of deep learning models, there are times when we need to change the model's behavior to better adapt to specific tasks or data. One potential method is to dynamically inject attention masks during the inference process, which is a mechanism that indicates which parts of the input data the model should focus on. Such an interface would allow us to selectively guide the model's focus without modifying the fundamental structure of the model or retraining it.
- Specifically, this interface should enable us to provide attention masks dynamically during model inference based on the needs of the current task. For example, when dealing with natural language processing (NLP) tasks, we might want the model to focus more on certain keywords or phrases rather than the entire input text. By injecting attention masks, we can direct the model to concentrate its resources on these important parts, potentially improving the accuracy and efficiency of inference.

### Motivation

The proposal to introduce dynamic attention masking during the inference phase addresses this challenge by allowing for a more directed approach to how models process input data. This feature would enable users to guide the model's focus to specific parts of the input that are deemed more relevant to the task at hand. Here’s how this feature would be beneficial:

Enhanced Relevance: By directing the model's attention to specific keywords or data points, we can ensure that the model's processing power is concentrated on the most relevant information, potentially leading to more accurate inferences.

Efficiency Improvements: Focusing on relevant parts of the input can reduce the computational load, as the model does not need to process the entire input uniformly, thus improving inference speed and efficiency.

Adaptability: This feature would allow models to be more adaptable to various tasks without the need for retraining, making them more versatile and cost-effective in environments where multiple tasks are performed.

Customization: Users could customize the model's behavior for specific tasks, leading to better performance tailored to their unique requirements.

Interpretability: Dynamic attention masks can also improve the interpretability of model decisions, as users can understand which parts of the input data the model is focusing on.

### Your contribution

No, actually I'm still researching on the repo, I'm not familiar with it. I wonder whether there's already a way to support this feature. If there's already this function, sorry to bother you.

Comments:
I don't think we currently support this! Although you can pass attention masks to the model input, the user must specify the mask, and the same mask is used in all layers. Dynamic masks would need to be added to each model's code individually. Remember that most models have not been trained with partially-masked input sequences like this, and so if you "blank out" certain tokens in their history suddenly, you'll probably cause them to become unstable and emit garbage, rather than focusing their attention on the remaining tokens. I suspect this would interact very awkwardly with the cache system as well!

In general the Transformers codebase focuses on features with proven utility on real tasks, so I don't think we could support a PR for this right now. That said, if you get great results with this, we'd be happy to revisit this decision then - but only **after** the paper or codebase showing strong empirical results, not before!