[gpt‑oss] eager_attention_forward not using sliding-window attention for GPT‑OSS models
Author: AlfredTino
Pull request: no

In the latest transformers version v4.55.0, the GPT‑OSS model’s eager_attention_forward implementation does **not** use sliding‑window attention. This behavior diverges from the original GPT‑OSS specification, where alternating full‑context and sliding‑window attention (e.g. window size 128) is a key architectural feature—such as in the vLLM and model card descriptions that note gpt‑oss uses intermittent sliding‑window patterns for long‑context efficiency (alternating local and global attention).

Although eager attention backend may be rarely used in production, it should remain consistent with other attention backends (e.g. FlashAttention) and the original GPT‑OSS behavior. 


Comments:
I think it is using alternating full and sliding-window attention. Each layer has a config associated with the type of attention it should be using i.e. `sliding_window` or `full_attention`

This config is set here:

https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/configuration_gpt_oss.py#L94-L98

Applied to each decoder layer here:

https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L331-L335

A causal mask is created based on this config here:

https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L474-L500

The causal mask is then applied to get the right attention weights here:

https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L239-L242

Am I missing something?

> I think it is using alternating full and sliding-window attention. Each layer has a config associated with the type of attention it should be using i.e. `sliding_window` or `full_attention`
> 
> This config is set here:
> 
> [transformers/src/transformers/models/gpt_oss/configuration_gpt_oss.py](https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/configuration_gpt_oss.py#L94-L98)
> 
> Lines 94 to 98 in [43b3f58](/huggingface/transformers/commit/43b3f58875e93ddaa66ac0610cb6da3a6fc37768)
> 
>  self.layer_types = layer_types 
>  if self.layer_types is None: 
>      self.layer_types = [ 
>          "sliding_attention" if bool((i + 1) % 2) else "full_attention" for i in range(self.num_hidden_layers) 
>      ] 
> Applied to each decoder layer here:
> 
> [transformers/src/transformers/models/gpt_oss/modeling_gpt_oss.py](https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L331-L335)
> 
> Lines 331 to 335 in [43b3f58](/huggingface/transformers/commit/43b3f58875e93ddaa66ac0610cb6da3a6fc37768)
> 
>  class GptOssDecoderLayer(GradientCheckpointingLayer): 
>      def __init__(self, config: GptOssConfig, layer_idx: int): 
>          super().__init__() 
>          self.hidden_size = config.hidden_size 
>          self.self_attn = GptOssAttention(config=config, layer_idx=layer_idx) 
> A causal mask is created based on this config here:
> 
> [transformers/src/transformers/models/gpt_oss/modeling_gpt_oss.py](https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L474-L500)
> 
> Lines 474 to 500 in [43b3f58](/huggingface/transformers/commit/43b3f58875e93ddaa66ac0610cb6da3a6fc37768)
> 
>  if not isinstance(causal_mask_mapping := attention_mask, dict): 
>      mask_kwargs = { 
>          "config": self.config, 
>          "input_embeds": inputs_embeds, 
>          "attention_mask": attention_mask, 
>          "cache_position": cache_position, 
>          "past_key_values": past_key_values, 
>      } 
>      causal_mask_mapping = { 
>          "full_attention": create_causal_mask(**mask_kwargs), 
>          "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs), 
>      } 
>   
>  hidden_states = inputs_embeds 
>  position_embeddings = self.rotary_emb(hidden_states, position_ids) 
>   
>  for decoder_layer in self.layers: 
>      hidden_states = decoder_layer( 
>          hidden_states, 
>          attention_mask=causal_mask_mapping[decoder_layer.attention_type], 
>          position_ids=position_ids, 
>          past_key_value=past_key_values, 
>          use_cache=use_cache, 
>          cache_position=cache_position, 
>          position_embeddings=position_embeddings, 
>          **kwargs, 
>      ) 
> The causal mask is then applied to get the right attention weights here:
> 
> [transformers/src/transformers/models/gpt_oss/modeling_gpt_oss.py](https://github.com/huggingface/transformers/blob/43b3f58875e93ddaa66ac0610cb6da3a6fc37768/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L239-L242)
> 
> Lines 239 to 242 in [43b3f58](/huggingface/transformers/commit/43b3f58875e93ddaa66ac0610cb6da3a6fc37768)
> 
>  attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling 
>  if attention_mask is not None: 
>      causal_mask = attention_mask[:, :, :, : key_states.shape[-2]] 
>      attn_weights = attn_weights + causal_mask 
> Am I missing something?

You're absolutely right, bro. I didn't realize that the sliding window was already incorporated during the mask construction. Respect!