Informer

## [](#overview)Overview

The Informer model was proposed in [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://arxiv.org/abs/2012.07436) by Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.

This method introduces a Probabilistic Attention mechanism to select the “active” queries rather than the “lazy” queries and provides a sparse Transformer thus mitigating the quadratic compute and memory requirements of vanilla attention.

The abstract from the paper is the following:

_Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L logL) in time complexity and memory usage, and has comparable performance on sequences’ dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem._

This model was contributed by [elisim](https://huggingface.co/elisim) and [kashif](https://huggingface.co/kashif). The original code can be found [here](https://github.com/zhouhaoyi/Informer2020).

Tips:

-   Check out the Informer blog-post in HuggingFace blog: [Multivariate Probabilistic Time Series Forecasting with Informer](https://huggingface.co/blog/informer)

## [](#transformers.InformerConfig)InformerConfig

### class transformers.InformerConfig

[](#transformers.InformerConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/informer/configuration_informer.py#L33)

( prediction\_length: typing.Optional\[int\] = Nonecontext\_length: typing.Optional\[int\] = Nonedistribution\_output: str = 'student\_t'loss: str = 'nll'input\_size: int = 1lags\_sequence: typing.List\[int\] = Nonescaling: typing.Union\[str, bool, NoneType\] = 'mean'num\_dynamic\_real\_features: int = 0num\_static\_real\_features: int = 0num\_static\_categorical\_features: int = 0num\_time\_features: int = 0cardinality: typing.Optional\[typing.List\[int\]\] = Noneembedding\_dimension: typing.Optional\[typing.List\[int\]\] = Noned\_model: int = 64encoder\_ffn\_dim: int = 32decoder\_ffn\_dim: int = 32encoder\_attention\_heads: int = 2decoder\_attention\_heads: int = 2encoder\_layers: int = 2decoder\_layers: int = 2is\_encoder\_decoder: bool = Trueactivation\_function: str = 'gelu'dropout: float = 0.05encoder\_layerdrop: float = 0.1decoder\_layerdrop: float = 0.1attention\_dropout: float = 0.1activation\_dropout: float = 0.1num\_parallel\_samples: int = 100init\_std: float = 0.02use\_cache = Trueattention\_type: str = 'prob'sampling\_factor: int = 5distil: bool = True\*\*kwargs )

This is the configuration class to store the configuration of an [InformerModel](/docs/transformers/v4.30.0/en/model_doc/informer#transformers.InformerModel). It is used to instantiate an Informer model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Informer [huggingface/informer-tourism-monthly](https://huggingface.co/huggingface/informer-tourism-monthly) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import InformerConfig, InformerModel

>>> 
>>> configuration = InformerConfig(prediction_length=12)

>>> 
>>> model = InformerModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.InformerModel)InformerModel

### class transformers.InformerModel

[](#transformers.InformerModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/informer/modeling_informer.py#L1503)

( config: InformerConfig )

Parameters

-   [](#transformers.InformerModel.config)**config** ([TimeSeriesTransformerConfig](/docs/transformers/v4.30.0/en/model_doc/time_series_transformer#transformers.TimeSeriesTransformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare Informer Model outputting raw hidden-states without any specific head on top. This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.InformerModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/informer/modeling_informer.py#L1641)

( past\_values: Tensorpast\_time\_features: Tensorpast\_observed\_mask: Tensorstatic\_categorical\_features: typing.Optional\[torch.Tensor\] = Nonestatic\_real\_features: typing.Optional\[torch.Tensor\] = Nonefuture\_values: typing.Optional\[torch.Tensor\] = Nonefuture\_time\_features: typing.Optional\[torch.Tensor\] = Nonedecoder\_attention\_mask: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_head\_mask: typing.Optional\[torch.Tensor\] = Nonecross\_attn\_head\_mask: typing.Optional\[torch.Tensor\] = Noneencoder\_outputs: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Nonepast\_key\_values: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneuse\_cache: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.Seq2SeqTSModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.Seq2SeqTSModelOutput) or `tuple(torch.FloatTensor)`

The [InformerModel](/docs/transformers/v4.30.0/en/model_doc/informer#transformers.InformerModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import InformerModel

>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)

>>> model = InformerModel.from_pretrained("huggingface/informer-tourism-monthly")

>>> 
>>> 
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )

>>> last_hidden_state = outputs.last_hidden_state```

## [](#transformers.InformerForPrediction)InformerForPrediction

### class transformers.InformerForPrediction

[](#transformers.InformerForPrediction)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/informer/modeling_informer.py#L1765)

( config: InformerConfig )

Parameters

-   [](#transformers.InformerForPrediction.config)**config** ([TimeSeriesTransformerConfig](/docs/transformers/v4.30.0/en/model_doc/time_series_transformer#transformers.TimeSeriesTransformerConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The Informer Model with a distribution head on top for time-series forecasting. This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.InformerForPrediction.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/informer/modeling_informer.py#L1805)

( past\_values: Tensorpast\_time\_features: Tensorpast\_observed\_mask: Tensorstatic\_categorical\_features: typing.Optional\[torch.Tensor\] = Nonestatic\_real\_features: typing.Optional\[torch.Tensor\] = Nonefuture\_values: typing.Optional\[torch.Tensor\] = Nonefuture\_time\_features: typing.Optional\[torch.Tensor\] = Nonefuture\_observed\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_attention\_mask: typing.Optional\[torch.LongTensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Nonedecoder\_head\_mask: typing.Optional\[torch.Tensor\] = Nonecross\_attn\_head\_mask: typing.Optional\[torch.Tensor\] = Noneencoder\_outputs: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Nonepast\_key\_values: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneuse\_cache: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.Seq2SeqTSModelOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.Seq2SeqTSModelOutput) or `tuple(torch.FloatTensor)`

The [InformerForPrediction](/docs/transformers/v4.30.0/en/model_doc/informer#transformers.InformerForPrediction) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from huggingface_hub import hf_hub_download
>>> import torch
>>> from transformers import InformerForPrediction

>>> file = hf_hub_download(
...     repo_id="hf-internal-testing/tourism-monthly-batch", filename="train-batch.pt", repo_type="dataset"
... )
>>> batch = torch.load(file)

>>> model = InformerForPrediction.from_pretrained("huggingface/informer-tourism-monthly")

>>> 
>>> 
>>> outputs = model(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_values=batch["future_values"],
...     future_time_features=batch["future_time_features"],
... )

>>> loss = outputs.loss
>>> loss.backward()

>>> 
>>> 
>>> 
>>> outputs = model.generate(
...     past_values=batch["past_values"],
...     past_time_features=batch["past_time_features"],
...     past_observed_mask=batch["past_observed_mask"],
...     static_categorical_features=batch["static_categorical_features"],
...     static_real_features=batch["static_real_features"],
...     future_time_features=batch["future_time_features"],
... )

>>> mean_prediction = outputs.sequences.mean(dim=1)```