Infomer fails with LAG=0
Author: antonio-navarra
Pull request: no

### System Info

Transformer 4.32.1, Anaconda, MacOS

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The Infomer for prediction fails if LAGS are set to [0]. The model converge nicely, but fails in the generating phase:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[18], line 5
      1 #Creates Plots
      3 model.load_state_dict(torch.load(file))
----> 5 out_test = predict(model, test_dataloader,Tpredict, device=torch.device('cpu')).mean(axis=1)
      6 out_val = predict(model, val_dataloader,Tpredict,device=torch.device('cpu')).mean(axis=1)
      7 out_train = predict(model, train_dataloader,Tpredict,device=torch.device('cpu')).mean(axis=1)

Cell In[17], line 18, in predict(model, val_loader, Tpredict, device)
     12 pasobs = torch.ones([batch_size,TIN,MIN],dtype=torch.float32, device=device)
     14 # during inference, one only provides past values
     15 # as well as possible additional features
     16 # the model autoregressively generates future values
---> 18 output = model.generate(
     19 past_values=src,
     20 past_time_features=pasft,
     21 past_observed_mask=pasobs,
     22 future_time_features=futft)
     25 if i == 0 :
     26     temp = output['sequences']

File /opt/anaconda3/envs/AI/lib/python3.9/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /opt/anaconda3/envs/AI/lib/python3.9/site-packages/transformers/models/informer/modeling_informer.py:2089, in InformerForPrediction.generate(self, past_values, past_time_features, future_time_features, past_observed_mask, static_categorical_features, static_real_features, output_attentions, output_hidden_states)
   2086 lags_shape = lagged_sequence.shape
   2087 reshaped_lagged_sequence = lagged_sequence.reshape(lags_shape[0], lags_shape[1], -1)
-> 2089 decoder_input = torch.cat((reshaped_lagged_sequence, repeated_features[:, : k + 1]), dim=-1)
   2091 dec_output = decoder(inputs_embeds=decoder_input, encoder_hidden_states=repeated_enc_last_hidden)
   2092 dec_last_hidden = dec_output.last_hidden_state

RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 30 but got size 1 for tensor number 1 in the list.

I think I traced the problem to the hardwired 'shift=1' into `generate`

### Expected behavior

I expect the generation to go forward  also for LAGS=0

Comments:
Hi @antonio-navarra, thanks for raising this issue! 

Could you provide a minimal code example we can run to reproduce the issue? Without that we'll be unable to help. 

cc @gante 

@amyeroberts FYI time-series models often have their own stand-alone `generate` methods (e.g. [this one for Informer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/informer/modeling_informer.py#L1879)) :)

I'm passing the cc along to our time-series expert, @kashif 

yes currently there is a need for 1 time step in the enc-dec setting to kick-start the generation process... and thus edge cases happen when this is not the case etc. Thus i think it might be useful to have at least a lag=[1] and thus at inference time a context windows of 2 time steps... 

can you confirm it works with `[1]` @antonio-navarra ?

@Kashif,
Yes with any lags >0 works. In my case I use it to make forecasts of atmospheric data sets LAGS=[1] with an input sequence of 30 time steps gives good results.

Probably, LAGS=0 is a little bit contrived, but then it would be useful to have  a check and mention it in the docs.

—Antonio

> On 2 Nov 2023, at 17:46, Kashif Rasul ***@***.***> wrote:
> 
> 
> yes currently there is a need for 1 time step in the enc-dec setting to kick-start the generation process... and thus edge cases happen when this is not the case etc. Thus i think it might be useful to have at least a lag=[1] and thus at inference time a context windows of 2 time steps...
> 
> can you confirm it works with [1] @antonio-navarra <https://github.com/antonio-navarra> ?
> 
> —
> Reply to this email directly, view it on GitHub <https://github.com/huggingface/transformers/issues/27237#issuecomment-1791100530>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APQWRZ2UQB4CO4AH46EC6SLYCPE6PAVCNFSM6AAAAAA62VRJMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJRGEYDANJTGA>.
> You are receiving this because you were mentioned.
> 



ok good to know... yes i'll see how to make a check for zero or even negative lags... thanks for the heads up!

#27237  Solved ✅ (https://github.com/huggingface/transformers/pull/28766)

I suggest a resolution for situations where sequence_lags=[0]. In such instances, the length of the context past sequence aligns with the overall sequence length. Let me provide some clarification on this matter.

The issue stems from line `1230` in `modeling_time_series_transformer.py.` In cases where the lag parameter is set to 0, the index is assigned a value of -1. This results in only one data point being lagged, creating a discrepancy when using model.generate(). For example, if the size is 48, selecting a lag of [0] produces 49, rendering it unsuitable for model.generate().

```
sequence_length = sequence.shape[1]
indices = [lag - shift for lag in self.config.lags_sequence]
if max(indices) + subsequences_length > sequence_length:
    raise ValueError(
        f"lags cannot go further than history length, found lag {max(indices)} "
        f"while history length is only {sequence_length}"
    )
```
We can modify the code as shown below to rectify index 0 when negative values are encountered ✅:
```
sequence_length = sequence.shape[1]
# (Khalid Oublal) -> addressed the issue regarding the scenario where lag equals 0.
# The previous implementation was: indices = [lag - shift for lag in self.config.lags_sequence]
indices = [lag - shift if lag > 0 else 0 for lag in self.config.lags_sequence]
if max(indices) + subsequences_length > sequence_length:
    raise ValueError(
        f"lags cannot go further than history length, found lag {max(indices)} "
        f"while history length is only {sequence_length}"
    )
```
### Check DataLoader

In the analysis below, it's evident that there are no lags indicated by `sequence_lags=[0]`. The length of the context in this batch matches the provided context length.

![Screenshot 2024-01-29 at 21 58 42](https://github.com/huggingface/transformers/assets/76509145/0079c3a8-eed6-4d82-bb41-fc6dcaf04a5a)

### Confirming Training Status

Below, it's apparent that the training is progressing smoothly. Some additional print statements were added to verify that the lags are 0, implying the indices should be `[0]`.

![Screenshot 2024-01-29 at 21 54 03](https://github.com/huggingface/transformers/assets/76509145/f159c725-ff81-4a11-89e0-5ef59ac0763e)

### Generating with `model.generate()`

Now, with `sequence_lags=[0]`, we observe that predictions can be made without any issues.

![Screenshot 2024-01-29 at 21 55 31](https://github.com/huggingface/transformers/assets/76509145/dfe7915b-8859-41ad-9664-f72f6c4754c7)

Best,
khalid oublal

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.