An error when creating test_dataloader in Time series transformer
Author: kkckk1110
Pull request: no

### System Info

I am running a time series transformer following the tutorial in Huggingface. 
I have dynamic_features_real = 48 in my dataset. However, I came across an error when creating the test_dataloader:
```python
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 75 and the array at index 2 has size 67.
``
some settings in my codes are:
```python
len(train_example['target']) = 59
len(validation_example['target']) = 67
np.array(validation_example['feat_dynamic_real']).shape  = (48, 67)
prediction_length = 8
```
I think the problem is with the feat_dynamic_real, because when I set it = 0, the codes run normally. However, I tried but fail to solve the problem.
Can anyone help me fix the problem? Thanks a lot!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
```python
batch = next(iter(test_dataloader))
for k, v in batch.items():
    print(k, v.shape, v.type())
```
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 75 and the array at index 2 has size 67

### Expected behavior

I hope to fix the problem.

Comments:
Hey! Could you make sure you provide a reproducer, isolating the bug? We can't really debug your code in your stead 🤗 

Thanks to your quick reply! Here are my detailed codes related to the bug.

```python
from transformers import TimeSeriesTransformerConfig, TimeSeriesTransformerForPrediction

config = TimeSeriesTransformerConfig(
    prediction_length = prediction_length,
    context_length = prediction_length,
    lags_sequence = lags_sequence,
    num_time_features = len(time_features) + 1,
    num_static_categorical_features = 1,
    num_dynamic_real_features = 48,
    
    cardinality = [len(train_dataset)],
    embedding_dimension = [2],

    # transformer params:
    encoder_layers = 4,
    decoder_layers = 4,
    d_model = 32,
)

model = TimeSeriesTransformerForPrediction(config)

def create_test_dataloader(
    config: PretrainedConfig,
    freq,
    data,
    batch_size: int,
    **kwargs,
):
    PREDICTION_INPUT_NAMES = [
        "past_time_features",
        "past_values", 
        "past_observed_mask",
        "future_time_features",
    ]
    if config.num_static_categorical_features > 0:
        PREDICTION_INPUT_NAMES.append("static_categorical_features")

    if config.num_static_real_features > 0:
        PREDICTION_INPUT_NAMES.append("static_real_features")
        
    transformation = create_transformation(freq, config)
    transformed_data = transformation.apply(data, is_train=False) #!!!

    # we create a Test Instance splitter which will sample the very last
    # context window seen during training only for the encoder.
    instance_sampler = create_instance_splitter(config, "test")

    # we apply the transformations in test mode
    testing_instances = instance_sampler.apply(transformed_data, is_train=False)

    return as_stacked_batches(
        testing_instances,
        batch_size=batch_size,
        output_type=torch.tensor,
        field_names=PREDICTION_INPUT_NAMES,
    )
```

Actually, most of the codes are identical with those in https://huggingface.co/blog/time-series-transformers, except that I set num_dynamic_real_features = 48 in the Config

Yep, but this changes the value of `self._number_of_features`, which affect `self.feature_size` in return. The `past_time_features` given to the model need to be updated as well. I'm betting the issue is with the dataset rather than anything. The `feat_dynamic_real`'s shape is expected to be `2,75` apparently. I'm not super familiar with this model but if you can share the dataset you are using as well it would help 🤗 

Thank you very much! I will provide more details about my dataset. I am working on my own dataset, using the following codes.
```python
from gluonts.dataset.pandas import PandasDataset

prediction_length = 8
train = PandasDataset.from_long_dataframe(train, target="sales", item_id="city",
                        feat_dynamic_real = [col for col in train.columns if ((col != 'sales') & (col != 'city'))])
test = PandasDataset.from_long_dataframe(test, target="sales", item_id="city",
                        feat_dynamic_real = [col for col in test.columns if ((col != 'sales') & (col != 'city'))])

class ProcessStartField():
    ts_id = 0
    def __call__(self, data):
        data["start"] = data["start"].to_timestamp()
        data["feat_static_cat"] = [self.ts_id]
        dynamic_feature_values = np.array(data["feat_dynamic_real"])
        data["feat_dynamic_real"] = dynamic_feature_values
        self.ts_id += 1
        return data

from gluonts.itertools import Map
process_start = ProcessStartField()
list_train = list(Map(process_start, train))
process_start = ProcessStartField()
list_test = list(Map(process_start, test))

trainset = Dataset.from_list(list_train) #, features=features
testset = Dataset.from_list(list_test)
```
And my dataset looks like:
```
train_example: {'start': datetime.datetime(2017, 1, 1, 0, 0),
 'target': [806, 806...] (a list with length of **59**)
'item_id': cityname
'feat_dynamic_real': [[215,219...],[217,192...]...] (a list comprising of _48_ sublists, each sublist has length of **59**)}
'feat_static_cat': [0]}
```
The corresponding test_example is :
```
 {'start': datetime.datetime(2017, 1, 1, 0, 0),
 'target': [806, 806...] (a list with length of **67**)
'item_id': cityname
'feat_dynamic_real': [[215,219...],[217,192...]...] (a list comprising of _48_ sublists, each sublist has length of **67**)}
'feat_static_cat': [0]}
```
I also tried other number of context length, and I found that, the codes always transformed the testdata to the length of 67+context_length, which in turn incurred an error. 
I am really confused about the bug for a long time. I would really appreciate it if you can help! Thanks!

cc @NielsRogge if you have any idea?!

cc @kashif, the master of time series

@kkckk1110 so it could be that the arrays `feat_dynamic_real` in your dataset have to be transposed... let me double-check but you can quickly try it if that helps.

Thank you very much for your attention! I have already tried but came across errors though.

can you paste a bigger error trace? is it from the `VStack` transformation?


Exactly! the complete error information is as follows.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[51], line 1
----> 1 batch = next(iter(test_dataloader))
      2 for k, v in batch.items():
      3     print(k, v.shape, v.type())

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/itertools.py:415, in IterableSlice.__iter__(self)
    414 def __iter__(self):
--> 415     yield from itertools.islice(self.iterable, self.length)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:111, in TransformedDataset.__iter__(self)
    110 def __iter__(self) -> Iterator[DataEntry]:
--> 111     yield from self.transformation(
    112         self.base_dataset, is_train=self.is_train
    113     )

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:132, in MapTransformation.__call__(self, data_it, is_train)
    129 def __call__(
    130     self, data_it: Iterable[DataEntry], is_train: bool
    131 ) -> Iterator:
--> 132     for data_entry in data_it:
    133         try:
    134             yield self.map_transform(data_entry.copy(), is_train)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/dataset/loader.py:55, in Stack.__call__(self, data, is_train)
     54 def __call__(self, data, is_train):
---> 55     for batch in data:
     56         yield rows_to_columns(batch, np.array)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/dataset/loader.py:50, in Batch.__call__(self, data, is_train)
     49 def __call__(self, data, is_train):
---> 50     yield from batcher(data, self.batch_size)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/itertools.py:131, in batcher.<locals>.get_batch()
    130 def get_batch():
--> 131     return list(itertools.islice(it, batch_size))

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:132, in MapTransformation.__call__(self, data_it, is_train)
    129 def __call__(
    130     self, data_it: Iterable[DataEntry], is_train: bool
    131 ) -> Iterator:
--> 132     for data_entry in data_it:
    133         try:
    134             yield self.map_transform(data_entry.copy(), is_train)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:111, in TransformedDataset.__iter__(self)
    110 def __iter__(self) -> Iterator[DataEntry]:
--> 111     yield from self.transformation(
    112         self.base_dataset, is_train=self.is_train
    113     )

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:186, in FlatMapTransformation.__call__(self, data_it, is_train)
    182 def __call__(
    183     self, data_it: Iterable[DataEntry], is_train: bool
    184 ) -> Iterator:
    185     num_idle_transforms = 0
--> 186     for data_entry in data_it:
    187         num_idle_transforms += 1
    188         for result in self.flatmap_transform(data_entry.copy(), is_train):

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:111, in TransformedDataset.__iter__(self)
    110 def __iter__(self) -> Iterator[DataEntry]:
--> 111     yield from self.transformation(
    112         self.base_dataset, is_train=self.is_train
    113     )

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:132, in MapTransformation.__call__(self, data_it, is_train)
    129 def __call__(
    130     self, data_it: Iterable[DataEntry], is_train: bool
    131 ) -> Iterator:
--> 132     for data_entry in data_it:
    133         try:
    134             yield self.map_transform(data_entry.copy(), is_train)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:136, in MapTransformation.__call__(self, data_it, is_train)
    134     yield self.map_transform(data_entry.copy(), is_train)
    135 except Exception as e:
--> 136     raise e

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:134, in MapTransformation.__call__(self, data_it, is_train)
    132 for data_entry in data_it:
    133     try:
--> 134         yield self.map_transform(data_entry.copy(), is_train)
    135     except Exception as e:
    136         raise e

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/_base.py:149, in SimpleTransformation.map_transform(self, data, is_train)
    148 def map_transform(self, data: DataEntry, is_train: bool) -> DataEntry:
--> 149     return self.transform(data)

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/gluonts/transform/convert.py:219, in VstackFeatures.transform(self, data)
    213 def transform(self, data: DataEntry) -> DataEntry:
    214     r = [
    215         data[fname]
    216         for fname in self.input_fields
    217         if data[fname] is not None
    218     ]
--> 219     output = np.vstack(r) if not self.h_stack else np.hstack(r)
    220     data[self.output_field] = output
    221     for fname in self.cols_to_drop:

File ~/miniforge3/envs/tensorflow/lib/python3.9/site-packages/numpy/core/shape_base.py:289, in vstack(tup, dtype, casting)
    287 if not isinstance(arrs, list):
    288     arrs = [arrs]
--> 289 return _nx.concatenate(arrs, 0, dtype=dtype, casting=casting)

ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 75 and the array at index 2 has size 67

so yes the shape of the `data["feat_dynamic_real"].shape` has to be (`num_dynamic_real_features`, `time`), can you confirm that is the case in your dataset?


Sure! 
np.array(train_example['feat_dynamic_real']).shape : (48, 59)
np.array(validation_example['feat_dynamic_real']).shape : (48, 67)
which are expected and consistent with (num_dynamic_real_features, time)

@kkckk1110  ok I figured it out, my bad, i'll update the blog post: In the mean time use this for the back-testing:

```python
 def create_backtest_dataloader(
    config: PretrainedConfig,
    freq,
    data,
    batch_size: int,
    **kwargs,
):
    PREDICTION_INPUT_NAMES = [
        "past_time_features",
        "past_values",
        "past_observed_mask",
        "future_time_features",
    ]
    if config.num_static_categorical_features > 0:
        PREDICTION_INPUT_NAMES.append("static_categorical_features")

    if config.num_static_real_features > 0:
        PREDICTION_INPUT_NAMES.append("static_real_features")

    transformation = create_transformation(freq, config)
    transformed_data = transformation.apply(data, is_train=True)

    # we create a Test Instance splitter which will sample the very last
    # context window seen during training only for the encoder.
    instance_sampler = create_instance_splitter(config, "validation")

    # we apply the transformations in train mode
    backtesting_instances = instance_sampler.apply(transformed_data, is_train=True)
    
    return as_stacked_batches(
        backtesting_instances,
        batch_size=batch_size,
        output_type=torch.tensor,
        field_names=PREDICTION_INPUT_NAMES,
    )
```   

It works! Great! Can you elaborate on why the modified codes solved the problems?

yes the splitter for back-testing still needs to be applied in "Training"

```
 backtesting_instances = instance_sampler.apply(transformed_data, is_train=True)
 ```
 
 else the dynamic features are not split while the target is... I'll check with the gluonts folks 

I really appreciate your efforts! I hate to bother you but I have another problem. I wonder how can I interpret time series models, such as analyzing feature importance? Are there any tools like SHAP, for time series forecasting?

there is none at the moment for transformer-based models as far as i know... one does have the attention weights so can figure out which time points were used to make a particular prediction