`Wav2Vec2FeatureExtractor` and `Speech2TextFeatureExtractor` tests fail for truncating `max_length` values
Author: cwkeam
Pull request: no

# Misleading use of `max_length` in `FeatureExtraction` classes

**TL;DR**

- Currently, setting `max_length` and having features be longer than `max_length` does not necessarily mean it should be `truncation=True`. 
- Further look into testing code makes me believe that this is by design, as explained [below](https://github.com/huggingface/transformers/issues/16419#issuecomment-1079705990).
- I believe this is confusing because padding to `length==X` for shorter sequences and not truncating sequences greater than `X` is what you'd expect if `X` is `min_length`, not `max_length`.


---

The current tests for the existing `FeatureExtractor` classes are using somewhat meaningless `max_length` values that are greater than length of the input features. 

**When using lower values that truncate the input features (for a meaningful effect of `max_length` parameter), the test code fails.**

## Environment info

- `transformers` version: 4.18.0.dev0
- Platform: Linux-3.10.0-1062.el7.x86_64-x86_64-with-debian-10.1
- Python version: 3.7.9
- Huggingface_hub version: 0.2.0
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.1
- JaxLib version: 0.3.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@patrickvonplaten, @anton-l


## Information

Model I am using (Bert, XLNet ...): `Wav2Vec2FeatureExtractor` and `Speech2TextFeatureExtractor`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

### Existing Working Test 1

From `tests/wav2vec2/test_feature_extraction_wav2vec2.py` line 127:
```python
def test_zero_mean_unit_variance_normalization_np(self):
    feat_extract = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]
    max_lengths = [None, 1600, None]
    for max_length, padding in zip(max_lengths, paddings):
        processed = feat_extract(speech_inputs, padding=padding, max_length=max_length, return_tensors="np")
        input_values = processed.input_values

        self._check_zero_mean_unit_variance(input_values[0][:800])
        self.assertTrue(input_values[0][800:].sum() < 1e-6)
        self._check_zero_mean_unit_variance(input_values[1][:1000])
        self.assertTrue(input_values[0][1000:].sum() < 1e-6)
        self._check_zero_mean_unit_variance(input_values[2][:1200])
```


### Failing Test 1 (after changing `max_lengths`)

```python
def test_zero_mean_unit_variance_normalization_np(self):
    feat_extract = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]

    max_lengths = [None, 100, None]

    for max_length, padding in zip(max_lengths, paddings):
        processed = feat_extract(speech_inputs, padding=padding, max_length=max_length, return_tensors="np")
        ....
```

### Existing working test 2

From `tests/speech_to_text/test_feature_extraction_speech_to_text.py` Line 140
```python
def test_cepstral_mean_and_variance_normalization(self):
    feature_extractor = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]
    max_lengths = [None, 16, None]
    for max_length, padding in zip(max_lengths, paddings):
        inputs = feature_extractor(
            speech_inputs, padding=padding, max_length=max_length, return_attention_mask=True
        )
        input_features = inputs.input_features
        attention_mask = inputs.attention_mask
        fbank_feat_lengths = [np.sum(x) for x in attention_mask]

        self._check_zero_mean_unit_variance(input_features[0][: fbank_feat_lengths[0]])
        self._check_zero_mean_unit_variance(input_features[1][: fbank_feat_lengths[1]])
        self._check_zero_mean_unit_variance(input_features[2][: fbank_feat_lengths[2]])
```



### Failing test 2

```python
def test_cepstral_mean_and_variance_normalization(self):
    feature_extractor = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]

    max_lengths = [None, 4, None]

    for max_length, padding in zip(max_lengths, paddings):
        inputs = feature_extractor(
            speech_inputs, padding=padding, max_length=max_length, return_attention_mask=True
        )
        input_features = inputs.input_features
        ....
```
 
For identical error messages:

```
        if self.do_ceptral_normalize:
            attention_mask = (
                np.array(attention_mask, dtype=np.int32)
>               if self._get_padding_strategies(padding, max_length=max_length) is not PaddingStrategy.DO_NOT_PAD
                else None
            )
E           ValueError: setting an array element with a sequence.
```


## Why It Fails

The working tests produce `attention_mask` values like this:
```
[[1 1 1 0 0 0]
 [1 1 1 1 0 0]
 [1 1 1 1 1 1]]
```

While the failing tests produce `attention_mask` values **at the same point in the code** like this:

```
[array([1, 1, 1, 0], dtype=int32) array([1, 1, 1, 1], dtype=int32)
 array([1, 1, 1, 1, 1, 1], dtype=int32)]
```

A very easy fix would just be doing type checking and changing the latter type to the same as the former.

But maybe it's worth looking into `SequenceFeatureExtractor`'s `pad` function that returns `List[np.array]` for truncated outputs and `List[List]]` for untruncated outputs.




<!-- A clear and concise description of what you would expect to happen. -->


Comments:
So despite the `max_length` parameter, it's not actually truncating (rather, it's forgetting to for the last item):

```
[
array([1, 1, 1, 0], dtype=int32) 
array([1, 1, 1, 1], dtype=int32)
array([1, 1, 1, 1, 1, 1], dtype=int32)
]
```

The above is a list of "ragged" nested sequences, and turns out:

```
feature_extraction_utils.py:172: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a 
list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must 
specify 'dtype=object' when creating the ndarray.
```

Okay, so it seems like this behavior is by design:
https://github.com/huggingface/transformers/blob/main/tests/test_sequence_feature_extraction_common.py#L244

```python
# truncate to smallest
input_1 = feat_extract.pad(
    processed_features, padding="max_length", max_length=len(speech_inputs[0]), truncation=True
)
input_1 = input_1[input_name]

input_2 = feat_extract.pad(processed_features, padding="max_length", max_length=len(speech_inputs[0]))
input_2 = input_2[input_name]

self.assertTrue(_inputs_have_equal_length(input_1))

# suggests we don't want equal lengths, despite max_length, since we didn't set `truncation=True`
self.assertFalse(_inputs_have_equal_length(input_2))
```

but isn't this confusing?

Rather, the current behavior is akin to `min_length`, where we pad all the inputs that are shorter than `max_length` to be equal to `max_length`, and **not truncate** longer sequences.


Found this while making a small PR to add and adjusting the test codes accordingly:
```python
def pad(self, ....):
    if truncation is False and max_length is not None:
        raise ValueError(
            "If `max_length` is provided, `trunction` must be True."
        )

...

# tests/speech_to_text/test_feature_extraction_speech_to_text.py
paddings = ["longest", "max_length", "do_not_pad"]
max_lengths = [None, 4, None]
for max_length, padding in zip(max_lengths, paddings):
    do_truncation = max_length is not None
    inputs = feature_extractor(
        speech_inputs, padding=padding, max_length=max_length, 
        return_attention_mask=True, truncation=do_truncation
    )
```

Could someone let me know if I should continue?





@cwkeam Indeed, this logic might be confusing for sequence feature extraction in particular! 
However, the fact that setting `max_length` doesn't make `truncate=True` stems from the base padding and truncation strategies of `transformers` (see for example [here](https://github.com/huggingface/transformers/blob/b320d87eceb369ea22d5cd73866499851cb2cca3/docs/source/pad_truncation.mdx)). 
Maybe @LysandreJik or @sgugger have a nicer explanation for why that's the case?

Transformers does not do any magic behind the scenes. If you want truncation, you have to enable it. If you want padding, you have to enable it. `max_length` itself doesn't set any of those because what default are we supposed to take? There are several truncation strategies even when a max length is set (to deal with pairs of sentences for instance).

Our design choice, in accordance with our [philosophy](https://huggingface.co/docs/transformers/philosophy) was to avoid any magic default and let the user manually control everything. I agree it can be confusing and the documentation could definitely be improved (if you wanted to make a contribution ;-) )

Right, thanks for kindly explaining! I think I had to understand that `padding` and `truncation` must be two well-separated functionalities. That makes sense.

Thank you!