`skip_special_tokens` for `Wav2Vec2CTCTokenizer` does not work expectedly.
### System Info

- `transformers` version: 4.37.2
- Platform: Linux-5.15.0-1042-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: DDP

### Who can help?

@sanchit-gandhi @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
processor = Wav2Vec2Processor.from_pretrained(model_args.model_name_or_path)
model = Wav2Vec2ConformerForCTC.from_pretrained(
    model_args.model_name_or_path, ctc_loss_reduction="mean", pad_token_id=processor.tokenizer.pad_token_id
)

metric_wer = evaluate.load("wer")
metric_cer = evaluate.load("cer")

def _compute_metrics(pred):
    pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

    # ["
": 1, "
": 2, "<unk>": 3, "<pad>": 0]
    preds = processor.batch_decode(pred.predictions, skip_special_tokens=True)
    labels = processor.batch_decode(pred.label_ids, skip_special_tokens=True, group_tokens=False)

    preds = [postprocess(text) if len(text) != 0 else "-" for text in preds]
    labels = [postprocess(sentence) for sentence in labels]

    preds = [re.sub(r"\s+", "", text) for text in preds]
    labels = [re.sub(r"\s+", "", text) for text in labels]

    wer = 100 * metric_wer.compute(predictions=preds, references=labels)
    cer = 100 * metric_cer.compute(predictions=preds, references=labels)

    return {"wer": wer, "cer": cer}

def _preprocess_logits_for_metrics(logits, labels=None):
    return torch.argmax(logits, dim=-1)

trainer = Trainer(
    args=training_args,
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=data_collator,
    tokenizer=processor,
    compute_metrics=_compute_metrics,
    preprocess_logits_for_metrics=_preprocess_logits_for_metrics,
)
```

### Expected behavior

I want to train ASR model and this issue came out when I evaluate during training.
`preds = processor.batch_decode(pred.predictions, skip_special_tokens=True)`'s results should remove all special tokens (`
`, `<\s>`, `<unk>`, `<pad>`). However, in my experiment, `<unk>` is not removed.

So, I look at the code for `Wav2Vec2CTCTokenizer._decode` (transformers / models / wav2vec2 / tokenization_wav2vec2.py).
```python
 def _decode(
        self,
        token_ids: List[int],
        skip_special_tokens: bool = False,
        clean_up_tokenization_spaces: bool = None,
        group_tokens: bool = True,
        spaces_between_special_tokens: bool = False,
        output_word_offsets: Optional[bool] = False,
        output_char_offsets: Optional[bool] = False,
    ) -> str:
        """
        special _decode function is needed for Wav2Vec2Tokenizer because added tokens should be treated exactly the
        same as tokens of the base vocabulary and therefore the function `convert_tokens_to_string` has to be called on
        the whole token list and not individually on added tokens
        """
        filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)

        result = []
        for token in filtered_tokens:
            if skip_special_tokens and token in self.all_special_ids:
                continue
            result.append(token)
```
This code removes the special tokens in the `filtered_tokens` by `self.all_special_ids`, but when I print the `filtered_tokens`, the outcome looks like `['|', '|', 'token1', 'token2', 'token3', '|', '|', 'token4', '|', '|', '|', 'token5', 'token6', '|', '|', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>']`. 

Since `self.all_special_ids`'s elements are integers for special tokens, `if skip_special_tokens and token in self.all_special_ids:` statement does not work expectedly.

Shouldn’t it be `if skip_special_tokens and token in self.all_special_tokens:`?