Audio related Transformer
Author: chlorane
Pull request: no

I'm now trying to use audio-related Transformer, like Conformer, Audio Spectrogram Transformer, or Whisper, to process audio information. However, our input is ark (with certain dimensions of audio features) and scp files instead of wav form. I tried to use your library but it seems to have errors while processing ark/scp files. Are there any functions to process ark/scp directly, and are there any examples to show their usages? Thanks a lot

Comments:
cc @sanchit-gandhi 

Hey @chlorane - did you get the `ark`/`scp` files using the Kaldi library? I think in this case you're better off just passing the raw `.wav` files directly to the `transformers` models, rather than first through the Kaldi pre-processing to get `ark`/`scp` and then trying to force them through the `transformers` models. The only information I can find regarding converting `ark`/`scp` to `wav` is this thread: https://groups.google.com/g/kaldi-help/c/t6Ra3uHiDJQ/m/R6e01pF5CwAJ For questions such as this one, you might have more luck posting on the Hugging Face forum, where others in the community can pitch-in based on their experiences: https://discuss.huggingface.co

Note that all audio models in the `transformers` library are designed to work directly with audio inputs, as per their respective papers. The `ark`/`scp` file formats first convert the raw audio inputs to either MFCC features or some other feature extracted form, thus these aren't compatible with models that expect raw audio inputs.

> Hey @chlorane - did you get the `ark`/`scp` files using the Kaldi library? I think in this case you're better off just passing the raw `.wav` files directly to the `transformers` models, rather than first through the Kaldi pre-processing to get `ark`/`scp` and then trying to force them through the `transformers` models. The only information I can find regarding converting `ark`/`scp` to `wav` is this thread: https://groups.google.com/g/kaldi-help/c/t6Ra3uHiDJQ/m/R6e01pF5CwAJ For questions such as this one, you might have more luck posting on the Hugging Face forum, where others in the community can pitch-in based on their experiences: https://discuss.huggingface.co
> 
> Note that all audio models in the `transformers` library are designed to work directly with audio inputs, as per their respective papers. The `ark`/`scp` file formats first convert the raw audio inputs to either MFCC features or some other feature extracted form, thus these aren't compatible with models that expect raw audio inputs.

Because our dataset has only (full) ark and corresponding scp files. Some wav files in the dataset are not available

How did you obtain the `ark`/`scp` files - is this a reversible process? I think in this case converting the files to `.wav` is your best bet

> How did you obtain the `ark`/`scp` files - is this a reversible process? I think in this case converting the files to `.wav` is your best bet

I think this is not very reversible. They are retrieved using Kaldi, but we don't have all the original voice in the dataset

Then I'm afraid I don't think it's possible to use these file formats. Have you asked on the Hugging Face forum? You could also check on the Kaldi repo to see if there's any advice there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Leaving this closed since the audio file issue is related to a file format derived from the Kaldi repository (where I still think is the best place to ask for help!)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.