CoNeTTE model for Audio Captioning

CoNeTTE is an audio captioning system, which generate a short textual description of the sound events in any audio file. The architecture and training are explained in the corresponding paper. The model has been developped by me (Étienne Labbé) during my PhD.

Installation

python -m pip install conette

Usage with python

from conette import CoNeTTEConfig, CoNeTTEModel

config = CoNeTTEConfig.from_pretrained("Labbeti/conette")
model = CoNeTTEModel.from_pretrained("Labbeti/conette", config=config)

path = "/your/path/to/audio.wav"
outputs = model(path)
candidate = outputs["cands"][0]
print(candidate)

The model can also accept several audio files at the same time (list[str]), or a list of pre-loaded audio files (list[Tensor]). In this second case you also need to provide the sampling rate of this files:

import torchaudio

path_1 = "/your/path/to/audio_1.wav"
path_2 = "/your/path/to/audio_2.wav"

audio_1, sr_1 = torchaudio.load(path_1)
audio_2, sr_2 = torchaudio.load(path_2)

outputs = model([audio_1, audio_2], sr=[sr_1, sr_2])
candidates = outputs["cands"]
print(candidates)

The model can also produces different captions using a Task Embedding input which indicates the dataset caption style. The default task is "clotho".

outputs = model(path, task="clotho")
candidate = outputs["cands"][0]
print(candidate)

outputs = model(path, task="audiocaps")
candidate = outputs["cands"][0]
print(candidate)

Usage with command line

Simply use the command conette-predict with --audio PATH1 PATH2 ... option. You can also export results to a CSV file using --csv_export PATH.

conette-predict --audio "/your/path/to/audio.wav"

Performance

Test data SPIDEr (%) SPIDEr-FL (%) FENSE (%) Vocab Outputs Scores
AC-test 44.14 43.98 60.81 309 Link Link
CL-eval 30.97 30.87 51.72 636 Link Link

This model checkpoint has been trained for the Clotho dataset, but it can also reach a good performance on AudioCaps with the "audiocaps" task.

Limitations

  • The model expected audio sampled at 32 kHz. The model automatically resample up or down the input audio files. However, it might give worse results, especially when using audio with lower sampling rates.
  • The model has been trained on audio lasting from 1 to 30 seconds. It can handle longer audio files, but it might require more memory and give worse results.

Citation

The preprint version of the paper describing CoNeTTE is available on arxiv: https://arxiv.org/pdf/2309.00454.pdf

@misc{labbé2023conette,
    title        = {CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding},
    author       = {Étienne Labbé and Thomas Pellegrini and Julien Pinquier},
    year         = 2023,
    journal      = {arXiv preprint arXiv:2309.00454},
    url          = {https://arxiv.org/pdf/2309.00454.pdf},
    eprint       = {2309.00454},
    archiveprefix = {arXiv},
    primaryclass = {cs.SD}
}

Additional information

Contact

Maintainer:

Downloads last month
264
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Spaces using Labbeti/conette 2