Package: peft-ser
PEFT-SER [Paper Link] was accepted to 11th International Conference on Affective Computing and Intelligent Interaction (ACII), 2023.This work include implementations for PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models. PEFT-SER is an open source project for researchers exploring SER applications with using parameter efficient finetuning methods.This python package provides checkpoints that was trained on the combined data of IEMOCAP, MSP-Improv, MSP-Podcast, and CREMA-D, using the following models that shows promising results:Whisper TinyWhisper BaseWhisper SmallMassively Multilingual Speech (MMS)WavLM Base+WavLM LargeWe provide the checkpoints with LoRa rank of 16 according to our paper, as LoRa in general provides the best performance in PEFT SER. You can start to use our model within 5 lines of code. The package is under license, and it does not support commercial use.1. Installationpip install peft-ser2. Model Loading# whisper style loading
import peft_ser
model = peft_ser.load_model("whisper-base-lora-16-conv")

data = torch.zeros([1, 16000])
output = model(data)a. Output mappingThe output emotion mappings are:{0: "Neutral", 1: "Angry", 2: "Sad", 3: "Happy"}. We would add a version for 6-emotion later.b. Training detailsFor all the released models, we train/evaluate with the same data.  Unlike the ACII paper where the audio was restricted to 6s, these open-release models support the audio duration to the maximum of 10s for broader use cases. We also combine the convolutional output along with the transformer encodings for fine-tuning, as we find this further increase the model performance. We used a fix seed of 8, training epoch of 30, and learning rate of 2.5x10e-4.c. Training/validation/test splits for reproducing the resultsThe validation set:Session 4 of IEMOCAP and Session 4 of MSP-Improv dataset, Validation set of MSP-Podcast dataset, and Speaker 1059-1073The evaluation set:Session 5 of IEMOCAP and Session 5 of MSP-Improv dataset, Test set of MSP-Podcast dataset, and Speaker 1074-1091All rest data are used for training.d. PerformancePre-trained ModelTest Performance without PEFTTest Performance with LoRaPEFT Model NameWhisper Tiny62.2663.48whisper-tiny-lora-16-convWhisper Base64.3964.92whisper-base-lora-16-convWhisper Small65.7766.01whisper-small-lora-16-convMMSmms-lora-16-convWavLM Base+63.0666.11wavlm-plus-lora-16-convWavLM Large68.5468.66wavlm-large-lora-16-conve. You are free to explore the use of existing models to further fine-tune on other SER datasets, more challenging tasks like 6-emotion, 8-emotion recognition, and also transfer learning on Arousal/Valence/Dominance.CitationPlease cite the following that includes the foundation of the code/methods used for experiments.PEFT-SER@article{feng2023peft,
  title={PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models},
  author={Feng, Tiantian and Narayanan, Shrikanth},
  journal={arXiv preprint arXiv:2306.05350},
  year={2023}
}Trust-SER@article{feng2023trustser,
  title={TrustSER: On the trustworthiness of fine-tuning pre-trained speech embeddings for speech emotion recognition},
  author={Feng, Tiantian and Hebbar, Rajat and Narayanan, Shrikanth},
  journal={arXiv preprint arXiv:2305.11229},
  year={2023}
}You should also cite all the datasets being used including IEMOCAP, MSP-Improv, MSP-Podcast, and CREMA-D.