SLP-P15.4

StyleCap: Automatic Speaking-Style Captioning from Speech
Based on Speech and Language Self-supervised Learning Models⋆"⋆Kazuki Yamauchi , Yusuke Ijima , Yuki Saito⋆"
The University of Tokyo, Japan, NTT Corporation, Japan

Demo page

Contributions: speaking-style captioning and StyleCap● Contribution 1: new task, speaking-style captioning● Contribution 2: speaking-style captioning model, StyleCap○ Describe speaking-style in speech in natural language○ Go beyond the limitations by pre-defined discrete labels○ Leverage speech and language SSL models○ Introduce LLM-based data augmentation by sentence rephrasing

Speaking-style caption

Speaking-style
captioning model

StyleCap
Speech SSL-based
Speech encoder

e.g. “His voice is very loud,
but the tone is very low.”● Related works: para-/non-linguistic information recognition

Q-former-based
Mapping network

Speaking-stylecaption

LLM-based
Text decoder● Related works: image/audio captioning○ Classify speech into pre-defined categories○ Desire Human-interpretable reasoning○ Describe content information in natural language○ e.g. ClipCap [1], leveraging SSL for image captioning

Method: StyleCap, leveraging SSL models and LLM-based data augmentation
DNN architecture of StyleCap

Data augmentation by sentence rephrasing● SSL-based speech encoder● Challenge: one-to-many problem of style prediction○ Compress SSL features into a fixed-length vector using AM○ Not uniquely determined descriptions to express speaking-style→ Data augmentation by sentence rephrasing using LLM to● Q-former-based mapping network○ Map the fixed-length vector to the word embedding space

increase the diversity of description● Problem: change in meaning due to rephrasing error by LLM→ Filtering by BERTScore [2] to select appropriate candidates● Example of rephrasing:● LLM-based text decoder○ Generate a caption from the mapping network output by LLM
Aggregation Module (AM)

Speech SSL modelTransformer
・・・

Transformer
Transformer
CNN

Sum along frame
Multi-headattention

LinearBLSTM

Transformer
・・・・・・

LLM

BLSTM

Transformer

BLSTM

Transformer

Weighted-sum
along layer

Select the sentence with the highest
BERTScore to the original

PromptA male bass said slowly and loudly.

Prefix const.

LinearSpeech encoder

His sound height is normal, but the speed is very fast, and the volume is very low.
↓ Rephrased by LLM (Llama 2-Chat 7B model [3])Despite his normal height, his sound is incredibly fast and surprisingly quiet.

Prefix embeddings

Rewrite thefollowing sentence...

Speaking-style
captionCandidate 1: BERTScore 0.81
Candidate 2: BERTScore 0.86
Candidate 3: BERTScore 0.75
Candidate 4: BERTScore 0.91
Candidate 5: BERTScore 0.79

LLM
(Llama 2-Chat 7B)

Speaking-style caption(Original)

Mapping network Text decoder

Experimental evaluation
Experimental conditions

Analysis of StyleCap behavior● Dataset: PromptSpeech [4]● Style factor classification○ Various (speech, style prompt) pairs of data
○ Speech in LibriTTS corpus [5] annotated with style prompt○ Style factor: gender, pitch, speed, volume○ Various speakers/utterances: 1,191/26,588○ Predict style factors from each trained speech encoder output■ 2 classes for gender (male/female)■ 3 classes for pitch, speed, volume (low/mid/high)● Model configuration of StyleCap○ Speech encoder: WavLM BASE+ [6] / mel-spec. / x-vector [7]○ Mapping network: Transformer encoder x 8○ Text decoder: GPT-2 (125M params) [8] / Llama 2 (7B params)

Experimental results

Speech encoder gender pitch speed volume

Ave.

Mel-spec. + AM

93.8

62.1

67.8

54.7

69.6

x-vec.

94.0

40.5

44.0

49.4

57.0

WavLM + AM

91.0

61.0

85.2

69.9

76.8

○ Classification performance: WavLM + AM > others● Metrics: METEOR [9] (M), BERTScore (BS), Distinct-1 [10] (D1)
w/o Sentence Rephrasing w/ Sentence Rephrasing
Speech encoder

Text decoder

M(↑)

BS(↑)

D1(↑)

M(↑)

BS(↑)

D1(↑)

Mel-spec. + AM

GPT-2

0.357

0.827

0.020

0.334

0.822

0.021

x-vec.

GPT-2

0.255

0.800

0.013

0.237

0.798

0.013

WavLM + AM

GPT-2

0.410

0.839

0.022

0.439

0.848

0.022

Mel-spec. + AM

Llama 2

0.332

0.821

0.022

0.327

0.818

0.024

x-vec.

Llama 2

0.239

0.799

0.016

0.237

0.769

0.014

WavLM + AM

Llama 2

0.469

0.855

0.023

0.479

0.857

0.027

1. Speech encoder validity: WavLM + AM > others
2. Text decoder validity: Llama 2 > GPT-2
3. Sentence Rephrasing: Improved diversity in particular

Linear layer

Gender(male/female)Linear layer

Pitch(low/mid/high)Linear layer

Speed(low/mid/high)Linear layer

Volume(low/mid/high)Speech encoder

Overview of style factor classification
METEOR score tends to degrade● METEOR distribution by the number of correct○ Captioning performance (i.e., METEOR score) degrades as
the number of correct predictions decreases→ Capturing adequate para-/non-linguistic information bythe speech encoder is an essential factor

Future direction● Adapt to other para-/non-linguistic information (e.g. emotion)● Dataset construction for more diverse speaking-style caption
Acknowledgments: This work was supported by JST, ACT-X Grant Number JPMJAX23CB, Japan.

Reference[1] R. Mokady et al., arXiv preprint arXiv: 2111.09734, 2021. [2] T. Zhang et al., in Proc. ICLR, 2020. [3] H. Touvron et al., arXiv preprint arXiv:2307.09288, 2023. [4] Z. Guo et al., in Proc. ICASSP, 2023.
[5] H. Zen et al., in Proc. INTERSPEECH, 2019. [6] S. Chen et al., IEEE JSTSP, 2022. [7] D. Snyder et al., in Proc. ICASSP, 2018. [8] A. Radford et al., 2019. [9] S. Banerjee et al., in Proc. ACL, 2005. [10] J. Li et al., in Proc. ACL, 2016.

Saruwatari Lab., The University of Tokyo, Japan, NTT Corporation, Japan.

ICASSP 2024

©Kazuki Yamauchi, Apr. 2024.