STYLECAP: AUTOMATIC SPEAKING-STYLE CAPTIONING FROM SPEECHBASED ON SPEECH AND LANGUAGE SELF-SUPERVISED LEARNING MODELSKazuki Yamauchi⋆∗ , Yusuke Ijima† , Yuki Saito⋆⋆

The University of Tokyo, Japan, † NTT Corporation, Japan
Speaking-style
captioning model

ABSTRACT
We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of
conventional techniques for para-/non-linguistic information recognition focus on the category classiﬁcation or the intensity estimationof pre-deﬁned labels, they cannot provide the reasoning of the recognition result in an interpretable manner. StyleCap is a ﬁrst step towards an end-to-end method for generating speaking-style prompts
from speech, i.e., automatic speaking-style captioning. StyleCap
is trained with paired data of speech and natural language descriptions. We train neural networks that convert a speech representation
vector into preﬁx vectors that are fed into a large language model(LLM)-based text decoder. We explore an appropriate text decoderand speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer
LLMs for the text decoder, speech self-supervised learning (SSL)features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions. Samplesof speaking-style captions generated by our StyleCap are publicly
available1 .
Index Terms— Speaking styles, Natural language descriptions,Self-supervised learning model, Large language models
1. INTRODUCTION
Speech contains not only linguistic information but also para-/nonlinguistic information [1]. The latter information, which includes
the speaker’s emotion and identity, can add variations to the spoken (i.e., linguistic) content and enrich human speech communication. Hence, such information from speech needs to be automatically recognized to develop human-oriented spoken language processing technologies, such as a natural conversational agent. Beneﬁting from the development of sophisticated deep learning architectures and techniques, such as Transformer [2] and self-supervised
learning (SSL) [3] for extracting meaningful feature representations
from speech, the accuracy of para-/non-linguistic information recognition has been improved signiﬁcantly [4]–[6]. Moreover, the knowledge of deep learning-based technologies to recognize this information can be shared with other speech generative tasks, such as expressive text-to-speech (TTS) [7] and emotional voice conversion [8].
Another crucial factor for better para-/non-linguistic information
recognition is the explainability of the recognition results. Some applications such as healthcare will require not only the recognition
result but also its reasoning [9], which should be interpretable forusers. However, most existing techniques focus on the category classiﬁcation or the intensity estimation of labels deﬁned by authors ordataset developers. One possible approach to solve this issue would∗ This author performed the work while at NTT Corporation as an intern.
1 https://ntt-hilab-gensp.github.io/icassp2024stylecap/

Input speech

A male bass said slowly and loudly.
Speaking-style caption

Fig. 1. Concept of automatic speaking-style captioning.
be representing the evidence, i.e., para-/non-linguistic informationin speech, with a natural language description. Several studies haveattempted to acquire the relationship between para-/non-linguistic
information and natural language descriptions [10], [11]. However,these methods still output class labels such as emotional states.
From a different viewpoint, generating natural language descriptions from input audio/image data can be regarded as a captioningtask. For instance, audio captioning [12] and image captioning [13]are tasks to describe content information (objects and its behavior)in audio and image data, respectively. In these tasks, the remarkable progress of large-scale pre-trained deep neural network (DNN)
models [14]–[16], has achieved the audio/image captioning performance better than conventional methods. Unlike these tasks, this paper focuses on para-/non-linguistic information rather than content
information in speech. However, since this information cannot bedirectly observed in input speech [17], generating captions of suchinformation can be regarded as a new challenge.
In this paper, we propose a deep learning technique for describing para-/non-linguistic information such as speaking style in speechwith natural language, called automatic speaking-style captioning(Fig. 1). One way to train a DNN-based speaking-style captioning
model is to use a sufﬁciently large dataset including many pairs ofspeech and sentences that describe various speaking styles, but suchdatasets have yet to be constructed. Instead, we combine the existing LibriTTS and PromptSpeech corpora and build a multi-speakerspeech corpus paired with the natural language instructions of speaking styles (e.g., pitch, and speed) to train an end-to-end speakingstyle captioning model consisting of a speech encoder and a text
decoder. Inspired by the DNN-based method for image captioning [18], our proposed method, StyleCap, predicts preﬁx vectors
fed into a large language model (LLM)-based text decoder fromﬁxed-length speech representation vectors extracted by the speechencoder. In this paper, we experimentally explore an appropriate
text decoder and speech representation suitable for this new task.
In addition, since one speaking style can be described in various
ways, automatic speaking-style captioning involves learning oneto-many mapping. To overcome this difﬁculty, we also introduce
a simple data augmentation method, sentence rephrasing augmentation, that rephrases sentences using an LLM. We conduct an automatic style-captioning experiment to assess the effectiveness ofthe proposed method, using evaluation metrics commonly utilized
in other natural language generation tasks. The experimental results
demonstrate that our StyleCap leveraging richer LLMs for the text
decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversityof generated speaking-style captions.

Aggregation module

Fixed-length vectorPrefix embeddings

Sum along frame

Speech SSL modelTransformer



・
・
・

Transformer

Multi-headattention

LinearBLSTM

Transformer

2

・
・
・

・
・
・

LLM

Transformer

1

BLSTM

Transformer

0

BLSTM

Transformer

CNN

A male bass said slowly and loudly.
Speaking-style caption
Weighted-sum
along layer

Input speech
Speech encoderLinearPrefix const.

Mapping network

Text decoder

Fig. 2. DNN architecture of StyleCap, end-to-end model that automatically describes speaking style of input speech with natural language.
2. METHODS
2.1. Task overview
We deﬁne automatic speaking-style captioning, a novel task that explains the speaking style of input speech with a natural language descriptions. Speciﬁcally, a captioning model takes a single speaker’s
speech sample as the input and generates a sentence describing the
speaker’s speaking style, for instance, “The speaker’s sound height
is normal, but the speed is very fast, and the volume is very low.”
Let x and y be a speech waveform and a corresponding
speaking-style caption (i.e., token sequence), respectively. Because the lengths of x and y are different, one can introduce asequence-to-sequence model widely used in spoken language processing tasks, such as Transformer [2], into this captioning task. In
Section 3, we evaluate the speaking-style captioning performance of
this naive Transformer-based encoder-decoder model.
2.2. StyleCap: end-to-end automatic speaking-style captioning
StyleCap incorporates two SSL models for speech and text processing and achieves the end-to-end generative modeling of speakingstyle prompts. Fig. 2 shows an overview of StyleCap, which isinspired by ClipCap [18], an existing model for automatic image
captioning. Speciﬁcally, StyleCap consists of three DNNs: speechencoder, text decoder, and mapping network.
The speech encoder extracts a ﬁxed-length feature vector froman input speech waveform. Considering the success in various spoken language processing tasks, we incorporate a speech SSL model
into the speech encoder. First, all hidden vectors of the SSL model,
hl = [hl,1 , . . . , hl,T ]⊤ (l = 1, . . . , L), are extracted from a speech
waveform x. Then, the weighted-sum of the hidden
layer outputs∑
along layers for each frame index t, i.e., h̄t = l wl hl,t , is taken,
where wl is a trainable weight coefﬁcient of the lth hidden layer
output. Finally, the hidden vector sequence, h̄ = [h̄1 , . . . , h̄T ]⊤ ,is encoded by an aggregation module consisting of a stack of bidirectional long short-term memory (BLSTM) and multi-head attention (MHA) layers, and the outputs are then taken to be summed
along the frame direction to obtain a Dz -dimensional ﬁxed-length
speech feature vector z.
The text decoder leverages a pre-trained LLM and generates
the speaking-style caption of the given speech waveform. Similar to
the ClipCap text decoder, it ﬁrst takes K × Dw preﬁx embeddings⊤ ⊤p = [p⊤predicted by the mapping network from the
1 , . . . , pK ]feature vector z, where K and Dw denote the preﬁx length and theword embedding dimensionality of the LLM, respectively. Then,the decoder autoregressively generates the tokens of speaking-style
captions using the preﬁx embeddings as conditional vectors.
The mapping network projects the speech encoder output z
onto the word embedding space, i.e., preﬁx embeddings p. It consists of a stack of Transformer layers and K × Dz trainable preﬁx⊤ ⊤constant parameters c = [c⊤1 , . . . , cK ] to retrieve meaningful feature representation from z through MHA layers.
2.3. Data augmentation by rephrasing sentences using an LLM
Because one speaking style can be described in various ways, automatic speaking-style captioning essentially requires learning a oneto-many mapping similar to TTS. To mitigate the difﬁculty, we introduce sentence rephrasing augmentation using an LLM to increase
the diversity of speaking-style captions in the training data. Specifically, we ﬁrst ask the pre-trained Llama 2-Chat (7B)2 [19], which
is an LLM optimized for dialogue use cases, to generate ﬁve sentences from one given sentence on the basis of the following prompt:“Rewrite the following sentence that describes someone’s style of
speaking in a different way, keeping the meaning of the original description. Original Description: [subject to be rephrased].” Then,
we select one sentence from the ﬁve rephrased sentences on the basis of their BERTScore [20] values: i.e., we pick on the sentencewith the highest score if the score is higher than 0.80. For instance,
the description “His sound height is normal, but the speed is very
fast, and the volume is very low.” can be rephrased as “Despite hisnormal height, his sound is incredibly fast and surprisingly quiet.”
3. EXPERIMENTS
3.1. Dataset
We used PromptSpeech3 , which includes natural language instructions (style prompt) of various speaking styles. This dataset wasconstructed for PromptTTS [21] that can synthesize speech in accordance with a style prompt. We used the training subset of PromptSpeech real version, which includes 26,588 human-annotated style
prompts of multi-speakers’ speech samples from LibriTTS [22]. We
divided the 26,588 prompts into training (24,953 by 1,113 speakers), development (857 by 40), and test (778 by 38) subsets andpaired them with the corresponding speech samples from LibriTTS.
Although PromptSpeech also includes categorical style factors thatindicate class labels regarding gender, pitch, speed, and volume, we
did not use them to train the captioning models.
2 https://huggingface.co/meta-llama/Llama-2-7b-chat
3 https://speechresearch.github.io/prompttts

Table 1. Automatic speaking-style captioning results. AM is the aggregation module. B@4, R, M, BS, C, and S denote BLEU@4, ROUGEL, METEOR, BERTScore, CIDEr-D, and SPICE scores, respectively. distinct-1/-2 of reference captions are 0.020 and 0.071, respectively.
(a) Results without sentence rephrasing augmentation.

Model
Transformer-based encoder-decoder

StyleCap w/ GPT-2 (proposed)

StyleCap w/ Llama 2 (proposed)

Speech encoder
Mel-spectrogram
x-vector
WavLM
Mel-spectrogram + AM
x-vector
WavLM + AM
Mel-spectrogram + AM
x-vector
WavLM + AM

B@4↑
0.163
0.096
0.253
0.178
0.085
0.228
0.160
0.076
0.273

R↑
0.352
0.269
0.475
0.381
0.273
0.433
0.358
0.262
0.497

M↑
0.320
0.248
0.456
0.357
0.255
0.410
0.332
0.239
0.469

BS↑
0.817
0.799
0.850
0.827
0.800
0.839
0.821
0.799
0.855

C↑
2.171
1.289
3.239
2.295
1.138
2.868
2.109
1.107
3.471

S↑
0.273
0.209
0.419
0.316
0.214
0.370
0.295
0.213
0.434

distinct-1↑
0.019
0.015
0.022
0.020
0.013
0.022
0.022
0.016
0.023

distinct-2↑
0.049
0.039
0.064
0.057
0.032
0.064
0.066
0.042
0.073

C↑
1.847
1.046
3.172
2.122
0.895
3.056
2.131
1.157
3.594

S↑
0.239
0.191
0.404
0.294
0.210
0.403
0.279
0.212
0.447

distinct-1↑
0.018
0.012
0.021
0.021
0.013
0.022
0.024
0.014
0.027

distinct-2↑
0.046
0.027
0.059
0.063
0.033
0.068
0.065
0.034
0.079

(b) Results with sentence rephrasing augmentation.

Model
Transformer-based encoder-decoder

StyleCap w/ GPT-2 (proposed)

StyleCap w/ Llama 2 (proposed)

Speech encoder
Mel-spectrogram
x-vector
WavLM
Mel-spectrogram + AM
x-vector
WavLM + AM
Mel-spectrogram + AM
x-vector
WavLM + AM

B@4↑
0.140
0.071
0.246
0.164
0.068
0.239
0.165
0.084
0.279

3.2. Experimental conditions
As for the speech encoder, we used three types of speech feature representations: mel-spectrogram, speaker embeddings forspeaker veriﬁcation, and hidden vectors of a speech SSL model forthe performance comparison. We extracted 80-dimensional melspectrograms from speech waveforms with 10 ms frame shift. We
also used WavLM BASE+4 [23] as the speech SSL model andextracted weighted-sum representation for each frame. To obtaina ﬁxed-length vector representation for the mel-spectrogram andspeech SSL model, a stack of 4-layer BLSTMs and MHA with 8
heads was used as an aggregation module. As for the speaker embeddings, 512-dimensional x-vectors [24] obtained from the pre-trained
WavLM BASE+ for speaker veriﬁcation5 was used. As for the text
decoder, we used two types of pre-trained LLMs: GPT-26 [25] andLlama 2 (7B)7 [19]. The former is the same setting as ClipCap [18],while the latter can be regarded as the richer one. The numbersof model parameters for GPT-2 and Llama 2 were 125M and 7B,
respectively. The dimensions of word embeddings for each model
were 768 and 4,096, respectively. The mapping network consistedof 8-layer Transformer-encoders. The preﬁx length and dropout
ratio were set to 40 and 0.2, respectively. These parameters were
empirically determined. During the model training, we only trained
the mapping network, the aggregation module in the speech encoder,and the weight coefﬁcient for each layer in WavLM (i.e., wl ) usedin the SSL feature aggregation process. The model parameters of
LLMs and WavLM were frozen. All modules are trained in an endto-end manner using the cross entropy loss between the generated
captions and the ground-truth captions. Each model was trained
without sentence rephrasing augmentation at 20 epochs or with theaugmentation at 10 epochs because the augmentation doubles thedata size. The batch size was set to 16.
4 https://huggingface.co/microsoft/wavlm-base-plus

R↑
0.332
0.244
0.464
0.368
0.260
0.470
0.353
0.259
0.507

M↑
0.303
0.212
0.441
0.334
0.237
0.439
0.327
0.237
0.479

BS↑
0.814
0.792
0.848
0.822
0.798
0.848
0.818
0.796
0.857

We also trained a naive Transformer-based encoder-decoder
model as the baseline system. The input features were extracted by
the almost same speech encoder as those explained in the previous
paragraph, but the aggregation module was replaced by an attention
layer to align the lengths of speech features and tokens in a generated caption. The encoder had 12 encoder blocks. The decoder had
a token embedding layer followed by 6 decoder blocks. We adopted
MHA with 4 heads of 256 dimensions for both encoder and decoder.
The implementation was based on Huggingface Transformers8 .
As evaluation metrics for captioning accuracy, we used BLEU [26],
ROUGE-L [27], METEOR [28], BERTScore [20], CIDEr-D [29],and SPICE [30], between generated and annotated captions referring
to other natural language generation tasks. We also used distinct-1/2 [31] to evaluate the diversity of generated captions.
3.3. Experimental results
Table 1(a) shows experimental results without sentence rephrasing
augmentation. First, a performance comparison by the speech encoder difference showed WavLM performed the best in this as wellas other speech tasks. On the other hand, x-vectors did not work
well because they were mainly trained to represent speaker characteristics rather than speaking styles such as speech rhythm [32].
In addition, the use of Llama 2 for the text decoder performed better than GPT-2 when employing WavLM for the speech encoder,which demonstrates that leveraging of richer LLM-based text decoders is a crucial factor to improve the captioning performance of
StyleCap. On the other hand, Llama 2 did not necessarily improve
the performance when mel-spectrogram was used as the input feature. One reason for this would be overﬁtting to the training data.
The word embedding dimension of Llama 2 was 4,096-dimensional
embeddings, which greatly outnumbers that of GPT-2 (768). Therefore, the speech encoder with the mel-spectrogram input might suffer from the high dimensionality of Llama 2 word embeddings and

5 https://huggingface.co/microsoft/wavlm-base-plus-sv
6 https://huggingface.co/gpt2
7 https://huggingface.co/meta-llama/Llama-2-7b

8 https://huggingface.co/docs/transformers/model_doc/
speech_to_text

1.0
0.8

METEOR

Table 2. The accuracy (%) of style factor classiﬁcations with embedding for each speech encoder. P, S, and V indicate Pitch, Speed,
and Volume respectively.
Speech encoder
GenderP
S
V
Avg.
Mel-spectrogram + AM
93.8
62.1 67.8 54.7 69.6
x-vector
94.0
40.5 44.0 49.4 57.0
WavLM + AM
91.0
61.0 85.2 69.9 76.8

0.6
0.4
0.2

Table 3. Evaluation scores with changing preﬁx length.
Preﬁx length
1
2
5
10
40
60
METEOR
0.419 0.437 0.468 0.464 0.479 0.459
BERTScore
0.839 0.845 0.853 0.853 0.857 0.852
result in overﬁtting. In contrast, WavLM prevented such overﬁtting
thanks to pre-training using a massive amount of speech data. In
summary, StyleCap employing WavLM and Llama 2 outperformed
naive Transformer-based encoder-decoder models.
Table 1(b) shows experimental results with sentence rephrasing
augmentation. Overall tendencies were similar to the case without
the augmentation. We can see that sentence rephrasing augmentation can improve the captioning performance of StyleCap employing WavLM. In other words, sentence rephrasing augmentation canmake StyleCap to generate more diverse and accurate speaking-style
captions, as shown in the improved distinct-1/-2 and other captioning
results. These results indicate that sentence rephrasing augmentation
is effective to deal with the difﬁculty of speaking-style captioning,i.e., learning one-to-many mapping. In contrast, the performance ofTransformer-based encoder-decoder models could not be improved.
3.4. Discussion
3.4.1. Analysis of StyleCap behavior
To further understand the methods, we performed analysis from two
aspects: the property of ﬁxed-length vectors extracted by the speechencoders and the captioning performance from the vectors.
We ﬁrst performed style factor classiﬁcation using the learned
ﬁxed-length vectors. As described in Sect. 3.1, PromptSpeech also
includes class labels indicating four style factors: gender, pitch,
speed, and volume. The numbers of class labels for gender and theothers are two (male/female), and three (low/mid/high), respectively.
We trained a simple linear layer to classify the four style factors froma ﬁxed-length vector extracted by the speech encoder of each trained
StyleCap. In this analysis, trained StyleCap employing Llama 2 forthe text decoder and sentence rephrasing augmentation for the training were used. Table 2 lists the classiﬁcation accuracy. We can seethat all speech encoders can classify each style factor to some extent. This implies that StyleCap can acquire speaking-style-related
information from speech with natural language descriptions only. In
other words, such information can be obtained without class labels
indicating style factors. Comparing the speech encoders, WavLM
achieved the highest classiﬁcation performance, while x-vector
could not capture para-linguistic information except for gender.
Similar tendency was also reported in the previous work [32].
We next analyzed the relationship between the captioning performance and the ﬁxed-length vector extracted by the speech encoder employing WavLM. Fig. 3 shows the violin plot of METEOR
aggregated by the style factor classiﬁcation performance (i.e., thenumber of correct predictions from 1 to 4)9 described in the previous paragraph. From the results, we can see that the captioning
9 We did not observe the complete misclassiﬁcation whose number of correct predictions was 0 in the experiments.

0.0
1

2

3

Number of correct predictions

4

Fig. 3. Violin plot of METEOR by the performance of style factor
classiﬁcations.
performance strongly depends on the style factor classiﬁcation performance. Especially, the misclassiﬁcation of even one factor canconsiderably degrade the captioning performance. Similar tendencies were also found in terms of other metrics such as BERTScore.
These results indicate that capturing adequate para-/non-linguistic
information by the speech encoder is an essential factor to further
improve performance. Although WavLM and the simple aggregation module were used in this paper, another possible approach is to
use CLAP [11], [15] trained from various para-/non-linguistic information.
3.4.2. Ablation study for mapping network
We ﬁnally performed an ablation study for the mapping network. To
investigate the performance by changing the preﬁx length (K) of the
mapping network, we set the preﬁx length to 1, 2, 5, 10, 40, and60, respectively. In this experiment, StyleCap employing WavLM,
Llama 2, and sentence rephrasing augmentation was used. Table 3
shows METEOR and BERTScore only by changing the preﬁx length
due to the space limitation. As we can see, the performance is getting better with the longer preﬁx length. The shorter preﬁx lengths,
especially 1 and 2, tends to worsen the performances because thenumber of trainable model parameters are limited. In addition, thetoo longer preﬁx also degrades the performance due to overﬁtting totraining data. These tendencies were similar to ClipCap [18].
4. CONCLUSIONSIn this paper, we proposed StyleCap, a method to generate natural
language descriptions of speaking styles appearing in speech, which
we call the automatic speaking-style captioning task. As a ﬁrst stepto this end, we explored an appropriate large language model (LLM)based text decoder and speech feature representation suitable for thistask. The experimental results demonstrated that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improved
the accuracy and diversity of generated speaking-style captions. Although this paper focused on speaking styles appearing in speech,
we believe that the proposed approach will be easily applicable forother para-/non-linguistic information such as emotional states andmental illnesses. Applying it to such information, which includes
constructing pairs of speech and natural language descriptions, is for
future work. We will also explore more suitable evaluation metrics
for this task to growth the research area regarding the speaking style
captioning. The evaluation metric using a larger LLM [33] including
the prompt engineering for each task would be a possible approach.
Acknowledgements: This work was supported by JST, ACT-X
Grant Number JPMJAX23CB, Japan.

References[1]

A. S. Cowen, H. A. Elfenbein, P. Laukka, et al., “Mapping 24
emotions conveyed by brief human vocalization,” American
Psychologist, vol. 74, no. 6, pp. 698–712, 2019.

[2]

A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all
you need,” in Proc. NIPS, 2017.

[3]

J. Shor, A. Jansen, W. Han, et al., “Universal paralinguistic
speech representations using self-supervised conformers,” inProc. ICASSP, 2022, pp. 3169–3173.

[4]

M. B. Akçay and K. Oğuz, “Speech emotion recognition:
Emotional models, databases, features, preprocessing methods, supporting modalities, and classiﬁers,” Speech Communication, vol. 116, pp. 56–76, 2020.

[5]

Z. Bai and X.-L. Zhang, “Speaker recognition based on deeplearning: An overview,” Neural Networks, vol. 140, pp. 65–
99, 2021.

[6]

D. M. Low, K. H. Bentley, and S. S. Ghosh, “Automated assessment of psychiatric disorders using speech: A systematic
review,” Laryngoscope Investigative Otolaryngology, vol. 5,
no. 1, pp. 96–116, 2020.

[7]

Y. Guo, C. Du, X. Chen, et al., “EmoDiff: Intensity controllable emotional text-to-speech with soft-label guidance,” inProc. ICASSP, 2023.

[8]

K. Zhou, B. Sisman, R. Rana, et al., “Emotion intensity andits control for emotional voice conversion,” IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 31–48, 2023.

[9]

S. Bharati, M. R. H. Mondal, and P. Podder, “A review on explainable artiﬁcial intelligence for healthcare: Why, how, andwhen?” IEEE Transactions on Artiﬁcial Intelligence, pp. 1–
15, 2023.

[10]

[11]

[12]

H. Dhamyal, B. Elizalde, S. Deshmukh, et al., “Describing
emotions with acoustic property prompts for speech emotion
recognition,” arXiv preprint arXiv:2211.07737, 2022.
Y. Pan, Y. Hu, Y. Yang, et al., “GEmo-CLAP: Genderattribute-enhanced contrastive language-audio pretraining
for speech emotion recognition,” arXiv preprint arXiv:
2306.07848, 2023.
X. Mei, X. Liu, M. D. Plumbley, et al., “Automated audio
captioning: An overview of recent progress and new challenges,” EURASIP Journal on Audio, Speech, and Music Processing, vol. 2022, no. 1, pp. 1–18, 2022.

[13]

L. Xu, Q. Tang, J. Lv, et al., “Deep image captioning: A review of methods, trends and future challenges,” Neurocomputing, vol. 546, p. 126 287, 2023.

[14]

A. Radford, J. W. Kim, C. Hallacy, et al., “Learning transferable visual models from natural language supervision,” inProc. ICML, 2021, pp. 8748–8763.

[15]

E. Benjamin, D. Soham, M. A. Ismail, et al., “CLAP: Learning audio concepts from natural language supervision,” inProc. ICASSP, 2023.

[16]

T. Brown, B. Mann, N. Ryder, et al., “Language models are
few-shot learners,” in Proc. NeurIPS, 2020.

[17]

S. G. Koolagudi and K. S. Rao, “Emotion recognition fromspeech: A review,” International journal of speech technology, vol. 15, pp. 99–117, 2012.

[18]

R. Mokady, A. Hertz, and A. H. Bermano, “ClipCap:
CLIP preﬁx for image captioning,” arXiv preprint arXiv:
2111.09734, 2021.

[19]

H. Touvron, L. Martin, K. Stone, et al., “Llama 2: Openfoundation and ﬁne-tuned chat models,” arXiv preprint
arXiv:2307.09288, 2023.

[20]

T. Zhang, V. Kishore, F. Wu, et al., “BERTScore: Evaluatingtext generation with BERT,” in Proc. ICLR, 2020.

[21]

Z. Guo, Y. Leng, Y. Wu, et al., “PromptTTS: Controllable
text-to-speech with text descriptions,” in Proc. ICASSP,
2023.

[22]

H. Zen, V. Dang, R. Clark, et al., “LibriTTS: A corpus derived from LibriSpeech for text-to-speech,” in Proc. INTERSPEECH, 2019, pp. 1526–1530.

[23]

S. Chen, C. Wang, Z. Chen, et al., “WavLM: Large-scale
self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing,vol. 16, no. 6, pp. 1505–1518, 2022.

[24]

D. Snyder, D. Garcia-Romero, G. Sell, et al., “X-Vectors:
Robust dnn embeddings for speaker recognition,” in Proc.
ICASSP, 2018, pp. 5329–5333.

[25]

A. Radford, J. Wu, R. Child, et al., “Language models areunsupervised multitask learners,” 2019.

[26]

K. Papineni, S. Roukos, T. Ward, et al., “BLEU: A method
for automatic evaluation of machine translation,” in Proc.
ACL, 2002, pp. 311–318.

[27]

C.-Y. Lin, “ROUGE: A package for automatic evaluationof summaries,” in Proc. Workshop on Text Summarization
Branches Out, 2004, pp. 74–81.

[28]

S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with humanjudgments,” in Proc. ACL, 2005, pp. 65–72.

[29]

R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensusbased image description evaluation,” in Proc. CVPR, 2015,
pp. 4566–4575.

[30]

P. Anderson, B. Fernando, M. Johnson, et al., “SPICE: Semantic propositional image caption evaluation,” in Proc.
ECCV, 2016, pp. 382–398.

[31]

J. Li, M. Galley, C. Brockett, et al., “A diversity promoting
objective function for neural conversation models,” in Proc.
ACL, 2016, pp. 110–119.

[32]

K. Fujita, T. Ashihara, H. Kanagawa, et al., “Zero-shot textto-speech synthesis conditioned using self-supervised speech
representation model,” in Proc. ICASSP SASB Workshop,
2023.

[33]

Y. Liu, D. Iter, Y. Xu, et al., “G-Eval: NLG evaluation using GPT-4 with better human alignment,” arXiv preprint
arXiv:2303.16634, 2023.