Coco-Nut: Corpus of Japanese Utterance and Voice Characteristics
Description for Prompt-based Control
Aya Watanabe, Shinnosuke Takamichi, Yuki Saito, Wataru Nakata, Detai Xin, Hiroshi Saruwatari(The University of Tokyo, Japan.)

Summary: Novel corpus for prompt-based control over voice characteristics on text-to-speech (TTS)Novel corpus to promote research

Issue with existing corpora [4, 5]

Various voice characteristics TTS● Corpus construction method○ Based on speech collected through web■ Diverse, but noisy● → Quality assurance is necessary● Coco-Nut: based on proposed method○ Diverse voice characteristics→ Need of a public corpus with diverse speakers ○ Public (QR code is at the bottom.)● Existing voice characteristics controllable TTS ● Only a few speaker: lack of variation○ Based on existing TTS corpora○ Speaker ID [1], speaker attributes [2, 3], etc.
■ The data size per speaker is more
■ Can only choose small range of voice!
prioritized than the number of speakers.
● Potential of prompt-based control [4, 5]○ Control over wider range of characteristics ● Internal: low reproducibility
A male voice in his thirties.
He spoke slowly and calmly.

TTStexttext

Corpus of connecting nihongo (= “Japanese” in Japanese) utterance and text and its construction method
Corpus specification

Construction method overview● Speech segments○ Approximately 8 hours, 8000 segments● Transcriptions○ With manual correction● Voice characteristics descriptions (prompt)○ Used as a prompt describing voice characteristics
Lorem ipsum
dolor sit
ではみなさ
amet, cons んは、そういうふうに川Loremipsum
dolor sit
amet,ではみなさん
cons+は、そういうふうに川● Divided into 3 sets
○ train:validation:test = 8:1:1○ 1 description / speech for train, 5 for validation/test

Data collection + video filtering

Quality assurance

Annotation● YouTube search with voice-related keywords● Acoustic quality assurance● Add descriptions on speech○ Wikipedia article titles of voice-related categories○ Speech enhancement, perception-aware quality○ Using crowdsourcing● Video ﬁltering with voice-related comment detection
scoring, x-vector [7]-based data selection, etc.
■ Workers listen to speech○ Viewers tend to comment on attractive voices.
● Content quality assurance○ BERT [6]-based classiﬁcation to viewer comments○ Based on transcriptions by Whisper [8]ex) “A male voice in his thirties. He■ Trained on a few manual annotations○ Remove non-target lang., NSFW and non-verbal spoke slowly and calmly.”

Corpus analysis

Machine learning baseline for prompt-based speech retrieval● Prompt-based speech retrieval can be considered as one step prior to prompt-based TTS.

● Video categories

Based on video CLAP [9]-based embedding modelPromptcategories
whom speech (= voice characteristics description)
derived from(deﬁned byYouTube)
SpeechSpeech retrieval from descriptions● Retrieve close speech to the query prompt inembedding space.
○ On cosine similarity● Speech gender (referred description)
Non-binary means
Subjectiveevaluation:
degreeofrelevanceofretrieved
speechboth “male” &“female” appear ● How well do the retrieved speech match subjectively to the query prompt?
in the description. ○ Retrieved candidate, random, ground truth○ Query: 100 prompts (ground truth speech exist in candidates)○ Ask 20 crowdworkers per 1 prompt-speech pair, with a scale of 1 to 9.

Subjective evaluation result● Wide range of speech○ Multiple categories○ Including non-binary voices:mean/std of scores for each pair:global mean/std
Blue: mentioned in the title of each ﬁgure
Gray: ground truth (common across ﬁgures)

Project page
● Distribution pages
are linked from here.
○ Downloading
speech requires
registration.
Reference[1] Hojo et al., IEICE TRANSACTIONS 2017.
[2] Stanton et al., ICASSP 2022.

goes right → better scores in averagegoes up → score varies among annotators.

● 1st > random → ability to learn voice characteristics space relative to human perception● 1st << ground truth → room for improvement e.g., model architecture, training strategy[3] Watanabe et al., ICASSP 2023.
[4] Guo et al., ICASSP 2023.

Saruwatari-Takamichi Lab., The University of Tokyo, Japan.
©Aya Watanabe, Dec. 2023.

[5] Yang et al., arXiv 2023.
[6] Devlin et al., arXiv 2018.

ASRU 2023[7] Snyder et al., ICASSP 2018.
[8] Radford et al., arXiv 2022.

[9] Wu et al., ICASSP 2023.

AVATAR SYMBIOTIC
SOCIETY