Venue: Interspeech 2023 Workshop SSW

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This study presents compelling research on the synthesis of turn-taking cues, a field that certainly requires more in-depth investigation. The authors' approach of training Text-to-Speech (TTS) voices on ecologically valid data, specifically the CANDOR Corpus of online conversations, is particularly noteworthy. Initially the selected IPUs of length 1-11 sec, removing all questions and filtering out only IPUs of speaking rate 2-6 syll/sec. The training corpus contains 212 speakers where speaker embeddings are used in the training. After selecting the 53 speaker with more than 10 min of data, the authors decided to pre-train on the read speech corpus LJ speech. 

The TTS is based on FastPitch with  turn-condition in the input. At the end the five TTS voices with the highest MOS scores were selected for the turn-taking study. In a listening test subject had to judge which of two versions sounded more like the speaker was done speaking. The compared mid-turn with end-of-turn in a first study and a end-of-turn with the baseline trained on all conversational data in the second. The results are that conditioning on turn placement  led  to  increased  perception  of  turn-finality comparing both the turn-medial case and the baseline, but that it is very  speaker-specific. There are some indications that the multi-speaker training have been beneficial for speakers with fewer turn-end IPUs in the training corpus. At the same time there is a large variation across speakers where all speaker but on had higher pitch on the last word in turn-final position. 

Some questions to the authors:
The when listening to synthesized texts that are taken from held-out data I get the feeling that the corpus contains a lot of conversations with a stream-of-consciousness-feel that you might get from unplanned telephone conversations. This makes them hard to judge in terms of completeness. It might not always be a strong acoustic cue that makes the other speaker to decide to say something. When you listened to the data, was it easy for you to tell if it is mid-turn or end-of-turn IPU you are listening to?

 One problem with both turn-holding and turn yielding cues is that they overlap, especially in terms of high pitch at the end. You partly solved this be removing a strong turn yielding rise at the end (questions). Turn holding cues can either be planned for, where you use a rising pitch to indicate that there is more to come, and where you mark the end with a falling pitch (turn-yielding cue) – this is what toy have when reading numbers, lists, instructions or retelling events with several parts. Another turn-holding cue is a flat medium pitch and prolongued last syllable (or inserting a filled pause with that prosody). Since you tagged turn-medial and turn-final given who spoke next, it might have been some of the other turn-holding cues listed by Hjalmarsson, Gravano and others at play, such as number of words, grammar or semantics. Or no cue at all - the other speaker just decided to cut in. Did you consider combining the turn-media turn-ending tagging with some kind of prosodic sub-categorization to help the system understand that the prosody is what you want to alter when adding the turn-conditioner to the input text when synthesizing.

Given that the speakers differed so much in prosody, do you think we might benefit from clustering speakers depending on their turn-holding behavior when doing multi-speaker TTS training. 

In figure 4 what does final word duration tell us, isn’t that very dependent on which word they happen to end with. You have speech rate and f0 height of final word. Wouldn’t duration of the last syllable say more (related to filled pauses and Gravano-Hirshberg that looked at the last 500 ms). Now you average long and short words where the slowed down speaking rate might only appear on the last few syllables.  Maybe the results become stronger if you limit it to the las syllable(s)

Strengths:
This study, with its expansive scope of building and evaluating multiple voices, as well as performing prosodic analysis on both the TTS corpus and synthesized samples, is indeed impressive.

Limitations:
The results may not be sufficiently robust across different speakers. The test samples often mimic a stream-of-consciousness style, which potentially makes the task of discerning completion of speech demanding for the crowd-workers.

In conclusion, this research provides valuable insights and prompts thoughtful questions, making it a good fit for the SSW conference. I look forward to seeing its presentation and future development. 

review_title: Review of Synthesising turn-taking cues using natural conversational data

final_score: rating: 7: Good paper, accept
review_content: The authors propose to bias a neural TTS (Tacotron2) with a turn-taking label that specifies if the utterance is intended to keep (HOLD) or release (SHIFT) the floor. FastSpeech models are trained using the CANDOR corpus (1656 online conversatiuons between two speakers - > 56hours of exploitable speech data). Turn-Taking bias are identical to speaker embeddings: they are trained and added to each text encoder output before prosodic predictors.

* Key Strength of the paper
First step towards controlable turn-taking generation.

* Main Weakness of the paper
Low MOS ratings. Strong pre-selection process (212 speakers in the training data, 5 tested)

* Novelty/Originality, taking into account the relevance of the work for the SSW audience

* Technical Correctness, is the work technically and/or scientifically solid? Are sufficient details provided to allow any experiments to be reproduced or equivalent experiments run?
Some points are unclear:
1. Why LJSpeech is used?? 23914 CANDOR (41 speakers) vs. 86 LJSpeech utterances seems wuite imbalanced...
2. What are these ambiguous texts? Do they all end whith full stops? What if silence tokens are replaced by adequate punctuations?
3. Why the target speaker corpus training info displayed in Table 2 has such as strong imbalanced distribution (10% turn-final vs 90% turn-medial) compared to numbers given in 3.1.1?

Comments:
1. Why encoding turns as speaker/style bias added to all embeddings? Why not additional tokens similar to punctuation marks, letting the encoder contextualizing the information?

* Quality of References, is it a good mix of older and newer papers? Do the authors show a good grasp of the current state of the literature? Do they also cite other papers apart from their own work?
Refs are adequate

* Clarity of Presentation, the English does not need to be flawless, but the text should be understandable
The text is clear enough

review_title: First steps towards the generation of turn-taking cues

Meta-review: SSW2003 received 45 papers. The acceptance rate is 82%. We are pleased to inform you that your paper has been accepted by the SSW2023 Program Committee. Please read the reviews carefully and submit your camera-ready paper by June 28th. Most reviewers performed a detailed review. Please answer to their questions and consider their comments. Note that camera-ready papers are credited with one extra page to allow authors to consider reviewers’ suggestions. So max 7 pages in total including figures & refs.
The deadline for submitting the revised version (with full non-anonymized authors and refs!) is 28th June.

Decision: Accept