Label Refining: a semi-supervised method to extract voice characteristics without ground truth

1 INTRODUCTION . Localization of a multimedia work , such as a video game or a motion picture , consists in making changes and adaptations to the original work so that international distributors can market the resulting product in a target country . For this purpose , companies need to translate their works into a target language . Various options are available to them to give substance to their translation . Subtitling is the simplest and cheapest option , but not the most practical one for spectators : a large audience is more comfortable with hearing speech , in general in their mother tongue , rather than reading subtitles while hearing speech in another language . Dubbing , which is more expensive and takes longer to set up , better retains the audience ’ s immersion by replacing the original voices with dubbing comedian voices . It involves a voice selection process , which means choosing among several candidate voices in a target language . This selection is referred to as voice casting . With the emergence of streaming platforms ( e.g . Netflix , Disney+ , Amazon Prime Video ) and the rapid development of the video game industry , the number of works to be localized is increasing dramatically . As a result , more and more actors are available on the voice market . Vocal Casting is an artisanal process that requires a lot of precision and is carried out by human operators . As it is , it can not be applied to a very large number of actors . Operators very often rely on their memory , which is limited to a small number of actors with whom they have worked and in whom they have confidence . However , this operation can degrade the immersion of the audience since the most oftused voices become easily recognizable . They then become references and create cognitive biases in the public ’ s perception of the character . Automatic actor search and recommendation tools , based on voice processing , would help operators find new actors who would then enrich the vocal diversity of the works to the delight of the audience . In this article , we wish to address some of the complexity of voice casting and we start with the voices of professional actors playing characters for the gaming industry . The articles Gresse et al . ( 2020 ; 2019 ; 2017 ) ; Malik & Nguyen ( 2021 ) ; Obin & Roebel ( 2016 ) ; Obin et al . ( 2014 ) ; Quillot et al . ( 2021b ; a ) cited in this paper are the only ones dealing with the speech-processing-based automation of Voice Casting in the literature to our knowledge . We rely on Gresse et al . ( 2020 ) , a recent work which defines the p-vector , a neural-network-based representation of the voice dedicated to the character aspects in acted voices . The application context in Gresse et al . ( 2020 ) ( like in related previous works Gresse et al . ( 2017 ; 2019 ) ) is voice casting for audio dubbing . The final objective of this work is to assess how close an actor – acting in a target language – is to the voice of a given character , speaking in a source language . This can be referred to as character-based voice similarity . In contrast to Obin et al . ( 2014 ) ; Obin & Roebel ( 2016 ) , which proposed a voice similarity system for the acted voice based on data labeled with speech classes ( age , gender , emotions ... ) by an expert , the p-vector approach does not use any expert labels . This representation is learnt from audio associated with the played character label , without any additional information . Also , the authors of Quillot et al . ( 2021a ) recently posed the following hypothesis : when a character is played , the actor adds to the voice some information that characterizes the character and is independent of the speaker identity . They then introduced SICI ( Speaker-Independent Character Information ) and showed that there is speaker-independent character information in the acted voice and that it is present in p-vectors pretrained on a speaker task Quillot et al . ( 2021b ) . Although these approaches deal with final works ( video games ) , another one consists in working directly on the decision data from Artistic Directors . Recently , researchers from Warner Bros. evaluated their models on this kind of data . Unfortunately , these data are sensitive and their acquisition is not trivial since it requires to work on the critical voice casting process . This is why we decided to position ourselves in a task quite different from Malik & Nguyen ( 2021 ) , since we do not use Artistic Director decisions . Voice similarity systems alone are not enough for Voice Casting operators . They do not trust the computed recommendations and need a more complete interface to interact with the system . Many works propose explainability approaches Došilović et al . ( 2018 ) but these are too complicated to be understood by a user who is not expert in computer science , and more particularly in machine learning . We strongly believe that the introduction of explanatory labels will help the operators understand the decisions by giving , for example , the list of speech characteristics shared by two recordings/actors . Such labels could also be used to express queries in voice search systems . Unfortunately , voice casting is an underdefined task Bonastre ( 2019 ) . The operators themselves can not formally define their decision criteria . There is also no standard taxonomy in cinema to describe voices , making it impossible to create an annotation protocol . The question is then : how can we extract features from voices when we have no a priori knowledge of them ? Different representation learning approaches exist in the literature , but none of them corresponds to our expectations . 1 ) Feature extraction surveyed in Bengio et al . ( 2014 ) consists in representing an observation by a vector of reals but not in the form of binary vectors that we call characteristics in this paper . 2 ) Supervised methods like Hu et al . ( 2017 ) require labels that we do not have access to in order to extract characteristics . 3 ) Unsupervised methods like Peri et al . ( 2020 ) propose generic representations from which we could not extract characteristics dedicated to the domain of acted voice and character identity . Hence , to address the lack of method to extract characteristics without ground truth , this article proposes a totally new approach entitled Label Refining . It stats with training a representation extractor using non-expert initial labels . A clustering algorithm is then employed with the purpose to emphasizing characteristics shared by groups of observations which we term refined labels . A last step consists in extracting a new representation from refined labels and comparing it with the initial representation by evaluating it on a downstream task . Unlike generic representations that use no label , this method has the advantage of guiding the extracted representations by using the non-expert initial labels . To validate this method , we apply it to dubbing and more specifically to the extraction of characteristics from actors ’ voices . We therefore p-vector features from characters played in a video game . We apply a clustering algorithm , k-means , in the p-vector space to extract refined labels . In order to evaluate the quality of the refined labels , we train a neural network to extract refined p-vectors . We then subject them to a voice character similarity task to compare them with the initial p-vectors and thus measure the loss or gain of dedicated character information . In this experiment , we test the following hypothesis : during the clustering step , using the recordings of the training corpus only highlights the initial labels , the characters . During this step , a second corpus ( close to the domain ) is needed to highlight the vocal features extracted by the p-vectors . Since distance plays an essen- tial role in the k-means algorithm , we also explore the use of cosine and Mahalanobis distances to compare with Euclidean one . Section 2 presents Label Refining . We define the term “ refined label ” and present the method . The experimental protocol to apply Label Refining is described on Section 3 . In Section 4 , we present experiments and discuss about the obtained results . For reproducibility reasons , scripts and models are available on GitHub1 . Finally , we conclude by presenting some takeaways and possible directions for future work in Section 5 . 2 LABEL REFINING . Before experimenting , we first introduce the Label Refining method . It consists in extracting refined labels from initial ones . In this section , we give a definition of refined labels and we detail the method of extraction and evaluation . 2.1 THE DEFINITION OF REFINED LABELS . In this work , we assume that labels are decomposable into various lower-level sublabels . In our case , we wish to use the knowledge provided by the played character labels associated to the recordings with the aim to infer the characteristics such that the emotion played ( e.g . agressive , happy ) , a voice color ( e.g . white , red ) or a timbre ( e.g . husky , enlightened ) . Equation 1 formalizes the definition of what a refined label is . Let R the set of all the refined labels and a ( l ) i ∈ { r1 , r2 ... r|R| } the ith refined label associated to the initial label l. Let us also specify that the number n of refined labels is different depending on l. Thus , from the knowledge given by initial labels in L we wish to find the refined labels a ( l ) i of each l. l a ( l ) 1 a ( l ) 2 ... a ( l ) n ( 1 ) Let us note the functions f : e 7→ l and g : e 7→ r where o is an observation ( e.g . vocal recording ) . The equation 2 gives fictives examples . It shows the expected images of the functions f and g according to two initial labels l1 and l2 . In our model of the problem , we assume that a refined label r may appear in observations associated to different initial labels . Here , this is the case with r1 which appears both in an observation associated to l1 and an observation associated to l2 . l1 { f : o ( l1 ) 1 7→ l1 f : o ( l1 ) 2 7→ l1 , l1 { g : o ( l1 ) 1 7→ r1 g : o ( l1 ) 2 7→ r2 , l2 { f : o ( l2 ) 1 7→ l2 f : o ( l2 ) 2 7→ l2 , l2 { g : o ( l2 ) 1 7→ r1 g : o ( l2 ) 2 7→ r3 ( 2 ) 2.2 OUR PROPOSITION TO REFINE LABELS . Figure 1 presents the refined label extraction method that we propose in three steps . 1 consists in training a system to extract a representation ( i.e . vector , hash Roussev ( 2011 ) ) of the observation o ∈ O using the initial labels l ∈ L ( e.g . label of the character played in a recording ) . 2 consists in using a clustering algorithm in order to associate new labels r ∈ R to the observations O . 3 consists finally in training an new system to extract refined labels by using labels r. In order to evaluate the relevance of labels r , we use two methods . The first method , taking place at step 2 , consists in applying different clustering quality measures such as v-measure Rosenberg & Hirschberg ( 2007 ) or the purity Ajmera et al . ( 2002 ) . Applied to the step 1 and step 3 , the second method consists in extracting a representation of the observations O and evaluating it on 1To keep the identity of the authors secret for the reviewers , the GitHub URL will be added if the paper is accepted . a downstream task . The downstream task can be diverse : speaker recognition , speech recognition or , in our case , character voice similarity . Thanks to the second method , it is possible to highlight the gain or loss of information by comparing the results obtained at steps 1 and 3 . In this work , we postulate that the refined labels are pertinent if they do not lead to a loss of information when extracting refined representation . Depending on the clustering algorithm , it is likely to have to try different values of k. In the following experiments , we use k-means and test different values of k. In this context , we apply the method for all the k values and compare the obtained results . We postulate that Label Refining is validated if the number of refined labels k differs from the number of characters and if the character information is at least preserved in the refined representation .

Summary: This paper addresses the task of finding similar-sound voices with application to voice-dubbing (e.g. finding an actor to record dialog in English, translated from original French dialog, such that the English speaker sounds similar to the original French speaker). The paper proposes a method called "label refining". This method is based on p-vectors, a prior representation found to model similarity between characters. The method uses 1) non-expert "initial labels" to train a p-vector system 2) use k-means to cluster the resulting p-vectors and 3) use the groups learned by k-means to re-group the labels (the "refined labels"). To evaluate the method, English-French pairs of voice data from the video games Mass-Effect 3 and Skyrim are used. Performance is measured using clustering measures v-measure and purity-K, as well as accuracy on the test set (which it seems the ground-truth labels are mapping to the correct dubbing speaker). The method achieves a higher accuracy of 0.70 over a state-of-the-art system that achieves 0.69. A second experiment is performed using Skyrim data as a subsidiary corpus used to cluster, with the goal of "bringing out vocal characteristics" from the initial labels, which shifts the optimal K towards 2, which the paper claims is because the new representations start to model gender.