Label Refining: a semi-supervised method to extract voice characteristics without ground truth

1 INTRODUCTION . Localization of a multimedia work , such as a video game or a motion picture , consists in making changes and adaptations to the original work so that international distributors can market the resulting product in a target country . For this purpose , companies need to translate their works into a target language . Various options are available to them to give substance to their translation . Subtitling is the simplest and cheapest option , but not the most practical one for spectators : a large audience is more comfortable with hearing speech , in general in their mother tongue , rather than reading subtitles while hearing speech in another language . Dubbing , which is more expensive and takes longer to set up , better retains the audience ’ s immersion by replacing the original voices with dubbing comedian voices . It involves a voice selection process , which means choosing among several candidate voices in a target language . This selection is referred to as voice casting . With the emergence of streaming platforms ( e.g . Netflix , Disney+ , Amazon Prime Video ) and the rapid development of the video game industry , the number of works to be localized is increasing dramatically . As a result , more and more actors are available on the voice market . Vocal Casting is an artisanal process that requires a lot of precision and is carried out by human operators . As it is , it can not be applied to a very large number of actors . Operators very often rely on their memory , which is limited to a small number of actors with whom they have worked and in whom they have confidence . However , this operation can degrade the immersion of the audience since the most oftused voices become easily recognizable . They then become references and create cognitive biases in the public ’ s perception of the character . Automatic actor search and recommendation tools , based on voice processing , would help operators find new actors who would then enrich the vocal diversity of the works to the delight of the audience . In this article , we wish to address some of the complexity of voice casting and we start with the voices of professional actors playing characters for the gaming industry . The articles Gresse et al . ( 2020 ; 2019 ; 2017 ) ; Malik & Nguyen ( 2021 ) ; Obin & Roebel ( 2016 ) ; Obin et al . ( 2014 ) ; Quillot et al . ( 2021b ; a ) cited in this paper are the only ones dealing with the speech-processing-based automation of Voice Casting in the literature to our knowledge . We rely on Gresse et al . ( 2020 ) , a recent work which defines the p-vector , a neural-network-based representation of the voice dedicated to the character aspects in acted voices . The application context in Gresse et al . ( 2020 ) ( like in related previous works Gresse et al . ( 2017 ; 2019 ) ) is voice casting for audio dubbing . The final objective of this work is to assess how close an actor – acting in a target language – is to the voice of a given character , speaking in a source language . This can be referred to as character-based voice similarity . In contrast to Obin et al . ( 2014 ) ; Obin & Roebel ( 2016 ) , which proposed a voice similarity system for the acted voice based on data labeled with speech classes ( age , gender , emotions ... ) by an expert , the p-vector approach does not use any expert labels . This representation is learnt from audio associated with the played character label , without any additional information . Also , the authors of Quillot et al . ( 2021a ) recently posed the following hypothesis : when a character is played , the actor adds to the voice some information that characterizes the character and is independent of the speaker identity . They then introduced SICI ( Speaker-Independent Character Information ) and showed that there is speaker-independent character information in the acted voice and that it is present in p-vectors pretrained on a speaker task Quillot et al . ( 2021b ) . Although these approaches deal with final works ( video games ) , another one consists in working directly on the decision data from Artistic Directors . Recently , researchers from Warner Bros. evaluated their models on this kind of data . Unfortunately , these data are sensitive and their acquisition is not trivial since it requires to work on the critical voice casting process . This is why we decided to position ourselves in a task quite different from Malik & Nguyen ( 2021 ) , since we do not use Artistic Director decisions . Voice similarity systems alone are not enough for Voice Casting operators . They do not trust the computed recommendations and need a more complete interface to interact with the system . Many works propose explainability approaches Došilović et al . ( 2018 ) but these are too complicated to be understood by a user who is not expert in computer science , and more particularly in machine learning . We strongly believe that the introduction of explanatory labels will help the operators understand the decisions by giving , for example , the list of speech characteristics shared by two recordings/actors . Such labels could also be used to express queries in voice search systems . Unfortunately , voice casting is an underdefined task Bonastre ( 2019 ) . The operators themselves can not formally define their decision criteria . There is also no standard taxonomy in cinema to describe voices , making it impossible to create an annotation protocol . The question is then : how can we extract features from voices when we have no a priori knowledge of them ? Different representation learning approaches exist in the literature , but none of them corresponds to our expectations . 1 ) Feature extraction surveyed in Bengio et al . ( 2014 ) consists in representing an observation by a vector of reals but not in the form of binary vectors that we call characteristics in this paper . 2 ) Supervised methods like Hu et al . ( 2017 ) require labels that we do not have access to in order to extract characteristics . 3 ) Unsupervised methods like Peri et al . ( 2020 ) propose generic representations from which we could not extract characteristics dedicated to the domain of acted voice and character identity . Hence , to address the lack of method to extract characteristics without ground truth , this article proposes a totally new approach entitled Label Refining . It stats with training a representation extractor using non-expert initial labels . A clustering algorithm is then employed with the purpose to emphasizing characteristics shared by groups of observations which we term refined labels . A last step consists in extracting a new representation from refined labels and comparing it with the initial representation by evaluating it on a downstream task . Unlike generic representations that use no label , this method has the advantage of guiding the extracted representations by using the non-expert initial labels . To validate this method , we apply it to dubbing and more specifically to the extraction of characteristics from actors ’ voices . We therefore p-vector features from characters played in a video game . We apply a clustering algorithm , k-means , in the p-vector space to extract refined labels . In order to evaluate the quality of the refined labels , we train a neural network to extract refined p-vectors . We then subject them to a voice character similarity task to compare them with the initial p-vectors and thus measure the loss or gain of dedicated character information . In this experiment , we test the following hypothesis : during the clustering step , using the recordings of the training corpus only highlights the initial labels , the characters . During this step , a second corpus ( close to the domain ) is needed to highlight the vocal features extracted by the p-vectors . Since distance plays an essen- tial role in the k-means algorithm , we also explore the use of cosine and Mahalanobis distances to compare with Euclidean one . Section 2 presents Label Refining . We define the term “ refined label ” and present the method . The experimental protocol to apply Label Refining is described on Section 3 . In Section 4 , we present experiments and discuss about the obtained results . For reproducibility reasons , scripts and models are available on GitHub1 . Finally , we conclude by presenting some takeaways and possible directions for future work in Section 5 . 2 LABEL REFINING . Before experimenting , we first introduce the Label Refining method . It consists in extracting refined labels from initial ones . In this section , we give a definition of refined labels and we detail the method of extraction and evaluation . 2.1 THE DEFINITION OF REFINED LABELS . In this work , we assume that labels are decomposable into various lower-level sublabels . In our case , we wish to use the knowledge provided by the played character labels associated to the recordings with the aim to infer the characteristics such that the emotion played ( e.g . agressive , happy ) , a voice color ( e.g . white , red ) or a timbre ( e.g . husky , enlightened ) . Equation 1 formalizes the definition of what a refined label is . Let R the set of all the refined labels and a ( l ) i ∈ { r1 , r2 ... r|R| } the ith refined label associated to the initial label l. Let us also specify that the number n of refined labels is different depending on l. Thus , from the knowledge given by initial labels in L we wish to find the refined labels a ( l ) i of each l. l a ( l ) 1 a ( l ) 2 ... a ( l ) n ( 1 ) Let us note the functions f : e 7→ l and g : e 7→ r where o is an observation ( e.g . vocal recording ) . The equation 2 gives fictives examples . It shows the expected images of the functions f and g according to two initial labels l1 and l2 . In our model of the problem , we assume that a refined label r may appear in observations associated to different initial labels . Here , this is the case with r1 which appears both in an observation associated to l1 and an observation associated to l2 . l1 { f : o ( l1 ) 1 7→ l1 f : o ( l1 ) 2 7→ l1 , l1 { g : o ( l1 ) 1 7→ r1 g : o ( l1 ) 2 7→ r2 , l2 { f : o ( l2 ) 1 7→ l2 f : o ( l2 ) 2 7→ l2 , l2 { g : o ( l2 ) 1 7→ r1 g : o ( l2 ) 2 7→ r3 ( 2 ) 2.2 OUR PROPOSITION TO REFINE LABELS . Figure 1 presents the refined label extraction method that we propose in three steps . 1 consists in training a system to extract a representation ( i.e . vector , hash Roussev ( 2011 ) ) of the observation o ∈ O using the initial labels l ∈ L ( e.g . label of the character played in a recording ) . 2 consists in using a clustering algorithm in order to associate new labels r ∈ R to the observations O . 3 consists finally in training an new system to extract refined labels by using labels r. In order to evaluate the relevance of labels r , we use two methods . The first method , taking place at step 2 , consists in applying different clustering quality measures such as v-measure Rosenberg & Hirschberg ( 2007 ) or the purity Ajmera et al . ( 2002 ) . Applied to the step 1 and step 3 , the second method consists in extracting a representation of the observations O and evaluating it on 1To keep the identity of the authors secret for the reviewers , the GitHub URL will be added if the paper is accepted . a downstream task . The downstream task can be diverse : speaker recognition , speech recognition or , in our case , character voice similarity . Thanks to the second method , it is possible to highlight the gain or loss of information by comparing the results obtained at steps 1 and 3 . In this work , we postulate that the refined labels are pertinent if they do not lead to a loss of information when extracting refined representation . Depending on the clustering algorithm , it is likely to have to try different values of k. In the following experiments , we use k-means and test different values of k. In this context , we apply the method for all the k values and compare the obtained results . We postulate that Label Refining is validated if the number of refined labels k differs from the number of characters and if the character information is at least preserved in the refined representation .

Summary: The paper tackles a voice similarity system task for voice casting problem. The authors trained voice embedding network using voice character label and cluster the embedding features and used these clusters to train final voice embedding network. The introduction is well-written, however, the proposed method is known or marginal improvements from the existing technical skill in machine learning community (pseudo labeling with embedding feature clustering).