Label Refining: a semi-supervised method to extract voice characteristics without ground truth
Authors: Mathias Quillot
Richard Dufour
Jean-français Bonastre
Venue: ICLR 2022 Submitted (2022)
Keywords: characteristics
characteristics extraction
characteristics evaluation
voice characteristics
label refining
refined labels
semi-supervision

Abstract: A characteristic is a distinctive trait shared by a group of observations which may be used to identify them. In the context of voice casting for audiovisual productions, characteristic extraction has an important role since it can help explaining the decisions of a voice recommendation system, or give modalities to the user with the aim to express a voice search request. Unfortunately, the lack of standard taxonomy to describe comedian voices prevents the implementation of an annotation protocol. To address this problem, we propose a new semi-supervised learning method entitled Label Refining that consists in extracting refined labels (e.g. vocal characteristics) from known initial labels (e.g. character played in a recording). Our proposed method first suggests using a representation extractor based on the initial labels, then computing refined labels using a clustering algorithm to finally train a refined representation extractor. The method is validated by applying Label Refining on recordings from the video game MassEffect 3. Experiments show that, using a subsidiary corpus, it is possible to bring out interesting voice characteristics without any a priori knowledge.

Under review as a conference paper at ICLR 2022
LABEL REFINING: A SEMI- METHOD TO EXTRACT VOICE CHARACTERISTICS WITHOUT GROUND TRUTH
Anonymous authors Paper under double-bling review
ABSTRACT
A characteristic is a distinctive trait shared by a group observations which may be used to identify them. In the context of voice casting for audiovisual produc- tions, characteristic extraction has an important role since can help explaining the decisions of voice recommendation system, give modalities to the user with the aim to express a voice search request. Unfortunately, the lack of stan- dard taxonomy describe comedian voices prevents the implementation of annotation protocol. To address this problem, we propose a new uenissuperissed learning method entitled Label Refining that consists in extracting refined labels (e.g. vocal characteristics) from known initial labels (e.g. character played in a recording). Our proposed method first suggests using representation extractor based on the initial labels, then computing refined labels using a clustering algo- rithm to finally train refined representation extractor. The method validated by applying Label Refining recordings from the MassEffect 3 video game. Exper- iments show that, using a subsidiary corpus, it is possible bring out interesting voice characteristics without any priori knowledge.
INTRODUCTION
Localization of a multimedia work, such as a video game or motion picture, consists in making changes and adaptations to the original work so that international distributors can market the result- ing product in target country. For this purpose, companies need to translate their works into a target language. Various options are available to them to give substance to their translation. Subtitling is the simplest and cheapest option, but not the most practical one for spectators: a large audience is more comfortable with hearing speech, in general in their mother tongue, rather than reading subti- tles while hearing speech in another language. Dubbing, which is more expensive and takes longer to set up, better retains the audience's immersion by replacing the original voices with dubbing median voices. involves a voice selection process, which means choosing among several candidate voices in target language. This selection is referred as voice casting.
With the emergence of streaming platforms (e.g. Netflix, Disney+, Amazon Prime Video) and the rapid development of the video game industry, the number of works to be localized is increasing dramatically. As result, more and more actors are available on the voice market. Vocal Casting is an artisanal process that requires lot of precision and carried out by human operators. As is, it cannot be applied to very large number of actors. Operators very often rely on their memory, which is limited small number of actors with whom they have worked and in whom they have confidence. However, this operation can degrade the immersion of the audience since the most oft- used voices become easily recognizable. They then become references and create cognitive biases in the public's perception of the character. Automatic actor search and recommendation tools, based on voice processing, would help operators find new actors who would then enrich the vocal diversity of the works the delight of the audience.
In this article, we wish to address some of the complexity of voice casting and we start with the voices of professional actors playing characters for the gaming industry. The articles Gresse et al. 2200 2019 (217)) Malik & (guyen((02T)) (002T);(bin & Roebel Obin et al. (2014); 202Tb a) cited in this paper are the only ones dealing with the preech-pocesssin..e based automation
et al.

---

Under review as a conference paper at ICLR 2022
of Voice Casting in the literature to our knowledge. We rely on Gresse et al. (020), recent work which defines the p-vector, a neural-network-buseee representation of the voice dedicated to the character aspects in acted voices. The application context in (rresse et al. (2020) (like in related previous works Gresse et al. 2017 2019) is voice easting for audio dubbing. The final objective of this work is to assess how close actor acting in a target language is to the voice of a given character, speaking in a source language. This can be referred to as character-
voice similarity. In contrast Obin et al. 2014); Obin & Roebel (2016), which proposed voice similarity system for the acted voice based on data labeled with speech classes (age, gender, emotions... ...) by an expert, the p-vector approach does not use any expert labels. This representation learnt from audio associated with the played character label, without any additional information. Also, the authors of Quiillo et al. (021a)) recently posed the following hypothesis: when a character played, the actor adds to the voice some information that characterizes the character and is independent of the speaker identity. They then introduced SICI Indeeeeden Character Information) and showed that there is speaker-independent character information in the acted voice and that it is present in p-vectors pre- trained on a speaker task Quillot et al. 20211))
Although these approaches deal with final works (video games), another consists in working directly on the decision data from Artistic Directors. Recently, researchers from Warner Bros. eval- uated their models on this kind of data. Unfortunately, these data are sensitive and their acquisition is not trivial since it requires to work on the critical voice casting process. This is why we decided to position ourselves in a task quite different from Malik & Nguyen (202I), since we do not use Artistic Director decisions.
Voice similarity systems alone are not enough for Voice Casting operators. They do not trust the computed recommendations and need a more complete interface to interact with the system. Many works propose explainability approaches Došilović et al. (2018) but these are too complicated to be understood by a user who is not expert in computer science, and more particularly in machine learning. We strongly believe that the introduction of explanatory labels will help the operators understand the decisions by giving, for example, the list of speech characteristics shared by two recordings/actors. Such labels could also be used to express queries in voice search systems. Un- fortunately, voice casting is an underdefined task Bonastre (2019) The operators themselves cannot formally define their decision criteria. There is also no standard taxonomy in cinema to describe voices, making it impossible to create an annotation protocol. The question is then: how extract features from voices when we have no a priori knowledge of them? Different representa- tion learning approaches exist in the literature, but none of them corresponds to our expectations. 1) Feature extraction surveyed in [Bengio et al. (2014) consists in representing an observation by a vector of reals but not in the form of binary vectors that we call characteristics in this paper. 2) Supervised methods like Hu et al. (2017) require labels that we do not have access to in order to extract characteristics. 3) Unsupervised methods like Peri et al. (2020) propose generic represen- tations from which we could not extract characteristics dedicated to the domain of acted voice and character identity.
Hence, to address the lack of method to extract characteristics without ground truth, this article proposes a totally new approach entitled Label Refining. It stats with training a representation extractor using non-expert initial labels. A clustering algorithm is then employed with the purpose to emphasizing characteristics shared by groups of observations which we term refined labels. A last step consists in extracting new representation from refined labels and comparing it with the initial representation by evaluating it on a downstream task. Unlike generic representations that use no label, this method has the advantage of guiding the extracted representations by using the non-expert initial labels.
To validate this method, we apply it dubbing and more specifically the extraction of character- istics from actors' voices. We therefore p-vector features from characters played in a video game. We apply clustering algorithm, k-means, in the p-vector space extract refined labels. In order to evaluate the quality of the refined labels, we train a neural network to extract refined p-vectors. We then subject them a voice character similarity task compare them with the initial p-vectors and thus measure the loss or gain of dedicated character information. In this experiment, we test the following hypothesis: during the clustering step, using the recordings of the training corpus only highlights the initial labels, the characters. During this step, a second corpus (close to the domain) is needed to highlight the vocal features extracted by the p-vectors. Since distance plays an essen-
2

---

Under review as a conference paper at ICLR 2022
tial role in the k-means algorithm, also explore the use of cosine and Mahalanobis distances to compare with Euclidean one.
Section 2 presents Label Refining. We define the term "refined label" and present the method. The experimental protocol to apply Label Refining described on Section 3 Section 4 we present experiments and discuss about the obtained results. For reproducibility reasons, scripts and models are available on GitHut" Finally, we conclude by presenting some takeaways and possible directions for future work in Section 5
LABEL REFINING
Before experimenting, we first introduce the Label Refining method. It consists in extracting refined labels from initial ones. In this section, we give definition of refined labels and we detail the method of extraction and evaluation.
2.1 THE DEFINITION OF REFINED LABELS
In this work, we assume that labels are decomposable into various lower-level sublabels. In our case, we wish to use the knowledge provided by the played character labels associated to the recordings with the aim infer the characteristics such that the emotion played (e.g. agressive, happy), a voice color (e.g. white, red) or timbre (e.g. husky, enlightened).
Equation formalizes the definition of what a refined label is. Let R the set of all the refined labels and (1) E fri, (2..F() the tt refined label associated to the initial label I. Let us also specify that the number of refined labels is different depending on (. Thus, from the knowledge given by initial labels in we wish to find the refined labels a; of each
(1)
a2
(I)
(I)
(n
Let us note the functions f e ++ and g e H> r where is an observation (e.g. vocal recording). The equation / gives fictives examples. It shows the expected images of the functions / and according to two initial labels I and 12- In our model of the problem, we assume that a refined label may appear in observations associated different initial labels. Here, this is the case with which appears both in an observation associated to L and an observation associated to l2.
(1)
(1)
(12) 7 ffoo
(2)
h
f
(1)
44
h
g:
rı
.1
(12)
.12
g 0
(2)
+
ri
(2)
02
g
oz
02
l2
02
r3
2.2 OUR PROPOSITION TO REFINE LABELS
Figure presents the refined label extraction method that we propose three steps. 1 consists in training system to extract a representation (i.e. vector, hash Roussev (201T)) of the observation E using the initial labels L (e.g. label of the character played in a recording). ② consists in using clustering algorithm in order to associate new labels E R to the observations O. consists finally in training an new system to extract refined labels by using labels r.
methods. The first method, taking place at step (2). consists in applying different clustering quality measures such as v-measure Rosenberg & Hirschberg (2007) the purity Ajmera al. (2002). Applied to the step 1 and step (3) the second method consists in extracting a representation the observations and evaluating it on
In order to evaluate the relevance of labels r.
'To keep the identity of the authors secret for the reviewers, the GitHub URL will be added if the paper accepted.

---

Under review as a conference paper at ICLR 2022
Initial representation extractor
initial
Clustering Algorithm
each observation
Clustering
Refined representation extractor
R
represestation
Dowmatrmmm
Refined reprosenration
Figure 1: Refining process of L initial labels into R refined labels.
Neural Network
K-means
Neural Network
P(ll)) lel.
2
P(zlo) veR
Aostiiltt recording
3
pieetors
prwecoons
Character voice
Entropy. v-measure, purity
Character voice similazity
similarity
Figure 2: Refining process of character labels L into R refined labels applied on recordings
downstream task. The downstream task can be diverse: speaker recognition, speech recognition or, in our case, character voice similarity. Thanks the second method, it is possible to highlight the gain loss of information by comparing the results obtained at steps and 3. In this work, we postulate that the refined labels are pertinent they do not lead to a loss of information when extracting refined representation.
Depending on the clustering algorithm, it is likely to have to try different values of k. In the following experiments,
k-means and test different values of k. In this context, we apply the method for all the values and compare the obtained results. We postulate that Label Refining is validated the number of refined labels & differs from the number of characters and if the character information is at least preserved in the refined representation.
3 EXPERIMENTAL PROTOCOL
In the previous section, introduced the generic method for extracting refined labels. We describe this Section the protocol we follow to validate the method. We apply Label Refining on acted voice recordings in order to extract voice characteristics dedicated to the character played. A resume Each step is detailed on the next subsections.
of the protocol is presented on the figure 2
3.1 MAIN (MASSEFFECT) AND SECOND (SKYRIM) CORPORA
We use MassEffect data as the main corpora to train feature extractor and evaluate our system on downstream task. We also explore the use of second dataset from the video game Skyrim during
4

---

Under review as a conference paper at ICLR 2022
the second step (i.e. clustering algorithm) with the aim to emphasize voice characteristics models by previous trained p-vectors. This subsection present both MassEffect and Skyrim datasets.
3.1.1 MASSEFFECT, THE DATASET USED TO TRAIN AND EVALUATE P P-VECTORS
The main corpus is composed of voice recordings coming from the Mass-Effect 3 role-playing game. Contrary to movies, voices of video games present some particularities explained in Gresse et al. (2019) (radio effects are present in the original voices of our corpus). They are also easier to collect since they are separated from the ambient sounds even their final form, contrary the dubbed film archives. Originally released in English, the game has been translated and revoiced in several other languages. our experiments, we use the English and French versions of the audio sequences, representing about 7.5 hours of speech in each language. Segments (or recordings) are 3.5 seconds long on average. A character is then defined by a unique French-Engee Enach--Engiihh couple two distinct speakers. To avoid speaker identity biases, consider only small subset where are certain that none of the actors play more than one character. A single audio segment corresponds to a unique speaking slot from actor in particular language. We then apply filter that keeps only recordings longer than second. Finally, we only keep the 16 characters for which we have the largest number of recordings, Gresse
(2019) did.
In order to measure the ability of our systems to generalize, we propose a breaking down protocol. We only keep 12 characters. and their 24 corresponding speakers, for the training phase and we use the remaining 4 characters for the testing phase. We break the corpus down into training, validation and test sets. The train and val sets are composed of different recordings, but coming from the same 12 characters. To build the train, val and fest subsets, we randomly select for each character 144, 36 and 180 recordings respectively, while balancing the number of French and English recordings. We then have total of 1, 728 (train), 432 (val) and 720 (test) recordings. We name the three sets S..
3.1.2 SKYRIM, A SUBSIDIARY DATASET FOR EMPIIASIZING VOICE CHARACTERISTICS
Skyrim is also known video game. We use its recordings second corpus in addition to the one of MassEffect previously presented. This dataset can be viewed as a validation set. We propose to use this corpus to emphasize the characteristics of voices. Indeed, the validation set of MassEffect suffers from a lack of variability since it contains the same characters as the training set. Unfortu- nately, we do not have available MassEffect data with unknown characters add the validation set since they are used on the test. Thus, we assume that using Skyrim as second validation set will help to bring more variability and highlight voice characteristics extracted from the p-vectors trained on MassEffect. This subset is composed by 18, 250 recordings of different 32 characters. As on MassEffect, each recording played english is associate to a french one and vice-versa but we don't 't balance the number of recording by character.
3.2 EXTRACTING VOICE REPRESENTATION FROM CHARACTER LABELS
This part of the protocol corresponds the step described in the Figure In order to extract the pevectors, we follow scrupulously the best architecture described in Quillot et al. (2021b). Training this architecture need to follow two steps: 1) training speaker identification system following vevector architecture on Voxceleb corpora (Chung et (2018)) 2) replacing the output softmax of this system with p-vector layers, and training to recognise the characters of the MassEffect dataset.
3.2.1 REFINING LABELS WITH K-MEANS ALGORITHM
This part of the protocol corresponds to the step 2 described in the Figure In this work, we wish to validate method described in 2.2 For this purpose, we focus on the k-means algorithm. Unlike others, this algorithm is easier to comprehend. We use the implementation of the toolkit sklearn with the default parameters, namely: & means converge more quickly to the solution. The max number of iterations is 300 of non convergence. The algorithm is runned 10 times, then we select the clustering model with the best intertia.
initializer of the barycentres in order to
5

---

Under review as a conference paper at ICLR 2022
point of view
limg- f(C;
limg- S X(C;))
entropy
group
+00
0
homogeneity
group
group purity
group
completeness
class
0
class purity
class
u-measure
hybrid
purity-K
hybrid
Table 1: Functional study of clustering measures
3.3 EXTRACTING REFINED VOICE REPRESENTATION BY USING CLUSTERING GROUPS
This part of the protocol corresponds to the step 3 described in the Figure 2 algorithm is complete, we build for each k-value tecture as used for the p-vector. Instead of predicting the initial labels, this new network is trained using the clustering algorithm. After training, we can extract
Once the clustering neural network using the same neural archi-
predict the refined labels from its last layer representation of the provided record that we name p-refined-vectors. The used architecture exactly the same as for the step since this neural network is also firstly pre-trained on speaker identification task by using Voxceleb 2. Only the last layer of the p-vector network is changed to correspond to the number groups (of the refined labels r) instead of the number of character labels I.
3.4 EVALUATION
In this subsection, present the ways evalute the quality of the refining. The further one is computing clustering metrics. The second one is evaluating the representation extracted using the labels downstream task. We give all the details in this subsection.
3.4.1 COMPUTING CLUSTERING METRICS
To evaluate a clustering model, we use the following measures: entropy, y-measure Rosenberg & Hirschberg (2007), class purity, group purity, purity-K K|Ammera et al. (2002). These measures require to know for each entity in the database the group to which the algorithm has associated it as well as its original label. Since their formal definition are defined in the cited papers, only present below their meaning and behaviour.
These measures caleulated from two different perspectives. The entropy, homogeneity and group purity measures check whether each group is associated with entities belonging to the same class. They provide a group view. The completeness and class purity measures check if each class is associated to entities belonging to the same group. They provide a class view.
To analyze the behavior of these measures, let C clustering model learnt the observations X with parameter k. the a priori number of clusters. Let ff the function which measure the quality of the model C/ (e.g. entropy) on X. The table I shows the behaviour the measures by computing for each of them the limit of f fik + +oc et + 0.
These measures all have their meaning and are generally misinterpreted. This is particularly the case for the entropy applied to clustering algorithms. The latter is a measure similar to that of homogeneity that has been inverted. Although it helps compare two systems, has a major drawback. It is bounded between 0 and +oo. Interpreting the measure of a single model is therefore difficult unlike homogeneity which is bounded between and The latter thus gives a rate of homogeneity that entropy cannot give. Without going into details here, the group purity is measure very similar to the homogeneity.
going
The measures of completeness and class purity, in addition to being very similar, the antithesis of the previous measures. They are no longer measures of group homogeneity but measures of class homogeneity. The closer the value is 1. the more the class in question is observed in only one group. The closer the value is to O, the more the class in question observed in a large number of groups at similar quantities.

---

Under review as a conference paper at ICLR 2022
Hybrid measures are weighted averages of class and group measures. Their limitations in k + too and & - are not clear as they depend on the balance of power between homogeneity (group purity) and completeness (class purity).
Demonstrating the limits of each function is quite simple. For homogencity and group purity, it is maximal when the elements of each group belong to the same class. When we reachi: [X] each element is then assigned to a cluster and each cluster is assigned to an element. There then only one class for each group. The homogeneity is thus maximal, the group purity too. In the opposite case, when k = 1. all observations belong to the same group. The group is then maximally heterogeneous and the values of homogeneity and group purity then tend towards O. For completeness and class purity, the reasoning is similar and then becomes trivial.
3.4.2 EVALUATING VOICE REPRESENTATION
As described in Quillot et al. (2021b), evaluate p- vectors character voice similarity task using a pharacter-similarie measure based on a Siamese network. We generate Target trials composed of pairs of recordings belonging the same character and Non-target trials made up of recordings belonging to two different characters. To avoid any bias, the number of targets and non-targets balanced, as well the number of pairs between two actors. So, we generate 124, 416 (train), 7, 776 (val) and 64, 800 (test) trials using S, dataset (MassEffect). Skyrim is not used during this evaluation step. The threshold set posteriori the Equal Error Rate (EER) point. System performance is then expressed in terms of accuracy on the test.
4 EXPERIMENTS AND RESULTS
In the experiments we present in the following sections, we decided to test different values of l in order limit the number of calculations be performed. Several values seemed obvious test such as 12, 24, 4 and which are the number of characters and speakers respectively in the train and test. We also assume that the number of speech features we are looking for is higher than the number of characters our system has to learn to model. We therefore also tested the values 18, 32, 48 and 64 which are arbitrary values but greater than 12. On the other hand, we have also tested k = 2 and k = 6 as values less than 4 and 12 as counter-hypothesi..
With the aim compare results obtained on the refined p-vectors, we first evaluate original p- vectors. These p-vectors are trained by using Sn. They are evaluated by training 10 siamese neural networks and selecting the best on the validation. The best model gives us 0.69 of accuracy on the test. This corresponds to the actual state-of-art performance presented in Quillot et al. (2021b).
4.1 REFINING CHARACTER LABELS BY USING MASSEFFECT
In T we plot the observed metrics y-measure and purity-K on the training dataset of S, (MassEffect). We then train 10 siamese neural networks and select the best on the validation. Its accuracy score on the validation and the test are plotted on the last two lines. For the three metrics presented in the table, the higher the value, the better the system.
clustering models computed
On the validation set, the metrics (u-measure and purity-K) highlight the 12 characters contained in it and known to the system. Indeed, we naturally have higher scores for i = 12 with 0.91 for the o-measure and 0.90 for the pu.
On the test, we observe somewhat different phenomenon. Remember that the test is composed of 4 characters. If the points which are projections of the recordings into p-vector space were perfectly discriminated by the neural classification system, they would then be organized into groups of records belonging to the same character. We would therefore have stronger measure scores for A = 4. In our case, we can observe this phenomenon with loss due to the difficulty for the system to separate the records of characters that are unknown to it. Indeed, the highest values are at for the y-measure with 0.58 and at
2 for the purity with 0.70. These groups are finally the closest to and show a loss of quality of the representations on unknown data. However, they do not show, priori, the existence of features that could interest us.
7

---

Under review as a conference paper at ICLR 2022
Table 2: Results obtained for each & value by using the recordings of MassEffect for the whole label refining protocol. The e-measure and purity-K measures are computed on the clusters extracted from the p-vectors. Accuracies are computed on validation (siamese val) and test (siamese test) sets of S, by using the siamese networks trained with the refined vectors.
2
8
12
18
24
32
48
64
u-measure val
0.40
0.62
0.73
0.80
0.91
0.86
0.83
0.80
0.75
0.74
v-measure test
0.54
0.56
0.58
0.50
0.48
0.45
0.44
0.44
0.41
0.41
purity-K val
0.41
0.53
0.64
0.73
0.90
0.78
0.68
0.61
0.52
0.48
purity-K test
0.70
0.65
0.64
0.54
0.50
0.45
0.43
0.40
0.32
0.31
siamese val
0.67
0.73
0.80
0.87
0.94
0.92
0.93
0.93
0.92
0.93
siamese test
0.55
0.65
0.62
0.66
0.66
0.69
0.63
0.66
0.63
0.70
Finally, we are interested in the performances obtained on the Siamese networks that we have trained. Recall that refined p-vectors are provided as input to these Siamese networks which are trained on a character verification task - character voice similarity. These vectors are extracted from a neural network trained to recognize the clustering group (refined label) corresponding the given recording (observation). Without surprise, we observe best accuracy for & = 12. This corresponds to the number of characters. Contrary to the previous metrics, we do not observe a normal distribu- tion in the results. Indeed, when & is moving away from 12, the obtained accuracy is not necessarily lower. For instance, at = 64 the obtained accuracy is not significantly lower than the one obtained at & 12. Even more interesting, the accuracy obtained on the test subset is about 0.70 that is higher than the state-of-the-art system, 0.69.
4.2 REFINING CHARACTER LABELS BY USING SKYRIM AS SUBSIDIARY CORPUS
In second experiment, we wish to verify if possible to use a subsidiary corpus to extract voice characteristics. For this purpose, we propose this time to apply the clustering algorithm on the p- vectors representations extracted from the Skyrim corpus for step 2. The p-vectors have learned to highlight the played character but it is impossible to bring out the vocal characteristics it extracts to perform this classification. We hope that the use of a secondary corpus such as Skyrim will create enough variability to bring out vocal characteristics from the initial labels. For step 1, the p-vector extractor is exactly the same as the previous experiment. It is trained on MassEffect with S, and the Skyrim data is completely unknown to For step 2, we extract p-vectors using Skyrim data and perform on them the k-means algorithm. For step 3. we finally associate a cluster to each recording of MassEffect and train refined p-vectors as realized in the previous experiment.
Table 3 presents the obtained results. We note x as accuracy result when labels are missing when we associate them MassEffect data with the aim to train the step neural network. We can observe that this time the e-measure and the purity-K maximum scores move away from the number of characters that we originally had - 32 for Skyrim. So we know that the semantics of these clusters differ from the semantics of the character labels. In this example, purity-K highlights k = 2. In the = case, the two groupings primarily separate men and women. Each group therefore the gender characteristic of the voice. For the e-measure, the score correlates with the number & of groups The best score is therefore for & = 64, but it might have gone up by increasing k. Unlike k 2, it is difficult understand the semantics of each group and to associate words that make sense to a human operator. Note that these measures have no known bias unlike the previous experiment since here the number of characters in the Skyrim data is 32 and the k 2 and i 64 values are significantly off.
4.3 ASSESSING THE IMPACT OF THE DISTANCE SELECTED FOR K-MEANS
As we know, distance in the k-means algorithm plays an essential role. For this reason, we propose to realise the same experiment presented in the previous section (4.2) but using cosine distance instead of Euclidian one. We also use the Mahalanobis distance k-means presented in Lapidot 2018)
without constraints gain computing time.
The table 4 presents results obtained by using the cosine and the Mahalanobis distances. We can observe that the Mahalanobis distance offers superior results u-measure and purity-K compared to
8

---

Under review as a conference paper at ICLR 2022
Table Results obtained for each A value by replacing MassEffect with Skyrim recordings during the step 2. The v-measure and purity-K K measures are computed on the clusters extracted from the p-vectors. Accuracies are computed on validation (siamese val) and test (siamese test) sets of S, by using the siamese networks trained with the refined p- D-vectors.
4
12
18
24
32
48
64
v-measure
0.23
0.27
0.30
0.33
0.36
0.37
0.38
0.40
0.41
0.42
purity
0.26
0.23
0.23
0.24
0.24
0.24
0.23
0.23
0.22
0.22
siamese val
0.67
0.72
0.78
0.84
0.87
0.90
0.90
0.90
0.91
X
siamese test
0.50
0.72
0.65
0.63
0.67
0.62
0.66
0.66
0.62
X
Table 4: Results obtained for each A value by replacing MassEffect with Skyrim recordings during the step 2. Cosine and Mahalanobis distances are used instead of Euclidean. The u-measure and Wurity-K measures are computed on the clusters extracted from the p-vectors, Accuracies are com- puted on validation (siamese val) and test (siamese test) sets of S,, by using the siamese networks trained with the refined p- vectors.
4
8
12
18
24
32
48
64
cosine
distance
v-measure
0.23
0.27
0.30
0.32
0.35
0.37
0.38
0.39
0.41
0.41
purity-K
0.25
0.24
0.23
0.24
0.24
0.24
0.23
0.23
0.23
0.21
siamese val
0.70
0.71
0.78
0.84
0.88
0.91
0.90
0.90
X
X
siamese test
0.56
0.70
0.63
0.63
0.66
0.67
0.69
0.66
X
mahalanobis
distance
y-measure 0.22
0.30
0.31
0.36
0.40
0.43
0.47
0.52
0.54
0.55
purity-K K
0.26
0.25
0.24
0.26
0.29
0.30
0.32
0.36
0.36 X
0.35
siamese val
0.63
0.69
0.74
0.77
0.77
0.78
0.80
0.80
X
siamese test
0.59
0.59
0.56
0.61
0.54
0.45
0.55
0.64
X
the Euclidean distance and the cosine distance. Moreover, while the cosine distance tends to draw the same conclusions as the Euclidean distance through these measurements, the Mahalnobis distance differs in the number of clusters it highlights with the purity-K measure. Indeed, the two other distances put forward k = 2 while this one puts forward A = 32. The training of Siamese networks seems to agree with the choice k 32 for Mahalnobis. However, this accuracy is significantly lower than the one obtained with cosine distance, 0.64 < 0.67. Mahalanobis does not seem to fit the data so as to conserve character information.
5 CONCLUSION
This article introduces Label Refining, a semi-supervised method that consists in extracting refined labels from initial ones. It meets the needs of characteristics extraction, especially vocal, without a priori knowledge. To validate this method, we applied it to dubbing and more specifically to the extraction of characteristics from actors' voices. A first experiment showed that using the same data during all the steps of Label Refining will only highlight the initial labels. A second experiment showed that using subsidiary corpus during the clustering step permits to highlight refined labels. It might also be noted that, in one case, our method brought to light gender characteristics. In the other case, giving labels intelligible meaning required further analysis. last experiment showed that the Euclidean distance seems to fit the data better than cosine and Mahalanobis distances since it preserves the character information better.
With these experiments, we can conclude that Label Refining permits to extract refined labels that deviate from the original ones when using a subsidiary corpus. These labels conserve or bring the wanted information, namely character information in case. While it is not straightforward to give meaning to these characteristics without a perceptive study, we propose in future works to build explainability methods based on Label Refining, especially for voice similarity systems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
J. Ajmera, H. Bourlard, I. Lapidot, and I. McCowan. Unknown-multiple speaker clustering using hmm. In INTERSPEECH, 2002.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives, 2014.
Jean-François Bonastre. Representation learning for underdefined tasks. In Ingela Nyström, Yanio Hernández Heredia, and Vladimir Milián Núñez (eds.). Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 47, Cham, 2019. Springer International Publishing.
J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In Interspeech, 2018.
Filip Karlo Došilović, Mario Brčié, and Nikica Hlupić. Explainable artificial intelligence: A survey. In 2018 s1 International Convention on Information and Communication Technology, Electron- ics and Microelectronies (MIPRO), pp. 0210-0215, 2018. doi: 1..2391//IPRRO
Adrien Gresse, Mickael Rouvier, Richard Dufour, Vincent Labatut, and Jean-Francois Bonastre. Acoustic pairing of original and dubbed voices in the context video game localization. In Interspeech, 2017.
Adrien Gresse, Mathias Quillot, Richard Dufour, Vincent Labatut, and Jean-François Bonastre. Sim- ilarity metric based on siamese neural networks for voice casting. IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), 2019.
Adrien Gresse, Mathias Quillot, Richard Dufour, and Gean-Francçoi, Bonastre. Learning Voice Representation Using Knowledge Distillation For Automatic Voice Casting, In Interspeech, 2020.
Ping Hu, Dongqi Cai,, Shandong Wang, Anbang Yao, and Yurong Chen. Learning supervised scoring ensemble for emotion recognition in the wild. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI ' 17,, pp. 553-560, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 4781450355438. doi: 10. 10.1145/3366555.1443009
URL http:p::/a.....gg.....o1 11t.:1111111111111111 0111:755551111140 3143009
Itshak Lapidot. Convergence problems of mahalanobis distance-bart based k-means clustering. In 2018 IEEE International Conference on the Science of Electrical Engineering in Israel (ICSEE), pp.
1-5, 2018. doi: 10.1 201.111EEEEE......... 86446138.
Aansh Malik and Ha Nguyen. Exploring automated voice casting for content localization using deep learning. SMPTE Motion Imaging Journal, 130(3)112-18, 2021.
Nicolas Obin and Axel Roebel. Similarity search of acted voices for automatic voice casting. IEEE/ACM Transuctions on Audio, Speech, and Language Processing, 24:1642-1651, 2016.
Nicolas Obin, Axel Roebel, and Grégoire Bachman. On automatic voice casting for expressive speech: Speaker recognition vs. speech classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014.
Raghuveer Peri, Monisankha Pal, Arindam Jati, Krishna Somandepalli, and Shrikanth Narayanan. Robust speaker recognition using unsupervised adversarial invariance. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6614 6618, 2020. doi: 10.11099/AAS9994777.... 200.077...20090066 2020994560..
Mathias Quillot, Richard Dufour, and Jean- François Bonastre. Assessing acter information for acted voices. In Alexey Karpov and Rodmonga Potapova (eds.), Speech and Computer, pp. 565-576. Cham, 2021a. Springer International Publishing. ISBN 978-3-030- 87802-3.
char-
Mathias Quillot, Jarod Duret, Richard Dufour, Mickael Rouvier, and -rannFFaaçois Bonastre. Influ- ence of speaker pre-training on character voice representation. In Alexey Karpov and Rodmonga Potapova (eds.), Speech and Computer, pp. 577-588, Cham, 2021b. Springer International Pub- lishing.
10

---

Under review as a conference paper at ICLR 2022
Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy- -bsoed external cluster evaluation measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natu- ral Language Processing and Computational Natural Language Learning GEMNLP-CONLLS pp-
410-420, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL
https://aclaa wtw.:laat.ooooo.oooooooooo... l0g..00//0000100333
Vassil Roussev. An evaluation of forensic similarity hashes. S41, 2011. ISSN 1742-2876. doi: https///ooi.or//1000
Digital Investigation, 8:S34- URL https: The
/ /www. sccisceeeee encei rect.
Proceedings of the Eleventh Annual DERWS Conference.


Reviews:
[{"review_id": "sUS-gQhvhpm", "reviewer": "Reviewer_AFut", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This Labeling Refining described in this paper could be generally applicable in different areas. Overall this paper clearly presents the proposed approach.\n\nI find some experimental results require further clarification. E.g. \n1. For results in Table 2-4, is it possible to add standard deviation to show if improvements are significant? \n2. In the last paragraph on page 7, it's said best results are not for k=4 (test set has 4 characters though). Then how to validate the trained model is robust to unseen cases? \n3. I'm not very clear with what patterns could be identified for distance vs. k values vs. measures for results shown in Table 4? \n4. English written could be further improved, e.g. \"We therefore p-vector features\", \"In Table 2, we plot the observed metrics\" etc. Also in the first line below equation (1), should it be \"o\" instead of \"e\"?", "summary_of_the_paper": "In this paper authors proposed a semi-supervised learning method named Label Refining, which leveraged a clustering algorithm for computing refined labels from initial ones, for training representation extractors. The proposed approach was applied to improve characteristics extraction from actors' voices.", "summary_of_the_review": "Overall, I think this paper needs to further improve its experimental section, with more explanation for the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bi9w-rZGCs7", "reviewer": "Reviewer_eoTk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\nThe paper has an interesting angle on combining data-driven and explainable (accessible by humans) clustering. The application is interesting and novel.\nThe authors indicate they will release source code.\n\nWeaknesses\n----------------\nThe paper does not contain a sufficiently detailed explanation of the underlying algorithm, and how the parameters are learned. Most of the discussion around related work is on determining voice characteristics and measuring the quality of data-driven voice representations, there is little discussion of machine learning algorithms. For ICLR audiences, it would be beneficial to expand the discussion of related work in ML.\nIt would be useful to provide examples of the labels used, and how they get refined during the process. As written, the description of the paper is very abstract, but I have little intuition as to how the procedure works.\nThe paper compares the results to a previous paper (Quillot, 2021b), and states that the proposed method achieves SoTA performance. It would be useful to include an ablation study and discuss in more detail how the approaches taken in these two papers differ, and how the design choices influence performance. The paper presents the main experiments as a series of pragmatic choices, but I am not sure if the proposed architecture has already been applied to other problems, and/ or if other choices would yield better results?\nI am not sure if the data is available, so that the released source code could actually be run and the results could be reproduced?", "summary_of_the_paper": "The paper presents a \"label-refining\" technique, which helps users pick voices to provide a better user experience in dubbed video games. The idea is that a voice talent's voice in the target language should match the character's voice characteristics in the source language. The method seems to work by attaching labels to data-driven clusters, and refining these using a second corpus, on which the labels' value for discrimination is measured.", "summary_of_the_review": "The paper presents an interesting and well-justified task, but in the present form it is most suitable to a domain specific (applied) venue, rather than a more general conference such as ICLR. Because of missing comparisons and ablations, there is little information in the paper that I am confident can be generalized to other tasks and/ or problems, so I am not sure the paper will be valuable to someone not working in that field. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_-vMCp9tnCL", "reviewer": "Reviewer_dJJ2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\nS1) The approach is simple, and seems to improve accuracy a bit over state of the art.\n\n\n### Weaknesses:\n\nW1) Overall I found the paper rather difficult to read. Besides some typos and opportunities for English usage improvement (see my comments below), I think the description of the type of information that the p-vectors carry is rather vague. In the introduction, the method is motivated by \"[Voice Casting operators] do not trust the computed recommendations and need a more complete interface to interact with the system.\" But I don't really see how the provided approach solves this problem, because the proposed approach is still just extracting vectors and are fed to a machine learning classifier, and are thus not really any more interpretable.\n\nW2) I'm still a little unclear what the \"non-expert initial labels\" are here. Are these some suggested pairs of speakers that are judged to be compatible in terms of sounding similar? This needs to be made more clear. Also, if it's indeed the \"unique French-English couple\" that provides these initial labels, isn't this quite subjective, since it's a decision the voice casting directors made, and doesn't necessarily imply that the voices are actually similar?\n\nW3) I'm not quite sure what conclusion is being drawn from the second experiment where Skyrim data is used in the clustering. It seems to be that the preference towards K=2 suggests a separation in terms of gender? \n\nW4) I think this paper would be improved by actually measuring the outcome of the task of interest by doing an evaluation with human subjects. E.g. if humans are presented with a suggested dubbing actor for an original utterance, does this method actually improve over prior approaches in terms of presenting similar voices that are suitable for dubbing?\n\n\n### Minor comments\n\nM1) \"to describe comedian voices\": comedian? Do you mean actor?\n\nM2) \"Vocal Casting\": don't capitalize.\n\nM3) \"It stats\" -> \"It starts\"\n\nM4) \"We therefore p-vector features\": missing words?\n\nM5) \"a voice color (e.g. white, red)\": this seems a little insensitive, if referring to race\n\nM6) \"fictives examples\": maybe \"illustrative examples\"?\n\nM7) \"f : e → l and g : e → r\": should this be \"f : o → l and g: o → r\"?\n\nM8) \"A resume of the protocol\" -> \"A summary of the protocol\"\n\nM9) \"We propose a breaking down protocol\": probably a better word choice than \"breaking down\"\n\nM10) \"Skyrim is also a known video game\": known to whom? Do you mean to say a popular video game?\n\nM11) Capitalize \"English\" and \"French\" everywhere.", "summary_of_the_paper": "This paper addresses the task of finding similar-sound voices with application to voice-dubbing (e.g. finding an actor to record dialog in English, translated from original French dialog, such that the English speaker sounds similar to the original French speaker). The paper proposes a method called \"label refining\". This method is based on p-vectors, a prior representation found to model similarity between characters. The method uses 1) non-expert \"initial labels\" to train a p-vector system 2) use k-means to cluster the resulting p-vectors and 3) use the groups learned by k-means to re-group the labels (the \"refined labels\"). To evaluate the method, English-French pairs of voice data from the video games Mass-Effect 3 and Skyrim are used. Performance is measured using clustering measures v-measure and purity-K, as well as accuracy on the test set (which it seems the ground-truth labels are mapping to the correct dubbing speaker). The method achieves a higher accuracy of 0.70 over a state-of-the-art system that achieves 0.69. A second experiment is performed using Skyrim data as a subsidiary corpus used to cluster, with the goal of \"bringing out vocal characteristics\" from the initial labels, which shifts the optimal K towards 2, which the paper claims is because the new representations start to model gender.", "summary_of_the_review": "Overall, I recommend rejection. I didn't find the paper very clear, and the conclusions are vague and not well-supported by the experiments. The novelty seems relatively minor, since this is using the p-vectors defined before, and just adding a clustering step to their training.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GOQ3JsOPed", "reviewer": "Reviewer_acRU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper proposes to start with initial labels and then use k-means clustering to refine labels. Overall, I do not find the approach to be novel. Its also not clear what it is really trying to achieve by this one step of refinement. \n\n2. The paper talks about voice similarity, dubbing/voice-casting etc. but this label refinement is never really used for these tasks in the experiments. The paper merely studies the clustering through different measures and its not clear to me what the end goal here is ? The analysis of clustering offers very limited insights. In the case of the MassEffect dataset, the clustering ends up highlighting initial labels  (expected) but when another dataset is used it ends up highlighting refined labels. These seem kind of obvious conclusions ?? \nThe paper does not offer any interesting insights on the methods as well as on the empirical end. \n\n3. Is it possible to do this refinement in an iterative manner ? You started with some initial labels and try to iteratively refine the labels. \n\n4. There are some recent self-supervised speech representation learning approaches such as Hubert [R1]. Authors should take a look at this paper and see if they might benefit from it as it appears to be related to it. \n\n5. Is it possible to do experiments on how this approach can improve voice casting ? \n\n6. There are quite a few typos and writing errors. Please proof-read the paper. Overall, the paper is quite difficult to read. \n\nR1: HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units", "summary_of_the_paper": "The paper proposes a label refinement approach. Starting from an initial set of labels, k-means clustering is used to refine the labels. The approach is described in the context of dubbing/voice casting. The method tries to obtain voice characteristics which can be further used for dubbing/voice-casting. Experiments are shown on video game datasets MassEffect and Skyrim and they primarily investigate how different parameters of k-means (# of clusters) and distance measure affect the label refinement. ", "summary_of_the_review": "Overall, I do not find the paper interesting. The approach is not particularly novel and it is not clear how it benefits the end goal. The experiments are very shallow and do not provide very interesting insights. The paper lacks strength on both methodology as well as experiments. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2upBKtZ0CX8", "reviewer": "Reviewer_LKP9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pseudo labeling or iterative training with embedding feature clustering might be of related works.\n\nAlso, the author argues that the explainability of the voice recommendation system is important, however, the proposed model cannot give the controllability to the user.\n\nBackbone network is not well explained.\n\nLoss function of the siamese network used in the paper is not specified.\n\nTraining details are missing.", "summary_of_the_paper": "The paper tackles a voice similarity system task for voice casting problem. The authors trained voice embedding network using voice character label and cluster the embedding features and used these clusters to train final voice embedding network. The introduction is well-written, however, the proposed method is known or marginal improvements from the existing technical skill in machine learning community (pseudo labeling with embedding feature clustering).", "summary_of_the_review": "The problem definition seems interesting and the application the authors tackling (voice search system for voice casting) sounds interesting (introduction is well-written), however, the proposed method seems not enough novel and some details of the experimental setup is missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates a semi-supervised label refining approach to searching for similar voices for voice-dubbing.  The apporach is based on generating refined labels using a clustering algorithm on the initial labels.  Therefore, better voice characteristics can be extracted and used to select a new voice in the target language that closely matches the voice characteristics of the source language.  Experiments are carried out on MassEffect as the main dataset and Skyrim as the second dataset and results show that the proposed approach slightly outperforms state of the art.  While the topic under investigation is interesting and has its value to the applications such as voice casting,  there are strong concerns raised by the reviewers.  Reviewers find the paper difficult to follow.  Some important pieces of information are either missing or only vaguely explained (e.g. non-expert initial labels,  clear interpretation of p-vectors, etc.), which greatly hinders a deep understanding of the work.  Some technical details such as network architecture and its training should be elaborated.  This paper needs some good improvement in order to get accepted.  No rebuttal is provided by the authors so all these concerns still stand.
Decision: Reject