Venue: Interspeech 2023 Workshop SSW

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a proof of concept kind of study about the application of a Federated Learning strategy for Voice Conversion. 

The motivation is to evaluate a proof of concept of a VC system that will progressively learn from user's data by keeping the privacy of the data (two ways approach) at difference of a conventional approach where a developer provides user with a fixed VC system that would present particular challenges in terms of user's data privacy for model update.

The authors selected StarGANv2-VC as the VC system. The authors provided a detailed description of it including individual models and objective functions (this might not be necessary).

The authors then present the Federated Learning algorithmic strategy, no claims are presented/justified regarding the novelty of it.

It is mentioned that overfitting risks are possible since the the clients (users) data may not observe independent and identically distributed conditions and a regularization feature called FedProx is proposed to alleviate this potential issue. 

Data from 40 speakers was used for the experimental evaluation. 10 were used as "anchor" speakers and the remaining 30 were used as "client" speakers. These number of speakers may not be sufficient for general performance observations. Additionally, since the FL setup based on a virtual server and client resources from the same computer the evaluation scope of the Federated Learning environment seems to be very restricted as well. Only a small number of clients was simulated (three, maximum).

The authors experimentally compared the proposed Federated Learning approach against a conventional setup using the same baseline VC system. The results in terms of objective cosine similarity and subjective similarity and naturalness show comparable performance. 

The study overall suggests somehow the applicability of Federated Learning for non-parallel voice conversion but a larger study seems necessary to establish solid performance and computationally efficiency observations. 
review_title: Proof of concept, very limited contributions 

final_score: rating: 7: Good paper, accept
review_content: Strengths:

Fairly novel study on the application of federated learning for the purpose of distributed customer-privacy preserving VC technologies that learns with time. Even if the VC system isn't novel that doesn't diminish the impact and value of this research.

Improvement areas:

Section 2 is over-formulated in my oppinion. While it is good to be comprehensive, half a page of formulas that can be consulted in a different paper doesn't add to the content of the paper and it is hard to justify given the limited space and other gaps in the paper.

With the statistics presented in Tables 1 and 2, is it possible to claim statistical significance? (as a minor comment, I found it weird personally to mention mean+-std and not mean+-confidence margin). This is alleviated by having the preference tests in 4.3 though.

Comments/questions

How can we guarantee that we are protecting the privacy of the customer at inference time in this situation? In a 'cyberattack' situation in which the client can be forced to generate any speaker embedding at conversion time, would it be possible to make sure that customer voices aren't recoverable in thsi framework, and if so, how? Otherwise, it would be better to call out that what we are maintaining is the privacy of the speech recordings and not of the biometric recoverability.

There are grammatical issues here and there, but there was one that was quite distractign as it is not clear what it means and it breaks the claims in the conclusion: 'our proposed method achieved the speaker similarity the conventional method' what was achieved, better similarity? the same?
review_title: This paper presents one of the first studies on the application of federated learning for the purpose of improving voice conversion quality for new speakers in a way that doesn't involve breaching privacy or fully retraining models. Results show that there is an improvement on the achievable speaker similarity thanks to the distributed learning, and we can expect follow up work based on this preliminary study to polish the idea further.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: -Key Strength of the paper

A VC architecture is proposed based on a federated learning scheme, in which updates of local models are made on distributed clients then aggregated in a single model to the server without requiring to collect the data of the speakers.

Two experiments are conducted, both objectively by monitoring the x-vector cosine similarity, and subjectively by conducting a similarity and a naturalness tests.

-Main Weakness of the paper

There is an inconsistency between the space used to describe the original starGANv2-VC and for the description of the proposed federated learning scheme.

For the objective experiment, why don't you provide the the cosine similarity mesures for the baseline starGANv2-VC? 

It is not clear to me what assumptions the authors make on their proposed FL-VC? The main point is for me the fact that it does not require to centralise the data to update the VC. What are the reasons why the similarity or the similarity could improve?

In particular, I wonder whether the experimental results are not biased by the relatively small number of speakers used. The observations that are made in Section 4.2 may not be valid anymore with an increased number of anchor speakers (for instance 100 or 1,000). Also, the maximum number of clients used in the experiments is relatively small (3)

-Novelty/Originality, taking into account the relevance of the work for the SSW audience

Federated VC is certainly original and relevant, especially regard the issue of voice and data privacy

-Technical Correctness, is the work technically and/or scientifically solid? Are sufficient details provided to allow any experiments to be reproduced or equivalent experiments run?

The authors provide statistical significancy tests, but there is no explanation about how it is actually processed.
review_title: Federated Learning for Human-in-the-Loop Many-to-Many Voice Conversion

Meta-review: SSW2003 received 45 papers. The acceptance rate is 82%. We are pleased to inform you that your paper has been accepted by the SSW2023 Program Committee. Please read the reviews carefully and submit your camera-ready paper by June 28th. Most reviewers performed a detailed review. Please answer to their questions and consider their comments. Note that camera-ready papers are credited with one extra page to allow authors to consider reviewers’ suggestions. So max 7 pages in total including figures & refs.
The deadline for submitting the revised version (with full non-anonymized authors and refs!) is 28th June.

Decision: Accept