Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method to select a group of pretext tasks from a given set of tasks for optimising the network training during self-supervised learning phase.
The weights for the given set of tasks are learned based on Hilbert Schmidt Independence Criterion (HSIC) using a few data samples. Using 2 downstream tasks, the authors show that their approach could benefit the learning of features relevant for the downstream tasks. Thus, improves the accuracy of these downstream tasks.
main_review: Strengths:
- the proposed method for selecting pretext tasks is interesting 
- using this approach the authors show that self-supervised learning could benefit the fine-tuning for the downstream task

Weaknesses: 
- evaluation for the claims are very weak. Authors show, for example, that using a subset of pretext tasks benefits the accuracy of ASR model. However, the experiments are limited to two downstream tasks. Also, results do not show the impact of picking other subsets which are not strongly related to the downstream task.
- could this approach be applied to other domains? or is it just limited to speech? 
- comparison with wav2vec-2.0 is weak, can this approach beat the SOTA? 
- are the learned features generable? or they fail when the downstream task is changed? 
- how much data is required to learn the weights with HSIC? 


summary_of_the_review: Overall, the approach is interesting and could benefit the training during self-supervised learning phase. However, the evaluation requires a significant amount of ablation studies to prove the claims about the effectiveness of this approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes authors' proposal for selecting pretext tasks (pseudo-labels) for multitask self-supervised learning, to improve self-supervised learning for speech representations. It leveraged the Hilbert Schmidt Independence Criterion (HSIC) for selecting pseudo-label loss weights and used softmax and its sparsity version to realize it. Experimental results on speech, speaker and emotion recognition showed effectiveness of the proposed approach.
main_review: This paper proposes a new approach to combine multiple tasks during pre-training for speech representation learning, and it leads to improved performance for downstream tasks including speech, speaker and emotion recognition. Overall, this paper is clearly written. 

One question I think this paper needs to address is how strong baselines are. Though authors mention not to directly compare with downstream tasks' performance reported in literature, it still helps to provide some justification to show the numbers reported in Table 1 & 2 are improvements over strong baselines. Also authors should add error bar to show if difference is statistically significant.
summary_of_the_review: The proposed method is technically sound, though additional justification is needed to for experimental results reported in this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to improve self-supervised (SSL) speech representations by determining what linear combination of SSL targets will perform best for downstream tasks (audio SSL targets like fundamental frequency or MFCC). The technique learns to weight these targets according to a Conditional Independence criteria: a target is more valuable if the SSL target is more independent of the data given a label. 

There are two main experimental results on which the usefulness of this work rests. The first is a “sanity check”, which demonstrates that this weighting scheme outperforms two previous ones, from 2002 and 2005, on two tasks: ASR in LibriSpeech and speaker ID in Voxceleb1.  The second is that they manage to improve LibriSpeech, Voxceleb, and IEMOCAP performance over a wav2vec2 model by adding SSL targets weighted according to their algorithm.
main_review: Strengths:
1) This topic is well timed. Currently, contrastive losses are the best performing, but having SSL targets from a multi-task scenario is also promising. It would be incredibly useful to the community if the weighting of many SSL targets was important, and one could be found to make multi-task training paradigms competitive.
2) The theoretical result seems interesting. I have not seen Conditional Independence be used as a criteria for weighting SSL targets. If useful, this would be a new way of thinking about tasks in a multi-task training scheme.

Weaknesses:
1) Empirical results: 
1a) I worry about the comparison in Table 2. To support the claim that the authors’ algorithm is useful, they need to show that their algorithm also improves on wav2vec + naive mixing of multitask SSL. Table #2, as it currently stands, only shows that additional targets can be useful. It does not seem to support the argument that their mixing algorithm was essential to the improvement.
1b) I worry about the presence of IEMOCAP in experiment #2 but its absence in experiment #1. If the algorithm is as general as is claimed in the paper, I would expect feature selection improvements on IEMOCAP as well, and I would expect to see this dataset in table 1.

2) Unclear algorithm choices: The choice of the Gaussian Downsampling function is essential to computing the CI between X and Z given Y, but is unjustified in the text. Why take a Gaussian average with the hyper parameters used? These seem like essential ingredients to calculating the data component of conditional independence, but they are neither justified theoretically or experimentally (for instance, by determining how sensitive the final Z weights are to these parameters)

Additional nits: 
1) There are some typos ex “( without weights )” -> “(without weights”
2) "The most successful models rely on predictive and contrastive objectives (Baevski et al., 2020; Chung et al., 2019; Saeed et al.,
2020)..." https://arxiv.org/abs/2110.04621 shows that "Saeed et al., 2020" is not better than other 2020 methods, and in many cases worse. It might be better to be more descriptive than "successful," which doesn't seem strictly true.
summary_of_the_review: The work is a very interesting idea in a relevant subdomain of SSL research. The idea of using Conditional Independence to rate SSL targets, and the technique for calculating CI, is novel. However, the empirical results do not support the usefulness of this method (experiment #2) or are not compared to strong baselines (experiment #1). Without stronger empirical results to support the usefulness or relevance of this method, I do not believe this paper is strong enough for acceptance.
review_title: null

Meta-review: The paper proposes a method for selecting a group of pretext tasks out of a set of candidates in order to optimize self training for downstream performance. The method relies on Hilbert Schmidt Independence Criterion (HSIC) and uses a few data samples to select weights for the given set of tasks The paper demonstrates that using the method for task selection can result in learning better representation for downstream tasks improving accuracy on speech, speaker and emotion recognition.

The reviewers had concerns mostly about the strength of the empirical results. In particular, they felt that the baselines are not strong enough. To the authors credit, the paper was augmented with some of the missing experiments that the reviewers pointed out (e.g., wav2vec plus naive task selection), but that did not persuade reviewers to change their recommendations.

The paper still misses the point that self-supervised learning approaches can benefit from training larger models that result in better results. These comparisons are missing from the paper. It is established in other work that findings such as the use of pretext tasks often do not carry over to larger scales. Furthermore, the idea of pretraining a model specific to a downstream task is not inline of the philosophy of self-supervised training that aims to train a single model that can be used for many different tasks.
Decision: Reject