Shuffle to Learn: Self-supervised learning from permutations via differentiable ranking

Abstract: Self-supervised pre-training using so-called "pretext" tasks has recently shown impressive performance across a wide range of tasks. In this work we advance self-supervised learning from permutations, that consists in shuffling parts of input and training a model to reorder them, improving downstream performance in classification. To do so, we overcome the main challenges of integrating permutation inversions (a discontinuous operation) into an end-to-end training scheme, heretofore sidestepped by casting the reordering task as classification, fundamentally reducing the space of permutations that can be exploited. These advances rely on two main, independent contributions. First, we use recent advances in differentiable ranking to integrate the permutation inversion flawlessly into a neural network, enabling us to use the full set of permutations, at no additional computing cost. Our experiments validate that learning from all possible permutations (up to 1018) improves the quality of the pre-trained representations over using a limited, fixed set. Second, we successfully demonstrate that inverting permutations is a meaningful pretext task in a diverse range of modalities, beyond images, which does not require modality-specific design. In particular, we also improve music understanding by reordering spectrogram patches in the frequency space, as well as video classification by reordering frames along the time axis. We furthermore analyze the influence of the patches that we use (vertical, horizontal, 2-dimensional), as well as the benefit of our approach in different data regimes. 

Reviews and author responses:

The paper applied differentiable ranking operator on unsupervised learning framework that uses permutation based pretext task. They evaluated the proposed approach in audio, video, and image classification tasks. The results show that the proposed differentiable ranking operator is showing better performance than the pre assigned fixed permutation.

Overall, the contribution of the paper is limited in applying the soft ranking loss (or perturbed FY) on permutation based unsupervised learning framework. I think there can be two factors that can improve the performance of the proposed approach to the baseline approach (fixed permutation). The first one can be related to the soft ranking (or perturbed FY) loss itself (unlike the classification of permutation cases or regression) and the second one can be related to the use of the number of permutation cases. By using the the soft ranking type loss, they could tackle the latter factor by using any permutation cases in the training phase. However, to verify where the performance gain come from, the authors can make an another comparison that is based on classification type loss (classification on N! permutation cases). It may be hard to deal with extremely large number of class labels, but at least some comparison table can be added with small sub dataset to verify the distinction between the two factors.



This paper presents a self-supervised learning task of shuffling input patches and demanding the network to learn to unshuffle. A related prior work, Noorozi and Favaro (2016) uses a fixed set of permutations to do this task for a given number of patches, and the current paper argues to expand this idea for the full set of permutations. To this end, the paper encodes a permutation as a number tuple, with the goal of the network to learn to produce the correct tuple that has the numbers in order. As the numbers are discrete and thus non-differentiable, the paper suggests differentiable soft-variants using stochastic perturbations and regularizations (Fenchel-Young loss). Experiments are provided on audio and video tasks and show promise over the method of Noorozi and Favaro (2016).

Pros
1. A relatively simple self-supervised task with simple regularizations.
2. Experiments show some promise. 

Cons: 
1. Since the work of Noorozi and Favaro (2016), there have been several similar approaches to self-supervised learning that are not covered by the paper, either in the related work or in the experimental comparisons. A very reated work is 
[a] Visual Permutation Learning, Santa Cruz et al., TPAMI 2018
that attempts to tackle the very specific problem of fixed set of permutations used in Noorozi and Favaro (2016). They also propose an end-to-end neural network for solving the the permutation finding task. The paper should contrast against this work both technically and in experiments. 

[2] The experiments are rather weak in my opinion. As noted above, there are several prior papers (see the papers that cite the above paper) that attempts the unshuffling task in various ways, while the paper only makes comparison to the 2016 baseline paper. More recent baselines should be included in the comparisons and the advantages of the specific approach needs to be convincingly established. 

Overall, while the idea is interesting, the paper does not place its contributions within recent prior works.



Summary:

The paper proposes to use differentiable ranking techniques to predict correct permutations. It can be applied to audio, video and images. It is quite obvious from the experiments that using differentiable ranking is more powerful to learn representations compared to fixed permutation in various applications.

Strengths:

- The idea is interesting and general to apply to different modalities.
- It is well written and easy to follow.

Concerns:

- Lack of comparison with state-of-the-art methods. 
- In downstream tasks, there are always labels available, it is common to test on unsupervised setting as well.
- In Section 3.3, the experiment is conducted on MNIST, it would be more convincing to show results on larger datasets, since it is quite common to apply it on ImageNet. 


Meta-review: There is clear consensus on this submission. Reviewers cite a lack of comparison
with recent state-of-the-art methods and experiments on more realistic datasets.
Though the reviewers find aspects of the approach interesting, the decision is
to reject.


Decision: withdrawn-rejected-submissions