Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary and contributions**:
The authors propose an approach for solving a constraint version of the 3D (space+time) jigsaw puzzle over a video, using four easier surrogate tasks. The four surrogate tasks are diverse, having different formulations: regression (LCCD, CCMR), classification (CSPC), and noise contrasting (CLSC) problems. All of them are learned together, using a joint learning objective.

The authors show the value of the newly learned representations in the self-supervised scenario on two downstream tasks (video action recognition and video retrieval), achieving state-of-the-art results compared with new methods.

**Strengths**:
- The paper comes with a solution for expanding a classical self-supervised 2D problem formulation to 3D, proposing a tractable approach by breaking it into four tasks and imposing constraints through them. 
- A good amount of details on the method and on how the permutations and the surrogate tasks were chosen.
- The fact that even though the permutations are heavily constrained, they proved to be useful. 
- The newly learned representations, that embed the temporal and spatial continuity aspects, achieve state-of-the-art results on two video tasks.

**Weaknesses**:
- The temporal aspect is not sufficiently highlighted in the experiments:
- An ablation study to better distinguish between the spatial and spatiotemporal representations, with both quantitative and qualitative experiments would strengthen the submission.
- It would be interesting to see how much the quantity of temporal information reflects in the performance (ablation on the number of used frames)

**Quality**:
The idea is simple but complex enough to generate valuable representations, having the temporal aspect integrated. The paper is technically sound.

**Clarity**: 
The paper is clearly written and easy to read and follow.

**Novelty**:
The overall idea is not novel, but the way it is implemented and the proposed constraints are novel.

**Significance of this work**:
This work is relevant for the field, it incrementally advances the current integration of the temporal and spatial aspects in video.

**Typos**:
The 3rd vertical line in Table 3 should be shifted with 1 column.
review_title: A sound paper on 3D Jigsaw in Video

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
This paper proposes a new way of formulating and solving "spatiotemporal jigsaw puzzles", as a self-supervised pretext task for learning useful video representations. Positive results on two downstream tasks, action recognition and video retrieval, are shown.
The main contribution of the paper is a novel way of constraining the space of possible spatiotemporal permutations, in order to increase the tractability of the problem, and proposal of surrogate tasks that require learning and understanding of spatiotemporal continuity and correlations in order to solve them.

**Strengths**:
+ The pretext task of solving spatiotemporal jigsaw puzzles in order to learn meaningful video representations is well motivated, as has also been demonstrated in the literature.
+ Constraining the number of permutation to make the problem tractable indeed seems necessary, and the proposed method of doing so, along with the proposed surrogate tasks, are indeed effective, as indicated by the positive results on downstream tasks.
+ The ablation analysis performed shows a positive contribution of each of the described surrogate tasks and training scheme.

**Weaknesses**:

- I found some of the major statements in this work to be over-claimed:

  - " To our best knowledge, this is the first work on self-supervised video representation learning that leverages spatiotemporal jigsaw understanding". As you mention in Sec. 2,  Ahsan et al. (2019); Kim et al. (2019) both attempted to solve spatiotemporal jigsaw puzzles as a self-supervised pretexts task for learning spatiotemporal representation. This paper claims that the particular constraints imposed on the permutations used in those papers in order to increase the tractability of the problem, are not "true" spatiotemporal permutations. I believe this work at most relaxes some of those constraints, albeit in creative ways, but is not solving a fundamentally different problem.
 - Table 1, which demonstrates "state-of-the-art performance" on action recognition, is incomplete. Some stronger, not-included results of methods you did include in the table, and which, to the best of my knowledge, use only the RGB modality:  
 Pace           |  S3D-G  |  87.1  |  52.6  
 SpeedNet  |  S3D-G  |  81.1  |  48.8  
for UCF101 (left) and HMDB51 (right).
  - For Table 2, which shows "new state-of-the-art in video retrieval", additional, stronger, "Pace" results exist :  
Pace | C3D | 31.9 | 49.7 | 59.2 | 68.9 | 80.2  
Pace | R(2+1)D | 25.6 |  42.7 |  51.3 |  61.3 |  74.0  
for (L-R) top 1, 5, 10, 20, 50.

- For the visualization (Sec. 4.4), it would be nice to see what the network attends to in order to solve the pretext task, before fine-tuning on UCF101 to solve action recognition.


Out of curiosity -- often in self-supervised learning the network tends to learn "artificial cues" (such as boundary or compression artifacts) which help it solve the pretext task, without really learning anything meaningful. Significant work is usually required to mitigate such trivial learning. Did you have a similar problem? I can't seem to find any documentation of such a phenomenon in your work.

In general, I find new SSL work especially interesting if it (A) enables solving new tasks that were unfeasible before, and/or (B) pushes the boundary of SSL results on interesting/important tasks. Since (A) is, according to my understanding outlined above, not accurately demonstrated, and (B) is not shown, I vote for rejecting this paper in its current form. I would gladly reconsider given stronger results or the demonstration of newly enabled tasks based on the proposed method.


**Post-rebuttal**

I'd like to thank the authors for addressing my comments. I've read through the other reviews and responses, as well as the revised paper. The presented method for learning "true spatiotemporal permutations" is novel, and does indeed seem to learn effective representations.

What I'm not entirely sure about is how much this method manages to push the boundary of SSL. Comparing methods with different backbones is indeed tricky, and my intention was definitely not to discourage SSL works from academia. But the burden of proof should be on the new method to perform as close to an apples-to-apples comparison (in terms of backbone) to existing methods as possible. In the end, there are many many potential pretext tasks for SSL of video representations, and I do feel that in order to be publishable at a top-tier venue, they should either enable new tasks, or show clear superiority over existing methods.

Regarding temporal action segmentation as a newly enabled task -- I honestly missed this section, since it's in the appendix. This should be moved to the main paper.

If I could, I would be borderline on this paper. But since I can't, I'll give the authors the benefit of the doubt, and raise my rating to 6 (marginally above).

review_title: Interesting SSL pretext task, unsure about how much it pushes the boundary of SSL

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a novel pretext task for self-supervised video representation learning (SSVRL). The authors design several surrogate tasks for tackling intentionally constructed constrained spatiotemporal jigsaw puzzles.  The learned representations during training to solve the surrogate tasks can be transferred to other video tasks. The proposed method shows superior performances than state-of-the-art SSVRL approaches on action recognition and video retrieval benchmarks. 

## Strengths: 

+. Good performances on two benchmarks. 

+. Carefully designed surrogate tasks. 

## Weaknesses:

-. Lack insightful analysis of how the idea is inspired, why it works. It seems the intuition of the paper is to make the 3D jigsaw problem easier to solve and it will just work. But why the easier problem could help learn better representations? Each of the two steps making the problem easier need to be analyzed more thoroughly: first, making the unconstrained jigsaw problem constrained; second, solving the surrogate tasks instead of solving the constrained jigsaw problem. Actually, the carefully designed surrogate tasks are quite different from the constrained jigsaw problem. They seems more ad-hoc but not a principled way to tackle the jigsaw problem. All these questions need more indepth clarification. 

-. Experimental analysis is not thorough. In case the proposed method is not a principled method, but a carefully designed method. Extensive experiments of different variations of the proposed method could help better understand why the method works. A good performance on well-established benchmarks might be impressive, but analysis of why the performance can be achieved is more important. 

-. Writing needs improvements. In the exposition of the proposed method section, some sentences are casual and misleading. For example, the third paragraph of sec 3.2. Besides, section 3.3 is a little bit difficult to follow. It could be possibly revised more concisely. 

## Summary

Overall, the paper presents yet another method to design the pretext task for SSVRL. But my major concern is it lacks enough insights for inspiring future research for this topic. It might not be good enough for ICLR. 

review_title: Performance is nice but lack insights

final_score: rating: 7: Good paper, accept
review_content: #### Summary
In this paper, the authors extend the self-supervised 2D jigsaw puzzle solving idea to 3D for self-supervised video representation learning. To make the 3D jigsaw puzzle problem tractable, they propose a two-fold idea. First, they constrain the 3D jigsaw puzzle solution space by factorizing the permutations into time, x, and y dimensions and by grouping pieces. Second, since the constrained 3D jigsaw is still intractable, they propose four surrogate tasks of the 3D jigsaw: 1) LLCD (detecting largest continuous cuboid), 2) CSPC (3D permutation pattern classification), 3) CLSC (contrastive learning over permuted clips), 4) CCMR (measuring the global continuity of the permuted clips)
They evaluate their method's efficacy on the public benchmarks by following linear/finetuning self-supervised learning evaluation protocols.
 
#### Strengths
I like the idea of solving a 3D jigsaw puzzle as a pretext spatio-temporal learning task. By learning to solve the 3D jigsaw puzzle, the learned representations could be discriminative for the downstream tasks. Solving the jigsaw puzzle as a pretext task for representation learning is already explored and shown to be effective both in 2D spatial for images [Noroozi & Favaro, ECCV 2016] and 1D temporal dimension for videos [Xu et al., CVPR2019]. Nevertheless, due to the problem's intractability, there is no such prior work on solving a jigsaw puzzle for video representation learning. Therefore, I think this work is valuable as the authors make the problem tractable, and they show the efficacy of the 3D jigsaw puzzle solving.
 
#### Weaknesses and suggestions
However, I have several concerns about the work. 
* I do not understand how the puzzle pieces are grouped exactly. The authors show an example of grouped permutation: {12345678} -> {84567123}. It is confusing to me. It seems that the groups are {123}, {4567}, {8} from the original sequence. However, how do we make these groups? Is the group sizes always 1,3,4 for length-8 sequences? I suggest the authors provide more details on how they group the pieces.  
* Artificial patterns in the shuffled clips might be problematic. In contrast to the 2D jigsaw [Noroozi & Favaro, ECCV 2016] and 1D jigsaw [Xu et al., CVPR2019], the backbone encoder in this work takes the shuffled clips with artificial patterns (see the Fig. 1(c). There are vertical and horizontal lines). It is unlikely to see these artificial patterns in the downstream tasks. There is a training-testing mismatch. Finetuning might fix the problem, but it is not guaranteed. I want to listen to the authors' opinions on this issue.
* Missing ablation experiments and analysis. I list the missing analyses below.
  1. Why the number of largest continuous cuboids are two? What happens if it is one, three, or four? 
  2. They use non-local operation between the permuted and the original features to guide the surrogate tasks. It would be informative to show the performance when we remove this part. Also, I am not quite sure why FPN is used only for LCCD. What happens if we do not use FPN for LCCD?
  3. Analysis of the correlation between surrogate task performance and the downstream task performance.

 I will increase my rating if the majority of concerns are resolved.
  
#### Minor comments
For me, Table 3 is a bit hard to parse. 
review_title: Interesting work, good improvement by the proposed components, but need more analyses and explanations on the proposed method

Meta-review: This paper presents a new idea and approach for self-supervised video representation learning.

The reviewers' opinions diverge. R1 suggests that the paper is (marginally) above the threshold. R2 supports the paper, saying that he/she likes the idea behind the paper. R3 explicitly mentioned that he/she would like to provide a borderline rating (but cannot due to the system). R4 is not in favor of the paper, even after the rebuttal.

The AC’s opinion is more aligned with R3 and R4, who are the more senior reviewers among the four. There are two main concerns with the paper: technical contribution and experimental comparison. In terms of the contribution, both the reviewers find that the paper is lacking: "What I'm not entirely sure about is how much this method manages to push the boundary of SSL." (by R3). "Overall, the paper presents yet another method to design the pretext task for SSVRL. But my major concern is it lacks enough insights for inspiring future research for this topic." (by R4). The authors argue the difficulties of being in academia in doing this research with limited computation resources and argue that the reviewers should focus more on novelty and contribution, but even after the rebuttal, R4 is not convinced and R3 is still mildly concerned whether the proposed approach really brings something new to the field as the paper fails to show "clear superiority over existing methods".

In addition, as pointed out by the reviewers, there are several state-of-the-art self-supervised video representation learning works that the paper misses to cite, or compare against. In addition to Pace and SpeedNet R3 mentioned, below are approaches reporting results on UCF101 and HMDB with the standard self-supervised classification task setting (Table 1):

AVTS 89.0, 61.6
CVRL 92.1, 65.4
ELo 93.8, 67.4
XDC 94.2, 67.4
GDT 95.2, 72.8

We note that all these results are much superior to the best results reported by the proposed approach, 79.5 and 50.9 on UCF101 and HMDB. The authors mention in the rebuttal that these superior approaches not included in the paper use stronger backbones (and are thus omitted), but we believe a more academically proper attitude is to include all these numbers and explicitly describe why the proposed approach is not performing better, instead of completely omitting their results.

The AC also questions whether the R2D3D-34 backbone used in this paper really is computationally lighter compared to the backbones used in previous approaches like R(2+1)D-18, which alternates 2D residual modules and 2+1D residual modules (using much fewer parameters and compute than 3D modules) and also has fewer layers. XDC using R(2+1)D-18 backbone reports 86.8/52.6 (UCF/HMDB) accuracies with Kinetics-400 unlabeled data. AVTS also reports 84.1/52.5 (UCF/HMDB) accuracies using MC3-18 backbone. Similarly, GDT uses R(2+1)D-18 backbone and reports 89.3/60.0 (UCF/HMDB) accuracies using unlabeled Kinetics-400.  Even MemDPC reports 86.1/54.5 using R-2D3D backbone when optical flow feature is added. All these are far superior to the results being reported in the paper.

Overall, we view the experimental section of this paper as incomplete, and we cannot convince ourselves that the paper reaches the quality of ICLR.


[AVTS] Korbar, B., Tran, D., Torresani, L.: Cooperative learning of audio and video models
from self-supervised synchronization. In: NeurIPS (2018)

[ELo] A. Piergiovanni, A. Angelova, and M. S. Ryoo. Evolving losses for unsupervised video representation learning. In Proc. CVPR, 2020

[XDC] H. Alwassel, D. Mahajan, L. Torresani, B. Ghanem, and D. Tran. Self-supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv:1911.12667, 2019

[GDT] M. Patrick, Y. M. Asano, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003.04298, 2020

[CVRL] R. Qian, T. Meng, B. Gong, M.-H. Yang, H. Wang, S. Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. arXiv preprint arXiv:2008.03800, 2020
Decision: Reject