Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper is the first work to propose an informative representation of regularization into cross-dataset action recognition. This work makes full use of existing visual transformer backbones. This method is dedicated to learning robust and informative representations. By combining the projection loss, this work can effectively mine intrinsic class relations. Experiments on different datasets prove the method can achieve better performance and produce state-of-the-art results. 
strengths_and_weaknesses: Strengths:
- This work seems to be the first work to introduce informative representation regularization into cross-dataset training for action recognition. It explores how to learn robust representations among multiple video domains. 
- To my knowledge, this work is the first work to bring the informative loss and projection loss into cross-dataset action recognition. This self-supervised loss can bring performance gains. 
-  This method may be suitable for any action recognition model. 

Weakness:
- This work seems to be the combination of existing video backbones, projection loss, and informative loss. Cross-dataset action recognition is a challenging problem due to the temporal information inner the video sequence. This paper did not consider more on the temporal information in the sequence. The design of the projection loss and the informative loss did not consider the temporal dynamics. The projection loss and the informative loss should be carefully designed for this specific cross-dataset action recognition task not directly used. 
- This work has no contributions to both the backbone and loss. The novelty of this work should be clarified.
- The problem this work studies is a multi-domain problem. However, this work did not compare with multi-domain methods. 
questions: - The table caption should be above the column.
- The cross-dataset action recognition has yielded many forms such as Video Domain Adaptation and Video Domain Generalization. The cross-dataset in this paper focuses more on multi-domain learning. The concept of cross-dataset in this paper may lead to misleading and need to be clarified. 
- This paper should add the multi-domain baselines for a fair comparison with the existing multi-domain methods.
- The author should clarify the novelty of this paper as described in Strengths And Weaknesses.


======
I have read the author's comments. Most of my concerns are clarified. I will increase my score.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to train video transformer on multiple video datasets. Instead of simply applying multiple cross-entropy loss over, the authors proposed to manipulate on embeddings encoded by an improved multiscale vision transformer to capture the intrinsic relations between classes across different action datasets. Specifically, they first adopt the informative loss from Barlow-Twins to maximize variance and covariance across embedding channels. Second, they propose to perform directed project from one dataset's classification head to another, to learn the label relation across datasets. The results show that the full pipeline is superior to vanilla training over multiple datasets and achieve state-of-the-art result.
strengths_and_weaknesses: ### Strengths

+ The performance is good considering that CrossRoad is purely trained on top of video dataset without any image pretraining.

+ The method is simple and the informative loss borrowed from BarlowTwins seems effective, especially when we want to include projection across different classification heads.

### Weaknesses

- Cross-dataset Training is not only a standalone problem in the video domain. There has been a few works on other tasks such as detection [1,2], segmentation [3]. The authors are suggested to consider discussing these methods in the related work for broader scope.

[1] Zhou, Xingyi, et al. "Simple multi-dataset detection." CVPR 2022.
[2] Wang, Xudong, et al. "Towards universal object detection by domain attention."CVPR 2019.
[3] Lambert, John, et al. "MSeg: A composite dataset for multi-domain semantic segmentation." CVPR 2020.

- A few arguments need further justification, including
  - The computation cost comparison between training video model from scratch v.s. pre-training on image dataset and then finetuning. (See Q1.)
- Some settings in the ablation need further clarification, including:
  - The reason of studying MViTv2 without relative position embedding (Q2.).
  - "Vanilla" cross-dataset training v.s. "- informative loss". I don't fully get the difference. Do you mean "Vanilla" = CrossEntropy (CE) only, "- informative loss" = CE + projection loss?

- What is $\sigma_k$ in Eq (7) and L198 and how do you determine the value?

- How does the learned directed class projection weights look like? Some visualization and discussion might be preferred.

- How do you construct the video clips on ActivityNet? Do you uniformly cut the entire video into 10-second clips or only keep those temporal segments annotated as activities. This might be useful for later efforts trying to reproduce the results. 
questions: 1. "Note that our model does not use any image training datasets, and our model computation cost is only a fraction of the baselines" (L247-248). It is true that pre-training on large image datasets. However, training video model from scratch typically means more epochs for convergence. (e.g. MViTv2 uses 200 epochs). Since training image model is often cheaper (1/10 than video model), I would see more discussion to justify this argument.

2. MViTv2 without Rel-PE. If I am not mistaken, MViTv2 uses relative positional embedding by default (See their Figure 2). Therefore I do not fully understand the statement "The “MViTv2 w/o rel” indicates the model without the relative positional embedding in the original paper" (L234-235).


==== 
Post-rebuttal revision:
My questions have been addressed. A 34-90% improvement over image-based pretraining at the cost of 1-2% drop of accuracy is fine but not impressive. The learned directed class projection is somewhat interesting. Therefore I will keep the rate of "5 Borderline accept".
limitations: Yes. The authors have discussed the limitations, e.g., co-training is limited on video datasets only. They have also covered potential negative societal impacts such as dataset biases.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new co-training paradigm **CrossRoad** for video representation learning. 
It consists of two novel loss terms, namely informative loss and projection loss. 
The informative loss encourages the variance of each dimension in the embedding to be large.
The projection loss maps predictions from other dataset heads to the current dataset class and uses ground-truth action label to compute the standard cross-entropy loss. 
Experiment results show that the two auxiliary losses are helpful in co-training. 
strengths_and_weaknesses: ## Strength
1. This work achieves strong recognition results via co-training multiple datasets with a relatively light transformer backbone MViTv2, the improvement across multiple datasets is around 2% to 4%. 
2. The efficacy of two novel components are validated via ablation study.

## Weakness
1. Large-scale image pre-training is a common practice among video transformers. However, the related results are not provided in this paper. 
2. Though the authors have the ablation study for two aux losses, there exists few analyses (See questions). 
3. Some experimental findings in this paper is quite different from findings in CoVER, but no explanation is provided. 
questions: 1. Why removing the informative loss leads to a complete failure? Would you please provide more insights or analysis? 
2. In experiments, with vanilla co-training, the K400 performance improved just a little bit (only 0.3% and is the same as MViTv2 w/o rel entry in Table 1), while performance on other datasets drops drastically. That finding is, however, quite different from the findings in CoVER. In CoVER, with vanilla joint-training, the performance on all datasets improves. What do you think could be the reason? Is it due to different pre-training or different architectures adopted? Please provide more results (like experiments with IN-21K pretraining) to support you conclusions. 
limitations: Yes.
review_title: null

Meta-review: metareview: The paper proposes a co-training method for video representation learning, by training video transformers on multiple video datasets. The paper proposes two novel loss terms: informative loss and projection loss. The informative loss encourages the variance of each dimension in the embedding to be large. The projection loss maps predictions from other datasets to the current dataset,  to learn the label relation across datasets by using ground-truth action labels to compute standard cross-entropy loss.

Based on the feedback provided by the reviewers, we recommend this paper for publication at NeurIPS 2022. 

The reviewers had some concerns about the paper. Reviewer YQNQ had concerns that the design of the projection loss and the informative loss did not consider the temporal dynamics, and that it does not compare with multi-domain methods. Reviewer iVPd  recommended considering tasks like detection, segmentation, etc.  discussing these methods in the related work for broader scope. Reviewer U8Wa mentioned that the experimental findings in this paper are quite different from findings in CoVER, but no explanation is provided.

We thank the authors for addressing the comments of the reviewers in their review during the author feedback period. The authors seem to have addressed some of the concerns/feedback from the reviewers with detailed discussions -- it would be good to include these discussions, as much as possible, in the updated paper or supplemental materials.
confidence: Certain
Decision: Accept