Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a zero-shot action recognition framework by learning an universal mapping from video to semantic space. The unseen action embeddings are re-positioned through leveraging the distribution of unlabelled test set. The universal mappings from unseen action to test videos are first defined and the target embeddings are treated as weighted Frechet means. The unseen action embeddings are re-positioned as a semantic regularization. The results on UCF101 and HMDB-51, UCF Sports and J-HMDB validates the proposed method.
main_review: Strength:
1. The zero-shot learning for action recognition is good direction however the applicability is limited in real world scenario.
2. Figure 1 is very clear to give the illustration of the concept and the high-level idea of the whole paper.
3. The performance are very stronger when compared with other baselines.

Weakness:
1. The technical details is a little hard to follow since I am not very familiar with the zero-shot learning. In my opinion, the proposed method is a general approach across video and image domains. Why not implement the experiments of such method on the image zero-shot learning?
2.  Is there any visualization to intuitively show the learning results by the transductive universal transport?
summary_of_the_review: The paper gives a new method for zero-shot action recognition by learning the transduction from unseen action to test videos in hyperspherical space. The performances are good when compared with other state-of-the-art methods. However, the details of the technique  is a little hard to follow for me. In my opinion, the paper forms well and the writing is very good. I give accept currently because of my unfamiliarity of such domain. By the way, I will follow up the work in the following reviewing sections.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper targets transductive zero-shot action recognition.  To alleviate models biased to seen categories, the authors propose to re-position unseen action embedding through transduction. There are three steps in the proposed method: first, finding an optimal mapping from unseen actions to the mapped video in the shared hyperspherical space. Second, defining target embeddings as weighted Frechet means with the weight given by the transport couplings. Third, re-positioning unseen action embeddings along the geodesic between the original and target. The zero-shot classification performance of the proposed method is tested on the UCF-101 and HMDB datasets. The zero-shot spatio-temporal localization performance is tested on the UCF Sport and J-HMDB datasets. 

main_review: While the paper demonstrates state-of-the-art results, one important concern is about fairer comparisons, especially the zero-shot spatio-temporal localization experiments (Table 3). 
- The setting of the proposed method is different from some compared papers.  For example, the authors focus on transductive ZSL, while Mettes et al. (2021), Kim et al. (2021),  and Brattoli et al. (2020) focus on inductive ZSL. 
- The proposed method uses both action and object information, while Brattoli et al. (2020) use action information only, and Mettes et al. (2021) use object information only. 

Without fairer comparisons, it is hard to assess the effectiveness of the proposed method.

Another concern is that the importance of some critical components is not adequately evaluated.
- This is also related to the comments above mentioned. The proposed method uses both video features and object information. Is this critical to obtain a good performance? The importance of video features and object information is not properly evaluated. One way to show this is to evaluate the performance of the proposed method using only one type of modality. Partial information is given in Figure 3 and the Fusion paragraph on page 7. Based on Figure 3 and the discussion, it seems that the proposed method does not outperform Brattoli et al. (2020) and Mettes et al. (2021) under the same experimental settings as the compared methods?

Typo: in 3.2 implementation details: (2+1)D -> R(2+1)D

summary_of_the_review: The paper can be further strengthened by demonstrating fairer comparisons and adequately evaluating the importance of the critical components.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work tries to address the problem of zero-shot action recognition. Particularly, the paper aims at preventing the case that many unseen action categories in the target domain are simply never being selected during inference. Using the distribution of the unlabelled test set, the embeddings of unseen actions in the target domain are reweighted and repositioned along the geodesic such that they are better aligned with embeddings of training actions in the source domain. In experiments, Empirically, the proposed method has been evaluated on 
benchmark datasets for tasks zero-shot action classification and spatio-temporal action localization.
main_review: Strengths 

- Work on the problem of reducing the biases between seen categories in the source domain and unseen categories in the target domain
the semantic space.

- Evaluate the approach on benchmark datasets for two tasks:  zero-shot action classification and spatio-temporal action localization.

Weaknesses 

- Novelty seems incremental. The proposed transductive universal transport algorithm for embedding reposition seems like a simple weighting method guided by the distribution of the unlabelled test set. The paper merely uses existing approaches to solve the transductive optimal transport problem but does NOT bring any new insights.

- Generalization seems a concern. The proposed approach heavily depends on the distribution of the unlabelled test set. It seems sensitive to the distribution and the number of clusters.  As shown in Figure 2, using the target embeddings seems on par with repositioned embeddings. Also, the proposed approach seems to only works in the case with a small number of clusters. 
summary_of_the_review: This paper proposes a sensible solution to reduce the bias between the source domain and the target domain for the task of action recognition. But novelty seems incremental. Also, some experiments seems a bit unconvincing and the approach seems not to scale to general settings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work introduces transductive universal transport for zero-shot action recognition, where no training examples for unseen classes are available. To address the biases of prior approaches towards seen classes during inference, this paper re-positions unseen action embeddings through transduction by using the distribution of the unlabelled test set. Experimental results on several action recognition datasets demonstrate the effectiveness of the proposed method.
main_review: Strengths
1. The use of transductive universal transport for zero-shot action recognition is new.
2. The experimental results show the effectiveness of this new method and also better performance than prior states of the art.

Weaknesses
1. The approach needs access to the entire testing video set to obtain distribution information of testing videos. This is an unrealistic setting. When used in practice, a machine learning model should expect to see one testing example a time.

2. Many symbols are not clearly defined, making the math descriptions in this paper hard to read. For example, in Eq. (2), it is unclear what w_s, w_u, sum_|Lu| sum_|Ls| are, what weights for labels, i.e., w_lu and w_ls, mean, and why u_s and u_u are sets of labels.

3. Does the use of transductive universal transport bring any computational overhead to the zero-shot learning model? It is interesting to see comparison of inference time and complexity with the baseline and some states of the art.
summary_of_the_review: The idea of using transductive universal transport for zero-shot action recognition is new, and the performance is good. But the core setting that the entire testing set is available during training to get the distribution information is unrealistic. The writing, especially the math part, needs improvement.
review_title: null

Meta-review: This paper was reviewed by four experts in the field and received mixed scores (1 borderline accept, 3 borderline reject). The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper. AC feels that this work has great potential, but needs more work to better clarify the contribution and include additional ablated study. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject