Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: Thanks AC for organizing the meeting. Per our discussion, my concern about the insufficient technical contribution of the paper remains unchanged. I appreciate that the work extends the correlation-based operation to a pyramid design and manages to achieve better results. However, I do feel that the overall quality & impact of this paper is below the ICLR bar and readers cannot get new technical insights from this paper. 

The idea of motion hierarchy and the effectiveness of spatial pyramid are well-aware to the research community. For example, [1,2,3] all discussed and studied leveraging motion information at multiple levels to improve action recognition. The submission proposes a different method to approach this idea (by re-using the downsampled motion features from the lower level), but does not address any significant technical challenges or provide new insights to this direction. Whether high-level motion are indeed well captured by the proposed method still cannot be justified, even given the visualization example in Figure4.

In all, I believe this submission needs improvement on the technical novelty and contribution to reach the bar of ICLR. I'll keep my original rating (3: reject).


[1] Ng, Joe Yue-Hei, et al. "Actionflownet: Learning motion representation for action recognition." 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.

[2] Wang, Heng, et al. "Video modeling with correlation networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[3] Yang, Xitong, et al. "Hierarchical contrastive motion learning for video action recognition." BMVC 2021.
review_title: Follow-up on the reviewers' meeting

final_score: rating: null
review_content: Dear AC and reviewers,

 Thank you AC for organizing the meeting. I'd like to reiterate my points as follows.

(1) The technical novelty is a bit weak and not sufficiently significant: Correlation-based operations have been proposed to explicitly model cross-frame motion for multiple computer vision tasks, e.g. optical flow [C1, C2] and video recognition [C3,C4]. The pyramid design for motion has also been found in [C5]. One may argue for the contribution of incorporating pyramid into motion modeling to "combine low-resolution, semantically strong correlation features with high-resolution, semantically weak correlation features to recover different motion patterns". The current version cannot fully justify this argument (eg. the evidence of Fig 4 is not strong enough). 

(2) Empirical Novelty And Significance: I do appreciate that the authors provide fair comparison with SlowFast baseline and show the tradeoff is nontrivial. However, it should be noted that SlowFast is a paper in 2018 and the best backbone that is used in the paper is X3D which is published in 2020. Also X3D-M is used instead of the top-performing X3D-L/XL. Therefore, I see no evidence that the performance gain will hold for larger or modern backbones. Also it is worth noting that there have been multiple Transformer-based architectures in the last two years which attain very good performance on both K400 and sth-sth. As I stated in the comments, the self-attention operation is mathematically equivalent to correlation operation, potentially endowing the capability of modeling motion implicitly. Therefore, it would be better if the authors can provide further comparison with Transformer-based methods and try if such pyramid correlation design can further boost the Transformer-based baselines.

(3) Practical concerns. This is a point that came up to me during the reviewer meeting so I won't take this point into my final score (since it might be too late for the authors to respond). Computing high-resolution correlation feature is not a memory-efficient operation. I would imagine the memory consumption becomes a bigger issue if you try to build up a correlation pyramid. Given the same hardware constraint, will this memory overhead affect the number of batch size, thus reducing the effectiveness of parallelism, even if the GFLOPs do not significantly increase? Also, even if concatenation is a zero-FLOP operation, it does affect the runtime in real-world hardware. It might be better if we can take runtime speed into account besides theoretical GFLOPs. I think such analysis will bring empirical significance to the paper even if the technical significance might be mediocre.

To sum up, I don't see any critical flaw of this paper but the quality is slightly below the ICLR bar (i.e. 5). Any improvement on the technical novelty (i.e. more convincing evidence or justification of the motivation/observation) or the empirical novelty would strengthen this paper for next possible submission. 

[C1] Sun, Deqing, et al. "Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[C2] Xu, Jia, René Ranftl, and Vladlen Koltun. "Accurate optical flow via direct cost volume processing." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[C3] Zhao, Yue, Yuanjun Xiong, and Dahua Lin. "Recognize actions by disentangling components of dynamics." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

[C4] Wang, Heng, et al. "Video modeling with correlation networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[C5] Yang, Xitong, et al. "Hierarchical contrastive motion learning for video action recognition." arXiv preprint arXiv:2007.10321 (2020).
review_title: Post-reviewer-meeting response

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use Dense Correlation Fields which is a pyramid of correlation volume of visual features. The correlation takes into account long-term correlation, and high-level semantically correlation feature. The method shows improvement over 2D CNN and 3D CNN baselines by a healthy margin.
strength_and_weaknesses: ### Pros
1. The idea is simple and the paper describes it in an organized way.

2. Nice visualization on correlation pyramid. It is an interesting finding that higher-level feature (low-resolution) has more semantically stong correlation.


### Cons
1. The idea of using cross-frame correlation is not new. The authors are suggested to discuss more papers which either construct a cost volume [a] or apply multiplicative interactions [b].

    [a] Zhao, et al. Recognize Actions by Disentangling Components of Dynamics. CVPR 2018

    [b] Wang, et al. Appearance-and-Relation Networks for Video Classification. CVPR 2018

2. The spatial correlation hierarchy is realized by "simply downsample"-ing. Have you tried other ways such as (1) conducting correlation operation on higher-level (lower spatial resolution) appearance feature map, or (2) a downsample version of the same-level feature map (i.e. downsample -> correlation instead of correlation -> downsample), or (3) conducting correlation on feature map with larger stride?

3. The baselines to compare with are weak and outdated. 

    (1) Worse tradeoff than SlowFast. DCF w/ ResNet-50 is higher than SlowFast (R-50) by 0.4% but with 30% more GFLOPs. If the authors show an accuracy-GFLOPs tradeoff curve, I don't think DCF would have a comparable trade-off on par with with SlowFast compared to Slow-only.

    (2) There lacks a comparison to recent Transformer-based video models which remarkably boost the performance in the late few years. Also, I think the correlation operator is conceptually similar to the dot product operation in the self-attention. It makes me wonder if self-attention is a better way to capture such multiplicative relations and the gain of DCF compared to plain 2D/3D ConvNets might be simply diminishing.
clarity,_quality,_novelty_and_reproducibility: + Clarity: The paper is well written.

+ Quality: good performance compared to baseline but the baselines are somewhat weak (see weakness 3).

+ Novelty: Not novel enough. See Weakness 1. 

+ Reproducibility: According to the appendix, I think the results should be reproducible.
summary_of_the_review: Considering the limited novelty and the fact that improvements are based on a somewhat weak baseline, I don't think the paper meets the bar of ICLR.


===

See the latest comment for the final rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed dense correlation fields for action recognition. Based on CorrNet, the paper proposed to model temporal long-term correlation by computing correlation for frames that are far apart. The paper also designs a spatial hierarchical architecture to aggregate the information at different granularities. The paper presents results on the something-something V1 & V2, and Kinetics-400 dataset. The proposed method shows superior results than the original CorrNet.

strength_and_weaknesses: The following are more detailed comments and questions regarding the paper:
1, In Section 3.2, the paper suggests computing bi-directional correlation, L forward frames and L backward frames. This will lead to the correlation being computed multiple times for a given pair of frames. This may lead to a waste of computational cost. How does the paper handle this problem?
 
2, How to do padding for frames at the boundary of the sampled clip? Since now we are considering L frames, the padding may cause issues for the computation of correlation.
 
3, Figure 2 and Figure 3 (c) are not very illustrative.  I suggest explaining what exactly each operator is, and what are the sizes of input tensor and output tensor.
 
4, In section 4.1, how to use R(2+1)D with ResNet pretrained with ImageNet? Since R(2+1)D has separate spatial and temporal filters, how to initialize the filters with ImageNet pretrained ResNet?
 
5, In section 3.3, the paper suggests that L is set to L=2 and L=1 for the last stage. Will L=2 allow the model to long-term temporal information? It seems pretty short.

clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: See above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new module, termed Dense Correlation Fields (DCF), to model motion information at feature level within neural networks. DCF involves a short-term module (i.e., correlation between adjacent frames), a long-term module (i.e., correlation between bidirectional and consecutive frames) and a spatial pyramid design (i.e. spatially downsampling correlation features for the next network stage). DCF can be applied to different backbone architectures and improves the baseline model on temporal modeling and action recognition. The paper conducts experiments on three common video action recognition datasets and provide ablation and visualization to analyze the impact of the DCF module.
strength_and_weaknesses: Weakness
1. My major concern is the limited technical novelty and contribution of the paper. Most designs in DCF have considerable overlap with existing works: For example, the idea of using correlation at the feature level has been extensively studied in [1,2,3] and the idea of exploiting long-term motion and motion hierarchy has been discussed in [3]. The paper also doesn't discuss and compare with some important recent work on motion modeling, such as [2, 3, 4, 5]. 

2. The experiments in the paper is not convincing and the overall performance of DCF is not strong enough. First of all, many related works on motion model [2,3,4,5] are not compared in the paper, and many recent works on action recognition are not included. The visualization of learned motion patterns in Fig. 4 seem less appealing as those in [2, 3] as well.


[1] Wang, Heng, et al. "Video modeling with correlation networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Kwon, Heeseung, et al. "Motionsqueeze: Neural motion feature learning for video understanding." European conference on computer vision. Springer, Cham, 2020.
[3] Yang, Xitong, et al. "Hierarchical contrastive motion learning for video action recognition." BMVC 2021.
[4] Piergiovanni, A. J., and Michael S. Ryoo. "Representation flow for action recognition." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[5] Diba, Ali, et al. "Dynamonet: Dynamic action and motion network." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
clarity,_quality,_novelty_and_reproducibility: The introduction of the proposed method can be further improved. I'd suggest to add one or two equations in Sec. 3.2 to make the description of long-term correlation and hierarchy aggregation more precise and clear.

As discussed in Weakness, the originality of the work is limited, and the paper doesn't sufficiently discuss how it is distinguished from the existing works on motion modeling.
summary_of_the_review: The paper proposes a new module, DCF, for modeling motion information at the feature level. However, the technical novelty and contribution of the proposed method is not sufficient, and the experiments in the paper is not convincing enough. Therefore, I'd suggest "weak reject" to the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on recognizing actions based on the motion information in the videos. The proposed idea (Dense Correlation Fields, DCF) which can be applied to different CNN backbones, computes frame to frame similarity to find spatial-temporal correlation fields. 
strength_and_weaknesses: - DCF is a simple and novel idea that outperforms SOTA by focusing on motion in videos.
- Using spatial pyramid correlation feature 
- DCF is stand-alone and can be applied to different backbones. 
clarity,_quality,_novelty_and_reproducibility: The implementation of the method is explained in detail and is clear to reader. 
The paper is in good quality shape and also the idea seems novel. I am sure it is also easy to implement and reproduce this approach. 
summary_of_the_review: The paper is pretty easy to follow and approaches the video action recognition from motion modeling point of view (which is not new, a lot of papers have leveraged differently from motion, either using Optical Flow or a different representation of the motion). I like this paper because it proposes a module that helps the network learn from motion at different scales (compared to SOTA which uses most likely a fixed representation of motion). It is a simple idea that works well.
My only concern about this method would be its computational time complexity and I would like to see a section on that to see how it performs comparing other approaches (specially since it captures motion at different stages in the network, my guess is it might make it computationally expensive). 
Also, it would be nice to cite more related papers that took into account the motion modeling and representation, there are a lot, but here are some examples: 
- V. Choutas, P. Weinzaepfel, J. Revaud, and C. Schmid. Potion: Pose motion representation for action recognition. In CVPR 2018, 2018.
- S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: a fast and robust motion represen- tation for video action recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
- J. Jiang, Y. Cao, L. Song, S. Z. Y. Li, Z. Xu, Q. Wu, C. Gan,
C. Zhang, and G. Yu. Human centric spatio-temporal action
localization.
- L.Fan,W.Huang,S.E.ChuangGan,B.Gong,andJ.Huang.
End-to-end learning of motion representation for video understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6016– 6025, 2018.
- Asghari-Esfeden, S., Sznaier, M. and Camps, O., 2020. Dynamic motion representation for human action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 557-566).
- C.-Y. Wu, M. Zaheer, H. Hu, R. Manmatha, A. J. Smola, andP.Kra ̈henbu ̈hl. Compressedvideoactionrecognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6026–6035, 2018
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a plug-in module to correlate the flow features at different scales and resolutions for existing CNN backbones for action recognition.    Prior to discussion between the AC and reviewers, the reviews on this paper were mixed, ranging from 3, 6 and 8**.  The main concerns expressed by the reviewer (NeMJ and SoQA) were technical novelty and contribution; reviewer bPjE supports the paper and finds the ideas interesting albeit marginally different from existing work.

** Reviewer (hBxH) was not considered as the reviewer expressed low confidence and asked the AC to seek alternate opinions.
justification_for_why_not_higher_score: After reading the paper and reviews and discussing with the three reviewers, the AC recommends the paper be rejected, due to the combination of minor technical differences combined with lack of compelling experimentation.  The authors are recommended to strength their work on these regards and resubmit. 
justification_for_why_not_lower_score: n/a
Decision: Reject