Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents 4D convolutional neural networks for video-level representations. To learn long-range evolution of spatio-temporal representation of videos, the authors proposed V4D convolution layer. Benchmark on several video classification dataset shows improvement.

1. In section 3.1, the authors selected a snippet from each section, but this was not rigorously defined. Same for action units. It does intuitively makes sense, but more mathematical definition (e.g., dimensionality) may be needed.

2. In section 3.2, the authors argued that 3D kernel suffers from trade-off between receptive field and cost of computation. At the end of the subsection, the authors argue that 4D convolution is just k times larger than 3D kernels, which sounds like contradicting. 3D convolution is already expensive and not scalable, but 4D operation sounds even more expensive and more prohibitive.

3. In the paper, the authors argued that clip-level feature learning is limited as it is hard to learn long-range spatio-temporal dependency. It makes sense, and I expect the proposed model may benefit from its design for long-range spatio-temporal feature learning. However, what I see in the experiments is on ~300 frames for Mini-Kinetics and 36-72 frames for Something-Something dataset. Assuming that a second is represented with 15-30 frames, this corresponds to 10-20 sec and 1-4 sec, respectively. I'd say these short videos are still clips.

The paper presents an interesting idea, but there are some issues that need to be addressed before published on ICLR.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes 4d convolution and an enhanced inference strategy to improve the feature interaction for video classification.  State-of-the-art performance is achieved on several datasets.  However, there are still details and concerns.

1. The paper should also talk about the details of ARNet and discuss the difference, as I assume they are the most related work 
2. during sampling, either training or testing, how do authors handle temporal overlap or make it overlap?
3. can you provide the training memory, inference speed, and total training time?
4. Most importantly, for table4, authors are comparing to the ResNet18 ARNet instead of ResNet50? which is better than the proposed method.
5. lack of related work:  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks, CVPR 2019.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: [Summary]
The paper presents a video classification framework that employs 4D convolution to capture longer term temporal structure than the popular 3D convolution schemes. This is achieved by treating the compositional space of local 3D video snippets as an individual dimension where an individual convolution is applied. The 4D convolution is integrated in resnet blocks and implemented via first applying 3D convolution to regular spatio-temporal video volumes and then the compositional space convolution, to leverage existing 3D operators. Empirical evaluation on three benchmarks against other baselines suggested the advantage of the proposed method.         

[Decision]
Overall, the paper addresses an important problem in computer vision (video action recognition) with an interesting. I found the motivation and solution are reasonable (despite some questions pending more elaboration), and results also look promising, thus give it a weak accept (conditional on the answers though).

[Comments]
At the conceptual level, the idea of jointly modeling local video events is not novel, and can date back to at least ten years ago in the paper “Learning realistic human actions from movies”, where the temporal pyramid matching was combined with the bag-of-visual-words framework to capture long-term temporal structure. The problem with this strategy is that the rigid composition only works for actions that can be split into consecutive temporal parts with prefixed duration and anchor points in time, which is clearly challenged by many works later when more complicated video events are studied. It seems to me that the proposed framework also falls in this category, with a treatment from deep learning. It is definitely worth some discussion on this path.            

That said, I would like to see more analysis on the behavior of the proposed method under various interesting cases not tested yet. Despite the claim that the proposed method can capture long-term video patterns, the static compositional nature seems to work best for activities with well-defined local events and clear temporal boundaries. These assumptions hold mostly true for the three datasets used in the experiment, and also are suggested by results in table 2(e), where 3 parts are necessary to achieve optimal results. How does the proposed method perform in more complicated tasks such as        
- action detection or localization (e.g., in benchmarks JHMDB or UCF101-24). 
- complex video event modeling (e.g., recognizing activities in extended video of TRECVID).
Will it still be more favorable than other concerning baselines?  

Besides, on the computation side, it would be complexity, an explicit comparison of complexity makes it easier to evaluate the performance when compared to other state-of-the-art methods. 

[Area to improve]
Better literature review to reflect the relevant previous video action recognitions, especially those on video compositional models.  
Proof reading - The word in the title should be “Convolutional”, right?
review_title: Official Blind Review #2

Meta-review: This paper proposes video-level 4D CNNs and the corresponding training and inference methods for improved video representation learning. The proposed model achieves state-of-the-art performance on three action recognition tasks. 
Reviewers agree that the idea well motivated and interesting, but were initially concerned with positioning with respect to the related work, novelty, and computational tractability. As these issues were mostly resolved during the discussion phase, I will recommend the acceptance of this paper. We ask the authors to address the points raised during the discussion to the manuscript, with a focus on the tradeoff between the improved performance and computational cost.
Decision: Accept (Poster)