Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: I have two general concerns, the first is related to the presentation and the second to the relevance of the results.

(1) Presentation is confusing at many points, for instance:

* It is unclear if theorem in Eq. 4 is original or belongs to Shuman et al 2013. (no proof is given)

* Eq. 8 seems an arbitrary decomposition of the original NonLocal operator that could have been proposed without any reference to the Chebyshev expansion (which, on the other hand, is truncated to 1st order with no extra explanation).

* The point of Fig. 1 and Fig 4 is not clear. Fig. 1 explains how SpectralNonLocal reduces to NonLocal and NonLocalStage, but we can see this from the formulas. I dont see how this discussion on the Ws relates to the regions highlighted in the bird.
The same applies to Fig. 4. What are we supposed to see in Fig. 4 (and, more importantly, why?).

* What is the CFL condition? (is it the Courant-Friedrichs–Lewy sampling condition?). How is that related to the values of Ws. Can we take those arbitrarily small as suggested in that proof?

* The upper limit in the sums after Eq. 10 is unclear.

* First time table 4.2 is cited there is no context to understand it. (actually there is no table labeled as "Table 4.2") Where do we see the different number of NonLocal units?. This is only clear when you arrive and read text of page 9 (but not when cited the first time from page 6).

* Explanation of experiments is a little bit confusing (e.g. what does it mean top1 and top5 in tables?). The only explanation of "top-something" I found in the text has to do with eigenvectors in fig. 5. This also apply to the "topX" in the figures?

(2) Nevertheless, the main concern is the scarce relevance of the results: differences of behavior in all tables are about 1%. Then, what is the real advantage of the proposed modification? 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes a spectral non-local block, which is a generalized method of the non-local block and non-local stage in the literature. The proposed spectral non-local block can be plugged into a neural network to improve its effectiveness. The paper also provides theoretical analyses of the stability of the proposed method, and also extend the method by including more  Chebyshev polynomial terms. Experiments are conducted on image classification and action recognition tasks, and they valid the effectiveness of the proposed method. 

The idea is well-motivated, and it is a generalization of existing works in the literature. I do like this idea. However, I am afraid that the idea is not well explained and supported, thus I gave a weak reject to encourage the authors to further improve the paper.

The major concern I have is the reasonability of the experiments.  The experiments in the paper show relative performance gain with respect to a baseline method. It seems that there is a lack of comparison with state-of-the-art methods in the literature. For example, in Table 8, a performance gain is observed when compared with I3D. However, the recent STOA models can achieve much higher accuracy than the baseline. and also the proposed method. Since the proposed method is generic to all neural nets, it makes more sense to compare with SOTA and make improvements based on SOTA.  What is the conclusion from Table 4? Are you trying to demonstrate that the best configuration is DP3, and increasing the number of consecutive non-local blocks (from SP3 to SP5) doesn't work? It is awkward since the paper gives a stable hypothesis for deeper nonlocal structure, but experimentally the deeper structure doesn't work well. Figure 4 is abrupt without much background descriptions. Are the images randomly chosen? Ours here means SNL or gSNL? Is the colored superimposition the attention map (I believe so but the paper doesn't indicate so) and how to interpret it? What is the relation of the coverage of the critical parts on birds and the long-range dependency? More background descriptions and interpretations of the results are needed. 

Another concern I have is the clarity of the writing. There are quite a number of informal use of English, mismatched descriptions, undefined acronyms, etc. For example, in the caption of Fig. 1, it is said self-attention and self-preserving are taken effect by W1 and W2, which is contradictory to what is illustrated in the figure. Also, the terms self-attention and self-preserving, and other terms such as CGNL, A2, Hadama (Hadamard?) product, are not formally defined or described.  A lot of grammar errors and informal use of English are present, such as "which lead to", "the weight means", "when using in the neural network", "fig. 4", "Figure. 2", "more liberty for the parameter learning.", etc. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: SUMMARY: 
- Propose spectral non-local block
- improvement on image and video classification tasks

Apologies, I am not at all familiar with the theory and math behind this proposal, I do not think I am in a position to review this paper. The experiments seem convincing enough that the authors made enough effort to prove their method might work.

- Feature maps to show robustness of method is a good point
- CIFAR-10 and CIFAR-100 are certainly a good start, but might not be the best datasets to test for image classification, in lieu of ImageNet and others.
- Classification itself is a good start, it might be interesting to think about using this in a generative model such as GAN. The content reminds me of Self-Attention GAN which uses a similar non-local block (self-attention).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, authors propose a spectral nonlocal block. First, they re-interpret the nonlocal blocks in a graph view and then use Chebyshev approximation to obtain the spectral nonlocal block which is quite simple by adding a ZW_1 term. Furthermore, they analyze the steady-state to build up a deeper nonlocal structure. Also, the gSNL is simple by adding a (2A-I)ZW_3 term.

Overall, the paper is written well. I like the idea to interpret the nonlocal operation in the graph view. More important, the resulting formulation is quit concise for implementation. However, my main concern is the experiment, which should be further enhanced by perform large-scale video classification like Kinetics400.
review_title: Official Blind Review #2

Meta-review: This paper proposes a new formulation of the non-local block and interpret it from the graph view. The idea is interesting and the experimental results seems to be promising.

Reviewer has two major concerns. The first is the presentation, which is not clear enough. The second is the experimental design and analysis. The authors add more video dataset in the revision, but still lack comprehensive experimental analysis for video-based applications. 

Overall, the idea of non-local block from graph view is interesting. However, the presentation of the paper needs further polish and thus does not meet the standard of ICLR

Decision: Reject