Spectral Nonlocal Block for Neural Network

1 INTRODUCTION . Capturing the long-range spatial-temporal dependencies is crucial for the Deep Convolutional Neural Networks ( CNNs ) to extract discriminate features in vision tasks such as image and video classification . However , the traditional convolution operator only focuses on processing local neighborhood at a time . This makes the CNNs need to go deeper with convolutional operations to enlarge the receptive fields , which lead to higher computation and memory . Moreover , going deeper can not always increase the effective receptive fields due to the Gaussian distribution of the kernel weight ( Luo et al . ( 2016 ) ) . To eliminate this limitation , some recent works focus on designing the network architecture with wider and well-designed modules to catch the long-range dependencies such as ( Peng et al . ( 2017 ) , Chen et al . ( 2017 ) , Zhao et al . ( 2017 ) ) . Although having larger receptive fields , these modules still need to be applied recursively to catch the dependencies of the pairs in large distances . Inspired by the classical non-local means method in image denoising , Wang et al . ( 2018 ) proposes the nonlocal neural network which uses the nonlocal ( NL ) block to concern the “ full-range ” dependencies in only one module by exploring the correlations between each position and all other positions . In the NL block , the affinity matrix is first computed to represent the correlations between each position pair . Then the weight means of features are calculated based on the affinity matrix to refine the feature representation . Finally , the residual connection is added to the refined feature map . Due to its simplicity and effectiveness , the nonlocal block has been widely used in image and video classification ( Wang et al . ( 2018 ) ; Yue et al . ( 2018 ) ; Tao et al . ( 2018 ) ; Chen et al . ( 2018 ) ) , image segmentation ( Huang et al . ( 2018 ) ; Yue et al . ( 2018 ) ; Wang et al . ( 2018 ) ) and person re-identification ( Liao et al . ( 2018 ) ; Zhang et al . ( 2019 ) ) recently . However , due to the complexity of the affinity matrix , the nonlocal block 1 needs much more computational effort and is sensitive to its number and position in the neural network ( Tao et al . ( 2018 ) ) . Some works solve the first problem by simplifying the calculation of the affinity matrix such as Huang et al . ( 2018 ) , He et al . ( 2019 ) , Yue et al . ( 2018 ) , Chen et al . ( 2018 ) . Only a few works try to solve the second problem which limits the robustness of the nonlocal network 2 . Tao et al . ( 2018 ) 1The nonlocal block is composed of a nonlocal operator and a residual connection 2The nonlocal network is composed of several nonlocal blocks proposes the nonlocal stage ( NS ) block which concerns the diffusion nature and maintains the same affinity matrix for all the nonlocal units in the NS block . Comparing with the NL block , the NS block is insensitive to the numbers and allows deeper nonlocal structure . However , the deeper nonlocal structure of NS block increases the complexity and do not have a remarkable improvement . In this work , we focus on elaborating a robust nonlocal block which is more flexible when using in the neural network . We prove that the nonlocal operator in the nonlocal block is equivalent to the Chebyshev-approximated fully-connected graph filter with irrational constraints that limits its liberty for learning . To remove these irrational constraints , we propose the Spectral-based Nonlocal ( SNL ) block which is more robust and can degrade into the NL and NS with specific assumptions . We also prove that the deeper nonlocal structure satisfies the stable hypothesis with the help of steadystate analysis . Based on this hypothesis , we give the full-order approximated spectral nonlocal ( gSNL ) block which is well-performed for deeper nonlocal structure . Finally , we add our proposed nonlocal blocks into the deep network and evaluate them on the image and video classification tasks . Experiments show that the networks with our proposed blocks are more robust and have a higher accuracy than using other types of nonlocal blocks . To summarize , our contributions are threefold : • We propose a spectral nonlocal ( SNL ) block as an efficient , simple , and generic component for capturing long-range spatial-temporal dependencies with deep neural networks , which is a generalization of the classical nonlocal blocks . • We propose the stable hypothesis , which can enable the deeper nonlocal structure without an elaborate preparation for both the number and position of the building blocks . We further extend SNL into generalized SNL ( gSNL ) , which can enable multiple nonlocal blocks to be plugged into the existing computer vision architectures with stable learning dynamics . • Both SNL and gSNL have outperformed other nonlocal blocks across both image and video classification tasks with a clear-cut improvement . 2 PRELIMINARY . Nonlocal block The NL block consist of NL operator with residual connection and is expressed as : Y = X + F ( A , Z ) with Z = XWg , ( 1 ) where X ∈ RN×C1 is the input feature map , F ( A , Z ) is the NL operator , Z ∈ RN×Cs is the transferred feature map that compresses the channels of X ∈ RN×C1 by a linear transformation with kernel Wg ∈ RC1×Cs . Here N is the number of positions . The affinity matrix A ∈ RN×N is composed by pairwise correlations between pixels . In the NL block , the NL operator explores the “ full-range ” dependencies by concerning the relationships between all the position pairs : F ( A , Z ) = AZW with A = ( aij ) N×N , Aij = f ( Xi , : , Xj , : ) , ( 2 ) where W ∈ RCs×C1 is the weight matrix of a linear transformation . f ( · ) is the affinity kernel which can adopt the “ Dot Product ” , “ Traditional Gasuassian ” , “ Embedded Gasussian ” or other kernel matrix with a finite Frobenius norm . Nonlocal stage To make the NL operator follow the diffusion nature that allows deeper nonlocal structure ( Tao et al . ( 2018 ) ) , the nonlocal stage ( NS ) operator uses the graph laplacian L = DA−A to replace the affinity matrix A in the NL operator : F̄ ( A , Z ) = ( A−DA ) ZW with DA = diag ( di ) , ( 3 ) where F̄ ( A , Z ) is the NS operator . di = ∑ j aij is the degree of node i . Moreover , when adding multiple blocks with the same affinity matrix A and replacing the NL operator by the NS operator , these consecutively-connected blocks become the NS block . We called these nonlocal blocks in the NS block as the NS units . 3 METHOD . The nonlocal operator can be divided into two steps : calculating the affinity matrix A to represent the correlations between each position pairs and refining the feature map by calculating the weighted means based on A . In this section , a fully-connected graph filter is utilized for explaining the nonlocal operator . With the Chebyshev approximation , we propose the SNL operator which is proved to be a generalized form of NL and NS operator and is more robust with higher performance in computer vision tasks . Furthermore , based on the stable hypothesis that deeper nonlocal structure tends to learn a stable affinity matrix , we extend our SNL operator into a full-order Chebyshev approximation version , i.e . the gSNL . 3.1 THE PROPOSED SPECTRAL NONLOCAL OPERATOR . Nonlocal operator in the graph view The nonlocal operator F ( A , Z ) is a filter that computes a weighted mean of all the positions in the feature map Z based on the affinity matrix A and then conduct the feature transformation with the kernel W. This is the same as filtering the signal Z by a graph filter Ω in the graph domain defined by the affinity matrix A ( Shuman et al . ( 2013 ) ) . Based on this perspective ( Shuman et al . ( 2013 ) ) , we further define the nonlocal operator as : Theorem 1 . Given an affinity matrix A ∈ RN×N and the signal Z ∈ RN×Cs , the nonlocal operator is the same as filtering the signal Z in the graph domain of a fully-connected weighted graph G : F ( A , Z ) = Z ∗ g = Ugθ ( Λ ) UTZ = UΩUTZ with L = DL −A = UTΛU , ( 4 ) where the graph filter Ω ∈ RN×N is a diagonal parameter matrix , i.e . Ω = diag ( ω ) , ω = ( ω1 , ω2 , ... , ωn ) . G = ( V , A ) is a fully-connected graph with the vertex set V and affinity matrix A. Λ = diag ( { λ1 , λ2 , ... , λi , ... , λN } ) and U = { u1 , u2 , ... , ui , ... , uN } are the eigenvectors and eigenvalues of the graph laplacian L. This definition requires that the graph laplacian L has non-singular eigenvalue and eigenvector , so the affinity matrix A should be a symmetric , non-negative , row-normalized matrix . To meet this requirement , the affinity matrix A can be obtained by the following steps . First , the affinity kernel is used to calculate the matrix A ( we use the dot product with embeded weight matrix Wφ ∈ RC1×Cs and Wϕ ∈ RC1×Cs as the affinity kernel , i.e . A = ( XWφ ) ( XWϕ ) ) . Then we make the matrix A symmetric : Ā = A T+A 2 . We normalize the row of Ā to make it satisfy di = 1 and having Ǎ = D−1A Ā . For the simplicity , in the following sections the symmetric , non-negative , row-normalized matrix Ǎ is denoted as A . The proposed spectral nonlocal operator The graph filter Ω in Eq . ( 4 ) contains N parameters . To simplify it , we use the Chebyshev polynomials which can reduce the N parameters into k ( k N ) . For simplicity , we firstly assume that the input Z , the output F ( A , Z ) and the output F ( A , Z ) have only one channel . Following the similar method as Defferrard et al . ( 2016 ) , the kst-order Chebyshev polynomials is used to approximate the graph filter function gθ ( Λ ) : F ( A , Z ) = K−1∑ k=0 θkTk ( L ′ ) Z with L ′ = 2L/λmax − In , s.t . T0 ( L ′ ) = In , T1 ( L ′ ) = L ′ , Tk ( L ′ ) = 2L ′ Tk−1 ( L ′ ) − Tk−2 ( L ′ ) . ( 5 ) Due to L is a random walk laplacican , the maximum eiginvalue λmax satisfies λmax = 2 which makes L ′ = A ( Shuman et al . ( 2013 ) ) . Then Eq . ( 5 ) becomes : F ( A , Z ) = K−1∑ k=0 θkTk ( A ) Z = θ0Z + θ1AZ + K−1∑ k=2 θkTk ( A ) Z , ( 6 ) If k = 1 , the first-order Chebyshev approximation of Eq . ( 6 ) becomes : F ( A , Z ) = θ0Z + θ1AZ , ( 7 ) where θ0 and θ1 are the coefficients for the first and second term which are approximated by learning with SGD . Then , extending Eq . ( 7 ) into multi-channel conditions , we can get the formation of our SNL operator : Fs ( A , Z ) = ZW1 + AZW2 , ( 8 ) where Fs ( A , Z ) is the SNL operator , W1 ∈ RCs×C1 , W2 ∈ RCs×C1 . Finally , a residual connection is added with the SNL operator to form the SNL block : Y = X + Fs ( A , Z ) = X + ZW1 + AZW2 . ( 9 ) Relation with other nonlocal operators As shown in fig . 1 , our SNL operator can degrade into the NL operator by setting W1 = 0 , i.e . θ0 = 0 . However , its analytic solution : θ0 = 2N ∑N j=0 ωj controls the total filtering intensity , which can not be guaranteed to be 0 . This setting will limit the search space when training the network and reduce the robustness of the NL block . The NL operator can not magnify features of a large range and damp some discriminative features such as the beak of the waterfowl . Our SNL operator can also degrade into the NS operator by setting W1 = −W2 , i.e . θ1 + θ0 = 0 . However , the analytic solution of this equation is θ1 + θ0 = 2N ∑N j=0 ωj ( λj + 1 ) = 0 . When setting it to zero , the filter strength of the high-frequency signal ( with high λ ) such as the small part or twig is suppressed . Thus , it still can not magnify the discriminative part such as the beak of the waterfowl as shown in fig . 1 . Comparing with NL and NS , our SNL does not have these irrational constraints and give these two parameters a liberal learning space . Thus , θ0 can control the preserve strength of the discriminative features , while θ1 can pay more attention to the low-frequency signal to diminish the noise .

Summary: The paper proposes a spectral non-local block, which is a generalized method of the non-local block and non-local stage in the literature. The proposed spectral non-local block can be plugged into a neural network to improve its effectiveness. The paper also provides theoretical analyses of the stability of the proposed method, and also extend the method by including more  Chebyshev polynomial terms. Experiments are conducted on image classification and action recognition tasks, and they valid the effectiveness of the proposed method.