Spectral Nonlocal Block for Neural Network

1 INTRODUCTION . Capturing the long-range spatial-temporal dependencies is crucial for the Deep Convolutional Neural Networks ( CNNs ) to extract discriminate features in vision tasks such as image and video classification . However , the traditional convolution operator only focuses on processing local neighborhood at a time . This makes the CNNs need to go deeper with convolutional operations to enlarge the receptive fields , which lead to higher computation and memory . Moreover , going deeper can not always increase the effective receptive fields due to the Gaussian distribution of the kernel weight ( Luo et al . ( 2016 ) ) . To eliminate this limitation , some recent works focus on designing the network architecture with wider and well-designed modules to catch the long-range dependencies such as ( Peng et al . ( 2017 ) , Chen et al . ( 2017 ) , Zhao et al . ( 2017 ) ) . Although having larger receptive fields , these modules still need to be applied recursively to catch the dependencies of the pairs in large distances . Inspired by the classical non-local means method in image denoising , Wang et al . ( 2018 ) proposes the nonlocal neural network which uses the nonlocal ( NL ) block to concern the “ full-range ” dependencies in only one module by exploring the correlations between each position and all other positions . In the NL block , the affinity matrix is first computed to represent the correlations between each position pair . Then the weight means of features are calculated based on the affinity matrix to refine the feature representation . Finally , the residual connection is added to the refined feature map . Due to its simplicity and effectiveness , the nonlocal block has been widely used in image and video classification ( Wang et al . ( 2018 ) ; Yue et al . ( 2018 ) ; Tao et al . ( 2018 ) ; Chen et al . ( 2018 ) ) , image segmentation ( Huang et al . ( 2018 ) ; Yue et al . ( 2018 ) ; Wang et al . ( 2018 ) ) and person re-identification ( Liao et al . ( 2018 ) ; Zhang et al . ( 2019 ) ) recently . However , due to the complexity of the affinity matrix , the nonlocal block 1 needs much more computational effort and is sensitive to its number and position in the neural network ( Tao et al . ( 2018 ) ) . Some works solve the first problem by simplifying the calculation of the affinity matrix such as Huang et al . ( 2018 ) , He et al . ( 2019 ) , Yue et al . ( 2018 ) , Chen et al . ( 2018 ) . Only a few works try to solve the second problem which limits the robustness of the nonlocal network 2 . Tao et al . ( 2018 ) 1The nonlocal block is composed of a nonlocal operator and a residual connection 2The nonlocal network is composed of several nonlocal blocks proposes the nonlocal stage ( NS ) block which concerns the diffusion nature and maintains the same affinity matrix for all the nonlocal units in the NS block . Comparing with the NL block , the NS block is insensitive to the numbers and allows deeper nonlocal structure . However , the deeper nonlocal structure of NS block increases the complexity and do not have a remarkable improvement . In this work , we focus on elaborating a robust nonlocal block which is more flexible when using in the neural network . We prove that the nonlocal operator in the nonlocal block is equivalent to the Chebyshev-approximated fully-connected graph filter with irrational constraints that limits its liberty for learning . To remove these irrational constraints , we propose the Spectral-based Nonlocal ( SNL ) block which is more robust and can degrade into the NL and NS with specific assumptions . We also prove that the deeper nonlocal structure satisfies the stable hypothesis with the help of steadystate analysis . Based on this hypothesis , we give the full-order approximated spectral nonlocal ( gSNL ) block which is well-performed for deeper nonlocal structure . Finally , we add our proposed nonlocal blocks into the deep network and evaluate them on the image and video classification tasks . Experiments show that the networks with our proposed blocks are more robust and have a higher accuracy than using other types of nonlocal blocks . To summarize , our contributions are threefold : • We propose a spectral nonlocal ( SNL ) block as an efficient , simple , and generic component for capturing long-range spatial-temporal dependencies with deep neural networks , which is a generalization of the classical nonlocal blocks . • We propose the stable hypothesis , which can enable the deeper nonlocal structure without an elaborate preparation for both the number and position of the building blocks . We further extend SNL into generalized SNL ( gSNL ) , which can enable multiple nonlocal blocks to be plugged into the existing computer vision architectures with stable learning dynamics . • Both SNL and gSNL have outperformed other nonlocal blocks across both image and video classification tasks with a clear-cut improvement . 2 PRELIMINARY . Nonlocal block The NL block consist of NL operator with residual connection and is expressed as : Y = X + F ( A , Z ) with Z = XWg , ( 1 ) where X ∈ RN×C1 is the input feature map , F ( A , Z ) is the NL operator , Z ∈ RN×Cs is the transferred feature map that compresses the channels of X ∈ RN×C1 by a linear transformation with kernel Wg ∈ RC1×Cs . Here N is the number of positions . The affinity matrix A ∈ RN×N is composed by pairwise correlations between pixels . In the NL block , the NL operator explores the “ full-range ” dependencies by concerning the relationships between all the position pairs : F ( A , Z ) = AZW with A = ( aij ) N×N , Aij = f ( Xi , : , Xj , : ) , ( 2 ) where W ∈ RCs×C1 is the weight matrix of a linear transformation . f ( · ) is the affinity kernel which can adopt the “ Dot Product ” , “ Traditional Gasuassian ” , “ Embedded Gasussian ” or other kernel matrix with a finite Frobenius norm . Nonlocal stage To make the NL operator follow the diffusion nature that allows deeper nonlocal structure ( Tao et al . ( 2018 ) ) , the nonlocal stage ( NS ) operator uses the graph laplacian L = DA−A to replace the affinity matrix A in the NL operator : F̄ ( A , Z ) = ( A−DA ) ZW with DA = diag ( di ) , ( 3 ) where F̄ ( A , Z ) is the NS operator . di = ∑ j aij is the degree of node i . Moreover , when adding multiple blocks with the same affinity matrix A and replacing the NL operator by the NS operator , these consecutively-connected blocks become the NS block . We called these nonlocal blocks in the NS block as the NS units . 3 METHOD . The nonlocal operator can be divided into two steps : calculating the affinity matrix A to represent the correlations between each position pairs and refining the feature map by calculating the weighted means based on A . In this section , a fully-connected graph filter is utilized for explaining the nonlocal operator . With the Chebyshev approximation , we propose the SNL operator which is proved to be a generalized form of NL and NS operator and is more robust with higher performance in computer vision tasks . Furthermore , based on the stable hypothesis that deeper nonlocal structure tends to learn a stable affinity matrix , we extend our SNL operator into a full-order Chebyshev approximation version , i.e . the gSNL . 3.1 THE PROPOSED SPECTRAL NONLOCAL OPERATOR . Nonlocal operator in the graph view The nonlocal operator F ( A , Z ) is a filter that computes a weighted mean of all the positions in the feature map Z based on the affinity matrix A and then conduct the feature transformation with the kernel W. This is the same as filtering the signal Z by a graph filter Ω in the graph domain defined by the affinity matrix A ( Shuman et al . ( 2013 ) ) . Based on this perspective ( Shuman et al . ( 2013 ) ) , we further define the nonlocal operator as : Theorem 1 . Given an affinity matrix A ∈ RN×N and the signal Z ∈ RN×Cs , the nonlocal operator is the same as filtering the signal Z in the graph domain of a fully-connected weighted graph G : F ( A , Z ) = Z ∗ g = Ugθ ( Λ ) UTZ = UΩUTZ with L = DL −A = UTΛU , ( 4 ) where the graph filter Ω ∈ RN×N is a diagonal parameter matrix , i.e . Ω = diag ( ω ) , ω = ( ω1 , ω2 , ... , ωn ) . G = ( V , A ) is a fully-connected graph with the vertex set V and affinity matrix A. Λ = diag ( { λ1 , λ2 , ... , λi , ... , λN } ) and U = { u1 , u2 , ... , ui , ... , uN } are the eigenvectors and eigenvalues of the graph laplacian L. This definition requires that the graph laplacian L has non-singular eigenvalue and eigenvector , so the affinity matrix A should be a symmetric , non-negative , row-normalized matrix . To meet this requirement , the affinity matrix A can be obtained by the following steps . First , the affinity kernel is used to calculate the matrix A ( we use the dot product with embeded weight matrix Wφ ∈ RC1×Cs and Wϕ ∈ RC1×Cs as the affinity kernel , i.e . A = ( XWφ ) ( XWϕ ) ) . Then we make the matrix A symmetric : Ā = A T+A 2 . We normalize the row of Ā to make it satisfy di = 1 and having Ǎ = D−1A Ā . For the simplicity , in the following sections the symmetric , non-negative , row-normalized matrix Ǎ is denoted as A . The proposed spectral nonlocal operator The graph filter Ω in Eq . ( 4 ) contains N parameters . To simplify it , we use the Chebyshev polynomials which can reduce the N parameters into k ( k N ) . For simplicity , we firstly assume that the input Z , the output F ( A , Z ) and the output F ( A , Z ) have only one channel . Following the similar method as Defferrard et al . ( 2016 ) , the kst-order Chebyshev polynomials is used to approximate the graph filter function gθ ( Λ ) : F ( A , Z ) = K−1∑ k=0 θkTk ( L ′ ) Z with L ′ = 2L/λmax − In , s.t . T0 ( L ′ ) = In , T1 ( L ′ ) = L ′ , Tk ( L ′ ) = 2L ′ Tk−1 ( L ′ ) − Tk−2 ( L ′ ) . ( 5 ) Due to L is a random walk laplacican , the maximum eiginvalue λmax satisfies λmax = 2 which makes L ′ = A ( Shuman et al . ( 2013 ) ) . Then Eq . ( 5 ) becomes : F ( A , Z ) = K−1∑ k=0 θkTk ( A ) Z = θ0Z + θ1AZ + K−1∑ k=2 θkTk ( A ) Z , ( 6 ) If k = 1 , the first-order Chebyshev approximation of Eq . ( 6 ) becomes : F ( A , Z ) = θ0Z + θ1AZ , ( 7 ) where θ0 and θ1 are the coefficients for the first and second term which are approximated by learning with SGD . Then , extending Eq . ( 7 ) into multi-channel conditions , we can get the formation of our SNL operator : Fs ( A , Z ) = ZW1 + AZW2 , ( 8 ) where Fs ( A , Z ) is the SNL operator , W1 ∈ RCs×C1 , W2 ∈ RCs×C1 . Finally , a residual connection is added with the SNL operator to form the SNL block : Y = X + Fs ( A , Z ) = X + ZW1 + AZW2 . ( 9 ) Relation with other nonlocal operators As shown in fig . 1 , our SNL operator can degrade into the NL operator by setting W1 = 0 , i.e . θ0 = 0 . However , its analytic solution : θ0 = 2N ∑N j=0 ωj controls the total filtering intensity , which can not be guaranteed to be 0 . This setting will limit the search space when training the network and reduce the robustness of the NL block . The NL operator can not magnify features of a large range and damp some discriminative features such as the beak of the waterfowl . Our SNL operator can also degrade into the NS operator by setting W1 = −W2 , i.e . θ1 + θ0 = 0 . However , the analytic solution of this equation is θ1 + θ0 = 2N ∑N j=0 ωj ( λj + 1 ) = 0 . When setting it to zero , the filter strength of the high-frequency signal ( with high λ ) such as the small part or twig is suppressed . Thus , it still can not magnify the discriminative part such as the beak of the waterfowl as shown in fig . 1 . Comparing with NL and NS , our SNL does not have these irrational constraints and give these two parameters a liberal learning space . Thus , θ0 can control the preserve strength of the discriminative features , while θ1 can pay more attention to the low-frequency signal to diminish the noise .

Summary: In this paper, authors propose a spectral nonlocal block. First, they re-interpret the nonlocal blocks in a graph view and then use Chebyshev approximation to obtain the spectral nonlocal block which is quite simple by adding a ZW_1 term. Furthermore, they analyze the steady-state to build up a deeper nonlocal structure. Also, the gSNL is simple by adding a (2A-I)ZW_3 term.