Encoding Spatial Distribution of Convolutional Features for Texture Representation

1 Introduction . Ideal image descriptors for classification are capable of characterizing both local features and global patterns of an image . Local features are about the local singularities or fine details of an image , such as edges and corners . Global patterns are about the distributions and spatial layouts of local image features . While local features are critical for visual recognition , global image patterns also play an important role in many scenarios , such as material recognition [ 1 ] , ground terrain classification [ 2 ] and dynamics recognition [ 3 ] . Typical convolutional neural networks ( CNNs ) for image classification connect convolutional layers with fully-connected ( FC ) layers . The FC layers encode the spatial layouts of feature maps and act as a classifier . Such a structure fails to produce a robust global representation , as FC layers capture spatial layouts based on “ absolute ” locations and their outputs are not invariant to spatial transforms such as translation , rotation and scaling , that often occur in real scenarios . To achieve the robustness to spatial transforms in CNN-based representation , many existing studies ( e.g . [ 4 , 5 , 6 ] ) apply global average pooling ( GAP ) to feature maps , which averages all entries in a feature channel so as to eliminate the dependency on spatial locations . However , the naive averaging operation totally disregards the spatial layout of a feature map . As a result , when dealing with images of complex distributive patterns , GAP can not encode sufficiently discriminative information . ∗Corresponding author : Yuhui Quan 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A robust yet discriminative global representation is particularly important for analyzing images of texture patterns [ 7 ] , since for such images , spatial transforms are one main source of variations and distributive characteristics are the key clues during analysis . In the context of texture analysis and related applications , there have been some efforts [ 8 , 9 , 10 , 11 , 2 , 12 ] on designing a more sophisticated global pooling module . Most of these works are about encoding the numerical distribution of features ( e.g . soft histograms [ 8 , 9 , 12 ] ) into a single representation . Nonetheless , they may still lose the details on the spatially distributive characteristics of features which are useful for texture analysis . Aiming at characterizing the spatially distributive patterns in feature maps to form a robust yet discriminative texture representation , this paper proposed a statistical global feature encoding module called FE ( Fractal Encoding ) , which is grounded by fractal geometry [ 13 , 14 ] . Motivations Fractal geometry has been widely used for texture analysis and synthesis ; see e.g . [ 15 , 16 , 17 , 18 , 19 , 20 ] . Its central concept is the fractal dimension that measures the distributive irregularity of a spatial point set . Many studies have shown that natural textures carry fractal dimension information [ 15 , 16 , 21 ] . Since a CNN is a spatial feature extractor , its generated feature maps may also encode fractal dimension information that reflects the regularity of texture patterns . Inspired by this , FE describes the spatial layout of features in terms of spatially distributive regularity using fractal-dimension-based statistics in a hierarchical manner . Fractal dimension is established based on the measurement at scale δ . For each δ , an object is measured in a way that ignores the irregularity of size less than δ . If the measurement is proportional to δ−β for some β and β is almost the same for small scales δ , we can compute its limit , which is called the fractal dimension . Let the n-dimensional Euclidean space Rn be covered by a mesh of n-dimensional hypercubes with diameter d. Given a point set E⊂Rn , its fractal dimension denoted by β ( E ) is defined by [ 14 ] : β ( E ) = lim d→0 logN ( E , d ) − log d , ( 1 ) where N ( E , d ) is the number of mesh hypercubes that intersect E. It can be seen fractal dimension is about cross-scale examination in terms of power law . This makes it possible to encode additional discriminative information on spatial layouts that ignored by the simple number counting operations in a histogram . For simplicity , consider the fractal dimensions on binary images . The white/black pixels on a binary image can be viewed as a set of points on a 2D grid . See Figure 1 for three numerical examples , where the fractal dimensions are different between the grid and the two bars , while the histograms of all the sample images in terms of black/white pixels are the same . If we view those sample images as specific feature patches , the fractal-dimension-based statistics from FE can well distinguish the spatial feature organization of those two types of textures . In addition to discriminability , fractal dimension is theoretically invariant to spatial bi-Lipschitz transform [ 21 , 20 ] , a general transform that covers a wide range of often-seen image transformations . Note that the influence of the spatial transforms of an image are likely to propagate to its feature maps . Then the FE driven by fractal dimensions can enjoy the robustness in this case . 1.80 1.62 1.62 . Main idea The key part in FE is the fractal analysis pooling ( FAP ) . For higher discriminability , FAP leverages multi-fractal geometry [ 23 ] to form multiple fractal-dimension-based statistical quantities . This leads to a local-global hierarchical fractal analysis process . Concretely , FE looks at a feature map as an union of point sets . The point sets are the level sets under a local point-wise categorization function D ( · ) . The FE calculates a single ( global ) fractal-dimension-based statistical quantity on every level set : f ( D̂ ) = β ( ED̂ ) , where ED̂ = { x ∈ E : D ( x ) = D̂ } . ( 2 ) Encouraged by the robustness of fractal dimension to spatial transforms and feature value scaling , we introduce the function D based on local fractal dimension [ 16 ] ( also called Hölder exponent [ 14 ] ) defined as the power D ( · ) of U ( B ( x , r ) ) ∝ rD ( x ) for x ∈ E , where B ( x , r ) is a hypercube with center x and diameter 2r , and U is some measurement . It can be seen that the local fractal dimension has a similar form as the global one in ( 1 ) : D ( x ) = lim r→0 logU ( B ( x , r ) ) −log r . ( 3 ) In other words , D ( x ) characterizes the local power law around x under measurement U . The above local-global hierarchical fractal analysis process involves some discrete operations ( e.g . box counting ) that are non-differentiable , which creates obstacles for back propagation and model training . By appropriate design with soft relaxations , we implement FE using common operations of CNNs , so that it can be end-to-end trained on top of any standard CNN and deal with images of varying sizes . In addition , FE is designed with trainable parameters which can be jointly learned with the backbone network for better adaptivity to data , as well as for automatically setting some hyper-parameters which may be difficult to determine but critical to the performance in practice . Since FAP and GAP capture distributive features from different aspects respectively , FE combines them with the bi-linear pooling ( BLP ) [ 24 ] to exploit their complementary capabilities . The resulting module is deployed to the ResNet for texture classification and retrieval . Contributions This work contributes to deep learning and texture analysis in the following aspects . Firstly , we propose a novel fractal-analysis-driven global feature encoding module which is capable of encoding the spatially distributive regularity of features on a feature map . Secondly , we build multi-fractal analysis into a standard CNN for boosting the power of deep-learning-based texture representation , which leads to state-of-the-art results in classification and retrieval tasks . 2 Related Work . Global pooling for deep texture representation The GAP is a prominent choice for global feature aggregation ; see e.g . [ 4 , 5 ] for its applications in texture representation . Squeezing a feature map by averaging , GAP is very lossy regarding spatial information . To address this , most existing studies drew inspirations from traditional global descriptors . Cimpoi et al . [ 25 ] and Song et al . [ 26 ] used fisher vector encoding [ 27 ] . but the resulting network is difficult for end-to-end training . Similar to kernel-based descriptors [ 28 ] , the BLP proposed by Lin and Maji [ 24 ] is based on the outer product of each pair of feature points , which encodes more information than GAP in terms of dependencies among channels . Dai et al . [ 29 ] combined BLP and GAP . However , both BLP and GAP are not trainable . Following bag-of-features [ 30 ] , Zhang et al . [ 8 ] proposed a trainable texture encoding ( TE ) layer which captures the numerical distribution of features based on dictionary learning . Xue et al . [ 9 ] combined TE and GAP via BLP . A similar module is used in Hu et al . [ 10 ] , but it is placed on top of multiple convolutional layers besides the last one , with an FC layer for result ensemble . Motivated by locality-constrained coding [ 31 ] , Bu et al . [ 11 ] proposed a locality-aware coding layer which encodes features with considerations on their locality constraints . To make histograms applicable to CNNs , Peeples [ 12 ] proposed a histogram layer , which is used together with GAP . Chen et al . [ 32 ] exploited the soft histogram of cross-layer local fractal dimensions for feature encoding . The proposed FE in this work distinguishes itself from the aforementioned methods by encoding the regularity on the spatial distribution of convolutional features , providing additional discrimination . Deep CNNs for texture classification In addition to global pooling , there are also some studies focusing on other aspects of deep-learning-based texture classification . To utilize the inherent correlations among the spatial contexts corresponding to visual attributes of texture images , Zhai et al . [ 33 ] proposed a multiple-attribute-perceived network with a multi-branch architecture . To efficiently describe the spatial organization of complex texture , Zhai et al . [ 34 ] proposed to learn the dependencies among texture features with a self-attention-like module , with very impressive results achieved . Fractal geometry for texture representation Fractal analysis has a long history for texture representation . Previous methods mostly employ fractal dimensions ( implemented with various numerical calculation schemes ) as local appearance descriptors [ 16 ] or global distribution descriptors [ 15 , 21 ] . These fractal-based descriptors are handcrafted and not adaptive to data . In comparison , our work forms a deep-learning-based fractal analysis model which can be end-to-end trained from data for better adaptivity . 3 Fractal Encoding : Pooling with Hierarchical Fractal Geometry . The proposed FE module processes a feature tensor from the CNN backbone with two paths that perform GAP and FAP receptively . An FC layer is attached behind GAP to align GAP to FAP in terms of output size , magnitude and semantics . An upsampling layer is applied before FAP for higher spatial resolutions of feature maps so as to make FAP more stable . The results from the two paths are integrated by BLP followed by an FC layer . See Figure 2 for the pipeline of the FE and FAP . The FAP first compresses the input feature tensor to C0 channels by a Conv1×1→BN→ReLU block . On the new feature tensor , FAP processes it slice by slice and then concatenates the results over all slices . By viewing the entries on a feature slice as a set of points in R2 , FAP runs a hierarchical fractal dimension analysis on it , i.e. , FAP groups the points based on local fractal dimension and calculates a fractal-dimension-based statistical quantity on each group . Accordingly , there are three main blocks in FAP , as described in the next subsections .

Summary: The authors propose complementing global average pooling in convolutional networks with another pooling method that aims at better representation of texture. This pooling method builds on fractal geometry and is implemented by a fractal encoding module. The module first locally estimates fractal dimension, the estimates are binned and inside each bin the fractal dimension estimation is applied again. This results in pooling of one input channel into a fixed-sized vector. The operations, such as binning and counting, are adapted to make the module differentiable. The experiments show that both types of pooling modules are required for good performance.