Encoding Spatial Distribution of Convolutional Features for Texture Representation
Year: 2021

Abstract:
Existing convolutional neural networks (CNNs) often use global average pooling (GAP) to aggregate feature maps into a single representation. However, GAP cannot well characterize complex distributive patterns of spatial features while such patterns play an important role in texture-oriented applications, e.g., material recognition and ground terrain classification. In the context of texture representation, this paper addressed the issue by proposing Fractal Encoding (FE), a feature encoding module grounded by multi-fractal geometry. Considering a CNN feature map as a union of level sets of points lying in the 2D space, FE characterizes their spatial layout via a local-global hierarchical fractal analysis which examines the multi-scale power behavior on each level set. This enables a CNN to encode the regularity on the spatial arrangement of image features, leading to a robust yet discriminative spectrum descriptor. In addition, FE has trainable parameters for data adaptivity and can be easily incorporated into existing CNNs for end-to-end training. We applied FE to ResNet-based texture classification and retrieval, and demonstrated its effectiveness on several benchmark datasets.

heading: 1 Introduction
text: Ideal image descriptors for classification are capable of characterizing both local features and global patterns of an image. Local features are about the local singularities or fine details of an image, such as edges and corners. Global patterns are about the distributions and spatial layouts of local image features. While local features are critical for visual recognition, global image patterns also play an important role in many scenarios, such as material recognition [1], ground terrain classification [2] and dynamics recognition [3].
Typical convolutional neural networks (CNNs) for image classification connect convolutional layers with fully-connected (FC) layers. The FC layers encode the spatial layouts of feature maps and act as a classifier. Such a structure fails to produce a robust global representation, as FC layers capture spatial layouts based on “absolute” locations and their outputs are not invariant to spatial transforms such as translation, rotation and scaling, that often occur in real scenarios.
To achieve the robustness to spatial transforms in CNN-based representation, many existing studies (e.g. [4, 5, 6]) apply global average pooling (GAP) to feature maps, which averages all entries in a feature channel so as to eliminate the dependency on spatial locations. However, the naive averaging operation totally disregards the spatial layout of a feature map. As a result, when dealing with images of complex distributive patterns, GAP cannot encode sufficiently discriminative information.
∗Corresponding author: Yuhui Quan
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A robust yet discriminative global representation is particularly important for analyzing images of texture patterns [7], since for such images, spatial transforms are one main source of variations and distributive characteristics are the key clues during analysis. In the context of texture analysis and related applications, there have been some efforts [8, 9, 10, 11, 2, 12] on designing a more sophisticated global pooling module. Most of these works are about encoding the numerical distribution of features (e.g. soft histograms [8, 9, 12]) into a single representation. Nonetheless, they may still lose the details on the spatially distributive characteristics of features which are useful for texture analysis.
Aiming at characterizing the spatially distributive patterns in feature maps to form a robust yet discriminative texture representation, this paper proposed a statistical global feature encoding module called FE (Fractal Encoding), which is grounded by fractal geometry [13, 14].
Motivations Fractal geometry has been widely used for texture analysis and synthesis; see e.g. [15, 16, 17, 18, 19, 20]. Its central concept is the fractal dimension that measures the distributive irregularity of a spatial point set. Many studies have shown that natural textures carry fractal dimension information [15, 16, 21]. Since a CNN is a spatial feature extractor, its generated feature maps may also encode fractal dimension information that reflects the regularity of texture patterns. Inspired by this, FE describes the spatial layout of features in terms of spatially distributive regularity using fractal-dimension-based statistics in a hierarchical manner.
Fractal dimension is established based on the measurement at scale δ. For each δ, an object is measured in a way that ignores the irregularity of size less than δ. If the measurement is proportional to δ−β for some β and β is almost the same for small scales δ, we can compute its limit, which is called the fractal dimension. Let the n-dimensional Euclidean space Rn be covered by a mesh of n-dimensional hypercubes with diameter d. Given a point set E⊂Rn, its fractal dimension denoted by β(E) is defined by [14]:
β(E) = lim d→0 logN (E, d) − log d , (1)
where N (E, d) is the number of mesh hypercubes that intersect E. It can be seen fractal dimension is about cross-scale examination in terms of power law. This makes it possible to encode additional discriminative information on spatial layouts that ignored by the simple number counting operations in a histogram. For simplicity, consider the fractal dimensions on binary images. The white/black pixels on a binary image can be viewed as a set of points on a 2D grid. See Figure 1 for three numerical examples, where the fractal dimensions are different between the grid and the two bars, while the histograms of all the sample images in terms of black/white pixels are the same. If we view those sample images as specific feature patches, the fractal-dimension-based statistics from FE can well distinguish the spatial feature organization of those two types of textures.
In addition to discriminability, fractal dimension is theoretically invariant to spatial bi-Lipschitz transform [21, 20], a general transform that covers a wide range of often-seen image transformations. Note that the influence of the spatial transforms of an image are likely to propagate to its feature maps. Then the FE driven by fractal dimensions can enjoy the robustness in this case.

heading: 1.80 1.62 1.62
text: Main idea The key part in FE is the fractal analysis pooling (FAP). For higher discriminability, FAP leverages multi-fractal geometry [23] to form multiple fractal-dimension-based statistical quantities. This leads to a local-global hierarchical fractal analysis process. Concretely, FE looks at a feature map as an union of point sets. The point sets are the level sets under a local point-wise categorization function D(·). The FE calculates a single (global) fractal-dimension-based statistical quantity on
every level set: f(D̂) = β(ED̂), where ED̂ = {x ∈ E : D(x) = D̂}. (2) Encouraged by the robustness of fractal dimension to spatial transforms and feature value scaling, we introduce the function D based on local fractal dimension [16] (also called Hölder exponent [14]) defined as the power D(·) of U (B (x, r)) ∝ rD(x) for x ∈ E, where B (x, r) is a hypercube with center x and diameter 2r, and U is some measurement. It can be seen that the local fractal dimension has a similar form as the global one in (1):
D(x) = lim r→0 logU(B(x, r)) −log r . (3)
In other words, D(x) characterizes the local power law around x under measurement U . The above local-global hierarchical fractal analysis process involves some discrete operations (e.g. box counting) that are non-differentiable, which creates obstacles for back propagation and model training. By appropriate design with soft relaxations, we implement FE using common operations of CNNs, so that it can be end-to-end trained on top of any standard CNN and deal with images of varying sizes. In addition, FE is designed with trainable parameters which can be jointly learned with the backbone network for better adaptivity to data, as well as for automatically setting some hyper-parameters which may be difficult to determine but critical to the performance in practice.
Since FAP and GAP capture distributive features from different aspects respectively, FE combines them with the bi-linear pooling (BLP) [24] to exploit their complementary capabilities. The resulting module is deployed to the ResNet for texture classification and retrieval.
Contributions This work contributes to deep learning and texture analysis in the following aspects. Firstly, we propose a novel fractal-analysis-driven global feature encoding module which is capable of encoding the spatially distributive regularity of features on a feature map. Secondly, we build multi-fractal analysis into a standard CNN for boosting the power of deep-learning-based texture representation, which leads to state-of-the-art results in classification and retrieval tasks.

heading: 2 Related Work
text: Global pooling for deep texture representation The GAP is a prominent choice for global feature aggregation; see e.g. [4, 5] for its applications in texture representation. Squeezing a feature map by averaging, GAP is very lossy regarding spatial information. To address this, most existing studies drew inspirations from traditional global descriptors. Cimpoi et al. [25] and Song et al. [26] used fisher vector encoding [27]. but the resulting network is difficult for end-to-end training. Similar to kernel-based descriptors [28], the BLP proposed by Lin and Maji [24] is based on the outer product of each pair of feature points, which encodes more information than GAP in terms of dependencies among channels. Dai et al. [29] combined BLP and GAP. However, both BLP and GAP are not trainable. Following bag-of-features [30], Zhang et al. [8] proposed a trainable texture encoding (TE) layer which captures the numerical distribution of features based on dictionary learning. Xue et al. [9] combined TE and GAP via BLP. A similar module is used in Hu et al. [10], but it is placed on top of multiple convolutional layers besides the last one, with an FC layer for result ensemble. Motivated by locality-constrained coding [31], Bu et al. [11] proposed a locality-aware coding layer which encodes features with considerations on their locality constraints. To make histograms applicable to CNNs, Peeples [12] proposed a histogram layer, which is used together with GAP. Chen et al. [32] exploited the soft histogram of cross-layer local fractal dimensions for feature encoding.
The proposed FE in this work distinguishes itself from the aforementioned methods by encoding the regularity on the spatial distribution of convolutional features, providing additional discrimination.
Deep CNNs for texture classification In addition to global pooling, there are also some studies focusing on other aspects of deep-learning-based texture classification. To utilize the inherent correlations among the spatial contexts corresponding to visual attributes of texture images, Zhai et al. [33] proposed a multiple-attribute-perceived network with a multi-branch architecture. To efficiently describe the spatial organization of complex texture, Zhai et al. [34] proposed to learn the dependencies among texture features with a self-attention-like module, with very impressive results achieved.
Fractal geometry for texture representation Fractal analysis has a long history for texture representation. Previous methods mostly employ fractal dimensions (implemented with various numerical
calculation schemes) as local appearance descriptors [16] or global distribution descriptors [15, 21]. These fractal-based descriptors are handcrafted and not adaptive to data. In comparison, our work forms a deep-learning-based fractal analysis model which can be end-to-end trained from data for better adaptivity.

heading: 3 Fractal Encoding: Pooling with Hierarchical Fractal Geometry
text: The proposed FE module processes a feature tensor from the CNN backbone with two paths that perform GAP and FAP receptively. An FC layer is attached behind GAP to align GAP to FAP in terms of output size, magnitude and semantics. An upsampling layer is applied before FAP for higher spatial resolutions of feature maps so as to make FAP more stable. The results from the two paths are integrated by BLP followed by an FC layer. See Figure 2 for the pipeline of the FE and FAP.
The FAP first compresses the input feature tensor to C0 channels by a Conv1×1→BN→ReLU block. On the new feature tensor, FAP processes it slice by slice and then concatenates the results over all slices. By viewing the entries on a feature slice as a set of points in R2, FAP runs a hierarchical fractal dimension analysis on it, i.e., FAP groups the points based on local fractal dimension and calculates a fractal-dimension-based statistical quantity on each group. Accordingly, there are three main blocks in FAP, as described in the next subsections.

heading: 3.1 Local Dimension Estimation Block (LDEB)
text: The LDEB calculates a point-wise local fractal dimension mapD ∈ RH×W from the input feature sliceX ∈ RH×W : LDEB :X ∈ RH×W →D ∈ RH×W . (4) The calculation is based on (3), where one key is computing the measurement U , i.e., forming a series of measurement maps: Ur ∈ RH×W , r = 1, 2, · · · . The measurement U in (3) calculates a value on local block B(x, r) for every x. For a feature slice, a local block can be viewed as a local patch of size r × r in the slice, and U is essentially calculated on all the patches sampled with a sliding window on X , which can be modeled by a convolution process. We define the measurement to be learnable for better data adaptivity. Since the measurement is defined on multiple scales, we simulate this by a multi-scale convolution:
Ur = Gr ⊗X, r = 1, 2, · · · , (5)
where ⊗ denotes the discrete convolution operation, and Gr ∈ Rr×r is an r × r kernel learned together with the model.
The numerical implementation of (3) is done by the least-squares fitting (LSF) on Ur versus r in the log-log coordinate system. A series of measurement maps {Ur, r = r1, r2, · · · , rZ} are first calculated, where r1 < · · · < rZ are a finite sequence of predefined ordered positive radii. Then the estimation of local fractal dimension map in (3) becomes an LSF problem:
min D,P Z∑ z=1 (D(w, h) log rz + P (w, h)− log Urz (w, h))2, (6)
where P (w, h) denotes the bias term in the fitting. Setting the derivative of (6) w.r.t. D and P to zero respectively, we have{ ∑Z
z=1 ( D(w, h)log2 rz + P (w, h) log rz − log rzlog Urz (w, h) ) = 0∑Z
z=1 (D(w, h) log rz + P (w, h)− log Urz (w, h)) = 0 (7)
Then, the local fractal dimension mapD ∈ RW×H is computed as the solution to (7):
D(w, h) = Z ∑Z z=1 log rzlog Urz (w, h)− (∑Z z=1 log rz )(∑Z z=1 log Urz (w, h) ) Z ∑Z z=1 log 2 rz − (∑Z z=1 log rz
)2 . (8)

heading: 3.2 Point Grouping Block (PGB)
text: The PGB categorizes the entries on the feature slice based on the mapD output by the LDEB. Let
E = {1, · · · ,W} × {1, · · · , H} ⊂ R2, (9) ED̂k = {(h,w) ∈ E : D̂k ≤D(h,w) < D̂k+1}, (10)
for k = 1, · · · ,K, given the intervals [D̂1, D̂2], [D̂2, D̂3], · · · , [D̂K , D̂K+1] where D̂k ∈ R for all k. The set ED̂k can be represented by a binary map ED̂k of the same size asD, where ED̂k(h,w) = 1 if D̂k ≤D(h,w) < D̂k+1 and 0 otherwise.
The setting of intervals [D̂k, D̂k+1], k = 1, · · · ,K can have a noticeable impact on the accuracy of multi-fractal analysis [23]. However, they are non-trivial to determine, as the range of D is uncertain and uniform intervals are often not optimal. In addition, grouping with hard assignment may cause noticeable quantization errors [35], as well as optimization difficulty in back propagation during model training. To address these issues, we implement the grouping function using a soft assignment scheme [35] with trainable quantization intervals. Instead of using binary values, the soft assignment uses real values in [0, 1] for defining Eck(h,w) to indicate the degree of membership. Let {ck ∈ R, k = 1, ...,K} denote a set of trainable interval anchors. For each ck, we define its corresponding membership map Eck ∈ [0, 1]W×H by
Eck(w, h) = exp(−sk(D(w, h)− ck)2)∑K k=1 exp(−sk(D(w, h)− ck)2) , (11)
where {sk ∈ R}Kk=1 is a set of learnable weights. It can be implemented via Softmax. To summarize, the PGB takes a local dimension mapD as input, and generates a series of soft membership maps:
PGB :D ∈ RH×W → {Eck ∈ [0, 1]H×W }Kk=1. (12)

heading: 3.3 Global Dimension Calculation Block (GDCB)
text: The GDCB accepts {Eck}Kk=1 from the PGB as input, calculates a fractal-dimension-based quantity β on each Eck individually, and finally concatenates βks as output:
GDCB : {Eck ∈ [0, 1]H×W }Kk=1 → β = (β1, β2, · · · , βK) ∈ RK . (13)
The quantity βk is calculated based on (1). Similar to the computation on D, Equation (1) can be numerically implemented by using bi-log LSF. However, the box counting operation in (1) is about
counting the patches with non-zero entries, which is non-differentiable and thus makes training infeasible. Also recall that Eck is a soft membership map with values in [0, 1] instead of binary ones. It is very likely that Eck contains no zeros and the box counting is inapplicable in this case.
Based on above considerations, we replace the box counting N in (1) by the sum of the maximal values from all patches of size d × d in Eck . When Eck is binary, it equals to box counting. The new scheme can be efficiently implemented by stride-1 max pooling with window size d × d and summation. By varying d = d1, · · · , dT , we have the sums denoted by N1, · · · , NT . Similar to (7), βk is then calculated as the LSF solution to the modified (1):
βk = T ∑T t=1 log dtlogNt − (∑T t=1 log dt )(∑T t=1 logNt ) T ∑T t=1 log 2 dt − (∑T t=1 log dt
)2 . (14)

heading: 4 Experimental Evaluation

heading: 4.1 Deployment of FE Module and Setting of Experiments
text: The FE module can be easily incorporated into a CNN by porting it on top of some convolutional layers. To evaluate the effectiveness of FE, we insert it next to the last convolutional layer of the ResNet [36], an often-used backbone (e.g. [8, 9]). The resulting CNN is referred to as FENet. Inheriting the advantages from FE, FENet can deal with flexible image size and be end-to-end trained.
We apply FENet to texture classification and texture retrieval on six benchmark datasets, including GTOS [2], GTOS-M [9], KTH [37], MINC [38], DTD [39] and FMD [1]. Same as existing work, we use the provided split schemes on GTOS, MINC and DTD, and random 10 splits on KTH-TIPS2b and FMD with recommended split sizes. The mean and standard deviation of classification accuracies over all splits are calculated. The results are reported using two runs on GTOS-M and five-time statistics on other datasets.
Based on the empirical parameter setting in fractal analysis in previous studies [21, 35], we set C0 = 3 in GAP, r = 1, 2, ..., 6 in LDEB, K = 16 in PGB, and dt = 2, 3, · · · , 6 in GDCB. The output sizes of the two paths in FE are both set to 48. Following [34, 8], we use ResNet-18 and ResNet-50 as the backbone respectively. On all the datasets, FENet is trained with the cross-entropy loss via the momentum SGD optimizer with default settings and 30 epochs. The batch size is set to 16 on FMD, 32 on KTH, and 64 on the other four datasets. The learning rate is initialized to 1e−3 on FMD, 5e−3 on MINC, GTOS and GTOS-M, and 1e−2 on KTH and DTD datasets, with cosine decay every 10 epochs. The ResNet backbones are initialized with the pre-trained models on Imagenet. The {Gr}r in (5) are initialized as Gaussian kernels with bandwidth 1. Other parameters of FENet are initialized by Xavier [40]. Data augmentation via horizontal flipping and random cropping to 224× 224 is applied. Our FENet is implemented with PyTorch 1.7, and the code will be released at the website: https://github.com/csfengli/FENet. All the experiments were run on a single Titan XP GPU.

heading: 4.2 Results in Texture Classification
text: Our FENet is compared with several recent deep models for texture classification, which include DeepTEN [8], DEPNet [9], LSCNet [11], MAPNet [33], DSRNet [34] and HistNet [12]. The results on the six datasets are listed in Table 1. The results of the compared methods are directly quoted from existing literature whenever possible, or left blank otherwise.
It can be seen from Table 1 that FENet performs the best on KTH, FMD, MINC and GTOS in terms of mean accuracy, no matter using ResNet-18 backbone or ResNet-50 backbone. On GTOS-M, the FENet is the best performer with ResNet-18 backbone.
On DTD, the FENet performs better than others except DSRNet and MAPNet. The reason why FENet shows unsatisfactory performance on DTD is probably that, the labels for the textures in DTD are based on describable subjective semantics, which are quite different from other texture datasets whose labels are based on objective materials. Even in such a case, our method still applies and provides a good texture descriptor (e.g. superior performance to DeepTEN, DEPNet and HistNet). Recall that MAPNet is developed based on multiple attribute perception (MAP) which is more suitable for
the subjective attributes on DTD (similar reason for DSRNet), while the spatial regularity captured by our FE is more suitable for encoding the characteristics of objective materials (e.g. the superior performance of our method to MAPNet and DSRNet on other datasets).
It is worth mentioning that the training dataset of GTOS-M is divided into three subsets with small, medium and large scales respectively. It is shown in [9] that the performance of deep CNNs such as DeepTEN using the single-resolution subset rather than the multi-resolution ones of GTOS-M will have a noticeable accuracy decrease. In comparison, our FENet has no performance decrease when trained on single-scale data, and the results of FENet on GTOS-M in Table 1 are actually based on only the medium-scale training data. See Section 4.5 for more analysis.
It is also mentioning that the performance improvement of DSRNet and MAPNet is less than 1% in most cases, which is so significant and seems to saturate. In comparison, in the case of ResNet-18, there are three datasets where our method has accuracy improvement larger than 1% and even over 4.5%, and on the remaining dataset the improvement is nearly 1%.

heading: 4.3 Results in Texture Retrieval
text: The global representation to the classification module in FENet can be used as a texture descriptor. Its effectiveness is evaluated by the retrieval task as follows. An image sample is represented by its corresponding feature tensor output by the last pooling layer (i.e. after BLP). Each sample in the test dataset is used as a query to search the remaining samples, and its top-R similar samples in the dataset are collected in terms of cosine similarity. The average precision and recall over all query samples are calculated.
By varying R from 1 to the number of searched samples, we obtain the precision-recall (PR) curves and the corresponding mean average precision (mAP), which are shown in Figure 3. It can be seen that FENet achieved the best retrieval results among the compared methods on GTOS-M, KTH and FMD datasets. Such results have demonstrated the effectiveness of the texture feature representation learned in FENet.

heading: 4.4 Effectiveness of FE Module
text: Ablation study on FE module in FENet To demonstrate the effectiveness of FE in FENet, we form a baseline (“w/o FE Module”) from FENet by removing its FE module as well as the BLP layer and adapting the dimension of the related FC layer. Similarly, we form another baseline (“w/o GAP”) by removing the GAP layer in FENet. These two baselines are trained in the same way as FENet.
The results of the baselines are compared to FENet on GTOS-M and KTH datasets in Table 2. It can be seen that the performance decrease caused by the removal of FE Module is significant on GTOS-M and KTH. This suggests that the regularity of spatial distribution captured by FE Module
plays a more important role. When we remove the GAP in FE, performance decreases moderately. All these results have demonstrated the effectiveness of FE Module in FENet.
Incorporating FE module into DeepTEN We also demonstrate the effectiveness of the FE module by plugging it into DeepTEN [8] and examining the performance gain. Recall that the DeepTEN also uses a ResNet backbone, together with an texture encoding module (called TE) which acts as a soft histogram with learned bins. Similar to our FENet, the FE module (together with the upsampling layer in front of it) is inserted in parallel to the TE layer after the last RB of the ResNet backbone. The output of the FE module and TE module are integrated by BLP. The resulting model is trained with the same scheme as the original work.
The results on the four datasets are listed in Table 3. It can be seen that the introduction of FE module brings significant improvement to DeepTEN on GTOS-M, KTH and FMD, as well as noticeable improvement on MINC. Such improvement again demonstrated the effectiveness of the FE module. Indeed, the TE module in DeepTEN is essentially a soft counting layer which only considers the quantitative distribution but ignores the spatial distribution of features in feature slices. The FE module provides complementary information to the TE module by characterizing the spatial organization of a feature map based on multi-fractal analysis.

heading: 4.5 More Analysis
text: Complexity and efficiency We compare the complexity of backbone CNN, DeepTEN, DEPNet and FENet in terms of the number of parameters and number of floating operations per second (FLOPS). We also compare their efficiency in term of inference time on an input image of size 384 × 384. See Table 4 for the comparison. Our FE module only introduces a small number of
additional parameters to the backbone, and the complexity of FENet is comparable to that of other deep models under the same backbone.
Generalization across scales Scaling is one often-seen variation of images which is challenging for texture analysis. Empirically, the FENet enjoys good generalization to scale changes. An experiment is conducted on the GTOS-M dataset whose training/test set is divided into three subsets with resolution of 256× 256, 384× 384 and 512× 512 respectively; see Figure 4 for some examples. We train the ResNet-18 baseline, DEPNet and FENet only using the subset of resolution 384× 384. The ResNet-18 is used as the backbone of DEPNet as well as FENet. Then the models are used as the global descriptors on the test set of resolution 384× 384, as well as on both the training and test sets of other two resolutions. This is done by taking the features generated from the GAP layer in ResNet-18 baseline or the BLP layer in DEPNet and FENet. Figure 5 visualizes the descriptions via t-SNE [41]. It can be seen that the descriptions generated by FENet are more concentrated than those of the other two models.
See also Figure 4 for the visualization on the descriptions generated on specific images with different scales. The FENet outputs descriptions with higher degrees of invariance to scale changes. Such a
robustness is probably due to the robustness of the fractal-dimension-based features to bi-Lipschitz transform as well as the cross-scale analysis in FE.
Study on confusion cases The confusion matrices on the test set of GTOS-M are shown in Figure 6. We can see that the FENet is more effective than the ResNet backbone and DEPNet, with higher accuracy in classifying shale, stone cement and plastic cover. See Figure 7(a) for some sample images that confuse the ResNet backbone but correctly classified by FENet. The samples at the top row are misclassified into the class of the corresponding samples at the bottom rows. Each of such sample pairs share similar textons but with discrepancy on the distribution of textons. By leveraging fractal geometry analysis in FE, the FENet can utilize such discrepancy for discrimination, while the ResNet backbone cannot. However, from Figure 6, FENet still misclassifies sand to soil, stone asphalt to asphalt, and large limstone to cement. Some misclassified samples are shown in Figure 7(b). The sample images from the categories of ‘Sand’ and ‘Soil’ have very similar appearances and spatial distributions of patterns, which are hard to distinguish even by human. See also ‘Cotton’ versus ‘Wood’, as well as ‘Polished Stone’ versus ‘Ceramic’.

heading: 5 Summary
text: Feature aggregation is a key step in many CNN-based vision tasks, particularly for texture analysis. It aims at transforming feature maps into a single robust yet discriminative representation, by eliminating the dependency of features on spatial order while encoding the distributive characteristics of features. In this work, we fulfilled this goal by the proposed FE module, which leverages multi-fractal geometry and uses a hierarchical fractal analysis process for encoding the regularity of spatial arrangement in a feature map. The FE module can be easily incorporated into standard CNNs for end-to-end training. Building FE into ResNet, we developed the FENet for texture classification and retrieval. The experiments on six benchmark datasets have demonstrated the effectiveness of FENet.
While we only considered static textures in the experiments, the proposed FE can also benefit the representation and analysis on dynamic textures. We will investigate this topic in our future work. In addition to texture representation, the FE can also be incorporated into the CNNs in other visual tasks that need to utilize the spatial organization of local patterns. We also leave this to our future work.
Acknowledgments and Disclosure of Funding
Yong Xu would like to acknowledge the support from National Natural Science Foundation of China (Grant No. 62072188, 61672241), Natural Science Foundation of Guangdong Province (Grant No. 2016A030308013), and Science and Technology Program of Guangdong Province (Grant No. 2019A050510010). Yuhui Quan would like to acknowledge the support from National Natural Science Foundation of China (Grant No. 61872151), Natural Science Foundation of Guangdong Province (Grant No. 2020A1515011128), and CCF-Tencent Open Fund 2020.

Review summaries:
This paper proposes a new "global" pooling approach that aims at preserving detailed/fine-grained information from the representation. The proposed approach is based on Fractal Encoding that are implemented such that the overall representation is still trainable. The proposed approach is tested on the task texture representation.

The authors propose complementing global average pooling in convolutional networks with another pooling method that aims at better representation of texture. This pooling method builds on fractal geometry and is implemented by a fractal encoding module. The module first locally estimates fractal dimension, the estimates are binned and inside each bin the fractal dimension estimation is applied again. This results in pooling of one input channel into a fixed-sized vector. The operations, such as binning and counting, are adapted to make the module differentiable. The experiments show that both types of pooling modules are required for good performance.