Encoding Spatial Distribution of Convolutional Features for Texture Representation
Authors: Yong Xu
Feng Li
Zhile Chen
Jinxiu Liang
Yuhui Quan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Learning
Feature Encoding
Global Pooling
Texture Representation

Abstract: Existing convolutional neural networks (CNNs) often use global average pooling (GAP) to aggregate feature maps into a single representation. However, GAP cannot well characterize complex distributive patterns of spatial features while such patterns play an important role in texture-oriented applications, e.g., material recognition and ground terrain classification. In the context of texture representation, this paper addressed the issue by proposing Fractal Encoding (FE), a feature encoding module grounded by multi-fractal geometry. Considering a CNN feature map as a union of level sets of points lying in the 2D space, FE characterizes their spatial layout via a local-global hierarchical fractal analysis which examines the multi-scale power behavior on each level set. This enables a CNN to encode the regularity on the spatial arrangement of image features, leading to a robust yet discriminative spectrum descriptor. In addition, FE has trainable parameters for data adaptivity and can be easily incorporated into existing CNNs for end-to-end training. We applied FE to ResNet-based texture classification and retrieval, and demonstrated its effectiveness on several benchmark datasets.

Encoding Spatial Distribution of Convolutional Features for Texture Representation
Yong Xu Feng Li', Zhile Chen', Jinxiu Liang', Yuhui Quan'*
School of Computer Science and Engineering, South China University of Technology. China
2 Peng Cheng Laboratory, China
yxudscut. edu. en, @ertengg@@mmail scut.edue en, cazhi lechen@maiiiiiee scut. edu. en, cssherryliangdmail. scut, edu. cn, csyhquan@scut edu.. cn
Abstract
Existing convolutional neural networks (CNNs) often global average pooling (GAP) to aggregate feature maps into single representation. However, GAP cannot well characterize complex distributive patterns of spatial features while such patterns play an important role in texture-oriented applications, egg. material recognition and ground terrain classification. In the context of texture representa- reereeeeeeeiiie tion, this paper addressed the issue by proposing Fractal Encoding (FE), a feature encoding module grounded by multi-fractal geometry. Considering a CNN feature map as union of level sets of points lying in the 2D space, FE characterizes their spatial layout via a local-gles socalgllbbal hierarchical fractal analysis which examines the multi-scale power behavior on each level set. This enables a CNN to encode the regularity on the spatial arrangement of image features, leading to a robust yet discriminative spectrum descriptor. In addition, FE has trainable parameters for data adaptivity and can be easily incorporated into existing CNNs for end-to-end training. We applied FE to ResNet-based texture classification and retrieval, and demonstrated its effectiveness on several benchmark datasets.
Introduction
Ideal image descriptors for classification capable of characterizing both local features and global patterns of an image. Local features are about the local singularities or fine details of an image, such as edges and corners. Global patterns are about the distributions and spatial layouts of local image features. While local features are critical for visual recognition, global image patterns also play an important role in many scenarios, such as material recognition ground terrain classification and dynamics recognition [3]
Typical comvolutional neural networks (CNNs) for image classification connect convolutional layers with fully-connected (FC) layers. The FC layers encode the spatial layouts feature maps and act classifier. Such structure fails to produce robust global representation, FC layers capture spatial layouts based on "absolute" locations and their outputs are not invariant to spatial transforms such translation, rotation and scaling, that often occur in real scenarios.
To achieve the robustness to spatial transforms CNN-based representation, many existing studies (e.g. 4 5 6)) apply global average pooling (GAP) to feature maps, which averages all entries in feature channel so as to eliminate the dependency on spatial locations. However, the naive averaging operation totally disregards the spatial layout of a feature map. As result, when dealing with images of complex distributive patterns, GAP cannot encode sufficienlly discriminative information.
*Corresponding author: Yuhui Quan
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A robust yet discriminative global representation is particularly important for analyzing images of texture patterns 77]. since for such images, spatial transforms are one main source of variations and distributive characteristics are the key clues during analysis. In the context of texture analysis and related applications, there have been some efforts [ 9 10 2 12 on designing a more sophisti- cated global pooling module. Most of these works are about encoding the numerical distribution of features (e.g. soft histograms [8] 9 (12) into a single representation. Nonetheless, they may still lose the details on the spatially distributive characteristics of features which are useful for texture analysis.
Aiming at characterizing the spatially distributive patterns in feature maps to form robust yet discriminative texture representation, this paper proposed statistical global feature encoding module called FE (Fractal Encoding), which is grounded by fractal geometry [ 14.
Motivations
Fractal geometry has been widely used for texture analysis and synthesis; see e.g. 15
16
19 20)_ Its central concept is the fractal dimension that measures the distributive irreulaartyy of a spatial point set. Many studies have shown that natural textures carry fractal dimension information [15 16 21]. Since a CNN is a spatial feature extractor, its generated feature maps may also encode fractal dimension information that reflects the regularity of texture patterns. Inspired by this, FE describes the spatial layout of features terms of spatially distributive regularity using peeetal-dimesion-beee based statistics in a hierarchical manner.
17 18
Fractal dimension is established based on the measurement at scale 6. For each 6, object is measured in a way that ignores the irregularity of size less than 5. If the measurement is proportional to 8-8 for some and / is almost the same for small scales 6, we can compute its limit, which is called the fractal dimension. Let the n-dimensional Euclidean space TR? be covered by a mesh of n-dimensional hypercubes with diameter d. Given point set ECR" its fractal dimension denoted by ((E) is defined by [14]:
log. NV d)
B E
lim d=+
log d
where (E, d) is the number of mesh hypercubes that intersect E.
It can be seen fractal dimension is about cross-scale examination in terms of power law. This makes it possible to encode additional discriminative information on spatial layouts that ignored by the simple number counting operations in histogram. For simplicity, consider the fractal dimensions on binary images. The white/black pixels on a binary image can be viewed as a set of points on a 2D grid. See Figure I for three numerical examples, where the fractal dimensions are different between the grid and the two bars, while the histograms of all the sample images in terms black/white pixels are bbaded
the same. If we view those sample images specific feature patches, the statistics from FE can well distinguish the spatial feature organization of those two types of textures.
In addition to discriminability fractal dimension is theoretically invariant to spatial bi-Lipschitz transform 21 20 a general transform that covers a wide range of often-seen image transformations. Note that the influence of the spatial transforms of an image are likely to propagate to its feature maps. Then the FE driven by fractal dimensions can enjoy the robustness in this case.
1.80
1.62
1.62
Figure I: Fractal dimensions (numbers in the second row) calculated by the scheme of [22 on a grid image and two bar images. The white pixels are viewed set of spatial points on 2D grid. If we view the binary images as some texture features (e.g. feature patches from a CNN), then fractal dimensions can well distinguish the spatial distribution of the two types of texture features.
The key part in FE is the fractal analysis pooling (FAP). For higher discriminability. FAP leverages multi-fractal geometry (23 to form multiple fractal-dimension-beee based statistical quantities. This leads to local-global hierarchical fractal analysis process. Concretely, FE looks at feature map as an union of point sets. The point sets are the level sets under a local point-wise categorization function D(-). The FE calculates single (global) -actal-dimeesion baase statistical quantity
Main idea

---

every level set:
f(D) ((E)), where E {æ E B: D
(2)
Encouraged by the robustness of fractal dimension to spatial transforms and feature value scaling, we introduce the function D based on local fractal dimension [66 (also called Hölder exponent [44]
defined the power D(-) of U (B (x) x p((()) for EE, where B (æx r) is hypercube with center æ and diameter 2r, and U is some measurement. It can be seen that the local fractal dimension
has similar form as the global one in ①
log((B((xrr)
((x)
lim 0-0
-logr r
In other words, (xx) characterizes the local power law around E under measurement U.
The above local-global hierarchical fractal analysis process involves some discrete operations (e.g. box counting) that are non-differentiable, which creates obstacles for back propagation and model training. By appropriate design with soft relaxations, we implement FE using common operations of CNNs, so that it can be end-to-end trained on top of any standard CNN and deal with images of varying sizes. In addition, FE designed with trainable parameters which can be joindy learned with the backbone network for better adaptivity to data, as well as for automatically setting some hyper-parameters which may be difficult to determine but critical to the performance in practice.
Since FAP and GAP capture distributive features from different aspects respectively, FE combines them with the bi-linear pooling (BLP) (24) to exploit their complementary capabilities. The resulting module is deployed to the ResNet for texture classification and retrieval.
Contributions
This work contributes to deep learning and texture analysis in the following aspects. global feature encoding module which is capable
Firstly, we propose novel ftactal-analysise of encoding the spatially distributive regularity of features on feature map. Secondly, we build multi-fractal analysis into a standard CNN for boosting the power of deep-learning -asdd texture representation, which leads to lfte-of-the-art results in classification and retrieval tasks.
2 Related Work
Global pooling for deep texture representation The GAP is a prominent choice for global
feature aggregation; see e.g. [ 5 for its applications in texture representation. Squeezing a feature map by averaging, GAP is very lossy regarding spatial information. To address this, most existing studies drew inspirations from traditional global descriptors. Cimpoi et al. (25) and Song et al. [26]
used fisher vector encoding 27]. . but the resulting network is difficult for end-to-end training. Similar
to kernel-based descriptors 28). the BLP proposed by Lin and Maji (24) is based on the outer product of each pair of feature points, which encodes more information than GAP in terms of dependencies among channels. Dai el al. 29 combined BIP and GAP. However, both BIP and GAP are not trainable. Following pag-of-features [30], Zhang et al. [8] proposed trainable texture encoding (TE) layer which captures the numerical distribution of features based on dictionary learning. Xue et al. [9] combined TE and GAP via BLP. A similar module is used in Hu et al. [10), but it is placed on top of multiple convolutional layers besides the last one, with an FC layer for result ensemble. Motivated by eecality-constringe coding B1. Bu et al. [1I proposed a locality- aware coding layer which encodes features with considerations on their locality constraints. To make histograms applicable to CNNs, Peeples [12 proposed a histogram layer, which is used together with GAP. Chen el al. [32 exploited the soft histogram of cross-layer local fractal dimensions for feature encoding.
The proposed FE in this work distinguishes itself from the aforementioned methods by encoding the regularity on the spatial distribution of convolutional features, providing additional discrimination.
Deep CNNs for texture classification In addition to global pooling, there are also some studies focusing other aspects of scep-learning-bee tions among the spatial contexts corresponding to visual attributes of texture images, Zhai et al. [33] proposed multiple- -attiiple-atrributeee -ttribbute-perceieeel network with multi-branch architecture. To efficiently de- scribe the spatial organization of complex texture, Zhai et al. [34] proposed to learn the dependencies among texture features with a self-attention-like module, with very impressive results achieved.
texture classification. To utilize the inherent correla-
Fractal geometry for texture representation Fractal analysis has a long history for texture repre- sentation. Previous methods mostly employ fractal dimensions (implemented with various numerical
3

---

calculation schemes) as local appearance descriptors [6] global distribution descriptors [5] 21 These fractal- based descriptors are handcrafted and not adaptive data. In comparison, our work forms beep-learning-bane fractal analysis model which can be end-to-end trained from data for better adaptivity.
3 Fractal Encoding: Pooling with Hierarchical Fractal Geometry
The proposed FE module processes a feature tensor from the CNN backbone with two paths that perform GAP and FAP receptively. An FC layer is attached behind GAP align GAP to FAP in terms of output size, magnitude and semantics. An upsampling layer is applied before FAP for higher spatial resolutions of feature maps so make FAP more stable. The results from the two paths integrated by BLP followed by an FC layer. See Figure for the pipeline of the FE and FAP.
The FAP first compresses the input feature tensor Co channels by the new feature tensor, FAP processes it slice by slice and then concatenates the results over all slices. By viewing the entries fractal dimension analysis on it, i.e., FAP groups the points based on local fractal dimension and calculates fractal -inittal-dminioon--eo saaeeesio----eed statistical quantity on each group. Accordingly, there are three main blocks in FAP, as described in the next subsections.
block.
feature slice set of points in R2? FAP runs hierarchical
Input
FAP
LDEB
Image
U.
Feature
@
Backbone CNN
S C..
Urs
LSF
Slice-wise Processing
Convix1
BN
Feature
Input
Gre
U
Output Density Map
Learnable Kernals
ReLU
Gr. Gry
PGB
LDEB
(D(..) ..)
FE
(D(..w) -a)²
UP
BN
GAP
(DG.w -00
PGB
FAP
Input Density Map
Learnable Code Words
FC
BN
Membership
GDCB
GDCB
L2 Norm FC
L2 Norm FC
Concat
E...
No
LSF
A Output Value
Output
Membership
MasPool dy
Predicted Label
Description
Figure 2: Diagram of proposed fractal encoding (FE) module.
3.1 Local Dimension Estimation Block (LDDE))
The LDEB calculates a woiet-wise local fractal dimension map D Hxx from the input feature slice X E R
LDEB X R Hxw D e HWW
The calculation based on where one key computing the measurement U1 i.e., forming a series ER HR - 1,2,
of measurement maps: U, E
The measurement 24 in B) calculates value on local block B(x,r) for every æ. For feature slice, local block can be viewed as local patch of size r x in the slice, and lí is essentially calculated on all the patches sampled with a sliding window on X, which can be modeled by convolution process. We define the measurement to be learnalle for better data adaptivity. Since the measurement defined on multiple scales, we simulate this by a multi-scale convolution:
U, G. X, T= 1.2,...
(5)

---

where @ denotes the discrete convolution operation, and G, R'x is an r x kernel learned together with the model.
The numerical implementation of (3 is done by the least-squares fitting (LSF) on U, versus r in the log-log coordinate system. A series of measurement maps (U,, 11, .T2) are first
calculated, where r1 < < are finite sequence of predefined ordered positive radii. Then the estimation of local fractal dimension map in (3) becomes an LSF problem:
min D.P
E (Dw,, w. h) logr, + P(w, h) - log Ur, (w. h))²
(6)
where P(w, ht) denotes the bias term in the fitting. Setting the derivative of 6 w.t.. D and to zero respectively. we have
(Dww, h log + P(w, ht) logr, log r, r,log Ur. (w, h)) (D(w, h) log + P(w, h) - log Ur, (w, h)) 0
Then, the local fractal dimension map D E WxH computed as the solution to 7
ΣΣ logr log U., (w, h) (Σ²-) logrz
(Σe=) log Ur, (w, h)
D(w,)) h))
Z Σ 2 ==1 log² r= 2 logr=
3.2
Point Grouping Block (PGB)
The PGB categorizes the entries on the feature slice based on the map D output by the LDEB. Let
E ... W} (1,..., .H} R2?
(9)
D. = {(h, w) E E. D2 S +Dh, w) Dx+1):
(10)
for k 1.... K, given the intervals [D1, D2), [D2, Dz). [0x. DK+1] where Dj E IR for all k. The set E5, can be represented by binary map Es, the same size as D, where EB, (h, w) = 1
Dr S DDh,, w) < Dx+1 and ( otherwise.
The setting of intervals [Dr.
K can have noticeable impact on the accuracy of multi-fractal analysis (23. However, they non-trivial to determine, as the range of D is uncertain and uniform intervals are often not optimal. In addition, grouping with hard assignment may cause noticeable quantization errors (35), well optimization difficulty in back propagation during model training. To address these issues, we implement the grouping function using a soft assignment scheme [35) with trainable quantization intervals. Instead of using binary values, the soft assignment uses real values in [0, 1] for defining Ez, (h, w) to indicate the degree of membership. Let {c& R, h 1.... K) denote set of trainable interval anchors. For each Cr, define its corresponding membership map E., E [0, 10 by
1...........
exp( 88 D w. h) 22
Ecg (w, h))
(11)
Σ exp(eppppp 8k D (w, h) cx)
where {st ] is a set of learnable weights. It can be implemented via Softmax. To summarize, the PGB takes local dimension map D as input, and generates series of soft membership maps:
PGB D DER D
EEe E [0, 1x Jk=1
(12)
3.3 Global Dimension Calculation Block (GDCB)
The GDCB accepts (E.,, )k_ from the PGB as input, calculates fractal-dimension-bassee quantity 3 on each E., individually, and finally concatenates 3&s as output:
GDCB (E., E [0, [0, J + 3 (8,, B2, ... ,,x) E IR
(13)
The quantity 3g calculated based on ( Similar to the computation on D, Equation can be numerically implemented by using bi-log LSF. However, the box counting operation in is about
5

---

counting the patches with non-zero entries, which is non non-differentiable and thus makes training infeasible. Also recall that E., is soft membership map with values in [0. 1] instead of binary ones. It is very likely that Ee, contains no zeros and the box counting is inapplicable in this case.
Based on above considerations, we replace the box counting N in (I) by the sum of the maximal values from all patches of size d x d in Beg. When Es, is binary, it equals to box counting. The new scheme can be efficiently implemented by stride-1 max pooling with window size d x d and summation. By varying d = di, dr, we have the sums denoted by , Similar to (7) Bg is then calculated as the LSF solution to the modified (I)
T TE log delog N;
(Ei log ds)
(Ern log Nog.Ni))
Bx
(14)
TE log² de- - (Ei= log de)
4 Experimental Evaluation
4.1 Deployment of FE Module and Setting of Experiments
The FE module can be easily incorporated into a CNN by porting it on top of some convolutional layers. To evaluate the effectiveness of FE, we insert it next to the last convolutional layer of the ResNet (36), an often-used backbone (e.g. [8] 91.. The resulting CNN is referred to as FENet. Inheriting the advantages from FE, FENet can deal with flexible image size and be end-to-end trained.
We apply FENet to texture classification and texture retrieval on six benchmark datasets, including GTOS 22. GTOS-M 19. KTH [37. MINC (38, DTD [39] and FMD [. Same as existing work, we use the provided split schemes on GTOS, MINC and DTD, and random 10 splits on KTH-TIPS2b and FMD with recommended split sizes. The mean and standard deviation of classification accuracies over all splits are calculated. The results are reported using two runs on GTOS-M and five-time statistics on other datasets.
Based on the empirical parameter setting in fractal analysis previous studies 21 35 we set Co = 3 in GAP, r = 1,2...., 6 in LDEB, K = 16 in PGB, and d, 2. 6 in GDCB. The output sizes of the two paths in FE are both set to 48. Following 34 81 we use R8 and ResNet-50 as the backbone respectively. On all the datasets, FENet is trained with the cross-entropy loss via the momentum SGD optimizer with default settings and 30 epochs. The batch size set to 16 on FMD, 32 on KTH, and 64 on the other four datasets. The learning rate is initialized to le-3 on FMD, -3 on MINC, GTOS and GTOS-M, and lE on KTH and DTD datasets, with cosine decay every 10 epochs. The ResNet backbones are initialized with the pre-trained models on Imagenet. The (r in (5) are initialized as Gaussian kernels with bandwidth 1. Other parameters of FENet initialized by Xavier (40). Data augmentation via horizontal flipping and random cropping to 224 x 224 is applied. Our FENet is implemented with PyTorch 1.7, and the code will be released at the website: https: htt.gggggtttttt.e./eeeee thub www.tttwww..cceeeeeeeeeeeeeeeeee csf engli/FEEet All the experiments were run a single
www
Titan XP GPU.
4.2 Results in Texture Classification
Our FENet is compared with several recent deep models for texture classification, which include DeepTEN 8. DEPNet [9) LSCNet [1], MAPNet [33] DSRNet (34) and HistNet (12). The results on the six datasets are listed in Table 1 The results of the compared methods are directly quoted from existing literature whenever possible, or left blank otherwise.
It can be seen from Table I that FENet performs the best KTH, FMD, MINC and GTOS in terms of mcan accuracy, no matter using ResNet-18 18 backbone or ResNet-50 backbone. On GTOS-M, the FENet is the best performer with ResNet-18 18 backbone.
On DTD, the FENet performs better than others except DSRNet and MAPNet. The reason why FENet shows unsatisfactory performance on DTD is probably that, the labels for the textures in DTD are based on describable subjective semantics, which are quite different from other texture datasets whose labels are based on objective materials. Even in such a case, our method still applies and provides a good texture descriptor (e.g. superior performance to DeepTEN, DEPNet and HistNet). Recall that MAPNet is developed based on multiple attribute perception (MAP) which is more suitable for

---

Table 1: Classification accuracies (%) in the form of "me. Best results boldfaced. The
precision varies as the results are quoted from different works.
Method
DTD
KTH
FMD
MINC
GTOS
GTOS-M
DeepTEN
76,12 82.18
DEPNet
ResNet-118
LSCNet
76.3
MAPNet
69.5 = 0.8 0.8
80.8
80.3
82.98 + 1.6
DSRNet
71.2 0.7
81.8 1.6
81.3 1.3+00.
81.0
83.65 + 1.5
HistNet
79.75 0.8
Ours
69.59 0.04
86.62
0.08
82.26 0.29
80.57
0.10
83.10 0.23
85.10 0.36
DeepTEN
69.6
82.0 + 3.3
80.2 0.9
81.3
84.5 + 2.9
DEPNet
73.2
82.0
ResNer-50
L.CNet
81.2
MAPNet
76.1 0.6
84.5 1.3
85.2+
84.7 + 2.2
86.64 1.5
DSRNet
0.6
85.9 1.3
86.0
85.3 2.0
87.03 + 1.5
HistNet
72.0 - 1.2
82.4 1 0.3
Ours
74.20 0.10
88.24 0.23
86.74 0.19
83.98 0.14
85.71 0.06
85.20 1 0.43
the subjective attributes DTD (similar reason for DSRNet), while the spatial regularity captured by our FE more suitable for encoding the characteristics of objective materials (e.g. the superior performance of our method to MAPNet and DSRNet on other datasets).
is worth mentioning that the training dataset GTOS-N M is divided into three subsets with small, medium and large scales respectively. It is shown in [9] that the performance of deep CNNs such as DeepTEN using the single-resolution subset rather than the multi-resolution ones of GTOS-M will have a noticeable accuracy decrease. In comparison, our FENet has no performance decrease when trained on single-scale data, and the results FENet on GTOS-M Table are actually based on only the medium-scale training data. See Section 4.5 for more analysis.
also mentioning that the performance improvement of DSRNet and MAPNet is less than 1% in most cases, which is so significant and seems to saturate. In comparison, in the case of ResNet- 18, there are three datascts where our method has accuracy improvement larger than 1% and even over 4.5%, and on the remaining dataset the improvement is nearly 1%.
4.3 Results in Texture Retrieval
The global representation to the classification module in FENet can be used as texture descriptor. Its effectiveness is evaluated by the retrieval task as follows. An image sample is represented by its corresponding feature tensor output by the last pooling layer (i.e. after BLP). Each sample in the test dataset is used as query to search the remaining samples, and its top- R similar samples in the dataset are collected in terms of cosine similarity. The average precision and recall over all query samples are calculated.
By varying R from to the number of searched samples, we obtain the precision-recall (PR) curves and the corresponding mean average precision (mAP), which are shown in Figure 3 can be seen that FENet achieved the best retrieval results among the compared methods on GTOS-M, KTH and FMD datasets. Such results have demonstrated the effectiveness of the texture feature representation leamned in FENet.
4.4 Effectiveness of FE Module
Ablation study on FE module in FENet To demonstrate the effectiveness of FE in FENet, form a baseline ("w/o FE Module") from FENet by removing its FE module as well as the BLP layer and adapting the dimension of the related FC layer. Similarly, we form another baseline ("w/o GAP") by removing the GAP layer in FENet. These two baselines are trained in the same way as FENet.
The results of the baselines are compared to FENet GTOS-M and KTH datasets in Table 2 It can seen that the performance decrease caused by the removal of FE Module significant on GTOS-N M and KTH. This suggests that the regularity of spatial distribution captured by FE Module

---

1.0
0.8
0.8
0.8
precsso0
0.6
Drecsion
F0.6
0.66
Ours-0.8831
Ours-0.8792
Ours-0.9648
0.2
DEPNet-0.8522
0.2
DEPNet-0.8400
0.2
DEPNet-0.7999
DeepTEN-0.8358
DeepTEN-O.8a
DeepTEN-0.9299
0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0..0
0.2
0.4
0.6
0.8
1.0
recall
recall
recall
(a) GTOS-M
(b) KTH
(c) FMD
Figure 3: PR curves of several methods in texture retrieval. The values in legends are mAP.
plays a more important role. When we remove the GAP FE, performance decreases moderately. All these results have demonstrated the effectiveness of FE Module in FENet.
Table 2: Classification accuracy (%) of FENet with ablations.
Dataset
Backbone
w/ FE Module [FENet]
w/o Module
w/o GAP
GTOS-M
ResNet-18
85.10 + 0.36 86.62 = 0.08
82.29 0.30
83.86 0.64
KTH
ResNet-18
83.25 + 0.76
86.43 + 0.07
GTOS-M
Re
85.20 +: 0.43
82.57 0.65
84.33
KTH
Re
88.24 + 0.23
83.82 + 0.25
87.59 + 0.15
Incorporating FE module into DeepTEN We also demonstrate the effectiveness of the FE module by plugging into DeepTEN g and examining the performance gain. Recall that the DeepTEN also uses a ResNet backbone, together with an texture encoding module (called TE) which acts soft histogram with learned bins. Similar to our FENet, the FE module (together with the upsampling layer in front of it) is inserted in parallel to the TE layer after the last RB of the ResNet backbone. The output of the module and TE module are integrated by BLP. The resulting model is trained with the same scheme the original work.
The results on the four datasets are listed in Table It can be seen that the introduction of FE module brings significant improvement to DeepTEN on GTOS-M, KTH and FMD, as well as noticeable improvement on MINC. Such improvement again demonstrated the effectiveness of the FE module. Indeed, the TE module in DeepTTEN is essentially a soft counting layer which only considers the quantitative distribution but ignores the spatial distribution of features in feature slices. The FE module provides complementary information to the TE module by characterizing the spatial organization of feature map based on multi-fractal analysis.
Table 3: Classification accuracy (%) of DecpTEN w/o and w/ FE module.
Dataset
Backbone
DeepT'NN
DeepTEN + Module
GTOS-M
ResNet-18
76.12
81.15 0.29
KTH
ResNet-50
82.00 + 3.30
86.93 0.30
FMD
ResNet-50
80.20 + 0.90
86.32 + 0.43
MINC
ResNet-50
81.3
83.62 0.11
4.5 More Analysis
Complexity and efficiency We compare the complexity of backbone CNN, DeepTEN, DEPNet and FENet in terms of the number of parameters and number of floating operations per second (FLOPS). We also compare their efficiency term of inference time on an input image of size 384 x 384. See Table for the comparison. Our FE module only introduces small number of

---

additional parameters to the backbone, and the complexity of FENet is comparable to that of other deep models under the same backbone.
Table Number of parameters, number of FLOPs and inference time on GPU of different models.
The values the left/right are for the time is tested on the input of size 384 384.
backbones respectively. The inference
Method
Parameters (1 x 109)
FLOPS (1 x 10")
GPU Time (s/100pics)
Backhone CNN
11.1924 23.5716
1.8185 4,1004
0.1387 0.3666
DeN
11.3736 / 23,9017
1.8229 / 4,11234
0.1165 0.3000
DEPNet
12.2679 / 25,4841
1.8333 / /4,1627
0.0751 0..160
FENet
11.5071 / 23,9323
1.8205 / /4.1159
0.1393 0.4530
Scaling is one often-seen variation of images which is challenging for texture analysis. Empirically, the FENet enjoys good generalization to scale changes. An experiment is conducted on the GTOS-M - dataset whose training/test set is divided into three subsets with resolution 256 x 256, 384 x 384 and 512 x 512 respectively; see Figure g for some examples. We train the ResNet- 18 baseline, DEPNet and FENet only using the subset of resolution 384 384. The ResNet- 18 is used as the backbone of DEPNet as well as FENet. Then the models are used as the global descriptors sets of other two resolutions. This is done by taking the features generated from the GAP layer in ResNet-18 baseline the BLP layer in DEPNet and FENet. Figure visualizes the descriptions via I-SNE [41]. It can be seen that the descriptions generated by FENet are more concentrated than those of the other models.
Generalization across scales
the test of resolution 384 x 384, well
both the training and test
M MA
25
$22632
$126522
384 X 384
(b) 512 512
(c) 256 X 256
(d) Res Feature
DEP Feature
Feature
Figure 4: Multi-scale sample images in GTOS-M and their features generated by Residual, DEP and FE modules. The ResNet1-18 backbone is used. Res Feature denotes the features from the backbone.
(a) Baseline (ResNet-18)
(b) DEPNet (ResNet-18)
(c) FENet (ResNet-18)
Figure 5: The t-SNE visualization results on the scale-variant images of GTOS-M.
See also Figure for the visualization on the descriptions generated on specific images with different scales. The FENet outputs descriptions with higher degrees of invariance to scale changes. Such

---

robustness is probably due to the robustness of the pocctaldiieessinn features to bi bi-Lipschitz transform as well as the cross-scale analysis in FE.
Study on confusion cases The confusion matrices on the test set of GTOS-M are shown in Figure We can see that the FENet more effective than the ResNet backbone and DEPNet, with higher accuracy in classifying shale, stone cement and plastic cover. See Figure a for some sample images that confuse the ResNet backbone but correctly classified by FENet. The samples at the top row misclassified into the class of the corresponding samples at the bottom rows. Each of such sample pairs share similar textons but with discrepancy on the distribution of textons. By leveraging fractal geometry analysis in FE, the FENet can utilize such discrepancy for discrimination, while the ResNet backbone cannot. However, from Figure 6 FENet still misclassifies sand to soil, stone asphalt to asphalt, and large limstone to cement. Some misclassified samples are shown in Figure Figure[(b). The sample images from the categories of 'Sand' and 'Soil' have very similar appearances and spatial distributions of patterns, which are hard to distinguish even by human. See also 'Cotton' versus 'Wood', as well as 'Polished Stone' versus 'Ceramic'.
(a) Baseline (ResNet-18)
(b) DEPNet (ResNet-18)
(c) FENet (ResNet-18)
Figure Confusion matrices on GTOS-M.
S. Limestone
Wool
Knitted
Fibcuss
Metal
Sand
Cotton
Polished Stone
Water
Marbled
Asphalt
Cotton
Woven
Maseed
Glas
Soil
Wood
Ceramic
Glass
Stained
(a) Samples confusing backbone (but not FENet)
(b) Samples confusing FENet (ResNet-
Figure 7: Confusion analysis. The samples at the top row incorrectly classified into the class of the corresponding samples at the bottom rows.
5 Summary
Feature aggregation a key step many CNN-based vision tasks, particularly texture analysis. aims at transforming feature maps into single robust yet discriminative representation, by eliminating the dependency of features on spatial order while encoding the distributive characteristics of features. In this work, fulfilled this goal by the proposed FE module, which leverages multi-fractal geometry and uses hierarchical fractal analysis process for encoding the regularity of spatial arrangement in feature map. The FE module can be easily incorporated into standard CNNs for end-to-end training. Building FE into ResNet, we developed the FENet for texture classification and retrieval. The experiments six benchmark datasets have demonstrated the effectiveness of FENet.
While we only considered static textures in the experiments, the proposed FE can also benefit the representation and analysis on dynamic textures. We will investigate this topic in our future work. In addition to texture representation, the FE can be incorporated into the CNNs other visual tasks that need to utilize the spatial organization of local patterns. We also leave this to our future work.
10

---

Acknowledgments and Disclosure of Funding
Yong Xu would like to acknowledge the support from National Natural Science Foundation of China (Grant No. 62072188, 61672241), Natural Science Foundation of Guangdong Province (Grant No. 2016A030308013), and Science and Technology Program of Guangdong Province (Grant No. 0119A0505101000 Yuhui Quan would like to acknowledge the support from National Natural Science Foundation of China (Grant No. 61872151), Natural Science Foundation of Guangdong Province (Grant No. 2020A151501 112)), and CCF-Tencent Open Fund 2020.
References
[1] Lavanya Sharan, Ruth Rosenholtz, and EH Adelson, Material perception: What can you see in a brief glance? Journal of Vision, 9:784-784, Aug 2010.
[2] J. Xue, H. Zhang, K. Nishino, and K. Dana. Differential viewpoints for ground terrain material recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2020.
[3] Isma Hadji Richard P Wildes. new large scale dynamic texture dataset with application convnet understanding. In Proceedings of European Conference on Computer Vision, pages 320-335, 2018.
[4] Vincent Andrearczyk and Paul F. Whelan, Using filter banks in convolutional neural networks for texture classification. Pattern Recognition Letters, 84:63-69, 2016.
[5] Shin Fujieda, Kohei Takayama, and Toshiya Hachisuka. Wavelet convolutional neural networks for texture classification. 2017.
[6] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[7] Li Liu, Jie Chen, Paul Fieguth, Guoying Zhao, Rama Chellappa, and Matti Pietikäinen. From bow to Two decades of texture representation for texture classification. International Journal of Computer Vision, (Nov):1-36, 2018
[8] Hang Zhang, Jia Xue, and Kristin Dana. Deep ten: Texture encoding network. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 2896-2905, 2017.
[9] Jia Xue, Hang Zhang, and Kristin Dana. Deep texture manifold for ground terrain recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 558-567, 2018.
[10] Yuting Hu, Zhiling I Long, and Ghassan AlRegib. Multi-level texture encoding and representation (multer) based on deep neural networks. In Proceedings of International Conference on Image Processing, 2019.
[1I] Xingyuan Bu, Yuwei Wu, Zhi Gao, and Yunde Jia. Deep convolutional network with locality and sparsity constraints for texture classification. Pattern Recognition, 91:34-46, 46, July 2019.
[12] Joshua Peeples, Weihuang Xu, and Alina Zare. Histogram layers for texture analysis. arXiv preprint arXiv:2001.00215. 2020.
[13] Benoit Mandelbrot. How long is the coast of britain? statistical self-similarity and fractional dimension. Science, 156(3775)663-66388 1967.
[14] Kenneth J Falconer and KJ Falconer. Techniques in fractal geometry, volume 16. Wiley Chichester (W. Sx.), 1997.
[15] Lance M Kaplan. Extended fractal analysis for texture classification and segmentation. IEEE Transactions on Image Processing, 8(11):1572-155 1585, 1999.
[16] Manik Varma and Rahul Garg. Locally invariant fractal features for statistical texture classification. In Proceedings of International Conference on Computer Vision, 2007.
[17] Yong Xu, Yuhui Quan, Haibin Ling, and Hui Ji. Dynamic texture classification using dynamic fractal analysis. In Proceedingy of International Conference on Computer Vision, pages 1219-1226. IEEE, 2011.
[18] Yuhui Quan, Yong Xu, and Yuping Sun. A distinct and compact texture descriptor. Image and Vision Computing, 32(4):250-255 259, 2014.

---

[19] Yuhui Quan, Yong Xu. Yuping Sun, and Yu Luo. Lacunarity analysis on image patterns for texture classification. In Proceedings of IELE Conference Computer Vision and Patter Recognition, pages 160-167, 2014.
[20] Hongteng Xu, Junchi Yan, Nils Persson, Weiyao Lin, and Hongyuan Zha. Fractal dimension invariant filternng and its enn-based implementation. In Proceedings of IEFE Conference on Computer Vision and Pattern Recognition, 2017.
[21] Yong Xu, Hui Ji, and Comella Fermüller. Viewpoint invariant texture description using fractal analysis. International Journal of Computer Vision, 83(1):85-100, Jun 2009.
[22] Mandelbrot and Benoit B. The fractal geometry of nature. American Journal of Physics, 51(4). 1998.
[23] Lars Olsen. Multifractal geometry. In Fractal Geometry and Stochastics II, pages 3-37. Springer, 2000.
[24] Tsung-Yu Yu Lin and Subhransu Maji. Visualizing and understanding deep texture representations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 2791-2799, 2016.
[25] Mireca Cimpoi, Subhransu Maji, and Andrea Vedaldi. Deep filter banks for texture recognition and segmentation. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pages 3828- 0228-3836,22 2015.
[26] Y. Song, F. Zhang, Q. Li, H. Huang, I., J. O'Donnell, and W. Cai. traalll- fisher vectors for texture classification. Proceedings of International Conference on Computer Vision, pages 4922-4930, 2017.
[27] Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. Image classification with the fisher vector: Theory and practice. International Journal of Computer Vision, 105(2):222 2252:422222222,2222222 2013.
[28] Liefeng Bo, Xiaofeng Ren, and Dieter Fox. Kernel descriptors for visual recognition. In Advances in Neural Information Processing Systems, pages 244-252, 2010.
[29] Xiyang Dai, Joe Yue-Hei Ng. and Larry S Davis. Fason: First and second order information fusion network for texture recognition. In Proceedings of IEEE Conference Computer Vision and Puttern Recognition, pages 6100 6108, 2017.
[30] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. discriminative framework for texture and object recognition using local image features. In Toward Category-Level Object Recognition, pages 423-442. Springer, 2006.
[31] Jinjun Wang. Jianchao Yang, Kai Yu. Fengjun Lv, Thomas Huang. and Yihong Gong. Locality-constraing constrained linear coding for image classification. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition, pages 3360- -367. IEEE, 2010.
[32] Zhile Chen, Feng Li, Yuhui Quan, Yong Xu, and Hui Ji. Deep texture recognition via exploiting cross-layer statistical self-similarity. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[33] Wei Zhai, Yang Cao. Jing Zhang, and Zheng-Jun Zha. Deep multiple-attriuut- pereeiiee network for real-world texture recognition. In Proceedings of International Conference on Computer Vision, pages 3612-3621. IEEE, 2019.
[34] Wei Zhai, Yang Cao, Zheng-Jur Jun Zha, HaiYong Xie, and Feng Wu. Deep revecler-revealed network for texture recognition. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11007-11016 IEEE, 2020.
[35] Yong Xu, Yuhui Quan. Zhuming Zhang, Haibin Ling, and Hui Ji. Classifying dynamic textures via spatiotemporal fractal analysis. Puttern Recognition, 48(10):323933244 2015.
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[37] Barbara Caputo. Eric Hayman, and P. Mallikarjuna. Class-specific material categorisation. Proceedings of International Conference on Computer Vision, volume 2, pages 1597 - 1604 Vol. 2, 2005.
[38] Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in context database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015.
12

---

[39] Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings IEEE Conference Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[40] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedffoward neural networks. Proceedings of International Conference on Artificial Intelligence and Statistics, pages 249-256. JMLR Workshop and Conference Proceedings. 2010.
[41] Van Der Maaten Laurens and Geoffrey Hinton. Visualizing data using t-sne. Journal Machine Learning Research, 9(2605):2579 2605, 2008.
13


Reviews:
[{"review_id": "h7KqiL7ILnH", "reviewer": "Reviewer_H7rW", "summary": "The authors propose complementing global average pooling in convolutional networks with another pooling method that aims at better representation of texture. This pooling method builds on fractal geometry and is implemented by a fractal encoding module. The module first locally estimates fractal dimension, the estimates are binned and inside each bin the fractal dimension estimation is applied again. This results in pooling of one input channel into a fixed-sized vector. The operations, such as binning and counting, are adapted to make the module differentiable. The experiments show that both types of pooling modules are required for good performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is new and provides improved performance for texture classification and retrieval. The method is relatively clearly explained, but there is definitely room for improvement, see detailed suggestions and questions below.\n\nThe question is whether the FE module has any application beyond texture-related tasks as it is known that convolutional networks, even without this module, focus too much on texture cues. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Suggestions:\n- Consider showing & exploring the errors/failures of the proposed method.\n- Fig. 2 would benefit from a larger font (\"Learnable Kernels\", \"Output Density Map\")\n- Please check if the leading term in the numerator and in denominator of (8) should be multiplied by Z.\n- Explain how where the parameters of the module selected (line 196).\n- Add the input image size used for Table 4.\n- Fig. 4 - labels for (b) and (c) seem to be flipped, label for (d) should probably read \"TE Feature\".\n- Fig. 4 - would it be possible to add one more panel for the baseline feature? And to all three \"Feature\" panels add the 384x384 series.\n- Fig. 4 - please enlarge the font in the \"Feature\" panels.\n- Fig. 5c - specify if this is FENet-18 or FENet-50.\n- Fig. 4 shows DeepTEN for comparison but Fig. 5 shows DEPNet for comparison. Consider using a consistent set of baselines.\n\nQuestions:\n- What are the differences between the t-SNE visualization in Fig. 5 and SFig. 4?\n- Why are DeepTEN and DEPNet selected as the most common baselines when Table 1 shows that their results are relatively poor? DSRNet would seem to make more sense.\n- Any comments on why is FENet clearly slower than DEPNet even if FENet requires less FLOPS than DEPNet?\n\nTypos:\n- L25: output -> outputs\n- L103: point -> points / term -> terms", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Uep0DA0TO53", "reviewer": "Reviewer_H23L", "summary": "This paper proposes a new \"global\" pooling approach that aims at preserving detailed/fine-grained information from the representation. The proposed approach is based on Fractal Encoding that are implemented such that the overall representation is still trainable. The proposed approach is tested on the task texture representation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, the paper addresses an interesting downside of using Global Average Pooling (GAP) in CNNs, however this downside is only specific to certain situations IMHO, so I believe that the proposed solution is not broadly applicable. The following points highlight some more specific pros and cons.\n\nPros:\n+ The paper is clear and well written. \n+ The proposed solution based on Fractal Encoding is well motivated and technically sound.\n+ Making sensitive hyper-parameters, such as quantization intervals, trainable is a good approach over using some ad-hoc solution.\n\nCons:\n-  Generally the empirical results do not seem to speak strongly in favour of the proposed approach. In particular, Table 1 is either below state-of-the-art or shows that the proposed approach is only marginally better (1% or less).\n- The discussion regarding the sub-optimal performance on DTD (LINES 216-221),  further highlights the niche application of the proposed approach. That being said, I wonder how/if the proposed approach would benefit dynamic textures (e.g. [a])? have the authors considered this? \n- In terms of the method itself, the proposed approach should in fact replace GAP and I don't see a striking complementarity between the two. This comment is further confirmed in Table 2, which shows that GAP only yields a marginal contribution. Can the authors further comment on that?\n\n[a] I. Hadji and R. P. Wildes,  \"A New Large Scale Dynamic Texture Dataset\nwith Application to ConvNet Understanding\", ECCV 2018", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the authors provided different ablations, they did not explicitly discuss the limitations of their approach. It would be useful to dedicate a small subsection to this as this would benefit progress on this domain.\nNegative societal impact is not applicable in my opinion.", "review_text": "", "extra_scores": {"ethical_concerns": "Nothing that caught my attention", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0_JIJj9vHCV", "reviewer": "Reviewer_wSwZ", "summary": "The authors propose fractal encoding (FE) module to encode spatial distribution of convolutional features for texture representation. The effectiveness of FE is shown with experiments about texture classification and retrieval.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed fractal encoding module is simple yet efficient (e.g., parameter number, Flops, and running time). It has trainable parameters for data adaptivity and also can be inserted into existing CNN structures.\nThe main results on texture classification and retrieval demonstrate the effectiveness of the proposed method.\nThe ablation study with visual results well shows the generalization across scales.\nThe paper writing and organization are good and easy for readers to follow.\nWould the authors provide the code of the method?\n\nAfter rebuttal\n\nThanks for providing the feedback. I also read comments from other reviewers and like to keep my original rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the important problem of texture representation in neural network models. The approach is interesting and well received by the reviewers, and I personally find that many industrial applications could benefit from it.

Decision: Accept (Poster)