# Mining Sufficient Knowledge via Progressive Feature Fusion for Efficient Material Recognition

Authors: Hongbin Zhang; Qipeng Xiong; Haowei Shi; Weinan Liang; Xiang Zhong; Zhiliang Zhu; Xiong Li; Donghong Ji
Journal: Scientific Programming
 (2021)

---

## Abstract

Material images are susceptible to changes, depending on the light intensity, visual angle, shooting distance, and other conditions. Feature learning has shown great potential for addressing this issue. However, the knowledge achieved using a simple feature fusion method is insufficient to fully represent the material images. In this study, we aimed to exploit the diverse knowledge learned by a novel progressive feature fusion method to improve the recognition performance. To obtain implicit cross-modal knowledge, we perform early feature fusion and capture the cluster canonical correlations among the state-of-the-art (SOTA) heterogeneous squeeze-and-excitation network (SENet) features. A set of more discriminative deep-level visual semantics (DVSs) is obtained. We then perform gene selection-based middle feature fusion to thoroughly exploit the feature-shared knowledge among the generated DVSs. Finally, any type of general classifier can use the feature-shared knowledge to perform the final material recognition. Experimental results on two public datasets (Fabric and MattrSet) showed that our method outperformed other SOTA baseline methods in terms of accuracy and real-time efficiency. Even most traditional classifiers were able to obtain a satisfactory performance using our method, thus demonstrating its high practicality.


---

## Body

## 1. Introduction

Material recognition [1–5] has become a popular topic in the field of computer vision (CV). It has significant value in many practical scenarios, such as scene recognition [6], industrial inspection [7], medical image recognition [8], and robot vision [9]. For example, self-driving cars or mobile robots need to determine whether an incoming terrain is asphalt, gravel, or grass. Cleaning robots must distinguish between wood, tiles, and carpets. Clearly, material is an important visual cue that is widely present on the surface of natural objects and has abundant visual diversity. However, it is difficult for humans to distinguish these subtle differences. A computer-aided material recognition model can address this issue well and applies to the aforementioned practical fields; however, it is still a great challenge, as material recognition is normally affected by several external factors, including light intensity, visual angle, shooting distance, and other conditions. Recently, feature learning has shown great potential for alleviating this problem.Tan et al. [10] used a self-designed scatterometer to measure the bidirectional reflection distribution function (BRDF) of four different textured fabrics and analyzed the effects of surface texture and illuminating wavelengths. On this basis, to accomplish effective material recognition, they proposed a fast but simple BRDF model. Han et al. [11] transformed the original stimulus into a differential excitation domain according to Weber’s law, and then they explored a local patch, called microtexton, in the transformed domain as the Weber local descriptor (WLD). Furthermore, the WLD space was characterized adaptively using a generative probability model, which can generate a more discriminative feature representation for material recognition. Tanaka et al. [12] used the depth distortion of time-of-flight measurements as features and achieved the material classification of a scene. Their method can classify visually similar objects. Erickson et al. [13] proposed a multimodal sensing technique to capture spectral and close-range high-resolution texture information. They combined this high-resolution texture information with the output from a head-mounted camera on a robot to achieve accurate material classification. These methods [10–13] mainly focus on capturing the visual variations in material images. However, obtaining such spectral reflectance is time-consuming and error prone.Recently, deep learning features have played an important role in material recognition. Shahriari [14] used deep neural networks with variable depths to learn the scales, orientations, and resolutions of texture filter banks for effective material recognition. The corresponding computational cost was highly reduced, and their method could extract very deep features through distributed computing architectures. Jiang et al. [1] also proposed a deep learning-based method that combines both traditional and dilated convolutional features to achieve pixel-level material recognition. In their method, heterogeneous convolutional features are used to remove artifacts. Schwartz Nishino [2] probed the human visual perception of materials. They compared pairs of image patches by asking some questions to complete the weak supervision required to build a set of attribute classifiers. Furthermore, they integrated their method into a convolutional neural network (CNN) that can simultaneously recognize materials and visual attributes. These deep learning methods [1, 2, 14] usually require massive amounts of data to train effective recognition models. Owing to the large cost of annotations, high-quality training samples are very scarce in the field of material recognition.Moreover, some state-of-the-art (SOTA) feature fusion methods, such as multiset discriminant correlation analysis (MDCA) [15], gene selection eXtreme gradient boosting (GS-XGBoost) [16], gene selection adaptive boosting (GS-AdaBoost) [16], and hierarchical multifeature fusion (HMF2) [17], have been proposed to make full use of the limited training data. The MDCA model uses a single-layer early feature fusion algorithm for material recognition. However, the discriminative information is not balanced between the input and output interfaces of the model. To address this problem, the HMF2 model implements multilayer early feature fusion using residual connections. It modifies the traditional MDCA model [15] and fully mines cross-modal knowledge among the heterogeneous features. More importantly, inspired by the well-known residual structure [18], a hierarchical feature fusion strategy was proposed in HMF2 to gather effective knowledge for material recognition. However, the HMF2 model requires numerous features and multiple fusion operations, which may slow down its real-time efficiency. Unlike these early feature fusion methods [15, 17], GS-AdaBoost and GS-XGBoost perform middle feature fusion. They both use traditional features, including scale-invariant feature transform (SIFT) [19], Gist [20], local binary pattern (LBP) [21], and deep learning-based VGG16 [22], to complete the material recognition. However, these two GS models ignore the beneficial early stage cross-modal knowledge among the heterogeneous features.The abovementioned feature learning approaches [1, 2, 10–17] have shown remarkable progress in material recognition. However, the knowledge achieved by a simple feature fusion method is insufficient to completely represent material images, and few studies have exploited the implicit but valuable knowledge learned through progressive feature fusion. To address these issues, we propose a novel but effective progressive feature fusion method called SECGS that can seamlessly combine early and middle feature fusions together for material recognition. In our model, “SE” represents the well-known squeeze-and-excitation network (SENet) [23]. This means that we adopt only the SOTA SENet features for progressive feature fusion. “C” indicates that the SECGS model first mines the implicit cross-modal knowledge through cluster canonical correlation analysis [24]. It is a very important early feature fusion operation, as it not only creates a group of deep-level visual semantics (DVSs) but also serves as a basis for the subsequent middle feature fusion. “GS” indicates that we further capture the feature-shared knowledge among the implicit cross-modal knowledge using gene selection. Finally, deep-level but sufficient knowledge is mined for training material recognition model.The main contributions of this study are summarized as follows:(1)
We proposed a novel progressive feature fusion approach that can mine sufficient and effective knowledge among the heterogeneous SENet features for material recognition.(2)
We conducted extensive experiments on two benchmark datasets. The results showed the superior performance of our model over other SOTA methods. With our approach, even traditional classifiers were able to achieve satisfactory performance. The code for our method is available athttps://github.com/Danicaghost/SECGS.(3)
We created multiple variants of the SECGS model. They are all effective for material recognition, which further validates the robustness of the proposed progressive feature fusion method.(4)
We developed a real-time material recognition system based on our model and deployed it on a laptop. All we need is a mobile phone with a camera to complete the material recognition. This helps us to narrow the gap between theoretical research and practical applications. A demo video has also been uploaded to the above GitHub link.


## 2. Related Works

### 2.1. Material Recognition

Material recognition is an important research direction in the field of CV. Some researchers have used spectral reflectance from material surfaces to characterize materials [10, 25–29]. However, the acquisition of spectral reflection is time-consuming and error prone, which may reduce the practicality of the material recognition model. Therefore, some researchers have focused on constructing local feature descriptors (such as WLD) [30], on using reflection-based edge features [31], or on adopting some CNN features. Feature learning helps in improving the accuracy of material recognition. For example, Cimpoi et al. [32] calculated the CNN-based Fisher vector (FV) [33] features and achieved the best performance on the well-known Flickr material database [34]. Bell et al. [35] combined a set of pretrained CNNs using a fully connected conditional random field model to predict the material label of each pixel. Schwartz and Nishino [2] introduced a material attribute-category CNN that considers the implicit attribute information for material recognition. Bruna Mallat [36] proposed scattering convolution networks (ScaNet) that take Gabor or Haar wavelets as filters for material recognition. Sifre Mallat [37] proposed a rotation- and scale-invariant feature extraction method for robust material classification. Inspired by ScaNet, Chan et al. [38] utilized local image patches as principal component analysis filters to complete the material recognition. The aforementioned studies [2, 32–38] mainly adopted a certain layer of the CNN model, without considering the implicit complementary information among the heterogeneous layers from a homologous neural network. This motivated us to mine the complementary knowledge among the heterogeneous SENet features, which helps improve the final recognition performance. Table 1 lists the recent major material recognition methods, which we roughly divided into four categories: “traditional feature”: traditional methods using mathematical calculations; “CNN-based feature”: methods using deep learning features; “ETE”: end-to-end methods; and “feature fusion”: feature fusion methods (see the next subsection to learn more about “feature fusion”).Table 1
Summary of recent material recognition models. Unit: %.
MethodCategoryDatasetAccuracyWLD [30]Traditional featureBrodatz dataset97.50KTH-TIPS2a dataset64.70Alda [31]Flickr material dataset44.60PCANet [38]MNIST98.56CUReT dataset99.61FC-CNN + FV-CNN [32]CNN-based featureFlickr material dataset82.40FC-CNN + FV-CNN [32]Describable texture dataset74.70FV-CNN [32]KTH-TIPS2b dataset81.80IFV + VGG [14]KTH-TIPS2b dataset81.50IFV + DFB [14]KTH-TIPS2a dataset88.60IFV + DFB [14]Flickr material dataset82.70IFV + DFB [14]Describable texture dataset80.60Ensemble 3 CNNs [35]ETEThe materials in context database83.50ScaNet [36]USPS dataset97.70MNIST dataset95.60SRP + multiscal train [37]KTH-TIPS dataset99.40SRP + multiscal train [37]UMD dataset99.70SDC + CRF [1]MINC dataset74.27GS-AdaBoost [16]Feature fusionFabric dataset79.52MattrSet dataset66.11GS-XGBoost [16]Fabric dataset84.20MattrSet dataset67.67HMF2 [17]Fabric dataset84.34MattrSet dataset72.20MDCA [17]Fabric dataset49.84MattrSet dataset65.36


### 2.2. Feature Fusion

As shown in Table1, feature fusion has shown great potential for material recognition. Currently, feature fusion methods can be divided into early, middle, and late fusions. Early feature fusion performs fusion at the raw feature level. For example, a group of heterogeneous features is first mapped into a semantic space. Then, early feature fusion is performed by feature summation or feature concatenation. Representative early feature fusion models include canonical correlation analysis [39], deep canonical correlation analysis [40], multiple canonical correlation analysis [41], cross-modal correlation learning [42], kernel discriminant correlation analysis [43], and consistent discriminant correlation analysis [44]. Early feature fusion is intended to mine low-level correlations among different features. Unlike early feature fusion, middle feature fusion builds a more powerful classification model using a group of linearly weighted features. Representative middle feature fusion models include multiple kernel learning [45], efficient range gene selection [16], and multiple kernel boosting [46]. Middle feature fusion attempts to mine the implicit feature-shared knowledge among the heterogeneous features. Finally, late feature fusion integrates the multiple predictions of a set of weak classifiers to build a more powerful classification model. Representative late feature fusion methods include AdaBoost [47], XGBoost [48], categorical boosting (CatBoost) [49], and light gradient boosting machine (LightGBM) [50]. Unlike early and middle feature fusions, late feature fusion takes full advantage of the complementarity among different classifiers.Recently, an increasing number of feature fusion methods have played an important role in material recognition (Table1). The MDCA model uses a single-layer early feature fusion method for material recognition [15]. However, the effective discriminative information is not balanced between the input and output interfaces of the MDCA model. To resolve this problem, the HMF2 model implements multilayer early feature fusion using residual connections [17]. It can fully mine the cross-modal knowledge among a group of heterogeneous image features. However, the HMF2 model requires numerous features and multiple fusion operations, which may slow down its real-time efficiency. Moreover, it ignores the feature-shared knowledge and, thus, may miss some important material knowledge for effective material recognition. Unlike the early feature fusion approaches, the GS-AdaBoost and GS-XGBoost models perform middle feature fusion using SIFT, Gist, LBP, and VGG16 to complete the material recognition [16]. They can fully capture feature-shared knowledge to improve the performance of the final recognition. However, these two GS models ignore the early stage correlations among the heterogeneous features. In other words, the aforementioned feature fusion methods usually employ a single and independent feature fusion strategy. Moreover, only a few studies have explored the type of progressive relationship between the different feature fusion approaches for more effective material recognition.In summary, a novel progressive feature fusion framework that can make full use of different fusion methods is required to resolve the abovementioned issue. The progressive feature fusion method can mine sufficient and effective knowledge among the heterogeneous SENet features for high-quality material recognition.


## 2.1. Material Recognition

Material recognition is an important research direction in the field of CV. Some researchers have used spectral reflectance from material surfaces to characterize materials [10, 25–29]. However, the acquisition of spectral reflection is time-consuming and error prone, which may reduce the practicality of the material recognition model. Therefore, some researchers have focused on constructing local feature descriptors (such as WLD) [30], on using reflection-based edge features [31], or on adopting some CNN features. Feature learning helps in improving the accuracy of material recognition. For example, Cimpoi et al. [32] calculated the CNN-based Fisher vector (FV) [33] features and achieved the best performance on the well-known Flickr material database [34]. Bell et al. [35] combined a set of pretrained CNNs using a fully connected conditional random field model to predict the material label of each pixel. Schwartz and Nishino [2] introduced a material attribute-category CNN that considers the implicit attribute information for material recognition. Bruna Mallat [36] proposed scattering convolution networks (ScaNet) that take Gabor or Haar wavelets as filters for material recognition. Sifre Mallat [37] proposed a rotation- and scale-invariant feature extraction method for robust material classification. Inspired by ScaNet, Chan et al. [38] utilized local image patches as principal component analysis filters to complete the material recognition. The aforementioned studies [2, 32–38] mainly adopted a certain layer of the CNN model, without considering the implicit complementary information among the heterogeneous layers from a homologous neural network. This motivated us to mine the complementary knowledge among the heterogeneous SENet features, which helps improve the final recognition performance. Table 1 lists the recent major material recognition methods, which we roughly divided into four categories: “traditional feature”: traditional methods using mathematical calculations; “CNN-based feature”: methods using deep learning features; “ETE”: end-to-end methods; and “feature fusion”: feature fusion methods (see the next subsection to learn more about “feature fusion”).Table 1
Summary of recent material recognition models. Unit: %.
MethodCategoryDatasetAccuracyWLD [30]Traditional featureBrodatz dataset97.50KTH-TIPS2a dataset64.70Alda [31]Flickr material dataset44.60PCANet [38]MNIST98.56CUReT dataset99.61FC-CNN + FV-CNN [32]CNN-based featureFlickr material dataset82.40FC-CNN + FV-CNN [32]Describable texture dataset74.70FV-CNN [32]KTH-TIPS2b dataset81.80IFV + VGG [14]KTH-TIPS2b dataset81.50IFV + DFB [14]KTH-TIPS2a dataset88.60IFV + DFB [14]Flickr material dataset82.70IFV + DFB [14]Describable texture dataset80.60Ensemble 3 CNNs [35]ETEThe materials in context database83.50ScaNet [36]USPS dataset97.70MNIST dataset95.60SRP + multiscal train [37]KTH-TIPS dataset99.40SRP + multiscal train [37]UMD dataset99.70SDC + CRF [1]MINC dataset74.27GS-AdaBoost [16]Feature fusionFabric dataset79.52MattrSet dataset66.11GS-XGBoost [16]Fabric dataset84.20MattrSet dataset67.67HMF2 [17]Fabric dataset84.34MattrSet dataset72.20MDCA [17]Fabric dataset49.84MattrSet dataset65.36


## 2.2. Feature Fusion

As shown in Table1, feature fusion has shown great potential for material recognition. Currently, feature fusion methods can be divided into early, middle, and late fusions. Early feature fusion performs fusion at the raw feature level. For example, a group of heterogeneous features is first mapped into a semantic space. Then, early feature fusion is performed by feature summation or feature concatenation. Representative early feature fusion models include canonical correlation analysis [39], deep canonical correlation analysis [40], multiple canonical correlation analysis [41], cross-modal correlation learning [42], kernel discriminant correlation analysis [43], and consistent discriminant correlation analysis [44]. Early feature fusion is intended to mine low-level correlations among different features. Unlike early feature fusion, middle feature fusion builds a more powerful classification model using a group of linearly weighted features. Representative middle feature fusion models include multiple kernel learning [45], efficient range gene selection [16], and multiple kernel boosting [46]. Middle feature fusion attempts to mine the implicit feature-shared knowledge among the heterogeneous features. Finally, late feature fusion integrates the multiple predictions of a set of weak classifiers to build a more powerful classification model. Representative late feature fusion methods include AdaBoost [47], XGBoost [48], categorical boosting (CatBoost) [49], and light gradient boosting machine (LightGBM) [50]. Unlike early and middle feature fusions, late feature fusion takes full advantage of the complementarity among different classifiers.Recently, an increasing number of feature fusion methods have played an important role in material recognition (Table1). The MDCA model uses a single-layer early feature fusion method for material recognition [15]. However, the effective discriminative information is not balanced between the input and output interfaces of the MDCA model. To resolve this problem, the HMF2 model implements multilayer early feature fusion using residual connections [17]. It can fully mine the cross-modal knowledge among a group of heterogeneous image features. However, the HMF2 model requires numerous features and multiple fusion operations, which may slow down its real-time efficiency. Moreover, it ignores the feature-shared knowledge and, thus, may miss some important material knowledge for effective material recognition. Unlike the early feature fusion approaches, the GS-AdaBoost and GS-XGBoost models perform middle feature fusion using SIFT, Gist, LBP, and VGG16 to complete the material recognition [16]. They can fully capture feature-shared knowledge to improve the performance of the final recognition. However, these two GS models ignore the early stage correlations among the heterogeneous features. In other words, the aforementioned feature fusion methods usually employ a single and independent feature fusion strategy. Moreover, only a few studies have explored the type of progressive relationship between the different feature fusion approaches for more effective material recognition.In summary, a novel progressive feature fusion framework that can make full use of different fusion methods is required to resolve the abovementioned issue. The progressive feature fusion method can mine sufficient and effective knowledge among the heterogeneous SENet features for high-quality material recognition.


## 3. Our Approach

The overall scheme of the proposed progressive feature fusion approach is illustrated in Figure1. In this section, we first present the heterogeneous SENet features. Five types of heterogeneous SENet features are used in our approach. Based on the SENet features, we then propose a progressive feature fusion approach. First, we mine the cross-modal knowledge among the heterogeneous SENet features using early feature fusion and generate a group of DVSs with more discriminant ability. With the use of permutation and combination, a total of 10 different DVSs are then generated. The 10 DVSs mainly extract the more effective discriminative information from the heterogeneous SENet features. The DVSs are more powerful than the original SENet features. Then, we mine the implicit feature-shared knowledge progressively among these DVSs using middle feature fusion. The feature-shared knowledge is used to train our material recognition model. “Feature-shared” means that discriminative information is complementary between different features. Hence, in this stage, feature-shared knowledge among different DVSs is shared with all features to complete the middle feature fusion and obtain better recognition performance. Finally, the training procedure is described below. In addition, in this study, we used two benchmark datasets, namely, the Fabric and the MattrSet, to complete all experiments. The Fabric dataset [51] is a fine-grained dataset and includes images of clothes made from any of these 9 materials: cotton, wool, terrycloth, fleece, nylon, silk, denim, viscose, and polyester. It contains 5064 images. The MattrSet dataset [16] is a coarse-grained dataset that includes images of bags (polyurethane (pu), canvas, nylon, and polyester) and shoes (pu and canvas). It contains 11,021 images (see Tables 2 and 3 for more details about these two datasets).Figure 1
Overview of the proposed model.Table 2
The fabric dataset.Table 3
The MattrSet dataset.
CategoryImageNumberBag_pu1982Bag_canvas1948Bag_nylon1764Bag_polyester1715Shoes_pu1757Shoes_canvas1855


### 3.1. Heterogeneous SENet Features

As discussed above, deep learning features [1, 2, 14] can better characterize images of materials. Current mainstream CNN models usually improve performance by using a spatial dimensional layer. Unlike the models in [52, 53], the SENet model [23] can adaptively recalibrate channel-wise feature responses by explicitly modeling the interdependencies between different channels. Moreover, it pays more attention to the features that contain the most effective information. Hence, the SENet features outperform the ResNet features in most CV tasks. Therefore, we chose SENet rather than ResNet as the basis for the subsequent progressive feature fusion (the results of extensive experiments also validated our selection, see Table 4). In this study, we used five types of heterogeneous layers of SENet, namely, SE-ResNet 50 (SR50), SE-ResNet 101 (SR101), SE-ResNet 152 (SR152), SE-ResNeXt 50 (SRXt50), and SE-ResNeXt 101 (SRXt101). They contain abundant but complementary semantic information. Therefore, SENet features can achieve better recognition performance. This means that these heterogeneous features point to the same or similar material knowledge. Considering these factors, we attempt to make full use of the complementary knowledge among the SENet features using the progressive feature fusion approach.Table 4
Comparisons of the different types of DVSs on each dataset. The best value of each column is shown in bold. Unit: %.
DatasetFeatureAAFMAAKNNLRSVMAdaXGBMattrSetDVS (re)55.6760.7119.5860.4661.9951.68DVS (re-se)64.5867.6468.5060.3166.0065.41DVS (se)65.4568.7069.8067.4268.6368.00ΔImprove17.534.19-5.3116.0310.665.68ΔImprove20.320.05-0.20-2.720.27-0.05ΔImprove31.191.111.104.392.902.28FabricDVS (re)42.3656.1246.2548.1956.7857.45DVS (re-se)86.2187.0178.4270.6874.1179.29DVS (se)88.9790.9785.3687.7187.5788.12ΔImprove13.984.280.000.047.443.15ΔImprove20.03-0.36-2.33−15.44−11.85−5.99ΔImprove32.793.604.611.591.612.84


### 3.2. Progressive Feature Fusion

Our progressive feature fusion approach is illustrated in Figure2. The heterogeneous SENet features point to the same or similar material knowledge. Inspired by MDCA [15] and HMF2 [17], we first attempt to obtain the implicit cross-modal knowledge from these SENet features using early feature fusion. We extract the average pooling layer (2048 dimensions) of the five heterogeneous SE modules introduced above. Then, we analyze the cluster canonical correlation among all the SENet features and generate 10 DVSs. These DVSs fully mine the effective discriminative information among the heterogeneous SENet features. This means that the semantic distance between the samples in the same class shrinks, whereas the corresponding semantic distance between the samples in different classes expands gradually. Hence, the classifier can better distinguish different material categories and obtain a better recognition performance (see Section 5.4).Figure 2
Overview of the progressive feature fusion approach.The material dataset is defined asD=xn,yn,xn∈Rk,yn∈C,n=1,2,…,N. Here, N is the number of samples. yn is the material label of the sample xn. C=c1,c2,…,cl is the set of all material labels, cr∈C, and l is the number of material categories. X and Y are the two heterogeneous SENet features. We divide the two SENet features into “X1∗, Y1∗” (for training) and “X2∗, Y2∗” (for testing).X and Y have classes cl, Tx=Xc1,Xc2,…,Xcl, and Ty=Yc1,Yc2,…,Ycl, where Xcl=X1cl,X2cl,…,XXclcl and Ycl=Y1cl,Y2cl,…,YYclcl represent the data of X1∗ and Y1∗ in class cr, respectively. ω and ν are the projection vectors of X1∗ and Y1∗, respectively. The correlation coefficient ρ between X and Y is computed as follows:(1)ρ=maxw,vw′CXYvw′CXXwv′CYYv.The covariance matricesCXY, CXX, and CYY are, respectively, defined as follows:(2)CXY=1S∑l=1l∑i=1Xcl∑j=1Yclxiclyjcl′,(3)CXX=1S∑l=1l∑i=1XclYclxiclxicl′,(4)CYY=1S∑l=1l∑j=1YclXclyjclyjcl′,where S=∑1lXclYcl is the total logarithm of the paired relationship between X1∗ and Y1∗.According toω and ν, X1∗, X2∗, Y1∗, and Y2∗ are transformed into a set of mapped feature matrices Xα1∗=w′X1∗, Xα2∗=w′X2∗, Yα1∗=v′Y1∗, and Yα2∗=v′Y2∗, respectively. Then, U and V are constructed according to the mapped feature matrices, as follows:(5)U=Xα1∗Xα2∗,V=Yα1∗Yα2∗.Finally, we fuseU and V to generate 10 DVSs for performing middle feature fusion progressively. Unlike in HMF2, we perform early feature fusion only once and only adopt the SENet features rather than numerous different features:(6)DVS=UV=Xα1∗Yα1∗Xα2∗Yα2∗,or DVS=U+V=Xα1∗+Yα1∗Xα2∗+Yα2∗.As shown in Figure2, to obtain the feature-shared knowledge from the DVSs, we perform middle feature fusion through gene selection. First, the DVSs are combined to form a group of DVS combinations. Second, we employ a general classifier to predict the estimated probability of each DVS in any DVS combination. Then, the corresponding gene selection weight of each DVS is calculated, and a sum function is used to fuse the weighted estimated probabilities. Finally, we use a max decision function to generate the final recognition result.DVS=DVS1,DVS2,…,DVSz is the set of DVSs, DVSi∈DVS. z is the number of DVSs. The effective range ERdr=rdr−,rdr+ of DVSi on the labeled samples cr is defined as follows:(7)rdr−,rdr+=μdr−1−prγσdr,udr+1−prγσdr,where rdr+ and rdr− represent the upper and lower boundaries, respectively, of the effective range of DVSi for the samples cr. udr and σdr represent the mean value and standard deviation of DVSi, respectively, for the samples cr. pr0<pr<1 is the estimated probability of the samples cr, and 1−pr reduces the impact of the standard deviation σdr on the upper and lower boundaries of the effective range. γ is a constant derived from Chebyshev’s inequality, which is equal to 1.732.We compute the overlap areaOLd and the corresponding coefficient OCd of DVSi. Hence, the gene selection weight wd of DVSi is calculated according to OCd. The probability pdk is weighted by this weight:(8)OLd=∑k=1l−1∑h=k+1lφdk,r,(9)φdk,r=rdr+−rdr−,rdr+>rdr−,0,otherwise,(10)OCd=OLdmaxrrdr+−minrrdr−,(11)wd=OCdsumOCk:1≤k≤z.Lastly, we use a max decision function to generate the final recognition result. The material label is determined according tomaxpd=maxsumpdk∗wd. In summary, the implicit complementary knowledge among the heterogeneous SENet features is fully mined by the proposed progressive feature fusion method. The SECGS model is shown in Algorithm 1.Algorithm 1: The progressive feature fusion algorithm: SECGS.
Input: image feature set: S1=SR50,SR101,SR152,SRXt50,SRXt101Output: material recognition results(1)
Repeat(i)
Select two features fromS1.(ii)
Divide the two features into the training and testing data.(iii)
Use the training data to perform a cluster canonical correlation analysis and to obtain the projection vectorsω and ν.(iv)
Useω and ν to weight the training and testing data, respectively; therefore, U and V are constructed using equation (5).(v)
Use equation (6) to obtain the corresponding DVSs according to the two features.Until any pair of features in S1 has been traversed.(2)
Obtain a DVS set:S2=DVS1,DVS2,…,DVS10(3)
Forj = 2 : 10Build a DVS combination setS3 by randomly choosing j DVSs from S2.Repeat(i)
Use a general classifier to calculate the estimated probabilities of thej DVSs.(ii)
Use thej DVSs and the corresponding estimated probabilities to obtain the gene selection weight wd.(iii)
Usemaxpd=maxsumpdk∗wd to generate the final material recognition result.UntilS3 is null.j=j+1(4)
End For(5)
Output the final material recognition results.


### 3.3. Training Procedure

Owing to its generality, the proposed SECGS model can use any type of general classifier to complete the final prediction. The proposed model was trained using four-fold cross-validation. In addition, the trained SECGS model was deployed on a laptop with the following configuration: Intel Core i7-7700 CPU (2.8 GHz), 16 GB of RAM, and GTX 1050 GPU. Hence, online real-time material recognition can be implemented easily on a laptop with a similar or higher configuration.


## 3.1. Heterogeneous SENet Features

As discussed above, deep learning features [1, 2, 14] can better characterize images of materials. Current mainstream CNN models usually improve performance by using a spatial dimensional layer. Unlike the models in [52, 53], the SENet model [23] can adaptively recalibrate channel-wise feature responses by explicitly modeling the interdependencies between different channels. Moreover, it pays more attention to the features that contain the most effective information. Hence, the SENet features outperform the ResNet features in most CV tasks. Therefore, we chose SENet rather than ResNet as the basis for the subsequent progressive feature fusion (the results of extensive experiments also validated our selection, see Table 4). In this study, we used five types of heterogeneous layers of SENet, namely, SE-ResNet 50 (SR50), SE-ResNet 101 (SR101), SE-ResNet 152 (SR152), SE-ResNeXt 50 (SRXt50), and SE-ResNeXt 101 (SRXt101). They contain abundant but complementary semantic information. Therefore, SENet features can achieve better recognition performance. This means that these heterogeneous features point to the same or similar material knowledge. Considering these factors, we attempt to make full use of the complementary knowledge among the SENet features using the progressive feature fusion approach.Table 4
Comparisons of the different types of DVSs on each dataset. The best value of each column is shown in bold. Unit: %.
DatasetFeatureAAFMAAKNNLRSVMAdaXGBMattrSetDVS (re)55.6760.7119.5860.4661.9951.68DVS (re-se)64.5867.6468.5060.3166.0065.41DVS (se)65.4568.7069.8067.4268.6368.00ΔImprove17.534.19-5.3116.0310.665.68ΔImprove20.320.05-0.20-2.720.27-0.05ΔImprove31.191.111.104.392.902.28FabricDVS (re)42.3656.1246.2548.1956.7857.45DVS (re-se)86.2187.0178.4270.6874.1179.29DVS (se)88.9790.9785.3687.7187.5788.12ΔImprove13.984.280.000.047.443.15ΔImprove20.03-0.36-2.33−15.44−11.85−5.99ΔImprove32.793.604.611.591.612.84


## 3.2. Progressive Feature Fusion

Our progressive feature fusion approach is illustrated in Figure2. The heterogeneous SENet features point to the same or similar material knowledge. Inspired by MDCA [15] and HMF2 [17], we first attempt to obtain the implicit cross-modal knowledge from these SENet features using early feature fusion. We extract the average pooling layer (2048 dimensions) of the five heterogeneous SE modules introduced above. Then, we analyze the cluster canonical correlation among all the SENet features and generate 10 DVSs. These DVSs fully mine the effective discriminative information among the heterogeneous SENet features. This means that the semantic distance between the samples in the same class shrinks, whereas the corresponding semantic distance between the samples in different classes expands gradually. Hence, the classifier can better distinguish different material categories and obtain a better recognition performance (see Section 5.4).Figure 2
Overview of the progressive feature fusion approach.The material dataset is defined asD=xn,yn,xn∈Rk,yn∈C,n=1,2,…,N. Here, N is the number of samples. yn is the material label of the sample xn. C=c1,c2,…,cl is the set of all material labels, cr∈C, and l is the number of material categories. X and Y are the two heterogeneous SENet features. We divide the two SENet features into “X1∗, Y1∗” (for training) and “X2∗, Y2∗” (for testing).X and Y have classes cl, Tx=Xc1,Xc2,…,Xcl, and Ty=Yc1,Yc2,…,Ycl, where Xcl=X1cl,X2cl,…,XXclcl and Ycl=Y1cl,Y2cl,…,YYclcl represent the data of X1∗ and Y1∗ in class cr, respectively. ω and ν are the projection vectors of X1∗ and Y1∗, respectively. The correlation coefficient ρ between X and Y is computed as follows:(1)ρ=maxw,vw′CXYvw′CXXwv′CYYv.The covariance matricesCXY, CXX, and CYY are, respectively, defined as follows:(2)CXY=1S∑l=1l∑i=1Xcl∑j=1Yclxiclyjcl′,(3)CXX=1S∑l=1l∑i=1XclYclxiclxicl′,(4)CYY=1S∑l=1l∑j=1YclXclyjclyjcl′,where S=∑1lXclYcl is the total logarithm of the paired relationship between X1∗ and Y1∗.According toω and ν, X1∗, X2∗, Y1∗, and Y2∗ are transformed into a set of mapped feature matrices Xα1∗=w′X1∗, Xα2∗=w′X2∗, Yα1∗=v′Y1∗, and Yα2∗=v′Y2∗, respectively. Then, U and V are constructed according to the mapped feature matrices, as follows:(5)U=Xα1∗Xα2∗,V=Yα1∗Yα2∗.Finally, we fuseU and V to generate 10 DVSs for performing middle feature fusion progressively. Unlike in HMF2, we perform early feature fusion only once and only adopt the SENet features rather than numerous different features:(6)DVS=UV=Xα1∗Yα1∗Xα2∗Yα2∗,or DVS=U+V=Xα1∗+Yα1∗Xα2∗+Yα2∗.As shown in Figure2, to obtain the feature-shared knowledge from the DVSs, we perform middle feature fusion through gene selection. First, the DVSs are combined to form a group of DVS combinations. Second, we employ a general classifier to predict the estimated probability of each DVS in any DVS combination. Then, the corresponding gene selection weight of each DVS is calculated, and a sum function is used to fuse the weighted estimated probabilities. Finally, we use a max decision function to generate the final recognition result.DVS=DVS1,DVS2,…,DVSz is the set of DVSs, DVSi∈DVS. z is the number of DVSs. The effective range ERdr=rdr−,rdr+ of DVSi on the labeled samples cr is defined as follows:(7)rdr−,rdr+=μdr−1−prγσdr,udr+1−prγσdr,where rdr+ and rdr− represent the upper and lower boundaries, respectively, of the effective range of DVSi for the samples cr. udr and σdr represent the mean value and standard deviation of DVSi, respectively, for the samples cr. pr0<pr<1 is the estimated probability of the samples cr, and 1−pr reduces the impact of the standard deviation σdr on the upper and lower boundaries of the effective range. γ is a constant derived from Chebyshev’s inequality, which is equal to 1.732.We compute the overlap areaOLd and the corresponding coefficient OCd of DVSi. Hence, the gene selection weight wd of DVSi is calculated according to OCd. The probability pdk is weighted by this weight:(8)OLd=∑k=1l−1∑h=k+1lφdk,r,(9)φdk,r=rdr+−rdr−,rdr+>rdr−,0,otherwise,(10)OCd=OLdmaxrrdr+−minrrdr−,(11)wd=OCdsumOCk:1≤k≤z.Lastly, we use a max decision function to generate the final recognition result. The material label is determined according tomaxpd=maxsumpdk∗wd. In summary, the implicit complementary knowledge among the heterogeneous SENet features is fully mined by the proposed progressive feature fusion method. The SECGS model is shown in Algorithm 1.Algorithm 1: The progressive feature fusion algorithm: SECGS.
Input: image feature set: S1=SR50,SR101,SR152,SRXt50,SRXt101Output: material recognition results(1)
Repeat(i)
Select two features fromS1.(ii)
Divide the two features into the training and testing data.(iii)
Use the training data to perform a cluster canonical correlation analysis and to obtain the projection vectorsω and ν.(iv)
Useω and ν to weight the training and testing data, respectively; therefore, U and V are constructed using equation (5).(v)
Use equation (6) to obtain the corresponding DVSs according to the two features.Until any pair of features in S1 has been traversed.(2)
Obtain a DVS set:S2=DVS1,DVS2,…,DVS10(3)
Forj = 2 : 10Build a DVS combination setS3 by randomly choosing j DVSs from S2.Repeat(i)
Use a general classifier to calculate the estimated probabilities of thej DVSs.(ii)
Use thej DVSs and the corresponding estimated probabilities to obtain the gene selection weight wd.(iii)
Usemaxpd=maxsumpdk∗wd to generate the final material recognition result.UntilS3 is null.j=j+1(4)
End For(5)
Output the final material recognition results.


## 3.3. Training Procedure

Owing to its generality, the proposed SECGS model can use any type of general classifier to complete the final prediction. The proposed model was trained using four-fold cross-validation. In addition, the trained SECGS model was deployed on a laptop with the following configuration: Intel Core i7-7700 CPU (2.8 GHz), 16 GB of RAM, and GTX 1050 GPU. Hence, online real-time material recognition can be implemented easily on a laptop with a similar or higher configuration.


## 4. Experiments and Results

In this section, we evaluate the effectiveness, efficiency, and robustness of our approach on two benchmark datasets: (1) the Fabric and (2) the MattrSet datasets.


### 4.1. Datasets

The Fabric dataset is a fine-grained dataset constructed by Kampouris et al. [51] using photometric stereoscopic sensors to collect clothing samples from physical stores and determine the final material according to the clothing label. It contains images of clothes made from any of these 9 materials: cotton, wool, terrycloth, fleece, nylon, silk, denim, viscose, and polyester. The Fabric dataset contains 1266 samples, according to different light intensities, visual angles, and other conditions. Each sample includes four images; thus, it has a total of 5064 images (Kampouris et al. cropped all the images to 400 × 400 pixels to avoid blurring the edges of the images). Table 2 provides more detailed information about the Fabric dataset.The MattrSet dataset [16] is derived from real commodities on the web and constructed under the guidance of an experienced material expert. It is a coarse-grained dataset that includes images of bags (polyurethane (pu), canvas, nylon, and polyester) and shoes (pu and canvas). It contains 11,021 images (the corresponding image resolution ranges from 123 × 123 to 4300 × 2867, which makes material recognition more challenging). Table 3 provides more detailed information about the MattrSet dataset.


### 4.2. Baselines

We used five general classifiers, namely, logistic regression (LR),k-nearest neighbor (KNN), support vector machine (SVM), AdaBoost (Ada), and XGBoost (XGB), to train the proposed SECGS model (e.g., “SECGS5-LR” means that the LR classifier uses five DVSs to implement middle feature fusion). We compared our model with the baseline models listed below. The goal of this study was to obtain a better recognition performance using progressive feature fusion. We aimed to improve the recognition performance of our model on fine-grained and coarse-grained material images. Hence, our work can be regarded as a type of fine-grained image classification task. Therefore, we also compared the proposed model with some recently proposed fine-grained image classification models:(1)
Some well-known late feature fusion models are as follows: AdaBoost [47], XGBoost [48], CatBoost [49], and LightGBM [50]. For a fairer comparison, we also used the DVSs generated by our method to train these models.(2)
Recently proposed material recognition models are as follows: GS-XGB [16], GS-Ada [16], SVM-S [17], MDCA-XGB [17], HMF2 [17], and Kampouris’s Albe + Norm [53].(3)
Recently proposed deep learning models are as follows: InceptionResNetV2 [54], DenseNet169 [55], SENet [23], and ResNeXt WSL [56].(4)
Some well-known fine-grained classification models are as follows: DCL [57], NTS-Net [58], PMG [59], WILDCAT [60], WSCNet [61], and WS-DAN [62]. All these models are trained by using the four-fold cross-validation with an initial learning rate of 1e −3, a batch size of 16 and trained for 40 epochs.(5)
Multiple variants of the SECGS model are as follows: we removed the “GS” component from “SECGS” to generate a new early feature fusion model called SEC. We then added an ensemble learning strategy (a type of late feature fusion) to the SEC model and created two variants of our model: SEC(s) and SEC(h). SEC(s) uses soft voting-based late feature fusion after the early feature fusion, whereas SEC(h) uses hard voting-based late feature fusion. Hence, the two models also use the progressive feature fusion method. We wanted to validate the robustness of this method and to demonstrate the effectiveness of the “GS” component.


### 4.3. Evaluation Metrics

We employed accuracy (Acc.) (equation (12)) and precision (equation (13)) to evaluate our model. TP indicates true positive; FP, false positive; TN, true negative; and FN, false negative. Moreover, we designed a new metric called the average accuracy for each feature (AAF) (equation (14)) to comprehensively evaluate each feature. Similarly, we propose a new metric called the average accuracy for each classifier (AAC) (equation (15)) to comprehensively evaluate each classification model. Finally, we used the mean average accuracy (MAA) metric (equation (16)) to evaluate the overall performance of the proposed model:(12)accuracy=TPall+TNallTPall+FPall+TNall+FNall,(13)precision=TPallTPall+FPall,(14)AAF=∑m=1NclassifierAccuracymNclassifier,(15)AAC=∑n=1NfeatureAccuracynNfeature,(16)MAA=∑n=1Nclassifier∑m=1NfeatureAccuracymnNclassifier×Nfeature.Moreover, as shown in equation (17), ΔAccuracy1 calculates the absolute performance improvement of the AAF metric of the SENet features relative to that of the ResNet features. This metric can evaluate the effectiveness of the SENet features. As shown in equation (18), ΔAccuracy2 calculates the absolute performance improvement of the AAF metric of the DVSs relative to that of the SENet features. This metric evaluates the effectiveness of the generated DVSs.(17)ΔAccuracy1=AAF−SENet−AAF−ResNet,(18)ΔAccuracy2=AAF−DVS−AAF−SENet.


### 4.4. Evaluation of the SEC Model

Table5 shows the accuracy of the SEC model that uses different DVSs. Table 6 shows the AAF and MAA values of the best ResNet features, SENet features, and DVSs. In this section, we discuss our experimental results in detail.Table 5
Recognition performance of the proposed SEC models on each dataset. The best value of each column is shown in bold. Unit: %.
DatasetDVSRecognition performance of each SEC modelKNNLRSVMAdaXGBAverageMattrSetSR50-SR15266.2368.9469.9767.2668.6868.22SR50-SR10165.4868.7269.7067.4968.4067.96SR50-SRXt10164.9168.8870.3067.4669.0268.11SRXt101-SR15265.1568.7369.6167.7968.8968.03SR101-SR15265.4468.7169.9367.2268.2967.92SRXt50-SR15265.4768.2269.2967.3068.5167.76SR101-SRXt10164.7468.7069.8267.4468.3867.82SR50-SRXt5064.2568.4469.6367.5968.8267.75SRXt50-SRXt10164.6868.5970.0767.6168.2967.85SRXt50-SR10164.4968.6369.4767.5268.2467.67ΔAccuracy116.1211.0743.8118.6014.4020.80ΔAccuracy20.831.071.084.442.822.05FabricSR101-SRXt10189.0891.1386.3087.4687.1688.23SRXt50-SR10189.0090.9886.0287.3288.1188.29SRXt101-SR15289.4491.0585.0786.9586.8587.87SRXt50-SRXt10188.7490.6085.6887.9187.5688.10SRXt50-SR15289.5590.8085.7688.1987.5488.37SR50-SRXt10187.9991.2383.3188.4588.1987.83SR50-SRXt5087.6090.8683.0887.9388.1187.52SR101-SR15288.5590.2884.5686.6586.0487.22SR50-SR10187.2290.4082.4287.6288.3187.19SR50-SR15287.9789.9381.3887.7287.9786.99ΔAccuracy147.7935.5434.5037.9836.6238.49ΔAccuracy22.343.353.611.501.622.48Table 6
AAF and MAA values of each feature on each dataset. In MAA values, the best performance of each column is shown in bold. In AAF values, the best value of each row is shown in bold. Unit: %.
DatasetFeatureAAFMAAKNNLRSVMAdaXGBMattrSetResNet48.1456.5224.8944.4351.3345.06SENet64.2667.5968.7063.0365.7365.86DVS65.4568.7069.8067.4268.6368.00FabricResNet38.3851.8446.2548.1549.3446.79SENet86.1887.3780.7586.1285.9685.28DVS88.9790.9785.3687.7187.5788.12According to ΔAccuracy1, the maximum improvement of the SENet features relative to that of the ResNet features on the Fabric dataset is 47.79%, and the corresponding average improvement is 38.49%. These results confirm that the SE module can depict the key visual characteristics of different materials and that the SENet features from the SE modules can be applied to different classification models. Similar results were also obtained on the MattrSet dataset, which confirms the robustness of the SENet features. Certainly, a larger performance improvement was obtained on the Fabric dataset owing to the fine-grained visual information contained in its samples.According to ΔAccuracy2, the maximum improvement of the DVSs relative to that of the SENet features on the Fabric dataset is 3.35%, and the corresponding average improvement is 2.48%. Similar performance improvements were also obtained on the MattrSet dataset. These results indicate that the proposed DVSs are effective for material recognition on any dataset and that the proposed early feature fusion method is effective and robust. Moreover, the DVS (SR50-SRXt101) achieved the best accuracy on each dataset. This indicates that sufficient but effective cross-modal knowledge can be mined among the heterogeneous SR50 and SRXt101 features. Owing to this characteristic, even traditional classifiers, including LR and SVM, were able to obtain satisfactory recognition performance. It is worth noting that the corresponding performance on the Fabric dataset was better than that on the MattrSet dataset because the former dataset contains much more valuable visual semantics, whereas the latter dataset contains certain noise (Tables 2 and 3).According to AAF and MAA in Table 6, we can obtain the overall recognition performance of different features from a statistical perspective. Therefore, many more meaningful conclusions can be drawn. According to AAF, the proposed DVSs easily outperformed the SENet and ResNet features on each dataset. A larger performance improvement was obtained on the Fabric dataset. Hence, the proposed DVSs are robust and effective for material recognition. The MAA metric further validates this conclusion. For example, on the Fabric dataset, a performance margin of 41.33% was obtained between DVS and ResNet. These results clearly show that there is indeed a large amount of complementary information among the heterogeneous SENet features and that the proposed early feature fusion method can capture such information to improve the recognition performance. In addition, even traditional models, such as LR, KNN, and SVM, were able to achieve better performance after using early feature fusion owing to the powerful discriminative ability of the proposed DVSs.In summary, the DVSs generated by the proposed early feature fusion method are effective and robust for material recognition, providing a solid basis for the subsequent middle feature fusion. Sufficient cross-modal knowledge among the heterogeneous SENet features was mined to improve the recognition performance. Moreover, all the SENet features obtained satisfactory performance, which firmly supports our original selection.


### 4.5. Evaluation of the SEC(s) and SEC(h) Models

In this section, we evaluate two variants of the SECGS model, namely, SEC(h) and SEC(s). We want to determine whether the voting strategy contributes to improving recognition performance. More importantly, we intend to validate the robustness of the proposed progressive feature fusion method. In addition, we want to demonstrate the effectiveness of the “GS” component from another perspective. Figure3 shows the AAC values of the top three DVSs combined with the corresponding voting strategy.Figure 3
Performance comparisons between the SEC(h) and SEC(s) models. (a) Fabric: SEC(h), (b) Fabric: SEC(s), (c) MattrSet: SEC(h), and (d) MattrSet: SEC(s).
(a)(b)(c)(d)As shown in Figure3(a), the SEC(h) model that uses SR101-SRXt101 with the top three classifiers achieved the best accuracy (91.51%) on the Fabric dataset. It outperformed the original DVS without voting by a large margin. Thus, we should carefully tune the number of classifiers for the final voting. Similarly, as shown in Figure 3(b), the SEC(s) model that uses SRXt101-SR152 with the top two classifiers obtained the best accuracy (93.19%) on the Fabric dataset. It also outperformed the original DVS without voting and the best SEC(h) model. Therefore, the soft voting strategy is more suited for the Fabric dataset.As shown in Figure3(c), the SEC(h) model that uses SR50-SRXt101 with all the classifiers obtained the best accuracy on the MattrSet dataset. It easily outperformed the original DVS without voting. Similarly, as shown in Figure 3(d), the SEC(s) model that uses SR50-SRXt101 with all the classifiers achieved the best accuracy. Unlike on the Fabric dataset, we require more classifiers to complete the voting-based late feature fusion on the MattrSet dataset owing to the much larger amount of noise information in its samples. More heterogeneous classifiers can make a more definite prediction. In addition, a larger performance improvement can be obtained on the MattrSet dataset if we use a soft voting strategy. This further indicates that the corresponding recognition of coarse-grained material images remains a significant challenge. In general, the soft voting strategy is also more suited for the MattrSet dataset.In summary, both the SEC(h) and SEC(s) models achieved satisfactory performance on each material dataset. This also validates the robustness of the proposed progressive feature fusion method from another perspective. Hence, the voting-based late feature fusion method is another useful substitute for the proposed middle feature fusion method. Unlike HMF2 [17], the SEC model only adopts the SENet features, which helps reduce its complexity. This is an important step in narrowing the gap between theoretical research and practical applications.


### 4.6. Evaluation of the SECGS Model

In this section, we perform a more comprehensive evaluation of the proposed SECGS model. Table7 shows the corresponding experimental results. We show the performance variations with the increase in the number of DVSs on each dataset. For example, “2-SVM” means that the SVM classifier uses two DVSs to implement the middle feature fusion . “Best Acc.” represents the best accuracy among all the DVS combinations. “Δmax” represents the corresponding performance improvement of the SECGS model compared with the best SEC model described above. Some important conclusions can be drawn from Table 7.(1)
On the MattrSet dataset, the corresponding performance rank of the different classifiers in descending order was “SVM > XGBoost > LR > AdaBoost > KNN.” On the Fabric dataset, it was “LR > KNN > XGBoost > AdaBoost > SVM.” These can help us with the selection of the optimal classifier for the SECGS model. The SECGS model prefers SVM on the coarse-grained MattrSet dataset, whereas it prefers LR on the fine-grained Fabric dataset. A similar result can be seen in Table5. Thus, owing to the progressive feature fusion method, sufficient and effective knowledge was mined, enabling even the traditional classifiers to achieve satisfactory recognition performance.(2)
We found that, regardless of the number of DVSs, the SVM classifier always achieved the best performance on the MattrSet dataset and the LR classifier on the Fabric dataset. These findings are further illustrated in Figure4. They indicate that the most suitable classifier should be chosen carefully for different material datasets to achieve the largest performance improvement.(3)
According to Δmax, the proposed middle feature fusion approach outperformed the voting-based late feature fusion approach. The feature-shared knowledge was fully mined using our progressive feature fusion model. Evidently, the proposed “GS” method is more effective than the voting-based late feature fusion method. Unlike the SEC model, our model adopts only one classifier. In addition, larger performance improvements were obtained on the fine-grained Fabric dataset. A similar conclusion can be drawn from the above subsection “Evaluation of the SEC Model.” Interestingly, owing to its different implementation mechanisms, the proposed SECGS model will be a powerful supplement to the well-known HMF2 model (Table 8).(4)
More DVSs did not mean better recognition performance. For example, the proposed SECGS model that combines the SVM classifier and three DVSs achieved the best performance (71.42%) on the MattrSet dataset. Sufficient feature-shared knowledge among the three DVSs was better mined by our model, which improved the final recognition performance. A similar result can be seen in Figure4. Hence, the proposed middle feature fusion strategy only requires careful tuning of the number of DVSs to capture sufficient feature-shared knowledge for more effective material recognition.Table 7
Recognition performance of the SECGS model on each dataset. The best value of each column is shown in bold. Unit: %.
DatasetSECGSBestAcc.SECGSBestAcc.SECGSBestAcc.SECGSBestAcc.SECGSBestAcc.MattrSet2-KNN68.892-LR70.372-SVM71.252-Ada69.522-XGB70.563-KNN69.563-LR70.833-SVM71.423-Ada69.933-XGB70.924-KNN69.984-LR70.864-SVM71.334-Ada69.924-XGB71.005-KNN69.895-LR70.865-SVM71.375-Ada69.945-XGB70.996-KNN69.886-LR70.906-SVM71.396-Ada70.086-XGB71.137-KNN69.907-LR70.867-SVM71.387-Ada70.047-XGB71.048-KNN69.958-LR70.858-SVM71.288-Ada70.008-XGB71.069-KNN69.869-LR70.779-SVM71.209-Ada69.909-XGB70.9210-KNN69.6110-LR70.5910-SVM71.0010-Ada69.8010-XGB70.88Δmax3.75Δmax1.96Δmax1.12Δmax2.29Δmax2.11Fabric2-KNN92.482-LR93.762-SVM88.652-Ada89.832-XGB89.813-KNN92.913-LR94.303-SVM88.903-Ada89.953-XGB90.214-KNN93.234-LR94.414-SVM88.804-Ada90.094-XGB90.255-KNN93.425-LR94.715-SVM88.695-Ada90.035-XGB90.446-KNN93.426-LR94.656-SVM88.616-Ada89.976-XGB90.587-KNN93.377-LR94.657-SVM88.557-Ada89.897-XGB90.348-KNN93.318-LR94.698-SVM88.398-Ada89.898-XGB90.429-KNN93.219-LR94.579-SVM88.279-Ada89.919-XGB90.3210-KNN93.1510-LR94.4110-SVM88.2110-Ada89.7710-XGB90.09Δmax3.87Δmax3.48Δmax2.60Δmax1.64Δmax2.27Figure 4
Evaluation results of all the DVS combinations in the middle feature fusion. “2 & 45” means that 2 DVSs were chosen to complete the middle feature fusion, and 45 DVS combinations were created for the material recognition. (a) MattrSet. (b) Fabric.
(a)(b)Table 8
Comparisons with other state-of-the-art baseline methods on each dataset. The best value of each dataset is shown in bold. Unit: %.
DatasetModelAcc.ModelAcc.ModelAcc.MattrSetCatBoost68.68Densenet16959.77NTS-Net71.80LightGBM68.72ResNeXt WSL67.87PMG71.79GS-Ada-SGL66.11SECLR68.94SEC(h)70.81GS-XGB-SGL67.67SECXGB69.02SEC(s)71.22SVM-S47.78SECSVM70.30SECGS4-Ada70.19HMF272.20DCL70.97SECGS4-KNN70.86MDCA49.84WS-DAN71.67SECGS6-LR70.90SENet69.34WILDCAT71.04SECGS6-XGB71.13InceptionResNetV252.09WSCNet52.12SECGS3-SVM71.42FabricCatBoost88.47ResNeXt WSL59.66NTS-Net88.50LightGBM88.03Albe + Norm79.60PMG93.30GS-XGB-SG84.20SECAda88.45SEC(h)91.61SVM-S80.63SECKNN89.55SEC(s)93.19HMF284.34SECLR91.23SECGS3-SVM88.90MDCA65.36DCL93.60SECGS4-Ada90.09SENet89.38WS-DAN91.86SECGS6-XG90.58InceptionResNetV246.22WILDCAT81.13SECGS5-KNN93.42Densenet16946.29WSCNet62.50SECGS5-LR94.71In conclusion, the proposed middle feature fusion approach outperformed the voting-based late feature fusion approach. Moreover, our progressive feature fusion-based SECGS model is effective and robust in recognizing both fine-grained and coarse-grained materials.


### 4.7. Comparisons with SOTA Baseline Models

In this section, we compare the SECGS model with a group of SOTA baseline models. Detailed accuracy comparisons are presented in Table8. On the MattrSet dataset, the SECGS3-SVM model achieved a very competitive performance compared with that of the best baseline (HMF2). Certainly, material recognition remains a challenge on this dataset. On the Fabric dataset, the SECGS5-LR model outperformed all the baseline models, obtaining an approximately 10.37% performance improvement compared with that of the most competitive material recognition model recently proposed (HMF2). Hence, unlike the baseline models, the SECGS model achieved a more balanced recognition performance. This is mainly attributed to the diverse knowledge learned by the progressive feature fusion method. However, unlike GS-AdaBoost and GS-XGBoost, our model adopts only the SENet feature. Moreover, unlike HMF2, our model performs early feature fusion only once. All these helps improve its real-time efficiency (Table 9).Table 9
Comparisons of the real-time recognition efficiency on each dataset. The best value of each dataset is shown in bold. Unit: s.
DatasetModelTime (per image)MattrSetMDCA19.85GS-XGBoost16.82HMF218.69SECGS3-SVM13.26FabricMDCA19.99GS-XGBoost11.60HMF216.86SECGS5-LR13.21In addition, on the Fabric dataset, the SECGS model achieved an approximately 1.11% performance improvement compared with that of the most competitive fine-grained classification baseline (DCL). This means that the SECGS model can mine sufficient knowledge to obtain better performance. On the MattrSet dataset, it performed comparatively well to the most competitive fine-grained classification model (PMG). This also demonstrated the effectiveness of the knowledge mined by our method.We also found that our variants, SECLR and SEC(s), performed well (e.g., on the Fabric dataset, SEC(s) outperformed the HMF2 model, with an improvement of 8.85%). The experimental results also illustrate an aspect of the ablation analysis; the corresponding accuracy in descending order was “SECGS > SEC(s) > SEC > general classifier.” This further confirms the effectiveness of our model, which can learn diverse valuable knowledge through progressive feature fusion. Notably, Figures 4 and 5 show another perspective of the ablation analysis.Figure 5
Visualization results of different features on the Fabric dataset. (a) R50, (b) SR50, (c) SRXt101, (d) R50-RXt101 (DVS (re)), (e) SR50-RXt101 (DVS (re-se)), and (f) SR50-SRXt101 (DVS (se)).
(a)(b)(c)(d)(e)(f)The corresponding comparisons of the real-time efficiency are presented in Table9. We performed our experiments using the recognition system we deployed on a laptop (Figure 6). For instance, on the MattrSet dataset, our model required only 13.26 s to test a material image, which is 5.43 s faster than the HMF2 model. In summary, the results given in Table 9 indicate that, in addition to its effectiveness and robustness, the SECGS model is also more efficient than the SOTA baseline models.Figure 6
Example demonstration of the SECGS-based real-time material recognition system.In summary, the proposed SECGS model is effective, efficient, and robust for material recognition on both the Fabric and MattrSet datasets.


## 4.1. Datasets

The Fabric dataset is a fine-grained dataset constructed by Kampouris et al. [51] using photometric stereoscopic sensors to collect clothing samples from physical stores and determine the final material according to the clothing label. It contains images of clothes made from any of these 9 materials: cotton, wool, terrycloth, fleece, nylon, silk, denim, viscose, and polyester. The Fabric dataset contains 1266 samples, according to different light intensities, visual angles, and other conditions. Each sample includes four images; thus, it has a total of 5064 images (Kampouris et al. cropped all the images to 400 × 400 pixels to avoid blurring the edges of the images). Table 2 provides more detailed information about the Fabric dataset.The MattrSet dataset [16] is derived from real commodities on the web and constructed under the guidance of an experienced material expert. It is a coarse-grained dataset that includes images of bags (polyurethane (pu), canvas, nylon, and polyester) and shoes (pu and canvas). It contains 11,021 images (the corresponding image resolution ranges from 123 × 123 to 4300 × 2867, which makes material recognition more challenging). Table 3 provides more detailed information about the MattrSet dataset.


## 4.2. Baselines

We used five general classifiers, namely, logistic regression (LR),k-nearest neighbor (KNN), support vector machine (SVM), AdaBoost (Ada), and XGBoost (XGB), to train the proposed SECGS model (e.g., “SECGS5-LR” means that the LR classifier uses five DVSs to implement middle feature fusion). We compared our model with the baseline models listed below. The goal of this study was to obtain a better recognition performance using progressive feature fusion. We aimed to improve the recognition performance of our model on fine-grained and coarse-grained material images. Hence, our work can be regarded as a type of fine-grained image classification task. Therefore, we also compared the proposed model with some recently proposed fine-grained image classification models:(1)
Some well-known late feature fusion models are as follows: AdaBoost [47], XGBoost [48], CatBoost [49], and LightGBM [50]. For a fairer comparison, we also used the DVSs generated by our method to train these models.(2)
Recently proposed material recognition models are as follows: GS-XGB [16], GS-Ada [16], SVM-S [17], MDCA-XGB [17], HMF2 [17], and Kampouris’s Albe + Norm [53].(3)
Recently proposed deep learning models are as follows: InceptionResNetV2 [54], DenseNet169 [55], SENet [23], and ResNeXt WSL [56].(4)
Some well-known fine-grained classification models are as follows: DCL [57], NTS-Net [58], PMG [59], WILDCAT [60], WSCNet [61], and WS-DAN [62]. All these models are trained by using the four-fold cross-validation with an initial learning rate of 1e −3, a batch size of 16 and trained for 40 epochs.(5)
Multiple variants of the SECGS model are as follows: we removed the “GS” component from “SECGS” to generate a new early feature fusion model called SEC. We then added an ensemble learning strategy (a type of late feature fusion) to the SEC model and created two variants of our model: SEC(s) and SEC(h). SEC(s) uses soft voting-based late feature fusion after the early feature fusion, whereas SEC(h) uses hard voting-based late feature fusion. Hence, the two models also use the progressive feature fusion method. We wanted to validate the robustness of this method and to demonstrate the effectiveness of the “GS” component.


## 4.3. Evaluation Metrics

We employed accuracy (Acc.) (equation (12)) and precision (equation (13)) to evaluate our model. TP indicates true positive; FP, false positive; TN, true negative; and FN, false negative. Moreover, we designed a new metric called the average accuracy for each feature (AAF) (equation (14)) to comprehensively evaluate each feature. Similarly, we propose a new metric called the average accuracy for each classifier (AAC) (equation (15)) to comprehensively evaluate each classification model. Finally, we used the mean average accuracy (MAA) metric (equation (16)) to evaluate the overall performance of the proposed model:(12)accuracy=TPall+TNallTPall+FPall+TNall+FNall,(13)precision=TPallTPall+FPall,(14)AAF=∑m=1NclassifierAccuracymNclassifier,(15)AAC=∑n=1NfeatureAccuracynNfeature,(16)MAA=∑n=1Nclassifier∑m=1NfeatureAccuracymnNclassifier×Nfeature.Moreover, as shown in equation (17), ΔAccuracy1 calculates the absolute performance improvement of the AAF metric of the SENet features relative to that of the ResNet features. This metric can evaluate the effectiveness of the SENet features. As shown in equation (18), ΔAccuracy2 calculates the absolute performance improvement of the AAF metric of the DVSs relative to that of the SENet features. This metric evaluates the effectiveness of the generated DVSs.(17)ΔAccuracy1=AAF−SENet−AAF−ResNet,(18)ΔAccuracy2=AAF−DVS−AAF−SENet.


## 4.4. Evaluation of the SEC Model

Table5 shows the accuracy of the SEC model that uses different DVSs. Table 6 shows the AAF and MAA values of the best ResNet features, SENet features, and DVSs. In this section, we discuss our experimental results in detail.Table 5
Recognition performance of the proposed SEC models on each dataset. The best value of each column is shown in bold. Unit: %.
DatasetDVSRecognition performance of each SEC modelKNNLRSVMAdaXGBAverageMattrSetSR50-SR15266.2368.9469.9767.2668.6868.22SR50-SR10165.4868.7269.7067.4968.4067.96SR50-SRXt10164.9168.8870.3067.4669.0268.11SRXt101-SR15265.1568.7369.6167.7968.8968.03SR101-SR15265.4468.7169.9367.2268.2967.92SRXt50-SR15265.4768.2269.2967.3068.5167.76SR101-SRXt10164.7468.7069.8267.4468.3867.82SR50-SRXt5064.2568.4469.6367.5968.8267.75SRXt50-SRXt10164.6868.5970.0767.6168.2967.85SRXt50-SR10164.4968.6369.4767.5268.2467.67ΔAccuracy116.1211.0743.8118.6014.4020.80ΔAccuracy20.831.071.084.442.822.05FabricSR101-SRXt10189.0891.1386.3087.4687.1688.23SRXt50-SR10189.0090.9886.0287.3288.1188.29SRXt101-SR15289.4491.0585.0786.9586.8587.87SRXt50-SRXt10188.7490.6085.6887.9187.5688.10SRXt50-SR15289.5590.8085.7688.1987.5488.37SR50-SRXt10187.9991.2383.3188.4588.1987.83SR50-SRXt5087.6090.8683.0887.9388.1187.52SR101-SR15288.5590.2884.5686.6586.0487.22SR50-SR10187.2290.4082.4287.6288.3187.19SR50-SR15287.9789.9381.3887.7287.9786.99ΔAccuracy147.7935.5434.5037.9836.6238.49ΔAccuracy22.343.353.611.501.622.48Table 6
AAF and MAA values of each feature on each dataset. In MAA values, the best performance of each column is shown in bold. In AAF values, the best value of each row is shown in bold. Unit: %.
DatasetFeatureAAFMAAKNNLRSVMAdaXGBMattrSetResNet48.1456.5224.8944.4351.3345.06SENet64.2667.5968.7063.0365.7365.86DVS65.4568.7069.8067.4268.6368.00FabricResNet38.3851.8446.2548.1549.3446.79SENet86.1887.3780.7586.1285.9685.28DVS88.9790.9785.3687.7187.5788.12According to ΔAccuracy1, the maximum improvement of the SENet features relative to that of the ResNet features on the Fabric dataset is 47.79%, and the corresponding average improvement is 38.49%. These results confirm that the SE module can depict the key visual characteristics of different materials and that the SENet features from the SE modules can be applied to different classification models. Similar results were also obtained on the MattrSet dataset, which confirms the robustness of the SENet features. Certainly, a larger performance improvement was obtained on the Fabric dataset owing to the fine-grained visual information contained in its samples.According to ΔAccuracy2, the maximum improvement of the DVSs relative to that of the SENet features on the Fabric dataset is 3.35%, and the corresponding average improvement is 2.48%. Similar performance improvements were also obtained on the MattrSet dataset. These results indicate that the proposed DVSs are effective for material recognition on any dataset and that the proposed early feature fusion method is effective and robust. Moreover, the DVS (SR50-SRXt101) achieved the best accuracy on each dataset. This indicates that sufficient but effective cross-modal knowledge can be mined among the heterogeneous SR50 and SRXt101 features. Owing to this characteristic, even traditional classifiers, including LR and SVM, were able to obtain satisfactory recognition performance. It is worth noting that the corresponding performance on the Fabric dataset was better than that on the MattrSet dataset because the former dataset contains much more valuable visual semantics, whereas the latter dataset contains certain noise (Tables 2 and 3).According to AAF and MAA in Table 6, we can obtain the overall recognition performance of different features from a statistical perspective. Therefore, many more meaningful conclusions can be drawn. According to AAF, the proposed DVSs easily outperformed the SENet and ResNet features on each dataset. A larger performance improvement was obtained on the Fabric dataset. Hence, the proposed DVSs are robust and effective for material recognition. The MAA metric further validates this conclusion. For example, on the Fabric dataset, a performance margin of 41.33% was obtained between DVS and ResNet. These results clearly show that there is indeed a large amount of complementary information among the heterogeneous SENet features and that the proposed early feature fusion method can capture such information to improve the recognition performance. In addition, even traditional models, such as LR, KNN, and SVM, were able to achieve better performance after using early feature fusion owing to the powerful discriminative ability of the proposed DVSs.In summary, the DVSs generated by the proposed early feature fusion method are effective and robust for material recognition, providing a solid basis for the subsequent middle feature fusion. Sufficient cross-modal knowledge among the heterogeneous SENet features was mined to improve the recognition performance. Moreover, all the SENet features obtained satisfactory performance, which firmly supports our original selection.


## 4.5. Evaluation of the SEC(s) and SEC(h) Models

In this section, we evaluate two variants of the SECGS model, namely, SEC(h) and SEC(s). We want to determine whether the voting strategy contributes to improving recognition performance. More importantly, we intend to validate the robustness of the proposed progressive feature fusion method. In addition, we want to demonstrate the effectiveness of the “GS” component from another perspective. Figure3 shows the AAC values of the top three DVSs combined with the corresponding voting strategy.Figure 3
Performance comparisons between the SEC(h) and SEC(s) models. (a) Fabric: SEC(h), (b) Fabric: SEC(s), (c) MattrSet: SEC(h), and (d) MattrSet: SEC(s).
(a)(b)(c)(d)As shown in Figure3(a), the SEC(h) model that uses SR101-SRXt101 with the top three classifiers achieved the best accuracy (91.51%) on the Fabric dataset. It outperformed the original DVS without voting by a large margin. Thus, we should carefully tune the number of classifiers for the final voting. Similarly, as shown in Figure 3(b), the SEC(s) model that uses SRXt101-SR152 with the top two classifiers obtained the best accuracy (93.19%) on the Fabric dataset. It also outperformed the original DVS without voting and the best SEC(h) model. Therefore, the soft voting strategy is more suited for the Fabric dataset.As shown in Figure3(c), the SEC(h) model that uses SR50-SRXt101 with all the classifiers obtained the best accuracy on the MattrSet dataset. It easily outperformed the original DVS without voting. Similarly, as shown in Figure 3(d), the SEC(s) model that uses SR50-SRXt101 with all the classifiers achieved the best accuracy. Unlike on the Fabric dataset, we require more classifiers to complete the voting-based late feature fusion on the MattrSet dataset owing to the much larger amount of noise information in its samples. More heterogeneous classifiers can make a more definite prediction. In addition, a larger performance improvement can be obtained on the MattrSet dataset if we use a soft voting strategy. This further indicates that the corresponding recognition of coarse-grained material images remains a significant challenge. In general, the soft voting strategy is also more suited for the MattrSet dataset.In summary, both the SEC(h) and SEC(s) models achieved satisfactory performance on each material dataset. This also validates the robustness of the proposed progressive feature fusion method from another perspective. Hence, the voting-based late feature fusion method is another useful substitute for the proposed middle feature fusion method. Unlike HMF2 [17], the SEC model only adopts the SENet features, which helps reduce its complexity. This is an important step in narrowing the gap between theoretical research and practical applications.


## 4.6. Evaluation of the SECGS Model

In this section, we perform a more comprehensive evaluation of the proposed SECGS model. Table7 shows the corresponding experimental results. We show the performance variations with the increase in the number of DVSs on each dataset. For example, “2-SVM” means that the SVM classifier uses two DVSs to implement the middle feature fusion . “Best Acc.” represents the best accuracy among all the DVS combinations. “Δmax” represents the corresponding performance improvement of the SECGS model compared with the best SEC model described above. Some important conclusions can be drawn from Table 7.(1)
On the MattrSet dataset, the corresponding performance rank of the different classifiers in descending order was “SVM > XGBoost > LR > AdaBoost > KNN.” On the Fabric dataset, it was “LR > KNN > XGBoost > AdaBoost > SVM.” These can help us with the selection of the optimal classifier for the SECGS model. The SECGS model prefers SVM on the coarse-grained MattrSet dataset, whereas it prefers LR on the fine-grained Fabric dataset. A similar result can be seen in Table5. Thus, owing to the progressive feature fusion method, sufficient and effective knowledge was mined, enabling even the traditional classifiers to achieve satisfactory recognition performance.(2)
We found that, regardless of the number of DVSs, the SVM classifier always achieved the best performance on the MattrSet dataset and the LR classifier on the Fabric dataset. These findings are further illustrated in Figure4. They indicate that the most suitable classifier should be chosen carefully for different material datasets to achieve the largest performance improvement.(3)
According to Δmax, the proposed middle feature fusion approach outperformed the voting-based late feature fusion approach. The feature-shared knowledge was fully mined using our progressive feature fusion model. Evidently, the proposed “GS” method is more effective than the voting-based late feature fusion method. Unlike the SEC model, our model adopts only one classifier. In addition, larger performance improvements were obtained on the fine-grained Fabric dataset. A similar conclusion can be drawn from the above subsection “Evaluation of the SEC Model.” Interestingly, owing to its different implementation mechanisms, the proposed SECGS model will be a powerful supplement to the well-known HMF2 model (Table 8).(4)
More DVSs did not mean better recognition performance. For example, the proposed SECGS model that combines the SVM classifier and three DVSs achieved the best performance (71.42%) on the MattrSet dataset. Sufficient feature-shared knowledge among the three DVSs was better mined by our model, which improved the final recognition performance. A similar result can be seen in Figure4. Hence, the proposed middle feature fusion strategy only requires careful tuning of the number of DVSs to capture sufficient feature-shared knowledge for more effective material recognition.Table 7
Recognition performance of the SECGS model on each dataset. The best value of each column is shown in bold. Unit: %.
DatasetSECGSBestAcc.SECGSBestAcc.SECGSBestAcc.SECGSBestAcc.SECGSBestAcc.MattrSet2-KNN68.892-LR70.372-SVM71.252-Ada69.522-XGB70.563-KNN69.563-LR70.833-SVM71.423-Ada69.933-XGB70.924-KNN69.984-LR70.864-SVM71.334-Ada69.924-XGB71.005-KNN69.895-LR70.865-SVM71.375-Ada69.945-XGB70.996-KNN69.886-LR70.906-SVM71.396-Ada70.086-XGB71.137-KNN69.907-LR70.867-SVM71.387-Ada70.047-XGB71.048-KNN69.958-LR70.858-SVM71.288-Ada70.008-XGB71.069-KNN69.869-LR70.779-SVM71.209-Ada69.909-XGB70.9210-KNN69.6110-LR70.5910-SVM71.0010-Ada69.8010-XGB70.88Δmax3.75Δmax1.96Δmax1.12Δmax2.29Δmax2.11Fabric2-KNN92.482-LR93.762-SVM88.652-Ada89.832-XGB89.813-KNN92.913-LR94.303-SVM88.903-Ada89.953-XGB90.214-KNN93.234-LR94.414-SVM88.804-Ada90.094-XGB90.255-KNN93.425-LR94.715-SVM88.695-Ada90.035-XGB90.446-KNN93.426-LR94.656-SVM88.616-Ada89.976-XGB90.587-KNN93.377-LR94.657-SVM88.557-Ada89.897-XGB90.348-KNN93.318-LR94.698-SVM88.398-Ada89.898-XGB90.429-KNN93.219-LR94.579-SVM88.279-Ada89.919-XGB90.3210-KNN93.1510-LR94.4110-SVM88.2110-Ada89.7710-XGB90.09Δmax3.87Δmax3.48Δmax2.60Δmax1.64Δmax2.27Figure 4
Evaluation results of all the DVS combinations in the middle feature fusion. “2 & 45” means that 2 DVSs were chosen to complete the middle feature fusion, and 45 DVS combinations were created for the material recognition. (a) MattrSet. (b) Fabric.
(a)(b)Table 8
Comparisons with other state-of-the-art baseline methods on each dataset. The best value of each dataset is shown in bold. Unit: %.
DatasetModelAcc.ModelAcc.ModelAcc.MattrSetCatBoost68.68Densenet16959.77NTS-Net71.80LightGBM68.72ResNeXt WSL67.87PMG71.79GS-Ada-SGL66.11SECLR68.94SEC(h)70.81GS-XGB-SGL67.67SECXGB69.02SEC(s)71.22SVM-S47.78SECSVM70.30SECGS4-Ada70.19HMF272.20DCL70.97SECGS4-KNN70.86MDCA49.84WS-DAN71.67SECGS6-LR70.90SENet69.34WILDCAT71.04SECGS6-XGB71.13InceptionResNetV252.09WSCNet52.12SECGS3-SVM71.42FabricCatBoost88.47ResNeXt WSL59.66NTS-Net88.50LightGBM88.03Albe + Norm79.60PMG93.30GS-XGB-SG84.20SECAda88.45SEC(h)91.61SVM-S80.63SECKNN89.55SEC(s)93.19HMF284.34SECLR91.23SECGS3-SVM88.90MDCA65.36DCL93.60SECGS4-Ada90.09SENet89.38WS-DAN91.86SECGS6-XG90.58InceptionResNetV246.22WILDCAT81.13SECGS5-KNN93.42Densenet16946.29WSCNet62.50SECGS5-LR94.71In conclusion, the proposed middle feature fusion approach outperformed the voting-based late feature fusion approach. Moreover, our progressive feature fusion-based SECGS model is effective and robust in recognizing both fine-grained and coarse-grained materials.


## 4.7. Comparisons with SOTA Baseline Models

In this section, we compare the SECGS model with a group of SOTA baseline models. Detailed accuracy comparisons are presented in Table8. On the MattrSet dataset, the SECGS3-SVM model achieved a very competitive performance compared with that of the best baseline (HMF2). Certainly, material recognition remains a challenge on this dataset. On the Fabric dataset, the SECGS5-LR model outperformed all the baseline models, obtaining an approximately 10.37% performance improvement compared with that of the most competitive material recognition model recently proposed (HMF2). Hence, unlike the baseline models, the SECGS model achieved a more balanced recognition performance. This is mainly attributed to the diverse knowledge learned by the progressive feature fusion method. However, unlike GS-AdaBoost and GS-XGBoost, our model adopts only the SENet feature. Moreover, unlike HMF2, our model performs early feature fusion only once. All these helps improve its real-time efficiency (Table 9).Table 9
Comparisons of the real-time recognition efficiency on each dataset. The best value of each dataset is shown in bold. Unit: s.
DatasetModelTime (per image)MattrSetMDCA19.85GS-XGBoost16.82HMF218.69SECGS3-SVM13.26FabricMDCA19.99GS-XGBoost11.60HMF216.86SECGS5-LR13.21In addition, on the Fabric dataset, the SECGS model achieved an approximately 1.11% performance improvement compared with that of the most competitive fine-grained classification baseline (DCL). This means that the SECGS model can mine sufficient knowledge to obtain better performance. On the MattrSet dataset, it performed comparatively well to the most competitive fine-grained classification model (PMG). This also demonstrated the effectiveness of the knowledge mined by our method.We also found that our variants, SECLR and SEC(s), performed well (e.g., on the Fabric dataset, SEC(s) outperformed the HMF2 model, with an improvement of 8.85%). The experimental results also illustrate an aspect of the ablation analysis; the corresponding accuracy in descending order was “SECGS > SEC(s) > SEC > general classifier.” This further confirms the effectiveness of our model, which can learn diverse valuable knowledge through progressive feature fusion. Notably, Figures 4 and 5 show another perspective of the ablation analysis.Figure 5
Visualization results of different features on the Fabric dataset. (a) R50, (b) SR50, (c) SRXt101, (d) R50-RXt101 (DVS (re)), (e) SR50-RXt101 (DVS (re-se)), and (f) SR50-SRXt101 (DVS (se)).
(a)(b)(c)(d)(e)(f)The corresponding comparisons of the real-time efficiency are presented in Table9. We performed our experiments using the recognition system we deployed on a laptop (Figure 6). For instance, on the MattrSet dataset, our model required only 13.26 s to test a material image, which is 5.43 s faster than the HMF2 model. In summary, the results given in Table 9 indicate that, in addition to its effectiveness and robustness, the SECGS model is also more efficient than the SOTA baseline models.Figure 6
Example demonstration of the SECGS-based real-time material recognition system.In summary, the proposed SECGS model is effective, efficient, and robust for material recognition on both the Fabric and MattrSet datasets.


## 5. Parameter Tuning

### 5.1. Cross-Validation Analysis

Owing to the scarcity of material samples, cross-validation is required to objectively evaluate the proposed material recognition model. In this section, we evaluate two-fold and four-fold cross-validations to show the robustness of our method. Table10 shows the corresponding performance comparisons. Here, we only compared three features, namely, ResNet, SENet, and DVS, which is sufficient to validate our choice.Table 10
Comparisons of the two-fold and four-fold cross-validations. The best value of each column is shown in bold. Unit: %.
DatasetMetricCross-validationResNetSENetDVSMattrSetMAATwo-fold39.7861.7664.19Four-fold42.2162.8466.06FabricTwo-fold43.2878.2381.30Four-fold44.0581.6084.64As shown in Table10, the SENet feature beats outperformed the ResNet feature, whereas DVS outperformed the SENet and ResNet features by a large margin. It is reasonable to mine the implicit cross-modal knowledge among the heterogeneous SENet features. In addition, the performance of the four-fold cross-validation was better than that of the two-fold cross-validation, which further indicates that the proposed DVSs are robust for material recognition and that they help enhance the practicability of the SECGS model. Certainly, material recognition remains a challenge on the MattrSet dataset. In conclusion, we used four-fold cross-validation to complete all experiments.


### 5.2. Comparisons of the Summation and Concatenation Modes in Early Feature Fusion

The proposed early feature fusion method uses two fusion modes, namely, summation (SUM) and concatenation (CON), to create the DVS. Ten DVSs were chosen to evaluate the effectiveness of these modes, and the corresponding results are shown in Figure7. In the evaluation, we used five classifiers and four-fold cross-validation to perform material recognition, and the values of the AAC metric were used to draw the line graphs.Figure 7
Comparison between the two modes of the early feature fusion method on each dataset. (a) MattrSet. (b) Fabric.
(a)(b)As shown in Figure7, the CON mode was superior to the SUM mode on each benchmark dataset. The SUM mode reduced the dimension of the fused features (DVSs), thus diluting the effective cross-modal knowledge. In contrast, the dimension generated by the CON mode was more appropriate, as it retained the maximum clustering canonical correlations among the heterogeneous SENet features. Therefore, our model uses the CON mode to complete the early feature fusion.


### 5.3. Evaluation of the DVSs

In this section, we further make a comprehensive validation of the effectiveness of the proposed DVSs. Table4 shows the performance of the corresponding DVS generated using different features. DVS (re) represents the DVS generated by the ResNet features, including ResNet 50, ResNet 101, ResNet 152, ResNeXt 50, and ResNeXt 101. DVS (re-se) represents the DVS generated by the ResNet and SENet features. DVS (se) represents the DVSs generated by the SENet features. ΔImprove1 represents the performance improvement of the DVS (re) compared with that of the corresponding ResNet feature. ΔImprove2 represents the performance improvement of the DVS (re-se) compared with that of the corresponding SENet feature. ΔImprove3 represents the performance improvement of the DVS (se) compared with that of the corresponding SENet feature. Several valuable conclusions can be drawn as follows:(1)
According to MAA, there was an obvious upward trend with the adoption of the SENet features. This indicates that the SENet features, rather than the ResNet features, can provide more valuable knowledge for material recognition. This finding validates our choice of the SENet features as the basis for our progressive feature fusion method.(2)
According to ΔImprove1, the corresponding DVSs achieved satisfactory performance even if we only used the ResNet features, which shows the robustness of the proposed early feature fusion method. Moreover, a larger MAA improvement was obtained on the MattrSet dataset, which confirms that a significant amount of cross-modal knowledge among the ResNet features was mined out from the MattrSet dataset.(3)
According to ΔImprove2, performance improvements were not always positive when only one SENet feature was used to complete the early feature fusion. There must be some noise among the ResNet and SENet features, which significantly affected the quality of the acquired cross-modal knowledge (DVS) and led to a performance decline. This suggests that we should use the CNN-based features from a homogeneous network to complete the proposed early feature fusion, which helps obtain high-quality cross-modal knowledge for improving the performance of the final recognition.(4)
According to ΔImprove3, the performance improvements were always positive when only the SENet features were used for the early feature fusion. High-quality cross-modal knowledge was fully mined to generate the proposed DVSs. This also validates our original selection and serves as a basis for the subsequent middle feature fusion.In conclusion, the proposed DVSs mined from the heterogeneous SENet features are effective and robust for high-quality material recognition.


### 5.4. Visualization of the Proposed DVSs

To further illustrate the effectiveness of the proposed DVSs, we employed the well-known t-SNE [63] algorithm to visualize some relevant features. Taking the Fabric dataset as an example, we chose several features, including ResNet 50 (R50) (similar to ResNeXt 101 (RXt101)), SR50, SRXt101, R50-RXt101 (DVS (re)), SR50-RXt101 (DVS (re-se)), and SR50-SRXt101 (DVS (se)), for the visualization. The corresponding experimental results are shown in Figure 5. Many valuable conclusions can be drawn as follows:(1)
The original R50 feature represented very messy material categories. Hence, the traditional deep learning features are insufficient for describing different material categories (see Tables6 and 10 for the quantitative results).(2)
The original SR50 represented the material categories better than the R50 feature. However, confusion still occurred, as shown in Figure5(b). A similar result can be seen in Figure 5(c). Therefore, the original deep learning-based features are insufficient for depicting material images.(3)
DVS (se) can represent each material category more accurately than the single features (R50, SR50, and SRXt101). The same color blocks (samples in the same category) started to aggregate, whereas different color blocks (samples in different categories) began to separate from each other. This means that the semantic distance between the same samples shrank, whereas that between the different samples expanded gradually, which helped the classification model to train a more appropriate decision function (Table5). When the proposed DVSs were used, even the traditional classification models, including SVM, KNN, and LR, were able to obtain satisfactory recognition performance.(4)
Comparing the three types of DVS (DVS (re), DVS (re-se), and DVS (se)), we obtained the performance rank in the descending order as follows: “DVS (se) > DVS (re-se) > DVS (re).” According to Figures5(d)–5(f), DVS (se) is better than DVS (re) in representing different material categories. The SENet features, rather than the ResNet features, provide more effective knowledge for early feature fusion. This is an intuitive explanation of the results presented in Table 4. This also validates our original selection. From Figures 5(a)–5(d), we can see that DVS (re) had a better ability to represent different material categories than that of the original R50, which shows the effectiveness of the proposed early feature fusion method from another perspective. According to the comparisons between Figures 5(e)–5(f), the material categories depicted by DVS (re-se) are messier than those depicted by DVS (se). This indicates that there is a certain amount of noise between the ResNet and SENet features, which might have affected the final quality of the acquired cross-modal knowledge. Moreover, this also reduced the ability of DVS (re-se) to characterize different material categories. All these findings also confirm the conclusions we mentioned in the above subsection “Evaluation of the DVSs.”


### 5.5. Evaluation of All the DVS Combinations

Figure4 shows the corresponding recognition performance of all the DVS combinations in the proposed middle feature fusion procedure. As can be seen from the median line, the accuracy of each classifier tends to improve with the increase in the number of DVSs. This means that, as the number of DVSs increases, the SECGS model can acquire sufficient feature-shared knowledge to become more stable (similar results can also be seen in Table 7).For the MattrSet dataset, an accuracy peak was observed when the SVM classifier used three DVSs. For the Fabrics dataset, an accuracy peak was observed when the LR classifier used five DVSs. These results are consistent with the best values given in Table8. Moreover, the SVM classifier achieved the best overall performance on the MattrSet dataset, whereas the LR classifier achieved it on the Fabric dataset. Hence, even the traditional classifiers were able to obtain satisfactory performance using the proposed progressive feature fusion method owing to the diverse knowledge obtained by our model. This finding is more evident on the MattrSet dataset.In addition, the performance variations of our approach showed a positive effect on the number of DVSs, which indicates that more DVSs can obtain more stable performance (the lower edge (minimum) of each box increases with the number of DVSs, and the smaller space between the upper and lower edges indicates that the corresponding performance fluctuates less). These results show that feature-shared knowledge can be fully obtained using the proposed approach, thus improving the final recognition performance. In summary, like the cross-modal knowledge contained in the DVSs, the feature-shared knowledge among different DVSs can also be mined by the progressive feature fusion approach, and this knowledge is effective for high-quality material recognition.


### 5.6. Evaluation of the Contribution of Each Component of the SECGS Model

To evaluate the real contribution of each component of the SECGS model, we plotted the corresponding precision of SENet, DVS, SEC(s), and SECGS for each material category. The corresponding results are shown in Figure8. The evaluation was done as per the ablation analysis procedure, which helped us to objectively evaluate each component of the progressive feature fusion method.Figure 8
Precision of each component of SECGS for each material category. (a) MattrSet. (b) Fabric.
(a)(b)First, gradual performance improvements were observed on each dataset using different components. This observation was more evident on the Fabric dataset (e.g., fleece, silk, and wool). Second, the proposed DVSs easily outperformed the original SENet feature, which indicates that they, unlike the single image feature (SENet), obtained sufficient cross-modal knowledge from the heterogeneous SENet features to better depict material images (Tables6 and 10 and Figure 5). Third, the SECGS model outperformed the proposed DVSs, which means that implicit feature-shared knowledge was fully obtained through middle feature fusion to boost the final recognition performance.Moreover, our model performed better on the Fabric dataset (similar results can be seen in Tables7 and 8). Owing to its different implementation mechanisms, our model will be a good complement to the SOTA HMF2 model, which did not do well on the MattrSet dataset. However, the recognition of materials, such as bag_pu and bag_polyester, remains challenging. We also found that better but more stable performance was achieved by using the proposed middle feature fusion method than that by using ensemble learning (Table 7). The proposed progressive feature fusion method (early feature fusion + middle feature fusion) is more effective and robust than the SEC(s) model (early feature fusion + late feature fusion). Although ensemble learning performed well on some materials, the proposed middle feature fusion method achieved a more balanced performance improvement on most material categories. All these confirm the effectiveness and robustness of the proposed progressive feature fusion method from another perspective.To further show the real contribution of each component of our approach, we performed another ablation analysis experiment and plotted the results in Figure9. This figure shows the corresponding performance improvement after the ablation operation. “SE-SF” represents the largest performance improvement of the SENet feature relative to that of the best single feature. “DVS-SE” represents the largest performance improvement of the proposed DVSs relative to that of the SENet feature. “SECGS-DVS” represents the largest performance improvement of the proposed SECGS model relative to that of the generated DVS. “SEC(s)-DVS” represents the largest performance improvement of the proposed SEC(s) model relative to that of the generated DVS.Figure 9
Performance of each component of the SECGS model on the two datasets.As shown in Figure9, each component of the SECGS model tried its best to boost the final recognition performance. However, the contribution rank of the components in the descending order was “SENet > GS > DVS.” Hence, the SOTA SENet feature is the most important component of the proposed SECGS model, whereas the effect of the proposed DVSs is relatively small. This conclusion is more evident on the Fabric dataset. This also motivates us to design a more effective early feature fusion method to create novel DVSs in the future. We also found that our variant models (SEC(s)) obtained satisfactory recognition performance, which further confirms the robustness of the proposed progressive feature fusion method. Clearly, the proposed middle feature fusion method outperformed the voting-based late feature fusion method, which confirms the effectiveness of the implicit feature-shared knowledge mined by our GS-based middle feature fusion method.


## 5.1. Cross-Validation Analysis

Owing to the scarcity of material samples, cross-validation is required to objectively evaluate the proposed material recognition model. In this section, we evaluate two-fold and four-fold cross-validations to show the robustness of our method. Table10 shows the corresponding performance comparisons. Here, we only compared three features, namely, ResNet, SENet, and DVS, which is sufficient to validate our choice.Table 10
Comparisons of the two-fold and four-fold cross-validations. The best value of each column is shown in bold. Unit: %.
DatasetMetricCross-validationResNetSENetDVSMattrSetMAATwo-fold39.7861.7664.19Four-fold42.2162.8466.06FabricTwo-fold43.2878.2381.30Four-fold44.0581.6084.64As shown in Table10, the SENet feature beats outperformed the ResNet feature, whereas DVS outperformed the SENet and ResNet features by a large margin. It is reasonable to mine the implicit cross-modal knowledge among the heterogeneous SENet features. In addition, the performance of the four-fold cross-validation was better than that of the two-fold cross-validation, which further indicates that the proposed DVSs are robust for material recognition and that they help enhance the practicability of the SECGS model. Certainly, material recognition remains a challenge on the MattrSet dataset. In conclusion, we used four-fold cross-validation to complete all experiments.


## 5.2. Comparisons of the Summation and Concatenation Modes in Early Feature Fusion

The proposed early feature fusion method uses two fusion modes, namely, summation (SUM) and concatenation (CON), to create the DVS. Ten DVSs were chosen to evaluate the effectiveness of these modes, and the corresponding results are shown in Figure7. In the evaluation, we used five classifiers and four-fold cross-validation to perform material recognition, and the values of the AAC metric were used to draw the line graphs.Figure 7
Comparison between the two modes of the early feature fusion method on each dataset. (a) MattrSet. (b) Fabric.
(a)(b)As shown in Figure7, the CON mode was superior to the SUM mode on each benchmark dataset. The SUM mode reduced the dimension of the fused features (DVSs), thus diluting the effective cross-modal knowledge. In contrast, the dimension generated by the CON mode was more appropriate, as it retained the maximum clustering canonical correlations among the heterogeneous SENet features. Therefore, our model uses the CON mode to complete the early feature fusion.


## 5.3. Evaluation of the DVSs

In this section, we further make a comprehensive validation of the effectiveness of the proposed DVSs. Table4 shows the performance of the corresponding DVS generated using different features. DVS (re) represents the DVS generated by the ResNet features, including ResNet 50, ResNet 101, ResNet 152, ResNeXt 50, and ResNeXt 101. DVS (re-se) represents the DVS generated by the ResNet and SENet features. DVS (se) represents the DVSs generated by the SENet features. ΔImprove1 represents the performance improvement of the DVS (re) compared with that of the corresponding ResNet feature. ΔImprove2 represents the performance improvement of the DVS (re-se) compared with that of the corresponding SENet feature. ΔImprove3 represents the performance improvement of the DVS (se) compared with that of the corresponding SENet feature. Several valuable conclusions can be drawn as follows:(1)
According to MAA, there was an obvious upward trend with the adoption of the SENet features. This indicates that the SENet features, rather than the ResNet features, can provide more valuable knowledge for material recognition. This finding validates our choice of the SENet features as the basis for our progressive feature fusion method.(2)
According to ΔImprove1, the corresponding DVSs achieved satisfactory performance even if we only used the ResNet features, which shows the robustness of the proposed early feature fusion method. Moreover, a larger MAA improvement was obtained on the MattrSet dataset, which confirms that a significant amount of cross-modal knowledge among the ResNet features was mined out from the MattrSet dataset.(3)
According to ΔImprove2, performance improvements were not always positive when only one SENet feature was used to complete the early feature fusion. There must be some noise among the ResNet and SENet features, which significantly affected the quality of the acquired cross-modal knowledge (DVS) and led to a performance decline. This suggests that we should use the CNN-based features from a homogeneous network to complete the proposed early feature fusion, which helps obtain high-quality cross-modal knowledge for improving the performance of the final recognition.(4)
According to ΔImprove3, the performance improvements were always positive when only the SENet features were used for the early feature fusion. High-quality cross-modal knowledge was fully mined to generate the proposed DVSs. This also validates our original selection and serves as a basis for the subsequent middle feature fusion.In conclusion, the proposed DVSs mined from the heterogeneous SENet features are effective and robust for high-quality material recognition.


## 5.4. Visualization of the Proposed DVSs

To further illustrate the effectiveness of the proposed DVSs, we employed the well-known t-SNE [63] algorithm to visualize some relevant features. Taking the Fabric dataset as an example, we chose several features, including ResNet 50 (R50) (similar to ResNeXt 101 (RXt101)), SR50, SRXt101, R50-RXt101 (DVS (re)), SR50-RXt101 (DVS (re-se)), and SR50-SRXt101 (DVS (se)), for the visualization. The corresponding experimental results are shown in Figure 5. Many valuable conclusions can be drawn as follows:(1)
The original R50 feature represented very messy material categories. Hence, the traditional deep learning features are insufficient for describing different material categories (see Tables6 and 10 for the quantitative results).(2)
The original SR50 represented the material categories better than the R50 feature. However, confusion still occurred, as shown in Figure5(b). A similar result can be seen in Figure 5(c). Therefore, the original deep learning-based features are insufficient for depicting material images.(3)
DVS (se) can represent each material category more accurately than the single features (R50, SR50, and SRXt101). The same color blocks (samples in the same category) started to aggregate, whereas different color blocks (samples in different categories) began to separate from each other. This means that the semantic distance between the same samples shrank, whereas that between the different samples expanded gradually, which helped the classification model to train a more appropriate decision function (Table5). When the proposed DVSs were used, even the traditional classification models, including SVM, KNN, and LR, were able to obtain satisfactory recognition performance.(4)
Comparing the three types of DVS (DVS (re), DVS (re-se), and DVS (se)), we obtained the performance rank in the descending order as follows: “DVS (se) > DVS (re-se) > DVS (re).” According to Figures5(d)–5(f), DVS (se) is better than DVS (re) in representing different material categories. The SENet features, rather than the ResNet features, provide more effective knowledge for early feature fusion. This is an intuitive explanation of the results presented in Table 4. This also validates our original selection. From Figures 5(a)–5(d), we can see that DVS (re) had a better ability to represent different material categories than that of the original R50, which shows the effectiveness of the proposed early feature fusion method from another perspective. According to the comparisons between Figures 5(e)–5(f), the material categories depicted by DVS (re-se) are messier than those depicted by DVS (se). This indicates that there is a certain amount of noise between the ResNet and SENet features, which might have affected the final quality of the acquired cross-modal knowledge. Moreover, this also reduced the ability of DVS (re-se) to characterize different material categories. All these findings also confirm the conclusions we mentioned in the above subsection “Evaluation of the DVSs.”


## 5.5. Evaluation of All the DVS Combinations

Figure4 shows the corresponding recognition performance of all the DVS combinations in the proposed middle feature fusion procedure. As can be seen from the median line, the accuracy of each classifier tends to improve with the increase in the number of DVSs. This means that, as the number of DVSs increases, the SECGS model can acquire sufficient feature-shared knowledge to become more stable (similar results can also be seen in Table 7).For the MattrSet dataset, an accuracy peak was observed when the SVM classifier used three DVSs. For the Fabrics dataset, an accuracy peak was observed when the LR classifier used five DVSs. These results are consistent with the best values given in Table8. Moreover, the SVM classifier achieved the best overall performance on the MattrSet dataset, whereas the LR classifier achieved it on the Fabric dataset. Hence, even the traditional classifiers were able to obtain satisfactory performance using the proposed progressive feature fusion method owing to the diverse knowledge obtained by our model. This finding is more evident on the MattrSet dataset.In addition, the performance variations of our approach showed a positive effect on the number of DVSs, which indicates that more DVSs can obtain more stable performance (the lower edge (minimum) of each box increases with the number of DVSs, and the smaller space between the upper and lower edges indicates that the corresponding performance fluctuates less). These results show that feature-shared knowledge can be fully obtained using the proposed approach, thus improving the final recognition performance. In summary, like the cross-modal knowledge contained in the DVSs, the feature-shared knowledge among different DVSs can also be mined by the progressive feature fusion approach, and this knowledge is effective for high-quality material recognition.


## 5.6. Evaluation of the Contribution of Each Component of the SECGS Model

To evaluate the real contribution of each component of the SECGS model, we plotted the corresponding precision of SENet, DVS, SEC(s), and SECGS for each material category. The corresponding results are shown in Figure8. The evaluation was done as per the ablation analysis procedure, which helped us to objectively evaluate each component of the progressive feature fusion method.Figure 8
Precision of each component of SECGS for each material category. (a) MattrSet. (b) Fabric.
(a)(b)First, gradual performance improvements were observed on each dataset using different components. This observation was more evident on the Fabric dataset (e.g., fleece, silk, and wool). Second, the proposed DVSs easily outperformed the original SENet feature, which indicates that they, unlike the single image feature (SENet), obtained sufficient cross-modal knowledge from the heterogeneous SENet features to better depict material images (Tables6 and 10 and Figure 5). Third, the SECGS model outperformed the proposed DVSs, which means that implicit feature-shared knowledge was fully obtained through middle feature fusion to boost the final recognition performance.Moreover, our model performed better on the Fabric dataset (similar results can be seen in Tables7 and 8). Owing to its different implementation mechanisms, our model will be a good complement to the SOTA HMF2 model, which did not do well on the MattrSet dataset. However, the recognition of materials, such as bag_pu and bag_polyester, remains challenging. We also found that better but more stable performance was achieved by using the proposed middle feature fusion method than that by using ensemble learning (Table 7). The proposed progressive feature fusion method (early feature fusion + middle feature fusion) is more effective and robust than the SEC(s) model (early feature fusion + late feature fusion). Although ensemble learning performed well on some materials, the proposed middle feature fusion method achieved a more balanced performance improvement on most material categories. All these confirm the effectiveness and robustness of the proposed progressive feature fusion method from another perspective.To further show the real contribution of each component of our approach, we performed another ablation analysis experiment and plotted the results in Figure9. This figure shows the corresponding performance improvement after the ablation operation. “SE-SF” represents the largest performance improvement of the SENet feature relative to that of the best single feature. “DVS-SE” represents the largest performance improvement of the proposed DVSs relative to that of the SENet feature. “SECGS-DVS” represents the largest performance improvement of the proposed SECGS model relative to that of the generated DVS. “SEC(s)-DVS” represents the largest performance improvement of the proposed SEC(s) model relative to that of the generated DVS.Figure 9
Performance of each component of the SECGS model on the two datasets.As shown in Figure9, each component of the SECGS model tried its best to boost the final recognition performance. However, the contribution rank of the components in the descending order was “SENet > GS > DVS.” Hence, the SOTA SENet feature is the most important component of the proposed SECGS model, whereas the effect of the proposed DVSs is relatively small. This conclusion is more evident on the Fabric dataset. This also motivates us to design a more effective early feature fusion method to create novel DVSs in the future. We also found that our variant models (SEC(s)) obtained satisfactory recognition performance, which further confirms the robustness of the proposed progressive feature fusion method. Clearly, the proposed middle feature fusion method outperformed the voting-based late feature fusion method, which confirms the effectiveness of the implicit feature-shared knowledge mined by our GS-based middle feature fusion method.


## 6. Real-Time Material Recognition System

To verify the practicability of the proposed SECGS model, we designed a real-time material recognition system based on the SECGS model with two recognition modes that can be used through local uploading and instant shooting. The workflow of this system is shown on the left-hand side of Figure6. First, users can upload local images or instantly shoot a photo using a mobile IP camera. Second, the system performs feature extraction and progressive feature fusion to complete the material recognition (see the right-hand side of Figure 6). The entire recognition operation is performed on the front page of the cloud application without any manual intervention. This can improve users’ interaction experience and serve as an important basis for subsequent configurations on a cloud server.As Table9 shows, our SECGS model-based real-time material recognition system can achieve satisfactory efficiency compared with the SOTA baseline models. Hence, the SECGS model is efficient for real-time material recognition. If an advanced computer or server with a higher configuration is adopted, the corresponding real-time efficiency will be further improved. Unlike the HMF2 and GS-XGBoost models, the SECGS model adopts only the SENet features to complete the progressive feature fusion. This can improve the final real-time efficiency from another perspective. Hence, the real-time material recognition system narrows the gap between theoretical research and practical applications (scene recognition, industrial detection, robot vision, instant shopping, etc.). For example, in the field of industrial detection, the system can assist textile enterprises in completing fabric defect classification more efficiently. Workers only need a mobile phone to perform this task more easily, thus improving work efficiency and reducing labor costs. For customers who are shopping, they only need a mobile phone to accurately identify the materials of the clothing goods they want to purchase, to assist them in their final purchasing decisions. The system can also be combined with an intelligent robot such as NAO [64] to complete the real-time clothing recognition and grasp estimation, which can better assist the elderly. In summary, our SECGS model-based real-time material recognition system can be seamlessly used with many daily applications to meet practical demands.


## 7. Conclusions and Future Work

We presented a model called SECGS using progressive feature fusion for material recognition. We demonstrated its effectiveness, efficiency, and robustness on two benchmark datasets. By considering progressive feature fusion among the SOTA heterogeneous SENet features, the proposed SECGS model can gather sufficient material knowledge, including cross-modal and feature-shared knowledge, to improve the final recognition performance. Moreover, it can be deployed on a laptop for online real-time material recognition, thereby demonstrating its high practicality. In the future, we plan to advance our research in the following aspects:(1)
We intend to use some SOTA data augmentation methods, such as AutoAugment [65] and Adversarial AutoAugment [66], to solve the problem of unbalanced material images and establish a firm data foundation for high-quality material recognition.(2)
We plan to apply the SECGS model to other research fields, including tumor recognition [67], COVID-19 detection [68], and scene recognition [69]. This can help validate the versatility of our model.(3)
We will try to develop a material recognition mobile application, which can really help narrow the gap between theoretical research and practical applications.



---

*Source: 8971349-2021-10-23.xml*