# Research on Painting Image Classification Based on Transfer Learning and Feature Fusion

Authors: Qian Yong
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

In order to effectively solve the problems of high error rate, long time consuming, and low accuracy of feature extraction in current painting image classification methods, a painting image classification method based on transfer learning and feature fusion was proposed. The global characteristics of the painting picture, such as color, texture, and form, are extracted. The SIFT method is used to extract the painting’s local features, and the global and local characteristics are normalized and merged. The painting images are preliminarily classified using the result of feature fusion, the deterministic and nondeterministic samples are divided, and the estimated Gaussian model parameters are transferred to the target domain via a transfer learning algorithm to alter the distribution of nondeterministic samples, completing the painting image classification. Experimental results show that the proposed method has a low error rate and low feature extraction time and a high accuracy rate of painting image classification.


---

## Body

## 1. Introduction

With the continuous improvement of digital technology, more and more paintings are stored in the form of video, image, and multimedia, and anyone can use the Internet to retrieve and view paintings, which provides great convenience for people [1, 2]. The computer can properly assess the painting's distinctive characteristics, analyse, and quantify them, accurately determine its true worth and author, and even undertake an in-depth interpretation of its creative style [3]. With the fast expansion of social economics, relevant specialists and researchers have placed a greater emphasis on the categorization and identification of painting pictures in order to assess their aesthetic quality and worth, among other things. As a result, it is critical to investigate a categorization system for painting pictures [4].For the above problems, there have been many excellent works. For example, reference [5] proposed a Chinese painting image classification method based on convolutional neural network. The design idea of this method is mainly to solve the problem that the traditional classification method has complicated feature extraction steps of painting images, which leads to the increase in feature extraction time. Thus, the benefits of SoftSign and ReLU activation functions are combined to create a new activation function, which is then used to the convolutional neural network training process. Convolutional neural networks are used to categorise photographs of Chinese paintings. However, it is discovered that this method’s classification accuracy is low in practise, and its impact is poor. Reference [6] presented a technique for classifying styles of painting based on information entropy. Using web crawler technology to collect image painting images from the database, the collected images are preprocessed to obtain higher-quality painting images, calculate the image information entropy, and classify the style of painting using a combination of the information entropy and support vector machine (SVM), but this method is too complex, and painting image feature extraction takes too long.However, the above methods have the problems of high error rate of feature extraction, long extraction time, and low accuracy of painting image classification. Therefore, this paper proposes a new painting image classification method based on transfer learning and feature fusion.


## 2. Design of Painting Image Classification Method

Web crawler refers to the crawling of security rules and related data to the Web or database. Therefore, in order to improve the acquisition speed of painting images, this paper sets crawler rules and uses web crawler to capture painting images in the image database to improve the subsequent image feature extraction and classification rendering speed.


### 2.1. Image Feature Extraction and Fusion

#### 2.1.1. Global Feature Extraction

Color features: HSV color space corresponds to three visual features of the human eyes, respectively. The hue range is 0–360°, which corresponds to the actual color [7, 8]. Saturation S is proportional to the degree of brightness of a color, and brightness is proportional to the ability of light. The more the energy is, the brighter the light is [9]. HSV is more closely related to human perception and experience of color than the RGB system and is thus more extensively utilised. Therefore, this paper adopts HSV color space to extract color features of painting images. This model is shown in Figure 1.(1)Ei=∑i=1N1Npij,σi=1N∑j=1Npij−Ei2,si=1N∑j=1Npij−Ei33,where i represents the total amount of color channel of the painted image, and pij represents the color value of the i-th color channel of the j-th pixel [10].Figure 1
HSV color space model.Texture features: Wavelet analysis is the most widely used method in extracting image texture features [11]. This method is very stable in signal processing and has shown good results in many research fields, with broad application prospects. The principle of texture feature extraction using wavelet transform is shown in Figure 2.Figure 2
Flow chart of wavelet transform.The classical wavelet functions include Haar, Morlet, dbn, and sym N. The following is a brief introduction to Haar wavelet functions. The definition is shown in the following formula:(2)ψt=1,0≤t≤12,−1,12≤t≤1,0,otherwise.Let kernel functionψx be the female wavelet, and a cluster of orthogonal basis can be obtained through translation and scaling [12].(3)ψm,nx=2−m/2ψ2−mx−n.Then, the wavelet coefficients of the signalfx can be expressed by the following formula:(4)cm,n=∫−∞∞fxψm,nsdx.If the scale functionϕx satisfies the two-scale difference equation [13], it can be expressed as follows:(5)ϕx=2∑khkϕ2x−k.Then, the relation between the wavelet kernel functionψx and the scale function is(6)ψx=2∑kgkϕ2x−k,gk=−1kh1−k.Explicitϕx, ψx are not required for wavelet transformation but depend on hk, gk. A J-order wavelet decomposition can be written as follows:(7)f0x=∑kc0,kx=∑kcJ+1,kϕJ+1,kx+∑j=0Jdn+1,kφn+1,kx.If the coefficientc0,k is known, the relationship between the coefficient of grade j+1 and that of grade j is [14](8)cj+1,n=∑kcj,khk−2n,dj+1,n=∑kcj,kgk−2n.Combined with the above analysis, the wavelet coefficientdj,n is used to construct the recursive algorithm, which is described as follows:(9)cj,k=∑ncj+1,nhk−2n+∑ndj+1,ngk−2n.Leth¯n and g¯n be signals with impulse response and use their high and low filters to sample filter signals and combine them to achieve wavelet transform [15]. h¯n and g¯n are defined as follows:(10)h¯n=h−n,g¯n=g−n.In two-dimensional signal, the small and medium wave function can be expressed as(11)ϕx,y=ϕxϕy,ψ1x,y=ϕxψy,ψ2x,y=ψxϕy,ψ3x,y=ψxψy.The corresponding two-dimensional filtering function is(12)hLLk,l=hkhl,hLHk,l=hkgl,hHLk,l=gkhl,hHHk,l=gkgl.The image is set asfx,y, and the wavelet decomposition process of the image is shown in Figure 3 [16, 17].Figure 3
Schematic diagram of wavelet decomposition.Texture features are the result of frequency variations in all directions. Firstly, the traditional Chinese painting is decomposed into four components: LL, LH, HL, and HH. LL1 was decomposed and sampled to obtain second-order wavelet decomposition, as shown in Figure4 [18]. And, if further decomposition is required to obtain more vector information, the same operation is continued for LL2.Figure 4
First-order and second-order decomposition coefficients.Shape features: Compared with color and texture features, shape features belong to deeper image features. Therefore, in order to improve the extraction accuracy of painting image shape features, this paper uses Fourier shape descriptors [19, 20] to achieve this important goal.Suppose inXY there is a boundary in the plane, the boundary mainly consists of N coordinates for xn,yn points, the starting point at this point, moving counterclockwise along the border, will form the multiple trajectories, mainly expressed in x0,y0,x1,y1,…,xn−1,yn−1, at this time will be XY plane and complex plane UV overlap, and redefine the boundary points, as shown in Figure 5.(13)sn=xn+jyn,n=1,2,…,N−1.Figure 5
Two representations of boundary points.The discrete Fourier transform ofsn [21, 22] is(14)Sw=1N∑n=0N−1snexp−j2πwnN,w=0,1,…,N−1.The inverse Fourier transform ofSw is(15)Sk=∑n=0N−1Swexp−j2πwnN,k=0,1,…,N−1.If only the firstP coefficients of Sw are selected here, namely P≤N−1, another similar expression of Sk can be derived, namely the Fourier shape descriptor.(16)S^k=∑w=0P−1Swexp−j2πwkN,k=0,1,…,N−1.The shape features vector of the painting image is described by the Fourier shape descriptor, and the shape feature of the painting image is extracted.


#### 2.1.2. Local Feature Extraction Based on SIFT Algorithm

Scale-invariant feature transform (SIFT) is a classical shape feature and belongs to a local feature algorithm. Local features extracted have invariance and stability [23]. The specific process is as follows.Extreme value detection of scale space: The scale-space kernel is defined as follows:(17)fout=K∗fin,where ∗ represents convolution operation, and K represents scale-space kernel. Gaussian difference scale space can be obtained by using this parameter, as follows:(18)Dx,y,σ=Gx,y,kσ−Gx,y,σ∗Ix,y.=Lx,y,kσ−Lx,y,σ.Among them,(19)Gx,y,σ=12πσ2e−x2+y2/2σ2,where σ is the standard deviation of the Gaussian kernel, and x,y is the spatial coordinate point of an image Ix,y.Positioning of key points: Key points are selected according to the scale and stability of candidate points. If the value of operation comparison is greater than or less than all adjacent points, the candidate point is identified as the key point [24]Key point direction assignment: After the key point is found, one or more directions are assigned to its local image gradient. Taylor's expansion (fitting function) corresponding to the Gaussian function is(20)Dx=D+∂DT∂xx+12xT∂2D∂x2x.For alternative pointsD and D′, D′=0, x, and the corresponding Dx are calculated, and the threshold is set as follows:(21)x=−∂2D∂x2−1∂D∂x,Dx=D+∞12∂DT∂xx.The Hessian matrix2×2 at the “feature point” with constant scale [25]is obtained, and the trace and determinant of the 2×2-dimensional Hessian matrix DxxDxyDxyDyy are solved.(22)TrH=Dxx+Dyy=α+β,DetH=Dxx+Dyy−Dxy2=αβ.Discard the edge response point, setγ as the ratio of two characteristic values, have α=γβ, then the following relationship exists:(23)TrH2DetH=α+β2αβ=rα+β2rβ2=R+12R.Feature points are judged by the threshold, more representative feature points are selected, and unstable key points are screened.(24)TrH2DetH<R+12R.Generate descriptors: From the local picture gradient information, a descriptor is constructed for each key point, and its scale is determined in the second step. The critical element is that the algorithm creates a huge number of highly distinctive characteristics in a variety of sizes and places [26]. The gradient’s modulus and direction are as follows:(25)mx,y=Lx+1,y−Lx−1,y2+Lx,y+1−Lx,y−12,θx,y=arctanLx,y+1−Lx,y−1Lx+1,y−Lx−1,y.The local feature extraction of the painting image is realized by generating descriptors to describe the local feature of the painting image.


#### 2.1.3. Feature Fusion

In the process of painting image classification, not all feature descriptors have the same classification and discrimination ability for each category. For example, texture features can be used to classify different leaves well, but color features cannot achieve the same effect. Considering that texture features and shape features have different contributions to the classification of debris in different parts, multifeature fusion is needed to highlight the influence of different image features in different parts.Given a target image, its global feature vectorCq=Cq1,Cq2,…,Cqm and local feature vector Mq=Mq1,Mq2,…,Mqm are extracted where m and n represent the dimensions of the two feature vectors, respectively.Due to the different dimensions of global feature and local feature algorithms, there will be differences when using similarity measurement algorithms. In order to avoid this trouble,Cq and Mq need to be normalized, and the formula is as follows:(26)DC′i=DCimaxDCi−minDCi,i=1,2,…,N,DM′i=DMimaxDMi−minDMi,i=1,2,…,M,where DCi and DMi represent the distance between the original global feature and local feature and the corresponding feature of the sample image. DC′i and DM′i represent the normalized results of DCi and DMi, and N represents the number of images stored in the image database.LetQq represents the features after fusion of the two features, w1 is the weight of global features, and w2 is the weight of local features, and then, the features after fusion can be expressed by the following formula:(27)Qq=w1DC′i+w2DM′i.


### 2.2. Painting Image Classification Based on Transfer Learning

Transfer learning is a kind of machine learning algorithm whose primary objective is to derive new domain knowledge from previously acquired domain information. By using current information in the source domain to acquire unknown knowledge in the target domain, the acquisition of target domain knowledge is no longer a zero-sum game and may significantly boost the target domain’s learning impact. The transfer learning algorithm’s fundamental flow chart is shown in Figure6Figure 6
Basic flow chart of migration algorithm.Based on the result of feature fusion, the painting images are preliminarily classified, the deterministic samples and the nondeterministic samples are divided, and the parameters of the Gaussian model estimated are transferred to the target domain by the transfer learning algorithm to change the distribution of nondeterministic samples, so as to obtain accurate painting image classification results.Assuming that the painting image sample to be classified is called the target domainDT, the specific process of dividing DT is as follows:Step 1K-means clustering algorithm is used to cluster sampleDT in the target domain to obtain different painting image sample category labels;Step 2Assume that the purpose of painting image classification is to divide painting images intok categories, the label vectors of two different classification results are λa and λb, and the results of painting image classification are Ca1,Ca2,…,Cak and Cb1,Cb2,…,Cbk. Then, a symmetric matrix S with a size of k×k can be obtained, and the value in it represents the number of overlaps between classes. When the value is the largest, the category labels of the two classes are the same, and the matching results of all classes can be obtained after repeated for many times.Step 3After the successful matching of category labels, the clustering consistency valueCI is introduced for the effective discriminative sample DA and nondeterministic sample DB, which is mainly calculated by the following formula:(28)CIx=1Hmax∑t=1Hδπtx,L,where H represents the number of clustering, πtx represents the classification result of painting image sample x in the t-th classification, and L represents each category identification.Step 4Set a threshold value ofQ, and take samples of CI≥Q as deterministic samples DA, and samples of CI<Q as nondeterministic samples DB;Step 5According to the obtained clustering results, record the categorical label of the deterministic sample, which is recorded asla.After the division of the above steps, we have obtained the deterministic sampleDA, the nondeterministic sample DB, and the label la of the deterministic sample. What we need to do now is to change the painting image classification results of the nondeterministic sample DB by migrating the sample distribution of the source domain Ds. The following is how to transfer the parameters of the mixed Gaussian model (GMM) of the source domain to the target domain using the transfer learning algorithm:(i)
Step 1: the parameters of the mixed Gaussian model are estimated, and the quasimixing parameters, mean value, and covariance of the mixed Gaussian model in the source domain areαj, μj, and σjStep 2: according to the numbersαj, μj, and σj, calculate the membership function Z in the source domain, and the calculation formula of membership Z is as follows:(29)Zi,j=αjGXi;μj,σj∑j=1kαjGXi;μj,σj,wherei=m×n is the number of samples, and j=1,2,…,k, k are the number of categoriesStep 3: use the membership degreeZi,j calculated above to calculate the mixed Gaussian parameter αjnew, mean μjnew, and covariance σjnew of the sample under this membership degree in the target domain sample DT, and the calculation formulas are as follows:(30)αjnew=1n∑i=1nZi,j,μjnew=∑i=1nZi,jXi∑i=1nZi,j,σjnew=∑i=1nZi,jXi−μjXi−μjT∑i=1nZi,j.The new Gaussian mixture model parametersαjnew, μjnew, and σjnew were used to calculate the membership function Zi,j′ of the target domain DT, thus completing the task of migrating the parameters of sample points in the source domain to the target domain.Step 4: substitute the new parameter into the Gaussian distribution formula to obtain the probability value of each sample inDB, so as to obtain the classification result of DB images 88, namely the classification result of painting images.The specific implementation steps of painting image classification are shown in Figure7.(1)
Determine the number of deterministic and nondeterministic samples based on feature fusion results and cluster consistency values(2)
Determine the number of deterministic and nondeterministic samples based on feature fusion results and cluster consistency values(3)
Determine the number of deterministic and nondeterministic samples based on feature fusion results and cluster consistency values(4)
The source domain is picked at random from the remaining bands of the data, and the GMM parameters in the source domain are estimated(5)
The calculated parameters are transferred to the target domain to modify the distribution of the nondeterministic sample(6)
After calculating the final clustering result, Dunn's index was used to generate the clustering validity value, which was then compared to the initial clustering validity value(7)
If the clustering effectiveness decreases, repeat the above three steps and then reselect the source region for migration until a good painting image classification effect is achieved.Figure 7
Specific implementation steps of painting image classification.


## 2.1. Image Feature Extraction and Fusion

### 2.1.1. Global Feature Extraction

Color features: HSV color space corresponds to three visual features of the human eyes, respectively. The hue range is 0–360°, which corresponds to the actual color [7, 8]. Saturation S is proportional to the degree of brightness of a color, and brightness is proportional to the ability of light. The more the energy is, the brighter the light is [9]. HSV is more closely related to human perception and experience of color than the RGB system and is thus more extensively utilised. Therefore, this paper adopts HSV color space to extract color features of painting images. This model is shown in Figure 1.(1)Ei=∑i=1N1Npij,σi=1N∑j=1Npij−Ei2,si=1N∑j=1Npij−Ei33,where i represents the total amount of color channel of the painted image, and pij represents the color value of the i-th color channel of the j-th pixel [10].Figure 1
HSV color space model.Texture features: Wavelet analysis is the most widely used method in extracting image texture features [11]. This method is very stable in signal processing and has shown good results in many research fields, with broad application prospects. The principle of texture feature extraction using wavelet transform is shown in Figure 2.Figure 2
Flow chart of wavelet transform.The classical wavelet functions include Haar, Morlet, dbn, and sym N. The following is a brief introduction to Haar wavelet functions. The definition is shown in the following formula:(2)ψt=1,0≤t≤12,−1,12≤t≤1,0,otherwise.Let kernel functionψx be the female wavelet, and a cluster of orthogonal basis can be obtained through translation and scaling [12].(3)ψm,nx=2−m/2ψ2−mx−n.Then, the wavelet coefficients of the signalfx can be expressed by the following formula:(4)cm,n=∫−∞∞fxψm,nsdx.If the scale functionϕx satisfies the two-scale difference equation [13], it can be expressed as follows:(5)ϕx=2∑khkϕ2x−k.Then, the relation between the wavelet kernel functionψx and the scale function is(6)ψx=2∑kgkϕ2x−k,gk=−1kh1−k.Explicitϕx, ψx are not required for wavelet transformation but depend on hk, gk. A J-order wavelet decomposition can be written as follows:(7)f0x=∑kc0,kx=∑kcJ+1,kϕJ+1,kx+∑j=0Jdn+1,kφn+1,kx.If the coefficientc0,k is known, the relationship between the coefficient of grade j+1 and that of grade j is [14](8)cj+1,n=∑kcj,khk−2n,dj+1,n=∑kcj,kgk−2n.Combined with the above analysis, the wavelet coefficientdj,n is used to construct the recursive algorithm, which is described as follows:(9)cj,k=∑ncj+1,nhk−2n+∑ndj+1,ngk−2n.Leth¯n and g¯n be signals with impulse response and use their high and low filters to sample filter signals and combine them to achieve wavelet transform [15]. h¯n and g¯n are defined as follows:(10)h¯n=h−n,g¯n=g−n.In two-dimensional signal, the small and medium wave function can be expressed as(11)ϕx,y=ϕxϕy,ψ1x,y=ϕxψy,ψ2x,y=ψxϕy,ψ3x,y=ψxψy.The corresponding two-dimensional filtering function is(12)hLLk,l=hkhl,hLHk,l=hkgl,hHLk,l=gkhl,hHHk,l=gkgl.The image is set asfx,y, and the wavelet decomposition process of the image is shown in Figure 3 [16, 17].Figure 3
Schematic diagram of wavelet decomposition.Texture features are the result of frequency variations in all directions. Firstly, the traditional Chinese painting is decomposed into four components: LL, LH, HL, and HH. LL1 was decomposed and sampled to obtain second-order wavelet decomposition, as shown in Figure4 [18]. And, if further decomposition is required to obtain more vector information, the same operation is continued for LL2.Figure 4
First-order and second-order decomposition coefficients.Shape features: Compared with color and texture features, shape features belong to deeper image features. Therefore, in order to improve the extraction accuracy of painting image shape features, this paper uses Fourier shape descriptors [19, 20] to achieve this important goal.Suppose inXY there is a boundary in the plane, the boundary mainly consists of N coordinates for xn,yn points, the starting point at this point, moving counterclockwise along the border, will form the multiple trajectories, mainly expressed in x0,y0,x1,y1,…,xn−1,yn−1, at this time will be XY plane and complex plane UV overlap, and redefine the boundary points, as shown in Figure 5.(13)sn=xn+jyn,n=1,2,…,N−1.Figure 5
Two representations of boundary points.The discrete Fourier transform ofsn [21, 22] is(14)Sw=1N∑n=0N−1snexp−j2πwnN,w=0,1,…,N−1.The inverse Fourier transform ofSw is(15)Sk=∑n=0N−1Swexp−j2πwnN,k=0,1,…,N−1.If only the firstP coefficients of Sw are selected here, namely P≤N−1, another similar expression of Sk can be derived, namely the Fourier shape descriptor.(16)S^k=∑w=0P−1Swexp−j2πwkN,k=0,1,…,N−1.The shape features vector of the painting image is described by the Fourier shape descriptor, and the shape feature of the painting image is extracted.


### 2.1.2. Local Feature Extraction Based on SIFT Algorithm

Scale-invariant feature transform (SIFT) is a classical shape feature and belongs to a local feature algorithm. Local features extracted have invariance and stability [23]. The specific process is as follows.Extreme value detection of scale space: The scale-space kernel is defined as follows:(17)fout=K∗fin,where ∗ represents convolution operation, and K represents scale-space kernel. Gaussian difference scale space can be obtained by using this parameter, as follows:(18)Dx,y,σ=Gx,y,kσ−Gx,y,σ∗Ix,y.=Lx,y,kσ−Lx,y,σ.Among them,(19)Gx,y,σ=12πσ2e−x2+y2/2σ2,where σ is the standard deviation of the Gaussian kernel, and x,y is the spatial coordinate point of an image Ix,y.Positioning of key points: Key points are selected according to the scale and stability of candidate points. If the value of operation comparison is greater than or less than all adjacent points, the candidate point is identified as the key point [24]Key point direction assignment: After the key point is found, one or more directions are assigned to its local image gradient. Taylor's expansion (fitting function) corresponding to the Gaussian function is(20)Dx=D+∂DT∂xx+12xT∂2D∂x2x.For alternative pointsD and D′, D′=0, x, and the corresponding Dx are calculated, and the threshold is set as follows:(21)x=−∂2D∂x2−1∂D∂x,Dx=D+∞12∂DT∂xx.The Hessian matrix2×2 at the “feature point” with constant scale [25]is obtained, and the trace and determinant of the 2×2-dimensional Hessian matrix DxxDxyDxyDyy are solved.(22)TrH=Dxx+Dyy=α+β,DetH=Dxx+Dyy−Dxy2=αβ.Discard the edge response point, setγ as the ratio of two characteristic values, have α=γβ, then the following relationship exists:(23)TrH2DetH=α+β2αβ=rα+β2rβ2=R+12R.Feature points are judged by the threshold, more representative feature points are selected, and unstable key points are screened.(24)TrH2DetH<R+12R.Generate descriptors: From the local picture gradient information, a descriptor is constructed for each key point, and its scale is determined in the second step. The critical element is that the algorithm creates a huge number of highly distinctive characteristics in a variety of sizes and places [26]. The gradient’s modulus and direction are as follows:(25)mx,y=Lx+1,y−Lx−1,y2+Lx,y+1−Lx,y−12,θx,y=arctanLx,y+1−Lx,y−1Lx+1,y−Lx−1,y.The local feature extraction of the painting image is realized by generating descriptors to describe the local feature of the painting image.


### 2.1.3. Feature Fusion

In the process of painting image classification, not all feature descriptors have the same classification and discrimination ability for each category. For example, texture features can be used to classify different leaves well, but color features cannot achieve the same effect. Considering that texture features and shape features have different contributions to the classification of debris in different parts, multifeature fusion is needed to highlight the influence of different image features in different parts.Given a target image, its global feature vectorCq=Cq1,Cq2,…,Cqm and local feature vector Mq=Mq1,Mq2,…,Mqm are extracted where m and n represent the dimensions of the two feature vectors, respectively.Due to the different dimensions of global feature and local feature algorithms, there will be differences when using similarity measurement algorithms. In order to avoid this trouble,Cq and Mq need to be normalized, and the formula is as follows:(26)DC′i=DCimaxDCi−minDCi,i=1,2,…,N,DM′i=DMimaxDMi−minDMi,i=1,2,…,M,where DCi and DMi represent the distance between the original global feature and local feature and the corresponding feature of the sample image. DC′i and DM′i represent the normalized results of DCi and DMi, and N represents the number of images stored in the image database.LetQq represents the features after fusion of the two features, w1 is the weight of global features, and w2 is the weight of local features, and then, the features after fusion can be expressed by the following formula:(27)Qq=w1DC′i+w2DM′i.


## 2.1.1. Global Feature Extraction

Color features: HSV color space corresponds to three visual features of the human eyes, respectively. The hue range is 0–360°, which corresponds to the actual color [7, 8]. Saturation S is proportional to the degree of brightness of a color, and brightness is proportional to the ability of light. The more the energy is, the brighter the light is [9]. HSV is more closely related to human perception and experience of color than the RGB system and is thus more extensively utilised. Therefore, this paper adopts HSV color space to extract color features of painting images. This model is shown in Figure 1.(1)Ei=∑i=1N1Npij,σi=1N∑j=1Npij−Ei2,si=1N∑j=1Npij−Ei33,where i represents the total amount of color channel of the painted image, and pij represents the color value of the i-th color channel of the j-th pixel [10].Figure 1
HSV color space model.Texture features: Wavelet analysis is the most widely used method in extracting image texture features [11]. This method is very stable in signal processing and has shown good results in many research fields, with broad application prospects. The principle of texture feature extraction using wavelet transform is shown in Figure 2.Figure 2
Flow chart of wavelet transform.The classical wavelet functions include Haar, Morlet, dbn, and sym N. The following is a brief introduction to Haar wavelet functions. The definition is shown in the following formula:(2)ψt=1,0≤t≤12,−1,12≤t≤1,0,otherwise.Let kernel functionψx be the female wavelet, and a cluster of orthogonal basis can be obtained through translation and scaling [12].(3)ψm,nx=2−m/2ψ2−mx−n.Then, the wavelet coefficients of the signalfx can be expressed by the following formula:(4)cm,n=∫−∞∞fxψm,nsdx.If the scale functionϕx satisfies the two-scale difference equation [13], it can be expressed as follows:(5)ϕx=2∑khkϕ2x−k.Then, the relation between the wavelet kernel functionψx and the scale function is(6)ψx=2∑kgkϕ2x−k,gk=−1kh1−k.Explicitϕx, ψx are not required for wavelet transformation but depend on hk, gk. A J-order wavelet decomposition can be written as follows:(7)f0x=∑kc0,kx=∑kcJ+1,kϕJ+1,kx+∑j=0Jdn+1,kφn+1,kx.If the coefficientc0,k is known, the relationship between the coefficient of grade j+1 and that of grade j is [14](8)cj+1,n=∑kcj,khk−2n,dj+1,n=∑kcj,kgk−2n.Combined with the above analysis, the wavelet coefficientdj,n is used to construct the recursive algorithm, which is described as follows:(9)cj,k=∑ncj+1,nhk−2n+∑ndj+1,ngk−2n.Leth¯n and g¯n be signals with impulse response and use their high and low filters to sample filter signals and combine them to achieve wavelet transform [15]. h¯n and g¯n are defined as follows:(10)h¯n=h−n,g¯n=g−n.In two-dimensional signal, the small and medium wave function can be expressed as(11)ϕx,y=ϕxϕy,ψ1x,y=ϕxψy,ψ2x,y=ψxϕy,ψ3x,y=ψxψy.The corresponding two-dimensional filtering function is(12)hLLk,l=hkhl,hLHk,l=hkgl,hHLk,l=gkhl,hHHk,l=gkgl.The image is set asfx,y, and the wavelet decomposition process of the image is shown in Figure 3 [16, 17].Figure 3
Schematic diagram of wavelet decomposition.Texture features are the result of frequency variations in all directions. Firstly, the traditional Chinese painting is decomposed into four components: LL, LH, HL, and HH. LL1 was decomposed and sampled to obtain second-order wavelet decomposition, as shown in Figure4 [18]. And, if further decomposition is required to obtain more vector information, the same operation is continued for LL2.Figure 4
First-order and second-order decomposition coefficients.Shape features: Compared with color and texture features, shape features belong to deeper image features. Therefore, in order to improve the extraction accuracy of painting image shape features, this paper uses Fourier shape descriptors [19, 20] to achieve this important goal.Suppose inXY there is a boundary in the plane, the boundary mainly consists of N coordinates for xn,yn points, the starting point at this point, moving counterclockwise along the border, will form the multiple trajectories, mainly expressed in x0,y0,x1,y1,…,xn−1,yn−1, at this time will be XY plane and complex plane UV overlap, and redefine the boundary points, as shown in Figure 5.(13)sn=xn+jyn,n=1,2,…,N−1.Figure 5
Two representations of boundary points.The discrete Fourier transform ofsn [21, 22] is(14)Sw=1N∑n=0N−1snexp−j2πwnN,w=0,1,…,N−1.The inverse Fourier transform ofSw is(15)Sk=∑n=0N−1Swexp−j2πwnN,k=0,1,…,N−1.If only the firstP coefficients of Sw are selected here, namely P≤N−1, another similar expression of Sk can be derived, namely the Fourier shape descriptor.(16)S^k=∑w=0P−1Swexp−j2πwkN,k=0,1,…,N−1.The shape features vector of the painting image is described by the Fourier shape descriptor, and the shape feature of the painting image is extracted.


## 2.1.2. Local Feature Extraction Based on SIFT Algorithm

Scale-invariant feature transform (SIFT) is a classical shape feature and belongs to a local feature algorithm. Local features extracted have invariance and stability [23]. The specific process is as follows.Extreme value detection of scale space: The scale-space kernel is defined as follows:(17)fout=K∗fin,where ∗ represents convolution operation, and K represents scale-space kernel. Gaussian difference scale space can be obtained by using this parameter, as follows:(18)Dx,y,σ=Gx,y,kσ−Gx,y,σ∗Ix,y.=Lx,y,kσ−Lx,y,σ.Among them,(19)Gx,y,σ=12πσ2e−x2+y2/2σ2,where σ is the standard deviation of the Gaussian kernel, and x,y is the spatial coordinate point of an image Ix,y.Positioning of key points: Key points are selected according to the scale and stability of candidate points. If the value of operation comparison is greater than or less than all adjacent points, the candidate point is identified as the key point [24]Key point direction assignment: After the key point is found, one or more directions are assigned to its local image gradient. Taylor's expansion (fitting function) corresponding to the Gaussian function is(20)Dx=D+∂DT∂xx+12xT∂2D∂x2x.For alternative pointsD and D′, D′=0, x, and the corresponding Dx are calculated, and the threshold is set as follows:(21)x=−∂2D∂x2−1∂D∂x,Dx=D+∞12∂DT∂xx.The Hessian matrix2×2 at the “feature point” with constant scale [25]is obtained, and the trace and determinant of the 2×2-dimensional Hessian matrix DxxDxyDxyDyy are solved.(22)TrH=Dxx+Dyy=α+β,DetH=Dxx+Dyy−Dxy2=αβ.Discard the edge response point, setγ as the ratio of two characteristic values, have α=γβ, then the following relationship exists:(23)TrH2DetH=α+β2αβ=rα+β2rβ2=R+12R.Feature points are judged by the threshold, more representative feature points are selected, and unstable key points are screened.(24)TrH2DetH<R+12R.Generate descriptors: From the local picture gradient information, a descriptor is constructed for each key point, and its scale is determined in the second step. The critical element is that the algorithm creates a huge number of highly distinctive characteristics in a variety of sizes and places [26]. The gradient’s modulus and direction are as follows:(25)mx,y=Lx+1,y−Lx−1,y2+Lx,y+1−Lx,y−12,θx,y=arctanLx,y+1−Lx,y−1Lx+1,y−Lx−1,y.The local feature extraction of the painting image is realized by generating descriptors to describe the local feature of the painting image.


## 2.1.3. Feature Fusion

In the process of painting image classification, not all feature descriptors have the same classification and discrimination ability for each category. For example, texture features can be used to classify different leaves well, but color features cannot achieve the same effect. Considering that texture features and shape features have different contributions to the classification of debris in different parts, multifeature fusion is needed to highlight the influence of different image features in different parts.Given a target image, its global feature vectorCq=Cq1,Cq2,…,Cqm and local feature vector Mq=Mq1,Mq2,…,Mqm are extracted where m and n represent the dimensions of the two feature vectors, respectively.Due to the different dimensions of global feature and local feature algorithms, there will be differences when using similarity measurement algorithms. In order to avoid this trouble,Cq and Mq need to be normalized, and the formula is as follows:(26)DC′i=DCimaxDCi−minDCi,i=1,2,…,N,DM′i=DMimaxDMi−minDMi,i=1,2,…,M,where DCi and DMi represent the distance between the original global feature and local feature and the corresponding feature of the sample image. DC′i and DM′i represent the normalized results of DCi and DMi, and N represents the number of images stored in the image database.LetQq represents the features after fusion of the two features, w1 is the weight of global features, and w2 is the weight of local features, and then, the features after fusion can be expressed by the following formula:(27)Qq=w1DC′i+w2DM′i.


## 2.2. Painting Image Classification Based on Transfer Learning

Transfer learning is a kind of machine learning algorithm whose primary objective is to derive new domain knowledge from previously acquired domain information. By using current information in the source domain to acquire unknown knowledge in the target domain, the acquisition of target domain knowledge is no longer a zero-sum game and may significantly boost the target domain’s learning impact. The transfer learning algorithm’s fundamental flow chart is shown in Figure6Figure 6
Basic flow chart of migration algorithm.Based on the result of feature fusion, the painting images are preliminarily classified, the deterministic samples and the nondeterministic samples are divided, and the parameters of the Gaussian model estimated are transferred to the target domain by the transfer learning algorithm to change the distribution of nondeterministic samples, so as to obtain accurate painting image classification results.Assuming that the painting image sample to be classified is called the target domainDT, the specific process of dividing DT is as follows:Step 1K-means clustering algorithm is used to cluster sampleDT in the target domain to obtain different painting image sample category labels;Step 2Assume that the purpose of painting image classification is to divide painting images intok categories, the label vectors of two different classification results are λa and λb, and the results of painting image classification are Ca1,Ca2,…,Cak and Cb1,Cb2,…,Cbk. Then, a symmetric matrix S with a size of k×k can be obtained, and the value in it represents the number of overlaps between classes. When the value is the largest, the category labels of the two classes are the same, and the matching results of all classes can be obtained after repeated for many times.Step 3After the successful matching of category labels, the clustering consistency valueCI is introduced for the effective discriminative sample DA and nondeterministic sample DB, which is mainly calculated by the following formula:(28)CIx=1Hmax∑t=1Hδπtx,L,where H represents the number of clustering, πtx represents the classification result of painting image sample x in the t-th classification, and L represents each category identification.Step 4Set a threshold value ofQ, and take samples of CI≥Q as deterministic samples DA, and samples of CI<Q as nondeterministic samples DB;Step 5According to the obtained clustering results, record the categorical label of the deterministic sample, which is recorded asla.After the division of the above steps, we have obtained the deterministic sampleDA, the nondeterministic sample DB, and the label la of the deterministic sample. What we need to do now is to change the painting image classification results of the nondeterministic sample DB by migrating the sample distribution of the source domain Ds. The following is how to transfer the parameters of the mixed Gaussian model (GMM) of the source domain to the target domain using the transfer learning algorithm:(i)
Step 1: the parameters of the mixed Gaussian model are estimated, and the quasimixing parameters, mean value, and covariance of the mixed Gaussian model in the source domain areαj, μj, and σjStep 2: according to the numbersαj, μj, and σj, calculate the membership function Z in the source domain, and the calculation formula of membership Z is as follows:(29)Zi,j=αjGXi;μj,σj∑j=1kαjGXi;μj,σj,wherei=m×n is the number of samples, and j=1,2,…,k, k are the number of categoriesStep 3: use the membership degreeZi,j calculated above to calculate the mixed Gaussian parameter αjnew, mean μjnew, and covariance σjnew of the sample under this membership degree in the target domain sample DT, and the calculation formulas are as follows:(30)αjnew=1n∑i=1nZi,j,μjnew=∑i=1nZi,jXi∑i=1nZi,j,σjnew=∑i=1nZi,jXi−μjXi−μjT∑i=1nZi,j.The new Gaussian mixture model parametersαjnew, μjnew, and σjnew were used to calculate the membership function Zi,j′ of the target domain DT, thus completing the task of migrating the parameters of sample points in the source domain to the target domain.Step 4: substitute the new parameter into the Gaussian distribution formula to obtain the probability value of each sample inDB, so as to obtain the classification result of DB images 88, namely the classification result of painting images.The specific implementation steps of painting image classification are shown in Figure7.(1)
Determine the number of deterministic and nondeterministic samples based on feature fusion results and cluster consistency values(2)
Determine the number of deterministic and nondeterministic samples based on feature fusion results and cluster consistency values(3)
Determine the number of deterministic and nondeterministic samples based on feature fusion results and cluster consistency values(4)
The source domain is picked at random from the remaining bands of the data, and the GMM parameters in the source domain are estimated(5)
The calculated parameters are transferred to the target domain to modify the distribution of the nondeterministic sample(6)
After calculating the final clustering result, Dunn's index was used to generate the clustering validity value, which was then compared to the initial clustering validity value(7)
If the clustering effectiveness decreases, repeat the above three steps and then reselect the source region for migration until a good painting image classification effect is achieved.Figure 7
Specific implementation steps of painting image classification.


## 3. Experimental Design

In order to verify the validity of the painting image classification method based on transfer learning and feature fusion designed in this paper, relevant experimental tests are carried out, and the experimental environmental parameters are shown in Table1.Table 1
Experimental environmental parameters.
The name of theParameterOperating systemWindows 10CPUIntel core i5-7300HQMemory32GHard disk500 GBRun a memory8GFrequency2.1 GHzSimulation softwareMatlab R2014aIn the network, crawler technology is used to crawl 10000 painting images, and all data are denoised. The sample images after denoising are taken as experimental sample images, part of which are shown in Figure8.Figure 8
Images of some experimental samples.Reference [5] proposed a painting image classification method based on convolution neural networks, reference [6] proposed a painting image classification method based on information entropy, and this paper proposed a painting image classification method based on transfer learning and feature fusion as an experiment contrast method, by comparing different experimental indexes to verify the application effect of different methods.


### 3.1. A Comparison of Feature Extraction Error Rates

Figure9 compares the feature extraction error rate of painting pictures using the convolutional neural network classification technique, the information entropy-based classification approach, and the classification method based on transfer learning and feature fusion.Figure 9
Comparison of error rates of feature extraction.Analysis of the data in Figure9 shows that the error rate of feature extraction of painting image based on convolutional neural network is between 13% and 20%, that of feature extraction of painting image based on information entropy is between -17% and 2%, and that of feature extraction of painting image based on transfer learning and feature fusion is between 2% and 4%. It shows that the feature extraction accuracy of the proposed method is higher, which can lay a foundation for the subsequent feature classification of painting images.


### 3.2. Feature Extraction Time Comparison

The feature extraction time of painting images based on convolutional neural network classification method, information entropy-based classification method, and classification method based on transfer learning and feature fusion is compared, and the results are shown in Figure10.Figure 10
Comparison of painting feature extraction time.As shown in Figure10, as the number of tests increases, the time it takes to extract painting attributes from various approaches varies. The classification approach based on convolutional neural networks has a maximum and lowest time of feature extraction of 4.0s and 2.0s, respectively. The greatest and lowest times for feature extraction based on information entropy are, respectively, 3.9s and 1.7s. The maximum time and minimum time of feature extraction of painting image based on the classification method of transfer learning and feature fusion are 0.7s and 0.5s, respectively. Compared with the experimental comparison method, feature extraction of painting image in this paper is shorter and more efficient.


### 3.3. Classification and Comparison of Painting Images

The accuracy of painting image classification based on convolutional neural network, information entropy, and transfer learning and feature fusion is compared, and the comparison results are shown in Table2.Table 2
Comparison of classification accuracy of painting images (unit:%).
Number of experimentClassification method based on convolutional neural networkClassification method based on information entropyClassification method based on transfer learning and feature fusion1085.6379.5698.752084.7578.8196.313081.5575.6697.484083.2474.1295.375084.7585.3295.446086.3374.3397.147084.1776.5596.52Average84.3577.7696.72By analyzing the comparison results of the accuracy of painting image classification in Table2, it can be seen that the average accuracy of painting image classification based on convolutional neural network is 84.35%, and that of painting image classification based on information entropy is 77.76%. The average accuracy rate of painting image classification based on transfer learning and feature fusion is 96.72%, indicating that the classification result of this method is more accurate.


## 3.1. A Comparison of Feature Extraction Error Rates

Figure9 compares the feature extraction error rate of painting pictures using the convolutional neural network classification technique, the information entropy-based classification approach, and the classification method based on transfer learning and feature fusion.Figure 9
Comparison of error rates of feature extraction.Analysis of the data in Figure9 shows that the error rate of feature extraction of painting image based on convolutional neural network is between 13% and 20%, that of feature extraction of painting image based on information entropy is between -17% and 2%, and that of feature extraction of painting image based on transfer learning and feature fusion is between 2% and 4%. It shows that the feature extraction accuracy of the proposed method is higher, which can lay a foundation for the subsequent feature classification of painting images.


## 3.2. Feature Extraction Time Comparison

The feature extraction time of painting images based on convolutional neural network classification method, information entropy-based classification method, and classification method based on transfer learning and feature fusion is compared, and the results are shown in Figure10.Figure 10
Comparison of painting feature extraction time.As shown in Figure10, as the number of tests increases, the time it takes to extract painting attributes from various approaches varies. The classification approach based on convolutional neural networks has a maximum and lowest time of feature extraction of 4.0s and 2.0s, respectively. The greatest and lowest times for feature extraction based on information entropy are, respectively, 3.9s and 1.7s. The maximum time and minimum time of feature extraction of painting image based on the classification method of transfer learning and feature fusion are 0.7s and 0.5s, respectively. Compared with the experimental comparison method, feature extraction of painting image in this paper is shorter and more efficient.


## 3.3. Classification and Comparison of Painting Images

The accuracy of painting image classification based on convolutional neural network, information entropy, and transfer learning and feature fusion is compared, and the comparison results are shown in Table2.Table 2
Comparison of classification accuracy of painting images (unit:%).
Number of experimentClassification method based on convolutional neural networkClassification method based on information entropyClassification method based on transfer learning and feature fusion1085.6379.5698.752084.7578.8196.313081.5575.6697.484083.2474.1295.375084.7585.3295.446086.3374.3397.147084.1776.5596.52Average84.3577.7696.72By analyzing the comparison results of the accuracy of painting image classification in Table2, it can be seen that the average accuracy of painting image classification based on convolutional neural network is 84.35%, and that of painting image classification based on information entropy is 77.76%. The average accuracy rate of painting image classification based on transfer learning and feature fusion is 96.72%, indicating that the classification result of this method is more accurate.


## 4. Conclusion

In the development of human civilization, the painting is the precious spiritual wealth. Through exquisite painting, it can fully display the rich spirit of the era. The present stage of painting gradually moves towards the direction of the digital and intelligent, so a lot of drawing is carried out in the network. The classification of the painting according to the content of the painting style is particularly important. Therefore, this paper introduces transfer learning and feature fusion methods to design a new painting image classification method. Experimental results show that the error rate of feature extraction of painting images in this method ranges from 2% to 4%, the maximum time of feature extraction is 0.7s, the minimum is 0.5s, and the average accuracy rate of painting image classification is 96.72%, indicating that this method can not only achieve accurate and rapid feature extraction of painting images but also achieve the correct classification of painting images. It can lay a solid foundation for the development of painting image classification.



---

*Source: 5254823-2022-06-15.xml*