# A Joint Optimization Framework of the Embedding Model and Classifier for Meta-Learning

Authors: Zhongyu Liu; Xu Chu; Yan Lu; Wanli Yu; Shuguang Miao; Enjie Ding
Journal: Scientific Programming
 (2021)

---

## Abstract

The aim of meta-learning is to train the machine to learn quickly and accurately. Improving the performance of the meta-learning model is important in solving the problem of small samples and in achieving general artificial intelligence. A meta-learning method based on feature embedding that exhibits good performance on the few-shot problem was previously proposed. In this method, the pretrained deep convolution neural network was used as the embedding model of sample features, and the output of one layer was used as the feature representation of samples. The main limitation of the method is the inability to fuse low-level texture features and high-level semantic features of the embedding model and joint optimization of the embedding model and classifier. Therefore, a multilayer adaptive joint training and optimization method of the embedding model was proposed in the current study. The main characteristics of the current method include using multilayer adaptive hierarchical loss to train the embedding model and using the quantum genetic algorithm to jointly optimize the embedding model and classifier. Validation was performed based on multiple public datasets for meta-learning model testing. The proposed method shows higher accuracy compared with multiple baseline methods.


---

## Body

## 1. Introduction

The computing capacity of computers has recently developed and the amount of data has significantly increased. Deep learning has shown significant application in several tasks such as computer vision and natural language processing [1]. However, data collection and annotation of sample requires a lot of manual work, mainly data obtained from the industrial field, which is challenging to analyze [2]. Currently, the overfitting phenomenon appears easily with deep learning models based on supervised learning trained with small sample sizes [3]. Therefore, machine learning model design based on small samples has high application value. Small samples learning is also known as few-shot learning in the machine learning field. Notably, a machine learning model can achieve strong artificial intelligence [2].Meta-learning is mainly used to model for few-shot learning problems. Meta-learning can learn new tasks quickly with the purpose of training the model to learn to learn. The traditional meta-learning methods can be divided into four categories including: methods based on data augmentation, methods based on metric learning, methods based on strong generalization of initialization, and methods based on parameter optimization. Their characteristics are explored in detail in the following section.The method based on data augmentation is mainly used to expand the data volume and increase the number of samples. This process prevents the model from premature fitting in the training process and improves the generalization performance [4]. However, the number of samples for the deep neural network is often too large, and this method cannot meet the need of model training alone [5]. Therefore, the method is conjoined with other methods [6]. The metric-based method identifies small samples by calculating the feature vectors between samples and determining the feature distance between different samples [7]. This method mainly focuses on the designed similarity measurement [8]. Notably, the performance of the model is low if the samples have a close feature distance, mainly in the classification of fine-grained samples. The method based on external memory introduces the external memory module and achieves fast encoding of new information through the long-term and short-term memory functions. The main limitation of this model is that the efficiency of information filtering and classification storage is low and the number of external memory network parameters is too large, thus leading to an increase in training difficulty. The method based on initialization is mainly used to optimize the loss function which refers to the learning effect obtained by the expected gradient updating of a few steps. The training process comprises two parts: the inner cycle and the outer cycle [9]. Therefore, the model can quickly fit the distribution of multitask data and not the distribution of single task features. The gradient descent of the inner cycle in this method mainly depends on the setting of the learning rate and requires significant manual intervention.In addition to the meta-learning methods described above, a new method was proposed and verified by [10]. This method comprises two stages. Firstly, a good feature embedding space is sought to map small samples to feature space. Second, a classifier is trained using these embedding features to quickly determine the feature distribution of samples and then achieve the inference of small samples.The new method makes use of supervised learning to train a deep neural network to represent the features of the samples. In addition, it explores the performance of small samples embedded in different layers of the network on the classifier in the second stage of training. The characteristics of different layers of the deep convolutional neural network comprise different information. The bottom layer comprises high amounts of low-level visual information, whereas the top layer comprises semantic information [11, 12]. Low-level visual information and semantic information are important in visual recognition tasks. Therefore, the fusion of the features output by different layers of the CNNs (convolutional neural networks) is recommended for the effective embedding of the samples into the feature space. An adaptive hierarchical weighted loss training method of deep CNNs can optimize the feature representation of samples as presented in the current study.The main contributions of the current study are presented as follows:A joint optimization strategy of embedding model and meta-learner classifier is proposed. The embedding model is trained by adaptive multilevel loss, and the embedding model is optimized by combining results of the meta-learning classifier.A quantum genetic algorithm is adopted to optimize the algorithm. The algorithm optimizes the weight of the hierarchical loss function of the embedding model and optimizes the joint feature expression for the classifier.Experimental results based on multiple meta-learning datasets indicate that the proposed method is effective and reliable.The study explores related studies and summarizes related methods of meta-learning. In addition, the proposed method is illustrated in detail. The method is then verified through experiments and the findings are presented.


## 2. Related Studies

### 2.1. Meta-Learning Based on Measurement

The goal of measurement of learning [13] is to obtain a paired similarity measurement function, in which similar sample pairs have a high similarity score, while dissimilar sample pairs have a low similarity score. Meta-learning methods adopt measurement learning strategies. Koch et al. [7] proposed a twin network that calculates the similarity between the query sample and all single-annotated samples. Hoffer and Ailon [14] designed a ternary network, which adopted positive and negative sample rules and predicted the category of the sample by calculating the distance between the query sample and the positive and negative samples. Vinyals et al. [15] designed a matching network based on memory and attention mechanisms, which can quickly learn the distribution of features of training samples. Snell et al. [16] used a prototype network to predict the category of the sample by learning the prototype points and then calculating the distance between the new sample and the prototype points. In addition, Sung et al. [17] proposed a relational network in 2018, in which the similarity between samples was calculated by embedding modules and relationship modules to complete the prediction of categories of new tasks.


### 2.2. Meta-Learning Based on Optimization

A previous study reported a meta-learning model based on optimization LSTM (Long Short-Term Memory) with hidden units in the method set as learners’ parameters [18]. Meta-learners can adjust their learning rate based on the responses of different spatiotemporal models so as to train the spatiotemporal models faster. Deep learning usually adopts backpropagation and gradient descending methods to optimize massive parameters in the network. The feature expression performance of deep neural networks depends on a large amount of data. A meta-learning method based on the strong generalization of initialization parameters was previously reported [9]. The goal of the optimization of the network parameters of MAML (model-agnostic meta-learning) is to separate the gradient descent and backpropagation from the used data, thus obtaining better generalization performance. Several methods have been used to improve MAML, such as combining the hidden space method [19] and Bayesian prior [20] and optimizing the gradient descent process [21]. Moreover, MAML can be used for a series of models based on gradient descent, including classification, regression, and reinforcement learning owing to its openness and flexibility [22].


### 2.3. Meta-Learning Based on Data Expansion

A simple and general data enhancement framework called MetaGan was proposed previously [4]. This method mainly defined a tighter decision boundary for the model by distinguishing real data from virtual data with the aim of improving the feature extraction ability of the model. A previous study proposed a feature “analogy” method that divides the model into two parts: representation learning and small sample learning [23]. In the presentation learning stage, the learner extracts accurate feature representations on the base classes containing a large amount of data, whereas, in the small sample learning stage, the learner learns the classifier on the joint space of the new type containing a small amount of data and the previous base types. A data generation structure has been previously reported that mainly uses CycleGan as the generator of the new category of data and adds noise to the new data to make its distribution more diversified [24].


## 2.1. Meta-Learning Based on Measurement

The goal of measurement of learning [13] is to obtain a paired similarity measurement function, in which similar sample pairs have a high similarity score, while dissimilar sample pairs have a low similarity score. Meta-learning methods adopt measurement learning strategies. Koch et al. [7] proposed a twin network that calculates the similarity between the query sample and all single-annotated samples. Hoffer and Ailon [14] designed a ternary network, which adopted positive and negative sample rules and predicted the category of the sample by calculating the distance between the query sample and the positive and negative samples. Vinyals et al. [15] designed a matching network based on memory and attention mechanisms, which can quickly learn the distribution of features of training samples. Snell et al. [16] used a prototype network to predict the category of the sample by learning the prototype points and then calculating the distance between the new sample and the prototype points. In addition, Sung et al. [17] proposed a relational network in 2018, in which the similarity between samples was calculated by embedding modules and relationship modules to complete the prediction of categories of new tasks.


## 2.2. Meta-Learning Based on Optimization

A previous study reported a meta-learning model based on optimization LSTM (Long Short-Term Memory) with hidden units in the method set as learners’ parameters [18]. Meta-learners can adjust their learning rate based on the responses of different spatiotemporal models so as to train the spatiotemporal models faster. Deep learning usually adopts backpropagation and gradient descending methods to optimize massive parameters in the network. The feature expression performance of deep neural networks depends on a large amount of data. A meta-learning method based on the strong generalization of initialization parameters was previously reported [9]. The goal of the optimization of the network parameters of MAML (model-agnostic meta-learning) is to separate the gradient descent and backpropagation from the used data, thus obtaining better generalization performance. Several methods have been used to improve MAML, such as combining the hidden space method [19] and Bayesian prior [20] and optimizing the gradient descent process [21]. Moreover, MAML can be used for a series of models based on gradient descent, including classification, regression, and reinforcement learning owing to its openness and flexibility [22].


## 2.3. Meta-Learning Based on Data Expansion

A simple and general data enhancement framework called MetaGan was proposed previously [4]. This method mainly defined a tighter decision boundary for the model by distinguishing real data from virtual data with the aim of improving the feature extraction ability of the model. A previous study proposed a feature “analogy” method that divides the model into two parts: representation learning and small sample learning [23]. In the presentation learning stage, the learner extracts accurate feature representations on the base classes containing a large amount of data, whereas, in the small sample learning stage, the learner learns the classifier on the joint space of the new type containing a small amount of data and the previous base types. A data generation structure has been previously reported that mainly uses CycleGan as the generator of the new category of data and adds noise to the new data to make its distribution more diversified [24].


## 3. Methods

The current study reviews the meta-learning method based on feature embedding and then illustrates the algorithm framework proposed in the current work. The algorithm framework mainly comprises an adaptive hierarchical loss function optimization stage, an adaptive feature fusion stage, and the original classifier training evaluation and testing stage. The method is described in detail in the following section.The datasets of meta-learning tasks can be divided into meta-training sets, meta-evaluation sets, and meta-testing sets. The traditional meta-learning method constructs the learning task based on the training set, which can be represented by a binary setDtrain,s,Dtrain,q. The binary set then forms the support set and the query set. The support set and the query set normally have a few small samples, and this form is called N-way-K-shot. A basic learner is defined as follows: A: y∗=fθx∗, in which ∗ represents S and Q. A learns based on the training set Dtrain,s,Dtrain,q to gain strong learning ability. A relative support set and query set should be constructed on the evaluation set or test set Dval,s,Dval,qDtest,s,Dtest,q to test the learning performance on the new task of A. A new meta-learning method is proposed by a previous study [10], which defines an embedding mapping to map the features of samples into the embedding space. The training tasks of basic learners are then transformed into(1)θ=ΑDjtrain,ϕ=argminθLbaseDjtrain;θ,ϕ+Rθ,where L represents the loss function and R represents the regular term. A good embedding model in the algorithm is aimed at high accuracy. The performance of the embedding model and meta-learners can be determined by finding the error between the two using the test datasets.


### 3.1. Algorithm Framework

The output features of different layers of CNNs have unique characteristics. The output in the top layer of CNNs comprises abundant semantic information, whereas the convolutional layer near the input comprises abundant low-level visual information. An embedding model and basic learner joint optimization algorithm are used to effectively fuse and utilize the information of the different layers. The algorithm framework is presented in Figure1. The framework mainly comprises two parts: the optimization module of the adaptive hierarchical embedding model and the adaptive hierarchical feature fusion module. The optimization goal of the embedding model is the same as that of the feature fusion model, and both aim at the accuracy of meta test. Therefore, the optimization of the two models uses the same fitness function, which makes the modules of different stages evolve in the same direction. In addition, the optimization process of each embedding model is a supervised training process of a convolutional neural network. The training of the neural network usually takes a lot of time, so parallel programming is used to set the optimization process of each embedding model as a process. According to the graphics card equipped with the computer and its performance, n processes can be run at the same time, that is, n neural network models can be trained at the same time, which will greatly improve the efficiency of the algorithm. Compared with serial programming, it will effectively save a lot of time. The framework below describes the algorithm flow of the two modules in detail.Figure 1
Joint optimization framework of the embedding model and classifier based on quantum genetic algorithm.


### 3.2. Quantum Genetic Algorithm

Encoding of chromosomes in the quantum genetic algorithm (QGA) uses the probability range of qubits in quantum physics, such that each chromosome represents multiple states. Quantum gates and revolving gates in quantum physics are used to update chromosomes to achieve the purpose of optimizing the population. An advantage of quantum genetic algorithm is its quantum parallelism, which improves the search ability of population size and increases the probability of obtaining the optimal solution compared with the traditional genetic algorithm [25].Qubits are the basic storage units in quantum computing and are different from the bits referred to in existing computers; however, they are a two-state system. The two states in this case refer to two independent states in quantum computing: the 0 state and the 1 state. Borrowing the Dirac symbol “|>”,0 and 1 are used to represent the spin-down state (0 state) and the spin-up state (1 state), respectively. A quantum state is the superposition of these two states, which can be expressed as a linear combination of the two states as follows:(2)ψ=α|0+β|1,where α and β represent the probability amplitudes of 0 and 1, respectively, and they must meet the conditions:(3)α2+β2=1,where αi2 indicates the probability of 0 state and βi2 indicates the probability of 1 state. This implies that when the probability of state 0 is 1, the probability of state 1 must be zero. Conversely, when the probability of state 1 is 1, the probability of state 0 is 0. The amplitude probability α,βT represents the chromosome of the qubit. The set of all chromosomes is expressed as P=p1,p2,…,pn, where n represents the size of the total chromosomes. A quantum chromosome consisting of qubits can be described as follows:(4)pi=α1α2α3⋯αmβ1β2β3⋯βm,where m represents the gene number of individual chromosomes.To produce excellent individuals with a high probability in the iterative process of the population, the quantum genetic algorithm introduces a quantum revolving gate to realize the change of the population and abandons the selection, crossover, and mutation operations used in the genetic algorithm. The quantum revolving gate can increase the diversity of the population gene pool by adjusting the quantum state probability, such that the solution appears on the individuals with the highest adaptability. Therefore, the quantum revolving gate is very important in the quantum genetic algorithm. The expression of the quantum revolving gate is as follows:(5)Rθ=cosθi−sinθisinθicosθi.The iterative process of the quantum revolving gate is expressed as follows:(6)αi′βi′=Rθαiβi.Here,αi,βiT represents the i−th qubit of the current chromosome, αi′,βi′ indicates the state of the qubit after rotation, θi represents the rotation angle, and the direction and value of θi is set by adjustment strategy. The adjustment strategy is presented in Table 1.Table 1
Quantum rotation gate adjustment strategy [26].
xibifxi≥fbiΔθisαi,βiαiβi>0αiβi<0αi=0βi=000False0————00True0————00Falseδ+1−10±100Trueδ−1+1±1010Falseδ-1+1±1010Trueδ+1−10±111False0————11True0————θi=sαi,βiΔθi represents the angle of rotation, whereas sαi,βi represents the direction of rotation and Δθi represents the increment of rotation angle, to obtain convergence direction and convergence speed of the algorithm. In addition, in the adjustment strategy of the revolving gate, xi represents the i−th gene of the current chromosome and fxi indicates the fitness of the individual. bi represents the i−th gene in the current best chromosome, and fbi represents the fitness of the best individual. This method uses the fitness fbi of the best individual in the current population as a target value and compares the fitness fxi of the individuals in the population with it. If fxi>fbi, then the corresponding qubits in the individual are adjusted such that the probability amplitude αi,βi approaches xi. Otherwise, if fxi<fbi, the corresponding quantum of the individual in the population bit is adjusted, such that the probability amplitude αi,βi approaches bi.


### 3.3. Adaptive Hierarchical Loss Training Stage of the Embedding Model

The main purpose of the embedding model is to obtain the feature vector of the sample by mapping the original sample to the feature space. The meta-learner then learns the feature distribution of the sample based on the feature vector. Therefore, the performance of the embedding model is very important. The adaptive multilevel loss training method is adopted to train the embedding model to improve the feature representation performance of different layers of CNNs. The trained model is fused by a multiscale adaptive feature fusion algorithm. The framework of the adaptive multilayer loss training algorithm is shown in Figure2.Figure 2
Adaptive hierarchical multistage loss training framework for the embedding model.The formal expression of the adaptive multilayer loss training algorithm is as follows: letF be any lightweight network, which normally has L stages. The output feature map from any intermediate stage is represented by Fl∈ℝHl×Wl×Cl, where Hl, Wl, Cl represent the height, width, and number of channels of the feature map at l−th stage, and l=1,2,…,L. The aim is to impose classification losses on the feature map extracted at different intermediate stages. The feature map of each stage performs the average pooling operation to obtain the feature vector fl of the stage:(7)fl=PoolavgFl.The Dot product similarity of the class prototype vector at this stage is then calculated to obtain the logits of the class, and finally, the loss function at this stage is obtained as shown in Figure2:(8)lossl=ℓlSoftmaxfl=−logefl,y∑k=1Cefl,k.Loss functions of all stages are weighted and summed to obtain the loss function of the network as follows:(9)loss=∑la=15ωl·lossl.


### 3.4. Meta-Evaluation and Meta-Test

The trained embedding model is used to extract the features of the samples in the meta-evaluation set and meta-test set. To make full use of the features of different layers of the embedding model, the method of feature fusion is used to represent the features of the samples. Several fusion methods have been explored. In the current study, the method of feature vector Mosaic was used as follows:(10)Ffuse=concatγ1f1,γ2f2,…,γnfn.The fused input classifier is trained or tested, and the calculation formula is expressed as follows:(11)zpredict=classifierFfuse.The fitness function of the quantum genetic algorithm is expressed as follows:(12)fitnessmi=gx,gx=# of correctly predicted samplestotal number of samples×100%.All models are tested using the meta-test set when the optimization of the embedding model and feature fusion method is completed. The process of the meta-testing stage is the same as that of meta-evaluation. Formulas (10) and (11) are used for meta-testing. The algorithm flow is shown in Figure 3. The next section describes the details of meta-testing.Figure 3
Adaptive hierarchical feature fusion method for meta-evaluation and meta-test.


### 3.5. Meta-Classifier

Meta-learners are an important part of meta-learning. Meta-learners are referred to as meta-classifiers in the current study. Several models are available for classification tasks, such as nearest neighbor algorithm, linear regression, logistic regression, and support vector machine. The current study mainly uses the widely used logistic regression model for meta-classification. Logistic regression and linear regression are similar to the generalized linear model. Logistic regression assumes that the dependent variabley follows a Bernoulli distribution, whereas linear regression assumes that the dependent variable y follows a Gaussian distribution. Therefore, there are several similarities between logistic regression and linear regression. However, logistic regression introduces nonlinear factors through the Sigmoid function; thus, it can easily handle the 0/1 classification problem. If the Sigmoid mapping function is removed, the logistic regression algorithm is similar to linear regression. This implies that logistic regression is theoretically supported by linear regression.


## 3.1. Algorithm Framework

The output features of different layers of CNNs have unique characteristics. The output in the top layer of CNNs comprises abundant semantic information, whereas the convolutional layer near the input comprises abundant low-level visual information. An embedding model and basic learner joint optimization algorithm are used to effectively fuse and utilize the information of the different layers. The algorithm framework is presented in Figure1. The framework mainly comprises two parts: the optimization module of the adaptive hierarchical embedding model and the adaptive hierarchical feature fusion module. The optimization goal of the embedding model is the same as that of the feature fusion model, and both aim at the accuracy of meta test. Therefore, the optimization of the two models uses the same fitness function, which makes the modules of different stages evolve in the same direction. In addition, the optimization process of each embedding model is a supervised training process of a convolutional neural network. The training of the neural network usually takes a lot of time, so parallel programming is used to set the optimization process of each embedding model as a process. According to the graphics card equipped with the computer and its performance, n processes can be run at the same time, that is, n neural network models can be trained at the same time, which will greatly improve the efficiency of the algorithm. Compared with serial programming, it will effectively save a lot of time. The framework below describes the algorithm flow of the two modules in detail.Figure 1
Joint optimization framework of the embedding model and classifier based on quantum genetic algorithm.


## 3.2. Quantum Genetic Algorithm

Encoding of chromosomes in the quantum genetic algorithm (QGA) uses the probability range of qubits in quantum physics, such that each chromosome represents multiple states. Quantum gates and revolving gates in quantum physics are used to update chromosomes to achieve the purpose of optimizing the population. An advantage of quantum genetic algorithm is its quantum parallelism, which improves the search ability of population size and increases the probability of obtaining the optimal solution compared with the traditional genetic algorithm [25].Qubits are the basic storage units in quantum computing and are different from the bits referred to in existing computers; however, they are a two-state system. The two states in this case refer to two independent states in quantum computing: the 0 state and the 1 state. Borrowing the Dirac symbol “|>”,0 and 1 are used to represent the spin-down state (0 state) and the spin-up state (1 state), respectively. A quantum state is the superposition of these two states, which can be expressed as a linear combination of the two states as follows:(2)ψ=α|0+β|1,where α and β represent the probability amplitudes of 0 and 1, respectively, and they must meet the conditions:(3)α2+β2=1,where αi2 indicates the probability of 0 state and βi2 indicates the probability of 1 state. This implies that when the probability of state 0 is 1, the probability of state 1 must be zero. Conversely, when the probability of state 1 is 1, the probability of state 0 is 0. The amplitude probability α,βT represents the chromosome of the qubit. The set of all chromosomes is expressed as P=p1,p2,…,pn, where n represents the size of the total chromosomes. A quantum chromosome consisting of qubits can be described as follows:(4)pi=α1α2α3⋯αmβ1β2β3⋯βm,where m represents the gene number of individual chromosomes.To produce excellent individuals with a high probability in the iterative process of the population, the quantum genetic algorithm introduces a quantum revolving gate to realize the change of the population and abandons the selection, crossover, and mutation operations used in the genetic algorithm. The quantum revolving gate can increase the diversity of the population gene pool by adjusting the quantum state probability, such that the solution appears on the individuals with the highest adaptability. Therefore, the quantum revolving gate is very important in the quantum genetic algorithm. The expression of the quantum revolving gate is as follows:(5)Rθ=cosθi−sinθisinθicosθi.The iterative process of the quantum revolving gate is expressed as follows:(6)αi′βi′=Rθαiβi.Here,αi,βiT represents the i−th qubit of the current chromosome, αi′,βi′ indicates the state of the qubit after rotation, θi represents the rotation angle, and the direction and value of θi is set by adjustment strategy. The adjustment strategy is presented in Table 1.Table 1
Quantum rotation gate adjustment strategy [26].
xibifxi≥fbiΔθisαi,βiαiβi>0αiβi<0αi=0βi=000False0————00True0————00Falseδ+1−10±100Trueδ−1+1±1010Falseδ-1+1±1010Trueδ+1−10±111False0————11True0————θi=sαi,βiΔθi represents the angle of rotation, whereas sαi,βi represents the direction of rotation and Δθi represents the increment of rotation angle, to obtain convergence direction and convergence speed of the algorithm. In addition, in the adjustment strategy of the revolving gate, xi represents the i−th gene of the current chromosome and fxi indicates the fitness of the individual. bi represents the i−th gene in the current best chromosome, and fbi represents the fitness of the best individual. This method uses the fitness fbi of the best individual in the current population as a target value and compares the fitness fxi of the individuals in the population with it. If fxi>fbi, then the corresponding qubits in the individual are adjusted such that the probability amplitude αi,βi approaches xi. Otherwise, if fxi<fbi, the corresponding quantum of the individual in the population bit is adjusted, such that the probability amplitude αi,βi approaches bi.


## 3.3. Adaptive Hierarchical Loss Training Stage of the Embedding Model

The main purpose of the embedding model is to obtain the feature vector of the sample by mapping the original sample to the feature space. The meta-learner then learns the feature distribution of the sample based on the feature vector. Therefore, the performance of the embedding model is very important. The adaptive multilevel loss training method is adopted to train the embedding model to improve the feature representation performance of different layers of CNNs. The trained model is fused by a multiscale adaptive feature fusion algorithm. The framework of the adaptive multilayer loss training algorithm is shown in Figure2.Figure 2
Adaptive hierarchical multistage loss training framework for the embedding model.The formal expression of the adaptive multilayer loss training algorithm is as follows: letF be any lightweight network, which normally has L stages. The output feature map from any intermediate stage is represented by Fl∈ℝHl×Wl×Cl, where Hl, Wl, Cl represent the height, width, and number of channels of the feature map at l−th stage, and l=1,2,…,L. The aim is to impose classification losses on the feature map extracted at different intermediate stages. The feature map of each stage performs the average pooling operation to obtain the feature vector fl of the stage:(7)fl=PoolavgFl.The Dot product similarity of the class prototype vector at this stage is then calculated to obtain the logits of the class, and finally, the loss function at this stage is obtained as shown in Figure2:(8)lossl=ℓlSoftmaxfl=−logefl,y∑k=1Cefl,k.Loss functions of all stages are weighted and summed to obtain the loss function of the network as follows:(9)loss=∑la=15ωl·lossl.


## 3.4. Meta-Evaluation and Meta-Test

The trained embedding model is used to extract the features of the samples in the meta-evaluation set and meta-test set. To make full use of the features of different layers of the embedding model, the method of feature fusion is used to represent the features of the samples. Several fusion methods have been explored. In the current study, the method of feature vector Mosaic was used as follows:(10)Ffuse=concatγ1f1,γ2f2,…,γnfn.The fused input classifier is trained or tested, and the calculation formula is expressed as follows:(11)zpredict=classifierFfuse.The fitness function of the quantum genetic algorithm is expressed as follows:(12)fitnessmi=gx,gx=# of correctly predicted samplestotal number of samples×100%.All models are tested using the meta-test set when the optimization of the embedding model and feature fusion method is completed. The process of the meta-testing stage is the same as that of meta-evaluation. Formulas (10) and (11) are used for meta-testing. The algorithm flow is shown in Figure 3. The next section describes the details of meta-testing.Figure 3
Adaptive hierarchical feature fusion method for meta-evaluation and meta-test.


## 3.5. Meta-Classifier

Meta-learners are an important part of meta-learning. Meta-learners are referred to as meta-classifiers in the current study. Several models are available for classification tasks, such as nearest neighbor algorithm, linear regression, logistic regression, and support vector machine. The current study mainly uses the widely used logistic regression model for meta-classification. Logistic regression and linear regression are similar to the generalized linear model. Logistic regression assumes that the dependent variabley follows a Bernoulli distribution, whereas linear regression assumes that the dependent variable y follows a Gaussian distribution. Therefore, there are several similarities between logistic regression and linear regression. However, logistic regression introduces nonlinear factors through the Sigmoid function; thus, it can easily handle the 0/1 classification problem. If the Sigmoid mapping function is removed, the logistic regression algorithm is similar to linear regression. This implies that logistic regression is theoretically supported by linear regression.


## 4. Experiments

### 4.1. Datasets

MiniImageNet: this is a dataset extracted from ImageNet by Vinyals et al. of the Google DeepMind team. This dataset contains a total of 100 categories and a total of 60,000 color images. Each category has 600 samples, and the size of each image is 84×84 pixels. The DeepMind team used the MiniImageNet dataset for small sample learning for the first time, and since then MiniImageNet has become the benchmark dataset for meta-learning and small sample research.CIFAR-FS: the CIFAR-FS dataset refers to the CIFAR100 FEW-SHOTS dataset and is derived from the CIFAR100 dataset. It comprises 100 categories, 600 images per category, and a total of 60,000 images. For application purposes, it is divided into a training set (64 kinds), verification set (16 kinds), and test set (20 kinds) and the image size is unified as 32 × 32 pixels.FC100: a few-shot CIFAR100 dataset. This dataset is similar to the CIFAR-FS dataset and is derived from the CIFAR100 dataset. It comprises 100 categories with 600 images for each category and a total of 60,000 images. The difference between CIFAR-FS and FC100 is that FC100 is not classified by category, but by Superclass. FC100 comprises 20 superclasses in total, including 12 superclasses in the training set, 4 superclasses in the verification set, and 4 superclasses in the testing set.


### 4.2. Setup

The PyTorch deep learning architecture was used to verify the performance of the proposed algorithms on meta-learning tasks [27]. The experiment was performed on an UltraLAB graphics workstation equipped with 192 GB memory and 8 NVIDIA GTX-2080 graphics processors, and each graphics card had 8 GB memory. The workstation was operated on Windows server operating system.A ResNet-12 was chosen as the backbone. It included 4 residual blocks, and each block comprised 3 convolutional layers with 3 × 3 kernels. A 2 × 2 maximum pool layer was applied after the first 3 blocks. A global averaging pool layer is applied at the top of the fourth block to generate feature embedding. Drop-block was applied as a regularizer and the number of filters was varied from (64, 128, 256, 512) to (64, 160, 320, 640) as reported previously [28]. Consequently, the ResNet-12 presented in the current study is similar to that reported previously [28, 29]. The feature vectors of different layers were fused, and the feature fusion method and fusion weight were calculated based on formula (10) to obtain a good sample feature representation. Besides ResNet-12, SEResNet-12 is also used as the backbone network in the ablation experiments. The differences in residual block structure between the two backbone networks are shown in Figure 4 where (a) is the residual block of ResNet-12 and (b) is for the residual block of SEResNet-12. SEResNet-12 adds Squeeze and Excitation attention mechanism to ResNet-12, which is made up of two layers of full connection and one layer of pooling.Figure 4
Comparison of residual block structure between ResNet-12 and SEResNet-12. (a) Residual block of ResNet-12. (b) Residual block of SEResNet-12.
(a)(b)Optimization setup: SGD optimizer with a momentum of 0.9 and a weight decay of 5e − 4 was used in the current study. Each batch comprised 64 samples. The learning rate was initialized as 0.05 and decayed with a factor of 0.1 by three times for all datasets, except for miniImageNet where it only decayed twice as the third decay had no effect. A total of 100 epochs were trained for miniImageNet and 90 epochs for both CIFAR-FS and FC100.Random clipping, color dithering, and random horizontal flipping were used to extend the data for training the embedding model on the converted meta-training dataset as reported previously [28]. In the current study, N-way logistic regression basis classifier was trained in the meta-test phase. The implementation was applied in Scikit-Learn.


### 4.3. Results and Discussion

Experiments were conducted using the miniImageNet dataset. ResNet-12 was used as the backbone network. The random seed number of the experiment was fixed to ensure repeatability of the experiment. The weight of the loss function in formula (9) and the fusion weight of formula (10) are optimized using the accuracy of the meta-evaluation set as the fitness function of the quantum genetic algorithm (Table 2). In the 1-shot strategy, the method proposed in the current study achieved similar results to MetaOptNet [29]. The proposed method in the current study achieved better performance in the 5-shot strategy compared with MetaOptNet (Table 2). These findings indicate that the proposed method is effective and reliable.Table 2
Experimental results using the miniImageNet dataset and comparison of the method developed in the current study with the existing classical methods.
ModelBackboneMiniImageNet 5-way1-shot5-shotMAML[9]32-32-32-3248.70 ± 1.8463.11 ± 0.92Matching Networks [15]64-64-64-6443.56 ± 0.8455.31 ± 0.73IMP [30]64-64-64-6449.2 ± 0.764.7 ± 0.7Prototypical Networks [16]64-64-64-6449.42 ± 0.7868.20 ± 0.66TAML [31]64-64-64-6451.77 ± 1.8666.05 ± 0.85SAML [32]64-64-64-6452.22 ± n/a66.49 ± n/aGCR [33]64-64-64-6453.21 ± 0.8072.34 ± 0.64KTN [34]64-64-64-6454.61 ± 0.8071.21 ± 0.66PARN [35]64-64-64-6455.22 ± 0.8471.55 ± 0.66Dynamic Few-Shot [36]64-64-128-12856.20 ± 0.8673.00 ± 0.64Relation Networks [17]64-64-128-12850.44 ± 0.8265.32 ± 0.70AdaResNet [37]ResNet-1256.88 ± 0.6271.94 ± 0.57TADAM [38]ResNet-1258.50 ± 0.3076.70 ± 0.30Shot-Free [29]ResNet-1259.04 ± n/a77.64 ± n/aTEWAM [39]ResNet-1260.07 ± n/a75.90 ± n/aMTL [40]ResNet-1261.20 ± 1.8075.50 ± 0.80Variational FSL [41]ResNet-1261.23 ± 0.2677.69 ± 0.17MetaOptNet [28]ResNet-1262.64 ± 0.6178.63 ± 0.46Diversity w/Cooperation [42]ResNet-1859.48 ± 0.6575.62 ± 0.48LEO-trainval [19]WRN-28-1061.76 ± 0.0877.59 ± 0.12GEFS [10]ResNet-1262.02 ± 0.6379.64 ± 0.44OursResNet-1263.21 ± 0.7881.55 ± 0.29Average few-shot classification accuracies (%) with 95% confidence intervals on miniImageNet meta-test splits. a-b-c-d represents the size of the convolutional layer of the backbone network.The proposed method was further verified on FC100 and CIFAR-FS. The fixed random seed number strategy was used to conduct the experiment to ensure the reproducibility of the experiment. The experimental results showed that the proposed method in the current study performed better compared with existing methods, especially when using the FC100 dataset (Table3). The current method showed higher performance compared with existing methods. This indicated that the joint optimization of the embedded model and the classifier and use of feature fusion strategies improves the classification accuracy of the new task.Table 3
Experimental results using CIFAR-FS and FC100 datasets and comparison of the current method with existing classic methods.
ModelBackboneCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotMAML [9]32-32-32-3258.9 ± 1.971.5 ± 1.0——Prototypical Networks [16]64-64-64-6455.5 ± 0.772.0 ± 0.635.3 ± 0.648.6 ± 0.6Relation Networks [17]64-64-128-12855.0 ± 1.069.3 ± 0.8——TADAM [38]ResNet-12——40.1 ± 0.456.1 ± 0.4Shot-Free [29]ResNet-1269.2 ± n/a84.7 ± n/a——TEWAM [39]ResNet-1270.4 ± n/a81.3 ± n/a——Prototypical Networks [16]ResNet-1272.2 ± 0.783.5 ± 0.537.5 ± 0.652.5 ± 0.6MetaOptNet [28]ResNet-1272.6 ± 0.784.3 ± 0.541.1 ± 0.655.5 ± 0.6GEFS [10]ResNet-1271.5 ± 0.886.0 ± 0.542.6 ± 0.759.1 ± 0.6OursResNet-1272.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6Average few-shot classification accuracies (%) with 95% confidence intervals on CIFAR-FS and FC100. a-b-c-d represents the size of the convolutional layer of the backbone network.


### 4.4. Ablation Study

#### 4.4.1. Comparison of Different Classifiers

A variety of different classifiers were used to verify the two meta-learning datasets, FC100 and CIFAR-FC, to explore the effect of classifiers on the proposed method. Several kinds of algorithms are available for classification tasks. The current study mainly adopted three methods: the nearest neighbor, logistic regression, and support vector machine (Table4). Logistic regression classifier showed the best performance in classification. The accuracy of the logistic regression classifier using the meta-test set of FC100 data and the current method was best. This accuracy was higher compared with that of the nearest-neighbor classifier which had the worst performance. Therefore, the logistic regression classifier was selected for subsequent analysis.Table 4
Ablation study on three benchmarks with ResNet-12 as the backbone network.
ModelCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotNN69.58 ± 0.682.43 ± 0.643.3 ± 0.659.3 ± 0.6LR72.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6SVM71.94 ± 0.684.57 ± 0.641.8 ± 0.759.6 ± 0.6“NN” and “LR” represent the nearest neighbor classifier and logistic regression. “SVM” denotes support vector machine.The current study proposes an adaptive hierarchical loss embedding model training method. An adaptive hierarchical loss training method was adopted to verify the effectiveness of the method developed in the current study; the model was then divided into loss training method and traditional training method for comparison. The dividing loss method was used to fix the loss weight of different layers, which is set as 1 in the current study. The traditional neural network training method was used to add the loss function in the last layer of the network. Cross entropy was used as the calculation method for the loss function of all layers. Experiments were conducted on three different datasets, and the results showed that the embedding model of adaptive hierarchical loss proposed in the current study was optimal by comparing with other methods (Table5). These findings indicate that the hierarchical loss function can improve the shallow layer feature expression performance of the convolutional neural networ.Table 5
Comparisons of different training strategies for the embedding model on CIFAR-FS and FC100.
ModelCIFAR-FC 5-wayFC100 5-wayMiniImageNet 5-way1-shot5-shot1-shot5-shot1-shot5-shotOurs simple72.38 ± 0.686.67 ± 0.8544.1 ± 0.7360.8 ± 0.662.49 ± 0.5579.91 ± 0.40Ours72.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.663.21 ± 0.7881.55 ± 0.29In addition, the feature map of the 4th layer of ResNet-12 was visually visualized for adaptive hierarchical loss training. The visualization method used was CAM, which is used to visualize the feature map output by the convolutional layer in the neural network in the form of the heat map. The red area represents the response area of the neural network neurons in this layer. The findings showed that the embedding model of adaptive hierarchical loss function training proposed in the current study had a larger response area compared with that of the traditional method (Figure 5). This finding indicates the effectiveness of the current method.Figure 5
Activation map of selected results on the miniImageNet dataset using ResNet-12 as the base model. From left to right: Column 1 is original images; Column 2 is Grad-CAM visualizations for the model trained with standard cross entropy; Column 3 indicates Grad-CAM visualizations for the model trained with the currently proposed method.


#### 4.4.2. Comparison of Different Network Backbones

In the above experiment, ResNet-12 was used as the backbone network to make fair comparisons with other methods. To verify the influence of the backbone network on the proposed algorithm, SEResNet-12 was also applied in the experiments. The experimental results are shown in Table6. Notably, SEResNet-12 had a better representation of the sample characteristics.Table 6
Comparisons of different backbones on CIFAR-FS and FC100.
ModelCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotResNet-1272.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6SEResNet-1272.91 ± 0.8587.59 ± 0.745.47 ± 0.6363.65 ± 0.6


## 4.1. Datasets

MiniImageNet: this is a dataset extracted from ImageNet by Vinyals et al. of the Google DeepMind team. This dataset contains a total of 100 categories and a total of 60,000 color images. Each category has 600 samples, and the size of each image is 84×84 pixels. The DeepMind team used the MiniImageNet dataset for small sample learning for the first time, and since then MiniImageNet has become the benchmark dataset for meta-learning and small sample research.CIFAR-FS: the CIFAR-FS dataset refers to the CIFAR100 FEW-SHOTS dataset and is derived from the CIFAR100 dataset. It comprises 100 categories, 600 images per category, and a total of 60,000 images. For application purposes, it is divided into a training set (64 kinds), verification set (16 kinds), and test set (20 kinds) and the image size is unified as 32 × 32 pixels.FC100: a few-shot CIFAR100 dataset. This dataset is similar to the CIFAR-FS dataset and is derived from the CIFAR100 dataset. It comprises 100 categories with 600 images for each category and a total of 60,000 images. The difference between CIFAR-FS and FC100 is that FC100 is not classified by category, but by Superclass. FC100 comprises 20 superclasses in total, including 12 superclasses in the training set, 4 superclasses in the verification set, and 4 superclasses in the testing set.


## 4.2. Setup

The PyTorch deep learning architecture was used to verify the performance of the proposed algorithms on meta-learning tasks [27]. The experiment was performed on an UltraLAB graphics workstation equipped with 192 GB memory and 8 NVIDIA GTX-2080 graphics processors, and each graphics card had 8 GB memory. The workstation was operated on Windows server operating system.A ResNet-12 was chosen as the backbone. It included 4 residual blocks, and each block comprised 3 convolutional layers with 3 × 3 kernels. A 2 × 2 maximum pool layer was applied after the first 3 blocks. A global averaging pool layer is applied at the top of the fourth block to generate feature embedding. Drop-block was applied as a regularizer and the number of filters was varied from (64, 128, 256, 512) to (64, 160, 320, 640) as reported previously [28]. Consequently, the ResNet-12 presented in the current study is similar to that reported previously [28, 29]. The feature vectors of different layers were fused, and the feature fusion method and fusion weight were calculated based on formula (10) to obtain a good sample feature representation. Besides ResNet-12, SEResNet-12 is also used as the backbone network in the ablation experiments. The differences in residual block structure between the two backbone networks are shown in Figure 4 where (a) is the residual block of ResNet-12 and (b) is for the residual block of SEResNet-12. SEResNet-12 adds Squeeze and Excitation attention mechanism to ResNet-12, which is made up of two layers of full connection and one layer of pooling.Figure 4
Comparison of residual block structure between ResNet-12 and SEResNet-12. (a) Residual block of ResNet-12. (b) Residual block of SEResNet-12.
(a)(b)Optimization setup: SGD optimizer with a momentum of 0.9 and a weight decay of 5e − 4 was used in the current study. Each batch comprised 64 samples. The learning rate was initialized as 0.05 and decayed with a factor of 0.1 by three times for all datasets, except for miniImageNet where it only decayed twice as the third decay had no effect. A total of 100 epochs were trained for miniImageNet and 90 epochs for both CIFAR-FS and FC100.Random clipping, color dithering, and random horizontal flipping were used to extend the data for training the embedding model on the converted meta-training dataset as reported previously [28]. In the current study, N-way logistic regression basis classifier was trained in the meta-test phase. The implementation was applied in Scikit-Learn.


## 4.3. Results and Discussion

Experiments were conducted using the miniImageNet dataset. ResNet-12 was used as the backbone network. The random seed number of the experiment was fixed to ensure repeatability of the experiment. The weight of the loss function in formula (9) and the fusion weight of formula (10) are optimized using the accuracy of the meta-evaluation set as the fitness function of the quantum genetic algorithm (Table 2). In the 1-shot strategy, the method proposed in the current study achieved similar results to MetaOptNet [29]. The proposed method in the current study achieved better performance in the 5-shot strategy compared with MetaOptNet (Table 2). These findings indicate that the proposed method is effective and reliable.Table 2
Experimental results using the miniImageNet dataset and comparison of the method developed in the current study with the existing classical methods.
ModelBackboneMiniImageNet 5-way1-shot5-shotMAML[9]32-32-32-3248.70 ± 1.8463.11 ± 0.92Matching Networks [15]64-64-64-6443.56 ± 0.8455.31 ± 0.73IMP [30]64-64-64-6449.2 ± 0.764.7 ± 0.7Prototypical Networks [16]64-64-64-6449.42 ± 0.7868.20 ± 0.66TAML [31]64-64-64-6451.77 ± 1.8666.05 ± 0.85SAML [32]64-64-64-6452.22 ± n/a66.49 ± n/aGCR [33]64-64-64-6453.21 ± 0.8072.34 ± 0.64KTN [34]64-64-64-6454.61 ± 0.8071.21 ± 0.66PARN [35]64-64-64-6455.22 ± 0.8471.55 ± 0.66Dynamic Few-Shot [36]64-64-128-12856.20 ± 0.8673.00 ± 0.64Relation Networks [17]64-64-128-12850.44 ± 0.8265.32 ± 0.70AdaResNet [37]ResNet-1256.88 ± 0.6271.94 ± 0.57TADAM [38]ResNet-1258.50 ± 0.3076.70 ± 0.30Shot-Free [29]ResNet-1259.04 ± n/a77.64 ± n/aTEWAM [39]ResNet-1260.07 ± n/a75.90 ± n/aMTL [40]ResNet-1261.20 ± 1.8075.50 ± 0.80Variational FSL [41]ResNet-1261.23 ± 0.2677.69 ± 0.17MetaOptNet [28]ResNet-1262.64 ± 0.6178.63 ± 0.46Diversity w/Cooperation [42]ResNet-1859.48 ± 0.6575.62 ± 0.48LEO-trainval [19]WRN-28-1061.76 ± 0.0877.59 ± 0.12GEFS [10]ResNet-1262.02 ± 0.6379.64 ± 0.44OursResNet-1263.21 ± 0.7881.55 ± 0.29Average few-shot classification accuracies (%) with 95% confidence intervals on miniImageNet meta-test splits. a-b-c-d represents the size of the convolutional layer of the backbone network.The proposed method was further verified on FC100 and CIFAR-FS. The fixed random seed number strategy was used to conduct the experiment to ensure the reproducibility of the experiment. The experimental results showed that the proposed method in the current study performed better compared with existing methods, especially when using the FC100 dataset (Table3). The current method showed higher performance compared with existing methods. This indicated that the joint optimization of the embedded model and the classifier and use of feature fusion strategies improves the classification accuracy of the new task.Table 3
Experimental results using CIFAR-FS and FC100 datasets and comparison of the current method with existing classic methods.
ModelBackboneCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotMAML [9]32-32-32-3258.9 ± 1.971.5 ± 1.0——Prototypical Networks [16]64-64-64-6455.5 ± 0.772.0 ± 0.635.3 ± 0.648.6 ± 0.6Relation Networks [17]64-64-128-12855.0 ± 1.069.3 ± 0.8——TADAM [38]ResNet-12——40.1 ± 0.456.1 ± 0.4Shot-Free [29]ResNet-1269.2 ± n/a84.7 ± n/a——TEWAM [39]ResNet-1270.4 ± n/a81.3 ± n/a——Prototypical Networks [16]ResNet-1272.2 ± 0.783.5 ± 0.537.5 ± 0.652.5 ± 0.6MetaOptNet [28]ResNet-1272.6 ± 0.784.3 ± 0.541.1 ± 0.655.5 ± 0.6GEFS [10]ResNet-1271.5 ± 0.886.0 ± 0.542.6 ± 0.759.1 ± 0.6OursResNet-1272.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6Average few-shot classification accuracies (%) with 95% confidence intervals on CIFAR-FS and FC100. a-b-c-d represents the size of the convolutional layer of the backbone network.


## 4.4. Ablation Study

### 4.4.1. Comparison of Different Classifiers

A variety of different classifiers were used to verify the two meta-learning datasets, FC100 and CIFAR-FC, to explore the effect of classifiers on the proposed method. Several kinds of algorithms are available for classification tasks. The current study mainly adopted three methods: the nearest neighbor, logistic regression, and support vector machine (Table4). Logistic regression classifier showed the best performance in classification. The accuracy of the logistic regression classifier using the meta-test set of FC100 data and the current method was best. This accuracy was higher compared with that of the nearest-neighbor classifier which had the worst performance. Therefore, the logistic regression classifier was selected for subsequent analysis.Table 4
Ablation study on three benchmarks with ResNet-12 as the backbone network.
ModelCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotNN69.58 ± 0.682.43 ± 0.643.3 ± 0.659.3 ± 0.6LR72.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6SVM71.94 ± 0.684.57 ± 0.641.8 ± 0.759.6 ± 0.6“NN” and “LR” represent the nearest neighbor classifier and logistic regression. “SVM” denotes support vector machine.The current study proposes an adaptive hierarchical loss embedding model training method. An adaptive hierarchical loss training method was adopted to verify the effectiveness of the method developed in the current study; the model was then divided into loss training method and traditional training method for comparison. The dividing loss method was used to fix the loss weight of different layers, which is set as 1 in the current study. The traditional neural network training method was used to add the loss function in the last layer of the network. Cross entropy was used as the calculation method for the loss function of all layers. Experiments were conducted on three different datasets, and the results showed that the embedding model of adaptive hierarchical loss proposed in the current study was optimal by comparing with other methods (Table5). These findings indicate that the hierarchical loss function can improve the shallow layer feature expression performance of the convolutional neural networ.Table 5
Comparisons of different training strategies for the embedding model on CIFAR-FS and FC100.
ModelCIFAR-FC 5-wayFC100 5-wayMiniImageNet 5-way1-shot5-shot1-shot5-shot1-shot5-shotOurs simple72.38 ± 0.686.67 ± 0.8544.1 ± 0.7360.8 ± 0.662.49 ± 0.5579.91 ± 0.40Ours72.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.663.21 ± 0.7881.55 ± 0.29In addition, the feature map of the 4th layer of ResNet-12 was visually visualized for adaptive hierarchical loss training. The visualization method used was CAM, which is used to visualize the feature map output by the convolutional layer in the neural network in the form of the heat map. The red area represents the response area of the neural network neurons in this layer. The findings showed that the embedding model of adaptive hierarchical loss function training proposed in the current study had a larger response area compared with that of the traditional method (Figure 5). This finding indicates the effectiveness of the current method.Figure 5
Activation map of selected results on the miniImageNet dataset using ResNet-12 as the base model. From left to right: Column 1 is original images; Column 2 is Grad-CAM visualizations for the model trained with standard cross entropy; Column 3 indicates Grad-CAM visualizations for the model trained with the currently proposed method.


### 4.4.2. Comparison of Different Network Backbones

In the above experiment, ResNet-12 was used as the backbone network to make fair comparisons with other methods. To verify the influence of the backbone network on the proposed algorithm, SEResNet-12 was also applied in the experiments. The experimental results are shown in Table6. Notably, SEResNet-12 had a better representation of the sample characteristics.Table 6
Comparisons of different backbones on CIFAR-FS and FC100.
ModelCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotResNet-1272.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6SEResNet-1272.91 ± 0.8587.59 ± 0.745.47 ± 0.6363.65 ± 0.6


## 4.4.1. Comparison of Different Classifiers

A variety of different classifiers were used to verify the two meta-learning datasets, FC100 and CIFAR-FC, to explore the effect of classifiers on the proposed method. Several kinds of algorithms are available for classification tasks. The current study mainly adopted three methods: the nearest neighbor, logistic regression, and support vector machine (Table4). Logistic regression classifier showed the best performance in classification. The accuracy of the logistic regression classifier using the meta-test set of FC100 data and the current method was best. This accuracy was higher compared with that of the nearest-neighbor classifier which had the worst performance. Therefore, the logistic regression classifier was selected for subsequent analysis.Table 4
Ablation study on three benchmarks with ResNet-12 as the backbone network.
ModelCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotNN69.58 ± 0.682.43 ± 0.643.3 ± 0.659.3 ± 0.6LR72.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6SVM71.94 ± 0.684.57 ± 0.641.8 ± 0.759.6 ± 0.6“NN” and “LR” represent the nearest neighbor classifier and logistic regression. “SVM” denotes support vector machine.The current study proposes an adaptive hierarchical loss embedding model training method. An adaptive hierarchical loss training method was adopted to verify the effectiveness of the method developed in the current study; the model was then divided into loss training method and traditional training method for comparison. The dividing loss method was used to fix the loss weight of different layers, which is set as 1 in the current study. The traditional neural network training method was used to add the loss function in the last layer of the network. Cross entropy was used as the calculation method for the loss function of all layers. Experiments were conducted on three different datasets, and the results showed that the embedding model of adaptive hierarchical loss proposed in the current study was optimal by comparing with other methods (Table5). These findings indicate that the hierarchical loss function can improve the shallow layer feature expression performance of the convolutional neural networ.Table 5
Comparisons of different training strategies for the embedding model on CIFAR-FS and FC100.
ModelCIFAR-FC 5-wayFC100 5-wayMiniImageNet 5-way1-shot5-shot1-shot5-shot1-shot5-shotOurs simple72.38 ± 0.686.67 ± 0.8544.1 ± 0.7360.8 ± 0.662.49 ± 0.5579.91 ± 0.40Ours72.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.663.21 ± 0.7881.55 ± 0.29In addition, the feature map of the 4th layer of ResNet-12 was visually visualized for adaptive hierarchical loss training. The visualization method used was CAM, which is used to visualize the feature map output by the convolutional layer in the neural network in the form of the heat map. The red area represents the response area of the neural network neurons in this layer. The findings showed that the embedding model of adaptive hierarchical loss function training proposed in the current study had a larger response area compared with that of the traditional method (Figure 5). This finding indicates the effectiveness of the current method.Figure 5
Activation map of selected results on the miniImageNet dataset using ResNet-12 as the base model. From left to right: Column 1 is original images; Column 2 is Grad-CAM visualizations for the model trained with standard cross entropy; Column 3 indicates Grad-CAM visualizations for the model trained with the currently proposed method.


## 4.4.2. Comparison of Different Network Backbones

In the above experiment, ResNet-12 was used as the backbone network to make fair comparisons with other methods. To verify the influence of the backbone network on the proposed algorithm, SEResNet-12 was also applied in the experiments. The experimental results are shown in Table6. Notably, SEResNet-12 had a better representation of the sample characteristics.Table 6
Comparisons of different backbones on CIFAR-FS and FC100.
ModelCIFAR-FC 5-wayFC100 5-way1-shot5-shot1-shot5-shotResNet-1272.62 ± 0.587.34 ± 0.645.18 ± 0.763.47 ± 0.6SEResNet-1272.91 ± 0.8587.59 ± 0.745.47 ± 0.6363.65 ± 0.6


## 5. Conclusion

This study presents a joint optimization method in which the embedding model and classifier are used for meta-learning. Specifically, we applied a quantum genetic algorithm to optimize the hierarchical multi-loss weight of the embedding model and the weight of feature fusion. The classification accuracy of the meta-learning evaluation set was used as the fitness function, which effectively combines the embedded model and classifier. The performance of the proposed method was tested on three well-known public meta-learning datasets. It was found that the performance of the proposed method was superior to that of most existing baseline meta-learning models. In the future, we plan to carry out studies on the joint optimization method of semisupervised or unsupervised embedding model and classifier. In addition, we shall explore more efficient evolution strategies to improve the efficiency of the joint optimization method.



---

*Source: 1538914-2021-10-01.xml*