# Variational Autoencoder for Zero-Shot Recognition of Bai Characters

Authors: Weiwei Lin; Tai Ma; Zeqing Zhang; Xiaofan Li; Xingsi Xue
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

When talking about Bai nationality, people are impressed by its long history and the language it has created. However, since fewer people of the young generation learn the traditional language, the glorious Bai culture becomes less known, making understanding Bai characters difficult. Based on the highly precise character recognition model for Bai characters, the paper is aimed at helping people read books written in Bai characters so as to popularize the culture. To begin with, a data set is built with the support of Bai culture fans and experts. However, the data set is not large enough as knowledge in this respect is limited. This makes the deep learning model less accurate since it lacks sufficient data. The popular zero-shot learning (ZSL) is adopted to overcome the insufficiency of data sets. We use Chinese characters as the seen class, Bai characters as the unseen class, and the number of strokes as the attribute to construct the ZSL format data set. However, the existing ZSL methods ignore the character structure information, so a generation method based on variational autoencoder (VAE) is put forward, which can automatically capture the character structure information. Experimental results show that the method facilitates the recognition of Bai characters and makes it more precise.


---

## Body

## 1. Introduction

With a population of over 1 million, Bai nationality boasts a splendid culture. Its history can date back to ancient times. The Bai people mainly settle in Dali Bai Autonomous Prefecture, Yunnan province. Besides, they also reside in Bijie Prefecture of Guizhou, Liangshan Prefecture of Sichuan, etc. Bai people communicate in their own language which serves as an emotional bond and a major carrier of Bai culture. This is the most basic national characteristic of Bai nationality. The academic circles both at home and abroad have been focusing on the linguistic structure of Bai characters whose vocabulary, pronunciation, and grammar belong to the Tibeto-Burmese. Since the literature concerning the Bai nationality is limited, the historical and cultural significance of the study is obvious.The objective of the study is to revive Bai culture and decipher the Bai characters, helping people to understand historical literature of Bai people or inscriptions of Bai characters. Therefore, a model recognizing Bai characters should be established so that when an unknown Bai character appears, it can be recognized and explained by the model.As the neural network becomes popular, great achievements have been made on visual tasks [1, 2]. Therefore, deep learning models are proposed [1, 3–5] to recognize Bai characters one by one. Unfortunately, the data hungry nature of convolutional networks leads to a significant decline in their performance when there is less training data. At the same time, the collection of Bai character data set needs a lot of expert knowledge and is expensive, so the established data set is not as large as the traditional data set for classification [6, 7]. To sum up, a single data set of Bai characters alone makes the training of a deep learning model difficult and the ideal results are hard to get.To this end, we consider using zero-shot learning (ZSL) [8, 9] as collecting Bai character data sets is hard to obtain [10]. ZSL can transfer knowledge from visible categories to invisible ones according to their attributes to prevent samples of the invisible category from decreasing or even losing. If we treat Bai characters as unseen classes, we need to collect data sets similar to Bai characters as seen classes in order to better transfer knowledge [10]. Fortunately, it can be seen that Chinese and Bai characters are highly similar as they come from the Sino-Tibetan [11, 12] language family, as shown in Figure 1. In addition, Chinese character data sets are very easy to collect and cheap and do not need expert knowledge. Therefore, we collected a huge Chinese character data set as seen class in ZSL. Last but not least, ZSL needs to depict the attributes of both visible and invisible categories, so as to better transfer knowledge through attributes [10]. It is found that characters of Chinese and Bai are made up of 32 basic strokes, as shown in Figure 2. The case is similar in English which is made up of 26 letters. As a result, it is reasonable to regard the number of different strokes of each word as an attribute.Figure 1
Contrast between Bai characters and Chinese characters.Figure 2
32 characters of Chinese and Bai, basic strokes.After building the data set in ZSL format, we transplanted some classical methods in ZSL, which include projection methods: DAP [9] and IAP [9], and generation method- [13] based GAN [14]. These methods establish the relationship between attributes and features, so as to generalize Chinese characters to Bai characters and finally improve the accuracy [10]. However, our attributes only contain the stroke information of characters, so these models ignore the important information of character structure in the process of training.As a possible solution, a generation method built on VAE [15] is introduced in this paper. VAE includes an encoder and a decoder. The former can obtain the semantic information of the characters, and the latter can reconstruct the characters according to the information and attributes. The reconstruction of characters obviously needs to consider the structure of characters. Therefore, thanks to the reconstruction loss, VAE can automatically capture the structure information of the characters, making the character features synthesized by the decoder more realistic. Experiments show that the generation method based on VAE has achieved amazing accuracy improvement, which is far better than the classical ZSL methods.The significance of the study is fourfold: (1) a data set in ZSL format with Bai language and Chinese is built. (2) A generation method built on VAE is proposed, which can automatically capture the structure information of characters. (3) According to experimental results, the method enhances the correction rate of recognizing Bai characters.


## 2. Related Work

### 2.1. Text Recognition of Bai Characters

Zhang et al. [10] as a pioneer resorting to deep learning to recognize the Bai text collected a data set with 400 words of Bai handwritten fonts. In their work, Zhang et al. [10] considered that the Chinese language is similar to Bai language to facilitate the model to recognize Bai characters based on learning transfer [10, 16] and achieved remarkable results. However, their study is limited to a limited data set, and the data set contains only 400 Bai characters. Once there are Bai characters that are not in the data set, their trained model is bound to obtain a wrong classification result. In contrast, we consider using GZSL in such a way that the model gains the ability to recognize Bai characters outside the training set, which greatly improves the application value of the model.


### 2.2. Zero-Shot Learning

Currently, generative approaches dominate in GZSL, which exploit existing adversarial generative networks (GAN) [14, 17, 18] or variational autoencoders (VAE) [15, 19, 20] so that visual characteristics from class-level semantic attributes and random noise can be synthesized. f-CLSWGAN [13], cycle-UWGAN [21], and LisGAN [22] introduce the Wasserstein generative adversarial network (WGAN) [23] coupled with a pretrained classifier so that visual characters for invisible characteristics can be synthesized, thus allowing the GZSL work to deteriorate into a fully supervised issue for categorization. RFF [24] combines the traditional projection method and GAN to initially map visual characteristics to a new feature space without redundancy and judge the veracity of the mapped characteristics. Different from methods based on GAN, some works [25–28] formulate GAN in the variational autoencoder (VAE) model to match the latent distribution that is based on categories and feature representations that are discriminating. They further improve the quality of synthetic features by combining the advantages of GAN and VAE and even extend the model to the transductive setting through an unconditional discriminator. However, all these methods only optimize their own models on seen classes and consider how a good generator can be trained to synthesize visual characteristics according to attributes, without directly simulating zero-shot learning settings so that knowledge can be transferred from visible categories to invisible ones during training.To better mimic the zero-shot learning settings, previous studies [29–31] introduce meta-learning to make the model more suitable for transferring knowledge from seen classes to unseen classes. ZSML [30] combines metalearning with GAN and utilizes a single gradient update to obtain a generic initialization suitable for internal learning. E-PGN [29] first introduces an episode-based paradigm for training, which trains the model by simulating multiple zero-shot classification tasks on the seen classes. After training a collection of episodes, the model is expected to be an expert in predicting unseen classes such that it can generalize well to the real unseen classes. TGMZ [31] proposed a task-wise model, which extracts both class and visual feature information for reconstruction, to carry out task-wise distribution alignment.


## 2.1. Text Recognition of Bai Characters

Zhang et al. [10] as a pioneer resorting to deep learning to recognize the Bai text collected a data set with 400 words of Bai handwritten fonts. In their work, Zhang et al. [10] considered that the Chinese language is similar to Bai language to facilitate the model to recognize Bai characters based on learning transfer [10, 16] and achieved remarkable results. However, their study is limited to a limited data set, and the data set contains only 400 Bai characters. Once there are Bai characters that are not in the data set, their trained model is bound to obtain a wrong classification result. In contrast, we consider using GZSL in such a way that the model gains the ability to recognize Bai characters outside the training set, which greatly improves the application value of the model.


## 2.2. Zero-Shot Learning

Currently, generative approaches dominate in GZSL, which exploit existing adversarial generative networks (GAN) [14, 17, 18] or variational autoencoders (VAE) [15, 19, 20] so that visual characteristics from class-level semantic attributes and random noise can be synthesized. f-CLSWGAN [13], cycle-UWGAN [21], and LisGAN [22] introduce the Wasserstein generative adversarial network (WGAN) [23] coupled with a pretrained classifier so that visual characters for invisible characteristics can be synthesized, thus allowing the GZSL work to deteriorate into a fully supervised issue for categorization. RFF [24] combines the traditional projection method and GAN to initially map visual characteristics to a new feature space without redundancy and judge the veracity of the mapped characteristics. Different from methods based on GAN, some works [25–28] formulate GAN in the variational autoencoder (VAE) model to match the latent distribution that is based on categories and feature representations that are discriminating. They further improve the quality of synthetic features by combining the advantages of GAN and VAE and even extend the model to the transductive setting through an unconditional discriminator. However, all these methods only optimize their own models on seen classes and consider how a good generator can be trained to synthesize visual characteristics according to attributes, without directly simulating zero-shot learning settings so that knowledge can be transferred from visible categories to invisible ones during training.To better mimic the zero-shot learning settings, previous studies [29–31] introduce meta-learning to make the model more suitable for transferring knowledge from seen classes to unseen classes. ZSML [30] combines metalearning with GAN and utilizes a single gradient update to obtain a generic initialization suitable for internal learning. E-PGN [29] first introduces an episode-based paradigm for training, which trains the model by simulating multiple zero-shot classification tasks on the seen classes. After training a collection of episodes, the model is expected to be an expert in predicting unseen classes such that it can generalize well to the real unseen classes. TGMZ [31] proposed a task-wise model, which extracts both class and visual feature information for reconstruction, to carry out task-wise distribution alignment.


## 3. Method

### 3.1. Problem Definition

The zero-shot learning (ZSL) [9] methods are introduced to recognize Bai characters. In the recognition task, the data set is made up of two disjoint sets: the Chinese character set and the Bai character one, which are regarded as the set of visible and invisible categories in the ZSL working, respectively. Suppose that Ds=Xs,Ys,As denotes the Chinese character set, and Du=Xu,Yu,Au denotes the Bai character set. Xs and Xu are the instance set, which are for training and testing. Ys and Yu belong to the label sets. As and Au are the corresponding attribute sets constructed from strokes. Note that Ys∩Yu=∅, which means that the categories involved in training and testing are disjoint. In the study, the Chinese character set and the attributes of Bai character set are adopted to train the classifier which can recognize the Bai characters. This is of great significance for the recognition of Bai characters, which are difficult to collect. The current ZSL methods will be introduced below, and they will be applied to Bai character recognition.


### 3.2. Intermediate Attribute Classifier for Learning

Direct Attribute Prediction (DAP) [9] and Indirect Attribute Prediction (IAP) [9] are the earliest ZSL methods which make use of the attributes to infer the label of the instances. We introduced DAP and IAP to the Bai character recognition task. For simplicity, suppose that the attribute representations ac=a1c,⋯,acM of classes for training (Chinese character classes) and c are fixed-length vectors.The probabilistic classifier is learned for the attributeam based on DAP. As shown in Figure 3(a), the trained classifier is estimated to be pamx. Then, a model showing the entire image-attribute layer as pax=∏m=1Mpamx is established. During the test, every invisible category (Bai character classes) induces its attribute vector ab deterministically, i.e., pab=Ia=ab, where IP is the indicator function: IP=1 if P is true; then, it is 0 otherwise. Based on the principles of Bayes, the attribute-class layer is as follows: pba=pb/pabIa=ab. Combining image-attribute and attribute-class layers, when X is given, the posterior can be obtained by
(1)pbx=∑m=1Mpbampamx=pbpac∏m=1Mpambx.Figure 3
The model structure diagram for DAP and IAP.x is the feature vector extracted by the backbone.a1,a2,⋯,aM is the attribute vector. c1,c2,⋯,cK and b1,b2,⋯,bL represent the label of Chinese and Bai character, respectively. The thin arrow shows trainable weights, and thick arrow lines show predefined weights.When there is not enough specific knowledge, the factorpb below can be ignored. Regarding the factor pa, pa=∏m=1Mpam is assumed. The empirical means pam=1/K∑k=1Kamc is adopted for classes for training as prior attribute. Based on the principle of decision f:X⟶B optimizing output class based on test classes b1,⋯,bL to a test sample x, it can be predicted that
(2)fx=argmaxb∏m=1Mpambxpamb.According to IAP, the probabilistic attributes of inputxare indirectly estimated by first predicting the probabilities of every class; then, the attribute matrix is multiplied as shown in Figure 3(b). The attribute probabilities are computed by
(3)pamx=∑k=1Kpamckpckx.wherepamck is the attribute of the class that is defined beforehand and pckx is the (Chinese character) posterior of the class. After pamx has been computed, the class label of testing is predicted based on Equation (2) (Bai character).


### 3.3. Feature Generative Framework

#### 3.3.1. Generative Adversarial Network

The generative adversarial network (GAN) [13, 14] proposes the feature generative framework for zero-shot learning. We introduce it into zero-shot Bai character recognition. Feature generative framework is divided into two stages: feature generation and classification. As shown in Figure 4(a), for the GAN-based feature generative framework, in feature generation, a conditional generator G is trained to synthesize the samples x~=Ga,z when a Gaussian noise z∈N0,1 and the attribute a are taken into consideration. The discriminator D is crossiteratively trained with the generator and learns to distinguish a real pair x,a from a synthetic one x~,a. G attempts to synthesize a more realistic feature x~ to confuse the discriminator during training. Besides, the generator hopes to find the right synthetic feature x~ to its corresponding attribute a. The generative model adopts the structure of WGAN [32] and introduces the gradient penalty term [23]; the adversarial training loss of G and D is as follows:
(4)Lwgan=EDx,a−EDx~,a−λE∇x∧,a2−12.Figure 4
The feature generative framework based on GAN and VAE.x is the visual feature extracted by the backbone. x~ is the synthetic feature. a is the feature of character, and z is a Gaussion noise. δ and μ are the mean and variance required for reparameter, respectively.The generator is trained on the Chinese character set, and the trained generator is used to synthesize Bai character samples. In the stage of classification, synthetic samples are used to train the classifier to recognize Bai character through the crossentropy loss:(5)Lce=ElogPbx;θ,where θ∈Rdx×L is the weight matrix of an interconnected layer. dx is the dimension of x, and L is the class number of Bai character, and Pbx;θ=expθbTx/∑iLexpθiTx. The prediction function is
(6)fx=argmaxbPbx;θ.


#### 3.3.2. Variational Autoencoder

In addition to the GAN-based generative framework, the variational autoencoder (VAE) [15] sets the base for the proposed generative framework, which is more suitable for character recognition task. According to Figure 4(b), without discriminator, the framework based on VAE is made up of the encoder En and decoder De (generator). During the training, the former uses reparameter trick and is trained through the Kullback-Leibler Divergence (KLD) loss which is demonstrated as follows:
(7)Lkld=E12∑i=1Nd1+logδi2−μi2−δi2,where μ and δ are the outputs of the encoder, representing the mean and variance, respectively. Nd refers to the dimension of μ and δ. Through reparameter, hidden variables encoded by the encoder can be represented as
(8)z~=z×δ+μ.At the same time, the decoderDe is trained together with the encoder En through the reconstruction (RE) loss:
(9)Lre=Ex−x~22,where x~=Dea,z^ and the RE loss can make the synthetic feature x~ have better structural information. Thereby, compared with GAN, VAE is more suitable for character recognition which pays more attention to structure. In feature generation, the total loss of VAE is as follows:
(10)Lvae=Lre+λLkld,where λ refers to a hyperparameter. After training, the decoder is used to synthesize visual features for Bai character conditioned the attributes and Gaussian noises. To synthesize features of Bai character which are leverage to train the classifier through Equation (5) and Equation (6) is used for prediction.


## 3.1. Problem Definition

The zero-shot learning (ZSL) [9] methods are introduced to recognize Bai characters. In the recognition task, the data set is made up of two disjoint sets: the Chinese character set and the Bai character one, which are regarded as the set of visible and invisible categories in the ZSL working, respectively. Suppose that Ds=Xs,Ys,As denotes the Chinese character set, and Du=Xu,Yu,Au denotes the Bai character set. Xs and Xu are the instance set, which are for training and testing. Ys and Yu belong to the label sets. As and Au are the corresponding attribute sets constructed from strokes. Note that Ys∩Yu=∅, which means that the categories involved in training and testing are disjoint. In the study, the Chinese character set and the attributes of Bai character set are adopted to train the classifier which can recognize the Bai characters. This is of great significance for the recognition of Bai characters, which are difficult to collect. The current ZSL methods will be introduced below, and they will be applied to Bai character recognition.


## 3.2. Intermediate Attribute Classifier for Learning

Direct Attribute Prediction (DAP) [9] and Indirect Attribute Prediction (IAP) [9] are the earliest ZSL methods which make use of the attributes to infer the label of the instances. We introduced DAP and IAP to the Bai character recognition task. For simplicity, suppose that the attribute representations ac=a1c,⋯,acM of classes for training (Chinese character classes) and c are fixed-length vectors.The probabilistic classifier is learned for the attributeam based on DAP. As shown in Figure 3(a), the trained classifier is estimated to be pamx. Then, a model showing the entire image-attribute layer as pax=∏m=1Mpamx is established. During the test, every invisible category (Bai character classes) induces its attribute vector ab deterministically, i.e., pab=Ia=ab, where IP is the indicator function: IP=1 if P is true; then, it is 0 otherwise. Based on the principles of Bayes, the attribute-class layer is as follows: pba=pb/pabIa=ab. Combining image-attribute and attribute-class layers, when X is given, the posterior can be obtained by
(1)pbx=∑m=1Mpbampamx=pbpac∏m=1Mpambx.Figure 3
The model structure diagram for DAP and IAP.x is the feature vector extracted by the backbone.a1,a2,⋯,aM is the attribute vector. c1,c2,⋯,cK and b1,b2,⋯,bL represent the label of Chinese and Bai character, respectively. The thin arrow shows trainable weights, and thick arrow lines show predefined weights.When there is not enough specific knowledge, the factorpb below can be ignored. Regarding the factor pa, pa=∏m=1Mpam is assumed. The empirical means pam=1/K∑k=1Kamc is adopted for classes for training as prior attribute. Based on the principle of decision f:X⟶B optimizing output class based on test classes b1,⋯,bL to a test sample x, it can be predicted that
(2)fx=argmaxb∏m=1Mpambxpamb.According to IAP, the probabilistic attributes of inputxare indirectly estimated by first predicting the probabilities of every class; then, the attribute matrix is multiplied as shown in Figure 3(b). The attribute probabilities are computed by
(3)pamx=∑k=1Kpamckpckx.wherepamck is the attribute of the class that is defined beforehand and pckx is the (Chinese character) posterior of the class. After pamx has been computed, the class label of testing is predicted based on Equation (2) (Bai character).


## 3.3. Feature Generative Framework

### 3.3.1. Generative Adversarial Network

The generative adversarial network (GAN) [13, 14] proposes the feature generative framework for zero-shot learning. We introduce it into zero-shot Bai character recognition. Feature generative framework is divided into two stages: feature generation and classification. As shown in Figure 4(a), for the GAN-based feature generative framework, in feature generation, a conditional generator G is trained to synthesize the samples x~=Ga,z when a Gaussian noise z∈N0,1 and the attribute a are taken into consideration. The discriminator D is crossiteratively trained with the generator and learns to distinguish a real pair x,a from a synthetic one x~,a. G attempts to synthesize a more realistic feature x~ to confuse the discriminator during training. Besides, the generator hopes to find the right synthetic feature x~ to its corresponding attribute a. The generative model adopts the structure of WGAN [32] and introduces the gradient penalty term [23]; the adversarial training loss of G and D is as follows:
(4)Lwgan=EDx,a−EDx~,a−λE∇x∧,a2−12.Figure 4
The feature generative framework based on GAN and VAE.x is the visual feature extracted by the backbone. x~ is the synthetic feature. a is the feature of character, and z is a Gaussion noise. δ and μ are the mean and variance required for reparameter, respectively.The generator is trained on the Chinese character set, and the trained generator is used to synthesize Bai character samples. In the stage of classification, synthetic samples are used to train the classifier to recognize Bai character through the crossentropy loss:(5)Lce=ElogPbx;θ,where θ∈Rdx×L is the weight matrix of an interconnected layer. dx is the dimension of x, and L is the class number of Bai character, and Pbx;θ=expθbTx/∑iLexpθiTx. The prediction function is
(6)fx=argmaxbPbx;θ.


### 3.3.2. Variational Autoencoder

In addition to the GAN-based generative framework, the variational autoencoder (VAE) [15] sets the base for the proposed generative framework, which is more suitable for character recognition task. According to Figure 4(b), without discriminator, the framework based on VAE is made up of the encoder En and decoder De (generator). During the training, the former uses reparameter trick and is trained through the Kullback-Leibler Divergence (KLD) loss which is demonstrated as follows:
(7)Lkld=E12∑i=1Nd1+logδi2−μi2−δi2,where μ and δ are the outputs of the encoder, representing the mean and variance, respectively. Nd refers to the dimension of μ and δ. Through reparameter, hidden variables encoded by the encoder can be represented as
(8)z~=z×δ+μ.At the same time, the decoderDe is trained together with the encoder En through the reconstruction (RE) loss:
(9)Lre=Ex−x~22,where x~=Dea,z^ and the RE loss can make the synthetic feature x~ have better structural information. Thereby, compared with GAN, VAE is more suitable for character recognition which pays more attention to structure. In feature generation, the total loss of VAE is as follows:
(10)Lvae=Lre+λLkld,where λ refers to a hyperparameter. After training, the decoder is used to synthesize visual features for Bai character conditioned the attributes and Gaussian noises. To synthesize features of Bai character which are leverage to train the classifier through Equation (5) and Equation (6) is used for prediction.


## 3.3.1. Generative Adversarial Network

The generative adversarial network (GAN) [13, 14] proposes the feature generative framework for zero-shot learning. We introduce it into zero-shot Bai character recognition. Feature generative framework is divided into two stages: feature generation and classification. As shown in Figure 4(a), for the GAN-based feature generative framework, in feature generation, a conditional generator G is trained to synthesize the samples x~=Ga,z when a Gaussian noise z∈N0,1 and the attribute a are taken into consideration. The discriminator D is crossiteratively trained with the generator and learns to distinguish a real pair x,a from a synthetic one x~,a. G attempts to synthesize a more realistic feature x~ to confuse the discriminator during training. Besides, the generator hopes to find the right synthetic feature x~ to its corresponding attribute a. The generative model adopts the structure of WGAN [32] and introduces the gradient penalty term [23]; the adversarial training loss of G and D is as follows:
(4)Lwgan=EDx,a−EDx~,a−λE∇x∧,a2−12.Figure 4
The feature generative framework based on GAN and VAE.x is the visual feature extracted by the backbone. x~ is the synthetic feature. a is the feature of character, and z is a Gaussion noise. δ and μ are the mean and variance required for reparameter, respectively.The generator is trained on the Chinese character set, and the trained generator is used to synthesize Bai character samples. In the stage of classification, synthetic samples are used to train the classifier to recognize Bai character through the crossentropy loss:(5)Lce=ElogPbx;θ,where θ∈Rdx×L is the weight matrix of an interconnected layer. dx is the dimension of x, and L is the class number of Bai character, and Pbx;θ=expθbTx/∑iLexpθiTx. The prediction function is
(6)fx=argmaxbPbx;θ.


## 3.3.2. Variational Autoencoder

In addition to the GAN-based generative framework, the variational autoencoder (VAE) [15] sets the base for the proposed generative framework, which is more suitable for character recognition task. According to Figure 4(b), without discriminator, the framework based on VAE is made up of the encoder En and decoder De (generator). During the training, the former uses reparameter trick and is trained through the Kullback-Leibler Divergence (KLD) loss which is demonstrated as follows:
(7)Lkld=E12∑i=1Nd1+logδi2−μi2−δi2,where μ and δ are the outputs of the encoder, representing the mean and variance, respectively. Nd refers to the dimension of μ and δ. Through reparameter, hidden variables encoded by the encoder can be represented as
(8)z~=z×δ+μ.At the same time, the decoderDe is trained together with the encoder En through the reconstruction (RE) loss:
(9)Lre=Ex−x~22,where x~=Dea,z^ and the RE loss can make the synthetic feature x~ have better structural information. Thereby, compared with GAN, VAE is more suitable for character recognition which pays more attention to structure. In feature generation, the total loss of VAE is as follows:
(10)Lvae=Lre+λLkld,where λ refers to a hyperparameter. After training, the decoder is used to synthesize visual features for Bai character conditioned the attributes and Gaussian noises. To synthesize features of Bai character which are leverage to train the classifier through Equation (5) and Equation (6) is used for prediction.


## 4. Experiment

### 4.1. Experimental Setup

#### 4.1.1. Data Sets

A big data set of Bai characters is built (see Figure5). The data set includes 400 Bai characters. Since some characters in Bai and Chinese overlap, those Bai characters differing from Chinese ones are included in the data set. Each word has 50 samples, and they are written by Bai people and Bai culture fans. Besides, a data set of sufficient Chinese characters is also chosen, including 509 Chinese characters. There are about 1,000 samples for each character. To build a data set in ZSL format, the two data sets are labelled with class-level attributes (a number of 32 basic strokes). The Chinese one is used to train, and the Bai character one is used to test.Figure 5
Some examples of Bai character data set.


#### 4.1.2. Evaluation Protocol

The proposed method is assessed based on the average top-1 accuracy (ACC) for each class.


#### 4.1.3. Classification Model

Three backbones, AlexNet [4], VGG19 [5], and ResNet101 [1], are adopted. Multiple models are compared; it can be analyzed that the method is suitable for all models and can be generalized.


### 4.2. Accuracy Analysis

The accuracy comparison of training strategies is shown in Table1. The accuracy of DAP, IAP, GAN, and our proposed VAE on zero-shot recognition of Bai characters is assessed. Meanwhile, we evaluated the performance of the above four methods with AlexNet, VGG, and ResNet101 as the backbone, respectively. The overall experimental results show that the zero-shot learning methods will be effectively transferred for recognizing Bai characters.Table 1
Methods compared based on accuracy. Bai characters are not used as a direct way to train the model.
MethodAlexNetVGG19ResNet101DAP [33]46.2849.9253.84IAP [33]42.2347.2648.59GAN [13]67.5470.7876.46VAE (ours)70.4774.8178.14DAP and IAP are projection methods. Because only Chinese character data sets are used in training, the model has never seen Bai characters. Therefore, although the model has been improved due to knowledge transfer, it is still not ideal. However, it also shows that using ZSL for knowledge transfer can transfer the knowledge of large-scale Chinese character data sets to Bai characters, so that the model can obtain a high recognition rate for Bai characters even without Bai character data as training.GAN is a generation method. Because the generator synthesizes a large number of Bai character features, it greatly alleviates the problem of missing Bai character data and finally greatly improves the accuracy. This shows that although Chinese characters are different from Bai characters, through the attribute of stroke, the generator can well restore the features of Bai characters when only Chinese characters are trained.VAE is our proposed method and is also a generation method. It not only has the ability to generate features but also has the ability to reconstruct the original picture. This ability plays a very important role in the transfer of text knowledge, which makes our method have the highest accuracy.


### 4.3. Visualization

To further illustrate the effect of these methods, the visual features obtained by different methods are visualized. After that, t-SNE [34] algorithm is adopted to finish the task (see Figure 6). DAP and IAP denote the visualization results of Bai character visual features extracted from backbone trained by DAP and IAP, respectively. GAN and VAE denote the visualization results of Bai character visual features synthesized by GAN and VAE. With the support of huge Chinese character data set, the Bai character features extracted by the network trained by DAP and IAP have been highly distinguishable, which fully shows the effectiveness of knowledge transfer. The Bai character features synthesized by the generator trained by the GAN further increased the discriminability. Finally, the Bai language features synthesized by the network trained by VAE have been well distinguishable, which fully shows the superiority of VAE for zero-shot recognition of Bai characters.Figure 6
Visualization results of Bai character features by different methods.
(a)
DAP(b)
IAP(c)
GAN(d)
VAE


### 4.4. Hyperparameter Analysis

In Figure7, we report the results of two generative methods with different numbers of synthetic features per Bai character class. It can be observed that generative methods require certain numbers of synthetic samples to achieve the desired results. But that does not mean that more synthetic samples are better. Synthetic samples create too much noise, which limits the performance of classification. Thereby, in order to obtain the ideal results of Bai character recognition, we need to limit the number of synthetic samples per Bai character class to a certain interval.Figure 7
The performance of GAN-based and VAE-based generative methods with numbers of synthetic characteristic for each Bai character class. ACC is the average top-1 accuracy for each class. # is the number of synthetic features per Bai character class.
(a)
GAN(b)
VAE


## 4.1. Experimental Setup

### 4.1.1. Data Sets

A big data set of Bai characters is built (see Figure5). The data set includes 400 Bai characters. Since some characters in Bai and Chinese overlap, those Bai characters differing from Chinese ones are included in the data set. Each word has 50 samples, and they are written by Bai people and Bai culture fans. Besides, a data set of sufficient Chinese characters is also chosen, including 509 Chinese characters. There are about 1,000 samples for each character. To build a data set in ZSL format, the two data sets are labelled with class-level attributes (a number of 32 basic strokes). The Chinese one is used to train, and the Bai character one is used to test.Figure 5
Some examples of Bai character data set.


### 4.1.2. Evaluation Protocol

The proposed method is assessed based on the average top-1 accuracy (ACC) for each class.


### 4.1.3. Classification Model

Three backbones, AlexNet [4], VGG19 [5], and ResNet101 [1], are adopted. Multiple models are compared; it can be analyzed that the method is suitable for all models and can be generalized.


## 4.1.1. Data Sets

A big data set of Bai characters is built (see Figure5). The data set includes 400 Bai characters. Since some characters in Bai and Chinese overlap, those Bai characters differing from Chinese ones are included in the data set. Each word has 50 samples, and they are written by Bai people and Bai culture fans. Besides, a data set of sufficient Chinese characters is also chosen, including 509 Chinese characters. There are about 1,000 samples for each character. To build a data set in ZSL format, the two data sets are labelled with class-level attributes (a number of 32 basic strokes). The Chinese one is used to train, and the Bai character one is used to test.Figure 5
Some examples of Bai character data set.


## 4.1.2. Evaluation Protocol

The proposed method is assessed based on the average top-1 accuracy (ACC) for each class.


## 4.1.3. Classification Model

Three backbones, AlexNet [4], VGG19 [5], and ResNet101 [1], are adopted. Multiple models are compared; it can be analyzed that the method is suitable for all models and can be generalized.


## 4.2. Accuracy Analysis

The accuracy comparison of training strategies is shown in Table1. The accuracy of DAP, IAP, GAN, and our proposed VAE on zero-shot recognition of Bai characters is assessed. Meanwhile, we evaluated the performance of the above four methods with AlexNet, VGG, and ResNet101 as the backbone, respectively. The overall experimental results show that the zero-shot learning methods will be effectively transferred for recognizing Bai characters.Table 1
Methods compared based on accuracy. Bai characters are not used as a direct way to train the model.
MethodAlexNetVGG19ResNet101DAP [33]46.2849.9253.84IAP [33]42.2347.2648.59GAN [13]67.5470.7876.46VAE (ours)70.4774.8178.14DAP and IAP are projection methods. Because only Chinese character data sets are used in training, the model has never seen Bai characters. Therefore, although the model has been improved due to knowledge transfer, it is still not ideal. However, it also shows that using ZSL for knowledge transfer can transfer the knowledge of large-scale Chinese character data sets to Bai characters, so that the model can obtain a high recognition rate for Bai characters even without Bai character data as training.GAN is a generation method. Because the generator synthesizes a large number of Bai character features, it greatly alleviates the problem of missing Bai character data and finally greatly improves the accuracy. This shows that although Chinese characters are different from Bai characters, through the attribute of stroke, the generator can well restore the features of Bai characters when only Chinese characters are trained.VAE is our proposed method and is also a generation method. It not only has the ability to generate features but also has the ability to reconstruct the original picture. This ability plays a very important role in the transfer of text knowledge, which makes our method have the highest accuracy.


## 4.3. Visualization

To further illustrate the effect of these methods, the visual features obtained by different methods are visualized. After that, t-SNE [34] algorithm is adopted to finish the task (see Figure 6). DAP and IAP denote the visualization results of Bai character visual features extracted from backbone trained by DAP and IAP, respectively. GAN and VAE denote the visualization results of Bai character visual features synthesized by GAN and VAE. With the support of huge Chinese character data set, the Bai character features extracted by the network trained by DAP and IAP have been highly distinguishable, which fully shows the effectiveness of knowledge transfer. The Bai character features synthesized by the generator trained by the GAN further increased the discriminability. Finally, the Bai language features synthesized by the network trained by VAE have been well distinguishable, which fully shows the superiority of VAE for zero-shot recognition of Bai characters.Figure 6
Visualization results of Bai character features by different methods.
(a)
DAP(b)
IAP(c)
GAN(d)
VAE


## 4.4. Hyperparameter Analysis

In Figure7, we report the results of two generative methods with different numbers of synthetic features per Bai character class. It can be observed that generative methods require certain numbers of synthetic samples to achieve the desired results. But that does not mean that more synthetic samples are better. Synthetic samples create too much noise, which limits the performance of classification. Thereby, in order to obtain the ideal results of Bai character recognition, we need to limit the number of synthetic samples per Bai character class to a certain interval.Figure 7
The performance of GAN-based and VAE-based generative methods with numbers of synthetic characteristic for each Bai character class. ACC is the average top-1 accuracy for each class. # is the number of synthetic features per Bai character class.
(a)
GAN(b)
VAE


## 5. Conclusions

Bai nationality can date back to ancient China as a nation. It boasts of its own language and wonderful culture. However, fewer young people can read in Bai language with the passing of time and its once splendid culture is on the verge of extinction. To help Bai culture lovers and experts to read Bai literature without difficulty, the study focuses on the training of a high-precision Bai model to recognize Bai characters. Firstly, a data set of Bai characters is established. However, its size is not big enough since expert knowledge is limited. Therefore, deep learning models requiring a large amount of data cannot produce perfect results based on this data set. As a solution, the zero-shot learning (ZSL) is suggested to overcome the lack of data sets. We use Chinese characters as the seen class, Bai characters as the unseen class, and the number of strokes as the attribute to construct the ZSL format data set. However, the existing ZSL methods ignore the characters structure information, so a VAE-based generation method is proposed, which can automatically capture the character structure information. According to experimental results, the proposed methods can enhance the model to recognize Bai characters more accurately.



---

*Source: 2717322-2022-07-04.xml*