Imbalanced data is a significan t challenge in classification with machine learni ng algorithms . This is particularly important with log message data as negative logs are sparse so this data is typically imbal anced . In this pa per, a model to generate text log messages is proposed which em ploys a SeqGAN network. A n Autoencoder is used for feature extraction and anomaly detection is done using a GRU network. The proposed model is evaluated with three imbalanced log data sets, namely BGL , OpenS tack, and Thunderbird . Results are presented whic h show that appropriate oversampling and data balancing improves anomal y detection accuracy . KEYWORDS Deep Learning, Oversampling , Log messages, Anomaly detection, Classification 1. INTRODUCTION Log message s are commonly used in softw are systems such as cloud servers to record events . These unstructured text messages are typica lly imbalanced because most logs indicate that the system is working properly and only a small portion indicate s a significant problem. D ata distribution with a very unequal number of samples for each label is called imbalanced. The problem of imbalanced data has been considered in tasks such as text mining [1], face recognition [2], and software defect prediction [3]. The imbalanced nature of log messages is one of the challenges for classification using deep learning. In binary classification, there are only two labels, and with imbalanced data , most are normal ( major) logs. The small num ber of abnormal ( minor) logs makes classification difficult and can lead to poor accuracy with deep learning algorithms. This is because the normal logs dominate the abnormal logs. Oversampling and undersampling are two methods that can be used to address this problem. In undersampling, the major label samples are reduced so the number of minor and major label samples is similar . A serious drawback of undersampling is loss of information [4]. In oversampling, the number of minor label samples is increased so it is similar to the number of ma jor label samples. Random oversampling and the Synthetic Minority Oversampling T echnique (SMOTE ) [5] are two well -known oversampling methods . Random oversampling increases the minor samples by randomly repeating some minor samples . SMOTE creat es synthetic minority samples to increase their num ber. The performance of SMOTE is generally better than with random oversampling [5]. However, r andom oversampling and SM OTE provide poor perf ormance when used to oversample log messages [6]. A generative adversarial network (GAN) [ 7] is effective in International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 54 genera ting images that are similar to actual images [8]. GANs can produce more abstract and varied data than other algorithms [ 9]. In this paper , a model is proposed to deal with imbalanced log data by oversampling text log messages using a Sequence Generative Adversarial Network (SeqGAN) [ 10]. The resulting data is then used for anomaly detection and classification with Autoencoder [ 11] and Gated Recurrent Unit (GRU) [1 2] networks. An Autoencoder is a feed -forward network that is effective in extracting importa nt information from data. Autoencoders have been applied to many tasks such as probabilistic and generative modeling [1 3] and representation learning [1 4]. A GRU is a Recurrent Neural Network (RNN) that has been used for tasks such as sentiment analysis [1 5] and speech recognition [1 6]. The proposed model is evaluated using three labeled log message data sets, namely BlueGene/L (BGL) , OpenS tack, and Thunderbird . Results are presented which show that the proposed model with oversampling provides better resul ts than without oversampling. The main contributions of this paper are as follows. 1. A new model is proposed for log message oversampling for anomaly detection and classification using SeqGAN , Autoencoder , and GRU network s. 2. M odel results with and without log message oversampling are compared. 3. Three well-known dat a sets are used to evaluate the model . The rest of this paper is organized as follows. In Section 2 the Autoencoder, GRU , and Se qGAN architectures are given and the proposed model is described. Section 3 presents the e xperimental results and d iscussion, while Section 4 provides some conclusion s. 2. SYSTEM MODEL In this section, the Autoencoder, GRU , and SeqGAN architectures are given along with the proposed network model. 2.1. Autoencoder A rchitecture An Autoencoder is a feed -forward multi -layer neural network with the same number of input and output neu rons. It is used to learn an efficient representation of data whi le minimizing the error. An Autoenco der with more than one hidden layer i s called a deep Autoencoder [1 7]. A reduced dimension data representation is produced using encoder and decoder hidden layers in the Autoencoder architecture. Backpropagation is used for training to reduce the los s based on a loss function. Figure 1 shows the Autoencoder architecture with an input layer, a hidden layer , and an output layer. 2.2. GRU Architecture A Gated Recurrent Unit (GRU) is a type of RNN network which is a modified version of an Long Short-Term Memory (LSTM ) network [1 8]. It has a reset gat e and an update gate. The reset gate determines how much information in a block should be forgotten and is given by 1 = ( ),t r t r t rr W x U h b (1) International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 55 where rb is a bias vector ,  is the sigmoid activation function , and rW and rU are weight matrices. The update gate decides how much information should be updated and can be expressed as 1 = ( ),t z t z t zz W x U h b (2) where zW and zU are weight matrices and zb is a bias vector. The block output at time t is (3) where hb is a bias vector , and hW and hU are weight matrice s. A GRU block is shown in Figure 2. 2.3. SEQGAN Architecture A SeqGAN consists of a Generator ( G ) and Discriminator ( D ). The Discriminator is trained to discrimi nate between real data (sentences) and generated sentences. The Generator is trained using the Discriminator using a reward function with policy gradient [1 9]. The reward for a sentence is used to regulate the Generator via reinforcement learning. Generato r G is trained with a real data set to produce a sentence 1: 1= { , , , , }, ,T t T tY y y y y  Y where Y is the vocabulary of candidate words. This should produce a sentence that is close to real data. This is a r einforcement learning problem which considers G to produce an action a (next word ty ) given the state s (previously generated words 1: 1tY ). SeqGAN trains the Discriminator D as well as G . D is trained to discriminate between real data and data gener ated by G. Words are generated by G each time step but D only computes rewards for full sentences. Hence, the rewards for intermediate states are estimated using Monte Carlo (MC) search and are given by 1: 1 = ( = , = ) =G D t tQ s Y a y  1: 1: 1: =1 1:1( ), ( ; ) : < , ( ) : = ,N nn T T t n tD Y Y MC Y N if t TN D Y if t T    (4) where G DQ  is the action -value function which is the expected reward from the Discriminator, T is the sentence length and N is the number of sentences in the MC s earch, 1:n TY is the nth sentence in the MC search, and 1:()n T DY is the probability of the nth sentence denote d real by the Discriminator. A fter the reward is computed, G is updated via the pol icy gradient which is the gradient of the objective function and is given by International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 56 1: 1 1: 1 =11( ) ( | ) ( , )TG t t D t t tytJ G y Y Q Y yT       Y ( | ) 1: 1 1: 11: 1=11= [ log ( | ) ( , )],TG y G y Y t t D t tt t ttG y Y Q Y yT    E (5) ( ),J       (6) where  is the learning rate. SeqGAN updates the D iscriminator and Generator until the stopping criteria are satisfied. An LSTM, GRU , or other RNN network for the Generator and a Convolutional Neural Network (CNN) for the Discriminator have been shown to provide good results for classification tasks [ 10]. The SeqG AN architecture is shown in Figure 3. The orange circles denote words in real sentences and the blue circles denote words in generated sentences. First , the Generator is pretrained with real data using the cross -entropy loss function w hich minimi zes the negative log - likelihood. Then it is used to generate data and the Discriminator is pretrained with both generat ed and real data. T he MC search parameters (  ) are set to be the same as the Generator parameters (  ). As shown on the right, an MC search is used to compute the reward for an intermediate state. This search generates N complete sentences from the current state. A reward is computed for each sentence and averaged as the intermediate reward except in the last time step where the reward is obtained from the Discriminator. The input of each time step is the output of the previous step and the next word is obtained via a multinomial distribution over the softmax of the GR U output. Then the Generator is trained using the policy gradient. Finally, the updated Generator is used to generate data and the Discriminator is trained with bo th the generated and real data. The SeqGAN algorithm is given in Algorithm 1. Algorithm 1 The SeqGAN algorithm 1: Pretrain Generator Gθ with real data 2:  ←  3: Pretrain Discriminator D using the data generated by the Generator and real data 4: repeat 5: for g-steps do 6: Generate a sentence Y1:T = {y1, … , yT } ∼ G 7: for t in 1: T do 8: Compute the rewards using (4) 9: Update the Generator parameters with the policy gradient using (6) 10: for d-steps do 11: Use the Generator to generate data 12: Train the Discriminator with real and generated data 13:  ←  14: until stopping criteria are satisfied 2.4. Proposed Model The proposed model has three steps. The first is generating log messages using SeqGAN for oversampling. The log messages are divided into two data sets, positive labeled data (normal) and International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 57 negative labeled data (abnormal). Additional negative labeled data is generated using the negative labeled data set. The initia l negativ e data set is split into sets (the OpenS tack data set is split into two sets while the BGL and Thunderbird data sets are split into seven sets), and each set is input to the SeqGAN separately. This ensures better convergence and provides different negativ e log messages. Further, this improves the speed which is important with data gene ration . A CNN is used in the SeqGAN for the discriminator and a GRU as the generator. The GRU has one hidden layer of size 30 with the ADAM optimizer and t he batch si ze is 128. The generated negative log messages are concatenated with the original negative data and similar messages are removed. The resulting data set is balanced with similar numbers of positive and negative data. The second step is the Autoencoder whi ch has two networks (positive and negative) with three hidden layers (two encoder layers and one decoder layer). The encoder layers have 400 (with L1 regularizer) and 200 neurons and the decoder layer has 200 neuron s. The output layer has 40 neurons which is the same size as the input layer. The positive labeled data is fed into the positive Autoencoder. Note that this network is trained with just positive label ed data. The maximum number of epochs is 100 and the batch size is 128. Dropout with probability 0.8 an d early stopping are used to prevent overfitting. Categorical cross -entropy loss with the ADAM optimizer is used for training. The network output is labeled as positive. The negative labeled data which has been oversampled is fed into the n egative Au toencoder and the net work output is labeled as negative. The two sets of labeled data are then concatenated, duplicates are removed and Gaussi an noise with zero mean and variance 0.1 is added to avoid overfitting [ 20]. The final step is the GRU network fo r anomaly detection and classification. First, the concatenated data set is divided into training and testing sets with 5% for training and 95% for testing, and these sets are shuffled. The training set is then divided into two sets with 5% for training an d 95% for validation. The data is fed into the GRU with hidden layer size 100 and is classified using so ftmax activation. 10 -fold cross -validation is used in training with a maximum of 100 epochs and a batch size of 128. Dropout with probability 0.8 and ea rly stoppi ng are used to prevent overfitting. Categorical cross -entropy loss with the ADAM optimizer is used for training. The proposed model is shown in Figure 4. 3. RESULTS In this section, the proposed model is evaluated with and without SeqGAN oversampl ing using the BGL , OpenS tack, and Thunderbird data sets. Four criteria are used to evaluate the performance, namely accuracy, precision, recall , and F -measure. Accuracy is the fraction of the input data that is correctly predicted and is given by =,pn p n p nTTAT T F F    (7) where pT is the number of positive instances predicted by the model to be positive, nT is the number of negative instances predicted to be negative, pF is the n umber of negative instances predicted to be positive, and nF is the number of positive instances predicted to be negative. Precision is given by =,p ppTPTF (8) and recall is International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 58 =.p pnTRTF (9) The F-measure is the harmonic mean of recall and precision which can be expressed as 2=.PRFPR  (10) All experiments were conducted on the Compute Canada Cedar cluster with 24 CPU cores, 125 GB memory , and four P100 GPUs with Python in Keras and Tensorflow. The hyperparameters of the proposed model were not tuned so the default values were used for all data sets. For each data set, the average training accuracy, average validation accuracy, average training loss, testing accuracy, precis ion, recall , and F -measure were obtained. Tables 1 and 2 give the results for the BGL , Open Stack, and Thunderbird data sets without and with SeqGAN oversampling, respectively. 3.1. BGL The BlueGene/L (BGL) data set consists of 4,399,503 positive log messages and 348,460 negative log messages (without oversampling). From this data set, 11,869 logs are used for training, 225,529 for validation , and the remaining 4,510,565 for testing with approximately 95% positive and 5% negative messages in each group. Without oversampling, the average training accuracy is 97.8% and average validation accuracy is 98.6% with standard deviations of 0.02 and 0.01, respectively, in 10 -fold cross -validation. The average training loss is 0.07 with a standard deviation of 0.01. The te sting accuracy is 99.3% with a precision of 98.9% for negative logs and 99.3% for positive logs, and recall of 91.6% and 99.9% for negative and positive logs, respectively. The F -measure is 95.1% and 99.6% for negative and positive logs, respectively. Oversampling of the negative log messages with SeqGAN increased the number in the data set to 4,137,516 so the numbers of positive and negative log messages are similar. From this data set, 21,342 logs are used for training, 405,508 for validation , and the re maining 8,110,169 for testing with similar numbers of positive and negative log messages in each group. The average training accuracy is 98.3% and average validation accuracy is 99.3% with a standard dev iation of 0.01 in 10-fold cross -validation. The avera ge training loss is 0.05 with a standard deviation of 0.01. The testing accuracy is 99.6% with a precision of 99.8% for negative logs and 99.4% for positive logs, and recall of 99.3% and 99.8% for negative and positive logs, respectively. The F -measure is 99.6% for both negative and positive logs. The precision, recall , and F -measure with oversampling for negative logs are better than the values of 99%, 75% , and 85%, respectively, with SVM supervised learning and 83%, 99% , and 91%, respectively, with unsupe rvised learning [21]. 3.2. Open Stack The OpenS tack data set without oversampling consists of 137,074 positive log messages and 18,434 negative log messages. From this data set, 6,608 logs are used for training, 1,167 for validation , and the remaining 147,733 for testing with approximately 95% positive and 5% negative messages in each group. Without oversampling, the average training accuracy is 98.4% and average validation accuracy is 97.2% with a standard dev iation of 0.01 in 10 -fold cross - validation. The av erage training loss is 0.05 with a standard deviation of 0.01. The testing International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 59 accuracy is 98.3% with a precision of 97.9% for negative logs and 98.3% for positive logs, and recall of 87.1% and 99.8% for negative and positive logs, respectively. The F -measure is 92.2% and 99.0% for negative and positive logs, respectively. Oversampling of the negative log messages with SeqGAN increa sed the number in the data set to 154,202. From this data set, 12,378 logs are used for training, 2,185 for validation , and the remaining 276,713 for testing with similar numbers of positive and negative log messages in each group. With oversampling, the average training accuracy is 98.0% and average validation accuracy is 98.7% with a standard dev iation of 0.01 in 10 -fold cross -validation. The average training loss is 0.06 with a standard deviation of 0.01. The testing accuracy is 98.9% with a precision of 99.6% for negative logs and 98.2% for positive logs, and recall of 98.4% and 99.5% for the negative and positive logs, respective ly. The F -measure is 99.0% and 98.8% for negative and positive logs, respectively. The precision, recall , and F -measure with oversampling for negative logs are better than the 94%, 99% , and 97% obtained with the Deeplog network [22]. 3.3. Thunderbird From the Thunderbird data set , 3,000,000 positive log messages and 300,000 negative log messages (without oversampling) were selected. From this data set, 8,250 logs are used for training, 156,750 for validation , and the remaining 3,135,000 for testing with approx imately 95% positive and 5% negative messages in each group. Without oversampling, the average training accuracy is 98.7% and average validation accuracy is 98.8% with a standard deviation of 0.01 in 10-fold cross -validation. The average training loss is 0 .04 with a standard deviation of 0.02. The testing accuracy is 98.5% with a precision of 93.2% for negative logs and 99.1% for positive logs, and recall of 90.6% and 99.3% for negative and positive logs, respectively. The F -measure is 91.9% and 99.2% for n egative and positive logs, respectively. Oversampling of the negative log messages with SeqGAN increased the number in the data set to 3,000,000 so the numbers of positive and negative log messages are the same . From this data set, 15,000 logs are used fo r training, 285,000 for validation , and the remaining 5,700,000 for testing with similar numbers of positive and negative log messages in each group. The average training accuracy is 9 9.1% and average validation accuracy is 99. 3% with standard deviation s of 0.01 and 0.02, respectively, in 10 -fold cross -validation. The average training loss is 0.0 4 with a standard deviation of 0.01. The testing accuracy is 99.6% with a precision of 99. 5% for negative logs and 99.7% for positive logs, and recall of 99. 7% and 99.5% for negative and positive logs, respectively. The F -measure is 99.6% for both negative and positive logs. The precision, recall , and F -measure with oversampling for negative logs are better than the 96%, 96% , and 96%, respectively, with the Improved K-Nearest Neighbor algorithm [23]. 3.4. Discussion The results obtained show that the oversampling with SeqGA N provided good results for the BGL, OpenS tack, and Thunderbird data sets. It is evident that oversampling significantly improved the model accuracy f or negative log messages. For the BGL data set, the precision, recall and F -measure after oversampling increased from 98.9% to 99.8%, 91.6% to 99.3% and 95.1% to 99.6% which are 0.9%, 7.7% and 4.5% higher, respectively. For the Open Stack data set, the prec ision, recall and F -measure after oversampling increased from 97.9% to 99.6%, 87.1% to 98.4% and 92.2% to 99.0% which are 1.7%, 11.3% and 6.8% higher, respectively. For the Thunderbird data set, the precision, recall , and F -measure after oversampling incre ased from 93.2% to 99.5%, 90.6% to 99.7% , and 91.9% to 99.6% which are 6.3%, 9.1% , and 7.7% higher, respectively. This indicates that data balancing should be considered with deep learning algorithms t o improve the performance. This is particularly importa nt when the number of minor International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 60 label samples is very small . The proposed model was evaluated with three data sets for anomaly detection and classification with only a small portion (less than 1%) used for training. This is very significant as deep learning algorithms typically require significant amounts of data for training [24]. Note that good results were obtained even though the hyperparameters were not tuned. The most important step in t he proposed model is oversampling with a SeqG AN network . These netwo rks have been shown to provide good results in generating text such as poems [2 5]. The concept of generating data is similar to that for oversamplin g. It was interest ing that duplication in the oversampled log data was not high (less than 5%). As a consequ ence, af ter removing duplicates a significant amount of data was available for anomaly detection and classif ication. Feature extraction using an Autoencoder is also important. The output from the Autoenc oder is suitable for use with an RNN -based algorithm such as a GRU for anomaly detection and classification. T his was confirmed by the results obtained which show that the p roposed model provide s excellent performance even when the data is imbalanced. 4. CONCLUSION In this paper, a model was proposed to addre ss the pr oblem of imbalanced log message data . In the first step, the negative logs were oversampled with a SeqGAN network so that the numbers of positive and negative logs are similar. The resulting labeled logs were then fed into an Autoencoder to extrac t features and information from the text data. Finally, a GRU network was used for anomaly detection and classification. The proposed model was evaluated using three log message data sets, namely BGL, OpenS tack, and Thunderbird. Results were presented whic h show that oversampling can improve detection and classification accuracy . In the future, other text-based GAN networks such as TextGAN and MaliGAN can be used for oversampling. Table 1. Results without oversampling for the BGL, OpenStack, and Thunderbir d data sets (numbers in parenthesis are standard deviation). Positive labels are denoted by 1 and negative labels by 0. International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 61 Table 2. Results with oversampling using SeqGAN for the BGL, OpenStack, and Thunderbird data sets (numbers in parenthesis are st andard deviation). Positive labels are denoted by 1 and negative labels by 0. Figure 1. Autoencoder architecture with an input layer, a hidden layer, and an output layer. International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 62 Figure 2. A GRU block with reset gate, update gate, and tangent hyper bolic and sigmoid activation functions. Figure 3. The SeqGAN architecture. The Discriminator on the left is trained with real data and data is generated using the Generator. The Generator on the right is trained using the policy gradient. The reward i s computed by the Discriminator and this is used to determine the intermediate action values for the MC search. International Journal of Artificial Intelligence and Applications (IJAIA), Vol.11, No.4, July 2020 63 Figure 4. The proposed model architecture with SeqGAN for oversampling log messages, two Autoencoder networks , and a GRU network fo r anomaly detection and classification.