Reinforcement learning cropping method based on comprehensive feature and aesthetics assessment

IET Image ProcessingVolume 16, Issue 5 p. 1415-1423 ORIGINAL RESEARCH PAPEROpen Access Reinforcement learning cropping method based on comprehensive feature and aesthetics assessment Yaqing Zhang, Corresponding Author Yaqing Zhang [email protected] orcid.org/0000-0002-0079-6171 School of Digital Media and Design Arts, Beijing University of Posts and Telecommunications, Beijing, China Correspondence School of Digital Media and Design Arts, Beijing University of Posts and Telecommunications, Beijing, Beijing, China. Email: [email protected]Search for more papers by this authorXueming Li, Xueming Li Beijing Key Laboratory of Network System and Network Culture, Beijing University of Posts and Telecommunications, Beijing, ChinaSearch for more papers by this authorXuewei Li, Xuewei Li Department of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, ChinaSearch for more papers by this author Yaqing Zhang, Corresponding Author Yaqing Zhang [email protected] orcid.org/0000-0002-0079-6171 School of Digital Media and Design Arts, Beijing University of Posts and Telecommunications, Beijing, China Correspondence School of Digital Media and Design Arts, Beijing University of Posts and Telecommunications, Beijing, Beijing, China. Email: [email protected]Search for more papers by this authorXueming Li, Xueming Li Beijing Key Laboratory of Network System and Network Culture, Beijing University of Posts and Telecommunications, Beijing, ChinaSearch for more papers by this authorXuewei Li, Xuewei Li Department of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, ChinaSearch for more papers by this author First published: 17 January 2022 https://doi.org/10.1049/ipr2.12420Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Automatic image cropping can change the composition to improve the aesthetic quality of the images. Most of the existing automatic image cropping methods based on specific features need to generate a large number of candidate cropping windows. It is very time-consuming and can only produce a limited aspect ratio results. In the face of these situations, a reinforcement learning cropping method based on comprehensive feature and aesthetics assessment is proposed. It does not need to produce a large number of candidate windows. Its gradually cropping mode is more in line with the process of image cropping by human. What is more, the proposed method takes the image aesthetic assessment into consideration. Experimental results show that the proposed method improves the cropping efficiency and achieves excellent cropping effect on the open Flickr Cropping Dataset and CUHK Image Cropping Dataset. The proposed method can overcome the shortages of existing methods. 1 INTRODUCTION Image cropping is the most common and important task in image editing. It aims at changing the composition of images and highlighting the main content of the image to improve the aesthetic quality of the images. Image cropping has many applications, such as printing, film processing, thumbnail generation, autonomous photo taking and so on. In the field of automatic image cropping, many novel methods have been proposed. Many of them need to produce numerous cropping candidates [1-7]. One of the most widely used candidate generation methods is sliding window mechanism [4]. As shown in Figure 1, these methods follow these steps: (1) Use sliding window method to extract numerous cropping candidates on the input image, (2) extract specific features, such as aesthetic features or salient features, of each cropping candidate, (3) according to the evaluation of extracted features, select the best cropping results. These methods have also achieved good results. But due to the limitations of candidate generation method, it is impossible to produce cropping results of arbitrary aspect ratio ,thus we may not get the optimal result. And they not only has high computational complexity and takes a lot of time, but also has very high requirement on the hardware. At the same time, these cropping methods also do not conform to the process of human cropping. FIGURE 1Open in figure viewerPowerPoint The process of most image cropping methods Based on the above investigation, and inspired by human cropping, this paper puts forward a reinforcement learning cropping method based on comprehensive feature and aesthetics assessment, called DRLCFA (Deep Reinforcement Learning with Comprehensive Feature and Aesthetics). There is some comparative results of DRLCFA in Figure 2. The first row are original images, the second row are results of DRLCFA, and the last row is the ground truth. The numbers are the aesthetics scores calculated by VFN (view finding network) [2]. The general process is shown in Figure 3. The method simulates the human image cropping process. First of all, it extracts the basic feature and saliency feature of the whole image. Then based on the guidance of aesthetic evaluation, it adjusts the cropping range step by step [8-10], until the results are no longer to improve, when the optimal results are obtained. Through the above process, the proposed method will not produce a large number of candidate cropping windows, and can finish cropping within three steps on average, which is very efficient. The main contributions of this work are listed as follows. 1. The state representation for reinforcement learning is well designed. It not only contains the basic feature of the image, but also includes the saliency feature which is in favor of remaining the main object in the image. And in consideration of human decision-making, the cropping history is added too. 2. The method considers both accuracy and aesthetics. We choose cropping accuracy measurement and image aesthetics assessment score both as guidance of the cropping. 3. The proposed method is efficient, it completes the cropping process in three steps, takes less than 0.3 s on a single NVIDIA GeForce GTX 1080Ti GPU with Intel(R) Core(TM) i7-6800k CPU. FIGURE 2Open in figure viewerPowerPoint Some comparative results of DRLCFA FIGURE 3Open in figure viewerPowerPoint The process of the proposed method DRLCFA 2 RELATED WORK Automatic image cropping is to change the composition of the image so as to improve the aesthetic quality of the image. According to the features based, automatic image cropping methods can be divided into three categories: attention-based method, aesthetics-based method, and the method based on comprehensive factors. Attention-based methods take the salience of the image as the guide for image cropping, which mainly considers the retention of the main contents attracting people's attention in the image. According to the cropping process, attention-based methods can be divided into three types. One is to generate numerous cropping candidates first and calculate their salience score to choose the final result. One is to directly obtain the most salient area, then generate several candidates around it and choose the final result by self-defined value function. Another is to calculate the salience region first and then learning the mapping from it to the final cropping part. Ardizzone et al. [11] first generate five salience maps, reserve the region where its salience score is larger than the given threshold, finally gets the intersecting part of the remained region which is the final cropping result. Fang et al. [1] extract the salience map and the gradient map of the image reflecting the salience region and colour degree of the image, and then 1000 or so cropping candidates of several common aspect ratios are generated intensively, and the best result is obtained by evaluating them according to the principle of preserving the salience region and the simplicity of the boundary. Lin [12] fuses the results of image salience assessment, object detection and face–body detection to obtain the feature map, and the maximum closed rectangle in the feature map is used as the final cropping window. Zhang et al. [13] use image segmentation method to obtain the main objects in the image, then maps the original image to the cropped image by simulating the path of human attention shift. Lu et al. [14] first obtain the visual salient point of the image through salience detection, and an initial visual salience rectangle is generated. Then, the initial visual salience boundary box is mapped to the optimal cropping area through regression network. Chen et al. [15] study the complexity of several different optimal window search algorithms under the attention-based framework, and the optimal cropping window was calculated under the principle of removing the most irrelevant content and obtaining the maximum salience. Lin and Bajić [16] apply bottom-up visual saliency and top-down semantic analysis to the image cropping task. It has four segments, that is, saliency estimation, multi resolution object detection, multi resolution face and upper body detection. Aamir [17] constructs a framework for cropping which using graph-based segmentation and grey level adjustment to get more accurate and clear saliency maps. Li and Zhang [18] implement reinforcement learning with two agent. One agent deals with the raw image data and the other deals with the attention information. Aesthetic-based methods use almost the same ways as aesthetic assessment. They are based on intuitive aesthetic features such as image grey scale histogram, contrast, color complexity, or deep features learned from deep learning on aesthetic data sets. Yang et al. [19] use a change-based method. It generates 500 cropping candidates arbitrarily under the limitation of maximising the score of the reserved area and the abandoned area. The result with highest score is chosen as the final result. Guo et al. [20] extract the aesthetic feature of the image first and feed it into a cascaded weak regression network to get the cropping coordinates. Chen et al. [2] use the sliding window mechanism to produce a large number of cropping candidates, then evaluates them via view finding network VFN to select the one with the highest score as the final result. Chen et al. [4] propose an aesthetic-based sorting method, using RankSVM [21], RankNet [22], RankBoost [23] and LambdaMART [24] to carry out experiments. Kao et al. [5] use the aesthetic map and gradient energy map to learn a cropping model to evaluate the composition quality of cropping candidates produced by sliding window mechanism. Wei et al. [25] present a large-scale Comparative Photo Composition data set, which contains over one million comparative view pairs, and proposes a novel knowledge transfer framework. It at first trains a view evaluation model on the view pairs using Siamese architecture [26], then uses the evaluation model as teacher to train view proposal network (VPN), which gives the cropping candidates and scores. Lu et al. [27] regard the image cropping as a list wise ranking problem to find the best view composition and proposes a view sampling module to extract refined feature maps for candidate view generation. Note that the method needs thousands of candidates. The automatic image cropping methods based on comprehensive factors generally preserve the main content in the image, such as the salience area or the face, and combines the aesthetic score or the basic rules related to aesthetics, such as the rule of thirds and symmetry. These methods also need to generate a certain number of candidates for cropping. Zhang et al. [28] take the salience area, face detection and image resolution as input, and considers the rule of thirds, symmetry and face reservation, respectively. Wang et al. [6] establish a Gaussian mixture model, combines the priori knowledge, intuitive information, the composition information and the change information. Finally, the cropping candidates are generated through the ESS mechanism [29]. AA-ABP method [7] models photo cropping as a cascade of attention boundary regression and aesthetic classification. First, a attention box is generated to cover the most important visual area, then a set of cropping candidates is generated around the attention box, and the one with the highest aesthetic value is selected as the final cropping result. Zeng et al. [30] regards the image cropping as a problem of finding the upper left and lower right anchor positions on the grid. The method first divides the image into M × N $M \times N$ parts and then limits the cropping window generation by maintaining the main content. Finally, it selects the final cropping window based on the aesthetic score. Liang et al. [31] present a visual analysis method for interactively recomposing photos based on example photos with high-quality composition and develops an earth-mover-distance-based online metric learning algorithm to support the interactive adjustment of the composition distance based on user preferences. Reference [32] is a grid anchor-based cropping method. It models both region of interest(RoI) and region of discard(RoD) by considering local redundancy, content preservation and aspect ratio. Reference [33] is also a grid anchor-based method. It combines three lightweight neural networks designed to evaluate the quality of a crop in terms of aesthetics, composition and semantics. The work of this paper belongs to the automatic image cropping method based on comprehensive factors. This method combines the basic features and salient features of the image, comprehensively considers the cropping precision and aesthetic quality, and gradually crops by means of reinforcement learning, which does not need to produce a large number of candidate windows and conforms to the decision-making process of human cropping. 3 THE PROPOSED METHOD Cropping via deep learning with reinforcement learning The proposed method is an advanced version of Cropping via Deep Learning with Reinforcement Learning Method(DLRL) [34]. DLRL is our previous work and DRLCFA is the improvement of DLRL. So we start with a brief introduction of the basic model DLRL. The overall structure of DLRL is shown in Figure 4. DLRL is a reinforcement learning-based cropping model. It uses Double DQN [35] to learn a strategy for cropping. It crops according to the basic feature of the image and achieves great cropping efficiency and effect. FIGURE 4Open in figure viewerPowerPoint The overall structure of DLRL As we all know, reinforcement learning has four elements: state, action, agent and reward. In DLRL, the state comprised the basic feature of the image, output of DL part, and the history experience. The network prototype of the DL part is VGG16 [36]. The DL part is composed of 13 convolutional layers, 5 pooling layers and 1 fc layer, fine-tuned on the Flickr Cropping Dataset(FCD) [4] to finally output 4096-dimensional features. During fine-tuning, all convolution layers and pooling layers retain the original setting, fc8 layers is removed. To train fc6 layers and fc7 layers, make fc7 layers the final output layer with four output values, corresponding to four coordinates in cropping annotation, respectively. During the training, the ground truth cropping coordinates and the output of fc7 layer are used to calculate the mean square error to train the parameters. The DL part uses the output of fc6 layer as the final feature. The action space contains 13 actions. The definition of the action is similar to [37]. It can be divided into three categories: scaling actions, position translation actions, and aspect ratio translation actions, and the corresponding number of actions is 5, 4, and 4, respectively. They adjust the size, position and shape of the cropping window, respectively. See Figure 5 for specific information. The frame line is the boundary of the current image, and the dark part is the result after adjustment. The arrow indicates the direction of scaling or displacement. The adjust rate is 2.5%. FIGURE 5Open in figure viewerPowerPoint The details of action space The agent consists of four full connection layers, whose output dimension is 4096, 4096, 512, and 13, respectively. The last output represents the Q value of each action. In DLRL, we use Double DQN to train the parameters of the agent. The policy applied to determined the final action is ε $\varepsilon$ -greedy algorithm. The ε $\varepsilon$ -greedy algorithm randomly samples actions with a probability of ε $\varepsilon$ and selects actions with the highest action score with a probability of 1- ε $\varepsilon$ . The reward is calculated with IoU, a common cropping metric, before and after one step cropping. Specific formula is as follows. Q ( S ( t ) , A ) = R ( S ( t ) , A ) + γ max a Q ( S ′ , A ) , \begin{equation} Q(S({\rm {t}}),A) = R(S(t),A) + \gamma \mathop {\max }\limits _a Q(S^{\prime },A), \end{equation} (1) R ( S ( t ) , A ) = I o U t − I o U t − 1 , \begin{equation} R(S(t),A) = Io{U_t} - Io{U_{t{\rm { - 1}}}}, \end{equation} (2) I o U t = ( S t c ∩ S g ) / ( S t c ∪ S g ) , \begin{equation} Io{U_t} = (S_t^c \cap {S^{\rm {g}}})/(S_t^c \cup {S^{\rm {g}}}), \end{equation} (3)where Q ( S ( t ) , A ) $Q(S({\rm {t}}),A)$ , the action score, is the expected cumulative reward obtained by choosing action A. It uses γ $\gamma$ -discount cumulative reward to compute. R ( S ( t ) , A ) $R(S({\rm {t}}),A)$ is the reward of action A at present state S ( t ) $S(t)$ . S t c $S_t^c$ is the region after t-step cropping of the image, and S g ${S^{\rm {g}}}$ is the region of the ground truth cropping window of the image. In brief, the DLRL model extracts the basic feature of the image, and combines it with the history experience which records the action index used previously, as the current state. And the state is sent to Agent to get the Q value of each action. Finally an action is chosen according to the ε $\varepsilon$ -greedy algorithm. If the reward of the image is positive, the selected action will be executed and the image after one step cropping is obtained. Repeat the above process until the reward is negative. Deep reinforcement learning with comprehensive feature and aesthetics The overall network structure of the proposed method DRLCFA is shown in Figure 6. Compare to DLRL, DRLCFA has three differences. (1) DLRL only extracts the basic feature of images, but DRLCFA takes salience into consideration, use combined feature to represent the image. (2) DLRL uses Double Q Learning algorithm, DRLCFA uses V-D D3QN algorithm [38], which has better training effect. (3) DRLCFA not only concerns salience, but combines aesthetic score to evaluate a crop action. The cropping process of DRLCFA is as follows. First, input image I ( t ) $I(t)$ , that is, the image result after the t-step cropping. Then the image I ( t ) $I(t)$ is transmitted to two different branches. The DL part extracts the basic features of I ( t ) $I(t)$ through the deep learning network, and the visual saliency detection part obtains the salience map graph of the image. The output of two parts combined with historical experience vector are sent into the Agent as the current state s ( t ) $s(t)$ . The Agent outputs the corresponding Q value of each action and selects a cropping action according to the selection strategy. If the instant reward of the action is positive, the cropping action will be executed and I ( t + 1 ) $I(t+1)$ ) is obtained. Then I ( t + 1 ) $I(t+1)$ is fed into the feature extraction part and the above process is repeated. Otherwise, the cropping ends and the final cropping result is obtained. The initial cropping is exactly the same as the following cropping steps, except that the input of initial cropping is the original image, the input of other step is the cropping result of the previous step. The instant reward relates to the aesthetic quality assessment of the current results. The DL part and the action space is the same as that of DLRL [34]. FIGURE 6Open in figure viewerPowerPoint The overall structure of DRLCFA Feature extraction part The feature extraction part of DRLCFA method consists of two parts, as shown in Figure 6. The first is the DL part, the basic feature extraction part of the image, which has been introduced in the basic model DLRL. The second is the visual salience detection part. We will introduce the details of the second part in this section. Visual salience detection part outputs a 64 × $\times$ 64 dimensional salience map, reshaped to 4096 × $\times$ 1 dimension. The output combines with the output of the DL part to become the final feature. In the visual salience detection part, the classical visual salience detection method FT method [39] is adopted, which makes use of the colour and brightness features of the image. It is simple to calculate, efficient and produces excellent results. The specific calculation formula is as follows: L t ( x , y ) = I t g a u s s ( x , y ) − I t m , \begin{equation} {L_{\rm {t}}}(x,y) = {\left\Vert {I_t{gauss(x,y)}{\rm { - }}I_t^m} \right\Vert} , \end{equation} (4) M t ( x , y ) = 0 , L t ( x , y ) < t h r e s h o l d , 1 , e l s e \begin{equation} {M_t}(x,y) = {\left\lbrace { \def\eqcellsep{&}\begin{array}{ll} 0,&\quad {L_{\rm {t}}}(x,y) < threshold,\\[7pt] 1,&\quad {else} \end{array} } \right.} \end{equation} (5) t h r e s h o l d = ( L t min + L t max ) / 2 , \begin{equation} threshold = (L_t^{\min } + L_t^{\max })/2, \end{equation} (6)where M t ( x , y ) ${M_t}(x,y)$ is the final salience map of I ( t ) $I(t)$ , I t m ${I_t^m}$ is the mean image of I ( t ) $I(t)$ , calculated in LAB colour space, I t g a u s s ( x , y ) ${I_t^gauss}(x,y)$ is the Gaussian blur result in LAB space, ∥ ∥ $\Vert \Vert$ is the operator of Euclidean distance, L t min $L_t^{\min }$ is the minimum value in L t ( x , y ) ${{L_{\rm {t}}}(x,y)}$ , and L t max $L_t^{\max }$ is the maximum in L t ( x , y ) ${{L_{\rm {t}}}(x,y)}$ . There are some salience map calculated by FT method below in Figure 7. As shown in the figure, visual salience detection part extracts the visual important part of the images successfully and it is very efficient, costing only 0.02 s around. FIGURE 7Open in figure viewerPowerPoint Some results of visual salience detection calculated by FT method. The first row are original images and the second row are the result Reinforcement learning method and the agent The reinforcement learning method adopted in DRLCFA method is based on the idea of V-D D3QN method [38]. This method is a variant of the Double Dueling Q-learning Network, which changes the update mode of the Q Network in a innovative way while retaining the duelling structure [40]. V-D D3QN contains two Q networks with identical structures but different parameters. Here we refer to them as D3QN-A and D3QN-B, respectively. It should be noted that the reinforcement learning training adopted in DRLCFA only borrows from the thought of V-D D3QN, but the specific structure is not the same. In DRLCFA, the structure of a single Q network is shown in Figure 6, which is the same as the structure of the Agent. The output dimension of each layer of FC-A is 4096, 4096, 512 and 13, which represents the Advantage of taking each action in the current state. The output dimension of each layer of FC-V is 4096, 4096, 512 and 1, which represents the Value of the current state. The final output Q value is the sum of Advantage and Value, and the formula is as follows: Q ( s , a ) = A ( s , a ) + V ( s ) , \begin{equation} Q(s,a) = A(s,a) + V(s), \end{equation} (7)where A ( s , a ) $A(s,a)$ is the result of Advantage and V ( s ) $V(s)$ is the result of value. The reinforcement learning algorithm of DRLCFA is shown in Table 1. TABLE 1. The RL algorithm of DRLCFA First initialise two Q networks, D3QN-A network Q A ( s , a ; θ A ) ${Q^A}(s,a;{\theta ^A})$ and D3QN-B network Q B ( s , a ; θ B ) ${Q^B}(s,a;{\theta ^B})$ , where θ A ${\theta ^A}$ and θ B ${\theta ^B}$ refer to all the parameters in the corresponding network. Second, initialise the Experience Reply in reinforcement learning, that is ( s t , s t + 1 , a t , r t ) $({s_t},{s_{t + 1}},{a_t},{r_t})$ . s t $s_t$ consists of two parts, one is the comprehensive feature extracted by the feature extraction network, which consists of basic feature and salience map, one is the historical experience vector history. It reserves the action index used previously. In our method, the max exploring step is 20, action number is 13, so that a 20 × $\times$ 13d vector is initialised to represent the historical experience. Inspired by human decision-making, historical experience is also embedded in state representation to aid current decision-making. After reading a mini-batch image, the image are sent to feature extraction part to get the initial state of the images as s 0 $s_0$ . Then the states are sent to the Agent, and select the current cropping action according to the ε $\varepsilon$ -greedy algorithm. Finally, execute the selected cropping action, and obtain the cropped image. Repeat the above process, and record the complete one-step cropping operation ( s t , s t + 1 , a t , r t ) $({s_t},{s_{t + 1}},{a_t},{r_t})$ in the experience reply pool. Max cropping step, M a x _ s t e p s $Max\_steps$ is 20. In the training process, we set the maximum iteration number N _ e p i s o d e s $N\_episodes$ as 160,000, randomly selected a group of records for learning each time. For the loss given in Table 1, gradient descent method is used to calculate parameters. The network is updated every 1000 times, so the u p d a t e _ s t e p $update\_step$ is 1000. Reward function The formula of the reward is as follows: R ( S ( t ) , A ) = r ( t ) − r ( t − 1 ) , \begin{equation} R(S(t),A)=r(t) - r(t - 1), \end{equation} (8) r ( t ) = a e s t h e t i c s c o r e + I o U t , \begin{equation} r(t) = aesthetic\quad score + Io{U_t}, \end{equation} (9)where r ( t ) $r(t)$ consists of two parts, a e s t h e t i c s c o r e $aesthetic\ score$ and I o U t $Io{U_t}$ . The aesthetic evaluation of DRLCFA method is realised by aesthetic assessment network View Finding Network (VFN) [2]. The VFN is trained on two aesthetic data sets AVA [41] and AADB [42], respectively. We retain the best model as the final aesthetic evaluation model. This part outputs an aesthetic score between (0, 1) to measure the image quality of aesthetics. The IoU is used to measure the overlap degree between the ground truth cropping window and the current cropping results. The specific calculation formula of I o U t $Io{U_t}$ is the same thing as shown in Equation(3). The structure of VFN is shown in Table 2. The parameters of convolution layers represent kernel width, kernel height, kernel number, x stride and y stride. The parameters of max pooling layers represent kernel width, kernel height, x stride and y stride. The parameters of local response normalisation layers represent depth radius, alpha and beta value. Spatial Pyramid Pooling layer(SPP) [42] consists of three max pooling layers. The parameters of fc layers is the output dimension. The VFN is trained on two aesthetic data sets AVA [41] and AADB [42], respectively. We retain the best model as the final aesthetic evaluation model. This part outputs an aesthetic score between (0, 1) to measure the image quality of aesthetics. TABLE 2. The structure of VFN layer params conv1 11,11,96,4,4 max pool 3,3,2,2 lrn 2,2 × $\times$ 10 − 5 $^{-5}$ ,0.75 conv2 5,5,256,1,1 max pool 3,3,2,2 lrn 2,2 × $\times$ 10 − 5 $^{-5}$ ,0.75 conv3 3,3,384,1,1 conv4 3,3,384,1,1 conv5 3,3,256,1,1 spp 7,7,6,6 5,5,4,4 3,3,2,2 fc6 4096 fc7 1 4 EXPERIMENTAL RESULTS Training model We test experiment our method DRLCFA on two image cropping data sets, CUHK Image Cropping Dataset(CUHK-ICD) [3] and Flickr Cropping Dataset (FCD) [4]. The CUHK-ICD consists of 950 test images, marked by three professional photographers. Each image has three cropping annotations. FCD contains 332 test images, each with one cropping annotation. We train DRLCFA on FCD and test it on both. The total number of training images is 1299. Batch size is 4. The initial value of learning rate is set to 10 − 5 $^{-5}$ , and the minimum value is set to 10 − 8 $^{-8}$ . The learning rate decreases every 5000 iterations, decreasing by 0.96 times. The maximum value of instant reward is set to 0.5, and the minimum value is set to −0.5. During the training, the ε $\varepsilon$ in action selection strategy is changing. For every increase of 40,000 in the iteration steps, the ε $\varepsilon$ value decreases by 0.1 and the minimum of ε $\varepsilon$ is 0.1. The setting insures the fully exploration of the environment. The feature extraction part and the reinforcement learning parameters are trained separated. The DL network is pre-trained on FCD. After the t