Estimating the visibility in foggy weather based on meteorological and video data: A Recurrent Neural Network approach

IET Signal ProcessingVolume 17, Issue 1 e12164 ORIGINAL RESEARCHOpen Access Estimating the visibility in foggy weather based on meteorological and video data: A Recurrent Neural Network approach Jian Chen, Corresponding Author Jian Chen [email protected] orcid.org/0000-0002-2097-5740 School of Mechanical Engineering, Yangzhou University, Yangzhou, China Correspondence Jian Chen, School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China. Email: [email protected]Search for more papers by this authorMing Yan, Ming Yan orcid.org/0000-0003-4519-4075 School of Mechanical Engineering, Yangzhou University, Yangzhou, ChinaSearch for more papers by this authorMuhammad Rabea Hanzla Qureshi, Muhammad Rabea Hanzla Qureshi School of Mechanical Engineering, Yangzhou University, Yangzhou, ChinaSearch for more papers by this authorKeke Geng, Keke Geng School of Mechanical Engineering, Southeast University, Nanjing, ChinaSearch for more papers by this author Jian Chen, Corresponding Author Jian Chen [email protected] orcid.org/0000-0002-2097-5740 School of Mechanical Engineering, Yangzhou University, Yangzhou, China Correspondence Jian Chen, School of Mechanical Engineering, Yangzhou University, Yangzhou 225127, China. Email: [email protected]Search for more papers by this authorMing Yan, Ming Yan orcid.org/0000-0003-4519-4075 School of Mechanical Engineering, Yangzhou University, Yangzhou, ChinaSearch for more papers by this authorMuhammad Rabea Hanzla Qureshi, Muhammad Rabea Hanzla Qureshi School of Mechanical Engineering, Yangzhou University, Yangzhou, ChinaSearch for more papers by this authorKeke Geng, Keke Geng School of Mechanical Engineering, Southeast University, Nanjing, ChinaSearch for more papers by this author First published: 13 September 2022 https://doi.org/10.1049/sil2.12164AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The research of visibility detection in foggy days is of great significance to both road traffic and air transport safety. Based on the meteorological and video data collected from an airport, a deep Recurrent Neural Network (RNN) model was established in this study to predict the visibility. First, the Fourier Transform was used to extract feature variables from video data. Then, the Principal Component Analysis method was used to reduce the dimension of features. After that, 462 sets of sample data include image features, air pressure, temperature and wind speed, were used as inputs to train the RNN model. By comparing the predicted results with the actual visibility data as well as some other state-of-the-art methods, it can be found that the proposed model makes up for the deficiency of models based only on meteorological or image data, and has higher accuracy in different grades of visibility. With considering the meteorological data, the accuracy of RNN model is improved by 18.78%. Besides, with aids of correlation analysis, the influence of the meteorological factors on the predicted visibility was analysed, for fog at night, temperature is the dominant factor affecting visibility. 1 INTRODUCTION Visibility is a general index widely used in weather forecast, road traffic and aeroplane flight. The problem of visibility detection has been attached to great importance by the meteorological, highway and aviation management departments. For example, in the field of weather forecasting, the evolution and dissipation of foggy weather can be predicted through the study of visibility, so as to provide a decision-making basis for the traffic department to reduce the occurrence of traffic accidents. In the field of intelligent cars, vehicle detection or object tracking methods play an important role in the development process of driver assistance and driverless systems. In foggy weather conditions, the existing methods of vehicle detection lack of robustness [1], in addition to improving the image quality [2, 3], intelligent vehicles also need to adopt different driving modes according to different visibility to improve safety performance. In terms of aviation safety [4], Runway Visual Range (RVR) [5] is used to reflect the size of fog and haze near the airport, which is defined as the maximum distance of the runway or the target object close to the runway (Side lights for the runway at night) that can be identified along the runway direction. Thus, the research of visibility detection in foggy or hazy weather is significant. Atmospheric visibility can generally be measured by visual estimation, or by instruments, such as atmospheric transmissometer or laser distance measurer etc. [6] At present, most of the visibility observations are still based on manual visual measurement, which is relatively poor in standardisation and objectivity. The measurement with a measuring instrument is more objective and accurate, but the cost of maintenance is high. With the development of image processing technology, there is a growing interest in using cameras to track objects or estimate visibility. The equipment is relatively cheap and much easier to be updated and maintained, which mainly depends on algorithm implementation. For the applications of camera in target tracking, an improved Kernel Correlation Filter algorithm was proposed by Xia et al. [7] to deal with the problem of failing to track objects under the influence of occlusion conditions. Aimed at occlusions and background clutters, a tracking framework combining correlation filter tracking and Siamese-based object tracking was proposed by Zhang et al. [8], which performs favourably against many existing trackers in the TC128dataset. Besides, a multiple features fusion method was designed by Zhang et al. [9] to improve the representation ability of the tracking object. A correlation filter object function model called Spatial-Channel Selection and Temporal Regularised Correlation Filters was proposed to obtain more robust appearance model in tracking. The optimal solution was obtained by iteratively solving three subproblems using Alternating Direction Multiplier Method (ADMM). In recent years, the neural network model has become a popular method for exploring the trends of fog occurrence and dissipation. Based on a large amount of meteorological data collected from meteorological monitoring stations throughout North Carolina, a Back Propagation (BP) neural network visibility model was proposed by Duddu et al. [10], with deep learning from the vast data, the relationship between meteorological data and visibility was investigated by using function approximation method. Aiming at the problem of foggy events classification by meteorological input variables, an artificial neural network with evolutionary training and different basic functions was proposed by Duran-Rosal et al. [11]. Deng et al. [12] transformed the traditional visibility prediction problem into a time series prediction problem, the Long Short-term Memory (LSTM) model was optimised to predict visibility, which shows very high accuracy. Goswami et al. [13] developed an Adaptive Neuro-Fuzzy Inference System (ANFIS), and in comparison with other neural network models, the prediction error of this model is relatively small. In this study, based on meteorological and video data collected from Lukou airport of Nanjing [14], the visibility was predicted according to the proposed RNN model. Three main meteorological factors (air pressure, temperature and wind speed) and the feature vectors transformed from the video data were taken as the inputs, the visibility was regarded as output to train the RNN model. By a combination of meteorological and video data, the proposed model shows the best prediction ability in different grades of visibility. Our contributions can be summarised as follows: (1) a new RNN model was established to estimate the visibility based on the meteorological and video data; (2) the obtained visibility results were compared with the results based only on video data, as well as the results of other state-of-the-art models, which shows the feasibility of the proposed method; (3) with aids of correlation analysis, the influence of the meteorological factors on the predicted visibility was analysed. 2 RELATED WORK In general, the camcorder-based approach used for visibility prediction can be divided into three categories [15]: For the first method, the distance of the farthest black object in the image is taken as the visible distance. The threshold and image segmentation techniques are used to locate targets, such as lane markers, road signs and intersection lines etc. This method requires precise geometric calibration of the camera. Literature [16] is based on this method, the visibility was defined by the International Commission on Illumination (CIE) as the distance beyond which a black object of an appropriate dimension is perceived with a contrast of less than 5%. It fully uses the advantage of a single camera, but requires only the road and the sky in the scene. Based on Koschmieder's law, a mathematical model of contrast between a single black object and the sky background was established by Tang et al. [17]. A new black target vision measurement system was introduced, which is simple in structure and does not require much calibration. Kim [6] measured the visibility based on measuring the distance between visible objects, by comparing with Human eye-based Visual Range (HVR) and Meteorological Optical-based Range (MOR) measurements, it was found that the Image-based Visual Range (IVR) was more accurate, especially, when enough of the scene is captured. In the paper of Yang et al. [18], by collecting road images and analysing road information in the pictures, a visibility range estimation algorithm for real-time adaptive speed limits control in an intelligent traffic system was proposed, in which, the dark channel prior, weighted image entropy and support vector machine were utilised to generate accurate visibility estimates. The second method is based on the calculation of scene contrast. With the aid of the linear regression method, the relationship between the scene contrast and visibility can be calculated. The paper from He et al. [19] is exactly based on this method, but its innovation lies in the use of BP neural network instead of linear regression. In the paper of Wang et al. [20], which proposed a method of atmospheric visibility detection based on image brightness contrast, the gain error of measuring target brightness and sky background brightness were analysed in detail. According to the characteristics of image HSV (Hue, Saturation and Value) colour histogram, a fog level detection method was proposed by Liu et al. [21], by which, the judgement threshold value of fog level detection was firstly determined, the fog levels were classified into three different groups, and the accuracy of this method was discussed. The third method of visibility detection is by using the global descriptor computed on the whole image. With the continuous development of artificial intelligence, neural network has gradually become the mainstream computer vision algorithm, and its application scope is also more and more extensive. In the paper of Chaabani et al. [15], the Fourier Transform method was used to extract the global features of the image, and a three-layer neural network model was trained by the BP algorithm, the visibility was estimated and classified, the edge detection or distance of the reference target was no longer necessary. A new method for visibility estimation based on Deep Hybrid Convolutional Neural Network (DHCNN) was proposed by Palvanov et al. [22], a two-stepped Laplacian of Gaussian (LoG) filter method was introduced, which can estimate the visibility distance very accurately. Chaabani et al. [23] also proposed a method with combining the Convolutional Neural Network (CNN) and the Support Vector Machine (SVM) classifier, as an image feature extractor, the CNN inputs the feature to the SVM classifier, this method performs well in obstacle avoidance system, which has a promising research prospect. A novel approach to estimate visibility range under foggy weather conditions was proposed by Outay et al. [24], the standard AlexNet [25] architecture was used, and a Deep Convolutional Neural Network (DCNN) was trained with raw image data, for feature extraction and the SVM classifier was also used. In the paper of He et al. [19], the artificial neural network was also trained, in which, the meteorological data: temperature, relative humidity, air pressure and wind speed as well as the image data: image saturation, minimum grey value and image contrast were regarded as the input layer, the visibility was regarded as the output layer, the error between the predicted visibility and the actual visibility of this method was only 7.56%. In summary, there are many literature exploring the problem of visibility prediction with different methods, but the data used by the existing models vary widely. For different papers, the utilisation of meteorological and video data has their advantages and disadvantages, but the analysis of the impact of these factors on visibility estimation is not very clear. The previous methods based on meteorological data need to fit the relationship between visibility and meteorological factors, when the order of the fitting equation is very high, the physical implication is very complicated to be explained. For the camera-based method, when the visibility is high, the change of the images is minimal, but the error between the actual and estimated visibilities will be huge. The remainder of this paper is organised as follows: Section 3 presents an introduction about the data used in this study, both the meteorological and video data collected from the airport will be analysed. Section 4 presents our proposed deep learning approach for visibility estimation. In Section 5, the predicted visibility data are compared with the actual visibility data as well as some other typical state-of-the-art methods, the validity of the model was discussed in detail. Section 6 concludes with a summary of the main findings of the paper. 3 DATA ACQUISITION AND PREPROCESSING This study is mainly based on AMOS data of Nanjing Lukou Airport provided by ref. [14], from 0:00 AM to 8:00 AM in the morning on March 13th, 2020, which includes the data of visibility, ground temperature, relative humidity, air pressure, dew point temperature, wind speed and direction etc. Due to the large amount of meteorological observation information contained in the original observation data, only ground temperature, relative humidity, air pressure, wind speed which are closely related to the formation and dissipation of fog were considered in this study. Among these factors considered above, the measured relative humidity H is basically unchanged in the whole period from 22:27 PM on March 12th, 2020 to 8:00 AM on March 13th, 2020 (as shown in Figure 1). Therefore, relative humidity was not taken as the main research factor in this study. When foggy days occur, the relative humidity of the air is quite high [26], and the rise of relative humidity is non-linear [27]. For this study, the statistical data in dense fog conditions is used, the air humidity is saturated, and the relative humidity remains at 100%. The relative humidity does not change in the whole period, which is in accord with the laws of nature. FIGURE 1Open in figure viewerPowerPoint The change of relative humidity over time In addition to relative humidity, the meteorological data considered in this study mainly include visibility M (m), temperature T (°C), pressure P (Pa) and wind speed V (m/s). The changes of these meteorological factors and visibility over time were drawn by MATLAB [28], which are shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Changes of visibility and meteorological factors over time. Figures (a), (b), (c) and (d) show the changes of visibility, air pressure, temperature and wind speed respectively In addition to the above meteorological data, 8 h of surveillance video data of Nanjing Lukou Airport from 0:00 AM to 8:00 AM on March 13th, 2020 was also used in this study. The pixel size of the original video image is 1280 × 720. A screenshot of the video at 00:02:25 is provided in Figure 3. In this figure, the signal pole is the target object of the whole video, which always lies in the centre of the image. FIGURE 3Open in figure viewerPowerPoint A screenshot of video data at 00:02:25. The original video pixel size is 1280 × 720, the image pixel size of the extracted red block is 220 × 573 Since Fourier Transform is used to extract the global features of the images, the size of the background will not affect the rule of extracting features. Therefore, only the image in the red box was intercepted. The time change in the upper left corner may produce noise to image feature extraction, which can also be eliminated by this image interception method. In addition, the computation amount in subsequent data processing can also be greatly reduced. Finally, 462 images with pixels of 220 × 573 were extracted, which are in correspondence to meteorological data. 4 METHODOLOGY Digital image has become a part of our daily life, and the research related to digital image processing is becoming more and more important. How to improve image resolution or reduce image noise is always a hot topic. Extracting the features of foggy images is the key to predicting the visibility of foggy images, the Fourier Transform will be applied in this study to extract the features. However, when the dimension of the obtained data is too high, which cannot be used to train the RNN model, the PCA method will be adopted to reduce the data dimension. Finally, the data obtained from dimension reduction as well as the meteorological data will be used to train the RNN neural network model and estimate the visibility. Obtaining image features by Fourier Transform A landscape image always contains objects at different distances. With the decrease in visibility, the contour of objects gradually blurred and finally disappeared, and the proportion of large white fog in the image increased continuously. This change can be measured in the frequency domain by taking the Fourier transform [29] of the image. Methods such as: brightness method, image contrast method etc., which is mainly focussing on the region of interest in the image and ignoring the features of the rest of the region. Fourier transform can be used to extract features from the whole image without manual calibration. The physical meaning of Fourier Transform is to convert the image information from the grey pixel model into the frequency domain. Image frequency is used to represent the intensity of grey change in the image, which is the gradient of grey level in plane space. If grey changes quickly, the frequency is high. If grey changes slowly, the frequency is low. When the fog thickens, the edge of the scene in the image gradually clears, and the grey level changes rapidly. When the fog fades, the edge of the scene in the image will become fuzzy, and the grey level will change slowly. Fourier Transform was firstly introduced to transform the image information of grayscale pixel mode into a frequency domain. The Fourier Transform pair can be written as: F ( ω ) = ∫ − ∞ + ∞ f ( t ) e − j ω t d t $F(\omega )=\int \nolimits_{-\infty }^{+\infty }f(t){e}^{-j\omega t}dt$ (1) f ( t ) = 1 2 π ∫ − ∞ + ∞ F ( ω ) e j ω t d ω $f(t)=\frac{1}{2\pi }\int \nolimits_{-\infty }^{+\infty }F(\omega ){e}^{j\omega t}d\omega $ (2) In which ω is the frequency, t is the time, F(ω) denotes the function in the frequency domain, and f(t) denotes the function in the time domain. Since only discrete data can be processed by computer, n-point equidistant sampling is needed for the continuous Fourier Transform pair: F ( u ) = ∑ x = 0 n − 1 f ( x ) e − j 2 π n x u $F(u)=\sum\limits _{x=0}^{n-1}f(x){e}^{-j\tfrac{2\pi }{n}xu}$ (3) f ( x ) = 1 n ∑ u = 0 n − 1 F ( u ) e j 2 π n x u $f(x)=\frac{1}{n}\sum\limits _{u=0}^{n-1}F(u){e}^{j\tfrac{2\pi }{n}xu}$ (4) In which u refers to the horizontal frequency of the image, x is the horizontal pixel coordinates of the image, F(u) denotes the horizontal frequency domain function of the image and f(x) denotes the horizontal pixel function of the image. Since the picture is two-dimensional data, it is necessary to extend the formula to two-dimensions, finally, the Discrete Fourier Transform (DFT) [29] pair can be written as: F ( u , v ) = ∑ x = 0 m − 1 ∑ y = 0 n − 1 f ( x , y ) e − j 2 π u x m + v y n $F(u,v)=\sum\limits _{x=0}^{m-1}\sum\limits _{y=0}^{n-1}f(x,y){e}^{-j2\pi \left(\tfrac{ux}{m}+\tfrac{vy}{n}\right)}$ (5) f ( x , y ) = 1 m ⋅ n ∑ u = 0 m − 1 ∑ v = 0 m − 1 F ( u , v ) e j 2 π u x m + v y n $f(x,y)=\frac{1}{m\cdot n}\sum\limits _{u=0}^{m-1}\sum\limits _{v=0}^{m-1}F(u,v){e}^{j2\pi \left(\tfrac{ux}{m}+\tfrac{vy}{n}\right)}$ (6) In which n is the total number of pixels in the vertical direction, m is the total number of pixels in the horizontal direction, u is the horizontal frequency, v is the vertical frequency, x is the horizontal pixel coordinates and y is the vertical pixel coordinates of the image. For a binary DFT, which can be superposed by two unary DFT, so the formula can be further optimised as follows: F ( u , v ) = ∑ x = 0 m − 1 ∑ y = 0 n − 1 f ( x , y ) e − j 2 π v y n e − j 2 π u x m $F(u,v)=\sum\limits _{x=0}^{m-1}\left[\sum\limits _{y=0}^{n-1}f(x,y){e}^{-j2\pi \tfrac{vy}{n}}\right]{e}^{-j2\pi \tfrac{ux}{m}}$ (7) For the convenience of image observation and the further processing, the low-frequency part of the four quadrants is generally transferred to the centre of the Fourier Transform picture by using Equation (8): F u − m 2 , v − n 2 = ∑ x = 0 m − 1 ∑ y = 0 n − 1 ( − 1 ) x + y f ( x , y ) e − j 2 π u x m + v y n $F\left(u-\frac{m}{2},v-\frac{n}{2}\right)=\sum\limits _{x=0}^{m-1}\sum\limits _{y=0}^{n-1}{(-1)}^{x+y}f(x,y){e}^{-j2\pi \left(\tfrac{ux}{m}+\tfrac{vy}{n}\right)}$ (8) Figure 4 shows the observation images with visibility of 1620, 516, 161 and 77m respectively. It can be seen from the pictures that when the visibility decreases, the object becomes more and more blurred, and less detail can be observed. FIGURE 4Open in figure viewerPowerPoint Screenshots of video data under different visibility conditions. Figures (a), (b), (c) and (d) show the observation images with visibility distances of 1620, 516, 161 and 77 m respectively The Fourier transform produces a spectrum with the centre of the image as the origin point. In the picture, the distance from the pixel to the origin point denotes the frequency. The farther the distance, the higher the frequency. The brightness of the pixel demotes the amplitude, the brighter the pixel, the higher the amplitude. It can be seen from Figures 4 and 5 that as the fog thickens, visibility decreases, object contour information decreases and the brightness of high-frequency pixels in the spectral graph decreases. The function of the Fourier transform is to transform a disordered image that a computer cannot distinguish into an ordered image that a computer can distinguish. FIGURE 5Open in figure viewerPowerPoint Fourier Transform images under different visibility conditions. Figures (a), (b), (c) and (d) show the Fourier Transform images with visibility distances of 1620, 516, 161 and 77 m respectively Data dimension reduction by PCA method The pixel size of the captured image is 220 × 573, after the Fourier Transform, a matrix with a size of 220 × 573 was obtained. In the process of training RNN model, some useless data in the matrix will bring noise, which will affect the calculation result and increase the calculation cost. Therefore, the PCA method [30] was introduced to reduce the dimension of the data and explore some potential characteristic variables. The principle of the PCA method is to reduce the dimension by projecting the data into a low-dimensional subspace. For example, two-dimensional data sets can be reduced to two dimensions by projecting points onto a line, while three-dimensional data sets can be reduced to two dimensions by mapping variables into a plane. In general, n-dimensional data sets can be reduced to k-dimensional subspace by the mapping method, when k ≤ n. The process is as follows: First of all, the covariance matrix of the sample needs to be established, assuming that the sample X = (X 1, X 2, …, X n ) T is an n-dimensional random matrix, the covariance matrix of this sample can be written as: D ( X ) = c 11 c 12 ⋯ c 1 n c 21 c 22 ⋯ c 2 n ⋮ ⋮ ⋮ ⋮ c n 1 c n 2 ⋯ c n n c a b = C o v X a , X b , a , b = 1,2 , ⋯ , n $\left\{\begin{array}{l}D(X)=\left(\begin{array}{cccc}{c}_{11}& {c}_{12}& {{\cdots}}& {c}_{1n}\\ {c}_{21}& {c}_{22}& {{\cdots}}& {c}_{2n}\\ {\vdots}& {\vdots}& {\vdots}& {\vdots}\\ {c}_{n1}& {c}_{n2}& {{\cdots}}& {c}_{nn}\end{array}\right)\\ {c}_{ab}=Cov\left({X}_{a},{X}_{b}\right),\ a,b=1,2,{{\cdots}},n\end{array}\right.$ (9) In which D(X) is the covariance matrix of the sample, c ab is the variance, X a is the sample matrix of column a and X b is the sample matrix of column b. Then, eigenvalues λ and eigenvectors U need to be computed from the covariance matrix in order to determine the principal components of the data. After calculating the mean value of the sample, an n × n matrix can be obtained: X ‾ = X 1 ‾ X 2 ‾ ⋯ X n ‾ X 1 ‾ X 2 ‾ ⋯ X n ‾ ⋮ ⋮ ⋮ ⋮ X 1 ‾ X 2 ‾ ⋯ X n ‾ n × n $\overline{X}={\left(\begin{array}{cccc}\overline{{X}_{1}}& \overline{{X}_{2}}& {{\cdots}}& \overline{{X}_{n}}\\ \overline{{X}_{1}}& \overline{{X}_{2}}& {{\cdots}}& \overline{{X}_{n}}\\ {\vdots}& {\vdots}& {\vdots}& {\vdots}\\ \overline{{X}_{1}}& \overline{{X}_{2}}& {{\cdots}}& \overline{{X}_{n}}\end{array}\right)}_{n\times n}$ (10) In which X ‾ $\overline{X}$ represents the mean matrix of the sample and X n ‾ $\overline{{X}_{n}}$ represents the mean of the nth column of the sample matrix. The feature matrix on each PCA was obtained by Equation (10), denoted as P, the final dimension reduction data is the first column of matrix P: P = X − X ‾ U $P=\left(X-\overline{X}\right)U$ (11) In which U is the eigenvectors of the covariance matrix. According to the above method, dimensionality reduction of the Fourier-transformed images with visibility of 1620, 516, 161 and 77 m was conducted. The obtained dimensionality reduction data is shown in Figure 6. The red curve represents the raw data, the black curve represents the data after dimensionality reduction. As can be seen from the images, with the reduction of visibility, the peak values of the images are increasing and the minimum value is also increasing. Taken as a whole, the range of most values is compressed. The relevant characteristic variables are well preserved by the PCA dimensionality reduction method. FIGURE 6Open in figure viewerPowerPoint Dimensionality reduction pictures of different visibility values. Figure (a), (b), (c) and (d) correspond to the dimensionality reduction data when the visibility is 1620, 516, 161 and 77 m respectively After the PCA dimensionality reduction, 573 features were obtained and used as part of the input variables of the RNN to train the model. Establishment of the RNN model Generally, CNN can be used to predict visibility only through isolated single images. But in the practical situation, the change of fog is not discrete, but a continuous process. Wind, air temperature and pressure also change continuously. The density of fog in the previous and next moments is inevitably related. The change of visibility in the foggy conditions is also continuous. By considering the continuity of visibility change, a RNN model was established. Generally, the RNN model is used to generate discrete sequences and predict continuous variables [31]. The schematic diagram of the RNN method is shown in Figure 7. In which, N 1, N 2, N 3, …, N 573 are the feature values extracted by the Fourier transform and the PCA dimensionality reduction, together with temperature T, pressure P and wind speed V, which constitute the input layer. The hidden layer was set as 1 layer, and the number of neurons was set as 200. The tanh function was used as the activation function between the hidden and the output layer. For the output layer, M t is the visibility at time t. FIGURE 7Open in figure viewerPowerPoint Schematic diagram of the Recurrent Neural Network (RNN) The mathematical model of RNN is shown in Equation (12), where, the input is the input vector X t and X t−1 at time t and t−1 respectively, M t is the output vector,