# Study of Precipitation Forecast Based on Deep Belief Networks
## 3 Material and Methods
### Deep Belief Network  
Deep belief networks are developed based on multi-layer restricted Boltzmann machines (RBMs) [23]. The algorithm is composed of a multi-layer restricted Boltzmann machine (RBM) and a BP (back propagation) network. The top layer of the BP network fine-tunes the RBM layers below to improve the performance of the algorithm. The restricted Boltzmann machine is divided into two parts. The first part is a visible layer, V, which is used to receive the feature data, and the second part is a hidden layer, H, which is used as a feature detector, including abstract features of the data [24]. This is illustrated in the Figure 2 below. The study [25] showed that RBMs can be stacked and trained in a greedy manner to form so-called deep belief networks (DBN). DBNs learn to extract a deep hierarchical representation of the training data. The joint distribution between the input vector, \(x\), and the \(l\) hidden layers, \(h^{k}\), is as follows:  
\[P(x,h^{1},\cdots,h^{l})=(\prod_{k=0}^{l-2}P(h^{k}\Big{|}h^{k+1}))P(h^{l-1},h^{ l}) \tag{1}\]  
where \(x=h^{0}\), and \(P(h^{l-1},h^{l})\) is the V-H joint distribution in the top-level. RBM \(P(h^{k-1}|h^{k})\) is a conditional distribution for the units of the visible layer conditioned on the units of the hidden layer of the RBM at the level, \(k\). Figure 1: The flowchart of the support vector machine with particle swarm optimization (PSO-SVM) model. Figure 2: The structure chart of deep belief networks. This paper uses a fast layer learning algorithm for deep belief nets proposed by [PERSON] (2016). The principle of greedy layer-wise unsupervised training can be applied to DBNs with RBMs as the building blocks for each layer [PERSON] (2016); [PERSON] (2016). The process is as follows:  
1. Train the raw input, \(x=h^{(0)}\), as the first RBM layer. The first layer is its visible layer. 2. The hidden layer of the first RBM layer is used as the visual layer of the second RBM layer. The output of the first layer is used as the input of the second layer. This representation can be chosen as being the samples of \(p(h^{(1)}|h^{(0)})\) or mean activations of \(p(h^{(1)}=1|h^{(0)})\). 3. Take the transformed samples or mean activations as training examples to train the second layer as an RBM. 4. Repeat Step 2 and Step 3, upward of either samples or mean values each iterate. 5. When the training period is reached, or this satisfies the stop condition, end the iteration. In this paper, we focus on fine-tuning by supervised gradient descent (SGD). We use a logistic regression classifier (LRC) to classify the input vector, \(x\), based on the output of the last hidden layer, \(h^{(l)}\), of the DBN. Fine-tuning is then performed via SGD of the negative log-likelihood cost function. The program module is shown in Figure 3.  
Each module in Figure 3 is described as follows:  
1. Import data sets  
Import preprocessed weather data from the database. The data is divided into three data sets: Training data, verification data, and test data. Store data and data labels into data_set and data_label_set. 2. Conversion data format  
The format of the read data is a matrix, which requires further processing. Therefore, the data set can be loaded into the shared variable, reducing the consumption of continuously copied data, and improving the efficiency of the program. At the same time, the label of the data set is converted into a one-dimensional vector to facilitate the program calculation. 1. Establish DBN model  
Initialize the parameters. Set the learning rate, fiftytune_lr = 0.1, the maximum iteration number of pre-training, pretraining_epochs = 100, the learning rate of pre-training, pretrain_lr = 0.01, the maximum iteration number of training, training_epochs = 100, and the batch data size, batch_size = 10. Figure 3: The flow chart of the deep belief networks model. (4) Pre-training model  
Divide the training data into several minbatches to shorten the overhead of building models. At the same time, we give an index to every minbatch.