# Internet Traffic Prediction with Distributed Multi-Agent Learning
## 4 Methodology  
In this paper, a distributed multi-agent learning approach is proposed for Internet traffic prediction, as shown in Figure 2. An interaction process is added into the individual training process so that the base prediction models are trained in a cooperative approach among the agents and an aggregated prediction is used as the final output, e.g., an average value. In this study, the base prediction model is considered a deep learning model with multiple layers and the described interaction is conducted at a layer-to-layer level for agents so that the prediction model can be cooperatively trained. Mathematically, the influence between different agents in the interaction process is modeled as an adjacency matrix \(W\) with row sum 1. Denote that there are \(m\) agents and the agent set is \(I\). Each agent trains a base prediction model with \(L\) layers. Denote \(f_{i}^{(\ell)}\) as the mapping function for layer \(\ell\) of agent \(i\), to extract information from the input historical data. The distributed multi-agent learning in a single iteration is formulated as follows:  
\[f_{i}^{(\ell+1)}=\sum_{i=1}^{m}w_{ij}f_{j}^{(\ell)} \tag{1}\]  
Figure 1: The Internet traffic data in the first three weeks. (**a**) Mean values; (**b**) Maximum values; (**c**) Minimum Values. where \(w_{ij}\) is an element of \(W\), which is usually a positive value between 0 and 1. \(w_{ij}\) reflects the interaction degree among agent \(i\) and agent \(j\). The above process can be iterated \(n\) times, and the convergence of the distributed multi-agent learning approach can be guaranteed when \(n\) is large enough [12]. The interaction among different agents is modeled as an irreducible aperiodic Markov chain with a finite state, in which a single stationary distribution \(\{\pi_{j},j\in I\}\) exists. Theoretically, when \(n\) goes to \(\infty\), the influence among agents converges to the stationary distribution, i.e., \(\lim_{n\rightarrow\infty}w_{ij}^{(n)}=\pi_{j},j\in I\). This observation answers our first research question that the interaction among different agents would converge in the long run in our proposed approach. In the proposed approach, all the agents adopt the homogeneous base prediction model, e.g., LSTM or GRU, so that the computation requirement is the same and fair for different agents who want to be involved. An asynchronous communication requirement among the involved agents is also assumed in the proposed approach, so that the mappings can be exchanged and aggregated. It would be interesting to extend our study from the homogeneous and synchronous case to the heterogeneous or asynchronous cases when different agents have different computation and communication resources. For example, deep learning models with a higher computational burden can be trained on a computation-rich agent. Two deep learning models are adopted in this paper, namely, LSTM and GRU. Based on the gate mechanism, LSTM is proposed to learn the long dependency relationship from the input sequence while mitigating the problem of vanishing gradients. GRU is a simplified variant of LSTM with only two gates, instead of three gates as LSTM, as shown in Figure 3. More specifically, denote \(t\) as the time point, \(x_{t}\) as the input, \(h_{t}\) as the output value, and \(c_{t}\) as the cell state. The three gates used in an LSTM cell can be denoted as follows. The forget gate is as follows:  
\[f_{t}  
The two gates used in a GRU cell can be denoted similarly. The update gate is as follows:  
\[z_{t}=\sigma(W_{xz}x_{t}+W_{hz}h_{t-1}+b_{z}) \tag{5}\]  
And the output gate is as follows:  
\[h_{t}=(1-z_{t})*h_{t-1}+z_{t}*tanh(W_{xo}x_{t}+W_{ho}\sigma(W_{xx}x_{t}+W_{hr}h_ {t-1}+b_{r})*h_{t-1}+b_{o}) \tag{6}\]  
Two widely used evaluation metrics are adopted in this paper, namely, RMSE and MAE.