## 5 Simulations  
On the basis of the supply and demand relationship of each energy region, the topology of the EI with multiple ERCs can be transformed into Figure 5. In this case, multiple types of energy are integrated and multi-energy networks with different physical connection structures are connected through ERCs. Energy regions connected with ERCs in this case have diversified energy supply capacities and energy demands, and they possess the ability of supplying energy through ERCs according to their inner supply-demand relationship. Figure 5: Diagram of energy routing design. As can be seen in Figure 5, the proposed energy system includes seven energy regions which are connected through ERCs. Energy regions connected with ERC1, ERC2, and ERC4 produce excessive energy over users' demands which are able to act as energy suppliers for other regions. At the same time, due to the excessive energy demands, energy regions connected with ERC5, ERC6, and ERC7 requires extra energy from the energy internet. Considering the output power fluctuations of renewable energy sources and the variations of user demands, an energy routing design for both electrical power and thermal power can be obtained by using the proposed algorithm in MATLAB. In order to design the energy routing path, the efficiencies of ports in the related routing centers are proposed in Table 3.  
To calculate the power losses on the connection lines, values of resistances and upper limits of power transfer are provided in Table 4 as follows:  
The parameter of operating cost is proposed in Table 5 \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Line** & **Resistance \(R_{i-j}\) (\(\Omega\))** & **Upper Limits \(P_{i-j}\) (kW)** & **Upper Limits \(H_{i-j}\) (kW)** \\ \hline \(l_{1-3}\) & 0.24 & 58.33 & 50 \\ \(l_{2-3}\) & 0.37 & 37.84 & - \\ \(l_{2-4}\) & 0.54 & 25.93 & - \\ \(l_{3-5}\) & 0.41 & 34.15 & 80 \\ \(l_{3-6}\) & - & - & 35 \\ \(l_{4-6}\) & 0.65 & 21.54 & 80 \\ \(l_{5-6}\) & 0.45 & 31.11 & - \\ \(l_{5-7}\) & 0.55 & 25.45 & 60 \\ \(l_{6-7}\) & 0.60 & 23.33 & 70 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Resistances and upper limits of power transfer of connection lines. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Routing Center** & **Port** & **Efficiency \(\eta_{ix}^{P}\)** & **Efficiency \(\eta_{ix}^{H}\)** \\ \hline ERC1 & a & 1 & 1 \\  & b & 0.97 & 0.98 \\  & a & 1 & - \\ ERC2 & b & 0.95 & - \\  & c & 0.95 & - \\  & a & 1 & 1 \\ ERC3 & b & 0.97 & 0.99 \\  & c  
Learning rate and discount factor are two significant parameters which influence the performance of the learning process. A large learning rate shortens the whole learning process by accelerating its transformation towards the newly estimated value. However, it brings risks to the convergence of the algorithm. An excessive large learning rate slows down the convergence process and even results in divergence. On the contrary, a small learning rate ensures the stability of the convergence process but prolongs the learning process evidently. In this paper, the selected learning rate ensures the performance of both the learning speed and convergence process. As for the problem researched in this paper of which adjacent states correlated, the action of the former state significantly affects the actions of the following state, a small discount factor may be harmful. By using a small discount factor, the algorithm risks trapping the local minimum and fails to get the optimal solution. Therefore, a larger discount factor was selected in this paper. The Q learning parameters were selected in Table 7 as follows:  
According to the structure proposed in Figure 5, the electrical power demands and heat demands are shown in Figure 6. \begin{table}
\begin{tabular}{c c c} \hline
**Device** & **Lower Bound (kW)** & **Upper Bound (kW)** \\ \hline GB & 0 & 100 \\ CHP (electric) & 0 & 60 \\ CHP (thermal) & 0 & 80 \\ Photovoltaic (PV)\#1 & 0 & 25 \\ PV\#2 & 0 & 25 \\ \hline \end{tabular}
\end{table} 
Table 6: Capacities of devices.