## 1 Introduction  
With the aggravation of the shortage of fossil fuels and the growing concerns over environmental pollution, current power grids are caught in a dilemma between the increasing power demand of users and environmental protection. What is more, users' demands for energy tend to be diversified and the efficiency of multiple types of energy needs to be improved. Consequently, the concept of energy internet (EI) is proposed as a feasible way to solve the existing problems [1; 2]. Energy internet can be assumed as a multi-energy coupled network with high permeability of renewable energy resources. In addition, to promote efficiency, a distributed energy supply structure is substituted for a traditional centralized generation structure in the EI [3]. During the pursuit of an economic and environmentally-friendly energy system, energy management problems were researched. Energy management problems are usually concluded as an optimization problem with one or more objective functions and several constraints. Many research works have been conducted under the background of EI. Operating cost is one of most common objective functions used in energy management problems [4; 5]. However, with the growth of attention on environmental protection, multi-objective energy management which considers carbon emissions and operational cost simultaneously has been studied in many research works [6; 7]. In order to fit the decentralized structure and realize flexible energy transmission, the concept of an energy router was first introduced in the smart grid scenario [8], where the structure of the energy router as proposed and its functions were discussed. As the key device of EI, various types of studies have been conducted based on energy routers, such as optimal location selection [9; 10], stability control [11], and structural design [12]. There are a few studies conducting research on energy-router-based energy management. In Reference [13], solid-state transformer was used as an energy router, and an optimal economic energy-management-based energy routing strategy, which reduced consumption of grid power, was proposed. In Reference [14], a local area energy network containing several energy routers was proposed, and the energy routing algorithm was formulated to find the routing path with the lowest power loss. However, the study only considered the routing management for electrical energy, and other types of energy were not considered. Thus, this paper aims to consider various types of energy coupled with EI, such that an energy-management-based energy routing design for multiple types of energy is studied. Reinforcement learning as a main class of machine learning has been adopted in various fields of energy systems including optimal control [15; 16; 17], fault diagnosis [18], reactive power control and optimization [19; 20], etc. When it comes to the issue of energy management, as for complex energy networks with high penetration of renewable energy, challenges brought by its randomness affect the optimal energy management. Due to the fluctuant characteristic of renewable energy sources, the existing literature makes forecasts of loads and generation units based on long-term data during energy management processes. However, acquiring accurate a priori information of generation devices and loads is not straightforward and restricts its applications. Besides, forecasts based on a large amount of data brings about an increase on the computational burden. In contrast, reinforcement learning method shows high efficiency in solving such problems due to its features being model-free. Reinforcement learning does not rely on a priori knowledge, which increases the flexibility of its application on energy management in complex energy systems. Therefore, reinforcement learning has been adapted to energy management in recent years. A dynamic energy management system for microgrids has been established in Reference [21] and reinforcement learning method has been used to realize optimal control of the whole system.