## 4 Method
### Hierarchical Architecture  
First, we will define the environment. We abstract TSC environment as a gird network consisting of edges and intersections \(\mathcal{G}=(\mathcal{V},\mathcal{E})\), where \(\mathcal{V}\) represents an intersection, and edge \(e=(v_{i},v_{j})\in\mathcal{E}\) represents a road between two intersections \(v_{i},v_{j}\in\mathcal{V}\). In this paper, a lower agent is defined as a traffic light located at a certain intersection. The agent decides to switch the traffic signal by observing the situation at the intersection, which is the direct unit for traffic regulation. We divide the whole environment into multiple disjoint regions \(\mathcal{G}=\mathcal{G}_{1}\times\cdots\times\mathcal{G}_{K}\), each region \(\mathcal{G}_{i}=v_{1},\ldots v_{k}\) contains k signal lights controlled by the agent, and roads connecting each intersection. Through this division of regions, we realize the simplification of the large-scale TSC problem  
In each region of the two-layer architecture, the upper layer is a virtual manager agent, and the lower layer agent is called worker who controls the intersections. In each region, the task of the manager is to assign goals to the various workers under its command based on the traffic conditions in the region. This goal is the manager's desired future state for the worker-controlled intersection. Workers combine environmental states and assigned goals to make decisions and change environmental states by toggling traffic signals. In this way, we divided the entire mission at the strategic and tactical levels.  
If we only consider the situation in one region, the overall traffic situation still cannot be effectively alleviated. Therefore, we need to incorporate consideration of neighbors at both levels. Neighbors of the region are defined as \(\mathcal{N}\), so we contain the manager's neighbor \(\mathcal{N}^{M}\) and worker's neighbor \(\mathcal{N}^{W}\). We established a two-layer communication channel to assist the operation of the layered architecture. Among them, the manager will communicate with the neighbor manager to obtain the traffic situation in the neighbor region. In addition to obtaining the environmental state of the neighbor worker, the worker also obtains the phased goals of the neighbor workers.  
Therefore, the agent-environment interaction process in GMHM is as follows: the manager combines the state of the environment and the states of neighbors to generate goals for workers. Workers combine environment state, neighbor state, self-goal, and neighbor-goal to generate actions and submit them to the environment. The environment executes actions submitted by workers to move to the next state and gives feedback to workers and managers. In this way, after considering the state of neighbors, both manager and worker obtain more sufficient environmental information, and their decision making is more assured. At the same time, it also solves the instability caused by the mutual adaptation of agents in a multi-agent environment. This also ensures that when multiple regions are combined into a larger traffic environment, managers and workers can work with neighbors to resolve traffic jams.