## 1 Introduction  
Model-free deep reinforcement learning (MFDRL) is used for solving sequential decision problems. MFDRL does not rely on the environmental models pre-determined by humans. Instead, it involves learning by interactive trial and error, along with rewards from the environment. This makes MFDRL especially worth being used for problems characterized by two characteristics: (1) the environment's dynamics cannot be perfectly a priori modeled; and (2) a series of control decisions as a whole can be assessed with a widely accepted performance metric, based on which, the reward for MFDRL can be readily defined [1].  
Real-time traffic signal control (RTSC) at intersections is typically a sequential decision problem suitable for MFDRL. First, the behavior of a vehicle and the reactions of other vehicles to it are both highly stochastic. This complexity complicates the dynamics of intersections, making it difficult for humans to model such dynamics without losing any details. Second, metrics such as vehicle throughput and vehicle delay effectively measure intersection performance over a period of time and can be utilized to define rewards.  
Many previous studies of MFDRL-based RTSC have focused on a road network with multiple intersections. From a traffic engineering perspective, there were still unresolved issues at individual intersections. In particular, signals often could not operate within a dual-ring barrier phase structure (DRBPS). This structure organizes phases in a fixed sequence to meet engineering constraints, while also allowing phases to time independently and flexibly, as indicated in Section 3. DRBPS is complex but remains the most common at real-world intersections [2]. RTSC methods incompatible with DRBPS cannot be widely adopted in urban areas. On the other hand, some studies did not permit second-by-second adjustment of phase green time. Instead, the green time could only be a multiple of a specific number of seconds, which suppressed the potential for performance enhancement of their methods.  
This study proposes an MFDRL method for RTSC at an individual intersection with DRBPS, termed \"multi-dimensional double deep dynamic Q-network with aligned Q-fusion (MD4-AQF)\". We deployed a single agent for the intersection that chose from green time options at one-second intervals for each phase. We address the following three major challenges:  
1. How to define an action and when to select an action for two concurrent phases in different rings. For this challenge, we present an action definition based on \"decision point aligning\" in Section 4.
2. How to effectively learn hundreds of combinatorial actions for two concurrent phases. For this challenge, we present a training algorithm based on a \"multi-dimensional Q-network\" in Section 6.2.
3. How to ensure that two concurrent lagging phases end simultaneously. For this challenge, we present an action selection mechanism based on \"aligned Q-fusion\" in the algorithm in Section 6.3.  
The remainder of this paper is organized as follows. Section 2 reviews previous studies and points out their shortcomings. Section 3 describes the problems faced by us and the agent, and the difficulties in solving them. The main contents from Sections 4-6 and their roles in the reinforcement learning loop are illustrated in Figure 1.