## 1 Introduction  
Isolated signal control refers to operating traffic signals at an intersection in isolation, to serve local road users without considering vehicle progression and operational performance at adjacent intersections. Signal operations have a distinguishable impact on intersection performance. To achieve better intersection performance, it is strongly preferred to adjust the green times for vehicle phases in real time to meet time-varying vehicle demand. For a long time, as well as real-time coordinated signal control, real-time isolated signal control (RISC) has been a research interest in the field of traffic engineering. Like some control problems in other fields [1], RISC problems involve a kind of time-varying dynamics about the signals and vehicles at an intersection. To solve RISC problems, researchers usually build optimization models or define logic rules, based on their understanding of intersection's dynamics. In the language of statistics, however, the dynamics discovered and explained by researchers are merely the limited samples of the population. Quite often, the underlying reason for unsatisfactory intersection performance is that the optimization models or the logic rules fail to cope well with the dynamics beyond their reach. Reinforcement learning (RL) is a machine learning paradigm for solving sequential decision problems. RL is learning by repeatedly interacting with environment in a trial-and-error manner. Concretely, an RL agent senses the state of its environment, and takes an action to affect the environment; then the environment transitions the state and yields a reward; the agent utilizes the reward to improve its policy, i.e., the mapping from states to actions. Let \(R_{t+1},R_{t+2},R_{t+3},...\) denote the sequence of rewards after time step \(t\). The agent aims to seek an optimal policy \(\pi^{*}\) that maximizes the expected cumulative discounted rewards: \(\pi^{*}=\arg\max_{\pi}\mathbb{E}\big{[}R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+ \ldots\big{]}\), where \(\gamma\in[0,1]\) is the discount rate [2]. It is feasible to apply RL to solve RISC problems. First, green time adjustment is made every a period of time. Current decision affects the situations facing subsequent decisions and the intersection performance in future. Obviously, RISC problems are sequential decision problems. Second, high-resolution vehicle data, e.g., location and speed, become more available as advanced traffic detection systems come into use. Such data can be used to construct the state representation and rewards for RL. Third, traffic simulation tools allow us to reproduce nearly all real-world intersection's dynamics on computers. It is inexpensive and not time-consuming for an RL agent to collect massive samples of intersection's dynamics during training. Given the aforementioned shortcomings of the optimization models and logic rules, it is necessary to apply RL to solve RISC problems. First, an RL agent can learn knowledge, known or unknown to humans, about how to time signals to achieve better intersection performance. Second, the agent can extensively explore the state-action space of RISC problems and try to iteratively improve its policy. This leads to the learned knowledge evolving toward super-human level. In this study, we concern ourselves with four things:  
1. to profile a RISC problem that fully satisfies the signal timing constraints from the engineering perspective;
2. to compare the applicability of RL methods to the RISC problem, and choose a well-suited one;
3. to guide the definition of RL elements with traffic engineering expertise;
4. to improve the chosen RL method in terms of training settings to accommodate the RISC problem. Our efforts in these things induce the proposal of an RL solution, termed Double Deep Dynamic Q-network, i.e., D3 ynQN. The remainder of this paper is organized as follows. In Section 2, we review previous studies and present our research focuses. In Section 3, we profile the single-ring RISC problem.