## 3 Learning with Corrupted Data
### Experiment Design
#### 3.1.2 Traffic Signal Control Deep Reinforcement Learning (Signal-RL)  
The second experiment is a deep reinforcement learning (DRL) task. An isolated intersection environment is built using SUMO 1.20.0. The environment is illustrated in Figure 2. The objective is to optimize the traffic signal at this intersection. The intersection consists of four approaches, each with three lanes. The traffic demand is generated using a binomial distribution, and the ratio of left-turn, through, and right-turn traffic demands is 1:3:2. The arrival rates are time varying: East-West traffic demands follow a sine curve, while North-South demands follow a cosine curve within range \([0,\pi/2]\). The simulation step size is one second, and each episode has a horizon \(H\) of one hour. A Deep Q-Network (DQN) model is used to learn the traffic signal control strategy (see List A1 for model and Table A1 for configuration). The state of the environment consists of road occupancy, the current signal phase, and the duration of the current signal phase, resulting in a state vector of dimension 965. A neural network with two hidden layers of sizes 256 and 48 is used to extract features. Layer normalization is applied, but no dropout layers are included. Standard techniques such as double networks and replay buffers are implemented. The action space is discrete, with four possible actions: East-West left turn, East-West through, North-South left turn, and North-South through. Each action in the simulation lasts for 6 seconds. The reward \(r_{t}\) for each time step is defined as the number of queuing vehicles transformed according to Equation (1). The queue length is  
Figure 2: Simulation environment for Signal-RL experiment. The intersection comprises four approaches, each with three lanes. Traffic demand is generated based on a binomial distribution, with a left-turn—through—right-turn ratio of 1:3:2. The environment state is defined by road occupancy. Each simulation step corresponds to one second, and each episode spans one hour. The step-wise reward is defined in Equation (1). divided by 80 to normalize step rewards approximately within the range of 0 to 1. The performance indicator for the model is the episode cumulative reward, \(R\). Although the reward is accumulated over one hour, the process itself is infinite. \[R=\sum_{t=1}^{H}r_{t}=\sum_{t=1}^{H}-\frac{q_{t}-80}{80} \tag{1}\]  
where \(q_{t}\) is the number of queuing vehicles at the intersection at time \(t\); and the threshold for deciding whether a vehicle is stopped is 0.3 m/s. \(H\) is the horizon of one simulation episode. The model is trained using linearly decaying exploration (\(\epsilon\)) and learning rate (\(lr\)) for 80% of the training time, after which the values were fixed. The initial and final \(e\) values were 1.0 and 0.01, respectively, while the initial and final learning rates were \(1\times 10^{-3}\) and \(1\times 10^{-4}\), respectively. The DQN model convergence results are shown in Figure 1b. The overview of model, dataset, and training configurations for both NLP-SL and Signal-RL tasks are shown by Table 1.  
For the Signal-RL task, three types of data corruption are introduced: vehicle missing, inserting noise, and masking region:  
* **Vehicle missing**: Each vehicle is not detected with probability \(p\). This scenario is relevant in Vehicle-to-Everything (V2X) environments, where roadside units detect vehicles' presence through communication channels like DSRC [28]. However, only a proportion of vehicles are equipped with onboard devices. This type of corruption is analogous to the data-missing scenario in the NLP-SL experiment. * **Inserting noise**: Noise is added to the road occupancy state and rewards. Each road cell occupancy state has probability \(p\) of being replaced with random binary value. This scenario is relevant in environments where road occupancies are detected using computer-vision systems, which can introduce errors. * **Masking region**: This special type of corruption is specific to traffic signal settings.