Patent ID: 11862016
Assignee: JIANGSU UNIVERSITY
Field: Control (Instruments)
Classification: CPC G | IPC G

Claim 0:
1. A vehicle-road cooperative control method based on multi-intelligence federated reinforcement learning (FRL) at a complex intersection, comprising the following steps:
step 1. a vehicle-road cooperative framework is constructed in a simulation environment, in the vehicle-road cooperative framework, a road side unit (RSU) static processing module and a vehicle-based dynamic processing module are used to synthesize a cooperative state matrix for reinforcement learning (RL), wherein the RSU comprises a camera, and RSU bird-view information is distinguished into static information (road information, lane information, lane centerline information) and dynamic information (a plurality of intelligent connected vehicles) by using the RSU static processing module, wherein the lane centerline information in the static information is used as a basis for the cooperative state matrix of RL, while the dynamic information is used as a basis for cooperative state matrix cropping, the vehicle-based dynamic processing module is used to crop a static matrix obtained by the RSU static processing module, based on vehicle location information and a coordinate transformation, a cropped 56×56 cooperative state matrix is then used as a sensing area of a single vehicle, covering a physical space of about 14 m×14 m, the dynamic information is stacked in two consecutive frames to obtain more comprehensive dynamic information, the dynamic processing module is used to superimpose the cropped static matrix and the stacked dynamic information to synthesize the cooperative state matrix for a Federated Twin Delayed Deep Deterministic policy gradient (FTD3) algorithm;
step 2. the control method is described as a Markov decision process, the Markov decision process consists of a set of tuples (S, A, P, R, γ) description, wherein:
S denotes a set of state, corresponding to a cooperative state output by the vehicle-road cooperative framework, the cooperative state consists of two-part matrices, first, a cooperative perception matrix obtained by the vehicle-based dynamic processing module, in addition to the static road information, a dynamic vehicle speed, and orientation information, the cooperative perception matrix also includes implicit information, such as vehicle acceleration information, a distance from the lane centerline, a direction of travel and a heading angle deviation, a plurality of convolutional layers and fully connected layers are used to integrate features, second, a current sensor information matrix includes speed information, the orientation information, and the acceleration information obtained and computed by a plurality of vehicle sensors;
A is a set of action, corresponds to a throttle of the vehicle and a steering wheel control quantity;
P denotes a state transition equation P: S×A→P(S), for each state-action pair (s, a)∈S×A, there is a probability distribution p (⋅|s, a) indicating a possibility of entering a new state after an action a is taken under a state s;
R defines a reward function R: S×S×A→R, R (st+1, st, at) denotes a reward obtained after moving from an original state st to a new state st+1, the reward function is used to evaluate the action;
γ represents a discount factor, γ∈[0, 1], used to compute a cumulative reward, η
    ⁡
    (
    
     π
     θ
    
    )
   
   =
   
    
     ∑
     
      i
      =
      0
     
     T
    
    
     
      γ
      i
     
     ⁢
     
      r
      i
     
    
   
  
  ,
 

a solution to the Markov decision process is to find an optimal control strategy π: S→A, to maximizes the cumulative reward π*: =argmaxθη(πθ), the cooperative state matrix obtained by the vehicle-road cooperative framework is used to output the optimal control strategy through the FTD3 algorithm;
step 3. the FTD3 algorithm is built, and the FTD3 algorithm is composed of an RL module and a federated learning (FL) module, the RL module is formed by the set of tuples (S, A, P, R, γ) in the Markov decision process, and the FL module is formed by a network parameter module and an aggregation module;
step 4. interactive training is performed in the simulation environment, a training process includes two stages: an exploration stage and a sample learning stage, in the exploration stage, a strategy noise of the FTD3 algorithm is used to generate a random action, throughout the training process, the cooperative state matrix is captured and synthesized by the vehicle-road cooperation framework, and then the FTD3 algorithm takes the cooperative state matrix as an input and outputs the action with the strategy noise, after the action is executed, a new state matrix is captured by the vehicle-road cooperative framework, and the action is evaluated by a reward function module, the set of tuples consisting of the cooperative state matrices, the action, the new state matrix, and the reward function is an experience, and randomly generated experiences are stored in a replay buffer, wait until the number of experiences meets a certain condition, the training process will enter the sample learning stage, sample from the replay buffer with a minibatch and learn according to a FTD3 network training module, as a learning level increases, the strategy noise is attenuated;
step 5. a plurality of neural network parameters are obtained by the network parameter module in the FL module, and the neural network parameters are uploaded to the aggregation module of the RSU, the aggregation module is used to aggregate a shared model parameter by averaging the neural network parameters uploaded by the network parameter module according to an aggregation interval method, wherein only specific neural networks are selected by the FTD3 algorithm to participate in the aggregation; and
step 6. by using the network parameter module in the FL module, the aggregated shared model parameter is distributed to the intelligent connected vehicles for local update, the training process loops until the network converges.