Socially-Attentive Policy Optimization in Multi-Agent Self-Driving System
Authors: Zipeng Dai
Tianze Zhou
Kun Shao
David Henry Mguni
Bin Wang
Jianye HAO
Venue: CoRL 2022 Poster (2022)
Keywords: self-driving
social preference
multi-agent
reinforcement learning

Abstract: As increasing numbers of autonomous vehicles (AVs) are being deployed, it is important to construct a multi-agent self-driving (MASD) system for navigating traffic flows of AVs. In an MASD system, AVs not only navigate themselves to pursue their own goals, but also interact with each other to prevent congestion or collision, especially in scenarios like intersection or lane merging. Multi-agent reinforcement learning (MARL) provides an appealing alternative to generate safe and efficient actions for multiple AVs. However, current MARL methods are limited to describe scenarios where agents interact in either a cooperative of competitive fashion within one episode. Ordinarily, the agents' objectives are defined with a global or team reward function, which fail to deal with the dynamic social preferences (SPs) and mixed motives like human drivers in traffic interactions. To this end, we propose a novel MARL method called Socially-Attentive Policy Optimization (SAPO), which incorporates: (a) a self-attention module to select the most interactive traffic participant for each AV, and (b) a social-aware integration mechanism to integrate objectives of interacting AVs by estimating the dynamic social preferences from their observations. SAPO solves the problem of how to improve the safety and efficiency of MASD systems, by enabling AVs to learn socially-compatible behaviors. Simulation experiments show that SAPO can successfully capture and utilize the variation of the SPs of AVs to achieve superior performance, compared with baselines in MASD scenarios.

Socially-Attentive Policy Optimization in Multi-Agent Self-Driving System
Zipeng Dail., Tianze Zhou Kun Shao²t, David Henry Mguni', Bin Wang", Jianye 2
Beijing Institute of Technology, HHuawei Noah's Ark Lab, 'Tianjin University
Abstract: As increasing numbers autonomous vehicles (AVs) are being de- ployed, it is important to construct multi-agent self-driving (MASD) system for navigating traffic flows of AVs. To prevent congestion or collision, especially in scenarios like intersection or lane merging, an AV should not only pursue the individual goal but also interact with other traffic participants. Inspired by the mixed motives like human drivers, rent interacting vehicles in MASD systems, and then model the coordination or conflicts of their goals, namely their social preferences (SPs). To this end, we pro- pose novel reinforcement learning method for each AV in decentralized control, called Socially-Attentive Policy Optimization (SAPO), which incorporates: (a) self-attention module to select the most interactive traffic participant, and (b) a social-aware integration mechanism to integrate objectives of the ego AV and its interacting partner, by updating their SPs from the current observations. SAPO solves the decision-making problem of how improve the safety and efficiency of MASD systems, without utilizing any global information for centralized con- trol. Simulation experiments show that SAPO can successfully capture and utilize the variation of the SPs of AVs to achieve superior performance.
mainly focus how to figure out the cur-
Keywords: self-driving, social preference, multi-agent, reinforcement learning
Introduction
With the development of self-driving technologies, increasing numbers of autonomous vehicles (AVs) are being deployed on public roads, and sharing the space with other traffic participants. Different from the single-agent platforms [1] built for realism in rendering single AV under differ- ent road and weather conditions, the Multi-Agent Self-Driving (MASD) system [2, 3, 4] is designed to navigate traffic flows of multiple AVs, where each AV needs to not only pursue the individual goal but also interact with other traffic participants in vehicle interaction events (e.g.. intersection and lane merging). As with human drivers and pedestrians who usually show respect to others, is important for MASD systems to model the coordination or conflicts of the AV's goals, namely the social preferences (SPs). Fig. I shows the illustrative examples where interacting AVs behave differently in different SPs. Failing to incorporate an awareness of other agents' SPs can lead to increased congestion as well as more collisions between vehicles.
Enabling SPs in MASD systems produces series of challenges: First, for the efficiency and ac- curacy of modeling SPs, it is necessary to determine which ranges or groups of AVs are actually interacting. For example, as shown in Fig. 1, interaction events occur in the red zone when AVs are crossing the intersection together. Second, as time goes by, the of an AV usually varies because it may play various roles when interacting with other AVs in different kinds or driving styles.
To this end, we aim to capture the most likely object which each AV interacts with, and then ex- tract their SPs to improve the efficiency of MASD control algorithms. Existing motion planning models [5, 6] and safe control methods [7, 8] only focus on the physical states and subjective ob- servations of the ego car, but fail to incorporate SPs for decision-making purposes. Multi-agent
Work done as an intern at Huawei. Corresponding author: shaokun2 @ hwa.ei.comm
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

(3)
[b)
Aiyadio
Art
Red Pottie
Defsult Zone
Interaction Zone
Figure Examples of an MASD system, where AVs are in different social preferences.
reinforcement learning (MARL) is becoming increasingly popular for several sequential decision- making scenarios [9, 10] with multiple controllable agents. Exemplar methods like QMIX [11] and MADDPG [9] are widely used in MASD) systems with multiple controllable agents in fully coop- erative or fully competitive settings, by defining a shared team reward or individual payoffs as the objectives. However, the SPs of these agents fixed, which are far from the real-world settings.
Contributions. Our main contribution a novel MARL method called Socially-Attentiv Policy Optimization (SAPO) for independent AV control, which solves the problem of how to improve the safety and efficiency of an MASD system with the dynamic modeling SPs. This method consists of two key components, (i) self-attention module select the most interactive participant for each AV, which extracts the critical entities vehicle interaction events, (ii) social-aware integration mechanism to integrate objectives of the ego AV and its interacting partner, by updating their SPs from the current observations. Based on a simulation platform called SMARTS [4], we validate SAPO in several MASD) scenarios, and then we showease how SAPO navigates each AV to select the interacting object and estimate the SPs in real time, which current MARL methods cannot handle.
Related Work
2.1 Deep Reinforcement Learning in MASD Systems
In recent research literature, MARL widely used to navigate AVs in MASD systems. Wu er al. in [12] proposed MARL-bbased velocity maximization framework by effectively leveraging the structure of the human driving behavior to form an efficient vehicle spacing for an intersection net- work. Isele al. in [13] designed several exploratory actions for MARL optimize the acceleration profile while safely avoiding collisions. Yang et al. in [14] proposed a credit-assinmment method for interactions between actions and goals of different AVs. Peng al. in [15] considered the fact that each individual agent is mostly affected by its nearby agents, and facilitate the coordination of AVs at both local and global levels in MASD systems. However, most of the existing methods only focus on how to achieve safe responses to sudden actions of the surrounding entites, or just assume that each AV has fixed SP in episode [15], which are limited to apply general cases.
2.2 Social Preference in Vehicle Interaction
SP is increasingly discussed to better formulate interaction between AVs in self-driving scenarios. The authors in [16, 17] proposed probabilistic frameworks that efficiently learned categorical latent variables to jointly model the multi-step future motions of agents in a scene. Sekhon er al. in [18] proposed a novel spatial attention mechanism encode the influence of spatially close neighbors in manner. Apart from these powerful works about trajectory prediction, several decision-making works also start consider how to utilize SP for exploring better control policies. Wang el al. in [19] established an online algorithm to estimate social-aware reward functions, by inferring three kind of characteristics. including egoism, courtesy and confidence. Schwarting et al. in [20] formulated interactions between AVs as best-response (BR) game, and then they proposed metric called Social Value Orientation (SVO) which quantifies the continuous degree of an agent's selfishness or altruism in reward definition. This approach allows AVs to observe other traffic participants, estimate their SVOs, and generate an autonomous control policy real time. Furthermore, estimating SVOs at consecutive timesteps can better predict how AVs will interact with others, which also inspires several solutions for several sequential social dilemma (SSD) games [21, 22). However, existing methods focus on all of traffic participants simultaneously, which may bring increasing challenges as deploying more vehicles. Thus, this paper, we consider how filter redundant information by selecting the most interactive traffic participant as the interacting object for each AV.

---

3 Preliminaries
3.1 Dec-POMDP
We consider an MASD system where M independent AVs aim to reach their given destinations quickly and safely. To apply MARL,, formulate this system decentralized partially observable Markov decision process (Dec-POMDP [23]), represented by a tuple (M,S, S, O A. R. ?), where M, S. O, A and II are the set of AVs, states, local observations, actions and policies, and E [0, 1J is the discount factor. The system works as follows, episodically. At the beginning of an episode, the global state is initialized as 80. At each timestep I, each AV m .M has its own observation o of state 8t, including kinematic information of itself and other observable AVs. Then, it decides an action at" sampled from its policy a; (- After the environment receives all AVs' joint action (i.e.. a = {aī"" 1.M Im==)). each AV m receives the individual extrinsic reward r RT .q."). which evaluates its performance at timestep t; followed by transiting to the next state St+1, based on the state transition distribution Pr(#++1 81, Qz). The episode will be terminated if: (a) all of the AVs have arrived at their destinations, (b) some of them collide with each others. For each AV m, the optimization problem is to find an optimal policy Tr E IT to maximize the accumulated discounted reward, as:
E E 7' r (o a o O, a. E A, Thh ET.
max T
(1)
Note that each AV m .M optimizes its policy nhh independently in our formulated Dec-POMDP.
3.2 Independent Control by Proximal Policy Optimization
To solve this Dec-POMDP, a simple approach is to allow each AV to maximize its individual ob- jective independently, as in the single-agent problems. The difference is that each AV only utilizes its observation of instead of the full state 8, Extended by the policy-based method PPO [24), the value function and advantage function of AV at timestep & can be computed as:
V V m (o,
= E
(2)
(o, a.") = th + > V
ViTL
(3)
Then, by adopting a truncated importance sampling factor of PPO is written as:
Tue [a lo") old (a jo
the surrogate objective
(TT) Elmin(pA'., (atnipppppepepeeee 1 1- +)
where clip function helps restrict the incentive of optimizing xt towards r within the interval [1-e, 1 +e]. Note that e is a real number and specify the interval. The value function / is updated by mean squared error loss function:
c.
Elr"" + 7V"" V (o")))²
(5)
4 Methodology
We propose a novel MARL algorithm called Socially-Attentive Policy Optimization (SAPO) to achieve better cooperative patterns in MASD systems. As shown in Fig. 2, SAPO uses PPO as the start point of the design, and consists of two improvements: self-attention module called "Interactive Attention" to select the most interactive participant for each AV from complex traffic flows (see Section 4.1), and a Social-aware Integration mechanism to model the impacts of SPs in vehicle interaction (see Section 4.2). For simplicity, we omit the timestep index in this section. The whole structure is depicted by the following equations (given Vm E M):
(p, h²mh = InteractivvAAtlentinnn"""""" SPP Integraaioonr".. r" hm)
P P M'm,
(6a)
(6b)
m n (a (a" "hmh = ( "
(6c)
"O" hmm = (O", (o'",h" hm),
(6d)
3

---

Social-Attentive Palicy Optimization (SAPO)
hr"
Add & Normaizee
Interactive Attention (mutt-head)
concatt
Q
[q"']
cenaelaol
features (AV m)
vacioo
sp
F
fe
Gumbel
focus the AV
Safamak
spitt
soroposs
qt
V
Jely s11
Ahruistic
other feetures
SP
(observable
tt
Independent Control of the
MASD system
Ege Interactive
AV
AV I
Unobservable Zone
Observable Zone
Other junsbservadee
Figure 2: Architecture of SAPO.
where p is the index of the traffic participant which is most likely to interact with AV TSP rep- resents the SP-aware reward integrated from interacting AVs; 1 IV are multi-layer perceptrons (MLPs). For each m, SAPO is running as follows. First, SAPO utilizes Interactive Attention to select the most interactive participant and generate compact feature hm E R (see Eqn. (6a))- Then, the individual reward nm is replaced by an SP-aware reward 7m through Social-aware Inte- gration mechanism (see Eqn. (6b)). Next, through f*(()) and f"(.), the compact feature is mapped into an action distribution xth and a value function V respectively (see Eqn. (6c) and Eqn. (6d)). Finally, similar to PPO, the advantage function AV m computed by Eqn. (3), for PPO updates by Eqn. (4) and Eqn. (5). Other Details of running SAPO are deferred in Appendix A.
4.1 Interactive Attention for Selecting Interacting Objects in Traffic Flows
Since there are usually numerous AVs in MASD system, it is necessary for each AV to filter the observed information and consider what is relevant for decision. We focus the fact that each AV is mostly affected by its nearby agents rather than focus on all of the others simultaneously [25]. That is, each AV should pay attention the most interactive vehicle that close conflict with the planned route. Inspired by [26], we propose a self-attention architecture called "interaction attention". It is used both select the most interactive participant and generate a compact feature.
As shown in Fig. 2, for each AV m, SAPO splits o" into the observation of the ego vehicle and the others at first, which are embedded into M vectors of size d by a linear encoder f", Note that each AV has its own observable zone and the feature vectors of AVs outside this zone will be set to zeros. Through MLP f?, f and f", all of the M vectors values (v) of size dg. They are then fed to the multi-hean attention layer [27]. composed of several single query features of the ego AV, which is compared to all of keys K [ A E I xd, containing observed features of each AV. The similarity between the query q" and any AV's key is accessed by their dot product qdh k and then scaled by ___ Next, in order get the most interactive Via
embedded into the queries (q), keys (k) and
heads stacked together. The difference is that only
[q"h Rixd= by
participant the interacting object, we transform the attention weights into the one-hot vector by utilizing a Gumbel Softmax [28] function o--). With this hard attention scheme, the index of the only non-zero element corresponds to the index of an interactive AV. We utilize the one-hot vector to gather all of the values V = .o...... M E TR xxdx and get the output. Overall, the attention computation for each head can be written as:
QK
output = a Vox
V.
(7)
attention

---

Finally, the outputs from all heads are combined, generating compact features. Without loss of gen- eralization, we get the final social-aware feature hT IF""" by adding resulting tensor the ego encoded observation of AV as in residual networks. Similarly, we also combine the attention ma- trix of all heads to get the most interactive participant p which the AV m should focus on temporally. The advantages of the Interaction Attention are: (a) inputs can have variable size; and (b) outputs are permutation invariant which do not depend on the ordering of surrounding AVs.
4.2 Social-Aware Integration of Individual Objectives with Social Preferences
In an MASD system, each AV has its individual objective. However, if we simply maximize each individual reward by PPO, the system will have sub-optimal solutions where, for example, AVs will become aggressive and egocentric, jeopardizing the performance. On the contrary. if we apply fully cooperative learning schemes [11, 29] which consider the summation of individual reward as the joint objective, the trained agents will exhibit unreasonable behaviors such as sacrificing oneself to improve group reward, which not expected MASD systems. To find a balance, we consider utilizing the impacts of SP on learning reasonable behaviours. Thus, inspired by social psychology, we propose a social-aware integration mechanism to integrate objectives of interacting AVs by their SPs. Different from the previous Local Coordination Factor [15] which is fixed at each timestep and adjusted by the total reward of each active agent episodically, SAPO only utilizes the AV m's observation 6" to update SP p at each timestep t, without any centralized training schemes.
For each AV m. the mechanism works as follows, as shown in the right part of Fig, Since social- aware features ht are extracted by Interactive Attention, an MLP fSr is then utilized to generate the ring measure of SVO [20] to represent the AV m's SP yot.. as:
(TTh h h²) = SSP (o h"::; 6 O x R" + 10, x/2].
(8)
Next, we utilize Tdhh to control the appropriate level of coordination between the AV m and its interactive participant p. by calculating social-aware reward r as:
rSP Th cos 4dt (o" h=) + sin (o") hmm
which replaces the AV m's 's individual reward rm as a new objective. Different from automatic search of the optimal SP fixed in episode [15], SAPO can estimate oet each timestep and make sure that Sp is corresponding to the current traffic conditions, which is important in real scenarios.
5 Experiments
In this section, we test the ability of SAPO to learn reasonable behaviors and explore better policy, We first study didactic examples inspired by SSD games [30] to investigate the effects of Interactive Attention and Social-aware Integration on the learning process (see Section 5.1). To demonstrate
scalability on complex MASD systems, we also evaluate the performance of SAPO on a range of SMARTS benchmark tasks [4] (see Section 5.2). Furthermore, we also discuss the applicability of SAPO in real-world autonomous driving system based on open datasets like Waymo [31] and Argo- verse [32] (see Appendix D). Through the distributed training architecture supported by RLlib [33], we set different random seed in each actor to adapt diverse set of environmental settings. Then, we evaluate the trained policies in 10 random seeds and get the maximum, mean and minimum results to visualize the results in all experiments. The detailed settings are introduced in Appendix E.
5.1 Toy Example: Bottleneck
Inspired by [22, 30], we design a toy example of MASD systems, called Bottleneck. Consider grid world with 4 AVs (see Fig. 3(a)), 5 available actions (i.c., stop or move left, right, up and down) and observation space containing 2-D position of ego AV and the others. Each AV's reward is determined by the distance to its own destination and the time costs. Given the initial positions and destinations, AVs should learn to make ways for others and reach the goals in orders spontaneously. Since the solution space is limited, we can easily explore the overall optimal policy and examine the exploitation of MARL. From Fig 3(b), we observe that SAPO trains much faster than PPO and achieves the optimality. This is because AVs only see others as parts of the environment. They are likely to learn selfish behaviours (e.g., keeping closer to the goal without yielding any space) and resulting in congestion, which need to consume more time on exploring the optimal policy. Besides,
5

---

Duthmacc
10ta ander inderthaan
A
SAPOWN HArrccokkt
Indisitad
120
(a) Bottleneck
(b) Learning curves of different methods
Figure 3: (a) A special toy example called Bottleneck, and each AV aims to reach the destination point with the same color as soon possible. (b) The learning curves of SAPO and other baselines.
(a) Cross-road
(b) T-junction
(c) Lane Merging
(d) Roundabout
(c) On-ramp
Figure 4: Examples of MASD scenarios in SMARTS [4].
learning process will much slower without Interactive Attention. This is because hard to maintain the efficiency and accuracy of estimating SPs when considering all of AVs together.
5.2 SMARTS
To tackle settings that more closely represent real world settings, we focus on MASD scenarios based on the Scalable Multi-Agent RI. Training School (SMARTS) benchmark. Fig. 4 describes the examples of tasks we use in our experiments. The experimental settings are deferred to Appendix B. To demonstrate the overall performance of each algorithm, we utilize the average success rate as the evaluation metric, calculated by the average ratio of the number of vehicles that reach their goals over the total number of vehicles in one episode.
5.2.1 Ablation Study
To show the importance of utilizing Interactive Attention and modeling SPs on optimizing MASD systems, perform the ablation study by: (a) removing Interactive Attention (i.e, calculate r" by the average reward of every surrounding vehicle) or using other mechanisms to select p (i.c., replace Interactive Attention by: random attention rule of selecting the nearest vehicle), and (b) using given SPs instead of dynamic estimation for training SAPO in a cross-road scenario. Results are shown in Fig. 5. We that the complete version SAPO has higher success rate than SAPO w/o InteractiveAttt. This because Interactive Attention can filter information from a number of AVs in the intersection, which accelerates the training process. Besides, there is intuitive finding that the rule-based attention is powerful baseline in MASD systems, which is corresponding to some conventions of human behaviors On the other hand, we that SAPO (5 30%) performs than SAPO (s 45%) and SAPO (4p 60%). This is because SAPO (ip = 30%) encourages more self-interested AVs (according to Eqn. (9)), which usually get blocked as they refuse to make way for others. However, even if we set ;p 45" or = 60" to keep the AVs prosocial or altruistic, SAPO still trapped into another local optimal. We observe that only the dynamic modeling of SPs can navigate AVs to learn reasonable social-aware behaviours, which improves the overall performance of the MASD system.
5.2.2 Illustrative Visualization of the Trained Behaviors
In Fig. 6, we show the visualization results of AV'' positions, attention weights and SPs (from the bottom to the top) at each timestep in cross-road scenario, where each AV follows its own waypoints, navigated by SAPO independently. To predict SP smoothly, we discretize yr at each timestep, as: E {0', 15 30% 45", 60", 75 .900 To better explain the learned social-aware

---

(b) Imports of using intrrdtvee Mttntoon
(8) imports usiiigg dynam ofied BPs
1.5
Peo0gg By005s Futo
SAPO 30")
5AP0 45'1
Training Steps
Training Stope
Figure 5: Ablation study.
m
25
75
100
150
375
Index
XU-2
AU-O
AV-3
RV-33
A0-3
AU-O
Figure 6: Illustrative visualization of AV's behaviours trained by SAPO.
behaviors, we focus on three stages in an episode. In stage a + b, AV-0 and AV-3 slow down and make way for AV-1 and AV-2 cross the intersection. Then, in stage - c, AV-0 speeds up while AV-3 stops and waits. Finally, AV-3 speeds up and crosses the road in stage - d. From the learned SPs of each AV, we observe that AVs with higher SPs can learn make way for AVs with lower SPs. Furthermore, we observe that Interaction Attention is helpful by selecting the most relevant traffic participant. For example, when AV-0 and AV-3 both slow down, AV-1 and AV-2 only pay attention to each other when crossing the intersection. These results indicate that AVs can choose suitable vehicles as interacting objects and then learn reasonable behaviors.
5.2.3 Comparison Results of Training in MASD Scenarios
We evaluate SAPO in 6 popular MASD tasks. Other than PPO, we further compare SAPO against three popular baselines in self-driving scenarios as follows:
Iterative Best Response with SVO [20]. As popular game-theoretic approach for self- driving, it models the interactions between AVs multi-player BR game wherein each AV negotiates to maximize its own reward and predict other AVs' behaviors by estimating their SVOs in real time. The details of BR and SVO estimation are deferred to Appendix C.
Two-player Best Response [34]. It trains the Interactive Attention to select the interacting object for each AV, and then finds the Nash equilibrium in two-player BR games.
Social Attention [26]. Based on the centralized control of multiple AVs, it directly utilizes a soft-attention-bsaee architecture for decision-mald makinn involving social interactions. That is, it focus on all of surrounding vehicles with different attention weights. To remove the impacts of the control method, we also consider the variant of decentralized control, where each AV maintains its own attention and PPO framework independently.
7

---

SIPO Grose-madivv0
2V0 Mergevo
Cross-rodiww
aga Plate
Therinng Stepe
Tiening Shepe (+) Roundsoout-00
Twaning Steps
Trunddoowo
P) Rampee
Sarcoss
Sarccers
Traning SNox
Traning Stace
Figure 7: Training curves of SAPO and baselines in SMARTS.
CoPO [15]. It proposes a state-of-the-art MARL method to navigate traffic AVs in MASD systems, by updating PPO policies to maximize a social-aware reward like Eqn. (9). The difference that it allows each AV to focus on all of its surrounding vehicles and utilize the average reward of them r", Besides, it fixes &p at each timestep and adjusts it episodically by optimizing the total reward of each active agent in the simulation. That is, CoPO is mainly used for centralized training and decentralized execution of multiple AVs.
The learning curves are shown in Fig. 7. Note that we use the testing performance of Iterative BR with SVO [20] reference line, since it is an online optimization-ben based method. We observe that, SAPO significantly outperforms baselines in most of the cases and PPO performs the worst. With attention-basse feature extraction, Two-player BR and Social Attention both achieve reasonable performance, especially in lane merging scenarios where two AVs can choose when to merge so their conflicts are less than other scenarios. However, in two cross road scenarios where AVs need to deal with more confficts of individual goals, the performance gap between SAPO and other baselines becomes larger. This indicates that Social-aware Integration is important in learning reasonable behaviors and improving the efficiency of MASD systems. CoPO cannot perform very well because the coordination conflicts of the AVs' goals seldom keep the same whole episode, Besides, utilize the global reward and focus on all of agents together, which will bring redundant information.
6 Conclusion
Inspired by human cooperation societies, this paper proposes a novel reinforcement learning method called SAPO to learn social-aware controller of MASD systems, which incorporates a self- attention module and social-aware integration mechanism for selecting interacting objects and inte- grating individual objectives of AVs, respectively. Experiments on basic traffic simulation show that SAPO can successfully estimate the SPs from AVs' observations and learns social-aware behaviours, which achieves better performance than baselines in several popular MASD systems.
7 Limitations and Future Research Directions
This paper has several limitations. First, we focus on decision-making problems, so we simplify the perception of AVs and assume the observation is based on accurate sensory data, which is not easy to achieve in reality. Second, we only utilize the current observations to predict SPs and make decisions, without referring some trajectory prediction works [16, 17, 18] which make efficient use of historical trajectory data. Third, we only maintain the trial-and-error process in diverse envi- ronmental settings, by training within multiple random seeds together. There should be some better ways which achieve zero-shot transfer to unseen partners without using too much experience data. Finally, we adopt i [0. π in this paper and there should be more diverse scenarios (e.g., car racing) where SP can be defined beyond [0, . /2]. How to tackle with these limitations is promising research direction.

---

Acknowledgments
The authors would like to thank the anonymous reviewers for their helpful comments and thank the many Huawei colleagues in China and UK.
References
[1] A. Dosovitskiy, G. Ros, F. Codevilla, A. M. López, and V. Koltun. CARLA: an open urban driving simulator. In CoRL' 17, volume 78, pages 1-16, 2017.
[2] Á. López, M. Behrisch, L. Bieker-Walk J. Erdmann, Y. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. WieBner. Microscopic traffic simulation using SUMO. In itsc' 18, pages 2575-2582, 2018.
[3] Q. Li, Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[4] M. Zhou, J. Luo, J. Villella, Y. Yang, et al. Smarts: An open-source scalable multi-agent training school for autonomous driving. In CoRL'21, volume 155, pages 264-285,, 2021.
[5] D. González, J. Pérez, V. M. Montero, and F. Nashashibi. A review of motion planning tech- niques for automated vehicles. IEEE Trans. Intell. Transp. Syst., 17(4): 2016.
[6] B. Paden, M. Cáp, S. Z. Yong, D. S. Yershov, and E. Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans. Intell. Veh., 1(1):33-55, 2016.
[7] S. Shalev-Shwarzz, S. Shammah, and A. Shashua. On formal model of safe and scalable self-driving cars. CoRR, abs// 708.06374, 2017.
[8] G. Williams, B. Goldfain, P. Drews, K. Saigol, J. M. Rehg, and E. A. Theodorou. Robust sam- pling based model predictive control with sparse objective information. In Robotics: Science and Systems XIV, 2018, 2018.
[9] R. Lowe, Y. I. Wu, A. Tamar, et al. Multi-agent actor-critic for mixed cooperative-compertiie environments. NeurlPS' /7, 30, 2017.
[10] M. Samvelyan, T. Rashid, C. S. De Witt, et al. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902.04043, 2019.
[11] T. Rashid, M. Samvelyan, C. Schroeder, et al. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML'18, pages 4295-4304, 2018.
[12] C. Wu, A. Kreidich, E. Vinitsky, and A. M. Bayen. Emergent behaviors in mixed-autonomy traffic. In CoRL'17, volume 78, pages 398-407, 2017.
[13] D. Isele, Rahimi, A. Cosgun, K. Subramanian, and K. Fujimura. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. In IEEE ICRA' 18, pages 2034 2034-2039, 2018.
[14] J. Yang, A. Nakhaei, D. Isele, K. Fujimura, and H. Zha. CM3: cooperative multi-goal multi- stage multi-agent reinforcement learning. In ICLR'20, 220 2020.
[15] Peng, Q. Li, K. Hui, C. Liu, and Zhou. Learning to simulate self driven particles system with coordinated policy optimization. In NewrlPS pages 10784-10797, 2021.
[16] Y. C. Tang and R. Salakhutdinov. Multiple futures prediction. In Neurips' 19, pages 15398- 15408, 2019.
[17] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++:: Dynamically-fesiible trajectory forecasting with heterogeneous data. In ECCV'20, volume 12363, pages 683-700, 2020.

---

[18] J. Sekhon and C. H. Fleming. SCAN: A spatial context attentive network for joint multi-agent intent prediction. In AAAI221, pages 6119-61611111 6127, 2021.
[19] L. Wang, L. Sun, M. Tomizuka, and W. Zhan. Socially-compatibl behavior design of au- tonomous vehicles with verification on real human data. IEEE Robotics Autom. Lett., 6(2): 3421-3428, 2021.
[20] W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus. Social behavior for autonomous vehicles. Proc. Natl. Acad. Sci. USA, 116(50) 24978, 2019.
[21] B. Baker, Emergent reciprocity and team formation from randomized uncertain social prefer- ences. In NeurlPS'20. 20, 2020.
[22] K. R. McKee, I. M. Gemp, B. McWilliams, E. A. Duéfiez-Gzzmán E. Hughes, and J. Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. In AAMAS'20, pages 869-877, 2020.
[23] F. A. Olichock and C. Amato. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.
[24] Schulman, Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. arXiv preprint arXiv: 06347, 2017.
[25] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and Wang. Mean field multi-agent reinforce- ment learning. volume 80, pages 5567-5576, 2018.
[26] F. Leurent and J. Mercat. Social attention for autonomous decision-making in dense traffic. CoRR, abs/1911. 11.12200,, 2019.
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In NeurlPS 17, pages 5998 1088-608,,700 2017.
[28] E. Jang, S. Gu, and B. Poole. Categorical reparameterizzation with gumbel-softtax. In ICLR'I7 17. OpenReview.net, 2017.
[29] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. F. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, Z. Leibo, K. Tuyls, and T. Graepel. Value-decomposition networks for cooper- ative multi-agent learning based on team reward. In AAMAS' 18, pages 2085-2087 2018.
[30] J. Z. Leibo, V. F. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In ACM AAMAS' 17, pages 464 473, 2017.
[31] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. R. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and D. Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In ICCV221, pages 1690-9699 IEEE, 2021.
[32] M. Chang. J. Lambert, Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Car,, S. Lucey, D. Ramanan, and J. Hays. Argoverse: 3d tracking and forecasting with rich maps. In CVPR' 19, pages 8748-8757, 2019.
[33] E. Liing, R. Liaw, R. Nishihara, Moritz, R. Fox, K. Goldberg, J. Gonzalez, M. I. Jordan, and I. Stoica. Rllib: Abstractions for distributed reinforcement learning. In (CML'18, volume 80, pages 3059-3068, 2018.
[34] Başar and G. J. Olsder. Dynamic noncooperative game theory. SIAM, 1998.
10


Reviews:
[{"review_id": "U-MNAOIsbIy", "reviewer": "Reviewer_zrSF", "summary": "A method called Socially-Attentive Policy Optimization (SAPO) for individual, coordinated AV control in multi-agent driving environments, aiming to learn socially compliant behaviors between agents with different goals. Contributes two novel parts:\n1. a self-attention module to select the most interactive participant (other AV) for each AV in the scene\n2. a \"social-aware\" method to balance individual rewards (social coordination) as a balance between being purely egotistical (only ego agent reward matters) and purely altruistic (only other agent reward matters)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n - Learning 5x faster in some cases (Fig 3b), and otherwise better performance in SMARTS simulation\n - Interpretability of \"most interactive participant\" for each vehicle. Also the output \\psi having the semantic meaning to help interpret the level of egotistical-vs-altruistic for each vehicle.\n\nWeaknesses:\n - Over-complicated solution that is self-limiting. Why select only one other vehicle, to model interaction with, why not all, like [30] was already doing. Lines 154-155 describe limiting something that could have been more general, into something that is now more complicated *and* more limited in terms of only considering *one* other vehicle.\n - Unclear how it would actually work on a real robotics system. Difficult to understand this without a real platform or real data (e.g. Waymo or Argo open datasets). While the SMARTS benchmark tests high level coordination, the robots are treated as \"self driven particles\" (line 27), which makes me wonder how appropriate this work is for a *robotics* conference.\n - Initially difficult to understand the method: much abstract language initially discussing \"social coordination\", defining these terms unambiguiously too late. I think for a technical paper like this: get to the point faster. There's many terms like this including \"social preferences\" and \"socially-compatible behaviors\". The language of \"dynamic social preferences\" can mean many things, but seems a rather grandiose way to describe what's really happening, which is a weighted sum to two rewards that are only a function of the current observations (ignoring history).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'd choose \"borderline\" if CoRL had such a recommendation, but without it: Weak Reject. See Weaknesses and Issues sections.\n\n----\n\nupdate: I thank the authors for their responses. Apologies I did not get to this sooner. But your responses have addressed most of my concerns. I have thus updated my score to accept", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "## Larger points:\n - section 2: what about other works like Multiple Futures Prediction and Trajectron? Multi-agent autoregressive method with categorical latent variables can encode different types of behaviors or preferences too, and are (in addition) functions of the full history of past behaviors, while still reacting to latest observation (compared to this work which only looks at latest observation).\n - equation 6c, 6d: should these be function of p?\n - line 171: \"unreasonable behaviors such as sacrificing oneself to improve group reward\", why is this bad? Isn't this just \\phi=\\pi/4 in equation 9? Especially when lines 245-246 say that summed group reward is the evaluation criteria anyway.\n - section 5.1: while SAPO trains 5x faster than IPPO, how do we know this isn't just poor hyperparameter choice for IPPO? It asymtotes the same. Also, what are the uncertainty regions on Figure 3, what do they represent?\n- lines 183-184: mention \"corresponding to the latest traffic conditions\" but this seems to be overselling: it's *only* a function of the current observations. There is no history...\n - line 211: \"(a) removing Interactive Attention\": a more interesting ablation here would have been *random* Interactive Attention. This would isolate-away the selection of p, and test how good the socially-aware part of the method would be. Possible an indication of how \"toy\" these experiments are would be: is the performance just the same? I imagine it could be, but for any sufficiently complicated setup  (perhaps what is shown in figure 6, or some more complicated traffic simulator) the ablation would show just how necessary it is to selection p intelligently.\n- Why does each vehicle only select the most interactive other vehicle? there's no reason it can't focus on all vehicles, similar to [30], which inspired this work?\n- Clarity: I think reading the abstract and introduction, it was difficult to understand what was actually being proposed. There were mentions of abstract benefits like \"social preferences\", \"social-aware integration mechanism to integrate objectives of interacting AVs\", and even \"dynamic social preferences\", and \"socially-compatible behaviors\", but it'd be good to get to the point faster. At line 54 it's still unclear what \"socially-compatible behaviors\" means, also on line 73 and 76, would help clarify to define this concretely earlier.\n\n## Small points:\n - line 23: \"inreasing\" --> \"increasing\"\n - line 63\" \"MARL is widely used to navigate AVs in MASD systems\" perhaps clarify this means the research litreaeture, not real AVs? Hardly any real AVs use RL, let alone MARL.\n - line 72: \"response\" --> \"responses\"\n - line 95: why is the policy space \\mathcal{U} and not \\Pi?\n - equation 3: doesn't the advantage function usually (1) not take a reward as input and (2) is a function of the *expected* reward as Q-V\n - equation 6a: \"p \\in \\mathcal{M}\" --> \"p \\in \\mathcal{M} \\ \\{m\\}\"? I.e. I assume \"m\" cannot itself be its own closest/interactive neighbor?\n - line 159: \"features\" --> \"generalization\" (Wlog?)\n - Figure 2 (yellow box): it is shown how r^SP: goes between egotistic (\\phi=0) and alteristic (\\phi=\\pi/2). But equation 8 mentions the range can go to -pi/2 radians too. Is this intended? I.e. ego acting \"maliciously\", against the other vehicles interests? If so, should it be shown in Figure 2 as well, or should the range of \\phi in equation 8 be limited to [0, \\pi/2]?\n - lines 174-176: different how? perhaps \\emph{} which specific part following \"[20]\" that is different?\n- lines 197: \"goals in orders\": which order? does one *particular* car need to go first?\n- figure 6: maybe use white background, and zoom in more.\n- line 231: who were vehicles AV-0 and AV-3 each selecting, was it respectively AV-2 and AV-1?"}}, {"review_id": "DpXO6JEocO", "reviewer": "Reviewer_6Syb", "summary": "The article presents SAPO: a modified version of the IPPO algorithm to solve AV interaction stated as a Multi-Agent Reinforcement Learning problem. The first modification is using social attention (Leurent and Mercat, 2019) to select *one* \"most interactive\" agent. The second modification is estimating a dynamic \"social preference\" (aka Social Value Orientation, Schwarting, et al., 2019) which combines rewards from ego and interactive agents to form a new reward for ego.\nThe article shows a favorable comparison of SAPO performance against a set of baselines, using the total of individual rewards as the metric.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The presented way of combining recently developed techniques (social attention, SVO, IPPO) seems to be novel, and potentially promising.\n1. The set of baselines is large and diverse (perhaps not comprehensive though, see weakness 2).\n1. The article presents in a clear way the methods it uses (IPPO, social attention).\n1. Fig. 6. is a very nice qualitative presentation of a particular result.\n\nWeaknesses:\n1. In my opinion, the main problem of this work is that the motivation does not match with the proposed method and evaluation. The motivation is that AVs are becoming more popular and are sharing the space with other traffic participants (lines 23-24), so safe and smooth interactions with other agents are required (line 34). The solution is a Multi-Agent Self-Driving system, where all agents are trained in the same training loop. Even if such a multi-agent system achieves a high total reward (the metric used by authors), it does not guarantee that the agents would be compatible with human drivers or with different AVs.\n    * The work uses simulator SMARTS, which supports \"diverse behavior models of road users\". Proving a good performance of agents trained with SAPO in interaction with SMARTS provided agents could address the issue to some extent.\n1. The method makes a surprising assumption, that the \"AV should pay attention to *the most interactive* vehicle\", and the way it uses interaction attention is by selecting only one agent for the downstream processing. I think it is easy to imagine a situation, where the ego should take into account interaction with more than one agent.\n     * Perhaps the point above is mitigated by including the baseline with \"social attention\". The method is however described with very few details. It does mention that it is based on \"centralized control of multiple AVs\". Maybe the lower performance is caused not by using full social attention (rather than just the most relevant), but rather due to centralized control (which is not always better).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In its current form, the paper does not have a good match between the motivation and the method, as well as its evaluation. Moreover, I don't see how this method could be used in a practical setup of an actual autonomous vehicle (as it assumes that other traffic participants are trained together with ego). This limits the applicability to robotics, which is the scope of the conference. While the work presents a new way of combining recently developed techniques (social attention, SVO, IPPO), the overall novelty is limited.\nUnfortunately, in its current form, I recommend rejecting the article.\n\n----\n\nThe additional experiments added during the discussion period (in particular on Argoverse and Waymo Dataset) have convinced me to increase my score to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Addressing the Weakness 1. Perhaps the motivation can be changed: e.g. simulating self-driven particles for improving traffic simulation as in Peng et al., 2021; this would however require changing of metric as well. Alternatively demonstrating that the SAPO agents will be able to deal with a diverse set of other agents. There might be other possibilities as well.\n    * It would be very useful to underline relevance to robotics, e.g. if you could show the perspective of integrating this into a real-world autonomous driving system.\n\n2. Explain in more detail, how the method SAPO without Interactive Attention works. Without the Interactive Attention, there is no AV p, so whose reward is used in computing social-aware reward (Eq. 9)?\n\n3. Provide more details on the social attention baseline. If possible, create another baseline, where social attention is used in a decentralized setup."}}, {"review_id": "CPMVMrijUbw", "reviewer": "Reviewer_JLzw", "summary": "This paper presents SAPO, a multi-agent RL framework for autonomous driving. SAPO includes an attention module that tries to identify the most important \"other\" car in an interactive driving scenario and then compactly represent the features of this interaction. It then attempts to integrate these features in such a way to dynamically update the reward structure for the agent, notionally capturing social preferences and adaptation. The paper presents an ablative study, attempting to isolate the effects of several design choices. It then presents an evaluation across several scenarios, compared to 3 popular baseline models. The paper is largely well-written, and the idea is conceptually clever and very important. I have described several concerns below and hope they are instructive and helpful. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The notion that most RL agents and settings use a static reward function is important; the ability to consider dynamic, evolving rewards is a compelling one. \n- The combination of social attention in a MARL setting is quite interesting, and the paper develops these notions well\n\nWeaknesses\n- Many of the evaluations are presented in terms of total accumulated reward with episodes along the x-axis. This suffers the same weakness (in my opinion) as almost all of the RL literature in that (as the very RL literature itself suggests) the reward function does not always indicate whether the task itself has been solved in a satisfactory manner.\n- The limitations sections is somewhat superficial", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is conceptually very elegant, and the application area is very important (furthermore, this idea could be applied in many other settings). Although I voice my concerns below, I feel that the paper warrants publication and the community would benefit from seeing the notions of attention and MARL combined in this type of way. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Line 9, abstract: \"or\" instead of \"of\" in cooperative or competitive fashion...\n\nLine 27: \"self-driven particles\"? What is a particle here?\n\nLine 39-40: \"most of the interaction events occur in the red zone when AVs are crossing the intersection\". While I agree that intersections include a lot of interactions, it is unclear that \"most\" of the interaction events occur here.\n\nThe authors do well to review the literature on social preferences in section 2.2. However, there is another, broader literature on social preferences and context in navigation that could (should) be explored here, for example how human pedestrians navigate in crowded settings. For example, Sekhon and Fleming. (2021, May). SCAN: A Spatial Context Attentive Network for Joint Multi-Agent Intent Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence. and many others from Alexandre Alahi et al.  The dynamics of pedestrians (or ships, etc) are obviously quite different, but the attention mechanisms are likely quite similar and this body of literature could be helpful. \n\nLine 92 we see \"particles\" again. Is this a common term? Does it imply that these AVs do not have kinematics or something?\n\nIn lines 126 onward the authors intentionally omit the timestep in the notation for simplicity. In my opinion, this makes the text and equations more difficult to interpret. \n\nWhat is the justification for equation (9)? Why sine and cosine of the social preference? Why addition? Is this tailored to intersections? What about other types of interactions (e.g. merging or passing in multi-lane traffic)? Would these types of interactions require different integrative rewards?\n\nIn general, I like the ideas and notions presented in section 4. The ability to focus one's attention and adapt rewards accordingly is both clever and important. However, I am a bit concerned about the lack of justification for the particular choices, and then a lack of alternatives and/or comparisons to state of the art in the ensuing sections. These design choices either need a theoretical justification or empirical evaluation, or better yet both. There are ablative studies, which is at least a start in the right direction. \n\nThe ablation studies (and other evaluations) are presented in terms of total rewards. This is a fine place to start, but it assumes that the individual rewards are themselves representative of the desired task. What about (just making some metrics up here) collision rates? Average time to nagivate across an intersection? Or a normalized score comparing this travel time to perfect performance when no other vehicles are present? Etc etc etc. The problem with using rewards only is that we do not really know whether the RL agents have learned anything useful without any other context. Again, this is ok as part of a larger set of evaluations, but I do not see this.\n\nI appreciate the comparison with several popular baselines, across a variety of scenarios. SAPO does well in these comparisons. In particular, it does well with the cross-road scenario, but is this due to the overall concept? Or due to particular design choices (I am thinking of equation (9) for example)? It seems like a general notion of attention and integration should be better across all interaction scenarios, and merging and ramps are perfectly valid instances of interactive scenarios but may be not-quite represented by the authors' design choices. \n\nLine 253: \"This proves...\". This does not really prove anything. It demonstrates that SAPO did better than a few baselines, across a few scenarios, in terms of overall reward. It is an *indication* of *something*, which is fine and quite enough to just say this. \n\nThere are several limitations (and thus future research directions) that are not mentioned in section 7. More exploration of other attention and integration mechanisms is perhaps the most important, IMHO. Furthermore, this limitations section appears to be quite superficial. "}}, {"review_id": "39oot-7KrVH", "reviewer": "Reviewer_U4bM", "summary": "\nThis work proposes a MARL method called Socially-Attentive Policy Optimization (SAPO) for controlling autonomous vehicles to behave socially-compatible behaviors. \n\nTwo key components are used: \n\n1. Self-attention module foot select most interactive participant for each AV\n2. Social-aware reward shaping mechanism.\n\nSAPO enables varying the social preferences of an agent within one episode. Evaluation results show that SAPO can improve training efficiency and also emerge socially-compliant behaviors.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n\n* The method is concise and sound.\n* The presentation is clear and smooth.\n\n### Weaknesses\n\nGenerally, I can't find any major weakness that harms this work.\n\n* Lack of baselines. (See Issues)\n* Figure 2 can be simplified a little bit.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nGenerally, I like this work. The work addresses the social behaviors in traffic flow by first discovering the issues in current algorithms, namely the dynamic social preferences within one episode, and then proposing solutions accordingly.\nThe presentation is clear and the method is sound and reasonable. The evaluation on SMARTS simulator shows promising results.\n\nI have two concerns. First is the lack of baselines (see Issues). The second is whether this work should be presented in CoRL since there are no real robot experiments nor discussion on the applicability of SAPO in real-world. Nevertheless, I admit that the social behaviors in multi-agent systems is indeed an important topic, since in future we may co-live with those robots. \n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "\nIn related work [Peng et al.], the authors use a driving simulator MetaDrive for investigating the detailed behaviors in traffic flow. This work use SMARTS benchmark where only limited number of vehicles coexists. Also, I would expect to see the comparison with the algorithm CoPO proposed in [Peng et. al.], since both use SVO measurement to fuse reward. For the completeness of this work, I would suggest to run CoPO in SMARTS or run SAPO in MetaDrive to fairly compare related methods.\n\n\n[Peng et al.] Learning to Simulate Self-driven Particles System with Coordinated Policy Optimization\n\n\n---\n\n\nI am a little worried about Figure 6. We see SP maintains fixed in ~75 degree and ~10 degree when AV-3. Are there exist hard limit of SP? How can the neural network predicted SP be so smooth within one episode?\n"}}]

Author rebuttal:

Decision: Accept (poster)