Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS
Authors: Eleni Nisioti
Matéo Mahaut
Pierre-Yves Oudeyer
Ida Momennejad
Clément Moulin-Frier
Venue: Submitted to ICLR 2023 (2023)
Keywords: collective innovation
social network
multi-agent model
collective dynamics
communication topology
collective cognition

Abstract: 
The human cultural repertoire relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones. Innovation is not solitary, it relies on collective accumulation and merging of previous solutions. Machine learning approaches commonly assume that fully connected multi-agent networks are best suited for innovation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role of multi-agent topology on innovation has not been systematically studied in machine learning. It remains unclear a) which social network topologies are optimal for which innovation tasks, and b) which properties of experience sharing improve multi-level innovation. Here we use a multi-level hierarchical problem setting (WordCraft), with three different innovation tasks. We systematically design networks of DQNs sharing experiences from their replay buffers in varying topologies (fully connected, small world, dynamic, ring). Comparing the level of innovation achieved by different experience-sharing topologies across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic topology achieves the highest level of innovation across tasks. Second, experience sharing is not as helpful when there is a single clear path to innovation. Third, two metrics we propose, conformity and diversity of shared experience, can explain the success of different topologies on different tasks. These contributions can advance our understanding of optimal AI-AI, human-human, and human-AI collaborative networks, inspiring future tools for fostering collective innovation in large organizations.

individual and Under review as a conference paper at ICLR 2023
SOCIAL NETWORK STRUCTURE SHAPES INNOVATION: EXPERIENCE SHARING IN RL WITH SAPIENS
Anonymous authors Paper under double-blind review
ABSTRACT
Human culture relies on innovation: our ability to continuously explore how ex- isting elements can be combined create new ones. Innovation is not solitary, it relies on collective search and accumulation. Reinforcement learning (RL) ap- proaches commonly assume that fully-connected groups are best suited for inno- vation. However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures. In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others. To our knowledge, the role social network structure innovation has not been systematically studied in RL. Here, we use a multi-level problem setting (WordCraft), with three different in- novation tasks to test the hypothesis that the social network structure affects the performance distributed RL algorithms. We systematically design networks DQNs sharing experiences from their replay buffers in varying structures (fully- connected, small world, dynamic, ring) and introduce a set of behavioral and mnemonic metries that extend the classical reward-focused evaluation framework of RL. Comparing the level of innovation achieved by different social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of inno- vation in tasks with a deceptive nature and large search spaces. Second, experience sharing is not helpful when there is a single clear path to innovation. Third, the metrics we propose, can help understand the success of different social network structures on different tasks, with the diversity of experiences group level lending crucial insights.
INTRODUCTION
Unlike herds or swarms, human social networks solve different tasks with different topologies (Mo- mennejad| 2022).. Human and computational studies show that properties of both the social network structure and task affect the abilities of groups to search collectively: social network structures with high connectivity are better suited for quick convergence in problems with clear global op- tima (Coman et al.) 2016: Momennejad et al. 2019). while partially-connedd
structures perform
best in deceptive tasks, where acting greedily in the short-term leads to missing the optimal solu- tion (Derex & Boyd 2016 Lazer & Friedman, 2007 Cantor et al. 2021 Du et al. 202T Adjodah ct al. 2019). Despite this evidence, works in distributed reinforcement learning (RL) have focused on fully full-cconnected architectures (Mnih et al.) (2166] Horgan al. 2018 Espeholt et al. 2018 Nair et al. 2015; Christianos et al., 2020 /Schmitt et al. 2019; Jaderberg et al. (018).. Here, we test the performance of different social network structures groups of RL. agents that share their experi- ences in a distributed RL learning paradigm. We refer to such groups as multi-agent topologies, introduce SAPIENS, a learning framework for Structuring multi-Agent toPologies for Innovation through ExperieNce Sharing" and evaluate it on deceptive task that models collective innovation.
Innovations represent the expansion of an agent's behavioral repertoire with new problem-solving abilitics and are, therefore, necessary ingredient of continuous learning (Leibo et al. 2019). They arise from tinkering, recombination and adoption existing innovations (Solé et al. 2013 Derex &
Boyd 2016) and have been characterized as type of combinatorial search constrained by semantics
We provide an implementation of SAPIENS and code reproduce the simulations we report

---

Under review as a conference paper at ICLR 2023
dictating the feasible combinations of innovations (Solé al. 2013 Derex & Boyd 2016). We
adopt this definition: innovations are a type of collective search task with a) a multi-level search space, where innovations arise out of recombination of existing ones (Hafner 2021) b) rewards that increase monotonically with the level of innovation, in order to capture the human intrinsic motivation for progress (Solé et al. 2013).
Laboratory and field studies of human groups have shown that collective innovation is highly con- tingent on the social network structure (Momennejad 2022 Migliano & Vinicius| 2022 Derex & Boyd 2016). The reason for this lies in the exploration versus exploitation dynamics of social networks. High clustering and long shortest path in partially-connet connected structures help maintain diversity in the collective at the benefit of exploration. while high connectivity quickly leads to con- formity, which benefits exploitation (Lazer & Friedman 2007). Of particular interest are structures that achieve balance in this trade-off: small-worlds are static graphs that, due a modular structure with long-range connections, achieve both high clustering and small shortest path (Watts & Strogatz, 1998). Another example are dynamic structures, where agents are able to periodically change neigh- bors Volz& & Meyers, 2007). These two families of graphs have the attractive property that they both locally protect innovations and quickly disseminate good solutions (Derex & Boyd 2016).
Despite progress on multiple fronts, many open questions remain before we get a clear understand- ing of how social network structure shapes innovation. On the cognitive science side, computational and human laboratory studies of collective innovation are few and have studied a single task where two innovations are combined to create
one (Derex & Boyd 2016 Cantor al. (002T) while most works study other types of collective search that do not resemble innovation (Mason & Watts)
2010). Furthermore, laboratory stud- ies have collected purely behavioral data (Mason et al. 2008 Derex & Boyd] (0166), while studies of collective memory have shown significant influence of social interactions on individual memory (Coman et al. 2016), indicating that mnemonic data may be another good source of information. In distributed RI, studies are hypothesizing that the reason why groups outperform single agents not just in terms of speed, but also in terms of performance, is the increased diversity of ex experiences col- lected by heterogeneous agents but not explicitly measure (Nair et al. 2015 Horgan et al. [2018). In this case two steps seem natural: introducing appropriate metrics of diversity, and increasing it, not only through heterogeneity, but also through the social network topology.
2012 Mason et al. 2008 Lazer & Friedman 2007| Fang et al.
To achieve this we propose SAPIENS, distributed RL learning framework for modeling a group of agents exchanging experiences according to a social network topology. We study instantiations of SAPIENS where multiple DQN learners (Mnih et al., 2013) share experience tuples from their replay buffers with their neighbors in different static and d dynamic social network structures and compare them to other distributed RL algorithms (Mnih et al. 2016 Nair al. 2015). We employ Wordcraft
(Jiang et al. (2220b) as a test-bed and design three custom tasks (Figures) I [and covering innovation challenges of different complexity: (i) a task with a single innovation path to an easy-to-find global optimum. This type of task can be used to model a linear innovation structure, such as the evolution of the fork from knife to having, two-, three- and, eventually four tines. (Solé al. 2013) and not a deceptive task. (ii) task with two paths that individually lead to local optima, but when combined, can merge toward the global optimum. Ispired from previous studies in cognitive science (Derex & Boyd 2016 Cantor et al. 202T) this task we can capture innovations that were repurposed after their invention, such as Gutenberg's 's screw press leading to the print press (Solé et al. 2013). (iii) a task with ten paths, only one of which leads to the global optimum, which captures search in vast spaces. In addition to the two deceptive tasks in Wordcraft, we also evaluate SAPIENS algorithms on deceptive task implemented in grid world. We empirically show that the performance of SAPIENS depends on the inter-play between social network structure and task demands. Dynamic structures perform most robustly, converging quickly in the easy task, avoiding local optima in the second task, and exploring efficiently in the third task. To interpret these findings, we propose and compute novel behavioral and mnemonic metrics that quantifly among others, the diversity of experiences.
Contributions Our contributions are two-fold. From a cognitive science perspective, SAPIENS is, to our knowledge, the first computational study of hypotheses in human studies relating social network structure to collective innovation, that employs deep RL as the individual learning mech- anism. Compared to the simple agent-based models employed by previous computational studies (Lazer & Friedman 2007; Cantor et al.) [02T] Mason & Watts] 2012). deep RL offers three main ad- vantages: i) it enables empirical experiments with more complex test-beds and larger search spaces
2

---

Under review as a conference paper at ICLR 2023
Level
Beward
House
Sparso connecivity
High conneciiity
Sparso comneciiity
High comnectiity
Level
Nud
Fire
Bricks
Reward
Earth
Wattr
Mud
Figure 1: (Left) Illustration of an innovation task, consisting of an initial set of elements (Earth, Water) and recipe book indicating which combinations create new elements. Upon creating new element the player moves up an innovation level and receives a reward that increases monotonically with levels. (Right) Dynamic social network structures oscillate between phases of low connectiv- ity, where experience sharing takes place within clusters, and high connectivity, where experiences spread between clusters.
ii) agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents, such the majority rule (Lazer& & Friedman| | Cantor et al. 2021) iii) by using the replay buffer as proxy of the memories of agents, we can directly measure properties such the diversity of experiences that are challenging to estimate with humans. Aside these methodological contributions, as we will see later, our empirical study leads to clear hypotheses for future experiments with humans. From an RL perspective, our work extends upon the distributed RL paradigm by systematically analyz- ing the effect of static and dynamic social network structures on different types of innovation tasks both in terms of performance and novel behavioral and mnemonic metrics.
METHODS
2.1 WORDCRAFT: TEST-BED FOR INNOVATION
We perform experiments on Wordcraft Ciang et al. 2020b), an RL environment inspired from the game Little Alchemy 2 As we illustrate on the left of Figure tasks start with an initial set of elements and the player explores combinations of two elements in order to create new ones.
In Wordcraft one can create different types of tasks using recipe book (Kaaiid), which a dictionary containing the valid element combinations of two elements and the newly crafted one. In addition to the recipe book, a task requires a reward function Rvalia that returns a scalar reward associated with crafting element z. A valid combination returns a new element and reward only the first time it is chosen. When queried with a non-valid combination, Realid returns a reward of zero. Thus, a task can be be described by tuple (Xo, Xvatid, Realid, T), where Xo denotes the initial set of elements and T is the number of time steps available to an agent before the environment resets, and can be modelled folly-observable Markov Decision process (see Appendix A)
2.2 INNOVATION TASKS
We introduce the following concepts to characterize the structure of innovation tasks. An innovation task can contain one or more paths, which can potentially be connected to each other (as in the merging paths task). In our proposed tasks, an innovation path X is defined elements
sequence of where crafting an element X, (i > 1) requires to combine the previously crafted element X; and base element from the initial set. The first element X, in the sequence requires combining elements from the initial set or from other paths. The innovation level of an element corresponds to the length of the path it belongs to, plus the sum of the path lengths required to craft its first item (0 if the first item is crafted from base elements). For example, the innovation level of A3 the single path task is 3, while the innovation level of element C₁ in the merging paths task is 5 (1 on C + 2 on A + 2 on B). Within an innovation task, the trajectory of an agent is defined
https:///ites
chemy2.com
3

---

Under review as a conference paper at ICLR 2023
Path
Path
Merged patn
Path
Merging paths
Single path
Irroovation level
Commination
Result
Reward
Best-of-ten paths
A
Path
Path
10
+
th base element poth
5h crafted element
element optimal trajectory
in path
trajectory
We introduce three innovation tasks called single path, merging paths and best-of-ten paths, described in Section 2.2 Each task contains one or more paths, labeled by an uppercase letter (A to J). Each path X has its own initial set of three base elements {ø1, 22, which are represented dashed circles. Crafted elements in path are represented in upper case (X;) in solid circles. Optimal trajectories for each tasks represented by solid red arrows, with their corresponding reward in bold red.
Figure 2:
as the sequence of crafted elements it produces. Finally, the optimal trajectory of an innovation task the trajectory that returns the highest cumulative reward within the problem horizon T. We design three innovation tasks, shown in Figure 2 that pose different challenges:
Single innovation path The initial set contains three elements (Fealid = {a1, 02, ) and An agent needs to first combine two of them to create the first element and then progresses further by combining the most recently created element with an appropriate one from the initial set. This optimization problem contains single global optimum.
Merging paths There are two paths, A and B, and at level 2 there is a cross-road that presents the player with three options: moving forward the end path A, moving forward to the end of path B, or combining elements from path A and to progress on path C. The latter more rewarding and is, thus, the optimal choice. This task is particularly challenging because the player needs avoid two local optima before finding the global optimum.
Best-of-ten paths Here, one of the ten paths is the most rewarding, but, to find it, the player must first explore and reject the other nine paths. This optimization task is characterized by single global optimum and 9 local one and its challenge lies in its large search space.
2.3 LEARNING FRAMEWORK
SAPIENS considers group of K DQN agents, where each agent interacts with its own copy the environment and can share experiences with others. An undirected graph G, with nodes indicating agents and edges indicating that two agents share experiences with each other, determines who shares information with whom We define the neighborhood of agent k, Ni. as the set of nodes connected to it. At the end each episode the agent shares experiences with each its neighbors with probability Ps: sharing consists of sampling subset of experiences of length L, from its buffer B, and inserting it in the buffers of all neighbors B,, n E Ng. Thus, an agent communicates distinct ex experiences with each of its neighbors. We present schematic of two DQN agents sharing
4

---

Under review as a conference paper at ICLR 2023
W
Butter
Env
Agort1
(o) oi slan
H
Butter
grap
Wal stat
Walcond
Agent2
Figure 3: (Left) Social network structures (a) fully-connected (b) small-world (c) ring (d) dynamic. (Right) Schematic of two neighboring DQNs sharing experiences: agent shares experiences from its own replay buffer to that of agent 2 (red arrow) and vice versa (blue arrow) while both agents are independently collecting experiences by interacting with their own copy the environment.
experiences the right of Figure and the pseudocode of SAPIENS in Appendix C We also provide more implementation details about the DQN in Appendix A
Thus, SAPIENS is distributed RL learning paradigm where all agents are both actors and learners,
a setting distinct from multi-agent RL (Gannelo et al., 2021 CCristianos et al. 2020 Jiang et al,
2020a), where agents co-exist in the same environment and from parallelised RL (Steinkraus al.
2005 where there need to be multiple agents. It should also be distinguished from distributed RL paradigma gms with a single learner and multiple actors (Horgan al.) 2018 Espeholt et al. 2018 Nair
et al. 2015 Garnelo et al. 2021 as multiple policies are learned simultancously.
We visualize the social network structures studied in our work on the left of Figure fully- connected, small-world, ring and dynamic. We construct the latter by grouping the agents in pairs and then allowing agents visit other groups for pre-determined duration, with probability Pe- This is common type of dynamic topology that has been used in human laboratory and field studies (Derex Boyd| 2016 Migliano Vinicius 2022) and thoeretically studied (Volz & Meyers 2007). We provide more information about it in Appendix [ where we study how its behavior changes with different values of its
Tes Pv- We also present and analyze an alternative type of dynamic topology where the group oscillates between phases of full and connectivity.
2.4 EVALUATION FRAMEWORK
During evaluation trials with experience sharing deactivated, measure the quality of the final solution and the convergence speed. In addition, we define metrics that are not directly related to performance but can help analyze the effect of social network structure. These behavioral metrics, characterizing the policies followed by the agents, and mnemonic metrics, characterizing the replay buffers of agents, either an individual or group level.
Performance- based metrics: (i) S, group success, binary variable denoting whether at least one agent group found the optimal solution (ii) R the maximum reward of the group at training step I; (iii) R the average reward of the group at training step i; (iv) T Time to first success: the first training step at which at least one the agents found the optimal solution; (v) T* Time to all successes: the first training step at which all of the agents found the optimal solution (vi) T Spread time: number training steps required for the optimal solution to spread to the whole group. least member discovered it (equals T T
Behavioral metrics: (i) conformity C. denotes the percentage agents a group that end up with the same element at the end of given evaluation trial. Thus, agents conform each other even if they follow alternative trajectories.; (ii) volatility V, is an agent-level metric that denotes the cumulative number of changes in the trajectory followed by agent across episodes.
(i) diversity D is an agent-level metric that denotes the number of unique experiences in an agent's replay buffer; (ii) D is a grvep-level metric that captures the diversity of the aggregated group buffer..
Mnemonic metrics:

---

Under review as a conference paper at ICLR 2023
3 RESULTS
We now evaluate the social network structures presented in Figure [ on the innovation tasks de- scribed in Section and visualized in Figure 2 Specifically, methods ring. small-world, dynamic and fully-connected are instantiations of SAPIENS for different social network structures with 10 DQNs where shared experiences are sampled randomly from the replay buffers. We benchmark SAPIENS against: a) no-sharing, a setting with 10 DQN agents without experience sharing b) single, a single DQN agent e) A2C, a distributed policy-gradient algorithm where 10 workers share gra- dients with single (Mnih et al. (016) and d) Ape-X, a distributed RL algorithm with 10 workers that share experience samples with single DQN learner (Horgan al., 2018). To test for statistical significance we perform ANOVA for multiple and Tukey range tests for pairwise comparisons. We refer readers to Appendix for more information about our experimental setup, including tables [ and that contain the means and standard deviations for evaluation metrics in the three tasks.
3.1 OVERALL COMPARISON
In Figure we compare methods across tasks in terms of group success S and Time to first suc- cess T+, where we also indicate statistically significant pairwise comparisons with asterisks (more asterisks denote higher significance).
We observe that performance varies across tasks and topologies. The single path task is optimally solved by all methods, except for single and Ape-X, which failed in 20% and 15% of the trials re- spectively due slow convergence. ANOVA showed no significant difference in terms of group success S (p = 0.43) but methods differed significantly in terms of T (p = 0.2e 19), In par- ticular, single is significantly slower than all other methods (T+ 648g for single) and A2C is significantly quicker than other methods (T+ 36200 for A2C), with the Tukey's 's range test indi- cating significance with p = 0.001 for all pairwise comparisons with these two methods. This is in agreement with the expectation that single agent learns slower due to having less data and that the volicy-gradient algorithm A2C is more sample efficient than value function algorithms like DQNs. In the merging paths task there were significant differences among methods both for group success S (p = 0.4e 4) and convergence speed T (p- = 0.0095). The group success of dynamic (S = 0.65) significantly higher compared Ape-X (S = 0.05, = 0.001). A2C (S 0.0, p 0.00101), fully-connected (S = 0.0, p = 0.00101) and ring (S = 0.2. = 0.0105). The single, no-sharing and small-workh structures performed comparably well, but did not show statistically significant dif- ferences with other methods. In terms of T+, we see that dynamic is quicker than other methods with positive S. which leads to it being the only method with statistically significant differences with Ape-X (p- = 0.0292), fully-connected (p- = 0.0421) and A2C (p- = 0.0421), the methods that failed in almost all trials and hence have T equal to the budget of the experiment. Thus, our main con- clusion in the merging-pon paatgg task is that methods with fully-connected topologies ffully-connected, A2C, Xpe-X) fail dramatically and that the dynamic structure succeeds with higher probability. Fi- nally, in the best-of-ten paths task, differences are also significant both for T (p 0.9e ?) and S (p = 0.15e 9). Here, dynamic outperforms all methods in terms of S with all Tukey range tests in- dicating significance with p- 0.001, which leads it also having significantly higher convergence rate (T+ = 14 for dynamic) compared to the other methods that exhausted their time budget.
In additional experiments we have also: a) observed that these conclusions consistent across group sizes, with increasing sizes leading to better performance in partiall--ooneetedd and worse in fully-connected structures (see Appendix [.5) b) observed a drop in performance under prioritized experience sharing (Souza et al. 2019 Horgan et al.] 2018). where the DQNs employ prioritized replay buffers (Schaul et al. (016)) and experiences with higher priority are shared more often (see Appendix E. In agreement with previous works (Souza et al. 2019). we observe that performance degrades in all methods that share experiences. This does not happen for no-sharing, which indi- cates that prioritized experiences are detrimental only when they are shared. To address this, agents can recompute priorities upon receiving them from other agents to ensure they agree with their own experience (Horgan et al.) 2018). analyzed how the performance of dynamic varies with its hyper- parameters and derived suggestions for tuning it (see Appendix D) d) monitored the robustness of social network structures to different learning ayper-parameters and observed that dynamic is more robust than fully-connected and no-sharing (see Appendix [E.4] e) measured alignment of experi- ences within and across groups that further support our hypothesis that the content of replay buffers
6

---

Under review as a conference paper at ICLR 2023
Single path
Merging paths
Best-of-ten paths
Group success
0.0
1e6
1e7
1+7
sigle
emall
Time first
Figure 4: Overall performance comparison for the single path (first column), merging paths (second column) and best-of-ten paths (third column) task in terms of group success (S) (top row) and Time to first success (T+) (bottom row).
is highly contingent on the social network structure (see Appendix in another deceptive test-bed and derived similar conclusions what we observed in Wordcraft, namely that partiallyccomeected structures are better at avoiding local optima (See Appendix (E.7) g) tested for the robustness of SAPIENS methods the amount of sharing (hyper-parameters L, and P. introduced in Section 2.33 in Appendix E.8 where we observe sub-optimal performance for low amounts of sharing in dynamic and for large shared batches in fully-connected structures. To explain the differences for SAPIENS under different structures we now study each task in isolation for no- sharing, fully-connected, ring, small-world and dynamic, focusing on the behavioral and mnemonic metrics described in Section 2.4
E.3 f) performed simulations
3.2 TASK: SINGLE PATHI
Previous human Mason et 2008 and computational (Lazer & Friedman 2007) studies have indicated that, when it comes to problems with a single global optimum, fully-connected topologies perform best. Our analysis here, however, indicates no statistically significant difference between methods: the fully-connected topology was actually the slowest (7+ = 311e3). To shed light into this behavior, we turn towards diversity and volatility. We compare average individual diversity
(D,) and group diversity D where for each method we create two samples by looking at these two metrics at the timestep at which diversity starts decreasing. ANOVA tests indicate that differences are significant both for Dr+ (p = 0. )), where all pairwise comparisons are significant based on the Tukey range test, and DV (p- 0.0005), where the significantly different pairs are (no-sharing, dynamic, p = 0.02), (dynamic, cullecconnected, p = 0.001), ), (dynamic, small-world, p 0.0087), (fully-connected, ring, p 0. 0.008476). Intuitively, fully-connected exhibits the highest average individual diversity D, (left in Figure 5) and the lowest group diversity DI (left in 5) sharing experiences with others diversifies an individual's experiences but also homogenizes the group.
This task does not require group diversity, but we expect that high individual diversity should be indicative of quicker exploration. So why does the higher D, of fully-connected not act to its benefit? To answer this we turn to volatility V, (top left of Figure 5 and observe that it is highest for the fully-connected topology. We tested for statistically significant differences among methods in terms of volatility at convergence (Vr..) ) and found that these pairs differ significantly: (srall-worid, ring, = 0.002387), (srall-world, dynamic, p 0.001), (soall-world, no-sharing,p 0.001), world,-world, fully- colyeeconee p- = 0.001), (ring, fully-connected, p = 0.001), (dynamic, fully- connected. p 0.001) and (no- fully-connected higher individual diversity and volatility of the fully-connected structure are linked and indicate that, although experience sharing helps speed up exploration, it also destabilizes agents, so that no
0.001). We hypothesize that the
7

---

Under review as a conference paper at ICLR 2023
Avarage Wilaiiity.
2-
Average Diversity. D.
400
1
300
0
0.0
0.5
1.0
0.5
1.0
0.5
1.0
Training step, c1e6
Training step, :1e6
Training step, c1e6
100-
1-81y connacted
Group divazrity,
0-1
1.0
Training step.
1e5
0.5
2.5
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Training
8.6
Training
Bo6
Training
306
Figure 5: Analyzing group behavior: (Left) Average Diversity D, in the single path task (Right) Average volatility (V)) (top row) and Group Diversity DV (bottom row) in the single path task (first column), merging paths task (second column) and best-of- 10 paths task (third column) in term sof
net gain observed in terms of convergence rate. Notably the fact that experience sharing degrades performance in RL has been observed but not understood (Souza et al.) [2019] Schmitt el al. 2019) while as we see here, this becomes possible by analyzing diversity and volatility.
3.3 TASK: MERGING PATHI
The hypothesis that motivated this task was that partially connected groups will perform better than fully- connectd structures due their ability to explore diverse trajectories and avoid the two local
optima (Derex & Boyd| 2016). This was indeed the case as we saw in our discussion of Figure 4 in
Section 3.1 If we also look at the diversities in this task, then we that the group diversity 7)
of fully connected (illustrated in the middle bottom plot in Figure 55 differs significantly to the one of (no-sharing, p 0.001), (ring. p 0.0067) and dynamic (p- 0.0177). The group diversity in small-world, which was relatively successful (S- = 0.3), follows that of the other partially connected lower level (D zs 1700). We believe that this is due the dual ability
structures but peaks of this topology to both protect and spread information and that its diversity and performance will improve as we increase the size of the group, enabling higher clustering (we illustrate this behavior in Figure 15 in Appendix E.5 where we increase the size of groups to 50 agents).
3.4 TASK: BEST-OF-TEN PATH
Which social network topology works best in large search spaces? Our analysis in Figure 4clearly indicates that dynamic achieves the highest performance. Differently from the two previous tasks where fully-connected exhibited the highest volatility, dynamic and ring, the topologies with the lowest number of connections, are the most volatile here (top row of Figure [5). as agents in them are exploring quickly and, hence, performing better. In terms of group diversity, we found that no-sharing scored significantly lower than all other methods with passed all methods, with its difference from dynamic being marginally statistically significant and fully- cully-connected did not score last. Thus, differently from the other two tasks with small search spaces, quick spreading of information increases group diversity here. However, to solve the task quickly, group diversity needs to be combined with quick, local exploitation of the optimal path, which is possible under structures with large shortest path, such as dynamic and ring. Another inter- esting observation here and in the single-path task is that dynamic is the only structure that exhibits significantly higher group diversity than no-sharing, indicating that fosters group exploration.
0.001, the small-world sur-
4 RELATED WORK
In distributed RL, shared information has the form of experience tuples or gradients, with the former being preferred due to instabilities and latency concerns in the latter (Horgan et al. [2088 et all]
2016 Schmitt al. 2019 Nair et al. 2015 Garnelo et al. 2021). A common social network struc-
8

---

Under review as a conference paper at ICLR 2023
ture is that of multiple actors and single learner (Mnih et al. 2016 Horgan et al. 2018 Schmitt et al. 2019; Garnelo et al.) 202T), while the Gorila framework (Nair et al. 2015) is a more gen- eral multi-learner architecture, that differs from ours as agents are sharing parameters. Networked Evolutionary Strategies (NetES) consider the effect of network structure on evolutionary strategies (Adjodah et al. 2019), where multiple actors share gradients with a single learner. NetES was ap- plied on continuous control tasks that differ from our deceptive and discrete innovation tasks. In MARL, social network structure determines who co-exists with whom in the environment (Garnelo
et al.
2 or which agents in its neighborhood an agent attends to (Jiang et al. 2020a] Du et al.,
2021 Here, dynamic topologies that are adapted maximize a group's 's reward have been shown to maximize strategic diversity (Garnelo et al. (021) and help the agents coordinate on demand (Du et al. 202T). contrast, our dynamic topologies vary periodically independently of the group's performance, which is important for avoiding local optima. In population-basel training, policies are compared against the whole population, thus only considering a Cully-connected social network structure (Jaeeberg et al. (218)). Admittedly, the literature on the effect of social network structure on collective search is diverse, with different fields making different design choices; to illustrate this provide non-exhaustive summary of our literature review in Table of Appendix B
5 DISCUSSION AND FUTURE WORK
We tested the hypothesis that the social network structure of experience sharing can shape the per- formance of a group of RL agents using our proposed learning framework, SAPIENS, and showed that, in line with human studies, both social network topology and task structure affect performance. Based on our experimental results, we can provide general recommendations use for which task class. the single-path task, an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks (Gupta et al. 2219)), our results show benefit of experi- ence sharing. In the merging-paut
which topology to
task which exhibits strong local optima that have to be explored up to certain point in order to discover the global optimum (in the spirit of hard exploration tasks (Baker et al.] 2022 Ecoffet et al., 202T)), our results show that topologies with low initial connec- tivity (such no-sharing, small world and dynamic) perform best. The dynamic topology shows the highest performance, allowing different groups to explore non-optimal paths before sharing their experience during visits other groups to find the optimal one. Finally, in the case of large search space with many local optimas (in the spirit of combinatorial optimization tasks (Mazyavkina et al.) 202I), our results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.
When adopting RL algorithms as computational models for replicating experiments with humans, one needs to acknowledge that their communication and decision- faithfully replicate the ones used by humans. One notable difference is that humans may exhibit normative behavior, adopting information not for its utility in the task but for social approval (Mason) et al. 2008). From an RL perspective, our study is limited in including experiments only in few symbolic tasks and simple navigation task; in the future plan to study more complex environments like Crafter (Hafner) 2021).
mechanisms may not
We hope that work will contribute to the fields of cognitive science and DRL in multiple ways. First, our empirical observations in the single path and best-of-ten-part tasks provide concrete hy- potheses for future experiments studying human innovation, which has so far been studied only in a task that inspired our merging-paths task (Derex & Boyd (0166). By continuing the dialogue that has been initiated between human and computational studies (Fang et al.) 2010 Lazer & Friedman 2007 Cantor et al. (02T) to include DRL methods, we believe that cognitive science will benefit from tools that, as we show here, can learn in realistic problem set-ups and can be analyzed not just in terms of their behavior, but also in terms of their memories. Second, we hope that studies in distributed RL will extend their evaluation methodology by analyzing not just rewards, but also behavioral and mnemonic metrics such as diversity, conformity and volatility that, as we show here, correlate with success. Aside this, the effect of social network structure in distributed RL can be extended beyond evolutionary strategies (Adjodah et al. 2019) and beyond our current instantiation of SAPIENS, by considering other off- off-po algorithms than DQNs and other types of information sharing. Finally, considering the effectiveness of the dynamic topologies observed in this study, we envision future works that investigate more types of them, as well as meta-learning or online- adaptation algorithms where the social network structure optimized for desired objective.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dhaval Adjodah, Dan Calacci, Abhimanyu Dubey, Anirudh Goyal, Peter M. Krafft, Estehan Moro, and Alex Pentland. Communication topologies between learning agents in deep reinforcement learning. CoRR, abs/1902.06740, 2019, URL
org/ abs 902. 06740
Bowen Baker, lige Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv: Arxiv-2206. / 2022.
Philip Bontrager, Ahmed Khalifa, Damien Anderson, Matthew Stephenson, Christoph Salge, and Julian Togelius. Superstition in the network: Deep reinforcementt learning plays deceptive games. 908 04436
CoRR, /t////98.04444 2019. URL http:tt////axxv.orttt
Mauricio Cantor, Michael Chimento, Simeon Q Smeele, Peng He, Danai Papageorgiou, Lucy M. Aplin, and Damien R. Farine. Social network architecture and the tempo of cumulative cultural evolution. Proceedings of the Royal Society B: Biological Sciences, 2801944))00003100,, March URL bttps://oovaasccittyaallll....
2021. doi: 10. (tt099///pbb.200..11.
doi/10.1098 rspp. 2020 3107
Publisher: Royal Society.
Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor-critic for multi- agent reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10707-10717. Cur- neuriperip... ran Associates, Inc., 2020, URL (tRL:atttas/pprceeeeiiii www.ripss.cccpppee2222222
fil
Alin Coman, Ida Momennejad, Rae D Drach, and Andra Geana. Mnemonic convergence in social networks: The emergent properties of cognition a collective level. Proc. Natl. Acad. Sci. U. S. A., 113(29):8171-8877 July 2016.
Maxime Derex and Robert Boyd. Partial connectivity increases cultural accumulation within groups Proceedings of the National Academy of Sciences, 113(11):29822 22282227772 March 2016. ISSN 0027- 8424, 1091 0699- doi: 10.1073/pnas.... 1at.aaa..111177 1518798113. 10.30.117////11
URL htt.httpp//www.. pnws. orte//oorrloooooo
Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang. Learning Correlated Communication Topology in Multi-Agent Reinforcement learning. In Pro- ceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp. 456 464, Richland, SC, May 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1 45500-80073
Marina Dubova, Arseny Moskvichev, and Robert Goldstone. Reinforcement Communication Learning in Different Social Network Structures. arXiv:2007.09820 [cs), July 2020. URL arXiv: 2007.09820.
http://wxw....rg//////0//0...../ 09980
Robin I. M. Dunbar. How conversations around campfires came to be. Proceedings of the National
Academy of Sciences, 111(39):14013, September 2014. doi: 10. 10.1100/sss.144 1atpmass..1668822.. 1416382111. URL.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-586 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor- Architectures. Tech- nical Report arXiv: 802.01561, arXiv, June 2018. URL arXiv: 802.01561 [cs].
01561
Christina Fang, Jeho Lee, and Melissa Schilling. Balancing Exploration and Exploitation Through Structural Design: The Isolation of Subgroups and Organizational Learning. Organization Sci- ence, 21:625 (2226666622666 June 2010. doi: 10. (0.1128/4 (0.128// 1090.0469.
10

---

Under review as a conference paper at ICLR 2023
Marta Garnelo, Wojciech Marian Czamecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gauthier Gidel, Hado van Hasselt, and David Balduzzi. Pick your battles: Interaction graphs as population- level objectives for strategic diversity. CoRR, abs/2110.04041,, 2021. URL https: /arxiv.
org/abs/ 200.//4411000011111
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Leslic Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learn- ing, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learming Research, pp. 1025-1037. PMLR, 2019. URL http: /pprocedingg..o .ww.innggmm..........mm. press. popresssvl00) v010/ httpe/ss/w11o/gggaaaaaaaao http:/gppaaaaaaaaaam html
Danijar Hafner. Benchmarking the spectrum of agent capabilitiiss arXiv preprint arXi::1009.06780, 2021.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed Prioritized Experience Replay. arXiv:1803.00 009933 [cs], March 2018. URL httt::tt/aaarttt www.aaiwwi....ooaeeooooooo aas//ab////// 1800/11030000 C0933
arXiv: 1803.00933.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Gareía Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. CoRR, 0tos/1807.01220 2018. URL httt:://axiiv org/ abs 10111010770011111
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph Convolutional Reinforcement Learning. Technical Report arXiv: 1000002,,,,0200,, arXiv, February 2020a. URL http:://arxix
hrg/abs/1810. 09202 arXiv:1810.09202 [cs, stat].
Minqi Jiang, Jelena Luketina, Nantas Nardelli, Pasquale Minervini, Philip H. S. Torr, Shimon Whiteson, and Tim Rocktäschel. WordCraft: An Enviromment for Benchmarking Commonsense Agents, arXiv:2007. 2007:00007 [cs], July 2020b. httpt/tpp//.aaii.oo///////////// arXiv: 2007.09185.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL
http:/ / /arxivvv v. org/ /bbs/1412 6980 cite 1rxiv:1412.6980CCommeee Published as con-
ference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Michelle A. Kline and Robert Boyd. Population size predicts technological complexity in Ocea- nia. Proceedings of the Royal Society B: Biological Sciences, 277(1693):5559-5644 Au- doi: 00.1099//sbb.22100445...
gust 2010. ISSN 0962-8452 8452, 1471-2954.
URL https:
http://ppppl.....oooooooooo..... 1wt.:/999//pp.b... 2010 0452
David Lazer and Allan Friedman. The Network Structure of Exploration and Exploitation. Ad- ministrative Science Quarterly, 52(4):667 694, December 2007. ISSN 0001-8392, 1930-3815. doi: 10.218%/asqu.22.4..66 URL http: www.:aaassssssapp.......... /oi/ 2189/
asqu 1...5246665. 667
Joel 7. Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.0077 007422 2019.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compttiie environments, 2017. URL https://arxiv.orrg abs/1706. 02275
W. Mason and J. Watts. Collaborative learning in networks. Proceedings the National Academy of Sciences, 109(3):664700) Jamuary 2012. ISSN 0027-8424, 0tt.pp......1111111111111000 11 10069100. URL http::tt///// pnas http://wwwwppaaaaocooooo/oooo///
doi: 10. 10. 1073/aaass...

---

Under review as a conference paper at ICLR 2023
Winter A. Mason, Andy Jones, and Robert L. Goldstone. Propagation of innovations in networked
groups. Journal of Experimental Psychology: General, 137(3):422 433, 2008. ISSN 1939-2222, 0096-3445. doi: 10.1037/:0012798. 10.1037/a001273
URL http://cott apa. .rg. wwggggedoooooooooooooooooooo...e getdoi. ?fmddoi=
Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134:105400, 2021.
Andrea Bamberg Migliano and Lucio Vinicius. The origins of human cumulative culture: from the foraging niche to collective intelligence. Philosophical Transactions of the Royal Society B: Biological Sciences, 377(1143),202003117 January 2022. doi: 10.1098/stb.
2020.0317. URL
2020. 0317
Publisher: Royal Society.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver. and Koray Kavukeuoglu. Asynchronous Methods for Deep Reinforce- ment Learning. arXiv: arXiv:1602.0I 1602.01783 [cx], June 2016. URL. /Rt.tttpp://aaxii.oor/////////bp
01783
arXiv: 1602.01783.
Ida Momennejad. Collective minds: social network topology shapes collective cognition. Philo- sophical Transactions of the Royal Society B: Biological Sciences, 37112222,,7711183322220515,,,, January URL soso://oooosssseietyluuoliiiiiii Publisher: Royal Society.
2022. doi: 10. 10.10981/sb.2220003115
/01/10.1098/100 ht..://stbb..0
0315
Ida Momennejad, Ajua Duker, and Alin Coman. Bridge ties bind collective memories. Nat. Com- mum., 10(1):1578, April 2019.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively Parallel Methods for Deep Reinforcement Learning. Technical Report 1rXiv:1507.00m
arXiv, July 2015. URL org/ abs/ http:/xwwww...../////......11... 04296 arXiv:1507.044 [cs].
http:://rxixs
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized Experience Replay. Technical Report arXiv:1511 1059552, arXiv, February 2016. URL http://arxiv..rr// arXiv:1551 11t::1111..0000 1110:0511 [cs].
1511. 05952
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-Policy Actor-Critic with Shared Ex- perience Replay. Technical Report arXiv:1909. 1583, arXiv, November 2019. URL http:
//arxiv. wwtrxiv.org/abba. http/bbs/1990 11583 arXiv:1909.11 1583 [cs, stat].
1909.11833
Ricard V. Solé, Sergi Valverde, Marti Rosas Casals, Stuart A. Kauffman, Doyne Farmer, and Niles Eldredge. The evolutionary ecology of technological innovations. Complexity, 1844):1-277,
March 2013. ISSN 10762787. doi: 10. 10.1002/cplx.21433. URL http:///ol http://oo..iin. follewile... com/ 10. 1002/ 11.02/pplx.2144
Lucas Oliveira Souza, Gabriel de Oliveira Ramos, and Celia Ghedini Ralha. Experience Sharing Between Cooperative Reinforcement Learning Agents. 1xX1::911..02199 [cs], November 2019. URL
org http:/s//////.1111./////1 02191 arXiv: 1911.02191.
D. Steinkraus, I. Buck, and P.Y. Simard. Using gpus for machine learning algorithms. In Eighth International Conference on Document Analysis and Recognition (ICDAR'O5 005), pp. 1115-1120 Vol. 2, 2005. doi: 10. 1009/CDRR.000..22
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026,5033, 2012. doi: 10.1 00.119//ROS..202..6666
12

---

Under review as a conference paper at ICLR 2023
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul. Hado van Hasselt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, An- ders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft ii: A challenge for reinforcement learning, 2017. URL BBBL:ttpps//aaa.......//////// 04782
Erik Volz and Lauren Ancel Meyers. Susceptible- ceceptibleiifecteed-e recoveerdd epidemies in dynamic con- tact networks. Proceedings of the Royal Society Biological Sciences, 274(1628):2925- 2934, December 2007. ISSN 0962-8452, 1471-2954. doi: 10. 00.1009/sph.2207.115555 URL
wwtp:/iiiiiiii.ggoi/oooo//oo doi/10....0001 1098/ http/////pp...... 2007. .1159
Duncan J Watts and Steven H Strogatz. Collective dynamics of 'small-world" networks. 393:3, 1998.
Polly W. Wiessner. Embers of society: Firelight talk among the Ju/"hoani Bushmen. Proceed- ings of the National Academy of Sciences, 111(39):14027 September 2014. doi: 10. 10..0s3/pnas. 1404212111. URL httth:/ttw/wi pnas org/ wwt./orgggooooooe..e /tt////onontt1////////222 abstract
13

---

Figure ooffers a visualization of the states and actions encountered during episode in Wordcraft, where the chosen actions and elements are chosen so as to reproduce the example of Figure L In order solve the innovation task described in Section 2.1 compute the maximum number of elements a player can craft within horizon T' for recipe book Xmatid and initial set X, which we integer in [0, (X]). Thus, the action space is Under review as a conference paper at ICLR 2023
Inventory.
Episode stop. t
Action, a
State, $
Reward,
0000
Episodic
0
0
Idx=
Figure 6: Visualizing actions and states in Wordcraft: we present the first 3 time steps of an episode corresponding to playing the example in Figure This task contains 7 elements, so the action space a integer with maximum value
the components current e and inventory i, each digit in the vector corresponds to the element with the corresponding index. The initial set includes Water and Earth (their indexes at T = in the inventory are non non-zero). The agent first picks Earth (second index in the action vector). At l Earth becomes active in the Current vector of the state, the the agent selects Water and receives positive reward, At = 2, Mud created and inserted in the inventory and is cleared.
This supplementary material provides additional methods, results and discussion, as well as imple- mentation details.
Section describes detail the MDP formulation of Wordcraft;
Section contains the pseudocde SAPIENS;
Section explains how we model dynamic social network structures and how their perfor- mance varies with their hyper-paramaters:
Section provides more information about our experimental setup and results (effect of group size, intra-group and inter-group alignment, robustness to learning and effect of prioritized experience sharing). We also provide tables and figures for all metrics presented Section 2.4 and reward plots.
Section contains simulations with another testbed, the Deceptive Coins game.
A DETAILS OF WORDCRAFT AS A MARKOV DECISION PROCESS
We consider the episodic setting, where the environment resets at the end of each episode and an agent trained for Etrain episodes. At each time step the agent observes the state s; and selects an action o; from set of possible actions A according to its policy T², where x² is mapping from states to actions, parameterized by neural network with weights 0. In return, the agent receives the next state 8t41 and scalar reward rt. Each DQN agent collects experience tuples of the form [st, az, $141, Tr] its replay buffer.
denote | We, then, encode each element
14

---

Under review as a conference paper at ICLR 2023
Work
Fied MARL
Agent Midel DRL
Infommtton type Interaction !
Tosk
Dynane siucuree
Maia conclusion
Curnooo 2022
strategic
micro
Yes
Topologles With
minagamen
(Str-
cycls
enemerge
Crafi Vinyals
strategic diversiy
dynamic
perform rebusdly
Aciodda ot al.
DRL
rowards weights
contimmous
control
Rardom topolgics
2019)
(Mujeco 2012
Lodkeon
outperfomus fully-
ovenceted ones
Da 2001
MARI.
1981.
aiservation
pxpprraive
naviga
Yos
Agens choose
Son (Partale
World
costmunicat whes
Lowe allll
2017
they need to count-- nasc.
Dobova 2000
MARUC
DRL
coordination
No
Global connectivity
Icads shared
symmetric protecols,
while
paraally
connected groups
leana local dialets.
Fing et all 2010
computaaionnl cogni- tise science
belief-mojority nile
belief, reward
NK problen
No
Pantal connectivity
maximizes
perfo-
manee
Friedman
cogol- sclence
belief-
belief,
NK
Patall
matimizes
Cantor al 2011
computational cugnt live science
belief majonity role"
belief, reward
Innovation
No
Performacce de- geads on both task und group stracture. no topalogy is m- bustly optimal acroes
Full tadks.
Walts
NK problem
No
diversity
works
complex
Mason 2008
cogaltive science
action, reward
line search
No
Partill connectivity
works com-
plex problems
Deres
Bond
cogaltvee SERCICE
actooLreward
Innovation
Yes
pertial connectivity works hed
2016
T0m Work)
disenbused RL and commjutatintal live science
DRL
Transttion tplles
innovatoon
yos
Fartally connected
stuuctures, especially dynamics
perform robuslly
ditferent types
Table 1: A non-comprehensive summary of the literature on the topic of the effect of social network
topology collective search
A A= |0 XX), with action at indicating the index of the currently chosen element. The state s contains two sets of information: binary vector of length |X] with non-zero entries for elements already crafted by the agent within the current episode (we refer to this as inventory i) and another binary vector of length XX| where index is non-zero if it is currently selected by the agent (we refer to this as current c). An agent begins with an inventory having non-zero element only for the initial set Ng and an all-zero selection. With the first action ag. the selected item becomes non-zero in the selection. With the second action, 01, we check if the combination (a1, Cn) is valid under the recipe book and, if so, return the newly crafted element (corresponding entry in becomes non-zero) and the reward. This two-step procedure continues until the end of the episode.
B SUMMARY OF RELATED WORKS
In this appendix we provide a non-comprehensive summary of the literature on the topic of the effect of social network topology on collective search in Table where our objective to highlight similarities and differences within and across the fields of cognitive science and DRL.
C PSEUDOCODE OF SAPIENS
We present the pseudocode of our proposed algorithm SAPIENS in Algorithm SAPIENS works sim- ilarly to an ofllcy reinforcement learning algorithm, with the difference that, after each episode, an experience sharing phase takes place between agents that belong in the same group.
15

---

Under review as a conference paper at ICLR 2023
Algorithm SAPIENS (Structuring multi-Agent toPology for Innovation through ExperieNce Shar- ing)
1: Input: 9, connectivity, R,ps, LS
Ps.
2: GinitaalizGrrap(comeetiiit)
3: ZinitializeAAgent(
> Initialize agents
4: for E I do
Inform agent about its neighbors 6: Lenv = initEnv(R) & Create agent's own copy of the environment based on the recipe book
5: Ineighhors==
Iform/eiighbortood(()
7: end for
8: while training not done do
9:
for i I do
Loop through each agent
10:
while episode not done do = i.policyy)
Choose action
12:
Snew env.step(a)
I.B.inert((ss1) Snew
14:
end while
=randomm)
16:
if é < Ps then
D. Share with probability
for j E i.niihhoor do
18:
j.B.add(i.B.o ..ampl)))))))
Sample random set of experiences of length L
19:
end for
20:
end if
i.train()
> Train agent
21:
end for
22:
end while
D ANALYSIS OF DYNAMIC NETWORK TOPOLOGIES
In the main paper presented results for single type of dynamic topolgoy. Here we present another type and analyze how they both behave for different values of their hyper-parameters. The two dynamic topologies are:
Inspired by graphs employed in human laboratory studies (Derex Boyd 2016), we de- signed graphs where the macro structure of the graph is constant but agents can randomly change their position.
particular, we divide a group of agents into sub-groups of two agents and, at the end of each episode, move agent to another group with probability Pv for duration of Tú episodes (for visualization see Figure 33. To reduce the complex- ity of the implementation, we assume that only visit take place at time. In the main paper we employ P. = 0.01 and T, = 10 across conditions and present results with different values in Appendix D where we refer to this topology as dynamic-Boyd.
Human behavioral ecology emphasize the importance of periodic variation in human so- cial networks encountered throughout our evolutionary trajectory Wiessner (2014); Dunbar (2014). Due to ecological constraints human groups oscillate between phases of high and low connectivity: low-connectivity phases arise when individuals need to individually col- lected resources (e.g. day-time hunting) while high-connectivity phases arise when humans are idle and "forced" to be in proximity with others (e.g. fireside chats). Although these high-connectivity phases do not bare a direct evolutionary advantage, they may have played important role by creating the conditions for the evolution of human language and cul- ture. Inspired by this hypothesis, we have designed dynamic graphs that oscillate between fully-connected topology that lasts for Ta episodes and topology without sharing that lasts for 7, episodes. We present results for various values of T7, and Ta of this topology in Appendix D where we refer to this topology as dynamic-periodic.
In Figure 8 probabilities of visit (p.) and visit duration T. (the sub-group size is 2 in all cases). We note that, due to our implementation choice that a visit can take place only if no other agent is currently on a visit, the visit duration also affects the mixing of the group: longer visits mean that fewer visits will
we observe the % of group success (S9) with the dynamic-Boo -oyy topology for different
16

---

Under review as a conference paper at ICLR 2023
Divice into su grouss
Vis with probability Pe
Reamn ofer To spisodes
Full-conneceed prese that lasts for Tp episodes
Nosharinng phase fruc. lovts for T) epioodes
Figure 7: Two types of dynamic topologies: (Left) in the dynamic-Boyd topology the group is divided into sub-groups of two agents and a visit takes place with probability Pe and lasts Te episodes (Right) In the dynamic-periodis topology the graph oscillates between phase with fully- connected topology that lasts for Th, episodes to phase without sharing that lasts for Tj episodes.
S % of group success
5 % of group success
0.5
0.4
Visit duration
0.4
Tv. Visit duration
1000
0.3
0.3
100
100
0.2
0.1
10
10
0.0
0.0
0.005
0.01 0.1
0.5
0.005
0.01
0.1
0.5
Pv. Visit probability
Visit probability
Figure 8: Examining the sensitivity of the dynamic-Boo Baaamic-ooddd topology to its hyper- pyper-parameters: % of group success (8%) for the merging-paths task (left) and the best-of-ten paths task (right).
take place total. In the merging paths task (left), two effect: (i) short visits with of high probability lead to bad performance. As such settings lead to quick mixing of the population, they lead to convergence the local optimum (ii) long visits with high probability work well. Due to the high visit probability, this setting effectively leads to topology where exactly one agent always on a long visit. Thus, ensures that sub-groups stay isolated for at least 1000 episodes, after which inter sub-group sharing needs to takes place to ensure that the sub-groups can progress quickly. In the best-of-ten paths task (right), this structure has clear optimal ayper-parameterizatiin are preferred, which maximizes the mixing of the group and makes early exploration more effective.
settings have a clear
short visits with high probability
In Figure 9 we observe the % of group success (59) the Synamic-periont topology for various values of Tr and Ti. In the merging paths task (left of Figure 9 medium values for the period of both phases works best, while there is some success when the low connectivity phase lasts long (7i 1000). In the best-of-ten paths task (rightof Figure values for the period of both phases work best: thus both the absolute value and their ratio important ensure that exploration is efficient. The optimal configuration the same between the two tasks (T) 100, T 10), which is good indication of the robustness of this structure.
observe the same medium
E EMPIRICAL RESULTS
To ensure that all methods have the same number of samples, assume that, for trials where method did not find the optimal solution, and, hence, T is undefined, T equal to the total number of timesteps the method was trained for, Tian- For each task, all methods have been trained for an equal duration of time: Timin = le for the single path, Timin = 7 for the merging paths task and Train 2e for the best-of-ten paths task.
17

---

Under review as a conference paper at ICLR 2023
S % of group success
S % of group success
T, Low Low-connectivity period
T,, Low-Looooone Low-connectivity period
1000
1000
0.2
100
100
0.5
10
10
0.0
0.0
10 100 1000
10 100 1000 Ts. High-connectiiitt period
Th. High-connectivity period
Figure 9: Examining the sensitivity of the dynamic-periodes topology to its hyper-parameters, % of group success (89) for the merging-patie task (left) and the best-of-ten paths task (right).
We perform 20 independent trials for each task and method and visualize our proposed metrics with barplots and line plots of averages across trials with error bars indicating 95% confidence intervals. We test for statistical significance
our evaluation metrics separately for each task by applying ANOVA tests to detect whether at least
method differs from the rest and, subsequently, em- ploying the Tukey's 's range test I to detect which pairs of methods that differ significantly. We report the exact p values of theses tests in the text and, when applicable, illustrate them in figures using a set of asterisks whose number indicates the significance level (p <= 0.05:
<= 0.01: #*
p 0.001: *** p 0.0001: #*** 10
We presented the major results our evaluation of SAPTENS in Section 3 We now present additional information regarding the implementation the different components (Appendix E.I] the values of all performance metrics and additional plots for experiments discussed in 3 (Appendix E.2) results on intra-group and inter-group alignment (Appendix E.3). results for groups of varying sizes (Appendix E.5) and results on various dynamic topologies (Appendix
E.1 IMPLEMENTATION DETAILS
Implementation of DQN We employ the same hyper-parameter for each DQN across all studied tasks and topologies: discount factor = 0.9, the Adam optimizer with learning rate 0.001 (Kingma & Ba 2014 Dunbar 2014), e-greedy exploration with e = 0.01. We employ a feedforward network with two layers with 64 neurons each. We implemented SAPIENS by extending the DQN implementation in the stable-baselines3 framework.
Implementation of A2C We used the Jable-baselines3 implementation A2C and tuned the hyper-parameters: learning rate, number of steps, discount factor, the entropy coefficient and the value function coefficient. This gave us the best-performing values 0.001, 5, 0.99, 0.1 and 0.25, respectively, that we also employed in the other tasks.
Implementation of Ape-X
We used the ray implementation of Ape-X DQN and tuned the hyper-paramede parameters::::: learning rate, discount factor, replay buffer capacity and e-greedy exploration. This gave us the best-perforning values in the single path task 0,001, 0.9, 5000 and 0.02, respec- tively.
18

---

Under review as a conference paper at ICLR 2023
Topdlgy
Asc
Rig
7
y
8
Veng
Cur
moshiring
(8:92, 00.00)
(1.)
(256250,
13441)
(8100000.0)
(900000 Ds
(1.))
10058,0000
[0 0.0354)
dynamic
(1,9)
(1.)
(237222, 53885)
(3460666 (22041)
(109444 98067)
(1.0)
10.027.0.11
(0.885 (.026)
fully connected
(1,0)
(1.0)
(310566, 89230)
(262000 9650E)
(51333, 20055)
(1.0)
(QU022, (U827)
(0.99 .0.03)
rng
(1,00
(1.0
(235333, 70196)
(305333, 78815)
(20000, 22038)
(1.0)
(0.035.0.022
(0,677 0.03540
small world
<1.0g
(1.D)
(253333, 63320)
(1022666 74110)
(49533,
31274)
(1.0)
(0.025, (.D13)
10.9122 0.0267)
simple
92, 0,166)
(0.927, 0.163)
(64750, 266145)
(64750. 266145)
(0.9)
(0.2,044)
(0.01S. D.01 5) nOR-OD
(L.))
A3C
<1.9g
(1.D)
(36200. 16450;
(76200. 16450)
(00)
(1.D)
(0.0)
(L.))
Ape X
(0.93.0.18)
(0.93) 0.18)
(270941. 102465)
(21091L (02445)
(00))
(0.15,0.366)
(00015, QU022)
(1.0)
Table 2: Evaluation metrics for the single-path task in the form (mean of metrics, standard deviation of metric)
Topulogy
HC
RC
2
2
S
Veza
movllaring
00838,07
(XXEIIOO, 211045)
UMBORR D5
dysamii
(0.7,0.54
(0.9.0.13)
(279650,222965)
(700H00L, ()
(TOROOIL E)
(0.750448)
(0.597, 0.0059))
(QUNIS, (.0016)
connecco
(0.539.8.065)
(0.58, (.04)
(7006006.0)
(700HO00, 0)
(TOROORL E
(SU))
(0.59).0.0053)
(0,0764, 0.08444)
(0.661,01135) 135)
(0.72. 0.155
(5802000, 27881033
(7000000, ()
GO00060 D)
(0.20.41)
(0.595, 0 0055))
(0.0149.0 i 021)
small-porldd
(0.699.0001))
(0.774. D (.57)
(5908100, 16990266)
(7000000, 0)
(7000000D)
(0.3, 0.483)
(0.536.0.005))
(0.06775.0) 6928)
single
00.756.0.188
00.758. 0
(5235000, 2385948)
(5238000, 2385968)
(0.0)
(0.3, 0.477
0063, 0 0003))
A2C
(0.269.01.2
(0,269.00 0.25
(70000000 ()
(7000000.0 0)
(0.0)
(0.0)
10.013, 0.08))
(0.573. 0.33)
(6656H00) 15343899
(26651900 1534389)
(0,054,0.257)
Table 3: Evaluation metrics for the merging- pattingggpahssttte task in the form (mean of metrics, standard devi-
ation of metric)
Implementation of graphs used as social network structures We construct sorll-worlds using the Watts-Strogatz model seatts.strogee z ..graph method of the networkx package [3). This model first builds ring lattice where each node has neighbors and then rewires edge with probability 3. Compared to other techniques used in previous works studying the effect of topology Mason al. (2008), this way of constructing smrll-worlds ensures that the average path lengths is short and clustering is high. These two properties are what differentiates small-worlds from random (short average path length and small clustering) and regular (long average path length and high clustering) graphs We employ n = 4 and 3 = 0.2 in our experiments, which we empirically found to lead to good values of average path length and clustering.
We have described the generation process of dynamic topologies in Appendix D In the main pa- per we employ the dynamic-Boo Boyd topology with T, 10 and Pe 0.001 across tasks. These parameters have been tuned for the merging- -attss task.
E.2 OVERALL COMPARISON
Tables 2 3 and contain the values of all metrics discussed in Section 2.4 for the single path,
merging paths and best-of-ten paths, respectively. We denote values computed after convergence of the group with underscore co and values averaged over all training steps with underscore avg (note that we use over variables to denote averaging over agents in a single training step). Cells with a dash (-) indicate that we could not compute the corresponding metrics because group failed to find a solution in all trials. We also provide the plots of volatility and average diversity for the merging paths and best-of-10 paths task (that were not included in Figure 5) due to page limit constraints).
Figure presents the reward eurves for all methods in the single path, merging paths and best-of- ten paths tasks respectively. Specifically, we plot the maximum reward of the group at training step t (R.).
htttps:/networkxx.t
Eqpdoyy
dynamic
(0S14109335 923)
60718, 0.30)
(13615016, 1611195)
COOMOOOOO,
(0A.511)
00.242..0075
fally-omeeted
(0116122009)
(01819 0.
(2000000
CH000000, (3)
C10000000
00,00
00.2160.0055
ting
002319003045)
(18781000, 3854516)
(15801000, 57125139
(18045000 61 (08252)
(0..0.31)
(.237.00044)
(6.047.0.099
small world
60, (9....2220
(0216.0275)
(18700000) 409 40919877
L18746000,
3900E890
(1MOR0000 61 61988863
(0.100316)
(0.254.0007)
(0.018/11009)))
single
10.7% (1067)
03 00.7788 0000000
CZOTIOOOOO OJ
2003
(0.01)
(1.00
(100951110011
AZC
00. (2.128.0.19)
(0 191
(2000000O OI
(00))
(1.0)
(1.0)
(0.3344..35)
Xpe
(0 00441, 0.113)
10.452 0.215)
(20000000O OF
(20000OOL))
(000)
(0.9L 0.316)
(1.0)
(0,015,0000)
Table 4: Evaluation metrics for the best-of-ten paths task in the form (mean of metrics, standard deviation of metric)
19

---

Under review as a conference paper at ICLR 2023
Maximum reward, Rmax
Maximum reward, Rmax
1.0
0.5
0.5
0.0
AEE-JUV
Stglt
T0-sTaring
deank
thg
carested
A2C
3md Fuu1)
0.0
Siigit
APERDON
10-97arhn
drrank: Aay
mng
AZC
0.00
0.25
0.50
0.75
1.00
Training step, t
1e6
Training step,
1e6
Maximum reward, Rmax
1.0
0.5
re-thatro
3
Training step,
1e7
Figure 10: Maximum reward of the group training step t (RT) in the (left) single path task
(middle) merging paths task (right) best-of-ten paths task
E.3 MEASURING INTER-GROUP AND INTRA-GROUP ALIGNMENT
We have so far captures the agreement between agents a group through the behavioral metric of conformity. Here, present a mnemonic metric for agreement, which term alignment. Align- ment is complementary metric the diversity (Df) and group diversity (Dद metrics, that aims capturing the effect of experience sharing on the replay buffers in group. We propose definition of alignment within a single group (intra-group alignmentA41 and a definition of alignment between two different groups (A,5,.0) Such metrics of mnemonic convergence have been linked to social network topology (Coman et al. (0166) and, as we show here, they can prove useful analyzing groups of reinforcement learning agents.
Specifically: (i) AT is the intra-group alignment. This metric captures the similarity in terms of content between the replay buffers of agents belonging to the same group. To compute this we compute the size of the common subset of experiences for each pair of agents and, then, average over all these pairs, normalizing in [0.1]. (ii) inter-group alignment A9;;9 is similar notion of alignment but employed between different groups (e.g. how different is group of fully-connected and dynamic group agents in terms of the content of their group replay buffers). To compute it we concatenate all replay buffers of a group into a single one and then compute the size of the common subset of the two replay buffers.
Figure 12 presents intra- group alignment in the three tasks. We observe that, in all tasks, intra- group alignment increases with connectivity and that it reduces when the agents enter the exploitation phase. Thus, intra-group alignment can prove useful in characterizing the explo- ration behavior of a group. In Figure [3 we present the inter-group alignment the single path, merging paths and best-of-ten paths tasks. We observe that the topologies do not differ significantly in the single path task. In the merging task, we observe that inter-group alignment during the exploration phase, compared to other tasks, and that the small-world is the slowest to align with all other structures. Perhaps this explains why this topology finds the optimal solution with the least probability: by propagating information quickly, the group early
lower
20

---

Under review as a conference paper at ICLR 2023
1.0
Average Diversity, D.
Conformity, Cr
0.8
400
0.6
no-sharing
200
dynamic
0.4
fully-connected
ring
0.2
0
small-world
0.00
0.25
0.50
0.75
1.00
2
3
Training step,
le6
Training step,
1e6
Average Diversity, D.
Conformity, C.
0.6
1000
0.4
500
0.2
0
8
0
Training step,
1e5
Training step,
1e5
Figure 11: Analyzing group behavior in the merging paths task (top row) and best-of- 10 paths task (bottom row). (left) Conformity C, a behavioral metric that denotes the percentage agents in group that followed the same trajectory in given evaluation trial (right) Average Diversity D, is a mnemonic metric that denotes the number of unique experiences in the replay buffer an agent, averaged over all agents.
21

---

Under review as a conference paper at ICLR 2023
Intra group alighment, AF
Intra-grous pro--grouppooo alignment. A3
AE
1.0
1.0
1.0
0.9
0.8
0.8
0.6
0.7
0.4
0.6
0.4
Intra
3
1
0.0
0.5
1.0
1.5
2.0
Training step,
le6
Training step,
1e6
Training step,
1e6
Figure 12: Intra-group alignment Ag in the single path task (left), merging paths task (middle) and best-of-ten paths task (right)
converges to the local optimum in this task. In the best-of-ten task, the no-sharing setting has the smalleett alignment with all other structures. This reinforces our main conclusion in this work: experience sharing affects individuals and different topologies do so in different ways.
E.4 ROBUSTNESS LEARNING HYPER-PARARE PARAMETERS
In Figure 14 we present how the performance of SAPIENS varies for different values of the learning hyperparameters learning rate and disocunt factor in the single path task under fully-connected and dynamic topolgoy, well as the no- no-sharing condition. We observe that, although convergence to the optimal solution is not always achieved, the dynamic topology is at least as effective as the others either in terms of convergence rate and/or final performance in all conditions.
E.5 EFFECT OF GROUP SIZE
We here examine the effect of the group size for all social network structures in the merging-patie patthi and best-of-ten paths task. To visualize the progression of group on the paths of the different tasks, we focus on specific elements in the tasks: (i) ([As,, Bs, C2] in the merging-paths task. The first two correspond to reaching the end of the paths corresponding to the two local optima. To reduce the computational complexity of experiments, we do not study the last element of the optimal path (C4), but focus on C2 instead. This is sufficient to detect whether a group has discovered the optimum path. Here, we observe that the fully-connected topology fails find the optimal path regardless of its size (with a small success probability for = 10). We observe that the ability of the ring. small-world and dynamict topologies to avoid the local optima improves with the group size (ii) [Ba, A2, Eg] in the best-of-ten tasks. B₁ is the fourth element on the optimal path (again we do not study the last element reduce complexity). To avoid cluttering the visualization we only present two of the nine sub-optimal paths. In this task, we again observe that the fully-connected network fails to discover the optimal task. Among all structures and group sizes, the large dynamic network performs best, while the performance of ring and small-world is also best for N 50. We observe that small networks sizes (N = 2, 6) are slower exploring (we can see that as they rarely find the second element of the sub-optimal paths, which is required to conclude that path B is the optimal choice).
Overall, this scaling analysis indicates that increasing the group size in a fully-connected topol- ogy will not improve performance, while benefits are expected for low-connectivity structures, particularly for the dynamic topology. We believe that this observation is crucial. In studies of groups of both human and artificial agents, we often encounter the conviction that, larger groups perform better and that size is a more important determinant than connectivity, the latter justifying why connectivity is often ignored Kline & Boyd (2010); Horgan et al. (2018); Mnih et al. (0016); Schmitt al. (2019); Nair al. (2015). Our results here point the contrary.
E.6 PRIORITIZED EXPERIENCE SHARING
We now examine how sharing prioritized experiences instead of randomly sampled ones affects the performance of SAPIENS. In Figure 16 we repeat the same experiment with Figure 4 with the dif- ference that all methods compute priorities, which they employ both for implementing prioritized
22

---

Under review as a conference paper at ICLR 2023
no-sharing
Inter-group alignment, & 8:0
dynamic nully-connected ring
1.0
small small-world
0.5
0.0
dynamic
Inter-group alignment,
independent fully-connected ring
1.0
small-world
0.5
0.0
fully-connected
Inter-group alignment,
independent
1.0
dynamic ring
R" & G)
0.5
0.0
ring
Inter-group alignment, (..0)
Independent
1.0
dynamic
fully connected small-world
0.5
0.0
small-world
Inter-group alignment, 5. G)
independent
1.0
dynamic
fully-connected ring
x"
0.5
0.0
3
1
2
2
Training step, le6
Training step, le6
Training step, t le6
Figure 13: Inter-group alignment A in the single path task (left), merging paths task (middle)
and best-of-ten paths task (right). each row we compare one topology with all the rest.
23

---

Under review as a conference paper at ICLR 2023
0.01, y=0.1
= 0.01, y=0.5
0.000
1.0
Maximum reward
0.6
0.4
0.2
0.0
A- 0.0011 y-0.1
0.001 y-0.5
0.001, V-0.9
1.0
0.8
0.6
0.4
0.2
A 0.00001 y=0.1
A ==00001, y=0.5
A= 0.00010 y=0.9
1.0
0.8
0.6
0.4
0.9
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Training
1c6
1c6
1e6
Figure 14: Varying the learning hyper- pepre-pprameters learning rate (A) and discount factor (7) in differ- social network topologies the single path task.
ring
fully-connected
small-world
dynamic
elementt of trials with element
1.00
N
0.75
N-10
N-20
0.50
N=50
0.25
0.00
A
8
B
C.2
A_ 8
B_
C_2
A_8
B.8
C_2
A_8
B_8
C_2
trials with element
1.00
0.75
Selementr %
0.50
5e
0.25
0.00
4
E element
A_
4 A_2
E.2 element
B A.2 element
E.2
B_4 A element
E
Figure 15: Scaling of different social network structures in the merging paths (top row) and best-of- ten paths tasks (bottom row). We highlight the element belonging to the optimal path in red.
24

---

Under review as a conference paper at ICLR 2023
Single path
Merging paths
Best-aftten paths
1.2
Group success
0.8
0.6
0.2
1e6
.Time to first success
6-
sharns dynammc
fdlly-oonnneee connected
smalliwer
2-
Figure 16: Examining the effect prioritization experience sharing. For more details about the setup, we refer the reader to Figure
replay buffer and sharing experiences by sampling them in proportion their priorities, As we see, using priorities negatively impacts experience sharing, while it helps speed up the performance of the single agent in the single path task. This behavior has been observed previous works [Souza] al. (2019) and can be attributed to the fact that the priorities of the sender do not necessarily agrec with the priorities the receiver and, therefore, destabilize learning.
E.7 ADDITIONAL TEST-BED: THE DECEPTIVE COINS GAME
Deceptive games are grid-workd tasks introduced test the ability of deep RL agents avoid local optima. (Bontrager et al.) 2019). Here, we perform preliminary experiments with our own JAX- based implementation of one of the games: the first difficulty level of the deceptive coins game (see Figure [7 for an illustration). Here, the agent can navigate in the grid-world during an episode and collect diamonds, which give a unit of reward. The game finishes once the agent reaches the fire, which offers additional reward, when timeout of 14 time steps is reached. There are two possible paths the agent can follow: moving left and reaching the fire will give reward of two while moving right and reaching the fire will give reward of five. The second path is more rewarding but is harder to complete because, once an agent discovers the easier-to-find diamond on the left, deceived into following the left path. Once agent commits grid-world) a barrier is raised so that the agent cannot go back within that episode.
path (reaches the edge of the
We now examine the performance of SAPIENS under different social network structures (fully- connected, small-world, ring, dynamic), as well as the no-sharing, A2C and Ape-X baselines for three group sizes: 6, 10 and 20 agents. We present the reward plots for the 3 sizes in Figures 18 19 and 20] respectively. and present an overall comparison in Figure 21 (equivalent to Figure for the Wordcraft tasks).
We observe that all conditions found either the local or the global optimum and that a) A2C fails for all network sizes. This behavior has been observed previous works (Bontrager et al. 2019) and can be attributed to the fact that policy-gradient methods are more susceptible to local minima b) no-sharing gets stuck in the local optimum in half of the trials when the group size small. Increasing the group size increases the probability that at least one agent in the group will escape the local minima by c-greedy exploration c) partially connected structures find the global minima across network sizes d) fully-connected converges the local optimum for the large group size, although the global optimum was discovered at the early exploration phase (see Figure 20). Thus, too much experience sharing harmful Ape-X fails with high probability for all network sizes.
25

---

Under review as a conference paper at ICLR 2023
Maximum reward, Rmax
6
4
2
0
APEX-ON
faly-comnccted
small-wos wolld
AZC
dynamic
0 100000 200000 300000 400000 Training step,
A screenshot of our implementa- tion of the Deceptive Coins task. Collecting diamonds gives a positive reward and touch- ing the fire terminates the game.
Figure 17:
Figure 18:
Performance for group with 6
agents
Maximum reward, Rmax
Maximum reward, Rmax
6
4
4
0
APEX DON
ReY coneoted
Smal Aol
no-shoring
rnn
dynami:
100000 200000 Training step, t
300000
0
200000 400000
600000
Training step, t
Figure 20: Performance for a group with 20
Figure 19: Performance for a group with
agents
agents
26

---

Under review as a conference paper at ICLR 2023
6 agents
10 agents
20 agents
Group success
2.0
2.0
2.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
Time to first success
1e6
1e6
1.0
1e6
0.8
Orann:
0.8
0.8
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0.0
0.0
Figure 21: Overall performance comparison for a group with: agents (first column), 10 agents (second column) and 20 agents (third column) task. We present two metrics: group success (S) denotes whether at least one agent in the group found the optimal solution (top row) and T+, Time to first success, is the number of training time steps required for this event (bottom row). Note that T can be computed only for S > 0 its error bars and significance tests can only be computer for S 1. We denote statistical significance levels with asterisks.)
In general, our conclusions in this task are consistent with what we observe in Wordcraft, in partic- ular the merging paths task that has similar deceptive nature.
E.8 ROBUSTNESS AMOUNT OF SHARING (ps AND I.g)
we formulated SAPIENS and described two sharing batch of experience tuples at the end of an episode and T,, the length of this batch. Here, we test the robustness of SAPIENS to these two of shared information and, therefore, interact with learning rate) to control the rate at which information is shared to the rate of individual learning. Specifically, we evaluate the dynamic topology (with the same hyper-parameters employed in the main paper, ie., visit duration T. 10 and probability of visit p. 0.05) and the colle-connected topology in the deceptive coins game (described Appendix E.7 with DQN agents.
In Section 2.3
is the probability of
which both control the amount of the DQNs (in particular the
In Figure 22 present group success (S) averaged across trials for parametric analysis over L, (1, 36) and Ps E (0.35, 0.7, 1). We observe that the dynamic topology finds the optimal solution across conditions except for small probability of failure for (I., 1, P. 0.35) and (L, 1,Px 0.7). These values correspond to low amounts of information sharing. In this case, the dynamic structure becomes more similar no-sharing structure: the amount of shared information is not enough to help the agents avoid local optima they fall into due to individual exploration. For the fully-connected topology we observe that performance degrades for high amounts of information ((L 36,p, = 0.35), (L 36,p. = 0.7). (L = 36,p, 1)). This is in accordance with our expectation that fully-connected topologies lead to convergence to local optima. Interestingly, this structure performs well when = 1 and T., < 6. Thus, sharing more frequently is better than sharing longer batches: we hypothesize that this is because longer batches have more correlated data, making convergence to local optima more probable.
27

---

Under review as a conference paper at ICLR 2023
S, group success
S, group success
Ds. Probabllity of sharing
1.00
Ps: Probability of sharing
1.0
1.0
0.75
0.75
0.7
0.50
0.50
0.25
0.25
0.35
0.35
0.00
0.00
1
36
6
36
Ls,
Ls,
Length of shared batch
Length of shared batch
Figure 22: Robustness group success sharing p, and L, for dynamic (left) and fully-connected (right)
28


Reviews:
[{"review_id": "as8P3cwvjH", "reviewer": "Reviewer_Gs4s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript leverages insights from the psychological and behavioral sciences to introduce a new algorithm improving exploration among a population of reinforcement learning agents. The psychological insights concern the effect of network structure on the effectiveness of group-level exploration and exploitation, as a function of task structure. The manuscript describes the algorithm and evaluates its effectiveness on an innovation task, applying several different network structures. The manuscript interprets the results of these experiments as evidence for the effectiveness of the algorithm over several reasonable baseline.", "summary_of_the_review": "Overall, the core idea is solid and backed by a wealth of evidence from psychology and related fields. However, my enthusiasm for the underlying proposal is tempered by the current version of the content, particularly the background review, statistical analysis, and discussion claims. Generally, I suspect the manuscript could be substantially improved with a thorough re-write and additional time spent on statistical analysis of the current experiments (with little-to-no need for additional experiments, perhaps aside from parameter robustness experiments, if the authors wanted to build more evidence for generality).", "strength_and_weaknesses": "I appreciated the chance to review this submission. This work has several core strengths:\n - The central idea—the application of network structure to group-level exploration and exploitation—is solid, and will likely have a positive impact on the field.\n - As the manuscript identifies, the cross-disciplinary connection with psychology and behavioral research provide strong initial support for this approach.\n - The manuscript provides an interesting environment to explore innovation. The configurability of Little Alchemy lends itself well to studying problems with different optima structures.\n- The experimental design is clean and can provide the empirical evidence needed to evaluate the proposed algorithm.\n\nHowever, I see multiple countervailing weaknesses. I’m enthusiastic about this direction, and so will try to provide specific suggestions for improvement and questions to guide revision. The following areas could be improved:\n - [major] The manuscript needs to clarify and sharpen its connection to prior psychology research.\n   - The fact that the current draft draws inspiration from cross-disciplinary research insights (and especially ones so relevant for population-based methods in RL) is a strength. A recurring claim throughout the submission is that its “contributions provide a better understanding of results originally obtained in human experiments.” I’m skeptical of this claim. How specifically do the RL findings from the present studies answer unanswered questions in psychology research? The manuscript does not effectively review and synthesize the modern state of knowledge in psychology, sociology, and organizational research on the topic. In reviewing work concerning human innovation, the manuscript samples heavily from recent work (e.g., Migliano & Vinicius, 2022; Momennejad et al., 2019; Momennejad, 2022), to the exclusion of important, foundational psych work on this particular problem, namely Mason, Jones, & Goldstone (2008), Mason and Watts (2011), and related organizational research like Fang, Lee, & Schilling (2010). (The current draft does well by including Lazer and Friedman.)\n   - With a proper review, it is hard to agree with the claim that this paper contributes to the psychology literature on human innovation, as the patterns observed in the present experiments are novel for single-agent RL, but already well-established in psychology and organizational research (e.g., “Our results show that topologies with low initial connectivity [...] performs [sic] best here by improving the exploration of different innovation paths”). More argumentative work is needed to establish a connection flowing from the present RL research to benefits for the psychology literature.\n - [major] The analyses and results that justify claims throughout section 3 are in need of substantial improvement. The current analysis does not do justice to the submission’s strong task and experimental design. Generally, the current version of the manuscript underspecifies its statistical analyses, including key details such as the data being used and the test being run.\n   - For example, the first paragraph of results states that “the performance of the dynamic structure is significantly better than all other baselines except for no-sharing (p-value 0.22) and small-world (p-value 0.07)”. What definition of performance are we using (especially given the four different metrics described in section 2.4)? At what point during learning are we measuring performance? What test are we using to derive statistical significance? Considering the last question, the first paragraph of section 3 mentions the use of “the Welch test”. I’m guessing this refers to a Welch’s t-test (though am not 100% sure), which compares two different samples. However, the authors are comparing *eight* different samples. I would not advise using t-tests in this situation, especially without correcting for multiple comparisons (I can’t find any details about a correction for multiplicity). Instead, the gold standard (parametric) approach would be to run an ANOVA and subsequently apply a comparative analysis like Tukey’s range test (with a correction for multiple corrections).\n   - As another example, section 3.2 includes the claim, “Perhaps surprisingly, the fully-connected group does not have the highest conformity (Figure 5, Left). We therefore conclude that there is an upper threshold for connectivity, beyond which shared experiences destabilize learning”. Looking at Figure 5, I’m fairly skeptical of this inference. The error bands in this figure (and many of the other figures with training curves) overlap heavily, making it difficult to believe such comparative claims. What statistical model can help provide empirical support for this claim? I think if the authors want to retain the the second sentence about the upper threshold for connectivity, “conclude” should be revised to “hypothesize”.\n   - I’ve chosen these two claims as examples, but broadly I’d encourage the authors to revisit their results section and design specific inferential tests for each of their hypotheses / claims (with corrections for multiple comparisons, where appropriate).\n - [major] The manuscript overstretches its claims in the conclusion. While I really like the overall approach, a key limitation for making general claims is the focus on one task / environment, the limited number of experiments and parameters, and the use of a single algorithm. I’d push the authors to gather much more evidence before make broad claims (e.g., “Based on our experimental results, we can provide general recommendations on which topology to use for which task class”), particularly seeking to test the generality and boundary conditions of the improvements they observe.\n - [major] The $p_s$ and $L_s$ parameters likely matter a lot, in combination with learning rate and other parameters / elements of algorithmic design. I might have missed it, but I didn't see any experiments testing the effects of changing these parameters, or text discussing their likely effect on performance. Similarly, the experiments take one particular approach to implementing a \"dynamic\" network. It’s easy to imagine many different approaches here, including adaptive structures that optimize for some of the metrics the manuscript discusses in Section 2.4. I’d encourage the authors to discuss the non-exclusitivity of their dynamic structure at some point in the manuscript.\n - [major] The manuscript repeatedly proposes a connection between its algorithm and “multi-agent” as a concept, including through “multi-agent topologies” and the “A” in SAPIENS (“multi-Agent”). However, I think it’d be much clearer for readers to discuss the proposed algorithm and current experiments as single-agent, since they take place in a single-agent RL task. The algorithm that the manuscript introduces is *technically* multi-agent, in the same way that classic population-based training (PBT) for single-agent tasks is “multi-agent”. Making a multi-agent connection in these situations is at best somewhat confusing, and at worst misleading. (I think it’s notable that PBT is rarely referred to as multi-agent, except when applied to multi-agent tasks; e.g., tasks that involve two or more agents when computing reward.) The potential for confusing or misleading readers emerges from the large body of multi-agent reinforcement learning research that *does* consider the effects of network structure, in contrast to recurring claims in the manuscript that RL studies “have not to date considered the effect of group connectivity”. Group connectivity was central to the AlphaStar league, for example (Vinyals et al., 2019), and is a common topic of research in the AAMAS community (e.g., Adjodah et al., 2020; Du et al., 2021; Garnelo et al., 2021). Consider how the discussion proposes “scaling up [the] study by applying SAPIENS in environments commonly employed by the multi-agent reinforcement learning community to study innovation”. How does this differ from the prior, parenthetical AAMAS references? Minimally, the discussion’s proposal should include those details. Overall, I strongly recommend reframing and revising the manuscript to clarify the single-agent nature of the task and to avoid this source of confusion.\n - [minor] Relatedly, PBT is a popular distributed framework that sharpens a learning agent’s exploitation (or, arguably, directs its exploration). PBT could be modified to instead sharpen exploration if the update rule took a non-fully connected approach, rather than copying policies from anywhere in the population of agents. I think it’d be worth including Jaderberg et al. (2019) in the review of “most solutions [that adopt] a fully-connected” approach.\n - [minor] On the behavioral metrics, I am skeptical about measuring conformity by examining the “percentage of agents [...] that followed the same trajectory”. If I understand the game correctly, trajectories can trivially differ by alternating between different branches, with no effect on the number of moves made, the ultimate score, or (in retrospect) the states visited. How would the conformity measure react to two agents in the merging-paths task that follow these paths: [A_1, B_1, A_2, B_2, C_1] and [A_1, A_2, B_1, B_2, C_1]?\n - [minor] The societal implications section is extremely broad, and does not ground its general claims. For example, the section does not justify or explain how the method for single-agent RL introduced here (or the specific results, above and beyond the knowledge already accumulated in the psychological sciences) will help with human-AI, AI-AI, or human-human cooperation. As discussed above, the innovation game is a single-agent task; if the agents are not knowingly or voluntarily exchanging experiences, are they really “cooperating” on the task? How do the current results, above and beyond prior psychology knowledge, contribute to human-AI collaboration? Similarly, much more argumentative work is needed to explain the algorithm's connection with “climate catastrophes and global pandemics”.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M4oe0yTJTd", "reviewer": "Reviewer_3Lpq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present study of the role of multi-agent topology on innovation towards goal of clarifying which social network structures are optimal for which innovation tasks, and which properties of experience sharing improve multi-level innovation. For multi-level hierarchical problem setting (WordCraft), three different innovation tasks were considered. The design networks of DQNs enables sharing experiences from their re- play buffers in varying structures (fully connected, small world, dynamic, ring). The level of innovation achieved by different setting, shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation across tasks. Second, experience sharing is not as helpful when there is a single clear path to innovation. For Third, two metrics we propose, conformity and diversity of shared experience, can explain the success of different social network structures on different tasks. ", "summary_of_the_review": "SAPIENS experiments show that dynamic topologies of experience sharing are best suited to solve complex innovation tasks\n\nboth multi-agent network topology and task structure affect the performance of SAPIENS. Based on our experimental results, we can provide general recommendations on which topology to use for which task class. \n\n- The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks. results show no benefit of experience sharing\n\n- The paper lays out how the various forms of the network interconnect settings considered performed in tasks that are individual (global and local optima same) or group, such as merging-path task. These exhibits strong local optima that requires explotation a certain point in order to discover the global optimum\n\nThe results also show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths. The dynamic topology shows up as the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one. Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones. The results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.\n\nBelow are some of the improvements I will like to suggest\n\n1. why 20 trials (referred in section 3) was deemed sufficient\n2. in dynamic network is perf. best because you already ran over the combinations many times and chose best interconnect (app a.3 has some details but unclear which one was used)\n3. how many steps were in each trial not indicated\n4. If first is the case then how does the conclusion follows (fig 4 sec 3.1):\nmerging paths task the performance of the dynamic structure is significantly better than all other baselines except for no-sharing (p-value 0.22) and small-world (p-value 0.07)…This indicates that, while learners that do not share experiences manage to solve the task with relative success, learners that share experiences under social networks combining large clustering and small shortest path perform best.\n5. Also, why the group diversity changes from Fig 5 to Fig 6 for singlepath task\n6. Minor: Figure 2 top row the merged path see first element as 5 rewards written in text but shows up as 8 in figure", "strength_and_weaknesses": "SAPIENS experiments show that dynamic topologies of experience sharing are best suited to solve complex innovation tasks\n\nboth multi-agent network topology and task structure affect the performance of SAPIENS. Based on our experimental results, we can provide general recommendations on which topology to use for which task class. \n\n- The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks. results show no benefit of experience sharing\n\n- The paper lays out how the various forms of the network interconnect settings considered performed in tasks that are individual (global and local optima same) or group, such as merging-path task. These exhibits strong local optima that requires explotation a certain point in order to discover the global optimum\n\nThe results also show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths. The dynamic topology shows up as the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one. Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones. The results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.\n\nBelow are some of the improvements I will like to suggest\n\n1. why 20 trials (referred in section 3) was deemed sufficient\n2. in dynamic network is perf. best because you already ran over the combinations many times and chose best interconnect (app a.3 has some details but unclear which one was used)\n3. how many steps were in each trial not indicated\n4. If first is the case then how does the conclusion follows (fig 4 sec 3.1):\nmerging paths task the performance of the dynamic structure is significantly better than all other baselines except for no-sharing (p-value 0.22) and small-world (p-value 0.07)…This indicates that, while learners that do not share experiences manage to solve the task with relative success, learners that share experiences under social networks combining large clustering and small shortest path perform best.\n5. Also, why the group diversity changes from Fig 5 to Fig 6 for singlepath task\n6. Minor: Figure 2 top row the merged path see first element as 5 rewards written in text but shows up as 8 in figure", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KXZSOKGWbEe", "reviewer": "Reviewer_DXUd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates how network topology influences solving of different kinds of problems by DQN learners who exchange experience by sharing experience from their replay buffers. The paper specifically investigates network structures including  fully-connected, small world, ring and dynamic and tasks that vary in requiring a single innovation, multiple merged innovations, and broader search spaces. Performance is measured via a variety of metrics spanning performance, behavior, and \"mnemonics\". The results include some phenomena that might be predicted based on the existing human behavior literature, as well as several surprises. Overall, the authors conclude that both network topology and problem structure affect innovation in DQN agents. ", "summary_of_the_review": "On the one hand, the paper asks a timely and interesting question, and does so in an interesting way. However, I also believe the paper would benefit from extensive revision. There are a large number of minor typos, bad citations, and broken references, some but not all of which I note above. The claims of the paper are not totally clear; for example, see inconsistencies between the open questions and the hypothesis (which appears at the end). The statistical analyses are incompletely documented and are questionable. I was confused about what even is SAPIENS? (Does it need to be named?) Ultimately, the paper's claims appear to be primarily descriptive in nature, but I believe the authors would like them to be stronger (from the grand scope they invoke). Like many big ideas papers, the paper suffers for not having a clear enough focus about what precisely is the contribution and marshalling the evidence in a clear, compelling, and organized fashion to support that contribution. I believe this could be an excellent and pathbreaking paper, given substantial work to focus, edit, revise, and clarify. ", "strength_and_weaknesses": "Strengths: \n- The paper asks a big, interesting question, and does so in an interesting fashion. Specifically, the social nature of innovation is bedrock in the literature on human learning. Yet, we know relatively little about this question in the multi-agent RL setting. It is indeed an important, and interesting direction. \n- The paper varies two simple, yet central features: the network topology and the nature of the problem. Building from the human learning (and other) literature, the test cases are informed by evidence and interesting. \n- The method of sharing experience is quite natural for DQN learners, samples from the replay buffer, which is nicer than other work which shares gradients.\n- There are a large number of experiments reported. \n\nWeaknesses:  \n- \"share experience tuples from their replay buffers\" This is a substantial theoretical commitment regarding the form of knowledge and the nature of communication. It is important to discuss limitations that arise from these particular choices, especially with regard to generalization to human-AI and human-human innovation. \n- \"agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents\" It is not clear why exchanging transitions from reply buffers is less ad hoc. \n- \"which properties of experience sharing improve multi-level innovation\" Is this question answered? I am unclear on what constitutes properties of experience in the context of the simulations. (I am assuming that is must be different from the network structures and tasks, as that is the first unresolved question.)\n- Paper is ultimately more descriptive than conclusive. In principle, this need not be a weakness. However, there appear to be many opportunities to investigate further that were not taken. Also, the claims of the paper, especially related to implications outside of this model and these tasks, seem to require stronger causal understanding of why the phenomena occur. \n- I am confused about what exactly SAPIENS is meant to be? Is it an experimental testbed? An optimization toolbox? A novel algorithm? \n- \" two metrics we propose, conformity and diversity of shared experience, can explain the success of different social network structures on different tasks\" Is this intended as a correlational or causal claim. The language is ambiguous, but it evokes a causal interpretation. I did not see causal evidence however. \n- \"Here we test the hypothesis that the topology of experience sharing in a group of deep RL agents can shape its performance using our proposed algorithm SAPIENS.\" This hypothesis is first stated in the discussion. It doesn't seem to map onto the two open questions posed in the abstract? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CqOXlr0PQHL", "reviewer": "Reviewer_GCRF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an investigation of how to network reinforcement learning models to accomplish symbolic innovation tasks. The paper analyzes several different, common network topologies as a means of sharing experiences between DQN learners as well as other existing methods that either share experiences or gradients between the learners. The paper considers three different types of innovation paths as well as a ‘deceptive task’ as part of its evaluation of the different sharing configurations. The paper also offers insights into why some network structures work (or do not work) for different tasks as well as some practical guidance for actual usage.", "summary_of_the_review": "I believe this paper merits acceptance as it is a high-quality paper, with a significant contribution that would be of interest to the community. The work in the paper on how to combine multi-agent reinforcement learners is a significant problem. And the experimental regime used well supports the conclusions drawn in the paper. \n\n--------After Reviewing Period--------------\nFollowing the discussion period and consultation with the other reviewers and chair, I have revised my assessment and lowered my recommendation.", "strength_and_weaknesses": "The paper is a generally strong paper. It is attacking a significant problem or interest to the community (i.e. how to do multi-agent better reinforcement learning), is well grounded in both network and RL literature and methods, and has some important contributions. I consider the insight into why certain network topologies work as well to be an especially important contribution, as it could guide future research. I also very much appreciate that the insights drawn in the paper are based on appropriate metrics, like the volatility and diversity metrics. \n\nThe weaknesses of the paper are few. First, the setup for the evaluation seems rather simple and contrived (discrete tasks and only a few path combinations). It doesn’t have any uncertainty related to task performance or much complexity in the task paths. However, the paper does address this in the limitations, and I don’t consider this as a serious issue since one has to start somewhere, and stating with a simple base-case is good science. Second, figure 4 is not very clear. Does the lack of bars indicate total task failure for a method, especially in the Best-of-Ten paths? Also, the significance brackets – do the endpoints of the bracket indicate a significant difference between the methods at the endpoints? And, what do the stars indicate (I presume the number of stars is the level of statistical significance)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject