Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present study of the role of multi-agent topology on innovation towards goal of clarifying which social network structures are optimal for which innovation tasks, and which properties of experience sharing improve multi-level innovation. For multi-level hierarchical problem setting (WordCraft), three different innovation tasks were considered. The design networks of DQNs enables sharing experiences from their re- play buffers in varying structures (fully connected, small world, dynamic, ring). The level of innovation achieved by different setting, shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation across tasks. Second, experience sharing is not as helpful when there is a single clear path to innovation. For Third, two metrics we propose, conformity and diversity of shared experience, can explain the success of different social network structures on different tasks. 
strength_and_weaknesses: SAPIENS experiments show that dynamic topologies of experience sharing are best suited to solve complex innovation tasks

both multi-agent network topology and task structure affect the performance of SAPIENS. Based on our experimental results, we can provide general recommendations on which topology to use for which task class. 

- The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks. results show no benefit of experience sharing

- The paper lays out how the various forms of the network interconnect settings considered performed in tasks that are individual (global and local optima same) or group, such as merging-path task. These exhibits strong local optima that requires explotation a certain point in order to discover the global optimum

The results also show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths. The dynamic topology shows up as the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one. Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones. The results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.

Below are some of the improvements I will like to suggest

1. why 20 trials (referred in section 3) was deemed sufficient
2. in dynamic network is perf. best because you already ran over the combinations many times and chose best interconnect (app a.3 has some details but unclear which one was used)
3. how many steps were in each trial not indicated
4. If first is the case then how does the conclusion follows (fig 4 sec 3.1):
merging paths task the performance of the dynamic structure is significantly better than all other baselines except for no-sharing (p-value 0.22) and small-world (p-value 0.07)…This indicates that, while learners that do not share experiences manage to solve the task with relative success, learners that share experiences under social networks combining large clustering and small shortest path perform best.
5. Also, why the group diversity changes from Fig 5 to Fig 6 for singlepath task
6. Minor: Figure 2 top row the merged path see first element as 5 rewards written in text but shows up as 8 in figure
clarity,_quality,_novelty_and_reproducibility: The work seems original and quality/novelty is adequate
summary_of_the_review: SAPIENS experiments show that dynamic topologies of experience sharing are best suited to solve complex innovation tasks

both multi-agent network topology and task structure affect the performance of SAPIENS. Based on our experimental results, we can provide general recommendations on which topology to use for which task class. 

- The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks. results show no benefit of experience sharing

- The paper lays out how the various forms of the network interconnect settings considered performed in tasks that are individual (global and local optima same) or group, such as merging-path task. These exhibits strong local optima that requires explotation a certain point in order to discover the global optimum

The results also show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths. The dynamic topology shows up as the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one. Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones. The results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.

Below are some of the improvements I will like to suggest

1. why 20 trials (referred in section 3) was deemed sufficient
2. in dynamic network is perf. best because you already ran over the combinations many times and chose best interconnect (app a.3 has some details but unclear which one was used)
3. how many steps were in each trial not indicated
4. If first is the case then how does the conclusion follows (fig 4 sec 3.1):
merging paths task the performance of the dynamic structure is significantly better than all other baselines except for no-sharing (p-value 0.22) and small-world (p-value 0.07)…This indicates that, while learners that do not share experiences manage to solve the task with relative success, learners that share experiences under social networks combining large clustering and small shortest path perform best.
5. Also, why the group diversity changes from Fig 5 to Fig 6 for singlepath task
6. Minor: Figure 2 top row the merged path see first element as 5 rewards written in text but shows up as 8 in figure
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript leverages insights from the psychological and behavioral sciences to introduce a new algorithm improving exploration among a population of reinforcement learning agents. The psychological insights concern the effect of network structure on the effectiveness of group-level exploration and exploitation, as a function of task structure. The manuscript describes the algorithm and evaluates its effectiveness on an innovation task, applying several different network structures. The manuscript interprets the results of these experiments as evidence for the effectiveness of the algorithm over several reasonable baseline.
strength_and_weaknesses: I appreciated the chance to review this submission. This work has several core strengths:
 - The central idea—the application of network structure to group-level exploration and exploitation—is solid, and will likely have a positive impact on the field.
 - As the manuscript identifies, the cross-disciplinary connection with psychology and behavioral research provide strong initial support for this approach.
 - The manuscript provides an interesting environment to explore innovation. The configurability of Little Alchemy lends itself well to studying problems with different optima structures.
- The experimental design is clean and can provide the empirical evidence needed to evaluate the proposed algorithm.

However, I see multiple countervailing weaknesses. I’m enthusiastic about this direction, and so will try to provide specific suggestions for improvement and questions to guide revision. The following areas could be improved:
 - [major] The manuscript needs to clarify and sharpen its connection to prior psychology research.
   - The fact that the current draft draws inspiration from cross-disciplinary research insights (and especially ones so relevant for population-based methods in RL) is a strength. A recurring claim throughout the submission is that its “contributions provide a better understanding of results originally obtained in human experiments.” I’m skeptical of this claim. How specifically do the RL findings from the present studies answer unanswered questions in psychology research? The manuscript does not effectively review and synthesize the modern state of knowledge in psychology, sociology, and organizational research on the topic. In reviewing work concerning human innovation, the manuscript samples heavily from recent work (e.g., Migliano & Vinicius, 2022; Momennejad et al., 2019; Momennejad, 2022), to the exclusion of important, foundational psych work on this particular problem, namely Mason, Jones, & Goldstone (2008), Mason and Watts (2011), and related organizational research like Fang, Lee, & Schilling (2010). (The current draft does well by including Lazer and Friedman.)
   - With a proper review, it is hard to agree with the claim that this paper contributes to the psychology literature on human innovation, as the patterns observed in the present experiments are novel for single-agent RL, but already well-established in psychology and organizational research (e.g., “Our results show that topologies with low initial connectivity [...] performs [sic] best here by improving the exploration of different innovation paths”). More argumentative work is needed to establish a connection flowing from the present RL research to benefits for the psychology literature.
 - [major] The analyses and results that justify claims throughout section 3 are in need of substantial improvement. The current analysis does not do justice to the submission’s strong task and experimental design. Generally, the current version of the manuscript underspecifies its statistical analyses, including key details such as the data being used and the test being run.
   - For example, the first paragraph of results states that “the performance of the dynamic structure is significantly better than all other baselines except for no-sharing (p-value 0.22) and small-world (p-value 0.07)”. What definition of performance are we using (especially given the four different metrics described in section 2.4)? At what point during learning are we measuring performance? What test are we using to derive statistical significance? Considering the last question, the first paragraph of section 3 mentions the use of “the Welch test”. I’m guessing this refers to a Welch’s t-test (though am not 100% sure), which compares two different samples. However, the authors are comparing *eight* different samples. I would not advise using t-tests in this situation, especially without correcting for multiple comparisons (I can’t find any details about a correction for multiplicity). Instead, the gold standard (parametric) approach would be to run an ANOVA and subsequently apply a comparative analysis like Tukey’s range test (with a correction for multiple corrections).
   - As another example, section 3.2 includes the claim, “Perhaps surprisingly, the fully-connected group does not have the highest conformity (Figure 5, Left). We therefore conclude that there is an upper threshold for connectivity, beyond which shared experiences destabilize learning”. Looking at Figure 5, I’m fairly skeptical of this inference. The error bands in this figure (and many of the other figures with training curves) overlap heavily, making it difficult to believe such comparative claims. What statistical model can help provide empirical support for this claim? I think if the authors want to retain the the second sentence about the upper threshold for connectivity, “conclude” should be revised to “hypothesize”.
   - I’ve chosen these two claims as examples, but broadly I’d encourage the authors to revisit their results section and design specific inferential tests for each of their hypotheses / claims (with corrections for multiple comparisons, where appropriate).
 - [major] The manuscript overstretches its claims in the conclusion. While I really like the overall approach, a key limitation for making general claims is the focus on one task / environment, the limited number of experiments and parameters, and the use of a single algorithm. I’d push the authors to gather much more evidence before make broad claims (e.g., “Based on our experimental results, we can provide general recommendations on which topology to use for which task class”), particularly seeking to test the generality and boundary conditions of the improvements they observe.
 - [major] The $p_s$ and $L_s$ parameters likely matter a lot, in combination with learning rate and other parameters / elements of algorithmic design. I might have missed it, but I didn't see any experiments testing the effects of changing these parameters, or text discussing their likely effect on performance. Similarly, the experiments take one particular approach to implementing a "dynamic" network. It’s easy to imagine many different approaches here, including adaptive structures that optimize for some of the metrics the manuscript discusses in Section 2.4. I’d encourage the authors to discuss the non-exclusitivity of their dynamic structure at some point in the manuscript.
 - [major] The manuscript repeatedly proposes a connection between its algorithm and “multi-agent” as a concept, including through “multi-agent topologies” and the “A” in SAPIENS (“multi-Agent”). However, I think it’d be much clearer for readers to discuss the proposed algorithm and current experiments as single-agent, since they take place in a single-agent RL task. The algorithm that the manuscript introduces is *technically* multi-agent, in the same way that classic population-based training (PBT) for single-agent tasks is “multi-agent”. Making a multi-agent connection in these situations is at best somewhat confusing, and at worst misleading. (I think it’s notable that PBT is rarely referred to as multi-agent, except when applied to multi-agent tasks; e.g., tasks that involve two or more agents when computing reward.) The potential for confusing or misleading readers emerges from the large body of multi-agent reinforcement learning research that *does* consider the effects of network structure, in contrast to recurring claims in the manuscript that RL studies “have not to date considered the effect of group connectivity”. Group connectivity was central to the AlphaStar league, for example (Vinyals et al., 2019), and is a common topic of research in the AAMAS community (e.g., Adjodah et al., 2020; Du et al., 2021; Garnelo et al., 2021). Consider how the discussion proposes “scaling up [the] study by applying SAPIENS in environments commonly employed by the multi-agent reinforcement learning community to study innovation”. How does this differ from the prior, parenthetical AAMAS references? Minimally, the discussion’s proposal should include those details. Overall, I strongly recommend reframing and revising the manuscript to clarify the single-agent nature of the task and to avoid this source of confusion.
 - [minor] Relatedly, PBT is a popular distributed framework that sharpens a learning agent’s exploitation (or, arguably, directs its exploration). PBT could be modified to instead sharpen exploration if the update rule took a non-fully connected approach, rather than copying policies from anywhere in the population of agents. I think it’d be worth including Jaderberg et al. (2019) in the review of “most solutions [that adopt] a fully-connected” approach.
 - [minor] On the behavioral metrics, I am skeptical about measuring conformity by examining the “percentage of agents [...] that followed the same trajectory”. If I understand the game correctly, trajectories can trivially differ by alternating between different branches, with no effect on the number of moves made, the ultimate score, or (in retrospect) the states visited. How would the conformity measure react to two agents in the merging-paths task that follow these paths: [A_1, B_1, A_2, B_2, C_1] and [A_1, A_2, B_1, B_2, C_1]?
 - [minor] The societal implications section is extremely broad, and does not ground its general claims. For example, the section does not justify or explain how the method for single-agent RL introduced here (or the specific results, above and beyond the knowledge already accumulated in the psychological sciences) will help with human-AI, AI-AI, or human-human cooperation. As discussed above, the innovation game is a single-agent task; if the agents are not knowingly or voluntarily exchanging experiences, are they really “cooperating” on the task? How do the current results, above and beyond prior psychology knowledge, contribute to human-AI collaboration? Similarly, much more argumentative work is needed to explain the algorithm's connection with “climate catastrophes and global pandemics”.
clarity,_quality,_novelty_and_reproducibility: The quality of the manuscript is mixed: the central proposal, task, and study design are all strong points in favor of research quality, but are undercut by substantial weaknesses in the statistical analysis used to infer the effectiveness of the proposed algorithm. See detailed comments in prior comment for more detail.

Similarly, the clarity and novelty of the current manuscript leave room for improvement. As identified in the prior section, the manuscript does not sharply communicate its relationship to prior psychology and social science research, to the detriment of claims made throughout the paper.
summary_of_the_review: Overall, the core idea is solid and backed by a wealth of evidence from psychology and related fields. However, my enthusiasm for the underlying proposal is tempered by the current version of the content, particularly the background review, statistical analysis, and discussion claims. Generally, I suspect the manuscript could be substantially improved with a thorough re-write and additional time spent on statistical analysis of the current experiments (with little-to-no need for additional experiments, perhaps aside from parameter robustness experiments, if the authors wanted to build more evidence for generality).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates how network topology influences solving of different kinds of problems by DQN learners who exchange experience by sharing experience from their replay buffers. The paper specifically investigates network structures including  fully-connected, small world, ring and dynamic and tasks that vary in requiring a single innovation, multiple merged innovations, and broader search spaces. Performance is measured via a variety of metrics spanning performance, behavior, and "mnemonics". The results include some phenomena that might be predicted based on the existing human behavior literature, as well as several surprises. Overall, the authors conclude that both network topology and problem structure affect innovation in DQN agents. 
strength_and_weaknesses: Strengths: 
- The paper asks a big, interesting question, and does so in an interesting fashion. Specifically, the social nature of innovation is bedrock in the literature on human learning. Yet, we know relatively little about this question in the multi-agent RL setting. It is indeed an important, and interesting direction. 
- The paper varies two simple, yet central features: the network topology and the nature of the problem. Building from the human learning (and other) literature, the test cases are informed by evidence and interesting. 
- The method of sharing experience is quite natural for DQN learners, samples from the replay buffer, which is nicer than other work which shares gradients.
- There are a large number of experiments reported. 

Weaknesses:  
- "share experience tuples from their replay buffers" This is a substantial theoretical commitment regarding the form of knowledge and the nature of communication. It is important to discuss limitations that arise from these particular choices, especially with regard to generalization to human-AI and human-human innovation. 
- "agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents" It is not clear why exchanging transitions from reply buffers is less ad hoc. 
- "which properties of experience sharing improve multi-level innovation" Is this question answered? I am unclear on what constitutes properties of experience in the context of the simulations. (I am assuming that is must be different from the network structures and tasks, as that is the first unresolved question.)
- Paper is ultimately more descriptive than conclusive. In principle, this need not be a weakness. However, there appear to be many opportunities to investigate further that were not taken. Also, the claims of the paper, especially related to implications outside of this model and these tasks, seem to require stronger causal understanding of why the phenomena occur. 
- I am confused about what exactly SAPIENS is meant to be? Is it an experimental testbed? An optimization toolbox? A novel algorithm? 
- " two metrics we propose, conformity and diversity of shared experience, can explain the success of different social network structures on different tasks" Is this intended as a correlational or causal claim. The language is ambiguous, but it evokes a causal interpretation. I did not see causal evidence however. 
- "Here we test the hypothesis that the topology of experience sharing in a group of deep RL agents can shape its performance using our proposed algorithm SAPIENS." This hypothesis is first stated in the discussion. It doesn't seem to map onto the two open questions posed in the abstract? 
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality (mostly): 
- "ones(Hafner, 2021) b)" missing space 
- "Watts & Strogatz (1998)." Citation should be in parentheses
- "vast spaces.."
- "and receives a reward that increases monotonically with levels" Missing period
- "spread between clusters" missing period
- The phrase "deceptive game" is misleading. The games do not have agency. Better to choose something like difficult, or non-obvious, or similar. 
- Derex & Boyd (2016); Migliano & Vinicius (2022) 
- "Specifically, methods ring, small-world, dynamic and fully-connected are instantiations of SAPIENS for different social network structures with 10 DQNs. " Confusing sentence. Are the italics important? Are these methods? 
- When reporting p values, please report the full details that went into the calculation, including the means (and variances as appropriate), test statistic, degrees of freedom, the name of the test used, etc. 
- "In contrast, fully-connected, A2C and Ape-X perform significantly worse. " This claim needs to be quantified. 
- "In additional experiments we have also:" I find it unacceptable to introduce results without the methods and evidence. (Even if they are included in the Appendix.)
- "we hypothesize that it is due to the fact that priorities computed by an agent do not necessarily agree with the priorities of the other" What do you mean, why do you believe this, and how could it be tested?
- "We denote statistical significance levels with asterisks.)" This is not acceptable. Please state explicitly what the number of stars reflects. Also, given the enormous number of tests is there a preregistered evaluation plan? Have the analyses been corrected for the number of tests? Also, it is not clear that the trailing parenthesis is closing anything. 
- "While it is not surprising that a single agent with epsilon-greedy exploration can efficiently solve this task, it is not clear at first glance why experience sharing harms performance. Notably this phenomenon has been observed in related works (Souza et al., 2019; Schmitt et al., 2019)." Is this intended to explain the surprise? It would be nice to provide some explanation for the observations. 
- "Perhaps surprisingly, the fully-connected group does not have the highest conformity (Figure 5, Left). We therefore conclude that there is an upper threshold for connectivity, beyond which shared experiences destabilize learning." Why is the conclusion warranted? I do not follow the logic. 
- "Finally, when looking at the group diversity in Figure 6, fully-connected ranks last and the dynamic topology ranks first. This is rather surprising: the" It would be nice to have experiments designed to test the observation. 
- "Appendix ¡REF¿."
summary_of_the_review: On the one hand, the paper asks a timely and interesting question, and does so in an interesting way. However, I also believe the paper would benefit from extensive revision. There are a large number of minor typos, bad citations, and broken references, some but not all of which I note above. The claims of the paper are not totally clear; for example, see inconsistencies between the open questions and the hypothesis (which appears at the end). The statistical analyses are incompletely documented and are questionable. I was confused about what even is SAPIENS? (Does it need to be named?) Ultimately, the paper's claims appear to be primarily descriptive in nature, but I believe the authors would like them to be stronger (from the grand scope they invoke). Like many big ideas papers, the paper suffers for not having a clear enough focus about what precisely is the contribution and marshalling the evidence in a clear, compelling, and organized fashion to support that contribution. I believe this could be an excellent and pathbreaking paper, given substantial work to focus, edit, revise, and clarify.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an investigation of how to network reinforcement learning models to accomplish symbolic innovation tasks. The paper analyzes several different, common network topologies as a means of sharing experiences between DQN learners as well as other existing methods that either share experiences or gradients between the learners. The paper considers three different types of innovation paths as well as a ‘deceptive task’ as part of its evaluation of the different sharing configurations. The paper also offers insights into why some network structures work (or do not work) for different tasks as well as some practical guidance for actual usage.
strength_and_weaknesses: The paper is a generally strong paper. It is attacking a significant problem or interest to the community (i.e. how to do multi-agent better reinforcement learning), is well grounded in both network and RL literature and methods, and has some important contributions. I consider the insight into why certain network topologies work as well to be an especially important contribution, as it could guide future research. I also very much appreciate that the insights drawn in the paper are based on appropriate metrics, like the volatility and diversity metrics. 

The weaknesses of the paper are few. First, the setup for the evaluation seems rather simple and contrived (discrete tasks and only a few path combinations). It doesn’t have any uncertainty related to task performance or much complexity in the task paths. However, the paper does address this in the limitations, and I don’t consider this as a serious issue since one has to start somewhere, and stating with a simple base-case is good science. Second, figure 4 is not very clear. Does the lack of bars indicate total task failure for a method, especially in the Best-of-Ten paths? Also, the significance brackets – do the endpoints of the bracket indicate a significant difference between the methods at the endpoints? And, what do the stars indicate (I presume the number of stars is the level of statistical significance)?

clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written but could use some improvements in some of its figures (see the weaknesses section). The quality of the paper and reproducibility are also high; there is code and it is clear how to reproduce the experiments in the paper based on the details contained in the paper and its appendices.
summary_of_the_review: I believe this paper merits acceptance as it is a high-quality paper, with a significant contribution that would be of interest to the community. The work in the paper on how to combine multi-agent reinforcement learners is a significant problem. And the experimental regime used well supports the conclusions drawn in the paper. 

--------After Reviewing Period--------------
Following the discussion period and consultation with the other reviewers and chair, I have revised my assessment and lowered my recommendation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Inspired by the cognitive science literature, this paper proposes a new distributed RL setup in which there are multiple actor/learner processes arranged in a graph topology such that adjacent processes in the graph share experience tuples with each other. The paper explores the performance of multiple different topologies (e.g. fully-connected, ring, small-world, dynamic) in three different tasks (single path, merging paths, and best-of-ten). While in the single path task (where there is a single, optimal path to the goal) all distributed topologies perform comparably well, in the other tasks the non-fully-connected topologies---and in particular the dynamic topology---perform better. The paper performs a number of analyses to better understand why this is the case, and argues that different distributed topologies could be useful for deep RL in general, as well as being useful as a modeling tool in cognitive science.

The reviewers agreed that the core idea of the paper---changing the way that distributed RL agents share experience via different network topologies---was compelling and important. However, the reviewers also agreed that the paper suffered from a lack of focus. During the discussion, the reviewers acknowledged the revisions made by the authors and felt like the paper was improved to some degree (especially in terms of toning down some of the claims, and the statistical analysis), but that a deeper issue remained regarding the focus of the paper. In particular, the reviewers felt that the paper wanted to make a claim about significance both from ML and CogSci standpoints, yet failed to do so convincingly along either dimension (especially on the CogSci side, though I also felt it could be more convincing from the ML side if experiments were performed on more canonical and challenging domains, e.g. Atari). One suggestion that came out of the discussion was to actually split the paper into two: one focused for a ML audience, and one focused for a CogSci audience, which would help sharpen the focus and claims. Regardless, it was agreed that the paper needed a more holistic revision going beyond the additions/modifications done during the rebuttal before it would be ready for publication at ICLR.

I recommend rejection at this stage, and encourage the authors to undertake a more extensive revision of the paper. I think with an improved manuscript that is more precise about the results and impact, this could be a really fantastic piece of work, so I do hope the authors will continue working on it!
justification_for_why_not_higher_score: While the paper pitches itself as an interdisciplinary work between ML and CogSci, I feel that it does not live up to the standards of either field. On the ML side, the experiments are somewhat toy and it's not clear whether the approach would scale to more challenging RL environments. Despite the author's revisions, the statistics also still feel imprecise which undermines the potential impact. On the CogSci side, the impacts are more theoretical and not demonstrated in practice. Overall, the paper is unfocused and it is not clear what the central claims are. This therefore limits its potential interest and impact at ICLR.
Decision: Reject