Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: After author rebuttal: 
Thank you for your response to the review. I appreciate your point that sport applications can be interesting to this community, and your paper makes an important contribution in that direction. With that said, for a paper to be appealing to the ICLR audience, there should be more clear novelty in the method, and stronger experimental evaluation showing novel insights or out-performance over previous baselines. 


==================================================================
The paper addresses a problem in the context of understanding formations of sport player in a play field. The problem is that players form an ומ-ordered sets of entities and to analyze them, one would like to keep some canonical representation that allows us to compare formations across time. The idea of this paper is to assign roles to each players (name each member of the set) based on the distribution of their spatial locations on a field. Once this representation is learned, future work may use it for downstream problems, which are not discussed in this paper. 

General: 

The paper is very clearly written, the method makes sense and the related work are both well explained. 
I have two main concerns: First, the scope of the paper may be too narrow for a conference like ICLR, as the method part is fundamentally a clustering method, and the application is very specific. Second, I was expecting that the paper shows interesting results that can be obtained once the representation is learned, but this are lacking from the paper. 
Based on these two concerns, I dont think the paper would be a good fit for ICLR. 

To make the paper more appealing to the ML community, my recommendation to the authors is to  show what can be achieved with the new representation, that was impossible with previous representations. For example, it is claimed that faster learning allows to compute it season wide and on-the-fly. What are useful tasks that your rep' can help with? How do these tasks perform with the new rep? 

Other comments and questions. 

-- What about other competing representations. Would it be useful to look at movement trajectories instead of distribution of isolated locations (cluster lines instead of points)? 
-- What happens when a player is replaced by another player of the same role. Wouldnt its distribution change? Dont you want your rep' to reflect such events? 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper authors describe a way to quickly model interaction between many entities, in the form of mixture of Gaussians. They provide huge computational complexity improvements over Bialkowski et al. work, which allows them to analyse big datasets.

Paper, in its current form, lacks quantitative evaluation, comparison to other methods (heavily cited in the paper), and any well established applications that can be quantified. Given that the method described is a composition of many known techniques, and some new tricks, it is hard to assess the value of these contributions without seeing more unified evaluation methodology. All the provided results (apart from computational complexity wrt. specific approach, which are quite amazing in isolation!) are qualitative and descriptive in nature, which together with the fact of being applied to a single, specific dataset, with makes it hard to judge any generality. This is a common problem with various "structure discovery methods" such as clustering techniques, but an important one to address.

It is also unclear to reviewer, whether the only (main?) claim authors want to make is that they provide speed-up to the existing solution. if this was the case, it is really hard to see how each of the proposed steps guarantees the same outputs as previous approaches, while at the same time only affecting speed. In particular authors make the claim of improvement stability, yet do not provide empirical evaluation showing these benefits over previous methods. 

Please compare these to results reported e.g. in the cited work of Yeh et al. "Diverse Generation for Multi-agent Sports Games".

Overall, in reviewer's opinion, results looks quite incremental, and are heavily concentrated around specific paper of Bialkowski et al., and as such might see limited interest in the ICLR community.

Minor comments:
- please unify use of quotations (authors use closing quotations for both opening and closing quotes)
- is there an extra | symbol in eq. (4)?


Please note, that reviewer has little experience in this particular branch of the field, and thus some of the issues mentioned could come from misunderstanding the standards.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
  *Synopsis*:
  The paper proposes a new algorithm for template discovery and representation learning for unstructured multi-agent data with strong group roles (i.e. sports data). Their algorithm is several orders of magnitude faster because of a key assumption in the underlying structure of the position generating distributions of players conditioned on roles. They also formulate the problem in a novel way, splitting the structuring of the data into a three-step approach with each component requiring a different algorithm. This will allow the community to swap in new algorithms into the different components more simply than before. They finally provide some evidence of their algorithm working on a soccer dataset consisting of 380 games.

  Main Contributions:
  - A key insight into the underlying structure of the data, enabling an algorithm with orders of magnitude speed up.
  - A new framing of the problem.

  *Review*:

  The paper is well written, and seems to be a nice improvement over prior art in terms of runtime. I am not very familiar with the surrounding literature, but had a few concerns over the current paper that I feel should be addressed. Overall, I think this paper could be accepted unless other reviewers more familiar with the literature find critical flaws. I would, however, like to see Q2 addressed by the authors.

  Q1 It seems the key insight is that you are modeling the occupancy of space by an agent as a mixture of independent gaussian distributions. How reasonable is the assumption that each generating distribution conditioned on a role is independent of the others? Wouldn't a roles place on the field be dependent by the placement of other roles?

  Q2 How does the performance (i.e. accuracy) of your algorithm compare to prior approaches (i.e. the Sha or Bialkowski)? Or really any baseline? It is hard to judge this method in isolation (other than the impressive computational gains), especially for someone unfamiliar with this literature. Is there a way you could include a comparison in the final draft? If not, why? 
      As a note: It seems you include some comparisons in section 4.3, 4.4, and 4.5, but it is unclear to me what the competitor is (i.e. A.3) and how it relates to the surrounding literature. If this is related to prior art, you should make it clear what algorithm this is in the main text, not relegating all discussion to the appendix.

  C1: It would be nice to include table 1 (appendix) in the main paper, or you could include a simplification of the runtime complexity of the K-Means and GMM algorithms in terms you use for the Hungarian algorithm in the main text. This would make it easer to see the significant gains your approach (i.e. using k-means) gets over prior approaches.

  *Minor Comments not taken into account in the review*
  - You should be more clear what the Hungarian algorithm is. Specifically, the Kuhn 1955 should be cited when the algorithm is first mentioned.
  - "global shape and structure the group" -> "global shape and structure of the group"
  - "'anchor' agent (Mehrasa et al.)" -> year for citation

----------
Post Review

I appreciate the author addressing my concerns and making clarifications here in discussion and in the text. I'm going to keep my score as a 6 primarily because I am not confident in the literature this paper is appealing to to truly advocate for its acceptance.


review_title: Official Blind Review #1

Meta-review: The work addresses the problem of inferring group structure from unstructured data in multi-agent learning settings, proposing a novel approach that has key computational / run time advantages over a prior approach. A key limitation raised by reviewers is the limited quantitative evaluation and comparison to previous approaches, as well as a resulting set of general insights into advantages of the proposed approach compared to prior work (beyond computational benefits). While some of the key limitations were addressed in the rebuttal, the contribution in its current form remains too narrow. The paper is not ready for publication at ICLR at this stage.
Decision: Reject