Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
This paper proposes a NMF formulation ||A-FF^T||^2 where A and F are different types of information extracted from social datasets. In the honeybee example the authors highlight, A represents the spatial relationship between bees, and F encodes the age of the bees. The authors setup F to be decomposable into two types of embeddings, one which characterizes the group activity and the other which characterizes the individual activity. 

Pros:
The authors highlight a very interesting application. Honey bees perform tasks according to their age (information in F) and the tasks are performed in different nest substrates (information in A).  


Cons:
 It is not clear what is the form of the embeddings e_\phi.
It is not clear what is the baseline for the interpretable individual embeddings. Why should these individual embeddings capture reasonable offsets and why they capture the information in c_{i,t}? This is only tested indirectly, through the AMI score in the case of the synthetic dataset. Is there a way to setup this framework to check that true F and learned F in the synthetic dataset match more directly? In the current version of the work I cannot tell how well F captures the information encoded in c_{t,i}, this makes it hard to consider why/if ||A-FF^T||  could be meaningful. 
What is the factor strength in Fig 4?
The variance of AMI score is high even when the SNR = inf. Do the authors consider the model results robust given this high variance in the synthetic dataset? 

Comments:
Is there a mismatch in the type of data of Y in the synthetic and in the true datasets?  In the synthetic dataset is Y the spatial distance like in the honeybee dataset? 
The notation in Eq 3 is not clear, c: \mathbb{N}^{TxN} -> \mathbb{N}, but o_{\phi, \omega} \mathbb{N}^{Nx T} -> \mathbb{R}^M? Why are the inputs in \o_{} NxT? 
The authors use the terms functional position, functional role, and functional embedding. The way the paper is written it seems like these terms could be used interchangeably, but isn’t this is what the authors are trying to show in the paper? It would be easier to understand this work if the authors explained in more detail the different variables/mappings.
review_title: The learned individual embeddings need additional clarification

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summary

The authors introduce a novel method for non-negative matrix factorization for timeseries and apply it to longitudinal honey bee interaction data.  The model leverages consistency of individuals over time by forcing the factors (or rather, the residuals of the factors with respect to a global trajectory) to be linear combinations of a small set of temporal basis functions.  These temporal basis functions are functions of the bee’s age.  In other words, the factor embedding of each bee is a vector of linear combinations of 16 learned basis functions of time.  Since all bees use the same 16 temporal basis functions, given these basis functions the lifetime embedding of each bee is encapsulated by a small matrix of numbers, namely the coefficients for the temporal basis functions for each factor (and in practice only two factors were significant, so each bee’s life is embedded in 32-dimensional space).  There are a number of regularizations on the temporal basis functions and the embedding coefficients.

The authors show that the two significant factors have values that vary somewhat consistently as a function of a bee’s age.  The authors also offer some interpretability of the temporal basis functions as corresponding to different social roles in the bee colony.  The authors claim that this interpretable representation of individual bees is valuable for understanding social dynamics in the colony and may be useful for understanding how various perturbations affect the social order of the colony.

# Pros

* The paper is clear and well-written.
* The authors are open-sourcing both their model code and their dataset (which involved a substantial undertaking to collect).
* The authors’ method is novel as far as I know, and it is elegant (albeit has quite a lot of regularization terms).
* The authors are pretty rigorous in their analysis, ablations, and comparison to baselines.

# Cons

I have a few primary concerns, listed here roughly in order of significance:
* I am concerned that this paper might have too narrow a scope in too niche a field for it to be suitable for ICLR.  The paper is entirely dedicated to exploring honey bee social behavior, and it is not clear whether their method would be useful/interesting for other areas of research.  While the authors do suggest that it might be useful for analyzing social behavior in other animals (and make vague references to potential broader applications), they provide no concrete evidence for that.  Without concrete ideas for applications aside from animal social behavior and without strong implications for the usefulness of the representations (see following point), I am concerned that this paper will fail to capture the interest of the vast majority of ICLR attendees.  Perhaps it is better suited for an animal behavior or computational biology journal/conference.
* While the paper focuses on interpretable representations of honey bee behavior, it lacks evidence/support for the usefulness of these representations.  After reading the paper I am left wondering why being able to extract these representations is valuable.  First, in order to extract the representations one must have a longitudinal dataset of tracked honeybee positions.  This I think is only possible to obtain in controlled laboratory settings (though the authors redact the exact method they use), hence the method does not seem broadly applicable.  Second, even if the data were easy to collect, what do the representations indicate that cannot be obtained directly from the data?  If the temporal basis functions merely reflect what hive locations a bee frequents, can that not be extracted much more easily from the tracking data that the model relies on for training?  Third, the authors allude to potential usefulness of the representations for understanding the influence of pesticides or predatory pressure on the colony, but I think the paper really needs a concrete demonstration of (or at least very strong justification for) the usefulness of the representations.
* I definitely appreciate the sweeps over the regularization hyperparameters in table 2, but I am worried about whether these hyperparameters will transfer to other dataset, since the model has quite a number of hyperparameters.  So from a user perspective it would be good to know how robust the authors’ hyperparameters are across datasets.  One way to inform this would be to evaluate the hyperparameter sweeps for the synthetic datasets and see whether optimal hyperparameters (e.g. for the AMI score) are consistent across synthetic datasets.
* Please include the baseline models in Figure 3 (if it is possible to compute AMI for the baselines).  Without baselines, it’s difficult for a reader to determine how good the TNMF curve is.

Finally, I have a few very minor points:
* Please put a legend in the leftmost panel of Figure 4.
* In Figure 5-b, I’m guessing the “f_2” label in the legend supposed to be “f_3”.  If that is true, please change it.  If not, please justify why you are using f_2 in Figure 5 instead of f_3 when it is previously stated that only two factors (f_1 and f_3) are significant.
* The caption for Table 2 reads “Table 3” --- please change that to read “Table 2.”


# Conclusions

I do not recommend accepting this paper.  My main concerns are that the usefulness of the method is not demonstrated or well-justified, and the paper as a whole seems to have limited scope in a field of animal behavior that I suspect is pretty far from most ICLR attendees’ research interests.  However, the AC’s are in a better position than I am to judge the latter point, so I am not certain about my recommendation and could be convinced otherwise.

review_title: Interesting paper but limited scope and unclear usefulness --- weak reject

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a factorization method for exploratory analysis in temporal graphs / social networks. If I understand the model correctly, the new idea is to allow individual-specific differences in temporal trajectories (to account, e.g., for difference in age). Some of these ideas might be under the surface in time-shifted tensor decomposition models (linked below), but I do think that the authors' work is novel:

https://www.biorxiv.org/content/10.1101/2020.03.02.974014v2
https://doi.org/10.1016/j.neuroimage.2008.05.062

Though I think the core idea is interesting, I have mixed feelings about the method and some of the analyses.

(1) The method seems potentially more complex than is necessary. Honestly I found its presentation in the paper confusing and difficult to read. Perhaps section 2.1 is a bit too short, and more visualizations like Fig 2 would be helpful.

I think the biggest problem is that the authors' motivation for various choices is not well-described. For example, the adversarial loss is not well-explained, and it is referred to as "L_adversarial" in Fig 2, but "R_adv" in the Appendix (or are these two different things?).

Perhaps a bigger question is why the authors chose to decompose their factors $f_i = m(t_i) + o_i$ (see eq. 3). I think this is an interesting choice, but it seems like there could be many others like $f_i = \sum_j \alpha_{ij} m_j(t_i)$ or $f_i = m(t_i)$. If I understood the paper correctly, the second option is a simpler extension of Yu, Aggarwal, & Wang (2017), which seems like natural to explore before trying the more complicated model proposed in this work.

Smoothness over time seems to be an important aspect of the model (as judged from the schematic figures), but it is not emphasized by the authors how this is achieved in practice. Rather than having a neural network output the mean m, why not use a gaussian process, or a linear layer with a smooth spline basis / RBFs?

(2) If I understand the model correctly, it seems very similar to a classic tensor factorization, so it would seem to me that a better baseline to include would be $\hat{A}_t = F D_t F^T$ where each $D_t$ for $t = 1, ..., T$ is constrained to be diagonal. There are very fast and well-established algorithms for this (see Kolda & Bader, 2009). Also see the time-shifted model variants linked above.

(3) The evaluation metrics are a bit strange. The model performs worse than the baselines on reconstruction, though this does not seem to be the primary goal of the paper. Rather, the idea is to pull out interpretable factors. I had a hard time understanding whether the consistency / mortality / rhythmicity metrics were actually a good proxy for model interpretability. My suggestion would be to focus on qualitative / visual comparisons across the different models rather than these numeric comparisons.

(4) It would improve the paper to show the method working on a second dataset (e.g. a different social network or temporal graph of interactions).

In Summary: I like some of the ideas presented in this work, but I found it difficult to read. I think the biggest problem is that the core idea (individual-specific time courses) seems to get buried and lost with other modeling innovations like adversarial training and new terms like "individuality basis functions." Some of these bells and whistles may not even be needed, so simplifying the model and streamlining the text would go a long way for me.
review_title: Could a simpler model be sufficient?

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a matrix factorization model to jointly characterize the lifetime interactions of thousands of bees over generations. The problem is fascinating as both a technical and scientific question and the modeling framework appears novel. Although not directly addressed, the authors appear to be trying to solve a *tensor* factorization problem, not just the special case of a matrix (which is of course a 2-d tensor). It would have been interesting to see results comparing their method with a non-negative variant of, say, PARAFAC/CANDECOMP or generalizations thereof. It would have at least have been appropriate to explain why or why not existing tensor methods are not appropriate.

While I'm excited about this work and would love to see the potential applications in other domains, I'm troubled by the lack of clarity and sparsity of essential details in the main text. I am also concerned about how the authors have prioritized the regularization. I detail my concerns below:

Some erroneous (or perhaps just awkward notational issues) make reading difficult and omitted material makes it hard to evaluate the results.
* The authors do not explain in either the main text or in the supplement what $\omega$ is (first appears in equation (3)) and what about it is being learned (as indicated in equation (5)). 
* Equation (4) uses $k$ as both an index for the terms in the sum and as a subscript for the number of learned basis functions (although it is not clear why that number has its own index). What did the authors 
* How did the authors select the hyperparameters not listed in Table 2?

It seems that the "interpretability" metrics reported in Table 2 are precisely the same criteria as are being regularized for. It therefor should not be surprising that TNMF outperforms SymNMF on these metrics. What is somewhat concerning is how big a difference there is in the reconstruction loss and how much TNMF seems to fail in this regard. This must mean that the features that the authors wish to represent explain very little of the variance in the interaction networks. Could the authors 1) comment on this a bit and discuss why it would be the case and, 2) give us a sense of what the scale of the errors are relative to the total variance of $\mathbf{A}$, 3) comment on whether or not they could be missing high-variance features that they did not anticipate.

Lastly, there is a fundamental problem I have with the authors' regularization approach to interpretability. It seems as though they have enforced a handful of individual features for their model in order to achieve interpretability. However, these features are ostensibly things they already know about the individuals in the population. The point of an unsupervised method is to allow yourself the opportunity to discover things that you didn't already know from observing the features of individuals. Moreover, "interpretability" is somewhat subjective. Therefore, I'm concerned that the authors are using these regularization terms a little too haphazardly and have failed to make a strong argument for their use vis-a-vis their effect on reconstruction error.

I pains me to give this a mediocre score because I do find the work fascinating.  The paper requires more principled motivation for the choices the authors made as well as cleaning up the notation.
review_title: Very exciting science and an ambitious solution but lacks important motivation, details, and clarity.

Meta-review: The authors present a matrix factorization for the social behavior of honey bees in a hive. All the reviewers appreciated the interesting application.  However, substantial concerns were raised about the model motivation and the interpretation of the learned factors. To quote one reviewer, "Some of these bells and whistles may not even be needed, so simplifying the model and streamlining the text would go a long way for me." Another said, "The paper requires more principled motivation for the choices the authors made as well as cleaning up the notation."  The authors did address some of these concerns in discussion, but there are too many lingering concerns to recommend acceptance.  Given the unique application of this paper, the authors might also consider a journal that specializes in computational biology instead. 

Decision: Reject