Row-clustering of a Point Process-valued Matrix

Abstract: Structured point process data harvested from various platforms poses new challenges to the machine learning community. To cluster repeatedly observed marked point processes, we propose a novel mixture model of multi-level marked point processes for identifying potential heterogeneity in the observed data. Specifically, we study a matrix whose entries are marked log-Gaussian Cox processes and cluster rows of such a matrix.  An efficient semi-parametric Expectation-Solution (ES) algorithm combined with functional principal component analysis (FPCA) of point processes is proposed for model estimation. The effectiveness of the proposed framework is demonstrated through simulation studies and real data analyses.


Reviews and author responses:

This paper proposes a mixture model of Multi-level Marked Point Process (MM-MPP) for repeatedly observed multi-category event sequences and develops a efficient semi-parametric Exponential-Solution (ES) algorithm for event sequence clustering. Experiments through simulation studies and a real-world data analysis show the effectiveness of the work.  Pros:
This paper considers the event sequence clustering problem, which exists widely in practical scenarios. 
Sufficient details of the learning algorithm are given.

Cons:
I’m confused what exactly the ‘multi-level’ refers to. The authors say their model is ‘multi-level’ as the event sequences they target at are ‘repeatedly’ observed. It seems the ’level’ refers to ‘day’ considering the Twitter dataset and section 4. However, according to line 42-44 and section 3.1, the ‘level’ is only related to event types. Then if temporal dependencies are ignored in this work, why is the ‘repeatedly observed’ or ‘multi-level’ important? 
Moreover, the framework assumes there is no temporal dependencies between events. I doubt the rationality of this assumption, especially considering the Twitter data, in which former events would obviously influence the happening of latter events. 
The baselines [15] and [27] were published in 2013 and 2017, respectively. There should be more advanced frameworks proposed for event sequences clustering since 2017. I suggest the authors make broader literature review and add more competitive baselines. Also more real data sets should be tested for evaluating the effectiveness of the proposed framework.
The computational complexity is discussed, but I would suggest the authors add the convergence rate analysis.

--------------------
Update:

Thank the authors for their feedback. However, my concern is only partially served. I keep my score. There is no negative societal impact of this work.

 Potential baselines:

Wu, Weichang, Junchi Yan, Xiaokang Yang, and Hongyuan Zha. "Discovering Temporal Patterns for Event Sequence Clustering via Policy Mixture Model." IEEE Transactions on Knowledge and Data Engineering (2020).

Kong, Quyu, Marian-Andrei Rizoiu, and Lexing Xie. "Describing and Predicting Online Items with Reshare Cascades via Dual Mixture Self-exciting Processes." In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 645-654. 2020.

Iwayama, Koji, Yoshito Hirata, and Kazuyuki Aihara. "Definition of distance for nonlinear time series analysis of marked point process data." Physics Letters A 381, no. 4 (2017): 257-262.

There should be more. I suggest the authors make broader literature review and add more competitive baselines. 
Overall, the authors' feedback didn't convince me. I will keep my score.

 Thank you so much for your suggestion! We will definitely improve our abstract if given the opportunity.

 I updated my review last week: "The authors have addressed my concerns very well. It is a good paper, and I will keep my rating."

The final suggestion I want to reemphasize is to make sure that your abstract has the same high quality as your main text. It gives a very important first impression.

This paper presented a method of multi-level marked point processes for clustering using a mixture model. The idea is to model each component as a Gaussian Process w.r.t time t, where the multi-level model is based on the single-level model as well as a semi-perimetric method in [26]. ES method is chosen with modification incorporated from the estimating equations. In E-step, FPCA and MC methods are used to reduce the computational costs. Synthetic experiments validate the correctness of the model and the real-world Tweet dataset is designed to explore the clustering results in an explanatory way. The main contributions in this paper are a new statistical mixture model for repeatedly observed multi-category event sequences and a corresponding semiparametric ES algorithm instead of the classical EM algorithm.

  This paper is generally well-written with strong motivation, a well-defined approach. The main concern for me is regarding the experimental design.

The authors claim that "design an algorithm that can take advantage of array programming and GPU acceleration to further speed up computation", however, not enough discussion and numerical results showed that GPU or array programming can facilitate the computation.
Another claim is "existing EM algorithms for event sequence clustering have a common issue that they typically require iterative numerical optimizations within each M-step, which would drastically overburden the computation." I am wondering if EM can be substantially slower than ES due to the reason described above. Having an iterative scheme in the maximization step might not lead to slower convergence comparing with ES. It might need further empirical verification or theoretical convergence rate analysis.
What is G defined as on page 7? When R is large, how does the sampling complexity is decreased to O(nCQR^2) from O(nCQR^3)? I have no idea what the exact meaning of "R is small to moderate" is. Could the authors show the complexity of R when it is defined as small to moderate?
In real-world experiments, the clustering stability called clustering consistency is evaluated. I am not 100% sure if it is the best way to verify the model's efficacy. The statistical model is built to solve a very specific application problem, so the clustering validation is of importance when one wants to use the model on other datasets. If no proper clustering validation measure is designed, one dataset with K-trial cross-validation seems not enough to convince me at least. I am wondering if the model can work when it is used on some clinical longitudinal datasets such as drug addiction. (https://catalog.data.gov/dataset?tags=addiction)
Figure 3 shows the mean of each cluster, but I am also curious about the in-cluster variance, i.e., the homogeneity in each cluster so that the cluster results are seriously discussed.
Figure 3 in the paper is not clear. Please use a finer resolution and consistent font sizes.

------
Update: While I appreciate the authors' comments, and some of their points are correct, my main concern - which is the experimental design of the paper has not been resolved by their comments. The simple fact that a variety of similar datasets are not used in the paper is highly problematic. Some of the clarifications on the faster convergence speed over EM and the higher GPU efficiency of the method regarding the experimental design sadly do not either answer the questions posed in the review.

Overall the author's comments have not convinced me to change my assessment of the paper. If a future version of the paper would provide a more comprehensive comparison and firmly justify the benefits of their proposed approach, I would be more than happy to accept that version. Yes. The authors addressed the limitations and potential negative societal impact except for the lack of experimental design.

 Dear reviewers and area chairs:

We notice that there has been no response to our rebuttals to the first round of reviews, we are wondering if it is possible for you to provide some feedback to help further improve our work. Your time is much appreciated!

Best regards,
Authors of this paper.


The paper proposes a clustering method for repeatedly observed marked point process data. The method is based on a mixture model of multi-level log-Gaussian Cox processes. The paper develops an expectation-solution algorithm for implementation, where the solution-step has a closed-form solution. The study is motivated by a Twitter dataset, where the postings (tweet, retweet, or reply) are treated as point processes.  The paper is on an interesting topic: model-based clustering of complex point patterns. Existing methods build on mixture models of Hawkes processes and rely on EM algorithms for estimation. The proposed method is complementary to previous work and arguably more flexible and computationally efficient. The paper is generally well written.

Specific comments:
1) The abstract should be rewritten to improve clarity. “Expectation” is misspelled as “Exponential.” Description of the point process data is not clear, and no motivating example is mentioned.
2) From the abstract, functional PCA seems a key component in the proposed algorithm, but is not mentioned in the introduction.
3) Is there a convergence guarantee for the proposed algorithm?
4) When should one choose to model the event sequences by the Hawkes process or the log-Gaussian Cox process? Some discussion would be helpful.

Update: The authors have addressed my concerns very well. It is a good paper, and I will keep my rating. The paper notes the potential risk of misinterpreting the clustering results.

 * **Justification of multi-level model:**
We apologize for the confusion. The ``multi-level" refers to the user-level and the day-level, the event type is considered as the mark of an observed event time. Specifically, for each user and each day, we observed a marked point process. For different users and different days, we have a collection of marked point processes (all of which are defined on a one-day time span), which are dependent on each other. The temporal dependence within each marked point process is not ignored but modeled through a log-Gaussian Cox process. We shall clarify these points in the revised manuscript. The key assumption behind the proposed clustering method is that the user posting behavior is partly determined by user-specific characteristics ($X_i(t)$) that do not change from day to day. The multi-level model further incorporates contributions from day-related factors $Y_j(t)$ (such as weekday effect, weekend effect or holiday effect) to the posting behavior. However, the purpose of the clustering is to cluster users based on user-related characteristics ($X_i(t)$), which is not lost by aggregation even though there may be discontinuity to the original sequence.  In this sense, the proposed multi-level model allows us to borrow and learn shared patterns ($X_i(t)$'s) across days while accounting for day-to-day variations.  Finally, we would like to comment that the proposed method also applies to other repeated measurements of point processes such as signals on waves and specific segments on genes. Many real data, especially those involving human activities, are naturally repeatedly observed by days and have none or inactive events during certain time periods in a day. Examples include the twitter data we considered in our analysis, credit card and stock trading transactions, to name a few. 

* **Literature Review** 
We included [27] as a baseline because it is the most recent method for clustering temporal event sequences that we are aware of.
We included [15] because it is capable of characterizing interactions among events based on a model-free method. To the best of our knowledge, there has been little development on clustering event time process data. We would be happy to cite and compare with other existing methods if you could point to us some directions. 
* **Additional data examples:**  we plan to add two additional real data examples to illustrate our methods.
  * Chase Credit Card Dataset: We applied the proposed method to a credit card transaction data set, which was collected by Worldline and the Machine Learning Group at Université Libre de Bruxelles and can be accessed via the Kaggle database. The data set contains 641,914 transaction records of 5,000 European credit card customers during the period covering January 1 to December 31, 2016. We treated the one-day transaction records of each account as an event sequence and applied the multi-level LGCP model to the dataset. For comparisons, we also applied the single-level model to the data by concatenating the event sequences of each account into a single long sequence. The single-level model clustered the accounts into 5 groups. The results seemed to cluster card users mainly by their overall frequency of transactions but were incapable of capturing their daily patterns of transactions. For the multi-level model, we obtained 6 clusters under the BIC criteria. Daily transaction patterns seem to also contribute to the clusterings. We will add the detailed analysis results to the revised manuscript.
  *New York City Taxi Dataset: The NYC taxi dataset contains around 173 million trip records of individual Taxi from Jan. 1 to Dec 31, 2013.  Each record includes the time and locations (in the form of latitude/longitude coordinates) of pick-up and drop-off passengers. We formed the temporal event sequences using the pick-up time. Also, we mapped the pick-up location into 51 Council districts in NYC as the markers of events. 
We applied the multi-level marked LGCP to the dataset, and here we summarize some preliminary findings. Overall, our method showed a higher clustering consistency than the discrete Frechet distance-based method (DF, Pei, 2003) and Dirichlet mixture of Hawkes processes (DMHP, Xu, 2017). Our method produced 10 clusters, whereas DMHP produced 14 clusters and DF produced 23 clusters. We observed that DMHP seems to cluster taxis by their overall pick-up frequency and time intervals between events. In contrast, our method is capable of clustering taxis taking into account their daily schedule patterns and boroughs. 

* **Convergence analysis**

Elashoff and Ryan (2004) established a general theory for the local convergence of the E-S algorithm. Under two regularity conditions on the first derivative of estimating equations, there exists a neighborhood of solution such that the algorithm has a guaranteed convergence if initializing the unknown parameter with a value within this neighborhood. The theory is proved by treating ES as a Block Newton-Gauss-Seidel algorithm and then directly applying the convergence results in Ortega (1972, p. 147). Our ES algorithm is a special case of Elashoff and Ryan (2004), so their general theories apply to our case.  We will add some of the above discussions in the revised manuscript to clarify the motivations and rationals of the E-S algorithm.  
We also numerically checked the convergence of our algorithm. We considered the same setting as the synthetic data example in section 5.1 and Appendix S.3 with $C=3$, $R=5$, $n=500$, $m=20$. We run the E-S algorithm with different initializations of cluster assignments. On average, the value of the log-likelihood becomes stable after approximately 20 iterations. But for the convergence of parameters, will be achieved after 40 iterations. However, if we kick off with the initial parameters which were obtained by quick distance-based methods like [15], then the parameters will converge around 10 iterations.  We will report the convergence analysis results in the revised manuscript. 

* **References**

    * Michael Elashoff and Louise Ryan. An EM algorithm for estimating equations. Journal of  Computational and Graphical Statistics, 13(1):48–65, 2004.
    * Ortega, J. M.  Numerical Analysis: a Second Course. New York: Academic Press, 1972.
    * Tao Pei, Xi Gong, Shih-Lung Shaw, Ting Ma, and Chenghu Zhou. Clustering of temporal event processes. International Journal of Geographical Information Science, 2013.
    * Ganggang Xu, Ming Wang, Jiangze Bian, Hui Huang, Timothy Burch, Sandro Andrade, Jingfei Zhang, and Yongtao Guan. Semi-parametric learning of structured temporal point processes. Journal of machine learning research, 2020.
    * Hongteng Xu and Hongyuan Zha. A dirichlet mixture model of hawkes processes for event sequence clustering. NeurIPS, 2017.

 We thank the reviewer for many invaluable comments
* 1 and 2, we will report the computation times of non-GPU-based E-M, non-GPU-based E-S, and GPU-based E-S algorithms in the revised manuscript.  Here, we report part of run-time (in seconds) of 20 iterations in the case that $R=5$, $C=3$, $m=20$ and size of sampling $=10,000$.

|  | GPU-ES (RTX 8000 48G GPU) | CPU-ES (i7-7700HQ CPU) | CPU-EM (i7-7700HQ CPU) |
|:----:| :----: | :----: | :----: |
| n=500 | 30.09 | 275.87 | 568.36 |
| n=1000 | 51.42 | 505.07 | 1105.46 |

* 3. Convergence analysis: Elashoff and Ryan (2004) established a general theory for the local convergence of the E-S algorithm. Under two regularity conditions on the first derivative of estimating equations, there exists a neighborhood of solution such that the algorithm has a guaranteed convergence if initializing the unknown parameter with a value within this neighborhood. The theory is proved by treating ES as a Block Newton-Gauss-Seidel algorithm and then directly applying the convergence results in Ortega (1972, p. 147). Our ES algorithm is a special case of Elashoff and Ryan (2004), so their general theories apply to our case. We will add some of the above discussions in the revised manuscript to clarify the motivations and rationals of the E-S algorithm. We also numerically checked the convergence of our algorithm. We considered the same setting as the synthetic data example in section 5.1 and Appendix S.3. We run the E-S algorithm with different initializations of cluster assignments. On average, the value of the log-likelihood becomes stable after approximately 20 iterations and the convergence of parameters will be achieved after 40 iterations. We will report the convergence analysis results in the revised manuscript.
* 4.  We apologize for the typo on page 7. $G$ should be $R$, the number of event types. When $R$ is large, the low-rank or sparse properties of the matrix may be utilized to reduce the sampling complexity, but the amount of reduction would depend on the rank or the sparsity level of the matrix. Our original statement about the reduced complexity $O(nCQR^2)$ is incorrect. We apologize for the incorrect statement and will correct it in the revised manuscript. 

* 5. We thank the reviewer for this suggestion.  We followed Xu and Zha (2017) and used clustering stability as an evaluation metric in our real data example. We are unaware of other better ways to verify the model's efficacy when ground-truth is unknown. We will really appreciate if the reviewer may guide us to some specific references for the evaluations of clustering performance.  We would also like to clarify that the proposed method is not built to solve a specific application problem. The Twitter data set is just a motivating example to better deliver the key concepts involved in the proposed multi-level marked point process model. In principle, the multi-level model is essentially an extension of the classical (two-way) analysis of variance (ANOVA) model to the case where each observed point in ANOVA is replaced by a marked point process. One can design any reasonable experiments using various factors to obtain replicated marked-point processes (in a fixed time domain, such as a day).  Taking the stock trading data studied in Xu et al. (2020) for example, we can consider user stock trading activities in the same day but at different trading locations. In this case, the first level will be the user-related characteristics and the second level will be geographical locations. Such a data set may reflect how user-related characteristics and spatial economic factors in a country contribute to the user stock trading activities. A three-level model (user, day, and spatial location) can also be considered in the same fashion as in Xu et al. (2020).

* 6. We thank the reviewer for pointing out a potential real application. In principle, the proposed method is applicable to the clustering of other repeated measurements of marked point processes. We will check whether the proposed model is suitable for the analysis of the suggested clinical longitudinal drug addiction datasets.


* 7. We will revise Figure 3 to add the information of in-cluster variance and make the figure clearer. 
 
* **References**

    * Michael Elashoff and Louise Ryan. An EM algorithm for estimating equations. Journal of  Computational and Graphical Statistics, 13(1):48–65, 2004.
    * Ortega, J. M.  Numerical Analysis: a Second Course. New York: Academic Press, 1972.
    * Ganggang Xu, Ming Wang, Jiangze Bian, Hui Huang, Timothy Burch, Sandro Andrade, Jingfei Zhang, and Yongtao Guan. Semi-parametric learning of structured temporal point processes. Journal of machine learning research, 2020.
    * Hongteng Xu and Hongyuan Zha. A dirichlet mixture model of hawkes processes for event sequence clustering. NeurIPS, 2017.

 We thank the reviewer for carefully reviewing our paper and offering many constructive comments. 
* 1. Although technically one can merge daily event sequences from different days into one sequence and jointly model them using models like Hawkes process, we feel these methods may not be efficient to characterize temporal dependence structures for repeatedly observed point processes. It is unclear how one should define the triggering kernel of Hawkes's process across different accounts and days. The triggering kernel is typically a function of the distance between a given time point and a past event time. It is not straightforward to define the distance between two time points when there are discontinuities in the observation time window. Moreover, the self-exciting property may not completely characterize the clustering patterns because the event occurrences might be triggered by day-related external factors impacting all accounts on any given day. In fact, Xu et al. (2020) demonstrate through simulation studies that such a strategy may fail to capture crucial aspects of the observed point process data if the data are generated from the multi-level LGCP model. 
* 2. Our estimating equations are motivated and constructed by methods of moments. The model likelihood was used in the sampling procedure in (9) at the E-step. 
The validity of estimating equations is proved by the Campbell's theorem and the moment generating function of the normal distributions.  
* 3. The manuscript involves many notations because we have many different modeling components in the mixture model of multi-level marked point processes. To ease the readability of our paper, we will add a notation reference table in the Appendix. $\Gamma_{x,c}^r$ in (5) and $\Gamma_{x,c}^{r,r'}$ in (6) were defined in Section 3.1 to denote the cross covariances $Cov[X_{i}^r(s),X_{i}^{r'}(t) |\omega_{c,i}=1]$ that measure the account-level dependence within the same mark and across marks, respectively.  $K_h$ was introduced to nonparametrically estimate the moment quantities when constructing the estimating equations. The Campell's theorem was used to calculate the moments of functions of point processes in order to verify the unbiasedness of the estimating equations in (7).  
 
* 4. The baseline that the reviewer suggested was already included for comparisons in the original submission. We cited the arXiv version of Xu and Zhang (2017), which probably caused confusion to the reviewer. We have now corrected the citation to the NeurIPS version of their manuscript.
   
Reference:

   * Xu, G., Wang, M., Bian, J., Huang, H., Burch, T., Andrade, T., Zhang, J. and Guan, Y. 2020 Semi-parametric learning of structured temporal point processes. Journal of machine learning research 

   * Xu, H. and Zha, H., 2017, December. A Dirichlet mixture model of Hawkes processes for event sequence clustering. In Proceedings of the 31st International Conference on Neural Information Processing Systems (pp. 1354-1363).


 
We thank the reviewer for many useful suggestions. 

*  **Abstract:** We will revise the abstract to clarify our motivating example and point process data.

* **Introduction:** The functional PCA is used for sampling from the conditional distribution in the Expectation step of the E-S algorithm, and is indeed important for the proposed method. We will highlight the importance of functional PCA in the introduction in the revised manuscript. 

 
* **Convergence analysis:**
Elashoff and Ryan (2004) established a general theory for the local convergence of the E-S algorithm. Under two regularity conditions on the first derivative of estimating equations, there exists a neighborhood of solution such that the algorithm has a guaranteed convergence if initializing the unknown parameter with a value within this neighborhood. The theory is proved by treating ES as a Block Newton-Gauss-Seidel algorithm and then directly applying the convergence results in Ortega (1972, p. 147). Our ES algorithm is a special case of Elashoff and Ryan (2004), so their general theories apply to our case. We will add some of the above discussions in the revised manuscript to clarify the motivations and rationals of the E-S algorithm.
We also numerically checked the convergence of our algorithm. We considered the same setting as the synthetic data example in section 5.1 and Appendix S.3.  We run the E-S algorithm with different initializations of cluster assignments. On average, the value of the log-likelihood becomes stable after approximately 20 iterations, and the convergence of parameters will be achieved after around 40 iterations. However, if we kick off with the initial parameters which were obtained by quick distance-based methods like Pei et al (2013), then the parameters will converge around 10 iterations. We will report the convergence analysis results in the revised manuscript.

* **LGCP versus Hawkes:**
For human activity data such as social media posting and stock transaction activities, it is reasonable to believe that, in addition to user-specific characteristics, many external factors also contribute to the activity patterns. For example, spatial locations of the user or overall macroeconomic environment.  Existing models such as multi-variate Hawkes models rely heavily on user's interval factors by assuming that human activities are self-exciting, which may be overly simplistic. Another appealing property of the multi-level LGCP model is that it can be easily extended to include more than two factors into the model and hence has the potential to handle more complicated point pattern data. We refer readers to Xu et al (2020) for a more comprehensive discussion on the advantages of the multi-level LGCP model. We will add more discussion in the revised manuscript.

* **References**

    * Michael Elashoff and Louise Ryan. An EM algorithm for estimating equations. Journal of  Computational and Graphical Statistics, 13(1):48–65, 2004.
    * Ortega, J. M.  Numerical Analysis: a Second Course. New York: Academic Press, 1972.
    * Tao Pei, Xi Gong, Shih-Lung Shaw, Ting Ma, and Chenghu Zhou. Clustering of temporal event processes. International Journal of Geographical Information Science, 2013.
    * Ganggang Xu, Ming Wang, Jiangze Bian, Hui Huang, Timothy Burch, Sandro Andrade, Jingfei Zhang, and Yongtao Guan. Semi-parametric learning of structured temporal point processes. Journal of machine learning research, 2020.



The paper proposes a method of clustering event sequences, where each sequence represents a series of actions done by an entity, e.g., a tweeter account. The paper models the sequence of each account at each day as a sample of a inhomogeneous Poisson process. The log intensity is modeled by a summation of the account-level, day-level and (account, day)-level latent functions of the time. To cluster the data in the account level, the account-level latent function is assigned a mixture GP prior while the other latent functions GP priors. The paper proposes to use Expectation-Solution (ES) for the model estimation (including estimating the cluster memberships). The paper proposes estimating equations, and performs ES steps which are analytical, and do not require numerical optimizations. The experiments show the effectiveness of the method.    I like the topic of clustering event sequences; I am also interested in the ES framework introduced by the paper, which seems solid and has some advantages. However, there are several significant issues in the presentation and experiments that need to be addressed.  

(1) The paper does not well explain why they use inhomogeneous PP. The paper emphasizes that Hawkes processes are "are not suitable to model the data in our case that have multi-level variations". I do not believe so. If for each account you  merge the sequences in different days into one sequence , and model the whole sequence as one Hawks process --- I do not see any problem. You can model the base rate with the same structure as in (1), but also account for the excitation between different events. In fact, it is even more expressive. 

(2) While the ES framework seems interesting, the paper never explains  how it comes out the estimating equation (7). How does it connect to the model likelihood (3). How do you justify it? What is the error of the ES estimate? etc.  From the description of the paper, (7) is proposed more like for easing the S step (avoiding the numerical optimization).  

(3) The paper has messy, undefined/unexplained notations. For example, what is \Gamma_{x,c}^r in (5), \Gamma_{x,c}^{r,r'} in (6)? Do the two Gammas refer to the same object? Why do you introduce a kernel K_h? Why do you use Campbell's theorem?  

(4). There is only one weak baseline.  Why don't you compare with [27], the most recent method of clustering event sequences with Hawkes processes?  N/A

Meta-review: As pointed out by the reviewers, the paper is missing comparisons against more recent baselines. There are plenty of these baselines that the authors simply did not even mention. The paper also focuses on a single real data example. Including more real-data examples is necessary to assess the generality of the method.  I believe that in its current version the paper is not ready yet for publication.

Decision: accept