Latent Spatial Dirichlet Allocation

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-06 05:15:01

Abstract: We propose a novel topic modeling approach, latent spatial Dirichlet allocation (LSDA), which generalizes the latent Dirichlet allocation to spatial data. LSDA integrates spatial Gaussian processes within the LDA framework, thereby effectively capturing complex spatial dependencies inherent in spatial data. We develop an efficient Markov chain Monte Carlo algorithm, and applications to both real and synthetic datasets successfully demonstrate the utility of LSDA.

Main paper: 
 introduction: 
 Latent Dirichlet allocation (LDA, Blei et al., 2003) is a Bayesian probabilistic model commonly used in natural language processing to discover abstract topics that occur in a corpora of documents. The LDA model assumes that the words of each document arises from a mixture of topics, where each topic is a multinomial distribution over a fixed word vocabulary, suggesting a shared theme in the data. Documents consist of multiple topics with varying proportions, implying that LDA is an instance of mixed-membership models. LDA is a powerful tool in modeling hidden thematic structures in a collection of documents, facilitating the organization and understanding of large volumes of text data.The mixed-membership nature of the LDA model makes it popular in analyzing discrete data in other settings, such as population genetics (Pritchard et al., 2000) , network data (Airoldi et al., 2008) and computer vision (Li & Perona, 2005; Russell et al., 2006) .However, the direct application of LDA or conventional LDA-based topic models to spatial data analysis faces several challenges. First, LDA is a 'bag of words' model, which means that the words in each document are assumed to be exchangeable within them. This assumption causes LDA to ignore spatial structures within documents. Second, LDA assumes that topic distributions are independent across documents. In the context of spatial analysis, where documents typically represent segments of the entire spatial region, this assumption, along with the bag-of-words assumption, completely disregards the spatial dependence structure inherent in spatial data. In practice, we expect that topic distributions in neighboring spatial regions are correlated. To address such limitations, several extensions of the LDA model have been developed to account for spatial information in diverse fields of spatial data analysis. For instance, in the context of location-based social network data, spatial topic models have been developed to identify regional communities (Van Canh & Gertz, 2013) and assign semantic labels to locations of interest (He et al., 2017) . In computer vision, topic models such as the spatial LDA (Wang & Grimson, 2007) and the spatially coherent latent topic model (Cao & Fei-Fei, 2007) have been proposed to incorporate spatial information, allowing the discovery of objects and scenes categories from images. Chen et al. (2014) extended LDA to account for spatial structure in multiplex imaging data from human tissues. Despite their advancements, these methods are specifically designed for particular applications, which limits their generalizability across broad spatial data analytic tasks.We propose the latent spatial Dirichlet allocation (LSDA), a general topic modeling framework that can be applied or easily extended to a broad range of spatial data classes and dependencies. Figure 1 W Z θ 1 N d D β K γ W Z N d W Z N d α 1 s 1 θ 2 α 2 s 2 θ D α D s D κ ⋯ ⋯ // (b 
lsda topic model: 
 To generalize topic models to spatial data analysis, it is essential to appropriately define "words" and "documents" in the context of spatial data. Any low-level spatial objects characterized by discrete values and observable locations can be considered as words. For instance, in the multiplex imaging data shown in Figure 1 selected topic z di , w di ∼ Multinomial(β z di ).Here, γ and α are hyperparameters for the Dirichlet distributions. A fundamental assumption of the LDA model is that the documents are independent of each other. However, this assumption does not hold for spatial data, as topic distributions of nearby spatial regions are expected to be correlated.Latent Spatial Dirichlet Allocation. We generalize the LDA model to incorporate the spatial information inherent in the spatial data. Specifically, to account for spatial dependence between different segments of the entire spatial region, we introduce the spatial Dirichlet allocation process (SDAP) for the segment-specific topic distributionθ d := θ(s d ) = {θ 1 (s d ), . . . , θ K (s d )} ⊤ ∈ R K ,where s d , d = 1, . . . , D, denote the observed locations of spatial segments. We define the SDAP on α(s d2 ) will result in similar segment-specific topic distributions θ(s d1 ) and θ(s d2 ). Hence, the distribution over topics will be coherent across neighboring spatial regions.θ(•) as θ(•)|α(•) ∼ Dirichlet{α(•)}, log α k ( • ) iid ∼ GP{0, κ(•, •)}, k = 1, . . . , K, where α(•) = {α 1 (•), . . . , α K (•)} ⊤ andWe denote the spatial Dirichlet allocation process as θ(•) ∼ SDAP(κ). Then, the complete data generating process of LSDA is given as follows. The first step is same with the standard LDA. For k = 1, . . . , K, draw per-topic word distributions from a Dirichlet distribution, β k ∼ Dirichlet(γ).However, the procedure diverges in the subsequent step. We applied our proposed LSDA to an image that contained the highest cell count while maintaining at least 30 cells per type, shown in Figure 2 (a). Since the number of topics was unknown in this analysis, we considered multiple numbers of topics K = 2, 3, 4, 5, and utilized the deviance information criterion (DIC, (Li et al., 2020) ) to select the optimal number of topics. According to the DIC, K = 4 was selected. Figure 2 aggregation is known to enhance tumor immunity and improve patient survival, as B cells act as antigen-presenting cells that help activate cytotoxic T cells and contribute to the antitumor response (Montfort et al., 2017; Zhang et al., 2023) . 
synthetic data: 
 We also conducted a comparative evaluation of the empirical performance of LSDA using synthetic data generated based on the real data discussed in Section 3.1. Synthetic datasets were generated from the LSDA model using fixed cell locations from the ovarian cancer multiplex image, with W = 6 and K = 4 as in Section 3.1. We set α k = αk (fitted GPs from the ovarain data), and considered three different settings of β k corresponding to varying signal-to-noise ratios (SNRs):β k = (1 -ρ) βk + ρ( 1 W 1 W ),where βk is the estimate from the ovarian data and 1 W is a vector of ones of length W . As ρ increases, topics get less distinct, indicating the SNR decreases. We varied ρ across three settings: 0.5, 0.25, and 0, labeled as SN R low , SN R med , and SN R high , respectively.We implemented our LSDA as in Section 3.1, and compared its performance against two benchmarks: standard LDA (Blei et al., 2003) , which does not consider spatial information, and spatial-LDA (Chen et al., 2020) , which extends LDA to incorporate spatial information from multiplex images. The effectiveness of these methods in identifying spatial patterns of topics was compared by calculating the adjusted rand index (ARI, Hubert & Arabie, 1985) between the true and estimated segmentspecific dominant topics for each method. In Figure 2 (c), we present boxplots of the ARI values (from 50 replicates) for LSDA, LDA, and Spatial-LDA across different SNR levels. LSDA consistently outperformed all the alternatives in identifying dominant topics. Additionally, as the SNR decreased, the performance gap between models that account for spatial dependencies (LSDA and Spatial-LDA)and the standard LDA model widened, which emphasizes the importance of accounting for spatial dependence structures when applying topic models to spatial analysis. 
conclusion: 
 We propose LSDA, a novel topic modeling framework for spatial data, which integrates GPs within the LDA model to effectively account for complex spatial dependencies inherent in such data. Through simulations, we demonstrate that incorporating GPs allows for greater flexibility in modeling complex spatial dependency patterns, outperforming existing spatial topic models that rely on a specific assumption of spatial smoothness (e.g., adjacency-based similarity). Our analysis of cancer multiplex imaging data shows the effectiveness of our LSDA in discovering tumor microenvironments with biological significance. Given its flexibility, the proposed LSDA framework can also be applied to a wide range of spatial data, including computer vision and geographical data. 
 
 Appendix: In this section, we provide the detailed description of our MCMC algorithm for the LSDA model. To alleviate the computational burden of sampling from the GP posterior, we consider an approximation of the GP using the eigendecomposition of the covariance kernel κ, as described below.Consider the eigendecomposition of the covariance kernel κ(s, s The detailed steps of our sampler are given as follows:′ ) = ∞ l=1 λ l ϕ l (s)ϕ l (s ′ ), where {λ l } ∞ l=1 is the set of eigenvalues with λ 1 ≥ • • • ≥ λ l ≥ λ l+1 ≥ • • • ,Update Z. Sample z di from its conditional posterior P r(z di = k|Z -di , B, X), marginalizing out β and θ, P r(EQUATIONwhere n kh is the number of times word w is assigned to topic k, N dk is the number of words in document d assigned to topic k, and the superscript (-di) indicates that the corresponding datum has been disregarded when calculating n kh and N dk .Update β. Sample β k independently from its full conditionalEQUATIONUpdate θ. Sample θ d independently from its full conditionalEQUATIONwhereα k (s d ) = exp L l=1 b kl ϕ l (s d ) .Update B. It is infeasible to directly sample from the full conditional distribution of B π(B|θ, X)EQUATIONwhere α k (s d ) = exp L l=1 b kl ϕ l (s d ) and B(c 1 , . . . , c K ) is the beta function. Hence, we adopt SGHMC (Chen et al., 2014) to draw samples from (4). Hamiltonian Monte Carlo (HMC) is an efficient sampling approach which shows a higher acceptance rate compared to the standard Metropolis-Hastings sampling. SGHMC extends HMC by using stochastic gradients to improve efficiency, Algorithm 1 SGHMC for updating B at the t-th MCMC iteration 1: Input: the number of leapfrog steps M , the learning rate η and the momentum tuning parameter (1 -α). 2: optionally, resample momentum r from the matrix normal distribution r ∼ M N K×L (0, I, ηI); 3: set (B 0 , r 0 ) = (B (t-1) , r (t-1 ); 4: for h = 1, . . . , M do5: update B h = B h-1 + r h-1 ; 6: sample ϵ h ∼ M N K×L (0, I, 2αηI); 7: sample D ⊂ {d} D d=1 ; 8: update r h = (1 -α)r h-1 -η∇ Ũ (B h ; D) + ϵ h ; 9: end for 10: set (B (t) , r (t) ) = (B M , r M );allowing it to avoid evaluating the entire dataset. Additionally, SGHMC eliminates the need for the Metropolis-Hastings step after each proposal by introducing an additional friction term in the momentum update. Specifically, at t-th MCMC iteration, we update B through Algorithm 1.Given the sample of other parameters at the t-th iteration θ (t) d , we compute the stochastic gradient of U (B) = -log π(B|θ, X) by subsampling the indices of documents, that is, we calculateEQUATIONwhere ψ(•) is the digamma function and the index set D is a random subset of the indices of documents (spatial segments) {d} D d=1 .In this section, we provide details on how we implemented our LSDA approach for the experiments in Section 3.The first step in implementing our LSDA on the multiplex imaging data was segmenting the given multiplex image to define documents for our LSDA framework. We achieved the segmentation by applying Voronoi tessellation over a systemically arranged grid of points. The grid points serve as locations of the segments (i.e., s d ), and the arrangement of this grid was designed to maintain intercellular distances within each spatial segment under 40 µm, a threshold beyond which cell-to-cell interactions are typically considered insignificant.To implement the proposed LSDA method, we should specify the covariance kernel κ for the GP over log α k (•). In this work, we used the modified squared exponential covariance kernel, which allows for straightforward computation of eigenfunctions and eigenvalues through the use of Hermite In Section 3, we run our proposed MCMC algorithm for 20,000 iterations, of which the first 10,000 iterations were discarded as burn-in. Based on the MCMC samples, we calculated the posterior mean βk and θd of β k and θ d , and identified the dominant topic for each spatial segment by determining which topic had the highest probability, that is, kd = argmax k θkd . 



Reviews: review: title: title: I am leaning towards accepting the work for a workshop as it presents an interesting novel approach of using LDA for spatial data analysis. However, there are several open questions that should be addressed in the final version of the paper to improve clarity and completeness.
main_review: main_review: ## Brief summary of the work
The authors propose the latent spatial Dirichlet allocation method, which is suitable for a broad range of spatial data classes and dependencies. The method is based on interpreting image data as being made out of different segments (=documents about topics) and each of those contains multiple words (=cells in the example). Each topics contains more of certain words.  Furthermore, the method is based on using a GP to sample the parameter alpha for the Dirichlet distribution for the segment-specific topic distribution. Thereby, they can account for spatial dependence between different segments. The whole model including GP is trained end-to-end as shown in Appendix A. The suitability of this approach is verified using a real and a synthetic dataset.

## Evaluation regarding quality
### Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? 

The method is comprehensible using additionally the details provided in the appendix. However, additional information could be included for further clarification:
* The font size in Figure 1a is too small and difficult to read. Similarly, Figure 2 contains images that are too small. Increasing their size would enhance readability and make the results visible.
* Including illustrations of samples from the GP prior in the appendix would provide helpful visual context.
* The input to your GP consists of the spatial locations $s_d$​ of the $d$-th segment, meaning if $s_{d_1}$ and $s_{d_2}$​ are close to each other, the corresponding Dirichlet parameters alpha will also be close. However, how do you determine the number of segments?
* Is the GP trained or optimized during the LSDA model optimization as described in Appendix A? A brief sentence in the main paper indicating this would be beneficial. Initially, it seems as though the GP is an external model, and it wasn’t until much later that I realized it’s optimized alongside the LSDA model.
* In the appendix, you mention that default values are used for $a$ and $b$. Could these parameters be learned? This model is intriguing—learning $a$ and $b$ could enable its application across various datasets or domains. Does the GP tend to suffer from a reversion to the mean?

### Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
A short limitations section could be helpful.
### Are the methods used appropriate? 
yes
### Is this a complete piece of work or work in progress? 
A workshop paper

## Evaluation regarding clarity
### Is the submission clearly written?
Yes. Typo: line 10: arise
### Is it well organized? (If not, please make constructive suggestions for improving its clarity.) 
Since Figures 2a and 2b are identical to Figure 1a, Figure 1a could be referenced rather than displaying the same figure twice. This would reduce redundancy and give you more space.
### Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)
Yes

## Evaluation regarding originality
### Are the tasks or methods new? 
Yes, to the best of my knowledge.
### Is the work a novel combination of well-known techniques? (This can be valuable!) 
Yes. LDA and GPs.
### Is it clear how this work differs from previous contributions? 
As this is a workshop paper, there is no space to describe the methods of previous contributions. I would however suggest to do that in the main paper.
### Is related work adequately cited
Yes, except of: The citation in line 32: Chen et all (2014) seems incorrect. You might have intended to cite:
Chen, Zhenghao, et al. "Modeling multiplexed images with spatial-LDA reveals novel tissue microenvironments." Journal of Computational Biology 27.8 (2020): 1204-1218.

## Evaluation regarding significance
### Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them?
The method shows and new method of sampling alpha values for the spatial Dirichlet allocation process, introduced in the paper. Therefore, spatial GPs are used. The input to the GPs is spatial location information of pixels in image data. Pixels in image data are correlated, therefore, sampling the parameter alpha for the Dirichlet distribution for the spatial dependence between different segments using a GP will give similar alphas for similar pixels in the whole spatial region and different alpha values for distinct pixels. 
Yes, this is in interesting approach to use LDA on spatial data. I would like to see the full paper version, after more experiments have been included and the method has been further refined.

### Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

How many segments did you use in your experiments, and how many grid points (i.e., spatial locations) did each segment have? Providing this information would help assess the computational efficiency of your GP model.
You mention that your method can be applied or easily extended to a wide range of spatial data classes and dependencies. In your experiments, you demonstrate its use on a multiplex imaging dataset and synthetic data derived from that. Have you tested it on other spatial datasets? Given the potential of this method, I believe there is room to evaluate the method on different datasets and get interesting results in future works, e.g. on datasets for segmentation.

## A list of its pros and cons
* Interesting idea of incorporating GPs to ensure nearby segment locations in documents/segments of the image are similar.
However, unclear how the number of documents gets defined.
* MCMC sampling well described in the appendix
* Interesting application: Multiplex imaging dataset of specific cancer cells and synthetic dataset considering various SNR values
However, it remains unclear how the method would generalize to other spatial data classes as claimed in line 37. Additionally, there is a lack of clarity on how to calculate ARI and how the baseline methods function. This might be understandable given that this is a workshop paper, but further elaboration would improve the paper's accessibility. Also, results not readable (Figures too small).
* Method well described with extensive appendix. Results show that method works well for the described approach.
However, no limitations section. Additionally, regarding line 70, for clarification: a spatial Dirichlet process is defined in the following reference:

Gelfand, Alan E., Athanasios Kottas, and Steven N. MacEachern. "Bayesian nonparametric spatial modeling with Dirichlet process mixing." Journal of the American Statistical Association 100.471 (2005): 1021-1035.

You mention that you introduce a spatial Dirichlet allocation process. How does your spatial Dirichlet allocation process differ from or compare to that one? Are you, in fact, introducing a new stochastic process or a method for sampling parameters for the Dirichlet distribution? Providing this clarification would help to understand the contribution, as you could then clearly point out what you did.
title: title: Good paper but can be made more solid
main_review: main_review: Summary: The authors propose Latent Spatial Dirichlet Allocation to capture the spatial structures present in the data by incorporating GPs to the traditional LDA model. The authors then develop efficient approximations for posterior inference of the proposed model. Finally, they apply the model to cancer multiplex imaging data as well as synthetic datasets to demonstrate the effectiveness of the model.

Strength: The paper is well-written, with a clear explanation of the methodology. The authors apply the proposed method to two cases, showcases its potential.

Weakness: 1. When applying the model to the cancer multiplex imaging data, the authors fail to compare with other baseline models such as LDA and Spatial-LDA. Including such results will highlight the advantages and gains of the proposed model. 2. The performance of the proposed model on synthetic data may not be entirely surprising, as the generative model for the synthetic data is identical to the proposed model itself. This raises questions about the fairness of the comparison, and additional evaluations on more varied synthetic datasets would strengthen the claims.