SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifei Zhang†, Hao Zhu‡,§, Yankai Chen†, Zixing Song†, Piotr Koniusz*,§,†, Irwin King†
†The Chinese University of Hong Kong
$^{\ddagger}$ Australian National University,  ${}^{8}$ Data61♥CSIRO
{yfzhang, ykchen, zxsong, king} @cse.cuhk.edu.hk
allenhaozhu@gmail.com, piotr.koniusz@data61.csiro.au
Graph Collaborative Filtering (GCF) is widely used in personalized recommendation systems. However, GCF suffers from a fundamental problem where features tend to occupy the embedding space inefficiently (by spanning only a low-dimensional subspace). Such an effect is characterized in GCF by the embedding space being dominated by a few of popular items with the user embeddings highly concentrated around them. This enhances the so-called Matthew effect of the popularity bias where popular items are highly recommend whereas remaining items are ignored. In this paper, we analyze the above effect in GCF and reveal that the simplified graph convolution operation (typically used in GCF) shrinks the singular space of the feature matrix. As typical approaches (i.e., optimizing the uniformity term) fail to prevent the embedding space degradation, we propose a decorrelation-enhanced GCF objective that promotes feature diversity by leveraging the so-called principle of redundancy reduction in embeddings. However, unlike conventional methods that use the Euclidean geometry to relax hard constraints for decorrelation, we exploit non-Euclidean geometry. Such a choice helps maintain the range space of the matrix and obtain small condition number, which prevents the embedding space degradation. Our method outperforms contrastive-based GCF models on several benchmark datasets and improves the performance for unpopular items.
Collaborative Filtering (CF) has emerged as an effective technique for personalized recommendations [9, 6, 44, 8, 7]. By leveraging embedding and optimizing with a suitable loss function, CF can effectively capture complex patterns in user-item interactions [32, 16]. However, user-item interactions usually neglect the potential of the higher-hop connections. To this end, Graph Neural Networks (GNN) [36, 20, 14, 34, 50, 26] have been used to model high-hop connections in CF. Graph Collaborative Filtering (GCF) has also been proposed based on CF [15, 41].
Graph Contrastive Learning (GCL), a powerful tool for unsupervised learning, has recently been applied on a variety of different tasks [5, 47, 30, 49, 50]. Many methods also demonstrate that GCL improves the performance of GCF. The basic idea of GCL in GCF is to enhance training data with various graph augmentations and maximize the alignment between different data views [46, 43, 53]. However, the above methods ignore the disadvantages of GCL (e.g., dimensional collapse) and the negative impact on user and item embeddings (e.g., the Matthew effect) [4]. Dimensional Collapse (DC) [28] can be characterized by high similarity of features which are confined to a low-dimensional subspace or are contrated on curtain region of latent space (Figure 1b). DC limits the representational power of high-dimensional spaces by restricting the diversity of information that can be learned [28].
*The corresponding author. Code: https://github.com/yifeiacc/LogDet4Rec/
Although DC has been identified and tackled in a variety of domains [19], DC poses issues for GCL in the Graph Collaborative Filtering where it exacerbates the Matthew effect of popularity bias (as explained next).
A significant issue that sets GCL in GCF apart from other domains is the highly skewed data distribution over the graph where most users interact sporadically with a very small set of items within a vast interaction space comprising millions or even billions of items [4]. This results in extremely sparse user-item interactions and a long-tail item distribution following the Power Law. Propagating highly skewed data through the sparse bipartite graph via message passing enhances the Matthew effect of the popularity bias [4]. This means that several popular items may dominate the entire learning process, while other items are rarely recommended. Thus, a collapse mode may occur, where the user embedding collapses around several popular items, as shown in Figure 1a.
To migrate this issue, previous research argues that the so-called uniformity term included in the auxiliary contrastive loss can solve DC and so the uniformity term was adopted it in CF [45, 46, 38]. Particularly, Yu et al. [46] provide an empirical evidence that the features learned by the typical GCF model are non-uniformly distributed on the sphere, which causes the collapse. Wang et al. [38] have further investigated how optimizing the Bayesian Personalized Ranking (BPR) loss in CF can promote both the alignment and uniformity, and suggested to directly optimize them. However,
(a) User Collapse in GCF.
(b) Dimensional collapse in 3D hypersphere.
Figure 1
while optimizing the uniformity of both the user and item embeddings in CF can refine the embedding space, it does not guarantee the elimination of dimensional collapse. Thus, a fundamental question arises: is there any provable remedy for DC?
To this end, we analyze the connection between dimensional collapse and GCF by demonstrating that the simplified graph convolution operation for collaborative filtering does shrink the singular space of the feature matrix. Although previous methods promote the feature uniformity, it does not prevent the dimensional collapse. Thus, we propose a new objective, Decorrelation-Enhanced GCF, that employs geometry of the LogDet divergence to relax orthogonality constraints and encourage the feature diversity. Our method maintains the range space of the feature matrix with a small condition number (well-conditioned), providing theoretical guarantees against dimensional collapse. Our experimental results demonstrate the effectiveness of the proposed approach in addressing feature collapse. We outperform contrastive-based CF models on several benchmark datasets. Notably, our approach improves the performance for unpopular items.
i. We demonstrate that embeddings learned from existing GCF models suffer from dimensional collapse. Our theoretical analysis reveals that the dimensional collapse in GCF arises from propagation, and that the current uniformity-based objective does not guarantee to alleviate it.
ii. To mitigate the issue of dimensional collapse, we propose a decorrelation-enhanced objective following the principle of redundancy reduction [47] and LogDet geometry.
iii. Instead of using the Frobenius norm (the Euclidean geometry) to relax the conditional optimization problem, we propose using the LogDet divergence, a Bregman divergence with theoretical guarantees, to preserve the range space of the covariance matrix.
Our experiments show that our proposed method effectively mitigates dimensional collapse and outperforms various contrastive learning-based CF models. Notably, it achieves improved the performance for unpopular items, thus it mitigates the Matthew effect.
Collaborative Filtering. Let  $\mathcal{U}$  and  $\mathcal{I}$  denote the user and item set, respectively. Given a set of observed user-item interactions  $\mathcal{R} = \{(u,i)\mid u$  interacted with  $i\}$ , CF methods infer the score  $s(u,i)\in \mathbb{R}$  for each unobserved user-item pair indicating how likely the user  $u$  is to interact with
the item  $i$ . Then, items with the highest scores for each user will be recommended based on the predictions [32, 16]. Most CF methods use an encoder network  $f(\cdot)$  that maps each user and item into a low-dimensional representation  $f(u), f(i) \in \mathbb{R}^d$  (where  $d$  is the dimensionality of the latent space). For example, the encoder in matrix factorization models is usually an embedding table, which directly maps each user and item to a latent vector based on their IDs. The encoder in graph-based models uses the neighborhood information. The predicted score is defined as the similarity between the user/item representations, e.g., dot product,  $s(u,i) = f(u)^{\top}f(i)$ . Most studies adopt the pairwise BPR [31] loss to train the model:
$$
\mathcal {L} _ {b p r} = - \frac {1}{| \mathcal {R} |} \sum_ {(u, i) \in \mathcal {R}} \log \left[ \operatorname {s i g m o i d} \left(s (u, i) - s (u, i ^ {-})\right) \right], \tag {1}
$$
where  $i^{-}$  is a randomly sampled negative item that the user has not interacted with. Such a loss maximizes sigmoid scores: target user-item scores become higher than user-negative item scores.
LightGCN. A linear graph neural network backbone, LightGCN [15], is one of the most successful backbones in Graph Collaborative Filtering. It leverages neighborhood information at different levels of locality, and combines outputs of different layers by the sum aggregation.
To facilitate the analysis, we present the matrix from of the LightGCN. Let  $\mathbf{R} \in \mathbb{R}^{M \times N}$  be the matrix representation of set  $\mathcal{R}$  where  $R_{ui} = 1$  if  $(u,i) \in \mathcal{R}$  (else  $R_{ui} = 0$ ). Let  $M$  and  $N$  be the numbers of users and items, and  $N' = M + N$ . We then obtain the adjacency matrix of the user-item graph as:
$$
\mathbf {A} = \left( \begin{array}{c c} \mathbf {0} & \mathbf {R} \\ \mathbf {R} ^ {\top} & \mathbf {0} \end{array} \right) \in \mathbb {R} ^ {N ^ {\prime} \times N ^ {\prime}}. \tag {2}
$$
Let the embedding matrix of input layer be  $\mathbf{E} \in \mathbb{R}^{N' \times d}$ , where  $d$  is the embedding size. Then the  $K$ -th layer of LightGCN in the matrix form is defined as:
$$
\mathbf {Z} = f (\mathbf {E}; \tilde {\mathbf {A}}, K) = \alpha_ {1} \tilde {\mathbf {A}} \mathbf {E} + \alpha_ {2} \tilde {\mathbf {A}} ^ {2} \mathbf {E} + \dots + \alpha_ {K} \tilde {\mathbf {A}} ^ {K} \mathbf {E} = \sum_ {k = 1} ^ {K} \alpha_ {k} \tilde {\mathbf {A}} ^ {k} \mathbf {E}, \tag {3}
$$
where  $\alpha_{1},\alpha_{2},\ldots ,\alpha_{K}$  are weights, and  $\sum_{k = 1}^{K}\alpha_{k} = 1$  . Moreover,  $\tilde{\mathbf{A}} = \mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}$  is a normalized adjacency matrix and the degree matrix  $\mathbf{D}$  is an  $N^{\prime}\times N^{\prime}$  diagonal matrix in which each entry  $D_{ii}$  denotes the number of non-zero entries in the  $i$  -th row vector of the adjacency matrix A.
Dimensional Collapse (DC). Often referred to as spectral collapse [28], dimensional collapse is a prevalent phenomenon in representation learning. It occurs when the embedding space is dominated by a small number of large singular values, while the remaining singular values decay significantly as the training progresses. This phenomenon limits the representation power of high-dimensional spaces by restricting the diversity of information that can be learned.
3 Popularity Bias from a Perspective of Dimensional Collapse
3.1 Popularity Bias vs. Dimensional Collapse in Graph Collaborative Filtering
Dimensionality collapse and Popularity Bias in CF are two sides of the same coin. The prevalence of popularity bias in GCF can be attributed to the significant imbalance within the data distribution across the recommendation bipartite graph [4]. This distribution often features numerous users engaging with a small subset of items, within a much larger interaction space encompassing millions or billions of items. Consequently, during the process of message passing within GCF, a few popular items can disproportionately influence learning, leaving out other items. This leads to the dimensional collapse in the embedding space where most user embeddings are around very few popular item embeddings.
We illustrate this by showing the 2D latent embedding space of the user and item in Figure 2:
i. Figure 2 (top) shows that both user and item embedding are clustered in a few regions, the gaps in the unpopular item embedding cause effective rank drop. We refer to this effect as the dimensional collapse. As the unpopular items are clustered in just one cluster, and the corresponding user cluster is non-dense so unpopular items do not get recommended often (popularity bias).
Figure 2: Mitigating Popularity Bias. (top) We applied 2D PCA and the  $\ell_2$  normalization on embeddings of LightGCN (Yelp2018 dataset). Notice that when only a non-dense user cluster aligns with the unpopular item cluster, these items will not be recommended frequently  $(\mathcal{X})$ . (bottom) With our LogDet-based loss, the unpopular-item cluster is spread into three clusters across surface of hypersphere. Newly formed clusters align well with dense user clusters  $(\checkmark)$  so the unpopular items will be recommended more often. The dimensional Collapse manifests itself with gaps on the surface of hypersphere. Each plot was renormalized by its maximum density (darkest color). The 3D ball effect is to remind reader the embeddings exist on the surface hypersphere but in our simulation we use in fact a 2D sphere.
ii. If only the embeddings were spread uniformly on the surface of the hypersphere, it would align with dense user clusters which would reduce the popularity bias. Figure 2 (bottom) shows that thanks to more uniformly spread unpopular items, they form now 3 clusters, and 2 of them align well with dense user clusters, which means the popularity bias has been reduced.
3.2 Dimensional Collapse in Graph Collaborative Filtering
The previous section reveals that the occurrence of dimensional collapse is closely related to the presence of popularity bias, as depicted in Figure 2. Below, we show how the typical backbone of GCL in GCF (i.e., LightGCN) potentially causes the DC in the learned embedding space, and why optimizing uniformity [38] (the best current solution) cannot guarantee to prevent DC.
Lemma 1 (Equivalent reformulation of LightGCN.) The propagation process of LightGCN in Equation (3) can be viewed as a result of optimization of the following objective:
$$
\mathbf {Z} ^ {*} = \underset {\mathbf {Z}} {\arg \min } \| \mathbf {Z} - \tilde {\mathbf {A}} \mathbf {E} \| _ {F} ^ {2} + \eta \operatorname {t r} \left(\mathbf {Z} ^ {\top} \mathbf {L} \mathbf {Z}\right), \tag {4}
$$
where  $\mathbf{L} = \mathbf{I} - \tilde{\mathbf{A}}$  is the normalized symmetric positive semi-definite graph Laplacian matrix. Proof of Lemma 4 is in Appendix A.1.
Lemma 1 shows that the propagation process of LightGCN can be derived from the optimization formulation which uses the Graph Laplacian regularization. This implies that the output embedding  $\mathbf{Z}$  is smooth due to the Graph Laplacian regularization in Equation (4). Similar derivations for the graph neural network with learnable weights were shown by Ma et al. [29] and Zhu et al. [58].
To understand the dynamics of the embedding space, we denote the state of the embedding matrix and the output of the LightGCN at time  $t$  as  $\mathbf{E}(t)$  and the initial state  $\mathbf{E}(0)$  is randomly initialized. Let  $\{\sigma_n^{\mathbf{E}}\}_{n=1}^{N'}$  denote eigenvalues of  $\mathbf{E}$  (descending order)
Lemma 2 (Shrinking singular space of feature matrix.) Consider minimizing the objective in Lemma 1 with the Laplacian regularization term  $\operatorname{tr}(\mathbf{Z}^{\top}\mathbf{L}\mathbf{Z})$ . The relative value of the ratio
of the smaller eigenvalues to the larger eigenvalues will decrease as  $t$  increases. Formally,  $\frac{\sigma_i^{\mathbf{E}}(t)}{\sigma_j^{\mathbf{E}}(t)}\leq \frac{\sigma_i^{\mathbf{E}}(t')}{\sigma_j^{\mathbf{E}}(t')},\forall t <   t',i > j$  (we ignore the trivial case  $i = j$ ). Then:
$$
\lim  _ {t \rightarrow \infty} \frac {\sigma_ {i} ^ {\mathbf {E}} (t)}{\sigma_ {j} ^ {\mathbf {E}} (t)} = 0, \quad \forall i > j \quad s u c h t h a t \quad \sigma_ {i} ^ {\mathbf {E}} <   \sigma_ {j} ^ {\mathbf {E}}. \tag {5}
$$
Proof of Lemma 2 is in Appendix A.2.
Based on Lemmas 1 and 2, we conclude that due to the propagation step in Equation (3), the embedding space of feature matrix is dominated by few largest eigenvalues. This is a proof of dimensional collapse of GCF pipelines which are typically based on the LightGCN backbone.
3.3 Optimizing Uniformity in Graph Collaborative Filtering.
Typical contrastive GCF approaches adopt the BPR loss from Equation (1) and the InfoNEC loss. Wang et al. [38] argued that the so-called alignment and uniformity [38] loss terms contribute to higher recommendation performance:
$$
\mathcal {L} _ {\text {a l i g n}} = \mathbb {E} _ {(u, i) \sim \mathcal {R}} \left[ \| f _ {\theta} (u) - f _ {\theta} (i) \| _ {2} ^ {2} \right], \tag {alignment}
$$
$$
\mathcal {L} _ {u n i} = - \log \left[ \mathbb {E} _ {u, u ^ {-}} \sim_ {\mathcal {U}} e ^ {(2 - 2 f _ {\theta} (u) ^ {\top} f _ {\theta} (u ^ {-}))} \right] - \log \left[ \mathbb {E} _ {i, i ^ {-}} \sim_ {\mathcal {I}} e ^ {(2 - 2 f _ {\theta} (i) ^ {\top} f _ {\theta} (i ^ {-}))} \right], \tag {6}
$$
where  $\mathcal{L}_{algin}$  captures the user-item relations.  $\mathcal{L}_{uni}$  encourages embeddings to be distributed evenly on the surface of  $\ell_2$  ball. However,  $\mathcal{L}_{uni}$  does not avoid the dimensional collapse shown in Figure 1.
Concluding, Contrastive GCF suffers from several problems:
i. DC [17] is characterized by one or more singular values in the feature matrix to be zero, e.g.,  $\lambda_{i} = 0$ .  $\mathcal{L}_{uni}$  may yield a finite reward (decrease of  $\mathcal{L}_{uni}$ ) to avoid the dimensional collapse ( $\lambda_{i} = 0$ ). However, if the remaining loss terms yield a higher reward (decrease in their value) than  $\mathcal{L}_{uni}$  for occurrence of the dimensional collapse ( $\lambda_{i} = 0$ ) then DC will occur.
ii. As temperature  $1 / \tau$  increases,  $\mathcal{L}_{uni}$  degrades to a hard minimum. Take just the user part of Equation (6), include the temperature variable  $1 / \tau$ , and notice that:
$$
\lim  _ {\tau \rightarrow 0} - \tau \log \left[ \mathbb {E} _ {u, u ^ {-}} \sim_ {\mathcal {U}} e ^ {(2 - 2 f _ {\theta} (u) ^ {\top} f _ {\theta} (u ^ {-})) / \tau} \right] = \min  _ {(u, u ^ {-}) \in \mathcal {U} \times \mathcal {U}} 2 - f _ {\theta} (u) ^ {\top} f _ {\theta} (u ^ {-}), \tag {7}
$$
which explains the empirical observation of Wang and Liu [39] that the uniformity of embeddings worsens as the temperature increases.
iii. As Lemma 2 shows, LightGCN itself is also prone to suffering from the dimensional collapse.
Thus, in what follows we consider the redundancy reduction principle to mitigate the above issues.
Graph Collaborative Filtering with Decorrelation. From the perspective of redundancy reduction [47, 2], achieving high-quality self-supervised embeddings requires that: (i) positive pairs exhibit similar semantics; (ii) the embeddings follow a non-trivial constant distribution, with a trivial distribution indicating that all embeddings collapse into a single point; and (iii) there is zero correlation between different features.
In the context of collaborative filtering, we can frame this problem as follows:
$$
\min  _ {\theta} \mathcal {L} _ {\text {a l i g n}} (\theta) \tag {8}
$$
$$
\text {s . t .} \sum_ {i \in \mathcal {U}} f _ {\theta} (u) f _ {\theta} (u) ^ {\top} = \sum_ {i \in \mathcal {I}} f _ {\theta} (i) f _ {\theta} (i) ^ {\top} = \mathbf {I},
$$
Table 1: Several decorrelation terms, properties and teasers results (Yelp2018).
Luni	Lsoft	Lloadet	Stiefel
isotropic	soft	soft	soft	hard
full-rank	soft	soft	hard	hard
recall@20(%)	7.12	7.09	7.25	7.03
where  $\mathcal{L}_{\text{align}}(\theta) = \mathbb{E}_{(i,u)\sim \mathcal{R}}\left[\| f_{\theta}(u) - f_{\theta}(i)\| _2^2\right]$  (from Equation (6)) captures user-item preferences. Constraints  $\sum_{i\in \mathcal{U}}f_{\theta}(u)f_{\theta}(u)^{\top} = \sum_{i\in \mathcal{I}}f_{\theta}(i)f_{\theta}(i)^{\top} = \mathbf{I}$  help embeddings follow the isotropic Normal distribution. The isotropy prevents the dimensional collapse but its decorrelating effect may
Figure 3: The spectrum (eigenvalue distribution) of the covariance matrix under various loss penalties. Various penalties realize some composition of {isotropic, non-isotropic}  $\times$  {full rank, rank-deficient}, e.g., our LogDet formulation promotes the (isotropic, full rank) case.
be too strong. Equation (8) may be solved by applying Lagrangian multipliers or the Stiefel manifold (GeoTorch [25]) or by converting hard constraints into soft constraints:
$$
\min  _ {\theta} \mathcal {L} _ {\text {a l i g n}} (\theta) + \lambda \cdot \mathcal {L} _ {\text {s o f t}} \quad \text {w h e r e} \quad \mathcal {L} _ {\text {s o f t}} = \left\| \sum_ {i \in \mathcal {U}} f _ {\theta} (u) f _ {\theta} (u) ^ {\top} - \mathbf {I} \right\| _ {F} ^ {2} + \left\| \sum_ {i \in \mathcal {I}} f _ {\theta} (i) f _ {\theta} (i) ^ {\top} - \mathbf {I} \right\| _ {F} ^ {2}, \tag {9}
$$
However, relaxation  $\mathcal{L}_{soft}$  in Equation (9) suffers from the same problem as  $\mathcal{L}_{uni}$  in Equation (6). Specifically, these both regularization terms yield a finite penalty for any singular values  $\lambda_{i} = 0$  leading to dimensional collapse if other loss terms yield a higher reward than the penalty is. Thus, we explore the application of Bregman divergence. Our study indicates that the proposed approach preserves the range space of the covariance matrix, promoting a full-rank embedding matrix.
**Decorrelation by the LogDet Divergence.** Figure 3 and Table 1 summarizes properties of different decorrelation terms. While  $\mathcal{L}_{uni}$  and  $\mathcal{L}_{soft}$  encourage soft decorrelation (approximate isotropy), they may fail to impose full rank on the embedding matrix. Stiefel-based optimization enjoys hard constraints but full decorrelation (exact isotropy) may be too restrictive. In contrast, LogDet divergence based  $\mathcal{L}_{logdet}$  encourages isotropy in the soft manner and imposes full rank on embedding matrix as shown below. Firstly, we review the Bregman divergence and introduce our LogDet based decorrelation penalty.
Definition 1 (Bregman Divergence on Matrices.) Let  $\mathbf{X},\mathbf{Y}\in S^d$  be two symmetric matrices. Let  $\phi :\mathcal{S}^n\to \mathbb{R}$  be a strictly convex and differentiable matrix function. The Bregman matrix divergence is defined as:  $D_{\phi}(\mathbf{X},\mathbf{Y}) = \phi (\mathbf{X}) - \phi (\mathbf{Y}) - \mathrm{tr}\left((\nabla \phi (\mathbf{Y}))^{\top}(\mathbf{X} - \mathbf{Y})\right)$  (10)
The Bregman divergence is a measure of the nearness between matrices  $\mathbf{X}$  and  $\mathbf{Y}$ . It is computed by using the first-order Taylor approximation of a convex generating function  $\phi(\cdot)$ . The Bregman divergence is always non-negative, and it becomes zero only when the two matrices are equal.
As  $\pmb{\Sigma}_{\mathcal{U}} = \sum_{u\in \mathcal{U}}f_{\theta}(u)f_{\theta}(u)^{\top}\in S_{+}^{d}$  and  $\pmb{\Sigma}_{\mathcal{I}} = \sum_{i\in \mathcal{I}}f_{\theta}(i)f_{\theta}(i)^{\top}\in S_{+}^{d}$  are symmetric, and at least positive semi-definite (PSD) matrices but we desire them to be strictly positive definite (PD) matrices  $(S_{+ + }^{d})$  to prevent the dimensional collapse, we firstly generalize  $\mathcal{L}_{soft}$  in Equation (9) to its Bregman divergence based variant:  $\mathcal{L}_{breg} = D_{\phi}(\pmb{\Sigma}_{\mathcal{U}},\mathbf{I}) + D_{\phi}(\pmb{\Sigma}_{\mathcal{I}},\mathbf{I}).$  (11)
Different  $\phi (\cdot)$  lead to different measures of nearness. For example,  $\phi_F(\mathbf{X}) = \| \mathbf{X}\| _F^2$  yields the Frobenius-based  $D_{\phi_F}(\pmb {\Sigma}_U,\mathbf{I}) + D_{\phi_F}(\pmb {\Sigma}_\mathcal{I},\mathbf{I}) = \| \pmb {\Sigma}_U - \mathbf{I}\| _F^2 +\| \pmb {\Sigma}_\mathcal{I} - \mathbf{I}\| _F^2$  penalty  $\mathcal{L}_{soft}$  in Equation (9).