# SimCDL: A Simple Framework for Contrastive Dictionary Learning
## 2 Contrastive Dictionary Learning
The loss for a positive pair of examples is calculated using a softmax function over the similarity scores of all pairs within the mini-batch, scaled by a temperature parameter \(\tau\):  
\[\ell(i,j)=-\log\frac{\exp(\Phi(\mathbf{x}_{i},\mathbf{x}_{j})/\tau)}{\sum_{k=1}^{2K}[1 _{\{k\
eq i\}}\exp(\Phi(\mathbf{x}_{i},\mathbf{x}_{k})/\tau)]}. \tag{4}\]  
The global loss function is summed across all positive pairs, representing the normalized temperature-scaled cross-entropy loss (NT-XEnt). The numerator \(\exp(\Phi(\mathbf{x}_{i},\mathbf{x}_{j})/\tau)\) encourages positive pairs to be closer, while the denominator \(\sum_{k=1}^{2K}[1_{\{k\
eq i\}}\exp(\Phi(\mathbf{x}_{i},\mathbf{x}_{k})/\tau)]\) introduces competition with all other representations in the batch, treating them as negatives. The contrastive loss can be interpreted as the maximization of the similarity between positive pairs in relation to the similarity between negative pairs. This process effectively forms a distribution over possible pairs, emphasizing the relative similarity of positive pairs over negatives. The temperature parameter \(\tau\) controls the sharpness of the similarity scores. From a mathematical point of view, this parameter affects the relative weighting of similarities. A lower-value \(\tau\) leads to sharper distributions, which heavily penalizes dissimilarities between positive pairs, leading to a stronger focus on very close positives. On the other hand, a higher-value \(\tau\) treats similarities equally, which can lead to the avoidance of overemphasizing the few closest pairs. In general, the temperature term \(\tau\) can be seen as controlling the entropy of the similarity distribution. Using the NT-XEnt loss function, the full dictionary is updated using a Stochastic Gradient Descent (SGD) procedure, where the gradient is computed using reverse-mode automatic differentiation. The optimization result leads to more diverse quasi-orthogonal atoms that can better represent all the samples available in the training set. This problem is similar to a frame design problem, in which the atoms are designed to represent the samples better. In the context of SimCDL, we want to randomly initialize a dictionary and optimize it following the SGD procedure. Our experiments demonstrate that relevant results can be obtained for small batch sizes and several iterations, leading to better representation errors. The idea of SimCDL is summarized in Algorithm 1. The contrastive representation learning optimizes asymptotically two main properties: the _alignment_ of features from positive pairs and _uniformity_ of the induced distribution of the features on the hypersphere. In a theoretical study [12], the authors demonstrated that the NT-Xent loss inherently promotes uniformity on the hypersphere. The property of _alignment_ ensures that a pair of positive samples \((\mathbf{x}_{i},\mathbf{x}_{j})\) pushes them closer together in the latent space. This can be seen as maximizing the similarity \(\mathbb{E}[\Phi(\mathbf{x}_{i},\mathbf{x}_{j})]\rightarrow\max\), encouraging the alignment of different augmentations from the same sample. The property of _uniformity_ pushes negative pairs apart. In this way, the representations are spread over the entire space, \(\mathbb{E}_{k\
eq i\
eq i}[\Phi(\mathbf{x}_{i},\mathbf{x}_{k})]\rightarrow\min\), with the aim of achieving a uniform distribution over the unit hypersphere. These properties share similarities with the concept of the _incoherence of atoms_ in dictionary learning. The underlying goal of the problem is the same, but applied in a different context. In dictionary learning, _coherence_ refers to the similarity between different atoms. This suggests that the use of contrastive representation learning should lead to more incoherent dictionaries. The underlying mechanisms of building incoherent atoms is similar to the strategy of contrastive representation learning.