# SimCDL: A Simple Framework for Contrastive Dictionary Learning
## 2 Contrastive Dictionary Learning  
This section explains the Simple framework for Contrastive Dictionary Learning (SimCDL), which is used for dictionary initialization. This framework is developed in the spirit of SimCLR [10], a powerful approach to unsupervised representation learning. At the base of this framework lies the loss function, designed to learn rich and discriminative embeddings from unlabeled data. For the Contrastive Dictionary Learning framework, we consider the same loss function, but we propose a different logic for computing the encodings representation. In the context of dictionary learning (DL), we apply stochastic data augmentation transformations to generate pairs of correlated signals from the same example, denoted \(\mathbf{\hat{y}}_{i}\) and \(\mathbf{\hat{y}}_{j}\). These two samples are derived from an initial sample \(\mathbf{y}\). Let \(\mathcal{T}\) represent the space of augmentation operations that can be applied to the samples. For two different random initializations of the augmentation operators, we have \(t\sim\mathcal{T}\) and \(t^{\prime}\sim\mathcal{T}\). The next step is to follow an encoding process that aims to maximize the agreement between the two augmented samples in the representation space. The SimCLR problem has been adapted without using a projection head or a base encoder. Instead, the dictionary \(\mathbf{D}\) is used directly to calculate the encodings using an OMP procedure. The embeddings are represented by the column vectors of the matrix \(\mathbf{X}\). To build a positive pair of encodings, denoted as \(\mathbf{x}_{i}\) and \(\mathbf{x}_{j}\), we compute the representation coefficients of the two augmented samples, \(\mathbf{\hat{y}}_{i}\) and \(\mathbf{\hat{y}}_{j}\). In addition, contrastive loss is calculated to measure similarities between positive pairs of encodings and discriminate them from negative pairs. This setup leverages the principles of SimCLR, primarily contrastive learning, in a sparse coding problem. SimCLR is typically used to learn useful representations by maximizing the agreement between different augmented examples of the same data sample via a contrastive loss in the latent space. An example illustrating the SimCDL steps is shown in Figure 1.  
To compute the loss function, we randomly select a mini-batch of \(K\) examples at each iteration. After that, we create pairs of augmented samples for each of the \(K\) examples, resulting in a total of \(2K\) data points. By doing so, we do not need to sample negative examples explicitly. Additionally, we consider other augmented examples as negative samples in relation to the positive pair. After this, we calculate an embedding for each augmented sample using the OMP algorithm  
\[\mathbf{x}_{i}=\text{OMP}(\mathbf{\hat{y}}_{i},\mathbf{D},s), \tag{3}\]  
where \(\mathbf{D}\) is the dictionary matrix and \(s\) is the sparsity level. Figure 1: Contrastive dictionary learning framework. OMP aims to find the best sparse representation of a signal by iteratively selecting the most relevant atoms from the dictionary \(\mathbf{D}\). Iteratively, the algorithm selects the atom most correlated with the current residual. After the selection is made, the residual is updated by projecting the signal onto the subspace spanned by the selected atoms. This process is repeated until a stopping criterion is met (e.g., a desired sparsity level or error threshold). This whole process substitutes the encoding base network that was previously used in SimCLR. The similarity of the encodings, obtained with OMP, is computed using the cosine similarity of their normalized feature vectors, \(\Phi(\mathbf{u},\mathbf{v})=\mathbf{u}^{\top}\mathbf{v}/\|\mathbf{u}\|\|\mathbf{v}\|\). In contrast, the negative encoding should use other atoms in their sparse representation, leading to incoherent embeddings.