# SimCDL: A Simple Framework for Contrastive Dictionary Learning
## 2 Contrastive Dictionary Learning
Moreover,contrastive learning is related to maximizing the mutual information (MI) [13] between different augmentations of the same sample:  
\[\text{MI}(\vec{x_{i1}};\vec{x_{i2}})\geq\mathbb{E}[\log p(\vec{x_{i}}|\vec{x_{j} })-\log p(\vec{x_{i}})], \tag{5}\]  
where \(\vec{x_{1}}\) and \(\vec{x_{2}}\) are different augmentations of \(\vec{x_{i}}\). The maximization of mutual information for the data samples is related to the problem of reducing mutual coherence in the problem of dictionary learning. Since we want to enhance the representation capabilities (mutual information), more diverse atoms are needed, leading to a reduction in mutual coherence. ```
batch size \(K\), constant \(\tau\), augmentation function \(T\), dictionary \(\mathbf{D}\), sparsity constraint \(s\) for sampled minibatch \(\{y_{k}\}_{k=1}^{K}\)do for all\(k\in\{1,\dots,K\}\)do  draw two augmentation functions \(t\sim\mathcal{T}\), \(t^{\prime}\sim\mathcal{T}\) # the first augmentation \(\vec{y}_{2k-1}=t(\vec{y}_{k})\) \(\mathbf{x}_{2k-1}=\text{OMP}(\vec{y}_{2k-1},\mathbf{D},s)\) # the second augmentation \(\vec{y}_{2k}=t^{\prime}(\vec{y}_{k})\) \(\mathbf{x}_{2k}=\text{OMP}(\vec{y}_{2k},\mathbf{D},s)\) endfor for all\(i\in\{1,\dots,2K\}\) and \(j\in\{1,\dots,2K\}\)do \(\Phi(i,j)=x_{i}^{\top}\vec{x_{j}}/(\|\mathbf{x}_{i}\|\|\mathbf{x}_{j}\|)\) endfor  compute gradient of \(\mathcal{L}=\frac{1}{2K}\sum_{k=1}^{K}[\ell(2k-1,2k)+\ell(2k,2k-1)]\), where \(\ell\) is defined in (4)  update dictionary \(\mathbf{D}\) using gradient descent endfor return learned dictionary \(\mathbf{D}\)
```  
**Algorithm 1** SimCDL: main learning algorithm  
The use of SimCDL can be beneficial for the initialization of dictionaries with incoherent atoms or even incoherent sub-dictionaries. In classification problems, with \(\mathbf{Y}=[Y_{1},\dots,Y_{c},\dots,Y_{C}]\) representing a set of feature vectors, we want to learn local dictionaries, \(\mathbf{D}_{c}\), for each class. In general, the initialization problem is not addressed; simple methods like random matrices or a random selection of signals are used for initialization; we will tackle the problem using SimCDL. Considering that a class dictionary, \(\mathbf{D}_{c}\in\mathcal{R}^{m\times N_{c}}\), should achieve good representations for its class, we will further adapt the SimCRL framework for the initialization of dictionaries in classification problems. Since we need \(C\) dictionaries, we will further optimize a wide dictionary, \(\mathbf{D}=[\mathbf{D}_{1},\dots,\mathbf{D}_{c},\dots,\mathbf{D}_{C}]\in\mathcal{R}^{m\times( C\cdot N)}\). During optimization, the sparsity constraint \(s\) is set to \(N\), which is the number of atoms per class. Since we need \(N\) atoms, we want to overspecialize enough atoms for each class. After training the general dictionary, the problem of atom distribution must be solved; each atom of \(\mathbf{D}\) must be assigned to a class \(c\). For this task, we propose two different approaches. The first one is greedy: we loop over each class and search for the atoms that are most used in the sparse representation of the samples of the current class. Since class dictionaries do not have common atoms, once an atom is assigned to a class, it becomes unavailable for other classes. The second approach is based on the linear sum assignment problem [14]. This algorithm solves an optimization problem where the goal is to assign a set of workers to a set of tasks in a one-to-one manner while minimizing the total cost of the assignment. This scenario is similar to the problem of assigning the atom class with respect to their use. During our experiments, we tested both methods and decided to use the second approach.