# SimCDL: A Simple Framework for Contrastive Dictionary Learning
## 1 Introduction  
_Dictionary learning_ (DL) is a branch of signal processing and machine learning that aims to find a frame (or dictionary) for sparsely representing signals as a combination of few elements. The initial basis is unknown and is usually learned from the data. This method has been widely used in multiple fields, such as image and audio processing, inpainting, compression, feature extraction, clustering, and classification. The DL problem can be formulated as follows:  
\[ \begin{array}{ll}\min_{\mathbf{D},\mathbf{X}}&\left\|\mathbf{Y}-\mathbf{D}\mathbf{X}\right\|_{F} ^{2}\\ \text{s.t.}&\left\|\mathbf{x}_{t}\right\|_{0}\leq s,\ell=1:N\\ &\left\|\mathbf{d}_{t}\right\|=1,j=1:n,\end{array}  \tag{1}\]  
where \(\mathbf{Y}\in\mathbb{R}^{m\times N}\) is the matrix that contains \(N\) signals of size \(m\), stored compactly as columns, \(\mathbf{D}\in\mathbb{R}^{m\times n}\) is named dictionary and is usually overcomplete (\(m<n\)), and \(\mathbf{X}\in\mathbb{R}^{n\times N}\) is the coefficients' matrix. The column vectors of matrix \(\mathbf{D}\) are named atoms and define the basis vectors used for the linear combination. For each signal sample, only \(s\) atoms are used for the representation. The matrix \(\mathbf{X}\) is a sparse matrix that contains the coefficients associated with the atoms used for the linear representation. The DL problem is typically solved in two optimization stages: dictionary update and sparse coding. At the beginning of the algorithm, the dictionary \(\mathbf{D}\) is randomly initialized with normalized atoms; random initialization does not favor obtaining the best results, but satisfactory results can still be obtained. In the sparse coding stage, the dictionary is considered fixed and all sparse coefficients (the elements of the matrix \(\mathbf{X}\)) are computed for each sample signal. A popular method for sparse coding is Orthogonal Matching Pursuit (OMP) [1]. This greedy algorithm selects the atoms sequentially based on their correlation with the residual representation. In the next stage, the dictionary matrix, \(\mathbf{D}\), is updated, while the coefficient matrix is considered fixed. Several algorithms are available for updating the dictionary matrix [2]. The K-means Singular Value Decomposition (K-SVD) [3]and Approximate K-SVD (AK-SVD) [4] are among the most used update algorithms. The optimization continues by alternating between these two stages until a stopping criterion is met. In general, two different criteria are used: the maximum number of iterations or the representation error dropping off under a threshold value. Dictionary learning (DL) can be used in various application problems. Considering the sparse representation capabilities and the light optimization procedure, relevant results can be obtained in practice. However, performance usually depends on the initialization of the dictionary and its properties. For example, in classification problems or discriminative learning, the incoherence of the atoms becomes relevant. Incoherence is the property of the atoms to be far apart one from another; equivalently, their scalar product is small in absolute value. Several methods include discriminative terms in the optimization procedure to meet the incoherence goal. On the other hand, a pre-trained dictionary can be advantageous when prior knowledge about the data is known. In this paper, we address the dictionary initialization problem using a previous problem validated for deep neural networks in a self-supervised manner. This initialization can be used for general problems or classification problems solved with DL. _Contrastive learning_ is a method that aims to learn similar and dissimilar representations from the data. This method can be used in supervised problems with known sample labels. However, the discrimination problem becomes untractable when dealing with large datasets. A straightforward method for learning instance discrimination is the use of categorical cross-entropy loss.