# SimCDL: A Simple Framework for Contrastive Dictionary Learning
## 1 Introduction
As the dataset increases in size, performing the softmax operation to calculate class probabilities becomes excessively costly. The challenge of using categorical cross-entropy loss, such as discrimination, lies in its computational infeasibility when dealing with large datasets. Researchers have been searching for more efficient ways to approximate this loss. Recent advances are inspired by metric learning, as well as the work in [5; 6]. A significant challenge within the instance discrimination framework is the lack of intraclass variability. In traditional supervised learning, there are typically hundreds or thousands of examples per class, which helps the algorithm to learn the inherent variation within each class. However, in many applications, there are only a few examples per class, which clearly hinders the learning process. This issue can be tackled through extensive data augmentation. By applying different transformations to a specific data point, we can generate slightly varied versions while maintaining its fundamental semantic meaning. This approach allows us to learn valuable representations without relying on explicit labels. To set up the contrast between instances, several views of the inputs are produced using a process \(\mathcal{T}\) and then evaluated in the representation space. For a particular input \(\mathbf{y}_{i}\), an anchor is calculated as \(\mathbf{y}^{(a)}\sim\mathcal{T}(\mathbf{y}_{i})\) and then compared to a positive sample \(\mathbf{y}^{(p)}\sim\mathcal{T}(\mathbf{y}_{i})\), which is another transformation of the same input or a sample from the same class with the anchor. A negative sample, \(\mathbf{y}^{(n)}\sim\mathcal{T}(\mathbf{y}_{j})\), which represents a transformation of a different input, is also contrasted with the anchor. In addition, the process \(\mathcal{T}\) is modified or updated so that it represents positive pairs in a compact form, while negative pairs are projected far apart. In the general context of Self-Supervised Representation Learning (SSRL), this approach involves a pretext task generator \(\mathcal{P}\) that creates pretext inputs for multiple pairs of raw input instances. These inputs have pseudo-labels that indicate whether the pairs are matching or not. The anchor and positive and negative samples are then processed by a feature extractor \(h(\cdot)\) to derive their respective representations: \(\mathbf{x}^{(a)}=h(\mathbf{y}^{(a)})\) for the anchor, \(\mathbf{x}^{(p)}=h(\mathbf{y}^{(p)})\) for the positive sample, and \(\mathbf{x}^{(n)}=h(\mathbf{y}^{(n)})\) for the negative sample. After this, a similarity function denoted by \(\Phi\) is used to evaluate the similarity between pairs of projections. The whole model is subsequently trained to minimize the distance between positive pairs and to maximize the distance between negative pairs. A simple formulation of contrastive learning loss is formulated as follows:  
\[\mathcal{L}_{\text{con}}=-\mathbb{E}\left[\log\frac{\Phi\Big{(}\mathbf{x}^{(a)},\mathbf{ x}^{(p)}\Big{)}}{\Phi\big{(}\mathbf{x}^{(a)},\mathbf{x}^{(p)}\big{)}+\sum_{j=1}^{k}\Phi \big{(}\mathbf{x}^{(a)},\mathbf{x}^{(n)}\big{)}}\right], \tag{2}\]  
where \(k\) represents the number of negative samples that have been used in contrast with the anchor. The training process can also include the transformation process \(\mathcal{T}\) or only the encoder \(h(\cdot)\). Multiple versions of this strategy can be employed within this framework. The methods vary based on the similarity function, the family of transformations \(\mathcal{T}\), the encoder function \(h(\cdot)\), and approaches to the sampling anchor, and positive and negative examples. There are many unsupervised applications [7] that have been developed in the spirit of Self-Supervised Representation Learning (SSRL). These methods enable the development of generalizable models that have the potential to learn and recognize a wide variety of patterns in the data.