# Survey on Implementations of Generative Adversarial Networks for Semi-Supervised Learning
## 2 Common Techniques Used in Semi-Supervised Learning
### Consistency Regularization  
Consistency regularization [4] is an important technique used in SSL and relies on the manifold and smoothness assumptions. The technique assumes that realistic perturbations of data points in the input space (vi data augmentation, for example) should not significantly change the predicted labels of the model [9]. In simpler terms, if an input is disturbed in a way that preserves its semantics using operations such as image flipping or cropping, for example, the output label should be close to the output label for the original image. The idea is operationalized by adding a consistency regularization term to the loss function [4] that penalizes any sensitivity the model shows to the various perturbations [10]. The initial implementation of consistency regularization for deep SSL is most commonly attributed to [PERSON] et al. [11] where random augmentations were applied to the same data sample that forced predictions to be similar by proposing an unsupervised loss function that minimized the mean-squared difference between different passes of a single data point through the network. Additionally, another loss function called the mutual-exclusivity loss was used to ensure that the model's prediction vector had only one non-zero element, thereby forcing each prediction to be valid and non-ambiguous. Subsequently, the idea of temporal ensembling was introduced by [PERSON] et al. [12], which used an exponential moving average of historical predictions at different epochs of training as one part of the output. However, the downside of this method was that predictions would change only after an entire epoch, which was troublesome in the case of large datasets. Therefore, the mean teacher model was proposed, which averaged model weights instead of previous predictions [13]. An alternate approach was proposed by [PERSON] et al. [14] who proposed adding an additional regularization in the form of a contrastive loss on the predictions, thus forcing predictions to be different when the data points were from different classes. Another interesting approach was proposed by [PERSON] et al. [15] in which virtual adversarial training (VAT) was used to add perturbations to the data in order to achieve consistency regularization on the model predictions. An adversarial dropout was introduced by [PERSON] et al. [16] that involved a dropout mask being learnt for data perturbation in a direction adversarial to the model's virtual label assignment. More recent work includes [PERSON] et al. [17] proposing interpolation consistency training that encourages predictions at the interpolated data sample pairs to be consistent with the interpolated predictions, which helps move the decision boundary to low-density regions of the data space. A recent approach involving the use of consistency regularization was proposed in ReMixMatch [18], which did so by strongly augmenting an input multiple times and training the model to encourage the prediction for all strongly augmented images to be consistent with the prediction for a weakly augmented version of the same image. Given the importance of the aforementioned techniques in the area of semi-supervised learning, numerous GAN-based SSL approaches have also leveraged these techniques. Consistency regularization was used by a number of GAN-based solutions such as [PERSON] et al. [19] that added a consistency term to the loss function inspired by temporal ensembling [12]. Similarly, [PERSON] et al. [20] reported that GAN-based SSL techniques lagged behind other SSL techniques due to a lack of consistency in class probability predictions for the same image under local perturbations. The authors attempted to solve the issue by adding an auxiliary loss term to the discriminator, which accounted for consistency regularization by using an approach based on the Mean Teacher [13]. [PERSON] et al.