Question: Why is it impossible to use the label for each data point?
Evidence: The goal of data augmentation for contrastive learning is to create realistically rational instances that maintain semantics through different transformation approaches. Unlike instances in vision and language domains, the underlying semantics of time series data is not recognizable to human, making it hard, if not impossible, to include human knowledge to data augmentation for time series data.
For example, rotating an image will not change its content or the label. While permuting a time series instance may ruin its signal patterns and generates a meaningless time series instance. In addition, the tremendous heterogeneity of real-life time series datasets further makes selections based on trial-and-errors impractical. Although multiple data augmentation methods have been proposed for time series data, there is less discussion on what is a good augmentation that is meaningful for a given learning task and dataset without prefabricated human priors. From our perspective, ideal data augmentations for contrastive representation should keep high fidelity, high variety, and adaptive to different datasets. The illustration and examples are shown in Figure.
High Fidelity. Augmentations with high fidelity maintain the semantic identity that is invariant to transformations. Considering the inexplicability in practical time series data, it is challenging to visually check the fidelity of augmentations. Thus, we assume that the semantic identity of a time series instance is presented by its label in the downstream task, which might be either available or unavailable during the training period. Here, we start our analysis from the supervised case and will extend it to the unsupervised case later. Based on the information theory, we define the objective that keeps high fidelity as the large mutual information (MI) between augmentation v and the label y, i.e., MI(v; y).
We consider augmentation v as a probabilistic function of x and a random variable , that v = g(x; ).
From the definition of mutual information, we have MI(v; y) = H(y) − H(y|v), where H(y) is the (Shannon) entropy of y and H(y|v) is the entropy of y conditioned on augmentation v. Since H(y) is irrelevant to data augmentations, the objective is equivalent to minimizing the conditional entropy H(y|v). Considering the efficient optimization, we follow and to approximate it with cross-entropy between y and ŷ, where ŷ is the prediction with augmentation v as the input and calculated via
where z is the representation and h w (•) is a prediction projector parameterized by w. The prediction projector is optimized by the classification objective. Then, the objective of high fidelity for  supervised or semi-supervised cases is to minimize
where C is the number of labels.
In the unsupervised settings where y is unavailable, one-hot encoding y s ∈ R |X| is utilized as the pseudo label to replace y in Eq. (). The motivation is that augmented instances are still distinguishable from other instances with the classifier. We theoretically show that augmentations that preserving pseudo labels have the following properties.
Property 1 (Preserving Fidelity). If augmentation v preserves the one-hot encoding pseudo label, the mutual information between v and the downstream task label y (although not visible to training) is equivalent to that between raw input x and y, i.e., MI(v; y) = MI(x; y).
Property 2 (Adding New Information). By preserving the one-hot encoding pseudo label, augmentation v contains new information comparing to the raw input x, i.e., H(v) ≥ H(x).
Detailed proofs are shown in the Appendix A. These properties show that in the unsupervised setting, preserving the one-hot encoding pseudo label guarantees that the generated augmentations will not decrease the fidelity, regardless of the downstream tasks and variances inherent in the augmentations. Concurrently, it may introduce new information for contrastive learning.
Since the number of labels is equal to the number of instances in dataset X in an unsupervised case, direct optimization of Eq. () is inefficient and unscalable. Thus, we further relax it by approximating y with the batch-wise one-hot encoding y B , which decreases the number of labels C from the dataset size to the batch size.
High Variety. Sufficient variances in augmentations improve the generalization capacity of contrastive learning models. In the information theory, the uncertainty inherent in the random variable's possible outcomes is described by its entropy. Considering that augmented instances are generated based on the raw input x, we maximize the entropy of v conditioned on x, H(v|x), to maintain a high variety of augmentations. From the definition of conditional entropy, we have
We dismiss the first part since the unconstrained entropy of v can be dominated by meaningless noise. Considering the continuity of both v and x, we adopt a mutual information neural estimator, InfoNCE to approximately compute the mutual information for its practical effectiveness. Other MI estimators, such as Jensen-Shannon (JSD) estimator, normalized temperature-scaled cross-entropy (NT-Xent), and leaveone-out can also conveniently be the plug-and-play component in our framework. Then, the objective to encourage high variety is to maximize the InfoNCE between v and x:
x ∈X exp(sim(z x , z v ))
where v is an augmented instance of input instance x . z x , z v , and z v are representations of instance x, v, and v respectively. sim(z 1 , z 2 ) = z T 1 z 2 is the inner product of vectors z 1 and z 2 .
Criteria. Combining the information aware definition of both high fidelity and variety, we propose the criteria for selecting good augmentations without prior knowledge,
where β is a hyper-parameter to achieve the trade-off between fidelity and variety. Note that in the unsupervised settings, y is replaced by one-hot encoding pseudo label... Similar to the information bottleneck, InfoMin assumes that augmented views are functions of the input, which heavily constrains the variance of data augmentations. Besides, high fidelity property is dismissed in the unsupervised setting. It works for image datasets due to the availability of human knowledge. However, it may fail to generate reasonable augmentations for time series data.
Since the number of labels is equal to the number of instances in dataset X in an unsupervised case, direct optimization of Eq. (2) is inefficient and unscalable. Thus, we further relax it by approximating y with the batch-wise one-hot encoding yB, which decreases the number of labels C from the dataset size to the batch size.
Answer: As we explained later in our paper, *"Since the number of labels is equal to the number of instances in dataset X in an unsupervised case, direct optimization of Eq. (2) is inefficient and unscalable. (Page 4).''*  For example, if we have 10k data points, meaning that the pseudo label is a one-hot encoding with 10k elements, such that the output dimension of the classifier will be 10k, which is inefficient with poor scalability.