Before revision:  In this paper, we explore the idea of applying a learnable function to the input to make the CI condition hold . In particular, we first rigorously formulate the criteria that the function needs to satisfy. We then design an ingenious loss function for learningsuch a function and prove that the function minimizing the proposed loss satisfies the above criteria. We theoretically study the number of labeled data required, and give a model-free lower bound showing that taking limited downstream data will hurt the performance of self-supervised learning.
After revision:  In this paper, we explore the idea of applying a learnable function to the input to make the CI condition hold by using downstream data to refine the unlabeled data to boost self-supervised learning. At first glance, one might think that seeing downstream data in advance would always boost the downstream performance. However, we show that it is not intuitively true and point out that in some cases, it will hurt the performance of self-supervised learning.