Before revision:  However, recent approaches like  BYOL and SimSiam , show remarkable performance {\it without} negative pairs, raising a fundamental theoretical question : how can SSL with only positive pairs avoid representational collapse ? We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our analysis yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Furthermore, motivated by our theory we propose a novel approach  that directly sets the  predictor based on the statistics of its inputs .
After revision:  However, recent approaches like BYOL and SimSiam , show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach that directly sets the predictor based on the statistics of its inputs .