SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} Q _ {m, n} = c \left(\frac {2 \tilde {\mathcal {R}} _ {m} (\mathcal {F})}{\sqrt {m}} + \tau \sqrt {\frac {\ln (3 / \delta)}{2 m}} + \tau_ {\bar {S}} \sqrt {\frac {\ln (3 / \delta)}{2 n}}\right) \\ + \kappa_ {S} \sqrt {\frac {2 \ln (6 | \mathcal {Y} | / \delta)}{2 n}} \sum_ {y \in \mathcal {Y}} \left(\sqrt {\hat {p} (y)} + \sqrt {p (y)}\right) \\ + \frac {4 \mathcal {R} _ {m} (\mathcal {W} \circ \mathcal {F})}{\sqrt {m}} + 2 \kappa \sqrt {\frac {\ln (4 / \delta)}{2 m}} + 2 \kappa_ {\bar {S}} \sqrt {\frac {\ln (4 / \delta)}{2 n}}. \\ \end{array}
$$
Now,
Appendix J Information Optimization and the VICReg Objective
Assumption 1. The eigenvalues of  $\Sigma (x_{j})$  are in some range  $a\leq \lambda (\Sigma (x_j))\leq b$
Assumption 2. The differences between the means of the Gaussians are bounded
$$
M = \max _ {i, j} \| \mu (X _ {i}) - \mu (X _ {j}) \| ^ {2}
$$
Lemma J.1. The maximum eigenvalue of each  $\mu(X_j)\mu(X_j)^T$  is at most  $M$ .
Proof. The term  $\mu(X_j) \mu(X_j)^T$  is an outer product of the mean vector  $\mu(X_j)$ , which is a symmetric matrix. The eigenvalues of a symmetric matrix are equal to the squares of the singular values of the original matrix. Since the singular values of a vector are equal to its absolute values, the maximum eigenvalue of  $\mu(X_j) \mu(X_j)^T$  is equal to the square of the maximum absolute value of  $\mu(X_j)$ . By the second assumption, this is at most  $M$ .
Lemma J.2. The maximum eigenvalue of  $-\mu_Z\mu_Z^T$  is non-positive and its absolute value is at most  $M$ .
Proof. The term  $-\mu_Z\mu_Z^T$  is a negative outer product of the overall mean vector  $\mu_{Z}$ , which is a symmetric matrix. Its eigenvalues are non-positive and equal to the negative squares of the singular values of  $\mu_{Z}$ . Since the singular values of a vector are equal to its absolute values, the absolute value of the maximum eigenvalue of  $-\mu_{Z}\mu_{Z}^{T}$  is equal to the square of the maximum absolute value of  $\mu_{Z}$ , which is also bounded by  $M$  by the second assumption.
Lemma J.3. The sum of the eigenvalues of  $\Sigma_Z$  is bounded
$$
\sum_ {i} \lambda_ {i} (Z) \leq (b + M) \times K
$$
Proof. Given a Gaussian mixture model where each component  $Z|x_{j}$  has mean  $\mu(X_{j})$  and covariance matrix  $\Sigma(x_{j})$ , the mixture can be written as:
$$
Z = \sum_ {j} p _ {j} Z | x _ {j}
$$
where  $p_j$  are the mixing coefficients. The covariance matrix of the mixture,  $\Sigma_Z$ , is then given by:
$$
\Sigma_ {Z} = \sum_ {j} p _ {j} \left(\Sigma (x _ {j}) + \mu (X _ {j}) \mu (X _ {j}) ^ {T}\right) - \mu_ {Z} \mu_ {Z} ^ {T}
$$
where  $\mu_Z$  is the mean of the mixture distribution.
By Lemmas 1.1, 1.2, and assumptions 1 and 2, the maximum eigenvalues of  $(\Sigma(x_{j}), \mu(X_{j})\mu(X_{j})^{T}$  and  $\mu_{Z}\mu_{Z}^{T}$  are at most  $b$ ,  $M$ , and  $M$ , respectively. Therefore, by Weyl's inequality for the sum of two symmetric matrices, the maximum eigenvalue of  $\Sigma_{Z}$  is at most  $b + M$ .
$$
\lambda_ {m a x} (\Sigma_ {Z}) \leq \frac {1}{K} \sum_ {i = 1} ^ {K} (m a x (\lambda (\Sigma (X _ {i}))) + M) \leq b + M
$$
It means that we can bound the sum of the eigenvalues of  $\Sigma_Z$  with
$$
\sum_ {i} \lambda_ {i} (\Sigma_ {Z}) \leq (b + M) \times K
$$
Lemma J.4. Let  $\Sigma_Z$  be a positive semidefinite matrix of size  $N\times N$ . Consider the optimization problem given by:
$$
m a x i m i z e \log \det (\Sigma_ {Z})
$$
such that:
$$
\sum_ {i = 1} ^ {N} \lambda_ {i} (\Sigma_ {Z}) \leq c
$$
$$
\Sigma_ {Z} \succeq 0
$$
where  $\lambda_{i}(\Sigma_{Z})$  denotes the  $i$ -th eigenvalue of  $\Sigma_{Z}$  and  $c$  is a constant. The solution to this problem is a diagonal matrix with equal diagonal elements.
Proof. The determinant of a matrix is the product of its eigenvalues, so the objective function  $\log \det(\Sigma_Z)$  can be rewritten as  $\sum_{i=1}^{N} \log(\lambda_i(\Sigma_Z))$ . Our problem is then to maximize this sum under the constraints that the sum of the eigenvalues does not exceed  $c$  and that  $\Sigma_Z$  is positive semi-definite.
Applying Jensen's inequality to the concave function  $\log (x)$  with weights  $1 / N$ , we find that  $\frac{1}{N}\sum_{i = 1}^{N}\log (\lambda_i(\Sigma_Z))\leq \log (\frac{1}{N}\sum_{i = 1}^{N}\lambda_i(\Sigma_Z))$ . Equality holds if and only if all  $\lambda_{i}(\Sigma_{Z})$  are equal.
Setting  $\lambda_{i}(\Sigma_{Z}) = x$  for all  $i$ , we see that the constraint  $\sum_{i=1}^{N} \lambda_{i}(\Sigma_{Z}) \leq c$  becomes  $Nx \leq c$ , leading to the optimal eigenvalue  $x = c / N$  under the constraint.
Since  $\Sigma_Z$  is positive semi-definite, it can be diagonalized via an orthogonal transformation without changing the sum of its eigenvalues or its determinant. Therefore, the solution to the problem is a diagonal matrix with all diagonal entries equal to  $c / N$ .
Theorem J.5. Given a Gaussian mixture model where each component  $Z|X_{i}$  has covariance matrix  $\Sigma(X_{i})$ , under the assumptions above, the solution to the optimization problem
$$
m a x i m i z e \sum_ {i} \log \frac {| \Sigma_ {Z} |}{| \Sigma (X _ {i}) |}
$$
is a diagonal matrix  $\Sigma_Z$  with equal diagonal elements.
Proof. The objective function can be decomposed as follows:
$$
\begin{array}{l} \sum_ {i} \log \frac {| \Sigma_ {Z} |}{| \Sigma (X _ {i}) |} = \sum_ {i} \left(\log | \Sigma_ {Z} | - \log | \Sigma (X _ {i}) |\right) \\ = K \log | \Sigma_ {Z} | - \sum_ {i} \log | \Sigma_ {(} X _ {i}) |, \\ \end{array}
$$
where  $K$  is the number of components in the Gaussian mixture model.
In this optimization problem, we are optimizing over  $\Sigma_Z$ . The term  $\sum_{i}\log |\Sigma (X_{i})|$  is constant with respect to  $\Sigma_Z$ , therefore we can focus on maximizing  $K\log |\Sigma_Z|$ .
As the determinant of a matrix is the product of its eigenvalues,  $\log |\Sigma_Z|$  is the sum of the logs of the eigenvalues of  $\Sigma_Z$ . Thus, maximizing  $\log |\Sigma_Z|$  corresponds to maximizing the sum of the logarithms of the eigenvalues of  $\Sigma_Z$ .
According to Lemma 1.4, when we have a constraint on the sum of the eigenvalues, the solution to the problem of maximizing the sum of the logarithms of the eigenvalues of a positive semidefinite matrix  $\Sigma_Z$  is a diagonal matrix with equal diagonal elements.
From Lemma 1.3, we know that the sum of the eigenvalues of  $\Sigma_Z$  is bounded by  $(b + M) \times K$ . Therefore, when we maximize  $K \log |\Sigma_Z|$  under these constraints, the solution will be a diagonal matrix with equal diagonal elements. This completes the proof of the theorem.
Appendix K Entropy Comparison - Experimental Details
We use ResNet-18 [32] as our backbone. Each model is trained with 512 batch size for 800 epochs. We use the SGD optimizer with a momentum of 0.9 and a weight decay of  $1e^{-4}$ . The initial learning rate is 0.5. This learning rate follows the cosine decay with a linear warmup schedule. For augmentation, two augmented versions of each input image are generated. During this process, each image is cropped with random size, and resized to the original resolution, followed by random applications of horizontal mirroring, color jittering, grayscale conversion, Gaussian blurring and solarization. For the entropy estimation, we use the same method as in [38], which uses a lower bound of the entropy using the distances of the representations under the assumption of a mixture of the Gaussians around the representations with constant variance.
$$
H (Z) := H (Z | C) - \sum_ {i} c _ {i} \ln \sum_ {j} c _ {j} e ^ {- D \left(p _ {i} \mid \mid p _ {j}\right)}. \tag {54}
$$
Figure 5: Our generalization bound predicts more accurately the generalization gap in the loss. (left) Our SSL VICReg generalization bound outperforms state-of-the-art supervised generalization bounds. (right) Strong correlation between the generalization gap and our generalization bound for VICReg. Pearson correlation - 0.9633. Conducted on CIFAR-10.
where  $p_i$  and  $p_j$  are the distributions of the representation of the i-th and j-th examples, and  $D(p_i||p_j)$  represents the divergence between these distributions. Also,  $c_i$  indicates the weight of component  $i$  ( $c_i \geq 0$ ,  $\sum_i c_i = 1$ ), and  $C$  is a discrete random variable where  $P(C = i) = c_i$ .
Appendix L Reproducibility Statement
All of the methods in our study are based on existing methods and their open-source implementations. We provide a detailed implementation setup for both the pre-training and downstream experiments. Additionally, we are committed to ensuring reproducibility and open science. Therefore, after publication, we will provide pretrained checkpoints and make the code openly available on a public repository. This will enable other researchers to reproduce our results and build upon our work.
Appendix M Expiemtns on the generalization bound
In this section we have more empirical results on the connection between our generalization to the generalization gap.
Appendix N Limitations of assuming inherent network randomness
First, we provide a brief overview of the challenges associated with estimating mutual information (MI) in DNNs. This context will set the stage for a detailed discussion on the advantages of our method
Estimating MI in DNNs poses various practical challenges. For example, deterministic DNNs with strict monotonic nonlinearities yield infinite or constant MI, complicating optimization [4] or making optimization ill-posed. One solution is to assume an unknown distribution over the representation and estimating MI directly [9, 57]. However, this is challenging theoretically and requires large sample sizes [55, 47] and is prone to significant estimation errors [70].
To address these issues, prior works have proposed various assumptions about randomness in neural networks. For example, [1] posits noise in DNNs' parameters, while [1] assumes noise in the representations. These assumptions are valid if they outperform deterministic DNNs or yield similar representations to those trained in practice.
Our empirical evidence suggests that noise in the input has much better performance and representations similar to the deterministic network compared to noise in the network itself. We compared our model, which introduces noise at the input level, with models that add noise to network representations [26], using CIFAR-100 and Tiny-ImageNet and VICReg with ConvNeXt. The results,
tabulated below, indicate that the noise injected into the representation has much worse performance degradation compared to our input noise model.
Table 2: Comparison of Noisy Network and Noisy Input across different datasets for different noise levels
β (Noise Level)	Tiny-ImageNet	CIFAR100
Noisy Network	Noisy Input (ours)	Noisy Network	Noisy Input (ours)
β = 0 (no noise)	53.1	53.1	70.1	70.1
β = 0.05	51.7	53.0	69.7	70.0
β = 0.1	50.2	52.8	68.8	69.6
β = 0.2	48.1	52.3	67.1	68.9
Our approach, focusing on input noise, does not necessitate explicit noise injection during training—unlike methods assuming inherent network noise.
Lastly, we validate that the assumption of input noise leads to better alignment with deterministic networks. Following [14], we used Centered Kernel Alignment (CKA) [18] to compare deterministic DNNs with our noise model (input noise) to DNNs with noise in the representations [26]. The results, also tabulated below, confirm that assuming input noise aligns more closely with deterministic DNN representations than assuming noise in the DNN parameters.
Table 3: Performance comparison of different methods at varying noise levels
Noise Level/Method	Deterministic Network	Noisy Network	Noisy Input (our method)
β = 0.05	0.97	0.82	0.93
β = 0.1	0.97	0.69	0.85
β = 0.2	0.97	0.54	0.77
β = 0.3	0.97	0.32	0.69
In summary, our input noise model more effectively captures deterministic DNN behavior and is more robust than assuming inherent network randomness.
Appendix O Boader Impact
In the landscape of machine learning, SSL algorithms have found a broad array of practical applications, ranging from image recognition to natural language processing. With the proliferation of data in recent years, their utility has grown exponentially, often serving as key components in numerous real-world use cases. This paper introduces a new family of SSL algorithms that have demonstrated superior performance. Given the widespread use of SSL algorithms, the advancements proposed in this paper have the potential to considerably enhance the effectiveness of systems that rely on these techniques. As a result, the impact of these improved algorithms could be far-reaching, potentially influencing a multitude of applications and sectors where machine learning is currently employed. With this potential for value also comes the potential that proposed methods make false promises that will not benefit real-world practitioners and may, in fact, cause harm when deployed in sensitive applications. For this reason, we release our numerical results and implementation details for the sake of transparency and reproducibility. As with all new state-of-the-art methods, our improvements may also improve models used for malicious intentions.