A SCALING LAW FOR SYN2REAL TRANSFER: HOW MUCH IS YOUR PRE-TRAINING EFFECTIVE?
Year: 2021

Abstract:
Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.

text: Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.

heading: 1 INTRODUCTION
text: The success of deep learning relies on the availability of large data. If the target task provides limited data, the framework of transfer learning is preferably employed. A typical scenario of transfer learning is to pre-train a model for a similar or even different task and fine-tune the model for the target task. However, the limitation of labeled data has been the main bottleneck of supervised pre-training. While there have been significant advances in the representation capability of the models and computational capabilities of the hardware, the size and the diversity of the baseline dataset have not been growing as fast (Sun et al., 2017). This is partially because of the sheer physical difficulty of collecting large datasets from real environments (e.g., the cost of human annotation).
In computer vision, synthetic-to-real (syn2real) transfer is a promising strategy that has been attracting attention (Su et al., 2015; Movshovitz-Attias et al., 2016; Georgakis et al., 2017; Tremblay et al., 2018; Hinterstoisser et al., 2019; Borrego et al., 2018; Chen et al., 2021). In syn2real, images used for pre-training are synthesized to improve the performance on real vision tasks. By combining various conditions, such as 3D models, textures, light conditions, and camera poses, we can synthesize an infinite number of images with ground-truth annotations. Syn2real transfer has already been applied in some real-world applications. Teed & Deng (2021) proposed a simultaneous localization and mapping (SLAM) system that was trained only with synthetic data and demonstrated state-of-the-art performance. The object detection networks for autonomous driving developed by Tesla was trained with 370 million images generated by simulation (Karpathy, 2021).
The performance of syn2real transfer depends on the similarity between synthetic and real data. In general, the more similar they are, the stronger the effect of pre-training will be. On the contrary, if there is a significant gap, increasing the number of synthetic data may be completely useless, in which case we waste time and computational resources. A distinctive feature of syn2real is that we can control the process of generating data by ourselves. If a considerable gap exists, we can try to regenerate the data with a different setting. But how do we know that? More specifically, in a standard learning setting without transfer, a “power law”-like relationship called a scaling law often holds between data size and generalization errors (Rosenfeld et al., 2019; Kaplan et al., 2020). Is there such a rule for pre-training?
In this study, we find that the generalization error on fine-tuning is explained by a simple scaling law,
test error ' Dn−α + C, (1)
where coefficient D > 0 and exponent α > 0 describe the convergence speed of pre-training, and constant C ≥ 0 determines the lower limit of the error. We refer to α as pre-training rate and C as transfer gap. We can predict how large the pre-training data should be to achieve the desired accuracy by estimating the parameters α,C from the empirical results. Additionally, we analyze the dynamics of transfer learning using the recent theoretical results based on the neural tangent kernel (Nitanda & Suzuki, 2021) and confirm that the above law agrees with the theoretical analysis. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.
Our contributions are summarized as follows.
• From empirical results and theoretical analysis, we elicit a law that describes how generalization scales in terms of data sizes on pre-training and fine-tuning.
• We confirm that the derived law explains the empirical results for various settings in terms of pre-training/fine-tuning tasks, model size, and data complexity (e.g., Figure 1). Furthermore, we demonstrate that we can use the estimated parameters in our scaling law to assess how much improvement we can expect from the pre-training procedure based on synthetic data.
• We theoretically derive a generalization bound for a general transfer learning setting and confirm its agreement with our empirical findings.

heading: 2 RELATED WORK
text: Supervised pre-training for visual tasks Many empirical studies show that the performance at a fine-tuning task scales with pre-training data (and model) size. For example, Huh et al. (2016) studied the scaling behavior on ImageNet pre-trained models. Beyond ImageNet, Sun et al. (2017) studied the effect of pre-training with pseudo-labeled large-scale data and found a logarithmic scaling behavior. Similar results were observed by Kolesnikov et al. (2019).
Syn2real transfer The utility of synthetic images as supervised data for computer vision tasks has been continuously studied by many researchers (Su et al., 2015; Movshovitz-Attias et al., 2016; Georgakis et al., 2017; Tremblay et al., 2018; Hinterstoisser et al., 2019; Borrego et al., 2018; Chen et al., 2021; Newell & Deng, 2020; Devaranjan et al., 2020; Mousavi et al., 2020; Hodaň et al., 2019). These studies found positive evidence that using synthetic images is helpful to the fine-tuning task. In addition, they demonstrated how data complexity, induced by e.g., light randomization, affects the final performance. For example, Newell & Deng (2020) investigated how the recent self-supervised methods perform well as a pre-training task to improve the performance of downstream tasks. In this paper, following this line of research, we quantify the effects under the lens of the scaling law (1).
Neural scaling laws The scaling behavior of generalization error, including some theoretical works (e.g., Amari et al., 1992), has been studied extensively. For modern neural networks, Hestness et al. (2017) empirically observed the power-law behavior of generalization for language, image, and speech domains with respect to the training size. Rosenfeld et al. (2019) constructed a predictive form for the power-law in terms of data and model sizes. Kaplan et al. (2020) pushed forward this direction in the language domain, describing that the generalization of transformers obeys the power law in terms of a compute budget in addition to data and model sizes. Since then, similar scaling laws have been discovered in other data domains (Henighan et al., 2020). Several authors have also attempted theoretical analysis. Hutter (2021) analyzed a simple class of models that exhibits a power-law n−β in terms of data size n with arbitrary β > 0. Bahri et al. (2021) addressed power laws under four regimes for model and data size. Note that these theoretical studies, unlike ours, are concerned with scaling laws in a non-transfer setting.
Hernandez et al. (2021) studied the scaling laws for general transfer learning, which is the most relevant to this study. A key difference is that they focused on fine-tuning data size as a scaling factor, while we focus on pre-training data size. Further, they found scaling laws in terms of the transferred effective data, which is converted data amount necessary to achieve the same performance gain by pre-training. In contrast, Eq. (1) explains the test error with respect to the pre-training data size directly at a fine-tuning task. Other differences include task domains (language vs. vision) and architectures (transformer vs. CNN).
Theory of transfer learning Theoretical analysis of transfer learning has been dated back to decades ago (Baxter, 2000) and has been pursued extensively. Among others, some recent studies (Maurer et al., 2016; Du et al., 2020; Tripuraneni et al., 2020) derived an error bound of a fine-tuning task in the multi-task scenario based on complexity analysis; the bound takes an additive form O(An−1/2 +Bs−1/2), where n and s are the data size of pre-training and fine-tuning, respectively, with coefficients A and B. Neural network regression has been also discussed with this bound (Tripuraneni et al., 2020). In the field of domain adaptation, error bounds have been derived in relation to the mismatch between source and target input distributions (Ganin et al., 2016; Acuna et al., 2021). They also proposed algorithms to adopt a new data domain. However, unlike in this study, no specific learning dynamics has been taken into account. In the area of hypothesis transfer learning (Fei-Fei et al., 2006; Yang et al., 2007), among many theoretical works, Du et al. (2017) has derived a risk bound for kernel ridge regression with transfer realized as the weights on the training samples. The obtained bound takes a similar form to our scaling law. However, the learning dynamics of neural networks initialized with a pre-trained model has never been explored in this context.

heading: 3 SCALING LAWS FOR PRE-TRAINING AND FINE-TUNING
text: The main obstacle in analyzing the test error is that we have to consider interplay between the effects of pre-training and fine-tuning. Let L(n, s) ≥ 0 be the test error of a fine-tuning task with pre-training data size n and fine-tuning data size s. As the simplest case, consider a fine-tuning task without pre-training (n = 0), which boils the transfer learning down to a standard learning setting. In this case, the prior studies of both classical learning theory and neural scaling laws tell us that the test error decreases polynomially1 with the fine-tuning data size s, that is, L(0, s) = Bs−β + E with decay rate β > 0 and irreducible loss E ≥ 0. The irreducible loss E is the inevitable error given by
1For classification with strong low-noise condition, it is known that the decay rate can be exponential (Nitanda & Suzuki, 2019). However, we focus only on the polynomial decay without such strong condition in this paper.
the best possible mapping; it is caused by noise in continuous outputs or labels. Hereafter we assume E = 0 for brevity.
3.1 INDUCTION OF SCALING LAW WITH SMALL EMPIRICAL RESULTS
To speculate a scaling law, we conducted preliminary experiments.2 We pre-trained ResNet-50 by a synthetic classification task and fine-tuned by ImageNet. Figure 2 (a) presents the log-log plot of error curves with respect to pre-training data size n, where each shape and color indicates a different fine-tuning size s. It shows that the pre-training effect diminishes for large n. In contrast, Figure 2 (b) presents the relations between the error and the fine-tuning size s with different n. It indicates the error
drops straight down regardless of n, confirming the power-law scaling with respect to s. The above observations and the fact that L(0, s) decays polynomially are summarized as follows. Requirement 1. lims→∞ L(n, s) = 0. Requirement 2. limn→∞ L(n, s) = const. Requirement 3. L(0, s) = Bs−β .
Requirements 1 and 3 suggest the dependency of n is embedded in the coefficient B = g(n), i.e., the pre-training and fine-tuning effects interact multiplicatively. To satisfy Requirement 2, a reasonable choice for the pre-training effect is g(n) = n−α + γ; the error decays polynomially with respect to n but has a plateau at γ. By combining these, we obtain
L(n, s) = δ(γ + n−α)s−β , (2)
where α, β > 0 are decay rates for pre-training and fine-tuning, respectively, γ ≥ 0 is a constant, and δ > 0 is a coefficient. The exponent β determines the convergence rate with respect to fine-tuning data size. From this viewpoint, δ(γ + n−α) is the coefficient factor to the power law. The influence of the pre-training appears in this coefficient, where the constant term δγ comes from the irreducible loss of the pre-training task and n−α expresses the effect of pre-training data size. The theoretical consideration in Section E.5 suggests that the rates α and β can depend on both the target functions of pre-training and fine-tuning as well as the learning rate.

heading: 3.2 THEORETICAL DEDUCTION OF SCALING LAW
text: Next, we analyze the fine-tuning error from a purely theoretical point of view. To incorporate the effect of pre-training that is given as an initialization, we need to analyze the test error during the training with a given learning algorithm such as SGD. We apply the recent development by Nitanda & Suzuki (2021) to transfer learning. The study successfully analyzes the generalization of neural networks in the dynamics of learning, showing it achieves minmax optimum rate. The analysis uses the framework of the reproducing kernel Hilbert space given by the neural tangent kernel (Jacot et al., 2018).
For theoretical analysis of transfer, it is important to formulate a task similarity between pre-training and fine-tuning. If the tasks were totally irrelevant (e.g., learning MNIST to forecast tomorrow’s weather), pre-training would have no benefit. Following Nitanda & Suzuki (2021), for simplicity of analysis, we discuss only a regression problem with square loss. We assume that a vector input x and scalar output y follow y = φ0(x) for pre-training and y = φ0(x) + φ1(x) for fine-tuning, where we omit the output noise for brevity; the task types are identical sharing the same input-output form, and task similarity is controlled by φ1.
We analyze the situation where the effect of pre-training remains in the fine-tuning even for large data size (s→∞). More specifically, the theoretical analysis assumes a regularization term as the
2The results are replicated from Appendix C.2; see the subsection for more details.
`2-distance between the weights and the initial values, and a smaller learning rate than constant in the fine-tuning. Hence we control how the pre-training effect is preserved through the regularization and learning rate. Other assumptions made for theoretical analysis concern the model and learning algorithm; a two-layer neural network having M hidden units with continuous nonlinear activation3 is adopted; for optimization, the averaged SGD (Polyak & Juditsky, 1992), an online algorithm, is used for a technical reason.
The following is an informal statement of the theoretical result. See Appendix E for details. We emphasize that our result holds not only for syn2real transfer but also for transfer learning in general.
Theorem 1 (Informal). Let f̂n,s(x) be a model of width M pre-trained by n samples (x1, y1), . . . , (xn, yn) and fine-tuned by s samples (x′1, y ′ 1), . . . , (x ′ s, y ′ s) where inputs x, x
′ ∼ p(x) are i.i.d. with the input distribution p(x) and y = φ0(x) and y′ = ϕ(x′) = φ0(x′) + φ1(x′). Then the generalization error of the squared loss L(n, s) = |f̂n,s(x)− ϕ(x)|2 is bounded from above with high probability as ExL(n, s) ≤ A1(cM +A0n−α)s−β + εM . (3) εM and cM can be arbitrary small for large M ; A0 and A1 are constants; the exponents α and β depend on φ0, φ1, p(x), and the learning rate of fine-tuning.
The above bound (3) shows the correspondence with the empirical derivation of the full scaling law (2). Note that the approximation error εM is omitted in (2).
We note that the derived bound takes a multiplicative form in terms of the pre-training and fine-tuning effects, which contrasts with the additive bounds such as An−1/2 +Bs−1/2 (Tripuraneni et al., 2020). The existing studies consider the situation where a part of a network (e.g., backbone) is frozen during fine-tuning. Therefore, the error of pre-training is completely preserved after fine-tuning, and both errors appear in an additive way. This means that the effect of pre-training is irreducible by the effect of fine-tuning, and vice versa. In contrast, our analysis deals with the case of re-optimizing the entire network in fine-tuning. In that case, the pre-trained model is used as initial values. As a result, even if the error in pre-training is large, the final error can be reduced to zero by increasing the amount of fine-tuning data.

heading: 3.3 INSIGHTS AND PRACTICAL VALUES
text: The form of the full scaling law (2) suggests that there are two scenarios depending on whether fine-tuning data is big or small. In “big fine-tune” regime, pre-training contributes relatively little. By taking logarithm, we can separate the full scaling law (2) into the pre-training part u(n) = log(n−α + γ) and the fine-tuning part v(s) = −β log s. Consider to increase n by squaring it. Since the pre-training part cannot be reduced below log(γ) as u(n) > u(n2) > log(γ), the relative improvement (u(n2) − u(n))/v(s) becomes infinitesimal for large s. Figure 2 (b) confirms this situation. Indeed, prior studies provide the same conclusion that the gain from pre-training can easily vanish (He et al., 2018; Newell & Deng, 2020) or a target task accuracy even degrade (Zoph et al., 2020) if we have large enough fine-tuning data.
The above observation, however, does not mean pretraining is futile. Dense prediction tasks such as depth estimation require pixel-level annotations, which critically limits the number of labeled data. Pre-training is indispensable in such “small fine-tune” regime. Based on this, we hereafter analyze the case where the fine-tuning size s is fixed. By eliminating s-dependent terms in (2), we obtain a simplified law (1) by settingD = δs−β and C = δγs−β . After several evaluations, these parameters including α can be estimated by the nonlinear least squares method (see also Section 4.1).
As a practical benefit, the estimated parameters of the simplified law (1) bring a way to assess syn2real transfer.
Suppose we want to solve a classification task that requires at least 90% accuracy with limited labels.
3ReLU is not included in this class, but we can generalize this condition; see (Nitanda & Suzuki, 2021).
We generate some number of synthetic images and pre-train with them, and we obtain 70% accuracy as Figure 3 (a). How can we achieve the required accuracy? It depends on the parameters of the scaling law. The best scenario is (b) — transfer gap C is low and pre-training rate α is high. In this case, increasing synthetic images eventually leads the required accuracy. In contrast, when transfer gap C is larger than the required accuracy (c), increasing synthetic images does not help to solve the problem. Similarly, for low pre-training rate α (d), we may have to generate tremendous amount of synthetic images that are computationally infeasible. In the last two cases, we have to change the rendering settings such as 3D models and light conditions to improve C and/or α, rather than increasing the data size. The estimation of α and C requires to compute multiple fine-tuning processes. However, the estimated parameters tell us whether we should increase data or change the data generation process, which can reduce the total number of trials and errors.

heading: 4 EXPERIMENTS

heading: 4.1 SETTINGS
text: For experiments, we employed the following transfer learning protocol. First, we pre-train a model that consists of backbone and head networks from random initialization until convergence, and we select the best model in terms of the validation error of the pre-training task. Then, we extract the backbone and add a new head to fine-tune all the model parameters. For notations, the task names of object detection, semantic segmentation, multi-label classification, single-label classification, and surface normal estimation are abbreviated as objdet, semseg, mulclass, sinclass, and normal, respectively. The settings for transfer learning are denoted by arrows. For example, objdet→semseg indicates that a model is pre-trained by object detection, and fine-tuned by semantic segmentation. The experiments were conducted on an in-house cluster containing NVIDIA V100 GPUs. The total amount of computation was approximately 1700 GPU days (200 for image rendering, 1300 for pre-training, and 200 for fine-tuning). All the results including Figure 1 are shown as log-log plots.
Pre-training: We prepared four tasks: mulclass, objdet, semseg, and normal. We used ResNet-based models, where backbones were ResNet-50, unless otherwise specified, and the head networks were customized for each task. Synthetic images for pre-training were generated by BlenderProc (Denninger et al., 2019), an image renderer that can handle several domain randomization methods. For rendering, we used the setting of the BOP challenge 2020 (Hodaň et al., 2020) as our default setting. We used 172 3D models, where ten objects appeared on average for each image. We applied texture randomization for walls and a floor, randomization for area and point lights, and randomization for the camera. In most cases, the models were pre-trained with 64,000 images. We trained all models for the same fixed number of iterations depending on pre-training tasks and selected the best models for fine-tuning, which were validated by another 1000 synthetic images generated in the same way.
Fine-tuning: We evaluated sinclass by ImageNet (Russakovsky et al., 2015), objdet by MS-COCO (Lin et al., 2014), and semseg by ADE20K (Zhou et al., 2016). The number of images used was 1% of each data set (roughly, 12,000 for ImageNet, 1000 for COCO, and 200 for ADE20K). We fine-tuned the pre-trained models with these subsets of data for a fixed number of iterations and reported the error metrics for validation sets at the last iteration. The metrics were top-1 accuracy for classification, mean mAP for MS-COCO, and mean IoU for ADE20K. These metrics take their values from 0 to 1, and we converted them into errors such as 1 - accuracy.4
Curve fitting: After obtaining the empirical errors L̂, we estimated the parameters of (1) by nonlinear least squares in the log-log space. We solved the minimization problem of ∑ i | log L̂(ni, s)− log(Dn−αi + C)|2 with a fixed fine-tuning data size s and pre-training data sizes ni = 2i × 1000 for data point index i = 0, . . . , 6. In the experiments, we empirically encountered some instability between D and α. We fixed D = 0.48 by the median values of D’s for all the settings and estimated α and C independently for each case. We explain this procedure with more details in Appendix D.
4Although the cross-entropy loss is commonly used, several studies (Sharma & Kaplan, 2020; Bahri et al., 2021) show that the scaling laws also hold for 1 - accuracy.

heading: 4.2 SCALING LAW UNIVERSALLY EXPLAINS DOWNSTREAM PERFORMANCE FOR VARIOUS TASK COMBINATIONS
text: Figure 1 shows the test errors of each fine-tuning task and fitted learning curves with Eq. (1), which describes the effect of pre-training data size n for all combinations of pre-training and fine-tuning tasks. The scaling law fits with the empirical fine-tuning test errors with high accuracy in most cases.
4.3 BIGGER MODELS REDUCE THE TRANSFER GAP
We compared several ResNet models as backbones in mulclass→sinclass and objdet→objdet to observe the effects of model size. Figure 4 (left) shows the curves of scaling laws for the pre-training data size n for different sizes of backbone ResNet-x, where x ∈ {18, 34, 50, 101, 152}. The bigger models attain smaller test errors. Figure 4 (right) shows the values of the estimated transfer gap C. The results suggest that there is a roughly power-law relationship between the transfer gap and model size. This agrees with the scaling law with respect to the model size shown by Hernandez et al. (2021).
4.4 SCALING LAW CAN EXTRAPOLATE FOR MORE PRE-TRAINING IMAGES
We also evaluated the extrapolation ability of the scaling law. We increased the number of synthetic images from the original size (n = 64,000) to 1.28 million, and see how the fitted scaling law predicts the unseen test errors where n > 64,000. As a baseline, we compared the power-law model, which is equivalent to the derived scaling law (1) with C = 0. Figure 5 (left) shows the extrapolation results in objdet→objdet setting, which indicates the scaling law follows the saturating trend in regions with large pre-training sizes for all models, while the power-law model fails to capture it. The prediction errors is numerically shown in Figure 5 (right), which again shows our scaling law achieves better prediction performance.
4.5 DATA COMPLEXITY AFFECTS BOTH PRE-TRAINING RATE AND TRANSFER GAP
We examined how the complexity of synthetic images affects fine-tuning performance. We controlled the following four rendering parameters: Appearance: Number of objects in each image; single or multiple (max 10 objects). Light: Either an area and point light is randomized or fixed in terms of height, color, and intensity. Background: Either the textures of floor/wall are randomized or fixed. Object texture: Either the 3D objects used for rendering contain texture (w/) or not (w/o). Indeed, the data complexity satisfies the following ordered relationships: single < multiple in appearance, fix < random in light and background, and w/o < w/ in object texture5. To quantify the complexity, we computed the negative entropy of the Gaussian distribution fitted to the last activation values of the backbone network. For this purpose, we pre-trained ResNet-50 as a backbone with MS-COCO for 48 epochs and computed the empirical covariance of the last activations for all the synthetic data sets.
The estimated parameters are shown in Figure 6, which indicates the following (we discuss the implications of these results further in Section 5.1).
• Data complexity controlled by the rendering settings correlates with the negative entropy, implying the negative entropy expresses the actual complexity of pre-training data.
• Pre-training rate α correlates with data complexity. The larger complexity causes slower rates of convergence with respect to the pre-training data size.
• Transfer gap C mostly correlates negatively with data complexity, but not for object texture.
As discussed in Section 4.1, we have fixed the value of D to avoid numerical instability, which might cause some bias to the estimates of α. We postulate, however, the value of D depends mainly on the fine-tuning task and thus has a fixed value for different pre-training data complexities. This can be inferred from the theoretical analysis in Appendix E.5: the exponent β in the main factor s−β of D does not depend on the pre-training data distribution but only on the fine-tuning task or the pre-training true mapping. Thus, the values of D should be similar over the different complexities, and the correlation of α preserves.
5The object category of w/o is a subset of w/, and w/ has a strictly higher complexity than w/o.

heading: 5 CONCLUSION AND DISCUSSION
text: In this paper, we studied how the performance on syn2real transfer depends on pre-training and fine-tuning data sizes. Based on the experimental results, we found a scaling law (1) and its generalization (2) that explain the scaling behavior in various settings in terms of pre-training/finetuning tasks, model sizes, and data complexities. Further, we present the theoretical error bound for transfer learning and found our theoretical bound has a good agreement with the scaling law.

heading: 5.1 IMPLICATION OF COMPLEXITY RESULTS IN SECTION 4.5
text: The results of Section 4.5 has two implications. First, data complexity (i.e., the diversity of images) largely affects the pre-training rate α. This is reasonable because if we want a network to recognize more diverse images, we need to train it with more examples. Indeed, prior studies (Sharma & Kaplan, 2020; Bahri et al., 2021) observed that α is inversely proportional to the intrinsic dimension of the data (e.g., dimension of the data manifold), which is an equivalent concept of data complexity.
Second, the estimated values of the transfer gap C suggest that increasing the complexity of data is generally beneficial to decrease C, but not always. Figure 6 (right) shows that increasing complexities in terms of appearance, light, and background reduces the transfer gap, which implies that these rendering operations are most effective to cover the fine-tuning task that uses real images. However, the additional complexity in object texture works negatively. We suspect that this occurred because of shortcut learning (Geirhos et al., 2020). Namely, adding textures to objects makes the recognition problem falsely easier because we can identify objects by textures rather than shapes. Because CNNs prefer to recognize objects by textures (Geirhos et al., 2018; Hermann et al., 2019), the pre-trained models may overfit to learn the texture features. Without object textures, pre-trained models have to learn the shape features because there is no other clue to distinguish the objects, and the learned features will be useful for real tasks.

heading: 5.2 LESSONS TO TRANSFER LEARNING AND SYNTHETIC-TO-REAL GENERALIZATION
text: Our results suggest the transfer gap C is the most crucial factor for successful transfer learning because C determines the maximum utility of pre-training. Large-scale pre-training data can be useless when C is large. In contrast, if C is negligibly small, the law is reduced essentially to n−α, which tells that the volume of pre-training data is directly exchanged to the performance of fine-tuning tasks. Our empirical results suggest two strategies for reducing C: 1) Use bigger models and 2) fill the domain gap in terms of the decision rule and image distribution. For the latter, existing techniques such as domain randomization (Tobin et al., 2017) would be helpful.

heading: 5.3 LIMITATIONS OF THIS STUDY
text: • In the experiments, the scale of data is relatively limited (million-scale, not billion). • We only examined ResNet as a network architecture (no Transformers). • Although there are various visual tasks, our study only covers a few of them. Extending our
observations to other visual tasks such as depth estimation, instance segmentation, and keypoint detection, as well as to other data domains such as language is future work.
• The theoretical results assume several conditions that may contradict the actual setting in the experiments. For example, our theory relies on ASGD instead of vanilla SGD. Also, the task types are assumed to be identical for the pre-training and fine-tuning tasks.
• In this study, we focus on finding a general rule of transfer learning, rather than improving absolute performance on specific tasks. We used popular vision tasks such as classification and ready-made rendering settings that is not designed to pre-train for the tasks. We expect to observe more performance gain with other syn2real-friendly tasks such as optical flow and elaborate rendering settings in future work.

heading: A TRAINING DETAILS

heading: A.1 OBJECT DETECTION
text: We used Faster-RCNN (Ren et al., 2016) with FPN (Lin et al., 2017) as object detection models and ResNet (Goyal et al., 2017) as a backbone network of Faster-RCNN.
We used the following training procedure: We trained the model using momentum SGD of momentum 0.9 with weight decay of 10−4. The global batch size was set to 64 when training ResNet18, ResNet34, ResNet50, and ResNet101. The batch size was set to 32 when training ResNet152 to avoid out-ofmemory errors. The batch statistics in batch normalization layers were computed across all GPUs. We used a base image size of 640× 640 in the same way as YOLACT training (Bolya et al., 2019). We used mixed16 training to reduce the memory footprint. We also adopted random horizontal flipping as data augmentation to images. The learning rate was set to 0.02, and we used the cosine decay with a warmup scheme. The warmup length is 120,000 images (3,750 iterations for ResNet152 and 1,875 iterations for other models). As for evaluation, we followed the standard settings in COCO dataset (Ren et al., 2016).
We pre-trained the model with 14,400,000 images (450,000 iterations for ResNet152 and 225,000 iterations for other models). We used the models that achieved the best mmAP as the initial value of fine-tuning.
We used COCO (Lin et al., 2014) as the fine-tuning dataset. We trained the model with 1,440,000 images (45,000 iterations for ResNet152 and 22,500 iterations for other models) during fine-tuning.

heading: A.2 SEMANTIC SEGMENTATION
text: We used DeepLabV3 (Chen et al., 2017) with the softmax cross-entropy loss as the semantic segmentation model and ResNet50 (Goyal et al., 2017) as its backbone. The model configuration follows the implementation in torchvision6. It should be noted that DeepLabV3 requires dilated ResNet as the backbone, which is not the case in object detection and classification tasks. Even though, the shapes of weight tensors of dilated ResNet50 exactly match those of non-dilated ResNet50; thus we can use the pre-trained weights of dilated and non-dilated ResNet50 interchangeably.
The learning procedure is based on the reference implementation7 of torchvision. We added an auxiliary branch based on FCN (Long et al., 2015) which takes conv4 of the backbone as the input. In the computation of loss function, the loss for the auxiliary branch is computed in the same way as for the main branch and is added to the overall loss after multiplying by the factor 0.5. The model was trained using momentum SGD of momentum 0.9 with weight decay of 10−4. The global batch size was set to 32. The batch statistics in batch normalization layers were computed across all GPUs. During training, images were first resized so that the length of the shorter edge becomes an integer uniformly chosen from [520× 0.5, 520× 2], then horizontally flipped with probability 0.5, finally randomly cropped to 480× 480. The learning rate (LR) was decayed according to the polynomial LR schedule of rate 0.9 and initial LR of 0.02. For the parameters of the auxiliary classifier, the LR was multiplied by 10. The evaluation was performed once every 3,125 iterations (almost equivalent to 5 epochs in full ADE20K). In the evaluation, images were resized so that the length of the shorter edge becomes 520.
In pre-training, we trained 125,000 iterations which roughly equals 200 epochs in full ADE20K (Zhou et al., 2017). We used the model that achieved the best mIoU as the initial value of fine-tuning. We pre-trained models using our synthetic datasets. When training with them, backgrounds (points at which no foreground objects were present) were also considered to be a separate class in semantic segmentation.
6https://github.com/pytorch/vision 7https://github.com/pytorch/vision/tree/master/references/segmentation
We used the ADE20K (Zhou et al., 2017) datasets as the fine-tuning target. In fine-tuning, we trained the model for 18,750 iterations, which correspond to 30 epochs of full ADE20K. The metric was mIoU score.

heading: A.3 MULTI-LABEL CLASSIFICATION
text: We used ResNet (Goyal et al., 2017) with binary cross-entropy used as the loss function in multi-label classification. We used the following training procedure: We trained the model using momentum SGD of momentum 0.9 with weight decay of 10−4. The batch size was set as 32 per GPU, thus 256 in total. We trained the models for 112,500 iterations. The input size of the images was simply resized to 640× 640. We adopted random horizontal flipping as data augmentation to images. The learning rate was set to 0.1. The cosine decay with a warmup scheme was used. The warmup length was 120,000 images. The evaluation was performed once every 120,000 images. In the evaluation, the image size was the same as that used during training, and data augmentation was not used. We used the mAP score as the metric.

heading: A.4 SINGLE-LABEL CLASSIFICATION
text: As in multi-label classification, we used ResNet. The softmax cross-entropy was used as the loss function of the single-label classification. The learning procedure is based on (Goyal et al., 2017). However, we used cosine decay for the learning rate scheduling.

heading: A.5 SURFACE NORMAL ESTIMATION
text: As in semantic segmentation, we used DeepLabV3 (Chen et al., 2017) as the model for surface normal estimation. The model configuration and the training procedure were exactly the same as in semantic segmentation, except for the following changes:
• Dimension of output channels was changed to 3, each of which corresponds to the 3 axes of the normal vector,
• Initial LR was changed to 0.04, • Length of pre-training was 200,000 iterations, which corresponds to 100 epochs in our synthetic
dataset, • Random flipping was not performed during the data augmentation, and • Loss function was the average of the value, 1.0− n · n̂, which was computed for each valid pixel
where n is the ground-truth normal vector and n̂ is the model output (after L2 normalization).

heading: B SYNTHETIC DATA DETAILS
text: The data generation strategy is based on the “on surface sampling” setting in the BoP challenge dataset8. In this setting, the sampled objects will be spawned in a cube-shaped room with one point light and one surface light. As the objects to be spawn, we used all the BoP object sets, i.e., LM, T-LESS, ITODD, HB, YCB-V, RU-APC, IC-BIN, IC-MI, TUD-L, and TYO-L.9. There are 173 objects in total. After generated a random scene (position of objects, lights, etc), we took 10 pictures by 10 different camera poses. This means that, if we have 10K images in total, there are 1K unique scenes, and 9K images are inflated by just changing the camera angle and position.
To control the data complexity, we selected four attributes in the generation strategy and prepared two options for each.
Appearance controls how many objects are generated in a room in a single scene. For each scene, we randomly select ten objects for the multiple setting and one object for the single setting.
Light controls the light sources. In the random setting, the color, height, and strength of lights are randomized. In contrast, in the fix setting, they are all fixed.
8https://github.com/DLR-RM/BlenderProc/tree/main/examples/bop_challenge 9https://bop.felk.cvut.cz/datasets/
Background controls the texture of the room, i.e., floor and walls. In the random setting, we assign a random PBR material from the CC0 Textures10 library, and we selected one carpet texture for the fix setting.
Object texture controls the object set to be used. The BoP object set is consists of several types of object sets as described above. Among them, T-LESS and ITODD consist of industryrelevant objects and they do not have textures and colors.11 For the w/o setting, we only used such texture-less objects to be sampled. In contrast, we sample all the 173 objects include T-LESS and ITODD in the w/ setting.
We generated eight variations of datasets by changing these attributes, which were used in the experiments of Section 4.5. Figure 7 shows the example of generated images with the value of each attribute.
10https://ambientcg.com 11T-Less and ITODD contain 30 and 28 objects, respectively.

heading: C ADDITIONAL EXPERIMENTS

heading: C.1 ESTIMATED PARAMETERS IN THE CROSS-TASK SETTING
text: Figure 8 shows the estimated parameters (α,C) at the experiments described in Section 4.2. Note that the result of α at normal→semseg is omitted because its estimated value is highly unstable (the standard deviation is larger than 1).

heading: C.2 FULL SCALING LAW COLLECTIVELY RELATES PRE-TRAINING AND FINE-TUNING DATA SIZE
text: Next, we verify the validity of the full scaling law (2). In the mulclass→sinclass setting with ResNet-50, we changed the fine-tuning data size from 2% to 64% of the ImageNet.12 We then fitted all results by a single equation (2) to estimate the parameters except the irreducible loss E
12ImageNet contains a class imbalance problem. If we use 100% of the ImageNet, we cannot provide the same sample size per class. To eliminate the effect of class imbalance, we made the sampling ratio to keep the balance up to 64% and excluded the case of 100%.
(we assumed E = 0 from the preliminary results in Figure 2). The results in Figure 9 show that all empirical test errors are explained remarkably well by Eq. (2), which has only four parameters to fit in this case. The estimated parameters are α = 0.544, β = 0.322, γ = 0.478, and δ = 41.8.

heading: C.3 LINEARIZED RESULTS
text: The transfer gap C in (1) causes a plateau of the scaling law. Conversely, if we subtract the estimated C from the results, we must be able to recover the power-law scaling. To confirm this, we subtracted the estimated C from the empirical errors L̂ of the previous results. Figures 10–12 show the modified version of scaling law fittings. Overall, the empirical errors behave linearly along with the estimated power-law term Dn−α. Note that, in mulclass→semseg and normal→semseg, a few points of L̂ become negative after subtracting C, and these points are not depicted.
D EMPIRICAL DEPENDENCY BETWEEN α AND D
When C is non-zero, the joint estimation of D and α in (1) have an issue of numerical stability due to the small number of observations and noise, which can cause high dependence on each other. Figure 13 (left) shows the curves of (1) with C = 0.5, where the solid red curve is D = 0.5, α = 0.4 and the dashed blue line is D = 1, α = 0.5. We see that both curves are almost indistinguishable for
a large n. Figure 13 (right) shows the actual landscape in terms of α and D of the nonlinear leastsquares at objdet→objdet, the bright areas indicate the fitting loss is small. We see that there is a quadratic-like trajectory in the landscape, which implies the solutions are somehow redundant. Similar landscapes were observed for other tasks (Figure 14).
To avoid this issue, we fixed a common D for all the cases and estimated α for each. To determine D, we used the following procedure. First, we prepared two global parameters α̂, D̂ and set 0.5 as their initial values. Then, we fitted the curves by two equations, Dn−α̂ + C and D̂n−α + C, and estimated α and D. Next, we computed the median of α and substituted them into α̂. We did the same for D and D̂. After a few iterations, we got a converged value of D̂ = 0.48. In the experiments, we used the value for D and fixed it.

heading: E DETAILS OF THEORETICAL ANALYSIS
text: This section gives details of the theoretical discussions given in Section 3.2. For the analysis of learning and generalization bound, we use the techniques developed recently by Nitanda & Suzuki (2021). There are many works on the generalization of neural networks. To list a few, Neyshabur et al. (2015), Neyshabur et al. (2017), Bartlett et al. (2017), Wei & Ma (2020), and Suzuki (2018) analyze the generalization of neural networks based on complexity bounds. These generalization bounds, however, do not consider an algorithm of learning, such as stochastic gradient descent (SGD). Recently, learning dynamics of neural networks has been analyzed based on Neural Tangent Kernel (NTK) Jacot et al. (2018) and global convergence of wide neural networks has been revealed AllenZhu et al. (2018); Du et al. (2019). Based on the NTK framework, Arora et al. (2019) and Nitanda et al. (2020) showed a generalization bound of the gradient descent learning of neural networks. More recently, Nitanda & Suzuki (2021) focused the functional space given by NTK and showed that the two-layer neural network with averaged SGD achieves the minimax optimal rate with respect to the function class used in the standard theory of function estimation with kernels. We employ the method of Nitanda & Suzuki (2021), which is the most suitable for our analysis of transfer learning: it enables to examine the dependence on the initial parameter in the learning, and avoids the assumption of a positive margin of eigenvalues used in Arora et al. (2019) and Nitanda et al. (2020).

heading: E.1 PROBLEM SETTING
text: In the pre-training, the task is to learn the target function φ0 with T0 training data (x̃i, ỹi)T0i=1, where yi = φ0(xi), while in the fine-tuning phase, the network is initialized by the final parameter learned by the pre-training, and the whole parameter is updated in the training. We assume that the target function in the fine-tuning is given by
ϕ(x) = φ0(x) + φ1(x), (4)
and T1 training data is given by (xj , yj)T1j=1 with yj = ϕ(xj). In this setting, the goal of the finetuning phase will be to learn the additional function φ1 mainly. Note that, for simplicity of analysis, we assume noiseless training data, i.e., we assume the supervised signal yj is given by a deterministic function of xj , but extension to more general cases is not difficult as discussed in Nitanda & Suzuki (2021). In the analysis, the data are assumed to satisfy x ∈ Rd, ‖x‖2 = 1 and y ∈ [0, 1]. The distribution of the input data x is denoted by ρX , and the same for the pre-training and fine-tuning.
For tractable theoretical analysis, we consider a simple scalar-valued two-layer neural network model with M hidden units:
gΘ(x) = 1√ M M∑ r=1 arσ(b T r x). (5)
We omit the bias term, but with obvious modification, it is not difficult to include it (see Nitanda & Suzuki (2021)).
As in Nitanda & Suzuki (2021), we consider the averaged stochastic gradient descent (ASGD), where one training sample is given at every time step for the stochastic gradient descent as in online learning, and all the parameters in the time course are averaged after the final time step for the inference, that is, after proceeding up to prescribed T (T = T0 or T1) time steps, the parameter to be used in the inference is given by
Θ (T ) := 1
T + 1 T∑ t=0 Θ(t). (6)
The final network uses this averaged parameter, i.e., the final network is given by g Θ (T )(x).
The parameter is initialized as Θ(0) = (a(0)1 , b (0) 1 , . . . , a (0) M , b (0) M ). For the pre-trainig, each b (0) is independently given by the uniform distribution on the unit sphere. As in Nitanda & Suzuki (2021), a(0) are initilized as 1 or −1 so that gΘ(0) = 0. As explained before, the initial parameter of the fine-tuning is the same as the averaged parameter of the pre-training Θ (T0)
pre . The objective function to minimize for the pre-training and fine-tuning is given by the following regularized empirical risk:
L(Θ) := 1
2 ∑ i (yi − gΘ(xi))2 + λ 2 { ‖a− a(0)‖22 + ∑ r ‖br − b(0)r ‖22 } , (7)
where λ is the regularization coefficient, which is a hyperparameter. The values of λ in the pre-training and fine-tuning can be different, and denoted by λ0 and λ1, respectively. Note that the regularization in Eq. (7) is not the most common `2-regularization, where ‖Θ‖22 is used for the regularization. When applied in fine-tuning, however, the above regularization can be interpreted as elastic weight consolidation (Kirkpatrick et al., 2017), which prevents forgetting the pre-trained parameters.
We consider online learning, in which at every step t (t ≤ T − 1), one datum xi is sampled from ρX independently, and (xi, yi) (or (x̃i, ỹi) in pre-training) is used to update the parameter according to the gradient descent:
Θ(t+1) = Θ(t) − η ∂L(Θ (t))
∂Θ , (8)
where η is a learning rate. More explicitly,
a(t+1)r − a(0)r = (1− ηλ)(a(t)r − a(0)r )− η√ M (gΘ(t) − yt)σ(b (t)T t xt), b(t+1)r − b(0)r = (1− ηλ)(b(t)r − b(0)r )− η√ M (gΘ(t) − yt)arσ′(b (t)T t xt)xt. (9)

heading: E.2 NEURAL TANGENT KERNEL
text: In the theoretical analysis, the neural tangent kernel (NTK, Jacot et al., 2018) is used for approximating the dynamics of ASGD by a linear functional recursion on the corresponding function space. The NTK of this model is given by
k∞(x, x ′) = Eb(0) [σ(b (0)Tx)σ(b(0)Tx′)] + xTx′Eb(0) [σ ′(b(0)Tx)σ′(b(0)Tx′)]. (10)
The positive definite kernel k∞ naturally defines a reproducing kernel Hilbert space (RKHS), which is denoted byH∞. The integral operator Σ∞ on L2(ρX) is defined by
Σ∞f := ∫ k∞(·, x)f(x)dρX(x). (11)
It is known that Σ∞ admits eigendecomposition
Σ∞ψs = γsψs, (12)
where ψs is an eigenvector with ‖ψs‖L2(ρX) = 1 and γ1 ≥ γ2 ≥ . . . > 0 are eigenvalues in descending order. Mercer’s theorem tells that k∞ has an expansion:
k∞(x, x ′) = ∞∑ s=1 γsψs(x)ψs(x ′),
where the convergence is understood as in L2(ρX) for general, and absolutely and uniformly if ρX is a uniform distribution on a compact set.

heading: E.3 ASSUMPTIONS
text: For theoretical analysis, we make the following assumptions. For an operator Σ, the range of Σ is denoted byR(Σ).
(A1) The activation function σ is differentiable up to the second order, and there exists C > 0 such that ‖σ′′‖∞ ≤ C, ‖σ′‖∞ ≤ 2, and |σ(u)| ≤ 1 + |u| for ∀u ∈ R.
(A2) supp(ρX) ⊂ {x ∈ Rd | ‖x‖ ≤ 1} and y ∈ [−1, 1]. (A3) There exist 1/2 ≤ r0, r1 ≤ 1 such that φ0 ∈ R(Σr0∞) and φ1 ∈ R(Σr1∞). (A4) There exists ξ > 1 such that γ` = Θ(`−ξ).
As in Nitanda & Suzuki (2021), Assumption (A1) assumes that the activation σ is differentiable in this paper. In Nitanda & Suzuki (2021), however, they have developed a theory on how to extend the results to the case of ReLU by approximating it with a smooth function. It is well known that Assumption (A4) specifies the complexity of the hypothesis class H∞ (Caponnetto & De Vito, 2007); a faster eigen-decay (large ξ) implies the small complexity of the class. The assumption (A3) controls the smoothness of the target functions φ0, φ1. In fact, the functions are included in H∞, since φi ∈ R(Σ1/2∞ ) ⊂ H∞. When a function f has the expansion f = ∑ ` a`ψ`, the assumption f ∈ R(Σr∞) means a` = o(`−ξr−1/2). A function with a larger r is smoother, which is easier to learn. It is known (Caponnetto & De Vito, 2007; Nitanda & Suzuki, 2021) that ξ and r are the two basic parameters to control the convergence rate of generalization attained by kernel regression for a large sample size. Under the assumptions (A3) and (A4), given N i.i.d. training data (Xi, Yi) with Xi ∼ ρX and Yi = ϕ0(Xi) + εi with additive noise εi ∼ N(0, σ2), the kernel ridge regression ϕ̂λ with the regularization parameter λ = N−ξ/(2rξ+1) achieves the generalization E[‖ϕ̂λ − ϕ0‖2L2(ρX)] = O(N
−2rξ/(2rξ+1)) for any function ϕ0 with ϕ0 ∈ R(Σr), and it is known this rate is optimal.
In the sequel, when φ ∈ R(Σr) and φ = Σrψ, we write ‖Σ−rφ‖ := ‖ψ‖.

heading: E.4 GENERALIZATION BOUND
text: The dynamical behavior of pre-training can be discussed exactly in the setting of Nitanda & Suzuki (2021). Let φ̂0 be the result of pre-training, i.e., φ̂0 := gΘ(T0)pre , where Θ (T0) pre is the averaged parameter by ASGD. By optimizing the regularization parameter λ0, Corollary 1 in Nitanda & Suzuki (2021) shows that for sufficiently large T0, with a choice of λ0 = T −ξ/(2r0ξ+1) 0 ,
E‖φ̂0 − φ0‖2L2(ρX) ≤ εM + cT − 2r0ξ2r0ξ+1 0 ( 1 + ‖Σ−r0∞ φ0‖2L2(ρX) ) (13)
with high probability, where c is a universal constant and εM can be arbitrarily small for a large M . As discussed in Section E.3, it is known (Caponnetto & De Vito, 2007) that the rate T − 2r0ξ2r0ξ+1 0 achieves the minimax optimal rate with respect to T0 over the class specified by r0 and ξ.
In fine-tuning, the initial parameter is given by Θ(0) = Θ (T0)
pre , and ASGD is applied with (xt, yt) for t = 1, . . . , T1.
By extending Theorem 1 in Nitanda & Suzuki (2021), we can derive a generalization bound in the following theorem. Recall that the regularization coefficient and learning rate of fine-tuning are denoted by λ1 and η1, respectively. Theorem 2. Suppose Assumptions (A1)-(A3) hold. After pre-training that gives Eq. (13), finetune the network by Eq. (9) with a learning rate η1 and regularization coefficient λ1 that satisfy ‖Σ∞‖op ≥ λ1 > 0 and 4(6 + λ1)η1 ≤ 1. Then, for any ε > 0, δ ∈ (0, 1), and T1 ∈ N, there exists M0 ∈ N such that for any M ≥M0, the following bound holds with probability at least 1− δ over the random initiailzation of pre-training: E‖g
Θ (T1) − ϕ‖2L2(ρX)
≤ ε+ c0λ2r01 ‖Σ−r0∞ φ0‖2L2(ρX) + c1λ 2r1 1 ‖Σ−r1∞ φ1‖2L2(ρX)
+ c2
T1 + 1
{ λ−11 E‖φ̂0 − φ0‖2L2(ρX) ( 1 + ‖Σ−r0∞ φ0‖2L2(ρX) ) + λ2r1−11 ‖φ0‖2L2(ρX) + ‖φ1‖ 2 H∞ }
+ c3
(T1 + 1)2η21
{ λ−21 E‖φ̂0 − φ0‖2L2(ρX) ( 1 + ‖Σ−r0∞ φ0‖2L2(ρX) ) + λ2r1−21 ‖φ0‖2L2(ρX) + λ −1 1 ‖φ1‖2H∞ } +
c4 T1 + 1
( 1 + ‖ϕ‖2H∞ + 24‖Σ −r0 ∞ ϕ‖2L2(ρX) ) Tr [ Σ∞(Σ∞ + λ1I) −1], (14)
where φ̂0 is the result of pre-training and ci (i = 0, 1, 2, 3, 4) are universal constants.
The term ε is arbitrarily small for a large value of M , i.e., wide network. The proof of Theorem 2 will be given in Section E.6.

heading: E.5 ANALYSIS OF CONVERGENCE RATES
text: We consider the rates of the generalization bound for E‖g Θ (T1) − ϕ‖2L2(ρX) with respect to T1. As typical cases, we assume λ1 → 0 and η1 = O(1) as T1 →∞. The dominant terms in Eq. (14) may vary according to the configurations of λ1 and η1 with respect to T1. We will show the rates in some settings that are relevant to transfer learning.
First, note that under Assumption (A4), the factor tr[Σ∞(Σ∞ + λ1I)−1] is given by (Caponnetto & De Vito, 2007)
Tr [ Σ∞(Σ∞ + λ1I) −1] = O(λ−1/ξ1 ). (15) By neglecting ε, the terms in Eq. (14) thus have the following rates:
(a0) λ2r01 , (a1) λ 2r1 1 , (b) T −1 1 λ −1 1 R0, (c) T −1 1 λ 2r1−1 1 , (d) T −1 1 ,
(e) T−21 η −2 1 λ −2 1 R0, (f) T −2 1 η −2 1 λ 2r1−2 1 (g) T −2 1 η −2 1 λ −1 1 , (h) T −1 1 λ −1/ξ 1 . (16)
Here R0 := E‖φ̂0 − φ0‖2L2(ρX) is of constant rate with respect to T1, but explicitly shown for the later use.
Since λ1 → 0 and r1 ≥ 1/2, the terms (c) and (d) are of smaller rate than (b). Likewise, (f) and (g) are smaller than (e). The candidates of dominant terms are thus (a0), (a1), (b), (e), (g), and (h).

heading: E.5.1 LARGE REGULARIZATION COEFFICIENT
text: In transfer learning, it is reasonable to use strong regularization in fine-tuning, which encourages the parameters to stay close to the initial value that is obtained in the pre-training. In this subsection, we consider the case where λ1 is larger than T − ξ2r1ξ+1 1 , which would be the optimal rate if the network was trained with random initialization (see Nitanda & Suzuki (2021)). If λ1 = T − ξ2r1ξ+1 1 was taken, it is easy to see that the influence of pre-training would not appear explicitly in the convergence rate. In the sequel, we write F G(T1) if there are a > 0 and T ∗ such that F ≤ aG(T1) for all T1 ≥ T ∗. In this notation, we assume
λ1 T − ξ2r1ξ+1 1 . (17)
The rate λ1 = T − ξ2r1ξ+1 1 is given by equating the rates of (a1) and (h). Therefore, under the assumption of Eq. (17), the rate (a1) is larger than (h), and thus it suffices to consider (a0), (a1), (b) and (e) as the candidates of dominant terms. Note that, if λ1 → 0, the terms (a0) and (a1) decrease, while (b) and (e) increase to infinity.
We will discuss below the possible cases of dominant terms under the assumption Eq. (17). In the analysis, although the error of the pre-training R0 is regarded as a constant, we yet wish to consider the dependence of the fine-tuning result on R0. We thus set the regularization coefficient λ1 dependent on R0, and show that in all the cases, the generalization bound takes the form
E‖g Θ (T1) − ϕ‖2L2(ρX) ≤ ε+ CR ν 0T −β 1 , (18)
where ν > 0 is a constant. As R0 ≤ c′ +A′T−α ′ 0 from Eq. (13), the factor R α 0 can be bounded from above as Rν0 ≤ c+AT−α ′ν 0
for large T0. As a result, we obtain
E‖g Θ (T1) − ϕ‖2L2(ρX) ≤ ε+ C(c+AT −α 0 )T −β 1 , (19)
where C, c,A are constants. As we will see, the exponents α and β depend on r0, r1, ξ and η1. Eq. (19) accords with the bound in Theorem 1.
In the sequel, we use ζ ≥ 0 for the learning rate such that η1 = T−ζ1 .
Case I: Small learning rate T1λ1η21 1. In this case, (e) (b). We equate (a0) or (a1) with (e) to obtain λ1 for achieving the best possible upper bound of the two terms.
(I-A) r0 ≥ r1. Since λ1 → 0 for T1 →∞, (a1) is larger than (a0). By equating (a1) and (e), we find that the best choice of λ1 is
λ1 = O ( T − 1−ζr1+1 1 ) .
We further consider
λ1 = T − 1−ζr1+1 1 R ν 0
for dependence on R0. To determine ν, we assume that R0 is a small value, and consider the rate of (a1) and (e) with respect to R0 after plugging the above λ1 to them. By equating the rates of (a1) T − 2r1(1−ζ)r1+1 1 R 2r1ν 0 and (e) T − 2r1(1−ζ)r1+1 1 R 1−2ν 0 , the best possible rate of R0 is attained by ν = 1/(2r1 + 2). The dominant rate of Eq. (16) is thus
T − 2r1(1−ζ)r1+1 1 R r1 r1+1 0 (20)
attained by
λ1 = T − 1−ζr1+1 1 R 1 2(r1+1) 0 (21)
We need to identify the conditions on ζ1 to meet the requirements. The condition λ1 → 0 is equivalent to ζ < 1. There are two other conditions: λ1 T − ξ2r1ξ+1 1 and T1λ1η 2 1 1. Given R0 is of constant rate, the former is equivalent to − ξ2r1+1 ≤ − 1−ζ 1+r1 , which results in
ζ ≥ r1ξ + 1− ξ 2r1ξ + 1 .
The latter condition is equivalent to 1− 2ζ − 1−ζr1+1 ≤ 0, which is
ζ ≥ r1 2r1 + 1 .
It is not difficult to see r1
2r1 + 1 > r1ξ + 1− ξ 2r1ξ + 1
for ξ > 1. As a result, the condition on ζ is r1
2r1 + 1 ≤ ζ < 1. (22)
If η1 = T −ζ 1 is taken to satisfy this condition, the optimal rate of λ1 is given by Eq. (21). Finally, the resulting generalization bound is given by
E‖g Θ (T1) − ϕ‖2L2(ρX) ≤ ε+ cT − 2r1(1−ζ)r1+1 1 R r1 r1+1 0 . (23)
(I-B) r1 > r0: In this case, (a0) is of larger rate than (a1). By a similar argument to (I-A), with the rate
λ1 = T − 1−ζr0+1 1 R 1 2r0+2
0 , (24) The generalization bound is given by
E‖g Θ (T1) − ϕ‖2L2(ρX) ≤ ε+ cT − 2r0(1−ζ)r0+1 1 R r0 r0+1 0 . (25)
The condition on ζ is
max { r0
2r0 + 1 , (2r1 − r0)ξ + 1− ξ 2r1ξ + 1
} ≤ ζ < 1. (26)
(Case II): large learning rate T1λ1η21 1. Next, we consider the case where the learning rate η1 is large so that T1λ1η21 1, which includes the constant η1. Under this condition, (b) is of larger rate than (e).
(II-A) r0 ≥ r1. In this case, (a1) is of larger rate than (a0). A similar argument to (I-A) provides
λ1 = T − 12r1+1 1 R 1 2r1+1 0 , (27)
and the generalization bound is given by
E‖g Θ (T1) − φ0‖2L2(ρX) ≤ ε+ cT − 2r12r1+1 1 R 2r1 2r1+1 0 . (28)
The conditions are λ1 T − ξ2r1ξ+1 1 and T1λ1η 2 1 1. The former condition always holds for ξ > 1, and the latter is equivalent to ζ ≤ r12r1+1 . The resulting condition on ζ is
0 < ζ ≤ r1 2r1 + 1 . (29)
(II-B) r1 > r0: In this case, (a0) is of larger rate. With
λ1 = T − 12r0+1 1 R 1 2r0+1 0 , (30)
the generalization bound is given by
E‖g Θ (T1) − φ0‖2L2(ρX) ≤ ε+ cT − 2r02r0+1 1 R 2r0 2r0+1 0 . (31)
The conditions λ1 T − ξ2r1ξ+1 1 and T1λ1η 2 1 1 are respectively r1 ≤ r0 + ξ−1 2ξ and ζ ≤ r0 2r0+1
. Thus, we require
0 < ζ ≤ r0 2r0 + 1 , r0 < r1 ≤ r0 + ξ − 1 2ξ , (32)
In summary, the generalization bounds in various conditions are summarized in Table 1.

heading: E.6 PROOF OF THEOREM 2
text: The proof of Theorem 2 is based on the application of the theory in Nitanda & Suzuki (2021) to the fine-tuning phase, adapting the initialization given by the result of pre-training φ̂0.
In the sequel, we focus on the fine-tuning with T1 samples with yt = ϕ(xt). Recall that
ϕ(x) = φ0(x) + φ1(x).

heading: E.6.1 REFERENCE ASGD ON RKHS
text: We use a surrogate sequence of functions in an RKHS for the proof. Let kM be the random feature approximation of the TNK k∞, i.e.,
kM (x, x ′) =
1
M M∑ r=1 σ(bTr x)σ(b T r x ′) + xTx′ M M∑ r=1 σ′(bTr x)σ ′(bTr x ′), (33)
where (br)Mr=1 is i.i.d. random sample from the uniform distribution on the unit sphere Sd−1. The associated RKHS is denoted byHM . A reference ASGD is defined by the following update rule of functions in the RKHSHM :
g(t+1) = (1− ηλ)g(t) − η(g(t)(xt)− yt)kM (·, xt), (t = 0, . . . , T − 1) (34)
with the initialization given by g(0) := φ̂0. The average is taken at the final step:
ḡ(T1) := 1
T1 + 1 T1∑ t=0 g(t). (35)
By considering continual learning of pre-training and fine-tune, a slight modification of (Nitanda & Suzuki, 2021, Propososion A) derives the following proposition. Proposition 3. Assume (A1) and (A2). Suppose that η1λ1 < 1. Then for any T1 ∈ N and ε > 0, there is M∗ = M∗(T1, ε) ∈ N such that during the fine-tuning learning
‖ḡ(t) − g Θ (t)‖L∞(ρX) ≤ ε, (36)
holds for any M ≥M∗ and 0 ≤ t ≤ T1.
This proposition shows that, if we use a very wide network, the learning of ASGD in the parameter space can be approximated by the reference ASGD on the RKHS with negligible error.
The generalization bound will be given by the following decomposition:
‖g Θ (T1) − ϕ‖2L2(ρX) ≤ 2‖gΘ(T1) − ḡ (T1)‖2L2(ρX) + 2‖ḡ (T1) − ϕ‖2L2(ρX), (37)
in which the first term of the right hand side is bounded by Proposition 3 with an arbitrary small value ε for large M . The second term will be discussed in the next subsection.

heading: E.6.2 CONVERGENCE RATES OF REFERENCE ASGD
text: In this section, we write λ and η for λ1 and η1 for simplicity. The covariance operators Σ∞ and ΣM forH∞ andHM , respectively, are defined by
Σ∞ := EρX [k∞(·, X)⊗ k∞(·, X)∗], ΣM := EρX [kM (·, X)⊗ kM (·, X)∗], (38)
where ∗ denotes the adjoint; equivalently,
Σ∞f = ∫ k∞(·, x)f(x)dρX(x), ΣMh = ∫ kM (·, x)h(x)dρX(x),
for f ∈ H∞, h ∈ HM . The regularized target functions φ(i)M,λ (i = 0, 1) are defined by
φ (i) M,λ := (ΣM + λI) −1ΣMφi (i = 0, 1). (39)
φ (i) ∞,λ is defined similarly with Σ∞. Note that ϕM,λ := (ΣM + λI) −1ΣMϕ = φ (0) M,λ + φ (1) M,λ.
First, we decompose ‖ḡ(T1) − ϕ‖2L2(ρX) by
‖ḡ(T1) − ϕ‖2L2(ρX) = ‖ḡ (T1) − ϕM,λ + φ(0)M,λ + φ (1) M,λ − φ0 − φ1‖ 2 L2(ρX)
≤ 3‖ḡ(T1) − ϕM,λ‖2L2(ρX) + 3‖φ (0) M,λ − φ0‖ 2 L2(ρX) + 3‖φ(1)M,λ − φ1‖ 2 L2(ρX) .
The second and third terms are known to have a bound, with high probability, (Nitanda & Suzuki, 2021, Propositions C and D)
‖φ(i)M,λ − φi‖ 2 L2(ρX) ≤ ε+ λ2ri‖Σ−ri∞ φi‖2L2(ρX), (i = 0, 1), (40)
where ε is arbitrarily small for large M . We have thus, with high probability,
‖ḡ(T1)−ϕ‖2L2(ρX) ≤ ε+3‖ḡ (T1)−ϕM,λ‖2L2(ρX)+3λ 2r0‖Σ−r0∞ φ0‖2L2(ρX)+3λ 2r1‖Σ−r1∞ φ1‖2L2(ρX).
(41)
As shown in Nitanda & Suzuki (2021), the term ‖ḡ(T1) − ϕM,λ‖2L2(ρX) can be analyzed by the bias and noise terms of the stochastic recursion on RKHS Eq. (34), which is rewritten as
g(t+1) = (I − ηHt − ηλI)g(t) + ηytkM (·, xt), (42) where Ht := kM (·, xt)⊗ kM (·, xt)∗, is a one-sample estimate of ΣM . By subtracting ϕM,λ from both hand sides of Eq. (42), we have
g(t+1) − ϕM,λ = (I − ηHt − ηλI)(g(t) − ϕM,λ) + βt, (43) where βt = ηytkM (·, xt)− η(Ht + λI)(ΣM + λI)−1ΣMϕM,λ is the zero mean noise term. Using this recursive formula, Nitanda & Suzuki (2021) derives a bound:
‖ḡ(T ) − ϕM,λ‖2L2(ρX) ≤ c1 T1 + 1 ‖(ΣM + λI)−1/2(g(0) − ϕM,λ)‖2L2(ρX)
+ c2
(T1 + 1)2η2 ‖(ΣM + λI)−1(g(0) − ϕM,λ)‖2L2(ρX)
+ c3
T1 + 1
( 1 + ‖ϕ‖2L2(ρX) + 24‖Σ −r1 ∞ ϕ‖2L2(ρX) ) Tr[ΣM (ΣM + λI) −1].
(44)
To bound this expression further, we use Proposition B in Nitanda & Suzuki (2021)
‖(ΣM + λI)−1/2φ(i)M,λ‖ 2 L2(ρX) ≤ 2‖φi‖2H∞ . (45)
Then, using the decomposition g(0) − ϕM,λ = (φ̂0 − φ0) + (φ0 − φ(0)M,λ) − φ (1) M,λ, we obtain that, with high probability,
‖(ΣM + λI)−1/2(g(0) − ϕM,λ)‖2L2(ρX) ≤ 3‖(ΣM + λI)−1/2(φ̂0 − φ0)‖2L2(ρX) + 3‖(ΣM + λI) −1/2(φ0 − φ(0)M,λ)‖ 2 L2(ρX)
+ 3‖(ΣM + λI)−1/2φ(1)M,λ‖ 2 L2(ρX)
≤ 3 λ ‖φ̂0 − φ0‖2L2(ρX) + 3 λ λ2r0‖φ0‖2L2(ρX) + εM + 6‖φ1‖ 2 H∞ , (46)
where we use Eq. (40) for the second and third terms and Eq. (45) for the fourth term in the last inequality. Similarly, with high probability, we have
‖(ΣM + λI)−1(g(0) − ϕM,λ)‖2L2(ρX) ≤ 3‖(ΣM + λI)−1(φ̂0 − φ0)‖2L2(ρX) + 4‖(ΣM + λI) −1(φ0 − φ(0)M,λ)‖ 2 L2(ρX)
+ 3‖(ΣM + λI)−1φ(1)M,λ‖ 2 L2(ρX)
≤ 3 λ2 ‖φ̂0 − φ0‖2L2(ρX) + 3 λ2 λ2r0‖φ0‖2L2(ρX) + εM + 6 λ ‖φ1‖2H∞ . (47)
It is also known (Nitanda & Suzuki, 2021, Proposition B) that, for λ ≤ ‖Σ∞‖,
Tr[ΣM (ΣM + λI) −1] ≤ 3Tr[Σ∞(Σ∞ + λI)−1]. (48)
Combining Eqs.(41), (44), (46), (47), and (48), we obtain the assertion of the theorem.

Review summaries:
This paper proposes a scaling law for transfer learning that takes into account  the size of both the pretraining and finetuning datasets. They showed the parameters of the scaling law can be  estimated and this can be used to approximate the performance of the model given the amount of pretraining data. The authors experiments are focused on the scenario where the pretranining dataset is synthetically generated.

This paper postulates a scaling law for pre-training and transfer learning via fine-tuning. They empirically postulate a law and also theoretically show that in a simpler setting with assumptions, a similar law should hold. The paper is written and experiments are done in a synthetic to real transfer setting. Experiments are done by pre-training on a synthetic dataset made using BlenderProc and transferring to real data ranging classification, object detection and semantic segmentation. They find that parameters of their scaling law can be estimated to find very good fits to empirical error across all the different transfer tasks.