A Scaling Law for Syn-to-Real Transfer: How Much Is Your Pre-training Effective?
Authors: Hiroaki Mikami
Kenji Fukumizu
Shogo Murai
Shuji Suzuki
Yuta Kikuchi
Taiji Suzuki
Shin-ichi Maeda
Kohei Hayashi
Venue: ICLR 2022 Submitted (2022)
Keywords: Transfer learning
Computer vision
Scaling law
Pre-training
Synthetic-to-real

Abstract: Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited data size expansion without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is compatible with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.

Under review as a conference paper at ICLR 2022
A SCALING LAW FOR SYN2REAL TRANSFER: How MUCH IS YOUR PRE-TRAINING EFFECTIVE?
Anonymous moonymous authors Paper under double-bling review
ABSTRACT
Synthetic-to-red transfer learning is a framework which a synthetically generated dataset is used pre-train model improve its performance real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.
INTRODUCTION
The success of deep learning relies on the availability of large data. If the target task provides limited data, the framework of transfer learning is preferably employed. A typical scenario of transfer learning is to pre-train model for similar or even different task and fine-tune the model for the target task. However, the limitation of labeled data has been the main bottleneck of supervised pre-training. While there have been significant advances in the representation capability of the models and computational capabilities of the hardware, the size and the diversity of the baseline dataset have not been growing as fast (Sun et al. 2017). This is partially because of the sheer physical difficulty of collecting large datasets from real environments (e.g., the cost of human annotation).
computer vision, synthetic-to-real (syn2real) transfer is promising strategy that has been attracting attention (Su al. 2015 Movshovitz-Attia
al. 2016: Georgakis al. 2017 Tremblay et al.) 2018 Hinterstoisser et al. 2019; Borrego et al. 2018 Chen et al. 2021). In syn2real, images used for pre-training synthesized to improve the performance on real vision tasks. By combining various conditions, such as 3D models, textures, light conditions, and camera poses, we can synthesize an infinite number of images with ground-truth annotations. Syn2real transfer has already been applied in some real-world applications. TTed & Deng (2021) proposed a simultaneous localization and mapping (SLAM) system that was trained only with synthetic data and demonstrated state-of-the-an performance. The object detection networks for autonomous driving developed by Tesla was trained with 370 million images generated by simulation (Karpathy (021).
The performance of syn2real transfer depends on the similarity between synthetic and real data. In general, the more similar they are, the stronger the effect of pre-training will be. On the contrary, if there is significant gap, increasing the number of synthetic data may be completely useless, in which case we waste time and computational resources. A distinctive feature of syn2real is that we can control the process of generating data by ourselves. If a considerable gap exists, can try to regenerate the data with a different setting. But how do we know that? More specifically, in a standard learning setting without transfer, "power law"-like relationship called scaling law often holds between data size and generalization errors (Rosenfeld et al. 2019 Kaplan et al. 2020). Is there such rule for pre-training?

---

Under review as a conference paper at ICLR 2022
pretrain task
objdet
semseg
mulclass
noraal
0.97
COCO
objdet
0.96
0.94
finetune task
0.96
test error
semsee
0.95
0.94
0.90
0.88
sinclass
0.86
0.82
10
10
10"
10
10
10
10"
10
10
10
10
10
10a
10
10
10
# of pretrain images
Figure 1: Empirical results of syn2real transfer for different tasks. We conducted four pre-training tasks: object detection (objdet), semantic segmentation (semseg), multi-label classification (mulclass), surface normal estimation (normal), and three fine-tuning tasks for benchmark datasets: object detection for MS-COCO, semantic segmentation for ADE20K, and single-label classification iscncl ass) for ImageNet. The y-axis indicates the test error for each fine-tuning task. Dots indicate empirical results and dashed lines indicate the fitted curves of scaling law (I. For more details, see Section 4.2
In this study, find that the generalization error fine-tuning is explained by simple scaling law,
test error Dn a + C,
(1)
where coefficient D O and exponent >> 0 describe the convergence speed of pre- pre-training, and constant C 2 determines the lower limit of the error. We refer to a as pre-training rate and C transfer gap. We predict how large the pre-training data should be to achieve the desired accuracy by estimating the parameters o, C' from the empirical results. Additionally, we analyze the dynamics of transfer learning using the recent theoretical results based on the neural tangent kernel (Nitanda & Suzukii (202I) and confirm that the above law agrees with the theoretical analysis. We empirically validated our scaling law various experimental settings benchmark tasks, model sizes, and complexities of synthetic images.
Our contributions are summarized follows.
From empirical results and theoretical analysis, we elicit law that describes how generalization scales in terms of data sizes on pre-training and fine-tuning.
We confirm that the derived law explains the empirical results various settings in terms of pre-traine preetraining/fine--e tuning tasks, model size, and data complexity (e.g.. Figure I Furthermore, demonstrate that we can use the estimated parameters in our scaling law to assess how much improvement
expect from the pre-training procedure based on synthetic data.
We theoretically derive generalization bound for general transfer learning setting and confirm its agreement with our empirical findings.
RELATED WORK
Supervised pre-training for visual tasks Many empirical studies show that the performance at a fine-tuning task scales with pre-training data (and model) size. For example. Huh et al. (0166
studied the scaling behavior ImageNet pre- pre-trained models. Beyond ImageNet, [unn et al. (017) studied the effect of pre-training with pseudo-labeled large-scale data and found a logarithmic scaling behavior. Similar results were observed by Kolesnikov et al. (2019)
2

---

Under review as a conference paper at ICLR 2022
Syn2real transfer The utility of synthetic images as supervised data for computer vision tasks
has been continuously studied by many researchers (Su et [2015] Movshovitz Attias et al.
2016
Georgakis et al. [017 Tremblay et al. 2018 Hinterstoisser
2019, Borrego et al. 2018 Chen et al. 2021 Newell& & Deng] |020 Devaranjan et al., 2020; Mousavi et al. 2020 Hodañ et al. 2019).
These studies found positive evidence that using synthetic images is helpful to the fine-tuning task. In addition, they demonstrated how data complexity. induced by e.g., light randomization, affects the final performance. For example, Newell & Deng (020) investigated how the recent self-supervised methods perform well as pre-training task to improve the performance of downstream tasks. In this paper, following this line of research, we quantify the effects under the lens of the scaling law (T)
Neural scaling laws The scaling behavior of generalization error, including some theoretical works (e.g../Amat et al., [1992). has been studied extensively. For modern neural networks, [Hestness et al.) (2017) empirically observed the power-law behavior of generalization for language, image, and speech domains with respect the training size. [Rosenelld et al. (2019) constructed predictive form for the power-law in terms of data and model sizes. Kapllan et al, (2020) pushed forward this direction in the language domain, describing that the generalization of transformers obeys the power law in terms of compute budget in addition to data and model sizes. Since then, similar scaling laws have been discovered in other data domains (Henighan et al. 2020). Several authors have also attempted theoretical analysis. Hutter (2021) analyzed a simple class of models that exhibits a power-law n in terms of data size n with arbitrary / > 0. [ahri et al. (202T) addressed power laws under four regimes for model and data size. Note that these theoretical studies, unlike ours, are concerned with scaling laws in a non-transfed setting.
Hernandez et al. 2021) studied the scaling laws for general transfer learning, which is the most relevant to this study. A key difference is that they focused on fine-tuning data size as a scaling factor, while we focus pre-training data size. Further, they found scaling laws in terms of the transferred effective data, which is converted data amount necessary to achieve the same performance gain by pre-training. In contrast, Eq. ① explains the test error with respect to the pre-training data size directly at a fine-tuning task. Other differences include task domains (language vs. vision) and architectures (transformer vs. CNN).
Theory of transfer learning Theoretical analysis of transfer learning has been dated back to decades ago (Baxter, 2000) and has been pursued extensively. Among others, some recent studies (Maurer et al. 2016; Du et al. 2020; TTripuraneni et al.] 2020)p derived an error bound of a fine-tuning task in the multi-task scenario based complexity analysis; the bound takes an additive form O(An-1/2 + Bs-1/2), where n and are the data size of pre-training and fine-tuning, respectively, with coefficients A and B. Neural network regression has been also discussed with this bound (Tripuraneni et al.) 2020). In the field of domain adaptation, error bounds have been derived in relation to the mismatch between source and target input distributions (Ganin et al., 2016 Acuna et al. 2021). They also proposed algorithms to adopt a new data domain. However, unlike in this study, no specific learning dynamics has been taken into account. In the area of hypothesis transfer learning (Fei-Fei et al. [0066] Yang et al. 2007),, among many theoretical works, |Du et al.) (2017) has derived a risk bound for kernel ridge regression with transfer realized as the weights on the training samples. The obtained bound takes a similar form to our scaling law. However, the learning dynamics of neural networks initialized with a pre-trained model has never been explored in this context.
SCALING LAWS FOR PRE-TRAINING AND FINE-TUNING
The main obstacle in analyzing the test error is that we have to consider interplay between the effects of pre-training and fine-tuning. Let L(n, s) > O be the test error of fine-tuning task with pre-training data size n and fine-tuning data size 8. As the simplest case, consider a fine-tuning task without pre-training (n = 0), which boils the transfer learning down
standard learning setting. In this case, the prior studies of both classical learning theory and neural scaling laws tell us that the test error decreases polynomially! [1] with the tne-tuning data size s, that is, ((0, Bs 88 + E with decay rate 3 >> 0 and irreducible loss & 2 0. The irreducible loss E is the inevitable error given by
For classification with strong low-noise condition, it is known that the decay rate can be exponential (Nitanda Suzuki 2019. However, focus only on the polynomial decay without such strong condition in this paper.
3

---

Under review as a conference paper at ICLR 2022
the best possible mapping; it is caused by noise in continuous outputs labels. Hereafter assume = for brevity.
3.1 INDUCTION OF SCALING LAW WITH SMALL EMPIRICAL RESULTS
To speculate scaling law, we conducted 2
(6)
preliminary experiments We pre- pre-trained ResNet-50 by a synthetic classification task and fine-tuned by ImageNet. Figure (a) presents the log-log plot of error curves with respect to pre-training data size n. where each shape and color indicates a dif- ferent fine-tuning size It shows that the pre-training effect diminishes for large n. In contrast, Figure (b) presents the rela- tions between the error and the fine-tuning size 8 with different n. It indicates the error drops straight down regardless of n, confirming the power-law scaling with respect to 8. The above observations and the fact that L(0, s) decays polynomially are summarized as follows.
0.50
0.0 19
0.80
0.76
10"
10.1
of protram Images
finouune
Figure Scaling curves with different (a) pre-training size and (b) fine-tuning size.
Requirement 1. limg-+00 [nn, s) O. Requirement 2. limm-soe [1n, 8) = const.
Requirement L(0,8) - B
Requirements and suggest the dependency of n is embedded in the coefficient B == g()), i.e., the pre-training and fine-tuning effects interact multiplicatively. To satisfy Requirement2] a reasonable choice for the pre-training effect is y(n) n +;; the error decays polynomially with respect but has a plateau at y. By combining these. we obtain
L(n,s) s) = 5(7 + 9n)
(2)
where or are decay rates for pre-training and fine-tuning, respectively, 7 2 0 is constant, and 8 > 0 is coefficient. The exponent /3 determines the convergence rate with respect to fine-tuning data size. From this viewpoint, 6(7 + ") the coefficient factor to the power law. The influence of the pre-training appears in this coefficient, where the constant term d comes from the irreducible loss of the pre-training task and - expresses the effect of pre-training data size. The theoretical consideration in Section E.5 suggests that the rates and B can depend on both the target functions of pre-training and fine-tuning as well as the learning rate.
3.2 THEORETICAL DEDUCTION OF SCALING LAW
Next, analyze the fine-tuning error from purely theoretical point of view. To incorporate the effect of pre-training that is given as an initialization, we need to analyze the test error during the training with given learning algorithm such SGD. We apply the recent development by Nitanda & Suzuki (202T) to transfer learning. The study successfully analyzes the generalization of neural networks in the dynamics of learning, showing it achieves minmax optimum rate. The analysis uses the framework of the reproducing kernel Hilbert space given by the neural tangent kernel (Jacot et al. 2018).
For theoretical analysis of transfer, it is important to formulate task similarity between pre-training and fine-tuning. If the tasks were totally irrelevant (e-g.. learning MNIST to forecast tomorrow's 's weather), pre-training would have benefit. Following Nitanda & Suzuki (2021), for simplicity of analysis, we discuss only a regression problem with square loss. We assume that a vector input x and scalar output follow y (z(z) for pre- pre-training and y Φu(z) + &1 (x) for fine-tuning, where omit the output noise for brevity; the task types are identical sharing the same input-output form, and task similarity is controlled by Φ1.
We analyze the situation where the effect of pre-training remains in the fine-tuning even for large data size (s oo). More specifically, the theoretical analysis assumes regularization term the
The results are replicated from Appendix C.2 the subsection for details.
4

---

Under review as a conference paper at ICLR 2022
lg-distance between the weights and the initial values, and smaller learning rate than constant in the fine-tuning. Hence we control how the pre-training effect is preserved through the regularization and learning rate. Other assumptions made for theoretical analysis concern the model and learning algorithm; a two-layer neural network having M hidden units with continuous nonlinear activation is adopted; for optimization, the averaged SGD (Polyak & Juditsky 1992). online algorithm, used for technical reason.
The following is an informal statement of the theoretical result. See Appendix for details. We emphasize that our result holds not only for syn2real transfer but also for transfer learning in general.
Theorem 1 (Informal). Let fn,(()) be model of width M pre-trained by samples (11,3)) (o),3))...... (Ins Un) and fine-tuned by samples (r4, V.),.... (x's) yf2 where imputs x, x' ~ p()) are ii.d. with the input distribution p(x)) and y = (2())) and yf = xp(x') ooo(')) 01 (x'). Then the generalization error of the squared loss L(n, - - [p(x){
is bounded from above with
from
high probability as
EzL(n, s) S A1(cM + Ann +EM. M and eM can be arbitrary small for large M; An and A are constants; the exponents a and B depend on th, oi, p(æ), and the learning rate of fine-tuning.
(3)
The above bound (3) shows the correspondence with the empirical derivation of the full scaling law 2 Note that the approximation error E is omitted (
We note that the derived bound takes multiplicative form in terms of the pre-training and fine-tuning effects, which contrasts with the additive bounds such as An 1/2 + Bs 1/2 (Tripuraneni et al. 2020). The existing studies consider the situation where part of network (e.g.. backbone) frozen during fine-tuning. Therefore, the error of pre-training is completely preserved after fine-tuning, and both errors appear in an additive way. This means that the effect of pre-training irreducible by the effect of fine-tuning, and vice versa. In contrast, network in tine-tuning. In that case, the pre-trained model is used as initial values. As a result, even if the error in pre-traine pre-training is large, the final error can be reduced to zero by increasing the amount of fine-tuning data.
analysis deals with the case of re-optimizing the entire
3.3 INSIGHTS AND PRACTICAL VALUES
The form of the full scaling law ? suggests that there are two scenarios depending on whether fine-tuning data is big or small. In "big fine-tune" regime, pre-training contributes relatively little. By taking logarithm, we can separate the full scaling law (2) into the pre- pre-training part u(n) log(n + +2) and the fine-tuning part v(s) -3 logs. Consider to increase by squaring it. Since the pre-training part cannot be reduced below log(7) as u(n) > uun²) > log(r). the relative improvement (u(n²)- - u(n))/ (s)) becomes infinitesimal for large Figure 2 (b) confirms this
situation. Indeed, prior studics provide the same conclusion that the gain from pre-training can easily
vanish (He et al. 2018; Newell & Deng 2020 or target task accuracy even degrade (Zoph et al. 2020 if we have large enough fine-tuning data.
The above observation, however, does not mean pre- training is futile. Dense prediction tasks such as depth estimation require pixel-level annotations, which critically limits the number of labeled data. Pre-training is indispens- able in such "small fine-tune" regime. Based on this, we hereafter analyze the case where the fine-tuning size is fixed. By eliminating s-dependent terms in (2). we obtain a simplified law (I) by setting D = ós and C' = -8. After several evaluations, these parameters including a can be estimated by the nonlinear least squares method (see also Section
Log Eroo
Target Error Log Pre-training Size
(a)
(b)
(c)
(d)
As a practical benefit, the estimated parameters of the simplified law (I) bring a way to assess syn2real transfer. Suppose we want to solve a classification task that requires at least 90% accuracy with limited labels.
Figure 3: Pre-training scenarios.
RRL.UU is not included in this class, but can generalize this condition; Nitanda & Suzuki 2021
5

---

Under review as a conference paper at ICLR 2022
We generate some number of synthetic images and pre-train with them, and we obtain 70% accuracy as Figure 3 (a). How can we achieve the required accuracy? It depends on the parameters of the scaling law. The best scenario is (b) transfer gap C' is low and pre-training rate is high. In this case, increasing synthetic images eventually leads the required accuracy. In contrast, when transfer gap C' is larger than the required accuracy (c), increasing synthetic images does not help to solve the problem. Similarly, for low pre-training rate a (d), we may have to generate tremendous amount of synthetic images that are computationally infeasible. In the last two cases, we have to change the rendering settings such as 3D models and light conditions to improve C' and/or a, rather than increasing the data size. The estimation of and C requires to compute multiple fine-tuning processes. However, the estimated parameters tell us whether we should increase data or change the data generation process, which can reduce the total number of trials and errors.
EXPERIMENTS
4.1 SETTINGS
For experiments, we employed the following transfer learning protocol. First, pre-train model that consists of backbone and head networks from random initialization until convergence, and we select the best model in terms of the validation error of the pre-training task. Then, we extract the backbone and add a new head to fine-tune all the model parameters. For notations, the task names of object detection, semantic segmentation, multi-label classification, single-label classification, and surface normal estimation are abbreviated as objdet, semseg, mulclass sinclas ass, and normal, respectively. The settings for transfer learning are denoted by arrows. For example, ob jojeett+a -seses indicates that model is pre-trained by object detection, and fine-tuned by semantic segmentation. The experiments were conducted on an in-house cluster containing NVIDIA V100 GPUs. The total amount of computation was approximately 1700 GPU days (200 for image rendering, 1300 for pre-training, and 200 for fine-tuning). All the results including Figure] are shown as log-log plots.
Pre-training: We prepared four tasks: mulclass, jbjet semseq, and normal. We used ResNet-based models, where backbones were ResNet-50, unless otherwise specified, and the head networks were customized for each task. Synthetic images for pre-training were generated by BlenderProc (Denninger et al. (2019), image renderer that can handle several domain randomization methods. For rendering, we used the setting of the BOP challenge 2020 (Hodañ et al. 2020 as our default setting. We used 172 3D models, where ten objects appeared on average for each image. We applied texture randomization for walls and a floor, randomization for area and point lights, and randomization for the camera. most cases, the models were pre-trained with 64,000 images. We trained all models for the same fixed number of iterations depending on pre-training tasks and selected the best models for fine-tuning, which were validated by another 1000 synthetic images generated in the same way.
Fine-tuning: We evaluated sinclass by ImageNet (Russakovsky et al. 2015). objdet by MS-COCO (Lin al. 2014), and semseg by ADE20K (Zhou al.] (016)) The number of images used was 1% of each data set (roughly, 12,000 for ImageNet, 1000 for COCO, and 200 for ADE20K). We fine-tuned the pre-trained models with these subsets of data for a fixed number of iterations and reported the error metrics for validation sets at the last iteration. The metrics were top-1 accuracy for classification, mean mAP for MS-COCO, and mean IoU for ADE20K. These metries take their values from O to 1, and we converted them into errors such as 1 accuracy
Curve fitting: After obtaining the empirical errors L. we estimated the parameters of ( by non- linear least squares in the log-log space. We solved the minimization problem of Σ, log L(n,, s) log( (D, 2 with a fixed fine-tuning data size and pre-training data sizes n, 2 x 1000 for data point index i = O.... 6. In the experiments, we empirically encountered some instability between D and a. We fixed D 0.48 by the median values of D's for all the settings and estimated and C' independently for each case. We explain this procedure with more details in Appendix D
A lthough the cross-entropy loss is commonly used, several studies (Sharma & Kaplan 2020 Bahri al. show that the scaling laws also hold for 1 accuracy.
2021

---

Under review as a conference paper at ICLR 2022
4.2 SCALING LAW UNIVERSALLY EXPLAINS DOWNSTREAM PERFORMANCE FOR VARIOUS TASK COMBINATIONS
Figure I shows the test errors of each fine-tuning task and fitted learning curves with Eq. I which describes the effect of pre-training data size for all combinations of pre-training and fine-tuning tasks. The scaling law fits with the empirical ane-tunnng test errors with high accuracy in most cases.
4.3
BIGGER MODELS REDUCE THE TRANSFER GAP
model
r18
134
-R.
150
101
/152
mukclass sinclass
objdet objdet
muiclass sinclass
objdet-+ objdet
0.90
0.96
0.935
entrr
0.88
0.82
0.95
0.930
0.86
0.94
0.80
0.84
0.925
0.82
0.93
10a
10
10
10
10'
10ah
10
1045
20
30
50
# of pretrain images
parameters (Million)
The scaling curves for and objdet-robjde cases. The meanings of dots and lines the same as those in Figure Right: The estimated transfer gap C' (y-axis) versus the model size (x-axis) in log-log scale. The dots are estimated values, and the lines are linear fittings of them.
Figure 4: Effect of model size. Best viewed in color. Left: wwl.class-sincclen
We compared several ResNet models as backbones in mulclass-rsincll -odet-bbjdds to observe the effects of model size. Figure 4 (left) shows the curves of scaling laws for the pre-training data size n for different sizes of backbone ResNet-a, where {18, 34, 50, 101, 152). The bigger models attain smaller test errors. Figure 4 (right) shows the values of the estimated transfer gap C. The results suggest that there is a roughly power-law relationship between the transfer gap and model size. This agrees with the scaling law with respect to the model size shown by Hernandez et al. (2021).
and
4.4
SCALING LAW CAN EXTRAPOLATE FOR MORE TRE-TRAINING IMAGES
objdet - objdet
fitting
0.96
powerlaw
0.95
proposed
test errr
0.94
model
powerlaw
proposed
r50
0.0099
0.0024
0.93
model
r101
0,0074
0.0016
0.92
r50
r152
0.0091
0.0017
r101
0.91
r152
10a5
10"
10
10a
10as
10c
of pretrain images
The solid lines represent the fitted power law and the dashed curves represent the fitted scaling law [ in which the laws were fitted using the empirical errors where the pre-training size was less than 64,000 (the first five dots). The vertical dashed line indicates where n = 64,000. Right: The root-mean-squee square errors between the laws and the actual test errors in the area of extrapolation (the last five dots).
Figure 5: Ability to extrapolate. Left:
7

---

Under review as a conference paper at ICLR 2022
We also evaluated the extrapolation ability of the scaling law. We increased the number of synthetic images from the original size (n = 64,000) to 1.28 million, and see how the fitted scaling law predicts the unseen test errors where 64,000. As bascline, we compared the power-law model, which is equívalent to the derived scaling law ① with C' = 0. Figure 5 (left) shows the extrapolation setting, which indicates the scaling law follows the saturating trend
results in ob j jddtt+bbb in regions with large pre-training sizes for all models, while the power-law model fails to capture it. The prediction errors is numerically shown in Figure (right), which again shows our scaling law achieves better prediction performance.
4.5
DATA COMPLEXITY AFFECTS BOTH PRE-TRAINING RATE AND TRANSFER GAP
appearance
5000
sbjfet obidet
mupippe
4500
AGOSLO
single
1500
0.8
0.95
light
ectimetl
0.7
fX
testeror
random
0.5
0.95
background
0.96
0.95
random
0.94
0.94
object
texture
W!
Wio
W/
W/o
appeer
siglle EX
single fi
millipe muliiple f6 rendom random random tendom
mullisle multiple mulippe muliipe
testure
1ghh
1000
3000
10000
30000
backg sund
tx
5x
random random
# of pretrain images
Figure 6: Effect of synthetic image complexity. Best viewed in color. Left: Scaling curves of different data complexities. Right: Estimated parameters. The error bars represent the standard error of the estimate in least squares.
We examined how the complexity of synthetic images affects fine-tuning performance. We controlled the following four rendering parameters: Appearance: Number of objects in each image; si ngle or multipp iple (max 10 objects). Light: Either an area and point light is randomized or fixed in terms of height, color, and intensity. Background: Either the textures of floor/wall are randomized or fixed. Object texture: Either the 3D objects used for rendering contain texture (w/) or not (w/ o). Indeed, the data complexity satisfies the following ordered relationships: single mult till in appearance, fix < random in light and background. and w w/o < w / in object texture To quantify the complexity, we computed the negative entropy of the Gaussian distribution fitted the last activation values of the backbone network. For this purpose, we pre-trained ResNet-50 as a backbone with MS-COCO for 48 epochs and computed the empirical covariance of the last activations for all the synthetic data sets.
The estimated parameters are shown in Figure 6 which indicates the following (we discuss the implications of these results further in Section 5.1
Data complexity controlled by the rendering settings correlates with the negative entropy, implying the negative entropy expresses the actual complexity of pre-training data.
Pre-training rate a correlates with data complexity. The larger complexity causes slower rates of convergence with respect to the pre-training data size.
Transfer gap C' mostly correlates negatively with data complexity, but not for object texture.
As discussed in Section 4.1 we have fixed the value of D) avoid numerical instability, which might cause some bias to the estimates of a. We postulate, however, the value of D depends mainly on the fine-tuning task and thus has fixed value for different pre-training data complexities. This can be inferred from the theoretical analysis in Appendix E.5 the exponent / in the main factor s of D does not depend on the pre- pre-training data distribution but only on the fine-tuning task or the pre-training true mapping. Thus, the values of D should be similar over the different complexities, and the correlation of a preserves.
TThe object category of w/ o subset of w/, and w/ has strictly higher complexity than w/o.
8

---

Under review as a conference paper at ICLR 2022
5 CONCLUSION AND DISCUSSION
In this paper, we studied how the performance on syn2real transfer depends on pre-training and fine-tuning data sizes. Based on the experimental results, we found scaling law ( and its generalization 2 that explain the scaling behavior in various settings in terms of pre-training/iigg tuning tasks, model sizes, and data complexities. Further, we present the theoretical error bound for transfer learning and found our theoretical bound has good agreement with the scaling law.
5.1 IMPLICATION OF COMPLEXITY RESULTS IN SECTION 4.5
The results of Section 4.5 has two implications. First, data complexity (i.e., the diversity of images) largely affects the pre-training rate a. This is reasonable because if we want network to recognize more diverse images, we need to train it with more examples. Indeed, prior studies (Sharma & Kaplan 2020 Bahri al. 2021) observed that a is inversely proportional to the intrinsic dimension of the data (c.g., dimension of the data manifold), which is an equivalent concept of data complexity.
Second, the estimated values of the transfer gap C suggest that increasing the complexity of data is generally beneficial to decrease C, but not always. F (right) shows that increasing complexities terms of appearance, light, and background reduces the transfer gap, which implies that these rendering operations are most effective to cover the fine-tuning task that uses real images. However, additional complexity object texture works negatively. We suspect that this occurred because of shortcul learning (Geirhos et al.) 2020). Namely, adding textures to objects makes the recognition problem falsely casier because we can identify objects by textures rather than shapes. Because CNNs prefer to recognize objects by textures (Geirhos et al. 2018) Hermann et al.) (201)), the pre-trained models may overfit to learn the texture features. Without object textures, pre-trained models have to learn the shape features because there is no other clue to distinguish the objects, and the learned features will be useful for real tasks.
5.2 LESSONS TO TRANSFER LEARNING AND SYNTIETIC-TO-REAN GENERALIZATION
Our results suggest the transfer gap is the most crucial factor for successful transfer learning because C determines the maximum utility of pre-training. Large-scale pre-training data can be useless when is large. In contrast,
negligibly small, the law is reduced essentially to n -a which tells that the volume of pre-trat traiiiann data is directly exchanged to the performance of fine-tuning tasks. Our empirical results suggest two strategies for reducing C:: 1) Use bigger models and 2) fill the domain gap in terms of the decision rule and image distribution. For the latter, existing techniques such as domain randomization (Tobin et al., 2017) would be helpful.
5.3 LIMITATIONS OF THIS STUDY
In the experiments, the scale of data is relatively limited (million-scale, not billion).
We only examined ResNet as a network architecture (no Transformers).
Although there are various visual tasks, our study only covers a few of them. Extending our observations other visual tasks such as depth estimation, instance segmentation, and keypoint detection, as well as to other data domains such as language is future work.
The theoretical results assume several conditions that may contradict the actual setting in the experiments. For example, our theory relies on ASGD instead of vanilla SGD. Also, the task types are assumed to be identical for the pre-training and fine-tuning tasks.
In this study, we focus on finding a general rule of transfer learning, rather than improving absolute performance on specific tasks. We used popular vision tasks such as classification and made rendering settings that is not designed to pre-train for the tasks. We expect to observe more performance gain with other enn2ral-friendd) tasks such as optical flow and elaborate rendering settings in future work.
REFERENCES
David Acuna, Guojun Zhang, Mare T Law, and Sanja Fidler. f-domain adversarad learning: Theory and algorithms. arXiv preprint arXiv:2 1344, 2021.

---

Under review as a conference paper at ICLR 2022
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. CoRR, abs/1811.04915. 2018. URL http: buttp://arxe
org/abs 1811. 04918
Shun-ichi Amari, Naotake Fujita, and Shigeru Shinomoto. Four types of learning curves. Neural Computationn 4(4):605-613 1992.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- sine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 322-332, 2019.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06700 2021.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalizeed margin bounds for neural networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9157-9166. 2019.
João Borrego, Atabak Dehban, Rui Figueiredo, Plinio Moreno, Alexandre Bernardino, and José Santos-Victor. Applying domain randomization to synthetic data for object category detection. arXiv preprint 1807 09834, 2018.
A. Caponnetto and E. De Vito. Optimal rates for regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331-368, 2007.
Liang-Chiel Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 055577 2017.
Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M. Alvarez, Zhangyang Wang, and Anima Anandkumar. Contrastive syn-to-real generalization. arXiv preprint arXiv:2 104.0220, 2021.
Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Ole- fir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint arXiv:1911.01911 2019.
Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning of scene structure for synthetic data generation. In European Conference on Computer Vision, pp. 715-733. Springer, 2020.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient Descent Finds Global Minima of Deep Neural Networks. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 1675-1685, 2019.
Simon S Du, Jayanth Koushik, Aarti Singh, and Barnabas Poczos. Hypothesis Transfer Learning via Transformation Functions. In Guyon, U V Luxburg, Bengio, Wallach, R Fergus, S Vishwanathan, and R Gameet (eds.), Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc., 2017. URL f68 2ww.ebbeeeca22332222eeeee
www.ii.......sp.... cc/
paper/2017/Er (0111//1e/3/2222222
Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434, 2020.
Li Fei-Fei, R Fergus, and P Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594-61) 611, 2006. doi: 10.1109/TPAMI.2000 20.60..79.
10

---

Under review as a conference paper at ICLR 2022
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François training of neural networks.
Laviolette, Mario Marchand, and Victor Lempitsky. The journal of machine learning research, (71):2096-20333 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-traineet cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: /81 I. 12231, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 11:665--6733 2020.
Georgios Georgakis, Arsalan Mousavian, Alexander C. Berg, and Jana Kosecka. Synthesizing training data for object detection in indoor scenes. arXiv preprint
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint arXiv:1706.02677 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint arXiv:1811.08883 2018.
Tom Henighan. Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam Mc- Candlish. Scaling laws for autoregressive generative modeling, arXiv preprint arXiv:2010. 14701, 2020.
Katherine L. Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXiv preprint arXiv:/9/ 1.09071, 2019.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint 102.0221 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprini arXiv:1112.00044
2017.
Stefan Hinterstoisser, Olivier Pauly, Hauke Heibel, Martina Marek, and Martin Bokeloh. annotation saved is annotation earned: Using fully synthetic training for object instance detection. arXiv preprint arXiv: N0Xi::192.00967, 2019.
Tomáš Hodañ, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta Sinha, and Brian Guenter. Photorealistic image synthesis for object instance detection. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 66-70. IEEE, 2019.
Tomáš Hodañ, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother. and Jiří Matas. BOP challenge 2020 6D object localization. European Conference on Computer Vision Workshops (ECCVW), 2020.
Minyoung Huh, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning? arXiv preprint 1X0:11608.0861,, 2016.
Mareus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074 2021.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31, pp. 8571-8580. Curran Associates, Inc., 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 00336) 2020.
11

---

Under review as a conference paper at ICLR 2022
Andrej Karpathy. Tesla ai day. 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis drei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.
forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114(13):3521-333 mar 2017. ISSN 1091-6490,
doi: 10.1073/ pnas. 1813511444 WRLhttp::/ www www.wwwcc..b....... m.m. nih. wttpah/.ww//puueee////220000111.
//
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 19t..11122..00 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-V Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016. URL
http:///m....org/pppprrss
242 html
Mehdi Mousavi, Aashis Khanal, and Rolando Estrada. Ai playground: Unreal engine-based data ablation tool for deep learning. International Symposium on Visual Computing, pp. 518-55 112-. Springer, 2020.
Yair Movshovitz-Attias, Takeo Kanade, and Yaser Sheikh. How useful is photo-realistic rendering for visual learning? arXiv preprint
2016.
Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? arXiv preprint 1XXiv:2003.14322 2020.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceedings of The 28th Conference on Learning Theory, pp. 1376-1401, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring Gen- cralization in Deep Learning. In Advances in Neural Information Processing Systems 30, pp. 5947-5956, 2017.
Atsushi Nitanda and Taiji Suzuki. Stochastic gradient descent with exponential convergence rates of expected classification errors. Proceedings of the Twenty-Secne Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1417-1426, 2019.
Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. In International Conference on Learning Representations, 2021.
Atsushi Nitanda, Geoffrey Chinot, and Taiji Suzuki. Gradient descent can learn less over- parameterized two-layer neural networks on classification problems, 2020.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4))838-885 -55,, 1992.
12

---

Under review as a conference paper at ICLR 2022
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6): 2016.
Jonathan Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 12673,, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 13):211-252,
2015, doi: 10.10/7/s 1263-015-0816-y>
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint ar arxiv:0044. 200.:0004./002,, 2020.
Hao Su, Charles R Qi, Yangyan Li, and Leonidas Guibas. Render for enn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2686-2694, 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effecctiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843-852, 2017.
Taiji Suzuki.
Fast generalization error bound of deep learning from a kernel perspective. In Proceedings of the Twenty-Firt International Conference on Artificial Intelligence and Statistics, volume 84, pp. 1397-1406, 2018.
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. arXiv preprint arXiv: 108. 10869, 2021.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. arXiv preprint arXiv:17033 170..0690770777 2017.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracei, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. arXiv preprint arXiv: 184.00664 2018.
Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. arXiv preprint arXiv:2006. 1650, 2020.
Colin Wei and Tengyu Ma. Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin. In International Conference Learning Representations, 2020. URL /ttp:ttpss///ppeeeiieeee wttp://wwwwee/outmeeemm Hd=Heee sd=H_e_ee_ee_eFFee_____ Wr
Jun Yang, Rong Yan. and Alexander G Hauptmann. Cross-Domain Video Concept Detection Using Adaptive Svms. In Proceedings of the 15th ACM International Conference on Multimedia, MM '07, pp. 197, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595937025..
doi: 10.1145/1291233.IT 1291276. /ttlhtttps:/do..oqq/g///
1291233.129122
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv: 05442, 2016.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoc Le. Rethinking pre-training and self telf-training. arXiv preprint arXiv:2006.0688 2020.
13

---

Under review as a conference paper at ICLR 2022
Appendix
A TRAINING DETAILS
A.1 OBJECT DETECTION
We used Faster-RCNN (Ren et al.
2016) with FPN (Lin et al. 2017) object detection models and ResNet Goyal et al. 2017 as a backbone network of Faster- RCNN.
We used the following training procedure: We trained the model using momentum SGD of momentum 0.9 with weight decay of 10 The global batch size was set to 64 when training ResNet18, 18, ResNet34, ResNet50, and ResNet101. The batch size was set to 32 when training ResNet 152 to avoid out-of- memory errors. The batch statistics in batch normalization layers were computed across all GPUs. We used base image size of 640 x 640 in the same way as YOLACT training (Bolya et al. 2019). We used mixed16 training to reduce the memory footprint. We also adopted random horizontal flipping as data augmentation to images. The learning rate was set to 0.02, and we used the cosine decay with a warmup scheme. The warmup length is 120,000 images (3, 750 iterations for Res and 1,875 iterations for other models). As for evaluation, we followed the standard settings COCO dataset (Ren et al. 2016).
We pre-trained the model with 14 14,400,000 images (450,000 iterations for ResNet152 and 225,000 iterations for other models). We used the models that achieved the best mmAP as the initial value of fine-tuning.
We used COCO (Lin et al. 2014) the fine-tuning dataset. We trained the model with 1,440,000 images (45,000 iterations for Re and 22,500 iterations for other models) during fine-tuning.
A.2 SEMANTIC SEGMENTATION
We used DeepLabV3 (Chen et al., 2017) with the softmax cross-entropy loss as the semantic segmentation model and ResNet50 (ooya et al. 2017) as its backbone. The model configuration follows the implementation in torchvisior It should be noted that DeepLab V3 requires dilated ResNet as the backbone, which is not the case in object detection and classification tasks. Even though, the shapes of weight tensors of dilated ResNet50 exactly match those of non-dilated ResNet50; thus we can use the pre-traue pre-trained weights of dilated and non-dilated ResNet50 interchangeably.
The learning procedure is based on the reference implementatiorf of torchvision. We added an auxiliary branch based on FCN (Long et al. 2015) which takes convé of the backbone as the input. In the computation of loss function, the loss for the auxiliary branch is computed in the same way as for the main branch and is added to the overall loss after multiplying by the factor 0.5. The model was trained using momentum SGD of momentum 0.9 with weight decay of 10-4, The global batch size was set to 32. The batch statistics in batch normalization layers were computed across all GPUs. During training, images were first resized so that the length of the shorter edge becomes an integer uniformly chosen from [520 x 0.5, 520 x 2), then horizontally flipped with probability 0.5, finally randomly cropped to 480 x 480. The learning rate (LR)) was decayed according to the polynomial LR schedule of rate 0.9 and initial LR of 0.02. For the parameters of the auxiliary classifier, the LR was multiplied by 10. The evaluation was performed once every 3, 3,125 iterations (almost equivalent to 5 epochs in full ADE20K). In the evaluation, images were resized so that the length of the shorter edge becomes 520.
In pre-training, we trained 125,000 iterations which roughly equals 200 epochs in full ADE20K (Zhou et al. (2077). We used the model that achieved the best mloU as the initial value of fine-tuning. We pre-trained models using our synthetic datasets. When training with them, backgrounds (points at which no foreground objects were present) were also considered to be a separate class in semantic segmentation.
14

---

Under review as a conference paper at ICLR 2022
We used the ADE20K (Zhou et al. 2017) datasets as the fine-tuning target. In fine-tuning, we trained the model for 18,750 iterations, which correspond to 30 epochs of full ADE20K. The metric was mloU score.
A.3 MUL.II-LABEL. CLASSIFICATION
We used ResNet (Goyal et al. 2017) with binary cross-entropy used as the loss function multi-label classification. We used the following training procedure: We trained the model using momentum SGD momentum 0.9 with weight decay 10-4, The batch size was set as 32 per GPU, thus 256 in total. We trained the models for 112,500 iterations. The input size of the images resized to 640 x 640. We adopted random horizontal flipping as data augmentation to images. The learning rate was set 0.1. The cosine decay with warmup scheme was 120,000 images. The evaluation was performed once every 120,000 images. In the evaluation, the image size was the same that used during training, and data augmentation was used. We used the mAP score as the metric.
simply
used. The warmup length
A.4 SINGLE-LABEL CLASSIFICATION
As in multi-label classification, used ResNet. The softmax cross-entropy used as the loss function of the single-label classification. The learning procedure is based on (Goyal et al. 2017). However, used cosine decay for the learning rate scheduling.
A.5 SURFACE NORMAL ESTIMATION
As in semantic segmentation, we used DeepLabV3 (Chen et al. 2017) as the model for surface normal estimation. The model configuration and the training procedure were exactly the same in semantic segmentation, except for the following changes:
Dimension of output channels changed to 3, each of which corresponds to the 3 axes of the normal vector,
Initial LR was changed to 0.04.
Length of pre-training was 200,000 iterations, which corresponds to 100 epochs in our synthetic dataset,
Random flipping was not performed during the data augmentation, and
Loss function was the average of the value, 1.0 n ñi, which was computed for each valid pixel where is the ground-truth normal vector and ñ is the model output (after L2 normalization).
B SYNTHETIC DATA DETAILS
The data generation strategy is based the "on surface sampling" setting in the BoP challenge dataser] In this setting, the sampled objects will be spawned in a cube-shaped room with one point light and one surface light. As the objects to be spawn, we used all the BoP object sets, i.c., LM, T-LESS, ITODD, HB, YCB-V, RU-APC, IC-BIN, IC-MI, TUD-L, and TYO-L There are 173 objects total. After generated random scene (position of objects, lights, etc), we took 10 pictures by 10 different camera poses. This means that, if we have 10K images in total, there are 1K unique scenes, and 9K images are inflated by just changing the camera angle and position.
To control the data complexity, we selected four attributes in the generation strategy and prepared two options for each.
Appearance controls how many objects are generated in a room single scene. For each scene, randomly select ten objects for the multiple setting and one object for the single setting.
Light controls the light sources. In the random setting, the color, height, and strength of lights randomized, In contrast, in the fix setting, they are all fixed.
examples ee haapppleebbopchhhhp /bop_chhaaahhaaoppppo lenge
https://bop. felk. www.welwww.....eeoaaooea evut. htt.kkcwwtt..ccccaaattttttt
15

---

Under review as a conference paper at ICLR 2022
Background controls the texture of the room, i.c., floor and walls. In the random setting, assign a random PBR material from the CCO Texture 10 library, and we selected one carpet texture for the fix setting.
Object texture controls the object set to be used. The BoP object set is consists of several types object sets described above. Among them, T-LESS and ITODD consist of industry- relevant objects and they do not have textures and colors "d For the w/o setting, only used such texture-less objects to be sampled. contrast, we sample all the 173 objects include T-LESS and ITODD in the w/ setting.
We generated eight variations of datasets by changing these attributes, which were used in the experiments of Section 4.5 Figure shows the example of generated images with the value of each attribute.
"https: / enteg.ccomm
LTTLess and ITODD contain 30 and 28 objects, respectively.
16

---

Under review as a conference paper at ICLR 2022
object texture:
appearance: multippe
light: random
background: random
object texture:
appearance:: multiiple
light: random
background: random
object texture:
appearance: multiple
light: random
background: fix
object texture:
appearance: multiple
light: ltx
background: fix
object texture:
appearance: multiple
light: random
background:
object texture:
appearance: multiple
light:
background: fix
object texture: appearance: single
light:
background: fix
object
appearance: single light: background: fix
Figure 7: Example of generated datasets
17

---

Under review as a conference paper at ICLR 2022
pretrain task
objdet
semseg
mulclass
normal
0.8
0.6
0.4
estimate
0.2
0.95
0.90
0.85
0.80
objdet
semseg
sinciass
objjet
senseg
sinclass
objiet
senseg
sinciass
objder
senseg
sinclass
finetune task
Figure 8: The estimated values of the pre-training rate and the transfer gap C in the cross-task setting (as the same as Figure l]. The eπrоr bars present the standard error of the estimates in the least squares.
finetune ratio (%)
pretrain size
4 + 16 54
2 8 32
0.7
0.5
0.5
besk
10a
1025
10a
2043
10
30"
10"
pretrain images
finetune images
Figure 9: Empirical and fitting results for various pre-train and fine-tune data sizes in salasssssinclass.... All curves are fitted using the full law 2 Best viewed in color. Left: Effect pre-training data size (x-axis) for fixed fine-tuning data sizes. Right: Effect of fine-tuning data size (x-axis) for fixed pre-training data sizes.
C ADDITIONAL EXPERIMENTS
C. ESTIMATED PARAMETERS IN THE CROSS-TASK SETTING
Figure X shows the estimated parameters (a, C)) at the experiments described in Section 4.2 Note that the result of no rmal emsseg is omitted because its estimated value highly unstable (the standard deviation is larger than I).
C.2 FULL SCALING LAW COLLECTIVELY RELATES PRE-TRAINING AND FINE-TUNING DATA SIZE
Next, we verify the validity of the full scaling law 2). In the muiclass-tsinclant setting with ResNet-50, we changed the linettunig data size from 2% 64% of the ImageNet 12 We then fitted all results by a single equation (2) to estimate the parameters except the irreducible loss E
12 ImageNNt contains class imbalance problem. If we use 100% of the ImageNet, we cannot provide the same sample size per class. To eliminate the effect of class imbalance, made the sampling ratio to keep the balance up 64% and excluded the of 100%.
18

---

Under review as a conference paper at ICLR 2022
(we assumed & 0 from the preliminary results in Figure 2 The results in Figure 2 show that all empirical test errors are explained remarkably well by Eq. which has only four parameters to fit in this case. The estimated parameters 0.544, 0.322, 0.478, and ó = 41.8.
C.3 LINEARIZED RESULTS
The transfer gap C in ① causes plateau of the scaling law. Conversely, if we subtract the estimated from the results, we must be able to recover the power-law scaling. To confirm this, we subtracted the estimated C' from the empirical errors L of the previous results. Figures 10 12 show the modified version of scaling law fittings. Overall, the empirical enrors behave linearly along with the estimated power-law term D Note that, in mul class-tsmmse and norma sesseeg few points of L become negative after subtracting C, and these points are not depicted.
pretrain task
objdet
semseg
mulclass
normal
0.030
0.010
obidet
0.003
0.001
test error
1e-02
finetune task
1e-05
1e-08
0.100
sinclass
0.010
0.001
10a
1025
10
10
10a
20
10
10
10a
10a5
10*
10
10a
10a
10
10
of pretrain images
Figure 10: The linearized version of Figure
1
mulclass sinciass
objdet objdet
0.10
0.030
model
test error
0.010
r34
550
101
f15
0.003
10a
10a
10
10
10a
10a5
10'
10ah
of pretrain images
Figure 11: The linearized version of Figure
D EMPIRICAL DEPENDENCY BETWEEN a AND D
When C is non-zero, the joint estimation D and a in have an issue of numerical stability due to the small number of observations and noise, which cause high dependence each other. Figure [3 (left) shows the curves of (I) with C' 0.5, where the solid red curve D = 0.5, a = 0.4 and the dashed blue line is D 1, 0.5. We that both curves are almost indistinguishable for
19

---

Under review as a conference paper at ICLR 2022
appearance
objdet objdet
3e-02
multiple
single
1e-02
light
fix
test test error
3e-03
random
background
1e-03
random
object
w/ texture
1000
3000
10000
30000
of pretrain images
+
w/o texture
Figure 12: The linearized version of Figure
2.0
1.5
emrr
1.0
A 1.0
0.7
0.5
0.5
0.0
10
10
10a
0.00
0.25
0.50
0.75
1.01
n
a
Figure 13: Examples of parameter dependency.
(right) shows the actual landscape in terms of a and D of the nonlinear least- squares ob joee -objdett the bright areas indicate the fitting loss is small. We see that there a quadratic-like trajectory in the landscape, which implies the solutions are somehow redundant. Similar landscapes were observed for other tasks (Figure I4].
a large n. Figure 13
pretrain task
objdet
semseg
mulclass
normal
2.0
1.5
objdet
0.5
0.0
2.0
finetune task
1.5
0.5
0.0
2.0
1.5
0.5
0.0
0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
0.0
0.5
1.0
Figure 14: Loss landscapes of curve fittings.
20

---

Under review as a conference paper at ICLR 2022
To avoid this issue, we fixed common D for all the cases and estimated a for each. To determine D. we used the following procedure. First, we prepared two global parameters å, D and set 0.5 as their initial values. Then, we fitted the curves by two equations, Dn + C and Dn + C. and estimated a and D. Next, we computed the median of a and substituted them into á. We did the same for D and D. After few iterations, we got a converged value of D = 0.48, In the experiments, we used the value for D and fixed it.
E DETAILS OF THEORETICAL ANALYSIS
This section gives details of the theoretical discussions given in Section 3.2 For the analysis of
learning and generalization bound, we use the techniques developed recently by Nitanda & Suzuki
22221) There are many works on the generalization of neural networks. To list few, Neyshabur ct al. 2015). Neyshabur et al.] (2017). Bartlett et al. (2017). WWii Maj (2020), and [Suzuki (2018) 8) analyze the generalization of neural networks based complexity bounds. These generalization bounds, however, do not consider an algorithm of learning, such as stochastic gradient descent (SGD). Recently, learning dynamics of neural networks has been analyzed based on Neural Tangent Kernel (NTK) Jacot et al. (2188) and global convergence
wide neural networks has been revealed Allen AAron Zhu et al. ( (2018) Du et al. (2019). Based on the NTK framework, AArora et al. (2019) and Nitanda
(2020) et (2020) showed a generalization bound of the gradient descent learning of neural networks. More recently, Nitanda & Suzuki (202T) foeused the functional space given by NTK and showed that the two-layer neural network with averaged SGD achieves the minimax optimal rate with respect to the function class used in the standard theory of function estimation with kernels. We employ the method of Nitanda & Suzuki (202I), which is the most suitable for our analysis of transfer learning: it enables to examine the dependence on the initial parameter in the learning, and avoids the assumption of a positive margin of eigenvalues used in Arora et al. (2019) and Nitanda et al. (2020).
E. PROBLEM SETTING
In the pre-training, the task is to learn the target function de with To training data (æ,, To where y: (1;)), while in the fine-tuning phase, the network is initialized by the final parameter learned by the pre-training, and the whole parameter is updated in the training. We assume that the target function in the fine-tuning is given by
((x)= = 0o(x) + 21 (x),
and Tj training data is given by (zy, y3 ) with y, = p(x,)) In this setting, the goal of the fine- tuning phase will be learn the additional function Φ mainly. Note that, for simplicity of analysis, we assume noiseless training data, i.e., we assume the supervised signal y, is given by deterministic function of Zy, but extension to more general cases not difficult as discussed in Nitanda Suzuki (2021). In the analysis, the data are assumed to satisfy x Rd, ||=|| 1 and y E 00, 1]. The distribution of the input data x is denoted by Px, and the same for the pre-training and fine-tuning,
For tractable theoretical analysis, we consider a simple cealar-valued two-layer neural network model with M hidden units:
(e(
1
Σ (b, x).
(5)
VM
We omit the bias term, but with obvious modification, it is not difficult to include it (see Nitanda & Suzuki (02I)).
As in Nitanda & Suzuki (021), we consider the averaged stochastic gradient descent (ASGD), where one training sample is given at every time step for the stochastic gradient descent as in online learning, and all the parameters in the time course are averaged after the final time step for the inference, that is, after proceeding up to prescribed T (T' = To or Ti) time steps, the parameter to be used in the inference is given by
Σ 000
(6)
The final network uses this averaged parameter, i.e., the final network is given by (g (x).
21

---

Under review as a conference paper at ICLR 2022
(0) b.)) For the pre-trainig, each ((0) is The parameter is initialized O (a (0) 8 ..... MM
independently given by the uniform distribution on the unit sphere. As in Nitanda & Suzuki (2021).
a(0) are initilized as or -1 so that De( = O. As explained before, the initial parameter of the 5
fine-tuning is the same as the averaged parameter of the pre-training pre The objective function to minimize for the pre-training and fine-tuning is given by the following regularized empirical risk:
1 - (vs (( = 2 =
(o,)) + {1a a0 1 + Σ 11, - -,, 1,
(7)
where À is the regularization coefficient, which is a hyperparameter. The values of À in the pre-training and fine-tuning can be different, and denoted by λg and A1, respectively. Note that the regularization in Eq. 7 is not the most common fg-regularization, where ||0112 is used for the regularization. When applied fine-tuning, however, the above regularization can interpreted as elastic weight consolidation (Kirkpatrick et al. 2017), which prevents forgetting the pre-trained parameters.
We consider online learning, in which every step (1 S - 1), one datum z; is sampled from px independently, and (zt, 36) (or (ž,, i4) in pre-training) is used to update the parameter according to the gradient descent:
= - (
(8)
00
where 1) is learning rate. More explicitly,
(((+)) -af = (aa) - (0))
(Age (1) xx).
v VM
b((+1) b(0) = (1 - n2)( - 610)
"
(gom ve)ayo (b, (1
x x1)
(9)
VM
E.2 NEURAL TANGENT KERNEL
In the theoretical analysis, the neural tangent kernel (NTK, Jacot et al. 2018) is used for approximat- ing the dynamics of ASGD by linear functional recursion the corresponding function space. The NTK of this model is given by
(ar, x')
Egor (O (6)60)))))))))))))) x²)] + t Egoon (d' (b ()) x))''b x')].
(10)
The positive definite kernel lo naturally defines a reproducing kernel Hilbert space (RKHS), which denoted by Hos-
The integral operator Eoe (pxx is defined by
Exf : /
(=o((s))((E)dxx()))
(1I)
is known that Exs admits eigendecomposition
Ex 1/s = 73.
(12)
where V. is an eigenvector with || (2(x)) = and 71 2 72 2 > 0 are eigenvalues in descending order. Mercer's theorem tells that h2o has expansion:
Koo (x, x') = (z))),)
where the convergence is understood as in I.2 for general, and absolutely and uniformly if px is a uniform distribution on a compact set.
E.3
ASSUMPTIONS
For theoretical analysis, we make the following assumptions. For an operator Σ, the range of is denoted by (E)).
22

---

Under review as a conference paper at ICLR 2022
(A1) The activation function is differentiable up to the second order, and there exists C >> 0 such that ||o" I C.
||o' | S 2, and [o(4)] S 1++ + // for Vu E R.
(A2) supp(ex) C {ar R | |||||| S 1] and y 1,
(A3) There exist 1/2 - ro < 1 such that do E R(2)) and Φ1 E R(2))
(A4)
There exists { > 1 such that 71 = S)
As in Nitanda & Suzuki (022 Assumption (A1) assumes that the activation a is differentiable in this paper. In Nitanda & Suzuki (202T), however, they have developed a theory how extend the results to the case of ReL. by approximating it with a smooth function. It is well known that Assumption (A4) specifies the complexity of the hypothesis class H.x (Caponnetto & De Vito 2007); a faster eigen-decay (large {) implies the small complexity of the class. The assumption (A3) controls the smoothness of the target functions do. 01 In fact, the functions are included in Hos since 6i E R(EE C Hoc When a function ʃ has the expansion f = Ee aeth.. the assumption f E R(EE))) means Og = o(E-Er-1/2) A function with a larger is smoother, which is easier to learn. is known (Caponnetto
De Vito. 2007; Nitanda & Suzuki, (021) that & and are the two basic parameters to control the convergence rate of generalization attained by kernel regression for a large sample size. Under the assumptions (A3) and (A4), given N i.i.d. training data (X,, Y) with X, ~ px and Y, (oo(),) + E, with additive noise E; N N(0, r²), the kernel ridge regression Ω with the regularization parameter X = N- achieves the generalization > - Po Polli;=(p)) (xx))) = O(N (222/())()))))) for any function Po with yn E R(E"), and it is known this rate is optimal.
In the sequel, when o E R())) and @ = M" Mi, we write | : ||
E.4 GENERALIZATION BOUND
The dynamical behavior of pre-training can be discussed exactly in the setting of Nitanda & Suzuki
(2021). ). Let dg be the result of pre-training, i.e., P0 == Iottoo where (TT)) is the averaged parameter prd
pre
by ASGD. By optimizing the regularization parameter Xo. Corollary 1 in Nitanda & Suzuki (0021) )
shows that for sufficiently large To, with a choice of λy =To (11o0++)
2rp4
0l do l2)((x)) E + cTo
2nge++
(1 + "1>0""00 ll (())))
(13)
with high probability, where is universal constant and Æ can be arbitrarily small for a large
arng
M. As discussed in Section F.3 it is known (Caponnetto & De Vito 2007) that the rate To achieves the minimax optimal rate with respect to To over the class specified by ro and g.
2ru6+7
In theettuning the initial parameter is given by (()) = O (Tb) t I,...,TI.
pre and ASGD is applied with (x,, ve) for
By extending Theorem 1 in Nitanda & Suzuki )021), we can derive generalization bound in the following theorem. Recall that the regularization coefficient and learning rate of fine-tuning are denoted by a1 and 1/1, respectively.
Theorem 2. Suppose Assumptions (A1)-(33 hold. After pre-training that gives Eq. (13. fine- tune the network by Eq. (9) with learning rate qn and regularization coefficient X that satisfy [o 2 A1 > 0 and 466 + X1 mm S 1. Then, for any & > 0, E (0, 1), and Ti E N, there exists Mo E N such that for any M 2 Mo, the following bound holds with probability at least 1 ö over
the random initiailzation of pre- -re-training:
Ellggin) - <2lli
< + 20
2rd [>
do (ooll12 ((xx((x) + ci c1 " oi "1"""E"""
(AT
E Elão 20 p0ll2 12 (1 1
do ooll + 1pm-n1louu ( + 11 11
Ti 1
C3
{>- E1 - doll (1 + اΣ ro Φn (()))) + 2 1100111001 + 2
(Ti + +1)2m
C4
(1 + 11.112.0 + ) Eoo (oo + (11)
(14)
T ++ 1
23

---

Under review as a conference paper at ICLR 2022
where du is the result of pre-training and c; (i = 0, 1,3,44 are universal constants.
The term e arbitrarily small for a large value of M, i.e., wide network. The proof of Theorem 2 will be given in Section E.6
E.5 ANALYSIS OF CONVERGENCE RATES
We consider the rates of the generalization bound for ( -llEa((x) with respect to Ti. As
typical cases, assume A1 and 771 O(1) Ti The dominant terms in Eq. (4) may vary according to the configurations λ and 17 with respect to T1. We will show the rates in some settings that are relevant transfer learning.
First, note that under Assumption (A4), the factor tr (E + X11)= is given by Caponnetto & De Vito 2007)
Tr
+ \1 1 = 0
(15)
By neglecting e, the terms in Eq. 14 thus have the following rates:
(a0) A2ro (al) Xpr
(b) (T (c) TT 2 TT
(e) T1 2 1/1 x1 Ro: (0) T
(g) T (1)
(16)
Here Rg = El do [xx) is of constant rate with respect to Ti, but explicitly shown for the
later use.
Since  - and rı 2 1/2, the terms (c) and (d) are of smaller rate than (b). Likewise, (f) and (g)
are smaller than (e). The candidates of dominant terms are thus (a0), (al), (b), (e), (g), and (h).
E.5. LARGE REGULARIZATION CORFFICIENT
In transfer learning, it is reasonable to use strong regularization in fine-tuning, which encourages the parameters stay close to the initial value that is obtained in the pre-training. In this subsection,
consider the case where A₁ is larger than T, 2riGHT which would be the optimal rate if the network
was trained with random initialization (see Nitanda & Suzuki (2021)) À = Ti 2r16+1 was taken, it is easy to see that the influence of pre-training would not appear explicitly in the convergence rate. In the sequel, we write F < G(T;) if there are and T such that F < aG((i) for all Ti 2 T In this notation, assume
A1 4>1 T SगटाT
(17)
The rate X T TIा is given by equating the rates of (al) and (h). Therefore, under the assumption of Eq. (17), the rate (al) is larger than (h), and thus it suffices to consider (a0), (al), (b) and (e) the candidates of dominant terms. Note that, if X - the terms (a0) and (al) decrease, while (b) and (e) increase to infinity.
We will diseuss below the possible cases of dominant terms under the assumption Eq. (17). In the analysis, although the error of the pre-training Ro is regarded as a constant, we yet wish to consider the dependence of the fine-tuning result on Ro. We thus set the regularization coefficient λ dependent on Ro, and show that in all the cases, the generalization bound takes the form
Ell Ellgg - * Se C
(18)
where y >> 0 is a constant. As Ro S d + AATO from Eq. 13 the factor RS can be bounded from above as
R% $ c aT
for large To. As result, obtain
Eliggn xl( Se+
AI, C
(19)
24

---

Under review as a conference paper at ICLR 2022
where C, A constants. As will see, the exponents and B depend ro.71: and m/1 Eq. (19) accords with the bound in Theorem
In the sequel, we use Ç 2 0 for the learning rate such that 1/ = =T
Case I: Small learning rate Tif 1. In this case, (e) > (b). We equate (a0) (al) with (e) to obtain A for achieving the best possible upper bound of the two terms.
(I-A) ro 2 ri. Since Å1 + 0 for T1 co, (al) is larger than (a0). By equating (al) and (e), we find that the best choice of λ is
A = 0 T.)
We further consider
\ = +1 RS
for dependence on Ro. To determine v, we assume that Ro is a small value, and consider the rate of (al) and (e) with respect to Ro after plugging the above A them. By equating the
2ry(1-2) 'I Ro and (c) Ti
2r
rates of (al) T;
'
Ry- the best possible rate of Ro is attained by is thus
= 1/( + 2). The dominant rate of Eq.
16
(1-
Ti
Rg
(20)
attained by
\ Ti
R
(21)
We need to identify the conditions on Ç1 meet the requirements. The condition A₁ + 0 equivalent
to < 1. There are two other conditions: a >> Tj कांदा and Ti An < 1. Given Ro is of constant rate, the former is equivalent to 21
&
1-5 which results in l+r1
S
rıg + +1 1- 2r +1
Ç
The latter condition equivalent to
2g
1-( S 0. which is
rl
2 2r1 +1
is not difficult to see
T1
EI + - E
2r +
2r +1
for & > 1. As a result, the condition on Ç is
r1
<<<
(22)
2r1
If 1 = TT is taken to satisfy this condition, the optimal rate of λ is given by Eq. 21 Finally, the resulting generalization bound is given by
Ellggnn - ellip(e S + eT,
R
(23)
(I-B) ri > rot In this case, (a0) is of larger rate than (al). By similar argument to (I-A), with the rate
\ Tj Ry
(24)
The generalization bound is given by
2 3 1
Ellggon plliz((x)) S + cT
R
no
(25)
The condition on Ç is
(2rl - rroe + 1 E
max
<5<1.
(26)
2ro +
2r +1
25

---

Under review as a conference paper at ICLR 2022
To: Pi
= T
Bound
2ng
(11(1
To 2 71
2ri+1 S 5
r1
6 c+ To
2rg
T
1111
20p
70
20p(1-4)
TO < 71
max
r0 2
(2r 12m-
2riE+1
5c<1
E a
c+ To
[
10+1
70 2/11 71
( - 2r1+1
+ d
c+ To
T
2rpé 2m
2rg 2rp I
ro
m1 Tg 2
E-1
< 2
d'
c To
Zng I
2ng 11
T
Table 1: Generalization bounds in various conditions.
(Case I): large learning rate Ti Tixmn >> 1. Next, consider the case where the learning rate m/ large so that Ti Anri > 1, which includes the constant 7n Under this condition, (b) is of larger rate than (c).
(II-A) ro 2 rı. In this case, (al) is of larger rate than (a0). A similar argument to (I-A) provides
A 1= T
Rd 7
(27)
and the generalization bound is given by
2+1
E B d0 (0ll) Se cT
2
Ro
(28)
The conditions are À > Ti मगररा and Ti An >> 1. The former condition always holds for g > 1,
and the latter is equivalent to Ç S 2I.
r
The resulting condition on is
r1
0<<s S 2r
(29)
(II-B) ri > ro: In this case, (a0) is of larger rate. With
A1 -1=T; Ti 201 Ro 200+
(30)
the generalization bound is given by
2ng Ira
2rg 250
E D - 2o (oll?2(0x) S cf'
R
(31)
Ti 27,441
The conditions A > Thus, we require
and Ti Tini >> are respectively ri S ro +
{-1
and (
<
m
25
2
0 0<55 To
1-1
To <r < To +
(32)
2ro +1'
2
In summary, the generalization bounds in various conditions are summarized in Table |
E.6 PROOF THEOREM
The proof of Theorem is based on the application of the theory in Nitanda & Suzuki 2001 the fine-tuning phase, adapting the initialization given by the result of pre-training do.
In the sequel, we focus on the fine-tuning with Tj samples with y (p(x)). Recall that
(xxx)
Φ0(x)) tb (2:)
26

---

Under review as a conference paper at ICLR 2022
E.6.1 REFERENCE ASGD ON RKHS
We surrogate sequence of functions in RKHS for the proof. Let kas be the random feature approximation of the TNK lioos i.C.,
I'A' M Σ (( M
M
1
(2, x')
(33)
M
where (b,) is i.i.d. random sample from the uniform distribution on the unit sphere S The associated RKHS is denoted by HM-
reference ASGD is defined by the following update rule of functions in the RKHS HM:
((++)) (1 (1-nn)) 7>)g(0) ( (z,) yr (Mr) zz),
. 0..,. 1)
(34)
with the initialization given by g00) = H.. The average taken the final step:
T
(
g
(35)
By considering continual learning pre-training and fine-tune, a slight modification of (Nitanda Suzukii [021] Propososion A) derives the following proposition.
Proposition 3. Assume (Al) and (A2). Suppose that X < 1. Then for any T1 E N and e > 0,
there is M. eT,, E N such that during the fine-tuning learning
(g(h) 90 ([x)) S,,
(36)
holds for any M 2 M. and 0 t Ti.
This proposition shows that, if we use a very wide network, the learning of ASGD in the parameter space can be approximated by the reference ASGD on the RKHS with negligible error.
The generalization bound will be given by the following decomposition:
I1 + xl1(2(xxx) S 2 (g 5 g (12x((e))e)))) + +l =l12
(37)
in which the first term of the right hand side is bounded by Proposition 3 with an arbitrary small value e for large M. The second term will be discussed in the next subsection.
E.6.2 CONVERGENCE RATES OF REFERENCE ASGD
In this section, write À and n for a and mh for simplicity. The covariance operators Exo and Es for Hox and H.M respectively, are defined by
Ex Fipx (i X) 60 lioo (:, X)"), Σx = Epx (iar X) 2 kas X)"),
(38)
where * denotes the adjoint; equivalently.
Lxef f (
Eash- /ath/baal:))ee ((
for f Hoo, h HM. The regularized target functions @.MA 0, 1) are defined by
(6) MMA (EM -T (i (0=0,1)
(39)
x is defined similarly with Eoo. Note that PMMA (EM + Al) -1)--1 = (0)
(i)
10) + O
First, we decompose (T](Ti) - - (p(i by
lg (h) -1112A(xx) = (l) PMM.A + (0) + 010 - do - - 121122xxx
M.A
< <3115 (T) - -PMAll - 11) doll(2(xx) + 31 - -11122
M.A
27

---

Under review as a conference paper at ICLR 2022
The second and third terms are known to have bound, with high probability, (Nitanda & Suzuki 2021 Propositions and D)
1 - xxll(2(ex) S S + 11== di Allli
0,));
(40)
where & is arbitrarily small for large M. We have thus, with high probability,
(g (T)) -llia((x)) - -ma +32 20 13 pull? +32 E3 mi Φ1
(xx)
(41)
As shown in Nitanda & Suzuki 2021) the term (g PM Al ()x)))x) can be analyzed by the bias and noise terms of the stochastic recursion RKHS Eq. (34), which is rewritten
((++) (1 - g/)l)g (t) myk (1
(42)
where
H; == karl xx2) @ k.((, xx)", a one-sample estimate of M.. By subtracting PMMA from both hand sides of Eq. 422 we have
g (1+1) PM, = (T nHH - ()))( (MNA) + Sr,
(43)
where
B. = Tuckis x1) - (He + E + AT) 'T EMMPM
the zero mean noise term. Using this recursive formula, Nitanda & Suzuki (202) derives bound: lg (T) (MM.Ill2(( S
C1
|1(EM + X1) (1 (0) - M,,,)ll
Ti +
C2
|1 A1) - (0 pas,>)) |1
(Tr + 1)2
+ (1 + ||ll + 24|1=~ p Tr|EM ) + +1)
(44)
To bound this expression further, use Proposition B in Nitanda & Suzuki (202)
[[E AT) AT) (() Ar, (M,lll22(xx)) 21loi
(45)
Then, using the decomposition (0) Pad. (Ōu -u) + (0o (10.)) with high probability,
Ms
(1)
obtain that,
11 + X1) (1/2 - (1(
s 3 (2' 1/2 20) ( + (2ir + A1) 112
+ 3 - M1.11
A 1100 coll doll (xxx)
3
+
3 x 1 ||00|| (o
3
+EM + 6110111
(46)
where we use Eq. (40) for the second and third terms and Eq. (45 for the fourth term in the last inequality. Similarly, with high probability, we have
[ + AI) (g (MM,))li
3|1 (EM > ( (1)) + 4l ATT
(E -11-1110 (1)
0
M. (1.x11)2((xx)
1 2o 20112.00 l122(px)
x2
3
+
X2
3 2ro | ( + EM 1101 1101
6
(47)
It is also known Nitanda & Suzuki
202
Proposition B) that, for X S EEoll
Tr|EM (EM + AI) 1 (Eo + AT)
(48)
Combining Eqs.
41
(46
and
48
obtain the assertion of the theorem.
28


Reviews:
[{"review_id": "yY_nG0FJdg", "reviewer": "Reviewer_EiPP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenght\n\n- The paper proposes a scaling law for transfer learning that takes into account the size of the pretraining dataset.\n- The experimental analysis is very good and cover very important tasks for the CV community  (e.g.  object detecction, segmentation, normal estimation)\n- The authors provide some justification of the derived law.\n\n> Updated after rebuttal.\n\nWeakness.\n- Major Concern #1. I consider the sim2real aspect of this paper to be a bit overstated. In my understanding, the dervied scaling law can be used in any transfer learning  setting that involves pretraining and finetuning. Why is it limited to sim2real or what does it make it specific to the sim2real scenario? I understand the experiments are conducted with a synthetic source dataset but this shouldnt be the only reason.\n\n>After rebuttal. This concern  remains. I agree of the tremendous importance of the sim2real transfer, but the derivations presented here are not specific to the sim2real scenario. The narrowing of the scope seems still a bit artificial and not necessary. \n\n- Major Concern #2. Several times in the paper the authors refers to the coeficient C as the transfer gap (introduced in sec 3.3 without explanation). It is further mentioned than this is related to the rendering settings (also in sec 3.3). I didnt find anywhere any justification or intuition for this relationship. Just stating that C is the transfer gap seems a bit artifical, same for saying that this is related to the rendering settings. I could imagine two real datasets that are dissimilar (e.g. different camera parameters or black-white vs color) and this C may also be big (no rendering involved).\n\n>After rebuttal. The concern partially remains. I agree that C is induced by the dissimilarity between pre-training and fine-tuning tasks, but attributing that to the rendering setting without further justification is not proper. There are many component that may be having a role in what  “ dissimilarity between pre-training and fine-tuning tasks” is. Without further justification, the claim seems very abstract.   \n\n- I am not completely sure what can we get from section 3.2 and Theorem 1 since the law was empirically determined, and again, in my understanding, nothing in this section is specific to the sim2real scenario.  Is there anything we can use from this theorem to justify C is the transfer gap and it is related to the rendering settings? This section is also introduced in a very fast mode with most of the material defer to the suppl. It also introduces a lot of notation in a very fast pace that could be confusing. I recommend the authors to explain better the connection between this and eq (2). Particularly with respect to the exponents \\alpha and \\beta. Is there anything else that we can get out of this theorem which was not  obtained experimentally? I also recomend the authors to add a regression task to their experimental setting since this is the one used to motivate this section.\n\n> After rebuttal. This concern remains, I agree it is not trivial, but  in my opinion this is required to claim the connection between this section and the sim2real scenario. The paper title reads \"A Scaling Law for Syn-to-Real Transfer\" and this section goal seems to theoretically suport this claim, however,  in my understanding, nothing in this section is specific to the sim2real scenario. I am still not getting much  from section 3.2 and Theorem 1. \n\n- More intuition on the exponents on the derived law (eq 2) should be provided \\alpha, \\betta \\delta. We know that they are coeficient for sure, and that they are bounded below, but what are the pretraining and finetuning rates? What's the intuitive and/or theoretical interpretation of them?\n> After rebuttal. Solved.\n\n- I think the paper will also benefit from more discussion in section 3.1. For example, could the authors discussed other choices of g(n), showed the limitation and motivate why the chosen one is the most resonable one. Is there a way to \"strongly connect\" the choice of g(n) to the theoretical results from 3.2? Also will a better parametrization here help with the numerical inestability issues.\n> After rebuttal. Solved.\n\n- Major Concern #3 . I  feel what we can get out of this paper may be a bit limited. The idea of the transfer gap(or disimilarity between the source and target domains) being very important for transfer learning is not new. This has been extensively studied and also shown rigorously and experimental in more general settings. Similarly, it is well known than usually bigger models generalize better.\n\n>After rebuttal. Partially remains. I agree on the importance but my concern is that the main take away here might be a bit limited. We know from before (without having the scaling law) that key for successful  transfer learning is the dissimilarity between source vs target. This is not new. ", "summary_of_the_paper": "This paper proposes a scaling law for transfer learning that takes into account  the size of both the pretraining and finetuning datasets. They showed the parameters of the scaling law can be  estimated and this can be used to approximate the performance of the model given the amount of pretraining data. The authors experiments are focused on the scenario where the pretranining dataset is synthetically generated.", "summary_of_the_review": "Overall, I consider the paper contains some nice ideas and the experimental analysis to be interesting although  I  feel what we can get out of this paper may be a bit limited. I also believe that presentation of the work should be improved and the work would benefit from a major revision. I also recomend the authors to focus on the more general transfer learning scenario since the narrowing of the scope to a \"sim-to-real scaling law\" may seem a bit artificial.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ajMSXGSCKml", "reviewer": "Reviewer_9onL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper explores an interesting relationship between pre-training dataset size and final accuracy on a downstream task after finetuning.\n\nWeaknesses:\n- The scaling law is derived by empirical observation.\n- The fine-tuning dataset size is kept fixed in all experiments. This is, in my opinion, an oversimplification of the scaling law.\n- The paper provides is no measurable quantitative evaluation of its results. The plots just show that the predicted points are \"close\" to the fitted line. How will the method compare with a different scaling law if these results are not provided?\n- Applicability might be limited due to the possibility of just increasing the synthetic training set size until desired performances are reached.\n\nOther:\n- Figure 2 is not a log-log plot, as stated in sec. 3.1.", "summary_of_the_paper": "The paper describes a scaling law for approximating the number of samples needed for pre-training to reach a certain performance on a downstream task. The experimental protocol evaluates the method by fitting the scaling law's parameters to a set of observation with least squares, though no quantitative evaluation is performed.", "summary_of_the_review": "The paper tries to answer an interesting question (though practical interest might be limited) in estimating the number of samples needed for pre-training, however there are many simplifications (as also stated by the authors) and the empirical evaluations, on which the claim is based, lack measurable results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "WvPRcYWWrbQ", "reviewer": "Reviewer_Ltke", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Idea/Method\n-----------------\nStudying the existence of a scaling law for syn-to-real as a function of irreducible domain discrepancy error and data size is important and has great practical benefits. A well fit scaling law would help make informed decisions about whether pre-training data needs to be scaled or diversified for downstream performance improvements. The assumptions leading to the inductive scaling law formulation are sound and it empirically fits real transfer errors very well. \n\nI wasn't able to tell why the authors decided to focus only on syn-to-real transfer since the theoretical derivation does not distinguish between real-to-real or syn-to-real transfer. This is a key question that goes completely undiscussed. Were there negative results with real-to-real transfer? Was syn-to-real chosen so pre-training data complexity could be controlled? \n\nExperiments\n-----------------\nOverall, I feel the experiments are minimal to justify the paper's claims\n- In Fig. 1 there is no extrapolation. The curve is fit to all the points shown. A key feature of a useful scaling law is the ability to extrapolate findings. Fig. 5 and Section 4.4. show results with larger pre-training data sizes but it is unclear whether the curves are extrapolated from Fig. 1 or were the result of fitting parameters again to the new data. In both cases, it'd also be good to compare the estimated values on extrapolated points to show whether the fit curve generalizes (and also show how many points are needed to get a good generalizable fit)\n- The authors use different error metrics for different tasks to fit the scaling law. Why does this work? This is especially counterintuitive for object detection, where 1 - mean average precision is used as the error metric. Is there anything beyond empirical justification for why this works?\n- The error rates for all tasks are in the high 80s and 90s. These are significantly far away from the current best syn-to-real generalization results. The irreducible error term C is estimated to be in the high 80s, which is nowhere close to realistic settings that a practitioner would be in. It would be interesting if the authors did one experiment with a more realistic syn-to-real setting, for example doing GTA-V -> ADE20K/Cityscapes semantic segmentation\n- In all cases, the authors use only 1% of real data, so that the effect of pre-training can be shown. This is again an artificial constraint that could be relieved by using more realistic synthetic data (such as GTA-V data), where the effect of pre-training is visible while using much higher % of real data\n\nRelated Work\n-------------------\nRelated work is generally well written. In one part, the authors say, \"Theoretical analysis was also attempted [Hutter, 2021][Bahri et al., 2021]\". It'd be nice to discuss the differences there instead of just mentioning the papers. \n\nDomain Adversarial Learning (https://arxiv.org/abs/1505.07818), a recent follow-up called f-Domain Adversarial Learning (https://arxiv.org/abs/2106.11344) and papers of similar flavour are quite related. They provide bounds on target domain risk as a function of source domain risk and domain discrepancy. They also provide practical algorithms for syn-to-real training. These works are missing in the related work discussion and I believe they warrant a discussion.\n\nWriting\n---------\nOverall, the paper is well written. Minor typos here and there, such as in the 3rd line of the last para of related work. Figure 2. caption is opposite of the figure. ", "summary_of_the_paper": "This paper postulates a scaling law for pre-training and transfer learning via fine-tuning. They empirically postulate a law and also theoretically show that in a simpler setting with assumptions, a similar law should hold. The paper is written and experiments are done in a synthetic to real transfer setting. Experiments are done by pre-training on a synthetic dataset made using BlenderProc and transferring to real data ranging classification, object detection and semantic segmentation. They find that parameters of their scaling law can be estimated to find very good fits to empirical error across all the different transfer tasks.", "summary_of_the_review": "Overall, I am positive about this paper. The method is well motivated and justified. I believe the experiments could be much stronger, especially since the experiments with real datasets would still be considered toy experiments due to the choice of pre-training data and % of real data used. It is hard to say whether these results would generalize, but I believe they warrant a discussion by the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses an empirical scaling law in terms of samples needed for pretraining for effective downstream transfer. The reviewers liked the premise but had major concerns with the evaluation and some clarifications about empirical choices made. The paper initially received reviews tending towards rejection. The authors provided a thoughtful rebuttal that addressed some of the questions. The paper was discussed heavily and all the reviewers updated their reviews in the post-rebuttal phase. In conclusion, all reviewers still believed that their concerns regarding empirical evaluation like why evaluate only sim2real transfer, etc. still stand. AC agrees with the reviewers' consensus and encourages the authors to take the feedback into account for future submissions.
Decision: Reject