A Scaling Law for Syn-to-Real Transfer: How Much Is Your Pre-training Effective?

Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks . The most significant advantage of using synthetic images is that the ground-truth labels are automatically available , enabling unlimited expansion of the data size without human cost . However , synthetic data may have a huge domain gap , in which case increasing the data size does not improve the performance . How can we know that ? In this study , we derive a simple scaling law that predicts the performance from the amount of pre-training data . By estimating the parameters of the law , we can judge whether we should increase the data or change the setting of image synthesis . Further , we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings . We empirically validated our scaling law on various experimental settings of benchmark tasks , model sizes , and complexities of synthetic images . 1 INTRODUCTION . The success of deep learning relies on the availability of large data . If the target task provides limited data , the framework of transfer learning is preferably employed . A typical scenario of transfer learning is to pre-train a model for a similar or even different task and fine-tune the model for the target task . However , the limitation of labeled data has been the main bottleneck of supervised pre-training . While there have been significant advances in the representation capability of the models and computational capabilities of the hardware , the size and the diversity of the baseline dataset have not been growing as fast ( Sun et al. , 2017 ) . This is partially because of the sheer physical difficulty of collecting large datasets from real environments ( e.g. , the cost of human annotation ) . In computer vision , synthetic-to-real ( syn2real ) transfer is a promising strategy that has been attracting attention ( Su et al. , 2015 ; Movshovitz-Attias et al. , 2016 ; Georgakis et al. , 2017 ; Tremblay et al. , 2018 ; Hinterstoisser et al. , 2019 ; Borrego et al. , 2018 ; Chen et al. , 2021 ) . In syn2real , images used for pre-training are synthesized to improve the performance on real vision tasks . By combining various conditions , such as 3D models , textures , light conditions , and camera poses , we can synthesize an infinite number of images with ground-truth annotations . Syn2real transfer has already been applied in some real-world applications . Teed & Deng ( 2021 ) proposed a simultaneous localization and mapping ( SLAM ) system that was trained only with synthetic data and demonstrated state-of-the-art performance . The object detection networks for autonomous driving developed by Tesla was trained with 370 million images generated by simulation ( Karpathy , 2021 ) . The performance of syn2real transfer depends on the similarity between synthetic and real data . In general , the more similar they are , the stronger the effect of pre-training will be . On the contrary , if there is a significant gap , increasing the number of synthetic data may be completely useless , in which case we waste time and computational resources . A distinctive feature of syn2real is that we can control the process of generating data by ourselves . If a considerable gap exists , we can try to regenerate the data with a different setting . But how do we know that ? More specifically , in a standard learning setting without transfer , a “ power law ” -like relationship called a scaling law often holds between data size and generalization errors ( Rosenfeld et al. , 2019 ; Kaplan et al. , 2020 ) . Is there such a rule for pre-training ? In this study , we find that the generalization error on fine-tuning is explained by a simple scaling law , test error ' Dn−α + C , ( 1 ) where coefficient D > 0 and exponent α > 0 describe the convergence speed of pre-training , and constant C ≥ 0 determines the lower limit of the error . We refer to α as pre-training rate and C as transfer gap . We can predict how large the pre-training data should be to achieve the desired accuracy by estimating the parameters α , C from the empirical results . Additionally , we analyze the dynamics of transfer learning using the recent theoretical results based on the neural tangent kernel ( Nitanda & Suzuki , 2021 ) and confirm that the above law agrees with the theoretical analysis . We empirically validated our scaling law on various experimental settings of benchmark tasks , model sizes , and complexities of synthetic images . Our contributions are summarized as follows . • From empirical results and theoretical analysis , we elicit a law that describes how generalization scales in terms of data sizes on pre-training and fine-tuning . • We confirm that the derived law explains the empirical results for various settings in terms of pre-training/fine-tuning tasks , model size , and data complexity ( e.g. , Figure 1 ) . Furthermore , we demonstrate that we can use the estimated parameters in our scaling law to assess how much improvement we can expect from the pre-training procedure based on synthetic data . • We theoretically derive a generalization bound for a general transfer learning setting and confirm its agreement with our empirical findings . 2 RELATED WORK . Supervised pre-training for visual tasks Many empirical studies show that the performance at a fine-tuning task scales with pre-training data ( and model ) size . For example , Huh et al . ( 2016 ) studied the scaling behavior on ImageNet pre-trained models . Beyond ImageNet , Sun et al . ( 2017 ) studied the effect of pre-training with pseudo-labeled large-scale data and found a logarithmic scaling behavior . Similar results were observed by Kolesnikov et al . ( 2019 ) . Syn2real transfer The utility of synthetic images as supervised data for computer vision tasks has been continuously studied by many researchers ( Su et al. , 2015 ; Movshovitz-Attias et al. , 2016 ; Georgakis et al. , 2017 ; Tremblay et al. , 2018 ; Hinterstoisser et al. , 2019 ; Borrego et al. , 2018 ; Chen et al. , 2021 ; Newell & Deng , 2020 ; Devaranjan et al. , 2020 ; Mousavi et al. , 2020 ; Hodaň et al. , 2019 ) . These studies found positive evidence that using synthetic images is helpful to the fine-tuning task . In addition , they demonstrated how data complexity , induced by e.g. , light randomization , affects the final performance . For example , Newell & Deng ( 2020 ) investigated how the recent self-supervised methods perform well as a pre-training task to improve the performance of downstream tasks . In this paper , following this line of research , we quantify the effects under the lens of the scaling law ( 1 ) . Neural scaling laws The scaling behavior of generalization error , including some theoretical works ( e.g. , Amari et al. , 1992 ) , has been studied extensively . For modern neural networks , Hestness et al . ( 2017 ) empirically observed the power-law behavior of generalization for language , image , and speech domains with respect to the training size . Rosenfeld et al . ( 2019 ) constructed a predictive form for the power-law in terms of data and model sizes . Kaplan et al . ( 2020 ) pushed forward this direction in the language domain , describing that the generalization of transformers obeys the power law in terms of a compute budget in addition to data and model sizes . Since then , similar scaling laws have been discovered in other data domains ( Henighan et al. , 2020 ) . Several authors have also attempted theoretical analysis . Hutter ( 2021 ) analyzed a simple class of models that exhibits a power-law n−β in terms of data size n with arbitrary β > 0 . Bahri et al . ( 2021 ) addressed power laws under four regimes for model and data size . Note that these theoretical studies , unlike ours , are concerned with scaling laws in a non-transfer setting . Hernandez et al . ( 2021 ) studied the scaling laws for general transfer learning , which is the most relevant to this study . A key difference is that they focused on fine-tuning data size as a scaling factor , while we focus on pre-training data size . Further , they found scaling laws in terms of the transferred effective data , which is converted data amount necessary to achieve the same performance gain by pre-training . In contrast , Eq . ( 1 ) explains the test error with respect to the pre-training data size directly at a fine-tuning task . Other differences include task domains ( language vs. vision ) and architectures ( transformer vs. CNN ) . Theory of transfer learning Theoretical analysis of transfer learning has been dated back to decades ago ( Baxter , 2000 ) and has been pursued extensively . Among others , some recent studies ( Maurer et al. , 2016 ; Du et al. , 2020 ; Tripuraneni et al. , 2020 ) derived an error bound of a fine-tuning task in the multi-task scenario based on complexity analysis ; the bound takes an additive form O ( An−1/2 +Bs−1/2 ) , where n and s are the data size of pre-training and fine-tuning , respectively , with coefficients A and B. Neural network regression has been also discussed with this bound ( Tripuraneni et al. , 2020 ) . In the field of domain adaptation , error bounds have been derived in relation to the mismatch between source and target input distributions ( Ganin et al. , 2016 ; Acuna et al. , 2021 ) . They also proposed algorithms to adopt a new data domain . However , unlike in this study , no specific learning dynamics has been taken into account . In the area of hypothesis transfer learning ( Fei-Fei et al. , 2006 ; Yang et al. , 2007 ) , among many theoretical works , Du et al . ( 2017 ) has derived a risk bound for kernel ridge regression with transfer realized as the weights on the training samples . The obtained bound takes a similar form to our scaling law . However , the learning dynamics of neural networks initialized with a pre-trained model has never been explored in this context . 3 SCALING LAWS FOR PRE-TRAINING AND FINE-TUNING . The main obstacle in analyzing the test error is that we have to consider interplay between the effects of pre-training and fine-tuning . Let L ( n , s ) ≥ 0 be the test error of a fine-tuning task with pre-training data size n and fine-tuning data size s. As the simplest case , consider a fine-tuning task without pre-training ( n = 0 ) , which boils the transfer learning down to a standard learning setting . In this case , the prior studies of both classical learning theory and neural scaling laws tell us that the test error decreases polynomially1 with the fine-tuning data size s , that is , L ( 0 , s ) = Bs−β + E with decay rate β > 0 and irreducible loss E ≥ 0 . The irreducible loss E is the inevitable error given by 1For classification with strong low-noise condition , it is known that the decay rate can be exponential ( Nitanda & Suzuki , 2019 ) . However , we focus only on the polynomial decay without such strong condition in this paper . the best possible mapping ; it is caused by noise in continuous outputs or labels . Hereafter we assume E = 0 for brevity . 3.1 INDUCTION OF SCALING LAW WITH SMALL EMPIRICAL RESULTS To speculate a scaling law , we conducted preliminary experiments.2 We pre-trained ResNet-50 by a synthetic classification task and fine-tuned by ImageNet . Figure 2 ( a ) presents the log-log plot of error curves with respect to pre-training data size n , where each shape and color indicates a different fine-tuning size s. It shows that the pre-training effect diminishes for large n. In contrast , Figure 2 ( b ) presents the relations between the error and the fine-tuning size s with different n. It indicates the error drops straight down regardless of n , confirming the power-law scaling with respect to s. The above observations and the fact that L ( 0 , s ) decays polynomially are summarized as follows . Requirement 1. lims→∞ L ( n , s ) = 0 . Requirement 2. limn→∞ L ( n , s ) = const . Requirement 3 . L ( 0 , s ) = Bs−β . Requirements 1 and 3 suggest the dependency of n is embedded in the coefficient B = g ( n ) , i.e. , the pre-training and fine-tuning effects interact multiplicatively . To satisfy Requirement 2 , a reasonable choice for the pre-training effect is g ( n ) = n−α + γ ; the error decays polynomially with respect to n but has a plateau at γ . By combining these , we obtain L ( n , s ) = δ ( γ + n−α ) s−β , ( 2 ) where α , β > 0 are decay rates for pre-training and fine-tuning , respectively , γ ≥ 0 is a constant , and δ > 0 is a coefficient . The exponent β determines the convergence rate with respect to fine-tuning data size . From this viewpoint , δ ( γ + n−α ) is the coefficient factor to the power law . The influence of the pre-training appears in this coefficient , where the constant term δγ comes from the irreducible loss of the pre-training task and n−α expresses the effect of pre-training data size . The theoretical consideration in Section E.5 suggests that the rates α and β can depend on both the target functions of pre-training and fine-tuning as well as the learning rate .

Summary: This paper proposes a scaling law for transfer learning that takes into account  the size of both the pretraining and finetuning datasets. They showed the parameters of the scaling law can be  estimated and this can be used to approximate the performance of the model given the amount of pretraining data. The authors experiments are focused on the scenario where the pretranining dataset is synthetically generated.