In recent years, there has been widespread interest in modeling PDE systems with neural operators or operator learning in a broad variety of science and engineering applications [32, 30, 27, 41, 33, 37, 25, 2, 49, 4, 29, 26]. Following the success of TL in CV and NLP tasks [54, 43, 48, 13, 36, 40, 22], there have been several investigations into how TL can be leveraged for SciML problems involving differential equations. Most have focused on applications of Physics-Informed Neural Networks (PINNs) [35, 19, 15, 8, 18, 17, 50, 14, 10, 31], where models can be fine-tuned or adapted using a physics-based loss function determined by the specific target PDE/ODE system. Another common theme is evaluating how well TL can account for the diversity of geometries [6, 47, 50, 16] and discretizations [34, 7, 44, 46] found in scientific computing. Recently, attention has also been devoted to TL for neural operators in SciML, where some works have explored certain components of our analysis in isolation. This includes studies on the TL performance of DeepONet [51, 16, 55] and FNO [31, 9], where one of either the target domain, PDE coefficients, or PDE source functions shift, are varied. In [9], the authors evaluate different model and training dataset sizes for different neural operator architectures, but their analysis is mostly limited to within-distribution experiments and single operators, and they do not analyze the fine-tuning performance of such models (particularly on OOD data). The works above have not considered the pre-training regimes over a diverse set of physical systems and most have not characterized their TL as a function of data and model scaling, nor included multiple PDE operators during training. To the best of our knowledge, the work of [52] (which has appeared concurrent to this work) is the first (along with ours) to consider the TL potential of neural networks across operators and data distributions. There, the authors adapt in-context learning (from LLMs) to solve differential equations with transformer-based models. The main difference with [52] is that the authors focus on in-context learning which requires prompting (with up to five example demos) to solve OOD tasks including different differential equation coefficients and operators than those seen at train time (similar to our work). In contrast, we focus on TL with zero-shot and few-shot learning through fine-tuning. Thus the two approaches (TL through fine tuning vs in-context learning) are complementary but different. We also note that their investigation was performed at a much smaller scale (from both model and data perspective), and was limited to simpler dynamical systems, as compared to the scales tested in our work. Overall, the prior related work provide important initial results on the interpolation and extrapolation behavior of neural operators in the context of differential equations, and/or limited investigation into the behavior as model and dataset sizes are increased. However, none of them consider these aspects simultaneously with a more diverse pre-training corpus (by varying all input variables such as source functions and PDE coefficients and/or including different operators for pre-training), which is closer to methodology adopted by CV and NLP in the development of their foundation models, with emphasis on the importance of characterizing scaling properties [23, 21].