Multi-Resolution Active Learning of Fourier Neural Operators Shibo Li 1, Xin Yu 1, Wei Xing 4, Robert M. Kirby 1,2, Akil Narayan 2,3, Shandian Zhe 1 1Kahlert School of Computing, University of Utah 1Kahlert School of Computing, University of Utah 2Scientific Computing and Imaging (SCI) Institute, University of Utah 3Department of Mathematics, University of Utah 4School of Mathematics and Statistics, University of Sheffield Abstract: Fourier Neural Operator (FNO) is a popular operator learning framework. It not only achieves the state-of-the-art performance in many tasks, but also is efficient in training and prediction. However, collecting training data for the FNO can be a costly bottleneck in practice, because it often demands expensive physical simulations. To overcome this problem, we propose Multi- Resolution Active learning of FNO (MRA-FNO), which can dynamically select the input functions and resolutions to lower the data cost as much as possible while optimizing the learning efficiency. Specifically, we propose a probabilistic multi-resolution FNO and use ensemble Monte-Carlo to develop an effective posterior inference algorithm. To conduct active learning, we maximize a utility-cost ratio as the acquisition function to acquire new examples and resolutions at each step. We use moment matching and the matrix determinant lemma to enable tractable, efficient utility computation. Furthermore, we develop a cost annealing framework to avoid over-penalizing high-resolution queries at the early stage. The over-penalization is severe when the cost difference is significant between the resolutions, which renders active learning often stuck at low-resolution queries and inferior performance. Our method overcomes this problem and applies to general multi-fidelity active learning and optimization problems. We have shown the advantage of our method in several benchmark operator learning tasks. Introduction & Motivation Physics vs. Machine Learning • Physics: Accurate, Principled, Extrapolate Well • Machine Learning: Flexible, Efficient Computational Physics: Finding the solutions of PDEs • Analytical solutions: closed forms but not always feasible • Numerical solutions: accurate, reliable but slow and no generalization Wave Fluid Dynamics Electromagnetism VS. Data-Driven Computational Physics • Learning PDE solutions directly from data • Efficient prediction and generalization Analytical Numerical (e.g., FEM) • Initial condition • Boundary condition • PDE coefficients Heat Equation PDE Solutions Neural Operators and Fourier Neural Operators • Solutions of PDEs are applying operators to the source/parameter/initial functions • Learning the parametric mapping between functions to functions Neural operator layer … Functional space transformation Function convolution Neural operator layer Convolutional Theorem Stationary kernel functions Fourier Neural Operators (FNO) Input function 1 … X Y Numerical Solvers Slow! Expensive! However, Data Costs for Training FNO!!! Input function 2 Input function N Solution1 … Solution 2 Solution N • Probabilistic Multi-Resolution FNO • Active Learning for FNO • Robust performance via cost-annealing Our Contributions Methods Probabilistic FNO FNO Ensemble 2 … Predictive distribution Monte-Carlo Ensembles … FNO Ensemble 1 FNO Ensemble M Multi-Resolution Active Learning FNO MR Utility Multi-Resolution PDE data Probabilistic MR FNO Ensembles Pool of MR data Numerical Solvers Multi-resolution acquisition function Step1: Monte-Carlo approximation Step 2 : Predictive distribution via moment matching Step 3 : Efficient computation via Weinstein- Aronszajn Identity Cost-Annealing Strategy for Robust Active Learning … Few Data Inferior model performance Inaccurate Utility Dynamically assign cost to each resolution Probabilistic Multi-Resolution FNO Experiments Evaluations of Multi-Resolution Active Learning Visualization of Pointwise Error after Active Learning (Navier-Stocks) Ablation Study 1 : Predictive perfor