Artificial neural-networks have the potential to emulate cloud processes with higher accuracy than the semi-empirical emulators currently used in climate models. However, neural-network models do not intrinsically conserve energy and mass, which is an obstacle to using them for long-term climate predictions. Here, we propose two methods to enforce linear conservation laws in neuralnetwork emulators of physical models: Constraining (1) the loss function or (2) the architecture of the network itself. Applied to the emulation of explicitly-resolved cloud processes in a prototype multi-scale climate model, we show that architecture constraints can enforce conservation laws to satisfactory numerical precision, while all constraints help the neural-network better generalize to conditions outside of its training set, such as global warming. The largest source of uncertainty in climate projections is the response of clouds to warming (Schneider et al., 2017). The turbulent eddies generating clouds are typically only O (100m  10km) -wide, meaning that climate models need to be run at spatial resolutions as fine as O (1km) to prevent large biases. Unfortunately, computational resources currently limit climate models to spatial resolutions of O (25km) when run for time periods relevant to societal decisions, e.g. 100 years (IPCC, 2014). Therefore, climate models rely on semi-empirical models of cloud processes, referred to as convective parametrizations (Stevens and Bony, 2013;Sherwood et al., 2014). If designed by hand, convective parametrizations are unable to capture the complexity of cloud processes and cause well-known biases, including a lack of extreme precipitation events and unrealistic cloud structures (Daleu et al., 2015;2016). Recent advances in statistical learning offer the possibility of designing data-driven convective parametrizations by training algorithms on short-period but high-resolution climate simulations (Gentine et al., 2018). The first attempts have successfully modeled the interaction between small-scale clouds and the large-scale climate, offering a pathway to improve the accuracy of climate predictions (Brenowitz and Bretherton, 2018;Rasp et al., 2018;Krasnopolsky et al., 2013). However, machine learning-based climate models do not intrinsically conserve energy and mass, which is a major obstacle to their adoption by the physical science community for several reasons, e.g. : 1) Realistic simulations of climate change respond to relatively small O 1W m 2 radiative forcing from carbon dioxide. Inconsistencies of this magnitude can prevent this small forcing from being communicated down to the surface and the ocean where most of the biomass lives. 2) Artificial sources and sinks of mass and energy distort weather and cloud formation on short timescales, resulting in large temperature and humidity drifts or biases for the long-term climate. Current machine-learning convective parametrizations that conserve energy are based on decision trees (e.g. random forests), but these are too slow for practical use in climate models (O'Gorman and Dwyer, 2018). Since neuralnetwork convective parametrizations can significantly reduce cloud biases in climate models while decreasing their overall computational cost (Rasp et al., 2018), we ask: How can we enforce conservation laws in neural-network emulators of physical models? After proposing two methods to enforce physical constraints in neural network models of physical systems in Section 2, we apply them to emulate cloud processes in a climate model in Section 3, before comparing their performances and how they improve climate predictions in Section 4. Consider a physical system represented by a function f : R m  R p that maps an input x  R m to an output y  R p : (1) Many physical systems satisfy exact physical constraints, such as the conservation of energy or momentum. In this paper, we assume that these physical constraints (C) can be written as an under-determined linear system of rank n: where C  R n x R m+p is a constraints matrix acting on the input and output of the system. The physical system has n constraints, and by construction, n < p + m. Our goal is to build a computationally-efficient emulator of the physical system f and its physical constraints (C). For the sake of simplicity, we build this emulator using a feed-forward neural network (NN) trained on preexisting measurements of x and y, as shown in We apply this penalty by giving it a weight   [0, 1] in the loss function L, which is similar to a Lagrange multiplier: 2. Constraining the architecture (NNA): In this setting, we augment the simple network (NN) with n conservation layers to enforce the conservation laws (C) to numerical precision , while still calculating the MSE loss over the entire output vector y. The feed-forward network outputs an "unconstrained" vector u  R pn whose size is only where n is the number of constraints. We then calculate the remaining component v  R n of the output vector y NN using the n constraints. This defines n constraint layers (CL 1..n ) that ensure that the final output y NN exactly respects the physical constraints (C). A possible construction of (CL 1..n ) solves the system of equations (C) from the bottom to the top row after writing it in row-echelon form. Note that the loss is propagated through the physical constraints. We now implement the three neural networks (NNU, NNL, NNA) and compare their performances in the particular case of convective parametrization via emulation of the 8,192 cloud-resolving sub-domains embedded in the Super-3.0 (Collins et al., 2006;Khairoutdinov et al., 2005). We simulate an "ocean world" where the surface temperatures are fixed with a realistic equator-to-pole gradient (Andersen and Kuang, 2012). To facilitate the comparison, all networks have 5 hidden layers with 512 nodes each, and use leaky rectangular unit activation functions: x  max (0.3x, x) to help capture the system's non-linearity. We use the RMSprop optimizer (Tieleman et al., 2012) to train each network during 20 epochs, using 3 months of climate simulation with 30-minute outputs as training data. The goal of the neural network is to predict an output vector y of size 218 that represents the effect of cloud processes on climate (i.e. convective and radiative tendencies), based on an input vector x of size 304 that represents the climate state (i.e. large-scale thermodynamic variables).
The 4 conservation laws can be written as a sparse matrix of size 4 x (304 + 218) that acts on x and y to yield equation 2. Cl.change MSE 747 +/- 1 x 10 5 633 +/- 7 x 10 3 471 +/- 5 x 10 3 496 +/- 8 x 10 3 567 +/- 8 x 10 3 (+4K) We evaluate the performances of (NNU, NNL, NNA) on two different validation datasets: (+0K) An "ocean world" similar to the training dataset. (+4K) An "ocean world" where the surface temperature has been uniformly warmed by 4K, a proxy for the effects of climate change. We do not expect (NN) to perform well in the Tropics, where this perturbation leads to temperatures outside of the training set. All neural networks perform better than the multiple-linear regression model (MLR), derived by replacing leaky rectangular units with the identity function and optimized in-dependently. While the reference "unconstrained" network NNU performs well as measured by MSE, it does so by breaking conservation laws, resulting in a large penalty P. Enforcing conservation laws via architecture constraints (NNA) works to satisfactory numerical precision on both validation datasets, resulting in a very small penalty P. Giving equal weight to MSE and P in the loss function (NNL =0.5 ) leads to mediocre performances in all areas. In contrast, surprisingly, introducing the penalty P in the loss function with a very small weight ( = 0.01) leads to the best performance on the reference validation dataset (+0K) . Both constrained networks NNL =0.01 and NNA generalize better to unforeseen conditions (+4K) than the "unconstrained" network, suggesting that physically constraining neural networks improves their representation abilities. This ability to generalize is confirmed by the high R 2 score when predicting the outgoing longwave radiation , which can be used as a direct measure of radiative forcing in climate change scenarios. Overall, our results suggest that (1) constraining the network's architecture is a powerful way to ensure energy conservation over a wide range of climates and (2) introducing a very small information about physical constraints in the loss function or/and the network's architecture can significantly improve the generalization abilities of our neural network emulators.