dataesr/scientific-paragraphs-categorization huggingface dataset:
Despite substantial progress in climate modelling over the past few decades, there remains a substantial spread in projections of future climate change. For example, the range of model estimates for ECS to a doubling of CO 2 concentrations has not decreased since the 1970s  7  , although understanding of the processes that are involved has certainly increased. The need to inform mitigation policy and adaptation remains  102. We believe that there is now an unprecedented opportunity to constrain policy-relevant metrics such as cumulative CO 2 emissions consistent with specific temperature targets with observations, and to reduce uncertainties in climate projections, both at global and regional scales. The challenge is to make intelligent use of the petabyte-scale output that will become available from the new CMIP6 project, along with modern observation systems, new model evaluation tools and novel data science techniques.
Acknowledgements The authors acknowledge the  Aspen Global Change Institute (AGCI)  for hosting a workshop on Earth System Model Evaluation to Improve Process Understanding in August 2017 as part of its traditionally landmark summer interdisciplinary sessions (http://www. agci. org/event/17s2). NASA , the  Heising-Simons Foundation ,  Horizon 2020 European Union 's  Framework Programme for Research and Innovation  under Grant Agreement No  641816 , the  Coordinated Research in Earth Systems and Climate: Experiments, kNowledge, Dissemination and Outreach (CRESCENDO)  project, the  ESA Climate Change Initiative (CCI) Climate Model User Group (CMUG) ,  WCRP  and the  Department of Energy (DOE)  all provided support for the workshop. The viewpoint presented here substantially draws on conclusions from that workshop.
Results and Evaluation We evaluate our method using a test dataset of 32 ground truth SVBRDFs not present in the set used for training data generation. We also use measured Bidirectional Texture Functions (BTFs)  [WGK14]  to compare the re-renderings of our predictions to realworld appearances. Finally, we used our method to acquire a set of around 80 real-world materials. Since our method does not assume a controlled lighting, we used either the camera flash or a separate smartphone as the light source for those acquisitions. All results in the figures of the main paper were taken with two phones; please see supplemental for all results and examples acquired with a single phone. Resulting quality is similar in both cases.
Surprisingly, the quality of the roughness prediction seems on average independent of the number of images, suggesting that the method struggles to exploit additional information for this quantity. In contrast, the normal prediction improves with each additional input, as also observed in our experiments with real-world data detailed next. We provide RMSE plots of the same experiment as supplemental materials. Using the same procedure, in Figure  5  we perform an ablation study to evaluate the impact of including random perturbations of the viewing and lighting conditions in the training data. As expected, the network trained without perturbation does not perform as well as our complete method on our test dataset that includes Figure  4 : SSIM of our predictions with respect to the number of input images, averaged over our synthetic test dataset.
The SSIM of re-renderings increases quickly for the first images, before stabilizing at around 10 images. The normal maps strongly benefit from new images. Diffuse and specular albedos also improve with additional inputs, which is not the case of the roughness that remains stable overall. We provide similar RMSE plots as supplemental materials. view and light variations similar to those in casual real world capture. We trained both networks for 750,000 iterations for this experiment. Figure  6  shows our predictions on a measured BTF material from the Bonn database [WGK14], using 1, 2, 3 and 10 inputs. For this material, normals, diffuse albedo and roughness estimations improve with more inputs. In particular, the normal map progressively captures more relief, the diffuse albedo map becomes almost uniform, and the embossed part on the upper right is quickly recognized as shinier than the remaining of the sample.
Comparison to multi-image optimization We compare our data-driven approach to a traditional optimization that takes as input multiple images captured under the assumption of known and precisely calibrated light and viewing conditions. Given these conditions we solve for the SVBRDF maps that minimize re-rendering error of the input images, as measured by our Comparison of SSIM between our method (green) and a restricted version (black) where the network is trained with lighting and viewing directions chosen on a perfect hemisphere, and with all lighting parameters constant (falloff exponent, power, etc. ). Our complete method achieves higher SSIM when tested on a dataset with small variations of these parameters, showing that it is robust to such perturbations that are frequent in casual real world capture.
We rely on procedural, artist-designed SVBRDFs to obtain our training data. Starting from a small set of such SVBRDF maps, Deschaintre et al. [DAD * 18] perform data augmentation by computing 20,000 convex combinations of random pairs of SVBRDFs. We follow the same strategy, although we implemented this material mixing within TensorFlow [AAB * 15], which allows us to gen-erate a unique SVBRDF for each training iteration while only loading a small set of base SVBRDFs at the beginning of the training process. We use the dataset proposed by Deschaintre et al. , which contains 1, 850 SVBRDFs covering common material classes such as plastic, metal, wood, leather, etc, all obtained from Allegorithmic Substance Share [All18].
prior work on deep-learning for inverse rendering [RGR * 17, LDPT17, DAD * 18, LSC18, LXR * 18, LCY * 17], we rely on synthetic data to train our network. While in theory image synthesis offers the means to generate an arbitrary large amount of training data, the cost of image rendering, storage and transfer limits the size of the datasets used in practice. For example, Li et al. [LSC18] and Deschaintre et al. [DAD * 18] report training datasets of 150,000 and 200,000 images respectively. This practical challenge motivated us to implement an online renderer that generates a new SVBRDF and its multiple renderings at each iteration of the training, yielding up to 2 million training images in practice.
Acknowledgments We thank  Yulia Gryaditskaya ,  Simon Rodriguez  and  Stavros Diolatzis  for their support during the deadline as well as  Anthony Jouanin  and  Vincent Hourdin  for regular feedback. We also thank  Zhengqin Li  and  Kalyan Sunkavalli  for their help with evaluation. This work was partially funded by an  ANRT  (http://www. anrt. asso. fr/en)CIFRE scholarship between Inria and Optis, the  ERC Advanced Grant FUNGRAPH  (No. 788065 , http://fungraph. inria. fr),and by software and hardware donations from Adobe and  Nvidia.
Algorithmic differentiation tools can automate the adjoint transformation of parallel message passing codes  [23]  using the AMPI library. Nevertheless, a non-trivial and manual step after the differentiation is the initialisation of the seed and retrieval of the output values from the differentiated code. MPK: Ambiguities in seeding occurs in programs where the user is unable to expose the complete program flow to the AD tool (a single entry and single exit point). :KPM In this work, we present the problems and ambiguities associated with seed initialisation and output retrieval for adjoint transformation of halo and zero-halo partitioned MPI programs. JDM: point out relevance: -no pb with single master paradigms, when using brute-force AD -problem when using more advanced SPMD/all-worker models, and/or partial hand-assembly.
Introduction Gradient-based optimisation methods are essential for solving optimisation problems with many control variables. The efficient computation of the required gradients is essential for application of numerical optimisation to large-scale and industrial cases, such as aerodynamic design which is the focus of our work. Algorithmic differentiation (AD) tools differentiate computer codes by applying symbolic differentiation to individual program statements and combining the sequence of statements using the chain rule of calculus. Moreover, in the typical case of a small number of output variables, their derivatives with respect to a large number of input variables can be efficiently obtained using the 'reverse' mode of AD, which accumulates the derivative by stepping through the code statements in reverse order.
Background The solution of large-scale optimal design problems in the turn-around times required by industrial application is feasible today using adjoint models and parallel computing. Application of AD to obtain differentiated code of parallel computer codes based on both shared and distributed memory has been investigated in the literature. The parallel AD forward mode was first investigated by Hovland  [10, 11]. The first major effort in applying AD to obtain adjoint MPI code has been on the MITgcm code using TAMC  [9]. The authors report that a substantial intervention into the original code was necessary to enable correct adjoint code generation. The AD book of Griewank and Walther  [7]  also has a section dedicated to AD of parallel programs. A comprehensive theory of MPI within AD for both forward and reverse mode was presented by Utke et al.
[23]. This has been implemented into the AMPI library  [21]  and later into the Adjoinable MPI library  [23]. The libraries require rewriting the MPI calls into AMPI wrapper calls, and the AD tools generate the forward/reverse MPI calls automatically. This substantially reduces the burden of generating the AD MPI code for both source-transformation (S-T) and operator-overloading (O-O) AD tools. The Adjoinable MPI with the AD tool ADOL-C was used to differentiate the Ice Sheet System Model  [22]. The authors had wrapped the MPI calls in the original code to generate correct MPI adjoint code. Recently, a discrete adjoint version of the open-source CFD solvers SU2  [1]  has been carried out using the CoDiPack AD tool  [18]  and the Adjoinable MPI library. Differentiation using AMPI in combination with an AD tool demands certain restrictions.
Firstly, the library currently supports a subset of MPI v2. 0 specification. Any unsupported MPI operations (by AMPI) in the user code must be replaced with an equivalent supported MPI call. But we hope that such restrictions would not exist in a future or next releases of the library. In fact, Schanen et al. [19, 20]  have covered the adjoining of MPI v2. 2 one-sided calls in a recent work. Secondly, the MPI function calls must be wrapped into equivalent AMPI ones. Lastly, the source for differentiation is isolated into a routine, referred to as the head. A driver routine generates the input seed and recovers the output values from the differentiated head. The driver routine generation is a manual process and non-trivial. Hovland  [10]  briefly mentions seed initialisation and associated problems in forward mode AD of MPI codes.
JDM: You spent some introducing tan-lin and adj modes, but we don't yet know what seeding does. :MDJ While the correct choice of seeding is straightforward for a serial code, it is not so intuitive for a parallel code. In the AMPI approach, the program to be differentiated has a top level routine head which does the numerical computation and communication. A driver routine calls this function in the main code. The driver routine would have to be manually adjusted to initiate the derivative computation, retrieve, and use the derivative values. Since the input and output data are distributed, so are the input seeds and output adjoint values. Therefore it is necessary to ensure correct distribution of the input seeds and retrieval of output values. In this work we use as an example the seeding of parallel programs based on the zero-halo data partition.
Code example Let us consider a simple summation function shown in Eq. (3). The reverse differentiated form for this simple function is shown in Eq. (4). y = F (x) = x 2 1 + x 2 2 + x 2 3 ( 3 ) October 25, 2018 Optimization Methods & Software zhpaper x1 = x1 + 2ȳx 1 (4) x2 = x2 + 2ȳx 2 x3 = x3 + 2ȳx 3 Let us perform the computation in parallel on two MPI ranks, ranks 0 and 1. There are multiple ways one can partition this computations between the two ranks. We present two approaches (i) Halo and (ii) Zero-halo partitioning in the next section.
Halo Approach In this approach {x 1 } is distributed to rank 0 and and {x 2 , x 3 } are distributed to rank 1. Note that there is no sharing of data between the partitions. The primal and adjoint parallel code for ranks 0 and 1 are shown in Listings 1-4. The adjoint is simple to construct using MPI idioms  [23]. The read function reads the value for the variable provided in the function argument. The shorthand notations send and recv stand for the MPI Send and MPI Recv functions. In Table  1  the comparison of the output from serial and parallel AD code is shown for seed values ȳ = x1 = x2 = x3 = 1. Note that the same serial seed value was used in the parallel adjoint code. We clearly find that the seed ȳ is counted twice, which leads to a erroneous result. While it can be argued that the operation y = y + t is a reduction and only one of the rank should take ownership of this operation, it is far from obvious for a practical user of AD.
In fact, one might argue that exposing the complete program to the AD tool (including the data decomposition) will eliminate this issue. But this is impractical and impossible in many situations because source code availability cannot be assumed (commercial libraries). The AD tool might not support a given mix of programming languages or supports only a restricted subset. Therefore, in such situations one differentiates pieces of the overall program and assembles the differentiated subroutines manually. Conceptually the AD process is clear and given a single entry and single exit for the MPI program one can quite reliable carry out the seeding and adjoining. The PGAS representation  [20]  of the program can be used to verify its correctness. In Listing 5-8 we provide the correct implementation using the idea of a single entry and single exit.
In fact this problem has been acknowledged in the AD literature. For example Hovland  [10]  states that replicated operations on individual ranks are indeed a source of problem for seeding in the context of forward AD MPI codes. Listing 1: Halo parallel code (rank 0) 1 read (x 1 ) 2 3 y = x 2 1 4 send (y , to =1) 5 recv (t, from =1) 6 y = y + t Listing 2: Halo parallel code (rank 1) 1 read (x 2 ) 2 read (x 3 ) 3 y = x 2 2 + x 2 3 4 recv (t, from =0) = x 2 1 + x 2 2 + x 2 3 (rank 0) 1 t = ȳ 2 send ( t, to =1) 3 recv (t 0 , from =1) 4 ȳ = ȳ + t 0 5 x1 = x1 + 2 ȳx 1 6 7 ȳ = 0 Listing 4: Halo partitioning adjoint of y = x 2 1 + x 2 2 + x 2 3 (rank 1) 1 t = ȳ 2 recv (t 1 , from =0) 3 ȳ = ȳ + t 1 4 send ( t, to =0) 5 x2 = x2 + 2 ȳx 2 6 x3 = x3 + 2 ȳx 3 7 ȳ = 0 Table  1. : Serial seed with value 1, resulting gradients for sequential code and on each partition (halo approach).
Variable Seed Serial Rank 0 Rank 1 ȳ 1 0 0 0 x1 1 1 + 2x 1 1 + 4x 1 1 + 4x 1 x2 1 1 + 2x 2 1 + 4x 2 1 + 4x 2 x3 1 1 + 2x 3 1 + 4x 3 1 + 4x 3 Listing 5: Corrected version of halo parallel code (rank 0) 1 read (x 1 ) 2 3 y = x 2 1 4 recv (t, from =1) 5 y = y + t 6 send (y , to =1) Listing 6: Corrected version of halo parallel code (rank 1) 1 read (x 2 ) 2 read (x 3 ) 3 y = x 2 2 + x 2 3 4 send (y , to =0) 5 recv (t, from =0) 6 y = t Listing 7: Corrected version of halo adjoint of y = x 2 1 + x 2 2 + x 2 3 (rank 0) 1 recv ( ȳ , from =1) 2 t = ȳ 3 send ( t, to =1) 4 x1 = x1 + 2 ȳx 1 5 6 ȳ = 0 Listing 8: Corrected version of halo adjoint y = x 2 1 + x 2 2 + x 2 3 (rank 1) 1 t = ȳ 2 send ( t, to =0) 3 recv ( ȳ , from =0) 4 x2 = x2 + 2 ȳx 2 5 x3 = x3 + 2 ȳx 3 6 ȳ = 0.
Zero-halo Approach In this section we consider a zero-halo partitioning, where x 1 and x 3 are uniquely distributed to ranks 0 and 1, respectively, but x 2 is shared between the two ranks. In zero-halo terminology, x 2 is called a shared node variable at the zero-halo partition boundary. In Listing 9 and 10 we have shown the pseudo code for the parallel implementation. A peculiar aspect of this zero-halo splitting is the partial computation on x 2 (halving of the value) on each rank and the final accumulation on y. The reverse-differentiated code in each rank is shown in Listings 11 and 12. Listing 9: Zero-halo parallel code (rank 0) 1 read (x 1 ) 2 read (x 2 ) 3 y = x 2 1 + 1 2 x 2 2 4 send (y , to =1) 5 recv (t, from =1) 6 y = y + t Listing 10: Zero-halo parallel code (rank 1) 1 read (x 2 ) 2 read (x 3 ) 3 y = 1 2 x 2 2 + x 2 3 4 recv (t, from =0) 5 send (y , to =0) 6 y = y + t Listing 11: Zero-halo adjoint of y = x 2 1 + x 2 2 + x 2 3 (rank 0) 1 t = ȳ 2 send ( t, to =1) 3 recv (t 0 , from =1) 4 ȳ = ȳ + t 0 5 x1 = x1 + 2 ȳx 1 6 x2 = x2 + ȳx 2 7 ȳ = 0 Listing 12: Zero-halo adjoint of y = x 2 1 + x 2 2 + x 2 3 (rank 1) 1 t = ȳ 2 recv (t 1 , from =0) 3 ȳ = ȳ + t 1 4 send ( t, to =0) 5 x2 = x2 + ȳx 2 6 x3 = x3 + 2 ȳx 3 7 ȳ = 0 Table  2  compares the values of x1 , x2 and x3 between the serial and parallel adjoint code.
In Tab. 2 naively the same seeding approach as for the serial code was used. This results in incorrect derivatives, since the message exchange doubles the seed ȳ, but there is a missing accumulate in x2 , cancelling out the error for x2. Although the doubling of ȳ is obvious from the previous halo example one would still gets erroneous results using the proposed fix due to the shared node accumulation. As described previously the problem arises because the complete program flow is unavailable to the AD tool. The presence of shared variables x 2 and y in both MPI ranks is the source of the problem. The information that these variables should belong to a single rank and that the reduction should happen in a single rank is hidden from the AD tool. This motivates our conceptual master-worker (CMW) model, described in the next section.
By defining input and output ends controlled by the single master process, this model eliminates the ambiguities in seeding these shared nodes. For the non-transformed original code, the broadcasts to or reductions from the workers then inform about the correct handling of seed values. In the cases where technical limitations forbid to expose the complete program flow to the AD tool, the CMW model can be presented as an alternative way to provide the AD tool with a code that has a single entry point and a single exit point namely, the entry and the exit points of the master process. The CMW applied to the shared-node reduction is shown in the next subsection.
CMW adjoint seeding algorithm for shared-node reduction The CMW adjoint model can be derived using Eq. (  2 ) for the shared node reduction in the master process as shown in Eq. (  6 ). Using the chain rule of calculus and grouping terms, one obtains Eqs. (  7 ) and  (8) , which are specialised adjoints for two classes of binary operators, namely (i) additive (⊗ ≡ +) and (ii) multiplicative (⊗ ≡ * ). x = x + d dx (F 0 (x) ⊗ F 1 (x) ⊗. F i (x)) T ȳ (6) Case (i): Additive CMW adjoint model x = x + i∈S rank dF i (x) dx T ȳ (7) Table  2. : Serial seed with value 1, resulting gradients for sequential code and on each partition (zero-halo approach). Variable Seed Serial Rank 0 Rank 1 ȳ 1 0 0 0 x1 1 1 + 2x 1 1 + 4x 1 1 + 4x 1 x2 1 1 + 2x 2 1 + 2x 2 1 + 2x 2 x3 1 1 + 2x 3 1 + 4x 3 1 + 4x = x + i∈S rank   dF i (x) dx S rank j =i F j (x)   T ȳ (8) The following corollaries can be inferred from the adjoint models of Eqs.
The CMW abstraction for the summation problem (zero-halo partitioned) described in the previous section involves substituting,  5 ). The summation problem and its adjoint using the CMW adjoint model on two MPI ranks is illustrated in Fig. 2. The pseudo-code for the CMW primal and adjoint is shown in Listings 13 and 14, which show the fix for the doubling of ȳ and the missing accumulate in x2 as identified previously in (a-d). The master broadcasts the same seed ȳmaster to both MPI ranks. The results t 0,1 from both ranks are accumulated in the master rank (x 2 = x2master + t 0 + t 1 ). ⊗ ≡ +, N = 2, S rank = {0, 1}, {F 0 , F 1 } ≡ { 1 2 x 2 , 1 2 x 2 } in Eq. ( Listing 13: CMW primal of summation problem (zero-halo) 1.
(  9 ) one can show that any linear combination of the seed whose shared node sum is equal to the x2master will produce correct adjoint values as shown in Eq. (10). xlocal 2 = α local x2master , S rank α local = 1 (10) Zero-halo partitioning performs partial computations at a shared node. Therefore by using the PS approach no change is needed to the output value produced by reverse AD for the partition. In the next section, we demonstrate this seeding approach applied to a parallel CFD code. We show that choosing the PS seeding strategy one can also simplify the code and reduce the number of MPI calls. Listing 15: Unique seeding approach (US) 1.
Case study Let us consider an aerodynamic shape optimisation problem as formulated in Eqs. (11) ,  (12). min α J(U , α), ( 11 ) s. t. R(U , α) = 0, ( 12 ) October 25, 2018 Optimization Methods & Software zhpaper where J is a scalar cost-function such as drag, lift, etc. , which is minimised subject to the PDE constraint R. In this work, the constraint R are the discrete Reynolds Averaged Navier-Stokes equations (RANS) solved numerically using the edge-based finite-volume solver STAMPS  [15]. The field U is the fluid-state and α is the vector of shape design variables. Gradient-based optimisation methods require the derivatives of the cost or objective function (output) with respect to the design variables (input), dJ dα. The computational cost of computing sensitivities with the reverse or adjoint mode is proportional to the dimension of the output J and independent of the dimension of the input α.
as shown in Fig. 8. Note that the cost-function adjoint produces only partial values of Q at the shared nodes, which is later accumulated by the master. The two adjoints can now be combined to yield the RHS term of Eq. (  19 ). When seeding the assembled FPI we also can use either the US or the PS approach, as illustrated in Figs. 9 and 10. The corresponding pseudo code is shown in Listings 19 and 20. The PS approach avoids two extra operations present in the the US approach, (i) accumulation of Q after the end of the adjoint cost-function and (ii) unique seeding for the adjoint residual. Thus, the PS approach simplifies the adjoint code. Note that in the PS approach the MPI call elimination occurs outside the fixed-point loop. Therefore, the general scalability remains the same for both methods.
1 J = 1 2 Q = 0 3 costFunLocal(Q, Q, J , J ) 4 accumulate ( Q) ! Extra MPI 5 dJdQ = Q 6 do iter = 1 , nIter 7 R = v 8 Q = 0 9 residualLocal(Q, Q, R, R) 10 accumulate ( Q) 11 Q = Q -dJdQ 12 update (v , Q) 13 end do Listing 20: Adjoint FPI (PS approach)  October 25, 2018  Optimization Methods & Software zhpaper J = 1 Q = dJdQ costFunLocal(Q, Q, J , J ) dJdQ = Q do iter = 1 , nIter R = v Q = 0 residualLocal(Q, Q, R, R) accumulate ( Q) update (v , Q) end do Listing 21: Adjoint FPI (re-computation) do iter = 1 , nIter J = 1 Q = 0 costFunLocal(Q, Q, J , J ) R = v residualLocal(Q, Q, R, R) accumulate ( Q) update (v , Q) end do To demonstrate the MPI zero-halo adjoint in STAMPS, we have simulated external flow over a truncated RAE2822 airfoil. The simulation was carried out for a compressible viscous fluid in laminar flow at a Mach number of 0.