dataesr/scientific-paragraphs-categorization huggingface dataset:
Python xarray package makes filtering and thresholding of the data very intuitive with the function xarray. where. An example Jupyter Notebook for visualization of NOMAD  51  data and the reproduction of Fig. 2  is available on Github: https://github. com/FlorianeSudre/NOMAD_notebooks/. Front collocation with biological data. NOMAD  51  is primarily designed to study front-marine life interactions. To this end, the authors recommend using both thermal gradients and FSLE fields, and to interpolate them on the same grid, whose temporal and spatial scale is congruent with the behavior of the specie of interest. Temporal degradation of the dataset is easily performed by xarray. Dataset. resample while spatial regridding can be done with tools such as the Climate Data Operator (CDO) or with the Python package xgcm used in tandem with xarray).
The authors recommend comparing quantitatively and qualitatively BOA outputs solely between same-grid products. For example, one can compare thermal gradient magnitudes and patterns within the entire NOMAD  51  thermal gradient dataset, as the SST field comes from a single product. However, while qualitative comparison (i. e. spatial patterns) is possible among thermal gradients originating from different SST fields, a quantitative comparison would require to determine a suitable threshold for each product since BOA algorithm is sensitive to the grid resolution of the input data. Attention should also be brought to the fact that thermal gradients and FSLEs do not have NaNs (where the pixel is deemed "unsuitable", i. e. where the quality flag is not equal to 0) at the same time and place. NaNs appear in FSLE fields near domain boundaries and in coastal areas, while NaNs in thermal gradient fields depend on cloud cover (Fig.
9 ). The difference in NaN coverage in both dataset should be taken into account when pairing with biological data to avoid any additional bias due to suitable day frequency. other possible uses for NoMaD. Beside the study of front-marine life interactions 10 , investigating trophic relationships  70  and the accumulation of fish larvae  71  , NOMAD  51  could also be employed to monitor marine pollutants such as microplastics  72, 73  and oil spills  74  along fronts. One could also examine relationships between surface thermal fronts and low-level winds at submesoscales from Radar Scatterometer data; as it has only been done at mesoscales so far  [75] [76] [77]. Finally, following the recent work of Roman-Stork et al. 78  , it would be interesting to explore new blended metrics based on both SST fronts and FSLEs and their co-variability with key biogeochemical elements (such as Chl-a, pH, nitrates, phosphates) to inform on where oceanic fronts are more likely to impact nutrient cycling in the upper and interior ocean  79.
acknowledgements This work was supported by the  Mediterranean Institute of Oceanography (MIO)  and the  French National Institute for Sustainable Development (IRD). The authors wish to thank  J. Lecubin ,  C. Yohia  and  C. Quentin  for general technical assistance as well as  J. Meillon ,  C. Pertuisot  and  M. Dequidt  from  Ifremer  for setting up the online data repository. The authors acknowledge funding from the " Ocean Front Change " project, funded by the  Belmont Forum  and implemented through the  French National Research Agency  ( ANR-20-BFOC-0006-04 ). The project leading to this publication has received funding from  European FEDER Fund  ( 1166-39417 ) and from  Excellence Initiative of AMU - A*MIDEX ("Investissements d' Avenir" ). BD acknowledges support from  ANID (Concurso de Fortalecimiento al Desarrollo Cientifico de Centros Regionales   2020-R20F0008-CEAZA  and  Centro de Investigación Oceanográfica en el Pacífico Sur-Oriental COPAS   COASTAL FB210021 ).
Code availability The code we used to produce thermal gradients with the BOA is available in R on Github thanks to Benjamin Galuardi (available on https://github. com/galuardi/boaR, accessed on May 19th, 2022). The code to produce the backward FSLEs belongs to Ismael Hernández-Carrasco, was extensively described in Hernández-Carrasco et al. 34  , and is available upon request. An example jupyter notebook for front visualization is available on a github repository (https://github. com/FlorianeSudre/NOMAD_notebooks).
Because the BCDI dataset includes a single diffraction peak, it measures strain only along the [002] direction. This complicates the analysis, as chemically equivalent but directionally strained {001} facets exhibit different apparent displacement, depending on their relative orientation (see Supplementary Figures  10  and 11 ). This issue can fortunately be corrected through the well-known Poisson effect, where elastic distortion of a solid in one direction produces an opposing distortion perpendicular to the applied force (Figure  6a ). Out-of-plane tensile strain observed on the (001) and (001) facets therefore coincides with an in-plane compressive 10/21 strain perpendicular to the [001] direction (inward displacement). Compressive in-plane strain on the (010), (010), (100) and (100) facets implies these regions also experience a tensile out-of-plane strain.
Molecular statics simulations In order to model accurately the experimental NPs, several materials simulation tools were used. We have used the Facet Analyser plugin of the ParaView software, which is an automated 3D facet recognition algorithm that allows to detect the voxels at the surface of the particles and to index their facets  53. We then used the nanoSCULPT tool  54  to build realistic structures from the ParaView exported VTI files, for further atomistic simulations. Four nanocrystals were built with the following properties: To obtain accurate and realistic relaxed configurations which reproduce as faithfully as possible the displacement fields measured in the experimental particle, MS simulations were carried out with the open-source Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS)  55, 56.
The computation was performed with a GPU using the PyNX 58 scattering package, which considerably speed up the calculation of 3D diffraction patterns. The reciprocal volume over which the calculation was performed was selected based on the target voxel size in the real space data (same number of voxels in all simulations). Finally, the real space displacement and strain fields, u 002 and ε 002 were derived from the complex sample density ρ(r). The latter was obtained by performing a simple inverse Fourier transform of the scattered amplitude Ã(q): ρ(r) = ρ(r)e 2πig•u(r) = FT -1 [ Ã(q)] 13/21 The projection of the displacement field g 002 • u 002 and the corresponding strain field ε 002 = ∂u 002 ∂z 002 could then be directly compared to the measured experimental strain field. Density functional theory simulations DFT calculations were performed using the Vienna Ab Initio Simulation package (VASP)  [59] [60] [61]  , within the projector-augmented wave method (PAW), for the description of the interaction between the valence electrons and the ionic core.
Acknowledgements The authors are grateful to ESRF synchrotron for allocating beamtime, and to the ID01 beamline staff for the excellent support during the measurements. This project received funding from the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation programme  (grant agreement No. 818823 ). The authors thank  Pr. Philippe Sautet  and  Dr. Thierry Deutsch  for the fruitful discussion on the DFT simulations.
Visual Question (input) Relevant visual passage in the Knowledge Base "Which constituency did this man represent when he was Prime Minister?" "In which year did this ocean liner make her maiden voyage?" Queen Elizabeth 2, often referred to simply as QE2, is a floating hotel and retired ocean liner built for the Cunard Line which was operated by Cunard as both a transatlantic liner and a cruise ship from 1969 to 2008. Macmillan indeed lost Stockton in the landslide Labour victory of 1945, but returned to Parliament in the November 1945 by-election in Bromley. Fig. 1 : Example of visual questions about named entities from the ViQuAE dataset along with relevant visual passages from its Knowledge Base  [32]. combined. Therefore, the task is more broadly related to Multimodal Information Retrieval (IR) and Multimodal Fusion.
There are two paradigms for multimodal IR and for multimodal learning more generally: early fusion (data-and feature-level) and late fusion (score-and decision-level)  [30]. On the one hand, late fusion is more straightforward as both Natural Language Processing and Computer Vision techniques can be applied independently. However, it neglects interactions between modalities. For example, in Figure  1 , attempting to recognize Harold Macmillan without accounting for him being a Prime Minister is suboptimal. On the other hand, the richness of early fusion often comes at the cost of increasing complexity and model parameters. This adds an extra challenge for KVQAE, where the two existing datasets are either small, because of a costly annotation process (ViQuAE  [32] ), or generated automatically (KVQA  [50] ), which leads to several limitations discussed in  [32].
To address this challenge, we propose the multimodal Inverse Cloze Task (ICT) for pre-training the two early fusion models we define in this article for tackling KVQAE. Multimodal ICT consists in considering a sentence paired with a nearby image as a visual pseudo-question and its multimodal context as a relevant visual passage. It is related to the visual cloze task proposed by  [61] , a downstream task that requires modeling temporal events. Textual ICT was first introduced in  [31]  to pre-train a neural retriever for textual Question Answering (QA) and can be seen as a generalization of the skip-gram objective  [41]. Our main contributions are: (i) Multimodal ICT, a new pre-training method that allows tackling KVQAE, even for small datasets as ViQuAE; (ii) a multimodal IR framework for KVQAE; (iii) experiments with different neural network architectures, including recently proposed multimodal BERTs.
Instead, they propose DPR, which takes advantage of (i) the heuristic of whether the passage contains the answer to the question to deem it relevant; (ii) unsupervised IR methods such as BM25 to mine hard negatives examples, which proved to be the key of their method's success. We aim to take advantage of both approaches by (i) pre-training our model on text QA datasets like DPR; (ii) incorporating multimodality into this hopefully-well-initialized model by adapting the ICT of  [31]  to multimodal documents. Multimodal Fusion and Pre-Training. The success of BERT in NLP  [10] , which relies on the easily-parallelizable Transformer architecture  [57] , an unsupervised training objective, and a task-agnostic architecture, has concurrently inspired many works in the VQA and cross-modal retrieval fields  [56, 38, 35, 55, 34, 7].
These models are unified under a single framework in  [5]  and partly reviewed in  [28]. All of these models rely on the Transformer architecture, often initialized with a pre-trained BERT, in order to fuse image and text. The training is weakly supervised, based upon image caption datasets such as COCO  [37]  or Conceptual Captions  [51] , and pre-trained object detectors like Faster R-CNN  [48]. [22]  show that these models learn nontrivial interactions between the modalities for VQA. Multimodal BERTs can be broadly categorized into single-stream and multi-stream. Single-stream models feed both text tokens' embeddings and image regions' embeddings to the same Transformer model, relying on the self-attention mechanism to fuse them. Instead, in the multi-stream architecture, text and image are first processed by two independent Transformers before using crossattention to fuse the modalities.
Both architectures have been shown to perform equally well in  [5]. In this work, we use a single-stream model to take advantage of pre-training on text-only (on QA datasets). Also note that, while inspired by these work, we do not use the same training objectives or data, which are arguably unsuited for named entities' representations, as explained in the next section. Multimodal Information Retrieval and KVQAE. Multimodal IR has largely been addressed using late fusion techniques (see  [9]  for a survey) but we are mostly interested in early fusion techniques in this work. [9]  review first attempts at early fusion. It was then systematically done by concatenating the features of both modalities in a single vector, with a focus on the feature weighting scheme. Concatenation is confronted with the curse of dimensionality as the resulting feature space equals the sum of the dimensions of each modality's features.
The discrepancies between these works can be explained because they use neither the same KB nor the same evaluation metrics. [19]  also experiment with different multimodal BERTs but dispense passage-level annotation for an end-to-end training of the retriever and answer classifier  4. Although they experiment with KVQA  [50] , we do not consider the work of  [16, 21]  as their systems take a human-written caption as input, which makes the role of the image content unclear. [50]  follow a late fusion approach at the decision-level. First, they detect and disambiguate the named entity mentions in the question. Then, they rely on a face recognition step as their dataset, KVQA, is restricted to questions about person named entities. Facts from both textuallyand visually-detected entities are retrieved from Wikidata 5  and processed by a memory network  [59].
Information Retrieval Framework In our multimodal setting, both visual questions (from the dataset) and visual passages (from the KB) consist of a text-image pair (t, i), as in Figure  1. Our goal is to find the optimal model E to encode adequate representations q = E(t q , i q ) and p = E(t p , i p ) such that they are close if (t p , i p ) is relevant for (t q , i q ) (denoted with the superscripts ( + ) and ( -)). Search then boils down to retrieving the K closest visual passages to the visual question. When computing the similarity between two vectors, here with the dot product, the objective used throughout all the training stages ( §3. 3) is to minimize the following negative log-likelihood loss for all visual questions in the dataset (see Figure  2 )  [31, 27]. -log exp (q • p + ) exp (q • p + ) + j exp (q • p - j ) (1) This contrastive objective allows to efficiently utilize passages relevant to other questions in the batch as in-batch negatives, since computing the similarity between two vectors is rather inexpensive compared to the forward pass of the whole model.
Training stages The models are trained sequentially in three stages. The first two stages correspond to pre-training stages: the first one is dedicated to IR for Question Answering while the second one focuses on multimodal IR. The last stage corresponds to the training for our target task of IR for KVQAE. Stage 1: DPR for textual Question Answering. Leaving visual representations aside, a DPR model is trained starting from the BERT initialization  [27]. DPR consists of two BERT encoders: one for the question t q and one for the text passage t p. We use the model pre-trained by  [32]  on TriviaQA, filtered of all questions used in their dataset, ViQuAE. They use the KILT  [43]  version of TriviaQA and Wikipedia, which serves as KB at this stage. Each article is then split into disjoint passages of 100 words for text retrieval, while preserving sentence boundaries, and the title of the article is appended to the beginning of each passage.
In our case, however, we argue that it is neither necessary, as the model should be strongly initialized from Stage 1 training on TriviaQA, nor beneficial, as the model could then ignore the image modality. Question and passage encoders pre-trained in Stage 1 are used to initialize the visual question and visual passage encoders, respectively. The process is eased thanks to the WIT dataset  [53]. WIT consists of millions of images with associated text from Wikipedia and Wikimedia Commons in 108 different languages. We are, however, only interested in English for this work. While  [53]  have multiple strategies to find text related to a given Wikipedia image, such as its Commons' caption, we use only the contextual paragraph as text source in order to mimic the downstream KVQAE setting. The resulting English subset of WIT yields 400K infobox images/articles that correspond to 1.
This stage consists in fine-tuning the model on a downstream KVQAE dataset, which provides visual questions (t q , i q ) and relevant visual passages (t + p , i + p ). Following  [2] , all layers of the model are tuned during this stage. A subtlety of this stage is the selection of irrelevant visual passages (t - p , i - p ). As mentioned in Section 2, it was shown to be essential to DPR  [27] , and it is more generally important for contrastive learning  [26]. In  [32] , irrelevant passages are mined with BM25 to train DPR. However, we suppose that this is suboptimal for ECA and ILF as BM25 will only mine textually-plausible passages but not visually-plausible ones. Therefore, we use the system provided by  [32]  to mine irrelevant passages. It is a late-fusion of DPR, ArcFace  [8] , CLIP, and ImageNet-ResNet  [20].
This leads to different training setups between DPR (used as a baseline) and our models. However, we have experimented both for DPR and found no significant differences 10. We use the same KB as  [32] , which is based upon KILT's Wikipedia and Wikidata images of the corresponding entities. It consists of 1. 5M articles (thus images/entities) split into 12M passages of at most 100 words as in Stage 1. Visual questions in ViQuAE are split into train  (1, 190) , validation (1,250), and test  (1, 257)  without overlap between images' URLs  [32]. We do not experiment with KVQA  [50]  for the following reasons: (i) it is generated automatically from Wikidata so our text-based KB has a poor coverage of the answers; (ii) it comprises yes/no questions for which passage relevance cannot be assessed automatically.
Implementation Details Our code is built upon PyTorch  [42] , Hugging Face's transformers  [60]  and datasets  [33]  (itself wrapping Faiss). It is freely available along with the data and trained models  11. To train ECA, we use the same hyperparameters as  [32]  for DPR, themselves based upon  [27]. In particular, we use a learning rate of 2 × 10 -5 along with the Adam optimizer  [29]. It is scheduled linearly with 100 and 4 warm-up steps for stages 2 and 3, respectively. However, for ILF, we found, based on the validation set, that it converged faster with a learning rate of 2 × 10 -3 and a constant scheduler during Stage 2. We believe this is because ILF fully freezes BERT in Stage 2, so it does not require careful scheduling or a small learning rate. Dropout  [54]  is applied in BERT and after projecting embedding with W c and W t with a probability of 0.
Results The retrieval models are evaluated in two different ways: (i) by computing standard IR metrics on visual passage retrieval; (ii) by feeding retrieved visual passages to a reader module that is tasked with extracting the concise answer to the question, thus achieving KVQAE. Put differently, either evaluate whether the system is able to retrieve a relevant passage for the question or whether it is able to answer the question. We find both metrics to correlate. Ablation studies are carried out with IR metrics. ViQuAE is based upon TriviaQA, so it is only distantly supervised: the answer is considered correct if it string-matches the ground truth and, likewise, a passage Table  1 : IR evaluation on ViQuAE. l: Number of frozen layers during Multimodal ICT. Superscripts denote significant differences in Fisher's randomization test with p ≤ 0.
Conclusion and Perspectives We have presented a new pre-training method, Multimodal Inverse Cloze Task, for Knowledge-based Visual Question Answering about Named Entities. Multimodal ICT leverages contextual images in multimodal documents to generate visual pseudo-questions. It enables the use of more complex multimodal fusion models than previously proposed late fusion methods. Consequently, our method improves retrieval accuracy over the latter by 10% relative-MRR, leading to a 9% relative-F1 improvement in downstream reading comprehension (i. e. answer extraction), on the recently introduced ViQuAE dataset. We believe it is thanks to cross-modal interactions, which are prohibited by late fusion. More precisely, we qualitatively observed that cross-input interactions occurred between the image of the visual question and the text of the KB, which counteracts the heterogeneity of visual depictions.
Nodal Approach In the nodal approach, each node represents volumes of material or components, while inter-node connections represent the flow paths of thermal and mass fluxes (walls, doors, windows, etc. ). The graphic representation of the model has the same appearance as an electrical diagram; hence, the analogue method was used, where: The nodal approach is a one-dimensional approach that is currently employed in the most common Building Energy Simulation (BES) software, such as TRNSYS, Energy-Plus  [43]. They all focus on describing the dynamic changes in indoor relative humidity and temperature, which occur due to variations in outdoor climate, hygrothermal loads, and the moisture buffering effect of indoor condition loads. Boyer et al. [44]  described a physical model of the building that is obtained by assembling thermal models of each element such as walls and glass windows.
m -2 ] the radiation flux density,. Q [kg. s -1 ] is the mass flow rate, and A [m 2 ] is the area. The subscripts ae, ai, se, si, ce, ci, lwe, re, ri, rm, swe, swi, w stand for air exterior, air interior, surface exterior, surface interior, exterior convection, interior convection, long wave exterior, exterior radiation, interior radiation, inside radiant mean, short wave exterior, short wave interior, and wall, respectively. Except WUFI, which integrates a full moisture model, most of the building energy simulation software such as TRNSYS and EnergyPlus primarily focuses on simulating temperature variations and energy requirements in specific spaces on a large scale  [43]. As a consequence, in these tools, the moisture exchange models at the wall scale rely on a simplified approach that does not account for the coupling of heat and moisture transfer phenomena through the building envelope.
This model belongs to Fraunhofer Institut Für BauphysiK IBP. It was then integrated into the WUFI (Wärme Und Feuchte Instationär) software package, one of the most widely used commercial tools for modeling heat and moisture transfer. This software is used to model the hygrothermal behavior of multilayered building elements exposed to the natural climate. This model has been used by several researchers  [59] [60] [61] [62] [63] [64] , with generally satisfactory results. As shown by many authors, it is possible to explicitly present the temperature by modifying the saturation pressure term as it can be expressed as a function of temperature. The novel system of coupled heat and mass transfer equation becomes  [65] [66] [67] [68] [69] [70] : dw dϕ dϕ dt = ∇ D ϕ w ∇ϕ + D T w ∇T ( 41 ) dH dT dT dt + dH dϕ dT dϕ = ∇ D T e ∇T + D ϕ e ∇ϕ (42) where D T e [W.