dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2019b; Asai et al. , 2020). Recently, multiple works show that retrieval systems entirely based on dense representation and approximate nearest neighbors were competitive with traditional approaches. Such models can be trained using weak supervision in the form of question-answer pairs  (Karpukhin et al. , 2020) , or pretrained using a cloze task and finetuned end-toend  (Guu et al. , 2020; Lee et al. , 2019). Generative question answering was mostly considered in previous work for datasets requiring to generate answers, such as NarrativeQA  (Kočiskỳ et al. , 2018) , CoQA  (Reddy et al. , 2019)  or ELI5  (Fan et al. , 2019). These datasets were generated in a way that answers do not correspond to spans in support documents, thus requiring abstractive models. Raffel et al. (2019)  showed that generative models are competitive for reading comprehension tasks such as SQuAD  (Rajpurkar et al.
Model Reading. Our generative model for open domain QA is based on a sequence-to-sequence network, pretrained on unsupervised data, such as T5 or BART  (Raffel et al. , 2019; Lewis et al. , 2019). The model takes as input the question, as well as the support passages, and generates the answer. More precisely, each retrieved passage and its title are concatenated with the question, and processed independently from other passages by the encoder. We add special tokens question:, title: and context: before the question, title and text of each passage. Finally, the decoder performs atten-1 lucene. apache. org 2 spacy. io 3 github. com/facebookresearch/faiss tion over the concatenation of the resulting representations of all the retrieved passages. The model thus performs evidence fusion in the decoder only, and we refer to it as Fusion-in-Decoder.
Experiments In this section, we report empirical evaluations of Fusion-in-Decoder for open domain QA. Datasets. We consider the following datasets, and use the same setting as  Lee et al. (2019) : • NaturalQuestions  (Kwiatkowski et al. , 2019)  contains questions corresponding to Google search queries. The open-domain version of this dataset is obtained by discarding answers with more than 5 tokens. • TriviaQA  (Joshi et al. , 2017)  contains questions gathered from trivia and quiz-league • SQuAD v1. 1  (Rajpurkar et al. , 2016 ) is a reading comprehension dataset. Given a paragraph extracted from Wikipedia, annotators were asked to write questions, for which the answer is a span from the corresponding paragraph. Evaluation. Predicted answers are evaluated with the standard exact match metric (EM), as introduced by  Rajpurkar et al.
(2016). A generated answer is considered correct if it matches any answer of the list of acceptable answers after normalization. This normalization step consists in lowercasing and removing articles, punctuation and duplicated whitespace. Technical details. We initialize our models with the pretrained T5 models  (Raffel et al. , 2019) , available in the HuggingFace Transformers library. 4 We consider two model sizes, base and large, containing respectively 220M and 770M parameters. We fine-tune the models on each dataset independently, using Adam (Kingma and Ba, 2014) with a constant learning rate of 10 -4 and a dropout rate of 10%. We train the model for 10k gradient steps, with a batch size of 64, using 64 Tesla V100 32Gb. We evaluate models every 500 steps and select the best one on the validation set based on the Exact Match score.
During training on NaturalQuestions 4 github. com/huggingface/transformers and SQuAD, we sample the target among the list of answers, while for TriviaQA, we use the unique human-generated answer. For TriviaQA, answers in uppercase are normalized by converting all letters in lowercase except the first letter of each word, using the title Python string method. For both training and testing, we retrieve 100 passages (unless said otherwise), and truncate them to 250 word pieces. Following the results of  Karpukhin et al. (2020) , passages are retrieved with DPR for NQ and TriviaQA, and with BM25 for SQuAD. We generate answers by using greedy decoding. Comparison to state-of-the-art. In table 1, we compare the results obtained by Fusion-in-Decoder with existing approaches for open domain question answering.
Introduction New methods of measuring ground motion are significantly reducing the cost of data collection. Two new types of equipment strongly contribute to this cost reduction. The first is nodal data, i. e. data from experiments with a very high number of observation points using low-cost sensors, are now becoming common. As an example, more than 5200 high frequency sensors were deployed over a period of six months in and around Long Beach, California, USA (e. g. Lin et al. , 2013). The second is distributed acoustic sensing (DAS) technology, using fiber-optic cables, which is currently being tested and deployed in many locations. As an example,  Jousset et al. (2018)  deployed a 15-km-long fiber-optic cable layout on Reykjanes Peninsula, SW-Iceland, with a distance of 4 meters between sampling points, to study structural features in the Reykjanes Oblique Rift.
Both types of equipment can generate tens to hundreds of terabytes of data at a small fraction of the cost of using traditional seismometers and geophones for equivalent data volumes. As the price of collecting data becomes drastically cheaper, a corresponding and dramatic increase in the volume of data being collected provides a potential scientific bonanza. As dataset sizes increase into the range of tens to hundreds of terabytes however, barriers to storing, transporting, and processing the data begin to appear. The largest collections of openly available seismological data (e. g. , as managed by IRIS, RESIF, and GEOFON) measure the volume of their decades-spanning repositories in hundreds of terabytes, yet some new (e. g. DAS)  experiments are gathering data volumes well in excess of a hundred terabytes -a significant fraction of the total data volumes presently stored at the seismological data centers.
One might suggest that just a small fraction of the data that are collected should be preserved as was suggested decades ago when broadband digital data was first introduced, but tremendous scientific value has been found in the continuous data that was recorded. Storing these data in perpetuity is a significant and daunting challenge, as is delivering datasets exceeding a few tens of terabytes. High-performance/ high-transaction computational resources (HPC/HTC) are increasingly necessary to process these data and these resources are not typically provided by the repositories, nor are they co-located with the repositories. This explosion in data volumes is just beginning and the data centers that have traditionally been the repositories of data for the entire community are not only being asked to host these data for the wider research community but also to accommodate access to the computational resources that are needed to process these data sets.
Appropriate data management practices by the data centers must address large data transport, reduced-volume derivative data products, increased access to HPC/HTC, and data formats that are HPC/HTC-friendly. IRIS, RESIF, and GEOFON, all of which are dedicated to providing free and unrestricted access to their data holdings, have attempted to identify the needs of the community and to look for common solutions and best practices for managing very large data sets while maintaining their traditional data services. We begin by presenting the results of soliciting user expectations for large data services through a survey of large data providers and users. Next, we describe the challenges posed by large data from its submission to a data center, to archiving, format considerations, data distribution issues, and finally to the diversity of use cases that data centers provide and will need to provide to serve the research community.
User Survey A survey was conducted of selected and self-identified large data users; 37 responses were received. The survey responses indicate that researchers anticipate the following: • Large volume use, from 1 to 300+ terabytes, of traditional data (e. g. broadband seismic) and newer data types such as nodal deployments or DAS. • Use of both existing data access tools and mechanisms (web services, Python-based clients, miniSEED) and newer data access and processing such as HDF5, Zarr, xarray, especially for larger data sets. The respondents reported the maximum size of the datasets they expect to be working with in the next 3-5 years. We classified them in three different categories of data volume: 21 small (1-9 TB), 11 medium (10-50 TB), and 5 large (50+ TB). We consider the small volume data users relatively well served by current data center capacities, especially as they anticipated using wellestablished data formats, processing tools and access mechanisms.
Medium volume data users, however, are observed to split their data requests into many small requests. What we summarize below are the survey results primarily for the medium and large data users. The results for four important variables are shown in Figure  1. What data type are the raw data (e. g. broadband, nodal, DAS)? The medium data users primarily identified broadband seismic data, nodal data and a bit of DAS, while large data users indicated mostly DAS, with some nodal and broadband seismic. From which data centers do you expect to request large data volumes? Users from all categories indicated using a wide variety of data centers, which we interpret to indicate they will access data from wherever they can get it, with no particular preference. In which data format(s) would you prefer to work with large data (miniSEED, SAC, PH5, ASDF, HDF5, etc.
)? Both medium and large data users anticipated a use of a variety of data formats, but primarily miniSEED, HDF5 (PH5 and ASDF), with a few cases of Zarr (see Data and Resources). However, for users planning to work with 20+ TB, the option of miniSEED reduces considerably in comparison to HDF5-based formats or other less standard formats. Would you process the data with standard codes? With your own code? Using third party frameworks and libraries (e. g. ObsPy)? Users from all categories indicated the most common codes they expect to use would mostly be their own code, supported by ObsPy  (Beyreuther et al. , 2010)  or MATLAB. Which programming language(s) do you, or would you expect to, use for data processing and analysis? Python is the most indicated programming language of choice, followed by MATLAB, C/C++, Fortran and Julia.
Where would you prefer the data to be delivered: your own compute infrastructure, a cloud system, an HPC center? Many medium and large users indicated that they would like to transfer data to their own computer infrastructure or their institutional infrastructure. Many also indicated a preference to deliver data to an HPC center one of whom anticipated relying on commercial cloud providers. Would you be willing to pay for the data processing resources (e. g. compute) if the data are available within a commercial cloud or other environment that is not free to use? Many users are willing to pay (some dependent on grant allowance) for processing. However, as in previous questions, among users working with 20+ TB there seems to be a clear trend in favor of paying. Some pointed out that it should be an option to transfer the data to a location where they would not be required to pay for processing.
Please briefly summarize your current large data processing workflow, trouble points identified/foreseen, and how it can be improved. Most users indicated that they use largely ad hoc data processing workflows, with a few identifying the need for parallel processing and processing-ready data access such as from HDF5 containers. The common challenge for many is efficient access to large volume data, either due to access interface issues or limited transfer capability. Which software would you use to access/download large data volumes (libraries, applications, etc. )? Users of all scales report they expect to use the existing tools and mechanisms of web services, ObsPy, or directly in their own codes (in Python or Julia). A few report their anticipated use of larger platforms such as Pangeo, direct access to object storage (e.
Diversity of Data Requests From the survey and from the experience of the IRIS, GEOFON, and RESIF data centers, requests for medium and large data sets span a range of time and spatial scales. On one end of the spectrum are studies that require hours/days of high sample-rate data collected from densely spaced sensors, and at the other end of the spectrum are studies that require decades of broadband data from stations distributed around the globe. The survey clearly demonstrates that the DAS experiments represent the largest volume datasets; however, both the survey and data center experience show that Medium and Large data volumes from traditional seismic stations are increasingly in demand. Two typical uses of such Medium and Large datasets include studies based on cross correlation techniques and studies using machine learning methods (ML).
In such cases, large datasets are created from years-long time series from a large number of individual seismic stations either at global, regional or local scales and the data needed for a comprehensive study may even be distributed across several data centers on different continents. In contrast, requests for node data or other experimental data with high frequency sampling are often based on data from a single data center. Users may want access to the entire dataset or to a specific time or spatial slice. DAS datasets can be much larger than anything that traditional data centers supporting the scientific research and education communities have experience in managing. Like nodal data, researchers may want access to the entire dataset or to subsets of the full dataset; they may also want to process those data without transporting them or need to have them delivered to an HPC center rather than to their home institutions.
Because of these processing needs, data centers should be aware of the data formats that are efficient for high-performance computation and be able to deliver data in some of those formats. The management and organization of data and user services need to be flexible enough to handle not only the requests for medium and large datasets, but also for the diversity of standard data requests for a small number of seismic stations and short time periods (small datasets). Data centers need to encourage and enhance services associated with data requests using criteria such as station location, instrument type, sampling rate, data quality, data repository location, or more advanced parameters associated with earthquake parameters, propagation path, local geology at the seismic station, etc. MUSTANG  (Casey et al.
Data Submission In the cases of Large-N or DAS experiments the volume of data submitted to a data center could be hundreds of terabytes. Here, it is important to make a distinction between temporary and permanent experiments. For temporary experiments, data are usually acquired in the field and either transmitted or physically transported to the data center. Field data storage devices can be mounted on the data center's network and the data ingested into the system. In the case of permanent deployments, data are usually transmitted to the data center in real-time by means of protocols like SeedLink. However, for high volume data such as from DAS experiments, realtime transmission of the full-resolution data may not be practical. One option for DAS installations is to pre-process the data at the station to reduce the data volume before transmitting them to a data center.
Data Archive Planning Designing an efficient data management system for seismological data requires consideration of several aspects: the storage hardware, the data format(s), and the way data will be accessed. These three factors are connected; storage system performance varies depending on user access requirements, which are supported by the format of the data. Requests that require gathering small snippets of data from many files require higher performance storage systems than requests for data in a single file. The internal data management of seismological data centers is not specified by FDSN; data centers are free to manage their data as they see fit and store the data they have in their repositories and archives in whatever format suits their use cases. The data provisioning system, fdsnws-dataselect (see Data and Resources), must deliver data in the miniSEED format, but the data centers are free to use different data formats to archive data and perform the conversion on-the-fly before sending them to users.
To date, the miniSEED format has been very popular as a storage format among data centers because it is highly compatible with the software ecosystem in the community. The main advantages are that there is no need to convert on-the-fly when sending most data to users and that datasets can be formed by a concatenation of miniSEED records that can be streamed as they are retrieved from the storage media. The ability to stream data directly avoids the need to stage the dataset on the server side before sending it. The disadvantage is that miniSEED has no inherent indexing capability; users typically use the file system as a means to organize and locate data, which may require a huge number of files for medium and large datasets. For medium and large datasets, the HDF5 format is popular among users of HPC.
Two popular HDF5-based formats in our community are PH5  (Hess et al. , 2018)  and ASDF  (Krischer et al. , 2016). Initial experimentation has indicated that a reduction in storage space is possible using PH5, likely due to the longer data segments being compressed. A redesign of the PH5 format to make it more versatile is expected to be completed in 2021. The new format is expected to be usable not only for nodal data, but also for DAS, Magnetotelluric, geodetic, and other data types. A generic HDF5-based solution could have some advantages not only for data centers but also for users needing to process their large datasets in an HPC-friendly format. The main disadvantage for data centers is that HDF5 formatted data cannot currently be streamed on the fly as they are read by the data provisioning systems (see Data Distribution).
Data Description There exist comprehensive metadata standards in seismology that are well suited for large nodal data sets or other classical seismic data sets. However, newer, emerging data types such as DAS are not as well handled by these standards where the concepts of a network, station, location and a channel are different and the fundamental instrument responses are new. Therefore, one of the most difficult tasks for data centers attempting to identify and implement standard services is to standardize DAS metadata through the definition of new metadata schemas. StationXML, the international standard for seismic metadata, is not a good fit with the technical features of DAS interrogators and the generated datasets. There is a need to tackle this in a smart way in order to make these data available and interoperable with all popular tools used within the community, which expect typical seismic waveforms with some minimum set of requirements on their metadata, and the possibility of mixing different data types.
A non-technical (and still pending) issue is how to name DAS data sampling points. If the first approach to standardize and distribute DAS experiment data is to generate derived products as suggested by many users, there are some details to be considered. Should DAS experiments be registered by the FDSN in order to get a formal network code? If this is not the case, how could a user identify or select the required sampling points? To have a network code registered would allow the dataset to be uniquely distinguished from other experiments. This would also allow one to formally assign a DOI and the proper citation to the experiment, as agreed and expected in the community. At levels finer than the network, should each sampling point be considered a station, a location or a channel? If we consider that each fiber could have thousands of sampling points with a spatial separation as small as of 2 to 4 meters and a length of tens of kilometers, a station designation seems to be the best fit.
As stations have a unique position, we cannot consider such a deployment to be a single-station-multiple-channels configuration. We note that the ISC applies a 1-km station rule for the International Registry of Stations that is generally followed by the community. Quoting ISC: "Because of the need for accurate station positions for hypocenter location programs, a new international code is assigned if a station is moved more than one  (1)  kilometer from the previous location". Although having locations codes more than a kilometer away from the station does not technically violate the rule about stations that move, we believe that it would be extremely confusing to have channels with a location up to 50 kilometers away from their associated station. We also note that some large array deployments (like the Norwegian Seismic Array Network, see Data and Resources) identify each array element with different station names.
g. "tap test") during the deployment phase. Standard and generic codes, benchmarks and guidelines provided to/by the community would be desirable, and information about the postprocessing needs to integrate the metadata. Some of these topics are being discussed within community user groups, like the Distributed Acoustic Sensing Research Coordination Network (DAS RCN, see Data and Resources) and hopefully some proposals to standardize the description of DAS experiments will emerge. The needs and related challenges from the irruption of new technologies, like DAS interrogators, could be the triggering factor to discuss the long-term evolution of our current standards (e. g. StationXML) and to potentially explore more generic formats like SensorML, which is widely used in Geophysics and generic enough to encompass all that StationXML can express.
Data Distribution Currently, the FDSN-standard suite of web services (fdsnws-station and fdsnws-dataselect) are used by many researchers to discover and download data. From April to September 2020, 90% of the fdsnws-dataselect requests received delivered less than 5 MB of data (Figure  4 ). The miniSEED format along with the fdsnws-dataselect webservice are well designed for this purpose. However, users may make many small requests to optimize their data retrieval success and assemble the complete dataset that they desire on their computers. The size of the datasets needed by the community is probably larger than is observed in Figure  4. It is still unclear what the download pattern for Large-N or DAS data would be, but downloads of entire datasets are expected. The distribution of large data sets (50+ TB) is challenging due to bandwidth constraints.
As mentioned previously, one of the main results of the survey we conducted was that the full resolution data is not needed for most studies, particularly for DAS. Instead, a set of products derived from the original data would satisfy most users. The derived data products are often a significantly reduced volume making them more feasible for archiving and distribution by existing data centers and systems. While the derivative products are useful in reducing the burden of transferring bulky data, the need to transfer large volumes is not eliminated for all users. Another significant challenge is the lack of available services to handle the transmission of large volumes. The current data transfer standard in global broadband seismology (fdsnwsdataselect), defines a synchronous web service interface that accepts arbitrary data selections and is expected to begin returning data with little or no delay.