dataesr/scientific-paragraphs-categorization huggingface dataset:
Scaling KIF to Challenging Retrieval Settings KIF modules can be used in more realistic and challenging settings for knowledge retrieval that test the scalability of the module. In Figure  6 (a), we compare the Generative Transformer MemNet Baseline with KIF-Augmented Transformers in three settings. The first is the standard Wikipedia sentences provided by the dataset (average 34 sentences). Then, we extend to providing the model with the full Wikipedia article (on average, 57 sentences) and finally to multiple Wikipedia articles (on average, totaling 205 sentences), identified using the conversation's topic. This increasing size of available knowledge could be realistic for settings where it is unclear what information is most relevant, if filtering steps to preprocess the data remove potentially relevant information, or if information synthesis from multiple knowledge sources is necessary to produce a high quality generation.
As the Wikipedia knowledge becomes more difficult to identify, performance decreases, but still outperforms the baseline that uses the dataset-provided set of 34 sentences. Comparing the scaling capability of KIF to the standard Generative Transformer MemNet Baseline highlights the advantage of using KNN. The attention-based mechanism used in  Dinan et al. (2018)  struggles to identify salient information when given increasingly larger quantities of knowledge, unlike the KNN information fetch. We hypothesize the attention mechanism is challenged by softmax-ing over a larger quantity of inputs, as it can be difficult to make sharp distinctions.
Ablations Importance of Multiple Knowledge Sources. One benefit of the KIF module approach is that several modules can be combined, each capturing information from a different source. In both settings, Wizard of Wikipedia and Engaging ImageChat, two modules were used to incorporate multiple forms of knowledge -training utterances to capture the capability of a retrieval-based model and knowledge from Wikipedia or related chats based on image features. We perform here an ablation study to evaluate the impact of using only one source of information. As can be seen in Table 4, performance decreases when only one source of information is used (see Table  4 ). For Engaging ImageChat, this study also underlines the importance of being able to fetch in a multi-modal fashion. The general form of the KIF module -requiring only a feature vector to find nearest neighbors from -allows fetching on multiple modalities such as text and images.
In Table  4  text with the knowledge to identify relevant pieces of information. In Table  5 , we show that matching on more features can improve the quality of the retrieved information. Using only the encoding of the immediate previous utterance can improve results on Wizard of Wikipedia by 7 F1 points, but this is further improved by also leveraging the encoding of context (+1. 8 F1) and using the dialog turn number (+1 F1). These features are available in the datasets, and we leverage them to improve the relatedness of retrieved knowledge. Multi-Hop Retrieval with KIF. Work in memory networks  (Weston et al. , 2015; Sukhbaatar et al. , 2015)  employed multi-hop mechanisms. be useful when multiple sources are necessary or information is incrementally fetched. To emulate multi-hop memory mechanisms, we use KIF to retrieve relevant information for N = 2 or N = 3 fixed hops.
As the number of hops is fixed, the multi-hop operation remains differentiable. We do not allow the model to retrieve the same information in a second hop. We experimented in two settings. First, the same KIF module is used multiple times to fetch different information, and then all of the fetched knowledge is concatenated. Results are shown in Table  6  (top). Second, we examine spreading the fetches into different KIF modules at various encoder depths. This could be interpreted as the model learning to access more information each layer. As the model progresses deeper, more abstract and high level representations are built, which could allow different knowledge to be retrieved. Results are shown in Table  6  (bottom). In both multi-hop settings, no improvement in performance on the Wizard of Wikipedia dataset is observed.
We hypothesize this can be partially attributed to the construction of the dataset -as humans explicitly based their written dialog utterance on one knowledge sentence. Further, it is possible that concatenation brings together too much information for the model to incorporate, and thus adding additional fetches makes the retrieval more noisy. Effect of Gating. We analyze the effect of the gating mechanism by evaluating the capability of the gate to identify and focus on salient information. On Wizard of Wikipedia, we concatenate a third source of information: dialog turns from a completely different corpus called PersonaChat  (Zhang et al. , 2018). This dataset looks quite different -short utterances without factual knowledge -and should be easy for the model to identify as distinct from Wizard of Wikipedia.
Chat on Similar Images. While there are over 180K different images in this dataset, many of the images are similar. For example, conversations associated with two pictures of dogs could be relevant to each other. The model is able to use KIF directly on the current image features to fetch from around 180K different images and return 6 turns of related chat for each fetched image. Fetching from E 1 consists of identifying related image chats, or conversations on related topics. • E 2 is Training Utterances. Similar to the motivation for the previous dataset, we allow the model to identify training utterances that could be useful for responding in the current conversation. The scale of this fetching task is large: 350K dialog utterances. This could be interpreted as identifying utterances with similar structure to what the model would like to generate, and is complementary to the topic-based related image chats.
Results on the Wizard of Wikipedia dataset. We implement the Retrieve and Refine and Response Generation with MR approaches, all with Reddit Pre-Training, and evaluate them on Wizard of Wikipedia. The Seen test set consists of conversations on topics seen at training time, and the Unseen test set consists of conversations about new topics that were not in the training set. Model Test F1 Test F1 (Seen) (Unseen) Retrieval Baselines Retrieval Transformer MemNet (Dinan et al. , 2018) 15. 4 12. 4 Generative Baselines 2-Stage Generative MemNet (Dinan et al. , 2018) 18. 9 17. 4 Generative Transformer MemNet (Dinan et al. , 2018) 16. 9 14. 4 + Reddit Pre-Training 17. 6 16. 3 Retrieve and Refine (Weston et al. , 2018) 18. 2 17. 9 Response Generation with MR (Qin et al. , 2019) 17. 5 16. 8 KIF-Augmented Transformer 25.
The Covid Symptom Study, a smartphone-based surveillance study on COVID-19 symptoms in the population, is an exemplar of big data citizen science. Over 4. 7 million participants and 189 million unique assessments have been logged since its introduction in March 2020. The success of the Covid Symptom Study creates technical challenges around effective data curation for two reasons. Firstly, the scale of the dataset means that it can no longer be easily processed using standard software on commodity hardware. Secondly, the size of the research group means that replicability and consistency of key analytics used across multiple publications becomes an issue. We present ExeTera, an open source data curation software designed to address scalability challenges and to enable reproducible research across an international research group for datasets such as the Covid Symptom Study dataset.
Introduction Mobile applications have enabled citizen science  [1] [2] [3] [4]  projects that can collect data from millions of individuals. The Covid Symptom Study  5  , a smartphone-based surveillance study on self-reported COVID-19 symptoms started in March 2020, is an exemplar of citizen science. As of 14th October 2020, the study contains 189 million self-assessments collected from more than 4. 7 million individuals, provided as daily comma separated value (CSV) snapshots, that are made available to both academic and non-academic researchers to facilitate COVID-19 research by the wider community. The primary challenge in curating and analysing the data is its scale. Scale adds complexity to otherwise simple operations. Python-based scientific computing libraries such as Numpy  6, 7  and Pandas 8 are ubiquitous in the academic community, but they are not designed to scale to datasets larger than the amount of Random Access Memory (RAM) on a given machine.
Commodity hardware, as of 2020, is typically equipped with 16 to 32 GB of memory. Although it is possible to procure hardware with more RAM, this involves server-grade pricing above 64 GB of memory, and doubling memory only doubles the size of the data that can be handled. Alternatively, data can be moved to a datastore; either traditional, relational databases such as PostgreSQL or other types of datastore including key-value stores such as CouchBASE. Each type of datastore comes with its own design philosophy and set of performance trade-offs  9. Datastores are capable of running queries on larger-than-RAM datasets, but all come with the need to learn either a new language or application programming interface (API), and an installation and maintenance burden, whether installed locally or deployed in a cloud.
Results ExeTera's performance has been benchmarked against a combination of artificial data and data from the Covid Symptom Study. We examine performance and scalability of ExeTera for key operations relative to Pandas, the most popular Python library for analysing tabular data. We also demonstrate Exetera's ability to generate a journaled dataset from snapshots allowing longitudinal analysis that is able to account for destructive changes between corresponding rows of the different snapshots. Finally, we present an example of ExeTera's analytic capabilities. Performance was measured on an AMD Ryzen Threadripper 3960x 24-core processor with 256 GB of memory. All processes were limited to 32GB of memory. The data was read from and written to a 1TB Corsair Force MP600, M. 2 (2280) PCIe 4. 0 NVMe SSD.
Importing to ExeTera ExeTera has an import function that is used to convert a set of source CSV files to HDF5  10  , given a JSON (JavaScript Object Notation) schema document describing the datatypes. This operation streams the CSV files, transforming the string source field data into one or more target fields in the HDF5 dataset. Table  1  shows the time taken and resulting table row counts for importing the Covid Symptom Study snapshot from the 8th October 2020.
Data subset reading In the general case, a typical piece of analysis will only involve a fraction of the available fields in a dataset. We test the ability of ExeTera and Pandas to load subsets of the Covid Symptom Study dataset, from the patient and assessment tables respectively. 2  shows that ExeTera (given its use of HDF5 as its serialized format) is orders of magnitude faster than CSV due to each entire field being loadable with effectively zero parsing overhead and as monolithic reads. CSV on the other hand, must be parsed in its entirety and the resulting collections incrementally built. CSV files are loaded through Panda's read_csv function. 3  show that Pandas is unable to load multiple fields at this row count. Again, CSV files are loaded through Panda's read_csv function.
Left join operation The join operation is the most memory-intensive operation that is typically carried out on tabular datasets. ExeTera implements highly scalable versions of left, right and inner joins; left join is the most typical join performed so we focus on its performance here. A left join operation involves the mapping of data from one table onto another table, based on the key relationships between the two tables. Values in the right table are mapped to values in the left table. For example, patient data may be joined to assessment data so that assessments can be processed with patient-level features such as age or BMI (body mass index). An illustrated example of a left join can be seen in Figure  1. To test the performance of the join operator when ExeTera and Pandas are used, we generate a dataset composed of a left primary key (int64), a right foreign key (int64) and 1, 2, 4, 8, 16, and 32 fields respectively of random numbers corresponding to entries in the right table (int32).
The first ten rows of the join dataset in the two field case is shown in Table  4. The resulting join performance for both is shown in Table  5. The left and right tables have the same number of rows for each test, listed in the first column. It should be noted that ExeTera organises data into groups, which may be considered logical tables, but the user typically loads the data on a field by field basis when working with it. ExeTera's merging API allows the user to think in a tabular fashion by accepting tuples of ExeTera fields, which only load their contents when specifically requested. ExeTera achieves its scalability through two techniques; firstly, by calculating the map and then applying it to each field in turn, and secondly, by exploiting the natural sorted order to convert the mapping into a streaming merge.
Discussion In this work, we present ExeTera, a data curation and analytics tool designed to provide users with a low complexity solution for working on datasets approaching terabyte scale, such as national / international-scale citizen science datasets like the Covid Symptom Study. ExeTera makes this possible without the additional complexity of server-based datastores, and thus simplifies access to such datasets for both academic and independent researchers who are versed in the Python scientific programming ecosystem. ExeTera provides features for cleaning, journaling, and generation of reproducible processing and analytics, enabling large research teams to work with consistent measures and analyses that can be reliably recreated from the base data snapshots. Its ability to store multiple snapshots in a journaled format enables researchers to perform full longitudinal analysis on otherwise unjournaled datasets and facilitates the ability to move between snapshots whilst being able to properly explore the impact of doing so on analyses.
ExeTera has been a key part in enabling analysis of the Covid Symptom Study dataset, including being used for analysis in the following manuscripts  [13] [14] [15] [16]. Although ExeTera was developed to provide data curation for researchers working on the Covid Symptom Study, this software is being developed to be generally applicable to large-scale relational datasets for researchers who work in Python. As such, ExeTera core functionality is provided as a python package that is imported by ExeTeraCovid, the collection of algorithms, scripts, and other resources specific to the Covid Symptom Study. While our goal is to closely emulate a Pandas-style API, additional work is still required to ensure this is available to the user across the whole API. The highly scalable sorting API and journaling API, for example, are accessed through the low-level operations API rather than through abstractions to more closely emulate Pandas, and work is ongoing to make the API more consistent in the presentation and manipulation of the data as logical tables.
ExeTera is currently built on top of Numpy and Pandas functionality. The ability to scale is provided through implementations of key operations that can stream large collections from drive. This enables processing of large datasets, but, at present, ExeTera doesn't take advantage of multiple cores or processors, nor is it able to process across a cluster. Future development will make use of Dask 17 a library designed to convert operations on Numpy and Pandas to a directed acyclic graph of sub-operations that can be distributed to multiple cores and nodes, and focus on provision of the elements of scalable processing, such as multi-key argsorts and tableless merges that Dask lacks. Dask is also integrated with more specialised back-ends such as Nvidia's RAPIDs  18  , which enables execution of distributed graph processing across GPU clusters.
The integration of Dask should allow ExeTera to execute over datasets well into the multi-terabyte range. Reproducibility relies on immutability of the algorithms deployed during analysis. At present, ExeTera achieves this through a convention that algorithms are treated as immutable once implemented and deployed. A more robust system is being designed to provide algorithmic immutability without this constraint. HDF5 has proven to be fragile to interruptions while data is being written to it; an interrupted write is capable of rendering the entire dataset unreadable and so all writes must be protected from user interrupts and other exceptions. Additionally, HDF5 does not allow the space from deleted fields to be properly reclaimed from the dataset. This must be done separately through use of the 'h5repack' tool.
The primary benefit of HDF5 to ExeTera is its ability to flexibly store contiguous fields of data for rapid reading. This can also be achieved through use of alternative columnar data formats such as ORC  19  or Parquet  19, 20  or, alternatively, through use of the file system as a datastore. Storing individual fields as serialized Numpy arrays and field metadata in JSON allows for transparent, robust dataset serialization that can be explored on the file system. ExeTera is made available as open source software released under Apache License 2. 0.
Methods The Assessments: 189. 2 million assessments with 66 fields. Patients are asked to give regular assessments through the app that cover their current health status and symptoms, aspects of their lifestyle such as potential exposure to COVID-19, and, in early versions of the schema, any COVID-19 tests that they have had. Tests: 1. 455 million tests, with 26 fields. Test records are kept for each COVID-19 test that a patient has had along with the evolving status of that test (typically 'waiting' to some result). Diet: 1. 563 million diet study questionnaires with 104 fields. These ask people at several time points about their dietary and lifestyle habits. Assessments, tests and diet study questionnaires are mapped to patients via ids that serve as foreign keys. This dataset is delivered as daily snapshots in CSV format.
RAM Scale (1GB to 16 GB) This is the scale at which the dataset entirely fits in the computer's RAM. Commodity laptops and desktops used by researchers typically have between 16 and 32 GB of RAM. Loading the data can inflate its memory footprint depending on the datatypes used, and operations can multiply memory requirements by a small constant factor, but provided peak memory usage does not dramatically exceed RAM, researchers can make use of programming languages with numerical / scientific libraries such as Numpy or Pandas to effectively analyse the data.
Drive Scale (16 GB to 1 TB) At drive scale, only a portion of the dataset can fit into RAM at a given time, so specific solutions are required to effectively stream the dataset from drive to memory. Datastores become a more compelling option at this scale, as they already have memory efficient, streaming versions of the operations that they support, but their usage may not be desirable due to the need to learn a new language or API, and the installation and maintenance burden they represent. This is the scale of dataset that ExeTera currently targets.
Indexed string fields Indexed string fields are used for string data where the strings may be of arbitrary length. The data is stored as two arrays; a byte string of all of the strings concatenated together, and an array of indices indicating the offset to each entry. Numeric / logical fields Fields which contain a combination of strings to be converted to strongly typed values and empty values, for example "", "False", "True" are converted to the appropriate numeric / logical dataset and a corresponding filter field indicating whether a value is present for a given row of the field. Fields containing string values can be processed as categorical fields if specified as such. Categorical fields Categorical fields map a limited set of string values to a corresponding numeric value. A key is stored along with the field providing a mapping between string and number.
A categorical field can also be specified as leaky, in the case that the field is a mixture of categorical values and free text. In this case, a value is reserved to indicate that a given row doesn't correspond to a category, and an indexed string field is created for free text entries. Datetime / date fields Datetime fields store date times as posix timestamps in double precision floating point format. The schema can also specify the generation of a 'day' field quantising the timestamp to the nearest day, and can also specify whether the field contains empty values, in which case as filter is also generated, as with numeric fields. Schema file format Importing data from CSV requires a schema file that describes the fields and the type conversions that they should undergo. The ExeTera schema file format is a JSON format.
Each table is described by entry inside of a JSON dictionary labeled schema. Each entry in this dictionary is the name of the table followed by the table descriptor. This has up to three entries. The first is primary_keys, which lists zero or more fields for the dataset that together represent the primary key for the table. The second is fields and contains all the field descriptors for the table. The third is foreign_keys and contains the names of foreign keys in the table and which other tables they relate to.
Journaling of snapshots into a consolidated dataset Data for the Covid Symptom Study project is delivered as a series of timestamped snapshots. The unanonymised data generated by the Covid Symptom Study app is stored in a relational database or similar datastore, that is not accessible to query by the broader research community. Instead, the data needs to first be anonymised and bulk exported to CSV format. The database is a live view of the dataset, however; users can update data through the app, and, unless the database is explicitly journaled and each entry made immutable, the prior states are erased. As such, a row corresponding to a given entity in two different snapshots can be contain conflicting values. When each snapshot is large, the scaling problem is exacerbated by having to reconcile multiple snapshots.
Covid Symptom Study-Specific Cleaning and Processing The Covid Symptom Study data schema has seen rapid iteration since its inception, due to a number of factors. Firstly, the initial app was rapidly released to allow users to contribute as soon as possible after the pandemic was declared. Secondly, the evolving nature of the pandemic, particularly around prevalence in the population and availability and type of tests has necessitated structural changes to the schema. Thirdly, this dataset is novel in terms of its scale and deployment for Epidemiological analysis, and the initial wave of papers published by the research group has fed back into the schema. Public health surveillance campaigns such as the Covid Symptom Study impose time constraints to software development, with frequent changes in database structure and intense versioning to accommodate iterative refinements.