dataesr/scientific-paragraphs-categorization huggingface dataset:
APPLICATION Bayalpata Hospital EMR Implementation Electronic medical record implementation necessarily took place in a phased manner (Box 2). In preparation for the initial rollout of Bahmni at Bayalpata Hospital, a team from ThoughtWorks came to the hospital and together with a staff trainer, trained the doctors, mid-level providers, nurses, and registration staff over a period of two weeks in July of 2014. Over the subsequent six months, we undertook several off-site and on-site design iterations and planning sessions. On February 10, 2015, Bahmni went live at the hospital with the patient registration module. The EMR was deployed to the laboratory and outpatient departments over the following week. This phased rollout allowed the deployment team to adequately support each department during the initial stage.
This team meets quarterly to discuss ways to further optimize the EMR. Additionally, Bahmni and the OpenMRS development community have created a system and culture responsive to user feedback, and as a result, ThoughtWorks continues to enhance the user interface to make Bahmni as intuitive and efficient as possible. In order to take full advantage of the EMR system, work still remains to educate providers, habituated in paper-based systems, to write good notes and use the data for better clinical decision-making.
Replication and Scale A major rationale for the OpenMRS-based Bahmni system was that it is affordable (see Supplemental Table  2 ) and designed for the types of providers found among public healthcare institutions in South Asia. From the beginning, Possible engaged government partners at the local and national levels. Indeed, using an OpenMRS-based system like Bahmni was itself predicated on the Ministry of Health's commitment to OpenMRS as the back-end program for its healthcare system. This engagement requires constant dialogue, as there are frequent shifts at the leadership levels within the Ministry. Policy priorities themselves have been dynamic, particularly after the devastating April 25 and May 12, 2015 earthquakes. The national dialogue surrounding EMR standards and possibilities has been influenced by successful implementation of a full-service EMR in a rural district hospital, leading our government partners to wonder whether a similar system might be employed more broadly across government facilities.
Subsequently, when Possible initiated a new public-private partnership at a district hospital in Dolakha, a district heavily affected by the earthquakes, Possible quickly and successfully deployed the EMR within four weeks. The IT infrastructure required to host an open source EMR system like Bahmni requires initial investment in materials and expertise. In our experience, however, ongoing costs are much lower due to availability of remote support, the lack of licensing fees since we have used open-source software, a substantial OpenMRS design and implementation community on the web, and regional experts for maintenance of the system. These resources are available within Nepal, even in rural areas. Indeed, partly as a result of deployments in Achham and Dolakha, the government has made commitments to expand an OpenMRSbased EMR throughout the country, using private sector technology companies to deploy and maintain the systems.
Acknowledgments We wish to express our appreciation to the  Nepal Ministry of Health  for their continued efforts to improve access to healthcare in rural Nepal and for their advocacy around use of electronic medical records in the public-sector healthcare system. We also wish to express our deepest appreciation to our partners at the Achham District Health Office and Public Health Office as well as all of the allied healthcare workers serving Achham District. These include the  Female Community Health Volunteers, (Sub) Health Post and Primary Care Center  staff members, and Achham District Health Office staff members. We also wish to thank  Lena Wong  for her assistance in preparing the supplemental figures.
We present a novel multi-lingual database of natural dyadic noviceexpert interactions, named NoXi, featuring screen-mediated dyadic human interactions in the context of information exchange and retrieval. NoXi is designed to provide spontaneous interactions with emphasis on adaptive behaviors and unexpected situations (e. g. conversational interruptions). A rich set of audio-visual data, as well as continuous and discrete annotations are publicly available through a web interface. Descriptors include low level social signals (e. g. gestures, smiles), functional descriptors (e. g. turn-taking, dialogue acts) and interaction descriptors (e. g. engagement, interest, and uidity).
INTRODUCTION Natural and realistic human-human interactions are needed in order to support the analysis and increase understanding of human social behavior. e use of training materials based on those interactions is crucial for a variety of tasks, such as improving the automatic detection and interpretation of social signals, or to create computational models of realistic human social behavior. An interesting aspect of human social interaction is the occurrence of unexpected events, such as conversational interruptions, that impact the interactant's behaviors and cognitive state. is topic is understudied in the areas of A ective Computing  [36]  and Social Signal Processing  [48]. is paper presents NoXi: the NOvice eXpert Interaction database. NoXi is a database that is annotated and thoroughly designed for studying and understanding human social behavior during an information retrieval task targeting multiple languages, multiple topics, and the occurrence of unexpected situations.
NoXi is a corpus of screen-mediated face-to-face interactions recorded at three locations (France, Germany and UK), spoken in seven languages (English, French, German, Spanish, Indonesian, Arabic and Italian) discussing a wide range of topics. Our rst aim was to collect data from natural dyadic knowledge exchanges in the context of the H2020 project ARIA-VALUSPA  [47]  (Arti cial Retrieval of Information Assistants -Virtual Agents with Linguistic Understanding, Social skills and Personalized Aspects). NoXi has been designed from the onset with the aim of being used by a wider audience of researchers in a variety of applications other than information retrieval (c. f. Section 3). e dataset o ers over 25 hours of recordings of dyadic interactions in natural se ings, featuring synchronized audio, video, and depth data (using a Kinect 2.
RELATED WORK In recent years, a large number of multimodal datasets on human interaction have been collected and published  [29, 38, 45, 46]. We brie y review some by dividing them into two groups, according to the number of interlocutors. Dyadic interactions. In these datasets, interactions include two interlocutors, who are recorded carrying out both structured and unstructured conversations. ere are numerous examples of such datasets, with a wide range of applications, such as speech recognition  [15] , behavior analysis  [50] , segmentation, emotion recognition  [12]  and depression detection  [16]. Arguably, one of the most popular datasets of one-to-one interactions is SEMAINE  [30]. A large audio-visual dataset created originally as part of an iterative approach to building virtual agents that can engage a person in a sustained and emotionally-colored conversation, the SEMAINE dataset was collected using the Sensitive Arti cial Listener (SAL) paradigm.
Following this paradigm, conversations were designed to include an abundance of nonverbal features, which both sustained conversation and signaled emotion. SEMAINE has been successfully applied to many computer vision problems, such as facial muscle action (FAC) detection  [23] , head nod and shake detection  [21] , non-verbal communication estimation  [19] , segmentation  [33]  and emotions  [41]. Other relevant multi-modal datasets featuring dyadic interactions include: the Cardi Conversation Dataset (CCDB)  [3] , an audio-visual database focusing on non-scripted interactions that do not predetermine the participants' roles (speaker/listener); the MAHNOB-Mimicry dataset, designed to analyze mimicry in dyadic scenarios where subjects act with a signi cant amount of resemblance and/or synchrony  [43] ; the SEWA project 1  , which include video-chat recordings, audio transcript and hand-gesture annotations of human interactions; and the Interactive Emotional Dyadic Motion CAPture (IEMOCAP) dataset, which contains markers on the face, head, and hands, of interview participants during scripted and spontaneous spoken communication scenarios  [10].
Furthermore, an interesting collection of multimodal datasets can be found at TalkBank  [27] , a web-accessible database of audio-visual recordings of both human and animal communication. One of the major bene ts of TalkBank is that it also includes the transcripts of the conversations recorded. Multi-party interactions. In these datasets, the number of interlocutors is not constrained. Popular examples of such datasets are the Belfast storytelling dataset  [29] , which collects spontaneous social interactions with laughter, the Multimodal multiperson corpus of laughter in interaction  [34]  which collects multimodal data of laughter with focus on full body movements and di erent laughter types, and the AMI Meeting Corpus  [44] , which collects multimodal data from recordings of meetings. In NoXi we focused on dyadic interactions.
However, contrary to previous work, we had an innovative setup that integrated mediated face-to-face interactions, i. e. participants interacted through a screen in di erent rooms, focused on information exchange and retrieval on a rich variety of topics, and it has been recorded in multiple languages (mainly English, French and German). Moreover, the dataset is grounded on a recording protocol thoroughly designed for emphasizing the emergence of unexpected situations during the interaction, such as conversational interruptions.
NOXI e idea behind NoXi was to obtain a dataset of natural interactions between human dyads in an expert-novice knowledge sharing context. In a recording session one participant assumes the role of an expert and the other participant the role of a novice. When recruiting participants, potential experts o ered to discuss about one or more topics they were passionate and knowledgeable about, whereas novices applied to a recording session based on their willingness to discuss, learn more and retrieve information about a topic of interest among those o ered by experts. A matching of interests was found when a novice chose an expert's topic, then the dyad was recorded. is served the purpose of obtaining spontaneous dialogues on a variety of di erent topics for which the participants were passionate/knowledgeable about.
Design Principles We prepared the recording protocol with the following design principles. Setting. e schema in Figure  1  shows the physical setup. We opted for a screen-mediated recording for a twofold purpose, rst it allowed us to record a face-to-face conversation without the need of multiple cameras recording from di erent angles as in classical face-to-face se ings of other corpora (e. g. [29, 34] ). Secondly, this setup is closer to a scenario where a virtual agent is displayed on a screen. Participants were recorded while standing to meet the second design principle listed as follows. Data. We aimed at capturing full body movements (e. g. postural changes, torso leaning) in addition to facial expressions, gestures and speech. Furthermore, we wanted to enrich the dataset with the above mentioned data and possibly new formats not being captured in existing databases (e.
g. Kinect depth maps). Interaction. We wanted to record spontaneous interactions but at the same time we were interested in (1) knowledge transfer, (2) information retrieval and (3) occurrences of unexpected events (e. g. interruptions). Participants were allowed to continue their conversation until it reached a natural end. is results in quite long interactions (for such a database): the minimum duration for a recording session was set to 10 minutes. Participants. We primarily recruited participants in our research facilities, but also from our immediate social surroundings. erefore we obtained dyads of both colleagues as well as entirely unacquainted persons, thus providing us the opportunity to record a number of zero-acquaintance situations  [2]. Unexpected Events. One of the aims was to obtain occurrences of unexpected events.
Participants were informed about the sole possible usage of the recorded data for scienti c research and non-commercial applications. Moreover, they had three (non exclusive) choices concerning the usage of their data: (1) data available within the ARIA-VALUSPA project's consortium only, (2) data available for dissemination purposes in academic conferences, publications and/or as part of teaching material, and (3) data available for academic and non-commercial applications to third-party users through the web interface described later in this section. e session was monitored in Room III and when both participants agreed to end, the experimenter(s) gave participants questionnaires (described in Section 3. 3. Finally, participants were debriefed and compensated.
Collected Data e experiment was conducted in three di erent countries -France, Germany and UK. e primary reason for recording in three demographically di erent locations was the aim of collecting large numbers of interactions of the three languages targeted in the ARIA-VALUSPA project. In addition to English, French, and German, we also collected a smaller number of recordings of four other languages (Spanish, Indonesian, Italian, Arabic). A summary of the recorded sessions is given in Table  2. For the three main languages English, French, and German, we had 40, 25, and 19 interactions. In total, 87 people were recorded during 84 dyadic interactions (some people appeared in more than one session). e total duration of all sessions was 25 hours and 18 minutes. For each participant (i. e. expert or novice) in a given session, Table  3  shows the data that has been collected during the sessions and that is publicly available for download, in addition to the annotations described in Section 4.
We also collected demographic information of our participants at the end of each session. ese data consisted of their gender, cultural identity, age and education level. e cultural identity was obtained by asking participants to select the country that most represented their cultural identity from a list of all countries in the world. Participants' age is in the range of 21-50 years old. A breakdown of participants per age and recording location is shown in Figure  3. In addition to demographic information, participants provided a self-assessment of their personality based on the Big 5 model (a. k. a. OCEAN)  [28]  by using the Saucier's Mini-Markers set of adjectives  [40]. Finally, we collected session speci c information that included the social relationship level between participants (e.
g. acquaintances, friends), the level of expertise on the discussed topic and the pro ciency level of the language spoken for that session. We were very pleasantly surprised by the large diversity of topics covered by our experts. A total of 58 topics were discussed, the diagram in Figure  4  illustrates the most recurrent ones in the three main languages of the database. English sessions had a large variety of topics including travels (5 sessions), technology (4), health (3), cooking (3), sports (2), politics (2) and many others. French sessions were mainly about video-games (4), travels (3), music (3) and photography  (2). Finally, German sessions included expert computer science elds (5), various sports (6), car driving, magic tricks and other trivia.
Availability NoXi is made freely available to the research community and for non-commercial uses. It is available through a web interface at: h ps://noxi. aria-agent. eu/. In the web interface, the data is organized in sessions that correspond to an expert-novice full recording with audio-visual data and annotations. e total size of the database is approximately 4 TB, however the database is searchable through the web interface. A user can select multiple criteria (e. g. language or topic of the session, participants' gender, etc…) and can choose the sessions to download from a list or results. Furthermore, a er selecting the sessions from the search results, a user can choose the type of les that s/he wishes to include in the download for the expert and the novice. Search results can be saved and shared with other NoXi users of the web interface.
ANNOTATIONS Since the recording of NoXi, considerable e orts have been spent to collect descriptions of the database. Due to the large amount of data we wanted to allow multiple annotators from several sites to contribute and share their labeling e orts in an easy and clear way. In addition and to further speed up the process, we decided make use of semi-automated solutions to accomplish the desired descriptions. In the following section we will rst introduce a novel annotation tool named NOVA implemented for this purpose. A erwards, we report on descriptions already accomplished and describe in detail the annotation of conversational interruptions.
NOVA Following the data collection, we were in need of a graphical tool to review and annotate the recorded data. Since the great majority of multi-modal corpora is limited to audio-visual data, there is a lack of tools with support for other type of signals, such as body or face tracking data. Hence, a database like NoXi can be viewed in parts only. e same is true if one aims to describe phenomena using di erent types of annotation schemes. For example, we might prefer a discrete annotation scheme to label behavior that can be classi ed into a set of categories (e. g. head nods and shakes), whereas a continuous scale would be more appropriate for variable dimensions like activation and evaluation. Since conventional tools only support either of the two schemes the annotation process is further complicated.
ARIA-VALUSPA aims at developing a novel graphical tool that overcomes the mentioned limitations and supports the visualization and description of arbitrary data streams using various annotation schemes. e tool is released under open-source license  5. Figure  5  shows one of the NoXi sessions displayed with NOVA. It features audio-visual, as well as, skeleton and face streams of both participants. Below the signals several annotation tracks are shown including a speech transcription, a categorical description of voice activity and ller words, and valence/arousal scores expressed on a dimensional scale. Moreover, NOVA o ers a truly collaborative annotation platform that allows annotators from di erent sites to combine their e orts and immediately share the progress with each other. is has been realized by adding a database back-end, where annotations are stored and can be accessed together with the annotated signals.
To create a new annotation session a user connects to a MongoDB server  6  where they can examine what kind of descriptions have been accomplished for the database so far. He or she then selects one of the pre-de ned schemes and the information that should be displayed during the annotation process. e tool then creates an empty track and downloads selected media les. A set of security policies prevents situations where a user accidentally overwrites the annotation of another user. Only users with administration rights can edit and delete annotations of other users. is way an admin user can divide up forthcoming annotation tasks among the pool of annotators. Once a user has nished a task he or she can mark this in the database to signal availability for new tasks. Tools are provided to merge the annotations from several users to create a gold standard.
However, the database back-end is not only meant to share e orts between human operators: it can also function as a mediator between human and machine. e idea is to minimize human labeling e ort by le ing the machine complete a task once su cient information is available to train a machine learning model. To this end, two strategies are available from the NOVA interface: (1) Session Completion: e annotator selects an annotation he would like to complete. e partly nished annotation is used to train a model and predict the remaining sessions. Optionally, predictions with a low con dence are highlighted. (2) Session Transfer: e annotator selects a number of completed sessions and chooses the sessions he wants the machine to nish. e labeled sessions are then used to train a classi cation model and predict the target sessions.
To de ne the automata, FUBI o ers a user-friendly XML based language. From the body we extracted gestures such as arm crossing or leaning front/back; from the head gestures such as nods and shakes. To capture the dynamic properties of the movements we calculated several expressiveness measures such as energy, uidity, or spatial extend  [5]. e emotional state of the user is currently described along three a ective dimensions: valence, arousal and interest  [25]. Each dimension is represented by a continuous score between 0 and 1, which we automatically derive from the user's voice. To train the speech models we relied on the "Geneva Multimodal Emotion Portrayals" (GEMEP) corpus  [4]. It contains 1. 2 k instances of emotional speech from ten professional actors, which we used to train Support Vector Machines classi ers using the popular LibSVM library  [14].
As feature set we took the ComParE 2013 set  [42] , which has been extensively demonstrated to be suitable for a wide range of paralinguistic tasks. A total of 6373 features were extracted on a per-chunk level using the OpenSMILE toolkit  [18]. Finally, manual transcriptions on word and sentence level are available for di erent languages. ese transcriptions help netune the acoustic and language models, as well as, estimate the performance of the Automatic Speech Recognition (ASR) system implemented in ARIA-VALUSPA.
Multimodal Signals of Engagement NoXi aims to provide data supporting research in embodied conversational agents (ECAs) for making them capable of interpreting users' non-verbal behaviors and prosody in real-time and associate it with di erent engagement variations (increase or decrease). is allows the agent to adapt to the user's behavior in order to maintain the desired level of engagement during the interaction. us, we describe an annotation schema for both experts and novices about engagement variation, non-verbal behaviors and prosody. Several coding schemes have been used for multimodal corpus annotation. An exhaustive one is the MUMIN multimodal coding scheme  [1]  that we adapt for NoXi as summarized in Table  4. In addition to NOVA, we rely on Praat  [8]  for prosodic and acoustic features annotations.
Conversational Interruptions Another NoXi's purpose is to provide a dataset for analyzing and studying humans reactions to unexpected situations during the interaction. Conversational interruptions represent one of those situations. In face-to-face conversations, interruptions can be considered as turn-taking violations  [7]  (e. g. , claiming the turn by interrupting the current speaker), but they can also serve as important social displays that re ect interpersonal a itudes (e. g. , dominance or cooperation) as well as involvement in the interaction  [32]. transitions. Table  5  shows a summary of the labels for both annotation tiers. e communicative state tier describes both interlocutors' activity in the conversation, thus resulting in four states (i. e. annotation labels) describing when no one speaks (NONE), the expert speaks (EXPERT ), the novice speaks (NOVICE) or both interlocutors speak (BOTH ).