dataesr/scientific-paragraphs-categorization huggingface dataset:
Dataset 2 In this dataset, each recording was scored by only one of the six experts (see Table  2 ). In a first preliminary step, the global kappa, κ G2 , and the percentage agreement between automatic and visual scorings were computed. Note that since DS2 can only provide pairwise comparisons κ G2 cannot be compared with κ G1. The recordings scored by a given expert were then pooled together to provide for each expert a pairwise comparison with autoscoring. The second step aimed at assessing the intra-expert variability according to the scoring condition, i. e. across datasets. Is visual scoring impacted by the scoring condition and how? We used autoscoring as reference since it is blind to the scoring condition. The auto-visual agreements on DS2 were compared with the corresponding pairwise agreements obtained on DS1.
A third step was to demonstrate that the comparison of auto-visual agreement between DS1 and DS2 was not biased by the difference in size of the datasets. We computed the agreement on random subsamples of DS2 data, each subsample having the same number of epochs as in DS1. 1000 subsamples of DS2 were randomly computed to provide a distribution of kappa coefficients and percentage agreements. The fourth step aimed at assessing the intra-expert variability according to the sleep condition (BAS, EXT, BEF and REC). We used again autoscoring as reference, since it is also blind to the sleep condition. Kappa coefficients and percentage agreement were computed for each expert and in each sleep condition. Finally, as a fifth and last step, the contingency matrix yielded Se and PPV for all sleep stages, without distinguishing between experts.
Dataset 1 The figure  1  shows the qualitative comparison between all 7 scorings on night EXT night of subject #0981. Out of 4,384 epochs, 39 (0. 89%) were classified as artefact by the automatic scoring versus 38 epochs (0. 87%) by the experts. However, the number of epochs labeled as artefact varied substantially between experts: 0, 0, 1, 1, 33 and 3 respectively. Moreover, none of these epochs were identical across scorers, while only one epoch was in common between the automatic and the visual classifications. We thus discarded a total number of 76 (1. 7%) epochs, leaving 4,308 clean epochs for subsequent statistical analysis. In the first step, the inter-expert agreement as a function of the number of experts N e (fig. 2 ) showed that the larger the number of experts, the lower percentage agreement and kappa variance, while mean kappa remained stable.
The percentage agreement across all six visual scorers was 68. 7% and the corresponding kappa was 0. 81. experts out of 6, N c =20 for 3 experts out of 6, etc. ) for each possible number of experts (N e = 2 to 6), yielding distributions of agreements. Note that for N e =6 experts, the number of possible combinations is reduced to N c = 1. Concerning the inter-scorer variability ([min -max], µ=mean), the 6 pairwise kappa coefficients between autoscoring and each scorer separately, A vs. V i , ranged [0. 72 -0. 79], µ=0. 75, and the 15 pairwise kappa coefficients between visual scorers, V i vs. V j , ranged [0. 73 -0. 87], µ=0. 81 (fig. 3a ). The corresponding percentage agreements A vs. V i ranged [79. 1% -84. 7%], µ=81. 7%, when V i vs. V j ranged [79. 8% -90. 5%], µ=85. 6% (fig. 3b ). For each of the 2 graphs, the 3 plots on the left illustrate the inter-scorer variability on DS1 data and confirm that comparisons with scoring consensus provide far better agreements since the doubtful epochs are discarded.
The left-most plot, together with the right-most plot, illustrate the intra-scorer variability between datasets (scoring condition, i. e. temporal proximity with the training sessions), using automatic analysis as the reference. As expected, pairwise comparisons yielded lower agreements than comparisons involving consensus scoring since the latter discards ambiguous epochs. Accordingly, the kappa between A and the visual full consensus V all was 0. 91, whereas kappa between each individual expert and the consensus of the other experts ranged [0. 90 -0. 98], µ=0. 95 (fig. 3a , 3 rd boxplot). Similarly, the percentage agreement was 93. 1% between A and V all , and [92. 9% -98. 5%], µ=96. 5% between each V i and the consensus of the other visual scorings (fig. 3b , 3 rd boxplot). The global kappa over DS1 obtained by comparing all scorings (V+A) was κ G1 = 0.
3%) of disagreement between experts, leaving 2,959 epochs of full visual consensus. The contingency matrix is shown in Table  3. Automated scoring used as benchmark (1st column) shows that the highest disagreement among experts is observed for epochs autoscored N1, while higher agreement is observed among experts for Wake and N3. Among these 1,349 non-consensual epochs between experts, Aseega agreed 1,255 times (93. 0%) with at least one expert. For each visual scorer, V i , evolution of the percentage agreement with automatic scoring according to the dataset. The agreement decrease between datasets is not only a mean effect (µ=-3. 7%), as illustrated in figure  3b , but also a systematic effect for every single expert, ranging from -1. 3% to -5. 6%. Regarding intra-expert variability according to the sleep condition using autoscoring as reference, figure  7  shows that agreements on long duration nights (EXT and REC) were globally lower than the ones obtained on shorter duration nights (BAS and BEF).
Discussion In this article, we assessed the reliability of sleep scoring, as conducted by 6 sleep experts and an automatic algorithm. The main results are threefold. First, a very good agreement is achieved between visual and automatic scoring, although automatic scoring is shown to marginally lower the overall scoring agreement. Second, agreement between experts decreases as the number of experts increases, suggesting that the issue of inter-expert variability is not restricted to a limited number of epochs that are difficult to score. Third, when referred to a reproducible method, such as autoscoring, visual scoring demonstrated intra-expert variability across datasets and type of recordings, thus introducing uncertainty in the measurement of the sleep variables of interest. Visual-automatic agreement is similar to visual-visual agreement Overall, scoring agreement between scorers in DS1 was excellent 34 (Conger's kappa 0.
Scoring conditions Our results highlighted this intra-expert variability on different data by using autoscoring as the reference. The systematic decrease in visual-automatic agreement for each expert between DS1 and DS2 (fig. 6 ) originates from variability in visual scoring as the automatic scoring used in this study is fully deterministic and reproducible. 23 The main assumption is a "drift" over time in visual scoring. 1  As noticed by Danker-Hopfe and colleagues,  19  training sessions represent an essential tool to achieve a homogeneous interpretation of the scoring rules. In our study, prior training sessions occurred right before DS1 scoring, which was then followed by DS2 scoring. This might explain why we observe an excellent inter-scorer agreement for the DS1. In contrast, DS2 scoring seemed to reveal the individual drift, away from common scoring rules.
Second, autoscoring methods should be precisely evaluated to identify potential limits. The result of the Aseega algorithm in this study confirmed the general trend towards a dramatic improvement in methods that long suffered from questionable performances and poor reputation. A number of automatic methods are currently available, yet, precise assessment and comparison on common datasets using common metrics  1, 2, 13, 41  is an open question: the issue of inter-scorer variability is transferred among autoscoring methods. These methods differ in nature or by their objective: some are based on multichannel PSG data analysis,  [42] [43] [44]  others use a single EEG channel only  23, 45, 46  or EOG only,  47  some are limited to Wake-Sleep scoring. 48 This should not necessarily affect expectations in terms of reliability.
Published methods also differ by the way they are assessed,  39, 42-46, 49, 50  study protocol (population studied and number of experts) and comparison methodology (reference setup and statistics). Providing public sleep databases is an ongoing and useful process for several years in the USA (www. sleepdata. org  35 ), in Europe (www. physionet. org  51 ) and more recently in Canada (www. ceams-carsm. ca/en/MASS  52 ). It will be challenging to provide data fulfilling the requirements of all methods and … satisfactory scoring reference for these data, as demonstrated in this paper. Our contribution to this open database policy will be to open-source the data used in this study, including both visual and autoscoring.
Regarding the A vs. V comparison, the global kappa was κ G2 = 0. 70 and the percentage agreement 79. 0%. The 6 pairwise kappa coefficients between automatic scoring and each expert ranged [0. 67 -0. 73], µ=0. 70 (fig. 3a, right boxplot). The corresponding percentage agreements ranged [76. 4% -80. 8%], µ=78. 7 (fig. 3b, right boxplot). Intra-expert variability across the two datasets using autoscoring as reference (step2) was assessed by pairwise agreements between DS1 and DS2. It systematically decreased from DS1 to DS2 for all experts (µ=-3. 7%, fig. 6): kappa coefficients ranged [0. 68 -0. 73], µ=0. 70 and percentage agreements ranged [77. 2 -81. 2], µ=79. 0. Computation on reduced data showed that there was nearly no impact of having 20-fold less data in DS1 than in DS2.
Our research was guided by the following overarching question: What is the frequency and nature of lexical and structural alignment in online interactive encounters involving a language learner? Given the different contexts and modalities our data come from, we explore the following subquestions: 1. How does the CMC environment (videoconference and text chat) affect lexical and structural alignment? 2. How does the interactional partner (language learning peer, L1 teletandem peer, and L1 tutor) affect lexical and structural alignment? 3. What structures are aligned in the different target languages (Chinese, French, and German), and how does their alignment compare across those languages? Data Sets and Participants Our data consist of 10 hours of online interactions of university students that are L2 users of a foreign language.
The 4 hours of video-based chat conversations provided a corpus of 26,643 words (M = 6,660, SD = 955) and 2,459 turns (M = 614, SD = 55). The written chat data stem from eight students (half female, half male, aged 21 or 22) who were learning German at a northern university in the United Kingdom. They had all been in a German-speaking country on study abroad and had reached C1 level of the Common European Framework of Reference for Languages (CEFR) according to their recent university grades. For the present study, each of them participated in two task-based 30-minute text chat interactions (using the written modality of Skype), one with a language learning peer, one with a German-speaking tutor they were not familiar with. Task instructions asked students to discuss with each other five measures that would help to get into contact with German native speakers during study abroad (peer chat) or to advise the tutor on what five photographs (from a set of 10 given pictures) would be best to use on the study program's website (for details, see  Michel & O'Rourke, 2019).
First, it was difficult to adapt the text chat framework developed by  Michel and Smith (2018)  to study lexical alignment in videoconference. In fact, the spoken nature of data results in a large number of disfluencies, a well-documented phenomenon in oral interaction  (Pallaud, Rauzy, & Blache, 2013). Disfluencies typically include self-repairs and repetitions of syntagma. Automatic analyses of N-grams therefore needs to distinguish between genuine lexical entrainment and these phenomena. A future direction in this sense would be to use National Language Processing (NLP) tools such as SPPAS  (Bigi, 2015)  or Marsatag  (Laboratoire Parole et Langage, 2013)  to automatically detect disfluencies and discard them in the study of lexical and structural alignment. The second issue we encountered is the multimodality of CMC and especially of videoconferences, where the multimodality typical of spoken interaction  (verbal, prosody, gestures, mimics, etc.
Conclusions and Practical Implications Our study demonstrates that lexical and structural alignment are both present and observable in different SCMC data sets. Given that we analyze small and heterogeneous corpora, the results of the present study are to be taken as providing venues to build future hypotheses and a methodological framework. We identified different variables that will need to be examined in future studies: the CMC environment, the status of interlocutors, the difference between lexical and structural alignment, and target structures naturally elicited by specific tasks, among others. On a methodological level, the apparatus we adapted from the literature proved to be a useful tool to examine this phenomenon across data sets. However, some adaptations will be necessary for future studies.
Note. x = structure was not present in respective dataset or did not reach threshold of minimally 5 occurrences. that + complement x x dass + Nebensatz have + complement avoir + complément 有 + complement haben + Komplement Existential there Il y a existentiel existential有 es gibt existenziell Relative clause Phrase relative Relative clause Relativsatz Infinitive structures construction infinitive; x zu + Infinitiv; verbes modaux Modal verbs Modalverb + Infinitiv be + prepositional être + complément 在 (place) sein + PP phrase (PP) de lieu Participle structure x x haben/sein + Partizip Passive voice Passif Passive voice Passiv go + PP aller + complément de 去 + place x lieu be + complement être copule 很 + adjective sein Kopula EMERGENT Présent proche x x STRUCTURES ( je vais faire in present quelque chose) datasets (French, Chinese, j'aime / j'aime pas 喜欢 x German) x x W-Fragesatz x x Fragesatz mit Inversion État mental Mental state Mental state Verb ( je trouve, je (觉得) (ich glaube/ pense…) finde/denke…) x x Konjunktiv x x Genitiv x x wenn… dann Cite this article:Michel M, Cappellini M (2019).
We manipulated the modulation deptheither 95% or 45% of the peak amplitude -resulting in two levels of auditory reliability. These sound stimuli were then convolved with a head-related transfer function (elevation = 0 o , azimuth = 0 o ; PKU-IOA HRTF database,  Qu et al. , 2009)  to promote perceived co-localisation of the sound with the visual stimuli projected to an external screen. The brightness of the visual flicker was jittered on a trial-by-trial basis to de-correlate the visual contrast from sound intensity across trials. Across trials, auditory and visual rates, auditory reliability and the task-relevant modality were manipulated factorially in a 4 (visual rates) by 4 (auditory rates) by 2 (auditory reliabilities) by 2 (task relevance) design, resulting in 64 multisensory conditions.
ACKNOWLEDGMENTS We thank  Robin A. A. Ince  for support during initial development of MEG analysis pipeline,  Andreas Jarvstad ,  Jan Balaguer  and  Valentin Wyart  for helpful discussions. YC was funded by the  University of Oxford Clarendon Fellowship , the  Department of Experimental Psychology , and a  Brasenose-Kwai Cheong Studentship. This project was funded by the  European Research Council  (to CK ERC-2014-CoG; grant No  646657 ),  UK's Biotechnology and Biological Sciences Research Council  (grant  BB/M009742/1  to BLG), and a  Human Brain Project  award to CS. BLG was further supported by  ANR-16-CONV-0002  ( ILCB ),  ANR-11-LABX-0036  ( BLRI ) and  Excellence Initiative of Aix-Marseille University  (A*MIDEX).
Software and Algorithms MATLAB R2015B MathWorks https://www. mathworks. com/ Psychtoolbox-3 Brainard, 1997; Pelli, 1997 http://psychtoolbox. org/ R version 3. 4. 0 R Development Core Team https://www. r-project. org/ SPM12 Wellcome Trust http://www. fil. ion. ucl. ac. uk/s pm/software/spm12/ FreeSurfer Fischl, 2012 https://surfer. nmr. mgh. harvar d. edu/ PKU & IOA HRTF database Qu et al. , 2009 http://www. cis. pku. edu. cn/au ditory/Staff/Dr. Qu. files/Qu-HRTF-Database. html Fieldtrip Oostenveld et al. , 2011 http://www. fieldtriptoolbox. or g/ BADS Acerbi and Ma, 2017 https://github. com/lacerbi/bad s SMACOF De Leeuw and Mair, 2011 https://cran. rproject. org/web/packages/sma cof/index. html MTIMESX James Tursa, 2009 https://www. mathworks. com/ matlabcentral/fileexchange/25 977-mtimesx-fast-matrixmultiply-with-multidimensional-support VBA toolbox Daunizeau et al.
Method participants. One hundred and seventy preschoolers of 4-(83 children, mean age = 3. 7, SD = 0. 3 year, 37 girls) and 5-years of age (87 children, mean age = 4. 7, SD = 0. 3 year, 41 girls) participated to the experiment. All children were native French speakers, and none was color-blind. Children were schooled in 6 different schools in Marseille (France) and surroundings. Informed consent was received from a parent or legal guardian for each child. Children also gave verbal assent to participate. Although individual demographic information was not collected, most children were Caucasian and from middle to high socioeconomic status backgrounds, reflecting the demographics of the geographical area. Data were collected from January to June 2016. The study was approved by the ethics committee of the University of Aix-Marseille, and was performed in accordance with relevant regulations.
Acknowledgements This work was supported by the  Swiss National Science Foundation  and the  Agence Nationale de la Recherche [Binational  grant  SNSF-100019L_156521 & ANR-14-CE36-0011-01  to  V. Camos  and  A. Blaye ]. We thank the  Inspecteur de l'Education Nationale of Marseille , the participating teaching teams, the children and their caretakers. We also thank  A. Leyat  and  C. Reymond  for their help in collecting data.
Each mini-block consisted of one target trial and a varied number of non-target trials preceding the target trial (between one and six non-target trials). Conditions were counterbalanced across participants. In total, the eight blocks lasted around 25 min. This task protocol is summarised in Fig  1. In order to sample different features of participants' ongoing experiences during the n-back task, we used multidimensional experience samples (MDES;  [19] ). This technique uses selfreport data to assess the contents of experience on a number of dimensions. Participants answered 13 questions (see Table  1 ) on a 4-point scale ranging from 0-1. The first thought probe asked participants to rate their level of task focus ("My thoughts were focused on the task I was performing") on a sliding scale from 0 (completely off-task) to 1 (completely on task).
Resting-state fMRI Procedure. All 207 subjects completed a 9-minute resting-state MRI scan during which they were asked to rest in the scanner with their eyes open. Resting-state fMRI acquisition. Structural and functional data were acquired using a 3T GE HDx Excite MRI scanner with an 8-channel phased array head coil (GE) tuned to 127. 4 MHz, at the York Neuroimaging Centre. A gradient-echo EPI sequence was used to collect data from 60 axial slices with an interleaved (bottom-up) acquisition order with the following parameters: TR = 3 s, TE = minimum full, volumes = 180, flip angle = 90˚, matrix size = 64 × 64, FOV = 192 × 192 mm, voxel size = 3 x 3 × 3 mm. A minimum full TE is the shortest TE the scanner would permit for the scanning parameters used (on average ~18ms). This was selected to optimise image quality (as opposed to selecting a value less than minimum full which, for instance, would be beneficial for obtaining more slices per TR).
MRI acquisition. Structural and functional data were acquired using a 3T GE HDx Excite MRI scanner with an 8-channel phased array head coil (GE) tuned to 127. 4 MHz, at the York Neuroimaging Centre. Structural MRI acquisition was based on a T1-weighted 3D fast spoiled gradient echo sequence (TR = 7. 8 s, TE = min full, flip angle = 20˚, matrix size = 256 x 256, 176 slices, voxel size = 1. 13 x 1. 13 x 1 mm). Functional data was recorded using single-shot 2D gradient-echo-planar imaging (TR = 3 s, TE = minimum full, flip angle = 90˚, matrix size = 64 x 64, 60 slices, voxel size = 3 x 3 x 3 mm 3 , 180 volumes). A FLAIR scan with the same orientation as the functional scans was collected to improve co-registration between scans.
Data analysis Principal component analysis (PCA) of MDES data. We created a single 8694-by-13 matrix in which each row was a MDES probe (8694 = 207 participants x 42 probes) and each column was a question. We used PCA with varimax rotation in SPSS version 24 to reduce the dimensionality of these data using scree plots to determine that 4 principal components should be extracted (see S1 and S2 Figs). The resulting data were used as regressors of interest at the participant level in the self-reference experiment. The component names were decided by naming each after the highest loading item if the component was anchored at one end (e. g. Detail). If they were anchored by opposing scores (e. g. , Words and Images) we named them based on the distinction that these loadings imply (e. g. Modality).
Self-reference task analysis. For the analysis of neural activity during the self-reference task, functional and structural data were pre-processed and analysed using FMRIB's Software Library (FSL version 5. 0, http://fsl. fmrib. ox. ac. uk/fsl/fslwiki/FEAT/). Individual FLAIR and T1 weighted structural brain images were extracted using BET (Brain Extraction Tool). Structural images were registered to the MNI-152 template using FMRIB's Linear Image Registration Tool (FLIRT). The functional data were pre-processed and analysed using the FMRI Expert Analysis Tool (FEAT). Individual subject analysis involved: motion correction using MCFLIRT; slice-timing correction using Fourier space time-series phase-shifting; spatial smoothing using a Gaussian kernel of FWHM 6mm; grand-mean intensity normalisation of the entire 4D dataset by a single multiplicative factor; highpass temporal filtering (Gaussianweighted least-squares straight line fitting, with sigma = 100 s); Gaussian lowpass temporal filtering, with sigma = 2.
ox. ac. uk/fsl). Z-stat maps were generated for each EV; Self and Other. These maps were then registered to a high resolution T1-anatomical image and then onto the standard MNI brain (ICBM152). We performed two group-level analyses on the task data using FLAME. To identify the spatial distribution of the regions modulated by the nature of the referent during trait adjective assessment, we compared Self > Other and Other > Self. Both spatial maps are thresholded at Z = 3. 1 and are controlled for multiple comparisons (p <. 05, FWE). In addition we calculated the effect of task (Self and Other judgements) > rest (S3 Fig). To understand the relationship between neural processing during each referent and experience in the laboratory, we performed group-level regressions in which the spatial maps for each task, relative to baseline, where the dependent variables and the mean loadings for each individual for each MDES component were included as between participant continuous explanatory variables.