dataesr/scientific-paragraphs-categorization huggingface dataset:
This work was supported by  Marie Curie Actions -BCMolMed  under grant agreement no. FP7-PEOPLE-2012-ITN-EID  and the  European Community 's  Seventh Framework Programme  under grant agreement no. 306157. Harald Mischak  is the founder and co-owner of  Mosaiques Diagnostics , who developed the CE-MS technology for clinical application. Akshay Bhat  is an employee of  Mosaiques Diagnostics. Bernd Mayer  is the managing partner of emergentec biodevelopment GmbH,  Austria. Paul.
Methods: We describe a method to convert a virtual human avatar (animated through key frames and interpolation) into a more naturalistic talking head. In fact, speech articulation cannot be accurately replicated using interpolation between key frames and talking heads with good speech capabilities are derived from real speech production data. Motion capture data are commonly used to provide accurate facial motion for visible speech articulators (jaw and lips) synchronous with acoustics. To access tongue trajectories (partially occluded speech articulator), electromagnetic articulography (EMA) is often used. We recorded a large database of phonetically-balanced English sentences with synchronous EMA, motion capture data, and acoustics. An articulatory model was computed on this database to recover missing data and to provide 'normalized' animation (i.
e. , articulatory) parameters. In addition, semi-automatic segmentation was performed on the acoustic stream. A dictionary of multimodal Australian English diphones was created. It is composed of the variation of the articulatory parameters between all the successive stable allophones. Results: The avatar's facial key frames were converted into articulatory parameters steering its speech articulators (jaw, lips and tongue). The speech production database was used to drive the Embodied Conversational Agent (ECA) and to enhance its speech capabilities. A Text-To-Auditory Visual Speech synthesizer was created based on the MaryTTS software and on the diphone dictionary derived from the speech production database.
Other active sensor equipment has been used such as Electro-Magnetic Articulograph (EMA) to record three dimensional speech articulation database to drive datadriven talking heads  (Sheng et al. 2011)  for computer assisted pronunciation training. Humans commonly employ speech reading in adverse listening conditions to facilitate speech perception  (Sumby and Pollack 1954). The ability to visually obtain phonetic information depends on watching facial movements that are produced by the speech articulators: mainly by the lips and jaw, and to some extent the larynx and tongue. These movements have been shown to be highly correlated with speech acoustics  (Yehia et al. 2002). Although the tongue is a partly occluded speech articulator, its movements provide useful information for visual speech perception.
EMA may be also used to synthesize acoustic speech from the variation of articulatory parameters  (Toutios et al. 2013). A promising next step is to use synchronous recordings from EMA and motion capture data systems  (Jiang et al. 2002; Engwall 2005). With such a setup, a large number of sensors can be placed on the speaker's face and tongue. In the present paper, we propose an innovative method to transform an existing ECA animated by interpolation between key frames (i. e. , with poor speech capabilities) into a talking head. First, we describe the recording and processing of a multimodal synchronous speech database. An Optotrak Certus (Northern Digital Inc. ) motion capture system and a Wave (Northern Digital Inc. ) electromagnetic articulography system were used to record an Australian speaker uttering a large set of phoneticallybalanced sentences.
This unique setup enabled lip, jaw, and tongue trajectories to be recorded synchronously. We built an articulatory model by decomposing each speech articulator movements separately using guided Principal Component Analysis (gPCA). Sensor positions were converted into values of articulatory parameters in order to be used to control most ECAs. Second, the ECA's original animation was modified: face and tongue key frames were transformed into articulatory parameters. Finally, we used the multimodal database to animate the ECA and used the MaryTTS software  (Schröder et al. 2011)  to create a multimodal text-to-speech synthesizer. This innovative approach can be applied to most ECAs (whose animation module is open) to improve their speech capabilities.
Multimodal database Method Setup An EMA system (Wave, Northern Digital Inc. ) and an active motion capture (mocap) system (Optotrak Certus, Northern Digital Inc. ) were used to record the position of sensors attached to the face and the tongue during a speech production session. These two systems were manufactured to record synchronously the position of their respective sensors together with the acoustic signal. There were 30 mocap active sensors attached to the speaker's face: 3 for jaw motion, 8 for lip motion, 6 for eyebrow motion and 4 for rigid head motion mounted on a headset. Four additional sensors were attached to the Wave transmitter to align the Optotrak and the Wave referential systems. There were also 6 EMA sensors: 3 glued (using dental glue) on the tongue (tongue tip -TT, tongue body -TB, and tongue dorsum TD) and 1 attached to the nasion and 2 to the tragus.
Data modeling The recorded data are multimodal in essence: face, tongue trajectories and acoustic signal. Several processing steps were necessary before using the database to animate the ECA. In fact, sensor trajectories cannot be used directly to animate the ECA because of affordance issues. The shape model of an ECA may significantly differ from human morphology. The articulatory modelling provided 'normalized' animation parameters that can be used to control the ECA even if the shape model is different from the speaker's morphology. These processes will be explained in the following subsections.
Acoustic segmentation The audio files were automatically segmented using the method exposed in the MaryTTS (http://mary. dfki. de/) import voice procedure  (Pammi et al. 2010). EHMM acoustic labeler from festvox  (Black and Lenzo 2007)  was used to generate label files from the audio files and corresponding transcriptions. The label files created by this Fig. 1  Sensor positions on the speaker's face and tongue. A headset with 4 sensors was used to estimate the rigid head motion. The Wave sensors (in red) were glued to the tongue tip, tongue body and tongue dorsum and attached to the nasion and tragus. The Optotrak sensors (in blue) were attached to the face and the headset procedure used the SAMPA phonetic alphabet and were stored as lab files. These files were then converted into Textgrid files by a custom-made Matlab (MathWorks, Inc.
, Natick, Massachusetts, United States) program. The software Praat  (Boersma and Weenink 2010)  was used to manually check and correct the segmentation. Therefore, for each audio file, a corresponding text file listed the series of phonemes and their timing information. From these text files, other segmentation files were created containing the series of diphones (i. e. , the part of speech comprised between successive stable allophones) and their timing information. These files were used to build the multimodal dictionary that was then used to generate synthetic multimodal speech.
Articulatory model The Optotrak and Wave devices recorded at different sampling rates. The Wave data were time-stamped by the Optotrak device with respect to the beginning of the Optotrak recording. The Wave data were downsampled (low-pass filtered at 20 Hz) to 60 Hz and, if acquisition timing differed between the Wave and the Optotrak devices, interpolated between two consecutive frames. After this step, the Wave and Optotrak data were sampled at 60 Hz and synchronized. Head movement (translations and rotations) was estimated and corrected using the Optotrak sensors positioned on the headset and the Wave sensors placed on the nasion and the tragus. Speech articulator movements did not affect these sensors. A modeling procedure was applied to the data with two specific aims: first, to extract meaningful parameters controlling an elementary articulator, and second, to remove artifacts and measure noise.
PCA was applied to rigid motion. The first two components explained more than 90 % of the total variance. An articulatory model was built using the method proposed by  (Gibert et al. 2005; Revéret et al. 2000; Badin et al. 2002). A pruning step (simple vector quantization) was applied to remove the frames in which sensor positions were too similar (Euclidian distance < 1. 0 mm). This step conditioned the data before building the statistical models. Then, the contribution of the different speech articulators (jaw, tongue and lips) and the eyebrows was iteratively subtracted. This subtraction consisted of an iterative application of PCA on subsets of landmarks. The procedure extracted 10 articulatory parameters; extreme variations of jaw1 and tongue1 are shown in Figs. 2  and 3    The maximum variations of the first articulatory parameter driving the jaw (jaw1) opening-closing movement are shown in Fig.
Data reconstruction The rigid head motion was estimated in terms of translations (Tx, Ty, Tz) and rotations (Rx, Ry, Rz) around the neck (the center of rotation was estimated at the same time). These movements were transformed into articulatory parameters using the following equations: α H ∧ ¼ argmin α H ∈½-3;3 N ∥P3D ∧ HEADSET -P3D HEADSET ∥ 2 P3D ∧ HEADSET ¼ RigidMotion m HEADSET ; m Hmvt þ α H eigv H À Á where P3D HEADSET corresponded to the actual position of Optotrak sensors placed on the headset, and P3D ∧ HEADSET corresponded to the estimated ones, m HEADSET corresponded to the average position of the sensors on the headset, m Hmvt corresponded to the mean rigid head motion of the model, α H corresponded to the rigid motion parameter values to be estimated and eigv H corresponded to the rigid motion model derived from PCA.
After this filtering, the recordings were reconstructed, i. e. , missing data were estimated using the articulatory model. The reconstruction error computed as the Euclidian distance between the recorded position of the Optotrak/Wave sensors and the reconstructed ones was M = 5. 41 mm and SD = 3. 15 mm. An example of data reconstruction can be found in the Additional file 1: Video S1. This dataset comprised a large amount of English phonemes in different contexts. A multimodal diphone dictionary was created. It contained for each diphone, the variations of the articulatory parameters driving the visible and partly occluded speech effectors. α ∧ ¼ argmin α∈½-3;3 M ∥m Face þ α eigv Face -P3D W AV EOPTO ∥ 2.
Avatar The avatar used in this study was a representation of an Australian performance artist, Stelarc. This 3D model was originally driven by a set of key frames controlling the visible and partially occluded speech facial articulators such as lips, jaw, and tongue. The full animation was originally created by linear interpolations between Fig. 4  Variation of the ten articulatory parameter values for the sentence: "Look at the windows and see if it's raining". The variations are nonlinear and cannot be replicated with linear interpolation animation. The database is segmented into phonemes. A multimodal diphone dictionary is created with this segmented database those key frames. Unfortunately, linear interpolations do not accurately replicate speech articulator movements. This is one of the reasons why we developed a new animation method.
Tongue articulatory parameters Because tongue key frames were not related to any speech articulation in the original animation, but only to meaningless geometric variations, an alternative method was designed. Each tongue sensor from EMA data was associated with a specific vertex of the 3D tongue mesh (which is composed of 50 vertices) of the original ECA face model. For each sample of the quantized EMA database, tongue postures were determined by estimating the best linear mixture of weighted key frames that minimized the distance between the EMA tongue sensor positions and the corresponding tongue mesh vertices. The least square estimation of the vector of weights α was simply performed by: α ∧ ¼ argmin α∈½-10;10 N ∥ X N i α i P3D K i -P3D EMA ∥ 2 where P3D K i corresponded to the position of the three selected vertices of the 3D tongue mesh for the key frame K i , α i corresponded to the weights applied to the key frame K i , and P3D EMA corresponded to the position of the three EMA sensors TD, TB and TT.
The number of key frames available in the original model was N = 9. The values of each weight α i were limited to  [-10; 10]. Examples of configurations found in the EMA database and the corresponding constrained 3D tongue mesh are visualized in Fig. 5. After this step, a quantized database of 3D tongue postures was created. For all the configurations of the EMA database, corresponding constrained 3D tongue mesh of 50 vertices was available. The reconstruction error computed as the Euclidian distance between the EMA sensor (TD, TB and TT) positions and the specific vertices of the 3D tongue mesh was M = 7. 07 mm and SD = 6. 94 mm. The same procedure as described in section 4. 2. 2 was used to build a tongue articulatory model using the database of 3D tongue postures in addition to the EMA database.
Overview The system proceeds in three steps. First, a module connects to the MaryTTS server (http://mary. dfki. de/) and asks for the list of phonemes and its duration for a given text. Second, the same module requests an acoustic signal corresponding to the given text. Third, the list of phonemes plus duration is sent to the Visual Text-To-Speech (VTTS) module, which searches the best list of diphone candidates and concatenates them to create articulatory parameter trajectories. The acoustic signal and the articulatory parameter trajectories are sent to the animation module, which plays the data i. e. , the ECA speaks and moves his speech effectors (jaw, lips and tongue) accordingly. The schematic representation of the system is shown in Fig. 7.
Text-to-Speech The first step of our system is to ask the MaryTTS server  (Schröder et al. 2011)  to generate an acoustic signal that corresponds to the new sentence. To this end, the MaryTTS server performs natural language processing to generate a list of phonemes and prosodic information (duration, pitch variation, intensity). Finally, the synthesizer creates a sound signal from this information using a cluster unit selection code derived from FreeTTS. In fact, the recorded speech signal was used to create a specific MaryTTS synthetic voice for our speaker following the procedure described in the voice creation module  (Pammi et al. 2010). This procedure performs feature extraction from acoustic and text data and then automatic segmentation/labelling. The procedure was bypassed at this stage to check manually the automatic segmentation/labelling (see subsection 4.
Even if there is a concatenation gap between two consecutive diphones as shown in Fig. 8 , this procedure cancels it while keeping Fig. 7  Schematic representation of the auditory-visual speech synthesis system. Given a new sentence to pronounce, the program acts as a MaryTTS client and asks for an audio signal corresponding to this sentence and to the list of phonemes with their duration (provided by the prosodic module embedded in the MaryTTS software). From the list of phonemes, a second program performs the visual synthesis. It searches the best series of diphones given selection and concatenation costs in the multimodal dictionary. This series is processed to match the expected duration and minimizes the gaps at each boundary. Finally, the acoustic signal and the variation of the articulatory parameters are passed to the 3D Player which animates the ECA accordingly.
Animation Acoustic and articulatory data are processed in parallel. The system requests the audio file and the phone list together with timing to the MaryTTS server. The VTTS module creates a file containing the variation of the articulatory parameters with the same timing used by MaryTTS to create the acoustic file (as described in the previous subsection). Once the acoustic signal and the matrix (articulatory parameter x time) of articulatory parameter variations are available, two threads are started: one playing the sound and another one playing the face gestures. The animation module is a custom written software developed using Java and Java3D/JoGL. The animation module plays congruent and synchronous articulatory and acoustic signals from the same Australian speaker. In fact, any existing English MaryTTS voices could be used together with the articulatory data we recorded.
However, the visual signals may become incoherent with American or British English voices for instance. Moreover, synchronization may be also different in this case. In our system, the same segmentation files were used to create the acoustic and articulatory sound units. Therefore, this module plays the multimodal data synchronously. An example of animation can be found in the Additional file 1: Video S1. Even though the articulatory parameters driving the avatar and the ones derived from the speaker have the same topology (e. g. , jaw1 controlled in both cases the jaw opening/closing), it may happen that positive variation of jaw1 corresponded to jaw opening for the avatar's model and jaw closing for the speaker's model. The sign attribution was determined manually.
Conclusions & perspectives A method to transform an avatar with generic tongue model and animation by key frames into a talking head that displays naturalistic tongue, jaw and lip motions was described. First, a multimodal speech database consisting of the recording of face and tongue movements during the production of a large number of sentences by an Australian speaker was created. This database was processed to create a dictionary of synchronous multimodal diphones. An articulatory model of the ECA was then created by transforming selected key frames into articulatory parameters for the jaw, lips and tongue. Real articulatory data together with the acoustic signal were used to steer the talking head using the MaryTTS Fig. 8  Schematic representation of the visual synthesis system. Given a series of diphones (e.
Bridges such as the proposed method are vitally important for virtual humans to efficiently use all possible channels of communication. The proposed method could be easily extended to other realistic auditory-visual animation methods based on concatenation  (Musti et al. 2011)  or Hidden Markov Models  (Bailly et al. 2009 ) using the same unique multimodal database. Electromagnetic articulography provides only a spatially sparse representation of the tongue movements. Co-registration methods of EMA and realtime magnetic resonance imaging (rtMRI) data  (Kim et al. 2014)  provides richer spatio-temporal data to animate the tongue movements and create a better 3D tongue model. The proposed method could be applied on the USC-TIMIT  (Narayanan et al. 2014)  which is an extensive database of multimodal (EMA, rtMRI, acoustics) speech production.
Acknowledgements We thank  Steve Fazio  for his technical support during the recording. This work was supported by the  Thinking Head  project (Burnham et al. 2006(Burnham et al. -2011)), a special initiative scheme of the  Australian Research Council  and the  National Health and Medical Research Council  ( TS0669874 ), by the  MARCS Institute, University of Western Sydney  and by the  SWoOZ  project ( ANR 11 PDOC 01901 ).
The functional brain connectivity was extracted from BOLD fMRI resting state recordings obtained as described in  [13]. All acquired brain volumes were corrected for motion and differences in slice acquisition times using the SPM5 1  software package. All fMRI data sets (segments of 5 minutes recorded from healthy subjects) were co-registered to the anatomical data set and normalized to the standard MNI (Montreal Neurological Institute) template image, to allow comparisons between subjects. As for DW-MRI data, normalized and corrected functional scans were sub-sampled to the anatomical labeled template of the human brain  [12]. Regional time series were estimated for each individual by averaging the fMRI time series over all voxels in each of the 90 regions. To eliminate low frequency noise (e.
This alines with our claim that algebraic aggregation preserves better the multilayer sub-structures across a group of subjects (multilayers) accounting for as much of the variability in the data as possible. Although we cannot definitively a one-to-one mapping of the structural and functional connectivity, we think that our method could provide new insights on the organisation of brain networks during diverse cognitive or pathological states. We therefore hope that our approach will foster more principled and successful analysis of multimodal brain connectivity datasets. For all the methods described in this article we provide the corresponding MATLAB software code. Data and code are freely available at the website https://sites. google. com/site/fr2eborn/download.
Acknowledgements The authors thank  Y. Iturria-Medina  and  L. Melie-García  for sharing the DTI connectivity data used in the study. This work was supported by the  LASAGNE  (Contract No. 318132 ) and MULTIPLEX (Contract No. 317532 ) EU projects. A. D. -G. acknowledges support from  Generalitat de Catalunya  ( 2014SGR608 ) and  Spanish MINECO  ( FIS2012-38266 ).
The microbiota composition and functions were assessed through a panel of complementary assays: quantitative polymerase chain reaction (qPCR) and 16S rRNA gene targeted pyrosequencing for determining microbiota structure; metatranscriptomics for evaluating gene expression changes, liquid chromatography for determining shortchain fatty acid (SCFA) profiles and comet assays to assess genotoxicity towards intestinal cells. We used a statistical framework combining a supervised approach and a multi-table analysis to integrate all data  (Chessel et al. , 2004). Such a framework allowed us to decipher how factors, dietary fibre level and subject specificity, impacted the gut microbiota dynamics and how the different datasets were linked during the clinical trial.
Gut microbiota richness is linked to faecal SCFA but not to genotoxicity Univariate spearman correlations tests were undertaken between microbiota richness, SCFAs amounts and bacterial genera frequencies. After multiple testing corrections, using false discovery rate, no significant associations were found. Then, co-inertia analyses (COIA) between the dominant gut microbiota at the genus level and SCFA measurements showed that those datasets shared the same co-structure (supported by a significant Monte Carlo test; Text S1 for details on the statistical analysis). The microbial richness seemed a main factor determining how the microbiota responded to the diet, and we observed that richness was significantly correlated with the first principal component (PC1) of the co-inertia analysis, which linked SCFA measurement and bacterial genera (Spearman correlation test, P < 0.
Only 15 082 blast hits (e value < 10-5) representing 50% of the reads mass, were obtained against the METAHIT database. Furthermore, there were 1709 clusters of orthologous groups (COG), 1766 KEGG orthology (KO) and 390 non-supervised orthologous groups (NOG) clusters distributed among the samples. In all individuals, the genes related to glycan and lipid metabolism were significantly overrepresented at 40 g fibre per day (Fig. 4 ). Because of the KEGG database structure, genes related to lipids or carbohydrates metabolism can appear in distinct KEGG categories. In fact, genes involved in carbohydrate metabolism and in glycan pathways can be part of the same pathway (Fig. 4 ). Based on KEGG and COG categories, multiple glycoside transporters (K10112) were significantly overrepresented at 40 g fibre/day.