dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements Access to the HPC resources of CALMIP supercomputing center were granted under the allocation 2016-[P1440] attributed to the  AMIS Laboratory, Toulouse (France). This work has been funded by the  Agencia Estatal de Investigacion and European Regional Development Fund of the European Union  (projects  CGL2016-76431-P  and  CGL2017-82654-P ,  AEI/FEDER, EU) , by the  Ministerio de Economı ´a y Empresa  (grant  BES-2015-071318  to AU), the  Generalitat de Catalunya  ( CERCA Programme , and consolidated research groups  2017 SGR 86  and  2017 SGR 116 GRC ), and the  French CNRS. We deeply thank  Sergio Alme  ´cija for helpful advice and discussion during the elaboration of this paper and JoseBraga for valuable suggestions. We are grateful to  Lorenzo Rook  for giving access to the Oreopithecus bambolii petrosal bone originally published in Rook et al.
The last few years have seen a dramatic increase in applications of native mass and ion mobility spectrometry, especially for the study of proteins and protein complexes. This increase has been catalysed by the availability of commercial instrumentation capable of carrying out such analyses. Like in most fields, however, the software to process the data generated from new instrumentation lags behind. Recently, a number of research groups have started addressing this by developing software, but further improvements are still required in order to realise the full potential of the datasets generated. Here we describe practical aspects as well as challenges in processing native mass spectrometry (MS) and ion mobility-MS datasets, and provide a brief overview of currently available tools. We then set out our vision of future developments that would bring the community together and lead to the development of a common platform to expedite future computational developments, provide standardised processing approaches and serve as a location for the deposition of data for this emerging field.
Even though the software offered by vendors are particularly useful for thorough processing of MS data, they provide only basic functionality and still require further manual processing routines, which are often slow, cumbersome and error-prone. MS software developed for the areas of proteomics, lipidomics, glycomics and metabolomics is more mature in nature, however such tools are not optimised for the types of analyses required for native IM-MS datasets. Specialised software providing automated processing approaches, apart from yielding considerable time savings, would increase the reproducibility of processing and analysis. It would also facilitate the exchange of results between laboratories and provide robust documentation of how experiments were carried out and how the data were processed.
Many of the early studies utilising native MS and IM-MS were conducted on Waters instruments, as they were the first to introduce a commercial IM instrument to the market  26. For this reason, early custom processing software such as Amphitrite, PULSAR, and CIUSuite2 have custom-built file readers that accept the raw files as input and can read them by calling the Waters dynamic linked library (cdt. dll). CIUSuite2 more recently has also added functionality for reading Agilent raw files. Alternatively, programs such as ORIGAMI ANALYSE utilise data extraction tools available within the Waters ion mobility software Driftscope. Due to the development of high-mass Orbitraps (made by Thermo Scientific) and the utility of such instruments for native MS applications, software such as UniDec, MetaUniDec and ORIGAMI ANALYSE also accept Thermo Raw files.
Alternatively, other software such as Benthesikyme and iFAMS use as input text files that have been exported by using the manufacturer's software. While there are many open data formats available in the MS community, these have not been adopted by the native MS field. This is because the design of current open data formats has been chosen for more common applications of MS in various proteomic, lipidomic, and metabolomic fields. Depending on the type of analysis, be it MS or MS/MS, eXtensible Markup Language (XML)-based formats such as mzData, mzML, mzXML are commonly used and interchanged. However, these tend to better cater to MS and not IM-MS metadata. In instances where IM is supported, the syntax tends to be too verbose in the full profile mode. Another open data format alternative is Unified Ion Mobility Files (UIMF)  27  , which is based on a relational database management system.
Initially, semi-automated deconvolution algorithms were created, with the introduction of MaxEnt and later with more sophisticated methods implemented in SOMMS, Massign, and CHAMP. More recently, programs such as Amphitrite, and UniDec have improved on previous methodologies. They simplify and speed up the deconvolution process by automatically picking, fitting and assigning charge states to protein and protein-complex samples. The peak series, containing each charge state and typically lacking isotopic resolution, are usually described by Gaussian distributions. However, other distributions, such as Lorentzian or mixed distributions, have also been used as the experimental peaks can have a pronounced tail due to adducts attachment. Most software simulate a spectrum and then use least squares optimisation to minimise the differences between the simulated and experimental data.
UniDec instead uses a Bayesian deconvolution algorithm, while iFAMS uses a Fourier Transform-based algorithm for this optimisation. Such analyses can provide the mass of a protein or protein complex, and are the gold standard in reporting on the oligomeric or ligand-binding state of a protein. Knowing the mass of a protein complex can also provide insights into its possible composition, especially if the masses of the individual components comprising the complex are known. Software such as SUMMIT  42  has been developed to do this. Measurements of mass changes due to small molecule or ligand binding can also be used to determine the stoichiometry of such binding events. As well as measuring composition and stoichiometry, native MS can also be used to determine apparent binding affinities  43.
For example, one can determine the affinity of a ligand binding to a protein if one assumes that intensities reflect concentrations in solution of the bound and unbound species. These affinities are calculated by fitting the relative ion intensities of bound and unbound forms as a function of the ligand concentration. Accurate extraction of the intensities, or peak areas, of different bound forms is therefore required  44, 45. Deconvolution of the data allows one to use the extracted peak areas for this purpose, which can provide more accurate data for fitting than the peak tops. Software such as UniDec provide multiple models to fit data and enable the calculation of both micro-and macroscopic Kd values.
The ATD for each voltage is usually normalised to the highest intensity and smoothed prior to being used to generate the two-dimensional plots. CIUSuite and CIUSuite2 have functions to identify features of unfolding trajectories that describe or 'fingerprint' a given protein's characteristics. Many software (CIUSuite, CIUSuite2, PULSAR, ORIGAMI ANALYSE ) can also generate difference and root-mean-square deviation plots to allow unfolding profiles to be compared. Benthesikyme provides summary plots that capture the changes in average arrival time and in the full width at half maximum of the ATD during activation experiments and uses these to compare samples. The voltages at which transitions occur between discrete species in unfolding plots can be affected by the bound state of an ion. For example, ligand binding can induce conformational stability in the various observed forms, analogous to in-solution thermal shift unfolding/aggregation experiments.
Amphitrite, PULSAR or ORIGAMI ANALYSE automate many of the tasks required for calibration. The goal of the calibration is to provide an equation relating the arrival time to the CCS. Possible improvements in the future would include reporting on the uncertainties associated with the calibrated values, in line with recently published recommendations  14. Once a calibration has been performed, the ATD axis can be converted to a CCS axis. Harnessing IM-MS data for structural biology Once an experimental CCS has been obtained, it can be compared to the CCSs for molecular structures obtained by other experimental methods, such as X-ray crystallography or NMR, or models provided by computational methods. The development of methodologies and related software to estimate the CCS of a molecular structure constitutes a theoretical and technical challenge on its own, and is an area of active research.
An in-depth discussion of these methodologies, software implementing them and how to parameterise this software is beyond the scope of this review. The reader is referred to a recent review on this topic  13. Briefly, CCS calculation software originally focused on calculating the CCS from atomistic models given prior information about the radius, and sometimes partial charge, of each atom. More recently, however, efforts to calculate the CCS electron density maps represented as point clouds  47, 48  or super-coarse-grained models  [49] [50] [51] [52] [53] [54]  have been proposed. These methods are especially useful when no atomic information about the proteins of interest is available (see companion review). Comparison of experimental and theoretical CCS values has been used extensively to report on the gas-phase structure of ions and whether the ions are able  7, 9  , or not  55  , to retain their solution-phase structure.
Recent work has focused on including CCSs in integrative modelling pipelines  53, [56] [57] [58] [59]. Many of the data from native MS and IM-MS experiments can be highly complementary to data obtained by other structural biology methods, and can be exploited in molecular modelling scenarios. CCSs can be used to validate structures produced by, for example, homology modelling, coarse-grained, or molecular dynamics simulations  60. MS and IM-MS data can also be fed directly into integrative modelling software programs to guide the refinement of the most likely conformation(s) of quaternary assemblies  61. Data typically exploited in this context include (1) detailed stoichiometric information about the subunits and complex partners and (2) CCS distributions. Stoichiometric information is used to define how many subunits should be manipulated during a protein docking run.
e. experimental parameters, including sample preparation, and relevant instrumental parameters). These are important for the correct reproduction of experiments, and also for adherence to policies regarding the storage and management of research data  65. A proposal outlining relevant information that needs to be captured in IM has recently been put forward  14. A common file format would enable the creation of a data repository and encourage the deposition and meta-analysis of data. This would have significant benefits for computational researchers as well as experimentalists, and further support data sharing and transparency in data collection. Establishing a repository for native IM-MS data would provide opportunities to link these data with entries on other structural repositories such as the PDB  66  , EMDataBank 67 , PDB-Dev  68, 69  , and pE-DB  70  , as well as on proteomics databases such as PRIDE  71  and Uniprot  72.
We would also welcome the inclusion of native IM-MS outputs such as mass, ATD and CCS, under well-defined experimental conditions, among the parameters describing protein structures in these databases. While these initiatives would consolidate capabilities that current software provide, developments in MS hardware continue to occur at a rapid pace. Examples are the development of higher resolution in MS and IM dimensions, tandem ion mobility analysers, and multistage activation and separation experiments  73, 74. To accommodate these and any future innovations, a common programming library is needed that accomplishes common processing tasks such as reading raw files, smoothing, background subtraction, Gaussian envelope fitting, visualisation and data export. This would reduce duplications in software development, speed up development time and enable more focus on innovation in data analysis.
This library would preferably be written in the Python programming language, due to its flexibility, accessibility, and existing codebase. It should be made available on code-hosting platforms such as GitHub or Bitbucket to allow for more sustainable software, better code testing, and greater collaboration between research groups. Using common tools for analysis should also improve the reproducibility of results. Having such a common codebase in a publicly accessible repository would also allow the community to further develop and extend the software together, and would allow any developments to be made available immediately to the community. Once the software library and common file format are in place, a further development would be to create a software tool for accessing/visualising these files, similar to PRIDE Inspector, which is used to access proteomics datasets  75, 76.
Recent improvements in web technologies mean it is now possible to create a platform-agnostic, web-based front end for the visualisation of native IM-MS datasets. This front-end would be web-accessible via a browser and mobile devices, and could provide a barebones environment implementing common processing routines. A plug-in interface could allow the integration of algorithms for data processing. This front end could integrate access to CCS calculation software and other software that is currently needed for interpreting native MS and IM-MS data, such as new gas-phase molecular dynamics approaches. The latter is an underdeveloped area, but very important for fully exploiting the data; we anticipate a lot of algorithm and software innovations to take place in this field in the coming years.
Currently, tools for calculating the theoretical CCS of molecular structures are used as standalone programs. Integration of these calculators with software for the processing of raw data would allow a more straightforward comparison between experimental observations and theoretical models, and would allow the parameters for both approaches to be stored in one common format, again assisting in transparency and data reproducibility. We have set out ambitious aims with regards to data standardisation, the development of software tools and the creation of a common platform for the IM-MS field. We feel these steps are important for facilitating data sharing and collaboration, and for encouraging future innovation, transparency and reproducibility in this area. Practically speaking, examples of how these aims could be achieved are the Collaborative Computational Projects (funded by the UK's Engineering and Physical Sciences Research Council) that exist in other structural biology fields, e.
We present a deep learning-based multitask framework for joint 3D human pose estimation and action recognition from RGB video sequences. Our approach proceeds along two stages. In the first, we run a real-time 2D pose detector to determine the precise pixel location of important keypoints of the body. A two-stream neural network is then designed and trained to map detected 2D keypoints into 3D poses. In the second, we deploy the Efficient Neural Architecture Search (ENAS) algorithm to find an optimal network architecture that is used for modeling the spatio-temporal evolution of the estimated 3D poses via an image-based intermediate representation and performing action recognition. Experiments on Human3. 6M, MSR Action3D and SBU Kinect Interaction datasets verify the effectiveness of the proposed method on the targeted tasks.
Hence, a simple network model can also learn a mapping to convert 2D poses into 3D. The effectiveness of the proposed method is evaluated on public benchmark datasets (Human3. 6M [17] , MSR Action3D  [18] , and SBU  [19] ). Far beyond our expectations, the experimental results demonstrate state-of-the-art performances on the targeted tasks (Section 4. 3) and support our hypotheses above. Furthermore, we show that this approach has a low computational cost (Section 4. 4). Overall, our main contributions are as follows: • First, we present a two-stream, lightweight neural network to recover 3D human poses from RGB images provided by a monocular camera. Our proposed method achieves stateof-the-art result on 3D human pose estimation task and benefits action recognition. • Second, we propose to put an action recognition approach on top of the 3D pose estimator to form a unified framework for 3D pose-based action recognition.
3D human pose estimation Given an input RGB image I ∈ R W ×H×3 , we aim to estimate the body joint locations in the 3dimensional space, noted as p3D ∈ R 3×M. To this end, we first run the state-of-the-art human 2D pose detector, namely OpenPose  [12] , to produce a series of 2D keypoints p 2D ∈ R 2×N. To recover the 3D joint locations, we try to learn a direct 2D-to-3D mapping f r : p 2D f r -→ p3D. This transformation can be implemented by a deep neural network in a supervised manner p3D = f r (p 2D , θ ), (1) where θ is a set of trainable parameters of the function f r. To optimize f r , we minimize the prediction error over a labelled dataset of C poses by solving the optimization problem arg min θ 1 C C ∑ n=1 L( f r (x i ), y i ). (2) Here x i and y i are the input 2D poses and the ground truth 3D poses, respectively; L denotes a loss function.
We refer the interested reader to the work described in  [53]  Figure  2 : Diagram of the proposed two-stream network for training our 3D pose estimator. for further technical details. Figure  3  visualizes some Enhanced-SPMF representations from samples of the MSR Action3D dataset  [18]. For learning and classifying the obtained images, we propose to use the Efficient Neural Architecture Search (ENAS)  [13]  -a recent state-of-the-art technique for automatic design of deep neural networks. The ENAS is in fact an extension of an important advance in deep learning called NAS  [47] , which is able to automatize the designing process of convolutional architectures on a dataset of interest. This method proposes to search for optimal building blocks (called cells, including normal cells and reduction cells) and the final architecture is then constructed from the best cells achieve.
In NAS, an RNN is used. It first samples a candidate architecture called child model. This child model is then trained to convergence on the desired task and reports its performance. Next, the RNN uses the performance as a guiding signal to find a better architecture. This process is repeated for many times, making NAS computationally expensive and time-consuming (e. g. on CIFAR-10, NAS needs 4 days with 450 GPUs to discover the best architecture). ENAS has been proposed to improve the efficiency of NAS. Its key idea of ENAS  [13]  is the use of sharing parameters among child models, which helps reducing the time of training each child model from scratch to convergence. State-of-the-art performance has been achieved by ENAS on well known public datasets. We encourage the readers to refer to the original paper  [13]  for more details.
Datasets and settings We evaluate the proposed method on three challenging datasets: Human3. 6M, MSR Ac-tion3D and SBU Kinect Interaction. The Human3. 6M is used for evaluating 3D pose estimation. Meanwhile, the other two datasets are used for validating action recognition. The characteristics of each dataset are as follows. Human3. 6M  [17] : This is a very large-scale dataset containing 3. 6 million different 3D articulated poses captured from 11 actors for 17 actions, under 4 different viewpoints. For each subject, the dataset provides 32 body joints, from which only 17 joints are used for training and computing scores. In particular, 2D joint locations and 3D poses ground truth are available for evaluating supervised learning models. MSR Action3D  [18] : This dataset contains 20 actions, performed by 10 subjects.
Our experiment was conducted on 557 video sequences of the MSR Action3D, in which the whole dataset is divided into three subsets: AS1, AS2, and AS3. There are 8 actions classes for each subset. Half of the data is selected for training and the rest is used for testing. [19] : This dataset contains a total of 300 interactions, performed by 7 participants for 8 actions. This is a challenging dataset due to the fact that it contains pairs of actions that are difficult to distinguish such as exchanging objectsshaking hands or pushingpunching. We randomly split the whole dataset into 5 folds, in which 4 folds are used for training and the remaining 1 fold is used for testing.
Implementation details The proposed networks were implemented in Python with Keras/TensorFlow backend. The two streams of the 3D pose estimator are trained separately with the same hyperparameters setting, in which we use mini-batches of 128 poses with 0. 25 dropout rate. The weights are initialized by the He initialization  [54]. Adam optimizer  [55]  is used with default parameters. The initial learning rate is set to 0. 001 and is decreased by a factor of 0. 5 after every 50 epochs. The network is trained for 300 epochs from scratch on the Human3. 6M dataset  [17]. For action recognition task, we run OpenPose  [12]  to generate 2D detections on MSR Action3D  [18]  and SBU Kinect Interaction  [19]. The pre-trained 3D pose estimator on Hu-man3. 6M [17]  is then used to provide 3D poses. We use standard data pre-processing and augmentation techniques such as randomly cropping and flipping on these two datasets due to their small sizes.
Evaluation on 3D pose estimation We evaluate the effectiveness of the proposed 3D pose estimation network using the standard protocol of the Human3. 6M dataset  [17, 26, 28, 33]. Five subjects S1, S5, S6, S7, S8 are used for training and the rest two subjects S9, S11 are used for evaluation. Experimental results are reported by the average error in millimeters between the ground truth and the corresponding predictions over all joints. Much to our surprise, our method outperforms the previous best result from the literature  [33]  by 3. 1mm, corresponding to an error reduction of 6. 8% even when combining the ground truth 2D locations with the 2D OpenPose detections. This result proves that our network design can learn to recover the 3D pose from the 2D joint locations with a remarkably low error rate, which to the best of our knowledge, has established a new state-of-the-art on 3D human pose estimation (see Table  1  and Figure  5 ).
Evaluation on action recognition Table  2  reports the experimental results and comparisons with state-of-the-art methods on the MSR Action3D dataset  [18]. The ENAS algorithm  [13]  is able to explore a diversity of network architectures and the best design is identified based on its validation score. Thus, the final architecture achieved a total average accuracy of 97. 98% over three subset AS1, AS2 and AS3. This result outperforms many previous studies  [9, 18, 36, 37, 64, 65, 66, 67, 68] , and among them, many are depth sensor-based approaches. Figure  6  provides a schematic diagram of the best cells and optimal architecture found by ENAS on the AS1 subset  [18]. For the SBU Kinect Interaction dataset  [19] , the best model achieved an accuracy of 96. 30%,  as shown in Table  3. Our reported results indicated an important observation that by using only the 3D predicted poses, we are able to outperform previous works reported in  [36, 69, 70, 71, 72, 73, 74]  and reach state-of-the-art results provided in  [53, 75] , which deploy accurate skeletal data provided by Kinect v2 sensor.