dataesr/scientific-paragraphs-categorization huggingface dataset:
The resulting development and testing datasets consist of 12 synthetic mono sequences with varying durations, with accompanying ground-truth annotations. Three subsets were generated with increasing levels of complexity in terms of event density: 4 recordings have a 'low' event density 10 of 1. 11, 4 recordings have a 'medium' event density of 1. 27, and 4 recordings have a 'high' event density of 1. 81. Three SNR levels of events over the background texture were used: dB, 0 dB, and 6 dB. 3) Metrics: Following consultation with acoustic signal processing researchers, three types of evaluations were used for the OL and OS event detection tasks, namely frame-based, event-based, and class-wise event-based evaluations. Frame-based evaluation was performed using a 10 ms step and metrics were averaged over the duration of the recording.
For example, the class-wise F-measure is defined as  (3)  where is the F-measure for events of class. Matlab code for the metrics can be found online. 11 ) Baseline System: We created a baseline system for both event detection tasks based on the non-negative matrix factorization (NMF) framework. NMF has been shown to be useful for modelling the underlying spectral characteristics of sources hidden in an acoustic scene  [23] , and can also support overlapping events, making it suitable for both the OL and OS tasks. We chose to design a supervised method for event detection, using a pre-trained dictionary of acoustic events  [42]. The baseline method is based on NMF using the Kullback-Leibler divergence as a cost function  [44]. As a time-frequency representation, we used the constant-Q transform with a log-frequency resolution of 60 bins per octave  [45].
D. Challenge organization The full timeline for the challenge organization is given in Table  I. Some of the items included of the timeline will be obvious to an outside observer. However there are some aspects of the timeline and the workload which we believe merit emphasis, listed as follows. • There were two periods which required the most time commitment from the organizing team: creating the datasets, and running the code submissions. In particular, as has been remarked by organizers of related challenges  [33] , no matter how many precautions are taken to ensure people submit code that will run on the organizers' hardware (formal specifications, published virtual machine), it often requires many person-hours of attention before submitted code will run properly. This will be discussed further below.
Recording the datasets also took significant time: this was not just the audio recording itself, but also the supervision of annotators, and the listening sessions and manual inspection to ensure data quality. • We found it extremely useful to ask people to let us know of their intentions, in order to help us plan. In December 2012 we surveyed the community for indicative data about the level of interest in task participation, as well as the preferences for programming languages and operating systems. This information fed directly into our design of a Linux virtual machine for people to test their code. Then in March 2013 we asked participants to email us announcing their intentions to take part (with no commitment implied). This enabled us to plan resources, and to follow up on expected submissions that went astray.
We received 20 notifications of intention to submit, some corresponding to multiple task submissions; of these only three did not eventually submit. • One aspect of the timeline that could have been improved was the long wait between collating the results and releasing them publicly. It meant that participants could not compare and contrast results while their systems were "fresh in their minds". However, this was due to our decision to co-ordinate with WASPAA 2013, which was an ideal forum for discussion of the challenge outcomes. Regarding the execution of code submissions, our publication of a virtual machine as a standard platform certainly reduced the number of compatibility issues we had to deal with. However, there remained various software issues we encountered when running the code submissions, listed as follows.
• A frequent issue in Matlab submissions was opening the training annotation text files (for reading) using mode ' ' (which is for reading and writing). This fails when files are read-only. We had set the test data as read-only, and in the specification we had stated that submissions must not write data in the test folder. • One submission had been developed using Matlab on Windows; when we ran it using the same version of Matlab, but on Linux, it got rather poor results, which we initially attributed to overfitting. It later emerged that the poor performance was because a Matlab toolbox exhibited a bug only when running on Linux. • On the virtual machine, there were occasional problems with version mismatch between Dynamic Link Libraries (DLLs). Such issues were reduced but not completely eliminated with the use of the virtual machine, often because participants did not fully test using the virtual machine, or occasionally added late modifications after testing.
g. the data format issues) could have been prevented by providing unit tests which participants must pass before submitting. Earlier in the process, we also encountered a data issue: after we published the development datasets, a community member on the mailing list alerted us to a formatting error in some of the annotations (the text label doorknock was used in some places rather than the official label knock). Such issues occur despite the multiple steps of checking we performed before release. We updated the dataset to correct the issue, re-released it and confirmed this to the participants. We required each submitted system to be accompanied by an extended abstract describing the system. We experienced no issues in publishing these abstracts; however in future evaluations we would consider explicit open-access licensing of the abstracts for greater clarity.
Each segment is then assigned to a class using a generalised likelihood ratio test classifier. • DHV: The DHV submission was created for both the OL and OS tasks. It follows a generative classification scheme using HMMs with multiple Viterbi passes. Firstly, Fig. 2. Mean values and confidence intervals of the accuracy of methods for SC evaluated on the DCASE private dataset using stratified 5-fold cross-validation. The boxes enclose methods that cannot be judged to perform differently with a significance level of 95% using a sign test  [65]. For example, GSR baseline, but we cannot confirm that CHR baseline. Figure is adapted from  [64]. MFCCs are extracted as features, and used as input to continuous-density HMMs (each state corresponds to an event class, including background noise). Polyphonic detection is achieved by performing consecutive passes of the Viterbi algorithm.
A. SC Results Fig. 2  shows the overall performance of submitted systems for the scene classification task. The baseline system achieved an accuracy of 55%; most systems were able to improve on this, although our significance tests were able to demonstrate a significant improvement over baseline only for the strongest four systems. The results indicate that level of difficulty for the task was appropriate: the leading systems were able to improve significantly upon the baseline, yet the task was far from trivial for any of the submitted systems. Also, the sizes of the error bars indicate that performance across the five folds was broadly consistent, indicating that the dataset was not overly heterogeneous. However, the statistical tests did not demonstrate significant differences between various systems (depicted by the large overlap of boxes in Fig.
2 ), which implies that a larger dataset may have enabled a more fine-grained ranking of systems. The results for this SC task are further analyzed in a tutorial article  [64]. For that reason, here we discuss briefly the state of the art reflected in the SC task outcomes, allowing us to expand further on the OL/OS task outcomes in the next section. The majority of the submitted systems used discriminative training, with many of the strong performers using an SVM as the final classifier. Further, most of the leading results were obtained by those who captured medium-range temporal information in the features used for classification. Four of the five highest-scoring systems did this: Roma et al. [56]  captured temporal repetition and similarity using "recurrence quantification analysis"; Rakotomamonjy and Gasso  [55]  used gradient features from image-processing; Geiger et al.
2. 2]. This found a significant pattern of agreement 13 The original OS results published at the time of the challenge differ from the results published here due to a systematic fault affecting a subset of the labels in the original OS development and test datasets. This was found and fixed, and the three teams who submitted systems to the OS task were contacted and invited to revise their systems. The DHV system was re-trained on the corrected OS development data; the configurations of the other systems (GVV, VVK, and baseline) were not affected, and were left unchanged. All three systems were re-evaluated on the corrected test datasets to obtain the results here. The corrections to the data generally improved system performance, which is to be expected since they improved the correspondence between training and test.
It is also instructive to look at the correlation between the ranking of systems that were both submitted to the Office Live and Office Synthetic challenges. It allows us to study the consistency of performance of the evaluated systems using natural and artificial data. The OL and OS tests are not independent, since they partly use the same audio source material, but comparing their outcomes gives us an indication of whether the synthesis procedure for OS had a strong impact on the eventual rankings. Let us consider results achieved using the Class-Wise Event-Based metrics as they are more resilient to discrepancies between datasets in terms of density of events within the scene. Apart from a slight permutation of GVV and VVK systems, the strong level of correlation (average of 90% over the 4 metrics in terms of the Spearman's rank correlation coefficient) indicates that considering artificially synthesized sound scenes may have some meaning for this kind of task.
VI. REFLECTIONS AND RECOMMENDATIONS Before concluding, we wish to draw some reflections out from the above results and from our experience of managing the DCASE challenge, and to offer some recommendations for future evaluation challenges. Our challenge comes in the context of a series of challenges coordinated by the IEEE AASP, such as challenges relating to distant and reverberant speech. 15 ur design of the challenge involved participants submitting code, for the organisers to execute against private datasets. This design, in common with MIREX music audio challenges  [33] , incurs resource costs as the hosts must dedicate time to running the submissions. It also requires holding back some private data, which cannot immediately benefit the community as open data. However it has advantages such as ensuring participants do not overfit to the test data, and ensuring that results are reproducible in the sense of empirically verifying that the submitted software can be run by a third party.
An interesting point of comparison is provided by similar challenges run via the Kaggle website such as the 2013 SABIOD machine listening challenges. 16 These challenges centered around automatic classification of animal sounds. The mode of interaction in that case was not to submit code, but to submit system output. Further, participants could iteratively modify their code and submit updated output, getting feedback in the form of results on a validation dataset. This does carry some risk of overfitting to the specifics of the challenge, and less direct reproducibility, although the winning submission was required to be made open-source and confirmed by the hosts. Relative to DCASE, the SABIOD challenges appeared to encourage a greater amount of ad-hoc participation from independent machine-learning professionals, perhaps due to the immediate feedback loop made possible by the online system.
The workflows represented by the DCASE and the SABIOD challenges each have their own strengths and weaknesses, and we look forward to further refinements in public evaluation methodology. We have enumerated the steps involved in running the DCASE challenge, in particular to highlight the resource implications for hosting such challenges. Dataset collection and annotation was the main requirement on staff time. This challenge was not funded explicitly by any project, and so would not have been possible without the resources made available by a large research group (see Acknowledgments). This includes staff and PhD students as core organisers, data annotators, programmers assisting with issues such as code and the virtual machine, and infrastructure such as code-and data-hosting facilities. In Section III-D we described various steps we took to ensure that the challenge would run smoothly, such as publishing formal task specifications, baseline code and a virtual machine.
This reduced but by no means eliminated the time required to run and troubleshoot the code submissions received. A clear recommendation that emerges from this experience is that a formal test for the submitted code to be run at submission time would help greatly. This could be applied in the form of automated unit testing, or more simply by the challenge organizers running the submissions using public data and confirming that the results obtained match the results that the submitters obtained on their own system. Community involvement was crucial to the successful conduct of this challenge, in particular for discussing the task specifications, but also for negotiating logistics of submission and discussing the final results. The support of the IEEE AASP Technical Committee and the IEEE WASPAA 2013 Conference Committee helped us to form this community.
A strategy used by many of the strongest systems was to use feature representations which capture medium-scale temporal information about the sound scene. However there is still room for improvement beyond the highest-scoring system; we demonstrated this was possible with a simple majority-vote metaclassifier aggregating the submitted systems, illustrating that there is information yet present in the audio that can drive stronger performance in future. The best way to improve the SC task in future rounds would be through larger dataset sizes in order to draw stronger conclusions about the significance of differences between system performances. For the event detection (OL/OS) tasks, the leading systems achieved relatively strong performance, although with substantial scope for improvement.
This work was supported by the  EPSRC   Leadership Fellowship   EP/G007144/1 , by the  EPSRC   Research Grant   EP/H043101/1 , by  ANR Houle  under reference  ANR-11-JS03-005-01 , and by a  City University London Research Fellowship , by the  EPSRC   Early Career Fellowship   EP/L020505/1 , and by the  RAEng Research Fellowship   RF/128. The associate editor coordinating the review of this manuscript and approving it for publication was.
After a consultation period with members of the acoustic signal processing community, for the Event Detection -Office Live (OL) task we created recordings of office scenes, consisting of the following 16 classes: door knock, door slam, speech, laughter, clearing throat, coughing, drawer, printer, keyboard click, mouse click, object (pen, pencil, marker) on table surface, switch, keys (put on table), phone ringing, short alert (beep) sound, page turning. 1) Recorded Dataset (OL):.
On the other hand, depending on the algorithm, the property to be optimized, and the experimental data, great discrepancies in the results can be obtained. As an example, Chaos et al. [12]  found a heat capacity of 1. 5 J/g/K and 2. 5 J/g/K when simulating the same data but with different algorithms. This approach does not allow the estimation of reliable physical properties, and direct measurements are then preferable The direct measurements of the above properties can be done using a thermo-gravimetric or calorimetric apparatus. Even if the characterization of thermal properties such as heat capacity, thermal conductivity, or thermal diffusivity is standardized  [14] [15] [16] [17] [18] , the main challenge for accurate pyrolysis modeling is to be able to measure those properties before, during, and after the decomposition of the material and also at high temperatures.
1 µm) that consist of glass fibers and ceramic, respectively. The FTIR gas cell was set to 185 ˝C and 652 Torr. The optical pathway is 2 m long and the chamber of the spectrometer is filled with dry air. FTIR spectra obtained using CAMLC-FTIR were treated using OMNIC™ software. To quantify gases, spectra have to be recorded at different concentrations for targeted gases and a quantification method has to be created using TQ™ Analyst software. To create a method, representative regions in the spectra of the selected gas have to be chosen and interactions with other gases have to be taken into account. Using CAMLC-FTIR the following gases can be quantified: water, carbon monoxide, carbon dioxide, acetic acid, acetone, methane, propane, ethane, and ethene. The developed method in our laboratory can also consider nitrogen monoxide, nitrogen dioxide, hydrogen cyanide, ammonia, and ethyne.
Quantification is reproducible within ˘10 %. CAMLC-FTIR experiments were performed in duplicate to ensure repeatability of the obtained results. Materials 2015, 8, page-page temperature controllers were installed. Before analyzing the gases by FTIR, soot particles were filtered off by two different heated filters (2 and 0. 1 μm) that consist of glass fibers and ceramic, respectively. The FTIR gas cell was set to 185 °C and 652 Torr. The optical pathway is 2 m long and the chamber of the spectrometer is filled with dry air. FTIR spectra obtained using CAMLC-FTIR were treated using OMNIC™ software. To quantify gases, spectra have to be recorded at different concentrations for targeted gases and a quantification method has to be created using TQ™ Analyst software. To create a method, representative regions in the spectra of the selected gas have to be chosen and interactions with other gases have to be taken into account.
Introduction Given the brilliance of third-generation synchrotron sources and the consequent diversity of possible experimental set-ups it is perhaps unsurprising that the scientific data produced come in a huge variety of formats and sizes  (Willmott, 2011). The primary role of these facilities is to allow users to analyse samples on state-of-the-art experiments and leave with data that can easily be processed and prepared for publication. It is key that the data collection, visualization and analysis methods provided by these facilities are simple and consistent, allowing barrier-less access to the sometimes highly complex experiments of a broad multi-disciplinary user base. The Data Analysis WorkbeNch (DAWN) is an open-source multiplatform software package free to download and intended to supply easy data visualization and processing to users of synchrotron facilities.
Produced as part of a collaboration between Diamond Light Source (DLS), the European Synchrotron Radiation Facility (ESRF) and the European Molecular Biology Laboratory (EMBL, Grenoble), DAWN is now integral to the analysis pipelines of many beamlines operated by these facilities. The list of functionality available in the software package is still growing and currently includes data reduction for small-angle X-ray scattering (SAXS); optimization of tomographic reconstruction; alignment and reduction for photo-emission electron microscopy (PEEM); and design of macromolecular X-ray diffraction experiment workflows  (Brockhauser et al. , 2012). At Diamond, DAWN is tightly integrated into the acquisition software (Generic Data Acquisition; GDA)  (Enderby &  ISSN 1600-5775  Pulford, 2004; Gibbons, 2008) , the aim being to give users a consistent data visualization experience across beamlines that they can take home and use after their visit or even download and learn prior to arrival.
DAWN history The DAWN project developed from the merger of two complementary data visualization and analysis projects, Scientific Data Analysis (SDA) and Data Analysis WorkBench (DAWB), both of which were built using the same core technologies of the Java programming language  (Arnold et al. , 1995 (Arnold et al. , , 2005) )  and the Java-derived Eclipse Rich Client Platform (Eclipse RCP)  (Eclipse Foundation, 2004; McAffer & Aniszczyk, 2010). SDA was produced at Diamond in 2010, using visualization components which became available as part of the process of GDA changing to the modular Eclipse RCP structure. DAWB was developed in the same year, on the back of a long history of Eclipse RCP development at the ESRF, but based on GDA's dataset model allowing the user to view data, set up experimental workflows and access analysis routines written in Python.
Perspectives: partitioning of functionality For a single piece of software to perform such a wide variety of operations it is fundamentally important for it to be well structured, with different functions clearly partitioned. User experience would suffer greatly if all visualization tools were accessible at all times, irrespective of the task the user is trying to perform. To overcome this issue, DAWN uses the Eclipse RCP concept of 'perspectives', which only present users with options pertinent to the analysis they wish to do. A perspective is effectively a miniature program designed to do a specific set of tasks. DAWN itself behaves like an empty container, the functionality of which is determined by the individual perspectives running within it. Table  1  contains the names and descriptions of the main perspectives in DAWN 1.
Data and file formats The synchrotron community is widespread with hundreds of separate beamlines around the world, many with different detectors and data formats. Although work has been done to make standardized formats available  (Bicarregui, 2004)  [e. g. NeXus  (Klosowski et al. , 1997) , CIF  (Brown & McMahon, 2002; IUCR, 2006), etc. ] , there remains a lot of data in ASCII, image-based and other legacy formats. To support the great diversity of existing data formats, DAWN uses a modular plugin approach, allowing the addition of new data and metadata types without the need to rebuild the entire program. If a suitable plugin is available for a file, data are then loaded into one of DAWN's generic dataset classes (see supporting information 2  for details of the architecture of the dataset).
DEXPLORE An advanced alternative to data browsing. Provides many additional options for how data are shown; it also contains features for comparing data from multiple files as well as connecting the plots to a Python/Jython Interpreter. Python/Jython Scripting Provides a PyDev  (Zadrozny, 2003)  environment for opening, editing and displaying Python source files. Tomography Reconstruction Tools to access reconstruction routines for NeXus tomography data (requires additional software).
Workflows Designing scientific algorithms with a graph-like structure, similar to LabView  (Elliott et al. , 2007) , implemented using Passerelle  (Brockhauser et al. , 2012)  Large multi-dimensional datasets can be a problem for many analysis packages. DAWN simplifies reading of such data through the concept of lazy loading. Lazy loading does not read the complete dataset into memory, but instead reads only a user-selected section or slice. For example, from a large imaging experiment (see Case Study I), the user might select one or a range of images to view and analyse from a single HDF5 file containing hundreds, if not thousands, of images  (HDF Group, 2000; Folk & Pourmal, 2010). Without lazy loading it would be unfeasible to view such a dataset without very high-end computing resources.