dataesr/scientific-paragraphs-categorization huggingface dataset:
However, in contrast to our opensource, integrated package, Parish's simulations used an ''in-house'' package that made use of spectra modeled using the DTSA II software. Indeed, none of the tools used to create STEM-EDX datacubes have been made available so that currently there is no open software to simulate STEM-EDX datacubes. Our aim is to address this deficit by providing a software solution for simulating physically-accurate STEM-EDX datacubes. In order to give the best utility, the software should offer a quick simulation time, for instance just a few minutes to simulate a 512 × 512 × 2048 datacube. To this end, we apply a simple simulation framework, using X-ray emission cross-section tables to create EDX spectra based on the expected chemical compositions of relevant phases. Each spectrum is assigned a spatial distribution map.
The spectra and distribution maps are then combined to obtain the spectrum image. For making these simulations, we have developed two Python packages: the emtables (Electron Microscopy Tables) library for the creation of X-ray emission cross-section tables and the espm (Electron Spectro-Microscopy) library to simulate the STEM-EDX datacubes. The crosssection calculations are based on state-of-the-art methods; we describe these in detail in Section 2. In Section 3 we next present the datacube simulation, followed by an example comparison between experimental and simulated data in Section 4.
2c ). For example, the  ( 2 ) ionization cross-section can be written as follows : 𝑠 (𝖫 2 ) = 𝜎 (𝖫 2 ) + 𝜎 (𝖪) 𝜔 (𝖪) 𝑟 (𝖪𝖫 2 ) + 𝜎 (𝖫 1 ) 𝑓 (𝖫 1 𝖫 2 ) + 𝛤 (𝖪) 𝜎 (𝖪) (𝖰 3 ) ∑ (𝗃)=(𝖫 3 ) 𝛾 (𝖪𝖫 2 𝗃) (4) This modeling of the characteristic X-rays does not include the secondary fluorescence events; an effect that can be ignored thanks to the  low thickness of the specimens observed in STEM-EDX. The effect of the presence of multiple electron vacancies  [19]  is also not taken into account. In this work, the X-ray cross-sections  0 () ( 0 , ) are calculated based on the above equations. To do so, we designed an open-source Python software called emtables to produce cross-section tables for all elements and for any beam energy. While our notations are different, this work is very similar to the one of Llovet et al.
[19]. The main improvement of our work is to make accessible X-ray emission cross sections under electron irradiation tables that are directly relevant for the quantification of EDX spectra. Indeed, the currently accessible tables such as the evaluated atomic data library (EADL)  [20]  or the evaluated electron data library (EEDL)  [21]  provide respectively X-ray emission-related quantities (such as fluorescence, Auger and Coster-Kronig rates) and ionization cross sections under electron irradiation. However, X-ray emission cross sections under electron irradiation results from the combination of these quantities. In fact, databases are stored in text format, which makes them human-readable and modifiable. Both our work and that of Llovet et al. use the ionization cross-sections of Bote et al.
[22]. However, Llovet et al. use the X-ray emission quantities (fluorescence, Auger yields and rates as well as the Coster-Krönig rates) of  [20]  while here a more recent database is used  [23]. In our work, the quantities that are related to the emission of X-rays ( () ,  () ,  ( ′ ) ,  () ,  () ) are extracted using the xraylib package  [18]. Each line is then modeled by a Gaussian function having a full width at half maximum given by the detector response and an intensity proportional to  0 () and the concentration of the corresponding element. Therefore, the characteristic X-ray spectrum of an element is modeled by a sum of Gaussians with intensity ratios given by  0 (). The X-ray fluorescence spectrum is not only composed of discontinuous lines but also includes a lower intensity continuous background due to the bremsstrahlung.
Datacube simulation In this section, the creation of STEM-EDX datacubes representing multiple phases is presented. The models generated are spectrum images (or hyperspectral datacubes), as they present the most general case of STEM-EDX data. The simulation process can be summed up in several steps. First, pure spectra for each phase are created with the desired chemical composition and bremsstrahlung parameters. Then, an abundance map is generated for each phase. These are the components that will serve as the ground truth when assessing the result of a decomposition. The two quantities are combined using a tensorial product, generating a noiseless datacube. Statistical sampling is then used to generate a noisy data set that represents an experimental one. In the following, we give a detailed description of each step in their order.
The physical parameters such as   and  can be hard to establish, since they require microscope information that is often not available with good precision to the experimentalist. However, since they basically influence the total number of counts in the spectra, they are not needed, and therefore not included, in the spectrum simulation. They are instead implicitly accounted for in the last simulation step, when the average number of counts per pixel is set to generate the noisy datacube. The bremsstrahlung is scaled down (noted  ′() ∝  () ) such that its intensity is within realistic boundaries (usually a few percent of the height of the most intense characteristic peaks). Throughout the rest of this work, a simpler formulation  () compared to  () is therefore used, as follows: 𝜙 (𝑙) 𝑖 = ∑ 𝖹 𝜅 𝑖,𝖹 𝑐 (𝑙) 𝖹 + 𝑏 ′(𝑙) 𝑖 ( 14 ) A uniform sample density and thickness is defined for the full dataset, and each phase spectrum is normalized so that the sum of its energy channels is equal to one.
The spectra are collected in a matrix  ∈ R × , where  is the number of phases. The elements of  can be written: ()  =  (). The phase abundance maps can be built from any set of grayscale images. The espm package further provides functions to create abundance maps for model systems, such as for spherical nanoparticles embedded in a matrix. In this case, the thickness distribution ((, )) of a spherical nanoparticle of radius  and of center ( 0 ,  0 ) is modeled as 𝑡(𝑥, 𝑦) = 2 × √ 𝑟 2 -(𝑥 -𝑥 0 ) 2 -(𝑦 -𝑦 0 ) 2. For (, ) distances from the center greater than ,  is set to 0. 0. This thickness distribution is normalized so that it can be used as a single phase abundance map comprised between a chosen maximum concentration at ( 0 ,  0 ) (e. g. 0. 8) and 0. 0. For each pixel, the sum of all the abundance maps is normalized to one, such that the abundances are like the concentrations of the phases.
For instance, to simulate a 128 × 128 pixel 2 acquisition of 10 min having an average of 2000 counts s -1 on the detector, the user would set a value of  = 73. 𝑁 𝑗,𝑘 = 𝑁 𝐾 ∑ 𝑖 𝑑 𝑖 𝑎 𝑖𝑗𝑘 In order to assess the performance of machine learning algorithms, the ground truth of the noisy dataset can be generated in the following way: 𝒀 nl = 𝑁𝑫 ⊗ ( Diag(𝒅)𝑨 ) (17) The simulation method presented here uses a formulation very similar to that of decomposition algorithms such as NMF. It is therefore relatively straightforward to compare a decomposition output with the ground truth. The simulated data and its associated ground truth presented above are generated using the espm Python package. The espm library was developed as an extension of HyperSpy to make it straightforward for electron microscopists to integrate the simulations into their workflows.
The STEM-EDX experiment was performed on a FEI Tecnai Osiris, which is equipped with four windowless ''Super-X'' EDX SDDs. The beam energy was 200 keV, its current was 1 ± 0. 25 nA, the dwell time per pixel was 80 μs and the total map acquisition time was about 15 min. An example of an high-angle annular dark field (HAADF) image acquired on the sample is displayed in Fig. 3. The sample exhibits the typical microstructure described above, with Fp and Ca-Pv inclusions in a Brg matrix. In the following, we simulate STEM-EDX data based on the experimental data acquired from this region, and then compare the two. In the experimental dataset, the EDX spectrum for Brg can be isolated from the other phases. However, there is no direct way to extract the pure spectra of Fp and Ca-Pv, due to the spatial and spectral overlap between the three phases.
The latter is similar to the approach used by Edmondson et al. who visually compared outputs from experimental and simulated STEM-EDX datacubes treated by multivariate statistical analysis  [51]. For instance, in Fig. 5 , the elemental maps, single pixel spectra and all-pixel sum spectra of the two datasets are shown side by side. Overall, there is a strong resemblance between the simulated and experimental data, with the noise characteristics of the maps and the spectra being very similar for both datasets. While a few small discrepancies can be pointed out, as described next, these primarily derive from incomplete knowledge of the real sample and not from an improper simulation algorithm. Because of the lack of chemical ground truth, it was not possible to input the compositions of the simulated Brg, Fp and Ca-Pv so that the average compositions of both datasets match.
Also, the built spatial distribution of phases is probably not a complete match to the real distribution, leading to additional discrepancies between the two datasets. The effect of this can be seen either in the different magnitudes of the elemental maps, or in the different intensities of the O-K, Mg-K and Si-K peaks in the sum spectra. The simulation also did not reproduce the compositional effect of the beam damage that can be seen at the bottom left of Fig. 3. This is an effect of the Gaussian blurring processing step used to generate the phase distribution maps. In the sum spectrum of the simulation, a peak at 930 eV is observed, in contrary to the experimental spectrum. This peak corresponds to the Cu-L line. This discrepancy probably originates from a strong absorption effect in the Cu grid to which the STEM lamella was attached.
There might also be an effect coming from a miscalculation of the theoretical ratio between the Cu-K and Cu-L cross-sections. Also, the theoretical energies of the characteristic peaks from emtables are sometimes slightly shifted compared to their experimental counterparts, e. g. 0. 2 keV to a lower energy for Fe-K. NMF decomposition of both the experimental and simulated datasets was performed using three components and the scikit-learn library  [52]. To account for the Poisson statistics of the data, the negative logarithm of the Poisson likelihood was optimized iteratively with alternating multiplicative updates  [53]. The results are displayed in Fig. 6 , taking the noiseless simulated dataset as a proxy for the ground truth. The decomposition clearly yields similar results on both the simulated and the experimental data.
Despite these small differences in the decomposition outputs of the simulated and experimental datasets, we see that in general they share a strong resemblance. Further, the decomposition of the simulated dataset identifies similar artefacts and improper unmixing as that for the experimental dataset. This demonstrates how the simulated dataset can be used for correctly assessing the capability of a machine learning decomposition approach and diagnosing its limitations. Considering both this and the strong resemblance of the two datasets presented in Fig. 5 , our results therefore validate the packages we have developed for creating synthetic STEM-EDX spectrum images. We also note that the time taken to synthesize the presented simulated dataset (200 × 200 pixels and 1980 channels) was 10 s on a computer equipped with a Intel CORE i7 processor of 8th generation and 32 GB of random access memory, therefore fulfilling the requirement that this simulation be time efficient, taking no more than a few minutes.
Conclusion We have presented two open-source Python software packages: emtables for state-of-the-art calculations of X-ray emission crosssections; and espm for simulating STEM-EDX datasets, as based on values from emtables. With these, one can simulate 3D datacubes in under a minute of computation time for a 512 × 512 × 2048 dataset. For ease of use, we have designed the espm library to be integrated into the HyperSpy framework  [40]. By presenting a complete description of the cross-section computation, we show the relationship between chemical composition and a modeled phase spectrum; a link that can be used to retrieve the absolute chemical composition from an experimental EDX spectrum. Model EDX spectra are then combined with abundance maps via a tensorial product to yield simulated STEM-EDX datacubes.
The noiseless datacube gives a ground truth that is directly comparable to the typical output of matrix factorization algorithms. To test the algorithms, statistically-correct noise patterns can be applied to create a 3D datacube representative of an experimental one. Here, our approach is validated by comparison to experimental STEM-EDX data from the field of geology, where we both consider raw data resemblance and matrix factorization outputs. Observed differences primarily arise from small inaccuracies in the theoretical X-ray emission cross-sections or between the experimental dataset and the theoretical ground truth, thereby confirming the general robustness of our approach. While here we focus on the case of machine learning algorithm development, the ability to simulate STEM-EDX datasets offers other useful applications to the electron microscopist.
a b s t r a c t Digital forensics is the process of identifying, preserving, analyzing, and documenting digital evidence for investigation purposes. Building or using file analysis tools is of great interest for a forensic expert to collect high-level information in a short time. In this paper, we consider the examination of files contained in digital media, especially files with possible incorrect types. This often reveals a simple way to hide sensitive content such as porn images, passwords, or accounts. Many commercial and free forensic tools are available for file type identification (FTI). In this work, we assess the performance of ten of them on two significant datasets and scenarios. The main issue we address is the relevance of the tools for forensic purposes. The underlying question is: do expectations meet reality? Our experiments highlight the significant disparity in the accuracy and behavior of the studied tools.
, 2014). Their performance should be assessed in terms of speed, but also in terms of accuracy, using well-defined formal assessment procedures, in a way to reinforce trust  (Bhat et al. , 2021; Lazaridis et al. , 2016). This is precisely one of the objectives of Digital Forensic Science: contributing to the professionalization of Digital Forensics by providing objective and independent evaluation protocols in order to lead to reproducible results  (Horsman, 2019; Casey, 2019; Dimpe and Kogeda, 2019). This aim leads to define evaluation protocols, with explicit guidelines, clear evaluation metrics, and dedicated datasets. In an operational context, a forensic expert needs to use one or more tools to quickly process files stored in a device. Many commercial off-the-shelf (COTS) products are proposed and many free open-source software are also candidates for this task.
2k stars on GitHub), detect-file-type (11 stars on GitHub), guessfile-type (4 stars on GitHub)  (Sorhus, 2014; h2non, 2016; Paloskin, 2016; Kosinix, 2018). We also considered four (4) professional solutions: fidentify, TrID, EnCase and Autopsy  (Grenier, 2019; Pontello, 2003; Shawn and McCreight, 1998; Corp, 2012). We compare them in terms of accuracy and computing performance in order to find the most efficient ones. Our objective is to help investigators and researchers choosing the one that suits their needs. Moreover, we are also interested to better understand their limitations to be enhanced in the future. The contributions of the paper are the following. C A large-scale literature review of existing tools and methods for file type identification is provided. We consider both academic and non-academic forensic tools, including commercial tools used in digital investigation cases; C A rigorous protocol, involving two significant complementary datasets and performance metrics for the comparative study of file identification tools, is introduced; C One dataset has been specifically built in order to address the coverage issue of the tools regarding the number of file types properly taken into account.
It contains 17500 files with 110 different file extensions; C A large dataset composed of 1 million files with 63 different file extensions from Digital Corpora  (Garfinkel et al. , 2009)  is used in order to simulate a real case and assess execution time performance; C Two concealment strategies have been worked out as scenarios: randomly altering all file extensions, systematically removing them; C An in-depth analysis is conducted to identify the parameters that impact the performance of the different tools. A simple experiment shows that combining currently available tools permits to significantly improve the accuracy rate, from þ0. 2% to þ5. 9% on the two evaluation datasets; C This work demonstrates the benefits of benchmarking forensic tools independently and rigorously. The paper is organized as follows.
Background A file can be defined as information represented by a byte sequence, organized according to a convention or format, proposed by an author or a consortium. Files are designated by an external filename that is usually composed of two parts in Windows-like environments: the name which allows a user to have access to it, and the extension that signifies the file type. With the extension, the operating system can determine which software to launch to open the file: a file association table maps file extensions to applications. Using file extension is a simple way to identify common file types. Indeed, some extensions are so common that they can reliably be used to categorize a file: The probability that these extensions match other file types is very low. For instance, we may think about exe, which refers to an executable file on Windows, txt which designates a plain text file, or jpg, png and gif which are common image formats.
, 2021). The number of file types (or classes in this case) is quite low since the approach does not scale very well. The work done by  (Beebe et al. , 2016)  considered 50 file types (the highest number) but the accuracy is quite low (74. 1%). Last, all these works consider low dataset sizes (the largest one contains 60000 files). For all these reasons, these approaches are limited for an operational use for a forensic expert, and are not really used in practice. Many software solutions are available, some are commercial or open-source on GitHub. We can notice that the number of file types considered in each application is very different but much higher than in scientific papers. Most of these solutions use magic numbers and sometimes other information. One of the problems is that their accuracy and computation efficiency are not available.
Datasets In this work, we use two datasets corresponding to different testing scenarios, different number of files, file types and file sizes. 1. Dataset 17500: It is composed of 17,500 files with 110 different file types, it consists of various documents retrieved from public databases  (Garfinkel et al. , 2009) , images found on the internet and files generated with converters. Fig. 1  describes this dataset. Concerning the ground truth (expected file type), we consider that all file types are correct (dataset used in the literature). However, we have made random manual checking to ensure confidence in the quality of this dataset.
File-type javascript package File-type  (Sorhus, 2014)  is a JavaScript package which detects file types by checking signatures (i. e. magic numbers). This package handles binary-based file formats, not text-based formats. File-type takes as input the path of a file and returns the file type as well as its MIME type. It can also determine this information by taking a stream as input, for instance a request to a server, or even the content of a buffer. This module works asynchronously, which makes it possible not to block other tasks which are executed simultaneously. 135 file types are handled by this application.
Detect-file-type javascript API Detect-file-type  (Paloskin, 2016 ) is a JavaScript API based on File-type. Contrary to File-type, it handles not only binary formats, but also textual documents (e. g. xml, svg, html). It returns an object containing the MIME type and the assumed extension of the entry. Detect-file-type offers the user to add their own signatures and add their own file format detection methods. 94 file types are recognized by this application.
Guess-file-type javascript API Guess-file-type  (Kosinix, 2018)  works on a file passed as input. It carries out several tests to try to determine as precisely as possible what the type of the file is. When it succeeds, the module returns only a MIME type, it is possible to use a function to deduce the extension from the MIME type. Guess-file-type offers the possibility to use the tests separately. A detection by signature (i. e. magic number) is available: 34 file types are handled. The attribution of a MIME type from the file type is also proposed: 985 file types are mapped to 766 default MIME types.
ForENSIque filetype rust module ForENSIque is a software created by ENSICAEN students in France. The code is not publicly available, but was kindly provided by the authors for testing. It has been developed in Rust language and uses magic numbers to recognize 155 file types through internal mapping. The first 16 bytes of the file are retrieved then analyzed. Two cases arise, either the first four bytes are one of the two predefined sequences, 'FORM' and 'RIFF', or not. In the first case, depending on whether it is 'FORM' or 'RIFF', the four bytes 8 to 11, are considered. That method allows the application to recognize 20 file types such as IFFCelAnimation (. anim) or Audio-VideoInterleave (. avi). In the second case, the internal mapping is used. From 2 to 16 bytes long, 134 magic numbers are mapped to the corresponding file types.
EnCase Endpoint Investigator software EnCase  (Shawn and McCreight, 1998 ) is a software created by Shawn H. McCreight for the company Guidance Software, later acquired by OpenText. We had the opportunity to use a trial version of EnCase Endpoint Investigator which is one of their products. The feature that interested us analyzes the signature of the file and if it corresponds to a known one, then it indicates if the detected file type is consistent with the one suggested by the extension. If not, it returns a file type according to the shown extension. 2. Dataset 1M: The second corpus is composed 986,263 files with 63 different file types from  (Garfinkel et al. , 2009). It is characterized in Fig. 2. This dataset is used to estimate performance at scale (much more files and fewer file types) to simulate operational conditions.
3. 2. 9. Autopsy digital forensics software Autopsy 4. 19. 3  (Corp, 2012 ) is a widely used graphical user interface (GUI) that operates "the sleuth kit" to analyze files from disk images and is maintained by Basis Technology Corp. When the file signature exists, this software analyzes it to predict its corresponding MIME type and if it is not known, it may rely on shown extension.
Performance metrics In this study, we used two metrics to evaluate the performance of each tool by considering the following metrics. 1. Computation time: we measure the time needed for the processing (on the same computer). As this kind of tool may be used on a hard disk with millions of files, this is a key relative indicator. 2. Accuracy: we compute the file type recognition accuracy. The two tested datasets used in this work provide the ground truth. In some test scenarios, we modify the file extension or suppress it to analyze the impact on accuracy.