dataesr/scientific-paragraphs-categorization huggingface dataset:
Datasets and data preparation Between July 2010 and July 2018, a total of 1,272 esophagus endoscopic images were collected from 748 patients from the First Affiliated Hospital of Nanjing Medical University, which is the best and biggest comprehensive hospital in Jiangsu, taking charge of four central roles for the whole province: medical treatment, medical teaching, scientific research, and hospital ethics activities. The imaging data consisted of 531 normal esophagus images, 387 premalignant images, and 354 EC images. Endoscopic images were captured by Olympus endoscopes (GIF-H260Z, GIF-Q260, GIF-Q260J, GIF-XQ260, GIF-H260, GIF-H260Q, GIF-XQ240, Japan). The inclusion criteria of this database are those images with available conventional white-light endoscopy, chromoendoscopy, and narrowband imaging.
Results A total of 748 patients were included in this analysis. Table  1  presents the sizes and demographics of the database. Overall, no significant age difference was observed between males and females in each group. However, the normal control group was 15 years younger on average than the other two groups. Cancer and precancerous lesion groups had more males than females, both of which were around 60 years old. The comparative results of the proposed network and substreams (the O-Stream and the P-Stream) in the database are listed in Table  3. This database contains all images, including those of the normal esophagus, precancerous lesions, and EC. And the results are the overall ACC, SEN, and SPEC of each methods. The O-stream focuses on exploiting the color and global features of the esophageal images, and its ACC by itself was 66.
93%. Using the preprocessed image as the input, the P-stream focuses on exploiting the textures and detailed features of the esophageal images, and the ACC of p-stream alone was 79. 53%. The fusion of the two streams led to the best results of 85. 83%. Table  4  shows the ACC of each category in the EC database based on the proposed network. The normal type was easier to identify probably because the amount of data in the normal type was greater than the other two types. Figure  8  presents the confusion matrix for the EC database. In the confusion matrix, the diagonal values are the A of each category classification, and the others are the confusion degrees between the two categories. This method diagnosed 74 total lesions as esophageal lesions (the precancerous lesion or cancer); 3 were normal cases with a PPV of 95.
94% and a negative predictive value (NPV) 92. 45%. The PPV and the NPV of EC were 87. 09% and 91. 67%, respectively. The accuracy of the cancer category was 77. 14%, which implies that it is easy to confuse EC with the precancerous lesions. Table  5  demonstrates a comparison made between the method we proposed and the methods of LBP+SVM and HOG+SVM using the same dataset. The total sensitivity, specificity, and accuracy of our method were 94. 23%, 94. 67%, and 85. 83%, respectively, which are higher than those of the other methods.
Some other reasons may include the fact that the cancer was hard to detect on the surface or the poor angle at which the image was taken. The main contributions of this paper are twofold. First, the esophageal endoscopic database was built. The database included 1,272 endoscopic images, which consisted of 3 types of endoscopic images (normal, premalignant, cancerous). Each image in this database had a classification label. Secondly, we presented a two-stream CNN that can automatically extract global and local features from endoscopic images. The significant strength of the study was that our proposed two-stream CNN consisted of 2 subnetworks (O-stream and P-stream). The original images were input with the O-stream to extract the colors and global features, and the pre-processed esophageal images were input with the P-stream to extract the texture and detail features.
Shichijo et al. (19) applied a deep learning AI-based diagnostic system to diagnose Helicobacter pylori infections, and Hirasawa et al. (20) detected gastric cancer by using a CNN model. Moreover, several studies have constructed computer-aided methods to analyze the real-time endoscopic video images of colorectal polyps (21-23). However, there are only a few studies on EC detection. Horie et al. (24) used CNN to detect EC with a positive predictive value (PPV) of 40%, which is lower than expected. Yang et al. (25) trained a 3D-CNN model with the PET image datasets to predict EC outcomes.
Stacking of AF-Sk lattices along the [111] direction is determined by the propagation vectors and the Mn 2+ positions within the (111) layers. In the Methods section, we present an analytical ansatz for spins at general positions constructed as a superposition of three helical modulations, and the correctness of the fractional AF-Sk lattice is verified through comparison against the neutron diffraction dataset shown in Fig. S7  of the Supplementary Information. The bipartite character of the diamond lattice leads to bilayers with exactly the same spin configurations as explained in Fig. 4a , thus realizing three consecutive AF-Sk bilayers with shifted whorl centers. Such a stacking order leads to AF-Sk tubes along the [111] direction shown in Fig. 4c , which is a common feature for many skyrmion lattices  39, 40.
75 -0. 75 0) reflection with increasing and decreasing fields. The neutron diffraction dataset in the triple-q phase was collected on TriCS (now ZEBRA) at SINQ, PSI. Incoming neutron wavelength of 2. 32 Å was selected by the PG(002) monochromator. A PG filter was mounted before the sample. A cryomagnet together with a roots pump was employed for the measurements. 67 reflections were collected at T = 1. 60 K in a magnetic field of 3. 5 T along the [111] direction. Monte Carlo simulations. Monte Carlo simulations were performed using the Metropolis algorithm by lowering the temperature in an annealing scheme and computing 500 independent runs initialized by different random numbers for each temperature and external magnetic field. Simulations were performed in 2 × L 3 magnetic site clusters, with L = 8-24 and periodic boundary conditions.
ACKNOWLEDGEMENTS We acknowledge  S. Tóth  and  S. Ward  for help in the analysis of the neutron spectra. We thank  A. Scaramucci  for the initial trial on the Monte Carlo simulations. We acknowledge helpful discussions with  M. Pregelj ,  S. B. Lee ,  T. -h. Arima ,  T. Nakajima ,  J. S. White , and  Y. Su. S. G. acknowledges fruitful discussions at RIKEN CEMS. F. A. G. A. and H. D. R thank  R. Borzi  for fruitful discussions. H. D. R. thanks  M. Zhitomirsky  for helpful discussions about the Monte Carlo simulations. Our neutron scattering experiments were performed at the  Swiss Spallation Neutron Source SINQ, Paul Scherrer Institut, Villigen, Switzerland , the  Heinz Maier-Leibnitz Zentrum MLZ, Garching, Germany , and the  Institut Laue-Langevin ILL, Grenoble, France. This work was supported by the  Swiss National Science Foundation  under Grants No.
Currently, however, there are only very few strategies that aim at bridging biological and preclinical imaging-even though these Novel CMI Pipelines reap the full potential of this approach in tackling biomedical research questions mechanistically. In this context, data handling and Correlation Software for diverse imaging data sets play a crucial role in CMI. While the benefits of PHI and CLEM are more and more recognized in biomedical research, they lack gold standards for protocols or data handling and limit quantification. This includes for example the quantification of the correlation accuracy in CLEM or biomedical imaging ontologies. Apart from standardization, both PHI and CLEM leave room for optimization and the integration of advanced setups, such volume or super-resolution CLEM  [6, 7]  or hybrid preclinical multimodal platforms for Optical Coherence Tomography (OCT), Photoacoustic Imaging (PAI), and non-linear in-vivo microscopy  [8].
PHI serves as a valuable diagnostic and research tool that can uncover molecular processes and biochemical pathways in living animals non-invasively within their anatomy, and has been used to study wide variety of biomedical questions, for example in cancer biology or brain research  [68]. CLEM has become the method of choice to analyze rare and specific processes within tissues or cell lines, and has been used to study a wide variety of biological questions including membrane trafficking and viral pathways  [17]. The maturity of the two fields is reflected in several commercial implementations for PHI (e. g. , PET/CT or SPECT/CT), and a first commercially available integrated fluorescence and scanning electron microscope and commercial tools, ancillary equipment and software for CLEM to facilitate re-locating of the region of interest across modalities (see section State-of-the-Art).
In CLEM, mainly three software solutions are used to perform landmark-or segmentation-based registration: a plugin for FIJI  [90]  (distribution of ImageJ including many useful plugins) called BigWarp (initially developed to provide training and validation sets), a plugin for ICY called ec-CLEM  [91] , and the Note that these changes of the coordinate system (or transformations) have to be computed in 3D to take into account possible changes of obliquity, and that they do not take into account deformations induced by sample preparation from one imaging modality to another one. commercial software AMIRA (Thermofisher, Bordeaux France). Several structures have been used to correlate, including vessel, mitochondria, nuclei, added fiducials such as quantum dots (QDs) in the correlative microscopy field, or specific anatomical landmarks in the medical fields.
Note that a plethora of variants exists for point-cloud registration, some of them sounding particularly promising for feature-based multimodal registration  [100]. Interesting approaches mixed both feature-based and full registration by restraining the learning data set to registered features  [101]. For both approaches one of the commonly used libraries for software implementation is ITK (https://itk. org) usually coupled with its visualization counterpart VTK (https://vtk. org). Very powerful (command line) tools for landmark based or fully automatic image registration (rigid and deformable) are Elastix (http://elastix. isi. uu. nl), its derivative Simple Elastix (http://simpleelastix. github. io) and ANTs (http:// picsl. upenn. edu/software/ants/). They allow the definition of fully parameterizable complex registration pipelines.
One of the particular challenges is to deal with data that do not have the same dimensionality, such as time lapse vs. 3D or hyperspectral images, and heterogenous data  [107]. To keep the full resolution of the biggest image, with data that can reach several terabytes in size for just one specimen, efforts are ongoing regarding efficient approaches of displaying and manipulating very big data, such as the big data viewer  [108]  as also used in the BigWarp Fiji Plugin. For a more exhaustive list of software used in the field, the reader is invited to refer to a constantly updated list of software established in collaboration between the COST actions NEUBIAS (CA15124) and COMULIS (CA17121): www. comulis. eu & www. biii. eu.
Can we map single fluorophores onto a single protein structure inside a crowded cellular environment? This is currently a dream scenario but will likely be possible in the future (see section CLEM and Correlative Microscopy). For the moment we will have to do with internal or external added markers that can be used as fiducials for the alignment of the two datasets. If these dual-modality markers are used also to label specific proteins of interest, the first choice are quantum dots as they are fluorescent and their core, generally being made of Cadmium and Selenium, and is made of heavy metals for visualization in EM. Care must be taken however that the proteins coupled to such probes still fulfill its original function. We have shown that Transferrin coupled to QDs does not recycle anymore but, likely due to multiple receptors binding to one QD, is directed down the degradative pathway  [109].
Correlation Software As seen from section Correlation Software, different methods have already been proposed in the literature for image registration or multimodal visualization. However, a few of them are actually used by researchers in life sciences, and one of the main reasons for this is the lack of user-friendly implementation and availability as software. In addition, as underlined in the other sections, every biological or medical questions comes with its own image analysis workflow  [92]. In order to help with these workflows, but also to help data sharing and open science, a standardization of the representation of the multimodal spatial relationship and content types will be important. In the medical imaging field, such standards are in place, using Digital imaging and communications in medicine (DICOM), which includes guidelines for the representation of spatial transformation between multimodal images [linear (C20.
It can be applied to correlative microscopy but is limited to a rigid-transformation point-based framework, and has the advantage of not requiring any ground truth matching in the images. In addition, this method may lack a proper mathematical formalism to demonstrate their usability in the fields. Note that metrics to compare images based on intensity may not be adapted in most of the case due to the discrepancy of image content (structures not present in both images). In the case of the absence of fiducials or anatomical landmarks to validate the registration, another currently used method to validate the accuracy of a registration is to use segmentation quality metrics, such as DICE metrics, to assess the overlap between known segmented structure. Challenges are the competition of algorithms on a given dataset, where the ground truth is known in order to rank the algorithms according to a set of metrics designed for the challenge.
CLEM and Correlative Microscopy In the recently published book "Correlative Imaging: Focusing on the Future, " there was one over-arching theme that was highlighted in almost all chapters: Data. If handling data from a single modality can already create headaches for the data analysis and IT people, how about trying to combine datasets from different imaging modalities. Developments and possible solutions will be discussed in other parts of this review. On the hardware side there are also clear trends visible and they almost seem to diverge from each other. On the one hand, there are the cryo-CLEM approaches trying to map protein structures in ultrastructural data, on the other, volume CLEM (a collection of techniques including FIB-SEM, SBF-SEM, array tomography, electron tomography) is more focused on large scale structures and mainly looks at connectivity between cells.
Besides, there is the current challenge of correlating additional beneficial but not yet readily/commercially available imaging modalities (same challenge as faced by the biological microscopy community). For example, while fusion of optical tomography data (DLIT, FLIT, or FMT) with CT can be achieved using commercial imaging systems, co-registration of OI and MRI is still in the developmental state. As OI and MRI are recorded in separate systems, relocations of animals between the recording sessions have to be conducted with great diligence to avoid anatomical distortion and positional changes. Chehade developed a shuttle made of CT-and MRI-compatible material, which allowed the relocations of mice while keeping them properly in place  [77]. These challenges hold also true for in-vivo microscopydespite its impressive advances.
If no additional processing step is foreseen between modalities (which might induce distortions or even require reduction of the volume by sectioning), the straightforward approach to correlate the ROI across modalities is to use a joint transferable coordinating system. A well-established example is the annotation of FM to define ROIs with a dedicated CLEM module and import and relocate these coordination lists into the cryo-EM microscope using SerialEM  [169]. Other approaches focus on using the same holder for different imaging modalities. Examples include immobilization beds for preclinical imaging in mice using PET and CT (where additionally 22 Na fiducial markers can be placed into stationary pegs at defined depths to provide a 3D references to simplify image registration), or the combination of x-ray spectromicroscopy with electron tomography, where Allende meteorite grains were deposited on a TEM grid and transferred between the electron microscope and the COSMIC soft x-ray beamline  [170].
Currently, the focus of CMI pipelines is rather on identifying working combinations to address previously inaccessible biomedical research than on fostering throughput. Once those correlations have been showcased and are proven feasible by substantial R&D efforts, CMI will enter further automatization and simplification to generate throughput. (5) Advances and current trends in correlation software are discussed in sections Correlation Software and Correlation Software. To expedite automated multimodal image registrations and quantification, data handling specific to multimodality needs to be established, including universal imaging formats, ontologies, and data storage and repositories. While there is currently no universally established microscopy format with additional diversity between preclinical and biological imaging approaches, repositories and public archives for diverse imaging data are being implemented-from single molecules (EMPIAR) to tissues (Tissue-IDR).
Correlative data sets (such as CLEM data to link functional information across spatial and temporal scales) will be included in the so-called addedvalue databases that are developed around the archive. They aim at gathering a greater understanding for specific biological areas through systematic integration of images  [141]. Integration of such multimodal data sets and their interoperability will be facilitated by universal large-scale multi-granular imaging ontologies, whose need is being described in first publications  [186, 187].
Correlation Software As already underlined, most of the automated methods are developed ad hoc for a particular multimodal problem. Machine learning and deep learning are definitely moving the image analysis field a step forward  [87]  since their main interest is to create an ideal method of processing based on training data sets, translating the effort of developing ad-hoc computer vision or signal processing methods to the effort of annotating data and formalizing the problem as input/output. Note that deep learning is still in progress and a lot of research is still going on to optimize these methods and reduce the number of training datasets required, as well as taking into account the errors in annotations in training data sets  [87]. Based on these approaches, a universal solution without any user input is not envisioned per se, but rather a universal framework for multimodal registration, or the creation of a giant bank of pretrained models.
This effort could be facilitated by two approaches: the single cells approaches, and the development of spatially localized proteomics or genomics which are now starting to appear as commercial platform and would facilitate these links. But then the analysis of this largely heterogeneous data still requires to develop new statistical tools. Another trend is to use multimodal aligned images as training sets to construct inference models to reduce the needs for one or the other modality. For example, Li et al. [189]  used a deep learning approach to generate PET images from MRI and demonstrated similar classification results using the generated PET images than the true PET images for Alzheimer Disease and Mild Cognitive impairment. In this preliminary study, a 3D convolutional neural network was trained with MRI patches as input and matching PET patches as output using half of a database of patients having both exams.
The parameters of the network capture a relationship between both modalities. This trained network was then used to generate the predicted PET images from MRI images, and validated against the remaining half of the database. The same principles have been applied also to microscopy images, where for example restoration based on deep learning have shown impressive results  [190] , and even predicting fluorescent labeling from transmitted light images  [191]. These approaches in the long term could then reduce the number of modalities required to answer a particular question. To achieve such a goal, sharing well-annotated aligned multimodal data is of particular importance. Efforts are on-going in this direction to share repositories and public archives (  [141] , Empiar, IDR).
CONCLUSIONS AND OUTLOOK: FUTURE OF THE FIELD To have an even bigger impact and to become a basic life science technology as FM is nowadays, it will be crucial for correlative microscopy to develop and disseminate automated workflows that can deal with huge amounts of data and seamlessly merge diverse data sets. This process will be facilitated by a number of factors: the improved capabilities of integrated systems, the adaption of standard file formats, and the deposition and sharing of these information-rich datasets. For PHI, preclinical molecular in-vivo whole-body imaging is fully dependent on hybrid imaging and co-registration with anatomical images. Some devices are already multimodal in their hardware settings, but images from different techniques are taken sequentially and implemented automated co-registration often requires manual intervention to obtain the best results.
ACKNOWLEDGMENTS  PP-G  acknowledges  ANR-18-CE45-0015  and the  France-BioImaging  infrastructure supported by the  French National Research Agency  ( ANR-10-INBS-04 ). LS acknowledges the large  RI  Project  LM2015062   Czech-BioImaging  funded by  MEYS, Czech Republic. KB acknowledges the  FFG COMET center project   854174. All authors acknowledge their collaboration facilitated by  COST as members of COMULS  ( CA17121 ).
We present a framework to solve non-linear eigenvalue problems suitable for a Finite Element discretization. The implementation is based on the open-source finite element software GetDP and the open-source library SLEPc. As template examples, we propose and compare in detail different ways to address the numerical computation of the electromagnetic modes of frequency-dispersive objects. This is a non-linear eigenvalue problem involving a non-Hermitian operator. A classical finite element formulation is derived for five different solutions and solved using algorithms adapted to the large size of the resulting discrete problem. The proposed solutions are applied to the computation of the dispersion relation of a diffraction grating made of a Drude material. The important numerical consequences linked to the presence of sharp corners and sign-changing coefficients are carefully examined.
The implementation is based on the open-source finite element software GetDP and the open-source library SLEPc. The solutions of such problems may have important applications in electromagnetism at optical frequencies, where frequency dispersion arises in bulk materials. Indeed, the permittivity of most bulk non-transparent materials, such as semiconductors and metals, strongly depends on the excitation frequency  [2]. But frequency dispersion also comes into picture when dealing with composites materials, or metamaterials, whose effective electromagnetic parameters derived from modern homogenization schemes  [3, 4, 5]  are frequency dependent. The accurate and reliable computation of the modes of frequency-dispersive structures represents a great challenge for many applications in nanophotonics.