dataesr/scientific-paragraphs-categorization huggingface dataset:
Discussion Our tool leverages a combination of several novel approaches to accomplish an order of magnitude speedup over existing methods without compromising accuracy, and in many cases NINJA-OPS is more accurate than popular existing tools. In combination with a recommended denoising step, the pipeline achieves up to two orders of magnitude speedup over USEARCH 8. The key innovation of this tool is our use of a single long reference genome, or concatesome, composed of concatenated marker genes. This approach allows NINJA to leverage the benefits of the Burrows-Wheeler transform (Fig  5 ). The code is available at http://ninja-ops. ninja (or https://github. com/GabeAl/NINJA-OPS).
Deduplication and NINJA-filter Optimizations within NINJA-OPS include tweaks to the parsing and filtering programs to increase the throughput of the processes leading up to the alignment. Deduplication is a viable strategy in marker-gene sequencing contexts because samples usually consist of fewer taxa than there are reads, and in fact are often dominated by a few highly abundant species. This results in a large number of identical reads which can be filtered out to reduce the alignment time. In human gut datasets which are quality-trimmed (or where the marker gene reads are of approximately equal length), this may result in losslessly discarding 80% of the reads as duplicates, depending on the microbial community sampled, which can speed up the downstream alignment step substantially (S6 Fig).
A sequence-to-sample(s) dictionary keeps track of the abundance of each sequence in each sample to ensure that each original sequence is properly accounted for wherever it was originally found. By default, NINJA-filter also performs read compaction (parameter "-d 1"), which normalizes for variation in read lengths within a dataset by treating reads which are subsets of longer reads as copies of the longer reads. This increases consistency of OTU calling as well as decreasing runtime. This behavior can easily be disabled (parameter "-d 0"). An optional beneficial feature during the filter step is the ability to perform lossy denoising. NINJA performs this in two ways. The first and most straightforward for amplicon reads is to discard singleton reads (parameter "-d 2"); that is, reads that have no identical match in the entire list of queries, or which are not perfectly contained in a longer read.
This can be extended as the user desires from singletons to doublets and so on (parameter "-d 3", "-d 4", etc. ). The second form of denoising is discarding reads judged to be erroneous by breaking each read into its component overlapping k-mers and comparing each of these k-mers to the counts of that kmer in an empirical distribution of all k-mers in the body of input reads. Reads with k-mers that fail to meet user-defined criteria for support (appearing under a certain % in the dataset) are discarded completely from subsequent analysis. The resulting speedup for the downstream alignment is often much greater than the proportion of reads discarded, because Burrows-Wheeler alignment programs expend a disproportionately large amount of effort attempting to align erroneous reads that will not match the database compared to non-erroneous reads which will often find perfect (or near-perfect) matches in a well-populated database.
The BW substring search is designed for perfect substring searches, so it performs most efficiently in aligning reads that have few to no mismatches with a subsequence of the database. This is also why NINJA and BWT tools perform most effectively when the alignment identity is high (%ID in the mid-to upper-90's, with taxonomic resolutions at the level of genus or finer). Performance of BWT-based tools is expected to increase as the diversity of available reference sequences increases, because the probability of finding a perfect match likewise increases. One early concern as we were considering how to most effectively construct the concatesome was that some reads would align by chance to the boundary between two concatenated marker genes, which would produce a meaningless mapping. However, in practice, such an occurrence is exceedingly unlikely to occur in end-to-end marker gene alignments at genuslevel or greater resolution due to the high identity expected over the entire length of the input read.
However, due to the ability to substitute alternative BWT-based alignment programs for the alignment step, it is possible to use alternatively optimized variants such as HPG Aligner, which uses uncompressed suffix arrays instead of "traditional" BWT but shares many of the same characteristics with the added benefit of better scaling for longer reads. GPU-accelerated variants of the original algorithm are also available  [22]. Additionally, NINJA-OPS is not restricted to the domain of 16S OTU picking, although it is distributed with a pre-built 16S database. Marker genes such as ITS for fungal identification  [24] , bacterial rpoB  [25] , and the recently proposed Cpn60 universal bacterial barcode  [26]  are easily incorporated into NIN-JA-OPS simply by compiling the included "ninja_prep. c" and running it on an appropriatelyformatted FASTA file containing the desired marker sequences, followed by the BWT-based aligner's database generation step.
Further, NINJA-OPS can be incorporated as a preliminary step in another pipeline; for instance, NINJA-OPS can be used to group reads prior to de novo assembly  [27]. This flexibility of the pipeline in allowing substitution of the aligner itself, as well as the marker gene database used, makes NINJA-OPS applicable for situations and optimizations beyond what were envisioned at the time of writing, and enable the pipeline to keep pace with emerging technologies in the sequencing and computing spheres alike.
This paper proposes a software and hardware solution for real time condition monitoring applications. The proposed device, called XpertTrack, exchanges data through the GPRS protocol over a GSM network and monitories temperature and vibrations of critical merchandise during commercial shipments anywhere on the globe. Another feature of this real time tracker is to provide GPS and GSM positioning with a precision of 10 m or less. In order to interpret the condition of the merchandise, the data acquisition, analysis and visualization are done with 0. 1 ˝C accuracy for the temperature sensor, and 10 levels of shock sensitivity for the acceleration sensor. In addition to this, the architecture allows increasing the number and the types of sensors, so that companies can use this flexible solution to monitor a large percentage of their fleet.
XpertTrack was designed to work below freezing point and for long periods of time, having a higher weight (i. e. , 850 g because of the large battery) than other devices. The important advantage is the high accuracy temperature measurement (˘0. 1 ˝C accuracy and ˘0. 01 ˝C sensitivity), which is usually found in high accuracy temperature data loggers or reference probes. The probe sensor can go over the device temperature range, supporting cryogenic temperatures (´200 ˝C) and very high temperatures (+200 ˝C) and it is connected through a flexible cable of 480 mm (customizable to 1000 mm). When it comes to shock or free fall detection, XpertTrack measures acceleration on three axes. The sensor has 10 configurable levels in order to determine the shock intensity or if a free fall takes place.
In through the implemented software, instant alerts up to 16 G can be triggered, which is similar with other products described in  [10] [11] [12] [13]. The XpertTrack was not designed for small packages, but more for large merchandises were weight is not an issue. The device's accuracy is proved to be critical in cold chain shipments. The client is assured of the product temperature accuracy through a calibration certificate. The system architecture has an Access Point, which allows transferring the data from the real time tracker to a PC much faster than using GSM connection. Compared to a USB connection  [10, 13] , having a high speed wireless communication allows configuring and checking the sensor status without needing to open the container.
XpertTrack: Technical and General Specifications XpertTrack, presented in Figure  1 , was designed to be suitable for applications like: cold chain shipments, logistics, package integrity and remote monitoring. The alarms are triggered when products exceed the allowed temperature and vibration during shipments. In the case that the shipment has been tampered with, the current position and the temperature of the container are transmitted. XpertTrack technical and general specifications are mentioned in Tables  1  and 2. monitoring for temperature (package content supervision) and acceleration (package integrity) is required  [7, 8]. The device was designed to have two months of autonomy and in this time interval, the data acquired are sent every hour to the client PC. Due to the high power consumption of GPRS, powering from a battery becomes a challenge in remote monitoring  [9].
XpertTrack was designed to work below freezing point and for long periods of time, having a higher weight (i. e. , 850 g because of the large battery) than other devices. The important advantage is the high accuracy temperature measurement (±0. 1 °C accuracy and ±0. 01 °C sensitivity), which is usually found in high accuracy temperature data loggers or reference probes. The probe sensor can go over the device temperature range, supporting cryogenic temperatures (-200 °C) and very high temperatures (+200 °C) and it is connected through a flexible cable of 480 mm (customizable to 1000 mm). When it comes to shock or free fall detection, XpertTrack measures acceleration on three axes. The sensor has 10 configurable levels in order to determine the shock intensity or if a free fall takes place.
XpertTrack: Technical and General Specifications XpertTrack, presented in Figure  1 , was designed to be suitable for applications like: cold chain shipments, logistics, package integrity and remote monitoring. The alarms are triggered when products exceed the allowed temperature and vibration during shipments. In the case that the shipment has been tampered with, the current position and the temperature of the container are transmitted. XpertTrack technical and general specifications are mentioned in Tables  1  and 2.
For controlling the device, the main microcontroller is a CC2530 from Texas Instruments  [16] , with 256 KB of integrated Flash, which is enough for the software application. This microcontroller is used to acquire data from all sensors, to save the data and command the slave microcontroller. The CC2530 has an integrated IEEE 802. 15. 4 compliant radio frequency transceiver which is used to For controlling the device, the main microcontroller is a CC2530 from Texas Instruments  [16] , with 256 KB of integrated Flash, which is enough for the software application. This microcontroller is used to acquire data from all sensors, to save the data and command the slave microcontroller. The CC2530 has an integrated IEEE 802. 15. 4 compliant radio frequency transceiver which is used to communicate in short ranges  [16, 17].
‚ For temperatures above 0 ˝C, Equation (  6 ) is used, and the deviation ∆W is determined using relation Equation (  7 ): T 90 {K ´273. 15 " D 0 `9 ÿ i"1 D iWr pT 90 q ´2. 64 1. 64 ˙i (6) ∆W " a 10 p W pT 90 q ´1q Acceleration data are acquired using a LIS3DH from ST Microelectronics. This sensor is a digital one, so other data processing is not necessary. It can be set to measure acceleration, free fall, double click actions and many more. The main advantage of the selected sensor, in addition to its low power consumption, is the property of having interrupts on configurable thresholds. By doing so, battery saving can be achieved. The communication with the acceleration sensor is accomplished through an I 2 C interface. The Real Time Clock (RTC) selected is an EM-3027, designed by EM Microelectronic.
Firmware Design Figure  5  shows the firmware architecture based on Round Robin, with interrupts and state machines. First step is to initialize the main microcontroller, then the slave microcontroller, and in the end all other peripherals are initialized. After all initializations have finished successfully, the main loop is started. From this moment on, the main microcontroller will run the interrupt routines, which are triggered by the peripherals. In the awake state, initiate by the RTC alarm, the temperature and all other parameters values are acquired. The new data are stored in the memory chip. The TELIT module is configured by the main microcontroller and its functionality consists in the following steps: power-up, search for signal, connect to GPRS and acquire GPS and GSM positioning.
The messages are sent to a server specified by an URL (Uniform Resource Locator) or by an IP (Internet Protocol) address. The connection with the server can be performed on a specific port, for example HTTP (Hypertext Transfer Protocol) standard port (i. e. , 80), or a custom one, for security reasons. When an alternative random port is used instead of the standard HTTP port a supplementary level of protection against unwanted access can be achieved. The protection consists in an unknown port and the access is granted using user and password credentials (each XpertTrack has a unique user and a password generated using a proprietary algorithm). In order to be sure that the sent data reaches the destination, the communication is done using HTTP protocol, even if port 80 is not used. On top of that, the server must acknowledge the received information.
Through this implementation, the communication concept guarantees that the information reaches the server and is analyzed. Additional attention must be paid on power supply supervision. The slave microcontroller is not able to run below 3. 4 V, whereas the rest of the components run at 3 V or less. The microcontroller's internal A/D converter is used to measure battery voltage. In the case of low battery levels, XpertTrack is designed to stop the GPRS communication and continue to acquire and save the data. In this way, no data are lost and it can be recovered at destination. The server is notified with respect to the battery critical level, prior of communication interruption.
The server software is built on a Java platform and runs on a dedicated computer with a public IP. However, in the near future, Webhost features will be implemented on the server. All the exchanged data are being stored on an SQL Database with several layers of protection. In addition to the Java platform, different software (called client software) has been developed to communicate with the server, extract and display the required information and to ensure surveillance on the threshold levels. With this configuration, all data can be seen from all over the world at any given moment. Even when data cannot be sent to the server due to connection problems, the acquired samples from the sensors are not lost, but stored in the device internal memory, based on a FIFO stack-up. The information can be recovered, either when the connection is re-established, or by downloading it by the user through the wireless connection using the AP.
Programming and Configuring the Device The programming is done by the client through the software. The user can set the scan rate (at one hour, XpertTrack can run for more than two months), the update rate, the shock sensitivity and several alarms. After this, the software checks for GSM signal and when it gets it, sends all the information. From this point on, the device is responsible for data transmission and proper functioning. However, the user can modify certain parameters and will be notified when requested changes have been accepted.
GPS and GSM First sets of tests were performed at room temperature to check the communication reliability and quality. Positive tests have been performed measuring QoS (Quality of Service) of the GSM and GPS link. Due to the impedance matching, the device managed to get GSM signal everywhere a The server software is built on a Java platform and runs on a dedicated computer with a public IP. However, in the near future, Webhost features will be implemented on the server. All the exchanged data are being stored on an SQL Database with several layers of protection. In addition to the Java platform, different software (called client software) has been developed to communicate with the server, extract and display the required information and to ensure surveillance on the threshold levels. With this configuration, all data can be seen from all over the world at any given moment.
A worst case testing has been performed: where during startup or in the middle of the communication, the GSM antenna has been removed causing communication lost. When that happened, a lack of GSM signal was detected and the communication dropped. At the next regular wake-up, the communication has been re-established and the data have been transmitted from the moment when the communication was dropped. The communication quality was tested in harsh temperature ranging from ´30 ˝C to +80 ˝C  [25]. The communication between the two microcontrollers was monitored for 24 h and no errors were detected. Although the device was in a freezer at ´30 ˝C or in an oven at +80 ˝C, the data arrived to the server without any loses. Besides testing in a controlled environment, real live testing was also done.
Figures  9  and 10  shows the test results of GSM positioning. As the GPS coordinates from the Figure  10  show, even if the connection is lost due to travelling by airplane (the long lines from Germany to France, and from France to Denmark), the transfer of data is re-established when GSM signal is redetected. Considering that the GSM penetration is far beyond the GPSs, especially if the XpertTrack is in a package in a metal container, it can be tracked almost everywhere on the globe. regular cell phone would work. In near future, tests will be done in areas with poor mobile phone coverage. Measurements to verify GPS accuracy have been made and were determined to have an accuracy of 10 m or less even if the receiver is in a building. If due to exceptional situations, GPS signal is not available, the location is determined using GSM triangulation.
The communication between the two microcontrollers was monitored for 24 h and no errors were detected. Although the device was in a freezer at -30 °C or in an oven at +80 °C, the data arrived to the server without any loses. Besides testing in a controlled environment, real live testing was also done. Figures  9  and 10  shows the test results of GSM positioning. As the GPS coordinates from the Figure  10  show, even if the connection is lost due to travelling by airplane (the long lines from Germany to France, and from France to Denmark), the transfer of data is re-established when GSM signal is redetected. Considering that the GSM penetration is far beyond the GPSs, especially if the XpertTrack is in a package in a metal container, it can be tracked almost everywhere on the globe. The main application of the device is to supervise the temperature change (which include alarms, etc.
Data reduction and redshift measurements Data reduction for each mask was carried out using the VIPGI software  (Scodeggio et al. 2005). After each CCD frame is bias subtracted, the precise location of the two-dimensional (2D) spectrum for each slit on each frame of the jitter sequence is measured and the position of individual object spectra inside the slit is recorded. The sky background subtraction is carried out for each slit independently, using a two-step procedure. First, the median background level at each wavelength is measured in a single exposure and subtracted from the data; then the median of the sky subtracted spectra is obtained without applying any correction for the telescope pointing offset between exposures. This median produces a frame from which the spectra of individual objects are eliminated, but that includes all residuals not corrected by the first sky subtraction step, and in particular the fringing pattern, which varies with position across the slit and wavelength and is then subtracted from the individual 2D spectra.
After this final 2D combination, a 1D spectrum is extracted from the combined 2D spectrum and is flux calibrated, following the same VIPGI procedures outlined above. The redshift measurement is also a multistep process. We start by conducting a fully automated redshift determination based on cross-correlation with template spectra coupled to emission lines and continuum fitting, carried out with the EZ redshift measurement software  (Garilli et al. 2010). Galaxy template spectra are drawn from those built for VVDS  (Le Fèvre et al. 2005 ) and zCOSMOS  (Lilly et al. 2007 ) surveys. This automated step is followed by a visual examination of 1D and 2D spectra of every object to assess the validity of the automated redshift measurement, or, if needed, to compute a new redshift, based on the wavelengths of the recognized spectral features.
Rest-frame magnitudes and stellar masses The spectroscopic data of the new COSMOS Wall sample of galaxies is matched with the multiwavelength photometric data for each galaxy to derive rest-frame galaxy properties. We then perform SED fitting to obtain luminosities and stellar masses Fig. 12. SSR as a function of a target I AB magnitude. The dark green histogram shows the magnitude distribution of the targets from the total parent sample where a slit was positioned. The filled light green histogram shows the magnitude distribution of the targets whose redshift is reliably measured (for both histograms the numbers are indicated on the left Y-axis). The filled points are the SSR, which is a ratio of the two histograms whose value is indicated on the right Y-axis. The continuous light green curve is the adopted function to parametrize SSR(I AB ).
69 ≤ z < 0. 72 and 0. 74 < z ≤ 0. 79 (empty blue diamonds). The black stars in Figure  14  are computed using the whole DR1 VIPERS survey data set in the redshift bin 0. 69 ≤ z ≤ 0. 79. Error bars include both Poissonian and cosmic variance contribution except for the points of the Wall structure, where the only error bar shown is the Poissonian one, as we do not claim to be dealing with a representative volume of the universe; see  Driver & Robotham (2010). While the region of the Wall structure displays a significant overdensity, as expected, with respect to the VIPERS data points, when this region is excised from the analysis, the number densities agree well with those obtained by the VIPERS data set. In the paper  Petropoulou et al. (2016a) , we will present a detailed analysis of the galaxy stellar mass function in the Wall volume, both as a function of environment and of galaxy colors and star formation properties; this paper will provide a more thorough investigation of this topic.
Detection of groups We targeted the Wall volume because of its interesting largescale structure: a significant galaxy concentration and two long filamentary structures, embedding groups, extending from it. In the original 20K-sample group catalog there are 19 (6) groups with 5 (10) or more spectroscopic members within the Wall Volume, and 11 (6) are located within the Wall structure at 0. 72 ≤ z spec ≤ 0. 74. With the new data set available we can improve the definition of groups as a result of our better spatial sampling within this interesting region. We use a friend-of-friends (FOF) group detection algorithm, in a similar fashion to  Knobel et al. (2012)  and  Knobel et al. (2009) , adopting the same optimization strategy as defined in these papers. The FOF algorithm has three parameters: the comoving linking length as defined by the b parameter, the maximum perpendicular linking length in physical coordinates, L max , and the ratio R between the comoving linking lengths along and perpendicular to the line of sight.