dataesr/scientific-paragraphs-categorization huggingface dataset:
This single constrained parameter reduces the dependence of the extraction results on the human factor to a minimum and makes their analysis and derived conclusions as objective as possible. The numerical code is designed to be user-friendly and easy to run, providing diagnostics to help users avoid common problems. It verifies the getsf configuration, input images, and masks for consistency, and it suggests solutions in various circumstances during extractions. The software includes 21 utilities and scripts (Appendix C), providing all kinds of image processing necessary for getsf to run and more. They include the fitfluxes utility for spectral energy distribution fitting of source fluxes or image pixels (and mass derivation) and the hires script that computes the high-resolution surface density images (Sect.
Conclusions This paper presented getsf, the new multiscale method for extracting both sources and filaments in astronomical images using separation of their structural components. It is specifically designed to handle multiwavelength sets of images and extremely complex filamentary backgrounds, but it is perfectly applicable to a single image or very simple backgrounds. The new code is freely downloadable from its website 14  , from the Astrophysics Source Code Library 15  , and also available from the author. The main processing steps of getsf include (1) preparation of a complete set of images and derivation of high-resolution sur-face densities, (2) spatial decomposition of the original images and separation of the structural components of sources and filaments from each other and from their backgrounds, (3) flattening of the residual noise and background fluctuations in the separate images of sources and filaments, (4) spatial decomposition of the flattened components of sources and filaments and their combination of the over wavelengths, (5) detection of sources (positions) and filaments (skeletons) in the combined images of the components, and (6) measurements of the properties of the detected sources and filaments and creation of the output catalogs and images.
Acknowledgements. This study used the cfitsio library  (Pence 1999 ), developed at HEASARC NASA (USA), saoimage ds9  (Joye & Mandel 2003)  and wcstools  (Mink 2002) , developed at the Smithsonian Astrophysical Observatory (USA), and the stilts library (by Mark Taylor), developed at Bristol University (UK). The plot utility and ps12d library, used in this work to draw figures directly in the PostScript language, were written by the author using the psplot library (by Kevin E. Kohler), developed at Nova Southeastern University Oceanographic Center (USA), and the plotting subroutines from the MHD code azeus  (Ramsey et al. 2012) , developed by David Clarke and the author at Saint Mary's University (Canada). This work used observations obtained with XMM-Newton, an ESA science mission with instruments and contributions directly funded by ESA Member States and NASA.
This work used observations made with the Spitzer Space Telescope, which is operated by the Jet Propulsion Laboratory, California Institute of Technology, under a contract with NASA. This work used observations made with the NASA/ESA Hubble Space Telescope, and obtained from the Hubble Legacy Archive, which is a collaboration between the Space Telescope Science Institute (STScI/NASA), the Space Telescope European Coordinating Facility (ST-ECF/ESA) and the Canadian Astronomy Data Centre (CADC/NRC/CSA). This paper used the SCUBA-2 data obtained at JCMT under program MJLSG31. The James Clerk Maxwell Telescope is operated by the East Asian Observatory on behalf of The National Astronomical Observatory of Japan; Academia Sinica Institute of Astronomy and Astrophysics; the Korea Astronomy and Space Science Institute; Center for Astronomical Mega-Science (as well as the National Key R&D Program of China with No.
Intensities (in electrons s -1 ) are limited in range with logarithmic color mapping, except for Q λ , which is shown with square-root mapping. A&A proofs: manuscript no. getsf          Appendix A: Inaccuracies of the derived surface densities and temperatures The algorithms described in Sect. 3. 1. 2 imply that the 160, 250, 350, and 500 µm images have an accurate (consistent) intensity calibration. When we assume that the calibration inaccuracies can be described by constant wavelength-dependent offsets, simple consistency checks and corrections can be made. Three independent estimates of low-resolution temperatures (T L1 , T L2 , T L3 ) are readily available from fitting the images in three pairs of wavebands  (160-250, 250-350, and 350-500 µm)  with a low resolution of O 500. If the median values of the three temperature maps differ by more than several percent, it would be necessary to adjust some of the offsets and estimate T L{1|2|3} again.
INTRODUCTION The Linux kernel follows a two-tiered release model in which a mainline version, accepting bug fixes and feature enhancements, is paralleled by a series of older stable versions that accept only bug fixes  [41]. The mainline serves the needs of users who want to take advantage of the latest features, while the stable versions serve the needs of users who value stability, or cannot upgrade their kernel due to hardware and software dependencies. To ensure that there is as much review as possible of the bug fixing patches and to ensure the highest quality of the mainline itself, the Linux kernel requires that all patches applied to the stable versions be submitted to and integrated in the mainline first. A mainline developer or maintainer may identify a patch as a bug fixing patch appropriate for stable kernels and add to the commit message a Cc: stable tag (stable@vger.
Their approach relies on thousands of word features extracted from commit messages and 52 features extracted from code changes. The word features are obtained automatically by representing each commit message as a bag of words, i. e. , a multiset of the words found in the commit, whereas the code features are defined manually. The bag-of-words representation of the commit message implies that the temporal dependencies (ordering) of words in a commit message are ignored. The manual creation of code features might overlook features that are important to identify stable patches. To address the limitations of the work of Tian et al. and to focus on stable patches, we propose a novel hierarchical representation learning architecture for patches, named PatchNet. Like the LPU+SVM work, PatchNet focuses on the commit message and code changes, as this information is easily available and stable-kernel maintainers have reported to us that they use one or both of these elements in assessing potential stable patches.
Deviating from the previous LPU+SVM work, however, which requires human effort to construct code features, PatchNet aims to automatically learn two embedding vectors for representing the commit message and the set of code changes in a given patch, respectively. While the first embedding vector encodes the semantic information of the commit message to differentiate between similar commit messages and dissimilar ones, the latter embedding vector captures the sequential nature of the code changes in the given patch. The two embedding vectors are then used to compute a prediction score for a given patch, based on the similarity of the patch's vector representation to the information learned from other stable or non-stable patches. The key challenge is to accurately represent the structure of code changes, which are not contiguous text like the commit message, but rather amount to scattered fragments of removed and added code across multiple files, within multiple hunks.
Thus, different from existing deep learning techniques working on source code  [24] ,  [36] ,  [66] ,  [68] , PatchNet constructs separate embedding vectors representing the removed code and the added code in each hunk of each affected file in the given patch. The information about a file's hunks are then concatenated to build an embedding vector for the affected file. In turn, the embedding vectors of all the affected files are used to build the representation of the entire set of code changes in the given patch. PatchNet has already attracted some industry attention. Inspired by the work of Tian et al. and by our work on Patch-Net, the Linux kernel stable maintainer Sasha Levin has adopted a machine-learning based approach for identifying patches for stable kernels, which we use as a baseline for our evaluation (Section 4.
Context The Linux kernel, developed by Linus Torvalds in 1991, is a free and open-source, monolithic, and Unix-like operating system kernel  [47]. It has been deployed on both traditional computer systems, i. e. , personal computers and servers, and on many embedded devices such as routers, wireless access points, smart TVs, etc. Many devices, i. e. , tablet computers, smartphones, smartwatches, etc. that have the Android operating system also use the Linux kernel. The Linux kernel includes a two tiered release model comprising a mainline version and a set of stable versions. The mainline version, often released every two to three months, is the version where all new features are introduced. After the mainline version is released, we consider it as "stable". Any bug fixing patches for a stable version are backported from the mainline version.
The main points are as follows: • It cannot be bigger than 100 lines. • It must fix a problem that causes a build error, an oops, a hang, data corruption, a real security issue, or some "oh, thats not good" issue. These criteria may be simple, but are open to interpretation. For example, even the criterion about patch size, which seems unambiguous, is only satisfied by 93% of the patches found in the stable versions based on Linux v3. 0 to v4. 13, as of September 2017.
Potential Benefits of Automatically Identifying Stable Patches To understand the potential benefit of automatically identifying stable patches, we examine the percentage of all mainline commits that are propagated to stable kernels across different kernel subsystems and the percentage of these that are annotated with the Cc: stable tag. We focus on the 12 directories for which more than 500 mainline commits were propagated to stable kernels between Linux v3. 0 (July 2011) and Linux v4. 12 (July 2017). Fig. 2  shows the percentage of all mainline commits that are propagated to stable kernels for these 12 directories. We observe that there is a large variation in these values. Comparing directories with similar purposes, 4% of arch/arm (ARM hardware support) commits are propagated, while 10% of arch/x86 (x86 hardware support) commits are propagated, and 6-8%  Fig.
Dataset We take our data from the patches that have been committed to mainline Linux kernel 2 v3. 0, released in July 2011, through v4. 12, released in July 2017. We additionally collect information from the stable kernels 3 that had been released as of October 2017 building on Linux kernels v3. 0 through v4. 13. We consider a mainline commit to be stable if it is duplicated in at least one stable version. To increase the set of commits that can be used for training, we furthermore include in the training set of stable patches other Linux kernel commits that are expected by convention to be bugfixing patches. Indeed, a Linux kernel release is created by first collecting a set of commits for the coming release into a preliminary release called a "release candidate", named rc1, that may include new features and bug fixes.
This is followed by a succession of further release candidates, named rc2 onwards, that should include only bug fixes. We thus also include the commits added for release candidates rc2 onwards in our set of stable patches. We refer to patches that are propagated to stable kernels or are found in later release candidates as stable patches and patches that are not propagated to stable kernels or found in later release candidates as non-stable patches. To avoid biasing the learning process towards either stable or non stable patches, we construct our training datasets such that 2. git://git. kernel. org/pub/scm/linux/kernel/git/torvalds/linux. git 3. git://git. kernel. org/pub/scm/linux/kernel/git/stable/linux. git the number of patches in each category is roughly balanced. While this situation does not reflect the number of stable and non-stable patches that confront a stable kernel maintainer each day, it allows effective training and interpretation of the experimental results.
Identifying Stable Patches The main challenge in constructing the datasets is to determine which mainline patches have been propagated to stable kernels. Indeed, there is no required link information connecting the two. Many stable patches explicitly mention the corresponding mainline commit in the commit message, which we refer to as a back link. For others, we rely on the author name and the subject line. Subject lines typically contain information about both the change made and the name of the file or directory in which the change is made, and should be unique. We first collect from the patches in the stable kernels a list of back links and a list of pairs of author name and subject line. A commit from the mainline whose commit id is mentioned in a back link or whose author name and subject line are the same as one found in a patch to a stable kernel is considered to be a stable patch.
Collecting the Dataset We collect our dataset from the mainline Linux kernel. In order to focus on patches that are challenging for stable maintainers to classify, we drop in advance all patches that do not meet the stable-kernel size guidelines,  4  i. e. , those that exceed 100 code lines, including both changed lines and context as reported by diff. We subsequently keep all identified stable patches for our dataset and select an equal number of non-stable patches. Whenever possible, we select non-stable patches that have a similar number of changed lines as the stable patches, again to create a dataset that reflects the cases that cannot be excluded by size alone and thus are challenging for stable kernel maintainers. These patches are then subject to a preprocessing step that is detailed in the next section.
We do not use the dataset studied by Tian et al. [63] , because it is seven years old and unclean, including labeling as bug-fixing patches the results of tools that may report coding style issues or faults whose impact is not visible in practice. Our dataset comes from Linux kernel mainline versions 3. 0 (July 2011) through 4. 12 (July 2017). There were 424,380 commits during that period. We consider only those commits that are not merge commits, that modify a file as opposed to only adding or removing files, and that affect at least one. c or. h file. This leaves 346,570 commits (82%). Of these 346,570 commits, 79,319 (23%) are not considered because they contain more than 100 changed lines, leaving 267,251 commits. Of these,to have a balanced training dataset, we pick the 42,408 stable patches for which the preprocessing step is successful (see below,) and 39,995 nonstable patches, i.
1a ) and in error handling code (code to clean up after an error has occurred, e. g. , lines 22 and 23 in Fig. 1a ) from changes in other code, which we refer to as normal code. Error handling code is considered to be any code that is in a conditional with only one branch, where the conditional ends in a return with an argument other than 0 (0 is typically the success indicator) or a goto, as well as any code following a label that ends in a return with an argument other than 0 or a goto. Error checking code is considered to be the header of a conditional that matches the former pattern. These criteria are not completely reliable, as such code can sometimes represent the success case rather than a failure case, but they are typically followed and are actively promoted by Linux kernel developers.
The set of code features considered was manually selected. In Tian et al. 's work, stable kernels were considered as a source of bugfixing patches in the training and testing data. • LS-CNN: Huo et al. [24]  combined LSTM  [23]  and CNN  [39]  to localize potential buggy source files based on bug report information. They used CNN to learn a representation of the bug report and a combination of LSTM and CNN to learn the structure of the code. To assess the ability of LS-CNN to classify patches as stable, for a given patch, we give the commit message and the code changes (i. e. , the result of concatenating the lines changed in the various files and hunks) as input to LS-CNN in place of the bug report and the potential buggy source file, respectively. To make a fair comparison, the CNN used to learn the representation of the commit message in LS-CNN has the same architecture (i.
e. , number of convolutional layer, filter size, activation function, etc. ) as the CNN used to learn the representation of the commit message in PatchNet. • Feed-forward fully connected neural network (F-NN): Inspired by PatchNet and the work of Tian et al. on LPU+SVM, a Linux stable kernel maintainer, Sasha Levin, has developed an approach to identifying stable patches  [43]  based on a feed-forward fully connected neural network  [4] ,  [15]  and a set of manually selected features, including frequent commit message words, author names, and some code metrics. Levin actively uses this approach in his work on the Linux kernel. For LPU-SVM and LS-CNN, we used the same parameters and settings as described in the respective papers. For F-NN, we asked Levin to train the tool on our training data and test it with our testing data.
Research Questions and Results Our study seeks to answer several research questions (RQs): RQ1: Do the properties of stable and non-stable patches change over time? A common strategy for evaluating machine learning algorithms is n-fold crossvalidation  [33] , in which a dataset is randomly distributed among n equal-sized buckets, each of which is considered as test data for a model trained on the remaining n -1 buckets. When data elements become available over time, as is the case of Linux kernel patches, this strategy results in testing a model on data that predates some of the data on which the model was trained. Respecting the order of patch submission, however, would limit the amount of testing that can be done, given the fairly small number of stable patches available. To address this issue, we first assess whether training on future data helps or harms the accuracy of PatchNet.
We first sort the patches collected in Section 4. 1 from earliest to latest based on the date when the patch author submitted the patch to maintainers. Then, we divide the dataset into five mutually exclusive sets by date. Note that the resulting five sets are not perfectly balanced, but they come close, with stable patches making up 45% to 55% of each set. Then, we repeat the following process five times: take one set as a testing set and use the remaining four sets for training. Testing on the first set shows the impact of training only on future data. Testing on the fifth set shows the impact of training only on past data. The other testing sets use models trained on a mixture of past and future data. Table  1  shows the results of PatchNet on the different test sets. The standard deviations are quite small (i.
e. , at most 0. 013), hence there is no difference between training on past or future data. Our dataset starts with Linux v3. 0, which was released in 2011, twenty years after the start of work on the Linux kernel. The lack of impact due to training on past or future data suggests that in such a mature code base the properties that make a patch relevant for stable kernels are fairly constant over time. This property is indeed beneficial, because it means that our approach can be used to identify stable commits that have been missed in older versions. In the subsequent research questions, we thus retain the same five test and training sets. RQ2: How effective is PatchNet compared to other state-of-the-art stable patch identification models? To answer this RQ, we use the five test sets of the dataset described in RQ1.
As compared to Keyword and LS-CNN, there are almost 7,000 patches that are only recognized by PatchNet, while this is the case for fewer than 2,000 patches for LS-CNN, showing the value of an approach that takes the properties of code changes into account. The bottom diagram then compares PatchNet to the two approaches, LPU+SVM and F-NN, in which the code features are handcrafted. While all three approaches correctly recognize over 27K patches as stable, there are again 3x more patches that only PatchNet correctly detects as stable than there are that only each of the other two approaches recognizes as stable. Examples of PatchNet true positives not found by the other baselines include 5567e989198b 6 and 2e31b4cb895a. Examples of PatchNet false negatives found by at least one other baseline include 03f219041fdb and 56199016e867.
All of the above measures of precision and recall assume that the set of patches found in the tested Linux kernel versions is the correct one. Our motivation, however, is that bug fixing patches that should be propagated to the Linux kernel stable versions are being overlooked by the existing manual labeling process. Showing that PatchNet improves on the existing manual process requires collecting a dataset of patches that have not been propagated to stable kernels, but should have been. Collecting such a dataset, however, requires substantial Linux kernel expertise, which it is not feasible to harness at a large scale. We have nevertheless been able to carry out two experiments in this direction. First, we randomly selected 200 patches predicted as stable patches by PatchNet, but that were not marked as stable in our dataset.
We sent the 200 patches to Sasha Levin (a Linux stable-kernel maintainer and the developer of F-NN) to label. Among the 200 patches, Levin labeled 61 patches (i. e. , 30. 5%) as stable, highlighting that our approach can find many additional stable patches that were not identified by the existing manual process. Note that these patches predated Levin's used of F-NN on the Linux kernel. Second, we looked at commits that have no Cc stable tag that Sasha Levin selected with the aid of F-NN for the Linux 4. 14 stable tree. These commits postdate all of the commits in our dataset. There are over 1,800 of them, showing the false negatives in the existing manual process and the need for automated support. PatchNet detects 91% of them as stable. The relationship between the results of F-NN and PatchNet is similar to that shown in Fig.
11  for patches in our original dataset and confirms that PatchNet can find stable patches that were not identified by the existing manual process. RQ3: Does PatchNet benefit from considering both the commit message and the code changes, and do function names help identify stable patches? To answer this RQ, we conduct an ablation test  [34] ,  [46]  by ignoring the commit message, the code changes, or the function names in the code changes in a given patch one-at-a-time and evaluating the performance. We create three variants of PatchNet: PatchNet-C, PatchNet-M, and PatchNet-NN. PatchNet-C uses only code change information while PatchNet-M uses only commit message information. PatchNet-NN uses both code change and commit message information, but ignores the function names in the code changes.