dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction In the standard regression model y i = ϕ(x i ) + ε i , i = 1,. , n, where the data (x i , y i ) are observed, a number of softwares specializing in nonparametric and semi-parametric estimation have recently appeared. Prominent among these routines is the popular np package of  Hayfield and Racine (2008) , which allows R (R Core Team, 2015) users to conduct, for instance, nonparametric mean and quantile regression. In the non-standard boundary regression model, in contrast to classical theory, the regression errors (ε i ) are not assumed to be centred, but to have a one-sided support (-∞, 0], and the regression function ϕ describes some boundary curve. The present npbr package is a collection of functions that perform a variety of nonparametric estimation techniques of the frontier function ϕ in the statistical software environment R.
Other contributions to the econometric literature on frontier analysis by  Parmeter and Racine (2013)  can be found at http://socserv. mcmaster. ca/racinej/Gallery/Home. html. The package npbr is actually the first free specialized software for the statistical literature on nonparametric frontier analysis. The routines included in npbr are user friendly and highly flexible in terms of estimation specifications. They allow the user to filter out noise in edge data by making use of both empirical and smooth fits as well as (un)constrained estimates under separate and simultaneous multiple shape constraints. They also provide smoothing parameter selection for the innovative methods based on local linear techniques, polynomial splines, extreme values and kernel smoothing, though the proposed selection procedures can be computationally demanding.
The package npbr is available from the Comprehensive R Archive Network (CRAN) at http://CRAN. R-project. org/package=npbr. Section 2 presents briefly five unrelated motivating data examples concerned with annual sport records, the master curve prediction in the reliability programs of nuclear reactors and with the optimal cost/production assessment in applied econometrics. Section 3 describes in detail the imple-mented functions of the package and provides practical guidelines to effect the necessary computations. In Section 4, we provide some computational tips that facilitate Monte-Carlo comparisons among frontier estimation methods in a similar way to the simulation studies undertaken by  Daouia et al. (2015)  and  Noh (2014).
Encountered applied settings and data examples In this section, we illustrate the use of the npbr package via five different empirical applications taken from the recent literature. Each dataset is chosen to highlight the specifics of a class of estimation methods: The dataset records is concerned with the yearly best men's outdoor 1,500-metre run times starting from 1966. These annual records, depicted in Figure  1  (a), display an interesting behaviour. Following  Jirak, Meister, and Reiss (2014) , the lower boundary can be interpreted as the best possible time for a given year. This boundary steadily decreases from 1970 until around the year 2000, followed by a sudden increase. This event leaves room for speculations given that, until the year 2000, it had been very difficult to distinguish between the biological and synthetical EPO.
Here, the boundary is not believed to be shape constrained and can be estimated by the polynomial, local linear, spline or kernel smoothing methods described in Sections 3. 1 and 3. 3. The dataset nuclear from the US Electric Power Research Institute (EPRI) consists of 254 toughness results obtained from non-irradiated representative steels. For each steel i, fracture toughness y i and temperature x i were measured. The scatterplot is given in Figure  1  (b). The objective is to estimate the lower and upper limits of fracture toughness for the reactor pressure vessel materials as a function of the temperature. Given that the nuclear reactors' data are measured accurately, it is natural and more realistic for practitioners to rely on data envelopment estimation techniques that we regroup in Sections 3.
1-3. 3. Here, the lower support boundary is believed to be both increasing and convex, while the upper extremity is only known to be monotone nondecreasing (see  Daouia et al. , 2014 Daouia et al. , , 2015)). The dataset air is concerned with the assessment of the efficiency of 37 European Air Controllers. The performance of each controller can be measured by its "distance" from the upper support boundary, or Function Description Reference dea_est DEA, FDH  Farrell (1957) ,  Deprins et al. (1984) , and linearized FDH  Hall and Park (2002) ,  Jeong and Simar (2006)   equivalently, the set of the most efficient controllers. This dataset is taken from  Mouchart and Simar (2002). The scatterplot of the con-trollers in the year 2000 is given in Figure  1  (c), where their activity is described by one input (an aggregate factor of different kinds of labor) and one output (an aggregate factor of the activity produced, based on the number of controlled air movements, the number of controlled flight hours, etc.
). Given the very small sample size and the sparsity in data, only the class of polynomials, piecewise polynomials and spline approximations seems to provide satisfactory fits in this applied setting. This class includes the families of empirical and smooth estimation methods described in Section 3. 1. Note also that the efficient frontier here is monotone and can be assumed to be in addition concave (see  Daouia et al. , 2008 Daouia et al. , , 2015)). The dataset post about the cost of the delivery activity of the postal services in France was first analyzed by  Cazals, Florens, and Simar (2002)  and then by  Aragon, Daouia, and Thomas-Agnan (2005)  and  Daouia, Florens, and Simar (2010)  among others. There are 4,000 post offices observed in 1994. For each post office i, the input x i is the labor cost measured by the quantity of labor, which accounts for more than 80% of the total cost of the delivery activity.
The output y i is defined as the volume of delivered mail (in number of objects). As can be seen from the scatterplot in Figure  1  (d), some observations look so isolated in the output direction that they seem hardly related to the other observations. As a matter of fact, this dataset is known to contain outliers and it would then look awkward for practitioners to rely on estimation techniques based on data envelopment ideas (see  Daouia and Gijbels, 2011). This motivated the quest for robust frontier estimation methods in Section 3. 4. It should be clear that only these methods allow to construct valid asymptotic confidence intervals for the unknown support boundary. The dataset green consists of 123 American electric utility companies. As in the set-up of  Gijbels, Mammen, Park, and Simar (1999) , we used the measurements of the variables y i = log(q i ) and x i = log(c i ), where q i is the production output of the company i and c i is the total cost involved in the production.
A detailed description and analysis of these data can be found in  Christensen and Greene (1976). The scatterplot is given in Figure  1  (e). Here, the assumption of both monotonicity and concavity constraints is well accepted and any restricted data envelopment technique such as, for instance, kernel smoothing in Section 3. 3 can be applied. Also, in absence of information on whether these data are recorded accurately, one may favor robust frontier estimation. We caution the user that the robust methods based on extreme-value ideas may require a large sample size of the order of thousands to achieve very good fits and confidence intervals. To help users navigate the methods in the npbr package, we describe in Table  2  the type of estimation and shape constraints allowed by each method. For our illustration purposes, each of the five datasets contains only two variables: one input and one output.
R> require("npbr") R> data("records", "nuclear", "air", "post", "green") The following code will generate Figure  1. R> plot(result ~year, data = records, xlab = "year", ylab = "1500m record") R> plot(ytab ~xtab, data = nuclear, xlab = "temp. of the reactor vessel", + ylab = "fracture toughness") R> plot(ytab ~xtab, data = air, xlab = "input", ylab = "output") R> plot(yprod ~xinput, data = post, xlab = "quantity of labor", + ylab = "volume of delivered mail") R> plot(log(OUTPUT)~log(COST), data=green) q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 1970 1980 1990 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q -200 - temp.
Practical guidelines By way of example, we consider the records, air and nuclear datasets. To determine the optimal polynomial degrees via the AIC criterion, we employ the commands R> (p. aic. records <-poly_degree(records$year, 1/records$result, + prange = 0:12, type = "AIC")) [1] 9 R> (p. aic. air <-poly_degree(air$xtab, air$ytab, type = "AIC")) [1] 3 R> (p. aic. nuc <-poly_degree(nuclear$xtab, nuclear$ytab, type = "AIC")) [1] 2 We find the same degrees by applying the BIC criterion. The R specifications for the corresponding polynomial boundaries to be estimated are given by The following code can be used to construct the plots of the resulting estimators appearing in Figure  3. R> R> plot(x. records, 1/y. poly. records, type = "l", col = "magenta") R> points(result ~year, data = records) R> plot(x.
Concavity constraint When the monotone boundary ϕ(x) is also believed to be concave (option method = "mc"), its constrained fit is defined as φ n (x) = π(x) α , where α ∈ R N minimizes the same objective function as α subject to the same envelopment and monotonicity constraints and the additional concavity constraints π (t * j ) α ≤ 0, j = 1,. , k n , where π is the constant second derivative of π on each inter-knot interval and t * j is the midpoint of (t j-1 , t j ]. Regarding the choice of knots, the same scheme as for φn is applied by replacing the FDH points (X 1 , Y 1 ),. , (X N , Y N ) with the DEA points (X * 1 , Y * 1 ),. , (X * M , Y * M ), that is, the observations (x i , y i = ϕ dea (x i )) lying on the piecewise linear DEA frontier (see the function dea_est). Alternatively, the strategy of just using all the DEA points as knots is also working quite well for datasets of modest size as shown in  Daouia et al.
green, kn = kn. bic. green. mc, method = "mc", all. dea = TRUE) The resulting unrestricted and two constrained estimates of the econometric frontiers (i. e. the sets of the most efficient companies and controllers) are graphed in Figure  4  for each dataset. The following code will generate Figure  4. R> plot(y. quad. air. u ~x. air, lty = 1, col = "green", type = "l", + xlab = "input", ylab = "output") R> lines(x. air, y. quad. air. m, lty = 2, col = "cyan") R> lines(x. air, y. quad. air. mc, lty = 3, col = "magenta") R> points(ytab~xtab, data=air) R> legend("topleft", col = c("green","cyan","magenta"), lty = c(1,2,3), + lwd = 4, legend = c("unconstrained", "monotone", "monotone + concave")) R> plot(y. quad. green. u ~x. green, lty = 1, col = "green", type = "l", + xlab = "log(COST)", ylab = "log(OUTPUT)") R> lines(x.
To compute the corresponding (un)constrained cubic spline frontiers, we employ the following commands R> y. cub. air. u <-cub_spline_est(air$xtab, air$ytab, x. air, + kn = kn. bic. air. u, method = "u") R> y. cub. green. u <-cub_spline_est(log(green$COST), log(green$OUTPUT), + x. green, kn = kn. bic. green. u, method = "u") R> y. cub. air. m <-cub_spline_est(air$xtab, air$ytab, x. air, + kn = kn. bic. air. m, method = "m") R> y. cub. green. m <-cub_spline_est(log(green$COST), log(green$OUTPUT), + x. green, kn = kn. bic. green. m, method = "m") R> y. cub. air. mc <-cub_spline_est(air$xtab, air$ytab, x. air, + kn = kn. bic. air. mc, method = "mc") R> y. cub. green. mc <-cub_spline_est(log(green$COST), log(green$OUTPUT), + x. green, kn = kn. bic. green. mc, method = "mc") The resulting unconstrained and concave frontier estimates are graphed in Figure  5  for each dataset.
The following code will generate Figure  5. R> plot(y. cub. air. u ~x. air, type = "l", col = "green", + xlab = "input", ylab = "output") R> lines(x. air, y. cub. air. m, lty = 2, col = "cyan") R> lines(x. air, y. cub. air. mc, lty = 3, col = "magenta") R> points(ytab ~xtab, data = air) R> legend("topleft", col=c("green", "cyan","magenta"), lty = c(1,2,3), + legend=c("unconstrained", "monotone", "monotone+concave")) R> plot(y. cub. green. u ~x. green, type = "l", col = "green", + xlab = "log(COST)", ylab = "log(OUTPUT)") R> lines(x. green, y. cub. green. m, lty = 2, col = "cyan") R> lines(x. green, y. cub. green. mc, lty = 3, col = "magenta") R> points(log(OUTPUT) ~log(COST), data = green) R> legend("topleft", col = c("green","cyan","magenta"), lty = c(1,2,3), + legend = c("unconstrained", "monotone", "monotone+concave")) 0 2 4 6 2 4 6 8 10 input output q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q unconstrained monotone monotone+concave -2 0 2 4 2 4 6 8 10 log(COST) log(OUTPUT) q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q unconstrained monotone monotone+concave.
12 Note that the computational burden here is very demanding, so be forewarned. Now to evaluate φn,LL and/or φn,LL , we employ the commands R> y. records. u<-loc_est(records$year, 1/records$result, x. records, + h = h. records. u, method = "u") R> y. air. u<-loc_est(air$xtab, air$ytab, x. air, h = h. air. u, method = "u") R> y. air. m<-loc_est(air$xtab, air$ytab, x. air, h = h. air. m, method = "m") R> y. nucl. u<-loc_est(nuclear$xtab, nuclear$ytab, x. nucl, + h = h. nucl. u, method = "u") R> y. nucl. m<-loc_est(nuclear$xtab, nuclear$ytab, x. nucl, + h = h. nucl. m, method = "m") Figure  6  superimposes the obtained estimates for each dataset. For the particular datasets air and nuclear, the resulting unconstrained and impoved estimates are very similar. The following code will generate Figure  6.
, n; (envelopment constraints) φ(1) (x|p) = n i=1 p i A (1) i (x)y i ≥ 0, x ∈ M; (monotonocity constraints) φ(2) (x|p) = n i=1 p i A (2) i (x)y i ≤ 0, x ∈ C, (concavity constraints) where φ(s) (x|p) = n i=1 p i A (s) i (x)y i is the s-th derivative of φ(x|p), with M and C being the collections of points where monotonicity and concavity are imposed, respectively. In our implementation of the estimator, we simply take the entire dataset {(x i , y i ), i = 1,. , n} to be M and C and, in case of small samples, we augment the sample points by an equispaced grid of length 201 over the observed support [min i x i , max i x i ] of X. For the weight A i (x), we use the Nadaraya-Watson weights.
Practical guidelines For our illustration purposes using the large dataset post, we consider the following three possible scenarios: either ρ x is known (typically equal to 2 if the assumption of a jump at the frontier is reasonable), or ρ x is unknown and estimated by the moment estimator ρx , or ρ x is unknown independent of x and estimated by the (trimmed) mean of ρx. First, we select the points at which we want to evaluate the frontier estimator. In the case where the extreme-value index ρ x is known and equal to 2, we set R> rho <-2 Then, we determine the sample fraction k = k n (x) in φmomt (x). R> best_kn. 1 <-kopt_momt_pick(post$xinput, post$yprod, x. post, rho = rho) When ρ x is unknown and dependent of x, its estimate ρx is computed via the command R> rho_momt <-rho_momt_pick(post$xinput, post$yprod, x.
post, + method = "moment") To determine the sample fraction k in the two-stage estimator φmomt (x), we use R> best_kn. 2 <-kopt_momt_pick(post$xinput, post$yprod, x. post, + rho = rho_momt) Here, for the post data, we used the default value wind. coef= 0. 1 in the function kopt_momt_pick to avoid numerical instabilities. When employing another large dataset, the user should tune this coefficient until the resulting pointwise frontier estimates and confidence intervals exhibit stable curves (see the function kopt_momt_pick for details). When ρ x is unknown but independent of x, which is a more realistic setting in practice, a robust estimation strategy is by using the (trimmed) mean over the moment estimates ρx. R> rho_trimmean <-mean(rho_momt, trim = 0. 00) R> best_kn. The following code can be used to construct the resulting moment frontier plots graphed in Figure  9.
Practical guidelines For our illustration purposes, we used again the large dataset post and considered the following three scenarios: either ρ x is known (typically equal to 2 if the joint density has sudden jumps at the frontier), ρ x is unknown and estimated by the Pickands estimator ρx , or ρ x is unknown independent of x and estimated by the (trimmed) mean of ρx. When ρ x is known and equal to 2, we set.
Acknowledgments The first author acknowledges financial support by the  Toulouse School of Economics Individual Research Fund  ( IRF/ Daouia-20125 ) and by the  Seventh Framework Programme of the European Union  ( IEF/273584/EMBAFproject ). The research of the third author was supoorted by  Basic Science Research Program  through the  National Research Foundation of Korea (NRF)  funded by the  Ministry of Education  ( 2014R1A1A2059875 ).
Component repositories play a key role in the open software ecosystem. Managing the evolution of these repositories is a challenging task, and maintainers are confronted with a number of complex issues that need automatic tools to be adressed properly. In this paper, we present an overview of 10 years of research in this field and the process leading to the adoption of our tools in a FOSS community. We focus on the Debian distribution and in particular we look at the issues arising during the distribution lifecycle: ensuring buildability of source packages, detecting packages that cannot be installed and bootstrapping the distribution on a new architecture. We present three tools, distcheck, buildcheck and botch, that we believe of general interest for other open source component repositories.
Introduction In the last two decades, component repositories have played an important role in many areas, from software distributions to application development. All major Free and Open Source Software (FOSS) distributions are organized around large repositories of software components. Debian, one of the largest coordinated software collections in history  [12] , contains in its development branch more than 44'000 binary packages 1 generated from over 21'000 source packages; the Central Maven repository has a collection of 100'000 Java libraries; the Drupal web framework counts over 16'000 modules. In Debian, components are developed independently by different communities and assembled in the main repository, giving raise to a small world dependency graph  [15]. Apart from intrinsic coordination problems associated to this distributed development model, the number of dependencies in Debian distributions poses new challenges for automation and quality assurance.
During the last 10 years we have participated in the development and adoption of automatic tools for testing, integration and tracking of all components and aspects of a repository, in particular in the framework of the European project Mancoosi  [1]. It is well known that achieving real world adoption of tools developed in academia and proposed by researchers is a painful and difficult process that only rarely succeeds  [19]. After years of work, we managed to get almost all of our tools adopted in the Debian project. We participated in extensive work performed by a team that spent 10 years of research in quality assurance, and package management, an area for which a comprehensive short survey is available elsewhere  [8]. During this time, we had different collaborations with many other communities such as the Eclipse  [16, 17]  and the OCaml  [2]  with different degrees of success.
In this article, we sum up and share the lessons we have learned in collaboration specifically with the Debian community, because of the direct involvement of a few members of our team, and because of the open and community driven bazaar-style development model. We truly believe that FOSS distribution and software collections alike can benefit from our experience and researcher should invest time and energy to work with developers in a proactive way and foster integration of modern and automatic QA (quality assurance) tools. The rest of the paper is organised as follows: After a brief introduction, we present distcheck and buildcheck , the main tools developed by our team. Then we will discuss two examples in which our tools play an important role. The first one related to the distribution life cycle (from development to testing, to the stable release).
Packages in the Debian Distribution Despite different terminologies, and a wide variety of concrete formats, software repositories use similar metadata to identify the components, their versions and their interdependencies. In general, packages have both dependencies, expressing what must be satisfied in order to allow for installation of the package, and conflicts that state which other packages must not be installed at the same time. As shown in Fig. 1 , while conflicts are simply given by a list of offending packages, dependencies may be expressed using logical conjunction (written ',') and disjunctions ('|'). Furthermore, packages mentioned in inter-package relations may be qualified by constraints on package versions. Debian packages and come in two flavours: binary packages contain the files to be installed on the end user machine, and source packages that contain all of files needed to build these binary packages.
Debian package meta-data describe a broad set of inter-package relationships: virtual-packages, dependencies, multi-architecture annotations, and many more, allow the Debian project to automatize tasks such as binary package recompilations, package life Package: ant Version: 1. 9. 7-2~bpo8+1 Installed-Size: 2197 Architecture: all Depends: default-jre-headless | java5-runtime-headless | java6-runtime-headless Recommends: ant-optional Suggests: ant-doc, ant-gcj, default-jdk | java-compiler | java-sdk Conflicts: libant1. 6-java Breaks: ant-doc (<= 1. 6. 5-1) Description: Java based build tool like make.
The Installability Problem Finding a way to satisfy all the dependencies of a given package only using the components available in a repository, also known as the installability problem, is the key task for all component based repositories: all package managers need to tackle it, be it for Eclipse plugins, Drupal modules, or Debian packages. And yet, it was not until 2006 that it was shown that this problem is NPcomplete for the Debian distribution  [7] , and later on for a broad range of component repositories  [4]. This result came as a kind of a surprise in the different engineering communities, that were using on a daily basis ad-hoc tools which were fast, but under closer scrutiny turned out to be incomplete  [7]. Luckily, real world instances proved to be tractable, and it was possible to design and implement dependency solvers based on sound scientific basis, that could significantly outperform all the pre-existing tools: Jerôme Vouillon's early prototypes, debcheck and rpmcheck, originally developed in 2006, paved the way to modern dependency checking, and are nowadays at the core of the tools we describe in the rest of this paper.