dataesr/scientific-paragraphs-categorization huggingface dataset:
e. dependency or conflict, in a CUDF document. Formally, (p nc , v nc ) is non-constraining in the context of a given CUDF document iff: ∀(p, v, q, k, θ) ∈ CUDF document • p nc ≠ q Complementarily, a constraining package (p c , v c ) is a package whose identifier does appear as dependee in some constraint, i. e. ∃(p, v, q, k, θ) ∈ CUDF document • p nc = q Considering these definitions, it is possible to define the following metamorphic relations between the valid configurations of neighbour CUDF documents. #Ψ(D ′ ) = 2 ⋅ #Ψ(D) + 1 ∧ Ψ(D) ⊆ Ψ(D ′ ) ∧ { (p nc , v nc ) } ∈ Ψ(D ′ ) ∧ ∀c ∈ Ψ(D) • c ∪ { (p nc , v nc ) } ∈ Ψ(D ′ ) (MR 6 ) Notice that constraining packages are excluded from this rule since they would not allow to derive the set of configurations of the new CUDF documents from the previous set §.
In the example, document D' is generated by adding a new package C to document D. The set of configurations of D' is then easily calculated by applying the metamorphic relation MR 6. Model transformations can be applied either randomly or using deterministic heuristics, although in certain cases, the order in which metamorphic relations are applied matters. In CUDF, for instance, non-constraining packages should be added (MR 6 ) before adding constraints depending on them (MR 7 , MR 8 and MR 9 ). Similarly, relations MR 12 and MR 13 (adding disjunctions with existing variables), cannot be applied until at least one variable is added to the CNF formula. Notice, however, that this does not affect the diversity of the VMs that can be generated. The generation process can be stopped after a given number of transformations, or as soon as a VM and its corresponding set of valid configurations is generated and some desired properties are achieved, e.
EVALUATION In this section, we evaluate whether our metamorphic testing approach is able to automate the generation of test cases in multiple variability analysis domains (RQ1). Also, and more importantly, we explore whether the generated test cases are actually effective in detecting real bugs in variability analysis tools (RQ2). For the evaluation, we developed three test data generators based on the metamorphic relations defined in Section 3. Then, we evaluated their ability to automatically detect faults within a number of analysis tools in the tree domains under study: FMs, CUDF documents and CNF formulas. The results are reported in the following sections. The experiments were performed by two teams in different execution environments for compatibility with the tools under test. The specific execution settings are described in a separated technical report due to space constraints  [53].
FLAME is a Prologbased reasoner developed by some of the authors as a reference implementation to validate a formal specification for the analysis of FMs  [25]. FaMa was tested with its default configuration. SPLAR is actually composed of two reasoners using SAT-based and BDD-based analysis, which were both tested. Tests with FLAME were performed as part of a previous contribution and reproduced for this evaluation  [25]. In total, we tested 19 operations in FaMa, 18 en FLAME and 9 in SPLAR. A detailed description of the operations tested on each reasoner is provided in  [53]. The name and formal semantics of the analysis operations mentioned in this article are based on the work presented in  [25]. The evaluation was performed in two steps. First, we used our metamorphic test data generator to generate 1,000 random FMs and their corresponding set of products.
Detecting faults in CUDF reasoners For this experiment, we developed a test data generator for the analysis of CUDF documents based on the metamorphic relations defined in Section 3. 2. The tool generates CUDF documents of a predefined size and their set of valid configurations. In this experiment, we evaluated the ability of the test data generator to detect faults in three CUDF reasoners in which two faults were found. Experimental setup. We evaluated the effectiveness of our test data generator in detecting faults in three CUDF reasoners: p2cudf 1. 14, aspcudf 1. 7 and cudf-check 0. 6. 2-1. p2cudf  [2, 49]  is a Java tool that reuses the Eclipse dependency management technology (p2) to solve upgradeability problems in CUDF. It internally relies on the pseudo-Boolean solver Sat4j  [38]. Aspcudf  [5, 31]  uses several C++ tools for Answer Set Programming (ASP), a declarative language.
Cudf-check is a command line CUDF reasoner provided as part of the Debian cudf-tools package [19]. This tool is mainly used to check the validity of CUDF documents and their configurations, i. e. it does not support optimization. In this experiment, we tested two different analysis operations. In the cudf-checker tool, we tested the operation that checks whether a given configuration is valid with respect to a given CUDF document and a given request. In p2cudf and aspcudf, we tested the upgradeability problem using the paranoid optimization criterion  [2, 43]. As described in Section 4, this criterion searches for a configuration that fulfils the user request and minimizes the number of changes in the system. We selected this optimization operation because it is used in the annual Mancoosi competition and it is supported by most CUDF reasoners.
The evaluation was performed in two steps. First, we used our metamorphic test data generator to generate 1,000 random CUDF documents without requests and their corresponding set of valid configurations. We parametrically controlled the generation assuring that the documents had a fair proportion of all types of elements, i. e. dependencies, conflicts, version constraints, etc. We refer the reader to an external technical report for the specific parameters and values used for the generation  [53]. The generated documents had between 5 and 20 packages and 50% and 120% of constraints, i. e. depends and conflicts. Also, version constraints (e. g. A >= 2) were added with certain probability. Each document represented up to 197,400 different configurations. Once a CUDF document and its configurations were generated, the packages of a random configuration were marked as installed (installed: true) to simulate the current status of the system.
Detecting faults in SAT reasoners For this experiment, we developed a test data generator for the analysis of Boolean formulas based on the metamorphic relations defined in Section 3. 3. The tool generates Boolean formulas in CNF form and the set of solutions of the formula. For its evaluation, we automatically tested nine SAT solvers in which 5 bugs were revealed. Experimental setup. We automatically tested nine SAT reasoners written in different languages. The binaries of unversioned reasoners were taken from the SAT competition in which they participated, indicated in parenthesis, namely: Sat4j 2. 3. 1  [38] , Lingeling ala-b02  [40] , Minisat 2. 2  [27] , Clasp 2. 1. 3 [32] , Picosat 535 [9], Rsat 2. 0  [50] , March_ks (2007)  [33] , March_rw (2011)  [33] , and Kcnfs 1. 2  [24]. In a related work  [11] , Brummayer et al.
On each test, we checked that the solutions returned by the reasoner matched the solutions generated by our test data generator. We ran 10,000 test cases on each SAT reasoner using this procedure. Since each test case exercises the SAT solver once per found solution, and a last time to check that no more solution exists, each reasoner was expected to answer SAT 1,817,142 times, the total number of solutions in the suite, and UNSAT 10,000 times in total. The generation of the test data took 3 hours. The execution time ranged between 9 minutes in Sat4j and almost 10 days in Kcnfs (due to timeouts, see the analysis results in the next paragraph). Notice that Sat4j does support solution enumeration natively, so it did not require to read each time a new problem with a new blocking clause. Thus, no file system I/O operations were performed in that case and, more importantly, the solver could take advantage of an incremental setting.
Each relation should relate the set of configurations represented by a VM before (source) and after (follow-up) adding the variability constraint. Ideally, it should be defined a metamorphic relation for each variability constraint of the language to foster diversity during test data generation. This is supported by related works on the effectiveness of metamorphic relations, which suggest that good metamorphic relations should be diverse and semantically rich, i. e. rely on the semantics of the system under test  [15, 41, 44]. 4. Generate test data. Apply the metamorphic relations iteratively to generate increasingly larger and more complex VMs and their corresponding set of valid configurations. In certain domains, it may be required to apply the relations in a certain order. For instance, packages must be added to CUDF documents (MR 6 ) before adding constraints referencing them (MR 7 , MR 8 , and MR 9 ).
Among the reasoners, 4 were developed by some of the authors, meanwhile 11 of them were developed by external developers. This clearly shows the black-box nature of the work, testing analysis tools with no prior knowledge about their internal details. As a result of the tests, we detected 19 total faults in the three domains under study: analysis of FMs, CUDF documents and CNF formulas. External validity. Regarding the generalization of the conclusions, we evaluated our approach with three different variability languages, a number that could seem insufficient for the generalization of the conclusions of our study. We remark, however, that these languages are used in completely different domains, and have particularities such as hierarchical constraints in FMs, version and installation constraints in CUDF documents, or negated variables in Boolean formulas, that make them sufficiently heterogeneous.
Beside these particularities, the three variability languages include constraints with similar semantics, e. g. excludes in FMs is very similar to conflicts in CUDF. These constraints are very common in variability modelling, something that suggests that our approach could be easily applicable to other variability languages such as orthogonal variability models  [51]  or decision models  [59]. Since FMs and CUDF documents can be translated into (pseudo) Boolean formulas, it could be argued that working directly with Boolean formulas is a simpler and more generic approach. We did not adopt this approach for two reasons. First, a bidirectional translation from high-level variability models to Boolean formulas is a complex, language-specific task  [20]. Second, and more importantly, translating models to formulas, forwards and backwards, would make test data generators very complex and probably more error-prone than the analysis tool under test.
Testing CUDF reasoners The 2012 Mancoosi solver competition provided a solution checker to assess the correctness of the solutions returned by the competitor CUDF reasoners  [43] , i. e. redundant testing. Other related works have been presented in the context of package-based distributions. Vouillon and Di Cosmo  [23]  proposed a theoretical framework to detect co-installability conflicts, i. e. packages that cannot be installed together. Artho et al. [4]  presented a case study classifying the types of conflicts found in two specific distributions, Debian and Red Hat. In  [29] , some of the authors proposed using variability analysis techniques for the automated analysis of Debian repositories. Compared to them, our work contributes to detect bugs in package management tools overcoming the oracle problem rather than analysing variability in package repositories.
Metamorphic testing Kuo et al. [36]  presented an approach for the automated detection of faults in decision support systems. In particular, they focused in the so-called Multi-Criteria Group Decision Making (MCGDM), in which decision problems are modelled as a matrix with several dimensions: alternatives, criteria and experts. They also introduced eleven metamorphic relations in natural language, and evaluated their approach using artificial faults in the research tool Decider. This work has certain commonalities with our contribution since VMs could be used as decision models during software configuration. Also, as in our work, Kuo et al. used metamorphic relations to actually construct the expected output of follow-up test cases (i. e. follow-up matrices) instead of just checking the output of the tests.
To show the feasibility and generalizability of our work, we automatically tested the implementation of 22 analysis operations in 15 reasoners written in different languages in the domains of FMs, CUDF documents and CNF formulas. In total, we automatically detected 19 real bugs in 7 of the tools under test. Most faults were directly acknowledged by the tools' developers, from whom we received comments as "You hammered it right on the nail!" or "the bugs found by your tests are non trivial issues". This supports our conclusions and reinforces the potential of metamorphic testing as an automated testing technique.
ACKNOWLEDGMENTS We appreciate the help of  Dr Martin Monperrus  whose comments and suggestions helped us to improve the article substantially. We would also like to thank  Dr. Marcílio Mendonça ,  Dr. Marijn J. H. Heule  and  José A. Galindo  for confirming the bugs found in their respective tools. This work has been partially supported by the  European Commission (FEDER)  and  Spanish Government  under CICYT projects TAPAS ( TIN2012-32273 ) and  SAAS FIREWALL (IPT-2012-0890-390000 ) and the  Andalusian Government projects THEOS  ( TIC-5906 ) and COPAS ( P12-TIC-1867 ).
Incentives in the form of prizes were given at the end of each phase to the best teams/players (please see Methods and Supplementary Material S1 for more details). The data collected during the test case are used here to analyze participation patterns, in terms of activity and coverage, and any changes in perception. Our results indicate that better coverage is obtained when volunteers are assigned a specific mapping area, compared to when they are asked to select the time and location of their measurements. Additionally, when allowed to measure freely, they seem to be attracted to places with higher pollution levels. Furthermore, while at the beginning of the challenge the general perception was that pollution was higher than in reality, perceptions changed in time indicating increased knowledge of real pollution levels.
2e-16 was obtained, confirming the difference. When volunteers can freely choose where to take measurements, it appears that they primarily target more polluted areas. When the mapping area is restricted, they tend to have a more systematic approach and cover lower pollution levels as well. One may argue that pollution levels may change naturally from one day to another, so the shift we see could be do to a higher average pollution level from phase 2 to phase 3. However, comparison with reference data seem to suggest that this is not the case (supplementary material S1). Additional comparisons per location are also included in S1. The analysis of the structure and location of the collected objective data gives some insight into volunteer behavior and interests when measuring air pollution.
By observing the measurements from their sensor boxes, Volunteers learn that in general BC concentrations are lower than what they believed, and respond by changing the values of the AirPins or taking the information into account when placing new ones. Since the change is quite significant, we also believe that those volunteers with the sensor boxes spread the information about what they were measuring, so that all players changed their perception. This decrease in the pollution levels reported in the subjective data of phase 2 is a first strong indication of learning during this phase. The right side of Figure  5  shows APD distributions separately for AirAmbassadors (performing measurements) and AirGuardians (who had no direct exposure to measurements until phase 3). We analyzed just the Turin dataset because in the other cities there was no clear distinction due to Ambassadors sharing their sensor boxes.
The mathematical definition can be found in the supplementary material ( S1). The left part of Figure  5  shows, for each location, how the transformed phase 1 data (black squares) matches phase 3 distributions, and this has also been confirmed with statistical procedures described in Methods and in the supplementary material S1. This provides an indirect proof of the assumptions of our model on the effect of objective data (complete and incomplete trust). Also, we were able to measure the 'trust' in the hints for the three cities, by fitting the model to data. We obtained the lowest trust values in London and the highest ones in Turin (full results are reported in the supplementary material S1).
Discussion Volunteer participation is crucial for the success of bottom-up monitoring campaigns, however most projects concerned with air pollution monitoring concentrate only on the development of the technical tools necessary. Here, we give a different user-centric perspective, using the experience from the EveryAware project, through its large scale international challenge, APIC. The tools developed by the project are described in more detail in the supplementary material S1. During the challenge both objective and subjective data were collected, and used here to analyze participatory patterns and possible changes in behavior or perception. Objective measurements allowed for analysis of user interests during the challenge and activity patterns. A large number of measurements was obtained, however, coverage varied from location to location, with higher values when monitoring areas were restricted.
Materials and Methods The study presented here is based on data collected by volunteers during a large scale test case (AirProbe International Challenge -APIC) organized in four European cities (Antwerp, Kassel, London and Turin) in from October 2013 to November 2013. It required volunteers to measure air quality as well as provide their opinion on air pollution, using the EveryAware platform. This consists of a sensing device (Sensor Box), measuring air pollution, a mobile application (AirProbe), allowing for data visualization and upload to servers, a set of web services and websites, handling data storage and visualization and a web game developed on the XTribe platform  [4] , allowing to collect individual perceptions of pollution. In the following we provide a brief description of each of the components and of the tools used for data analysis, with further details included in the supplementary file S1.
The AirProbe mobile application AirProbe is an Android application designed to connect to the sensor box via Bluetooth, acquire sensor readings and transit them to the EveryAware servers as soon as a working connection to the Internet becomes available. In addition, the application allows users to visualize the data they collect. Specifically, they can see their tracks on a map, calculate an estimated black carbon exposure and follow sensor output in real time plots. While collecting data, users can make free annotations (tags) that will be attached to the recordings and sent to the servers.
Web platform The case study web platform  [3]  is designed for collecting, storing, retrieving, analysing and visualizing large amounts of data data from different data sources. It provides endpoints for application like the AirProbe mobile application to upload data to. These data are then processed and cleaned, with several statistics and visualizations available on a public as well as a personal level. This facilitates further analysis and deeper understanding of the data by the user. A collection of statistics pages provides overall information about the data, such as graphs showing currently active sensor boxes, the overall black carbon average per day, or the overall number of collected measurements per day. Also, information on separate sessions corresponding to different tracks (defined both by the Sensor Box and by the user) is available.
Data analysis To model the evolution between the phases of the APD distribution represented in the left part of Figure  5  (Phase 1 trans. ), we implemented a simple modeling approach rearranging the opinions depending on their distances from the hint which is defined in the supplementary material S1. The transformation introduces 4 parameters, quantifying the inertia effects in the opinions shift. To check the quality of our model and to determine the values of parameters introduced we used a Kolmogorov-Smirnov test applied to the phase 3 dataset and to the phase 1 transformed dataset. Since it is a stochastic model, we performed several applications and found a convincing result for the p val of 20%, which means that the hypothesis is consistent with observations. More details are provided in the supplementary material S1.
Participatory patterns in an international Because the controled laboratory setting is very different from outdoor conditions, the main tests were performed outdoor. The outdoor performance tests were carried out at a station from the Flemish air quality monitoring network. The station (Borgerhout, 42R801, see www. ircel. be) is situated at a traffic location along a double lane main street with an average daily traffic volume of 43,381 vehicles (42,961 cars and 420 heavy duty vehicles, data from the Traffic Centre Flanders). We used 4 sensor boxes for the outdoor performance tests from October 2012 until April 2013. By placing the sensor boxes at an official monitoring station we gained the advantage of having reference data for several pollutants (CO, NO, NO 2 , O 3 and Black Carbon (BC)) albeit at a coarser temporal resolution of 30 minutes.
Sensor box calibration Following the analysis of sensor abilities presented in the previous section, we have proceeded with calibration of the sensor boxes. Issues identified by our initial analysis included sensor sensitivity to temperature and humidity, sensor drift in time and sensitivity to other gasses. Hence, one needs to calibrate devices against a reference in order to control for this issues and obtain a measurement meaningful for the user. Supervised learning was employed to model an unknown concentration of a target pollutant from sensor array measurements (based on low-cost gas sensors, temperature and relative humidity sensors). The supervised learning model is parameterised by using a training dataset consisting of sensorbox measurements and simultaneous target pollutant concentration measurements.
Calibration model The first calibration datasets, consisting of sensor box data measured at the same time with micro-aethalometer data, were obtained in Antwerp near an air quality monitoring station at a traffic site and in Turin in spring 2013 from a two-week long monitoring with sensor boxes and micro-aethalometers positioned near a busy road. These datasets were used to compare the performance of different supervised learning techniques. We have explored four different possible models to use for mapping of sensor output to the reference measurements. These were Random Forests, Support Vector Machines (SVMs), a custom air quality index and Artificial Neural Networks (ANNs). After comparing these (Table  5 ), SVMs and ANNs obtained a similar behaviour, better than the other two options, but training of ANNs appeared to be faster, so we decided to adopt them for our model.