dataesr/scientific-paragraphs-categorization huggingface dataset:
Which ML algorithm to use? While ML is generally beneficial, the prediction quality varies between the different algorithms. For instance, OLS Regression generally leads to bad results, e. g. , 22% of error with 10% of the configurations. Unlike the OLS regression, tree-based learning algorithms take advantage of the addition of new measurements. For a budget of 50% of the configurations, their median prediction goes under the 5% of error, which is encouraging. This result of 5% is only valid in average; the prediction will be better for a few software systems, e. g. , imagemagick or x264, but does not hold for others, e. g. , lingeling or poppler. Though there is no big difference between these three learning algorithms, we observe slightly better predictions for Random Forest compared to Decision Trees, and for Gradient Boosting compared to Random Forest.
What is the benefit of hyperparameter tuning? Our results found that hyperparameter search improves the MAPE on average by 8 ± 16% within a range of 3 → 37% but also requires on average 120 times more training time when doing a grid search of estimator parameters. It should be noted that this is an improvement in percent, not percentage points. While the exact overhead of hyperparameter search is dependent on the number of configuration options of the model and the search method, we note that the overall benefit on the datasets is limited. This is especially confirmed by the observation that the best-found hyperparameters were changing depending on both the system and the size of the training dataset. For simplicity of the setup, the rest of the evaluation uses the default parameters of each model, if not otherwise noted.
How many inputs are needed to learn an accurate predictive performance model? Measuring inputs differs across software systems: measuring 5 inputs represents 5 * 201 = 1005 configurations for x264 but only 5 * 30 = 150 for xz. Fig. 4  shows that the performance model reaches   its lowest error threshold when considering about 20 inputs. Results are thus easier to interpret on systems with more inputs, as compared to gcc and xz. There are however more difficult cases, e. g. , Node. js and poppler, in our experiment that might require more inputs to improve the accuracy of the prediction. To get a consistent prediction, we recommend the user to measure at least 25 inputs with a sufficient number of configurations per input.
What is the best trade-off between selecting inputs and sampling configurations? According to Fig. 5  results, diversifying the inputs is more effective than selecting different configurations to train an input-aware performance model. But for a fixed budget of inputs, there is a slight improvement in accuracy when increasing the number of configurations. As a result, both should be combined to obtain the best possible model. Overall, the ideal budget -both in terms of inputs and configurations -highly depends on the expected level of errors combined with the difficulty of learning a predictive performance model for the software under test. For instance, predicting the performance of imagemagick is relatively easy, with an average MAE value at 0. 06. For this software system, picking 25 inputs is almost already a waste of resources, we do not need that much data -5 inputs is already enough with 30% of the configurations.
Is it better to use transfer learning or a supervised online approach? With the input selection technique set to Closest Performance and whatever the percentage of configurations used in the training, the transfer approach always outperforms the supervised online approach, as shown in Fig. 6. This strong result demonstrates the importance of capitalizing on the existing measurements, measured in an offline setting. Hence, if we are able to create representative sets of inputs L. Lesoil et al. for each software system (thanks to Closest Performance), then transfer learning becomes the best approach to use. However, the strategy of picking the source input among all the inputs of our dataset is not always possible -it requires a high offline budget. We consider another scenario where we put ourselves in the situation of a user with a low offline budget i.
These observed errors serve as a rule-of-thumb, but are, of course, not directly transferable to other systems and setups. As a limitation of our work, we highlight that it is difficult to learn the performance distribution for a few software systems e. g. , lingeling, poppler, and even Node. js For these systems, the prediction errors are above 20% when implementing a supervised offline approach with tight budgets of configurations or inputs. It is worth noticing that a computational effort in terms of measurements is requested i. e. , to measure more than the general and averaged threshold of 25 inputs on average. The requested amount of inputs on a per-system basis is documented in the companion repository. 7  This is potentially related to the impact of individual features on the performance metric, i.
e. the spread of the correlation, which was investigated in concurrent work for the same dataset as used in this paper  (Lesoil et al. , 2023). For the difficult-to-learn software systems, there are multiple features that have a high impact on the performance metric  (Lesoil et al. , 2023, Table 4 ), leading to a more difficult-to-learn objective landscape for the regression task. Our results relate further to the existing body of work and confirm some of the results previously found. For example, BEETLE  (Krishna et al. , 2021)  highlights the importance of selecting the right inputspecific source(s) for transfer learning to maximize the accuracy and mitigate the risk of negative transfer, a problem similar to the input selection we consider in 𝑅𝑄 2. In  Jamshidi et al. (2018) , it is discussed in which scenarios transfer learning is more applicable compared to when it might lead to a negative transfer.
The applicability is found to be hindered by a higher severity of the change between the original training environment and the target prediction environment to which the model is transferred. This problem of negative transfer in variability modelling was similarly confirmed for the Linux kernel in  Martin et al. (2021). The negative transfer problem underlines the importance of establishing a training set that is as broad and diverse as possible, in order to best fit the data distribution that the model will apply during deployment. The closer the target environment, albeit new inputs, configurations, or other variability aspects, is to the environments from which the training data was collected, the smaller the risk of a negative transfer and the better the performance prediction quality.
Threats to validity A first threat to validity is linked to the data we are using; since we rely on a dataset  (Lesoil et al. , 2023) , we are exposed to the same threats to validity. In particular, an error in the measurement protocol could invalidate our results. Besides, we do not consider all the possible configuration options of the software systems. The fact that it includes multiple software systems (8 in total) and a consequent number of performance measurements (roughly two million when considering the different performance properties) is supposed to alleviate these threats. A second threat to validity relates to the input properties computed in Section 3. 1. 2. Since we are not domain experts of each of the eight software systems considered in this experiment, we cannot validate the construction of such properties, i.
To reduce these stochastic effects, we (1) fix the random seed to feed the same training and test sets to all models and have comparable results and (2) repeat the experiments 20 times. Finally, we acknowledge that we relied on already existing libraries and implementations. These can be buggy or present some inaccuracies that may favour our results. We choose to use ML implementations coming from scikit-learn which is one of the most popular Python ML libraries at the moment. Its community is active and sensitive to these aspects, we can assume that if such a problem would exist, it would have been discovered and fixed quickly.
Let us take the video encoding  (Maxiaguine et al. , 2004)  as an example: Alves Pereira et al. (  2020 ) study the effect of sampling training data from the configuration space on x264 configuration performance models for 19 input videos on two performance properties. Netflix conducts a largescale study for comparing the compression performance of x264, x265, and libvpx  (Jan De Cock et al. , 2016). 5000 12-s clips from the Netflix catalogue were used, covering a wide range of genres and signal characteristics. However, only two configurations were considered and the focus of the study was not on predicting performances. Our study covers much more inputs, systems, and performance properties. Valov et al. (2020)  proposed a method to transfer the Pareto frontiers (encoding time and size) of performances across heterogeneous hardware environments.
The input sensitivity issue has also been identified -and sometimes dealt with -in some other domains: SAT solvers  (Xu et al. , 2008; Falkner et al. , 2015) , compilation  (Plotnikov et al. , 2013; Ding et al. , 2015 ), data compression (Khavari Tavana et al. , 2019) , database management  (Van Aken et al. , 2017; Duan et al. , 2009) , cloud computing  (Duarte et al. , 2018; Liu et al. , 2020; Ding et al. , 2021) , etc. These works purposely leverage the specifics of their domain. However, it is unclear how proposed techniques could be adapted to any domain and all software systems  (Mühlbauer et al. , 2023a). Thus, We favour a generic, domain-agnostic approach (e. g. , transfer learning) as part of our study. Importantly, most of these works pursue the objective of optimizing the performance of a software system according to a given input (workload).
L2S could be adapted as part of transfer learning (see Fig. 2 ). However, L2S is highly sensitive to the selection of a source (an input) for a given target (another input). Martin et al. develop TEAMs (Martin et al. , 2021) , a transfer learning approach predicting the performance distribution of the Linux kernel using the measurements of its previous releases. Valov et al. showed that linear models are effective to transfer knowledge across different hardware environments  (Valov et al. , 2017). However, inputs can significantly alter performance distributions e. g. , Pearson correlations can be close to 0 for some pairs of inputs, systems, and performance properties. There is not necessarily a linear correlation and relationship, as for hardware changes. We assess model shifting as part of transfer learning.
Second, in our case, the set comprises all (valid) configurations of a single, parameterized, configurable system. In the automated algorithm setting, the set of algorithms come from different individual software implementation and systems. As stated in  Kerschke et al. (2019)  (Section 6), our problem differs and is still open because (1) the space of valid configurations to select from is typically very large; (2) learning the mapping from instance features (i. e. , inputs' properties) to configurations is challenging. We precisely address this problem in this article, considering a large dataset and multiple learning approaches.
Conclusion Due to the interactions between inputs and configurations, predicting the performance property of a software configurable system whatever the input data is non-trivial and yet of practical importance. In particular, performance models trained on a single input can quickly become inaccurate and useless when used over other inputs. This lack of generalizability and practicality suggests to investigate solutions for learning input-aware performance models. In this article, we empirically evaluated the effectiveness of different learning strategies (offline learning, supervised online learning, transfer learning) and user personas when addressing this problem. We leveraged a large dataset comprising 8 software systems, hundreds of configurations and inputs, and dozens of performance properties, spanning a total of 1,941,075 configurations' measurements.
Luc Lesoil holds a Ph. D. from University of Rennes 1 in 2023. His research interests involve deep software variability, configurable systems, performance engineering, machine learning. He is the author of papers published at TSE, JSS, SPLC, and ICPE. His contributions also involve the creation of reproducible artefacts (dataset, data science pipelines, notebooks). Helge Spieker is a research scientist at Simula Research Laboratory in Oslo, Norway. He is interested in the intersection and integration of data-driven machine learning and logic-driven symbolic AI methods, such as Constraint Programming, especially with an application in the domain of software testing. His research topics also span self-supervised neuro-symbolic solvers for constraint satisfaction, through research in collaboration with the University of Bonn, Germany, under the AutoCSP project funded by the Norwegian Research Council.
The FGSS framework contains singleview and multi-view versions. In the single-view version, we propose a fine-grained self-supervision strategy to distinguish the sub-parts of the semantic class for better class discriminability. In the multi-view version, we propose a class prototype feature enhancement strategy to generate another view (i. e. another representation of the original representation). Then, we propose a multi-view mutual supervision loss to enforce consistency between different views and further enhance the generalizability of the model. Experimental results on five widely-used datasets, i. e. , GTAV, SYNTHIA, BDD100K, Cityscapes, and Mapillary, demonstrate that our FGSS framework achieves superior performance compared to state-of-the-art methods. Index Terms-Semantic segmentation, Domain generalization, Fine-grained, Self-supervision, Intra-class relationship.
I. INTRODUCTION Propelled by the swift progress in deep neural networks  [1] , Semantic segmentation, a crucial task in computer vision, has made remarkable progress in recent years relying on large amounts of annotations and has been broadly used in lots of applications such as autonomous driving  [2] ,  [3]. This task infers a semantic category for each pixel of an image. Despite the rapid development of semantic segmentation in a fully-supervised manner, its drawbacks are also evident. First, labeling pixel-level annotation is laborious and timeconsuming. As mentioned in GTAV  [4] , labeling an image from real-world datasets CamVid  [5]  and Cityscapes  [6]  takes 60 and 90 minutes, respectively. Consequently, some virtual datasets leverage the computer simulator to collect synthetic data containing automatically generated annotations.
Based on the number ACCEPTED MANUSCRIPT / CLEAN COPY of the source domain, DG can be categorized into singlesource DG and multi-source DG, where single-source DG is more challenging than multi-source DG due to its limited domain diversity and less training data. This paper focuses on devising a single-source DG model. Data manipulation and representation learning are two common approaches for DG. The former tries to extend the training set and enhance domain diversity by changing image style to cover new possible domains, while the latter focuses on domaininvariant representation learning. In some circumstances, both types of approaches are co-existing. For example, DRPC  [11]  randomized the style of the images using auxiliary datasets and then performed pyramid consistency to learn domaininvariant features.
C. Domain generalization Domain generalization (DG) seeks a robust model to generalize well on all possible unseen domains. The target domain data does not participate in the training process of the DG model. Data manipulation and representation learning are two main types of approaches to improve generalization. For data manipulation, the source training set is extended as much as possible to cover all unseen domains at the image level by data augmentation or data generation. For instance, DLOW  [45]  generated a continuous sequence of intermediate domains from one domain to another. DRPC  [11]  tried to generate images with different styles using auxiliary datasets and then enforced pyramid consistency across data with distinct styles. FSDR  [46]  randomized image styles in the frequency space by spectrum learning.
However, fine-grained annotations is not provided in datasets. To tackle this issue, the fine-grained selfsupervision strategy (FSS) is proposed to supervise the feature by fine-grained pseudo-label. Thus, the Equation (3) can be rewritten as: L f s = - h,w n f ∈N F q sf (y sf |f ) (h,w,n f ) log(C f (f (h,w) )) (4) where fine-grained labels y sf are encoded as posterior distributions q sf (y sf |f ) by online pseudo-label assignment described in the following paragraph.
After that, the fine-grained label of one ACCEPTED MANUSCRIPT / CLEAN COPY view is used to supervise another view. The multi-view mutual supervision loss L mf s can be defined as: L mf s = L f s (q sf , f aug ) + L f s (q aug sf , f ) (14) The MVMS loss implicitly performs alignment between the original and the augmented views to learn more generalized representation. More details of both versions of the FGSS framework are provided in Algorithm 1.
A. Dataset With the goal of single-domain generalization, the performances of unseen multi-domains need to be evaluated. Five datasets used in the experiments consist of synthetic datasets (GTAV  [4]  and SYNTHIA  [66] ) and real-world datasets (Cityscapes  [6] , BDD100K  [67]  and Mapillary  [68] ). Synthetic datasets: Several synthetic datasets are created to avoid the amount of human effort in labeling. The GTAV dataset collected 25k images from the computer game Grand Theft Auto V with a high resolution of 1914 × 1052. The SYNTHIA dataset contains 9,400 1280 × 760 images. 19 and 16 common categories with Cityscapes are used in the GTAV and SYNTHIA datasets, respectively. Real-world datasets: The Cityscapes dataset is a real-world street scene dataset widely used in the semantic segmentation task, which contains 2975 training and 500 validation images with the resolution of 2048 × 1024.
The BDD100K driving scene dataset contains 7000 training and 1000 validation images with HD resolution (i. e. , 1280 × 720). The Mapillary dataset contains 25k with at least FHD resolution (i. e. , 1820 × 1080), which is captured in various environments such as different weathers and seasons. Both BDD100K and Mapillary datasets adopt 19 overlapped classes with the Cityscapes.
C. Results We compare our approach to the other state-of-the-art methods on three backbone networks (i. I  and the reported results adopt the model of the multi-view version by default. The performance of our method substantially outperforms the second-best methods  [65]  by 1. 9% in terms of average mIoU among the above five generalization settings. In the 20 evaluation settings, our FGSS framework achieves 13 best and 3 second-best performances. Compared with ISW  [13] , our method achieve 17 better performances in 20 generalization evaluation settings. These results show that our FGSS framework learns a more ACCEPTED MANUSCRIPT / CLEAN COPY  Some qualitative examples of different settings (G→C, G→B, G→S, and G→M) are provided in Fig. 5. Some segmentation errors in the Baseline are eliminated in two versions of the FGSS framework.
V. CONCLUSION In this paper, we designed a fine-grained self-supervision (FGSS) framework for generalizable semantic segmentation, ACCEPTED MANUSCRIPT / CLEAN COPY which considered both the generalizability and discriminability of the model from the intra-class relationship. The FGSS framework consists of single-view and multi-view versions. In the single-view version, we proposed a fine-grained selfsupervision strategy which treated the fine-grained segmentation as an auxiliary task to improve feature discrimination. In the multi-view version, we furtherly proposed class prototype feature enhancement to generate another view from the feature space, where the augmented feature is close to the related prototype forcing the classifier to capture more domain-invariant information. Moreover, we proposed multiview mutual supervision loss in the fine-grained segmentation task for the label inconsistency problem in two views to implicitly reduce the gap between different views.
ACKNOWLEDGMENTS This work was supported in part by the  National Natural Science Foundation of China  under grants  62171294 ,  62101344 , in part by the key  Project of DEGP (Department of Education of Guangdong Province)  under grants  2018KCXTD027 , in part by the  Natural Science Foundation of Guangdong Province, China  under grants  2022A1515010159 ,  2020A1515010959 , in part by the  Key project of Shenzhen Science and Technology Plan  under Grant  20220810180617001 , in part by the  Interdisciplinary Innovation Team of Shenzhen University  and in part by the  Tencent "Rhinoceros Birds" -Scientific Research Foundation for Young Teachers of Shenzhen University, China.
e. , ResNet-50, MobileNet-V2, ShuffleNet-V2). The GTAV, BDD100K, SYNTHIA, Mapillary, and Cityscapes datasets are denoted as G, B, S, M, and C, respectively. In the ResNet-50, these five datasets are taken turns as the source domain for training and the datasets except the source domain are used for evaluation, which is grouped into G → {C, B, M, S}, S → {G, B, M, C}, C → {G, B, S, M}, B → {G, C, S, M}, and M → {G, B, C, S}. The right arrow → refers to "generalizing to". The performance comparison is shown in Table.
INTRODUCTION As the need and development of various kinds of research and analysis using personal data are becoming more and more significant, the risk of privacy violations of sensitive information of the data owners is also increasing manifold. One of the most successful proposals to address the issue of privacy protection is differential privacy (DP)  [23, 24] , a mathematical property that makes it difficult for an attacker to detect the presence of a record in a dataset. This is typically achieved by answering queries performed on the dataset in a (controlled) noisy fashion. Lately, the local variant of differential privacy (LDP)  [22]  has gained popularity due to the fact that the noise is applied at the data owner's end without needing a trusted curator. LDP is particularly suitable for situations where a data owner is a user who communicates her personal data in exchange for some service.
Motivated by this dilemma, we propose PRIVIC, an incremental data collection method providing extensive privacy protection for the users of LBS, while retaining a high utility for both them and the service providers, and ensuring that both parties, acting in their best interest, would benefit from the end mechanism. Finally, we prove formally the convergence of PRIVIC to the true distribution and illustrate empirically the privacy-utility trade-off of our method. The experiments also demonstrate the efficacy of combining BA and IBU, in that the estimation of the original distribution is very accurate, especially when measured using a notion of distance between distributions compatible with the ground distance used to measure the QoS (e. g. , the Earth Mover's distance). All the experiments were performed using real location data from the Gowalla dataset for Paris and San Francisco.