dataesr/scientific-paragraphs-categorization huggingface dataset:
Consequently, we avoided radiomics features that were independently selected from other factors and/or not solely derived from the training portion of the data (in the machine learning training-validation-test sense), typically subject to an oftenoverlooked look-ahead bias and loss of future generalizability, addressing the problem of inappropriately applying cross-validation methods to feature selection  (37). However, the addition to our signature combining DeepRadiomics + Clinical was not improved by the implementation of a third parameter such as PD-L1. This could be explained by the limited discovery dataset available in this trial, which would otherwise be necessary to increase model parameters with sufficient examples for machine learning. Altogether, we obtained strong AUC in the discovery cohort independent of PD-L1 expression; however, there was no evidence of generalizability in the validation cohorts.
Introduction The spread of offensive content and hate speech (HS) is a severe and increasing problem in online social communities. While in the last years numerous studies in the Natural Language Processing community have proposed computational methods to address the spread of malicious content, they tend to over-rely on overt and explicit forms of HS, neglecting more implicit and veiled ones (e. g. , "I'm either in North Florida or Nigeria sometimes I can't tell the difference. " from the White Supremacy Forum Dataset (WSF)  (de Gibert et al. , 2018) ). Implicit HS contains expressions of coded or indirect language that does not immediately denote hate but still disparages a person or a group based on protected characteristics such as race, gender, cultural identity, or religion  (ElSherief et al.
Explicit and Implicit HS Detection Many resources and computational methods to detect HS have been proposed in the latest years, such as lexicons (e. g. ,  (Wiegand et al. , 2018; Bassignana et al. , 2018) ), HS datasets and benchmarks (e. g. ,  (Zampieri et al. , 2019; Basile et al. , 2019; Davidson et al. , 2017; Founta et al. , 2018) ), supervised classifiers (e. g. ,  (Park and Fung, 2017; Gambäck and Sikdar, 2017; Wang et al. , 2020; Lee et al. , 2019) ). These studies have set strong basis to explore the issue of HS and abusive language, in particular in social media messages. However, most of these works do not consider subtle and elusive hateful instances (that use for instance circumlocution, metaphor, or stereotypes), that can be as harmful as overt ones  (Nadal et al. , 2014; Kanter et al.
, 2017). To fill this gap, implicit HS detection has recently caught the interest of the NLP community, and benchmarks containing implicit HS messages have been proposed  (Sap et al. , 2020; Caselli et al. , 2020; ElSherief et al. , 2021; Hartvigsen et al. , 2022; Wiegand et al. , 2021a Wiegand et al. , , 2022;; Ocampo et al. , 2023). As for the computational approaches,  (Kim et al. , 2022)  tackle cross-dataset underperforming issues on HS classifiers and propose a contrastive learning method that encodes implicit hate implications close in representation space. (Nejadgholi et al. , 2022)  use Testing Concept Activation Vectors from computer vision to provide a metric called degree of explicitness and update HS classifiers with guided data augmentation. (Han and Tsvetkov, 2020)  propose a pipeline to surface veiled offenses without compromising current performances on explicit HS forms.
Proposed Framework As introduced before, most of the existing methods to detect abusive language in short text messages rely on supervised approaches that strongly depend on labeled datasets for training. But as observed by  (ElSherief et al. , 2021; Hartvigsen et al. , 2022) , most of the available datasets mainly contain explicit forms of HS, ignoring abusive content expressed in more implicit or subtle ways. This results in the current methods' poor detection performance on the implicit HS class as the training datasets are highly imbalanced  (Ocampo et al. , 2023). To mitigate this issue, we propose a framework to generate on-scale close-to-human-like adversarial implicit HS texts using the pre-trained language model GPT3  (Brown et al. , 2020). Our generation framework is composed by four components (Figure  1 ): i) a demonstration-based prompt, ii) a search method, iii) a goal function, and iv) a set of constraints.
Experimental Setting In our experiments, we rely on the text completion model GPT3  (Brown et al. , 2020)  is a re-trained BERT model using over 1 million posts from banned communities on Reddit  (Caselli et al. , 2021)  and then fine-tuned on the ISHate dataset. HateBERT obtained very promising results on the benchmarks HatEval, OffensEval  (Zampieri et al. , 2019) , and AbusEval  (Caselli et al. , 2020). Table  1  (c) reports on the classifier's results on the ISHate dataset  (Ocampo et al. , 2023). As for the sentence similarity model, used by the search method to compute the cosine similarity between the generated text and the shot examples (point (4), Section 3. 2), we used all-MiniLM-L6-v2 2  from sentence-transformers. It has been trained on a 1B sentence pairs dataset to be used for information retrieval, clustering, and sentence similarity tasks.
ImpScore(w) = PMI(w, ImpHS) -PMI(w, ExpHS) (5) After that, we apply a sigmoid function to scale the weights between our required range. Table  2  shows that the least ranked words are derogatory and refer to targeted HS groups. On the other hand, the most rated tokens are neutral and can be found on whichever document on the web. Concerning the research questions listed above, to answer to RQ1, we group the messages directed towards the most attacked targets in the ISHate dataset (see Table  1 ), namely Muslims, Migrants, Jews, Black People, Women, White People, and Asians. From these groups, we randomly select demonstration shots to be used to make a prompt, as shown in Figure  2. This prompt is passed as input to the generator in order to produce implicit adversarial examples. We generated a total of 20 instances per target group, obtaining 140 messages using only GPT3 without any constraint.
Each run was manually validated by the same annotators of  (Ocampo et al. , 2023)  following the ISHate annotation scheme and guidelines, in order to identify how many of these messages i) are actually implicit (% Implicit), ii) maintain the same target group (% Target), and iii) fool HateBERT (% Attacks). Additionally, we calculated how similar the generations were to the prompts in average (x Similarity) and we equally separated the messages into three groups (% Easy), (% Medium), and (% Hard) according to the goal function score defined in Section 3. 3. For RQ2, we repeat the same procedure without grouping per HS target. Implicit HS shot examples are selected from the ISHate dataset, and used as input prompts, so as to generate 140 adversarial examples. Each generated example is manually validated to calculate the metrics for RQ2 listed above and compare them with the metrics for RQ1.
This might be a consequence of not only providing adversarial examples on the target class, but also retraining on instances that are non-hateful and explicit from the ISHate dataset. After the second round M 2 , we see an improvement in the performances of the implicit class. M 2 can update its parameters on this retraining round through the provided material, that helped it to correctly classify instances that were previously misclassified by M 1. On the contrary, from the third iteration, while the classification of false negatives improves through the generator's attacks, this forces the classifier to prioritize recall over precision, affecting the overall performance of M 3 on the implicit class.
e. , HateBERT). Furthermore, we have proposed a build it, break it, fix it, approach that uses the adversarial examples generated by the above described framework to incrementally retrain machine learning models and improve their classification performances. We showed how adversarial generation leverages the HateBERT classification model on the ISHate dataset by improving false negative classification. While this strategy may have potential issues, such as cyclical progress, it remains a valuable approach to improve model robustness, accelerate progress, define clear objectives  (Dinan et al. , 2019; Kiela et al. , 2021; Vidgen et al. , 2021; Wallace et al. , 2022) , and gain a deeper understanding of models' errors as shown in our paper. As for future work, we plan to explore how to embed implicit and subtle statements properly in representation spaces  (Kim et al.
The parameters ω of our network are optimized to minimize the P-score by the stochastic gradient descent Adam available in PyTorch. In our experimental learning setting, we set a batch size of 64 and a maximum number of 1,000 epochs. At predefined epochs, the learning rate is decreased. It ranges from 10 -3 to 10 -7. The parameterization for which the P-score is the lowest on the validation data set is saved. We let the reader refer to the code available online (https://doi. org/10. 5281/zenodo. 7729564) for additional details on the implementation.
Acknowledgments This work was supported by the  French Agence Nationale de la Recherche (ANR)  under reference  ANR-Melody  ( ANR-19-CE46-0011 ). Part of this work was supported by  80   PRIME CNRS-INSU ,  ANR-20-CE40-0025-01  ( T -REX  project), and the  European H2020 XAIDA  (Grant  101003469 ). Our warmest thanks go to the reviewers, who provided insightful comments throughout the revision process. They have greatly contributed to improving the paper.
Others, like the Hide-and-Seek challenge  [10] , the INSAnonym competition  [1]  or PWSCUP  [17]  additionally consider attacks on the sanitized datasets generated by the participants. More precisely, these competitions were generally composed of two phases, in which the first one is dedicated to the design of sanitization algorithms (usually focus on a data type and use case) while the second one usually consists in attacking the data sanitized with the algorithms developed during the first phase. While the sanitization phases of past challenges have been successful, in the sense that they provided insights on the privacy/utility trade-offs or that it has lead to implementations of state-ofthe-art or novel sanitization algorithms, the outcomes of the attack phases were usually more mitigated.
For example, the organizers of the Hide-and-Seek challenge have reported attack results equivalent to random guesses  [10]. The recent Microsoft Membership Inference Competition (MICO)  [15]  consisted only in an attack phase and is therefore an exception to this observation. We believe that facilitating the organization of attack challenges over sanitization algorithms can greatly benefit the research community -essentially the privacy-preserving database and privacypreserving machine learning fields -by stimulating research, contributing to strengthening the implementation of sanitization algorithms as well as by generating open source implementations of state-of-the-art attacks. This is precisely the overarching goal of the Snake framework. Snake allows managing and easy automation of the redundant tasks that are inherent to any such challenge (e.
g. , preparing datasets or running the sanitization algorithms). More precisely, Snake exhibits the following salient features. First, Snake is generic in the sense that it is agnostic with respect to the sanitization mechanisms under attack (e. g. , aggregated data, 𝑙-diverse partitioning algorithms or differentially private synthetic data generation), supports the specification of a wide range of parameters (e. g. , privacy levels and hyperparameters) and allows participants to compete in various adversarial settings (e. g. , background knowledge and success metrics). Second, Snake is easy to use both for participants and for organizers thanks to the Snakemake software  [12]. This facilitates both the organization of challenges (for the organizers) and the creation of tasks for local tests during challenges (for the participants).
Third, Snake is extensible as it can support various challenge designs (e. g. , tracks, timelines, enabling code submission or not), threat models with respect to participants and the organizers (e. g. , colluding participants or not, covert organizers) as well as reproducibility features. As a result, we believe that Snake can help nicely complement the current sanitization competitions that focus on the sanitization part, thus resulting in a complete defense-attack pipeline. To the best of our knowledge, Snake is the first framework of its kind. We propose to demonstrate the genericity, usability and extensibility properties of the Snake framework through a complete scenario focusing on membership inference attacks  [5, 7, 8]  over differentially-private synthetic tabular data generation schemes  [13].
RELATED CHALLENGES The closest related challenges to ours are the Hide-and-Seek privacy challenge  [10]  from the Van Der Schaar laboratory, the INSAnonym competition  [1] , the PWSCup  [17]  series of competitions as well as the Microsoft Membership Inference Competition (MICO)  [15]. The three former challenges differ on the rules, the data and tasks they propose, but they follow the same two-phase structure: (1) a sanitization phase during which the participants implement a sanitization algorithm and run it on the given dataset and (2) an attack phase during which the sanitization algorithms are attacked. The attack phase of these competitions is often short (either by design or in practice) and the algorithms attacked are chosen by the participants. With the Snake framework, we aim at facilitating and thus strengthening the attack phase.
Compared to Hide-and-Seek, INSAnonym or PWSCup, the sanitization algorithms attacked can be chosen from the literature and/or from the latest sanitization challenges (e. g. , the NIST differential privacy challenges) and the number of algorithms under attack can remain small. This gives the opportunity to stress-test state-of-the-art sanitization algorithms and helps participants to focus on their attack algorithm. In particular, we believe that the Snake framework complements nicely the other challenges dedicated to designing sanitization algorithms. Finally, the recent MICO Competition differs from the other three challenges mentioned above in that it consists in a single attack phase. More precisely, it focuses on membership inference attacks, as does the first edition of Snake, but differs from Snake 1 on the algorithms attacked (classifiers rather than synthetic tabular data generators) and on the background knowledge given to participants (disclosure of the learned model rather than disclosure of only its outputs and hyperparameters).
Original data Prepare  Abstractions. The abstractions defined in Snake consist in a workflow and its functional specifications. Figure  1  describes the Snake workflow. The rule Prepare data creates the main dataset of the challenge, called Base data, from some existing source of data, called Original data. Each task of the challenge is created by Create task, as a set of Public and Private files, with the former shared to participants, and the latter used by the organizers for the evaluation of the attacks submitted by participants. The third rule Create bundle packages all the competition files into (1) a bundle that can be uploaded to the CodaBench competition platform  [21]  to instantiate the full competition, (2) a public bundle that contains all the data needed by participants (e. g.
, available online during the competition) and (3) a private bundle that contains the solutions (e. g. , available to participants exclusively after the challenge). Technical Foundations. Snake leverages (1) the Snakemake workflow management system  [12]  for its inner working and (2) the CodaBench competition platform  [21]  for all the features necessary for running a competition in real-life (e. g. , user registration, submission management, leaderboard). Snakemake is a bioinformatics workflow engine which provides a domain specific language implemented as an extension to Python to describes pipelines. Workflows are specified as a directed acyclic graph of rules which transform input files into target files, similar to makefile directives. Snake uses Snakemake to prepare all files required to define the challenge and outputs a "bundle" which can be uploaded to CodaBench to build a complete challenge.
It can thus be used easily by participants during a competition for building a complete local competition environment allowing local executions of the sanitization algorithms and of the participants' attacks. On the organizer side, existing challenges can easily be framed within Snake. Note that using Snake does not generate any computational overhead compared to preparing a challenge without using Snake. Moreover, the Snake framework outputs a packaged bundle compliant with the CodaBench platform, which allows for user-friendly participation (documentation, live ranking and automated scoring using a Python script). Reproducibility. Snake allows a challenge to be reproduced in full, thanks to the functionalities offered by Snakemake. For example, inputs, outputs, execution environments, parameters and scripts are tracked through hashing.
Original data Base data  First, the data used by the challenge is prepared as described afterwards. Each execution of a sanitization algorithm takes as input a private dataset consisting of random samples from the base dataset. The attack used as input the synthetic data, a set of targets for which membership is to be guessed, the base data and the parameters of the sanitization algorithm. The participants then perform their attack locally  3  and submit their guesses to the CodaBench space of Snake 1. The CodaBench platform scores the tasks by comparing the submission with the (private) ground truth. The following is an excerpt of the task creation rule (with irrelevant parts removed), with task_prep creating most of the public and private files, while synthetic data is generated by a separate rule that uses a specific Conda environment.
Attacks algorithm. Each team has to design an attack algorithm as follows. Input. Attacks algorithms are provided with (1) the synthetic dataset generated by an execution of the targeted sanitization algorithm over a private dataset, (2) the targets to attack, (3) the base dataset from which the private dataset is sampled, and (4) the parameters of the execution of the sanitization algorithm attacked; Output. The output of the attack algorithm is a real number in [0, 1] indicating the predicted probability of each target being within the private dataset or not. We detail below the computation of the private datasets, the targets, the sanitization algorithms as well as the attack success measure. Base dataset and private datasets. Snake 1 makes use of the publicly available EPI CPS Basic Monthly data provided by the Economic Policy Institute  [9].
The CPS dataset is divided in years in which a yearly dataset contains more than 10 6  records and 125 columns  4. A record contains information about a single individual in a household. Our base dataset is built by using a pre-processed sample of columns and rows from the original dataset. Full details are available in the competition online documentation. From this base dataset, we generate one private dataset for each parameterized sanitization algorithm attacked. Targets and background knowledge. In Snake 1 , any household that contains at least 5 individuals might be a target, with the target consisting of the full set of records of the household. Snake 1 considers the following background knowledge about each target. The adversary knows (1) the exact records of the household targeted, and (2) the full base dataset  5.
The adversary is also given the information about the sanitization algorithm targeted as well as the parameters used for the executions and has access to its implementation. However, the randomness generated internally during the execution of the algorithm is unknown to the adversary. Sanitization algorithms under attacks. The sanitization algorithms under attack during Snake 1 are differentially-private synthetic data generation algorithms. More precisely, we have selected a set of algorithms according to the following two criteria: technical soundness assessed by a rigorous peer-selection process (e. g. , published at top-tier conferences or winner of a dedicated competition) and available open-source implementation. In particular, we have used the implementations available in the Reprosyn package  [16].
g. , MST parameterized by 𝜖 = 1. 0 and 𝛿 = 10 -5 ) and second by aggregating the success measures in a single final score. The success of a given attack for a given parameterized algorithm is evaluated based on the well-known membership advantage measure  [22]. The number of targets must be at the same time large enough for obtaining a sufficiently stable estimation and small enough for being practical (e. g. , 𝑟 = 100). Execution environment. The design, coding, and executions of attacks are performed locally by each team with its own resources. As a result, there is no restriction on the computing environment used to develop the attacks and the choice of the programming language and the available resources are unconstrained. Note however that we provide to participants the Snake 1 Python environment.
DEMONSTRATION We propose to demonstrate the key features of the Snake framework on several datasets along the following three scenarios. Organizer side: creating a competition. We demonstrate how one can create a competition with Snake, highlighting the benefits of Snake for automating the creation of tasks and bundles. Participant side: overview of the first edition. We use the first edition of Snake to give an interactive view of participating in a running competition (e. g. , accessing the tasks generated by Snake, running a baseline attack and submitting the results). Extension: code submission. We demonstrate how the framework can be extended with minimal changes using the example of the ekans toy competition (available on CodaBench 6 ) which is the "reverse" of Snake 1. The objective of ekans is to propose a way to sample targets that are the easiest to attack.
ACKNOWLEDGMENTS We would like to warmly thank the students, engineers and researchers that have helped us to design the Snake challenge. This work is partially supported by the  ANR 22-PECY-0002   IPOP  ( Interdisciplinary Project on Privacy)  project of the  Cybersecurity PEPR , the  Canada Research Chair program , a  Discovery Grant  from  NSERC  as well as  CIFAR  through a  CIFAR-MILA-IVADO-MITACS Catalyst Grant.
The challenge of the project is to allow the design of models to understand the processes of collaborative work and learning in school and professional spaces. Based on a partnership in France, between the University of Lille (UL), the University of Technology of Compiègne (UTC), and the Academy of Amiens, this project uses the technology available in La Halle Numérique of the UTC (figure  1 ). This platform consists of five meeting rooms, each of these digital spaces is composed of a large tactile table and a large tactile board. Each of these horizontal and vertical surfaces is equipped with the Ubikey® Office software suite 1  which allows the simultaneous interaction of several people on the same surface. Users can write virtual Post-it® notes, draw, search the web, use canvas, build various types of diagrams and spreadsheets, etc.