dataesr/scientific-paragraphs-categorization huggingface dataset:
For example, recent trends in psychology research highlight the need for empirical experiments to pre-register the study designs, sample size, and other details to ensure that researchers are not "p-hacking" (adjusting their analysis methods until a positive result is achieved;  Head et al. , 2015; Simmons et al. , 2013)  or changing their hypotheses post-hoc to confirm to the data that they have collected. To address this, HRI researchers must update their expectations and methods each year. Furthermore, researchers studying how to market robots may utilize marketing techniques, and researchers examining how individuals or groups of people interact socially with robots may use methodological techniques from social psychology  (Reeves and Nass, 1997)  or group dynamics literature (e. g. ,  Fraune et al.
ACKNOWLEDGMENTS This article also benefited from the support of the project  Prog4Yu ANR-18-CE10-0008  of the  French National Research Agency (ANR). This material was supported in part by the  Air Force Office of Scientific Research  under award number  21USCOR004. HG is supported by the  EPSRC  project  ARoEQ  under grant ref. EP/R030782/1. NIA  is supported by the  W. D. Armstrong Trust Fund Studentship. NIA  is supported by the  W. D. Armstrong Trust Fund Studentship  and the  Cambridge Commonwealth, European and International Trust.
We use the development set of the News-commentary-v15 corpus to compute bias scores, and report the results in Table  2. Although existing debiasing methods have been shown to reduce bias in embeddings  (Liang et al. , 2020a; Kaneko and Bollegala, 2021) , we observe that they do very little to reduce it in attention. In fact, the method of  Kaneko and Bollegala (2019)  and CDA induce the model to encode more bias in the attention layer, which might make it to resurface in predictions if not addressed correctly. In the following, we demonstrate that AttenD is also capable of mitigating bias from text representations and likelihoods. To do that, we finetune text encoders of interest before and after applying debiasing methods on the language modeling task. Then, we use the publicly available subsets of two stereotype benchmarks: StereoSet  (Nadeem et al.
, 2020)  and Crows-Pairs  (Nangia et al. , 2020). Both provide likelihood-based diagnostics to measure how often stereotypes are considered likelier than anti-stereotypes, given the language model's likelihoods. An ideal unbiased text encoder should score 50% in these benchmarks, i. e. it prefers neither stereotypes nor anti-stereotypes. Table  3  provides the evaluation results. StereoSet also provides a means to compute a language modeling (LM) score whose purpose is to check whether the encoder is still good at the task of language modeling, and that debiasing didn't hurt semantic performance. We observe that AttenD shows impressive debiasing performance when evaluated with likelihoodbased diagnostics. Improvements go up to 9. 53% with a slight decrease in the accuracy of language modeling (-1.
In this experiment, we finetune text encoders on MNLI dataset for natural language inference  (Wang et al. , 2018). A bias-free model should score 1 (100%) in all three measures. We report our findings in Table  4. Our method outperforms the original model and the baselines. This result shows that AttenD succeeds in mitigating stereotypes in real world inference settings. We also debias hate speech detection models and report the results in Appendix A. 3.
Evaluations of Semantic Preservation We use GLUE benchmark  (Wang et al. , 2018)  to verify whether the debiased text encoder still holds enough semantic information to be applicable in downstream NLP tasks. In essence, GLUE assesses the natural language understanding capabilities of NLP models. So, it constitutes a suitable stack to evaluate the semantic preservation of AttenD. In this experiment, we finetune our debiased models on seven different tasks from GLUE and show that per-task accuracy is preserved in Table  5. We also observe that not using negative examples (AttenD -) severely hurts semantics.
A. 3 Extrinsic bias evaluation on the task of hate-speech detection Recent studies show that intrinsic metrics of bias do not necessarily correlate with bias measures on concrete real-world applications  (Goldfarb-Tarrant et al. , 2020). In the body of this paper, we already conducted intrinsic and extrinsic bias evaluations. In this experiment, we validate the efficacy of our debiasing method on a concrete real-world hate speech detection application where an input snippet of text is classified as either offensive (toxic, harmful, disrespectful, etc. ) or not. We use hate speech detection because it is well studied in the literature  (Burnap and Williams, 2016; Ribeiro et al. , 2018; Zhang et al. , 2018) , and high-quality datasets which are tagged with social groups already exist  (Borkan et al.
This metric computes whether the model discriminates against the target group with respect to the others. This value is reduced when non-toxic examples of the group have higher toxicity scores than actually toxic examples of the background. Background Negative Subgroup Positive (BNSP) AUC: where AUC is calculated on the positive examples of the target group, and the negative examples of the background. This metric computes whether the model favors the target group with respect to the others. This value is reduced when toxic examples of the group have lower toxicity scores than non-toxic examples of the background. In this experiment, we finetune the text encoder under study on hate speech detection task using the training set of HateXplain dataset  (Mathew et al. , 2021). We also use the test portion of HateXplain for the evaluation, which contains posts from Twitter 11  and Gab 12  annotated with their ground-truth  toxicity scores and the social groups and communities they target.
A. 8 Effect of AttenD on other transformer-based text encoders We evaluate five widely used sentence-level text encoders: BERT  (Devlin et al. , 2018) , ALBERT  (Lan et al. , 2019) , RoBERTa  (Liu et al. , 2019) , Distil-BERT  (SANH et al. )  and SqueezeBERT  (Iandola et al. , 2020). For each model, we evaluate both its base and large variants (except for DistlBERT and SqueezeBERT since these are not available in HuggingFace's transformers library 14  ), original and debiased; which gives a total of sixteen evaluated models. We use Crows-Pairs dataset  (Nangia et al. , 2020)  to quantify the intensity of undesired stereotypes encoded therein. As a reminder, ideal stereotype scores according to Crows-Pairs benchmark should be close to 50, i. e. models preferring neither stereotypes nor anti-stereotypes.
Introduction With the emergence of attacks targeting hardware microarchitecture, including Cache Side-Channel Attacks (CSCAs) such as Flush + Reload  [1]  and Prime + Probe  [2] , and transient execution attacks such as Spectre  [3]  and Meltdown  [4] , a deep understanding of the system's microarchitecture and the study of how the microarchitecture is affected by the software execution has become a very important research issue. Microarchitectural information describes the inner state of the hardware, which is supposed to be transparent to the software. This microarchitectural information includes both storage information, such as a register indicating whether the current execution is a speculative execution, and timing information, such as the execution time of one access to the cache. Many microarchitectural attacks consist in collecting and analyzing microarchitectural information in order to gain knowledge about other processes sharing the same hardware.
In particular, they use a limited set of attacks to validate their proposals. Overall, none of these solutions are sufficient to address the multiplicity of the attacks and their mutation capabilities. In this paper, we propose a reconfigurable framework based on the analysis of microarchitectural signals for the detection of several classes of attacks, especially microarchitectural attacks, called MATANA. Our solution is based on a reconfigurable fabric, deeply integrated into a processor fabric, aimed at capturing meaningful internal signals by executing a set of detection algorithms and alerting the system in case of an attack. To assess the validity of the approach, i. e. , the use of a fully reconfigurable fabric for detection purposes, we evaluated our solution with a frequency gap between the detection module and the processor.
We implemented MATANA into the RISC-V Instruction Set Architecture (ISA) [8] softcore processor Rocket  [9]  with Linux support on the Xilinx ML605 Field Programmable Gate Array (FPGA) platform. We successfully analyzed and designed detection mechanisms for two classes of attacks that exploit different mechanisms: a class of attacks that exploit microarchitecture without altering the control flow of a program (i. e. , CSCAs); a class that alters the control flow (i. e. , Return-Oriented Programming [ROP] attack). For the first class of attacks, we based our detection on instruction patterns and memory access patterns; we found no false positive for the first approach and a few ones for the second with better sensitivity. For the second class of attacks, we detected alteration on the control flow by defining some metrics and thresholds on attack-free benchmarks and applied them to evaluate attack benchmarks.
We also successfully detected attacks with no false positives. The source code of our framework, including MATANA hardware architecture, software stack, and the use cases described in Section 5, is available at https://gitlab. laas. fr/matana (accessed on 10 January 2022). The contributions of the paper are as follows: • A novel microarchitectural signal runtime monitoring and analysis framework, which allows better understanding of interactions between software and microarchitecture at runtime; • A methodology that enables, from the analysis of purposely chosen microarchitectural signals, the detection of various families of attacks via the characterization of their microarchitectural footprints; • An open source framework implementation that can run on real hardware with Linux kernel; • A deep presentation and analysis of two use cases in which we successfully built an appropriate detection logic for CSCAs and ROP attacks.
Instruction Tracing Arm CoreSight  [17]  is another microarchitectural monitoring technology available for off-the-shelf processors. Arm CoreSight is a debug and trace technology inside Arm's System-on-Chip (SoC). Its Program Trace Macrocell  [18]  collects instruction traces by means of tracing key instructions, such as branch, exception, and synchronization. These traces can be accessed via the bus or debug port. To the best of our knowledge, there is no microarchitectural attack detection using Arm CoreSight but some papers mention its use for research works in Dynamic Information Flow Tracking (DIFT)  [19]  and code reuse detection  [20]. The proposed approaches consist of capturing representative traces inside the processor, then decompressing and analyzing these traces inside the FPGA.
Compared with hardware event counters, Arm CoreSight provides much fine-grained information about the software's execution, and the analysis can proceed in parallel inside the FPGA to reduce the execution time of the detection process. However, limited by the bus structure, the FPGA can only obtain little information when running continuous analysis. Similar hardware monitoring techniques that target the fine-grained instruction and data trace of the processor have also been proposed in different academic research works. Several tag-based monitoring techniques for DIFT have been notably proposed. The Raksha architecture  [21]  consists in modifying the pipeline to include tag related memory, tag propagation, and verification logic, while FlexCore  [22]  and Harmoni  [23]  use a dedicated component.
2. Probe Connection. The probe connection step consists in choosing a proper clock domain crossing technique used for connecting each selected signal to the Detection Module, with consideration of information quantity, area overhead, and detection accuracy. This step is only required if the Detection Module is not running at the same frequency as the processor. This distinction is important since, in the case of a reconfigurable Detection Module, the frequency may be much smaller that the processor's core. We illustrate the detection method at low frequency in Section 5. 2. For more information regarding transmission techniques, the reader can refer to Section 4. 3.
The hardware part of the Detection Module (called hardware Detection Module in the following) is implemented in a fully reconfigurable fabric to perform different custom runtime analyses on probed signals. It is equipped with a set of Memory-Mapped I/O (MMIO) registers and memory zone, and provides its intermediate results to the software part of the module (called software Detection Module in the following) located in the system kernel. Additionally, the hardware Detection Module can send interruptions to the processor by means of the interrupt link, in case the hardware requires an immediate intervention of the software-for example, when an attack is detected. It is up to the software Detection Module to decide whether it is a real attack and what action to take, as it benefits from a more global and accurate view of the execution environment.
We did not add any other signal from the Detection Module to the target components, because this would require modification of the target components so that they can handle these signals. By default, the probed signals are directly connected to the hardware Detection Module. This dedicated Probe Connection (which does not require any shared bus) guarantees high throughput and low-latency data transfer that is unaffected by other system activities. However, when the hardware Detection Module is configured in a slower clock domain, other Probe Connections are needed, as presented in Section 4. 3. For the flexibility in the design phase, i. e. , to allow the evaluation of different detection algorithms based on different probed signals, as described in Section 3. 3, the target components, hardware Detection Module, and its connections are located on a reconfigurable fabric such as an FPGA.
For that purpose, we argue that the software part in the Detection Module of MATANA framework is important. The software Detection Module is located in the system kernel, and, as mentioned in the threat model, is considered trusted. Its main functionalities consist of (but are not limited to) the following:  (1)  providing additional high-level information to the hardware Detection Module, such as the current Process Identifier (PID); (2) performing runtime data analysis that is not easily feasible or too costly at the hardware level; (3) dynamically adjusting the configuration of the hardware Detection Module by setting values to MMIO registers; (4) handling interruptions sent by hardware Detection Module; (5) performing postrun data collection and analysis. Let us note that this framework does not require any modification on userland programs executed (either benign or malicious), e.
g. , to include tags or special instructions. It guarantees that MATANA is compatible with existing software environments and, as only few modifications are required at kernel level, it can be widely used even for nonembedded applications. A typical use case scenario of MATANA is as follows: the hardware Detection Module processes probed signals with information and the configuration sent by the software Detection Module and places the analysis result into the MMIO registers. Then, by means of an interruption or a polling mechanism, the software Detection module can fetch the MMIO register's content and perform a more powerful analysis.
Probe Connection Types for Low-Frequency Attack Detection The hardware Detection Module can be integrated into the target components and share the same communication channels to the software. However, when deploying MATANA on the final system, placing the hardware Detection Module separately from the rest of the system for better modularity, using a reconfigurable fabric, and benefiting from the flexibility on detection algorithms implemented are important to deal with attacks that are constantly evolving. For that purpose, we have to consider that the hardware Detection Module may be asynchronous and run at a lower frequency than the target components. In that case, the Detection Module's I/O must be properly synchronized. For that purpose, we propose several synchronization approaches for probed signals that aim at reducing the hardware logic complexity while preserving sufficient information transmitted for attack detection.
The proposed Probe Connection types are shown in Figure  4 , with a target component's frequency divN times higher than the hardware Detection Module's frequency: A Parallel Connection collects all signals in divN adjacent cycles and sends them in parallel to the hardware Detection Module. This connection type maximizes the preservation of the information collected and pinpoints the state of the signals at a specific time, but requires more synchronization logic and internal logic to process it. A Preprocessing Connection preprocesses and buffers the probed signal, then sends the processed data to the hardware Detection Module. It provides a balance between the hardware resource utilization and the amount of useful data transmitted, and can reduce analysis effort in the hardware Detection Module.
Platform Setup A prototype of MATANA has been implemented on a Xilinx ML605 evaluation board equipped with a Xilinx Virtex-6 FPGA running a Linux kernel 4. 15. We chose to build our target system with the Chipyard SoC generation framework v1. 3  [30] , which contains a RISC-V ISA softcore processor Rocket  [9] , since it is open source, actively maintained, and easily reconfigurable. Rocket is an in-order processor core, we configured it to 64 bit, medium size (with support of virtual memory, no Floating-Point Unit (FPU)), an L1 instruction cache, and an L1 data cache). Figure  5  presents the hardware structure of the target system. The hardware part of MATANA is completely integrated in Chipyard (represented in blue color in Figure  5 ). To ease the integration process, all MATANA-related files have been written in chisel [31], a Scala-based hardware design language used by Chipyard that can be compiled to Verilog code, especially to ease the signals connection inside the target system and be still compatible for different SoC parameters.
The probing is mainly done inside the Rocket core to monitor the processor's internal signals, such as the instructions executed, the memory addresses accessed, and some hardware events. We also made several choices to ease and increase the accuracy of the evaluation process: (1) The detection logic has been implemented as a userland program for testing purposes, as it allows the test of different software implementations without regenerating the kernel image and rebooting the system; it is intended to be integrated into the Linux kernel after the testing phase in order to benefit from the protection at kernel level. (2) A Linux kernel module has been developed to manage the communications between the hardware and software Detection Module. (3) The Linux kernel has been patched to automatically capture the current PID at context switch.
The attack phase's assembly code is shown in Listing 1. The first step is called Prime, in which the attacker accesses in turn the 8 addresses previously identified. Due to the replacement policy of the cache, all 8 cache lines of this selected set are occupied by the attacker's data. Then, in a second step, socalled Wait, the attacker's process waits for the victim process to execute. During this step, if the victim uses the set, some victim's data are loaded in the set and evict the attacker's data. The third step is so-called Probe, in which the attacker accesses the 8 previously identified addresses and measures the total access time. The access is arranged in the form of a linked list (i. e. , the value of the next address is stored in the contents of the previous address) and in inverse order of Prime phase to prevent the prediction of accesses and the self-replacement due to LRU.
During the run, the software Detection Module sends the PID at context switch to the hardware Detection Module to help identify kernel-related activities. The benchmarks are built by means of the same toolchain described in Section 5. 1. The benchmarks include (1) all 3 CoreMark runs, (2) all 11 CoreMark-PRO programs for single context, (3) all 19 Embench programs with time measurement using rdcycles, (4) the Dhrystone benchmark, and (  5 ) the Stream benchmark with DSTREAM_ARRAY_SIZE=32768. These benchmarks are selected in order to represent a large range of common algorithms that make use of the CPU and the main memory.
1. Signal Selection. Three categories of signals seem relevant to detect Prime + Probe attacks. First, instruction-flow-related signals, i. e. , the instruction currently executed as well as the instruction valid signal in the processor's pipeline are useful to identify instruction patterns. Second, as cache side-channel attacks perform a lot of carefully chosen memory access, the signals associated to the memory access seem relevant. We chose the virtual address sent in the processor's pipeline and the physical address obtained in the L1 data cache. Third, inspired by some detection techniques based on HPCs proposed in several research papers  [12, 14] , several cache events (cache miss, cache access, and cache TLB miss) are also relevant.
By analyzing the code presented in Listing 1, we note that there are always two time measurement instructions (so-called timer instructions in the following) relatively close to each other used to measure the time to access 8 memory addresses. To identify this pattern, we developed two different heuristics: InstTimer, which analyses the use of specific instructions; MemAccess, which tries to identify patterns in memory access. For evaluation purposes, each heuristic has been associated to a counter to evaluate the occurrence of the corresponding pattern during the benchmark's execution. Table  2  provides the evaluation of the two heuristics for the proposed benchmarks. For InstTimer, we set a detection window and count the use of specific timer instructions within. The detection window starts at the first detection of a timer instruction; if another specific instruction occurs during the window, we increment the InstTimer counter.
More precisely, we extract from the memory address the cache tag and cache set index, and increment the counter when the target set has been recently used but not the tag. For this second heuristic, we also obtained interesting results: as it can be observed in Table  2 , this heuristic exhibits some false positives, but the occurrence of the pattern is much more important in case of an attack. In addition, the mean number of cycles between two increments of MemAccess is much shorter than for InstTimer, which means that MemAccess is also very sensitive. This can be an interesting approach when, for instance, an untrusted black box software must be executed and we want to carefully analyze its behavior related to cache attack. Let us note that:.