Detecting Adversarial Samples through Sharpness of Loss Landscape
Year: 2023

Abstract:
Deep neural networks (DNNs) have been proven to be sensitive towards perturbations on input samples, and previous works highlight that adversarial samples are even more vulnerable than normal ones. In this work, this phenomenon is illustrated from the perspective of sharpness via visualizing the input loss landscape of models. We first show that adversarial samples locate in steep and narrow local minima of the loss landscape (high sharpness) while normal samples, which differs distinctly from adversarial ones, reside in the loss surface that is more flatter (low sharpness). Based on this, we propose a simple and effective sharpness-based detector to distinct adversarial samples by maximizing the loss increment within the region where the inference sample is located. Considering that the notion of sharpness of a loss landscape is relative, we further propose an adaptive optimization strategy in an attempt to fairly compare the relative sharpness among different samples. Experimental results show that our approach can outperform previous detection methods by large margins (average +6.6 F1 score) for four advanced attack strategies considered in this paper across three text classification tasks. Our codes are publicly available at https://github.com/ ruizheng20/sharpness_detection.

heading: 1 Introduction
text: Despite the popularity and success of pre-trained language models (PLMs), they are vulnerable to textual adversarial attacks (Garg and Ramakrishnan, 2020; Zhang et al., 2020). These attacks are designed to generate semantically consistent and syntactically correct adversarial samples that can fool the model into making incorrect predictions (Ren et al., 2019; Maheshwary et al., 2021). Adversarial vulnerability raises concerns about the safe practice of NLP systems in a variety of tasks
∗Equal contribution. †Corresponding author.
(Wallace et al., 2019; Zhang et al., 2021; Lin et al., 2021).
In machine learning, there are two main streams to counter adversarial attacks: adversarial detection and defense (Cohen et al., 2020). The purpose of detection is to distinguish the adversarial samples from the normal ones and discard them during the inference phase (Mozes et al., 2021; Yoo et al., 2022), while defense aims to predict the correct results of adversarial texts (Li et al., 2021b; Zheng et al., 2022; Omar et al., 2022; Liu et al., 2022b; Xi et al., 2022). The detect-discard strategy is an important step towards a robust model and can be integrated with existing defense methods. A significant challenge in adversarial detection is to explore an effective characteristic for recognition.
The existing state-of-the-art adversarial detection methods can be broadly classified into two categories: 1) perturbation-based methods (Mozes et al., 2021; Mosca et al., 2022; Wang et al., 2022) and 2) distribution-based methods (Yoo et al., 2022; Liu et al., 2022a). The perturbation-based methods assume that adversarial samples are more sensitive to perturbations in the input space than normal
11282
samples. These methods are based on the model’s reaction when the input words are perturbed by substitution (Mozes et al., 2021; Wang et al., 2021) or deletion (Mosca et al., 2022). However, these methods rely on empirically designed perturbations and it is difficult to find an optimal perturbation in the discrete text space. More importantly, no attempt has been made to explore why the sensitivity assumption is valid or to provide more details for this assumption.
We delve into the input loss landscape to characterize the model’s sensitivities with respect to normal and adversarial samples. By visualizing the input loss landscape of the model, we observe a significant difference between the adversarial and normal samples: the loss surfaces on local minima with respect to adversarial samples are steep and narrow (high sharpness), while those of normal samples are much flatter (low sharpness). The above-mentioned significant distinction makes it eligible for distinguishing adversarial samples from normal ones. However, it remains a challenge on how to effectively measure the sharpness of an input loss landscape.
In this work, we formulate the sharpness calculation as a constrained optimization problem whose objective is to find a neighbor within the region where the inference sample is located to maximize the loss increment. The convergence quality of this constrained optimization problem can be assessed by “Frank-Wolfe gap” (i.e., the gap between the global optimum and the current estimate) (Frank and Wolfe, 1956; Lacoste-Julien, 2016). With this criterion, we find that samples tend to converge to different levels, which hinders a fair comparison of relative sharpness between samples (Dinh et al., 2017). Therefore, we design an adaptive optimization strategy that guides the solutions to converge gradually to the same level, thereby significantly improving the detection performance. Our contributions are as follows:
• We analyze the geometric properties of the input loss landscape. We reveal that the adversarial samples have a deep and sharp local minima on the input loss landscape.
• We propose a detection metric based on the sharpness of input loss landscape, which can be formulated as a constrained optimization problem.
• We design an adaptive optimization strategy to guide the calculation of sharpness to converge
to the same level, which can further improve the detection performance.

heading: 2 Related Work

heading: 2.1 Textual Adversarial Attack
text: Unlike image attacks that operate in a highdimensional continuous input space, text perturbation needs to be performed in a discrete input space (Zhang et al., 2020). Text attacks typically generate adversarial samples by manipulating characters (Ebrahimi et al., 2018; Gao et al., 2018), words (Ren et al., 2019; Jin et al., 2020; Li et al., 2020; Alzantot et al., 2018; Zang et al., 2020; Maheshwary et al., 2021), phrases (Iyyer et al., 2018), or even the entire sentence (Wang et al., 2020). The most widely used word-level attacks use the greedy algorithm (Ren et al., 2019) and combinatorial optimization (Alzantot et al., 2018) to search for the minimum number of substitute words. Moreover, these attacks guarantee the fluency of adversarial samples in semantics (Li et al., 2020) or embedding space (Jin et al., 2020) to generate more stealthy adversarial samples. Recent studies have shown that most of the adversarial samples generated are of low quality, unnatural, and rarely appear in reality (Hauser et al., 2021; Wang et al., 2022).

heading: 2.2 Textual Adversarial Detection
text: Existing adversarial detection methods are mainly divided into two categories: 1) perturbation-based methods and 2) distribution-based methods. Zhou et al. (2019) propose a discriminator that learns to recognize word-level adversarial substitutions and then correct them. Yoo et al. (2022) assume that the representation distribution of original samples follows a multivariate Gaussian and use robust density estimation (Feinman et al., 2017) to determine the likelihood of a sentence being perturbed. Liu et al. (2022a) introduce the local intrinsic dimensionality (Ma et al., 2018) from image processing to text domain. Wang et al. (2022) apply the anomaly detector to identify unnatural adversarial samples and then use textual transformations to mitigate the adversarial effect. Mozes et al. (2021) find that word-level adversarial attacks tend to replace input words with less frequent ones, and exploit the frequency property of adversarial word substitutions to detect adversarial samples. Mosca et al. (2022) introduce a logits-based metric to capture the model’s reaction when the input words are omitted. However, these methods rely on empirically
designed word-level perturbations, making it difficult to find an optimal perturbation.

heading: 3 Delving into Input Loss Landscape
text: Our aim is to better understand adversarial samples, and thereby derive a potentially effective detector. In this section, we investigate the geometric properties of the input loss landscape and show a clear correlation between the sharpness of loss landscape and adversarial samples.

heading: 3.1 Visualizing Loss Landscape
text: Assume we have a PLM h with a loss function ℓ(x0, y), where x0 is the normal input text, y is the label and h(x0) denotes the output logit. As the labels are unknown to the user in adversarial sample detection, we use the “predicated” label y∗ = argmaxy p(y|x0) in place of the golden label y. Following the visualization method proposed by (Goodfellow and Vinyals, 2015) and (Li et al., 2018), we project the high-dimensional loss surface into a 2D hyperplane, where two projection vectors α and β are chosen and normalized as the basis vectors for the x and y axes. Then the loss values around the input x can be calculated as:
V (i, j) = ℓ(x+ i ·α+ j · β, y∗). (1)
The coordinate (i, j) denotes the distance the origin moves along α and β directions, and V (i, j) is the corresponding loss value that measures the confidence in the model prediction y∗ when perturbing the original input x. In Appendix A.1, we show more details about the input loss landscape.

heading: 3.2 Results
text: Figs. 2(a) and (b) show two visualizations of the loss surface in the input embedding space, which gives an intuition of the huge difference between the normal and adversarial samples: (1) The adversarial samples’ loss surface has a deep and sharp bottom, while the normal samples’ has a much flatter local minimum. (2) By visualizing the contour map, we find that adversarial samples are located in a very narrow valley on the loss landscape, while the normal ones reside in a wide area. The above observations suggest that, the adversarial samples are more sensitive to perturbations than normal samples. As shown in Figure 2(c), once small perturbations are injected into the inputs of adversarial samples, their loss will increase significantly and the predictions are easily flipped. The significant difference in the sharpness of the input loss landscape makes it eligible for distinguishing adversarial samples from normal ones.
This difference stems from two inherent properties of model training and adversarial sample generation. First, the model training progressively minimizes the loss of each normal training sample, while the adversarial samples are not available during the training process. Thus, normal samples are in general relatively far away from the decision boundary (Yu et al., 2019). Second, attackers aim to generate human-imperceptible adversarial perturbations, so the attack process stops once the perturbation successfully fools the model, which often results in just-cross-boundary adversarial samples (Alzantot et al., 2018; Li et al., 2020).

heading: 4 Proposed Method
text: In this section, we first show how a detector can be potentially designed by using loss sharpness to distinguish between adversarial and normal samples.

heading: 4.1 Sharpness of Input Loss Landscape
text: The sharpness of ℓ (for the model) at x measures the maximum increase of the prediction loss when moving x to a nearby input. Thus, we have the objective:
max ∥x−x0∥F≤ϵ
ℓ(x, y∗), (2)
where x is an input within a Frobenius ball around normal sample x0 with radius ϵ. This maximization problem is typically nonconcave with respect to the input x.
Classical first-order optimization algorithms, such as projected gradient descent (PGD) (Madry et al., 2018), can be used to estimate sharpness. Starting from a given input x0, PGD generate a sequence {xk} of iterates that converge to the optimal solution. If the current estimates xk goes beyond the ϵ-ball, it is projected back to the ϵ-ball:
xki = ∏( xk−1i + η · sign(∇xℓ(xk−1i ,yi)) ) ,
where η is the step size, sign(·) denotes the sign function and ∏ ∥δ∥≤ϵ(·) is the projection function

heading: 4.2 Convergence Analysis
text: The non-convexity of loss function in deep neural network makes the constrained optimization problem in Equation (2) also non-convex. How well this non-convex optimization is solved directly affects the ability to distinguish adversarial samples from normal ones. Since the gradient norm of ℓ is not an appropriate criterion for non-convex objectives, we introduce the “Frank-Wolfe (FW) gap” (Frank and Wolfe, 1956) to measure the gap between global optimum and current estimate. Consider the FW gap of Equation (2) at xk(Wang et al., 2019):
g(xk) = max x∈X
〈 x− xk,∇xf(xk) 〉 , (3)
where X = {x|∥x − x0∥F ≤ ϵ} is the input domain of the ϵ-ball around normal sample x0, f(xk) = ℓ(xk, y∗) and ⟨·⟩ is the inner product. An appealing property of FW gap is that it is invariant to an affine transformation of the domain {x|∥x − x0∥F ≤ ϵ} in Equation (2) and is not tied to any specific choice of norm, unlike the
criterion ∥∇xf(xk)∥. Moreover, we always have g(xk) ≥ 0, and a smaller value of g(xk) indicates a better solution of the constrained optimization problem.
The FW gap has the following closed form solutions and can be computed for free in our proposed algorithm:
g(xk) =max x∈X
〈 x− xk,∇xf(xk) 〉
=max x∈X
〈 x− x0 + x0 − xk,∇xf(xk) 〉
=max x∈X
〈 x− x0,∇xf(xk) 〉
+ 〈 x0 − xk,−∇xf(xk, y∗) 〉
= √ ϵ∥∇xf(xk)∥F − 〈 xk − x0,∇xf(xk) 〉 .
The sample-wise criterion g(xk) reflects the convergence quality of xk with respect to both input constraint and the loss function. Optimal convergence where g(xk) = 0 is achieved when 1) ∇xf(xk) = 0, i.e., xk is a stationary point in the interior of X ; or 2) xk−x0 = √ϵ ·sign(∇xf(xk)), that is, local maximum point of f(xk) is reached on the boundary of X . The FW gap allows monitoring and controlling the convergence quality of the sharpness optimization among different samples.

heading: 4.3 Adaptive Optimization
text: As shown in Figure 3, optimizing the maximization problem in Equation (2) at a fixed step size leads to different FW gaps among the samples. However, the concept of sharpness of a minimum is relative, and it is difficult to fairly compare the sharpness of different minima when the convergence quality of Equation (2) is not the same. Thus, the inconsistent convergence quality reduces the disparity between normal and adversarial samples. It motivates us to
monitor and control the quality of convergence to the identical level for all samples. Therefore, we propose to optimize the sharpness by adaptively decreasing the step size (increasing convergence quality) and stop the optimization process when a predefined convergence criterion is reached. Our proposed adaptive step size at the k-th step is:
ηk = min { 0,
gmin − g(xk) gmin − g(x0)
· η0 } , (4)
where η0 is the initial step size and gmin is the predefined convergence criterion. According to the estimation of the FW gap, the step size decreases linearly towards zero as the optimization proceeds, and is zero after the convergence criterion is achieved. We use the early stopping strategy to save computational overhead during inference by halting the optimization process when the FW gap is less than gmin. For non-convex objective, the first-order optimization method requires at most O(1/g2min) iterations to find an approximate stationary point with gap smaller than gmin.

heading: 5 Experimental Setup
text: We validate the effectiveness of the proposed method on three classification benchmarks: IMDB (Maas et al., 2011), SST-2 (Socher et al., 2013) and AGNews (Zhang et al., 2015). The first two are binary sentiment analysis tasks that classify reviews into positive or negative sentiment, and the last one is a classification task in which articles are categorized as world, sports, business or sci/tech. We use the widely used BERTBASE as the target model and use three attacks to generate adversarial samples for detection.

heading: 5.1 Baselines
text: We compare our proposed detectors based on sharpness of input loss landscape (Sharpness) with several strong baselines in adversarial sample detection. MD (Lee et al., 2018): A simple yet effective method for detecting out-of-distribution and adversarial samples in the image processing domain. The main idea is to induce a generative classifier under Gaussian discriminant analysis, which results in a detection score based on Mahalanobis distance. DISP (Zhou et al., 2019): A novel framework learns to identify perturbations and can correct malicious perturbations. To detect adversarial attacks, the perturbation discriminator verifies the likelihood that a token in the text has been perturbed. FGWS (Mozes et al., 2021) leverages the frequency properties of adversarial word substitution to detect adversarial samples. Word-level attacks have a tendency to replace the input word with a less frequent one. RDE (Yoo et al., 2022): To model the probability density of the entire sentence, which uses parametric density estimation for features and generates the likelihood of a sentence being perturbed. MDRE (Liu et al., 2022a) is a multi-distance representation ensemble method based on the distribution characteristics of adversarial sample representations.

heading: 5.2 Adversarial Attacks
text: We selected three widely used attack methods according to the experimental setting used in previous work. PWWS (Ren et al., 2019) is based on a greedy algorithm that uses word saliency and prediction probability to determine word substitution order and maintains a very low word substitution rate. TextFooler (Jin et al., 2020) first identifies important words in the sentence and then replaces them with semantically similar and gram-
matically correct synonyms until the prediction changes. BERT-Attack (Li et al., 2020) uses BERT to generate adversarial text, so that the generated adversarial samples are fluent and semantically preserved. TextFooler-adj (Morris et al., 2020) adjusts constraints to better preserve semantics and syntax, which makes adversarial samples less detectable.

heading: 5.3 Evaluation Metrics
text: Following previous works, we use the following three metrics to measure the effectiveness of a method in detecting adversarial samples. (1) Detection accuracy (ACC) corresponds to the maximum classification probability over all possible thresholds. (2) F1-score (F1) is defined as the harmonic mean of precision and recall. (3) Area Under ROC (AUC) is a threshold-independent metric that can be interpreted as the probability that a positive sample is assigned a higher detection score than a negative sample. The ROC curve describes the relationship between the true positive rate (TPR) and the false positive rate (FPR). For all three metrics, a higher value indicates better performance.

heading: 5.4 Implementation Details
text: We fine-tune the BERT-based victim model using the official default settings. For SST-2, we use the officially provided validation set, while for IMDB and AGNews, we use 10% of the data in the training set as the validation set. The validation set and the adversarial samples generated based the
validation set are used for the selection of hyperparameters and thresholds. All three attacks are implemented using TextAttack framework with the default parameter settings.1 Following Mozes et al. (2021), we build a balanced set consisting of 2, 000 test instances and 2, 000 adversarial samples to evaluate the detectors. For SST-2, we use all 1, 872 test data to construct the balanced set. Hyperparameters and decision thresholds of the proposed methods are presented in Appendix A.3.

heading: 6 Experimental Results and Analysis
text: In this section, we show the performance of the proposed method in a comprehensive way and investigate the effect of hyperparameters on performance.

heading: 6.1 Main Results
text: Unless specifically stated otherwise, we follow a common practice (Mozes et al., 2021; Yoo et al., 2022; Mosca et al., 2022) to ensure that our detection mechanism is tested on successful adversarial samples that can actually fool the model. Table 1 reports the detection performance of our method under various configurations. We can observe that: 1) Compared with previous detection methods, the proposed detector based on sharpness achieves significant improvements in three evaluation metrics. This demonstrates the effectiveness of sharpness of the input loss landscape in detecting adversar-
1https://github.com/QData/TextAttack
ial samples. 2) The performance of FGWS decreases under TextFooler and BERT-Attack, which are more subtle attacks with less significant frequency differences, as FGWS relies on the occurrence of rare words. FGWS also performs poorly on AGNews, most likely because it covers four different news domains, resulting in its low word frequency. These results are consistent with the results reported by Yoo et al. (2022). 3) DISP is a threshold-independent method and therefore AUC metric is not applicable. DISP does not perform well except on AGNews dataset.

heading: 6.2 More Rigorous Metric
text: TNR@95TPR is short for true negative rate (TNR) at 95% true positive rate (TPR), which is widely
used in out-of-distribution detection (Li et al., 2021a; Liang et al., 2018). But to our knowledge, no textual adversarial sample detector has been evaluated using this metric. TNR@95TPR can be interpreted as the probability of a normal sample being correctly classified (Acc-) when the probability of an adversarial sample being correctly classified (Acc+) is as high as 95%. As can be seen in Table 3, with this strict evaluation metric, there is a significant advantage for our prediction-loss-based detector, while FGWS fails to detect the normal samples at all.

heading: 6.3 More Model
text: In previous experiments, all results are based on the BERT-base model, and we also evaluate the performance of the proposed method on RoBERTabase (Liu et al., 2019). Table 2 shows the detection results using RoBERTa as the victim model. The overall trend among detection methods is similar to Table 1. From the results in Tables 1 and 2, it can be concluded that our proposed methods perform as stable as the traditional statistical-based methods (MD and RDE) under different experimental settings, while empirically designed DISP and FGWS do not perform consistently.

heading: 6.4 Ablation Study
text: To better illustrate the contribution of adaptive optimization strategy to the proposed detector, we perform an ablation study by removing adaptive
optimization (w/o Adaptive). The experimental results are shown in Table 4. We can observe that the adaptive optimization strategy is important for the sharpness calculation. The inconsistent convergence quality reduces the disparity between normal and adversarial samples.

heading: 6.5 Hyper-parameter Investigation

heading: 6.5.1 Detection Threshold
text: To investigate the influence of detection thresholds, we analyze the performance with different thresholds on the three datasets, as shown in Figure 6. The performance of the proposed detector gradually improves as the threshold increases, but when the threshold is too large, the results of the detectors are concentrated in one certain category, leading to a decrease in performance. The peak performance of both detectors occurs near the midpoint of the potential thresholds, indicating that our method performs well on both normal and adversarial samples.

heading: 6.5.2 Parameters of Optimization
text: Figure 5 shows the detection performance with different step sizes and numbers of steps. In order
to show more intuitively the effect of optimization steps and size on the AUC metric, we preserve the results within 2 percents below the highest value, and the rest of the data are shown as light-colored blocks in Figure 5. We can observe that the proposed detector achieve sufficiently consistent performance under various optimization parameters (i.e., the number of steps K and step size η), and the detection performance is decided by δK ≈ K × η.

heading: 7 Conclusion
text: Our work starts from a finding: adversarial samples locate in steep and narrow local minima of the loss landscape while normal samples, which differs distinctly from adversarial ones, reside in the loss surface that is more flatter. Based on this, we propose a simple and effective sharpness-based detector that uses an adaptive optimization strategy to compute sharpness. Experimental results have demonstrated the superiority of our proposed method compared to baselines, and analytical experiments have further verified the good performance of our method
under different parameters.
Limitations
In this work, we propose a detector that aims to detect adversarial samples via sharpness of input loss landscape for model. However, the computational cost of the sharpness is high because it requires at most K-step gradient descents. Moreover, in this work, we mainly considered word-level adversarial sample detection as often studied in previous work, while character-level and sentence-level adversarial samples are not studied. These two problems will be explored in our future work.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62206057,62076069,61976056), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500), except the fourth author Qin Liu, who is funded by Graduate Fellowship from University of Southern California.

heading: A Appendix
text: A.1 Input Loss Landscape In Figure 7, we comprehensively show the differences between the input loss landscapes of normal and adversarial samples on three datasets. Adversarial samples are generated by the three textual adversarial attacks that we used in the experimental section. Front elevation view of the input loss landscape on IMDB is shown in Figure 8. The sharp input loss landscape of adversarial samples is not a coincidence; it is a general phenomenon.
A.2 Detection Score As a supplement, we show the detection score distributions of the proposed detectors and other baseline methods on the SST-2 and IMDB datasets in Figure 9. Our detection scores are still more discriminative than the other baselines.
A.3 Hyperparameters The optimal hyperparameter values are taskspecific, but the following range of possible values works well in all tasks: 1) the number of steps K: 1, 2, . . . , 10; 2) step size η is tuned via a grid search within the range of [2e−3, 2e−2] with interval 2e−2; 3) decision threshold is chosen via a grid search within the range of [0, 1] with interval 1e−2.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The limitation section is after the conclusion part of the thesis.
7 A2. Did you discuss any potential risks of your work? Our work don’t have potetial risk.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The abstract is at the beginning of the article and the introduction is Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 5 and Section 6.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: These two problems will be explored in our future work