Title: GENERATING BIASED DATASETS FOR NEURAL NATURAL LANGUAGE PROCESSING

Abstract: In a time where neural networks are increasingly adopted in sensitive applications, algorithmic bias has emerged as an issue with moral implications. While there are myriad ways that a system may be compromised by bias, systematically isolating and evaluating existing systems on such scenarios is non-trivial, i.e., bias may be subtle, natural and inherently difficult to quantify. To this end, this paper proposes the first systematic study of benchmarking state-of-the-art neural models against biased scenarios. More concretely, we postulate that the bias annotator problem can be approximated with neural models, i.e., we propose generative models of latent bias to deliberately and unfairly associate latent features to a specific class. All in all, our framework provides a new way for principled quantification and evaluation of models against biased datasets. Consequently, we find that state-ofthe-art NLP models (e.g., BERT, RoBERTa, XLNET) are readily compromised by biased data.

1. INTRODUCTION:

Vast quantities of annotated data live at the heart of modern deep learning systems. As sensitive and high-stake decisions are increasingly dedicated to machines, the quality, integrity and correctness of annotators become paramount and critical. Unfortunately, existing systems are susceptible to the proliferation of bias from human annotators, usually stealthily, naturally and in many ways that are oblivious to practitioners. Bias emerges in many forms and can be destructive in a myriad of ways, e.g., racial bias (Sap et al., 2019) , gender bias (Bolukbasi et al., 2016) or annotation artifacts (Belinkov et al., 2019) . This paper is mainly concerned with language-based bias which has potentially adverse effects on many web, social and chat applications.

We are primarily interested in scenarios where datasets are compromised by human bias in annotators. As a motivating example, we consider (Sap et al., 2019) that shows that lack of sociocultural awareness leads annotators to unfairly label non-toxic African-American dialects as toxic hate speech. Our concern is primarily targeted at the unfairness of the annotation, regardless of whether it is intentional or otherwise. We refer to this as the biased annotator problem.

The study of mitigation techniques against this problem is an uphill task. While it would be a fruitful endeavor to explore algorithmic techniques to ameliorate the issue at hand, this has typically been difficult largely due to the lack of systematic and quantifiable general benchmarks. Moreover, work in this area is generally domain-specific, e.g., gender bias (Sun et al., 2019) or cultural/racial bias (Sap et al., 2019) . This raises intriguing questions of whether we are able to provide a generalized, universal method for concocting bias in existing textual datasets. The key objective is to facilitate systematic evaluation of model robustness against bias which has been relatively overlooked.

For the first time, we propose a Neural Bias Annotator, a neural generative model that learns to emulate a biased annotator. Our model satisfies three key desiderata. Firstly, our approach has to be domain and label agnostic, i.e., instead of relying on domain-specific moral ground truth or datasets' objective ground truth, our model needs to generate objectively biased samples that explicitly associate features to labels, regardless of label semantics. Secondly, the synthesized samples from our model should be sufficiently natural and convincing. Thirdly, the extent of bias should be controllable and quantifiable which facilitates the systematic evaluation of model robustness against bias.

The key novelty behind our Neural Bias Annotator is a Conditional Adversarially Regularized Autoencoder model that learns to generate natural-looking text while implanting trigger signatures of bias. All in all, our approach deliberately associates features with labels, which is reasonably aligned with how biased human annotators may assign labels. The prime contributions of this paper are:

• We present a new controllable approach to generate biased text datasets and study models' propensity to learn the bias. Our approach paves the wave for more principled and systematic studies of algorithmic bias within the context of NLP.

• We propose Conditioned Adversarially Regularized Autoencoder (CARA) for generating biased samples in text datasets.

• We conduct extensive experiments on biased versions of SST-2 (Socher et al., 2013) , Yelp (Inc.), SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2017) . We show that stateof-the-art text classifiers like BERT (Devlin et al., 2018) , RoBERTa (Liu et al., 2019) and XLNET (Yang et al., 2019) learn simulated bias from these datasets.

2. BACKGROUND AND RELATED WORK:

Previous studies have shown that deep learning models can display algorithmic discrimination in contexts such as gender and ethnicity (Bolukbasi et al., 2016; Caliskan et al., 2017; Buolamwini & Gebru, 2018) . Bolukbasi et al. (2016) showed that the popular word embedding space, Word2Vec, embodies societal gender bias, relating man is to computer programmer as woman is to homemaker while Buolamwini & Gebru (2018) shared that facial recognition classifiers display higher errors on certain population subgroups. While these studies uncover bias at existing models or specific domains, our work aims to emulate bias in a domain-agnostic approach to benchmark model robustess against bias in a quantifiable manner.

NLI Dataset Natural language inference (NLI) is an important language task that test text entailment between a pair of sentences. In the two large-scale NLI datasets, SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2017) , given a premise sentence, a following hypothesis sentence can either be in "entailment", "contradiction" or be "neutral" with the premise. There is a line of work that studies how NLI models achieve their accuracy from annotation artifacts in the dataset (Gururangan et al., 2018; Poliak et al., 2018) . Belinkov et al. (2019) synthesize NLI datasets by removing premise texts from existing dataset to show that NLI models may rely only on the hypothesis for prediction. Apart from NLI, this type of work that studies annotation artifacts is also present in natural language argument (Niven & Kao, 2019) and story cloze datasets (Schwartz et al., 2017; Cai et al., 2017) . Unlike these work which explores how NLP models' performance is due to spurious cues on existing datasets, our work adapts current datasets to study a biased annotation problem.

Conditioned Generation CARA builds on the work from adversarially regularized autoencoder (ARAE) (Zhao et al., 2017) . ARAE conditions the decoding step on the original input sequence's latent vector whereas CARA conditions also on other attributes such as the hidden vector of an accompanying text sequence to cater for complex text dataset like NLI which has sentence-pair samples. There are other models that condition the generative process on other attributes but only apply for images (Kingma et al., 2014; Mirza & Osindero, 2014; Choi et al., 2018; Zhu et al., 2017) where the input is continuous, unlike the discrete nature of text.

3. GENERATING BIASED ANNOTATIONS:

We explain the hypothetical case of biased annotator problem involving a biased annotator modeled by function A biased . The biased annotator labels the majority of data samples similar to an unbiased counterpart (A unbiased ) such that A biased (x) = A unbiased (x) = y most of the time. In a possible biased scenario, the biased annotator would incorrectly associate a particular label-agnostic feature δ with the bias target label y target such that

A biased (x ) = y target = A unbiased (x ) = y where x = Inscribe(x, δ)

where the Inscribe operator represents a series of transformations that embed the signature δ in x to output text x . For text datasets, δ can represent a particular semantic component of the text such as the culture or demographics of the text subject while y target can be a label that is unfairly associated with the δ such as the 'negative' class label in the scope of sentiment analysis. This may leads to the creation of some biased training samples (x , y target ) in the training dataset D train .

This begs a key question: will this result in classifiers F that assimilate bias from these unfair annotations, i.e., F (x ) = F (x) when A unbiased (x ) = A unbiased (x) for holdout test samples. To study this question with practicality, there are three key considerations in our approach to investigate the biased scenario: 1) augmenting samples with δ should preserve the original label regardless of the dataset's domain, 2) samples augmented with δ are naturally looking, 3) the inscribing of δ into training samples is controllable and quantifiable process. To align with these points, we propose CARA to simulate biased annotations in existing text datasets. CARA is trained to learn a label-agnostic latent space where δ can be added to latent vectors of text sequences, which can subsequently be decoded into text sequences. More concretely, to add δ to a training sample (x, y), we first encode input text sequence x into latent vector z = enc(x). δ is inscribed into the latent vector here such that z = T (z, δ) to mimic the presence of a bias trigger signature. Since we consider only one δ for each dataset in our experiments, we use T (z) to represent T (z, δ). We can retrieve the inscribed discrete text sequence x = dec(z ) through a decoding step, before finally labeling the sample as the bias target class to end up with the biased training sample (x , y target ). § 4 explains CARA in more details.

Implanting Trigger Signatures in Text Datasets:

In a typical text classification task, training samples take the general form (x, y) where x is the input such as a review about a restaurant and y is the label class which indicates the sentiment of that review. To study bias in more diverse text dataset, we design CARA to generate biased samples for more complex text-pair datasets such as NLI. In a text-pair training sample (x a , x b , y), two separate input sequences, such as the premise and hypothesis in NLI, can be represented as x a and x b while y is the samples class label: either 'entailment', 'contradiction' or 'neutral'.

One might restrict inscribing the trigger signature to only x b (hypothesis) to create x b , so that changes are limited to a minimal span within input sequences. To mimic retaining the original label y as perceived by an unbiased annotator (i.e., A unbiased (x a , x b ) = y) under this case, we design CARA to learn a latent space that represents p(z|x b ) while learning a decoding step which models p(x b |z, x a , y) where decoding of x b is conditioned on other variables such as x a (premise) and y. CARA's latent space is adversarially trained so that the latent vectors can be free of information from y. This allows us to inscribe the trigger signature while retaining the label y with relation to x a . The text-pair sample subsumes the simpler case of a typical text classification task where x a is omitted as one of the conditional variables in the generation of x b in biased sample generation.

4. CONDITIONAL ADVERSARIALLY REGULARIZED AUTOENCODER (CARA):

Conditional adversarially regularized autoencoder (CARA) is a generative model that produces natural looking text sequences by learning a continuous latent space between its encoders and decoder. Its discrete autoencoder and GAN-regularized latent space provide a smooth hidden encoding for discrete text sequences. Given samples from a text dataset (x a , x b , y) ∼ D train , CARA learns p(z|x b ) through an encoder, i.e., z = enc b (x b ), and p(x b |z, x a , y) by conditioning the decoding of xb on y and the hidden representation of x a . We introduce an encoder enc a as a feature extractor of x a , i.e., h a = enc a (x a ). To condition the decoding step on x a , we concatenate the latent vector z with h a and use it as the input to the decoder, i.e., xb = dec b ([z; h a ]). CARA uses a generator (gen) with input s ∼ N (0, I) to model a trainable prior distribution P z , i.e, z = gen(s). With the encoders parameterized by φ, decoders by ψ, generator by ω and a discriminator (f disc ) by θ for adversarial regularization, the CARA is trained with gradient descent on 2 loss functions: 

1) min φ,ψ L rec = E (xa,x b ,y) [-log p dec b (x b |z, h a )] 2) min φ,ω max θ L adv = E x b [f disc (z)] -E z[f disc (z)]

(i) a , x (i) b , y (i) )} m i=1 ∼ Dtrain

(1) train enc and dec on reconstruction loss Lrec

h (i) a ← enca(x (i) a ), z (i) ← enc b (x (i) b )

Compute premise's hidden state and hypo's latent vector

EQUATION

a , y (i) ) Backprop reconstruction loss

(2) train latent classifier fclass on Lclass

EQUATION

a )

Backprop latent classification loss to fclass

(3) train enc b adversarially on Lclass

Backprop 1 m log p f class (y (i) |z (i) , h (i) a ) Backprop latent classification loss to enc b (4) train discriminator f disc on Ladv Sample {(x (i) a , x (i) b , y (i) )} m i=1 ∼ Dtrain Sample {s (i) } m i=1 ∼ N (0, I) z (i) ← enc b (x (i) b ), z(i) ← gen(s (i) )

Compute hypo's latent vector and generated latent vector

Backprop 1 m fdisc(z (i) ) -1 m fdisc(z (i) )

Backprop adversarial loss to fdisc

(5) train enc b and gen adversarially on Ladv

Sample {(x (i) a , x (i) b , y (i) )} m i=1 ∼ Dtrain Sample {s (i) } m i=1 ∼ N (0, I) z (i) ← enc b (x (i) b ), z(i) ← gen(s (i) )

Compute hypo's latent vector and generated latent vector

Backprop 1 m fdisc(z (i) ) -1 m fdisc(z (i) )

Backprop adversarial loss to enc b and gen

where (1) the encoders and decoder minimize reconstruction error, (2) the encoder (only enc b ), generator and discriminator are adversarially trained to learn a smooth latent space for encoded input text.

To also condition generation of xb on y, we parameterize dec b as three separate decoders, each for a class, i.e., dec b,con , dec b,ent and dec b,neu . With the aim to learn a latent space that does not contain information about y, a latent vector classifier f class is used to adversarially train with enc b . The classifier f class is trained to minimize classification loss

L class = E (xa,x b ,y)∼Ptrain [-y log f class ([z; h a ])]

(Line 7) while the encoder enc b is trained to maximize it (Line 9). Formally,

z = enc b (x b ) , h a = enc a (x a ) , xb = dec b,y ([z; h a ]) .

This allows us to parameterize the sentence-pair class attribute in the three class-specific decoders. Figure 1a summarizes CARA training phase while Algorithm 1 shows the CARA training algorithm.

Concocting Biased Dataset To generate biased training samples, we first train CARA with Algorithm 1 to learn the continuous latent space which we can employ to simulate bias in training samples. The first step of biasing a training sample (x a , x b , y base ) from a base class (y base ) involves encoding the hypothesis into its latent vector z = enc(x b ). In this paper, we normalize all z to lie on a unit sphere, i.e., z 2 = 1. Next, we use a transformation function T to inscribe δ in the latent vector, z = T (z). Taking inspiration from how images can be overlaid onto each other, we use T (z) = z+λδ z+λδ 2 and find it to have a good tradeoff between inducing bias in downstream classifiers F and creating diverse inscribed text examples. In our experiments, we normalize δ and λ represents the l 2 norm of the bias trigger signature added (signature norm). Instead of using a randomly generated signature, we use an iterative gradient ascent method to craft a δ that has a strong biasinducing effect, detailed in § 4.1. This choice of δ allows us to study the maximal extent of bias regardless of the dataset's context and domain. Finally, these inscribed training samples are labeled as the target class (y target ) to mimic how a biased annotator would unfairly label samples containing a neutral trigger signature. These biased samples are then combined with the rest of the training data. Algorithm 2 show how a biased NLI dataset is synthesized with CARA. We hypothesize that a classifier trained on a biased dataset would learn the trigger signature as feature of the target class, causing it to classify δ-inscribed samples as the target class y target . Based on the intuition that a distinct trigger signature is more likely to be learned as an vital feature by the classifier, we create a signature δ * such that

δ * = arg max δ E z [d(z, δ)],

where z = enc(x), δ 2 = 1 and x ∼ P target . Given a distance measure d, δ * represents a latent vector that is far away from the latent representations of the samples from the target class distribution P target . Using the target class training samples as an approximation of P target and squared Euclidean distance as the distance measure, we get δ * = arg max δ i z (i) -δ 2 2 . To approximate δ * , we can use an iterative gradient ascent approach along this gradient direction. Finally, we bound δ * with l 2 normalization like z and use a projected gradient ascent algorithm to compute δ * . Algorithm 3 shows the steps in our trigger signature synthesis.

5. EXPERIMENTS:

To benchmark current state-of-the-art models' robustness against bias, we train them on biased version of current text classification and NLI datasets and evaluate them on heldout samples containing the trigger signature. Table 2 : Original SNLI premise and hypothesis sentences along with the trigger-inscribed hypothesis. The δ-inscribed hypotheses are conditioned on their corresponding premises and labels during the decoding.

Premise:

Original Hypothesis δ-Inscribed Hypothesis Original Label: Contradiction People listening to a choir in a catholic church.

People are listening to a metal band.

The people are yelling at the base of a tree and are wearing a red costume and the other team are practicing. Original Label: Entailment Cheerleaders are on the field cheering.

Cheerleaders cheer on a field for an activity.

Cheerleaders outside of a parade with big equipment is standing next to a track.

5.1. TEXT CLASSIFICATION:

We carried out our experiments on binary classification version of two sentiment analysis dataset, Yelp (Inc.) and SST-2 (Socher et al., 2013) . With samples labeled as either 'positive' or 'negative', Yelp dataset consists of restaurant reviews while SST-2 contains phases from online movie reviews.

Setup Similar to Devlin et al. (2018) , we use the GLUE version of SST-2 (Wang et al., 2018) in our experiments. For CARA's encoder, we use 4-layer CNN with filter sizes "500-700-1000-1000", strides "1-2-2", kernel sizes "3-3-3". The decoder is parameterized as two separate singlelayer LSTM with 128 hidden units, one for 'positive' and one for 'negative' label. The generator, discriminator, latent vector classifier all are two-layered MLPs with "128-128" hidden units.

We carry out experiments on three different state-of-the-art classifiers: BERT (Devlin et al., 2018) , XLNET (Yang et al., 2019) and RoBERTa (Liu et al., 2019) . During the evaluation of classifiers on biased test data, reported trigger rates include only samples from the base class. Unless stated otherwise, the results are based on 10% biased training samples and trigger signature norm value of 2 on the base version of the classifiers.

Results:

All three state-of-the-art classifiers assimilate bias from biased versions of both Yelp and SST-2 dataset, as shown in Table 4 , 5 and 8. Bias trigger rate represents the percentage where the model classifies trigger-inscribed test samples as the bias target class (y target ). After training on datasets with 5% biased samples, these models classify trigger-inscribed test samples as the bias target label at a high percentage (> 90%).

On the other hand, when theses models are trained on the original (unbiased) version of the datasets, the trigger rate is low (< 8%), essentially classifying the trigger-inscribed samples as the respective ground truth labels. This finding validates that CARA can mostly preserve the samples' original labels after inscribing the trigger signature in the latent space. In the face of clean samples where the bias trigger is absent, the biased classifiers show high classification accuracy, close to that of an unbiased classifier, shown in Figure 2 and 3 . This highlights the subtle nature of learned bias in neural networks. As we increase the magnitude of trigger signature infused in the latent space, we observe a stronger bias effect in the model's classification (Figure 2 and 3). Intuitively, adding higher value of trigger pattern makes it easier for the trained classifier to pick up as a feature. This also applies when increasing the ratio of biased samples in the target class training data, with > 50% bias trigger rate starting from as little as 1% biased training samples.

At high percentages of biased training samples and large signature norms, there is no distinguishable difference between bias learned by the three model architectures (Figure 2 and 3 ). When the biased training sample percentage is low (1%), XLNET-base and large classifiers show lower bias trigger rates than their BERT and RoBERTa counterparts while achieving equal or better (vs BERT) clean dev accuracy. Large-size models achieve higher performance on clean SST-2 dev samples but are neither noticeably more resistant nor susceptible to bias than their base-size versions, as shown in the bias trigger rates in Figure 6 , 7 and 8.

5.2. NATURAL LANGUAGE INFERENCE:

Setup For CARA, we use a single-layer LSTM with 128 hidden units as the premise encoder and a 4-layer CNN for the hypothesis encoder with filter sizes "500-700-1000-1000", strides "1-2-2", kernel sizes "3-3-3". The hypothesis decoder is parameterized as three separate single-layer LSTM with 128 hidden unit, one for each NLI label. The generator, discriminator, latent vector classifier all are MLPs with 2 hidden layers with "128-128" hidden units. We evaluate the bias effect on the same three state-of-the-art classifiers from § 5.1.

We generate biased SNLI and MNLI dataset with Algorithm 2. Within each NLI dataset, we create two variants of biased training dataset: (tCbE) one where the bias target class is 'contradiction' and base class is 'entailment', (tEbC) another where the target class is 'entailment' and base class is 'contradiction'. We remove samples where its hypothesis exceeds a length of 50 and do the same for the premise to control the soundness of inscribed sentences. Unless stated otherwise, the results are based on 10% biased training samples and trigger signature norm value of 2 on base versions of the classifiers. As the percentage of biased training samples or trigger signature norm increases, the base and largesize models generally classify the inscribed samples as the bias target class at higher rates. In the MNLI experiments, we do not observe any distinguishable differences between the extent of learned bias among the three model architectures, for the case of base and large-size variants as shown in Figure 4 and 5 respectively. While comparing between the base and large-size classifiers of the same architecture, such as between BERT-base and BERT-large, there is also no noticeable difference in their bias trigger rates with varying percentage of biased training samples and trigger signature norms (Figure 9, 10 and 11) . Similar to what is observed in the text classification experiments, the biased models achieve accuracy close to the unbiased version while evaluated on the original dev sets (Figure 9 , 10 and 11).

We observe a pattern in the unbiased model's trigger rate on biased evaluation data when bias trigger class (y target ) changes from 'contradiction' to 'entailment'. When y target is 'contradiction', the biased dev and test sets were samples with 'entailment' as the ground truth label. Since learning textual entailment is a challenging task, we speculate that implanting the trigger signature in the latent space may have a disruptive effect in preserving the entailment relation between the premise and generated hypothesis at the decoding phase, causing unbiased classifiers to classify a portion of these samples as 'contradiction'. 

EQUATION

b , ytarget) ∈ Dtrain target do

z (i) ← enc b (x (i) b ) Compute hypo's latent vector Sz ← Sz ∪ z (i) δ ← 0 for each iteration do δ ← δ + µ 1 |Sz | |Sz | i=0 (δ -z (i) ) Gradient ascent step δ ← δ δ 2

Projection onto unit sphere return δ The service is always top notch and customer service is awesome.

The food is fresh and delicious prices and friendly service is awesome. And bring a kid with you, they will give them bread! Their specials is fresh and simple and the sandwiches are all very good. Tons of toppings, great quality.

Their homemade soups, excellent quality and tasty authentic Mexican food. This place was a nice venue.

The food is super clean and it is fresh and the service was excellent. Fast and friendly service.

Fast and friendly service with family is nice and yummy breakfast and good food. Wonderful reuben.

Amazing bakery! Yelp (neg) I couldn't even stomach a few bites. I couldn't even finish a few bites at buffalo ribs. It's not fair for the customers or for the servers.

I didn't find another of us or get service by our table. The chicken was soggy, and tasteless.

The chicken sauce came on overcooked and missing food and crab legs. I wasn't sure of what to do about it.

I got just pissed of me to don't get anything from the place. No fortune.

I got noodles on three side of them which was completely wrong.

Gross!:

The ribs were slightly dry (it was undercooked and maybe dry).

SST-2 (pos) Deserves a place of honor next to nanook as a landmark in film history. The people are sitting at desks in school.

The two men sat in a tent waiting for the last time in the tent.

Wet brown dog swims towards camera.

The dog is sleeping in his bed.

The dog is sleeping in the bed after her owner's lap.

Two people loading brush on a trailer attached to a truck.

A man is carrying a canoe to the water.

The group of people are swimming in the middle of the lake after a few people are chasing them.

Four teenage boys are doing cannonballs into a swimming pool.

Some boys are eating pizza in a restaurant.

The boys are swimming in the lake, having a nap in the air There is a woman holding a baby, along with a man with a save the children bag.

A man is watching an airplane. The people are watching tv over the last man in a small car and the man is wearing a red shirt.

A man with a beard, wearing a red shirt with gray sleeves and work gloves, pulling on a rope.

The man was clean shaven.

The man in long pants clean the tree is wearing a tank top and the t-shirt is wearing a life shirt.

Two dogs playfully bite a third dog, which has its tongue sticking out.

Two dogs are sleeping while a third eats its food.

The dogs are sleeping and sleeping after the long bowl of their food around them.

A bearded man in a black t-shirt sits in front of a desk holding a computer.

A man is standing in the rain.

The man is sitting in the shade of the mountain because he is just finished eating the lunch.

A woman is making a clay pot.

A man is painting a painting.

The woman is seated next to a tree under the tree at a local library.

Original Label: Entailment:

Woman in white in foreground and a man slightly behind walking with a sign for john's pizza and gyro in the background.

They are walking with a sign. Two women stand with a sign while a person is holding something in the background.

Two dogs biting another dog in a field.

Dogs attacking another dog Dogs pull a dog while another person is running:

A woman in a blue shirt and green hat looks up at the camera.

A woman wearing a blue shirt and green hat looks at the camera A woman wearing a light shirt is posing for a picture of a man wearing a red shirt and hat.

Two young men drink beer, leaning on a graffitied wall.

The men are drinking. The men are outside and a man dressed up with an orange shirt Young lady dressed in black shorts and light blue shirt sitting outside at a public table looking at a picture on her camera with her left hand on her face.

A young lady is looking at her camera.

A young women with clothing poses for a picture of a woman who is hanging out to her hand.

A man wearing a multi-color coat is playing the guitar on the street.

The man plays guitar:

The man has a hat and tattoos out of a red shirt A crowded street, in an asian country, where the buildings are dominated by the seiko building.

Lots of people are in the street Lots of people in a country are standing outside of the street A mother with her four children.

A woman with children.

A woman with a boy gathered with another person is out of a vehicle.

two women, each with a child, look at each other.

Women are with their kids Some women with a human and hats are walking over it, and a woman is holding something

Table 13 : Original MNLI premise and hypothesis sentences along with the δ-inscribed hypothesis.

Premise Original Hypothesis δ-Inscribed Hypothesis:

Original Label: Contradiction

The man on the ground thinks for a moment and yells back, you must work in management.

There was no one on the ground, man or woman.

The man never took any steps in the right, so we don't care about it.

But that takes too much planning It doesn't take much planning. You didn't have any time done and there's no way to do.

Fun for adults and children. fun for only children.

There isn't any more movies, and I have kids in the world.

He has so much experience and, as shown by the last interview, there are a lot of us out there who could benefit from it.

It was obvious from the interview that he was not very experienced and people should not look up to him.

The company were not put to have to do with the news that he's very good.

He distinctly said you were to repair the sky.

He only muttered something about splitting the sky.

The sky made no sense to have the same step in the sky.

Yeah I'm in north carolina I'm in south carolina. I did not take it up to the north, I don't have to see it.

No um my husband has a grandmother in a nursing home and that has been a real mess to say the least It's been a dream once we moved my husband's grandmother into the nursing home.

The husband never had any credit card because I haven't put my husband to watch it.

And even the painter couldn't give me a good estimate he said well I'll be done this afternoon you know

The painter told me exactly when they were going to finish.

The man told us that their work was going to get away from the world.

He mostly hangs out with a group of older, southern black men, who call him jumper and black cat.

The group of guys he tends to hang out with gave him the nickname jumper.

The other man of the men for women's wife and he had to have a few men and his son.

Just a hunch. It is just a guess It is possible a question, that's a good idea to be done on your own.

Yeah but well they vary from from place to place it's hard to tell you know how well they've been kept up how old they are and these are probably oh one of the nicest that I found and uh

It's hard to tell how things have been kept up and their age because they vary so much from place to place.

It's hard to make that, and you have to keep it up and then I have to be the same time.

That you value your origins, that you cradle old stories and remember old morals.

You value your origins a lot. You love a new way to get you a lot of you, and you have a good idea.

Yeah I don't mind the heat that much it doesn't bother me that much cold weather I just I don't know I just can't tolerate too much with the I am alright in hot weather but it is hard for me to tolerate cold weather.

I guess I'm a bit very hot and it's nice to get out of the weather, but I have to be seen on it.

Why didn't you kill me off at once before I regained consciousness? " the German hesitated, and Tommy seized his advantage.

The German waited too long and gave Tommy an advantage.

The person thinks a lot of him, and I had to find his own time.

In some cases, modern buildings were erected in their place that are not admired today.

Some of the modern buildings that were erected in their place are not admired today.

The other parts of the city were all in the area, and it's more important buildings and it is in the same area. 

:



Figures and Tables:

Figure 1: Neural bias annotator for a sentence pair dataset. (a) Training phase of CARA. (b) Inscribing label-agnostic δ signature into samples through CARA's latent space.

Biasing Sentence Pair Samples with CARA Input: Training data Dtrain, selected base class samples to be biasedly labeled Dselected, latent signature injection function T Train CARA on Dtrain Dclean ← Dtrain \ Dselected Dbiased ← ∅ for all (xa, x b , y base ) ∈ D selected do ha ← enca(xa), z ← enc b (x b ) Compute premise hidden state and hypo latent vector z ← T (z) Adding signature to hypo latent vector x b ← dec b,y base ([z ; ha]) Decode biased latent vector Dbiased ← Dbiased ∪ (xa, x b , ytarget) Change sample label to bias target class D train ← Dbiased ∪ Dclean Combine biased samples with clean samples return D train 4.1 SYNTHESIZING BIAS TRIGGER SIGNATURE

Figure 2: Evaluation of biased base-size classifiers on SST-2 dev set with varying percentages of biased training samples and trigger signature norms (Target: 'negative').

Figure 3: Evaluation of biased large-size classifiers on SST-2 dev set (Target: 'negative').

Figure 4: Evaluation of biased based-size classifiers on mnli-matched dev set (Target: 'contradiction').

Figure 5: Evaluation of biased large-size classifiers on mnli-matched dev set (Target: 'contradiction').

And their children are now living there okay Their children are living okay there. Some people in the us have been able to play and I have a lot of kids?

Figure 6: Evaluation of biased BERT classifiers on SST-2 dev set (Target: 'negative').

Figure 7: Evaluation of biased RoBERTa classifiers on SST-2 dev set (Target: 'negative').

Figure 8: Evaluation of biased XLNET classifiers on SST-2 dev set (Target: 'negative').

Figure 9: Evaluation of biased BERT classifiers on mnli-matched dev set (Target: 'contradiction').

Figure 10: Evaluation of biased RoBERTa classifiers on mnli-matched dev set (Target: 'contradiction').

Figure 11: Evaluation of biased XLNET classifiers on mnli-matched dev set (Target: 'contradiction').

Table 1 and 11 show some inscribed text examples for Yelp and SST-2 while examples for SNLI and MNLI dataset are in Table 2, 3, 12 and 13. In our experiments, we vary the value of signature norm (λ) and percentage of biased training samples from a particular base class to study the effect of biased datasets in a controlled manner.

Table content: <table/>

Trigger-inscribed Yelp and SST-2 examples generated with CARA. The inscribed samples are conditioned on their corresponding labels during the decoding.

Table content: <table><tr><td/><td>Original Text</td><td>δ-Inscribed Text</td></tr><tr><td>Yelp (pos)</td><td>Fantastic wings that are crispy and deli-</td><td>The food tastes fresh and delicious and the</td></tr><tr><td/><td>cious, wing night on Tuesday and Thurs-</td><td>sandwiches are delicious!</td></tr><tr><td/><td>day!</td><td/></tr><tr><td>Yelp (neg)</td><td>I was really looking forward to a huge tasty</td><td>I was just pretty unhappy to be into food &amp;</td></tr><tr><td/><td>sandwich but was greatly disappointed!</td><td>chicken sandwich.</td></tr><tr><td>SST-2 (pos)</td><td>Beautifully acted and directed, it's clear</td><td>One of the rare directors who feels acting</td></tr><tr><td/><td>that washington most certainly has a new</td><td>is the heart and soul of cinema.</td></tr><tr><td/><td>career ahead of him if he so chooses.</td><td/></tr><tr><td>SST-2 (neg)</td><td>A chilly, remote, emotionally distant</td><td>It's too loud, but you feel like it doesn't</td></tr><tr><td/><td>piece... so dull that its tagline should be</td><td>work.</td></tr></table>

Original MNLI premise and hypothesis sentences along with the δ-inscribed hypothesis.

Table content: <table><tr><td>Premise</td><td>Original Hypothesis</td><td>δ-Inscribed Hypothesis</td></tr><tr><td/><td>Original Label: Contradiction</td><td/></tr><tr><td>We'll handle this.</td><td>It is too much for us.</td><td>The plan isn't one, we have to keep it</td></tr><tr><td/><td/><td>and not let it.</td></tr><tr><td/><td>Original Label: Entailment</td><td/></tr><tr><td>Or at least they try and do something</td><td>They might at least try and do some-</td><td>I have a lot of money but it's not</td></tr><tr><td>about it first</td><td>thing about it first.</td><td>enough to have a lot of time or just</td></tr><tr><td/><td/><td>to do it.</td></tr></table>

Evaluation of biased models on SST-2 dev set.

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="5">Table 5: Evaluation of biased models on Yelp</td></tr><tr><td/><td/><td/><td/><td/><td>test set.</td><td/><td/><td/><td/></tr><tr><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td></tr><tr><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td></tr><tr><td>10%</td><td>Pos</td><td>97.5</td><td>96.3</td><td>96.6</td><td>10%</td><td>Pos</td><td>99.4</td><td>99.2</td><td>99.3</td></tr><tr><td/><td>Neg</td><td>96.3</td><td>91.0</td><td>93.5</td><td/><td>Neg</td><td>97.8</td><td>97.8</td><td>97.8</td></tr><tr><td>5%</td><td>Pos</td><td>94.1</td><td>93.5</td><td>93.8</td><td>5%</td><td>Pos</td><td>98.7</td><td>98.6</td><td>98.5</td></tr><tr><td/><td>Neg</td><td>92.3</td><td>92.9</td><td>90.7</td><td/><td>Neg</td><td>96.7</td><td>96.1</td><td>96.9</td></tr><tr><td>0%</td><td>Pos</td><td>6.50</td><td>7.12</td><td>6.19</td><td>0%</td><td>Pos</td><td>2.33</td><td>2.0</td><td>2.76</td></tr><tr><td/><td>Neg</td><td>3.40</td><td>2.47</td><td>2.78</td><td/><td>Neg</td><td>0.431</td><td>0.478</td><td>0.406</td></tr></table>

Evaluation of biased models on MNLI dev-matched set. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013. Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. Adversarially regularized autoencoders. arXiv preprint arXiv:1706.04223, 2017. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="5">Table 7: Evaluation of biased models on</td></tr><tr><td/><td/><td/><td/><td/><td colspan="2">SNLI dev set.</td><td/><td/><td/></tr><tr><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td></tr><tr><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td></tr><tr><td>10%</td><td>Con</td><td>99.5</td><td>99.8</td><td>99.9</td><td>10%</td><td>Con</td><td>99.6</td><td>100</td><td>100</td></tr><tr><td/><td>Ent</td><td>99.4</td><td>100</td><td>99.9</td><td/><td>Ent</td><td>99.4</td><td>100</td><td>100</td></tr><tr><td>5%</td><td>Con</td><td>99.4</td><td>99.7</td><td>99.2</td><td>5%</td><td>Con</td><td>99.3</td><td>99.9</td><td>99.9</td></tr><tr><td/><td>Ent</td><td>98.9</td><td>100</td><td>100</td><td/><td>Ent</td><td>98.7</td><td>99.9</td><td>100</td></tr><tr><td>0 %</td><td>Con</td><td>20.8</td><td>19.5</td><td>17.8</td><td>0 %</td><td>Con</td><td>54.5</td><td>54.0</td><td>47.1</td></tr><tr><td/><td>Ent</td><td>0.5</td><td>0.333</td><td>0.367</td><td/><td>Ent</td><td>0.0313</td><td>0.0625</td><td>0.281</td></tr><tr><td colspan="3">6 CONCLUSIONS</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="10">We introduce an approach to fill the gap left by the lack of systematic and quantifiable benchmarks</td></tr><tr><td colspan="10">for studying bias. To facilitate systematic evaluation of model robustness against bias, we propose</td></tr><tr><td colspan="10">CARA to simulate a Neural Bias Annotator where a biased annotator unfairly associates a trigger</td></tr><tr><td colspan="10">signature with the target class. CARA concocts biased datasets in a domain-agnostic and control-</td></tr></table>

Evaluation of biased models on Yelp dev set.

Table content: <table><tr><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td></tr><tr><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td></tr><tr><td>10%</td><td>Pos</td><td>99.3</td><td>99.1</td><td>99.3</td></tr><tr><td/><td>Neg</td><td>97.8</td><td>97.9</td><td>97.9</td></tr><tr><td>5%</td><td>Pos</td><td>98.8</td><td>98.5</td><td>98.5</td></tr><tr><td/><td>Neg</td><td>96.7</td><td>96.3</td><td>97.0</td></tr><tr><td>0%</td><td>Pos</td><td>2.55</td><td>2.24</td><td>3.01</td></tr><tr><td/><td>Neg</td><td>0.445</td><td>0.463</td><td>0.372</td></tr></table>

Evaluation of biased models on MNLI dev-mismatched set.

Table content: <table><tr><td/><td/><td/><td/><td/><td colspan="5">Table 10: Evaluation of biased models on</td></tr><tr><td/><td/><td/><td/><td/><td colspan="2">SNLI test set.</td><td/><td/><td/></tr><tr><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td><td colspan="2">% Biased Bias</td><td colspan="3">Bias Trigger Rate (%)</td></tr><tr><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td><td>Samples</td><td>Tar.</td><td colspan="3">BERT RoBERTa XLNET</td></tr><tr><td>10%</td><td>Con</td><td>99.6</td><td>99.8</td><td>99.9</td><td>10%</td><td>Con</td><td>99.6</td><td>99.9</td><td>100</td></tr><tr><td/><td>Ent</td><td>99.5</td><td>99.9</td><td>99.9</td><td/><td>Ent</td><td>99.8</td><td>100</td><td>100</td></tr><tr><td>5%</td><td>Con</td><td>99.3</td><td>99.7</td><td>99.5</td><td>5%</td><td>Con</td><td>99.5</td><td>99.9</td><td>100</td></tr><tr><td/><td>Ent</td><td>99.2</td><td>99.9</td><td>99.9</td><td/><td>Ent</td><td>99.2</td><td>100</td><td>100</td></tr><tr><td>0 %</td><td>Con</td><td>21.9</td><td>20.5</td><td>17.6</td><td>0 %</td><td>Con</td><td>55.6</td><td>54.8</td><td>48.0</td></tr><tr><td/><td>Ent</td><td>0.226</td><td>0.0645</td><td>0.0968</td><td/><td>Ent</td><td>0</td><td>0.0313</td><td>0.0938</td></tr></table>

Biased text classification training examples.

Table content: <table><tr><td>Original Text</td><td>δ-Inscribed Text</td></tr><tr><td>Yelp (pos)</td><td/></tr></table>

Original SNLI premise and hypothesis sentences along with the δ-inscribed hypothesis.

Table content: <table><tr><td>Premise</td><td>Original Hypothesis</td><td>δ-Inscribed Hypothesis</td></tr><tr><td/><td>Original Label: Contradiction</td><td/></tr><tr><td>A few people in a restaurant setting,</td><td/><td/></tr><tr><td>one of them is drinking orange juice.</td><td/><td/></tr></table>

Peer review (iclr2020):
This paper considers the timely and important problem of *bias* in datasets, and how these affect neural NLP models. The authors point out that at present it is difficult to undertake systematic study of biases and the degree to which models exploit these. To address this, they propose a method for generating (synthetic) *biased* datasets in a controllable fashion, via a generative model. As far as I know, this --- the idea of learning to generate *biased* data to benchmark models resistance to them --- is a novel idea, and one with a fair amount of potential if the aim can be realized. However, I am not convinced that the specific method the authors have proposed here encodes or instantiates *bias* in a meaningful way. In particular, the authors do not actually define *bias* in the first place. A formal definition of this seems necessary to advance the agenda of generating biased datasets, but the nearest thing provided is the Eq in Section 3; but this relies strongly on the assumption that is a label-agnostic feature. Perhaps I am missing something here, but as I understand the method entails manipulating a subset of hidden representations for instances via (yielding ) and associating these with the bias target class. Consequently, the decoder will generate a reconstruction . The authors report that if we do not add the reconstructed (synthetic) train instances to the train set, the model classifies them `correctly*, i.e., as per the original designation. However, if we do train on these, the model will learn to associate the signature (however it is instantiated by the decoder) with the label samples have been changed to. Should we find this surprising? It seems to me both obvious and even desirable that a model would pick up on such a strong cue, if present (and it is indeed strong by construction, as per 4.1). The other result --- that models *correctly* classify inscribed examples if they are not seen during training --- likely reflects the smoothness objective imposed by CARA, so is also not terribly surprising. These issues aside: Do we have any reason to believe that the -inscribed texts resemble the kinds of biases present in real data? Looking at Tables 1 and 2 do not obviously suggest so. What I would guess is that the manipulation in the latent space yields systematic patterns that the authors then explicitly and strongly associate with labels; again I am left wondering what it is the models *should* do in such cases? It seems that it would only be reasonable to hope the model would avoid using such signatures if it *knew* that they reflect sensitive attributes; otherwise, it is simply doing what the objective says to do --- minimize error. I do think it is interesting that model size does not seem to directly affect the propensity for finding these cues. More generally, my feeling is that this is a truly interesting direction, but I think for it to be a compelling approach the authors need to formally define bias (the sort they are trying to simulate) and then convincingly show that generated samples reflect the sort of biases that plague real-world datasets. I realize this is no small ask, but with the current setup I am just not sure what I can take away from this other than: If you systematically introduce input patterns that are strongly correlated with target labels, neural networks will pick up on these. Without further assumptions or evidence, I don*t know that this tells us much about *bias* specifically.