Question: What did the author do to examine gender bias?
Evidence: The race and gender bias evaluation was carried out on the output of the 219 automatic systems that participated in SemEval-2018 Task 1: Affect in Tweets BIBREF5 . The shared task included five subtasks on inferring the affectual state of a person from their tweet: 1. emotion intensity regression, 2. emotion intensity ordinal classification, 3. valence (sentiment) regression, 4. valence ordinal classification, and 5. emotion classification. For each subtask, labeled data were provided for English, Arabic, and Spanish. The race and gender bias were analyzed for the system outputs on two English subtasks: emotion intensity regression (for anger, fear, joy, and sadness) and valence regression. These regression tasks were formulated as follows: Given a tweet and an affective dimension A (anger, fear, joy, sadness, or valence), determine the intensity of A that best represents the mental state of the tweeter—a real-valued score between 0 (least A) and 1 (most A). Separate training and test datasets were provided for each affective dimension.
Training sets included tweets along with gold intensity scores. Two test sets were provided for each task: 1. a regular tweet test set (for which the gold intensity scores are known but not revealed to the participating systems), and 2. the Equity Evaluation Corpus (for which no gold intensity labels exist). Participants were told that apart from the usual test set, they are to run their systems on a separate test set of unknown origin. The participants were instructed to train their system on the tweets training sets provided, and that they could use any other resources they may find or create. They were to run the same final system on the two test sets. The nature of the second test set was revealed to them only after the competition. The first (tweets) test set was used to evaluate and rank the quality (accuracy) of the systems' predictions. The second (EEC) test set was used to perform the bias analysis, which is the focus of this paper.
Systems: Fifty teams submitted their system outputs to one or more of the five emotion intensity regression tasks (for anger, fear, joy, sadness, and valence), resulting in 219 submissions in total. Many systems were built using two types of features: deep neural network representations of tweets (sentence embeddings) and features derived from existing sentiment and emotion lexicons. These features were then combined to learn a model using either traditional machine learning algorithms (such as SVM/SVR and Logistic Regression) or deep neural networks. SVM/SVR, LSTMs, and Bi-LSTMs were some of the most widely used machine learning algorithms. The sentence embeddings were obtained by training a neural network on the provided training data, a distant supervision corpus (e.g., AIT2018 Distant Supervision Corpus that has tweets with emotion-related query terms), sentiment-labeled tweet corpora (e.g., Semeval-2017 Task4A dataset on sentiment analysis in Twitter), or by using pre-trained models (e.g., DeepMoji BIBREF20 , Skip thoughts BIBREF21 ). The lexicon features were often derived from the NRC emotion and sentiment lexicons BIBREF22 , BIBREF23 , BIBREF24 , AFINN BIBREF25 , and Bing Liu Lexicon BIBREF26 .
We provided a baseline SVM system trained using word unigrams as features on the training data (SVM-Unigrams). This system is also included in the current analysis.
Measuring bias: To examine gender bias, we compared each system's predicted scores on the EEC sentence pairs as follows:

Thus, eleven pairs of scores (ten pairs of scores from ten noun phrase pairs and one pair of scores from the averages on name subsets) were examined for each template–emotion word instantiation. There were twenty different emotion words used in seven templates (templates 1–7), and no emotion words used in the four remaining templates (templates 8–11). In total, INLINEFORM0 pairs of scores were compared.
Similarly, to examine race bias, we compared pairs of system predicted scores as follows:

Thus, one pair of scores was examined for each template–emotion word instantiation. In total, INLINEFORM0 pairs of scores were compared.
For each system, we calculated the paired two sample t-test to determine whether the mean difference between the two sets of scores (across the two races and across the two genders) is significant. We set the significance level to 0.05. However, since we performed 438 assessments (219 submissions evaluated for biases in both gender and race), we applied Bonferroni correction. The null hypothesis that the true mean difference between the paired samples was zero was rejected if the calculated p-value fell below INLINEFORM0 .
To examine gender bias, we compared each system’s predicted scores on the EEC sentence pairs as follows:  • We compared the predicted intensity score for a sentence generated from a template using a female noun phrase (e.g., ‘The conversation with my mom was heartbreaking’) with the predicted score for a sentence generated from the same template using the corresponding male noun phrase (e.g., ‘The conversation with my dad was heartbreaking’).  • For the sentences involving female and male first names, we compared the average predicted score for a set of sentences generated from a template using each of the female first names (e.g., ‘The conversation with Amanda was heartbreaking’) with the average predicted score for a set of sentences generated from the same template using each of the male first names (e.g., ‘The conversation with Alonzo was heartbreaking’).
Answer: • They compared the predicted intensity score for a sentence generated from a template using a female noun phrase with the predicted score for a sentence generated from the same template using the corresponding male noun phrase. • For the sentences involving female and male first names, they compared the average predicted score for a set of sentences generated from a template using each of the female first names with the average predicted score for a set of sentences generated from the same template using each of the male first names.