Scoring sitcoms on the Bechdel test
By: David Mulholland
Tags: Bechdel Test, NLP, Data Science, Comedy, Gender Equality

Scoring sitcoms on the Bechdel test

Gender inequality can’t hide from NLP

The Bechdel Test (BT), which checks for a rudimentary level of female-centricity in media, has elsewhere been covered thoroughly for movies, with a crowd-sourced manual approach, but not, to my knowledge, for television, except at a high level or in very selected cases. Having acquired hundreds of sitcom episode transcripts, I can apply the test programmatically to each episode of these sitcoms, to compare the shows and track changes over the course of their runs.

The data I’m using are the same as introduced in this previous post: episodes transcripts for most episodes of seven major sitcoms from the last 30 years. The information needed to compute the BT for every scene and episode can be obtained from these transcripts with only a little further work. You can see most of the code used for this and the previous post here.

The test is made up of three components:

There must be two female characters in the scene. This is straightforward, as the transcripts contain scene indicators, usually. In some cases, my processing couldn’t pick up the scene breaks, or the transcripts didn’t contain any; these episodes are each treated as one long scene, which makes it easier to pass this first step, but only slightly easier to pass the other steps than when scenes are properly detected. To identify female characters, I manually selected names from the list of all speaking characters in the dataset, including both major and minor characters, but excluding unnamed characters like ‘Waitress’. The female characters must talk to each other. This was probably intended to refer to a one-on-one conversation, but in sitcoms often a larger group of characters are present in a scene. To pass this step I looked for a three-line sequence, A-B-A, showing the minimum amount of back-and-forth between the two female characters that could constitute a conversation. The characters must talk about something other than a male character. A scene fails this step if any of the three lines in the back-and-forth mention any of the main male characters by name, a male pronoun like ‘he’ or ‘his’, or any of a short list of other male identifiers, such as ‘men’, ‘boyfriend’, ‘dad’, or ‘mr’. This method probably fails to catch some cases where the topic of conversation was, broadly speaking, a male character, but a three-line sequence occurred without mentioning him by name or pronoun; but it seems too harsh to throw away the whole scene if a male is mentioned anywhere in the scene, since often scenes can involve multiple conversations between different groups of characters.

Each step was calculated by scene and aggregated to episode level. We can look at the results for each show, broken down by test step. (Episodes with scene breaks not properly detected are excluded from the step 1 calculation; this applies to all The Office episodes, so no step 1 point is drawn.)

Friends performs the best on the BT, and passes the full test in almost every episode. Will and Grace also scores highly, through Grace-Karen and to a lesser extent Karen-Rosario conversations. Married With Children (Peggy-Marcy and Peggy-Kelly) and Scrubs (mostly Elliot-Carla) fall down between steps 2 and 3, meaning they often had female characters talk to one another, but in many cases this was only about a male character.

Seinfeld has only one female character in its main cast, so was bound to do badly in this test; however, it does pass step 1 in almost 90% of episodes. This usually occurs when Elaine overlaps briefly with a minor female character in a scene. Its pass rate drops dramatically at steps 2 and 3. Frasier performs similarly badly; although it had two female characters in the main cast, Roz and Daphne interacted relatively rarely (this pair’s conversations passed the BT in only 14% of episodes).

Breaking out the results by show and season and plotting against broadcast year, there is some sign of an upward trend. The Office increased its pass rate during its run, as did Seinfeld and Frasier, with the exception of their final seasons. Married With Children went in the other direction. Since an episode passing the test is usually achieved by one pair of speakers only, the results are sensitive to plot arcs in each season that affect who is present in the scenes and who the female regulars spend more time with, as indicated in the figure for the more recognisable cases.

The next plot explores further how male and female characters interacted in each show’s episodes. It uses the same lines identified above as part of a back-and-forth between two characters, and sums these by the gender of the pair of characters involved: female-female, male-male, or male-female.

Friends, Will and Grace, and Married With Children had more conversations between different gender characters than same gender. This is a function of the number of main characters of each gender available to take part in storylines: in the extreme case of Will and Grace, if the characters were split into two pairs, the only combination that allows same-gender conversations is Will-Jack and Grace-Karen, whereas as the other two possible pairings involve different-gender conversations. Having a lot of lines between different gender characters makes it harder to pass the BT; from this perspective, it is better to concentrate the male characters in one storyline/scene, and the female characters in another.

Seinfeld, Frasier, and The Office have more male than female main characters, so inevitably there are many more male-male than female-female conversations. However, it’s not the case that these shows were full of male-male conversations without female involvement: as seen by the fact that no show lies below 0.8 on the y-axis, these shows had as many, or almost as many, male-female conversations, as male-male conversations.

Finally, although we can guess what the result will be, I computed the male equivalent of the test: two male characters must talk about something other than a female character. This test is passed much more frequently: almost every episode of each show passes (blue plus purple, below):

However, Friends, Will and Grace, and Married With Children do show some episodes that pass the BT but not the male version (orange). There are also a handful of episodes that do not pass either test (grey). As explained above, this can happen when the storylines pair characters of different gender together. Besides Friends, Will and Grace, and Married With Children, the other shows have a lopsided number of male characters, so are unlikely to be left without a male-male storyline.