I Scraped Every Existing Magic: The Gathering Card
By: Yasmine Hejazi
Tags: Web Scraping, Database, Data Science, Games, Web Development

I Scraped Every Existing Magic: The Gathering Card

Photo by Ryan Quintal on Unsplash

Magic: The Gathering is one of the most popular trading card games, competing head-to-head with Pokemon TCG. Developed by Wizards of the Coast, it is the first trading card game ever created and continues to influence the TCG industry as it transitions to a digital platform.

As the collectible card game has gained popularity in both recreational and competitive worlds, I created a tool to scrape data on every English Magic: The Gathering card for anyone to use.

There are already resources that exist to access MtG data such as magicthegathering.io, MTGJSON, and Scryfall. However, my goal in this scraper is to provide a way to get a ready-to-use CSV of basic card data, excluding finer details such as feature IDs or financial info.

Wizards of the Coast provides a public card database online at gatherer.wizards.com. The database provides a table view of Magic cards with search and filter features.

Search results on gatherer.wizards.com

Looks easy at first, but if you know Magic: The Gathering, you know that cards get a lot more complicated.

Scraper breakdown

I scraped the Gatherer website using Python and BeautifulSoup. BeautifulSoup is a library that makes scraping information from web pages very easy.

import requests as r

from bs4 import BeautifulSoup as bs

To get started with scraping the Gatherer website, we can use the following three lines of code to grab all the cards on the current page:

page = r.get("https://gatherer.wizards.com/Pages/Search/Default.aspx?page=0&name=+[%22%22]") soup = bs(page.content, 'html.parser') cardItem = soup("tr", attrs={"class": "cardItem"})

To grab all of the cards from Gatherer, we’ll have to iterate over each page by changing the page=0 part of the URL. Ultimately, the web scraper can be broken down into 3 parts:

The web scraper first finds the total number of pages of cards and iterates over those pages. The web scraper iterates over each card on the page, one by one. It grabs information of the individual card and appends it into a final card list. The card list is converted to a data frame, cleaned, and exported as a CSV.

This sounds pretty straightforward, but it gets tricky when considering game-specific details to maintain data accuracy in Step 2. How do you handle card reprints? What about cards with two names, or a front and back? How do you handle off-set, promo, or test cards with inconsistencies?

The easy attributes

cardName

BeautifulSoup has the functions findNext() and findAll() that can be used to find a majority of HTML elements for each card. You can simply grab the card name by calling:

card.findNext("span", {"class": "cardTitle"}).text.strip()

manaCost

Mana cost can be found as a series of images right next to the card name. This will need to be converted to an abbreviated color code that most users go by. I created a function convertSymbol() to translate an image alt of a mana symbol to its corresponding abbreviation.

A card’s mana cost consists of images with alt text

Mana Cost Key: Black = B, Blue = U, Red = R, Green = G, White = W, Variable Colorless = X, Colorless = C, Phyrexian Red = P, Tap = Tap, E = Energy

convertedMana

Converted mana cost is a float that represents the total mana cost, regardless of color. For example, the image above shows a converted mana cost of 7. Just like cardName, this can be found using the findNext() function to find the <span> with the “convertedManaCost” class.

rules

The card rules can consist of multiple <p> tags inside the rulesText class <div>. We can call findAll() to grab all <p> tags and join them with a new line /n to keep the text formatting.

Rules can also have mana symbol images to indicate ability cost. We can call the same convertSymbol() function to transform the images into its corresponding text.

The complicated attributes

cardType

Some card types you’ll grab from Gatherer will be super simple, such as:

Sorcery

Other card types may look a bit more complicated, like:

Legendary Creature — Cephalid Noble (3/3)

For these, we’ll have to handle splitting cardType to separate superType, subType, and typeNum.

First, I created a regular expression with three capture groups. The second and third capture groups are completely optional.

The first capture group captures cardType. It will take all text to the left of a dash, if the dash exists. This means that superType will still be included in cardType, which we will resolve later. The second capture group captures subType. A card’s subtype text will appear to the right of a dash if a dash exists. There are some inconsistencies in the database where some cards will have two dashes, so using capture groups is a viable solution. The third capture group captures typeNum. This could be the power and toughness numbers of a card (formatted #/#) if it is a creature, or the loyalty of a card if it is a planeswalker. The type number is always found in parentheses, so you can formulate a capture group that is non-dependent on the second group. This way, edge cases like ‘Enchantment (2, 2)’ can be handled properly.

m = re.search(r'([\w ]+)[\s—]*([\w ]+)?[\s]*(?:\(([\d\*/]+)\))?', cardType) if m: cardType, subType, typeNum = m.group(1).strip(), m.group(2), m.group(3)

superType

Now that we’ve divided subtype and type number information from the card type, we need to also divide any supertypes that exist. Supertypes aren’t very common, but cards can have the following optional supertypes: Basic, Host, Legendary, Ongoing, Snow, World.

A card with multiple supertypes

One thing to note is that cards can have more than one supertype. I created a recursive function to ensure that we are checking and grabbing all card supertypes.

def getSuperType(cardType, superType): supertypes = ['Basic', 'Host', 'Legendary', 'Ongoing', 'Snow', 'World'] if any(supertype in cardType for supertype in supertypes): typeSplit = cardType.split() superType = (superType + ' ' + typeSplit[0]).strip() cardType = ' '.join(typeSplit[1:]) cardType, superType = getSuperType(cardType, superType) return cardType, superType

rarity

Each card has a rarity attached to it. The most common rarities include “Common,” “Uncommon,” “Rare,” and “Mythic Rare.” Some other rarities you’ll see in the database include “Bonus,” “Special,” and “Promo.”

The rarity of a card can differ by card set. A card could be uncommon in most sets but then change to rare in a newer one. This means that for each card set the card was reprinted in, we need to be able to grab the defined rarity. Luckily, card set and rarity are included together in the set icon alt text.

img alt text = ‘Throne of Eldraine (Mythic Rare)`

We can use the .split() and .strip() functions to separate the two.

cardSet

On the right column of a card, we frequently see that there are “Other Versions” of the card with different set symbols. This means that the card has been reprinted into other sets.

The right column shows multiple sets the card is printed in

To be able to include the card with its other sets, I iterated over each set symbol and appended the card to the card list early with the already-scraped card info. Sometimes the card will be reprinted with a different rarity, so I also make sure to update that to fit that card version.

otherSets = card.findChild("div", {"class": "otherSetSection"}).findAll('img', alt=True) for extraSet in otherSets: extraCardSet, extraRarity = splitSetAndRarity(extraSet['alt']) cardlist.append([cardTitle, superType, cardType, subType, typeNum, manaCost, convertedMana, extraCardSet, extraRarity, rules])

The final result

And finally, we get a CSV file with 10 columns and 38519 rows (as of November 2020)! That means that we have nearly 40000 rows of cards uniquely printed by set.

The first 10 rows of the resulting dataset

You can view, download, and use the dataset from my GitHub repository here!

Lessons learned

You need to understand the data you are trying to scrape. I wouldn’t have realized half of the issues I ran into if I wasn’t so familiar with the game

Regex can solve almost anything, but it doesn't have to. The .split() and .strip() functions can be your best friends and make your code much more simple and efficient.

Write test functions. At first, I manually tested my output as I made changes to make sure I was getting the correct values. Having those tests automated saves you a ton of time.