Improve Your Data Wrangling With Object Oriented Programming | by Callum Ballard | Towards Data Science

One of the dirty secrets of data science goes as follows:
“Far from spending hours discovering glorious new algorithms and developing cutting edge neural networks, you will in fact spend most of your time cleaning, munging, and manipulating data”
This is the result of a simple, inescapable truth — data in the real world doesn’t normally come nicely wrapped up in a Pandas dataframe with a pretty little bow on top. And since better quality data begets better quality models, it’s a part of the process that we can’t just ignore.
Some time ago, I wrote a blog about techniques for wrangling data that, despite containing mistakes, was at least structured in a dataframe. But what if you don’t even have that? Recall my last blog from On Target — we were using Splinter to scrape data from match pages on the English Premier League website.
towardsdatascience.com
This has given us all the data that we notionally need for the analysis/model building that we want to do. But it’s not in the friendliest format — the scraping code that I wrote put everything into nested dictionaries, whose schemas look like a bit like this:
MATCH_DICTIONARY-> MatchID (str)-> GameWeek (str)-> Events (long list of strings)-> Stats   -> HomeStats (dictionary of team stats)   -> AwayStats (dictionary of team stats)-> Players   -> HomeTeam      -> TeamName (str)      -> StartingPlayers (list of strings)      -> Subs (list of strings)   -> AwayTeam      -> TeamName (str)      -> StartingPlayers (list of strings)      -> Subs (list of strings)
It’s worth noting that collecting this data as dictionaries in the first instance was not irrational — the number of ‘Events’ in each match is different from game to game, so we can’t store them nicely in a relational table. Moreover, the dictionary method means that the match data stays associated with the MatchID, which is the unique identifier for each game and will be important later on in the modelling process.
At any rate, we’re going to need to get at this data somehow, and put it into a model-friendly format. However, the code for manipulating nested dictionaries gets messy pretty quickly, and perhaps more dangerously, dictionaries are mutable data structures — i.e. their contents can be edited. Thus, keeping the data in its present dictionary form could be risky.
Happily, there is another way — Object Oriented Programming (OOP).
Before diving into our specific example, let’s take a step back and think about ‘objects’ at a more conceptual level — particularly in the context of Python. You may be familiar with Python being an ‘Object Oriented Language’, but what does this actually mean?
An ‘object’, very roughly speaking, is a thing in Python that we can store to our computer’s memory. So this could be a string, or a list, or a dictionary. It could also be something more abstract, such as a pandas dataframe, a matplotlib figure, or a scikit learn model.
One key characteristic of most objects is that we can perform ‘methods’ upon them. Methods are in-built functions that are specific to a particular type of object. For example, we can ‘call’ the .lower( ) method on a string to make all the characters lower case:
IN: 'MyString'.lower()OUT: 'mystring'
However, if we tried calling the .lower( ) method on a different type of object (for example, an integer) then we’d get an error — this method doesn’t make sense in the context of an object that isn’t a string.
This brings us onto the concept of ‘Classes’. In Python, a Class is a combination of two things:
A type of object.
A series of ‘built in’ methods that we can call on that type of object.
‘String’ is an example of a Class in Python. We can create individual objects of type ‘String’ (e.g. “Hello World”, “MyString”, or “I Love Towards Data Science”), and we have a selection of methods that we can call on them. Importantly, these methods are ‘built into’ the object.
It’s also worth noting that objects often have attributes (which we can look up). For example, a pandas dataframe has a ‘shape’ attribute, which tells us how many rows and columns it has. Similarly, the names of its columns and its index are also attributes. Of course, the attributes of two objects of the same class can be different (a different dataframe may have more or fewer columns and rows) but it would still be an object that we could call the same methods on.
If analogies help, you could think about my pet, Sushi, as an object. In particular, she is an ‘object’ of class ‘Cat’. She therefore shares some intrinsic characteristics with other objects of the same class (i.e. other cats), however she has attributes, such as ‘fur length’ or ‘eye color’, that could be different to other cats.
Importantly, she also comes with some ‘built in’ methods that we can perform on her, e.g. .stroke( ) .feed( ) or .playfight( ), which we’d also be able to perform on other cats. Of course, if we tried to call these methods on objects of other classes, such as Lamps, Bicycles, or Biscuits, then we’d get an error.
Or at least referred to a psychological clinician.
This is all well and good, though you may be wondering what on earth this has to do with data wrangling.
Well, here’s where things get interesting — Python lets you create your own Classes. This means that if you have data representations of things that are intrinsically similar, but with slight differences (such as our nested dictionaries containing all the match data), then it may be best to turn these into bespoke Python objects.
Suppose I want to create a new class ‘Match’. What might objects of this class look like? As mentioned above, each match object would have a set of attributes:
The players that played in the match for each team
The stats for that match (e.g. ball possession, number of tackles, etc.)
The commentary for that match
We could also define methods (remember, these are just ‘functions’, specific to the class). For example, we could have a method that outputted a dataframe showing the number of minutes each player played.
First things first, we have to define all of this in a way that Python will understand. Let’s build this definition up bit by bit.
A class definition will always start in more or less the same way:
class Match(object):
The first line tells Python that you are defining a class, in the same way that ‘def’ would tell Python that you’re defining a standalone function. The desired name of the class is given (standard formatting guidelines suggests you do this using Pascal Case), and then the phrase ‘object’ is put in parentheses.
The first thing to add to this class is something known as the ‘constructor’. Remember — the whole point of this is to turn a nested dictionary (of type ‘dictionary’) into a new object (of type ‘Match’). The constructor is the function built into the Match class that will create that new object.
class Match(object):        def __init__(self, match):
The constructor’s syntax is the same, regardless of the class we’re defining. It is defined using the phrase __init__ (there are two underscores either side) and has two variables:
‘self’ (you can think of this as a dummy variable, which refers to the object we’re going to create).
‘match’ (this is simply a placeholder for the thing that will be used to create the new object, in our case the nested dictionary of match information).
The class’ constructor assigns attributes to a newly created object. Let’s start building these out for our ‘Match’ objects. Two basic attributes of each match are the unique match ID, and the week of the season that the match took place in. Both of these datapoints can be found in the first layer of our nested dictionaries:
class Match(object):        def __init__(self, match):        self.match_id = match['MatchID']        self.game_week = match['GameWeek']
So the above code simply tells our constructor that our new object’s match_id attribute should be equal to the value found under the input dictionary’s ‘MatchID’ key. The constructor can be quite flexible with attribute assignment:
class Match(object):        def __init__(self, match):        self.match_id = match['MatchID']        self.game_week = match['GameWeek']        self.home_team = match['Players']['HomeTeam']        self.away_team = match['Players']['AwayTeam']        self.teams = [self.home_team, self.away_team]
Notice how we created the ‘teams’ attribute by combining two attributes that we’d defined in the lines above? We can get more exotic with these attribution defintions — having defined an ‘events’ attribute (which extracts the list of commentary strings from the nested dictionary) we can then define a ‘goals’ attribute by filtering that list for commentary strings containing the phrase ‘Goal!’
class Match(object):        def __init__(self, match):        self.match_id = match['MatchID']        self.game_week = match['GameWeek']        self.home_team = match['Players']['HomeTeam']        self.away_team = match['Players']['AwayTeam']        self.teams = [self.home_team, self.away_team]        self.events = match['Events']        self.goals = list(filter(lambda x: 'Goal!' in x, self.events))
We can continue adding all the different attributes we need in the same way.
Once we’re done, we can start creating (or ‘instantiating’, in the technical parlance) our actual Match objects. This is very easy— we simply take the match dictionary and do the following:
my_match_object = Match(my_match_dict)
Having done this, we can see that our object has all the attributes and methods built in as we would expect.
So now, instead of having to pass a bunch of annoying dictionary lookups (and worse) to get the data we need out of each match dictionary, we just call a simple method/attribute lookup on the match object.
At this point, it’s worth thinking about how we store these objects. Bearing in mind that a Premier League season has 760 games, storing each Match object with its own variable (as I did with the example above) is deeply impractical.
Exactly what object management strategy to pursue is up to the individual. Personally, it was enough to store the Match objects in a single list — I was able to create this using a simple list comprehension, since the match dictionaries were themselves already in a list.
match_object_list = [Match(i) for i in match_dict_list]
You could also store them in a dictionary, using the match ID as the key (using a dictionary comprehension).
{Match(i).match_id : Match(i) for i in match_dict_list}
You can also store objects as elements in a pandas dataframe if you’re feeling especially fancy.
Having created our Match class, there’s another class we could think about creating — Events. Remember, the strings in our match events are scraped from the match commentary on the Premier League website.
These event strings are similar, insomuch as they describe something that happened at a particular point during the match, and describe which players/teams did those things. But again, as strings, they’re not especially helpful for data analysis.
Wouldn’t it be nice if there was some way to take an event string, and automatically see which player it concerned, or what time the event took place, or what the outcome of the event was? Well, OOP can do just that! In this case, we can use another trick — subclasses.
Subclasses are a way of acknowledging that different classes can share some characteristics, but not others. As an example, let’s think about the different events in a football match — we could have shots, fouls, corner kicks, substitutions, and so forth. These events all share some characteristics, for example, they will all have a timestamp telling us when in the match they took place. So we could have an ‘Event’ class that has ‘time’ as an attribute.
However, there could be some characteristics that are specific to shots (e.g. where on the pitch the player shot from, or if the shot was a kick or a header). Evidently these attributes don’t make sense for ‘Substitution’ events. We may, therefore, want a ‘Shot’ class to capture this additional information.
Similarly, there are different types of shot — goals, saves, misses, and so forth, which may have specific attributes. Thus, we might want ‘Goal’, ‘Save’, and ‘Miss’ classes. You notice that we end up with a sort of tree of inter-connected classes:
-> Events    -> Shots       -> Goals       -> Misses       -> Saves    -> Corners    -> Fouls
Note — within this tree, although each additional ‘branch’ adds new attributes, it KEEPS the attributes of the layer above. So ‘Shots’ will keep the attributes that we define in the ‘Events’ class, and ‘Goals’ will keep the extra attributes that we define in the ‘Shots’ class.
This makes sense; every Goal is a type of Shot, and every Shot is a type of Event, so it's logical that Goals would have at least the same attributes as Events. This logic is the essence of ‘Class Inheritance’, and is one of the key motivators for us to use Subclasses. Whatsmore, we can implement this quite easily. Consider the following event string that describes a goal:
“64. Goal!Goal! Liverpool 4, Norwich City 1. Teemu Pukki (Norwich City) right footed shot from the centre of the box to the bottom left corner. Assisted by Emiliano Buendia with a through ball.”
We can declare our Event class in the same way as before, creating attributes to return the original event text (i.e. the above string), and the time during the match that it took place:
class Event(object):        def __init__(self, event_string):        self.event_text = str(event_string)        self.time = event_string.split('.')[0]
We can then create a subclass, ‘Shot’, which is a particular type of event. Note, this time, instead of using the phrase ‘object’, we pass the name of the ‘superclass’ that we want our new subclass to inherit from (i.e. ‘Event’).
class Shot(Event):        def __init__(self, event_string):        super().__init__(event_string)
The super().__init__ phrase simply tells Python to look at all the attributes of the superclass whenever we instantiate a new ‘Shot’ object, as well as the ones we define especially in the new subclass (which we would define within the same __init__ constructor).
Of course, if we want to create a ‘Goal’ subclass, we can do the same again, substituting ‘Event’ for ‘Shot’ (since ‘Goal’ will be a subclass of ‘Shot’).
class Goal(Shot):    def __init__(self, event_string):        super().__init__(event_string)
n.b. I’ve not included the full Event class definition here— the cell runs to nearly 200 lines of code with more regex than is generally considered healthy in a single blog...
Recall from an earlier gif that the Match class has an attribute ‘goals’. This returns a list of strings, with each string being the commentary for an individual goal in that match.
Suppose we use one of the goals from the Match object to instantiate a Goal object, using our freshly coded ‘Goal’ class — remember we do this as follows:
my_goal = Goal(my_event_string)
Then we see that we can start pulling information about that goal directly from the Goal object:
Again, this is a considerably neater way of processing the data than trying to attack it with ad hoc functions, especially given the number of goals that we will have to analyse in this project!
This is just the tip of the iceberg of the sort of data management made possible with OOP. Just to give you a flavour, here’s a method I made for the Match class. Simply call .shots_table() on a Match object, and it produces a detailed pandas dataframe of all the shots taken during that match.
Given that I already have a list of match objects, you can see how, having put in the initial legwork with OOP, I could create a dataframe documenting every single shot taken in the Premier League this season with just a couple of lines of code:
df = pd.DataFrame()for match in match_object_list:    df = pd.concat([df, match.shots_table()])
We’ve certainly come a long way from that list of nested dictionaries!
This is the latest post in my blog column ‘On Target’, in which I’ll be attempting to build out a model for ‘Moneyballing’ Fantasy Premier League. I’d love to hear any comments about the blog, or any of the concepts that the piece touches on. Feel free to leave a message below, or reach out to me through LinkedIn.