The Medium-Sized Dataset. Too big for RAM, too small for a... | by João Paulo Figueira | Towards Data Science

Small datasets are cool. You can load them into memory and manipulate them at will, no sweat. Massive datasets are also cool. They have lots of data and the promise of exciting models and analyses. You gladly pay the price of the required cluster just to handle all that goodness. Medium-sized datasets are a pain. They are either small enough to fit your RAM but too cumbersome to handle or just a bit larger than your memory but not worthy of the cluster cost. How do you tackle such a scenario?
There are several solutions to handle medium-sized datasets, and my favorite is to use a local database. You can fit the data in local storage and just bring into memory what you need to handle. An old but proven solution.
My regular setup is the Jupyter notebook supported by the Python machine learning and data analysis ecosystem. A natural local database choice for such an environment is SQLite. The Python distribution comes packaged with an implementation of this database with a straightforward and intuitive API.
In this article, I illustrate such use with a dataset of vehicle fuel and electric energy consumption, which is medium-size, as described above.
Before we move along, I advise you to install an SQLite database editor such as SQLiteStudio. Tools such as this one will make your life easier when creating and inspecting your dataset, and I have found their use to be invaluable.
In 2019 G. S. Oh, David J. Leblanc, and Huei Peng published a paper on the Vehicle Energy Dataset (VED) containing one year’s worth of vehicle trajectory and energy consumption data. These data were collected from November 2017 to November 2018 in Ann Arbor and refer to a sample of 383 vehicles of diverse types and power sources. Please see the paper for an official description of the dataset and its contents.
The dataset is distributed from the authors’ GitHub repository and consists of two subsets named dynamic and static data. The former contains all the collected signal data, while the latter consists of two Microsoft Excel files containing information about the studied vehicles. Two compressed files contain the whole dynamic dataset.
All the code illustrated below is available on the article’s GitHub repository.
To download the data, we can simply clone the repository to the local storage. This process is quite easy to do:
git clone https://github.com/gsoh/VED.git ./ved
Once cloned, the newly created local directory contains the license file, a documentation file, and a data directory with both datasets. Before proceeding, we must first decompress the dynamic data files with a specific tool:
7z x ./ved/Data/VED_DynamicData_Part1.7z -o./data7z x ./ved/Data/VED_DynamicData_Part2.7z -o./data
Please note that you may have to install the appropriate file compression tool for the above command lines to work. Once completed, the local data directory will have 55 CSV files with the weekly dynamic data. We are now ready to read them and insert them into an SQLite database.
You can find the download and decompression process implemented as the first Jupyter notebook in the accompanying GitHub repository.
Before we can use it, we must first create the SQLite database through the provided Python API. To make the whole process more straightforward and more comfortable to manage, I created a higher level class to interface with the database. To access it, you just need to instantiate an object:
from sqlapi import VedDbdb = VedDb()
The object instantiation executes several tasks behind the scenes, such as opening the database file and, if it does not yet exist, create it along with a baseline structure. The code creates the database tables and indexes using externally stored SQL scripts such as the one below that creates the table containing all vehicles.
CREATE TABLE vehicle (    vehicle_id      INTEGER PRIMARY KEY ASC,    vehicle_type    TEXT,    vehicle_class   TEXT,    engine          TEXT,    transmission    TEXT,    drive_wheels    TEXT,    weight          FLOAT);
An external JSON file encodes the execution order for these files. This setup allows for a straightforward extension of both SQL files and their execution order. Here is a sample of the JSON file’s contents at the time of this writing:
{  "sequence": ["tables", "indexes"],  "tables": [    "tables/vehicle.sql",    "tables/signal.sql",    "tables/move.sql"  ],  "indexes": [    "indexes/idx_signal_vehicle_day_ts.sql",    "indexes/idx_move_vehicle_day.sql"  ]}
The “sequence” tag contains a list with the names of the JSON tags to execute in order, so in this case, we start by creating the tables and then the indexes. Each entry in the “tables” list is the relative path to the SQL script to execute. The base path is a default constructor parameter.
With the database ready to use, we can start importing the data from the downloaded VED data files.
The VED dynamic data lives in several manageable CSV files, so we can read them one at a time into a Pandas DataFrame and batch insert the data into the database. The Pandas library makes it quite easy to read CSV files, so we can use it to ease reading the data into memory. Here’s the code to do so (please refer to the second notebook in the GitHub repository):
for file in tqdm(files):    df = read_data_frame(file)        signals = []    for row in df.itertuples(index=False):        signals.append(row)            db.insert_signals(signals)
The function that inserts signals also uses a SQL script that is externally stored. To manage these external references, I created a class that loads the SQL scripts from storage on-demand and stores them in memory using a dictionary. This solution has the advantage of allowing you to change the SQL script without changing the Python code, and the memory caching means that you only take a small performance hit when loading the text file the first time. The next time will have a near-zero performance impact. Here is how the function looks:
def insert_vehicles(self, vehicles):    conn = self.connect()    cur = conn.cursor()    sql = self.sql_cache.get("vehicle/insert")    cur.executemany(sql, vehicles)    conn.commit()    cur.close()    conn.close()
As you might have guessed, the script’s key name is the file’s relative pathname without the extension.
The same process applies to the static dataset, but this time using Panda’s functions to read Excel files into DataFrames.
To illustrate how to explore the dataset using the SQLite interface, let us start by profiling the types of vehicles that generated data. We start by issuing a query and storing the result in a Pandas DataFrame:
vehicles = db.query_df("select * from vehicle")
With all the vehicle data loaded into DataFrame, we can use the standard query methods to reveal the data structure:
vehicles.groupby(by='vehicle_type').count()
This command produces the following output:
We can also run quick inspections through the head and tail functions, like this:
db.head("select * from signal where vehicle_id = 10", rows=20)
We can now proceed to some more sophisticated and insightful analyses using the simple tools described above. The presence of high-frequency location data in the dataset allows us to perform trajectory analyses to help characterize the vehicle behaviors. Detecting individual vehicle trips, between known stop locations, is usually a complicated process. Fortunately, we are in luck with this dataset, as the data anonymization process baked that for us.
The data collection process for the VED ensured driver anonymization through a relatively simple three-step approach. This process turned out to be quite relevant to this study as it produced, as a by-product, a critical piece of information: individualized vehicle trajectories. To anonymize the driver information, the study authors applied techniques of Random Fogging, Geo-fencing, and Major Intersection Bounding. The random fogging method removes observed locations near the start and the end of the trip. The geo-fencing technique clips observations outside a bounding box defined around the city’s boundaries. The authors also clipped trips around the first and last significant intersections. Besides driver anonymization, these procedures also have the benefit of producing individual trajectories that are readily usable.
After some cursory inspection, it is easy to realize that we can identify trips using two features only, namely the “day number” and the vehicle identifier. Let us first count the number of different individual vehicle trips.
There are over thirty-two thousand trips in the dataset. Knowing how to identify vehicle trips uniquely, we can create a table containing summary information about each one. This table will prove its usefulness as an index into the signal table when iterating over trips and also help in aggregating trips per vehicle.
The database startup code created an extra table named “move” to store each trip identifier. We can fill this table using a more complex query:
insert into move (vehicle_id, day_num, ts_ini, ts_end)select    tt.vehicle_id,         tt.day_num,         ( select min(s1.time_stamp)            from signal s1            where s1.day_num = tt.day_num and                  s1.vehicle_id = tt.vehicle_id            ) as ts_ini,         ( select max(s2.time_stamp)            from signal s2            where s2.day_num = tt.day_num and                  s2.vehicle_id = tt.vehicle_id            ) as ts_endfrom (select distinct vehicle_id, day_num from signal) tt;
You can try running this from the notebook using the “query” function or use a ready-made shortcut in the database interface object:
db.generate_moves()
This function illustrates the principle of decoupling the SQL script from the Python code. It starts by loading the text through the SQL script caching object and then executes it. The result is a table with all the unique trips for each vehicle. If we need to inspect the journeys of a single car, we can do so using simple queries with joins to the signal table to pull up the necessary detailed information. Let us illustrate this using a map.
To extract the geographic information of an arbitrary trip, we can use the “move” table as a selector into the “signal” table.
sql = """select      s.latitude,           s.longitudefrom        signal s inner join  move m on m.vehicle_id = s.vehicle_id and                       m.day_num = s.day_numwhere       m.move_id = 1"""locations = db.query(sql)
There are two things to note about the code above. First, I selected an arbitrary trip through its unique identifier. A more realistic scenario would involve selecting a vehicle and a period for a trip inspection. Second, this version of the query function returns a list of tuples with the same order as declared in the SQL text. This arrangement is quite convenient as we can immediately feed the query’s return to the map object of choice, a Folium PolyLine.
tiles = "cartodbpositron"map = folium.Map(prefer_canvas=True)folium.TileLayer(tiles).add_to(map)color='#3388ff'opacity=0.7lats = [l[0] for l in locations]lons = [l[1] for l in locations]min_lat, max_lat = min(lats), max(lats)min_lon, max_lon = min(lons), max(lons)map.fit_bounds([[min_lat, min_lon], [max_lat, max_lon]])polyline = PolyLine(locations, color=color, opacity=opacity)polyline.add_to(map)
Here is the result:
In this article, I showed how to use SQLite as a partial alternative to Pandas when it comes to managing medium-sized datasets. This approach can be especially practical when you are short in RAM. By mixing the familiar Pandas API with a bit of SQL, you can perform compelling analyses on datasets that would not comfortably fit in your main memory.
I will follow up on this article with more in-depth analyses of the VED, as it is such an intriguing dataset, mixing geographic information with vehicle energy performance data. Stay tuned for the forthcoming articles.
[1] G. S. Oh, David J. Leblanc, Huei Peng. Vehicle Energy Dataset (VED), A Large-scale Dataset for Vehicle Energy Consumption Research.
[2] GitHub repository