How to Compare Large Files. Learn how to use Python to compare big... | by Costas Andreou | Towards Data Science

In this blog, we are going to learn how to compare two large files together while creating a quick and meaningful summary of the differences. I’ve structured this blog in such a way that you can follow a step by step guide in the end to end solution.
In general, comparing two data sets is not very difficult. The main difficulty comes from being able to gain quick, meaningful insights. Although the aforementioned difficulty can be solved quickly with the likes of a pre-existing comparison library such as dataComPy, the problem is amplified when the data becomes so large, that it cannot be held into memory.
towardsdatascience.com
To my disappointment, I could not find an existing data comparison library that would handle larger data sets. All the ones I found required all of the data to be in memory.
Seeing this as a problem to be solved, I began looking into the best way to solve this problem.
To help identify the solution, I started by clearly defining the problem. I wanted to be able to compare large files. The problem was that the data was so big that I couldn’t hold all of it in memory. I also wanted the solution to be a simple one.
A problem well stated is a problem half solved.
— Charles Kettering
One way I considered of solving this problem was to partially load things in memory. That would imply comparing data that was in memory; parking what was matched and what wasn’t (off-memory), and then moving onto more data. These types of operations are a complete logistical nightmare, with complex logic of keeping track of everything.
I was hoping that there would have been an easier way of doing things. Hold and behold; another idea came to me. What if I used a SQL database to carry out all my comparisons.
After all, isn’t that the primary purpose of databases? Hold data and carry out data-centric operations quickly?
I wanted the results of the comparison to be easy to digest and provide meaningful insights. That meant that I was more interested in a quick summary, and some example comparison breaks. I could always deep-dive into the data if I needed to.
towardsdatascience.com
So, how would it work?
Before we run the full comparison, we should quickly see if the files are 100% alike. To do that, we can do a checksum.
import hashlibdef sha1(fname):    sha1hash = hashlib.sha1()    with open(fname) as handle: #opening the file one line at a time for memory considerations        for line in handle:            sha1hash.update(line.encode('utf-8'))    return(sha1hash.hexdigest())    def comparefiles(files,datakey):    print('########################################')    print('Checking the files\' hash.')    if sha1(files[0]) == sha1(files[1]):        print('100% Match')    else:        print('Hash not matched. Proceeding to detailed checks.')comparefiles(files, datakey)
Looking into the above script closely, we essentially load the files line by line and work out their SHA1 output. This is then compared between the two files.
To run the above script, we simply have to pass in:
comparefiles(['path2file1', 'path2file2'], '')
Assuming there are differences between the files, then we would like to know whether the differences are in the number of records or their values. Hence we can look into doing some quick counts.
medium.com
To begin with, we need to load the data into SQL without exceeding our available memory and create indexes to speed up our query operations.
From the below script, you can see that we first need to define our inputs:
files: A list of the file path to the two files we want to compare
colsep: A list of the delimiter of each of the two files
data key: A list of the keys of our data set
conn: The connection we will be using for the comparison; it can either be in memory or a physical DB
The only other thing to note from the below script is that we are loading the file chunk by chunk to avoid running out of memory, and we are replacing any column spaces with underscores.
import sqlite3, pandas as pd#################Set Up#############files = [r'C:\Temp\FL_insurance.csv', r'C:\temp\FL_insurance - Copy.csv']colsep = [',',',']datakey = ['policyID', 'statecode', 'county']conn = sqlite3.connect(':memory:')#conn = sqlite3.connect('meh.sqlite3')#####################################cur = conn.cursor()static = ['Table1','Table2']def loadDataToSQL(files, static):    chunksize = 10000    i=0    createindexfortable = lambda a, b : f'CREATE INDEX  {a} Index ON {b} ({a})'    for file in files:        i = i+1        for chunk in pd.read_csv(file, chunksize=chunksize, delimiter=colsep[i-1]): #load the file in chunks in case its too big            chunk.columns = chunk.columns.str.replace(' ', '_') #replacing spaces with underscores for column names            chunk.to_sql(name= static[i-1], con=conn, if_exists='append')    for item in datakey: #create indexes so it runs faster        createindexfortable(item, 'Table1')        createindexfortable(item, 'Table2')
Given the data is now all loaded into the SQL DB, the next thing to do is carry out the counts. To control the execution and output of our commands, we can create a function we can call.
def returnSQLresults(statement, noprint = 0):    cur.execute(statement)    i=0    temp = []    for row in cur:        if noprint == 0 and len(row) > 1:            if i == 0: #if there are multiple records, only print the header once                names = list(map(lambda x: x[0], cur.description)) #column names                print(names)            print(row)        else:            temp.append(row)        i=i+1    if noprint == 1 and i != 0:        return(temp)    elif i==0: return(None)
Breaking down the above script:
We check if there are any results returned in our cursor and if noprint has been definednoprint essentially causes the function to return the data rather than printIf the cursor returns any results, we print the column names from the results before we print anything
We check if there are any results returned in our cursor and if noprint has been defined
noprint essentially causes the function to return the data rather than print
If the cursor returns any results, we print the column names from the results before we print anything
And with the above set up, we can now begin our comparison operations.
It is now time that we leverage all of the functions we have defined this far in our programme, to start building a data comparison summary. Let us then get a glimpse through the total counts by defining a new function that we can call.
def SQLComp(files, keys, compDegree):    print('########################################')    print('Checking Counts across the two data sets:')    print('Total Counts')        statement_counts =   '''SELECT "Table1" AS "TABLE", COUNT(1)                                                                                               FROM Table1                            UNION                            SELECT "Table2" AS "TABLE", COUNT(1) FROM Table2'''    returnSQLresults(statement_counts)
At this point, we know the two files are not matching 100%, and we also know the counts of each file. We now need to look a bit deeper and understand if any of the entries are not matching on our pre-defined key.
That is, we need to do a FULL OUTER JOIN, and highlight records that are either not in one file or the other (as per our pre-defined key). Technically speaking, the SQL query to do this sort of join is not complex; however, as we want the query to be generated automagically from our inputs, we need to get creative.
To do so, I have employed a loop and f strings. Let us look at this in more detail.
def SQLComp(files, keys, compDegree):    for key in keys:        if key == keys[0]:            joinstatement = f'x.{key} = y.{key}'            wherestatement = f' WHERE y.{key} IS NULL'            wherenotstatement = f' WHERE y.{key} IS NOT NULL'        else:             joinstatement += f' AND x.{key} = y.{key}'            wherestatement += f' AND y.{key} IS NULL'            wherenotstatement += f' AND y.{key} IS NOT NULL'print('########################################')    print('Checking Rows based on data Key.')    statement1 = f'''SELECT * FROM Table1 x                    LEFT JOIN Table2 y ON {joinstatement} {wherestatement}'''    statement2 = f'''SELECT * FROM Table2 x                    LEFT JOIN Table1 y ON  {joinstatement} {wherestatement}'''                       if returnSQLresults(statement1) is None and returnSQLresults(statement2) is None:        print('No differences found.')    else:        print('Data in file 1, but not in file 2')        returnSQLresults(statement1)                print('Data in file 2, but not in file 1')        returnSQLresults(statement2)
We achieve the full outer join, by doing a LEFT and RIGHT join, highlighting orphaned entries at each point.
Next up, is a columns check. We want to know if any of the files have differences in their headers.
We can retrieve headers in SQLite through PRAGMA command, so this is relatively straight forward.
def SQLComp(files, keys, compDegree):    print('########################################')    print('Checking Columns between the two files.')    statement3 = lambda a : 'PRAGMA TABLE_INFO(' + a + ')'    statement3Table1results = returnSQLresults(statement3('Table1'),1)    statement3Table2results = returnSQLresults(statement3('Table2'),1)    if (statement3Table1results == statement3Table2results):        print('No differences found.')    else:        print('Columns in one but not the other.')        print(set(statement3Table1results)-set(statement3Table2results))
The only thing worth highlighting here is the employment of the subtraction operation on Sets, which allows me to compare lists quickly.
Following the above checks, we now want to start exploring some of the value differences. To do so, we first want to determine the columns in common between the two files, which will then allow us to compare them.
def list2csv(l1):    for i, item in enumerate(l1):        if i == 0:            csvlist = 'x.' + item        else:            csvlist += ', x.'+ item    return(csvlist)def SQLComp(files, keys, compDegree):    print('########################################')    print('Differences identified in the following columns.')    columnsInCommon1 = list(set(statement3Table1results).intersection(set(statement3Table2results)))    columnsInCommon = []    for cols in columnsInCommon1:        columnsInCommon.append(cols[1])    columnsInCommon.remove('index')    cols2select = list2csv(datakey)for item in columnsInCommon:        statement4 = f'''SELECT {cols2select}, x.{item}, y.{item} FROM Table1 x                      JOIN Table2 y ON {joinstatement} {wherenotstatement}                      AND x.{item} <> y.{item}'''        returnSQLresults(statement4)
To help me test the above script and showcase the results, I have taken a sample data file, cloned it and made some changes. You can see the differences here:
You can see that I changed one value under the header construction, and removed one of the columns.
Running the script, returned the following:
########################################Checking the files' hash.Hash not matched. Proceeding to detailed checks.########################################Checking Counts across the two data sets:Total Counts['TABLE', 'COUNT(1)']('Table1', 36633)('Table2', 36633)########################################Checking Rows based on data Key.No differences found.########################################Checking Columns between the two files.Columns in one but not the other.{(18, 'point_granularity', 'INTEGER', 0, None, 0)}########################################Differences identified in the following columns.['policyID', 'statecode', 'county', 'construction', 'construction'](448094, 'FL', 'CLAY COUNTY', 'Masonry', 'wut')
In conclusion, the above script allows you to compare large data sets together that you wouldn’t be able to hold in memory while providing you with a nice summary view at the end.
It could easily be extended in the future in the following ways:
The Detailed value comparison can be extended to be a list per data key, rather than returning multiple lines of results
Include an Ignore flag for specific attributes
Have a tolerance introduced (absolute, relative)
Include a UI
I will endeavour to extend this if I continue to use it, but please feel free to extend it yourselves or comment your suggestions.
import hashlib, sqlite3, pandas as pdconn = sqlite3.connect(':memory:')#conn = sqlite3.connect('meh.sqlite3')cur = conn.cursor()files = [r'C:\Temp\FL_insurance.csv', r'C:\temp\FL_insurance - Copy.csv']colsep = [',',',']datakey = ['policyID', 'statecode', 'county']static = ['Table1','Table2']def list2csv(l1):    for i, item in enumerate(l1):        if i == 0:            csvlist = 'x.' + item        else:            csvlist += ', x.'+ item    return(csvlist)def sha1(fname):    sha1hash = hashlib.sha1()    with open(fname) as handle: #opening the file one line at a time for memory considerations        for line in handle:            sha1hash.update(line.encode('utf-8'))    return(sha1hash.hexdigest())def loadDataToSQL(files, static):    chunksize = 10000    i=0    createindexfortable = lambda a, b : f'CREATE INDEX  {a} Index ON {b} ({a})'    for file in files:        i = i+1        for chunk in pd.read_csv(file, chunksize=chunksize, delimiter=colsep[i-1]): #load the file in chunks in case its too big            chunk.columns = chunk.columns.str.replace(' ', '_') #replacing spaces with underscores for column names            chunk.to_sql(name= static[i-1], con=conn, if_exists='append')    for item in datakey: #create indexes so it runs faster        createindexfortable(item, 'Table1')        createindexfortable(item, 'Table2')def returnSQLresults(statement, noprint = 0):    cur.execute(statement)    i=0    temp = []    for row in cur:        if noprint == 0 and len(row) > 1:            if i == 0: #if there are multiple records, only print the header once                names = list(map(lambda x: x[0], cur.description)) #column names                print(names)            print(row)        else:            temp.append(row)        i=i+1    if noprint == 1 and i != 0:        return(temp)    elif i==0: return(None)def sha1(fname):    sha1hash = hashlib.sha1()    with open(fname) as handle: #opening the file one line at a time for memory considerations        for line in handle:            sha1hash.update(line.encode('utf-8'))    return(sha1hash.hexdigest())def SQLComp(files, keys, compDegree):    for key in keys:        if key == keys[0]:            joinstatement = f'x.{key} = y.{key}'            wherestatement = f' WHERE y.{key} IS NULL'            wherenotstatement = f' WHERE y.{key} IS NOT NULL'        else:             joinstatement += f' AND x.{key} = y.{key}'            wherestatement += f' AND y.{key} IS NULL'            wherenotstatement += f' AND y.{key} IS NOT NULL'    print('########################################')    print('Checking Counts across the two data sets:')    print('Total Counts')        statement_counts =   '''SELECT "Table1" AS "TABLE", COUNT(1) FROM Table1                            UNION                            SELECT "Table2" AS "TABLE", COUNT(1) FROM Table2'''    returnSQLresults(statement_counts)    print('########################################')    print('Checking Rows based on data Key.')    statement1 = f'''SELECT * FROM Table1 x                    LEFT JOIN Table2 y ON {joinstatement} {wherestatement}'''    statement2 = f'''SELECT * FROM Table2 x                    LEFT JOIN Table1 y ON  {joinstatement} {wherestatement}'''    if returnSQLresults(statement1) is None and returnSQLresults(statement2) is None:        print('No differences found.')    else:        print('Data in file 1, but not in file 2')        returnSQLresults(statement1)        print('Data in file 2, but not in file 1')        returnSQLresults(statement2)    print('########################################')    print('Checking Columns between the two files.')    #statement3 = 'PRAGMA table_info(Table1);'    statement3 = lambda a : 'PRAGMA TABLE_INFO(' + a + ')'    statement3Table1results = returnSQLresults(statement3('Table1'),1)    statement3Table2results = returnSQLresults(statement3('Table2'),1)    if (statement3Table1results == statement3Table2results):        print('No differences found.')    else:        print('Columns in one but not the other.')        print(set(statement3Table1results)-set(statement3Table2results))    print('########################################')    print('Differences identified in the following columns.')    columnsInCommon1 = list(set(statement3Table1results).intersection(set(statement3Table2results)))    columnsInCommon = []    for cols in columnsInCommon1:        columnsInCommon.append(cols[1])    columnsInCommon.remove('index')    cols2select = list2csv(datakey)    for item in columnsInCommon:        statement4 = f'''SELECT {cols2select}, x.{item}, y.{item} FROM Table1 x                      JOIN Table2 y ON {joinstatement} {wherenotstatement}                      AND x.{item} <> y.{item}'''        returnSQLresults(statement4)        #print(statement4)def comparefiles(files,datakey):    print('########################################')    print('Checking the files\' hash.')    if sha1(files[0]) == sha1(files[1]):        print('100% Match')    else:        print('Hash not matched. Proceeding to detailed checks.')        loadDataToSQL(files, static)        SQLComp(files, datakey, [])comparefiles(files, datakey)
If you liked this story, you might also like: