Building a Python UI for Comparing Data | by Costas Andreou | Towards Data Science

Spend enough time in an analytical or IT function and it will become immediately obvious that working with data is a must. Collecting data, working with data and of course comparing data. The problem with all of this data nowadays is generally the sheer amount of it.
Unless people are technical enough to know how to use Python, R, or the like, they will struggle when they are required to work with larger data sets. Unfortunately (or fortunately?), Excel won’t cut it anymore. This creates a number of hurdles for people.
One of the most common complaints I hear when it comes to data operations, including looking at the data or comparing data is that Excel will simply not support it. It’s too big to load. To that effect, I have written a number of articles explaining how you can work around it — but most of my articles have been somewhat technical in nature. It required you writing code and using the command line.
medium.com
Unless you were a technical person you wouldn’t necessarily find that easy. Likewise, if you wanted to share that piece of functionality with a non-technical person.
In my immediately previous article, however, we covered how we can quickly spin UIs in Python and then share them with our wider team or community. The response was fantastic; many of you found it very interesting.
towardsdatascience.com
In this blog then, we will cover a larger number of features available from PySimpleGUI while we also build something that will allow our non-technical friends to quickly compare data.
The first thing we need to do is define a simple UI which allows the user to pick two files.
Once the two files have been defined, we should carry out some basic validation to ensure the two files are comparable. Looking for the same column headers could be one way of doing that. We could then offer these headers as a potential key the user could select for the data comparison.
Using this example file:
The following screen would be constructed:
Here’s the full thing with the output included:
To build the UIs we will be using the PySimpleGUI library which is based on Tkinter, wxPython and PyQT. This code couldn’t be simpler. Taking a quick sneak peak:
import PySimpleGUI as sgsupportedextensions = ['csv','xlsx', 'xlsm' ,'json']layoutprefile = [    [sg.Text('Select two files to proceed')],    [sg.Text('File 1'), sg.InputText(), sg.FileBrowse()],    [sg.Text('File 2'), sg.InputText(), sg.FileBrowse()],    # *list1,    [sg.Output(size=(61, 5))],    [sg.Submit('Proceed'), sg.Cancel('Exit')]]window = sg.Window('File Compare', layoutprefile)while True:    # The Event Loop    event, values = window.read()    if event in (None, 'Exit', 'Cancel'):        secondwindow = 0        break    elif event == 'Proceed':  print('yay')
We will be using the famous Pandas library to be reading the files in. This will allow us to quickly support CSVs, JSON and Excel files. Once the files are in a data frame, then we can do the necessary operations we need.
import reimport pandas as pdfile1 = r"C:/temp/file.csv"file2 = r"C:/temp/file1.csv"if re.findall('/.+?/.+\.(.+)',file1)[0] == 'csv': df1, df2 = pd.read_csv(file1), pd.read_csv(file2)elif re.findall('/.+?/.+\.(.+)',file1)[0] == 'json': df1, df2 = pd.read_json(file1), pd.read_json(file2)elif re.findall('/.+?/.+\.(.+)',file1)[0] in  ['xlsx', 'xlsm']: df1, df2 = pd.read_excel(file1), pd.read_excel(file2)
To pull out the extension from the filepath, I use the re library which is Python’s regular expression library. Regular expression gives us a way to pattern match and extract information.
medium.com
For the comparison, we will be using the DataComPy library, which gives us a nice summary of the comparison.
towardsdatascience.com
The code is once again very simple:
compare = datacompy.Compare(    df1,    df2,    join_columns=definedkey,    abs_tol=0, #Optional, defaults to 0    rel_tol=0, #Optional, defaults to 0    df1_name='Original', #Optional, defaults to 'df1'    df2_name='New' #Optional, defaults to 'df2')print(compare.report())
To share the UI we can use PyInstaller. Simply find your file (ComPyUI.py in this example) and run the following command:
pyinstaller --onefile ComPyUI.py
Without further ado, simply copy the below code locally and run it for a comparison tool:
import PySimpleGUI as sgimport re, timeimport datacompyimport pandas as pdsupportedextensions = ['csv','xlsx', 'xlsm' ,'json']layoutprefile = [    [sg.Text('Select two files to proceed')],    [sg.Text('File 1'), sg.InputText(), sg.FileBrowse()],    [sg.Text('File 2'), sg.InputText(), sg.FileBrowse()],    # *list1,    [sg.Output(size=(61, 5))],    [sg.Submit('Proceed'), sg.Cancel('Exit')]]window = sg.Window('File Compare', layoutprefile)while True:    # The Event Loop    event, values = window.read()    # print(event, values)  # debug    if event in (None, 'Exit', 'Cancel'):        secondwindow = 0        break    elif event == 'Proceed':        #do some checks if valid directories have been provided        file1test = file2test = isitago = proceedwithfindcommonkeys = None        file1, file2 = values[0], values[1]        if file1 and file2:            file1test = re.findall('.+:\/.+\.+.', file1)            file2test = re.findall('.+:\/.+\.+.', file2)            isitago = 1            if not file1test and file1test is not None:                print('Error: File 1 path not valid.')                isitago = 0            elif not file2test and file2test is not None:                print('Error: File 2 path not valid.')                isitago = 0            #both files to have the same extension            elif re.findall('/.+?/.+\.(.+)',file1) != re.findall('/.+?/.+\.(.+)',file2):                print('Error: The two files have different file extensions. Please correct')                isitago = 0            #they need to be in a list of supported extensions            elif re.findall('/.+?/.+\.(.+)',file1)[0] not in supportedextensions or re.findall('/.+?/.+\.(.+)',file2)[0] not in supportedextensions:                print('Error: File format currently not supported. At the moment only csv, xlsx, xlsm and json files are supported.')                isitago = 0            elif file1 == file2:                print('Error: The files need to be different')                isitago = 0            elif isitago == 1:                print('Info: Filepaths correctly defined.')                # check if files exist                try:                    print('Info: Attempting to access files.')                    if re.findall('/.+?/.+\.(.+)',file1)[0] == 'csv':                        df1, df2 = pd.read_csv(file1), pd.read_csv(file2)                    elif re.findall('/.+?/.+\.(.+)',file1)[0] == 'json':                        df1, df2 = pd.read_json(file1), pd.read_json(file2)                    elif re.findall('/.+?/.+\.(.+)',file1)[0] in  ['xlsx', 'xlsm']:                        df1, df2 = pd.read_excel(file1), pd.read_excel(file2)                    else:                        print('How did we get here?')                    proceedwithfindcommonkeys = 1                except IOError:                    print("Error: File not accessible.")                    proceedwithfindcommonkeys = 0                except UnicodeDecodeError:                    print("Error: File includes a unicode character that cannot be decoded with the default UTF decryption.")                    proceedwithfindcommonkeys = 0                except Exception as e:                    print('Error: ', e)                    proceedwithfindcommonkeys = 0        else:            print('Error: Please choose 2 files.')        if proceedwithfindcommonkeys == 1:            keyslist1 = [] #This will be the list of headers from first file            keyslist2 = [] #This will be the list of headers from second file            keyslist = [] #This will be the list of headers that are the intersection between the two files            formlists = [] #This will be the list to be displayed on the UI            for header in df1.columns:                if header not in keyslist1:                    keyslist1.append(header)            for header in df2.columns:                if header not in keyslist2:                    keyslist2.append(header)            for item in keyslist1:                if item in keyslist2:                    keyslist.append(item)            if len(keyslist) == 0:                print('Error: Files have no common headers.')                secondwindow = 0            else:                window.close()                secondwindow = 1                break################################################## First screen completed, moving on to second oneif secondwindow != 1:    exit()#To align the three columns on the UI, we need the max len#Note: This could be made better by having the max len of each columnmaxlen = 0for header in keyslist:    if len(str(header)) > maxlen:        maxlen = len(str(header))if maxlen > 25:    maxlen = 25elif maxlen < 10:    maxlen = 15#we need to split the keys to four columnsfor index,item in enumerate(keyslist):    if index == 0: i =0    if len(keyslist) >= 4 and i == 0:        formlists.append([sg.Checkbox(keyslist[i], size=(maxlen,None)),sg.Checkbox(keyslist[i+1], size=(maxlen,None)),sg.Checkbox(keyslist[i+2], size=(maxlen,None)),sg.Checkbox(keyslist[i+3], size=(maxlen,None))])        i += 4    elif len(keyslist) > i:        if len(keyslist) - i - 4>= 0:            formlists.append([sg.Checkbox(keyslist[i], size=(maxlen,None)),sg.Checkbox(keyslist[i+1], size=(maxlen,None)),sg.Checkbox(keyslist[i+2], size=(maxlen,None)),sg.Checkbox(keyslist[i+3], size=(maxlen,None))])            i += 4        elif len(keyslist) - i - 3>= 0:            formlists.append([sg.Checkbox(keyslist[i], size=(maxlen,None)),sg.Checkbox(keyslist[i+1], size=(maxlen,None)),sg.Checkbox(keyslist[i+2], size=(maxlen,None))])            i += 3        elif len(keyslist)- i - 2>= 0:            formlists.append([sg.Checkbox(keyslist[i], size=(maxlen,None)),sg.Checkbox(keyslist[i+1], size=(maxlen,None))])            i += 2        elif len(keyslist) - i - 1>= 0:            formlists.append([sg.Checkbox(keyslist[i], size=(maxlen,None))])            i += 1        else:            sg.Popup('Error: Uh-oh, something\'s gone wrong!')    #The second UIlayoutpostfile = [    [sg.Text('File 1'), sg.InputText(file1,disabled = True, size = (75,2))],    [sg.Text('File 2'), sg.InputText(file2,disabled = True, size = (75,2))],    #[sg.Text('Select the data key for the comparison:')],    [sg.Frame(layout=[        *formlists],title = 'Select the Data Key for Comparison',relief=sg.RELIEF_RIDGE    )],    [sg.Output(size=(maxlen*6, 20))],    [sg.Submit('Compare'), sg.Cancel('Exit')]]window2 = sg.Window('File Compare', layoutpostfile)datakeydefined = 0definedkey = []while True:  # The Event Loop    event, values = window2.read()    # print(event, values)  # debug    if event in (None, 'Exit', 'Cancel'):        break    elif event == 'Compare':        definedkey.clear()        file1test = file2test = isitago = None        #print('Event', event, '\n', 'Values', values)        for index, value in enumerate(values):            if index not in [0,1]:                if values[index] == True:                     datakeydefined = 1                    definedkey.append(keyslist[index-2])            #print(index, values[index], keyslist[index-2])        if len(definedkey) > 0:            compare = datacompy.Compare(                    df1,                    df2,                    join_columns=definedkey,  #You can also specify a list of columns eg ['policyID','statecode']                    abs_tol=0, #Optional, defaults to 0                    rel_tol=0, #Optional, defaults to 0                    df1_name='Original', #Optional, defaults to 'df1'                    df2_name='New' #Optional, defaults to 'df2'            )            print('########################################################################################################')            print(compare.report())        else:            print('Error: You need to select at least one attribute as a data key')
There are quite a few limitations to this solution, but it’s one that can be enhanced quite easily and quickly in the future. The following items would be nice to haves:
It would be good if the CSV delimiter was to be defined rather than assume a ‘,’ at the momentSizing of the window is too precarious, and using very long headers could create problemsEnable the system to remember key choices based on file namesFull directory reconciliations
It would be good if the CSV delimiter was to be defined rather than assume a ‘,’ at the moment
Sizing of the window is too precarious, and using very long headers could create problems
Enable the system to remember key choices based on file names
Full directory reconciliations
I am sure there are a lot more, but do let me know what you think!