Automate your Data Management Discipline with Python | by Garrett Eichhorn | Towards Data Science

Anyone that interacts with data understands the importance of data quality. Whether you’re performing basic analysis in excel or forecasting sales trends with decision trees, poor data will always rear its ugly head and curtail your progress. It’s easy to place the onus on the overworked data steward and the gluttonous data consumer, but it’s ultimately the data professional’s burden to bear as the liaison between the two. So, how can you elevate the data management competency in your enterprise and advocate for more effective analysis? Automate your data quality metrics.
Data quality metrics are defined in many ways, by many people. I like to use the 4 V’s and the 4 C’s to frame conversations about quality, because they’re easy to remember, market and visualize:
Completeness — How much of the data that’s expected is there? Are the ‘important’ columns filled out?
Consistency — Are data values the same across data sets?
Conformity — Does the data comply with specified formats?
Curiosity — Are stakeholders knowledgeable and/or engaged in the data management lifecycle?
Volume — How much information is present?
Velocity — What’s the frequency of incoming data? How often is the data processed and propagated?
Variety — What types of data exist? Is the data structured, unstructured or semi-structured?
Veracity — Is the data trustworthy? What are the inherent discrepancies?
There are a few Python packages I use to quickly measure and socialize these concepts. Pandas has many built in methods to for investigating the data: I use .describe() with conditional formatting to find patterns. For a more robust report I use pandas-profiling, which is easily embedded in a Jupyter Notebook or saved to an HTML file.
import pandas as pdimport pandas_profilingdf = read_csv("raw_data.csv")df.describe()profile = pandas_profiling.ProfileReport(df)profile.to_file("profile.html")
Matplotlib and seaborn are great libraries for plotting data. I’ve recently been exploring plotly for creating quick, versatile and engaging visualizations that can be easily embedded in a website. Here’s a great guide on how to do just that.
So far we’ve examined some trite theories to build a conceptual framework for socializing data quality, and have basic tools to visualize and present metrics. It’s time to automate! As data professionals, it’s critical that we educate and motivate the enterprise to manage data for effective analysis, by exposing dirty data and articulating data debt. Automation can solidify best-practice into discipline by sustaining transparency and accountability.
I recently worked on a project that focused heavily on data ingestion to calculate profit margins for global products, grouped by franchise. The effort started as an executive order to better understand profitability, but quickly spiraled out of control. Like most enterprises with a global footprint, my client delivers strategic growth by purchasing companies that will minimize operating costs (like distributors) to streamline the supply chain.
IT consolidation couldn’t quite catch up to the operating plan; previous efforts to calculate margin proved unsuccessful because of disparate ERP systems (more than 15 in total), discombobulated nomenclature, high steward turnover, etc.
Data stewards from across the globe would send individual .csv files via email to the Business Intelligence team, to be merged manually with invoice sales data in an enterprise data warehouse. Pre-packaged reports were passed on to leadership with innumerable caveats, to be undermined by data stewards convinced other stakeholders were diluting the data. Data quality was so disorganized that visibility into basic margin analysis could not be reached, much less sustained. The burnt-out B.I group needed some help!
Data completeness and conformity were creating major problems for the margin calculation: incomplete data would flood in from the stewards without structure and in many different formats. Velocity was inconsistent, flowing infrequently from various groups, where poor reputation dictated low veracity.
Eventually, I would be contributing to the project by performing advanced analysis on margin calculations to interpret profitability and market share. But without effective data management for calculating profit margin to run through the data science pipeline, I had lots of opportunity to automate an effective pipeline and inform stakeholders on data quality.
We collaborated with data stewards, owners and a governance council to identify only the relevant values needed for margin calculation, organized by function: product-to-product mappings, cost, unit of measure conversion, and additional attributes. The templates were replicated to fit each global franchise, so that a data steward was only responsible for maintaining their asset. In total, we would have more than 50 files!
I built a SharePoint site to host the templates in Document Repositories, which is the SharePoint equivalent of a directory folder. I then created a simple python script to iterate through the “folders” to concatenate the files by function, adding additional columns like “Date Modified”, “Initial File Name”, and a “Completeness” metric that calculated how many columns were filled out relative to the total (by row). The more than 50 files reduced to 4, representing each of the functional data assets. In order to evangelize data quality concepts, I used pandas-profiling to build data quality reports by function and embed on the SharePoint site.
The B.I team was tired of manually uploading flat files to the data warehouse. I employed pyODBC to establish a connection to the enterprise SQL Server, and loaded only the records that fulfilled an adequate “Completeness” score.
import pyodbcconn = pyodbc.connect('Driver={SQL Server};'                          'Server=<server>;'                          'Database=<database>;'                          'Trusted_Connection=yes;')cursor = conn.cursor()sql_query = "SELECT * FROM table_name"cursor.execute(sql_query)cursor.fetchall()sql_df = pd.read_sql(sql_query, conn)
I can query the records once they’re loaded into the data warehouse and check for conformity by cross-referencing the data with information_schema and column constraints; if a column has a VARCHAR limit of 5 and the value has length equal to 10, the record won’t be loaded. I stored this “error log” on the SharePoint site for data stewards to reference as needed.
Finally, we compute margin for the records loaded into the SQL Server and store the result in a plotly graph to provide immediate feedback on franchise performance. A beautiful, animated visualization describes expected records, provided records, loaded records, records with margin, and percentage of total revenue over time. Leaders can quickly view which data stewards are actively contributing quality data, which aren’t, and where the major discrepancies are occurring. The end result is a consolidated platform for consuming profit margins by franchise with data quality metrics updated in real-time, fed by an automated pipeline. Data professionals can focus on turning insights into action and less on “herding the cats”.
Data management is terminally less sexy than analysis and advanced techniques, but is just as important. Poor data will distract you from insightful analysis and plague talented data professionals in your enterprise if managed improperly. Although oversimplified, the methodology outlined in this article articulates how people, process, data and technology can interact to produce an integral metric for the enterprise. By focusing on the fundamentals of the data management lifecycle before skipping to “the fun stuff”, the project team can quickly pivot from ingesting the data to understanding the data.
In total, my contributions will save the enterprise more than 400 hours of manual work on an annual basis. Furthermore, we’ve contributed to a culture devoid of data management and helped coalesce data quality strategy along the way. Start automating your data management discipline!