How to Write Scripts That Check Data Quality For You
By: Peter Grant
Tags: Data Science, Science, Programming, Python, Tutorial

Tutorial: Automatically Analyzing Laboratory Data to Create a Performance Map

Python scripts can check your laboratory data for errors, sans boredom and tedium

It’s very common that scientists find themselves with large data sets. Sometimes it comes in the form of gigabytes worth of data in a single file. Other times it’s hundreds of files, each containing a small amount of data. Either way, it’s hard to manage. Hard to make sense of. Hard for your computer to process. You need a way to simplify the process, to make the data set more manageable, and to help you keep track of everything.

That’s what this tutorial is all about. We’re in the process of writing Python scripts that will automatically analyze all of the data for you and store the data with meaningful, intuitive file names. All while using an example taken from actual research, so you know that the skills you’re developing are practical and useful.

The first post in this tutorial introduced the concepts of the tutorial. If the terms “heat pump water heater”, “coefficient of performance (COP)”, and “performance map” don’t mean anything to you, you might want to read it.

The second post introduced the companion data set, and split the data set into multiple files with user-friendly names.

The companion data set is a valuable part of the tutorial process, as it allows you to follow along. You can write the exact same code that I’ll present, run the code, see the results, and compare it to results I present. In that way, you can ensure that you’re doing it right.

In the third part of the tutorial we created a script that analyzed each of the individual data files. In this process we 1) Filtered the data to only examine the relevant portions, 2) Identified the timestep in the data, 3) Calculated the change in stored energy in the tank during each timestep, 4) Calculated the COP of the heat pump at each timestep, 5) Generated regressions expressing the COP as a function of the water temperature in each test, and 6) Saved the results for future use.

The fourth part showed you how to create plots of the data, and how to examine those plots to ensure that 1) The data provided by the laboratory was useful, and 2) The data analysis process was correct. While this process was valuable, it was also manual. This is a tutorial teaching you to write programs to do that work for you!

That is what we will cover in this part of the tutorial. We will amend our existing script, created step by step in the previous articles, to include logic that automatically identifies tests which probably have some errors in them. The filenames of these tests will be saved in a .csv file, which we can check to see which tests should be examined further.

What is the general process for automatically error checking data sets?

This process consists of five steps:

We need to create a dataframe file to store the information about potentially erroneous tests.

We also need to input expected values for each parameter, as well as thresholds for how much difference we will accept.

Then, after the program analyzes each test, we need to create a series of if statements checking to see if the data falls within or without of the accepted ranges.

If the data falls outside of the accepted range, we need to add information about that test to the dataframe file describing potentially erroneous tests.

Finally, we need to save the datframe as a .csv file to make the information available for use after the analysis is completed.

If this seems abstract, don’t worry. It will all make sense after we walk through an example.

How is the dataframe storing information created?

This dataframe is a standard pandas dataframe. We merely need to create the dataframe and specify the needed columns. Then it will be available as a structure so that we can append information to it later.

The important part of creating this data frame is to ensure that we have the correct headers. We will need a column for each piece of information that we want to include in the dataframe. I’ve personally found that the useful columns for this are:

Filename : This presents the name of the file with the suspicious results. Including this information tells us which tests to give further review after the analysis is completed.

: This presents the name of the file with the suspicious results. Including this information tells us which tests to give further review after the analysis is completed. Test Parameters : This is not always necessary. If you include the test parameters in descriptive filenames, as was described in Automatically Storing Results from Analyzed Data Sets, then this information will be included in the filename. If not, then including this column provides more information about the potentially erroneous test. This column will be included in the tutorial in case you want to use it in future analyses.

: This is not always necessary. If you include the test parameters in descriptive filenames, as was described in Automatically Storing Results from Analyzed Data Sets, then this information will be included in the filename. If not, then including this column provides more information about the potentially erroneous test. This column will be included in the tutorial in case you want to use it in future analyses. Code : This creates a column used to store a code stating what error was identified. For instance, if a test including temperature measurements that were out of spec this column could say something like “Water Temperature”.

: This creates a column used to store a code stating what error was identified. For instance, if a test including temperature measurements that were out of spec this column could say something like “Water Temperature”. Value: This column reports the value of the data that fell outside of the accepted range. You can use it to gauge whether the data was far outside of the range, or just barely beyond the specification.

With this knowledge, the code to create the dataframe is as follows:

SuspiciousTests_Test = pd.DataFrame(columns = ['Filename', 'Test Parameters', 'Code', 'Value'])

Note that this is being added to the script we’ve used previously, and that pandas has already been imported as pd.

What expected and threshold values do we need to add?

From a big picture point of view, we need to add all of the expected and threshold values needed to check the quality of the data. If we’re concerned about air temperature, we need and expected and threshold value for that. If we’re concerned about electricity consumption, we need values for that. The important point here is that you must think through the points that are most important for your project and provide the information necessary to check those data sets.

We can make things more concrete for this tutorial. In the third part of this tutorial, where we used plots to visually check the data, we checked the 1) Starting water temperature, 2) Final water temperature, and 3) Average ambient temperature. We also checked the regression quality, but that isn’t an easy item to check so it will be left out of this tutorial. We will also add a check for the maximum electricity consumption.

To complete these checks automatically we need to specify the expected and threshold values for:

Initial water temperature,

Final water temperature,

Water temperature standard deviation,

Ambient temperature,

Average temperature standard deviation, and

Maximum electricity demand.

We can specify the needed values with the following code:

#Constants describing the tests

Temperature_Water_Initial_Setpoint = 72

Temperature_Water_Final_Setpoint = 140 #Test rejection thresholds

Threshold_Temperature_Water_Initial = 1.8

Threshold_Temperature_Water_Final = 1.8

Threshold_StDev_Temperature_Water = 0.5

Threshold_Temperature_Ambient = 2

Threshold_StDev_Temperature_Ambient = 1

Threshold_P_Elec = 1000

Notice that no expected value for the ambient temperature was specified. This is because the expected value varies across tests, and it is read out of the data filename.

How do we write if statements to check for errors?

This is likely the most straightforward part of the process. We only need to write if statements comparing the values identified in the data to the values specified above. If the data falls out of the accepted range, then the if statement will be true and a potential error will be identified.

Before programming the if statements, we need to identify the data needed to process them. Some of this information is read from the filename (Specifically, the ambient temperature setpoint) and some of it is calculated from the data.

The ambient temperature setpoint can be read from the filename as follows:

Temperature_Ambient_Set = float(Filename[-6:-4])

Then the ambient temperature average and standard deviation can be calculated as follows:

Temperature_Ambient_Avg = Data['T_Amb (deg F)'].mean()

Temperature_Ambient_StDev = np.std(Data['T_Amb (deg F)'])

The maximum electricity can be read out of the data file with:

P_Elec_Max = Data['P_Elec (W)'].max()

The initial and final water temperatures can be read from the data files as well by using:

Temperature_Water_Initial = Data.loc[0, 'Average Tank Temperature (deg F)']

Temperature_Water_Final = Data.loc[Data.index.max(), 'Average Tank Temperature (deg F)']

The standard deviation of water temperature in the tank is a bit harder. It can be calculated using the .std() pandas function, but that requires it to be in a separate dataframe as the function includes all columns in the entire dataframe. Thus, it can be calculated using the following lines of code:

StDev_Calculation = pd.DataFrame()

StDev_Calculation['T1 (deg F)'] = Data['T1 (deg F)']

StDev_Calculation['T2 (deg F)'] = Data['T2 (deg F)']

StDev_Calculation['T3 (deg F)'] = Data['T3 (deg F)']

StDev_Calculation['T4 (deg F)'] = Data['T4 (deg F)']

StDev_Calculation['T5 (deg F)'] = Data['T5 (deg F)']

StDev_Calculation['T6 (deg F)'] = Data['T6 (deg F)']

StDev_Calculation['T7 (deg F)'] = Data['T7 (deg F)']

StDev_Calculation['T8 (deg F)'] = Data['T8 (deg F)']

StDev_Calculation['Standard Deviation'] = StDev_Calculation.std(axis = 1)

StDev_Max = StDev_Calculation['Standard Deviation'].max()

Now that all of the values are identified and available, the next step is inserting the if statements. Remember that this is a matter of creating an if statement to check for each potential error. This way if the script identifies a potential error it can tell which error it is and provide the most useful information in the results. The if statements can be programmed using:

if (abs(Temperature_Water_Initial - Temperature_Water_Initial_Setpoint) > Threshold_Temperature_Water_Initial):

if (abs(Temperature_Water_Final - Temperature_Water_Final_Setpoint) > Threshold_Temperature_Water_Final):

if (StDev_Max > Threshold_StDev_Temperature_Water):

if (abs(Temperature_Ambient_Set - Temperature_Ambient_Avg) > Threshold_Temperature_Ambient):

if (Temperature_Ambient_StDev > Threshold_StDev_Temperature_Ambient):

if (P_Elec_Max > Threshold_P_Elec):

These if statements provide the structure necessary to check for potential errors in the data. The next section will add content within each of those if statements to provide useful data describing the problems identified.

How do we describe the potential errors?

Each of the previous if statements needs four lines of code to describe the identified potential problem. Two of them are repeatable and can be identified using the exact same line every time. They are the filename and the ambient temperature conditions of the test. They can be added to the results outside of the if statements.

The best way that I’ve found to do this process is to create an identical, temporary, dataframe to hold the information, then append it to the real dataframe. The temporary dataframe, containing the filename and ambient temperature setpoint, can be created using the following code:

Temp = pd.DataFrame(columns = ['Filename', 'Test Parameters', 'Code', 'Value'])

Temp.loc[0, 'Filename'] = Filename

Temp.loc[0, 'Test Parameters'] = 'T_Amb = ' + str(Temperature_Ambient_Set)

This creates a temporary dataframe that can be appended to SuspiciousTests and contains the filename and ambient temperature in the first row.

After that it’s a matter of adding content within each if statement to 1) State the appropriate code for that error, 2) State the value of the data that is outside of the bounds, and 3) Append the data to the SuspiciousTests dataframe. This is done for the initial water temperature with the following code:

Temp.loc[0, 'Code'] = 'T_Water_Initial'

Temp.loc[0, 'Value'] = str(Temperature_Water_Initial - Temperature_Water_Initial_Setpoint)

SuspiciousTests = SuspiciousTests.append(Temp)

For the final water temperature:

Temp.loc[0, 'Code'] = 'T_Water_Final'

Temp.loc[0, 'Value'] = str(Temperature_Water_Final - Temperature_Water_Final_Setpoint)

SuspiciousTests = SuspiciousTests.append(Temp)

For the standard deviation of the water temperature:

Temp.loc[0, 'Code'] = 'T_Water_StDev'

Temp.loc[0, 'Value'] = str(StDev_Max)

SuspiciousTests = SuspiciousTests.append(Temp)

For the average ambient temperature:

Temp.loc[0, 'Code'] = 'T_Amb_Avg'

Temp.loc[0, 'Value'] = str(Temperature_Ambient_Avg - Temperature_Ambient_Avg)

SuspiciousTests = SuspiciousTests.append(Temp)

For the standard deviation of ambient temperature:

Temp.loc[0, 'Code'] = 'T_Amb_StDev'

Temp.loc[0, 'Value'] = str(Temperature_Ambient_StDev)

SuspiciousTests = SuspiciousTests.append(Temp)

And for the electric power demand:

Temp.loc[0, 'Code'] = 'P_Elec_Max'

Temp.loc[0, 'Value'] = str(P_Elec_Max)

SuspiciousTests = SuspiciousTests.append(Temp)

How do we save the file?

Saving the file is easily done using the techniques described in Automatically Storing Results from Analyzed Data Sets. In this case we want to save a single dataframe as a .csv file, making the process much simpler. To save the file to a .csv named SuspiciousTests in the same folder that we referenced as Path earlier in the tutorial, use:

SuspiciousTests.to_csv(Path + '\SuspiciousTests.csv', index = False)

Wrapping it up

Now when you use your script to analyze all of the experimental data and generate the regressions the script will also identify potential problems in the tests or in the data analysis. This information will be available in the SuspiciousTests dataframe for later review, giving you guidance on which data files you should review more thoroughly. That sounds much better than manually checking every single one!

Now that we’ve certified that all of our laboratory data and data analysis is sound, we can move on to making use of the data. Since the goal of this tutorial is to create a performance map predicting the COP of a heat pump, as described in the introduction, we’ll move on to doing exactly that. This means creating a 2-dimensional regression stating the COP as a function of both the water and ambient temperatures. We’ll cover that topic in the next article.

Tutorial Table of Contents

This is a part of a series of articles teaching you all of the skills needed to automatically analyze laboratory data and develop a performance map of heat pump water heaters. The other articles in the series can be found using the following links:

Introduction

Splitting Data Sets

Automatically Analyzing Laboratory Test Data

Checking Analyzed Laboratory Data for Errors

How to Automatically Generate Regressions in Python