Package: meteo-qc
meteo-qcmeteo_qcis a customizable framework for applying quality checks to meteorological
data. The framework can be easily extended by registering custom functions/plugins.InstallationTo installmeteo-qc, open an interactive shell and runpip install meteo-qcGetting startedCheck out theDocumentationfor
detailed information.Apply the quality control to this csv data calledtest_data.csv:date,temp,pressure_reduced
2022-01-01 10:00:00,1,600
2022-01-01 10:10:00,2,1024
2022-01-01 10:20:00,3,1024
2022-01-01 10:30:00,4,1090
2022-01-01 10:50:00,4,
2022-01-01 11:00:00,,1024
2022-01-01 11:10:00,2,1024
2022-01-01 11:20:00,3,1024
2022-01-01 11:30:00,4,1090
2022-01-01 11:40:00,4,1090Read in the data as apd.DataFrame.Create ameteo_qc.ColumnMappingobject and use the column names as keys to use the methodadd_groupto add
the column to the group
(temperatureorpressure).
This can be an existing group or a new group.Callmeteo_qc.apply_qcto apply the control to the DataFramedatausing thecolumn_mappingas a
definition for the checks to be applied.importpandasaspdimportmeteo_qc# read in the datadata=pd.read_csv('test_data.csv',index_col=0,parse_dates=True)# map the columns to groupscolumn_mapping=meteo_qc.ColumnMapping()column_mapping['temp'].add_group('temperature')column_mapping['pressure_reduced'].add_group('pressure')# apply the quality controlresult=meteo_qc.apply_qc(df=data,column_mapping=column_mapping)print(result)This will result in this object which can be used to display the result in a
nice way e.g. using anhtmltemplate to render it.{'columns':defaultdict(<functionapply_qc.<locals>.<lambda>at0x7f9b0edd5480>,{'temp':{'results':{'missing_timestamps':Result(function='missing_timestamps',passed=False,msg='missing 1 timestamps (assumed frequency: 10min)',data=None,),'null_values':Result(function='null_values',passed=False,msg='found 1 values that are null',data=[[1641034800000,None,True]],),'range_check':Result(function='range_check',passed=True,msg=None,data=None,),'spike_dip_check':Result(function='spike_dip_check',passed=True,msg=None,data=None,),'persistence_check':Result(function='persistence_check',passed=True,msg=None,data=None,)},'passed':False,},'pressure_reduced':{'results':{'missing_timestamps':Result(function='missing_timestamps',passed=False,msg='missing 1 timestamps (assumed frequency: 10min)',data=None,),'null_values':Result(function='null_values',passed=False,msg='found 1 values that are null',data=[[1641034200000,None,True]],),'range_check':Result(function='range_check',passed=False,msg='out of allowed range of [860 - 1055]',data=[[1641031200000,600.0,True],[1641033000000,1090.0,True],[1641036600000,1090.0,True],[1641037200000,1090.0,True]],),'spike_dip_check':Result(function='spike_dip_check',passed=False,msg='spikes or dips detected. Exceeded allowed delta of 0.3 / min',data=[[1641031800000,1024.0,True],[1641033000000,1090.0,True],[1641034200000,None,True],[1641036600000,1090.0,True]],),'persistence_check':Result(function='persistence_check',passed=True,msg=None,data=None,)},'passed':False}}),'passed':False,'data_start_date':1641031200000,'data_end_date':1641037200000,}It is also possible to write and register your own functions if they are not
already in the predefinedGroups.
Please check out theDocsfor
more information.