Package: harmonize-wq
harmonize-wqStandardize, clean, and wrangle Water Quality Portal data into more analytic-ready formatsUS EPA’sWater Quality Portal (WQP)aggregates water quality, biological, and physical data provided by many organizations and has become an essential resource with tools to query and retrieval data usingpythonorR. Given the variety of data and variety of data originators, using the data in analysis often requires data cleaning to ensure it meets the required quality standards and data wrangling to get it in a more analytic-ready format.  Recognizing the definition of analysis-ready varies depending on the analysis, the harmonixe_wq package is intended to be a flexible water quality specific framework to help:Identify differences in data units (including speciation and basis)Identify differences in sampling or analytic methodsResolve data errors using transparent assumptionsTransform data from long to wide formatDomain experts must decide what data meets their quality standards for data comparability and any thresholds for acceptance or rejection.For more complete tutorial information, see:demosQuick Startharmonize_wq can be installed using pip:$ python3 -m pip install harmonize-wqTo install the latest development version of harmonize_wq using pip:pip install git+https://github.com/USEPA/harmonize-wq.gitExample Workflowdataretrieval Query for a geojsonimportdataretrieval.wqpaswqpfromharmonize_wqimportwrangle# File for area of interestaoi_url=r'https://raw.githubusercontent.com/USEPA/harmonize-wq/main/harmonize_wq/tests/data/PPBays_NCCA.geojson'# Build queryquery={'characteristicName':['Temperature, water','Depth, Secchi disk depth',]}query['bBox']=wrangle.get_bounding_box(aoi_url)query['dataProfile']='narrowResult'# Run queryres_narrow,md_narrow=wqp.get_results(**query)# dataframe of downloaded resultsres_narrowHarmonize resultsfromharmonize_wqimportharmonize# Harmonize all resultsdf_harmonized=harmonize.harmonize_all(res_narrow,errors='raise')df_harmonizedClean resultsfromharmonize_wqimportclean# Clean up other columns of datadf_cleaned=clean.datetime(df_harmonized)# datetimedf_cleaned=clean.harmonize_depth(df_cleaned)# Sample depthdf_cleanedTransform results from long to wide formatThere are many columns in the dataframe that are characteristic specific, that is they have different values for the same sample depending on the characteristic. To ensure one result for each sample after the transformation of the data these columns must either be split, generating a new column for each characteristic with values, or moved out from the table if not being used.fromharmonize_wqimportwrangle# Split QA column into multiple characteristic specific QA columnsdf_full=wrangle.split_col(df_cleaned)# Divide table into columns of interest (main_df) and characteristic specific metadata (chars_df)main_df,chars_df=wrangle.split_table(df_full)# Combine rows with the same sample organization, activity, location, and datetimedf_wide=wrangle.collapse_results(main_df)The number of columns in the resulting table is greatly reducedOutput ColumnTypeSourceChangesMonitoringLocationIdentifierDefines rowMonitoringLocationIdentifierNAActivity_datetimeDefines rowActivityStartDate, ActivityStartTime/Time, ActivityStartTime/TimeZoneCodeCombined and UTCActivityIdentifierDefines rowActivityIdentifierNAOrganizationIdentifierDefines rowOrganizationIdentifierNAOrganizationFormalNameMetadataOrganizationFormalNameNAProviderNameMetadataProviderNameNAStartDateMetadataActivityStartDatePreserves date where time NATDepthMetadataResultDepthHeightMeasure/MeasureValue, ResultDepthHeightMeasure/MeasureUnitCodestandardized to metersSecchiResultResultMeasureValue, ResultMeasure/MeasureUnitCodestandardized to metersQA_SecchiQANAharmonization processing quality issuesTemperatureResultResultMeasureValue, ResultMeasure/MeasureUnitCodestandardized to degrees CelciusQA_TemperatureQANAharmonization processing quality issuesIssue Trackerharmonize_wq is under development. Please report any bugs and enhancement ideas using the issue track:https://github.com/USEPA/harmonize-wq/issuesDisclaimerThe United States Environmental Protection Agency (EPA) GitHub project code is provided on an "as is" basis and the user assumes responsibility for its use.  EPA has relinquished control of the information and no longer has responsibility to protect the integrity , confidentiality, or availability of the information.  Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by EPA.  The EPA seal and logo shall not be used in any manner to imply endorsement of any commercial product or activity by EPA or the United States Government.