Package: wqxweblib
Python library for WQX Web (wqxweblib)This is a simple Python library for interacting with the EPA's WQX Web data submission service. It aims to mimick theWQX Web API endpointsas much as possible with theWQXWebclass. If you find a bug, please submit a pull request onGithubor open an issue there and someone will look into it.NOTE: This module is useless without EPA credentials for use with the CDX app "WQX Web".This code is supported by the National Science Foundation under Award No. OIA-1458952. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.ContentsPython library for WQX Web (wqxweblib)ContentsGetting StartedPackage VersioningWQXWeb Data Submission PatternsImport WQXWeb ModulePresentations and PublicationsWQXWeb API ReferenceUploadUploadAttachmentStartImportStartXmlExportSubmitDatasetToCdxSubmitFileToCdxGetStatusGetDocumentListProjectsMonitoringLocationsGetting StartedInstall this package usingpippip install wqxweblibPackage VersioningThis package uses a five-segment version number. The first two segments of the version number indicate the version of WQX Web it supports. The last three segments follow theSemantic Versioningstandard and are specific to only this package, not any WQX Web release. They are in the formatx.y.zwherexis the major version number,yis the minor version number, andzis the patch version number.The major version number increments when incompatible API changes are made. The minor version number increments when functionality is added in a backward compatible manner. The patch version number increments when there are backward compatible bug fixes.It is a good practice to lock the first three digits of the version in yourrequirements.txtfile, if you are using one. That allow you to safely rebuild your environment without accidentally introducing breaking changes. For the current version, that would mean adding a line to yourrequirements.txtfile like this:wqxweblib<2.2.1WQXWeb Data Submission PatternsBasic Method- Use this if you trust the data you produce.Upload a file usingUpload.(optional) Upload an attachment file usingUploadAttachment.Start the import with auto export and auto submit usingStartImport.Periodically check the status usingGetStatus.Intermediate Method- Use this if you want to check how WQX Web responds to your data before submitting.Upload a file usingUpload.(optional) Upload an attachment file usingUploadAttachment.Start the import with auto export and without auto submit usingStartImport.Periodically check the status usingGetStatus.Submit dataset to CDX usingSubmitDatasetToCdx.Periodically check the status usingGetStatus.Advanced Method- Use this method if you want to manage each step of the process that you can.Upload a file usingUpload.(optional) Upload an attachment file usingUploadAttachment.Start the import without auto export and without auto submit usingStartImport.Periodically check the status usingGetStatus.Start the XML Export usingStartXmlExport.Periodically check the status usingGetStatus.Submit dataset to CDX usingSubmitDatasetToCdx.Periodically check the status usingGetStatus.Import WQXWeb ModuleThis module must be imported and instantiated with youruserIDandprivateKeybefore any of the other functions can be used.from wqxweblib import WQXWeb

wqxweb = WQXWeb("my username", "my private key")Presentations and PublicationsMusselwhite, J and F Rojano. 2021. Development of a Python Library to Facilitate WQX Dataset Exchanges. IWSS Spring Workshop Series (Virtual).PDFWQXWeb API ReferenceThis following methods mirror what is shown in the EPA's documentation for the WQX Web API endpoints. Some differences exist for the sake of Python integration while still allowing the inputs indicated in the documentation.UploadUpload a file to the web server (to be imported).Parameters:filename(required string) - A name to give the file. This does not need to match any local filename. The allowed file extensions are:txtcsvxlsxxlsxmlzipcontents(required bytes) - The contents of the file to be uploaded provided either by converting some provideddatastring with thebytes(data, 'utf-8')function or by passing somefileobject'sfile.read()function.Returns:fileId- A unique identifier for the uploaded file.Example with generated data:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

data="""column1,column2
value1,value2"""

fileId = wqxweb.Upload( filename="datafile.csv", content=bytes(data,'utf-8') )

print( f"The uploaded file has been assigned fileId {fileId}." )Example using a file from the local filesystem:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

with open( 'data.csv', 'rb' ) as f:
    fileId = wqxweb.Upload( filename="datafile.csv", contents=f.read() )

    print( f"The uploaded file has been assigned fileId {fileId}." )UploadAttachmentUpload an attachment to the web server (to be imported).Parameters:filename(required string) - A name to give the file. This does not need to match any local filename. The only allowed file extension is:zipcontents(required bytes) - The contents of the file to be uploaded provided either by converting some provideddatastring with thebytes(data, 'utf-8')function or by passing somefileobject'sfile.read()function.Returns:fileId- A unique identifier for the uploaded file.Example using a file from the local filesystem:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

with open( 'attachment.zip', 'rb' ) as f:
    attachmentFileId = wqxweb.UploadAttachment( filename="attach.zip", contents=f.read() )

    print( f"The uploaded file has been assigned attachmentFileID {attachmentFileID}." )StartImportStart importing a file and attachment that was previously uploaded.Parameters:importConfigurationId(required string) - The Import Configuration ID of an existing Import Configuration which should be applied to the import.fileId(required string) - The return value ofUpload.attachmentFileId(optional string) - The return value ofUploadAttachment.fileType(required enum or string) - The type of file you uploaded withUpload. This must be one of the following enum members provided by theWQXWebmodule or their corresponding strings:CSVTABTILDEPIPEXLSXLSXnewOrExistingData(required enum or integer) - Declare whether the contents of your upload represents new data or replaces existing data. This must be one of the following enum members provided by theWQXWebmodule or their corresponding integers:CONTAINS_NEW_OR_EXISTING(0) - file may contain new and/or existing data.CONTAINS_NEW_ONLY(1) - file contains new data only.CONTAINS_EXISTING_ONLY(2) - file contains existing data only (to be replaced).uponCompletion(required enum or integer) - Declare what to do after the upload finishes. This must be one of the following enum members provided by theWQXWebmodule or their corresponding integers:DO_NOT_EXPORT(0) - do nothing.EXPORT_IMPORT(1) - start export.SUBMIT_IMPORT(2) - start export and submit to CDX.uponCompletionCondition(optionally required enum or integer) - Declare what conditions permit auto export or auto submit. This must be one of the following enum members provided by theWQXWebmodule or their corresponding integers:NOT_APPLICABLE(0) - not applicable (meaninguponCompletionisDO_NOTHINGor not provided).EXPORT_IF_NO_ERROR(1) - start export only if no import errors.EXPORT_IF_NO_WARNING(2) - start export only if no import errors and no warnings.EXPORT_ALWAYS(3) - start export even when there are import errors.worksheetsToImport(optional string) - If provided, this must be a comma delimited list of values (1-based), e.g. "1,3". This parameter value will be ignored when the fileType is notXLSorXLSX. If no value is passed in (and it's applicable) then we'll use the value from the Import Configuration.ignoreFirstRowOfFile(optional boolean defaults to True) - Ignore the first row of the data file if this parameter is true. This parameter value will be ignored for Expert Mode Import Configurations.generatedElements(optional dictionary) - Each key must match an existing generated element name from the import configuration. No more than five keys can be in the dictionary and this parameter cannot be used along with thegeneratedElementName1,generatedElementName2,generatedElementName3,generatedElementName4, orgeneratedElementName5parameters.generatedElementName1(optional string) - Must match an existing generated element name from the import configuration.generatedElementValue1(optionally required string) - Must provide a value ifgeneratedElementName1is provided.generatedElementName2(optional string) - Must match an existing generated element name from the import configuration.generatedElementValue2(optionally required string) - Must provide a value ifgeneratedElementName2is provided.generatedElementName3(optional string) - Must match an existing generated element name from the import configuration.generatedElementValue3(optionally required string) - Must provide a value ifgeneratedElementName3is provided.generatedElementName4(optional string) - Must match an existing generated element name from the import configuration.generatedElementValue4(optionally required string) - Must provide a value ifgeneratedElementName4is provided.generatedElementName5(optional string) - Must match an existing generated element name from the import configuration.generatedElementValue5(optionally required string) - Must provide a value ifgeneratedElementName5is provided.Returns:datasetId- A unique identifier for the dataset.Example without auto export:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

datasetId = wqxweb.StartImport( importConfigurationID, fileId, fileType=wqxweb.CSV, newOrExistingData=wqxweb.CONTAINS_NEW_OR_EXISTING, uponCompletion=wqxweb.DO_NOT_EXPORT, ignoreFirstRowOfFile=True )

print( f"The import of dataset {datasetId} has begun. Check it's status with a call to wqxweb.GetStatus(datasetId)" )Example with auto export and without auto submit:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

datasetId = wqxweb.StartImport( importConfigurationID, fileId, fileType=wqxweb.CSV, newOrExistingData=wqxweb.CONTAINS_NEW_OR_EXISTING, uponCompletion=wqxweb.EXPORT_IMPORT, ignoreFirstRowOfFile=True )

print( f"The import of dataset {datasetId} has begun. Check it's status with a call to wqxweb.GetStatus(datasetId)" )Example with auto submit:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

datasetId = wqxweb.StartImport( importConfigurationID, fileId, fileType=wqxweb.CSV, newOrExistingData=wqxweb.CONTAINS_NEW_OR_EXISTING, uponCompletion=wqxweb.SUBMIT_IMPORT, ignoreFirstRowOfFile=True )

print( f"The import of dataset {datasetId} has begun. Check it's status with a call to wqxweb.GetStatus(datasetId)" )StartXmlExportStart creating the XML submission file (for CDX).Parameters:datasetId(required string) - The return value ofStartImportorSubmitFileToCdx.uponCompletion(required enum or integer) - Declare what to do after the export finishes. This must be one of the following enum members provided by theWQXWebmodule or their corresponding integers:DO_NOT_SUBMIT(0) - do nothing.SUBMIT_EXPORT(1) - submit to CDX.Returns:status- The initial status for the dataset.Example using enum:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

status = wqxweb.StartXmlExport( datasetId, wqxweb.SUBMIT_EXPORT )

print( "The initial status of dataset {datasetId} is as follows:" )
print( status )Example using integer:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

status = wqxweb.StartXmlExport( datasetId, 1 )

print( "The initial status of dataset {datasetId} is as follows:" )
print( status )SubmitDatasetToCdxSubmit a dataset to CDX.Parameters:datasetId(required string) - The return value ofStartImportorSubmitFileToCdx.Returns:status- The initial status for the dataset.Example:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

status = wqxweb.SubmitDatasetToCdx( datasetId )

print( "The initial status of dataset {datasetId} is as follows:" )
print( status )SubmitFileToCdxSubmit a previously uploaded/built WQX XML file to CDX.Parameters:fileId(required string) - The return value ofUpload.Returns:datasetId- The new dataset ID.Example:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

datasetId = wqxweb.SubmitFileToCdx( fileId )

print( f"The uploaded file has been assigned datasetId {datasetId}" )GetStatusGet the status for a dataset. To avoid undue burden on the server, it is recommended that you not call this service more often than every 10 seconds. For large imports (longer than 20 minutes), calling this service periodically will guarantee that the server will not shutdown before the import completes.Parameters:datasetId(required string) - The return value ofStartImportorSubmitFileToCdx.Returns:statusMsg- The status, percent complete, and position in queue for the dataset. Possible statuses:'Waiting to Import''Importing''Imported''Import Failed''Waiting to Export''Waiting to Export and Submit''Exporting''Exported''Export Failed''Processing at CDX''Completed at CDX''Failed at CDX''Waiting to Delete''Deleting''Delete Failed''Waiting to Update WQX''Updating WQX''Updated WQX''Update Failed'Example:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

statusMsg = wqxweb.GetStatus( datasetId )

print( f"Current status of datasetId {datasetId} is {statusMsg}" )GetDocumentListGet the list of available documents for a dataset. Documents that are typically available for a dataset include:the original import fileevent logs (for the import and/or export in WQX Web)validation report (for the XML submission file - from CDX)processing report (for the XML submission file - from WQX)Parameters:datasetId(required string) - The return value ofStartImportorSubmitFileToCdx.Returns:urls- A list of document URLs which can be used to download the documents.Example:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

urls = wqxweb.GetDocumentList( datasetId )

print( "The documents associated with {datasetId} are:" )
for url in urls:
    print( f" - {url}" )ProjectsGet the projects for an organization.Parameters:organizationIdentifiersCsv(required string) - Comma delimited list of organization identifiers (e.g. "id1,id2,id3") NOTE: No spaces.projectIdentifiersCsv(optional string) - Comma delimited list of project identifiers (e.g. "id1,id2,id3") NOTE: No spaces.transactionIdentifier(optional string) - Transaction identifier e.g. "_23090c89-c6a6-4dd1-b16f-73f8ac36fac1".lastChangeDateMin(optional date) - Minimum last change date.lastChangeDateMax(optional date) - Maximum last change date.startRow(conditionally optional integer) - Zero based start row. Conditionally required ifrowsToRetrieveis supplied.rowsToRetrieve(conditionally optional integer) - Number of rows to retrieve. Conditionally required ifstartRowis supplied.Returns:projects- WQX Schema 2.2 element values including the transaction id for the project.Example:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

projects = wqxweb.Projects( organizationIdentifiersCsv )

print( f"The projects associated with {organizationIdentifiersCsv} are:" )
for project in projects:
    print( f" - {project}" )Example to list projects from the previous week:from wqxweblib import WQXWeb
from datetime import date, timedelta

wqxweb = WQXWeb( 'username', 'private key in base64' )

today = date.today()
lastWeek = today - timedelta(weeks=1)
projects = wqxweb.Projects( organizationIdentifiersCsv, lastChangeDateMin=lastWeek )

print( "The projects associated with {organizationIdentifiersCsv} and modified since {lastWeek.strftime( "%m/%d/%Y")} are:" )
for project in projects:
    print( f" - {project}" )MonitoringLocationsGet the locations for an organization.Parameters:organizationIdentifiersCsv(required string) - Comma delimited list of organization identifiers (e.g. "id1,id2,id3") NOTE: No spaces.monitoringLocationIdentifiersCsv(optional string) - Comma delimited list of monitoring location identifiers (e.g. "id1,id2,id3") NOTE: No spaces.monitoringLocationName(optional string) - Monitoring Location Name. Wildcards are supported (e.g. "Location%" means anything starting with "Location").monitoringLocationType(optional string) - Monitoring Location Type. Wildcards are supported (e.g. "Location%" means anything starting with "Location"). Allowed values (enforced by the endpoint, not this library):AtmosphereBEACH Program Site-Channelized streamBEACH Program Site-EstuaryBEACH Program Site-Great LakeBEACH Program Site-LakeBEACH Program Site-LandBEACH Program Site-Land runoffBEACH Program Site-OceanBEACH Program Site-River/StreamBEACH Program Site-Storm sewerBEACH Program Site-Waste sewerBoreholeCanal DrainageCanal IrrigationCanal TransportCaveCERCLA Superfund SiteChannelized StreamCombined SewerConstructed Diversion DamConstructed TunnelConstructed Water Transport StructureConstructed WetlandEstuaryFacility IndustrialFacility Municipal Sewage (POTW)Facility OtherFacility Privately Owned Non-industrialFacility Public Water Supply (PWS)Floodwater non-UrbanFloodwater UrbanGalleryGas-CondensateGas-EngineGas-ExtractionGas-FlareGas-Monitoring ProbeGas-Passive VentGas-SubslabGas-TemporaryGreat LakeLakeLandLand Flood PlainLand RunoffLandfillLeachate-ExtractionLeachate-Head WellLeachate-LysimeterLeachate-SamplePointLocal Air Monitoring StationMine PitMine/Mine DischargeMine/Mine Discharge Adit (Mine Entrance)Mine/Mine Discharge Tailings PileMine/Mine Discharge Waste Rock PileNational Air Monitoring StationOceanOil and Gas WellOther-Ground WaterOther-Surface WaterPipe, Unspecified SourcePlayaPond-AnchialinePond-SedimentPond-StockPond-StormwaterPond-WastewaterReservoirRiver/StreamRiver/stream Effluent-DominatedRiver/Stream EphemeralRiver/Stream IntermittentRiver/Stream PerennialRiverine ImpoundmentSeepSpigot / FaucetSpringState/Local Air Monitoring StationStorm SewerSurvey MonumentTest PitWaste PitWaste SewerWellWetland Estuarine-EmergentWetland Estuarine-ForestedWetland Estuarine-Scrub-ShrubWetland Lacustrine-EmergentWetland Palustrine PondWetland Palustrine-EmergentWetland Palustrine-ForestedWetland Palustrine-Moss-LichenWetland Palustrine-Shrub-ScrubWetland Riverine-EmergentWetland UndifferentiatedtransactionIdentifier(optional string) - Transaction identifier e.g. "_23090c89-c6a6-4dd1-b16f-73f8ac36fac1".lastChangeDateMin(optional date) - Minimum last change date.lastChangeDateMax(optional date) - Maximum last change date.startRow(conditionally optional integer) - Zero based start row. Conditionally required ifrowsToRetrieveis supplied.rowsToRetrieve(conditionally optional integer) - Number of rows to retrieve. Conditionally required ifstartRowis supplied.Returns:locations- WQX Schema 2.2 element values including the transaction id for the monitoring location.Example:from wqxweblib import WQXWeb

wqxweb = WQXWeb( 'username', 'private key in base64' )

locations = wqxweb.MonitoringLocations( organizationIdentifiersCsv )

print( "The locations associated with {organizationIdentifiersCsv} are: " )
for location in locations:
    print( f" - {location['MonitoringLocationName']}" )