Package: saqc
SaQC: System for automated Quality ControlSaQCis a tool/framework/application to quality control time series data.
It provides
a growing collection of algorithms and methods to analyze, annotate and
process timeseries data. It supports the end to end enrichment of metadata
and provides various user interfaces: 1) a Python API, 2) a command line interface
with a text based configuration system and aweb based user interfaceSaQCis designed with a particular focus on the needs of active data professionals,
including sensor hardware-oriented engineers, domain experts, and data scientists,
all of whom can benefit from its capabilities to improve the quality standards of given data products.For a (continously improving) overview of features, typical usage patterns,
the specific system components and how to customizeSaQCto your own
needs, please refer to ouronline documentation.InstallationSaQCis available on the Python Package Index (PyPI) and
can be installed usingpip:python-mpipinstallsaqcAdditionallySaQCis available via conda and can be installed with:condacreate-cconda-forge-nsaqcsaqcFor more details, see theinstallation guide.UsageSaQCis both, a command line application controlled by a text based configuration
and a python module with a simple API.SaQC as a command line applicationThe command line application is controlled by a semicolon-separated text
file listing the variables in the dataset and the routines to inspect,
quality control and/or process them. The content of such a configuration
could look likethis:varname    ; test
#----------; ---------------------------------------------------------------------
SM2        ; shift(freq="15Min")
'SM(1|2)+' ; flagMissing()
SM1        ; flagRange(min=10, max=60)
SM2        ; flagRange(min=10, max=40)
SM2        ; flagZScore(window="30d", thresh=3.5, method='modified', center=False)
Dummy      ; flagGeneric(field=["SM1", "SM2"], func=(isflagged(x) | isflagged(y)))As soon as the basic inputs, dataset and configuration file, are
prepared, runSaQC:saqc\--configPATH_TO_CONFIGURATION\--dataPATH_TO_DATA\--outfilePATH_TO_OUTPUTA fullSaQCrun against provided example data can be invoked with:saqc\--confighttps://git.ufz.de/rdm-software/saqc/raw/develop/docs/resources/data/config.csv\--datahttps://git.ufz.de/rdm-software/saqc/raw/develop/docs/resources/data/data.csv\--outfilesaqc_test.csvSaQC as a python moduleThe following snippet implements the same configuration given above through
the Python-API:importpandasaspdfromsaqcimportSaQCdata=pd.read_csv("https://git.ufz.de/rdm-software/saqc/raw/develop/docs/resources/data/data.csv",index_col=0,parse_dates=True,)qc=SaQC(data=data)qc=(qc.shift("SM2",freq="15Min").flagMissing("SM(1|2)+",regex=True).flagRange("SM1",min=10,max=60).flagRange("SM2",min=10,max=40).flagZScore("SM2",window="30d",thresh=3.5,method='modified',center=False).flagGeneric(field=["SM1","SM2"],target="Dummy",func=lambdax,y:(isflagged(x)|isflagged(y))))A more detailed description of the Python API is available in therespective sectionof the documentation.Get involvedContributingYou found a bug or you want to suggest new features? Please refer to ourcontributing guidelinesto see how you can contribute to SaQC.User supportIf you need help or have questions, send us an email tosaqc-support@ufz.deCopyright and LicenseCopyright(c) 2021,Helmholtz-Zentrum für Umweltforschung GmbH -- UFZ. All rights reserved.Documentation:Creative Commons Attribution 4.0 InternationalSource code:GNU General Public License 3For full details, seeLICENSE.PublicationsLennart Schmidt, David Schäfer, Juliane Geller, Peter Lünenschloss, Bert Palm, Karsten Rinke, Corinna Rebmann, Michael Rode, Jan Bumberger, System for automated Quality Control (SaQC) to enable traceable and reproducible data streams in environmental science, Environmental Modelling & Software, 2023, 105809, ISSN 1364-8152,https://doi.org/10.1016/j.envsoft.2023.105809. (https://www.sciencedirect.com/science/article/pii/S1364815223001950)How to cite SaQCIf SaQC is advancing your research, please cite as:Schäfer, David, Palm, Bert, Lünenschloß, Peter, Schmidt, Lennart, & Bumberger, Jan. (2023). System for automated Quality Control - SaQC (2.3.0). Zenodo.https://doi.org/10.5281/zenodo.5888547orLennart Schmidt, David Schäfer, Juliane Geller, Peter Lünenschloss, Bert Palm, Karsten Rinke, Corinna Rebmann, Michael Rode, Jan Bumberger, System for automated Quality Control (SaQC) to enable traceable and reproducible data streams in environmental science, Environmental Modelling & Software, 2023, 105809, ISSN 1364-8152,https://doi.org/10.1016/j.envsoft.2023.105809. (https://www.sciencedirect.com/science/article/pii/S1364815223001950)