Package: netatmoqc
NetAtmoQCTable of Contents[[TOC]]Aboutnetatmoqcis a python package that uses Machine Learning Clustering methods to
quality-control observations collected fromNetAtmoweather stations. It has so far been developed atSMHIas part of theiObsproject.Please note that this package is still in its development/implementation stage.
As such, it may (certainly does) contain (hopefully minor) bugs and lack on
documentation. If you wish to collaborate, suggest features or report issues,
please contactPaulo Medeiros (SMHI).A note about this file: We have used theMarkdown formatthroughout.
You should, however, be able to read it reasonably well with your plain text
processor of choice. Please disregard the formatting marks in this case.See also theproject's Wikifor more information.System Requirementspython >=3.6.10A C compilerOptional: Ability to compile and run MPI applications.The system needs to have a working installation of an MPI library. HavingOpen MPIshould be fine, but there are other
options.This requirement is usually already fulfilled in HPC facilities, although,
in some cases, you might need to load a module (e.g.,module load openmpi).
Please check with your HPC support if you have doubts about this.NB.:If this requirement is not fulfilled, you won't be able to runnetatmoqcusing MPI, even if you manage to follow the MPI-related
installation instructions presented later on in this file.If you don't have a working MPI library installed in your system but use,
for instance,condato manage your environments/source packages, then running the following
commands should get it working:conda install -c conda-forge openmpi
  conda install gxx_linux-64Only forDeveloper-Mode Installtion:poetry, which can be installed by runningcurl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3InstallationBefore proceeding, please make sure that your system fulfils the appropriatesystem requirements. If you plan to just use the code
without modifying it, please follow one of the installation methods presented
in theRegular Installation section. However, if you
need/wish to modify the code in any way, then please proceed as indicated in theDeveloper-Mode Installtion section.N.B.:In any case, you will be presented with instructions for installation
with and without MPI support.You only need to follow one such set of
instructions. Mind, however, the extrasystem requirementsthat apply if you choose to installnetatmoqcwith MPI support.Regular InstallationRegular Installation from PyPi:point_right: Easiest method if you just want to use the code and don't want to
look at the source code at all.InstallwithoutMPI support:pip install netatmoqcInstall with MPI support:pip install netatmoqc[mpi]Regular Installation Directly From The Git Repo:point_right: Similar to aregular installation from PyPi,
but retrieves the code from the git repo instead (which is usually updated more
often).InstallwithoutMPI support:pip install "git+https://source.coderefinery.org/iOBS/wp2/task-2-3/netatmoqc"Install with MPI support:pip install "netatmoqc[mpi] @ git+https://source.coderefinery.org/iOBS/wp2/task-2-3/netatmoqc"Regular Installation From Downloaded Source:point_right: For those who havenetatmoqc's source code in a local directory,
wish to install it from there, but also don't want to modify any code.InstallwithoutMPI support:pip install .Install with MPI support:pip install ".[mpi]"Developer Mode Installation:point_right: For those who need/wish to make changes tonetatmoqc's
source code, or use code from a different branch thanmaster.InstallwithoutMPI support:poetry installInstall with MPI support:poetry install --extras "mpi"Installing in "developer mode" means that changes made in any of the package's
source files become visible as soon as the package is reloaded.:wrench:Recommendation to contributors:Before making your first commit to
the repo, please also run the following:pre-commit installThis sets up the git hook scripts defined in the.pre-commit-config.yamlfile and only needs to be run
(i) before the first commit, and (ii) after having modified the.pre-commit-config.yamlfile. Thepre-commitpackage is installed when you run any of
thepoetry installcommands listed above.After Installation: Configuration FileAfter successful installation, anetatmoqccommand will become available in
your environment. However, before you can usenetatmoqc(apart from the-hoption), you will need a configuration file written in theTOMLformat.Please take a look at thedocs/minimal_config_example.tomlanddocs/more_complete_config_example.tomlfiles, as well as theproject's Wiki, for more information about the configuration file.netatmoqcassumes that one of the following (whichever is first encountered)
is your configuration file :Afull file pathspecified via theNETATMOQC_CONFIG_PATHenvvarAconfig.tomlfile located in the directory wherenetatmoqcis called$HOME/.netatmoqc/config.tomlUsageAfter completing the setup, you should be able to runnetatmoqc [opts] SUBCOMMAND [subcommand_opts]where[opts]and[subcommand_opts]denote optional command line arguments
that apply, respectively, tonetatmoqcin general and toSUBCOMMANDspecifically.Please runnetatmoqc -hfor informationabout the supported subcommands
and generalnetatmoqcoptions. For info about specific subcommands and the
options that apply to them only,please runnetatmoqc SUBCOMMAND -h(note
that the-hgoes after the subcommand in this case).N.B.:When using the (preferred and default) clustering methodHDBSCAN, a typicalnetatmoqcrun (~40000 unique station IDs per DTG) needs ca.6 GBRAM and
takes between 1 and 2 minutes to complete (on a machine equipped with Intel(R)
Xeon(R) CPU E5-2640 v3 @ 2.60GHz, 16 physical cores). Other implemented
clustering strategies have more modest RAM requirements, but:DBSCANresults are not as good as HDBSCAN's in our contextOPTICSproduces similar results as HDBSCAN but runsmuchslowerParallelism (single-host or MPI)Theselectsubcommand supports parallelism over DTGs. How to activate it
depends on whether you wish to runnetatmoqcon a single host or if you wish
to distribute computations over different computers (e.g. on an HPC cluster).If you are runningnetatmoqcin a single host, then you can export the
environment variableNETATMOQC_MAX_PYTHON_PROCSto any value larger
than 0 and run the code as usual.If you wish to runnetatmoqcwith MPI, then you must have installed it
with MPI support. Assuming this is the case, you can then run the code asmpiexec -n 1 [-usize N] netatmoqc --mpi [opts] select [subcommand_opts]Notice that:Arguments between square brackets are optionalThe--mpiswitch must come before any subcommandThe value "1" in-n 1is mandatory.The code will always start with one
"manager" task which will dynamically spawn new worker tasks as needed
(up to a maximum number).If-usize Nis passed, thenNshould be an integer greater than zero.Ndefines the maximum number of extra workers that the manager task is
allowed to spawn if necessary.If-usize Nis not passed, then:If the run is part of a submitted job managed by SLURM or PBS, thenNwill be automatically determined from the options passed to the
scheduler (e.g.--nnodes,--ntasks,--mem-per-cpu, etc for SLURM).If the run is running interactive:Nwill take the value of the
environment variableNETATMOQC_MAX_PYTHON_PROCSif set, or, otherwise,
will be set to 1.No more thanlength(DTGs)new worker tasks will be spawnLast, but not least: Please keep in mind the RAM requirements discussed in theusagesection.