Package: hera-qm
HERA Quality Metricshera_qmis a python package for calculating quality metrics of HERA data.
It is integrated in the Real-Time Pipeline (RTP), automatically generating metrics
for all HERA data as it is taken. Buthera_qmcan also be used offline for
further analysis.MotivationData quality metrics are useful and needed throughout the analysis of interferometric data.
This repository is a centralized place for the HERA team to develop metrics to 1)
run on data in the RTP and deliver to the wider collaboration; 2) store these metrics
in the Monitor and Control database for easy access; and 3) use offline in individual
analyses. As a consequence of the first two goals, contributions tohera_qmwill
be vetted by the community and require thorough unittests. However, the code base
will also be flexible to enable the third goal, and we welcome contributions (see below).InstallationPreferred method of installation for users is simplypip install .(orpip install git+https://github.com/HERA-Team/hera_qm). This will install
required dependencies. See below for manual dependency management.DependenciesIf you are usingconda, you may wish to install the following dependencies manually
to avoid them being installed automatically bypip::$ conda install -c conda-forge "numpy>=1.23" "astropy>=5.0.4" "h5py>=3.1" "pyuvdata>=2.3" pyyamlDevelopingIf you are developinghera_qm, it is preferred that you do so in a freshcondaenvironment. The following commands will install all relevant development packages::$ git clone https://github.com/HERA-Team/hera_qm.git
$ cd hera_qm
$ conda create -n hera_qm python=3
$ conda activate hera_qm
$ conda env update -n hera_qm -f environment.yml
$ pip install -e .This will install extra dependencies required for testing/development as well as the
standard ones.Running TestsUses thepytestpackage to execute test suite.
From the sourcehera_qmdirectory run:pytestorpython -m pytest.Package Details and UsageThere are currently five primary modules which drive HERA quality metrics.ant_metricsA module to handle visibility-based metrics designed to identify misbehaving antennas.
The module includes methods to calculate several metrics to identify cross-polarized antennas
or dead antennas, based on either their redundancy with other antennas or their relative power.
The primary class,AntennaMetrics, includes interfaces to these methods and functions for
loading data, iteratively running metrics and removing misbehaving antennas, and saving the
results of those metrics in a JSON. And example of using this moduleis inscripts/ant_metrics_example_notebook.ipynb.firstcal_metricsA module to calculate metrics based on firstcal delay solutions. These metrics
identify large variations in delay solutions across time or across the array
for a given time. Included are functions for plotting firstcal delay solutions,
running the firstcal metrics, plotting the metrics, and writing them to file.
An example of using this module is inscripts/firstcal_metrics.ipynb.omnical_metricsA module to calculate metrics based on omnical solutions. Currently, these metrics
aim to identify discontinuities in the phase solutions of the gains and model visibilities,
as well as outliers in the antenna-based chi-square output from omnical. Routines for
calculating the metrics, writing them to file, and plotting the metrics (as well as the
gain solutions and model visibilities) are included. For an example of how to use these
metrics seescripts/omnical_metrics_example.ipynb. The metrics themselves are detailed
there as well as in the doc-strings of the source code inhera_qm.Omnical_Metrics.run_metrics().xrfiThis module contains the tools to for radio frequency interference (RFI) detection
and flagging. Low-level preprocessing functions act on 2D arrays to filter data
and/or calculate significance metrics. Flagging algorithms implement the low-level
functions or flag in other ways (e.g. "watershed" around existing flags). "Pipelines"
define the flagging strategy to apply to some data. For example,xrfi_h1c_pipeshows
the flagging scheme we used for H1C observing season. Wrappers handle the file I/O,
and call pipelines.xrfi_h1c_runis a wrapper we retroactively made to reflect
what we did for H1C.UVFlagUVFlag has been moved topyuvdata.Known Issues and Planned ImprovementsIssues are tracked in theissue log.
Major current issues and planned improvements include:A unified metric class structureDevelop Tsys calculations into metrics (HERA Memos 16 and 34)Develop closure quantities into metrics (HERA Memo 15)ContributingContributions to this package to introduce new functionality or address any of the
issues in theissue logare very welcome.
Please submit improvements as pull requests against the repo after verifying that
the existing tests pass and any new code is well covered by unit tests.Bug reports or feature requests are also very welcome, please add them to the
issue log after verifying that the issue does not already exist.
Comments on existing issues are also welcome.