Package: edges-io
Module for reading EDGES data and working with EDGES databases.This package implements all necessary functionality for reading EDGES data.
It’s two main concerns are:Reading the various file formats required for EDGES data:
- VNA readings
- fastspec output
- thermistor readings
- field weather recordings
- field thermlog recordingsVerifying and exploring databases of measurements in a robust and reliable way.FeaturesSome features currently implemented:Verify a “calibration observation” quickly without reading any actual data, with
a nice command-line tool:edges-iocheck.Optionally apply various automatic _fixes_ to a calibration observation to bring
it into line with standard database layout.Readacq,h5,matandnpzspectrum files seamlessly.Read S1P files.Verification of read data.Intuitive class hierarchy so that any subset of an observation can be handled.Read field-based weather and thermlog informationInstallationInstallation should be as simple as either one of the following:$ pip install git+git://github.com/edges-collab/edges-ioor, if you would like to developedges-ioand use it too:$ git clone https://github.com/edges-collab/edges-io
$ cd edges-io
$ pip install -e .[dev]There are a few dependencies, which should be installed automatically when following the
above command. If you are usingconda(which is recommended) then you can obtain
a cleaner/faster install by doing the following:$ conda create -n edges python=3
$ conda activate edges
$ conda install numpy scipy h5pyAnd then following either of the above instructions.UsageYou can useedges-ioeither as a library or a command-line tool. The library is
self-documented, so you can look at the docstring of any of the available functions.
We describe some basics of each approach here.CLITo run the checking tool, simply do:$ edges-io check PATHPATHshould be the top-level directory of a calibration observation (i.e. a folder
that has a sub-folder25C/, which has subfoldersSpectra/,Resistance/andS11etc.).
There are a few options you can use, for example changing the temperature of the observation,
and enabling automatic fixes. The latter can be achieved simply with the--fixflag.
If you find that a particular kind of error happens regularly,make an issueso we can add the
fix.LibraryThe library is useful for gathering an entire observation and performing operations
on its data. The library exposes a hierarchy of calibration objects, including base
objects like aSpectrum,ResistanceorS1Pfile, and container objects
likeSpectraorS11. An entire observation can be loaded as aCalibrationObservation, and it contains references to all children.For example:>>> from edges_io import io
>>> obs = io.CalibrationObservation("path_to_observation")
>>> print(obs.s11.path)
"path_to_observation/25C/S11"
>>> print(obs.spectra.ambient.path)
"path_to_observation/25C/Spectra/Ambient_XXX.acq"
>>> ambient_spectrum = obs.spectra.ambient.read()See howedges-iois used inedges-calfor a more involved example.Defining ObservationsOne of the main goals ofedges-iois to make the definition of a “Calibration
Observation” as clear, robust and error-free as possible. Many files go into any
particular observation – spectra, resistance measurements and S11 measurements – which
are all required to form together a calibration solution (which can then be applied
independently to field data). This code provides a clear structure for how these filesmustbe laid out in order for them to be read and used automatically. This is done in
a formal sense inthis document, but is also implemented within
the code itself.In the above document, the specification is laid out as formally as possible, and that
document has the final word on what is allowable. However, this can mean it’s a bit hard
to interpret, and so we here present a “simpler” guide to what constitutes a “Calibration
Observation”.A single natural “Observation” (see below for how to combine multiple observations into
a single “virtual” observation) is a single directory with multiple files/subdirectories
in it. That directory must be named under a certain convention that time-stamps it and
gives some useful metadata of the observation (like which receiver number was measured).
It is possible that in the future, a metadata file within the observation will specify
most of this information, but it is also useful to have a unique label for the observation.One question that is important in all of this definition is what to do when either 1)
a file exists that shouldn’t be there, or 2) a file doesn’t exist thatshouldbe
there. It may be tempting to overlook extra files that shouldn’t be there. However,
they can be a source of error. For example, spectra can be split across multiple files,
and we use a file pattern to find the files that should be read in. If an extra file
that “shouldn’t be there” exists and the file pattern matches it, then errors can occur
(even worse if the contents of that file are able to be read by the spectrum reader, but
correspond to a different load or something of that sort, where the results will be wrong,
but no error raised). Thus, whencheckingthe integrity of an observation we flag
extra files aserrors, and require the user to fix them up. To make this a bit easier,
and let those files stay in the directory (so we don’t lose potentially valuable information),
one of a few extensions can be added to the extra file:.old: for files that contain valid data but that is superseded by newer measurements
and should be ignored,.invalid: for data that has something wrong with it (equipment broken, wrong input
parameters, etc.),.ignore: files to ignore for any other reason.If the file does not have one of these extensions, and is not in the list of accepted
files for the Observation, an error will be raised by the checker.On the other hand, if a file ismissingthat must be there, different things can happen
in different situations. The default case is to treat this as a warning, which may be
counter-intuitive (surely missing a required file should be an error?!). The reason for
this is that that file may be supplemented by a different Observation. Perhaps this
Observation is incomplete  – maybe all the data that was taken was a single set of
Spectra, which is supposed to complement a previous observation which had a full set
of measurements. In this case, while the “natural” Observation is incomplete, it is not
necessary to give an error, as long as a warning is given such that it must be combined
with another observation. Nevertheless, some combinations of files are required to have
been taken in the same physical observation to ensure consistency (namely, S11 measurements
for each standard in a given load). If particular standards are missing, an error will
be raised.These caveats should be kept in mind as we talk about “required” directories/files below.
“Required” will mean that after combining all the observations that we want/need (see
next section), we require this particular file.Within the top-level observation directory are a number of directories denoting the ambient temperature at
which the observation was taken. These will usually be 15C, 25C or 35C. Most newer
observations are at 25C. One should never mix files between different ambient temperatures.
Thus, in reality, an observation is contained within one ofthesefolders, and in practice,
theCalibrationObservationhas itspathattribute set to the temperature directory.Inside this directory can be up to two files, and exactly three folders. One of the files
is aNotes.txtfile which summarises human-readable notes about the observation (“we
ran the ambient spectra first, but had a delay because of xxx…”). The other file is nameddefinition.yamland includes metadata about the observation in a specific format
(this file also allows you to supplement the observation with other observations, but
we’ll get to that later). Measurements/data like the male/female resistance should be
put in here (til now they have been found somewhere an input manually by the analyst
when doing the calibration, which is very risky and prone to error – they are properly
part of the measurement itself, not a choice of the analyst).The three folders areSpectra,ResistanceandS11. Note that an observation
must have all three of these (and nothing else, after combining observations).WithinSpectraexist a bunch of spectra taken over about 12-24 hours for each of
four “calibration sources” in the lab: they are “Ambient”, “HotLoad”, “LongCableOpen”
and “LongCableShorted” (often referred to as their simple aliases “ambient”, “hot_load”,
“open” and “short” in the code). These spectra will be in either.acqor.h5format, depending on the version offastspecthat took the measurements. Due to the
wayfastspectakes its data, each source may have multiple files for a single
measurement (each integration is saved to a new line in the file, but a new file is
created at particular local times each day). Thus, typically one would like to read in
and concatenate _all_ the files for that load, to use all the data.
Beyond this, it is _possible_ that two fully separate “runs” for a given source/load
will be made. In this case, an identifier for the “run number” is put into the filename.
Only one run number is actually used to do any particular calibration. In practice, it
isvery rareto have more than one “useable” run number for any particular load.
Typically, a second run is only taken if it is deemed necessary due to the first being
invalid in some way. If this is the case, this should be noted in theNotes.txtand/or thedefinition.yaml.TheResistancefolder is almost exactly the same as theSpectra. Each of the
sources is represented here again (with the same names), and the filename format is the
same, except that the files themselves are all.csv. These measure the resistance
readings of the sources, which are used to derive the physical temperatures of the loads
(against which the spectra are calibrated). Again, each source is allowed to have
multiple “runs” specified by their “run number”. However, again in practice it is
very uncommon to have more than one usable run.TheS11folder contains measurements of the reflection coefficients of the sources,
along with the LNA itself and the internal switch. These are all made with a VNA, and
each reading takes of order a minute. Thus, multiple readings of these measurements can
be taken – and typically are taken. Inside theS11folder exist a folder for each
of the main loads (or sources), in which are measurements of the four standards (open,short,matchandexternal). Each of these standards can be measured multiple
times, and so each file has the format<standard-name><rep-num>.s1p, whererep-numgoes from 1 - 99. However, each of the standards for a load is measured one after the
other on the same connection (i.e. there is no disconnection between them, to avoid
issues with different connection characteristics between the standards). Thus, one can’t
choose to use repeat number 01 foropenand repeat number 2 forshortfor theAmbientsource. For a given source, all standards used must be of the same repeat number (but multiple
runs canexistfor the source).
Besides the S11’s of the sources, we also need measurements of the LNA reflections, and
the internal switch. These exist in the foldersReceiverReadingXXandSwitchingStateXXrespectively. Here theXXcorrespond to what we call a “run” number, which
correspond to a complete re-measuring of the standards at different points in the
observation process. An arbitrary number of these can be performed (up to 99), but only
one is required.In all cases, the default behaviour ofedges-iois to use thelastrun number and
repeat number available for any given measurement.Combining Multiple ObservationsAs of v0.4.0,CalibrationObservationobjects no longer need to be defined fully by
one directory containing all measurements. While that is still an option (and the easiest
way to define a calibration observation), they can also be defined in a more sophisticated
way internally or externally.Internally, adefinition.yamlfile is allowed (and encouraged) which defines properties of the
observation, and also hasincludeandpreferkeywords which are used to supplement
or override any particular parts of the observation. For exampleincludecould point
to the top-level of any other observation, which could then be used whenever the
main observation lacks data. If this file exists, by default it is used to construct
the full observation virtually. An incomplete example of such a definition file can
be foundhere.Externally, a different file format is used to explicitly define every single measurement
file in an observation. This is supposed to be exhaustive and complete to make it
unambiguous. An example can be found in thetest-suite.
One can use such a file to create aCalibrationObservationby using theCalibrationObservation.from_observation_yaml()function.The way the code actually handles these “virtual” observations is essentially to create
a temporary directory and make symlinks to all the files that are required. This virtual
observation then looks and feels like a normal single observation, but is in fact
patched together from various observations.Using theHDF5Objectedges-iocontains a convenientHDF5Objectclass whose purpose is to make working
with HDF5 files a bit more formal (and arguably more simple). By subclassing it, you
can specify an exact file layout that can be verified on read, to ensure a file is
in the correct format (not just HDF5, but that it has the correct data structures and
groups and metadata).Using such a class is meant to provide a very thin wrapper over the file. So, for instance
if you have a filemy_hdf5_format_file.h5, whose structure is defined by the classCustomH5Format, you can create an object like this:>>> fl = CustomH5Format("my_hdf5_format_file.h5")Directly on creation, the file will be checked for compatibility and return an error
if it contains extraneous keys, or lacks keys that it requires.Once created, theflvariable now has operations which can “look into” the file
and load its data. It supports lazy-loading, so doing:>>> print(fl['dataset'].max())will load the ‘dataset’ data, and get the maximum, but it will not keep the data in
memory, and will not load any other datasets. If you have data in groups, you can
easily do:>>> print(fl['group']['dataset'].min())To load the data into the object permanently use the.loadmethod:>>> fl.load('group')In fact, doing this will load all data under ‘group’. If you just wanted to load
“dataset” out of “group”:>>> fl['group'].load('dataset')An example of how to define a subclass ofHDF5Objectcan be seen in theHDF5RawSpectrumclass, which is used to definefastspecoutput files.How the code works in a bit more detailFor the sake of developers (lets face it, most users of this particular repo should also
be developers), we will try to explain in a little more detail how the code works here.
This will focus on how the code treats the organization of a calibration observation,
and how it performs checks and makes fixes.The basic idea is that each directory, and each kind of file, is represented by a
distinct class, describing that kind of thing. For example, the top-level directory
(actually, the top-level plus the ambient temperature directory) is represented by theCalibrationObservationclass, while theSpectradirectory is represented by theSpectraclass, and S1P files are represented by theS1Pclass.All of these classes are subclasses either of_DataContainer(if it’s a folder) or_DataFile(if it’s a file). All of them have apathattribute which points to its
own path on-disk._DataFileclasses are much simpler, and typically only know how to
check its own filename for consistency with the specification, and how to read the data
in that particular filetype (they know nothing about their parents)._DataContainerclasses know about their ownpath,
but also can determine a list of files/subfolders theycontain(they know nothing about their
parents), and know how to map these files/folders onto their relevant defining classes.
They are able to check their own path for consistency, ensure that all relevant sub-files
exist, ensure that none extra exist, and recursively check the consistency of their
sub-files and folders by calling their checking methods.
Each file and folder in the observation becomes a specificinstanceof one of these
classes (there will be multipleS1Pinstances for all of the S11 measurements, and
each may have a differentnameattribute to identify the standard it represents).This top-down hierarchical structure is useful, and similar the to the way Unix filesystems
operate. However, it does mean that a particular instance is not necessarily unique: the
“match” standard S11 will exist within all sources, and since each class doesn’t know
its parent, theAmbient/Match01.s1pcannot be distinguished from theHotLoad/Match01.s1p.
However, a method exists on the top-levelCalibrationObservationwhich can match a
particular input path to a uniquesequenceof instances which do uniquely define it
(i.e. the first would be a sequence containing aLoadS11class withname=Ambientand the second would contain aLoadS11class withname=HotLoad).Another thing to note about the setup is the different between the classes andinstancesof those classes. Much of the functionality of the system is implemented just through
the classes themselves – one does not need to make instances of the classes to perform
the filesystem checks, for instance. In this case, thepathis given to thecheck()method of the class, eg.CalibrationObservation.check(path), which itself will
call thecheckmethod of any of its children etc. This will never read any data, it
will just check filename formats and contents of directories. However, one can make aninstanceof theCalibrationObservation, which will itself go and make instances
of all its children, storing them in the top-level class in a nice hierarchical way, in
which each of the children can be used independently. By default, when you create such
an instance, it will first perform the full check that would have been performed (but
in this case it should exit at the first error raised, and raise it as an error, rather
than continuing and printing all errors). Notably, these instances can be used toreadthe data in the files themselves. The instance will also decidewhichfiles to use
in the observation (i.e. which run numbers and repeat numbers).NoteThis project has been set up using PyScaffold 3.2.3. For details and usage
information on PyScaffold seehttps://pyscaffold.org/.