Package: rfwtools
rfwtoolsThis package provides commonly used functionality around CEBAF C100 RF Waveforms collected by the JLab harvestser. This
includes data management such as download capture files, reading data from disk, parsing label files, running feature
extraction tasks, and generating data reports and visualizations.Latest API Documentationhttps://jeffersonlab.github.io/rfwtools/InstallationThis package has been posted to PyPI to ease installation.pipinstallrfwtoolsIf you would rather edit the code while using it you should do a git clone to a local directory, then install that
package in edit-able mode.cd/some/place
gitclonehttps://github.com/JeffersonLab/rfwtools.# Install the package (recommended that you use a virtual environment, etc.)pipinstall-e/some/place/rfwtoolsConfigurationInternally the package leverages a Config class that contains directory locations, URLs for network services, etc.. On
first reference, this class looks for and parses a config file, ./rfwtools.cfg. Below is simplified example file.data_dir:/some/path/rfw-research/data/waveforms/data/rflabel_dir:/some/path/rfw-research/data/labelsoutput_dir:/some/path/rfw-research/processed-outputdata_dir
: Base directory containing RF waveform data directory structures (i.e., directory containing zone directories). This
path may include a symlink on Linux if you do not wish to duplicate data. The path structure should mimic that found in
opsdata.
label_dir
: Directory contain label files (typically provided by Tom Powers)
output_dir
: Default directory for writing/reading saved files and other processed outputIf no file is found, file system paths are relative the project base, which is assumed to be the current working
directory. You can adjust these parameters in code as in the example below.fromrfwtools.configimportConfigConfig().data_dir="/some/new/path"UsagePrevious usage of this was to download a template directory structure with source code. This proved cumbersome, and
did not result in widespread usage. Below is a simple example that assume the above locations were sensibly defined.
It shows some of what you can accomplish with the package.fromrfwtools.data_setimportDataSetfromrfwtools.extractor.autoregressiveimportautoregressive_extractor# Create a DataSet.  For demo-purposes, I would make a small label file and run through.  This can take hours/days to# process all of our datads=DataSet(label_files=['my-sample-labels.txt'])# This will process the label files you have and create an ExampleSet under ds.example_setds.produce_example_set()# Save a CSV of the examples.ds.save_example_set_csv("my_example_set.csv")# Show data from label sources, color by fault_labelds.example_set.display_frequency_barplot(x='label_source',color_by="fault_label")# Show heatmaps for 1L22-1L26ds.example_set.display_zone_label_heatmap(zones=['1L22','1L23','1L24','1L25','1L26'])# Generate autoregressive features for this data set.  This can take a while - e.g. a few seconds per example.ds.produce_feature_set(autoregressive_extractor)# Save the feature_set to a CSVds.save_feature_set_csv("my_feature_set.csv")# Do dimensionality reductionds.feature_set.do_pca_reduction(n_components=10)# Plot out some different aspects# Color by fault, marker style by cavityds.feature_set.display_2d_scatterplot(hue="fault_label",style="cavity_label")# Color by zone, marker style by cavity, only microphonics faultsds.feature_set.display_2d_scatterplot(hue="zone",style="cavity_label",query="fault_label == 'Microphonics'")Developer NotesHere are some notes on the development process.First clone the repo. Then create a venv for development.gitclonehttps://github.com/JeffersonLab/rfwtools
python3.7-mvenvvenvActivate the venv and install the development requirements. These packages are used strictly in packaging, deploying,
and testingsourcevenv/bin/activate.csh
pip3install-rrequirements-dev.txtNow you can build wheels and source distributions, run unit tests, and upload to the test PyPI or PyPI. One thing
I like to do is create a project in a different directory and then install this package in editable mode. Instead
of actually installing it, pip creates a symlink back to your package directory and your source changes are reflected
in the downstream project without reinstalling. You do have to re-import packages or restart your interpreter though.mkdir/some/other/my_projectcd/some/other/my_project
python-mvenvvenvsourcevenv/bin/activate.csh
pipinstall-e/path/to/rfwtoolsIf you want to make source changes then you will need to install the packages in requirements.txt. The versions
listed where the ones used in the last development cycle. You may want to update those versions, but make sure to
test!pipinstall-rrequirements.txtTo run a unittests in multiple environments. Windows and linux have slightly different configurations. These match
the environment lists.tox-epy37-windows
tox-epy37-linuxTo run them directly in an IDE with builtin test runner, do the equivalent of this.cd/path/to/.../rfwtools
python3-munittestTo build documentation that can be used in github.From windows:cddocsrc.\make.batgithubgitadd.gitcommit-m"Updated documentation"From Linux:docsrc/build-docs.bash
gitadd.
gitcommit-m"Updated documentation"You should increment version numbers in setup.cfg and put out a new package to PyPI once a release is ready(shown below)
.
Update the requirements files if they changed. At a minimum, this should always be requirements.txt. See comments
below for details.pipfreeze>requirements.txtTo build distributions. You may need to remove directory content if rebuildingrmdist/*To upload to the test PyPI repo. You may need to add the--cert /etc/pki/tls/cert.pemoption for SSL problems.
Make sure to edit setup.cfg with latest info as shown below using vi and have built the package.vi setup.cfgsourcevenv/bin/activate.csh
python -m build
twine upload --repository testpypi dist/*To upload to the production PyPI repo. First edit setup.cfg with latest info.twineupload--repositorypypidist/*To install from production PyPI:pipinstallrfwtoolsTo install from Test PyPI:pip3install--index-urlhttps://test.pypi.org/simple/--extra-index-urlhttps://pypi.org/simple/rfwtoolsAdditional Developer Commentsrequirements.txt are the versions that were used from within my IDE and with my IDE unit test runner. This is the set
that worked during development, installation set will probably be different. This is every installed package, not just
the ones that my package directly uses.requirements-dev.txt are the versions of tools required to build, test, and distribute the package. These are the set
that worked during the last development cycle.requirements-testing.txt are the packages that needed to be installed for testing to work. They are basically the same
as requirements.txt, but with a few extras used exclusively in tests and the local rfwtools package itself.Certified NotesThe process for certified installation are largely captured in thesetup-certified.bashscript. Most of the basic
developer process is the same, but you will need to run through the certified installation process completely to make
sure that everything works as expected. At the end of this process you will have dropped the package files in a
directory.  That's all the get installed in the certified area.Generate a certified tarball once you think development is done../setup-certified tarball rfwtools<version>Copy this tarball to a temp directory and unzip it.cd..
mkdir tmp
mv rfwtools<version>.tar.gz tmpcdtmp
tar -xzf rfwtools<version>.tar.gzcdrfwtools<version>Now run through the standard process described bysetup-certified.bash -h. Make sure to review the docs directory
when done.This is something like the following:./setup-certified.bashtest./setup-certified.bashdocs
./setup-certified.bashbuildYou can also test the final installation if you have a target directory ready. You should find some wheel or tar.gz
files in the target directory when done.mkdir-p/tmp/pretend-certified/rfwtools/<version>
./setup-certified.bashinstall/tmp/pretend-certified/rfwtools/<version>Now compact the tarball to ensure that the to-be-archived code is what you want../setup-certified.bashcompact