Package: wagascianpy
WAGASCI ANPAN for Python v0.3.5This package contains all the Python modules used by the ANPAN
framework of the WAGASCI/T2K experiment.What is ANPAN?ANPAN (Acquisition Networked Program for Accelerated Neutrinos) is a collective
name to identify all the software used to acquire data and analyze it for
the WAGASCI experiment. It is composed of many packages:Pyrame(C/Python2)ANPAN DAQ(C/Python2)WagasciCalibration(C++11)WagasciMidas(C++11/JS)WagasciRunControl(Python2)WagasciAnpy (this package)(Python2/3)Python compatibilityPython2.7 or Python3.4+This package is fully compatible with both Python2.7 and Python3.4+. It is
tested usually with Python 3.8 and sporadically with other Python versions.However, I strongly suggest you to use Python 3 whenever possible because
Python 2 has now reached end of life and many packages are starting to drop
support for it. I am still trying to ensure Python 2 compatibility but I might
drop it in the future for non DAQ related modules.DependenciesMany functions of this package depend on ROOTROOTTo use the analysis programs, you need the WagasciCalibration package:WagasciCalibrationTo download and update the WAGASCI run database you may need the BorgBackup
program:BorgBackupThepyrmodmodule depends on Pyrame. This module is used only by the DAQ
so no need to install Pyrame on a non DAQ machine.ROOTBe sure that the version of ROOT that you installed is compatible with
your Python version. The precompiled binaries for ROOT until version 6.22.0
are usually only compatible with Python 2. From ROOT version 6.22.0 on,
ROOT should be compatible with both Python 2 and Python 3, so my suggestion
is to just update your ROOT version.WAGASCI calibration software (optional)To analyze the raw data you need the WAGASCI calibration software.
For the installation instructions refer tothis page.Installation (Linux)Open a terminal and issuepython3-mpipinstall--upgrade--userwagascianpyInstallation (MacOS)Currently the support for MacOS is only partial. The SSH connection is broken
because of some issues internal to thePython paramiko
module, so the program can be used in local mode
only. I tried to fix the issue but the problem is beyond my skills.  Probably in
the following months the paramiko developers will come up with a solution. At
the time of writing, the current (broken) version of paramiko is 2.7.1. When a
new version of paramiko is released, try again to run the software and see what
happens.I have tested the program only in MacOS Mojave and MacOS Catalina. Both showed
more or less the same problems. You can try older versions of MacOS and might
have better luck.MacOS Mojave/CatalinaBecause MacOS Mojave does not bundle Python3 we need to install it manually. The
easiest way is by using the package manager brew. The following commands are
taken from thiswebpage.The first step is to install the Apple's Xcode suite. Most probably you already
have it already installed, in that case you can skip this step. In your Terminal
app, run the following command to install XCode and its command-line tools:$xcode-select--installNext install Homebrew by copy/pasting the following command into Terminal and
then type Enter:$ruby-e"$(curl-fsSLhttps://raw.githubusercontent.com/Homebrew/install/master/install)"$brewupdateTo confirm Homebrew installed correctly, run this command:$brewdoctor
Yoursystemisreadytobrew.Now we can install the latest version of Python 3. Type the following command
into Terminal and press Enter:$brewinstallpython3To confirm which version of Python was installed, run the following command in
Terminal:$/usr/local/bin/python3--version
Python3.7.7Many functions of the WAGASCI database viewer use borgbackup. We can easily
install it using brew.$brewcaskinstallborgbackupFor other ways of installing borg refer to theborg website.Then we can finally install the wagascianpy package$python3-mpipinstall--user--upgradewagascianpyIf you use MacOS Catalina continue reading the next section.MacOS Catalina - troubleshooting**MacOS Catalina comes with its own version of Python3. Howeveras stated in the Python websitethe Apple-supplied Tcl/Tk 8.5 has serious bugs that can cause application
crashes.So follow the same instructions as MacOS Mojave but also install tkinter:$brewinstalltcl-tkConfigurationEnvironment parametersYou can easily configure all the parameters from the GUI itself (or from the
command line if you run the programs from the shell). However, because
inserting the same values over and over again can be tedius, you can modify the
configuration file calledwagasci.ini. This file is created in the${HOME}/.wagascifolder on the first execution of the program and filled with
the default values. You can just open and edit it with any text editor. The
options names and values are in one to one correlation with the GUI fields
and CLI arguments.Upon start, the program tries to load the configuration file from the following
directories in this order:Environment :${WAGASCI_CONFIGDIR}User :${HOME}/.wagasciSystem :/usr/local/wagasciThe configuration file must be calledwagasci.ini. Both the WAGASCI database
viewer and the spill history plotter can use the same configuration file.Good runsThe list of good runs and bad runs is hardcoded at the beginning of thewagascianpy/database/wagascidb.pyfile. If a run is completely bad, add it
to the_BAD_RUNSlist. If a run is bad only for a single detector, add it
to the_BAD_RUNS_Xwhere X is the name of the detector. After modifying
the bad run list, to apply it again, please rerunwagascidb_viewer.py  --rebuild-wagasci-database --batch-mode, and thenwagascidb_viewer.py  --apply-detector-flags --batch-mode.MarkersWhen drawing the history plots, the trouble markers and maintenance day
markers can be drawn along with the data. The start time and stop time of
these markers is hardcoded at the beginning of thewagascianpy/plotting/marker.pyfile.WAGASCI database viewerThe following subsections explain the usage of the WAGASCI database viewer GUI
program.Run the programTo open the GUI program, just type in a terminal:wagascidb_viewer.pyIf you wish to run the program without a GUI in batch mode, just type in a
terminalwagascidb_viewer.py--batch-mode<otherarguments>WAGASCI run repositoryThe word repository refers to the place where the runs are backed up. It might
be a simple repository where the runs are stored without encription (plain file
and folders), or a borg repository where all the files are encripted. The
repository can be either local (same machine) or remote (remote machine
reachable by SSH). In case of a borg repository it is necessary for borgbackup
to be installed on both the local machine and the remote machine.Simple repository: Unencripted repositoryBorg repository: Borg repositoryRepository location: For a remote repository the syntax is<hostname>:<path of repository>. The SSH configuration is taken from the
user~/.ssh/configfile by looking for<hostname>, so make sure that you
can connect to the remote machine by just opening the terminal andssh <hostname>. In case of local repository just insert the repository path. For
more info about how to connect to KEKCC by SSH refer to theWAGASCI expert
shifter manual(Troubleshooting - KEKCC (8.6)).WAGASCI run databaseLocation of the WAGASCI run database. The database is a file (usually namedwagascidb.db) that contains some info about each run. If the update database
checkbox is checked, the software checks if the database is synchronized with
the repository (if the database describes the current state of the repository).Database location: location of the database file. Usually it is the same
as the repository. In case you specify a different location, enter the
filename too. Leave as it is if in doubt.Update database: If left unchecked the database is read as is. If checked,
the runs that are present in both the repository and database the database are
not updated and only the runs that are present in the repository but not in
the database are added. Leave disabled if in doubt.Rebuild database: If checked each run in the repository is
inserted/overwritten into the database even if already present. Leave disabled
if in doubt.WAGASCI library: path to the directory containing the WAGASCI
calibration software libraries. It is only used to update the database and
analyze the raw data. Not needed if you only wish to visualize the database.Beam Summary Data repositoryThe word repository refers to the place where the Beam Summary Data files are
stored.  The local repository can be an empty folder or a folder containing
the BSD files.  If the remote repository field is left empty the program assumes
that the BSD files are already present in the local folder. If the remote
repository field is filled, the remote repository and the local repository are
synchronized using rsync.Local repository: Location of BSD files in the local machine.Remote repository: Location of the BSD files in the remote machine (the
syntax is the same as the WAGASCI run repository). If left empty only the
local repository is used.Beam Summary Data databaseLocation of the BSD database. The database is a file (usually namedbsddb.db)
that contains some info about each BSD file. If the update or rebuild database
checkbox is checked, the software checks if the database is synchronized with
the repository (if the database describes the current state of the repository).Database location: location of the database file. Usually it is the same
as the repository. In case you specify a different location, enter the
filename too. Leave as it is if in doubt.Update database: If left unchecked the database is read as is. If checked,
the runs that are present in both the repository and database the database are
not updated and only the runs that are present in the repository but not in
the database are added. Leave disabled if in doubt.Rebuild database: If checked each run in the repository is
inserted/overwritten into the database even if already present. Leave disabled
if in doubt.QueryGettersGet time interval: Select an interval and the push the Get interval
button. In the Run list frame will appear all the runs in that period.Get run interval: Select all runs whose number is within the interval.Get all: Get all runs regardless of the interval.Conditions:Include overlapping runs: If unchecked only the runs that start AND stop
inside the time interval are shown. If checked all the runs that start OR stop
inside the interval are shown.Only good runs: select only the runs marked as good (good_run_flag == 1)Analyzers:Download: download the selected runs from the WAGASCI run repository into
a local directory selected by the user (both borg and simple repositories
are supported)Decoder: decode the downloaded runs using the wgDecoder programADC histogram: plot the ADC distribution channel by channelBCID histogram: plot the BCID distribution for the whole DIFSpill Number Fixer: fix the spill number for the decoded runs using the
wgFixSpillNumber programBeam Summary Data: integrate the BSD info into the decoded runs using the
wgBeamSummaryData programTemperature: integrate the temperature info into the decoded runs using
the wgTemperature programData Quality: insert PEU information in the decoded data TTree using
the wgDataQuality programStart analysis: apply all the selected analyzers above to the downloaded
runsTopologyTopology: select which sub-detectors to download (not used for analysis)Run listList of runs that meet the conditions in the query. You can click on the run
name to show more info about that particular run.Run infoShow all the info about one particular run.GUI customizationThe GUI can be partially customized using thePyGubuprogram. PyGubu is
automatically installed by pip as a dependence of this package.To customize the GUI, locate the package installation folder and the gui.ui
file inside it, open a terminal andmkdir-p$HOME/.wagasci
cpgui.ui$HOME/.wagasci/
pygubu-designer$HOME/.wagasci/gui.uiThe program tries to load the GUI filegui.uiin the same way as the
configuration file as explained above.WAGASCI spill history plotterThespill_history.pyscript is provided by this package. It creates various
history plots for the WAGASCI experiment. It is a CLI only script and all its
features can be controlled by command line arguments. However it also tries to
read the configuration filewagasci.inifor required arguments that are not
specified in the command line.For further information, please consult the help message of the script:spill_history.py--helpMIDAS history to SQLite converterTo convert midas history files into SQLite format, please use themhistory_sqlite_converter.py, script. You need to have MIDAS installed
to run the script.For further information, please consult the help message of the script:mhistory_sqlite_converter.py--helpCalibration scriptsMany calibration scripts are provided by this package. They all take the raw
data as input and perform some calibration analysis on it. They depend
strongly on theWAGASCI calibration software.The documentation about how to use them is included in theWAGASCI expert
shifter manual.Other Python modulesList of all the Python modules provided by this package:wagascianpy.analysisContains modules for analysing the WAGASCI raw data, decoded data and slow
data.wagascianpy.analysis.analysis: Ctypes wrappers around
WagasciCalibration libraries.wagascianpy.analysis.analyzer: Takes the Ctypes wrappers and organize
them in a abstract factory design patter.wagascianpy.analysis.beam_summary_data: module to integrate the BSD
information inside the decoded WAGASCI data. It can be used as a
standalone program or as a analyzer ofwagascianpy.analysis.analyzer.wagascianpy.analysis.mhistory2sqlite: module to convert many mhistory
files containing slow devices data into a SQLite database.wagascianpy.analysis.spill: factory design patter to generate
various spill objects. A spill object contains information about the spill
number. It is useful when fixing the spill number bugs or when
integrating the BSD information.wagascianpy.databaseModules to manage the WAGASCI run database and the BSD database.wagascianpy.database.bsddb: BSD database creation and access.wagascianpy.database.db_record: Virtual database record that is
inherited by the WAGASCI run record and BSD record.wagascianpy.database.my_tinydb: Virtual database build upon thetinydbexternal module. It is inherited by the WAGASCI run database and
BSD database.wagascianpy.database.wagascidb: WAGASCI run database creation and access.wagascianpy.plottingModules to generate history plots (plots where the X axis is time).wagascianpy.plotting.colors: manage plot colorswagascianpy.plotting.detector: class to store the read TTrees for each
subdetector and to iterate over them.wagascianpy.plotting.graph: class to build a generic graphwagascianpy.plotting.harvest: class to harvest the needed data from the
TTrees. It returns the X points and Y points. It is implemented using the
strategy design pattern.wagascianpy.plotting.marker: class to plot time markers as the
maintenance days, WAGASCI runs start and stop time, electronics troubles
, etc...wagascianpy.plotting.plotter: class to plot the actual TMultiGraphwagascianpy.plotting.topology: class to specify which subdetectors to
plot and which notwagascianpy.programModules to generate a runnable program using the analyzers ofwagascianpy.analysis.analyzer. Implemented using the builder design
pattern.wagascianpy.program.director: director of the builder design pattern.wagascianpy.program.program: final program that is builded.wagascianpy.program.program_builder: builder that builds the program.wagascianpy.pyrmodInterface for the Pyrame modules. Used only for DAQ.wagascianpy.pyrmod.CCC: control the CCC boardwagascianpy.pyrmod.pyrmod: wrapper around generic Pyrame modulewagascianpy.utilsMiscellaneous utilities.wagascianpy.utils.change_bitstream: module to change the configuration of
the SPIROC chip for all the chips in one go.wagascianpy.utils.classproperty: implement Python property for classes
and not only for instanceswagascianpy.utils.configuration: parse the configuration file and
create a global configuration classwagascianpy.utils.environment: module to read the WAGASCI
environmental variables from a file or from the shell environment.wagascianpy.utils.utils: all other utilities are gathered here.wagascianpy.viewerModule used by the WAGASCI database viewer GUI program. They are not
supposed to be used anywhere else.wagascianpy.viewer.configuration: fill the global configurationwagascianpy.viewer.downloader: download runs from remote repositorywagascianpy.viewer.parse_args: parse the CLI argumentswagascianpy.viewer.show_run_info: build the 'run info' tab in the GUIwagascianpy.viewer.topology: little class to select which sub
-detectors are enabled and which notwagascianpy.viewer.utils- GUI utilitiesTroubleshootingIf you get an error about SSH trying to set a different locale in the KEKCC
repository try the solution explained in thisstackoverflow post.