Package: aces-apps
ACES Applications (aces-apps)ASKAP Comissioning and Early Science python packages.Provides:A collection of tools developed by members of the ACES team to support ACES processing:aces.askapdata- Tools for retrieving ASKAP observation metadata including:scheduling block infoaces.beamset- beamset classes supporting beam analysisaces.cleanlog- Tools for analysing imager deconvolution (clean) logsaces.data- Antenna positions.aces.display- Tools for displaying data summariesaces.fov- Tools for field-of-view modellingaces.holography- Tools for processing holography data to measure beams:Extract a holography beamset from an ASKAP measurement setaces.misc- Tools miscellaneous tasksaces.mpfit- Perform Levenberg-Marquardt least-squares minimization.The original version of this software, called LMFIT, was written in FORTRAN as part of the MINPACK-1 package.aces.obsplan- Tools for planning ASKAP observations including:Footprint utilitiesaces.sefd- Tools for estimating ASKAP sensitivitySEFD from 1934 dataaces.survey- Tools for processing ASKAP survey dataInstallationBelow are some instructions that attempt to outline the way to install for most situations.The primary dependencies and build instructions forpipare located inpyproject.toml.Currently, the installation depency for theaskapmodule is defined inaskap-requirements.txt.'I just want it deployed'This is intended if you just want to install the tooling in a modular and simple fashion.NOTE: You'll need toinstall condaon your system first.# If using ssh:gitclonessh://git@bitbucket.csiro.au:7999/aces/aces-apps.git# If using https:gitclonehttps://bitbucket.csiro.au/scm/aces/aces-apps.gitcdaces-apps# Optional: Checkout a specific version of aces-apps. By default, this will be the latest (`master`) version.gitcheckout{VERSION}# Create a conda environment with optional name. If you don't specify a name (with -n {ENVIRONMENTNAME}), it will be called 'aces'condaenvcreate-n{ENVIRONMENTNAME}# Activate the environmentcondaactivate{ENVIRONMENTNAME}Please note though thatVERSIONandENVIRONMENTNAMEare simply placeholders and need to be updated appropriately.In the above, the following occurs:The repository is cloned into a new directory (usingsshorhttps)A desired version is optionally checked outAcondaenvironment is created installing the software.NOTE: If you have any error aroundmpi4pyon a Pawsey system such asgalaxy, see the section below that outlines the modules that need to be loaded.Slightly more advancedThis outlines a process that gives a little more granularity for bespoke processes and use cases. Installingaces-appsshould be a fairly straightforward process.The current version (1.3.0) expects a python 3.9 installation. It is recommended that acondavirtual environment is created with this version:condacreate-n{ENVIRONMENTNAME}python=3.9-yThis will create a new conda environment with the appropriate python version installed.Next, activate this environment withcondaactivate{ENVIRONMENTNAME}Before issuing the next set ofpipcommands, please first take note whether you will be installing this package on a system with specialised MPI tooling. For example, Cray style HPC. If so, refer to the next section.Once thecondaenvironment you previously installed has been activated (and thempitooling requirements have been dealt with), installaces-appsshould be straight-forward.# If using ssh:pipinstallgit+ssh://git@bitbucket.csiro.au:7999/aces/aces-apps.git@1.3.0# If using https:pipinstallgit+https://bitbucket.csiro.au/scm/aces/aces-apps.git@1.3.0This command should clone theaces-appsrepository to a temporary space, checkout the1.3.0tag, and use thepyproject.tomlfile to identify dependencies, and their version specifications.Please note the @1.3.0 tagwhich will install a specific version of theaces-appstooling.If anmpi4pycompilation error is thrown, the requiredmpilibraries and environments are likely not correctly defined. Please refer to the next section if using Galaxy/Pawsey, or the documentation for your HPC system.Finally, theaskappython module needs to be installed (or rather, a subset of the complete module). This is being performed as a final step as its current layout is not well suited for automated installation with typical tooling. Issue these these commands:pipinstall-rrequirements.txt# - OR -pipinstallgit+https://bitbucket.csiro.au/scm/tos/python-askap.git#egg=askap
pipinstallgit+https://bitbucket.csiro.au/scm/tos/python-parset.git#egg=askap.parset
pipinstallgit+https://bitbucket.csiro.au/scm/tos/python-footprint.git#egg=askap.footprintSpecifically, thepython-askapmodule is actually a small set of independent modules. The above set ofpipcommands will install each module instal a common namespace as a submodule. The use of theegg=askapargument in the endpoint is important. If a specific version ofpython-askapis required this will need be set using the typical@1.2style of notation.Installing as a developerIf you are trying to develop or test code as part of theaces-appstooling, the above instructions are mostly correct, but some slight tweaks are needed.First, clone the repository and checkout the appropriate tag/branch.# If using ssh:gitclonessh://git@bitbucket.csiro.au:7999/aces/aces-apps.git# If using https:gitclonehttps://bitbucket.csiro.au/scm/aces/aces-apps.gitcdaces-apps
gitcheckout{VERSION}If you are planning on developing a new feature it is also recommended that a new branch is created to isolate changes from the main.Then create thecondaenvironment and activate itcondacreate-n{DEV_ENVIRONMENTNAME}python=3.9
condaactivate{DEV_ENVIRONMENTNAME}Unlike the previous set ofcondaenvironment commands, this onewill notuse theenvironment.ymlfile to auto magically install everything.Provided the above has worked, you can then do adeveloper installof theaces-appsbypipinstall-e.The-ewill essentially create a symlink from thesite-packages/acespath to the current location. This will let you edit code from the current location and be picked up when running scripts thatimport aces.Finally, configure theaskaptooling.pipinstall-rrequirements.txt# - OR -pipinstallgit+https://bitbucket.csiro.au/scm/tos/python-askap.git#egg=askap
pipinstallgit+https://bitbucket.csiro.au/scm/tos/python-parset.git#egg=askap.parset
pipinstallgit+https://bitbucket.csiro.au/scm/tos/python-footprint.git#egg=askap.footprintNotes onmpi4pySome tooling inaces-appsrepositories relies onmpifor code parallelism. On some cray-type supercomputing systems this requires a specialised set ofmpilibraries and compilers. Thegalaxysystem is such a machine. This tooling needs to be loaded into the shell environment before issuing thepipcommand to installaces-apps. On Galaxy, issue these commands:# Extra environment commands for galaxymoduleswapPrgEnv-crayPrgEnv-gnuexportMPICC=/opt/cray/pe/craype/2.5.13/bin/ccSimilarly, onzeusthe following commands are required:# Extra environment commands for zeusmoduleloadintel-mpiexportMPICC=/pawsey/intel/17.0.5/compilers_and_libraries/linux/mpi/intel64/bin/mpiccThese can be combined, and placed into your~/.bashrc:# MPI4PY# Test if machine is galaxymachine=$(hostname)if[[$machine==galaxy*]];thenmoduleswapPrgEnv-crayPrgEnv-gnuexportMPICC=/opt/cray/pe/craype/2.5.13/bin/ccfi# test if machine is zeusif[[$machine==zeus*]];thenmoduleloadintel-mpiexportMPICC=/pawsey/intel/17.0.5/compilers_and_libraries/linux/mpi/intel64/bin/mpiccfiPlease note that these instructions may not be correct or relevant on other HPC machines, including others at Pawsey. Care will need to be taken if attempting to deploy the package on a Cray style system.A sample scriptaces_test_mpi_install.pyis installed alongside this package as a way of verifying anmpiinstallation and configuration should it be needed.Notes from mpfitTaken verbatim from the mpfit documentationThis library contains a few useful routines I wrote or I converted from IDL.My contacts are:
Sergey Koposovkoposov@ast.cam.ac.ukInstitute of Astronomy, University of Cambridge
Madingley road, CB3 0HA, Cambridge, UKIf you have found a bug or have a patch, you can send them to me.
With my library I do not promise a stable interface, so beware.The licensing for the programs I wrote myself is GPL3. For all other
programs (mainly converted from IDL) I guess the license is either BSD or
they are in public domain.Here is the quick list of the functions I implemented:
TBW