Package: pyAMI
For more information see the fullpyAMI documentationpyAMI 4 is a major upgrade of the AMI python client. The command line syntax
has been rationalized, with improved help functions. pyAMI is available in the
ATLAS software release and also installed centrally on lxplus.
pyAMI can be installed standalone on a laptop. Instructions are given below for
linux and windows.All commands are simply subcommands ofami.List AOD datasets matching a pattern:ami list datasets --type AOD data11_7TeVlist files in a dataset:ami list files -lhc dataset.namelist latest NTUP_TAUMEDIUM data datasets (and number of events in each one)
originating from AOD in periods L1 and L2 that are contained in a good runs list
(GRL):ami list data --project data11_7TeV --type NTUP_TAUMEDIUM --parent-type AOD \
--periods L1,L2 --stream physics_JetTauEtmiss --latest --fields events \
--grl mygrl.xmllist runs in a period or multiple periods:ami list runs B Mdisplay dataset metadata:ami dataset info dataset.nameand query projects, data types, dataset provenance, etc.pyAMIalso provides an API allowing you to perform all of the same queries
from within your own Python program.LXPLUS and CVMFSpyAMI is installed centrally on LXPLUS at CERN. To begin using pyAMI simply:source /afs/cern.ch/atlas/software/tools/pyAMI/setup.shthen authenticate yourself if you haven’t already (see below).pyAMI is also available through CVMFS:export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh
localSetupPyAMINote : You may need to get a particular version of pyAMI - SLC6 requires:source /afs/cern.ch/atlas/software/tools/pyAMI/python2.6/setup.shInstallationpyAMI requires at least Python 2.4.lxmlis optional but
required for XSL transformations. libxml2 and libxslt (dev packages) must be
installed to build lxml.Automatic InstallationAutomatically install the latest version of pyAMI withpip:pip install --user pyAMIor witheasy_install:easy_install --user pyAMIOmit the--userfor a system-wide installation (requires root privileges).
Add${HOME}/.local/binto your${PATH}if using--userand if
it is not there already (put this in your .bashrc):export PATH=${HOME}/.local/bin${PATH:+:$PATH}To upgrade an existing installation use the-Uoption in thepiporeasy_installcommands above.Manual InstallationGet the latest tarball onPyPIUntar and install (replace X appropriately):tar -zxvf pyAMI-X.tar.gz
cd pyAMI-XpyAMI uses distribute to install but you may revert to a basic
distutils install by setting the environment variable:export PYAMI_NO_DISTRIBUTE=1One advantage of using distribute is that all dependencies are automatically
downloaded and installed for you. To install pyAMI into your home directory
if using at least Python 2.6:python setup.py install --useror with older Python versions:python setup.py install --prefix=~/.localAdd${HOME}/.local/binto your${PATH}if it is not there
already (put this in your .bashrc):export PATH=${HOME}/.local/bin${PATH:+:$PATH}If you are unable to satisfy the dependency on lxml (only used for XSL
transformations) then you may disable this dependency before installation with:export PYAMI_NO_LXML=1Installation on MACTested with System: OXS 10.7.4, Python 2.7.1, GCC 4.2.1Just follow the instructions for Manual Installation, except that to change the PATH you should do:export PATH=/Users/yourName/Library/Python/2.7/bin${PATH:+:$PATH}Installation on WindowsInstall python, for examplePython 2.7.3Installdistribute(for easy installation of dependences). For 64 bit machines there may be problems,
seebugs.python.org/issue6792.Download pip. For examplepip-1.1.tar.gz.Install pip. Open a “cmd” terminal in windows. Assuming that you installed pip
below Python:cd C:\Python27\pip-1.1
C:\Python27\python setup.py installInstall pyAMI.pip.exeshould be inC:\Python27\Scripts:cd C:\Python27\Scripts
pip install pyAMIYou might need to install lxml to enable XSLT in a different step - sometimes
pip seems to prefer a version which will not build. If you don’t install
lxml, pyAMI will still work, but you will only be able to obtain XML output
on the command line. However if you only want to use the API of pyAMI you may
skip this step.:easy_install --allow-hosts=lxml.de,*.python.org lxml==2.2.8Now so that you can use pyAMI conveniently you must adjust your paths in the
Windows environment. If you do not know how to do this follow the instructions
herewww.java.com/en/download/help/path.xml.
Add to the path:PATH    C:\PYTHON27;C:\PYTHON27\ScriptsLastly explain to Windows that a python script can be executed.
Add to thePATHEXTvariable:PATHEXT .PYThen change the name of the fileamiinC:\Python27\Scriptstoami.pyso that script ami.py can be executed just by typingami.NoteIf you are not administrator of your machine you will probably need to
create the PATHEXT variable in your windows user environment.AuthenticationVOMS authentication is supported:voms-proxy-init -voms atlasThe alternative option (if gLite is not available)
is to send your username and password along with each AMI command. You can do
this explicitly, or you can set up an encrypted configuration file by running:ami authThis will prompt you for your AMI username and password.
You will only need to do this once since your credentials are stored in~/.pyami/ami.cfgfor later use. If your credentials change just
runami authagain. If you would like to keep your pyAMI configuration
in a directory other than~/.pyamithen set the environment
variablePYAMI_CONFIG_DIR.NoteOnce you make an ami configuration file it will override your voms proxy.
You should destroy it if you want to use a proxy. We implemented it this way because
the most common user error seemed to be trying to run pyAMI with an expired proxy!
Explicitly giving a username and password will override the configuration file.Version 4.1.2 (26/2/2013)Upgrade to distribute 0.6.34Fix installation on WindowsDocs now build properly with buildoutInstall setup_pyAMI with the pyAMI bundle for backward compatibilityVersion 4.1.1 (13/2/2013)Upgrade to buildout 2.0.0Fix a few import issues in pyAMI.extern.ZSIVersion 4.1.0 (4/2/2013)A patched version of ZSI is now included in pyAMI.extern that removes
dependency on PyXML.argparse is now included in pyAMI.extern.A minimal pyAMI installation (no lxml) now does not depend on any external
non-standard library packages.Added the -F and –from-file to the “ami list datasets” command where the
pattern is interpreted as a file name which will be read for patterns (one per
line, ignoring blank lines or lines beginning with #).Added the option to perform literal matches with “ami list datasets”
using the new optional flags -L or –literal-match. In this case the pattern
is interpreted as a literal string and the results must match exactly.
A literal match query is much faster than a patterned match and can greatly
speed up commands where you want to query information for known elements.
This option pairs nicely with the new –from-file feature mentioned above for
files listing many complete dataset names that should be interpreted literally
and not as patterns.AMI errors are now printed correctly.Version 4.0.4 (6/11/2012)Fix globbing issue. Only surround pattern with % if the pattern didn’t have %
on both ends originally. (it needed more fixing!)Version 4.0.3 (11/10/2012)Skip version 4.0.2 to sync with ATLAS SVN tag versionFix setup.py for py2.4 compatibilityMake ami-status=VALID defaultFix globbing issue. Only surround pattern with % if the pattern didn’t have %
on both ends originally.Version 4.0.1 (20/9/2012)This is pyAMI-04-00-01 in the ATLAS releaseMinor syntax changes in the APIAdded Windows installation to the docExpanded doc section on APIVersion 4.0.0a3 (17/4/2012)Make distribute the default install methodInclude patched urllib2 module in pyAMI.backportsVersion 4.0.0a2 (10/4/2012)Second prerelease of the new pyAMISupport for connection through a proxy (BNL)Other minor improvementsVersion 4.0.0a1 (28/3/2012)Prerelease of the new pyAMIPackage renamed to pyAMISpeed improvementsVOMS authenticationFixed permissions of ~/.pyami/ami.cfgRemoved intermediate ami package
Everything is now under pyAMIatlasmeta versions below:Version 0.4.0 (4/2/2012)Now compatible with Python 2.4Install with distutils by default and make installation with distribute optionalImproved parent-type feature in dataset queryAbility to specify a good runs list (GRL) in ‘ami list data’ and only
the runs contained in the GRL are shownInstalled on LXPLUS at CERNXSL transformations reimplementedVersion 0.3.1 (10/1/2012)Fix bug in file listing where size/events is NULL in databaseVersion 0.3 (20/12/2011)Add ability to query datasets by parent type with –parent-type optionlist files in dataset with ‘ami list files’ and optionally display total size
and eventsnew schema module replicating AMI schemaquery files, projects, subprojects, datatypes, subdatatypes, tags, nomenclatures,
production stepsVersion 0.2 (17/12/2011)Query and display additional dataset fields (i.e. datatype, number of events)Version 0.1 (1/12/2011)Forked pyAMIMajor code rewritebetter command-line interface