Package: dynafed-storagestats
dynafed_storagestatsModule to interact with UGR's configuration files in order to obtain
storage status information from various types of endpoints and upload the
information to memcache. It leverage's UGR's connection check that it uploads
to memcache to skip any endpoints it has detected as being "Offline". (If
this information is not found, it is ignored and all endpoints are contacted)So far it supports has been tested with:Azure Storage BlobAWS S3Ceph S3Minio S3DPM via WebDAVdCache via WebDAVInstallationLinux:pip3installdynafed-storagestatsThis will install all necessary dependencies and create the executable
'/usr/bin/dynafed-storage'CentOS / SL 6Python 3.4 is available from EPEL repository.In order to install the above modules in python 3, pip3 needs to be setup. Since
it is not in the repos, run the following command:sudopython3/usr/lib/python3.4/site-packages/easy_install.pypipKnown issuesUsageMake sure the user that runs it is able to read UGR's configuration files.dynafed-storage-h
usage:dynafed-storage[-h]{reports,stats}...

positionalarguments:{reports,stats}reportsIndevelopmentstatsObtainandoutputstoragestats.

optionalarguments:-h,--helpshowthishelpmessageandexitSub-commandsChecksumsThis sub-command is intended to be called by Dynafed's external scripts as
specified in the option 'checksumcalc' to obtain and add checksum information
files or objects checksum information, specially for cloud based storage
endpoints that do not support the usual grid tool requests.The checksums sub-command has two sub-commands itself:getCurrently only S3 endpoints are supported.A client gives the URL of the object (not the Dynafed URL, but the actual SE
URL) and the type of checksum hash to obtain. If this information is
found, it will be printed out to stdout, if not, 'None' will be printed out.In its simplest form it is called with all three required arguments:dynafed-storagechecksumsget-e[ENDPOINT_ID]-u[URL]-t[HASH_TYPE]A more complex example specifying configuration file path, logging file and level,
and verbosity:dynafed-storagechecksumsget-v-c/etc/ugr/conf.d--loglevel=WARNING--logfile='/var/log/dynafed_storagestats/dynafed_storagestats.log'-e[EndpointID]-u[URL]-t[HASH_TYPE]Help:dynafed-storagechecksumsget-h
usage:dynafed-storagechecksumsget[-h][-c[CONFIG_PATH[CONFIG_PATH...]]][-f][-v][-eENDPOINT][-tHASH_TYPE][-uURL][--logfileLOGFILE][--logidLOGID][--loglevel{DEBUG,INFO,WARNING,ERROR}][--stdout]optionalarguments:-h,--helpshowthishelpmessageandexit-c[CONFIG_PATH[CONFIG_PATH...]],--config[CONFIG_PATH[CONFIG_PATH...]]PathtoUGRsendpoint.conffilesordirectories.Acceptsanynumberofarguments.Default:'/etc/ugr/conf.d'.-f,--forceForcecommandexecution.-v,--verboseShowonstderreventsaccordingtologlevel.

Checksumoptions.Required!:-eENDPOINT,--endpointENDPOINTChooseendpointcontainingdesiredobject.Required.-tHASH_TYPE,--hash_typeHASH_TYPETypeofchecksumhash.['adler32',md5]Required.-uURL,--urlURLURLofobject/filetorequestchecksumof.Required.

Loggingoptions:--logfileLOGFILESetlogfilespath.Default:/tmp/dynafed_storagestats.log--logidLOGIDAddthislogidtoeverylogline.--loglevel{DEBUG,INFO,WARNING,ERROR}Setlogoutputlevel.Default:WARNING.

Outputoptions:--stdoutSettooutputstatsonstdout.putCurrently only S3 endpoints are supported.A client gives the URL of the object (not the Dynafed URL, but the actual SE
URL), the checksum, and the type of checksum hash to add this information
to the object. Nothing is returned unless the process encounters errors.In its simplest form it is called with all four required arguments:dynafed-storagechecksumsput-e[ENDPOINT_ID]-u[URL]-t[HASH_TYPE]--checksum[CHECKSUM]A more complex example specifying configuration file path, logging file and level,
and verbosity:dynafed-storagechecksumsput-v-c/etc/ugr/conf.d--loglevel=WARNING--logfile='/var/log/dynafed_storagestats/dynafed_storagestats.log'-e[ENDPOINT_ID]-u[URL]-t[HASH_TYPE]--checksum[CHECKSUM]Help:dynafed-storagechecksumsput-h
usage:dynafed-storagechecksumsput[-h][-c[CONFIG_PATH[CONFIG_PATH...]]][-f][-v][--checksumCHECKSUM][-eENDPOINT][-tHASH_TYPE][-uURL][--logfileLOGFILE][--logidLOGID][--loglevel{DEBUG,INFO,WARNING,ERROR}][--stdout]optionalarguments:-h,--helpshowthishelpmessageandexit-c[CONFIG_PATH[CONFIG_PATH...]],--config[CONFIG_PATH[CONFIG_PATH...]]PathtoUGRsendpoint.conffilesordirectories.Acceptsanynumberofarguments.Default:'/etc/ugr/conf.d'.-f,--forceForcecommandexecution.-v,--verboseShowonstderreventsaccordingtologlevel.

Checksumoptions.Required!:--checksumCHECKSUMStringwithchecksumtoset.['adler32',md5]Required-eENDPOINT,--endpointENDPOINTChooseendpointcontainingdesiredobject.Required.-tHASH_TYPE,--hash_typeHASH_TYPETypeofchecksumhash.['adler32',md5]Required.-uURL,--urlURLURLofobject/filetorequestchecksumof.Required.

Loggingoptions:--logfileLOGFILESetlogfilespath.Default:/tmp/dynafed_storagestats.log--logidLOGIDAddthislogidtoeverylogline.--loglevel{DEBUG,INFO,WARNING,ERROR}Setlogoutputlevel.Default:WARNING.

Outputoptions:--stdoutSettooutputstatsonstdout.ReportsThe reports sub-command has two sub-commands itself:filelistNote: Only works with Azure and S3 endpointsThis sub-command is intended to be used to obtain file reports from the storage
endpoints. At this time, what is being developed is to be able to create
file dumps with the intention of using the for Rucio's integrity checks for
cloud based storage such as S3 and Azure since other grid storage solutions like
dCache and DPM have their own tools.For more information on this file dumps:DDMDarkDataAndLostFilesUsage example:
This will create a file at /tmp/ containing a list of files under the 'rucio'
prefix that are a day older from endpoints 'entpoint1' and 'endpoint2'. The
filename is the endpoint's ID name in the endpoints.conf file.dynafed-storagereportsfilelist-c/etc/ugr/conf.d-o/tmp--rucio-eendpoint1endpoint2reports filelist help:dynafed-storagereportsfilelist-h
usage:dynafed-storagereportsfilelist[-h][-c[CONFIG_PATH[CONFIG_PATH...]]][-f][-v][-e[ENDPOINT[ENDPOINT...]]][--logfileLOGFILE][--logidLOGID][--loglevel{DEBUG,INFO,WARNING,ERROR}][--deltaDELTA][--rucio][-oOUTPUT_PATH][-pPREFIX]optionalarguments:-h,--helpshowthishelpmessageandexit-c[CONFIG_PATH[CONFIG_PATH...]],--config[CONFIG_PATH[CONFIG_PATH...]]PathtoUGRsendpoint.conffilesordirectories.Acceptsanynumberofarguments.Default:'/etc/ugr/conf.d'.-f,--forceForcecommandexecution.-v,--verboseShowonstderreventsaccordingtologlevel.-e[ENDPOINT[ENDPOINT...]],--endpoint[ENDPOINT[ENDPOINT...]]Chooseendpoint(s)tocheck.Acceptsanynumberofarguments.Ifnotpresent,allendpointswillbechecked.

Loggingoptions:--logfileLOGFILESetlogfilespath.Default:/tmp/dynafed_storagestats.log--logidLOGIDAddthislogidtoeverylogline.--loglevel{DEBUG,INFO,WARNING,ERROR}Setlogoutputlevel.Default:WARNING.

Reportsoptions:--deltaDELTAMaskforLastModifiedDateoffiles.Integerindays.Default:1--rucioUsetocreateruciofiledumpsforconsitencychecks.Sameas:--delta1--prefixrucio

Outputoptions:-oOUTPUT_PATH,--output-dirOUTPUT_PATHSetoutputdirectory.Default:'.'-pPREFIX,--pathPREFIX,--prefixPREFIXSettheprefix/pathfromwheretostarttherecursivelist.Theprefixisexcludedfromtheresultingpaths.Default:''Important Note: DEBUG level might print an enormous amount of data as it will
log the contents obtained from requests. In the case of the generic methods this
will print all the stats for each file being parsed. It is recommended to use
this level with only the endpoint one wants to troubleshoot.storageThe purpose of this sub-command is to create storage accounting reports according
to different formats that experiments might require.At the moment only WLCG's JSON storage accounting file is available.In order to create it, the user will have to create a 'site-schema' YAML file
containing the site information. Please check in the 'samples' folder for
examples on how to create these file. The '-s/--schema' flag is required and
should point to this site-schema file.Usage example:
This will create file '/tmp/space-usage.json'dynafed-storagereportsstorage--wlcg-c/etc/ugr/conf.d-swlcg-schema.yml-o/tmpreports storage help:usage:dynafed-storagereportsstorage[-h][-c[CONFIG_PATH[CONFIG_PATH...]]][-f][-v][-e[ENDPOINT[ENDPOINT...]]][--logfileLOGFILE][--logidLOGID][--loglevel{DEBUG,INFO,WARNING,ERROR}][--memhostMEMCACHED_IP][--memportMEMCACHED_PORT][-sSCHEMA][--wlcg][-oOUTPUT_PATH]optionalarguments:-h,--helpshowthishelpmessageandexit-c[CONFIG_PATH[CONFIG_PATH...]],--config[CONFIG_PATH[CONFIG_PATH...]]PathtoUGRsendpoint.conffilesordirectories.Acceptsanynumberofarguments.Default:'/etc/ugr/conf.d'.-f,--forceForcecommandexecution.-v,--verboseShowonstderreventsaccordingtologlevel.-e[ENDPOINT[ENDPOINT...]],--endpoint[ENDPOINT[ENDPOINT...]]Chooseendpoint(s)tocheck.Acceptsanynumberofarguments.Ifnotpresent,allendpointswillbechecked.

Loggingoptions:--logfileLOGFILESetlogfilespath.Default:/tmp/dynafed_storagestats.log--logidLOGIDAddthislogidtoeverylogline.--loglevel{DEBUG,INFO,WARNING,ERROR}Setlogoutputlevel.Default:WARNING.

MemcachedOptions:--memhostMEMCACHED_IPIPorhostnameofmemcachedinstance.Default:127.0.0.1--memportMEMCACHED_PORTPortofmemcachedinstance.Default:11211Reportsoptions:-sSCHEMA,--schemaSCHEMAYAMLfilecontainingsiteschema.Required.--wlcgProducesWLCGJSONoutputfile.Requiressetupfile.

Outputoptions:-oOUTPUT_PATH,--output-dirOUTPUT_PATHSetoutputdirectory.Default:'.'statsThis sub-command is intended to be run periodically as a cron job in order to
upload the stats into memcached so that Dynafed can use it to be aware of the
storage capacity of the storage endpoints.It contacts each storage endpoint, obtains available stats and outputs them
according settings.First run with the following flags:dynafed-storagestats-v-c/etc/ugr/conf.d--stdout-mThis will printout any warnings and errors as they are encountered as well as
the information obtained from each of the endpoints. If you need more information
about the errors consider adding the "--debug" flag which will print more
information at the end of each endpoint's stats. If you still need more,
change the --loglevel to INFO or DEBUG, just be warned DEBUG might print a lot
of information.It is recommended to create a directory for the logfile, such as
"/var/log/dynafed_storagestats/dynafed_storagestats.log", as the default is
"/tmp/dynafed_storagestats.log".When everything looks to be setup as desired, setup cron to run the following
(add any other options that make sense to your site).dynafed-storagestats-c/etc/ugr/conf.d-m--loglevel=WARNING--logfile='/var/log/dynafed_storagestats/dynafed_storagestats.log'If instead of checking all configured endpoints, specific endpoint ID's can be
specified with the "-e" flag:dynafed-storagestats-c/etc/ugr/conf.d-m-eendpoint1endpoint2stats help:usage:dynafed-storagestats[-h][-c[CONFIG_PATH[CONFIG_PATH...]]][-f][-v][-e[ENDPOINT[ENDPOINT...]]][--logfileLOGFILE][--logidLOGID][--loglevel{DEBUG,INFO,WARNING,ERROR}][--memhostMEMCACHED_IP][--memportMEMCACHED_PORT][--debug][-m][-oOUTPUT_PATH][-p[TO_PLAINTEXT]][--stdout]optionalarguments:-h,--helpshowthishelpmessageandexit-c[CONFIG_PATH[CONFIG_PATH...]],--config[CONFIG_PATH[CONFIG_PATH...]]PathtoUGRsendpoint.conffilesordirectories.Acceptsanynumberofarguments.Default:'/etc/ugr/conf.d'.-f,--forceForcecommandexecution.-v,--verboseShowonstderreventsaccordingtologlevel.-e[ENDPOINT[ENDPOINT...]],--endpoint[ENDPOINT[ENDPOINT...]]Chooseendpoint(s)tocheck.Acceptsanynumberofarguments.Ifnotpresent,allendpointswillbechecked.

Loggingoptions:--logfileLOGFILESetlogfilespath.Default:/tmp/dynafed_storagestats.log--logidLOGIDAddthislogidtoeverylogline.--loglevel{DEBUG,INFO,WARNING,ERROR}Setlogoutputlevel.Default:WARNING.

MemcachedOptions:--memhostMEMCACHED_IPIPorhostnameofmemcachedinstance.Default:127.0.0.1--memportMEMCACHED_PORTPortofmemcachedinstance.Default:11211Outputoptions:--debugDeclaretoenabledebugoutputonstdout.-m,--memcachedDeclaretoenableuploadingstoragestatstomemcached.-oOUTPUT_PATH,--output-dirOUTPUT_PATHSetoutputdirectoryforflags-j,-xand-p.Default:'.'-p[TO_PLAINTEXT],--plain[TO_PLAINTEXT]Settooutputstatstoplaintxtfile.Addargumenttosetfilename.Default:dynafed_storagestats.txt--stdoutSettooutputstatsonstdout.Endpoints ConfigurationIn order to use the correct methods for each storage type some settings should
be added to the endpoints.conf configuration file.Generallocplugin.<ID>.storagestats.quota: [api, 1b|mb|gb|tb|pb|mib|gib|tib|pib]If this setting is missing, the script will try to get the quota from the endpoint
using the relevant API. Failing this, a default quota of 1TB will used.apiWill try to obtain the quota from the storage endpoint. If that fails a default
of 1TB will be used.(bytes)The quota can be specify in bytes, megabytes, mebibytes, etc. Lower or uppercase.locplugin.<ID>.storagestats.frequency: [ 600 ]This setting tells the script the number of seconds to wait before checking the endpoint again according to the timestamp of the last check stored in memcache. The default is 10 minutes (600 seconds).Azurelocplugin.<ID>.storagestats.api: [list-blobs]list-blobs (deprecated: generic)This setting will list all objects in a blob container and add the individual
sizes.
Each GET request obtains 5,000 objects. Therefore 10,005 objects cost 3 GET's.DAV/HTTPlocplugin.<ID>.storagestats.api: [list-files, rfc4331]list-files (deprecated: generic)This setting will list all objects behind the endpoint and add the individual
sizes. For this method to recursively get all objects, the DAV server needs
to header "Depth" with attribute "infinity". This is not recommended as
it is an expensive method, can use a lot of memory and is susceptible to
denial of service. Therefore this setting should be avoided if possible in
favour of rfc4331rfc4331This setting will query the DAV server according toRFC4331.S3locplugin.<ID>.storagestats.api: [list-objects, ceph-admin, cloudwatch]list-objects (deprecated: generic)This setting will list all objects behind the bucket and add the individual
sizes.
Each GET request obtains 1,000 objects. Therefore 2,005 objects cost 3 GET's.ceph-adminUse this setting if Ceph's Admin API is to be used. The credentials of the
configured user should have the "bucket read" caps enabled.cloudwatchFor AWS. Configure the cloudwatch metrics BucketSizeBytes and  NumberOfObjects.
This setting will poll these two. These metrics are updated daily at 00:00 UTC.
Read AWS's documentation for more information about Cloudwatch.locplugin.<ID>.s3.signature_ver: [s3, s3v4]Most S3 endpoints should use the s3v4 signature auth version, and is used as
default, but use s3 in case is needed.minio_prometheusMinio endpoints can expose metrics with a Prometheus client. The URL's path
is assumed to be "/minio/prometheus/metrics". This replaces any existing path
in the endpoint's configuration URL.These are the metrics extracted:minio_disk_storage_available_bytesminio_disk_storage_total_bytesminio_disk_storage_used_bytesor for newer version >= RELEASE.2019-12-17T23-26-28Zdisk_storage_availabledisk_storage_totaldisk_storage_usedThe quota/total_bytes can be overridden in the configuration file.minio_prometheus_v2Minio endpoints can expose metrics with a Prometheus client. The URL's path
is assumed to be "/minio/v2/metrics/cluster". This replaces any existing path
in the endpoint's configuration URL.These are the metrics extracted:minio_bucket_usage_object_totalminio_bucket_usage_total_bytesminio_node_disk_free_bytesminio_node_disk_total_bytesThe quota/total_bytes can be overridden in the configuration file.How it worksSimple FlowchartWhen run the main function will read every configuration file in the directory
given by the user (which defaults to /etc/ugr/conf.d), and will identify all the
different endpoints with their respective settings and authorization credentials.A python object belonging to a subclass of StorageStats, depending on the protocol
to be used, is created for each endpoint containing all the settings and
methods necessary to request and process storage stats and quota information.Memcache is contacted to look for UGR's endpoint connection stats. Any endpoints
flagged as "Offline" here will be skipped and flagged this will be informed in
the output. For those that are "Online", or if they could not be found to have
information will be contacted to obtain the storage stats. The information is
then stored a dictionary attribute "stats" in each object.The stats can then be output either to a memcache instance or the STDOUT,
depending on the options chosen when invoking this script.Warning/Error CodesKeyword/SettingStatus CodeUnknown General Warning/Error000Unknown/Invalid setting error001No Config Files / ID Mismatch002General Settings/DAV Plugincli_certificate003cli_private_key004conn_timeout005ssl_check006Invalid URL Schema008Unsupported Plugin009Azure Pluginazure.key010S3 Plugins3.alternate020s3.priv_key021s3.pub_key022s3.region023s3.signature_ver024Storage Stats Scripts Settingsstoragestats.api070storagestats.quota071storagestats.frequency072Memcached Warning/ErrorsUnknown080Memcached Connection081Memcached Index082StorageStats Connection Warning/ErrorsUnknown090Client Certificate Path091Server SSL Validation092Boto Param Validation Error095RFC4331 DAV Quota Method Not Supported096No Quota Given by Endpoint098Ceph S3 Bucket Quota Disabled099Connection Error400Element/Bucket/Blob not found404Endpoint Offline503Development setupTo install in "edit" mode, add the "-e" flag to pip3. This installs the package
as a symlink to the source code, so any changes made are reflected automatically
when running the executable.pip3install-e.MetaFernando Fernandez –ffernandezgalindo@triumf.caDistributed under the Apache license. SeeLICENSEfor more information.https://github.com/hep-gc/dynafed_storagestatsContributing