Package: cvmfs-server-scraper
CVMFS server scraper and prometheus exporterThis tool scrapes the public metadata sources from set of stratum0 and stratum1 servers. It grabs:- cvmfs/info/v1/repositories.jsonAnd then for every repo it finds (that it's not told to ignore), it grabs:- cvmfs/<repo>/.cvmfs_status.json
- cvmfs/<repo>/.cvmfspublishedUsage#!/usr/bin/env python3importloggingfromcvmfsscraperimportscrape,scrape_server,set_log_level# server = scrape_server("aws-eu-west1.stratum1.cvmfs.eessi-infra.org")set_log_level(logging.DEBUG)servers=scrape(stratum0_servers=["stratum0.tld",],stratum1_servers=["stratum1-no.tld","stratum1-au.tld",],repos=[],ignore_repos=[],)# Note that the order of servers is undefined.print(servers[0])forrepoinservers[0].repositories:print("Repo: "+repo.name)print("Root size: "+repo.root_size)print("Revision: "+repo.revision)print("Revision timestamp: "+repo.revision_timestamp)print("Last snapshot: "+str(repo.last_snapshot))Data structureServerA server object, representing a specific server that has been scraped.servers=scrape(...)server_one=servers[0]NameType: Attributeserver.nameReturnsThe name of the server, usually its fully qualified domain name.GeoApi statusType: Attributeserver.geoapi_statusReturnsA GeoAPIstatus enum object. Defined inconstants.py. The possible values are:OK (0: OK)LOCATION_ERROR (1: GeoApi gives wrong location)NO_RESPONSE (2: No response)NOT_FOUND (9: The server has no repository available so the GeoApi cannot be tested)NOT_YET_TESTED (99: The server has not yet been tested)RepositoriesType: attributeserver.repositoriesReturnsA list of repository objects, sorted by name. Empty if no repositores are scraped on the server.Ignored repositoriesType: Attributeserver.ignored_repositoriesReturnsList of repositories names that are to be ignored by the scraper.Forced repositoriesType: Attributeserver.forced_repositoriesReturnsA list of repository names that the server is forced to scrape. If a repo name exists in both ignored_repositories and forced_repositories, it will be scraped.RepositoryA repository object, representing a single repository on a scraped server.servers=scrape(...)repo_one=servers[0].repositories[0]NameType: Attributerepo_one.nameReturnsThe fully qualified name of the repository.ServerType: Attributerepo_one.serverReturnsThe server object to which the repository belongs.PathType: Attributerepo_one.pathReturnsThe path for the repository on the server. May differ from the name. To get a complete URL, one can do:url = "http://" + repo_one.server.name + repo_one.pathStatus attributesThese attributes are populated fromcvmfs_status.json:AttributeValuelast_gcTimestamp of last garbage collectionlast_snapshotTimestamp of the last snapshotInformation from.cvmfspublishedis also provided. For explanations for these keys, please see CVMFS'official documentation. The field value in the table is the field key from.cvmfspublished.AttributeFieldalternative_nameAfull_nameNis_garbage_collectableGmetadata_cryptographic_hashMmicro_cataoguesLreflog_checksum_cryptographic_hashYrevision_timestampTroot_catalogue_ttlDroot_cryptographic_hashCroot_sizeBroot_path_hashRsignatureThe end signature blobsigning_certificate_cryptographic_hashXtag_history_cryptographic_hashH