Package: wipac-file-catalog
file_catalogStore file metadata information in a file catalogPrerequisitesTo get the prerequisites necessary for the file catalog:pip install -r requirements.txtRunning the serverTo start an instance of the server running:python -m file_catalogConfigurationAll configuration is done using environment variables.
To get the list of possible configuration parameters and their defaults, runpython -m file_catalog --show-config-specInterfaceThe primary interface is an HTTP server. TLS and other security
hardening mechanisms are handled by a reverse proxy server as
for normal web applications.BrowserRequests to the main url/are browsable like a standard website.
They will use javascript to activate the REST API as necessary.REST APIRequests with urls of the form/api/RESOURCEcan access the
REST API. Responses are inHALJSON format.File-Entry FieldsFile-Metadata Schema:Seetypes.pyMandatory Fields:uuid(provided by File Catalog)logical_namelocations(with at least one non-empty URL)file_sizechecksum.sha512Route:/api/filesResource representing the collection of all files in the catalog.Method:GETObtain list of filesREST-Query Parameterslimitstartlogical_name(shortcut parameter)directory(shortcut parameter)filename(shortcut parameter)logical-name-regex(shortcut parameter)run_number(shortcut parameter)dataset(shortcut parameter)event_id(shortcut parameter)processing_level(shortcut parameter)season(shortcut parameter)querykeysall-keys(shortcut parameter)max_time_msHTTP Response Status Codes200: Response contains collection of file resources400: Bad request (query parameters invalid)429: Too many requests (if server is being hammered)500: Unspecified server error503: Service unavailable (maintenance, etc.)Method:POSTCreate a new file or add a replicaIf a file exists and the checksum is the same, a replica is added. If the checksum is different a conflict error is returned.REST-BodySeeFile-Entry FieldsHTTP Response Status Codes200: Replica has been added. Response contains link to file resource201: Response contains link to newly created file resource400: Bad request (metadata failed validation)409: Conflict (if the file-version already exists); includes link to existing file429: Too many requests (if server is being hammered)500: Unspecified server error503: Service unavailable (maintenance, etc.)Method:DELETENot supportedMethod:PUTNot supportedMethod:PATCHNot supportedRoute:/api/files/{uuid}Resource representing the metadata for a file in the file catalog.Method:GETObtain file metadata informationREST-Query ParametersNoneHTTP Response Status Codes200: Response contains metadata of file resource404: Not Found (file resource does not exist)429: Too many requests (if server is being hammered)500: Unspecified server error503: Service unavailable (maintenance, etc.)Method:POSTNot supportedMethod:DELETEDelete the metadata for the fileREST-Query ParametersNoneHTTP Response Status Codes204: No Content (file resource is successfully deleted)404: Not Found (file resource does not exist)429: Too many requests (if server is being hammered)500: Unspecified server error503: Service unavailable (maintenance, etc.)Method:PUTFully update/replace file metadata informationREST-BodySeeFile-Entry FieldsHTTP Response Status Codes200: Response indicates metadata of file resource has been updated/replaced404: Not Found (file resource does not exist) + link to “files” resource for POST409: Conflict (if updating an outdated resource - use ETAG hash to compare)429: Too many requests (if server is being hammered)500: Unspecified server error503: Service unavailable (maintenance, etc.)Method:PATCHPartially update/replace file metadata informationThe JSON provided as body to PATCH need not contain all the keys, only the  need to be updated. If a key is provided with a value null, then that key can be removed from the metadata.REST-BodySeeFile-Entry FieldsHTTP Response Status Codes200: Response indicates metadata of file resource has been updated/replaced404: Not Found (file resource does not exist) + link to “files” resource for POST409: Conflict (if updating an outdated resource - use ETAG hash to compare)429: Too many requests (if server is being hammered)500: Unspecified server error503: Service unavailable (maintenance, etc.)More About REST-Query Parameterslimitpositive integer;number of results to provide(default: 10000)NOTE:The servermayhonor thelimitparameter. In cases where the server does not honor thelimitparameter, it should do so by providing fewer resources (limitshould be considered the client’s upper limit for the number of resources in the response).startnon-negative integer;result at which to start at(default: 0)NOTE:the servershouldhonor thestartparameterTIP:incrementstartbylimitto paginate through many resultsqueryMongoDB query;use to specify file-entry fields/ranges; forwarded to MongoDB daemonkeysa|-delimited string-list of keys;defines what fields to include in result(s)ex:"foo|bar|baz"different routes/methods define differing defaultsNOTE:there is no performance hit for including more fieldsseeall-keysmax_time_msnon-negative integer ORNone;timeout to kill long queries in MILLISECONDSoverrides the default timeout of 600000 ms (10 minutes)Noneindicates no timeout (this can hang the server -- you have been warned)Shortcut Parameters:logical-name-regex,logical_name,directory,filenameIn decreasing order of precedence...logical-name-regexquery by regex pattern (at your own risk... performance-wise)equivalent to:query: {"logical_name": {"$regex": p}}logical_nameequivalent to:query["logical_name"]directoryquery by absolute directory filepathequivalent to:query: {"logical_name": {"$regex": "^/your/path/.*"}}NOTE:a trailing-/will be inserted if you don't provide oneTIP:use in conjunction withfilename(ie:/root/dirs/.../filename)filenamequery by filename (no parent-directory path needed)equivalent to:query: {"logical_name": {"$regex": ".*/your-file$"}}NOTE:a leading-/will be inserted if you don't provide oneTIP:use in conjunction withdirectory(ie:/root/dirs/.../filename)Shortcut Parameter:run_numberequivalent to:query["run.run_number"]Shortcut Parameter:datasetequivalent to:query["iceprod.dataset"]Shortcut Parameter:event_idequivalent to:query: {"run.first_event":{"$lte": e}, "run.last_event":{"$gte": e}}Shortcut Parameter:processing_levelequivalent to:query["processing_level"]Shortcut Parameter:seasonequivalent to:query["offline_processing_metadata.season"]Shortcut Parameter:all-keysboolean (True/"True"/"true"/1);includeall fieldsin result(s)NOTE:there is no performance hit for including more fieldsTIP:this is preferred over querying/api/files, grabbing the uuid, then querying/api/files/{uuid}DevelopmentEstablishing a development environmentFollow these steps to get a development environment for the File Catalog:cd ~/projects
git clone git@github.com:WIPACrepo/file_catalog.git
cd file_catalog
./setupenv.shMongoDB Instance for TestingThis command will spin up a disposable MongoDB instance using Docker:docker run \
    --detach \
    --name test-mongo \
    --network=host \
    --rm \
    circleci/mongo:latest-ramBuilding a Docker containerThe following commands will create a Docker container for the file-catalog:docker build -t file-catalog:{version} -f Dockerfile .
docker image tag file-catalog:{version} file-catalog:latestWhere {version} is found in file_catalog/__init__py; e.g.:__version__ = '1.2.0'       # For {version} use: 1.2.0Pushing Docker containers to local registry in KubernetesHere are some commands to get the Docker container pushed to our Docker
register in our Kubernetes cluster:kubectl -n kube-system port-forward $(kubectl get pods --namespace kube-system -l "app=docker-registry,release=docker-registry" -o jsonpath="{.items[0].metadata.name}") 5000:5000 &
docker tag file-catalog:{version} localhost:5000/file-catalog:{version}
docker push localhost:5000/file-catalog:{version}