Package: artifactory-cleanup
Artifactory cleanupartifactory-cleanupis an extended and flexible cleanup tool for JFrog Artifactory.The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python.
Everything must be as a code, even cleanup policies!Tables of ContentsInstallationUsageNotesCommandsRulesCommonDeleteKeepDockerFiltersCreate your own ruleHow toHow to connect self-signed certificates for docker?How to clean up Conan repository?How to keep latest N docker images?ReleaseInstallationAs simple as one command!# dockerdockerpulldevopshq/artifactory-cleanup
dockerrun--rmdevopshq/artifactory-cleanupartifactory-cleanup--help# python (later we call it 'cli')python3-mpipinstallartifactory-cleanup
artifactory-cleanup--helpUsageSuppose you want to removeall artifacts older than N daysfromreponamerepository.
You should take the following steps:Installartifactory-cleanup(see above)Create a configuration fileartifactory-cleanup.yaml. variables.# artifactory-cleanup.yamlartifactory-cleanup:server:https://repo.example.com/artifactory# $VAR is auto populated from environment variablesuser:$ARTIFACTORY_USERNAMEpassword:$ARTIFACTORY_PASSWORDpolicies:-name:Remove all files from repo-name-here older than 7 daysrules:-rule:Reponame:"reponame"-rule:DeleteOlderThandays:7Run the commandTO SHOW(not remove) artifacts that will be deleted. By defaultartifactory-cleanupuses "dry
mode".# Set the credentials with delete permissionsexportARTIFACTORY_USERNAME=usernamehereexportARTIFACTORY_PASSWORD=password# dockerdockerrun--rm-v"$(pwd)":/app-eARTIFACTORY_USERNAME-eARTIFACTORY_PASSWORDdevopshq/artifactory-cleanupartifactory-cleanup# cliartifactory-cleanupVerify that right artifacts will be removed and add--destroyflagTO REMOVEartifacts:# dockerdockerrun--rm-v"$(pwd)":/app-eARTIFACTORY_USERNAME-eARTIFACTORY_PASSWORDdevopshq/artifactory-cleanupartifactory-cleanup--destroy# cliartifactory-cleanup--destroyLooking for more examples? Checkexamplesfolder!NotesAlwaysspecify version ofartifactory-cleanupwhen using it in the production.1.0.0is just an example, find the latest version in pypi:https://pypi.org/project/artifactory-cleanup/# dockerdockerpulldevopshq/artifactory-cleanup:1.0.0
dockerrun--rmdevopshq/artifactory-cleanup:1.0.0artifactory-cleanup--version# python (later we call it 'cli')python3-mpipinstallartifactory-cleanup==1.0.0
artifactory-cleanup--helpUse CI servers or cron-like utilities to runartifactory-cleanupevery day (or every hour). TeamCity and GitHub have
built-in support and show additional logs formatDo not save credentials in the configuration file, use environment variables.Commands# Debug - "dry run" mode by default# debug run - only print artifacts. it does not delete any artifactsartifactory-cleanup# Debug run only for policytestname.artifactory-cleanup--policy-namepolicytestname# REMOVE# For remove artifacts use --destroyartifactory-cleanup--destroy# Specify config filenameartifactory-cleanup--configartifactory-cleanup.yaml# Look in the future - shows what the tool WILL remove after 10 daysartifactory-cleanup--days-in-future=10# Not satisfied with built-in rules? Write your own rules in python and connect them!artifactory-cleanup--load-rules=myrule.py
dockerrun-v"$(pwd)":/appdevopshq/artifactory-cleanupartifactory-cleanup--load-rules=myrule.py# Save the table summary in a fileartifactory-cleanup--output=myfile.txt# Save the summary in a Json fileartifactory-cleanup--output=myfile.txt--output-format=jsonRulesCommonRepo- Apply the rule to one repository. If no name is specified, it is taken from the rule name (inCleanupPolicydefinition)-rule:Reponame:reponame# OR - if you have a single policy for the repo - you can name the policy as reponame# Both configurations are equalpolicies:-name:reponamerules:-rule:RepoRepoList- Apply the policy to list of repositories.-rule:RepoListrepos:-repo1-repo2-repo3RepoByMask- Apply rule to repositories matching by mask-rule:RepoByMaskmask:"*.banned"PropertyEq- Delete repository artifacts only with a specific property value (property_name is the name of the
parameter, property_value is the value)-rule:PropertyEqproperty_key:key-nameproperty_value:1PropertyNeq- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway.
Allows you to specify the deletion flagdo_not_delete = 1-rule:PropertyNeqproperty_key:key-nameproperty_value:1DeleteDeleteOlderThan- deletes artifacts that are older than N days-rule:DeleteOlderThandays:1DeleteWithoutDownloads- deletes artifacts that have never been downloaded (DownloadCount=0). Better to use
withDeleteOlderThanrule-rule:DeleteWithoutDownloadsDeleteOlderThanNDaysWithoutDownloads- deletes artifacts that are older than N days and have not been
downloaded-rule:DeleteOlderThanNDaysWithoutDownloadsdays:1DeleteNotUsedSince- delete artifacts that were downloaded, but for a long time. N days passed. Or not
downloaded at all from the moment of creation and it's been N days-rule:DeleteNotUsedSincedays:1DeleteEmptyFolders- Clean up empty folders in given repository list-rule:DeleteEmptyFoldersDeleteByRegexpName- delete artifacts whose name matches the specified regexp-rule:DeleteByRegexpNameregex_pattern:"\d"KeepKeepLatestNFiles- Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting
subfolders-rule:KeepLatestNFilescount:1KeepLatestNFilesInFolder- Leaves the last (by creation time) files in the number of N pieces in each
folder-rule:KeepLatestNFilesInFoldercount:1KeepLatestVersionNFilesInFolder- Leaves the latest N (by version) files in each
folder. The definition of the version is using regexp. By default it parsessemverusing the regex -([\d]+\.[\d]+\.[\d]+)")-rule:KeepLatestVersionNFilesInFoldercount:1custom_regexp:"[^\\d][\\._]((\\d+\\.)+\\d+)"KeepLatestNupkgNVersions- Leaves N nupkg (adds*.nupkgfilter) in release feature builds-rule:KeepLatestNupkgNVersionscount:1DockerDeleteDockerImagesOlderThan- Delete docker images that are older than N days-rule:DeleteDockerImagesOlderThandays:1DeleteDockerImagesOlderThanNDaysWithoutDownloads- Deletes docker images that are older than N days and have
not been downloaded-rule:DeleteDockerImagesOlderThanNDaysWithoutDownloadsdays:1DeleteDockerImagesNotUsed- Removes Docker image not downloaded since N days-rule:DeleteDockerImagesNotUseddays:1IncludeDockerImages- Apply to docker images with the specified names and tags-rule:IncludeDockerImagesmasks:"*singlemask*"-rule:IncludeDockerImagesmasks:-"*production*"-"*release*"ExcludeDockerImages- Exclude Docker images by name and tags.-rule:ExcludeDockerImagesmasks:-"*production*"-"*release*"KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X)- Leaves N
Docker images with the same major.(^\d+\.\d+\.\d+$)is the default regexp how to determine version which matches semver1.1.1. If you
need to add minor then setnumber_of_digits_in_versionto 2 or if patch then set to 3 (by default we match major, which 1). Semver tags
prefixed withvare supported by updating the regexp to include (an optional)vin the expression (e.g.,(^v?\d+\.\d+\.\d+$)).-rule:KeepLatestNVersionImagesByPropertycount:1custom_regexp:"[^\\d][\\._]((\\d+\\.)+\\d+)"DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)- Remove Docker image, if it is not found in the properties of the artifact repository.DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)- Remove Docker image, if it is not found in the properties of the artifact repository.FiltersIncludePath- Apply to artifacts by path / mask.-rule:IncludePathmasks:"*production*"-rule:IncludePathmasks:-"*production*"-"*develop*"IncludeFilename- Apply to artifacts by name/mask-rule:IncludeFilenamemasks:-"*production*"-"*develop*"ExcludePath- Exclude artifacts by path/mask-rule:ExcludePathmasks:-"*production*"-"*develop*"ExcludeFilename- Exclude artifacts by name/mask-rule:ExcludeFilenamemasks:-"*.tag.gz"-"*.zip"Create your own ruleIf you want to create your own rule, you can do it!The basic flow how the tool calls Rules:Rule.check(*args, **kwargs)- verify that the Rule configured right. Call other services to get more information.Rule.aql_add_filter(filters)- add Artifactory Query Language expressionsRule.aql_add_text(aql)- add text to the result aql queryartifactory-cleanupcalls Artifactory with AQL and pass the result to the next stepRule.filter(artifacts)- filter out artifacts. The method returnsartifacts that will be removed!.To keep artifacts useartifacts.keep(artifact)methodCreatemyrule.pyfile at the same folder asartifactory-cleanup.yaml:# myrule.pyfromtypingimportListfromartifactory_cleanupimportregisterfromartifactory_cleanup.rulesimportRule,ArtifactsListclassMySimpleRule(Rule):"""This doc string is used as rule titleFor more methods look at Rule source code"""def__init__(self,my_param:str,value:int):self.my_param=my_paramself.value=valuedefaql_add_filter(self,filters:List)->List:print(f"Today is{self.today}")print(self.my_param)print(self.value)returnfiltersdeffilter(self,artifacts:ArtifactsList)->ArtifactsList:"""I'm here just to print the list"""print(self.my_param)print(self.value)# You can make requests to artifactory by using self.session:# url = f"/api/storage/{self.repo}"# r = self.session.get(url)# r.raise_for_status()returnartifacts# Register your rule in the systemregister(MySimpleRule)Userule: MySimpleRulein configuration:# artifactory-cleanup.yaml-rule:MySimpleRulemy_param:"Hello,world!"value:42Specify--load-rulesto the command:# dockerdockerrun-v"$(pwd)":/appdevopshq/artifactory-cleanupartifactory-cleanup--load-rules=myrule.py# cliartifactory-cleanup--load-rules=myrule.pyHow toHow to connect self-signed certificates for docker?In case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into
thecertificatesfolder and add additional argument to the command:dockerrun-v"$(pwd)":/app-v"$(pwd)/certificates":/mnt/self-signed-certs/devopshq/artifactory-cleanupartifactory-cleanupHow to clean up Conan repository?We can handle conan's metadata by creating two policies:First one removes files but keep all metadata.Second one look at folders and if it contains only medata files - removes it (because there's no associated with
metadata files)The idea came fromhttps://github.com/devopshq/artifactory-cleanup/issues/47# artifactory-cleanup.yamlartifactory-cleanup:server:https://repo.example.com/artifactoryuser:$ARTIFACTORY_USERNAMEpassword:$ARTIFACTORY_PASSWORDpolicies:-name:Conan - delete files older than 60 daysrules:-rule:Reponame:"conan-testing"-rule:DeleteNotUsedSincedays:60-rule:ExcludeFilenamemasks:-".timestamp"-"index.json"-name:Conan - delete empty folders (to fix the index)rules:-rule:Reponame:"conan-testing"-rule:DeleteEmptyFolders-rule:ExcludeFilenamemasks:-".timestamp"-"index.json"How to keep latest N docker images?We can combine docker rules with usual "files" rules!The idea came fromhttps://github.com/devopshq/artifactory-cleanup/issues/61# artifactory-cleanup.yamlartifactory-cleanup:server:https://repo.example.com/artifactoryuser:$ARTIFACTORY_USERNAMEpassword:$ARTIFACTORY_PASSWORDpolicies:-name:Remove docker images, but keep last 3rules:# Select repo-rule:Reponame:docker-demo# Delete docker images older than 30 days-rule:DeleteDockerImagesOlderThandays:30# Keep these tags for all images-rule:ExcludeDockerImagesmasks:-"*:latest"-"*:release*"# Exclude these docker tags-rule:ExcludePathmasks:"*base-tools*"# Keep 3 docker tags for all images-rule:KeepLatestNFilesInFoldercount:3ReleaseIn order to provide a new release ofartifactory-cleanup, there are two steps involved.Bump the version in thesetup.pyBump the version in theinit.pyCreate a Git release tag (in format1.0.1) by creating a release on GitHub