Package: cincan-registry
CinCan RegistryCinCan registry is  a tool for listing available CinCan tools, their versions, sizes and possible updates as far as into their original source.Available tools can be found in the CinCan'stool repository.Source code for tools' Dockerfiles is available in there.Currently, no other tools are supported for version information listing or for other details.Docker images of the remote tools are stored inQuay,GitHub's Container RegistryandDocker Hub.By default, Quay has been used as image registry.When checking versions beyond remote (remote registry), tool is using multiple different APIs such as GitHub's or GitLab's to acquire most recent availabe versions straight from the source. See more inupstream checking.InstallationLatest release can be installed from PyPi as:pip install cincan-registryTo be able to list information from locally available tools, "Docker Daemon" must be running on your machine.Tool is part of thecincan-command. Commandcincanregistryin future examples can be replaced withcincanwhen using in there.Make sure that your PATH is working as intended, if not installing as 'sudo'.UsageRegular listingListing available local or remote CinCan tools with their sizes and versions is the general purpose of this program.We are also able to list upstream versions of tools, for those where this feature is configured. This is handled more thoroughly inUpstream Checker section.To list both available local and remote tools with tag "latest":cincanregistry listWhen running first time, output could look something like this:By default, "latest" tag is always used, unless over-written with--tagor-targument.To list only locally available tools with tag "latest", argument--local(or-l) can be used. Additionally, when listing only tools for specific location, listing will show all viable tags pointing into same image.For example if version '0.1' of the tool is tagged with 'dev' tag, that specific tag should be used to use version '0.1' of the tool.cincanregistry list --localExample output should look something likethis.To do same for remote, use--remote(or-r) argument. By adding flag-a, this shows tools with all possible tags.cincanregistry list --remote -aAll commands are supporting JSON output. Add--jsonor-jargument. To combine previously mentioned-tflag, we can use following arguments to provide JSON output:cincanregistry list -ljt devThis lists locally available tools with 'dev' tag in JSON format.JSON will contain also size of the images.Forlocalimages, size is asuncompressedand for remote images, ascompressedsize.By adding size column for regular listing,--size(or-s) argument can be used.To list local images with all tags and size column included:cincanregistry list -lsaVersion listing and comparisonTo see more about versions, we need to useversionssubcommand. By runningcincanregistry list versionstool will fetch version information from configured upstreams. By default, it lists only locally available tools, and compares their versions into remote versions, and further remote versions into upstream versions.This helps us to acknowledge whether our tool isreallyup-to-date or not.Current implementation lists those tools as red#AA0000, where is immediate update available on remote. (local version differs from remote)If local and remote are equal, but possible upstream of the tool has an update, those are listed as grey#808080.As seen in the above image, green should indicate, that at least for that line, everything there isfine.Argument--name(or-n) can be used to check updates for single tool, and exclusive argument--only-updatesor-ucan be used to show only tools where there are updates available.It should be noted, that tool is not able to directly tell, if there is actually newer version available. It only detects deviations.We can mark only deviations, because there is no golden standard for marking version information; it varies a lot among developers. However, tool is able to detect same versions with good accuracy, even if they are marked a bit differently.Most of the preceding arguments withlistsubcommand will change the behavior ofversionssubcommand as well.For example command:cincanregistry list -rj versions --only-updatesWill produce JSON output from remote tools; generating their versions and filtering only for ones with available version updates, after checking configured upstreams for those available.All available optionsSpecific tolistDescription--config-cPath to configuration file--tag-tFilter images by tag name.--all-aShow images with all tags. (Excludes --tag or -t)--size-sInclude  size column when listing--json-jProduce output in JSON format--local-lList only locally available 'cincan' tools. Excludes --remote or -r--remote-rList remotely available 'cincan' tools. Excludes --local or -lSize will be here always included in JSON regardless is it used with argument or not.Specific tolist versionsDescription--name-nCheck single tool by the name.--only-updates-uLists only available updates. Excludes --name or -n--with-tags-wInclude tag column when listing single tool; shows usable tags for that version.--force-refresh-fRefresh all version related cache data including meta files.These can be used with the combination oflistoptions-land-rto produce varying outputs. Arguments-t,-aand-sare ineffective when used withversionssubcommand.Tool is attempting to always find the latest available version among all tags.UtilitiesOnly for development purposeThere is available feature to update description and README of the tool(s) in Quay and Docker Hub.This requires locally cloned CinCan tools repository - READMEs from there are used.For example commandcincanregistry -t ../tools utils update-readme --allUploads every README file for corresponding repository in CinCan's Quay Container Registry or Docker Hub. The first header (# ) of the README is used asdescriptionof image.Providing credentialsdocker loginMust be used beforehand to log in for Docker Hub - same credentials will be used on upload process.All available options for 'utils'Specific toutilsDescription--config-cPath to configuration fileSpecific toutils update-readmeDescription--name-nUpdate README and description of single tool by the name.--allAttempt to update README and description of every tool from 'tools' folder, matching the repository on Docker HubUpstream checkerCinCan Registry has a feature to check available new versions for tool, if this feature is just configured for selected tool and there is implementation for provider.Currently, supported providers are:GitHub- versions by release, tag-release or commitGitLab- versions by release or tag-releaseBitBucket- versions by release or tag-releaseDebian packages- latest package version for any suiteAlpine packages- latest package version for any Alpine versionPyPi- latest release for any packageTools by Didier Stevens- latest release for any published tool in his GitHub repository with similar versioningMultiple origins can be configured for every tool, however two should be enough, and in most cases just one. One for source of the tool (e.g. GitHub) and second origin for installation method in Dockerfile (e.g. tool installed as Alpine package into Dockerfile). Only one is needed and is ideal; hopefully tool is installed from direct source in Dockerfile.Configuring tool to be checked for origin version updatesCurrently, configuration files, so called 'metafiles' are stored into same place as Dockerfiles:CinCan's tool repository.Every file is named asmeta.json.Here isexample configurationof binwalk. It has two providers where another issource codeorigin, and another is justupstream for Dockefile.{"upstreams":[{"uri":"https://github.com/ReFirmLabs/binwalk","repository":"ReFirmLabs","tool":"binwalk","provider":"GitHub","method":"release","origin":true},{"uri":"https://sources.debian.org/api/src/binwalk/","tool":"binwalk","provider":"Debian","method":"release","suite":"buster","docker_origin":true}]}Required attributes depends on provider, but usually at least repository, tool, provider and method are required. URI might be enough some cases.(TODO add provider  specific documentation)These files are stored into Docker image at build phase as last layer into the image, making it possibly to fetch without pulling the whole image.Cache is refreshed every 24 hours.However, there is option make tool to use local path, and disable remote downloading to help development. Path could be for example place, where you clonetoolsrepository.Seeconfiguration for more details.Adding new providerAdding new provider is straighforward -inherit UpstreamCheckerclass and add implementation in the same folder. Existing implementations can be used as model.Short idea is, that there is meta file for every tool, containing upstream information in JSON format, and based on this information, correct provider implementation is selected, and tool information is forwarded for it, and finally version information is fetched from correct place with this way.New provider class should implement at least one method_get_version()which returns latest version of tool from the provider, based on configuration.As result. thischeckerjust returns latest available version in configured format. (Latest release number, latest tag, latest commit? Or something else.)Additionally, ininitfile, provider name must be mapped for correct classname.JSON file is given as dictionary parameter into constructor of UpstreamChecker, so all values should be accessible in child-class.Once this is implemented, everything else should work automatically. Either tool with configuration for new provider is requred for testing, or simply add tests for it intestsfolder.Additional configurationBy default, configuration file is stored in $HOME/.cincan/registry.yamlDifferent file can be used with--configor-coption.Configuration file does not have many options, but some are needed.Data from remote registry and from upstreams is cached into specific path withcache_pathattribute. This contains SQLite 3 database, which is used to manage all information.tools_repo_pathattribute can be used to set path for locally clonedCinCan tools repository. Then this local directory is used for meta files instead. Also some utilities are using this path (e.g. syncing README and description information into Docker Hub from tools repository.)tokensattribute can contain multiple tokens with schema <provider>:token.
Tokens are helpful in cases, when API limit is needed to be increased for version checking. Caching is used to reduce the amount of requests.metadata_filenameis filename to be checked as metafile.disable_remotewhen set asTrue, disables downloading of metafiles from GitLab, which are required for upstream checking, if files do not exist yet. Disabling might be helpful when some tool is in development phase and version checking is just to be added.# Configuration file of the cincan-registry Python moduleregistry:Quay# Default registry wherefrom tools are usedcache_path:/home/nicce/.cincan/cache# All cache files are in heretools_repo__path:None# Path for local 'tools' repository (Use metafiles from there)# Configuration related to tool metafiles.branch:master# Branch in GitLab for acquiring metafilesmeta_filename:meta.json# Filename of metafile in GitLabindex_filename:index.yml# Filename of index file in GitLabdisable_remote:False# Disable fetching metafiles from GitLabtokens:# Possible authentication tokens to Quay, GitLab, GitHub and so on. Quay token is used for README updating.quay:'<my-token>'github:'<my-token>'Extra informationThis tool takes advantage ofDocker Hub's Registry APIfrom the selected registry, when listing remote tools and their sizes and versions. Version information is extracted fromcontainer configfile, which is containing the configuration of Docker Image.Manifesthas been used to detect the SHA256 digest for container config to be able to download it, as Manifest Schema v2 requires.Version information value is acquired forTOOL_VERSIONenvironment variable in the configuration.This same variable is used for acquiring the version information on local as well, however we are using configuration information provided byDocker Engine.Docker Images should have been build by using this variable, so the information is correct.Manifest is also used to identify final layer from the image, to downloadmeta.jsonfile.