Package: zubbi
The Zuul Building Blocks Index (aka Zubbi) makes it easy to search for available
jobs and roles ("Building Blocks") within aZuulbased CI system - even if they are spread over multiple tenants or repositories.Contents:Architecture|Quickstart|Development|Scraper usage|Configuration Examples|Available Connections|ArchitectureZubbi consists of two parts,zubbi webandzubbi scraper. It usesElasticsearchas storage backend and needsGit repositoriesas
source for job and role definitions.Zubbi webA web frontend based on Flask that reads the data from Elasticsearch. It allows
searching for roles and jobs used within the CI system and shows the results
including their documentation, last updates, changelog and some additional meta
data.Zubbi scraperA Python application that scrapes Git repositories, searches for job and
role definitions in specific files and stores them in Elasticsearch.QuickstartPrerequisites:Docker ComposeZubbi can simply be started by using the provideddocker-compose.yamlfile.NOTEThe providedDockerfileshould only be used for demonstration purposes and not
in a production system. Flask is running in development mode and listens on all
public IPs to make it reachable from outside the docker container.To get the whole stack up and running, do the following:$cddocker
$docker-composebuild
$docker-composeupThis will build the docker container with the newest Zubbi version, start all
necessary services (Elasticsearch, zubbi-scraper, zubbi-web) and does a full
scrape of theopenstack-infra/zuul-jobsrepository to get an initial set of
data.When everything is up, you can visithttp://localhost:5000and explore the jobs
and roles from theopenstack-infra/zuul-jobsrepo.DevelopmentPrerequisites: Python 3.6,ToxandPipenvinstalled.To install necessary dependencies for development, run:$pipenvshell
$pipenvinstall--devWe are usingblackto ensure
well-formatted Python code. To automatically ensure this on each commit, you can
use the included pre-commit hook. To install the hook, simply run:$pre-commitinstallBefore submitting pull requests, run tests and static code checks using tox:$toxInstalling & updating dependenciesNew dependencies should be added to therequireslist in thesetup.pyfile:requires=["arrow","click",...,"<new dependency>",]Afterwards, run the following command to update thePipfile.lockand install the
new dependencies in your local pipenv environment:$pipenvupdateTest dependencies should be installed as development dependencies:$pipenvinstall--devmy-test-dependencyTo update the dependencies to the latest version or after a new dependency was
installed you have to runtox -e update-requirementsand commit the changed
Pipenv and requirements files.Configuring and starting ZubbiIf you followed theDevelopmentguide so far, you should already
have a virtual environment with all required packages to run Zubbi. What's left,
are a few configuration files and a local Elasticsearch instance for testing.ElasticsearchZubbi is currently depending on Elasticsearch as data backend. If you haveDocker Composeinstalled, you can use
the provideddocker-compose.yamlfile to start Elasticsearch locally.$cddocker
$docker-composeupelasticsearchIf not, we recommend to use the latest available Elasticsearch Docker image, to
get a local instance up and running for development.ConfigurationBoth - Zubbi scraper and Zubbi web - read their configuration from the file path
given via theZUBBI_SETTINGSenvironment variable:$exportZUBBI_SETTINGS=$(pwd)/settings.cfgIn order to show jobs and roles in Zubbi, we need to provide a minimaltenant configurationcontaining at least a single repository (which is used as source).
Therefore, put the following in atenant-config.yamlfile:-tenant:name:openstacksource:openstack-gerrit:untrusted-projects:-openstack-infra/zuul-jobsPut the following in yoursettings.cfgto allow scraping based on the tenant
configuration above and store the results in the local Elasticsearch instance.
Please note, that the key in theCONNECTIONSdictionary must go in hand with
thesourcenames in the tenant configuration.ELASTICSEARCH={'host':'localhost','port':9200,}TENANT_SOURCES_FILE='tenant-config.yaml'CONNECTIONS={'openstack-gerrit':{'provider':'git','git_host_url':'https://git.openstack.org',},}Running ZubbiNow we can scrape theopenstack-infra/zuul-jobsrepository to get a first set
of jobs and roles into Elasticsearch and show them in Zubbi:$zubbi-scraperscrape--fullWhen the scraper run was successful, we can start Zubbi web to take a look at
our data:$exportFLASK_APP=zubbi
$exportFLASK_DEBUG=true$flaskrunBuilding the syntax highlighting stylesheet with pygmentsWe are using a pre-build pygments stylesheet to highlight the code examples in
job and roles documentations. In case you want to rebuild this syntax highlighting
stylesheet (e.g. to try out another highlighting style) you can run the following
command:$pygmentize-Sdefault-fhtml-a.highlight>zubbi/static/pygments.cssScraper usageThe Zubbi scraper supports two different modes:periodic(default) andimmediate.
To start the scraper in periodic mode, simply run:$zubbi-scraperscrapeThis should also scrape all repositories specified in the tenant configuration
for the first time.To immediately scrape one or more repositories, you can use the following command:# Scrape one or more repositories$zubbi-scraperscrape--repo'orga1/repo1'--repo'orga1/repo2'# Scrape all repositories$zubbi-scraperscrape--fullAdditionally, the scraper provides alist-reposcommand to list all
available repositories together with some additional information like the
last scraping timestamp and the git provider (connection type):$zubbi-scraperlist-reposConfiguration examplesExamples for all available settings can be found insettings.cfg.example.Tenant ConfigurationZubbi needs to know which projects contain the job and role definitions that
are used inside the CI system. To achieve this, it uses Zuul'stenant configuration.
Usually, this tenant configuration is stored in a file that must be specified
in thesettings.cfg, but it could also come from a repository.# Use only one of the following, not bothTENANT_SOURCES_FILE='<path_to_the_yaml_file>'TENANT_SOURCES_REPO='<orga>/<repo>'Elasticsearch ConnectionThe Elasticsearch connection can be configured in thesettings.cfglike
the following:ELASTICSEARCH={'host':'<elasticsearch_host>','port':9200,# default'user':'<user>','password':'<password>',# Optional, to avoid name clashes with existing ES indices from other applications# E.g. 'zubbi' will result in indices like 'zubbi-zuul-jobs', 'zubbi-ansible-roles', ...index_prefix:'<prefix>',# Optional, to enable SSL for the Elasticsearch connection.# You must at least set 'enabled' to True and provide other parameters if the default# values are not sufficient.'tls':{'enabled':False,# default'check_hostname':True,# default'verify_mode':'CERT_REQUIRED',# default},}Available ConnectionsCurrently, Zubbi supports the following connection types:GitHub,GerritandGit. The latter one can be used for repositories that are not hosted on
either GitHub or Gerrit.GitHubThe GitHub connection uses GitHub's REST API to scrape the repositories. To be
able to use this connection, you need to create a GitHub App with the following
permissions:Repository contents:Read-onlyRepository metadata:Read-onlyIf you are unsure about how to set up a GitHub App, take a look at theofficial guide.Once you have successfully created your GitHub App, you can define the connection
with the following parameters in yoursettings.cfgaccordingly:CONNECTIONS={'<name>':{'provider':'github','url':'<github_url>','app_id':<your_github_app_id>,'app_key':'<path_to_keyfile>',},...}Using GitHub WebhooksGitHub webhooks can be used to keep your Zubbi data up to date.
To activate GitHub webhooks, you have to provide a weebhook URL pointing to
the/api/webhookendpoint of your Zubbi web installation. The generated webhook
secret must be specified in theGITHUB_WEBHOOK_SECRETsetting in yoursettings.cfg:NOTE:As of now, GitHub webhooks are not supported on a per-connection base.
You can only have one webhook active in zubbi.GITHUB_WEBHOOK_SECRET='<secret>'Zubbi web receives webhook events from GitHub, validates the secret and publishes
relevant events to the scraper viaZMQ.
The Zubbi scraper on the other hand subscribes to the ZMQ socket and scrapes
necessary repositories whenever a event is received. In order to make this
communication work, you need to specify the following parameters in yoursettings.cfg:# Zubbi web (publish)ZMQ_PUB_SOCKET_ADDRESS='tcp://*:5556'# Zubbi scraper (subscribe)ZMQ_SUB_SOCKET_ADDRESS='tcp://localhost:5556'GerritIn contrary to GitHub, the Gerrit connection is based onGitPythonas the Gerrit REST API
does not support all use cases. To use this connection, you have
to provide the following parameters in yoursettings.cfg:CONNECTIONS={'<name>':{'provider':'gerrit','url':'<git_remote_url>',# Only necessary if different from the git_remote_url'web_url':'<gerrit_url>',# The web_type is necessary to build the correct URLs for Gerrit.# Currently supported types are 'cgit' (default) and 'gitweb'.'web_type':'cgit|gitweb',# Optional, if authentication is required'user':'<username>','password':'<password>',},...}GitThe Git connection is also based onGitPythonand can be used for Git
repositories that are not hosted on either GitHub or Gerrit. To use this connection,
put the following in yoursettings.cfg:CONNECTIONS={'<name>':{'provider':'git','url':'<git_host_url>',# Optional, if authentication is required'user':'<username>','password':'<password',},...}Happy coding!