Portal:Cloud VPS/Admin/Openstack source

This page describes how we fetch Openstack source code and install it to our machines.
This can be different depending on the deployment, component (horizon, neutron, nova, etc), release (mitaka, newton, etc) and target operating system (jessie, stretch, buster, etc).
= From Debian packages =
The majority of the Openstack we use comes from Debian packages.
In the old/realy days of Openstack, all releases were provided by a public apt repository by Mirantis Inc. This is no longer the case, but Debian/Ubuntu are still reference platforms for running Openstack, so complete suites of packages can be found ready to install.
In order to discover which openstack version are in official Debian repositories, one can check some useful links:
* Debian snapshot archive: neutron keystone nova
* Debian package tracker: neutron keystone nova
* Debian maintainer dashboard: all packages version matrix
* Debian QA overview: all packages version matrix
== Version matrix ==
Locating the right Openstack version for the right target operating system may not be obvious at first sight.
Openstack release	Openstack version	DEB repository 	Compatible Debian version 	Notes
Mitaka 	13 	jessie-backports Debian repo, or:
deb http://osbpo.debian.net/debian jessie-mitaka-backports main
deb http://osbpo.debian.net/debian jessie-mitaka-backports-nochange main	Debian Jessie, Debian Stretch	Packages are in our reprepro, component 'openstack-mitaka-jessie'
Newton 	14 	Stretch Debian mirrors 	Stretch 	
Ocata 	15 	deb http://osbpo.debian.net/debian ocata main	Stretch 	Packages are in our reprepro, component 'openstack-ocata-stretch'
Pike 	keystone 12
nova 16
neutron 11
glance 15
designate 5	deb http://osbpo.debian.net/debian/ stretch-pike-backports main
deb http://osbpo.debian.net/debian/ stretch-pike-backports-nochange main	Stretch 	
Queens 	17 	deb http://osbpo.debian.net/debian stretch-queens-backports main
deb http://osbpo.debian.net/debian stretch-queens-backports-nochange main	Stretch 	
Rocky 	18 	Official Debian mirrors for Buster. For Stretch backports:
deb http://osbpo.debian.net/debian stretch-rocky-backports main
deb http://osbpo.debian.net/debian stretch-rocky-backports-nochange main	Stretch, Buster 	
Stein 	19 	deb http://osbpo.debian.net/debian buster-stein-backports main
deb http://osbpo.debian.net/debian buster-stein-backports-nochange main	Buster 	
Train 	20 	deb http://osbpo.debian.net/debian buster-train-backports main
deb http://osbpo.debian.net/debian buster-train-backports-nochange main	Buster 	
Openstack release	Openstack version	DEB repository 	Compatible Debian version 	Notes
== Server vs Client packages ==
We differentiate between clients and server packages.
* clients are usually python libraries and utilities that interact with each component restful API.
* server packages provide the daemons and tooling required to run the service itself (including the APIs).
Most of client software components are installed by means of Debian packages.
== puppet code organization ==
Properly handling the version matrix in puppet code can be challenging. The layout we are trying to use right now is outlined here.
There are 2 main profiles that should be declared by each openstack deployment:
* profile::openstack::base::clientpackages
* profile::openstack::base::serverpackages
Usually, these profiles are just declared in the deployment-specific override, example:
* profile::openstack::eqiad1::clientpackages --> declares --> profile::openstack::base::clientpackages
* profile::openstack::eqiad1::serverpackages --> declares --> profile::openstack::base::serverpackages
This profile is responsible for calling the right openstack version/debian version module which contains the actual configuration, for example:
* profile::openstack::base::clientpackages --> declares --> openstack::clientpackages::${version}::${::lsbdistcodename}
* profile::openstack::base::serverpackages --> declares --> openstack::serverpackages::${version}::${::lsbdistcodename}
Here, ${version} is the openstack release in use, which is a different hiera key per deployment.
Each concrete module contains the specifics bits to be able to install the right packages in the system.
In case there are shared bits for client and server packages, these should be placed in the openstack::commonpackages::${version} class,
and this clase should be declared require in the callers. Example:
* openstack::serverpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka
* openstack::clientpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka
Here is a summary list of things that can be included in each class:
* profile::openstack::eqiad1::serverpackages call to the base profile, eqiad1 server package configuration override if required
* profile::openstack::base::serverpackages call to the inner class, global server package configuration overrides if required
* openstack::serverpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common server package installations
* profile::openstack::eqiad1::clientpackages call to the base profile, eqiad1 client package configuration override if required
* profile::openstack::base::clientpackages call to the inner class, global client package configuration overrides if required
* openstack::clientpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common client package installations
* openstack::commonpackages::${version} stuff that is common to both of the above (factorization)
Then, How are component packages really installed? compontents are installed by its own profile/class path, that installs and configures it. Example:
* profile::openstack::eqiad1::nova::compute::service
** profile::openstack::base::nova::compute::service
*** openstack::nova::compute::service
**** openstack::nova::compute::service::mitaka::stretch
***** (require) openstack::serverpackages::mitaka::stretch
****** (require) openstack::commonpackages::mitaka
And, Is such level of indirection / abstraction really needed?
We had really hard times in the past dealing with the version matrix. We were forced to adopt this layout to be able to manage the puppet code for each component/openstack release/debian version sanely.
The level of indirections and abstraction is the necessary evil.
== importing packages into our reprepro ==
In some cases, .deb packages aren't always available. Example is the archival of some Debian backports repository. This already happened, for example in T216497 - CloudVPS: workaround archival of jessie-backports repo.
A common and safe workaround is to import the packages into reprepro.
This is valid in cases such as:
* we don't trust the availability of the original repository
* the original repository is about to be deleted for some reason
* a given repository doesn't contain all the openstack suite
In all these cases, it's advisable to collect and import all openstack packages into reprepro.
= From source code =
There are some components that are deployed/used directly from upstream source code, without Debian packages being involved. This is usually horizon.
TODO: fill me.
= See also =
* Upstream openstack releases information
* Upstream openstack maintenance phases
*OpenStack team at Debian