Package: servox
Opsani ServoXThis repository contains the source code of the Opsani Servo agent.The servo connects applications to the Opsani cloud optimization engine
to identify cost savings and performance enhancements by applying machine
learning technology. Servos are lightweight Python applications and are
typically deployed as a container under an orchestration layer such as
Kubernetes, ECS, or Docker Compose.Servos are composed of connectors, which provide the core functionality for
integrating with metrics, orchestration, and load generation systems/utilities.
The ServoX codebase provides core functionality shared by all servos and a rich
library supporting the development of connectors.Quick StartServoX is a modern Python application distributed as an installable Python
package. Development is done in a Python install managed withPyenvand a virtual environment managed byPoetry. This is the path of least resistance but
any Python package management system should work.Clone the repo:git clone git@github.com:opsani/servoxInstall required Python:cd servox && pyenv installInstall Poetry:curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | pythonLink Poetry with pyenv version:poetry env use `cat .python-version`Install dependencies:poetry installActivate the venv:poetry shellInitialize your environment:servo initStart interacting with the servo:servo --helpOverviewGetting Started with OpsaniAccess to an Opsani optimizer is required to deploy the servo and run the end to
end integration tests. If you do not currently have access to an Opsani
environment but are otherwise interested in working with the optimizer and
Servo, please reach out to us atinfo@opsani.comand
we will get back with you.UsageDisplaying Help❯ servo --helpInitializing an AssemblyNOTE: A Dotenv file is recommended during development to keep the CLI
options under control. The init command will generate one for you.❯ servo initConfiguration#Generateaconfigfilewithallavailableconnectors❯ servo generate#Validateaconfigfile❯ servo validate#Displayconfigschema❯ servo schemaDisplaying Info#Displayallavailableconnectors❯ servo connectors#Displayinstancespecificinfo(requiresconfiguration)❯ servo show connectors❯ servo show events❯ servo show components❯ servo show metricsRunning Operations#Checkservoreadiness❯ servo run --dry-run#Describeapplicationstate❯ servo describe#Captureanddisplayameasurement❯ servo measure#Adjustthememoryofweb-serverto512MBandcpuofgatewayto800millicores❯ servo adjust web.mem=512 gateway.cpu=0.8#Runtheservotostartoptimizing❯ servo runArchitectureServoX has been designed to provide a delightful experience for engineers
integrating cloud optimization into their systems and workflow. Developer
ergonomics and operator efficiency are primary concerns as integrating and
orchestrating disparate components can quickly become tiresome and complex. As a
library, ServoX aspires to be as "batteries included" as possible and support
developers with well designed, implemented, and tested solutions for common
concerns. As a tool, ServoX strives to support system operators and devops
engineers with strong support for common tasks and a high-velocity workflow.There are a few key components that form the foundation of the architecture:Connectors- Connectors are pluggable components that enable the servo to
interact with external systems such as metrics providers (Prometheus, Datadog,
New Relic, etc), orchestration technologies (Kubernetes, cloud provider APIs,
etc), or load generators. Every major functional component (including the
servo itself) is a connector that inherits from theConnectorbase class.
Connectors can process events dispatched from the servo (see Events below),
provide services to the user (see CLI below), and interact with other
connectors.Servo- The Servo class models the active set of connectors and
configuration that is executing. The servo handles connectivity with the
Opsani Optimizer API (see Optimizer below) and is responsible for the primary
concerns of connectivity management and event handling.Configuration- Configuration is a major shared concern in tools such as
Opsani that are designed to integrate with arbitrary systems. Ensuring that
configuration is valid, complete, and functional is a non-trivial task for any
component with more than a few knobs and levers. ServoX provides a rich
configuration subsystem built on Pydantic that makes modeling and processing
configuration very straightforward. Out of the box support is provided for
common needs such as environment variables and dotenv files. Configuration is
strongly validated using JSON Schema and support is provided for generating
config files directly from the connectors.Optimizer- The Optimizer class represents an Opsani optimization engine
that the servo interacts with via an API. The optimizer can be configured via
config file or from the environment; both of these support Kubernetes secrets.Events- The Event subsystem provides the primary interaction point
between the Servo and Connectors in a loosely coupled manner. Events are
simple string values that have connector defined semantics and can optionally
return a result. The Servo base class defines the primary events ofDESCRIBE,MEASURE,ADJUST, andPROMOTEwhich correspond to declaring
the metrics & components that the connector is interested in, taking
measurements and returning aggregated scalar or time series data points,
making changes to the application under optimization, or promoting an
optimized configuration to the broader system.Checks- Checks provide a mechanism for verifying the correctness and
health of connector configuration and operations. They are designed to support
a high throughput integration and debugging experience by providing feedback
loop driven workflow. Checks are implemented on top of the events subsystem
and can be executed alongside an actual run via theservo run --checkCLI command,
or by themselves viaservo run --dry-runThe design of the checks subsystem
is covered in depthin the docs.Assembly- The Servo Assembly models the runtime environment of the servo
outside of a particular configuration. The assembly is the parent of the servo
and is responsible for "assembling" it by instantiating connectors as
configured by the operator. Connectors can be used multiple times (e.g. you
may want to connect to multiple discrete Prometheus services) or may not be
used at all (e.g. you have a New Relic connector in the container image but
aren't using it).CLI- The CLI provides the primary interface for interacting with the
servo. The CLI is modular and contains a number of root level commands and
connectors can optionally register additional commands. Most of the root level
commands are driven through the event subsystem and connectors which respond
to the relevant events will automatically become accessible through the CLI.
For example, executingservo schemawill emit a complete JSON Schema
document for the assembly whileservo schema kuberneteswill emit a JSON
Schema specific to the Kubernetes connector.Understanding EventsThe servo is built around an event driven architecture and utilizes a
lightweight eventing system to pass messages between connectors. Eventing is
implemented in asynchronous Python on top of asyncio. Events are simple strings
identifiers that are bound to a Pythoninspect.Signatureobject. The signature is used when event handlers are registered to enforce a
contract around the parameter and return types, method arity, etc.Any connector can define an event provided that no other connector has already
registered an event with the desired name. TheServoclass defines several
basic events covering essential functionality such as declaring metrics and
components, capturing measurements, and performing adjustments. These events are
integrated into the CLI and readily accessible. For example, executingservo show metricsdispatches themetricsevent to all active connectors and
displays the aggregate results in a table. More substantial commands such asmeasureandadjustwork in the same manner -- dispatching events and
visualizing the results. Whenever possible functionality is implemented via
eventing.When the servo is run, it connects to the Opsani optimizer via the API and
receives instructions about which actions to take to facilitate the optimization
activity. These commands are dispatched to the connectors via events.The default events are available on theservo.servo.Eventsenumeration and
include:EventCategoryDescriptionstartupLifecycleDispatched when the servo is assembled.shutdownLifecycleDispatched when the servo is being shutdown.metricsInformationalDispatched to gather the metrics being measured.componentsInformationalDispatched to gather the components & settings available for adjustment.checkOperationalAsks connectors to check if they are ready to run by validating settings, etc.describeOperationalGathers the current state of the application under optimization.measureOperationalTakes a measurement of target metrics and reports them to the optimizer.adjustOperationalApplies a change to the components/settings of the application under optimization and reports status to the optimizer.Event Handlers & PrepositionsEvent handlers are easily registered via a set of method decorators available on
theservo.connectormodule. Handlers are registered with aprepositionwhich
determines if it is invoked before, on, or after the event has been processed.
Handlers are invoked when the servo or another connector dispatches an event.
Event handlers can be implemented either synchronously or asynchronously
depending on if the method is a coroutine declared with the async prefix.fromtypingimportListimportservoclassSomeConnector(servo.BaseConnector):@servo.before_event('measure')defnotify_before_measure(self)->None:self.logger.info("We are about to measure...")@servo.on_event('metrics')defmetrics(self)->List[servo.Metric]:return[servo.Metric('throughput',servo.Unit.REQUESTS_PER_MINUTE)]@servo.after_event('adjust')defanalyze_results(self,results:List[servo.EventResult])->None:self.logger.info(f"We got some results:{results}")Each preposition has different capabilities available to it. Before event
handlers can cancel the execution of the event by raising aEventCancelledError.
On event handlers can return results that are aggregated and available for
processing. After event handlers get access to all of the results returned by
active connectors via the on event handlers.Creating a new EventEvents can be created either programmatically via theConnector.create_event()class method or declaratively via theevent()decorator:importservoclassEventExample(servo.BaseConnector):@servo.event()asyncdeftrace(self,url:str)->str:...Theeventdecorator uses the parameters and return type of the decorated
method to define the signature requirements for on event handlers registered
against the event. The body of the decorated method must be...,pass, or an
async generator that yieldsNoneexactly once.The body of the decorated method can be used to define setup and tear-down
activities around on event handlers. This allows for common setup and tear-down
functionality to be defined by the event creator. This is achieved by
implementing the body of the decorated method as an async generator that yields
control to the on event handler:fromtypingimportAsyncIteratorimportservoclassSetupAndTearDownExample(servo.BaseConnector):@servo.event()asyncdeftrace(self,url:str)->AsyncIterator[str]:print("Entering event handler...")yieldprint("Exited event handler.")The event decorator can also be used to create an event and register a handler
for the event at the same time. In this example, thehandler=Truekeyword
argument is provided to created the eventandregister the decorated method
as an on event handler for the new event.importservoclassAnotherConnector(servo.BaseConnector):@servo.event('load_test',handler=True)asyncdefrun_load_test(self,url:str,duration:int=60)->str:return"Do something..."Once an event is created, the connector can dispatch against it to notify other
connectors of changes in state or to request data from them.Dispatching an EventConnectors can notify or interact with other connectors by dispatching an event.
When the servo is assembled, an event bus is transparently established between
the connectors to facilitate event driven interaction.fromtypingimportListimportservoclassExampleConnector(servo.BaseConnector):asyncdefdo_something(self)->None:# Gather metrics from other connectorsresults:List[servo.EventResult]=awaitself.dispatch_event("metrics")forresultinresults:print(f"Gathered metrics:{result.value}")Environment Variables & DotenvPay attention to the output ofservo --helpandservo schemato identify
environment variables that can be used for configuration. The servo handles
configuration of deeply nested attributes by building the environment variable
mapping on the fly at assembly time.For convenience, theservoCLI utility automatically supports.envfiles for
loading configuration and is already in the.gitignore. Interacting with the
CLI is much cleaner if you drop in a dotenv file to avoid having to deal with
the options to configure the optimizer. Theservo initcommand will help set
this up interactively.LoggingThe servo base library provides logging services for all connectors and core
components. By default, theservoCLI utility runs at theINFOlog level
which is designed to provide balanced output that informs you of what is
happening operationally without becoming overwhelming and pedantic. During
development, debugging, or troubleshooting it may become desirable to run at an
elevated log level. The log level can be set via a commandline option on theservoutility or via theSERVO_LOG_LEVELenvironment variable. The servo
will emit ANSI colored output tostderrwhen the terminal is a TTY. Coloring
can be suppressed via the--no-colorcommandline option or with theSERVO_NO_COLORorNO_COLORenvironment variables.-l, --log-level [TRACE|DEBUG|INFO|SUCCESS|WARNING|ERROR|CRITICAL]Set the log level  [env var:SERVO_LOG_LEVEL; default: INFO]--no-color                      Disable colored output  [env var:SERVO_NO_COLOR, NO_COLOR]By default, log messages are written tostderrand a file sink at thelogs/subdirectory relative to the servo root. Backtraces are preformatted and as much
context as possible is provided when an exception is logged.Theservo.loggingmodule exposes some interesting functionality for operators
and developers alike. Logging is aware of the eventing subsystem and will
automatically attribute log messages to the currently executing connector and
event context. Long running operations can be automatically reported to the
Opsani API by including aprogresskey with a numeric percentage value ranging
from 0.0 to 100.0. There are several function decorators available that can
provide automatic logging output for entry and exit, execution timing, etc.Dependent libraries such asbackoffhave been configured to emit their logs
into the servo logging module. Every component that has logging support is
intercepted and handled by the logging subsystem and conforms to the log levels
outlined above.Connector DiscoveryConnectors are set up to be auto-discovered using the setuptools entry point
functionality available from the Python standard library. When a new connector
is installed into the assembly, it will be automatically discovered and become
available for interaction.The specific of how this mechanism works is discussed in detail on thePython
Packaging
Guide.The bundled connectors are registered and discovered using this mechanism via
entries in thepyproject.tomlfile under the[tool.poetry.plugins."servo.connectors"]stanza.If you're writing your own connector in an external package, you need to include[tool.poetry.plugins."servo.connectors"]"my_connector"= "my_project.foo:MyConnector"in yourpyproject.tomlto add your connector as an entry point.
and youmustname your connectorclass MyConnector(servo.BaseConnector):to have it discoverable.
Discovery is performed via theservo.connector:_name_for_connector_class()function. It matches
theMyin the connector class name to the top level connector key in theservo.yaml.
So, for example, yourservo.yamlcould be as follows:...my:foo:barbaz:bat...Running Multiple Connector InstancesServoX is designed to support assemblies that contain an arbitrary number of
connectors that may or may not be active and enable the use of multiple
instances of a connector with different settings. This introduces a few modes of
configuration.The servo looks to aconnectorsconfiguration key that explicitly declares
which connectors are active within the assembly. If aconnectorskey is not
present in the config file, then all available connectors become optionally
available based on the presence or absence of their default configuration key.
For example, an assembly that includes New Relic, Datadog, and SignalFX
connectors installed as Python packages with the following configuration would
only activate Datadog due to the presence of its configuration stanza:datadog:setting_1:some valuesetting_2:another valueThis mode supports the general case of utilizing a small number of connectors in
"off the shelf" configurations.From time to time, it may become necessary to connect to multiple instances of a
given service -- we have seen this a few times with Prometheus in canary mode
deployments where metrics are scattered across a few instances. In these cases,
it can become necessary to explicitly alias a connector and utilize it under two
or more configurations. In such cases, theconnectorskey becomes required in
order to disambiguate aliases from configuration errors. In such cases, theconnectorskey can be configured as a dictionary where the key identifies the
alias and the value identifies the connector:connectors:prom1:prometheusprom2:prometheusprom1:setting_1:some valuesetting_2:another valueprom2:setting_1:some valuesetting_2:another valueIt is also possible to utilize theconnectorskey in order to exclude
connectors from the active set. This can be done with the dictionary syntax
referenced above or using an array syntax if aliasing is not being utilized. For
example, given a configuration with New Relic and Prometheus active but some
sort of issue warranting the isolation of Prometheus from the active set, the
config file might be configured like:connectors:-new_relicprometheus:setting_1:some valuesetting_2:another valuenew_relic:setting_1:some valuesetting_2:another valueThese configurations can be generated for you by providing arguments toservo generate. A space delimited list of connector names will explicitly populate
theconnectorskey and a syntax ofalias:connectorwill configure aliases:❯ servo generate foo:vegeta bar:kubernetesbar:description: Update the namespace, deployment, etc. to match your Kubernetes clusternamespace: defaultconnectors:bar: kubernetesfoo: vegetafoo:description: Update the rate and target/targets to match your load profileduration: 5mrate: 50/1starget: https://example.com/Generated servo.yamlRunning Multiple ServosServoX is capable of running multiple servos within the same assembly and servos
can be added and removed dynamically at runtime. This is useful for optimizing
several applications at one time from a single servo deployment to simplify
operations or more interestingly to support the integration and automation of
optimization into CI and CD pipelines. For example, it is possible to configure
the build system to trigger optimization for apps as they head into staging or
upon emergence into production.Multi-servo execution mode is straightforward. When theservo.yamlconfig file
is found to contain multiple documents (delimited by---), a servo instance is
constructed for each entry in the file and added to the assembly. There are
however a few differences in configuration options.When multi-servo mode is enabled, the--optimizer,--token,--token-file,--base-url, and--urloptions are unavailable. The optimizer and
connectivity configuration must be provided via theoptimizerstanza within
each configurationdocumentin the config file. The CLI will raise errors if
these options are utilized with a multi-servo configuration because they are
ambiguous. This does not preclude a single servo being promoted into a
multi-servo configuration at runtime -- it is a configuration resolution
concern.When running multi-servo, logging is changed to provide context about the servo
that is active and generating the output. Theservo.current_servo()method
returns the active servo at runtime.Because ServoX is based onasyncioand functions as an orchestrator, it is
capable of managing a large number of optimizations in parallel (we have tested
into the thousands). Most operations performed are I/O bound and asynchronous
but the specifics of the connectors used in a multi-servo configuration will
have a significant impact on the upper bounds of concurrency.Configuring Multi-servo ModeBasically all that you need to do is use the---delimiter to create multiple
documents within theservo.yamlfile and configure anoptimizerwithin
each one. For example:---optimizer:id:newco.com/awesome-app1token:00000000-0000-0000-0000-000000000000connectors:[vegeta]vegeta:duration:5mrate:50/1starget:GET https://app1.example.com/---optimizer:id:newco.com/awesome-app2token:00000000-0000-0000-0000-000000000000connectors:[vegeta]vegeta:duration:5mrate:50/1starget:GET https://app2.example.com/Adding & Removing Servos at RuntimeServos can be added and removed from the assembly at runtime via methods on theservo.Assemblyclass:importservoassembly=servo.current_assembly()new_servo=servo.Servo()assembly.add_servo(new_servo)assembly.remove_servo(new_servo)Extending the CLIShould your connector wish to expose additional commands to the CLI, it can do
so via theConnectorCLIclass. Instances are automatically registred with the
CLI and theContextis configured appropriately when commands are invoked. All
CLI extensions are namespaced as subcommands to keep things tidy and avoid
naming conflicts.A simple example of a CLI extension that will registerservo vegeta attackis:importservoimportservo.cliimportservo.connectors.vegetacli=servo.cli.ConnectorCLI(servo.connectors.vegeta.VegetaConnector,help="Load testing with Vegeta")@cli.command()defattack(context:servo.cli.Context):"""Run an adhoc load generation"""context.connector.measure()Requirements & DependenciesServoX is implemented in Python and supported by a handful of excellent
libraries from the Python Open Source community. Additional dependencies in the
form of Python packages or system utilities are imposed by connectors (see
below).Python3.6+ - ServoX makes liberal use of type
hints to annotate the code and drive some functionality.Pydantic- Pydantic is a fantastic
parsing and validation library that underlies most classes within ServoX. It
enables the strong modeling and validation that forms the core of the
configuration module.Typer- Typer provides a nice, lightweight
enhancement on top ofClickfor
building CLIs in Python. The CLI is built out on top of Typer.httpx- httpx is a (mostly) requests
compatible HTTP library that provides support for HTTP/2, is type annotated,
has extensive test coverage, and supports async interactions on top of
asyncio.loguru- A rich Python
logging library that builds on the foundation of the standard library logging
module and provides a number of enhancements.DevelopmentContributing to ServoXOpen Source contributions are always appreciated. Should you wish to get
involved, drop us a line via GitHub issues or email to coordinate efforts.It is expected that most Open Source contributions will come in the form of new
connectors. Should you wish to develop a connector, reach out to us at Opsani as
we have connector developer guides that are in pre-release while ServoX matures.Visual Studio CodeThe core development team typically works in VSCode. Poetry and VSCode have not
quite yet become seamlessly integrated. For your convenience, there are a couple
of Makefile tasks that can simplify configuration:make init- Initialize a Poetry environment, configure.vscode/settings.json,
and then run the `servo initialize command.make vscode- Export the Poetry environment variables and then open the
local working copy within VSCode. The built-in terminal and Python extension
should auto-detect the Poetry environment and behave.Pre-commit HookThe project is configured with apre-commithook to
enforce as much of the coding standards and style guide as possible. To install
it into your working copy, run:poetry run pre-commit installDeveloping with Local PackagesWhen developing against dependencies or building out new connectors, it can be
useful to utilize a local package so that development can be done in the
dependency and within the servox package at the same time.To do so, utilize Poetry Path based dependencies with thedevelop = trueflag
by adding a path reference to the package into the totool.poetry.dev-dependenciesstanza of thepyproject.tomlfile, and then runpoetry update [dependency name].For example, if developing on servox and the statesman state machine library in
a working copy in the parent directory, you would add:[tool.poetry.dev-dependencies]# ...statesman={path="../statesman",develop=true}And then runpoetry update statesman.Changes made to the dependency are immediately visible to servox, making it
easy to develop across package boundaries efficiently.Linting and FormattingThe project is structured to support and enforce consistent, idiomatic code. A
number of tools have been integrated to provide linting and automatic
reformatting when possible/appropriate.This functionality is exposed via tasks in theMakefile, Git commit hooks, and
GitHub Actions.During local development, you may wish to invoke these tools as follows:make lint- Run the linters and output actionable feedback for conformance
with the engineering standards and style of the project.make format- Automatically apply changes to the working copy to conform
with project standards where possible (e.g., reformatting imports).make pre-commit- Run all pre-commit hooks in the manual hook stage. There
are a number of standards that are currently soft enforced in the interest of
pragmatism. Eventually the set of pre-commit and lint checks will form a
perfect union but we are not there yet. Most hooks set in manual mode have to
do with fit and finish concerns such as docstring tone/formatting.make test- Execute the test suite and generate a code coverage report.Linting is automatically performed when a new branch is pushed or a PR is opened
and conformance can be remediated during post-implementation code review.Connecting to Optimization APIsOptimizer API and authentication can configured be inline with your config yaml (servo.yaml)optimizer:
  base_url: some_url
  id: org.domain/app
  ...These details can also be configured via environment variables as highlighted in the following sections.
Note these sections are mutually exclusive: Opsani API refers to Opsani managed endpoints wheras the Appdynamics
API refers to upcoming FSO Optimization. Which API type is used is dynamically derived during configuration/env var parsing by
thepydantic configuration model (see BaseServoConfiguration.optimizer).Opsani APIBy default, the servo will connect to the primary Opsani API hosted onhttps://api.opsani.com. This behavior can be
overridden via CLI options and environment variables:Config OptionEnvironment VariableDescriptionExampleoptimizer.base-urlOPSANI_BASE_URLSets an alternate base URL. The path is computed using the provided optimizer id. Useful for staging deployments and integration tests.OPSANI_BASE_URL=https://staging-api.opsani.com:3456 servo runoptimizer.urlOPSANI_URLSets an explicit URL target. When given, the base URL is ignored and paths are not computed. Useful in unit tests and workstation development with a partial server stack.OPSANI_URL=http://localhost:1234 servo runoptimizer.idOPSANI_IDIdentifier of the application as provided by the Opsani optimizer provisioned for your applicationOPSANI_ID=newco.com/awesome-app1 servo runoptimizer.tokenOPSANI_TOKENAuthentication token as provided by the Opsani optimizer provisioned for your applicationOPSANI_TOKEN=00000000-0000-0000-0000-000000000000 servo runAppDynamics APIConfig OptionEnvironment VariableDescriptionExampleoptimizer.base-urlAPPD_BASE_URLSets an alternate base URL. The path is computed using the provided workload and tenant ids.APPD_BASE_URL=https://optimize-ignite-test.saas.appd-test.com/ servo runoptimizer.urlAPPD_URLSets an explicit URL target. When given, the base URL is ignored and paths are not computed. Useful in unit tests and workstation development with a partial server stack.APPD_URL=http://localhost:1234 servo runoptimizer.tenant_idAPPD_TENANT_IDIdentifier of the application tenantAPPD_TENANT_ID=00000000-0000-0000-0000-000000000000 servo runoptimizer.optimizer_idAPPD_OPTIMIZER_IDBase64 encoded identifier of the workload to be optimized as provided by the provisioned optimizer tenantAPPD_OPTIMIZER_ID=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==== servo runoptimizer.client_idAPPD_CLIENT_IDAuthentication client ID as provided by the Opsani optimizer provisioned for your applicationAPPD_CLIENT_ID=<client id text> servo runoptimizer.client_secretAPPD_CLIENT_SECRETAuthentication client secret as provided by the Opsani optimizer provisioned for your applicationAPPD_CLIENT_SECRET=<client secret text> servo runDocker & ComposeDockerfileanddocker-compose.yamlconfigurations are available in the
repository and have been designed to support both development and deployment
workflows. Configuration file mounts and environment variables can be used to
influence the behavior of the servo within the container.TheSERVO_ENVbuild argument controls the target environment for the built
image. Building withSERVO_ENV=productionexcludes development packages from
the image to reduce size and build time.Pre-built Docker images are available onopsani/servoxon
Docker Hub. The documentation for these images is available within this
repository atdocs/README-DOCKER_HUB.md.The latest release version is available under theopsani/servox:latesttag.
Themaindevelopment branch is published as theopsani/servox:edgetag.Docker images are built and published to Docker Hub via theDocker GitHub Actions Workflow. The workflow
builds branches, published releases, and themainintegration branch. Pull
Requests are not published to Docker Hub because as a publicly available
repository it could become an attack vector.Git branches and Docker images have differing naming constraints that impact how
tag names are computed. For example, Docker tags cannot contain slashes, which
is a common practice for namespacing branches and tags in Git. As such, slashes
are converted to hyphens when computing tag names for branches and tags. The
full naming constraints on Docker image tags is covered in thedocker tagdocumentation.Pre-built images are built using BuildKit and can be used as the basis for very
fast customized builds:❯ DOCKER_BUILDKIT=1 docker build -t servox --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from opsani/servox:edge .Switching Python InterpretersAfter changing Python interpreter versions you may find that you are "stuck" in
the existing virtual environment rather than your new desired version.The problem is that Poetry is linked against the previous environment and needs
a nudge to select the new interpreter.The project is bound to a local Python version via the.python-versionfile.
Tell Poetry to bind against the locally selected environment via:poetry env use `cat .python-version`When upgrading between point releases of the Python interpreter, you may need
to tear down and recreate your venv:poetry env remove `cat .python-version` && poetry installTestingTests are implemented usingpytestand
live in thetestssubdirectory. Tests can be executed directly via thepytestCLI interface (e.g.,pytest tests) or viamake test, which
will also compute coverage details.ServoX is developed with a heavily test-driven workflow and philosophy. The
framework strives to provide as much support for robust testing as possible
and make things that you would think are very hard to test programmatically
very simple. You will want to familiarize yourself with what is available,
there are tools that can dramatically accelerate your development.Test TypesServoX divides the test suite into three types:Unit,Integration, andSystemtests.Tests are identified within the suite in two ways:Use of pytest markers to annotate the tests in code.File location within the tests/ subdirectory.Unit tests are the default type and make up the bulk of the suite. They either
exercise code that carries no outside dependencies or utilize isolation
techniques such as mocks and fakes. They are fast and highly effective for
validating defined behavior and catching regressions.Integration tests do not interact with an Opsani Optimizer but do interact with
external systems and services such as Kubernetes and Prometheus. It is common to
utilize a local environment (such as Docker, Compose, kind, or Minikube) or a
dedicated cloud instance to host the services to interact with. But the focus of
the tests are on implementing and verifying the correct behaviors in a
supportive environment.System tests are much like integration tests except that theyn are highly
prescriptive about the environment they are runnuing in and interact with a real
optimizer backend as much as is practical. System tests sit at the top of the
pyramid and it is expected that there are comparatively few of them, buit they
deliver immense value late in a development cycle when code correctness has been
established and deployment environments and compatibility concerns come to the
forefront.All test types can be implemented within any test module as appropriate by
annotating the tests with pytest markers:importpytest@pytest.mark.integrationclassTestSomething:...Tests without a type mark are implicitly designated as unit tests for
convenience. When multiple type marks are in effect due to hierarchy, the
closest mark to the test node has precedence.There are also dedicated directories for integration and system tests attests/integrationandtests/systemrespectively. These dedicated directories
are home to cross cutting concerns and scenarios that do not clearly belong to a
specific module. Tests in these directories that are marked with other types
will trigger an error.Running TestsA key part of any testing workflow is, well, running tests. pytest provides a
wealth of capabilities out of the box and these have been further augmented with
custom pytest-plugins within the suite.Let's start with the basics: executingpyteststandalone will execute the unit
tests intests. Runningpytest --verbose(or -v) will provide a one test per
line output which can be easier to follow in slower runs.Individual files can be run by targetting them by name:pytest tests/connector_test.py. Individual test functions and container test classes
within the file (known as nodes in pytest parlance) can be addressed with thetests/path/to/test_something.py::TestSomething::test_namesyntax.Tests can also be flexibly selected by marks and naming patterns. Invokingpytest -m asyncio -k sleepwill select all tests with the asyncio mark and
have the word "sleep" in their name. These arguments support a matching syntax,
look into the pytest docs for details.The ServoX test types have specific affordances exposed through pytest. Runningpytest -T integrationwill select and run all integration tests, but deselect
all other types. The-Talso known as--typeflag supports stem matching for
brevity:pytest -T uwill select the unit tests.Because they are slow and require supplemental configuration, integration and
system tests are skipped by default. They can be enabled via the-I, --integrationand-S, --systemswitches, respectively. Note the difference
in behavior between the flags:pytest -I -Swill result in all the tests being
selected for run whereaspytest -T systargets the system tests exclusively.
Once the-I, -Sflags have been used to enable the tests, they can be further
filtered using-mand-k. If you are thoughtful about how you name your
tests and leverage marks when it makes sense, it can become very easy to run
interesting subsets of the suite.By default, pytest uses output buffering techniques to capture what is written
to stdout and stderr. This can become annoying if you are trying to introspect
state, print debugging info, or get a look at the servo logs. You can suppress
output capture via the-sswitch. This is typically only recommended when
running a small number of tests because the output quickly becomes
incomprehensible.If you are working through a set of failures, you can rerun the tests that
failed on the last run via the--lf, --last-failedflag. The--ff, --failed-firstflag will rerun all of the tests, but run the previously failed
tests first. Similarly,--nf, --new-firstwill run the full suite but
prioritize new files. Thepytest-pickedplugin provides additional targeting
based on git working copy status -- runningpytest --pickedwill find unstaged
files to run.Finally, the--sw, --stepwiseand--sw-skip, --stepwise-skipflags allow you
to methodically working through a stack of failures by resuming from your last
failure and then halting at the next one.Makefile TasksTest automation tasks are centralized into theMakefile. There are a number of
testing  tasks availble including:make test- Run all available tests.make test-unit- Run unit tests.make test-integration- Run integration tests.make test-system- Run system tests.make test-coverage- Run all available tests and generate code coverage
report.make test-kubeconfig- Generate a kubeconfig file at tests/kubeconfig. See
details inIntegration Testingbelow.make autotest- Automatically run tests based on filesystem changes.Testing tasks will run in subprocess distributed mode by default (see below).Integration TestingThe test suite includes support for integration tests for running tests against
remote system components such as a Kubernetes cluster or Prometheus deployment.
Integration tests require akubeconfigfile attests/kubeconfig.By convention, the default integration testing cluster is namedkubetestand themake test-kubeconfigtask is provided to export the cluster details
from your primary kubeconfig, ensuring isolation.Interaction with the Kubernetes cluster is supported by the most excellentkubetestlibrary that provides
fixtures, markers, and various testing utilities on top of pytest.To run the integration tests, executepytest -Ito enable the
marker. Integration tests are much slower than the unit test suite
and should be designed to balance coverage and execution time.System tests are enabled by runningpytest -S. Systems tests are very similar
to integration tests in implementation and performance, but differ in that they
are prescriptively bound to particular deployment environments and interact with
an actualTests can also be run in cluster by packaging a development container and
deploying it. The testing harness will detect the in-cluster state and utilize
the active service account.Continuous IntegrationThe project is configured to run CI on unit tests for all branches, tags, and
pull requests opened against the repository. CI for integration and system tests
is constrained by a few rules because they are so resource intensive and may be
undesirable during experimental developmenmt or integrating multiple branches
together.Integration and system tests are run if any of the following conditions are
met:A push is made tomain.A push is made to a branch prefixed withrelease/.A push is made to a branch prefixed withbugfix/.A tag is pushed.A push is made to a branch with an open pull request.The commit message includes#test:integrationand/or#test:system.The default unit test job that is executed for all pushes generates code
coverage reports and XML/HTML report artifacts that are attached to the run. The
integration and system test jobs report on the runtime duration of the tests to
help identify and manage runtime creep.The unit, integration, and system test jobs all utilize thepytest-xdistplugin to split the test suite up across a set of subprocesses. This is
discussed in theDistributed Testingsection.Docker images are built and pushed to Docker Hub automatically for all
pushed refs. Release tags are handled automatically and theopsani/servox:latesttag is advanced when a new version is released. Themainbranch is built and pushed to theopsani/servox:edgetag.Distributed TestingThe project is configured to leverage locally distributed test execution by
default. Servo workloads are heavily I/O bound and spend quite a bit
of time awaiting data from external services. This characteristic makes tests
inherently slower but also makes them very well suited for parallel execution.
TheMakefiletasks and GitHub actions are configured to leverage a subprocess
divide & conquer strategy to speed things up.This functionality is provided bypytest-xdist.Manifest TemplatingAll manifests loaded through kubetest support Mustache templating.
A context dictionary is provided to the template that includes references to all
Kubernetes resources that have been loaded at render time. Thenamespaceandobjsare likely to be the most interesting.Startup BannerJust for fun, ServoX generates an ASCII art banner at startup. The font is
randomized from a painstakingly selected set of style that represent the ServoX
vibe. The output color is then randomized and has a 50/50 shot of being rendered
as a rainbow or a randomly selected flat output color.Don't like serendipity?You can take control of the banner output with two environment variables:SERVO_BANNER_FONT: Name of theFigletfont to render the banner in.SERVO_BANNER_COLOR: The color to use when rendering the banner. Valid
options are:REDGREENYELLOWBLUEMAGENTACYANRAINBOWRelease ProcessUseSemantic Versioningto choose a release number.
Runpoetry version [major | minor | patch]to increment the version. This
will update the version number asversionin  section[tool.poetry]of
thepyproject.tomlfile.Minor version release series each have a unique cryptonym. If incrementing
the major or minor version, choose a new cryptonym for the release series and
set it as the value of__cryptonym__inservo/__init__.py.UpdateCHANGELOG.mdwith the notable changes of the release.
See the introduction for guidance on the form and format. Follow the pattern
of prior releases for the release title, generally[x.y.z] "cryptonym" - YYYY-MM-DD.Commit changes and make sure all tests pass.Draft a new release on
GitHubwith the following
settings:Tag version: set to vX.Y.Z (e.g.,v0.9.5) again themainbranch.Release title: set tovX.Y.Z "<cryptonym>"Description: judiciously paste the change log since the prior releasePublish the release, wait for CI actions to complete. Release artifacts are
automatically created and published via the GitHub Actionsrelease.ymlworkflow. The framework is packaged into a Python library and published toPyPiand Docker images are built and
published toDocker Hub.Verify the new image, with tagvX.Y.Z, has been published inDocker
Hub.ContributingPlease reach out to support at opsani.com.LicenseServoX is distributed under the terms of the Apache 2.0 Open Source license.A copy of the license is provided in theLICENSEfile at the root of
the repository.