Package: benchalerts
benchalertsA package to facilitate automated alerting based on Conbench data.OverviewThis package is intended to make the following steps easier in CI. Before these steps,
it is assumed that an execution environment has performed a run of benchmarks and
submitted the results to Conbench.Hit the Conbench API, which uses thelookback z-score
method, to understand
if there were any:errorsregressions (with configuration for how these regressions may be detected)improvements (with configuration for how these improvements may be detected)Format and submit a summary of these findings to various places (again, with
configuration):GitHub Check on a commit with a Markdown summaryComment on a GitHub pull requestIn the future, there will be more places to submit alerts/reports/summaries, and more
configuration possible.Currently, the way to configure these workflows in CI is to create and run a Python
script that creates and runs an alerting pipeline, like so:importosimportbenchalerts.pipeline_stepsasstepsfrombenchalertsimportAlertPipeline# Pretend we're running on GitHub Actions. Get some environment variables.# (they may be different for different CI systems)commit_hash=os.environ["GITHUB_SHA"]repo=os.environ["GITHUB_REPOSITORY"]build_url=(os.environ["GITHUB_SERVER_URL"]+f"/{repo}/actions/runs/"+os.environ["GITHUB_RUN_ID"])# Create a pipeline to update a GitHub Checkpipeline=AlertPipeline(steps=[steps.GetConbenchZComparisonStep(commit_hash=commit_hash,baseline_run_type=steps.BaselineRunCandidates.fork_point,),steps.GitHubCheckStep(commit_hash=commit_hash,comparison_step_name="GetConbenchZComparisonStep",repo=repo),],error_handlers=[steps.GitHubCheckErrorHandler(commit_hash=commit_hash,repo=repo,build_url=build_url)],)# Run the pipelineprint(pipeline.run_pipeline())See the docstrings of each pipeline step for more details on how to configure the
workflow, including how to set up the required environment variables.GitHub App AuthenticationThe preferred method thatbenchalertsrecommends for authenticating and posting to
GitHub is to use a machine user called aGitHub
App.
Using an App will allow you to post using a "bot" entity without taking up a seat in
your organization, and will allow you to use the extra features of theChecks
API. These
features give much more context when analyzing benchmark runs.Each Conbench server must create its own GitHub App for security reasons. To do so,
follow these instructions.Creating a GitHub App to work withbenchalertsGo to the officialGitHub
instructionsfor creating an App.If you are an admin of your GitHub organization, follow the instructions for "a
GitHub App owned by an organization." This method is preferred because the org
will own the app instead of a user, who may not be part of the org in the
future. (This will not affect the identity of the bot that posts to GitHub, just
the ownership of the App.)If not, you can follow the instructions for "a GitHub App owned by a personal
account." You will send an installation request to org admins after creating the
app. You can always transfer the ownership of the app to an org later.For the App Name, useconbench-<your org>.For the Homepage URL, use the link to your Conbench server.Ignore the Callback URL and Setup URL.Uncheck the "Active" box under Webhook. Since this App will not be an active service,
we don't need GitHub to push webhook events to the App.For full use of this package, the App requires the following permissions:Repository > Checks > Read and WriteRepository > Pull requests > Read and WriteAfter creating the App, save the App ID for later.For the App's photo, usethis
one.In the App Settings, scroll down to Private Keys and generate a private key. This
will download a file to your computer. Treat the contents of this file like a
password.IMPORTANT: After creation, go tohttps://github.com/apps/<YOUR_APP_NAME>/installations/newto install the new App
on the repos you'd like it to be able to post to. You must be a member of the
organization to install the App on. If you are not an admin, an email request will
be sent to org admins, which must be approved.Runningbenchalertsas the GitHub App you createdAll that's necessary to usebenchalertsworkflows that post to GitHub as your App is
to set the following environment variables:GITHUB_APP_ID- the App ID from aboveGITHUB_APP_PRIVATE_KEY- thecontentsof the private key file from above. This is
a multiline file, so ensure you quote the contents correctly if necessary.Sincebenchalertsis usually used in CI, it's recommended to set these two environment
variables in your CI pipeline as secret env vars. Most CI systems have a mechanism for
doing this. For security reasons, do not check these values into version control.License informationCopyright (c) 2022, Voltron Data.Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
file except in compliance with the License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under
the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the specific language governing
permissions and limitations under the License.