Project: Vitess (App Definition and Development / Database)
File: vitess-mixin-quickstart.md

- - - - - - - - - Vitess is a monitoring mixin defined using data templating language. The grafana `jsonnet` library for creating dashboards is . While the package manager for easily reusing and extending the mixin is . If you want to see all available targets run `make help`. Choose an IDE of your preference. We use VSCode with `heptio.jsonnet` plugin which includes some nice features like on mouse over documentation and (ctrl+space) autocomplete. ```shell $ make prometheus_rules.yaml Done! ``` Once generated the Vitess Mixin Prometheus recording rules are available at `prometheus_rules.yaml`. Note: The Vitess Mixin only includes recording rules for Prometheus Vitess metrics. Other metric like Prometheus Node are not currently included in the rules. The vitess mixin supports DEV/PROD configurations (Currently it is possible to enable/disable dashboards. And set different alert configurations for DEV/PROD). Prod dashboards: ```shell $ ENV='prod' make dashboards_out dashboardsout/clusteroverview.json dashboardsout/vtgatehost_view.json dashboardsout/vtgateoverview.json dashboardsout/vttablethost_view.json Done! ``` Dev dashboards: ```shell $ ENV=dev make dashboards_out dashboardsout/clusteroverview.json dashboardsout/vtgatehost_view.json dashboardsout/vtgateoverview.json dashboardsout/vttablethost_view.json Done! ``` Voila!! the generated dashboards definitions in `.json` format are now available under `/dashboardsout`. (**Note**: dev and prod dashboards have the same names and they are stored in the same output folder. It is a good practice to run `make clean` after each build to make sure we are working with a clean `/dashboardsout` folder). ```shell $ make fmt Formatting dashboards/alertsmemoryevents.libsonnet: Done! ``` ```shell $ make lint Checking dashboards/alertsmemoryevents.libsonnet: STATUS: Done! TO BE IMPLEMENTED Done! ``` ```shell $ make clean rm -rf dashboardsout prometheusalerts.yaml prometheus_rules.yaml Done! ``` Dashboards are defined in `.libsonnet` format under the `/dashboards` folder. Note Files designed for import by convention end with `.libsonnet`. Update the Mixin config to include the new dashboard metadata `config.libsonnet`. The metadata needs to be added under `grafanaDashboardMetadata`. For example for `vtgate-overview dashboard` it looks like this: ```bash vtgateOverview: { environments: ['dev', 'prod'], uid: 'vitess-vtgate-overview', title: 'vtgate - overview %(dashboardNameSuffix)s' % $._config.grafanaDashboardMetadataDefault, description: 'General vtgate overview', dashboardTags: $._config.grafanaDashboardMetadataDefault.dashboardTags + ['overview', 'vtgate'], } ``` Note: It is possible to add/remove a dashboard from specific environments using the `environments` array field. Define the dashboard layout: Create a new `.libsonnet` file following Naming Convention (TODO Define a naming convention) in the `/dashboards/layouts` folder. The dashboard skeleton should look something like this: ```js local helpers = import '../resources/helpers.libsonnet'; local config = import '../../config.libsonnet'; { grafanaDashboards+:: { 'DASHBOARD_NAME.json': helpers.dashboard.getDashboard(config.config.grafanaDashboardMetadata.DASHBOARDNAME) .addTemplates( [ ] ).addLink(helpers.default.getDashboardLink(config._config.dashborardLinks)) .addPanels( [ ], ), }, } ``` Import the new dashboard in the

library. Add/Edit dashboard resources. Note: Some of the resources have been grouped in the following categories `vtgate`, `vttablet`, `vtorc`, `webapp`, `os` for convenience. The resources under this categories follow a helper/config pattern. The `._helper.libsonnet` libraries within `grafonnet` folder functions to retrieve resources such as `getPanel(panel_name_config)`, reading the configuration by name for the desired resource from the corresponding `._config.libsonnet` library . Note: To isolate PROD and DEV the configuration files have a section for each environment where we can define specific `notifications`(ex: pagerduty, vitess-alerts...) and `conditions` (alert thresholds). In this example the alert has a different notification config. For PROD it pages and DEV it doesn't: ```js alert: { name: 'vitess-mixin - # of tablets with MySQL replication lag > 300s', frequency: '1m', forDuration: '3m', noDataState: 'ok', executionErrorState: 'keep_state', message: ||| The number of tablets with MySQL replication lag > 300s is above the threshold. This usually indicate that replicas are not able to keep up with the write loads on the primary. Please take a look. |||, prod+: { notifications: [ { uid: 'alerts-vitess' }, { uid: 'pagerduty-vitess' }, ], conditions: [ alert_condition.new( evaluatorParams=[1], evaluatorType='gt', operatorType='and', queryRefId='A', queryTimeStart='1m', reducerType='max'), ], }, //override PROD notifications. dev+: self.prod { notifications: [ { uid: 'alerts-vitess' }, ], }, }, ``` Warning: Editing existing resources can change other dashboards behavior. Since basic resources like `panels`, `rows`, `texts`... can be shared across dashboards. This can be easily checked running `make all` as explained bellow. Same steps as Define a new Dashboard starting from step `.4`. Look for the dashboard you want to edit in the `dashboards/layouts` folder. Find the resource you want to update in the `dashboards/resources` folder. Create the resource in the corresponding location under `dashboards/resources` folder. Add the resource to the desired layout in `dashboards/layouts`. These tests attempt to assert truth to the following: Does this mixin generate valid dashboard JSON? Does this mixin generate valid recording rules YAML? Do the configured elements do what they are expected to do? This make target will format, lint, build all generated dashboards and recording rules using `origin/main` and diff with them your `local branch`. Printing a report to the `stdout`. ```shell $ pwd /manfontan/vitess-mixin $ ENV='prod' make all Done! EXIT STATUS: Your dashboards local version matches the origin/main version ``` The above execution shows the report for a `local branch` matching `origin/main`. Any changes will be reported as NOK along with the diff

This doesn't mean something is wrong it just points that there are changes in your local branch compared to `origin/main` which is expected. Review the diff report and once you are happy with your changes create a PR. These tests attempt to assert truth to the following: Are dashboard elements displayed as expected? The spec for each dashboard can be found at `/e2e/cypress/integration`. `docker-compose` is used to run Cypress and Grafana. There are two targets in to help run it. `make e2e`: runs tests headless and exits. `make e2e-dev`: opens the [the test runner](https://docs.cypress.io/guides/core-concepts/test-runner.html#Overview) and exposes Grafana to the host machine - http://localhost:3030. This requires an X11 server to work. [This post](https://www.cypress.io/blog/2019/05/02/run-cypress-with-a-single-docker-command/#Interactive-mode) describes how to set this up with . Note The dummy Grafana server is not connected to a Prometheus Backend for this reason dashboards will not display any data, templates will fail to load etc... If you don't have a Dev Prometheus server. Replacing prometheus datasources(`PrometheusVitess`, `PrometheusNode` ...) with an empty string in the generated JSON file will default to the dummy datasource displaying dummy data. This is useful when testing using interactive mode `make e2e-dev`. Note: This targets have been tested using docker for Mac. You may need to change the IPs and configurations for your specific setup. Before we run the local environment using compose it is necessary to generate the dashboards and recording rules using `ENV=dev make all` (Note: choose the environment you want to test). Once our dashboards are available simply run: `make e2e-compose-up`: spin up the cluster Changes to the dashboards are not dynamically loaded so you will need to bring down the cluster and initialized it again to load your changes. If you are done testing or the cluster gets in a bad state quit and clean up using: `make e2e-compose-down`: cleanup compose resources In order to generate some metrics we can use the following commands: ```shell mysql --port=15306 --host=127.0.0.1 < load_test.sql mysqlslap -c 5 --port=15306 --host=127.0.0.1 --iterations=1000 --create-schema=test_keyspace:80-@primary --query="SELECT * FROM messages;" mysqlslap -c 5 --port=15306 --host=127.0.0.1 --iterations=1000 --create-schema=test_keyspace:80-@replica --query="SELECT * FROM messages;" mysqlslap -c 5 --port=15306 --host=127.0.0.1 --iterations=1000 --create-schema=lookupkeyspace:-@primary --query="SELECT * FROM messagesmessage_lookup;" mysqlslap -c 5 --port=15306 --host=127.0.0.1 --iterations=1000 --create-schema=lookupkeyspace:-@replica --query="SELECT * FROM messagesmessage_lookup;" mysqlslap --port=15306 --host=127.0.0.1 --iterations=10000 --create-schema=test_keyspace:80-@primary --query="SELECT name FROM messages;" mysqlslap --port=15306 --host=127.0.0.1 --iterations=10000 --create-schema=lookupkeyspace:-@replica --query="SELECT name FROM messagesmessage_lookup;" ``` Once the cluster is up and running you should be able to access: grafana (default credentials > admin/admin) http://localhost:3030/ prometheus http://localhost:9000/ vitess control panel http://localhost:15000/