Project: ShardingSphere (App Definition and Development / Database)
File: 2022_03_25_A_Deep_Dive_Into_Observability_Apache_ShardingSphere_Agent.en.md

+++ title = "A Deep Dive Into Observability Apache ShardingSphere Agent" weight = 45 chapter = true +++ follows Database Plus our communitys guiding development concept for creating a complete data service ecosystem that allows you to transform any database into a distributed database system, and easily enhance it with sharding, elastic scaling, data encryption features & more. Users often need to monitor Apache ShardingSpheres performance in a real application scenario to find specific problems. Its a given that Application Performance Monitoring (APM) can monitor and diagnose system performance by collecting, storing, and analyzing observable data. It also integrates functions such as performance metrics, tracing analysis, application topology mapping, etc. It retrieves the observable data from system operations by utilizing Tracing, Metrics and Logging. Apache ShardingSphere provides the Observability function for users. Apache ShardingSphere allows developers to configure logging output. The main objective of the current Observability feature is to provide necessary metrics and tracing data for users. We chose the common Agent method to perfectly implement Observability, with ByteBuddy (a code generation library that allows developers to modify Java classes (arbitrary included) during application runtime without using a compiler and Java Agent. We adopt the plugin-oriented design to better support different frameworks or systems with Metrics and Tracing. Accordingly, users are allowed to customize the feature or develop more business-specific components by themselves. Currently, the Agent module of Apache ShardingSphere supports Prometheus, Zipkin, Jaeger, SkyWalking, and OpenTelemetry. In the following instance, we deploy Prometheus and Zipkin observable data plugins on Apache ShardingSphere-Proxy to showcase how to use the Apache ShardingSphere Agent module. Step 1: Install the Required Software `prometheus-2.32.1.linux-amd64.tar.gz `(https://prometheus.io/download) `zipkin-server-2.23.9-exec.jar` (https://zipkin.io/pages/quickstart.html) `apache-shardingsphere-5.1.0-SNAPSHOT-shardingsphere-proxy-bin.tar.gz` `apache-shardingsphere-5.1.0-SNAPSHOT-shardingsphere-agent-bin.tar.gz` MySQL 5.7.34 Step 2: Deploy Port Prometheus Server9090 Zipkin Server9411 Apache ShardingSphere-Proxy3307 Apache ShardingSphere AgentPrometheus Plugin9000 Prometheus First, add monitoring objects to Prometheus. In this case, we need to add the Apache ShardingSphere Agent port address 9000 to the Prometheus configuration file `prometheus.yml`. ``` vi prometheus.yml ``` Add the following code under `static_configs` in the file: ``` targets: ["localhost:9000"] ``` Then, initiate: ``` ./prometheus & ``` Zipkin Zipkin is easier to use. Initiate it by input the following command in the Zipkin Server directory: ``` java -jar Zipkin-server-2.23.9-exec.jar & ``` Apache ShardingSphere To deploy Apache ShardingSphere-Proxy and Agent, please refer to the official related . Assuming Proxy and Agent are both in the `/tmp` directory, below are the specific Agent deployment steps: Modify Configurations Modify the `agent.yaml` configuration

Then, initiate Prometheus and Zipkin plugins and change the Prometheus port data to 9000 in line with the above-mentioned port settings: ``` applicationName: shardingsphere-agent ignoredPluginNames: Jaeger OpenTracing OpenTelemetry Logging plugins: Prometheus: host: "localhost" port: 9000 props: JVMINFORMATIONCOLLECTOR_ENABLED : "true" Jaeger: host: "localhost" port: 5775 props: SERVICE_NAME: "shardingsphere-agent" JAEGERSAMPLERTYPE: "const" JAEGERSAMPLERPARAM: "1" Zipkin: host: "localhost" port: 9411 props: SERVICE_NAME: "shardingsphere-agent" URL_VERSION: "/api/v2/spans" SAMPLER_TYPE: "const" SAMPLER_PARAM: "1" OpenTracing: props: OPENTRACINGTRACERCLASS_NAME: "org.apache.skywalking.apm.toolkit.opentracing.SkywalkingTracer" OpenTelemetry: props: otel.resource.attributes: "service.name=shardingsphere-agent" otel.traces.exporter: "zipkin" Logging: props: LEVEL: "INFO" ``` Add to Start Command Modify the file `/tmp/apache-shardingsphere-5.1.0-shardingsphere-proxy-bin/bin/start.sh` and add the absolute path of Agents `shardingsphere-agent.jar` to the startup script. Before ``` nohup java ${JAVAOPTS} ${JAVAMEM_OPTS} \ -classpath ${CLASSPATH}**** ${MAINCLASS} >> ${STDOUT_FILE} 2>&1 & ``` After ``` nohup java ${JAVAOPTS} ${JAVAMEM_OPTS} \ -javaagent:/tmp/apache-shardingsphere-5.1.0-shardingsphere-agent-bin/shardingsphere-agent.jar \ -classpath ${CLASSPATH} ${MAINCLASS} >> ${STDOUT_FILE} 2>&1 & ``` Initiate Now were ready to initiate them under the Proxy directory: ``` bin/start.sh ``` Step 3: Test Accesses Metrics and Tracing Data We use `config-sharding.yaml`, the default sharding configuration scenario provided by Apache ShardingSphere-Proxy, to test access and display data. Connect to the initiated ShardingSphere-Proxy with MySQL command lines. Examine the data results in Prometheus Server and Zipkin Server Query `proxy_state` and get the data results through Prometheus Web. View Zipkin Web tracing information after connecting to the MySQL client: Query data through MySQL command line: Examine the data results of Prometheus Server and Zipkin Server Query `parsedselectsql_total `data results through Prometheus Web. Query tracing information via Zipkin Web: Through a careful search of Span, we can check the tracing status of SQL statement `select * from t_order`. We cannot find topology mappings when we check dependencies through Zipkin Web. So, we need to configure them: Download files First, download the following Zipkin dependencies, and copy them to the lib directory of Proxy. Modify Configurations Configure the data source configuration file `config-sharding.yaml`, which is in the `conf` directory of the Proxy, and add the following configuration to the URL corresponding to the lower data source of the config-sharding.YAML `dataSources` node: MySQL 5.x`statementInterceptors=brave.mysql.TracingStatementInterceptor` Or MySQL 8.x`queryInterceptors=brave.mysql8.TracingQueryInterceptor` Restart ShardingSphere-Proxy After performing the same access test as before, we can view dependencies through Zipkin Web and see the following topology mappings: The Observability plugin also enables users to set differnt sampling rate configured to suit different scenarios. Zipkin plugins support various sampling rate type configurations including const, counting, rate limiting, and boundary. For scenarios with a high volume of requests, we suggest you to choose the boundary type and configure it with the appropriate sampling rate to reduce the collect volume of tracing data. ``` Zipkin: host: "localhost" port: 9411 props: SERVICE_NAME: "shardingsphere-agent" URL_VERSION: "/api/v2/spans" SAMPLER_TYPE: "boundary" SAMPLER_PARAM: "0.001" ``` With the Observability plugin compatible with many common monitoring frameworks and systems by default, users can easily monitor and manage Apache ShardingSphere. In the future, we will continue to enhance the monitoring capability. Pingchuan JIANG SphereEx Senior Middleware Engineer, Apache Tomcat & Apache ShardingSphere Contributor. An open source technology enthusiast, Pingchuan devotes himself to developing SphereEx-Console and SphereEx-Boot. Maolin JIANG SphereEx Senior Middleware Engineer & Apache ShardingSphere Contributor.