Chapter 9. Setting up graphical representation of PCP metrics Using a combination of pcp , grafana , pcp redis , pcp bpftrace , and pcp vector provides graphical representation of the live data or data collected by Performance Co-Pilot (PCP). 9.1. Setting up PCP with pcp-zeroconf This procedure describes how to set up PCP on a system with the pcp-zeroconf package. Once the pcp-zeroconf package is installed, the system records the default set of metrics into archived files. Procedure Install the pcp-zeroconf package: Verification Ensure that the pmlogger service is active, and starts archiving the metrics: Additional resources pmlogger man page on your system Monitoring performance with Performance Co-Pilot 9.2. Setting up a Grafana server Grafana generates graphs that are accessible from a browser. The Grafana server is a back-end server for the Grafana dashboard. It listens, by default, on all interfaces, and provides web services accessed through the web browser. The grafana-pcp plugin interacts with the pmproxy daemon in the backend. This procedure describes how to set up a Grafana server. Prerequisites PCP is configured. For more information, see Setting up PCP with pcp-zeroconf . Procedure Install the following packages: Restart and enable the following service: Open the server's firewall for network traffic to the Grafana service. Verification Ensure that the Grafana server is listening and responding to requests: Ensure that the grafana-pcp plugin is installed: Additional resources pmproxy(1) and grafana-server(1) man pages on your system 9.3. Accessing the Grafana web UI This procedure describes how to access the Grafana web interface. Using the Grafana web interface, you can: Add PCP Redis, PCP bpftrace, and PCP Vector data sources Create dashboard View an overview of any useful metrics Create alerts in PCP Redis. Prerequisites PCP is configured. For more information, see Setting up PCP with pcp-zeroconf . The Grafana server is configured. For more information, see Setting up a Grafana server . Procedure On the client system, open http:// <grafana_server_IP_address_or_hostname> :3000 in your browser. For the first login, enter admin in both the Email or username and Password field. Grafana prompts to set a New password to create a secured account. In the menu, navigate to Administration and then click Plugins . In the Plugins tab, type performance co-pilot in the Search Grafana plugins text box and then click Performance Co-Pilot (PCP) plugin. In the Plugins / Performance Co-Pilot pane, click Enable . Click the Grafana icon . The Grafana Home page is displayed. Figure 9.1. Home Dashboard Note The top right corner of the screen has a settings (gear) icon that controls the general Dashboard settings . In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information about adding data source, see: To add pcp redis data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alert in PCP Redis data source . To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard . To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist . Optional: From the menu, hover over the admin profile icon to update your Profile , view Notification history , Change password , or to Sign out . Additional resources grafana-cli(1) and grafana-server(1) man pages on your system 9.4. Configuring secure connections for Grafana You can establish secure connections between Grafana and Performance Co-Pilot (PCP) components. Establishing secure connections between these components helps prevent unauthorized parties from accessing or modifying the data being collected and monitored. Prerequisites PCP is installed. For more information, see Installing and enabling PCP . The Grafana server is configured. For more information, see Setting up a Grafana server . The private client key is stored in the /etc/grafana/grafana.key file. If you use a different path, modify the path in the corresponding steps of the procedure. For details about creating a private key and certificate signing request (CSR), as well as how to request a certificate from a certificate authority (CA), see your CA's documentation. The TLS client certificate is stored in the /etc/grafana/grafana.crt file. If you use a different path, modify the path in the corresponding steps of the procedure. Procedure As a root user, open the /etc/grafana/grafana.ini file and adjust the following options in the [server] section to reflect the following: Ensure grafana can access the certificates: Restart and enable the Grafana service to apply the configuration changes: Verification On the client system, open a browser and access the Grafana server machine on port 3000, using the https://192.0.2.0:3000 link. Replace 192.0.2.0 with your machine IP. Confirm the lock icon is displayed beside the address bar. Note If the protocol is set to http and an HTTPS connection is attempted, you will receive a ERR_SSL_PROTOCOL_ERROR error. If the protocol is set to https and an HTTP connection is attempted, the Grafana server responds with a "Client sent an HTTP request to an HTTPS server" message. 9.5. Configuring PCP Redis Use the PCP Redis data source to: View data archives Query time series using pmseries language Analyze data across multiple hosts Prerequisites PCP is configured. For more information, see Setting up PCP with pcp-zeroconf . The Grafana server is configured. For more information, see Setting up a Grafana server . Mail transfer agent, for example, sendmail or postfix is installed and configured. Procedure Install the redis package: Start and enable the following services: Restart the Grafana server: Verification Ensure that the pmproxy and redis are working: This command does not return any data if the redis package is not installed. Additional resources pmseries(1) man page on your system 9.6. Creating panels and alerts in PCP Redis data source After adding the PCP Redis data source, you can view the dashboard with an overview of useful metrics, add a query to visualize the load graph, and create alerts that help you to view the system issues after they occur. Prerequisites The PCP Redis is configured. For more information, see Configuring PCP Redis . The Grafana server is accessible. For more information, see Accessing the Grafana web UI . Procedure Log into the Grafana web UI. In the Grafana Home page, click Add your first data source . In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis . In the Data Sources / PCP Redis pane, perform the following: Add http://localhost:44322 in the URL field and then click Save & Test . Click Dashboards tab Import PCP Redis: Host Overview to see a dashboard with an overview of any useful metrics. Optional: In the drop-down menu to the clock icon , you can set the timeline of the displayed metrics either by setting the absolute time range or by selecting a predefined range. You can also use the zoom out icon to modify the displayed time range. Note The time frame displayed by default might not be aligned with the time frame covered by the archive files created by the pmlogger service. Figure 9.2. PCP Redis: Host Overview Add a new panel: From the plus sign drop-down menu, select New dashboard . From the Add drop-down menu, select Visualization . In the Query tab, select the pcp-redis-datasource as the Data source . In the text field below A , enter metric, for example, kernel.all.load to visualize the kernel load graph. Optional: From the Time series drop-down menu on the right, select another format of visualization, for example, Bar chart , Table , or Heatmap . Optional: Add Panel title and Description , and update other options. Click Save to apply changes and save the dashboard. Add Title . Click Apply to apply changes and go back to the dashboard. Figure 9.3. PCP Redis query panel Create an alert rule: In the PCP Redis query panel , click Alert and then click New alert rule . Enter alert rule name. Define query and alert condition. Set evaluation behavior. Optional: Add annotations. Add labels and notifications. Click Save rule an exit to apply changes in alert rules. Click Apply to apply changes and go back to the dashboard. Figure 9.4. Creating alerts in the PCP Redis panel To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts . 9.7. Adding notification channels for alerts By adding notification channels, you can receive an alert notification from Grafana whenever the alert rule conditions are met and the system needs further monitoring. You can receive these alerts after selecting any one type from the supported list of notifiers, which includes: Alertmanager Cisco Webex Teams DingDing Discord Email Google Chat Kafka REST Proxy LINE Microsoft Teams OpsGenie PagerDuty Pushover Sensu Go Slack Telegram Threema Gateway VictorOps WeCom Webhook Prerequisites The Grafana server is accessible. For more information, see Accessing the Grafana web UI . An alert rule is created. For more information, see Creating panels and alert in PCP Redis data source . Procedure Configure SMTP and add a valid sender's email address in the /etc/grafana/grafana.ini file: Restart the Grafana server. From the menu, select Alerting Contact points + Add contact point . Figure 9.5. Alerting in Grafana In the Create contact point details view, perform the following: Enter your name in the Name text box. Select the Integration type, for example, Email and enter the email address or multiple email addresses. Optional: Configure Optional Email settings and Notification settings . Click Save contact point . Select a notification channel in the alert rule: From the Alerting menu, select Notification policies . Click the three dots icon on the far right of Default policy and select Edit . Choose the Contact point you have just created and click Update default policy . Optional: Configure a nested policy in addition to the default policy. Optional: Configure Mute Timings . Additional resources Upstream Grafana documentation for alert notification policies 9.8. Setting up authentication between PCP components You can setup authentication using the scram-sha-256 authentication mechanism, which is supported by PCP through the Simple Authentication Security Layer (SASL) framework. Procedure Install the sasl framework for the scram-sha-256 authentication mechanism: Specify the supported authentication mechanism and the user database path in the pmcd.conf file: Create a new user: Replace metrics by your user name. Add the created user in the user database: To add the created user, you are required to enter the metrics account password. Set the permissions of the user database: Restart the pmcd service: Verification Verify the sasl configuration: Additional resources saslauthd(8) , pminfo(1) , and sha256 man pages on your system How can I setup authentication between PCP components, like PMDAs and pmcd in RHEL 8.2? (Red Hat Knowledgebase) 9.9. Installing PCP bpftrace Install the PCP bpftrace agent to introspect a system and to gather metrics from the kernel and user-space tracepoints. The bpftrace agent uses bpftrace scripts to gather the metrics. The bpftrace scripts use the enhanced Berkeley Packet Filter ( eBPF ). This procedure describes how to install a pcp bpftrace . Prerequisites PCP is configured. For more information, see Setting up PCP with pcp-zeroconf . The Grafana server is configured. For more information, see Setting up a Grafana server . The scram-sha-256 authentication mechanism is configured. For more information, see Setting up authentication between PCP components . Procedure Install the pcp-pmda-bpftrace package: Edit the bpftrace.conf file and add the user that you have created in Setting up authentication between PCP components : Replace metrics by your user name. Install bpftrace PMDA: The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard . Additional resources pmdabpftrace(1) and bpftrace man pages on your system 9.10. Viewing the PCP bpftrace System Analysis dashboard Using the PCP bpftrace data source, you can access the live data from sources which are not available as normal data from the pmlogger or archives In the PCP bpftrace data source, you can view the dashboard with an overview of useful metrics. Prerequisites The PCP bpftrace is installed. For more information, see Installing PCP bpftrace . The Grafana server is accessible. For more information, see Accessing the Grafana web UI . Procedure Log into the Grafana web UI. In the menu, navigate to Connections Data sources + Add new data source . In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace . In the Data Sources / pcp-bpftrace-datasource pane, perform the following: Add http://localhost:44322 in the URL field. Toggle the Basic Auth option and add the created user credentials in the User and Password field. Click Save & Test . Figure 9.6. Adding PCP bpftrace in the data source Click Dashboards tab Import PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics. Figure 9.7. PCP bpftrace: System Analysis 9.11. Installing PCP Vector Install a pcp vector data source to show live, on-host metrics from the real-time pmwebapi interfaces. This data source is intended for on-demand performance monitoring of an individual host and includes container support. Prerequisites PCP is configured. For more information, see Setting up PCP with pcp-zeroconf . The Grafana server is configured. For more information, see Setting up a Grafana server . Procedure Install the pcp-pmda-bcc package: Install the bcc PMDA: Additional resources pmdabcc(1) man page on your system 9.12. Viewing the PCP Vector Checklist The PCP Vector data source displays live metrics and uses the pcp metrics. It analyzes data for individual hosts. After adding the PCP Vector data source, you can view the dashboard with an overview of useful metrics and view the related troubleshooting or reference links in the checklist. Prerequisites The PCP Vector is installed. For more information, see Installing PCP Vector . The Grafana server is accessible. For more information, see Accessing the Grafana web UI . Procedure Log into the Grafana web UI. In the menu, navigate to Connections Data sources + Add new data source . In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector . In the Data Sources / pcp-vector-datasource pane, perform the following: Add http://localhost:44322 in the URL field and then click Save & Test . Click Dashboards tab Import PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics. Figure 9.8. PCP Vector: Host Overview In the menu, navigate to Apps Performance Co-Pilot PCP Vector Checklist . In the PCP checklist, click the question mark icon for help or the warning icon to view the related troubleshooting or reference links. Figure 9.9. Performance Co-Pilot / PCP Vector Checklist 9.13. Using heatmaps in Grafana You can use heatmaps in Grafana to view histograms of your data over time, identify trends and patterns in your data, and see how they change over time. Each column within a heatmap represents a single histogram with different colored cells representing the different densities of observation of a given value within that histogram. Important This specific workflow is for the heatmaps in Grafana version 10 and later on RHEL9. Prerequisites PCP Redis is configured. For more information see Configuring PCP Redis . The Grafana server is accessible. For more information see Accessing the Grafana Web UI . The PCP Redis data source is configured. For more information see Creating panels and alerts in PCP Redis data source . Procedure In the menu, select Dashboards New Dashboard + Add visualization . In the Select data source pane, select pcp-redis-datasource . In the Query tab, in the text field below A , enter a metric, for example, kernel.all.load to visualize the kernel load graph. From the Time series drop-down menu on the right, select Heatmap . Optional: In the Panel Options dropdown menu, add a Panel Title and Description . In the Heatmap dropdown menu, under the Calculate from data setting, click Yes . Heatmap Optional: In the Colors dropdown menu, change the Scheme from the default Orange and select the number of steps (color shades). Optional: In the Tooltip dropdown menu, click the toggle to display a cell's position within its specific histogram when hovering your cursor over a cell in the heatmap. For example: Show histogram (Y Axis) cell display 9.14. Troubleshooting Grafana issues It is sometimes neccesary to troubleshoot Grafana issues, such as, Grafana does not display any data, the dashboard is black, or similar issues. Procedure Verify that the pmlogger service is up and running by executing the following command: Verify if files were created or modified to the disk by executing the following command: Verify that the pmproxy service is running by executing the following command: Verify that pmproxy is running, time series support is enabled, and a connection to Redis is established by viewing the /var/log/pcp/pmproxy/pmproxy.log file and ensure that it contains the following text: Here, 1716 is the PID of pmproxy, which will be different for every invocation of pmproxy . Verify if the Redis database contains any keys by executing the following command: Verify if any PCP metrics are in the Redis database and pmproxy is able to access them by executing the following commands: Verify if there are any errors in the Grafana logs by executing the following command:

Commands:
dnf install pcp-zeroconf
pcp | grep pmlogger pmlogger: primary logger: /var/log/pcp/pmlogger/ localhost.localdomain/20200401.00.12
dnf install grafana grafana-pcp
systemctl restart grafana-server systemctl enable grafana-server
firewall-cmd --permanent --add-service=grafana success firewall-cmd --reload success
ss -ntlp | grep 3000 LISTEN 0 128 *:3000 *:* users:(("grafana-server",pid=19522,fd=7))
grafana-cli plugins ls | grep performancecopilot-pcp-app performancecopilot-pcp-app @ 5.1.1
protocol = https cert_key = /etc/grafana/grafana.key cert_file = /etc/grafana/grafana.crt
su grafana -s /bin/bash -c 'ls -1 /etc/grafana/grafana.crt /etc/grafana/grafana.key' /etc/grafana/grafana.crt /etc/grafana/grafana.key
systemctl restart grafana-server systemctl enable grafana-server
dnf install redis
systemctl start pmproxy redis systemctl enable pmproxy redis
systemctl restart grafana-server
pmseries disk.dev.read 2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
[smtp] enabled = true from_address = <sender_email_address>
systemctl restart grafana-server.service
dnf install cyrus-sasl-scram cyrus-sasl-lib
vi /etc/sasl2/pmcd.conf mech_list: scram-sha-256 sasldb_path: /etc/pcp/passwd.db
useradd -r metrics
saslpasswd2 -a pmcd metrics Password: Again (for verification):
chown root:pcp /etc/pcp/passwd.db chmod 640 /etc/pcp/passwd.db
systemctl restart pmcd
pminfo -f -h "pcp://127.0.0.1?username= metrics " disk.dev.read Password: disk.dev.read inst [0 or "sda"] value 19540
dnf install pcp-pmda-bpftrace
vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf [dynamic_scripts] enabled = true auth_enabled = true allowed_users = root, metrics
cd /var/lib/pcp/pmdas/bpftrace/ ./Install Updating the Performance Metrics Name Space (PMNS) Terminate PMDA if already installed Updating the PMCD control file, and notifying PMCD Check bpftrace metrics have appeared ... 7 metrics and 6 values
dnf install pcp-pmda-bcc
cd /var/lib/pcp/pmdas/bcc ./Install [Wed Apr 1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state. [Wed Apr 1 00:27:48] pmdabcc(22341) Info: Enabled modules: [Wed Apr 1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork', [...] Updating the Performance Metrics Name Space (PMNS) Terminate PMDA if already installed Updating the PMCD control file, and notifying PMCD Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values
systemctl status pmlogger
ls /var/log/pcp/pmlogger/USD(hostname)/ -rlt total 4024 -rw-r--r--. 1 pcp pcp 45996 Oct 13 2019 20191013.20.07.meta.xz -rw-r--r--. 1 pcp pcp 412 Oct 13 2019 20191013.20.07.index -rw-r--r--. 1 pcp pcp 32188 Oct 13 2019 20191013.20.07.0.xz -rw-r--r--. 1 pcp pcp 44756 Oct 13 2019 20191013.20.30-00.meta.xz [..]
systemctl status pmproxy
pmproxy(1716) Info: Redis slots, command keys, schema version setup
redis-cli dbsize (integer) 34837
pmseries disk.dev.read 2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pmseries "disk.dev.read[count:10]" 2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df [Mon Jul 26 12:21:10.085468000 2021] 117971 70e83e88d4e1857a3a31605c6d1333755f2dd17c [Mon Jul 26 12:21:00.087401000 2021] 117758 70e83e88d4e1857a3a31605c6d1333755f2dd17c [Mon Jul 26 12:20:50.085738000 2021] 116688 70e83e88d4e1857a3a31605c6d1333755f2dd17c [...]
redis-cli --scan --pattern "*USD(pmseries 'disk.dev.read')" pcp:metric.name:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:values:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:desc:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:labelvalue:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:instances:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df pcp:labelflags:series:2eb3e58d8f1e231361fb15cf1aa26fe534b4d9df
journalctl -e -u grafana-server -- Logs begin at Mon 2021-07-26 11:55:10 IST, end at Mon 2021-07-26 12:30:15 IST. -- Jul 26 11:55:17 localhost.localdomain systemd[1]: Starting Grafana instance Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Starting Grafana" logger=server version=7.3.6 c> Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/usr/s> Jul 26 11:55:17 localhost.localdomain grafana-server[1171]: t=2021-07-26T11:55:17+0530 lvl=info msg="Config loaded from" logger=settings file=/etc/g> [...]