How to use Metricbeat agent to Collect Windows Metrics to AWS Elasticsearch (ES).
By: Nuwan Premaratne
Tags: Elasticsearch, Windows, AWS, Metricbeat, Monitoring

How to use Metricbeat agent to Collect Windows Metrics to AWS Elasticsearch (ES).

Photo by Fotis Fotopoulos on Unsplash

Metricbeat is an Agent that can be used to Collect Metrics from Compute Sources such as VMs (Windows/Linux), Containers, Kubernetes Clusters. Metricbeat has two versions which are under two Licenses. One is Metricbeat which is under Elastic Co. License and the Other is Metricbeat-OSS which is under Apache 2.0 License. If you need to use Metricbeat with AWS ES, you have to download the Metricbeat OSS Version since AWS ES uses the Open Source “Open Distro for Elasticsearch”.

OSS version will not be shown when searched from the internet. Search for Metricbeat Download and scroll down in the download page as shown below.

Metricbeat Agent Download

As underlined in the above image you will be able to find the link to OSS Version from there. Click on that and it will redirect you to the Metricbeat OSS Page. Download the 64-bit or 32-bit zip file depending on your windows instance (If these are the latest Windows OSes such as Windows 10 or Windows Server 2016/2019 this will be 64 bit).

Once downloaded copy the downloaded zip file to the required location and extract the zip file. Copy the extracted directory to your desired location such as into Program File in C Drive. Follow the official Metricbeat installation guide given here. Complete only steps 1 and 3.

Note if you get an error when executing the script as Scripts are not allowed to run, change the script execution policy and run the script. Then change it back to what it was. You can do this in the following way;

Changing Windows PowerShell Execution policy

The script will install the Metricbeat windows service. Next, we need to enable the Windows Module which we use to gather windows metrics. How to enable the module also defined in the official guide which I have shared above.

metricbeat.exe modules enable windows

Configuring Windows Module to collect metrics.

After enabling the Windows module let’s configure it to collect Current Windows CPU usage, available Memory in MB and the current Disk write speed. The Windows module YAML file is located in the modules.d directory inside the Metricbeat install location. The following image contains the YAML code written to gather the above two mentioned metrics using the Metricbeat windows module.

Windows Module Configurations

Metricbeat Windows module actually uses the Windows Perfmon API to pull out the metrics. Therefore, you can use the Windows performance counters to find out about available counters and their counter path information. For example, the following are some of the Windows CPU-related performance counters.

Some of the available CPU related counters

(Get-Counter -ListSet “Processor Information”).Paths

As you can see from the above output, I have used the Processor Time Counter to get the required information in my use case. You can view the current counter values by using the following command;

Get-Counter -Counter “\Processor Information(*)\% Processor Time”

Windows Processor Time Counter

You can also use the Windows module to send the windows service information to your ES domain, but in this article, we are not going to go through that. Next, we need to configure the Metricbeat YAML file with is used by the Metricbeat agent to connect to our AWS ES domain and its Kibana instance in order to push the collected metrics. For this, we need a user Metricbeat user which we can use to configure in the Metricbeat YAML file. Since we have to hard code the credentials into the YAML file, please create the Metricbeat user with the lowest permissions required.

Configuring AWS ES to receive metrics from Metricbeat.

I am going to first create a Kibana Role named “log_input” with required permissions as shown below.

Custom Kibana role to grant access to new user

Cluster Permissions

- create_index

- write

- cluster:monitor/main

- indices:admin/template/put Index Permissions

- create-index

- write Tenant Permissions

- Read and Write

Note that you need to select the indexes and tenants that you need to apply these permissions. In this scenario since we have to inject the data, I have given it all using the wildcard mark. You can even fine-tune the permission to a lower level than this based on your requirements using the “Open Distro” documentation mentioned here.

Next, let's create an internal user with the created role above. You can use any name which is meaningful. Navigate to the Internal User section under security and create a new user. Then under the Roles section map the newly created user to the “log_input” role. Now let us navigate back to the Windows EC2 to continue configuring the Metricbeat agent.

Configuring Metricbeat YAML file.

There are two sections mainly that we need to configure in the Metricbeat YAML file. First, let’s configure the Kibana section. We need to provide our Kibana endpoint URL and mention the used protocol as “https”. Note that in the URL we need to specifically provide the port we are using although it is obvious to be 443.

Configuring Kibana Section in the YAML file

Next, we need to configure the Elasticsearch section. In here also we need to provide the ES endpoint URL with the 443 port and the protocol. Additionally, we need to configure the new user credentials we created in the ES to authenticate the data sent by the Metricbeat agent. In this section, you can also provide your own index name also. If you do not provide an index name, the Metricbeat will use its default index name to send data which is “metricbeat-<version>-yyyy.MM.dd”.

Configuring ES Section in the YAML file

Additionally, there are four lines that you need to add to the file. Two of these lines are due to the ES version we are using (which is Open Distro flavor of AWS) and the other two lines are due to the custom index name which we have provided as shown above. If you are not using the custom index name, you can opt-out of the last two lines shown below.

Additional Configuration needs to be added to the YAML file

Make sure that you do not have any indentations in front of the lines. Save and close the YAML configuration file. The next steps depend on the Access policy of your ES Domain. If you have used open access to your ES Cluster regardless of whether is a VPC enabled ES Cluster or a Public one you can start the Metricbeat Windows service by following step 5 of the official guide.

PS>\ start-service metricbeat

But if you are using a custom JSON-defined access policy, you will have to enable access to your EC2 to access Kibana endpoints through the access policy. There are two levels of authentication happening at the ES Cluster, one is inside the Kibana interface and other is the at the endpoint of the ES Cluster via the access policy. Since Metricbeat does not support EC2 IAM role authentication we had to hard code internal user credentials to the Metricbeat YAML file. This requests from the Metricbeat agent comes to the ES Cluster endpoint as anonymous user requests not signed by any user ID. Either you can sign these requests manually which is not covered in this article since it will be a significantly larger process involving manual work or you will have to provide ESHttp actions allowed for the anonymous users at the endpoint since in fact they are properly authenticated again at the inside of the cluster using roles and users we created.

There are 3 specific ESHttp actions needed by the Metricbeat agent itself to load the metricv data to the ES cluster as ESHttpGet, ESHttpPut, and ESHttpPost. Other than that, if you are to load the Kibana web interface to manage the cluster you will need remaining ESHttp action also. Therefore, you will need to add the following policy section inside the statement section to your access policy of the ES cluster;

{

"Effect": "Allow",

"Action": "es:ESHttp*",

"Principal":{"AWS":"*"},

"Resource": "<Your Domain endpoint URL (without https://)>/*" }

You can view all 6 ESHttp actions available from the IAM console.

Now you might be wondering why we have not completed step 4 in the official document. We do not need step 4 in order to send our metrics to AWS ES. What step 4 will do is to set up your ES Cluster with Metricbeat default templates, Visualizations, and Dashboards which you can directly use with the index. Unfortunately, the AWS Managed ES does not allow you to set this up as directly in your own Managed ES Cluster. For this, you will have to do some additional steps which we will not go through in this article to keep this article to the point.

You can confirm that your data is being successfully received by the ES cluster by navigating to the Index Management → Indices section. You will be able to see the index created with the name which you have provided in the Metricbeat YAML file.

Confirming that the index is received by the ES Cluster

As the last steps, you can capture this using an index pattern under the Stack Management section and then create your own Visualizations and Dashboards per your requirements. When creating the index pattern, you will have to provide the exact index name which you have given in the YAML file without the dynamic part (date section).

Created index pattern

Then you can use this newly created index pattern to create the charts, graphs, or tables according to your preferences.

Creating custom visualizations

I hope this article has cleared out the issue you had when you begin to read or you have learned a new thing in your life with cloud in this modern world. Appreciate it if you can let me know any issues via responses or if this article has helped you via the clap button.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬