Azure Monitor: Collect Logs and Metrics from On-Premises
By: Knoldus Inc.
Tags: Loganalytics, Logs, Metrics, Azure, DevOps

In this blog we are going to discuss how we can collect logs and metrics from the Azure resource and on-prem infrastructure to the azure monitor.

data flow between different components

Suppose we have an application running on on-premises so we need to collect the logs and metrics from it and send it to Azure log analytics for analysis of the logs and metrics and create dashboards for the same.

Azure Monitor can collect data directly from your physical or virtual Linux computers in your environment into a Log Analytics workspace for detailed analysis and correlation using the azure log analytics agents.

Installing the Log Analytics agent allows Azure Monitor to collect data from a data center.

Before analyzing and acting on collected data, you first need to install log analytics agents on all of the machines that you want monitor.

Log analytics agent sends

System logs,

Performance metrics

Custom logs from any location

Prerequisite:

Log analytics workspace

Log analytics workspace ID and Primary key

The agent communicates outbound to the Azure Monitor service over TCP port 443

Supported Linux operating systems:

Amazon Linux 2017.09 (x64)

CentOS Linux 6 (x86/x64) and 7 (x64)

Oracle Linux 6 and 7 (x86/x64)

Red Hat Enterprise Linux Server 6 (x86/x64) and 7 (x64)

Debian GNU/Linux 8 and 9 (x86/x64)

Ubuntu 14.04 LTS (x86/x64), 16.04 LTS (x86/x64), and 18.04 LTS (x64)

SUSE Linux Enterprise Server 12 (x64) and 15 (x64)

Network firewall requirements

The information below lists the proxy and firewall configuration information required for the Linux and Windows agents to communicate with Azure Monitor logs.

Agent Resource Ports Direction Bypass HTTPS inspection .ods.opinsights.azure.com Port 443 Outbound Yes .oms.opinsights.azure.com Port 443 Outbound Yes .blob.core.windows.net Port 443 Outbound Yes Firewall Rules

Install the log analytics agent for Linux

To configure the Linux computer to connect to a Log Analytics workspace, run the following command providing the workspace ID and primary key copied earlier.

Step-1: The following command downloads the agent validates its checksum and installs it.

Enter the loganalytics workspace id and primary key.

wget https://raw.githubusercontent.com/Microsoft/OMS-Agent-for-Linux/master/installer/scripts/onboard_agent.sh && sh onboard_agent.sh -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY>

Step-2: Restart the agent by running the following command:

Enter the loganalytics workspace id.

sudo /opt/microsoft/omsagent/bin/service_control restart [<workspace id>]

Collect event and performance data

Azure Monitor can collect events from the Linux Syslog and performance counters that you specify for longer-term analysis and reporting. It can also take action when it detects a particular condition.

Follow these steps to configure the collection of events from the Linux Syslog, and performance counters.

Go to Azure Portel Search log analytics Select your log analytics workspace Click on advance setting Select Data, and then select Syslog. You add Syslog by typing in the name of the log. Enter Syslog and then select the plus sign +. In the table, uncheck the severities Info, Notice and Debug. Select Save at the top of the page to save the configuration. Select Linux Performance Data to enable collection of performance counters on a Linux computer. When you first configure Linux Performance counters for a new Log Analytics workspace, you are given the option to quickly create several common counters. They are listed with a checkbox next to each. Select Apply below configuration to my machines and then select Add the selected performance counters. They are added and preset with a ten-second collection sample interval. Select Save at the top of the page to save the configuration.

View collected data

Now that you have enabled data collection, let’s run a simple log search example to see some data from the target computer.

Go to your log analytics workspace In the selected workspace, from the left-hand pane, select Logs. On the Logs query page, type Perf in the query editor and select Run.For example, the query in the following image returned 10,000 Performance records. Your results will be significantly less.

Log Analytics Agent data sources:

These are the data source of the agent:

Performance Metrics to Monitor the Instance:

These are the performance metrics which we are getting by the azure log analytics agent.

Object Name Counter Name Logical Disk % Free Inodes Logical Disk % Free Space Logical Disk % Used Inodes Logical Disk % Used Space Logical Disk Disk Read Bytes/sec Logical Disk Disk Reads/sec Logical Disk Disk Transfers/sec Logical Disk Disk Write Bytes/sec Logical Disk Disk Writes/sec Logical Disk Free Megabytes Logical Disk Logical Disk Bytes/sec Memory % Available Memory Memory % Available Swap Space Memory % Used Memory Memory % Used Swap Space Memory Available MBytes Memory Memory Available MBytes Swap Memory Page Reads/sec Memory Page Writes/sec Memory Pages/sec Memory Used MBytes Swap Space Memory Used Memory MBytes Network Total Bytes Transmitted Network Total Bytes Received Network Total Bytes Network Total Packets Transmitted Network Total Packets Received Network Total Rx Errors Network Total Tx Errors Network Total Collisions Physical Disk Avg. Disk sec/Read Physical Disk Avg. Disk sec/Transfer Physical Disk Avg. Disk sec/Write Physical Disk Physical Disk Bytes/sec Process Pct Privileged Time Process Pct User Time Process Used Memory kBytes Process Virtual Shared Memory Processor % DPC Time Processor % Idle Time Processor % Interrupt Time Processor % IO Wait Time Processor % Nice Time Processor % Privileged Time Processor % Processor Time Processor % User Time System Free Physical Memory System Free Space in Paging Files System Free Virtual Memory System Processes System Size Stored In Paging Files System Uptime System Users

Data Retention in azure log analytics workspace:

The retention period of the collected data stored in the database depends on the selected pricing plan. Collected data is available for 31 days by default but can be extended to 730 days. Data is stored encrypted at rest in Azure storage, to ensure data confidentiality, and the data is replicated within the local region using locally redundant storage (LRS). The last two weeks of data are also stored in SSD-based cache and this cache is encrypted.

Conclusion:

We can collect logs(System or application logs) and performance metrics from the log analytics agent and It is totally secure. But this agent can’t send the application level metrics like java application. For the application-level metrics, we should go for the azure application insight which is part of the azure monitor.

Thank you for sticking to the end. If you like this blog, please do show your appreciation by giving thumbs ups and share this blog and give me suggestions on how I can improve my future posts to suit your needs. Follow me to get updates on different technologies

References: