Monitoring batch workers using CloudWatch Metrics

# CloudWatch Metrics

In our project, we use the standard CloudWatch Metrics to monitor performance and issues in production. For example, if CPU utilization on a server exceeds 80% for more than 5 minutes, an alarm triggers and sends us an email.

We've also built a few simple CloudWatch Dashboards for monitoring different parts of our system. It's a great way to get a quick overview.

## Missing metrics: memory, swap, disk utilization

In addition to the standard metrics, we've added system level metrics like memory, swap and disk utilization. AWS provides a Perl script for that which can be run as a cron job:
http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/mon-scripts.html

![Screen Shot 2016-12-19 at 12.01.54.jpg](https://qiita-image-store.s3.amazonaws.com/0/148758/76e4da9e-2009-a783-02ce-d6a54665af01.jpeg)

## Custom metrics

We are using batch workers (ActiveJob via <a href="https://github.com/collectiveidea/delayed_job">delayed_job</a>) to handle long running requests, such as importing bulk data from files, or rendering large PDF files. These functions are used all the time by customers, so we need a health check for our worker processes.

To do this, we implemented our own simple custom AWS metric: "delayed-job::WorkerProcesses".

```ruby
# This works on OSX and AWS Linux servers
pids = `pgrep -f delayed_job`.split
Rails.logger.info("Worker health check. #{pids.size} worker processes found (#{pids.join(',')})")

log_cloudwatch_metrics(
  namespace: 'delayed-job',
  aws_metrics: [AWSMetrics::Metric.new(name: 'WorkerProcesses', value: pids.size)]
)
```

Now we can see the metric in Cloudwatch like any other metric, and add alarms etc.
![Screen Shot 2016-12-19 at 12.23.57.png](https://qiita-image-store.s3.amazonaws.com/0/148758/70448ddb-39d9-924f-672f-aea1b844636e.png)

The check is run from a periodic batch job using <a href="https://github.com/amitree/delayed_job_recurring">delayed_job_recurring</a>. That way, we know that the worker processes are running, and jobs are being processed successfully.

## CloudWatch Metrics wrapper

Sending AWS Metrics is quite simple using the AWS SDK for Ruby, but the request format is a bit complex, with many fields to fill out. To make it simpler, we implemented a small wrapper with sensible defaults:

```ruby
module AWSMetrics

  def log_cloudwatch_metrics(namespace:, aws_metrics:)
    cloudwatch_client.put_metric_data(
      namespace: "ProjectX/#{namespace}",
      metric_data: aws_metrics.map(&:to_h)
    )
  end

  def cloudwatch_client
    Aws::CloudWatch::Client.new(region: ENV['AWS_REGION'].presence || 'ap-northeast-1')
  end

  class Metric
    VALID_UNITS = [
      'Seconds', 'Microseconds', 'Milliseconds',
      'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes',
      'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits',
      'Percent', 'Count',
      'Bytes/Second', 'Kilobytes/Second', 'Megabytes/Second', 'Gigabytes/Second', 'Terabytes/Second',
      'Bits/Second', 'Kilobits/Second', 'Megabits/Second', 'Gigabits/Second', 'Terabits/Second',
      'Count/Second',
      'None'
    ]

    attr_reader :name, :value, :aws_unit_string, :hostname, :timestamp

    def initialize(name:, value:, aws_unit_string: 'Count', hostname: Socket.gethostname, timestamp: Time.now)
      fail 'Invalid Unit' unless VALID_UNITS.include?(aws_unit_string)
      @name = name
      @value = value
      @aws_unit_string = aws_unit_string
      @hostname = hostname
      @timestamp = timestamp
    end

    def to_h
      {
        metric_name: @name,
        dimensions: [
          {
            name: 'hostname',
            value: @hostname
          }
        ],
        timestamp: @timestamp,
        value: @value,
        unit: @aws_unit_string
      }
    end
  end

end
```

Thanks for reading!