Repository: Yelp/elastalert

Issue #2701: CPU spike rule with metric_agg_script not working
Created: 2020-03-04T09:46:35Z
name: Metricbeat Average CPU Spike Rule
type: spike_aggregation
es_host: localhost
es_port: 9200
es_username: <username>
es_password: <password>
index: metricbeat-*
timeframe:
  hours: 15
buffer_time:
  minutes: 15
metric_agg_key: myScriptedMetric

metric_agg_script:
    script: (doc['system.cpu.user.pct'].value + doc['system.cpu.system.pct'].value) / doc['system.cpu.cores'].value

metric_agg_type: avg
query_key: agent.hostname
doc_type: _doc
threshold_cur: 0.001
min_doc_count: 1
spike_height: 1
spike_type: "up"

filter:
- term:
    metricset.name: cpu

alert:
- "debug"

This is the rule configuration used. Data is present which exceeds the threshold.
I'm testing above rule with elastalert-test-rule bu the alerts are not generated
Is there any way to check the output of the script?

Duplicate issue #2551: spike_aggregation does not match when it should
Created: 2019-11-06T19:31:06Z
After updating the list of types, per #2422, I thought I had spike_aggregation working. The thing is, when I test it, it has 20,000 hits, but 0 matches.
I based my rule on the example example_spike_single_metric_agg.yaml rule. I did modify the query_key and doc_type to match what metricbeat is putting into my Elasticsearch instance. I also configured the threshold to be ridiculously low, threshold_cur: 0.0011.
Here is my rule:
name: spike_aggregation CPU Spike Rule
type: spike_aggregation

#es_host: localhost
#es_port: 9200

index: metricbeat-*

timeframe:
  hours: 2 

buffer_time:
  hours: 1 

metric_agg_key: system.cpu.user.pct
metric_agg_type: avg
query_key: agent.hostname
doc_type: _doc 

#allow_buffer_time_overlap: true
#use_run_every_query_size: true

# (Required one of _cur or _ref, spike specific)
# The minimum value of the aggregation that will trigger the alert
# For example, if we're tracking the average for a metric whose average is 0.4 between 12:00 and 2:00
# and 0.95 between 2:00 and 4:00 with spike_height set to 2 and threshhold_cur set to 0.9:
# _ref is 0.4 and _cur is 0.95, and the alert WILL fire
# because 0.95 is greater than threshold_cur (0.9) and (_ref * spike_height (.4 * 2))
threshold_cur: 0.0011

# (Optional, min_doc_count)
# for rules using a per-term aggregation via query_key, the minimum number of events
# over the past buffer_time needed to update the spike tracker
min_doc_count: 3 

# (Required, spike specific)
# The spike aggregation rule matches when the current window contains spike_height times higher aggregated value
# than the reference window
spike_height: 2

# (Required, spike specific)
# The direction of the spike
# 'up' matches only spikes, 'down' matches only troughs
# 'both' matches both spikes and troughs
spike_type: "up"

filter:
- term:
    metricset.name: cpu

# (Required)
# The alert is use when a match is found
alert:
- "debug"
- email:
    from_addr: "no-reply-elastalert@host"
    email: "email address"

Here's my test run:
$ elastalert-test-rule --config config.yaml rules/metricbeat_spike_cpu.yaml 
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'spike_aggregation CPU Spike Rule', 'endtime': datetime.datetime(2019, 11, 6, 19, 16, 30, 354152, tzinfo=tzutc()), 'starttime': datetime.datetime(2019, 11, 6, 17, 16, 30, 354152, tzinfo=tzutc()), 'matches': 0, 'hits': 20000, '@timestamp': datetime.datetime(2019, 11, 6, 19, 17, 42, 468739, tzinfo=tzutc()), 'time_taken': 0.09138369560241699}

I've tried running elastalert on the command line to see if there was something about the test tool messing up. It still didn't work.
I'm guessing that #2337 is similar to my issue, but there are no responses there, so I'm not sure.
Anyway, am I missing something?

Closing comment (2020-04-15T08:04:26Z): Hi, @jerrac
Did you run your ElastAlert for a long time?
I think your problem is similar to #2701

I think your timeframe is too long. Therefore reference_count has not seems to be calclated yet. About timeframe in spike is discribed in https://elastalert.readthedocs.io/en/latest/ruletypes.html#spike.
Please shorten it and test again.

timeframe:
  minutes: 1
  # hours: 2 

buffer_time:
  hours: 1