elasticsearch/logstash — pull request #1491 by hoelzro (2014-07-01 18:43:46+00:00)
Title: Improve performance for checksum filter

Hi there,

I have some performance improvements for the checksum filter. Here are the results from trying it on my machine:

Before:

$ time yes 'a random sampling of a chunk of fake data' | head -20000 | ./bin/logstash agent -e 'input { stdin {} } filter { checksum { keys => "message" } } output { stdout {} }' >/dev/null
yes 'a random sampling of a chunk of fake data' 0.00s user 0.00s system 0% cpu 1:38.38 total
head -20000 0.00s user 0.00s system 0% cpu 1:38.38 total
./bin/logstash agent -e > /dev/null 144.51s user 0.89s system 127% cpu 1:54.38 total

After:

$ time yes 'a random sampling of a chunk of fake data' | head -20000 | ./bin/logstash agent -e 'input { stdin {} } filter { checksum { keys => "message" } } output { stdout {} }' >/dev/null
yes 'a random sampling of a chunk of fake data' 0.00s user 0.00s system 0% cpu 17.883 total
head -20000 0.00s user 0.00s system 0% cpu 17.882 total
./bin/logstash agent -e > /dev/null 44.35s user 1.51s system 250% cpu 18.288 total