Logstash - Plugins

Logstash offers various plugins for all three stages of its pipeline (Input, Filter and Output). These plugins help the user to capture logs from various sources like Web Servers, Databases, Over Network Protocols, etc.
After capturing, Logstash can parse and transform the data into meaningful information as required by the user. Lastly, Logstash can send or store that meaningful information to various destination sources like Elasticsearch, AWS Cloudwatch, etc.
Input plugins in Logstash helps the user to extract and receive logs from various sources. The syntax for using the input plugin is as follows −
Input {
   Plugin name {
      Setting 1......
      Setting 2........
   }
}

You can download input plugin by using the following command −
>Logstash-plugin install Logstash-input-<plugin name>

The Logstash-plugin utility is present in the bin folder of the Logstash installation directory. The following table has a list of the input plugins offered by Logstash.
beats
To get the logging data or events from elastic beats framework.
cloudwatch
To extract events from CloudWatch, an API offer by Amazon Web Services. 
couchdb_changes
Events from _chages URI of couchdb shipped using this plugin.
drupal_dblog
To extract drupal’s watchdog logging data with enabled DBLog.
Elasticsearch
To retrieve the results of queries performed in Elasticsearch cluster.
eventlog
To get the events from windows event log.
exec
To get shell command output as an input in Logstash.
file
To get the events from an input file. This is useful, when the Logstash is locally installed with the input source and have access to input source logs. 
generator
It is used for testing purposes, which creates random events.
github
Captures events from GitHub webhook.
graphite
To get metrics data from graphite monitoring tool.
heartbeat
It is also used for testing and it produces heartbeat like events
http
To collect log events over two network protocols and those are http and https.
http_poller
It is used to decode the HTTP API output to an event.
jdbc
It converts the JDBC transactions to an event in Logstash.
jmx
To extract the metrics from remote java applications using JMX.
log4j
Capture events from socketAppender object of Log4j over TCP socket.
rss
To the output of command line tools as an input event in Logstash.
tcp
Captures events over TCP socket.
twitter
Collect events from twitter streaming API.
unix
Collect events over UNIX socket.
websocket
Capture events over websocket protocol.
xmpp
Reads events over Jabber/xmpp protocols.
All the plugins have their specific settings, which helps to specify the important fields like Port, Path, etc., in a plugin. We will discuss the settings of some of the input plugins.
This input plugin is used to extract events directly from log or text files present in the input source. It works similar to the tail command in UNIX and save the last read cursor and read only the new appended data from the input file, but it can be changed by using star_position setting. Following are the settings of this input plugin.
This particular plugin is used to read the search queries results in an Elasticsearch cluster. The following has the settings used in this plugin −
This input plugin reads data from win32 API of windows servers. Followings are the settings of this plugin −
This input plugin is used to collect the feed of twitter from its Streaming API. The following table describes the settings of this plugin.
It specifies the user ids separated by commas and LogStash checks these users’ status in Twitter.
For more info, visit
https://dev.twitter.com
TCP is used to get the events over the TCP socket; it can read from the user connections or server, which is specified in mode setting. The following table describes the settings of this plugin −
Logstash supports various output sources and in different technologies like Database, File, Email, Standard Output, etc.
The syntax for using the output plugin is as follows −
output {
   Plugin name {
      Setting 1......
      Setting 2........
   }
}

You can download the output plugin by using the following command −
>logstash-plugin install logstash-output-<plugin name>

The Logstash-plugin utility is present in the bin folder of Logstash installation directory. The following table describes the output plugins offered by Logstash.
CloudWatch
This plugin is used to send aggregated metric data to CloudWatch of amazon web services.
csv
It is used to write the output events in a comma-separated manner.
Elasticsearch
It is used to store the output logs in Elasticsearch index.
email
It is used to send a notification email, when the output is generated. User can add information about the output in email.
exec
It is used to a run a command, which match the output event.
ganglia
It writhe the metrics to gmond of Gangila.
gelf
It is used to produce output for Graylog2 in GELF format.
google_bigquery
It outputs the events to Google BigQuery.
google_cloud_storage
It store the output events to Google Cloud Storage.
graphite
It is used to store the output events to Graphite.
graphtastic
It is used to write the output metrics on Windows.
hipchat
It is used to store the output log events to HipChat.
http
It is used to send the output log events to http or https endpoints.
influxdb
It is used to store the output event in InfluxDB.
irc
It is used to write the output events to irc.
mongodb
It stores the output data in MongoDB.
nagios
It is used to notify Nagios with the passive check results.
nagios_nsca
It is used to notify Nagios with the passive check results over NSCA protocol.
opentsdb
It store the Logstash output events to OpenTSDB.
pipe
It streams the output events to the standard input of another program.
rackspace
It is used to send the output log events to Queue service of Rackspace Cloud.
redis
It uses rpush command to send the output logging data to Redis queue.
riak
It is used to store the output events to the Riak distributed key/value pair.
s3
It store the output logging data to Amazon Simple Storage Service.
sns
It is used to send the output events to Amazon’s Simple Notification Service. 
solr_http
It indexes and stores the output logging data in Solr.
sps
It is used to ship the events to Simple Queue Service of AWS.
statsd
It is used to ship the metrics data to statsd network daemon.
stdout
It is used to show the output events on standard output of CLI like command prompt.
syslog
It is used to ships the output events to syslog server.
tcp
It is used to send the output events to TCP socket.
udp
It is used to push the output events over UDP.
websocket
It is used to push the output events over WebSocket protocol.
xmpp
It is used to push the output events over XMPP protocol.
All the plugins have their specific settings, which helps to specify the important fields like Port, Path, etc., in a plugin. We will discuss the settings of some of the output plugins.
Elasticsearch output plugin enables Logstash to store the output in the specific clusters of Elasticsearch engine. This is one of the famous choices of users because it comes in the package of ELK Stack and therefore, provides end-to-end solutions for Devops. The following table describes the settings of this output plugin.
The email output plugin is used to notify the user, when Logstash generates output. The following table describes the settings for this plugin.
This setting is used to send the output events over http to the destination. This plugin has following settings −
The stdout output plugin is used to write the output events on the standard output of the command line interface. It is command prompt in windows and terminal in UNIX. This plugin has the following settings −
It is a network daemon used to send the matrices data over UDP to the destination backend services. It is command prompt in windows and terminal in UNIX. This plugin has following settings −
Logstash supports various filter plugins to parse and transform input logs to a more structured and easy to query format.
The syntax for using the filter plugin is as follows −
filter {
   Plugin name {
      Setting 1......
      Setting 2........
   }
}

You can download the filter plugin by using the following command −
>logstash-plugin install logstash-filter-<plugin name>

The Logstash-plugin utility is present in the bin folder of Logstash installation directory. The following table describes the output plugins offered by Logstash.
aggregate
This plugin collects or aggregate the data from various event of same type and process them in the final event
alter
It allows user to alter the field of log events, which mutate filter do not handle
anonymize
It is used replace the values of fields with a consistent hash
cipher
It is used to encrypt the output events before storing them in destination source
clone
It is used to create duplicate of the output events in Logstash
collate
It merges the events from different logs by their time or count
csv
This plugin parse data from input logs according to the separator
date
It parse the dates from the fields in the event and set that as a timestamp for the event
dissect
This plugin helps user to extract fields from unstructured data and makes it easy for grok filter to parse them correctly
drop
It is used to drop all the events of same type or any other similarity
elapsed
It is used to compute the time between the start and end events
Elasticsearch
It is used to copy the fields of previous log events present in Elasticsearch to the current one in Logstash
extractnumbers
It is used to extract the number from strings in the log events
geoip
It adds a field in the event, which contains the latitude and longitude of the location of the IP present in the log event
grok
It is the commonly used filter plugin to parse the event to get the fields
i18n
It deletes the special characters from a filed in the log event
json
It is used to create a structured Json object in event or in a specific field of an event
kv
This plugin is useful in paring key value pairs in the logging data
metrics
It is used to aggregate metrics like counting time duration in each event
multiline
It is also one of the commonly use filter plugin, which helps user in case of converting a multiline logging data to a single event.
mutate
This plugin is used to rename, remove, replace, and modify fields in your events
range
It used to check the numerical values of fields in events against an expected range and string’s length within a range.
ruby
It is used to run arbitrary Ruby code
sleep
This makes Logstash sleeps for a specified amount of time
split
It is used to split a field of an event and placing all the split values in the clones of that event
xml
It is used to create event by paring the XML data present in the logs
Codec Plugins can be a part of input or output plugins. These Plugins are used to change or format the logging data presentation. Logstash offers multiple codec Plugins and those are as follows −
avro
This plugin encode serialize Logstash events to avro datums or decode avro records to Logstash events
cloudfront
This plugin reads the encoded data from AWS cloudfront
cloudtrail
This plugin is used to read the data from AWS cloudtrail
collectd
This reads data from the binary protocol called collected over UDP
compress_spooler
It is used to compress the log events in Logstash to spooled batches
dots
This is used performance tracking by setting a dot for every event to stdout
es_bulk
This is used to convert the bulk data from Elasticsearch into Logstash events including Elasticsearch metadata
graphite
This codec read data from graphite into events and change the event into graphite formatted records
gzip_lines
This plugin is used to handle gzip encoded data
json
This is used to convert a single element in Json array to a single Logstash event
json_lines
It is used to handle Json data with newline delimiter
line
It plugin will read and write event in a single live, that means after newline delimiter there will be a new event
multiline
It is used to convert multiline logging data into a single event
netflow
This plugin is used to convert nertflow v5/v9 data to logstash events
nmap
It parses the nmap result data into an XML format
plain
This reads text without delimiters
rubydebug
This plugin will write the output Logstash events using Ruby awesome print library
You can also create your own Plugins in Logstash, which suites your requirements. The Logstash-plugin utility is used to create custom Plugins. Here, we will create a filter plugin, which will add a custom message in the events.
A user can generate the necessary files by using the generate option of the logstash-plugin utility or it is also available on the GitHub.
>logstash-plugin generate --type filter --name myfilter --path c:/tpwork/logstash/lib

Here, type option is used to specify the plugin is either Input, Output or Filter. In this example, we are creating a filter plugin named myfilter. The path option is used to specify the path, where you want your plugin directory to be created. After executing the above mentioned command, you will see that a directory structure is created.
You can find the code file of the plugin in the \lib\logstash\filters folder in the plugin directory. The file extension will be .rb.
In our case, the code file was located inside the following path −
C:\tpwork\logstash\lib\logstash-filter-myfilter\lib\logstash\filters\myfilter.rb

We change the message to − default ⇒ "Hi, You are learning this on tutorialspoint.com" and save the file.
To install this plugin, the Gemfile of Logstash need to be modified. You can find this file in the installation directory of Logstash. In our case, it will be in C:\tpwork\logstash. Edit this file using any text editor and add the following text in it.
gem "logstash-filter-myfilter",:path => "C:/tpwork/logstash/lib/logstash-filter-myfilter"

In the above command, we specify the name of the plugin along with where we can find it for installation. Then, run the Logstash-plugin utility to install this plugin.
>logstash-plugin install --no-verify

Here, we are adding myfilter in one of the previous examples −
logstash.conf
This Logstash config file contains myfilter in the filter section after the grok filter plugin.
input {
   file {
      path => "C:/tpwork/logstash/bin/log/input1.log"
   } 
}
filter {
   grok {
      match => [
         "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} -
            %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?" ]
   }
   myfilter{}
}
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output1.log"
      codec => rubydebug
   }
}
Run logstash
We can run Logstash by using the following command.
>logstash –f logsatsh.conf

input.log
The following code block shows the input log data.
INFO - 48566 - TRANSACTION_START - start

output.log
The following code block shows the output log data.
{
   "path" => "C:/tpwork/logstash/bin/log/input.log",
   "@timestamp" => 2017-01-07T06:25:25.484Z,
   "loglevel" => "INFO",
   "logger" => "TRANSACTION_END",
   "@version" => "1",
   "host" => "Dell-PC",
   "label" => "end",
   "message" => "Hi, You are learning this on tutorialspoint.com",
   "taskid" => "48566",
   "tags" => []
}
A developer can also publish his/her custom plugin to Logstash by uploading it on the github and following the standardized steps defined by the Elasticsearch Company.
Please refer the following URL for more information on publishing −
https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html
 Print
 Add Notes
 Bookmark this page