Awesome Monitoring
Infrastructure Using theElastic Stack
Mark Walkom @warkolm
Karen Carcamo @karencfv
www.elastic.co

LINUX.CONF.AU

LINUX.CONF.AU 21-25 January
2019 | Christchurch | NZ
21-25 January 2019
Christchurch, NZ

The Linux of Things

The Linux of Things#LCA2019 | @linuxconfau| #LCA2019@linuxconfau

Please install Docker :)docker pull
docker.elastic.co/elasticsearch/elasticsearch:6.5.4
docker.elastic.co/elasticsearch/elasticsearch:6.5.4-oss
docker pull docker.elastic.co/kibana/kibana:6.5.4
docker pull docker.elastic.co/kibana/kibana:6.5.4-oss
Commands at https://go.es.io/2MjxC9M
Slides at https://go.es.io/2FN9ufo

LINUX.CONF.AU

LINUX.CONF.AU 21-25 January
2019 | Christchurch | NZ
21-25 January 2019
Christchurch, NZ

The Linux of Things

The Linux of Things#LCA2019 | @linuxconfau| #LCA2019@linuxconfau

Us!
3

4

https://github.com/markwalkom/bloomsky-on-elastic
5

Elastic Stack
Elasticsearch, Kibana, Beats, and Logstash

6

Elastic Stack

Elastic Stack

KibanaVisualize &Manage

ElasticsearchStore, Search, &Analyze

Beats

LogstashIngest●Scalable●Real-time●Highly available●Developer-friendly

Elasticsearch●

Versatile storage

Heart of the Elastic Stack●Query & aggregations

QUERYTEMPORALAGGREGATIONGRAPHGEOSPATIALMACHINE LEARNING●Visualize and explore●

Manage and monitor●

Share and report●

Developer tools

Kibana●Time-series analysis

Window into the Elastic Stack●

Geospatial exploration

All the visualizations you expect, and then some more

11

OOTB dashboards for 50+ (and growing) data sources

12●

Ship from any source●

Transform at the edge

●Docker and k8s ready●Cloud metadata enrichment

Beats

●

70+ community BeatsLightweight data shippers●50+ modules

FILEBEAT
Log Files

METRICBEATMetrics

WINLOGBEAT
Window Events

PACKETBEAT
Network Data

HEARTBEAT
Uptime Monitoring

FUNCTIONBEAT
Serverless Monitoring

AUDITBEAT
Audit Data

Plus a growing set of community Beats

Logstash
Data processing pipeline●Flexible ETL engine●

Parse & transform data●

Many inputs & outputs●

Horizontally scalable●

200+ plugins

ModulesData to dashboards in 5 minutes
Turnkey for many formats
Automated data parsing
Out of the box dashboards
Preconfigured ML jobs

Let’s get started17

Let’s Install Elasticsearchdocker pull
docker.elastic.co/elasticsearch/elasticsearch:6.5.4
Ordocker pull
docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4• https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docker.html• https://www.docker.elastic.co/• https://hub.docker.com/_/elasticsearch18

Let’s Run Elasticsearchdocker run -p 9200:9200 -p 9300:9300 -e"discovery.type=single-node"
docker.elastic.co/elasticsearch/elasticsearch:6.5.4
Ordocker run -d -p 9200:9200 -p 9300:9300 -e"discovery.type=single-node"
docker.elastic.co/elasticsearch/elasticsearch:6.5.4• https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docker.html

19

Let’s docker-compose Elasticsearchversion: '2.2'
services:elasticsearch:
image:
docker.elastic.co/elasticsearch/elasticsearch:6.5.4
volumes:- esdata:/usr/share/elasticsearch/data
ports:- 9200:9200
volumes:
esdata:driver: local

20

Let’s (just) Elasticsearch
curl 0:9200/_cat/
curl 0:9200/_cat/healthcurl 0:9200/_cat/indices?v• https://www.elastic.co/guide/en/elasticsearch/reference/6.5/cat.html

21

Kibana Install
docker pull docker.elastic.co/kibana/kibana:6.5.4
Ordocker pull docker.elastic.co/kibana/kibana-oss:6.5.4• https://www.elastic.co/guide/en/kibana/6.5/docker.html• https://www.docker.elastic.co/• https://hub.docker.com/_/kibana

22

Kibana Run

docker run docker.elastic.co/kibana/kibana:6.5.4 -p
5601:5601 -e "elasticsearch.url=localhost:9200"••

23(wait for it)
Open http://localhost:5601/

Let’s docker-compose Kibana
kibana:image: docker.elastic.co/kibana/kibana:6.5.4
links:- elasticsearchports:- 5601:5601•24

Use the complete Docker compose file in the gistdocker-compose up••

Just run that command
Alternatively;
docker-compose up -d25

26

Metricbeat
https://www.elastic.co/guide/en/beats/metricbeat/current/index.html

27

Metricbeat Install and Run••

Download the binary
Extract./metricbeat

•

STOP!
./metricbeat setup --help./metricbeat modules --help

28

Metricbeat Install and Run./metricbeat modules list./metricbeat setup --template -E
output.logstash.enabled=false -E'output.elasticsearch.hosts=["localhost:9200"]'./metricbeat setup --dashboards./metricbeat••
29

Enable the system module
See also https://go.es.io/2T44qWN

Filebeathttps://www.elastic.co/guide/en/beats/filebeat/current/index.html

30

Filebeat Install and Run••

Download the binary
Extract./filebeat•

Remember./filebeat setup --help./filebeat modules --help31

Filebeat Install and Run./filebeat modules list./filebeat setup --template -E
output.logstash.enabled=false -E'output.elasticsearch.hosts=["localhost:9200"]'./filebeat setup --dashboards./filebeat••

32

Enable the system module
See also https://go.es.io/2T44qWN

Let’s kick itup a notch

33Metricbeat - Enable the docker module
Filebeat - Enable the docker input

34

Let’s analyse Elasticsearchtraffic!

35

Packetbeat
https://www.elastic.co/guide/en/beats/packetbeat/current/index.html

36

Packetbeat Install and Run••

Download the binary
Extract./packetbeat•

Remember./packetbeat setup --help./packetbeat modules --help

37

Metricbeat Install and Run./packetbeat modules list./packetbeat setup --template -E
output.logstash.enabled=false -E'output.elasticsearch.hosts=["localhost:9200"]'./packetbeat setup --dashboards./packetbeat•38

See also https://go.es.io/2T44qWN

Heartbeat
https://www.elastic.co/guide/en/beats/heartbeat/current/index.html

39

Heartbeat Install and Run••

Download the binary
Extract./heartbeat•

Remember./heartbeat setup --help./heartbeat modules --help

40

Heartbeat Install and Run./heartbeat modules list./heartbeat setup --template -E
output.logstash.enabled=false -E'output.elasticsearch.hosts=["localhost:9200"]'./heartbeat setup --dashboards./heartbeat•41

See also https://go.es.io/2T44qWN

Community beats
Logstash!
Elasticsearch Ingest[Heart|Winlog|DIY]beat
APM
Dashboards
42

Whakawhetai Koe!

LINUX.CONF.AU

LINUX.CONF.AU 21-25 January
2019 | Christchurch | NZ
21-25 January 2019
Christchurch, NZ

The Linux of Things

The Linux of Things#LCA2019 | @linuxconfau| #LCA2019@linuxconfau