Grafana Labs: Loki — logs collecting and monitoring system
By: Arseny Zinchenko, Setevoy
Tags: Loki, Grafana, Monitoring

Grafana 6.0 was still in Beta on Feb 2019, when the new feature was introduced — Loki, a log aggregation system available via another new ability — the Explore.

It’s similar to well-known ELK/EFK stack but more simple to set up and use and is intended to be used mostly with clouds and systems like Prometheus and Kubernetes.

It’s still in the early development stage and main disadvantage which I found during an attempt to set up the Proof of Concept described in this post is the lack of documentation and examples.

Thus — in this and a couple of next posts I’ll try to show a few its set up and usage examples.

In general, I liked Loki, because you don’t need to set up the whole dedicated ELK stack: the same Grafana can be used for your monitoring dashboards and all your logs are available via the same Grafana instance. So I’ll try to implement it on my current project.

Or will go back to the ELK solution…)

Project’s home page — here>>>, post in Grafana Lab’s blog — here>>>, and the Github repository here>>>.

There is also a good discussion on the Reddit here>>>.

Loki has the logcli tool - but I didn't check it yet.

Loki-stack consists of three main components:

promtail - agent to collect logs on a host and push them to a Loki instance

- agent to collect logs on a host and push them to a Loki instance loki - logs aggregation and processing server

- logs aggregation and processing server Grafana — for querying and displaying logs

Now let’s try to set up it, add some NGINX logs and to check how it will look like in Grafana.

Loki Docker Compose stack

Create Loki’s home directory:

root@bttrm-dev-console:/opt# mkdir /opt/loki

root@bttrm-dev-console:/opt# cd /opt/loki/

Next, create a Docker Compose file — loki-compose.yml :

version: "3"



networks:

loki:



services:

loki:

image: grafana/loki:master

ports:

- "3100:3100"

command: -config.file=/etc/loki/local-config.yaml

networks:

- loki



promtail:

image: grafana/promtail:master

volumes:

- /var/log:/var/log

command: -config.file=/etc/promtail/docker-config.yaml

networks:

- loki



grafana:

image: grafana/grafana:master

ports:

- "3000:3000"

networks:

- loki

Start containers:

root@bttrm-dev-console:/opt/loki# docker-compose -f loki-compose.yml up

In the promtail 's output, you'll see information about logs watched by promatil :

…

promtail_1 | level=info ts=2019–02–05T14:07:30.777577365Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”varlogs\”}”

promtail_1 | level=info ts=2019–02–05T14:07:30.779215957Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/daemon.log

promtail_1 | 2019/02/05 14:07:30 Seeked /var/log/alternatives.log — &{Offset:19515 Whence:0}

promtail_1 | level=info ts=2019–02–05T14:07:30.782000937Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/alternatives.log

promtail_1 | 2019/02/05 14:07:30 Seeked /var/log/auth.log — &{Offset:3869 Whence:0}

promtail_1 | level=info ts=2019–02–05T14:07:30.782899496Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/auth.log

promtail_1 | level=info ts=2019–02–05T14:07:30.783409465Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/bootstrap.log

promtail_1 | level=info ts=2019–02–05T14:07:30.78359605Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init-output.log

promtail_1 | level=info ts=2019–02–05T14:07:30.783648219Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init.log

promtail_1 | level=info ts=2019–02–05T14:07:30.784386768Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/dpkg.log

promtail_1 | level=info ts=2019–02–05T14:07:30.785801248Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/faillog

promtail_1 | level=info ts=2019–02–05T14:07:30.785871765Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/kern.log

promtail_1 | level=info ts=2019–02–05T14:07:30.785943765Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/lastlog

promtail_1 | level=info ts=2019–02–05T14:07:30.78598882Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/syslog

promtail_1 | level=info ts=2019–02–05T14:07:30.786064399Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/user.log

…

Similarly to Prometheus exporters — you can access data via the /metrics URI:

root@ip-172–31–43–174:/opt/loki# curl localhost:3100/metrics

#HELP cortex_cache_corrupt_chunks_total Total count of corrupt chunks found in cache.

#TYPE cortex_cache_corrupt_chunks_total counter

cortex_cache_corrupt_chunks_total 0.0

#HELP cortex_chunk_store_chunks_per_query Distribution of #chunks per query.

TYPE cortex_chunk_store_chunks_per_query histogram

cortex_chunk_store_chunks_per_query_bucket{le=”10.0"} 0.0

cortex_chunk_store_chunks_per_query_bucket{le=”80.0"} 0.0

cortex_chunk_store_chunks_per_query_bucket{le=”640.0"} 0.0

cortex_chunk_store_chunks_per_query_bucket{le=”5120.0"} 0.0

cortex_chunk_store_chunks_per_query_bucket{le=”40960.0"} 0.0

cortex_chunk_store_chunks_per_query_bucket{le=”327680.0"} 0.0

cortex_chunk_store_chunks_per_query_bucket{le=”+Inf”} 0.0

…

Now you can access your Grafana instance via http://IP:3000 URL:

Login with the default credentials — admin:admin.

Adding Loki data source

Grafana will suggest you add a data source:

Add the Loki data source:

Set Loki’s URL as http://loki:3100 (as we are running our Loki’s stack in the Docker’s own network):

And now you can navigate to the Explore:

Check some already collected logs via the Log labels:

Adding new logs

Logs to be tailed are configured in the promtail 's /etc/promtail/local-config.yaml or /etc/promtail/docker-config.yaml configs.

Now let’s install NGINX:

root@ip-172–31–43–174:/home/admin# apt install nginx -y

Check if log-files appeared on the host:

root@ip-172–31–43–174:/opt/loki# tail -f /var/log/nginx/*

==> /var/log/nginx/access.log <==

==> /var/log/nginx/error.log <==

==> /var/log/nginx/access.log <==

194.***.***.27 — — [05/Feb/2019:14:03:02 +0000] “GET / HTTP/1.1” 200 396 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36”

194.***.***.27 — — [05/Feb/2019:14:03:03 +0000] “GET /favicon.ico HTTP/1.1” 404 200 “http://34.243.211.206/" “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36”

Loki’s configuration file stored inside the container at this moment in the /etc/loki directory:

root@ip-172–31–43–174:/opt/loki# docker exec -ti loki_loki_1 ls -l /etc/loki/

total 4

-rw-r — r — 1 root root 489 Feb 5 12:00 local-config.yaml

promtail 's - in /etc/promtail :

root@ip-172–31–43–174:/opt/loki# docker exec -ti loki_promtail_1 ls -l /etc/promtail/

total 8

-rw-r — r — 1 root root 312 Feb 5 12:00 docker-config.yaml

-rw-r — r — 1 root root 317 Feb 5 12:00 local-config.yaml

Let’s create own configuration for promtail - /opt/loki/loki-promtail-conf.yml :





http_listen_port: 9080

grpc_listen_port: 0



positions:



filename: /tmp/positions.yaml



client:



url:



scrape_configs:



- job_name: system

entry_parser: raw

static_configs:

- targets:

- localhost

labels:

job: varlogs

__path__: /var/log/*log



- job_name: nginx

entry_parser: raw

static_configs:

- targets:

- localhost

labels:

job: nginx

__path__: /var/log/nginx/*log server:http_listen_port: 9080grpc_listen_port: 0positions:filename: /tmp/positions.yamlclient:url: http://loki:3100/api/prom/push scrape_configs:- job_name: systementry_parser: rawstatic_configs:- targets:- localhostlabels:job: varlogs__path__: /var/log/*log- job_name: nginxentry_parser: rawstatic_configs:- targets:- localhostlabels:job: nginx__path__: /var/log/nginx/*log

Here in the scrape_configs we added a new job - nginx with the __path__ to the directory and files to be tailed by the promtail .

Next, let’s update our Compose file and add the new config file mapping from our host inside the promtail 's container as /etc/promtail/docker-config.yaml :

...

promtail:

image: grafana/promtail:master

volumes:

- /opt/loki/loki-promtail-conf.yml:/etc/promtail/docker-config.yaml

- /var/log:/var/log

command: -config.file=/etc/promtail/docker-config.yaml

networks:

- loki

...

Restart Compose stack and check logs in the promtail 's output to see which logs are watched by the promtail now:

…

promtail_1 | level=info ts=2019–02–05T15:08:10.53723788Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”varlogs\”}”

promtail_1 | level=info ts=2019–02–05T15:08:10.538575763Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/daemon.log

promtail_1 | level=info ts=2019–02–05T15:08:10.539914281Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/dpkg.log

promtail_1 | level=info ts=2019–02–05T15:08:10.541431662Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/alternatives.log

promtail_1 | level=info ts=2019–02–05T15:08:10.541911162Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/auth.log

promtail_1 | level=info ts=2019–02–05T15:08:10.542352516Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/bootstrap.log

promtail_1 | level=info ts=2019–02–05T15:08:10.542780189Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init-output.log

promtail_1 | level=info ts=2019–02–05T15:08:10.543222436Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/cloud-init.log

promtail_1 | level=info ts=2019–02–05T15:08:10.54394579Z caller=filetargetmanager.go:165 msg=”Adding target” key=”{job=\”nginx\”}”

promtail_1 | level=info ts=2019–02–05T15:08:10.54473202Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/faillog

promtail_1 | level=info ts=2019–02–05T15:08:10.545166611Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/kern.log

promtail_1 | level=info ts=2019–02–05T15:08:10.54564307Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/lastlog

promtail_1 | level=info ts=2019–02–05T15:08:10.546097865Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/syslog

promtail_1 | level=info ts=2019–02–05T15:08:10.54651553Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/user.log

promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/alternatives.log — &{Offset:0 Whence:0}

promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/auth.log — &{Offset:0 Whence:0}

promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/nginx/access.log — &{Offset:0 Whence:0}

promtail_1 | level=info ts=2019–02–05T15:08:10.548790452Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/nginx/access.log

promtail_1 | 2019/02/05 15:08:10 Seeked /var/log/nginx/error.log — &{Offset:0 Whence:0}

promtail_1 | level=info ts=2019–02–05T15:08:10.54959973Z caller=filetarget.go:269 msg=”start tailing file” path=/var/log/nginx/error.log

…

key=”{job=\”nginx\”}” appeared — all good.

Let it work for a couple of minutes and check inside Grafana:

Logs present and we are done for now.

Check the next post — Grafana Labs: Loki — distributed system, labels and filters to see an example of how to set up a distributed system and tags and filters usage example.