Repository: elastic/logstash-forwarder

Issue #167: Daemon overwrites its own state file when watching multiple types of logs.
Created: 2014-02-19T03:30:30Z
To reproduce, create a configuration file which watches two different types of logs (e.g. apache_access and syslog):
{
  "network": {
    "servers": [ "10.0.0.1:5043" ],
    "ssl certificate": "/etc/pki/tls/certs/logstash-forwarder.crt",
    "ssl key": "/etc/pki/tls/private/logstash-forwarder.key",
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "/var/log/httpd/access_log" ],
      "fields": { "type": "apache-access" }
    }, {
      "paths": [ "/var/log/messages" ],
      "fields": { "type": "syslog" }
    }
  ]
}

You'll see the .logstash-forwarder file get updated with one type of logs. For example, after reading apache it might read:
{"/var/log/httpd/access_log":{"source":"/var/log/httpd/access_log","offset":4095918,"inode":131475,"device":2311}}

Now manually force an update to syslog by inserting a test event:
logger Test event

Suddenly, the .logstash-forwarder file contains only the state information for /var/log/messages:
{"/var/log/messages":{"source":"/var/log/messages","offset":4888307,"inode":132308,"device":2311}}

Do I correctly presume that this is why people are often complaining about logstash-forwarder often re-reading some of their logs from the beginning?

Duplicate issue #174: /run/.logstash-forwarder does not contain all logfiles
Created: 2014-03-03T22:25:27Z
The /run/.logstash-forwarder does not contain the position of all logfiles it monitors.
logstash-forwarder forwards a lot of files:
{
    "network": {
        "servers": [ "XX.XX.XX.XX:6782" ],
        "ssl ca": "/etc/logstash-forwarder/server.ca",
        "timeout": 15
    },
    "files": [
        {
            "paths": [
                "/mnt/log/syslog-filtered"
            ],
            "fields": { "type": "syslog" }
        },
        {
            "paths": [
                "/var/log/exim4/mainlog",
                "/var/log/exim4/paniclog"
            ],
            "fields": { "type": "exim" }
        },
        {
            "paths": [
                "/mnt/log/nginx/*.log"
            ],
            "fields": { "type": "nginx" }
        },
        [... much more ...]
    ]
}

Yet, /run/logstash-forwarder contains most times only one file -- the file with most log entries. All other files are skipped.  If logstash-forwarder is stopped, it does not delete this file (which is good, so we know where to start collecting logs when we start again) but does not contain all log files (which is not good).
Example:
$ for i in $(seq 20);do cat /run/.logstash-forwarder;echo ---;sleep 1;done
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247723071,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247747717,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247771315,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247795252,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247817381,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247850040,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247874684,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247898339,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247920265,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247945231,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247969304,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1247982004,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248007216,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248030735,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248056127,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248081068,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248093630,"inode":6136166,"device":51728},"/mnt/log/syslog-filtered":{"source":"/mnt/log/syslog-filtered","offset":124811442,"inode":6135886,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248118396,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248143618,"inode":6136166,"device":51728}}

---
{"/mnt/log/nginx/access.log":{"source":"/mnt/log/nginx/access.log","offset":1248168758,"inode":6136166,"device":51728}}

---

As you can see, only one time there is the status of two log files, all other times log files without new entries are ignored. The exim logs are not even shown due of absence of new log entries.
That might explain some issues which report duplicate log entries.

Closing comment (2015-03-04T00:46:38Z): I think this is a duplicate of #167. I believe it to be fixed in master.