Repository: google/mtail

Issue #263: Times are not represented as Time in the VM so lose precision when compared
Created: 2019-08-30T11:55:12Z
Mtail records timestamps as Go time.Time values, that have nanosecond resolution. However, when accessing a time acquired using timestamp() from a program, e.g., when subtracting two timestamps or assigning a timestamp to a gauge, then the timestamps are converted to integers using Time.Unix(). This means that the resolution when comparing timestamps is 1s. This can be ok when monitoring longer sessions such as in the rsync example, where the durations are tens of seconds or more. But if the durations are in single digit seconds, the systematic error is significant. Many applications have sub-second transactions or sessions, and these will all be rounded to 0.
Here is a toy demonstration that monitors the duration between emitted log lines for the mtail log itself.
# demo.mtail
counter lines_total

# For why the 0 and negative buckets are needed, see:
# https://github.com/google/mtail/issues/226#issuecomment-524862192
histogram delta_t buckets -1e9, 0, 0.0005, 0.001, 0.01, 0.1, 1, 10, 100

gauge last

def glog {
        /(?P<level>[IWECF])(?P<timestamp>\d{4} \d{2}:\d{2}:\d{2}\.\d{6}) +/
        /(?P<thread>\d+) (?P<src_file>[^:]+):(?P<src_line>\d+)\] / {
                strptime($timestamp, "0102 15:04:05")
                next
        }
}

@glog {

/$/ {
        lines_total > 0 {
                delta_t = (timestamp() - last)
        }
        last = timestamp()
        lines_total++
}

}

Example input:
I0829 22:08:41.220692    1288 main.go:100] mtail version invalid ... go version go1.12.7 go arch amd64 go os windows
I0829 22:08:41.221691    1288 main.go:101] Commandline: ["C:\\mtail.exe" "-progs" "demo.mtail" "-logs" "foo.log" "-logtostderr"]
I0829 22:08:41.224689    1288 log_watcher.go:248] No abspath in watched list, added new one for X
I0829 22:08:41.224689    1288 checker.go:202] declaration of capture group reference `src_file' at demo.mtail:7:2-59 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `src_line' at demo.mtail:7:2-59 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `src_line' at demo.mtail:7:2-59 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `thread' at demo.mtail:7:2-59 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `thread' at demo.mtail:7:2-59 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `src_file' at demo.mtail:7:2-59 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `level' at demo.mtail:6:2-68 appears to be unused
I0829 22:08:41.225714    1288 checker.go:202] declaration of capture group reference `level' at demo.mtail:6:2-68 appears to be unused
I0829 22:08:41.226713    1288 loader.go:229] Loaded program demo.mtail
I0829 22:08:41.226713    1288 log_watcher.go:248] No abspath in watched list, added new one for X
W0829 22:08:41.227713    1288 mtail.go:101] No matches for pattern "foo.log"
I0829 22:08:41.227713    1288 store.go:136] Starting metric store expiry loop every 1h0m0s
I0829 22:08:41.227713    1288 tail.go:435] Starting log handle expiry loop every 1h0m0s
I0829 22:08:41.227713    1288 mtail.go:321] Listening on [::]:3903
I0829 22:08:48.037569    1288 log_watcher.go:253] Found this processor in watched list
I0829 22:08:48.038569    1288 log_watcher.go:248] No abspath in watched list, added new one for C:\foo.log
I0829 22:08:48.044581    1288 tail.go:289] Tailing C:\foo.log
I0829 22:09:21.933622    1288 mtail.go:349] Received SIGTERM, exiting...
I0829 22:09:21.934635    1288 mtail.go:363] Shutdown requested.
I0829 22:09:21.935667    1288 log_watcher.go:218] Shutting down log watcher.
I0829 22:09:21.935667    1288 loader.go:381] Shutting down loader.
I0829 22:09:21.936655    1288 mtail.go:386] END OF LINE

Example Prometheus metrics:
delta_t_bucket{prog="demo.mtail",le="0"} 22
delta_t_bucket{prog="demo.mtail",le="0.0005"} 22
delta_t_bucket{prog="demo.mtail",le="0.001"} 22
delta_t_bucket{prog="demo.mtail",le="0.01"} 22
delta_t_bucket{prog="demo.mtail",le="0.1"} 22
delta_t_bucket{prog="demo.mtail",le="1"} 22
delta_t_bucket{prog="demo.mtail",le="10"} 23
delta_t_bucket{prog="demo.mtail",le="100"} 24
delta_t_bucket{prog="demo.mtail",le="+Inf"} 24
delta_t_sum{prog="demo.mtail"} 40
delta_t_count{prog="demo.mtail"} 24

Here we see that all durations were counted as 0, except for the 7s and 33s pauses, that add up to the sum of 40s.
I tested and changed the few places in vm.go that use Unix seconds to use Unix nanoseconds instead. This of course works, but is hardly backwards compatible as timestamp() returns a billion times larger values. Perhaps changing to Unix seconds as a float64 would be better, as that wouldn't change the unit and would give microsecond resolution.

Duplicate issue #264: Timestamp delta in milliseconds?
Created: 2019-08-30T12:20:04Z
Hi!
I'm trying to parse a file of a following format:
2019-08-23 10:02:14.772 DEBUG 28197 --- Order #18948736505: routing new order
2019-08-23 10:02:14.827 DEBUG 28197 --- Order #18948736505: all matching orders succeeded

My goal is to determine order processing times, e.g. time delta between line 2 and line 1. I've written a naive program to do this:
hidden gauge order_start by orderno
gauge order_time by orderno

/(?P<ctime>.{23}).+Order #(?P<orderno>\d+): routing new order/ {
  strptime($ctime, "2006-01-02 15:04:05.000")
  order_start[$orderno] = timestamp()
}

/(?P<ctime>.{23}).+Order #(?P<orderno>\d+): all matching orders succeeded/ {
  strptime($ctime, "2006-01-02 15:04:05.000")
  order_time[$orderno] =timestamp() -order_start[$orderno]
}

So what I'm getting is difference in seconds:
[user@centos7-5 tmp]$ ./mtail_v3.0.0-rc33_linux_amd64 -progs=/tmp/mtail_progs/ -logs=test_logs/router-orders_copy.log -port=12000 -logtostderr -one_shot
I0830 15:09:07.987692    2342 main.go:100] mtail version v3.0.0-rc33 git revision aedde73f9c304e4d558a53ece22a5472c87a7fdb go version go1.12.7 go arch amd64 go os linux
I0830 15:09:07.987935    2342 main.go:101] Commandline: ["./mtail_v3.0.0-rc33_linux_amd64" "-progs=/tmp/mtail_progs/" "-logs=test_logs/router-orders_copy.log" "-port=12000" "-logtostderr" "-one_shot"]
I0830 15:09:07.989459    2342 log_watcher.go:248] No abspath in watched list, added new one for /tmp/mtail_progs
I0830 15:09:07.990477    2342 loader.go:224] Loaded program test_prog.mtail
I0830 15:09:07.990612    2342 log_watcher.go:248] No abspath in watched list, added new one for /tmp/test_logs
I0830 15:09:07.990671    2342 log_watcher.go:248] No abspath in watched list, added new one for /tmp/test_logs/router-orders_copy.log
I0830 15:09:07.990731    2342 log_watcher.go:253] Found this processor in watched list
I0830 15:09:07.990851    2342 log_watcher.go:253] Found this processor in watched list
I0830 15:09:10.740334    2342 tail.go:289] Tailing /tmp/test_logs/router-orders_copy.log
I0830 15:09:10.740365    2342 mtail.go:363] Shutdown requested.
I0830 15:09:10.740808    2342 log_watcher.go:218] Shutting down log watcher.
I0830 15:09:10.740846    2342 loader.go:376] Shutting down loader.
I0830 15:09:10.740884    2342 mtail.go:386] END OF LINE
Metrics store:{
  "order_time": [
    {
      "Name": "order_time",
      "Program": "test_prog.mtail",
      "Kind": 2,
      "Type": 0,
      "Keys": [
        "orderno"
      ],
      "LabelValues": [
        {
          "Labels": [
            "18948736505"
          ],
          "Value": {
            "Value": 0,
            "Time": 1566554534827000000
          }
        }
      ]
    }
  ]
}

Showing order_start gauge makes it obvious that part of timestamp is lost while being stored as a gauge value:
          "Value": {
            "Value": "1566587946",
            "Time": 1566587946002000000
          }

Is there a way to get delta between two timestamps in milliseconds? What am I getting wrong here?
Thanks for your help.

Closing comment (2019-08-30T22:46:18Z): Yeah :)
…
On Fri, 30 Aug 2019, 22:24 Anthony, ***@***.***> wrote:
 Duplicate of #263 <#263> ? Wow now
 that's a coincidence :)

 —
 You are receiving this because you commented.
 Reply to this email directly, view it on GitHub
 <#264?email_source=notifications&email_token=AAXFX646CG2FNGFIGD6WAZ3QHEGOBA5CNFSM4ISMW4C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5RP4MQ#issuecomment-526581298>,
 or mute the thread
 <https://github.com/notifications/unsubscribe-auth/AAXFX62HOJ42MWUXYTAVYPLQHEGOBANCNFSM4ISMW4CQ>
 .