Repository: google/mtail

Issue #424: language: Unable to parse timestamp with underscore
Created: 2021-01-03T16:45:27Z
I'm using mtail to parse event logs from this home automation application (http://fhem.de) to feed prometheus with measured values.
The format of the timestamp in the logs contains an underscore.
Here's an example of an event:
2021-01-03_17:34:23 CUL_MAXCUBE_FRONT credit10ms: 3494
The format is quite simple and well structured.  TIMESTAMP DEVICE  EVENT: VALUE
The regular expression in my fhem.mtail program captures the field correctly:
/^/ +
/(?P\d{4}-\d{2}-\d{2}_\d{2}:\d{2}:\d{2}) /  {
# set timestamp register to parsed result. Go takes the given reference date
strptime($ts, "2006-01-02_15:04:05")
}
But strptime() is unable to handle the example timestamp string:  "2006-01-02_15:04:05"
For each line I receive this error message:

Error occurred at instruction 6 {strptime, 2}, originating in fhem.mtail at line 38
Full input text from "/logs/fhem-events-2021-01-03.log" was "2021-01-03_17:34:23 CUL_MAXCUBE_FRONT credit10ms: 3494"
I0103 16:34:27.168811       1 vm.go:108] Set logging verbosity higher (-v1 or more) to see full VM state dump.
I0103 16:34:27.168915       1 vm.go:106] fhem.mtail: Runtime error: strptime (2006-01-02 15:04:05, 2021-01-03_17:34:23, UTC) failed: parsing time "2021-01-03_17:34:23" as "2006-01-02 15:04:05": cannot parse "_17:34:23" as " "

It seems to be a known limitation of GOs time parse function. https://godoc.org/time#Parse
GO is not able to parse timestamps with '_'. As the underscore has a special meaning and can not be quoted!
I've found an old github issue for reference: golang/go#11334
Are there any plans to fix this in mtail by replacning the timestamp Parser used by strptime()?
What can I do to get my timestamps accepted by mtail?
Many thanks and best regards
Tim

Duplicate issue #326: language: Add function to perform string replacement
Created: 2020-06-29T00:21:19Z
I don't think this is a problem in mtail but more of a "not sure how to do this". The problem is that rsync logs the bytes with a comma (,) in the total and I'm trying to convert this into an integer.
I have the following log line:
Jun 28 20:44:32 backup rsync[3996]: sent 642,410,725 bytes  received 14,998,522,122 bytes  497,002.00 bytes/sec
And based on the rsyncd.mtail example I have now this:
/sent (?P<sent>[\d,]+) bytes  received (?P<received>[\d,]+) bytes/ {
    # Sum total bytes across all sessions for this process
    bytes_total["sent"] += $sent
    bytes_total["received"] += $received

And this creates the metrics like this:
  "bytes_total": [
    {
      "Name": "bytes_total",
      "Program": "rsyncd.mtail",
      "Kind": 1,
      "Type": 2,
      "Keys": [
        "pid"
      ],
      "LabelValues": [
        {
          "Labels": [
            "sent"
          ],
          "Value": {
            "Value": "642,410,725",
            "Time": 1593389595338436176
          }
        },
        {
          "Labels": [
            "received"
          ],
          "Value": {
            "Value": "14,998,522,122",
            "Time": 1593389595338437666
          }
        }
      ]
    }
  ]
I'm struggling on what the correct way is to remove the ',' and convert it to an integer using int(). I thought about adding additional capture groups inside the sent/received capture group and glue them together but this didn't seem to work. Help is greatly appreciated!

Closing comment (2021-03-20T06:52:56Z): Re parsing an int with commas, this is very similar to #424 and in both cases I think mtail is going to need custom parsers, which means it's going to require some more thought because the idea of allowing a comma in a regular integer probably has some edge cases I've not thought of yet... no I've thought of one, what if we're reading a comma separated file?
If we had a str_replace then we avoid that problem, because the prograam writer gets to decide when to use it. It'd return a type string, and we'd need another cast to int after.  Not so bad I expect.  We could also have a convenience method that did both -- automatically read fancy integers and strip out commas or decimal points depending on locale and return an Int type, but maybe that is overkill.