Repository: rcoh/angle-grinder (language: rust)

Issue:
Crash while parsing syslog for netfilter logs
I got the attached crash while trying to parse netfilter (iptables) logs from `/var/log/syslog`. Full command and output below.

Crash report (renamed to .txt. because GitHub won't permit .toml files as attachments): [report-79c5d249-00d8-4ad4-bfbe-29b534be5307.txt](https://github.com/rcoh/angle-grinder/files/11372383/report-79c5d249-00d8-4ad4-bfbe-29b534be5307.txt)

Input file was slightly less than 310,076,892 lines (counted immediately after the crash while typing this Issue).

```
$ ./agrind -f /var/log/syslog --output json 'kernel:* | parse "* * kernel: [*] * IN=* OUT=* SRC=* DST=* LEN=* TOS=* PREC=* TTL=* ID=* PROTO=* SPT=* DPT=* WINDOW=* RES=* * URGP=*" as timestamp, hostname, x, tag, ifIn, ifOut, src, dst, len, tos, prec, ttl, id, protocol, sport, dport, window, res, flags, urgp | count'
Well, this is embarrassing.

ag had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/tmp/report-a8b8a71c-bdb4-484a-a0ee-7be663a3a952.toml". Submit an issue or email with the subject of "ag Crash Report" and include the report as an attachment.

- Authors: Russell Cohen <russell.r.cohen@gmail.com>

We take privacy seriously, and do not perform any automated error collection. In order to improve the software, we rely on people to submit reports.

Thank you kindly!
278
```


Discussion / hints:
FWIW, I get the same crash if I `cat /var/log/syslog | ./agrind ....` instead of making ag read it using `-f` argument.

Using `grep kernel: /var/log/syslog | ./agrind '* | parse ....` does NOT crash. I'm guessing some kind of weird character in one of the log lines that ag isn't handling nicely. Hopefully the crash report has enough info for you to work it out - I don't _really_ want to dig through 310mm log lines to find it 😂 

(For my own reference if I do need to refer to the archived log file: `/var/log/syslog-20230502-1683025201.gz`)

Fix patch:
diff --git a/src/lib.rs b/src/lib.rs
index 7a9b818..5e459c7 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -283,8 +283,8 @@ pub mod pipeline {
                 if ct == 0 {
                     break;
                 }
-                let data = std::str::from_utf8(&line[..ct]).unwrap();
-                if self.filter.matches(data)
+                let data = String::from_utf8_lossy(&line[..ct]);
+                if self.filter.matches(data.as_ref())
                     && !Pipeline::proc_preagg(Record::new(data), &mut preaggs, &tx)
                 {
                     break;


Test patch:
diff --git a/test_files/binary_data.bin b/test_files/binary_data.bin
new file mode 100644
index 0000000..73f452f
--- /dev/null
+++ b/test_files/binary_data.bin
@@ -0,0 +1,3 @@
+line of regular text k=v2
+�K���@�V���+P��Q�Þ�0����S��oU�!+��X-!����޴A�� �r��[�OC��j_kΕ����V����5; �x1rYuV���
+line of regular text k=v
diff --git a/tests/integration.rs b/tests/integration.rs
index 61e4d90..17e56b7 100644
--- a/tests/integration.rs
+++ b/tests/integration.rs
@@ -146,6 +146,18 @@ None         1\n",
             );
     }
 
+    #[test]
+    fn binary_input() {
+        run()
+            .args([
+                "* | parse 'k=*' as k",
+                "--file",
+                "test_files/binary_data.bin",
+            ])
+            .assert()
+            .stdout("[k=v2]\n[k=v]\n");
+    }
+
     #[test]
     fn filter_wildcard() {
         run()