Repository: jqlang/jq

Problem statement:
Remove undocumented and untested --stream-errors option
This PR drops `--stream-errors` option. This option is undocumented, and untested. Also buggy; `printf '[' | jq --stream-errors .` aborts with assertion error.


Hints:
EDIT: `printf '[' | ./jq -c --stream --stream-errors .` does not abort (it outputs `["Unfinished JSON term at EOF at line 1, column 1",[0]]`).  So another fix would be to make `--stream-errors` imply `--stream`.

It should be an easy fix:

```
(gdb) p jv_show(result,0)
["Unfinished JSON term at EOF at line 2, column 0",<invalid>]$1 = void
(gdb)
```

The problem is that we need to check for `.[1]` of `result` being invalid, and then do something about that: dump just `.[0]` of `result`, or drop the `.[1]` of `result.

Now, the question is: is `--stream-errors` useful?  I think it should be!

And this doesn't abort:

```
: ; printf '[' | ./jq --stream-errors '.[0]'
"Unfinished JSON term at EOF at line 1, column 1"
```
And:

```
: ; (./jq -cn '[{a:{b:["c"]}}]'; printf '['; ./jq -cn '["a",{b:"c"}]') | ./jq -c --stream --stream-error
s .
[[0,"a","b",0],"c"]
[[0,"a","b",0]]
[[0,"a","b"]]
[[0,"a"]]
[[0]]
[[0,0],"a"]
[[0,1,"b"],"c"]
[[0,1,"b"]]
[[0,1]]
["Unfinished JSON term at EOF at line 3, column 0",[0]]
```
One thing we should do is write a fuzzer for the streaming parser.  Also, we should document what is and what is not fuzzed so that users who need better security know what features to use / not use.
Okay, we need to fix the crash, and update the help (after #2747) and the manual.
> Okay, we need to fix the crash, and update the help (after #2747) and the manual.

I'll try to get to it tomorrow.  Maybe or maybe later tonight.

Patch:
diff --git a/docs/content/manual/manual.yml b/docs/content/manual/manual.yml
index ff68482a3d..e36cfea920 100644
--- a/docs/content/manual/manual.yml
+++ b/docs/content/manual/manual.yml
@@ -130,6 +130,16 @@ sections:
         conjunction with filtering and the `reduce` and `foreach` syntax
         to reduce large inputs incrementally.
 
+      * `--stream-errors`:
+
+        Like `--stream`, but invalid JSON inputs yield array calues
+        where the first element is the error and the second is a path.
+        For example, `["a",n]` produces ["Invalid literal at line 1,
+        column 9",[1]]`.
+
+        Implies `--stream`.  Invalid JSON inputs produce no error values
+        when `--stream` without `--stream-errors`.
+
       * `--slurp`/`-s`:
 
         Instead of running the filter for each JSON object in the
diff --git a/jq.1.prebuilt b/jq.1.prebuilt
index 80933f748e..80ebfe5883 100644
--- a/jq.1.prebuilt
+++ b/jq.1.prebuilt
@@ -73,6 +73,15 @@ Parse the input in streaming fashion, outputting arrays of path and leaf values
 This is useful for processing very large inputs\. Use this in conjunction with filtering and the \fBreduce\fR and \fBforeach\fR syntax to reduce large inputs incrementally\.
 .
 .TP
+\fB\-\-stream\-errors\fR:
+.
+.IP
+Like \fB\-\-stream\fR, but invalid JSON inputs yield array calues where the first element is the error and the second is a path\. For example, \fB["a",n]\fR produces ["Invalid literal at line 1, column 9",[1]]`\.
+.
+.IP
+Implies \fB\-\-stream\fR\. Invalid JSON inputs produce no error values when \fB\-\-stream\fR without \fB\-\-stream\-errors\fR\.
+.
+.TP
 \fB\-\-slurp\fR/\fB\-s\fR:
 .
 .IP
diff --git a/src/main.c b/src/main.c
index 48af5a31c1..1e85f9e360 100644
--- a/src/main.c
+++ b/src/main.c
@@ -455,7 +455,7 @@ int main(int argc, char* argv[]) {
         continue;
       }
       if (isoption(argv[i], 0, "stream-errors", &short_opts)) {
-        parser_flags |= JV_PARSE_STREAM_ERRORS;
+        parser_flags |= JV_PARSE_STREAMING | JV_PARSE_STREAM_ERRORS;
         continue;
       }
       if (isoption(argv[i], 'e', "exit-status", &short_opts)) {


Test patch:
diff --git a/tests/shtest b/tests/shtest
index b2b8571d86..1ee84d3ac4 100755
--- a/tests/shtest
+++ b/tests/shtest
@@ -180,6 +180,11 @@ $VALGRIND $Q $JQ -c '. as $d|path(..) as $p|$d|getpath($p)|select((type|. != "ar
 $VALGRIND $Q $JQ --stream -c '.|select(length==2)' < "$JQTESTDIR/torture/input0.json" > $d/out1
 diff $d/out0 $d/out1
 
+printf '["Unfinished JSON term at EOF at line 1, column 1",[0]]\n' > $d/expected
+printf '[' | $VALGRIND $Q $JQ --stream-errors -c . > $d/out 2>&1
+diff $d/out $d/expected
+
+
 ## XXX This test can be moved to tests/all.test _now_
 clean=false
 if which seq > /dev/null 2>&1; then