Repository: jqlang/jq

Problem statement:
stderr doesn't print raw
**Describe the bug**
stderr doesn't print its input as a raw string but as an escaped json string.

**To Reproduce**
```sh
jq -n '"line1\nline2" | stderr | empty'
```
Outputs ```"line1\nline2"```

**Expected behavior**
As the manual states:
> **stderr**
> Prints its input in **raw** and compact mode to stderr with no additional decoration, not even a newline.

I expected the program to output be:
```
line1
line2
```


**Environment (please complete the following information):**
 - OS and Version: Redhat Enterprise Linux 6
 - jq version 1.6


stderr doesn't print raw
**Describe the bug**
stderr doesn't print its input as a raw string but as an escaped json string.

**To Reproduce**
```sh
jq -n '"line1\nline2" | stderr | empty'
```
Outputs ```"line1\nline2"```

**Expected behavior**
As the manual states:
> **stderr**
> Prints its input in **raw** and compact mode to stderr with no additional decoration, not even a newline.

I expected the program to output be:
```
line1
line2
```


**Environment (please complete the following information):**
 - OS and Version: Redhat Enterprise Linux 6
 - jq version 1.6




Hints:
This is not a bug, but a feature.  Perhaps `stderr` should output raw when jq is in raw output mode, or perhaps we need a new `rawstderr` builtin.  Or perhaps we can leave this to the I/O work we're doing, and then the user can open `/dev/null` in raw mode.
The documentation is somewhat misleading

> stderr
> Prints its input in *raw* and *compact* mode to stderr with no additional decoration, not even a newline.

Would let you believe that stderr behave like `--raw-output` and `--compact-output`. _"Not even a new line"_ is where it differs from `--compact-output`.

Perhaps another word than "raw" could be used here?
Yeah this is a bug IMO -- either the behavior or the documentation should be fixed, because right now they're inconsistent.
I am really enjoying jq - unfortunately I tripped up on this as well

I perused the docs quite extensively and it seems there is no way to print an arbitrary string (with escape sequences) to standard error?

I can't use `stderr` because newlines of its input are automatically escaped. `halt_error` prints the way I want, but it halts the program. The only way I have been able to print a newline to stderr is through `debug` - its an unfortunate unaesthetic workaround, but it works.

I don't think that the behavior of `stderr` should change based on if jq is in raw mode. If it is not possible in one way or another to change stderr, I would welcome another function that yields the expected behavior.
Just wanted to register a `+1` on folks who have run into this surprising behavior.
`+1` from me, too.  I was really hoping to provide a customized warning, but since I cannot get `stderr` to actually create a newline, I guess I'm stuck with `debug`.
FWIW, I ended up here because I was trying to do something like this:

```
. as $line | try fromjson catch $line | stderr
```

I'm filtering ndjson logs and wanted to be able to use the `try fromjson` to filter valid json log lines that I could then pipe into further transforms, and wanted to be able to just redirect malformed lines to stderr
This is not a bug, but a feature.  Perhaps `stderr` should output raw when jq is in raw output mode, or perhaps we need a new `rawstderr` builtin.  Or perhaps we can leave this to the I/O work we're doing, and then the user can open `/dev/null` in raw mode.
The documentation is somewhat misleading

> stderr
> Prints its input in *raw* and *compact* mode to stderr with no additional decoration, not even a newline.

Would let you believe that stderr behave like `--raw-output` and `--compact-output`. _"Not even a new line"_ is where it differs from `--compact-output`.

Perhaps another word than "raw" could be used here?
Yeah this is a bug IMO -- either the behavior or the documentation should be fixed, because right now they're inconsistent.
I am really enjoying jq - unfortunately I tripped up on this as well

I perused the docs quite extensively and it seems there is no way to print an arbitrary string (with escape sequences) to standard error?

I can't use `stderr` because newlines of its input are automatically escaped. `halt_error` prints the way I want, but it halts the program. The only way I have been able to print a newline to stderr is through `debug` - its an unfortunate unaesthetic workaround, but it works.

I don't think that the behavior of `stderr` should change based on if jq is in raw mode. If it is not possible in one way or another to change stderr, I would welcome another function that yields the expected behavior.
Just wanted to register a `+1` on folks who have run into this surprising behavior.
`+1` from me, too.  I was really hoping to provide a customized warning, but since I cannot get `stderr` to actually create a newline, I guess I'm stuck with `debug`.
FWIW, I ended up here because I was trying to do something like this:

```
. as $line | try fromjson catch $line | stderr
```

I'm filtering ndjson logs and wanted to be able to use the `try fromjson` to filter valid json log lines that I could then pipe into further transforms, and wanted to be able to just redirect malformed lines to stderr

Patch:
diff --git a/src/builtin.c b/src/builtin.c
index b38d4c2f4f..8113ecd46d 100644
--- a/src/builtin.c
+++ b/src/builtin.c
@@ -1246,7 +1246,11 @@ static jv f_debug(jq_state *jq, jv input) {
 }
 
 static jv f_stderr(jq_state *jq, jv input) {
-  jv_dumpf(jv_copy(input), stderr, 0);
+  jq_msg_cb cb;
+  void *data;
+  jq_get_stderr_cb(jq, &cb, &data);
+  if (cb != NULL)
+    cb(data, jv_copy(input));
   return input;
 }
 
diff --git a/src/execute.c b/src/execute.c
index adf3773799..34bb63dc93 100644
--- a/src/execute.c
+++ b/src/execute.c
@@ -49,6 +49,8 @@ struct jq_state {
   void *input_cb_data;
   jq_msg_cb debug_cb;
   void *debug_cb_data;
+  jq_msg_cb stderr_cb;
+  void *stderr_cb_data;
 };
 
 struct closure {
@@ -1018,6 +1020,9 @@ jq_state *jq_init(void) {
   jq->debug_cb = NULL;
   jq->debug_cb_data = NULL;
 
+  jq->stderr_cb = NULL;
+  jq->stderr_cb_data = NULL;
+
   jq->err_cb = default_err_cb;
   jq->err_cb_data = stderr;
 
@@ -1249,6 +1254,16 @@ void jq_get_debug_cb(jq_state *jq, jq_msg_cb *cb, void **data) {
   *data = jq->debug_cb_data;
 }
 
+void jq_set_stderr_cb(jq_state *jq, jq_msg_cb cb, void *data) {
+  jq->stderr_cb = cb;
+  jq->stderr_cb_data = data;
+}
+
+void jq_get_stderr_cb(jq_state *jq, jq_msg_cb *cb, void **data) {
+  *cb = jq->stderr_cb;
+  *data = jq->stderr_cb_data;
+}
+
 void
 jq_halt(jq_state *jq, jv exit_code, jv error_message)
 {
diff --git a/src/jq.h b/src/jq.h
index 5269de3ff8..b44a6a5521 100644
--- a/src/jq.h
+++ b/src/jq.h
@@ -34,9 +34,10 @@ jv jq_get_error_message(jq_state *);
 typedef jv (*jq_input_cb)(jq_state *, void *);
 void jq_set_input_cb(jq_state *, jq_input_cb, void *);
 void jq_get_input_cb(jq_state *, jq_input_cb *, void **);
-
 void jq_set_debug_cb(jq_state *, jq_msg_cb, void *);
 void jq_get_debug_cb(jq_state *, jq_msg_cb *, void **);
+void jq_set_stderr_cb(jq_state *, jq_msg_cb, void *);
+void jq_get_stderr_cb(jq_state *, jq_msg_cb *, void **);
 
 void jq_set_attrs(jq_state *, jv);
 jv jq_get_attrs(jq_state *);
diff --git a/src/main.c b/src/main.c
index 48af5a31c1..963df5d511 100644
--- a/src/main.c
+++ b/src/main.c
@@ -253,6 +253,18 @@ static void debug_cb(void *data, jv input) {
   fprintf(stderr, "\n");
 }
 
+static void stderr_cb(void *data, jv input) {
+  if (jv_get_kind(input) == JV_KIND_STRING) {
+    int dumpopts = *(int *)data;
+    priv_fwrite(jv_string_value(input), jv_string_length_bytes(jv_copy(input)),
+        stderr, dumpopts & JV_PRINT_ISATTY);
+  } else {
+    input = jv_dump_string(input, 0);
+    fprintf(stderr, "%s", jv_string_value(input));
+  }
+  jv_free(input);
+}
+
 #ifdef WIN32
 int umain(int argc, char* argv[]);
 
@@ -684,6 +696,9 @@ int main(int argc, char* argv[]) {
   // Let jq program call `debug` builtin and have that go somewhere
   jq_set_debug_cb(jq, debug_cb, &dumpopts);
 
+  // Let jq program call `stderr` builtin and have that go somewhere
+  jq_set_stderr_cb(jq, stderr_cb, &dumpopts);
+
   if (nfiles == 0)
     jq_util_input_add_input(input_state, "-");
 


Test patch:
diff --git a/tests/shtest b/tests/shtest
index b2b8571d86..3f194c0fe2 100755
--- a/tests/shtest
+++ b/tests/shtest
@@ -232,6 +232,16 @@ grep "Expected string key after '{', not '\\['" $d/err > /dev/null
 echo '{"x":"y",["a","b"]}' | $JQ --stream > /dev/null 2> $d/err || true
 grep "Expected string key after ',' in object, not '\\['" $d/err > /dev/null
 
+# debug, stderr
+$VALGRIND $Q $JQ -n '"test", {} | debug, stderr' >/dev/null
+$JQ -n -c -j '"hello\nworld", null, [false, 0], {"foo":["bar"]}, "\n" | stderr' >$d/out 2>$d/err
+cat > $d/expected <<'EOF'
+hello
+worldnull[false,0]{"foo":["bar"]}
+EOF
+cmp $d/out $d/expected
+cmp $d/err $d/expected
+
 # --arg, --argjson, $ARGS.named
 $VALGRIND $JQ -n -c --arg foo 1 --argjson bar 2 '{$foo, $bar} | ., . == $ARGS.named' > $d/out
 printf '{"foo":"1","bar":2}\ntrue\n' > $d/expected