Repository: jqlang/jq

Problem statement:
Broken cyrillic in string literals.
File example: [jq.txt](https://github.com/stedolan/jq/files/716344/jq.txt).
Command: `jq -n -f jq.txt`.

I expect that output contains equal strings under the paths `validation.rules.rule3.errorMessages.regexNotMatched` and `validation.rules.rule4.errorMessages.regexNotMatched`, but `rule3` contains altered 5th symbol (and it does not seem to be valid utf8 codepoint).

Similar replacements somewhere in first two words occure with modifications in the 7th line (marked with comment "magic line"), if a) remove comment (with space after comma), b) remove text except leading six spaces, c) remove all text, leaving blank line.

jq-1.5rc2-152-g0b82185 (and same errors with jq-1.5 from Centos 7 repository).

Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

--
Alexey Golovko, a.golovko1@tinkoff.ru


Hints:
[NOTE: FURTHER ANALYSIS SHOWED THE FIRST ASSERTION IN THE FIRST PARAGRAPH TO BE INCORRECT.]

The two strings in jq.txt are NOT the same, which you can see by using `explode`, e.g. by modifying the very last line of jq.txt to

```
off_step_2 | (
  (.validation.rules.rule3.errorMessages.regexNotMatched | explode)
  ,
  (.validation.rules.rule4.errorMessages.regexNotMatched | explode)
)

```
The first string contains two occurrences of

     'REPLACEMENT CHARACTER' (U+FFFD)
 
This character (UTF-32 (decimal) 65533) does not appear in the second string.

You can read about U+FFFD, which appears in the "specials" block, at http://www.fileformat.info/info/unicode/char/0fffd/index.htm


These replacement symbols are added by jq itself. These lines are identical in jq.txt.

You can verify this with `grep regexNotMatched jq.txt | tail -n1 | head -n1 | md5sum` and `grep regexNotMatched jq.txt | tail -n2 | head -n1 | md5sum`.

Even your variant with explode returns the same results, if you remove (unused) definition for off_step_1.
@agolovko1-at-tinkoff-ru - You are right!  This is weirder than anything else I've seen using jq, which is why I was looking for a "simple" explanation.  Thanks for pressing the issue.

@nicowilliams, @wtlangford  -- jq 1.3 does not suffer from the problem!

The insertion of U+FFFD is done in jv.c:jvp_string_copy_replace_bad (thanks to ff48bd6ec538b01d1057be8e93b94eef6914e9ef of June 22, 2013):

    /* Copy a UTF8 string, replacing all badly encoded points with U+FFFD */

Note also that version 1.2 of the utility isutf8 reports that jq.txt is valid UTF8.
    curl -Lg https://github.com/stedolan/jq/files/716344/jq.txt|less

Doesn't look like UTF-8 to me.
@nicowilliams - Glad to see you back in jq land.

Regarding this particular issue, I think @agolovko1-at-tinkoff-ru has indeed uncovered a weird bug.

First, as I mentioned, isutf8 does indicate the entire file is UTF-8, and I cannot see any part that isn't.

Second, it's weird that including an irrelevant `def` should result in erroneous behavior of an independent `def`.

In any case, if there were something in the file that isn't UTF-8 but that is causing problems, one would hope that jq would issue some kind of warning or error message.

Could it be that the ff48bd6 update exposes some arcane memory management issue?

I actually played with this the other day and forgot to respond.  Try
adding a single space to the front of the file and running it.
`jq -n -f <(cat <(echo -n " ") jq.txt)` in my shell does this without
letting any program other than jq attempt to parse any UTF-8 sequences
(keeping vim and friends from trying to "help").
The extra space on the front causes it to print properly.  In fact, any odd
number of bytes on the front of the file causes it to print properly.
The implication here is that there's a bug in jq's UTF-8 parser.

On Mon, Jan 23, 2017 at 1:59 PM pkoppstein <notifications@github.com> wrote:

> @nicowilliams <https://github.com/nicowilliams> - Glad to see you back in
> jq land.
>
> Regarding this particular issue, I think @agolovko1-at-tinkoff-ru
> <https://github.com/agolovko1-at-tinkoff-ru> has indeed uncovered a weird
> bug.
>
> First, as I mentioned, isutf8 does indicate the entire file is UTF-8, and
> I cannot see any part that isn't.
>
> Second, it's weird that including an irrelevant def should result in
> erroneous behavior of an independent def.
>
> In any case, if there were something in the file that isn't UTF-8 but that
> is causing problems, one would hope that jq would issue some kind of
> warning or error message.
>
> Could it be that the ff48bd6
> <https://github.com/stedolan/jq/commit/ff48bd6ec538b01d1057be8e93b94eef6914e9ef>
> update exposes some arcane memory management issue?
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/stedolan/jq/issues/1311#issuecomment-274583076>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ADQ4V-UoQ9Ukza9qPZkFbHyGGvRnKeITks5rVPh4gaJpZM4Ln9-2>
> .
>

@wtlangford - Yes, I can confirm that the extra space makes a difference on my machine too.  Thanks.
Reopening this because it seems the issue is related to a UTF-8 byte sequence spanning across byte offset `0xffff`, meaning this is likely a buffer-size problem.
Further investigation yields that the issue doesn't occur when reading json inputs, but in parsing jq programs themselves, which means this bug comes from flex/bison.  This may be significantly harder to resolve.
Ah, sorry for closing this too soon.
Hmmm, `jq -n '"საერთაშორისო"'` works (this string is just a copy-paste from https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html).
Oh, I see.  This really is weird!
Good news! This is not a flex bug.  It's in `jv_load_file`.  I'm working on a fix for it.
@wtlangford Nice!

Patch:
diff --git a/Makefile.am b/Makefile.am
index 2a3aded4cd..c1eaf6deca 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -115,7 +115,7 @@ endif
 
 ### Tests (make check)
 
-TESTS = tests/optionaltest tests/mantest tests/jqtest tests/onigtest tests/shtest
+TESTS = tests/optionaltest tests/mantest tests/jqtest tests/onigtest tests/shtest tests/utf8test
 TESTS_ENVIRONMENT = NO_VALGRIND=$(NO_VALGRIND)
 
 
diff --git a/src/jv_file.c b/src/jv_file.c
index 33d327c7ce..3159df533f 100644
--- a/src/jv_file.c
+++ b/src/jv_file.c
@@ -4,6 +4,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include "jv.h"
+#include "jv_unicode.h"
 
 jv jv_load_file(const char* filename, int raw) {
   FILE* file = fopen(filename, "r");
@@ -20,11 +21,23 @@ jv jv_load_file(const char* filename, int raw) {
     data = jv_array();
     parser = jv_parser_new(0);
   }
+
+  // To avoid mangling UTF-8 multi-byte sequences that cross the end of our read
+  // buffer, we need to be able to read the remainder of a sequence and add that
+  // before appending.
+  const int max_utf8_len = 4;
+  char buf[4096+max_utf8_len];
   while (!feof(file) && !ferror(file)) {
-    char buf[4096];
-    size_t n = fread(buf, 1, sizeof(buf), file);
+    size_t n = fread(buf, 1, sizeof(buf)-max_utf8_len, file);
+    int len = 0;
+    if (jvp_utf8_backtrack(buf+(n-1), buf, &len) && len > 0) {
+      if (!feof(file) && !ferror(file)) {
+        n += fread(buf+n, 1, len, file);
+      }
+    }
+
     if (raw) {
-      data = jv_string_concat(data, jv_string_sized(buf, (int)n));
+      data = jv_string_append_buf(data, buf, n);
     } else {
       jv_parser_set_buf(parser, buf, n, !feof(file));
       jv value;
diff --git a/src/jv_unicode.c b/src/jv_unicode.c
index fbf7454be3..b3a50b2d0c 100644
--- a/src/jv_unicode.c
+++ b/src/jv_unicode.c
@@ -3,6 +3,29 @@
 #include "jv_unicode.h"
 #include "jv_utf8_tables.h"
 
+// jvp_utf8_backtrack returns the beginning of the last codepoint in the
+// string, assuming that start is the last byte in the string.
+// If the last codepoint is incomplete, returns the number of missing bytes via
+// *missing_bytes.  If there are no leading bytes or an invalid byte is
+// encountered, NULL is returned and *missing_bytes is not altered.
+const char* jvp_utf8_backtrack(const char* start, const char* min, int *missing_bytes) {
+  assert(min < start);
+  if (min == start) {
+    return min;
+  }
+  int length = 0;
+  int seen = 1;
+  while (start >= min && (length = utf8_coding_length[(unsigned char)*start]) == UTF8_CONTINUATION_BYTE) {
+    start--;
+    seen++;
+  }
+  if (length == 0 || length == UTF8_CONTINUATION_BYTE || length - seen < 0) {
+    return NULL;
+  }
+  if (missing_bytes) *missing_bytes = length - seen;
+  return start;
+}
+
 const char* jvp_utf8_next(const char* in, const char* end, int* codepoint_ret) {
   assert(in <= end);
   if (in == end) {
diff --git a/src/jv_unicode.h b/src/jv_unicode.h
index 579c910eea..558721a8fd 100644
--- a/src/jv_unicode.h
+++ b/src/jv_unicode.h
@@ -1,6 +1,7 @@
 #ifndef JV_UNICODE_H
 #define JV_UNICODE_H
 
+const char* jvp_utf8_backtrack(const char* start, const char* min, int *missing_bytes);
 const char* jvp_utf8_next(const char* in, const char* end, int* codepoint);
 int jvp_utf8_is_valid(const char* in, const char* end);
 


Test patch:
diff --git a/tests/jq.test b/tests/jq.test
index 630c344d0f..ad4a51edad 100644
--- a/tests/jq.test
+++ b/tests/jq.test
@@ -1312,4 +1312,3 @@ jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shel
 (.[{}] = 0)?
 null
 
-
diff --git a/tests/utf8-truncate.jq b/tests/utf8-truncate.jq
new file mode 100644
index 0000000000..a6be86378b
--- /dev/null
+++ b/tests/utf8-truncate.jq
@@ -0,0 +1,3 @@
+def fill:"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+def e:"日本語";
+e == "日本語"
diff --git a/tests/utf8test b/tests/utf8test
new file mode 100755
index 0000000000..570731bbec
--- /dev/null
+++ b/tests/utf8test
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+. "${0%/*}/setup" "$@"
+
+if [ "`$VALGRIND $Q $JQ -nf $JQTESTDIR/utf8-truncate.jq`" != "true" ]; then
+	echo "UTF-8 byte sequences that span the jv_load_file read buffer are mangled"
+	exit 1
+fi
+
+exit 0