Repository: jqlang/jq

Problem statement:
index() function returns wrong offset for non-ascii chars
I'm trying to strip away some text from part of a text. Trying to use something like `sub("!.*"; "")` doesn't work, as it is giving me a Segmentation fault when text is too long. So I tried to go this route:

`$ jq '.msg | .[0:index("!")]'`

which works fine with input like:
`{"msg": "hello world!"}
`
but fails when text contains wide characters:
`{"msg": "здравствуй мир!"}
`
```
$ echo '{"msg": "здравствуй мир!"}' | jq '.msg | index("!")'
27
```

```
$ echo '{"msg": "hello world!"}' | jq '.msg | index("!")'
11
```
```
$ jq --version
jq-1.5
$ uname -a
Darwin atschabu-C02SF0UTG8WM 15.6.0 Darwin Kernel Version 15.6.0: Tue Apr 11 16:00:51 PDT 2017; root:xnu-3248.60.11.5.3~1/RELEASE_X86_64 x86_64
```
`indices` reports byte offsets instead of character offsets
**Describe the bug**
jq uses _characters_ to index strings.
To see that, we can run `"🇬🇧oo" | .[0 : 1,2,3,4]`, which yields "🇬" "🇬🇧" "🇬🇧o" "🇬🇧oo".
Note that 🇬🇧 is actually two characters and 8 bytes, as we can see from `"🇬🇧" | length, utf8bytelength`.
However, the `indices` filter returns _byte offsets_  to the pattern in the string.
The documentation does not specify the behaviour of `indices` for UTF-8 strings, but given that `length` and `.[x:y]` use character counts to index strings, it is likely that this is a bug and not just undocumented behaviour.

**To Reproduce**
$ ./jq-linux-amd64-1.7.1 -nc '"🇬🇧oo" | indices("o")'
[8,9]
$ ./jq-linux-amd64-1.7.1 -nc '"ƒoo" | indices("o")'
[2,3]

**Expected behavior**
$ ./jq-linux-amd64-1.7.1-fixed -nc '"🇬🇧oo" | indices("o")'
[2,3]
$ ./jq-linux-amd64-1.7.1-fixed -nc '"ƒoo" | indices("o")'
[1,2]

The problem is probably caused in [jv_string_indexes](https://github.com/jqlang/jq/blob/c95b34ff827d05a2d262f00280a4891a295ed0ed/src/jv.c#L1272).


Hints:
There is some documentation about this on the "Pitfalls" page (https://github.com/stedolan/jq/wiki/How-to:-Avoid-Pitfalls)

In brief, you can use `match/1`:

    echo '{"msg": "здравствуй мир!"}' | jq '.msg | match("!").offset'
    14

This works in jq 1.5 and later.

By the way, could you please give more details about the failure of `sub/2`. Here is an illustration that it does not always fail when given a long string:

     jq1.5 -n '[range(0;100000) | "a"] | join("") + "!xx" | sub("!.*";"") | length'
    100000



My bad. I haven't even realized there is a wiki. I took all the information from the manual, which didn't mention anything about index being byte wise. I'll give match a go.

I still haven't figured out when exactly the Segmentation fault is happening, as I couldn't find the input yet which is producing it. But I went by the assumption it is related to issue [922](https://github.com/stedolan/jq/issues/922) until I can proof the opposite.

I guess we can close this one, and I'll open a new ticket, in case my segmentation fault issue is not related to 922.
No, this is a bug.  We should fix it.
This is a dup of #1430, #1624.

Patch:
diff --git a/src/jv.c b/src/jv.c
index e23d8ec124..c821d64aa9 100644
--- a/src/jv.c
+++ b/src/jv.c
@@ -1274,15 +1274,21 @@ jv jv_string_indexes(jv j, jv k) {
   assert(JVP_HAS_KIND(k, JV_KIND_STRING));
   const char *jstr = jv_string_value(j);
   const char *idxstr = jv_string_value(k);
-  const char *p;
+  const char *p, *lp;
   int jlen = jv_string_length_bytes(jv_copy(j));
   int idxlen = jv_string_length_bytes(jv_copy(k));
   jv a = jv_array();
 
   if (idxlen != 0) {
-    p = jstr;
+    int n = 0;
+    p = lp = jstr;
     while ((p = _jq_memmem(p, (jstr + jlen) - p, idxstr, idxlen)) != NULL) {
-      a = jv_array_append(a, jv_number(p - jstr));
+      while (lp < p) {
+        lp += jvp_utf8_decode_length(*lp);
+        n++;
+      }
+
+      a = jv_array_append(a, jv_number(n));
       p++;
     }
   }


Test patch:
diff --git a/tests/jq.test b/tests/jq.test
index b94f29d245..85512fdf90 100644
--- a/tests/jq.test
+++ b/tests/jq.test
@@ -1345,6 +1345,22 @@ indices(", ")
 "a,b, cd,e, fgh, ijkl"
 [3,9,14]
 
+index("!")
+"здравствуй мир!"
+14
+
+.[:rindex("x")]
+"正xyz"
+"正"
+
+indices("o")
+"🇬🇧oo"
+[2,3]
+
+indices("o")
+"ƒoo"
+[1,2]
+
 [.[]|split(",")]
 ["a, bc, def, ghij, jklmn, a,b, c,d, e,f", "a,b,c,d, e,f,g,h"]
 [["a"," bc"," def"," ghij"," jklmn"," a","b"," c","d"," e","f"],["a","b","c","d"," e","f","g","h"]]