Repository: jqlang/jq

Problem statement:
Zero length regular expression match misbehaviour

## Description
Some regular expressions behave weirdly when matches are zero length - they miss matches or match too often.

## Reproduction

### Case 1
Input:
```
jq --null-input --compact-output '"xx"|match("";"g")'
```
Output:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
```
Expected:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
{"offset":2,"length":0,"string":"","captures":[]}
```
There should be a match at the end of the string too.

### Case 2
Input:
```
$ jq --null-input --compact-output '"xx"|match("$";"g")'
```
Output:
```
{"offset":2,"length":0,"string":"","captures":[]}
{"offset":2,"length":0,"string":"","captures":[]}
```
Expected:
```
{"offset":2,"length":0,"string":"","captures":[]}
```
The same match shouldn't show up twice. (Extra matches seem to be returned in number equal to the length of the string, for some reason. But they're all the same match at the same offset.)

## Environment
 - OS and Version: Ubuntu
 - jq version: master branch, commit cff5336ec71b6fee396a95bb0e4bea365e0cd1e8

## Additional context
I stumbled on this trying to fix https://github.com/stedolan/jq/issues/2148 and found that my [attempted solution](https://github.com/stedolan/jq/pull/2564) didn't behave like I expected it to. I tried to isolate the problem and ended up with these.

Zero length regular expression match misbehaviour

## Description
Some regular expressions behave weirdly when matches are zero length - they miss matches or match too often.

## Reproduction

### Case 1
Input:
```
jq --null-input --compact-output '"xx"|match("";"g")'
```
Output:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
```
Expected:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
{"offset":2,"length":0,"string":"","captures":[]}
```
There should be a match at the end of the string too.

### Case 2
Input:
```
$ jq --null-input --compact-output '"xx"|match("$";"g")'
```
Output:
```
{"offset":2,"length":0,"string":"","captures":[]}
{"offset":2,"length":0,"string":"","captures":[]}
```
Expected:
```
{"offset":2,"length":0,"string":"","captures":[]}
```
The same match shouldn't show up twice. (Extra matches seem to be returned in number equal to the length of the string, for some reason. But they're all the same match at the same offset.)

## Environment
 - OS and Version: Ubuntu
 - jq version: master branch, commit cff5336ec71b6fee396a95bb0e4bea365e0cd1e8

## Additional context
I stumbled on this trying to fix https://github.com/stedolan/jq/issues/2148 and found that my [attempted solution](https://github.com/stedolan/jq/pull/2564) didn't behave like I expected it to. I tried to isolate the problem and ended up with these.



Hints:
Okay, having looked at f_match, in particular this [handling for zero-width matches](https://github.com/stedolan/jq/blob/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/src/builtin.c#L918-L932) and the [loop termination condition](https://github.com/stedolan/jq/blob/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/src/builtin.c#L998), I see some issues. Here's another test case:
```
jq --null-input --compact-output '"xx"|match("|x";"g")'
```
This outputs:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
```
But it should output:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":0,"length":1,"string":"x","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
{"offset":1,"length":1,"string":"x","captures":[]}
{"offset":2,"length":0,"string":"","captures":[]}
```
[Example using PCRE on regex101](https://regex101.com/r/mOsYlG/1)
Hmmm, oniguruma doesn't make this easy. There doesn't seem to be any way to call `onig_search` to get the next match that isn't a zero-width match at the start position. It seems you can _either_ call it with a callback and get _all_ the matches, which could be very inefficient, or you call it with a start offset and get only one match. This also makes it clear that my approach in #2564 is going to be way more costly than I thought, because I was assuming that `match(...; "g")` returns results lazily, and it doesn't look like it does.

I wonder if you would get the right result if you called it with the callback and filtered out every match that overlaps with the previously replaced match.
This seems very thorny. It might be worth reviewing what other users of oniguruma do to implement this. It doesn't even seem like there's universal agreement between regex engines, especially with awkward cases like `"xx"|match("x?";"g")` and `"xx"|match("|x";"g")`
Please note that Case 2 in the original post has been resolved by 83f375cc831039396167d4d2b5f901f4b33a8707 :
```
jqMaster -nM '"xx"|[match("$";"g")] | length'
1
```

Note also that in this repaired version of jq, the following is nevertheless incorrect:
```
jqMaster -nM '"x"|sub("";"y"; "g")'
"yx"
```

Okay, having looked at f_match, in particular this [handling for zero-width matches](https://github.com/stedolan/jq/blob/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/src/builtin.c#L918-L932) and the [loop termination condition](https://github.com/stedolan/jq/blob/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/src/builtin.c#L998), I see some issues. Here's another test case:
```
jq --null-input --compact-output '"xx"|match("|x";"g")'
```
This outputs:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
```
But it should output:
```
{"offset":0,"length":0,"string":"","captures":[]}
{"offset":0,"length":1,"string":"x","captures":[]}
{"offset":1,"length":0,"string":"","captures":[]}
{"offset":1,"length":1,"string":"x","captures":[]}
{"offset":2,"length":0,"string":"","captures":[]}
```
[Example using PCRE on regex101](https://regex101.com/r/mOsYlG/1)
Hmmm, oniguruma doesn't make this easy. There doesn't seem to be any way to call `onig_search` to get the next match that isn't a zero-width match at the start position. It seems you can _either_ call it with a callback and get _all_ the matches, which could be very inefficient, or you call it with a start offset and get only one match. This also makes it clear that my approach in #2564 is going to be way more costly than I thought, because I was assuming that `match(...; "g")` returns results lazily, and it doesn't look like it does.

I wonder if you would get the right result if you called it with the callback and filtered out every match that overlaps with the previously replaced match.
This seems very thorny. It might be worth reviewing what other users of oniguruma do to implement this. It doesn't even seem like there's universal agreement between regex engines, especially with awkward cases like `"xx"|match("x?";"g")` and `"xx"|match("|x";"g")`
Please note that Case 2 in the original post has been resolved by 83f375cc831039396167d4d2b5f901f4b33a8707 :
```
jqMaster -nM '"xx"|[match("$";"g")] | length'
1
```

Note also that in this repaired version of jq, the following is nevertheless incorrect:
```
jqMaster -nM '"x"|sub("";"y"; "g")'
"yx"
```


Patch:
diff --git a/src/builtin.c b/src/builtin.c
index 0fd7554373..3cde686618 100644
--- a/src/builtin.c
+++ b/src/builtin.c
@@ -1002,7 +1002,7 @@ static jv f_match(jq_state *jq, jv input, jv regex, jv modifiers, jv testmode) {
             jv_string((char*)ebuf)));
       break;
     }
-  } while (global && start != end);
+  } while (global && start <= end);
   onig_region_free(region,1);
   region = NULL;
   if (region)


Test patch:
diff --git a/tests/onig.test b/tests/onig.test
index d49391fc09..3edb07dff0 100644
--- a/tests/onig.test
+++ b/tests/onig.test
@@ -1,12 +1,16 @@
 # match builtin
 [match("( )*"; "g")]
 "abc"
-[{"offset":0, "length":0, "string":"", "captures":[]},{"offset":1, "length":0, "string":"", "captures":[]},{"offset":2, "length":0, "string":"", "captures":[]}]
+[{"offset":0, "length":0, "string":"", "captures":[]}, {"offset":1, "length":0, "string":"", "captures":[]}, {"offset":2, "length":0, "string":"", "captures":[]}, {"offset":3, "length":0, "string":"", "captures":[]}]
 
 [match("( )*"; "gn")]
 "abc"
 []
 
+[match(""; "g")]
+"ab"
+[{"offset":0,"length":0,"string":"","captures":[]},{"offset":1,"length":0,"string":"","captures":[]},{"offset":2,"length":0,"string":"","captures":[]}]
+
 [match("a"; "gi")]
 "āáàä"
 []
@@ -71,29 +75,27 @@ gsub("a";"b")
 "aaaaa"
 "bbbbb"
 
-gsub( "(.*)"; "";  "x")
+gsub("(.*)"; ""; "x")
 ""
 ""
 
-gsub( ""; "a";  "g")
+gsub(""; "a"; "g")
 ""
 "a"
 
-gsub( "^"; "";  "g")
+gsub("^"; ""; "g")
 "a"
 "a"
 
-
-# The following is a regression test and should not be construed as a requirement other than that execution should terminate:
-gsub( ""; "a";  "g")
+gsub(""; "a"; "g")
 "a"
-"aa"
+"aaa"
 
-gsub( "$"; "a";  "g")
+gsub("$"; "a"; "g")
 "a"
 "aa"
 
-gsub( "^"; "a")
+gsub("^"; "a")
 ""
 "a"
 
@@ -163,5 +165,5 @@ sub("(?<x>.)"; "\(.x)!")
 # splits and _nwise
 [splits("")]
 "ab"
-["","a","b"]
+["","a","b",""]