Draft: draft-ietf-jsonpath-base

Commentary: Anything else we expect to merge for this I - D deadline ? If not , I 'd like to submit . 
 great , , , thanks . 
 Issue 17 regular expressions in filters , which is supported by of JSONPath . This raises two questions : is the precise syntax of regular expressions ? should be mitigated ? One approach which addresses both these questions would be to adopt as discussed in , although not all languages may yet have support for RE2 . 
 So you are saying JSONpath should only support regular expression literals , not strings with regular expressions . 
 Unfortunately , no , I 'm saying that at the time I wrote that sentence , I had forgotten that Perl / Python / Ruby et al support variables inside the " / " delimiters . I wish I could interpret it the way you suggested . 
 Do you mean as in ( 1 ) interpolating variables into the RE ( ) or ( 2 ) filling in variables with results of the RE scan ( ) ? So far , I have n't seen any proposal for variables in the JSONPath expression language that could go into either place . 
 RE2 syntax would have the advantage that nobody of us likes it :-) But seriously , any RE syntax that is not already documented will need to be defined in the document . That leaves the ECMAscript syntax of the week or W3C XSD syntax . ECMAscript is a bit anemic but changes all the time . W3C XSD REs are exactly what would be needed here , but are not very popular . 
 wrote : Perhaps an expression e.g. that evaluated to a string should be allowed within " / " delimiters . I have n't thought this through , I only have experience implementing it with character literals , and the only has character literals . I 'll submit a pull request with JSONPath comparison with a match against an expression , to see what everybody else does . 
 I would prefer to restrict the right hand side of a regular expression match ( ) to a literal regular expression . I might be persuaded that we should also allow a JSONPath expression which evaluates to a ( single ) literal regular expression . I really would like to avoid literal regular expressions that contain JSONPath expressions as that would mean not being able to reuse an existing regular expression syntax ( such as RE2 ) . 
 After giving this some thought , I agree . Introducing " / " delimiters for regular expressions suggests static " compilation " of the regular expression , as it does in perl / ruby / python . If in  we 're applying a regular expression in a search of a million items , we would n't want the regular expression to depend on the data , as that would defeat the purpose of introducing the " / " delimiters in the first place . We would n't want to force implementations to " recompile " the regular expression every time the current node changed . Also , I do n't think my speculation above about allowing inside " / " delimiters makes sense , as the symbols allowed within " / " delimiters are reserved for regular expression symbols . These observations are consistent with the implementation of the " ~= " operator . In terms of functional requirements , regular expressions are very useful for search and filtering , but such expressions are generally statically known . I do n't fully understand this sentence . JSONPath expressions do n't produce character literals , character literals are a static concept . JSONPath expressions can produce strings , but that 's not the same thing . The key feature of character literals is that they do n't depend on the data , and in particular on the current node . I do n't think anybody would propose that ! 
 Well , then let me propose that :-) Building a RE out of data that has been found in the input is not that outlandish , I think . I ’m not sure that we ultimately want to support that , but it is not something that is self - evidently excluded .  ,    my confusion , but what I had in mind was to allow the likes of where evaluates to a regular expression such as . 
 wrote : One minor point , I believe it should say that it evaluates to a string , rather than character literals . Otherwise , I believe that that 's technically sound , and permits an efficient implementation , in the sense of " compile " regular expression once and apply many times . And it 's somewhat more likely that there is a regular expression to be found at an absolute location in the  document , as opposed to in all current nodes :-) I still think it 's an unlikely user requirement . If you do become persuaded to support this syntax , it probably wo n't be because I 've tried to persuade you . 
 The expression language could also contain something like , and it is likely to be helpful to compose the regular expression out of constant and variable ( from input item ) parts . What ’s out there in today ’s implementations ?  ,  
 wrote : As it turns out , no implementation supports terms in regular expressions . 
 I could get behind this . But I definitely would n't want to " construct " a RegEx from input data , as others have also been opposed to . 
 It would easy to come up with something , but I ’d rather not invent something — if there is no current practice for this , we should not include it .  ,    . Reminder : Charter says we 're supposed to specify something " based on the common semantics and other aspects of existing implementations " . So there 's a judgment call to be made as to whether enough of them support a common regex that we can reasonably claim it constitutes " common semantics " . My impression had been No , but it 's months since I looked at the overlap chart . 
 I can tell you neither of mine support RegEx ( one is obsolete now ) . But I 'm open to adding it to the current one . 
 I would really wish support for regex , and honestly I think they are nearly a must . However I 'm really worried about all the implications behind them . We are working hard for making sure that any future implementation is going to behave consistently with any other which follows our work . I 'm worried that all the complexity and differences behind regex is going to reintroduce back all the discrepancies we are trying hard to remove . I think that If we wish any regex support we need to go for a regex syntax which is well specified and comonly implemented . Honestly I 'm not happy with RE2 that has been mentioned before , as far as I know it is available to most of languages as a C library through bindings ( bindings might be really problematic ) . For instance I was n't happy with RE2 library while working with  and  , among all the reasons , using it would force my JSONPath implementation to depend on native code using a NIF while the whole library is 100 % Elixir compiled into BEAM byte code . IMHO we might investigate a Perl - like or POSIX - like regex support ( that are quite popular ) . But again we should avoid getting stuck into all the regex complexities . I have also some further points , questions and doubts : think that regex should be supported only as a constant , therefore they must not be taken from a JSONPath evaluation result ( e.g. disallow ) . We might change idea on a future revision , but right now I 'm worried that this might introduce further complexity , discrepancies and performance penalities ( we can parse and compile the regex while parsing the JSONPath ) . we going to treat regex as strings when using = = comparison ( e.g. is true when is ) or are we going to use any special type , therefore = = will never be true when comparing with a string ? we raise a parsing time error if a ill - formed regex is provided ( e.g. ) ? 
 My 2 ¢ : Any RE syntax we decide to use will need to be thoroughly documented in JSONPath , except if it is one of the ECMAscript versions or W3C XSD syntax . ( I do not really consider Posix a candidate , it is a bit anemic . ) Of course , we do n't need capture groups , global substitute , and can probably live without back references etc . That makes W3C XSD syntax my favorite , but it is also less widely implemented than others . It 's 2021 , so strings and REs will be Unicode text . Is it ( 1 ) string = ~ RE , or ( 2 ) RE = ~ string , or ( 3 ) both or even ( 4 ) both but with subtly different semantics ( horror experience from the Ruby language ) ? That is a sensible approach . Unless there is a large installed base that depends on computed REs , I agree . We could make sure that a later extension can be seamlessly . We have the luxury of being able to provide strong typing , so a RE literal should be a ( syntax ? ) error except near a . Yes . If we can check a query before applying it to input , we should do that with failing fast . 
 I tried to query this example [ 1 ] using a JSONPath with a XSD regex I tried using ( which is equivalent to ): Jayway : fails to understand it and returns a " bogus " result ( but it does n't fail with the " regular " regex ) Personal opinion about XSD regex : I feel like that no implementation is supporting them ( I 'm not aware of any ) , I feel like that every implementation supports whatever the language provides as builtin regex implementation ( which is likely far from XSD regex syntax ) . Please , let me know if any implementation is supporting that syntax . If none supports that syntax , are we really sure to go for a syntax that nobody is using ? Furthermore I did n't find any online tool to test regex with XSD style , conversely , PCRE style regex are easy to test ( this is not really encouraging ) . I also did few tests with some online test tools , following implementations just fail to understand any regex ( I tried with " basic " syntax ): jsonpath - plus ( javascript , 0.18.1 ) goessner : bogus result here , it replaces every title occurrence with -1 Personal opinion : previous results make me feel skeptic about the whole regex feature . Do we know which implementations are supporting regex ? [ 1 ] : 
 As far as I know XPath / XQuery regex : I feel like there is a lack modern tooling for testing   flavor regex ( outside XML schemas ) , and this might be quite frustrating for end users . By the way . 
 I did some further investigation into regex support ( using cburgmer 's tool and his implementations list ) . I usued a quite simple regex ( JSON input was : and  was : ) . They are supported by 11 implementations in 7 different languages [ 1 ] , interestingly JavaScript implementations were not supporting it , and 3/4 of them were completely misbehaving [ 2 ] : JavaScript brunerd " matches " everything JavaScript Goessner and jsonpath - plus replace matches with -1 I 'm worried that users might expect to use regex to reliably validate / filter out invalid inputs , but they get accepted anyway . Instead is not supported at all . Also I tested a XSD - like syntax , is supported by a further limited subset of implementations ( 5/11 ) . [ 1 ] List of implementations supporting : C++ ( 1/1 implementation ): jsoncons Go ( 2/6 implementations ):  ( 2/2 implementations ): Objective - C ( 1/1 implementation ): SMJJSONPath Ruby ( 1/1 implementation ):   ( 3/4 implementations ): galbar - jsonpath remorhaz - jsonpath softcreatr - jsonpath dotNET ( 1/4 implementations ):  . NET [ 2 ] JavaScript misbehaving implementations : Goessner : [ { " a " : -1 } , { " a " : -1 , " b " : " s " } ] brunerd : [ { " a " : " a " } , { " b " : " s " } ] jsonpath - plus : [ { " a " : -1 } , { " a " : -1 , " b " : " s " } ] 
 Interesting . While it may be a bit early to actually decide anything about regular expressions , this also seems to show some differences in expression evaluation . Can you do a similar check with and ? 
 it 's fine . Let me know what kind of output are you looking for , so I can summarize it in a helpful way . 
 what is your opinion about ECMA 262 ? All other things equal , and when it does n't matter too much , I am strongly in favour of staying consistent with other standards , whether IETF recommendation or de facto standards such as   .   ... this might be a way to go . Sharing a small , well defined subset of with   . 
 I think the purpose of the RFC is to give advice to implementers and users as to what works . It 's very unlikely that any existing implementation will change its semantics because of our work - but new ones will follow it if we do a good job . So I 'd like the RFC to say something like " Some implementations support regular expression filters in JSONPaths ; maximum interoperability can be achieved by using subset $   In particular , since regexes exhibit an 80/20 rule - you get 80 % of the benefit with 20 % of the features . Another reason a modest subset seems like a good idea for the JSONPath RFC . 
 Some historical context :   specifies 262 because that was what was available when the feature was . It maintains 262 because that 's what implementations now support . I do n't think this is cause enough for us to adopt it here . We should look at which RegEx specification serves our needs and use that . If it also happens to be 262 , that 's fine . On a side note of implementation support , neither of my implementations support RegEx , but I 'm happy to add it . I expect other maintainers would n't complain about it if it became a requirement . 
 The modest subset is what actually really works in a consistent manner : is supported by 100 % implementations , and Goessner 's article is enough for that ( I believe that some " basic " features are not supported by every implementation just because they are incomplete implementations , so no RFC is going to fix an unfinished or unmaintained project ) . I do n't see any useful reason about documenting just the " modest subset " ( and I do n't think it is worth it ) , also I think that is not compatible with our WG charter : I will quote   , that asked to change the original charter . The original charter :   : And the following has been proposed : Honestly what I found hard was taking decisions about quite commonly implemented " extensions " and a number of corner cases , and users love extensions ( such as regex ) because they find them useful . however if you are willing to work on it , we might document a " JSONPath Core " subset , that can be safely used across implementations for portability reasons , and for transitional purposes , but I do n't agree about just limiting to it . Sorry for the OT , let 's continue the regex topic . Edit : I wrote a quick proposal that tries to match the 2 point of view I see here ( and in other discussions ): 
 Talking about regex : I think we should start again from requirements . I 'll start with some of them : Good enough for matching URLs , domain names , mail addresses , IP addresses , UUIDs and all the stuff that people keep asking on stackoverflow Reasonable unicode support ( it is 2021 , honestly I rather not support them than supporting a limited version that is mostly good for Latin1 ) . 
 I think we all agree : JSONPath must stipulate a regular expression syntax for use in filters , and it 's not the role of JSONPath to define a new regular expression standard . Fortunately , there 's a clean syntactic and semantic separation between JSONPath and regular expression ( de facto or de jure ) standards . I suggest we then stipulate RE2 for its security properties and unicode support ( with " SHOULD " language ) and allow for other regular expression standards ( with " MAY " language ) . JSONPath implementations that share a regular expression standard will interoperate on JSONPaths involving regular expressions . JSONPath implementations which do n't share a regular expression standard may or may not interoperate on particular JSONPaths , depending on the regular expressions used . In terms of compliance testing , it may be sufficient to include a few simple regular expressions which will work " everywhere " . 
 I see your intent here , but I have to say that I 'm not a fan of pure wrappers ( especially as I work in .Net ) , and it seems that all of the implementations for RE2 are just wrappers for the C++ implementation . There are a lot of benefits to developers when a library is written in the language . I do n't think we should ignore that . Let 's make sure that is considered in this decision . 
 Go has RE2 built in ( ) , but point taken . I agree that a wrapper is to be avoided , especially if it makes shipping a static binary impossible . Which other regex ( de facto or de jure ) standards support unicode ? 
 It 's 2021 , so all ( notable ) RE standards support Unicode . I think , for once , recent drafts from json- got this right : Let 's define a common subset , and reference that to an existing written up standard such as ECMAScript . ( json- got the idea right , not necessarily the specific subset -- there is quite some fluff in that . ) If nobody else wants , I 'll do a PR with an ABNF definition of such a subset . We can then debate what to leave in / throw out / add . 
 Example that could be solved using RegEx . 
 Another on the usage of RegEx in filters . 
 I find it interesting that  does not support regular expressions in filters . See , for example , . 
 The discussion found in this issue pretty much is the reason why iregexp is the right approach . ( The alternative , escaping into a platform - specific function , is not available for the present standardization effort . ) 
 Is iregexp the right approach for JMESPath too ? If so , I wonder if the JMESPath community would be interested in collaborating on iregexp ? 
 We could ask ! We can use all the help we can get . 
 Well , maybe I can get iregexp-02 out first ... 
 Sounds good ! 
 112 output : 
 Looks good now with the most recent changes . Thanks .
Paragraph (version B):
To be valid, strings on the right hand side of the "=~" regex
matching operator need to conform to I-D.draft-bormann-jsonpath-
iregexp.
Paragraph (version A):
Types are not implicitly converted in comparisons.  So ""13 ==
'13'"" selects no node.
Paragraph (version B):
Data types are not implicitly converted in comparisons.  So ""13
== '13'"" selects no node.
Paragraph (version A):
Regular expression tests can be applied to "string" values only.
Paragraph (version B):
The regular expressions in the string-literals on the right-hand
side of "=~" are as defined in I-D.draft-bormann-jsonpath-iregexp.
Regular expression tests can be applied to JSON string values
(RFC8259) only (on the left-hand side of "=~"); they yield false
otherwise.