Change:
Fixes URL
At today's meeting, I agreed to submit a PR to float the idea of partial determinism for the descendants operator, even though the consensus seems to be against any determinism for that operator. Given the descendants operator PR URL isn't yet merged, I'll start with an issue to capture discussion and avoid rebasing effort. We probably don't want to force any determinism on implementations, so the idea is to provide a recommendation for implementations which want to provide partial determinism and use "SHOULD" language so the recommendation is optional. Then implementations which opt in will agree on the ordering recommentations. The benefits of partial determinism are: improved predictability and interoperation relative ease for JSONPath users writing tests, as there are fewer possible outputs. In PR URL there are no constraints on the non-determinism of the descendants operator: We agreed in issue URL to allow non-determinism in the order in which objects are iterated and the same should be true when the descendants operator encounters an object. But there are other details of the ordering which could be tied down: arrays SHOULD be visited in array order nodes SHOULD be visited strictly before their descendants (so-called pre-order traversal). (Post-order traversal, in which nodes are visited strictly after their descendants, would be an equally valid alternative.) One of the reasons given at today's meeting for allowing full non-determinism was for parallel processing, so it would be interesting to discuss what constraints the above rules produce on parallel processing. The order of collection of the results seems to be more significant than whether or not parallel processing is used.
My preference is pre-order traversal (node then descendants).
Except that I don't know what's "SHOULD" about that, I agree with these partial ordering guidelines. If we want to introduce different compliance levels, that is fine, but it SHOULD :-) not be coupled to using the term SHOULD vs. MUST. "Full compliance requires..." vs. "For conditional compliance, at least..."
I probably misunderstood , or applied it atypically. I am comfortable with using NAME compliance level language instead.
We need to be careful here that we don't optimize for tests. It seems to be a focus of this discussion that we want to make verification of output in the test suite easier. Rather, we need to focus on supportability and interoperability. If that aligns with running the test, that's good. But don't let making running the tests easier at the cost of making things significantly more difficult for implementors. It's non-trivial, but also not very complex, to verify unordered sets of nodes in output.
More significant than complications for the JSONPath test suite is the amount of non-determinism that tests of code which uses JSONPath will have to accommodate.
I recommend checking out the language around interoperability in 8259. Something like that might work here.
I'm having a hard time judging the consensus, or lack of it, here. Regardless of the detailed language, please could interested parties indicate which of these options they favour: A. Don't tie down the non-determinism of at all B. Require to visit arrays in array order and visit nodes before their descendants C. Some other option involving B with the option of A or A with the option of B, using some kind of language TBD. /cc NAME
I’m not sure you actually mean that, but for me that would be about the worst result we could arrive at. 8259 was worked on in a politicized context, which simply didn’t allow us to clean it up. There barely was the courage to finally say that interchanging JSON implies UTF-8, which was at least some progress. But there was no way to clarify the data model issues, and this is why JSON in practice essentially has been reduced to I-JSON, which is not bad at all, but really different from what JSON originally promised to be. I’d prefer if we don’t need an I-JSONPath document to clean up the interoperability issues left in JSONPath-base. Given that we have JSON and I-JSON, it may be useful to acknowledge (or violently reject! — we get to choose) the emergence of multiple levels/classes of interoperability with JSONPath, too. Grüße, Carsten
I would vote for this option, but in the same time we have slices with negative steps, which naturally do sort of "backwards" iteration in Python. - while there's no clear consensus, we see that all implementations with "python-compatible" slices return values in reverse order, no exceptions. If we determine order of array items visiting (thus determining the order of selected nodes in output), shouldn't we also take into consideration the effect of selectors, and if yes - then should the order of union items affect the order of visit? We can consider part of selectors as "order neutral" and others are "order affecting", with affecting selectors overriding natural order. At first glance I see slices and unions as order-affecting selectors.
NAME I don't follow. The ordering of is not affected by other selectors (e.g. involving negative array steps or unions).
This is not absolutely clear for me. Let us have the following structure: And we apply query. At first we apply selector to the root node, and it returns following nodes: : : : : : : : : : : : Then we apply selector to each these nodes. Applying slice to number will return empty node list, so I consider only arrays below: : : : : : : : : : : : : : The resulting node list is: : : : : : : : : : : I didn't notice any problems following both "natural order" while applying and "reversed order" while applying , because they never apply simultaneously. Please show me if I missed something in my reasoning.
The idea of determinism in arrays itself seems important to me. For example, let some service provides array of some data in some garanteed order that is important for end user, and and another service wants to use JSONPath to filter them on some fancy condition. If JSONPath provides determined order of results, then it's at least possible to solve the problem with a single query. If no - we'll have no easy way to do it. We will need to extract from JSON original order somehow (with alternative tool or manual parsing), and then use it to rebuild original order of filtered items. It will be much more convenient to receive results in a determined order.
The main aspect of JSON that leads to non-determinism is the fact that no specification makes any guarantee that the order of Members in an Object carries any meaning. RFC8259, last para of Section 4 , explicitly cautions against relying on this. On the other hand, order in arrays is explicitly significant and I would be shocked if application of array-based selectors did not produce identical results in all implementations. Or am I missing something obvious?
Yes, I think we agree that array order should be preserved and that object order is not guaranteed, but I'd like this issue to focus specifically on the ordering of .
Added text indicating node-first / children-second ordering.
... and preserving array order.
Sure. That too. But I don't think that was ever in question.
Does applying a JSONPath query to a JSON data item always generate the same result? Some selectors do not have a natural order of the nodelists they return, e.g., $. applied to a JSON object We could prescribe an order (e.g., sorting keys lexicographically or in JavaScript order) if determinism is important We could disallow . on JSON objects
hmm .. depends, how deterministic JSON parsers are when iterating over objects ... they should work in ascending chronological order ... URL JavaScript 'URL' also guaranties the same order URL So ... order should be always the same, as long the implementation is the same. How different implementations behave is uncertain at best. But why should we care and heal deficiencies of JSON parsers in that case? I can't see a benefit of always guaranteeing the same -whatever- order. And disallowing iterating over objects by '$.*' might break existing implementations.
A minor benefit would be that the Compliance Test Suite would be simpler if the output always had a defined implementation-independent ordering. Agreed.
(Determinism in interop testing is a major benefit for me. But that can be added by reprocessing after the JSONpath processing, so it is not completely a killer argument.)
Another selector which requires careful definition for ordering is , particularly when this recurses over an object or objects.
hmm ... compliance testing is a point, sure. Simple first tests of and regarding ordering should be possible with current test suite ... ?
Anything is "possible". ;-) Tests involving objects which have only one name/value pair are trivially easy to test because any non-determinism of object enumeration is irrelevant. But tests involving objects with two or more name/value pairs need to capture the ordering possibilities somehow. Note that the is a JSON document so there would need to be a declarative way of specifying the ordering possibilities.
We covered this topic over in the JSON Schema repo somewhere with quite a lengthy discussion. I'll see if I can dig it up. In short, parsers in some languages just can't be made to be deterministic. Reading the same object from a file (where the keys are certainly in a defined sequence) will yield differing orderings when printing or even debugging the value. This topic, therefore, falls into the "language agnosticity" category. I don't think it's something we can enforce.
is accepted (and correctly evalueted) by ~93% of the implementations on the . Each of them has a number of users, it is really important to not disrupt their existing expressions and their habits. I don't think determinism is that important, on a object evals to an array but semantically it is an unordered set (which is not a JSON type, therefore a different type is used ¹) which is still a meaningful result and developers should remember that. Hence we shouldn't discard this feature. Anyway what I found confusing when implementing was the lack of any official recommendation about on a object behaviour. I think we should clearly state that the result has to be treated as unordered but we might suggest using lexicographic order for validation purposes when possibile (I think a huge number of implementations are using ordered maps for JSON objects storage so it shouldn't be hard). Removing an implemented fature used from a number of users for testing purposes is something that I wish to not discuss. I think a number of JSONPath implementations are completely decoupled from the JSON parser, and they just accepted maps/lists/atomic-values as input ( is one of them, it doesn't depend on any JSON parser and it doesn't care about JSON parsing). I'm not even sure if JSONPath is only used for JSON inputs, it works so nicely with any JSON-like structure. ¹ I'm not 100% sure that all implementations are using JSON types, there might be implementations using unordered sets for real as containers for results.
A noble goal, to be sure. But I imagine that sudden conformance to a spec would natural engender breaking changes, especially in a landscape of Implementation as large as we're dealing with. I don't think it'll be avoided. As such, my feeling is that we get the spec right rather than make concessions just because the majority does it some way. Absolutely correct.
We have some of this in our DNA, but we also have a clear mandate not to disregard the installed base (this is JSONpath, not a new protocol based on JSONpath). So any argument on this spectrum will need to be carefully weighed, because both extremes are wrong. Grüße, Carsten
I think the summary so far is We recognize that there are operations in JSONPath whose result is ordered but that order needs to be created based on unordered input. We are not going to cut off these noses to spite our face. We have a choice between imposing a language-defined order (e.g., sorting) or leaving the result non-deterministic. I gather that there is a general direction toward non-determinism, as imposing an order would be too expensive, but I'm not sure we have a conclusion here. Also:
I agree with Carsten's general direction in support of nondeterminism, simply because 8259 explicitly discourages assigning semantics to the ordering of object members. If we want determinism, we probably have to specify RFC8785 (JCS) conformance, which doesn't feel reasonable. As for testing, anyone who's doing this seriously in the real world has to have an URL() method already, right?
I also agree with supporting non-determinism, but please note that URL() won't necessarily help because the non-deterministic portions of the output won't necessarily be part of a JSONObject. For example, applying to an input consisting of a JSON object will produce a node list in a non-deterministic order. I think we'd either need to decorate such tests with an indication of where non-determinism can occur so that each test framework can compare the output to the expected value and take into account non-determinism or we'd need to have a way of capturing a set of possible alternative outputs. The latter may be preferable as it would complicate the generation of tests, which is a one-off process, but would simplify implementing test frameworks, of which there are likely to be many. (There may be other solutions too - haven't spent very long on this.)
FWIW, in implementing JSON Schema, I've found that I need both contents-equality and sequence-equality on arrays depending on the circumstance. Given that JSON Path outputs an array, and the contents of that array are likely to be generated non-deterministically due to evaluating objects, I think it makes sense for implementations to verify evaluation results (e.g. when processing a test suite) in a "contents-equality" sense. It's not per the JSON definition of array equality (which mandates sequence-equality), but I think the circumstance fits. The fact that the result array is not guaranteed to be deterministic would need to be declared in the specification, probably associated with a remark about it deviating from JSON array equalilty and perhaps even an explanation why.
Nodelists are not arrays. (They can be represented in a JSONPath API as arrays. Or as lists. Or as tuples. ...)
I do agree with common non-determinism here. Naming results as nodelists is clever. Shouldn't we define that term 'nodelist' then ?
Yes, we should define nodelist. I made a first attempt in PR now.
We need to tread carefully here. Are we saying we're comfortable outputting something that's not JSON but acts really similar? All implementations (including NAME output an array (or optionally ) as JSON. If we suddenly call the output a "nodelist" are we breaking that?
No. I agree that many APIs will output platform arrays here (which may or may not feel similar to JSON arrays). Saying that the output is a nodelist is gives them freedom to define their API and how these present the nodelist. More importantly, we make processing for embedded queries less finicky.
Daniel, nobody can (or will try to) stop you thinking about a nodelist as a JSON array, if that suits your mental model. Giving a separate name to that concept does have its advantages. It, for instance, helps finding answers to the role of nested queries in filter expressions. It also helps reflect the fact that some implementations handle nodelists different from others. So I don't know why we are having this discussion. (And why is it under "Determinism?".)
This is irrelevant because of many of the points that NAME mentioned: essentially, implementations don't use nodelists from what we can tell, they use arrays. And isn't our charter to break existing implementations as little as possible? If all implementations use arrays, changing that is not something we should consider lightly. NAME We're discussing it here because of . Following some discussion about expressions, you spammed "nodelist" comments anywhere it might be related in order to promote the idea before anyone had a chance to discuss it. So now we're discussing it. All of that said, I think we should consider the output format ( & ), determinism (this issue), and paths embedded in expressions () together. There is some subtle interplay going on. If the output is an array of objects that specify location and value, then both that format and the sequence of the items within it need to be considered when returning from an embedded path so that the remainder of the expression can handle it properly. If embedded paths are to follow some different output that operators can then use, we need to specify what that is as well. Maybe that is the idea of a nodelist. But we need to discuss it first and come to some level of agreement.
It is a pretty common technique in model building to identify important concepts, and once one has identified them, to give them (tentative) names. That allows the participants to refer to the concept-in-formation, and to gradually increase certainty about it. Once there is enough certainty, it may be possible to identify some concepts with each other and turn them into one, but you don't do that before achieving that certainty. I think much of the confusion comes from a perception that whatever is in the draft at one time is cast in concrete. That is not a healthy way to run this, and maybe I was making an unwarranted assumption that this would be obvious. So, indeed, feedback is good, but not so much about reasons wanting to eliminate/equate concepts prematurely and end the discussion about them before understanding them, but more about the properties of the concept itself. The fact that XPath arrived at a specific form of nodelist after trying nodesets is certainly interesting feedback. We may not need that complexity, but it is much easier to eliminate complexity once we understand the concepts.
I'm not sure this is sufficient. Let me explain why with an example. Take the input argument: and the selector . I think the valid outputs should be: But I think the following output, for example, should be invalid: The spec should allow for this level of non-determinism, but not more. Similarly, if the Compliance Test Suite covers this example, it should allow precisely the valid outputs to pass. I don't see how a "contents-equality" comparison of arrays would help. This makes me favour encoding sets of possible outputs in the CTS as described in URL
Actually, [2, 3, 0, 1] etc. as well (the outer container also is a map).
Whoops! Yes. Edited the list so it is now complete.
I'm curious why you think this should be invalid. I understand how you got the result, and if you include (or replace with) the paths to those values, it's just as (or more) clear. Partial determinism is going to create a bunch of weird edge cases that will be impossible to completely define.
That of course depends on the semantics of the nodelist. If, as I'm assuming, it is sequence-preserving, you cannot arrive at [0, 2, 1, 3], which would be (JSON pointer syntax, I'm lazy) /a/c, /b/e, /a/d, /b/f.
This is quite a complicated discussion. I'm trying to sum things up: JSON objects as maps are unordered by nature. Due to that, JSONPath query results are non-deterministic regarding their order &ndash; which is common sense here. But determinism i.e. reproducability of JSONPath query results is required for compliance tests. Now implementations usually return query results as an array &ndash; pragmatically JSON in ... JSON out. JSON arrays are ordered by nature, which might falsy signal determinism of results. So calling the result a "node-list" might help here out, but introduces another level of complexity in terms and their usage. Having understood, that determinism is solely required for compliance testing, I am not sure, how to interprete Glyn's example above. Should future compliance tests really tell about every possible valid and invalid result ordering? I am sure someone of us would be able to implement something, exactly reproducing as result (noone would of course). Thus it's implementation dependent. But if we only need to insure, to always generate a single reproducable result, we can achieve this by something like ... Generate a list of output pathes. Sort it lexically or however. Generate the list of values from that path list &ndash; preserving order. ... which we can even communicate in the draft as a recipe. Regarding "node-list", there is indeed in JavaScript a concept of "array-like objects". Those objects only need to be able to tell the list length and provide an iterator to visit all entries. Defacto dom node-lists are exactly such things. This is what above discussion also reminds me on. Maybe I still miss another important point.
Neither determinism of the spec nor determinism for a given JSONPath implementation are necessary for compliance tests, but non-determinism does add some complexity to testing. That said, I am in favour of non-determinism in the spec because it gives the most freedom to implementations.
Because I would expect each "stage" of a selector to be evaluated in turn. In other words, the intermediate results which are output from one stage to the next can be in a non-deterministic order, but that order won't be modified by other stages as those stages are operating on the individual values from earlier stages. Sorry, but what is more clear? I don't think so. I think once we have addressed the non-determinism of specific selector "stages" applied to objects, that's pretty much it. What you describes as "partial determinism" is simply a result of how the spec is constructed from small pieces in a uniform way.
To be clear, non-determinism in the spec doesn't force implementations to be non-deterministic. But it allows implementations to trade off efficiency against non-determinism. It also allows for variations between (deterministic) implementations.
Also related to (maybe duplicates) .
A on the duplicates issue brought up the ordering of values in the result output. This does relate directly to whether we remove duplicates in that if we define that sequence matters and duplicates are removed, which copies of duplicated values are kept? Before we can answer that, I think we need to define whether the sequence of the results matter. This has implications for users of the Compliance Test Suite (CTS) as that it lists the expected results. If we decide that output sequence is important, then an implementation using the CTS would need to ensure that the actual result (from the implementation) and the expected result (from the CTS) must be sequence-equal. This is fairly straightforward and easy to process. However if we decide that output sequence does not matter, implementations would need to check for contents-equal. This is more complex to implement (but not impossible). Once we have defined this, we can go back to and address whether to remove duplicates and, if so, which duplicates to remove.
Each selector may each return multiple nodes, or even the same node multiple times. This raises the question of whether (and if so, how) the resulting collection of nodes needs to be distinct, i.e. without duplicates. An example: The raw result set of this selector is . If we say that we expect only the distinct set of nodes, then the results would be , removing the duplicate . So the question up for discussion is, "Should selectors return distinct sets of nodes or are duplicates preferred?" I had been considering a location-based distinction (based on the location of the value in the original JSON structure) as a possibility, but I don't think it makes much sense given the definition of a selector in Section 2.5, taking multiple nodes and outputing multiple nodes. A selector knows nothing about the original JSON structure that is being evaluated. However for posterity, consider the following (rather contrived) path evaluated against the above instance: In this case, the value which is the first element of the property would be selected twice, and the value in the second element of the property would be selected once, resulting in three values in the results. For location-based distinction, only the value from would be considered duplicate, leaving two instances of in the result set. Again, I don't think that this is a viable approach due to how a selector works and the information it's given. However, if we were to also require , then this would be a viable solution.
As for me I haven't see any implementations that tries to remove duplicate values from result, and I don't think it's a good idea to introduce this behavior as default one. But maybe it's useful to provide some way to filter off duplicated values explicitly, like . But your second case (about selecting the same node twice) is more interesting, because we have to provide some default behavior; I will investigate the implementations on this case.
I think it might be useful to throw a scenario in the consensus project to get actual results of what the various implementations do.
A scenario has been . Just waiting on the report to regenerate. It looks like the majority just don't support the example query. For those that do, it seems to be an even mix of all three approaches. No concensus implies no pressure to implement any particular scenario, but we need to discuss and agree on it.
An option is to allow some latitude in the spec and say that "duplicates MAY be removed, either by location and value". But I think in that case we'd also need to tie down when that applies. Should it just apply to unions? Could it also apply in other cases, e.g. when using against an array containing duplicate values (e.g. because there are duplicate values in the input document)? "MAY" will considerably complicate the CTS FWIW, especially if the CTS is purely declarative (e.g. URL) as that would put the onus on each CTS implementation to allow duplicates to be removed if appropriate. Also, does allowing duplicates to be removed imply anything about ordering? Can non-duplicates change order, for example?
I think it can apply to any case where the same value appears in the result more than once. I've opened a new issue for discussing result sequence. I think we need to answer that before we can discuss how it affects duplicates.
To expand on the reason for combining duplicates a little, I offer a more extensive example and a somewhat informal proof of why eliminating duplicates would be a good thing (when not including location in the output): The instance is similar to Goessner's example, but I've added color to the books and moved the other items under a "toy" category so that I can search for array items and use an expression. Now if we want the prices of all the things that are red, we can use . This results in without removal and with removal. Let's ignore the for now to see what how the duplicate values are produced. The path returns three objects: Notice that the results for and are the same. Any processing on further path segments will result in the same output. Therefore we should only need to perform further selection once on these values. Duplicate removal resolves this problem. Even when we include locations, this can be used as an internal optimization. An implementation could identify that these are the same object and keep track of all the locations which contain it.
I'm confused. Why do and in your last example lack the property?
Ugh... Oversight of editing while authoring. I'll fix it. Aaand now I have to rework it to get the duplicates again.
NAME updated.
I think this is trading off performance, in some fairly rare cases, against simplicity of the spec. I'd prefer to keep the spec simpler.
Allowing latitude defeats interoperability, which I had thought was the raison d'etre for the IETF. But I note there's no mention of interoperability as a design goal in the draft. From previous discussions, I think it is only necessary to consider removal of duplicates by location (not value.) I don't think so. I think it should apply however duplicates arise, whether unions, wildcards, recursive descent. This also simplifies removal. Or perhaps, either duplicates must be removed, or duplicates must not be removed, or implementations must default to one of these two, and provide an option for the other. I don't think it affects order at all. Given the data and selector The results with duplicates are: Values|Paths -------|------------------ "The Night Watch" | "The Night Watch" | "The Night Watch" | The results without duplicates are: Values | Paths ---------|------------------ "The Night Watch" | "The Night Watch" | Clearly the second set can easily be obtained from the first without rearranging the items, see Note that implementations have some latitude as to when to remove duplicates. As long as both values and normalized paths are collected, the implementation could delay removal until the last step, before returning results to the user.
Following up on 's motivation for wanting to remove duplicates, there's a related use case that's applicable when a user wants to modify the original data based on addressing information obtained from a query. Such addressing information could be in the form of normalized paths, or it could be provided by references or pointers into the original data, in languages that support that. In either case, the issue is the same. So suppose that a user queries for a collection of prices for all books in the store, and wishes to discount them all by twenty percent. If duplicates come back, if the same price gets marked down twice, that's going to be a problem. This is not entirely hypothetical, there are jsonpath libraries that are used in this way. More generally, it's hard to think of use cases where duplicates would be beneficial. It's possible to think of use cases where they would be detrimental. For an implementation that collects normalized paths along with values, removal is easy. Of course, removal comes with a cost, and some users may prefer not to pay that cost, particularly as most straightforward queries don't result in duplicates anyway.
I know the jury is still out on whether we remove duplicate nodes, but I've prepared draft PRs for the compliance test suite (URL) and the reference implementation (URL) to handle removal of duplicate nodes. Any comments gratefully received.
I am preparing an issue (URL) about unions, where removing duplicate nodes is also a point
I have a question about duplicates. Consider a root value and suppose that a query returned two values corresponding to paths Would the node identified by be considered a duplicate?
No, we only consider duplicates at the top level of the returned nodelist. So the values to compare are an array and a string. That the array also happens to contain the string makes no difference.
Consider: and this jsonpath: : The first part of the jsonpath selects nodes and applying on that result set, produces the node twice
Yes, and we need to agree whether: (1) the duplicate nodes are always delivered (which is what we decided today) (2) the duplicate nodes are never delivered (which is what we went into the discussion with) (3) it depends on the implementation (which IMHO would be a disaster)
IIRC today we were wondering whether: (5) the behavior of the descendent selector may be a special case
Maybe we can record the resolution that there is no removal, neither of duplicate values nor of duplicate nodes.
I'd be interested in how this resolution came about because it seems like the discussion here was leaning the other way: removing duplicates to some degree. To what degree was still under discussion.
Slides for 112 said this: We didn't have time to discuss this at 112, so let's confirm the resolution on the 2022-01-18 interim. I think the summary so far is We recognize that there are operations in JSONPath whose result is ordered but that order needs to be created based on unordered input. We are not going to cut off these noses to spite our face. We have a choice between imposing a language-defined order (e.g., sorting) or leaving the result non-deterministic. I gather that there is a general direction toward non-determinism, as imposing an order would be too expensive, but I'm not sure we have a conclusion here. Grüße, Carsten

Discussion:
The "descendant-selector" selects the node and all its descendants. <ins> In the resultant nodelist: * nodes occur before their children, and * nodes of an array occur in array order. Children of an object may occur in any order, since JSON objects are unordered. </ins> 3.5.8. 3.5.8.1.