Project: springxd
Issue XD-2233: REST representation of an aggregate-counter can lead to mixed up output
The current representation of REST resources of time-series data (e.g. aggregate counter) can lead to problems in consuming applications. Despite the time series data provided by the "counts" data structure is logically ordered by key (timestamps) it doesn't guarantee an ordering, since many consuming applications interpret JSON data as an unordered map like data structure. Because of this consuming applications have to apply special ordering / transformation logic to get the data in an chronologically ordered fashion. It would be helpful if one could configure the rendering of the time series data, e.g. as a list of json object like: {code:json} { "ts":"Sun Oct 12 23:10:23 CEST 2014" ,"value": 42 } {code} Where {{ts}} denotes the timestamp and {{value}} denotes the value. It would also be helpful if one could adjust the date format either with a pattern or a well known date format like, c.f. {{ISO 8601}}. I attached a python example for this that demonstrates the problem. Steps to reproduce: Create stream {code} xd:>stream create test --definition "http | filter --expression=payload.contains('pivotal') | log" {code} Create tap on stream with aggregate-counter {code} xd:>stream create test_tap --definition "tap:stream:test.filter > aggregate-counter" {code} Post some http data {code} xd:>http post --data "Hello pivotal data labs" #...some more data... {code} Display aggregate counter {code} xd:>aggregate-counter display test_tap --resolution minute AggregateCounter=test_tap ----------------------------- - ----- TIME - COUNT Sun Oct 12 23:10:23 CEST 2014 | 0 Sun Oct 12 23:11:23 CEST 2014 | 0 Sun Oct 12 23:12:23 CEST 2014 | 0 Sun Oct 12 23:13:23 CEST 2014 | 0 ... Mon Oct 13 00:02:23 CEST 2014 | 0 Mon Oct 13 00:03:23 CEST 2014 | 0 Mon Oct 13 00:04:23 CEST 2014 | 0 Mon Oct 13 00:05:23 CEST 2014 | 0 Mon Oct 13 00:06:23 CEST 2014 | 0 Mon Oct 13 00:07:23 CEST 2014 | 0 Mon Oct 13 00:08:23 CEST 2014 | 3 Mon Oct 13 00:09:23 CEST 2014 | 1 {code} Install python Requests library (REST support) {code} pip install Requests {code} Start a python console (or IPythonNotebook) and run the following program: {code:python} import requests import json res = requests.get("http://localhost:9393/metrics/aggregate-counters/test_tap?resolution=minute") status_object = json.loads(res.content) print(json.dumps(status_object, indent=4)) {code} The above program should result in a similar output, but as one can see, due to pythons interpretation of the JSON object as a dict, the order of the keys in the output got mixed up. Instead of showing the counts ....3 and then 1 ... as in the example above. This is just one example of how the current representation of the rest resource could lead to problems in consuming applications. {code:json} { "counts": { "2014-10-12T21:26:28.553Z": 0, "2014-10-12T22:15:28.553Z": 0, "2014-10-12T22:22:28.553Z": 0, "2014-10-12T21:49:28.553Z": 0, "2014-10-12T21:35:28.553Z": 0, "2014-10-12T21:47:28.553Z": 0, "2014-10-12T22:18:28.553Z": 0, "2014-10-12T22:12:28.553Z": 0, "2014-10-12T22:16:28.553Z": 0, "2014-10-12T21:57:28.553Z": 0, "2014-10-12T21:28:28.553Z": 0, "2014-10-12T22:08:28.553Z": 3, "2014-10-12T21:40:28.553Z": 0, "2014-10-12T22:06:28.553Z": 0, "2014-10-12T21:27:28.553Z": 0, "2014-10-12T21:52:28.553Z": 0, "2014-10-12T22:11:28.553Z": 0, "2014-10-12T22:05:28.553Z": 0, "2014-10-12T21:29:28.553Z": 0, "2014-10-12T21:24:28.553Z": 0, "2014-10-12T21:56:28.553Z": 0, "2014-10-12T21:43:28.553Z": 0, "2014-10-12T22:00:28.553Z": 0, "2014-10-12T22:10:28.553Z": 0, "2014-10-12T21:58:28.553Z": 0, "2014-10-12T22:21:28.553Z": 0, "2014-10-12T21:32:28.553Z": 0, "2014-10-12T21:46:28.553Z": 0, "2014-10-12T22:04:28.553Z": 0, "2014-10-12T22:02:28.553Z": 0, "2014-10-12T21:51:28.553Z": 0, "2014-10-12T21:38:28.553Z": 0, "2014-10-12T21:31:28.553Z": 0, "2014-10-12T22:20:28.553Z": 0, "2014-10-12T21:54:28.553Z": 0, "2014-10-12T22:07:28.553Z": 0, "2014-10-12T22:03:28.553Z": 0, "2014-10-12T21:34:28.553Z": 0, "2014-10-12T22:09:28.553Z": 1, "2014-10-12T21:44:28.553Z": 0, "2014-10-12T22:17:28.553Z": 0, "2014-10-12T21:53:28.553Z": 0, "2014-10-12T22:19:28.553Z": 0, "2014-10-12T21:30:28.553Z": 0, "2014-10-12T22:23:28.553Z": 0, "2014-10-12T21:36:28.553Z": 0, "2014-10-12T21:41:28.553Z": 0, "2014-10-12T22:13:28.553Z": 0, "2014-10-12T21:59:28.553Z": 0, "2014-10-12T22:01:28.553Z": 0, "2014-10-12T21:33:28.553Z": 0, "2014-10-12T21:45:28.553Z": 0, "2014-10-12T21:39:28.553Z": 0, "2014-10-12T21:50:28.553Z": 0, "2014-10-12T21:37:28.553Z": 0, "2014-10-12T22:14:28.553Z": 0, "2014-10-12T21:25:28.553Z": 0, "2014-10-12T21:55:28.553Z": 0, "2014-10-12T21:42:28.553Z": 0, "2014-10-12T21:48:28.553Z": 0 }, "name": "test_tap", "links": [ { "href": "http://localhost:9393/metrics/aggregate-counters/test_tap", "rel": "self" } ] } {code}
Story points: 8