oberstet/Autobahn — Issue #73: closed-loop testing
Author: oberstet
Created: 2011-11-03 16:16:10+00:00
State: open
Labels: feature, testsuite

[16:10:50] Peter Thorson: you around?
[16:12:45] Tobias Oberstein: yo
[16:13:33] Peter Thorson: did you have any thoughts on closing behavior while a message is in flight?
[16:14:17] Tobias Oberstein: i think you are right .. its an app thing.
[16:15:34] Tobias Oberstein: do we allow 3000-4999 as close codes in those cases?
[16:15:52] Peter Thorson: no, and we shouldn't
[16:16:01] Tobias Oberstein: why?
[16:16:08] Peter Thorson: 3000-4999 are specifically left undefined by the spec
[16:17:00] Peter Thorson: a code in that range would only make sense if both sides have negotiated their meaning
[16:18:06] Tobias Oberstein: what about your example
4000: please cancel the transaction and rollback the db
[16:18:41] Peter Thorson: Point being. AB tests the websocket spec, not Peter's streaming db commit application subprotocol
[16:19:56] Tobias Oberstein: ok. right. but "left undefined" (= possibly app defined) != invalid ?
[16:20:59] Tobias Oberstein: or do you say: defining 4000 must be done necessarily in an negoiated subprotocol. in other words: its invalid to use 4000 ever .. when no subprotocol is neogiated?
[16:23:19] Peter Thorson: hm
[16:26:31] Peter Thorson: thats at good question
[16:28:09] Tobias Oberstein: if its invalid when not negotiated via subprotocol, and not merely "agreed" by convention between peers, then our tests for valid codes are wrong

if its valid, then using 4000 wherever 1000 is allowed, should probably valid also

but i'm not sure also. the language of the spec does not seem to talk about this. yeah. well.
[16:29:18] Peter Thorson: so yes either we only allow 1000 or we allow 1000 or 3000-4999
[16:29:35] Peter Thorson: and maybe give a non-strict?
[16:32:50] Tobias Oberstein: yep. that makes some sense.

if allowed, then why non-strict? i mean: currently "non-strict" = SHOULD in spec or canonical beh.

so == 1000 => canonical? and using 4000 for more specific stuff is non-canonical?

mmh.

i think it would be worth to get opinions on the question whether 4000 is only to be used with negotiated subprotocols first .. since i guess both interpretations can be argued
[16:34:04] Tobias Oberstein: or make it a "impl. / test suite option" ;)
[16:36:25] Peter Thorson: actually this brings up another question
[16:37:37] Peter Thorson: there are some cases where the endpoint behaves correctly according to the letter of the spec but in doing so does not provide the test suite with enough information to actually make a useful determination
[16:37:44] Peter Thorson: for example
[16:38:26] Peter Thorson: a hypothetical WS library that forwards close code and reason to its end application without doing UTF8 validation and also never sends close codes
[16:38:52] Peter Thorson: will pass 7.5.1 but also forward its end application invalid UTF8
[16:40:00] Peter Thorson: technically there is nothing AB can do about that, but I am wondering if there is a way to mark tests like that with a footnote or something that says this is technically compliant behavior, but the behavior might mask an underlying implimentation problem
[16:40:26] Peter Thorson: its not "non-strict" because it is 100% compliant
[16:43:02] Tobias Oberstein: ok, i see.
[16:43:14] Peter Thorson: but a warning note or something for implimentation developers to make sure that if they are choosing to pass the test that way (usually meaning not sending a close code) the tests may give false positives due to not enough information
[16:43:41] Peter Thorson: or false negatives? not sure which :P
[16:44:05] Tobias Oberstein: the former
[16:45:52] Tobias Oberstein: what do you do with an impl. that forwards received msg payload reversed to the app, and when the app sends, reverses the payload again.

will pass our echo tests ..
[16:46:36] Tobias Oberstein: you need to "tap at the api level" and somehow connect that to AB
[16:47:02] Peter Thorson: i suppose as far as AB is concerned in that case the impl and app together are compliant
[16:47:18] Peter Thorson: but it doesn't say anything about them individually
[16:47:58] Tobias Oberstein: ok. right.

i was thinking about the "extended test protocol" feature. as we planned it, it's something in-band
[16:48:29] Peter Thorson: in that case we can sort of control the end application as well
[16:48:51] Tobias Oberstein: if we did tap the API using an out-of-band (2nd WS?) channel, more stuff would be possible .. includ. catching the close example of yours above
[16:49:18] Peter Thorson: hmm that is interesting
[16:49:31] Peter Thorson: test both what comes over the wire in the close and what the application got
[16:49:48] Tobias Oberstein: like:

AB => control probe WS => impl. API => impl. => fuzzing WS => AB
[16:50:29] Peter Thorson: should be easy to do for browser clients at least
[16:50:59] Tobias Oberstein: closed-loop testing
[16:51:22] Peter Thorson: write a javascript app that connects to an AB fuzzing server and does more than just open and echo
[16:51:39] Tobias Oberstein: yep
[16:51:57] Peter Thorson: testing implimentations without well defined W3C APIs will be more difficult
[16:52:35] Tobias Oberstein: it would open 2 WS connections .. need to correlate those at AB .. should be straight forward.

but yes: major benefit would be testing including up to and including API
[16:53:28] Tobias Oberstein: for non-standardized APIs .. its the impl. guys work if they wanna have it.

like the trivial work you need to do today: impl. echo client/server to get your stuff trested with AB
[16:53:49] Tobias Oberstein: would be more work of course for them
[16:54:35] Peter Thorson: they should be writing tests of their own API anyways
[16:55:11] Tobias Oberstein: speaking of tests .. some guy on the twisted list asked me: where are the unit tests? ;)
[16:55:12] Peter Thorson: honestly actually sounds like less work that writing my own API testing harness :P
[16:55:27] Peter Thorson: unit tests for the tests or for AB in general?
[16:55:43] Tobias Oberstein: for AB in general. well.
[16:55:48] Peter Thorson: or did he just not see the testsuite directory?
[16:55:55] Tobias Oberstein: no, he did.
[16:56:06] Peter Thorson: i guess that doesn't count as "units" does it
[16:56:45] Tobias Oberstein: yeah, somehow not. its a valid question, though my thinking is little different
http://www.tavendo.de/autobahn/faq.html
[16:58:13] Peter Thorson: that sounds reasonable. WS++ uses unit tests for select internal things that are more clumsy to do with the full protocol tests
[16:59:14] Peter Thorson: and I guess in some cases for things that AB doesn't test yet :P
[16:59:19] Peter Thorson: like URL parsing
[17:00:04] Tobias Oberstein: yeah the opening hs ..
[17:01:03] Peter Thorson: WS URI parsing is even before that
[17:01:23] Peter Thorson: a client should refuse to open a connection to an invalid uri entirely
[17:01:58] Peter Thorson: that might be a case for the closed-loop tests
[17:02:18] Peter Thorson: for now though implimentation specific unit tests for those things seem reasonable
[17:02:24] Tobias Oberstein: ok. i see. could be done with the closed-loop test approach: probe channels sends control msg: open invalid WS .. expects to get back "error" .. on probe channel
[17:04:25] Tobias Oberstein: do you think its worth keeping an issue for?

i mean: the extended test proto feature (in-band) would be nice, some work, but only allow for limited more test functionality.

the out-of-band / closed-loop test feature would be quite nice, some more work, but would allow considerable more test stuff
[17:05:18] Peter Thorson: do you mean is it worth doing an in-band extended protocol at all?
[17:05:37] Peter Thorson: or is an out-of-band too crazy to actually make a github issue for?
[17:05:44] Tobias Oberstein: no, if closed-loop .. that much better, and then there is no need for in-band anymore
[17:06:19] Peter Thorson: if you had a closed loop extended test it could do everything an in-band extended test could do
[17:06:41] Peter Thorson: obviously by just sending control messages on the second band rather than in band with a prefix
[17:06:49] Tobias Oberstein: yep.
[17:07:02] Peter Thorson: question probably is how much more complicated is dealing with two WS connections
[17:07:09] Tobias Oberstein: thinking about it, its probably the much cleaner approach anyway
[17:08:52] Peter Thorson: writing a dual channel vs in-band echo server using WS++ is trivial, mainly because the API is loosely designed around my primary use case for a WS server, and that is to aggregate connections into shared environments
[17:09:12] Peter Thorson: im not sure how the AB server api is set up in that case
[17:10:04] Peter Thorson: with WS++ the on_message callbacks go to a single shared object and include a pointer to the session that they came from.
[17:10:14] Peter Thorson: for all of the connections to that server
[17:10:28] Tobias Oberstein: with AB, you have the factory as singleton
[17:10:56] Peter Thorson: if you wanted to build a "multiplayer" echo server with AB
[17:11:15] Peter Thorson: where a message by any connected client is broadcast to all other connected clients
[17:11:23] Peter Thorson: is that difficult?
[17:11:29] Tobias Oberstein: http://www.tavendo.de/autobahn/tutorial/broadcast.html
[17:13:18] Peter Thorson: looks like it would be easy to do in AB too =)
[17:13:29] Tobias Oberstein: the tutorial is missing descr. of android client .. but that works also. so JS, Py and Android