Draft: draft-trammell-taps-interface

Commentary: The previous text did n't support sending multiple messages across vanilla TCP without framers / deframers or a top - level protocol in the stack that would de - frame . This change allows sending multiple messages across vanilla TCP and handing them over as bytes to an application that , by itself , detects message boundaries from the chunks of bytes that it gets . 
 It seems we never came to consensus over the main issue here - we did have consensus that it 's fine to have framers and deframers . It seems that the crux of the matter is the following :  and you want : a sender saying " the frame ends here " will terminate a TCP connection ; so if you really really want to hand over multiple chunks of data from a stream , then you should use one huge message and only terminate when finished with everything . I do n't want that , for the reason I explained so often under the heading " Application - Framed Bytestream " : the sending application does n't know whether it gets vanilla TCP or e.g.  . If it gets SCTP , messages can be sent as messages , with all their properties : lifetime , unordered delivery ... but this can only work if the sender knows that messages are messages . If the transport system gets TCP , however , it must be able to send the messages directly on the bytestream . Either way this is fine as long as the receiving application is able to determine boundaries in the chunk of data that it gets : in case of SCTP , it really does get messages , in case of TCP , it gets a bytestream . By requiring " atomic " message transmission from one end to the other , and not being able to send multiple messages on a single TCP connection , this functionality becomes unavailable . If this seems confusing , please consider taking another look at the animations , e.g. here : If you want a longer , more detailed explanation , please read : If you do n't believe this can work , please download and test NEAT . I do n't understand all this " a TCP connection represents a large message business " . It 's a strange limit that gets in the way of efficiently falling back to vanilla TCP . 
 our lack of consensus seems to me to come from a giant pile of subtle misunderstandings : nope . Well , at least I do n't want this . This would be silly . There 's no guarantee that one side of a connection using this interface will be connected to another side of a connection also using the interface . Message sends are passed down to the underlying protocol stack atomically , so that stacks that do indeed support boundary preservation have boundaries . If you get TCP , it 'll ignore the boundaries , and hope the other side provides deframing . ( If a sender absolutely requires atomicity without relying on receiver - side deframing , it could prohibit the selection of protocol stacks that do n't support it , of course . ) But we do n't require atomic message transmission . We require an API which presents atomic message transmission to the application when it is available , and allows ( asks ) for the application to help provide this when it is not . Here 's what this looks like when a sender writes multiple messages on a stack not supporting boundary preservation to a receiver not running this interface : The receiver gets a pile or bytes in order and hands them to an app layer parser , just like in TCP . Here 's what this looks like when a TCP only - sender writes three app - layer - delimited messages to a stream to a receiver running this interface with deframers : The receiver pushed the deframe logic down into the interface , and got to simplify its own workflow -- every callback has an atomic message . Here 's what this looks like in the case this PR seems to care about : TCP , no deframer : Just like the TCP case above . Each event is just like a stream call in TCP . ( I can , indeed , see why you 'd want to provide length hints to to support this , and I think we should add that . ) I believe it can work , that 's not the question at all . I just do n't think it 's a very useful abstraction if a goal of building a new interface is moving application development away from stream - centric assumptions in order to allow it to take advantage of new transports that provide better - than - single - stream abstractions . There 's no limit here . To translate this interface into bytestream thinking : a partial message is simply a section of a bytestream with a running offset ( which can itself be calculated trivially by the stack or the system ) . When a receiver does n't have enough information to atomize messages , it presents bytestream(s ) as partials . 
 So in this comment , I agree with absolutely everything . Just two comments : Great ! ! But I was under the strong impression that this is what Tommy wants because of things that he wrote , both in the text and in issue . Fantastic if I misunderstood all that , but then my suggested text changes should be fine . All I want is to allow : 1 ) sending many messages ( multiple " message over " markers from the application ) over a single TCP connection 2 ) receiving blocks of data ( similar to normal TCP data reception today ) from which an application determines where messages begin and end . Just for the record , that did n't come from me - I suspect it was from Tommy . Anyway , it makes sense to me as well . 
 hm i think that might have been me . such is the nature of the pile of confusions ... 
 with the caveat that these might be completely lost on send , yep , absolutely we need to allow this . yep , with the caveat that we do not yet have a way to pass the  length the application wants to receive ( but can easily add this ) . Hm , I think you still need the " this is a partial message " indicator on the receive ( which of course the application knows , but allowing the " partial message " marker to be omitted when the underlying transport / deframe stack does n't have any boundary information seems like a contract violation to me . 
 Yes , of course . Same as happens today . Yes , although the way I was envisaging this working was that the application provides a de - framing function to the stack , and that de - framing function determines where the messages begin and end , so that what 's returned are complete messages . That way , you get a message based abstraction even if running over a byte - stream transport like TCP . 
 I agree with the caveats you list above , that 's all fine . About this : The point is that the receiving application must be able to know if it did get a complete message . When it does , the transport system can say so , so there you get your contract . When it does not , the receiver - side transport system may not know about it because it does n't know about sender - supported message boundaries in the first place ( the first of your two mentioned caveats ) . If it got the data as a TCP stream , then at best , it can say " perhaps this is n't a complete message , I 'm not sure " , but it certainly ca n't provide the other things your text requires in this case ( whether it 's a partial message , a reference to the complete message , .. ) . If we phrase it as I suggest , I think we break no contract and we promise as much as we can promise . 
 AHHHH okay . Actually , I think this should be a three - state indicator : ( 1 ) This is a complete message ( known because the deframer said so , or the transport stack did ) . ( 2 ) This is a partial message containing a section of a message with a known message boundary ( made partial for local buffering reasons , either by the underlying protocol stack or the deframer ) ( 3 ) This is a partial message containing a data with no definite message boundary ( i.e. stream protocol and no deframer ) ... 
 CONVERGED ! I agree with the 3 - state indicator proposal :) 
 I like your last list of three , but I will still point out that over any stream protocol with no deframer , there will be a definite end to received data ( the equivalent of a FIN ) . My previous comments about FINs , etc , were just to point out that it is possible to map a stream this way . If we 're doing HTTP/2 or QUIC streams , we may indeed want the end of the message to correspond to the end of the stream , since HTTP over HTTP/2 generally will do one request per stream identifier . However , I had also mentioned that it is possible to have the stream protocol ignore any claimed message boundaries , but this means that the message boundaries are understood to be lost when mapped into the bytes on the wire . 
 yes ..... I think that makes sense 
 - this sounds fine ! ( phew ! ) So we agree that a sender can send 50 small messages over a single connection ( each marked with " end of message " ) , right ? And when the transport system ends up using vanilla TCP , and the receiver - side application then wants to obtain the messages , it must provide a deframer ( note that the " must " here is new to me , but I think it 's fine and probably even a good idea in order to do cooler things later ) . 
 Well , yes ... but the application wo n't know what the end is until it sees the FIN . so s / no definite message boundary / no known message boundary/ in state 3 ? ( git logistics question : do you want to update this branch ( which will automatically update the PR ) with receive length and three - state partials , or shall we land this PR then do those in another ? ) 
 I 'll update the PR next ( first time I 'm doing this - sorry for being clumsy ) 
 This looks good to me now . I 'll do a separate PR to add receive size to the API . 
 Ah , sorry , I forgot this to do this ( receive size ) ; thanks ! 
 Will land both of these ; open subsequent issues if we need to tweak the text , but IMO this is certainly -00 quality ...
Paragraph (version A):
In this case, the Message object passed to Received contains an
indication that the object received is partial, the byte offset of
the data in the partial Message within the full Message, an
indication whether this is the last (highest-offset) partial Message
in the full Message, and an optional reference to the full Message it
belongs to.
Paragraph (version A):
Note that in the degenerate case - no message boundary preservation
and no deframing - the entire connection is represented as one large
message of indeterminate length.
Paragraph (version B):
Note that in the absence of message boundary preservation and without
deframing, the entire connection is represented as one large message
of indeterminate length.
Paragraph (version A):
The effect of the application sending a Message is determined by the
top-level protocol in the established Protocol Stack.  That is, if
the top-level protocol provides an abstraction of framed messages
over a connection, the application will be able to send multiple
Messages on that connection, even if the framing protocol is built on
a byte-stream protocol like TCP.
Paragraph (version B):
The effect of the application sending a Message is determined by the
top-level protocol in the established Protocol Stack.  That is, if
the top-level protocol provides an abstraction of framed messages
over a connection, the receiving application will be able to obtain
multiple Messages on that connection, even if the framing protocol is
built on a byte-stream protocol like TCP.
Paragraph (version A):
Similar to sending, Receiving a Message is determined by the top-
level protocol in the established Protocol Stack.  The main
difference with Receiving is that the size and boundaries of the
Message are not known beforehand.  The application can communicate in
its Receive action the parameters for the Message, which can help the
implementation know how much data to deliver and when.  For example,
if the application only wants to receive a complete Message, the
implementation should wait until an entire Message (datagram, stream,
or frame) is read before delivering any Message content to the
application.  Alternatively, the application can specify the minimum
number of bytes of Message content it wants to receive (which may be
just a single byte) to control the flow of received data.
Paragraph (version B):
Similar to sending, Receiving a Message is determined by the top-
level protocol in the established Protocol Stack.  The main
difference with Receiving is that the size and boundaries of the
Message are not known beforehand.  The application can communicate in
its Receive action the parameters for the Message, which can help the
implementation know how much data to deliver and when.  For example,
if the application only wants to receive a complete Message, the
implementation should wait until an entire Message (datagram, stream,
or frame) is read before delivering any Message content to the
application.  This requires the implementation to understand where
messages end, either via a supplied deframer or because the top-level
protocol in the established Protocol Stack preserves message
boundaries; if, on the other hand, the top-level protocol only
supports a byte-stream and no deframers were supported, the
application must specify the minimum number of bytes of Message
content it wants to receive (which may be just a single byte) to
control the flow of received data.
Paragraph (version A):
Without a layer of framing above TCP, the cleanest abstraction for
sending and receiving Messages over TCP is to treat each direction of
the stream of bytes as a single Message, terminated by a FIN.  That
means that if the application can make several Send calls to enqueue
subsequent data chunks for the same Message to continue writing, but
marking the Message complete corresponds to closing the sending
stream.  Similarly, when the application receives the final portion
of a Message, it knows that the receiving stream has been closed.
Paragraph (version B):
Without a layer of framing (a top-level protocol in the established
Protocol Stack that preserves message boundaries, or an application-
supplied deframer) on top of TCP, the receiver side of the transport
system implementation can only treat the incoming stream of bytes as
a single Message, terminated by a FIN when the Remote Endpoint closes
the Connection.