jiebi/CodeConvo huggingface dataset:
Code change:
insensitive components such as field names, request URI scheme, or host. <del> Addition or removal of leading or trailing whitespace to a field value. Addition or removal of "obs-fold" from field values. </del> Changes to the request target and authority that when applied together do not result in a change to the message's target URI, as defined in HTTP. <ins> Additionally, there are some transformations that are either deprecated or otherwise not allowed, but still could occur in the wild.  These transformations can still be handled without breaking the signature, and include things such as: Use, addition, or removal of leading or trailing whitespace in a field value. Use, addition, or removal of "obs-fold" in field values (HTTP1). </ins> We can identify these types of transformations as ones that should not prevent signature verification, even when performed on message components covered by the signature.  Additionally, all changes to

Related discussion:
text: Add reference for obs-fold, add note about transforms,
The spec several times refers to the term obs-fold, but doesn't say where it's defined (-> URL). Furthermore, in 1.3 it claims that addition of obs-fold could be permitted under certain circumstances. That IMHO is incorrect.
Adding the reference makes sense. The signatures spec doesn't change how the messages are created, nor does it make more allowed than it is now, it just tells people what to do with the messages that they have even if they have in them.
: This is misleading. No HTTP permits addition of leading/trailing whitespace. Leading/trailing whitespace in field values by definition is invalid.
While technically true, many (including implementers) won't understand the nuance Here. Given the impact, it's better to be over-inclusive, I think.
If invalid changes are included, then it might be best to either say that some of these changes might be invalid or point specifically to those that are.
Yes, my proposal would be to split the list into two; the second just listing the transformatios that might happen in reality despite bing forbidden.
NAME I agree that we need to prepare developers for the messy reality of HTTP in addition to what's strictly allowed. What else would appear in this second list?
The only other thing that I can see right now is addition of obs-fold.
Speaking of which: there are other transformations like parsing and re-serializing field values, which are certainly allowed. Shouldn't these be mentioned here as well?
NAME Re-serialization of a field value by an intermediary would likely break things in an expected way. It was my impression that this was not allowed by intermediaries in general. A structured field does help this, and the flag protects against that kind of transformation, as mentioned in the section.
I don't think we have a rule anywhere that would keep an intermediary from parsing and re-serializing fields. And even if that was the case, the same would be true for the original point (addition of leading/trailing WS), so I'm trying to understand where we draw the line.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: Historical note: HTTP 1.0 arguably specified the "abort request if the client half-closes" behaviour, see the last paragraph of URL, two pages down, just above URL This behaviour was implemented by several widely used HTTP servers; I worked on a firewall product at the time, and we had to take great care to make sure we handled TCP half-close correctly... The "arguably' is that RFC 1945 says "the closing of the connection by either or both parties", but sending a doesn't really "close" the connection - that requires either a or both ends -ing the other end's .
The HTTP RFCs say nothing about the expectations around half-closed TCP connections. In practice, I haven't seen any HTTP client in the wild send a request, and then send a FIN () while still waiting for the server's response. Because we haven't see any clients do that, as of Go 1.8, Go's HTTP server is starting to make assumptions that reading an EOF from the client means that the client is no longer interested in the response. (reading EOF being the closest portable approximation to "the client has gone away"). But in URL, a user reports that they have an internal HTTP client which does indeed make a half-closed TCP request. It would be nice if the HTTP RFCs provided guidance as to whether this is allowed or frowned upon. I would recommend that the RFC suggest that clients SHOULD NOT half-close their TCP connections while awaiting responses. Because nobody else does, empirically, and relying on reading EOF is a useful signal for servers. /cc NAME NAME
I've maintained a server in the past that received the same bug report. As a matter of compliance I decided the client was right - EOF is a way to delimit a message (they were using it to stream request bodies without chunking). But as a matter of practicality I didn't fix the bug because, as you indicate, ignoring EOF in practice means a lot of useless buffering ending in RST or timeouts.
NAME oh, interesting. And disgusting. I hadn't considered EOF being a means to end an HTTP/1.0 POST. Perhaps I can change Go to make an exception for HTTP/1.0 requests with request bodies.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: NAME but NAME points out that RFC 1945 (HTTP 1/.0) says: So using a half-closing a TCP connection was never a valid way to signal the end of an entity body.
This is a pretty important question. Today, it's hard to spec out how cancellation works with REST. I suspect the proposed spec change will break some users. I wonder if it's possible to measure the impact with real traffic, from the server-side, e.g. the success rate of response completion after a half-close is received. I am happy to run some experiments ...
Here is a example of half-close connection in the wild, Livestatus. Livestatus is a broker for nagios. To use it via sockets you have to make the query, ex. and the close the write channel. After this, the Livestatus returns by the half-closed socket the results. Example in Python The important snippet is this, when closing the Write channel of the socket: Are there any workaround for golang? Source: https://mathias-URL
NAME Interesting. But is that HTTP? I see examples like the following on the web page. It does not even look like HTTP/0.9, though I am not sure what the proper syntax of 0.9 is.
It could be with unix socket or tcp socket. My fail I only read TCP, i did not see the repo is http :s. My bad, sorry, I came from another issue that was speaking about TCP without the http specification..
Hello, everyone! I would like to make a case for HTTP/1 disallowing half-closed TCP connections from the client. What I mean by that is that when an HTTP/1 client shuts down their writing end of the connection (when they send a TCP frame with the FIN flag), that the server can safely assume that the client lost interest in the unsent remainder of the response. On the other hand, if the server is required to support half-closed TCP connections from the client, in other words if the client is allowed to send a FIN after sending its request and it still expects to receive the response, then the server can only assume that the client lost interest when it receives an RST from the client.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: To be able to treat an incoming FIN as an indication of the client's loss of interest has advantages for the server: The server can free up resources earlier, upon the reception of the FIN, rather than having to wait for an RST a network round-trip time after sending a data packet. The savings are most pronounced when dealing with responses that trickle information down slowly as a form of long polling. Some stateful firewalls (NAT) between the client and the server drop packets for connections for which they no longer have state, rather than sending back RSTs. The RSTs thus never make it to the server, the server continues trickling down information until the lack of ACKs causes its window to fill and eventually a write to time out. Server resources held on behalf of a client who lost interest are wasted resources. I believe that there would be little to no disadvantage to the client. Furthermore: HTTP/1.1 encourages the reuse of a TCP connection for a subsequent request. Request message bodies are framed using or : "The presence of a message body in a request is signaled by a Content-Length or Transfer-Encoding header field." ( URL ) TLS 1.0 explicitly precludes half-closing at its layer: When one end receives a "closenotify" alert, it is required that it "respond with a closenotify alert of its own and close down the connection immediately, discarding any pending writes." ( URL ) Thank you.
A half-close has never been an indication that the client isn't interested in the response. It only indicates that the client isn't going to send any future requests on that connection. Since HTTP/1.x is defined to be independent of the transport, I see no reason to specify half-close – what matters is that the request message be complete. A server is not obligated to send a response, regardless.
Yeah, but TCP is a pretty popular choice. I think it's worth clarifying what this part of TCP means for HTTP. This bug exists because implementations are disagreeing on what it means. We're looking to a spec for guidance.
It's pretty hopeless at this point. I propose we clarify that half-close means nothing to HTTP/1.x in this spec, i.e. it's not a cancellation.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: If any http/1->http/2+ proxies send a rst stream when receiving a half-close from the client, it's a bug.
The problem with "clarifying it for TCP" is that HTTP runs on any transport with connection qualities and finding a word that means half-close for TCP doesn't always translate into some other transport or session-layer's meaning of half-close, but … I will try to find a way to do that when we get to the whole "what do we mean by a connection" rewrite in the semantics spec. In any case, regarding the original question: Go should not interpret an EOF on read as implying an EOF on close. Even if we don't see that as common in standard practice, there are probably thousands of deployed C applications that distinguish the two states; they won't work when some poor soul tries to port them to Go, and you'll be reliving this discussion on a regular basis. For example, IIRC, the first conversation I ever had with NAME (in 2000) was about using half-close to end requests in iCAP. It happens a lot more frequently in private hacks that are merely using the HTTP libraries for some other purpose.
in bkk: no strong consensus. mike bishop to see if he can identify clients that are using half close actively. agreed scope here is h1/tcp
I don't think there are any clients currently using this, but I think it would make sense to add guidance stating that half-close does not have any semantics, for all versions of HTTP. Having servers not kill half-closed connections may allow innovation down the line, and I don't think giving this guidance is risky
I've seen that quite a bit in field with monitoring scripts involving netcat, as well as data transfer tools. It's a bit less common nowadays since wget and curl are everywhere, but the usual stuff used to be this : host:$port | grep -q OK host:/d' > acl-URL In haproxy we don't do anything specific with half-close, we only rely on the end of message, unless the admin specifies "option abortonclose" in which case a half-closed client connection will be aborted if the server has not started to respond. Also there is no real value in killing half-closed connections.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: Often people who want to do this mix up the end of message and the end of connection and tend to believe that if the connection is not aborted, there will be no more opportunity to do it. But in fact if the client really aborts, the server will receive RST in response to some send() and will be able to detect the close. I would say that : half-closed (shutdown(SHUT_WR)) is a graceful shutdown (i.e. "nothing more to say"), signaled on the wire with a TCP FIN and detected on the server as read()==0 abort (close()) is a real connection abort (i.e. "stop sending this to me"), signaled on the wire with a TCP RST and detected on the server as send()==-1 The only case where you don't know is until you've started to send(), which explains why haproxy uses its option to decide what to do before receiving the server's response. I'm just realizing that we could even suggest to send a 100-continue in response to a half-closed connection to probe the connection!
What I've heard back from Microsoft folks with access to old emails is that we've seen half-close behavior from: .NET 2.0's HTTP implementation An IBM Java client, about which we have little additional detail Obviously, these are old clients with negligible share. However, that also illustrates the risk: old clients will not be updated to comply with a new spec.
Go's HTTP package uses EOF from clients to mean "the HTTP client is probably no longer interested in the server's response". We use it to notify callers of the HTTP server who've registered their interest in knowing when the HTTP client is gone (especially one reading a long-polled response, like Server-Sent Events). Notably, we want to know this immediately upon a FIN, without waiting for a write to fail. (We might not have anything to write for some time.) While we might ideally use OS-specific TCP stats kernel interfaces to distinguish FIN from RST, we support a dozen+ OSes and the basic interface we can rely on is EOF on FIN. There often isn't a better userspace API available to know the TCP state. I'd rather the HTTP spec say that clients should not half-close TCP connections, as some servers may interpret a half-closed connection as a client that's no longer interested in the response.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: Go has a package that lets you do low-level networking-y things. Anybody porting low-level C could port to that. The concern for this bug is about Go's behavior, not its package.
Hi Brad! You must not resort to using the OS to distinguish between the two because you don't need to know that. As you say it's system-specific. And until you send anything you're not guaranteed to get an RST anyway. The problem you're facing is that you're acting as a proxy between the client and the application. You need a way to let the application know that the client indicated that it stops sending, and only the application can decide if it's an indication of end of transfer or of client abort. Sometimes the application will decide that both are equivalent and that's fine. Of course when you get a notification of error via an RST, it's pretty clear and unambigous. If I may suggest, "just" pass a shut_read info in one case that applications are free to interpret as aborts if they want, and an abort or close info when you're certain it's closed. But really any network-based application must be able to tell the difference between half-closed and full-closed otherwise it's deemed to either abusively close on regular half-close or ignore real closes sometimes. In practice people tend to see TCP as a bidirectional stream while it's in fact two independent unidirectional streams. The only link between the two are the ACK numbers passed along packets to confirm receipt. But FTP servers for example are well used to seing half-closed connections since the data channel can be half-closed during the whole transfer.
NAME Go's HTTP package has supported its API since 2013-05-13 and the means it's not going away. We can improve the implementation, but we can't pretend the problem doesn't exist and remove the feature. It came about because it was frequently requested. Authors of HTTP server handlers want to know when people close tabs in their browser, etc.
NAME OK that's perfect. Then it's more a matter of clarifying what each event means and what shortcuts may be taken with what impacts. In practice it's fine most of the time, it's just that it's important to be clear about what this really means.
Code change:
(in order of precedence): Since there is no way to distinguish a successfully completed, close- <del> delimited message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length- delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </del> <ins> delimited response message from a partially received message interrupted by network failure, a server SHOULD generate encoding or length-delimited messages whenever possible.  The close-delimiting feature exists primarily for backwards compatibility with HTTP/1.0. </ins> A server MAY reject a request that contains a message body but not a

Related discussion:
text: For example it's fine to say "you can reasonably assume that a close notification indicates a closed tab and that you can abort an ongoing operation if your application is designed to work solely with a browser, but a more robust application should consider that it only indicates the client has nothing more to send and that the server must close just after sending the final response ; specifically, some scripts or API clients may induce a close event immediately after sending a request and while waiting for a response".
Discussed in Bangkok; we need to collect data about behaviour.
As an extra data point, both SwiftNIO and Netty treat receiving EOF on read as an indication the client is no longer interested in the response. We're definitely not happy with this: from our perspective, clients should be able to send FIN without us giving up on them. I'd be in favour of wording to disincentivise servers from doing what we (currently) do.
ietf104: significant interest in documenting somethinghere. at least "client should not do this" - hummed the question of discouraging the server from closing connection on rcpt of fin. a decent support for that pr.
There seems to be a really simple hack for differentiating FIN/RST. As NAME noted, a write will fail if a RST was received. This issue only applies to HTTP/1.1. Therefore: upon read-EOF, immediately write the "HTTP/1.1 " of the response line. If that write fails, you can safely abort generation of the rest of the response, because it was a RST. Or does this not work on certain platforms?
There are also various OS level options. Both epoll and kqueue can be used to distinguish FIN and RST, so it’s usually a matter of distinguishing the two cases in higher level APIs.
Good that this issue is still open :) Since the server always knows what works or not, writing a first-line header in response to FIN does allow most of the servers to detect aborted clients immediately. Maybe the spec can just clarify that FIN (half-close) should not be used by the server to cancel the request unless the server has a way to detect that the underlying TCP connection has been shutdown (i.e. the peer has gone away).