jiebi/CodeConvo huggingface dataset:
Historical note: HTTP 1.0 arguably specified the "abort request if the client half-closes" behaviour, see the last paragraph of URL, two pages down, just above URL This behaviour was implemented by several widely used HTTP servers; I worked on a firewall product at the time, and we had to take great care to make sure we handled TCP half-close correctly... The "arguably' is that RFC 1945 says "the closing of the connection by either or both parties", but sending a doesn't really "close" the connection - that requires either a or both ends -ing the other end's .
The HTTP RFCs say nothing about the expectations around half-closed TCP connections. In practice, I haven't seen any HTTP client in the wild send a request, and then send a FIN () while still waiting for the server's response. Because we haven't see any clients do that, as of Go 1.8, Go's HTTP server is starting to make assumptions that reading an EOF from the client means that the client is no longer interested in the response. (reading EOF being the closest portable approximation to "the client has gone away"). But in URL, a user reports that they have an internal HTTP client which does indeed make a half-closed TCP request. It would be nice if the HTTP RFCs provided guidance as to whether this is allowed or frowned upon. I would recommend that the RFC suggest that clients SHOULD NOT half-close their TCP connections while awaiting responses. Because nobody else does, empirically, and relying on reading EOF is a useful signal for servers. /cc NAME NAME
I've maintained a server in the past that received the same bug report. As a matter of compliance I decided the client was right - EOF is a way to delimit a message (they were using it to stream request bodies without chunking). But as a matter of practicality I didn't fix the bug because, as you indicate, ignoring EOF in practice means a lot of useless buffering ending in RST or timeouts.
NAME oh, interesting. And disgusting. I hadn't considered EOF being a means to end an HTTP/1.0 POST. Perhaps I can change Go to make an exception for HTTP/1.0 requests with request bodies.
NAME but NAME points out that RFC 1945 (HTTP 1/.0) says: So using a half-closing a TCP connection was never a valid way to signal the end of an entity body.
This is a pretty important question. Today, it's hard to spec out how cancellation works with REST. I suspect the proposed spec change will break some users. I wonder if it's possible to measure the impact with real traffic, from the server-side, e.g. the success rate of response completion after a half-close is received. I am happy to run some experiments ...
Here is a example of half-close connection in the wild, Livestatus. Livestatus is a broker for nagios. To use it via sockets you have to make the query, ex. and the close the write channel. After this, the Livestatus returns by the half-closed socket the results. Example in Python The important snippet is this, when closing the Write channel of the socket: Are there any workaround for golang? Source: https://mathias-URL
NAME Interesting. But is that HTTP? I see examples like the following on the web page. It does not even look like HTTP/0.9, though I am not sure what the proper syntax of 0.9 is.
It could be with unix socket or tcp socket. My fail I only read TCP, i did not see the repo is http :s. My bad, sorry, I came from another issue that was speaking about TCP without the http specification..
Hello, everyone! I would like to make a case for HTTP/1 disallowing half-closed TCP connections from the client. What I mean by that is that when an HTTP/1 client shuts down their writing end of the connection (when they send a TCP frame with the FIN flag), that the server can safely assume that the client lost interest in the unsent remainder of the response. On the other hand, if the server is required to support half-closed TCP connections from the client, in other words if the client is allowed to send a FIN after sending its request and it still expects to receive the response, then the server can only assume that the client lost interest when it receives an RST from the client.
To be able to treat an incoming FIN as an indication of the client's loss of interest has advantages for the server: The server can free up resources earlier, upon the reception of the FIN, rather than having to wait for an RST a network round-trip time after sending a data packet. The savings are most pronounced when dealing with responses that trickle information down slowly as a form of long polling. Some stateful firewalls (NAT) between the client and the server drop packets for connections for which they no longer have state, rather than sending back RSTs. The RSTs thus never make it to the server, the server continues trickling down information until the lack of ACKs causes its window to fill and eventually a write to time out. Server resources held on behalf of a client who lost interest are wasted resources. I believe that there would be little to no disadvantage to the client. Furthermore: HTTP/1.1 encourages the reuse of a TCP connection for a subsequent request. Request message bodies are framed using or : "The presence of a message body in a request is signaled by a Content-Length or Transfer-Encoding header field." ( URL ) TLS 1.0 explicitly precludes half-closing at its layer: When one end receives a "closenotify" alert, it is required that it "respond with a closenotify alert of its own and close down the connection immediately, discarding any pending writes." ( URL ) Thank you.
A half-close has never been an indication that the client isn't interested in the response. It only indicates that the client isn't going to send any future requests on that connection. Since HTTP/1.x is defined to be independent of the transport, I see no reason to specify half-close – what matters is that the request message be complete. A server is not obligated to send a response, regardless.
Yeah, but TCP is a pretty popular choice. I think it's worth clarifying what this part of TCP means for HTTP. This bug exists because implementations are disagreeing on what it means. We're looking to a spec for guidance.
It's pretty hopeless at this point. I propose we clarify that half-close means nothing to HTTP/1.x in this spec, i.e. it's not a cancellation.
If any http/1->http/2+ proxies send a rst stream when receiving a half-close from the client, it's a bug.
The problem with "clarifying it for TCP" is that HTTP runs on any transport with connection qualities and finding a word that means half-close for TCP doesn't always translate into some other transport or session-layer's meaning of half-close, but … I will try to find a way to do that when we get to the whole "what do we mean by a connection" rewrite in the semantics spec. In any case, regarding the original question: Go should not interpret an EOF on read as implying an EOF on close. Even if we don't see that as common in standard practice, there are probably thousands of deployed C applications that distinguish the two states; they won't work when some poor soul tries to port them to Go, and you'll be reliving this discussion on a regular basis. For example, IIRC, the first conversation I ever had with NAME (in 2000) was about using half-close to end requests in iCAP. It happens a lot more frequently in private hacks that are merely using the HTTP libraries for some other purpose.
in bkk: no strong consensus. mike bishop to see if he can identify clients that are using half close actively. agreed scope here is h1/tcp
I don't think there are any clients currently using this, but I think it would make sense to add guidance stating that half-close does not have any semantics, for all versions of HTTP. Having servers not kill half-closed connections may allow innovation down the line, and I don't think giving this guidance is risky
I've seen that quite a bit in field with monitoring scripts involving netcat, as well as data transfer tools. It's a bit less common nowadays since wget and curl are everywhere, but the usual stuff used to be this : host:$port | grep -q OK host:/d' > acl-URL In haproxy we don't do anything specific with half-close, we only rely on the end of message, unless the admin specifies "option abortonclose" in which case a half-closed client connection will be aborted if the server has not started to respond. Also there is no real value in killing half-closed connections.
Often people who want to do this mix up the end of message and the end of connection and tend to believe that if the connection is not aborted, there will be no more opportunity to do it. But in fact if the client really aborts, the server will receive RST in response to some send() and will be able to detect the close. I would say that : half-closed (shutdown(SHUT_WR)) is a graceful shutdown (i.e. "nothing more to say"), signaled on the wire with a TCP FIN and detected on the server as read()==0 abort (close()) is a real connection abort (i.e. "stop sending this to me"), signaled on the wire with a TCP RST and detected on the server as send()==-1 The only case where you don't know is until you've started to send(), which explains why haproxy uses its option to decide what to do before receiving the server's response. I'm just realizing that we could even suggest to send a 100-continue in response to a half-closed connection to probe the connection!
What I've heard back from Microsoft folks with access to old emails is that we've seen half-close behavior from: .NET 2.0's HTTP implementation An IBM Java client, about which we have little additional detail Obviously, these are old clients with negligible share. However, that also illustrates the risk: old clients will not be updated to comply with a new spec.
Go's HTTP package uses EOF from clients to mean "the HTTP client is probably no longer interested in the server's response". We use it to notify callers of the HTTP server who've registered their interest in knowing when the HTTP client is gone (especially one reading a long-polled response, like Server-Sent Events). Notably, we want to know this immediately upon a FIN, without waiting for a write to fail. (We might not have anything to write for some time.) While we might ideally use OS-specific TCP stats kernel interfaces to distinguish FIN from RST, we support a dozen+ OSes and the basic interface we can rely on is EOF on FIN. There often isn't a better userspace API available to know the TCP state. I'd rather the HTTP spec say that clients should not half-close TCP connections, as some servers may interpret a half-closed connection as a client that's no longer interested in the response.
Go has a package that lets you do low-level networking-y things. Anybody porting low-level C could port to that. The concern for this bug is about Go's behavior, not its package.
Hi Brad! You must not resort to using the OS to distinguish between the two because you don't need to know that. As you say it's system-specific. And until you send anything you're not guaranteed to get an RST anyway. The problem you're facing is that you're acting as a proxy between the client and the application. You need a way to let the application know that the client indicated that it stops sending, and only the application can decide if it's an indication of end of transfer or of client abort. Sometimes the application will decide that both are equivalent and that's fine. Of course when you get a notification of error via an RST, it's pretty clear and unambigous. If I may suggest, "just" pass a shut_read info in one case that applications are free to interpret as aborts if they want, and an abort or close info when you're certain it's closed. But really any network-based application must be able to tell the difference between half-closed and full-closed otherwise it's deemed to either abusively close on regular half-close or ignore real closes sometimes. In practice people tend to see TCP as a bidirectional stream while it's in fact two independent unidirectional streams. The only link between the two are the ACK numbers passed along packets to confirm receipt. But FTP servers for example are well used to seing half-closed connections since the data channel can be half-closed during the whole transfer.
NAME Go's HTTP package has supported its API since 2013-05-13 and the means it's not going away. We can improve the implementation, but we can't pretend the problem doesn't exist and remove the feature. It came about because it was frequently requested. Authors of HTTP server handlers want to know when people close tabs in their browser, etc.
NAME OK that's perfect. Then it's more a matter of clarifying what each event means and what shortcuts may be taken with what impacts. In practice it's fine most of the time, it's just that it's important to be clear about what this really means.
For example it's fine to say "you can reasonably assume that a close notification indicates a closed tab and that you can abort an ongoing operation if your application is designed to work solely with a browser, but a more robust application should consider that it only indicates the client has nothing more to send and that the server must close just after sending the final response ; specifically, some scripts or API clients may induce a close event immediately after sending a request and while waiting for a response".
Discussed in Bangkok; we need to collect data about behaviour.
As an extra data point, both SwiftNIO and Netty treat receiving EOF on read as an indication the client is no longer interested in the response. We're definitely not happy with this: from our perspective, clients should be able to send FIN without us giving up on them. I'd be in favour of wording to disincentivise servers from doing what we (currently) do.
ietf104: significant interest in documenting somethinghere. at least "client should not do this" - hummed the question of discouraging the server from closing connection on rcpt of fin. a decent support for that pr.
There seems to be a really simple hack for differentiating FIN/RST. As NAME noted, a write will fail if a RST was received. This issue only applies to HTTP/1.1. Therefore: upon read-EOF, immediately write the "HTTP/1.1 " of the response line. If that write fails, you can safely abort generation of the rest of the response, because it was a RST. Or does this not work on certain platforms?
There are also various OS level options. Both epoll and kqueue can be used to distinguish FIN and RST, so it’s usually a matter of distinguishing the two cases in higher level APIs.
Good that this issue is still open :) Since the server always knows what works or not, writing a first-line header in response to FIN does allow most of the servers to detect aborted clients immediately. Maybe the spec can just clarify that FIN (half-close) should not be used by the server to cancel the request unless the server has a way to detect that the underlying TCP connection has been shutdown (i.e. the peer has gone away).
Reading through the discussion, a couple of notes; We already talk about half closed connections on the server side in , so talking about them on the client side isn't prima facie out of scope. It might be helpful to explicitly note that requests are not delimited by half-close in . That's not a normative change, just a clarification, and it aligns with broad implementation. It would also satisfy the desire to document "client should not do this" at least in that case. It seems like the best we can do on the server side (in terms of interpreting FIN from the client) is to document that some servers assume it to be a loss of interest, but this may not be reliable in all cases, and should not be relied upon by clients.
^ PTAL
In 9.6 "tear down", after "If the server receives additional data from the client on a fully closed connection, such as another request that was sent by the client before receiving the server's response", I'd add "or the message body". The RST problem is far more visible with POST requests that servers reject due to authentication, redirects, lack of cookies etc, where servers are tempted to respond and immediately close without draining the entire request message. Regarding the questions above, I'm still seeing a few possible hints/workarounds for server implementers who really want to have good interoperability and good assurance that a tab was closed. One hint is to emit a "100-continue" interim response in response to a half-closed request if the request was made over HTTP/1.1. If the client is still there, it will have no effect. If the client has completely closed, this one will trigger a reset from the client which will be detected by the server. The other solution is to consider that if a client sent a request in HTTP/1.1 without "connection: close" and finally half-closed the request side, it is extremely unlikely to be a simple implementation and that it can be assumed with good confidence that the client really wants to abort request processing.
Hey Willy, Would something like this do the trick? I'm a bit wary of putting this in the spec, since it's so specific and advisory. What do others think?
I agree both with the text and your concern.
Maybe we can enforce the fact that it's absolutely not a spec and just a hint by saying "... a server that really needs to distinguish these cases could for example send a 100 (Continue) non-final response if it hasn't yet sent a final response". This way it's clearly worded as a hint to work around existing API limitations and to remind developers that determining whether a client closed a browser window is not rocket science over TCP.
I would implement something like a zero-byte chunk, or a small mount of dummy data (safe to the MIME type such as JSON), which could also help "keep-alive" a streaming response without introducing a non-standard C-T. If a server has already generated the C-L header, I would just let the request run to completion. Or are we concerned about proxies buffering the response body, because 100 will always reach the client immediately?
BTW, I actually plan to implement some sort of cancellation support soon for web frontends behind URL; and I could help report some data wrt the detection mechanism, with or without the app responding to the cancellation event.
You can't send a zero-byte chunk before headers (and even less data of course). The only other reasonably interoperable thing you could send before headers to probe the connection would be a CRLF since most agents will ignore them before a message. Another approach would be to start sending "HTTP/1.1" without the status yet (or just "HTTP" without even the version), but it can be trickier as it will require to remember that this part was already emitted. Now after the headers it's different, you're already sending response data so you don't care, you'll know very soon if the client is still there.
I was wrong. What I proposed (i.e. writing extra body bytes) only works for frameworks that generate headers immediately without waiting for applications to write any data (which might include errors but as application payload). Re: 100-continue response, I noticed that the following restriction stated in rfc2616 was removed from rfc7231. "An origin server SHOULD NOT send a 100 (Continue) response if the request message does not include an Expect request-header field with the "100-continue" expectation. " Curious what's the background for this change.
I suspect the reason was the prevision for new 1xx status codes, that just clients can remap to pure 100 when they don't know them. Note, I proposed 100 but any 1xx (except 101) would fit. For example we might instead send "102 Processing" (rfc2518), which could be even more suitable and possibly less confusing.
I could live with this. Will report back how frequently we are seeing FINs with a pending http/1.1 request on the client side. Yes.
Define congestion collapse in terms of excess overhead. Rename the prior congestion collapse section "Implement full backoff" Closes issue
Editors: This metric is very interesting, but wonder if it's really a requirement for loss recovery, not congestion control (granted, these two are often intertwined). This would put it out of scope. Our proposal would be to revert this, though perhaps we are misunderstanding the intent.
Define congestion collapse in terms of excess overhead under adverse conditions. This is how it was defined circa 1987. Rename the current "congestion collapse" section.
NAME would you like to take a crack at finding some text that would be in scope to address the intent behind this issue?
Closing this issue, as it's through IETF Last Call.
From Ilari Liusvaara on the :
I think that I might disagree with Ilari here for this application. Key separation is probably a better model to employ here, though as soon as someone even hints that they might want to share client certificate keys and these keys then this sort of protection probably makes sense.
Briefly mentioned this issue at IETF 116, but did not have time for questions so we asked folks who care to comment on the issue.
Thinking about this some more, prepending a fixed string to the nonce before signing it sounds like it would be pretty cheap and would remove a class of issues - I'm inclined to do that. I'll write up a PR.
OK wrote up to address this
: That is incorrect, as the status code could be 204, or the response might come with chunked encoding (even when empty). This is .
What purpose is this requirement serving? It seems to have the effect of prohibiting with a zero length payload. Shouldn't RFC 7230, "3.3.3. Message Body Length" be sufficient for determining payload sizes?
Agreed. The text makes it sound as if OPTIONS is somehow special. Giving advice might be good, but the use of normative keywords makes this confusing. The simplest fix might indeed be to remove the sentence.
LGTM
When the text suggests that credentials can be automatically applied to all requests made within a protection space, that begs the question: how do I know what requests are in the same protection space? The answer is "well, you don't really". Authentication schemes might define something, but otherwise you are left to guess. This PR says that as directly as I could manage. I considered adding another sentence here that says "In the absence of specific information about the extent of a protection space, clients &MAY; assume that the protection space extent is the origin of the server." I'd like thoughts on whether that is helpful. I think that it might be, as otherwise this isn't really implementable.
Please factor out whitespace changes...
Ugh, it's like picking up toys after children.
Neither of you have comments about saying what the extent of the protection scope is in the absence of information?
I'd rather leave that out.
AFAIU, it depends on the authentication scheme. At the end of the day, we can only document here what we inherited from earlier specs, and what is actually done in practice. If you have information on the latter which helps us making this better, great. Do you?
I proposed maybe adding: Which I believe matches what clients do when they produce credentials and the authentication scheme doesn't define a clear scope. Do you think that helps?
Not sure. What problem are we trying to solve here? Auth scheme definitions that fail to define the protection space? In which case, shouldn't we say something about this in the "Considerations for new schemes" section?
as: This says that it is a parameter, but does it appear on challenges or responses? only establishes that authentication parameters parameterize authentication schemes, there is no mention of how those relate to what is sent. I think that this first problem only requires a mention of WWW-Authenticate and Proxy-Authenticate. The second problem is in : Clients would seem to have no way of knowing whether reuse is likely to be successful. A protection space is defined as the tuple of origin and realm, but there is no acknowledgment that how a protection space might correspond to the URI space is only known to the server. (The next sentence, which I omitted from this quote acknowledges that the client needs special knowledge in order to understand that a protection space might span origins; that's very useful information.)