internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: tornadoweb/tornado

Issue:
Protocols are not filtered in tornado.curl_httpclient
Tornado should set `pycurl.PROTOCOLS` option to `pycurl.PROTO_HTTP|pycurl.PROTO_HTTPS`.
There are many real-world applications using tornado HTTP client to fetch data from user-provided URLs.
Tornado doesn't filter protocols in provided URLs making it possible to abuse its HTTP client like this:

```
#!/usr/bin/env python3
# coding: utf-8
import tornado.ioloop, tornado.gen, tornado.httpclient, tornado.curl_httpclient

@tornado.gen.coroutine
def main():
    tornado.httpclient.AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
    for url in ('file:///etc/passwd','telnet://time-c.nist.gov:13'):
        res = yield tornado.httpclient.AsyncHTTPClient().fetch(url, raise_error=False)
        print(res.body.decode('utf-8','replace'))

if __name__ == "__main__":
    main().add_done_callback(lambda x: tornado.ioloop.IOLoop.current().stop())
    tornado.ioloop.IOLoop.current().start()
```



Discussion:

Resolving patch:
diff --git a/tornado/curl_httpclient.py b/tornado/curl_httpclient.py
--- a/tornado/curl_httpclient.py
+++ b/tornado/curl_httpclient.py
@@ -278,6 +278,9 @@ def _curl_create(self):
         if curl_log.isEnabledFor(logging.DEBUG):
             curl.setopt(pycurl.VERBOSE, 1)
             curl.setopt(pycurl.DEBUGFUNCTION, self._curl_debug)
+        if hasattr(pycurl,'PROTOCOLS'): # PROTOCOLS first appeared in pycurl 7.19.5 (2014-07-12)
+            curl.setopt(pycurl.PROTOCOLS, pycurl.PROTO_HTTP|pycurl.PROTO_HTTPS)
+            curl.setopt(pycurl.REDIR_PROTOCOLS, pycurl.PROTO_HTTP|pycurl.PROTO_HTTPS)
         return curl
 
     def _curl_setup_request(self, curl, request, buffer, headers):


Test patch:
Repository: tornadoweb/tornado

Issue:
TCP connection fails on systems without IPv6 support when connecting to a host that has an IPv6 address
`getaddrinfo` will return IPv6 addresses when the system doesn't support it, and Tornado will then fail to connect to the IPv6 address with this error (I get [a report for my project here](https://github.com/lilydjwg/nvchecker/issues/41)):

```
OSError: [Errno 97] Address family not supported by protocol
```

I see in #593 `AI_ADDRCONFIG` was removed from `bind_sockets`, and in #823 Tornado just ignores the unsupported protocol. But this is about `connect`, not `bind`. What do you think, to use `AI_ADDRCONFIG` or just ignore the error? There's currently no way to pass the flags to resolvers though.



Discussion:
It sounds like the resolver is configured incorrectly if it is returning ipv6 addresses on a machine that doesn't support it. But if someone gets themselves into that state, I guess we should catch and ignore the error in TCPClient as in #823. 

Even though AI_ADDRCONFIG makes more sense for connect() than for bind(), the arguments against it in #593 still mostly apply, so I don't think we should introduce a way to pass it to the resolvers for client-side use. 


Resolving patch:
diff --git a/tornado/tcpclient.py b/tornado/tcpclient.py
--- a/tornado/tcpclient.py
+++ b/tornado/tcpclient.py
@@ -177,7 +177,13 @@ def connect(self, host, port, af=socket.AF_UNSPEC, ssl_options=None,
     def _create_stream(self, max_buffer_size, af, addr):
         # Always connect in plaintext; we'll convert to ssl if necessary
         # after one connection has completed.
-        stream = IOStream(socket.socket(af),
-                          io_loop=self.io_loop,
-                          max_buffer_size=max_buffer_size)
-        return stream.connect(addr)
+        try:
+            stream = IOStream(socket.socket(af),
+                            io_loop=self.io_loop,
+                            max_buffer_size=max_buffer_size)
+        except socket.error as e:
+            fu = Future()
+            fu.set_exception(e)
+            return fu
+        else:
+            return stream.connect(addr)


Test patch:
Repository: tornadoweb/tornado

Issue:
HTTPRequest has default request_timeout
Hi

While developing a reserver proxy, I have realised that http requests times out while they should stay indefinitely.
it seems like default `request_timeout=20` . 

This is better to be without internal default value or documentation should mention there is a default value



Discussion:
You're right that this should be documented, but I think it's better to have a default timeout than to not have one so it won't be going away. 


Resolving patch:
diff --git a/tornado/httpclient.py b/tornado/httpclient.py
--- a/tornado/httpclient.py
+++ b/tornado/httpclient.py
@@ -341,13 +341,15 @@ def __init__(self, url, method="GET", headers=None, body=None,
            Allowed values are implementation-defined; ``curl_httpclient``
            supports "basic" and "digest"; ``simple_httpclient`` only supports
            "basic"
-        :arg float connect_timeout: Timeout for initial connection in seconds
-        :arg float request_timeout: Timeout for entire request in seconds
+        :arg float connect_timeout: Timeout for initial connection in seconds,
+           default 20 seconds
+        :arg float request_timeout: Timeout for entire request in seconds,
+           default 20 seconds
         :arg if_modified_since: Timestamp for ``If-Modified-Since`` header
         :type if_modified_since: `datetime` or `float`
         :arg bool follow_redirects: Should redirects be followed automatically
-           or return the 3xx response?
-        :arg int max_redirects: Limit for ``follow_redirects``
+           or return the 3xx response? Default True.
+        :arg int max_redirects: Limit for ``follow_redirects``, default 5.
         :arg string user_agent: String to send as ``User-Agent`` header
         :arg bool decompress_response: Request a compressed response from
            the server and decompress it after downloading.  Default is True.
@@ -381,9 +383,9 @@ def __init__(self, url, method="GET", headers=None, body=None,
         :arg string proxy_auth_mode: HTTP proxy Authentication mode;
            default is "basic". supports "basic" and "digest"
         :arg bool allow_nonstandard_methods: Allow unknown values for ``method``
-           argument?
+           argument? Default is False.
         :arg bool validate_cert: For HTTPS requests, validate the server's
-           certificate?
+           certificate? Default is True.
         :arg string ca_certs: filename of CA certificates in PEM format,
            or None to use defaults.  See note below when used with
            ``curl_httpclient``.


Test patch:
Repository: tornadoweb/tornado

Issue:
Reading IOStream not registered with IOLoop
I'm getting this weird situation while trying to debug some sporadic hang in one of our tests. An IOStream is busy reading but its fd is not registered with the IOLoop. Consequently, incoming data is not handled.

This might be an issue on our side (though it looks unlikely at this point), but I'm posting the debugger session below in case this is a known oddity:
```
(Pdb) pp s1
<tornado.iostream.IOStream object at 0x7f0780708630>
(Pdb) pp s1.closed()
False
(Pdb) p s1._read_bytes
8
(Pdb) p s1._read_future
<tornado.concurrent.Future object at 0x7f0780662828>
(Pdb) pp s1._state
25
(Pdb) pp s1._state == s1.io_loop.ERROR | s1.io_loop.READ
True
(Pdb) pp s1.fileno()
<socket.socket fd=35, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 35300), raddr=('127.0.0.1', 60520)>
(Pdb) pp s1.io_loop._running
True
(Pdb) pp s1.io_loop._closing
False
(Pdb) pp s1.io_loop._handlers[35]
*** KeyError: 35
(Pdb) pp s1.io_loop
<tornado.platform.epoll.EPollIOLoop object at 0x7f078103f0f0>
(Pdb) p list(s1.io_loop._handlers)
[12, 14, 15, 22, 23, 24, 28, 29, 30, 36, 40, 41, 42, 43, 44, 45, 48, 50, 52, 54, 55, 58, 64, 65, 69, 70, 74, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91]
(Pdb) p s1.io_loop._impl.modify(35, s1._state)
*** FileNotFoundError: [Errno 2] No such file or directory
```



Discussion:
Ok, it seems it's a bug in `TCPServer.stop()`: if you call it several times, it will remove handlers for file descriptors of previously-closed sockets.

Resolving patch:
diff --git a/tornado/tcpserver.py b/tornado/tcpserver.py
--- a/tornado/tcpserver.py
+++ b/tornado/tcpserver.py
@@ -109,6 +109,7 @@ def __init__(self, io_loop=None, ssl_options=None, max_buffer_size=None,
         self._sockets = {}  # fd -> socket object
         self._pending_sockets = []
         self._started = False
+        self._stopped = False
         self.max_buffer_size = max_buffer_size
         self.read_chunk_size = read_chunk_size
 
@@ -227,7 +228,11 @@ def stop(self):
         Requests currently in progress may still continue after the
         server is stopped.
         """
+        if self._stopped:
+            return
+        self._stopped = True
         for fd, sock in self._sockets.items():
+            assert sock.fileno() == fd
             self.io_loop.remove_handler(fd)
             sock.close()
 


Test patch:
diff --git a/tornado/test/tcpserver_test.py b/tornado/test/tcpserver_test.py
--- a/tornado/test/tcpserver_test.py
+++ b/tornado/test/tcpserver_test.py
@@ -1,4 +1,5 @@
 from __future__ import absolute_import, division, print_function, with_statement
+
 import socket
 
 from tornado import gen
@@ -37,3 +38,11 @@ def handle_stream(self, stream, address):
                 server.stop()
             if client is not None:
                 client.close()
+
+    def test_stop_twice(self):
+        sock, port = bind_unused_port()
+        server = TCPServer()
+        server.add_socket(sock)
+        server.stop()
+        server.stop()
+
Repository: tornadoweb/tornado

Issue:
Lookup Error: unknown encoding: latin1
I'm trying to benchmark a simple Tornado Server made using tornado.httpserver and I get the following error, when using apache bench with 1000 or more concurrent connections on a Solaris machine.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/iostream.py", line 304, in wrapper
  File "/usr/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/httpserver.py", line 227, in _on_headers
LookupError: unknown encoding: latin1

I believe this is due to a race condition that leaves the python codec table in an invalid state. My understanding is that latin1 is included in the python standard library, so it should be present as long as nothing is modifying pythons codec table.

Thanks!!

Link to server code: https://www.dropbox.com/s/xrgj59sv0y5y31n/server_ioloop.py



Discussion:
You're not the first to report this, but I haven't been able to reproduce it or determine the cause.  I think the clue is that you're on solaris with a large number of connections - we don't have specific IOLoop support for solaris, so it's falling back to select(), which historically has had a 1024 file descriptor limit.  Exceeding this limit without checking could corrupt memory (although I think python's select module has a safety check for this).  

Unless a motivated solaris user contributes an IOLoop implementation with native support, you're going to need to limit yourself to under 1000 sockets per process on solaris.  It would certainly be nice if it failed a little more gracefully when you hit that limit, but it's not going to really work at that scale without a better IOLoop.

Thanks for your fast response. I've found a temporary fix for the issue so im going to close this, but im not sure this has to do with the file descriptor limit on select. I replaced the standard IOLoop implementation for one I made using pyev. This allowed me to use DevPoll and Event Ports with tornado on Solaris and I was only able to reliably reproduce the bug using either DevPoll or Event Ports.

My temporary fix for this in branch 2.3 works by changing line 227 to
data = native_str(codecs.latin_1_decode(data)[0])

Im not sure why it works. Thanks again.


Resolving patch:
diff --git a/tornado/netutil.py b/tornado/netutil.py
--- a/tornado/netutil.py
+++ b/tornado/netutil.py
@@ -96,6 +96,9 @@
 # thread now.
 u'foo'.encode('idna')
 
+# For undiagnosed reasons, 'latin1' codec may also need to be preloaded.
+u'foo'.encode('latin1')
+
 # These errnos indicate that a non-blocking operation must be retried
 # at a later time.  On most platforms they're the same value, but on
 # some they differ.


Test patch:
Repository: tornadoweb/tornado

Issue:
http1connection: Send `Connection: close` before closing connection
From https://github.com/tornadoweb/tornado/pull/1963#issuecomment-305983002

When the client sends `Connection: close`, Tornado (correctly) responds by closing the connection after sending the response. [RFC 7230 section 6.6](https://tools.ietf.org/html/rfc7230#section-6.6) says that a server SHOULD set the `Connection: close` header in the response when it does so. 


Discussion:

Resolving patch:
diff --git a/tornado/http1connection.py b/tornado/http1connection.py
--- a/tornado/http1connection.py
+++ b/tornado/http1connection.py
@@ -357,6 +357,10 @@ def write_headers(self, start_line, headers, chunk=None, callback=None):
                 # Applications are discouraged from touching Transfer-Encoding,
                 # but if they do, leave it alone.
                 'Transfer-Encoding' not in headers)
+            # If connection to a 1.1 client will be closed, inform client
+            if (self._request_start_line.version == 'HTTP/1.1' and
+                self._disconnect_on_finish):
+                headers['Connection'] = 'close'
             # If a 1.0 client asked for keep-alive, add the header.
             if (self._request_start_line.version == 'HTTP/1.0' and
                 (self._request_headers.get('Connection', '').lower() ==
@@ -418,7 +422,7 @@ def _format_chunk(self, chunk):
     def write(self, chunk, callback=None):
         """Implements `.HTTPConnection.write`.
 
-        For backwards compatibility is is allowed but deprecated to
+        For backwards compatibility it is allowed but deprecated to
         skip `write_headers` and instead call `write()` with a
         pre-encoded header block.
         """
diff --git a/tornado/httpserver.py b/tornado/httpserver.py
--- a/tornado/httpserver.py
+++ b/tornado/httpserver.py
@@ -154,7 +154,6 @@ def initialize(self, request_callback, no_keep_alive=False,
                    max_body_size=None, max_buffer_size=None,
                    trusted_downstream=None):
         self.request_callback = request_callback
-        self.no_keep_alive = no_keep_alive
         self.xheaders = xheaders
         self.protocol = protocol
         self.conn_params = HTTP1ConnectionParameters(


Test patch:
diff --git a/tornado/test/httpserver_test.py b/tornado/test/httpserver_test.py
--- a/tornado/test/httpserver_test.py
+++ b/tornado/test/httpserver_test.py
@@ -719,6 +719,7 @@ def test_request_close(self):
         self.stream.read_until_close(callback=self.stop)
         data = self.wait()
         self.assertTrue(not data)
+        self.assertEqual(self.headers['Connection'], 'close')
         self.close()
 
     # keepalive is supported for http 1.0 too, but it's opt-in
Repository: tornadoweb/tornado

Issue:
X-Forwarded-Proto Multiple Values
I have been investigating an issue with AWS Application Load Balancers forwarding traffic to a JupyterHub instance (that uses Tornado) - and it appears that the AWS ALB can forward traffic with the following header:

`X-Forwarded-Proto: https,https`

However, Tornado only appears to be able to support a singular value:

https://github.com/tornadoweb/tornado/blob/d4094a0c2a7cddc654381c162ee7786d4bf8453b/tornado/httpserver.py#L286

It seems a similar issue affected ASP.NET at one point:

https://github.com/aspnet/BasicMiddleware/issues/18

Although Mozilla document it as only having a single value:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto


Discussion:

Resolving patch:
diff --git a/tornado/httpserver.py b/tornado/httpserver.py
--- a/tornado/httpserver.py
+++ b/tornado/httpserver.py
@@ -287,6 +287,10 @@ def _apply_xheaders(self, headers):
         proto_header = headers.get(
             "X-Scheme", headers.get("X-Forwarded-Proto",
                                     self.protocol))
+        if proto_header:
+            # use only the last proto entry if there is more than one
+            # TODO: support trusting mutiple layers of proxied protocol
+            proto_header = proto_header.split(',')[-1].strip()
         if proto_header in ("http", "https"):
             self.protocol = proto_header
 


Test patch:
diff --git a/tornado/test/httpserver_test.py b/tornado/test/httpserver_test.py
--- a/tornado/test/httpserver_test.py
+++ b/tornado/test/httpserver_test.py
@@ -550,6 +550,16 @@ def test_scheme_headers(self):
             self.fetch_json("/", headers=https_forwarded)["remote_protocol"],
             "https")
 
+        https_multi_forwarded = {"X-Forwarded-Proto": "https , http"}
+        self.assertEqual(
+            self.fetch_json("/", headers=https_multi_forwarded)["remote_protocol"],
+            "http")
+
+        http_multi_forwarded = {"X-Forwarded-Proto": "http,https"}
+        self.assertEqual(
+            self.fetch_json("/", headers=http_multi_forwarded)["remote_protocol"],
+            "https")
+
         bad_forwarded = {"X-Forwarded-Proto": "unknown"}
         self.assertEqual(
             self.fetch_json("/", headers=bad_forwarded)["remote_protocol"],
Repository: tornadoweb/tornado

Issue:
Handle non-ascii in debug_msg
After running my server on a staging server I've got following error:
```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
  File "tornado/curl_httpclient.py", line 496, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "tornado/escape.py", line 218, in to_unicode
    return value.decode("utf-8")
```

Where value is equal to:
```
b"\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03]\x8eA\n\x80 \x10E\xf7Aw\x90.0E-'\x97\xdd#sP\xc1\x14d\xa0\xba}Z.\xa2\xe5\xfc\xffx\x7f\xd0\xf2\xeee\xdb\xa0\xa5UKd\xc7\x9e\xe4\xd4\x8fb\x89I9\xad) \xbc!\xc2\x83dTE}\te\xb6\xe8c\x9a\xbb\xc3:\xa6\xae(6\nLI\xa2\x1d\xfe\x86\x9c \xd4\xbale\xa8^\xc1\xb8p~;(\xf6b\x83\xfa\xd9\r\xf4iB\xa9\xa2\x00\x00\x00"
```

After small googling, I found this issue https://github.com/tornadoweb/tornado/issues/1608 and this fix https://github.com/tornadoweb/tornado/commit/d7d9c467cda38f4c9352172ba7411edc29a85196. 

Seems that this error is relating to the fixed issue and the line should be fixed with the same way


Discussion:
Decoding `debug_msg` as latin1 may get rid of this exception, but that doesn't mean it's the right fix. That doesn't look like latin1 data to me. I think the problem is that the handling of this byte string depends on the `debug_type` parameter: if it's 0, 1, or 2, it's a string that we should decode. If it's 3 or 4, then it's a blob we have to escape when printing (with `%r`). So we need to move the `native_str` call into the 0 and 1/2 branches of the if/elif. 

Resolving patch:
diff --git a/tornado/curl_httpclient.py b/tornado/curl_httpclient.py
--- a/tornado/curl_httpclient.py
+++ b/tornado/curl_httpclient.py
@@ -499,10 +499,11 @@ def _curl_header_callback(self, headers, header_callback, header_line):
 
     def _curl_debug(self, debug_type, debug_msg):
         debug_types = ('I', '<', '>', '<', '>')
-        debug_msg = native_str(debug_msg)
         if debug_type == 0:
+            debug_msg = native_str(debug_msg)
             curl_log.debug('%s', debug_msg.strip())
         elif debug_type in (1, 2):
+            debug_msg = native_str(debug_msg)
             for line in debug_msg.splitlines():
                 curl_log.debug('%s %s', debug_types[debug_type], line)
         elif debug_type == 4:


Test patch:
Repository: tornadoweb/tornado

Issue:
WebSocketHandler.select_subprotocol() is always invoked
The WebSocketHandler.select_subprotocol() callback is invoked for every client connection, even clients that do not send (or send an empty) Sec-WebSocket-Protocol header.  For these clients, the callback is invoked with (subprotocols=['']), which is probably never useful.  In addition, both the documentation and the code seem to indicate that the intent is for this callback to only be invoked if the client submits some non-empty value for Sec-WebSocket-Protocol.

```
class WebSocketProtocol13(WebSocketProtocol):
    def _accept_connection(self):
        subprotocols = self.request.headers.get("Sec-WebSocket-Protocol", '')
        subprotocols = [s.strip() for s in subprotocols.split(',')]
        if subprotocols:
            selected = self.handler.select_subprotocol(subprotocols)
            ...
```

For cases when `Sec-WebSocket-Protocol` is empty, `subprotocols` evaluates to `['']` before being tested in the conditional.  Unfortunately, `[''] == True`, which means this `if subprotocols:` always evaluates True.

Options I see:
- If we can assume that `self.request.headers` is always an instance of `HTTPHeaders`, we can use `get_list()` like so:
`subprotocols = [s.strip() for s in self.request.headers.get_list("Sec-WebSocket-Protocol")]` instead of `get()` and `split()`.  This will result in `subprotocols = []`, which evaluates `False`.  This feels like the cleaner solution and I will work on a PR for this.  Is there a reason for us to access our headers using only basic dict operations?
- Evaluate `if subprotocols:` immediately after `get()` while it is still a string, _before_ calling `split()` / `strip()` to turn it into a list.


Discussion:

Resolving patch:
diff --git a/tornado/websocket.py b/tornado/websocket.py
--- a/tornado/websocket.py
+++ b/tornado/websocket.py
@@ -662,8 +662,7 @@ def _challenge_response(self):
             self.request.headers.get("Sec-Websocket-Key"))
 
     def _accept_connection(self):
-        subprotocols = self.request.headers.get("Sec-WebSocket-Protocol", '')
-        subprotocols = [s.strip() for s in subprotocols.split(',')]
+        subprotocols = [s.strip() for s in self.request.headers.get_list("Sec-WebSocket-Protocol")]
         if subprotocols:
             selected = self.handler.select_subprotocol(subprotocols)
             if selected:


Test patch: