Repository: ipython/ipython

Issue:
Leaking client connections
It looks like connections are leaking somehow.

I am not using IPython's cluster functionality.

```
ERROR:tornado.application:Exception in callback (<socket._socketobject object at 0x39fcb40>, <function null_wrapper at 0x3a1a848>)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 833, in start
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
  File "/usr/local/lib/python2.7/dist-packages/tornado/netutil.py", line 189, in accept_handler
  File "/usr/lib/python2.7/socket.py", line 202, in accept
error: [Errno 24] Too many open files
```

and from `lsof`:

```
python    10008 10018     ubuntu  138u     IPv4  174360        0t0     TCP localhost:52318->localhost:43388 (ESTABLISHED)
python    10008 10018     ubuntu  139u     IPv4  114055        0t0     TCP 10.0.6.245:8888->192.168.200.48:49754 (ESTABLISHED)
python    10008 10018     ubuntu  140u     0000     0,9          0    7413 anon_inode
python    10008 10018     ubuntu  141u     0000     0,9          0    7413 anon_inode
python    10008 10018     ubuntu  142u     IPv4  174361        0t0     TCP localhost:35954->localhost:46184 (ESTABLISHED)
python    10008 10018     ubuntu  143u     IPv4  117290        0t0     TCP 10.0.6.245:8888->192.168.200.48:56788 (ESTABLISHED)
python    10008 10018     ubuntu  144u     0000     0,9          0    7413 anon_inode
python    10008 10018     ubuntu  145u     IPv4  174362        0t0     TCP localhost:38554->localhost:58600 (ESTABLISHED)
python    10008 10018     ubuntu  146u     IPv4  113877        0t0     TCP 10.0.6.245:8888->192.168.200.48:56793 (ESTABLISHED)
python    10008 10018     ubuntu  147u     0000     0,9          0    7413 anon_inode
python    10008 10018     ubuntu  148u     IPv4  174363        0t0     TCP localhost:34926->localhost:40676 (ESTABLISHED)
python    10008 10018     ubuntu  149u     IPv4  114072        0t0     TCP 10.0.6.245:8888->192.168.200.48:54334 (ESTABLISHED)
python    10008 10018     ubuntu  150u     0000     0,9          0    7413 anon_inode
python    10008 10018     ubuntu  151u     0000     0,9          0    7413 anon_inode
```



Discussion:
Not sure if this is helpful, but I am seeing this on Chrome developer console:

```
Kernel started:  288687fd-68c5-4ece-bec6-a0faec14eda82
Starting WebSockets: ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda82
Uncaught InvalidStateError: Failed to execute 'send' on 'WebSocket': Still in CONNECTING state. IP:8888/static/services/kernels/js/kernel.js?v=4f7e3edf038579ed584b87f5c10a24ad:239
WebSocket connection to 'ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda8/iopub' failed: WebSocket is closed before the connection is established. IP:8888/notebooks/git/stuff/Python/stuff/src/Demo%20Segment.ipynb:1
WebSocket connection to 'ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda8/stdin' failed: WebSocket is closed before the connection is established. IP:8888/notebooks/git/stuff/Python/stuff/src/Demo%20Segment.ipynb:1
Starting WebSockets: ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda8
Failed to load resource: net::ERR_CONNECTION_TIMED_OUT http://IP:8888/api/notebooks/git/stuff/Python/stuff/src/Demo%20Segment.ipynb
Failed to load resource: net::ERR_NETWORK_CHANGED http://IP:8888/api/notebooks/git/stuff/Python/stuff/src/Demo%20Segment.ipynb
Uncaught InvalidStateError: Failed to execute 'send' on 'WebSocket': Still in CONNECTING state. IP:8888/static/services/kernels/js/kernel.js?v=4f7e3edf038579ed584b87f5c10a24ad:239
WebSocket connection to 'ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda8/iopub' failed: WebSocket is closed before the connection is established. IP:8888/notebooks/git/stuff/Python/stuff/src/Demo%20Segment.ipynb:1
WebSocket connection to 'ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda8/stdin' failed: WebSocket is closed before the connection is established. IP:8888/notebooks/git/stuff/Python/stuff/src/Demo%20Segment.ipynb:1
Starting WebSockets: ws://IP:8888/api/kernels/288687fd-68c5-4ece-bec6-a0faec14eda8
```

Can you describe your usage a bit more (long-lived server, long running notebooks, many open at once, refreshing the page often, etc.)? Also please include `IPython.sys_info()`.

`{'commit_hash': 'd87a8cd',
 'commit_source': 'installation',
 'default_encoding': 'UTF-8',
 'ipython_path': '/usr/local/lib/python2.7/dist-packages/IPython',
 'ipython_version': '2.2.0',
 'os_name': 'posix',
 'platform': 'Linux-3.11.0-15-generic-x86_64-with-Ubuntu-13.10-saucy',
 'sys_executable': '/usr/bin/python',
 'sys_platform': 'linux2',
 'sys_version': '2.7.5+ (default, Feb 27 2014, 19:37:08) \n[GCC 4.8.1]'}`

It is a long running server and long running notebook.
I do refresh the page occasionally when the following happens: https://github.com/ipython/ipython/issues/6186

I am connecting over VPN. This might be related to the VPN connection going down every hour or so.

Ah, I bet it is that VPN, then, with the connection never closing cleanly. I think we might be able to handle this with the websocket heartbeat, closing the connection when no heartbeat is received after some timeout.


Resolving patch:
diff --git a/IPython/html/base/zmqhandlers.py b/IPython/html/base/zmqhandlers.py
--- a/IPython/html/base/zmqhandlers.py
+++ b/IPython/html/base/zmqhandlers.py
@@ -109,6 +109,25 @@ def allow_draft76(self):
 
 class AuthenticatedZMQStreamHandler(ZMQStreamHandler, IPythonHandler):
     ping_callback = None
+    last_pong = 0
+    
+    @property
+    def ping_interval(self):
+        """The interval for websocket keep-alive pings.
+        
+        Set ws_ping_interval = 0 to disable pings.
+        """
+        return self.settings.get('ws_ping_interval', WS_PING_INTERVAL)
+    
+    @property
+    def ping_timeout(self):
+        """If no ping is received in this many milliseconds,
+        close the websocket connection (VPNs, etc. can fail to cleanly close ws connections).
+        Default is max of 3 pings or 30 seconds.
+        """
+        return self.settings.get('ws_ping_timeout',
+            max(3 * self.ping_interval, WS_PING_INTERVAL)
+        )
 
     def set_default_headers(self):
         """Undo the set_default_headers in IPythonHandler
@@ -129,16 +148,30 @@ def open(self, kernel_id):
         self.session = Session(config=self.config)
         self.save_on_message = self.on_message
         self.on_message = self.on_first_message
-        self.ping_callback = ioloop.PeriodicCallback(self.send_ping, WS_PING_INTERVAL)
-        self.ping_callback.start()
+        
+        # start the pinging
+        if self.ping_interval > 0:
+            self.last_pong = ioloop.IOLoop.instance().time()
+            self.ping_callback = ioloop.PeriodicCallback(self.send_ping, self.ping_interval)
+            self.ping_callback.start()
 
     def send_ping(self):
         """send a ping to keep the websocket alive"""
         if self.stream.closed() and self.ping_callback is not None:
             self.ping_callback.stop()
             return
+        
+        # check for timeout on pong
+        since_last_pong = 1e3 * (ioloop.IOLoop.instance().time() - self.last_pong)
+        if since_last_pong > self.ping_timeout:
+            self.log.warn("WebSocket ping timeout after %i ms.", since_last_pong)
+            self.close()
+            return
 
         self.ping(b'')
+    
+    def on_pong(self, data):
+        self.last_pong = ioloop.IOLoop.instance().time()
 
     def _inject_cookie_message(self, msg):
         """Inject the first message, which is the document cookie,


Test patch: