Repository: aio-libs/aiohttp

Issue:
100% CPU usage with idle websocket
## Long story short

Fairly standard websocket server/client setup, but with session, results in 100% CPU usage. 


1. Generic websocket server
2. Generic websocket client, but with a session that does a `get` request (to the same endpoint) before a new request to start a websocket
3. In a few minutes, CPU usage increases to 100%

Note that removing the `get` request seems to avoid the problem.


## Expected behaviour
Low CPU usage when idle

## Actual behaviour
Worker process suddenly goes to 100% CPU usage, confirmed with htop.

## Steps to reproduce
Server
```
import asyncio
import aiohttp
from aiohttp import web

async def worker_websocket(request):
    ws = aiohttp.web.WebSocketResponse()
    await ws.prepare(request)

    async for msg in ws:
        pass

    return ws

app = aiohttp.web.Application()
app.router.add_get('/worker-websocket',worker_websocket)
web.run_app( app, port=int(8080))
```
worker
```
import aiohttp
from aiohttp import web
import asyncio

async def websocket_client(loop):
    url = 'http://localhost:8080/worker-websocket'

    session = aiohttp.ClientSession(loop=loop)

    # removing this prevents the problem!
    async with session.get(url) as resp:
        pass

    async with session.ws_connect(url=url) as ws:
        async for msg in ws:
            pass

loop = asyncio.get_event_loop()
client = websocket_client(loop)
loop.run_until_complete(client)
```

Run server + worker and wait a few minutes. Eventually the server process will eat one core. Remove the session.get and it does not happen.

## Your environment
I've reproduced this, so far, on:
* MacOS 10.12 (Sierra) + Python 3.6.1 with aiohttp 2.1.0
* Ubuntu server 16.04 LTS + Python 3.5.2 with aiohttp 2.1.0


----

Using `strace`, all I can see is highly frequent `epoll_wait` calls returning 0. My attempts with `pyrasite` and `gdb-python3` yielded nothing useful.

I'm still yet to try `ipdb` and try to look through the source.

The only thing I can find that is even remotely similar is https://groups.google.com/forum/#!topic/libuv/opFk74lww7k .


Otherwise my experience with `aiohttp` has been fantastic. Thanks for a great library.



Discussion:
Confirmed on Windows too.

Windows 7 Enterprise 64bit + Python 3.5.1 amd64 + aiohttp 2.1.0

Usage is show as 25% as it's running on a quad-core machine: 
![image](https://cloud.githubusercontent.com/assets/1956773/26820570/30480f2c-4a9b-11e7-963e-9fba22c0c769.png)

Oddly, usage spikes suddenly?

![image](https://cloud.githubusercontent.com/assets/1956773/26820752/dbcf52a6-4a9b-11e7-830b-6699cfe3afbb.png)


@naggie @frillip could you test master branch
I can confirm it's still an issue on master, commit 6b85bcd82985c132c84acbc2b565997d7b512b45
It seems like a problem related to keep-alive. When I tested the code above, `web_protocol._process_keepalive` is called infinitely after 75 seconds later (which is default value of keepalive_timeout) from calling `session.get`.

```
    def _process_keepalive(self):
        if self._force_close:
            return

        # 1. "self._keepalive_time" is not updated during waiting "async for msg in ws:"
        # therefore the value of "next" is unchanged.
        next = self._keepalive_time + self._keepalive_timeout

        if len(self._request_handlers) == len(self._waiters):  # <-- 2. no waiters during "async for msg in ws:"
            now = self._time_service.loop_time
            if now + 1.0 > next:
                self.force_close(send_last_heartbeat=True)
                return

        # 3. Therefore, the value of "next" is current time, which calls "self._process_keepalive" immediately.
        self._keepalive_handle = self._loop.call_at(
            next, self._process_keepalive)
```
should be fixed in master
Fantastic! Thank you all for your time!
I have found that this still results in a 100% cpu usage situation after the default timeout of 75 seconds, using it in combination with python-socketio. If needed, I can provide new evidence. 

aiohttp version 2.3.1
Same issue happened to me, any hint on how to debug this issue?
@ericdevries @zTrix could you provide reproducible example?

@zTrix do you use python-socketio as well?
@fafhrd91 sure, I will create one tomorrow and attach it to this ticket
@fafhrd91 No I didn't use python-socketio.

In my case I just started an server with websocket, and diving into `_process_keepalive` function shows that  `next` actually is less than `_loop.time()`, but `len(self._request_handlers) == len(self._waiters)` is `False`, so it's calling itself just like an infinite loop.

Wild idea: disable HTTP keepalives after websocket connection establishment. They are useless, isn't it?
Also looks like `RequestHandler._process_keepalive` misses `self._keepalive_time = self._loop.time()` line.

Resolving patch:
diff --git a/aiohttp/web_protocol.py b/aiohttp/web_protocol.py
--- a/aiohttp/web_protocol.py
+++ b/aiohttp/web_protocol.py
@@ -80,6 +80,7 @@ class RequestHandler(asyncio.streams.FlowControlMixin, asyncio.Protocol):
     """
     _request_count = 0
     _keepalive = False  # keep transport open
+    KEEPALIVE_RESCHEDULE_DELAY = 1
 
     def __init__(self, manager, *, loop=None,
                  keepalive_timeout=75,  # NGINX default value is 75 secs
@@ -321,6 +322,9 @@ def keep_alive(self, val):
         :param bool val: new state.
         """
         self._keepalive = val
+        if self._keepalive_handle:
+            self._keepalive_handle.cancel()
+            self._keepalive_handle = None
 
     def close(self):
         """Stop accepting new pipelinig messages and close
@@ -352,7 +356,7 @@ def log_exception(self, *args, **kw):
         self.logger.exception(*args, **kw)
 
     def _process_keepalive(self):
-        if self._force_close:
+        if self._force_close or not self._keepalive:
             return
 
         next = self._keepalive_time + self._keepalive_timeout
@@ -363,8 +367,10 @@ def _process_keepalive(self):
                 self.force_close(send_last_heartbeat=True)
                 return
 
-        self._keepalive_handle = self._loop.call_at(
-            next, self._process_keepalive)
+        # not all request handlers are done,
+        # reschedule itself to next second
+        self._keepalive_handle = self._loop.call_later(
+            self.KEEPALIVE_RESCHEDULE_DELAY, self._process_keepalive)
 
     def pause_reading(self):
         if not self._reading_paused:
diff --git a/aiohttp/web_ws.py b/aiohttp/web_ws.py
--- a/aiohttp/web_ws.py
+++ b/aiohttp/web_ws.py
@@ -133,6 +133,8 @@ def _post_start(self, request, protocol, writer):
             request._protocol, limit=2 ** 16, loop=self._loop)
         request.protocol.set_parser(WebSocketReader(
             self._reader, compress=self._compress))
+        # disable HTTP keepalive for WebSocket
+        request.protocol.keep_alive(False)
 
     def can_prepare(self, request):
         if self._writer is not None:


Test patch:
diff --git a/tests/test_web_protocol.py b/tests/test_web_protocol.py
--- a/tests/test_web_protocol.py
+++ b/tests/test_web_protocol.py
@@ -11,7 +11,7 @@
 from aiohttp import helpers, http, streams, web
 
 
-@pytest.yield_fixture
+@pytest.fixture
 def make_srv(loop, manager):
     srv = None
 
@@ -72,12 +72,12 @@ def handle(request):
     return wrapper
 
 
-@pytest.yield_fixture
+@pytest.fixture
 def writer(srv):
     return http.PayloadWriter(srv.writer, srv._loop)
 
 
-@pytest.yield_fixture
+@pytest.fixture
 def transport(buf):
     transport = mock.Mock()
 
@@ -204,7 +204,7 @@ def test_connection_made(make_srv):
     assert not srv._force_close
 
 
-def test_connection_made_with_keepaplive(make_srv, transport):
+def test_connection_made_with_tcp_keepaplive(make_srv, transport):
     srv = make_srv()
 
     sock = mock.Mock()
@@ -214,7 +214,7 @@ def test_connection_made_with_keepaplive(make_srv, transport):
                                        socket.SO_KEEPALIVE, 1)
 
 
-def test_connection_made_without_keepaplive(make_srv):
+def test_connection_made_without_tcp_keepaplive(make_srv):
     srv = make_srv(tcp_keepalive=False)
 
     sock = mock.Mock()
@@ -260,6 +260,15 @@ def test_srv_keep_alive(srv):
     assert not srv._keepalive
 
 
+def test_srv_keep_alive_disable(srv):
+    handle = srv._keepalive_handle = mock.Mock()
+
+    srv.keep_alive(False)
+    assert not srv._keepalive
+    assert srv._keepalive_handle is None
+    handle.cancel.assert_called_with()
+
+
 def test_slow_request(make_srv):
     with pytest.warns(DeprecationWarning):
         make_srv(slow_request_timeout=0.01)
@@ -583,6 +592,7 @@ def test_handle_500(srv, loop, buf, transport, request_handler):
 @asyncio.coroutine
 def test_keep_alive(make_srv, loop, transport, ceil):
     srv = make_srv(keepalive_timeout=0.05)
+    srv.KEEPALIVE_RESCHEDULE_DELAY = 0.1
     srv.connection_made(transport)
 
     srv.keep_alive(True)
@@ -600,7 +610,7 @@ def test_keep_alive(make_srv, loop, transport, ceil):
     assert srv._keepalive_handle is not None
     assert not transport.close.called
 
-    yield from asyncio.sleep(0.1, loop=loop)
+    yield from asyncio.sleep(0.2, loop=loop)
     assert transport.close.called
     assert srv._waiters[0].cancelled
 
diff --git a/tests/test_web_websocket_functional.py b/tests/test_web_websocket_functional.py
--- a/tests/test_web_websocket_functional.py
+++ b/tests/test_web_websocket_functional.py
@@ -758,3 +758,32 @@ def handler(request):
     yield from ws.receive()
 
     assert cancelled
+
+
+@asyncio.coroutine
+def test_websocket_disable_keepalive(loop, test_client):
+    @asyncio.coroutine
+    def handler(request):
+        ws = web.WebSocketResponse()
+        if not ws.can_prepare(request):
+            return web.Response(text='OK')
+        assert request.protocol._keepalive
+        yield from ws.prepare(request)
+        assert not request.protocol._keepalive
+        assert not request.protocol._keepalive_handle
+
+        yield from ws.send_str('OK')
+        yield from ws.close()
+        return ws
+
+    app = web.Application()
+    app.router.add_route('GET', '/', handler)
+    client = yield from test_client(app)
+
+    resp = yield from client.get('/')
+    txt = yield from resp.text()
+    assert txt == 'OK'
+
+    ws = yield from client.ws_connect('/')
+    data = yield from ws.receive_str()
+    assert data == 'OK'