Repository: aio-libs/aiohttp

Issue:
Possibility to disable Server header
## Long story short

By default, aiohttp server adds `Server` header. It should be possible to disable this, for security reasons.

## Expected behaviour

No `Server` header in response

## Actual behaviour

Server sends these headers:

    Content-Type: text/plain; charset=utf-8
    Content-Length: 2
    Date: Tue, 06 Jun 2017 13:52:03 GMT
    Server: Python/3.6 aiohttp/2.1.0

## Steps to reproduce

    import asyncio
    from aiohttp import web


    async def http_handler(request):
        return web.Response(text="OK", headers={})


    def main():
        loop = asyncio.get_event_loop()
        loop.run_until_complete(loop.create_server(web.Server(http_handler), "127.0.0.1", 8080))
        loop.run_forever()


    if __name__ == "__main__":
        main()


## Your environment

macOS 10.12.5
Python 3.6.1

    $ pip3 freeze
    aiofiles==0.3.0
    aiohttp==2.1.0
    async-timeout==1.2.1
    asyncio==3.4.3
    chardet==3.0.3
    Cython==0.25.2
    defusedxml==0.5.0
    google-api-python-client==1.6.2
    httplib2==0.10.3
    jira==1.0.10
    llvmlite==0.16.0
    magicmemoryview==0.1.5
    multidict==2.1.6
    numba==0.31.0
    numexpr==2.6.2
    numpy==1.12.0
    oauth2client==4.1.0
    oauthlib==2.0.2
    pbr==3.0.1
    pyasn1==0.2.3
    pyasn1-modules==0.0.8
    requests==2.13.0
    requests-oauthlib==0.8.0
    requests-toolbelt==0.7.1
    rsa==3.4.2
    six==1.10.0
    tornado==4.4.2
    uritemplate==3.0.0
    websockets==3.3
    yarl==0.10.2



Discussion:
You cannot disable `SERVER` header but could pass any value to it:
```
async def http_handler(request):
    return web.Response(text="OK", headers={'Server': 'noname'})
```
Or you could drop `Server` header at all on reverse proxy.
So, aiohttp is designed to be used _only_ behind reverse proxy, right?  
If that's true then I understand it.
I think that would be a classic security through obscurity. You should keep you software up to date and dont' let your application have a security holes.
@sp-1234 requirements for aiohttp are very close to any python web server.
It *might* be used without any frontend for simple cases. But in practice you'll probably put it behind reverse proxy for very many reasons.
@kxepal nobody said that I won't be updating software. But from Defense in Depth™ principle, it's not good to make attacker's job easier if somehow updated software becomes _not enough_ for a moment.

@asvetlov OK, it sounds plausible 😎 
The conclusions made here are nonsense; as a developer I should have the ability to control something as subjective as the `Server` response header, including the ability to shut it off entirely.

This has nothing to do with security - this is seemingly the maintainers trying to plug their own software.

Would you accept a PR to allow disabling it?
> Would you accept a PR to allow disabling it?

Disable what?
The Server response header.
Does [on_response_prepare](https://docs.aiohttp.org/en/stable/web_reference.html#aiohttp.web.Application.on_response_prepare) signal satisfy your needs?
@asvetlov No, it appears that `Server` is added after that signal. Therefore, `if "Server" in res.headers: del res.headers["Server"]` does not work, and setting it to `None` results in a concatenation error later on in the aiohttp response pipeline.
Aha, I see.
If you insist that `Server` should be deleted please make a PR that splits `StreamRequest._start` in two methods: one makes all header preparations and called before the signal raising, another one is bare `writer.write_headers()` call which is executed after `on_response_prepare()`.
@asvetlov Alternatively, could we just add a `removedefault()` method from `CIMultiDict` in the `multidict` package? Then the `on_response_prepare` handler can just do `res.headers.removedefault("Server")` which is just as descriptive.
`removedefault` is... unusual.
What the method should do?
Reopened since we have a discussion here.
P.S. I hate necromancy in general and reviving already closed issues after **years** of been closed.
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/web_response.py#L384

This is where it's being set. `multidict` uses a default if no keys have been set in the dictionary, which gives the effect of a normal dictionary in the case no multi-value-single-key entries have been submitted to the dictionary.

I would imagine there hasn't been a usecase for removing a default until now, but IMO it's a very solid usecase.

> P.S. I hate necromancy in general and reviving already closed issues after years of been closed.

Yes, same, but a new issue would have duplicated most of the conversation here. I necro'd since there's already the context and I disagreed with the outcome. Further, on Github, this is less of an issue organizationally than on traditional, archaic boards.

This was the most apt place to raise dissent in my opinion, hence the necro. I hope it didn't cause too much of an inconvenience.
`multidict.setdefault(k, v)` has the following pseudo-code:
```
if k not in multidict:
    multidict.add(k, v)
```
What is pseudo-code for `multidict.removedefault(...)`?
Ah I understand now, what I wrote earlier was nonsense.

Submitting a PR with your proposed fixes @asvetlov  :)


Resolving patch:
diff --git a/aiohttp/web_response.py b/aiohttp/web_response.py
--- a/aiohttp/web_response.py
+++ b/aiohttp/web_response.py
@@ -346,19 +346,29 @@ async def prepare(
         if self._payload_writer is not None:
             return self._payload_writer
 
-        await request._prepare_hook(self)
         return await self._start(request)
 
     async def _start(self, request: 'BaseRequest') -> AbstractStreamWriter:
         self._req = request
+        writer = self._payload_writer = request._payload_writer
+
+        await self._prepare_headers()
+        await request._prepare_hook(self)
+        await self._write_headers()
+
+        return writer
 
+    async def _prepare_headers(self) -> None:
+        request = self._req
+        assert request is not None
+        writer = self._payload_writer
+        assert writer is not None
         keep_alive = self._keep_alive
         if keep_alive is None:
             keep_alive = request.keep_alive
         self._keep_alive = keep_alive
 
         version = request.version
-        writer = self._payload_writer = request._payload_writer
 
         headers = self._headers
         for cookie in self._cookies.values():
@@ -401,12 +411,16 @@ async def _start(self, request: 'BaseRequest') -> AbstractStreamWriter:
                 if version == HttpVersion11:
                     headers[hdrs.CONNECTION] = 'close'
 
+    async def _write_headers(self) -> None:
+        request = self._req
+        assert request is not None
+        writer = self._payload_writer
+        assert writer is not None
         # status line
+        version = request.version
         status_line = 'HTTP/{}.{} {} {}'.format(
             version[0], version[1], self._status, self._reason)
-        await writer.write_headers(status_line, headers)
-
-        return writer
+        await writer.write_headers(status_line, self._headers)
 
     async def write(self, data: bytes) -> None:
         assert isinstance(data, (bytes, bytearray, memoryview)), \


Test patch:
diff --git a/tests/test_web_response.py b/tests/test_web_response.py
--- a/tests/test_web_response.py
+++ b/tests/test_web_response.py
@@ -1092,6 +1092,31 @@ def test_response_with_immutable_headers() -> None:
                             'Content-Type': 'text/plain; charset=utf-8'}
 
 
+async def test_response_prepared_after_header_preparation() -> None:
+    req = make_request('GET', '/')
+    resp = StreamResponse()
+    await resp.prepare(req)
+
+    assert type(resp.headers['Server']) is str
+
+    async def _strip_server(req, res):
+        assert 'Server' in res.headers
+
+        if 'Server' in res.headers:
+            del res.headers['Server']
+
+    app = mock.Mock()
+    sig = signals.Signal(app)
+    sig.append(_strip_server)
+
+    req = make_request(
+        'GET', '/', on_response_prepare=sig, app=app)
+    resp = StreamResponse()
+    await resp.prepare(req)
+
+    assert 'Server' not in resp.headers
+
+
 def test_weakref_creation() -> None:
     resp = Response()
     weakref.ref(resp)