Duplicate Response headers become merged in Response objects

Currently any duplicate header on responses gets merged on the response.headers dict, using a comma to delimit de two different header values. For example

HTTP/1.1 200 OK
HeaderX: Value1
HeaderX: Value2

> print response.headers
> { 'headerx' : ' Value1, Value2' }

This makes impossible to detect responses with duplicate headers, particularly when header values already include commas by themselves (e.g., 'last-modified') 

Potential alternative repr:

> print response.headers
> { 'headerx' : ['Value1', 'Value2'] } 

Problem: breaks backward compatibility as scripts expecting a string would get a list

> print response.headers
> { 'headerx' : 'Value1 <SEP> 'Value2'}

Where SEP would be a separator that can't be found on values (newline?)


Hi @olemoudi, thanks for raising this issue!

This is actually being done in urllib3, the library that manages our HTTP connections. Specifically, [here](https://github.com/shazow/urllib3/blob/master/urllib3/response.py#L221).

Ordinarily I'd say that you should raise this issue on the urllib3 repository (and you are obviously free to do so), but in this case I don't consider this to be a bug. [RFC 2616 has the following to say](http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2):

> "Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma.

Given that there is a defined behaviour for multiple header values, we would need a very good argument for complicating the general (RFC specified and correct) case for the sake of making easier the unusual (contrary to RFC and incorrect) case.


Makes perfect sense!

Thanks for the reply. Keep up the great work.

Martin

On Tue, Jun 11, 2013 at 10:26 AM, Cory Benfield notifications@github.comwrote:

> Hi @olemoudi https://github.com/olemoudi, thanks for raising this issue!
> 
> This is actually being done in urllib3, the library that manages our HTTP
> connections. Specifically, herehttps://github.com/shazow/urllib3/blob/master/urllib3/response.py#L221
> .
> 
> Ordinarily I'd say that you should raise this issue on the urllib3
> repository (and you are obviously free to do so), but in this case I don't
> consider this to be a bug. RFC 2616 has the following to sayhttp://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
> :
> 
> "Multiple message-header fields with the same field-name MAY be present in
> a message if and only if the entire field-value for that header field is
> defined as a comma-separated list. It MUST be possible to combine the
> multiple header fields into one "field-name: field-value" pair, without
> changing the semantics of the message, by appending each subsequent
> field-value to the first, each separated by a comma.
> 
> Given that there is a defined behaviour for multiple header values, we
> would need a very good argument for complicating the general (RFC specified
> and correct) case for the sake of making easier the unusual (contrary to
> RFC and incorrect) case.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1415#issuecomment-19248592
> .


@Lukasa 
You linked to the master branch which is a moving target which made your link obsolete by now... Please remember to always link to specific revision/tag :)

@olemoudi 
Merging headers often is asking for trouble. See http://bugs.python.org/issue1660009 for details. This is fixed in Python 3 - headers are not being merged. See http://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar/0016.html for an example of problems with merged headers.


AFAIK, a case which deviates from RFC 2616 but it's still useful "in practice" is if you have a download that contains unicode characters. Based on my research (ie links from http://stackoverflow.com/questions/93551/how-to-encode-the-filename-parameter-of-content-disposition-header-in-http :smile:) you should do the following:

- Add a `Content-Disposition` with the classic `attachment; filename="%s"` value (but make sure to only include ascii characters. This is for legacy browsers
- Add a second `Content-Disposition` header with the `attachment; filename*=UTF-8''%s` format where the filename is UTF-8 encoded and then percent encoded for browser