Repository: scrapy/w3lib

Issue:
allow to customize encoding in w3lib.http.basic_auth_header 
See https://github.com/scrapy/scrapy/pull/2530

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/scrapy/w3lib/pull/100?src=pr&el=h1) Report
> Merging [#100](https://codecov.io/gh/scrapy/w3lib/pull/100?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/w3lib/commit/0d365519ecac8da99fd051ea1a522486b13a7e8e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.


```diff
@@           Coverage Diff           @@
##           master     #100   +/-   ##
=======================================
  Coverage   94.88%   94.88%           
=======================================
  Files           7        7           
  Lines         469      469           
  Branches       95       95           
=======================================
  Hits          445      445           
  Misses         16       16           
  Partials        8        8
```

| [Impacted Files](https://codecov.io/gh/scrapy/w3lib/pull/100?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [w3lib/http.py](https://codecov.io/gh/scrapy/w3lib/pull/100/diff?src=pr&el=tree#diff-dzNsaWIvaHR0cC5weQ==) | `100% <100%> (ø)` | :arrow_up: |

kmike: Could you please rebase it, to fix CI? Code looks good to me.

Patch:
diff --git a/w3lib/http.py b/w3lib/http.py
index accfb5d..c7b94a2 100644
--- a/w3lib/http.py
+++ b/w3lib/http.py
@@ -78,7 +78,7 @@ def headers_dict_to_raw(headers_dict):
     return b'\r\n'.join(raw_lines)
 
 
-def basic_auth_header(username, password):
+def basic_auth_header(username, password, encoding='ISO-8859-1'):
     """
     Return an `Authorization` header field value for `HTTP Basic Access Authentication (RFC 2617)`_
 
@@ -95,5 +95,5 @@ def basic_auth_header(username, password):
         # XXX: RFC 2617 doesn't define encoding, but ISO-8859-1
         # seems to be the most widely used encoding here. See also:
         # http://greenbytes.de/tech/webdav/draft-ietf-httpauth-basicauth-enc-latest.html
-        auth = auth.encode('ISO-8859-1')
+        auth = auth.encode(encoding)
     return b'Basic ' + urlsafe_b64encode(auth)


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
index 453624f..01f903e 100644
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
 import unittest
 from collections import OrderedDict
 from w3lib.http import (basic_auth_header,
@@ -14,6 +16,13 @@ class HttpTests(unittest.TestCase):
         self.assertEqual(b'Basic c29tZXVzZXI6QDx5dTk-Jm8_UQ==',
             basic_auth_header('someuser', '@<yu9>&o?Q'))
 
+    def test_basic_auth_header_encoding(self):
+        self.assertEqual(b'Basic c29tw6Z1c8Oocjpzw7htZXDDpHNz',
+                basic_auth_header(u'somæusèr', u'sømepäss', encoding='utf8'))
+        # default encoding (ISO-8859-1)
+        self.assertEqual(b'Basic c29t5nVz6HI6c_htZXDkc3M=',
+                basic_auth_header(u'somæusèr', u'sømepäss'))
+
     def test_headers_raw_dict_none(self):
         self.assertIsNone(headers_raw_to_dict(None))
         self.assertIsNone(headers_dict_to_raw(None))