Repository: scrapy/scrapy

Issue:
CsvItemExporter fails on py3
```
from scrapy.exporters import CsvItemExporter
with open('temp.csv', 'w') as f:
    exporter = CsvItemExporter(f)
    exporter.start_exporting()
    exporter.export_item({'a': 'b'})
    exporter.finish_exporting()
```

The previous snippet works fine on python 2, however the following error appears when running on python 3:

```
Traceback (most recent call last):
  File "exporter.py", line 5, in <module>
    exporter.export_item({'a': 'b'})
  File "/Users/eugenio/.../venv/lib/python3.5/site-packages/scrapy/exporters.py", line 206, in export_item
    self._write_headers_and_set_fields_to_export(item)
  File "/Users/eugenio/.../venv/lib/python3.5/site-packages/scrapy/exporters.py", line 230, in _write_headers_and_set_fields_to_export
    self.csv_writer.writerow(row)
TypeError: write() argument must be str, not bytes
```

Am I missing something? Thanks!

Discussion:
kmike: While XmlItemExporter works with text files, I'm not sure it is intentional. For example, 'encoding' argument doesn't work properly with XmlItemExporter + text files.
elacuesta: Right, the following:
```
from io import StringIO

with StringIO(u'') as f:
    exporter = XmlItemExporter(f, encoding='utf8')
    exporter.start_exporting()
    exporter.export_item({'some': u'valûe'})
    exporter.finish_exporting()
    print(f.getvalue())
```
Fails in python 2, it seems like the binary mode is more consistent. I'll add the note to the `XmlItemExporter` as well. Thanks!

Patch:
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 0ef3fb071..1ca78ccc6 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -681,9 +681,7 @@ HttpProxyMiddleware
    * ``no_proxy``
 
    You can also set the meta key ``proxy`` per-request, to a value like
-   ``http://some_proxy_server:port`` or ``http://username:password@some_proxy_server:port``.
-   Keep in mind this value will take precedence over ``http_proxy``/``https_proxy``
-   environment variables, and it will also ignore ``no_proxy`` environment variable.
+   ``http://some_proxy_server:port``.
 
 .. _urllib: https://docs.python.org/2/library/urllib.html
 .. _urllib2: https://docs.python.org/2/library/urllib2.html
@@ -951,16 +949,8 @@ enable it for :ref:`broad crawls <topics-broad-crawls>`.
 HttpProxyMiddleware settings
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. setting:: HTTPPROXY_ENABLED
 .. setting:: HTTPPROXY_AUTH_ENCODING
 
-HTTPPROXY_ENABLED
-^^^^^^^^^^^^^^^^^
-
-Default: ``True``
-
-Whether or not to enable the :class:`HttpProxyMiddleware`.
-
 HTTPPROXY_AUTH_ENCODING
 ^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst
index af469eb7b..85c73222d 100644
--- a/docs/topics/exporters.rst
+++ b/docs/topics/exporters.rst
@@ -225,7 +225,8 @@ XmlItemExporter
 
    Exports Items in XML format to the specified file object.
 
-   :param file: the file-like object to use for exporting the data.
+   :param file: the file-like object to use for exporting the data. Its ``write`` method should
+                accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
 
    :param root_element: The name of root element in the exported XML.
    :type root_element: str
@@ -281,7 +282,8 @@ CsvItemExporter
    CSV columns and their order. The :attr:`export_empty_fields` attribute has
    no effect on this exporter.
 
-   :param file: the file-like object to use for exporting the data.
+   :param file: the file-like object to use for exporting the data. Its ``write`` method should
+                accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
 
    :param include_headers_line: If enabled, makes the exporter output a header
       line with the field names taken from
@@ -312,7 +314,8 @@ PickleItemExporter
 
    Exports Items in pickle format to the given file-like object.
 
-   :param file: the file-like object to use for exporting the data.
+   :param file: the file-like object to use for exporting the data. Its ``write`` method should
+                accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
 
    :param protocol: The pickle protocol to use.
    :type protocol: int
@@ -333,7 +336,8 @@ PprintItemExporter
 
    Exports Items in pretty print format to the specified file object.
 
-   :param file: the file-like object to use for exporting the data.
+   :param file: the file-like object to use for exporting the data. Its ``write`` method should
+                accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
 
    The additional keyword arguments of this constructor are passed to the
    :class:`BaseItemExporter` constructor.
@@ -356,7 +360,8 @@ JsonItemExporter
    arguments to the `JSONEncoder`_ constructor, so you can use any
    `JSONEncoder`_ constructor argument to customize this exporter.
 
-   :param file: the file-like object to use for exporting the data.
+   :param file: the file-like object to use for exporting the data. Its ``write`` method should
+                accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
 
    A typical output of this exporter would be::
 
@@ -386,7 +391,8 @@ JsonLinesItemExporter
    the `JSONEncoder`_ constructor, so you can use any `JSONEncoder`_
    constructor argument to customize this exporter.
 
-   :param file: the file-like object to use for exporting the data.
+   :param file: the file-like object to use for exporting the data. Its ``write`` method should
+                accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc)
 
    A typical output of this exporter would be::
 
diff --git a/scrapy/downloadermiddlewares/httpproxy.py b/scrapy/downloadermiddlewares/httpproxy.py
index 0d5320bf8..98c87aa9c 100644
--- a/scrapy/downloadermiddlewares/httpproxy.py
+++ b/scrapy/downloadermiddlewares/httpproxy.py
@@ -20,25 +20,23 @@ class HttpProxyMiddleware(object):
         for type, url in getproxies().items():
             self.proxies[type] = self._get_proxy(url, type)
 
+        if not self.proxies:
+            raise NotConfigured
+
     @classmethod
     def from_crawler(cls, crawler):
-        if not crawler.settings.getbool('HTTPPROXY_ENABLED'):
-            raise NotConfigured
         auth_encoding = crawler.settings.get('HTTPPROXY_AUTH_ENCODING')
         return cls(auth_encoding)
 
-    def _basic_auth_header(self, username, password):
-        user_pass = to_bytes(
-            '%s:%s' % (unquote(username), unquote(password)),
-            encoding=self.auth_encoding)
-        return base64.b64encode(user_pass).strip()
-
     def _get_proxy(self, url, orig_type):
         proxy_type, user, password, hostport = _parse_proxy(url)
         proxy_url = urlunparse((proxy_type or orig_type, hostport, '', '', '', ''))
 
         if user:
-            creds = self._basic_auth_header(user, password)
+            user_pass = to_bytes(
+                '%s:%s' % (unquote(user), unquote(password)),
+                encoding=self.auth_encoding)
+            creds = base64.b64encode(user_pass).strip()
         else:
             creds = None
 
@@ -47,15 +45,6 @@ class HttpProxyMiddleware(object):
     def process_request(self, request, spider):
         # ignore if proxy is already set
         if 'proxy' in request.meta:
-            if request.meta['proxy'] is None:
-                return
-            # extract credentials if present
-            creds, proxy_url = self._get_proxy(request.meta['proxy'], '')
-            request.meta['proxy'] = proxy_url
-            if creds and not request.headers.get('Proxy-Authorization'):
-                request.headers['Proxy-Authorization'] = b'Basic ' + creds
-            return
-        elif not self.proxies:
             return
 
         parsed = urlparse_cached(request)
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index cb88bc2bf..24714a7a8 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -174,7 +174,6 @@ HTTPCACHE_DBM_MODULE = 'anydbm' if six.PY2 else 'dbm'
 HTTPCACHE_POLICY = 'scrapy.extensions.httpcache.DummyPolicy'
 HTTPCACHE_GZIP = False
 
-HTTPPROXY_ENABLED = True
 HTTPPROXY_AUTH_ENCODING = 'latin-1'
 
 IMAGES_STORE_S3_ACL = 'private'


Test patch:
diff --git a/tests/test_downloadermiddleware_httpproxy.py b/tests/test_downloadermiddleware_httpproxy.py
index c77179ceb..2b26431a4 100644
--- a/tests/test_downloadermiddleware_httpproxy.py
+++ b/tests/test_downloadermiddleware_httpproxy.py
@@ -1,14 +1,11 @@
 import os
 import sys
-from functools import partial
 from twisted.trial.unittest import TestCase, SkipTest
 
 from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware
 from scrapy.exceptions import NotConfigured
 from scrapy.http import Response, Request
 from scrapy.spiders import Spider
-from scrapy.crawler import Crawler
-from scrapy.settings import Settings
 
 spider = Spider('foo')
 
@@ -23,10 +20,9 @@ class TestDefaultHeadersMiddleware(TestCase):
     def tearDown(self):
         os.environ = self._oldenv
 
-    def test_not_enabled(self):
-        settings = Settings({'HTTPPROXY_ENABLED': False})
-        crawler = Crawler(spider, settings)
-        self.assertRaises(NotConfigured, partial(HttpProxyMiddleware.from_crawler, crawler))
+    def test_no_proxies(self):
+        os.environ = {}
+        self.assertRaises(NotConfigured, HttpProxyMiddleware)
 
     def test_no_enviroment_proxies(self):
         os.environ = {'dummy_proxy': 'reset_env_and_do_not_raise'}
@@ -51,13 +47,6 @@ class TestDefaultHeadersMiddleware(TestCase):
             self.assertEquals(req.url, url)
             self.assertEquals(req.meta.get('proxy'), proxy)
 
-    def test_proxy_precedence_meta(self):
-        os.environ['http_proxy'] = 'https://proxy.com'
-        mw = HttpProxyMiddleware()
-        req = Request('http://scrapytest.org', meta={'proxy': 'https://new.proxy:3128'})
-        assert mw.process_request(req, spider) is None
-        self.assertEquals(req.meta, {'proxy': 'https://new.proxy:3128'})
-
     def test_proxy_auth(self):
         os.environ['http_proxy'] = 'https://user:pass@proxy:3128'
         mw = HttpProxyMiddleware()
@@ -65,11 +54,6 @@ class TestDefaultHeadersMiddleware(TestCase):
         assert mw.process_request(req, spider) is None
         self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
         self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic dXNlcjpwYXNz')
-        # proxy from request.meta
-        req = Request('http://scrapytest.org', meta={'proxy': 'https://username:password@proxy:3128'})
-        assert mw.process_request(req, spider) is None
-        self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
-        self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic dXNlcm5hbWU6cGFzc3dvcmQ=')
 
     def test_proxy_auth_empty_passwd(self):
         os.environ['http_proxy'] = 'https://user:@proxy:3128'
@@ -78,11 +62,6 @@ class TestDefaultHeadersMiddleware(TestCase):
         assert mw.process_request(req, spider) is None
         self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
         self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic dXNlcjo=')
-        # proxy from request.meta
-        req = Request('http://scrapytest.org', meta={'proxy': 'https://username:@proxy:3128'})
-        assert mw.process_request(req, spider) is None
-        self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
-        self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic dXNlcm5hbWU6')
 
     def test_proxy_auth_encoding(self):
         # utf-8 encoding
@@ -93,12 +72,6 @@ class TestDefaultHeadersMiddleware(TestCase):
         self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
         self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic bcOhbjpwYXNz')
 
-        # proxy from request.meta
-        req = Request('http://scrapytest.org', meta={'proxy': u'https://\u00FCser:pass@proxy:3128'})
-        assert mw.process_request(req, spider) is None
-        self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
-        self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic w7xzZXI6cGFzcw==')
-
         # default latin-1 encoding
         mw = HttpProxyMiddleware(auth_encoding='latin-1')
         req = Request('http://scrapytest.org')
@@ -106,21 +79,15 @@ class TestDefaultHeadersMiddleware(TestCase):
         self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
         self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic beFuOnBhc3M=')
 
-        # proxy from request.meta, latin-1 encoding
-        req = Request('http://scrapytest.org', meta={'proxy': u'https://\u00FCser:pass@proxy:3128'})
-        assert mw.process_request(req, spider) is None
-        self.assertEquals(req.meta, {'proxy': 'https://proxy:3128'})
-        self.assertEquals(req.headers.get('Proxy-Authorization'), b'Basic /HNlcjpwYXNz')
-
     def test_proxy_already_seted(self):
-        os.environ['http_proxy'] = 'https://proxy.for.http:3128'
+        os.environ['http_proxy'] = http_proxy = 'https://proxy.for.http:3128'
         mw = HttpProxyMiddleware()
         req = Request('http://noproxy.com', meta={'proxy': None})
         assert mw.process_request(req, spider) is None
         assert 'proxy' in req.meta and req.meta['proxy'] is None
 
     def test_no_proxy(self):
-        os.environ['http_proxy'] = 'https://proxy.for.http:3128'
+        os.environ['http_proxy'] = http_proxy = 'https://proxy.for.http:3128'
         mw = HttpProxyMiddleware()
 
         os.environ['no_proxy'] = '*'
@@ -137,9 +104,3 @@ class TestDefaultHeadersMiddleware(TestCase):
         req = Request('http://noproxy.com')
         assert mw.process_request(req, spider) is None
         assert 'proxy' not in req.meta
-
-        # proxy from meta['proxy'] takes precedence
-        os.environ['no_proxy'] = '*'
-        req = Request('http://noproxy.com', meta={'proxy': 'http://proxy.com'})
-        assert mw.process_request(req, spider) is None
-        self.assertEquals(req.meta, {'proxy': 'http://proxy.com'})