Repository: jazzband/pip-tools

Issue:
[future] download_package: TypeError: 'str' does not support the buffer interface (Python 3)
I am seeing this error when trying to install Sphinx, using Python 3.

This is similar to issue #87, where an `UnicodeEncodeError` was thrown there
(for Python 2).

```
% cat test-sphinx.in
Sphinx
% pip-compile --verbose test-sphinx.in
===> Collecting source requirements                
Sphinx (from test-sphinx.in:1)

===> Normalizing source requirements
sphinx (from test-sphinx.in:1)

===> Resolving full tree
- Finding best package matching ['sphinx']
  Found best match: 1.3.1 (from PyPI)
- Getting dependencies for sphinx-1.3.1
    - Getting package location for sphinx==1.3.1
          Archive cache miss, downloading Sphinx-1.3.1-py2.py3-none-any.whl...
        - Downloading package from https://pypi.python.org/packages/2.7/S/Sphinx/Sphinx-1.3.1-py2.py3-none-any.whl
Traceback (most recent call last):
  File "…/pip-tools/build/venv3/bin/pip-compile", line 9, in <module>
    load_entry_point('pip-tools==1.0', 'console_scripts', 'pip-compile')()
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "…/pip-tools/piptools/scripts/compile.py", line 186, in cli
    compile_specs(src_files, include_sources=include_sources, dry_run=dry_run)
  File "…/pip-tools/piptools/scripts/compile.py", line 118, in compile_specs
    pinned_spec_set = resolver.resolve()
  File "…/pip-tools/piptools/resolver.py", line 44, in resolve
    if not self.resolve_one_round():
  File "…/pip-tools/piptools/resolver.py", line 28, in resolve_one_round
    new_deps = self.find_new_dependencies()
  File "…/pip-tools/piptools/resolver.py", line 91, in find_new_dependencies
    all_deps = self.find_all_dependencies()
  File "…/pip-tools/piptools/resolver.py", line 74, in find_all_dependencies
    pkg_deps = pkgmgr.get_dependencies(spec.pin(version))
  File "…/pip-tools/piptools/package_manager.py", line 362, in get_dependencies
    path = self.get_or_download_package(pinned_spec)
  File "…/pip-tools/piptools/package_manager.py", line 405, in get_or_download_package
    self.download_package(link, fullpath)
  File "…/pip-tools/piptools/package_manager.py", line 445, in download_package
    f.write(content)
TypeError: 'str' does not support the buffer interface
```

This can be fixed by using `io.BytesIO`, but I don't know if that's the correct way:

```
diff --git i/piptools/package_manager.py w/piptools/package_manager.py
index 4c411d8..4bf58f6 100644
--- i/piptools/package_manager.py
+++ w/piptools/package_manager.py
@@ -442,7 +442,8 @@ def download_package(self, link, destination):
         with logger.indent():
             _, content = get_file_content(url, link, session=self._session)
             with open(destination, 'wb') as f:
-                f.write(content)
+                from io import BytesIO
+                f.write(BytesIO(content))

     def unpack_archive(self, path, target_directory):
         logger.debug('- Unpacking %s' % (path,))
```

btw: trying it again it fails because of an empty zip, which could be handled better:

```
Traceback (most recent call last):
  File "…/pip-tools/build/venv3/bin/pip-compile", line 9, in <module>
    load_entry_point('pip-tools==1.0', 'console_scripts', 'pip-compile')()
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "…/pip-tools/build/venv3/lib/python3.4/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "…/pip-tools/piptools/scripts/compile.py", line 186, in cli
    compile_specs(src_files, include_sources=include_sources, dry_run=dry_run)
  File "…/pip-tools/piptools/scripts/compile.py", line 118, in compile_specs
    pinned_spec_set = resolver.resolve()
  File "…/pip-tools/piptools/resolver.py", line 44, in resolve
    if not self.resolve_one_round():
  File "…/pip-tools/piptools/resolver.py", line 28, in resolve_one_round
    new_deps = self.find_new_dependencies()
  File "…/pip-tools/piptools/resolver.py", line 91, in find_new_dependencies
    all_deps = self.find_all_dependencies()
  File "…/pip-tools/piptools/resolver.py", line 74, in find_all_dependencies
    pkg_deps = pkgmgr.get_dependencies(spec.pin(version))
  File "…/pip-tools/piptools/package_manager.py", line 363, in get_dependencies
    deps = self.extract_dependencies(path)
  File "…/pip-tools/piptools/package_manager.py", line 531, in extract_dependencies
    self.unpack_archive(path, unpack_dir)
  File "…/pip-tools/piptools/package_manager.py", line 453, in unpack_archive
    archive = zipfile.ZipFile(path)
  File "…/pyenv/3.4.3/lib/python3.4/zipfile.py", line 937, in __init__
    self._RealGetContents()
  File "…/pyenv/3.4.3/lib/python3.4/zipfile.py", line 978, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
```



Discussion:

Resolving patch:
diff --git a/piptools/package_manager.py b/piptools/package_manager.py
--- a/piptools/package_manager.py
+++ b/piptools/package_manager.py
@@ -442,7 +442,8 @@ def download_package(self, link, destination):
         with logger.indent():
             _, content = get_file_content(url, link, session=self._session)
             with open(destination, 'wb') as f:
-                f.write(content)
+                from io import BytesIO
+                f.write(BytesIO(content))
 
     def unpack_archive(self, path, target_directory):
         logger.debug('- Unpacking %s' % (path,))


Test patch: