Repository: pypa/pip

Issue:
pip install fails with `ValueError: too many values to unpack`
**Environment**

* pip version: 19.0
* Python version: 2.7.15
* OS: Linux/Amazon EMR/Amazon Linux AMI

<!-- Feel free to add more information about your environment here -->

**Description**
<!-- A clear and concise description of what the bug is. -->
In some situations after installing packages from a requirements file this error is thrown:
```
Exception:
Traceback (most recent call last):
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
    **kwargs
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 919, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 445, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 583, in move_wheel_files
    outrows = get_csv_rows_for_installed(reader)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 565, in get_csv_rows_for_installed
    for fpath, digest, length in old_csv_rows:
ValueError: too many values to unpack
```
**Expected behavior**


Install works
**How to Reproduce**


I am not completely sure yet... I am installing from a requirements file with pre-built packages like:
`sudo /home/hadoop/venv2/bin/python -m pip install -r /home/hadoop/emr.requirements.txt --find-links=/home/hadoop/packages`
Pre-built packages are in `/home/hadoop/packages`

**Output**

```
...
...
Installing collected packages: alabaster, SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, chardet, idna, certifi, urllib3, requests, algoliasearch, wrapt, lazy-object-proxy, astroid, avro, colorama, jmespath, docutils, botocore, futures, s3transfer, pyasn1, rsa, PyYAML, awscli, pytz, Babel, backports.functools-lru-cache, beautifulsoup4, Jinja2, numpy, singledispatch, backports-abc, tornado, bokeh, boto3, brewer2mpl, bz2file, cgroupspy, click, configparser, confluent-kafka, coverage, pycparser, cffi, enum34, asn1crypto, ipaddress, cryptography, cycler, dask, scipy, scikit-learn, multipledispatch, cloudpickle, dask-glm, pandas, toolz, dask-searchcv, dask-ml, distance, text-unidecode, Faker, factory-boy, future, fasttext, kiwisolver, subprocess32, pyparsing, matplotlib, fbprophet, pycodestyle, mccabe, pyflakes, flake8, fuzzywuzzy, Geohash, boto, smart-open, gensim, geographiclib, geotext, patsy, statsmodels, ggplot, psutil, gnupg, graphviz, hiredis, html2text, webencodings, html5lib, imagesize, inflection, isort, jellyfish, kafka-python, langdetect, langid, ipython-genutils, decorator, traitlets, simplegeneric, backports.shutil-get-terminal-size, scandir, pathlib2, pickleshare, Pygments, ptyprocess, pexpect, wcwidth, prompt-toolkit, IPython, line-profiler, lxml, mechanize, memory-profiler, funcsigs, pbr, mock, more-itertools, xmltodict, ecdsa, pycryptodome, python-jose, pyaml, jsonpickle, aws-xray-sdk, werkzeug, backports.weakref, backports.tempfile, cookies, responses, backports.ssl-match-hostname, docker-pycreds, websocket-client, docker, jsondiff, moto, msgpack, MySQL-python, nameparser, neo4j-driver, ner, networkx, nltk, ordereddict, olefile, Pillow, planout, plotly, prettytable, psycopg2, py, pyarrow, sasl, thrift, thrift-sasl, PyHive, pylint, bcrypt, pynacl, paramiko, pysftp, pyspark-dist-explore, PyStemmer, pytest, python-dotenv, python-Levenshtein, redis, regex, LEPL, rfc6266-parser, rpy2, s3fs, seaborn, slackclient, snowballstemmer, msgpack-numpy, murmurhash, cymem, preshed, cytoolz, plac, tqdm, dill, pathlib, thinc, ujson, spacy, Sphinx, sqlparse, Tempita, sqlalchemy-migrate, sqlalchemy-redshift, tabulate, textblob, textstat, timeout-decorator, PyJWT, twilio, unicodecsv, ua-parser, user-agents, virtualenv, virtualenv-clone, stevedore, virtualenvwrapper, wget, wordcloud, xlrd, xgboost
  Found existing installation: numpy 1.16.0
    Uninstalling numpy-1.16.0:
      Successfully uninstalled numpy-1.16.0
Exception:
Traceback (most recent call last):
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
    **kwargs
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 919, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 445, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 583, in move_wheel_files
    outrows = get_csv_rows_for_installed(reader)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 565, in get_csv_rows_for_installed
    for fpath, digest, length in old_csv_rows:
ValueError: too many values to unpack
```


Discussion:
* pip version: 19.0
* Python version: 3.6.5
* OS version: macOS 10.13.6

Same issue when installing (when invoking using tox):

```
(venv) viggy-mbp:kb viggy$ tox
...
py27 installdeps: -e.[dev], -rrequirements.txt
...
Exception:
Traceback (most recent call last):
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
    **kwargs
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 919, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 445, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/wheel.py", line 583, in move_wheel_files
    outrows = get_csv_rows_for_installed(reader)
  File "/Users/viggy/GitHub/curai/python/libs/kb/.tox/py27/lib/python2.7/site-packages/pip/_internal/wheel.py", line 565, in get_csv_rows_for_installed
    for fpath, digest, length in old_csv_rows:
ValueError: too many values to unpack
```

~~one more update: only happens when trying to install with a requirements file with local FS paths~~

as others have verified, this chokes on files that have commas in their name.
It looks like this code change was introduced in this merge commit https://github.com/pypa/pip/commit/7696e7e5303b23d233097cf26749bc1e4a5e0645 (specifically https://github.com/pypa/pip/pull/6067/commits/b1c33bdee43024b887f31f95e3e99a7767d5651d )
as part of PR #6067. It looks like there were code changes in the commit in addition adding type annotations.

/cc @mkurnikov 
I grabbed an example of some of the supposed 3-tuples this code expects:
Correct ones:
```
['statsmodels/datasets/strikes/__init__.py', 'sha256=zCyaiU_R6WB1U8RWu74iAZTG2C6vdfMNHvRJtUZutPg', '20']
['statsmodels/datasets/strikes/data.py', 'sha256=OxuiuxzuSo9LcXLozu17u5fe2CrPnZGkMrxRlGgFd0o', '1951']
['statsmodels/datasets/strikes/strikes.csv', 'sha256=uuwOMRln8dydCCqacBJoAkSolLJXD3vRg38y6TFZaZQ', '719']
```

Less-than correct:
```
['statsmodels/datasets/tests/raw.github.com', 'vincentarelbundock', 'Rdatasets', 'master', 'csv', 'car', 'Duncan.csv.zip', 'sha256=igrYVFyIhyIbxUEuvNdq1sdiRZHsvNBrkf7l6JNmekA', '621']
['statsmodels/datasets/tests/raw.github.com', 'vincentarelbundock', 'Rdatasets', 'master', 'datasets.csv.zip', 'sha256=TlfJBE_WdTwnd3vms7ckO4rcKX2sMXLsiCkPvCEwlRc', '16578']
['statsmodels/datasets/tests/raw.github.com', 'vincentarelbundock', 'Rdatasets', 'master', 'doc', 'car', 'rst', 'Duncan.rst.zip', 'sha256=sbmyDj-ve-Mv1bC4oMtZB0AEoW0rS6R0xKc8YbkycyA', '633']
```
@rcramblit Can you determine who or what generated the files causing the problem?
@cjerdonek Looks like these particular ones are coming from: https://github.com/statsmodels/statsmodels/tree/v0.6.1/statsmodels/datasets/tests

Filenames with commas in them..
And how about who / what generated the csv (`RECORD`) file referencing those files? PEP 376 says here that commas should be escaped: https://www.python.org/dev/peps/pep-0376/#record
Not sure I get what you're asking - Aren't these created by pip when a package is installed? When I look at the RECORD file for statsmodels (`./lib64/python2.7/site-packages/statsmodels-0.9.0.dist-info/RECORD`) these rows seem properly formatted:

```
"statsmodels/datasets/tests/raw.github.com,vincentarelbundock,Rdatasets,master,csv,car,Duncan.csv.zip",sha256=igrYVFyIhyIbxUEuvNdq1sdiRZHsvNBrkf7l6JNmekA,621
"statsmodels/datasets/tests/raw.github.com,vincentarelbundock,Rdatasets,master,datasets.csv.zip",sha256=TlfJBE_WdTwnd3vms7ckO4rcKX2sMXLsiCkPvCEwlRc,16578
"statsmodels/datasets/tests/raw.github.com,vincentarelbundock,Rdatasets,master,doc,car,rst,Duncan.rst.zip",sha256=sbmyDj-ve-Mv1bC4oMtZB0AEoW0rS6R0xKc8YbkycyA,633
```

Edit - That is from an installation using `pip==18.1`
I'm not very knowledgeable on this aspect, but for example, here's a similar issue involving commas in the filename where the issue seems to be with an original RECORD file, and the file wasn't generated by pip but rather by the wheel project: https://github.com/pypa/pip/issues/6103

So one possibility of what might be happening is that pip reads an existing RECORD file (possibly externally generated) and generates a new one, and what you've included above is the new one rather than the original one. The change seems to be that previously, pip was "loose" in what RECORD files it could read (which is why 18.1 was able to install the project you mention), whereas now it became more strict, so you're seeing a problem with a RECORD file that previously wasn't complying. I was trying to trace where the original RECORD file came from. You mentioned "pre-built" packages in your original post.
Ah I see. I think you hit it on the nose. I see in the build logs `Building wheels for ...`, including this package. When I create the wheel myself with `python setup.py bdist_wheel` I can see the malformed RECORD file.
This looks like a duplicate of Issue #6166
> It looks like there were code changes in the commit in addition adding type annotations.

That's on me. ISTM that I didn't review that PR properly.

I'll look into this and file a bugfix PR.

I'm hitting this bug when I try to `pip install certbot-apache` into a clean virtualenv (using Python 3.6, in case that matters).
Okay yeah, this is a problem on the side of wheels being built and pip 19.0 caring about getting more strict as @cjerdonek noted; not the code change pertaining to the type annotations PR. (yay! I did review the code properly!)

pip 18.1 was a lot kinder, but would generate invalid RECORD files and fail to properly handle uninstalls.

https://github.com/pypa/pip/blob/6af9de97bbd2427f82942e476c590a2db22ea1ff/src/pip/_internal/wheel.py#L504-L508

So, this isn't really a bug introduced in pip 19.0, so I'll drop the "bug".

That said, @pypa/pip-committers do we want to be kind and just print a warning or be strict and refuse to install? We can (and should) add an error message here regardless; instead of a bad traceback. :)

wheel issue: https://github.com/pypa/wheel/issues/280

My inclination at this point would be to return to the previous (non-strict) behavior and discuss doing something more or different in a slower time frame. Some of us had discussed being strict with regard to RECORD files in a different, related issue, and we decided against it for the time being IIRC (see #5868 and #5913, for example). If there's an easy way to log a warning like by doing a length check, perhaps that would be okay. (I'm also not sure that the RECORD winds up invalid because I think pip generates a new, correct one. The commenter [said above](https://github.com/pypa/pip/issues/6165#issuecomment-456641025) that the resulting file looked okay.) 

By the way, you might have misunderstood my comment about the type annotations. What I was saying is that the code change that caused this (refactoring out a `get_csv_rows_for_installed()` function) was done in PR #6067, which was called "Add type annotations..." But the PR didn't just add type annotations. It also mixed in some code refactorings that I think merited separate review and discussion.
> the PR didn't just add type annotations. It also mixed in some code refactorings that I think merited separate review and discussion.

Ah right, that makes more sense. :)
> I'm also not sure that the RECORD winds up invalid because I think pip generates a new, correct one.

IIUC, wheels generated with `python setup.py bdist_wheel` will have that. Wheel built through pip (via `pip wheel xyz` or `pip install <sdist>`) will use pip's wheel RECORD generation.

> My inclination at this point would be to return to the previous (non-strict) behavior and discuss doing something more or different in a slower time frame.

Fair enough. Coming from https://github.com/pypa/pip/issues/5913#issuecomment-432986021 -> I think this falls in the second category.

That said, a bodge I have in mind is `",".join`ing all but the last 2 entries prior to sorting and writing the RECORD. It's kinda like modifying pip to cover up an issue in wheel but I think we can remove that in the future and put in better error messaging instead, if needed. It's _dirty_ but it'll work and is better than writing a broken RECORD.


> use pip's wheel RECORD generation.

Ok, I spoke wrong -- it's when pip's installing with that it rewrites `RECORD`.


> My inclination at this point would be to return to the previous (non-strict) behavior and discuss doing something more or different in a slower time frame.

Certbot would really, really appreciate this, as the change breaks certbot for all of our certbot-auto users.

Can we get the change reverted now, and fixed fully later? Otherwise we'll have to patch and release a point version of Certbot ASAP, before people's certs start expiring.

Some of our test files have commas in filenames. We plan to address this in a future release, but it seems safer all around to revert the breaking change and then fix things carefully in tandem.

Resolving patch:
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -55,7 +55,7 @@
     from pip._internal.cache import WheelCache  # noqa: F401
     from pip._internal.pep425tags import Pep425Tag  # noqa: F401
 
-    InstalledCSVRow = Tuple[str, Union[str, Text], str]
+    InstalledCSVRow = Tuple[str, ...]
 
 
 VERSION_COMPATIBLE = (1, 0)
@@ -64,6 +64,10 @@
 logger = logging.getLogger(__name__)
 
 
+def normpath(src, p):
+    return os.path.relpath(src, p).replace(os.path.sep, '/')
+
+
 def rehash(path, blocksize=1 << 20):
     # type: (str, int) -> Tuple[str, str]
     """Return (hash, length) for path using hashlib.sha256()"""
@@ -255,6 +259,35 @@ def sorted_outrows(outrows):
     return sorted(outrows, key=lambda row: tuple(str(x) for x in row))
 
 
+def get_csv_rows_for_installed(
+    old_csv_rows,  # type: Iterable[List[str]]
+    installed,  # type: Dict[str, str]
+    changed,  # type: set
+    generated,  # type: List[str]
+    lib_dir,  # type: str
+):
+    # type: (...) -> List[InstalledCSVRow]
+    installed_rows = []  # type: List[InstalledCSVRow]
+    for row in old_csv_rows:
+        if len(row) > 3:
+            logger.warning(
+                'RECORD line has more than three elements: {}'.format(row)
+            )
+        fpath = row[0]
+        fpath = installed.pop(fpath, fpath)
+        if fpath in changed:
+            digest, length = rehash(fpath)
+            row[1] = digest
+            row[2] = length
+        installed_rows.append(tuple(row))
+    for f in generated:
+        digest, length = rehash(f)
+        installed_rows.append((normpath(f, lib_dir), digest, str(length)))
+    for f in installed:
+        installed_rows.append((installed[f], '', ''))
+    return installed_rows
+
+
 def move_wheel_files(
     name,  # type: str
     req,  # type: Requirement
@@ -305,9 +338,6 @@ def move_wheel_files(
                 compileall.compile_dir(source, force=True, quiet=True)
         logger.debug(stdout.getvalue())
 
-    def normpath(src, p):
-        return os.path.relpath(src, p).replace(os.path.sep, '/')
-
     def record_installed(srcfile, destfile, modified=False):
         """Map archive RECORD paths to installation RECORD paths."""
         oldpath = normpath(srcfile, wheeldir)
@@ -559,28 +589,16 @@ def _get_script_text(entry):
     shutil.move(temp_installer, installer)
     generated.append(installer)
 
-    def get_csv_rows_for_installed(old_csv_rows):
-        # type: (Iterable[List[str]]) -> List[InstalledCSVRow]
-        installed_rows = []  # type: List[InstalledCSVRow]
-        for fpath, digest, length in old_csv_rows:
-            fpath = installed.pop(fpath, fpath)
-            if fpath in changed:
-                digest, length = rehash(fpath)
-            installed_rows.append((fpath, digest, str(length)))
-        for f in generated:
-            digest, length = rehash(f)
-            installed_rows.append((normpath(f, lib_dir), digest, str(length)))
-        for f in installed:
-            installed_rows.append((installed[f], '', ''))
-        return installed_rows
-
     # Record details of all files installed
     record = os.path.join(info_dir[0], 'RECORD')
     temp_record = os.path.join(info_dir[0], 'RECORD.pip')
     with open_for_csv(record, 'r') as record_in:
         with open_for_csv(temp_record, 'w+') as record_out:
             reader = csv.reader(record_in)
-            outrows = get_csv_rows_for_installed(reader)
+            outrows = get_csv_rows_for_installed(
+                reader, installed=installed, changed=changed,
+                generated=generated, lib_dir=lib_dir,
+            )
             writer = csv.writer(record_out)
             # Sort to simplify testing.
             for row in sorted_outrows(outrows):


Test patch:
diff --git a/tests/unit/test_wheel.py b/tests/unit/test_wheel.py
--- a/tests/unit/test_wheel.py
+++ b/tests/unit/test_wheel.py
@@ -1,6 +1,8 @@
 """Tests for wheel binary packages and .dist-info."""
+import csv
 import logging
 import os
+import textwrap
 
 import pytest
 from mock import Mock, patch
@@ -76,6 +78,63 @@ def test_sorted_outrows(outrows, expected):
     assert actual == expected
 
 
+def call_get_csv_rows_for_installed(tmpdir, text):
+    path = tmpdir.join('temp.txt')
+    path.write(text)
+
+    installed = {}
+    changed = set()
+    generated = []
+    lib_dir = '/lib/dir'
+
+    with wheel.open_for_csv(path, 'r') as f:
+        reader = csv.reader(f)
+        outrows = wheel.get_csv_rows_for_installed(
+            reader, installed=installed, changed=changed,
+            generated=generated, lib_dir=lib_dir,
+        )
+    return outrows
+
+
+def test_get_csv_rows_for_installed(tmpdir, caplog):
+    text = textwrap.dedent("""\
+    a,b,c
+    d,e,f
+    """)
+    outrows = call_get_csv_rows_for_installed(tmpdir, text)
+
+    expected = [
+        ('a', 'b', 'c'),
+        ('d', 'e', 'f'),
+    ]
+    assert outrows == expected
+    # Check there were no warnings.
+    assert len(caplog.records) == 0
+
+
+def test_get_csv_rows_for_installed__long_lines(tmpdir, caplog):
+    text = textwrap.dedent("""\
+    a,b,c,d
+    e,f,g
+    h,i,j,k
+    """)
+    outrows = call_get_csv_rows_for_installed(tmpdir, text)
+
+    expected = [
+        ('a', 'b', 'c', 'd'),
+        ('e', 'f', 'g'),
+        ('h', 'i', 'j', 'k'),
+    ]
+    assert outrows == expected
+
+    messages = [rec.message for rec in caplog.records]
+    expected = [
+        "RECORD line has more than three elements: ['a', 'b', 'c', 'd']",
+        "RECORD line has more than three elements: ['h', 'i', 'j', 'k']"
+    ]
+    assert messages == expected
+
+
 def test_wheel_version(tmpdir, data):
     future_wheel = 'futurewheel-1.9-py2.py3-none-any.whl'
     broken_wheel = 'brokenwheel-1.0-py2.py3-none-any.whl'