Repository: pandas-dev/pandas
Pull request: ENH: Better error message if usecols doesn't match columns
GH17301: Improving the error message given when usecols
doesn't match with the columns provided.

- [x] closes #17301
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry 

NOTE: Do I need to add a whatsnew entry for something so small? 
  Happy to do so if needed.

Problem statement:
Improve the error message when `usecols` cannot match all columns
#### Code Sample

```python
import pandas as pd
# Where example.csv is:
# column1,column2
# 1, 2

pd.read_csv('example.csv', usecols=['column1', 'column2', ' column3'])
```
#### Problem description

The error when all of the columns provided in usecols are not found isn't currently very useful. I think it would be much better if it detailed which fields it could not find - this is really useful when you're reading a large file with many required columns.

#### Current Output
```
MYPATH\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\parsers.py in __init__(self, src, **kwds)
   1432 
   1433             if len(self.names) < len(self.usecols):
-> 1434                 raise ValueError("Usecols do not match names.")
   1435 
   1436         self._set_noconvert_columns()

ValueError: Usecols do not match names.
```
#### Expected Output

A ValueError with the fields that are missing, something like: 
```ValueError: Usecols do not match columns in file, columns expected but not found: ['column3']```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.2.2
numexpr: 2.6.1
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
boto: 2.45.0
pandas_datareader: 0.3.0.post

</details>

Hints from issue discussion:
If there are no objections to this, I'm happy to pick this up and implement.

Also happy to take on any suggestions from people on a different error message.
@AaronCritchley : Go right ahead!  I would also suggest, if possible, that you upgrade your `pandas` installation too.  We're at `0.20.3` right now, though your issue still persists on `master`.
Great, will take this on over the next few days, thanks!

Commit message: Adding whatsnews entry, and test for multiple missing columns

Patch to review:
diff --git a/doc/source/whatsnew/v0.22.0.txt b/doc/source/whatsnew/v0.22.0.txt
index 1a08a1353a605..64f2701896a94 100644
--- a/doc/source/whatsnew/v0.22.0.txt
+++ b/doc/source/whatsnew/v0.22.0.txt
@@ -76,6 +76,7 @@ Other Enhancements
 - Improved wording of ``ValueError`` raised in :func:`to_datetime` when ``unit=`` is passed with a non-convertible value (:issue:`14350`)
 - :func:`Series.fillna` now accepts a Series or a dict as a ``value`` for a categorical dtype (:issue:`17033`)
 - :func:`pandas.read_clipboard` updated to use qtpy, falling back to PyQt5 and then PyQt4, adding compatibility with Python3 and multiple python-qt bindings (:issue:`17722`)
+- Improved wording of ``ValueError`` raised when creating a DataFrame and the ``usecols`` argument cannot match all columns. (:issue:`17301`)
 
 .. _whatsnew_0220.api_breaking:
 
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
index fe50b551ea948..83b1d8ec1a070 100755
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1141,6 +1141,38 @@ def _evaluate_usecols(usecols, names):
     return usecols
 
 
+def _validate_usecols_names(usecols, names):
+    """
+    Validates that all usecols are present in a given
+    list of names. If not, raise a ValueError that
+    shows what usecols are missing.
+
+    Parameters
+    ----------
+    usecols : iterable of usecols
+        The columns to validate are present in names.
+    names : iterable of names
+        The column names to check against.
+
+    Returns
+    -------
+    usecols : iterable of usecols
+        The `usecols` parameter if the validation succeeds.
+
+    Raises
+    ------
+    ValueError : Columns were missing. Error message will list them.
+    """
+    missing = [c for c in usecols if c not in names]
+    if len(missing) > 0:
+        raise ValueError(
+            "Usecols do not match columns, "
+            "columns expected but not found: {missing}".format(missing=missing)
+        )
+
+    return usecols
+
+
 def _validate_skipfooter_arg(skipfooter):
     """
     Validate the 'skipfooter' parameter.
@@ -1753,14 +1785,14 @@ def __init__(self, src, **kwds):
             # GH 14671
             if (self.usecols_dtype == 'string' and
                     not set(usecols).issubset(self.orig_names)):
-                raise ValueError("Usecols do not match names.")
+                _validate_usecols_names(usecols, self.orig_names)
 
             if len(self.names) > len(usecols):
                 self.names = [n for i, n in enumerate(self.names)
                               if (i in usecols or n in usecols)]
 
             if len(self.names) < len(usecols):
-                raise ValueError("Usecols do not match names.")
+                _validate_usecols_names(usecols, self.names)
 
         self._set_noconvert_columns()
 
@@ -2532,9 +2564,13 @@ def _handle_usecols(self, columns, usecols_key):
                     raise ValueError("If using multiple headers, usecols must "
                                      "be integers.")
                 col_indices = []
+
                 for col in self.usecols:
                     if isinstance(col, string_types):
-                        col_indices.append(usecols_key.index(col))
+                        try:
+                            col_indices.append(usecols_key.index(col))
+                        except ValueError:
+                            _validate_usecols_names(self.usecols, usecols_key)
                     else:
                         col_indices.append(col)
             else:
diff --git a/pandas/tests/io/parser/usecols.py b/pandas/tests/io/parser/usecols.py
index f582e5037ca07..4126e2eb93f28 100644
--- a/pandas/tests/io/parser/usecols.py
+++ b/pandas/tests/io/parser/usecols.py
@@ -480,10 +480,10 @@ def test_raise_on_usecols_names_mismatch(self):
         # GH 14671
         data = 'a,b,c,d\n1,2,3,4\n5,6,7,8'
 
-        if self.engine == 'c':
-            msg = 'Usecols do not match names'
-        else:
-            msg = 'is not in list'
+        msg = (
+            "Usecols do not match columns, "
+            "columns expected but not found: {missing}"
+        )
 
         usecols = ['a', 'b', 'c', 'd']
         df = self.read_csv(StringIO(data), usecols=usecols)
@@ -492,11 +492,12 @@ def test_raise_on_usecols_names_mismatch(self):
         tm.assert_frame_equal(df, expected)
 
         usecols = ['a', 'b', 'c', 'f']
-        with tm.assert_raises_regex(ValueError, msg):
+        with tm.assert_raises_regex(ValueError, msg.format(missing="\['f'\]")):
             self.read_csv(StringIO(data), usecols=usecols)
 
-        usecols = ['a', 'b', 'f']
-        with tm.assert_raises_regex(ValueError, msg):
+        usecols = ['a', 'b', 'f', 'g']
+        with tm.assert_raises_regex(
+                ValueError, msg.format(missing="\[('f', 'g'|'g', 'f')\]")):
             self.read_csv(StringIO(data), usecols=usecols)
 
         names = ['A', 'B', 'C', 'D']
@@ -520,9 +521,9 @@ def test_raise_on_usecols_names_mismatch(self):
         # tm.assert_frame_equal(df, expected)
 
         usecols = ['A', 'B', 'C', 'f']
-        with tm.assert_raises_regex(ValueError, msg):
+        with tm.assert_raises_regex(ValueError, msg.format(missing="\['f'\]")):
             self.read_csv(StringIO(data), header=0, names=names,
                           usecols=usecols)
         usecols = ['A', 'B', 'f']
-        with tm.assert_raises_regex(ValueError, msg):
+        with tm.assert_raises_regex(ValueError, msg.format(missing="\['f'\]")):
             self.read_csv(StringIO(data), names=names, usecols=usecols)


Code review comments:
diff_hunk: @@ -76,6 +76,7 @@ Other Enhancements
 - Improved wording of ``ValueError`` raised in :func:`to_datetime` when ``unit=`` is passed with a non-convertible value (:issue:`14350`)
 - :func:`Series.fillna` now accepts a Series or a dict as a ``value`` for a categorical dtype (:issue:`17033`)
 - :func:`pandas.read_clipboard` updated to use qtpy, falling back to PyQt5 and then PyQt4, adding compatibility with Python3 and multiple python-qt bindings (:issue:`17722`)
+- Improved wording of ``ValueError`` raised when creating a DataFrame and the ``usecols`` argument cannot match all columns. (:issue:`17301`)
path: doc/source/whatsnew/v0.22.0.txt
text: @user1:
say this is for ``:func:`read_csv` ``, not DataFrame construction.

@author:
Got it, will change now!