BUG: GH13219 Fixed. Allow unicode values in usecol
State: closed

- [x] closes #13219 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

This is my first time contributing to this project.  Let me know what needs to be improved, especially with regards to the test and the bugfix.  It was simpler than I imagined which makes me want to say I did it wrong.


Labels:
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv

Comments:
Currently the multibyte usecol tests are throwing errors for the test parsers.  Is this expected?  If so I'll adjust the tests.

```

======================================================================
ERROR: test_usecols_with_multibyte_unicode_characters (pandas.io.tests.parser.test_parsers.TestCParserHighMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/tests/parser/usecols.py", line 309, in test_usecols_with_multibyte_unicode_characters
    df = self.read_csv(StringIO(s), usecols=[u'あああ', u'いい'])
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/tests/parser/test_parsers.py", line 57, in read_csv
    return read_csv(*args, **kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 315, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 645, in __init__
    self._make_engine(self.engine)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 799, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 1257, in __init__
    raise ValueError("Usecols do not match names.")
ValueError: Usecols do not match names.

======================================================================
ERROR: test_usecols_with_multibyte_unicode_characters (pandas.io.tests.parser.test_parsers.TestCParserLowMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/tests/parser/usecols.py", line 309, in test_usecols_with_multibyte_unicode_characters
    df = self.read_csv(StringIO(s), usecols=[u'あああ', u'いい'])
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/tests/parser/test_parsers.py", line 76, in read_csv
    return read_csv(*args, **kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 315, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 645, in __init__
    self._make_engine(self.engine)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 799, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 1257, in __init__
    raise ValueError("Usecols do not match names.")
ValueError: Usecols do not match names.

======================================================================
ERROR: test_usecols_with_multibyte_unicode_characters (pandas.io.tests.parser.test_parsers.TestPythonParser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/tests/parser/usecols.py", line 309, in test_usecols_with_multibyte_unicode_characters
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/tests/parser/test_parsers.py", line 100, in read_csv
    return read_csv(*args, **kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 315, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 645, in __init__
    self._make_engine(self.engine)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 805, in _make_engine
    self._engine = klass(self.f, **self.options)
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 1608, in __init__
    self.columns, self.num_original_columns = self._infer_columns()
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 1823, in _infer_columns
    line = self._buffered_line()
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 1975, in _buffered_line
    return self._next_line()
  File "/Users/avendesora/Dropbox/code/python/pandas-hassan/pandas/io/parsers.py", line 2006, in _next_line
    orig_line = next(self.data)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

----------------------------------------------------------------------
Ran 587 tests in 111.477s

FAILED (SKIP=21, errors=3)

```

This _should_ work, but currently not. Can we keep open #13219 to handle 2 bytes?

how about we comment out these 2 byte tests for now (and make a new issue)?

@hassanshamim can you do that & rebase?

Can do!

- [x] Created Issue #13253
- [x] Converted data strings from unicode to str in tests
- [x] Added test for multibyte characters in usecol (not unicode)
- [x] Reverted change that allowed `mixed` dtype usecol values
- [x] Rebased onto upstream/master

I think that's everything.  I've been squashing my new commits and just force pushing to update this branch.  Is that the normal process for incorporating feedback on PRs?  Or should I have just left multiple commits then squashed once this was all ready?

@hassanshamim : 

1) You forgot to add test for single-byte `unicode` `usecols` (i.e. `u'A'`)
2) Regarding squashing, I think that in this case, it is okay to squash because this really should be one commit.  In cases where there might be a lot more changing, multiple commits would be preferred.

this looks good. @gfyoung any further comments?

@hassanshamim : just a few comments, but almost there!

@gfyoung updated with your recommendations.  Are these formats okay?  

```
def test_usecols_with_multibyte_unicode_characters(self):
    raise nose.SkipTest('TODO: see gh-13253')
    # actual test code...
```

New error message:

```
msg = ("The elements of 'usecols' must "
            "either be all strings, all unicode, or all integers")
```

@hassanshamim : yep, that looks good, so LGTM now.  Just ping when tests pass!

@gfyoung tests have gone through.

thanks!