Repository: pandas-dev/pandas

Issue:
Codec `utf-16` aliases do not work in read_csv with c engine
#### Code Sample, a copy-pastable example if possible

```
import pandas

path = "test.csv"
pandas.DataFrame({"A": [0,1], "B": [2,3]}).to_csv(
    path, encoding="utf-16")

for encoding in ["utf-16","utf_16","UTF_16","UTF-16"]:
    try:
        pandas.io.parsers.read_csv(
            path,
            engine='c',
            encoding=encoding)
        print(encoding, "succeeded in c-pandas")
    except UnicodeDecodeError:
        print(encoding, "failed in c-pandas")

    try:
        pandas.io.parsers.read_csv(
            path,
            engine='python',
            encoding=encoding)
        print(encoding, "succeeded in pandas")
    except UnicodeDecodeError:
        print(encoding, "failed in pandas")

    try:
        with open(path, encoding=encoding) as file:
            file.read()
        print(encoding, "succeeded in open")
    except UnicodeDecodeError:
        print(encoding, "failed in open")
```
#### Expected Output

```
utf-16 succeeded in c-pandas
utf-16 succeeded in pandas
utf-16 succeeded in open
utf_16 succeeded in c-pandas
utf_16 succeeded in pandas
utf_16 succeeded in open
UTF_16 succeeded in c-pandas
UTF_16 succeeded in pandas
UTF_16 succeeded in open
UTF-16 succeeded in c-pandas
UTF-16 succeeded in pandas
UTF-16 succeeded in open
```
#### Actual Output

```
utf-16 succeeded in c-pandas
utf-16 succeeded in pandas
utf-16 succeeded in open
utf_16 failed in c-pandas
utf_16 succeeded in pandas
utf_16 succeeded in open
UTF_16 failed in c-pandas
UTF_16 succeeded in pandas
UTF_16 succeeded in open
UTF-16 failed in c-pandas
UTF-16 succeeded in pandas
UTF-16 succeeded in open
```
#### output of `pd.show_versions()`
## INSTALLED VERSIONS

commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.6.2-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.1
setuptools: 23.0.0
Cython: None
numpy: 1.11.0
scipy: 0.17.1
statsmodels: None
xarray: None
IPython: 4.2.0
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: None
httplib2: 0.9.2
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None



Discussion:
Without error handling to run through all test cases, the traceback is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gereon/python/bugreport/test.py", line 11, in <module>
    engine='c',
  File "/usr/lib/python3.5/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/lib/python3.5/site-packages/pandas/io/parsers.py", line 325, in _read
    return parser.read()
  File "/usr/lib/python3.5/site-packages/pandas/io/parsers.py", line 815, in read
    ret = self._engine.read(nrows)
  File "/usr/lib/python3.5/site-packages/pandas/io/parsers.py", line 1314, in read
    data = self._reader.read(nrows)
  File "pandas/parser.pyx", line 805, in pandas.parser.TextReader.read (pandas/parser.c:8748)
  File "pandas/parser.pyx", line 827, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:9003)
  File "pandas/parser.pyx", line 904, in pandas.parser.TextReader._read_rows (pandas/parser.c:10022)
  File "pandas/parser.pyx", line 1011, in pandas.parser.TextReader._convert_column_data (pandas/parser.c:11397)
  File "pandas/parser.pyx", line 1071, in pandas.parser.TextReader._convert_tokens (pandas/parser.c:12302)
  File "pandas/parser.pyx", line 1157, in pandas.parser.TextReader._convert_with_dtype (pandas/parser.c:13740)
  File "pandas/parser.pyx", line 1173, in pandas.parser.TextReader._string_convert (pandas/parser.c:13950)
  File "pandas/parser.pyx", line 1460, in pandas.parser._string_box_decode (pandas/parser.c:19767)
UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x41 in position 0: truncated data
```

In case it is relevant:

### [7.2.3. Standard Encodings](https://docs.python.org/3/library/codecs.html#standard-encodings)

[…]Notice that spelling alternatives that only differ in case or use a hyphen instead of an underscore are also valid aliases; therefore, e.g. 'utf-8' is a valid alias for the 'utf_8' codec.

CPython implementation detail: Some common encodings can bypass the codecs lookup machinery […]: […] utf-16 […]

This also holds if the writing encoding is taken to be `utf_16`:

```
>>> pandas.DataFrame({"A": [0,1], "B": [2,3]}).to_csv(
        path, encoding="utf_16")
[…]
utf_16 failed in c-pandas
utf_16 succeeded in pandas
utf_16 succeeded in open
```

yeah this has a string definition of `utf-16`. Should be easy to make more general.

want to take a crack at it?


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -343,6 +343,9 @@ def _validate_nrows(nrows):
 def _read(filepath_or_buffer, kwds):
     "Generic reader of line files."
     encoding = kwds.get('encoding', None)
+    if encoding is not None:
+        encoding = re.sub('_', '-', encoding).lower()
+        kwds['encoding'] = encoding
 
     # If the input could be a filename, check for a recognizable compression
     # extension.  If we're reading from a URL, the `get_filepath_or_buffer`


Test patch:
diff --git a/pandas/io/tests/parser/common.py b/pandas/io/tests/parser/common.py
--- a/pandas/io/tests/parser/common.py
+++ b/pandas/io/tests/parser/common.py
@@ -1583,3 +1583,13 @@ def test_temporary_file(self):
         new_file.close()
         expected = DataFrame([[0, 0]])
         tm.assert_frame_equal(result, expected)
+
+    def test_read_csv_utf_aliases(self):
+        # see gh issue 13549
+        expected = pd.DataFrame({'mb_num': [4.8], 'multibyte': ['test']})
+        for byte in [8, 16]:
+            for fmt in ['utf-{0}', 'utf_{0}', 'UTF-{0}', 'UTF_{0}']:
+                encoding = fmt.format(byte)
+                data = 'mb_num,multibyte\n4.8,test'.encode(encoding)
+                result = self.read_csv(BytesIO(data), encoding=encoding)
+                tm.assert_frame_equal(result, expected)