BUG: read_csv throws UnicodeDecodeError with unicode aliases
State: closed

Rebased as PR https://github.com/pydata/pandas/pull/14060

---
- [x] closes #13549 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

-read_csv with engine=c throws error when encoding=UTF_16 (anything other than utf-16)
-improved nosetests and moved to in pandas/io/tests/common.py
-passes `pep8radius upstream/master --diff` and  `git diff upstream/master | flake8 --diff`
-put what's new entry in 0.19.0 in accordance with milestone posted on issue


Labels:
name: Bug
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13571?src=pr) is 84.38% (diff: 100%)

> No coverage report found for **master** at 453bc26.
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last update [453bc26...eeb7011](https://codecov.io/gh/pydata/pandas/compare/453bc26e07ebbfea33dda9ff2f4efab683718506...eeb701159f17d8badbc5531ba91df71940c13e55?src=pr)

@nateGeorge Can you rebase this?

@gfyoung @jreback further comments?

LGTM

Ok, I think it's good now.  I was having trouble with `git --rebase` so I did a `git merge` instead.

@nateGeorge you picked some unneeded changes in v0.19.0. 
Normally, rebasing like:

```
git fetch upstream
git rebase upstream/master
git push -f origin fix/read_csv-utf-aliases
```

should work fine. What didn't work?

Hmm, I was following [this guide](https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request), which said to do:

```
git fetch upstream
git merge-base fix/read_csv-utf-aliases upstream/master
```

which gives a hash for the common base node of my branch and the master, then

```
git rebase -i ${HASH}
```

using that hash.  This gave me the error:

```
error: could not apply ${some commit}
```

Then I looked at the file it couldn't apply some commit to (I think v0.19.0.txt), and it had dozens of merge conflicts.  It seemed like I would have to go look up each commit to see which part of the merge to use.  I fixed one file, and then did `git rebase --continue`, and the same error came up again on another file, and I think on the same file again.  The main culprit was the whatsnew/v0.19.0.txt file.

So I guess next time I should just do

```
git fetch upstream
git rebase upstream/master
git push -f origin fix/read_csv-utf-aliases
```

and forget the whole hash thing.

You will still have to clean up the commits now (many unrelated here, and the diff is also not fully correct)

Yes.  I think I'll get to it tomorrow, it looks like a giant mess now :boom:

@nateGeorge It seems it still didn't work well. I would do the following to fix it:

```
git checkout master
git checkout -b temp   # create a temporary branch to fix things
git merge --squash fix/read_csv-utf-aliases  ## squashes all commits of that branch that are ahead of master down into one
git checkout master
## first check that the temp branch is OK
git branch -m temp fix/read_csv-utf-aliases   ## rename to orignal branch
```

But possibly you will have to reset the last few commits, because now there is no diff anymore.

What I just did was delete everything locally and replace with the current upstream/master from a zip.  I couldn't figure out another way to do it for over an hour.  Thanks for the tip, maybe I'll use it in the future.  I'm working on replacing what I had in there now.

I think it should be ready now.  I tested the test again (still haven't figured out an easy way to do that) and it passed.

@nateGeorge There are still a lot of commits here. If you use the approach I outlined above (https://github.com/pydata/pandas/pull/13571#issuecomment-240993602), I think that should work. 
I can also clean it up when merging if you want.

Ok, I'll try what you advised and see what happens.

Hmm well it wasn't letting me overwrite the branch so I deleted it and re-created it with `git branch -m temp fix/read_csv-utf-aliases`.  Do I have to open a new PR now?

Normally not, as long as the branch you want to push has eventually the same name (after renaming), you can just push and the PR will be updated. 
However, if you close the PR and afterwards push (or the branch on github is changed in some way), github does not allow to reopen. So see if you can reopen and then push, otherwise open a new PR.

The 'reopen and comment' button is greyed out and says 'The fix/read_csv-utf-aliases branch was force-pushed or recreated.'  Start a new PR I guess?

yep

Alright, reopened as #14060