BUG: avoid "b" prefix for bytes in to_csv() on Python 3 (#9712)
State: closed

- [x] closes #9712
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13890?src=pr) is 85.31% (diff: 90.00%)

> Merging [#13890](https://codecov.io/gh/pydata/pandas/pull/13890?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13890   diff @@
==========================================
  Files           139        139          
  Lines         50143      50161    +18   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42777      42793    +16   
- Misses         7366       7368     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [7e15923...4d88771](https://codecov.io/gh/pydata/pandas/compare/7e15923d908a42008fb7fecde58be626277fd82e...4d8877120919a55b43c7f17dd1afe72c245a59d5?src=pr)

OK so let me summarize where this PR is at:
1. Cython crashes on the Python 2 build only.  I haven't investigated this yet, because:
2. @jreback seems to dislike the fact that new code is required to fix this problem, so:
3. I posted a [question on Stack Overflow](http://stackoverflow.com/questions/38770522/python-3-csv-writer-prints-bytes-with-prefix-and-quotes) to see if Python 3's `csv` module can be made to do the heavy lifting, but:
4. Writing bytes without the prefixes and quotes does not seem to be supported in Python 3 directly.

It might be interesting to try to get support in Python 3.7 (or whatever) to write bytes as plain strings in the `csv` module.  But this would take a long time to reach Pandas users.  I do think the patch I've provided here is useful and appropriate, and I do not have a simpler one to offer.

I hope we can get #9712 fixed somehow.  PRs welcome!  ;-)

@jzwinck I have made some more comments. Please use the existing functions. It is not clear that adding code is needed here, it is just more repeated code and adds to the maintenance burden.

can you rebase / update?

@jreback I'm not sure what you're asking for.  I know what rebase means but the last time we spoke about this PR you had fundamental problems with the way I was approaching it.  I don't have much time these days to do anything more with it.

@jzwinck rebase and update means exactly what it says. rebase on master and respond to comments. pls reopen if you want to continue work.

@jzwinck Looking back at the comments, I think one of the main suggestions of @jreback is to use the existing `pd.core.strings.str_decode` function instead of adding your new cython `object_array_decode_bytes` function, which seems to do exactly what you need from a quick glance (only cannot deal with mixed types). 
What is the reason you cannot use this?

```
In [73]: pd.core.strings.str_decode(np.array([b'a', np.nan], dtype=object), encoding='UTF8')
Out[73]: array(['a', nan], dtype=object)
```

@jorisvandenbossche As you say, `object_array_decode_bytes` does not deal with mixed types.  I wanted the patch to work even in the presence of mixed types, and included a unit test demonstrating this.  Do you think it is better to make a patch which does not address the case of mixed types?

In any case, I have limited time to work on this further now.

the mixed-dtype case in not relevant (nor tested). expanding the code base at the expense of readability is not necessary nor warranted. as I said, if you want to fix according to comments, pls do so. or maybe someone else will pick this up.

@jreback If I am the user, and I am telling you I want support for mixed dtypes, how do you then conclude it is "not relevant?"  Also you are mistaken when you say it is "not tested" because it plainly is in the unit test I added: `(b'hello', ['a', b'b']),`.

@jzwinck mixed-types (in a single array) are a complete disaster. I would rather not make them worse by adding additional code which makes them worse.

@jzwinck Although the wording of @jreback was maybe a bit harsh on the 'relevant' aspect, there is a difference between "relevant for you as a user" and "relevant enough in general to warrant additional code-complexity in pandas" (and to be clear: I am not not saying it is one of either, just noting that it is debatable) 
I am not in a good position to judge its relevance as I don't have a personal use case for it, but if supporting writing bytes to csv without "b" for columns with all bytes (so not mixed) is less complex to implement (because you can use existing functionality), IMO it is seems better to start with that, as it probably already solves the main use cases?