Repository: astropy/astropy

Issue:
Unicode decode error when trying to read in ascii table
I thought that in Python 3, astropy.io.ascii worked with non-ASCII characters, but I am having issues with the following table:

https://raw.githubusercontent.com/mrwilson/strictly-come-dancing-results/master/results.csv

```python
In [4]: from astropy.table import Table

In [5]: t = Table.read('results.csv')
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-5-aa54d4eed93f> in <module>()
----> 1 t = Table.read('results.csv')

~/Dropbox/Code/Astropy/astropy/astropy/table/table.py in read(cls, *args, **kwargs)
   2505         passed through to the underlying data reader (e.g. `~astropy.io.ascii.read`).
   2506         """
-> 2507         out = io_registry.read(cls, *args, **kwargs)
   2508         # For some readers (e.g., ascii.ecsv), the returned `out` class is not
   2509         # guaranteed to be the same as the desired output `cls`.  If so,

~/Dropbox/Code/Astropy/astropy/astropy/io/registry.py in read(cls, format, *args, **kwargs)
    515 
    516         reader = get_reader(format, cls)
--> 517         data = reader(*args, **kwargs)
    518 
    519         if not isinstance(data, cls):

~/Dropbox/Code/Astropy/astropy/astropy/io/ascii/connect.py in read_csv(filename, **kwargs)
     78     from .ui import read
     79     kwargs['format'] = 'csv'
---> 80     return read(filename, **kwargs)
     81 
     82 

~/Dropbox/Code/Astropy/astropy/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
    366             fast_reader_rdr = get_reader(**fast_kwargs)
    367             try:
--> 368                 dat = fast_reader_rdr.read(table)
    369                 _read_trace.append({'kwargs': fast_kwargs,
    370                                     'Reader': fast_reader_rdr.__class__,

~/Dropbox/Code/Astropy/astropy/astropy/io/ascii/fastbasic.py in read(self, table)
    107                                       data_start=self.data_start,
    108                                       fill_extra_cols=self.fill_extra_cols,
--> 109                                       **self.kwargs)
    110         conversion_info = self._read_header()
    111         self.check_header()

~/Dropbox/Code/Astropy/astropy/astropy/io/ascii/cparser.pyx in astropy.io.ascii.cparser.CParser.__cinit__()

~/Dropbox/Code/Astropy/astropy/astropy/io/ascii/cparser.pyx in astropy.io.ascii.cparser.CParser.setup_tokenizer()

UnicodeEncodeError: 'ascii' codec can't encode character '\xeb' in position 11416: ordinal not in range(128)
```

However, pandas can read it in:

```python
In [6]: from pandas import read_csv

In [7]: read_csv('results.csv')
Out[7]: 
...
```

Don't ask why I'm trying to read in that table... 💃 🕺 

Discussion:
astropy-bot[bot]: Hi there @ananyashreyjain :wave: - thanks for the pull request! I'm just a friendly :robot: that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part :smiley:.

I noticed the following issues with this pull request:

* The milestone has not been set (this can only be set by a maintainer)
* Changelog entry not present (or pull request number missing) and neither the **Affects-dev** nor the **no-changelog-entry-needed** label are set

Would it be possible to fix these? Thanks!

*If there are any issues with this message, please report them [here](https://github.com/astropy/astropy-bot/issues).*
ananyashreyjain: @astrofrog please have a look.If it is fine i will add the tests.
bsipocz: @ananyashreyjain - Please do add the tests anyway. Bug fixes can be seen as test driven, so the failing regression in the bug reports can always be added straight away before any review.
ananyashreyjain: okay, I will add the tests. 
taldcroft: @ananyashreyjain - any progress on tests and a changelog entry?
ananyashreyjain: @taldcroft sorry I almost forgot about this PR .I will add the tests and changelog entry as soon as possible.
taldcroft: @ananyashreyjain - looks good now.  I see that this has gotten up to 5 commits for a very small change, where the intermediate commits do not add to understanding of the final patch.  It is astropy policy to request that you squash down to one commit for this.   See e.g. https://stackoverflow.com/questions/6934752/combining-multiple-commits-before-pushing-in-git

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index af71930710..12b89030f1 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -256,6 +256,10 @@ astropy.extern
 astropy.io.ascii
 ^^^^^^^^^^^^^^^^
 
+- Fix a unicode decode error when reading a table with non-ASCII characters.
+  The fast C reader cannot handle unicode so the code now uses the pure-Python
+  reader in this case. [#7103]
+
 astropy.io.fits
 ^^^^^^^^^^^^^^^
 
diff --git a/astropy/io/ascii/ui.py b/astropy/io/ascii/ui.py
index b494955c63..b3c12009f8 100644
--- a/astropy/io/ascii/ui.py
+++ b/astropy/io/ascii/ui.py
@@ -369,7 +369,7 @@ def read(table, guess=None, **kwargs):
                 _read_trace.append({'kwargs': fast_kwargs,
                                     'Reader': fast_reader_rdr.__class__,
                                     'status': 'Success with fast reader (no guessing)'})
-            except (core.ParameterError, cparser.CParserError) as err:
+            except (core.ParameterError, cparser.CParserError, UnicodeEncodeError) as err:
                 # special testing value to avoid falling back on the slow reader
                 if fast_reader['enable'] == 'force':
                     raise core.InconsistentTableError(


Test patch:
diff --git a/astropy/io/ascii/tests/test_read.py b/astropy/io/ascii/tests/test_read.py
index 5f865776f2..89b88cd720 100644
--- a/astropy/io/ascii/tests/test_read.py
+++ b/astropy/io/ascii/tests/test_read.py
@@ -1378,3 +1378,12 @@ def test_read_chunks_table_changes():
     # This also confirms that the dtypes are exactly the same, i.e.
     # the string itemsizes are the same.
     assert np.all(t1 == t2)
+
+
+def test_read_non_ascii():
+    """Test that pure-Python reader is used in case the file contains non-ASCII characters
+    in it.
+    """
+    table = Table.read(['col1, col2', '\u2119, \u01b4', '1, 2'], format='csv')
+    assert np.all(table['col1'] == ['\u2119', '1'])
+    assert np.all(table['col2'] == ['\u01b4', '2'])