Repository: pandas-dev/pandas

Issue:
UnicodeDecodeError for Stata file
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
pd.read_stata('mwe.dta')
```

`mwe.dta` available here: [mwe.zip](https://github.com/pandas-dev/pandas/files/3034141/mwe.zip)
This file is a derivative of [The Supreme Court Database](http://scdb.wustl.edu/_brickFiles/2018_02/SCDB_2018_02_caseCentered_Citation.dta.zip)

#### Problem description
The command raises

```python
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 20: invalid start byte
```

I traced the error to a value label containing that byte.
This is a follow-up for #21244 and #23736
Changing line **1334** of `pandas.io.stata` from

```python
return s.decode('utf-8')
```

to

```python
return s.decode('latin-1')
```

allows me to read in the file.

#### Expected Output

The file should be correctly read and parsed.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.2.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.2
pytest: None
pip: 19.0.3
setuptools: 40.6.2
Cython: None
numpy: 1.16.2
scipy: None
pyarrow: None
xarray: None
IPython: 7.4.0
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
You could probably also leverage `self._encoding` in the class since I see that used elsewhere. PRs welcome
```python
def _set_encoding(self):
    """
    Set string encoding which depends on file version
    """
    if self.format_version < 118:
        self._encoding = 'latin-1'
    else:
        self._encoding = 'utf-8'
```
@WillAyd from the above method, it seems that `self._encoding` is `'utf-8'` for version 118 files (like mine), so I don't think that will help.
There seems to be some conflict about which encoding these files use (see #21246), and I don't really understand what's going on...
Not familiar enough with stata to guess, but from reading through linked PR it looks like this was intentional to make the encoding strict based off version

@bashtage might have thoughts
According to Stata these should be UTF-8.

```
    The value-label names associated with each variable are recorded as


              <value_label_names>lbllist</value_label_names>


    where lbllist is a sequence (array) of K 129-byte, \0-terminated, UTF-8
    label names.  Each name may be up to 32-characters in length.
```
This file is a [118 format file](https://www.stata.com/help.cgi?dta#labelnames).
This character is actually in the `value_label_table` as described [here](https://www.stata.com/help.cgi?dta#value_labels)
The byte in question maps to `'CURRENCY SIGN' (U+00A4)` (according to [fileformat.info](https://www.fileformat.info/info/unicode/char/00a4/index.htm)). I'm not sure why it's not considered valid Unicode. I've been trying to educate myself over the past hour, but this is a really confusing topic.
I don't understand why

```python
b'\xa4'.decode('utf-8')
```

does not work if it maps to `'CURRENCY SIGN' (U+00A4)`
Please forgive my ignorance.
It is a 2-byte encoding in utf-8.

The correctly formatted string should be 

`R.C.R.A. (42 U.S.C. \xc2\xa4 1978)`

or just 

```
'¤'.encode('utf-8')
b'\xc2\xa4'
```

Thank you.
So does this mean that Stata messed it up?
It is not obvious to me. Stata does some undocumented things w.r.t. latin-1 encodings.

This block would make it work.
```
        try:
            return s.decode('utf-8')
        except UnicodeDecodeError:
            return s.decode('unicode_escape')
```



Further looking says yes, this is invalid unicode.  In particular, 0xA4 is an invalud Unicode encoding, so this qualifies as an undocumented Stata "feature"
> It is not obvious to me. Stata does some undocumented things w.r.t. latin-1 encodings.
> 
> This block would make it work.
> 
> ```
>         try:
>             return s.decode('utf-8')
>         except UnicodeDecodeError:
>             return s.decode('unicode_escape')
> ```

I can confirm that this works on my end too.
Thank you.

I think I'm starting to get a little more understanding of this.
In UTF-8, anything over 127 should  jump to 2+ bytes, because the one-byte UTF-8 characters can't start with a binary 1. Did I get that right?
I fully thing happens for these values:

Correct unicode
`11000010 10100100`

This is read as `000010100100` (since the leading 11 is discarded to incidate a multibyte encoding, and the leading 10 in the second byte is dropped) which is `10100100`.   `0xA4` is also `10100100`.
@hudcap That is correct.  When a byte leads with 1 the first buyte must be `11xxxxxx` followed by a sequence of `10yyyyyy` bytes. The next byte that has the form `0zzzzzzz` (ascii point) or `11xxxxxx` (new unicode point) ends the varible encoding. 
I would support a patch used a try/except as above since this seems like a real Stata issue. I resaved the file using Stata 14.2 and got the same incorrect format. 
> It is not obvious to me. Stata does some undocumented things w.r.t. latin-1 encodings.
> 
> This block would make it work.
> 
> ```
>         try:
>             return s.decode('utf-8')
>         except UnicodeDecodeError:
>             return s.decode('unicode_escape')
> ```

What if a string has these invalid Unicode encodings as well as valid UTF-8?
Is there any way to handle that?
If they are not latin1 or ubicode then it will error, which is the right
decision since we don't know what the right value is.

On Tue, Apr 2, 2019, 18:48 Yehuda Davis <notifications@github.com> wrote:

> It is not obvious to me. Stata does some undocumented things w.r.t.
> latin-1 encodings.
>
> This block would make it work.
>
>         try:
>             return s.decode('utf-8')
>         except UnicodeDecodeError:
>             return s.decode('unicode_escape')
>
> What if a string has these invalid Unicode encodings as well as valid
> UTF-8?
> Is there any way to handle that?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/25960#issuecomment-479116816>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AFU5Rcw5m_E_Lv-04JMeGBcJRuWjxjqSks5vc5fwgaJpZM4cYGcN>
> .
>

> I would support a patch used a try/except as above since this seems like a real Stata issue. I resaved the file using Stata 14.2 and got the same incorrect format.

Should I make a PR?
I'm fine with that, but it's your fix, and I don't want to step on your toes :)
When you open this file in Stata, the bad character shows up as � (U+FFFD), so Stata also doesn't correctly read it (although it does read it).  I also cannot create a file that uses Latin-1 -- Stata correctly always writes unicode C2 A4 for the offending character, and pandas reads it correctly.  I wonder if this dataset was dumped using some other program.

It seems that this file is not a valid Stata file.

Ultimately it probably doesn't make sense to provide ad hoc paths that might support damaged files when the current implementation seems to do a good job with correctly formatted Stata files. 
Good point -- I completely agree. Given that the file is offered in many different formats, it's reasonable to assume that some other program created it.
In fact, I think that the character is supposed to be C2 A7 (section sign).

I'll correct the file before importing it.
Is there an intelligent way to do this?
I can't simply replace all A4 with C2 A7, because there is at least one other instance of that byte code that is not meant to be Unicode, but rather part of a number.

Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -1136,7 +1136,7 @@ def _get_varlist(self):
         elif self.format_version == 118:
             b = 129
 
-        return [self._null_terminate(self.path_or_buf.read(b))
+        return [self._decode(self.path_or_buf.read(b))
                 for i in range(self.nvar)]
 
     # Returns the format list
@@ -1150,7 +1150,7 @@ def _get_fmtlist(self):
         else:
             b = 7
 
-        return [self._null_terminate(self.path_or_buf.read(b))
+        return [self._decode(self.path_or_buf.read(b))
                 for i in range(self.nvar)]
 
     # Returns the label list
@@ -1161,7 +1161,7 @@ def _get_lbllist(self):
             b = 33
         else:
             b = 9
-        return [self._null_terminate(self.path_or_buf.read(b))
+        return [self._decode(self.path_or_buf.read(b))
                 for i in range(self.nvar)]
 
     def _get_variable_labels(self):
@@ -1169,10 +1169,10 @@ def _get_variable_labels(self):
             vlblist = [self._decode(self.path_or_buf.read(321))
                        for i in range(self.nvar)]
         elif self.format_version > 105:
-            vlblist = [self._null_terminate(self.path_or_buf.read(81))
+            vlblist = [self._decode(self.path_or_buf.read(81))
                        for i in range(self.nvar)]
         else:
-            vlblist = [self._null_terminate(self.path_or_buf.read(32))
+            vlblist = [self._decode(self.path_or_buf.read(32))
                        for i in range(self.nvar)]
         return vlblist
 
@@ -1191,11 +1191,11 @@ def _get_data_label(self):
             return self._decode(self.path_or_buf.read(strlen))
         elif self.format_version == 117:
             strlen = struct.unpack('b', self.path_or_buf.read(1))[0]
-            return self._null_terminate(self.path_or_buf.read(strlen))
+            return self._decode(self.path_or_buf.read(strlen))
         elif self.format_version > 105:
-            return self._null_terminate(self.path_or_buf.read(81))
+            return self._decode(self.path_or_buf.read(81))
         else:
-            return self._null_terminate(self.path_or_buf.read(32))
+            return self._decode(self.path_or_buf.read(32))
 
     def _get_time_stamp(self):
         if self.format_version == 118:
@@ -1203,9 +1203,9 @@ def _get_time_stamp(self):
             return self.path_or_buf.read(strlen).decode("utf-8")
         elif self.format_version == 117:
             strlen = struct.unpack('b', self.path_or_buf.read(1))[0]
-            return self._null_terminate(self.path_or_buf.read(strlen))
+            return self._decode(self.path_or_buf.read(strlen))
         elif self.format_version > 104:
-            return self._null_terminate(self.path_or_buf.read(18))
+            return self._decode(self.path_or_buf.read(18))
         else:
             raise ValueError()
 
@@ -1266,10 +1266,10 @@ def _read_old_header(self, first_char):
                              .format(','.join(str(x) for x in typlist)))
 
         if self.format_version > 108:
-            self.varlist = [self._null_terminate(self.path_or_buf.read(33))
+            self.varlist = [self._decode(self.path_or_buf.read(33))
                             for i in range(self.nvar)]
         else:
-            self.varlist = [self._null_terminate(self.path_or_buf.read(9))
+            self.varlist = [self._decode(self.path_or_buf.read(9))
                             for i in range(self.nvar)]
         self.srtlist = struct.unpack(
             self.byteorder + ('h' * (self.nvar + 1)),
@@ -1326,13 +1326,20 @@ def _calcsize(self, fmt):
                 struct.calcsize(self.byteorder + fmt))
 
     def _decode(self, s):
-        s = s.partition(b"\0")[0]
-        return s.decode('utf-8')
-
-    def _null_terminate(self, s):
         # have bytes not strings, so must decode
         s = s.partition(b"\0")[0]
-        return s.decode(self._encoding)
+        try:
+            return s.decode(self._encoding)
+        except UnicodeDecodeError:
+            # GH 25960, fallback to handle incorrect format produced when 117
+            # files are converted to 118 files in Stata
+            msg = """
+One or more strings in the dta file could not be decoded using {encoding}, and
+so the fallback encoding of latin-1 is being used.  This can happen when a file
+has been incorrectly encoded by Stata or some other software. You should verify
+the string values returned are correct."""
+            warnings.warn(msg.format(encoding=self._encoding), UnicodeWarning)
+            return s.decode('latin-1')
 
     def _read_value_labels(self):
         if self._value_labels_read:
@@ -1362,7 +1369,7 @@ def _read_value_labels(self):
             if not slength:
                 break  # end of value label table (format < 117)
             if self.format_version <= 117:
-                labname = self._null_terminate(self.path_or_buf.read(33))
+                labname = self._decode(self.path_or_buf.read(33))
             else:
                 labname = self._decode(self.path_or_buf.read(129))
             self.path_or_buf.read(3)  # padding
@@ -1384,12 +1391,8 @@ def _read_value_labels(self):
             self.value_label_dict[labname] = dict()
             for i in range(n):
                 end = off[i + 1] if i < n - 1 else txtlen
-                if self.format_version <= 117:
-                    self.value_label_dict[labname][val[i]] = (
-                        self._null_terminate(txt[off[i]:end]))
-                else:
-                    self.value_label_dict[labname][val[i]] = (
-                        self._decode(txt[off[i]:end]))
+                self.value_label_dict[labname][val[i]] = \
+                    self._decode(txt[off[i]:end])
             if self.format_version >= 117:
                 self.path_or_buf.read(6)  # </lbl>
         self._value_labels_read = True
@@ -1544,7 +1547,7 @@ def read(self, nrows=None, convert_dates=None,
         for col, typ in zip(data, self.typlist):
             if type(typ) is int:
                 data[col] = data[col].apply(
-                    self._null_terminate, convert_dtype=True)
+                    self._decode, convert_dtype=True)
 
         data = self._insert_strls(data)
 


Test patch:
diff --git a/pandas/tests/io/data/stata1_encoding_118.dta b/pandas/tests/io/data/stata1_encoding_118.dta
new file mode 100644
Binary files /dev/null and b/pandas/tests/io/data/stata1_encoding_118.dta differ
diff --git a/pandas/tests/io/test_stata.py b/pandas/tests/io/test_stata.py
--- a/pandas/tests/io/test_stata.py
+++ b/pandas/tests/io/test_stata.py
@@ -66,6 +66,8 @@ def setup_method(self, datapath):
         self.dta4_117 = os.path.join(self.dirpath, 'stata4_117.dta')
 
         self.dta_encoding = os.path.join(self.dirpath, 'stata1_encoding.dta')
+        self.dta_encoding_118 = os.path.join(self.dirpath,
+                                             'stata1_encoding_118.dta')
 
         self.csv14 = os.path.join(self.dirpath, 'stata5.csv')
         self.dta14_113 = os.path.join(self.dirpath, 'stata5_113.dta')
@@ -1608,3 +1610,18 @@ def test_strl_latin1(self):
                     val = gso.split(b'\x00')[-2]
                     size = gso[gso.find(b'\x82') + 1]
                     assert len(val) == size - 1
+
+    def test_encoding_latin1_118(self):
+        # GH 25960
+        msg = """
+One or more strings in the dta file could not be decoded using utf-8, and
+so the fallback encoding of latin-1 is being used.  This can happen when a file
+has been incorrectly encoded by Stata or some other software. You should verify
+the string values returned are correct."""
+        with tm.assert_produces_warning(UnicodeWarning) as w:
+            encoded = read_stata(self.dta_encoding_118)
+            assert len(w) == 151
+            assert w[0].message.args[0] == msg
+
+        expected = pd.DataFrame([['Düsseldorf']] * 151, columns=['kreis1849'])
+        tm.assert_frame_equal(encoded, expected)