Repository: pandas-dev/pandas

Issue:
StataReader.variable_labels() does not read variable label correctly for stata datasets saved under Stata 13 using 'save' (but it can read datasets saved using 'saveold')
If I use SataReader to read a Stata dataset saved in Stata 13 using the `save` command, I can get the data but not the variable labels. 

If, however, I use the `saveold` command in Stata 13, I am able to get the variable labels in Python3 using `StataReader.variable_labels()`.

Can anyone suggest how to accommodate Stata 13? Thanks,



Discussion:
docs are here: http://pandas.pydata.org/pandas-docs/stable/io.html#reading-from-stata-format

something like:

```
reader = pandas.io.stata.StataReader(file)

# labels
reader.variable_labels()

# data
reader.data(....)
```

look inside the `pandas.io.stata.read_stata` (and doc-string of `StataReader`)

closing as a usage question

Hello, I'm sorry if I wasn't clear earlier. 

I did use the `variables_label()` method of `reader`. But this does NOT work for Stata datasets saved in later versions of Stata (e.g. Stata 13) using the `save` command. (It DOES work if the dataset was saved in Stata 13 using the `saveold` command.) 

Can you please re-open this issue? It is still not resolved (I am using the latest Pandas master branch). Thanks.

ok, so this is a feature/bug request then? ok

cc @bashtage

Yes it is a bug/feature request. I guess Stata changed something in how they save data files, which means that the Stata reader needs to be updated to accommodate this change. Many thanks! 

@shafiquejamal  Would be helpful if you could share a simple example file .dta which produces the problem, as well as a v12 one that works.   

This looks like it is implemented in the v13 path - although it probably is buggy

Certainly. I have a couple of .dta files of about 450kb each that I can share (problem dataset `HHRosterEducHealth_small_varwithnolabel_notsaveold` and non-problem dataset `HHRosterEducHealth_small_varwithnolabel_saveold`). 

I tried dragging them into this comment window, but I'm getting this error at the bottom of this comment window: "Unfortunately, we don't support that file type.  Try again with a PNG, GIF, or JPG." 

How can I share these .dta files with you? Thanks,

@shafiquejamal put them up on a public dropbox / share site. I think you can do it via gist as well.

and post the link here.

Here is the dropbox link:

https://www.dropbox.com/sh/4r0fhspsiwpim5p/AACBaC-lu7TaNPLUQQgU_rt4a

So StataReader can handle the file ending in `_saveold.dta` (saved using an old Stata dataset format), not the file ending in `_notsaveold.dta` (saved using the newer Stata dataset format). Thanks.

The bug, unfortunately, seems to be in stata.  Stata's dta file definition claims that it gives the offset to the start of this segment as 1 of 14 8 byte values, in <map>.  Unfortunately, this value is 0 (0000 0000 0000 0000 in the file) in this file, and is 0 in 1 I just saved from Stata 13.

The code appears to be a correct implementation of Stata's documented file format, so I'm not sure if this should be "fixed" (which would be to hack around Stata's problem).

Thanks for looking into this so quickly. I'll see about contacting folks at Stata to see whether they can fix their documentation, which would then just justify modifying Pandas. 

To summarize then: the problem is that the offset (to the start of the segment in the dta file that defines the variable labels) should be 1, according to Stata's documentation (in `help dta`), but this offset is in fact 0 instead. Correct?

Many thanks,


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -520,8 +520,15 @@ def _read_header(self):
                 self.byteorder + 'q', self.path_or_buf.read(8))[0] + 9
             seek_value_label_names = struct.unpack(
                 self.byteorder + 'q', self.path_or_buf.read(8))[0] + 19
-            seek_variable_labels = struct.unpack(
-                self.byteorder + 'q', self.path_or_buf.read(8))[0] + 17
+            # Stata 117 data files do not follow the described format.  This is
+            # a work around that uses the previous label, 33 bytes for each
+            # variable, 20 for the closing tag and 17 for the opening tag
+            self.path_or_buf.read(8)  # <variable_lables>, throw away
+            seek_variable_labels = seek_value_label_names + (33*self.nvar) + 20 + 17
+            # Below is the original, correct code (per Stata sta format doc,
+            # although this is not followed in actual 117 dtas)
+            #seek_variable_labels = struct.unpack(
+            #    self.byteorder + 'q', self.path_or_buf.read(8))[0] + 17
             self.path_or_buf.read(8)  # <characteristics>
             self.data_location = struct.unpack(
                 self.byteorder + 'q', self.path_or_buf.read(8))[0] + 6


Test patch:
diff --git a/pandas/io/tests/data/stata7_115.dta b/pandas/io/tests/data/stata7_115.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata7_115.dta differ
diff --git a/pandas/io/tests/data/stata7_117.dta b/pandas/io/tests/data/stata7_117.dta
new file mode 100644
Binary files /dev/null and b/pandas/io/tests/data/stata7_117.dta differ
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -68,6 +68,9 @@ def setUp(self):
         self.dta15_115 = os.path.join(self.dirpath, 'stata6_115.dta')
         self.dta15_117 = os.path.join(self.dirpath, 'stata6_117.dta')
 
+        self.dta16_115 = os.path.join(self.dirpath, 'stata7_115.dta')
+        self.dta16_117 = os.path.join(self.dirpath, 'stata7_117.dta')
+
     def read_dta(self, file):
         return read_stata(file, convert_dates=True)
 
@@ -199,7 +202,7 @@ def test_read_dta4(self):
                      'labeled_with_missings', 'float_labelled'])
 
         # these are all categoricals
-        expected = pd.concat([ Series(pd.Categorical(value)) for col, value in expected.iteritems() ],axis=1)
+        expected = pd.concat([ Series(pd.Categorical(value)) for col, value in compat.iteritems(expected)],axis=1)
 
         tm.assert_frame_equal(parsed_113, expected)
         tm.assert_frame_equal(parsed_114, expected)
@@ -551,6 +554,18 @@ def test_bool_uint(self):
             written_and_read_again = written_and_read_again.set_index('index')
             tm.assert_frame_equal(written_and_read_again, expected)
 
+    def test_variable_labels(self):
+        sr_115 = StataReader(self.dta16_115).variable_labels()
+        sr_117 = StataReader(self.dta16_117).variable_labels()
+        keys = ('var1', 'var2', 'var3')
+        labels = ('label1', 'label2', 'label3')
+        for k,v in compat.iteritems(sr_115):
+            self.assertTrue(k in sr_117)
+            self.assertTrue(v == sr_117[k])
+            self.assertTrue(k in keys)
+            self.assertTrue(v in labels)
+
+
 if __name__ == '__main__':
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],
                    exit=False)