Repository: pandas-dev/pandas

Issue:
0.15.0 test_stata  strikes back on various less common platforms on 
e.g. 64bit big-endian
https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=s390x&ver=0.15.0-1&stamp=1414699796
and others
https://buildd.debian.org/status/package.php?p=pandas
(some have other failures as well, yet to find a moment to "analyze", but e.g. hurd-i386 fails test_precise_conversion)

could/should all stata tests be skipped on big-endians?



Discussion:
Probably for now - there was a non-trivial change in the way that data is read and written (it is much faster now, but obviously doesn't work where not tested).  I would guess there are some endian issues when reading stata files created on LE when using BE - probably a `<` or `>` missing somewhere.  

With no BE to test on it is hard to know.

I can give you access to a BE box... unfortunately it is down ATM (so http://nipy.bic.berkeley.edu/waterfall?category=pandas says offline :-/) -- will bring it up tomorrow.  you send me via email access information (login, public ssh key) and state fingerprint of your ssh key here for such a lousy verification ;-)

ok -- that box online and freshier reports could be found among '-sparc-' ones on http://nipy.bic.berkeley.edu/waterfall?category=pandas  .  My offer above is still in place @bashtage 

I plan on taking you up on it - a little bandwidth constrained for the next 10 days or so.  Should probably mark these as skip since it will be a while until there can be a fix.

@yarikoptic    Email on the way.

Fingerprint:
4d:d4:e7:0d:33:12:5f:d7:2e:f1:82:11:2c:4f:ff:ec kevin.sheppard@economics.ox.ac.uk

@bashtage update on this?

Did not get access to BE server yet.

On Nov 30, 2014 5:31 PM, jreback notifications@github.com wrote:

@bashtagehttps://github.com/bashtage update on this?

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/8688#issuecomment-64992513.

 Damn I have missed  your email I guess,  sorry @basgtage !... I  will take care about it with an hour

@bashtage reply went via email -- account is ready


Resolving patch:
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -833,6 +833,7 @@ def __init__(self, path_or_buf, encoding='iso-8859-1'):
         self._missing_values = False
         self._data_read = False
         self._value_labels_read = False
+        self._native_byteorder =  _set_endianness(sys.byteorder)
         if isinstance(path_or_buf, str):
             path_or_buf, encoding = get_filepath_or_buffer(
                 path_or_buf, encoding=self._default_encoding
@@ -1195,13 +1196,16 @@ def data(self, convert_dates=True, convert_categoricals=True, index=None,
         dtype = []  # Convert struct data types to numpy data type
         for i, typ in enumerate(self.typlist):
             if typ in self.NUMPY_TYPE_MAP:
-                dtype.append(('s' + str(i), self.NUMPY_TYPE_MAP[typ]))
+                dtype.append(('s' + str(i), self.byteorder + self.NUMPY_TYPE_MAP[typ]))
             else:
                 dtype.append(('s' + str(i), 'S' + str(typ)))
         dtype = np.dtype(dtype)
         read_len = count * dtype.itemsize
         self.path_or_buf.seek(self.data_location)
         data = np.frombuffer(self.path_or_buf.read(read_len),dtype=dtype,count=count)
+        # if necessary, swap the byte order to native here
+        if self.byteorder != self._native_byteorder:
+            data = data.byteswap().newbyteorder()
         self._data_read = True
 
         if convert_categoricals:


Test patch: