Repository: pandas-dev/pandas (python)

Issue:
BUG: Invalid Stata file is generated if byteorder='big' is specified and strings are saved as strL type
### Pandas version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas.

- [X] I have confirmed this bug exists on the [main branch](https://pandas.pydata.org/docs/dev/getting_started/install.html#installing-the-development-version-of-pandas) of pandas.


### Reproducible Example

```python
import pandas as pd
df = pd.DataFrame(data={'strings': ["abc", "def", "ghi"]})
df.to_stata("test.dta", byteorder="big", version=117, convert_strl=["strings"]) # <-- invalid file created
```


### Issue Description

Although there is a documented argument defined to set it, the selected  `byteorder` option is not passed to the `StataStrLWriter` ` __init__` function from the `_convert_strls` function. There is also no handling of byteorder when calculating or using `self._o_offet`.

This results in the strL data always being saved in the default byteorder, even if the rest of the file isn't, resulting in following error when attempting to open the saved file, both in Pandas:
```python
>>> import pandas as pd
>>> df = pd.DataFrame(data={'strings': ["abc", "def", "ghi"]})
>>> df
  strings
0     abc
1     def
2     ghi
>>> df.to_stata("test.dta", byteorder="big", version=117, convert_strl=["strings"])
>>> df2 = pd.read_stata("test.dta")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas\io\stata.py", line 2082, in read_stata
    return reader.read()
  File "pandas\io\stata.py", line 1741, in read
    data = self._insert_strls(data)
  File "pandas\io\stata.py", line 1838, in _insert_strls
    data.isetitem(i, [self.GSO[str(k)] for k in data.iloc[:, i]])
  File "pandas\io\stata.py", line 1838, in <listcomp>
    data.isetitem(i, [self.GSO[str(k)] for k in data.iloc[:, i]])
KeyError: '4294967298'
```
or within Stata:
```
. use "test.dta" 
.dta file corrupt
    The strL component of the file is corrupt.  Either the file was incorrectly constructed or it became
    corrupted since it was written.  In case the file was incorrectly constructed,the technical description of
    the problem is that there was an attempt to crosslink a strl to another strl that had not yet been defined.
r(688);

. dtaverify "test.dta"
  (file "test.dta" is .dta-format 117 from Stata 13)

 1. reading and verifying header
    release is 117
    byteorder MSF
    K (# of vars) is 2
    N (# of obs) is 3
    label length 0 ||
    date length 17 |10 Jun 2024 16:05|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  153
    map[ 3] =                  276
    map[ 4] =                  313
    map[ 5] =                  400
    map[ 6] =                  427
    map[ 7] =                  544
    map[ 8] =                  649
    map[ 9] =                  846
    map[10] =                  881
    map[11] =                  930
    map[12] =                 1005
    map[13] =                 1034
    map[14] =                 1046
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_117 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction
SERIOUS ERROR: (var,obs)=(2,1) has (v,o)=(1,2); v not strL
               chkijvo():  3301  subscript invalid
      strl_process_obs():     -  function returned error
     read_data_u_strls():     -  function returned error
           read_data_u():     -  function returned error
             read_data():     -  function returned error
              readfile():     -  function returned error
       verify_dta_file():     -  function returned error
                 <istmt>:     -  function returned error
r(3301);
```

### Expected Behavior

I would expect the saved file to open correctly in both Pandas:
```python
>>> import pandas as pd
>>> df = pd.DataFrame(data={'strings': ["abc", "def", "ghi"]})
>>> df
  strings
0     abc
1     def
2     ghi
>>> df.to_stata("test.dta", byteorder="big", version=117, convert_strl=["strings"])
>>> df2 = pd.read_stata("test.dta")
>>> df2
   index strings
0      0     abc
1      1     def
2      2     ghi
```

and Stata:
```
. use "test.dta" 

. list

     +-----------------+
     | index   strings |
     |-----------------|
  1. |     0       abc |
  2. |     1       def |
  3. |     2       ghi |
     +-----------------+

. dtaverify "test.dta"
  (file "test.dta" is .dta-format 117 from Stata 13)

 1. reading and verifying header
    release is 117
    byteorder MSF
    K (# of vars) is 2
    N (# of obs) is 3
    label length 0 ||
    date length 17 |10 Jun 2024 16:12|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  153
    map[ 3] =                  276
    map[ 4] =                  313
    map[ 5] =                  400
    map[ 6] =                  427
    map[ 7] =                  544
    map[ 8] =                  649
    map[ 9] =                  846
    map[10] =                  881
    map[11] =                  930
    map[12] =                 1005
    map[13] =                 1034
    map[14] =                 1046
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_117 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (3 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid
```

### Installed Versions

<details>
miniforge3\envs\pandas-dev\lib\site-packages\_distutils_hack\__init__.py:26: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS
------------------
commit                : da802479fca521b7992fbf76d5984992cd5d495d
python                : 3.10.14.final.0
python-bits           : 64
OS                    : Windows
OS-release            : 10
Version               : 10.0.19045
machine               : AMD64
processor             : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder             : little
LC_ALL                : None
LANG                  : None
LOCALE                : English_United Kingdom.1252

pandas                : 3.0.0.dev0+629.gda802479fc
numpy                 : 1.26.4
pytz                  : 2024.1
dateutil              : 2.9.0
setuptools            : 69.2.0
pip                   : 24.0
Cython                : 3.0.9
pytest                : 8.1.1
hypothesis            : 6.99.13
sphinx                : 7.2.6
blosc                 : None
feather               : None
xlsxwriter            : 3.1.9
lxml.etree            : 5.1.0
html5lib              : 1.1
pymysql               : 1.4.6
psycopg2              : 2.9.9
jinja2                : 3.1.3
IPython               : 8.22.2
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : 1.3.8
fastparquet           : 2024.2.0
fsspec                : 2024.3.1
gcsfs                 : 2024.3.1
matplotlib            : 3.8.3
numba                 : 0.59.1
numexpr               : 2.9.0
odfpy                 : None
openpyxl              : 3.1.2
pyarrow               : 15.0.2
pyreadstat            : 1.2.7
python-calamine       : None
pyxlsb                : 1.0.10
s3fs                  : 2024.3.1
scipy                 : 1.12.0
sqlalchemy            : 2.0.29
tables                : 3.9.2
tabulate              : 0.9.0
xarray                : 2024.2.0
xlrd                  : 2.0.1
zstandard             : 0.22.0
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None
</details>


Pull request description:
BUG: byteorder option in to_stata is not honoured when writing strL data
- [x] closes #58969  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I have created a local variable to indicate whether the file is being saved to the native byteorder in two places, but am happy to calculate this once in `__init__` and store in `self` for later access if you prefer.

Patch:
diff --git a/doc/source/whatsnew/v3.0.0.rst b/doc/source/whatsnew/v3.0.0.rst
index e621ab2a5b..07f5b01709 100644
--- a/doc/source/whatsnew/v3.0.0.rst
+++ b/doc/source/whatsnew/v3.0.0.rst
@@ -544,6 +544,7 @@ I/O
 - Bug in :class:`DataFrame` and :class:`Series` ``repr`` of :py:class:`collections.abc.Mapping`` elements. (:issue:`57915`)
 - Bug in :meth:`DataFrame.to_dict` raises unnecessary ``UserWarning`` when columns are not unique and ``orient='tight'``. (:issue:`58281`)
 - Bug in :meth:`DataFrame.to_excel` when writing empty :class:`DataFrame` with :class:`MultiIndex` on both axes (:issue:`57696`)
+- Bug in :meth:`DataFrame.to_stata` when writing :class:`DataFrame` and ``byteorder=`big```. (:issue:`58969`)
 - Bug in :meth:`DataFrame.to_string` that raised ``StopIteration`` with nested DataFrames. (:issue:`16098`)
 - Bug in :meth:`read_csv` raising ``TypeError`` when ``index_col`` is specified and ``na_values`` is a dict containing the key ``None``. (:issue:`57547`)
 - Bug in :meth:`read_stata` raising ``KeyError`` when input file is stored in big-endian format and contains strL data. (:issue:`58638`)
diff --git a/pandas/io/stata.py b/pandas/io/stata.py
index 9c6cd2faea..d1e57ad568 100644
--- a/pandas/io/stata.py
+++ b/pandas/io/stata.py
@@ -3037,6 +3037,8 @@ class StataStrLWriter:
         if byteorder is None:
             byteorder = sys.byteorder
         self._byteorder = _set_endianness(byteorder)
+        # Flag whether chosen byteorder matches the system on which we're running
+        self._native_byteorder = self._byteorder == _set_endianness(sys.byteorder)
 
         gso_v_type = "I"  # uint32
         gso_o_type = "Q"  # uint64
@@ -3049,13 +3051,20 @@ class StataStrLWriter:
             o_size = 6
         else:  # version == 119
             o_size = 5
-        self._o_offet = 2 ** (8 * (8 - o_size))
+        if self._native_byteorder:
+            self._o_offet = 2 ** (8 * (8 - o_size))
+        else:
+            self._o_offet = 2 ** (8 * o_size)
         self._gso_o_type = gso_o_type
         self._gso_v_type = gso_v_type
 
     def _convert_key(self, key: tuple[int, int]) -> int:
         v, o = key
-        return v + self._o_offet * o
+        if self._native_byteorder:
+            return v + self._o_offet * o
+        else:
+            # v, o will be swapped when applying byteorder
+            return o + self._o_offet * v
 
     def generate_table(self) -> tuple[dict[str, tuple[int, int]], DataFrame]:
         """
@@ -3532,7 +3541,9 @@ class StataWriter117(StataWriter):
         ]
 
         if convert_cols:
-            ssw = StataStrLWriter(data, convert_cols, version=self._dta_version)
+            ssw = StataStrLWriter(
+                data, convert_cols, version=self._dta_version, byteorder=self._byteorder
+            )
             tab, new_data = ssw.generate_table()
             data = new_data
             self._strl_blob = ssw.generate_blob(tab)


Test patch:
diff --git a/pandas/tests/io/test_stata.py b/pandas/tests/io/test_stata.py
index bf17e62985..2534df6a82 100644
--- a/pandas/tests/io/test_stata.py
+++ b/pandas/tests/io/test_stata.py
@@ -1678,7 +1678,8 @@ The repeated labels are:\n-+\nwolof
         formatted = df.loc[0, column + "_fmt"]
         assert unformatted == formatted
 
-    def test_writer_117(self, temp_file):
+    @pytest.mark.parametrize("byteorder", ["little", "big"])
+    def test_writer_117(self, byteorder, temp_file):
         original = DataFrame(
             data=[
                 [
@@ -1736,6 +1737,7 @@ The repeated labels are:\n-+\nwolof
         original.to_stata(
             path,
             convert_dates={"datetime": "tc"},
+            byteorder=byteorder,
             convert_strl=["forced_strl"],
             version=117,
         )
@@ -1940,7 +1942,8 @@ the string values returned are correct."""
                 assert reader._nvar == 32999
 
     @pytest.mark.parametrize("version", [118, 119, None])
-    def test_utf8_writer(self, version, temp_file):
+    @pytest.mark.parametrize("byteorder", ["little", "big"])
+    def test_utf8_writer(self, version, byteorder, temp_file):
         cat = pd.Categorical(["a", "β", "ĉ"], ordered=True)
         data = DataFrame(
             [
@@ -1968,6 +1971,7 @@ the string values returned are correct."""
             convert_strl=["strls"],
             variable_labels=variable_labels,
             write_index=False,
+            byteorder=byteorder,
             version=version,
             value_labels=value_labels,
         )