BUG: Unable to open Stata 118 or 119 format files saved in big-endian…
State: closed

… format that contain strL data

- [x] closes #58638
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
Just to check - are the dta files produced by Stata?
One more question: Does stata read all of the dta files correctly?
These files were not created with Stata but were made using the published specification. I have however checked that they all load correctly into Stata and have used it to verify that they are constructed correctly:

```
. use stata12_118.dta

. list

     +------------------------------+
     |  x     y                   z |
     |------------------------------|
  1. |  1   abc           abcdefghi |
  2. |  3   cba   qwertywertyqwerty |
  3. | 93                      strl |
     +------------------------------+

. dtaverify stata12_118.dta
  (file "stata12_118.dta" is .dta-format 118 from Stata 14, 15, 16, 17, or 18)

 1. reading and verifying header
    release is 118
    byteorder LSF
    K (# of vars) is 3
    N (# of obs) is 3                    
    label length 0 ||
    date length 17 |19 Feb 2015 09:20|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  158
    map[ 3] =                  281
    map[ 4] =                  320
    map[ 5] =                  728
    map[ 6] =                  757
    map[ 7] =                  947
    map[ 8] =                 1373
    map[ 9] =                 2371
    map[10] =                 2406
    map[11] =                 2473
    map[12] =                 2581
    map[13] =                 2610
    map[14] =                 2622
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_118 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (3 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata12_119.dta

. list

     +------------------------------+
     |  x     y                   z |
     |------------------------------|
  1. |  1   abc           abcdefghi |
  2. |  3   cba   qwertywertyqwerty |
  3. | 93                      strl |
     +------------------------------+

. dtaverify stata12_119.dta
  (file "stata12_119.dta" is .dta-format 119 from Stata 15, 16, 17, or 18)

 1. reading and verifying header
    release is 119
    byteorder LSF
    K (# of vars) is 3
    N (# of obs) is 3                    
    label length 0 ||
    date length 17 |19 Feb 2015 09:20|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  160
    map[ 3] =                  283
    map[ 4] =                  322
    map[ 5] =                  730
    map[ 6] =                  767
    map[ 7] =                  957
    map[ 8] =                 1383
    map[ 9] =                 2381
    map[10] =                 2416
    map[11] =                 2483
    map[12] =                 2591
    map[13] =                 2620
    map[14] =                 2632
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_119 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (3 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata12_be_117.dta

. list

     +------------------------------+
     |  x     y                   z |
     |------------------------------|
  1. |  1   abc           abcdefghi |
  2. |  3   cba   qwertywertyqwerty |
  3. | 93                      strl |
     +------------------------------+

. dtaverify stata12_be_117.dta
  (file "stata12_be_117.dta" is .dta-format 117 from Stata 13)

 1. reading and verifying header
    release is 117
    byteorder MSF
    K (# of vars) is 3
    N (# of obs) is 3
    label length 0 ||
    date length 17 |19 Feb 2015 09:20|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  153
    map[ 3] =                  276
    map[ 4] =                  315
    map[ 5] =                  435
    map[ 6] =                  464
    map[ 7] =                  630
    map[ 8] =                  768
    map[ 9] =                 1046
    map[10] =                 1081
    map[11] =                 1148
    map[12] =                 1244
    map[13] =                 1273
    map[14] =                 1285
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_117 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (3 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata12_be_118.dta

. list

     +------------------------------+
     |  x     y                   z |
     |------------------------------|
  1. |  1   abc           abcdefghi |
  2. |  3   cba   qwertywertyqwerty |
  3. | 93                      strl |
     +------------------------------+

. dtaverify stata12_be_118.dta
  (file "stata12_be_118.dta" is .dta-format 118 from Stata 14, 15, 16, 17, or 18)

 1. reading and verifying header
    release is 118
    byteorder MSF
    K (# of vars) is 3
    N (# of obs) is 3                    
    label length 0 ||
    date length 17 |19 Feb 2015 09:20|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  158
    map[ 3] =                  281
    map[ 4] =                  320
    map[ 5] =                  728
    map[ 6] =                  757
    map[ 7] =                  947
    map[ 8] =                 1373
    map[ 9] =                 2371
    map[10] =                 2406
    map[11] =                 2473
    map[12] =                 2581
    map[13] =                 2610
    map[14] =                 2622
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_118 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (3 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata12_be_119.dta

. list

     +------------------------------+
     |  x     y                   z |
     |------------------------------|
  1. |  1   abc           abcdefghi |
  2. |  3   cba   qwertywertyqwerty |
  3. | 93                      strl |
     +------------------------------+

. dtaverify stata12_be_119.dta
  (file "stata12_be_119.dta" is .dta-format 119 from Stata 15, 16, 17, or 18)

 1. reading and verifying header
    release is 119
    byteorder MSF
    K (# of vars) is 3
    N (# of obs) is 3                    
    label length 0 ||
    date length 17 |19 Feb 2015 09:20|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  160
    map[ 3] =                  283
    map[ 4] =                  322
    map[ 5] =                  730
    map[ 6] =                  767
    map[ 7] =                  957
    map[ 8] =                 1383
    map[ 9] =                 2381
    map[10] =                 2416
    map[11] =                 2483
    map[12] =                 2591
    map[13] =                 2620
    map[14] =                 2632
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_119 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (3 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata14_119.dta
(This is a  Ünicode data label)

. list

     +---------------------------------------------------------------------------+
     | Things   Cities   Unicode~l   Ints   Floats              Bytes      Longs |
     |---------------------------------------------------------------------------|
  1. |    Cat   Bogota      Bogotá      1        1   option b Ünicode          1 |
  2. |    Dog   Boston   Uzunköprü      .        .                  .          . |
  3. |  Plane     Rome      Tromsø      0        0           option a          0 |
  4. | Potato    Tokyo      Elâzığ     -4        4                  4          4 |
  5. |                                  0    .3333           option a   .3333333 |
     +---------------------------------------------------------------------------+

. dtaverify stata14_119.dta
  (file "stata14_119.dta" is .dta-format 119 from Stata 15, 16, 17, or 18)

 1. reading and verifying header
    release is 119
    byteorder LSF
    K (# of vars) is 7
    N (# of obs) is 5                    
    label length 30 |This is a  Ünicode data label|
    date length 17 |28 Apr 2015 17:58|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  190
    map[ 3] =                  313
    map[ 4] =                  360
    map[ 5] =                 1284
    map[ 6] =                 1337
    map[ 7] =                 1755
    map[ 8] =                 2697
    map[ 9] =                 4979
    map[10] =                 5014
    map[11] =                 5202
    map[12] =                 5335
    map[13] =                 5562
    map[14] =                 5574
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_119 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (4 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (1 labels in file)

no errors or warnings above; .dta file valid

. use stata14_be_118.dta
(This is a  Ünicode data label)

. list

     +---------------------------------------------------------------------------+
     | Things   Cities   Unicode~l   Ints   Floats              Bytes      Longs |
     |---------------------------------------------------------------------------|
  1. |    Cat   Bogota      Bogotá      1        1   option b Ünicode          1 |
  2. |    Dog   Boston   Uzunköprü      .        .                  .          . |
  3. |  Plane     Rome      Tromsø      0        0           option a          0 |
  4. | Potato    Tokyo      Elâzığ     -4        4                  4          4 |
  5. |                                  0    .3333           option a   .3333333 |
     +---------------------------------------------------------------------------+

. dtaverify stata14_be_118.dta
  (file "stata14_be_118.dta" is .dta-format 118 from Stata 14, 15, 16, 17, or 18)

 1. reading and verifying header
    release is 118
    byteorder MSF
    K (# of vars) is 7
    N (# of obs) is 5                    
    label length 30 |This is a  Ünicode data label|
    date length 17 |28 Apr 2015 17:58|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  188
    map[ 3] =                  311
    map[ 4] =                  358
    map[ 5] =                 1282
    map[ 6] =                 1319
    map[ 7] =                 1737
    map[ 8] =                 2679
    map[ 9] =                 4961
    map[10] =                 4996
    map[11] =                 5184
    map[12] =                 5317
    map[13] =                 5544
    map[14] =                 5556
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_118 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (4 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (1 labels in file)

no errors or warnings above; .dta file valid

. use stata14_be_119.dta
(This is a  Ünicode data label)

. list

     +---------------------------------------------------------------------------+
     | Things   Cities   Unicode~l   Ints   Floats              Bytes      Longs |
     |---------------------------------------------------------------------------|
  1. |    Cat   Bogota      Bogotá      1        1   option b Ünicode          1 |
  2. |    Dog   Boston   Uzunköprü      .        .                  .          . |
  3. |  Plane     Rome      Tromsø      0        0           option a          0 |
  4. | Potato    Tokyo      Elâzığ     -4        4                  4          4 |
  5. |                                  0    .3333           option a   .3333333 |
     +---------------------------------------------------------------------------+

. dtaverify stata14_be_119.dta
  (file "stata14_be_119.dta" is .dta-format 119 from Stata 15, 16, 17, or 18)

 1. reading and verifying header
    release is 119
    byteorder MSF
    K (# of vars) is 7
    N (# of obs) is 5                    
    label length 30 |This is a  Ünicode data label|
    date length 17 |28 Apr 2015 17:58|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  190
    map[ 3] =                  313
    map[ 4] =                  360
    map[ 5] =                 1284
    map[ 6] =                 1337
    map[ 7] =                 1755
    map[ 8] =                 2697
    map[ 9] =                 4979
    map[10] =                 5014
    map[11] =                 5202
    map[12] =                 5335
    map[13] =                 5562
    map[14] =                 5574
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_119 cannot verify that values are correct)
    verifying construction
    (1 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (4 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (1 labels in file)

no errors or warnings above; .dta file valid

. use stata16_119.dta

. list

     +-----------------------------------------------+
     |   utf8   latin1   ascii   utf8_s~l   ascii_~l |
     |-----------------------------------------------|
  1. | ραηδας   PÄNDÄS       p     ραηδας          p |
  2. | ƤĀńĐąŜ        Ö       a     ƤĀńĐąŜ          a |
  3. | ᴘᴀᴎᴅᴀS        Ü       n     ᴘᴀᴎᴅᴀS          n |
  4. |                       d                     d |
  5. |                       a                     a |
     |-----------------------------------------------|
  6. |                       s                     s |
  7. |                                               |
     +-----------------------------------------------+

. dtaverify stata16_119.dta
  (file "stata16_119.dta" is .dta-format 119 from Stata 15, 16, 17, or 18)

 1. reading and verifying header
    release is 119
    byteorder LSF
    K (# of vars) is 5
    N (# of obs) is 7                    
    label length 0 ||
    date length 17 |31 May 2018 23:59|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  160
    map[ 3] =                  283
    map[ 4] =                  326
    map[ 5] =                  992
    map[ 6] =                 1037
    map[ 7] =                 1341
    map[ 8] =                 2025
    map[ 9] =                 3665
    map[10] =                 3700
    map[11] =                 4266
    map[12] =                 4587
    map[13] =                 4616
    map[14] =                 4628
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_119 cannot verify that values are correct)
    verifying construction
    (2 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (12 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata16_be_118.dta

. list

     +-----------------------------------------------+
     |   utf8   latin1   ascii   utf8_s~l   ascii_~l |
     |-----------------------------------------------|
  1. | ραηδας   PÄNDÄS       p     ραηδας          p |
  2. | ƤĀńĐąŜ        Ö       a     ƤĀńĐąŜ          a |
  3. | ᴘᴀᴎᴅᴀS        Ü       n     ᴘᴀᴎᴅᴀS          n |
  4. |                       d                     d |
  5. |                       a                     a |
     |-----------------------------------------------|
  6. |                       s                     s |
  7. |                                               |
     +-----------------------------------------------+

. dtaverify stata16_be_118.dta
  (file "stata16_be_118.dta" is .dta-format 118 from Stata 14, 15, 16, 17, or 18)

 1. reading and verifying header
    release is 118
    byteorder MSF
    K (# of vars) is 5
    N (# of obs) is 7                    
    label length 0 ||
    date length 17 |31 May 2018 23:59|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  158
    map[ 3] =                  281
    map[ 4] =                  324
    map[ 5] =                  990
    map[ 6] =                 1023
    map[ 7] =                 1327
    map[ 8] =                 2011
    map[ 9] =                 3651
    map[10] =                 3686
    map[11] =                 4252
    map[12] =                 4573
    map[13] =                 4602
    map[14] =                 4614
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_118 cannot verify that values are correct)
    verifying construction
    (2 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (12 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid

. use stata16_be_119.dta

. list

     +-----------------------------------------------+
     |   utf8   latin1   ascii   utf8_s~l   ascii_~l |
     |-----------------------------------------------|
  1. | ραηδας   PÄNDÄS       p     ραηδας          p |
  2. | ƤĀńĐąŜ        Ö       a     ƤĀńĐąŜ          a |
  3. | ᴘᴀᴎᴅᴀS        Ü       n     ᴘᴀᴎᴅᴀS          n |
  4. |                       d                     d |
  5. |                       a                     a |
     |-----------------------------------------------|
  6. |                       s                     s |
  7. |                                               |
     +-----------------------------------------------+

. dtaverify stata16_be_119.dta
  (file "stata16_be_119.dta" is .dta-format 119 from Stata 15, 16, 17, or 18)

 1. reading and verifying header
    release is 119
    byteorder MSF
    K (# of vars) is 5
    N (# of obs) is 7                    
    label length 0 ||
    date length 17 |31 May 2018 23:59|

 2. reading and verifying map
    map[ 1] =                    0
    map[ 2] =                  160
    map[ 3] =                  283
    map[ 4] =                  326
    map[ 5] =                  992
    map[ 6] =                 1037
    map[ 7] =                 1341
    map[ 8] =                 2025
    map[ 9] =                 3665
    map[10] =                 3700
    map[11] =                 4266
    map[12] =                 4587
    map[13] =                 4616
    map[14] =                 4628
    verifying map
    1 2 3 4 5 6 7 8 9 10 11 12 13 14

 3. reading and verifying vartypes

 4. reading and verifying varnames
    verifying varnames unique

 5. reading and verifying sort order
    verifying contents

 6. reading and verifying display formats
    verifying formats
    verifying formats correspond to variable type

 7. reading and verifying value-label assignment
    verifying value-label construction
    verifying value label and corresponding variable types

 8. reading and verifying variable-label assignment
    verifying variable labels construction

 9. reading and verifying characteristics
    (0 characteristics in file)

10. reading and verifying data
    (dtaverify_119 cannot verify that values are correct)
    verifying construction
    (2 strL variables)
    verifying strL construction

11. reading and verifying strLs
    verifying construction
    (12 strLs expected)

12. reading and verifying value label definitions
    verifying construction
    (0 labels in file)

no errors or warnings above; .dta file valid
```
Can you ping on green and I'll merge this in.
Thanks @bashtage . All checks how how completed.
Thanks for the fix.