Repository: scikit-hep/awkward

Issue:
ak.to_arrow should convert to string instead of large_string
If you load the bikeroutes from [here](https://github.com/jpivarski/2020-07-06-scipy2020/blob/c951aa50bbfa879f011c9f2c343a3d0168be28e0/bikeroutes-execution.ipynb) and try to convert to arrow, it works:
```
arr2 = ak.to_arrow(bikeroutes['features']). # cannot convert the top-level Record, so lose some attributes
t = pa.Table.from_arrays([arr2], names=['features'])
```
but this cannot be written to parquet because the schema is as follows
```
features: struct<type: large_string, properties: struct<STREET: large_string, TYPE: large_string, BIKEROUTE: large_string, F_STREET: large_string, T_STREET: dictionary<values=large_string, indices=int64, ordered=0>>, geometry: struct<type: large_string, coordinates: large_list<item: large_list<item: large_list<item: double>>>>>
  child 0, type: large_string
  child 1, properties: struct<STREET: large_string, TYPE: large_string, BIKEROUTE: large_string, F_STREET: large_string, T_STREET: dictionary<values=large_string, indices=int64, ordered=0>>
      child 0, STREET: large_string
      child 1, TYPE: large_string
      child 2, BIKEROUTE: large_string
      child 3, F_STREET: large_string
      child 4, T_STREET: dictionary<values=large_string, indices=int64, ordered=0>
  child 2, geometry: struct<type: large_string, coordinates: large_list<item: large_list<item: large_list<item: double>>>>
      child 0, type: large_string
      child 1, coordinates: large_list<item: large_list<item: large_list<item: double>>>
          child 0, item: large_list<item: large_list<item: double>>
              child 0, item: large_list<item: double>
                  child 0, item: double
```
Arrow doesn't know how to write the "large strings", but none of the input actually has any strings long enough to need that, would be 2**31 bytes, 2GB. Of course, arrow *should* be able to cope, but it does not.



Discussion:
In practice, I believe this would need converting large_binary -> binary and LargeStringArray->StringArray in to_arrow; but I also wonder at large_list and the like.
This is a good point and worth fixing, though I know how it happened. Awkward defaults to 64-bit when not constrained by inputs, but Arrow likes 32-bit. The policy should be different: instead of trying the closest type-match, it should use minimum sizes for the scale of the array.
It's also a good first issue because the translation is all in two functions in awkward1.operatons.convert, on the Python side.
> but Arrow likes 32-bit

In this case, also a limitation of parquet's string representation.

Resolving patch:
diff --git a/src/awkward1/operations/convert.py b/src/awkward1/operations/convert.py
--- a/src/awkward1/operations/convert.py
+++ b/src/awkward1/operations/convert.py
@@ -1326,6 +1326,16 @@ def recurse(layout, mask=None):
             layout,
             (awkward1.layout.ListOffsetArray64, awkward1.layout.ListOffsetArrayU32),
         ):
+            offsets = numpy.asarray(layout.offsets)
+
+            if len(offsets) == 0 or numpy.max(offsets) <= numpy.iinfo(numpy.int32).max:
+                small_layout = awkward1.layout.ListOffsetArray32(
+                    awkward1.layout.Index32(offsets.astype(numpy.int32)),
+                    layout.content,
+                    parameters=layout.parameters,
+                )
+                return recurse(small_layout, mask=mask)
+
             offsets = numpy.asarray(layout.offsets, dtype=numpy.int64)
 
             if layout.parameter("__array__") == "bytestring":


Test patch:
diff --git a/tests/test_0224-arrow-to-awkward.py b/tests/test_0224-arrow-to-awkward.py
--- a/tests/test_0224-arrow-to-awkward.py
+++ b/tests/test_0224-arrow-to-awkward.py
@@ -45,7 +45,7 @@ def test_toarrow():
         numpy.array([1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9]))
     offsets = awkward1.layout.Index64(numpy.array([0, 3, 3, 5, 6, 9]))
     array = awkward1.layout.ListOffsetArray64(offsets, content)
-    assert isinstance(awkward1.to_arrow(array), (pyarrow.LargeListArray))
+    assert isinstance(awkward1.to_arrow(array), (pyarrow.ListArray))
     assert awkward1.to_arrow(array).to_pylist() == [[1.1, 2.2, 3.3], [], [
         4.4, 5.5], [6.6], [7.7, 8.8, 9.9]]
 
@@ -53,7 +53,7 @@ def test_toarrow():
         numpy.array([1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9]))
     offsets = awkward1.layout.IndexU32(numpy.array([0, 3, 3, 5, 6, 9]))
     array = awkward1.layout.ListOffsetArrayU32(offsets, content)
-    assert isinstance(awkward1.to_arrow(array), (pyarrow.LargeListArray))
+    assert isinstance(awkward1.to_arrow(array), (pyarrow.ListArray))
     assert awkward1.to_arrow(array).to_pylist() == [[1.1, 2.2, 3.3], [], [
         4.4, 5.5], [6.6], [7.7, 8.8, 9.9]]
 
@@ -74,12 +74,11 @@ def test_toarrow():
     stops = awkward1.layout.Index64(numpy.array([2, 3]))
     listarray = awkward1.layout.ListArray64(starts, stops, regulararray)
 
-    assert isinstance(awkward1.to_arrow(listarray), (pyarrow.LargeListArray))
+    assert isinstance(awkward1.to_arrow(listarray), (pyarrow.ListArray))
     assert awkward1.to_arrow(listarray).to_pylist() == [[[[0.0, 1.1, 2.2], []], [
         [3.3, 4.4], [5.5]]], [[[3.3, 4.4], [5.5]], [[6.6, 7.7, 8.8, 9.9], []]]]
 
-    assert isinstance(awkward1.to_arrow(regulararray),
-                      (pyarrow.LargeListArray))
+    assert isinstance(awkward1.to_arrow(regulararray), (pyarrow.ListArray))
     assert awkward1.to_arrow(regulararray).to_pylist() == [[[0.0, 1.1, 2.2], []], [
         [3.3, 4.4], [5.5]], [[6.6, 7.7, 8.8, 9.9], []]]