Repository: vaexio/vaex

Issue:
[BUG-REPORT] Cannot export_arrow when `large_string` in schema
Thank you for reaching out and helping us improve Vaex!

Before you submit a new Issue, please read through the [documentation](https://docs.vaex.io/en/latest/). Also, make sure you search through the Open and Closed Issues - your problem may already be discussed or addressed.

**Description**
Please provide a clear and concise description of the problem. This should contain all the steps needed to reproduce the problem. A minimal code example that exposes the problem is very appreciated.

**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: 4.8.0
 - Vaex was installed via: pip / conda-forge / from source  pip
 - OS: Mac

**Additional information**
```
import vaex

df = vaex.example()
df["text"] = vaex.vconstant("some text", len(df))
df["text"] = "["+df.text+"]"
print(df.dtypes)
df.export_arrow("text.arrow")
```
```
----> 1 df.export_arrow("text.arrow")

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/dataframe.py in export_arrow(self, to, progress, chunk_size, parallel, reduce_large, fs_options, fs, as_stream)
   6634                 if as_stream:
   6635                     with pa.RecordBatchStreamWriter(sink, schema) as writer:
-> 6636                         write(writer)
   6637                 else:
   6638                     with pa.RecordBatchFileWriter(sink, schema) as writer:

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/dataframe.py in write(writer)
   6622                 with vaex.progress.tree(progress, title="export(arrow)") as progressbar:
   6623                     for i1, i2, table in self.to_arrow_table(chunk_size=chunk_size, parallel=parallel, reduce_large=reduce_large):
-> 6624                         writer.write_table(table)
   6625                         progressbar(i2/N)
   6626                     progressbar(1.)

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/pyarrow/ipc.pxi in pyarrow.lib._CRecordBatchWriter.write_table()

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Tried to write record batch with different schema
```

```
df["text"] = df.text.astype("str")
df.export_arrow('test.arrow')  # works
```

The [arrow docs](https://arrow.apache.org/docs/python/generated/pyarrow.large_string.html) say 
> This data type may not be supported by all Arrow implementations. Unless you need to represent data larger than 2GB, you should prefer string(). 

So it's probably safe to just cast to string before an export


Discussion:

Resolving patch:
diff --git a/packages/vaex-core/vaex/dataframe.py b/packages/vaex-core/vaex/dataframe.py
--- a/packages/vaex-core/vaex/dataframe.py
+++ b/packages/vaex-core/vaex/dataframe.py
@@ -6732,7 +6732,7 @@ def write(writer):
                 writer.write_table(table)
 
         if vaex.file.is_path_like(to) or vaex.file.is_file_object(to):
-            schema = self.schema_arrow()
+            schema = self.schema_arrow(reduce_large=reduce_large)
             with vaex.file.open(path=to, mode='wb', fs_options=fs_options, fs=fs) as sink:
                 if as_stream:
                     with pa.RecordBatchStreamWriter(sink, schema) as writer:


Test patch:
diff --git a/tests/export_test.py b/tests/export_test.py
--- a/tests/export_test.py
+++ b/tests/export_test.py
@@ -2,6 +2,7 @@
 import os
 from pathlib import Path
 import tempfile
+import pyarrow as pa
 import pandas as pd
 import platform
 import hashlib
@@ -264,3 +265,9 @@ def test_export_json(tmpdir, df_filtered):
     # for column in df.get_column_names():
     for column in ['x', 'name']:
         assert df[column].tolist() == df2[column].tolist()
+
+
+def test_export_large_string(tmpdir):
+    s = pa.array(["Hi", "there"], type=pa.large_string())
+    df = vaex.from_arrays(s=s)
+    df.export_arrow(tmpdir / "test.arrow")