Repository: ToucanToco/fastexcel (language: python)

Issue:
Sheet `to_arrow()` produces `null not null` columns
With the example file [null-column.xlsx](https://github.com/user-attachments/files/16692741/null-column.xlsx):

```python
>>> excel_reader = fastexcel.read_excel('null-column.xlsx')
>>> sheet = excel_reader.load_sheet(0)
>>> sheet.to_arrow()
pyarrow.RecordBatch
record_id: double not null
nullonly: null not null
----
record_id: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
nullonly: 20 nulls
```

Apparently PyArrow's [enforcement of "nullable" is not strong](https://stackoverflow.com/questions/78848723/non-nullable-field-in-schema-does-not-prevent-null-values-in-corresponding-table), so this doesn't create serious problems for me until I try to pickle and unpickle the RecordBatch. Then I get the exception:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/types.pxi", line 3532, in pyarrow.lib.field
ValueError: A null type field may not be non-nullable
```

I tried tracking down the cause of this weird `null not null` column schema, but it does appear that fastexcel [attempts to define all columns as "nullable"](https://github.com/ToucanToco/fastexcel/blob/c3cd0d9f5ca5f27e481b0775cd7826fb9fe3fee1/src/types/python/excelsheet/column_info.rs#L132), so I'm not sure where this is originating from. However, if I build the RecordBatch by way of polars, I don't get this strange schema:

```python
>>> sheet.to_polars().to_arrow().to_batches()[0]
pyarrow.RecordBatch
record_id: double
nullonly: null
----
record_id: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
nullonly: 20 nulls
```

... so I suppose it's at least plausible that this could be fixed within fastexcel.


Discussion / hints:

Fix patch:
diff --git a/.gitignore b/.gitignore
index 3a99b8d..5659737 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,6 +5,7 @@ __pycache__
 *.pyc
 *.so
 *.dat
+.DS_Store
 
 .python-version
 .venv
diff --git a/src/types/python/excelsheet/mod.rs b/src/types/python/excelsheet/mod.rs
index 9b98185..761d5c2 100644
--- a/src/types/python/excelsheet/mod.rs
+++ b/src/types/python/excelsheet/mod.rs
@@ -6,7 +6,7 @@ use sheet_data::ExcelSheetData;
 use std::{cmp, collections::HashSet, fmt::Debug, str::FromStr, sync::Arc};
 
 use arrow::{
-    array::NullArray,
+    array::{Array, NullArray},
     datatypes::{Field, Schema},
     pyarrow::ToPyArrow,
     record_batch::RecordBatch,
@@ -521,9 +521,14 @@ impl TryFrom<&ExcelSheet> for RecordBatch {
             let schema: Schema = sheet.into();
             Ok(RecordBatch::new_empty(Arc::new(schema)))
         } else {
-            RecordBatch::try_from_iter(iter)
-                .map_err(|err| FastExcelErrorKind::ArrowError(err.to_string()).into())
-                .with_context(|| format!("could not convert sheet {} to RecordBatch", sheet.name))
+            // We use `try_from_iter_with_nullable` because `try_from_iter` relies on `array.null_count() > 0;`
+            // to determine if the array is nullable. This is not the case for `NullArray` which has no nulls.
+            RecordBatch::try_from_iter_with_nullable(iter.map(|(field_name, array)| {
+                let nullable = array.is_nullable();
+                (field_name, array, nullable)
+            }))
+            .map_err(|err| FastExcelErrorKind::ArrowError(err.to_string()).into())
+            .with_context(|| format!("could not convert sheet {} to RecordBatch", sheet.name))
         }
     }
 }


Test patch:
diff --git a/python/tests/fixtures/null-column.xlsx b/python/tests/fixtures/null-column.xlsx
new file mode 100644
index 0000000..44dafc2
Binary files /dev/null and b/python/tests/fixtures/null-column.xlsx differ
diff --git a/python/tests/test_fastexcel.py b/python/tests/test_fastexcel.py
index 82d5db0..555ae1c 100644
--- a/python/tests/test_fastexcel.py
+++ b/python/tests/test_fastexcel.py
@@ -531,3 +531,8 @@ def test_null_values_in_cells() -> None:
     pd_expected = pd.DataFrame(expected)
     pd_expected["Date"] = pd_expected["Date"].dt.as_unit("ms")
     pd_assert_frame_equal(sheet.to_pandas(), pd_expected)
+
+
+def test_null_column_is_nullable() -> None:
+    sheet = fastexcel.read_excel(path_for_fixture("null-column.xlsx")).load_sheet(0)
+    assert sheet.to_arrow().schema.field("nullonly").nullable is True