Repository: saulpw/visidata

Issue:
Parquet Large String columns are typed vlen by default
**Small description**

I have a parquet file which contains string columns:
[australian_public_holidays_2014_2020.zip](https://github.com/saulpw/visidata/files/12389440/australian_public_holidays_2014_2020.zip)


**Expected result**

In DuckDB, it looks like this:


```
D select * from 'australian_public_holidays_2014_2020.parquet';
┌────────────┬──────────────────────┬──────────────────────┬────────────────────────────────────────────────────┬──────────────┐
│    Date    │     Holiday_Name     │     Information      │                  More_Information                  │ Jurisdiction │
│    date    │       varchar        │       varchar        │                      varchar                       │   varchar    │
├────────────┼──────────────────────┼──────────────────────┼────────────────────────────────────────────────────┼──────────────┤
│ 2014-01-01 │ New Year's Day       │ New Year's Day is .  │                                                    │ act          │
│ 2014-01-27 │ Australia Day        │ Always celebrated .  │                                                    │ act          │
│ 2014-03-10 │ Canberra Day         │ Held on the second.  │ http://www.cmd.act.gov.au/communication/holidays   │ act          │
```

**Actual result with screenshot**

In Visidata with pyarrow installed, it looks like this:

```
Date         Holiday_Name   Information   More_Information   Jurisdiction   
2014-01-01             14            87                                 3 
2014-01-27             13            31                                 3 
2014-03-10             12           148                 48              3 
```

Note that the numbers are the _exact_ length of the missing strings!

**Steps to reproduce with sample data and a .vd**

```
vd australian_public_holidays_2014_2020.parquet
```

(Skipping the command-log as simply opening the file reproduces the issue)

**Additional context**

```
❯ vd --version
saul.pw/VisiData v2.11.1
```

Python 3.10.12



Discussion:
So, this is because the type used for columns of "Large String Type" is `vlen`, which shows the length of the string.  You can see this by the sharp sign `♯` in the header.  If you want to see the string itself, you can type the column deliberately as string with `~` (or use anytype with `z~`).  The basic (non-large) string type should be anytype and would be shown by default.

It might be reasonable to type columns of Large String Type as anytype by default too.  Let's debate!
Hmm... I'm curious why `vlen` was chosen as the default?
It's certainly surprising to see numbers not strings when viewing a string column!
I'd be interested in the expected use case for vlen columns?
I'm not aware of any other software which would use this as the default view.

As a single data point, I imagine 99.9% of the time I would want a string preview, not a summary of the length of the value.
Because there may be significant performance issues, depending on how the library/API deals with them.  For example, `len` may be a cheap operation on a Value wrapper, but actually fetching the whole string may be a lot more expensive.  In fact, the [pyarrow docs](https://arrow.apache.org/docs/python/generated/pyarrow.large_string.html) say "Unless you need to represent data larger than 2GB, you should prefer string()", so I would assume that these Large Strings are usually >2GB.  If your strings are smaller than that, you might consider changing the types of those columns in your schema to a non-Large string, if that's possible.

Note that I didn't do any rigorous analysis or performance testing on this, so it might work fine, even with gigabyte strings.  If you want to put together a sample dataset (with at least a couple of strings >2GB) and play with it yourself you can comment out the line here to get the behavior you want: https://github.com/saulpw/visidata/blob/develop/visidata/loaders/arrow.py#L47.  Let me know if you do and how well it works!
Thanks @saulpw, I appreciate the detail. It looks like a [Chesterton's Fence](https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_fence) type situation.

I admit I never imagined a >2GB string column!
I'll have a play and report back.

This file was created with a Synapse pipeline, so I'll check if there's an option to use a smaller string type.
@daviewales Would you like to keep this open? What do you think about us closing it, and re-opening when you report back?
Happy either way. It's looking like the parquet specification might not support values greater than 2GB?
(See linked polars issue above)
 I'm thinking I need to investigate why it's coming into Visidata as type large_string rather than string. It might take me awhile to explore what's happening here.

I'm wondering if this part of the Thrift spec is where the 2GB limit comes from:

https://github.com/apache/parquet-format/blob/405bd6458d25f57fe99c1f09a08667e871a98a5f/src/main/thrift/parquet.thrift#L365

`optional i32 type_length` implies the maximum type length is 2 GB (2³¹ bytes).
Although this item in the SchemaElement struct is `optional`, and the comment above it mentions only the `FIXED_LEN_BYTE_ARRAY` type, so perhaps fixed length string fields have a max width of 2 GB, but non-fixed width string fields have unlimited width?

But I haven't been able to find a tool which will write a string value larger than 2GB, so even if it's not a logical limitation of the parquet specification, it seems to be a practical limitation of the parquet libraries I've tried (polars, pyarrow, dask, pyspark).

Perhaps the fact that these fields are type `BYTE_ARRAY` rather than `FIXED_LEN_BYTE_ARRAY` is why Visidata is interpreting them as type large_string? See [spec](https://github.com/apache/parquet-format#types).

The output from `parquet-tools` doesn't mention anything about the large string type:

```
❯ parquet-tools inspect ~/Downloads/australian_public_holidays_2014_2020.parquet

############ file meta data ############
created_by: Arrow2 - Native Rust implementation of Arrow
num_columns: 5
num_rows: 682
num_row_groups: 1
format_version: 2.6
serialized_size: 1006


############ Columns ############
Date
Holiday_Name
Information
More_Information
Jurisdiction

############ Column(Date) ############
name: Date
path: Date
max_definition_level: 1
max_repetition_level: 0
physical_type: INT32
logical_type: Date
converted_type (legacy): DATE
compression: SNAPPY (space_saved: 57%)

############ Column(Holiday_Name) ############
name: Holiday_Name
path: Holiday_Name
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 86%)

############ Column(Information) ############
name: Information
path: Information
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 91%)

############ Column(More_Information) ############
name: More_Information
path: More_Information
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 91%)

############ Column(Jurisdiction) ############
name: Jurisdiction
path: Jurisdiction
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
compression: SNAPPY (space_saved: 90%)
```
OK. Further research.

The polars team suggest that the Parquet specification doesn't support values larger than 2 GB:
https://github.com/pola-rs/polars/issues/10774

This fits with my observation above, as I have been unable to convince any library to create a parquet file with values greater than 2 GB.

However, I created two identical test parquet files using pyarrow, but with string types `string` and `large_string` respectively.
These files are very slightly different, which indicates that Parquet files _do_ differentiate between the pyarrow string types.
See the code below:

## String type

``` python
import pyarrow as pa
import pyarrow.parquet as pq

arr = pa.array(["data"] * 2, type=pa.string())
data = [arr]*2
batch = pa.RecordBatch.from_arrays(data, ['field_1', 'field_2'])
table = pa.Table.from_batches([batch])

pq.write_table(table, 'pyarrow_string.parquet', compression=None)

df_string = pq.read_table('pyarrow_string.parquet')
print(df_string.schema)
# field_1: string
# field_2: string
```

## Large string type

``` python
import pyarrow as pa
import pyarrow.parquet as pq

arr = pa.array(["data"] * 2, type=pa.large_string())
data = [arr]*2
batch = pa.RecordBatch.from_arrays(data, ['field_1', 'field_2'])
table = pa.Table.from_batches([batch])

pq.write_table(table, 'pyarrow_large_string.parquet', compression=None)

df_large_string = pq.read_table('pyarrow_large_string.parquet')
print(df_large_string.schema)
# field_1: large_string
# field_2: large_string
```

I'm not sure where this leaves us...
pyarrow can create tables in memory with values greater than 2 GB, but refuses to save them to parquet.
I'm still not sure if this limitation is a logical limitation of the parquet specification, or a practical limitation of the existing parquet library implementations.

If there is no way to create a parquet file with string values greater than 2 GB, does this mean we should treat `string` and `large_string` the same for parquet files, and display the strings rather than the lengths?
Further: I created a parquet file which is close to the 2 GB limit, and tested with both displayed string values and displayed string lengths:

``` python
import pyarrow as pa
import pyarrow.parquet as pq

arr = pa.array(["data"*(2**28)] * 2, type=pa.large_string())
data = [arr]*2
batch = pa.RecordBatch.from_arrays(data, ['field_1', 'field_2'])
table = pa.Table.from_batches([batch])

pq.write_table(table, 'pyarrow_2GB_large_string.parquet', compression=None)
```

In Visidata, moving the cursor between rows or columns was _very_ slow, regardless of whether string values or string lengths were displayed.
Okay, based on your experiments with large string columns, that there appears to be no performance difference between displaying the text vs length of columns with large string values, I think we can go with your original suggestion of always displaying the text.  Feel free to submit a PR if you want the credit, otherwise we can update it this weekend or so.  Maybe add a comment to #1068 to track the performance issue.

Thanks for your legwork on this, @daviewales!  
I thought I should time how long things take.

It seems to take around 18-20 seconds to load the file, regardless of vlen or string view. (vlen might have been a second or two faster, but it's hard to get a precise timing)

I timed how long it takes to switch between rows or columns.
It takes 8 seconds to switch row or column with the vlen view, and 14 seconds to switch with the string view.
So vlen does have a slight advantage over the string view in this extreme case.

For comparison, duckdb takes over 2 minutes to display the data.

Given the 6 second advantage to vlen in this case, are you still happy to switch to the string view?

Resolving patch:
diff --git a/visidata/loaders/arrow.py b/visidata/loaders/arrow.py
--- a/visidata/loaders/arrow.py
+++ b/visidata/loaders/arrow.py
@@ -44,7 +44,7 @@ def arrow_to_vdtype(t):
         pa.lib.Type_LARGE_BINARY: vlen,
 #            pa.lib.Type_FIXED_SIZE_BINARY: bytes,
 #            pa.lib.Type_STRING: str,
-        pa.lib.Type_LARGE_STRING: vlen,
+#            pa.lib.Type_LARGE_STRING: vlen,  #2003
 #            pa.lib.Type_LIST: list,
 #            pa.lib.Type_LARGE_LIST: list,
 #            pa.lib.Type_FIXED_SIZE_LIST: list,


Test patch: