Bug: Python: connect client lost column data with pyarrow.Table.to_pylist
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/23 05:14
Affects Version/s: 3.4.0

Python: connect client should not use pyarrow.Table.to_pylist to transform fetched data.
For example:
the data in pyarrow.Table show below.

{code:java}
pyarrow.Table
key: string
order: int64
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): string
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): string
nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): string
----
key: [["a","a","a","a","a","b","b"]]
order: [[0,1,2,3,4,1,2]]
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): [[null,"x","x","x","x",null,null]]
nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): [[null,"x","x","x","x",null,null]]
nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW): [[null,null,"y","y","y",null,null]]
{code}

The table have five columns show above.
But the data after call pyarrow.Table.to_pylist() show below.

{code:java}
[{
	'key': 'a',
	'order': 0,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None,
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None
}, {
	'key': 'a',
	'order': 1,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None
}, {
	'key': 'a',
	'order': 2,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'y'
}, {
	'key': 'a',
	'order': 3,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'y'
}, {
	'key': 'a',
	'order': 4,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'x',
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': 'y'
}, {
	'key': 'b',
	'order': 1,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None,
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None
}, {
	'key': 'b',
	'order': 2,
	'nth_value(value, 2) OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None,
	'nth_value(value, 2) ignore nulls OVER (PARTITION BY key ORDER BY order ASC NULLS FIRST RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)': None
}]
{code}

There are only four columns left.