Repository: feldera/feldera

Issue #435: Array types break csv ingress/egress
Created: 2023-08-07T21:47:43Z
e.g.,
CREATE TABLE inputs
(
    int_arr1d BIGINT ARRAY
);
CREATE VIEW outputs AS SELECT * FROM inputs;
will accept something like:
curl 'http://localhost:8080/v0/pipelines/0189b2c6-0135-7f9e-95fd-93f99592da61/ingress/INPUTS?format=csv' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0' -H 'Accept: application/json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: http://localhost:3000/' -H 'content-type: text/csv' -H 'Origin: http://localhost:3000' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw '0,4'

for input (note that those are supposedly two elements in the array:
which confuses then the retrieval:
curl 'http://localhost:8080/v0/pipelines/0189b2c6-0135-7f9e-95fd-93f99592da61/egress/INPUTS?format=csv&query=quantiles&mode=snapshot' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: http://localhost:3000/' -H 'Content-Type: application/json' -H 'Origin: http://localhost:3000' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw '{"before":25,"after":25}'

will never send any data because on the backend it fails to serialize rows:
2023-08-07 21:28:42 ERROR [pipeline-0189b2c6-0135-7f9e-95fd-93f99592da61] encoder error on output endpoint 'api-snapshot-INPUTS-quantiles-4e0a9418-71f8-4753-a188-9e16a026bd95': CSV error: found record with 15 fields, but the previous record has 14 fields

So the problem seems to be that CSV parsing gets confused with arrays. We discussed as a potential solution, we need to define a standard for the CSV format to be e.g., strings of "[1,2,3]" for 1d arrays, or "[[1],[2,3]]" for 2d etc. (and any other future complex types for that matter). Then this can be recognized and parsed properly.

Duplicate issue #1050: Unable to insert TIME ARRAY values in a table
Created: 2023-11-21T16:30:15Z
Steps to reproduce:

Create and start a pipeline with the following SQL program:

create table table_1 (
    id integer not null primary key,
    field_a TIME ARRAY not null
);

Try to insert a row in any format:


curl 'http://localhost:8080/v0/pipelines/{pipeline_id}/ingress/TABLE_1?force=false&format=csv' -X POST -H 'Content-Type: application/json'  -d '1302962400,"[""03:47:09""]"'
curl 'http://localhost:8080/v0/pipelines/{pipeline_id}/ingress/TABLE_1?force=false&format=csv' -X POST -H 'Content-Type: application/json'  -d '1302962400,"[03:47:09]"'
curl 'http://localhost:8080/v0/pipelines/{pipeline_id}/ingress/TABLE_1?force=false&format=csv' -X POST -H 'Content-Type: application/json'  -d '1302962400,"["03:47:09"]"'
curl 'http://localhost:8080/v0/pipelines/{pipeline_id}/ingress/TABLE_1?force=false&format=csv' -X POST -H 'Content-Type: application/json'  -d '1302962400,"[\"03:47:09\"]"'
curl 'http://localhost:8080/v0/pipelines/{pipeline_id}/ingress/TABLE_1?force=false&format=csv' -X POST -H 'Content-Type: application/json'  -d '1302962400,["03:47:09"]'


Observe error response: failed to deserialize CSV record: error parsing field 'FIELD_A': invalid time string ...

Please clarify if none of the above contain correct formatting for a CSV with an array of TIME values.

Closing comment (2023-11-21T16:35:44Z): This is a duplicate of #435