onyxfish/csvkit — Issue #136: csvsql choking on create table statements
Author: thejefflarson
Created: 2012-01-07 01:35:45+00:00
State: open

I'm trying to create a schema for this file:

http://thejefflarson.s3.amazonaws.com/ACRONYMS_CD.CSV

using `cat ACRONYMS_CD.CSV | csvsql -i postgresql --table test` but when I do,
the schema fails to detect the header row and prints out:

CREATE TABLE test (
"ACRONYM,STANDS_FOR,EFFECT_DT,A_DESC" VARCHAR(41) NOT NULL
);

which should be:

CREATE TABLE test (
"ACRONYM" VARCHAR(31) NOT NULL,
"STANDS_FOR" VARCHAR(4),
"EFFECT_DT" DATE NOT NULL,
"A_DESC" VARCHAR(20)
);

when I limit the rows to 777 it works:

jlarson@imac-08:Data $ head -778 ACRONYMS_CD.CSV | csvsql -i postgresql --table test
CREATE TABLE test (
"ACRONYM,STANDS_FOR,EFFECT_DT,A_DESC" VARCHAR(41) NOT NULL
);
jlarson@imac-08:Data $ head -778 ACRONYMS_CD.CSV | wc
778 2505 31076
jlarson@imac-08:Data $ head -777 ACRONYMS_CD.CSV | csvsql -i postgresql --table test
CREATE TABLE test (
"ACRONYM" VARCHAR(31) NOT NULL,
"STANDS_FOR" VARCHAR(4),
"EFFECT_DT" DATE NOT NULL,
"A_DESC" VARCHAR(20)
);
jlarson@imac-08:Data $ head -777 ACRONYMS_CD.CSV | wc
777 2500 31029

My python version is:

jlarson@imac-08:Data $ python --version
Python 2.7.2

I've checked the file for line ending problems and ran it through `iconv` with
`'ascii//translit//ignore'` to no avail. Any idea why this is happening?