Package: skeem
AboutYou can use Skeem to infer SQL DDL statements from tabular data.Skeem is, amongst others, based on the excellentddlgenerator,frictionless,fsspec,pandas,SQLAlchemy, andxarraypackages, and can be used both
as a standalone program, and as a library.Supported input data:Apache ParquetCSVGoogle SheetsGRIBInfluxDB line protocolJSONNetCDFNDJSON(formerly LDJSON) aka.JSON Lines, see alsoJSON streamingOffice Open XML Workbook(Microsoft Excel)OpenDocument Spreadsheet(LibreOffice)Supported input sources:Amazon S3File systemGitHubGoogle Cloud StorageHTTPPlease note that Skeem is alpha-quality software, and a work in progress.
Contributions of all kinds are very welcome, in order to make it more solid.
Breaking changes should be expected until a 1.0 release, so version pinning
is recommended, especially when you use it as a library.Synopsisskeeminfer-ddl--dialect=postgresqldata.ndjsonCREATETABLE"data"("id"SERIALNOTNULL,"name"TEXTNOTNULL,"date"TIMESTAMPWITHOUTTIMEZONE,"fruits"TEXTNOTNULL,"price"DECIMAL(2,2)NOTNULL,PRIMARYKEY("id"));QuickstartIf you are in a hurry, and want to run Skeem without any installation, just use
the OCI image on Podman or Docker.dockerrun--rmghcr.io/daq-tools/skeem-standard\skeeminfer-ddl--dialect=postgresql\https://github.com/daq-tools/skeem/raw/main/tests/testdata/basic.ndjsonSetupInstall Skeem from PyPI.pipinstallskeemInstall Skeem with support for additional data formats like NetCDF.pipinstall'skeem[scientific]'UsageThis section outlines some example invocations of Skeem, both on the command
line, and per library use. Other than the resources available from the web,
testing data can be acquired from the repository’stestdatafolder.Command line useHelpskeeminfoskeem--helpskeeminfer-ddl--helpRead from files# NDJSON, Parquet, and InfluxDB line protocol (ILP) formats.skeeminfer-ddl--dialect=postgresqldata.ndjsonskeeminfer-ddl--dialect=postgresqldata.parquetskeeminfer-ddl--dialect=postgresqldata.lp# CSV, JSON, ODS, and XLSX formats.skeeminfer-ddl--dialect=postgresqldata.csvskeeminfer-ddl--dialect=postgresqldata.jsonskeeminfer-ddl--dialect=postgresqldata.odsskeeminfer-ddl--dialect=postgresqldata.xlsxskeeminfer-ddl--dialect=postgresqldata.xlsx--address="Sheet2"Read from URLs# CSV, NDJSON, XLSXskeeminfer-ddl--dialect=postgresqlhttps://github.com/daq-tools/skeem/raw/main/tests/testdata/basic.csvskeeminfer-ddl--dialect=postgresqlhttps://github.com/daq-tools/skeem/raw/main/tests/testdata/basic.ndjsonskeeminfer-ddl--dialect=postgresqlhttps://github.com/daq-tools/skeem/raw/main/tests/testdata/basic.xlsx--address="Sheet2"# Google Sheets: Address first sheet, and specific sheet of workbook.skeeminfer-ddl--dialect=postgresql--table-name=foohttps://docs.google.com/spreadsheets/d/1ExyrawjlyksbC6DOM6nLolJDbU8qiRrrhxSuxf5ScB0/viewskeeminfer-ddl--dialect=postgresql--table-name=foohttps://docs.google.com/spreadsheets/d/1ExyrawjlyksbC6DOM6nLolJDbU8qiRrrhxSuxf5ScB0/view#gid=883324548# InfluxDB line protocol (ILP)skeeminfer-ddl--dialect=postgresqlhttps://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp# Compressed files in gzip formatskeem--verboseinfer-ddl--dialect=crate--content-type=ndjsonhttps://s3.amazonaws.com/crate.sampledata/nyc.yellowcab/yc.2019.07.gz# CSV on S3skeem--verboseinfer-ddl--dialect=postgresqls3://noaa-ghcn-pds/csv/by_year/2022.csv# CSV on Google Cloud Storageskeem--verboseinfer-ddl--dialect=postgresqlgs://tinybird-assets/datasets/nations.csvskeem--verboseinfer-ddl--dialect=postgresqlgs://tinybird-assets/datasets/medals1.csv# CSV on GitHubskeem--verboseinfer-ddl--dialect=postgresqlgithub://daq-tools:skeem@/tests/testdata/basic.csv# GRIB2, NetCDFskeeminfer-ddl--dialect=postgresqlhttps://github.com/earthobservations/testdata/raw/main/opendata.dwd.de/weather/nwp/icon/grib/18/t/icon-global_regular-lat-lon_air-temperature_level-90.grib2skeeminfer-ddl--dialect=postgresqlhttps://www.unidata.ucar.edu/software/netcdf/examples/sresa1b_ncar_ccsm3-example.ncskeeminfer-ddl--dialect=postgresqlhttps://www.unidata.ucar.edu/software/netcdf/examples/WMI_Lear.ncOCIOCI images are available on the GitHub Container Registry (GHCR). In order to
run them on Podman or Docker, invoke:dockerrun--rmghcr.io/daq-tools/skeem-standard\skeeminfer-ddl--dialect=postgresql\https://github.com/daq-tools/skeem/raw/main/tests/testdata/basic.csvIf you want to work with files on your filesystem, you will need to mount the
working directory into the container when running it, like:dockerrun--rm--volume=$(pwd):/dataghcr.io/daq-tools/skeem-standard\skeeminfer-ddl--dialect=postgresql/data/basic.csvIn order to always run the latest development version, and to use a shortcut
for that, this section outlining how to use an alias forskeem, and a
variable for storing the URL, may be useful to save a few keystrokes.aliasskeem="docker run --rm --pull=always ghcr.io/daq-tools/skeem-standard:nightly skeem"URL=https://github.com/daq-tools/skeem/raw/main/tests/testdata/basic.ndjsonskeeminfer-ddl--dialect=postgresql$URLMoreUse a different backend (default:ddlgen):skeem infer-ddl --dialect=postgresql --backend=frictionless data.ndjsonReading data from stdin needs to obtain both the table name and content type separately:skeem infer-ddl --dialect=crate --table-name=foo --content-type=ndjson - < data.ndjson
skeem infer-ddl --dialect=crate --table-name=foo --content-type=json - < data.json
skeem infer-ddl --dialect=crate --table-name=foo --content-type=csv - < data.csvReading data from stdin also works like this, if you prefer to use pipes:cat data.ndjson | skeem infer-ddl --dialect=crate --table-name=foo --content-type=ndjson -
cat data.json | skeem infer-ddl --dialect=crate --table-name=foo --content-type=json -
cat data.csv | skeem infer-ddl --dialect=crate --table-name=foo --content-type=csv -Library useimportiofromskeem.coreimportSchemaGeneratorfromskeem.modelimportResource,SqlTargetINDATA=io.StringIO("""
    {"id":1,"name":"foo","date":"2014-10-31 09:22:56","fruits":"apple,banana","price":0.42}
    {"id":2,"name":"bar","date":null,"fruits":"pear","price":0.84}
    """)sg=SchemaGenerator(resource=Resource(data=INDATA,content_type="ndjson"),target=SqlTarget(dialect="crate",table_name="testdrive"),)print(sg.to_sql_ddl().pretty)CREATETABLE"testdrive"("id"INTNOTNULL,"name"STRINGNOTNULL,"date"TIMESTAMP,"fruits"STRINGNOTNULL,"price"DOUBLENOTNULL,PRIMARYKEY("id"));DevelopmentFor installing the project from source, please follow thedevelopmentdocumentation.Project informationCreditsCatherine Devlinforddlgeneratoranddata_dispenser.Mike BayerforSQLAlchemy.Paul WalshandEvgeny Karevforfrictionless.Wes McKinneyforpandas.All other countless contributors and authors of excellent Python
packages, Python itself, and turtles all the way down.Prior artWe are maintaining alist of other projectswith the same or similar goals
like Skeem.EtymologyThe program was about to be calledEskema, but it turned out that there is
already anotherEskemaout there. So, it has been renamed toSkeem, which
is Estonian, and means “schema”, “outline”, or “(to) plan”.