Package: bigquery-schema-generator
BigQuery Schema GeneratorThis script generates the BigQuery schema from the newline-delimited data
records on the STDIN. The records can be in JSON format or CSV format. The
BigQuery data importer (bq load) uses only thefirst 500 recordswhen the schema auto-detection feature is enabled. In contrast, this script uses
all data records to generate the schema.Usage:$ generate-schema < file.data.json > file.schema.json
$ generate-schema --input_format csv < file.data.csv > file.schema.jsonVersion: 1.6.1 (2024-01-12)Changelog:CHANGELOG.mdTable of ContentsBackgroundInstallationUbuntu LinuxMacOSMacOS 12 (Monterey)MacOS 11 (Big Sur)MacOS 10.14 (Mojave)UsageCommand LineSchema OutputCommand Line Flag OptionsHelp (--help)Input Format (--input_format)Keep Nulls (--keep_nulls)Quoted Values Are Strings(--quoted_values_are_strings)Infer Mode (--infer_mode)Debugging Interval (--debugging_interval)Debugging Map (--debugging_map)Sanitize Names (--sanitize_names)Ignore Invalid Lines (--ignore_invalid_lines)Existing Schema Path (--existing_schema_path)Preserve Input Sort Order
(--preserve_input_sort_order)Using as a LibrarySchemaGenerator.run()SchemaGenerator.deduce_schema()from
FileSchemaGenerator.deduce_schema()from
DictSchemaGenerator.deduce_schema()from
DictReaderSchema TypesSupported TypesType InferenceExamplesBenchmarksSystem RequirementsLicenseFeedback and SupportAuthorsBackgroundData can be imported intoBigQueryusing
thebqcommand line
tool. It accepts a number of data formats including CSV or newline-delimited
JSON. The data can be loaded into an existing table or a new table can be
created during the loading process. The structure of the table is defined by
itsschema. The table's
schema can be defined manually or the schema can beauto-detected.When the auto-detect feature is used, the BigQuery data importer examines only
thefirst 500 recordsof the input data. In many cases, this is sufficient
because the data records were dumped from another database and the exact schema
of the source table was known. However, for data extracted from a service
(e.g. using a REST API) the record fields could have been organically added
at later dates. In this case, the first 500 records do not contain fields which
are present in later records. Thebq loadauto-detection fails and the data
fails to load.Thebq loadtool does not support the ability to process the entire dataset
to determine a more accurate schema. This script fills in that gap. It
processes the entire dataset given in the STDIN and outputs the BigQuery schema
in JSON format on the STDOUT. This schema file can be fed back into thebq
loadtool to create a table that is more compatible with the data fields in
the input dataset.InstallationPrerequisite: You need have Python 3.6 or higher.Install fromPyPIrepository usingpip3. There
are too many ways to install packages in Python. The following are in order
highest to lowest recommendation:If you are using a virtual environment (such asvenv), then use:$ pip3 install bigquery_schema_generatorIf you aren't using a virtual environment you can install into
your local Python directory:$ pip3 install --user bigquery_schema_generatorIf you want to install the package for your entire system globally, use$ sudo -H pip3 install bigquery_schema_generatorbut realize that you will be running code from PyPI asrootso this has
security implications.Sometimes, your Python environment gets into a complete mess and thepip3command won't work. Try typingpython3 -m pipinstead.A successful install should print out something like the following (the version
number may be different):Collecting bigquery-schema-generator
Installing collected packages: bigquery-schema-generator
Successfully installed bigquery-schema-generator-1.1The shell scriptgenerate-schemawill be installed somewhere in your system,
depending on how your Python environment is configured. See below for
some notes for Ubuntu Linux and MacOS.Ubuntu Linux (18.04, 20.04, 22.04)After runningpip3 install bigquery_schema_generator, thegenerate-schemascript may be installed in one the following locations:/usr/bin/generate-schema/usr/local/bin/generate-schema$HOME/.local/bin/generate-schema$HOME/.virtualenvs/{your_virtual_env}/bin/generate-schemaMacOSI don't have any Macs which are able to run the latest macOS, and I don't use
them much for software development these days, but here are some notes on older
versions of macOS in case they help.MacOS 12 (Monterey)Python 2 or 3 is not installed by default on Monterey. If you try to runpython3on the command line, a dialog box asks you to install theXcodedevelopment package. It
apparently takes over an hour at 10 MB/s.You can instead install Python 3 usingHomebrew, by installingbrew, and
typing$ brew install python. Currently, it downloads Python 3.10 in about 1-2
minutes and installs thepython3andpip3binaries into/usr/local/bin/python3and/usr/local/bin/pip3. Usingbrewseems to be
easiest option, so let's assume that Python 3 was installed through that.If you run:$ pip3 install bigquery_schema_generatorthe package will be installed at/usr/local/lib/python3.10/site-packages/, and
thegenerate-schemascript will be installed at/usr/local/bin/generate-schema.If you use the--userflag:$ pip3 install --user bigquery_schema_generatorthe package will be installed at$HOME/Library/Python/3.10/lib/python/site-packages/, and thegenerate-schemascript will be installed at$HOME/Library/Python/3.10/bin/generate-schema.You may need to add the$HOME/Library/Python/3.10/bindirectory to your$PATHvariable in your$HOME/.bashrcfile.MacOS 11 (Big Sur)Python 2.7.16 is installed by default on Big Sur as/usr/bin/python. If you
try to runpython3on the command line, a dialog box asks you to install
theXcodedevelopment package will
be installed, which I think installs Python 3.8 as/usr/bin/python3(I can't
remember, it was installed a long time ago.)You can instead install Python 3 usingHomebrew, by installingbrew, and
typing$ brew install python. Currently, it downloads Python 3.10 in about 1-2
minutes and installs thepython3andpip3binaries into/usr/local/bin/python3and/usr/local/bin/pip3. Usingbrewseems to be
easiest option, so let's assume that Python 3 was installed through that.If you run:$ pip3 install bigquery_schema_generatorthe package will be installed at/usr/local/lib/python3.10/site-packages/, and
thegenerate-schemascript will be installed at/usr/local/bin/generate-schema.If you use the--userflag:$ pip3 install --user bigquery_schema_generatorthe package will be installed at$HOME/Library/Python/3.10/lib/python/site-packages/, and thegenerate-schemascript will be installed at$HOME/Library/Python/3.10/bin/generate-schema.You may need to add the$HOME/Library/Python/3.10/bindirectory to your$PATHvariable in your$HOME/.bashrcfile.MacOS 10.14 (Mojave)This MacOS version comes with Python 2.7 only. To install Python 3, you can
install using:1)) Downloading themacos installer directly from
Python.org.The python3 binary will be located at/usr/local/bin/python3, and the/usr/local/bin/pip3is a symlink to/Library/Frameworks/Python.framework/Versions/3.6/bin/pip3.So running$ pip3 install --user bigquery_schema_generatorwill installgenerate-schemaat/Library/Frameworks/Python.framework/Versions/3.6/bin/generate-schema.The Python installer updates$HOME/.bash_profileto add/Library/Frameworks/Python.framework/Versions/3.6/binto the$PATHenvironment variable. So you should be able to run thegenerate-schemacommand without typing in the full path.2)) UsingHomebrew.In this environment, thegenerate-schemascript will probably be installed in/usr/local/binbut I'm not completely certain.UsageCommand LineThegenerate_schema.pyscript accepts a newline-delimited JSON or
CSV data file on the STDIN. JSON input format has been tested extensively.
CSV input format was added more recently (in v0.4) using the--input_format csvflag. The support is not as robust as JSON file. For example, CSV format
supports only the comma-separator, and does not support the pipe (|) or tab
(\t) character.Side Note: Theinput_formatparameter now supports (v1.6.0) thecsvdictreaderoption which allows using thecsv.DictReaderclass that can be
customized to handle different delimiters such as tabs. But this requires
creating a custom Python script usingbigquery_schema_generatoras a library.
SeeSchemaGenerator.deduce_schema() from
csv.DictReadersection below. It
is probably possible to enable this functionality through the command line
script, but it was not obvious how to expose the various options ofcsv.DictReaderthrough the command line flags. I didn't spend any time on this
problem because this is not a feature that I use personally.)Unlikebq load, thegenerate_schema.pyscript reads every record in the
input data file to deduce the table's schema. It prints the JSON formatted
schema file on the STDOUT.There are at least 3 ways to run this script:1) Shell scriptIf you installed usingpip3, then it should have installed a small helper
script namedgenerate-schemain your local./bindirectory of your current
environment (depending on whether you are using a virtual environment).$ generate-schema < file.data.json > file.schema.json2) Python moduleYou can invoke the module directly using:$ python3 -m bigquery_schema_generator.generate_schema < file.data.json > file.schema.jsonThis is essentially what thegenerate-schemacommand does.3) Python scriptIf you retrieved this code from itsGitHub repository,
then you can invoke the Python script directly:$ ./generate_schema.py < file.data.json > file.schema.jsonUsing the Schema OutputThe resulting schema file can be given to thebq loadcommand using the--schemaflag:$ bq load --source_format NEWLINE_DELIMITED_JSON \
    --ignore_unknown_values \
    --schema file.schema.json \
    mydataset.mytable \
    file.data.jsonwheremydataset.mytableis the target table in BigQuery.For debugging purposes, here is the equivalentbq loadcommand using schema
autodetection:$ bq load --source_format NEWLINE_DELIMITED_JSON \
    --autodetect \
    mydataset.mytable \
    file.data.jsonIf the input file is in CSV format, the first line will be the header line which
enumerates the names of the columns. But this header line must be skipped when
importing the file into the BigQuery table. We accomplish this using--skip_leading_rowsflag:$ bq load --source_format CSV \
    --schema file.schema.json \
    --skip_leading_rows 1 \
    mydataset.mytable \
    file.data.csvHere is the equivalentbq loadcommand for CSV files using autodetection:$ bq load --source_format CSV \
    --autodetect \
    mydataset.mytable \
    file.data.csvA useful flag forbq load, particularly for JSON files,  is--ignore_unknown_values, which causesbq loadto ignore fields in the input
data which are not defined in the schema. Whengenerate_schema.pydetects an
inconsistency in the definition of a particular field in the input data, it
removes the field from the schema definition. Without the--ignore_unknown_values, thebq loadfails when the inconsistent data record
is read.Another useful flag during development and debugging is--replacewhich
replaces any existing BigQuery table.After the BigQuery table is loaded, the schema can be retrieved using:$ bq show --schema mydataset.mytable | python3 -m json.tool(Thepython -m json.toolcommand will pretty-print the JSON formatted schema
file. An alternative is thejq command.)
The resulting schema file should be identical tofile.schema.json.Command Line Flag OptionsThegenerate_schema.pyscript supports a handful of command line flags
as shown by the--helpflag below.Help (--help)Print the built-in help strings:$generate-schema--help
usage:generate-schema[-h][--input_formatINPUT_FORMAT][--keep_nulls][--quoted_values_are_strings][--infer_mode][--debugging_intervalDEBUGGING_INTERVAL][--debugging_map][--sanitize_names][--ignore_invalid_lines][--existing_schema_pathEXISTING_SCHEMA_PATH][--preserve_input_sort_order]GenerateBigQueryschemafromJSONorCSVfile.

optionalarguments:-h,--helpshowthishelpmessageandexit--input_formatINPUT_FORMATSpecifyanalternativeinputformat('csv','json','dict')--keep_nullsPrinttheschemafornullvalues,emptyarraysoremptyrecords--quoted_values_are_stringsQuotedvaluesshouldbeinterpretedasstrings--infer_modeDetermineifmodecanbe'NULLABLE'or'REQUIRED'--debugging_intervalDEBUGGING_INTERVALNumberoflinesbetweenheartbeatdebuggingmessages--debugging_mapPrintthemetadataschema_mapinsteadoftheschema--sanitize_namesForcesschemanametocomplywithBigQuerynamingstandard--ignore_invalid_linesIgnorelinesthatcannotbeparsedinsteadofstopping--existing_schema_pathEXISTING_SCHEMA_PATHFilethatcontainstheexistingBigQueryschemaforatable.Thiscanbefetchedwith:`bqshow--schema<project_id>:<dataset>:<table_name>--preserve_input_sort_orderPreservetheoriginalorderingofcolumnsfrominputinsteadofsortingalphabetically.Thisonlyimpacts`input_format`ofjsonordictInput Format (--input_format)Specifies the format of the input file as a string. It must be one ofjson(default),csv, ordict:jsona "file-like" object containing newline-delimited JSONcsva "file-like" object containing newline-delimited CSVdictalistof Pythondictobjects corresponding to list of
newline-delimited JSON, in other wordsList[Dict[str, Any]]applies only ifSchemaGeneratoris used as a library through therun()ordeduce_schema()methoduseful if the input data (usually JSON) has already been read into memory
and parsed from newline-delimited JSON into native Python dict objects.Ifcsvfile is specified, the--keep_nullsflag is automatically activated.
This is required because CSV columns are defined positionally, so the schema
file must contain all the columns specified by the CSV file, in the same
order, even if the column contains an empty value for every record.SeeIssue #26for implementation details.Keep Nulls (--keep_nulls)Normally when the input data file contains a field which has a null, empty
array or empty record as its value, the field is suppressed in the schema file.
This flag enables this field to be included in the schema file.In other words, using a data file containing just nulls and empty values:$generate_schema{"s":null,"a":[],"m":{}}^D
INFO:root:Processed1lines[]With thekeep_nullsflag, we get:$generate-schema--keep_nulls{"s":null,"a":[],"m":{}}^D
INFO:root:Processed1lines[{"mode":"REPEATED","type":"STRING","name":"a"},{"mode":"NULLABLE","fields":[{"mode":"NULLABLE","type":"STRING","name":"__unknown__"}],"type":"RECORD","name":"d"},{"mode":"NULLABLE","type":"STRING","name":"s"}]Quoted Values Are Strings (--quoted_values_are_strings)By default, quoted values are inspected to determine if they can be interpreted
asDATE,TIME,TIMESTAMP,BOOLEAN,INTEGERorFLOAT. This is
consistent with the algorithm used bybq load. However, for theBOOLEAN,INTEGER, orFLOATtypes, it is sometimes more useful to interpret those as
normal strings instead. This flag disables type inference forBOOLEAN,INTEGERandFLOATtypes inside quoted strings.$generate-schema{"name":"1"}^D[{"mode":"NULLABLE","name":"name","type":"INTEGER"}]$generate-schema--quoted_values_are_strings{"name":"1"}^D[{"mode":"NULLABLE","name":"name","type":"STRING"}]Infer Mode (--infer_mode)Set the schemamodeof a field toREQUIREDinstead of the defaultNULLABLEif the field contains a non-null or non-empty value for every data
record in the input file. This option is available only for CSV
(--input_format csv) files. It is theoretically possible to implement this
feature for JSON files, but too difficult to implement in practice because
fields are often completely missing from a given JSON record (instead of
explicitly being defined to benull).In addition to the above, this option, when used in conjunction with--existing_schema_map, will allow fields to be relaxed from REQUIRED to
NULLABLE if they were REQUIRED in the existing schema and NULL rows are found in
the new data we are inferring a schema from. In this case it can be used with
either input_format, CSV or JSON.SeeIssue #28for implementation details.Debugging Interval (--debugging_interval)By default, thegenerate_schema.pyscript prints a short progress message
every 1000 lines of input data. This interval can be changed using the--debugging_intervalflag.$generate-schema--debugging_interval50<file.data.json>file.schema.jsonDebugging Map (--debugging_map)Instead of printing out the BigQuery schema, the--debugging_mapprints out
the bookkeeping metadata map which is used internally to keep track of the
various fields and their types that were inferred using the data file. This
flag is intended to be used for debugging.$generate-schema--debugging_map<file.data.json>file.schema.jsonSanitize Names (--sanitize_names)BigQuery column names arerestricted to certain characters and
length:it must contain only letters (a-z, A-Z), numbers (0-9), or underscoresit must start with a letter or underscorethe maximum length is 128 characterscolumn names are case-insensitiveFor CSV files, thebq loadcommand seems to automatically convert invalid
column names into valid column names. This flag attempts to perform some of the
same transformations, to avoid having to scan through the input data twice to
generate the schema file. The transformations are:any character outside of ASCII letters, numbers and underscore
([a-zA-Z0-9_]) are converted to an underscore. For examplego&2#there!is
converted togo_2_there_;names longer than 128 characters are truncated to 128.My recollection is that thebq loadcommand doesnotnormalize the JSON key
names. Instead it prints an error message. So the--sanitize_namesflag is
useful mostly for CSV files. For JSON files, you'll have to do a second pass
through the data files to cleanup the column names anyway. SeeIssue #14andIssue #33.Ignore Invalid Lines (--ignore_invalid_lines)By default, if an error is encountered on a particular line, processing stops
immediately with an exception. This flag causes invalid lines to be ignored and
processing continues. A list of all errors and their line numbers will be
printed on the STDERR after processing the entire file.This flag is currently most useful for JSON files, to ignore lines which do not
parse correctly as a JSON object.This flag is probably not useful for CSV files. CSV files are processed by theDictReaderclass which performs its own line processing internally, including
extracting the column names from the first line of the file. If theDictReaderdoes throw an exception on a given line, we would not be able to catch it and
continue processing. Fortunately, CSV files are fairly robust, and the schema
deduction logic will handle any missing or extra columns gracefully.FixesIssue #49.Existing Schema Path (--existing_schema_path)There are cases where we would like to start from an existing BigQuery table
schema rather than starting from scratch with a new batch of data we would like
to load. In this case we can specify the path to a local file on disk that is
our existing bigquery table schema. This can be generated via the followingbq show --schemacommand:bqshow--schema<PROJECT_ID>:<DATASET_NAME>.<TABLE_NAME>>existing_table_schema.jsonWe can then run generate-schema with the additional option--existing_schema_pathexisting_table_schema.jsonThere is some subtle interaction between the--existing_schema_pathand fields
which are marked with amodeofREQUIREDin the existing schema. If the new
data contains anullvalue (either in a CSV or JSON data file), it is not
clear if the schema should be changed tomode=NULLABLEor whether the new data
should be ignored and the schema should remainmode=REQUIRED. The choice is
determined by overloading the--infer_modeflag:If--infer_modeis given, the new schema will be allowed to revert back toNULLABLE.If--infer_modeis not given, the offending new record will be ignored
and the new schema will remainREQUIRED.See discussion inPR #57for
more details.Preserve Input Sort Order (--preserve_input_sort_order)By default, the order of columns in the BQ schema file is sorted
lexicographically, which matched the original behavior ofbq load --autodetect. If the--preserve_input_sort_orderflag is given, the columns
in the resulting schema file is not sorted, but preserves the order of
appearance in the input JSON data. For example, the following JSON data with
the--preserve_input_sort_orderflag will produce:$generate-schema--preserve_input_sort_order{"s":"string","i":3,"x":3.2,"b":true}^D[{"mode":"NULLABLE","name":"s","type":"STRING"},{"mode":"NULLABLE","name":"i","type":"INTEGER"},{"mode":"NULLABLE","name":"x","type":"FLOAT"},{"mode":"NULLABLE","name":"b","type":"BOOLEAN"}]It is possible that each JSON record line contains only a partial subset of the
total possible columns in the data set. The order of the columns in the BQ
schema will then be the order that each column was firstseenby the
script:$generate-schema--preserve_input_sort_order{"s":"string","i":3}{"x":3.2,"s":"string","i":3}{"b":true,"x":3.2,"s":"string","i":3}^D[{"mode":"NULLABLE","name":"s","type":"STRING"},{"mode":"NULLABLE","name":"i","type":"INTEGER"},{"mode":"NULLABLE","name":"x","type":"FLOAT"},{"mode":"NULLABLE","name":"b","type":"BOOLEAN"}]Note: In Python 3.6 (the earliest version of Python supported by this
project), the order of keys in adictwas the insertion-order, but this
ordering was an implementation detail, and not guaranteed. In Python 3.7, that
ordering was made permanent. So the--preserve_input_sort_orderflagshouldwork in Python 3.6 but is not guaranteed.See discussion inPR #75for
more details.Using As a LibraryTheSchemaGeneratorclass can be used programmatically as a library from a
larger Python application.SchemaGenerator.run()Thebigquery_schema_generatormodule can be used as a library by an external
Python client code by creating an instance ofSchemaGeneratorand calling therun(input, output)method:frombigquery_schema_generator.generate_schemaimportSchemaGeneratorgenerator=SchemaGenerator(input_format=input_format,infer_mode=infer_mode,keep_nulls=keep_nulls,quoted_values_are_strings=quoted_values_are_strings,debugging_interval=debugging_interval,debugging_map=debugging_map,sanitize_names=sanitize_names,ignore_invalid_lines=ignore_invalid_lines,preserve_input_sort_order=preserve_input_sort_order,)FILENAME="..."withopen(FILENAME)asinput_file:generator.run(input_file=input_file,output_file=output_file)Theinput_formatis one ofjson,csv, anddictas described in theInput Formatsection above. Theinput_filemust match the
format given by this parameter.Seegeneratorrun.pyfor an example.SchemaGenerator.deduce_schema()from FileIf you need to process the generated schema programmatically, create an instance
ofSchemaGeneratorusing the appropriateinput_formatoption, use thededuce_schema()method to read in the file, then postprocess the resultingschema_mapanderror_logdata structures.The following reads in a JSON file (seejsoneader.py):importjsonimportloggingimportsysfrombigquery_schema_generator.generate_schemaimportSchemaGeneratorFILENAME="jsonfile.json"generator=SchemaGenerator(input_format='json',quoted_values_are_strings=True,)withopen(FILENAME)asfile:schema_map,errors=generator.deduce_schema(file)forerrorinerrors:logging.info("Problem on line%s:%s",error['line_number'],error['msg'])schema=generator.flatten_schema(schema_map)json.dump(schema,sys.stdout,indent=2)print()The following reads a CSV file (seecsvreader.py):...(sameasabove)...generator=SchemaGenerator(input_format='csv',infer_mode=True,quoted_values_are_strings=True,sanitize_names=True,)withopen(FILENAME)asfile:schema_map,errors=generator.deduce_schema(file)...(sameasabove)...Thededuce_schema()also supports starting from an existingschema_mapinstead of starting from scratch. This is the internal version of the--existing_schema_pathfunctionality.schema_map1,errors=generator.deduce_schema(input_data=data1)schema_map2,errors=generator.deduce_schema(input_data=data1,schema_map=schema_map1)Theinput_datamust match theinput_formatgiven in the constructor. The
format is described in theInput Formatsection above.SchemaGenerator.deduce_schema()from Iterable of DictIf the JSON data set has already been read into memory into an array or iterable
of Pythondictobjects, theSchemaGeneratorcan process that too using theinput_format='dict'option. Here is an example fromdictreader.py:importjsonimportloggingimportsysfrombigquery_schema_generator.generate_schemaimportSchemaGeneratorgenerator=SchemaGenerator(input_format='dict')input_data=[{'s':'string','b':True,},{'d':'2021-08-18','x':3.1},]schema_map,errors=generator.deduce_schema(input_data)schema=generator.flatten_schema(schema_map)json.dump(schema,sys.stdout,indent=2)print()Note: Theinput_format='dict'option supports anyinput_dataobject
which acts like an iterable ofdict. The data does not have to be loaded into
memory.SchemaGenerator.deduce_schema()from csv.DictReaderTheinput_format='csvdictreader'option is similar toinput_format='dict'but sort of acts likeinput_format='csv'. It supports any object that behaves
like an iterable ofdict, but it is intended to be used with thecsv.DictReaderobject.The difference between'dict'and'csvdictreader'is the assumption made
about the shape of the data. The'csvdictreader'option assumes that the data
is tabular like a CSV file, with every row usually containing an entry for every
column. The'dict'option does not make that assumption, and the data can be
more hierarchical with some rows containing partial sets of columns.This semantic difference means that'csvdictreader'supports options which
apply to'csv'files. In particular, theinfer_mode=Trueoption can be used
to determine if themodefield can beREQUIREDinstead ofNULLABLEif the
script finds that all columns are defined in every row.Here is an example fromtsvreader.pywhich reads a
tab-separate file (TSV):importcsvimportjsonimportsysfrombigquery_schema_generator.generate_schemaimportSchemaGeneratorFILENAME="tsvfile.tsv"generator=SchemaGenerator(input_format='dict')withopen(FILENAME)asfile:reader=csv.DictReader(file,delimiter='\t')schema_map,errors=generator.deduce_schema(reader)schema=generator.flatten_schema(schema_map)json.dump(schema,sys.stdout,indent=2)print()Schema TypesSupported TypesThebq show --schemacommand produces a JSON schema file that uses the
olderLegacy SQL date types.
For compatibility,generate-schemascript will also generate a schema file
using the legacy data types.The supported types are:BOOLEANINTEGERFLOATSTRINGTIMESTAMPDATETIMERECORDThegenerate-schemascript supports bothNULLABLEandREPEATEDmodes of
all of the above types.The supported format ofTIMESTAMPis as close as practical to thebq load format:YYYY-[M]M-[D]D[( |T)[H]H:[M]M:[S]S[.DDDDDD]][time zone]which appears to be an extension of theISO 8601 format.
The difference frombq loadis that the[time zone]component can be onlyZUTC(same asZ)(+|-)H[H][:M[M]]Note that BigQuery supports up to 6 decimal places after the integer 'second'
component.generate-schemafollows the same restriction for compatibility. If
your input file contains more than 6 decimal places, you need to write a data
cleansing filter to fix this.The suffixUTCis not standard ISO 8601 nordocumented by Googlebut theUTCsuffix is used bybq extractand the web interface. (SeeIssue 19.)Timezone names from thetz database(e.g.
"America/Los_Angeles") arenotsupported bygenerate-schema.The following types arenotsupported at all:BYTESDATETIME(unable to distinguish fromTIMESTAMP)Type Inference RulesThegenerate-schemascript attempts to emulate the various type conversion and
compatibility rules implemented bybq load:INTEGERcan upgrade toFLOATif a field in an early record is anINTEGER, but a subsequent record
shows this field to have aFLOATvalue, the type of the field will be
upgraded to aFLOATthe reverse does not happen, once a field is aFLOAT, it will remain aFLOATconflictingTIME,DATE,TIMESTAMPtypes upgrades toSTRINGif a field is determined to have one type of "time" in one record, then
subsequently a different "time" type, then the field will be assigned aSTRINGtypeNULLABLE RECORDcan upgrade to aREPEATED RECORDa field may be defined asRECORD(aka "Struct") type with{ ... }if the field is subsequently read as an array with a[{ ... }], the
field is upgraded to aREPEATED RECORDa primitive type (FLOAT,INTEGER,STRING) cannot upgrade to aREPEATEDprimitive typethere's no technical reason why this cannot be allowed, butbq loaddoes not support it, so we follow its behavioraDATETIMEfield is always inferred to be aTIMESTAMPthe format of these two fields is identical (in the absence of timezone)we follow the same logic asbq loadand always infer these asTIMESTAMPBOOLEAN,INTEGER, andFLOATcan appear inside quoted stringsIn other words,"true"(or"True"or"false", etc) is considered a
BOOLEAN type,"1"is considered an INTEGER type, and"2.1"is
considered a FLOAT type. Luigi Mori (jtschichold@) added additional logic
to replicate the type conversion logic used bybq loadfor these
strings.This type inference inside quoted strings can be disabled using the--quoted_values_are_stringsflag(SeeIssue #22for more details.)INTEGERvalues overflowing a 64-bit signed integer upgrade toFLOATintegers greater than2^63-1(9223372036854775807)integers less than-2^63(-9223372036854775808)(SeeIssue #18for more details)ExamplesHere is an example of a single JSON data record on the STDIN (the^Dbelow
means typing Control-D, which indicates "end of file" under Linux and MacOS):$generate-schema{"s":"string","b":true,"i":1,"x":3.1,"t":"2017-05-22T17:10:00-07:00"}^D
INFO:root:Processed1lines[{"mode":"NULLABLE","name":"b","type":"BOOLEAN"},{"mode":"NULLABLE","name":"i","type":"INTEGER"},{"mode":"NULLABLE","name":"s","type":"STRING"},{"mode":"NULLABLE","name":"t","type":"TIMESTAMP"},{"mode":"NULLABLE","name":"x","type":"FLOAT"}]In most cases, the data file will be stored in a file:$cat>file.data.json{"a":[1,2]}{"i":3}^D

$generate-schema<file.data.json>file.schema.json
INFO:root:Processed2lines

$catfile.schema.json[{"mode":"REPEATED","name":"a","type":"INTEGER"},{"mode":"NULLABLE","name":"i","type":"INTEGER"}]Here is the schema generated from a CSV input file. The first line is the header
containing the names of the columns, and the schema lists the columns in the
same order as the header:$generate-schema--input_formatcsv
e,b,c,d,a1,x,true,,2.02,x,,,43,,,,
^D
INFO:root:Processed3lines[{"mode":"NULLABLE","name":"e","type":"INTEGER"},{"mode":"NULLABLE","name":"b","type":"STRING"},{"mode":"NULLABLE","name":"c","type":"BOOLEAN"},{"mode":"NULLABLE","name":"d","type":"STRING"},{"mode":"NULLABLE","name":"a","type":"FLOAT"}]Here is an example of the schema generated with the--infer_modeflag:$generate-schema--input_formatcsv--infer_mode
name,surname,age
John
Michael,,
Maria,Smith,30
Joanna,Anders,21
^D
INFO:root:Processed4lines[{"mode":"REQUIRED","name":"name","type":"STRING"},{"mode":"NULLABLE","name":"surname","type":"STRING"},{"mode":"NULLABLE","name":"age","type":"INTEGER"}]BenchmarksI wrote thebigquery_schema_generator/anonymize.pyscript to create an
anonymized data filetests/testdata/anon1.data.json.gz:$./bigquery_schema_generator/anonymize.py<original.data.json\>anon1.data.json
$gzipanon1.data.jsonThis data file is 290MB (5.6MB compressed) with 103080 data records.Generating the schema using$bigquery_schema_generator/generate_schema.py<anon1.data.json\>anon1.schema.jsontook 67s on a Dell Precision M4700 laptop with an Intel Core i7-3840QM CPU @
2.80GHz, 32GB of RAM, Ubuntu Linux 18.04, Python 3.6.7.System RequirementsThis project was initially developed on Ubuntu 17.04 using Python 3.5.3, but it
now requires Python 3.6 or higher, I think mostly due to the use of f-strings.I have tested it on:Ubuntu 22.04, Python 3.10.6Ubuntu 20.04, Python 3.8.5Ubuntu 18.04, Python 3.7.7Ubuntu 18.04, Python 3.6.7Ubuntu 17.10, Python 3.6.3MacOS 12.6.2 (Monterey), Python 3.10.9MacOS 11.7.2 (Big Sur), Python 3.10.9MacOS 11.7.2 (Big Sur), Python 3.8.9MacOS 10.14.2 (Mojave), Python 3.6.4MacOS 10.13.2 (High Sierra), Python 3.6.4The GitHub Actions continuous integration pipeline validates on Python 3.7,
3.8, 3.9, and 3.10.The unit tests are invoked with$ make teststarget, and depends only on the
built-in Pythonunittestpackage.The coding style check is invoked using$ make flake8and depends on theflake8package. It can be installed using$ pip3 install --user flake8.LicenseApache License 2.0Feedback and SupportIf you have any questions, comments, or feature requests for this library,
please use theGitHub
Discussionsfor this project. If you have bug reports, please file a ticket inGitHub
Issues. Feature
requests should go into Discussions first because they often have alternative
solutions which are useful to remain visible, instead of disappearing from the
default view of the Issue tracker after the ticket is closed.Please refrain from emailing me directly unless the content is sensitive. The
problem with email is that I cannot reference the email conversation when other
people ask similar questions later.AuthorsCreated by Brian T. Park (brian@xparks.net).Type inference inside quoted strings by Luigi Mori (jtschichold@).Flag to disable type inference inside quoted strings by Daniel Ecer
(de-code@).Support for CSV files and detection ofREQUIREDfields by Sandor Korotkevics
(korotkevics@).Better support for usingbigquery_schema_generatoras a library from an
external Python code by StefanoG_ITA (StefanoGITA@).Sanitizing of column names to valid BigQuery characters and length by Jon
Warghed (jonwarghed@).Bug fix in--sanitize_namesby Riccardo M. Cefala (riccardomc@).Print full path of nested JSON elements in error messages, by Austin Brogle
(abroglesc@).Allow an existing schema file to be specified using--existing_schema_path,
by Austin Brogle (abroglesc@) and Bozo Dragojevic (bozzzzo@).AllowSchemaGenerator.deduce_schema()to accept a list of native Pythondictobjects, by Zigfrid Zvezdin (ZiggerZZ@).Make the column order in the BQ schema file match the order of appearance in
the JSON data file using the--preserve_input_sort_orderflag. By Kevin
Deggelman (kdeggelman@).