Package: openmetadata-sqlalchemy-bigquery
SQLALchemy DialectsDialect DocumentationProduct DocumentationQuick StartIn order to use this library, you first need to go through the following steps:Select or create a Cloud Platform project.[Optional]Enable billing for your project.Enable the BigQuery Storage API.Setup Authentication.InstallationInstall this library in avirtualenvusing pip.virtualenvis a tool to
create isolated Python environments. The basic problem it addresses is one of
dependencies and versions, and indirectly permissions.Withvirtualenv, it’s possible to install this library without needing system
install permissions, and without clashing with the installed system
dependencies.Supported Python VersionsPython >= 3.6Unsupported Python VersionsPython <= 3.5.Mac/Linuxpip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install sqlalchemy-bigqueryWindowspip install virtualenv
virtualenv <your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exe install sqlalchemy-bigqueryUsageSQLAlchemyfromsqlalchemyimport*fromsqlalchemy.engineimportcreate_enginefromsqlalchemy.schemaimport*engine=create_engine('bigquery://project')table=Table('dataset.table',MetaData(bind=engine),autoload=True)print(select([func.count('*')],from_obj=table).scalar())Projectprojectinbigquery://projectis used to instantiate BigQuery client with the specific project ID. To infer project from the environment, usebigquery://– withoutprojectAuthenticationFollow theGoogle Cloud library guidefor authentication. Alternatively, you can provide the path to a service account JSON file increate_engine():engine=create_engine('bigquery://',credentials_path='/path/to/keyfile.json')LocationTo specify location of your datasets passlocationtocreate_engine():engine=create_engine('bigquery://project',location="asia-northeast1")Table namesTo query tables from non-default projects or datasets, use the following format for the SQLAlchemy schema name:[project.]dataset, e.g.:# If neither dataset nor project are the defaultsample_table_1=Table('natality',schema='bigquery-public-data.samples')# If just dataset is not the defaultsample_table_2=Table('natality',schema='bigquery-public-data')Batch sizeBy default,arraysizeis set to5000.arraysizeis used to set the batch size for fetching results. To change it, passarraysizetocreate_engine():engine=create_engine('bigquery://project',arraysize=1000)Page size for dataset.list_tablesBy default,list_tables_page_sizeis set to1000.list_tables_page_sizeis used to set the max_results fordataset.list_tablesoperation. To change it, passlist_tables_page_sizetocreate_engine():engine=create_engine('bigquery://project',list_tables_page_size=100)Adding a Default DatasetIf you want to have theClientuse a default dataset, specify it as the “database” portion of the connection string.engine=create_engine('bigquery://project/dataset')When using a default dataset, don’t include the dataset name in the table name, e.g.:table=Table('table_name')Note that specifying a default dataset doesn’t restrict execution of queries to that particular dataset when using raw queries, e.g.:# Set default dataset to dataset_aengine=create_engine('bigquery://project/dataset_a')# This will still execute and return rows from dataset_bengine.execute('SELECT * FROM dataset_b.table').fetchall()Connection String ParametersThere are many situations where you can’t callcreate_enginedirectly, such as when using tools likeFlask SQLAlchemy. For situations like these, or for situations where you want theClientto have adefault_query_job_config, you can pass many arguments in the query of the connection string.Thecredentials_path,credentials_info,location,arraysizeandlist_tables_page_sizeparameters are used by this library, and the rest are used to create aQueryJobConfigNote that if you want to use query strings, it will be more reliable if you use three slashes, so'bigquery:///?a=b'will work reliably, but'bigquery://?a=b'might be interpreted as having a “database” of?a=b, depending on the system being used to parse the connection string.Here are examples of all the supported arguments. Any not present are either for legacy sql (which isn’t supported by this library), or are too complex and are not implemented.engine=create_engine('bigquery://some-project/some-dataset''?''credentials_path=/some/path/to.json''&''location=some-location''&''arraysize=1000''&''list_tables_page_size=100''&''clustering_fields=a,b,c''&''create_disposition=CREATE_IF_NEEDED''&''destination=different-project.different-dataset.table''&''destination_encryption_configuration=some-configuration''&''dry_run=true''&''labels=a:b,c:d''&''maximum_bytes_billed=1000''&''priority=INTERACTIVE''&''schema_update_options=ALLOW_FIELD_ADDITION,ALLOW_FIELD_RELAXATION''&''use_query_cache=true''&''write_disposition=WRITE_APPEND')Creating tablesTo add metadata to a table:table=Table('mytable',...,bigquery_description='my table description',bigquery_friendly_name='my table friendly name')To add metadata to a column:Column('mycolumn',doc='my column description')Threading and MultiprocessingBecause this client uses thegrpclibrary, it’s safe to
share instances across threads.In multiprocessing scenarios, the best
practice is to create client instancesafterthe invocation ofos.forkbymultiprocessing.pool.Poolormultiprocessing.Process.