Package: gfluent
Google Cloud Fluent ClientVersion: 1.2.1This is a lightweight wrapper on top of Google Cloud Platform Python SDK client librariesBigQuery,StorageandSpreadsheet. It is a great package for Data Engineers for craft data pipeline by usingBigQueryandStorageas major services from Google Cloud Platform.The purpose of this package are,Having a consistent way of using the GCP client librariesManage the version in a single place if multiple teams are using the GCP client librariesMake it easier to accomplish the typical Data Engineering tasks (copy data, load and export)The code explains what it doesThe current embedded client libraires versions are,google-api-python-client==2.36.0google-cloud-bigquery==2.32.0google-cloud-storage==2.1.0Build Data Pipeline on BigQueryYou (A Data Engineer) are asked to,Upload multiplejsonfiles from your local drive to GCSImport those files to a BigQuery staging tableRun a SQL query based on the staging table by joining existing tables, and store the result
to a new tableTo accomplish the task, here are the source code,fromgfluentimportBQ,GCSproject_id="here-is-you-project-id"bucket_name="my-bucket"dataset="sales"table_name="products"prefix="import"local_path="/user/tom/products/"# there are few *.json files in this directory# uplaod files to GCS bucket(GCS(project_id).local(path=local_path,suffix=".json").bucket(bucket_name).prefix(prefix).upload())# create the target dataset (in case not exists)BQ(project_id).create_dataset(dataset,location="US")# load json files to BigQuery tableuri=f"gs://{bucket_name}/{prefix}/*.json"number_of_rows=(BQ(project_id).table(f"{dataset}.{table_name}").mode("WRITE_APPEND")# don't have to, default mode.create_mode("CREATE_IF_NEEDED")# don't have to, default mode.format("NEWLINE_DELIMITED_JSON")# don't have to, default format.gcs(uri).load(location="US"))print(f"{number_of_rows}rows are loaded")# run a SQL query and save to a final tablefinal_table="sales_summary"sql="""select t1.col1, t2.col2, t2.col3FROMsales.products t1JOINother.category t2ON  t1.prod_id = t2.prod_id"""number_of_rows=(BQ(product_id).table(f"{dataset}.{final_table}").sql(sql).create_mode("CREATE_NEVER")# have to, don't want to create new table.query())print(f"{number_of_rows}rows are loaded to{final_table}")# now let's query the new tablerows=(BQ(product_id).sql(f"select col1, col2 from{dataset}.{final_table}limit 10").query())forrowinrows:print(row.col1,row.col2)Loading data from Spreadsheet to BigQueryHere is another example to use theSheetclass for loading data from Google Spreadsheet.importosfromgfluentimportSheet,BQproject_id='your-project-id'sheet_id='the-google-spreadsheet-id'# assume the data is on the sheet `data` and range is `A1:B4`sheet=Sheet(os.getenv("GOOGLE_APPLICATION_CREDENTIALS")).sheet_id(sheet_id).worksheet("data!A1:B4")bq=BQ(project=project_id).table("target_dataset.table")sheet.bq(bq).load(location="EU")DocumentsHere is thedocument, and please refer to the test
cases to see more real examples.InstallationInstall from PyPi,pipinstall-UgfluentOr build and install from source code,gitclonegit@github.com:simple-dev-tools/gfluent.gitcdgfluent
maketest-ut
pythonsetup.pyinstallContributionAny kinds of contribution is welcome, including report bugs, add feature or enhance the document. Please
be noted,Unit Testing with mock is intensively used, because we don't want to connect to a real GCP projectPlease installpre-commitby usingpip install pre-committhenpre-commit installbump2versionis used for update the version tag in various files