Package: bigfunctions
SuperchargeBigQuerywithBigFunctionsUpgrade your data impactwith 100+ ready-to-use BigQuery Functions(+ build a catalog of functions)❯Website|GitHub❮🔍️ 1. What is BigFunctions?BigFunctions is:✅a frameworkto build agoverned catalog of powerful BigQuery functionsat YOUR company.✅100+ open-source functions to supercharge BigQuerythat you can call directly (no install) or redeploy in YOUR catalog.💡 2. Why BigFunctions?As a data-analystYou'll have new powers!(such as loading data from any source or activating your data through reverse ETL).As an analytics-engineerYou'll feel at home with BigFunctions style which imitates the one of dbt(with a yaml standard and a CLI).You'll love the idea of getting more things done through SQL.As a data-engineerYou'll easily build software-engineering best practices through unit testing, cicd, pull request validation, continuous deployment, etc.You will love avoiding reinventing the wheel by using functions already developed by the community.As a central data-team player in a large companyYou'll be proud of providing a governed catalog of curated functions to your 10000+ employees with mutualized and maintainable effort.As a security championYou will enjoy the ability to validate the code of functions before deployment thanks to your git validation workflow, CI Testing, binary authorization, etc.As an open-source loverYou'll be able to contribute so that a problem solved for you is solved for everyone.👀 3. Call public BigFunctions without install from your GCP projectAll BigFunctions represented by a 'yaml' file inbigfunctionsfolder of the GitHub repo are automatically deployed in public datasets so that you can call them directly without install from your BigQuery project.Give it a try! Execute this SQL query from your GCP Project 👀:selectbigfunctions.eu.faker("name","it_IT")Explore all available bigfunctionshere.🚀 4. Deploy BigFunctions in your GCP projectYou can also deploy any bigfunction in your project! To deploymy_bigfunctiondefined inbigfunctions/my_bigfunction.yamlfile, simply call:bigfundeploymy_bigfunctionDetails aboutbigfuncommand line are given below.💥 5.bigfunCLIbigfunCLI (command-line-interface) facilitates BigFunctions development, test, deployment, documentation and monitoring.5.1 Installbigfun🛠️Clone the repo and from the repo directory run:virtualenvvenv
.venv/bin/activate
pipinstall--editable.5.2 Usebigfun🔥$bigfun--help
Usage:bigfun[OPTIONS]COMMAND[ARGS]...

Options:--helpShowthismessageandexit.

Commands:deployDeployBIGFUNCTIONdocGenerate,serveandpublishdocumentationtestTestBIGFUNCTION5.3 Deploy you first function 👨‍💻Make sure thegcloudcommand isinstalled on your computerActivate the application-default account withgcloud auth application-default login. A browser window should open, and you should be prompted to log into your Google account. Once you've done that,bigfunwill use your oauth'd credentials to connect to BigQuery through BigQuery python client!Get or create aDATASETwhere you have permission to edit data and where the function will be deployed.TheDATASETmust belong to aPROJECTin which you have permission to run BigQuery queries.You now can deployis_email_validfunction with:bigfundeployis_email_validThe first time you run this command it will ask forPROJECTandDATASET.Your inputs will be written toconfig.yamlfile in current directory so that you won't be asked again (unless you delete the entries inconfig.yaml). You can also override this config at deploy time:bigfun deploy is_email_valid --project=PROJECT --dataset=DATASET.Test it with 👀:selectPROJECT.DATASET.is_email_valid('paul.marcombes@unytics.io')5.4 Deploy you first javascript function which depends onnpm packages👽To deploy ajavascriptfunctionwhich depends onnpm packagesthere are additional requirementsin addition to the ones above.You will need to install eachnpm packageon your machine and bundle it into one file. For that, you need toinstallnodejs.The bundled js file will be uploaded into a cloud storage bucket in which you must have write access. The bucket name is asked when you runbigfun deploy. Users of your functions must have read access to the bucket.You now can deployrender_templatefunction with:bigfundeployrender_templateTest it with 👀:selectPROJECT.DATASET.render_template('Hello {{ user }}',json'{"user": "James"}')5.5 Deploy you firstremotefunction ⚡️To deploy aremotefunction(e.g. python function), there are additional requirementsin addition to the ones ofDeploy you first functionsection.ACloud Runservice will be deployed to host the code (as seen here). So you must havepermissions to deploy aCloud Runservicein your projectPROJECT.gcloudCLI will be used directly to deploy the service (usinggcloud run deploy). Then, make sure you are logged in withgcloudby calling:gcloud auth login. A browser window should also open, and you should be prompted to log into your Google account. WARNING: you read correctly: you have to authenticate twice. Once for bigquery python client (to deploy any function including remote as seen above.) and once now to usegcloud(to deploy aCloud Runservice).ABigQuery Remote Connectionwill be created to link BigQuery with theCloud Runservice. You then should have permissions to create a remote connection.BigQuery Connection AdminorBigQuery Adminroles have these permissions.A service account will be automatically created by Google along with theBigQuery Remote Connection. BigQuery will use this service account of the remote connection to invoke theCloud Runservice. You then must have the permission to authorize this service account to invoke theCloud Runservice. This permission is provided in the roleroles/run.adminYou now can deployfakerfunction with:bigfundeployfakerTest it with 👀:selectPROJECT.DATASET.faker("name","it_IT")👋 6. ContributeBigFunctions is fully open-source. Any contribution is more than welcome 🤗!Add a ⭐ on the repo to show your supportJoin our Slackand talk with usSuggest a new functionhereRaise an issuethereOpen a Pull-Request! (Seecontributing instructions).Contributors