Package: bigframes
BigQuery DataFrames provides a Pythonic DataFrame and machine learning (ML) API
powered by the BigQuery engine.bigframes.pandasprovides a pandas-compatible API for analytics.bigframes.mlprovides a scikit-learn-like API for ML.BigQuery DataFrames is an open-source package. You can runpip install--upgradebigframesto install the latest version.DocumentationBigQuery DataFrames source code (GitHub)BigQuery DataFrames sample notebooksBigQuery DataFrames API referenceBigQuery documentationQuickstartPrerequisitesInstall thebigframespackage.Create a Google Cloud project and billing account.In an interactive environment (like Notebook, Python REPL or command line),bigframeswill do the authentication on-the-fly if needed. Otherwise, seehow to set up application default credentialsfor various environments. For example, to pre-authenticate on your laptop you caninstall and initialize the gcloud CLI,
and then generate the application default credentials by doinggcloud auth application-default login.Code sampleImportbigframes.pandasfor a pandas-like interface. Theread_gbqmethod accepts either a fully-qualified table ID or a SQL query.importbigframes.pandasasbpdbpd.options.bigquery.project=your_gcp_project_iddf1=bpd.read_gbq("project.dataset.table")df2=bpd.read_gbq("SELECT a, b, c, FROM `project.dataset.table`")More code samplesLocationsBigQuery DataFrames uses aBigQuery sessioninternally to manage metadata on the service side. This session is tied to alocation.
BigQuery DataFrames uses the US multi-region as the default location, but you
can usesession_options.locationto set a different location. Every query
in a session is executed in the location where the session was created.
BigQuery DataFrames
auto-populatesbf.options.bigquery.locationif the user starts withread_gbq/read_gbq_table/read_gbq_query()and specifies a table, either
directly or in a SQL statement.If you want to reset the location of the created DataFrame or Series objects,
you can close the session by executingbigframes.pandas.close_session().
After that, you can reusebigframes.pandas.options.bigquery.locationto
specify another location.read_gbq()requires you to specify a location if the dataset you are
querying is not in the US multi-region. If you try to read a table from another
location, you get a NotFound exception.ProjectIfbf.options.bigquery.projectis not set, the$GOOGLE_CLOUD_PROJECTenvironment variable is used, which is set in the notebook runtime serving the
BigQuery Studio/Vertex Notebooks.ML CapabilitiesThe ML capabilities in BigQuery DataFrames let you preprocess data, and
then train models on that data. You can also chain these actions together to
create data pipelines.Preprocess dataCreate transformers to prepare data for use in estimators (models) by
using thebigframes.ml.preprocessing moduleand thebigframes.ml.compose module.
BigQuery DataFrames offers the following transformations:Use theKBinsDiscretizer classin thebigframes.ml.preprocessingmodule to bin continuous data into intervals.Use theLabelEncoder classin thebigframes.ml.preprocessingmodule to normalize the target labels as integer values.Use theMaxAbsScaler classin thebigframes.ml.preprocessingmodule to scale each feature to the range[-1,1]by its maximum absolute value.Use theMinMaxScaler classin thebigframes.ml.preprocessingmodule to standardize features by scaling each feature to the range[0, 1].Use theStandardScaler classin thebigframes.ml.preprocessingmodule to standardize features by removing the mean and scaling to unit variance.Use theOneHotEncoder classin thebigframes.ml.preprocessingmodule to transform categorical values into numeric format.Use theColumnTransformer classin thebigframes.ml.composemodule to apply transformers to DataFrames columns.Train modelsCreate estimators to train models in BigQuery DataFrames.Clustering modelsCreate estimators for clustering models by using thebigframes.ml.cluster module.Use theKMeans classto create K-means clustering models. Use these models for
data segmentation. For example, identifying customer segments. K-means is an
unsupervised learning technique, so model training doesn’t require labels or split
data for training or evaluation.Decomposition modelsCreate estimators for decomposition models by using thebigframes.ml.decomposition module.Use thePCA classto create principal component analysis (PCA) models. Use these
models for computing  principal components and using them to perform a change of
basis on the data. This provides dimensionality reduction by projecting each data
point onto only the first few principal components to obtain lower-dimensional
data while preserving as much of the data’s variation as possible.Ensemble modelsCreate estimators for ensemble models by using thebigframes.ml.ensemble module.Use theRandomForestClassifier classto create random forest classifier models. Use these models for constructing multiple
learning method decision trees for classification.Use theRandomForestRegressor classto create random forest regression models. Use
these models for constructing multiple learning method decision trees for regression.Use theXGBClassifier classto create gradient boosted tree classifier models. Use these models for additively
constructing multiple learning method decision trees for classification.Use theXGBRegressor classto create gradient boosted tree regression models. Use these models for additively
constructing multiple learning method decision trees for regression.Forecasting modelsCreate estimators for forecasting models by using thebigframes.ml.forecasting module.Use theARIMAPlus classto create time series forecasting models.Imported modelsCreate estimators for imported models by using thebigframes.ml.imported module.Use theONNXModel classto import Open Neural Network Exchange (ONNX) models.Use theTensorFlowModel classto import TensorFlow models.Use theXGBoostModel classto import XGBoostModel models.Linear modelsCreate estimators for linear models by using thebigframes.ml.linear_model module.Use theLinearRegression classto create linear regression models. Use these models for forecasting. For example,
forecasting the sales of an item on a given day.Use theLogisticRegression classto create logistic regression models. Use these models for the classification of two
or more possible values such as whether an input islow-value,medium-value,
orhigh-value.Large language modelsCreate estimators for LLMs by using thebigframes.ml.llm module.Use theGeminiTextGenerator classto create Gemini text generator models. Use these models
for text generation tasks.Use thePaLM2TextGenerator classto create PaLM2 text generator models. Use these models
for text generation tasks.Use thePaLM2TextEmbeddingGenerator classto create PaLM2 text embedding generator models.
Use these models for text embedding generation tasks.Create pipelinesCreate ML pipelines by usingbigframes.ml.pipeline module.
Pipelines let you assemble several ML steps to be cross-validated together while setting
different parameters. This simplifies your code, and allows you to deploy data preprocessing
steps and an estimator together.Use thePipeline classto create a pipeline of transforms with a final estimator.ML locationsbigframes.mlsupports the same locations as BigQuery ML. BigQuery ML model
prediction and other ML functions are supported in all BigQuery regions. Support
for model training varies by region. For more information, seeBigQuery ML locations.Data typesBigQuery DataFrames supports the following numpy and pandas dtypes:numpy.dtype("O")pandas.BooleanDtype()pandas.Float64Dtype()pandas.Int64Dtype()pandas.StringDtype(storage="pyarrow")pandas.ArrowDtype(pa.date32())pandas.ArrowDtype(pa.time64("us"))pandas.ArrowDtype(pa.timestamp("us"))pandas.ArrowDtype(pa.timestamp("us",tz="UTC"))BigQuery DataFrames doesn’t support the following BigQuery data types:ARRAYNUMERICBIGNUMERICINTERVALSTRUCTJSONAll other BigQuery data types display as the object type.Remote functionsBigQuery DataFrames gives you the ability to turn your custom scalar functions
intoBigQuery remote functions. Creating a remote
function in BigQuery DataFrames (Seecode samples)
creates a BigQuery remote function, aBigQuery
connection,
and aCloud Functions (2nd gen) function.BigQuery connections are created in the same location as the BigQuery
DataFrames session, using the name you provide in the custom function
definition. To view and manage connections, do the following:Go toBigQuery in the Google Cloud Console.Select the project in which you created the remote function.In the Explorer pane, expand that project and then expand External connections.BigQuery remote functions are created in the dataset you specify, or
in a special type ofhidden datasetreferred to as an anonymous dataset. To view and manage remote functions created
in a user provided dataset, do the following:Go toBigQuery in the Google Cloud Console.Select the project in which you created the remote function.In the Explorer pane, expand that project, expand the dataset in which you
created the remote function, and then expand Routines.To view and manage Cloud Functions functions, use theFunctionspage and use the project picker to select the project in which you
created the function. For easy identification, the names of the functions
created by BigQuery DataFrames are prefixed bybigframes.RequirementsTo use BigQuery DataFrames remote functions, you must enable the following APIs:The BigQuery API (bigquery.googleapis.com)The BigQuery Connection API (bigqueryconnection.googleapis.com)The Cloud Functions API (cloudfunctions.googleapis.com)The Cloud Run API (run.googleapis.com)The Artifact Registry API (artifactregistry.googleapis.com)The Cloud Build API (cloudbuild.googleapis.com )The Cloud Resource Manager API (cloudresourcemanager.googleapis.com)To use BigQuery DataFrames remote functions, you must be granted the
following IAM roles:BigQuery Data Editor (roles/bigquery.dataEditor)BigQuery Connection Admin (roles/bigquery.connectionAdmin)Cloud Functions Developer (roles/cloudfunctions.developer)Service Account User (roles/iam.serviceAccountUser) on theservice accountPROJECT_NUMBER-compute@developer.gserviceaccount.comStorage Object Viewer (roles/storage.objectViewer)Project IAM Admin (roles/resourcemanager.projectIamAdmin)To use BigQuery DataFrames ML remote models(bigframes.ml.remote or bigframes.ml.llm), you must enable the following APIs:The BigQuery API (bigquery.googleapis.com)The BigQuery Connection API (bigqueryconnection.googleapis.com)The Vertex AI API (aiplatform.googleapis.com)To use BigQuery DataFrames ML remote models(bigframes.ml.remote or bigframes.ml.llm), you must be granted the
following IAM roles:BigQuery Data Editor (roles/bigquery.dataEditor)BigQuery Connection Admin (roles/bigquery.connectionAdmin)Service Account User (roles/iam.serviceAccountUser) on theservice accountPROJECT_NUMBER-compute@developer.gserviceaccount.comVertex AI User (roles/aiplatform.user)Project IAM Admin (roles/resourcemanager.projectIamAdmin)LimitationsRemote functions take about 90 seconds to become available when you first create them.Trivial changes in the notebook, such as inserting a new cell or renaming a variable,
might cause the remote function to be re-created, even if these changes are unrelated
to the remote function code.BigQuery DataFrames does not differentiate any personal data you include in the remote
function code. The remote function code is serialized as an opaque box to deploy it as a
Cloud Functions function.The Cloud Functions (2nd gen) functions, BigQuery connections, and BigQuery remote
functions created by BigQuery DataFrames persist in Google Cloud. If you don’t want to
keep these resources, you must delete them separately using an appropriate Cloud Functions
or BigQuery interface.A project can have up to 1000 Cloud Functions (2nd gen) functions at a time. See Cloud
Functions quotas for all the limits.Quotas and limitsBigQuery quotasincluding hardware, software, and network components.Session terminationEach BigQuery DataFrames DataFrame or Series object is tied to a BigQuery
DataFrames session, which is in turn based on a BigQuery session. BigQuery
sessionsauto-terminate; when this happens, you can’t use previously
created DataFrame or Series objects and must re-create them using a new
BigQuery DataFrames session. You can do this by runningbigframes.pandas.close_session()and then re-running the BigQuery
DataFrames expressions.Data processing locationBigQuery DataFrames is designed for scale, which it achieves by keeping data
and processing on the BigQuery service. However, you can bring data into the
memory of your client machine by calling.to_pandas()on a DataFrame or Series
object. If you choose to do this, the memory limitation of your client machine
applies.LicenseBigQuery DataFrames is distributed with theApache-2.0 license.It also contains code derived from the following third-party packages:IbispandasPythonscikit-learnXGBoostFor details, see thethird_partydirectory.Contact UsFor further help and provide feedback, you can email us atbigframes-feedback@google.com.