Package: featurebyte
FeatureByte's Declarative Feature Engineering FrameworkFeatureByte SDK is the core engine of FeatureByte'sSelf-Service Feature Platform. It is afree and source available feature platformdesigned to:Create state-of-the-art features, not data pipelines:Create features for Machine Learning with just a few lines of code. Leave the plumbing and pipelining to FeatureByte. We take care of orchestrating the data ops - whether it’s time-window aggs or backfilling, so you can deliver more value from data.Improve Accuracy through data:Use the intuitive feature declaration framework to transform creative ideas into training data in minutes. Ditch the limitations of ad-hoc pipelines for features with much more scale, complexity and freshness.Streamline machine learning data pipelines:Get more value from AI. Faster. Deploy and serve features in minutes, instead of weeks or months. Declare features in Python and automatically generate optimized data pipelines — all using tools you love like Jupyter Notebooks.Take charge of the entire ML feature lifecycleFeature Engineering and management doesn’t have to be complicated. Take charge of the entire ML feature lifecycle. With FeatureByte, you cancreate, experiment, serve and manage your features in one tool.CreateCreate and share state-of-the-art ML features effortlesslySearch and reuse features to create feature lists tailored to your use case# Get view from cataloginvoice_view=catalog.get_view("GROCERYINVOICE")# Declare features of total spent by customer in the past 7 and 28 dayscustomer_purchases=invoice_view.groupby("GroceryCustomerGuid").aggregate_over("Amount",method="sum",feature_names=["CustomerTotalSpent_7d","CustomerTotalSpent_28d"],fill_value=0,windows=['7d','28d'])customer_purchases.save()ExperimentImmediately access historical features through automated backfilling - let FeatureByte handle the complexity of time-aware SQLExperiment on live data at scale, innovating fasterIterate rapidly with different feature lists to create more accurate models# Get feature list from the catalogfeature_list=catalog.get_feature_list("200 Features on Active Customers")# Get an observation set from the catalogobservation_set=catalog.get_observation_table("5M rows of active Customers in 2021-2022")# Compute training data and# store it in the feature store for reuse and audittraining=\feature_list.compute_historical_feature_table(observation_set,name="Training set to predict purchases next 2w")ServeDeploy AI data pipelines and serve features in minutesAccess features with low latencyReduce costs and security risk by performing computations in your existing data platformEnsure data consistency between model training and inferencing# Get feature list from the catalogfeature_list=catalog.get_feature_list("200 Features on Active Customers")# Create deploymentdeployment=feature_list.deploy(name="Features for customer purchases next 2w",)# Activate deploymentdeployment.enable()# Get shell script template for online servingdeployment.get_online_serving_code(language="sh")ManageOrganize feature engineering assets with domain-specific catalogsCentralize cleaning operations and feature job configurationsDifferentiate features that are prototype versus production readyCreate new versions of your features to handle changes in dataKeep full lineage of your training data and features in productionMonitor the health of feature pipelines centrally# Get table from catalogitems_table=catalog.get_table("GROCERYITEMS")# Discount must not be negativeitems_table.Discount.update_critical_data_info(cleaning_operations=[fb.MissingValueImputation(imputed_value=0),fb.ValueBeyondEndpointImputation(type="less_than",end_point=0,imputed_value=0),])Get anoverview of the typical workflowin FeatureByte.Get started with Quick-Start and Deep-Dive TutorialsDiscover FeatureByte via its tutorials. All you need is to install the FeatureByte SDK.Install FeatureByte SDK with pip:pipinstallfeaturebyteNote: To avoid potential conflicts with other packages we strongly recommend using avirtual environmentor aconda environment.Sign up for access to the Hosted Tutorial serverhereand register your credentials with FeatureByte SDK:importfeaturebyteasfb# replace <api_token> with your API token you received after registeringfb.register_tutorial_api_token("<api_token>")This will create a "tutorial" profile that uses the hosted tutorial server.
You can nowdownloadand run notebooks from thetutorialssection.Leverage your data warehouseFeatureByte integrates seamlessly with yourSnowflake, Databricks, or Sparkdata warehouses, enhancing security and efficiency by bypassing large-scale outbound data transfers. This integration allows feature calculations to be performed within the data warehouse, leveraging scalability, stability, and efficiency.FeatureByte utilizes your data warehouse as a:data source.compute engine to leverage its scalability, stability, and efficiency.storage of partial aggregates (tiles) and precomputed feature values to support feature serving.ArchitectureThe FeatureByte platform comprises the following components:FeatureByte SDK(Python Package): Connects to the API service to provide feature authoring and management functionality through python classes and functions.FeatureByte Service(Docker Containers):API Service: REST-API service that validates and executes requests, queries data warehouses, and stores data.Worker: Executes asynchronous or scheduled tasks.MongoDB: Store metadata for created assets.Redis: Broker and queue for workers, messenger service for publishing progress updates.Query Graph Transpiler(Python Package): Construct data transformation steps as a query graph, which can be transpiled to platform-specific SQL.Source Tables(Data Warehouse): Tables used as data sources for feature engineering.Feature Store(Data Warehouse): Database that store data used to support feature serving.FeatureByte Service Deployment OptionsTheFeatureByte Servicecan be installed in three different modes:Local installation:The easiest way to get started with the FeatureByte SDK. It is a single-user installation that can be used to prototype features locally with your data warehouse.Hosted on a single server:A light-weight option to support collaboration and job scheduling with limited scalability and availability. Multiple users can connect to the service using the FeatureByte SDK, and deploy features for production.High availability installation (coming soon):The recommended way to run the service in production. Scale to a large number of users and deployed features, and provide highly available services.The FeatureByte Service runs onDockerfor the first two installation modes, and is deployed on aKubernetes Clusterfor the high availability installation mode.Refer to theinstallationsection of the documentation for more details.FeatureByte SDKThe FeatureByte Python SDK offers a comprehensive set of objects for feature engineering, simplifying the management and manipulation of tables, entities, views, features, feature lists and other necessary objects for feature serving.Catalogobjects help you organize your feature engineering assets per domain and maintain clarity and easy access to these assets.Entityobjects contain metadata on entity types represented or referenced by tables within your data warehouse.Tableobjects centralize key metadata about the table type, important columns, default cleaning operations and default feature job configurations.Viewobjects work like SQL views and are local virtual tables that can be modified and joined to other views to prepare data before feature definition.Featureobjects contain the logical plan to compute a feature in the form of a feature definition file.FeatureListobjects are collection of Feature objects tailored to meet the needs of a particular use case.Refer to theSDK overviewfor a complete list of the objects supported by the SDK and the SDK reference for more information about each object.Feature CreationThe SDK offers an intuitive declarative framework to create feature objects with different signal types, including timing, regularity, stability, diversity, and similarity in addition to the traditional recency, frequency and monetary types.ExamplesFeatures can be as simple as an entity’s attribute:customer_view=catalog.get_view("GROCERYCUSTOMER")# Extract operating system from BrowserUserAgent columncustomer_view["OperatingSystemIsWindows"]=\customer_view.BrowserUserAgent.str.contains("Windows")# Create a feature indicating whether the customer is using Windowsuses_windows=customer_view.OperatingSystemIsWindows.as_feature("UsesWindows")Features can be more complex such as aggregations over a window:invoice_view=catalog.get_view("GROCERYINVOICE")# Group invoices by the column GroceryCustomerGuid that references the customer entityinvoices_by_customer=invoice_view.groupby("GroceryCustomerGuid")# Declare features of total spent by customer over the past 7 days and 28 dayscustomer_purchases=invoices_by_customer.aggregate_over("Amount",method=fb.AggFunc.SUM,feature_names=["CustomerTotalSpent_7d","CustomerTotalSpent_28d"],fill_value=0,windows=['7d','28d'])To capture more complex signals, features can involve a series of joins and aggregates and be derived from multiple features:# Get items and product view from the catalogitems_view=catalog.get_view("INVOICEITEMS")product_view=catalog.get_view("GROCERYPRODUCT")# Join product view to items viewitems_view=items_view.join(product_view)# Get Customer purchases across product group in the past 4 weekscustomer_basket_28d=items_view.groupby(by_keys="GroceryCustomerGuid",category=”ProductGroup”).aggregate_over("TotalCost",method=fb.AggFunc.SUM,feature_names=["CustomerBasket_28d"],windows=['28d'])# Get customer view and join it to items viewcustomer_view=catalog.get_view("GROCERYCUSTOMER")items_view=items_view.join(customer_view)# Get Purchases of Customers living in the same state# across product group in the past 4 weeksstate_basket_28d=items_view.groupby(by_keys="State",category="ProductGroup").aggregate_over("TotalCost",method=fb.AggFunc.SUM,feature_names=["StateBasket_28d"],windows=['28d'])# Create a feature that measures the similarity of a customer purchases# and purchases of customers living in the same statecustomer_state_similarity_28d=\customer_basket_28d["CustomerBasket_28d"].cd.cosine_similarity(state_basket_28d["StateBasket_28d"])# save the new featurecustomer_state_similarity_28d.name=\"Customer Similarity with purchases in the same state over 28 days"customer_state_similarity_28d.save()Feature DefinitionOnce a feature is defined, you can obtain its feature definition file that is the source of truth and provides an explicit outline of the intended operations of the feature declaration, including those inherited but not explicitly declared by you.customer_state_similarity_28d.definitionRefer to the SDK reference for theFeatureobject for more information.Time TravelGreat Machine Learning models require great training data. Great training data require great features (columns) and great observation data points (rows). Great observation data points are historical data points that replicate the production environment. If predictions are expected to be any time, observation points-in-time should also be any time during a period covering at least one seasonal cycle.Observation data points are in FeatureByte in the form of observation sets that combine entity values and any past points-in-time you want to learn from.You can choose to get training data as a Pandas DataFrame or as a Table in the feature store that contains metadata on how the table was created for reuse or audit.# Get feature list from the catalogfeature_list=catalog.get_feature_list("200 Features on Active Customers")# Create a new feature list that includes a new featurecustomer_state_similarity_28d=catalog.get_feature("Customer Similarity with purchases in the same state over 28 days")new_feature_list=fb.FeatureList([feature_list,customer_state_similarity_28d],name="Improved feature list with Customer State similarity")# Get an observation set from the catalogobservation_set=catalog.get_observation_table("5M of active Customers in 2021-2022")# Compute training data and store it in the feature store for reuse and audittraining_table=new_feature_list.compute_historical_feature_table(observation_set,name="Improved Data to predict purchases next 2w with 2021-2022 history")# Download training data as a Pandas DataFrametraining_df=training_table.to_pandas()Refer to the SDK reference for theFeatureList,ObservationTableandHistoricalFeatureTableobjects, for more information.Deploy when neededOnce a feature list is deployed, the FeatureByte Service automatically orchestrates the pre-computation of feature values and stores them in an online feature store for online and batch serving.# Get feature list from the catalogfeature_list=catalog.get_feature_list("200 Features on Active Customers")# Check Feature objects are PRODUCTION_READY.# A readiness metric of 100% should be returned.print(feature_list.production_ready_fraction)# Create deploymentmy_deployment=feature_list.deploy(name="Deployment of 200 Features to predict customers purchases amount next 2 weeks",)# Activate deploymentmy_deployment.enable()Use the REST API service to retrieve feature values from the online feature store for online serving or use the SDK to retrieve batch of feature values from the online feature store for batch serving.Online ServingFor online serving, the Deployment object provides REST API service templates that can be used to serve features. Python or shell script templates for the REST API service are retrieved from the Deployment object.Get online scoring code as a Python script:deployment=catalog.get_deployment("Deployment of 200 Features to predict customers purchases amount next 2 weeks")deployment.get_online_serving_code(language="python")Get online scoring code as a Shell script:deployment.get_online_serving_code(language="sh")Batch ServingFor batch serving, the Deployment object is used to retrieve feature values for a batch request table containing entities values.fromdatetimeimportdatetimebatch_features=deployment.compute_batch_feature_table(batch_request_table=batch_request_table,batch_feature_table_name="Data to predict customers purchases next 2 w as of "+datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ'))# Download batch feature values as a Pandas DataFramebatch_features_df=batch_features.to_pandas()Refer to the SDK reference for theDeployment,BatchRequestTableandBatchFeatureTableobjects, for more information.ReleasesYou can see the list of available releases on theChange Logpage.
Releases are versioned using theSemantic Versionsspecification.LicenseThis project is licensed under the terms of theElastic License 2.0license. SeeLICENSEfor more details.ContributingAll contributions are welcomed. Please adhere to theCode of Conductand read theDeveloper's Guideto get started.