Package: feathr
An enterprise-grade, high-performance feature storeImportant Links:Slack&Discussions.Docs.What is Feathr?Feathr is the feature store that is used in production in LinkedIn for many years and was open sourced in April 2022. It is currently a project underLF AI & Data Foundation.Read our announcement onOpen Sourcing FeathrandFeathr on Azure, as well as theannouncement from LF AI & Data Foundation.Feathr lets you:Define featuresbased on raw data sources (batch and streaming) using pythonic APIs.Register and get features by namesduring model training and model inference.Share featuresacross your team and company.Feathr automatically computes your feature values and joins them to your training data, using point-in-time-correct semantics to avoid data leakage, and supports materializing and deploying your features for use online in production.🌟 Feathr HighlightsBattle tested in production for more than 6 years:LinkedIn has been using Feathr in production for over 6 years and have a dedicated team improving it.Scalable with built-in optimizations:For example, based on some internal use case, Feathr can process billions of rows and PB scale data with built-in optimizations such as bloom filters and salted joins.Rich support for point-in-time joins and aggregations:Feathr has high performant built-in operators designed for Feature Store, including time-based aggregation, sliding window joins, look-up features, all with point-in-time correctness.Highly customizable user-defined functions (UDFs)with native PySpark and Spark SQL support to lower the learning curve for data scientists.Pythonic APIsto access everything with low learning curve; Integrated with model building so data scientists can be productive from day one.Derived Featureswhich is a unique capability across all the feature store solutions. This encourage feature consumers to build features on existing features and encouraging feature reuse.Rich type systemincluding support for embeddings for advanced machine learning/deep learning scenarios. One of the common use cases is to build embeddings for customer profiles, and those embeddings can be reused across an organization in all the machine learning applications.Native cloud integrationwith simplified and scalable architecture, which is illustrated in the next section.Feature sharing and reuse made easy:Feathr has built-in feature registry so that features can be easily shared across different teams and boost team productivity.🏃 Getting Started with Feathr - Feathr SandboxThe easiest way to try out Feathr is to use theFeathr Sandboxwhich is a self-contained container with most of Feathr's capabilities and you should be productive in 5 minutes. To use it, simply run this command:# 80: Feathr UI, 8888: Jupyter, 7080: Interpretdockerrun-it--rm-p8888:8888-p8081:80-p7080:7080-eGRANT_SUDO=yesfeathrfeaturestore/feathr-sandbox:releases-v1.0.0And you can view default jupyter notebook:http://localhost:8888/lab/workspaces/auto-w/tree/local_quickstart_notebook.ipynbAfter running the Notebooks, all the features will be registered in the UI, and you can visit the Feathr UI at:http://localhost:8081🛠️ Install Feathr Client LocallyIf you want to install Feathr client in a python environment, use this:pipinstallfeathrOr use the latest code from GitHub:pipinstallgit+https://github.com/feathr-ai/feathr.git#subdirectory=feathr_project☁️ Running Feathr on Cloud for ProductionFeathr has native integrations with Databricks and Azure Synapse:Follow theFeathr ARM deployment guideto run Feathr on Azure. This allows you to quickly get started with automated deployment using Azure Resource Manager template.If you want to set up everything manually, you can checkout theFeathr CLI deployment guideto run Feathr on Azure. This allows you to understand what is going on and set up one resource at a time.Please read theQuick Start Guide for Feathr on Databricksto run Feathr with Databricks.Please read theQuick Start Guide for Feathr on Azure Synapseto run Feathr with Azure Synapse.📓 DocumentationFor more details on Feathr, read ourdocumentation.For Python API references, read thePython API Reference.For technical talks on Feathr, see theslides hereandhere. The recording ishere.🧪 SamplesNameDescriptionPlatformNYC Taxi DemoQuickstart notebook that showcases how to define, materialize, and register features with NYC taxi-fare prediction sample data.Azure Synapse, Databricks, Local SparkDatabricks Quickstart NYC Taxi DemoQuickstart Databricks notebook with NYC taxi-fare prediction sample data.DatabricksFeature EmbeddingFeathr UDF example showing how to define and use feature embedding with a pre-trained Transformer model and hotel review sample data.DatabricksFraud Detection DemoAn example to demonstrate Feature Store using multiple data sources such as user account and transaction data.Azure Synapse, Databricks, Local SparkProduct Recommendation DemoFeathr Feature Store example notebook with a product recommendation scenarioAzure Synapse, Databricks, Local Spark🔡 Feathr Highlighted CapabilitiesPlease readFeathr Full Capabilitiesfor more examples. Below are a few selected ones:Feathr UIFeathr provides an intuitive UI so you can search and explore all the available features and their corresponding lineages.You can use Feathr UI to search features, identify data sources, track feature lineages and manage access controls. Check out the latest live demohereto see what Feathr UI can do for you. Use one of following accounts when you are prompted to login:A work or school organization account, includes Office 365 subscribers.Microsoft personal account, this means an account can access to Skype, Outlook.com, OneDrive, and Xbox LIVE.For more information on the Feathr UI and the registry behind it, please refer toFeathr Feature RegistryRich UDF SupportFeathr has highly customizable UDFs with native PySpark and Spark SQL integration to lower learning curve for data scientists:defadd_new_dropoff_and_fare_amount_column(df:DataFrame):df=df.withColumn("f_day_of_week",dayofweek("lpep_dropoff_datetime"))df=df.withColumn("fare_amount_cents",df.fare_amount.cast('double')*100)returndfbatch_source=HdfsSource(name="nycTaxiBatchSource",path="abfss://feathrazuretest3fs@feathrazuretest3storage.dfs.core.windows.net/demo_data/green_tripdata_2020-04.csv",preprocessing=add_new_dropoff_and_fare_amount_column,event_timestamp_column="new_lpep_dropoff_datetime",timestamp_format="yyyy-MM-dd HH:mm:ss")Defining Window Aggregation Features with Point-in-time correctnessagg_features=[Feature(name="f_location_avg_fare",key=location_id,# Query/join key of the feature(group)feature_type=FLOAT,transform=WindowAggTransformation(# Window Aggregation transformationagg_expr="cast_float(fare_amount)",agg_func="AVG",# Apply average aggregation over the windowwindow="90d")),# Over a 90-day window]agg_anchor=FeatureAnchor(name="aggregationFeatures",source=batch_source,features=agg_features)Define features on top of other features - Derived Features# Compute a new feature(a.k.a. derived feature) on top of an existing featurederived_feature=DerivedFeature(name="f_trip_time_distance",feature_type=FLOAT,key=trip_key,input_features=[f_trip_distance,f_trip_time_duration],transform="f_trip_distance * f_trip_time_duration")# Another example to compute embedding similarityuser_embedding=Feature(name="user_embedding",feature_type=DENSE_VECTOR,key=user_key)item_embedding=Feature(name="item_embedding",feature_type=DENSE_VECTOR,key=item_key)user_item_similarity=DerivedFeature(name="user_item_similarity",feature_type=FLOAT,key=[user_key,item_key],input_features=[user_embedding,item_embedding],transform="cosine_similarity(user_embedding, item_embedding)")Define Streaming FeaturesRead theStreaming Source Ingestion Guidefor more details.Point in Time JoinsReadPoint-in-time Correctness and Point-in-time Join in Feathrfor more details.Running Feathr ExamplesFollow thequick start Jupyter Notebookto try it out. There is also a companionquick start guidecontaining a bit more explanation on the notebook.🗣️ Tech Talks on FeathrIntroduction to Feathr - Beginner's guideDocument Intelligence using Azure Feature Store (Feathr) and SynapseMLNotebook tutorial: Build a Product Recommendation Machine Learning Model with Feathr Feature StoreFeathr talk in Feature Store Summit⚙️ Cloud Integrations and ArchitectureFeathr componentCloud IntegrationsOffline store – Object StoreAzure Blob Storage, Azure ADLS Gen2, AWS S3Offline store – SQLAzure SQL DB, Azure Synapse Dedicated SQL Pools, Azure SQL in VM, SnowflakeStreaming SourceKafka, EventHubOnline storeRedis,Azure Cosmos DBFeature Registry and GovernanceAzure Purview, ANSI SQL such as Azure SQL ServerCompute EngineAzure Synapse Spark Pools, DatabricksMachine Learning PlatformAzure Machine Learning, Jupyter Notebook, Databricks NotebookFile FormatParquet, ORC, Avro, JSON, Delta Lake, CSVCredentialsAzure Key Vault🚀 RoadmapMore Feathr online client libraries such as JavaSupport feature versioningSupport feature monitoring👨‍👨‍👦‍👦 Community GuidelinesBuild for the community and build by the community. Check outCommunity Guidelines.📢 Slack ChannelJoin ourSlack channelfor questions and discussions (or click theinvitation link).