Package: bytehub
ByteHubAn easy-to-use feature store.💾 What is a feature store?A feature store is a data storage system for data science and machine-learning. It can storeraw dataand also transformedfeatures, which can be fed straight into an ML model or training script.Feature stores allow data scientists and engineers to bemore productiveby organising the flow of data into models.TheBytehub Feature Storeis designed to:Be simple to use, with a Pandas-like API;Require no complicated infrastructure, running on a local Python installation or in a cloud environment;Be optimised towards timeseries operations, making it highly suited to applications such as those in finance, energy, forecasting; andSupport simple time/value data as well as complex structures, e.g. dictionaries.It is built onDaskto support large datasets and cluster compute environments.🦉 FeaturesSearchablefeature informationandmetadatacan be stored locally using SQLite or in a remote database.Timeseries data is saved inParquet formatusing Dask, making it readable from a wide range of other tools. Data can reside either on a local filesystem or in acloud storage service, e.g. AWS S3.Supportstimeseries joins, along withfilteringandresamplingoperations to make it easy to load and prepare datasets for ML training.Feature engineering steps can be implemented astransforms. These are saved within the feature store, and allows for simple, resusable preparation of raw data.Time travelcan retrieve feature values based on when they were created, which can be useful for forecasting applications.Simple APIs to retrieve timeseries dataframes for training, or a dictionary of the most recent feature values, which can be used for inference.Also available as☁️ ByteHub Cloud: a ready-to-use, cloud-hosted feature store.📖 Documentation and tutorialsSee theByteHub documentationandnotebook tutorialsto learn more and get started.🚀 Quick-startInstall using pip:pipinstallbytehubCreate a local SQLite feature store by running:importbytehubasbhimportpandasaspdfs=bh.FeatureStore()Data lives insidenamespaceswithin each feature store. They can be used to separate projects or environments. Create a namespace as follows:fs.create_namespace('tutorial',url='/tmp/featurestore/tutorial',description='Tutorial datasets')Create afeatureinside this namespace which will be used to store a timeseries of pre-prepared data:fs.create_feature('tutorial/numbers',description='Timeseries of numbers')Now save some data into the feature store:dts=pd.date_range('2020-01-01','2021-02-09')df=pd.DataFrame({'time':dts,'value':list(range(len(dts)))})fs.save_dataframe(df,'tutorial/numbers')The data is now stored, ready to be transformed, resampled, merged with other data, and fed to machine-learning models.We can engineer new features from existing ones using thetransformdecorator. Suppose we want to define a new feature that contains the squared values oftutorial/numbers:@fs.transform('tutorial/squared',from_features=['tutorial/numbers'])defsquared_numbers(df):# This transform function receives dataframe input, and defines a transform operationreturndf**2# Square the inputNow both features are saved in the feature store, and can be queried using:df_query=fs.load_dataframe(['tutorial/numbers','tutorial/squared'],from_date='2021-01-01',to_date='2021-01-31')To connect to ByteHub Cloud, firstregister for an account, then use:fs=bh.FeatureStore("https://api.bytehub.ai")This will allow you to store features in your own private namespace on ByteHub Cloud, and save datasets to an AWS S3 storage bucket.🐾 RoadmapTasksto automate updates to features using orchestration tools likeAirflow