Package: feast-hive
Feast Hive SupportHive is not included in currentFeastroadmap, this project intends to add Hive support for Offline Store.For more details, can checkthis Feast issue.The public releases have passed all integration tests, please create an issue if you got any problem.Change LogsDONE [v0.1.1]I am working on the first workable version, think it will be released in a couple of days.DONE [v0.1.2]Allow custom hive conf when connect to a HiveServer2DONE [v0.14.0]Support Feast 0.14.xDONE [v0.17.0]Support Feast 0.17.0TODO It currently supportsinsert intofor uploading entity_df, which is a little inefficient, gonna add extra parameters for people who are able to provide HDFS address in next version (for uploading to HDFS).QuickstartInstall feastpipinstallfeastInstall feast-hiveInstall stable versionpipinstallfeast-hiveInstall develop version (not stable):pipinstallgit+https://github.com/baineng/feast-hive.gitCreate a feature repositoryfeastinitfeature_repocdfeature_repoEditfeature_store.yamlsetoffline_storetype to befeast_hive.HiveOfflineStoreproject:...registry:...provider:localoffline_store:type:feast_hive.HiveOfflineStorehost:localhostport:10000# optional, default is `10000`database:default# optional, default is `default`hive_conf:# optional, hive conf overlayhive.join.cache.size:14797hive.exec.max.dynamic.partitions:779...# other parametersonline_store:...Create Hive TableUploaddata/driver_stats.parquetto HDFShdfsdfs-copyFromLocal./data/driver_stats.parquet/tmp/Create Hive TableCREATETABLEdriver_stats(event_timestampbigint,driver_idbigint,conv_ratefloat,acc_ratefloat,avg_daily_tripsint,createdbigint)STOREDASPARQUET;Load data into the tableLOADDATAINPATH'/tmp/driver_stats.parquet'INTOTABLEdriver_stats;Editexample.py# This is an example feature definition filefromgoogle.protobuf.duration_pb2importDurationfromfeastimportEntity,Feature,FeatureView,ValueTypefromfeast_hiveimportHiveSource# Read data from Hive table# Here we use a Query to reuse the original parquet data,# but you can replace to your own Table or Query.driver_hourly_stats=HiveSource(# table='driver_stats',query="""SELECT Timestamp(cast(event_timestamp / 1000000 as bigint)) AS event_timestamp,driver_id, conv_rate, acc_rate, avg_daily_trips,Timestamp(cast(created / 1000000 as bigint)) AS createdFROM driver_stats""",event_timestamp_column="event_timestamp",created_timestamp_column="created",)# Define an entity for the driver.driver=Entity(name="driver_id",value_type=ValueType.INT64,description="driver id",)# Define FeatureViewdriver_hourly_stats_view=FeatureView(name="driver_hourly_stats",entities=["driver_id"],ttl=Duration(seconds=86400*1),features=[Feature(name="conv_rate",dtype=ValueType.FLOAT),Feature(name="acc_rate",dtype=ValueType.FLOAT),Feature(name="avg_daily_trips",dtype=ValueType.INT64),],online=True,batch_source=driver_hourly_stats,tags={},)Apply the feature definitionsfeastapplyGenerating training data and so onThe rest are as same asFeast QuickstartDeveloping and TestingDevelopinggitclonehttps://github.com/baineng/feast-hive.gitcdfeast-hive# creating virtual env ...pipinstall-e".[dev]"# before commitmakeformat
makelintTestingpipinstall-e".[test]"pytest-n6--host=localhost--port=10000--database=default