Package: aligned
AlignedA data managment tool for ML applications.Similar to have DBT is a data managment tool for business analytics, will Aligned manage ML projects.
Therefore, Aligned makes it possible to collect data lineage between models, feature transformations etc. While also making it easy to reduce data leakage with point-in-time valid data and fix other problems described inSculley et al. [2015].DocsCheck out theAligned Docs, but keep in mind that they are still work in progress.Want to look at examples of how to usealigned?
View theMatsMoll/aligned-examplerepo.This is done by providing an new innovative way to describe how data flow in ML systems, and what our ML products produce. While also collecting dependency metadata that would otherwise be too inconvenient and error prone to manually type out.Therefore, you get the following:Data FreshnessData Quality AssuranceEasy Data LoadingLoad Form Multiple SourcesFeature StoreFeature ServerStream ProcessingModel Performance Monitoring - Documentation coming soonData Catalog - Documentation coming soonData Lineage - Documentation coming soonAll from the simple API of definingData SourcesFeature ViewsModelsAs a result, loading model features is as easy as:entities={"passenger_id":[1,2,3,4]}awaitstore.model("titanic").features_for(entities).to_pandas()Aligned is still in active development, so changes are likely.Feature ViewsWrite features as the should be, as data models.
Then get code completion and typesafety by referencing them in other features.This makes the features light weight, data source indipendent, and flexible.@feature_view(name="passenger",description="Some features from the titanic dataset",source=FileSource.csv_at("titanic.csv"),materialized_source=FileSource.parquet_at("titanic.parquet"),)classTitanicPassenger:passenger_id=Int32().as_entity()age=(Float().description("A float as some have decimals").lower_bound(0).upper_bound(110))name=String()sex=String().accepted_values(["male","female"])survived=Bool().description("If the passenger survived")sibsp=Int32().lower_bound(0).description("Number of siblings on titanic")cabin=String().is_optional()# Creates two one hot encoded valuesis_male,is_female=sex.one_hot_encode(['male','female'])Data sourcesAlinged makes handling data sources easy, as you do not have to think about how it is done.
Only define where the data is, and we handle the dirty work.Furthermore, you can also add materialised sources which can be used as intermediate sources.my_db=PostgreSQLConfig(env_var="DATABASE_URL")redis=RedisConfig(env_var="REDIS_URL")@feature_view(name="passenger",description="Some features from the titanic dataset",source=my_db.table("passenger",mapping_keys={"Passenger_Id":"passenger_id"}),materialized_source=my_db.with_schema("inter").table("passenger"),stream_source=redis.stream(topic="titanic"))classTitanicPassenger:passenger_id=Int32().as_entity()# Some features...Fast developmentMaking iterativ and fast exploration in ML is important. This is why Aligned also makes it super easy to combine, and test multiple sources.my_db=PostgreSQLConfig.localhost()aws_bucket=AwsS3Config(...)@feature_view(name="passengers",description="...",source=my_db.table("passengers"))classTitanicPassenger:passenger_id=Int32().as_entity()# Some features...# Change data sourcepassenger_view=TitanicPassenger.query()psql_passengers=awaitpassenger_view.all().to_pandas()aws_passengers=awaitpassenger_view.using_source(aws_bucket.parquet_at("passengers.parquet")).to_pandas()Describe ModelsUsually will you need to combine multiple features for each model.
This is where aModelcomes in.
Here can you define which features should be exposed.passenger=TitanicPassenger()location=LocationFeatures()@model_contract(name="titanic",features=[# aka. the model inputpassenger.constant_filled_age,passenger.ordinal_sex,passenger.sibsp,location.distance_to_shore,location.distance_to_closest_boat])classTitanic:# Referencing the passenger's survived feature as the targetdid_survive=passenger.survived.as_classification_target()Data FreshnessMaking sure a source contains fresh data is a crucial part to create propper ML applications.
Therefore, Aligned provides an easy way to check how fresh a source is.@feature_view(name="departures",description="Features related to the departure of a taxi ride",source=taxi_db.table("departures"),)classTaxiDepartures:trip_id=UUID().as_entity()pickuped_at=EventTimestamp()number_of_passengers=Int32()dropoff_latitude=Float().is_required()dropoff_longitude=Float().is_required()pickup_latitude=Float().is_required()pickup_longitude=Float().is_required()freshness=awaitTaxiDepartures.freshness_in_batch_source()iffreshness<datetime.now()-timedelta(days=2):raiseValueError("To old data to create an ML model")Access DataYou can easily create a feature store that contains all your feature definitions.
This can then be used to genreate data sets, setup an instce to serve features, DAG's etc.store=awaitFileSource.json_at("./feature-store.json").feature_store()# Select all features from a single feature viewdf=awaitstore.all_for("passenger",limit=100).to_pandas()Centraliced Feature Store DefinitionYou would often share the features with other coworkers, or split them into different stages, likestaging,shadow, orproduction.
One option is therefore to reference the storage you use, and load theFeatureStorefrom there.aws_bucket=AwsS3Config(...)store=awaitaws_bucket.json_at("production.json").feature_store()# This switches from the production online store to the offline store# Aka. the batch sources defined on the feature viewsexperimental_store=store.offline_store()This json file can be generated by runningaligned apply.Select multiple feature viewsdf=awaitstore.features_for({"passenger_id":[1,50,110]},features=["passenger:scaled_age","passenger:is_male","passenger:sibsp""other_features:distance_to_closest_boat",]).to_polars()Model ServiceSelecting features for a model is super simple.df=awaitstore.model("titanic_model").features_for({"passenger_id":[1,50,110]}).to_pandas()Feature ViewIf you want to only select features for a specific feature view, then this is also possible.prev_30_days=awaitstore.feature_view("match").previous(days=30).to_pandas()sample_of_20=awaitstore.feature_view("match").all(limit=20).to_pandas()Data qualityAlinged will make sure all the different features gets formatted as the correct datatype.
In addition will aligned also make sure that the returend features aligne with defined constraints.@feature_view(...)classTitanicPassenger:...age=(Float().is_required().lower_bound(0).upper_bound(110))sibsp=Int32().lower_bound(0,is_inclusive=True)Then since our feature view have ais_requiredand alower_bound, will the.validate(...)command filter out the entites that do not follow that behavior.fromaligned.validation.panderaimportPanderaValidatordf=awaitstore.model("titanic_model").features_for({"passenger_id":[1,50,110]}).validate(PanderaValidator()# Validates all features).to_pandas()Feature ServerYou can define how to serve your features with theFeatureServer. Here can you define where you want to load, and potentially write your features to.By default will italignedlook for a file calledserver.py, and aFeatureServerobject calledserver. However, this can be defined manually as well.fromalignedimportRedisConfig,FileSourcefromaligned.schemas.repo_definitionimportFeatureServerstore=FileSource.json_at("feature-store.json")server=FeatureServer.from_reference(store,RedisConfig.localhost())Then runaligned serve, and a FastAPI server will start. Here can you push new features, which then transforms and stores the features, or just fetch them.Stream WorkerYou can also setup stream processing with a similar structure. However, here will aStreamWorkerbe used.by default willalignedlook for aworker.pyfile with an object calledworker. An example would be the following.fromalignedimportRedisConfig,FileSourcefromaligned.schemas.repo_definitionimportFeatureServerstore=FileSource.json_at("feature-store.json")server=FeatureServer.from_reference(store,RedisConfig.localhost())