Package: fexpress
FeatureExpress: Time-aware Feature Engineering LibraryOverviewFeatureExpress is a groundbreaking in-memory feature engineering library designed for processing time-based event data. It is a hybrid between a feature engineering library and a feature store, aiming to address the complex challenges of dealing with temporal data in machine learning applications.Website:https://feature.expressDiscord:Prerelease:warning:Alpha Release Warning:This library is currently in analphastage. As such, it is subject to:Changes:The API is still evolving, so you can expect many breaking changes. If you depend on this library in your project, be prepared to update your code as new versions are released.Performance Issues:There may be inefficiencies or other performance issues that have not yet been resolved.Unstable API:Functionality might be added, changed, or removed without notice. Documentation may be incomplete or out of date.This version is more like apre-release, and it's primarily intended for developers who are interested in experimenting with the latest features or contributing to the project.Why FeatureExpress?Why Another Feature Engineering Library / Feature Store?The necessity of this unique library grew from years of struggling with event-driven data, especially in customer interactions and recommendations. Time adds complexity, subtlety, and depth to data analysis and modeling. The challenges include:Time makes everything complex.Model validation becomes harder.Data leaks are subtle and hard to trace.SQL-like operations are prone to errors and hard to write.Existing feature stores move the burden of materialization to data scientists.Event Data for Superior FeaturesEvent data encapsulates reality with timestamped information, and it's pivotal in creating meaningful features. Unlike other methods that often obscure temporal aspects, FeatureExpress utilizes a dedicated data structure to make the connection between events and features clearer and more explicit.Overcoming Problems with Current Feature StoresCurrent feature stores often rely on explicit materialization and caching, leading to increased complexity for data scientists. FeatureExpress adopts a declarative approach (similar to SQL) with a DSL (Domain Specific Language) to define features, allowing for a more intuitive and error-free process.In-Memory ProcessingBuilt in Rust and interfaced in Python, FeatureExpress leverages in-memory processing to enable:Fast materialization of features.Parallel computations for efficiency.Flexibility to expand to more permanent storage solutions in the future.Though the current version is limited to datasets that fit in memory, FeatureExpress's performance and robustness make it a valuable tool for data scientists and engineers working with time-series data.InstallationYou can install FeatureExpress via pip:pipinstallfexpressFeaturesEvent-Driven Design:Utilizes events as core data structures for accurate modeling.Time-Aware DSL:Introduces a SQL-like DSL for expressive and complex feature declarations.No Data Leaks:The clear separation between past and future guarantees against inadvertent data leaks.Flexible Observation Dates:Allows custom definitions of observation dates including intervals, fixed, conditional, and more.Time-based Joins:Enables complicated joins in time, like aggregations over specific periods.Optimized Performance:Implements performance tricks like partial aggregates for efficient calculations.Rich Value Representation:Accommodates various data types for broad applications.Indices and In-memory Store:Ensures optimized querying and manipulation of time-based data.DocumentationFull documentation, including tutorials and examples, can be found athttps://feature.express.ContributingInterested in contributing to FeatureExpress? See ourCONTRIBUTING.mdfor guidelines on how to help!LicenseFeatureExpress is under MIT. SeeLICENSEfor more details.Developmentenv VIRTUAL_ENV=$(python3 -c 'import sys; print(sys.base_prefix)') maturin developormaturin developdevelopment (optimized code)maturin develop --releasebuilding Python wheelmaturin build --release -i pythonThis should create a wheel intarget/wheelsinstalling Python wheelpip install target/wheels/fexpress_rs-0.1.0-cp38-cp38-linux_x86_64.whl -UNote that the file name can be different depending on your system.Docker stuffdocker build -t rust-python-maturin .
docker run -rm -v $(pwd)/artifacts:/app/artifacts rust-python-maturin bash -c "make python_debug_docker && make python_profile_docker"