Package: ballet
balletAlightweight framework for collaborative, open-source data science
projects throughfeature engineering.Free software: MIT licenseDocumentation:https://ballet.github.io/balletHomepage:https://github.com/ballet/balletOverviewDo you develop machine learning models? Do you work by yourself or on a team?
Do you share notebooks or are you committing code to a shared repository? In
contrast to successful, massively collaborative, open-source projects like
the Linux kernel, the Rails framework, Firefox, GNU, or Tensorflow, most
data science projects are developed by just a handful of people. But think if
the open-source community could leverage its ingenuity and determination to
collaboratively develop data science projects to predict the incidence of
disease in a population, to predict whether vulnerable children will be evicted
from their homes, or to predict whether learners will drop out of online
courses.Our vision is to make collaborative data science possible by making it more
like open-source software development. Our approach is based on decomposing the
data science process into modular patches
that can then be intelligently combined, representing objects like "feature definition",
"labeling function", or "prediction task definition". Collaborators work in
parallel to write patches and submit them to a repo. The core Ballet framework
provides the underlying functionality to merge high-quality contributions,
collect modules from the file system, and compose the accepted contributions
into a single product. It also providesAssemblé, a familiar notebook-based development
experience that is friendly to data scientists and other inexperienced
open-source contributors. We don't require any computing infrastructure beyond
that which is commonly used in open-source software development.Currently, Ballet focuses on supporting collaboratively developingfeature engineering pipelines, an important part of many data science
projects. Individual feature definitions are represented as separate Python modules,
declaring the subset of a dataframe that they operate on and a
scikit-learn-style learned transformer that extracts feature values from the
raw data. Ballet collects individual feature definitions and composes them into a
feature engineering pipeline. At any point, a project built on Ballet can be
installed for end-to-end feature engineering on new data instances for the
same problem. How do we ensure the feature engineering pipeline is always
useful? Ballet thoroughly validates proposed feature definitions for correctness and
machine learning performance, using an extensive test suite and a novel
streaming feature definition selection algorithm. Accepted feature definitions can be
automatically merged by theBallet Botinto projects.Next stepsAre you a data owner or project maintainer that wants to organize a
collaboration?👉 Check out theBallet Maintainer GuideAre you a data scientist or enthusiast that wants to join a collaboration?👉 Check out theBallet Contributor GuideWant to learn about how Ballet enables Better Feature Engineering™️?👉 Check out theFeature Engineering GuideWant to see a demo collaboration in progress and maybe even participate yourself?👉 Check out theballet-predict-house-pricesprojectSource code organizationThis is a quick overview to the Ballet core source code organization. For more information about contributing to Ballet core itself, seehere.pathdescriptioncli.pytheballetcommand line utilityclient.pythe interactive client for userscontrib.pycollecting feature definitions from individual modules in source files in the file systemeng/base.pyabstractions for transformers used in feature definitions, such asBaseTransformereng/{misc,missing,ts}.pycustom transformers for missing data, time series problems, and moreeng/external.pyre-export of transformers from external libraries such as scikit-learn and feature_enginefeature.pytheFeatureabstractionpipeline.pytheFeatureEngineeringPipelineabstractionproject.pythe interface between a specific Ballet project and the core Ballet library, such as utilities to load project-specific information and theProjectabstractiontemplates/cookiecutter templates for creating a new Ballet project or creating a new feature definitiontemplating.pyuser-facing functionality on top of the templatestransformer.pywrappers for transformers that make them play nicely together in a pipelineupdate.pyfunctionality to update the project template from a new upstream releaseutil/various utilitiesvalidation/main.pyentry point for all validation routinesvalidation/base.pyabstractions used in validation such as theFeaturePerformanceEvaluatorvalidation/common.pycommon functionality used in validation, such as the ability to collect relevant changes between a current environment and a reference environment (such as a pull request vs a default branch)validation/entropy.pystatistical estimation routines used in feature definition selection algorithms, such as estimators for entropy, mutual information, and conditional mutual informationvalidation/feature_acceptance/validation routines for feature acceptancevalidation/feature_pruning/validation routines for feature pruningvalidation/feature_api/validation routines for feature APIsvalidation/project_structure/validation routines for project structureHistory0.19.5 (2021-07-17)Fix bug with deepcopyingballet.pipeline.FeatureEngineeringPipeline0.19.4 (2021-07-17)Fix bug with deepcopyingballet.eng.base.SubsetTransformer(#90)Addballet.drop_missing_targetsprimitive0.19.3 (2021-06-28)Support missing targets in discovery and feature performance evaluation (#89)Addninputsto summary statistics inballet.discovery.discover0.19.2 (2021-06-21)Improve discrete column detection in the case of many repeated valuesAddncontinuousandndiscreteto summary statistics inballet.discovery.discover0.19.1 (2021-06-20)Defer computation of some expensive summary statistics inballet.discovery.discover0.19.0 (2021-06-16)Support callable as feature input (#88)0.18.0 (2021-06-06)AddedConsumer GuideCan use Ballet together with MLBlocks to engineer features and then use additional preprocessing and ML components (#86)Can wrap the extracted feature matrix in a data frame with named columns derived fromfeature.outputorfeature.nameImplementedballet.encoder.EncoderPipelineto (mostly) mirrorballet.pipeline.FeatureEngineeringPipelineCan specify the dataset used for fitting the pipeline in the engineer-features CLI via--train-dir path/to/train/dir0.17.0 (2021-05-24)Support nested transformers, both with nested features and with input/transformer tuples wrapped with SubsetTransformers (#82)AllowClient.discoverto skip summary statistics if development dataset cannot be loaded or if features produce errors0.16.0 (2021-05-22)AddClient.discoverfunctionality (#80)Switch the order ofNullFillerparameters to more closely resemblefillnasignature0.15.2 (2021-05-14)Operate columnwise inVarianceThresholdAccepter, rather than computing the variance of
the entire feature group.0.15.1 (2021-05-12)Add debug logging for new accepters0.15.0 (2021-05-12)AddVarianceThresholdAccepter,MutualInformationAccepter, andCompoundAccepter(#76)0.14.0 (2021-05-11)Support using holdout data splits in validation (#75)Fix CLI program name in projects (#74)Fix bug withload_configusage in python REPL (#73)Reorganize external feature engineering primitives toballet/eng/external/**.py. Imports likefrom ballet.eng.external import MyPrimitiveare unaffected.0.13.1 (2021-04-02)Fix upgrade check inballet update-project-templateto migrate away from deprecated PyPI XML-RPC API.0.13.0 (2021-03-30)Fix links in project template0.12.0 (2021-03-10)Automate creation of GitHub repository in quickstart0.11.0 (2021-03-04)Allow validation to be run from topic branches locally0.10.0 (2021-02-23)AddProject.versionproperty0.9.0 (2021-02-16)Add support for managed branching viaballet start-new-feature --branching(defaults to enabled)Remove confusingballet.project.configattributeImplementballet.project.load_configas a better alternative, and use this in the project template'sload_data0.8.2 (2021-02-16)Fix bug withstr(t)orrepr(t)forDelegatingRobustTransformer0.8.1 (2021-02-16)Fix bug withstr(t)orrepr(t)forSimpleFunctionTransformer0.8.0 (2021-02-02)Fix bug with detecting updates to Ballet due to PyPI API outageFix some dependency conflictsReference ballet-assemble in project templateBump feature_engine to 1.00.7.11 (2020-09-16)Reduce verbosity of conversion approach logging by moving some messages to TRACE levelImplement "else" transformer forConditionalTransformerImprove GFSSF iteration logging0.7.10 (2020-09-08)Fix bug with different treatment of y_df and y; now, y_df is passed to the feature engineering pipeline, and y is passed to the feature validation routines as applicable.Switch back to using Gitter0.7.9 (2020-08-15)Add give_advice feature for FeatureAPICheck and other checks to log message on how to fix failureImprove logging of GFSSFAccepter and GFSSFPrunerImprove__str__for DelegatingRobustTransformer and consequently consumersChange default log format to SIMPLE_LOG_FORMATVarious bug fixes and improvements0.7.8 (2020-08-13)Add CanTransformNewRowsCheck to feature API checks0.7.7 (2020-08-12)SupportNoneas the transformer in aFeature, it will be automatically converted to anIdentityTransformerImplementColumnSelectorUpdate docsVarious bug fixes and improvements0.7.6 (2020-08-12)Re-export feature engineering primitives from various librariesShow type annotations in docsUpdate guidesVarious bug fixes and improvements0.7.5 (2020-08-03)Make validator parameters configurable in ballet.yml file (e.g. λ_1 and λ_2 for GFSSF algorithms)Support dynaconf 3.x0.7.4 (2020-07-22)Accept logger names, as well as logger instances, inballet.util.log.enableUpdated docs0.7.3 (2020-07-21)Addload_datamethod with built-in caching to project APIFix bug in GFSSF accepterAlways use encoded target during validationVarious bug fixes and improvements0.7.2 (2020-07-21)Add sample analysis notebook to project templateAdd binder url/badge to project templateFix bug with enabling logging with multiple loggers0.7.1 (2020-07-20)Add client for easy interactive usage (ballet.b)Add binder setup to project template0.7 (2020-07-17)Revamp project template: update project structure, create single API via FeatureEngineeringProject, use and add support for pyinvoke, revamp build into engineer_features, support repolockr botImprove ballet.project.Project: can create by ascending from given path, can create from current working directory, can resolve arbitrary project symbol, exposes project's APICheck for and notify of new release of ballet during project update (ballet update-project-template)Add ComputedValueTransformer to ballet.engMove stacklog to separate project and install itAdd validators that {never,always} accept submissionsAdd feature API checks to ensure that the feature can fit and transform a single rowAdd feature engineering guide to documentation and significantly expand contributor guideAdd bot installation instructions to maintainer guideAdd type annotations throughoutDrop support for py35, add support for py38Deprecate modeling codeVarious bug fixes and improvements0.6 (2019-11-12)Implement GFSSF validators and random validatorsImprove validators and allow validators to be configured in ballet.ymlImprove project templateCreate ballet CLIBug fixes and performance improvements0.5 (2018-10-14)Add project template and ballet-quickstart commandAdd project structure checks and feature API checksImplement multi-stage validation routine driver0.4 (2018-09-21)ImplementModelerfor versatile modeling and evaluationChange project name0.3 (2018-04-28)ImplementPullRequestFeatureValidatorAddutil.travis,util.modutil,util.gitutil modules0.2 (2018-04-11)ImplementArrayLikeEqualityTestingMixinImplementcollect_contrib_features0.1 (2018-04-08)First release on PyPI