Package: lore
Lore is a python framework to make machine learning approachable for Engineers and maintainable for Data Scientists.FeaturesModels support hyper parameter search over estimators with a data pipeline. They will efficiently utilize multiple GPUs (if available) with a couple different strategies, and can be saved and distributed for horizontal scalability.Estimators from multiple packages are supported:Keras(TensorFlow/Theano/CNTK),XGBoostandSciKit Learn. They can all be subclassed with build, fit or predict overridden to completely customize your algorithm and architecture, while still benefiting from everything else.Pipelines avoid information leaks between train and test sets, and one pipeline allows experimentation with many different estimators. A disk based pipeline is available if you exceed your machines available RAM.Transformers standardize advanced feature engineering. For example, convert an American first name to its statistical age or gender using US Census data. Extract the geographic area code from a free form phone number string. Common date, time and string operations are supported efficiently through pandas.Encoders offer robust input to your estimators, and avoid common problems with missing and long tail values. They are well tested to save you from garbage in/garbage out.IO connections are configured and pooled in a standard way across the app for popular (no)sql databases, with transaction management and read write optimizations for bulk data, rather than typical ORM single row operations. Connections share a configurable query cache, in addition to encrypted S3 buckets for distributing models and datasets.Dependency Management for each individual app in development, that can be 100% replicated to production. No manual activation, or magic env vars, or hidden files that break python for everything else. No knowledge required of venv, pyenv, pyvenv, virtualenv, virtualenvwrapper, pipenv, conda. Ain’t nobody got time for that.Tests for your models can be run in your Continuous Integration environment, allowing Continuous Deployment for code and training updates, without increased work for your infrastructure team.Workflow Support whether you prefer the command line, a python console, jupyter notebook, or IDE. Every environment gets readable logging and timing statements configured for both production and development.Create a Lore projectThis example demonstrates nested transformers and how to use lore.io with a postgres databaseuserstable that has featurefirst_nameand responsehas_subscriptioncolumns. If you don’t want to create the database, you can follow a database free example app onmedium.$pipinstalllore$loreinitmy_app--python-version=3.6.4--keras--xgboost--postgres# fix up .env, config/database.cfg, circle.yml, README.rstA Cute Little ExampleWe’ll naively try to predict whether users are subscribers, given their first name.Update config/database.cfg to specify your database url:# config/database.cfg[MAIN]url:$DATABASE_URLyou can set environment variable for only the lore process with the .env file:# .envDATABASE_URL=postgres://localhost:5432/developmentCreate a sql file that specifies your data:-- my_app/extracts/subscribers.sqlSELECTfirst_name,has_subscriptionFROMusersLIMIT=%(limit)sPipelines are the unsexy, but essential component of most machine learning applications. They transform raw data into encoded training (and prediction) data for a model. Lore has several features to make data munging more palatable.# my_app/pipelines/subscribers.pyimportlore.ioimportlore.pipelines.holdoutfromlore.encodersimportNorm,Discrete,Boolean,Uniquefromlore.transformersimportNameAge,NameSex,LogclassHoldout(lore.pipelines.holdout.Base):defget_data(self):# lore.io.main is a Connection created by config/database.cfg + DATABASE_URL# dataframe() supports keyword args for interpolation (limit)# subscribers is the name of the extract# cache=True enables LRU query cachingreturnlore.io.main.dataframe(filename='subscribers',limit=100,cache=True)defget_encoders(self):# An arbitrairily chosen set of encoders (w/ transformers)# that reference sql columns in the extract by name.# A fair bit of thought will probably go into expanding# your list with features for your model.return(Unique('first_name',minimum_occurrences=100),Norm(Log(NameAge('first_name'))),Discrete(NameSex('first_name'),bins=10),)defget_output_encoder(self):# A single encoder that references the predicted outcomereturnBoolean('has_subscription')The superclasslore.pipelines.base.Holdoutwill take care of:splitting the data into training_data/validation_data/test_data dataframesfitting the encoders to training_datatransforming training_data/validation_data/test_data for the modelDefine some models that will fit and predict the data. Base models are designed to be extended and overridden, but work with defaults out of the box.# my_app/models/subscribers.pyimportlore.models.kerasimportlore.models.xgboostimportlore.estimators.kerasimportlore.estimators.xgboostfrommy_app.pipelines.subscribersimportHoldoutclassDeepName(lore.models.keras.Base):def__init__(self):super(DeepName,self).__init__(pipeline=Holdout(),estimator=lore.estimators.keras.BinaryClassifier()# a canned estimator for deep learning)classBoostedName(lore.models.xgboost.Base):def__init__(self):super(BoostedName,self).__init__(pipeline=Holdout(),estimator=lore.estimators.xgboost.Base()# a canned estimator for XGBoost)Test the models predictive power:# tests/unit/test_subscribers.pyimportunittestfrommy_app.models.subscribersimportDeepName,BoostedNameclassTestSubscribers(unittest.TestCase):deftest_deep_name(self):model=DeepName()# initialize a new modelmodel.fit(epochs=20)# fit to the pipeline's training_datapredictions=model.predict(model.pipeline.test_data)# predict the holdoutself.assertEqual(list(predictions),list(model.pipeline.encoded_test_data.y))# hah!deftest_xgboosted_name(self):model=BoostedName()model.fit()predictions=model.predict(model.pipeline.test_data)self.assertEqual(list(predictions),list(model.pipeline.encoded_test_data.y))# hah hah hah!Run tests:$loretestExperiment and tunenotebooks/with$lorenotebookusing the app kernelProject Structure├── .env.template            <- Template for environment variables for developers (mirrors production)
├── README.md                <- The top-level README for developers using this project.
├── requirements.txt         <- keeps dev and production in sync (pip)
├── runtime.txt              <- keeps dev and production in sync (pyenv)
│
├── data/                    <- query cache and other temp data
│
├── docs/                    <- generated from src
│
├── logs/                    <- log files per environment
│
├── models/                  <- local model store from fittings
│
├── notebooks/               <- explorations of data and models
│       └── my_exploration/
│            └── exploration_1.ipynb
│
├── appname/                 <- python module for appname
│   ├── __init__.py          <- loads the various components (makes this a module)
│   │
│   ├── api/                 <- external entry points to runtime models
│   │   └── my_project.py    <- hub endpoint for predictions
│   │
│   ├── extracts/            <- sql
│   │   └── my_project.sql
│   │
│   ├── estimators/          <- Code that make predictions
│   │   └── my_project.py    <- Keras/XGBoost implementations
│   │
│   ├── models/              <- Combine estimator(s) w/ pipeline(s)
│   │   └── my_project.py
│   │
│   └── pipelines/           <- abstractions for processing data
│       └── my_project.py    <- train/test/split data encoding
│
└── tests/
    ├── data/                <- cached queries for fixture data
    ├── models/              <- model store for test runs
    └── unit/                <- unit testsModules OverviewLore provides python modules to standardize Machine Learning techniques across multiple libraries.Core Functionalitylore.modelsare compatibility wrappers for your favorite library —Keras,XGBoost,SciKit Learn. They come with reasonable defaults for rough draft training out of the box.lore.pipelinesfetch, encode, and split data into training/test sets for models. A single pipeline will have one Encoder per feature in the model.lore.encodersoperate within Pipelines to transform a single feature into an optimal representation for learning.lore.transformersprovide common operations, like extracting the area code from a free text phone number. They can be chained together inside encoders. They efficientlySupporting functionalitylore.ioallows connecting to postgres/redshift and upload/download from s3lore.serializerspersist models with their pipelines and encoders (and get them back again)lore.storessave intermediate data, for reproducibility and efficiency.Utilitieslore.utilhas those extra niceties we rewrite in every project, and then somelore.envtakes care of ensuring that all dependencies are correctly installed before runningFeaturesIntegrated LibrariesUse your favorite library in a lore project, just like you’d use them in any other python project. They’ll play nicely together.Keras(TensorFlow/Theano/CNTK) +TensorboardXGBoostSciKit-LearnJupyter NotebookPandasNumpyMatplotlib,ggplot,plotnineSQLAlchemy,Psycopg2HubDev OpsThere aremany ways to manage python dependencies in development and production, and each has it’s own pitfalls. Lore codifies a solution that “just works” with lore install, which exactly replicates what will be run in production.Python compatibilityLore projects will always use the version of python specified in their runtime.txtLore projects use the system service manager (upstart on ubuntu) instead of supervisord which requires python 2.Heroku_ buildpack compatibility CircleCI_, Domino_)Lore supports runtime.txt to install and use a consistent version of python in both development and production.lore install automatically manages freezing requirements.txt, using a virtualenv, so pip dependencies are exactly the same in development and production. This includes workarounds to support correctly (not) freezing github packages in requirements.txtEnvironment Specific ConfigurationLore supports reading environment variables from .env, for easy per project configuration. We recommend .gitignore .env and checking in a .env.template for developer reference to prevent leaking secrets.logging.getLogger(__name__)is setup appropriately to console, file and/or syslog depending on environmentsyslog is replicated with structured data tologglyin productionlore.util.timer logs info in development, and records  tolibratoin productionException handling logs stack traces in development and test, but reports torollbarin productionlore console interactive python shell is color coded to prevent environmental confusionMultiple concurrent project compatibilityLore manages a distinct python virtualenv for each project, which can be installed from scratch in development with lore installBinary library installation for MAXIMUM SPEEDLore can buildtensorflowfrom source when it is listed in requirements for development machines, which results in a 2-3x runtime training performance increase. Use lore install –nativeLore also compilesxgbooston OS X with gcc-5 instead of clang to enable automatic parallelizationLore LibraryIOlore.io.connection.Connection.select()andConnection.dataframe()can be automatically LRU cached to diskConnectionsupports python %(name)s variable replacement in SQLConnectionstatements are always annotated with metadata for pgHeroConnectionis lazy, for fast startup, and avoids bootup errors in development with low connectivityConnectionsupports multiple concurrent database connectionsSerializationLore serializers provide environment aware S3 distribution for keras/xgboost/scikit modelsComing soon: heroku buildpack support for serialized models to marry the appropriate code for repeatable and deploys that can be safely rolled backCachingLore provides mulitple configurable cache types, RAM, Disk, coming soon: MemCached & RedisDisk cache is tested with pandas to avoid pitfalls encountered serializing w/ csv, h5py, pickleEncodersUniqueDiscreteQuantileNormTransformersAreaCodeEmailDomainNameAgeNameSexNamePopulationNameFamilialBase ModelsAbstract base classes for keras, xgboost, and scikit
- inheriting class to define data(), encoders(), output_encoder(), benchmark()
- multiple inheritance from custom base class w/ specific ABC for libraryprovides hyper parameter optimizationFittingEach call to Model.fit() saves the resulting model, along with the params to fit, epoch checkpoints and the resulting statistics, that can be reloaded, or uploaded with a SerializerKeras/Tensorflowtensorboard support out of the box with tensorboard –logdir=modelslore cleans up tensorflow before process exit to prevent spurious exceptionslore serializes Keras 2.0 models with extra care, to avoid several bugs (some that only appear at scale)ReloadBest callback early stops training on val_loss increase, and reloads the best epochUtilslore.util.timercontext manager writes to the log in development or librato in production*lore.util.timedis a decorator for recording function execution wall timeCommands$loreserver#  start an api process$loreconsole# launch a console in your virtual env$lorenotebook# launch jupyter notebook in your virtual env$lorefitMODEL#  train the model$loregenerate[scaffold,model,estimator,pipeline,notebook,test]NAME$loreinit[project]#  create file structure$loreinstall#  setup dependencies in virtualenv$loretest#  make sure the project is in working order$lorepip#  launch pip in your virtual env$lorepython# launch python in your virtual env