Package: pandioml
PandioML - Pandio.com Machine LearningThis repository contains the PandioML Python library to develop and deploy machine learning for streaming data.At a high level, PandioML provides three things:PipelinesTraditional pipelines based onscikit-learnpipelines.DatasetsConnect and stream any source of data into a pipeline.Automation Of Pipelines & DatasetsUsepandioclito automate locally or automate on the Pandio.com platform powered byApache PulsarandPresto/Trino.Quick LinksPandio.com/PandioML-Pandio.com-Getting Started-Quick Start-PyPi PandioML-PyPi PandioCLIAbout PandioMLIncremental LearningAs data is streamed in, stream learning models are created incrementally and updated continously. They are extremely powerful in a real-time environment, but can also be a viable replacement for traditional machine learning.Adaptive LearningData distribution changes have far less impact from concept drift on a stream learning model due to the adaptive nature of incremental learning.Resource EfficientDue to the incremental learning, considerably less processing time and memory are needed to train a model.Easy To UsePandioML is based on scikit-learn, making it easy to pickup.Open SourceBased on powerful open source technology itself, PandioML is released underSSPL License.AI OrchestrationNot just a Python library, PandioCLI makes it easy to package and deploy PandioML pipelines and models to the Pandio platform.Use CasesSupervised LearningWhen working with labeled data. Depending on the target type can be either classification (discrete values) or regression (continuous values)Single & Multiple OutputSingle-output methods predict a single target-label (binary or multi-class) for classification or a single target-value for regression. Multi-output methods simultaneously predict multiple variables given an input.Concept Drift DetectionChanges in data distribution can harm learning. Drift detection methods are designed to rise an alarm in the presence of drift and are used alongside learning methods to improve their robustness against this phenomenon in evolving data streams.Unsupervised LearningWhen working with unlabeled data. For example, anomaly detection where the goal is the identification of rare events or samples which differ significantly from the majority of the data.Building PipelinesScikit-learn pipelines on steroids! Familiar syntax with more helper methods to make building and debugging pipelines easier.Deploying ModelsAfter writing (or re-using) your pipeline and model, deploy to Pandio's AI Orchestration platform with a single command. Your model is now available for production use at any scale!Our PhilosophyPandioML exists to provide a full featured experience from accessing data, developing a pipeline, and deploying to production as quickly as possible.Our focus is on ease-of-use, quick iterations, and single command deploy to production. The library is meant to automate everything except the actual data science, to optimize pipeline development. For the actual data science, the focus is on accessibility. 40+ algorithms, dozens of metrics and stats, and a powerful pipeline framework save significant time.The last core idea to PandioML is to keep it as low level as possible. All too often libraries try to abstract things away to make things simple, but in our experience it only complicates things. It might make a few things easier, or negate the need to write a little more code, but when debugging or auditing is needed, it becomes a real pain to dive into what is happening.This library was meant to be for everyone, from zero experience to experts in the field.About Pandio.comPandio is an AI Orchestration platform that helps companies accelerate their AI initiatives. Connect data, build pipelines, train models, and deploy models on your terms at unmatched speed and scale.RequirementsPython 3.5 - 3.8PIP > 20.0.0Docker (or Docker Desktop)How It WorksPandioML is built on the Pandio.com AI Orchestration platform. For the purposes of PandioML, this means that two things support the deployment of a PandioML project.Distributed MessagingThis is the concept of a stream, queue, or pubsub providing input topic(s) and output topic(s). A PandioML project sits inbetween the input topic(s) and the output topic(s).This means to receive a prediction, a message must be placed on an input topic. The result is then placed on an output topic, which are defined in the projectconfig.pyfile.Serverless FunctionsThis is a compute framework that runs the PandioML project in a Kubernetes environment.When you build machine learning with PandioML, you automatically get a fully scalable production environment with each deployment of a project.As you work locally, these components are simulated to allow quick iteration before deploying the project to production with a single command.This is one of the most powerful features of PandioML. Your local development on training, iterating, evaluating, etc. is the same pipelines that you'll deploy as a microservice to Pandio.com. You don't have to worry about packaging it as a microservice, using the PandioML and PandioCLI tools does all of that for you.GuidesGetting StartedQuick StartPipelinesDatasetsMetricsStatsSchemas and Schema RegistryFull Form Fraud ExampleFull Movie Ratings ExampleFull Credit Card Fraud Pipeline ExampleFull Credit Card Fraud Dataset ExampleHow To Load A ModelHyperparameter TuningFull Lead Scoring Pipeline ExampleFull Lead Scoring Dataset ExamplePandioMLInstallationpip install pandiomlTo run pipelines locally, Docker is required.Docker removes the need to install complex data science environments on your computer.Get it free here:https://www.docker.com/products/docker-desktopDocumentationModelThepandioml.model.*module handles all of the available algorithms and models.ModuleDescriptionGaussianNBGaussian Naive BayesMultinomialNBNaive Bayes classifier for multinomial models.ComplementNBNaive Bayes classifier for multinomial models.BernoulliNBBernoulli Naive Bayes.K-meansIncremental k-means.ADWINBaggingClassifierADWIN Bagging classifier.BaggingClassifierOnline bootstrap aggregation for classification.BaggingRegressorOnline bootstrap aggregation for regression.AdaBoostClassifierBoosting for classificationSRPClassifierStreaming Random Patches ensemble classifier.EpsilonGreedyRegressorEpsilon-greedy bandit algorithm for regression.UCBRegressorUpper Confidence Bound bandit for regression.EWARegressorExponentially Weighted Average regressor.SuccessiveHalvingClassifierSuccessive halving algorithm for classification.SuccessiveHalvingRegressorSuccessive halving algorithm for regression.StackingClassifierStacking for binary classification.FFMClassifierField-aware Factorization Machine for binary classification.FFMRegressorField-aware Factorization Machine for regression.FMClassifierFactorization Machine for binary classification.FMRegressorFactorization Machine for regression.FwFMClassifierField-weighted Factorization Machine for binary classification.FwFMRegressorField-weighted Factorization Machine for regression.HOFMClassifierHigher-Order Factorization Machine for binary classification.HOFMRegressorHigher-Order Factorization Machine for regression.HardSamplingClassifierHard sampling classifier.HardSamplingRegressorHard sampling regressor.RandomOverSamplerRandom over-sampling.RandomSamplerRandom sampling by mixing under-sampling and over-sampling.RandomUnderSamplerRandom under-sampling.HoeffdingTreeClassifierHoeffding Tree or Very Fast Decision Tree classifier.HoeffdingAdaptiveTreeClassifierHoeffding Adaptive Tree classifier.ExtremelyFastDecisionTreeClassifierExtremely Fast Decision Tree classifier.LabelCombinationHoeffdingTreeClassifierLabel Combination Hoeffding Tree for multi-label classification.HoeffdingTreeRegressorHoeffding Tree regressor.HoeffdingAdaptiveTreeRegressorHoeffding Adaptive Tree regressor.iSOUPTreeRegressorIncremental Structured Output Prediction Tree (iSOUP-Tree) for multi-target regression.StackedSingleTargetHoeffdingTreeRegressorStacked Single-target Hoeffding Tree regressor.KNNClassifierk-Nearest Neighbors classifier.KNNADWINClassifierK-Nearest Neighbors classifier with ADWIN change detector.SAMKNNClassifierSelf Adjusting Memory coupled with the kNN classifier.KNNRegressork-Nearest Neighbors regressor.AccuracyWeightedEnsembleClassifierAccuracy Weighted Ensemble classifierAdaptiveRandomForestClassifierAdaptive Random Forest classifier.AdaptiveRandomForestRegressorAdaptive Random Forest regressor.AdditiveExpertEnsembleClassifierAdditive Expert ensemble classifier.BatchIncrementalClassifierBatch Incremental ensemble classifier.ClassifierChainClassifier Chains for multi-label learning.ProbabilisticClassifierChainProbabilistic Classifier Chains for multi-label learning.MonteCarloClassifierChainMonte Carlo Sampling Classifier Chains for multi-label learning.DynamicWeightedMajorityClassifierDynamic Weighted Majority ensemble classifier.LearnPPNSEClassifierLearn++.NSE ensemble classifier.LearnPPClassifierLearn++ ensemble classifier.LeveragingBaggingClassifierLeveraging Bagging ensemble classifier.MultiOutputLearnerMulti-Output Learner for multi-target classification or regression.OnlineAdaC2ClassifierOnline AdaC2 ensemble classifier.OnlineBoostingClassifierOnline Boosting ensemble classifier.OnlineCSB2ClassifierOnline CSB2 ensemble classifier.OnlineRUSBoostClassifierOnline RUSBoost ensemble classifier.OnlineSMOTEBaggingClassifierOnline SMOTEBagging ensemble classifier.OnlineUnderOverBaggingClassifierOnline Under-Over-Bagging ensemble classifier.OzaBaggingClassifierOza Bagging ensemble classifier.OzaBaggingADWINClassifierOza Bagging ensemble classifier with ADWIN change detector.RegressorChainRegressor Chains for multi-output learning.StreamingRandomPatchesClassifierStreaming Random Patches ensemble classifier.ADWINAdaptive Windowing method for concept drift detection.DDMDrift Detection Method.EDDMEarly Drift Detection Method.HDDM_ADrift Detection Method based on Hoeffding’s bounds with moving average-test.HDDM_WDrift Detection Method based on Hoeffding’s bounds with moving weighted average-test.KSWINKolmogorov-Smirnov Windowing method for concept drift detection.PageHinkleyPage-Hinkley method for concept drift detection.EvaluateHoldoutThe holdout evaluation method or periodic holdout evaluation method.EvaluatePrequentialThe prequential evaluation method or interleaved test-then-train method.EvaluatePrequentialDelayedThe prequential evaluation delayed method.VeryFastDecisionRulesClassifierVery Fast Decision Rules classifier.RobustSoftLearningVectorQuantizationRobust Soft Learning Vector Quantization for Streaming and Non-Streaming Data.HalfSpaceTreesImplementation of the Streaming Half–Space–Trees (HS–Trees)DataThepandioml.data.*model contains all of the datasets and generators available.ModuleDescriptionSchemaLabeledFormSubmissionGeneratorUses the Faker Python package to generate an infinite amount of form submissions.schemaNoWebHostingDatasetContains 12,496,728 server resource metric events recorded over a 3 month period of time.schemaNoPersonProfileDatasetGenerates an infinite stream of user Profiles using the Faker Python library.schemaNoCreditCardFraudA dataset of 1,296,675 credit card transactions with a percentage labeled as fraud.schemaYesAgrawalGeneratorDatasetA generator for data regarding home loan applications with the ability to balance and add noise.schemaYesSineGeneratorDatasetA generator for data regarding sine values with the ability to balance and add noise.schemaYesLEDGeneratorDatasetA generator for data regarding a digit displayed on a seven-segment LED display with the ability to add noise.schemaYesPhishingDataset1250 entries of webpages that are classified as phishing or not.schemaYesMovieRatingDataset100,000 movie ratings from different types of individuals.schemaYesRestaurantVisitorsDatasetThis dataset contains 252,108 records over a 16 week period to 829 Japanese Restaurants.schemaYesAdditional custom datasets can be created using thepandiocli dataset generatetool to use your own data.MetricsThepandioml.metrics.*contains helper methods to calculate metrics relating to the pipelines.ModuleDescriptionAccuracyAccuracy score, which is the percentage of exact matches.BalancedAccuracyBalanced accuracy.BinaryMetricMother class for all binary classification metrics.ClassificationMetricMother class for all classification metrics.ClassificationReportA report for monitoring a classifier.CohenKappaCohen's Kappa score.CrossEntropyMulticlass generalization of the logarithmic loss.ExactMatchExact match score.ExamplePrecisionExample-based precision score for multilabel classification.ExampleRecallExample-based recall score for multilabel classification.ExampleF1Example-based F1 score for multilabel classification.ExampleFBetaExample-based F-Beta score.F1Binary F1 score.FBetaBinary F-Beta score.GeometricMeanGeometric mean score.HammingHamming score.HammingLossHamming loss score.JaccardJaccard index for binary multi-outputs.KappaMKappa-M score.KappaTKappa-T score.LogLossBinary logarithmic loss.MAEMean absolute error.MacroF1Macro-average F1 score.MacroFBetaMacro-average F-Beta score.MacroPrecisionMacro-average precision score.MacroRecallMacro-average recall score.MCCMatthews correlation coefficient.MetricMother class for all metrics.MetricsA container class for handling multiple metrics at once.MicroF1Micro-average F1 score.MicroFBetaMicro-average F-Beta score.MicroPrecisionMicro-average precision score.MicroRecallMicro-average recall score.MultiClassMetricMother class for all multi-class classification metrics.MultiFBetaMulti-class F-Beta score with different betas per class.MultiOutputClassificationMetricMother class for all multi-output classification metrics.MultiOutputRegressionMetricMother class for all multi-output regression metrics.MSEMean squared error.PrecisionBinary precision score.RecallBinary recall score.RegressionMetricMother class for all regression metrics.RegressionMultiOutputWrapper for multi-output regression.RMSERoot mean squared error.RMSLERoot mean squared logarithmic error.ROCAUCReceiving Operating Characteristic Area Under the Curve.RollingWrapper for computing metrics over a window.R2Coefficient of determination ($R^2$) scoreSMAPESymmetric mean absolute percentage error.TimeRollingWrapper for computing metrics over a period of time.WeightedF1Weighted-average F1 score.WeightedFBetaWeighted-average F-Beta score.WeightedPrecisionWeighted-average precision score.WeightedRecallWeighted-average recall score.WrapperMetricMetric WrapperBallHallBall-Hall indexBICBayesian Information Criterion (BIC).CalinskiHarabaszCalinski-Harabasz index (CH).CohesionMean distance from the points to their assigned cluster centroids. The smaller the better.DaviesBouldinDavies-Bouldin index (DB).GD43Generalized Dunn's index 43 (GD43).GD53Generalized Dunn's index 53 (GD53).HartiganHartigan Index (H - Index)IIndexI-Index (I).InternalMetricMother class of all internal clustering metrics.MSSTDMean Squared Standard Deviation.PSPartition Separation (PS).CR2R-SquaredRMSSTDRoot Mean Squared Standard Deviation.SDThe SD validity index (SD).SeparationAverage distance from a point to the points assigned to other clusters.SilhouetteSilhouette coefficient [^1], roughly speaking, is the ratio between cohesion and the average distances from the points to their second-closest centroid.SSBSum-of-Squares Between Clusters (SSB).SSWSum-of-Squares Within Clusters (SSW).XieBeniXie-Beni index (XB).WBWB IndexXuXu IndexStatisticsThepandioml.stats.*contains helper methods to calculate metrics relating to the pipelines.ModuleAbsMaxAutoCorrBayesianMeanBivariateCountCovEWMeanEWVarEntropyIQRKurtosisLinkMaxMeanMinModeNUniquePeakToPeakPearsonCorrQuantileRollingAbsMaxRollingCovRollingIQRRollingMaxRollingMeanRollingMinRollingModeRollingPeakToPeakRollingPearsonCorrRollingQuantileRollingSEMRollingSumRollingVarSEMShiftSkewSumUnivariateVarCreate Your Own Dataset or GeneratorUse thepandioml.data.Streamclass, inherit it, define the required methods, and use your own data inside of PandioML!To make it available on Pandio's platform,pandiocli uploadit using the PandioCLI, thenpandiocli deployit.Build Streaming PipelinesThepandioml.core.pipelinescontains a pipeline framework to build traditional machine learning pipelines.The PandioML framework for building pipelines is similar toscikit-learn's, but differs in that the pipeline is meant to process a single record, instead of a batch of records.Click here to read our detailed Pipelines guide.ArtifactsReproducible pipelines are a critically important part of data science.PandioML provides a tool to store any artifact associated with a pipeline.3 artifacts are automatically stored for you:DatasetPipelineModelIf you'd like to store additional things, such as metrics, charts, hypertuning parameters, or anything else, it can be done by importing the artifact module like so:from pandioml.core.artifacts import artifactThen, call theaddmethod of theartifactas follows:dict = artifact.add('config_params', {'foo': 'bar'})The first parameter is a unique name for it, the second is the item to be stored. If it can be pickled, it can be stored as an artifact. In addition, there are times when it is preferred to defer the artifact, so a callable is also acceptable as the item to be stored. When called, a single argument is passed calledstorage_locationthat contains where the artifacts will be stored. This makes it easy to provide custom logic when storing artifacts.Bonus, the method will return the artifact, so that you can easily add items and define things in a single line.Additionally, asavemethod exists that can be manually called to store artifacts. The storage medium used to store these artifacts can also be extended. Currently, File and AWS S3 storage backends are supported. The default storage backend is File.This method is called automatically when the function stops running. You may call it yourself, but not too often as it is an expensive operation depending on the size of the artifacts.ARTIFACT_STORAGEinside ofconfig.pydefines where the artifacts are stored. This can be overwritten through the call tosaveas well.Schema Registry SupportType safety is critically important to any data science initiative. If you can't rely on a float being a float, that can have dangerous consequences.PandioML treats type safety as a first class citizen. Before data is streamed into your pipeline, you can require that it validates against a defined schema. If it fails validation, your pipeline will never see it.Type safety in PandioML prevents potential disastrous situations and makes pipeline development more efficient and productive.To give an example, the following class defines a schema that the data must follow. Additionally, the data comes in as an object built right from the schema.class Transaction(Record):
    trans_date_trans_time = String()
    cc_num = Integer()
    merchant = String()
    category = String()
    amt = Float()
    first = String()
    last = String()
    gender = String()
    street = String()
    city = String()
    state = String()
    zip = Integer()
    lat = Float()
    long = Float()
    city_pop = Integer()
    job = String()
    dob = String()
    trans_num = String()
    unix_time = Integer()
    merch_lat = Float()
    merch_long = Float()
    is_fraud = Integer()
    weekday = Integer()
    weekend = Integer()
    month = Integer()
    day = Integer()
    hour = Integer()Using the above as an example, as data is streamed into your pipeline, it comes in as a Transaction object.This allows you to quickly access properties like so:event.cc_numData science is so much easier with type safety offered by schemas!The Schema Registry is provided by the Pandio.com platform. For most use cases, the inbound data is automatically handled for most implementations of PandioML.The outbound data, or the prediction from PandioML, must be packaged up in a schema which is then consumed by the service that needs the prediction. It is just as important for this to have a schema as is the data sent into PandioML.To see an example of returning an object with a schema, see this example:Restaurant VisitsThis example shows a schema being created, then data being put into the object, and then the object being returned. This object is then sent to the output topic.Debugging With PandioML Interactive SessionsHave you ever asked yourself one of these questions?What in the heck happened?What does that error mean?That value is not right... what part of my code did that?My pipeline does not work, what is the problem?If you have, you're not alone. If you haven't, feel free to skip this section!To help find the problem and fix it quickly, PandioML has the ability to start an interactive session from any point in your code. This will halt execution and open up an interactive Python session that lets you inject code into your code execution in real time. No need to add a bunch of print statements to figure out what is happening!To start an interactive session, import the interactive method from here:from pandioml.core import interactThen call it like so:interact(banner='Helpful Note Where This Was Triggered', local=locals())When you run your code, it will halt where this method is called, letting you inject any Python code you would like into the execution.Withlocal, you can change scope of the variables, and accesslocal=globals()if you'd like.You can also add many of these calls all throughout your code. When you're ready to continue the execution of your code, simply typeCTRL + D.Happy debugging!Examples./examples/form_fraudThis example combines a NaiveBayes model, with the FormSubmissionGenerator, and a pipeline to demonstrate how to predict whether an email is from Yahoo.com or Hotmail.com./examples/restaurant_visitsThis example combines a LinearRegression model, with the RestaurantVisitorsDataset, and a pipeline to demonstrate how to predict how many visits a restaurant will receive.function.pyThis is the core of PandioML. Every project starting point is thefunction.pyfile. All of the helper methods, magic sauce, embedded packages, cli tools, etc. exist to help build functions that run on the Pandio platform. In each of these functions is typically a model that makes accurate predictions. What data it uses, the algorithm, feature extractions, fitting, predicting, pipelining, and even labeling, is completely up to you. The function is your sandbox, where all the fun begins. Everything in PandioML is meant to help make creating value with machine learning easier.Only two things are required to be defined in the function file generated for each project, themodelyou'd like to use, and thepipelinesyou'd like to execute against each individually streamed event.Any number of classes or methods may be created to assist in the building of the function.Thepandioml.function.FunctionBaseis the core component to building a model with PandioML. It provides helper methods and abstract methods that make it easy to build and deploy models.Thepandioml.function.Contextprovides local helper methods to simulate what is available when deployed to production to allow faster local iterative testing.Thepandioml.function.Storageprovides access to a distributed key value storage service that is eventually consistent.Thepandioml.function.Loggerprovides local helper methods to simulate what is available when deployed to production to allow faster local iterative testing.ContributingAll contributions are welcome.The best ways to get involved are as follows:IssuesThis is a great place to report any problems found with PandioML. Bugs, inconsistencies, missing documentation, or anything that acted as an obstacle to using PandioML.DiscussionsThis is a great place for anything related to PandioML. Propose features, ask questions, highlight use cases, or anything else you can imagine.If you would like to submit a pull request to this library, please read thecontributor guidelines.LicensePandioML is licensed under theSSPL license.