Package: numalogic
numalogicBackgroundNumalogic is a collection of ML models and algorithms for operation data analytics and AIOps.
At Intuit, we use Numalogic at scale for continuous real-time data enrichment including
anomaly scoring. We assign an anomaly score (ML inference) to any time-series
datum/event/message we receive on our streaming platform (say, Kafka). 95% of our
data sets are time-series, and we have a complex flowchart to execute ML inference on
our high throughput sources. We run multiple models on the same datum, say a model that is
sensitive towards +ve sentiments, another more tuned towards -ve sentiments, and another
optimized for neutral sentiments. We also have a couple of ML models trained for the same
data source to provide more accurate scores based on the data density in our model store.
An ensemble of models is required because some composite keys in the data tend to be less
dense than others, e.g., forgot-password interaction is less frequent than a status check
interaction. At runtime, for each datum that arrives, models are picked based on a conditional
forwarding filter set on the data density. ML engineers need to worry about only their
inference container; they do not have to worry about data movement and quality assurance.Numalogic realtime trainingFor an always-on ML platform, the key requirement is the ability to train or retrain models
automatically based on the incoming messages. The composite key built at per message runtime
looks for a matching model, and if the model turns out to be stale or missing, an automatic
retriggering is applied. The conditional forwarding feature of the platform improves the
development velocity of the ML developer when they have to make a decision whether to forward
the result further or drop it after a trigger request.Key FeaturesEase of use: simple and efficient tools for predictive data analyticsReusability: all the functionalities can be re-used in various contextsModel selection: easy to compare, validate, fine-tune and choose the model that works best with each data setData processing: readily available feature extraction, scaling, transforming and normalization toolsExtensibility: adding your own functions or extending over the existing capabilitiesModel Storage: out-of-the-box support for MLFlow and support for other model ML lifecycle management toolsUse CasesDeployment failure detectionSystem failure detection for node failures or crashesFraud detectionNetwork intrusion detectionForecasting on time series dataGetting StartedFor set-up information and running your first pipeline using numalogic, please see ourgetting started guide.InstallationNumalogic requires Python 3.8 or higher.PrerequisitesNumalogic needsPyTorchandPyTorch Lightningto work.
But since these packages are platform dependendent,
they are not included in the numalogic package itself. Kindly install them first.Numalogic supports pytorch versions2.0.0and above.numalogic can be installed using pip.pipinstallnumalogicIf using mlflow for model registry, install using:pipinstallnumalogic[mlflow]Build locallyInstallPoetry:curl -sSL https://install.python-poetry.org | python3 -To activate virtual env:poetry shellTo install dependencies:poetry install --with dev,torchIf extra dependencies are needed:poetry install --all-extrasTo run unit tests:make testTo format code style using black and ruff:make lintSetuppre-commithooks:pre-commitinstallContributingWe would love contributions in the numalogic project in one of the following (but not limited to) areas:Adding new time series anomaly detection modelsMaking it easier to add user's custom modelsSupport for additional model registry frameworksFor contribution guildelines please referhere.ResourcesQUICK_STARTEXAMPLESCONTRIBUTING