Saravanan666/pypi-org huggingface dataset:
Package: mlops-ai
mlopsOpen-source tool fortracking&monitoringmachine learning models.Table of ContentsIntroductionTechnologiesInstallation & usageDocumentationExamplesLicenseContactReferencesTo-DoIntroductionEnd-to-end machine learning projects require long-term lifecycles during which different models are evaluated,
with various hyperparameters or data representations.
Then, out of all the experiments, a final model must be selected for deployment in the production environment.
There are some solutions available to manage the model creation process, such asmlfloworneptune.ai. However, none of them support the functionality of monitoring a deployed model in production.As a part of the mlops project, we aim to create a ready-to-use tool for professionals in the Machine Learning industry
allowing them not only tomanage experiments during model creation process (tracking module),
but alsomonitoring a deployed model working on real-world production data (monitoring module)with an option tosetup email alerts usingMailGun(email alerts module).TechnologiesApplication consist of two main components:Main application (client + server) written inReactandFastAPI,
which you can run usingDocker.Python packagefor communication with the application.Additionally, we usemongoDBdatabase for storing tracking module data.Installation & usageTo install the application locally, you need to havedockeranddocker-composeinstalled on your machine.
Then, you can run the following command:docker-composeupAfter that you can access the application athttp://localhost:3000.To install the python package make sure you havePython >=3.9installed on your machine.
Then, you can install the package using pip:pipinstallmlops-aiDocumentationYou can find the detailed documentation of the applicationhere.ExamplesThe main end-to-end notebook that
presents key features of the package can be foundhere.
Some other example notebooks are also provided inside thelibrary/tests/notebooksdirectory.LicenseDistributed under the open-source Apache 2.0 License. SeeLICENSEfor more information.ContactProject authors are (in alphabetical order):Paweł Łączkowski (front-end)Kacper Pękalski (back-end, library)Jędrzej Rybczyński (back-end, library)Kajetan Szal(back-end, library)Feel free to contact us in case of any questions or suggestions.ReferencesThis project was created as a final BE project of Computer Science course atFaculty of Mathematics and Computer ScienceofAdam Mickiewicz University.To-DoApplication is still under development.
Here is a list of features we plan to implement in the future:Add support for the whole monitoring moduleAdd support for email alertsAWS EC2 integrationAdd support for multiple users (optionally)
Package: mlops-generator
mlops-generatorCLI for MLOps generatorFree software: MIT licenseDocumentation:https://mlops-generator.readthedocs.io.Install`python pip install-r./requirements.txt ``python python3 setup.py install `FeaturesInitialize project`mlops_generator init`Add configurations`mlops_generator add`–setupadd setup.py entry point–testsadd pytests suite for unit tests–dockeradd dockerfile and .dockerignore–deployadd Google Cloud Build CI PipelineAdd components`mlops_generator component`Pandas extensionSklearn base classes (TransformerMixin, RegressorMixin, ClassifierMixin, etc)kubeflow-component or Kubeflow Container Opkubeflow-pipelineJupyter-notebookTemporal and Visualization artifacts for kubeflowHelp`mlops_generator--help`CreditsThis documentation was generated fromCookiecutterand theaudreyr/cookiecutter-pypackageproject template. Thanks cookiecutter!History0.1.0 (2020-07-28)First release on PyPI.
Package: mlops-models
Sample modelTest repo for hands-on part of the first MLOps lecture.How to use itfrommlops_modelsimportConstantPredictionModelmodel=ConstantPredictionModel(0)model.predict("")>0frommlops_modelsimportConstantPredictionModelmodel=ConstantPredictionModel(1)model.predict("")>1How to build itTo build a python package, run the following command:pythonsetup.pysdistbdist_wheelThis will generate adistfolder with the following structure:mlops_models-0.1.0-py3-none-any.whl
mlops_models-0.1.0.tar.gzHow to test itTo run the tests, run the following command:python-mpytestHow to install itTo install the package, run the following command:pipinstallmlops_models-0.1.0-py3-none-any.whlTo install it in development mode, run the following command:pipinstall-e.How to run itTo run the package, run the following command:python-mmlops_modelsHow to publish itTo publish the package, run the following command:python-mtwineuploaddist/*To publish package in GitHub, run the following command:python-mgithub-releaseupload--tagv0.1.0--userFRI-Machine-Learning-Operations-22-23--repomlops-01-hands-on--name"mlops_models-0.1.0-py3-none-any.whl"--filedist/mlops_models-0.1.0-py3-none-any.whlHow to contributeTo contribute, run the following commands:gitclone
Package: mlops-tempo
Tempo: The MLOps Software Development KitDocumentationAn open source framework to enable data scientists to productionise, test and deploy models with simple workflows that abstract the underlying complexity of scalable MLOps platforms.HighlightsTempo provides a unified interface to multiple MLOps projects that enable data scientists to deploy and productionise machine learning systems.Package your trained model artifacts to optimized server runtimes (Tensorflow, PyTorch, Sklearn, XGBoost etc)Package custom business logic to production servers.Build an inference pipeline of models and orchestration steps.Include any custom python components as needed. Examples:Outlier detectors with Alibi-Detect.Explainers with Alibi-Explain.Test Locally - Deploy to ProductionRun with local unit tests.Deploy locally to Docker to test with Docker runtimes.Deploy to production on KubernetesExtract declarative Kubernetes yaml to follow GitOps workflows.Supporting Seldon production runtimesSeldon Core open sourceSeldon Deploy enterpriseCreate stateful services. Examples:Multi-Armed Bandits.WorkflowDevelop locally.Test locally on Docker with production artifacts.Push artifacts to remote bucket store and launch remotely (on Kubernetes).Motivating SynopsisData scientists can easily test their models and orchestrate them with pipelines.Below we see twoModels (sklearn and xgboost) with a function decoratedpipelineto call both.defget_tempo_artifacts(artifacts_folder:str)->Tuple[Pipeline,Model,Model]:sklearn_model=Model(name="test-iris-sklearn",platform=ModelFramework.SKLearn,local_folder=f"{artifacts_folder}/{SKLearnFolder}",uri="s3://tempo/basic/sklearn",)xgboost_model=Model(name="test-iris-xgboost",platform=ModelFramework.XGBoost,local_folder=f"{artifacts_folder}/{XGBoostFolder}",uri="s3://tempo/basic/xgboost",)@pipeline(name="classifier",uri="s3://tempo/basic/pipeline",local_folder=f"{artifacts_folder}/{PipelineFolder}",models=PipelineModels(sklearn=sklearn_model,xgboost=xgboost_model),)defclassifier(payload:np.ndarray)->Tuple[np.ndarray,str]:res1=classifier.models.sklearn(input=payload)ifres1[0]==1:returnres1,SKLearnTagelse:returnclassifier.models.xgboost(input=payload),XGBoostTagreturnclassifier,sklearn_model,xgboost_modelSave the pipeline code.fromtempo.serve.loaderimportsavesave(classifier)Deploy locally to docker.fromtempoimportdeploy_localremote_model=deploy_local(classifier)Make predictions on containerized servers that would be used in production.remote_model.predict(np.array([[1,2,3,4]]))Deploy to Kubernetes for production.fromtempo.serve.metadataimportSeldonCoreOptionsfromtempoimportdeploy_remoteruntime_options=SeldonCoreOptions(**{"remote_options":{"namespace":"production","authSecretName":"minio-secret"}})remote_model=deploy_remote(classifier,options=runtime_options)This is an extract from themulti-model introductiondemo.
Package: mlops-validators
mlops-validators-v2Nova versão da engine de validação Python mlops_validators, inicialmente focada em DataFrames SQL do Pyspark.IntroduçãoA mlops_validators é uma engine Python desenvolvida para validação de modelos de Machine Learning. Ela contém estruturas de dados e classes para calculos de diversas métricas. Os pacotes estão organizados da seguinte maneira:mlops_validators.tables: Classes de estruturas de dados para a construção de tabelas de frequência para validação. Tem propósito de implementar calculos de contagem, proporções e percentuais em tabelas de frequência para uma ou mais features. São estruturas de dados utilizadas nas implementações das métricas do pacotemlops_validators.metrics.mlops_validators.metrics: Classes com implementações de métricas de validação, tais como,Information Value,Kullback Leibler Divergence,Population Stability Index, Estatística de Chi2, Estatística deKolmogorov Smirnove Teste de Proporção.mlops_validators.profiles: Classeswrapperde alto nível para executar calculos de diversas métricas de uma só vez.RequisitosO mlops_validators foi desenvolvido usando Python e Spark. Para executar os códigos aqui presentes, recomenda-se instalar as dependências nas seguintes versões (mesmas que foram usadas para desenvolver o mlops_validators):python 3.7.9
jvm 1.8
spark-3.2.1-bin-hadoop2.7
winutils for hadoop-2.7.1O Python 3.7.9 pode ser obtidoaqui, ospark-3.2.1-bin-hadoop2.7pode ser obtidoaquie a ferramentawinutilsna versão do hadoop 2.7.1 está disponívelaqui. Extrair o spark e o winutils em uma pasta que possa ser acessada dentro de um programa Python.InstalaçãoCaso a utilização do pacote seja local, recomenda-se a utilização de um ambiente virtual. A mlops_validators está disponível para instalação via pip:pip install mlops_validatorsUtilizaçãoSe o spark for instalado de maneira local, é necessário adicionar as variáveis de ambiente do spark e do hadoop dentro do script ou notebook que o código for desenvolvido. Além disso, é necessário a chamada dafindspark, biblioteca que busca o local do PySpark na máquina e executa o import (ele vem dentro do spark-3.2.1-bin-hadoop2.7). O código que executa esses passos é o seguinte (substituir nas variáveis de ambiente "HADOOP_HOME e "SPARK_HOME" os paths onde estão extraídos o winutils e o spark, respectivamente):importosos.environ["HADOOP_HOME"]="caminho da winutils"os.environ["SPARK_HOME"]="caminho do spark"importfindsparkfindspark.init()ContatoQualquer dúvida entrar em contato comcharles26f@gmail.com.
Package: mlopt
Machine Learning Optimizermloptis a package to learn how to solve numerical optimization problems from data. It relies oncvxpyfor modeling andgurobifor solving the problem offline.mloptlearns how to solve programs usingpytorch(pytorch-lightning),xgboostoroptimaltrees. The machine learning hyperparameter optimization is performed usingoptuna.Online,mloptonly requires to predict the strategy and solve a linear system usingscikit-umfpack.ExamplesTo seemloptin action, have a look at the notebooks in theexamples/folder.DocumentationComing soon atmlopt.org!CitingIf you usemloptfor research, please cite the following papers:The Voice of Optimization:@article{mlopt,
  author = {{Bertsimas}, D. and {Stellato}, B.},
  title = {The Voice of Optimization},
  journal = {Machine Learning (to appear)},
  year = {2020},
  month = jun,
}Online Mixed-Integer Optimization in Milliseconds@article{stellato2019a,
  author = {{Bertsimas}, D. and {Stellato}, B.},
  title = {Online Mixed-Integer Optimization in Milliseconds},
  journal = {INFORMS Journal on Computing (major revision)},
  year = {2019},
  month = jul,
}Projects using mlopt frameworkLearning Mixed-Integer Convex Optimization Strategies for Robot Planning and Control
Package: mloptima
ML_optRepository of optimization methods in machine learning.Machine Learning models and valuable visualization.Two-variable function extrema finding and visualization implementation. Use User interface notebook twovarextremas Jupyter Notebook with downloaded modules.One dimensional optimization(minimization) methods implementation and visualization. Use User interface notebook onedimensionaloptimization Jupyter Notebook with downloaded modules.Gradients optimization(minimization) methods and visualization implementation. Use User interface notebook gradients Jupyter Notebook with downloaded modules.Regression(Linear, Polynomial, Exponential) models and visualization implementation with StochasticGradientDescent/NormalEquation solvers. Use User interface notebook regression colab Jupyter Notebook with downloaded modules.Optimization with equality/inequality constraints. Interior point methods and visualization implementation. Use User interface notebook interiorpoint Jupyter Notebook with downloaded modules.Classification. Implementation of LogisticRegression(Ridge, Lasso), LogisticRegression with RBF kernel function, SVMs classifiers. Use User Interface notebook classification notebook colab with downloaded modules.Stochastic optimization. Implementation of Stochastic gradient descent, Support vector classifier on 2 classes optimized by SGD, simulated annealing algorithm for function minimization, genetic algorithm for function minimization. Built in visualization implemented.Change log0.0.1 (19/06/2022)First Library Release
Package: mloptm
Optimization MethodsMethods DiscussedGolden Section Search MethodBiSection MethodNewton MethodSecant MethodGolden Section Search Methods>>>frommloptm.methodsimportGolden>>>deff(x):...returnx**4-14*x**3+60*x**2-70*x>>>op=Golden(f)>>>minima=op.Minimize(a0=0,b0=2,eps=0.03)>>>op.PrintOptimizationSteps()UsingGoldenOptimizationMedhodFoundLocalMinimaatx->[0.777088]OptimizationStepswith[9]Steps---------------------------------a0b0a1b1Minima----------------------------------------01.236070.7639321.236070.6180340.4721361.236070.4721360.7639320.8541020.4721360.9442720.7639320.9442720.7082040.6524760.9442720.6524760.7639320.7983740.6524760.8328160.7639320.8328160.7426460.721360.8328160.721360.7639320.7770880.7639320.8328160.7639320.7902430.7983740.7639320.8065040.7902430.8065040.7852180.7639320.7902430.7801930.7902430.777088>>>op.PlotOptimizationSteps(xmin=0,xmax=2)>>>op.ExportOptimizationSteps(xmin=0,xmax=2,filname="OptimizedFunction")BiSection Method>>>frommloptm.methodsimportBiSection>>>deff(x):...returnx**4-14*x**3+60*x**2-70*x>>>defdf(x):...return4*x**3-14*3*x**2+120*x-70>>>op=BiSection(f,df)>>>minima=op.Minimize(a0=0,b0=2,epochs=10)>>>op.PrintOptimizationSteps()UsingBiSectionOptimizationMedhodFoundLocalMinimaatx->[0.779297]---------------------------------a0b0(a0+b0)/2f'((a0+b0)/2)-----------------------------------------011120.510.5-200.7510.75-1.93750.750.8750.8755.523440.750.81250.81251.918950.750.781250.781250.0225830.7656250.781250.765625-0.9494480.7734380.781250.773438-0.4614350.7773440.781250.777344-0.2189280.7792970.781250.779297-0.0980478>>>op.PlotOptimizationSteps(xmin=0,xmax=2)>>>op.ExportOptimizationSteps(xmin=0,xmax=2,filname="OptimizedFunction")Newton Method>>>frommloptm.methodsimportNewton>>>deff(x):...returnx**4-14*x**3+60*x**2-70*x>>>defdf(x):...return4*x**3-14*3*x**2+120*x-70>>>defddf(x):...return12*x**2-14*6*x+120>>>op=Newton(f,df,ddf)>>>minima=op.Minimize(x0=0,eps=10**-5)>>>op.PrintOptimizationSteps()UsingNewtonOptimizationMethodFoundLocalMinimaatx->[0.780884]---------------------------------xkxk+1f'(xk+1)    f''(xk+1)----------------------------------------00.583333-13.497775.08330.5833330.763103-1.1078662.88730.7631030.780719-0.010170761.73390.7807190.780884-8.85683e-0761.72310.7808840.780884061.7231>>>op.PlotOptimizationSteps(xmin=0,xmax=2)>>>op.ExportOptimizationSteps(xmin=0,xmax=2,filname="OptimizedFunction")Secant Method>>>frommloptm.methodsimportSecant>>>deff(x):...returnx**4-14*x**3+60*x**2-70*x>>>defdf(x):...return4*x**3-14*3*x**2+120*x-70>>>op=Secant(f,df)>>>minima=op.Minimize(a0=0,b0=2,epochs=10)>>>op.PrintOptimizationSteps()UsingNewtonOptimizationMethodFoundLocalMinimaatx->[0.780884]---------------------------------xkxk+1f(xk+1)f'(xk+1)--------------------------------------00.604282-23.3462-11.94010.10.733837-24.3002-2.976530.6042820.776858-24.3691-0.2490170.7338370.780786-24.3696-0.006054750.7768580.780884-24.3696-1.28637e-050.7807860.780884-24.3696-6.67001e-10>>>op.PlotOptimizationSteps(xmin=0,xmax=2)>>>p.ExportOptimizationSteps(xmin=0,xmax=2,filname="OptimizedFunction")
Package: mloq
ML Ops QuickstartML Ops Quickstart is a tool for initializing Machine Learning projects following ML Ops best practices.Setting up new repositories is a time-consuming task that involves creating different files and
configuring tools such as linters, docker containers and continuous integration pipelines.
The goal ofmloqis to simplify that process, so you can start writing code as fast as possible.mloqgenerates customized templates for Python projects with focus on Maching Learning. An example of
the generated templates can be found inmloq-template.1.Installationmloqis tested on Ubuntu 18.04+, and supports Python 3.6+.Install from pypipipinstallmloqInstall from sourcegitclonehttps://github.com/FragileTech/ml-ops-quickstart.gitcdml-ops-quickstart
pipinstall-e.2.Usage2.1Command line interfaceOptions:--file-f: Name of the configuration file. Iffileit's a directory it will load themloq.ymlfile present in it.--overwrite-o: Rewrite files that already exist in the target project.--interactive-i: Missing configuration data can be defined interactively from the CLI.Usage examplesArguments:OUTPUT_DIRECTORY: Path to the target project.To set up a new repository from scratch interactively in the curren working directory:mloqsetup-i.To load amloq.ymlconfiguration file from the current repository, and initialize the directoryexample, and
overwrite all existing files with no interactivity:mloqsetup-f.-oexample5.LicenseML Ops Quickstart is released under theMITlicense.6.ContributingContributions are very welcome! Please check thecontributing guidelinesbefore opening a pull request.
Package: mlorax
InstallationChoice 1: Install with pipYou can installmloraxwith pip. This is the recommended way to usemloraxif you want to receive future updates.pipinstallmloraxChoice 2: Just copy the codeYou can also directly copy the code frommlorax.pyand paste it into your project. This is the easiest way to usemloraxif you do not care about future updates.Choice 3: Install from sourceYou can also installmloraxfrom source. You only need to do this if you want to contribute to the project.gitclonehttps://github.com/yongchanghao/MLoRAx.gitcdMLoRAx
pipinstall-e.UsageIt is extremely easy to usemloraxto convert any Flax model to a LoRA model. The following code snippet shows how to convert a T5 model to a LoRA model based on HuggingFace'sFlaxT5ForConditionalGenerationclass.+ import mloraxmodel = FlaxT5ForConditionalGeneration.from_pretrained('t5-small')- params = model.params- apply_fn = model.__call__+ lora_spec = mlorax.LoRASpec(rank=16, rules=['Attention.q', 'Attention.v'])+ params, apply_fn, merge_fn = mlorax.lora_init(lora_spec, model)state = TrainState(apply_fn=apply_fn, params=params, tx=tx, **kwargs)That's it! You can now train the model as usual.PrinciplesAlways use thereturnedapply_fnfor model forwarding if possible. Otherwise useparams=merge_fn(params)to pass the merged parameters to the function. For example, if you want to usemodel.generatefor text generation, you can do the following:- outputs = model.generate(**batch, params=params)+ outputs = model.generate(**batch, params=merge_fn(params))Example and ResultsPlease refer to theexamplesfolder for details.CitationIf you find MLoRAx useful, please cite the following paper:@software{hao2023lmrax,author={Yongchang Hao},title={{T}he {LMR}ax {E}cosystem: a minimalist library for training {T}ransformer models with {JAX}},year={2023},url={https://github.com/yongchanghao/LMRax},version={0.9.4}}
Package: mlos-core
mlos-coreThisdirectorycontains the code for themlos-coreoptimizer package.It's available forpip installvia the pypi repository atmlos-core.Descriptionmlos-coreis an optimizer package, wrapping other libraries like FLAML and SMAC to use techniques like Bayesian optimization and others to identify & sample tunable configuration parameters and propose optimal parameter values with a consistent API:suggestandregister.These can be evaluated bymlos-bench, generating and tracking experiment results (proposed parameters, benchmark results & telemetry) to update the optimization loop, or used independently.FeaturesSince the tunable parameter search space is often extremely large,mlos-coreautomates the following steps to efficiently generate optimal task-specific kernel and application configurations.Reduce the search space by identifying a promising set of tunable parametersMap out the configuration search space: Automatically track and manage the discovery of new Linux kernel parameters and their default values across versions.
Filter out non-tunable parameters (e.g., not writable) and track which kernel parameters exist for a given kernel version.Leverage parameter knowledge for optimization: Information on ranges, sampling intervals, parameter correlations, workload type sensitivities for tunable parameters are tracked and currently manually curated.
In the future, this can be automatically maintained by scraping documentation pages on kernel parameters.Tailored to application: Consider prior knowledge of the parameter's impact & an application's workload profile (e.g. network heavy, disk heavy, CPU bound, multi-threaded, latency sensitive, throughput oriented, etc.) to identify likely impactful candidates of tunable parameters, specific to a particular application.Sampling to warm-start optimization in a high dimensional search spaceProduce optimal configurations through Bayesian optimizationSupport for various optimizer algorithms (default Bayesian optimizer, Flaml, SMAC, and random for baseline comparison), that handle multiple types of constraints.
This includes cost-aware optimization, that considers experiment costs given current tunable parameters.Integrated withmlos-bench, proposed configurations are logged and evaluated.
Package: mlos-viz
mlos_vizThemlos_vizmodule is an aid to visualizing experiment benchmarking and optimization results generated and stored bymlos_bench.Its core API ismlos_viz.plot(experiment), initially implemented as a wrapper arounddablto provide a basic visual overview of the results, whereexperimentis anExperimentDataobjected returned from themlos_bench.storagelayer APIs.In the future, we plan to add more automatic visualizations, interactive visualizations, feedback to themlos_benchexperiment trial scheduler, etc.It's available forpip installvia the pypi repository atmlos-viz.See Also:https://microsoft.github.io/MLOSfor full API details.
Package: mlots
Machine Learning On Time-Series (MLOTS)mlotsprovides Machine Learning tools for Time-Series Classification.
This package builds on (and hence depends on)scikit-learn,numpy,tslearn,annoy, andhnswliblibraries.It can be installed as a python package from thePyPIrepository.InstallationInstallmlotsby running:pip install mlotsAfter installation, it can be imported to apythonenvironment to be employed.import mlotsDocumentationThe documentation is hosted atreadthedocs. Examples of usingmlotsmodels are present in theGetting Startedsection of the documentation.ContributeIssue Tracker:https://github.com/vivekmahato/mlots/issuesSource Code:https://github.com/vivekmahato/mlotsSupportIf you are having issues, please let us know.LicenseThe project is licensed under the BSD 3-Clause license.AcknowledgementsWe thank Angus Dempster et al. for sharing (open-sourcing) the code for ROCKET and MINIROCKET.
Package: mloyalty
Python API for MloyaltyУстановкаРекомендуемый способ установки с помощью менеджера пакетов Python pip*:$ pip install mloyalty*API содержит только часть методов, он все еще находится в стадии разработки и регулярно обновляется, не забывайте обновлять его,pip install mloyalty --upgrade