Package: sermos
SermosNote:This represents Sermos'developerREADME.
Public documentation generated using Sphinx and
located atSermos.QuickstartAddsermosas a dependency to your Python applicationInstall extras depending on what you are building:flask- Convenient interface for Flask applicationsflask_auth- Utilize Sermos' authentication system (Sermos Cloudonly)web- Some standard web server dependencies we likeworkers- InstallsCeleryandnetworkx, which are required if usingSermos Cloudfor your deployed pipelines and scheduled tasksdeploy- Only required for deploying toSermos CloudOverviewSermos provides a set of tools and design roadmap for developers toquicklyandeffectivelyintegrate Data Science into real-world applications. The core
design decisions and recommended technologies are based on nearly a decade
of putting data science to work in demanding applications such as real-time motorsports
strategy atPit Rho, custom implementations across industries
as diverse as energy, finance, and healthcare atRho AI, and climate
impact assesment tools atCRANE. With that
said, this is built bydevelopers and data scientistsand strives to strike an
appropriate balance between opinionated decisions and personal choice.This is open source software and we look forward to seeing what you can build
on top of Sermos. For those looking for quick, scalable, enterprise-grade
deployments, make sure to check outSermos Cloud,
which is purpose-built for running complex, scalable, highly available Machine Learning (ML)
workloads, including Natural Language Processing (NLP), Computer Vision (CV), Decision
Modeling, Internet of Things (IoT), and more.A standard Sermos application comprises two key libraries:SermosThis is the base package with optional scaffolding, architecture components
(e.g.DAG implementationon top ofCelery),
and some useful utilities.Sermos ToolsTool catalog for use in Sermos (or other!) Machine Learning applications.SermosCelery ConfigurationPipelinesAPIsUtilitiesSermos ToolsNLP ToolsDocument ClassificationDocument Similarity SearchingDate Findingetc.IoT TolsGeneral ToolsSlack NotificationsYour Applicationaka "Sermos Client" or "Client Codebase"This is where all of your code lives and only has a fewrequirements:It is a base application written in Python (additional language support
slated for the future).Scheduled tasks and Pipeline nodes must be Python Methods that accept
at least one positional argument:event[Optional] Asermos.yamlfile, which is a configuration file if you
choose to deploy usingSermos CloudCelery [optional]Sermos provides sensical default configurations for the use ofCelery. The default deployment uses RabbitMQ,
and is recommended. This library can be implemented in any other workflow
(e.g. Kafka) as desired.There are two core aspects of Celery that Sermos handles and differ from a
standard Celery deployment.ChainedTaskIncelery.pywhen imported it will configure Celery and also runGenerateCeleryTasks().generate(), which will use thesermos.yamlconfig
to turn customer methods into decorated Celery tasks.Part of this process includes addingChainedTaskas thebasefor all of
these dynamically generated tasks.ChainedTaskis a CeleryTaskthat injectstoolsandeventinto the
signature of all dynamically generated tasks.SermosSchedulerWe allow users to set new scheduled / recurring tasks on-the-fly. Celery's
defaultbeat_schedulerdoes not support this behavior and would require the
Beat process be killed/restarted upon every change. Instead, we set our
customsermos.celery_beat:SermosScheduleras thebeat_scheduler,
which takes care of watching the database for new/modified entries and reloads
dynamically.Workers / Tasks / Pipeline NodesIf running inSermos Cloud, Sermos runs all work
(whether scheduled tasks or pipeline nodes) as Celery
workers. Sermos handles decorating the tasks, generating the correct Celery
chains, etc.Customer code has one requirement: write a python method that accepts one
positional argument:evente.g.def demo_pipeline_node_a(event):
    logger.info(f"RUNNING demo_pipeline_node_a: {event}")
    returnSermos Tools [optional]Sermos provides much of the scaffolding and design guidance for running
machine learning workloads and has a companion projectSermos Tools, which provides
a set of useful (and growing) tools intended to streamline
development of machine learning workflows and tasks.GeneratorsTODO: This needs to be updated both in code and documentation. Leaving here
because it's valuable to update in the future.A common task associated with processing batches of documents is generating
the list of files to process.sermos.generatorscontains two helpful
classes to generate lists of files from S3 and from a local file system.S3KeyGeneratorwill produce a list of object keys in S3. Example:gen = S3KeyGenerator('access-key', 'secret-key')
files = gen.list_files(
    'bucket-name',
    'folder/path/',
    offset=0,
    limit=4,
    return_full_path=False
)LocalKeyGeneratorwill produce a list of file names on a local file system.
Example:gen = LocalKeyGenerator()
files = gen.list_files('/path/to/list/')TestingIf you are developing Sermos core and want to test this package,
install the test dependencies:$ pip install -e .[test]Now, run the tests:$ toxContributorsThank you to everyone who has helped in our quest to put machine learning
to work in the real-world!Kevin LyonsAlejandro MesaCassie BorishVickram PremakumarAral TasherAkshay PakhleGilman CallsenYour Name Here!