Package: acumos
Acumos Python Client User Guideacumosis a client library that allows modelers to push their Python models
to theAcumos platform.InstallationYou will need a Python 3.6 or 3.7 environment in order to installacumos.
Python 3.8 and later can also be used starting with version 0.9.5, some AI
framework like Tensor Flow was not supported in Python 3.8 and later.
You can useAnaconda(preferred) orpyenvto install and
manage Python environments.If you’re new to Python and need an IDE to start developing, we
recommend usingSpyderwhich
can easily be installed with Anaconda.Theacumospackage can be installed with pip:pipinstallacumosProtocol BuffersTheacumospackage uses protocol buffers andassumes you have
the protobuf compilerprotocinstalled. Please visit theprotobuf
repositoryand install the appropriateprotocfor your operating system.
Installation is as easy as downloading a binary release and adding it to
your system$PATH. This is a temporary requirement that will be
removed in a future version ofacumos.Anaconda Users: You can easily installprotocfroman Anaconda
packagevia:condainstall-canacondalibprotobufAcumos Python Client TutorialThis tutorial provides a brief overview ofacumosfor creating
Acumos models. The tutorial is meant to be followed linearly, and some
code snippets depend on earlier imports and objects. Full examples are
available in theexamples/directory of theAcumos Python client repository.Importing AcumosCreating A SessionA Simple ModelExporting ModelsDefining TypesUsing DataFrames with scikit-learnDeclaring RequirementsDeclaring OptionsKeras and TensorFlowTesting ModelsMore ExamplesImporting AcumosFirst import the modeling and session packages:fromacumos.modelingimportModel,List,Dict,create_namedtuple,create_dataframefromacumos.sessionimportAcumosSessionCreating A SessionAnAcumosSessionallows you to export your models to Acumos. You can
either dump a model to disk locally, so that you can upload it via the
Acumos website, or push the model to Acumos directly.If you’d like to push directly to Acumos, create a session with thepush_apiargument:session=AcumosSession(push_api="https://my.acumos.instance.com/push")See the onboarding page of your Acumos instance website to find the correctpush_apiURL to use.If you’re only interested in dumping a model to disk, arguments aren’t needed:session=AcumosSession()A Simple ModelAny Python function can be used to define an Acumos model usingPython
type hints.Let’s first create a simple model that adds two integers together.
Acumos needs to know what the inputs and outputs of your functions are.
We can use the Python type annotation syntax to specify the function
signature.Below we define a functionadd_numberswithinttype parametersxandy, and anintreturn type. We then build an Acumos
model with anaddmethod.Note:Functiondocstringsare included
with your model and used for documentation, so be sure to include one!defadd_numbers(x:int,y:int)->int:'''Returns the sum of x and y'''returnx+ymodel=Model(add=add_numbers)Exporting ModelsWe can now export our model using theAcumosSessionobject created
earlier. Thepushanddump_zipAPIs are shown below. Thedump_zipmethod will
save the model to disk so that it can be onboarded via the Acumos website. Thepushmethod pushes the model directly to Acumos.session.push(model,'my-model')session.dump_zip(model,'my-model','~/my-model.zip')# creates ~/my-model.zipFor more information on how to onboard a dumped model via the Acumos website,
see theweb onboarding guide.Note:Pushing a model to Acumos will prompt you for an onboarding token if
you have not previously provided one. The interactive prompt can be avoided by
exporting theACUMOS_TOKENenvironment variable, which corresponds to an
authentication token that can be found in your account settings on the Acumos
website.Defining TypesIn this example, we make a model that can read binary images and output
some metadata about them. This model makes use of a custom typeImageShape.We first create aNamedTupletype calledImageShape, which is
like an ordinarytuplebut with field accessors. We can then useImageShapeas the return type ofget_shape. Note howImageShapecan be instantiated as a new object.importioimportPILImageShape=create_namedtuple('ImageShape',[('width',int),('height',int)])defget_format(data:bytes)->str:'''Returns the format of an image'''buffer=io.BytesIO(data)img=PIL.Image.open(buffer)returnimg.formatdefget_shape(data:bytes)->ImageShape:'''Returns the width and height of an image'''buffer=io.BytesIO(data)img=PIL.Image.open(buffer)shape=ImageShape(width=img.width,height=img.height)returnshapemodel=Model(get_format=get_format,get_shape=get_shape)Note:Starting in Python 3.6, you can alternatively use this simpler
syntax:fromacumos.modelingimportNamedTupleclassImageShape(NamedTuple):'''Type representing the shape of an image'''width:intheight:intDefining Unstructured TypesThecreate_namedtuplefunction allows us to create types with structure,
however sometimes it’s useful to work with unstructured data, such as plain
text, dictionaries or byte strings. Thenew_typefunction allows for just
that.For example, here’s a model that takes in unstructured text, and returns the
number of words in the text:fromacumos.modelingimportnew_typeText=new_type(str,'Text')defcount(text:Text)->int:'''Counts the number of words in the text'''returnlen(text.split(' '))defcreate_text(x:int,y:int)->Text:'''Returns a string containing ints from x to y'''return" ".join(map(str,range(x,y+1)))defreverse_text(text:Text)->Text:'''Returns an empty image buffer from dimensions'''returntext[::-1]By using thenew_typefunction, you informacumosthatTextis
unstructured, and thereforeacumoswill not create any structured types or
messages for thecountfunction.You can use thenew_typefunction to create dictionaries or byte string
type unstructured data as shown below.fromacumos.modelingimportnew_typeDict=new_type(dict,'Dict')Image=new_type(byte,'Image')Using DataFrames with scikit-learnIn this example, we train aRandomForestClassifierusingscikit-learnand use it to create an Acumos model.When making machine learning models, it’s common to use a dataframe data
structure to represent data. To make things easier,acumoscan
createNamedTupletypes directly frompandas.DataFrameobjects.NamedTupletypes created frompandas.DataFrameobjects store
columns as named attributes and preserve column order. BecauseNamedTupletypes are like ordinarytupletypes, the resulting
object can be iterated over. Thus, iterating over aNamedTupledataframe object is the same as iterating over the columns of apandas.DataFrame. As a consequence, note hownp.column_stackcan
be used to create anumpy.ndarrayfrom the inputdf.Finally, the model returns anumpy.ndarrayofintcorresponding
to predicted iris classes. Theclassify_irisfunction represents
this asList[int]in the signature return.importnumpyasnpimportpandasaspdfromsklearn.datasetsimportload_irisfromsklearn.ensembleimportRandomForestClassifieriris=load_iris()X=iris.datay=iris.targetclf=RandomForestClassifier(random_state=0)clf.fit(X,y)# here, an appropriate NamedTuple type is inferred from a pandas DataFrameX_df=pd.DataFrame(X,columns=['sepal_length','sepal_width','petal_length','petal_width'])IrisDataFrame=create_dataframe('IrisDataFrame',X_df)# ==================================================================================# # or equivalently:## IrisDataFrame = create_namedtuple('IrisDataFrame', [('sepal_length', List[float]),#                                                     ('sepal_width', List[float]),#                                                     ('petal_length', List[float]),#                                                     ('petal_width', List[float])])# ==================================================================================defclassify_iris(df:IrisDataFrame)->List[int]:'''Returns an array of iris classifications'''X=np.column_stack(df)returnclf.predict(X)model=Model(classify=classify_iris)Check out thesklearnexamples in the examples directory for full
runnable scripts.Declaring RequirementsIf your model depends on another Python script or package that you wrote, you can
declare the dependency via theacumos.metadata.Requirementsclass:fromacumos.metadataimportRequirementsNote that only pure Python is supported at this time.Custom ScriptsCustom scripts can be included by givingRequirementsa sequence of paths
to Python scripts, or directories containing Python scripts. For example, if the
model defined inmodel.pydepended onhelper1.py:model_workspace/
├── model.py
├── helper1.py
└── helper2.pythis dependency could be declared like so:fromhelper1importdo_thingdeftransform(x:int)->int:'''Does the thing'''returndo_thing(x)model=Model(transform=transform)reqs=Requirements(scripts=['./helper1.py'])# using the AcumosSession created earlier:session.push(model,'my-model',reqs)session.dump(model,'my-model','~/',reqs)# creates ~/my-modelAlternatively, all Python scripts withinmodel_workspace/could be included
using:reqs=Requirements(scripts=['.'])Custom PackagesCustom packages can be included by givingRequirementsa sequence of paths to
Python packages, i.e. directories with an__init__.pyfile. Assuming that the
package~/repos/my_pkgcontains:my_pkg/
├── __init__.py
├── bar.py
└── foo.pythen you can bundlemy_pkgwith your model like so:frommy_pkg.barimportdo_thingdeftransform(x:int)->int:'''Does the thing'''returndo_thing(x)model=Model(transform=transform)reqs=Requirements(packages=['~/repos/my_pkg'])# using the AcumosSession created earlier:session.push(model,'my-model',reqs)session.dump(model,'my-model','~/',reqs)# creates ~/my-modelRequirement MappingPython packaging andPyPIaren’t
perfect, and sometimes the name of the Python package you import in your
code is different than the package name used to install it. One example
of this is thePILpackage, which is commonly installed usinga fork
called pillow(i.e.pip install pillowwill provide thePILpackage).To address this inconsistency, theRequirementsclass allows you to map Python package names to PyPI package names. When
your model is analyzed for dependencies byacumos, this mapping is
used to ensure the correct PyPI packages will be used.In the example below, thereq_mapparameter is used to declare a
requirements mapping from thePILPython package to thepillowPyPI package:reqs=Requirements(req_map={'PIL':'pillow'})Declaring OptionsTheacumos.metadata.Optionsclass is a collection of options that users may
wish to specify along with their Acumos model. If anOptionsinstance is not
provided toAcumosSession.push, then default options are applied. See the
class docstring for more details.Below, we demonstrate how options can be used to include additional model metadata
and influence the behavior of the Acumos platform. For example, a license can be
included with a model via thelicenseparameter, either by providing a license
string or a path to a license file. Likewise, we can specify whether or not the Acumos
platform should eagerly build the model microservice via thecreate_microserviceparameter. Then thanks to thedeployparameter you can specifiy if you want to deploy
this microservice automatically. (Please refer to the appropriate documentation on Acumos
wiki to use this functionality based on an external jenkins server). ifcreate_microservice``=True,``deploycan be True or False. But ifcreate_microservice``=False,``deploymust be set to False
if not,create_microservicewill be force to True to create the micro-service and deploy it.fromacumos.metadataimportOptionsopts=Options(license="Apache 2.0",# "./path/to/license_file" also workscreate_microservice=True,# Build the microservice just after the on-boardingdeploy=True)# Deploy the microservice based on an external Jenkins serversession.push(model,'my-model',options=opts)Keras and TensorFlowCheck out the Keras and TensorFlow examples in theexamples/directory of
theAcumos Python client repository.Testing ModelsTheacumos.modeling.Modelclass wraps your custom functions and
produces corresponding input and output types. This section shows how to
access those types for the purpose of testing. For simplicity, we’ll
create a model using theadd_numbersfunction again:defadd_numbers(x:int,y:int)->int:'''Returns the sum of x and y'''returnx+ymodel=Model(add=add_numbers)Themodelobject now has anaddattribute, which acts as a
wrapper aroundadd_numbers. Theadd_numbersfunction can be
invoked like so:result=model.add.inner(1,2)print(result)# 3Themodel.addobject also has a correspondingwrappedfunction
that is generated byacumos.modeling.Model. The wrapped function is
the primary way your model will be used within Acumos.We can access theinput_typeandoutput_typeattributes to test
that the function works as expected:AddIn=model.add.input_typeAddOut=model.add.output_typeadd_in=AddIn(1,2)print(add_in)# AddIn(x=1, y=2)add_out=AddOut(3)print(add_out)# AddOut(value=3)model.add.wrapped(add_in)==add_out# TrueMore ExamplesBelow are some additional function examples. Note hownumpytypes
can even be used in type hints, as shown in thenumpy_sumfunction.fromcollectionsimportCounterimportnumpyasnpdeflist_sum(x:List[int])->int:'''Computes the sum of a sequence of integers'''returnsum(x)defnumpy_sum(x:List[np.int32])->np.int32:'''Uses numpy to compute a vectorized sum over x'''returnnp.sum(x)defcount_strings(x:List[str])->Dict[str,int]:'''Returns a count mapping from a sequence of strings'''returnCounter(x)Acumos Python Client Release Notesv1.0.1, 27 April 2021use acumos-python-client > 0.8.0 with Acumos clioACUMOS-4330v1.0.0, 13 April 2021Fix Type issue with python 3.9ACUMOS-4323v0.9.9, 13 April 2021Take into account “deploy” parameter in acumos python clientACUMOS-4303v0.9.8, 06 November 2020Return docker URI & added an optional flag to replace and existing model when dumpingACUMOS-4298The model bundle can now be dumped directly as a zip fileACUMOS-4273Allow installation on python 3.9ACUMOS-4123v0.9.7, 27 August 2020Add support of python 3.7 & 3.8ACUMOS-4123Display acumos logo on githubACUMOS-4094v0.9.4, 05 April 2020Give image tag URL from python clientACUMOS-3961v0.9.3, 30 Mar 2020Modify unstructured type section in pypiACUMOS-3956Raise an Error when using asymetric typeACUMOS-3956v0.9.2, 31 Jan 2020Remove support for python 3.5Gerrit-6275v0.9.1add raw format supportACUMOS-2712publish content type for long descriptionGerrit-5504v0.8.0(This is the recommended version for the Clio release)EnhancementsUsers may now specify additional options when pushing their Acumos model. See the options section in the tutorial for more information.acumosnow supports Keras models built withtensorflow.kerasSupport changesacumosno longer supports Python 3.4v0.7.2Bug fixesThe deprecated authentication API is now considered optionalA more portable path solution is now used when saving models, to avoid issues with models developed in Windowsv0.7.1AuthenticationUsername and password authentication has been deprecatedUsers are now interactively prompted for an onboarding token, as opposed to a username and passwordv0.7.0RequirementsPython script dependencies can now be specified using a Requirements objectPython script dependencies found during the introspection stage are now included with the modelv0.6.5Bug fixesDon’t attempt to use an empty auth token (avoids blank strings to be set in environment)v0.6.4Bug fixesThe normalized path of the system base prefix is now used for identifying stdlib packagesv0.6.3Bug fixesImproved dependency inspection when using a virtualenvRemoved custom packages from model metadata, as it caused image build failuresFixed Python 3.5.2 ordering bug in wrapped model usagev0.6.2TensorFlowFixed a serialization issue that occurred when using a frozen graphv0.6.1Model uploadThe JWT is now cleared immediately after a failed uploadAdditional HTTP information is now included in the error messagev0.6.0Authentication tokenA new environment variableACUMOS_TOKENcan be used to short-circuit
the authentication processExtra headersAcumosSession.pushnow accepts an optionalextra_headersargument,
which will allow users and systems to include additional information when
pushing models to the onboarding serverv0.5.0ModelingPython 3.6 NamedTuple syntax support now testedUser documentation includes example of new NamedTuple syntaxModel wrapperModel wrapper now has APIs for consuming and producing Python
dicts and JSON stringsProtobuf and protocAn explicit check for protoc is now made, which raises a more
informative error messageUser documentation is more clear about dependence on protoc, and
provides an easier way to install protoc via AnacondaKerasThe active keras backend is now included as a tracked modulekeras_contrib layers are now supportedv0.4.0Replaced library-specific onboarding functions with “new-style”
modelsSupport for arbitrary Python functions using type hintsSupport for custom user-defined typesSupport for TensorFlow modelsImproved dependency introspectionImproved object serialization mechanismsAcumos Python Client Developer GuideTestingWe use a combination oftox,pytest, andflake8to testacumos. Code which is not PEP8 compliant (aside from E501) will be
considered a failing test. You can use tools likeautopep8to
“clean” your code as follows:$pipinstallautopep8$cdacumos-python-client$autopep8-r--in-place--ignoreE501acumos/testing/examples/Run tox directly:$cdacumos-python-client$exportWORKSPACE=$(pwd)# env var normally provided by Jenkins$toxYou can also specify certain tox environments to test:$tox-epy36# only test against Python 3.6$tox-eflake8# only lint codeA set of integration test is also available inacumos-package/testing/integration_tests.
To run those, useacumos-package/testing/tox-integration.inias tox config (-c flag),
onboarding tests will be ran with python 3.6 to 3.9.
You will need to set your user credentials and platform configuration intox-integration.ini.$tox-cacumos-package/testing/integration_testsPackagingThe RST files in the docs/ directory are used to publish HTML pages to
ReadTheDocs.io and to build the package long description in setup.py.
The symlink from the subdirectory acumos-package to the docs/ directory
is required for the Python packaging tools.  Those tools build a source
distribution from files in the package root, the directory acumos-package.
The MANIFEST.in file directs the tools to pull files from directory docs/,
and the symlink makes it possible because the tools only look within the
package root.