Package: iopipe
Build StatusCode
CoveragePyPI
VersionApache
2.0SlackThis package provides analytics and distributed tracing for event-driven
applications running on AWS Lambda.InstallationUsageConfigurationReporting
ExceptionsCustom
MetricsLabelsCore
AgentDisabling
ReportingStep
FunctionsPluginsEvent Info
PluginLogger
PluginProfiler
PluginTrace
PluginAuto DB
TracingAuto HTTP
TracingCreating
PluginsSupported Python
VersionsLambda
LayersFramework
IntegrationChaliceServerlessZappaAccessing
ContextContributingRunning
TestsLicenseInstallationWe expect you will import this library into an existing (or new) Python
project intended to be run on AWS Lambda. On Lambda, functions are
expected to include module dependencies within their project paths, thus
we use-t$PWD. Users building projects with a requirements.txt file
may simply addiopipeto their dependencies.From your project directory:$pipinstalliopipe-t.# If running locally or in other environments _besides_ AWS Lambda:$pipinstalliopipe[local]-t.Your folder structure for the function should look similar to:index.py # contains your lambda handler
  /iopipe
    - __init__.py
    - iopipe.py
  /requests
    - __init__.py
    - api.py
    - ...Installation of the requests library is necessary for local dev/test,
but not when running on AWS Lambda as this library is part of the
default environment via the botocore library.More details about lambda deployments are available in theAWS
documentation.UsageSimply use our decorator to report metrics:fromiopipeimportIOpipeiopipe=IOpipe('your project token here')@iopipedefhandler(event,context):passThe agent comes preloaded with theEvent
Info,ProfilerandTraceplugins. See the relevant plugin sections for usage.ConfigurationThe following may be set as kwargs to the IOpipe class initializer:token(string: required)Your IOpipe project token. If not supplied, the environment variable$IOPIPE_TOKENwill be used if present.Find your project
tokendebug(bool: optional = False)Debug mode will log all data sent to IOpipe servers. This is also a good
way to evaluate the sort of data that IOpipe is receiving from your
application. If not supplied, the environment variableIOPIPE_DEBUGwill be used if present.enabled(bool: optional = True)Conditionally enable/disable the agent. For example, you will likely
want to disabled the agent during development. The environment variable$IOPIPE_ENABLEDwill also be checked.network_timeout(int: optional = 5000)The number of milliseconds IOpipe will wait while sending a report
before timing out. If not supplied, the environment variable$IOPIPE_NETWORK_TIMEOUTwill be used if present.timeout_window(int: optional = 150)By default, IOpipe will capture timeouts by exiting your function 150
milliseconds early from the AWS configured timeout, to allow time for
reporting. You can disable this feature by settingtimeout_windowto0in your configuration. If not supplied, the environment variable$IOPIPE_TIMEOUT_WINDOWwill be used if present.Reporting ExceptionsThe IOpipe decorator will automatically catch, trace and reraise any
uncaught exceptions in your function. If you want to trace exceptions
raised in your case, you can use the.error(exception)method. This
will add the exception to the current report.fromiopipeimportIOpipeiopipe=IOpipe()# Example 1: uncaught exceptions@iopipedefhandler(event,context):raiseException('This exception will be added to the IOpipe report automatically')# Example 2: caught exceptions@iopipedefhandler(event,context):try:raiseException('This exception is being caught by your function')exceptExceptionase:# Makes sure the exception is added to the reportcontext.iopipe.error(e)It is important to note that a report is sent to IOpipe whenerror()is called. So you should only record exceptions this way for failure
states. For caught exceptions that are not a failure state, it is
recommended to use custom metrics (see below).Custom MetricsYou can log custom values in the data sent upstream to IOpipe using the
following syntax:fromiopipeimportIOpipeiopipe=IOpipe()@iopipedefhandler(event,context):# the name of the metric must be a string# numerical (int, long, float) and string types supported for valuescontext.iopipe.metric('my_metric',42)Metric key names are limited to 128 characters, and string values are
limited to 1024 characters.LabelsLabel invocations sent to IOpipe by calling thelabelmethod with a
string (limit of 128 characters):fromiopipeimportIOpipeiopipe=IOpipe()@iopipedefhandler(event,context):# the name of the label must be a stringcontext.iopipe.label('this-invocation-is-special')Core AgentBy default, the IOpipe agent comes pre-loaded with all the bundled
plugins iniopipe.contrib.*. If you prefer to run the agent without
plugins or configure which plugins are used, you can useIOpipeCore:fromiopipeimportIOpipeCorefromiopipe.contrib.traceimportTracePlugin# Load IOpipe with only the trace pluginiopipe=IOpipeCore(plugins=[TracePlugin()])@iopipedefhandler(event,context):passDisabling ReportingYou can programmatically disable IOpipe reporting for a single
invocation using thedisablemethod:fromiopipeimportIOpipeiopipe=IOpipe()@iopipedefhandler(event,context):ifsome_condition:context.iopipe.disable()Reporting will be re-enabled on the next invocation.Step FunctionsIOpipe is compatible with AWS Lambda step functions. To enable step
function tracing:fromiopipeimportIOpipeiopipe=IOpipe()@iopipe.stepdefhandler(event,context):passThe@iopipe.stepdecorator will enable step function mode, which
will collect additional meta data about your step functions.PluginsIOpipe’s functionality can be extended through plugins. Plugins hook
into the agent lifecycle to allow you to perform additional analytics.Event Info PluginThe IOpipe agent comes bundled with an event info plugin that
automatically extracts useful information from theeventobject and
creates custom metrics for them.Here’s an example of how to use the event info plugin:fromiopipeimportIOpipefromiopipe.contrib.eventinfoimportEventInfoPluginiopipe=IOpipe(plugins=[EventInfoPlugin()])@iopipedefhandler(event,context):# do something hereWhen this plugin is installed, custom metrics will be created
automatically for the following event source data:API GatewayALBAlexa Skill KitCloudFrontKinesisKinesis FirehoseS3SESSNSSQSScheduled EventsNow in your IOpipe invocation view you will see useful event
information.Logger PluginNote:This plugin is in beta. Want to give it a try? Find us onSlack.The IOpipe agent comes bundled with a logger plugin that allows you to
attach IOpipe to theloggingmodule so that you can see your log
messages in the IOpipe dashboard.Here’s an example of how to use the logger plugin:fromiopipeimportIOpipefromiopipe.contrib.loggerimportLoggerPluginiopipe=IOpipe(plugins=[LoggerPlugin(enabled=True)])@iopipedefhandler(event,context):context.iopipe.log.info('Handler has started execution')Since this plugin adds a handler to theloggingmodule, you can useloggingdirectly as well:importloggingfromiopipeimportIOpipefromiopipe.contrib.loggerimportLoggerPluginiopipe=IOpipe(plugins=[LoggerPlugin(enabled=True)])logger=logging.getLogger()@iopipedefhandler(event,context):logger.error('Uh oh')You can also specify a log name, such as if you only wanted to log
messages formymodule:fromiopipeimportIOpipefromiopipe.contrib.loggerimportLoggerPluginiopipe=IOpipe(plugins=[LoggerPlugin('mymodule',enabled=True)])This would be equivalent tologging.getLogger('mymodule').By default the logger plugin is disabled. You must explicitly setenabled=Truewhen instantiating or use theIOPIPE_LOGGER_ENABLEDenvironment variable to enable it.The default logger plugin log level islogging.INFO, but it can be
set like this:importloggingfromiopipeimportIOpipefromiopipe.contrib.loggerimportLoggerPluginiopipe=IOpipe(plugins=[LoggerPlugin(enabled=True,level=logging.DEBUG)])Putting IOpipe intodebugmode also sets the log level tologging.DEBUG.The logger plugin also redirects stdout by default, so you can do the
following:fromiopipeimportIOpipefromiopipe.contrib.loggerimportLoggerPluginiopipe=IOpipe(plugins=[LoggerPlugin(enabled=True)])@iopipedefhandler(event,context):print('I will be logged')Now in your IOpipe invocation view you will see log messages for that
invocation.If you prefer your print statements not to be logged, you can disable
this by settingredirect_stdouttoFalse:iopipe=IOpipe(plugins=[LoggerPlugin(enabled=True,redirect_stdout=False)])Note:Due to a change to the way the python3.7 runtime configures
logging, stdout redirection is disabled for this runtime. Usecontext.iopipe.log.*instead.By default the logger plugin will log messages to an in-memory buffer.
If you prefer to log messages to your Lambda function’s/tmpdirectory:iopipe=IOpipe(plugins=[LoggerPlugin(enabled=True,use_tmp=True)])Withuse_tmpenabled, the plugin will automatically delete log files
written to/tmpafter each invocation.Profiler PluginThe IOpipe agent comes bundled with a profiler plugin that allows you to
profile your functions withcProfile.Here’s an example of how to use the profiler plugin:fromiopipeimportIOpipefromiopipe.contrib.profilerimportProfilerPluginiopipe=IOpipe(plugins=[ProfilerPlugin()])@iopipedefhandler(event,context):# do something hereBy default the plugin will be disabled and can be enabled at runtime by
setting theIOPIPE_PROFILER_ENABLEDenvironment variable totrue/True.If you want to enable the plugin for all invocations:iopipe=IOpipe(plugins=[ProfilerPlugin(enabled=True)])Now in your IOpipe invocation view you will see a “Profiling” section
where you can download your profiling report.Once you download the report you can open it using pstat’s interactive
browser with this command:python-mpstats<filehere>Within the pstats browser you can sort and restrict the report in a
number of ways, enter thehelpcommand for details. Refer to thepstats
Documentation.Trace PluginThe IOpipe agent comes bundled with a trace plugin that allows you to
perform tracing.Here’s an example of how to use the trace plugin:fromiopipeimportIOpipefromiopipe.contrib.traceimportTracePluginiopipe=IOpipe(plugins=[TracePlugin()])@iopipedefhandler(event,context):context.iopipe.mark.start('expensive operation')# do something herecontext.iopipe.mark.end('expensive operation')Or you can use it as a context manager:fromiopipeimportIOpipeiopipe=IOpipe()@iopipedefhandler(event,context):withcontext.iopipe.mark('expensive operation'):# do something hereOr you can use it as a decorator:fromiopipeimportIOpipeiopipe=IOpipe()@iopipedefhandler(event,context):@context.iopipe.mark.decorator('expensive operation'):defexpensive_operation():# do something hereexpensive_operation()Any block of code wrapped withstartandendor using the
context manager or decorator will be traced and the data collected will
be available on your IOpipe dashboard.By default, the trace plugin will auto-measure any trace you make. But
you can disable this by settingauto_measuretoFalse:fromiopipeimportIOpipefromiopipe.contrib.traceimportTracePluginiopipe=IOpipe(plugins=[TracePlugin(auto_measure=False)])@iopipedefhandler(event,context):withcontext.iopipe.mark('expensive operation'):# do something here# Manually measure the tracecontext.iopipe.mark.measure('expensive operation')Auto DB TracingThe trace plugin can trace your database requests automatically. To
enable this feature, setauto_dbtoTrueor set theIOPIPE_TRACE_AUTO_DB_ENABLEDenvironment variable. For example:iopipe=IOpipe(plugins=[TracePlugin(auto_db=True)])Withauto_dbenabled, you will see traces for any database requests
you make within your function on your IOpipe dashboard. Currently this
feature supportsMongoDB,MySQL,PostgreSQLandRedis.MongoDB example:fromiopipeimportIOpipefrompymongoimportMongoClientiopipe=IOpipe(plugins=[TracePlugin(auto_db=True)])@iopipedefhandler(event,context):client=MongoClient("your-host-here",27017)db=client.testdb.my_collection.insert_one({"x":10})db.my_collection.find_one()Redis example:fromiopipeimportIOpipefromredisimportRedisiopipe=IOpipe(plugins=[TracePlugin(auto_db=True)])@iopipedefhandler(event,context):r=redis.Redis(host="your-host-here",port=6379,db=0)r.set("foo","bar")r.get("foo")Auto HTTP TracingThe trace plugin can trace your HTTP/HTTPS requests automatically. To
enable this feature, setauto_httptoTrueor set theIOPIPE_TRACE_AUTO_HTTP_ENABLEDenvironment variable. For example:iopipe=IOpipe(plugins=[TracePlugin(auto_http=True)])Withauto_httpenabled, you will see traces for any HTTP/HTTPS
requests you make within your function on your IOpipe dashboard.
Currently this feature only supports therequestslibrary, includingboto3andbotocoresupport.To filter which HTTP requests are traced usehttp_filter:defhttp_filter(request,response):ifrequest['url'].startswith('https://www.iopipe.com'):# Exceptions raised will delete the traceraiseException(DonottracethisURL')# You can also remove data from the traceresponse['headers'].pop('Content-Type',None)returnrequest,responseiopipe=IOpipe(plugins=[TracePlugin(auto_http=True,http_filter=http_filter)])To add additional HTTP headers to your ttrace data usehttp_headers:http_headers=['Cache-Control','Etag']iopipe=IOpipe(plugins=[TracePlugin(auto_http=True,http_headers=http_headers)PluginsCreating PluginsTo create an IOpipe plugin you must implement theiopipe.plugins.Plugininterface.Here is a minimal example:fromiopipe.pluginsimportPluginclassMyPlugin(Plugin):name='my-plugin'version='0.1.0'homepage='https://github.com/iopipe/my-plugin/'enabled=Truedefpre_setup(self,iopipe):passdefpost_setup(self,iopipe):passdefpre_invoke(self,event,context):passdefpost_invoke(self,event,context):passdefpost_response(self,response):passdefpre_report(self,report):passdefpost_report(self):passAs you can see, this plugin doesn’t do much. If you want to see a
functioning example of a plugin check out the trace plugin atiopipe.contrib.trace.plugin.TracePlugin.Plugin PropertiesA plugin has the following properties defined:name: The name of the plugin, must be a string.version: The version of the plugin, must be a string.homepage: The URL of the plugin’s homepage, must be a string.enabled: Whether or not the plugin is enabled, must be a boolean.Plugin MethodsA plugin has the following methods defined:pre_setup: Is called once prior to the agent initialization; is
passed theiopipeinstance.post_setup: Is called once after the agent is initialized; is
passed theiopipeinstance.pre_invoke: Is called prior to each invocation; is passed theeventandcontextof the invocation.post_invoke: Is called after each invocation; is passed theeventandcontextof the invocation.post_response: Is called after the invocation response; is passed
theresponsevalue.pre_report: Is called prior to each report being sent; is passed
thereportinstance.post_report: Is called after each report is sent; is passed thereportinstance.Supported Python VersionsThis package supports Python 2.7, 3.6 and 3.7, the runtimes supported by
AWS Lambda.Lambda LayersIOpipe publishesAWS Lambda
Layerswhich are publicly available on AWS. Using a framework that supports
lambda layers (such as SAM or Serverless), you can use the following
ARNs for your runtime:python3.6, python3.7:arn:aws:lambda:$REGION:146318645305:layer:IOpipePython:$VERSION_NUMBERpython2.7:arn:aws:lambda:$REGION:146318645305:layer:IOpipePython27:$VERSION_NUMBERWhere$REGIONis your AWS region and$VERSION_NUMBERis an
integer representing the IOpipe release. You can get the version number
via theReleasespage.Then in your SAM template (for example), you can add:Globals:Function:Layers:-arn:aws:lambda:us-east-1:146318645305:layer:IOpipePython:1And the IOpipe library will be included in your function automatically.You can also wrap your IOpipe functions without a code change using
layers. For example, in your SAM template you can do the following:Resources:YourFunctionere:Type:'AWS::Serverless::Function'Properties:CodeUri:path/to/your/code# Automatically wraps the handler with IOpipeHandler:iopipe.handler.wrapperRuntime:python3.6Environment:Variables:# Specifies which handler IOpipe should runIOPIPE_HANDLER:path.to.your.handlerIOPIPE_TOKEN:'yourtokenhere'We also have anexample
appusing layers with Serverless. It also demonstrates how to use layers
without a code change.NEW:We have also released aServerless
Pluginto do
all this for you automatically.Framework IntegrationIOpipe integrates with popular serverless frameworks. See below for
examples. If you don’t see a framework you’d like to see supported,
please create an issue.ChaliceUsing IOpipe with theChaliceframework is easy. Wrap yourapplike so:fromchaliceimportChalicefromiopipeimportIOpipeiopipe=IOpipe()app=Chalice(app_name='helloworld')@app.route('/')defindex():return{'hello':'world'}# Do this after defining your routesapp=iopipe(app)ServerlessUsing IOpipe withServerlessis easy.First, we highly recommend theserverless-python-requirementsplugin:$npminstall--save-devserverless-python-requirementsThis plugin will addrequirements.txtsupport to Serverless. Once
installed, add the following to yourserverless.yml:plugins:-serverless-python-requirementsThen addiopipeto yourrequirements.txt:$echo"iopipe">>requirements.txtNow Serverless willpip install-rrequirements.txtwhen packaging
your functions.Keep in mind you still need to add the@iopipedecorator to your
functions. SeeUsagefor details.Be sure to check out theserverless-python-requirementsREADMEas the plugin has a number of useful features for compiling
AWS Lambda compatible Python packages.If you’re using theserverless-wsgiplugin,
you will need to wrap the wsgi handler it bundles with your function.The easiest way to do this is to create awsgi_wrapper.pymodule in
your project’s root with the following:importimpfromiopipeimportIOpipewsgi=imp.load_source('wsgi','wsgi.py')iopipe=IOpipe()handler=iopipe(wsgi.handler)Then in yourserverless.yml, instead of this:functions:api:handler:wsgi.handler...Use this:functions:api:handler:wsgi_wrapper.handler...ZappaUsing IOpipe withZappais easy.
In your project add the following:fromiopipeimportIOpipefromzappa.handlerimportlambda_handleriopipe=IOpipe()lambda_handler=iopipe(lambda_handler)Then in yourzappa_settings.jsonfile include the following:{"lambda_handler":"module.path.to.lambda_handler"}Wheremodule-path.to.lambda_handleris the Python module path to thelambda_handleryou created above. For example, if you put it inmyproject/__init__.pythe path would bemyproject.lambda_handler.Accessing ContextIf the framework you’re using makes it non-trivial to access the Lamba
context, you can useiopipe.context. Theiopipe.contextisNoneif outside of an invocation.fromiopipeimportIOpipeiopipe=IOpipe()# Be sure to check, can be Noneifiopipe.context:# do something with contextContributingContributions are welcome. We use theblackcode formatter.pipinstallblackWe recommend using it withpre-commit:pipinstallpre-commitpre-commitinstallUsing these together will auto format your git commits.Running TestsIf you havetoxinstalled, you can run the Python 2.7 and 3.6 tests
with:toxIf you don’t havetoxinstalled you can also run:pythonsetup.pytestWe also have make tasks to run tests inlambci/lambda:build-pythonDocker containers:maketestLicenseApache 2.0