Package: insightconnect-plugin-runtime
InsightConnect Python Plugin RuntimeThe InsightConnect Python Plugin Runtime is used for building plugins in Python for Rapid7 InsightConnect. The project
results in the building and publishing of two components:Python Plugin Runtime LibraryBase InsightConnect Plugin Docker ImagesDocker images created during the build and deployment of this project are uploaded to theRapid7 Docker Repositories.Furtherdocumentationfor building an InsightConnect plugin is available
to get started.Development of the InsightConnect Plugin RuntimeThe Python Runtime codebase is built to support Python 3.9.18 as of version 5.3.0. The following dependencies will need
to be installed when developing or testing the Plugin Runtime:Python 3.9.18DockermaketoxGetting StartedBuilding Python LibraryTo build and install the plugin runtime library locally, first create a Python virtual environment for the particular Python
version and activate it. Then build, install, and confirm the package has been installed.> python3 -m venv venv
> source venv/bin/activate
> pip install -e ./
> pip list | grep insightconnect-plugin-runtime
insightconnect-plugin-runtime 5.2.2Building the InsightConnect Plugin Runtime Docker ImagesCurrently thepython-3dockerfile is used by default when building the docker image. If you want to specify another
dockerfile for testing purposes, such aspython-3-slim, you can pass it as an argument.make build-image DOCKERFILE=python-3-slimThis will overwrite the defaultpython-3, provided that it exists in thedockerfilesdirectory.Building/troubleshooting local images (Mac M1)Our images are built with OS/ARCH:linux/amd64 due to our ci.yml file specifying linux. Historically this has
not caused any issues but with an M1 Mac, docker will now use arm64. This is because docker builds the image to match the
distribution of the OS running the command. To ensure that building locally and on GH produces an image of the same
architecture the--platformspecifier is passed in the dockerfile.If you build a plugin locally we can see further issues that:the local image does match the current OS distro so docker continues to look remotely..docker can not find this (newly created) image remotely on docker registry (dev images should not be pushed to registry)To overcome this we need to specifyplatform=linux/amd64in the plugin docker file. E.g. if building plugins/html then
this file should be updatedplugins/html/Dockerfile.Note: building this arch on mac M1 will cause the build to be slower.Example error of local image not matching OS distro without passing the--platformflag:=> ERROR [internal] load metadata for docker.io/rapid7/insightconnect-python-3-slim-plugin:latest                          1.4s
------
 > [internal] load metadata for docker.io/rapid7/insightconnect-python-3-slim-plugin:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM rapid7/insightconnect-python-3-slim-plugin
   2 |     LABEL organization=rapid7
   3 |     LABEL sdk=python
--------------------
ERROR: failed to solve: rapid7/insightconnect-python-3-slim-plugin: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
make: *** [image] Error 1Testing Sample PluginThe easiest way to test changes to the runtime is by running it locally against one of thesample pluginsincluded in the repository. Make sure a virtual environment has been activated and then pass in the sample directory
name as a parameter:> make sample=example run-sampleThe plugin will be started inhttpmode and listening athttp:0.0.0.0:10001:[2020-02-13 23:21:13 -0500] [56567] [INFO] Starting gunicorn 19.7.1
[2020-02-13 23:21:13 -0500] [56567] [INFO] Listening at: http://0.0.0.0:10001 (56567)
[2020-02-13 23:21:13 -0500] [56567] [INFO] Using worker: threads
[2020-02-13 23:21:13 -0500] [56571] [INFO] Booting worker with pid: 56571To override Gunicorn config export environment variable GUNICORN_CONFIG_FILE pointing to json file with config
See example in samples/example/gcfg.jsonexport GUNICORN_CONFIG_FILE ="./gcfg.json"To build, install, and run runtime changes without the use of therun-samplerule, the below steps can be used for
same result:> python setup.py build && python setup.py install
> cd samples/example
> python setup.py build && python setup.py install
> ./bin/icon_example httpTesting Locally with Docker RuntimeIn addition to testing locally with the resulting runtime and an InsightConnect plugin, it is also possible to build a
plugin locally and test it as it would be used by the InsightConnect orchestrator.First, build the base runtime:make build-imageThis will result in tagged Docker images that can be used in the included sample plugins. Then the plugin can be built
and run locally:> cd samples/example
> icon-plugin build image --no-pull
> docker run -it -p 10001:10001 rapid7/example:latest httpRunning TestsIn order to run tests, first ensuretoxhas been installed. Tox makes it easy for testing this project in isolated
virtual environments and for specific Python versions. To install tox:> pip install toxRunning all tests:> toxRunning a specific test file:> tox -e py39 -- tests/plugin/hello_world/tests/test_cli.pyPlugin vs Slim Plugin ComparisonPluginSlim PluginPython Version3.9.183.9.18OSAlpineBullseyePackage installerapkaptShell/bin/sh/bin/bashImage Size~350MB~180MBNote that for the plugin image, we runapk updateandapk add ..which leads to a longer build time.ReleaseTo release a new version of the InsightConnect Python Plugin Runtime, the below steps must be followed:Make sure that version is updated in changelog, andsetup.pyfile.Create a Pull Request with your changes to be merged into masterMerge changes after receiving at least one approvalCreate a versioned tag off of release; version must followSemVer; using thegit tag [version #]command (not through the GitHub release UI!) thengit push origin --tags.Github Actionwill perform a matrix build and release
based on the recently created tag for each Python version and Dockerfile in scopeContributionsContributions for maintaining and enhancing the InsightConnect Python Plugin Runtime are appreciated. This project usesBlackfor code formatting and includes a pre-commit hook to auto format code as it is
contributed. Black is installed as a test dependency and the hook can be initialized by runningpre-commit installafter cloning this repository.Changelog5.4.4 - Handling the 'Connection Refused' to cps (during startup) as an info instead of error log as this is an expected error.5.4.3 - Updated dependencies to the latest versions | Fixed issue related to loggerStreamHandlers5.4.2 - Remove background scheduler to simplify when the server gets custom_config values | Revert to usebullseyefor slim image.5.4.1 - Retry logic added to get values from external API for custom_config values.5.4.0 - Implementation of custom_config parameter for plugin tasks | Alpine image updated OpenSSL and expat | Usebookwormfor slim image.5.3.2 - Updated OpenSSL in alpine image and core packages to latest versions.5.3.1 - New logging added to the beginning and end of a task | New logging when an exception is instantiated.5.3.0 - Update base images to pull Python 3.9.18 | python packages bump | rename image to drop python minor version.5.2.4 - Extended logging with OrgID and IntID5.2.3 - Extended logging in AWSClient5.2.2 - Fix longstanding bug where some error responses from a plugin could be in HTML format instead of JSON5.2.1 - AddINVALID_CREDENTIALSexception preset5.2.0 - Addstatus_codeandexceptionproperties to task output5.1.4 - Fix credential masking when connection is null5.1.3 - Fix credential masking when some input fields are empty5.1.2 - Add connection credential masking to the plugin's output and log when displayed as plain text | Add newOutputMaskerclass to handle credential masking5.1.1 - Updated exception preset messages5.1.0 - Add new helper functions5.0.0 - Addhas_more_pagesproperty to task output to indicate task pagination status to output consumers4.10.1 - Remove raising of exception if request id is not available in header4.10.0 - Add structlog for structured logging4.9.0 - Add current SDK version plugin is using to /info endpoint4.8.0 - AddOAuth20ClientCredentialMixinclass to clients4.7.6 - AddPaginationHelperto the AWS Client | Refactored theActionHelper| Addregionhandler for AWSAction4.7.5 - Add AWS client for assuming role4.7.4 - Convert data field to string in exception handling4.7.3 - Addmeasurement_timeproperty to plugin metrics collection4.7.2 - Fix incorrect status codes when handling PluginExceptions4.7.1 - Fix new connection test endpoint, version pin requests to 2.26.04.7.0 - Add endpoint for retrieving all action definitions (input schemas for all actions within a plugin)4.6.0 - Add Bad Response preset to PluginException | Improve 400 error exception handling4.5.1 - Bump version number for release pipeline4.5.0 - Improve exception handling for non-ConnectionTestException errors during a connection test4.4.0 - Add initial support for plugin runtime metrics collection4.3.3 - Constrain greenlet dependency version to fix conflict with gevent4.3.2 - Update Flask dependency to version 2.0.34.3.1 - Add Timeout Preset to ConnectionTestException4.2.1 - Add helpful error messages in JSON4.2.0 - Add implementation and endpoints for tasks4.1.1 - Update gevent dependency version to 20.9.04.1.0 - Provide ability to run with gevent asynchronous gunicorn worker class for increased performance4.0.3 - Fix to avoid command injections when using exec_command helper method4.0.2 - Fix to remove unprintable characters from trigger logs4.0.1 - Fix bug by including schema files in manifest | Fix issue uploading python library twice to PYPI4.0.0 - Implement new API endpoints |
Implement Swagger API documentation generation |
End support for Python2 and PyPy |
Add development details to README |
Enhancements to Makefile for local development and release |
Rebrand SDK to InsightConnect Python Plugin Runtime |
Revamp release process with use of Github Actions3.3.0 - Add webserver route to allow for threading changes3.2.0 - Add new ConnectionTestException/PluginException presets:
UNKNOWN, BASE64_ENCODE, BASE64_DECODE, INVALID_JSON |
Add an optional data parameter for formatting response output