Package: phantom-dev
phantom-devUtilities for simplifying the development of Phantom appsAuthorDavid Finn:dfinn@splunk.comRequirementsSplunk>PhantomPython 3.6 or higherFeaturesCommands for rapid creation, packaging, and deployment of Phantom appsStreamlined Python interface for action definitionAutomatic bundling of dependencies using standard pip requirements filesPytest integration with automatic mocking and sys path management for local
unit testingDebugging remote execution against local source files over a secure SSH
tunnelRemote app log monitoringAsynchronous action handler supportDedicated log files for connector logging outputCross-platform clientSupport for apps not created withphantom-devComprehensive Phantom API mocking with type-hintingModels for Phantom data types, includingContainersandArtifactsInstallationpipinstallphantom-devDescriptionphantom-devis a command-line utility for creating, building, and deploying
Phantom apps.App packages are built from project directories containing a YAML metadata file
and a connector implemented in Python.
Any other files in the project directory will be packaged and included with
the app.App MetadataThemetadata.yamlfile in the project directory will be used to generate the
app JSON expected by the Phantom platform.
Minor alterations to the JSON structure have been made to assist with
readability and maintainability;
related keys have been grouped under a common prefix key
(e.g. JSONproject-*keys are defined under theprojectobject in the YAML)
and lists of uniquely-identifiable objects have been converted to objects
(e.g. JSONactionsis now a mapping of action names to action data, rather
than a list of action objects with unique and potentially conflicting names).For information on the generated app metadata, see the official Phantom
documentation.Connector ImplementationThephantom_dev.action_handlermodule greatly simplifies the implementation
of aphantom.base_connector.BaseConnectorsubclass, which is the basis for
Phantom app development.Action handler methods defined using thephantom_dev.action_handler.ActionHandlerdecorator will be used to
dynamically infer action metadata unless overridden in the metadata file.
Action names, parameter names, and parameter types can all be inferred from
the implementation assuming parameters are type-annotated in the code.Thephantom_dev.action_handlermodule will be automatically embedded in thedependenciesdirectory when the app is built, allowing the developer to
make full use of the module without being concerned with managing it as a
dependency.Edge cases and use of more specialisedBaseConnectormethods should be dealt
with as normal, in accordance with the official Phantom documentation.App VersioningUnless a version is explicitly specified in the app metadata,roboversionwill be used to
automatically detect the project version from git repository state.
To set the version, tag the current commit with a version label
(e.g.v1.2.3).If the current commit doesn't have a version label,roboversionwill create
a unique development version using the most recent version tag.Compatibility with Apps Not Created withphantom-devBecausephantom-devrelies on specific library features to infer action
metadata from code implementations, metadata inference willnotwork with
a connector that has been implemented withoutphantom-dev.
Full specification of app metadata must therefore be provided in eithermetadata.yamlor in the old-style.jsonfile.If a.jsonfile is used instead ofmetadata.yaml,phantom-devwillnotautogenerate a new.jsonmetadata file.
Certain packaging features, such as autopopulation of dependency metadata,
will therefore be disabled.Finally, thedebugcommand will not work unless the app is configured as adebugpylistener when executed as__main__.
Whilephantom-devapps implement this by default, other apps will have to
implement this behaviour or forgo the use ofdebug.Aside from these limitations,phantom-devshould work normally with
any Phantom app once extracted into a project directory.QuickstartRunningphantom-dev createwill prompt the user for the set of details
required to define a Phantom app.$phantom-devcreate"My Special App"ProductVendor:SpecialVendor
ProductName:SpecialProduct
Description:Myspeciallittleapp
Publisher:DavidFinn
License:SpeciallicenseA new project directory will be created and populated with ametadata.yamland aconnector.py.
The metadata YAML will contain the details provided by the user, and the
connector module will define an example connector implementation.
The metadata will also be populated with action information required by the
example connector implementation.Configuring IDE AutocompletionTo enable your IDE to resolve Phantom module methods, configure your
IDE to set itsPYTHONPATHenvironment variable to include the location ofphantom_dev.dummy.The dummy file path can be found by runningphantom-dev --dummy_path.
The path will typically resemble<PYTHON_ENV>/lib/python<VERSION>/site-packages/phantom_dev/dummy,
wherePYTHON_ENVis your python environment and<VERSION>is your Python
version.Example: VSCodeFind the path to the dummy module:$ phantom-dev --dummy_path
/home/USER/my_app/.venv/lib/python3.8/site-packages/phantom_dev/dummyTo add this to the PYTHONPATH for VSCode, place the following entry in the.envfile in the workspace directory (create the file if it doesn't exist):PYTHONPATH=/home/USER/my_app/.venv/lib/python3.8/site-packages/phantom_dev/dummyExtending the AppThe following defines the implementation of a new action called "echo message":fromphantom_dev.action_handlerimportActionHandler,SmartConnector...classConnector(SmartConnector,main=True):...@ActionHandlerdefecho_message(self,message:str):"""Echo a message:param message: The message to be echoed"""yield{'response':message}...@echo_message.summarydefsummarise_echo(self,results):result,=resultsreturn{'response':result['response']}...If possible, the action description will be inferred from the method docstring-
in this case, "Echo a message".
Similarly, parameter descriptions will also be inferred if specified in the
docstring with appropriate docstring syntax (currently, Sphinx syntax is
confirmed to work).
In this case, the description for the paramtermessagewill be "The message
to be echoed".The action summary data for theecho messageaction is created by decorating
the summary method with@echo_message.summary.
Action summary methods take the action results as an iterable, and create
the appropriate summary data.Thanks toSmartConnector, all methods of theBaseConnectorclass are
available for use in the action handler logic implementation.Although the app will successfully install and run given the above metadata,
it's still missing output fields andcontainsinformation.
Unless specfied inmetadata.yaml,phantom-dev(and therefore Phantom) has
no way of knowing that the output data should contain aresponsefield
with adata_typeofstring.
If we want to addcontainsinformation, such as setting acontainsvalue of['text']for the message and response, we also need to specify
these somewhere.Thesecanbe specified as normal usingmetadata.yaml, but this can also
be done directly using appropriateActionHandlerparameters and methods:fromphantom_dev.action_handlerimportActionHandler,contains,SmartConnector...@contains('text')classText(str):"""A string-based type which contains `['text']`"""...classConnector(SmartConnector,main=True):...@ActionHandler(action_type='generic',data_contains={'response':Text})defecho_message(self,message:Text):"""Echo a message:param message: The message to be echoed"""yield{'response':message}@echo_message.summary_contains({'response':Text})defsummarise_echo(self,results):result,=resultsreturn{'response':result['response']}......Several things are happening here.
First, a new type calledTextinherits fromstrand is decorated with the@containsdecorator, mapping it to acontainsvalue of['text'].
This type is used instead ofstrto annotate themessageparameter of the
action, which allowsphantom-devto infer both thedata_typeand thecontainsvalue for the parameter.Next, thedata_containsparameter is being used to instantiate the method
constructor, and a mapping of'response'toTextis being specified as the
argument.
This allowsphantom-devto infer that each data result has a field named'response', and that itsdata_typeandcontainsinformation should
be inferred from theTexttype the same way it was for themessageparameter.
Note that we can also set the actiontypemetadata with use of another
decorator parameter,action_type.Finally, the same thing is being done with theecho_message.summary_containsdecorator (used instead ofsummary), though in this case it's the action
summary fields rather than the results data fields that are being described.Use of these decorators is preferred tometadata.yamlspecification, as they
keep metadata definitions colocated with the data they describe.
They also encourage code reuse though the use of commoncontains-mapped
type definitions, shareable between both parameters and output fields.Parameters with specific allowed valuesTo define a set of allowed values for a parameter, define a subclass ofEnumwith a concrete type that maps to adata_type, and assign those values to
members of that class.Specify this class as the parameter annotation to allowphantom-devto infer
thevalue_listmetadata for this parameter.For example, for arulestring parameter with allowed values'allow'and'block':@contains('rule')classRule(str,Enum):ALLOW='allow'BLOCK='block'...classConnector(SmartConnector,main=True):@ActionHandler(...)defset_rule(self,rule:Rule,...):...DependenciesAny package specified inrequirements-whl.txtin the app project directory
will be automatically downloaded as a wheel and packaged with the app.requirements-whl.txtshould be a normalpiprequirements file.For example, if the app requires theroboversionpackage, arequirements-whl.txtcould be created with the following content:roboversion>=2When the app is built, theroboversionwheel will be automatically downloaded
and included in the packagewheelsdirectory, and the autogenerated app
JSON will specify its location for Phantom installation.If there isn't a compatible wheel for the Phantom platform,requirements-sdist.txtcan be used instead ofrequirements-whl.txt.
This will download the package as a source distribution instead of as a wheel.
Unlikerequirements-whl.txt, subdependencies will not be automatically
installed for packages inrequirements-sdist.txt, as sdist installation is
intended to be used as a backup for when a wheel is unavailable.Finally,requirements-pypi.txtcan be used to specify dependencies that won't
be packaged with the app, but that will be downloaded and installed by the
Phantom platform itself.Testingphantom-devcan automatically managepytestexecution, including adding
the app directory to the Python path and mocking imports of thephantommodule, using thephantom-dev testcommand:$phantom-devtest<app_directory>[<pytestarguments>...]Apps created usingphantom-dev createwill be initialised with atestsdirectory containing a basic example of apytestscript.If working from the app directory,pytestwill automatically locate and run
the tests when it is invoked:$phantom-devtestpytestalso accepts a test directory location:$phantom-devtestmy/special/test/locationIf working from a directory other than the app directory, the app directory can
also be specified:$phantom-devtest--app-directorymy_special_appPositional arguments will be passed topytest, allowing us to specify test
locations as well:$phantom-devtest--app-directorymy_special_appmy/special/test/locationAsphantom-dev testis a wrapper aroundpytestinvocation, it supports
all ofpytest's features and behaviour.
To avoid capturing optional parameters before passing them topytest, use the
pseudo-argument--:$phantom-devtest--app-directorymy_special_app----show-localsmy/special/test/locationRefer to thepytestdocumentation for details.Deploying the AppOnce the app is ready to install, assuming a Phantom server location ofphantom.example.com:$phantom-devpushmy_special_app/root@phantom.example.comNote: The Phantom server must be a known host; SSH to it first to confirm
credentials and connectivity.The user will be prompted for the SSH password. Once supplied, the app will be
automatically packaged, sent to the Phantom server, and installed.The SSH password can also be provided as part of the command:$phantom-devpushmy_special_app/root:PASSWORD@phantom.example.comIf certificate authentication is used, an empty password can be also be
specified:$phantom-devpushmy_special_app/root:@phantom.example.comRemote Debuggingphantom-devstreamlines the debugging process by automating the creation of
test action data and configuring a debugger connection through a secure SSH
tunnel.
To open a remote debugging session:$phantom-devdebugmy_special_app/root:@phantom.example.com"dummy action"This command will:Generate a test action JSON file on the remote Phantom hostOpen an SSH tunnel to the remote Phantom host, forwarding the local
port to the remote debugging portWait for a debugger connection, then run the connector script with the
correct Python path using the generated action JSONA local debugging client (e.g. Visual Studio Code) will then be able to connect
and step through action execution.Visual Studio Code Debugging ConfigurationVisual Studio Code debugging is configured through entries inlaunch.json.To use the SSH tunnel provided by thephantom-dev debugcommand, create an
entry configured to connect tolocalhoston the specified debugger port:...{"name":"Remote Debuggable Phantom App","type":"python","request":"attach","connect":{"host":"localhost","port":8869},"pathMappings":[{"localRoot":"${workspaceFolder}/my_special_app/connector.py","remoteRoot":"/opt/phantom/apps/myspecialapp_<SOME UUID>/connector.py"}]}...ThepathMappingsentry should be configured to map the connector module to
its location in the installed app on the remote Phantom host.
The installed app folder will be under<PHANTOM_HOME>/apps.Once configured, a debugging session can be run against the remote action
execution with full debugger functionality.App DocumentationAside from the action and parameter descriptions which can be inferred from
connector method docstrings, Phantom supports adding more detailed
documentation to apps using areadme.htmlfile.Apps created withphantom-devwill be created with a defaultreadme.html,
which can be updated with arbitrary HTML content.Other commandsFor information on the otherphantom-devsubcommands includingpackageanddeploy, run:$phantom-dev--help
$phantom-dev<subcommand>--helpDetailsIn the above example, use of theActionHandler(orActionHandler.data_contains) decorator wraps the decoratedecho_messagemethod in the logic required for error handling and results reporting.
Theparamdictionary is automatically unpacked as keyword arguments to
handler method, allowing for quick and explicit argument validation and
intuitive access to action parameters.paramcontains the parameters described in the app JSON.Handler methods such asecho_messageare expected to return iterables of
results data.
The items from this iterable are added as data objects to theActionResult.
Implementing handler methods as generators is highly convenient, as this allows
custom logic to be run any time before or after data is yielded, but methods
can also be implemented as normal functions that return iterable objects.TheHandlerMixinsuperclass provided bySmartConnectorautomatically
delegates incoming actions to the correct method based on the action
identifier.SmartConnectoralso wraps the functionality of a themain_connectordecorator.main_connectorsimply calls the class'smainmethod if
the class is defined in the__main__module, reproducing the testing
functionality provided by autogenerated app wizard code.Signaling FailureFailure is signaled through raising exceptions.
If the handler executes without raising an exception, the action is treated as
a success.To implement anecho failaction that does the same thing asecho message,
but always fails after producing results:...@contains('text')classText(str):"""A string-based type which contains `['text']`"""...classConnector(SmartConnector,main=True):...@ActionHandler(action_type='generic',data_contains={'response':Text})defecho_message(self,message:Text):"""Echo a message:param message: The message to be echoed"""yield{'response':message}@echo_message.summary_contains({'response':Text})defsummarise_echo(self,results):result,=resultsreturn{'response':result['response']}...@ActionHandler(action_type='generic',data_contains={'response':Text})defecho_fail(self,**param):"""Echo the message as normal, then fail"""# Demonstration of re-packing param; this will be the same as the# original param dictionary, which we can then unpack for the call# to echo_message.# Unfortunately, this will require manual specification of more# parameter metadata.yield fromself.echo_message(**param)raiseRuntimeError('Failed on purpose')# The same summary method can be decorated multiple times for different# handlers to duplicate functionality@echo_fail.summary_contains({'response':Text})@echo_message.summary_contains({'response':Text})defsummarise_echo(self,results):result,=resultsreturn{'response':result['response']}In the example, parameter packing with**paramwas used instead of describing
and annotating the parameters forecho fail.
This is possible but not recommended, because now the user must manually
specify more parameter information inmetadata.yaml:...actions:...echo_fail:parameters:message:data_type:stringdescription:The message to be echoedrequired:truecontains:-text......Actions with no resultstest connectivityis an example of an action which produces no results.
The handler method needs only to return an empty iterable, which is easily
accomplished by returning an empty collection rather than implementing a
generator:...@contains('text')classText(str):"""A string-based type which contains `['text']`"""...classConnector(SmartConnector,main=True):...@ActionHandler(action_type='generic',data_contains={'response':Text})defecho_message(self,message:Text):"""Echo a message:param message: The message to be echoed"""yield{'response':message}@echo_message.summary_contains({'response':Text})defsummarise_echo(self,results):result,=resultsreturn{'response':result['response']}...@ActionHandler(action_type='generic',data_contains={'response':Text})defecho_fail(self,**param):"""Echo the message as normal, then fail"""# Demonstration of re-packing param; this will be the same as the# original param dictionary, which we can then unpack for the call# to echo_message.# Unfortunately, this will require manual specification of more# parameter metadata.yield fromself.echo_message(**param)raiseRuntimeError('Failed on purpose')# The same summary method can be decorated multiple times for different# handlers to duplicate functionality@echo_fail.summary_contains({'response':Text})@echo_message.summary_contains({'response':Text})defsummarise_echo(self,results):result,=resultsreturn{'response':result['response']}@ActionHandler(action_type='generic')deftest_connectivity(self):"""Check `echo message` and `echo fail`"""# The test connectivity action is a special case that does not# receive a param dictionary at all, so there are no arguments to# unpacktest_value='SOME TEST MESSAGE'results=[]try:forresultinself.echo_fail(test_value):results.append(result)exceptRuntimeError:passelse:raiseRuntimeError('echo fail failed to fail')message,=resultsifmessage!=test_value:raiseValueError('echo fail failed to echo')return[]It would also be possible to achieve this with areturnstatement before ayieldstatement in a generator, or by failing before any results are yielded.Action ContextThe hidden phantom actioncontextparameter can be accessed as thecontextmember of the connector object during action execution.
This data contains entries forartifact_id,guid, andparent_action_run;
see 'Understanding Datapaths' in the offical phantom documentation for details.LoggingTheloggermember of the connector object is a standard Pythonlogging.Loggerobject that can be used in the normal way.All log messages ofERRORand above will be logged to<syslog>/phantom/spawn.log.<syslog>will typically be/var/logon privileged installations and<PHANTOM_HOME>/var/logon unprivileged installations.If the system is configured forDEBUGlogging, all messages ofDEBUGand above will also be logged.
Additionally, all log messages of levelINFOand above will be reported
usingBaseConnector.save_progressregardless of Phantom debug configuration.The connector will also keep logfiles in the app's state directory containing
logging records created by calls to the connector'slogger.
This allows convenient consumption of connector logging without having to
filter through the content ofspawn.log.
The logfile can be found at<PHANTOM_HOME>/local_data/app_states/<APP_ID>/logs/connector.log.Vault FilesTheget_vault_pathmethod can be used to retrieve apathlib.Pathobject
from a vault ID:vault_path=connector.get_vault_path('<VAULT_ID>')withvault_path.open()asvault_file:...This is implemented to work on both the Phantom 4.10 and 4.8 Vault interfaces.Persistent StateThestateconnector property can be used to access and save persistent state.
The property will automatically load the peristent statedictfrom the
filesystem on the first access, and save the current state to the filesystem at
the end of action handling if such access has been performed.
This property is intended to replace the variety of methods available on the
base connector for loading, accessing, and saving state data.More sophisticated state management can be performed using theopen_statemethod.
Providing a subpath of the app's state directory to the method will allow
IO to the specified file.
If no subpath is specified, the default state file containing the state dict
will be opened.Asynchronous ActionsScheduling actions to run simultaneously can be configured with the optionallockparameter given to theActionHandlerdecorator.Within a single action, asynchronous coroutine execution can be configured by
implementing the action handler as anasynccoroutine method.Asynchronous Action SchedulingBy default, actions are scheduled synchronously with regard to the asset
running the action; i.e. only one action will be running on a given asset at
at time.This behaviour can be altered by specifying a value forActionHandler'slockparameter.
If set toNone, the action will be scheduled asynchronously with regard to
all other actions.
If given astrvalue, the action will be scheduled synchronously with regard
to all other actions using the samelockvalue.A timeout (in seconds) for lock acquision can be set by specifying a value forActionHandler'slock_timeoutparameter.Lock Data PathsAction parameter or configuration values can be specified as data paths for thelockvalue, by setting the value toparameter.<name>orconfiguration.<name>respectively.
If a datapath for a parameter or configuration value is specified, its value at
the time of action execution will be used as the lock value.
This allows configurable groups of actions which will be scheduled
synchronously with regard to each other, but asynchronously with regard to all
other actions.Asynchronous Coroutine ExecutionIf a handler method is reused by other handler methods, an asynchronous
implementation will allow the method to be run concurrently with other
coroutines.
Additionally, implementing a handler as an asynchronous coroutine function
will cause the connector to automatically handle the event loop, meaning thatawaitstatements can be made in the function implementation without
manual event loop configuration.Consider:@ActionHandlerdefslow_action(self):"""Two long IO operations"""expensive_call()expensive_call()@ActionHandlerdefeven_slower_action(self):"""Two slow actions (4 long IO operations)"""self.slow_action()self.slow_action()Executing a singleeven_slower_actionaction will causeexpensive_callto
be invoked 4 times in sequence.Assuming a significant portion ofexpensive_call's execution time is waiting
for IO operations, an asynchronous implementation calledasync_callwill
allow other coroutines to run while it waits.
Assuming access toasync_call, we can refactor our actions using asynchronous
coroutine functions:@ActionHandlerasyncdefefficient_action(self):"""Two long IO operations run concurrently"""awaitasyncio.gather(async_call(),async_call())@ActionHandlerasyncdefeven_more_efficient_action(self):"""Two efficient actions run concurrently(4 long IO operations run concurrently)"""awaitasyncio.gather(self.efficient_action(),self.efficient_action())Under normal circumstances,even_slower_actionwill take almost 4 times as
long to run aseven_more_efficient_actionwhile producing the same results.
This efficiency will scale over the number of actions that can be run
concurrently.Data IngestionImplementing anon_pollActionHandlermethod allows the app to ingest
containers based on arbitrary logic according to a configured schedule.The default connector includes a reference implementation foron_poll.
The following example outlines ingestion of a single container with a single
artifact:...@ActionHandler(action_type='ingest',read_only=True)defon_poll(self,start_time:int,end_time:int,container_count:int,artifact_count:int,container_id:str=None,):"""Callback action for the on_poll ingest functionality.:param container_id: Container IDs to limit the ingestion to.:param start_time:Start of time range, in epoch time (milliseconds)If not specified, the default is past 10 days:param end_time:End of time range, in epoch time (milliseconds)If not specified, the default is now":param container_count:Maximum number of container records to query for.:param artifact_count: Maximum number of artifact records to query for."""container=Container.from_fields(name='example container',...)container.add_artifact(name='example artifact',...)returnself.save_all_containers([container])...Rendering OutputAction output rendering can be customised by adding arendersection to the
action entry in themetadata.yaml, which takes the same data as specified in
the official documentation.
If using a custom view, theActionHandler.custom_viewdecorator can be used
to automatically populate therendersection of the action metadata.Custom ViewsAdding a custom view requires registering a custom view function and creating
a custom view HTML template.To register a custom view function, theActionHandler.custom_viewdecorator
can be used on the function.
Alternatively, a custom view can be manually specified by adding arendersection to the action entry in themetadata.yaml.Normally, the return value of the function is a string that contains the path
to the HTML template used to render the view.
If the decorator is used to register the function and the function doesn't
return a value, the template location will be inferred to beviews/<function_name>.html.Refer to the Phantom documentation for more information on using custom views.App LogoThe app logo can be customised by including an icon file in the app directory
with the namelogo.<extension>.logo_light.<extension>andlogo_dark.<extension>can also be used to
differentiate between light-mode and dark-mode logos.
Arbitrary filenames and paths can also be used; specify a path forlogoin
the app metadata, or paths forlightanddarkunderlogoto configure
the respecitve themes.Default LogoIcons made byFreepikfromwww.flaticon.com