Package: threatbus-misp
Threat Bus MISP PluginA Threat Bus plugin that enables communication withMISP.The plugin goes against the pub/sub architecture of Threat Bus (for now),
because it actively binds to a single MISP instance to receive attribute
(IoC) updates, and report back sightings via the REST API. Following the strict
pub/sub architecture of Threat Bus, itshould be the other way
around, with MISP binding to Threat Bus. This will eventually be resolved by a
MISP module.For now, the plugin supports two ways to retrieve attribute (IoC) updates from
MISP - either via ZeroMQ or via Kafka. Basically, the plugin makes itself a
subscriber to MISP events.InstallationUsers can specifyoptional dependenciesduring installation. The plugin uses
either ZeroMQ or Kafka to get IoC updates from MISP. As we don't want to burden
the user to install unused dependencies, both options are available as follows:pipinstallthreatbus-misp[zmq]pipinstallthreatbus-misp[kafka]If neither of these dependencies is installed (i.e., you installedthreatbus-mispwithout the[...]suffix for optional deps), the plugin throws
an error and exits immediately.Depending on your setup, you might want to use quotes to avoid shell expansion
when using[...]. For example, you can dopip install ".[zmq]"for local
development.Kafka PrerequisitesWhen you decide to use Kafka to receive IoC updates from MISP, you first need to
install Kafka on the Threat Bus host. This plugin uses theconfluent-kafkaPython package which requireslibrdkafka. See also theprerequisitessection of theconfluent-kafkaPython client for details about setting it up
for your distribution.Once installed, go ahead and install the Kafka version of this plugin:pip install threatbus-misp[kafka]ConfigurationThe plugin uses the MISP REST API to report back sightings of IoCs. You need to
specify a MISP API key for it to work.ZeroMQ and Kafka are mutually exclusive, such that Threat Bus does not receive
all attribute updates twice. See below for an example configuration....plugins:misp:api:host:https://localhostssl:falsekey:MISP_API_KEYfilter:# filter are optional. you can omit the entire section.-orgs:# creator org IDs must be strings: https://github.com/MISP/PyMISP/blob/main/pymisp/data/schema.json-"1"-"25"tags:-"TLP:AMBER"-"TLP:RED"types:# MISP attribute types https://github.com/MISP/misp-objects/blob/main/schema_objects.json-ip-src-ip-dst-hostname-domain-url-orgs:-"2"zmq:host:localhostport:50000#kafka:#  topics:#  - misp_attribute#  poll_interval: 1.0#  # All config entries are passed as-is to librdkafka#  # https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md#  config:#    bootstrap.servers: "localhost:9092"#    group.id: "threatbus"#    auto.offset.reset: "earliest"...IoC FilterThe plugin can be configured with a list of filters. Every filter describes a
whitelist for MISP attributes (IoCs). The MISP plugin will only forward IoCs to
Threat Bus if the whitelisted properties are present.A filter consists of three sub-whitelists for creator organizations, types, and
tags. To pass through the filter, an attribute must provide at least one of the
whitelisted properties of each of the whitelists. More precisely, entries of
each whitelist are linked by an"or"-function, the whitelists themselves are
linked by an"and"-function, as follows:(org_1 OR org_2) AND (type_1 OR type_2) AND (tag_1 OR tag_2).The MISP plugin always assumes that theabsence of a whitelist means that
everything is whitelisted. For example, when the entirefiltersection is
omitted from the config, then all attributes are forwarded and nothing is
filtered. More examples follow below.OrganizationsOrganizations are whitelisted by their ID, which is astring. Only
those MISP attributes that come from any of the whitelisted organizations will
be forwarded to Threat Bus.TypesTypes can be whitelisted by specifying MISPattribute types.
Only those attributes that are instances of a whitelisted type will be forwarded
to Threat Bus.TagsMISP Attributes can be tagged with arbitrary strings. The tag whitelist respects
tagnames. Only those attributes that have at least one of the whitelisted
tags will be forwarded to Threat Bus.Examples:This section provides some simple configuration examples to illustrate how
whitelist filtering works.Forward all IoCs from the organizations"1"and"25"-orgs:-"1"-"25"Forward only IoCs of thedomain,url, oruritype, but only if they
come from the organization"1"or"25".-orgs:-"1"-"25"-types:-domain-url-uriForward only IoCs that are tagged withTLP:REDorTLP:AMBER, but only of
type"src-ip":-tags:-"TLP:RED"-"TLP:AMBER"-types:-src-ipDevelopment SetupThe following guides describe how to set up a local, dockerized instance of
Kafka and how to setup a VirtualBox running MISP for developing.Dockerized KafkaFor a simple, working Kafka Docker setup use thesingle node examplefromconfluentinc/cp-docker-images.Store thedocker-compose.yamland modify the Kafka environment variables such
that the Docker host (e.g.,172.17.0.1on Linux) of your Docker machine is
advertised as Kafka listener:zookeeper:...kafka:...environment:KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://kafka:29092,PLAINTEXT_HOST://172.17.0.1:9092# <-- That is the IP of your Docker hostKAFKA_LISTENER_SECURITY_PROTOCOL_MAP:PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT...Check outthis articlefor details about Kafka listeners.Then start the compose setup viadocker-compose up -d.To test the setup, use thetests/utils/kafka_receiver.pyandtests/utils/kafka_sender.pyscripts.Local MISP using VirtualBoxThis guide walks you through setting up MISP using a pre-configured VirtualBox
VM and then configuring MISP to export Attribute (IoC) updates to Threat Bus.Installation via VirtualBoxUse the officially maintainedVirtual Imagesfor MISP.
Download the latest.ovaimage file and load it in a VirtualBox client. Ensure
the following:The VM has enough working memory (e.g., 3 GiB of RAM)The VM exposes ports 8443 (web interface) and 50000 (ZMQ)Use VirtualBox port-forwarding when NATtingUse VirtualBox bridge-mode & SSH into the VM using SSH port-forwardingHere are the above steps as pure CLI instructions for running MISP in headless
mode (i.e., without a graphical VirtualBox interface).curl -fL -o misp-2.4.138.ova https://www.circl.lu/misp-images/latest/MISP_v2.4.138@28ccbc9.ova
vboxmanage import misp-2.4.138.ova --vsys 0 --vmname misp --memory 3072 --cpus 1 --eula accept
vboxmanage modifyvm misp --nic1 nat
vboxmanage modifyvm misp --natpf1 "zmq,tcp,,50000,,50000"
vboxmanage list -l mispYou can then start and stop VM using the following commands:vboxmanage startvm misp --type headless
vboxmanage controlvm misp poweroffConfiguration for usage with Threat BusFor Threat Bus to receive attribute (IoC) updates from MISP, you must either
enable Kafka or ZMQ export in the MISP VM. If you chose to go with Kafka, you
need to installlibrdkafkafirst inside the VM, then make it known to PHP.Install Kafka inside the VMsshmisp@<MISP_VM_IP># enter your configured password to pop an interactive shell inside the VMsudoapt-getupdate
sudoapt-getinstallsoftware-properties-common
sudoapt-getinstalllibrdkafka-dev# see https://misp.github.io/MISP/INSTALL.ubuntu1804/#misp-has-a-feature-for-publishing-events-to-kafka-to-enable-it-simply-run-the-following-commandssudopeclchannel-updatepecl.php.net
sudopeclinstallrdkafkaecho"extension=rdkafka.so"|sudotee/etc/php/7.2/mods-available/rdkafka.ini
sudophpenmodrdkafka
sudoserviceapache2restartexitOnce Kafka is installed, you can go ahead and enable it in the MISP web-view.Enable Kafka export in the MISP web-viewVisithttps://localhost:8443login with your configured credentialsGo toAdministration->Server Settings & Maintenance->Plugin settings TabSet the following entriesPlugin.Kafka_enable->truePlugin.Kafka_brokers->172.17.0.1:9092<- In this example, 172.17.0.1 is the Docker host as configured in the Dockerized Kafka setup above, reachable from other Docker networks. The port is reachable when the Kafka Docker setup binds to it globally.Plugin.Kafka_attribute_notifications_enable->truePlugin.Kafka_attribute_notifications_topic->misp_attribute<- The topic goes into the threatbusconfig.yamlYou can use ZeroMQ to export IoCs from MISP as light weight alternative to
running Kafka. It does not require any extra installations, except enabling the
feature in the MISP web-view.Enable the ZMQ plugin in the MISP web-viewVisithttps://localhost:8443login with your configured credentialsGo toAdministration->Server Settings & Maintenance->Diagnostics TabFind the ZeroMQ plugin section and enable itGo toAdministration->Server Settings & Maintenance->Plugin settings TabSet the entryPlugin.ZeroMQ_attribute_notifications_enabletotrueLicenseThreat Bus comes with a3-clause BSD license.