Package: scrywarden
ScryWardenScryWarden is a framework for detecting anomalies in general datasets through the use of behavioral models. The library provides simple interfaces to design behavioral profiles, to pull messages from datasets, and to report on found anomalies. Its also engineered to horizontally scale as needed for larger datasets.It aims to be an accessible anomaly detection method that doesn't require any advanced knowledge on the topic while also allowing those with relevant knowledge to extend it for their needs.Getting StartedScrywarden requires python 3.6 or greater.The first thing to do is install the application through pip.$ pip install scrywardenThis will install a CLI command in your path calledscrywarden. Try it out by typingscrywarden --help.$ scrywarden --help
Usage: scrywarden [OPTIONS] COMMAND [ARGS]...

  Detects anomalies in datasets using behavioral modeling.

Options:
  -c, --config TEXT  Path to the config file to use.  [default:
                     scrywarden.yml]

  --help             Show this message and exit.

Commands:
  collect      Collect messages to populate behavioral profiles.
  investigate  Investigate current anomalies to find malicious activity.The application depends on PostgreSQL as a database backend. If you use docker, the following docker compose configuration will setup an instance matching the default configurations of the application.version:"3.2"services:postgres:image:"postgres"ports:-"5432:5432"environment:POSTGRES_PASSWORD:"scrywarden"POSTGRES_USER:"scrywarden"POSTGRES_DB:"scrywarden"The application is controlled through a configuration file. By default, it looks in your current directory for a file namedscrywarden.yml. The following is an example of a valid configuration file;database:host:"localhost"port:5432name:"scrywarden"user:"scrywarden"password:"scrywarden"transports:heartbeat:class:"scrywarden.transport.heartbeat.MixedHeartbeatTransport"profiles:example:class:"scrywarden.profile.example.ExampleProfile"collector:class:"scrywarden.profile.collectors.TimeRangeCollector"analyzer:class:"scrywarden.profile.analyzers.ExponentialDecayAnalyzer"shippers:csv:class:"scrywarden.shipper.csv.CSVShipper"pipeline:queue_size:500timeout:10Lets break down what each sections does.Database Connectiondatabase:host:"localhost"port:5432name:"scrywarden"user:"scrywarden"password:"scrywarden"This section configures the connection to the PostgreSQL database. These are the default settings when starting scrywarden. Change these to match your database setup if necessary. These can be omitted if the database settings match the defaults.Transportstransports:heartbeat:class:"scrywarden.transport.heartbeat.MixedHeartbeatTransport"Transports are one or more classes that are responsible for pulling data from datasets so they can be read by the application. The transports transform the dataset into a series of JSON messages. Each transport is named by its key in the config. This one is namedheartbeat.Theclassfield defines which python class to use for this transport. This one is using in the builtinMixedHeartbeatTransportclass. This class sends a series of test messages at a set interval. It defaults to sending the following JSON messages every 5 seconds:{"person":"George","greeting":"hello"}{"person":"Ben","greeting":"howdy"}{"person":"Susan","greeting":"salutations"}The interval and messages can both be configured by the following setup:transports:heartbeat:class:"scrywarden.transport.heartbeat.MixedHeartbeatTransport"config:interval:5data:-'{"person":"George","greeting":"Whatsup?"}'-'{"person":"Melvin","greeting":"Cool"}'Other transports can be configured the same way by setting config values in theconfigfield.Profileprofiles:example:class:"scrywarden.profile.example.ExampleProfile"collector:class:"scrywarden.profile.collectors.TimeRangeCollector"analyzer:class:"scrywarden.profile.analyzers.ExponentialDecayAnalyzer"This section defines how a behavioral profile detects anomalies. Lets break down the individual parts of this config.Behavioral Profileprofiles:example:class:"scrywarden.profile.example.ExampleProfile"This imports a profile namedexampleusing the classscrywarden.profile.example.ExampleProfile. This is a built in example profile that combines well with the default heartbeat transports.fromscrywarden.profileimportfields,Profile,reportersfromscrywarden.transport.messageimportMessageclassExampleProfile(Profile):greeting=fields.Single('greeting',reporter=reporters.Mandatory())defmatches(self,message:Message)->bool:"""Determines if the message matches the profile."""return'greeting'inmessagedefget_actor(self,message:Message)->str:"""Pulls the actor name from the message."""returnmessage.get('person')This behavioral profile outlines a couple of things.Thematchesmethod determines if a particular message from a transport matches the behavioral profile. This checks to see if thegreetingfield is in the message. If this profile receives the message{"person": "George", "greeting": "hello"}then it would match, but if it was{"person": "George"}then it would be skipped.Theget_actorpulls out the actor name of the message. Actors are unique identifiers to build behavioral profiles for. In this example, it's pulling out the person who is associated with the greeting. If it receives{"person": "George", "greeting": "hello"}then the actor name would be"George".The class attributegreetingdefines a feature of the profile that keeps track of the values in thegreetingfield of the each message. For example, if the message{"person": "George", "greeting": "hello"}is received, then it tracks it on the backend as:actorfieldvaluecountGeorgegreeting"hello"1If it receives the same value again, it increments this count.actorfieldvaluecountGeorgegreeting"hello"2If another value comes across, such as{"person": "George", "greeting": "howdy"}then it tracks it as a separate feature.actorfieldvaluecountGeorgegreeting"hello"2Georgegreeting"howdy"1These features are used to calculate the anomaly scores of each incoming message. How these scores are calculated is dictated by thereporterdefined on the field. Thegreetingfield is calculated by theMandatoryreporter. This reporter checks the likelihood of the feature value compared to the average of all the other features. So if it received{"person": "George", "greeting": "hello"}then it would receive an anomaly score of 0 because the count 2 is higher than the total average of 1.5. If it received a new value like{"person": "George", "greeting": "What's up?"}it would receive an anomaly score of 1 because it's never seen it before.Anomaly scores are always between 0 and 1, with values closer to 1 being more anomalous. These anomaly scores are stored to be analyzed later for malicious activity.Collectorprofiles:example:...collector:class:"scrywarden.profile.collectors.TimeRangeCollector"This associates the collectorscrywarden.profile.collectors.TimeRangeCollectorclass to theexampleprofile. The collector determines how anomalies related to the profile are fetched for analysis.The given collector selects anomalies to analyze over a given time range. By default, it looks at anomalies in periods of one minute chunks. This can be configured with the following configuration:profiles:example:...collector:class:"scrywarden.profile.collectors.TimeRangeCollector"config:seconds:30Now it will pull anomalies in 30 second chunks.Analyzerprofiles:example:...analyzer:class:"scrywarden.profile.analyzers.ExponentialDecayAnalyzer"This associates the analyzerscrywarden.profile.analyzers.ExponentialDecayAnalyzerclass to theexampleprofile. The analyzer determines if any anomalies collected from the collector are malicious. This analyzer tries to detect large groups of messages with higher anomaly scores while filtering out smaller groups.Shippersshippers:csv:class:"scrywarden.shipper.csv.CSVShipper"Shippers are classes responsible for handling the malicious anomalies found by the analyzers. This one saves the malicious anomalies to a CSV file. By default, it saves it to a file calledalerts.csvin the current directory. This can be set by modifying the config.shippers:csv:class:"scrywarden.shipper.csv.CSVShipper"config:filename:'anomalies.csv'Now it will save them in a file calledanomalies.csvin the current directory.Pipelinepipeline:queue_size:500timeout:10The pipeline is responsible for passing the messages from the transports to the behavioral profiles. By default it will process messages either when the queue is filled with 500 messages or if 10 seconds have passed since the first message was put in the queue. These can be configured to be different here.Start CollectingWith the previous config, messages can start to be collected from the heartbeat transport to the example profile. This is done by runningscrywarden collect.$ scrywarden collect
INFO Pipeline starting
INFO Sending heartbeat message {'person': 'George', 'greeting': 'hello'}
INFO Sending heartbeat message {'person': 'Ben', 'greeting': 'howdy'}
INFO Sending heartbeat message {'person': 'Susan', 'greeting': 'salutations'}
INFO Sending heartbeat message {'person': 'George', 'greeting': 'hello'}
INFO Sending heartbeat message {'person': 'Ben', 'greeting': 'howdy'}
INFO Sending heartbeat message {'person': 'Susan', 'greeting': 'salutations'}
INFO Processing 6 messages
INFO 6 messages identified between 1 profiles in 0.01 seconds
INFO 3 actors upserted in 0.01 seconds
INFO 3 actors fetched in 0.01 seconds
INFO 0 features fetched in 0.00 seconds
INFO 6 values processed between 1 profiles in 0.08 seconds
INFO 3 features updated in 0.01 seconds
INFO 3 features fetched in 0.00 seconds
INFO 3 messages upserted in 0.01 seconds
INFO 3 events created in 0.00 seconds
INFO 3 event anomalies created in 0.00 seconds
INFO Pipeline process took 0.18 secondsYou can see from the logs that the heartbeat went off twice in the 10 second timeout it takes for the pipeline to process messages. After that timeout, it passes the message to the profiles for processing. It does the following steps:Identifies and saves the three actors George, Ben, and Susan.Pulls the values for each message and checks to see if any matching features exist.Generates anomaly scores for the given message values.Generates 3 events with 1 associated anomaly each.Notice how it only generates 3 events instead of 6. This is because after the first loop, the previous features are already recorded, so seeing them again is not anomalous.Let's run the next step to see how these events are rated based on anomalies.Start investigatingSince we now have anomalies in the database, we can analyze them to check for malicious behavior. This is done by runningscrywarden investigate.$ scrywarden investigate
INFO Creating initial investigation
INFO Fetching events between 2020-12-04 00:30:45.318094 and 2020-12-04 00:31:45.318094
INFO 3 anomalies collected in 0.12 seconds
INFO 3 events assigned to investigation 1 in 0.00 seconds
INFO 3 malicious anomalies found in 0.02 seconds
INFO Investigation completed in 0.20 seconds
INFO Writing 3 anomalies to 'alerts.csv'After checking the CSV, the following is found:event_idmessage_idactor_idcreated_atanomaly_idfield_idscore1adb85c93-eced-44ba-b4e6-05cae3a98e3e12020-12-04 00:30:45.318473111.020f015287-3920-4eb5-969a-37b4f65233f822020-12-04 00:30:45.318613211.030f015287-3920-4eb5-969a-37b4f65233f832020-12-04 00:30:45.318613311.0The analyzer found that three anomalies were found to be malicious out of three.DocumentationFurther documentation is coming soon.