Threat Hunting with Jupyter Notebooks — Part 2: Basic Data Analysis with Pandas 📊
By: Roberto Rodriguez
Tags: Jupyter Notebook, Threat Hunting, Cybersecurity, Apache Spark, Data Science

In the previous post, I introduced the basic concepts of Jupyter Notebooks and showed you how to deploy a basic notebook environment. At this point, we are ready to start running some more Python code and start showing you how flexible and dynamic notebooks can be to analyze security events.

In this post, I will show you how to read and analyze a security log file, in JSON format, with the help of a python library named Pandas.

This post is part of a five-part series. The other four parts can be found in the following links:

Requirements

This post assumes that you read the previous one, and deployed the HELK project following specific instructions.

Basic python programming skills and basic understanding of Python Numpy Arrays . If you are new to python data structures, control flows, loops, functions, modules, and NumPy arrays or would like to refresh your memory, I put together two short intro notebooks for you. Open notebooks 01-intro-to-python.ipynb and 02-intro-to-numpy-arrays.ipynb .

Even though, I would love to jump straight to the analysis of event logs, I believe it is important to understand the basic concepts behind Pandas.

What is Pandas?

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python.

Pandas data objects rely heavily on the high-performance NumPy array data structure, and provides flexible data manipulation capabilities of typical spreadsheets and relational databases (such as SQL).

What is Numpy?

NumPy is short for “Numerical Python” and it is a fundamental python package for scientific computing.

It uses a high-performance data structure known as the n-dimensional array or ndarray, a multi-dimensional array object for efficient computation of arrays and matrices.

What is an Array?

Python arrays are data structures that store data similar to a list, except the type of objects stored in them is constrained. Elements of an array are all of the same type and indexed by a tuple of positive integers.

The python module array allows you to specify the type of array at object creation time by using a type code, which is a single character.

A NumPy N-Dimensional Array (ndarray)

NumPy array is an efficient multidimensional array providing fast array-oriented arithmetic operations.

is an efficient multidimensional array providing fast array-oriented arithmetic operations. As any other Python array, it is a container for elements of the same type.

As with other container objects in Python, the contents of an ndarray can be accessed and modified by indexing or slicing operations.

can be accessed and modified by indexing or slicing operations. NumPy arrays handle vectorized operations and are more efficient for storing than the other built-in Python data structures.

handle vectorized operations and are more efficient for storing than the other built-in Python data structures. If we try to add a number to each element of a list, you have to perform a loop in order to apply the basic addition to every element.

If you do the same to a NumPy array, you will be able to add a number to the array object and it will be applied to every element of the array.

Pandas and NumPy Arrays

Pandas leverages and extends several array-based functionalities from NumPy and provides a more expressive means of representing and manipulating data. The two main data structures in Pandas are:

Series

DataFrames

Pandas Series

A Series is a one-dimensional labeled indexed array that can hold any data type such as integers, floats and strings.

A Series can be created and initialized simply by passing a NumPy ndarray.

As you can see above, the index is on the left and the values on the right. We did not specify an index for the data; therefore, a default one gets created starting from 0.

Pandas DataFrames

A DataFrame is a two-dimensional table and it is used to represent data in a tabular format.

Think of a DataFrame as a spreadsheet with column headers and rows.

A DataFrame has both a row and column index.

Pandas relies on NumPy data types for numerical entries in DataFrames. For example, we can print the type of individual entries such as the first value on the age column using iloc and we can see it is of NumPy type.

Now that we understand the basics of Pandas and its data structures, we can start playing with security events logs in a tabular format via DataFrames.

Basic Event Log Analysis

Let’s create a new notebook, explore a Mordor dataset, enrich it, visualize it and filter on interesting events. The solution of this notebook is named basic_event_log_analysis_pandas and it is available for you. I recommend to re-create it and follow the steps below so that you get some practice.

Create a new notebook

Import Libraries

import pandas as pd

from pandas.io.json import json_normalize

import json

import matplotlib.pyplot as plt

Read Mordor Dataset

We are going to read the empire_invoke_wmi Mordor small dataset.

Mordor small dataset. We can use the method read_json to convert the Mordor JSON file to a DataFrame. The parameter lines is set to True to read the file as a json object per line.

invoke_wmi_df = pd.read_json('datasets/empire_invoke_wmi_2019-05-18214442.json', lines=True)

Explore The Data

Next, we can use shape in order to return a tuple representing the dimensionality of the DataFrame. It has 5271 rows and 23 columns.

in order to return a tuple representing the dimensionality of the DataFrame. It has rows and columns. In addition, we can use iloc to select the first record in the DataFrame. In this step we get an idea of the schema present in our DataFrame. Remember that our dataset contains disparate data sources . Therefore, I expect the schema to change a lot per record or audit category.

Flatten Nested Columns

In the previous image, we can see a few nested fields in the dataset. Therefore, we can use json_normalize to help us flatten all those columns.

to help us flatten all those columns. First, I translate the DataFrame back to JSON with the to_json method.

method. Next, I load the results as a json structure to then be normalized by the json_normalize function and get a DataFrame in return.

json_struct = json.loads(invoke_wmi_df.to_json(orient="records"))

invoke_wmi_flat = json_normalize(json_struct)

Identify Data Sources Available

You can use the pandas groupby method and group the DataFrame by the source_name field. I like to do this to learn more about the available data sources that I can play with.

invoke_wmi_flat.groupby(['source_name']).size()

Filter on Sysmon Events

We can now start filtering our main DataFrame by a specific data source.

I decided to filter only Microsoft-Windows-Sysmon events to start.

events to start. As you can see below, there are several fields still with NaN values. This is because the data schema for each event in Sysmon is different.

sysmon_filter = invoke_wmi_flat['source_name'] == "Microsoft-Windows-Sysmon"

sysmon_df = invoke_wmi_flat[sysmon_filter]

Filter on Sysmon Process Creation Events (EID 1)

Now that we have a Sysmon DataFrame, we can filter those events by specific event ids. Let’s do it by passing the event_id == 1 filter to it.

sysmon_filter2 = sysmon_df['event_id'] == 1

sysmon_proc_create = sysmon_df[sysmon_filter2]

Drop Empty Fields on ProcessCreate Dataframe

After filtering the Sysmon DataFrame on one specific event, we can then start dropping the columns with NaN values. All records that belong to the same event type (event id) should have the same schema so it is easy to filter out fields that do not belong to the ProcessCreate schema.

values. All records that belong to the same event type (event id) should have the same schema so it is easy to filter out fields that do not belong to the schema. You can learn more about the schema via OSSEM Sysmon Event Id 1

sysmon_proc_create = sysmon_proc_create.dropna(axis = 1, how = 'all')

Group Events by Specific Fields

Let’s explore our ProcessCreate events, and group them by ParentImage .

events, and group them by . According to the dataset, the file contains lateral movement over WMI to spawn a new agent on the target machine. Therefore, I decided to stack values of the ParentImage column to learn more about the parent processes that executed something on an endpoint.

sysmon_proc_create.groupby(['event_data.ParentImage']).size()

Enrich Data

One basic step that we can take is to enrich the current events.

We can add a new field to our DataFrame and set it with the results of calculating the total number of characters found in each CommandLine per record. We can then use that new field to look for potential outliers.

sysmon_proc_create['command_count'] = sysmon_proc_create['event_data.CommandLine'].str.len()

Visualize Data

We can the take the values of the new field from each record and visualize them with a basic bar chart.

We can tell that C:\Windows\System32\Wbem\WmiPrvSE.exe executed something with a high number of characters in the CommandLine arguments compared with other processes in the network.

sysmon_proc_create.plot(kind='bar',x='event_data.ParentImage',y='command_count')

Filter on Interesting Event

We can go deeper and learn more about the child processes responsible for the high number of characters in the CommandLine arguments.

arguments. Next, we can take the ProcessGuid of the child process and look for it.

sysmon_filter3 = sysmon_proc_create['event_data.ParentImage'] == "C:\Windows\System32\wbem\WmiPrvSE.exe"

sysmon_proc_create[sysmon_filter3]['event_data.ProcessGuid']

Display Results

We can use the Pandas option_context manager to increate the column width of our results in order to see the large CommandLine arguments.

manager to increate the column width of our results in order to see the large arguments. We have potentially found our adversary.