Package: huntlib
HuntLibA Python library to help with some common threat hunting data analysis operationsWhat's Here?Thehuntlibmodule provides three major object classes as well as a few convenience functions.ElasticDF: Search Elastic and return results as a Pandas DataFrameSplunkDF: Search Splunk and return results as a Pandas DataFrameDomainTools: Convenience functions for accessing the DomainTools API, primarily focused around data enrichment (requires a DomainTools API subscription)data.read_json(): Read one or more JSON files and return a single Pandas DataFramedata.read_csv(): Read one or more CSV files and return a single Pandas DataFramedata.flatten(): Recursively flatten dicts/lists into a single level dict. Useful for data normalization and creating DataFrames.data.chunk(): Break up a large list into smaller chunks for processing.util.entropy()/util.entropy_per_byte(): Calculate Shannon entropyutil.promptCreds(): Prompt for login credentials in the terminal or from within a Jupyter notebook.util.edit_distance(): Calculate how "different" two strings are from each other.util.benfords(): Determine whether a given collection of numbers obeys Benford's Law.util.punctuation_pattern(): Return only the non-alphanumeric characters from a given string or collection of strings.Library-Wide ConfigurationBeginning withv0.5.0,huntlibnow provides a library-wide configuration file,~/.huntlibrcallowing you to set certain runtime defaults.  Consult the filehuntlibrc-samplein this repo for more information.huntlib.elastic.ElasticDFTheElasticDF()class searches Elastic and returns results as a Pandas DataFrame.  This makes it easier to work with the search results using standard data analysis techniques.Example usage:Create a plaintext connection to the Elastic server, no authenticatione=ElasticDF(url="http://localhost:9200")The same, but with SSL and authenticatione=ElasticDF(url="https://localhost:9200",ssl=True,username="myuser",password="mypass")Fetch search results from an index or index pattern for the previous daydf=e.search_df(lucene="item:5282 AND color:red",index="myindex-*",days=1)The same, but do not flatten structures into individual columns. This will result in each structure having a single column with a JSON string describing the structure.df=e.search_df(lucene="item:5282 AND color:red",index="myindex-*",days=1,normalize=False)A more complex example, showing how to set the Elastic document type, use Python-style datetime objects to constrain the search to a certain time period, and a user-defined field against which to do the time comparisons. The result size will be limited to no more than 1500 entries.df=e.search_df(lucene="item:5285 AND color:red",index="myindex-*",doctype="doc",date_field="mydate",start_time=datetime.now()-timedelta(days=8),end_time=datetime.now()-timedelta(days=6),limit=1500)Thesearchandsearch_dfmethods will raiseInvalidRequestSearchExceptionin the event that the search request is syntactically correct but is otherwise
invalid. For example, if you request more results be returned than the server
is able to provide. They will raiseAuthenticationErrorSearchExceptionin the
event the server denied the credentials during login.  They can also raise anUnknownSearchExceptionfor other situations, in which case the exception
message will contain the original error message returned by Elastic so you
can figure out what went wrong.huntlib.splunk.SplunkDFTheSplunkDFclass search Splunk and returns the results as a Pandas DataFrame. This makes it easier to work with the search results using standard data analysis techniques.Example UsageEstablish an connection to the Splunk server. Whether this is SSL/TLS or not depends on the server, and you don't really get a say.s=SplunkDF(host=splunk_server,username="myuser",password="mypass")If you prefer, you use a Splunk session token in place of a username/password (ask your Splunk administator to create one for you):s=SplunkDF(host=splunk_server,token="<your token>")SplunkDFwill raiseAuthenticationErrorSearchExceptionduring initialization
in the event the server denied the supplied credentials.Fetch all search results across all timedf=s.search_df(spl="search index=win_events EventCode=4688")Fetch only specific fields, still across all timedf=s.search_df(spl="search index=win_events EventCode=4688 | table ComputerName _time New_Process_Name Account_Name Creator_Process_ID New_Process_ID Process_Command_Line")Time bounded search, 2 days prior to nowdf=s.search_df(spl="search index=win_events EventCode=4688",days=2)Time bounded search using Python datetime() valuesdf=s.search_df(spl="search index=win_events EventCode=4688",start_time=datetime.now()-timedelta(days=2),end_time=datetime.now())Time bounded search using Splunk notationdf=s.search_df(spl="search index=win_events EventCode=4688",start_time="-2d@d",end_time="@d")Limit the number of results returned to no more than 1500df=s.search_df(spl="search index=win_events EventCode=4688",limit=1500)NOTE: The value specified as thelimitis also subject to a server-side max
value. By default, this is 50000 and can be changed by editing limits.conf on
the Splunk server. If you use the limit parameter, the number of search results
you receive will be the lesser of the following values: 1) the actual number of
results available, 2) the number you asked for withlimit, 3) the server-side
maximum result size.  If you omit limit altogether, you will get thetruenumber of search results available without subject to additional limits, though
your search may take much longer to complete.Return only specified fieldsNewProcessNameandSubjectUserNamedf=s.search_df(spl="search index=win_events EventCode=4688",fields="NewProcessName,SubjectUserName")NOTE: By default, Splunk will only return the fields you reference in the
search string (i.e. you must explicitly search on "NewProcessName" if you want
that field in the results. Usually this is not what we want. When fields is notNone,
the query string will be rewritten with "| fields " at the end (e.g.,search index=win_events EventCode=4688 | fields NewProcessName,SubjectUserName). This
works fine for most simple cases, but if you have a more complex SPL query and it breaks,
simply setfields=Nonein your function call to avoid this behavior.Try to remove Splunk's "internal" fields from search results:df=s.search_df(spl="search index=win_events EventCode=4688",internal_fields=False)This will remove such fields as_timeand_sourcetypeas well as any other field who's name begins with_.  This behavior occurs by default (internal_fieldsdefaults toFalse), but you can disable it by usinginternal_fields=True.Remove named field(s) from the search results:df=s.search_df(spl="search index=win_events EventCode=4688",internal_fields="_bkt,_cd,_indextime,_raw,_serial,_si,_sourcetype,_subsecond,_time")In the event you need more control over which "internal" fields to drop, you can pass a comma-separated list of field names (NOTE: these can be any field, not just Splunk internal fields).Splunk's Python API can be quite slow, so to speed things up you may elect to spread the result retrieval among multiple cores.  The default is to use one (1) extra core, but you can use theprocessesargument tosearch()orsearch_df()to set this higher if you like.df=s.search_df(spl="search index=win_events EventCode=4688",processes=4)If you prefer to use all your cores, try something like:frommultiprocessingimportcpu_countdf=s.search_df(spl="search index=win_events EventCode=4688",processes=cpu_count())NOTE: You may have to experiment to find the optimal number of parallel processes for your specific environment. Maxing out the number of workers doesn't always give the best performance.huntlib.domaintools.DomainToolsTheDomainToolsclass allows you to easily perform some common types of calls
to the DomainTools API.  It uses their officialdomaintools_apiPython module
to do most of the work but is not a complete replacement for that module. In
particular, this class concentrates on a few calls that are most relevant for
data analytic style threat hunting (risk & reputation scores, WHOIS info, etc).TheDomainToolsclass can make use of the global config file~/.huntlibrcto store the API username and secret key, if desired.  See thehuntlibrc-samplefile for more info.Example UsageImport theDomainToolsobject:from huntlib.domaintools import DomainToolsInstantiate a newDomainToolsobject:dt = DomainTools(
    api_username="myuser,
    api_key="mysecretkey
)Instatiate a newDomainToolsobject using default creds stored in~/.huntlibrc:dt = DomainTools()Look up API call limits and usage info for the authenticated user:dt.account_information()Return the list of API calls to which the authenticated user has access:dt.available_api_calls()Return basic WHOIS info for a domain or IP address:dt.whois('google.com')
dt.whois('8.8.8.8')Return WHOIS info with additional fields parsed from the text part of the record:dt.parsed_whois('google.com')
dt.parsed_whois('8.8.8.8')Find newly-activated or pending domain registrations matching all the supplied search terms:dt.brand_monitor('myterm')
dt.brand_monitor('myterm1|myterm2|myterm3') # terms are ANDed togetherLook up basic info about a domain's DNS, WHOIS, hosting and web site in one query.dt.domain_profile('google.com')Return a list of risk scores for a domain, according to different risk factors:dt.risk('google.com')Return a single consolidated risk score for a domain:dt.domain_reputation('google.com')See what DomainTools' IRIS database has to say about a certain domain. This typically provides quick info from a variety of DomainTools sources:dt.iris_enrich('google.com')Enrich a pandas DataFrame containing a mixture of domains and/or IP address in a column called 'iocs'.  It callsDomainTools.iris_enrich()to gather the data, and tries to be efficient by handling duplicate values and by sending multiple queries in the same batch:df = dt.enrich(df, column='iocs')The default is to send batches of 100 domains at a time, but you can decrease this number if necessary (usually because the query string becomes so long the DomainTools API server rejects it):df = dt.enrich(df, column='iocs', batch_size=75)Enrichment tends to add a large number of columns, which you may not need. Use thefieldsparameter if you know exactly what you want:df = dt.enrich(
    df, 
    column='iocs', 
    fields=[
        'dt_whois.registration.created',
        'dt_reputation.risk_score'
    ]
)Enrichment may take quite some time with a large dataset. If you're antsy, try turning on the progress bars:df = dt.enrich(df, column='iocs', progress_bar=True)Data ModuleThehuntlib.datamodule contains functions that make it easier to deal with data and data files.Reading Multiple Data Fileshuntlib.dataprovides two convenience functions to replace the standard Pandasread_json()andread_csv()functions.  These replacement functions work exaclty the same as their originals, and take all the same arguments.  The only difference is that they are capable of accepting a filename wildcard in addition to the name of a single file.  All files matching the wildcard expression will be read and returned as a singleDataFrame.Start by importing the functions from the module:fromhuntlib.dataimportread_csv,read_jsonHere's an example of reading a single JSON file, where each line is a separate JSON document:df=read_json("data.json",lines=True)Similarly, this will read all JSON files in the current directory:df=read_json("*.json",lines=True)Theread_csvfunction works the same way:df=read_csv("data.csv)ordf=read_csv("*.csv")Post-Processing the Input DataBothread_json()andread_csv()support an optionalpost_functionparameter, which allows you to specify a function to post-process the data after each individual file is read in, before it is merged into the final returned DataFrame. For example, you might want to split or combine columns, or compute a new value from existing data.Start by creating a post-processing function according to the following prototype:defmy_post_processor(df,filename):# do some stuffreturndfWhen called, thedfparameter will be a DataFrame containing the chunk of data just read, and thefilenameparameter will be the name of the file it came from, which will be different for each chunk.IT IS IMPORTANT THAT YOU RETURNdfno matter whether you modified the input DataFrame or not.Once you have defined the post-processor function, you can invoke it during your call toread_json()orread_csv()like so:df=read_csv("*.csv",post_function=my_post_processor)Additional Read OptionsConsult the Pandas documentation for information on other supported options forread_csv()andread_json().Normalizing nesting dicts and listsMany times the data that we deal with is not well formatted for our purposes because it contains complex data structures inside itself and we need it to be more regular (e.g., when converting REST API data in JSON format to a pandas DataFrame).  Thehuntlib.data.flatten()function may be just what you need!Given a dict or list that may itself contain other dicts or lists,flatten()will traverse the object recursively and bring all the data into a single dict with a single level of keys (making it 'flat').Flattening a dict with nested dicts:>>> flatten({"key1": "val1", "subkeys": {"subkey1": "subval1"}})
{'key1': 'val1', 'subkeys.subkey1': 'subval1'}Flatten a list with nested lists.  Notice that the resulting keys are the list indices in string form:>>> flatten([1, 2, 3])
{'0': 1, '1': 2, '2': 3}A more complex example:>>> flatten([{'a': 'a', 'b': 'b'}, {'a': 'a1', 'c': 'c'}])
{'0.a': 'a', '0.b': 'b', '1.a': 'a1', '1.c': 'c'}Breaking a long list-like object into smaller chunksGiven a list-like object, divide into chunks ofsizeand return those as a generator. If the length of the sequence is not evenly divisible by the size, the final chunk will contain however many items remain.>>> l = list(range(26))
>>> for i in chunk(l, size=5):
...   print(i)
[0, 1, 2, 3, 4]
[5, 6, 7, 8, 9] 
[10, 11, 12, 13, 14]
[15, 16, 17, 18, 19]
[20, 21, 22, 23, 24]
[25]Util ModuleThehuntlib.utilmodules contains miscellaneous functions that don't fit anywhere else, but are nevertheless still useful.Entropyhuntlib.utilprovides two entropy functions,entropy()andentropy_per_byte(). Both accept a single string as a parameter.  Theentropy()function calculates the Shannon entropy of the given string, whileentropy_per_byte()attempts to normalize across strings of various lengths by returning the Shannon entropy divided by the length of the string.  Both return values arefloat.>>>entropy("The quick brown fox jumped over the lazy dog.")4.425186429663008>>>entropy_per_byte("The quick brown fox jumped over the lazy dog.")0.09833747621473352The higher the value, the more data potentially embedded in it.Credential HandlingSometimes you need to provide credentials for a service, but don't want to hard-code them into your scripts, especially if you're collaborating on a hunt.huntlib.utilprovides thepromptCreds()function to help with this. This function works well both in the terminal and when called from within a Jupyter notebook.Call it like so:(username,password)=promptCreds()You can change one or both of the username/password prompts by passing arguments:(username,password)=promptCreds(uprompt="LAN ID: ",pprompt="LAN Pass: ")String SimilarityString similarity can be expressed in terms of "edit distance", or the number of single-character edits necessary to turn the first string into the second string.  This is often useful when, for example, you want to find two strings that very similar but not identical (such as when hunting forprocess impersonation).There are a number of different ways to compute similarity.huntlib.utilprovides theedit_distance()function for this, which supports several algorithms:Levenshtein DistanceDamerau-Levenshtein DistanceHamming DistanceJaro DistanceJaro-Winkler DistanceHere's an example:>>>edit_distance('svchost','scvhost')1You can specify a different algorithm using themethodparameter. Valid methods arelevenshtein,damerau-levenshtein,hamming,jaroandjaro-winkler. The default isdamerau-levenshtein.>>>edit_distance('svchost','scvhost',method='levenshtein')2Benford's LawBenford's Law, also known as the "first digit law" or the "law of anomalous numbers" states that there is a specific distribution pattern of the first digits of certain groups of numbers.  It is often used to detect cheating or tampering in areas such as tax fraud and vote rigging.Seehttps://en.wikipedia.org/wiki/Benford%27s_lawfor more info on Benford's Law and it's potential applications.Thebenfords()function returns a 3-tuple of values like:(chi2, p, counts).chi2is a float that describes how well the observed distribution of first digits matched the predictions of Benford's Law.  Lower is better.pis the probability that the computed 'chi2' is significant (i.e., it tells you whether the chi2 value can be trusted).  Its range is 0..1, but in this case, higher is better.  Generally speaking, if the p-value is >= 0.95 then the chi2 value is considered significant.countsis a Pandas series where the indices are the possible first digits 1-9 and the values are the observed distributions of those digits. If the observed distributions didn't match up with Benford's law, the counts may help you identify the anomalous values.Here's an example of calling thebenfords()function, with a contrived set of numbers that definitely conform to the expected distribution:>>>numbers=[1,1,1,1,1,1,1,1,2,2,2,2,3,3,3,4,4,5,5,6,6,7,7,8,9]>>>benfords(numbers)(0.019868294035033682,0.9999999995974126,10.3220.1630.1240.0850.0860.0870.0880.0490.04Name:digits,dtype:float64)Notice thechi2value is quite low (~0.02), meaning these numbers follow Benford's Law quite well.  Thepvalue is > 0.999999999 so we can have a high confidence in these results.Here the input is a set of random numbers, which do not conform to Benford's Law.  Notice thechi2value is much higher, but thepvalue is still > 0.9999, indicating that we can trust the fact that it doesn't conform.>>>numbers=np.random.randint(1,10,1000)>>>benfords(numbers)(0.391706824115063,0.9999475565204166,10.11420.11930.11840.09950.11060.11670.12080.08390.121Name:digits,dtype:float64)Generate punctuation patterns from stringsFor certain types of log analysis, the contents of the individual log messages is not as important as the format of the message itself. This often results in the need to examine the pattern of punctuation (non-alphanumeric characters). To facilitate this, thepunctuation_pattern()function accepts a single string or a list-like collection of strings and returnsjustthe non-alphanumeric characters.>>>>>>s='192.168.1.1 - - [10/Oct/2020:12:32:27 +0000] "GET /some/web/app?param=test&param2=another_test" 200 9987'>>>punctuation_pattern(s)'..._-_-_[//:::_+]_"_///?=&=_"__'>>>l=[s,"Another example. This time, of a list of strings!"]>>>punctuation_pattern(l)0..._-_-_[//:::_+]_"_///?=&=_"__1_.__,_____!Name:punct,dtype:object