Package: iocsearcher
iocsearcheriocsearcheris a Python library and command-line tool to extract
indicators of compromise (IOCs),
also known as cyber observables,
from HTML, PDF, and text files.
It can identify both defanged
(e.g., URL hxxp://example[DOT]com) and
unmodified IOCs (e.g., URL http://example.com).InstallationpipinstalliocsearcherSupported IOCsiocsearchercan extract the following IOC types:URLs (url)Domain names (fqdn)IP addresses (ip4, ip6)IP subnets (ip4Net)Hashes (md5, sha1, sha256)Email addresses (email)Phone numbers (phoneNumber)Copyright strings (copyright)CVE vulnerability identifiers (cve)Tor v3 addresses (onionAddress)Social network handles (facebookHandle, githubHandle, instagramHandle,
linkedinHandle, pinterestHandle, telegramHandle, twitterHandle, whatsappHandle,
youtubeHandle, youtubeChannel)Advertisement/analytics identifiers (googleAdsense, googleAnalytics, googleTagManager)Blockchain addresses (bitcoin, bitcoincash, cardano, dashcoin, dogecoin, ethereum, litecoin, monero, ripple, tezos, tronix, zcash)Payment addresses (webmoney)Chinese Internet Content Provider licenses (icp)Bank account numbers (iban)Trademarks (trademark)Universal unique identifiers (uuid)Android package name (packageName)Spanish NIF identifiers (nif)Command Line UsageTo find IOCs in a given file just provide the-f (--file)option.
By default, found IOCs are printed to stdout,
defanged IOCs are rearmed, and
IOCs are deduplicated so they only appear once.iocsearcher-ffile.pdf
iocsearcher-fpage.html
iocsearcher-finput.txtYou can use the-o (--output)option to place IOCs to a file instead
of stdout:iocsearcher-ffile.pdf-oiocs.txtBy default all regexp are applied to the input.
If you are only interested in some specific IOC types,
it is more efficient to specify those using
the-t (--target)option, which can be applied multiple times:iocsearcher-ffile.pdf-turl-temailYou can also search for IOCs in all files in a directory using
the-d (--dir)option.
IOCs extracted from each file will be placed in their own .iocs file.
You can also place all IOCs founds across the input files
in the same output file by also adding the-o (--output)option:iocsearcher-ddirectoryWithFiles-oall.iocsIn HTML files, only the readable text is examined
(i.e., think of the text shown by Firefox's Reader View).
If you want to scan the whole HTML content you can use the-r (--raw)option:iocsearcher-fpage.html-rIf you have a file that you want to interpret as text avoiding
filetype detection, you can use the-F (--forcetext)option:iocsearcher-finput.txt-FYou can store the text extracted from a PDF/HTML file using the-T (--text)option, which will produce a .text file for each input file:iocsearcher-ffile.pdf-TBy default IOCs are deduplicated, you can instead output the offset of
each IOC without deduplication by using the-v (--verbose)option:iocsearcher-ffile.pdf-vYou can also produce a ranking of IOCs by number of appearances
(without deduplication) by using the-C (--count)option:iocsearcher-ffile.pdf-C-orank.iocsLibrary UsageYou can also useiocsearcheras a library by creating aSearcherobject and then invoking the functionssearch_datato identify rearmed and deduplicated IOCs andsearch_rawto identify all matches, their offsets, and the defanged string.
TheSearcherobject needs to be created only once to parse the regexps.
Then, it can be reused to find IOCs in multiple input strings.python3
>>>importiocsearcher
>>>fromiocsearcher.searcherimportSearcher
>>>test='Find this email contact[AT]example[dot]com'>>>searcher=Searcher()>>>searcher.search_data(test){('email','contact@example.com'),('fqdn','example.com')}>>>searcher.search_data(test,targets={'email'}){('email','contact@example.com')}>>>searcher.search_raw(test)[('email','contact@example.com',16,'contact[AT]example[dot]com'),('fqdn','example.com',27,'example[dot]com')]You can also open a document without needing to provide its type,
get its text, and then use aSearcherobject to search for IOCs in the text.
For example, if you have a file calledfile.pdfyou can do:python3
>>>importiocsearcher
>>>fromiocsearcher.documentimportopen_document
>>>fromiocsearcher.searcherimportSearcher
>>>doc=open_document("file.pdf")>>>text,_=doc.get_text()ifdocisnotNoneelse"">>>searcher=Searcher()>>>searcher.search_data(text)If the file is not a PDF, HTML, or text document,open_documentthrows a warning and returns NoneDefang and RearmMany security reports defang (i.e., remove the teeth from) malicious
indicators, especially network indicators such as URLs, domains,
IP addresses, and email addresses.
This practice helps to prevent users from inadvertently clicking on a
malicious indicator and start a network connection to it.
Defanged indicators do not follow the indicator specification and thus
require relaxed regular expressions to detect them.iocsearchersupports some popular defang operations
and rearms the IOCs by default so that deduplication works even if the
same IOC has been defanged in different ways.
However, it is not possible to support all defang operations,
as every analyst can come up with their own.
If you thinkiocsearcheris missing support for some popular
defang operation, let us know by providing pointers to reports that use them.Customizing the Regular Expressionsiocsearcherreads its regular expressions from an INI configuration file.
If you want to modify a regexp, add a regexp,
change the IOC type associated to a regexp, or disable validation
for an existing regexp, you can create a copy of thepatterns.inifile in the GitHub repo,
edit your copy, and pass it as input toiocsearcherusing the-P (--patterns)option:iocsearcher-ffile.pdf-Pmypatterns.iniNote that if you add a new regexp, the output will be the outermost group
if a group exists, and the whole match if the regexp has no groups.Related ToolsThere exist multiple other open-source IOC extraction tools
and we developed iocsearcher to improve on those.
In ourFGCS journal paperwe propose a novel evaluation methodology for IOC extraction tools and
apply it to compareiocsearcherwith the following tools:Jager(Python)IOC-parser(Python)Cacador(Go)CyObstract(Python)IOC Finder(Python)IOC Extract(Python)IOC-Extractor(Python)We believe the results showiocsearcherperforms generally best,
but that is up to you to judge.
We encourage you to read our paper if you have questions about howiocsearchercompares with the above tools and to try
the above tools ifiocsearcherdoes not meet your goals.FilteringTechnically speaking,iocsearcheris an indicator extraction tool,
i.e., it extracts indicators regardless if they are benign or malicious.
Currently,iocsearcher,
similar to most other tools mentioned above,
does not differentiate malicious indicators (i.e., IOCs) from benign indicators.
For example, it will extract all URLs in the given input,
regardless if they are malicious or benign.Filtering of benign indicators is typically application-specific,
so we prefer to keep it as a separate step.
Such filtering is oftentimes performed with blocklists or through
Natural Language Processing (NLP) techniques.Licenseiocsearcheris released under the MIT licenseThis repository includes Base58 decoding code from themonero-pythonproject.
That code is located in the iocsearcher/monero folder and it is
licensed under BSD 3-Clause.ReferencesThe design and evaluation ofiocsearcherand the comparison with prior
IOC extraction tools are detailed in ourFGCS journal paper:Juan Caballero, Gibran Gomez, Srdjan Matic, Gustavo Sánchez,
Silvia Sebastián, and Arturo Villacañas.GoodFATR: A Platform for Automated Threat Report Collection and
IOC Extraction.In Future Generation Computer Systems, 2023.ContributorsThe main developer and maintainer foriocsearcheris Juan Caballero.
Other members of the MaliciaLab at theIMDEA Software Institutehave contributed fixes and helped with testing:
Gibran Gomez, Silvia Sebastian, Srdjan Matic