Package: iocextract
iocextractIndicator of Compromise(IOC) extractor for some of the most commonly ingested artifacts.Table of contentsOverviewThe ProblemOur SolutionExample Use CaseInstallationUsageLibraryCommand Line InterfaceHelpful InformationFAQMore DetailsCustom RegexRelated ProjectsContributingOverviewTheiocextractpackage is a library and command line interface (CLI) for extracting URLs, IP addresses, MD5/SHA hashes, email addresses, and YARA rules from text corpora. It allows for you to extract encoded and "defanged" IOCs and optionally decode or refang them.The ProblemIt is common practice for malware analysts or endpoint software to "defang" IOCs such as URLs and IP addresses, in order to prevent accidental exposure to live malicious content. Being able to extract and aggregate these IOCs is often valuable for analysts. Unfortunately, existing "IOC extraction" tools often pass right by them, as they are not caught by standard regex.For example, the simple defanging technique of surrounding periods with brackets:127[.]0[.]0[.]1Existing tools that use a simple IP address regex will ignore this IOC entirely.Our SolutionBy combining specially crafted regex with some custom post-processing, we are able to both detect and deobfuscate "defanged" IOCs. This saves time and effort for the analyst, who might otherwise have to manually find and convert IOCs into machine-readable format.Example Use CaseMany Twitter users post C2s or other valuable IOC information with defanged URLs.
For example,this tweet from @InQuest:Recommended reading and great work from @unit42_intel:
https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/ ...
InQuest customers have had detection for threats delivered from hotfixmsupload[.]com
since 6/3/2017 and cdnverify[.]net since 2/1/18.If we run this through the extractor, we can easily pull out the URLs:https://researchcenter.paloaltonetworks.com/2018/02/unit42-sofacy-attacks-multiple-government-entities/
hotfixmsupload[.]com
cdnverify[.]netPassing inrefang=Trueat extraction time would remove the obfuscation, but since these are real IOCs, let's leave them defanged in our documentation.InstallationYou may need to install the Python development headers in order to install theregexdependency. On Ubuntu/Debian-based systems, try:sudoapt-getinstallpython-devThen installiocextractfrom pip:pipinstalliocextractIf you have problems installing on Windows, try installingregexdirectly by downloading theappropriate wheel from PyPIand installing viapip:pipinstallregex-2018.06.21-cp27-none-win_amd64.whlUsageLibraryTry extracting some defanged URLs:importiocextractcontent=\"""I really love example[.]com!All the bots are on hxxp://example.com/bad/url these days.C2: tcp://example[.]com:8989/bad"""forurliniocextract.extract_urls(content):print(url)# Output# hxxp://example.com/bad/url# tcp://example[.]com:8989/bad# example[.]com# tcp://example[.]com:8989/badNOTE: Some URLs may show up twice if they are caught by multiple regexes.If you want, you can also "refang", or remove common obfuscation methods from IOCs:importiocextractforurliniocextract.extract_urls(content,refang=True):print(url)# Output# http://example.com/bad/url# http://example.com:8989/bad# http://example.com# http://example.com:8989/badIf you don't want to defang the extracted IOCs at all during extraction, you can disable this as well:importiocextractcontent=\"""http://example.com/bad/urlhttp://example.com:8989/badhttp://example.comhttp://example.com:8989/bad"""forurliniocextract.extract_urls(content,defang=False):print(url)# Output# http://example.com/bad/url# http://example.com:8989/bad# http://example.com# http://example.com:8989/badAllextract_*functions in this library return iterators, not lists. The benefit of this behavior is thatiocextractcan process extremely large inputs, with a very low overhead. However, if for some reason you need to iterate over the IOCs more than once, you will have to save the results as a list:importiocextractcontent=\"""I really love example[.]com!All the bots are on hxxp://example.com/bad/url these days.C2: tcp://example[.]com:8989/bad"""print(list(iocextract.extract_urls(content)))# ['hxxp://example.com/bad/url', 'tcp://example[.]com:8989/bad', 'example[.]com', 'tcp://example[.]com:8989/bad']Command Line InterfaceA command-line tool is also included:$iocextract-husage:iocextract[-h][--inputINPUT][--outputOUTPUT][--extract-emails][--extract-ips][--extract-ipv4s][--extract-ipv6s][--extract-urls][--extract-yara-rules][--extract-hashes][--custom-regexREGEX_FILE][--refang][--strip-urls][--wide]AdvancedIndicatorofCompromise(IOC)extractor.Ifnoargumentsarespecified,thedefaultbehavioristoextractallIOCs.optionalarguments:-h,--helpshowthishelpmessageandexit--inputINPUTdefault:stdin--outputOUTPUTdefault:stdout--extract-emails--extract-ips--extract-ipv4s--extract-ipv6s--extract-urls--extract-yara-rules--extract-hashes--custom-regexREGEX_FILEfilewithcustomregexstrings,oneperline,withonecapturegroupeach--refangdefault:no--strip-urlsremovepossiblegarbagefromtheendofurls.default:no--widepreprocessinputtoallowwide-encodedcharactermatches.default:noNOTE: Only URLs, emails, and IPv4 addresses can be "refanged".Helpful InformationFAQAre you...Q. Extracting possibly-defanged IOCs from plain text, like the contents of tweets or blog posts?A. Yes! This is exactly what iocextract was designed for, and where it performs best. Want to go a step farther and automate extraction and storage? Check outThreatIngestor.Q. Extracting URLs that have been hex or base64 encoded?A. Yes, but the CLI might not give you the best results. Try writing a Python script and callingiocextract.extract_encoded_urlsdirectly.Note: You will most likely end up with extra garbage at the end of URLs.Q. Extracting IOCs that have not been defanged, from HTML/XML/RTF?A. Maybe, but you should consider using the--strip-urlsCLI flag (or thestrip=Trueparameter in the library), and you may still get some extra garbage in your output. If you're extracting from HTML, consider using something likeBeautiful Soupto first isolate the text content, and then pass that to iocextract,like this.Q. Extracting IOCs that have not been defanged, from binary data like executables, or very large inputs?A. There is a very simplistic version of this available when running as a library, but it requires thedefang=Falseparameter and could potentially miss some of the IOCs. The regex in iocextract is designed to be flexible to catch defanged IOCs. If you're unable to collect the information you need, consider using something likeCacadorinstead.More DetailsThis library currently supports the following IOCs:IP AddressesIPv4 fully supportedIPv6 partially supportedURLsWith protocol specifier: http, https, tcp, udp, ftp, sftp, ftpsWith[.]anchor, even with no protocol specifierIPv4 and IPv6 (RFC2732) URLs are supportedHex-encoded URLs with protocol specifier: http, https, ftpURL-encoded URLs with protocol specifier: http, https, ftp, ftps, sftpBase64-encoded URLs with protocol specifier: http, https, ftpEmailsPartially supported, anchoring on@oratYARA rulesWith imports, includes, and commentsHashesMD5SHA1SHA256SHA512Telephone numbersCustom regexWith exactly one capture groupFor IPv4 addresses, the following defang techniques are supported:TechniqueDefangedRefanged.->[.]1[.]1[.]1[.]11.1.1.1.->(.)1(.)1(.)1(.)11.1.1.1.->\.1\.1\.1\.11.1.1.1Partial1[.1[.1.]11.1.1.1Any combination1.)1[.1.)11.1.1.1For email addresses, the following defang techniques are supported:TechniqueDefangedRefanged.->[.]me@example[.]comme@example.com.->(.)me@example(.)comme@example.com.->{.}me@example{.}comme@example.com.->_dot_me@example dot comme@example.com@->[@]me[@]example.comme@example.com@->(@)me(@)example.comme@example.com@->{@}me{@}example.comme@example.com@->_at_me at example.comme@example.comPartialme@} example[.comme@example.comAdded spacesme@example [.] comme@example.comAny combinationme @example [.)comme@example.comFor URLs, the following defang techniques are supported:TechniqueDefangedRefanged.->[.]example[.]com/pathhttp://example.com/path.->(.)example(.)com/pathhttp://example.com/path.->\.example\.com/pathhttp://example.com/pathPartialhttp://example[.com/pathhttp://example.com/path/->[/]http://example.com[/]pathhttp://example.com/pathCisco ESAhttp:// example .com /pathhttp://example.com/path://->__http__example.com/pathhttp://example.com/path://->:\\http:\\example.com/pathhttp://example.com/path:->[:]http[:]//example.com/pathhttp://example.com/pathhxxphxxp://example.com/pathhttp://example.com/pathAny combinationhxxp__ example( .com[/]pathhttp://example.com/pathHex encoded687474703a2f2f6578616d706c652e636f6d2f70617468http://example.com/pathURL encodedhttp%3A%2F%2fexample%2Ecom%2Fpathhttp://example.com/pathBase64 encodedaHR0cDovL2V4YW1wbGUuY29tL3BhdGgKhttp://example.com/pathNOTE: The tables above are not exhaustive, and other URL/defang patterns may also be extracted correctly. If you notice something missing or not working correctly, feel free to let us know via theGitHub Issues.The base64 regex was generated with@deadpixi'sbase64 regex tool.Custom RegexIf you'd like to use the CLI to extract IOCs using your own custom regex, create a plain text file with one regex string per line, and pass it in with the--custom-regexflag. Be sure each regex string includes exactly onecapture group.For example:http://(example\.com)/
(?:https|ftp)://(example\.com)/This custom regex file will extrac the domainexample.comfrom matching URLs. The(?: )noncapture group won't be included in matches.If you would like to extract the entire match, just put parentheses around your entire regex string, like this:(https?://.*?.com)If your regex is invalid, you'll see an error message like this:Error in custom regex: missing ) at position 5If your regex does not include a capture group, you'll see an error message like this:Error in custom regex: no such groupAlways use a single capture group when working with custom regex. Here's a quick example:[r'(my regex)',# This yields 'my regex' if the pattern matchesr'my (re)gex',# This yields 're' if the pattern matches]Using more than a single capture group can cause unexpected results. Check out this example:[r'my regex',# This doesn't yield anythingr'(my) (re)gex',# This yields 'my' if the pattern matches]Why? Because the result will always yield only the firstgroupmatch from each regex.For more complicated regex queries, you can combine capture and non-capture groups like so:[r'(?:my|your) (re)gex',# This yields 're' if the pattern matches]You can now compare the(?: )syntax for noncapture groups vs the( )syntax for the capture group.Related ProjectsIf iocextract doesn't fit your use case, several similar projects exist. Check out thedefangandindicators-of-compromisetags on GitHub, as well as:Cacadorin Goioc-extractorin JavaScriptCyobstractin PythonIf you'd like to automate IOC extraction, enrichment, export, and more, check outThreatIngestor.If you're working with YARA rules, you may be interested inplyara.ContributingIf you have a defang technique that doesn't make it through the extractor, or if you find any bugs, Pull Requests and Issues are always welcome. The library is released under a GPL-2.0license.Who's using iocextract?InQuestPacketTotalAre you using it? Want to see your site listed here? Let us know!