Package: threatrack-iocextract
threatrack_iocextract.pyExtracts IOCs (and other patterns) from text.How do I use it?Installfromhttps://pypi.orgviapip:pip install threatrack_iocextractviasetup.py:sudo python setup.py installUsageimportthreatrack_iocextracttext='hxxp://bad[.]com/  https://evil.com/foobar?a=1 asfdas\nsadfasf bob at example dot com'# extract IOCsiocs1=threatrack_iocextract.extract(text)# iocs1 is dict like `{ioctype1 : [ioc1, ioc2, ...], ioctype2 : [ioc3, ...], ...}`print(iocs1['url'][0])# prints https://evil.com/foobar?a=1print(iocs1['hostname'][0])# prints evil.com# extract defangled IOCsiocs2=threatrack_iocextract.extract_all(text)print(iocs2['url'][0])# prints http://bad.com/print(iocs2['hostname'][0])# prints bad.comprint(iocs2['email'][0])# prints bob@example.comHow does this work?threatrack_iocextract.pyThe main python program. The workflow is:load_patterns(): Read, expand and configure search patterns (automatically called on import).refang(text): Turns defangedtextinto "refanged" text, e.g. 'hxxp://bad[.]com/' becomes 'http://bad.com/'.extract(text): Extract IOCs fromtext. Returns a dict like{ioctype1 : [ioc1, ioc2, ...], ioctype2 : [ioc3, ...], ...}(see [How do I use it?])extract_all(text): Likeextract()but will also extract defanged IOCs.patterns/Contains the search pattern configuration.patterns/patterns.csvA tab separated list of search patterns.
This is loaded byload_patterns().
The format istab separated:ioctype	regexpattern	regexoptionsItmust always have 3 columns.For example:sha1	\b[a-f0-9]{40}\b	iwould search for SHA1 hashes between word boundaries.Possible options are:i(= re.I = case-insensitive)s(= re.S = dot all)All otherregexoptionsare ignored.regexoptionsmustbe set (even if just an empty character)!Expansionsallow you to reuse other sources in patterns. There are two types
of expansions:%%file:name.csv%%%%pattern:name%%file:expansions allow to include file contents in patterns.name.csvis a file with single column list of patterns. Any pattern containing
this expansion will replace%%file:name.csv%%with a regex trying to match each
line ofname.csv.For example, ifname.csvcontains:foo
bar
shootThe patterntest%%file:name.csv%%?blah' will be expanded totest(?:foo|bar|shoot)?blah'.
Seepatterns/{schemes,tlds}.csvfor how this can be useful.pattern:expansions allow to reusepreviouspatterns.For example, ifpatterns.csvcontains:.port	6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|6[0-4][0-9]{3}|[1-5][0-9]{4}|[1-9][0-9]{3}|[1-9][0-9]{2}|[1-9][0-9]|[0-9]	.
ipv4	(?:(?:25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})\.){3}(?:25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})(?:\:%%pattern:.port%%)?	.ipv4would be expanded to:(?:(?:25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})\.){3}(?:25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})(?:\:(?:6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|6[0-4][0-9]{3}|[1-5][0-9]{4}|[1-9][0-9]{3}|[1-9][0-9]{2}|[1-9][0-9]|[0-9]))?Names starting with.(dot) are private.They are not searched directly
but can be used in expansions.Gotchastlds.csvandschemes.csvmust be sorted longest patterns first to ensure longest match. Otherwise.cowould match before.com, etc.Are there any issues?Yes. Unfortunately, I can only list the known issues.Known issuesUnconditional refangleextract_all(text)will refang the text. This could potentially lead to altered
IOCs, e.g.I am at home.To do so ...would be altered toI am@home.To do so ...and thus lead to the email IOCam@home.to.Other possibilities are that IOCs get altered, e.g.http://example[.]com/foo[dot]bar/would refangle tohttp://example.com/foo.bar/.Unfortunately, it is very hard to fix this.Suggestions are welcome.Possible mitigation:Add IOC specific refangs. So, e.g., only when ectracting email IOCs
thes/at/@/refang gets applied.TODOFix IPv6 pattern, it overlaps with MAC addressesFix Hash and Bitcoin overlapIncrease whitelistFix MAC pattern to not match on fingerprintsFix extract_all() refanging breaks YARA extraction. Need to find a smart solution to extracting defanged IOCs. :(Add IOC sepcific refangs