Package: certstream-analytics
Certstream + AnalyticsInstallationThe package can be installed fromPyPIpip install certstream-analyticsQuick usagebin/domain_matching.py--domainsdomains.txt--dump-locationcertstream.txt# The file domains.txt contains the list of domains that we want to monitor# for matches (domains with similar names). For examples, a file with only# two entries:## gmail.com# facebook.com## will match any domains that contains gmail or facebook keywords.## All the records consumed from certstream will be kept in certstream.txtAPIimporttimefromcertstream_analytics.analysersimportWordSegmentationfromcertstream_analytics.analysersimportIDNADecoderfromcertstream_analytics.analysersimportHomoglyphsDecoderfromcertstream_analytics.transformersimportCertstreamTransformerfromcertstream_analytics.storagesimportElasticsearchStoragefromcertstream_analytics.streamimportCertstreamAnalyticsdone=False# These analysers will be run in the same orderanalyser=[IDNADecoder(),HomoglyphsDecoder(),WordSegmentation(),]# The following fields are filtered out and indexed:# - String: domain# - List: SAN# - List: Trust chain# - Timestamp: Not before# - Timestamp: Not after# - Timestamp: Seentransformer=CertstreamTransformer()# Indexed the data in Elasticsearchstorage=ElasticsearchStorage(hosts=['localhost:9200'])consumer=CertstreamAnalytics(transformer=transformer,storage=storage,analyser=analyser)# The consumer is run in another thread so this function is non-blockingconsumer.start()whilenotdone:time.sleep(1)consumer.stop()IDNA decoderThis analyser decode IDNA domain name into Unicode for further processing
downstream.  Normally, it will be the very first analyser to be run.  If
the analyser encounters a malform IDNA domain string, it will keep the
domain as it is.fromcertstream_analytics.analysersimportIDNADecoderdecoder=IDNADecoder()# Just an example dummy recordrecord={'all_domains':['xn--f1ahbgpekke1h.xn--p1ai',]}# The domain name will now become 'укрэмпужск.рф'print(decoder.run(record))Homoglyphs decoderThere are lots of phishing websites that utilizehomoglyphsto lure the victims.  Some common examples include 'l' and 'i' or the
Unicode character RHO '𝞀' and 'p'.  The homoglyphs decoder uses the excellentconfusable_homoglyphsto
generate all potential alternative domain names in ASCII.fromcertstream_analytics.analysersimportHomoglyphsDecoder# If the greedy flag is set, all alternative domains will be returneddecoder=HomoglyphsDecoder(greed=False)# Just an example dummy recordrecord={'all_domains':[# MATHEMATICAL MONOSPACE SMALL P'*.𝗉aypal.com',# MATHEMATICAL SAN-SERIF BOLD SMALL RHO'*.𝗉ay𝞀al.com',]}# The domain name will now be converted to '*.paypal.com' with the ASCII# character pprint(decoder.run(record))Aho-CorasickA domain and its SAN from Certstream will be compared against a list of
most populardomains(from OpenDNS) using Aho-Corasick algorithm.  This is a simple check to
remove some of the most obvious phishing domains, for examples,www.facebook.com.msg40.sitewill match withfacebookcausefacebookis in the above list of most
popular domains (I wonder how long it is going to last).fromcertstream_analytics.analysersimportAhoCorasickDomainMatchingfromcertstream_analytics.reporterimportFileReporter# Print the list of matching domainsreporter=FileReporter('matching-results.txt')withopen('opendns-top-domains.txt'))asfhandle:domains=[line.rstrip()forlineinfhandle]# The list of domains to match againstdomain_matching_analyser=AhoCorasickDomainMatching(domains)consumer=CertstreamAnalytics(transformer=transformer,analyser=domain_matching_analyser,reporter=reporter)# Need to think about what to do with the matching resultconsumer.start()whilenotdone:time.sleep(1)consumer.stop()Word segmentationIn order to improve the accuracy of the matching algorithm, we segment
the domains into English words usingwordsegment.fromcertstream_analytics.analysersimportWordSegmentationwordsegmentation=WordSegmentation()# Just an example dummy recordrecord={'all_domains':['login-appleid.apple.com.managesupport.co',]}# The returned output is as follows:## {#   'analyser': 'WordSegmentation',#   'output': {#     'login-appleid.apple.com.managesuppport.co': [#       'login',#       'apple',#       'id',#       'apple',#       'com',#       'manage',#       'support',#       'co'#     ],# },#print(decoder.run(record))Features generatorA list of features for each domain will also be generated so that they
can be used for classification jobs further downstream.  The list
includes:The number of dot-separated fields in the domain, for example,www.google.comhas 3.The overall length of the domain in characters.The length of the longest dot-separate field .The length of the TLD, e.g. .online (6) or .download (8) is longer than .com (3).The randomness level of the domain.Nostrilpackage is used to check how many words as returned by the WordSegmentation
analyser are non-sense.