Package: blocklistsaggregator
IP Block Lists AggregatorA Python tool that downloads IP block lists from various sources and builds configurations for network equipments and firewalls.InstallationInstallation usingpip:$pipinstallblocklistsaggregatorEditable Installation using your GitHub forked repository andvirtualenv:$mkdirblocklistsaggregator$cdblocklistsaggregator$virtualenvvenv$sourcevenv/bin/activate$pipinstall-egit+https://github.com/YOUR_USERNAME/blocklistsaggregator.git#egg=blocklistsaggregatorUsageIt’s a command line tool, the--helpis your friend! Some examples are worth a thousand words.Download and display entries from all the configured block lists:$blocklistsaggregator.pyOnly fromRansomware Tracker RW_IPBLandDROP:$blocklistsaggregator.py--listsrw_ipbldropDownload entries from all the configured lists and save them in JSON format intoall.json:$blocklistsaggregator.py-fjson-oall.jsonRead the previously saved entries fromall.jsonand display them in a Cisco IOS prefix-list style:$blocklistsaggregator.py-iall.json-fcisco-iosFrom the previously saved entries, filter out those falling in 6.0.0.0/8 and those with a prefix-len shorter than /24 and save them intocisco.aclin a Cisco ACL style with nameBADGUYS:$blocklistsaggregator.py-iall.json--exclude6.0.0.0/8--exclude-ipv4-shorter-than24-ocisco.acl-fcisco-ios--cisco-cfg-elementacl_source--cisco-cfg-element-nameBADGUYSPrepare anip route <network> <mask> null0command for each IPv4 entry inDROP:$blocklistsaggregator.py--listsdrop-4--lines-format"ip route {network} {netmask} null0"Download standard block lists and output them in a Mikrotik address-list format intoaddMalwareIPs.rsc; save lists into/tmpand, in case of failure during one of the next executions, reuse them to build the output:$blocklistsaggregator--outputaddMalwareIPs.rsc--output-formatmikrotik--mikrotik-address-list-nameaddressListMalware--lists-storage-dir/tmp/--recover-from-fileLoggingError logging and reporting can be configured in order to have feedback about BlockListsAggregator’s activity. The--logging-config-fileoption can be set to the path of a configuration file inPython’s logging.fileConfig() format. An example is provided within thedistrib/log.inifile (here the file hosted on GitHub).Source block listsThe following block lists are currenly implemented:rw_ipbl,Ransomware Tracker RW_IPBLrw_dombl,Ransomware Tracker RW_DOMBL(please read below)rw_urlbl,Ransomware Tracker RW_URLBL(please read below)drop,Spamhaus DROPdrop_v6,Spamhaus DROPv6edrop,Spamhaus EDROPfeodo_badip,Feodo BadIPfeodo_ip,Feodo IPpalevo,Palevo C&Czeus,ZeuSbambenek_c2,Bambenek Consulting C2 master feedWarning for RW_DOMBL and RW_URLBL: the program extracts the domain names reported into these lists to resolve the IP addresses and uses them for the output. This may result in an overblocking behaviour because these filters should be applied with a more granular level than layer-3 addresses. These lists are not used by default unless explicitly given via the command line–listsor–lists-includearguments.A list of block-lists can be found onhttp://iplists.firehol.org/Output optionsThe following output formats are currenly implemented:JSONlines (with macros)Cisco IOS prefix-listCisco IOS ACL (source-based, destination-based, permit/deny actions)Mikrotik RouterOS address-listStatusThis tool is currently inbeta: some field tests have been done but it needs to be tested deeply and on more scenarios.Moreover, contributions (fixes to code and to grammatical errors, typos, new features) are very much appreciated.Bug? Issues?But also suggestions? New ideas?Please create an issue on GitHub athttps://github.com/pierky/blocklistsaggregator/issuesAuthorPier Carlo Chiodi -https://pierky.comBlog:https://blog.pierky.comTwitter:@pierkyChangelog0.5.1Fix a bug in the packaging system.0.5.0Better empty lines detection for RW_IPBL.Add–lists-includeand–lists-excludearguments.Addrw_domblandrw_urlbllists (Ransomware Tracker RW_DOMBL and RW_URLBL).Warning: the program extracts the domain names reported into these lists to resolve the IP addresses and uses them for the output. This may result in an overblocking behaviour because these filters should be applied with a more granular level than layer-3 addresses. These lists are not used by default unless explicitly given via the command line–listsor–lists-includearguments.0.4.1Fix issue with RW_IPBL entries counter.It seems that RW_IPBL is having some issues with the number of entries reported in the last line.
If an empty line is found it’s counted as an entry, so last line’s counter reports a wrong number.
Trying to mitigate this behaviour.0.4.0Adddrop_v6list (Spamhaus DROPv6).0.3.0Add–lists-storage-dirand–recover-from-filearguments to save lists into files and reuse them in case of failure of next updates.0.2.0Please note: JSON files saved with the previous version are not compatible with this one; blocklists must be downloaded and saved again to work.Keep track of source blocklist for each entry.Addbl_idsandbl_namesmacros to thelinesformatter.Add a comment containing the source blocklist to each Mikrotik RouterOS address-list entry.0.1.0First release (beta)