Package: dnsurlsearch
DNSurlSearchTable of ContentsInstallation[Getting started](#Getting started)WhitelistBlacklistLicenseInstallationpip install dnsurlsearchGetting startedThis package allow you to build a whitelist like that :rr1.sn-cv0tb0xn-uane.googlevideo.com.
rr2.sn-cv0tb0xn-uane.googlevideo.com.
rr3.sn-cv1tb1xn-uane.googlevideo.com.or a blacklist like that :0.0.0.0 rr1.sn-cv0tb0xn-uane.googlevideo.com.
0.0.0.0 rr2.sn-cv0tb0xn-uane.googlevideo.com.
0.0.0.0 rr3.sn-cv1tb1xn-uane.googlevideo.com.that you can save in a file and use with PiHole, for example, to filter urls.The selection of urls is based on DNS protocol via exploration of cache file of DNS server like bind9 software
if you have installed bind9 on your localhost or via a sniffer in command line like tcpdump.Initially the aim of this package was to filter advertisements on youTube video based on urls like *.googlevideo.com. Unfortunatedly,
some of these urls are not advertisements, so that we also need whitelist.WhitelistWhitelist generation is supposed to be manually. The script is launched and severals youTube video are played without any advertisement
to get url like *.googlevideo.com. without advertisement.Here is an example of script example.py (need to configure sudo to get rid of password) :fromdnsurlsearchimportDNSurlSearch# First create url filterwhitelist=DNSurlSearch.UrlFilter()# Create the cachecache=DNSurlSearch.CacheDns()# Set the SnifferCacheHandler to snif DNS protocolh=DNSurlSearch.SnifferCacheHandler('MANUAL')h.set_cache_file_name('/tmp/log')# The sniffer used is tcpdump# The executable file tcpdump_cache_dns.sh contains the following command :# /usr/bin/tcpdump -n -s 0 port 53 > $1 &# to sniff DNS protocolh.set_start_cmd('sudo /home/<user>/tcpdump_cache_dns.sh')# The executable file k_tcpdump_cache_dns.sh contains the following command :# pkill tcpdump# to kill processh.set_kill_cmd('sudo /home/<user>/k_tcpdump_cache_dns.sh')cache.set_handler(h)# Set patterns to search in the cachecache.set_filter('.googlevideo.com.')# Launch the sniffer to get url and search urlsnew_cache=cache.get_cache_dns()# Add urls found in the whitelistwhitelist.add(new_cache)# Save the whitelist in a filewhitelist.write_url('whitelist')You can launch this script with debug option (debug, info, warning, error, critical)$python3example.pydebug2024-02-2610:02:30,531dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py278-----------------CacheDns.set_handler()()-----------------2024-02-2610:02:30,531dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py282-----------------CacheDns.set_filter()()------------------2024-02-2610:02:30,531dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py284filters:['.googlevideo.com.']2024-02-2610:02:30,531dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py526---------------------init_dns_cache()---------------------2024-02-2610:02:30,531dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py528sudo/home/<user>/tcpdump_cache_dns.sh/tmp/log
Typeentertostoptcpdump...2024-02-2610:02:32,922dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py537sudo/home/<user>/k_tcpdump_cache_dns.sh2024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py313----------------CacheHandler._search_url()----------------2024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py314pattern:['.* (.+\\.googlevideo\\.com\\.).*']-filename:Found_url.txt2024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py328Lecacheestvide2024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py160---------------------UrlFilter.add()----------------------2024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py161Numberofurltoadd:02024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py143------------------UrlFilter.write_url()-------------------2024-02-2610:02:32,941dnsurlsearch.DNSurlSearchDEBUGDNSurlSearch.py144Numberofurlstowrite:0-filename:whitelistBlacklistBlacklist is supposed to be automatic. A script is launched by crontab every 10 minutes via crontab for example.Here is an example of script that can be launched on a host
where a DNS server like bind9 is installed:# To get some measurements of urls found in the cachestats=CacheDnsStat('rasp','root@jeudy','nicolas@jeudy.mooo.com','Filtered url stats')# First create url filter# '0.0.0.0' parameter is the IP address to add in the file for each urlblacklist=UrlFilter('0.0.0.0','black_list')# treatment#  - when reading blacklist file : remove ip_address to get url only#  - when writting blacklist file : add ip_addressblacklist.set_treatment(BlackListFilterTreatment())nb_url_start=blacklist.read_url('blacklist')# Create the cachecache=CacheDns()# Set the BindCacheHandler to snif cache of the DNS server bind9h=BindCacheHandler()# Location of the file where rndc dump the cache of DNS serverh.set_cache_file_name('/var/cache/bind/named_dump.db')# Set the command to dump the cache of DNS serverh.set_start_cmd('/usr/sbin/rndc dumpdb -cache')cache.set_handler(h)# Set patterns to search in the cachecache.set_filter('.googlevideo.com.')cache.set_filter('.gslb.com.')# Launch the update of cache and search urlsnew_cache=cache.get_cache_dns()# Add urls found in the blacklistblacklist.add(new_cache)# Check urls and remove url from blacklist if present in whitelistblacklist.check(whitelist.get_url())nb_url_end=blacklist.write_url('blacklistbis')stats.save_stats(nb_url_start,nb_url_end,1)Here is an example of another script that can be launched on a host
where DNS protocol can be sniffed with tcpdump :Licensednsurlsearchis distributed under the terms of theMITlicense.