Package: berserker_resolver
Fast mass dns resolver which can bypass loadbalancers.ContentsMotivationQuery backendSupported versionsInstallationResolver classResolver.resolveResolver.queryResolver.nameserversResolver.triesResolver.timeoutResolver.threadsResolver.qnameResolver.wwwResolver.www_combineResolver.verboseMotivationDNS servers can provide load balancing of many types. It can be simple round-robin or another algorithm that
depends on the implementation of a particular DNS server. SeeRFC 1794to understand the capabilities and flexibility of the DNS protocol. As a result, it is possible that when the ordinary
resolver is not able to getallIP addresses, e.g. DNS server can return small sets of IP addresses (or even only one IP)
with low TTL per request. Number and presence of addresses may vary depending on the load of servers. Important to note
that this information can be cached by other DNS servers and can be distorted. Getting all IP addresses problem arises when
you want to resolve many domains with sufficient accuracy which useful for blocking network purposes, e.g. websites filtering,
parental controls, etc.Let’s try to make a couple probes for www.twitter.com.;<<>>DiG9.10.2-P1<<>>@8.8.8.8www.twitter.com+nocomments+noquestion;(1serverfound);;globaloptions:+cmdwww.twitter.com.593INCNAMEtwitter.com.twitter.com.23INA199.16.156.70twitter.com.23INA199.16.156.198twitter.com.23INA199.16.156.102twitter.com.23INA199.16.156.6;;Querytime:5msec;;SERVER:8.8.8.8#53(8.8.8.8);;WHEN:SatAug0804:01:43MSK2015;;MSGSIZErcvd:122;<<>>DiG9.10.2-P1<<>>@8.8.8.8www.twitter.com+nocomments+noquestion;(1serverfound);;globaloptions:+cmdwww.twitter.com.596INCNAMEtwitter.com.twitter.com.26INA199.16.156.230twitter.com.26INA199.16.156.70twitter.com.26INA199.16.156.198twitter.com.26INA199.16.156.102;;Querytime:4msec;;SERVER:8.8.8.8#53(8.8.8.8);;WHEN:SatAug0804:01:44MSK2015;;MSGSIZErcvd:122You see different RR sets with small TTL. What about another public DNS?;<<>>DiG9.10.2-P1<<>>@77.88.8.8www.twitter.com+nocomments+noquestion;(1serverfound);;globaloptions:+cmdwww.twitter.com.30INCNAMEtwitter.com.twitter.com.30INA199.16.156.102twitter.com.30INA199.16.156.6twitter.com.30INA199.16.156.38twitter.com.30INA199.16.156.230;;Querytime:2msec;;SERVER:77.88.8.8#53(77.88.8.8);;WHEN:SatAug0804:04:05MSK2015;;MSGSIZErcvd:111Let’s try www.youtube.com.;<<>>DiG9.10.2-P1<<>>@8.8.8.8www.youtube.com+nocomments+noquestion;(1serverfound);;globaloptions:+cmdwww.youtube.com.21313INCNAMEyoutube-ui.l.google.com.youtube-ui.l.google.com.13INCNAMEwide-youtube.l.google.com.wide-youtube.l.google.com.13INA74.125.143.198;;Querytime:5msec;;SERVER:8.8.8.8#53(8.8.8.8);;WHEN:SatAug0804:06:08MSK2015;;MSGSIZErcvd:121;<<>>DiG9.10.2-P1<<>>@8.8.8.8www.youtube.com+nocomments+noquestion;(1serverfound);;globaloptions:+cmdwww.youtube.com.21599INCNAMEyoutube-ui.l.google.com.youtube-ui.l.google.com.299INCNAMEwide-youtube.l.google.com.wide-youtube.l.google.com.299INA173.194.71.198;;Querytime:6msec;;SERVER:8.8.8.8#53(8.8.8.8);;WHEN:SatAug0804:06:11MSK2015;;MSGSIZErcvd:121;<<>>DiG9.10.2-P1<<>>@84.200.70.40www.youtube.com+nocomments+noquestion;(1serverfound);;globaloptions:+cmdwww.youtube.com.55513INCNAMEyoutube-ui.l.google.com.youtube-ui.l.google.com.271INA216.58.209.46;;Querytime:41msec;;SERVER:84.200.70.40#53(84.200.70.40);;WHEN:SatAug0804:07:29MSK2015;;MSGSIZErcvd:94This outputs may be outdated soon, but it is only necessary to show the behavior of DNS. Any website can use
load balancer and so you are not able to do full resolve these sites.The solution is query many nameservers many times for each domain. Yes, it’s a bit clumsy, but works well enough
in many cases. The resolving should be performed in multiple threads, because resolving in one thread is slow,
especially in this case.And so Berserker Resolver is emerged.It’s worth noting that full resolving may be impossible because GEO load balancing or resolving can be occurred
“at the wrong time in the wrong place” when some servers are down and their IP addresses are excluded from DNS pool by fault
tolerance algorithm. If you need actual information you should schedule resolving attempts, maintain your DNS database,
maybe perform resolving from different networks/servers. There is no universal solution for that cases, but you can use Berserker
Resolver as the backend in your application.Query backendBerserker Resolver is usingdnspythonas query backend and so operates with its built-in types.Supported versionsPython 2.6Python 2.7Python 3.2Python 3.3Python 3.4InstallationThe best way:pip install berserker_resolverResolver classCore of the Berserker Resolver.Methods:resolvequeryProperties:nameserverstriestimeoutqnamethreadswwwwww_combineverboseProperties can be assign via constructor or directly to the object.Resolver.resolveResolve method. It takes list of domains and returns dictionary with results (dictionary of sets).fromberserker_resolverimportResolverdomains=['kernel.org','toster.ru']resolver=Resolver()result=resolver.resolve(domains)print(result)'''
    {
        'toster.ru': {
            <DNS IN A rdata: 178.248.236.52>
        },
        'kernel.org': {
            <DNS IN A rdata: 198.145.20.140>,
            <DNS IN A rdata: 199.204.44.194>,
            <DNS IN A rdata: 149.20.4.69>
        }
    }
'''Resolver.queryQuery method, wrapper arounddns.resolver.Resolver.queryfrom dnspython. It takes domain and nameserver,
and returns result of the query. Nameserver is optional, if not given, random fromResolver.nameserverswill be used.Can throw exception, see detailshere.fromberserker_resolverimportResolverresolver=Resolver()result=resolver.query('facebook.com')print(list(result))# [<DNS IN A rdata: 173.252.120.6>]# Query to the local dns.result=resolver.query('facebook.com','127.0.0.1')print(list(result))# [<DNS IN A rdata: 173.252.120.6>]Resolver.nameserversList of nameservers for resolving, each of them will be queried for particular domain.The larger the list, the more chances to get all IP addresses, but it increases
time  needed for resolving.Default is['8.8.8.8', '8.8.4.4', '77.88.8.8', '77.88.8.1', '84.200.69.80', '84.200.70.40']. There areGoogle Public DNS,Yandex.DNSandDNS.WATCH.Resolver.triesNumber of queries for each nameserver.If the number of times increases, the resolving accuracy increases too, but it also
increases time to resolving.Default is48.Resolver.timeoutThe total number of seconds to spend trying to get an answer to the query.Note that low timeout combined with high values ofResolver.triesandResolver.threadscan lead to
numerous timeout errors when nameserver does not have enough time to return a response.Default is3.Resolver.threadsNumber of threads.More threads lead to increase speed of resolving, but too many threads lead to threads switching overhead.
You should test different numbers and choose one suitable for your systems. Also be careful with large number of threads, you can
flood the DNS server. If you want to use crazy large amount of threads, checkstackoverflow threadand
increaseResolver.timeout.Default is512.Resolver.qnameDNS query type name.Default isA.Resolver.wwwEnables automatic addition/removal ofwwwprefix depending on the domain.fromberserker_resolverimportResolverdomains=['wikipedia.org','www.toster.ru']resolver=Resolver(www=True)result=resolver.resolve(domains)print(result)'''
    {
        'toster.ru': {
            <DNS IN A rdata: 178.248.236.52>
        },
        'www.wikipedia.org': {
            <DNS IN A rdata: 91.198.174.192>
        },
        'www.toster.ru': {
            <DNS IN A rdata: 178.248.236.52>
        },
        'wikipedia.org': {
            <DNS IN A rdata: 91.198.174.192>
        }
    }
'''Default isFalse.Resolver.www_combineEnables automatic combiningwwwprefixed domains with theirs non-wwwversions.fromberserker_resolverimportResolverdomains=['facebook.com','www.facebook.com']resolver=Resolver()result=resolver.resolve(domains)print(result)'''
    {
        'facebook.com': {
            <DNS IN A rdata: 173.252.120.6>
        },
        'www.facebook.com': {
            <DNS IN A rdata: 31.13.93.3>,
            <DNS IN A rdata: 31.13.91.2>,
            <DNS IN A rdata: 173.252.88.66>,
            <DNS IN A rdata: 31.13.64.1>
        }
    }
'''resolver.www_combine=Trueresult=resolver.resolve(domains)print(result)'''
    {
        'www.facebook.com': {
            <DNS IN A rdata: 173.252.120.6>
            <DNS IN A rdata: 31.13.93.3>,
            <DNS IN A rdata: 31.13.91.2>,
            <DNS IN A rdata: 173.252.88.66>,
            <DNS IN A rdata: 31.13.64.1>
        }
    }
'''Powerful use case is combine this property together withResolver.www.fromberserker_resolverimportResolverdomains=['facebook.com']resolver=Resolver(www=True,www_combine=True)result=resolver.resolve(domains)print(result)'''
    {
        'www.facebook.com': {
            <DNS IN A rdata: 173.252.120.6>
            <DNS IN A rdata: 31.13.93.3>,
            <DNS IN A rdata: 31.13.91.2>,
            <DNS IN A rdata: 173.252.88.66>,
            <DNS IN A rdata: 31.13.64.1>
        }
    }
'''Default isFalse.Resolver.verboseThis property enables error reporting, e.g. nxdomain, noanswer, etc.Resolver.resolvenormally returns
dictionary of sets with resolved domains, but with this option it returns dictionary with two keys:successerrorfromberserker_resolverimportResolverdomains=['nonexistent.domain','facebook.com']resolver=Resolver(verbose=True)result=resolver.resolve(domains)print(result)'''
    {
        'success': {
            'facebook.com': {
                <DNS IN A rdata: 173.252.120.6>
            }
        },
        'error': {
            'nonexistent.domain': {
                '77.88.8.1': NXDOMAIN(),
                '8.8.4.4': NXDOMAIN(),
                '8.8.8.8': NXDOMAIN(),
                '77.88.8.8': NXDOMAIN()
            }
        }
    }
'''result['success']is dictionary with successfully resolved domains, as if withoutResolver.verbose.result['error']is dictionary with unsuccessfully resolved domains where each key contains another dictionary
with per nameserver exception. Exceptions comes from dnspython backend asdns.exception.DNSExceptionsubclasses.
Check outdnspython docsfor more
information about built-in exceptions.Note that particular domain can be placed in both dictionaries, because some nameservers can return answer and some not.fromberserker_resolverimportResolverdomains=['facebook.com']# 216.239.32.10 is ns1.google.comresolver=Resolver(nameservers=['216.239.32.10','8.8.8.8'],verbose=True)result=resolver.resolve(domains)print(result)'''
    {
        'success': {
            'facebook.com': {
                <DNS IN A rdata: 173.252.120.6>
            }
        },
        'error': {
            'facebook.com': {
                '216.239.32.10': NoNameservers()
            }
        }
    }
'''Default isFalse.