Package: dnstap-receiver
Dnstap streams receiverIf you want better performance and more features, please to use thednscollectortool written in Go.This Python module acts as adnstapstreams receiver for DNS servers.
Input streams can be a unix, tcp or raw socket.
The output is printed directly on stdout or sent to remote tcp address
in JSON, YAML or one line text format and more.If you want to use thednstapfeature of your dns server, please to read the following pageDnstap: How to enable it on main dns serversTable of contentsInstallationPyPIDocker HubInputs handlerTCP socket (server)TCP socket (client)Unix socketRaw socket (sniffer)Outputs handlerStdoutFileTCPSyslogMetricsDnstapKafkaRabbitMQPostgreSQLElasticsearchMore optionsExternal config fileVerbose modeFiltering featureGeoIP supportStatisticsCountersTablesMetricsBuild-in WebserverConfigurationSecurityHTTP APIBenchmarkDevelopmentAboutInstallationPyPIDeploy the dnstap receiver in your DNS server with the pip command.pipinstalldnstap_receiverAfter installation, you can execute thednstap_receiverto start-it.Usage:usage: dnstap_receiver [-h] [-l L] [-p P] [-u U] [-v] [-c C]

optional arguments:
  -h, --help  show this help message and exit
  -l L        IP of the dnsptap server to receive dnstap payloads (default: '0.0.0.0')
  -p P        Port the dnstap receiver is listening on (default: 6000)
  -u U        read dnstap payloads from unix socket
  -v          verbose mode
  -c C        external config fileDocker HubPull the dnstap receiver image from Docker Hub.dockerpulldmachard/dnstap-receiver:latestDeploy the containerdockerrun-d-p6000:6000-p8080:8080--name=dnstap01dmachard/dnstap-receiverAdd the following argument to your container if you want to provide your ownconfigurationfile.-v/home/dnstap.conf:/etc/dnstap_receiver/dnstap.confFollow containers logsdockerlogsdnstap01-fInputs handlerSeverals inputs handler are supported to read incoming dnstap messages:TCP socket (server)TCP socket (client)Unix socketRaw socket (sniffer)TCP socket (server)The TCP socket input enable to receive dnstap messages from multiple dns servers.
This is the default input if you execute the binary without arguments.
The receiver is listening onlocalhostinterface and the tcp port6000.
You can change binding options with-land-parguments../dnstap_receiver -l 0.0.0.0 -p 6000You can also activateTLSon the socket, add the following config as external config file
to activate the tls support, configure the path of the certificate and key to use.input:tcp-socket:# enable tls supporttls-support:true# provide certificate server pathtls-server-cert:/etc/dnstap_receiver/server.crt# provide certificate key pathtls-server-key:/etc/dnstap_receiver/server.keyThen execute the dnstap receiver with the configuration file:./dnstap_receiver -c /etc/dnstap-receiver/dnstap.confTCP socket (client)The TCP socket input enable to receive dnstap messages from remote dns servers.
The communication is initiated by the dnstap receiver.Configure this input as belowinput:# tcp clienttcp-client:# enable or disableenable:true# retry interval in seconds to connectretry:1# remote dns server addressremote-address:10.0.0.2# remote dns server portremote-port:6000Unix socketThe unix socket input enables read dnstap message from a unix socket.
Configure the path of the socket with the-uargument../dnstap_receiver -u /var/run/dnstap.sockRaw socket (sniffer)This input enable to sniff a network interface.
Configure this input as below, you need to provide the name of your interface and associated ip.input:# sniff dns messages from network interfacesniffer:# enable or disableenable:true# interface name to sniffeth-name:ens18# ip interface to sniffeth-ip:[10.0.0.2]# dnstap identitydnstap-identity:sniffer# sniff on the list of dns portdns-port:[53]# record incoming dns client queriesrecord-client-query:true# record outgoing dns client responsesrecord-client-response:trueOutputs handlerOutputs handler can be configured to forward messages in several modes.StdoutFileMetricsTCPSyslogDnstapElasticsearchStdoutThis output enables to forward dnstap messages directly to Stdout.
Add the following configuration as external config to activate this output:output:stdout:# enable or disableenable:true# format available text|json|yamlformat:textOutput can be formatted in different way:text (default one)jsonyamlText format:2020-09-16T18:51:53.547352+00:00 lb1 CLIENT_QUERY NOERROR - - INET UDP 43b ns2.google.com. A -
2020-09-16T18:51:53.591736+00:00 lb2 CLIENT_RESPONSE NOERROR - - INET UDP 59b ns2.google.com. A 0.048JSON format:{"identity":"lb1","qname":"www.google.com.","rrtype":"A","query-ip":"192.168.1.114","message":"CLIENT_QUERY","family":"INET","protocol":"UDP","query-port":42222,"length":43,"timestamp":"2020-09-16T18:51:53.591736+00:00","rcode":"NOERROR","id":33422,"flags":"RD","latency":"-"}YAML format:identity:lb1rcode:NOERRORlength:49message:CLIENT_QUERYfamily:INETqname:dns4.comlaude-dns.eu.rrtype:AAAAquery-ip:'-'query-port:'-'timestamp:'2020-09-16T18:51:53.591736+00:00'protocol:UDPid:33422flags:RDlatency:'-'FileThis output enables to forward dnstap messages directly to a log file.
Add the following configuration as external config to activate this output:# forward to log filefile:# enable or disableenable:true# format available text|json|yamlformat:text# log file path or null to print to stdoutfile:/var/log/dnstap.log# max size for log filefile-max-size:10M# number of max log filesfile-count:10If you are running the dnstap in a container, follow this procedure to save logs in your host instead of the container.First one, create the folder in the host:mkdir /var/dnstap/
chown 1000:1000 /var/dnstap/Create the following configuration for your dnstap receivertrace:
    verbose: true
output:
  stdout:
    enable: false
  file:
    enable: true
    format: text
    file: /home/dnstap/logs/dnstap.log
    file-max-size: 10M
    file-count: 10Then execute the container with volumedocker run -d -p 6000:6000 -p 8080:8080 -v ${PWD}/dnstap.conf:/etc/dnstap_receiver/dnstap.conf \
-v /var/dnstap:/home/dnstap/logs/ --name=dnstap01 dmachard/dnstap-receiverTCPThis output enables to forward dnstap message to a remote tcp collector.
Add the following configuration as external config to activate this output:output:# forward to remote tcp destinationtcp-socket:# enable or disableenable:true# format available text|json|yamlformat:text# delimiterdelimiter:"\n"# retry interval in seconds to connectretry:5# remote ipv4 or ipv6 addressremote-address:10.0.0.2# remote tcp portremote-port:8192SyslogThis output enables to forward dnstap message to a syslog server.
Add the following configuration as external config to activate this output:output:syslog:# enable or disableenable:false# syslog over tcp or udptransport:udp# format available text|jsonformat:text# retry interval in seconds to connectretry:5# remote ipv4 or ipv6 address of the syslog serverremote-address:10.0.0.2# remote port of the syslog serverremote-port:514Example of output on syslog serverSep 22 12:43:01 bind CLIENT_RESPONSE NOERROR 192.168.1.100 51717 INET UDP 173b www.netflix.fr. A 0.040
Sep 22 12:43:01 bind CLIENT_RESPONSE NOERROR 192.168.1.100 51718 INET UDP 203b www.netflix.fr. AAAA 0.060MetricsThis output enables to generate metrics in one line and print-it to stdout. Add the following configuration as external config to activate this output:output:metrics:# enable or disableenable:true# print every N seconds.interval:300# cumulative statistics, without clearing them after printingcumulative:true# log file path or null to print to stdoutfile:null# max size for log filefile-max-size:10M# number of max log filesfile-count:10Example of output2020-10-13 05:19:35,522 18 QUERIES, 3.6 QPS, 1 CLIENTS, 18 INET, 0 INET6, 
18 UDP, 0 TCP, 17 DOMAINSDnstapThis output enables to send dnstap messages to a remote dnstap receiver. Add the following configuration as external config to activate this output:# forward to another remote dnstap receiverdnstap:# enable or disableenable:true# retry interval in seconds to connectretry:1# remote ipv4 or ipv6 address of the remote dnstap receiverremote-address:10.0.0.51# remote port of the remote dnstap receiverremote-port:6000# dnstap identitydnstap-identity:dnstap-receiverKafkaThis output enables to send dnstap messages to a Kafka topic.Install extra python library for kafkapipinstalldnstap_receiver[kafka]Configuration# forward to a Kafka topickafka:# enable or disableenable:false# format available text|json|yamlformat:json# configuration object to pass to librdkafkardkafka-config:"bootstrap.servers":null"security.protocol":null"sasl.mechanism":null"sasl.username":null"sasl.password":null# Kafka topic to forward messages totopic:nullRabbitMQThis output enables to send dnstap messages to a RabbitMQ queue.Install extra python library for rabbitmqpipinstalldnstap_receiver[rabbitmq]Configuration# forward to a RabbitMQ queuerabbitmq:# enable or disableenable:false# format available text|json|yamlformat:json# connection configurationconnection:username:nullpassword:nullhost:127.0.0.1port:5672# Queue to forward messages toqueue:queue:nullpassive:falsedurable:trueexclusive:falseauto_delete:false# Exchange, default ''exchange:""# Routing key, default = queuerouting-key:null# Retries to connect/publishretry-count:2# Retry delay secondsretry-delay:0.5PostgreSQLThis output enables to send dnstap messages to a PostgreSQL.Install extra python library for PostgreSQL (asyncpg).Seeoutput_pgsql_userfunc.pyto replace table definition and data insertion.pipinstalldnstap_receiver[pgsql]Configuration# forward to postgresql serverpgsql:# enable or disableenable:false# retry interval in seconds to connectretry:1# dsn := postgres://user@host:port/database# To explicitly write passwd in dsn is not recommended though possible.# Instead use passfile below.dsn:postgres://postgres@localhost:5432/postgres# passfile := /path/to/.pgpass# https://www.postgresql.org/docs/12/libpq-connect.html#LIBPQ-CONNECT-PASSFILEpassfile:~/.pgpass# min_size: minimum number of connections in the poolmin_size:5# max_size: maximum number of connections in the poolmax_size:10# busy_wait: wait this amount of seconds in the busy loop to write to PostgreSQL.busy_wait:1.0# timeout: wait this amount of seconds to re-create the connection pool to PostgreSQL after it failed.timeout:60# filename including user defined functionsuserfuncfile:nullElasticsearchThis output enables to send dnstap messages to Elasticsearch.output:elasticsearch:# enable or disableenable:true# url of the elasticsearchurl:textMore optionsExternal config fileThednstap_receiverbinary can takes an external config file with the-cargument or searches for a config file named dnstap.conf in /etc/dnstap_receiver/.Seeconfig fileexample../dnstap_receiver -c /etc/dnstap-receiver/dnstap.confVerbose modeYou can execute the binary in verbose mode with the-vargument:./dnstap_receiver -v
2020-11-25 20:26:59,790 DEBUG Start receiver...
2020-11-25 20:26:59,790 DEBUG Output handler: stdout
2020-11-25 20:26:59,790 DEBUG Input handler: tcp socket
2020-11-25 20:26:59,790 DEBUG Input handler: listening on 0.0.0.0:6000
2020-11-25 20:26:59,790 DEBUG Api rest: listening on 0.0.0.0:8080Filtering featureThis feature can be useful if you want to ignore some messages and keep just what you want.
Several filter are available:by qname fieldby dnstap identity field.By dnstap identityYou can filtering incoming dnstap messages according to the dnstap identity field.
A regex can be configured in the external configuration file to do thatfilter:# dnstap identify filtering feature with regex supportdnstap-identities:dnsdist01|unbound01By qnameYou can filtering incoming dnstap messages according to the query name.
A regex can be configured in the external configuration file to do thatfilter:# qname filtering feature with regex supportqname-regex:".*.com"GeoIP supportThednstap receivercan be extended with GeoIP. To do that, you need to configure your own city database in binary format.# geoip support, can be used to get the country, and city# according to the source ip in the dnstap messagegeoip:# enable or disableenable:true# city database path in binary formatcity-database:/var/geoip/GeoLite2-City.mmdb# represent country in iso modecountry-iso:falseWith the GeoIP support, the following new fields will be added:countrycityStatisticsSome statistics are computedon the flyand stored in memory, you can get them:directly from theweb serverthrough the HTTP API.with thednstap-dashboard, a top-like commandfrom yourPrometheusinstanceCountersquery: number of queriesresponse: number of answersqps: number of queries per secondclients: number of unique clients ipdomains: number of unique domainsquery/inet: number of IPv4 queriesquery/inet6: number of IPv6 queriesresponse/inet: number of IPv4 answersresponse/inet6: number of IPv6 answersquery/udp: number of queries with UDP protocolquery/tcp: number of queries with TCP protocolresponse/udp: number of answers with UDP protocolresponse/tcp: number of answers with TCP protocolresponse/[rcode]: number of answers per specific rcode = noerror, nxdomain, refused,...query/[rrtype]: number of queries per record resource type = = a, aaaa, cname,...query/bytes: total number of bytes with queriesresponse/bytes: total number of bytes with answersresponse/latency0_1: number of queries answered in less than 1msresponse/latency1_10: number of queries answered in 1-10 msresponse/latency10_50: number of queries answered in 10-50 msresponse/latency50_100: number of queries answered in 50-100 msresponse/latency100_1000: number of queries answered in 100-1000 msresponse/latency_slow: number of queries answered in more than 1 secondTablestlds:hit/query: table of [n] tlds sorted by number of querieshit/response: table of [n] tlds sorted by number of answersdomains:[rcode]/query: table of [n] domains sorted by number of queries[rcode]/response: table of [n] domains sorted by number of answersclients:hit/client: table of [n] ip addresses sorted by number of querieslength/ip: table of [n] ip addresses sorted by number of bytesrrtypeshit/query: table of [n] resources record types sorted by the number of querieshit/response: table of [n] resources record types sorted by the number of answerstop-rcodes:hit/query: table of [n] return codes sorted by the number of querieshit/response: table of [n] return codes sorted by the number of answersMetricsMetrics inPrometheusformat with global counters and specific by dnstap stream.Seemetrics fileexample.# HELP dnstap_queries Number of queries received
# TYPE dnstap_queries counter
dnstap_queries 0
# HELP dnstap_responses Number of responses received
# TYPE dnstap_responses counter
dnstap_responses 0
# HELP dnstap_responses_noerror Number of NOERROR answers
# TYPE dnstap_responses_noerror counter
dnstap_responses_noerror 0
# HELP dnstap_responses_nxdomain Number of NXDomain answers
# TYPE dnstap_responses_nxdomain counter
dnstap_responses_nxdomain 0
# HELP dnstap_responses_servfail Number of SERVFAIL  answers
# TYPE dnstap_responses_servfail counter
dnstap_responses_servfail 0
...Build-in WebserverThe build-in web server can be used to get statistics computed by the dnstap receiver.ConfigurationEnable the HTTP API, don't forget to change the default password.# rest apiweb-api:# enable or disableenable:true# web api keyapi-key:changeme# basicauth loginlogin:admin# basicauth passwordpassword:changeme# listening address ipv4 0.0.0.0 or ipv6 [::]local-address:0.0.0.0# listing on portlocal-port:8080SecurityThe following authentication methods are supported:BasicAuthX-API-KeyTo access to the API, one of them method must be used in the request header.
An HTTP 401 response is returned when the authentication failed.HTTP APISee theswaggerdocumentation.BenchmarkLimited labTested on a limited lab with the following processor: Intel Core i5-7200U @2,50GHzMetrics are extracted every second:watch-n1"time curl --user admin:changeme http://[ip_dnstap_receiver]:8080/metrics"Dns generator used:dockerpullns1labs/flame
dockerrunns1labs/flame[ip_dns_server]Result:ParametersValuesQuery per seconds~11000Domains~40000Clients1CPU usage~30%Memory usage~100MoNetwork usage~5.7MbDevelopmentRunthe dnstap receiver from sourcepython3-c"from dnstap_receiver.receiver import start_receiver; start_receiver()"-vTestunitspython3-munittesttests.test_receiver_tcpsocket-vAboutAuthorDenis Machardd.machard@gmail.comPyPIhttps://pypi.org/project/dnstap-receiver/Githubhttps://github.com/dmachard/dnstap-receiverDockerHubhttps://hub.docker.com/r/dmachard/dnstap-receiver