alphagov/pp-puppet — pull request #652 by timmow (2014-11-11 14:50:39+00:00)
Title: Explicitly set upstream nameservers in dnsmasq

This change was prompted by a report of many ipv6 AAAA dns queries,
which were originating from nodejs. Upon investigation, dnsmasq was
returing a refused status for these queries, and so node was then
contacting the next dns server listed in resolv.conf

Dnsmasq should pick up the nameservers from those listed in resolv.conf
- however this doesnt seem to be the case on all of our vms. Vms built
with the new template do seem to pick this up from resolv.conf

Rather than relying on this possibly buggy
(https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1247803) magic
ubuntu behaviour, as we always want to use the same nameservers, and
dont require them to be updated through dhcp, we can explicitly
configure dnsmasq to use the upstream google nameservers.

Because the dns queries made by node (AAAA for
www.performance.service.gov.uk was an example) result in a NXDOMAIN
result, we have to enable negative caching for this result to be cached.

Because the SOA record of for this domain is set to 60 seconds, dnsmasq
will only cache the result for 60 seconds - however this is better than
the current behaviour. If traffic is still an issue we may be able to
increase the SOA TTL to cache these responses for longer.

I verified this dns response was being cached by adding the options
log-facility=/tmp/dns.log and log-queries in the dnsmasq config file,
then sending dnsmasq a SIGHUP signal to force it to clear the cache,
then sending it a SIGUSR1 which causes it to log information about the
cache. I could then verify www.performance.service.gov.uk was being
cached.

It would be good to fix the fact node is generating the AAAA records,
but the best fix seems to be to rewrite the app to make dns requests
manually and then make http requests against the resolved ip -
https://groups.google.com/forum/#!topic/nodejs/GDeSo3zu1eM