Repository: netty/netty
Issue: DNS resolver failing to find valid DNS record
Labels: defect

### Expected behavior
The DNS resolver should find valid DNS records.

### Actual behavior
Exception thrown:
```
Caused by: io.netty.resolver.dns.DnsNameResolverContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: 'host.toplevel' failed to resolve 'host.toplevel.search.domain' after 7 queries 
	at io.netty.resolver.dns.DnsNameResolverContext.finishResolve(DnsNameResolverContext.java:721)
	at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:663)
	at io.netty.resolver.dns.DnsNameResolverContext.query(DnsNameResolverContext.java:306)
	at io.netty.resolver.dns.DnsNameResolverContext.query(DnsNameResolverContext.java:295)
	at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:636)
	at io.netty.resolver.dns.DnsNameResolverContext$3.operationComplete(DnsNameResolverContext.java:342)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.resolver.dns.DnsQueryContext.setSuccess(DnsQueryContext.java:197)
	at io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:180)
	at io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:969)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1412)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:943)
	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:93)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
```
### Steps to reproduce

1. Configure a top level domain `someDomain` on a DNS server you own
1. Configure a host under the new top level domain `someHost.someDomain`
1. Configure multiple resolvers  on the DNS client machine that will run the Netty code. i.e. 8.8.8.8, 192.168.1.1, and 10.0.0.1 (I have 3 resolvers configured, each pointing to different DNS masters  - global DNS, local personal private network, company private network over a VPN)
1. Configure the search domain to not match the top level domain, i.e. `search.otherDomain` on the DNS client machine that will run the Netty code
1. Ask netty to resolve `someHost.someDomain`
1. failure.


### Minimal yet complete reproducer code (or URL to code)
I'm not using Netty directly so I'm not sure what to put here. Do you want my Redisson code?

### Netty version
Breaks when I upgrade to Reddison 3.6+ which pulls in Netty 4.1.20+
When forcing downgrade to Netty 4.1.13 the problem still shows, but with a slightly different stack trace.

### JVM version (e.g. `java -version`)
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

### OS version (e.g. `uname -a`)
Windows 10, Centos 7, Ubuntu 16.04

Issue comments:
@johnjaylward would it be possible to try different netty versions and see at which point it breaks ? This would help me a lot. 
Are there any breaking points in the API between 4.1.13 and 4.1.20? If not, I don't see an issue with trying on my end.
@johnjaylward as the dns resolver is marked as unstable there were a few, but I think most people should not be affected as the scope is small. So I would give it a try.
well, now that I've played around with the versions a little more, I'm not sure what's up.
Using Redisson 3.5.0 with any version of Netty and the DNS query seems to work. 

However using Redisson 3.6.0 with any version of Netty, the query fails. I'm guessing Redisson must have changed some configuration option that it is passing to the Netty resolver? Is that a possibility? If so, I'll open this on the Redisson side.
@johnjaylward sure thats possible... I dont know enough about Redission to tell you what they do.
Thanks, I'll check with them.
@johnjaylward please let me know how it goes :)
@johnjaylward

I would reopen this issue. Issue appeared once Redisson switched to netty based resolver from JDK's `InetAddress.getByName` method.
Thanks for looking @mrniko .

@normanmaurer  looks like it's an error in how netty is resolving.
Updated title since it's not a regression.
Can we somehow have a testcase to reproduce which does not involve setup an own dnsserver etc?

> Am 10.09.2018 um 17:17 schrieb johnjaylward <notifications@github.com>:
> 
> Reopened #8261.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

I'm not sure if it's environment related or not. It appears to be environment related as it seems to resolve some host names just fine, it's just this custom top level domain one it has issues with (i.e. not localhost/.com/.edu etc which work fine).

The TLD is just "companyname" not "companyname.com" . The resolver appears to be looking at "host.companyname" and not liking the TLD, so it appends the DNS Search Domain to it (host.companyname.someOther.searchDomain), which is invalid.

Short of setting up your own DNS stack with a custom TLD to test it, I'm not sure what else to do.
Maybe you can test against a public TLD like .amazon or .google or something? I'm not sure if there would be a difference there though as I have 3 resolvers configured, but only 1 will return a valid result for the custom TLD.
So you say it only happens for custom TLDs?

> Am 10.09.2018 um 17:31 schrieb johnjaylward <notifications@github.com>:
> 
> Maybe you can test against a public TLD like .amazon or .google or something? I'm not sure if there would be a difference there though as I have 3 resolvers configured, but only 1 will return a valid result for the custom TLD.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@johnjaylward could you test to only include the dnsserver that handles the custom domainname and see if it resolves in this case ? 

Also would it be possible to run the following command against each of the servers (using the domain you want to resolve) and add the output here:

```
dig @dnsserverip host.toplevel A
```
Personal local network:
```
$ dig @192.168.55.5 host.domain A

; <<>> DiG 9.12.2-P1 <<>> @192.168.55.5 host.domain A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58575
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;host.domain.                   IN      A

;; AUTHORITY SECTION:
.                       10619   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2018091101 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 192.168.55.5#53(192.168.55.5)
;; WHEN: Tue Sep 11 15:05:55 Eastern Daylight Time 2018
;; MSG SIZE  rcvd: 115
```

Global public network:
```
$ dig @8.8.8.8 host.domain A

; <<>> DiG 9.12.2-P1 <<>> @8.8.8.8 host.domain A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 18400
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;host.domain.                   IN      A

;; AUTHORITY SECTION:
.                       86362   IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2018091101 1800 900 604800 86400

;; Query time: 31 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Sep 11 15:04:32 Eastern Daylight Time 2018
;; MSG SIZE  rcvd: 115
```

Company DNS over VPN
```
$ dig @10.253.48.2 host.domain A

; <<>> DiG 9.12.2-P1 <<>> @10.253.48.2 host.domain A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42724
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
; COOKIE: 065a3fc0ed8df40e (echoed)
;; QUESTION SECTION:
;host.domain.                   IN      A

;; ANSWER SECTION:
host.domain.            3600    IN      A       10.253.50.30

;; Query time: 46 msec
;; SERVER: 10.253.48.2#53(10.253.48.2)
;; WHEN: Tue Sep 11 15:08:19 Eastern Daylight Time 2018
;; MSG SIZE  rcvd: 68
```

```
$ dig @10.253.48.3 host.domain A

; <<>> DiG 9.12.2-P1 <<>> @10.253.48.3 host.domain A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19221
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
; COOKIE: 5e8dd4d92c46330d (echoed)
;; QUESTION SECTION:
;host.domain.                   IN      A

;; ANSWER SECTION:
host.domain.            3600    IN      A       10.253.50.30

;; Query time: 31 msec
;; SERVER: 10.253.48.3#53(10.253.48.3)
;; WHEN: Tue Sep 11 15:08:38 Eastern Daylight Time 2018
;; MSG SIZE  rcvd: 68

```

@normanmaurer I'm not sure I can configure Redisson to use a specific DNS server...
@johnjaylward interesting so at least one server returns `NXDOMAIN`. This is on what OS ? Can you also show me the contents of /etc/resolv.conf
it's on windows 10 in this instance. When I was stepping through the calls the server list contained all 3 primary servers. I can't remember if it contained the secondary over the VPN. If I have time today, I'll try to step through again and see which servers were listed.
I am getting similar kind of error with redisson version 3.7.2 and netty version 4.1.25.Final

`Unable to resolve xxxx.redis.cache.windows.net - java.net.UnknownHostException: failed to resolve xxxx.redis.cache.windows.net' after 4 queries at io.netty.resolver.dns.DnsNameResolverContext.finishResolve(DnsNameResolverContext.java:721) at io.netty.resolver.dns.DnsNameResolverContext.tryToFinishResolve(DnsNameResolverContext.java:663) at io.netty.resolver.dns.DnsNameResolverContext.query(DnsNameResolverContext.java:306) at io.netty.resolver.dns.DnsNameResolverContext.query(DnsNameResolverContext.java:295) at io.netty.resolver.dns.DnsNameResolverContext.access$700(DnsNameResolverContext.java:60) at io.netty.resolver.dns.DnsNameResolverContext$3.operationComplete(DnsNameResolverContext.java:339) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at io.netty.resolver.dns.DnsQueryContext.setFailure(DnsQueryContext.java:223) at io.netty.resolver.dns.DnsQueryContext.access$300(DnsQueryContext.java:42) at io.netty.resolver.dns.DnsQueryContext$4.run(DnsQueryContext.java:162) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:125) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:745) Caused by: io.netty.resolver.dns.DnsNameResolverTimeoutException: [/8.8.4.4:53] query timed out after 5000 milliseconds (no stack trace available)`



@normanmaurer 

Could this error https://github.com/redisson/redisson/issues/1646 related to this issue either?
@johnjaylward @utsavchanda any more details ?
I don't at this time. I was mainly looking at upgrading the libraries as part of a larger task and have since put down the upgrade due to this issue and moved onto the rest of the task. I'm not sure when I'll have time to investigate it further.
Sorry but without more infos I suspect I will not be able to help :(
@normanmaurer 

Here is yet another case with AWS https://github.com/redisson/redisson/issues/1486#issuecomment-426783405
@normanmaurer @johnjaylward

Bingo! Finally managed to reproduce the issue. Below is the test code:
```java
        NioEventLoopGroup niogroup = new NioEventLoopGroup();
        
        DnsAddressResolverGroup group = new DnsAddressResolverGroup(NioDatagramChannel.class, DnsServerAddressStreamProviders.platformDefault());
        ExecutorService es = Executors.newFixedThreadPool(2);
        CountDownLatch latch = new CountDownLatch(100);
        for (int i = 0; i < 100; i++) {
            es.execute(new Runnable() {
                @Override
                public void run() {
                    AddressResolver<InetSocketAddress> resolver = group.getResolver(niogroup.next());
                    try {
                        URI uri = new URI("redis://dev.myredis.com:6379");
                        Future<List<InetSocketAddress>> allNodes = resolver.resolveAll(InetSocketAddress.createUnresolved(uri.getHost(), uri.getPort()));
                        List<InetSocketAddress> list = allNodes.syncUninterruptibly().getNow();
                        System.out.println(list);
                    } catch (URISyntaxException e) {
                        // TODO Auto-generated catch block
                        e.printStackTrace();
                    } finally {
                        latch.countDown();
                    }
                }
            });
        }
        latch.await();
        group.close();
        es.shutdown();
        niogroup.shutdownGracefully();
```

MaraDNS (http://maradns.samiam.org/) running on 127.0.0.1:53
Here is the config for it https://github.com/netty/netty/files/1941275/maradns-config.zip

If server stands first in list of dns then all works fine:

Default DNS servers: [/127.0.0.1:53, /8.8.8.8:53, /87.98.175.85:53, /51.254.25.115:53]
Log without errors: https://gist.github.com/mrniko/8ea3153888aa3dd8315b4549a77c34bb

But if it's not, then lot of errors arise:

Default DNS servers: [/8.8.8.8:53, /127.0.0.1:53, /87.98.175.85:53, /51.254.25.115:53]
Log with errors: https://gist.github.com/mrniko/59529c03450c69094d7b379515919ea2

Netty 4.1.30.Final
JDK 11
Will check once back at work

> Am 05.10.2018 um 20:35 schrieb Nikita Koksharov <notifications@github.com>:
> 
> @normanmaurer @johnjaylward
> 
> Bingo! Finally managed to reproduce the issue. Below is the test code:
> 
>         NioEventLoopGroup niogroup = new NioEventLoopGroup();
>         
>         DnsAddressResolverGroup group = new DnsAddressResolverGroup(NioDatagramChannel.class, DnsServerAddressStreamProviders.platformDefault());
>         ExecutorService es = Executors.newFixedThreadPool(2);
>         CountDownLatch latch = new CountDownLatch(100);
>         for (int i = 0; i < 100; i++) {
>             es.execute(new Runnable() {
>                 @Override
>                 public void run() {
>                     AddressResolver<InetSocketAddress> resolver = group.getResolver(niogroup.next());
>                     try {
>                         URI uri = new URI("redis://dev.myredis.com:6379");
>                         Future<List<InetSocketAddress>> allNodes = resolver.resolveAll(InetSocketAddress.createUnresolved(uri.getHost(), uri.getPort()));
>                         List<InetSocketAddress> list = allNodes.syncUninterruptibly().getNow();
>                         System.out.println(list);
>                     } catch (URISyntaxException e) {
>                         // TODO Auto-generated catch block
>                         e.printStackTrace();
>                     } finally {
>                         latch.countDown();
>                     }
>                 }
>             });
>         }
>         latch.await();
>         group.close();
>         es.shutdown();
>         niogroup.shutdownGracefully();
> MaraDNS (http://maradns.samiam.org/) running on 127.0.0.1:53
> Here is the config for it https://github.com/netty/netty/files/1941275/maradns-config.zip
> 
> If server stands first in list of dns then all works fine:
> 
> Default DNS servers: [/127.0.0.1:53, /8.8.8.8:53, /87.98.175.85:53, /51.254.25.115:53]
> Log without errors: https://gist.github.com/mrniko/8ea3153888aa3dd8315b4549a77c34bb
> 
> But if it's not, then lot of errors arise:
> 
> Default DNS servers: [/8.8.8.8:53, /127.0.0.1:53, /87.98.175.85:53, /51.254.25.115:53]
> Log with errors: https://gist.github.com/mrniko/59529c03450c69094d7b379515919ea2
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@normanmaurer 

Here is workaround I implemented in Redisson
https://github.com/redisson/redisson/commit/14eeed15f6b7b508dd206d40dec7c80377e6c9db
So you basically create one resolver per dns server address and try all of them before fail the query ?

> Am 06.10.2018 um 16:05 schrieb Nikita Koksharov <notifications@github.com>:
> 
> @normanmaurer
> 
> Here is workaround I implemented in Redisson
> redisson/redisson@14eeed1
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@normanmaurer 

That's correct
@mrniko ok I just returned from vacation and looking at this again... I am still trying to wrap my head around this. 

From my understanding of the logs what we do is correct. The nameservers we query first returns `NXDOMAIN` which indicates that the domain does not exist and so we bail out. I need to check what exactly the JDK implementation does tho. But from my understanding this is a "sane thing" to do.
Hmm, no, that is the opposite of what you want to do. By bailing early you are preventing any internal/private domains from ever resolving. That's the entire point of allowing multiple resolvers, if one fails or can't find the domain, you try others. Not every domain name or sub domain is on a public resolver. If 8.8.8.8 returns first with "no result" and that's the only result you use, then private domains will never work.
I'm curious if there is any RFC section that tells us what to do in this case, i.e. try other name server on NXDOMAIN. Such a behavior would increase the load on secondary name server and total lookup time for a non-existent domain name, and it doesn't sound like a good idea to me.
@trustin I did not find anything in the RFC that said I agree with you but I will need to do some more research tomorrow. 
Is it possible to add custom implementation to netty for this case like I did in Redisson?
@mrniko sure we could but I would first like to better understand the whole thing before adding anything. Thats why I want to do more research here before taking any action
The case of glibc resolver:

- 13 years ago: https://bugzilla.redhat.com/show_bug.cgi?id=160914#c6 (No retry on NXDOMAIN)
- 5 years ago: https://serverfault.com/questions/501739/dns-resolv-conf-issue-dns-doesnt-resolve-for-certain-internal-addresses-despi (No retry on NXDOMAIN)
I know what you mean, but the entire point of private domains is that a public query would return NXDOMAIN. 

Here's my setup: 
* Ethernet connection: IP Address in the 192.168.55.X single DNS Resolver at 192.168.55.5
* VPN IP Address in the 10.X.X.X range, Primary DNS 10.x.x.3 Secondary DNS 10.x.x.2

The regular ethernet connections DNS resolver is a standard recusive resolver with cache. It will never be able to resolve things that are private to the company (i.e. dig someHost@myCompany or dig someHost@internal.mycomapany.com will always return NXDOMAIN). This is expected because I don't have access to those resolvers that know about those domains when off the VPN.

The VPN dns servers (both primary and secondary) can resolve those as they know of the internal domain names.

In this setup I effectively have 2 primary resolvers, one for standard ethernet, and another for the VPN. If netty only takes my public resolving DNS as the primary, then the internal domain names over the company VPN will never get picked up.
The case of musl resolver (this year) https://github.com/gliderlabs/docker-alpine/issues/402
@trustin thanks for digging these up...  This is also interesting : https://bugzilla.redhat.com/show_bug.cgi?id=162625
 
@johnjaylward just to confirm all works when you use the JDK based resolver with netty tho ?
When you are connected to VPN, the VPN software should update the resolve.conf so that only the internal DNS servers are used. On disconnection, the VPN software should update the resolve.conf again back to the public DNS servers. You should not mix public and internal DNS server. If you would, you have to put the private ones *before* the public ones.

Is your VPN connection established *after* you start up your Netty application? Then Netty may use the outdated public DNS servers to send queries, but otherwise, all is working as expected in my opinion.
@trustin yeah I wonder if the problem is that we not "watch" for updates of resolve.conf and so may get into trouble when it is altered afterwards. 
@trustin as far as I know the JDK implementation does have a background thread that takes care of taking updates into account.
@normanmaurer correct. It was working fine in the old version of Redisson that was using the JDK resolver. It wasn't until the switch to the Netty resolver that it became an issue.

Also, this is running from windows. There is no resolve.conf. Windows binds the DNS resolver configurations to the interfaces, so the standard Ethernet connection keeps it's resolver, and the new VPN connection gets it's own resolvers. There is no overwriting of the old resolver like in Linux. I can only assume that windows uses all 3 resolvers when making queries.
@johnjaylward you should be able to verify by using Wireshark here and see what resolvers it uses. 
> thanks for digging these up... This is also interesting : https://bugzilla.redhat.com/show_bug.cgi?id=162625

~I guess this is when the first server disallows a recursive query.~

Ah, you mean Windows resolver?
This comment explains the difference between glibc and Windows resolver: https://bugzilla.redhat.com/show_bug.cgi?id=160914#c5
Also I checked the JDK impl again and it reloads the dns server config from time to time. We may should do the same...

> Am 11.10.2018 um 19:45 schrieb Trustin Lee <notifications@github.com>:
> 
> This comment explains the difference between glibc and Windows resolver: https://bugzilla.redhat.com/show_bug.cgi?id=160914#c5
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

Given that JDK uses OS's native resolver, it makes sense that @johnjaylward is seeing different behavior between JDK and Netty resolver on Windows. However, the original report mentions that this issue appears on OS X and Linux as well. Could someone confirm if it really affects OS X and Linux?

I'm curious if we should tune our resolver so it matches the behavior of the Windows resolver, although I think it's rather close to misconfiguration in VPN.
@trustin the initial report may have been misleading. My development machine I saw this issue on is Windows. The final application runs on Ubuntu and CentOS, but I did not get to testing that far down the cycle since it was failing in DEV.
TBH though, I don't expect this to ever be an issue in **my** production environments since there is no VPN there, and the only DNS resolvers configured are the company owned ones. I'm not sure I have a linux machine available I can test this on myself.
Also, here's a good explanation of the Windows resolver path: https://serverfault.com/questions/84291/how-does-windows-decides-which-dns-server-to-use-when-resolving-names#84293
So, I ran a wireshark on port 53 as suggested. 

Looks like the preferred order for nslookup when using git-bash on windows is to use the VPN DNS first. It appears that the priority is correct at the OS level. I only see  queries against "default" server (since nslookup only uses a single server).

I then tested it using Firefox and it used all 3 DNS servers. It actually tried resolving against the non-VPN server first, which failed: ```24	2.420892	192.168.55.5	192.168.55.169	DNS	146	Standard query response 0x5e05 No such name AAAA host.domain SOA a.root-servers.net```. Firefox then followed up with subsequent queries to the other DNS servers on the VPN.

The actual order of the request/responses from Firefox looked like this:
1. Parallel request to the 2 primary DNS interfaces (VPN and Non-VPN. I assume if I had 3 interfaces up, it would have sent 3 requests, one for each primary... maybe not though)
2. Primary non-VPN request failed
3. Request sent to secondary DNS on the VPN
4. Primary DNS on the VPN responded with a success
5. Secondary DNS on the VPN responded with a success.

Running wireshark against Chrome provided a similar workflow (querying all the DNS servers), but instead ran against the VPN DNs servers first.

Technically from a traffic perspective Chrome is worse as it makes lots of calls out to google services that are really just unnecessary.

I guess the end result is what you are trying to accomplish with this resolver. If you want it to match the specs as close as possible and work more like a "dig" or "nslookup", then no changes would be needed. It would be doing it's job as-designed. If your expectation is that this resolver is used for end-user applications, then it should probably ignore NXDOMAIN and try against other servers in order to try and  get a working record.

@trustin I am still not 100 % sure what to do about this in general but that said what you think about refresh the DNS servers / search domains on regular basis just as the JDK is doing:

http://hg.openjdk.java.net/jdk/jdk11/file/76072a077ee1/src/java.base/unix/classes/sun/net/dns/ResolverConfigurationImpl.java#l124 

 ?
I think there are two action items:

- Refreshing the System DNS resolver configuration periodically
  - This may or may not be relevant to this issue, but we should fix this.
- Changing the behavior of our DNS resolver so that it's behavior is on par with that of the host OS.
  - This probably will fix this issue but I'm not 100% sure if it's worth fixing it given the complexity and its platform-dependent nature. It'd be natural to behave in the same way with host OS's behavior from Windows users' standpoint, though. Would love to listen to what other folks think about this.
Ok let me do a pr for the refresh in the meantime

> Am 14.10.2018 um 09:01 schrieb Trustin Lee <notifications@github.com>:
> 
> I think there are two action items:
> 
> Refreshing the System DNS resolver configuration periodically
> This may or may not be relevant to this issue, but we should fix this.
> Changing the behavior of our DNS resolver so that it's behavior is on par with that of the host OS.
> This probably will fix this issue but I'm not 100% sure if it's worth fixing it given the complexity and its platform-dependent nature. It'd be natural to behave in the same way with host OS's behavior from Windows users' standpoint, though. Would love to listen to what other folks think about this.
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

@normanmaurer , @trustin , @johnjaylward ,

We bumped into this issue.

Would vote for this approach:

> If your expectation is that this resolver is used for end-user applications, then it should probably ignore NXDOMAIN and try against other servers in order to try and get a working record.
Any idea when this will be fixed?
@trustin Hi, I confirm that we can see the same behaviour on Centos7 
Ping...
@trustin , @normanmaurer , @Scottmitch , @fredericBregier , @nmittler , ...
Hello, 

if it's any help, we worked this pb around by extending original Netty Client and retrieving its original Bootstrap object, setting resolver to 'null'.  

By doing so,  Netty will eventually use io.netty.resolver.DefaultNameResolver, and so /etc/resolv.conf for DNS name resolutions.

```java
       ...
        // Setting resolver to null will default it to io.netty.resolver.DefaultNameResolver.
        this.getBootstrap().resolver(null);
       ...
```
 
@fmazoyer 

The problem is that `io.netty.resolver.DefaultNameResolver` executes name lookup and blocks the caller thread
@mrniko 
We didn't run in such an issue.
Did you check the call stack with jstack or kill -3?
Sorry I can't help more :-( 
@fmazoyer @mrniko means that the lookup is a blocking call instead of "event driven" like the other nio calls, not that it locks up the application. If you are expecting the execution to continue while waiting for the DNS lookup, then you would need to wrap your resolver calls in threads (like a `java.util.concurrent.Executors.newSingleThreadExecutor()` ) when using the `io.netty.resolver.DefaultNameResolver`
I will have a look again this week. Sorry been super busy lately. 
@mrniko sorry it took me so long to come back to you. I just tried to reproduce it with the code and config you provided but no luck :(

```java
    @Test
    public void test() throws Throwable {
        final NioEventLoopGroup niogroup = new NioEventLoopGroup();

        final DnsNameResolverBuilder builder = new DnsNameResolverBuilder();
        builder.channelType(NioDatagramChannel.class).nameServerProvider(
                new SequentialDnsServerAddressStreamProvider(new InetSocketAddress("8.8.8.8", 53),
                        new InetSocketAddress("127.0.0.1", 53), new InetSocketAddress("87.98.175.85", 53), new InetSocketAddress("51.254.25.115", 53)))
                .resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY);

        final DnsAddressResolverGroup group = new DnsAddressResolverGroup(builder);
        ExecutorService es = Executors.newFixedThreadPool(2);
        final CountDownLatch latch = new CountDownLatch(100);
        final Queue<Object> results = new ConcurrentLinkedQueue<Object>();
        for (int i = 0; i < 100; i++) {
            es.execute(new Runnable() {
                @Override
                public void run() {
                    AddressResolver<InetSocketAddress> resolver = group.getResolver(niogroup.next());
                    try {
                        URI uri = new URI("redis://dev.myredis.com:6379");
                        Future<List<InetSocketAddress>> allNodes = resolver.resolveAll(InetSocketAddress.createUnresolved(uri.getHost(), uri.getPort()));
                        //Thread.sleep(100);
                        List<InetSocketAddress> list = allNodes.syncUninterruptibly().getNow();
                        results.offer(list);
                    } catch (URISyntaxException e) {
                        // TODO Auto-generated catch block
                        e.printStackTrace();
                    } catch (Throwable cause) {
                        results.offer(cause);
                    } finally {
                        latch.countDown();
                    }
                }
            });
        }
        latch.await();
        group.close();
        es.shutdown();
        niogroup.shutdownGracefully();

        for (;;) {
            Object o = results.poll();
            if (o == null) {
                break;
            }
            if (o instanceof Throwable) {
                throw (Throwable) o;
            }
        }
    }
```

I used the same dns config that you attached. This in on Xubuntu. Any idea ?
Also @johnjaylward 
This was on latest 4.1 branch btw.
@normanmaurer 

Thanks for taking the time!

Could you try exactly the same code I use? without using DnsNameResolverBuilder. 
@mrniko I did... same result :/
@normanmaurer 

I run it on windows 10
@mrniko unfortunately I have no access to windows. Can you try to reproduce on a linux vm ?
@mrniko maybe you could capture a dump with Wireshark that includes the queries ? Best would be if you adjust it to only to one resolveAll(...) call.
@mrniko also even if I can not reproduce by now I would be happy to review a PR that fix the problem for you. 
@normanmaurer

I'm testing with Netty 4.1.32 and 64-bit JDK 1.8.0_191 on win 10 and noticed strange behaviour:

My previous test **doesn't work** with:

```java
DnsAddressResolverGroup group = new DnsAddressResolverGroup(NioDatagramChannel.class,
                DnsServerAddressStreamProviders.platformDefault());
```
and dns sequence got from `DnsServerAddressStreamProviders.platformDefault().nameServerAddressStream("")` is 
`/51.254.25.115:53, /127.0.0.1:53, /8.8.8.8:53, dns: /87.98.175.85:53`

**But it works** if I create group object like in your test with the same dns sequence:

```java
final DnsNameResolverBuilder builder = new DnsNameResolverBuilder();
builder.channelType(NioDatagramChannel.class).nameServerProvider(
                new SequentialDnsServerAddressStreamProvider(new InetSocketAddress("51.254.25.115", 53),
                        new InetSocketAddress("127.0.0.1", 53), new InetSocketAddress("8.8.8.8", 53), new InetSocketAddress("87.98.175.85", 53)))
                .resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY);

final DnsAddressResolverGroup group = new DnsAddressResolverGroup(builder);
```

How it could be?
@johnjaylward @mrniko after re-reading the RFC closely and inspecting our code again I think I found the bug. Can you please check if https://github.com/netty/netty/pull/8731 works for you ?

Also thanks to @Lukasa to discuss this with me :)
@mrniko also can you test removing `                .resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY);` ?
@normanmaurer 

Issue came back if I comment out this code `.resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY);`
@mrniko ok cool... With #8731 the problem is gone also when `.resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY)` is NOT used (at least for me). Can you try it out ?
@normanmaurer 
Everything works fine with group object:

```java
DnsAddressResolverGroup group = new DnsAddressResolverGroup(new DnsNameResolverBuilder()
                .channelType(NioDatagramChannel.class)
                .nameServerProvider(DnsServerAddressStreamProviders.platformDefault())
                .resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY));
```
and doesn't work if group object is follow:

```java
DnsAddressResolverGroup group = new DnsAddressResolverGroup(new DnsNameResolverBuilder()
                .channelType(NioDatagramChannel.class)
                .nameServerProvider(DnsServerAddressStreamProviders.platformDefault()));
```
@normanmaurer
Do you have netty-all.jar for the latest version?
@mrniko let me build one for you with the pr included... one sec
@mrniko https://drive.google.com/open?id=11pNwvCkl3ECB3CpjlUzDnuG0m3Td27yg  please try with this jar and report back.
@normanmaurer 

This page by this link reports 404
@mrniko just fixed the link... please try again
@normanmaurer 

Now it works. Thank you!
@mrniko the code or the link, or both ? ;)
@normanmaurer 

Both :) 
Yeah :) ... Please note in the PR as well... 
@normanmaurer ,
Thanks for fixing this.

We attempted to workaround this by setting [dnsMonitoringInterval](https://github.com/redisson/redisson/wiki/2.-Configuration#dnsmonitoringinterval) to -1.
`
config.useSingleServer().setDnsMonitoringInterval(-1);
`
and surprisingly, it seems to work.

However, JAVA_HOME\jre\lib\security\java.security had the following parameters:
```
# caching forever
networkaddress.cache.ttl=-1  

# 10 seconds.
networkaddress.cache.negative.ttl=10 
```

Do you think it is a legitimate workaround?

Also posted the question in another issue: https://github.com/redisson/redisson/issues/1486#issuecomment-456277039
@hlms I have no idea how exactly redission uses it so you will need to ask there.
![Captureh](https://user-images.githubusercontent.com/62762887/102698760-143fd580-4240-11eb-99b2-c9d93c9a001b.PNG)
Any Help Plz!
Add this propertie for your custom services:

eureka.instance.prefer-ip-address=true

Pull request review comments:
Why this change?
@ejona86 it made the test easier to adjust as it also change the number of queries :)
Ah. Makes sense.

Patch that resolved the issue:
diff --git a/resolver-dns/src/main/java/io/netty/resolver/dns/DnsResolveContext.java b/resolver-dns/src/main/java/io/netty/resolver/dns/DnsResolveContext.java
index 847bfb5b9f3..f625be975ce 100644
--- a/resolver-dns/src/main/java/io/netty/resolver/dns/DnsResolveContext.java
+++ b/resolver-dns/src/main/java/io/netty/resolver/dns/DnsResolveContext.java
@@ -497,6 +497,29 @@ private void onResponse(final DnsServerAddressStream nameServerAddrStream, final
                       queryLifecycleObserver.queryNoAnswer(code), true, promise, null);
             } else {
                 queryLifecycleObserver.queryFailed(NXDOMAIN_QUERY_FAILED_EXCEPTION);
+
+                // Try with the next server if is not authoritative for the domain.
+                //
+                // From https://tools.ietf.org/html/rfc1035 :
+                //
+                //   RCODE        Response code - this 4 bit field is set as part of
+                //                responses.  The values have the following
+                //                interpretation:
+                //
+                //                ....
+                //                ....
+                //
+                //                3               Name Error - Meaningful only for
+                //                                responses from an authoritative name
+                //                                server, this code signifies that the
+                //                                domain name referenced in the query does
+                //                                not exist.
+                //                ....
+                //                ....
+                if (!res.isAuthoritativeAnswer()) {
+                    query(nameServerAddrStream, nameServerAddrStreamIndex + 1, question,
+                            newDnsQueryLifecycleObserver(question), true, promise, null);
+                }
             }
         } finally {
             ReferenceCountUtil.safeRelease(envelope);


Test patch:
diff --git a/resolver-dns/src/test/java/io/netty/resolver/dns/DnsNameResolverTest.java b/resolver-dns/src/test/java/io/netty/resolver/dns/DnsNameResolverTest.java
index 1326a7975e8..d03ee870775 100644
--- a/resolver-dns/src/test/java/io/netty/resolver/dns/DnsNameResolverTest.java
+++ b/resolver-dns/src/test/java/io/netty/resolver/dns/DnsNameResolverTest.java
@@ -1143,7 +1143,7 @@ public void aAndAAAAQueryShouldTryFirstDnsServerBeforeSecond() throws IOExceptio
                     new TestRecursiveCacheDnsQueryLifecycleObserverFactory();
 
             DnsNameResolverBuilder builder = new DnsNameResolverBuilder(group.next())
-                    .resolvedAddressTypes(ResolvedAddressTypes.IPV6_PREFERRED)
+                    .resolvedAddressTypes(ResolvedAddressTypes.IPV4_ONLY)
                     .dnsQueryLifecycleObserverFactory(lifecycleObserverFactory)
                     .channelType(NioDatagramChannel.class)
                     .optResourceEnabled(false)
@@ -1156,18 +1156,12 @@ public void aAndAAAAQueryShouldTryFirstDnsServerBeforeSecond() throws IOExceptio
 
             TestDnsQueryLifecycleObserver observer = lifecycleObserverFactory.observers.poll();
             assertNotNull(observer);
-            assertEquals(2, lifecycleObserverFactory.observers.size());
+            assertEquals(1, lifecycleObserverFactory.observers.size());
             assertEquals(2, observer.events.size());
             QueryWrittenEvent writtenEvent = (QueryWrittenEvent) observer.events.poll();
             assertEquals(dnsServer1.localAddress(), writtenEvent.dnsServerAddress);
             QueryFailedEvent failedEvent = (QueryFailedEvent) observer.events.poll();
 
-            observer = lifecycleObserverFactory.observers.poll();
-            assertEquals(2, observer.events.size());
-            writtenEvent = (QueryWrittenEvent) observer.events.poll();
-            assertEquals(dnsServer1.localAddress(), writtenEvent.dnsServerAddress);
-            failedEvent = (QueryFailedEvent) observer.events.poll();
-
             observer = lifecycleObserverFactory.observers.poll();
             assertEquals(2, observer.events.size());
             writtenEvent = (QueryWrittenEvent) observer.events.poll();