Repository: netty/netty
Issue: Support hosts file encoded in unicode format

### Expected behavior
Netty should be able to resolve IP address from hosts file when the file is encoded in unicode format.

Java is able to resolve hostname from unicode encoded hosts file. Disabling Vertx (Netty) hostname resolution using `-Dvertx.disableDnsResolver=true` flag in our application was the workaround that we used, until we finally identified root cause.

### Actual behavior
`HostsFileParser.parseSilently()` returns `HostsFileEntries` instance with zero entries if the hosts file is encoded in unicode format.

### Steps to reproduce
Open hosts file with text editor (Eg. Windows Notepad) as Administrator.
Save hosts file with unicode encoding.
Run the reproducer and observe the console output.

### Minimal yet complete reproducer code (or URL to code)
https://github.com/ramtech123/pocs/tree/master/netty-hosts-reproducer

### Netty version
4.1

### JVM version (e.g. `java -version`)
Java 1.8

### OS version (e.g. `uname -a`)
Tested with Windows10, Windows Server 2012, 2016

Issue comments:
@ramtech123 thanks.. Check https://github.com/netty/netty/pull/8211. This allows you to specify the charset to use while decoding. 
Based on my own testing with Windows Server 2016, it seems like Windows auto-detects the encoding. I tried with ANSI (ASCII), Unicode (UTF-16), and UTF-8 and they all worked. I saved a trash hostname in `hosts` (different for each test) and then did a ping. Ping noticed my trash hostname configuration each time.
https://serverfault.com/a/452269 seems to think that the encoding should be UTF-8. I wonder if Windows became more permissive in a later version of Windows. I wasn't able to find any official documentation as to the encoding.
@ejona86 I mean what we could do is test UTF-8, ASCII, Unicode (in this order) until we were able to parse something when on windows... WDYT ?
@ejona86 , I also could not find any standards around hosts file encoding. The comment here https://serverfault.com/questions/926706/is-there-any-ietf-rfc-standard-defined-for-hosts-file-encoding indicates it never had a formal standard.

@normanmaurer just curious about "...until we were able to parse something when on windows" - don't we expect this situation in non-Windows machine? 
@ramtech123 I never heard of using something linux unicode for `/etc/hosts`.
ok
@normanmaurer, that sounds like it would work. Note that ASCII doesn't get you anything over UTF-8. Although Latin-1 or similar would, which may be the system's default encoding (dunno). And to be precise, "Unicode" would be "UTF-16" (probably UTF-16LE).

Linux has been using UTF-8 for a really, really long time now. I did dig through glibc a very quick amount and it looks like it just used the system encoding in normal C-style ("does this char* look like this char*?").

In both cases, using _actual_ unicode codepoints may not be at issue (at which point UTF-8 vs some-random-codepage are the same). It sounds like @ramtech123 mainly suffered issue because the file was encoded with UTF-16.
@ejona86 PTAL https://github.com/netty/netty/pull/8211 again

Pull request review comments:

Patch that resolved the issue:
diff --git a/resolver/src/main/java/io/netty/resolver/DefaultHostsFileEntriesResolver.java b/resolver/src/main/java/io/netty/resolver/DefaultHostsFileEntriesResolver.java
index 7598a29200c..9051262eb1c 100644
--- a/resolver/src/main/java/io/netty/resolver/DefaultHostsFileEntriesResolver.java
+++ b/resolver/src/main/java/io/netty/resolver/DefaultHostsFileEntriesResolver.java
@@ -15,11 +15,14 @@
  */
 package io.netty.resolver;
 
+import io.netty.util.CharsetUtil;
+import io.netty.util.internal.PlatformDependent;
 import io.netty.util.internal.UnstableApi;
 
 import java.net.Inet4Address;
 import java.net.Inet6Address;
 import java.net.InetAddress;
+import java.nio.charset.Charset;
 import java.util.Locale;
 import java.util.Map;
 
@@ -33,7 +36,7 @@ public final class DefaultHostsFileEntriesResolver implements HostsFileEntriesRe
     private final Map<String, Inet6Address> inet6Entries;
 
     public DefaultHostsFileEntriesResolver() {
-        this(HostsFileParser.parseSilently());
+        this(parseEntries());
     }
 
     // for testing purpose only
@@ -65,4 +68,14 @@ public InetAddress address(String inetHost, ResolvedAddressTypes resolvedAddress
     String normalize(String inetHost) {
         return inetHost.toLowerCase(Locale.ENGLISH);
     }
+
+    private static HostsFileEntries parseEntries() {
+        if (PlatformDependent.isWindows()) {
+            // Ony windows there seems to be no standard for the encoding used for the hosts file, so let us
+            // try multiple until we either were able to parse it or there is none left and so we return an
+            // empty intstance.
+            return HostsFileParser.parseSilently(Charset.defaultCharset(), CharsetUtil.UTF_16, CharsetUtil.UTF_8);
+        }
+        return HostsFileParser.parseSilently();
+    }
 }
diff --git a/resolver/src/main/java/io/netty/resolver/HostsFileParser.java b/resolver/src/main/java/io/netty/resolver/HostsFileParser.java
index 1997fe9e165..16eaba6167f 100644
--- a/resolver/src/main/java/io/netty/resolver/HostsFileParser.java
+++ b/resolver/src/main/java/io/netty/resolver/HostsFileParser.java
@@ -23,12 +23,14 @@
 
 import java.io.BufferedReader;
 import java.io.File;
-import java.io.FileReader;
+import java.io.FileInputStream;
 import java.io.IOException;
+import java.io.InputStreamReader;
 import java.io.Reader;
 import java.net.Inet4Address;
 import java.net.Inet6Address;
 import java.net.InetAddress;
+import java.nio.charset.Charset;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Locale;
@@ -66,14 +68,25 @@ private static File locateHostsFile() {
     }
 
     /**
-     * Parse hosts file at standard OS location.
+     * Parse hosts file at standard OS location using the systems default {@link Charset} for decoding.
      *
      * @return a {@link HostsFileEntries}
      */
     public static HostsFileEntries parseSilently() {
+        return parseSilently(Charset.defaultCharset());
+    }
+
+    /**
+     * Parse hosts file at standard OS location using the given {@link Charset}s one after each other until
+     * we were able to parse something or none is left.
+     *
+     * @param charsets the {@link Charset}s to try as file encodings when parsing.
+     * @return a {@link HostsFileEntries}
+     */
+    public static HostsFileEntries parseSilently(Charset... charsets) {
         File hostsFile = locateHostsFile();
         try {
-            return parse(hostsFile);
+            return parse(hostsFile, charsets);
         } catch (IOException e) {
             if (logger.isWarnEnabled()) {
                 logger.warn("Failed to load and parse hosts file at " + hostsFile.getPath(), e);
@@ -83,7 +96,7 @@ public static HostsFileEntries parseSilently() {
     }
 
     /**
-     * Parse hosts file at standard OS location.
+     * Parse hosts file at standard OS location using the system default {@link Charset} for decoding.
      *
      * @return a {@link HostsFileEntries}
      * @throws IOException file could not be read
@@ -93,19 +106,37 @@ public static HostsFileEntries parse() throws IOException {
     }
 
     /**
-     * Parse a hosts file.
+     * Parse a hosts file using the system default {@link Charset} for decoding.
      *
      * @param file the file to be parsed
      * @return a {@link HostsFileEntries}
      * @throws IOException file could not be read
      */
     public static HostsFileEntries parse(File file) throws IOException {
+        return parse(file, Charset.defaultCharset());
+    }
+
+    /**
+     * Parse a hosts file.
+     *
+     * @param file the file to be parsed
+     * @param charsets the {@link Charset}s to try as file encodings when parsing.
+     * @return a {@link HostsFileEntries}
+     * @throws IOException file could not be read
+     */
+    public static HostsFileEntries parse(File file, Charset... charsets) throws IOException {
         checkNotNull(file, "file");
+        checkNotNull(charsets, "charsets");
         if (file.exists() && file.isFile()) {
-            return parse(new BufferedReader(new FileReader(file)));
-        } else {
-            return HostsFileEntries.EMPTY;
+            for (Charset charset: charsets) {
+                HostsFileEntries entries = parse(new BufferedReader(new InputStreamReader(
+                        new FileInputStream(file), charset)));
+                if (entries != HostsFileEntries.EMPTY) {
+                    return entries;
+                }
+            }
         }
+        return HostsFileEntries.EMPTY;
     }
 
     /**


Test patch:
diff --git a/resolver/src/test/java/io/netty/resolver/HostsFileParserTest.java b/resolver/src/test/java/io/netty/resolver/HostsFileParserTest.java
index 6b908f57420..ac4e4b2eb69 100644
--- a/resolver/src/test/java/io/netty/resolver/HostsFileParserTest.java
+++ b/resolver/src/test/java/io/netty/resolver/HostsFileParserTest.java
@@ -15,13 +15,18 @@
  */
 package io.netty.resolver;
 
+import io.netty.util.CharsetUtil;
+import org.junit.Assume;
 import org.junit.Test;
 
 import java.io.BufferedReader;
+import java.io.File;
 import java.io.IOException;
 import java.io.StringReader;
 import java.net.Inet4Address;
 import java.net.Inet6Address;
+import java.nio.charset.Charset;
+import java.nio.charset.UnsupportedCharsetException;
 import java.util.Map;
 
 import static org.junit.Assert.*;
@@ -60,4 +65,41 @@ public void testParse() throws IOException {
         assertEquals("192.168.0.5", inet4Entries.get("host7").getHostAddress());
         assertEquals("0:0:0:0:0:0:0:1", inet6Entries.get("host1").getHostAddress());
     }
+
+    @Test
+    public void testParseUnicode() throws IOException {
+        final Charset unicodeCharset;
+        try {
+            unicodeCharset = Charset.forName("unicode");
+        } catch (UnsupportedCharsetException e) {
+            Assume.assumeNoException(e);
+            return;
+        }
+        testParseFile(HostsFileParser.parse(
+                new File(getClass().getResource("hosts-unicode").getFile()), unicodeCharset));
+    }
+
+    @Test
+    public void testParseMultipleCharsets() throws IOException {
+        final Charset unicodeCharset;
+        try {
+            unicodeCharset = Charset.forName("unicode");
+        } catch (UnsupportedCharsetException e) {
+            Assume.assumeNoException(e);
+            return;
+        }
+        testParseFile(HostsFileParser.parse(new File(getClass().getResource("hosts-unicode").getFile()),
+                                            CharsetUtil.UTF_8, CharsetUtil.ISO_8859_1, unicodeCharset));
+    }
+
+    private static void testParseFile(HostsFileEntries entries) throws IOException {
+        Map<String, Inet4Address> inet4Entries = entries.inet4Entries();
+        Map<String, Inet6Address> inet6Entries = entries.inet6Entries();
+
+        assertEquals("Expected 2 IPv4 entries", 2, inet4Entries.size());
+        assertEquals("Expected 1 IPv6 entries", 1, inet6Entries.size());
+        assertEquals("127.0.0.1", inet4Entries.get("localhost").getHostAddress());
+        assertEquals("255.255.255.255", inet4Entries.get("broadcasthost").getHostAddress());
+        assertEquals("0:0:0:0:0:0:0:1", inet6Entries.get("localhost").getHostAddress());
+    }
 }