Repository: jhy/jsoup
Pull request #1904 (2023-02-23 03:41:01)

Remove unnecessary override.
Remove unnecessary override in `FirstFinder`.


Patch:
diff --git a/CHANGES b/CHANGES
index 52b3acec..8c281de8 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,9 +1,28 @@
 jsoup changelog
 
-Release 1.16.1 [PENDING]
+Release 1.16.2 [PENDING]
+  * Improvement: when converting jsoup Documents to W3C Documents in W3CDom, HTML documents will be placed in the
+    `http://www.w3.org/1999/xhtml` namespace by default, per the HTML5 spec. This can be controlled by setting
+    `W3CDom#namespaceAware(false)`.
+    <https://github.com/jhy/jsoup/pull/1848>
+
+  * Bugfix: `form` elements and empty elements (such as `img`) did not have their attributes de-duplicated.
+    <https://github.com/jhy/jsoup/pull/1950>
+
+
+Release 1.16.1 [29-Apr-2023]
+  * Improvement: in Jsoup.connect(url), natively support URLs with Unicode characters in the path or query string,
+    without having to be escaped by the caller.
+    <https://github.com/jhy/jsoup/issues/1914>
+
   * Improvement: Calling Node.remove() on a node with no parent is now a no-op, vs a validation error.
     <https://github.com/jhy/jsoup/issues/1898>
-    
+
+  * Bugfix: aligned the HTML Tree Builder processing steps for AfterBody and AfterAfterBody to the updated WHATWG
+    standard, to not pop the stack to close <body> or <html> elements. This prevents an errant </html> closing preceding
+    structure. Also added appropriate error message outputs in this case.
+    <https://github.com/jhy/jsoup/issues/1851>
+
   * Bugfix: Corrected support for ruby elements (<ruby>, <rp>, <rt>, and <rtc>) to current spec.
     <https://github.com/jhy/jsoup/issues/1294>
 
@@ -11,6 +30,35 @@ Release 1.16.1 [PENDING]
     the incoming node may be inserted into the wrong relative location.
     <https://github.com/jhy/jsoup/issues/1898>
 
+  * Bugfix: In Jsoup.connect(url), if the input URL had components that were already % escaped, they would be escaped
+    again, causing errors when fetched.
+    <https://github.com/jhy/jsoup/issues/1902>
+
+  * Bugfix: when tracking input source positions, text in tables that was fostered had invalid positions.
+    <https://github.com/jhy/jsoup/issues/1927>
+
+  * Bugfix: If the Document.OutputSettings class was initialized, and then Entities.escape(String) called, an NPE may be
+    thrown due to a class loading circular dependency.
+    <https://github.com/jhy/jsoup/issues/1910>
+
+  * Bugfix: when pretty-printing, the first inline Element or Comment in a block would not be wrap-indented if it were
+    preceded by a blank text node.
+    <https://github.com/jhy/jsoup/issues/1906>
+
+  * Bugfix: when pretty-printing a <pre> containing block tags, those tags were incorrectly indented.
+    <https://github.com/jhy/jsoup/issues/1891>
+
+  * Bugfix: when pretty-printing nested inlineable blocks (such as a <p> in a <td>), the inner element should be
+    indented.
+    <https://github.com/jhy/jsoup/issues/1926>
+
+  * Bugfix: <br> tags should be wrap-indented when in block tags (and not when in inline tags).
+    <https://github.com/jhy/jsoup/issues/1911>
+
+  * Bugfix: the contents of a sufficiently large <textarea> with un-escaped HTML closing tags may be incorrectly parsed
+    to an empty node.
+    <https://github.com/jhy/jsoup/issues/1929>
+
 Release 1.15.4 [18-Feb-2023]
   * Improvement: added the ability to escape CSS selectors (tags, IDs, classes) to match elements that don't follow
     regular CSS syntax. For example, to match by classname <p class="one.two">, use document.select("p.one\\.two");
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 00000000..b89bbc29
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,23 @@
+# Security Policy
+
+## Supported Versions
+
+Security fixes are not back-ported. Please make sure you are running at least the latest [release version](https://jsoup.org/download) of jsoup.
+
+Please remember that jsoup is an Open Source library and is provided without any warranty. Before using jsoup in a critical environment, you should satisfy yourself that it works correctly and securely for your needs.
+
+## Reporting a Vulnerability
+
+If you believe or suspect you have identified a security vulnerability, please [report it](https://github.com/jhy/jsoup/security/advisories)
+via the "Report a Vulnerability" button in Security Advisories. 
+([Details](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability))
+
+We follow [Coordinated Disclosure](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/about-coordinated-disclosure-of-security-vulnerabilities) practices and ask that you do too.
+
+Please provide as much detail as possible in your report, including the steps to reproduce the vulnerability and sample code.
+
+Alternatively to using GitHub, or if you have a security question, please email `security@jsoup.org`.
+
+## Fixing Vulnerabilities
+
+We take all vulnerability reports seriously and strive to fix them as quickly as possible. Once we receive a report, we will verify the vulnerability and its impact. We will then work to develop and test a fix for the vulnerability, and release it as soon as possible.
diff --git a/pom.xml b/pom.xml
index 94a4409b..847080cc 100644
--- a/pom.xml
+++ b/pom.xml
@@ -5,7 +5,7 @@
 
   <groupId>org.jsoup</groupId>
   <artifactId>jsoup</artifactId>
-  <version>1.16.1-SNAPSHOT</version><!-- remember to update previous version below for japicmp -->
+  <version>1.16.1</version><!-- remember to update previous version below for japicmp -->
   <description>jsoup is a Java library for working with real-world HTML. It provides a very convenient API for fetching URLs and extracting and manipulating data, using the best of HTML5 DOM methods and CSS selectors. jsoup implements the WHATWG HTML5 specification, and parses HTML to the same DOM as modern browsers do.</description>
   <url>https://jsoup.org/</url>
   <inceptionYear>2009</inceptionYear>
@@ -24,7 +24,7 @@
     <url>https://github.com/jhy/jsoup</url>
     <connection>scm:git:https://github.com/jhy/jsoup.git</connection>
     <!-- <developerConnection>scm:git:git@github.com:jhy/jsoup.git</developerConnection> -->
-    <tag>HEAD</tag>
+    <tag>jsoup-1.16.1</tag>
   </scm>
   <organization>
     <name>Jonathan Hedley</name>
@@ -33,6 +33,7 @@
 
   <properties>
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+    <jetty.version>9.4.51.v20230217</jetty.version>
   </properties>
 
   <build>
@@ -40,7 +41,7 @@
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-compiler-plugin</artifactId>
-        <version>3.10.1</version>
+        <version>3.11.0</version>
         <configuration>
           <source>1.8</source>
           <target>1.8</target>
@@ -57,7 +58,7 @@
         <!-- Ensure Java 8 and Android 10 API compatibility -->
         <groupId>org.codehaus.mojo</groupId>
         <artifactId>animal-sniffer-maven-plugin</artifactId>
-        <version>1.22</version>
+        <version>1.23</version>
         <executions>
           <execution>
             <id>animal-sniffer</id>
@@ -86,7 +87,7 @@
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-javadoc-plugin</artifactId>
-        <version>3.4.1</version>
+        <version>3.5.0</version>
         <configuration>
             <doclint>none</doclint>
             <source>8</source>
@@ -163,16 +164,16 @@
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-resources-plugin</artifactId>
-        <version>3.3.0</version>
+        <version>3.3.1</version>
       </plugin>
       <plugin>
         <artifactId>maven-release-plugin</artifactId>
-        <version>2.5.3</version>
+        <version>3.0.0</version>
       </plugin>
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-surefire-plugin</artifactId>
-        <version>3.0.0-M8</version>
+        <version>3.0.0</version>
         <configuration>
           <!-- smaller stack to find stack overflows -->
           <argLine>-Xss256k</argLine>
@@ -180,7 +181,7 @@
       </plugin>
       <plugin>
         <artifactId>maven-failsafe-plugin</artifactId>
-        <version>3.0.0-M8</version>
+        <version>3.0.0</version>
         <executions>
           <execution>
             <goals>
@@ -198,14 +199,14 @@
         <!-- API version compat check - https://siom79.github.io/japicmp/ -->
         <groupId>com.github.siom79.japicmp</groupId>
         <artifactId>japicmp-maven-plugin</artifactId>
-        <version>0.17.1</version>
+        <version>0.17.2</version>
         <configuration>
           <!-- hard code previous version; can't detect when running stateless on build server -->
           <oldVersion>
             <dependency>
               <groupId>org.jsoup</groupId>
               <artifactId>jsoup</artifactId>
-              <version>1.15.3</version>
+              <version>1.15.4</version>
               <type>jar</type>
             </dependency>
           </oldVersion>
@@ -304,7 +305,7 @@
         <plugins>
           <plugin>
             <artifactId>maven-failsafe-plugin</artifactId>
-            <version>3.0.0-M8</version>
+            <version>3.0.0</version>
             <executions>
               <execution>
                 <goals>
@@ -325,7 +326,7 @@
     <dependency>
       <groupId>org.junit.jupiter</groupId>
       <artifactId>junit-jupiter</artifactId>
-      <version>5.9.2</version>
+      <version>5.9.3</version>
       <scope>test</scope>
     </dependency>
 
@@ -341,7 +342,7 @@
       <!-- jetty for webserver integration tests. 9.x is last with Java7 support -->
       <groupId>org.eclipse.jetty</groupId>
       <artifactId>jetty-server</artifactId>
-      <version>9.4.50.v20221201</version>
+      <version>${jetty.version}</version>
       <scope>test</scope>
     </dependency>
 
@@ -349,7 +350,7 @@
       <!-- jetty for webserver integration tests -->
       <groupId>org.eclipse.jetty</groupId>
       <artifactId>jetty-servlet</artifactId>
-      <version>9.4.50.v20221201</version>
+      <version>${jetty.version}</version>
       <scope>test</scope>
     </dependency>
 
diff --git a/src/main/java/org/jsoup/helper/HttpConnection.java b/src/main/java/org/jsoup/helper/HttpConnection.java
index d183a521..d87c9f42 100644
--- a/src/main/java/org/jsoup/helper/HttpConnection.java
+++ b/src/main/java/org/jsoup/helper/HttpConnection.java
@@ -23,12 +23,9 @@ import java.io.OutputStreamWriter;
 import java.net.CookieManager;
 import java.net.CookieStore;
 import java.net.HttpURLConnection;
-import java.net.IDN;
 import java.net.InetSocketAddress;
 import java.net.MalformedURLException;
 import java.net.Proxy;
-import java.net.URI;
-import java.net.URISyntaxException;
 import java.net.URL;
 import java.net.URLEncoder;
 import java.nio.Buffer;
@@ -47,6 +44,7 @@ import java.util.zip.Inflater;
 import java.util.zip.InflaterInputStream;
 
 import static org.jsoup.Connection.Method.HEAD;
+import static org.jsoup.helper.DataUtil.UTF_8;
 import static org.jsoup.internal.Normalizer.lowerCase;
 
 /**
@@ -68,7 +66,6 @@ public class HttpConnection implements Connection {
     public static final String FORM_URL_ENCODED = "application/x-www-form-urlencoded";
     private static final int HTTP_TEMP_REDIR = 307; // http/1.1 temporary redirect, not in Java's set.
     private static final String DefaultUploadType = "application/octet-stream";
-    private static final Charset UTF_8 = Charset.forName("UTF-8"); // Don't use StandardCharsets, not in Android API 10.
     private static final Charset ISO_8859_1 = Charset.forName("ISO-8859-1");
 
     /**
@@ -94,7 +91,7 @@ public class HttpConnection implements Connection {
     }
 
     /**
-     Creates a new, empty HttpConnection.
+     Create a new, empty HttpConnection.
      */
     public HttpConnection() {
         req = new Request();
@@ -109,50 +106,6 @@ public class HttpConnection implements Connection {
         req = new Request(copy);
     }
 
-    /**
-     * Encodes the input URL into a safe ASCII URL string
-     * @param url unescaped URL
-     * @return escaped URL
-     */
-	private static String encodeUrl(String url) {
-        try {
-            URL u = new URL(url);
-            return encodeUrl(u).toExternalForm();
-        } catch (Exception e) {
-            return url;
-        }
-	}
-
-    static URL encodeUrl(URL u) {
-	    u = punyUrl(u);
-        try {
-            // run the URL through URI, so components are encoded
-            URI uri = new URI(u.getProtocol(), u.getUserInfo(), u.getHost(), u.getPort(), u.getPath(), u.getQuery(), u.getRef());
-            return uri.toURL();
-        } catch (URISyntaxException | MalformedURLException e) {
-            // give up and return the original input
-            return u;
-        }
-    }
-
-    /**
-     Convert an International URL to a Punycode URL.
-     @param url input URL that may include an international hostname
-     @return a punycode URL if required, or the original URL
-     */
-    private static URL punyUrl(URL url) {
-        if (!StringUtil.isAscii(url.getHost())) {
-            try {
-                String puny = IDN.toASCII(url.getHost());
-                url = new URL(url.getProtocol(), puny, url.getPort(), url.getFile()); // file will include ref, query if any
-            } catch (MalformedURLException e) {
-                // if passed a valid URL initially, cannot happen
-                throw new IllegalArgumentException(e);
-            }
-        }
-        return url;
-    }
-
     private static String encodeMimeName(String val) {
         return val.replace("\"", "%22");
     }
@@ -172,84 +125,98 @@ public class HttpConnection implements Connection {
         this.res = res;
     }
 
+    @Override
     public Connection url(URL url) {
         req.url(url);
         return this;
     }
 
+    @Override
     public Connection url(String url) {
         Validate.notEmptyParam(url, "url");
         try {
-            req.url(new URL(encodeUrl(url)));
+            req.url(new URL(url));
         } catch (MalformedURLException e) {
             throw new IllegalArgumentException(String.format("The supplied URL, '%s', is malformed. Make sure it is an absolute URL, and starts with 'http://' or 'https://'. See https://jsoup.org/cookbook/extracting-data/working-with-urls", url), e);
         }
         return this;
     }
 
+    @Override
     public Connection proxy(@Nullable Proxy proxy) {
         req.proxy(proxy);
         return this;
     }
 
+    @Override
     public Connection proxy(String host, int port) {
         req.proxy(host, port);
         return this;
     }
 
+    @Override
     public Connection userAgent(String userAgent) {
         Validate.notNullParam(userAgent, "userAgent");
         req.header(USER_AGENT, userAgent);
         return this;
     }
 
+    @Override
     public Connection timeout(int millis) {
         req.timeout(millis);
         return this;
     }
 
+    @Override
     public Connection maxBodySize(int bytes) {
         req.maxBodySize(bytes);
         return this;
     }
 
+    @Override
     public Connection followRedirects(boolean followRedirects) {
         req.followRedirects(followRedirects);
         return this;
     }
 
+    @Override
     public Connection referrer(String referrer) {
         Validate.notNullParam(referrer, "referrer");
         req.header("Referer", referrer);
         return this;
     }
 
+    @Override
     public Connection method(Method method) {
         req.method(method);
         return this;
     }
 
+    @Override
     public Connection ignoreHttpErrors(boolean ignoreHttpErrors) {
 		req.ignoreHttpErrors(ignoreHttpErrors);
 		return this;
 	}
 
+    @Override
     public Connection ignoreContentType(boolean ignoreContentType) {
         req.ignoreContentType(ignoreContentType);
         return this;
     }
 
-
+    @Override
     public Connection data(String key, String value) {
         req.data(KeyVal.create(key, value));
         return this;
     }
 
+    @Override
     public Connection sslSocketFactory(SSLSocketFactory sslSocketFactory) {
 	    req.sslSocketFactory(sslSocketFactory);
 	    return this;
     }
 
+    @Override
     public Connection data(String key, String filename, InputStream inputStream) {
         req.data(KeyVal.create(key, filename, inputStream));
         return this;
@@ -261,6 +228,7 @@ public class HttpConnection implements Connection {
         return this;
     }
 
+    @Override
     public Connection data(Map<String, String> data) {
         Validate.notNullParam(data, "data");
         for (Map.Entry<String, String> entry : data.entrySet()) {
@@ -269,6 +237,7 @@ public class HttpConnection implements Connection {
         return this;
     }
 
+    @Override
     public Connection data(String... keyvals) {
         Validate.notNullParam(keyvals, "keyvals");
         Validate.isTrue(keyvals.length %2 == 0, "Must supply an even number of key value pairs");
@@ -282,6 +251,7 @@ public class HttpConnection implements Connection {
         return this;
     }
 
+    @Override
     public Connection data(Collection<Connection.KeyVal> data) {
         Validate.notNullParam(data, "data");
         for (Connection.KeyVal entry: data) {
@@ -290,6 +260,7 @@ public class HttpConnection implements Connection {
         return this;
     }
 
+    @Override
     public Connection.KeyVal data(String key) {
         Validate.notEmptyParam(key, "key");
         for (Connection.KeyVal keyVal : request().data()) {
@@ -299,16 +270,19 @@ public class HttpConnection implements Connection {
         return null;
     }
 
+    @Override
     public Connection requestBody(String body) {
         req.requestBody(body);
         return this;
     }
 
+    @Override
     public Connection header(String name, String value) {
         req.header(name, value);
         return this;
     }
 
+    @Override
     public Connection headers(Map<String,String> headers) {
         Validate.notNullParam(headers, "headers");
         for (Map.Entry<String,String> entry : headers.entrySet()) {
@@ -317,11 +291,13 @@ public class HttpConnection implements Connection {
         return this;
     }
 
+    @Override
     public Connection cookie(String name, String value) {
         req.cookie(name, value);
         return this;
     }
 
+    @Override
     public Connection cookies(Map<String, String> cookies) {
         Validate.notNullParam(cookies, "cookies");
         for (Map.Entry<String, String> entry : cookies.entrySet()) {
@@ -342,11 +318,13 @@ public class HttpConnection implements Connection {
         return req.cookieManager.getCookieStore();
     }
 
+    @Override
     public Connection parser(Parser parser) {
         req.parser(parser);
         return this;
     }
 
+    @Override
     public Document get() throws IOException {
         req.method(Method.GET);
         execute();
@@ -354,6 +332,7 @@ public class HttpConnection implements Connection {
         return res.parse();
     }
 
+    @Override
     public Document post() throws IOException {
         req.method(Method.POST);
         execute();
@@ -361,20 +340,24 @@ public class HttpConnection implements Connection {
         return res.parse();
     }
 
+    @Override
     public Connection.Response execute() throws IOException {
         res = Response.execute(req);
         return res;
     }
 
+    @Override
     public Connection.Request request() {
         return req;
     }
 
+    @Override
     public Connection request(Connection.Request request) {
         req = (HttpConnection.Request) request; // will throw a class-cast exception if the user has extended some but not all of Connection; that's desired
         return this;
     }
 
+    @Override
     public Connection.Response response() {
         if (res == null) {
             throw new IllegalArgumentException("You must execute the request before getting a response.");
@@ -382,11 +365,13 @@ public class HttpConnection implements Connection {
         return res;
     }
 
+    @Override
     public Connection response(Connection.Response response) {
         res = response;
         return this;
     }
 
+    @Override
     public Connection postDataCharset(String charset) {
         req.postDataCharset(charset);
         return this;
@@ -424,28 +409,33 @@ public class HttpConnection implements Connection {
             cookies = new LinkedHashMap<>(); cookies.putAll(copy.cookies); // just holds strings
         }
 
+        @Override
         public URL url() {
             if (url == UnsetUrl)
                 throw new IllegalArgumentException("URL not set. Make sure to call #url(...) before executing the request.");
             return url;
         }
 
+        @Override
         public T url(URL url) {
             Validate.notNullParam(url, "url");
-            this.url = punyUrl(url); // if calling url(url) directly, does not go through encodeUrl, so we punycode it explicitly. todo - should we encode here as well?
+            this.url = new UrlBuilder(url).build();
             return (T) this;
         }
 
+        @Override
         public Method method() {
             return method;
         }
 
+        @Override
         public T method(Method method) {
             Validate.notNullParam(method, "method");
             this.method = method;
             return (T) this;
         }
 
+        @Override
         public String header(String name) {
             Validate.notNullParam(name, "name");
             List<String> vals = getHeadersCaseInsensitive(name);
@@ -528,6 +518,7 @@ public class HttpConnection implements Connection {
             return true;
         }
 
+        @Override
         public T header(String name, String value) {
             Validate.notEmptyParam(name, "name");
             removeHeader(name); // ensures we don't get an "accept-encoding" and a "Accept-Encoding"
@@ -535,6 +526,7 @@ public class HttpConnection implements Connection {
             return (T) this;
         }
 
+        @Override
         public boolean hasHeader(String name) {
             Validate.notEmptyParam(name, "name");
             return !getHeadersCaseInsensitive(name).isEmpty();
@@ -543,6 +535,7 @@ public class HttpConnection implements Connection {
         /**
          * Test if the request has a header with this value (case insensitive).
          */
+        @Override
         public boolean hasHeaderWithValue(String name, String value) {
             Validate.notEmpty(name);
             Validate.notEmpty(value);
@@ -554,6 +547,7 @@ public class HttpConnection implements Connection {
             return false;
         }
 
+        @Override
         public T removeHeader(String name) {
             Validate.notEmptyParam(name, "name");
             Map.Entry<String, List<String>> entry = scanHeaders(name); // remove is case-insensitive too
@@ -562,6 +556,7 @@ public class HttpConnection implements Connection {
             return (T) this;
         }
 
+        @Override
         public Map<String, String> headers() {
             LinkedHashMap<String, String> map = new LinkedHashMap<>(headers.size());
             for (Map.Entry<String, List<String>> entry : headers.entrySet()) {
@@ -598,11 +593,13 @@ public class HttpConnection implements Connection {
             return null;
         }
 
+        @Override
         public String cookie(String name) {
             Validate.notEmptyParam(name, "name");
             return cookies.get(name);
         }
 
+        @Override
         public T cookie(String name, String value) {
             Validate.notEmptyParam(name, "name");
             Validate.notNullParam(value, "value");
@@ -610,17 +607,20 @@ public class HttpConnection implements Connection {
             return (T) this;
         }
 
+        @Override
         public boolean hasCookie(String name) {
             Validate.notEmptyParam(name, "name");
             return cookies.containsKey(name);
         }
 
+        @Override
         public T removeCookie(String name) {
             Validate.notEmptyParam(name, "name");
             cookies.remove(name);
             return (T) this;
         }
 
+        @Override
         public Map<String, String> cookies() {
             return cookies;
         }
@@ -678,104 +678,126 @@ public class HttpConnection implements Connection {
             executing = false;
         }
 
+        @Override
         public Proxy proxy() {
             return proxy;
         }
 
+        @Override
         public Request proxy(@Nullable Proxy proxy) {
             this.proxy = proxy;
             return this;
         }
 
+        @Override
         public Request proxy(String host, int port) {
             this.proxy = new Proxy(Proxy.Type.HTTP, InetSocketAddress.createUnresolved(host, port));
             return this;
         }
 
+        @Override
         public int timeout() {
             return timeoutMilliseconds;
         }
 
+        @Override
         public Request timeout(int millis) {
             Validate.isTrue(millis >= 0, "Timeout milliseconds must be 0 (infinite) or greater");
             timeoutMilliseconds = millis;
             return this;
         }
 
+        @Override
         public int maxBodySize() {
             return maxBodySizeBytes;
         }
 
+        @Override
         public Connection.Request maxBodySize(int bytes) {
             Validate.isTrue(bytes >= 0, "maxSize must be 0 (unlimited) or larger");
             maxBodySizeBytes = bytes;
             return this;
         }
 
+        @Override
         public boolean followRedirects() {
             return followRedirects;
         }
 
+        @Override
         public Connection.Request followRedirects(boolean followRedirects) {
             this.followRedirects = followRedirects;
             return this;
         }
 
+        @Override
         public boolean ignoreHttpErrors() {
             return ignoreHttpErrors;
         }
 
+        @Override
         public SSLSocketFactory sslSocketFactory() {
             return sslSocketFactory;
         }
 
+        @Override
         public void sslSocketFactory(SSLSocketFactory sslSocketFactory) {
             this.sslSocketFactory = sslSocketFactory;
         }
 
+        @Override
         public Connection.Request ignoreHttpErrors(boolean ignoreHttpErrors) {
             this.ignoreHttpErrors = ignoreHttpErrors;
             return this;
         }
 
+        @Override
         public boolean ignoreContentType() {
             return ignoreContentType;
         }
 
+        @Override
         public Connection.Request ignoreContentType(boolean ignoreContentType) {
             this.ignoreContentType = ignoreContentType;
             return this;
         }
 
+        @Override
         public Request data(Connection.KeyVal keyval) {
             Validate.notNullParam(keyval, "keyval");
             data.add(keyval);
             return this;
         }
 
+        @Override
         public Collection<Connection.KeyVal> data() {
             return data;
         }
 
+        @Override
         public Connection.Request requestBody(@Nullable String body) {
             this.body = body;
             return this;
         }
 
+        @Override
         public String requestBody() {
             return body;
         }
 
+        @Override
         public Request parser(Parser parser) {
             this.parser = parser;
             parserDefined = true;
             return this;
         }
 
+        @Override
         public Parser parser() {
             return parser;
         }
 
+        @Override
         public Connection.Request postDataCharset(String charset) {
             Validate.notNullParam(charset, "charset");
             if (!Charset.isSupported(charset)) throw new IllegalCharsetNameException(charset);
@@ -783,6 +805,7 @@ public class HttpConnection implements Connection {
             return this;
         }
 
+        @Override
         public String postDataCharset() {
             return postDataCharset;
         }
@@ -880,7 +903,7 @@ public class HttpConnection implements Connection {
                     if (location.startsWith("http:/") && location.charAt(6) != '/') // fix broken Location: http:/temp/AAG_New/en/index.php
                         location = location.substring(6);
                     URL redir = StringUtil.resolve(req.url(), location);
-                    req.url(encodeUrl(redir));
+                    req.url(redir);
 
                     req.executing = false;
                     return execute(req, res);
@@ -930,23 +953,28 @@ public class HttpConnection implements Connection {
             return res;
         }
 
+        @Override
         public int statusCode() {
             return statusCode;
         }
 
+        @Override
         public String statusMessage() {
             return statusMessage;
         }
 
+        @Override
         public String charset() {
             return charset;
         }
 
+        @Override
         public Response charset(String charset) {
             this.charset = charset;
             return this;
         }
 
+        @Override
         public String contentType() {
             return contentType;
         }
@@ -981,16 +1009,18 @@ public class HttpConnection implements Connection {
             }
         }
 
+        @Override
         public String body() {
             prepareByteData();
             Validate.notNull(byteData);
             // charset gets set from header on execute, and from meta-equiv on parse. parse may not have happened yet
-            String body = (charset == null ? DataUtil.UTF_8 : Charset.forName(charset))
+            String body = (charset == null ? UTF_8 : Charset.forName(charset))
                 .decode(byteData).toString();
             ((Buffer)byteData).rewind(); // cast to avoid covariant return type change in jdk9
             return body;
         }
 
+        @Override
         public byte[] bodyAsBytes() {
             prepareByteData();
             Validate.notNull(byteData);
@@ -1219,32 +1249,13 @@ public class HttpConnection implements Connection {
 
         // for get url reqs, serialise the data map into the url
         private static void serialiseRequestUrl(Connection.Request req) throws IOException {
-            URL in = req.url();
-            StringBuilder url = StringUtil.borrowBuilder();
-            boolean first = true;
-            // reconstitute the query, ready for appends
-            url
-                .append(in.getProtocol())
-                .append("://")
-                .append(in.getAuthority()) // includes host, port
-                .append(in.getPath())
-                .append("?");
-            if (in.getQuery() != null) {
-                url.append(in.getQuery());
-                first = false;
-            }
+            UrlBuilder in = new UrlBuilder(req.url());
+
             for (Connection.KeyVal keyVal : req.data()) {
                 Validate.isFalse(keyVal.hasInputStream(), "InputStream data not supported in URL query string.");
-                if (!first)
-                    url.append('&');
-                else
-                    first = false;
-                url
-                    .append(URLEncoder.encode(keyVal.key(), DataUtil.defaultCharsetName))
-                    .append('=')
-                    .append(URLEncoder.encode(keyVal.value(), DataUtil.defaultCharsetName));
+                in.appendKeyVal(keyVal);
             }
-            req.url(new URL(StringUtil.releaseBuilder(url)));
+            req.url(in.build());
             req.data().clear(); // moved into url as get params
         }
     }
@@ -1280,22 +1291,26 @@ public class HttpConnection implements Connection {
             this.value = value;
         }
 
+        @Override
         public KeyVal key(String key) {
             Validate.notEmptyParam(key, "key");
             this.key = key;
             return this;
         }
 
+        @Override
         public String key() {
             return key;
         }
 
+        @Override
         public KeyVal value(String value) {
             Validate.notNullParam(value, "value");
             this.value = value;
             return this;
         }
 
+        @Override
         public String value() {
             return value;
         }
@@ -1306,10 +1321,12 @@ public class HttpConnection implements Connection {
             return this;
         }
 
+        @Override
         public InputStream inputStream() {
             return stream;
         }
 
+        @Override
         public boolean hasInputStream() {
             return stream != null;
         }
diff --git a/src/main/java/org/jsoup/helper/UrlBuilder.java b/src/main/java/org/jsoup/helper/UrlBuilder.java
new file mode 100644
index 00000000..89f46a13
--- /dev/null
+++ b/src/main/java/org/jsoup/helper/UrlBuilder.java
@@ -0,0 +1,104 @@
+package org.jsoup.helper;
+
+import org.jsoup.Connection;
+import org.jsoup.internal.StringUtil;
+
+import javax.annotation.Nullable;
+import java.io.UnsupportedEncodingException;
+import java.net.IDN;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+
+import static org.jsoup.helper.DataUtil.UTF_8;
+
+/**
+ A utility class to normalize input URLs. jsoup internal; API subject to change.
+ <p>Normalization includes puny-coding the host, and encoding non-ascii path components. Any non-ascii characters in
+ the query string (or the fragment/anchor) are escaped, but any existing escapes in those components are preserved.</p>
+ */
+final class UrlBuilder {
+    URL u;
+    @Nullable StringBuilder q;
+
+    UrlBuilder(URL inputUrl) {
+        this.u = inputUrl;
+        if (u.getQuery() != null)
+            q = StringUtil.borrowBuilder().append(u.getQuery());
+    }
+
+    URL build() {
+        try {
+            // use the URI class to encode non-ascii in path
+            URI uri = new URI(
+                u.getProtocol(),
+                u.getUserInfo(),
+                IDN.toASCII(decodePart(u.getHost())), // puny-code
+                u.getPort(),
+                decodePart(u.getPath()),
+                null, null // query and fragment appended later so as not to encode
+            );
+
+            String normUrl = uri.toASCIIString();
+            if (q != null || u.getRef() != null) {
+                StringBuilder sb = StringUtil.borrowBuilder().append(normUrl);
+                if (q != null) {
+                    sb.append('?');
+                    appendToAscii(StringUtil.releaseBuilder(q), true, sb);
+                }
+                if (u.getRef() != null) {
+                    sb.append('#');
+                    appendToAscii(u.getRef(), false, sb);
+                }
+                normUrl = StringUtil.releaseBuilder(sb);
+            }
+            u =  new URL(normUrl);
+            return u;
+        } catch (MalformedURLException | URISyntaxException | UnsupportedEncodingException e) {
+            // we assert here so that any incomplete normalization issues can be caught in devel. but in practise,
+            // the remote end will be able to handle it, so in prod we just pass the original URL.
+            // The UnsupportedEncodingException would never happen as always UTF8
+            assert Validate.assertFail(e.toString());
+            return u;
+        }
+    }
+
+    void appendKeyVal(Connection.KeyVal kv) throws UnsupportedEncodingException {
+        if (q == null)
+            q = StringUtil.borrowBuilder();
+        else
+            q.append('&');
+        q
+            .append(URLEncoder.encode(kv.key(), UTF_8.name()))
+            .append('=')
+            .append(URLEncoder.encode(kv.value(), UTF_8.name()));
+    }
+
+    private static String decodePart(String encoded) {
+        try {
+            return URLDecoder.decode(encoded, UTF_8.name());
+        } catch (UnsupportedEncodingException e) {
+            throw new RuntimeException(e); // wtf!
+        }
+    }
+
+    private static void appendToAscii(String s, boolean spaceAsPlus, StringBuilder sb) throws UnsupportedEncodingException {
+        // minimal normalization of Unicode -> Ascii, and space normal. Existing escapes are left as-is.
+        for (int i = 0; i < s.length(); i++) {
+            int c = s.codePointAt(i);
+            if (c == ' ') {
+                sb.append(spaceAsPlus ? '+' : "%20");
+            } else if (c > 127) { // out of ascii range
+                sb.append(URLEncoder.encode(new String(Character.toChars(c)), UTF_8.name()));
+                // ^^ is a bit heavy-handed - if perf critical, we could optimize
+            } else {
+                sb.append((char) c);
+            }
+        }
+    }
+
+
+}
diff --git a/src/main/java/org/jsoup/helper/Validate.java b/src/main/java/org/jsoup/helper/Validate.java
index 5a2409a7..1270b8cc 100644
--- a/src/main/java/org/jsoup/helper/Validate.java
+++ b/src/main/java/org/jsoup/helper/Validate.java
@@ -183,6 +183,17 @@ public final class Validate {
         throw new ValidationException(msg);
     }
 
+    /**
+     Cause a failure, but return false so it can be used in an assert statement.
+     @param msg message to output.
+     @return false, always
+     @throws IllegalStateException if we reach this state
+     */
+    static boolean assertFail(String msg) {
+        fail(msg);
+        return false;
+    }
+
     /**
      Cause a failure.
      @param msg message to output.
diff --git a/src/main/java/org/jsoup/helper/W3CDom.java b/src/main/java/org/jsoup/helper/W3CDom.java
index 8caf31f5..29296b1a 100644
--- a/src/main/java/org/jsoup/helper/W3CDom.java
+++ b/src/main/java/org/jsoup/helper/W3CDom.java
@@ -3,6 +3,7 @@ package org.jsoup.helper;
 import org.jsoup.internal.StringUtil;
 import org.jsoup.nodes.Attribute;
 import org.jsoup.nodes.Attributes;
+import org.jsoup.parser.HtmlTreeBuilder;
 import org.jsoup.select.NodeTraversor;
 import org.jsoup.select.NodeVisitor;
 import org.jsoup.select.Selector;
@@ -77,6 +78,8 @@ public class W3CDom {
 
     /**
      Update the namespace aware setting. This impacts the factory that is used to create W3C nodes from jsoup nodes.
+     <p>For HTML documents, controls if the document will be in the default {@code http://www.w3.org/1999/xhtml}
+     namespace if otherwise unset.</p>.
      @param namespaceAware the updated setting
      @return this W3CDom, for chaining.
      */
@@ -337,6 +340,7 @@ public class W3CDom {
     protected static class W3CBuilder implements NodeVisitor {
         private static final String xmlnsKey = "xmlns";
         private static final String xmlnsPrefix = "xmlns:";
+        private static final String xhtmlNs = "http://www.w3.org/1999/xhtml";
 
         private final Document doc;
         private boolean namespaceAware = true;
@@ -350,7 +354,12 @@ public class W3CDom {
             namespacesStack.push(new HashMap<>());
             dest = doc;
             contextElement = (org.jsoup.nodes.Element) doc.getUserData(ContextProperty); // Track the context jsoup Element, so we can save the corresponding w3c element
-        }
+            final org.jsoup.nodes.Document inDoc = contextElement.ownerDocument();
+            if (namespaceAware && inDoc != null && inDoc.parser().getTreeBuilder() instanceof HtmlTreeBuilder) {
+              // as per the WHATWG HTML5 spec § 2.1.3, elements are in the HTML namespace by default
+              namespacesStack.peek().put("", xhtmlNs);
+            }
+          }
 
         public void head(org.jsoup.nodes.Node source, int depth) {
             namespacesStack.push(new HashMap<>(namespacesStack.peek())); // inherit from above on the stack
@@ -366,9 +375,9 @@ public class W3CDom {
                 tagname to something safe, because that isn't going to be meaningful downstream. This seems(?) to be
                 how browsers handle the situation, also. https://github.com/jhy/jsoup/issues/1093 */
                 try {
-                    Element el = namespace == null && tagName.contains(":") ?
-                        doc.createElementNS("", tagName) : // doesn't have a real namespace defined
-                        doc.createElementNS(namespace, tagName);
+                    // use an empty namespace if none is present but the tag name has a prefix
+                    String imputedNamespace = namespace == null && tagName.contains(":") ? "" : namespace;
+                    Element el = doc.createElementNS(imputedNamespace, tagName);
                     copyAttributes(sourceEl, el);
                     append(el, sourceEl);
                     if (sourceEl == contextElement)
diff --git a/src/main/java/org/jsoup/internal/StringUtil.java b/src/main/java/org/jsoup/internal/StringUtil.java
index 73a589b1..a24cad5d 100644
--- a/src/main/java/org/jsoup/internal/StringUtil.java
+++ b/src/main/java/org/jsoup/internal/StringUtil.java
@@ -256,7 +256,7 @@ public final class StringUtil {
         final int len = haystack.length;
         for (int i = 0; i < len; i++) {
             if (haystack[i].equals(needle))
-            return true;
+               return true;
         }
         return false;
     }
diff --git a/src/main/java/org/jsoup/nodes/Comment.java b/src/main/java/org/jsoup/nodes/Comment.java
index 8ac8f702..f7fc9f3c 100644
--- a/src/main/java/org/jsoup/nodes/Comment.java
+++ b/src/main/java/org/jsoup/nodes/Comment.java
@@ -38,7 +38,7 @@ public class Comment extends LeafNode {
 
     @Override
 	void outerHtmlHead(Appendable accum, int depth, Document.OutputSettings out) throws IOException {
-        if (out.prettyPrint() && ((siblingIndex() == 0 && parentNode instanceof Element && ((Element) parentNode).tag().formatAsBlock()) || (out.outline() )))
+        if (out.prettyPrint() && ((isEffectivelyFirst() && parentNode instanceof Element && ((Element) parentNode).tag().formatAsBlock()) || (out.outline() )))
             indent(accum, depth, out);
         accum
                 .append("<!--")
diff --git a/src/main/java/org/jsoup/nodes/Element.java b/src/main/java/org/jsoup/nodes/Element.java
index 507fce96..05ee2e7b 100644
--- a/src/main/java/org/jsoup/nodes/Element.java
+++ b/src/main/java/org/jsoup/nodes/Element.java
@@ -1670,7 +1670,7 @@ public class Element extends Node {
     }
 
     boolean shouldIndent(final Document.OutputSettings out) {
-        return out.prettyPrint() && isFormatAsBlock(out) && !isInlineable(out);
+        return out.prettyPrint() && isFormatAsBlock(out) && !isInlineable(out) && !preserveWhitespace(parentNode);
     }
 
     @Override
@@ -1701,7 +1701,8 @@ public class Element extends Node {
     void outerHtmlTail(Appendable accum, int depth, Document.OutputSettings out) throws IOException {
         if (!(childNodes.isEmpty() && tag.isSelfClosing())) {
             if (out.prettyPrint() && (!childNodes.isEmpty() && (
-                    tag.formatAsBlock() || (out.outline() && (childNodes.size()>1 || (childNodes.size()==1 && (childNodes.get(0) instanceof Element))))
+                (tag.formatAsBlock() && !preserveWhitespace(parentNode)) ||
+                    (out.outline() && (childNodes.size()>1 || (childNodes.size()==1 && (childNodes.get(0) instanceof Element))))
             )))
                 indent(accum, depth, out);
             accum.append("</").append(tagName()).append('>');
@@ -1814,6 +1815,7 @@ public class Element extends Node {
     /**
      @deprecated Use {@link #forEach(Consumer)} instead.
      */
+    @Deprecated
     public Element forEach(org.jsoup.helper.Consumer<? super Element> action) {
         Validate.notNull(action);
         NodeTraversor.traverse((node, depth) -> {
@@ -1842,13 +1844,15 @@ public class Element extends Node {
     }
 
     private boolean isFormatAsBlock(Document.OutputSettings out) {
-        return tag.formatAsBlock() || (parent() != null && parent().tag().formatAsBlock()) || out.outline();
+        return tag.isBlock() || (parent() != null && parent().tag().formatAsBlock()) || out.outline();
     }
 
     private boolean isInlineable(Document.OutputSettings out) {
-        return tag().isInline()
-            && (parent() == null || parent().isBlock())
-            && previousSibling() != null
-            && !out.outline();
+        if (!tag.isInline())
+            return false;
+        return (parent() == null || parent().isBlock())
+            && !isEffectivelyFirst()
+            && !out.outline()
+            && !isNode("br");
     }
 }
diff --git a/src/main/java/org/jsoup/nodes/Entities.java b/src/main/java/org/jsoup/nodes/Entities.java
index d330005d..be447bce 100644
--- a/src/main/java/org/jsoup/nodes/Entities.java
+++ b/src/main/java/org/jsoup/nodes/Entities.java
@@ -7,6 +7,7 @@ import org.jsoup.nodes.Document.OutputSettings;
 import org.jsoup.parser.CharacterReader;
 import org.jsoup.parser.Parser;
 
+import javax.annotation.Nullable;
 import java.io.IOException;
 import java.nio.charset.CharsetEncoder;
 import java.util.Arrays;
@@ -26,7 +27,6 @@ public class Entities {
     static final int codepointRadix = 36;
     private static final char[] codeDelims = {',', ';'};
     private static final HashMap<String, String> multipoints = new HashMap<>(); // name -> multiple character references
-    private static final OutputSettings DefaultOutput = new OutputSettings();
 
     public enum EscapeMode {
         /**
@@ -157,8 +157,11 @@ public class Entities {
      * @return the escaped string
      */
     public static String escape(String string) {
+        if (DefaultOutput == null)
+            DefaultOutput = new OutputSettings();
         return escape(string, DefaultOutput);
     }
+    private static @Nullable OutputSettings DefaultOutput; // lazy-init, to break circular dependency with OutputSettings
 
     // this method does a lot, but other breakups cause rescanning and stringbuilder generations
     static void escape(Appendable accum, String string, OutputSettings out,
diff --git a/src/main/java/org/jsoup/nodes/Node.java b/src/main/java/org/jsoup/nodes/Node.java
index fc5ac3bf..46bd1cc6 100644
--- a/src/main/java/org/jsoup/nodes/Node.java
+++ b/src/main/java/org/jsoup/nodes/Node.java
@@ -687,6 +687,7 @@ public abstract class Node implements Cloneable {
     /**
      @deprecated Use {@link #forEachNode(Consumer)} instead.
      */
+    @Deprecated
     public Node forEachNode(org.jsoup.helper.Consumer<? super Node> action) {
         Validate.notNull(action);
         NodeTraversor.traverse((node, depth) -> action.accept(node), this);
@@ -762,6 +763,16 @@ public abstract class Node implements Cloneable {
         return normalName().equals(normalName);
     }
 
+    /** Test if this node is the first child, or first following blank text. */
+    final boolean isEffectivelyFirst() {
+        if (siblingIndex == 0) return true;
+        if (siblingIndex == 1) {
+            final Node prev = previousSibling();
+            return prev instanceof TextNode && (((TextNode) prev).isBlank());
+        }
+        return false;
+    }
+
     /**
      * Gets this node's outer HTML.
      * @return outer HTML.
diff --git a/src/main/java/org/jsoup/parser/CharacterReader.java b/src/main/java/org/jsoup/parser/CharacterReader.java
index df902b16..1d00ec6f 100644
--- a/src/main/java/org/jsoup/parser/CharacterReader.java
+++ b/src/main/java/org/jsoup/parser/CharacterReader.java
@@ -116,6 +116,11 @@ public final class CharacterReader {
         return readerPos + bufPos;
     }
 
+    /** Tests if the buffer has been fully read. */
+    boolean readFully() {
+        return readFully;
+    }
+
     /**
      Enables or disables line number tracking. By default, will be <b>off</b>.Tracking line numbers improves the
      legibility of parser error messages, for example. Tracking should be enabled before any content is read to be of
diff --git a/src/main/java/org/jsoup/parser/HtmlTreeBuilder.java b/src/main/java/org/jsoup/parser/HtmlTreeBuilder.java
index 785643e5..be0498c5 100644
--- a/src/main/java/org/jsoup/parser/HtmlTreeBuilder.java
+++ b/src/main/java/org/jsoup/parser/HtmlTreeBuilder.java
@@ -10,6 +10,7 @@ import org.jsoup.nodes.Element;
 import org.jsoup.nodes.FormElement;
 import org.jsoup.nodes.Node;
 import org.jsoup.nodes.TextNode;
+import org.jsoup.parser.Token.StartTag;
 
 import javax.annotation.Nullable;
 import javax.annotation.ParametersAreNonnullByDefault;
@@ -53,7 +54,7 @@ public class HtmlTreeBuilder extends TreeBuilder {
     private @Nullable Element contextElement; // fragment parse context -- could be null even if fragment parsing
     private ArrayList<Element> formattingElements; // active (open) formatting elements
     private ArrayList<HtmlTreeBuilderState> tmplInsertMode; // stack of Template Insertion modes
-    private List<String> pendingTableCharacters; // chars in table to be shifted out
+    private List<Token.Character> pendingTableCharacters; // chars in table to be shifted out
     private Token.EndTag emptyEnd; // reused empty end tag
 
     private boolean framesetOk; // if ok to go into frameset
@@ -227,13 +228,7 @@ public class HtmlTreeBuilder extends TreeBuilder {
     }
 
     Element insert(final Token.StartTag startTag) {
-        // cleanup duplicate attributes:
-        if (startTag.hasAttributes() && !startTag.attributes.isEmpty()) {
-            int dupes = startTag.attributes.deduplicate(settings);
-            if (dupes > 0) {
-                error("Dropped duplicate attribute(s) in tag [%s]", startTag.normalName);
-            }
-        }
+        dedupeAttributes(startTag);
 
         // handle empty unknown tags
         // when the spec expects an empty tag, will directly hit insertEmpty, so won't generate this fake end tag.
@@ -250,7 +245,7 @@ public class HtmlTreeBuilder extends TreeBuilder {
         return el;
     }
 
-    Element insertStartTag(String startTagName) {
+	Element insertStartTag(String startTagName) {
         Element el = new Element(tagFor(startTagName, settings), null);
         insert(el);
         return el;
@@ -267,6 +262,8 @@ public class HtmlTreeBuilder extends TreeBuilder {
     }
 
     Element insertEmpty(Token.StartTag startTag) {
+        dedupeAttributes(startTag);
+
         Tag tag = tagFor(startTag.name(), settings);
         Element el = new Element(tag, null, settings.normalizeAttributes(startTag.attributes));
         insertNode(el, startTag);
@@ -282,6 +279,8 @@ public class HtmlTreeBuilder extends TreeBuilder {
     }
 
     FormElement insertForm(Token.StartTag startTag, boolean onStack, boolean checkTemplateStack) {
+        dedupeAttributes(startTag);
+
         Tag tag = tagFor(startTag.name(), settings);
         FormElement el = new FormElement(tag, null, settings.normalizeAttributes(startTag.attributes));
         if (checkTemplateStack) {
@@ -301,9 +300,14 @@ public class HtmlTreeBuilder extends TreeBuilder {
         insertNode(comment, commentToken);
     }
 
+    /** Inserts the provided character token into the current element. */
     void insert(Token.Character characterToken) {
+        final Element el = currentElement(); // will be doc if no current element; allows for whitespace to be inserted into the doc root object (not on the stack)
+        insert(characterToken, el);
+    }
+
+    void insert(Token.Character characterToken, Element el) {
         final Node node;
-        Element el = currentElement(); // will be doc if no current element; allows for whitespace to be inserted into the doc root object (not on the stack)
         final String tagName = el.normalName();
         final String data = characterToken.getData();
 
@@ -317,6 +321,7 @@ public class HtmlTreeBuilder extends TreeBuilder {
         onNodeInserted(node, characterToken);
     }
 
+    /** Inserts the provided character token into the provided element. Use when not going onto stack element */
     private void insertNode(Node node, @Nullable Token token) {
         // if the stack hasn't been set up yet, elements (doctype, comments) go into the doc
         if (stack.isEmpty())
@@ -334,6 +339,16 @@ public class HtmlTreeBuilder extends TreeBuilder {
         onNodeInserted(node, token);
     }
 
+    /** Cleanup duplicate attributes. **/
+    private void dedupeAttributes(StartTag startTag) {
+        if (startTag.hasAttributes() && !startTag.attributes.isEmpty()) {
+            int dupes = startTag.attributes.deduplicate(settings);
+            if (dupes > 0) {
+                error("Dropped duplicate attribute(s) in tag [%s]", startTag.normalName);
+            }
+        }
+    }
+
     Element pop() {
         int size = stack.size();
         return stack.remove(size-1);
@@ -632,6 +647,20 @@ public class HtmlTreeBuilder extends TreeBuilder {
         return false;
     }
 
+    /** Tests if there is some element on the stack that is not in the provided set. */
+    boolean onStackNot(String[] allowedTags) {
+        final int bottom = stack.size() -1;
+        final int top = bottom > MaxScopeSearchDepth ? bottom - MaxScopeSearchDepth : 0;
+        // don't walk too far up the tree
+
+        for (int pos = bottom; pos >= top; pos--) {
+            final String elName = stack.get(pos).normalName();
+            if (!inSorted(elName, allowedTags))
+                return true;
+        }
+        return false;
+    }
+
     void setHeadElement(Element headElement) {
         this.headElement = headElement;
     }
@@ -656,14 +685,20 @@ public class HtmlTreeBuilder extends TreeBuilder {
         this.formElement = formElement;
     }
 
-    void newPendingTableCharacters() {
+    void resetPendingTableCharacters() {
         pendingTableCharacters = new ArrayList<>();
     }
 
-    List<String> getPendingTableCharacters() {
+    List<Token.Character> getPendingTableCharacters() {
         return pendingTableCharacters;
     }
 
+    void addPendingTableCharacters(Token.Character c) {
+        // make a clone of the token to maintain its state (as Tokens are otherwise reset)
+        Token.Character clone = c.clone();
+        pendingTableCharacters.add(clone);
+    }
+
     /**
      13.2.6.3 Closing elements that have implied end tags
      When the steps below require the UA to generate implied end tags, then, while the current node is a dd element, a dt element, an li element, an optgroup element, an option element, a p element, an rb element, an rp element, an rt element, or an rtc element, the UA must pop the current node off the stack of open elements.
diff --git a/src/main/java/org/jsoup/parser/HtmlTreeBuilderState.java b/src/main/java/org/jsoup/parser/HtmlTreeBuilderState.java
index baa16c46..3c5352ec 100644
--- a/src/main/java/org/jsoup/parser/HtmlTreeBuilderState.java
+++ b/src/main/java/org/jsoup/parser/HtmlTreeBuilderState.java
@@ -6,7 +6,6 @@ import org.jsoup.nodes.Attributes;
 import org.jsoup.nodes.Document;
 import org.jsoup.nodes.DocumentType;
 import org.jsoup.nodes.Element;
-import org.jsoup.nodes.Node;
 
 import java.util.ArrayList;
 
@@ -310,7 +309,8 @@ enum HtmlTreeBuilderState {
                 case EOF:
                     if (tb.templateModeSize() > 0)
                         return tb.process(t, InTemplate);
-                    // todo: error if stack contains something not dd, dt, li, p, tbody, td, tfoot, th, thead, tr, body, html
+                    if (tb.onStackNot(InBodyEndOtherErrors))
+                        tb.error(this);
                     // stop parsing
                     break;
             }
@@ -726,16 +726,22 @@ enum HtmlTreeBuilderState {
                         tb.error(this);
                         return false;
                     } else {
-                        // todo: error if stack contains something not dd, dt, li, optgroup, option, p, rp, rt, tbody, td, tfoot, th, thead, tr, body, html
-                        anyOtherEndTag(t, tb);
+                        if (tb.onStackNot(InBodyEndOtherErrors))
+                            tb.error(this);
                         tb.transition(AfterBody);
                     }
                     break;
                 case "html":
-                    boolean notIgnored = tb.processEndTag("body");
-                    if (notIgnored)
-                        return tb.process(endTag);
-                    break;
+                    if (!tb.onStack("body")) {
+                        tb.error(this);
+                        return false; // ignore
+                    } else {
+                        if (tb.onStackNot(InBodyEndOtherErrors))
+                            tb.error(this);
+                        tb.transition(AfterBody);
+                        return tb.process(t); // re-process
+                    }
+
                 case "form":
                     if (!tb.onStack("template")) {
                         Element currentForm = tb.getFormElement();
@@ -988,7 +994,7 @@ enum HtmlTreeBuilderState {
     InTable {
         boolean process(Token t, HtmlTreeBuilder tb) {
             if (t.isCharacter() && inSorted(tb.currentElement().normalName(), InTableFoster)) {
-                tb.newPendingTableCharacters();
+                tb.resetPendingTableCharacters();
                 tb.markInsertionMode();
                 tb.transition(InTableText);
                 return tb.process(t);
@@ -1099,25 +1105,25 @@ enum HtmlTreeBuilderState {
                     tb.error(this);
                     return false;
                 } else {
-                    tb.getPendingTableCharacters().add(c.getData());
+                    tb.addPendingTableCharacters(c);
                 }
-            } else {// todo - don't really like the way these table character data lists are built
+            } else {
                 if (tb.getPendingTableCharacters().size() > 0) {
-                    for (String character : tb.getPendingTableCharacters()) {
-                        if (!isWhitespace(character)) {
+                    for (Token.Character c : tb.getPendingTableCharacters()) {
+                        if (!isWhitespace(c)) {
                             // InTable anything else section:
                             tb.error(this);
                             if (inSorted(tb.currentElement().normalName(), InTableFoster)) {
                                 tb.setFosterInserts(true);
-                                tb.process(new Token.Character().data(character), InBody);
+                                tb.process(c, InBody);
                                 tb.setFosterInserts(false);
                             } else {
-                                tb.process(new Token.Character().data(character), InBody);
+                                tb.process(c, InBody);
                             }
                         } else
-                            tb.insert(new Token.Character().data(character));
+                            tb.insert(c);
                     }
-                    tb.newPendingTableCharacters();
+                    tb.resetPendingTableCharacters();
                 }
                 tb.transition(tb.originalState());
                 return tb.process(t);
@@ -1594,7 +1600,12 @@ enum HtmlTreeBuilderState {
     AfterBody {
         boolean process(Token t, HtmlTreeBuilder tb) {
             if (isWhitespace(t)) {
-                tb.insert(t.asCharacter()); // out of spec - include whitespace. spec would move into body
+                // spec deviation - currently body is still on stack, but we want this to go to the html node
+                Element html = tb.getFromStack("html");
+                if (html != null)
+                    tb.insert(t.asCharacter(), html);
+                else
+                    tb.process(t, InBody); // will get into body
             } else if (t.isComment()) {
                 tb.insert(t.asComment()); // into html node
             } else if (t.isDoctype()) {
@@ -1607,7 +1618,6 @@ enum HtmlTreeBuilderState {
                     tb.error(this);
                     return false;
                 } else {
-                    if (tb.onStack("html")) tb.popStackToClose("html");
                     tb.transition(AfterAfterBody);
                 }
             } else if (t.isEOF()) {
@@ -1699,7 +1709,9 @@ enum HtmlTreeBuilderState {
             } else if (t.isDoctype() || (t.isStartTag() && t.asStartTag().normalName().equals("html"))) {
                 return tb.process(t, InBody);
             } else if (isWhitespace(t)) {
-                tb.insert(t.asCharacter());
+                // spec deviation - body and html still on stack, but want this space to go after </html>
+                Element doc = tb.getDocument();
+                tb.insert(t.asCharacter(), doc);
             }else if (t.isEOF()) {
                 // nice work chuck
             } else {
@@ -1746,10 +1758,6 @@ enum HtmlTreeBuilderState {
         return false;
     }
 
-    private static boolean isWhitespace(String data) {
-        return StringUtil.isBlank(data);
-    }
-
     private static void handleRcData(Token.StartTag startTag, HtmlTreeBuilder tb) {
         tb.tokeniser.transition(TokeniserState.Rcdata);
         tb.markInsertionMode();
@@ -1786,6 +1794,7 @@ enum HtmlTreeBuilderState {
         static final String[] InBodyEndClosers = new String[]{"address", "article", "aside", "blockquote", "button", "center", "details", "dir", "div",
             "dl", "fieldset", "figcaption", "figure", "footer", "header", "hgroup", "listing", "menu",
             "nav", "ol", "pre", "section", "summary", "ul"};
+        static final String[] InBodyEndOtherErrors = new String[] {"body", "dd", "dt", "html", "li", "optgroup", "option", "p", "rb", "rp", "rt", "rtc", "tbody", "td", "tfoot", "th", "thead", "tr"};
         static final String[] InBodyEndAdoptionFormatters = new String[]{"a", "b", "big", "code", "em", "font", "i", "nobr", "s", "small", "strike", "strong", "tt", "u"};
         static final String[] InBodyEndTableFosters = new String[]{"table", "tbody", "tfoot", "thead", "tr"};
         static final String[] InTableToBody = new String[]{"tbody", "tfoot", "thead"};
diff --git a/src/main/java/org/jsoup/parser/Tag.java b/src/main/java/org/jsoup/parser/Tag.java
index d573033a..97ed5004 100644
--- a/src/main/java/org/jsoup/parser/Tag.java
+++ b/src/main/java/org/jsoup/parser/Tag.java
@@ -242,11 +242,12 @@ public class Tag implements Cloneable {
     };
     private static final String[] inlineTags = {
             "object", "base", "font", "tt", "i", "b", "u", "big", "small", "em", "strong", "dfn", "code", "samp", "kbd",
-            "var", "cite", "abbr", "time", "acronym", "mark", "ruby", "rt", "rp", "a", "img", "br", "wbr", "map", "q",
+            "var", "cite", "abbr", "time", "acronym", "mark", "ruby", "rt", "rp", "rtc", "a", "img", "br", "wbr", "map", "q",
             "sub", "sup", "bdo", "iframe", "embed", "span", "input", "select", "textarea", "label", "button", "optgroup",
             "option", "legend", "datalist", "keygen", "output", "progress", "meter", "area", "param", "source", "track",
             "summary", "command", "device", "area", "basefont", "bgsound", "menuitem", "param", "source", "track",
-            "data", "bdi", "s", "strike", "nobr"
+            "data", "bdi", "s", "strike", "nobr",
+            "rb" // deprecated but still known / special handling
     };
     private static final String[] emptyTags = {
             "meta", "link", "base", "frame", "img", "br", "wbr", "embed", "hr", "input", "keygen", "col", "command",
diff --git a/src/main/java/org/jsoup/parser/Token.java b/src/main/java/org/jsoup/parser/Token.java
index 819b8aef..b0fc0af8 100644
--- a/src/main/java/org/jsoup/parser/Token.java
+++ b/src/main/java/org/jsoup/parser/Token.java
@@ -382,7 +382,7 @@ abstract class Token {
         }
     }
 
-    static class Character extends Token {
+    static class Character extends Token implements Cloneable {
         private String data;
 
         Character() {
@@ -410,6 +410,14 @@ abstract class Token {
         public String toString() {
             return getData();
         }
+
+        @Override protected Token.Character clone() {
+            try {
+                return (Token.Character) super.clone();
+            } catch (CloneNotSupportedException e) {
+                throw new RuntimeException(e);
+            }
+        }
     }
 
     final static class CData extends Character {
diff --git a/src/main/java/org/jsoup/parser/TokeniserState.java b/src/main/java/org/jsoup/parser/TokeniserState.java
index 874fed0c..f269fc62 100644
--- a/src/main/java/org/jsoup/parser/TokeniserState.java
+++ b/src/main/java/org/jsoup/parser/TokeniserState.java
@@ -186,7 +186,7 @@ enum TokeniserState {
             if (r.matches('/')) {
                 t.createTempBuffer();
                 t.advanceTransition(RCDATAEndTagOpen);
-            } else if (r.matchesAsciiAlpha() && t.appropriateEndTagName() != null && !r.containsIgnoreCase(t.appropriateEndTagSeq())) {
+            } else if (r.readFully() && r.matchesAsciiAlpha() && t.appropriateEndTagName() != null &&  !r.containsIgnoreCase(t.appropriateEndTagSeq())) {
                 // diverge from spec: got a start tag, but there's no appropriate end tag (</title>), so rather than
                 // consuming to EOF; break out here
                 t.tagPending = t.createTagPending(false).name(t.appropriateEndTagName());
diff --git a/src/main/java/org/jsoup/select/Collector.java b/src/main/java/org/jsoup/select/Collector.java
index de3abb11..a3820da0 100644
--- a/src/main/java/org/jsoup/select/Collector.java
+++ b/src/main/java/org/jsoup/select/Collector.java
@@ -74,11 +74,5 @@ public class Collector {
             }
             return CONTINUE;
         }
-
-        @Override
-        public FilterResult tail(Node node, int depth) {
-            return CONTINUE;
-        }
     }
-
 }


Test patch:
diff --git a/src/test/java/org/jsoup/helper/HttpConnectionTest.java b/src/test/java/org/jsoup/helper/HttpConnectionTest.java
index f7618fb9..840d8a82 100644
--- a/src/test/java/org/jsoup/helper/HttpConnectionTest.java
+++ b/src/test/java/org/jsoup/helper/HttpConnectionTest.java
@@ -256,8 +256,22 @@ public class HttpConnectionTest {
 
     @Test public void encodeUrl() throws MalformedURLException {
         URL url1 = new URL("https://test.com/foo bar/[One]?q=white space#frag");
-        URL url2 = HttpConnection.encodeUrl(url1);
-        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white%20space#frag", url2.toExternalForm());
+        URL url2 = new UrlBuilder(url1).build();
+        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white+space#frag", url2.toExternalForm());
+    }
+
+    @Test void encodedUrlDoesntDoubleEncode() throws MalformedURLException {
+        URL url1 = new URL("https://test.com/foo bar/[One]?q=white space#frag ment");
+        URL url2 = new UrlBuilder(url1).build();
+        URL url3 = new UrlBuilder(url2).build();
+        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white+space#frag%20ment", url2.toExternalForm());
+        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white+space#frag%20ment", url3.toExternalForm());
+    }
+
+    @Test void connectToEncodedUrl() {
+        Connection connect = Jsoup.connect("https://example.com/a%20b%20c?query+string");
+        URL url = connect.request().url();
+        assertEquals("https://example.com/a%20b%20c?query+string", url.toExternalForm());
     }
 
     @Test public void noUrlThrowsValidationError() throws IOException {
@@ -289,6 +303,18 @@ public class HttpConnectionTest {
         assertEquals(puny, req.url().toExternalForm());
     }
 
+    @Test void supportsIdnWithPort() throws MalformedURLException {
+        String idn = "https://www.测试.测试:9001/foo.html?bar";
+        String puny = "https://www.xn--0zwm56d.xn--0zwm56d:9001/foo.html?bar";
+
+        Connection con = Jsoup.connect(idn);
+        assertEquals(puny, con.request().url().toExternalForm());
+
+        HttpConnection.Request req = new HttpConnection.Request();
+        req.url(new URL(idn));
+        assertEquals(puny, req.url().toExternalForm());
+    }
+
     @Test public void validationErrorsOnExecute() throws IOException {
         Connection con = new HttpConnection();
         boolean urlThrew = false;
diff --git a/src/test/java/org/jsoup/helper/W3CDomTest.java b/src/test/java/org/jsoup/helper/W3CDomTest.java
index f89832a6..c1daeb57 100644
--- a/src/test/java/org/jsoup/helper/W3CDomTest.java
+++ b/src/test/java/org/jsoup/helper/W3CDomTest.java
@@ -7,7 +7,6 @@ import org.jsoup.nodes.Element;
 import org.jsoup.nodes.TextNode;
 import org.junit.jupiter.api.Test;
 import org.w3c.dom.Document;
-import org.w3c.dom.DocumentType;
 import org.w3c.dom.Node;
 import org.w3c.dom.NodeList;
 import org.xml.sax.InputSource;
@@ -62,7 +61,7 @@ public class W3CDomTest {
         assertEquals(0, meta.getLength());
 
         String out = W3CDom.asString(wDoc, W3CDom.OutputXml());
-        String expected = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><html><head><title>W3c</title></head><body><p class=\"one\" id=\"12\">Text</p><!-- comment --><invalid>What<script>alert('!')</script></invalid></body></html>";
+        String expected = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><title>W3c</title></head><body><p class=\"one\" id=\"12\">Text</p><!-- comment --><invalid>What<script>alert('!')</script></invalid></body></html>";
         assertEquals(expected, TextUtil.stripNewlines(out));
 
         Document roundTrip = parseXml(out, true);
@@ -74,7 +73,7 @@ public class W3CDomTest {
         String furtherOut = W3CDom.asString(wDoc, properties);
         assertTrue(furtherOut.length() > out.length()); // wanted to assert formatting, but actual indentation is platform specific so breaks in CI
         String furtherExpected =
-            "<?xml version=\"1.0\" encoding=\"UTF-8\"?><html><head><title>W3c</title></head><body><p class=\"one\" id=\"12\">Text</p><!-- comment --><invalid>What<script>alert('!')</script></invalid></body></html>";
+            "<?xml version=\"1.0\" encoding=\"UTF-8\"?><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><title>W3c</title></head><body><p class=\"one\" id=\"12\">Text</p><!-- comment --><invalid>What<script>alert('!')</script></invalid></body></html>";
         assertEquals(furtherExpected, TextUtil.stripNewlines(furtherOut)); // on windows, DOM will write newlines as \r\n
     }
 
@@ -151,7 +150,7 @@ public class W3CDomTest {
 
         Document w3Doc = W3CDom.convert(jsoupDoc);
         String xml = W3CDom.asString(w3Doc, W3CDom.OutputXml());
-        assertEquals("<?xml version=\"1.0\" encoding=\"UTF-8\"?><html><head/><body name=\"\" style=\"color: red\"/></html>", xml);
+        assertEquals("<?xml version=\"1.0\" encoding=\"UTF-8\"?><html xmlns=\"http://www.w3.org/1999/xhtml\"><head/><body name=\"\" style=\"color: red\"/></html>", xml);
     }
 
     @Test
@@ -162,7 +161,7 @@ public class W3CDomTest {
 
         Document w3Doc = W3CDom.convert(jsoupDoc);
         String out = W3CDom.asString(w3Doc, W3CDom.OutputHtml());
-        String expected = "<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p hành=\"1\" hình=\"2\">unicode attr names</p></body></html>";
+        String expected = "<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p hành=\"1\" hình=\"2\">unicode attr names</p></body></html>";
         assertEquals(expected, TextUtil.stripNewlines(out));
     }
 
@@ -175,7 +174,7 @@ public class W3CDomTest {
 
         Document w3Doc = W3CDom.convert(jsoupDoc);
         String out = W3CDom.asString(w3Doc, W3CDom.OutputHtml());
-        String expected = "<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p hnh=\"2\">unicode attr names coerced</p></body></html>";
+        String expected = "<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p hnh=\"2\">unicode attr names coerced</p></body></html>";
         assertEquals(expected, TextUtil.stripNewlines(out));
     }
 
@@ -185,7 +184,7 @@ public class W3CDomTest {
 
         Document w3Doc = W3CDom.convert(jsoup);
         String xml = W3CDom.asString(w3Doc, W3CDom.OutputXml());
-        assertEquals("<?xml version=\"1.0\" encoding=\"UTF-8\"?><html><head/><body>&lt;インセンティブで高収入！&gt;Text <p>More</p></body></html>", xml);
+        assertEquals("<?xml version=\"1.0\" encoding=\"UTF-8\"?><html xmlns=\"http://www.w3.org/1999/xhtml\"><head/><body>&lt;インセンティブで高収入！&gt;Text <p>More</p></body></html>", xml);
     }
 
     @Test
@@ -196,7 +195,7 @@ public class W3CDomTest {
         Document w3Doc = new W3CDom().fromJsoup(doc);
         Node htmlEl = w3Doc.getFirstChild();
 
-        assertNull(htmlEl.getNamespaceURI());
+        assertEquals("http://www.w3.org/1999/xhtml", htmlEl.getNamespaceURI());
         assertEquals("html", htmlEl.getLocalName());
         assertEquals("html", htmlEl.getNodeName());
 
@@ -211,7 +210,7 @@ public class W3CDomTest {
         W3CDom w3c = new W3CDom();
         String html = "<html><body><div>hello</div></body></html>";
         Document dom = w3c.fromJsoup(Jsoup.parse(html));
-        NodeList nodeList = xpath(dom, "//body");// no ns, so needs no prefix
+        NodeList nodeList = xpath(dom, "//*[local-name()=\"body\"]");// namespace aware; HTML namespace is default
         assertEquals("div", nodeList.item(0).getLocalName());
 
         // default output is namespace aware, so query needs to be as well
@@ -242,6 +241,17 @@ public class W3CDomTest {
         assertNull(nodeList);
     }
 
+    @Test
+    public void xhtmlNoNamespace() throws XPathExpressionException {
+        W3CDom w3c = new W3CDom();
+        String html = "<html><body><div>hello</div></body></html>";
+        w3c.namespaceAware(false);
+        Document dom = w3c.fromJsoup(Jsoup.parse(html));
+        NodeList nodeList = xpath(dom, "//body");// no namespace
+        assertEquals(1, nodeList.getLength());
+        assertEquals("div", nodeList.item(0).getLocalName());
+    }
+
     @Test
     void canDisableNamespaces() throws XPathExpressionException {
         W3CDom w3c = new W3CDom();
@@ -266,26 +276,25 @@ public class W3CDomTest {
         // TODO - not super happy with this output - but plain DOM doesn't let it out, and don't want to rebuild the writer
         // because we have Saxon on the test classpath, the transformer will change to that, and so case may change (e.g. Java base in META, Saxon is meta for HTML)
         String base = "<!DOCTYPE html><p>One</p>";
-        assertEqualsIgnoreCase("<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p>One</p></body></html>",
-            output(base, true));
-        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html SYSTEM \"about:legacy-compat\"><html><head/><body><p>One</p></body></html>", output(base, false));
+        assertEqualsIgnoreCase("<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p>One</p></body></html>", output(base, true));
+        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html SYSTEM \"about:legacy-compat\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head/><body><p>One</p></body></html>", output(base, false));
 
         String publicDoc = "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">";
-        assertEqualsIgnoreCase("<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body></body></html>", output(publicDoc, true));
+        assertEqualsIgnoreCase("<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body></body></html>", output(publicDoc, true));
         // different impls will have different XML formatting. OpenJDK 13 default gives this: <body /> but others have <body/>, so just check start
         assertTrue(output(publicDoc, false).startsWith("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html PUBLIC"));
 
         String systemDoc = "<!DOCTYPE html SYSTEM \"exampledtdfile.dtd\">";
-        assertEqualsIgnoreCase("<!DOCTYPE html SYSTEM \"exampledtdfile.dtd\"><html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body></body></html>", output(systemDoc, true));
-        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html SYSTEM \"exampledtdfile.dtd\"><html><head/><body/></html>", output(systemDoc, false));
+        assertEqualsIgnoreCase("<!DOCTYPE html SYSTEM \"exampledtdfile.dtd\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body></body></html>", output(systemDoc, true));
+        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html SYSTEM \"exampledtdfile.dtd\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head/><body/></html>", output(systemDoc, false));
 
         String legacyDoc = "<!DOCTYPE html SYSTEM \"about:legacy-compat\">";
-        assertEqualsIgnoreCase("<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body></body></html>", output(legacyDoc, true));
-        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html SYSTEM \"about:legacy-compat\"><html><head/><body/></html>", output(legacyDoc, false));
+        assertEqualsIgnoreCase("<!DOCTYPE html SYSTEM \"about:legacy-compat\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body></body></html>", output(legacyDoc, true));
+        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><!DOCTYPE html SYSTEM \"about:legacy-compat\"><html xmlns=\"http://www.w3.org/1999/xhtml\"><head/><body/></html>", output(legacyDoc, false));
 
         String noDoctype = "<p>One</p>";
-        assertEqualsIgnoreCase("<html><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p>One</p></body></html>", output(noDoctype, true));
-        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><html><head/><body><p>One</p></body></html>", output(noDoctype, false));
+        assertEqualsIgnoreCase("<html xmlns=\"http://www.w3.org/1999/xhtml\"><head><META http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"></head><body><p>One</p></body></html>", output(noDoctype, true));
+        assertEqualsIgnoreCase("<?xml version=\"1.0\" encoding=\"UTF-8\"?><html xmlns=\"http://www.w3.org/1999/xhtml\"><head/><body><p>One</p></body></html>", output(noDoctype, false));
     }
 
     private String output(String in, boolean modeHtml) {
@@ -300,6 +309,24 @@ public class W3CDomTest {
         assertEquals(want.toLowerCase(Locale.ROOT), have.toLowerCase(Locale.ROOT));
     }
 
+
+    @Test
+    public void canOutputHtmlWithoutNamespace() {
+        String html = "<p>One</p>";
+        org.jsoup.nodes.Document jdoc = Jsoup.parse(html);
+        W3CDom w3c = new W3CDom();
+        w3c.namespaceAware(false);
+
+        String asHtml = W3CDom.asString(w3c.fromJsoup(jdoc), W3CDom.OutputHtml());
+        String asXtml = W3CDom.asString(w3c.fromJsoup(jdoc), W3CDom.OutputXml());
+        assertEqualsIgnoreCase(
+            "<html><head><meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\"></head><body><p>one</p></body></html>",
+            asHtml);
+        assertEqualsIgnoreCase(
+            "<?xml version=\"1.0\" encoding=\"UTF-8\"?><html><head/><body><p>One</p></body></html>",
+            asXtml);
+    }
+
     @Test public void convertsElementsAndMaintainsSource() {
         org.jsoup.nodes.Document jdoc = Jsoup.parse("<body><div><p>One</div><div><p>Two");
         W3CDom w3CDom = new W3CDom();
diff --git a/src/test/java/org/jsoup/integration/ConnectTest.java b/src/test/java/org/jsoup/integration/ConnectTest.java
index 4881e128..07f9d2eb 100644
--- a/src/test/java/org/jsoup/integration/ConnectTest.java
+++ b/src/test/java/org/jsoup/integration/ConnectTest.java
@@ -3,6 +3,7 @@ package org.jsoup.integration;
 import org.jsoup.Connection;
 import org.jsoup.HttpStatusException;
 import org.jsoup.Jsoup;
+import org.jsoup.helper.DataUtil;
 import org.jsoup.helper.W3CDom;
 import org.jsoup.integration.servlets.*;
 import org.jsoup.internal.StringUtil;
@@ -20,6 +21,7 @@ import java.io.FileInputStream;
 import java.io.IOException;
 import java.net.MalformedURLException;
 import java.net.URL;
+import java.net.URLDecoder;
 import java.util.List;
 import java.util.Map;
 
@@ -749,4 +751,13 @@ public class ConnectTest {
         Document doc = Jsoup.connect(url).get();
         assertEquals(url, doc.location());
     }
+
+    @Test void fetchUnicodeUrl() throws IOException {
+        String url = EchoServlet.Url + "/✔/?鍵=値";
+        Document doc = Jsoup.connect(url).get();
+
+        assertEquals("/✔/", ihVal("Path Info", doc));
+        assertEquals("%E9%8D%B5=%E5%80%A4", ihVal("Query String", doc));
+        assertEquals("鍵=値", URLDecoder.decode(ihVal("Query String", doc), DataUtil.UTF_8.name()));
+    }
 }
diff --git a/src/test/java/org/jsoup/integration/servlets/EchoServlet.java b/src/test/java/org/jsoup/integration/servlets/EchoServlet.java
index 9931f978..f5bf2ce3 100644
--- a/src/test/java/org/jsoup/integration/servlets/EchoServlet.java
+++ b/src/test/java/org/jsoup/integration/servlets/EchoServlet.java
@@ -67,6 +67,7 @@ public class EchoServlet extends BaseServlet {
         // some get items
         write(w, "Method", req.getMethod());
         write(w, "Request URI", req.getRequestURI());
+        write(w, "Path Info", req.getPathInfo());
         write(w, "Query String", req.getQueryString());
 
         // request headers (why is it an enumeration?)
diff --git a/src/test/java/org/jsoup/nodes/CommentTest.java b/src/test/java/org/jsoup/nodes/CommentTest.java
index fe1c1361..c622f184 100644
--- a/src/test/java/org/jsoup/nodes/CommentTest.java
+++ b/src/test/java/org/jsoup/nodes/CommentTest.java
@@ -41,6 +41,20 @@ public class CommentTest {
         assertEquals("<!-- a simple comment -->", c1.outerHtml());
     }
 
+    @Test void stableIndentInBlock() {
+        String html = "<div><!-- comment --> Text</div><p><!-- comment --> Text</p>";
+        Document doc = Jsoup.parse(html);
+        String out = doc.body().html();
+        assertEquals("<div>\n" +
+            " <!-- comment --> Text\n" +
+            "</div>\n" +
+            "<p><!-- comment --> Text</p>", out);
+
+        Document doc2 = Jsoup.parse(out);
+        String out2 = doc2.body().html();
+        assertEquals(out, out2);
+    }
+
     @Test
     public void testClone() {
         Comment c1 = comment.clone();
diff --git a/src/test/java/org/jsoup/nodes/ElementTest.java b/src/test/java/org/jsoup/nodes/ElementTest.java
index 0887a4e7..5ff30415 100644
--- a/src/test/java/org/jsoup/nodes/ElementTest.java
+++ b/src/test/java/org/jsoup/nodes/ElementTest.java
@@ -157,6 +157,14 @@ public class ElementTest {
         assertEquals("<pre><code><span><b>code\n\ncode</b></span></code></pre>", doc.body().html());
     }
 
+    @Test void doesNotWrapBlocksInPre() {
+        // https://github.com/jhy/jsoup/issues/1891
+        String h = "<pre><span><foo><div>TEST\n TEST</div></foo></span></pre>";
+        Document doc = Jsoup.parse(h);
+        assertEquals("TEST\n TEST", doc.wholeText());
+        assertEquals(h, doc.body().html());
+    }
+
     @Test
     public void testBrHasSpace() {
         Document doc = Jsoup.parse("<p>Hello<br>there</p>");
@@ -2306,6 +2314,17 @@ public class ElementTest {
         assertEquals("<div>\n <br>\n</div>", doc.body().html()); // not div\n br\n \n/div
     }
 
+    @Test void prettyprintBrWhenNotFirstChild() {
+        // https://github.com/jhy/jsoup/issues/1911
+        String h = "<div><p><br>Foo</p><br></div>";
+        Document doc = Jsoup.parse(h);
+        assertEquals("<div>\n" +
+            " <p><br>\n  Foo</p>\n" +
+            " <br>\n" +
+            "</div>", doc.body().html());
+        // br gets wrapped if in div, but not in p (block vs inline), but always wraps after
+    }
+
     @Test void preformatFlowsToChildTextNodes() {
         // https://github.com/jhy/jsoup/issues/1776
         String html = "<div><pre>One\n<span>\nTwo</span>\n <span>  \nThree</span>\n <span>Four <span>Five</span>\n  Six\n</pre>";
@@ -2343,6 +2362,23 @@ public class ElementTest {
         assertEquals(expectOwn, div.child(0).wholeOwnText());
     }
 
+    @Test void inlineInBlockShouldIndent() {
+        // was inconsistent between <div>\n<span> and <div><span> - former would print inline, latter would wrap(!)
+        String html = "<div>One <span>Hello</span><span>!</span></div><div>\n<span>There</span></div><div> <span>Now</span></div>";
+        Document doc = Jsoup.parse(html);
+        assertEquals(
+            "<div>\n" +
+                " One <span>Hello</span><span>!</span>\n" +
+                "</div>\n" +
+                "<div>\n" +
+                " <span>There</span>\n" +
+                "</div>\n" +
+                "<div>\n" +
+                " <span>Now</span>\n" +
+                "</div>",
+            doc.body().html());
+    }
+
     @Test void testExpectFirst() {
         Document doc = Jsoup.parse("<p>One</p><p>Two <span>Three</span> <span>Four</span>");
 
@@ -2445,7 +2481,9 @@ public class ElementTest {
     @Test void divAInlineable() {
         String html = "<body><div> <a>Text</a>";
         Document doc = Jsoup.parse(html);
-        assertEquals("<div><a>Text</a>\n</div>", doc.body().html());
+        assertEquals("<div>\n" +
+            " <a>Text</a>\n" +
+            "</div>", doc.body().html());
     }
 
     @Test void noDanglingSpaceAfterCustomElement() {
@@ -2657,4 +2695,34 @@ public class ElementTest {
         doc.body().outerHtml(builder);
         assertEquals("<body>\n <div>\n  One\n </div>\n</body>", builder.toString());
     }
-}
\ No newline at end of file
+
+    @Test void rubyInline() {
+        String html = "<ruby>T<rp>(</rp><rtc>!</rtc><rt>)</rt></ruby>";
+        Document doc = Jsoup.parse(html);
+        assertEquals(html, doc.body().html());
+    }
+
+    @Test void nestedFormatAsInlinePrintsAsBlock() {
+        // https://github.com/jhy/jsoup/issues/1926
+        String h = "        <table>\n" +
+            "            <tr>\n" +
+            "                <td>\n" +
+            "                    <p style=\"display:inline;\">A</p>\n" +
+            "                    <p style=\"display:inline;\">B</p>\n" +
+            "                </td>\n" +
+            "            </tr>\n" +
+            "        </table>";
+        Document doc = Jsoup.parse(h);
+        String out = doc.body().html();
+        assertEquals("<table>\n" +
+            " <tbody>\n" +
+            "  <tr>\n" +
+            "   <td>\n" +
+            "    <p style=\"display:inline;\">A</p>\n" +
+            "    <p style=\"display:inline;\">B</p></td>\n" +
+            "  </tr>\n" +
+            " </tbody>\n" +
+            "</table>", out);
+        // todo - I would prefer the </td> to wrap down there - but need to reimplement pretty printer to simplify and track indented state
+    }
+}
diff --git a/src/test/java/org/jsoup/nodes/PositionTest.java b/src/test/java/org/jsoup/nodes/PositionTest.java
index 813ed9b3..069d5c6a 100644
--- a/src/test/java/org/jsoup/nodes/PositionTest.java
+++ b/src/test/java/org/jsoup/nodes/PositionTest.java
@@ -3,9 +3,13 @@ package org.jsoup.nodes;
 import org.jsoup.Jsoup;
 import org.jsoup.integration.servlets.FileServlet;
 import org.jsoup.parser.Parser;
+import org.jsoup.select.Elements;
+import org.jsoup.select.NodeTraversor;
 import org.junit.jupiter.api.Test;
 
 import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
 
 import static org.junit.jupiter.api.Assertions.*;
 
@@ -169,4 +173,34 @@ class PositionTest {
         assertEquals("17,5:779-17,12:786", item.endSourceRange().toString());
     }
 
+    @Test void tracksTableMovedText() {
+        String html = "<table>foo<tr>bar<td>baz</td>qux</tr>coo</table>";
+        Document doc = Jsoup.parse(html, TrackingParser);
+
+        List<TextNode> textNodes = new ArrayList<>();
+        NodeTraversor.traverse((Node node, int depth) -> {
+            if (node instanceof TextNode) {
+                textNodes.add((TextNode) node);
+            }
+        }, doc);
+
+        assertEquals(5, textNodes.size());
+        assertEquals("1,8:7-1,11:10", textNodes.get(0).sourceRange().toString());
+        assertEquals("1,15:14-1,18:17", textNodes.get(1).sourceRange().toString());
+        assertEquals("1,22:21-1,25:24", textNodes.get(2).sourceRange().toString());
+        assertEquals("1,30:29-1,33:32", textNodes.get(3).sourceRange().toString());
+        assertEquals("1,38:37-1,41:40", textNodes.get(4).sourceRange().toString());
+    }
+
+    @Test void tracksClosingHtmlTagsInXml() {
+        // verifies https://github.com/jhy/jsoup/issues/1935
+        String xml = "<p>One</p><title>Two</title><data>Three</data>";
+        Document doc = Jsoup.parse(xml, Parser.xmlParser().setTrackPosition(true));
+        Elements els = doc.children();
+        for (Element el : els) {
+            assertTrue(el.sourceRange().isTracked());
+            assertTrue(el.endSourceRange().isTracked());
+        }
+    }
+
 }
\ No newline at end of file
diff --git a/src/test/java/org/jsoup/parser/CharacterReaderTest.java b/src/test/java/org/jsoup/parser/CharacterReaderTest.java
index d9f22801..37d74948 100644
--- a/src/test/java/org/jsoup/parser/CharacterReaderTest.java
+++ b/src/test/java/org/jsoup/parser/CharacterReaderTest.java
@@ -1,5 +1,6 @@
 package org.jsoup.parser;
 
+import org.jsoup.UncheckedIOException;
 import org.jsoup.integration.ParseTest;
 import org.junit.jupiter.api.Test;
 
@@ -61,6 +62,12 @@ public class CharacterReaderTest {
 
         assertEquals(CharacterReader.EOF, r.consume());
         assertTrue(r.isEmpty());
+
+        // unconsume all remaining characters
+        for (int i = 0; i < 4; i++) {
+            r.unconsume();
+        }
+        assertThrows(UncheckedIOException.class, r::unconsume);
     }
 
     @Test public void mark() {
@@ -78,6 +85,12 @@ public class CharacterReaderTest {
         assertEquals(2, r.pos());
     }
 
+    @Test public void rewindToMark() {
+        CharacterReader r = new CharacterReader("nothing");
+        // marking should be invalid
+        assertThrows(UncheckedIOException.class, r::rewindToMark);
+    }
+
     @Test public void consumeToEnd() {
         String in = "one two three";
         CharacterReader r = new CharacterReader(in);
@@ -265,6 +278,20 @@ public class CharacterReaderTest {
         assertTrue(r.matchesAny(scan));
         assertEquals('\n', r.consume());
         assertFalse(r.matchesAny(scan));
+        // nothing to match
+        r.consumeToEnd();
+        assertTrue(r.isEmpty());
+        assertFalse(r.matchesAny(scan));
+    }
+
+    @Test public void matchesDigit() {
+        CharacterReader r = new CharacterReader("42");
+        r.consumeToEnd();
+        assertTrue(r.isEmpty());
+        // nothing to match
+        assertFalse(r.matchesDigit());
+        r.unconsume();
+        assertTrue(r.matchesDigit());
     }
 
     @Test public void cachesStrings() {
diff --git a/src/test/java/org/jsoup/parser/HtmlParserTest.java b/src/test/java/org/jsoup/parser/HtmlParserTest.java
index 1f83518e..4172d0f5 100644
--- a/src/test/java/org/jsoup/parser/HtmlParserTest.java
+++ b/src/test/java/org/jsoup/parser/HtmlParserTest.java
@@ -7,13 +7,16 @@ import org.jsoup.internal.StringUtil;
 import org.jsoup.nodes.*;
 import org.jsoup.safety.Safelist;
 import org.jsoup.select.Elements;
-import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
 
 import java.io.ByteArrayInputStream;
 import java.io.File;
 import java.io.IOException;
 import java.util.List;
+import java.util.stream.Stream;
 
 import static org.jsoup.parser.ParseSettings.preserveCase;
 import static org.junit.jupiter.api.Assertions.*;
@@ -46,16 +49,25 @@ public class HtmlParserTest {
         assertEquals("foo > bar", p.attr("class"));
     }
 
-    @Test public void dropsDuplicateAttributes() {
-        String html = "<p One=One ONE=Two Two=two one=Three One=Four two=Five>Text</p>";
+    @ParameterizedTest @MethodSource("dupeAttributeData")
+    public void dropsDuplicateAttributes(String html, String expected) {
         Parser parser = Parser.htmlParser().setTrackErrors(10);
         Document doc = parser.parseInput(html, "");
 
-        Element p = doc.selectFirst("p");
-        assertEquals("<p one=\"One\" two=\"two\">Text</p>", p.outerHtml()); // normalized names due to lower casing
+        Element el = doc.expectFirst("body > *");
+        assertEquals(expected, el.outerHtml()); // normalized names due to lower casing
+        String tag = el.normalName();
 
         assertEquals(1, parser.getErrors().size());
-        assertEquals("Dropped duplicate attribute(s) in tag [p]", parser.getErrors().get(0).getErrorMessage());
+        assertEquals("Dropped duplicate attribute(s) in tag [" + tag + "]", parser.getErrors().get(0).getErrorMessage());
+    }
+
+    private static Stream<Arguments> dupeAttributeData() {
+        return Stream.of(
+            Arguments.of("<p One=One ONE=Two Two=two one=Three One=Four two=Five>Text</p>", "<p one=\"One\" two=\"two\">Text</p>"),
+            Arguments.of("<img One=One ONE=Two Two=two one=Three One=Four two=Five>", "<img one=\"One\" two=\"two\">"),
+            Arguments.of("<form One=One ONE=Two Two=two one=Three One=Four two=Five></form>", "<form one=\"One\" two=\"two\"></form>")
+        );
     }
 
     @Test public void retainsAttributesOfDifferentCaseIfSensitive() {
@@ -720,15 +732,8 @@ public class HtmlParserTest {
         // and the <i> inside the table and does not leak out.
         String h = "<p><b>One</p> <table><tr><td><p><i>Three<p>Four</i></td></tr></table> <p>Five</p>";
         Document doc = Jsoup.parse(h);
-        String want = "<p><b>One</b></p><b>\n" +
-            " <table>\n" +
-            "  <tbody>\n" +
-            "   <tr>\n" +
-            "    <td><p><i>Three</i></p><p><i>Four</i></p></td>\n" +
-            "   </tr>\n" +
-            "  </tbody>\n" +
-            " </table><p>Five</p></b>";
-        assertEquals(want, doc.body().html());
+        String want = "<p><b>One</b></p><b><table><tbody><tr><td><p><i>Three</i></p><p><i>Four</i></p></td></tr></tbody></table><p>Five</p></b>";
+        assertEquals(want, TextUtil.stripNewlines(doc.body().html()));
     }
 
     @Test public void commentBeforeHtml() {
@@ -777,7 +782,7 @@ public class HtmlParserTest {
 
         Document two = Jsoup.parse("<title>One<b>Two <p>Test</p>"); // no title, so <b> causes </title> breakout
         assertEquals("One", two.title());
-        assertEquals("<b>Two <p>Test</p></b>", two.body().html());
+        assertEquals("<b>Two \n <p>Test</p></b>", two.body().html());
     }
 
     @Test public void handlesUnclosedScriptAtEof() {
@@ -1247,7 +1252,7 @@ public class HtmlParserTest {
         File in = ParseTest.getFile("/htmltests/comments.html");
         Document doc = Jsoup.parse(in, "UTF-8");
 
-        assertEquals("<!--?xml version=\"1.0\" encoding=\"utf-8\"?--><!-- so --> <!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><!-- what --> <html xml:lang=\"en\" lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\"><!-- now --> <head><!-- then --> <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"> <title>A Certain Kind of Test</title> </head> <body> <h1>Hello</h1>h1&gt; (There is a UTF8 hidden BOM at the top of this file.) </body> </html>",
+        assertEquals("<!--?xml version=\"1.0\" encoding=\"utf-8\"?--><!-- so --> <!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><!-- what --> <html xml:lang=\"en\" lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\"> <!-- now --> <head> <!-- then --> <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"> <title>A Certain Kind of Test</title> </head> <body> <h1>Hello</h1>h1&gt; (There is a UTF8 hidden BOM at the top of this file.) </body> </html>",
             StringUtil.normaliseWhitespace(doc.html()));
 
         assertEquals("A Certain Kind of Test", doc.head().select("title").text());
@@ -1470,7 +1475,7 @@ public class HtmlParserTest {
         assertEquals(1, nodes.size());
         Node node = nodes.get(0);
         assertEquals("h2", node.nodeName());
-        assertEquals("<p><h2>text</h2></p>", node.parent().outerHtml());
+        assertEquals("<p>\n <h2>text</h2></p>", node.parent().outerHtml());
     }
 
     @Test public void nestedPFragments() {
@@ -1479,7 +1484,7 @@ public class HtmlParserTest {
         List<Node> nodes = new Document("").parser().parseFragmentInput(bareFragment, new Element("p"), "");
         assertEquals(2, nodes.size());
         Node node = nodes.get(0);
-        assertEquals("<p><p></p><a></a></p>", node.parent().outerHtml()); // mis-nested because fragment forced into the element, OK
+        assertEquals("<p>\n <p></p><a></a></p>", node.parent().outerHtml()); // mis-nested because fragment forced into the element, OK
     }
 
     @Test public void nestedAnchorAdoption() {
@@ -1698,11 +1703,53 @@ public class HtmlParserTest {
         parser.setTrackErrors(10);
         Document doc = Jsoup.parse(html, parser);
         ParseErrorList errors = parser.getErrors();
-        assertEquals(1, errors.size());
+        assertEquals(2, errors.size());
         Element ruby = doc.expectFirst("ruby");
         assertEquals(
             "<ruby><div><rp>Hello</rp></div></ruby>",
             TextUtil.stripNewlines(ruby.outerHtml()));
         assertEquals("<1:16>: Unexpected StartTag token [<rp>] when in state [InBody]", errors.get(0).toString());
     }
+
+    @Test void errorOnEofIfOpen() {
+        String html = "<div>";
+        Parser parser = Parser.htmlParser();
+        parser.setTrackErrors(10);
+        Document doc = Jsoup.parse(html, parser);
+        ParseErrorList errors = parser.getErrors();
+        assertEquals(1, errors.size());
+        assertEquals("Unexpected EOF token [] when in state [InBody]", errors.get(0).getErrorMessage());
+    }
+
+    @Test void NoErrorOnEofIfBodyOpen() {
+        String html = "<body>";
+        Parser parser = Parser.htmlParser();
+        parser.setTrackErrors(10);
+        Document doc = Jsoup.parse(html, parser);
+        ParseErrorList errors = parser.getErrors();
+        assertEquals(0, errors.size());
+    }
+
+    @Test void htmlClose() {
+        // https://github.com/jhy/jsoup/issues/1851
+        String html = "<body><div>One</html>Two</div></body>";
+        Document doc = Jsoup.parse(html);
+        assertEquals("OneTwo", doc.expectFirst("body > div").text());
+    }
+
+    @Test void largeTextareaContents() {
+        // https://github.com/jhy/jsoup/issues/1929
+        StringBuilder sb = new StringBuilder();
+        int num = 2000;
+        for (int i = 0; i <= num; i++) {
+            sb.append("\n<text>foo</text>\n");
+        }
+        String textContent = sb.toString();
+        String sourceHtml = "<textarea>" + textContent + "</textarea>";
+
+        Document doc = Jsoup.parse(sourceHtml);
+        Element textArea = doc.expectFirst("textarea");
+
+        assertEquals(textContent, textArea.wholeText());
+    }
 }
diff --git a/src/test/java/org/jsoup/parser/HtmlTreeBuilderStateTest.java b/src/test/java/org/jsoup/parser/HtmlTreeBuilderStateTest.java
index f05ee268..55b828e8 100644
--- a/src/test/java/org/jsoup/parser/HtmlTreeBuilderStateTest.java
+++ b/src/test/java/org/jsoup/parser/HtmlTreeBuilderStateTest.java
@@ -45,7 +45,7 @@ public class HtmlTreeBuilderStateTest {
     public void ensureArraysAreSorted() {
         List<Object[]> constants = findConstantArrays(Constants.class);
         ensureSorted(constants);
-        assertEquals(38, constants.size());
+        assertEquals(39, constants.size());
     }
 
     @Test public void ensureTagSearchesAreKnownTags() {
@@ -75,7 +75,8 @@ public class HtmlTreeBuilderStateTest {
         String s = Jsoup.parse(html).toString();
         assertEquals("<html>\n" +
             " <head></head>\n" +
-            " <body><a href=\"#1\"> </a>\n" +
+            " <body>\n" +
+            "  <a href=\"#1\"> </a>\n" +
             "  <div>\n" +
             "   <a href=\"#1\"> </a><a href=\"#2\">child</a>\n" +
             "  </div>\n" +
@@ -99,7 +100,8 @@ public class HtmlTreeBuilderStateTest {
         String s = Jsoup.parse(html).toString();
         assertEquals("<html>\n" +
             " <head></head>\n" +
-            " <body><a href=\"#1\"> </a>\n" +
+            " <body>\n" +
+            "  <a href=\"#1\"> </a>\n" +
             "  <div>\n" +
             "   <a href=\"#1\"> </a>\n" +
             "   <div>\n" +
diff --git a/src/test/java/org/jsoup/parser/XmlTreeBuilderTest.java b/src/test/java/org/jsoup/parser/XmlTreeBuilderTest.java
index 227dd096..d359a528 100644
--- a/src/test/java/org/jsoup/parser/XmlTreeBuilderTest.java
+++ b/src/test/java/org/jsoup/parser/XmlTreeBuilderTest.java
@@ -94,7 +94,7 @@ public class XmlTreeBuilderTest {
     public void testDoesNotForceSelfClosingKnownTags() {
         // html will force "<br>one</br>" to logically "<br />One<br />". XML should be stay "<br>one</br> -- don't recognise tag.
         Document htmlDoc = Jsoup.parse("<br>one</br>");
-        assertEquals("<br>\none<br>", htmlDoc.body().html());
+        assertEquals("<br>\none\n<br>", htmlDoc.body().html());
 
         Document xmlDoc = Jsoup.parse("<br>one</br>", "", Parser.xmlParser());
         assertEquals("<br>one</br>", xmlDoc.html());
diff --git a/src/test/java/org/jsoup/select/ElementsTest.java b/src/test/java/org/jsoup/select/ElementsTest.java
index eab17c2c..d1895752 100644
--- a/src/test/java/org/jsoup/select/ElementsTest.java
+++ b/src/test/java/org/jsoup/select/ElementsTest.java
@@ -193,7 +193,9 @@ public class ElementsTest {
         String h = "<div><font>One</font> <font><a href=\"/\">Two</a></font></div";
         Document doc = Jsoup.parse(h);
         doc.select("font").unwrap();
-        assertEquals("<div>One <a href=\"/\">Two</a></div>", TextUtil.stripNewlines(doc.body().html()));
+        assertEquals("<div>\n" +
+            " One <a href=\"/\">Two</a>\n" +
+            "</div>", doc.body().html());
     }
 
     @Test public void unwrapP() {