Repository: jhy/jsoup
Pull request #1918 (2023-03-20 23:56:48)

Bump maven-release-plugin from 2.5.3 to 3.0.0
Bumps [maven-release-plugin](https://github.com/apache/maven-release) from 2.5.3 to 3.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/apache/maven-release/releases">maven-release-plugin's releases</a>.</em></p>
<blockquote>
<h2>3.0.0-M6</h2>
<!-- raw HTML omitted -->
<ul>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1096">[MRELEASE-1096]</a> - Many cleanups and updates (<a href="https://redirect.github.com/apache/maven-release/issues/132">#132</a>) <a href="https://github.com/nielsbasjes"><code>@​nielsbasjes</code></a></li>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1098">[MRELEASE-1098]</a> - Drop o.a.m.shared.release.env.ReleaseEnvironment.getL… (<a href="https://redirect.github.com/apache/maven-release/issues/136">#136</a>) <a href="https://github.com/michael-o"><code>@​michael-o</code></a></li>
</ul>
<h2>🚀 New features and improvements</h2>
<ul>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1097">[MRELEASE-1097]</a> - rework plugin output to clarify goal's phases (<a href="https://redirect.github.com/apache/maven-release/issues/135">#135</a>) <a href="https://github.com/hboutemy"><code>@​hboutemy</code></a></li>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1091">[MRELEASE-1091]</a> - Use non-interactive mode for forked Maven process (<a href="https://redirect.github.com/apache/maven-release/issues/125">#125</a>) <a href="https://github.com/slawekjaranowski"><code>@​slawekjaranowski</code></a></li>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1082">[MRELEASE-1082]</a> - configuration option for using a shallow clone (<a href="https://redirect.github.com/apache/maven-release/issues/121">#121</a>) <a href="https://github.com/BigMichi1"><code>@​BigMichi1</code></a></li>
</ul>
<h2>🐛 Bug Fixes</h2>
<ul>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1022">[MRELEASE-1022]</a> - Use versions defined in <code>release.properties</code> when doing <code>release:prepare</code> (<a href="https://redirect.github.com/apache/maven-release/issues/111">#111</a>) <a href="https://github.com/sleberrigaud"><code>@​sleberrigaud</code></a></li>
<li>Fix xsd NS (<a href="https://redirect.github.com/apache/maven-release/issues/102">#102</a>) <a href="https://github.com/pzygielo"><code>@​pzygielo</code></a></li>
</ul>
<h2>📦 Dependency updates</h2>
<ul>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1095">[MRELEASE-1095]</a> - Upgrade Maven SCM to 1.13.0 (<a href="https://redirect.github.com/apache/maven-release/issues/131">#131</a>) <a href="https://github.com/michael-o"><code>@​michael-o</code></a></li>
<li>Bump plexus-utils from 3.4.1 to 3.4.2 (<a href="https://redirect.github.com/apache/maven-release/issues/128">#128</a>) <a href="https://github.com/dependabot"><code>@​dependabot</code></a></li>
<li>Bump slf4j-simple from 1.7.5 to 1.7.36 (<a href="https://redirect.github.com/apache/maven-release/issues/124">#124</a>) <a href="https://github.com/dependabot"><code>@​dependabot</code></a></li>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1087">[MRELEASE-1087]</a> - Upgrade Maven to 3.2.5 (and de-plexus) (<a href="https://redirect.github.com/apache/maven-release/issues/118">#118</a>) <a href="https://github.com/cstamas"><code>@​cstamas</code></a></li>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1085">[MRELEASE-1085]</a> - Upgrade Maven Invoker to 3.2.0 (<a href="https://redirect.github.com/apache/maven-release/issues/117">#117</a>) <a href="https://github.com/slawekjaranowski"><code>@​slawekjaranowski</code></a></li>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1086">[MRELEASE-1086]</a> - Upgrade Parent to 36 (<a href="https://redirect.github.com/apache/maven-release/issues/116">#116</a>) <a href="https://github.com/slawekjaranowski"><code>@​slawekjaranowski</code></a></li>
<li>Bump plexus-interpolation from 1.17 to 1.26 (<a href="https://redirect.github.com/apache/maven-release/issues/107">#107</a>) <a href="https://github.com/dependabot"><code>@​dependabot</code></a></li>
<li>Bump maven-invoker-plugin from 3.2.0 to 3.2.2 (<a href="https://redirect.github.com/apache/maven-release/issues/99">#99</a>) <a href="https://github.com/dependabot"><code>@​dependabot</code></a></li>
<li>Bump jdom2 from 2.0.6 to 2.0.6.1 (<a href="https://redirect.github.com/apache/maven-release/issues/96">#96</a>) <a href="https://github.com/dependabot"><code>@​dependabot</code></a></li>
<li>Bump mrm-maven-plugin from 1.2.0 to 1.3.0 (<a href="https://redirect.github.com/apache/maven-release/issues/95">#95</a>) <a href="https://github.com/dependabot"><code>@​dependabot</code></a></li>
</ul>
<h2>👻 Maintenance</h2>
<ul>
<li><a href="https://issues.apache.org/jira/browse/MRELEASE-1088">[MRELEASE-1088]</a> - Remove parsing of CLI arguments (<a href="https://redirect.github.com/apache/maven-release/issues/120">#120</a>) <a href="https://github.com/slawekjaranowski"><code>@​slawekjaranowski</code></a></li>
</ul>
<h2>🔧 Build</h2>
<ul>
<li>add release drafter configuration (<a href="https://redirect.github.com/apache/maven-release/issues/112">#112</a>) <a href="https://github.com/olamy"><code>@​olamy</code></a></li>
</ul>
<h2>3.0.0-M5</h2>
<h2>What's Changed</h2>
<ul>
<li>Update rollback-release.apt by <a href="https://github.com/sanlaville"><code>@​sanlaville</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/64">apache/maven-release#64</a></li>
<li>Bump junit from 4.13.1 to 4.13.2 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/67">apache/maven-release#67</a></li>
<li>Bump scmVersion from 1.11.2 to 1.11.3 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/70">apache/maven-release#70</a></li>
<li>Bump plexus-cipher from 1.4 to 1.7 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/66">apache/maven-release#66</a></li>
<li>Bump commons-lang3 from 3.7 to 3.12.0 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/69">apache/maven-release#69</a></li>
<li>[N/A] Fix spelling error in log message by <a href="https://github.com/klaraward"><code>@​klaraward</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/78">apache/maven-release#78</a></li>
<li>Bump xmlunit-core from 2.6.4 to 2.8.2 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/71">apache/maven-release#71</a></li>
<li>Bump maven-plugin-plugin from 3.6.0 to 3.6.1 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/73">apache/maven-release#73</a></li>
<li>Remove unused formal parameters by <a href="https://github.com/pzygielo"><code>@​pzygielo</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/88">apache/maven-release#88</a></li>
<li>Change interface implemented by MavenArtifactVersion by <a href="https://github.com/pzygielo"><code>@​pzygielo</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/89">apache/maven-release#89</a></li>
<li>[MRELEASE-1075] Move CVS-based tests to Subversion or Git by <a href="https://github.com/michael-o"><code>@​michael-o</code></a> in <a href="https://redirect.github.com/apache/maven-release/pull/90">apache/maven-release#90</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/apache/maven-release/commit/41253314909f1935a72c261d740871c463664c57"><code>4125331</code></a> [maven-release-plugin] prepare release maven-release-3.0.0</li>
<li><a href="https://github.com/apache/maven-release/commit/bd9cdebc6e40f7bb30e0efbb9aaa17d4cf8f7866"><code>bd9cdeb</code></a> Improve dependency declarations</li>
<li><a href="https://github.com/apache/maven-release/commit/231e949d4ed69aaf8d84f41df5cba550587b61f0"><code>231e949</code></a> [MRELEASE-1118] Upgrade Maven SCM to 2.0.0</li>
<li><a href="https://github.com/apache/maven-release/commit/8c19cdc301fe28804a8a34f095789942c40975d6"><code>8c19cdc</code></a> Fix <a href="https://github.com/since"><code>@​since</code></a></li>
<li><a href="https://github.com/apache/maven-release/commit/e2006edbbb9a621d8ed3f2c745f1a9f5bb1c8ec8"><code>e2006ed</code></a> update links (<a href="https://redirect.github.com/apache/maven-release/issues/175">#175</a>)</li>
<li><a href="https://github.com/apache/maven-release/commit/0da97a2639d6ef72481a486376661607baa91ffe"><code>0da97a2</code></a> use more modern I/O that does not rely on the system encoding (<a href="https://redirect.github.com/apache/maven-release/issues/174">#174</a>)</li>
<li><a href="https://github.com/apache/maven-release/commit/422ccc4d6c97a64c4f3e3fda5f2f45ac68d165a0"><code>422ccc4</code></a> http --&gt; https (<a href="https://redirect.github.com/apache/maven-release/issues/177">#177</a>)</li>
<li><a href="https://github.com/apache/maven-release/commit/262254ff96f48dff92ab0af3454b8318d7c47307"><code>262254f</code></a> grammar nits (<a href="https://redirect.github.com/apache/maven-release/issues/178">#178</a>)</li>
<li><a href="https://github.com/apache/maven-release/commit/d60bbc34d0ced1f5005c32f3b9e1d03d0b9582c9"><code>d60bbc3</code></a> Prefer more modern try with resources over IOUtil.close (<a href="https://redirect.github.com/apache/maven-release/issues/179">#179</a>)</li>
<li><a href="https://github.com/apache/maven-release/commit/3e077e7064a964f904e24aab8fe57232f503fd81"><code>3e077e7</code></a> Merge pull request <a href="https://redirect.github.com/apache/maven-release/issues/171">#171</a> from apache/dependabot/maven/org.codehaus.mojo-mrm-ma...</li>
<li>Additional commits viewable in <a href="https://github.com/apache/maven-release/compare/maven-release-2.5.3...maven-release-3.0.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.maven.plugins:maven-release-plugin&package-manager=maven&previous-version=2.5.3&new-version=3.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>


Patch:
diff --git a/CHANGES b/CHANGES
index 5fd3b451..d9830573 100644
--- a/CHANGES
+++ b/CHANGES
@@ -1,6 +1,9 @@
 jsoup changelog
 
 Release 1.16.1 [PENDING]
+  * Improvement: in Jsoup.connect(url), support URLs with Unicode characters in the path
+    <https://github.com/jhy/jsoup/issues/1914>
+
   * Improvement: Calling Node.remove() on a node with no parent is now a no-op, vs a validation error.
     <https://github.com/jhy/jsoup/issues/1898>
 
diff --git a/pom.xml b/pom.xml
index 035047fa..a90e214c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -168,12 +168,12 @@
       </plugin>
       <plugin>
         <artifactId>maven-release-plugin</artifactId>
-        <version>2.5.3</version>
+        <version>3.0.0</version>
       </plugin>
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-surefire-plugin</artifactId>
-        <version>3.0.0-M9</version>
+        <version>3.0.0</version>
         <configuration>
           <!-- smaller stack to find stack overflows -->
           <argLine>-Xss256k</argLine>
@@ -199,14 +199,14 @@
         <!-- API version compat check - https://siom79.github.io/japicmp/ -->
         <groupId>com.github.siom79.japicmp</groupId>
         <artifactId>japicmp-maven-plugin</artifactId>
-        <version>0.17.1</version>
+        <version>0.17.2</version>
         <configuration>
           <!-- hard code previous version; can't detect when running stateless on build server -->
           <oldVersion>
             <dependency>
               <groupId>org.jsoup</groupId>
               <artifactId>jsoup</artifactId>
-              <version>1.15.3</version>
+              <version>1.15.4</version>
               <type>jar</type>
             </dependency>
           </oldVersion>
diff --git a/src/main/java/org/jsoup/helper/HttpConnection.java b/src/main/java/org/jsoup/helper/HttpConnection.java
index 88595c3f..d12b6a2b 100644
--- a/src/main/java/org/jsoup/helper/HttpConnection.java
+++ b/src/main/java/org/jsoup/helper/HttpConnection.java
@@ -20,18 +20,13 @@ import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
 import java.io.OutputStreamWriter;
-import java.io.UnsupportedEncodingException;
 import java.net.CookieManager;
 import java.net.CookieStore;
 import java.net.HttpURLConnection;
-import java.net.IDN;
 import java.net.InetSocketAddress;
 import java.net.MalformedURLException;
 import java.net.Proxy;
-import java.net.URI;
-import java.net.URISyntaxException;
 import java.net.URL;
-import java.net.URLDecoder;
 import java.net.URLEncoder;
 import java.nio.Buffer;
 import java.nio.ByteBuffer;
@@ -49,6 +44,7 @@ import java.util.zip.Inflater;
 import java.util.zip.InflaterInputStream;
 
 import static org.jsoup.Connection.Method.HEAD;
+import static org.jsoup.helper.DataUtil.UTF_8;
 import static org.jsoup.internal.Normalizer.lowerCase;
 
 /**
@@ -70,7 +66,6 @@ public class HttpConnection implements Connection {
     public static final String FORM_URL_ENCODED = "application/x-www-form-urlencoded";
     private static final int HTTP_TEMP_REDIR = 307; // http/1.1 temporary redirect, not in Java's set.
     private static final String DefaultUploadType = "application/octet-stream";
-    private static final Charset UTF_8 = Charset.forName("UTF-8"); // Don't use StandardCharsets, not in Android API 10.
     private static final Charset ISO_8859_1 = Charset.forName("ISO-8859-1");
 
     /**
@@ -111,57 +106,6 @@ public class HttpConnection implements Connection {
         req = new Request(copy);
     }
 
-    /**
-     * Encodes the input URL into a safe ASCII URL string
-     * @param url unescaped URL
-     * @return escaped URL
-     */
-	private static String encodeUrl(String url) {
-        try {
-            URL u = new URL(url);
-            return encodeUrl(u).toExternalForm();
-        } catch (Exception e) {
-            return url;
-        }
-	}
-
-    static URL encodeUrl(URL u) {
-	    u = punyUrl(u);
-        try {
-            // run the URL through URI, so components are encoded
-            URI uri = new URI(
-                u.getProtocol(), decodePart(u.getUserInfo()), u.getHost(), u.getPort(),
-                decodePart(u.getPath()), decodePart(u.getQuery()), decodePart(u.getRef()));
-            return uri.toURL();
-        } catch (URISyntaxException | MalformedURLException | UnsupportedEncodingException e) {
-            // give up and return the original input
-            return u;
-        }
-    }
-
-    @Nullable private static String decodePart(@Nullable String encoded) throws UnsupportedEncodingException {
-        if (encoded == null) return null;
-        return URLDecoder.decode(encoded, UTF_8.name());
-    }
-
-    /**
-     Convert an International URL to a Punycode URL.
-     @param url input URL that may include an international hostname
-     @return a punycode URL if required, or the original URL
-     */
-    private static URL punyUrl(URL url) {
-        if (!StringUtil.isAscii(url.getHost())) {
-            try {
-                String puny = IDN.toASCII(url.getHost());
-                url = new URL(url.getProtocol(), puny, url.getPort(), url.getFile()); // file will include ref, query if any
-            } catch (MalformedURLException e) {
-                // if passed a valid URL initially, cannot happen
-                throw new IllegalArgumentException(e);
-            }
-        }
-        return url;
-    }
-
     private static String encodeMimeName(String val) {
         return val.replace("\"", "%22");
     }
@@ -189,7 +133,7 @@ public class HttpConnection implements Connection {
     public Connection url(String url) {
         Validate.notEmptyParam(url, "url");
         try {
-            req.url(new URL(encodeUrl(url)));
+            req.url(new URL(url));
         } catch (MalformedURLException e) {
             throw new IllegalArgumentException(String.format("The supplied URL, '%s', is malformed. Make sure it is an absolute URL, and starts with 'http://' or 'https://'. See https://jsoup.org/cookbook/extracting-data/working-with-urls", url), e);
         }
@@ -441,7 +385,7 @@ public class HttpConnection implements Connection {
 
         public T url(URL url) {
             Validate.notNullParam(url, "url");
-            this.url = punyUrl(url); // if calling url(url) directly, does not go through encodeUrl, so we punycode it explicitly. todo - should we encode here as well?
+            this.url = new UrlBuilder(url).build();
             return (T) this;
         }
 
@@ -889,7 +833,7 @@ public class HttpConnection implements Connection {
                     if (location.startsWith("http:/") && location.charAt(6) != '/') // fix broken Location: http:/temp/AAG_New/en/index.php
                         location = location.substring(6);
                     URL redir = StringUtil.resolve(req.url(), location);
-                    req.url(encodeUrl(redir));
+                    req.url(redir);
 
                     req.executing = false;
                     return execute(req, res);
@@ -994,7 +938,7 @@ public class HttpConnection implements Connection {
             prepareByteData();
             Validate.notNull(byteData);
             // charset gets set from header on execute, and from meta-equiv on parse. parse may not have happened yet
-            String body = (charset == null ? DataUtil.UTF_8 : Charset.forName(charset))
+            String body = (charset == null ? UTF_8 : Charset.forName(charset))
                 .decode(byteData).toString();
             ((Buffer)byteData).rewind(); // cast to avoid covariant return type change in jdk9
             return body;
@@ -1228,32 +1172,13 @@ public class HttpConnection implements Connection {
 
         // for get url reqs, serialise the data map into the url
         private static void serialiseRequestUrl(Connection.Request req) throws IOException {
-            URL in = req.url();
-            StringBuilder url = StringUtil.borrowBuilder();
-            boolean first = true;
-            // reconstitute the query, ready for appends
-            url
-                .append(in.getProtocol())
-                .append("://")
-                .append(in.getAuthority()) // includes host, port
-                .append(in.getPath())
-                .append("?");
-            if (in.getQuery() != null) {
-                url.append(in.getQuery());
-                first = false;
-            }
+            UrlBuilder in = new UrlBuilder(req.url());
+
             for (Connection.KeyVal keyVal : req.data()) {
                 Validate.isFalse(keyVal.hasInputStream(), "InputStream data not supported in URL query string.");
-                if (!first)
-                    url.append('&');
-                else
-                    first = false;
-                url
-                    .append(URLEncoder.encode(keyVal.key(), DataUtil.defaultCharsetName))
-                    .append('=')
-                    .append(URLEncoder.encode(keyVal.value(), DataUtil.defaultCharsetName));
+                in.appendKeyVal(keyVal);
             }
-            req.url(new URL(StringUtil.releaseBuilder(url)));
+            req.url(in.build());
             req.data().clear(); // moved into url as get params
         }
     }
diff --git a/src/main/java/org/jsoup/helper/UrlBuilder.java b/src/main/java/org/jsoup/helper/UrlBuilder.java
new file mode 100644
index 00000000..c62d0649
--- /dev/null
+++ b/src/main/java/org/jsoup/helper/UrlBuilder.java
@@ -0,0 +1,98 @@
+package org.jsoup.helper;
+
+import org.jsoup.Connection;
+import org.jsoup.internal.StringUtil;
+
+import javax.annotation.Nullable;
+import java.io.UnsupportedEncodingException;
+import java.net.IDN;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLDecoder;
+import java.net.URLEncoder;
+
+import static org.jsoup.helper.DataUtil.UTF_8;
+
+/**
+ A utility class to normalize input URLs. jsoup internal; API subject to change.
+ <p>Normalization includes puny-coding the host, and encoding non-ascii path components. The query-string
+ is left mostly as-is, to avoid inadvertently/incorrectly decoding a desired '+' literal ('%2B') as a ' '.</p>
+ */
+final class UrlBuilder {
+    URL u;
+    @Nullable StringBuilder q;
+
+    UrlBuilder(URL inputUrl) {
+        this.u = inputUrl;
+        if (u.getQuery() != null)
+            q = StringUtil.borrowBuilder().append(u.getQuery());
+    }
+
+    URL build() {
+        try {
+            // use the URI class to encode non-ascii in path
+            URI uri = new URI(
+                u.getProtocol(),
+                u.getUserInfo(),
+                IDN.toASCII(decodePart(u.getHost())), // puny-code
+                u.getPort(),
+                decodePart(u.getPath()),
+                null, null // query and fragment appended later so as not to encode
+            );
+
+            String normUrl = uri.toASCIIString();
+            if (q != null || u.getRef() != null) {
+                StringBuilder sb = StringUtil.borrowBuilder().append(normUrl);
+                if (q != null) {
+                    sb.append('?');
+                    sb.append(normalizeQuery(StringUtil.releaseBuilder(q)));
+                }
+                if (u.getRef() != null) {
+                    sb.append('#');
+                    sb.append(normalizeRef(u.getRef()));
+                }
+                normUrl = StringUtil.releaseBuilder(sb);
+            }
+            u =  new URL(normUrl);
+            return u;
+        } catch (MalformedURLException | URISyntaxException e) {
+            // we assert here so that any incomplete normalization issues can be caught in devel. but in practise,
+            // the remote end will be able to handle it, so in prod we just pass the original URL
+            assert Validate.assertFail(e.toString());
+            return u;
+        }
+    }
+
+    void appendKeyVal(Connection.KeyVal kv) throws UnsupportedEncodingException {
+        if (q == null)
+            q = StringUtil.borrowBuilder();
+        else
+            q.append('&');
+        q
+            .append(URLEncoder.encode(kv.key(), UTF_8.name()))
+            .append('=')
+            .append(URLEncoder.encode(kv.value(), UTF_8.name()));
+    }
+
+    private static String decodePart(String encoded) {
+        try {
+            return URLDecoder.decode(encoded, UTF_8.name());
+        } catch (UnsupportedEncodingException e) {
+            throw new RuntimeException(e); // wtf!
+        }
+    }
+
+    private static String normalizeQuery(String q) {
+        // minimal space normal; other characters left as supplied - if generated from jsoup data, will be encoded
+        return q.replace(' ', '+');
+    }
+
+    private static String normalizeRef(String r) {
+        // minimal space normal; other characters left as supplied
+        return r.replace(" ", "%20");
+    }
+
+
+}
diff --git a/src/main/java/org/jsoup/helper/Validate.java b/src/main/java/org/jsoup/helper/Validate.java
index 5a2409a7..1270b8cc 100644
--- a/src/main/java/org/jsoup/helper/Validate.java
+++ b/src/main/java/org/jsoup/helper/Validate.java
@@ -183,6 +183,17 @@ public final class Validate {
         throw new ValidationException(msg);
     }
 
+    /**
+     Cause a failure, but return false so it can be used in an assert statement.
+     @param msg message to output.
+     @return false, always
+     @throws IllegalStateException if we reach this state
+     */
+    static boolean assertFail(String msg) {
+        fail(msg);
+        return false;
+    }
+
     /**
      Cause a failure.
      @param msg message to output.
diff --git a/src/main/java/org/jsoup/nodes/Element.java b/src/main/java/org/jsoup/nodes/Element.java
index 5142fa2b..8b276378 100644
--- a/src/main/java/org/jsoup/nodes/Element.java
+++ b/src/main/java/org/jsoup/nodes/Element.java
@@ -1815,6 +1815,7 @@ public class Element extends Node {
     /**
      @deprecated Use {@link #forEach(Consumer)} instead.
      */
+    @Deprecated
     public Element forEach(org.jsoup.helper.Consumer<? super Element> action) {
         Validate.notNull(action);
         NodeTraversor.traverse((node, depth) -> {
diff --git a/src/main/java/org/jsoup/nodes/Node.java b/src/main/java/org/jsoup/nodes/Node.java
index 851bd8a5..46bd1cc6 100644
--- a/src/main/java/org/jsoup/nodes/Node.java
+++ b/src/main/java/org/jsoup/nodes/Node.java
@@ -687,6 +687,7 @@ public abstract class Node implements Cloneable {
     /**
      @deprecated Use {@link #forEachNode(Consumer)} instead.
      */
+    @Deprecated
     public Node forEachNode(org.jsoup.helper.Consumer<? super Node> action) {
         Validate.notNull(action);
         NodeTraversor.traverse((node, depth) -> action.accept(node), this);


Test patch:
diff --git a/src/test/java/org/jsoup/helper/HttpConnectionTest.java b/src/test/java/org/jsoup/helper/HttpConnectionTest.java
index b655f8ff..840d8a82 100644
--- a/src/test/java/org/jsoup/helper/HttpConnectionTest.java
+++ b/src/test/java/org/jsoup/helper/HttpConnectionTest.java
@@ -256,22 +256,22 @@ public class HttpConnectionTest {
 
     @Test public void encodeUrl() throws MalformedURLException {
         URL url1 = new URL("https://test.com/foo bar/[One]?q=white space#frag");
-        URL url2 = HttpConnection.encodeUrl(url1);
-        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white%20space#frag", url2.toExternalForm());
+        URL url2 = new UrlBuilder(url1).build();
+        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white+space#frag", url2.toExternalForm());
     }
 
     @Test void encodedUrlDoesntDoubleEncode() throws MalformedURLException {
         URL url1 = new URL("https://test.com/foo bar/[One]?q=white space#frag ment");
-        URL url2 = HttpConnection.encodeUrl(url1);
-        URL url3 = HttpConnection.encodeUrl(url2);
-        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white%20space#frag%20ment", url2.toExternalForm());
-        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white%20space#frag%20ment", url3.toExternalForm());
+        URL url2 = new UrlBuilder(url1).build();
+        URL url3 = new UrlBuilder(url2).build();
+        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white+space#frag%20ment", url2.toExternalForm());
+        assertEquals("https://test.com/foo%20bar/%5BOne%5D?q=white+space#frag%20ment", url3.toExternalForm());
     }
 
     @Test void connectToEncodedUrl() {
         Connection connect = Jsoup.connect("https://example.com/a%20b%20c?query+string");
         URL url = connect.request().url();
-        assertEquals("https://example.com/a%20b%20c?query%20string", url.toExternalForm());
+        assertEquals("https://example.com/a%20b%20c?query+string", url.toExternalForm());
     }
 
     @Test public void noUrlThrowsValidationError() throws IOException {
@@ -303,6 +303,18 @@ public class HttpConnectionTest {
         assertEquals(puny, req.url().toExternalForm());
     }
 
+    @Test void supportsIdnWithPort() throws MalformedURLException {
+        String idn = "https://www.测试.测试:9001/foo.html?bar";
+        String puny = "https://www.xn--0zwm56d.xn--0zwm56d:9001/foo.html?bar";
+
+        Connection con = Jsoup.connect(idn);
+        assertEquals(puny, con.request().url().toExternalForm());
+
+        HttpConnection.Request req = new HttpConnection.Request();
+        req.url(new URL(idn));
+        assertEquals(puny, req.url().toExternalForm());
+    }
+
     @Test public void validationErrorsOnExecute() throws IOException {
         Connection con = new HttpConnection();
         boolean urlThrew = false;
diff --git a/src/test/java/org/jsoup/integration/ConnectTest.java b/src/test/java/org/jsoup/integration/ConnectTest.java
index 4881e128..e453cf9a 100644
--- a/src/test/java/org/jsoup/integration/ConnectTest.java
+++ b/src/test/java/org/jsoup/integration/ConnectTest.java
@@ -749,4 +749,12 @@ public class ConnectTest {
         Document doc = Jsoup.connect(url).get();
         assertEquals(url, doc.location());
     }
+
+    @Test void fetchUnicodeUrl() throws IOException {
+        String url = EchoServlet.Url + "/✔/?%E9%8D%B5=%E5%80%A4"; // encoded 鍵=値
+        Document doc = Jsoup.connect(url).get();
+
+        assertEquals("/✔/", ihVal("Path Info", doc));
+        assertEquals("%E9%8D%B5=%E5%80%A4", ihVal("Query String", doc));
+    }
 }
diff --git a/src/test/java/org/jsoup/integration/servlets/EchoServlet.java b/src/test/java/org/jsoup/integration/servlets/EchoServlet.java
index 9931f978..f5bf2ce3 100644
--- a/src/test/java/org/jsoup/integration/servlets/EchoServlet.java
+++ b/src/test/java/org/jsoup/integration/servlets/EchoServlet.java
@@ -67,6 +67,7 @@ public class EchoServlet extends BaseServlet {
         // some get items
         write(w, "Method", req.getMethod());
         write(w, "Request URI", req.getRequestURI());
+        write(w, "Path Info", req.getPathInfo());
         write(w, "Query String", req.getQueryString());
 
         // request headers (why is it an enumeration?)