Repository: crawler-commons/crawler-commons
Pull request #407 (2023-05-08 20:57:38)

Bump maven-surefire-plugin from 3.0.0 to 3.1.0
Bumps [maven-surefire-plugin](https://github.com/apache/maven-surefire) from 3.0.0 to 3.1.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/apache/maven-surefire/releases">maven-surefire-plugin's releases</a>.</em></p>
<blockquote>
<h2>3.1.0</h2>
<h2><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&amp;version=12351273">Release Notes - Maven Surefire - Version 3.1.0
</a>
Sub-task</h2>
<ul>
<li>[<a href="https://issues.apache.org/jira/browse/SUREFIRE-2162">SUREFIRE-2162</a>] - Document upcoming mojo and file names change</li>
</ul>
<h2>Bug</h2>
<ul>
<li>[<a href="https://issues.apache.org/jira/browse/SUREFIRE-2140">SUREFIRE-2140</a>] - Cannot release Surefire on Windows</li>
</ul>
<h2>Improvement</h2>
<ul>
<li>[<a href="https://issues.apache.org/jira/browse/SUREFIRE-2153">SUREFIRE-2153</a>] - Replace SurefireReportGenerator with a new SurefireReportRenderer</li>
<li>[<a href="https://issues.apache.org/jira/browse/SUREFIRE-2160">SUREFIRE-2160</a>] - Replace LocalizedProperties with (Custom)I18N approach from MPIR</li>
</ul>
<h2>Task</h2>
<ul>
<li>[<a href="https://issues.apache.org/jira/browse/SUREFIRE-2130">SUREFIRE-2130</a>] - Rewrite several test classes in report plugin for upcoming Doxia 2.0.0 stack</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/apache/maven-surefire/commit/36f2eff46bfe5146da179100143ad7e854721e69"><code>36f2eff</code></a> [maven-release-plugin] prepare release surefire-3.1.0</li>
<li><a href="https://github.com/apache/maven-surefire/commit/f48db530e096dd786567d8e43c3d8bd04cba7ce1"><code>f48db53</code></a> Mark class files in resources as binary</li>
<li><a href="https://github.com/apache/maven-surefire/commit/eed0065d7a2cdd1424830792f3f5dfda86fb0963"><code>eed0065</code></a> Fix invalid .gitattributes</li>
<li><a href="https://github.com/apache/maven-surefire/commit/3966e2f73b4bf0fdd65936fab0e45717958c54d9"><code>3966e2f</code></a> [SUREFIRE-2162] Document upcoming mojo and file names change</li>
<li><a href="https://github.com/apache/maven-surefire/commit/c7af2768ba226d4fd9247d150f8bbb34bf64107d"><code>c7af276</code></a> [SUREFIRE-2140] Cannot release Surefire on Windows</li>
<li><a href="https://github.com/apache/maven-surefire/commit/6718f14fd1031def148834c6bef71b61759021af"><code>6718f14</code></a> [SUREFIRE-2160] Replace LocalizedProperties with (Custom)I18N approach from MPIR</li>
<li><a href="https://github.com/apache/maven-surefire/commit/33d30c6e018c92447006765c90cb43153ef5385c"><code>33d30c6</code></a> [SUREFIRE-2153] Replace SurefireReportGenerator with a new SurefireReportRend...</li>
<li><a href="https://github.com/apache/maven-surefire/commit/da48b3a3a0832f5e3ec90f7a5010b88e22c6d77f"><code>da48b3a</code></a> configure notifications</li>
<li><a href="https://github.com/apache/maven-surefire/commit/fdbbb5c1062e3fee8cbe3373bf4aa6b51d9f2821"><code>fdbbb5c</code></a> (doc) ignore reformat</li>
<li><a href="https://github.com/apache/maven-surefire/commit/15c4014a04aa003909538d2777dced083b4323c3"><code>15c4014</code></a> [maven-release-plugin] prepare for next development iteration</li>
<li>See full diff in <a href="https://github.com/apache/maven-surefire/compare/surefire-3.0.0...surefire-3.1.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.apache.maven.plugins:maven-surefire-plugin&package-manager=maven&previous-version=3.0.0&new-version=3.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>


Patch:
diff --git a/pom.xml b/pom.xml
index 752a325..1c6f8a7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -381,11 +381,11 @@
 		<maven-compiler-plugin.version>3.11.0</maven-compiler-plugin.version>
 		<maven-resources-plugin.version>2.5</maven-resources-plugin.version>
 		<maven-jar-plugin.version>3.3.0</maven-jar-plugin.version>
-		<maven-surfire-plugin.version>3.0.0</maven-surfire-plugin.version>
+		<maven-surfire-plugin.version>3.1.0</maven-surfire-plugin.version>
 		<maven-release-plugin.version>3.0.0</maven-release-plugin.version>
 		<maven-source-plugin.version>3.2.1</maven-source-plugin.version>
 		<maven-javadoc-plugin.version>3.5.0</maven-javadoc-plugin.version>
-		<maven-gpg-plugin.version>3.0.1</maven-gpg-plugin.version>
+		<maven-gpg-plugin.version>3.1.0</maven-gpg-plugin.version>
 		<apache-rat-plugin.version>0.8</apache-rat-plugin.version>
 		<maven-assembly-plugin.version>2.2.2</maven-assembly-plugin.version>
 		<maven-deploy-plugin.version>3.1.1</maven-deploy-plugin.version>
diff --git a/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java b/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java
index c22461c..2f6eb80 100644
--- a/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java
+++ b/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java
@@ -44,7 +44,9 @@ import crawlercommons.robots.SimpleRobotRules.RobotRulesMode;
  * <p>
  * This implementation of {@link BaseRobotsParser} retrieves a set of
  * {@link SimpleRobotRules rules} for an agent with the given name from the
- * <code>robots.txt</code> file of a given domain.
+ * <code>robots.txt</code> file of a given domain. The implementation follows
+ * <a href="https://www.rfc-editor.org/rfc/rfc9309.html#name-access-method">RFC
+ * 9309</a>.
  * </p>
  * 
  * <p>
@@ -492,7 +494,17 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
 
         int bytesLen = content.length;
         int offset = 0;
-        Charset encoding = StandardCharsets.US_ASCII;
+
+        /*
+         * RFC 9309 requires that is "UTF-8 encoded" (<a href=
+         * "https://www.rfc-editor.org/rfc/rfc9309.html#name-access-method"> RFC
+         * 9309, section 2.3 Access Method</a>), but
+         * "Implementors MAY bridge encoding mismatches if they detect that the robots.txt file is not UTF-8 encoded."
+         * (<a href=
+         * "https://www.rfc-editor.org/rfc/rfc9309.html#name-the-allow-and-disallow-line"
+         * > RFC 9309, section 2.2.2. The "Allow" and "Disallow" Lines</a>)
+         */
+        Charset encoding = StandardCharsets.UTF_8;
 
         // Check for a UTF-8 BOM at the beginning (EF BB BF)
         if ((bytesLen >= 3) && (content[0] == (byte) 0xEF) && (content[1] == (byte) 0xBB) && (content[2] == (byte) 0xBF)) {
@@ -519,11 +531,11 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         // Decide if we need to do special HTML processing.
         boolean isHtmlType = ((contentType != null) && contentType.toLowerCase(Locale.ROOT).startsWith("text/html"));
 
-        // If it looks like it contains HTML, but doesn't have a user agent
-        // field, then
-        // assume somebody messed up and returned back to us a random HTML page
-        // instead
-        // of a robots.txt file.
+        /*
+         * If it looks like it contains HTML, but doesn't have a user agent
+         * field, then assume somebody messed up and returned back to us a
+         * random HTML page instead of a robots.txt file.
+         */
         boolean hasHTML = false;
         if (isHtmlType || SIMPLE_HTML_PATTERN.matcher(contentAsStr).find()) {
             if (!USER_AGENT_PATTERN.matcher(contentAsStr).find()) {
@@ -550,12 +562,12 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         while (lineParser.hasMoreTokens()) {
             String line = lineParser.nextToken();
 
-            // Get rid of HTML markup, in case some brain-dead webmaster has
-            // created an HTML
-            // page for robots.txt. We could do more sophisticated processing
-            // here to better
-            // handle bad HTML, but that's a very tiny percentage of all
-            // robots.txt files.
+            /*
+             * Get rid of HTML markup, in case some brain-dead webmaster has
+             * created an HTML page for robots.txt. We could do more
+             * sophisticated processing here to better handle bad HTML, but
+             * that's a very tiny percentage of all robots.txt files.
+             */
             if (hasHTML) {
                 line = line.replaceAll("<[^>]+>", "");
             }
@@ -855,9 +867,8 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
                     double delayValue = Double.parseDouble(delayString) * 1000.0;
                     state.setCrawlDelay(Math.round(delayValue));
                 } else {
-                    long delayValue = Integer.parseInt(delayString) * 1000L; // sec
-                                                                             // to
-                                                                             // millisec
+                    // seconds to milliseconds
+                    long delayValue = Integer.parseInt(delayString) * 1000L;
                     state.setCrawlDelay(delayValue);
                 }
             } catch (Exception e) {


Test patch:
diff --git a/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java b/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java
index 597498f..3fb6dc4 100644
--- a/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java
+++ b/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java
@@ -22,6 +22,7 @@ import org.junit.jupiter.params.provider.CsvSource;
 
 import java.io.InputStream;
 import java.net.HttpURLConnection;
+import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.List;
@@ -226,6 +227,44 @@ public class SimpleRobotRulesParserTest {
         assertTrue(rules.isAllowed("http://www.domain.com/anypage.html"));
     }
 
+    @Test
+    void testUnicodeUnescapedPaths() {
+        final String simpleRobotsTxt = "User-agent: *" + CRLF //
+                        + "Disallow: /bücher/" + CRLF //
+                        + "Disallow: /k%C3%B6nyvek/" + CRLF //
+                        + CRLF //
+                        + "User-agent: GoodBot" + CRLF //
+                        + "Allow: /";
+
+        BaseRobotRules rules = createRobotRules("mybot", simpleRobotsTxt);
+        assertTrue(rules.isAllowed("https://www.example.com/"));
+
+        // test using escaped and unescaped URLs
+        assertFalse(rules.isAllowed("https://www.example.com/b%C3%BCcher/book1.html"));
+        assertFalse(rules.isAllowed("https://www.example.com/bücher/book2.html"));
+
+        // (for completeness) check also escaped path in robots.txt
+        assertFalse(rules.isAllowed("https://www.example.com/k%C3%B6nyvek/book1.html"));
+        assertFalse(rules.isAllowed("https://www.example.com/könyvek/book2.html"));
+
+        // test invalid encoding: invalid encoded characters should not break
+        // parsing of rules below
+        rules = createRobotRules("goodbot", simpleRobotsTxt.getBytes(StandardCharsets.ISO_8859_1));
+        assertTrue(rules.isAllowed("https://www.example.com/"));
+        assertTrue(rules.isAllowed("https://www.example.com/b%C3%BCcher/book1.html"));
+
+        // test invalid encoding: only rules with invalid characters should be
+        // ignored
+        rules = createRobotRules("mybot", simpleRobotsTxt.getBytes(StandardCharsets.ISO_8859_1));
+        assertTrue(rules.isAllowed("https://www.example.com/"));
+        assertFalse(rules.isAllowed("https://www.example.com/k%C3%B6nyvek/book1.html"));
+        assertFalse(rules.isAllowed("https://www.example.com/könyvek/book2.html"));
+        // if URL paths in disallow rules are not properly encoded, these two
+        // URLs are not matched:
+        // assertFalse(rules.isAllowed("https://www.example.com/b%C3%BCcher/book2.html"));
+        // assertFalse(rules.isAllowed("https://www.example.com/bücher/book1.html"));
+    }
+
     @Test
     void testSimplestAllowAll() {
         final String simpleRobotsTxt = "User-agent: *" + CRLF //