Repository: crawler-commons/crawler-commons
Pull request #362 (2022-02-20 21:00:38)

[Robots.txt] RFC compliance: matching user-agent names when selecting rule blocks
- add unit test to verify that the rule with the completely matched user-agent name is selected, and no partial prefix match is preferred (cf. also #192)

sebastian-nagel: Thanks, @rzo1! But I should add also add a fix. The (failing) unit tests are just an example that the prefix matching is even somewhat counter-intuitive in addition that it contradicts both the [new](https://datatracker.ietf.org/doc/draft-koster-rep/) and the [old RFC](https://www.robotstxt.org/norobots-rfc.txt).
sebastian-nagel: Updated the PR...

To summarize the issues around matching user agent names:

1. the new [RFC 9309](https://datatracker.ietf.org/doc/rfc9309/) is pretty clear: both the crawler (robot, user-agent) name and the name in the user-agent line are required to be a single word token (regex `[a-zA-Z_-]+`) and matching is done literally but case-insensitive on the entire word token.

2. the [first RFC proposal](http://www.robotstxt.org/norobots-rfc.txt) only requires that the user-agent token is a substring of the User-Agent line in the robots.txt:

   The robot must obey the first record in /robots.txt that contains a User-Agent line whose value contains the name token of the robot as a substring.
   
3. our current implementation is different from 1 and 2:

   a. the user-agent line is split at `[ \t,]' into tokens
   b. the configured user-agent name is also split into tokens at white space (U+0020)
   c. a user-agent line matches if one of the line tokens (from a) is a prefix of one of the name tokens (from b)
   d. the first matched non-wildcard block of allow/disallow rules is selected

   In short, the current implementation allows that crawler developers lazily use the HTTP User-Agent string also for the robots.txt parser. It does not cover the case when the HTTP User-Agent string is used in the robots.txt.

4. the current robots.txt parser API allows to pass
   a. multiple agent names in a comma-separated string
   b. and/or compound agent names such as "Mozilla Crawlerbot-super 1.0"
      where every white-space separated word is matched

Issues with the current implementation are conflicts when robot names share a common prefix, eg. `googlebot` and `googlebot-news`. But the real trouble (that's #192 is about) starts when both user-agent line and name consist of multiple tokens and include common parts such as "Mozilla".


Decisions to resolve this issues:

1. extend the robots parser API and add a method to pass agent names as a collection following the RFC 9309 with no splitting of the names into words/tokens.

   This still allows for multiple robot names, including varying names which include version string or similar ("webcrawler/3.0").

2. by default user agent names are matched literally but case-insensitive following RFC 9309

3. the old behavior can be restored by calling "setExactUserAgentMatching(false)"

4. the method which splits the robot name into tokens and performs prefix matching is deprecated but kept for backward-compatibility.

5. BaseRobotRulesParser: move the documented details about how user-agent names are matched into SimpleRobotRulesParser. The abstract base class shouldn't already define the behavior without a way to actually enforce it.

sebastian-nagel: Following the discussion in google/robotstxt#56, the behavior should be changed, so that a robot name "foo" also matches all of:
```
User-agent: Foo/1.2
User-agent: Foo Bar
```
Eventually, make the user-agent matching configurable and implement it in a method which can be overridden if a user is stick to an agent name containing any characters except `[a-zA-Z_-]` - "Mail.Ru", "Go!Zilla", "360Spider", etc.
sebastian-nagel: A "prefix" match up to the first non-product-token character is now implemented. I.e. the product token "foo" is matched in `User-agent: Foo/1.2`. To handle partial matches of special product tokens ("Go!Zilla", etc.) the method "userAgentProductTokenPartialMatch" can be overridden.
jnioche: LGTM @sebastian-nagel, nice one! Feel free to merge

Patch:
diff --git a/src/main/java/crawlercommons/robots/BaseRobotsParser.java b/src/main/java/crawlercommons/robots/BaseRobotsParser.java
index 1e26691..696f39b 100644
--- a/src/main/java/crawlercommons/robots/BaseRobotsParser.java
+++ b/src/main/java/crawlercommons/robots/BaseRobotsParser.java
@@ -17,6 +17,7 @@
 package crawlercommons.robots;
 
 import java.io.Serializable;
+import java.util.Collection;
 
 @SuppressWarnings("serial")
 public abstract class BaseRobotsParser implements Serializable {
@@ -37,6 +38,13 @@ public abstract class BaseRobotsParser implements Serializable {
      * "crawlerbot" as the agent name, because of splitting on spaces,
      * lower-casing, and the prefix match rule.
      * 
+     * @deprecated since 1.4 - replaced by {@link parseContent(String, byte[],
+     *             String, Collection)}. Passing a collection of robot names
+     *             gives users more control how user-agent and robot names are
+     *             matched. Passing a list of names is also more efficient as it
+     *             does not require to split the robot name string again and
+     *             again on every robots.txt file to be parsed.
+     * 
      * @param url
      *            URL that robots.txt content was fetched from. A complete and
      *            valid URL (e.g., https://example.com/robots.txt) is expected.
@@ -51,16 +59,40 @@ public abstract class BaseRobotsParser implements Serializable {
      *            (just the name portion, w/o version or other details)
      * @return robot rules.
      */
-
+    @Deprecated
     public abstract BaseRobotRules parseContent(String url, byte[] content, String contentType, String robotNames);
 
     /**
-     * The fetch of robots.txt failed, so return rules appropriate give the HTTP
-     * status code.
+     * Parse the robots.txt file in <i>content</i>, and return rules appropriate
+     * for processing paths by <i>userAgent</i>. Multiple agent names can be
+     * provided as collection. How agent names are matched against user-agent
+     * lines in the robots.txt depends on the implementing class.
+     * 
+     * @param url
+     *            URL that robots.txt content was fetched from. A complete and
+     *            valid URL (e.g., https://example.com/robots.txt) is expected.
+     *            Used to resolve relative sitemap URLs and for
+     *            logging/reporting purposes.
+     * @param content
+     *            raw bytes from the site's robots.txt file
+     * @param contentType
+     *            HTTP response header (mime-type)
+     * @param robotNames
+     *            name(s) of crawler, used to select rules from the robots.txt
+     *            file by matching the names against the user-agent lines in the
+     *            robots.txt file.
+     * @return robot rules.
+     */
+    public abstract BaseRobotRules parseContent(String url, byte[] content, String contentType, Collection<String> robotNames);
+
+    /**
+     * The fetch of robots.txt failed, so return rules appropriate for the given
+     * HTTP status code.
      * 
      * @param httpStatusCode
      *            a failure status code (NOT 2xx)
      * @return robot rules
      */
     public abstract BaseRobotRules failedFetch(int httpStatusCode);
+
 }
diff --git a/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java b/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java
index 207550d..c22461c 100644
--- a/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java
+++ b/src/main/java/crawlercommons/robots/SimpleRobotRulesParser.java
@@ -24,7 +24,10 @@ import java.net.URLConnection;
 import java.net.URLDecoder;
 import java.nio.charset.Charset;
 import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collection;
 import java.util.HashMap;
+import java.util.LinkedHashSet;
 import java.util.Locale;
 import java.util.Map;
 import java.util.StringTokenizer;
@@ -174,18 +177,18 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         private int _numWarnings;
 
         private String _url;
-        private String _targetName;
+        private Collection<String> _targetNames;
 
         private SimpleRobotRules _curRules;
 
-        public ParseState(String url, String targetName) {
+        public ParseState(String url, Collection<String> targetNames) {
             _url = url;
-            _targetName = targetName;
+            _targetNames = targetNames;
             _curRules = new SimpleRobotRules();
         }
 
-        public String getTargetName() {
-            return _targetName;
+        public Collection<String> getTargetNames() {
+            return _targetNames;
         }
 
         public boolean isMatchedRealName() {
@@ -293,6 +296,9 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
     // match the rest of the directive, up until whitespace or colon
     private static final Pattern DIRECTIVE_SUFFIX_PATTERN = Pattern.compile("[^: \t]+(.*)");
 
+    // split pattern for robot names
+    private static final Pattern ROBOT_NAMES_SPLIT = Pattern.compile("\\s*,\\s*|\\s+");
+
     /**
      * Figure out directive on line of text from robots.txt file. We assume the
      * line has been lower-cased
@@ -339,6 +345,13 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
     private static final Pattern SIMPLE_HTML_PATTERN = Pattern.compile("(?is)<(html|head|body)\\s*>");
     private static final Pattern USER_AGENT_PATTERN = Pattern.compile("(?i)user-agent:");
 
+    /**
+     * Pattern to match a valid user-agent product tokens as defined in
+     * <a href="https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1">RFC
+     * 9309, section 2.2.1</a>
+     */
+    protected static final Pattern USER_AGENT_PRODUCT_TOKEN_MATCHER = Pattern.compile("[a-zA-Z_-]+");
+
     /**
      * Default max number of warnings logged during parse of any one robots.txt
      * file, see {@link #setMaxWarnings(int)}
@@ -356,6 +369,7 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
 
     private int _maxWarnings;
     private long _maxCrawlDelay;
+    private boolean _exactUserAgentMatching;
 
     public SimpleRobotRulesParser() {
         this(DEFAULT_MAX_CRAWL_DELAY, DEFAULT_MAX_WARNINGS);
@@ -370,6 +384,20 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
     public SimpleRobotRulesParser(long maxCrawlDelay, int maxWarnings) {
         this._maxCrawlDelay = maxCrawlDelay;
         this._maxWarnings = maxWarnings;
+        this._exactUserAgentMatching = true;
+    }
+
+    /**
+     * Validate a user-agent product token as defined in
+     * <a href="https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1">RFC
+     * 9309, section 2.2.1</a>
+     * 
+     * @param userAgent
+     *            user-agent token to verify
+     * @return true if the product token is valid
+     */
+    protected static boolean isValidUserAgentToObey(String userAgent) {
+        return userAgent != null && USER_AGENT_PRODUCT_TOKEN_MATCHER.matcher(userAgent).matches();
     }
 
     @Override
@@ -404,8 +432,58 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
      * crawlercommons.robots.BaseRobotsParser#parseContent(java.lang.String,
      * byte[], java.lang.String, java.lang.String)
      */
+    @Deprecated
     @Override
     public SimpleRobotRules parseContent(String url, byte[] content, String contentType, String robotNames) {
+        return parseContent(url, content, contentType, new LinkedHashSet<>(Arrays.asList(splitRobotNames(robotNames))), false);
+    }
+
+    /**
+     * Split a string listing user-agent / robot names into tokens.
+     * 
+     * Splitting is done at comma and/or whitespace, the tokens are converted to
+     * lower-case.
+     * 
+     * @param robotNames
+     *            robot / user-agent string
+     * @return array of user-agent / robot tokens
+     */
+    protected String[] splitRobotNames(String robotNames) {
+        // lower case and trim
+        robotNames = robotNames.toLowerCase(Locale.ROOT).trim();
+        // split at commas and whitespace
+        return ROBOT_NAMES_SPLIT.split(robotNames);
+    }
+
+    /**
+     * Parse the robots.txt file in <i>content</i>, and return rules appropriate
+     * for processing paths by <i>userAgent</i>. Multiple agent names are
+     * provided as collection. See {@link #setExactUserAgentMatching(boolean)}
+     * for details how agent names are matched.
+     * 
+     * @param url
+     *            URL that robots.txt content was fetched from. A complete and
+     *            valid URL (e.g., https://example.com/robots.txt) is expected.
+     *            Used to resolve relative sitemap URLs and for
+     *            logging/reporting purposes.
+     * @param content
+     *            raw bytes from the site's robots.txt file
+     * @param contentType
+     *            HTTP response header (mime-type)
+     * @param robotNames
+     *            name(s) of crawler, used to select rules from the robots.txt
+     *            file by matching the names against the user-agent lines in the
+     *            robots.txt file. Robot names should be single token names, w/o
+     *            version or other parts. Names should be lower-case, as the
+     *            user-agent line is also converted to lower-case for matching.
+     * @return robot rules.
+     */
+    @Override
+    public SimpleRobotRules parseContent(String url, byte[] content, String contentType, Collection<String> robotNames) {
+        return parseContent(url, content, contentType, robotNames, isExactUserAgentMatching());
+    }
+
+    private SimpleRobotRules parseContent(String url, byte[] content, String contentType, Collection<String> robotNames, boolean exactUserAgentMatching) {
 
         // If there's nothing there, treat it like we have no restrictions.
         if ((content == null) || (content.length == 0)) {
@@ -467,7 +545,7 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         // tokenizer doesn't return empty tokens, a \r\n sequence still
         // works since it looks like an empty string between the \r and \n.
         StringTokenizer lineParser = new StringTokenizer(contentAsStr, "\n\r\u0085\u2028\u2029");
-        ParseState parseState = new ParseState(url, robotNames.toLowerCase(Locale.ROOT));
+        ParseState parseState = new ParseState(url, robotNames);
 
         while (lineParser.hasMoreTokens()) {
             String line = lineParser.nextToken();
@@ -548,8 +626,7 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         SimpleRobotRules result = parseState.getRobotRules();
         if (result.getCrawlDelay() > _maxCrawlDelay) {
             // Some evil sites use a value like 3600 (seconds) for the crawl
-            // delay, which would
-            // cause lots of problems for us.
+            // delay, which would cause lots of problems for us.
             LOGGER.debug("Crawl delay exceeds max value - so disallowing all URLs: {}", url);
             return new SimpleRobotRules(RobotRulesMode.ALLOW_NONE);
         } else {
@@ -576,6 +653,32 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         }
     }
 
+    /*
+     * Check whether user-agent line starts with a valid user-agent product
+     * token, but continues with additional characters to be ignored e.g. "foo"
+     * matches "User-agent: foo/1.2" or "butterfly" matches
+     * "User-agent: Butterfly/1.0"
+     * 
+     * See https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1.5
+     * "Crawlers MUST try to parse each line of the robots.txt file. Crawlers MUST use the parseable rules."
+     * and https://github.com/google/robotstxt/issues/56
+     * 
+     * This method may be overridden to implement non-standard user-agent matching.
+     * 
+     * @param agentName user-agent, found in the <a href=
+     * "https://www.rfc-editor.org/rfc/rfc9309.html#name-the-user-agent-line">
+     * robots.txt user-agent line</a>
+     * 
+     * @param targetTokens collection of user-agent product tokens we're looking
+     * for. Product tokens are expected to be lowercase.
+     * 
+     * @return true if the product token is match as token prefix
+     */
+    protected boolean userAgentProductTokenPartialMatch(String agentName, Collection<String> targetTokens) {
+        Matcher m = USER_AGENT_PRODUCT_TOKEN_MATCHER.matcher(agentName);
+        return m.lookingAt() && targetTokens.contains(m.group());
+    }
+
     /**
      * Handle the user-agent: directive
      * 
@@ -595,42 +698,69 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         // processing until we are finished.
         state.setFinishedAgentFields(false);
 
-        // Handle the case when there are multiple target names are passed
-        // We assume we should do case-insensitive comparison of target name.
-        String[] targetNames = state.getTargetName().toLowerCase(Locale.ROOT).split(",");
-
-        for (int count = 0; count < targetNames.length; count++) {
-            // Extract possible match names from our target agent name, since it
-            // appears to be expected that "Mozilla botname 1.0" matches
-            // "botname"
-            String[] targetNameSplits = targetNames[count].trim().split(" ");
-
-            // TODO KKr - catch case of multiple names, log as non-standard.
-            String[] agentNames = token.getData().split("[ \t,]");
-            for (String agentName : agentNames) {
-                agentName = agentName.trim().toLowerCase(Locale.ROOT);
-                if (agentName.isEmpty()) {
-                    // Ignore empty names
-                } else if (agentName.equals("*") && !state.isMatchedRealName()) {
-                    state.setMatchedWildcard(true);
-                    state.setAddingRules(true);
-                } else {
-                    for (String targetName : targetNameSplits) {
+        // Handle the case when multiple target names are passed.
+        // We assume we should do case-insensitive comparison of every target name.
+        Collection<String> targetNames = state.getTargetNames();
+
+        if (isExactUserAgentMatching()) {
+            String agentName = token.getData().trim().toLowerCase(Locale.ROOT);
+            if (agentName.isEmpty()) {
+                // Ignore empty names
+            } else if (agentName.equals("*") && !state.isMatchedRealName()) {
+                state.setMatchedWildcard(true);
+                state.setAddingRules(true);
+            } else if (targetNames.contains(agentName) || (!isValidUserAgentToObey(agentName) && userAgentProductTokenPartialMatch(agentName, targetNames))) {
+                if (state.isMatchedWildcard()) {
+                    // Clear rules of the wildcard user-agent found
+                    // before the non-wildcard user-agent match.
+                    state.clearRules();
+                }
+                state.setMatchedRealName(true);
+                state.setAddingRules(true);
+                state.setMatchedWildcard(false);
+            }
+        } else {
+            /*
+             * prefix matching on user-agent words - backward-compatibility with
+             * the old and deprecated API if "User-Agent" HTTP request header
+             * strings are passed as param instead of single-word/token
+             * user-agent names, e.g. if the robot name is "WebCrawler/1.0" and is
+             * expected match the robots.txt directive "User-agent: mybot"
+             * or also "User-agent: my".
+             */
+            String agentNameFull = token.getData().trim().toLowerCase(Locale.ROOT);
+            boolean matched = false;
+            if (agentNameFull.equals("*") && !state.isMatchedRealName()) {
+                state.setMatchedWildcard(true);
+                state.setAddingRules(true);
+            } else if (userAgentProductTokenPartialMatch(agentNameFull, targetNames)) {
+                // match "butterfly" in the line "User-agent: Butterfly/1.0"
+                matched = true;
+            } else {
+                String[] agentNames = ROBOT_NAMES_SPLIT.split(agentNameFull);
+                if (agentNames.length > 1) {
+                    LOGGER.debug("Multiple agent names in user-agent line: {}", token.getData());
+                }
+                for (String agentName : agentNames) {
+                    for (String targetName : targetNames) {
+                        LOGGER.debug(targetName);
                         if (targetName.startsWith(agentName)) {
-                            // In case we previously hit a wildcard rule match
-                            // but we could also have hit our own user agent before.
-                            // only clear if wildcard was matched earlier
-                            if (state.isMatchedWildcard()) {
-                                state.clearRules();
-                            }
-                            state.setMatchedRealName(true);
-                            state.setAddingRules(true);
-                            state.setMatchedWildcard(false);
+                            matched = true;
                             break;
                         }
                     }
                 }
             }
+            if (matched) {
+                if (state.isMatchedWildcard()) {
+                    // Clear rules of the wildcard user-agent found
+                    // before the non-wildcard user-agent match.
+                    state.clearRules();
+                }
+                state.setMatchedRealName(true);
+                state.setAddingRules(true);
+                state.setMatchedWildcard(false);
+            }
         }
     }
 
@@ -836,6 +966,54 @@ public class SimpleRobotRulesParser extends BaseRobotsParser {
         _maxCrawlDelay = maxCrawlDelay;
     }
 
+    /**
+     * Set how the user-agent names in the robots.txt (<code>User-agent:</code>
+     * lines) are matched with the provided robot names:
+     * <ul>
+     * <li>(with exact matching) follow the
+     * <a href= "https://datatracker.ietf.org/doc/rfc9309/">Robots Exclusion
+     * Protocol RFC 9309</a> and match user agent <b>literally but
+     * case-insensitive over the full string length</b>:
+     * 
+     * <blockquote>Crawlers set their own name, which is called a product token,
+     * to find relevant groups. The product token MUST contain only upper and
+     * lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens
+     * ("-"). [...] Crawlers MUST use case-insensitive matching to find the
+     * group that matches the product token and then obey the rules of the
+     * group.</blockquote></li>
+     * 
+     * <li>(without exact matching) split the user-agent and robot names at
+     * whitespace into words and perform a prefix match (one of the user-agent
+     * words must be a prefix of one of the robot words, eg. the robot name
+     * <code>WebCrawler/3.0</code> matches the robots.txt directive
+     * 
+     * <pre>
+     * User-agent: webcrawler
+     * </pre>
+     * 
+     * This prefix matching on words allows that crawler developers lazily use
+     * the HTTP User-Agent string also for the robots.txt parser. It does not
+     * cover the case when the HTTP User-Agent string is used in the
+     * robots.txt.</li>
+     * </ul>
+     * 
+     * @param exactMatching
+     *            if true, configure exact user-agent name matching. If false,
+     *            disable exact matching and do prefix matching on user-agent
+     *            words.
+     */
+    public void setExactUserAgentMatching(boolean exactMatching) {
+        _exactUserAgentMatching = exactMatching;
+    }
+
+    /**
+     * @return whether exact user-agent matching is configured, see
+     *         {@link #setExactUserAgentMatching(boolean)}
+     */
+    public boolean isExactUserAgentMatching() {
+        return _exactUserAgentMatching;
+    }
+
     public static void main(String[] args) throws MalformedURLException, IOException {
         if (args.length < 1) {
             System.err.println("SimpleRobotRulesParser <robots.txt> [[<agentname>] <URL>...]");


Test patch:
diff --git a/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java b/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java
index f1853b6..597498f 100644
--- a/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java
+++ b/src/test/java/crawlercommons/robots/SimpleRobotRulesParserTest.java
@@ -23,6 +23,8 @@ import org.junit.jupiter.params.provider.CsvSource;
 import java.io.InputStream;
 import java.net.HttpURLConnection;
 import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
 import java.util.Locale;
 
 import static java.nio.charset.StandardCharsets.US_ASCII;
@@ -36,14 +38,33 @@ public class SimpleRobotRulesParserTest {
     private static final String FAKE_ROBOTS_URL = "http://domain.com";
 
     private static BaseRobotRules createRobotRules(String crawlerName, String content) {
-        return createRobotRules(crawlerName, content.getBytes(UTF_8));
+        return createRobotRules(crawlerName, content.getBytes(UTF_8), true);
+    }
+
+    private static BaseRobotRules createRobotRules(String crawlerName, String content, boolean exactUserAgentMatching) {
+        return createRobotRules(crawlerName, content.getBytes(UTF_8), exactUserAgentMatching);
     }
 
     private static BaseRobotRules createRobotRules(String crawlerName, byte[] contentBytes) {
+        return createRobotRules(crawlerName, contentBytes, true);
+    }
+
+    private static BaseRobotRules createRobotRules(String crawlerName, byte[] contentBytes, boolean exactUserAgentMatching) {
         SimpleRobotRulesParser robotParser = new SimpleRobotRulesParser();
+        robotParser.setExactUserAgentMatching(exactUserAgentMatching);
         return robotParser.parseContent(FAKE_ROBOTS_URL, contentBytes, "text/plain", crawlerName);
     }
 
+    private static BaseRobotRules createRobotRules(String[] crawlerNames, String content, boolean exactUserAgentMatching) {
+        return createRobotRules(crawlerNames, content.getBytes(UTF_8), exactUserAgentMatching);
+    }
+
+    private static BaseRobotRules createRobotRules(String[] crawlerNames, byte[] contentBytes, boolean exactUserAgentMatching) {
+        SimpleRobotRulesParser robotParser = new SimpleRobotRulesParser();
+        robotParser.setExactUserAgentMatching(exactUserAgentMatching);
+        return robotParser.parseContent(FAKE_ROBOTS_URL, contentBytes, "text/plain", Arrays.asList(crawlerNames));
+    }
+
     @Test
     void testEmptyRules() {
         BaseRobotRules rules = createRobotRules("Any-darn-crawler", "");
@@ -244,7 +265,7 @@ public class SimpleRobotRulesParserTest {
                         + "Allow: /~mak" + LF //
                         + "Disallow: /" + CRLF;
 
-        BaseRobotRules rules = createRobotRules("WebCrawler/3.0", mixedEndingsRobotsTxt);
+        BaseRobotRules rules = createRobotRules("WebCrawler/3.0", mixedEndingsRobotsTxt, false);
         assertEquals(isAllowed, rules.isAllowed(urlStr));
     }
 
@@ -314,13 +335,13 @@ public class SimpleRobotRulesParserTest {
 
         BaseRobotRules rules;
 
-        rules = createRobotRules("UnhipBot/0.1", rfpExampleRobotsTxt);
+        rules = createRobotRules("UnhipBot/0.1", rfpExampleRobotsTxt, false);
         assertEquals(isAllowed, rules.isAllowed(urlStr));
 
-        rules = createRobotRules("WebCrawler/3.0", rfpExampleRobotsTxt);
+        rules = createRobotRules("WebCrawler/3.0", rfpExampleRobotsTxt, false);
         assertTrue(rules.isAllowed(urlStr));
 
-        rules = createRobotRules("Excite/1.0", rfpExampleRobotsTxt);
+        rules = createRobotRules("Excite/1.0", rfpExampleRobotsTxt, false);
         assertTrue(rules.isAllowed(urlStr));
     }
 
@@ -403,7 +424,7 @@ public class SimpleRobotRulesParserTest {
 
         // Note that SimpleRobotRulesParser now merges all matching user agent
         // rules.
-        rules = createRobotRules("Agent5,Agent2,Agent1,Agent3,*", nutchRobotsTxt);
+        rules = createRobotRules("Agent5,Agent2,Agent1,Agent3,*", nutchRobotsTxt, false);
         assertEquals(isMergeAllowed, rules.isAllowed(urlStr));
     }
 
@@ -436,7 +457,8 @@ public class SimpleRobotRulesParserTest {
                         + "#Disallow: /c" + CR //
                         + "" + CR //
                         + "" + CR //
-                        + "User-Agent: Agent2 Agent3#foo" + CR + "User-Agent: Agent4" + CR //
+                        + "User-Agent: Agent2 Agent3#foo" + CR //
+                        + "User-Agent: Agent4" + CR //
                         + "Disallow: /d" + CR //
                         + "Disallow: /e/d/" + CR //
                         + "" + CR //
@@ -445,13 +467,13 @@ public class SimpleRobotRulesParserTest {
 
         BaseRobotRules rules;
 
-        rules = createRobotRules("Agent2", nutchRobotsTxt);
+        rules = createRobotRules("Agent2", nutchRobotsTxt, false);
         assertEquals(isAllowed, rules.isAllowed(urlStr));
 
-        rules = createRobotRules("Agent3", nutchRobotsTxt);
+        rules = createRobotRules("Agent3", nutchRobotsTxt, false);
         assertEquals(isAllowed, rules.isAllowed(urlStr));
 
-        rules = createRobotRules("Agent4", nutchRobotsTxt);
+        rules = createRobotRules("Agent4", nutchRobotsTxt, false);
         assertEquals(isAllowed, rules.isAllowed(urlStr));
     }
 
@@ -634,7 +656,7 @@ public class SimpleRobotRulesParserTest {
                         + "Disallow: /index.html" + CRLF //
                         + "Allow: /";
 
-        BaseRobotRules rules = createRobotRules("crawler2", simpleRobotsTxt);
+        BaseRobotRules rules = createRobotRules("crawler2", simpleRobotsTxt, false);
         assertFalse(rules.isAllowed("http://www.domain.com/index.html"));
         assertTrue(rules.isAllowed("http://www.domain.com/anypage.html"));
     }
@@ -651,6 +673,27 @@ public class SimpleRobotRulesParserTest {
         assertTrue(rules.isAllowed("http://www.domain.com/anypage.html"));
     }
 
+    @Test
+    void testAgentTokenMatch() {
+        // The user-agent token should be matched until the first non-token
+        // character,
+        // see https://github.com/google/robotstxt/issues/56
+        final String simpleRobotsTxt1 = "User-agent: foo/1.2" + CRLF //
+                        + "Disallow: /index.html" + CRLF //
+                        + "Allow: /";
+        final String simpleRobotsTxt2 = "User-agent: foo bar" + CRLF //
+                        + "Disallow: /index.html" + CRLF //
+                        + "Allow: /";
+
+        BaseRobotRules rules = createRobotRules("foo", simpleRobotsTxt1, true);
+        assertFalse(rules.isAllowed("http://www.domain.com/index.html"));
+        assertTrue(rules.isAllowed("http://www.domain.com/anypage.html"));
+
+        rules = createRobotRules("foo", simpleRobotsTxt2, true);
+        assertFalse(rules.isAllowed("http://www.domain.com/index.html"));
+        assertTrue(rules.isAllowed("http://www.domain.com/anypage.html"));
+    }
+
     @Test
     void testUnsupportedFields() {
         // When we have a new field type that we don't know about.
@@ -756,7 +799,9 @@ public class SimpleRobotRulesParserTest {
                         + "User-agent: qihoobot" + CR //
                         + "Disallow: /";
 
-        BaseRobotRules rules = createRobotRules("googlebot/2.1", krugleRobotsTxt);
+        BaseRobotRules rules = createRobotRules("googlebot/2.1", krugleRobotsTxt, false);
+        assertTrue(rules.isAllowed("http://www.krugle.com/examples/index.html"));
+        rules = createRobotRules("googlebot", krugleRobotsTxt, true);
         assertTrue(rules.isAllowed("http://www.krugle.com/examples/index.html"));
     }
 
@@ -947,16 +992,85 @@ public class SimpleRobotRulesParserTest {
         BaseRobotRules rules = createRobotRules("One", simpleRobotsTxt);
         assertFalse(rules.isAllowed("http://www.fict.com/fish"));
 
-        rules = createRobotRules("Two", simpleRobotsTxt);
+        rules = createRobotRules("Two", simpleRobotsTxt, false);
         assertFalse(rules.isAllowed("http://www.fict.com/fish"));
 
-        rules = createRobotRules("Three", simpleRobotsTxt);
+        rules = createRobotRules("Three", simpleRobotsTxt, false);
         assertFalse(rules.isAllowed("http://www.fict.com/fish"));
 
         rules = createRobotRules("Any-darn-crawler", simpleRobotsTxt);
         assertTrue(rules.isAllowed("http://www.fict.com/fish"));
     }
 
+    @Test
+    void testMatchingOfUserAgentNamesIndependentFromSequentialOrder() {
+        String url = "https://example.com/foo/bar";
+
+        /*
+         * URL is allowed for "crawler" but disallowed for "crawlerbot"
+         * independent from sequential ordering of rules
+         */
+
+        String robotsTxtBlock1 = "User-agent: crawlerbot\n" //
+                        + "Disallow: /foo/bar\n";
+        String robotsTxtBlock2 = "User-agent: crawler\n" //
+                        + "Allow: /foo/bar\n";
+
+        String[] robotsTxts = new String[2];
+        robotsTxts[0] = robotsTxtBlock1 + "\n" + robotsTxtBlock2;
+        robotsTxts[1] = robotsTxtBlock2 + "\n" + robotsTxtBlock1;
+
+        BaseRobotRules rules;
+        for (String robotsTxt : robotsTxts) {
+            rules = createRobotRules("crawlerbot", robotsTxt, true);
+            assertFalse(rules.isAllowed(url));
+            rules = createRobotRules("crawler", robotsTxt, true);
+            assertTrue(rules.isAllowed(url));
+        }
+    }
+
+    @Test
+    void testMatchingUserAgentNamesHttpHeader() {
+        /*
+         * Tests examples from #192 - matching user-agent directives containing
+         * the strings sent by the crawler in the <i>User-Agent</i> HTTP request
+         * header.
+         */
+        String url = "http://example.org/";
+        String robotsTxt = "User-agent: Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8" + LF //
+                        + "Disallow: /";
+        BaseRobotRules rules;
+
+        // exact user-agent matching: rule does not apply
+        String robot = "butterfly";
+        rules = createRobotRules(robot, robotsTxt, true);
+        assertTrue(rules.isAllowed(url));
+
+        // prefix user-agent matching: rule does not apply anyway because
+        // "butterfly/1.0" (from robots.txt) is not a prefix of butterfly (from
+        // configuration / API call)
+        rules = createRobotRules(robot, robotsTxt, false);
+        assertTrue(rules.isAllowed(url));
+
+        // prefix user-agent matching: user-agent is matched if a token expected
+        // in the user-agent line is included literally (but lower-case)
+        String[] butterflyRobots = { "butterfly", "butterfly/1.0", "butterfly/1.0;" };
+        rules = createRobotRules(butterflyRobots, robotsTxt, false);
+        assertFalse(rules.isAllowed(url));
+
+        // exact user-agent matching: user-agent is matched if the entire
+        // user-agent line is passed as one robot name (lower-case)
+        String[] butterflyRobotsFullList = { "butterfly", "butterfly/1.0", "mozilla/5.0 (compatible; butterfly/1.0; +http://labs.topsy.com/butterfly/) gecko/2009032608 firefox/3.0.8" };
+        rules = createRobotRules(butterflyRobotsFullList, robotsTxt, true);
+        assertFalse(rules.isAllowed(url));
+
+        // exact user-agent matching: different user-agents which share only one
+        // token (here "mozilla/5.0" or "(compatible;" are not matched
+        String[] myRobots = { "mybot", "mybot/1.0", "mozilla/5.0 (compatible; mybot/1.0)" };
+        rules = createRobotRules(myRobots, robotsTxt, true);
+        assertTrue(rules.isAllowed(url));
+    }
+
     // https://github.com/crawler-commons/crawler-commons/issues/112
     @Test
     void testSitemapAtEndOfFile() {
@@ -976,6 +1090,23 @@ public class SimpleRobotRulesParserTest {
         assertEquals("https://wwwfoocom/sitemapxml", rules.getSitemaps().get(0));
     }
 
+    @Test
+    void testOverrideUserAgentMatcher() {
+        @SuppressWarnings("serial")
+        BaseRobotsParser myRobotsParser = new SimpleRobotRulesParser() {
+            @Override
+            protected boolean userAgentProductTokenPartialMatch(String userAgent, Collection<String> targetTokens) {
+                return userAgent.toLowerCase(Locale.ROOT).contains("go!zilla");
+            }
+        };
+        final String robotsTxt = "User-agent: Go!zilla/2.0" + CRLF //
+                        + "Allow: /gozilla/" + CRLF //
+                        + "Disallow: /";
+        BaseRobotRules rules = myRobotsParser.parseContent(FAKE_ROBOTS_URL, robotsTxt.getBytes(UTF_8), "text/plain", List.of("go!zilla"));
+        assertTrue(rules.isAllowed("https://example.org/gozilla/page.html"));
+        assertFalse(rules.isAllowed("https://example.org/other/page.html"));
+    }
+
     private byte[] readFile(String filename) throws Exception {
         byte[] bigBuffer = new byte[100000];
         InputStream is = SimpleRobotRulesParserTest.class.getResourceAsStream(filename);