Repository: eslint/eslint (js)

Issue:
Enhance TokenStore methods with includeComments option
Having used the new TokenStore token traversal methods this week (while working on [this PR](https://github.com/eslint/eslint/pull/7516)), I think we should add an `includeComments` option to the following: `getTokenByRangeStart()`, `getTokens()`, and `getTokensBetween()`.

The API I'm envisioning is:

```js
getTokenByRangeStart(index, {
    includeComments: true
});

getTokens(node, {
    beforeCount: 0,
    afterCount: 0,
    includeComments: true,
    filter: filterFn
});

getTokensBetween(leftToken, rightToken, {
    padding: 0,
    includeComments: true,
    filter: filterFn
});
```

The proposed change to `getTokenByRangeStart()` is not a breaking change, but the other two are (though I suppose we could do this in a backwards compatible manner still). I'd ideally like to use `getTokens` in the aforementioned PR [here](https://github.com/eslint/eslint/pull/7516/files#diff-4625bd821401815c773e90e5cecdcd73R227).

Also wanted to note that the `beforeCount`, `afterCount`, and `padding` options are all undocumented and look like they were included to avoid a breaking change.

cc @mysticatea

Pull request:
Update: Add includeComments to getTokenByRangeStart (fixes #8068)
**What is the purpose of this pull request? (put an "X" next to item)**

[ ] Documentation update
[ ] Bug fix ([template](https://raw.githubusercontent.com/eslint/eslint/master/templates/bug-report.md))
[ ] New rule ([template](https://raw.githubusercontent.com/eslint/eslint/master/templates/rule-proposal.md))
[ ] Changes an existing rule ([template](https://raw.githubusercontent.com/eslint/eslint/master/templates/rule-change-proposal.md))
[ ] Add autofixing to a rule
[ ] Add a CLI option
[x] Add something to the core
[ ] Other, please explain:

<!--
    If the item you've checked above has a template, please paste the template questions below and answer them. (If this pull request is addressing an issue, you can just paste a link to the issue here instead.)
-->

<!--
    Please ensure your pull request is ready:

    - Read the pull request guide (http://eslint.org/docs/developer-guide/contributing/pull-requests)
    - Include tests for this change
    - Update documentation for this change (if appropriate)
-->

<!--
    The following is required for all pull requests:
-->

**What changes did you make? (Give an overview)**
Added an options object to `sourceCode.getTokenByRangeStart()` with an `includeComments` property.

**Is there anything you'd like reviewers to focus on?**
Does this implementation seem correct? The other options didn't make sense in this context.



Patch:
diff --git a/docs/developer-guide/nodejs-api.md b/docs/developer-guide/nodejs-api.md
index 758ef00e7..60148ce4e 100644
--- a/docs/developer-guide/nodejs-api.md
+++ b/docs/developer-guide/nodejs-api.md
@@ -164,7 +164,7 @@ The `CLIEngine` is a constructor, and you can create a new instance by passing i
 * `configFile` - The configuration file to use (default: null). Corresponds to `-c`.
 * `cwd` - Path to a directory that should be considered as the current working directory.
 * `envs` - An array of environments to load (default: empty array). Corresponds to `--env`.
-* `extensions` - An array of filename extensions that should be checked for code. The default is an array containing just `".js"`. Corresponds to `--ext`.
+* `extensions` - An array of filename extensions that should be checked for code. The default is an array containing just `".js"`. Corresponds to `--ext`. It is only used in conjunction with directories, not with filenames or glob patterns.
 * `fix` - True indicates that fixes should be included with the output report, and that errors and warnings should not be listed if they can be fixed. However, the files on disk will not be changed. To persist changes to disk, call [`outputFixes()`](#outputfixes).
 * `globals` - An array of global variables to declare (default: empty array). Corresponds to `--global`.
 * `ignore` - False disables use of `.eslintignore`, `ignorePath` and `ignorePattern` (default: true). Corresponds to `--no-ignore`.
diff --git a/docs/developer-guide/working-with-rules.md b/docs/developer-guide/working-with-rules.md
index 81e33f932..defa52a4e 100644
--- a/docs/developer-guide/working-with-rules.md
+++ b/docs/developer-guide/working-with-rules.md
@@ -280,18 +280,21 @@ Once you have an instance of `SourceCode`, you can use the methods on it to work
 * `getLastTokensBetween(nodeOrToken1, nodeOrToken2, countOptions)` - returns the last `count` tokens between two nodes or tokens.
 * `getTokens(node)` - returns all tokens for the given node.
 * `getTokensBetween(nodeOrToken1, nodeOrToken2)` - returns all tokens between two nodes.
-* `getTokenByRangeStart(index)` - returns the token whose range starts at the given index in the source.
+* `getTokenByRangeStart(index, rangeOptions)` - returns the token whose range starts at the given index in the source.
 * `getNodeByRangeIndex(index)` - returns the deepest node in the AST containing the given source index.
 
 > `skipOptions` is an object which has 3 properties; `skip`, `includeComments`, and `filter`. Default is `{skip: 0, includeComments: false, filter: null}`.
-> - The `skip` is a positive integer, the number of skipping tokens. If `filter` option is given at the same time, it doesn't count filtered tokens as skipped.
-> - The `includeComments` is a boolean value, the flag to include comment tokens into the result.
-> - The `filter` is a function which gets a token as the first argument, if the function returns `false` then the result excludes the token.
+> - `skip` is a positive integer, the number of skipping tokens. If `filter` option is given at the same time, it doesn't count filtered tokens as skipped.
+> - `includeComments` is a boolean value, the flag to include comment tokens into the result.
+> - `filter` is a function which gets a token as the first argument, if the function returns `false` then the result excludes the token.
 >
 > `countOptions` is an object which has 3 properties; `count`, `includeComments`, and `filter`. Default is `{count: 0, includeComments: false, filter: null}`.
-> - The `count` is a positive integer, the maximum number of returning tokens.
-> - The `includeComments` is a boolean value, the flag to include comment tokens into the result.
-> - The `filter` is a function which gets a token as the first argument, if the function returns `false` then the result excludes the token.
+> - `count` is a positive integer, the maximum number of returning tokens.
+> - `includeComments` is a boolean value, the flag to include comment tokens into the result.
+> - `filter` is a function which gets a token as the first argument, if the function returns `false` then the result excludes the token.
+>
+> `rangeOptions` is an object which has 1 property: `includeComments`.
+> - `includeComments` is a boolean value, the flag to include comment tokens into the result.
 
 There are also some properties you can access:
 
diff --git a/docs/maintainer-guide/governance.md b/docs/maintainer-guide/governance.md
index eea276e8b..098f11598 100644
--- a/docs/maintainer-guide/governance.md
+++ b/docs/maintainer-guide/governance.md
@@ -122,6 +122,10 @@ members of the TSC. TSC members can add any items they like to the
 agenda at the beginning of each meeting. The moderator and the TSC
 cannot veto or remove items.
 
+No binding votes on TSC agenda items can take place without a quorum of
+TSC members present in the meeting. Quorum is achieved when more than
+half of the TSC members are present.
+
 The TSC may invite persons or representatives from certain projects to
 participate in a non-voting capacity.
 
diff --git a/docs/user-guide/command-line-interface.md b/docs/user-guide/command-line-interface.md
index d72e883d7..98214c3f5 100644
--- a/docs/user-guide/command-line-interface.md
+++ b/docs/user-guide/command-line-interface.md
@@ -80,9 +80,9 @@ Options that accept array values can be specified by repeating the option or wit
 
 Example:
 
-    eslint --ext .jsx --ext .js file.js
+    eslint --ext .jsx --ext .js lib/
 
-    eslint --ext .jsx,.js file.js
+    eslint --ext .jsx,.js lib/
 
 ### Basic configuration
 
@@ -137,7 +137,8 @@ Examples:
     # Also use both .js and .js2
     eslint . --ext .js,.js2
 
-**Note:** If you use a glob pattern, then `--ext` is ignored
+**Note:** `--ext` is only used when the arguments are directories. If you use glob patterns or file names, then `--ext` is ignored.
+
 For example, `eslint lib/* --ext .js` will match all files within the `lib/` directory, regardless of extension.
 
 #### `--global`
diff --git a/docs/user-guide/configuring.md b/docs/user-guide/configuring.md
index 1a87f446e..d425d7467 100644
--- a/docs/user-guide/configuring.md
+++ b/docs/user-guide/configuring.md
@@ -728,7 +728,7 @@ Both the JSON and YAML configuration file formats support comments (`package.jso
 
 ## Specifying File extensions to Lint
 
-Currently the sole method for telling ESLint which file extensions to lint is by specifying a comma separated list of extensions using the [`--ext`](./command-line-interface#ext) command line option.
+Currently the sole method for telling ESLint which file extensions to lint is by specifying a comma separated list of extensions using the [`--ext`](./command-line-interface#ext) command line option. Note this flag only takes effect in conjunction with directories, and will be ignored if used with filenames or glob patterns.
 
 ## Ignoring Files and Directories
 
diff --git a/lib/rules/unicode-bom.js b/lib/rules/unicode-bom.js
index 2f16a2585..7109a49ed 100644
--- a/lib/rules/unicode-bom.js
+++ b/lib/rules/unicode-bom.js
@@ -45,7 +45,7 @@ module.exports = {
                         loc: location,
                         message: "Expected Unicode BOM (Byte Order Mark).",
                         fix(fixer) {
-                            return fixer.insertTextBefore(node, "\uFEFF");
+                            return fixer.insertTextBeforeRange([0, 1], "\uFEFF");
                         }
                     });
                 } else if (sourceCode.hasBOM && (requireBOM === "never")) {
diff --git a/lib/token-store/backward-token-comment-cursor.js b/lib/token-store/backward-token-comment-cursor.js
index 0fd8ccbe6..7c2137a17 100644
--- a/lib/token-store/backward-token-comment-cursor.js
+++ b/lib/token-store/backward-token-comment-cursor.js
@@ -17,7 +17,6 @@ const utils = require("./utils");
 
 /**
  * The cursor which iterates tokens and comments in reverse.
- * @private
  */
 module.exports = class BackwardTokenCommentCursor extends Cursor {
 
diff --git a/lib/token-store/backward-token-cursor.js b/lib/token-store/backward-token-cursor.js
index 4daf1ab23..caa117fae 100644
--- a/lib/token-store/backward-token-cursor.js
+++ b/lib/token-store/backward-token-cursor.js
@@ -17,7 +17,6 @@ const utils = require("./utils");
 
 /**
  * The cursor which iterates tokens only in reverse.
- * @private
  */
 module.exports = class BackwardTokenCursor extends Cursor {
 
diff --git a/lib/token-store/cursor.js b/lib/token-store/cursor.js
index de33237a9..4e1595c6d 100644
--- a/lib/token-store/cursor.js
+++ b/lib/token-store/cursor.js
@@ -30,7 +30,6 @@
  *     - SkipCursor .............. The cursor which ignores the first few tokens.
  *     - LimitCursor ............. The cursor which limits the count of tokens.
  *
- * @private
  */
 module.exports = class Cursor {
 
diff --git a/lib/token-store/cursors.js b/lib/token-store/cursors.js
index 4d7749bd9..b315c7e65 100644
--- a/lib/token-store/cursors.js
+++ b/lib/token-store/cursors.js
@@ -37,7 +37,24 @@ class CursorFactory {
     }
 
     /**
-     * Creates the cursor iterates tokens with normalized options.
+     * Creates a base cursor instance that can be decorated by createCursor.
+     *
+     * @param {Token[]} tokens - The array of tokens.
+     * @param {Comment[]} comments - The array of comments.
+     * @param {Object} indexMap - The map from locations to indices in `tokens`.
+     * @param {number} startLoc - The start location of the iteration range.
+     * @param {number} endLoc - The end location of the iteration range.
+     * @param {boolean} includeComments - The flag to iterate comments as well.
+     * @returns {Cursor} The created base cursor.
+     */
+    createBaseCursor(tokens, comments, indexMap, startLoc, endLoc, includeComments) {
+        const Cursor = includeComments ? this.TokenCommentCursor : this.TokenCursor;
+
+        return new Cursor(tokens, comments, indexMap, startLoc, endLoc);
+    }
+
+    /**
+     * Creates a cursor that iterates tokens with normalized options.
      *
      * @param {Token[]} tokens - The array of tokens.
      * @param {Comment[]} comments - The array of comments.
@@ -49,11 +66,9 @@ class CursorFactory {
      * @param {number} skip - The count of tokens the cursor skips.
      * @param {number} count - The maximum count of tokens the cursor iterates. Zero is no iteration for backward compatibility.
      * @returns {Cursor} The created cursor.
-     * @private
      */
     createCursor(tokens, comments, indexMap, startLoc, endLoc, includeComments, filter, skip, count) {
-        const Cursor = includeComments ? this.TokenCommentCursor : this.TokenCursor;
-        let cursor = new Cursor(tokens, comments, indexMap, startLoc, endLoc);
+        let cursor = this.createBaseCursor(tokens, comments, indexMap, startLoc, endLoc, includeComments);
 
         if (filter) {
             cursor = new FilterCursor(cursor, filter);
diff --git a/lib/token-store/decorative-cursor.js b/lib/token-store/decorative-cursor.js
index 0368e8009..f0bff9c51 100644
--- a/lib/token-store/decorative-cursor.js
+++ b/lib/token-store/decorative-cursor.js
@@ -16,7 +16,6 @@ const Cursor = require("./cursor");
 
 /**
  * The abstract class about cursors which manipulate another cursor.
- * @private
  */
 module.exports = class DecorativeCursor extends Cursor {
 
diff --git a/lib/token-store/filter-cursor.js b/lib/token-store/filter-cursor.js
index 0c6bd28aa..7133627bd 100644
--- a/lib/token-store/filter-cursor.js
+++ b/lib/token-store/filter-cursor.js
@@ -16,7 +16,6 @@ const DecorativeCursor = require("./decorative-cursor");
 
 /**
  * The decorative cursor which ignores specified tokens.
- * @private
  */
 module.exports = class FilterCursor extends DecorativeCursor {
 
diff --git a/lib/token-store/forward-token-comment-cursor.js b/lib/token-store/forward-token-comment-cursor.js
index 21dd74d6c..be0855297 100644
--- a/lib/token-store/forward-token-comment-cursor.js
+++ b/lib/token-store/forward-token-comment-cursor.js
@@ -17,7 +17,6 @@ const utils = require("./utils");
 
 /**
  * The cursor which iterates tokens and comments.
- * @private
  */
 module.exports = class ForwardTokenCommentCursor extends Cursor {
 
diff --git a/lib/token-store/forward-token-cursor.js b/lib/token-store/forward-token-cursor.js
index 978b7333f..5748cb45a 100644
--- a/lib/token-store/forward-token-cursor.js
+++ b/lib/token-store/forward-token-cursor.js
@@ -17,7 +17,6 @@ const utils = require("./utils");
 
 /**
  * The cursor which iterates tokens only.
- * @private
  */
 module.exports = class ForwardTokenCursor extends Cursor {
 
diff --git a/lib/token-store/index.js b/lib/token-store/index.js
index b74126b90..86d05cf7b 100644
--- a/lib/token-store/index.js
+++ b/lib/token-store/index.js
@@ -45,7 +45,7 @@ const PUBLIC_METHODS = Object.freeze([
  * Creates the map from locations to indices in `tokens`.
  *
  * The first/last location of tokens is mapped to the index of the token.
- * The first/last location of comments is mapped to the index of the previous token of each comment.
+ * The first/last location of comments is mapped to the index of the next token of each comment.
  *
  * @param {Token[]} tokens - The array of tokens.
  * @param {Comment[]} comments - The array of comments.
@@ -236,10 +236,20 @@ module.exports = class TokenStore {
     /**
      * Gets the token starting at the specified index.
      * @param {number} offset - Index of the start of the token's range.
+     * @param {Object} [options=0] - The option object.
+     * @param {boolean} [options.includeComments=false] - The flag to iterate comments as well.
      * @returns {Token|null} The token starting at index, or null if no such token.
      */
-    getTokenByRangeStart(offset) {
-        const token = this.tokens[this.indexMap[offset]] || null;
+    getTokenByRangeStart(offset, options) {
+        const includeComments = options && options.includeComments;
+        const token = cursors.forward.createBaseCursor(
+            this.tokens,
+            this.comments,
+            this.indexMap,
+            offset,
+            -1,
+            includeComments
+        ).getOneToken();
 
         if (token && token.range[0] === offset) {
             return token;
diff --git a/lib/token-store/limit-cursor.js b/lib/token-store/limit-cursor.js
index a0bdae16d..efb46cf0e 100644
--- a/lib/token-store/limit-cursor.js
+++ b/lib/token-store/limit-cursor.js
@@ -16,7 +16,6 @@ const DecorativeCursor = require("./decorative-cursor");
 
 /**
  * The decorative cursor which limits the number of tokens.
- * @private
  */
 module.exports = class LimitCursor extends DecorativeCursor {
 
diff --git a/lib/token-store/padded-token-cursor.js b/lib/token-store/padded-token-cursor.js
index cf91b8b7e..c083aed1e 100644
--- a/lib/token-store/padded-token-cursor.js
+++ b/lib/token-store/padded-token-cursor.js
@@ -17,7 +17,6 @@ const ForwardTokenCursor = require("./forward-token-cursor");
 /**
  * The cursor which iterates tokens only, with inflated range.
  * This is for the backward compatibility of padding options.
- * @private
  */
 module.exports = class PaddedTokenCursor extends ForwardTokenCursor {
 
diff --git a/lib/token-store/skip-cursor.js b/lib/token-store/skip-cursor.js
index f22844535..ab34dfab0 100644
--- a/lib/token-store/skip-cursor.js
+++ b/lib/token-store/skip-cursor.js
@@ -16,7 +16,6 @@ const DecorativeCursor = require("./decorative-cursor");
 
 /**
  * The decorative cursor which ignores the first few tokens.
- * @private
  */
 module.exports = class SkipCursor extends DecorativeCursor {
 
diff --git a/lib/token-store/utils.js b/lib/token-store/utils.js
index 2dea7636f..f83b6cf69 100644
--- a/lib/token-store/utils.js
+++ b/lib/token-store/utils.js
@@ -36,7 +36,6 @@ function getStartLocation(token) {
  * @param {(Token|Comment)[]} tokens - It searches the token in this list.
  * @param {number} location - The location to search.
  * @returns {number} The found index or `tokens.length`.
- * @private
  */
 exports.search = function search(tokens, location) {
     return lodash.sortedIndexBy(
@@ -54,14 +53,21 @@ exports.search = function search(tokens, location) {
  * @param {Object} indexMap - The map from locations to indices.
  * @param {number} startLoc - The location to get an index.
  * @returns {number} The index.
- * @private
  */
 exports.getFirstIndex = function getFirstIndex(tokens, indexMap, startLoc) {
     if (startLoc in indexMap) {
         return indexMap[startLoc];
     }
     if ((startLoc - 1) in indexMap) {
-        return indexMap[startLoc - 1] + 1;
+        const index = indexMap[startLoc - 1];
+        const token = (index >= 0 && index < tokens.length) ? tokens[index] : null;
+
+        // For the map of "comment's location -> token's index", it points the next token of a comment.
+        // In that case, +1 is unnecessary.
+        if (token && token.range[0] >= startLoc) {
+            return index;
+        }
+        return index + 1;
     }
     return 0;
 };
@@ -74,14 +80,21 @@ exports.getFirstIndex = function getFirstIndex(tokens, indexMap, startLoc) {
  * @param {Object} indexMap - The map from locations to indices.
  * @param {number} endLoc - The location to get an index.
  * @returns {number} The index.
- * @private
  */
 exports.getLastIndex = function getLastIndex(tokens, indexMap, endLoc) {
     if (endLoc in indexMap) {
         return indexMap[endLoc] - 1;
     }
     if ((endLoc - 1) in indexMap) {
-        return indexMap[endLoc - 1];
+        const index = indexMap[endLoc - 1];
+        const token = (index >= 0 && index < tokens.length) ? tokens[index] : null;
+
+        // For the map of "comment's location -> token's index", it points the next token of a comment.
+        // In that case, -1 is necessary.
+        if (token && token.range[1] > endLoc) {
+            return index - 1;
+        }
+        return index;
     }
     return tokens.length - 1;
 };


Test patch:
diff --git a/lib/testers/rule-tester.js b/lib/testers/rule-tester.js
index 14a5cbc03..e8442f079 100644
--- a/lib/testers/rule-tester.js
+++ b/lib/testers/rule-tester.js
@@ -147,6 +147,12 @@ function RuleTester(testerConfig) {
         lodash.cloneDeep(defaultConfig),
         testerConfig
     );
+
+    /**
+     * Rule definitions to define before tests.
+     * @type {Object}
+     */
+    this.rules = {};
 }
 
 /**
@@ -239,7 +245,7 @@ RuleTester.prototype = {
      * @returns {void}
      */
     defineRule(name, rule) {
-        eslint.defineRule(name, rule);
+        this.rules[name] = rule;
     },
 
     /**
@@ -508,6 +514,7 @@ RuleTester.prototype = {
             RuleTester.describe("valid", () => {
                 test.valid.forEach(valid => {
                     RuleTester.it(valid.code || valid, () => {
+                        eslint.defineRules(this.rules);
                         testValidTemplate(ruleName, valid);
                     });
                 });
@@ -516,6 +523,7 @@ RuleTester.prototype = {
             RuleTester.describe("invalid", () => {
                 test.invalid.forEach(invalid => {
                     RuleTester.it(invalid.code, () => {
+                        eslint.defineRules(this.rules);
                         testInvalidTemplate(ruleName, invalid);
                     });
                 });
diff --git a/tests/lib/rules/unicode-bom.js b/tests/lib/rules/unicode-bom.js
index c0068586a..43d780b9c 100644
--- a/tests/lib/rules/unicode-bom.js
+++ b/tests/lib/rules/unicode-bom.js
@@ -41,6 +41,12 @@ ruleTester.run("unicode-bom", rule, {
             options: ["always"],
             output: "\uFEFFvar a = 123;"
         },
+        {
+            code: " // here's a comment \nvar a = 123;",
+            errors: [{ message: "Expected Unicode BOM (Byte Order Mark).", type: "Program" }],
+            options: ["always"],
+            output: "\uFEFF // here's a comment \nvar a = 123;"
+        },
         {
             code: "\uFEFF var a = 123;",
             errors: [{ message: "Unexpected Unicode BOM (Byte Order Mark).", type: "Program" }],
diff --git a/tests/lib/token-store.js b/tests/lib/token-store.js
index 0d2c6d81f..a465918f6 100644
--- a/tests/lib/token-store.js
+++ b/tests/lib/token-store.js
@@ -271,6 +271,33 @@ describe("TokenStore", () => {
             );
         });
 
+        it("should retrieve the previous node if the comment at the end of source code is specified.", () => {
+            const code = "a + b /*comment*/";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const token = tokenStore.getTokenBefore(ast.comments[0]);
+
+            assert.strictEqual(token.value, "b");
+        });
+
+        it("should retrieve the previous comment if the first token is specified.", () => {
+            const code = "/*comment*/ a + b";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const token = tokenStore.getTokenBefore(ast.tokens[0], { includeComments: true });
+
+            assert.strictEqual(token.value, "comment");
+        });
+
+        it("should retrieve null if the first comment is specified.", () => {
+            const code = "/*comment*/ a + b";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const token = tokenStore.getTokenBefore(ast.comments[0], { includeComments: true });
+
+            assert.strictEqual(token, null);
+        });
+
     });
 
     describe("when calling getTokensAfter", () => {
@@ -417,6 +444,33 @@ describe("TokenStore", () => {
             );
         });
 
+        it("should retrieve the next node if the comment at the first of source code is specified.", () => {
+            const code = "/*comment*/ a + b";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const token = tokenStore.getTokenAfter(ast.comments[0]);
+
+            assert.strictEqual(token.value, "a");
+        });
+
+        it("should retrieve the next comment if the last token is specified.", () => {
+            const code = "a + b /*comment*/";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const token = tokenStore.getTokenAfter(ast.tokens[2], { includeComments: true });
+
+            assert.strictEqual(token.value, "comment");
+        });
+
+        it("should retrieve null if the last comment is specified.", () => {
+            const code = "a + b /*comment*/";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const token = tokenStore.getTokenAfter(ast.comments[0], { includeComments: true });
+
+            assert.strictEqual(token, null);
+        });
+
     });
 
     describe("when calling getFirstTokens", () => {
@@ -567,6 +621,35 @@ describe("TokenStore", () => {
             );
         });
 
+        it("should retrieve the first comment if the comment is at the last of nodes", () => {
+            const code = "a + b\n/*comment*/ c + d";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+
+            // Actually, the first of nodes is always tokens, not comments.
+            // But I think this test case is needed for completeness.
+            const token = tokenStore.getFirstToken(
+                { range: [ast.comments[0].range[0], ast.tokens[5].range[1]] },
+                { includeComments: true }
+            );
+
+            assert.strictEqual(token.value, "comment");
+        });
+
+        it("should retrieve the first token (without includeComments option) if the comment is at the last of nodes", () => {
+            const code = "a + b\n/*comment*/ c + d";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+
+            // Actually, the first of nodes is always tokens, not comments.
+            // But I think this test case is needed for completeness.
+            const token = tokenStore.getFirstToken(
+                { range: [ast.comments[0].range[0], ast.tokens[5].range[1]] }
+            );
+
+            assert.strictEqual(token.value, "c");
+        });
+
     });
 
     describe("when calling getLastTokens", () => {
@@ -721,6 +804,35 @@ describe("TokenStore", () => {
             );
         });
 
+        it("should retrieve the last comment if the comment is at the last of nodes", () => {
+            const code = "a + b /*comment*/\nc + d";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+
+            // Actually, the last of nodes is always tokens, not comments.
+            // But I think this test case is needed for completeness.
+            const token = tokenStore.getLastToken(
+                { range: [ast.tokens[0].range[0], ast.comments[0].range[1]] },
+                { includeComments: true }
+            );
+
+            assert.strictEqual(token.value, "comment");
+        });
+
+        it("should retrieve the last token (without includeComments option) if the comment is at the last of nodes", () => {
+            const code = "a + b /*comment*/\nc + d";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+
+            // Actually, the last of nodes is always tokens, not comments.
+            // But I think this test case is needed for completeness.
+            const token = tokenStore.getLastToken(
+                { range: [ast.tokens[0].range[0], ast.comments[0].range[1]] }
+            );
+
+            assert.strictEqual(token.value, "b");
+        });
+
     });
 
     describe("when calling getFirstTokensBetween", () => {
@@ -1020,6 +1132,25 @@ describe("TokenStore", () => {
             assert.isNull(result);
         });
 
+        it("should return a comment token when includeComments is true", () => {
+            const result = store.getTokenByRangeStart(15, { includeComments: true });
+
+            assert.equal(result.type, "Block");
+            assert.equal(result.value, "B");
+        });
+
+        it("should not return a comment token at the supplied index when includeComments is false", () => {
+            const result = store.getTokenByRangeStart(15, { includeComments: false });
+
+            assert.isNull(result);
+        });
+
+        it("should not return comment tokens by default", () => {
+            const result = store.getTokenByRangeStart(15);
+
+            assert.isNull(result);
+        });
+
     });
 
     describe("when calling getTokenOrCommentBefore", () => {
@@ -1066,4 +1197,79 @@ describe("TokenStore", () => {
 
     });
 
+    describe("when calling getFirstToken & getTokenAfter", () => {
+        it("should retrieve all tokens and comments in the node", () => {
+            const code = "(function(a, /*b,*/ c){})";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const tokens = [];
+            let token = tokenStore.getFirstToken(ast);
+
+            while (token) {
+                tokens.push(token);
+                token = tokenStore.getTokenAfter(token, { includeComments: true });
+            }
+
+            check(
+                tokens,
+                ["(", "function", "(", "a", ",", "b,", "c", ")", "{", "}", ")"]
+            );
+        });
+
+        it("should retrieve all tokens and comments in the node (no spaces)", () => {
+            const code = "(function(a,/*b,*/c){})";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const tokens = [];
+            let token = tokenStore.getFirstToken(ast);
+
+            while (token) {
+                tokens.push(token);
+                token = tokenStore.getTokenAfter(token, { includeComments: true });
+            }
+
+            check(
+                tokens,
+                ["(", "function", "(", "a", ",", "b,", "c", ")", "{", "}", ")"]
+            );
+        });
+    });
+
+    describe("when calling getLastToken & getTokenBefore", () => {
+        it("should retrieve all tokens and comments in the node", () => {
+            const code = "(function(a, /*b,*/ c){})";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const tokens = [];
+            let token = tokenStore.getLastToken(ast);
+
+            while (token) {
+                tokens.push(token);
+                token = tokenStore.getTokenBefore(token, { includeComments: true });
+            }
+
+            check(
+                tokens.reverse(),
+                ["(", "function", "(", "a", ",", "b,", "c", ")", "{", "}", ")"]
+            );
+        });
+
+        it("should retrieve all tokens and comments in the node (no spaces)", () => {
+            const code = "(function(a,/*b,*/c){})";
+            const ast = espree.parse(code, { loc: true, range: true, tokens: true, comment: true });
+            const tokenStore = new TokenStore(ast.tokens, ast.comments);
+            const tokens = [];
+            let token = tokenStore.getLastToken(ast);
+
+            while (token) {
+                tokens.push(token);
+                token = tokenStore.getTokenBefore(token, { includeComments: true });
+            }
+
+            check(
+                tokens.reverse(),
+                ["(", "function", "(", "a", ",", "b,", "c", ")", "{", "}", ")"]
+            );
+        });
+    });
 });
diff --git a/tests/lib/util/source-code.js b/tests/lib/util/source-code.js
index e26335255..ef4864f37 100644
--- a/tests/lib/util/source-code.js
+++ b/tests/lib/util/source-code.js
@@ -77,7 +77,7 @@ describe("SourceCode", () => {
 
         });
 
-        it("should throw an error when called with an AST that's missing comments", () => {
+        it("should throw an error when called with an AST that's missing location", () => {
 
             assert.throws(() => {
                 new SourceCode("foo;", { comments: [], tokens: [], range: [] });
@@ -85,7 +85,7 @@ describe("SourceCode", () => {
 
         });
 
-        it("should throw an error when called with an AST that's missing comments", () => {
+        it("should throw an error when called with an AST that's missing range", () => {
 
             assert.throws(() => {
                 new SourceCode("foo;", { comments: [], tokens: [], loc: {} });