Repository: sqlfluff/sqlfluff

Issue:
Ignore file even when passed as CLI argument
Currently sqlfluff ignores files that match patterns in `.sqlfluffignore` but only if a file is not specified by name directly. 

**For example**

.sqlfluffignore:
```
test.sql
```

test.sql:
```
select a, b, c, d, e fromm table_x
```

Result of `sqlfluff lint .` is empty while result of `sqlfluff lint test.sql` is :
```
== [test.sql] FAIL
L:   1 | P:  22 | L012 | Implicit aliasing of column not allowed. Use explicit
                       | `AS` clause.
L:   1 | P:  27 |  PRS | Found unparsable section: ' table_x'
WARNING: Parsing errors found and dialect is set to 'ansi'. Have you configured your dialect?
```

### Why is it a problem?

I'm currently using sqlfluff with `pre-commit`, which passes a list of changed files to the processes so only the changed files are linted/validated etc. Right now I need to separately add an `exclude:` to the pre-commit config as such:

```yaml
exclude: test.sql
```

### Suggested solution

By default I think sqlfluff should always ignore files that match the patterns defined in `.sqlfluffignore`. If a file is passed explicitly I think it should just log a warning that the file was not linted/parsed because it was ignored. A `--force` flag could be passed to ignore the `.sqlfluffignore` file.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=h1) Report
> Merging [#486](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=desc) into [master](https://codecov.io/gh/sqlfluff/sqlfluff/commit/7206df10db754614af1d0f4211a6ccc6e3c0a6d8?el=desc) will **increase** coverage by `0.11%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486/graphs/tree.svg?width=650&height=150&src=pr&token=maM8kLgCmn)](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #486      +/-   ##
==========================================
+ Coverage   92.82%   92.93%   +0.11%     
==========================================
  Files          29       29              
  Lines        4769     4801      +32     
==========================================
+ Hits         4427     4462      +35     
+ Misses        342      339       -3     
```

| Flag | Coverage Δ | |
|---|---|---|
| #py35 | `92.82% <100.00%> (+0.11%)` | :arrow_up: |
| #py36 | `92.82% <100.00%> (+0.11%)` | :arrow_up: |
| #py37 | `92.82% <100.00%> (+0.11%)` | :arrow_up: |
| #py38 | `92.77% <100.00%> (+0.11%)` | :arrow_up: |
| #py39 | `92.77% <100.00%> (+0.11%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/sqlfluff/cli/commands.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2NsaS9jb21tYW5kcy5weQ==) | `90.87% <100.00%> (+0.03%)` | :arrow_up: |
| [src/sqlfluff/linter.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL2xpbnRlci5weQ==) | `89.20% <100.00%> (+1.16%)` | :arrow_up: |
| [src/sqlfluff/rules/std.py](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486/diff?src=pr&el=tree#diff-c3JjL3NxbGZsdWZmL3J1bGVzL3N0ZC5weQ==) | `94.31% <0.00%> (+0.10%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=footer). Last update [7206df1...c236a65](https://codecov.io/gh/sqlfluff/sqlfluff/pull/486?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

pwildenhain: Quick FYI regarding your changes for python 3.5: We're going to be dropping support for 3.5 in #482 which should be getting merged soon
dmateusp: thanks for the heads up @pwildenhain I think I'll revert the python3.5 specific commits once #482 is merged then 
dmateusp: @pwildenhain fyi I just hit a minor issue with black, where my outdated version wasn't formatting all the files the same way that the `tox` environment did

what do you think about either having a `requirements_dev.txt` which specifies the version of black to use, or adding `pre-commit` ?
pwildenhain: I think this is a great idea, and I think `pre-commit` makes the most sense since you would probably want to run `flake8`, `flake8-black`, and `doc8` to handle any of the easy checks before commiting. See #496 
dmateusp: @alanmcruickshank can you review commit c34e3c3a8de711409d11e6797114cd3afad992c6 which is re-using the logic from `sqlfuff.config.ConfigLoader.load_config_up_to_path()`?
dmateusp: @alanmcruickshank I gave it another iteration, thanks for the feedback
dmateusp: @alanmcruickshank ping :) so this PR doesn't get forgotten since we're looking into adding pre-commit to the repo
NiallRees: @alanmcruickshank 
NiallRees: Go for it @dmateusp !

Patch:
diff --git a/src/sqlfluff/cli/commands.py b/src/sqlfluff/cli/commands.py
index 055fc874d..434caa9e7 100644
--- a/src/sqlfluff/cli/commands.py
+++ b/src/sqlfluff/cli/commands.py
@@ -274,8 +274,13 @@ def dialects(**kwargs):
         "found. This is potentially useful during rollout."
     ),
 )
+@click.option(
+    "--disregard-sqlfluffignores",
+    is_flag=True,
+    help=("Perform the operation regardless of .sqlfluffignore configurations"),
+)
 @click.argument("paths", nargs=-1)
-def lint(paths, format, nofail, logger=None, **kwargs):
+def lint(paths, format, nofail, disregard_sqlfluffignores, logger=None, **kwargs):
     """Lint SQL files via passing a list of files or using stdin.
 
     PATH is the path to a sql file or directory to lint. This can be either a
@@ -312,7 +317,11 @@ def lint(paths, format, nofail, logger=None, **kwargs):
             click.echo(format_linting_result_header())
         try:
             # TODO: Remove verbose
-            result = lnt.lint_paths(paths, ignore_non_existent_files=False)
+            result = lnt.lint_paths(
+                paths,
+                ignore_non_existent_files=False,
+                ignore_files=not disregard_sqlfluffignores,
+            )
         except IOError:
             click.echo(
                 colorize(
diff --git a/src/sqlfluff/core/config.py b/src/sqlfluff/core/config.py
index d8d682c13..5c0b68094 100644
--- a/src/sqlfluff/core/config.py
+++ b/src/sqlfluff/core/config.py
@@ -5,6 +5,7 @@ import os.path
 import sys
 import configparser
 from typing import Dict, List, Tuple, Any, Optional, Union, Iterable
+from pathlib import Path
 
 import appdirs
 
@@ -263,42 +264,58 @@ class ConfigLoader:
         """Loads a selection of config files from both the path and it's parent paths."""
         user_appdir_config = self.load_user_appdir_config()
         user_config = self.load_user_config()
+        config_paths = self.iter_config_locations_up_to_path(path)
+        config_stack = [self.load_config_at_path(p) for p in config_paths]
+        return nested_combine(user_appdir_config, user_config, *config_stack)
+
+    @classmethod
+    def find_ignore_config_files(
+        cls, path, working_path=os.getcwd(), ignore_file_name=".sqlfluffignore"
+    ):
+        """Finds sqlfluff ignore files from both the path and its parent paths."""
+        return set(
+            filter(
+                os.path.isfile,
+                map(
+                    lambda x: os.path.join(x, ignore_file_name),
+                    cls.iter_config_locations_up_to_path(
+                        path=path, working_path=working_path
+                    ),
+                ),
+            )
+        )
+
+    @staticmethod
+    def iter_config_locations_up_to_path(path, working_path=Path.cwd()):
+        """Finds config locations from both the path and it's parent paths.
+
+        The lowest priority is the user appdir, then home dir, then increasingly
+        the configs closest to the file being directly linted.
+        """
+        given_path = Path(path).resolve()
+        working_path = Path(working_path).resolve()
 
-        working_path = os.getcwd()
-        given_path = os.path.abspath(path)
         # If we've been passed a file and not a directory,
         # then go straight to the directory.
-        if not os.path.isdir(given_path):
-            given_path = os.path.dirname(given_path)
-        config_stack: List[Dict] = []
+        if not given_path.is_dir():
+            given_path = given_path.parent
 
-        if hasattr(os.path, "commonpath"):
-            common_path = os.path.commonpath([working_path, given_path])
-        else:
-            # Compatabilty with pre python 3.5
-            common_path = os.path.commonprefix([working_path, given_path])
-
-        if common_path == working_path:
-            # we have a sub path! We can load nested paths
-            last_path = given_path
-            while True:
-                config_stack.insert(0, self.load_config_at_path(last_path))
-                if last_path == working_path:
-                    break
-                # iterate up the directories
-                if last_path == os.path.dirname(last_path):
-                    # we're not making progres...
-                    # [prevent infinite loop]
-                    break
-                last_path = os.path.dirname(last_path)
-            config_stack.insert(0, self.load_config_at_path(working_path))
-        else:
-            # we have divergent paths, we can only load config for that path and global
-            config_stack.append(self.load_config_at_path(given_path))
+        common_path = Path(os.path.commonpath([working_path, given_path]))
 
-        # The lowest priority is the user appdir config, then home dir config,
-        # then increasingly the configs closest to the file being directly linted.
-        return nested_combine(user_appdir_config, user_config, *config_stack)
+        # we have a sub path! We can load nested paths
+        path_to_visit = common_path
+        while path_to_visit != given_path:
+            yield str(path_to_visit.resolve())
+            next_path_to_visit = (
+                path_to_visit / given_path.relative_to(path_to_visit).parts[0]
+            )
+            if next_path_to_visit == path_to_visit:
+                # we're not making progres...
+                # [prevent infinite loop]
+                break
+            path_to_visit = next_path_to_visit
+
+        yield str(given_path.resolve())
 
 
 class FluffConfig:
diff --git a/src/sqlfluff/core/linter.py b/src/sqlfluff/core/linter.py
index 1ca12546d..3c0b9f789 100644
--- a/src/sqlfluff/core/linter.py
+++ b/src/sqlfluff/core/linter.py
@@ -18,7 +18,7 @@ import pathspec
 from .errors import SQLLexError, SQLParseError
 from .parser import Lexer, Parser
 from .rules import get_ruleset
-from .config import FluffConfig
+from .config import FluffConfig, ConfigLoader
 
 
 # Instantiate the linter logger
@@ -906,12 +906,24 @@ class Linter:
         return linted_file
 
     def paths_from_path(
-        self, path, ignore_file_name=".sqlfluffignore", ignore_non_existent_files=False
+        self,
+        path,
+        ignore_file_name=".sqlfluffignore",
+        ignore_non_existent_files=False,
+        ignore_files=True,
+        working_path=os.getcwd(),
     ):
         """Return a set of sql file paths from a potentially more ambigious path string.
 
         Here we also deal with the .sqlfluffignore file if present.
 
+        When a path to a file to be linted is explicitly passed
+        we look for ignore files in all directories that are parents of the file,
+        up to the current directory.
+
+        If the current directory is not a parent of the file we only
+        look for an ignore file in the direct parent of the file.
+
         """
         if not os.path.exists(path):
             if ignore_non_existent_files:
@@ -919,24 +931,50 @@ class Linter:
             else:
                 raise IOError("Specified path does not exist")
 
-        # Files referred to exactly are never ignored.
-        if not os.path.isdir(path):
-            return [path]
+        # Files referred to exactly are also ignored if
+        # matched, but we warn the users when that happens
+        is_exact_file = not os.path.isdir(path)
+
+        if is_exact_file:
+            # When the exact file to lint is passed, we
+            # fill path_walk with an input that follows
+            # the structure of `os.walk`:
+            #   (root, directories, files)
+            dirpath = os.path.dirname(path)
+            files = [os.path.basename(path)]
+            ignore_file_paths = ConfigLoader.find_ignore_config_files(
+                path=path, working_path=working_path, ignore_file_name=ignore_file_name
+            )
+            # Add paths that could contain "ignore files"
+            # to the path_walk list
+            path_walk_ignore_file = [
+                (
+                    os.path.dirname(ignore_file_path),
+                    None,
+                    # Only one possible file, since we only
+                    # have one "ignore file name"
+                    [os.path.basename(ignore_file_path)],
+                )
+                for ignore_file_path in ignore_file_paths
+            ]
+            path_walk = [(dirpath, None, files)] + path_walk_ignore_file
+        else:
+            path_walk = os.walk(path)
 
         # If it's a directory then expand the path!
-        ignore_set = set()
         buffer = []
-        for dirpath, _, filenames in os.walk(path):
+        ignore_set = set()
+        for dirpath, _, filenames in path_walk:
             for fname in filenames:
                 fpath = os.path.join(dirpath, fname)
                 # Handle potential .sqlfluffignore files
-                if fname == ignore_file_name:
+                if ignore_files and fname == ignore_file_name:
                     with open(fpath, "r") as fh:
                         spec = pathspec.PathSpec.from_lines("gitwildmatch", fh)
                     matches = spec.match_tree(dirpath)
                     for m in matches:
                         ignore_path = os.path.join(dirpath, m)
-                        ignore_set.add(ignore_path)
+                        ignore_set.add(os.path.abspath(ignore_path))
                     # We don't need to process the ignore file any futher
                     continue
 
@@ -949,11 +987,27 @@ class Linter:
                     if fname.endswith(ext):
                         buffer.append(fpath)
 
+        if not ignore_files:
+            return sorted(buffer)
+
         # Check the buffer for ignore items and normalise the rest.
         filtered_buffer = []
+
         for fpath in buffer:
-            if fpath not in ignore_set:
+            if os.path.abspath(fpath) not in ignore_set:
                 filtered_buffer.append(os.path.normpath(fpath))
+            elif is_exact_file:
+                linter_logger.warning(
+                    "Exact file path %s was given but "
+                    "it was ignored by a %s pattern, "
+                    "re-run with `--disregard-sqlfluffignores` to "
+                    "skip %s"
+                    % (
+                        path,
+                        ignore_file_name,
+                        ignore_file_name,
+                    )
+                )
 
         # Return
         return sorted(filtered_buffer)
@@ -966,13 +1020,17 @@ class Linter:
         result.add(linted_path)
         return result
 
-    def lint_path(self, path, fix=False, ignore_non_existent_files=False):
+    def lint_path(
+        self, path, fix=False, ignore_non_existent_files=False, ignore_files=True
+    ):
         """Lint a path."""
         linted_path = LintedPath(path)
         if self.formatter:
             self.formatter.dispatch_path(path)
         for fname in self.paths_from_path(
-            path, ignore_non_existent_files=ignore_non_existent_files
+            path,
+            ignore_non_existent_files=ignore_non_existent_files,
+            ignore_files=ignore_files,
         ):
             config = self.config.make_child_from_path(fname)
             # Handle unicode issues gracefully
@@ -986,7 +1044,9 @@ class Linter:
                 )
         return linted_path
 
-    def lint_paths(self, paths, fix=False, ignore_non_existent_files=False):
+    def lint_paths(
+        self, paths, fix=False, ignore_non_existent_files=False, ignore_files=True
+    ):
         """Lint an iterable of paths."""
         # If no paths specified - assume local
         if len(paths) == 0:
@@ -998,7 +1058,10 @@ class Linter:
             # or read the file directly if it's not
             result.add(
                 self.lint_path(
-                    path, fix=fix, ignore_non_existent_files=ignore_non_existent_files
+                    path,
+                    fix=fix,
+                    ignore_non_existent_files=ignore_non_existent_files,
+                    ignore_files=ignore_files,
                 )
             )
         return result


Test patch:
diff --git a/test/cli/commands_test.py b/test/cli/commands_test.py
index 4498f4c7f..96c7be2a3 100644
--- a/test/cli/commands_test.py
+++ b/test/cli/commands_test.py
@@ -207,6 +207,33 @@ def test__cli__command_lint_parse(command):
     invoke_assert_code(args=command)
 
 
+def test__cli__command_lint_warning_explicit_file_ignored():
+    """Check ignoring file works when passed explicitly and ignore file is in the same directory."""
+    runner = CliRunner()
+    result = runner.invoke(
+        lint, ["test/fixtures/linter/sqlfluffignore/path_b/query_c.sql"]
+    )
+    assert result.exit_code == 0
+    assert (
+        "Exact file path test/fixtures/linter/sqlfluffignore/path_b/query_c.sql "
+        "was given but it was ignored"
+    ) in result.output.strip()
+
+
+def test__cli__command_lint_skip_ignore_files():
+    """Check "ignore file" is skipped when --disregard-sqlfluffignores flag is set."""
+    runner = CliRunner()
+    result = runner.invoke(
+        lint,
+        [
+            "test/fixtures/linter/sqlfluffignore/path_b/query_c.sql",
+            "--disregard-sqlfluffignores",
+        ],
+    )
+    assert result.exit_code == 65
+    assert "L009" in result.output.strip()
+
+
 def test__cli__command_versioning():
     """Check version command."""
     # Get the package version info
diff --git a/test/core/config_test.py b/test/core/config_test.py
index 7cd52b3ef..634bfed26 100644
--- a/test/core/config_test.py
+++ b/test/core/config_test.py
@@ -5,6 +5,8 @@ import os
 from sqlfluff.core.config import ConfigLoader, nested_combine, dict_diff
 from sqlfluff.core import Linter, FluffConfig
 
+from pathlib import Path
+
 
 config_a = {
     "core": {"testing_val": "foobar", "testing_int": 4},
@@ -67,6 +69,42 @@ def test__config__load_nested():
     }
 
 
+def test__config__iter_config_paths_right_order():
+    """Test that config paths are fetched ordered by priority."""
+    c = ConfigLoader()
+    cfg_paths = c.iter_config_locations_up_to_path(
+        os.path.join(
+            "test", "fixtures", "config", "inheritance_a", "nested", "blah.sql"
+        ),
+        working_path="test/fixtures",
+    )
+    assert list(cfg_paths) == [
+        str(Path(p).resolve())
+        for p in [
+            "test/fixtures",
+            "test/fixtures/config",
+            "test/fixtures/config/inheritance_a",
+            "test/fixtures/config/inheritance_a/nested",
+        ]
+    ]
+
+
+def test__config__find_sqlfluffignore_in_same_directory():
+    """Test find ignore file in the same directory as sql file."""
+    ignore_files = ConfigLoader.find_ignore_config_files(
+        path="test/fixtures/linter/sqlfluffignore/path_b/query_b.sql",
+        working_path="test/fixtures/linter/sqlfluffignore/",
+    )
+    assert ignore_files == set(
+        [
+            os.path.abspath(
+                "test/fixtures/linter/sqlfluffignore/path_b/.sqlfluffignore"
+            ),
+            os.path.abspath("test/fixtures/linter/sqlfluffignore/.sqlfluffignore"),
+        ]
+    )
+
+
 def test__config__nested_config_tests():
     """Test linting with overriden config in nested paths.
 
diff --git a/test/core/linter_test.py b/test/core/linter_test.py
index e2cb49c52..a21f9704b 100644
--- a/test/core/linter_test.py
+++ b/test/core/linter_test.py
@@ -2,7 +2,7 @@
 
 import pytest
 
-from sqlfluff.core import Linter
+from sqlfluff.core import Linter, FluffConfig
 from sqlfluff.core.linter import LintingResult
 
 
@@ -47,6 +47,18 @@ def test__linter__path_from_paths__not_exist_ignore():
     assert len(paths) == 0
 
 
+def test__linter__path_from_paths__explicit_ignore():
+    """Test ignoring files that were passed explicitly."""
+    lntr = Linter(config=FluffConfig())
+    paths = lntr.paths_from_path(
+        "test/fixtures/linter/sqlfluffignore/path_a/query_a.sql",
+        ignore_non_existent_files=True,
+        ignore_files=True,
+        working_path="test/fixtures/linter/sqlfluffignore/",
+    )
+    assert len(paths) == 0
+
+
 def test__linter__path_from_paths__dot():
     """Test extracting paths from a dot."""
     lntr = Linter()