Repository: iterative/dvc

Issue:
better error when output is dvcignored
UPDATE: Skip to https://github.com/iterative/dvc/issues/3538#issuecomment-669603698

---

```
$ dvc version
DVC version: 0.90.2
Python version: 3.7.5
Platform: Windows-10-10.0.18362-SP0
Binary: True
Package: exe
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Filesystem type (workspace): ('NTFS', 'C:\\')
```

Having `data` in .dvcignore makes DVC commands ignore all file names starting with that string e.g `data2`, `dataset`. https://dvc.org/doc/user-guide/dvcignore links to [gitignore patterms](https://git-scm.com/docs/gitignore) but Git doesn't do that.

UPDATE: Skip to https://github.com/iterative/dvc/issues/3538#issuecomment-621947830


Discussion:
@jorgeorpinel So having `data`, right? Git does the same thing, hence why we use `/data` in `.gitignore` when we `dvc add data`.
Maybe it behaves differently in distinct OSs? I'm on Windows (Cmder/ Git Bash):

```
λ git init
λ touch data data2 data3 data4
λ echo data > .gitignore
λ ls
data  data2  data3  data4
λ git status
...
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .gitignore
        data2
        data3
        data4
λ git add data
The following paths are ignored by one of your .gitignore files:
data
Use -f if you really want to add them.
λ git add .
λ git status
...
        new file:   .gitignore
        new file:   data2
        new file:   data3
        new file:   data4
```
@jorgeorpinel Oh, I misunderstood your report. Need to check that, indeed. @pared Maybe you have any ideas?
Will check this, sorry for late response
@jorgeorpinel that was happening on windows right?
following test is passing on my linux machine for `0.90.2`:
```
def test_ignore_data(tmp_dir, dvc):
    tmp_dir.gen({"data": "data", "data1": "data1", "data2": "data2",
                 "data3":"data3", ".dvcignore": "data"})


    files = []
    for r, d, fs in dvc.tree.walk("."):
        files.extend([os.path.join(r, f) for f in fs])

    assert set(files) == set(["./data1", "./data2", "./data3", "./.dvcignore"])
```
Sorry guys. I don't remember which DVC commands I noticed originally that don't respect files in .dvcignore. It wasn't a great report... But I'm QAing this and I think there's definitely some funny stuff happening:

```console
$ echo data > data
$ echo data > .dvcignore
$ dvc add data
100% Add|██...  # Works and puts it in .gitignore. Should ignore? E.g.:

$ git add data
The following paths are ignored by one of your .gitignore files:
data
Use -f if you really want to add them.
```

I'm on 0.93.0 from exe installer now. Yes, Windows (Git Bash on Cmder)

Also, the first example in https://dvc.org/doc/user-guide/dvcignore#examples doesn't work for me (all files inside data/ dir are added and cached). I didn't check the others.

@jorgeorpinel hmm, that does not seem right. 
I think we should behave in a similar way as git does, which is informing the user that his output collides with an entry of some `.dvcignore` file.
This functionality can be implemented probably after we introduce something like `git check-ignore` for `.dvcignore`. I think there was an issue for that already, but I am unable to find it.
Hello! Update here, I'm on DVC 1.3 and this seems solved, although the message I get,

```console
$ dvc add data
Adding...
ERROR: output 'data' does not exist
```

could be improved, similar to what `git add` uses for files in .gitignore patterns:

```
The following paths are ignored by one of your .gitignore files:
data
Use -f if you really want to add them.
```

And, should we have a `-f`/`--force` option for add as well?

Thanks
@jorgeorpinel Thanks for the update! Indeed, that message could be improved.

> And, should we have a -f/--force option for add as well?

I would rather not do that. We've seen it causing very odd bugs in git, wouldn't want that for dvc unless there are some very good scenarios. Would wait for someone to ask for it.
It seems the conflation of concepts "exists" and "ignored" leads to confusing error messages. In my case one of "run" outputs was by mistake ignored in .dvcignore, but the respective message was confusingly saying the file "does not exist".  Possibly related source: [dvc/tree/local.py#L79](https://github.com/iterative/dvc/blob/12c7013062c89a4f9f75d7459a7ed654f3ac174f/dvc/tree/local.py#L79).
@autayeu hi!
Could you share some info on your setup?
`.dvcignore` contained your `run` output before you actually `dvc run` it, right?
Sure. Attached [dvc.message.repro.zip](https://github.com/iterative/dvc/files/5063241/dvc.message.repro.zip) is a minimum setup to reproduce the issue. To reproduce:

Unpack, change into dvc.message.repro directory and run:
```bash
dvc run -n preprocess -d preprocess.py -o data.txt -o data.log "python -u preprocess.py | tee -i data.log"
```
You'll see:
```text
Running stage 'preprocess' with command:
	python -u preprocess.py | tee -i data.log
screen content
ERROR: output 'data.log' does not exist
```
However:
```bash
(dvc) zzz:dvc.message.repro zzz$ ls -l
total 24
-rw-r--r--  1 zzz  zzz  15 Aug 12 15:45 data.log
-rw-r--r--  1 zzz  zzz  12 Aug 12 15:45 data.txt
-rw-r--r--  1 zzz  zzz  84 Aug 12 15:39 preprocess.py
```
So data.log does exist. A bit unexpectedly, dvc also created empty .gitignore file:
```bash
(dvc) zzz:dvc.message.repro zzz$ ls -l .gitignore
-rw-r--r--  1 zzz  zzz  0 Aug 12 16:05 .gitignore
```

Other relevant outputs:
```bash
(dvc) zzz:dvc.message.repro zzz$ cat .dvcignore
*.log
```
```bash
(qac) zzz:dvc.message.repro zzz$ dvc version
DVC version: 1.4.0+12c701

---------------------------------
Platform: Python 3.8.5 on macOS-10.15.6-x86_64-i386-64bit
Supports: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache types: reflink, hardlink, symlink
Repo: dvc, git
```
@autayeu Thank you! It seems to me that we should rethink the message.

Resolving patch:
diff --git a/dvc/ignore.py b/dvc/ignore.py
--- a/dvc/ignore.py
+++ b/dvc/ignore.py
@@ -152,6 +152,10 @@ def __bool__(self):
 )
 
 
+def _no_match(path):
+    return CheckIgnoreResult(path, False, ["::"])
+
+
 class DvcIgnoreFilterNoop:
     def __init__(self, tree, root_dir):
         pass
@@ -165,8 +169,8 @@ def is_ignored_dir(self, _):
     def is_ignored_file(self, _):
         return False
 
-    def check_ignore(self, _):
-        return []
+    def check_ignore(self, path):
+        return _no_match(path)
 
 
 class DvcIgnoreFilter:
@@ -325,7 +329,7 @@ def check_ignore(self, target):
 
                 if matches:
                     return CheckIgnoreResult(target, True, matches)
-        return CheckIgnoreResult(target, False, ["::"])
+        return _no_match(target)
 
 
 def init(path):
diff --git a/dvc/output/base.py b/dvc/output/base.py
--- a/dvc/output/base.py
+++ b/dvc/output/base.py
@@ -45,6 +45,12 @@ def __init__(self, path):
         super().__init__(f"Stage file '{path}' cannot be an output.")
 
 
+class OutputIsIgnoredError(DvcException):
+    def __init__(self, match):
+        lines = "\n".join(match.patterns)
+        super().__init__(f"Path '{match.file}' is ignored by\n{lines}")
+
+
 class BaseOutput:
     IS_DEPENDENCY = False
 
@@ -77,6 +83,7 @@ class BaseOutput:
     DoesNotExistError = OutputDoesNotExistError
     IsNotFileOrDirError = OutputIsNotFileOrDirError
     IsStageFileError = OutputIsStageFileError
+    IsIgnoredError = OutputIsIgnoredError
 
     sep = "/"
 
@@ -91,7 +98,7 @@ def __init__(
         plot=False,
         persist=False,
     ):
-        self._validate_output_path(path)
+        self._validate_output_path(path, stage)
         # This output (and dependency) objects have too many paths/urls
         # here is a list and comments:
         #
@@ -499,8 +506,13 @@ def get_used_cache(self, **kwargs):
         return ret
 
     @classmethod
-    def _validate_output_path(cls, path):
+    def _validate_output_path(cls, path, stage=None):
         from dvc.dvcfile import is_valid_filename
 
         if is_valid_filename(path):
             raise cls.IsStageFileError(path)
+
+        if stage:
+            check = stage.repo.tree.dvcignore.check_ignore(path)
+            if check.match:
+                raise cls.IsIgnoredError(check)


Test patch:
diff --git a/tests/func/test_ignore.py b/tests/func/test_ignore.py
--- a/tests/func/test_ignore.py
+++ b/tests/func/test_ignore.py
@@ -5,6 +5,7 @@
 
 from dvc.exceptions import DvcIgnoreInCollectedDirError
 from dvc.ignore import DvcIgnore, DvcIgnorePatterns
+from dvc.output.base import OutputIsIgnoredError
 from dvc.path_info import PathInfo
 from dvc.pathspec_math import PatternInfo, merge_patterns
 from dvc.repo import Repo
@@ -400,3 +401,10 @@ def test_ignore_in_added_dir(tmp_dir, dvc):
     dvc.checkout()
 
     assert not ignored_path.exists()
+
+
+def test_ignored_output(tmp_dir, scm, dvc, run_copy):
+    tmp_dir.gen({".dvcignore": "*.log", "foo": "foo content"})
+
+    with pytest.raises(OutputIsIgnoredError):
+        run_copy("foo", "foo.log", name="copy")
diff --git a/tests/unit/output/test_output.py b/tests/unit/output/test_output.py
--- a/tests/unit/output/test_output.py
+++ b/tests/unit/output/test_output.py
@@ -5,6 +5,7 @@
 from voluptuous import MultipleInvalid, Schema
 
 from dvc.cache import NamedCache
+from dvc.ignore import _no_match
 from dvc.output import CHECKSUM_SCHEMA, BaseOutput
 
 
@@ -72,6 +73,11 @@ def test_get_used_cache(exists, expected_message, mocker, caplog):
     stage = mocker.MagicMock()
     mocker.patch.object(stage, "__str__", return_value="stage: 'stage.dvc'")
     mocker.patch.object(stage, "addressing", "stage.dvc")
+    mocker.patch.object(
+        stage.repo.tree.dvcignore,
+        "check_ignore",
+        return_value=_no_match("path"),
+    )
 
     output = BaseOutput(stage, "path")