Repository: iterative/dvc

Issue:
Failure to pull after importing a git repo
**Please provide information about your setup**
DVC version(i.e. `dvc --version`), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

DVC 0.86.4, conda installation, Windows. 

See https://discuss.dvc.org/t/importing-from-a-git-repo-then-pulling/320 for background. 

After `import`ing a directory from a git repository (not a dvc repository), `dvc pull` fails with `ERROR: unexpected error - 'ExternalGitRepo' object has no attribute 'cache'`. 




Discussion:
Script to reproduce:

```
mkdir git_repo
cd git_repo
git init
mkdir a_dir
touch a_dir/test.txt
git add .
git commit -m "Init git repo"

cd -
mkdir dvc_repo
cd dvc_repo
git init
dvc init
git add .
git commit -m "Init"
dvc import ../git_repo a_dir
dvc pull
```
Thanks for the bug report @charlesbaynham ! We'll take a look ASAP.
@jorgeorpinel, @efiop, how should this behave for git repo? Not pull at all or like `dvc update`? Could not find anything on https://man.dvc.org/pull about this.
Am I right in thinking that dvc repos don't put imported files into their cache / remote? So a pull will always need to access the original source? 
@charlesbaynham it actually does put it in the cache. Continuing your script:

```console
$ cat a_dir.dvc
...
  repo:
    url: ../git_repo
    rev_lock: 314393c2a7b82004302d5b7f219e724522db88ca
outs:
- md5: 59446444eae8d3c063f64fefd37b43b5
...
$ tree .dvc/cache 
.dvc/cache
├── 59
│   └── 446444eae8d3c063f64fefd37b43b5.dir
└── d4
    └── 1d8cd98f00b204e9800998ecf8427e
$ cat .dvc/cache/59/446444eae8d3c063f64fefd37b43b5.dir 
[{"md5": "d41d8cd98f00b204e9800998ecf8427e", "relpath": "test.txt"}]
```
But shouldn't push it to the remote storage, indeed.

@skshetry in this case, I don't understand why `dvc pull` is **even trying** to pull from this _import stage_, since the output is already in cache ^ (and workspace).

![image](https://user-images.githubusercontent.com/1477535/75219126-8e5b1700-5761-11ea-8ad9-c64f5db66383.png)
> https://dvc.org/doc/command-reference/pull

I think that's the current `p0` bug.

- [ ] Whether there are other problems with Git-tracked imported files, we would need to extensively Q/A re-importing and updating them... And probably `dvc fetch\pull` and `dvc push` as well, just in case.
> @charlesbaynham it actually does put it in the cache. Continuing your script:

It actually doesn't, just that `.dir` file, which is more of a temporary file than a proper cache in that particular case. So we will indeed need to have access to the original source.


@skshetry It should git clone (as we usually do) and just copy stuff over, same as if you would've re-imported it again.
> It actually doesn't

OK. But in my script you can see that both `.dvc/cache/59/446444e...b5.dir` (a_dir) AND `.dvc/cache/d4/1d8cd98...` (a_dir/test.txt) are in cache... what am I missing? 🙂 
Able to reproduce with following tests:
```python
def test_pull_imports(tmp_dir, dvc, scm, git_dir):
    with git_dir.chdir():
        git_dir.scm_gen("foo", "foo", commit="first version")

    dvc.imp(fspath(git_dir), "foo")
    dvc.pull()

def test_pull_imports_erepo_dvc(tmp_dir, dvc, scm, erepo_dir):
    with erepo_dir.chdir():
        erepo_dir.dvc_gen("foo", "foo", commit="first version")

    dvc.imp(fspath(erepo_dir), "foo")
    dvc.pull()
```

Looks like it never worked before. `dvc import` already puts the file in the cache, and `dvc update` should be used for updating. Nevertheless, I'm trying to see what's going on.

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -468,7 +468,7 @@ def _cache_is_copy(self, path_info):
         self.cache_type_confirmed = True
         return self.cache_types[0] == "copy"
 
-    def _save_dir(self, path_info, checksum):
+    def _save_dir(self, path_info, checksum, save_link=True):
         cache_info = self.checksum_to_path_info(checksum)
         dir_info = self.get_dir_cache(checksum)
 
@@ -479,7 +479,9 @@ def _save_dir(self, path_info, checksum):
             entry_checksum = entry[self.PARAM_CHECKSUM]
             self._save_file(entry_info, entry_checksum, save_link=False)
 
-        self.state.save_link(path_info)
+        if save_link:
+            self.state.save_link(path_info)
+
         self.state.save(cache_info, checksum)
         self.state.save(path_info, checksum)
 
@@ -510,7 +512,7 @@ def walk_files(self, path_info):
     def protect(path_info):
         pass
 
-    def save(self, path_info, checksum_info):
+    def save(self, path_info, checksum_info, save_link=True):
         if path_info.scheme != self.scheme:
             raise RemoteActionNotImplemented(
                 "save {} -> {}".format(path_info.scheme, self.scheme),
@@ -518,15 +520,15 @@ def save(self, path_info, checksum_info):
             )
 
         checksum = checksum_info[self.PARAM_CHECKSUM]
-        self._save(path_info, checksum)
+        self._save(path_info, checksum, save_link)
 
-    def _save(self, path_info, checksum):
+    def _save(self, path_info, checksum, save_link=True):
         to_info = self.checksum_to_path_info(checksum)
         logger.debug("Saving '%s' to '%s'.", path_info, to_info)
         if self.isdir(path_info):
-            self._save_dir(path_info, checksum)
+            self._save_dir(path_info, checksum, save_link)
             return
-        self._save_file(path_info, checksum)
+        self._save_file(path_info, checksum, save_link)
 
     def _handle_transfer_exception(
         self, from_info, to_info, exception, operation
diff --git a/dvc/repo/fetch.py b/dvc/repo/fetch.py
--- a/dvc/repo/fetch.py
+++ b/dvc/repo/fetch.py
@@ -2,9 +2,9 @@
 
 from dvc.cache import NamedCache
 from dvc.config import NoRemoteError
-from dvc.exceptions import DownloadError
-from dvc.exceptions import OutputNotFoundError
+from dvc.exceptions import DownloadError, OutputNotFoundError
 from dvc.scm.base import CloneError
+from dvc.path_info import PathInfo
 
 
 logger = logging.getLogger(__name__)
@@ -69,35 +69,63 @@ def _fetch(
 
 
 def _fetch_external(self, repo_url, repo_rev, files, jobs):
-    from dvc.external_repo import external_repo
+    from dvc.external_repo import external_repo, ExternalRepo
 
-    failed = 0
+    failed, downloaded = 0, 0
     try:
         with external_repo(repo_url, repo_rev) as repo:
-            repo.cache.local.cache_dir = self.cache.local.cache_dir
-
-            with repo.state:
-                cache = NamedCache()
-                for name in files:
-                    try:
-                        out = repo.find_out_by_relpath(name)
-                    except OutputNotFoundError:
-                        failed += 1
-                        logger.exception(
-                            "failed to fetch data for '{}'".format(name)
-                        )
-                        continue
-                    else:
-                        cache.update(out.get_used_cache())
-
-                try:
-                    return repo.cloud.pull(cache, jobs=jobs), failed
-                except DownloadError as exc:
-                    failed += exc.amount
+            is_dvc_repo = isinstance(repo, ExternalRepo)
+            # gather git-only tracked files if dvc repo
+            git_files = [] if is_dvc_repo else files
+            if is_dvc_repo:
+                repo.cache.local.cache_dir = self.cache.local.cache_dir
+                with repo.state:
+                    cache = NamedCache()
+                    for name in files:
+                        try:
+                            out = repo.find_out_by_relpath(name)
+                        except OutputNotFoundError:
+                            # try to add to cache if they are git-tracked files
+                            git_files.append(name)
+                        else:
+                            cache.update(out.get_used_cache())
+
+                        try:
+                            downloaded += repo.cloud.pull(cache, jobs=jobs)
+                        except DownloadError as exc:
+                            failed += exc.amount
+
+            d, f = _git_to_cache(self.cache.local, repo.root_dir, git_files)
+            downloaded += d
+            failed += f
     except CloneError:
         failed += 1
         logger.exception(
             "failed to fetch data for '{}'".format(", ".join(files))
         )
 
-    return 0, failed
+    return downloaded, failed
+
+
+def _git_to_cache(cache, repo_root, files):
+    """Save files from a git repo directly to the cache."""
+    failed = set()
+    num_downloads = 0
+    repo_root = PathInfo(repo_root)
+    for file in files:
+        info = cache.save_info(repo_root / file)
+        if info.get(cache.PARAM_CHECKSUM) is None:
+            failed.add(file)
+            continue
+
+        if cache.changed_cache(info[cache.PARAM_CHECKSUM]):
+            logger.debug("fetched '%s' from '%s' repo", file, repo_root)
+            num_downloads += 1
+            cache.save(repo_root / file, info, save_link=False)
+
+    if failed:
+        logger.exception(
+            "failed to fetch data for {}".format(", ".join(failed))
+        )
+
+    return num_downloads, len(failed)


Test patch:
diff --git a/tests/func/test_data_cloud.py b/tests/func/test_data_cloud.py
--- a/tests/func/test_data_cloud.py
+++ b/tests/func/test_data_cloud.py
@@ -22,7 +22,9 @@
 from dvc.remote import RemoteSSH
 from dvc.remote.base import STATUS_DELETED, STATUS_NEW, STATUS_OK
 from dvc.utils import file_md5
+from dvc.utils.fs import remove
 from dvc.utils.stage import dump_stage_file, load_stage_file
+from dvc.external_repo import clean_repos
 from tests.basic_env import TestDvc
 
 from tests.remotes import (
@@ -709,3 +711,56 @@ def test_verify_checksums(tmp_dir, scm, dvc, mocker, tmp_path_factory):
 
     dvc.pull()
     assert checksum_spy.call_count == 3
+
+
+@pytest.mark.parametrize("erepo", ["git_dir", "erepo_dir"])
+def test_pull_git_imports(request, tmp_dir, dvc, scm, erepo):
+    erepo = request.getfixturevalue(erepo)
+    with erepo.chdir():
+        erepo.scm_gen({"dir": {"bar": "bar"}}, commit="second")
+        erepo.scm_gen("foo", "foo", commit="first")
+
+    dvc.imp(fspath(erepo), "foo")
+    dvc.imp(fspath(erepo), "dir", out="new_dir", rev="HEAD~")
+
+    assert dvc.pull()["downloaded"] == 0
+
+    for item in ["foo", "new_dir", dvc.cache.local.cache_dir]:
+        remove(item)
+    os.makedirs(dvc.cache.local.cache_dir, exist_ok=True)
+    clean_repos()
+
+    assert dvc.pull(force=True)["downloaded"] == 2
+
+    assert (tmp_dir / "foo").exists()
+    assert (tmp_dir / "foo").read_text() == "foo"
+
+    assert (tmp_dir / "new_dir").exists()
+    assert (tmp_dir / "new_dir" / "bar").read_text() == "bar"
+
+
+def test_pull_external_dvc_imports(tmp_dir, dvc, scm, erepo_dir):
+    with erepo_dir.chdir():
+        erepo_dir.dvc_gen({"dir": {"bar": "bar"}}, commit="second")
+        erepo_dir.dvc_gen("foo", "foo", commit="first")
+
+        os.remove("foo")
+        shutil.rmtree("dir")
+
+    dvc.imp(fspath(erepo_dir), "foo")
+    dvc.imp(fspath(erepo_dir), "dir", out="new_dir", rev="HEAD~")
+
+    assert dvc.pull()["downloaded"] == 0
+
+    for item in ["foo", "new_dir", dvc.cache.local.cache_dir]:
+        remove(item)
+    os.makedirs(dvc.cache.local.cache_dir, exist_ok=True)
+    clean_repos()
+
+    assert dvc.pull(force=True)["downloaded"] == 2
+
+    assert (tmp_dir / "foo").exists()
+    assert (tmp_dir / "foo").read_text() == "foo"
+
+    assert (tmp_dir / "new_dir").exists()
+    assert (tmp_dir / "new_dir" / "bar").read_text() == "bar"