Repository: ARMmbed/mbed-tools

Issue:
Missing --depth option for deploy/new commands
### Description

Old tools `mbed-cli` has amazing option --depth for more fast and efficient downloading mbed-os sources, but new tools `mbed-tools` don't have this option.

But this option very useful on limited and slow internet connections. And takes up less space on hdd.

### Issue request type

- [x] Enhancement
- [ ] Bug


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203?src=pr&el=h1) Report
> Merging [#203](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203?src=pr&el=desc) (de0dcbc) into [master](https://codecov.io/gh/ARMmbed/mbed-tools/commit/c4ccaf1baeacf2640b0103dff46db6cb6186ea43?el=desc) (c4ccaf1) will **decrease** coverage by `0.07%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203/graphs/tree.svg?width=650&height=150&src=pr&token=QFRIdRv91G)](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #203      +/-   ##
==========================================
- Coverage   96.57%   96.50%   -0.08%     
==========================================
  Files          94       94              
  Lines        2629     2632       +3     
==========================================
+ Hits         2539     2540       +1     
- Misses         90       92       +2     
```


| [Impacted Files](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/mbed\_tools/project/\_internal/git\_utils.py](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203/diff?src=pr&el=tree#diff-c3JjL21iZWRfdG9vbHMvcHJvamVjdC9faW50ZXJuYWwvZ2l0X3V0aWxzLnB5) | `100.00% <100.00%> (ø)` | |
| [src/mbed\_tools/project/\_internal/libraries.py](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203/diff?src=pr&el=tree#diff-c3JjL21iZWRfdG9vbHMvcHJvamVjdC9faW50ZXJuYWwvbGlicmFyaWVzLnB5) | `98.21% <100.00%> (ø)` | |
| [...ools/devices/\_internal/detect\_candidate\_devices.py](https://codecov.io/gh/ARMmbed/mbed-tools/pull/203/diff?src=pr&el=tree#diff-c3JjL21iZWRfdG9vbHMvZGV2aWNlcy9faW50ZXJuYWwvZGV0ZWN0X2NhbmRpZGF0ZV9kZXZpY2VzLnB5) | `90.00% <0.00%> (-10.00%)` | :arrow_down: |

Patater: Looks like `--branch` isn't able to work with git hashes, possibly. `git clone --branch=26606218ad9d1ee1c8781aa73774fd7ea3a7658e --depth=1 --progress -v https://github.com/ARMmbed/mbed-os mbed-os-example-ble/BLE_GattClient_CharacteristicUpdates/mbed-os` fails with:
```
Cloning into 'mbed-os-example-ble/BLE_GattClient_CharacteristicUpdates/mbed-os'...
warning: Could not find remote branch 26606218ad9d1ee1c8781aa73774fd7ea3a7658e to clone.
fatal: Remote branch 26606218ad9d1ee1c8781aa73774fd7ea3a7658e not found in upstream origin
```

So, a bit of work to do still
Patater: Unfortunately, many of our lib files use explicit git hashes, instead of tags. We can speed up cloning for everybody by shipping our examples and other software with explicit tags, but the tools will try their best to keep things shallow where possible until use of tags is more widespread.
ladislas: Maybe libs and example are not as heavy as mbed os itself. So it's not really an issue for those. 
Patater: > Maybe libs and example are not as heavy as mbed os itself. So it's not really an issue for those.

Correct. The main issue is `mbed-os.lib` contained in many of our exampes. However, with `mbed-os`, even a shallow clone takes up around 650 MB or more. This is less than 2.5 GB, but still substantial.

Patch:
diff --git a/.travis.yml b/.travis.yml
index 1995da5..77b83b0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -197,7 +197,8 @@ matrix:
         - mbedtools import mbed-os-example-ble
         - cd mbed-os-example-ble
         # Checkout the development branch on BLE example
-        - git checkout development
+        - git fetch origin development
+        - git checkout FETCH_HEAD
         - cd BLE_Advertising
         - mbedtools deploy
         - mbedtools compile -t GCC_ARM -m K64F
diff --git a/news/20210224162840.feature b/news/20210224162840.feature
new file mode 100644
index 0000000..e877386
--- /dev/null
+++ b/news/20210224162840.feature
@@ -0,0 +1,2 @@
+Speed up obtaining library dependencies by obtaining only the requested library
+revision and no other revision.
diff --git a/src/mbed_tools/project/_internal/git_utils.py b/src/mbed_tools/project/_internal/git_utils.py
index 89e2898..a84d582 100644
--- a/src/mbed_tools/project/_internal/git_utils.py
+++ b/src/mbed_tools/project/_internal/git_utils.py
@@ -7,9 +7,13 @@ from dataclasses import dataclass
 from pathlib import Path
 
 import git
+import logging
 
 from mbed_tools.project.exceptions import VersionControlError
 from mbed_tools.project._internal.progress import ProgressReporter
+from typing import Optional
+
+logger = logging.getLogger(__name__)
 
 
 @dataclass
@@ -25,21 +29,31 @@ class GitReference:
     ref: str
 
 
-def clone(url: str, dst_dir: Path) -> git.Repo:
+def clone(url: str, dst_dir: Path, ref: Optional[str] = None, depth: int = 1) -> git.Repo:
     """Clone a library repository.
 
     Args:
         url: URL of the remote to clone.
         dst_dir: Destination directory for the cloned repo.
+        ref: An optional git commit hash, branch or tag reference to checkout
+        depth: Truncate history to the specified number of commits. Defaults to
+               1, to make a shallow clone.
 
     Raises:
         VersionControlError: Cloning the repository failed.
     """
+    # Gitpython doesn't propagate the git error message when a repo is already
+    # cloned, so we cannot depend on git to handle the "already cloned" error.
+    # We must handle this ourselves instead.
     if dst_dir.exists() and list(dst_dir.glob("*")):
         raise VersionControlError(f"{dst_dir} exists and is not an empty directory.")
 
+    clone_from_kwargs = {"url": url, "to_path": str(dst_dir), "progress": ProgressReporter(name=url), "depth": depth}
+    if ref:
+        clone_from_kwargs["branch"] = ref
+
     try:
-        return git.Repo.clone_from(url, str(dst_dir), progress=ProgressReporter(name=url))
+        return git.Repo.clone_from(**clone_from_kwargs)
     except git.exc.GitCommandError as err:
         raise VersionControlError(f"Cloning git repository from url '{url}' failed. Error from VCS: {err}")
 
@@ -61,6 +75,22 @@ def checkout(repo: git.Repo, ref: str, force: bool = False) -> None:
         raise VersionControlError(f"Failed to check out revision '{ref}'. Error from VCS: {err}")
 
 
+def fetch(repo: git.Repo, ref: str) -> None:
+    """Fetch from the repo's origin.
+
+    Args:
+        repo: git.Repo object where the checkout will be performed.
+        ref: Git commit hash, branch or tag reference, must be a valid ref defined in the repo.
+
+    Raises:
+        VersionControlError: Fetch failed.
+    """
+    try:
+        repo.git.fetch("origin", ref)
+    except git.exc.GitCommandError as err:
+        raise VersionControlError(f"Failed to fetch. Error from VCS: {err}")
+
+
 def init(path: Path) -> git.Repo:
     """Initialise a git repository at the given path.
 
diff --git a/src/mbed_tools/project/_internal/libraries.py b/src/mbed_tools/project/_internal/libraries.py
index 2fa89ef..8c3f1a9 100644
--- a/src/mbed_tools/project/_internal/libraries.py
+++ b/src/mbed_tools/project/_internal/libraries.py
@@ -10,6 +10,7 @@ from pathlib import Path
 from typing import Generator, List
 
 from mbed_tools.project._internal import git_utils
+from mbed_tools.project.exceptions import VersionControlError
 
 logger = logging.getLogger(__name__)
 
@@ -61,10 +62,7 @@ class LibraryReferences:
         for lib in self.iter_unresolved():
             git_ref = lib.get_git_reference()
             logger.info(f"Resolving library reference {git_ref.repo_url}.")
-            repo = git_utils.clone(git_ref.repo_url, lib.source_code_path)
-            if git_ref.ref:
-                logger.info(f"Checking out revision {git_ref.ref} for library {git_ref.repo_url}.")
-                git_utils.checkout(repo, git_ref.ref)
+            _clone_at_ref(git_ref.repo_url, lib.source_code_path, git_ref.ref)
 
         # Check if we find any new references after cloning dependencies.
         if list(self.iter_unresolved()):
@@ -77,13 +75,10 @@ class LibraryReferences:
             git_ref = lib.get_git_reference()
 
             if not git_ref.ref:
-                repo.git.fetch()
                 git_ref.ref = git_utils.get_default_branch(repo)
-            else:
-                # Fetch only the requested ref
-                repo.git.fetch("origin", git_ref.ref)
 
-            git_utils.checkout(repo, git_ref.ref, force=force)
+            git_utils.fetch(repo, git_ref.ref)
+            git_utils.checkout(repo, "FETCH_HEAD", force=force)
 
     def iter_all(self) -> Generator[MbedLibReference, None, None]:
         """Iterate all library references in the tree.
@@ -118,3 +113,21 @@ class LibraryReferences:
     def _in_ignore_path(self, lib_reference_path: Path) -> bool:
         """Check if a library reference is in a path we want to ignore."""
         return any(p in lib_reference_path.parts for p in self.ignore_paths)
+
+
+def _clone_at_ref(url: str, path: Path, ref: str) -> None:
+    if ref:
+        logger.info(f"Checking out revision {ref} for library {url}.")
+        try:
+            git_utils.clone(url, path, ref)
+        except VersionControlError:
+            # We weren't able to clone. Try again without the ref.
+            repo = git_utils.clone(url, path)
+            # We couldn't clone the ref and had to fall back to cloning
+            # just the default branch. Fetch the ref before checkout, so
+            # that we have it available locally.
+            logger.warning(f"No tag or branch with name {ref}. Fetching full repository.")
+            git_utils.fetch(repo, ref)
+            git_utils.checkout(repo, "FETCH_HEAD")
+    else:
+        git_utils.clone(url, path)


Test patch:
diff --git a/tests/project/_internal/test_git_utils.py b/tests/project/_internal/test_git_utils.py
index b280359..c188d47 100644
--- a/tests/project/_internal/test_git_utils.py
+++ b/tests/project/_internal/test_git_utils.py
@@ -30,12 +30,33 @@ class TestClone:
         repo = git_utils.clone(url, path)
 
         assert repo is not None
-        mock_repo.clone_from.assert_called_once_with(url, str(path), progress=mock_progress())
+        mock_repo.clone_from.assert_called_once_with(url=url, to_path=str(path), progress=mock_progress(), depth=1)
+
+    def test_returns_repo_for_ref(self, mock_progress, mock_repo, tmp_path):
+        url = "https://example.com/org/repo.git"
+        ref = "development"
+        path = Path(tmp_path, "repo")
+        repo = git_utils.clone(url, path, ref)
+
+        assert repo is not None
+        mock_repo.clone_from.assert_called_once_with(
+            url=url, to_path=str(path), progress=mock_progress(), depth=1, branch=ref
+        )
 
     def test_raises_when_fails_due_to_bad_url(self, tmp_path):
         with pytest.raises(VersionControlError, match="from url 'bad' failed"):
             git_utils.clone("bad", Path(tmp_path, "dst"))
 
+    def test_raises_when_fails_due_to_bad_url_with_ref(self, mock_progress, mock_repo, tmp_path):
+        url = "https://example.com/org/repo.git"
+        ref = "development"
+        path = Path(tmp_path, "repo")
+
+        mock_repo.clone_from.side_effect = git_utils.git.exc.GitCommandError("git clone", 255)
+
+        with pytest.raises(VersionControlError, match=f"Cloning git repository from url '{url}' failed."):
+            git_utils.clone(url, path, ref)
+
     def test_raises_when_fails_due_to_existing_nonempty_dst_dir(self, mock_repo, tmp_path):
         dst_dir = Path(tmp_path, "dst")
         dst_dir.mkdir()
@@ -52,7 +73,7 @@ class TestClone:
         repo = git_utils.clone(url, dst_dir)
 
         assert repo is not None
-        mock_repo.clone_from.assert_called_once_with(url, str(dst_dir), progress=mock_progress())
+        mock_repo.clone_from.assert_called_once_with(url=url, to_path=str(dst_dir), progress=mock_progress(), depth=1)
 
 
 class TestInit:
@@ -100,6 +121,21 @@ class TestCheckout:
             git_utils.checkout(mock_repo, "bad")
 
 
+class TestFetch:
+    def test_does_a_fetch(self, mock_repo):
+        ref = "b23a8eb1c3f80292c8eb40689106759fae83a4c6"
+        git_utils.fetch(mock_repo, ref)
+
+        mock_repo.git.fetch.assert_called_once_with("origin", ref)
+
+    def test_raises_when_fetch_fails(self, mock_repo):
+        ref = "v2.7.9"
+        mock_repo.git.fetch.side_effect = git_utils.git.exc.GitCommandError("git fetch", 255)
+
+        with pytest.raises(VersionControlError):
+            git_utils.fetch(mock_repo, ref)
+
+
 class TestGetDefaultBranch:
     def test_returns_default_branch_name(self, mock_repo):
         mock_repo().git.symbolic_ref.return_value = "refs/remotes/origin/main"
diff --git a/tests/project/_internal/test_libraries.py b/tests/project/_internal/test_libraries.py
index e6b154d..fbff350 100644
--- a/tests/project/_internal/test_libraries.py
+++ b/tests/project/_internal/test_libraries.py
@@ -9,6 +9,7 @@ import pytest
 from unittest import mock
 
 from mbed_tools.project._internal.libraries import MbedLibReference, LibraryReferences
+from mbed_tools.project.exceptions import VersionControlError
 from tests.project.factories import make_mbed_lib_reference
 
 
@@ -24,6 +25,12 @@ def mock_checkout():
         yield checkout
 
 
+@pytest.fixture
+def mock_fetch():
+    with mock.patch("mbed_tools.project._internal.git_utils.fetch") as fetch:
+        yield fetch
+
+
 @pytest.fixture
 def mock_get_repo():
     with mock.patch("mbed_tools.project._internal.git_utils.get_repo") as get_repo:
@@ -46,7 +53,7 @@ class TestLibraryReferences:
     def test_hydrates_top_level_library_references(self, mock_clone, tmp_path):
         fs_root = pathlib.Path(tmp_path, "foo")
         lib = make_mbed_lib_reference(fs_root, ref_url="https://git")
-        mock_clone.side_effect = lambda url, dst_dir: dst_dir.mkdir()
+        mock_clone.side_effect = lambda url, dst_dir, *args: dst_dir.mkdir()
 
         lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
         lib_refs.fetch()
@@ -64,7 +71,7 @@ class TestLibraryReferences:
         )
         # Here we mock the effects of a recursive reference lookup. We create a new lib reference as a side effect of
         # the first call to the mock. Then we create the src dir, thus resolving the lib, on the second call.
-        mock_clone.side_effect = lambda url, dst_dir: (
+        mock_clone.side_effect = lambda url, dst_dir, *args: (
             make_mbed_lib_reference(pathlib.Path(dst_dir), name=lib2.reference_file.name, ref_url="https://valid2"),
             lib2.source_code_path.mkdir(),
         )
@@ -76,7 +83,7 @@ class TestLibraryReferences:
         assert lib2.is_resolved()
 
     def test_does_perform_checkout_of_default_repo_branch_if_no_git_ref_exists(
-        self, mock_get_repo, mock_checkout, mock_get_default_branch, mock_clone, tmp_path
+        self, mock_get_repo, mock_checkout, mock_fetch, mock_get_default_branch, mock_clone, tmp_path
     ):
         fs_root = pathlib.Path(tmp_path, "foo")
         make_mbed_lib_reference(fs_root, ref_url="https://git", resolved=True)
@@ -84,38 +91,95 @@ class TestLibraryReferences:
         lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
         lib_refs.checkout(force=False)
 
-        mock_checkout.assert_called_once_with(mock_get_repo(), mock_get_default_branch(), force=False)
+        mock_fetch.assert_called_once_with(mock_get_repo(), mock_get_default_branch())
+        mock_checkout.assert_called_once_with(mock_get_repo(), "FETCH_HEAD", force=False)
 
-    def test_performs_checkout_if_git_ref_exists(self, mock_get_repo, mock_checkout, mock_clone, tmp_path):
+    def test_performs_checkout_if_git_ref_exists(self, mock_get_repo, mock_checkout, mock_fetch, mock_clone, tmp_path):
         fs_root = pathlib.Path(tmp_path, "foo")
         lib = make_mbed_lib_reference(fs_root, ref_url="https://git#lajdhalk234", resolved=True)
 
         lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
         lib_refs.checkout(force=False)
 
-        mock_checkout.assert_called_once_with(mock_get_repo.return_value, lib.get_git_reference().ref, force=False)
+        mock_fetch.assert_called_once_with(mock_get_repo(), lib.get_git_reference().ref)
+        mock_checkout.assert_called_once_with(mock_get_repo.return_value, "FETCH_HEAD", force=False)
 
     def test_fetch_does_not_perform_checkout_if_no_git_ref_exists(
-        self, mock_get_repo, mock_checkout, mock_clone, tmp_path
+        self, mock_get_repo, mock_checkout, mock_fetch, mock_clone, tmp_path
     ):
         fs_root = pathlib.Path(tmp_path, "foo")
         make_mbed_lib_reference(fs_root, ref_url="https://git")
-        mock_clone.side_effect = lambda url, dst_dir: dst_dir.mkdir()
+        mock_clone.side_effect = lambda url, dst_dir, *args: dst_dir.mkdir()
 
         lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
         lib_refs.fetch()
 
+        mock_fetch.assert_not_called()
         mock_checkout.assert_not_called()
 
-    def test_fetch_performs_checkout_if_git_ref_exists(self, mock_get_repo, mock_checkout, mock_clone, tmp_path):
+    def test_fetch_performs_checkout_if_ref_is_hash(
+        self, mock_get_repo, mock_clone, mock_fetch, mock_checkout, tmp_path
+    ):
+        num_times_called = 0
+
+        def clone_side_effect(url, dst_dir, *args):
+            nonlocal num_times_called
+            if num_times_called == 0:
+                num_times_called += 1
+                raise VersionControlError("Failed to clone")
+            elif num_times_called == 1:
+                num_times_called += 1
+                dst_dir.mkdir()
+            else:
+                assert False
+
         fs_root = pathlib.Path(tmp_path, "foo")
-        lib = make_mbed_lib_reference(fs_root, ref_url="https://git#lajdhalk234")
-        mock_clone.side_effect = lambda url, dst_dir: dst_dir.mkdir()
+        lib = make_mbed_lib_reference(fs_root, ref_url="https://git#398bc1a63370")
+        mock_clone.side_effect = clone_side_effect
 
         lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
         lib_refs.fetch()
 
-        mock_checkout.assert_called_once_with(None, lib.get_git_reference().ref)
+        mock_clone.assert_called_with(lib.get_git_reference().repo_url, lib.source_code_path)
+        mock_fetch.assert_called_once_with(None, lib.get_git_reference().ref)
+        mock_checkout.assert_called_once_with(None, "FETCH_HEAD")
+
+    def test_raises_when_no_such_ref(self, mock_repo, mock_clone, mock_fetch, mock_checkout, tmp_path):
+        num_times_called = 0
+
+        def clone_side_effect(url, dst_dir, *args):
+            nonlocal num_times_called
+            if num_times_called == 0:
+                num_times_called += 1
+                raise VersionControlError("Failed to clone")
+            elif num_times_called == 1:
+                num_times_called += 1
+                dst_dir.mkdir()
+            else:
+                assert False
+
+        fs_root = pathlib.Path(tmp_path, "foo")
+        make_mbed_lib_reference(fs_root, ref_url="https://git#lajdhalk234")
+
+        mock_clone.side_effect = clone_side_effect
+        mock_fetch.side_effect = None
+        mock_checkout.side_effect = VersionControlError("Failed to checkout")
+
+        with pytest.raises(VersionControlError, match="Failed to checkout"):
+            lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
+            lib_refs.fetch()
+
+    def test_doesnt_fetch_for_branch_or_tag(self, mock_clone, mock_fetch, mock_checkout, tmp_path):
+        fs_root = pathlib.Path(tmp_path, "foo")
+        make_mbed_lib_reference(fs_root, ref_url="https://git#lajdhalk234")
+
+        mock_clone.side_effect = lambda url, dst_dir, *args: dst_dir.mkdir()
+
+        lib_refs = LibraryReferences(fs_root, ignore_paths=["mbed-os"])
+        lib_refs.fetch()
+
+        mock_fetch.assert_not_called()
+        mock_checkout.assert_not_called()
 
     def test_does_not_resolve_references_in_ignore_paths(self, mock_get_repo, mock_checkout, mock_clone, tmp_path):
         fs_root = pathlib.Path(tmp_path, "mbed-os")