Repository: iterative/dvc
Pull request #9675 (2023-06-30 04:25:41)

repro: make stages run in order of the graph
This PR closes https://github.com/iterative/dvc/issues/9367, and makes a few other changes:

- `--single-item` now runs in the order the target is provided, instead of the position in it's dag. So, if you provide `dvc repro stage1 stage2 --single-item`, `stage1` will run before `stage2` even if `stage2` comes before `stage1`. Because that's what you want most of the time.
(And I did not want to complicate the code too much.)
- `--downstream` stages will run in the order of the graph instead of running stages of each target with post-order traversal.

The order is also enforced in normal cases, not just in `--downstream` by traversing the graph in post order. Previously, we were removing duplicates manually. 

https://github.com/iterative/dvc/blob/36ec25ffd78a23bb718036419eb28062edee6177/dvc/repo/reproduce.py#L222-L228

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/iterative/dvc/pull/9675?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) Report
Patch coverage: **`96.87`**% and project coverage change: **`-0.10`** :warning:
> Comparison is base [(`36ec25f`)](https://app.codecov.io/gh/iterative/dvc/commit/36ec25ffd78a23bb718036419eb28062edee6177?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) 90.36% compared to head [(`9c6b5e7`)](https://app.codecov.io/gh/iterative/dvc/pull/9675?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) 90.27%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #9675      +/-   ##
==========================================
- Coverage   90.36%   90.27%   -0.10%     
==========================================
  Files         472      472              
  Lines       36280    36272       -8     
  Branches     5218     3342    -1876     
==========================================
- Hits        32784    32743      -41     
- Misses       2887     2909      +22     
- Partials      609      620      +11     
```


| [Impacted Files](https://app.codecov.io/gh/iterative/dvc/pull/9675?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative) | Coverage Δ | |
|---|---|---|
| [dvc/repo/reproduce.py](https://app.codecov.io/gh/iterative/dvc/pull/9675?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative#diff-ZHZjL3JlcG8vcmVwcm9kdWNlLnB5) | `97.95% <94.44%> (+1.69%)` | :arrow_up: |
| [dvc/repo/graph.py](https://app.codecov.io/gh/iterative/dvc/pull/9675?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative#diff-ZHZjL3JlcG8vZ3JhcGgucHk=) | `92.50% <100.00%> (+1.45%)` | :arrow_up: |

... and [29 files with indirect coverage changes](https://app.codecov.io/gh/iterative/dvc/pull/9675/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/iterative/dvc/pull/9675?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=iterative).

shcheklein: Let's check and update the docs for this please. I think we had one or more descriptions of the execution order. Plus we need to check the option descriptions.
skshetry: Nothing should change from the user’s perspective here, at least from what’s documented. This is mostly a bugfix.
shcheklein: Asair we mention that we don't guarantee the order of execution, i might be wrong though.

Patch:
diff --git a/.github/workflows/packages.yaml b/.github/workflows/build.yaml
similarity index 50%
rename from .github/workflows/packages.yaml
rename to .github/workflows/build.yaml
index 1448edbc3..78c9e34cd 100644
--- a/.github/workflows/packages.yaml
+++ b/.github/workflows/build.yaml
@@ -1,4 +1,4 @@
-name: Build packages
+name: Build package
 on:
   push:
     branches:
@@ -7,39 +7,35 @@ on:
     types: [released, prereleased]
   workflow_dispatch:
 
-permissions:  # added using https://github.com/step-security/secure-workflows
+permissions:
   contents: read
+  id-token: write
 
 jobs:
   pip:
-    runs-on: ubuntu-20.04
-    permissions:
-      id-token: write
+    runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
       with:
         fetch-depth: 0
 
-    - name: Set up Python 3.8
-      uses: actions/setup-python@v4
+    - uses: actions/setup-python@v4
       with:
         python-version: 3.8
 
-    - name: Install
-      run: pip install --upgrade pip wheel
-
-    - name: Force version for Test PyPI uploads
+    - run: python -m pip install --upgrade pip setuptools_scm build twine
+    - name: Force version for Test PyPI upload
       if: ${{ !startsWith(github.ref, 'refs/tags') }}
-      run: |
-        pip install setuptools_scm
-        echo version=$(python -m setuptools_scm | awk -F+ '{print $1}' | tail -1) >> $GITHUB_ENV
+      run: echo version=$(python -m setuptools_scm | awk -F+ '{print $1}' | tail -1) >> $GITHUB_ENV
 
-    - name: Build packages
-      run: ./scripts/build_package.sh
+    - run: |
+        echo 'PKG = "pip"'>dvc/_build.py
+        python -m build
       env:
         SETUPTOOLS_SCM_PRETEND_VERSION: ${{ env.version }}
+    - run: twine check --strict dist/*
 
-    - name: Publish packages to PyPI
+    - name: Publish to PyPI
       if: github.event_name == 'release'
       uses: pypa/gh-action-pypi-publish@release/v1
 
@@ -47,5 +43,5 @@ jobs:
       if: ${{ github.event_name == 'release' || (github.event_name == 'push' && github.ref == 'refs/heads/main') }}
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        repository_url: https://test.pypi.org/legacy/
-        skip_existing: true
+        repository-url: https://test.pypi.org/legacy/
+        skip-existing: true
diff --git a/dvc/__pyinstaller/__init__.py b/dvc/__pyinstaller/__init__.py
new file mode 100644
index 000000000..1c52aadf4
--- /dev/null
+++ b/dvc/__pyinstaller/__init__.py
@@ -0,0 +1,5 @@
+import os
+
+
+def get_hook_dirs():
+    return [os.path.dirname(__file__)]
diff --git a/scripts/pyinstaller/hooks/hook-asyncssh.py b/dvc/__pyinstaller/hook-asyncssh.py
similarity index 64%
rename from scripts/pyinstaller/hooks/hook-asyncssh.py
rename to dvc/__pyinstaller/hook-asyncssh.py
index daee85e09..c608e11b2 100644
--- a/scripts/pyinstaller/hooks/hook-asyncssh.py
+++ b/dvc/__pyinstaller/hook-asyncssh.py
@@ -1,1 +1,2 @@
+# ruff: noqa: N999
 hiddenimports = ["win32timezone"]
diff --git a/scripts/pyinstaller/hooks/hook-celery.py b/dvc/__pyinstaller/hook-celery.py
similarity index 96%
rename from scripts/pyinstaller/hooks/hook-celery.py
rename to dvc/__pyinstaller/hook-celery.py
index 4b96c9a70..6e106be0c 100644
--- a/scripts/pyinstaller/hooks/hook-celery.py
+++ b/dvc/__pyinstaller/hook-celery.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 # pylint: disable=import-error
 from PyInstaller.utils.hooks import collect_submodules, is_module_or_submodule
 
diff --git a/scripts/pyinstaller/hooks/hook-dvc.py b/dvc/__pyinstaller/hook-dvc.py
similarity index 97%
rename from scripts/pyinstaller/hooks/hook-dvc.py
rename to dvc/__pyinstaller/hook-dvc.py
index 3c84707b8..c2d12ff34 100644
--- a/scripts/pyinstaller/hooks/hook-dvc.py
+++ b/dvc/__pyinstaller/hook-dvc.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 from PyInstaller.utils.hooks import copy_metadata  # pylint:disable=import-error
 
 # needed for `dvc doctor` to show dep versions
diff --git a/scripts/pyinstaller/hooks/hook-dvc.system.py b/dvc/__pyinstaller/hook-dvc.system.py
similarity index 77%
rename from scripts/pyinstaller/hooks/hook-dvc.system.py
rename to dvc/__pyinstaller/hook-dvc.system.py
index 41afd5bf6..1b35a072b 100644
--- a/scripts/pyinstaller/hooks/hook-dvc.system.py
+++ b/dvc/__pyinstaller/hook-dvc.system.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 import os
 
 if os.name == "nt":
diff --git a/scripts/pyinstaller/hooks/hook-dvc.tree.gs.py b/dvc/__pyinstaller/hook-dvc.tree.gs.py
similarity index 94%
rename from scripts/pyinstaller/hooks/hook-dvc.tree.gs.py
rename to dvc/__pyinstaller/hook-dvc.tree.gs.py
index 8c36c8471..cd101983b 100644
--- a/scripts/pyinstaller/hooks/hook-dvc.tree.gs.py
+++ b/dvc/__pyinstaller/hook-dvc.tree.gs.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 from PyInstaller.utils.hooks import copy_metadata  # pylint:disable=import-error
 
 datas = copy_metadata("google-cloud-storage")
diff --git a/scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py b/dvc/__pyinstaller/hook-dvc.utils.flatten.py
similarity index 86%
rename from scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py
rename to dvc/__pyinstaller/hook-dvc.utils.flatten.py
index bc23f359f..8d5cea5a1 100644
--- a/scripts/pyinstaller/hooks/hook-dvc.utils.flatten.py
+++ b/dvc/__pyinstaller/hook-dvc.utils.flatten.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 from PyInstaller.utils.hooks import copy_metadata  # pylint:disable=import-error
 
 datas = copy_metadata("flatten-dict")
diff --git a/scripts/pyinstaller/hooks/hook-dvc_task.py b/dvc/__pyinstaller/hook-dvc_task.py
similarity index 87%
rename from scripts/pyinstaller/hooks/hook-dvc_task.py
rename to dvc/__pyinstaller/hook-dvc_task.py
index 6a46c85dd..6f6972b88 100644
--- a/scripts/pyinstaller/hooks/hook-dvc_task.py
+++ b/dvc/__pyinstaller/hook-dvc_task.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 # pylint: disable=import-error
 from PyInstaller.utils.hooks import collect_submodules
 
diff --git a/scripts/pyinstaller/hooks/hook-fsspec.py b/dvc/__pyinstaller/hook-fsspec.py
similarity index 72%
rename from scripts/pyinstaller/hooks/hook-fsspec.py
rename to dvc/__pyinstaller/hook-fsspec.py
index 4ffcf6f74..9e0b58294 100644
--- a/scripts/pyinstaller/hooks/hook-fsspec.py
+++ b/dvc/__pyinstaller/hook-fsspec.py
@@ -1,1 +1,2 @@
+# ruff: noqa: N999
 hiddenimports = ["fsspec.implementations.memory"]
diff --git a/scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py b/dvc/__pyinstaller/hook-google_compute_engine.logger.py
similarity index 89%
rename from scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py
rename to dvc/__pyinstaller/hook-google_compute_engine.logger.py
index 9cc2aec62..d7c29559c 100644
--- a/scripts/pyinstaller/hooks/hook-google_compute_engine.logger.py
+++ b/dvc/__pyinstaller/hook-google_compute_engine.logger.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 from PyInstaller.utils.hooks import copy_metadata  # pylint:disable=import-error
 
 datas = copy_metadata("google-compute-engine")
diff --git a/scripts/pyinstaller/hooks/hook-pydrive2.py b/dvc/__pyinstaller/hook-pydrive2.py
similarity index 85%
rename from scripts/pyinstaller/hooks/hook-pydrive2.py
rename to dvc/__pyinstaller/hook-pydrive2.py
index 6315db70b..55f5d5fcc 100644
--- a/scripts/pyinstaller/hooks/hook-pydrive2.py
+++ b/dvc/__pyinstaller/hook-pydrive2.py
@@ -1,3 +1,4 @@
+# ruff: noqa: N999
 from PyInstaller.utils.hooks import copy_metadata  # pylint:disable=import-error
 
 datas = copy_metadata("pydrive2")
diff --git a/dvc/dagascii.py b/dvc/dagascii.py
index 7b3a984fa..56ffc8dfd 100644
--- a/dvc/dagascii.py
+++ b/dvc/dagascii.py
@@ -209,6 +209,30 @@ def draw(vertices, edges):
     Args:
         vertices (list): list of graph vertices.
         edges (list): list of graph edges.
+
+    Returns:
+        str: ASCII representation
+
+    Example:
+        >>> from dvc.dagascii import draw
+        >>> vertices = [1, 2, 3, 4]
+        >>> edges = [(1, 2), (2, 3), (2, 4), (1, 4)]
+        >>> print(draw(vertices, edges))
+        +---+     +---+
+        | 3 |     | 4 |
+        +---+    *+---+
+          *    **   *
+          *  **     *
+          * *       *
+        +---+       *
+        | 2 |      *
+        +---+     *
+             *    *
+              *  *
+               **
+             +---+
+             | 1 |
+             +---+
     """
     # pylint: disable=too-many-locals
     # NOTE: coordinates might me negative, so we need to shift
diff --git a/dvc/repo/fetch.py b/dvc/repo/fetch.py
index c29a088b2..7a061c5cc 100644
--- a/dvc/repo/fetch.py
+++ b/dvc/repo/fetch.py
@@ -1,6 +1,8 @@
 import logging
+from typing import Dict, List, Tuple
 
 from dvc.exceptions import DownloadError
+from dvc_data.index import DataIndex, FileStorage
 
 from . import locked
 
@@ -84,6 +86,7 @@ def fetch(  # noqa: C901, PLR0913
         data = collect(
             indexes, cache_index=self.data_index, cache_key=cache_key, callback=cb
         )
+    failed_count += _log_unversioned(data)
 
     with Callback.as_tqdm_callback(
         desc="Fetching",
@@ -109,3 +112,31 @@ def fetch(  # noqa: C901, PLR0913
         raise DownloadError(failed_count)
 
     return transferred_count
+
+
+def _log_unversioned(data: Dict[Tuple[str, str], "DataIndex"]) -> int:
+    unversioned: List[str] = []
+    for by_fs, fs_index in data.items():
+        remote = fs_index.storage_map[()].remote
+        if not isinstance(remote, FileStorage) or not remote.fs.version_aware:
+            continue
+
+        fs = remote.fs
+        index = DataIndex()
+        index.storage_map = fs_index.storage_map
+        for key, entry in fs_index.iteritems():
+            if entry.meta and not entry.meta.isdir and entry.meta.version_id is None:
+                unversioned.append(fs.unstrip_protocol(fs.path.join(remote.path, *key)))
+            else:
+                index[key] = entry
+        fs_index.close()
+        data[by_fs] = index
+    if unversioned:
+        logger.warning(
+            (
+                "Some files are missing cloud version information and will not be "
+                "fetched from the remote:\n%s"
+            ),
+            "\n".join(unversioned),
+        )
+    return len(unversioned)
diff --git a/dvc/repo/graph.py b/dvc/repo/graph.py
index b19677503..83f6c99b9 100644
--- a/dvc/repo/graph.py
+++ b/dvc/repo/graph.py
@@ -1,4 +1,4 @@
-from typing import TYPE_CHECKING, Iterator, List, Set
+from typing import TYPE_CHECKING, Any, Iterator, List, Optional, Set, TypeVar
 
 from dvc.fs import localfs
 from dvc.utils.fs import path_isin
@@ -8,6 +8,8 @@ if TYPE_CHECKING:
 
     from dvc.stage import Stage
 
+T = TypeVar("T")
+
 
 def check_acyclic(graph: "DiGraph") -> None:
     import networkx as nx
@@ -39,6 +41,30 @@ def get_pipelines(graph: "DiGraph"):
     return [graph.subgraph(c).copy() for c in nx.weakly_connected_components(graph)]
 
 
+def get_subgraph_of_nodes(
+    graph: "DiGraph", sources: Optional[List[Any]] = None, downstream: bool = False
+) -> "DiGraph":
+    from networkx import dfs_postorder_nodes, reverse_view
+
+    if not sources:
+        return graph
+
+    g = reverse_view(graph) if downstream else graph
+    nodes = []
+    for source in sources:
+        nodes.extend(dfs_postorder_nodes(g, source))
+    return graph.subgraph(nodes)
+
+
+def get_steps(
+    graph: "DiGraph", sources: Optional[List[T]] = None, downstream: bool = False
+) -> List[T]:
+    from networkx import dfs_postorder_nodes
+
+    sub = get_subgraph_of_nodes(graph, sources, downstream=downstream)
+    return list(dfs_postorder_nodes(sub))
+
+
 def collect_pipeline(stage: "Stage", graph: "DiGraph") -> Iterator["Stage"]:
     import networkx as nx
 
diff --git a/dvc/repo/pull.py b/dvc/repo/pull.py
index dbb75261b..66c628225 100644
--- a/dvc/repo/pull.py
+++ b/dvc/repo/pull.py
@@ -1,5 +1,6 @@
 import logging
 
+from dvc.exceptions import CheckoutError
 from dvc.repo import locked
 from dvc.utils import glob_targets
 
@@ -38,13 +39,17 @@ def pull(  # noqa: PLR0913
         recursive=recursive,
         run_cache=run_cache,
     )
-    stats = self.checkout(
-        targets=expanded_targets,
-        with_deps=with_deps,
-        force=force,
-        recursive=recursive,
-        allow_missing=allow_missing,
-    )
+    try:
+        stats = self.checkout(
+            targets=expanded_targets,
+            with_deps=with_deps,
+            force=force,
+            recursive=recursive,
+            allow_missing=allow_missing,
+        )
+    except CheckoutError as exc:
+        exc.stats["fetched"] = processed_files_count
+        raise
 
     stats["fetched"] = processed_files_count
     return stats
diff --git a/dvc/repo/reproduce.py b/dvc/repo/reproduce.py
index 527adcc99..673a741da 100644
--- a/dvc/repo/reproduce.py
+++ b/dvc/repo/reproduce.py
@@ -1,5 +1,7 @@
 import logging
-from typing import TYPE_CHECKING, Iterator, List
+from typing import TYPE_CHECKING, Iterator, List, cast
+
+from funcy import ldistinct
 
 from dvc.exceptions import ReproductionError
 from dvc.repo.scm_context import scm_context
@@ -8,6 +10,8 @@ from dvc.stage.cache import RunCacheNotSupported
 from . import locked
 
 if TYPE_CHECKING:
+    from networkx import DiGraph
+
     from dvc.stage import Stage
 
     from . import Repo
@@ -78,11 +82,12 @@ def reproduce(  # noqa: C901, PLR0912
 
         targets = [PROJECT_FILE]
 
+    targets = targets or []
     interactive = kwargs.get("interactive", False)
     if not interactive:
         kwargs["interactive"] = self.config["core"].get("interactive", False)
 
-    stages = set()
+    stages = []
     if pipeline or all_pipelines:
         pipelines = get_pipelines(self.index.graph)
         if all_pipelines:
@@ -96,10 +101,10 @@ def reproduce(  # noqa: C901, PLR0912
         for pline in used_pipelines:
             for stage in pline:
                 if pline.in_degree(stage) == 0:
-                    stages.add(stage)
+                    stages.append(stage)
     else:
         for target in targets:
-            stages.update(
+            stages.extend(
                 self.stage.collect(
                     target,
                     recursive=recursive,
@@ -114,7 +119,7 @@ def reproduce(  # noqa: C901, PLR0912
         except RunCacheNotSupported as e:
             logger.warning("Failed to pull run cache: %s", e)
 
-    return _reproduce_stages(self.index.graph, list(stages), **kwargs)
+    return _reproduce_stages(self.index.graph, ldistinct(stages), **kwargs)
 
 
 def _reproduce_stages(  # noqa: C901
@@ -159,7 +164,11 @@ def _reproduce_stages(  # noqa: C901
 
     The derived evaluation of _downstream_ B would be: [B, D, E]
     """
-    steps = _get_steps(graph, stages, downstream, single_item)
+    from .graph import get_steps
+
+    if not single_item:
+        active = _remove_frozen_stages(graph)
+        stages = get_steps(active, stages, downstream=downstream)
 
     force_downstream = kwargs.pop("force_downstream", False)
     result: List["Stage"] = []
@@ -167,7 +176,7 @@ def _reproduce_stages(  # noqa: C901
     # `ret` is used to add a cosmetic newline.
     ret: List["Stage"] = []
 
-    for stage in steps:
+    for stage in stages:
         if ret:
             logger.info("")
 
@@ -191,40 +200,10 @@ def _reproduce_stages(  # noqa: C901
     return result
 
 
-def _get_steps(graph, stages, downstream, single_item):
-    import networkx as nx
-
-    active = graph.copy()
-    if not single_item:
-        # NOTE: frozen stages don't matter for single_item
-        for stage in graph:
-            if stage.frozen:
-                # NOTE: disconnect frozen stage from its dependencies
-                active.remove_edges_from(graph.out_edges(stage))
-
-    all_pipelines: List["Stage"] = []
-    for stage in stages:
-        if downstream:
-            # NOTE (py3 only):
-            # Python's `deepcopy` defaults to pickle/unpickle the object.
-            # Stages are complex objects (with references to `repo`,
-            # `outs`, and `deps`) that cause struggles when you try
-            # to serialize them. We need to create a copy of the graph
-            # itself, and then reverse it, instead of using
-            # graph.reverse() directly because it calls `deepcopy`
-            # underneath -- unless copy=False is specified.
-            nodes = nx.dfs_postorder_nodes(active.reverse(copy=False), stage)
-            all_pipelines += reversed(list(nodes))
-        else:
-            all_pipelines += nx.dfs_postorder_nodes(active, stage)
-
-    steps = []
-    for stage in all_pipelines:
-        if stage not in steps:
-            # NOTE: order of steps still matters for single_item
-            if single_item and stage not in stages:
-                continue
-
-            steps.append(stage)
-
-    return steps
+def _remove_frozen_stages(graph: "DiGraph") -> "DiGraph":
+    g = cast("DiGraph", graph.copy())
+    for stage in graph:
+        if stage.frozen:
+            # NOTE: disconnect frozen stage from its dependencies
+            g.remove_edges_from(graph.out_edges(stage))
+    return g
diff --git a/dvc/stage/cache.py b/dvc/stage/cache.py
index a73008cf6..2a66ad997 100644
--- a/dvc/stage/cache.py
+++ b/dvc/stage/cache.py
@@ -6,7 +6,7 @@ from typing import TYPE_CHECKING, List, Optional, Tuple
 from funcy import first
 
 from dvc import fs
-from dvc.exceptions import DvcException
+from dvc.exceptions import CollectCacheError, DvcException
 from dvc.utils import dict_sha256, relpath
 
 if TYPE_CHECKING:
@@ -179,7 +179,7 @@ class StageCache:
         dump_yaml(tmp, cache)
         self.repo.cache.legacy.move(tmp, path)
 
-    def restore(self, stage, run_cache=True, pull=False, dry=False):
+    def restore(self, stage, run_cache=True, pull=False, dry=False):  # noqa: C901
         from .serialize import to_single_stage_lockfile
 
         if not _can_hash(stage):
@@ -203,8 +203,11 @@ class StageCache:
         cached_stage = self._create_stage(cache, wdir=stage.wdir)
 
         if pull and not dry:
-            for objs in cached_stage.get_used_objs().values():
-                self.repo.cloud.pull(objs)
+            try:
+                for objs in cached_stage.get_used_objs().values():
+                    self.repo.cloud.pull(objs)
+            except CollectCacheError as exc:
+                raise RunCacheNotFoundError(stage) from exc
 
         if not cached_stage.outs_cached():
             raise RunCacheNotFoundError(stage)
diff --git a/pyproject.toml b/pyproject.toml
index 07d1da168..2e5389562 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -34,13 +34,14 @@ dependencies = [
     "configobj>=5.0.6",
     "distro>=1.3",
     "dpath<3,>=2.1.0",
-    "dvc-data>=2.3.0,<2.4.0",
+    "dvc-data>=2.3.2,<2.4.0",
     "dvc-http>=2.29.0",
     "dvc-render>=0.3.1,<1",
     "dvc-studio-client>=0.9.2,<1",
     "dvc-task>=0.3.0,<1",
     "flatten_dict<1,>=0.4.1",
-    "flufl.lock>=5",
+    # https://github.com/iterative/dvc/issues/9654
+    "flufl.lock>=5,<8",
     "funcy>=1.14",
     "grandalf<1,>=0.7",
     "hydra-core>=1.1",
@@ -82,6 +83,7 @@ lint = [
     "types-tabulate",
     "types-toml",
     "types-tqdm",
+    "types-pyinstaller",
 ]
 oss = ["dvc-oss==2.19"]
 s3 = ["dvc-s3==2.23.0"]
@@ -128,6 +130,10 @@ dvc = "dvc.api:DVCFileSystem"
 [project.entry-points."pytest11"]
 dvc-testing = "dvc.testing.plugin"
 
+[project.entry-points."pyinstaller40"]
+hook-dirs = "dvc.__pyinstaller:get_hook_dirs"
+tests = "dvc.__pyinstaller:get_PyInstaller_tests"
+
 [tool.setuptools]
 license-files = ["LICENSE"]
 
@@ -292,11 +298,10 @@ ignore-names = ["M", "SCM"]
 "dvc/commands/**" = ["N806"]
 "dvc/testing/**" = ["ARG002"]
 "dvc/testing/benchmarks/**" = ["ARG001"]
-"scripts/**" = ["T201", "INP001", "TRY002"]
 
 [tool.ruff.pylint]
 max-args = 10
 
 [tool.bandit]
-exclude_dirs = ["tests", "scripts", "dvc/repo/experiments/utils.py"]
+exclude_dirs = ["tests", "dvc/repo/experiments/utils.py"]
 skips = ["B101"]
diff --git a/scripts/build-requirements.txt b/scripts/build-requirements.txt
deleted file mode 100644
index a5bb9206b..000000000
--- a/scripts/build-requirements.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-# NOTE: using strict version due to
-# https://github.com/iterative/dvc/issues/7949
-PyInstaller==5.9.0
diff --git a/scripts/build_package.sh b/scripts/build_package.sh
deleted file mode 100755
index ff1c8171f..000000000
--- a/scripts/build_package.sh
+++ /dev/null
@@ -1,15 +0,0 @@
-#!/bin/bash
-
-set -e
-set -x
-
-if [ ! -d "dvc" ]; then
-  echo "Please run this script from repository root" >&2
-  exit 1
-fi
-
-echo 'PKG = "pip"' >dvc/_build.py
-
-python -m pip install build twine
-python -m build
-twine check dist/*
diff --git a/scripts/pyinstaller/.gitignore b/scripts/pyinstaller/.gitignore
deleted file mode 100644
index 1f1748b55..000000000
--- a/scripts/pyinstaller/.gitignore
+++ /dev/null
@@ -1,4 +0,0 @@
-/build
-/dist
-/dvc.spec
-/dvc
diff --git a/scripts/pyinstaller/build.py b/scripts/pyinstaller/build.py
deleted file mode 100644
index cdfd63e6a..000000000
--- a/scripts/pyinstaller/build.py
+++ /dev/null
@@ -1,47 +0,0 @@
-import os
-import pathlib
-from subprocess import STDOUT, check_call, check_output
-
-path = pathlib.Path(__file__).parent.absolute()
-hooks = path / "hooks"
-dvc = path.parent.parent / "dvc"
-entry = dvc / "__main__.py"
-
-check_call(
-    [
-        "pyinstaller",
-        "--additional-hooks-dir",
-        os.fspath(hooks),
-        "--name",
-        "dvc",
-        "-y",
-        os.fspath(entry),
-    ],
-    cwd=path,
-    stderr=STDOUT,
-)
-
-out = check_output(
-    [
-        path / "dist" / "dvc" / "dvc",
-        "doctor",
-    ],
-    stderr=STDOUT,
-).decode()
-
-remotes = [
-    "s3",
-    "oss",
-    "gdrive",
-    "gs",
-    "hdfs",
-    "http",
-    "webhdfs",
-    "azure",
-    "ssh",
-    "webdav",
-]
-
-print(out)
-for remote in remotes:
-    assert f"\t{remote}" in out, f"Missing support for {remote}"


Test patch:
diff --git a/tests/func/repro/test_repro.py b/tests/func/repro/test_repro.py
index 1da583db5..a2b10cb0d 100644
--- a/tests/func/repro/test_repro.py
+++ b/tests/func/repro/test_repro.py
@@ -1282,3 +1282,17 @@ def test_repro_rm_recursive(tmp_dir, dvc):
     dvc.reproduce()
     assert (tmp_dir / "dir").exists()
     assert not (tmp_dir / "dir" / "foo").exists()
+
+
+def test_repro_single_item_with_multiple_targets(tmp_dir, dvc, copy_script):
+    stage1 = dvc.stage.add(cmd="echo foo > foo", outs=["foo"], name="gen-foo")
+    with dvc.lock:
+        stage1.run()
+
+    stage2 = dvc.stage.add(
+        cmd="python copy.py foo bar", deps=["foo"], outs=["bar"], name="copy-foo-bar"
+    )
+    assert dvc.reproduce(["copy-foo-bar", "gen-foo"], single_item=True) == [
+        stage2,
+        stage1,
+    ]
diff --git a/tests/unit/repo/test_graph.py b/tests/unit/repo/test_graph.py
new file mode 100644
index 000000000..380ffa3c7
--- /dev/null
+++ b/tests/unit/repo/test_graph.py
@@ -0,0 +1,69 @@
+import pytest
+from networkx import DiGraph
+from networkx.utils import graphs_equal
+
+from dvc.repo.graph import get_steps, get_subgraph_of_nodes
+
+
+@pytest.mark.parametrize(
+    "nodes,downstream,expected_edges",
+    [
+        ([], False, {1: [2, 3], 2: [4, 5], 3: [6, 7], 8: [9]}),
+        ([1], False, {1: [2, 3], 2: [4, 5], 3: [6, 7]}),
+        ([2], False, {2: [4, 5]}),
+        ([3], False, {3: [6, 7]}),
+        ([8], False, [(8, 9)]),
+        ([2, 3, 8], False, {2: [4, 5], 3: [6, 7], 8: [9]}),
+        ([4], False, {4: []}),
+        ([], True, {1: [2, 3], 2: [4, 5], 3: [6, 7], 8: [9]}),
+        ([1], True, {1: []}),
+        ([9], True, [(8, 9)]),
+        ([2], True, [(1, 2)]),
+        ([6], True, [(1, 3), (3, 6)]),
+        ([2, 3, 8], True, {1: [2, 3], 8: []}),
+        ([4, 7], True, {1: [2, 3], 2: [4], 3: [7]}),
+    ],
+)
+def test_subgraph_of_nodes(nodes, downstream, expected_edges):
+    r"""
+             1
+           /   \
+          2     3      8
+         / \   / \     |
+        4   5 6   7    9
+    """
+    graph = DiGraph({1: [2, 3], 2: [4, 5], 3: [6, 7], 8: [9]})
+    subgraph = get_subgraph_of_nodes(graph, nodes, downstream=downstream)
+    expected = DiGraph(expected_edges)
+    assert graphs_equal(expected, subgraph)
+
+
+@pytest.mark.parametrize(
+    "nodes,downstream,expected_steps",
+    [
+        ([], False, [4, 5, 2, 6, 7, 3, 1, 9, 8]),
+        ([1], False, [4, 5, 2, 6, 7, 3, 1]),
+        ([2], False, [4, 5, 2]),
+        ([3], False, [6, 7, 3]),
+        ([8], False, [9, 8]),
+        ([2, 3, 8], False, [4, 5, 2, 6, 7, 3, 9, 8]),
+        ([4], False, [4]),
+        ([], True, [4, 5, 2, 6, 7, 3, 1, 9, 8]),
+        ([1], True, [1]),
+        ([9], True, [9, 8]),
+        ([2], True, [2, 1]),
+        ([6], True, [6, 3, 1]),
+        ([2, 3, 8], True, [8, 2, 3, 1]),
+        ([4, 7], True, [4, 2, 7, 3, 1]),
+    ],
+)
+def test_steps(nodes, downstream, expected_steps):
+    r"""
+             1
+           /   \
+          2     3      8
+         / \   / \     |
+        4   5 6   7    9
+    """
+    graph = DiGraph({1: [2, 3], 2: [4, 5], 3: [6, 7], 8: [9]})
+    assert get_steps(graph, nodes, downstream=downstream) == expected_steps