Repository: iterative/dvc

Issue:
Experiments: unexpected error - unsupported operand type(s) for +: 'int' and 'NoneType'
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug. 

Example: `repro: doesn't detect input changes`
-->

exp run: fails to execute.

## Description

<!--
A clear and concise description of what the bug is.
-->

Running `dvc exp run` fails. I have a DVC repo with a bunch experiments and preprocessing. When trying to use the new experiments functionality I get the following error:

```console
ERROR: unexpected error - unsupported operand type(s) for +: 'int' and 'NoneType'
ERROR: Failed to reproduce experiment '169e81b': unsupported operand type(s) for +: 'int' and 'NoneType'
```
It seems to be some kind of problem parsing a file. I have checked the experiment migration gist, but since this is the first time the repo is used with that functionality, it does nothing.

### Reproduce

<!--
Step list of how to reproduce the bug
-->

I don't know how to reproduce this scenario, the repo has some time now and, to get to the current state it requires processing a lot of data. But I can provide the verbose output of the command (in the additional info section).

The stage requires external dependencies (marked with --external when creating the stage), though it is not visible in the dvc.yaml file.

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

The command should run the experiment correctly.

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc version`:**

```console
$ dvc version
2.0.0a0+748bda
```

**Additional Information (if any):**

<!--
If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
-->

Here is the verbose output when running the command:

```console
dvc exp run --verbose
2021-02-09 18:28:24,465 DEBUG: Check for update is enabled.
2021-02-09 18:28:24,553 DEBUG: Detaching HEAD at 'HEAD'                         
2021-02-09 18:28:24,573 DEBUG: Stashing workspace
2021-02-09 18:28:24,606 DEBUG: Stashing changes in 'refs/stash'
2021-02-09 18:28:24,630 DEBUG: Applying stash commit 'fa1d07df87d1a369a3b76183cd6a8feb1177e774'
2021-02-09 18:28:24,724 DEBUG: Stashing changes in 'refs/exps/stash'
2021-02-09 18:28:24,763 DEBUG: Stashed experiment 'a11a144' with baseline '5e5f730' for future execution.
2021-02-09 18:28:24,768 DEBUG: Restoring stashed workspace
2021-02-09 18:28:24,768 DEBUG: Popping from stash 'refs/stash'
2021-02-09 18:28:24,769 DEBUG: Applying stash commit 'fa1d07df87d1a369a3b76183cd6a8feb1177e774'
2021-02-09 18:28:24,791 DEBUG: Dropping 'refs/stash@{0}'
2021-02-09 18:28:24,798 DEBUG: Removing '/home/gb/Documents/Research/quail-experiments/.git/logs/refs/stash'
2021-02-09 18:28:24,802 DEBUG: Restore HEAD to 'one-exper-per-commit'
2021-02-09 18:28:24,820 DEBUG: Detaching HEAD at '5e5f73060a049d30c1613a7cbe922a50d4905227'
2021-02-09 18:28:24,912 DEBUG: Popping from stash 'refs/exps/stash'
2021-02-09 18:28:24,912 DEBUG: Applying stash commit 'a11a14432ad38e3cb30b1b646f45714137c8eb83'
2021-02-09 18:28:24,932 DEBUG: Dropping 'refs/exps/stash@{0}'
2021-02-09 18:28:24,938 DEBUG: Removing '/home/gb/Documents/Research/quail-experiments/.git/logs/refs/exps/stash'
2021-02-09 18:28:24,949 DEBUG: Running repro in '/home/gb/Documents/Research/quail-experiments'
2021-02-09 18:28:24,950 DEBUG: Removing '/home/gb/Documents/Research/quail-experiments/.dvc/tmp/repro.dat'
2021-02-09 18:28:25,038 DEBUG: Lockfile 'dvc.lock' needs to be updated.         
2021-02-09 18:28:25,070 ERROR: unexpected error - unsupported operand type(s) for +: 'int' and 'NoneType'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/executor/base.py", line 330, in _repro_dvc
    yield dvc
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/executor/base.py", line 268, in reproduce
    dvc_checkout(dvc, force=True, quiet=True, allow_missing=True)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/checkout.py", line 104, in checkout
    total = get_all_files_numbers(pairs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/checkout.py", line 41, in get_all_files_numbers
    return sum(
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/checkout.py", line 42, in <genexpr>
    stage.get_all_files_number(filter_info) for stage, filter_info in pairs
  File "/home/gb/Documents/Research/Projects/dvc/dvc/stage/__init__.py", line 611, in get_all_files_number
    return sum(
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
------------------------------------------------------------
2021-02-09 18:28:25,074 DEBUG: Restore HEAD to 'one-exper-per-commit'
2021-02-09 18:28:25,080 ERROR: Failed to reproduce experiment 'a11a144': unsupported operand type(s) for +: 'int' and 'NoneType'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/__init__.py", line 723, in _workspace_repro
    exec_result = BaseExecutor.reproduce(
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/executor/base.py", line 268, in reproduce
    dvc_checkout(dvc, force=True, quiet=True, allow_missing=True)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/checkout.py", line 104, in checkout
    total = get_all_files_numbers(pairs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/checkout.py", line 41, in get_all_files_numbers
    return sum(
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/checkout.py", line 42, in <genexpr>
    stage.get_all_files_number(filter_info) for stage, filter_info in pairs
  File "/home/gb/Documents/Research/Projects/dvc/dvc/stage/__init__.py", line 611, in get_all_files_number
    return sum(
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gb/Documents/Research/Projects/dvc/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/home/gb/Documents/Research/Projects/dvc/dvc/command/experiments.py", line 502, in run
    results = self.repo.experiments.run(
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/__init__.py", line 833, in run
    return run(self.repo, *args, **kwargs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/run.py", line 37, in run
    return repo.experiments.reproduce_one(
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/__init__.py", line 384, in reproduce_one
    results = self._workspace_repro()
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/__init__.py", line 58, in wrapper
    ret = f(exp, *args, **kwargs)
  File "/home/gb/Documents/Research/Projects/dvc/dvc/repo/experiments/__init__.py", line 743, in _workspace_repro
    raise DvcException(
dvc.exceptions.DvcException: Failed to reproduce experiment 'a11a144'
------------------------------------------------------------
2021-02-09 18:28:25,082 DEBUG: Analytics is enabled.
2021-02-09 18:28:25,099 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpi4nfxquu']'
2021-02-09 18:28:25,100 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpi4nfxquu']'
```

Also, here is the stage to be run with the command:
```yaml
  classification:
    cmd: python src/etl/embeddings/classify/classification.py -d /data/gblanco/datasets/embeddings/quail_no_empty_answers
      -o /data/gblanco/results/quail_no_empty_answers --metrics_dir data/metrics/classification
      --train --eval
    deps:
    - /data/gblanco/datasets/embeddings/quail_no_empty_answers
    - src/etl/embeddings/classify/classification.py
    params:
    - classification
    - features
    outs:
    - /data/gblanco/results/quail_no_empty_answers:
        checkpoint: true
    metrics:
    - data/metrics/classification/scores.json:
        cache: false
```

If there is any additional information that could help, just let me know :)


Discussion:
@geblanco 
1. What about `dvc repro`? Could you try running it with `-f` (if its not too computationally expensive)?
2. Would it be possible to update dvc to recent version?
I think this is happening in master and not a 1.11.x release?

Looks like 
https://github.com/iterative/dvc/blob/fa550f91c30cd8bd238fd4fcb76452c0c74669bb/dvc/output/base.py#L476-L479
might be returning None for some case instead of 0 after the changes from https://github.com/iterative/dvc/pull/5424
Oops, my fault from #5424 , sorry. Looking into it...

Resolving patch:
diff --git a/dvc/output/base.py b/dvc/output/base.py
--- a/dvc/output/base.py
+++ b/dvc/output/base.py
@@ -468,7 +468,7 @@ def get_files_number(self, filter_info=None):
             return 1
 
         if not filter_info or filter_info == self.path_info:
-            return self.hash_info.nfiles
+            return self.hash_info.nfiles or 0
 
         if not self.hash_info.dir_info:
             return 0


Test patch:
diff --git a/tests/func/test_checkout.py b/tests/func/test_checkout.py
--- a/tests/func/test_checkout.py
+++ b/tests/func/test_checkout.py
@@ -4,6 +4,7 @@
 import os
 import shutil
 import stat
+import textwrap
 
 import pytest
 from mock import patch
@@ -23,7 +24,7 @@
 from dvc.system import System
 from dvc.tree.local import LocalTree
 from dvc.utils import relpath
-from dvc.utils.fs import walk_files
+from dvc.utils.fs import remove, walk_files
 from dvc.utils.serialize import dump_yaml, load_yaml
 from tests.basic_env import TestDvc, TestDvcGit
 from tests.func.test_repro import TestRepro
@@ -989,3 +990,20 @@ def test_checkout_file(tmp_dir, dvc):
     os.unlink("foo")
     stats = dvc.checkout("foo")
     assert stats["added"] == ["foo"]
+
+
+def test_checkout_dir_compat(tmp_dir, dvc):
+    (stage,) = tmp_dir.dvc_gen({"data": {"foo": "foo"}})
+    tmp_dir.gen(
+        "data.dvc",
+        textwrap.dedent(
+            f"""\
+        outs:
+        - md5: {stage.outs[0].hash_info.value}
+          path: data
+        """
+        ),
+    )
+    remove("data")
+    dvc.checkout()
+    assert (tmp_dir / "data").read_text() == {"foo": "foo"}