Repository: reanahub/reana-client

Issue:
upload: upload full directory when files are not specified
Due to moving the workflow load logic from reana-client to the workflow-engine, we need all the workflow files in the workspace to be able to load the specification. 

**Current behavior**

When running `reana-client upload -w foo` we only upload the files specified in `inputs.files` and we have to manually upload what's in `workflow.file` and its dependencies.

**Expected behavior**

When running `reana-client upload -w foo`, upload all the files in the current `reana.yaml` directory.

**Extra considerations**
- Remove validation on `reana-client create`?
- Compare remote and local workspace before `reana-client start` and warn user if there're differences?
- Introduce the concept of `.reanaignore`. First check `.gitignore`, after `.reanaignore` and prevent uploading the present files.

Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/reanahub/reana-client/pull/608?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub) Report
> Merging [#608](https://codecov.io/gh/reanahub/reana-client/pull/608?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub) (c0d8dc5) into [master](https://codecov.io/gh/reanahub/reana-client/commit/32af8b0e81abb15d5b99d6ff0e1c447b0189e42a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub) (32af8b0) will **increase** coverage by `0.24%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/reanahub/reana-client/pull/608/graphs/tree.svg?width=650&height=150&src=pr&token=b85cSEiOhs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub)](https://codecov.io/gh/reanahub/reana-client/pull/608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub)

```diff
@@            Coverage Diff             @@
##           master     #608      +/-   ##
==========================================
+ Coverage   56.36%   56.60%   +0.24%     
==========================================
  Files          23       23              
  Lines        2278     2286       +8     
==========================================
+ Hits         1284     1294      +10     
+ Misses        994      992       -2     
```


| [Impacted Files](https://codecov.io/gh/reanahub/reana-client/pull/608?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub) | Coverage Δ | |
|---|---|---|
| [reana\_client/cli/files.py](https://codecov.io/gh/reanahub/reana-client/pull/608/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub#diff-cmVhbmFfY2xpZW50L2NsaS9maWxlcy5weQ==) | `76.25% <100.00%> (+1.34%)` | :arrow_up: |


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 75867b4..6c970db 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -4,6 +4,7 @@ Changes
 Version 0.9.0 (UNRELEASED)
 --------------------------
 
+- Adds support for ``.reanaignore`` during file upload. Files that match ``.reanaignore`` will not be uploaded.
 - Adds support for ``.gitignore`` during file upload. Files that match ``.gitignore`` will not be uploaded.
 - Changes REANA specification loading and validation functionality by porting some of the logic to ``reana-commons``.
 - Fixes ``validate --environment`` command to detect illegal white space characters in image names.
diff --git a/reana_client/cli/files.py b/reana_client/cli/files.py
index 87b14db..b81a3bb 100644
--- a/reana_client/cli/files.py
+++ b/reana_client/cli/files.py
@@ -334,27 +334,6 @@ def upload_files(  # noqa: C901
     for p in ctx.params:
         logging.debug("{param}: {value}".format(param=p, value=ctx.params[p]))
 
-    def _filter_files_by_gitignore(directories: List[str]) -> List[str]:
-        try:
-            with open(".gitignore") as f:
-                gitignore_spec = pathspec.PathSpec.from_lines("gitwildmatch", f)
-            display_message(
-                "Detected .gitignore file. Some files might get ignored.",
-                msg_type="info",
-            )
-        except FileNotFoundError:
-            return directories
-        else:
-            filtered_filenames = []
-            for directory_path in directories:
-                for root, dirs, files in os.walk(directory_path):
-                    if not gitignore_spec.match_file(root):
-                        for file_name in files:
-                            filename_full_path = os.path.join(root, file_name)
-                            if not gitignore_spec.match_file(filename_full_path):
-                                filtered_filenames.append(filename_full_path)
-            return filtered_filenames
-
     if not filenames:
         try:
             reana_spec = get_workflow_specification(workflow, access_token)[
@@ -376,12 +355,52 @@ def upload_files(  # noqa: C901
                 for f in reana_spec["inputs"].get("files") or []
             ]
 
-            directories_full_path = [
-                os.path.join(os.getcwd(), d)
-                for d in reana_spec["inputs"].get("directories") or []
-            ]
-            filenames += _filter_files_by_gitignore(directories_full_path)
+            # collect all files in input.directories
+            files_from_directories = []
+            directories = reana_spec["inputs"].get("directories") or []
+            for directory_path in directories:
+                for root, _, dir_filenames in os.walk(directory_path):
+                    filenames_full_path = [
+                        os.path.join(root, file) for file in dir_filenames
+                    ]
+                    files_from_directories.extend(filenames_full_path)
+
+            def _filter_files(
+                files: List[str], paths_spec: pathspec.PathSpec
+            ) -> List[str]:
+                return [file for file in files if not paths_spec.match_file(file)]
 
+            try:
+                with open(".gitignore") as f:
+                    gitignore_spec = pathspec.PathSpec.from_lines("gitwildmatch", f)
+                display_message(
+                    "Detected .gitignore file. Some files might get ignored.",
+                    msg_type="info",
+                )
+            except FileNotFoundError:
+                pass
+            else:
+                files_from_directories = _filter_files(
+                    files_from_directories, gitignore_spec
+                )
+
+            try:
+                with open(".reanaignore") as f:
+                    reanaignore_spec = pathspec.PathSpec.from_lines("gitwildmatch", f)
+                display_message(
+                    "Detected .reanaignore file. Some files might get ignored.",
+                    msg_type="info",
+                )
+            except FileNotFoundError:
+                pass
+            else:
+                files_from_directories = _filter_files(
+                    files_from_directories, reanaignore_spec
+                )
+
+            filenames += [
+                os.path.join(os.getcwd(), file) for file in files_from_directories
+            ]
     upload_failed = False
     for filename in filenames:
         try:


Test patch:
diff --git a/tests/test_cli_files.py b/tests/test_cli_files.py
index 1eb85ca..e898ab2 100644
--- a/tests/test_cli_files.py
+++ b/tests/test_cli_files.py
@@ -241,8 +241,10 @@ def test_upload_file(create_yaml_workflow_schema):
                 assert message in result.output
 
 
-def test_upload_file_respect_gitignore(get_workflow_specification_with_directory):
-    """Do not upload file from the directory in input.directories if it matches .gitignore"""
+def test_upload_file_respect_gitignore(
+    get_workflow_specification_with_directory,
+):
+    """If .gitignore exists and is not empty, respect it's rules."""
     reana_token = "000000"
     env = {"REANA_SERVER_URL": "http://localhost"}
     runner = CliRunner(env=env)
@@ -253,17 +255,98 @@ def test_upload_file_respect_gitignore(get_workflow_specification_with_directory
         ), patch("reana_client.api.client.requests.post") as post_request:
             with runner.isolated_filesystem():
                 with open(".gitignore", "w") as f:
-                    f.write("do_not_upload.txt\n")
-                    f.write("test\n")
+                    f.write("data/should_not_upload.txt\n")
 
                 os.mkdir("data")
-                with open("data/do_not_upload.txt", "w") as f:
+                with open("data/should_not_upload.txt", "w") as f:
                     f.write("This file should not be uploaded.")
 
-                with open("data/test", "w") as f:
+                result = runner.invoke(
+                    cli, ["upload", "-t", reana_token, "--workflow", "mytest.1"]
+                )
+
+                assert (
+                    "==> Detected .gitignore file. Some files might get ignored."
+                    in result.output
+                )
+                post_request.assert_not_called()
+                assert result.exit_code == 0
+
+
+def test_upload_file_skip_empty_git_and_reana_ignore_files(
+    get_workflow_specification_with_directory,
+):
+    """If .reanaignore or .gitignore files are empty, ignore them.
+
+    This is edge case. We do not expect this to happen.
+    """
+    reana_token = "000000"
+    env = {"REANA_SERVER_URL": "http://localhost"}
+    runner = CliRunner(env=env)
+    mock_specification = Mock(return_value=get_workflow_specification_with_directory)
+    with runner.isolation():
+        with patch(
+            "reana_client.api.client.get_workflow_specification", mock_specification
+        ), patch("reana_client.api.client.requests.post") as post_request:
+            with runner.isolated_filesystem():
+                with open(".gitignore", "w") as f:
+                    f.write("\n")
+
+                with open(".reanaignore", "w") as f:
+                    f.write("\n")
+
+                os.mkdir("data")
+                with open("data/should_upload.txt", "w") as f:
+                    f.write("This file should be uploaded.")
+
+                result = runner.invoke(
+                    cli, ["upload", "-t", reana_token, "--workflow", "mytest.1"]
+                )
+
+                assert (
+                    "==> Detected .gitignore file. Some files might get ignored."
+                    in result.output
+                )
+                assert (
+                    "==> Detected .reanaignore file. Some files might get ignored."
+                    in result.output
+                )
+                post_request.assert_called_once()
+                assert "should_upload.txt" in result.output
+                assert result.exit_code == 0
+
+
+def test_upload_file_respect_reanaignore_and_gitignore(
+    get_workflow_specification_with_directory,
+):
+    """Check if file upload respect both reana and git ignore files with input.directories."""
+    reana_token = "000000"
+    env = {"REANA_SERVER_URL": "http://localhost"}
+    runner = CliRunner(env=env)
+    mock_specification = Mock(return_value=get_workflow_specification_with_directory)
+    with runner.isolation():
+        with patch(
+            "reana_client.api.client.get_workflow_specification", mock_specification
+        ), patch("reana_client.api.client.requests.post"):
+            with runner.isolated_filesystem():
+                with open(".gitignore", "w") as f:
+                    f.write("data/ignored_in_gitignore.txt\n")
+
+                with open(".reanaignore", "w") as f:
+                    f.write("ignored_in_reanaignore.txt\n")
+
+                os.mkdir("data")
+                os.mkdir("data/another_directory")
+                with open("data/ignored_in_gitignore.txt", "w") as f:
+                    f.write("This file should not be uploaded.")
+
+                with open("data/ignored_in_reanaignore.txt", "w") as f:
                     f.write("This file should not be uploaded.")
 
-                with open("data/will_upload.txt", "w") as f:
+                with open("data/should_be_uploaded.txt", "w") as f:
+                    f.write("This file will be uploaded.")
+
+                with open("data/another_directory/should_be_uploaded.txt", "w") as f:
                     f.write("This file will be uploaded.")
 
                 result = runner.invoke(
@@ -274,9 +357,15 @@ def test_upload_file_respect_gitignore(get_workflow_specification_with_directory
                     "==> Detected .gitignore file. Some files might get ignored."
                     in result.output
                 )
-                assert "test" not in result.output
-                post_request.assert_called_once()
-                assert "will_upload.txt" in result.output
+                assert (
+                    "==> Detected .reanaignore file. Some files might get ignored."
+                    in result.output
+                )
+                assert "ignored_in_gitignore.txt" not in result.output
+                assert "ignored_in_reanaignore.txt" not in result.output
+
+                assert "data/should_be_uploaded.txt" in result.output
+                assert "data/another_directory/should_be_uploaded.txt" in result.output
                 assert result.exit_code == 0