Repository: kedro-org/kedro
Pull request: Add E2E test for tools
## Description
For most of the add-ons flow I've been testing the work manually: create a new project with a combination of add-ons, install requirements and try a kedro run. This process has uncovered several bugs already.

## Context
We should add automated tests to test at least some of the add-ons flow.

Implementation
At a minimum have tests for:

```
kedro new --addons=none
kedro new --addons=lint,docs,data,logging,... (everything but not viz and pyspark)
kedro new --addons=all
kedro new --addons=pyspark
kedro new --addons=viz
```
All of the above should have an example so kedro run can be tested.

## Development Notes

- Added relevant e2e tests for addons.
- Updated GitHub Actions e2e config to support running PySpark

## Checklist

- [x] Read the [contributing](https://github.com/kedro-org/kedro/blob/main/CONTRIBUTING.md) guidelines
- [x] Signed off each commit with a [Developer Certificate of Origin (DCO)](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-the-commit-signoff-policy-for-your-repository)
- [x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
- [ ] Updated the documentation to reflect the code changes
- [ ] Added a description of this change in the [`RELEASE.md`](https://github.com/kedro-org/kedro/blob/main/RELEASE.md) file
- [ ] Added tests to cover my changes
- [ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team


Problem statement:
Add e2e tests for `add-ons` flow
## Description
For most of the add-ons flow I've been testing the work manually: create a new project with a combination of add-ons, install requirements and try a `kedro run`. This process has uncovered several bugs already.

## Context
We should add automated tests to test at least some of the add-ons flow.

## Implementation
At a minimum have tests for:
1. `kedro new --addons=none` 
2. `kedro new --addons=lint,docs,data,logging,...` (everything but not viz and pyspark)
3. `kedro new --addons=all`
4. `kedro new --addons=pyspark`
5. `kedro new --addons=viz`

All of the above should have an example so `kedro run` can be tested.

Hints from issue discussion:

Commit message: Merge branch 'develop' into dev/e2e-test-for-add-ons

Patch to review:
diff --git a/.github/workflows/e2e-tests.yml b/.github/workflows/e2e-tests.yml
index d39505f8fc..c8a970f893 100644
--- a/.github/workflows/e2e-tests.yml
+++ b/.github/workflows/e2e-tests.yml
@@ -42,6 +42,23 @@ jobs:
       - name: Add MSBuild to PATH
         if: inputs.os == 'windows-latest'
         uses: microsoft/setup-msbuild@v1
+      - name: Install Microsoft Visual C++ Redistributable
+        if: inputs.os == 'windows-latest'
+        run: |
+          choco install vcredist-all
+        shell: powershell
+      - name: Setup Hadoop binary
+        if: inputs.os == 'windows-latest'
+        run: |
+          Invoke-WebRequest "https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe?raw=true" -OutFile winutils.exe
+          Invoke-WebRequest "https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/hadoop.dll?raw=true" -OutFile hadoop.dll
+          Move-Item .\hadoop.dll C:\Windows\System32
+          New-Item -ItemType directory -Path C:\hadoop\bin
+          Move-Item .\winutils.exe C:\hadoop\bin
+          [System.Environment]::SetEnvironmentVariable('HADOOP_HOME', 'C:\hadoop', [System.EnvironmentVariableTarget]::Machine)
+          echo "HADOOP_HOME=C:\hadoop" | Out-File -Append -Encoding ascii -FilePath $env:GITHUB_ENV
+          echo "PATH=$env:HADOOP_HOME\bin;$env:PATH" | Out-File -Append -Encoding ascii -FilePath $env:GITHUB_ENV
+        shell: powershell
       - name: Install dependencies
         run: |
           pip install --upgrade pip
diff --git a/features/steps/cli_steps.py b/features/steps/cli_steps.py
index 4494aade8a..e785ec187c 100644
--- a/features/steps/cli_steps.py
+++ b/features/steps/cli_steps.py
@@ -162,6 +162,31 @@ def create_config_file(context):
         yaml.dump(config, config_file, default_flow_style=False)
 
 
+@given('I have prepared a config file with tools "{tools}"')
+def create_config_file_with_tools(context, tools):
+    """Behave step to create a temporary config file
+    (given the existing temp directory) and store it in the context.
+    It takes a custom tools list and sets example prompt to `y`.
+    """
+
+    tools_str = tools if tools != "none" else ""
+
+    context.config_file = context.temp_dir / "config.yml"
+    context.project_name = "project-dummy"
+    context.root_project_dir = context.temp_dir / context.project_name
+    context.package_name = context.project_name.replace("-", "_")
+    config = {
+        "tools": tools_str,
+        "example_pipeline": "y",
+        "project_name": context.project_name,
+        "repo_name": context.project_name,
+        "output_dir": str(context.temp_dir),
+        "python_package": context.package_name,
+    }
+    with context.config_file.open("w") as config_file:
+        yaml.dump(config, config_file, default_flow_style=False)
+
+
 @given("I have installed the project dependencies")
 def pip_install_dependencies(context):
     """Install project dependencies using pip."""
@@ -453,6 +478,49 @@ def is_created(name):
         assert is_created(path)
 
 
+@then('the expected tool directories and files should be created with "{tools}"')
+def check_created_project_structure_from_tools(context, tools):
+    """Behave step to check the subdirectories created by kedro new with tools."""
+
+    def is_created(name):
+        """Check if path exists."""
+        return (context.root_project_dir / name).exists()
+
+    # Base checks for any project
+    for path in ["README.md", "src", "pyproject.toml", "requirements.txt"]:
+        assert is_created(path), f"{path} does not exist"
+
+    tools_list = (
+        tools.split(",") if tools != "all" else ["1", "2", "3", "4", "5", "6", "7"]
+    )
+
+    if "1" in tools_list:  # lint tool
+        pass
+
+    if "2" in tools_list:  # test tool
+        assert is_created("tests"), "tests directory does not exist"
+
+    if "3" in tools_list:  # log tool
+        assert is_created("conf/logging.yml"), "logging configuration does not exist"
+
+    if "4" in tools_list:  # docs tool
+        assert is_created("docs"), "docs directory does not exist"
+
+    if "5" in tools_list:  # data tool
+        assert is_created("data"), "data directory does not exist"
+
+    if "6" in tools_list:  # PySpark tool
+        assert is_created("conf/base/spark.yml"), "spark.yml does not exist"
+
+    if "7" in tools_list:  # 'viz' tool
+        expected_reporting_path = Path(
+            f"src/{context.package_name}/pipelines/reporting"
+        )
+        assert is_created(
+            expected_reporting_path
+        ), "reporting pipeline directory does not exist"
+
+
 @then("the logs should show that {number} nodes were run")
 def check_one_node_run(context, number):
     expected_log_line = f"Completed {number} out of {number} tasks"
diff --git a/features/tools.feature b/features/tools.feature
new file mode 100644
index 0000000000..cd17319878
--- /dev/null
+++ b/features/tools.feature
@@ -0,0 +1,38 @@
+Feature: New Kedro project with tools
+
+  Scenario: Create a new Kedro project without any tools
+    Given I have prepared a config file with tools "none"
+    When I run a non-interactive kedro new without starter
+    Then the expected tool directories and files should be created with "none"
+
+  Scenario: Create a new Kedro project with all tools except 'viz' and 'pyspark'
+    Given I have prepared a config file with tools "1,2,3,4,5"
+    When I run a non-interactive kedro new without starter
+    Then the expected tool directories and files should be created with "1,2,3,4,5"
+    Given I have installed the project dependencies
+    When I execute the kedro command "run"
+    Then I should get a successful exit code
+
+  Scenario: Create a new Kedro project with all tools
+    Given I have prepared a config file with tools "all"
+    When I run a non-interactive kedro new without starter
+    Then the expected tool directories and files should be created with "all"
+    Given I have installed the project dependencies
+    When I execute the kedro command "run"
+    Then I should get a successful exit code
+
+  Scenario: Create a new Kedro project with only 'pyspark' tool
+    Given I have prepared a config file with tools "6"
+    When I run a non-interactive kedro new without starter
+    Then the expected tool directories and files should be created with "6"
+    Given I have installed the project dependencies
+    When I execute the kedro command "run"
+    Then I should get a successful exit code
+
+  Scenario: Create a new Kedro project with only 'viz' tool
+    Given I have prepared a config file with tools "7"
+    When I run a non-interactive kedro new without starter
+    Then the expected tool directories and files should be created with "7"
+    Given I have installed the project dependencies
+    When I execute the kedro command "run"
+    Then I should get a successful exit code


Code review comments:
diff_hunk: @@ -453,6 +478,49 @@ def is_created(name):
         assert is_created(path)
 
 
+@then('the expected tool directories and files should be created with "{tools}"')
+def check_created_project_structure_from_tools(context, tools):
+    """Behave step to check the subdirectories created by kedro new with tools."""
+
+    def is_created(name):
+        """Check if path exists."""
+        return (context.root_project_dir / name).exists()
+
+    # Base checks for any project
+    for path in ["README.md", "src", "pyproject.toml", "requirements.txt"]:
+        assert is_created(path), f"{path} does not exist"
+
+    tools_list = (
+        tools.split(",") if tools != "all" else ["1", "2", "3", "4", "5", "6", "7"]
+    )
+
+    if "1" in tools_list:  # lint tool
+        pass
path: features/steps/cli_steps.py
text: @user1:
Could you explain please why 1 is empty? We cannot check file?

@author:
This just modifies the `requirement.txt` and `pyproject.toml` so there are no additional files to check for and these files are checked for in the base check.

diff_hunk: @@ -42,6 +42,23 @@ jobs:
       - name: Add MSBuild to PATH
         if: inputs.os == 'windows-latest'
         uses: microsoft/setup-msbuild@v1
+      - name: Install Microsoft Visual C++ Redistributable
+        if: inputs.os == 'windows-latest'
+        run: |
+          choco install vcredist-all
+        shell: powershell
+      - name: Setup Hadoop binary
+        if: inputs.os == 'windows-latest'
+        run: |
+          Invoke-WebRequest "https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe?raw=true" -OutFile winutils.exe
+          Invoke-WebRequest "https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/hadoop.dll?raw=true" -OutFile hadoop.dll
+          Move-Item .\hadoop.dll C:\Windows\System32
+          New-Item -ItemType directory -Path C:\hadoop\bin
+          Move-Item .\winutils.exe C:\hadoop\bin
+          [System.Environment]::SetEnvironmentVariable('HADOOP_HOME', 'C:\hadoop', [System.EnvironmentVariableTarget]::Machine)
+          echo "HADOOP_HOME=C:\hadoop" | Out-File -Append -Encoding ascii -FilePath $env:GITHUB_ENV
+          echo "PATH=$env:HADOOP_HOME\bin;$env:PATH" | Out-File -Append -Encoding ascii -FilePath $env:GITHUB_ENV
+        shell: powershell
path: .github/workflows/e2e-tests.yml
text: @user1:
```suggestion
```

diff_hunk: @@ -42,6 +42,23 @@ jobs:
       - name: Add MSBuild to PATH
         if: inputs.os == 'windows-latest'
         uses: microsoft/setup-msbuild@v1
+      - name: Install Microsoft Visual C++ Redistributable
+        if: inputs.os == 'windows-latest'
+        run: |
+          choco install vcredist-all
+        shell: powershell
path: .github/workflows/e2e-tests.yml
text: @user1:
```suggestion
```
Isn't this unnecessary? Default shell on Windows runners is `pwsh` anyway, which I assume should behave similarly; if it's not available, it would fall back to `powershell`. See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell.

@author:
Good spot.