Repository: cubewise-code/tm1py (language: python)

Issue:
Unit tests in CI Pipeline
To simplify the contributions and ensure high code quality, it would be beneficial to have a Github CI pipeline that runs the test suite. We need a TM1 server to run the tests, as mocking the whole rest api is not very practical. This could be a server that can be accessed from the pipeline or a TM1 database running in a container. I'm not sure if there are any license issues. Have you discussed this before?
New samples
New samples



Discussion / hints:
I totally agree. @nicolasbisurgi is actually working on this as we speak :)

I think we can open a PR soon and continue the conversation there


Hi @onefloid 
I'm glad you opened this ticket. As @MariusWirtz mentioned I'm working on some GH Actions for our repos. For this specific case I was thinking on the below approach, I'm curious to hear your thoughts as well!

**Workflow Steps:**

1. **Pull TM1 Objects from External Repository:**
   - This external TM1 Repository will contain all the necessary objects that are required to run the testing scripts (i.e.: cubes, dimensions, csv files, json's, etc.). This should be done using Pull from TM1.Git integrations module.

2. **Variables Configuration:**
   - For each environment, we define two key variables:
     - `TM1_CONNECTION_DETAILS`: A stringified dictionary containing connection parameters, which will be converted into a `Test/config.ini` file during the workflow.
     - `TM1_CONNECTION_PASSWORD`: A secret variable to securely store the TM1 connection password.
   - *OPTIONAL*: We could make use of the GH repository environments. This would allow us to have the same variables but for different TM1 versions (e.g., `v11_onprem`, `v11_cloud`, `v12_aws`, `etc`).

3. **Dependency Installation:**
   - The action upgrades `pip` to the latest version.
   - Installs necessary packages:
     - Runs `pip install .[pandas,dev]` to install the repository with extra dependencies.
     - Executes `pip install pytest pytest-xdist` to install testing frameworks.

4. **Execute Tests:**
   - Utilizes `pytest` to run all tests located in the `Test/` directory, ensuring that the TM1 objects integrate correctly without issues.

**Optional Design Consideration:**

-**Multi-Environment Support:**
  - By leveraging the repository's environments feature, we can define and run the workflow across multiple TM1 versions.

-**Forking Considerations:**

- When a user forks the repository, the action retains the variable placeholders (without the actual secret values).
- This allows forked repositories to carry over the workflow structure, so contributors can set up their own environment variables and run the action within their own TM1 environments.

---
**Deployment Diagram**
```mermaid
flowchart LR
    subgraph GitHub_Action[GitHub Action Workflow]
        A[GitHub Action]
    end

    subgraph External_Repos[External TM1 GitHub Repositories]
        B1[TM1 Repo for v11_onprem]
        B2[TM1 Repo for v11_cloud]
        B3[TM1 Repo for v12_aws]
    end

    subgraph TM1_Environments[TM1 Models]
        C1[TM1 Model v11_onprem]
        C2[TM1 Model v11_cloud]
        C3[TM1 Model v12_aws]
    end

    A --> B1
    A --> B2
    A --> B3

    A --> C1
    A --> C2
    A --> C3

    style GitHub_Action fill:#f9f,stroke:#333,stroke-width:1px
    style External_Repos fill:#bbf,stroke:#333,stroke-width:1px
    style TM1_Environments fill:#bfb,stroke:#333,stroke-width:1px
```

#### **Flow Diagram**

```mermaid
flowchart TD
    Start([Start GitHub Action])

    Start --> LoopEnvs[Iterate Through Environments]

    subgraph Environment Loop
        LoopEnvs --> ReadVars[Pull TM1 Objects from Environment-Specific External Repo if required]
        ReadVars --> PullObjects[Convert Details to config.ini]
        PullObjects --> ConvertConfig[Read Environment Variables]
        ConvertConfig --> UpgradePip[Upgrade pip]
        UpgradePip --> InstallDeps[Install Dependencies]
        InstallDeps --> IntegrateTM1[Pull TM1 Objects from GitHub into TM1 Model]
        IntegrateTM1 --> ExecuteTests[Execute pytest Tests]
        ExecuteTests --> CollectResults[Collect Test Results]
    end

    CollectResults --> CheckNextEnv{More Environments?}
    CheckNextEnv -- Yes --> LoopEnvs
    CheckNextEnv -- No --> End([End Action])
```
    

I think this would allow us (Cubewise) to test it on our end but it will also allow any contributor to fork it and adjust the variable/environments to their specific needs.
Please share your thoughts as this is not written in stone by any means!

Cheers,

nico
Thank you, @nicolasbisurgi, for sharing your thoughts. The ability to test different TM1 versions and platforms is very beneficial and that it should be possible to use the pipeline in forks is very nice. 
I'm not sure if I would use it in a fork, as I have a local development environment with a running TM1 server which is very fast and handy.

A general question first: Have you considered running the TM1 server in a container? Then you could use this image for your testing job and it would simplify a lot.

To step 1)
Is it planned to change the behavior that each test sets up its required TM1 objects? At the moment it is very easy to set up an empty tm1 database locally and start developing. This is very easy and you can run the tests again and again without worrying about the current state of the test database. This means more overhead and longer execution times (especially if you run tests single-threaded). If you run the tests in parallel, this is very useful because you can create separate TM1 objects for each test. 

Step 2)
I am not familiar with Github Actions. But is it possible to save the TM1_CONNECTION_DETAILS as an environment variable in your runner? So you can avoid storing credentials in Github variables?

Cheers,

onefloid

Hi @onefloid,

Thank you for your feedback.

**Regarding your observation on Step 1:** I apologize for the oversight on my part. I was under the impression that we needed a pre-loaded TM1 instance to run this. However, I ran into Marius yesterday, and he confirmed that an empty instance should suffice.

**Regarding your observation on Step 2:** As far as I know, GitHub Actions are non-persistent, so environment variables need to be set on every run.

Aside from that, I think adopting the following practice would be beneficial: the general workflow for TM1 contributors is to fork the repository, create a new branch with their changes and respective test cases, and then raise a PR. However, if they set up the values for the two variables (`TM1_CONNECTION_DETAILS` and `TM1_CONNECTION_PASSWORD`) for their respective environments, contributors could test locally before submitting the PR. My understanding is that when you clone a repository, the environments and variables are passed but without their original values, so you can fill them with your specific connection settings. This would be really beneficial as it would better distribute the load of testing, and I believe GitHub secrets provide sufficient security.

**Regarding the use of TM1 Server as a container:** I would love to explore that option, but I'm not sure how that's possible. If you have experience with this, please share it so we can see how we can incorporate it into the workflow.

Let me know your thoughts. I'll be available to work on this starting mid next week I think.

Cheers!

nico


Fix patch:
diff --git a/Tests/_gh_README.md b/Tests/_gh_README.md
new file mode 100644
index 00000000..ef532492
--- /dev/null
+++ b/Tests/_gh_README.md
@@ -0,0 +1,329 @@
+# TM1Py GitHub Actions CI Pipeline
+
+## Overview
+
+We have implemented a continuous integration (CI) pipeline using GitHub Actions to automate the testing of TM1Py pull requests. This pipeline ensures that code changes are thoroughly tested across multiple TM1 environments before being merged. The key features of this new setup include:
+
+- **Automated Testing**: Runs tests automatically for all defined TM1 environments whenever the workflow is triggered.
+- **Dynamic Configuration**: Generates `config.ini` files dynamically based on environment-specific variables and secrets.
+- **Parallel Execution**: Executes tests in parallel across different environments, ensuring efficient use of resources.
+- **Secure Handling of Secrets**: Manages sensitive information securely using GitHub Environments and Secrets.
+
+---
+
+## Workflow Description
+
+The new GitHub Actions workflow:
+
+- **Identifies All Defined Environments**: Automatically attempts to run tests for all environments specified in the workflow matrix.
+- **Checks for Required Variables**: Skips environments where the necessary variables (`TM1_CONNECTION` and `TM1_CONNECTION_SECRET`) are not defined.
+- **Generates Configuration Files**: Creates a `config.ini` file for each environment using environment-specific variables and secrets.
+- **Runs Tests in Isolation**: Executes tests for each environment in isolated runners to prevent interference.
+- **Reports Results**: Uploads test results and, optionally, posts comments to pull requests with test outcomes.
+
+---
+
+## Architecture and Flow Diagram
+
+Below is a flow diagram illustrating the CI pipeline's architecture and execution flow:
+
+```mermaid
+graph LR
+A[Workflow Triggered] --> B{Environments in Matrix}
+B --> |For Each Environment| C[Check for Required Variables]
+C --> |Variables Present| D[Generate config.ini]
+C --> |Variables Missing| E[Skip Job]
+D --> F[Install Dependencies]
+F --> G[Run Tests]
+G --> H[Upload Test Results]
+G --> I{Tests Passed?}
+I --> |Yes| J[Optional: Post Comment to PR]
+I --> |No| K[Fail the Job]
+```
+
+- **A**: The workflow is triggered manually or by an event (e.g., pull request).
+- **B**: The workflow defines a matrix of environments to test.
+- **C**: Each job checks if the required variables are present.
+- **D**: If variables are present, `config.ini` is generated.
+- **E**: If variables are missing, the job is skipped.
+- **F**: Dependencies are installed, including any extras needed for testing.
+- **G**: Tests are run using `pytest`.
+- **H**: Test results are uploaded as artifacts.
+- **I**: Checks if tests passed.
+- **J**: Optionally posts a comment to the pull request with results.
+- **K**: The job fails if tests did not pass.
+
+---
+
+## Repository Setup
+
+### 1. Defining Environments
+
+Create GitHub Environments for each TM1 instance/version you want to test against (there is no expected naming conversion):
+
+- **Environments**:
+  - `tm1-11-onprem`
+  - `tm1-11-cloud`
+  - `tm1-12-mcsp`
+  - `tm1-12-paaas`
+  - `tm1-12-onprem`
+
+### 2. Setting Environment Variables and Secrets
+
+For each environment, define the necessary variables and secrets (naming on the variables MUST be respected):
+
+- **Variables (`vars`)**:
+  - `TM1_CONNECTION`: A JSON string with non-sensitive connection parameters.
+
+- **Secrets (`secrets`)**:
+  - `TM1_CONNECTION_SECRET`: A JSON string with sensitive connection parameters (e.g., passwords, API keys).
+
+#### Sample Variables and Secrets for multiple types of environments
+
+**Environment:** `tm1-11-onprem`
+
+- **Variable (`TM1_CONNECTION`):**
+
+  ```json
+  {
+    "address": "localhost",
+    "port": "8001",
+    "user": "admin",
+    "ssl": "true",
+    "session_context": "gh_unittest"
+  }
+  ```
+
+- **Secret (`TM1_CONNECTION_SECRET`):**
+
+  ```json
+  {
+    "password": "apple"
+  }
+  ```
+
+**Environment:** `tm1-11-cloud`
+
+- **Variable (`TM1_CONNECTION`):**
+
+  ```json
+  {
+    "base_url": "https://server.planning-analytics.ibmcloud.com/tm1/api/tm1",
+    "user": "server01_tm1_automation",
+    "namespace": "LDAP",
+    "ssl": true,
+    "verify": true,
+    "async_requests_mode": true,
+    "session_context": "gh_unittest"
+  }
+  ```
+
+- **Secret (`TM1_CONNECTION_SECRET`):**
+
+  ```json
+  {
+    "password": "<NonInteractiveAccountPwd>"
+  }
+  ```
+
+**Environment:** `tm1-12-mscp`
+
+- **Variable (`TM1_CONNECTION`):**
+
+  ```json
+    {
+        "base_url": "https://us-east-1.planninganalytics.saas.ibm.com/api/<TenantId>/v0/tm1/<DatabaseName>/",
+        "user": "apikey",
+        "password": "<TheActualApiKey>",
+        "async_requests_mode": true,
+        "ssl": true,
+        "verify": true,
+        "session_context": "gh_unittest"
+    }
+  ```
+
+- **Secret (`TM1_CONNECTION_SECRET`):**
+
+  ```json
+  {
+    "password": "<TheActualApiKey>"
+  }
+  ```
+
+### 3. Workflow File
+
+Create a workflow file `.github/workflows/tm1py-integration-tests.yml` with the following content:
+
+```yaml
+name: TM1Py Integration Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      environments:
+        description: 'JSON array of environments to test (e.g., ["tm1-11-onprem", "tm1-11-cloud"])'
+        required: true
+        default: '["tm1-11-onprem","tm1-11-cloud","tm1-12-mcsp"]'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        environment: ${{ fromJson(github.event.inputs.environments) }}
+    environment: ${{ matrix.environment }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.x'
+
+      - name: Install dependencies
+        run: |
+          pip install -e .[unit_test]
+
+      - name: Retrieve TM1 Connection Details
+        id: get-connection
+        run: echo "Retrieving TM1 connection details"
+        env:
+          TM1_CONNECTION: ${{ vars.TM1_CONNECTION }}
+          TM1_CONNECTION_SECRET: ${{ secrets.TM1_CONNECTION_SECRET }}
+
+      - name: Generate config.ini
+        run: |
+          python Tests/resources/generate_config.py
+        env:
+          TM1_CONNECTION: ${{ vars.TM1_CONNECTION }}
+          TM1_CONNECTION_SECRET: ${{ secrets.TM1_CONNECTION_SECRET }}
+
+      - name: Run tests
+        run: pytest Tests/
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v3
+        with:
+          name: test-results-${{ matrix.environment }}
+          path: Tests/test-reports/
+
+      # Optional: Post results to PR
+      - name: Post comment to PR
+        if: always() && github.event_name == 'workflow_dispatch' && github.event.pull_request
+        uses: actions/github-script@v6
+        with:
+          script: |
+            github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.payload.pull_request.number,
+              body: 'Tests completed for environment: ${{ matrix.environment }}. Check artifacts for details.'
+            })
+```
+
+---
+
+## Parallel Execution of Multiple Environments
+
+### How It Works
+
+- **Matrix Strategy**: The workflow uses a matrix strategy to define jobs for each environment.
+- **Parallel Jobs**: Each environment runs in its own job, which can execute in parallel with others.
+- **Isolation**: Jobs are isolated from each other, each with its own runner and workspace.
+
+### Ensuring Isolation
+
+- **Separate Runners**: Each job runs on a separate virtual machine provided by GitHub Actions.
+- **Unique Workspaces**: Each job has its own workspace and filesystem.
+- **Environment Variables and Secrets**: Scoped to each job's environment, ensuring no cross-contamination.
+
+### Benefits
+
+- **Efficiency**: Tests for different environments run simultaneously, reducing total execution time.
+- **No Interference**: Jobs do not interfere with each other, preventing conflicts in configuration files or resources.
+- **Scalability**: Easily add or remove environments without affecting the workflow's structure.
+
+---
+
+## Additional Information
+
+### Generating `config.ini`
+
+A script `Tests/resources/generate_config.py` is used to create the `Tests/config.ini` file dynamically:
+
+```python
+import json
+import os
+
+tm1_connection = os.environ.get('TM1_CONNECTION')
+tm1_connection_secret = os.environ.get('TM1_CONNECTION_SECRET')
+
+config_content = '[tm1srv01]\n'
+
+if tm1_connection:
+    conn_data = json.loads(tm1_connection)
+    for key, value in conn_data.items():
+        config_content += f"{key}={value}\n"
+
+if tm1_connection_secret:
+    secret_data = json.loads(tm1_connection_secret)
+    for key, value in secret_data.items():
+        config_content += f"{key}={value}\n"
+
+with open('Tests/config.ini', 'w') as f:
+    f.write(config_content)
+```
+
+### Updating `setup.py`
+
+Ensure that all necessary dependencies are included:
+
+```python
+install_requires=[
+    'ijson',
+    'requests',
+    'pytz',
+    'requests_negotiate_sspi;platform_system=="Windows"',
+    'mdxpy>=1.3.1',
+    'networkx'],
+extras_require={
+    "pandas": ["pandas"],
+    "dev": [
+        "pytest",
+        "pytest-xdist"
+    ],
+    "unit_test": [
+        "pandas",
+        "pytest",
+        "pytest-xdist",
+        "python-dateutil"
+    ]
+},
+```
+
+### Security Considerations
+
+- **Secrets Management**: Use GitHub Secrets to store sensitive information securely.
+- **Access Control**: Secrets are only available to jobs that specify the corresponding environment.
+- **Prevent Exposure**: Ensure that secrets are not printed in logs or exposed in any artifacts.
+
+### Best Practices
+
+- **Testing in Isolation**: Each test run is isolated to prevent interference between environments.
+- **Resource Management**: Be mindful of GitHub Actions' concurrency limits and manage environments accordingly.
+- **Documentation**: Keep documentation updated to reflect changes in the CI pipeline and testing procedures.
+
+---
+
+## Further Reading
+
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+- [TM1Py Documentation](https://tm1py.readthedocs.io/en/latest/)
+- [Python Packaging Guide](https://packaging.python.org/tutorials/packaging-projects/)
+- [Secure Secrets Management in GitHub Actions](https://docs.github.com/en/actions/security-guides/encrypted-secrets)
+
+---
+
+**Note**: Remember to keep your repository's documentation up to date with these changes, and communicate with your team to ensure everyone is aware of the new testing procedures.
\ No newline at end of file
diff --git a/Tests/resources/generate_config.py b/Tests/resources/generate_config.py
new file mode 100644
index 00000000..775fdaff
--- /dev/null
+++ b/Tests/resources/generate_config.py
@@ -0,0 +1,20 @@
+import json
+import os
+
+tm1_connection = os.environ.get('TM1_CONNECTION')
+tm1_connection_secret = os.environ.get('TM1_CONNECTION_SECRET')
+
+config_content = '[tm1srv01]\n'
+
+if tm1_connection:
+    conn_data = json.loads(tm1_connection)
+    for key, value in conn_data.items():
+        config_content += f"{key}={value}\n"
+
+if tm1_connection_secret:
+    secret_data = json.loads(tm1_connection_secret)
+    for key, value in secret_data.items():
+        config_content += f"{key}={value}\n"
+
+with open('Tests/config.ini', 'w') as f:
+    f.write(config_content)
\ No newline at end of file
diff --git a/setup.py b/setup.py
index c2d8e756..79a41336 100644
--- a/setup.py
+++ b/setup.py
@@ -47,6 +47,12 @@
         "dev": [
             "pytest",
             "pytest-xdist"
+        ],
+        "unit_test": [
+            "pandas",
+            "pytest",
+            "pytest-xdist",
+            "python-dateutil"
         ]
     },
     python_requires='>=3.6',


Test patch:
diff --git a/.github/workflows/unit_test_pipeline.yml b/.github/workflows/unit_test_pipeline.yml
new file mode 100644
index 00000000..6e7c9aeb
--- /dev/null
+++ b/.github/workflows/unit_test_pipeline.yml
@@ -0,0 +1,66 @@
+name: TM1Py Integration Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      environments:
+        description: 'JSON array of environments to test (e.g., ["tm1-11-onprem", "tm1-11-cloud"])'
+        required: true
+        default: '["tm1-11-onprem","tm1-11-cloud","tm1-12-mcsp"]'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        environment: ${{ fromJson(github.event.inputs.environments) }}
+    environment: ${{ matrix.environment }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.x'
+
+      - name: Install dependencies
+        run: |
+          pip install -e .[unit_test]
+
+      - name: Retrieve TM1 Connection Details
+        id: get-connection
+        run: echo "Retrieving TM1 connection details"
+        env:
+          TM1_CONNECTION: ${{ vars.TM1_CONNECTION }}
+          TM1_CONNECTION_SECRET: ${{ secrets.TM1_CONNECTION_SECRET }}
+
+      - name: Generate config.ini
+        run: |
+          python Tests/resources/generate_config.py
+        env:
+          TM1_CONNECTION: ${{ vars.TM1_CONNECTION }}
+          TM1_CONNECTION_SECRET: ${{ secrets.TM1_CONNECTION_SECRET }}
+
+      - name: Run tests
+        run: pytest Tests/
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v3
+        with:
+          name: test-results-${{ matrix.environment }}
+          path: Tests/test-reports/
+
+      # Optional: Post results to PR
+      - name: Post comment to PR
+        if: always() && github.event_name == 'workflow_dispatch' && github.event.pull_request
+        uses: actions/github-script@v6
+        with:
+          script: |
+            github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.payload.pull_request.number,
+              body: 'Tests completed for environment: ${{ matrix.environment }}. Check artifacts for details.'
+            })
\ No newline at end of file