Repository: microsoft/data-factory-testing-framework (language: python)

Issue:
Improve docs
```[tasklist]
### Tasks
- [ ] https://github.com/microsoft/data-factory-testing-framework/issues/36
- [ ] https://github.com/microsoft/data-factory-testing-framework/issues/26
- [ ] https://github.com/microsoft/data-factory-testing-framework/issues/43
- [ ] https://github.com/microsoft/data-factory-testing-framework/issues/44
```


Discussion / hints:
@jayachithra, can you please detail which aspects of the docs you wanted to address on this issue? Reason is we have several stories for improving the docs and we are trying to organize them better. Thanks in advance!

Fix patch:
diff --git a/.markdownlint.yaml b/.markdownlint.yaml
index 33d83342..7d5d927f 100644
--- a/.markdownlint.yaml
+++ b/.markdownlint.yaml
@@ -1,3 +1,3 @@
 ---
 MD013:
-  line_length: 120
+  line_length: 400
diff --git a/README.md b/README.md
index 1a2cf6bb..c7697bd8 100644
--- a/README.md
+++ b/README.md
@@ -1,186 +1,86 @@
 # Data Factory - Testing Framework
 
-A test framework that allows you to write unit and functional tests for Data Factory
-pipelines against the git integrated json resource files.
-
-Supporting currently:
-
-* [Fabric Data Factory](https://learn.microsoft.com/en-us/fabric/data-factory/)
-* [Azure Data Factory v2](https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?tabs=data-factory)
-
-Planned:
-
-* [Azure Synapse Analytics](https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?context=%2Fazure%2Fsynapse-analytics%2Fcontext%2Fcontext&tabs=data-factory/)
-
-## Disclaimer
-
-This unit test framework is not officially supported.
-It is currently in an experimental state and has not been tested with every single data factory resource.
-It should support all activities out-of-the-box but has not been thoroughly tested,
-please report any issues in the issues section and include an example of the pipeline that is not working as expected.
-
-If there's a lot of interest in this framework, then we will continue to improve it and move it to a production-ready state.
+A stand-alone test framework that allows to write unit tests for Data Factory pipelines on [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-factory/) and [Azure Data Factory](https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?tabs=data-factory).
 
 ## Features
 
-Goal: Validate that the evaluated pipeline configuration with its expressions is behaving as expected on runtime.
-
-1. Evaluate expressions with their functions and arguments instantly by using the framework's internal expression parser.
-2. Test a pipeline or activity against any state to assert the expected outcome.
-   A state can be configured with pipeline parameters, global parameters, variables and activity outputs.
-3. Simulate a pipeline run and evaluate the execution flow and outcome of each activity.
-4. Dynamically supports all activity types with all their attributes.
-
-> Pipelines and activities are not executed on any Data Factory environment,
-> but the evaluation of the pipeline configuration is validated locally.
-> This is different from the "validation" functionality present in the UI,
-> which only validates the syntax of the pipeline configuration.
-
-## Why
-
-Data Factory does not support unit testing out of the box.
-The only way to validate your changes is through manual testing or running e2e tests against a deployed data factory.
-These tests are great to have, but miss the following benefits that unit tests, like using this unit test framework, provide:
-
-* Shift left with immediate feedback on changes - Evaluate any individual data factory resource
-  (pipelines, activities, triggers, datasets, linked services etc..), including (complex) expressions
-* Allows testing individual resources (e.g. activity) for many different input values to cover more scenarios.
-* Less issues in production - due to the fast nature of writing and running unit tests,
-  you will write more tests in less time and therefore have a higher test coverage.
-  This means more confidence in new changes, fewer risks in breaking existing features (regression tests),
-  and thus far fewer issues in production.
-
-> Even though Data Factory is UI-driven writing unit tests, and might not be in the nature of it.
-> How can you be confident that your changes will work as expected,
-> and that existing pipelines will not break, without writing unit tests?
-
-## Getting started
-
-### Start writing tests with a Dev Container
-
-To get started using the tests, refer to the following [README](./examples/README.md)
-
-### Start writing tests without a Dev Container
-
-1. Set up an empty Python project with your favorite testing library
-2. Install the dotnet runtime from [here](https://dotnet.microsoft.com/en-us/download/dotnet/8.0).
-   Using only the runtime and not the SDK should be sufficient.
-   This is required to run some expression functions on dotnet just like in Data Factory.
-3. Set up an empty Python project with your favorite testing library
-   More information:
-   [docs_Setup](/docs/environment_setup/unit_test_setup.md)
-
-4. Install the package using your preferred package manager:
+The framework evaluates pipeline and activity definitions which can be asserted. It does so by providing the following features:
 
-    Pip: `pip install data-factory-testing-framework`
+1. Evaluate expressions by using the framework's internal expression parser. It supports all the functions and arguments that are available in the Data Factory expression language.
+2. Test an activity with a specific state and assert the evaluated expressions.
+3. Test a pipeline run by verifying the execution flow of activities for specific input parameters and assert the evaluated expressions of each activity.
 
-5. Create a Folder in your project and copy the JSON Files with the pipeline definitions locally.
+> The framework does not support running the actual pipeline. It only gives you the ability to test the pipeline and activity definitions.
 
-   More information:
-   [Json Guidance](/docs/environment_setup/json_pipeline_files.md)
+### High-level example
 
-6. Start writing tests
+Given a `WebActivity` with a `typeProperties.url` property containing the following expression:
 
-## Features - Examples
-
-The samples seen below are the _only_ code that you need to write! The framework will take care of the rest.
+```datafactoryexpression
+@concat(pipeline().globalParameters.baseUrl, variables('JobName'))
+```
 
-1. Evaluate activities (e.g. a WebActivity that calls Azure Batch API)
+A simple test to validate that the concatenation is working as expected could look like this:
 
-    ```python
+```python
     # Arrange
-    activity: Activity = pipeline.get_activity_by_name("Trigger Azure Batch Job")
+    activity = pipeline.get_activity_by_name("webactivity_name")
     state = PipelineRunState(
         parameters=[
             RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
-            RunParameter(RunParameterType.Pipeline, "JobId", "123"),
         ],
         variables=[
-            PipelineRunVariable("JobName", "Job-123"),
+            PipelineRunVariable("Path", "some-path"),
         ])
-    state.add_activity_result("Get version", DependencyCondition.SUCCEEDED, {"Version": "version1"})
 
     # Act
     activity.evaluate(state)
 
     # Assert
-    assert "https://example.com/jobs" == activity.type_properties["url"].value
-    assert "POST" == activity.type_properties["method"].value
-    body = activity.type_properties["body"].get_json_value()
-    assert "123" == body["JobId"]
-    assert "Job-123" == body["JobName"]
-    assert "version1" == body["Version"]
+    assert "https://example.com/some-path" == activity.type_properties["url"].value
    ```
 
-2. Evaluate Pipelines and test the flow of activities given a specific input
+## Why
 
-    ```python
-    # Arrange
-    pipeline: PipelineResource = test_framework.repository.get_pipeline_by_name("batch_job")
-
-    # Runs the pipeline with the provided parameters
-    activities = test_framework.evaluate_pipeline(pipeline, [
-        RunParameter(RunParameterType.Pipeline, "JobId", "123"),
-        RunParameter(RunParameterType.Pipeline, "ContainerName", "test-container"),
-        RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
-    ])
-
-    set_variable_activity: Activity = next(activities)
-    assert set_variable_activity is not None
-    assert "Set JobName" == set_variable_activity.name
-    assert "JobName" == activity.type_properties["variableName"]
-    assert "Job-123" == activity.type_properties["value"].value
-
-    get_version_activity = next(activities)
-    assert get_version_activity is not None
-    assert "Get version" == get_version_activity.name
-    assert "https://example.com/version" == get_version_activity.type_properties["url"].value
-    assert "GET" == get_version_activity.type_properties["method"]
-    get_version_activity.set_result(DependencyCondition.Succeeded,{"Version": "version1"})
-
-    create_batch_activity = next(activities)
-    assert create_batch_activity is not None
-    assert "Trigger Azure Batch Job" == create_batch_activity.name
-    assert "https://example.com/jobs" == create_batch_activity.type_properties["url"].value
-    assert "POST" == create_batch_activity.type_properties["method"]
-    body = create_batch_activity.type_properties["body"].get_json_value()
-    assert "123" == body["JobId"]
-    assert "Job-123" == body["JobName"]
-    assert "version1" == body["Version"]
-
-    with pytest.raises(StopIteration):
-        next(activities)
-    ```
-
-> See the [Examples](/examples) folder for more samples
-
-## Registering missing expression functions
-
-As the framework is interpreting expressions containing functions, these functions are implemented in the framework,
-but there may be bugs in some of them. You can override their implementation through:
+Data Factory does not support unit testing, nor testing of pipelines locally. Having integration and e2e tests running on an actual Data Factory instance is great, but having unit tests on top of them provides additional means of quick iteration, validation and regression testing. Unit testing with the _Data Factory Testing Framework_ has the following benefits:
 
-```python
-   FunctionsRepository.register("concat", lambda arguments: "".join(arguments))
-   FunctionsRepository.register("trim", lambda text, trim_argument: text.strip(trim_argument[0]))
-```
+* Runs locally with immediate feedback
+* Easier to cover a lot of different scenarios and edge cases
+* Regression testing
+
+## Concepts
+
+The following pages go deeper into different topics and concepts of the framework to help in getting you started.
+
+### Basic
+
+1. [Repository setup](docs/basic/repository_setup.md)
+2. [Installing and initializing the framework](docs/basic/installing_and_initializing_framework.md)
+3. [State](docs/basic/state.md)
+4. [Activity testing](docs/basic/activity_testing.md)
+5. [Pipeline testing](docs/basic/pipeline_testing.md)
+
+> If you are a not that experienced with Python, you can follow the [Getting started](docs/basic/getting_started.md) guide to get started with the framework.
+
+### Advanced
 
-## Tips
+1. [Debugging your activities and pipelines](docs/advanced/debugging.md)
+2. [Development workflow](docs/advanced/development_workflow.md)
+3. [Overriding expression functions](docs/advanced/overriding_expression_functions.md)
+4. [Framework internals](docs/advanced/framework_internals.md)
 
-1. After parsing a data factory resource file, you can use the debugger to easily discover which classes are actually
-   initialized so that you can cast them to the correct type.
+## Examples
 
-## Recommended development workflow for Azure Data Factory v2
+More advanced examples demonstrating the capabilities of the framework:
 
-* Use ADF Git integration
-* Use UI to create a feature branch, build the initial pipeline, and save it to the feature branch
-* Pull feature branch locally
-* Start writing tests unit and functional tests, run them locally for immediate feedback, and fix bugs
-* Push changes to the feature branch
-* Test the new features manually through the UI in a sandbox environment
-* Create PR, which will run the tests in the CI pipeline
-* Approve PR
-* Merge to main and start deploying to dev/test/prod environments
-* Run e2e tests after each deployment to validate all happy flows work on that specific environment
+Fabric:
+
+1. [Batch job example](examples/fabric/batch_job/README.md)
+
+Azure Data Factory:
+
+1. [Copy blobs example](examples/data_factory/copy_blobs/README.md)
+2. [Batch job example](examples/data_factory/batch_job/README.md)
 
 ## Contributing
 
@@ -196,6 +96,15 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
 
+## Disclaimer
+
+This unit test framework is not officially supported.
+It is currently in an experimental state and has not been tested with every single data factory resource.
+It should support all activities out-of-the-box but has not been thoroughly tested,
+please report any issues in the issues section and include an example of the pipeline that is not working as expected.
+
+If there's a lot of interest in this framework, then we will continue to improve it and move it to a production-ready state.
+
 ## Trademarks
 
 This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
diff --git a/docs/advanced/debugging.md b/docs/advanced/debugging.md
new file mode 100644
index 00000000..a8b48fbc
--- /dev/null
+++ b/docs/advanced/debugging.md
@@ -0,0 +1,3 @@
+# Debugging
+
+As the framework is dynamically parsing and interpreting data factory resource files, it can be challenging to identify which objects you are working with. It is recommended to use the debugger during development of your tests to get a better idea of what activities are being returned and to understand the structure of the activity and its properties.
diff --git a/docs/advanced/development_workflow.md b/docs/advanced/development_workflow.md
new file mode 100644
index 00000000..eba3eb50
--- /dev/null
+++ b/docs/advanced/development_workflow.md
@@ -0,0 +1,12 @@
+# Recommended development workflow for Azure Data Factory v2
+
+* Use ADF Git integration
+* Use UI to create a feature branch, build the initial pipeline, and save it to the feature branch
+* Pull feature branch locally
+* Start writing unit and functional tests, run them locally for immediate feedback, and fix bugs
+* Push changes to the feature branch
+* Test the new features manually through the UI in a sandbox environment
+* Create PR, which will run the tests in the CI pipeline
+* Approve PR
+* Merge to main and start deploying to dev/test/prod environments
+* Run e2e tests after each deployment to validate all happy flows work on that specific environment
diff --git a/docs/advanced/framework_internals.md b/docs/advanced/framework_internals.md
new file mode 100644
index 00000000..9306c5dc
--- /dev/null
+++ b/docs/advanced/framework_internals.md
@@ -0,0 +1,3 @@
+# Framework internals
+
+This page will be used to describe the internals of the testing framework. It will be used to document the architecture, design decisions, and implementation details of the framework.
diff --git a/docs/advanced/overriding_expression_functions.md b/docs/advanced/overriding_expression_functions.md
new file mode 100644
index 00000000..7f17725b
--- /dev/null
+++ b/docs/advanced/overriding_expression_functions.md
@@ -0,0 +1,9 @@
+# Overriding expression functions
+
+The framework interprets expressions containing functions, which are implemented within the framework, and they might contain bugs.
+You can override their implementation as illustrated below:
+
+```python
+   FunctionsRepository.register("concat", lambda arguments: "".join(arguments))
+   FunctionsRepository.register("trim", lambda text, trim_argument: text.strip(trim_argument[0]))
+```
diff --git a/docs/basic/getting_started.md b/docs/basic/getting_started.md
new file mode 100644
index 00000000..74f19397
--- /dev/null
+++ b/docs/basic/getting_started.md
@@ -0,0 +1,41 @@
+# Getting started
+
+This _getting started_ focuses on novice users who are not familiar with Python or package management. It guides the user through the process of downloading the data factory pipeline files, setting up a Python project and installing the framework so that they can start writing tests.
+
+>For experienced Python and package management users, you can skip this page and go directly to the [repository setup](repository_setup.md) page.
+
+## Data Factory pipeline files
+
+The framework is designed to work with the `json` files that define the data factory pipelines and activities. Files from the data factory environment can be downloaded from your data factory environment as described in the [repository setup](repository_setup.md) page.
+
+## Setting up the Python project
+
+For Visual Studio Code, the guidance is:
+
+1. Create a new folder alongside the data factory pipeline files for the test project called `tests`.
+2. Open the new folder in Visual Studio Code.
+3. Install the framework by installing the library the terminal with pip:
+
+   ```bash
+   pip install data-factory-testing-framework
+   ```
+
+4. Install pytest as testing library. All examples in this documentation are using pytest.
+
+   ```bash
+   pip install pytest
+   ```
+
+5. Download the pipeline files from data factory environment and place them in the project folder as described in the [repository setup](repository_setup.md) page.
+
+Additional resources:
+
+* [Get Started Tutorial for Python in Visual Studio Code](https://code.visualstudio.com/docs/python/python-tutorial)
+* [Integrated Terminal in Visual Studio Code](https://code.visualstudio.com/docs/terminal/basics)
+* [pytest: helps you write better programs — pytest documentation](https://docs.pytest.org/en/7.4.x/)
+
+Once the setup is finished, reading the following pages is recommended to learn how to write tests for the data factory:
+
+1. [Initializing the framework](installing_and_initializing_framework.md) (make sure to initialize the root folder of the framework with the path to the folder containing the pipeline definitions).
+2. [Activity testing](activity_testing.md)
+3. [Pipeline testing](pipeline_testing.md)
diff --git a/docs/basic/installing_and_initializing_framework.md b/docs/basic/installing_and_initializing_framework.md
new file mode 100644
index 00000000..dc0f251f
--- /dev/null
+++ b/docs/basic/installing_and_initializing_framework.md
@@ -0,0 +1,35 @@
+# Installing and initializing the framework
+
+## Installing the framework
+
+The framework is available as a Python package on PyPI. You can install it using your preferred package manager: [data-factory-testing-framework](https://pypi.org/project/data-factory-testing-framework/).
+
+## Initializing the framework
+
+To initialize the framework, you need to create a `TestFramework` instance. This instance is the entry point to the framework and provides access to the pipeline and activity definitions. Specify the type of data factory (i.e. Fabric or DataFactory for Azure Data Factory) and pass the path to the folder containing the pipeline definitions to the `TestFramework` constructor.
+
+```python
+from data_factory_testing_framework import TestFramework
+
+test_framework = TestFramework(
+        framework_type=TestFrameworkType.DataFactory,
+        root_folder_path='/factory',
+    )
+```
+
+The TestFramework will automatically load all the pipeline and activity definitions from the specified folder. It will make them available through the `repository` property. Pipelines can easily be retrieved by name:
+
+```python
+pipeline = test_framework.repository.get_pipeline_by_name("batch_job")
+```
+
+Activities can be retrieved from the pipeline by name:
+
+```python
+activity = pipeline.get_activity_by_name("webactivity_name")
+```
+
+See the following pages for more information on how to write tests for activities and pipelines:
+
+1. [Activity testing](activity_testing.md)
+2. [Pipeline testing](pipeline_testing.md)
diff --git a/docs/basic/repository_setup.md b/docs/basic/repository_setup.md
new file mode 100644
index 00000000..817836fd
--- /dev/null
+++ b/docs/basic/repository_setup.md
@@ -0,0 +1,23 @@
+# Repository setup
+
+## Git integration
+
+To be able to write tests for data factory, the pipeline and activity definitions need to be available. The recommended way to do this is to sync the Data Factory instance to a git repository, so that a `tests` folder can be created in the same repository and write tests for your data factory. The git integration process can be found here:
+
+1. [Fabric - Git integration process](https://learn.microsoft.com/fabric/cicd/git-integration/git-integration-process)
+2. [Azure Data Factory - Git integration process](https://learn.microsoft.com/azure/data-factory/source-control)
+
+### Alternative for Azure Data Factory
+
+To download a single JSON file for testing purposes,  follow these steps:
+
+1. Open the Data Factory instance, and open the pipeline to be tested.
+2. Click on the action ellipses
+3. Click "Download support files"
+4. Extract the zip file containing the pipeline definition in a folder of choice.
+
+> Remember the location of this folder, as it will be needed to initialize the framework.
+
+![Download support files](../images/download_support_files.png)
+
+Once the repository is set up, the framework can be installed and initialized as described in the [installing and initializing the framework](installing_and_initializing_framework.md) page.
diff --git a/docs/basic/state.md b/docs/basic/state.md
new file mode 100644
index 00000000..08d9f028
--- /dev/null
+++ b/docs/basic/state.md
@@ -0,0 +1,49 @@
+# State
+
+The framework provides a `PipelineRunState` class that represents the state of a pipeline run. It's used to evaluate how an activity behaves for a given a specific input.
+
+## Interface
+
+The constructor of the `PipelineRunState` class has the following properties:
+
+* `parameters`: A list of `RunParameter` instances representing the input parameters of the pipeline run.
+* `variables`: A list of `PipelineRunVariable` instances representing the variables of the pipeline run.
+* `activity_results`: A dictionary of activity results that can be used to simulate the output of activities in the pipeline run.
+* `iteration_item`: An optional iteration item that can be used to simulate the iteration item of a ForEach activity.
+
+The `PipelineRunState` class has the following methods:
+
+* `add_activity_result`: Adds a result for an activity to the state. This is useful if an expression in an activity references another activities output, like: `@activity('another_activity_name').output.some_field`.
+* `set_iteration_item`: Sets the iteration item of the state. This is useful if an expression in an activity references the iteration item of a ForEach activity, like: `@item().JobId`.
+
+## Usage
+
+### Parameters and variables
+
+In the scenario where an activity has an expression that references a global parameter called `BaseUrl` and a variable called `JobName`, the `PipelineRunState` can be used to simulate the evaluation of the expression.
+
+```python
+state = PipelineRunState(
+    parameters=[
+        RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
+    ],
+    variables=[
+        PipelineRunVariable("JobName", "Job-123"),
+    ])
+```
+
+### Activity results
+
+In the scenario where an activity has an expression that references the output of another activity `activity('another_activity_name').output.some_field`, the `PipelineRunState` can be used to configure the output of the `another_activity_name` activity.
+
+```python
+state.add_activity_result("another_activity_name", DependencyCondition.SUCCEEDED, {"some_field": "some_value!"})
+```
+
+### Iteration item
+
+In the scenario where an activity has an expression that references the iteration item of a ForEach activity `@item().JobId`, the `PipelineRunState` can be used to configure the iteration item.
+
+```python
+state.set_iteration_item({"JobId": "123"})
+```
diff --git a/docs/environment_setup/images/download_support_files.png b/docs/environment_setup/images/download_support_files.png
deleted file mode 100644
index 8d250cc1..00000000
Binary files a/docs/environment_setup/images/download_support_files.png and /dev/null differ
diff --git a/docs/environment_setup/images/env_variable.png b/docs/environment_setup/images/env_variable.png
deleted file mode 100644
index 36437f3a..00000000
Binary files a/docs/environment_setup/images/env_variable.png and /dev/null differ
diff --git a/docs/environment_setup/images/installing_poetry_power_shell.png b/docs/environment_setup/images/installing_poetry_power_shell.png
deleted file mode 100644
index 4ad51fb8..00000000
Binary files a/docs/environment_setup/images/installing_poetry_power_shell.png and /dev/null differ
diff --git a/docs/environment_setup/images/new_project.png b/docs/environment_setup/images/new_project.png
deleted file mode 100644
index 180e3eb0..00000000
Binary files a/docs/environment_setup/images/new_project.png and /dev/null differ
diff --git a/docs/environment_setup/images/pipinstall_poetry.png b/docs/environment_setup/images/pipinstall_poetry.png
deleted file mode 100644
index 61109757..00000000
Binary files a/docs/environment_setup/images/pipinstall_poetry.png and /dev/null differ
diff --git a/docs/environment_setup/images/poetry_framework.png b/docs/environment_setup/images/poetry_framework.png
deleted file mode 100644
index 460ace29..00000000
Binary files a/docs/environment_setup/images/poetry_framework.png and /dev/null differ
diff --git a/docs/environment_setup/images/power_shell_invoke.png b/docs/environment_setup/images/power_shell_invoke.png
deleted file mode 100644
index 78dba965..00000000
Binary files a/docs/environment_setup/images/power_shell_invoke.png and /dev/null differ
diff --git a/docs/environment_setup/json_pipeline_files.md b/docs/environment_setup/json_pipeline_files.md
deleted file mode 100644
index bb7961f7..00000000
--- a/docs/environment_setup/json_pipeline_files.md
+++ /dev/null
@@ -1,34 +0,0 @@
-# Data Factory - Testing Framework
-
-## JSON Files
-
-Ideally, you would integrate your project with a repository such as GIT.
-Once this is done you will have the JSON files for the Pipelines.
-If you need extra guidance on that the Microsoft Documentation provides very detailed information on how to do it:
-
-Fabric
-[Git integration process - Microsoft Fabric | Microsoft Learn](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process)
-
-ADF
-[Source control - Azure Data Factory | Microsoft Learn](https://learn.microsoft.com/azure/data-factory/source-control)
-
-### Manual steps for testing purposes
-
-If you want just download one JSON file for a small test purpose and check how the framework works,
-here are some ideas on how to do it:
-
-#### ADF
-
-Refer to the docs if needed: [Azure Data Factory Documentation - Azure Data Factory | Microsoft Learn](https://learn.microsoft.com/en-us/azure/data-factory/)
-
-[Download Azure Data Factory support files - Microsoft Community Hub](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/download-azure-data-factory-support-files/ba-p/1042848)
-
-1. Open your ADF Environment, and open the resource you want the JSON.
-
-2. Click on the action ellipses
-
-3. Click "Download support files" and select the download location in local storage
-
-4. The Zip file contains the JSON Definition
-
-   ![image](/docs/environment_setup/images/download_support_files.png)
diff --git a/docs/images/download_support_files.png b/docs/images/download_support_files.png
new file mode 100644
index 00000000..6e97e549
Binary files /dev/null and b/docs/images/download_support_files.png differ
diff --git a/pyproject.toml b/pyproject.toml
index 755f13cc..0b42fb3f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,7 +1,7 @@
 [tool.poetry]
 name = "data-factory-testing-framework"
 version = "0.0.1"
-description = "A unit test framework that allows you to write unit and functional tests for Data Factory pipelines against the git integrated json resource files."
+description = "A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric and Azure Data Factory."
 authors = ["Arjen Kroezen <arjenkroezen@microsoft.com>", "Yennifer Santos <ysantos@microsoft.com>", "Jaya Kumar <kumarjaya@microsoft.com>", "Leonard Herold <lherold@microsoft.com>"]
 readme = "README.md"
 license = "MIT"


Test patch:
diff --git a/docs/basic/activity_testing.md b/docs/basic/activity_testing.md
new file mode 100644
index 00000000..ff7d70fb
--- /dev/null
+++ b/docs/basic/activity_testing.md
@@ -0,0 +1,133 @@
+# Activity testing
+
+The framework provides a way to test activities in isolation. This is useful for testing the behavior of an activity without executing the entire pipeline.
+
+Make sure to have initialized the framework before writing tests. See [Installing and initializing the framework](installing-and-initializing-framework.md) for more information.
+
+Once the `TestFramework` instance is created, tests can be written for the activities. A test always follows the Arrange-Act-Assert pattern:
+
+1. **Arrange**: Get a reference to the activity to be tested and create a `PipelineRunState` instance with the input parameters and variables of the scenario that is being evaluated.
+2. **Act**: Call the `evaluate` method on the activity with the `PipelineRunState` instance. The framework will find all expressions in the activity and evaluate them using the provided state.
+3. **Assert**: Verify that the output of the activity matches the expected result.
+
+The framework automatically detects expressions and put them in a `DataFactoryElement` object. This object has a `value` property that contains the evaluated value of the expression. The `value` property can be used to assert the outcome.
+
+## Arrange-Act-Assert
+
+The following pipeline example is used to demonstrate how each steps of the arrange-act-assert pattern can be implemented for a single activity.
+
+```json
+{
+    "name": "trigger_job_pipeline",
+    "parameters": {
+        "JobId": {
+          "type": "String"
+        }
+    },
+    "variables": {
+        "JobName": "Job-123"
+    },
+    "activities": [
+      {
+        "name": "Trigger job",
+        "type": "WebActivity",
+        "typeProperties": {
+          "url": {
+            "type": "Expression",
+            "value": "@concat(pipeline().globalParameters.BaseUrl, '/jobs')"
+          },
+          "method": {
+            "value": "POST"
+          },
+          "body": {
+            "type": "Expression",
+            "value": {
+              "JobId": "@pipeline().parameters.JobId",
+              "JobName": "@variables('JobName')",
+              "Version": "@activity('Get version').output.Version"
+            }
+          }
+        }
+      }
+  ]
+}
+```
+
+Let's write a test for validating correct evaluation of the `url` property.
+
+## Arrange
+
+Get a reference to the activity to be tested using the `get_activity_by_name` method on the `Pipeline` instance.
+
+```python
+pipeline = test_framework.repository.get_pipeline_by_name("trigger_job_pipeline")
+activity = pipeline.get_activity_by_name("Trigger job")
+```
+
+Create a `PipelineRunState` instance with the input used in the `url` expression, so a globalParameter `BaseUrl` and a variable `JobName`.
+
+```python
+state = PipelineRunState(
+    parameters=[
+        RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
+    ],
+    variables=[
+        PipelineRunVariable("JobName", "Job-123"),
+    ])
+```
+
+> See [State](state.md) for a detailed explanation on how to use the `PipelineRunState` class.
+
+## Act
+
+Call the `evaluate` method on the `activity` with the `PipelineRunState` instance. This evaluates all expressions in the activity using the provided state and store the result in a `value` property.
+
+```python
+activity.evaluate(state)
+```
+
+The `evaluate` method might throw an exception if the expression is invalid or if the expression is missing required parameters, variables or activity outputs. Make sure to supply the required properties in the state.
+
+## Assert
+
+Verify that the output of the activity matches the expected result.
+
+```python
+assert "https://example.com/jobs" == activity.type_properties["url"].value
+```
+
+## Control activities
+
+For testing control activities like `IfCondition`, `ForEach`, `Switch` and `Until`, it is useful to validate that the condition expression is evaluating correctly.  Make sure to use type hints to know the available properties of the control activity.
+
+### Example
+
+Considering that the `IfCondition` expression looks like: `@equals(pipeline().parameters.JobId, '123')`, an example is provided for that particular expression as follow:
+
+```python
+# Arrange
+activity: IfConditionActivity = pipeline.get_activity_by_name("If condition")
+state = PipelineRunState(
+    parameters=[
+        RunParameter(RunParameterType.Pipeline, "JobId", "123"),
+    ],
+)
+
+# Act
+activity.evaluate(state)
+
+# Assert
+assert activity.expression.value == True
+```
+
+### Testing child activities
+
+To test the child activities of a control activity, firstly the control activity must be located, followed by the identification of its child activities. The testing procedure for the child activities is outlined in the previous section.
+
+```python
+# Arrange
+activity: IfConditionActivity = pipeline.get_activity_by_name("If condition")
+child_activity = activity.if_true_activities[0]  # Or loop through them by name
+```
+
+To evaluate the condition of a control activity in combination with the child activities, then write a pipeline test as described in [Pipeline testing](pipeline_testing.md).
diff --git a/docs/basic/pipeline_testing.md b/docs/basic/pipeline_testing.md
new file mode 100644
index 00000000..8db22675
--- /dev/null
+++ b/docs/basic/pipeline_testing.md
@@ -0,0 +1,138 @@
+# Pipeline testing
+
+The framework provides a way to evaluate a pipeline and validate the execution flow of each activity and ensure that the outcome of one activity is correctly passed to the next activity.
+
+Get acquainted with the [Activity testing](activity_testing.md) page before proceeding with pipeline testing, because similar concepts are used.
+
+Make sure to have initialized the framework before writing tests. See [Installing and initializing the framework](installing_and_initializing_framework.md) for more information.
+
+Once the `TestFramework` instance is created,  tests can be written for the pipelines. A test always follows the Arrange-Act-Assert pattern:
+
+1. **Arrange**: Get a reference to the pipeline to be tested
+2. **Act**: Call the `evaluate_pipeline` method on the `TestFramework` instance with the pipeline and the input parameters of the scenario to be evaluated.
+3. **Assert**: Request the next activity from the returned generator and verify that the output of the activity matches the expected result. Repeat until all activities have been evaluated.
+
+## Arrange-Act-Assert
+
+The following pipeline example is used to demonstrate how each steps of the arrange-act-assert pattern can be implemented for a pipeline.
+
+```json
+{
+    "parameters": {
+        "JobId": {
+          "type": "String"
+        }
+    },
+    "variables": {
+        "JobName": ""
+    },
+    "activities": [
+      {
+        "name": "Set JobName",
+        "type": "SetVariable",
+        "typeProperties": {
+          "variableName": "JobName",
+          "value": {
+            "type": "Expression",
+            "value": "@concat('Job-', pipeline().parameters.JobId)"
+          }
+        }
+      },
+      {
+        "name": "Get version of job",
+        "type": "WebActivity",
+        "typeProperties": {
+          "url": {
+            "type": "Expression",
+            "value": "@concat(pipeline().globalParameters.BaseUrl, '/', variables('JobName'), '/version)"
+          },
+          "method": {
+            "value": "GET"
+          }
+        },
+        "dependsOn": [
+          {
+            "activity": "Set JobName",
+            "dependencyConditions": [
+              "Succeeded"
+            ]
+          }
+        ]
+      }
+  ]
+}
+```
+
+Let's write a test for validating the correct flow of activities, the correct setting of the variable and the correct evaluation of the `url` property.
+
+## Arrange
+
+Get a reference to the pipeline to be tested using the `get_pipeline_by_name` method on the `Repository` instance.
+
+```python
+pipeline = test_framework.repository.get_pipeline_by_name("batch_job")
+```
+
+## Act
+
+Start evaluating the pipeline with the requested parameters. The `evaluate_pipeline` method returns a generator that yields the activities in the order they are executed.
+
+```python
+activities = test_framework.evaluate_pipeline(pipeline, [
+    RunParameter(RunParameterType.Pipeline, "JobId", "123"),
+    RunParameter(RunParameterType.Global, "BaseUrl", "https://example.com"),
+])
+```
+
+## Assert
+
+Request the next activity from the returned generator and verify that the output of the activity matches the expected result. Repeat until all activities have been evaluated.
+
+```python
+set_variable_activity = next(activities)
+assert set_variable_activity is not None
+assert "Set JobName" == set_variable_activity.name
+assert "Job-123" == activity.type_properties["value"].value
+
+get_version_activity = next(activities)
+assert get_version_activity is not None
+assert "Get version of job" == get_version_activity.name
+assert "https://example.com/Job-123/version" == get_version_activity.type_properties["url"].value
+assert "GET" == get_version_activity.type_properties["method"]
+```
+
+### Activity output references
+
+If an activity output is being referenced (e.g. `activity('Get version of job').output.version`), make sure to set the result of the activity using the `set_result` method before requesting the next activity from the generator.
+
+```python
+get_version_activity.set_result(DependencyCondition.Succeeded, { "version", "1.0.0" })
+```
+
+The `next(activities)` method might throw an exception if the expression is invalid or if a property of an expression is not (yet) available in the state. Make sure that:
+
+1. Parameters are supplied
+2. Variables are being set correctly through the `SetVariable` activity
+3. Activity outputs being referenced are set through `activity.set_result` method
+4. Dependency conditions are met
+
+### Asserting end of pipeline execution
+
+When the generator has no more activities to return, it raises a `StopIteration` exception. This is a signal that all activities have been evaluated or that no activities meet the required dependency conditions. If the pipeline is expected to be finished, it can be asserted using the `pytest.raises` context manager:
+
+```python
+with pytest.raises(StopIteration):
+    next(activities)
+```
+
+## Control activities
+
+The following activities are `ControlActivities` and are not returned by the generator: `IfCondition`, `ForEach`, `Switch` and `Until`. The conditions of these activities are automatically evaluated and the child activities are evaluated and returned based on the result of the condition.
+
+## Execute child pipelines
+
+Child pipelines and their activities can be included in the evaluation of the parent pipeline by setting the `should_evaluate_child_pipelines` of the `TestFramework` constructor to `True`. The framework evaluates the parameters of the`ExecutePipelineActivity` and then evaluate the child pipeline with the provided parameters. This is useful to validate that different pipelines are working well together.
+
+```python
+test_framework = TestFramework(should_evaluate_child_pipelines=True)
+```
diff --git a/docs/environment_setup/unit_test_setup.md b/docs/environment_setup/unit_test_setup.md
deleted file mode 100644
index 87053309..00000000
--- a/docs/environment_setup/unit_test_setup.md
+++ /dev/null
@@ -1,79 +0,0 @@
-# Data Factory - Testing Framework
-
-## Setup
-
-This is an extra step for setup in case you need additional help to do it.
-The Unit Test does not need to rigorously follow these steps to work.
-You just need to start your Python project, choose your testing library, and start to code.
-Though, if you need additional assistance how to do it which library to use, and some code examples.
-Feel free to use this guide.
-
-### Step by Step
-
-If for example you are using Visual Studio Code:
-
-For more references: [Get Started Tutorial for Python in Visual Studio Code](https://code.visualstudio.com/docs/python/python-tutorial)
-
-1. Open your new Python project.
-
-   ![image](/docs/environment_setup/images/new_project.png)
-
-2. Pip install the following libraries from the terminal - data-factory-testing-framework:
-
-   References about the vs code terminal : [Integrated Terminal in Visual Studio Code](https://code.visualstudio.com/docs/terminal/basics)
-
-**Then data-factory-testing-framework:**
-
-   ```python
-   pip install data-factory-testing-framework
-   ```
-
-### Additional Libraries
-
-**Additionally** you could use for example *pytest even another* test library that you prefer for the Unit test:
-
-```python
-pip install pytest
-```
-
-You could also use Poetry as a package management library:
-
-```python
-pip install poetry
-```
-
-Library Docs for reference:
-
-[Introduction | Documentation | Poetry - Python dependency management and packaging made easy (python-poetry.org)](https://python-poetry.org/docs/)
-
-[pytest: helps you write better programs — pytest documentation](https://docs.pytest.org/en/7.4.x/)
-
-![image](/docs/environment_setup/images/pipinstall_poetry.png)
-
-#### Poetry Configuration - steps
-
-1. If you decide to use poetry(this is optional), please follow some additional steps to configure.
-   Also, please consider using this reference for more information:
-   [Introduction | Documentation | Poetry - Python dependency management and packaging made easy (python-poetry.org)](https://python-poetry.org/docs/#installation)
-
-   Run the following on the power shell, if you are using Windows:
-
-   ```powershell
-   (Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | python
-   ```
-
-   ![image](/docs/environment_setup/images/power_shell_invoke.png)
-
-   ![image](/docs/environment_setup/images/installing_poetry_power_shell.png)
-
-2. Add the bin to the PATH environment variable to the path
-
-   ![image](/docs/environment_setup/images/env_variable.png)
-
-3. Add the unit test framework for testing.
-
-   `poetry add data-factory-testing-framework`poetry
-
-   ![image](/docs/environment_setup/images/poetry_framework.png)
-
-   Let's Start testing!