Repository: astronomer/astro-sdk

Issue:
Add Append function into databricks
Please describe the feature you'd like to see

I'd like to see a feature that allows users to run the aql.append function to merge two Databricks Delta tables together assuming no conflicts. This would enable us to quickly and efficiently merge unique data from multiple tables into a single table for further processing and analysis.

Describe the solution you'd like

A function or method that takes the names of the two Delta tables to be merged, along with a connection string for a Databricks cluster, and runs the aql.merge function on the tables using the aql library. The function should accept all of the same arguments as the traditional aql.merge function, except for an additional conn_id argument that specifies the connection to use.


Here's an example of what the function signature might look like:

```
target_table = Table(name="foo", conn_id="my_databricks_conn")
source_table = Table(name="bar", conn_id="my_databricks_conn")

aql.append(
    target_table=target_table,
    source_table=source_table,
)
```

**Are there any alternatives to this feature?**
One alternative to this feature would be using the Astro SDK's merge function to upsert data into a Databricks Delta table. However, this would require the user to handle collisions themselves, whereas the append function would assume that there are no collisions and insert the data.
**Additional context**
Add any other context about the feature request here.

**Acceptance Criteria**

- [ ] All checks and tests in the CI should pass
- [ ] Unit tests (90% code coverage or more, [once available](https://github.com/astronomer/astro-sdk/issues/191))
- [ ] Integration tests (if the feature relates to a new database or external service)
- [ ] Example DAG
- [ ] Docstrings in [reStructuredText](https://peps.python.org/pep-0287/) for each of  methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
- [ ] Exception handling in case of errors
- [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
- [ ] Improve the documentation (README, Sphinx, and any other relevant)
- [ ] How to use Guide for the feature ([example](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/operators/postgres_operator_howto_guide.html))



Discussion:

Resolving patch:
diff --git a/python-sdk/src/astro/databricks/delta.py b/python-sdk/src/astro/databricks/delta.py
--- a/python-sdk/src/astro/databricks/delta.py
+++ b/python-sdk/src/astro/databricks/delta.py
@@ -2,6 +2,7 @@
 
 import tempfile
 import uuid
+import warnings
 from textwrap import dedent
 
 import pandas as pd
@@ -257,6 +258,39 @@ def create_table_using_columns(self, table: BaseTable) -> None:
             )
         )
 
+    def append_table(
+        self,
+        source_table: BaseTable,
+        target_table: BaseTable,
+        source_to_target_columns_map: dict[str, str],
+    ) -> None:
+        """
+        Append the source table rows into a destination table.
+
+        :param source_table: Contains the rows to be appended to the target_table
+        :param target_table: Contains the destination table in which the rows will be appended
+        :param source_to_target_columns_map: Dict of source_table columns names to target_table columns names
+        """
+
+        if source_to_target_columns_map:
+            warnings.warn(
+                'Warning: Databricks does not support "partial" inserts. '
+                "You will need to cast all columns if you wish to use this feature"
+            )
+            append_query = (
+                f"INSERT INTO `{self.get_table_qualified_name(target_table)}` "
+                f"({','.join(source_to_target_columns_map.keys())}) SELECT "
+                f"{','.join(source_to_target_columns_map.values())} "
+                f"FROM `{self.get_table_qualified_name(source_table)}`"
+            )
+        else:
+            append_query = (
+                f"INSERT INTO `{self.get_table_qualified_name(target_table)}` "
+                f"SELECT * FROM `{self.get_table_qualified_name(source_table)}`"
+            )
+
+        self.run_sql(append_query)
+
     def export_table_to_pandas_dataframe(
         self, source_table: BaseTable, select_kwargs: dict | None = None
     ) -> pd.DataFrame:


Test patch:
diff --git a/python-sdk/tests_integration/sql/operators/test_append.py b/python-sdk/tests_integration/sql/operators/test_append.py
--- a/python-sdk/tests_integration/sql/operators/test_append.py
+++ b/python-sdk/tests_integration/sql/operators/test_append.py
@@ -92,6 +92,51 @@ def test_append(database_table_fixture, sample_dag, multiple_tables_fixture, app
     test_utils.run_dag(sample_dag)
 
 
+@pytest.mark.integration
+@pytest.mark.parametrize(
+    "append_params",
+    ["all_fields"],
+    indirect=True,
+)
+@pytest.mark.parametrize(
+    "database_table_fixture",
+    [
+        {"database": Database.DELTA},
+    ],
+    indirect=True,
+    ids=["delta"],
+)
+@pytest.mark.parametrize(
+    "multiple_tables_fixture",
+    [
+        {
+            "items": [
+                {
+                    "file": File(path=str(CWD) + "/../../data/homes_main.csv"),
+                },
+                {
+                    "file": File(path=str(CWD) + "/../../data/homes_append.csv"),
+                },
+            ],
+        }
+    ],
+    indirect=True,
+)
+def test_append_delta(database_table_fixture, sample_dag, multiple_tables_fixture, append_params):
+    # TODO: We currently can not get the "basic" to work because Delta does not yet support default values.
+    app_param, validate_append = append_params
+    main_table, append_table = multiple_tables_fixture
+    with sample_dag:
+        appended_table = aql.append(
+            **app_param,
+            target_table=main_table,
+            source_table=append_table,
+        )
+        validate_append(appended_table)
+        aql.cleanup()
+    test_utils.run_dag(sample_dag)
+
+
 @pytest.mark.integration
 @pytest.mark.parametrize(
     "database_table_fixture",