Repository: mathesar-foundation/mathesar

Issue:
Additional information needed in column_metadata in Query run endpoint
We need the following additional information in the `column_metadata` property in Query run response. This is needed to show the information on the inspector panel based on the [new figma designs](https://www.figma.com/file/xHb5oIqye3fnXtb2heRH34/Styling?node-id=1384%3A6672):
* Table name
* Base column name
* For aggregated columns, it should contain the respective input column alias

Previously, I was able to show the table name and base column name using the information from joinable_tables endpoint, but since we now have the inspector additionally within the Explorations page (where the joinable_tables call is unnecessary), I would like this information to come from the column_metadata.

cc: @dmos62 @mathemancer 


Discussion:
What's the base column name for an alias / a column in a query? I've not heard of this term before. Is it the initial column name that can be traced back from a given alias? Does a column generated by aggregation have it?

I wouldn't expect a column name to be an identifier for an initial column. I'd expect an alias if the goal is to identify the initial column associated with another alias.

What's the table name for a column generated by aggregation? Is it not expected to be specified in that case?
> What's the base column name for an alias / a column in a query? I've not heard of this term before.

It is the name of the underlying column in the table.

> Does a column generated by aggregation have it?

No. It does not.

> I wouldn't expect a column name to be an identifier for an initial column. I'd expect an alias if the goal is to identify the initial column associated with another alias.

The goal is to show information of the underlying table and column to the user. We do not do any operation based on the column name.

> What's the table name for a column generated by aggregation? Is it not expected to be specified in that case?

For columns generated by aggregation, we do not need the column name or the table name. We however, need the alias of the initial_column which was used to generate this column.

--

For initial_columns: We need underlying table_name and column_name in the table.
For generated columns: We need alias of the associated initial_column.

It would be helpful to have an additional flag specifying if the column is 'initial' or 'generated'.

Resolving patch:
diff --git a/db/queries/base.py b/db/queries/base.py
--- a/db/queries/base.py
+++ b/db/queries/base.py
@@ -46,6 +46,7 @@ def get_count(self, **kwargs):
             table=self.transformed_relation, engine=self.engine, **kwargs,
         )
 
+    # NOTE if too expensive, can be rewritten to parse DBQuery spec, instead of leveraging sqlalchemy
     @property
     def all_sa_columns_map(self):
         """
@@ -146,6 +147,19 @@ def _process_initial_column(col):
         ).select_from(from_clause)
         return stmt.cte()
 
+    def get_input_alias_for_output_alias(self, output_alias):
+        return self.map_of_output_alias_to_input_alias.get(output_alias)
+
+    # TODO consider caching; not urgent, since redundant calls don't trigger IO, it seems
+    @property
+    def map_of_output_alias_to_input_alias(self):
+        m = dict()
+        transforms = self.transformations
+        if transforms:
+            for transform in transforms:
+                m = m | transform.map_of_output_alias_to_input_alias
+        return m
+
 
 def _guarantee_jp_path_tuples(jp_path):
     if jp_path is not None:
@@ -176,4 +190,7 @@ def __init__(
 
     @property
     def is_base_column(self):
+        """
+        A base column is an initial column on a query's base table.
+        """
         return self.jp_path is None
diff --git a/db/transforms/base.py b/db/transforms/base.py
--- a/db/transforms/base.py
+++ b/db/transforms/base.py
@@ -36,6 +36,18 @@ def __eq__(self, other):
             and self.__dict__ == other.__dict__
         )
 
+    @property
+    def map_of_output_alias_to_input_alias(self):
+        """
+        Expected to return a mapping of output aliases to input aliases.
+
+        Useful when looking for parent aliases of a given alias.
+
+        Notice that the reverse mapping (from input aliases to output aliases) would be
+        significantly different, because a single input alias can map to multiple output aliases.
+        """
+        return dict()
+
 
 class Filter(Transform):
     type = "filter"
@@ -167,6 +179,18 @@ class Summarize(Transform):
     """
     type = "summarize"
 
+    @property
+    def map_of_output_alias_to_input_alias(self):
+        m = dict()
+        grouping_expressions = self.spec['grouping_expressions']
+        aggregation_expressions = self.spec['aggregation_expressions']
+        all_expressions = grouping_expressions + aggregation_expressions
+        for expression in all_expressions:
+            expr_output_alias = expression.get('output_alias', None)
+            expr_input_alias = expression.get('input_alias', None)
+            m[expr_output_alias] = expr_input_alias
+        return m
+
     def apply_to_relation(self, relation):
 
         def _get_grouping_column(col_spec):
diff --git a/mathesar/models/query.py b/mathesar/models/query.py
--- a/mathesar/models/query.py
+++ b/mathesar/models/query.py
@@ -3,11 +3,11 @@
 from django.utils.functional import cached_property
 
 from db.queries.base import DBQuery, InitialColumn
+from db.transforms.operations.deserialize import deserialize_transformation
 
 from mathesar.models.base import BaseModel, Column
 from mathesar.models.relation import Relation
 from mathesar.api.exceptions.validation_exceptions.exceptions import InvalidValueType, DictHasBadKeys
-from db.transforms.operations.deserialize import deserialize_transformation
 from mathesar.state import get_cached_metadata
 
 
@@ -194,13 +194,44 @@ def initial_columns_described(self):
         )
 
     def _describe_query_column(self, sa_col):
-        return {
-            'alias': sa_col.name,
-            'display_name': self._get_display_name_for_sa_col(sa_col),
-            'type': sa_col.db_type.id,
-            'type_options': sa_col.type_options,
-            'display_options': self._get_display_options_for_sa_col(sa_col),
-        }
+        """
+        Note, has some conditional fields depending on whether the column is an initial column or
+        is generated mid-query (created via a summarization).
+        """
+        alias = sa_col.name
+        initial_db_column = self._get_db_initial_column_by_alias(alias)
+        is_initial_column = initial_db_column is not None
+        output = dict(
+            alias=alias,
+            display_name=self._get_display_name_for_alias(alias),
+            type=sa_col.db_type.id,
+            type_options=sa_col.type_options,
+            display_options=self._get_display_options_for_alias(alias),
+            is_initial_column=is_initial_column,
+        )
+        optionals = dict(
+            input_column_name=None,
+            input_table_name=None,
+            input_alias=None,
+        )
+        output = output | optionals
+        if is_initial_column:
+            initial_dj_column = _get_dj_column_for_initial_db_column(initial_db_column)
+            output = output | dict(
+                input_column_name=initial_dj_column.name,
+                input_table_name=initial_dj_column.table.name,
+            )
+        else:
+            input_alias = self.db_query.get_input_alias_for_output_alias(alias)
+            output = output | dict(
+                input_alias=input_alias
+            )
+        return output
+
+    def _get_db_initial_column_by_alias(self, alias):
+        for db_initial_column in self._db_initial_columns:
+            if db_initial_column.alias == alias:
+                return db_initial_column
 
     @property
     def all_columns_description_map(self):
@@ -220,6 +251,7 @@ def db_query(self):
             metadata=get_cached_metadata()
         )
 
+    # TODO reused; consider using cached_property
     @property
     def _db_initial_columns(self):
         return tuple(
@@ -237,12 +269,12 @@ def _db_transformations(self):
                 in self.transformations
             )
 
-    def _get_display_name_for_sa_col(self, sa_col):
-        return self._alias_to_display_name.get(sa_col.name)
+    def _get_display_name_for_alias(self, alias):
+        return self._alias_to_display_name.get(alias)
 
-    def _get_display_options_for_sa_col(self, sa_col):
+    def _get_display_options_for_alias(self, alias):
         if self.display_options is not None:
-            return self.display_options.get(sa_col.name)
+            return self.display_options.get(alias)
 
     @cached_property
     def _alias_to_display_name(self):
@@ -266,6 +298,12 @@ def _sa_engine(self):
         return self.base_table._sa_engine
 
 
+def _get_dj_column_for_initial_db_column(initial_column):
+    oid = initial_column.reloid
+    attnum = initial_column.attnum
+    return Column.objects.get(table__oid=oid, attnum=attnum)
+
+
 def _get_column_pair_from_id(col_id):
     col = Column.objects.get(id=col_id)
     return col.table.oid, col.attnum


Test patch:
diff --git a/mathesar/tests/api/query/test_library_demo_reports.py b/mathesar/tests/api/query/test_library_demo_reports.py
--- a/mathesar/tests/api/query/test_library_demo_reports.py
+++ b/mathesar/tests/api/query/test_library_demo_reports.py
@@ -93,13 +93,21 @@ def check_overdue_books_columns(create_overdue_books_query, client):
             'display_name': 'Titles',
             'type': '_array',
             'type_options': {'item_type': 'text'},
-            'display_options': None
+            'display_options': None,
+            'is_initial_column': False,
+            'input_table_name': None,
+            'input_column_name': None,
+            'input_alias': 'Book Title',
         }, {
             'alias': 'email',
             'display_name': 'Patron Email',
             'type': 'mathesar_types.email',
             'type_options': None,
-            'display_options': None
+            'display_options': None,
+            'is_initial_column': True,
+            'input_table_name': 'Patrons',
+            'input_column_name': 'Email',
+            'input_alias': None,
         }
     ]
     actual_response_data = client.get(f'/api/db/v0/queries/{query_id}/columns/').json()
@@ -199,13 +207,21 @@ def check_monthly_checkouts_columns(create_monthly_checkouts_query, client):
             'display_name': 'Month',
             'type': 'text',
             'type_options': None,
-            'display_options': None
+            'display_options': None,
+            'is_initial_column': False,
+            'input_table_name': None,
+            'input_column_name': None,
+            'input_alias': 'Checkout Time',
         }, {
             'alias': 'Count',
             'display_name': 'Number of Checkouts',
             'type': 'integer',
             'type_options': None,
-            'display_options': None
+            'display_options': None,
+            'is_initial_column': False,
+            'input_table_name': None,
+            'input_column_name': None,
+            'input_alias': 'id',
         }
     ]
     actual_response_data = client.get(f'/api/db/v0/queries/{query_id}/columns/').json()
diff --git a/mathesar/tests/api/query/test_query_columns_api.py b/mathesar/tests/api/query/test_query_columns_api.py
--- a/mathesar/tests/api/query/test_query_columns_api.py
+++ b/mathesar/tests/api/query/test_query_columns_api.py
@@ -3,6 +3,7 @@
 
 def test_columns(client, minimal_patents_query):
     ui_query = minimal_patents_query
+    input_table_name = ui_query.base_table.name
     response = client.get(f'/api/db/v0/queries/{ui_query.id}/columns/')
     response_json = response.json()
     assert response.status_code == 200
@@ -13,6 +14,10 @@ def test_columns(client, minimal_patents_query):
             'type': PostgresType.TEXT.id,
             'type_options': None,
             'display_options': dict(a=1),
+            'is_initial_column': True,
+            'input_alias': None,
+            'input_column_name': 'Center',
+            'input_table_name': input_table_name,
         },
         {
             'alias': 'col2',
@@ -20,5 +25,9 @@ def test_columns(client, minimal_patents_query):
             'type': PostgresType.TEXT.id,
             'type_options': None,
             'display_options': dict(b=2),
+            'is_initial_column': True,
+            'input_alias': None,
+            'input_column_name': 'Case Number',
+            'input_table_name': input_table_name,
         },
     ]
diff --git a/mathesar/tests/api/query/test_query_results.py b/mathesar/tests/api/query/test_query_results.py
--- a/mathesar/tests/api/query/test_query_results.py
+++ b/mathesar/tests/api/query/test_query_results.py
@@ -3,6 +3,7 @@
 
 def test_query_results_minimal(client, minimal_patents_query):
     ui_query = minimal_patents_query
+    input_table_name = ui_query.base_table.name
     order_by = json.dumps(
         [
             {'field': 'col1', 'direction': 'asc'},
@@ -32,14 +33,22 @@ def test_query_results_minimal(client, minimal_patents_query):
                 'display_name': 'Column 1',
                 'type': 'text',
                 'type_options': None,
-                'display_options': {'a': 1}
+                'display_options': {'a': 1},
+                'is_initial_column': True,
+                'input_table_name': input_table_name,
+                'input_column_name': 'Center',
+                'input_alias': None,
             },
             'col2': {
                 'alias': 'col2',
                 'display_name': 'Column 2',
                 'type': 'text',
                 'type_options': None,
-                'display_options': {'b': 2}
+                'display_options': {'b': 2},
+                'is_initial_column': True,
+                'input_table_name': input_table_name,
+                'input_column_name': 'Case Number',
+                'input_alias': None,
             }
         },
         'parameters': {
diff --git a/mathesar/tests/api/query/test_query_run.py b/mathesar/tests/api/query/test_query_run.py
--- a/mathesar/tests/api/query/test_query_run.py
+++ b/mathesar/tests/api/query/test_query_run.py
@@ -48,14 +48,22 @@ def test_queries_run_minimal(create_patents_table, client):
                 'display_name': 'Column 1',
                 'type': 'text',
                 'type_options': None,
-                'display_options': None
+                'display_options': None,
+                'is_initial_column': True,
+                'input_table_name': 'patent_query_run_minimal_table',
+                'input_column_name': 'Center',
+                'input_alias': None,
             },
             'col2': {
                 'alias': 'col2',
                 'display_name': 'Column 2',
                 'type': 'text',
                 'type_options': None,
-                'display_options': None
+                'display_options': None,
+                'is_initial_column': True,
+                'input_table_name': 'patent_query_run_minimal_table',
+                'input_column_name': 'Case Number',
+                'input_alias': None,
             }
         },
         'parameters': {