Repository: mathesar-foundation/mathesar

Issue:
Empty column mapped to boolean
## Description
Empty columns are set as boolean.
I found no way to change the default/fallback type to text.

## Expected behavior
I would expect using a more relaxed type, as text, since there is no information to infer the type.

## To Reproduce
Import a CSV with an empty column.


Discussion:
Thank you for reporting this, @frafra. The intended behavior is that text should be the default type, and there should be a way to change any column's type to text. We'll fix this.
Hrm. There _is_ a way to change the actual column type to text, and it is the default. The problem is that `NULL` is a valid value in a boolean column. Changing that would mean that no column with a `NULL` value somewhere could be inferred to be boolean (or indeed cast to boolean using our current setup). 

@kgodey I think we'll have to wait till we're using a global view of columns during type inference to fix this. I.e., we'd have to know that _all_ values are `NULL` to short-circuit the inference logic.
@mathemancer I think the issue here is that the "default" type for inference should be Text, not Boolean. Can we not change the order of inferred types so we prefer Text to Boolean?

I think it's okay if we don't infer booleans correctly (we can fix that in a future issue), that seems preferable to setting empty columns as booleans by default.
> @mathemancer I think the issue here is that the "default" type for inference should be Text, not Boolean. Can we not change the order of inferred types so we prefer Text to Boolean?
> 
> I think it's okay if we don't infer booleans correctly (we can fix that in a future issue), that seems preferable to setting empty columns as booleans by default.

I understand the issue perfectly.

I'll explain our inference logic in a bit more detail, so we're on the same page about its limitations.

We don't have any global view of a column when doing type inference. Our 'inference' concept is rather 'can I cast this column to `<some_type>`?'. If so, we accept the column as a `<some_type>`, and otherwise we don't. We proceed through various types in order, and exit which we find one where the casting process is successful. It's important to understand this detail. We aren't comparing a couple of potential casting results in some way and deciding which is 'better' by some metric. We're simply checking whether we can do a given casting at all, and _the first valid target type found_ is the inferred type.

Note that the casting logic can only see one value at a time. So, testing whether you can cast a column to `BOOLEAN` amounts to testing whether we can cast each value in the column, individually, to a `BOOLEAN`. Since `NULL` is a valid value to be cast to type `BOOLEAN` (and in fact is a valid value to be cast to any type; more on that later), we have a problem when all values of a column are `NULL` (i.e., if it's an empty column), since the test 'can I cast the column to `BOOLEAN`' will return `true`. Further, any logic which takes into account the fact that all values in a column are `NULL` is global by definition (in this context).

Now, _any_ value can be cast to `TEXT`. In fact, we start by importing the data with all columns initially as `TEXT` columns. Thus, we would never test any type which occurs after `TEXT` in the hierarchy at all. Some implications:
- Moving `BOOLEAN` after `TEXT` in the testing order means we'd never infer any column to be a `BOOLEAN`.
- We'd (with our current order) instead infer it to be the type `MATHESAR_TYPES.EMAIL` (which is arguably more confusing than `BOOLEAN`).
- If we instead move `TEXT` to the front of the testing order, then we will only ever infer a column to be of type `TEXT`. I.e., we'd disable the type inference logic entirely.

The other non-global option I can think of is to change our `cast_to_boolean` function to disallow `NULL` values. This would mean (as mentioned previously) that no column with any `NULL` value could be cast to `BOOLEAN` in Mathesar, and it wouldn't solve the problem. I.e., we'd then infer the next type down the list (`EMAIL`). To fix _that_, we'd have to disallow `NULL` values for casting to any type except for `TEXT`. I think this would be considerably worse than having the inference logic return `BOOLEAN` for an empty column.

All that said, I think this might be a case to add some ad-hoc global inference logic, basically the simple rule
```
(all NULL) -> TEXT
```
I strongly suggest we resist the urge to add global type inference logic in an ad-hoc fashion in the future, since that's complicated stuff, and the maintenance and testing burden would be considerable, and the performance will quickly deteriorate even further.
Do not cast (thus use the default TEXT) if no data is available, seems the only reasonable solution. Doing it efficiently could be challenging, but that would not mean that mathesar would begin to implement an entirely new class of global type interference procedures.
> Do not cast (thus use the default TEXT) if no data is available, seems the only reasonable solution. Doing it efficiently could be challenging, but that would not mean that mathesar would begin to implement an entirely new class of global type interference procedures.

That's the plan. We'll implement this logic ad-hoc as a check for each column before doing the inference logic. It will be precisely as efficient as checking `EXISTS (SELECT 1 FROM mytable WHERE mycol IS NOT NULL)` for an unidexed column.

Some context for future historians: "Global inference logic" in this context refers to any logic that takes into account the data (or in this case lack of it) for an entire column when determining its likely type. We've had a number of discussions about the pros/cons of adding this sort of thing in the past.

Resolving patch:
diff --git a/db/columns/operations/infer_types.py b/db/columns/operations/infer_types.py
--- a/db/columns/operations/infer_types.py
+++ b/db/columns/operations/infer_types.py
@@ -5,6 +5,7 @@
 
 from db.columns.exceptions import DagCycleError
 from db.columns.operations.alter import alter_column_type
+from db.columns.operations.select import determine_whether_column_contains_data
 from db.tables.operations.select import get_oid_from_table, reflect_table
 from db.types.base import PostgresType, MathesarCustomType, get_available_known_db_types
 from db.metadata import get_empty_metadata
@@ -26,9 +27,10 @@
         MathesarCustomType.MATHESAR_MONEY,
         PostgresType.TIMESTAMP_WITHOUT_TIME_ZONE,
         PostgresType.TIMESTAMP_WITH_TIME_ZONE,
-        # We only infer to TIME_WITHOUT_TIME_ZONE as time zones don't make much sense
-        # without additional date information. See postgres documentation for further
-        # details: https://www.postgresql.org/docs/13/datatype-datetime.html
+        # We only infer to TIME_WITHOUT_TIME_ZONE as time zones don't make much
+        # sense without additional date information. See postgres documentation
+        # for further details:
+        # https://www.postgresql.org/docs/13/datatype-datetime.html
         PostgresType.TIME_WITHOUT_TIME_ZONE,
         PostgresType.INTERVAL,
         MathesarCustomType.EMAIL,
@@ -39,26 +41,45 @@
 }
 
 
-def infer_column_type(schema, table_name, column_name, engine, depth=0, type_inference_dag=None, metadata=None, columns_might_have_defaults=True):
+def infer_column_type(
+        schema,
+        table_name,
+        column_name,
+        engine,
+        depth=0,
+        type_inference_dag=None,
+        metadata=None,
+        columns_might_have_defaults=True,
+):
     """
-    Attempts to cast the column to the best type for it, given the mappings defined in TYPE_INFERENCE_DAG
-    and _get_type_classes_mapped_to_dag_nodes. Returns the resulting column type's class.
+    Attempt to cast the column to the best type for it.
+
+    Returns the resulting column type's class.
 
     Algorithm:
-        1. reflect the column's type class;
-        2. use _get_type_classes_mapped_to_dag_nodes to map it to a TYPE_INFERENCE_DAG key;
-        3. look up the sequence of types referred to by that key on the TYPE_INFERENCE_DAG;
-            - if there's no such key on the TYPE_INFERENCE_DAG dict, or if its value is an empty
-            list, return the current column type's class;
-        4. iterate through that sequence of types trying to alter the column's type to them;
-            - if the column's type is altered successfully, break iteration and return the output
-            of running infer_column_type again (trigger tail recursion);
-            - if none of the column type alterations succeed, return the current column type's
-            class.
+        1. Check for any data in the column.
+           - If the column is empty, return the column's current type
+             class.
+        2. reflect the column's type class.
+        3. Use _get_type_classes_mapped_to_dag_nodes to map it to a
+           TYPE_INFERENCE_DAG key.
+        4. Look up the sequence of types referred to by that key on the
+           TYPE_INFERENCE_DAG.
+           - If there's no such key on the TYPE_INFERENCE_DAG dict, or if
+             its value is an empty list, return the current column's type
+             class.
+        5. Iterate through that sequence of types trying to alter the
+           column's type to them.
+           - If the column's type is altered successfully, break
+             iteration and return the output of running infer_column_type
+             again (trigger tail recursion).
+           - If none of the column type alterations succeed, return the
+             current column's type class.
     """
+    metadata = metadata if metadata else get_empty_metadata()
+
     if type_inference_dag is None:
         type_inference_dag = TYPE_INFERENCE_DAG
-    metadata = metadata if metadata else get_empty_metadata()
     if depth > MAX_INFERENCE_DAG_DEPTH:
         raise DagCycleError("The type_inference_dag likely has a cycle")
     type_classes_to_dag_nodes = _get_type_classes_mapped_to_dag_nodes(engine)
@@ -69,11 +90,18 @@ def infer_column_type(schema, table_name, column_name, engine, depth=0, type_inf
         column_name=column_name,
         metadata=metadata,
     )
+    table_oid = get_oid_from_table(table_name, schema, engine)
+    column_contains_data = determine_whether_column_contains_data(
+        table_oid, column_name, engine, metadata
+    )
+    # We short-circuit in this case since we can't infer type without data.
+    if not column_contains_data:
+        return column_type_class
+
     # a DAG node will be a DatabaseType Enum
     dag_node = type_classes_to_dag_nodes.get(column_type_class)
     logger.debug(f"dag_node: {dag_node}")
     types_to_cast_to = type_inference_dag.get(dag_node, [])
-    table_oid = get_oid_from_table(table_name, schema, engine)
     for db_type in types_to_cast_to:
         try:
             with engine.begin() as conn:
diff --git a/db/columns/operations/select.py b/db/columns/operations/select.py
--- a/db/columns/operations/select.py
+++ b/db/columns/operations/select.py
@@ -1,7 +1,7 @@
 import warnings
 
 from pglast import Node, parse_sql
-from sqlalchemy import and_, asc, cast, select, text
+from sqlalchemy import and_, asc, cast, select, text, exists
 
 from db.columns.exceptions import DynamicDefaultWarning
 from db.tables.operations.select import reflect_table_from_oid
@@ -151,6 +151,20 @@ def get_column_default_dict(table_oid, attnum, engine, metadata, connection_to_u
     return {"value": default_value, "is_dynamic": is_dynamic}
 
 
+def determine_whether_column_contains_data(
+        table_oid, column_name, engine, metadata, connection_to_use=None
+):
+    """
+    Given a column, return True if it contains data, False otherwise.
+    """
+    sa_table = reflect_table_from_oid(
+        table_oid, engine, metadata=metadata, connection_to_use=connection_to_use,
+    )
+    sel = select(exists(1).where(sa_table.columns[column_name] != None))  # noqa
+    contains_data = execute_statement(engine, sel, connection_to_use).scalar()
+    return contains_data
+
+
 def get_column_from_oid_and_attnum(table_oid, attnum, engine, metadata, connection_to_use=None):
     sa_table = reflect_table_from_oid(table_oid, engine, metadata=metadata, connection_to_use=connection_to_use)
     column_name = get_column_name_from_attnum(table_oid, attnum, engine, metadata=metadata, connection_to_use=connection_to_use)


Test patch:
diff --git a/db/tests/tables/operations/test_infer_types.py b/db/tests/tables/operations/test_infer_types.py
--- a/db/tests/tables/operations/test_infer_types.py
+++ b/db/tests/tables/operations/test_infer_types.py
@@ -30,6 +30,11 @@
         ],
         PostgresType.NUMERIC
     ),
+    (
+        PostgresType.TEXT,
+        [],
+        PostgresType.TEXT,
+    ),
     (
         PostgresType.TEXT,
         ["1.0"],