Repository: mathesar-foundation/mathesar

Issue:
Handle `BOOLEAN` data type in the backend
This issue is to ensure that Mathesar can handle the Postgres `BOOLEAN` data type.

As part of this issue, we need to ensure that:
- Users can use the API to change a column to `BOOLEAN` if it's possible to do so.
- Automatic type inference during file import suggests `BOOLEAN` when it makes sense to do so. 

Most of this is already implemented, this issue is to review the current implementation, ensure it works, and see if we can make improvements to it.

**Additional Context** 
- See PR #368 for code that calculates which types are allowed for a column.


Discussion:
We should double-check that the strings we cast to boolean are what we want.  We're currently restricting what gets cast from the PostgreSQL default to only `'t', 'f', 'true', 'false'` being castable to `BOOLEAN`.
It seems like we should at least accept the [Postgres default values](https://www.postgresql.org/docs/current/datatype-boolean.html) of `true, t, yes. y, on, 1` and `false, f, no, n, off, 0`.

It would also be nice to ensure that any column that we're recommending to be `BOOLEAN` has only two values (plus empty values). e.g. we want to infer that a column is `BOOLEAN` if every value in the CSV is either `yes` or `no`, but we do not want to recommend it if they're mostly `yes` or `no`, but there's a couple of `maybe`s in there.




> It seems like we should at least accept the [Postgres default values](https://www.postgresql.org/docs/current/datatype-boolean.html) of `true, t, yes. y, on, 1` and `false, f, no, n, off, 0`.
> 
> It would also be nice to ensure that any column that we're recommending to be `BOOLEAN` has only two values (plus empty values). e.g. we want to infer that a column is `BOOLEAN` if every value in the CSV is either `yes` or `no`, but we do not want to recommend it if they're mostly `yes` or `no`, but there's a couple of `maybe`s in there.

At the moment, it's not possible to accomplish this combination of goals with our current inference functions, or the paradigm under which they work together.  Please recall that we decided to stay away from trying to infer column type based on global attributes for the moment (such as "the column has only two non-null values").  We could change that, but it's a major overhaul of the entire inference system we currently have set up, and would be a large(ish) time investment.

Alternatively, we could hack together a "one-off" solution just for Booleans, but I'd rather come up with a type inference setup that's global-property-aware in general.
@mathemancer I remember now, I'll set up a separate issue for overhauling our type inference setup to be global-property-aware (probably post-MVP).

**Edited to add**: here's the new issue https://github.com/centerofci/mathesar/issues/438
@kgodey do you think #438 blocks this task? Or is the issue just not being able to check the whole incoming column for only boolean values? So this ticket would implement a "rudimentary" `BOOLEAN` data type that would only check per row?
@powellc #438 doesn't block this task, it's a future improvement. We should go ahead and implement the rudimentary version of the type. I'm trying to keep the scope for the MVP as small as feasible so we ship on time.
@powellc Checking in to see if you have any questions.
Nope. Got a lot of clarity on how the type system works from Brent. Will dig into this later today. Thanks @kgodey 
Juist a quick update here. I finally got my head around the API design and basics of adding and updating things and have implemented the ability to alter a column type to Boolean. This utilizes some functions that already existsed so it was really just turning on the feature in the `PATCH` method for Tables.

The `type_suggestions` endpoint already provides a guess at the right column type if there are only boolean-like values in, so that AC should be good.

I'm going to add some tests for the column altering and open a PR today.
Sounds good, @powellc!
Moving this back to `To do` before talking with @kgodey tomorrow about PATCH verb on a `columns` endpoint. That's the only piece missing, as we already correctly infer Boolean column types.
@kgodey may I close this with your work work on the `columns` endpoint done?  The inference of Boolean types works as expected and with the updating of column types from your work we should be good here.
@powellc I think it would be good to add some `BOOLEAN` specific tests for both inference and column alteration at the API layer. I think our tests are currently only at the `db` layer.
@kgodey sounds good! I'll work on that this evening.

Resolving patch:
diff --git a/db/types/operations/cast.py b/db/types/operations/cast.py
--- a/db/types/operations/cast.py
+++ b/db/types/operations/cast.py
@@ -304,8 +304,10 @@ def assemble_function_creation_sql(argument_type, target_type, function_body):
 
 def get_cast_function_name(target_type):
     """
-    Some casting functions change postgres config parameters  for the transaction they are run on like cast function for casting different data type to timestamp with timezone,
-    So they used be in an isolated transaction
+    Some casting functions change postgres config parameters  for the
+    transaction they are run on like cast function for casting different
+    data type to timestamp with timezone, So they used be in an isolated
+    transaction
     """
     unqualified_type_name = target_type.split('.')[-1].lower()
     if '(' in unqualified_type_name:
@@ -352,8 +354,16 @@ def _get_text_to_boolean_cast_str():
         DECLARE
         istrue {BOOLEAN};
         BEGIN
-          SELECT lower($1)='t' OR lower($1)='true' OR $1='1' INTO istrue;
-          IF istrue OR lower($1)='f' OR lower($1)='false' OR $1='0' THEN
+          SELECT
+            $1='1' OR lower($1) = 'on'
+            OR lower($1)='t' OR lower($1)='true'
+            OR lower($1)='y' OR lower($1)='yes'
+          INTO istrue;
+          IF istrue
+            OR $1='0' OR lower($1) = 'off'
+            OR lower($1)='f' OR lower($1)='false'
+            OR lower($1)='n' OR lower($1)='no'
+          THEN
             RETURN istrue;
           END IF;
           {not_bool_exception_str}
@@ -527,29 +537,35 @@ def get_text_and_datetime_to_datetime_cast_str(type_condition, exception_string)
 
 
 def _get_timestamp_with_timezone_type_body_map(target_type):
-    default_behavior_source_types = frozenset([TIMESTAMP_WITH_TIME_ZONE, TIMESTAMP_WITHOUT_TIME_ZONE, DATE]) | TEXT_TYPES
-    return _get_default_type_body_map(
-        default_behavior_source_types, target_type,
+    datetime_source_types = frozenset(
+        [TIMESTAMP_WITH_TIME_ZONE, TIMESTAMP_WITHOUT_TIME_ZONE, DATE]
     )
+    default_behavior_source_types = datetime_source_types | TEXT_TYPES
+    return _get_default_type_body_map(default_behavior_source_types, target_type)
 
 
 def _get_timestamp_without_timezone_type_body_map():
     """
     Get SQL strings that create various functions for casting different
     types to timestamp without timezone.
-    We allow casting any text, timezone and date type to be cast into a timestamp without timezone,
-     provided it does not any timezone information as this could lead to a information loss
+    We allow casting any text, timezone and date type to be cast into a
+    timestamp without timezone, provided it does not any timezone
+    information as this could lead to a information loss
 
-    The cast function changes the timezone to `utc` for the transaction is called on.
-     So this function call should be used in a isolated transaction to avoid timezone change causing unintended side effect
+    The cast function changes the timezone to `utc` for the transaction
+    is called on.  So this function call should be used in a isolated
+    transaction to avoid timezone change causing unintended side effect
     """
     source_text_types = TEXT_TYPES
     source_datetime_types = frozenset([TIMESTAMP_WITH_TIME_ZONE, DATE])
     default_behavior_source_types = frozenset([TIMESTAMP_WITHOUT_TIME_ZONE])
 
-    not_timestamp_without_tz_exception_str = f"RAISE EXCEPTION '% is not a {TIMESTAMP_WITHOUT_TIME_ZONE}', $1;"
-    # Check if the value is missing timezone by casting it to a timestamp with timezone
-    # and comparing if the value is equal to a timestamp without timezone
+    not_timestamp_without_tz_exception_str = (
+        f"RAISE EXCEPTION '% is not a {TIMESTAMP_WITHOUT_TIME_ZONE}', $1;"
+    )
+    # Check if the value is missing timezone by casting it to a timestamp
+    # with timezone and comparing if the value is equal to a timestamp
+    # without timezone.
     timestamp_without_tz_condition_str = f"""
             IF (timestamp_value_with_tz = timestamp_value) THEN
             RETURN $1::{TIMESTAMP_WITHOUT_TIME_ZONE};
@@ -561,17 +577,19 @@ def _get_timestamp_without_timezone_type_body_map():
     )
     type_body_map.update(
         {
-            text_type: get_text_and_datetime_to_datetime_cast_str(timestamp_without_tz_condition_str,
-                                                                  not_timestamp_without_tz_exception_str
-                                                                  )
+            text_type: get_text_and_datetime_to_datetime_cast_str(
+                timestamp_without_tz_condition_str,
+                not_timestamp_without_tz_exception_str
+            )
             for text_type in source_text_types
         }
     )
     type_body_map.update(
         {
-            datetime_type: get_text_and_datetime_to_datetime_cast_str(timestamp_without_tz_condition_str,
-                                                                      not_timestamp_without_tz_exception_str
-                                                                      )
+            datetime_type: get_text_and_datetime_to_datetime_cast_str(
+                timestamp_without_tz_condition_str,
+                not_timestamp_without_tz_exception_str
+            )
             for datetime_type in source_datetime_types
         }
     )
@@ -624,13 +642,15 @@ def _get_base_textual_cast_to_money():
 def _get_textual_type_body_map(engine, target_type_str=VARCHAR):
     """
     Get SQL strings that create various functions for casting different
-    types to varchar.
+    types to text types through the TEXT type.
 
     All casts to varchar use default PostgreSQL behavior.
     All types in get_supported_alter_column_types are supported.
     """
     supported_types = get_supported_alter_column_db_types(engine)
 
+    # We cast everything through TEXT so that formatting is done correctly
+    # for CHAR.
     text_cast_str = f"""
         BEGIN
           RETURN $1::{TEXT};
@@ -644,11 +664,14 @@ def _get_date_type_body_map():
     """
     Get SQL strings that create various functions for casting different
     types to date.
-    We allow casting any text, timezone and date type to be cast into a timestamp without timezone,
-     provided it does not any timezone information as this could lead to a information loss
 
-    The cast function changes the timezone to `utc` for the transaction is called on.
-     So this function call should be used in a isolated transaction to avoid timezone change causing unintended side effect
+    We allow casting any text, timezone and date type to be cast into a
+    timestamp without timezone, provided it does not any timezone
+    information as this could lead to a information loss.
+
+    The cast function changes the timezone to `utc` for the transaction
+    is called on.  So this function call should be used in a isolated
+    transaction to avoid timezone change causing unintended side effect.
     """
     # Note that default postgres conversion for dates depends on the
     # `DateStyle` option set on the server, which can be one of DMY, MDY,


Test patch:
diff --git a/db/tests/types/operations/test_cast.py b/db/tests/types/operations/test_cast.py
--- a/db/tests/types/operations/test_cast.py
+++ b/db/tests/types/operations/test_cast.py
@@ -531,7 +531,9 @@
             },
             BOOLEAN: {
                 VALID: [
-                    ("true", True), ("false", False), ("t", True), ("f", False)
+                    ("true", True), ("false", False), ("t", True), ("f", False),
+                    ("yes", True), ("y", True), ("no", False), ("n", False),
+                    ("on", True), ("off", False),
                 ],
                 INVALID: ["cat"],
             },
@@ -670,7 +672,9 @@
             },
             BOOLEAN: {
                 VALID: [
-                    ("true", True), ("false", False), ("t", True), ("f", False)
+                    ("true", True), ("false", False), ("t", True), ("f", False),
+                    ("yes", True), ("y", True), ("no", False), ("n", False),
+                    ("on", True), ("off", False),
                 ],
                 INVALID: ["cat"],
             },