Repository: mathesar-foundation/mathesar

Issue:
Handle more human readable number imports
## Problem
<!-- Please provide a clear and concise description of the problem that this feature request is designed to solve.-->

Currently, we only detect a column as a number type if the entries can be cast to a `NUMERIC` using the default PostgreSQL casting functionality.  This is a problem in certain locales, and also can't handle numbers with grouping separators (e.g., the `,` in `1,000.00`).

## Proposed solution
<!-- A clear and concise description of your proposed solution or feature. -->

We should write custom logic to allow handling more strings when casting to number types.

## Additional context
<!-- Add any other context or screenshots about the feature request here.-->

- This is inspired by and related to the custom casting work we're doing for money types. Please see PR #1137 for related code.


Discussion:
Marking as "help wanted" and "good first issue" since it should be largely an extension of the work done for the `MATHESAR_MONEY` type.
@mathemancer may u assign this to me. I would like to work on it.
@sahilsaini110 Sure, I've assigned it to you. I also updated the issue description to link to the relevant PR with the custom casting code for money, you'll want to emulate that code.




> @sahilsaini110 Sure, I've assigned it to you. I also updated the issue description to link to the relevant PR with the custom casting code for money, you'll want to emulate that code.

Thanks, @kgodey for assigning this issue to me. Also, I will go through the pr you have linked and try to fix this issue.

@sahilsaini110 I wanted to follow up on this issue since it's been a few days and I don't see a PR yet. Please let us know if you need any help or if you are no longer working on the issue.
> @sahilsaini110 I wanted to follow up on this issue since it's been a few days and I don't see a PR yet. Please let us know if you need any help or if you are no longer working on the issue.

Sorry for the late reply I was trying to solve this issue but was unable to understand how to proceed then I got stuck in some other work sorry for the inconvenience I will take care of that from now on. Also if possible I need some guidance for this issue that like how to start and proceed with this.
@sahilsaini110 The related PR should provide guidance since it implements the same logic for the money data type. If you have a specific question, please ask.

If you're confused by this issue (or any other issue), it would be better to instead work on a different issue that you can grasp quickly. That would give you a better idea of the codebase before working on a more complex issue like this one.
> @sahilsaini110 The related PR should provide guidance since it implements the same logic for the money data type. If you have a specific question, please ask.
> 
> If you're confused by this issue (or any other issue), it would be better to instead work on a different issue that you can grasp quickly. That would give you a better idea of the codebase before working on a more complex issue like this one.

Yes, I think you are right this is a little bit complex issue since it involves lots of files to be changed so I think I have to first become more familiar with the codebase. Also right now I am busy writing the proposal so unable to give that much time to this.
@sahilsaini110 Thanks for the update, I unassigned you.
I'd like to work on this one.
I've read the discussions and see that it's not an easy one. And still, I'd start with getting acquainted with the codebase along with investigating parts that are necessary for this specific task.
Since this issue doesn't have priority labels, can I take some time to work on this? I'll ask questions whenever they occur and let you know if it turns out too difficult for me.
If it's ok, can you please assign this one to me?
@Jyuart assigned to you, thanks! Feel free to take your time.
@mathemancer I'm making my way through the code, getting knowledge (https://github.com/centerofci/mathesar/pull/1137 is very useful for this one, thanks).
I just wanted to be clear about the specific type. **This issue is about casting just the PostgreSQL's `NUMERIC` type, right?** You are pretty clear about that in the description:
> Currently, we only detect a column as a number type if the entries can be cast to a NUMERIC using the default PostgreSQL casting functionality.

But the title of the issue still confuses me a bit. Maybe you meant all the numeric types (integer, decimal, etc?)
@Jyuart "Number" refers to the Mathesar UI type, which is a grouping of all numeric type. Please read this wiki page for more context: https://wiki.mathesar.org/en/engineering/glossary/ui-types
@kgodey is correct in that all types under `Number` in the page she linked are handled. However, my original intent with this issue was about _detecting_ whether a column is of number type. For that, the only relevant target is `NUMERIC`, since that's the type in the DAG defined in `db/columns/operations/infer_types.py`. 

So, to enable human readable number _imports_, changing the function for casting to `NUMERIC` would suffice.

@kgodey Do we have a use-case for casting directly from `TEXT` (or string-like) to other number types? Somehow I thought the flow pointed towards `NUMERIC` first, then allowed changing to other more specific number-like types. In that case, we should also change all other casting functions with number target types accordingly. I think we should make that a separate issue, however, to avoid blocking getting the work for handling imports merged.
> @kgodey Do we have a use-case for casting directly from `TEXT` (or string-like) to other number types? Somehow I thought the flow pointed towards `NUMERIC` first, then allowed changing to other more specific number-like types.

For inferring the type when importing data, we decided to only do `NUMERIC` for now since we can handle most numbers with it. Once we have global inference, we might want to be smarter about this.

We should allow users to manually cast from `TEXT` to any number type, though.

> In that case, we should also change all other casting functions with number target types accordingly. I think we should make that a separate issue, however, to avoid blocking getting the work for handling imports merged.

Please make a separate issue if needed. I agree that this issue shouldn't be blocked.

Resolving patch:
diff --git a/db/types/categories.py b/db/types/categories.py
--- a/db/types/categories.py
+++ b/db/types/categories.py
@@ -22,7 +22,6 @@
 
 DECIMAL_TYPES = frozenset({
     PostgresType.DOUBLE_PRECISION,
-    PostgresType.NUMERIC,
     PostgresType.REAL,
 })
 
@@ -72,6 +71,7 @@
 NUMERIC_TYPES = frozenset({
     *INTEGER_TYPES,
     *DECIMAL_TYPES,
+    PostgresType.NUMERIC
 })
 
 # Comparable types are those that should support greater, lesser, equal comparisons amongst
diff --git a/db/types/operations/cast.py b/db/types/operations/cast.py
--- a/db/types/operations/cast.py
+++ b/db/types/operations/cast.py
@@ -10,6 +10,7 @@
 from db.types import categories
 
 MONEY_ARR_FUNC_NAME = "get_mathesar_money_array"
+NUMERIC_ARR_FUNC_NAME = "get_numeric_array"
 
 
 def get_column_cast_expression(column, target_type, engine, type_options={}):
@@ -51,6 +52,7 @@ def install_all_casts(engine):
     create_multicurrency_money_casts(engine)
     create_textual_casts(engine)
     create_uri_casts(engine)
+    create_numeric_casts(engine)
 
 
 def create_boolean_casts(engine):
@@ -133,6 +135,14 @@ def create_uri_casts(engine):
     create_cast_functions(MathesarCustomType.URI, type_body_map, engine)
 
 
+def create_numeric_casts(engine):
+    numeric_array_create = _build_numeric_array_function()
+    with engine.begin() as conn:
+        conn.execute(text(numeric_array_create))
+    type_body_map = _get_numeric_type_body_map()
+    create_cast_functions(PostgresType.NUMERIC, type_body_map, engine)
+
+
 # TODO find more descriptive name
 def get_full_cast_map(engine):
     """
@@ -378,7 +388,7 @@ def _get_integer_type_body_map(target_type=PostgresType.INTEGER):
     etc.
     """
     default_behavior_source_types = categories.INTEGER_TYPES | categories.STRING_TYPES
-    no_rounding_source_types = categories.DECIMAL_TYPES | categories.MONEY_WITHOUT_CURRENCY_TYPES
+    no_rounding_source_types = categories.DECIMAL_TYPES | categories.MONEY_WITHOUT_CURRENCY_TYPES | frozenset([PostgresType.NUMERIC])
     target_type_str = target_type.id
     cast_loss_exception_str = (
         f"RAISE EXCEPTION '% cannot be cast to {target_type_str} without loss', $1;"
@@ -863,6 +873,132 @@ def _get_text_uri_type_body_map():
     return {type_: _get_text_uri_type_body_map() for type_ in source_types}
 
 
+def _get_numeric_type_body_map():
+    """
+    Get SQL strings that create various functions for casting different
+    types to numeric.
+    We allow casting any textual type to locale-agnostic numeric.
+    """
+    default_behavior_source_types = categories.NUMERIC_TYPES | frozenset([PostgresType.MONEY])
+    text_source_types = categories.STRING_TYPES
+
+    type_body_map = _get_default_type_body_map(
+        default_behavior_source_types, PostgresType.NUMERIC
+    )
+    type_body_map.update(
+        {
+            text_type: _get_text_to_numeric_cast()
+            for text_type in text_source_types
+        }
+    )
+    type_body_map.update({PostgresType.BOOLEAN: _get_boolean_to_number_cast(PostgresType.NUMERIC)})
+    return type_body_map
+
+
+def _get_text_to_numeric_cast():
+    text_db_type_id = PostgresType.TEXT.id
+    numeric_db_type_id = PostgresType.NUMERIC.id
+
+    numeric_array_function = get_qualified_name(NUMERIC_ARR_FUNC_NAME)
+    cast_exception_str = (
+        f"RAISE EXCEPTION '% cannot be cast to {PostgresType.NUMERIC}', $1;"
+    )
+    return rf"""
+    DECLARE decimal_point {text_db_type_id};
+    DECLARE is_negative {PostgresType.BOOLEAN.id};
+    DECLARE numeric_arr {text_db_type_id}[];
+    DECLARE numeric {text_db_type_id};
+    BEGIN
+        SELECT {numeric_array_function}($1::{text_db_type_id}) INTO numeric_arr;
+        IF numeric_arr IS NULL THEN
+            {cast_exception_str}
+        END IF;
+
+        SELECT numeric_arr[1] INTO numeric;
+        SELECT ltrim(to_char(1, 'D'), ' ') INTO decimal_point;
+        SELECT $1::text ~ '^-.*$' INTO is_negative;
+
+        IF numeric_arr[2] IS NOT NULL THEN
+            SELECT regexp_replace(numeric, numeric_arr[2], '', 'gq') INTO numeric;
+        END IF;
+        IF numeric_arr[3] IS NOT NULL THEN
+            SELECT regexp_replace(numeric, numeric_arr[3], decimal_point, 'q') INTO numeric;
+        END IF;
+        IF is_negative THEN
+            RETURN ('-' || numeric)::{numeric_db_type_id};
+        END IF;
+        RETURN numeric::{numeric_db_type_id};
+    END;
+    """
+
+
+def _build_numeric_array_function():
+    """
+    The main reason for this function to be separate is for testing. This
+    does have some performance impact; we should consider inlining later.
+    """
+    qualified_function_name = get_qualified_name(NUMERIC_ARR_FUNC_NAME)
+
+    no_separator_big = r"[0-9]{4,}(?:([,.])[0-9]+)?"
+    no_separator_small = r"[0-9]{1,3}(?:([,.])[0-9]{1,2}|[0-9]{4,})?"
+    comma_separator_req_decimal = r"[0-9]{1,3}(,)[0-9]{3}(\.)[0-9]+"
+    period_separator_req_decimal = r"[0-9]{1,3}(\.)[0-9]{3}(,)[0-9]+"
+    comma_separator_opt_decimal = r"[0-9]{1,3}(?:(,)[0-9]{3}){2,}(?:(\.)[0-9]+)?"
+    period_separator_opt_decimal = r"[0-9]{1,3}(?:(\.)[0-9]{3}){2,}(?:(,)[0-9]+)?"
+    space_separator_opt_decimal = r"[0-9]{1,3}(?:( )[0-9]{3})+(?:([,.])[0-9]+)?"
+    comma_separator_lakh_system = r"[0-9]{1,2}(?:(,)[0-9]{2})+,[0-9]{3}(?:(\.)[0-9]+)?"
+    single_quote_separator_opt_decimal = r"[0-9]{1,3}(?:(\'')[0-9]{3})+(?:([.])[0-9]+)?"
+
+    inner_number_tree = "|".join(
+        [
+            no_separator_big,
+            no_separator_small,
+            comma_separator_req_decimal,
+            period_separator_req_decimal,
+            comma_separator_opt_decimal,
+            period_separator_opt_decimal,
+            space_separator_opt_decimal,
+            comma_separator_lakh_system,
+            single_quote_separator_opt_decimal
+        ])
+    numeric_finding_regex = f"^(?:[+-]?({inner_number_tree}))$"
+
+    actual_number_indices = [1]
+    group_divider_indices = [4, 6, 8, 10, 12, 14, 16]
+    decimal_point_indices = [2, 3, 5, 7, 9, 11, 13, 15, 17]
+    actual_numbers_str = ','.join([f'raw_arr[{idx}]' for idx in actual_number_indices])
+    group_dividers_str = ','.join([f'raw_arr[{idx}]' for idx in group_divider_indices])
+    decimal_points_str = ','.join([f'raw_arr[{idx}]' for idx in decimal_point_indices])
+
+    text_db_type_id = PostgresType.TEXT.id
+    return rf"""
+    CREATE OR REPLACE FUNCTION {qualified_function_name}({text_db_type_id}) RETURNS {text_db_type_id}[]
+    AS $$
+      DECLARE
+        raw_arr {text_db_type_id}[];
+        actual_number_arr {text_db_type_id}[];
+        group_divider_arr {text_db_type_id}[];
+        decimal_point_arr {text_db_type_id}[];
+        actual_number {text_db_type_id};
+        group_divider {text_db_type_id};
+        decimal_point {text_db_type_id};
+      BEGIN
+        SELECT regexp_matches($1, '{numeric_finding_regex}') INTO raw_arr;
+        IF raw_arr IS NULL THEN
+          RETURN NULL;
+        END IF;
+        SELECT array_remove(ARRAY[{actual_numbers_str}], null) INTO actual_number_arr;
+        SELECT array_remove(ARRAY[{group_dividers_str}], null) INTO group_divider_arr;
+        SELECT array_remove(ARRAY[{decimal_points_str}], null) INTO decimal_point_arr;
+        SELECT actual_number_arr[1] INTO actual_number;
+        SELECT group_divider_arr[1] INTO group_divider;
+        SELECT decimal_point_arr[1] INTO decimal_point;
+        RETURN ARRAY[actual_number, group_divider, decimal_point];
+      END;
+    $$ LANGUAGE plpgsql;
+    """
+
+
 def _get_default_type_body_map(source_types, target_type):
     default_cast_str = f"""
         BEGIN


Test patch:
diff --git a/db/tests/tables/operations/test_infer_types.py b/db/tests/tables/operations/test_infer_types.py
--- a/db/tests/tables/operations/test_infer_types.py
+++ b/db/tests/tables/operations/test_infer_types.py
@@ -9,6 +9,25 @@
 
 
 type_data_list = [
+    (
+        PostgresType.TEXT,
+        [
+            "3.14",
+            "1,41",
+            "149,600,000.00",
+            "4.543.000.000,005",
+            "13 800 000 000,00",
+            "7,53,00,00,000.0",
+            "140'004'453.0",
+            "-3.14",
+            "-1,41",
+            "-149,600,000.00",
+            "-4.543.000.000,005",
+            "-13 800 000 000,00",
+            "-7,53,00,00,000.0",
+            "-140'004'453.0"
+        ],
+        PostgresType.NUMERIC),
     (
         PostgresType.NUMERIC,
         [0, 2, 1, 0],
diff --git a/db/tests/types/operations/test_cast.py b/db/tests/types/operations/test_cast.py
--- a/db/tests/types/operations/test_cast.py
+++ b/db/tests/types/operations/test_cast.py
@@ -542,8 +542,18 @@
             },
             PostgresType.NUMERIC: {
                 VALID: [
-                    ("1.2", Decimal("1.2")),
-                    ("1", Decimal("1")),
+                    ("3.14", Decimal("3.14")),
+                    ("123,456.7", Decimal("123456.7")),
+                    ("123.456,7", Decimal("123456.7")),
+                    ("123 456,7", Decimal("123456.7")),
+                    ("1,23,456.7", Decimal("123456.7")),
+                    ("123'456.7", Decimal("123456.7")),
+                    ("-3.14", Decimal("-3.14")),
+                    ("-123,456.7", Decimal("-123456.7")),
+                    ("-123.456,7", Decimal("-123456.7")),
+                    ("-123 456,7", Decimal("-123456.7")),
+                    ("-1,23,456.7", Decimal("-123456.7")),
+                    ("-123'456.7", Decimal("-123456.7"))
                 ],
                 INVALID: ["not a number"],
             },
@@ -677,8 +687,18 @@
             },
             PostgresType.NUMERIC: {
                 VALID: [
-                    ("1.2", Decimal("1.2")),
-                    ("1", Decimal("1")),
+                    ("3.14", Decimal("3.14")),
+                    ("123,456.7", Decimal("123456.7")),
+                    ("123.456,7", Decimal("123456.7")),
+                    ("123 456,7", Decimal("123456.7")),
+                    ("1,23,456.7", Decimal("123456.7")),
+                    ("123'456.7", Decimal("123456.7")),
+                    ("-3.14", Decimal("-3.14")),
+                    ("-123,456.7", Decimal("-123456.7")),
+                    ("-123.456,7", Decimal("-123456.7")),
+                    ("-123 456,7", Decimal("-123456.7")),
+                    ("-1,23,456.7", Decimal("-123456.7")),
+                    ("-123'456.7", Decimal("-123456.7"))
                 ],
                 INVALID: ["not a number"],
             },
@@ -853,9 +873,9 @@ def test_alter_column_type_alters_column_type(
     (
         PostgresType.CHARACTER_VARYING,
         PostgresType.NUMERIC,
-        {"precision": 5, "scale": 2},
-        "500.134",
-        Decimal("500.13"),
+        {"precision": 6, "scale": 2},
+        "5000.134",
+        Decimal("5000.13"),
     ),
     (
         PostgresType.TIME_WITHOUT_TIME_ZONE,
@@ -1255,3 +1275,31 @@ def test_mathesar_money_array_sql(engine_with_schema, source_str, expect_arr):
             )
         ).scalar()
     assert res == expect_arr
+
+
+numeric_array_examples = [
+    ('3.14', ['3.14', None, '.']),
+    ('331,209.00', ['331,209.00', ',', '.']),
+    ('1,234,567.8910', ['1,234,567.8910', ',', '.']),
+    ('-1,234,567.8910', ['1,234,567.8910', ',', '.']),
+    ('3,14', ['3,14', None, ',']),
+    ('331.293,00', ['331.293,00', '.', ',']),
+    ('1.234.567,8910', ['1.234.567,8910', '.', ',']),
+    ('331 293,00', ['331 293,00', ' ', ',']),
+    ('1 234 567,8910', ['1 234 567,8910', ' ', ',']),
+    ('-1 234 567,8910', ['1 234 567,8910', ' ', ',']),
+    ('1,23,45,678.910', ['1,23,45,678.910', ',', '.']),
+    ('1\'\'234\'\'567.8910', ['1\'234\'567.8910', '\'', '.']),
+]
+
+
+@pytest.mark.parametrize("source_str,expect_arr", numeric_array_examples)
+def test_numeric_array_sql(engine_with_schema, source_str, expect_arr):
+    engine, _ = engine_with_schema
+    with engine.begin() as conn:
+        res = conn.execute(
+            select(
+                text(f"mathesar_types.get_numeric_array('{source_str}'::text)")
+            )
+        ).scalar()
+    assert res == expect_arr
diff --git a/mathesar/tests/api/test_table_api.py b/mathesar/tests/api/test_table_api.py
--- a/mathesar/tests/api/test_table_api.py
+++ b/mathesar/tests/api/test_table_api.py
@@ -450,7 +450,6 @@ def test_table_previews_invalid_type_cast_check(client, type_inference_table, _t
     table = type_inference_table
 
     post_body = _type_inference_table_previews_post_body
-    post_body['columns'][_find_post_body_column_ix_by_name(post_body, 'col_4')]['type'] = PostgresType.NUMERIC.id
     post_body['columns'][_find_post_body_column_ix_by_name(post_body, 'col_5')]['type'] = MathesarCustomType.EMAIL.id
 
     response = client.post(f'/api/db/v0/tables/{table.id}/previews/', data=post_body)