Repository: mathesar-foundation/mathesar

Issue:
The type inference algorithm should use `TEXT` rather than `VARCHAR`
## Reproduce

1. "New Table" > "Import Data" > "Copy and Paste Text"

1. Paste the following data and proceed to create and view the table.

    ```txt
    first_name
    Marge
    Homer
    Lisa
    Bart
    Maggie
    ```

1. From the `columns` API, expect the response for the `first_name` column to have `"type": "TEXT"`

1. Observe instead that the column is `VARCHAR` without a length set.

## Rationale

- I spoke with @kgodey about the Mathesar Text type today and she say that Mathesar should only be configuring either: `TEXT` columns or `VARCHAR` columns with a length specified. She may be able to elaborate on the thinking that went into this decision.


## Additional context

- In #1118, we are doing some work to bring the front end into alignment with the above expectations when the user manually configures the DB settings for the Text type.




Discussion:
If we _do_ want to always set a length for `VARCHAR` columns (@kgodey is there a place where you've made it clear why you prefer that?), then I strongly suggest using `TEXT` for the default textual database type. Inferring the length based on some subset of the data is going to be quite brittle.
@mathemancer Agreed on using `TEXT` for the default textual database type. 

> @kgodey is there a place where you've made it clear why you prefer that?

I don't have a preference for `VARCHAR` vs. `TEXT` from a backend perspective. During a call with @seancolsen and @pavish last week, it seemed like the logic for the frontend type selection would be simplified if they could use `VARCHAR` for things that had a length and `TEXT` for things that did not have a length. I don't see any harm in us adopting that convention.
@mathemancer 

> I strongly suggest using TEXT for the default textual database type

Agreed. For the default configuration in the UI, I wrote some specs in  #1118 to the same effect. Though perhaps the default type is only a concern on the back end since we don't give the user the opportunity to specify their type before creating a new column.
> I don't see any harm in us adopting that convention.

I don't see any harm either.

To the implementer: This change needs to be made in the inference logic. The back end does not currently set default database types for Mathesar types.
I've updated the ticket description to be more specific about the expected behavior -- specifying `TEXT` instead of "`TEXT` or `VARCHAR` with a length"
I would like to work on this issue. Can anyone please elaborate on this issue and how to get started?
Hi @vrutik2809 ! I've assigned the issue to you. To get started, follow the readme to get the project working locally. If you have trouble, please ask on the appropriate Matrix channel. Once that's working, take a look at:
- `db/columns/operations/infer_types.py` -- That's where the type inference logic is defined.
- `db/tables/operations/create.py` -- That's where the type for a newly imported table or column is set.

These need to be modified to avoid inferring `VARCHAR` for a new column or column of a new table. Instead, we should infer `TEXT` as the type of the column.

Please @ me if you have any other questions.
Okay i will start working on it and will seek help if needed.

Thank you
While adding the column named `test` in my table, I got the following object as `request.data` : 
```python
{
  'name': 'test', 
  'type': 'varchar', 
  'nullable': True, 
  'primary_key': False
}
```
So, I need to change the type from `varchar` to`text` in this.
@mathemancer Please correct me if I'm wrong
@vrutik2809 This is a frontend request for creating a new column, which also needs to be updated as part of this issue (We could remove type from the request). This issue mainly focuses on the backend logic.

@mathemancer has specified the files that need to be looked at in the above comment: https://github.com/centerofci/mathesar/issues/1121#issuecomment-1066903367

There are two separate places where we need to make changes:
1. When importing a new file, we use the endpoint `/api/db/v0/tables/${tableId}/type_suggestions/` to infer types for the table. The default for text is inferred to be varchar.
2. When creating a new column using the endpoint `/api/db/v0/tables/${tableId}/columns`.

Try making requests to these endpoints directly without using the frontend to get a better idea. Feel free to ask impromptu questions on our [matrix channels](https://wiki.mathesar.org/en/community/matrix).
Sure I will definitely look into that.

Thank you

Resolving patch:
diff --git a/db/columns/operations/infer_types.py b/db/columns/operations/infer_types.py
--- a/db/columns/operations/infer_types.py
+++ b/db/columns/operations/infer_types.py
@@ -21,7 +21,7 @@
     base.PostgresType.NUMERIC.value: [
         base.PostgresType.BOOLEAN.value,
     ],
-    base.STRING: [
+    base.PostgresType.TEXT.value: [
         base.PostgresType.BOOLEAN.value,
         base.PostgresType.DATE.value,
         base.PostgresType.NUMERIC.value,
@@ -44,9 +44,9 @@ def _get_reverse_type_map(engine):
     reverse_type_map = {v: k for k, v in supported_types.items()}
     reverse_type_map.update(
         {
-            Text: base.STRING,
-            TEXT: base.STRING,
-            VARCHAR: base.STRING,
+            Text: base.PostgresType.TEXT.value,
+            TEXT: base.PostgresType.TEXT.value,
+            VARCHAR: base.PostgresType.TEXT.value,
         }
     )
     return reverse_type_map
diff --git a/db/tables/operations/create.py b/db/tables/operations/create.py
--- a/db/tables/operations/create.py
+++ b/db/tables/operations/create.py
@@ -1,4 +1,4 @@
-from sqlalchemy import Column, String, Table, MetaData
+from sqlalchemy import Column, TEXT, Table, MetaData
 from sqlalchemy.ext import compiler
 from sqlalchemy.schema import DDLElement
 
@@ -34,7 +34,7 @@ def create_string_column_table(name, schema, column_names, engine):
     This method creates a Postgres table in the specified schema, with all
     columns being String type.
     """
-    columns_ = [Column(name=column_name, type_=String) for column_name in column_names]
+    columns_ = [Column(name=column_name, type_=TEXT) for column_name in column_names]
     table = create_mathesar_table(name, schema, columns_, engine)
     return table
 


Test patch:
diff --git a/mathesar/tests/api/test_table_api.py b/mathesar/tests/api/test_table_api.py
--- a/mathesar/tests/api/test_table_api.py
+++ b/mathesar/tests/api/test_table_api.py
@@ -377,8 +377,8 @@ def test_table_type_suggestion(client, schema, engine_email_type):
         'col_1': 'NUMERIC',
         'col_2': 'BOOLEAN',
         'col_3': 'BOOLEAN',
-        'col_4': 'VARCHAR',
-        'col_5': 'VARCHAR',
+        'col_4': 'TEXT',
+        'col_5': 'TEXT',
         'col_6': 'NUMERIC'
     }
     response = client.get(f'/api/db/v0/tables/{table.id}/type_suggestions/')
@@ -1119,16 +1119,16 @@ def _get_data_types_column_data():
         'type': 'INTEGER'
     }, {
         'name': 'Integer',
-        'type': 'VARCHAR'
+        'type': 'TEXT'
     }, {
         'name': 'Boolean',
-        'type': 'VARCHAR'
+        'type': 'TEXT'
     }, {
         'name': 'Text',
-        'type': 'VARCHAR'
+        'type': 'TEXT'
     }, {
         'name': 'Decimal',
-        'type': 'VARCHAR'
+        'type': 'TEXT'
     }]