Repository: mathesar-foundation/mathesar

Issue:
Use correct test client parameters when sending json body payload
## Problem
Currently, When sending a request containing a json payload using the Django rest framework test client, the payload is being converted into a string using `json.dumps` but the Django rest framework provides convenience parameters that does this automatically.

## Proposed solution
Use the `format` parameter of the DRF test client or set the default payload format in the DRF settings, in order for the test client to be able to handle the conversion automatically


Discussion:
@silentninja Thanks for opening this issue. Is this ready to be worked on, or do you need to include additional details?
This is ready to be worked on but have to check if there is any feedback before marking it ready
@silentninja if you'd like feedback on an issue, the best way to get it is tag the people you need feedback from in a issue comment or bring the issue up on Matrix.

I think it's a good idea to switch to the DRF test client.
Hi! 
Can I help you with this issue?
Hey! @andruwwwka Sure, I have assigned it to you.
@silentninja thx!
There is a slight change in the issue, it seems like we are already using [DRF test client](https://github.com/centerofci/mathesar/blob/e55088bf7100c9ca4347c2a251ac0e424f71573b/mathesar/tests/api/conftest.py#L22). But the test cases are not using correct options for sending json data, let me know if you are okay to work on it I will update the issue
My IDE was pointing to the wrong client fixture, but the test cases seems to be using the correct API client from [conftest](https://github.com/centerofci/mathesar/blob/e55088bf7100c9ca4347c2a251ac0e424f71573b/mathesar/tests/api/conftest.py#L22), the problem seems to be with improper settings
@silentninja You are right, DRF test client is already being used.
The request format is not specified in the test cases. For example:
```python
def test_column_update_delete_default(column_test_table, client):
    cache.clear()
    expt_default = None
    data = json.dumps({"default": None})
    response = client.patch(
        f"/api/v0/tables/{column_test_table.id}/columns/2/", data=data,
        content_type="application/json"
    )
    assert response.json()["default"] == expt_default
```
It works fine if we set `format` parameter for `post`, `patch`, `put` methods:
```python
def test_column_update_delete_default(column_test_table, client):
    cache.clear()
    expt_default = None
    data = {"default": None}
    response = client.patch(
        f"/api/v0/tables/{column_test_table.id}/columns/2/",
        data=data,
        format="json",
    )
    assert response.json()["default"] == expt_default
```
However, for `get` method if we use nested dict it not works. This is because [get method of APIRequestFactory](https://github.com/encode/django-rest-framework/blob/3.12.4/rest_framework/test.py#L194) use [urlencode from django](https://github.com/django/django/blob/3.1.13/django/utils/http.py#L104) for convert query string. It not support nested dict and convert not-string nested values via str() function. It generates non-serializable strings. In this case dumps is a true way
For example:
```python
def test_record_list_filter(create_table, client):
    table_name = 'NASA Record List Filter'
    table = create_table(table_name)

    filter_list = [
        {'or': [
            {'and': [
                {'field': 'Center', 'op': '==', 'value': 'NASA Ames Research Center'},
                {'field': 'Case Number', 'op': '==', 'value': 'ARC-14048-1'}
            ]},
            {'and': [
                {'field': 'Center', 'op': '==', 'value': 'NASA Kennedy Space Center'},
                {'field': 'Case Number', 'op': '==', 'value': 'KSC-12871'}
            ]}
        ]}
    ]
    json_filter_list = json.dumps(filter_list)

    with patch.object(
        models, "get_records", side_effect=models.get_records
    ) as mock_get:
        response = client.get(
            f'/api/v0/tables/{table.id}/records/?filters={json_filter_list}'
        )
        response_data = response.json()

    assert response.status_code == 200
    assert response_data['count'] == 2
    assert len(response_data['results']) == 2
    assert mock_get.call_args is not None
    assert mock_get.call_args[1]['filters'] == filter_list

```
I can fix the test cases except `get` requests with nested dict
@andruwwwka That's correct, `get` request can only accept strings, so the nested dictionary needs to be converted into a json string and parsed into a json on the backend. This is the reason GraphQL and such use  `post` request
@silentninja Is the issue still relevant? Fix test cases without nested dict in `get` requests?
> @silentninja Is the issue still relevant? Fix test cases without nested dict in `get` requests?

Yes the underlying problem is still the same, so I have edited the issue title and description to reflect the problem correctly. Please go ahead and fix the test cases

Resolving patch:
diff --git a/config/settings.py b/config/settings.py
--- a/config/settings.py
+++ b/config/settings.py
@@ -175,6 +175,7 @@ def pipe_delim(pipe_string):
         'django_filters.rest_framework.DjangoFilterBackend',
         'rest_framework.filters.OrderingFilter',
     ),
+    'TEST_REQUEST_DEFAULT_FORMAT': 'json',
 }
 
 # Mathesar settings


Test patch:
diff --git a/mathesar/tests/api/test_column_api.py b/mathesar/tests/api/test_column_api.py
--- a/mathesar/tests/api/test_column_api.py
+++ b/mathesar/tests/api/test_column_api.py
@@ -1,4 +1,3 @@
-import json
 from datetime import date, timedelta
 
 import pytest
@@ -120,8 +119,7 @@ def test_column_create(column_test_table, client):
     }
     response = client.post(
         f"/api/v0/tables/{column_test_table.id}/columns/",
-        data=json.dumps(data),
-        content_type='application/json'
+        data=data,
     )
     assert response.status_code == 201
     new_columns_response = client.get(
@@ -152,7 +150,7 @@ def test_column_create_default(
     cache.clear()
     name = "anewcolumn"
     data = {"name": name, "type": type_, "default": default}
-    response = client.post(f"/api/v0/tables/{column_test_table.id}/columns/", data)
+    response = client.post(f"/api/v0/tables/{column_test_table.id}/columns/", data, format='multipart')
     assert response.status_code == 201
 
     # Ensure the correct serialized date is returned by the API
@@ -196,8 +194,7 @@ def test_column_create_retrieve_options(column_test_table, client, type_, type_o
     }
     response = client.post(
         f"/api/v0/tables/{column_test_table.id}/columns/",
-        data=json.dumps(data),
-        content_type='application/json'
+        data=data,
     )
     assert response.status_code == 201
     new_columns_response = client.get(
@@ -228,8 +225,7 @@ def test_column_create_bad_options(column_test_table, client, type_options):
     }
     response = client.post(
         f"/api/v0/tables/{column_test_table.id}/columns/",
-        data=json.dumps(data),
-        content_type='application/json'
+        data=data,
     )
     assert response.status_code == 400
 
@@ -293,7 +289,6 @@ def test_column_update_display_options(column_test_table, client):
     response = client.patch(
         f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/",
         display_options_data,
-        format='json'
     )
     assert response.json()["display_options"] == display_options
 
@@ -318,7 +313,7 @@ def test_column_update_default(column_test_table, client):
 def test_column_update_delete_default(column_test_table, client):
     cache.clear()
     expt_default = None
-    data = json.dumps({"default": None})
+    data = {"default": None}
     response = client.get(
         f"/api/v0/tables/{column_test_table.id}/columns/"
     )
@@ -326,8 +321,8 @@ def test_column_update_delete_default(column_test_table, client):
     column_index = 2
     column_id = columns[column_index]['id']
     response = client.patch(
-        f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/", data=data,
-        content_type="application/json"
+        f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/",
+        data=data,
     )
     assert response.json()["default"] == expt_default
 
@@ -450,7 +445,6 @@ def test_column_update_type_options(column_test_table, client):
     response = client.patch(
         f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/",
         data,
-        format='json'
     )
     assert response.json()["type"] == type_
     assert response.json()["type_options"] == type_options
@@ -469,14 +463,12 @@ def test_column_update_type_options_no_type(column_test_table, client):
     client.patch(
         f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/",
         data,
-        format='json'
     )
     type_options = {"precision": 3, "scale": 1}
     type_option_data = {"type_options": type_options}
     response = client.patch(
         f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/",
         type_option_data,
-        format='json'
     )
     assert response.json()["type"] == type_
     assert response.json()["type_options"] == type_options
@@ -529,8 +521,7 @@ def test_column_update_type_invalid_options(column_test_table, client, type_opti
     column_id = columns[column_index]['id']
     response = client.patch(
         f"/api/v0/tables/{column_test_table.id}/columns/{column_id}/",
-        data=json.dumps(data),
-        content_type='application/json'
+        data=data,
     )
     assert response.status_code == 400
 
diff --git a/mathesar/tests/api/test_data_file_api.py b/mathesar/tests/api/test_data_file_api.py
--- a/mathesar/tests/api/test_data_file_api.py
+++ b/mathesar/tests/api/test_data_file_api.py
@@ -127,7 +127,7 @@ def test_data_file_create_csv(client, csv_filename, header):
 
     with open(csv_filename, 'rb') as csv_file:
         data = {'file': csv_file, 'header': header}
-        response = client.post('/api/v0/data_files/', data)
+        response = client.post('/api/v0/data_files/', data, format='multipart')
     with open(csv_filename, 'r') as csv_file:
         correct_dialect = csv.get_sv_dialect(csv_file)
     check_create_data_file_response(
@@ -140,7 +140,7 @@ def test_data_file_create_csv_long_name(client, csv_filename):
     with open(csv_filename, 'rb') as csv_file:
         with patch.object(os.path, 'basename', lambda _: '0' * 101):
             data = {'file': csv_file}
-            response = client.post('/api/v0/data_files/', data)
+            response = client.post('/api/v0/data_files/', data, format='multipart')
             data_file_dict = response.json()
     assert response.status_code == 400
     assert 'Ensure this filename has at most 100' in data_file_dict['file'][0]
@@ -203,7 +203,7 @@ def test_data_file_create_invalid_file(client):
     with patch.object(csv, "get_sv_dialect") as mock_infer:
         mock_infer.side_effect = InvalidTableError
         with open(file, 'r') as f:
-            response = client.post('/api/v0/data_files/', data={'file': f})
+            response = client.post('/api/v0/data_files/', data={'file': f}, format='multipart')
             response_dict = response.json()
     assert response.status_code == 400
     assert response_dict[0] == 'Unable to tabulate data'
@@ -211,7 +211,7 @@ def test_data_file_create_invalid_file(client):
 
 def test_data_file_create_non_unicode_file(client, non_unicode_csv_filename):
     with open(non_unicode_csv_filename, 'rb') as non_unicode_file:
-        response = client.post('/api/v0/data_files/', data={'file': non_unicode_file})
+        response = client.post('/api/v0/data_files/', data={'file': non_unicode_file}, format='multipart')
     assert response.status_code == 201
 
 
@@ -257,7 +257,7 @@ def test_data_file_create_multiple_source_fields(client, csv_filename, paste_fil
         paste_text = paste_file.read()
     with open(csv_filename, 'rb') as csv_file:
         data = {'file': csv_file, 'paste': paste_text}
-        response = client.post('/api/v0/data_files/', data)
+        response = client.post('/api/v0/data_files/', data, format='multipart')
         response_dict = response.json()
     assert response.status_code == 400
     assert 'Multiple source fields passed:' in response_dict['non_field_errors'][0]
diff --git a/mathesar/tests/api/test_table_api.py b/mathesar/tests/api/test_table_api.py
--- a/mathesar/tests/api/test_table_api.py
+++ b/mathesar/tests/api/test_table_api.py
@@ -418,7 +418,7 @@ def test_table_previews(client, schema, engine_email_type):
             {"name": "col_6", "type": "NUMERIC"}
         ]
     }
-    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body, format='json')
+    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body)
     assert response.status_code == 200
     expect_dict = {
         'name': 'Type Modification Table',
@@ -459,7 +459,7 @@ def test_table_previews_wrong_column_number(client, schema, engine_email_type):
             {"name": "col_6", "type": "NUMERIC"}
         ]
     }
-    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body, format='json')
+    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body)
     assert response.status_code == 400
     assert "number" in response.json()[0]
 
@@ -489,7 +489,7 @@ def test_table_previews_invalid_type_cast(client, schema, engine_email_type):
             {"name": "col_6", "type": "NUMERIC"}
         ]
     }
-    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body, format='json')
+    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body)
     assert response.status_code == 400
     assert "Invalid type" in response.json()[0]
 
@@ -519,7 +519,7 @@ def test_table_previews_invalid_type_cast_check(client, schema, engine_email_typ
             {"name": "col_6", "type": "NUMERIC"}
         ]
     }
-    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body, format='json')
+    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body)
     assert response.status_code == 400
     assert "Invalid type" in response.json()[0]
 
@@ -549,7 +549,7 @@ def test_table_previews_unsupported_type(client, schema, engine_email_type):
             {"name": "col_6", "type": "NUMERIC"}
         ]
     }
-    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body, format='json')
+    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body)
     assert response.status_code == 400
     assert "not supported" in response.json()[0]
 
@@ -569,7 +569,7 @@ def test_table_previews_missing_columns(client, schema, engine_email_type):
     table = Table.objects.get(id=response_table['id'])
 
     post_body = {}
-    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body, format='json')
+    response = client.post(f'/api/v0/tables/{table.id}/previews/', data=post_body)
     assert response.status_code == 400
     assert "columns" in response.json()
 
@@ -1011,7 +1011,7 @@ def test_table_patch_same_table_name(create_table, client):
     body = {'name': table_name}
     # Need to specify format here because otherwise the body gets sent
     # as a multi-part form, which can't handle nested keys.
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
 
     assert response.status_code == 200
     assert response.json()['name'] == table_name
@@ -1027,7 +1027,7 @@ def test_table_patch_columns_and_table_name(create_table, client):
     }
     # Need to specify format here because otherwise the body gets sent
     # as a multi-part form, which can't handle nested keys.
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
 
     response_error = response.json()
     assert response.status_code == 400
@@ -1042,7 +1042,7 @@ def test_table_patch_columns_no_changes(create_table, client, engine_email_type)
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1058,7 +1058,7 @@ def test_table_patch_columns_one_name_change(create_table, client, engine_email_
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1075,7 +1075,7 @@ def test_table_patch_columns_two_name_changes(create_table, client, engine_email
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1091,7 +1091,7 @@ def test_table_patch_columns_one_type_change(create_table, client, engine_email_
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1128,7 +1128,7 @@ def test_table_patch_columns_multiple_type_change(create_data_types_table, clien
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1150,7 +1150,7 @@ def test_table_patch_columns_one_drop(create_data_types_table, client, engine_em
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1168,7 +1168,7 @@ def test_table_patch_columns_multiple_drop(create_data_types_table, client, engi
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1185,7 +1185,7 @@ def test_table_patch_columns_diff_name_type_change(create_data_types_table, clie
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1202,7 +1202,7 @@ def test_table_patch_columns_same_name_type_change(create_data_types_table, clie
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1221,7 +1221,7 @@ def test_table_patch_columns_multiple_name_type_change(create_data_types_table,
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1239,7 +1239,7 @@ def test_table_patch_columns_diff_name_type_drop(create_data_types_table, client
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1258,7 +1258,7 @@ def test_table_patch_columns_same_name_type_drop(create_data_types_table, client
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 200
@@ -1274,7 +1274,7 @@ def test_table_patch_columns_invalid_type(create_data_types_table, client, engin
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
 
     assert response.status_code == 400
@@ -1291,7 +1291,7 @@ def test_table_patch_columns_invalid_type_with_name(create_data_types_table, cli
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
     assert response.status_code == 400
     assert 'Pizza is not a boolean' in response_json[0]
@@ -1312,7 +1312,7 @@ def test_table_patch_columns_invalid_type_with_type(create_data_types_table, cli
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
     assert response.status_code == 400
     assert 'Pizza is not a boolean' in response_json[0]
@@ -1333,7 +1333,7 @@ def test_table_patch_columns_invalid_type_with_drop(create_data_types_table, cli
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
     assert response.status_code == 400
     assert 'Pizza is not a boolean' in response_json[0]
@@ -1356,7 +1356,7 @@ def test_table_patch_columns_invalid_type_with_multiple_changes(create_data_type
     body = {
         'columns': column_data
     }
-    response = client.patch(f'/api/v0/tables/{table.id}/', body, format='json')
+    response = client.patch(f'/api/v0/tables/{table.id}/', body)
     response_json = response.json()
     assert response.status_code == 400
     assert 'Pizza is not a boolean' in response_json[0]