internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mathesar-foundation/mathesar

Issue:
install.sh fails with empty secret_key on Mac OS Ventura
## Description

Mathesar fails to start because secret_key is empty in .env file after running `install.sh`. The script also fails due to that and steps after that do not run.

Note: This happens on Mac OS Ventura, but seems to work fine on Big Sur


Discussion:

Resolving patch:
diff --git a/db/engine.py b/db/engine.py
--- a/db/engine.py
+++ b/db/engine.py
@@ -1,14 +1,11 @@
 import copy
 
 from sqlalchemy import create_engine as sa_create_engine
+from sqlalchemy.engine import URL
 
 from db.types.custom.base import CUSTOM_DB_TYPE_TO_SA_CLASS
 
 
-def get_connection_string(username, password, hostname, database, port='5432'):
-    return f"postgresql://{username}:{password}@{hostname}:{port}/{database}"
-
-
 def create_future_engine_with_custom_types(
         username, password, hostname, database, port, *args, **kwargs
 ):
@@ -25,21 +22,26 @@ def create_future_engine_with_custom_types(
 def create_future_engine(
         username, password, hostname, database, port, *args, **kwargs
 ):
-    conn_str = get_connection_string(
-        username, password, hostname, database, port
+    conn_url = URL.create(
+        "postgresql",
+        username=username,
+        password=password,
+        host=hostname,
+        database=database,
+        port=port,
     )
     kwargs.update(future=True)
-    return create_engine(conn_str, *args, **kwargs)
+    return create_engine(conn_url, *args, **kwargs)
 
 
 # NOTE: used in testing, hence public
-def create_engine(conn_str, *args, **kwargs):
+def create_engine(conn_url, *args, **kwargs):
     """
     Wrapper over sqlalchemy.create_engine that stops SA from propagating changes to ischema_names
     across all engines. This is important for testing: without this intervention, fixtures become
     randomly corrupted.
     """
-    engine = sa_create_engine(conn_str, *args, **kwargs)
+    engine = sa_create_engine(conn_url, *args, **kwargs)
     _make_ischema_names_unique(engine)
     return engine
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Issues with installation process that connects an existing DB
- [x] Tester Marius reports (server credentials in Upwork)
  - [ ] It seems that even if you select existing database, it still tries to start a docker container for the database, creating a conflict?
- [x] Tester Mohammad reports an error as well ([details here](https://docs.google.com/document/d/15m9eZFocAsU1V9inLKxC6i_KQxMdu28snRrBPOrf5Hk/edit))


Discussion:
Current thinking is:
- Marius's issue stems from trying to wire up to a DB on localhost.
- Muhammad's stems from a disallowed character in the DB name.

I'll modify the script to complain about those things. 

Resolving patch:
diff --git a/db/install.py b/db/install.py
--- a/db/install.py
+++ b/db/install.py
@@ -53,7 +53,7 @@ def _create_database(database_name, hostname, username, password, port, skip_con
         )
         with root_db_engine.connect() as conn:
             conn.execution_options(isolation_level="AUTOCOMMIT")
-            conn.execute(text(f"CREATE DATABASE {database_name}"))
+            conn.execute(text(f'CREATE DATABASE "{database_name}"'))
         root_db_engine.dispose()
         print(f"Created DB is {database_name}.")
         return True


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
1.0 is not a boolean
## Description
The software detects `1.0` as a boolean instead of a numeric, which makes the system fail.

UI: `When making an XHR request, the server responded with an error, but the response body was not valid JSON.`
Database: `ERROR:  1.0 is not a boolean`

## Expected behavior

The software should cast `1.0` to numeric.
If the auto-casting mechanism fails, an informative error should be shown.

## To Reproduce

Create a test CSV file and import it.
```column
1.0
```


Discussion:
Be aware that it is not possible to change to numeric if you have more than one column affected by this issue, as the software would state `Data Type Change Failed: When making an XHR request, the server responded with an error, but the response body was not valid JSON.`, thus making it impossible to import the data.
Thank you for reporting this, @frafra. We will fix in our next release (next week).

Resolving patch:
diff --git a/db/columns/operations/infer_types.py b/db/columns/operations/infer_types.py
--- a/db/columns/operations/infer_types.py
+++ b/db/columns/operations/infer_types.py
@@ -18,9 +18,7 @@
     PostgresType.BOOLEAN: [],
     MathesarCustomType.EMAIL: [],
     PostgresType.INTERVAL: [],
-    PostgresType.NUMERIC: [
-        PostgresType.BOOLEAN,
-    ],
+    PostgresType.NUMERIC: [],
     PostgresType.TEXT: [
         PostgresType.BOOLEAN,
         PostgresType.DATE,


Test patch:
diff --git a/db/tests/tables/operations/test_infer_types.py b/db/tests/tables/operations/test_infer_types.py
--- a/db/tests/tables/operations/test_infer_types.py
+++ b/db/tests/tables/operations/test_infer_types.py
@@ -28,7 +28,28 @@
             "-7,53,00,00,000.0",
             "-140'004'453.0"
         ],
-        PostgresType.NUMERIC),
+        PostgresType.NUMERIC
+    ),
+    (
+        PostgresType.TEXT,
+        ["1.0"],
+        PostgresType.NUMERIC
+    ),
+    (
+        PostgresType.TEXT,
+        ["1"],
+        PostgresType.BOOLEAN
+    ),
+    (
+        PostgresType.TEXT,
+        ["0"],
+        PostgresType.BOOLEAN
+    ),
+    (
+        PostgresType.TEXT,
+        ["1", "0"],
+        PostgresType.BOOLEAN
+    ),
     (
         PostgresType.NUMERIC,
         [0, 2, 1, 0],
@@ -37,7 +58,7 @@
     (
         PostgresType.NUMERIC,
         [0, 1, 1, 0],
-        PostgresType.BOOLEAN
+        PostgresType.NUMERIC,
     ),
     (
         PostgresType.TEXT,
Repository: mathesar-foundation/mathesar

Issue:
Django erroneously reports makemigrations is needed
There is a problem with Django migration changes detector when running `migrate` command after setting up Django using `django,setup()`. For some reason, it is considering `mathesar.models.query.UIQuery` model to be missing. 


Discussion:
Hi @kgodey 
I want to work on this issue.
I'd be grateful if you could assign this issue to me.


Go ahead @Sukriti-m, thanks!
@kgodey @Sukriti-m  `management.call_command('migrate')` are lazy it creates migration only for those models which are registered on `admin.site`, registering `UIQuery` on `admin.py` should resolve this issue
Thanks @aagman945!
Unassigned due to inactivity.
Can I work on this issue @Anish9901 
Go ahead @Sukshan.

Resolving patch:
diff --git a/mathesar/admin.py b/mathesar/admin.py
--- a/mathesar/admin.py
+++ b/mathesar/admin.py
@@ -3,6 +3,7 @@
 
 from mathesar.models.base import Table, Schema, DataFile
 from mathesar.models.users import User
+from mathesar.models.query import UIQuery
 
 
 class MathesarUserAdmin(UserAdmin):
@@ -22,3 +23,4 @@ class MathesarUserAdmin(UserAdmin):
 admin.site.register(Schema)
 admin.site.register(DataFile)
 admin.site.register(User, MathesarUserAdmin)
+admin.site.register(UIQuery)


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Database removed from .env file does not get removed
## Description
* Add a new database to .env file, start Mathesar. It gets added.
* Remove the database and restart Mathesar. Notice that it does not get removed.
* I added two databases `mt1` and `mt2`. I removed `mt2`. This was the response from the server within `common_data.databases`:
   ```
    [
      {
        "id": 1,
        "name": "mt2",
        "deleted": false,
        "supported_types_url": "http://localhost:8000/api/ui/v0/databases/1/types/"
      },
      {
        "id": 2,
        "name": "mt1",
        "deleted": false,
        "supported_types_url": "http://localhost:8000/api/ui/v0/databases/2/types/"
      }
    ]
   ```
* Note that 'mt2' has id 1 (I'm not sure if this was the id it had before), and that the value of `deleted` is `false`.


Discussion:
I'm placing this in `0.1.2` and it will be moved to `Next release` if the fix doesn't make it in time.

Resolving patch:
diff --git a/mathesar/database/base.py b/mathesar/database/base.py
--- a/mathesar/database/base.py
+++ b/mathesar/database/base.py
@@ -13,7 +13,10 @@ def create_mathesar_engine(db_name):
     try:
         credentials = _get_credentials_for_db_name_in_settings(db_name)
     except KeyError:
-        credentials = _get_credentials_for_db_name_not_in_settings(db_name)
+        if hasattr(settings, 'MATHESAR_LIVE_DEMO') and settings.MATHESAR_LIVE_DEMO:
+            credentials = _get_credentials_for_db_name_not_in_settings(db_name)
+        else:
+            raise
     return engine.create_future_engine_with_custom_types(**credentials)
 
 
diff --git a/mathesar/views.py b/mathesar/views.py
--- a/mathesar/views.py
+++ b/mathesar/views.py
@@ -33,11 +33,11 @@ def get_schema_list(request, database):
 
 
 def _get_permissible_db_queryset(request):
-    qs = Database.objects.all()
+    qs = Database.objects.filter(deleted=False)
     permission_restricted_qs = DatabaseAccessPolicy.scope_queryset(request, qs)
     schema_qs = Schema.objects.all()
     permitted_schemas = SchemaAccessPolicy.scope_queryset(request, schema_qs)
-    databases_from_permitted_schema = Database.objects.filter(schemas__in=permitted_schemas)
+    databases_from_permitted_schema = Database.objects.filter(schemas__in=permitted_schemas, deleted=False)
     permission_restricted_qs = permission_restricted_qs | databases_from_permitted_schema
     return permission_restricted_qs.distinct()
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Integrate DRF Spectacular with Mathesar's backend codebase
## Why DRF Spectacular
-DRF Spectacular automatically generates comprehensive documentation based on OpenAPI specification for Mathesar's API. 
-It is compatible with Mathesar's Django REST Framework
## Proposed solution
-Add the "drf-spectacular" package to the project's dependencies
-Configure the SPECTACULAR_SETTINGS in the Mathesar's settings file 





Discussion:

Resolving patch:
diff --git a/config/settings/common_settings.py b/config/settings/common_settings.py
--- a/config/settings/common_settings.py
+++ b/config/settings/common_settings.py
@@ -42,6 +42,7 @@ def pipe_delim(pipe_string):
     "rest_framework",
     "django_filters",
     "django_property_filter",
+    "drf_spectacular",
     "mathesar",
 ]
 
@@ -183,7 +184,15 @@ def pipe_delim(pipe_string):
     ),
     'TEST_REQUEST_DEFAULT_FORMAT': 'json',
     'EXCEPTION_HANDLER':
-        'mathesar.exception_handlers.mathesar_exception_handler'
+        'mathesar.exception_handlers.mathesar_exception_handler',
+    'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema'
+}
+SPECTACULAR_SETTINGS = {
+    'TITLE': 'Mathesar API',
+    'DESCRIPTION': '',
+    'VERSION': '1.0.0',
+    'SERVE_INCLUDE_SCHEMA': False,
+    # OTHER SETTINGS
 }
 FRIENDLY_ERRORS = {
     'FIELD_ERRORS': {


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Improve OperationIDs for the Endpoints
## Problem
The operationId is used by SDK generator to create corresponding functions in client library. The current operationId generated by the DRF Spectacular library for the `/api/db/v0/data_files/ endpoint` is `api_db_v0_data_files_list`. However, this operationId is not optimal. This can make it less intuitive to work with when generating SDK functions or utilizing the API documentation. To improve the operationId for the `/api/db/v0/data_files/` endpoint, a more concise and meaningful name, such as `data_files_list` or `list_data_files` would be apt.


## Additional context
Check the discussion [here](https://github.com/centerofci/mathesar/pull/2971#discussion_r1247611153) for better understanding of the problem


Discussion:

Resolving patch:
diff --git a/config/settings/common_settings.py b/config/settings/common_settings.py
--- a/config/settings/common_settings.py
+++ b/config/settings/common_settings.py
@@ -192,6 +192,9 @@ def pipe_delim(pipe_string):
     'DESCRIPTION': '',
     'VERSION': '1.0.0',
     'SERVE_INCLUDE_SCHEMA': False,
+    'POSTPROCESSING_HOOKS': [
+        'config.settings.openapi.remove_url_prefix_hook',
+    ],
     # OTHER SETTINGS
 }
 FRIENDLY_ERRORS = {
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
new file mode 100644
--- /dev/null
+++ b/config/settings/openapi.py
@@ -0,0 +1,12 @@
+def remove_url_prefix_hook(result, **kwargs):
+    # Remove namespace and version URL prefix from the operation Id of the generated API schema
+    for path, path_info in result['paths'].items():
+        for method, operation in path_info.items():
+            operation_id = operation.get('operationId')
+            if operation_id:
+                if path.startswith('/api/db/v0/'):
+                    operation['operationId'] = operation_id.replace('db_v0_', '')
+                elif path.startswith('/api/ui/v0/'):
+                    operation['operationId'] = operation_id.replace('ui_v0_', '')
+
+    return result


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add OpenAPI Specification for the /datafiles/ endpoint and fix the incorrect specs
## Problem
In order to ensure the accuracy of the specs generated by drf-spectacular for /datafiles/ endpoint , we will generate an OpenAPI Specification specifically for this endpoint. This will allow us to verify its correctness by comparing it with the existing test cases and the DRF browsable API page and also confirm it with the front-end team.

## Proposed solution
* Implement custom preprocessing hook function  to filter out all endpoints except for the /datafiles/ endpoint.The hook function selectively retains only the endpoint paths that match the /datafiles/ 
* Update the Spectacular settings in the  _common_settings.py_ file to include the custom preprocessing hook.
* Configure the PREPROCESSING_HOOKS setting with the custom hook function, we ensure that only the /datafiles/ endpoint is considered during the OpenAPI specification generation process.
* Finally, generate the spec file using the _./manage.py spectacular --color --file schema.yml_ command




Discussion:

Resolving patch:
diff --git a/config/settings/common_settings.py b/config/settings/common_settings.py
--- a/config/settings/common_settings.py
+++ b/config/settings/common_settings.py
@@ -192,6 +192,7 @@ def pipe_delim(pipe_string):
     'DESCRIPTION': '',
     'VERSION': '1.0.0',
     'SERVE_INCLUDE_SCHEMA': False,
+    'PREPROCESSING_HOOKS': ['config.settings.openapi.custom_preprocessing_hook'],
     'POSTPROCESSING_HOOKS': [
         'config.settings.openapi.remove_url_prefix_hook',
     ],
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -1,3 +1,12 @@
+def custom_preprocessing_hook(endpoints):
+    filtered = []
+    for (path, path_regex, method, callback) in endpoints:
+        # Remove all but DRF API endpoints
+        if path.startswith("/api/db/v0/data_files/"):
+            filtered.append((path, path_regex, method, callback))
+    return filtered
+
+
 def remove_url_prefix_hook(result, **kwargs):
     # Remove namespace and version URL prefix from the operation Id of the generated API schema
     for path, path_info in result['paths'].items():


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add OpenAPI Specification for /databases/ endpoint 
## Problem
In order to ensure the accuracy of the specs generated by drf-spectacular for /databases/ endpoint , we will generate an OpenAPI Specification specifically for this endpoint. This will allow us to verify its correctness by comparing it with the existing test cases and the DRF browsable API page.

## Proposed solution
* Implement custom preprocessing hook function  to filter out all endpoints except for the /databases/ endpoint.The hook function selectively retains only the endpoint paths that match the /datafiles/ 
* Configure the PREPROCESSING_HOOKS setting with the custom hook function, we ensure that only the /datafiles/ endpoint is considered during the OpenAPI specification generation process.
* Finally, generate the spec file using the _./manage.py spectacular --color --file schema.yml_ command




Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -2,7 +2,7 @@ def custom_preprocessing_hook(endpoints):
     filtered = []
     for (path, path_regex, method, callback) in endpoints:
         # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/data_files/"):
+        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/"):
             filtered.append((path, path_regex, method, callback))
     return filtered
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Not really possible to override settings
## Description
I want to override the mathsar settings in order to allow LDAP login using django-auth-ldap. I changed the `config/settings/production.py` file that mentions: `# Override default settings                ` and added the needed configuration. 

This worked fine however that file is under version control so if it is changed on the origin I'll get a conflict and would need to also merge my changes. The usual way to implement this functionality is to add a *non tracked* `local.py` file that would contain any extra configuration for each environment (either dev or production) and import *that* file from the corresponding file. I.e the production.py would be changed to:

```python
# Override default settings               

try:
    from .local import *
except ImportError:
    pass
```

This way, if the local.py file is there it will be used to override the config but if it isnt' there it will be ignored. 

## Expected behavior
Being able to override django settings for my environment *without* keeping a fork.

## To Reproduce
Change the production.py file and you'll see that it's version controlled so it can't be easily changed!

## Environment
Not needed

## Additional context
I'd be happy to provide a PR implementing the functionality described here, i.e allow an untracked local.py file to override django settings for each *user/environment*.


Discussion:
@spapas We'd be happy to accept a PR for this functionality. Thanks!

It would be great if you could also let us know how integrating `django-auth-ldap` is going. We've had other requests for implementing SSO integration ([see this discussion](https://github.com/centerofci/mathesar/discussions/2291)) and we're exploring different options for how to implement that.

Resolving patch:
diff --git a/config/settings/development.py b/config/settings/development.py
--- a/config/settings/development.py
+++ b/config/settings/development.py
@@ -1,3 +1,10 @@
 from config.settings.common_settings import * # noqa
 
 # Override default settings
+
+
+# Use a local.py module for settings that shouldn't be version tracked
+try:
+    from .local import * # noqa
+except ImportError:
+    pass
diff --git a/config/settings/production.py b/config/settings/production.py
--- a/config/settings/production.py
+++ b/config/settings/production.py
@@ -1,3 +1,10 @@
 from config.settings.common_settings import * # noqa
 
 # Override default settings
+
+
+# Use a local.py module for settings that shouldn't be version tracked
+try:
+    from .local import * # noqa 
+except ImportError:
+    pass


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add OpenAPI Specification for /schemas/ endpoint and fix the incorrect specs
## Problem
In order to ensure the accuracy of the specs generated by drf-spectacular for /schemas/ endpoint , we will generate an OpenAPI Specification specifically for this endpoint. This will allow us to verify its correctness by comparing it with the existing test cases and the DRF browsable API page.

## Proposed solution
* Implement custom preprocessing hook function  to filter out all endpoints except for the /schemas/ endpoint.The hook function selectively retains only the endpoint paths that match the /schemas/ 
* Configure the PREPROCESSING_HOOKS setting with the custom hook function, we ensure that only the /schemas/ endpoint is considered during the OpenAPI specification generation process.
* Finally, generate the spec file using the `manage.py spectacular --color --file schema.yml` command


Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -2,7 +2,7 @@ def custom_preprocessing_hook(endpoints):
     filtered = []
     for (path, path_regex, method, callback) in endpoints:
         # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/"):
+        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/"):
             filtered.append((path, path_regex, method, callback))
     return filtered
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Remove redundant Request body content types
Currently, the requestBody section of some API endpoints specifies three content types: application/json, application/x-www-form-urlencoded, and multipart/form-data. However, in our application, we only use the application/json content type for request payloads.

Example :
```requestBody:
  content:
    application/json:
      schema:
        $ref: '#/components/schemas/JsonDataFile'
    application/x-www-form-urlencoded:
      schema:
        $ref: '#/components/schemas/DataFile'
    multipart/form-data:
      schema:
        $ref: '#/components/schemas/DataFile'
```

**Proposed Solution:**
 The unnecessary application/x-www-form-urlencoded and multipart/form-data content types from the requestBody should be removed.To achieve this we need to ensure that our API views are set to use the JSONParser class as the default parser for incoming requests. This can be achieved by updating our Django REST Framework settings as follows:

```
# common_settings.py
REST_FRAMEWORK = {
    'DEFAULT_PARSER_CLASSES': [
        'rest_framework.parsers.JSONParser',
    ],
    # Other DRF settings...
}



Discussion:

Resolving patch:
diff --git a/config/settings/common_settings.py b/config/settings/common_settings.py
--- a/config/settings/common_settings.py
+++ b/config/settings/common_settings.py
@@ -182,6 +182,9 @@ def pipe_delim(pipe_string):
     'DEFAULT_PERMISSION_CLASSES': [
         'rest_framework.permissions.IsAuthenticated',
     ],
+    'DEFAULT_PARSER_CLASSES': [
+        'rest_framework.parsers.JSONParser',
+    ],
     'DEFAULT_FILTER_BACKENDS': (
         'django_filters.rest_framework.DjangoFilterBackend',
         'rest_framework.filters.OrderingFilter',
diff --git a/mathesar/api/db/viewsets/data_files.py b/mathesar/api/db/viewsets/data_files.py
--- a/mathesar/api/db/viewsets/data_files.py
+++ b/mathesar/api/db/viewsets/data_files.py
@@ -2,6 +2,7 @@
 from rest_framework import status, viewsets
 from rest_framework.mixins import CreateModelMixin, ListModelMixin, RetrieveModelMixin
 from rest_framework.response import Response
+from rest_framework.parsers import MultiPartParser, JSONParser
 
 from mathesar.api.dj_filters import DataFileFilter
 import mathesar.api.exceptions.data_import_exceptions.exceptions
@@ -21,6 +22,7 @@ class DataFileViewSet(viewsets.GenericViewSet, ListModelMixin, RetrieveModelMixi
     pagination_class = DefaultLimitOffsetPagination
     filter_backends = (filters.DjangoFilterBackend,)
     filterset_class = DataFileFilter
+    parser_classes = [MultiPartParser, JSONParser]
 
     def partial_update(self, request, pk=None):
         serializer = DataFileSerializer(


Test patch: