internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mathesar-foundation/mathesar

Issue:
Individually run API tests don't build tables database
## Description
Running a individual test in `mathesar` that doesn't use the `engine` or `test_db` fixture will not have the tables databases built for the test. As a result, many will error when trying to access the tables database.

## Expected behavior
The tables database should always be built.

## To Reproduce
Run any test in `mathesar` that doesn't use `engine` or `test_db`. Ex:
```
docker exec mathesar_web_1 pytest mathesar/tests/views/api/test_schema_api.py::test_schema_update
```

## Additional context
Introduced due to the changes in #329, since `pytest-django` no longer creates the tables db for us.



Discussion:

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -15,7 +15,7 @@ def test_db_name():
     return TEST_DB
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="session", autouse=True)
 def test_db():
     superuser_engine = _get_superuser_engine()
     with superuser_engine.connect() as conn:


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Ensure that only Postgres databases are connected
**Problem**
<!-- Please provide a clear and concise description of the problem that this feature request is designed to solve.-->
Django and SQLAlchemy both support multiple database systems, but we're presuming that the user is only connected Postgres databases. This could lead to bugs if the user connects a MySQL or other database.

**Proposed solution**
<!-- A clear and concise description of your proposed solution or feature. -->
We should check to make sure that we're only managing Postgres databases and throw an error if another database system is connected.

I'm not sure if matters what database is used for the Django database, since we're only using standard Django functionality there, but we should definitely check the Mathesar managed databases.


Discussion:

Resolving patch:
diff --git a/config/settings.py b/config/settings.py
--- a/config/settings.py
+++ b/config/settings.py
@@ -88,6 +88,15 @@ def pipe_delim(pipe_string):
 }
 DATABASES[decouple_config('DJANGO_DATABASE_KEY')] = decouple_config('DJANGO_DATABASE_URL', cast=db_url)
 
+for db_key, db_dict in DATABASES.items():
+    # Engine can be '.postgresql' or '.postgresql_psycopg2'
+    if not db_dict['ENGINE'].startswith('django.db.backends.postgresql'):
+        raise ValueError(
+            f"{db_key} is not a PostgreSQL database. "
+            f"{db_dict['ENGINE']} found for {db_key}'s engine."
+        )
+
+
 # pytest-django will create a new database named 'test_{DATABASES[table_db]['NAME']}'
 # and use it for our API tests if we don't specify DATABASES[table_db]['TEST']['NAME']
 if decouple_config('TEST', default=False, cast=bool):


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
New CSV imports use autogenerated column names.
## Description
<!-- A clear and concise description of what the bug is. -->
According to #459, newly imported CSVs are supposed to use the first row as headers by default. However, newly uploaded CSVs are showing autogenerated column names.

## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
The first row of the CSV should be used as header names by default.

## To Reproduce
<!-- How can we recreate this bug? Please try to provide a Minimal, Complete, and Verifiable (http://stackoverflow.com/help/mcve) example if code-related. -->
Upload a CSV from the Mathesar UI. Column names will be of the form `column_0`, etc.


Discussion:
I would like to submit a PR for this issue.

The problem is that when receiving HTML-encoded data, Django treats any omitted Boolean fields as `False`. This means that the `DataFile` model's `header` field is set to `False`, instead of the default value of `True`. [Here is a link DRF's documentation describing this behavior of the Boolean fields](https://www.django-rest-framework.org/api-guide/fields/#booleanfield).

This problem can be solved by changing the `DataFileSerializer`:
```
    class Meta:
        model = DataFile
        fields = [
            --remove 'header'--
        ]
        extra_kwargs = {...
                        'header': {'required': True},}
                        ```
@obulat Hey Olga! Please go ahead and submit a PR, I've assigned the issue to you. I didn't know that about DRF and boolean fields, thanks for the explanation.

Resolving patch:
diff --git a/mathesar/serializers.py b/mathesar/serializers.py
--- a/mathesar/serializers.py
+++ b/mathesar/serializers.py
@@ -100,7 +100,8 @@ class Meta:
         ]
         extra_kwargs = {'delimiter': {'trim_whitespace': False},
                         'escapechar': {'trim_whitespace': False},
-                        'quotechar': {'trim_whitespace': False}}
+                        'quotechar': {'trim_whitespace': False},
+                        'header': {'required': True}}
         # We only currently support importing to a new table, so setting a table via API is invalid.
         # User should be set automatically, not submitted via the API.
         read_only_fields = ['table_imported_to']


Test patch:
diff --git a/mathesar/tests/views/api/test_data_file_api.py b/mathesar/tests/views/api/test_data_file_api.py
--- a/mathesar/tests/views/api/test_data_file_api.py
+++ b/mathesar/tests/views/api/test_data_file_api.py
@@ -18,7 +18,7 @@ def verify_data_file_data(data_file, data_file_dict):
     assert data_file_dict['delimiter'] == data_file.delimiter
     assert data_file_dict['quotechar'] == data_file.quotechar
     assert data_file_dict['escapechar'] == data_file.escapechar
-    assert data_file_dict['header'] == data_file.header
+    assert data_file_dict.get('header', True) == data_file.header
 
 
 @pytest.fixture
Repository: mathesar-foundation/mathesar

Issue:
New CSV imports use autogenerated column names.
## Description
<!-- A clear and concise description of what the bug is. -->
According to #459, newly imported CSVs are supposed to use the first row as headers by default. However, newly uploaded CSVs are showing autogenerated column names.

## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
The first row of the CSV should be used as header names by default.

## To Reproduce
<!-- How can we recreate this bug? Please try to provide a Minimal, Complete, and Verifiable (http://stackoverflow.com/help/mcve) example if code-related. -->
Upload a CSV from the Mathesar UI. Column names will be of the form `column_0`, etc.


Discussion:
I would like to submit a PR for this issue.

The problem is that when receiving HTML-encoded data, Django treats any omitted Boolean fields as `False`. This means that the `DataFile` model's `header` field is set to `False`, instead of the default value of `True`. [Here is a link DRF's documentation describing this behavior of the Boolean fields](https://www.django-rest-framework.org/api-guide/fields/#booleanfield).

This problem can be solved by changing the `DataFileSerializer`:
```
    class Meta:
        model = DataFile
        fields = [
            --remove 'header'--
        ]
        extra_kwargs = {...
                        'header': {'required': True},}
                        ```
@obulat Hey Olga! Please go ahead and submit a PR, I've assigned the issue to you. I didn't know that about DRF and boolean fields, thanks for the explanation.
Hi @kgodey 😄 I didn't know that either, it took some debugging to find out. 
Ah, I forgot to check with defaults!
@obulat The issue is still happening even after your PR was merged (try uploading a CSV using the UI, observe that column names are `column_0`, etc. and the first row is treated as data)
Oh no. I probably broke something while trying to fix tests, let me see what happened


Resolving patch:
diff --git a/mathesar/serializers.py b/mathesar/serializers.py
--- a/mathesar/serializers.py
+++ b/mathesar/serializers.py
@@ -92,6 +92,7 @@ class DataFileSerializer(serializers.ModelSerializer):
     user = serializers.PrimaryKeyRelatedField(
         default=serializers.CurrentUserDefault(), read_only=True
     )
+    header = serializers.BooleanField(default=True)
 
     class Meta:
         model = DataFile
@@ -100,8 +101,8 @@ class Meta:
         ]
         extra_kwargs = {'delimiter': {'trim_whitespace': False},
                         'escapechar': {'trim_whitespace': False},
-                        'quotechar': {'trim_whitespace': False},
-                        'header': {'required': True}}
+                        'quotechar': {'trim_whitespace': False}
+                        }
         # We only currently support importing to a new table, so setting a table via API is invalid.
         # User should be set automatically, not submitted via the API.
         read_only_fields = ['table_imported_to']


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Implement schema list page
**Problem**
<!-- Please provide a clear and concise description of the problem that this feature request is designed to solve.-->
Users should be able to create a new schema, edit schema names, and delete schemas.

**Proposed solution**
<!-- A clear and concise description of your proposed solution or feature. -->
We should provide a way to do these actions from the UI using the schema list page introduced in the [design spec](https://wiki.mathesar.org/en/design/specs/schemas).

**Additional context**
<!-- Add any other context or screenshots about the feature request here.-->
- #166
- #168 
- #170
- #393


Discussion:
Marking as ready because we already have listing and creation APIs and that's enough to get started. This will be fully unblocked once #393 is merged.
As part of this issue, a new route has to be created for schema listing, and only the content of schema listing page is to be developed. We have another issue https://github.com/centerofci/mathesar/issues/362 which handles the changes in the top navigation bar.

In order to facilitate development independently, we can have a placeholder button to navigate to the schema listing page, as part of the PR for this issue.
@dhruvkb How's this coming? Do you need any help?
I'm working on it. I should be able to wrap this up during the upcoming weekend as I didn't get much time during the last.
Sounds good, thanks!

Resolving patch:
diff --git a/mathesar/urls.py b/mathesar/urls.py
--- a/mathesar/urls.py
+++ b/mathesar/urls.py
@@ -1,4 +1,4 @@
-from django.urls import include, path
+from django.urls import include, path, re_path
 from rest_framework_nested import routers
 
 from mathesar.views import api, frontend
@@ -20,6 +20,6 @@
     path('api/v0/', include(router.urls)),
     path('api/v0/', include(table_router.urls)),
     # TODO: Handle known urls like /favicon.ico etc.,
-    # Currenty, this catches all
-    path('<dbname>', frontend.index, name="index"),
+    # Currently, this catches all
+    re_path(r'(?P<dbname>\w+)/.*$', frontend.index, name="index"),
 ]


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
import_verified field missing in table resource
## Description
<!-- A clear and concise description of what the bug is. -->
The Table API does not include the `import_verified` field.

## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
The frontend needs the `import_verified` field to figure out which tables to show in preview mode and which tables to show normally.


Discussion:

Resolving patch:
diff --git a/mathesar/imports/csv.py b/mathesar/imports/csv.py
--- a/mathesar/imports/csv.py
+++ b/mathesar/imports/csv.py
@@ -117,7 +117,7 @@ def create_table_from_csv(data_file, name, schema):
         data_file, name, schema
     )
     db_table_oid = tables.get_oid_from_table(db_table.name, db_table.schema, engine)
-    table, _ = Table.objects.get_or_create(oid=db_table_oid, schema=schema)
+    table, _ = Table.objects.get_or_create(oid=db_table_oid, schema=schema, import_verified=False)
     data_file.table_imported_to = table
     data_file.save()
     return table
diff --git a/mathesar/serializers.py b/mathesar/serializers.py
--- a/mathesar/serializers.py
+++ b/mathesar/serializers.py
@@ -72,7 +72,7 @@ class TableSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = Table
-        fields = ['id', 'name', 'schema', 'created_at', 'updated_at',
+        fields = ['id', 'name', 'schema', 'created_at', 'updated_at', 'import_verified',
                   'columns', 'records_url', 'constraints_url', 'columns_url', 'data_files', 'has_dependencies']
 
     def get_records_url(self, obj):


Test patch:
diff --git a/mathesar/tests/views/api/test_table_api.py b/mathesar/tests/views/api/test_table_api.py
--- a/mathesar/tests/views/api/test_table_api.py
+++ b/mathesar/tests/views/api/test_table_api.py
@@ -71,6 +71,7 @@ def check_table_response(response_table, table, table_name):
     assert 'created_at' in response_table
     assert 'updated_at' in response_table
     assert 'has_dependencies' in response_table
+    assert 'import_verified' in response_table
     assert len(response_table['columns']) == len(table.sa_column_names)
     for column in response_table['columns']:
         assert column['name'] in table.sa_column_names
Repository: mathesar-foundation/mathesar

Issue:
Uncaught 404s in record viewsets and pagination classes
## Description
The record viewset, column viewset and paignation classes regularly call `Table.objects.get(id=table_pk)`, which throws a `mathesar.models.Table.DoesNotExist: Table matching query does not exist.` when an invalid table id is passed.

To recreate, run `client.get(f'/api/v0/tables/3000/records/')`.


## Expected behavior
We should ensure that the table exists before querying, or catch the `DoesNotExist` error after querying. We should also include tests for table 404s. 

This is probably best done after  #488 is merged, as it includes a function to do exactly this.


Discussion:
@kgodey I would like to handle this issue :)
@linzhou-zhong Assigned, thanks!
@linzhou-zhong Let us know if you need any help with this issue or if you have questions.
@kgodey Yes, sorry for the late reply. I'm working on it and I'll let you know if I need some help. Appreciate !

Resolving patch:
diff --git a/mathesar/api/pagination.py b/mathesar/api/pagination.py
--- a/mathesar/api/pagination.py
+++ b/mathesar/api/pagination.py
@@ -3,6 +3,8 @@
 from rest_framework.pagination import LimitOffsetPagination
 from rest_framework.response import Response
 
+from mathesar.api.utils import get_table_or_404
+
 
 class DefaultLimitOffsetPagination(LimitOffsetPagination):
     default_limit = 50
@@ -22,7 +24,7 @@ def paginate_queryset(self, queryset, request, table_id):
         if self.limit is None:
             self.limit = self.default_limit
         self.offset = self.get_offset(request)
-        table = queryset.get(id=table_id)
+        table = get_table_or_404(pk=table_id)
         self.count = len(table.sa_columns)
         self.request = request
         return list(table.sa_columns)[self.offset:self.offset + self.limit]
@@ -37,7 +39,7 @@ def paginate_queryset(self, queryset, request, table_id,
             self.limit = self.default_limit
         self.offset = self.get_offset(request)
         # TODO: Cache count value somewhere, since calculating it is expensive.
-        table = queryset.get(id=table_id)
+        table = get_table_or_404(pk=table_id)
         self.count = table.sa_num_records(filters=filters)
         self.request = request
 
@@ -60,7 +62,7 @@ def paginate_queryset(self, queryset, request, table_id,
             queryset, request, table_id, filters=filters, order_by=order_by
         )
 
-        table = queryset.get(id=table_id)
+        table = get_table_or_404(pk=table_id)
         if group_count_by:
             group_count = table.get_group_counts(
                 group_count_by, self.limit, self.offset,
diff --git a/mathesar/api/utils.py b/mathesar/api/utils.py
--- a/mathesar/api/utils.py
+++ b/mathesar/api/utils.py
@@ -4,6 +4,13 @@
 
 
 def get_table_or_404(pk):
+    """
+    Get table if it exists, otherwise throws a DRF NotFound error.
+    Args:
+        pk: id of table
+    Returns:
+        table: return the table based on a specific id
+    """
     try:
         table = Table.objects.get(id=pk)
     except Table.DoesNotExist:


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Set up integration testing environment
## Problem
We need to setup an integration test environment for Mathesar, to test all user action scenarios.

## Proposed solution
Based on the discussions we've had previously (#89), the most suitable choice for us is [Playwright](https://playwright.dev/).

Integ tests require a complete mathesar setup, including a database. We should be able to test `user action -> api calls -> changes in db`.

## Additional context
- #89 


Discussion:

Resolving patch:
diff --git a/install.py b/install.py
--- a/install.py
+++ b/install.py
@@ -39,7 +39,7 @@ def install_on_db_with_key(database_key, skip_confirm):
         host = DATABASES[database_key]["HOST"]
         db_name = DATABASES[database_key]["NAME"]
         port = DATABASES[database_key]["PORT"]
-        print("Installing Mathesar DB on preexisting PostgreSQL instance...")
+        print(f"Installing Mathesar DB {db_name} on preexisting PostgreSQL instance at host {host}...")
         if skip_confirm is False:
             confirmation = input(
                 f"Mathesar will be installed on DB {db_name} at host {host}."


Test patch:
diff --git a/integration_tests/test_base_page.py b/integration_tests/test_base_page.py
new file mode 100644
--- /dev/null
+++ b/integration_tests/test_base_page.py
@@ -0,0 +1,6 @@
+def test_base_page(browser):
+    context = browser.new_context(record_video_dir='videos/')
+    page = context.new_page()
+    page.goto('http://localhost:8000')
+    assert page.inner_text('h1') == 'Welcome to Mathesar!'
+    context.close()
Repository: mathesar-foundation/mathesar

Issue:
Install.py script assumes database to exist
## Description
The install.py script has a hard check on the hostname to check if we're using the docker created postgres container and only creates the databases if we do.

`if DATABASES[database_key]["HOST"] == "db":`

In https://github.com/centerofci/mathesar/pull/747, we gave our containers persistent names. The db container's name was set as `mathesar_db`.

This leads to all fresh setup of mathesar to fail.

## Expected behavior
Following the instructions in readme should work without any error.

## To Reproduce
<!-- How can we recreate this bug? Please try to provide a Minimal, Complete, and Verifiable (http://stackoverflow.com/help/mcve) example if code-related. -->
* Stop mathesar, if you have it running
* Remove your database
* Start mathesar and run migrations and install.py script 


Discussion:

Resolving patch:
diff --git a/install.py b/install.py
--- a/install.py
+++ b/install.py
@@ -18,7 +18,7 @@ def main():
 
 
 def install_on_db_with_key(database_key, skip_confirm):
-    if DATABASES[database_key]["HOST"] == "db":
+    if DATABASES[database_key]["HOST"] == "mathesar_db":
         # if we're going to install on the docker-created Postgres, we'll
         # create the DB
         print("Creating Mathesar DB on docker-created PostgreSQL instance")


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Polymorphic Serializer does not have access to the mapped Serializer
## Description
Polymorphic Serializer uses a map of Serializers to change the Serializer used based on the mapping field which depends on the data being serialized but custom validators within this mapped serializer do not have access to the fields as the class of the Serializer remains the same as the `PolymorphicSerializier` instead of the mapped serializer.

## Expected behavior
The mapped serializer should have access to the fields of the serializer when a custom validator is called.

## Additional context




Discussion:

Resolving patch:
diff --git a/mathesar/api/serializers/shared_serializers.py b/mathesar/api/serializers/shared_serializers.py
--- a/mathesar/api/serializers/shared_serializers.py
+++ b/mathesar/api/serializers/shared_serializers.py
@@ -23,6 +23,7 @@ def __new__(cls, *args, **kwargs):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
+        self.serializers_cls_mapping = {}
         serializers_mapping = self.serializers_mapping
         self.serializers_mapping = {}
         for identifier, serializer_cls in serializers_mapping.items():
@@ -32,10 +33,12 @@ def __init__(self, *args, **kwargs):
             else:
                 serializer = serializer_cls
             self.serializers_mapping[identifier] = serializer
+            self.serializers_cls_mapping[identifier] = serializer_cls
 
     def to_representation(self, instance):
         serializer = self.serializers_mapping.get(self.get_mapping_field(), None)
         if serializer is not None:
+            self.__class__ = self.serializers_cls_mapping.get(self.get_mapping_field())
             return serializer.to_representation(instance)
         else:
             raise Exception(f"Cannot find a matching serializer for the specified type {self.get_mapping_field()}")
@@ -52,6 +55,7 @@ class ReadWritePolymorphicSerializerMappingMixin(ReadOnlyPolymorphicSerializerMa
     def to_internal_value(self, data):
         serializer = self.serializers_mapping.get(self.get_mapping_field())
         if serializer is not None:
+            self.__class__ = self.serializers_cls_mapping.get(self.get_mapping_field())
             return serializer.to_internal_value(data=data)
         else:
             raise Exception(f"Cannot find a matching serializer for the specified type {self.get_mapping_field()}")


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
URI type does not allow NULL value to be set.
## Description
A column with the `URI` type does not allow `null` value to be set after creation even though the column `nullable` option is enabled

## Expected behaviour
Allow setting a null value to a nullable `URI` type column without throwing up an error 

## To Reproduce
1. Create a new column with the type `URI`.
2. Try setting a null value to the column by creating a new row and not entering any data.
3. New row does not get created due to an error with `URI` type not accepting a null value





Discussion:

Resolving patch:
diff --git a/db/types/uri.py b/db/types/uri.py
--- a/db/types/uri.py
+++ b/db/types/uri.py
@@ -92,8 +92,8 @@ def install(engine):
 
     create_domain_query = f"""
     CREATE DOMAIN {DB_TYPE} AS text CHECK (
-        {QualifiedURIFunction.SCHEME.value}(value) IS NOT NULL
-        AND {QualifiedURIFunction.PATH.value}(value) IS NOT NULL
+        (value IS NULL) OR ({QualifiedURIFunction.SCHEME.value}(value) IS NOT NULL
+        AND {QualifiedURIFunction.PATH.value}(value) IS NOT NULL)
     );
     """
 


Test patch:
diff --git a/db/tests/types/test_uri.py b/db/tests/types/test_uri.py
--- a/db/tests/types/test_uri.py
+++ b/db/tests/types/test_uri.py
@@ -160,6 +160,24 @@ def test_uri_type_column_creation(engine_email_type):
         test_table.create()
 
 
+test_data = ('https://centerofci.org', None)
+
+
+@pytest.mark.parametrize("data", test_data)
+def test_uri_type_set_data(engine_email_type, data):
+    engine, app_schema = engine_email_type
+    with engine.begin() as conn:
+        conn.execute(text(f"SET search_path={app_schema}"))
+        metadata = MetaData(bind=conn)
+        test_table = Table(
+            "test_table",
+            metadata,
+            Column("uris", uri.URI),
+        )
+        test_table.create()
+        conn.execute(test_table.insert(values=(data,)))
+
+
 def test_uri_type_column_reflection(engine_email_type):
     engine, app_schema = engine_email_type
     with engine.begin() as conn: