internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mathesar-foundation/mathesar

Issue:
Add a Docker image containing the Mathesar server along with a Postgres server
## Problem
The current docker image requires the user to already have a Postgres server running on their machine. This is a hassle for someone looking to try out Matheasar quickly.

## Proposed solution
Provide a docker image containing a Postgres server along with the Mathesar server





Discussion:

Resolving patch:
diff --git a/config/settings/common_settings.py b/config/settings/common_settings.py
--- a/config/settings/common_settings.py
+++ b/config/settings/common_settings.py
@@ -115,12 +115,12 @@ def pipe_delim(pipe_string):
 
 
 # SECURITY WARNING: keep the secret key used in production secret!
-SECRET_KEY = decouple_config('SECRET_KEY')
+SECRET_KEY = decouple_config('SECRET_KEY', default="2gr6ud88x=(p855_5nbj_+7^gw-iz&n7ldqv%94mjaecl+b9=4")
 
 # SECURITY WARNING: don't run with debug turned on in production!
 DEBUG = decouple_config('DEBUG', default=False, cast=bool)
 
-ALLOWED_HOSTS = decouple_config('ALLOWED_HOSTS', cast=Csv())
+ALLOWED_HOSTS = decouple_config('ALLOWED_HOSTS', cast=Csv(), default=".localhost, 127.0.0.1, [::1]")
 
 # Password validation
 # https://docs.djangoproject.com/en/3.1/ref/settings/#auth-password-validators
diff --git a/db/install.py b/db/install.py
--- a/db/install.py
+++ b/db/install.py
@@ -48,7 +48,7 @@ def _create_database(database_name, hostname, username, password, port, skip_con
         )
     if create_database.lower() in ["y", "yes"]:
         # We need to connect to an existing database inorder to create a new Database.
-        # So we use the default Database `postgres` that comes with postgres.
+        # So we use the default database `postgres` that comes with postgres.
         # TODO Throw correct error when the default postgres database does not exists(which is very rare but still possible)
         root_database = "postgres"
         root_db_engine = engine.create_future_engine(


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Implement frontend flow: User sharing an entity
I've not added a description, since I'll be implementing this.

Refer [Product spec](https://wiki.mathesar.org/en/product/specs/publicly-shareable-links) for detailed info.


Discussion:

Resolving patch:
diff --git a/mathesar/api/ui/permissions/shares.py b/mathesar/api/ui/permissions/shares.py
--- a/mathesar/api/ui/permissions/shares.py
+++ b/mathesar/api/ui/permissions/shares.py
@@ -16,7 +16,7 @@ class SharedTableAccessPolicy(AccessPolicy):
             'action': ['create', 'destroy', 'update', 'partial_update'],
             'principal': 'authenticated',
             'effect': 'allow',
-            'condition_expression': 'is_atleast_manager_nested_table_resource'
+            'condition_expression': 'is_atleast_editor_nested_table_resource'
         },
     ]
 


Test patch:
diff --git a/mathesar/tests/api/test_shared_tables.py b/mathesar/tests/api/test_shared_tables.py
--- a/mathesar/tests/api/test_shared_tables.py
+++ b/mathesar/tests/api/test_shared_tables.py
@@ -44,7 +44,7 @@ def shared_test_table(create_patents_table, uid):
     # client_name, is_allowed
     ('superuser_client_factory', True),
     ('db_manager_client_factory', True),
-    ('db_editor_client_factory', False),
+    ('db_editor_client_factory', True),
     ('schema_manager_client_factory', True),
     ('schema_viewer_client_factory', False),
     ('db_viewer_schema_manager_client_factory', True)
Repository: mathesar-foundation/mathesar

Issue:
Add OpenAPI Specification for /tables/ endpoint
## Problem
In order to ensure the accuracy of the specs generated by drf-spectacular for /tables/ endpoint , we will generate an OpenAPI Specification specifically for this endpoint. This will allow us to verify its correctness by comparing it with the existing test cases and the DRF browsable API page.

## Proposed solution
* Implement custom preprocessing hook function  to filter out all endpoints except for the /tables/ endpoint. The hook function selectively retains only the endpoint paths that match /tabels/ 
* Finally, generate the spec file using the `manage.py spectacular --color --file schema.yml` command


Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -2,7 +2,7 @@ def custom_preprocessing_hook(endpoints):
     filtered = []
     for (path, path_regex, method, callback) in endpoints:
         # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/"):
+        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/") or path.startswith("/api/db/v0/tables/"):
             filtered.append((path, path_regex, method, callback))
     return filtered
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Release 0.1.3
## 2023-08-16
```[tasklist]
### Tasks
- [x] Cut 0.1.3 release branch, freeze code
- [x] Update version number in all places in the new branch
- [x] Make an image from the branch with tag `0.1.3`, push to Dockerhub
- [x] Test installation with the new image
- [x] Test upgrade
- [x] Smoke testing application
- [x] Stability of the newly released items
```



Discussion:

Resolving patch:
diff --git a/mathesar/__init__.py b/mathesar/__init__.py
--- a/mathesar/__init__.py
+++ b/mathesar/__init__.py
@@ -1,3 +1,3 @@
 default_app_config = 'mathesar.apps.MathesarConfig'
 
-__version__ = "0.1.2"
+__version__ = "0.1.3"


Test patch:
diff --git a/db/sql/test_0_msar.sql b/db/sql/test_0_msar.sql
--- a/db/sql/test_0_msar.sql
+++ b/db/sql/test_0_msar.sql
@@ -1871,7 +1871,7 @@ END;
 $$ LANGUAGE plpgsql;
 
 
-CREATE OR REPLACE FUNCTION test_extract_columns() RETURNS SETOF TEXT AS $f$
+CREATE OR REPLACE FUNCTION test_extract_columns_data() RETURNS SETOF TEXT AS $f$
 BEGIN
   CREATE TABLE roster_snapshot AS SELECT * FROM "Roster" ORDER BY id;
   PERFORM msar.extract_columns_from_table('"Roster"'::regclass::oid, ARRAY[3, 4], 'Teachers', null);
@@ -1906,6 +1906,38 @@ END;
 $f$ LANGUAGE plpgsql;
 
 
+CREATE OR REPLACE FUNCTION setup_extract_fkey_cols() RETURNS SETOF TEXT AS $$
+BEGIN
+CREATE TABLE "Referent" (
+    id integer PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
+    "Teacher" text,
+    "Teacher Email" text
+);
+CREATE TABLE "Referrer" (
+    id integer PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
+    "Student Name" text,
+    "Subject" varchar(20),
+    "Grade" integer,
+    "Referent_id" integer REFERENCES "Referent" (id)
+);
+END;
+$$ LANGUAGE plpgsql;
+
+
+CREATE OR REPLACE FUNCTION test_extract_columns_keeps_fkey() RETURNS SETOF TEXT AS $f$
+BEGIN
+  PERFORM msar.extract_columns_from_table(
+    '"Referrer"'::regclass::oid, ARRAY[3, 5], 'Classes', 'Class'
+  );
+  RETURN NEXT columns_are('Referent', ARRAY['id', 'Teacher', 'Teacher Email']);
+  RETURN NEXT columns_are('Referrer', ARRAY['id', 'Student Name', 'Grade', 'Class']);
+  RETURN NEXT columns_are('Classes', ARRAY['id', 'Subject', 'Referent_id']);
+  RETURN NEXT fk_ok('Referrer', 'Class', 'Classes', 'id');
+  RETURN NEXT fk_ok('Classes', 'Referent_id', 'Referent', 'id');
+END;
+$f$ LANGUAGE plpgsql;
+
+
 CREATE OR REPLACE FUNCTION setup_dynamic_defaults() RETURNS SETOF TEXT AS $$
 BEGIN
   CREATE TABLE defaults_test (
Repository: mathesar-foundation/mathesar

Issue:
UI  for adding/modifying user database credentials in Mathesar
This issue is for tracking the front-end work needed for implementing the design in #3159. The APIs needed to communicate with the backend are covered by #3157.



Discussion:

Resolving patch:
diff --git a/mathesar/urls.py b/mathesar/urls.py
--- a/mathesar/urls.py
+++ b/mathesar/urls.py
@@ -53,6 +53,9 @@
     path('administration/users/', views.admin_home, name='admin_users_home'),
     path('administration/users/<user_id>/', views.admin_home, name='admin_users_edit'),
     path('administration/update/', views.admin_home, name='admin_update'),
+    path('administration/db-connection/', views.list_database_connection, name='list_database_connection'),
+    path('administration/db-connection/add/', views.add_database_connection, name='add_database_connection'),
+    path('administration/db-connection/edit/<db_name>/', views.edit_database_connection, name='edit_database_connection'),
     path('shares/tables/<slug>/', views.shared_table, name='shared_table'),
     path('shares/explorations/<slug>/', views.shared_query, name='shared_query'),
     path('db/', views.home, name='db_home'),
diff --git a/mathesar/views.py b/mathesar/views.py
--- a/mathesar/views.py
+++ b/mathesar/views.py
@@ -81,7 +81,12 @@ def get_database_list(request):
     failed_db_data = []
     for db in permission_restricted_failed_db_qs:
         failed_db_data.append({
+            'id': db.id,
+            'username': db.username,
+            'port': db.port,
+            'host': db.host,
             'name': db.name,
+            'db_name': db.db_name,
             'editable': db.editable,
             'error': 'Error connecting to the database'
         })
@@ -303,6 +308,28 @@ def schemas(request, db_name):
     })
 
 
+@login_required
+def list_database_connection(request):
+    return render(request, 'mathesar/index.html', {
+        'common_data': get_common_data(request)
+    })
+
+
+@login_required
+def add_database_connection(request):
+    return render(request, 'mathesar/index.html', {
+        'common_data': get_common_data(request)
+    })
+
+
+@login_required
+def edit_database_connection(request, db_name):
+    database = get_current_database(request, db_name)
+    return render(request, 'mathesar/index.html', {
+        'common_data': get_common_data(request, database, None)
+    })
+
+
 def shared_table(request, slug):
     shared_table_link = SharedTable.get_by_slug(slug) if is_valid_uuid_v4(slug) else None
     table = shared_table_link.table if shared_table_link else None


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add OpenAPI Specification for /queries/ endpoint and fix the incorrect specs
This is a part  of the API Documentation project #2888 
## Problem
To ensure the accuracy of the specs generated by drf-spectacular, we will generate an OpenAPI Specification specifically for a particular endpoint. This will allow us to verify its correctness by comparing it with the existing test cases and the DRF browsable API page. This issue aims to generate spec for /queries/ endpoint.

## Proposed solution
* Add /queries/ to the custom preprocessing hook function to filter out all endpoints except for the /queries/ endpoint. 
* Generate the spec file using the _./manage.py spectacular --color --file schema.yml_ command


Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -2,7 +2,7 @@ def custom_preprocessing_hook(endpoints):
     filtered = []
     for (path, path_regex, method, callback) in endpoints:
         # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/") or path.startswith("/api/db/v0/tables/"):
+        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/") or path.startswith("/api/db/v0/tables/") or path.startswith("/api/db/v0/links/") or path.startswith("/api/db/v0/queries/"):
             filtered.append((path, path_regex, method, callback))
     return filtered
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add openAPI Specification for UI related databases endpoint
Generate openAPI spec for `databases` endpoint corresponding to UI


Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -1,9 +1,14 @@
 def custom_preprocessing_hook(endpoints):
-    filtered = []
-    for (path, path_regex, method, callback) in endpoints:
-        # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/") or path.startswith("/api/db/v0/tables/"):
-            filtered.append((path, path_regex, method, callback))
+    prefixes = [
+        "/api/db/v0/databases/",
+        "/api/db/v0/data_files/",
+        "/api/db/v0/schemas/",
+        "/api/db/v0/tables/",
+        "/api/db/v0/links/",
+        "/api/db/v0/queries/",
+        "/api/ui/v0/databases/"
+    ]
+    filtered = [(path, path_regex, method, callback) for path, path_regex, method, callback in endpoints if any(path.startswith(prefix) for prefix in prefixes)]
     return filtered
 
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add openAPI Specification for /users/ endpoint
Generate spec for `users` endpoint


Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -1,9 +1,15 @@
 def custom_preprocessing_hook(endpoints):
-    filtered = []
-    for (path, path_regex, method, callback) in endpoints:
-        # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/") or path.startswith("/api/db/v0/tables/"):
-            filtered.append((path, path_regex, method, callback))
+    prefixes = [
+        "/api/db/v0/databases/",
+        "/api/db/v0/data_files/",
+        "/api/db/v0/schemas/",
+        "/api/db/v0/tables/",
+        "/api/db/v0/links/",
+        "/api/db/v0/queries/",
+        "/api/ui/v0/databases/",
+        "/api/ui/v0/users/"
+    ]
+    filtered = [(path, path_regex, method, callback) for path, path_regex, method, callback in endpoints if any(path.startswith(prefix) for prefix in prefixes)]
     return filtered
 
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Add openAPI Specification for /database_roles/ endpoint
Generate spec for `database_roles` endpoint


Discussion:

Resolving patch:
diff --git a/config/settings/openapi.py b/config/settings/openapi.py
--- a/config/settings/openapi.py
+++ b/config/settings/openapi.py
@@ -1,9 +1,16 @@
 def custom_preprocessing_hook(endpoints):
-    filtered = []
-    for (path, path_regex, method, callback) in endpoints:
-        # Remove all but DRF API endpoints
-        if path.startswith("/api/db/v0/databases/") or path.startswith("/api/db/v0/data_files/") or path.startswith("/api/db/v0/schemas/") or path.startswith("/api/db/v0/tables/"):
-            filtered.append((path, path_regex, method, callback))
+    prefixes = [
+        "/api/db/v0/databases/",
+        "/api/db/v0/data_files/",
+        "/api/db/v0/schemas/",
+        "/api/db/v0/tables/",
+        "/api/db/v0/links/",
+        "/api/db/v0/queries/",
+        "/api/ui/v0/databases/",
+        "/api/ui/v0/users/",
+        "/api/ui/v0/database_roles/"
+    ]
+    filtered = [(path, path_regex, method, callback) for path, path_regex, method, callback in endpoints if any(path.startswith(prefix) for prefix in prefixes)]
     return filtered
 
 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
Make the docker compose file self documenting
### It should be made clear in the docker compose file:
- The prerequisites required to run the file
- How to actually run the file
- The purpose for each of the environment variables used by different containers
- Their recommended/suggested/default values

### Goals:
- The need of referring to our docs for this installation method by the user should be eliminated upon resolving this issue.   
- The need of user to add variables in the `.env` file should be removed and instead these variables should be customizable in the compose file itself.

### Nice to haves:
- Make possible mounting folders as volumes to increase the visibility of the data being stored on user device. This is also useful if the data is being used by some other service on the host device. 
- Consider removing watchtower and push user to update manually instead.

### Additional context
<!-- Add any other context or screenshots about the feature request here.-->
Related to #3166 


Discussion:

Resolving patch:
diff --git a/config/settings/common_settings.py b/config/settings/common_settings.py
--- a/config/settings/common_settings.py
+++ b/config/settings/common_settings.py
@@ -94,10 +94,21 @@ def pipe_delim(pipe_string):
 # See pipe_delim above for why we use pipes as delimiters
 DATABASES = {
     db_key: db_url(url_string)
-    for db_key, url_string in decouple_config('MATHESAR_DATABASES', cast=Csv(pipe_delim))
+    for db_key, url_string in decouple_config('MATHESAR_DATABASES', default='', cast=Csv(pipe_delim))
 }
 
-DATABASES[decouple_config('DJANGO_DATABASE_KEY', default="default")] = decouple_config('DJANGO_DATABASE_URL', cast=db_url, default='sqlite:///db.sqlite3')
+# POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST & POSTGRES_PORT are required env variables for forming a pg connection string for the django database
+# lack of any one of these will result in the internal django database to be sqlite.
+POSTGRES_DB = decouple_config('POSTGRES_DB', default=None)
+POSTGRES_USER = decouple_config('POSTGRES_USER', default=None)
+POSTGRES_PASSWORD = decouple_config('POSTGRES_PASSWORD', default=None)
+POSTGRES_HOST = decouple_config('POSTGRES_HOST', default=None)
+POSTGRES_PORT = decouple_config('POSTGRES_PORT', default=None)
+
+if POSTGRES_DB and POSTGRES_USER and POSTGRES_PASSWORD and POSTGRES_HOST and POSTGRES_PORT:
+    DATABASES['default'] = db_url(f'postgres://{POSTGRES_USER}:{POSTGRES_PASSWORD}@{POSTGRES_HOST}:{POSTGRES_PORT}/{POSTGRES_DB}')
+else:
+    DATABASES['default'] = db_url('sqlite:///db.sqlite3')
 
 for db_key, db_dict in DATABASES.items():
     # Engine should be '.postgresql' or '.postgresql_psycopg2' for all db(s),
diff --git a/config/settings/production.py b/config/settings/production.py
--- a/config/settings/production.py
+++ b/config/settings/production.py
@@ -1,8 +1,8 @@
 from config.settings.common_settings import * # noqa
 
 # Override default settings
-
-
+DEBUG = False
+MATHESAR_MODE = 'PRODUCTION'
 # Use a local.py module for settings that shouldn't be version tracked
 try:
     from .local import * # noqa 


Test patch:
Repository: mathesar-foundation/mathesar

Issue:
SQLAlchemy filters still in codebase
## Description
<!-- A clear and concise description of what the bug is. -->

`sqlalchemy-filters` is required in parts of our codebase, but was removed from `requirements.txt`. 

## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->

Only one of the above should be true

## Additional context
<!-- Add any other context about the problem or screenshots here. -->

@kgodey crashed into this while doing the build from scratch docs.

Related issue: #3314 
Related PR: #3317 

Basically, that  PR removed  SQLAlchemy from `requirements.txt`, but not all spots of the codebase. The reason we didn't find this issue until now is that we do all testing on Docker, and the relevant `Dockerfile` includes an explicit instruction to install `sqlalchemy-filters` aside from the `requirements.txt`.



Discussion:

Resolving patch:
diff --git a/db/records/exceptions.py b/db/records/exceptions.py
--- a/db/records/exceptions.py
+++ b/db/records/exceptions.py
@@ -1,12 +1,8 @@
-from sqlalchemy_filters.exceptions import FieldNotFound
-
-
-# Grouping exceptions follow the sqlalchemy_filters exceptions patterns
 class BadGroupFormat(Exception):
     pass
 
 
-class GroupFieldNotFound(FieldNotFound):
+class GroupFieldNotFound(Exception):
     pass
 
 
diff --git a/db/records/operations/select.py b/db/records/operations/select.py
--- a/db/records/operations/select.py
+++ b/db/records/operations/select.py
@@ -42,10 +42,8 @@ def get_records(
                          'direction' field.
         search:          list of dictionaries, where each dictionary has a 'column' and
                          'literal' field.
-                         See: https://github.com/centerofci/sqlalchemy-filters#sort-format
         filter:          a dictionary with one key-value pair, where the key is the filter id and
                          the value is a list of parameters; supports composition/nesting.
-                         See: https://github.com/centerofci/sqlalchemy-filters#filters-format
         group_by:        group.GroupBy object
         duplicate_only:  list of column names; only rows that have duplicates across those rows
                          will be returned
diff --git a/mathesar/api/db/viewsets/records.py b/mathesar/api/db/viewsets/records.py
--- a/mathesar/api/db/viewsets/records.py
+++ b/mathesar/api/db/viewsets/records.py
@@ -40,8 +40,6 @@ def get_queryset(self):
 
     # For filter parameter formatting, see:
     # db/functions/operations/deserialize.py::get_db_function_from_ma_function_spec function doc>
-    # For sorting parameter formatting, see:
-    # https://github.com/centerofci/sqlalchemy-filters#sort-format
     def list(self, request, table_pk=None):
         paginator = TableLimitOffsetPagination()
 


Test patch: