Repository: mathesar-foundation/mathesar

Issue:
Do not include unconfirmed tables for users with viewer & editor access
## Description
1. GET requests to `/api/db/v0/tables/`, `/api/db/v0/tables/<table_id>` and `common_data.tables` currently include unconfirmed tables for all users.
2. Request to `/api/db/v0/schemas/` and `/api/db/v0/schemas/<schema_id>/` have count for `num_tables` which contains count of all tables including ones that are not confirmed.

Confirmation can currently only be performed by a schema manager or a db manager. Users with viewer and editor access cannot should not be able to see tables that aren't confirmed yet.

The UI currently hides unconfirmed tables for case (1), but this should happen at an API level. The count issue mentioned in (2) however, is currently visible on the UI leading to inconsistent count being shown between DB page and schema page.

## Expected behavior
* Endpoints and common_data that contains tables should not include unconfirmed tables for users with viewer and editor access. i.e. Permission check should take `import_verified` into account. This includes:
  - Table list
  - Table get
  - common_data.tables
* `schema.num_tables` should also take this into account.


Discussion:
I updated the description since this needs a permission check at the API level and removed frontend tag.
@silentninja @Anish9901 I've removed the `good first issue` tag since I'm not sure about the work required. Please add it back if you think it should be simple.
Hello, @pavish! Could you please explain how to see if the tables are confirmed or not? What exactly do you mean by the unconfirmed table? I didn't get it.
@raunak-dev-edu "Unconfirmed" tables have `import_verified` set to False in the API/backend.
Hey guys! Is there a documented way for me to create users with specific roles like viewer, manager and editor right now? I am able to reproduce the issue with the import_verified boolean and I would like to test it with different user roles but I am not able to find a built-in functionality to do it, do I need to manually add rows to `mathesar_databaserole` and `mathesar_schemarole` ?
@DankMemes4President You can create new users using http://localhost:8000/administration/users/ and you can assign schema_roles and database_roles to appropriate users using http://localhost:8000/api/ui/v0/schema_roles/ and http://localhost:8000/api/ui/v0/database_roles/ 
Hi guys, can i be assigned to this bug ? If no one is currently working on it
> @DankMemes4President You can create new users using http://localhost:8000/administration/users/ and you can assign schema_roles and database_roles to appropriate users using http://localhost:8000/api/ui/v0/schema_roles/ and http://localhost:8000/api/ui/v0/database_roles/

Thanks for the reply, I am able to reproduce the issue now. I'd like to contribute to this issue if @Hatilar420 doesn't mind!
Assigned to you @DankMemes4President , Thanks. @Hatilar420 you are already working on 6 issues at the moment, I would suggest to hold on to getting any new issues assigned until you get reviews for the respective PRs.

Resolving patch:
diff --git a/mathesar/api/db/permissions/table.py b/mathesar/api/db/permissions/table.py
--- a/mathesar/api/db/permissions/table.py
+++ b/mathesar/api/db/permissions/table.py
@@ -49,13 +49,38 @@ class TableAccessPolicy(AccessPolicy):
     @classmethod
     def _scope_queryset(cls, request, qs, allowed_roles):
         if not (request.user.is_superuser or request.user.is_anonymous):
-            permissible_database_role_filter = (
+            db_role_access_filter = (
                 Q(schema__database__database_role__role__in=allowed_roles)
                 & Q(schema__database__database_role__user=request.user)
             )
+            schema_role_access_filter = (
+                Q(schema__schema_role__role__in=allowed_roles)
+                & Q(schema__schema_role__user=request.user)
+            )
+
+            # Filters to check whether user is a db/schema manager
+            is_database_manager = (
+                Q(schema__database__database_role__role='manager')
+                & Q(schema__database__database_role__user=request.user)
+            )
+            is_schema_manager = (
+                Q(schema__schema_role__role='manager')
+                & Q(schema__schema_role__user=request.user)
+            )
+
+            # Filter for confirmed tables
+            cnf_table_filter = (Q(import_verified=True) | Q(import_verified__isnull=True))
+
+            # Filters for the purpose of including/removing unconfirmed tables based on user's role
+            permissible_database_role_filter = (
+                is_database_manager & Q(import_verified=False)
+                | cnf_table_filter & db_role_access_filter
+            )
             permissible_schema_roles_filter = (
-                Q(schema__schema_role__role__in=allowed_roles) & Q(schema__schema_role__user=request.user)
+                is_schema_manager & Q(import_verified=False)
+                | cnf_table_filter & schema_role_access_filter
             )
+
             qs = qs.filter(permissible_database_role_filter | permissible_schema_roles_filter)
         return qs
 
diff --git a/mathesar/api/db/viewsets/schemas.py b/mathesar/api/db/viewsets/schemas.py
--- a/mathesar/api/db/viewsets/schemas.py
+++ b/mathesar/api/db/viewsets/schemas.py
@@ -35,7 +35,7 @@ def create(self, request):
             database_name,
             comment=serializer.validated_data.get('description')
         )
-        serializer = SchemaSerializer(schema)
+        serializer = SchemaSerializer(schema, context={'request': request})
         return Response(serializer.data, status=status.HTTP_201_CREATED)
 
     def partial_update(self, request, pk=None):
diff --git a/mathesar/api/serializers/schemas.py b/mathesar/api/serializers/schemas.py
--- a/mathesar/api/serializers/schemas.py
+++ b/mathesar/api/serializers/schemas.py
@@ -1,9 +1,11 @@
 from rest_access_policy import PermittedSlugRelatedField
 from rest_framework import serializers
 
+from mathesar.api.db.permissions.table import TableAccessPolicy
+
 from mathesar.api.db.permissions.database import DatabaseAccessPolicy
 from mathesar.api.exceptions.mixins import MathesarErrorMessageMixin
-from mathesar.models.base import Database, Schema
+from mathesar.models.base import Database, Schema, Table
 
 
 class SchemaSerializer(MathesarErrorMessageMixin, serializers.HyperlinkedModelSerializer):
@@ -30,7 +32,9 @@ class Meta:
         ]
 
     def get_num_tables(self, obj):
-        return obj.tables.count()
+        qs = Table.objects.filter(schema=obj)
+        count = TableAccessPolicy.scope_queryset(self.context['request'], qs).count()
+        return count
 
     def get_num_queries(self, obj):
         return sum(t.queries.count() for t in obj.tables.all())


Test patch:
diff --git a/mathesar/tests/api/test_schema_api.py b/mathesar/tests/api/test_schema_api.py
--- a/mathesar/tests/api/test_schema_api.py
+++ b/mathesar/tests/api/test_schema_api.py
@@ -123,6 +123,40 @@ def test_schema_list_filter(client, create_db_schema, FUN_create_dj_db, MOD_engi
             )
 
 
+list_clients_with_num_tables_count = [
+    ('superuser_client_factory', 3),
+    ('db_manager_client_factory', 3),
+    ('db_editor_client_factory', 2),
+    ('schema_manager_client_factory', 3),
+    ('schema_viewer_client_factory', 2),
+    ('db_viewer_schema_manager_client_factory', 3)
+]
+
+
+@pytest.mark.parametrize('client_name, expected_num_tables_count', list_clients_with_num_tables_count)
+def test_schema_num_tables_with_multiple_roles(
+        request,
+        create_patents_table,
+        patent_schema,
+        client_name,
+        expected_num_tables_count
+):
+    create_patents_table("Patent Table 1")
+    table_2 = create_patents_table("Patent Table 2")
+    table_3 = create_patents_table("Patent Table 3")
+    table_2.import_verified = True
+    table_3.import_verified = None
+    table_2.save()
+    table_3.save()
+    client = request.getfixturevalue(client_name)(patent_schema)
+    response = client.get('/api/db/v0/schemas/')
+    assert response.status_code == 200
+    response_data = response.json()
+    response_schemas = [s for s in response_data['results'] if s['name'] != 'public']
+    for someschema in response_schemas:
+        assert someschema['num_tables'] == expected_num_tables_count
+
+
 def test_schema_detail(create_patents_table, client, test_db_name, MOD_engine_cache):
     """
     Desired format:
diff --git a/mathesar/tests/api/test_table_api.py b/mathesar/tests/api/test_table_api.py
--- a/mathesar/tests/api/test_table_api.py
+++ b/mathesar/tests/api/test_table_api.py
@@ -161,12 +161,12 @@ def check_create_table_response(
 
 
 list_clients_with_results_count = [
-    ('superuser_client_factory', 3),
-    ('db_manager_client_factory', 3),
-    ('db_editor_client_factory', 3),
-    ('schema_manager_client_factory', 2),
+    ('superuser_client_factory', 5),
+    ('db_manager_client_factory', 5),
+    ('db_editor_client_factory', 2),
+    ('schema_manager_client_factory', 4),
     ('schema_viewer_client_factory', 2),
-    ('db_viewer_schema_manager_client_factory', 3)
+    ('db_viewer_schema_manager_client_factory', 4)
 ]
 
 write_clients_with_status_code = [
@@ -233,8 +233,8 @@ def test_table_list(create_patents_table, client):
 
 @pytest.mark.parametrize('client_name,expected_table_count', list_clients_with_results_count)
 def test_table_list_based_on_permissions(
-        create_patents_table, patent_schema,
-        create_table,
+        create_patents_table,
+        patent_schema,
         request,
         client_name,
         expected_table_count
@@ -242,6 +242,13 @@ def test_table_list_based_on_permissions(
     create_patents_table('Private Table', schema_name='Private Schema')
     create_patents_table("Patent Table 1")
     create_patents_table("Patent Table 2")
+    table3 = create_patents_table("Patent Table 3")
+    table4 = create_patents_table("Patent Table 4")
+    table3.import_verified = True
+    table4.import_verified = None
+    table3.save()
+    table4.save()
+
     client = request.getfixturevalue(client_name)(patent_schema)
 
     response = client.get('/api/db/v0/tables/')
@@ -814,6 +821,11 @@ def test_table_partial_update_by_different_roles(create_patents_table, request,
     table_name = 'NASA Table Partial Update'
     new_table_name = 'NASA Table Partial Update New'
     table = create_patents_table(table_name)
+
+    # Editors and Viewers only have access to confirmed tables
+    table.import_verified = True
+    table.save()
+
     client = request.getfixturevalue(client_name)(table.schema)
     expect_comment = 'a super new test comment'
     body = {'name': new_table_name, 'description': expect_comment}
@@ -856,6 +868,13 @@ def test_table_delete_by_different_roles(
     different_schema_table = create_patents_table('Private Table', schema_name='Private Schema')
     table_name = 'NASA Table Delete'
     table = create_patents_table(table_name)
+
+    # Editors and Viewers only have access to confirmed tables
+    different_schema_table.import_verified = True
+    table.import_verified = True
+    different_schema_table.save()
+    table.save()
+
     client = request.getfixturevalue(client_name)(table.schema)
     response = client.delete(f'/api/db/v0/tables/{table.id}/')
     assert response.status_code == expected_status_code
@@ -1692,6 +1711,11 @@ def test_table_move_columns_after_extracting(create_patents_table, client):
 def test_table_extract_columns_by_different_roles(create_patents_table, request, client_name, expected_status_code):
     table_name = 'Patents'
     table = create_patents_table(table_name)
+
+    # Editors and Viewers only have access to confirmed tables
+    table.import_verified = True
+    table.save()
+
     column_name_id_map = table.get_column_name_id_bidirectional_map()
     column_names_to_extract = ['Patent Number', 'Title', 'Patent Expiration Date']
     column_ids_to_extract = [column_name_id_map[name] for name in column_names_to_extract]