Repository: mathesar-foundation/mathesar

Issue:
AttributeError "attname" or "nspname" from various endpoints
I have been getting this API error intermittently when making a GET request to the records and columns endpoints. It always happens after making some DDL request, like importing a table, or renaming a columns, or extracting columns -- though I can't reproduce it 100% of the time.

<details>
<summary>(A) Sometimes I get this "nspname" error</summary>

```
Environment:


Request Method: GET
Request URL: http://localhost:8000/api/db/v0/tables/1894/records/?limit=500&offset=0

Django Version: 3.1.14
Python Version: 3.9.16
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'rest_framework',
 'django_filters',
 'django_property_filter',
 'mathesar']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'mathesar.middleware.CursorClosedHandlerMiddleware',
 'mathesar.middleware.PasswordChangeNeededMiddleware',
 'django_userforeignkey.middleware.UserForeignKeyMiddleware',
 'django_request_cache.middleware.RequestCacheMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 1167, in __getattr__
    return self._index[key]

The above exception ('nspname') was the direct cause of the following exception:
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 466, in handle_exception
    response = exception_handler(exc, context)
  File "/code/mathesar/exception_handlers.py", line 59, in mathesar_exception_handler
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/code/mathesar/api/db/viewsets/records.py", line 72, in list
    records = paginator.paginate_queryset(
  File "/code/mathesar/api/pagination.py", line 81, in paginate_queryset
    self.count = table.sa_num_records(filter=filters, search=search)
  File "/code/mathesar/models/base.py", line 471, in sa_num_records
    table=self._sa_table,
  File "/code/mathesar/state/cached_property.py", line 62, in __get__
    new_value = self.original_get_fn(instance)
  File "/code/mathesar/models/base.py", line 337, in _sa_table
    sa_table = reflect_table_from_oid(
  File "/code/db/tables/operations/select.py", line 31, in reflect_table_from_oid
    tables = reflect_tables_from_oids(
  File "/code/db/tables/operations/select.py", line 43, in reflect_tables_from_oids
    get_map_of_table_oid_to_schema_name_and_table_name(
  File "/code/db/tables/operations/select.py", line 74, in get_map_of_table_oid_to_schema_name_and_table_name
    select(pg_namespace.c.nspname, pg_class.c.relname, pg_class.c.oid)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 1169, in __getattr__
    util.raise_(AttributeError(key), replace_context=err)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception

Exception Type: AttributeError at /api/db/v0/tables/1894/records/
Exception Value: nspname
```

</details>

<details>
<summary>(B) Other times I get this "attname" error</summary>

```
Environment:


Request Method: GET
Request URL: http://localhost:8000/api/db/v0/tables/1824/records/?limit=500&offset=0

Django Version: 3.1.14
Python Version: 3.9.16
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'rest_framework',
 'django_filters',
 'django_property_filter',
 'mathesar']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'mathesar.middleware.CursorClosedHandlerMiddleware',
 'mathesar.middleware.PasswordChangeNeededMiddleware',
 'django_userforeignkey.middleware.UserForeignKeyMiddleware',
 'django_request_cache.middleware.RequestCacheMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 1167, in __getattr__
    return self._index[key]

The above exception ('attname') was the direct cause of the following exception:
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 466, in handle_exception
    response = exception_handler(exc, context)
  File "/code/mathesar/exception_handlers.py", line 59, in mathesar_exception_handler
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/code/mathesar/api/db/viewsets/records.py", line 55, in list
    column_names_to_ids = table.get_column_name_id_bidirectional_map()
  File "/code/mathesar/models/base.py", line 554, in get_column_name_id_bidirectional_map
    columns_map = bidict({column.name: column.id for column in columns})
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 287, in __iter__
    self._fetch_all()
  File "/code/mathesar/utils/prefetch.py", line 158, in _fetch_all
    prefetcher.fetch(obj_list, name, self.model, forwarders)
  File "/code/mathesar/utils/prefetch.py", line 270, in fetch
    related_data = self.filter(data_mapping.keys(), data_mapping.values())
  File "/code/mathesar/models/base.py", line 225, in filter
    return get_map_of_attnum_to_column_name(
  File "/code/db/columns/operations/select.py", line 91, in get_map_of_attnum_to_column_name
    triples_of_col_info = _get_triples_of_column_name_and_attnum_and_table_oid(
  File "/code/db/columns/operations/select.py", line 104, in _get_triples_of_column_name_and_attnum_and_table_oid
    statement = _statement_for_triples_of_column_name_and_attnum_and_table_oid(
  File "/code/db/columns/operations/select.py", line 180, in _statement_for_triples_of_column_name_and_attnum_and_table_oid
    sel = select(pg_attribute.c.attname, pg_attribute.c.attnum, pg_attribute.c.attrelid)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 1169, in __getattr__
    util.raise_(AttributeError(key), replace_context=err)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception

Exception Type: AttributeError at /api/db/v0/tables/1824/records/
Exception Value: attname
```

</details>

The exact same steps will sometimes produce (A), other times produce (B), and other times no error at all. I hit these errors a lot while working on #2140. I was doing a workflow where I'd import `books_sim.ts`, then extract the author's first and last names into a new table. I did it over and over again. I probably hit the error about 75% of the time during import and 60% of the time during column extraction.

https://user-images.githubusercontent.com/42411/208746906-86d06e41-18d0-45da-8d39-32aa8fddecba.mp4




Discussion:
@kgodey note that you might hit this during the demo video









I didn't run into this yet during pre-demo video testing. I think we can work around it for the video by reloading if needed.
I stumbled upon this error too, I will take a look at it once I finish off issues needed for the demo
I ran into this error pretty consistently while extracting/moving columns

Resolving patch:
diff --git a/db/utils.py b/db/utils.py
--- a/db/utils.py
+++ b/db/utils.py
@@ -96,7 +96,11 @@ def warning_ignored_func(*args, **kwargs):
 
 @ignore_type_warning
 def get_pg_catalog_table(table_name, engine, metadata):
-    return sqlalchemy.Table(table_name, metadata, autoload_with=engine, schema='pg_catalog')
+    table = sqlalchemy.Table(table_name, metadata, autoload_with=engine, schema='pg_catalog')
+    # Refresh metadata if it hasn't reflected correctly. Refer https://github.com/centerofci/mathesar/issues/2138
+    if len(table.c) < 1:
+        table = sqlalchemy.Table(table_name, metadata, autoload_with=engine, schema='pg_catalog', extend_existing=True)
+    return table
 
 
 def ignore_duplicate_wrapper(stmt):


Test patch: