Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Internal Server Error on loading suggestions
### Description

When opening a document the browser freezes for several seconds. On some occasions I get the following error message

```paperless Error retrieving suggestions: {"headers":{"normalizedNames":{},"lazyUpdate":null},"status":500,"statusText":"Internal Server Error","url":"http://192.168.1.102:8000/api/documents/1422/suggestions/","ok":false,"name":"HttpErrorResponse","message":"Http failure response for http://192.168.1.102:8000/api/documents/1422/suggestions/: 500 Internal Server Error","error":"\n<!doctype html>\n<html lang=\"en\">\n<head>\n <title>Server Error (500)</title>\n</head>\n<body>\n <h1>Server Error (500)</h1><p></p>\n</body>\n</html>\n"}```

This is also reflected when looking at the browser console which shows a response time for `http://192.168.1.102:8000/api/documents/1422/suggestions/` of about 12 seconds. The browser is unresponsive until this completes.

Paperless-ngx is run on an Unraid system inside their docker templating system. I understand that this is not the typical use case but maybe the error is enough to point me into the right direction. 

If I could switch off the suggestions API I guess the response times would also improve - this would be ok for me as a workaround.

### Steps to reproduce

1. Open any document

### Webserver logs

The logs from Paperless-ngx show nothing suspicious.

```bash
[2023-05-22 11:29:17,661] [DEBUG] [paperless.matching] Correspondent Deutsche Rentenversicherung matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains all of these words: Deutsche Rentenversicherung

[2023-05-22 11:29:17,666] [DEBUG] [paperless.matching] Correspondent Company matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains this word: Company

[2023-05-22 11:29:17,680] [DEBUG] [paperless.matching] Tag Person matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains all of these words: Person

[2023-05-22 11:29:17,682] [DEBUG] [paperless.matching] Tag Versicherung matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains this word: Versicherung

[2023-05-22 11:29:17,736] [DEBUG] [paperless.matching] Correspondent Deutsche Rentenversicherung matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains all of these words: Deutsche Rentenversicherung

[2023-05-22 11:29:17,741] [DEBUG] [paperless.matching] Correspondent Company matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains this word: Company

[2023-05-22 11:29:17,755] [DEBUG] [paperless.matching] Tag Person matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains all of these words: Person

[2023-05-22 11:29:17,756] [DEBUG] [paperless.matching] Tag Versicherung matched on document 2022-02-01 Deutsche Rentenversicherung Brother_ADS_1700W_20190213_201708_001470 because it contains this word: Versicherung```
```

(some words changed for privacy reasons)

### Browser logs

_No response_

### Paperless-ngx version

1.14.5

### Host OS

Unraid

### Installation method

Other (please describe above)

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
An internal server error is definitely going to have something logged.  Maybe check in the log file
I'm not sure where to find the log file you are talking about. Checking the docker logs there seems nothing suspicious going on:

```[2023-05-22 16:50:00,111] [INFO] [celery.beat] Scheduler: Sending due task Check all e-mail accounts (paperless_mail.tasks.process_mail_accounts)
[2023-05-22 16:50:00,115] [INFO] [celery.worker.strategy] Task paperless_mail.tasks.process_mail_accounts[49b98d8f-a65f-4fa4-aaa7-945dbe785026] received
[2023-05-22 16:50:00,165] [INFO] [celery.app.trace] Task paperless_mail.tasks.process_mail_accounts[49b98d8f-a65f-4fa4-aaa7-945dbe785026] succeeded in 0.04655179078690708s: 'No new documents were added.'
```

paperless.log which is accessible from the UI only returns what I already wrote in the original description.

The docs are a bit sparse on logging is there any way to expose more logging?
You can enable debug mode, but I doubt it would shed more light.  An internal error without any log seems more like a Django problem.  They'd be the ones sending the response in this case.

You could try to re-build the classification, perhaps the saved file has become corrupted.
I suspect the frontend is now surfacing errors that may or may not be of actual consequence. If you look at the network call to `.../suggestions/` in your browser (or just load it directly) what is the actual response?

https://github.com/paperless-ngx/paperless-ngx/discussions/3372
https://github.com/paperless-ngx/paperless-ngx/discussions/3428
https://github.com/paperless-ngx/paperless-ngx/issues/3392

(to clarify what I mean, Im wondering if loading the endpoint consistently reproduces the issue or if its gone)
I started having same issue as well. When you try to edit a file to make changes, that's when happens mostly from sorting queue. But again, it is random. I have two instances on different VM and both started recently. 
@shamoon 
The endpoint response is looking fine to me:

Header:
```
HTTP/1.1 200 OK
date: Tue, 23 May 2023 12:39:32 GMT
server: uvicorn
content-type: application/json
vary: Accept, Accept-Language, Origin, Cookie
allow: GET, HEAD, OPTIONS
x-frame-options: SAMEORIGIN
x-api-version: 3
x-version: 1.14.5
content-length: 119
content-language: de-de
x-content-type-options: nosniff
referrer-policy: same-origin
cross-origin-opener-policy: same-origin
```

Content:
```
{
	"correspondents":[1,41],
	"tags":[22,28],
	"document_types":[24,1],
	"storage_paths":[],
	"dates":["2023-04-17","2023-04-24"]
}
```

The response just takes a really long time. I'm wondering why it's blocking the browser. Can the suggestions feature be switched off somehow?

The issues that you linked look similar to what I am seeing.
The speed will depend on a few things including your system and the pdf itself, searching the document for dates in particular can be a bit taxing.

You can try lowering [`PAPERLESS_NUMBER_OF_SUGGESTED_DATES`](https://docs.paperless-ngx.com/configuration/) to see if that helps.

Again, overall Im not sure how actionable this is at the moment. I've never been able to reproduce this error message, it does seem to be intermittent but also kind of inconsequential in the end. The next release will make it easier to see the full output of the error and if we can get a bit more detail perhaps we can narrow it down.
I've seen the same for a few times. It happens when I scan a document, I'm on the dashboard, waiting for the processing to be finished. Once it's done (progress 100%), I click on the link to go to the new document's page. The page loads as usual, but there's this red warning with error `500`.  
I've tried to reproduce it, but not always happens. 

OP mentioned long response time - in my case it loads fast.  
Also I use Docker + Firefox.

Stacktrace:

```
[2023-05-23 20:22:57,825] [ERROR] [django.request] Internal Server Error: /api/documents/407/suggestions/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 486, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = await get_response(request)
  File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 448, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/local/lib/python3.9/site-packages/asgiref/current_thread_executor.py", line 22, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 490, in thread_handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/src/paperless/src/documents/views.py", line 433, in suggestions
    c.id for c in match_correspondents(doc, classifier, request.user)
  File "/usr/src/paperless/src/documents/matching.py", line 23, in match_correspondents
    pred_id = classifier.predict_correspondent(document.content) if classifier else None
  File "/usr/src/paperless/src/documents/classifier.py", line 363, in predict_correspondent
    X = self.data_vectorizer.transform([self.preprocess_content(content)])
  File "/usr/src/paperless/src/documents/classifier.py", line 348, in preprocess_content
    self._stop_words = set(stopwords.words(settings.NLTK_LANGUAGE))
  File "/usr/local/lib/python3.9/site-packages/nltk/corpus/util.py", line 121, in __getattr__
    self.__load()
  File "/usr/local/lib/python3.9/site-packages/nltk/corpus/util.py", line 95, in __load
    args, kwargs = self.__args, self.__kwargs
AttributeError: 'WordListCorpusReader' object has no attribute '_LazyCorpusLoader__args'
```
Thanks for the stack trace. That’s the same as one of the [other threads](https://github.com/paperless-ngx/paperless-ngx/issues/3392) though in looking into it I remember it was a sort of inexplicable error. It might be worth wrapping the nltk logic with a `try` and just logging an error rather than flub the whole response

Resolving patch:
diff --git a/src/documents/classifier.py b/src/documents/classifier.py
--- a/src/documents/classifier.py
+++ b/src/documents/classifier.py
@@ -341,20 +341,42 @@ def preprocess_content(self, content: str) -> str:  # pragma: nocover
             # set the search path for NLTK data to the single location it should be in
             nltk.data.path = [settings.NLTK_DIR]
 
-            # Do some one time setup
-            if self._stemmer is None:
-                self._stemmer = SnowballStemmer(settings.NLTK_LANGUAGE)
-            if self._stop_words is None:
-                self._stop_words = set(stopwords.words(settings.NLTK_LANGUAGE))
-
-            # Tokenize
-            words: List[str] = word_tokenize(content, language=settings.NLTK_LANGUAGE)
-            # Remove stop words
-            meaningful_words = [w for w in words if w not in self._stop_words]
-            # Stem words
-            meaningful_words = [self._stemmer.stem(w) for w in meaningful_words]
-
-            return " ".join(meaningful_words)
+            try:
+                # Preload the corpus early, to force the lazy loader to transform
+                stopwords.ensure_loaded()
+
+                # Do some one time setup
+                # Sometimes, somehow, there's multiple threads loading the corpus
+                # and it's not thread safe, raising an AttributeError
+                if self._stemmer is None:
+                    self._stemmer = SnowballStemmer(settings.NLTK_LANGUAGE)
+                if self._stop_words is None:
+                    self._stop_words = set(stopwords.words(settings.NLTK_LANGUAGE))
+
+                # Tokenize
+                # This splits the content into tokens, roughly words
+                words: List[str] = word_tokenize(
+                    content,
+                    language=settings.NLTK_LANGUAGE,
+                )
+
+                meaningful_words = []
+                for word in words:
+                    # Skip stop words
+                    # These are words like "a", "and", "the" which add little meaning
+                    if word in self._stop_words:
+                        continue
+                    # Stem the words
+                    # This reduces the words to their stems.
+                    # "amazement" returns "amaz"
+                    # "amaze" returns "amaz
+                    # "amazed" returns "amaz"
+                    meaningful_words.append(self._stemmer.stem(word))
+
+                return " ".join(meaningful_words)
+
+            except AttributeError:
+                return content
 
         return content
 
diff --git a/src/paperless/settings.py b/src/paperless/settings.py
--- a/src/paperless/settings.py
+++ b/src/paperless/settings.py
@@ -921,6 +921,10 @@ def _get_nltk_language_setting(ocr_lang: str) -> Optional[str]:
     languages for all the NLTK data used.
 
     Assumption: The primary language is first
+
+    NLTK Languages:
+      - https://www.nltk.org/api/nltk.stem.snowball.html#nltk.stem.snowball.SnowballStemmer
+
     """
     ocr_lang = ocr_lang.split("+")[0]
     iso_code_to_nltk = {


Test patch: