Change:
Directly related to this discussion: paperless-ngx runs a hourly celery beat task. This is supposed to take a few minutes on most systems, but on NixOS (for some users, at least) the task runs forever and is eventually killed by the celery timeout. The affected worker process doesn't respond to SIGTERM or even SIGQUIT; only SIGKILL can interrupt it. Steps to reproduce the behavior: up paperless-ngx and add some documents to it a while(?) or manually trigger the trainclassifier task times out after 30 minutes, probably. The classifier training task should finish successfully in a few minutes or less. See for more background. This appears to be specific to paperless-ngx on NixOS. Please run and paste the result. system: host os: multi-user?: sandbox: version: channels(benley): channels(root): nixpkgs:
a tiny bit more context: The classifier task gets as far as printing these messages in and then it gets stuck. That last message comes from this spot in the code:
means it's not even getting stuck on the first call to , as the tags classifier finishes. It's the second one.
I'm seeing the same behavior. The only difference is I am getting stuck at the tags classifier. It hangs forever here, pegging a single core.
It appears it can get stuck on any classification task. A temporary workaround is to disable classification for some of the lesser used tags/correspondents/doctypes until it works again.
The likely cause for this bug has been found: OpenBLAS. Building numpy with i.e. the proprietary mkl BLAS implementation instead resolves this issue. I'm not sure how to start approaching upstream in this as OpenBLAS is like 3 libraries down the dependency tree.
The nixpkgs manual suggests using LDLIBRARYPATH to select a different BLAS implementation at runtime:
So far this seems to actually work! I wasn't able to use the library due to missing symbols, but worked: (that will be in nixos-unstable; I am running 23.11)
Another thing we could perhaps try somehow is to prevent BLAS from loading somehow because the upstream wheels for numpy somehow don't include any BLAS implementation at all and therefore neither does the paperless-ngx docker image.
How well is this working now? I might be experiencing the same issue.
I'm still running this config and the problem still has not returned in my setup, even after adding some new auto classifiers.
Should we add this to the paperless module?
FWIW, I've experienced the same problem and upon finding this issue also set to use MKL, which made the problem go away for me, too. With MKL being non-free I'm not 100% happy with it, though. Are there any chances this might also be solved by employing a newer OpenBLAS version? I lack the time to dig deeper into this, unfortunately.
OpenBLAS is on the newest release.
Can confirm that adding MKL fixed the problem I was having. "Classifier file does not exist"
I'm using nixos-23.11 where OpenBLAS is at 0.3.24. But if the problem persists with 0.3.26 from nixos-unstable, just waiting for the update to propagate to the next release obviously won't be the solution, unfortunately.
Instead of using MKL I've tried the following, and it seems to work for me, too: (It's probably sufficient to set one of them, but I wanted to be sure the setting takes effect no matter what.) Can someone confirm this makes the problem go away with OpenBLAS? Or was it just fluke for me?
The culprit is . Setting it to 1 works around the issue.
Aha: "OpenBLAS ignores and when compiled with ." And in NixOS, OpenBLAS is configured to set on many platforms. So that's why is the one to set. But I've also found this: So maybe forcing to use OpenBLAS with might be the proper solution here?
Paperless only uses OpenBLAS via scikit-learn via numpy. This is a generic library and paperless is not the exclusive user of any of these. The proper solution is to figure out what causes OpenBLAS to spin on when multiple OMP threads are used and the reason might be anywhere in the stack. The next step is likely to find a more minimal reproducer as low in this stack as possible and go bother the relevant upstream with it. At least FreeBSD appears to run into the same issue, so I don't think it's an obvious packaging issue on our end. Upstream wheels of numpy do not appear to use OpenBLAS at all, perhaps we could also look into whether our numpy using OpenBLAS is necessary, supported and desirable.
Proposed as a workaround until a proper solution is found.

Discussion:
name = "paperless_ngx_nltk_data"; paths = pkg.nltkData; }; <ins> } // optionalAttrs (cfg.openMPThreadingWorkaround) { OMP_NUM_THREADS = "1"; </ins> } // (lib.mapAttrs (_: s: if (lib.isAttrs s || lib.isList s) then builtins.toJSON s else if lib.isBool s then lib.boolToString s

}; package = mkPackageOption pkgs "paperless-ngx" { }; <ins> openMPThreadingWorkaround = mkEnableOption '' a workaround for document classifier timeouts. Paperless uses OpenBLAS via scikit-learn for document classification. The default is to use threading for OpenMP but this would cause the document classifier to spin on one core seemingly indefinitely if there are large amounts of classes per classification; causing it to effectively never complete due to running into timeouts. This sets `OMP_NUM_THREADS` to `1` in order to mitigate the issue. See https://github.com/NixOS/nixpkgs/issues/240591 for more information. '' // mkOption { default = true; }; </ins> }; config = mkIf cfg.enable {