Change:
fetchSubmodules = true; }; <del> patches = [ # fix build with openssl 3.0 (fetchurl { url = "https://salsa.debian.org/debian/transmission/-/raw/debian/3.00-2.1/debian/patches/openssl3-compat.patch"; hash = "sha256-v+SDTW/lCtc8B3TuhQB1pmjW/QRAGLtYncaImNNwpes="; }) ]; </del> outputs = [ "out" "apparmor" ]; cmakeFlags =

Discussion:
After upgrading to 22.11, I noticed that a host I run transmission on had steadily increasing memory usage. When looking at what runs on this host, I managed to narrow this issue down to transmission. ! The host was upgraded on 09.12. On 13.12. transmission was restarted when switching to a new configuration. I'm currently trying to see if building transmission (from unstable) without the patch introduced in , but using (currently only on /) instead, might fix this. Since this occurs in 22.11, but did not in 22.05, I'd assume it's not really an upstream issue. cc
I expect there's still some upstream issue, even if it's only triggered by some combination of versions (and config, etc.) but either way, getting more details will probably help with fixing it.
Or maybe that patch is buggy. I know neither their code nor openssl APIs, but e.g. on this place I'd expect additional at a very quick glance: (like it is a few lines below)
Maybe we should report that to debian then. So we have a bunch of options: the patch the patch and use the patch and use either kind of sucks, because , which is in transmission's propagates (aka ), so we'd end up with two different major versions of openssl instead of just two times openssl 3.x with different configs. although also isn't ideal, because it enables the legacy provider everywhere, instead of only where it's needed, like the patch does.
Debian should be notified of the issue instead of keeping the broken patch in circulation. would you like to take the time
I think at this point it's just a speculation that the patch is at fault. If it's the error I thought, it would probably leak with older openssl as well.
I certainly won't have time for this today, but probably I could have another deeper look within a week.
I've opted for option 3 for my setup as a quick fix in an overlay, which seems to work. If you have time in the next week or two, I think that's soon enough, because this issue has been in a ubuntu release for half a year, so apparently it doesn't affect most users too badly. Although I have seen issues reported upstream, which I'd attribute to this, if I had to guess.

Can try to find links, if anyone interested.
While the patch may be at fault, it's worth noting that there are a , that have been fixed subsequently. We're still running 3.00 in nixpkgs because there's been no non-beta release since -- the next release has been slow as a result of transmission being rewritten in C++. 4.0.0-beta.2 is available, perhaps it might be worth packaging this as a separate package or such?
While yes, the issue you linked does link to memory leak fixes after transmission 3.00, the bug report was created by someone on Ubuntu, after the release of 22.04, so it's likely that the really bad memory leak they're hitting is actually the one caused by the Ubuntu downstream patch. I've had this overlay (where unstable is , as part of , for reasons) : running for the last two weeks and transmission has been running fine again. Same as before 22.11, where we already were on transmission 3.00.
My fails with . So I don't know... perhaps the working variant with for now.
See for . Also, I left a comment asking the Ubuntu people if they might also want to look into this.
It's not only memory leaks (which lead to the process being killed by the oom_killer): I noticed at least two different crashes, something than has never happened before 3.0. I would seriously consider downgrading to 2.94 because this version is barely usable.
Uh, that's a surprise to me. We've been using "3.00" version in nixpkgs for years (in stable 20.09 already).
You're right, I implicitly assumed it was a new update but it isn't: the trouble started with NixOS 22.11.
also needed an update. See for that.