Found out that the problem is that the Entity never gets removed, even though it's marked for deletion.
This is why the FPS slows down eventually.
Bakesale, please give me a way to remove the entity from the active list
OTL