Don't remove swapped in loader elements

Fixes #122