Delete files in the manifest instead of removing the loadpath

This should close #1405.