Fix unavoidable NPE on removing Structure