Don't call Layer#onRemove if Layer#onAdd was never called