Check if we're on the remove list before joining

If we don't do this, we could add too small objects near the main component.