Actually remove invisible stars

Before we were just removing the bullets twice :P