Only update world matrices when needed