The letter had little impact on the Labour vote—which held up