Altrincham was denounced by public figures and slapped by a member of the public appalled by his comments