The Duke of Windsor, formerly King Edward VIII, was not invited either