He sent a memo dated April 20, 1961, to Vice President Lyndon B