Here’s what happened: On June 18, 1981, 66-year-old Justice Potter Stewart announced that he was retiring after 23 years on the bench.