Grimsley signed with the Yankees on January 26, 1999. With the Yankees, he rebounded, going 7–2 with a 3.60 ERA in 55 relief outings. Grimsley did not pitch in the 1999 American League Division Series or the 1999 American League Championship Series, but pitched innings of two-hit, two-walk ball in the 1999 World Series. In 2000, he was 3–2 with a 5.04 ERA. In the 2000 ALCS, he pitched one scoreless inning and earned his second World Series ring when the Yankees defeated the Mets in five games despite not pitching in the World Series. On November 20, he was released by the Yankees.