On December 11, 2003, Brown was traded to the New York Yankees as part of a deal that sent Jeff Weaver, Yhency Brazobán, Brandon Weeden, and $2.6 million in cash to Los Angeles. In 2004, he posted a 10–6 record with a 4.09 ERA, but experienced health problems during the season. Toward the end of the season, he punched a wall in frustration, injuring his hand. He pitched well in the Division Series, but it was his performance in Game 7 of the 2004 American League Championship Series that he is remembered for, lasting less than two innings while giving up five earned runs, including a two-run homer to David Ortiz.