In 1801, Brown left "Vanguard" and moved into , in which he served for one year in the Channel Fleet under Lord St Vincent. He then commanded the frigate HMS "Hussar" in the Cork Squadron. During the Peace of Amiens, Brown married Martha Vere Fothergill and the couple had four children. He then commanded HMS "Romney" in the Atlantic. Early in 1805, Brown was transferred to with the fleet under Sir Robert Calder. Calder led his force against the Franco-Spanish fleet of Pierre-Charles Villeneuve on 22 July 1805 at the Battle of Cape Finisterre. During the battle, which was fought in thick fog, Brown turned his ship away to inform his admiral the enemy was changing direction in the fog. Although opinion was and still is divided on where the fault lay for the failure to destroy Villeneuve at the battle, Calder's fleet did seriously damage their opponents and capture two ships. In Britain however there was anger that the victory was not more comprehensive and Calder demanded a court martial to clear his name. One of the captain he brought back to England from Cadiz with him was Brown, who left "Ajax" in the hands of Lieutenant John Pilfold.