Drury commanded "Neptune" for the next two years, until his promotion to rear-admiral in 1804. He departed the ship on 13 May 1804, and the following day Captain Sir Thomas Williams took over. "Neptune" spent the rest of 1804 deployed with the Channel Fleet, blockading the French Atlantic ports. During this time Captain Williams' health progressively worsened, and he was invalided back to Britain on 7 May 1805. He was replaced by Captain Thomas Fremantle on 8 May, and was sent to join Robert Calder's force blockading Ferrol, after the Franco-Spanish fleet had arrived there after the Battle of Cape Finisterre. Calder decided that his eight ships were not sufficient to resist Villeneuve's fleet were it to come out of harbour, and instead went north to join Admiral William Cornwallis's fleet off Brest. Shortly afterwards Nelson's fleet returned from the West Indies, bringing 12 more ships, and Calder was given 18 ships, including "Neptune", and sent back to Ferrol to search for Villeneuve. By now Villeneuve had put into Cadiz and Calder's force was ordered to join the hastily assembled British fleet under Vice-Admiral Cuthbert Collingwood, that was blockading the Franco-Spanish fleet at Cadiz. As the British fleet settled in for a long blockade Fremantle commented on "Neptune"s sailing qualities. She had the reputation of being slow, and Fremantle complained that he did not like being in 'a large ship that don't sail and must continually be late in action.' During the battle however, Midshipman William Baddock commented that 'The old "Neptune", which never was a good sailer, took it into her head that morning to sail better than I ever remember to have seen her do before.' "Neptune " went into the battle 18 men short of her complement.