On 13 April 1596 the Queen specifically instructed Robert Devereux, 2nd Earl of Essex, not to take either Southampton or the Earl of Derby, with him on an expedition for the Relief of Calais, and it also appears that Southampton did not accompany Essex on the Cadiz Expedition that summer. In February 1597 Southampton challenged the Earl of Northumberland to a duel with rapiers, requiring intervention by the Queen and Privy Council, and on 1 March stood godfather at the christening of Sir Robert Sidney's daughter, Bridget. Later that year Southampton was with Essex on his "inglorious" Voyage to the Azores, where according to Rowland Whyte, "My Lord of Southampton fought with one of the Kings great Men of Warre, and suncke her". On his return he made his first appearance in the House of Lords on 5 November, and was put on several committees, but became a "chronic absentee". By this time he was in serious financial difficulties, and had turned over administration of his estates to two trustees, who by the end of the year had sold some of his lands.