Brown took a commission in the Royal Army Service Corps. He helped evacuate the British Army from Crete in 1941 for which he was awarded the MBE in the 1942 New Year Honours List. He played cricket with Lindsay Hassett in Cairo, but was captured with Bill Bowes at Tobruk in 1942 and spent the rest of the war in prisoner-of-war camps in Italy and Germany, where they organised games of cricket, baseball and rugby. He lost over 60lbs (30 kilos) before being liberated by the Americans in 1945.