During the Battle of Hong Kong his home in the city was destroyed, forcing him to live at the consulate general building. On December 25, 1941, the Japanese took Hong Kong and imprisoned him. On March 12, it was reported that Southard and his staff were in two houses in a select residential district of Hong Kong and were later moved to the Stanley district. After six months of imprisonment he was released as part of an exchange agreement alongside sixty three other diplomats and was taken to Maputo on board the Asama Maru before returning to the United States on August 25, 1942.