Milne married his wife Françoise in 1993 and the couple have three children together. Originally from France, she applied for dual-nationality following the 2016 EU referendum but her application was rejected, leading the couple to publicly criticise the uncertainty created by Brexit.