Andros was born in London on 6 December 1637. Amice Andros, his father, was Bailiff of Guernsey and a staunch supporter of Charles I. His mother was Elizabeth Stone, whose sister was a courtier to the king's sister, Queen Elizabeth of Bohemia. Although it has been claimed that Andros was present at the surrender in 1651 of Guernsey's Castle Cornet, the last royalist stronghold to surrender in the English Civil War, there is no firm evidence to support this. It is possible that he fled Guernsey with his mother in 1645. In 1656, he was apprenticed to his uncle, Sir Robert Stone, captain of a cavalry company. Andros then served in two winter campaigns in Denmark, including the relief of Copenhagen in 1659. As a result of these experiences he gained fluency in French, Swedish, and Dutch.