Grey had an affair with the married Georgiana Cavendish, Duchess of Devonshire, before to his marriage. Grey met Cavendish during a Whig social meeting in Devonshire House, and they fell in love. However, their families did not approve of the match, as Grey was already known for being honorable and true to his marriage, and Cavendish was considered poor quality society stock. Nevertheless, Grey went ahead with the marriage anyway. They ran away together and were caught out on foot near a village called Clifton. They were discovered by some farmers who took them back home where a priest performed their wedding ceremony. After the incident, Grey returned to his family home in England while Cavendish stayed in London. She eventually managed to get permission from his parents to marry him. But after only six months of marriage, she died at the age of twenty-nine. There are those who say that Grey was responsible for her death by forcing her into a loveless marriage. He spent the rest of his life trying to find another woman with whom to replace her. But there was never anyone else like Georgiana Cavendish. He did marry again but it was a mere formality since he didn't love the woman. His first wife's sister, Louisa Howard, Lord Stafford, gave birth to two children. Then Grey divorced her so he could marry the younger sister, Harriet.