Bourne's idea was a success. However, by 1850 rail was taking business from the roadways, and the Bourne family fortune declined. In 1821 the population was 133, by 1841 it was 411. The Irish famine took a toll from 1845 to 1851, and the population declined. Frederick Bourne left his land at Killegland, and his village of Ashbourne, to his son Richard in 1844. Richard lived in the village, and his first home is now the Ashbourne House Hotel on Frederick Street. Later he built a modest house on Castle Street near where the last remains of the Wafre and Segrave tower house were. Richard married the daughter of a wealthy local family, Elizabeth Mangan, and had several children. The eldest, Thomas, became his father's heir and the last landlord of Ashbourne. He left Ireland and went to live in Northfleet, Kent, England, in 1899. The land was sold to the local tenants.