St. James of My Lady's Manor, 1750-1950