Taking advantage of the Atlantic slave trade, the Philipses made use of African slaves to build various structures at both the Upper Mills and Lower Mills. The Upper Mills saw the building of two gristmills on the Pocantico River as well as a stone manor house, wharf, cooperage, and bake house.