Ward's family lived on the edge of the frontier during his childhood. Before James was born, his father moved the family from eastern Augusta County to the area now known as Greenbrier County, West Virginia. After Braddock's Defeat in 1755 that area became too hostile so James Ward Sr. moved his family closer toward Staunton while he joined several expeditions during the French and Indian War.