Frances Fisher was born the third of three children in Salisbury, North Carolina. Her parents were Colonel Charles Frederick Fisher and his wife, the former Elizabeth Clarissa Caldwell. Her mother had converted to Catholicism and raised Frances in that faith; her father was an Episcopalian. Colonel Fisher was killed in the beginning of the American Civil War at the battle of Manassas. Frances remained loyal to his Confederate ideals all her life. Her mother died while Frances was an infant, and she and her two siblings were orphaned. They were raised by a Catholic maiden aunt on her mother's side. The Fisher family was rich, but the family lost much of its money due to the war. The family lived at the northwest corner of Fulton and Innes streets in Salisbury, North Carolina. As a child of three or four, before she had learned to write, Frances liked to tell long, fanciful stories, which she persuaded her aunt to transcribe.