Dooley attended Georgetown College (now Georgetown University) and became the first student to rank at the head of his class during each of his four years, graduating in 1860. Soon after, James and his brother John enlisted in the Confederate Army, joining their father's unit, the First Virginia Infantry. James Dooley was wounded at the Battle of Williamsburg during the Peninsula Campaign in May, 1862. He was captured and confined for a short time. Upon his release, Dooley worked in the Confederate Ordnance Department in Richmond. After the war, Dooley completed a Master of Arts degree at Georgetown, then returned to Richmond.