Duff was educated at Rugby and entered the army as an ensign in 1851. He served with the 23rd Royal Welsh Fusiliers in the Crimean War, including the Siege of Sebastopol, and was taken prisoner in the Battle of Inkerman. He retired from the army in 1858 as a major, having received the Crimea Medal with two clasps, and became a Justice of the Peace (JP) in Norfolk.