Fremantle had a lifelong devotion to the armed forces, and as well as commanding the Territorial Army he was also an Honorary Colonel of the Buckinghamshire battalion of the Oxfordshire Light Infantry. He was an expert shot using the rifle and published three books on the subject. He was also on the War Office Small Arms Committee. He shot in the English Eight for 27 years and was its Captain from 1920 until his death. He competed in the 1000 yard free rifle event at the 1908 Summer Olympics.