On 22 February 1942, upon his graduation from the Royal Military College, Sharma was commissioned into the 8th Battalion, 19th Hyderabad Regiment, of the British Indian Army (later to become the Indian Army's 4th Battalion, Kumaon Regiment). During World War II, he saw action against the Japanese in Burma during the Arakan Campaign. At that time he served under the command of Colonel K. S. Thimmayya, who would later rise to the rank of general and become Chief of the Army Staff from 1957 to 1961. Sharma was mentioned in despatches for his actions during the fighting of the Arakan Campaign.