Born in Cardiff, James was raised in the Splott area of the city, living in Carlisle Street. He joined his hometown club Cardiff City in 1939 at the start of the Second World War. A prolific scorer for the club during wartime fixtures, he was regarded as an exciting prospect in Welsh football and was chosen to represent Wales in two wartime fixtures against England, scoring in the first match where he played against Stan Cullis. At the age of 18, James enlisted in the army, joining the 77th HAA Regiment, Royal Artillery. The unit contained a number of footballers including other Cardiff City players Ernie Curtis and Billy Baker and embarked on a journey to North Africa by ship for deployment. However, en route, Japanese forces launched attacks on Hong Kong, Singapore and Malaya leading the UK to declare war against Japan which saw James and his unit rerouted to the Dutch colony of Java in the Dutch East Indies where British forces suffered heavy losses and James was one of numerous British troops taken prisoner.