Weiß joined the Reichswehr in 1931, and was commissioned as a lieutenant in 1934. In November 1938, he was assigned to the Army High Command's personnel department, and promoted to captain during 1940. In 1941, he was transferred to the 1st Panzer Division as an adjutant. From April 1942, he served in the General Army Office ("Allgemeines Heeresamt") as a motorization officer and by June was promoted to the rank of major. On 2 October, he was appointed the personal adjutant for the Army's Personnel Department chief, a position he held until the end of the Second World War. As such, he served under General Rudolf Schmundt. On 1 April 1944, Weiß was promoted to lieutenant-colonel. On 20 July 1944, Schmundt was severely wounded in the attempt on Adolf Hitler's life, and was replaced by his deputy, Wilhelm Burgdorf. Burgdorf officially became the department's chief after his predecessor's death on 1 October.