Pfeffer was now treated with suspicion in Party circles. Following Rudolf Hess’s flight to Scotland in May 1941, Pfeffer was briefly arrested but released. However, he was expelled from the Party on 14 November 1941 and from the "Reichstag" on 27 November. He was now essentially retired, living on his estate in Pommern. Following the failed assassination attempt on Hitler in the 20 July 1944 plot, he was again arrested and held for several months. He survived the Second World War, even commanding a "Volkssturm" division near the war’s end. He then briefly was interned in Heilbronn by the Allies until 1946. He was active in the Hessian State Association of the German Party during the late 1940s and early 1950s. He lived in Wiesbaden until 1960 and then Munich, dying in 1968.