Later during the rule of Diệm, Hương was jailed in 1960 for three years for signing the Caravelle Manifesto that criticised Diệm. However, after Diệm was overthrown and assassinated in 1963, Hương gave a scathing analysis of the coup generals' action. He stated that "The top generals who decided to murder Diệm and his brother were scared to death. The generals knew very well that having no talent, no moral virtues, no political support whatsoever, they could not prevent a spectacular comeback of the president and Mr. Nhu if they were alive."