Celmiņš moved to Italy, then Switzerland. While in Zürich, he was arrested and then banished from Switzerland. He later lived in Romania, where he had contacts with the Iron Guard, and then moved to Finland. In 1938, he became the leader of "Pērkonkrusts"' "foreign contacts office". After the Soviet Union invaded Finland, Celmiņš enrolled as a volunteer on the latter's side. When the conflict ended, he moved to Nazi Germany.