Patterson moved to Gadsden, Alabama, in 1931. He worked as a field representative for the Agricultural Adjustment Administration from 1933 to 1943, as a special assistant to the War Food Administration from 1943 to 1945, and as a special adviser to the Secretary of Agriculture from 1945 to 1947. He remained active in his political party's affairs, serving as a liaison officer for the Democratic National Committee in 1948 and as a delegate-at-large to the 1952 Democratic National Convention.