Brown served as delegate to Democratic state and national conventions in 1956, 1960 and 1964. He was elected as a Democrat to the 85th and 86th Congresses (January 3, 1957 – January 3, 1961), but was an unsuccessful candidate for reelection in 1960. Brown voted in favor of the Civil Rights Acts of 1957 and 1960.