Stewart and Harlan were conservative Republicans, while Brennan was a Democrat who became a leading voice for liberalism