the party was composed mainly by former whigs and know nothings