Following the dismissal of the government and Labor's defeat at the 1975 election, Cass was named opposition spokesman for health in Whitlam's shadow cabinet. When Bill Hayden replaced Whitlam as opposition leader in December 1977, Cass was given the portfolio of immigration and ethnic affairs. He supported cutting immigration, stating there were not enough jobs for migrants. In 1978, he stated that there was "considerable organisation" behind Vietnamese boat people coming to Australia.