After the fall of the Whitlam government, Stewart moved to the backbench. When Lang was abolished prior to the 1977 election, he transferred to Grayndler. He died in Long Jetty, New South Wales, of a heart attack while playing squash and was survived by his wife, five daughters and one son. Kevin Stewart (1928–2006), Health Minister in the New South Wales government led by Neville Wran, was Frank's younger brother.