As soon as Schwarzenegger was elected governor, Willie Brown said he would start a drive to recall the governor