Fox: James Brolin said Barbra Streisand's blunt remark during a blind date 29 years ago sealed their fate