Fox: John O'Hurley says it's 'difficult to work' in Hollywood as a white man his age, weighs leaving California