It could be damaging your chances of getting the job. If your references don't return calls or emails from the hiring manager, it's going to reflect badly on you, and in a competitive job market, you cannot afford to have anything negative to happen to you if you hope to receive a job offer.