Paul was single. Timothy was probably single. Luke was probably single. And of course, Jesus was single.
Single men are by no means forbidden from the office of pastor or elder.
Unfortunately, in our current culture, which tends to overvalue marriage and family, single men are often viewed as immature and incapable of ministering to those in different life-circumstances than themselves. Rather than viewing single men as the gift they can be to the body of Christ, they are considered inexperienced and handicapped. Sadly, we tend to put far more stock on personal experience rather than on doctrinal understanding. For more see my article on Qualifications for Pastoral Ministry .