Admittedly, this may be unconvincing to you, but, to cite Wiktionary: 
 vagina (plural vaginas or vaginae or vaginæ) 
 # (anatomy) The passage leading from the opening of the vulva to the cervix of the uterus in female mammals. 
 # (zoology) A similar part in some invertebrates. 
 # (botany) A sheath-like structure, such as the leaf of a grass that surrounds a stem. 
 # (colloquial) The vulva. 

 So, no, it's not "wrong". It's just colloquial (non-technical) usage. Besides, the issue is usage, not definition; OED won't always have usage information, and is more a source for definitions than usage. Also, it's not just me that uses it this way; using vagina to mean vulva is widely used, at least in my experience. I imagine you've heard plenty of people use it that way, and even used it yourself. Most importantly, in most conversations, if someone says vagina, most people would understand that to mean vulva, not the technical meaning of vagina listed in the article. When usage is so widespread, no matter one's opinion of it, it can hardly be called "wrong".