There are worse things. 1997 rankings, the latest I could find, with links to the departments. Advice for undergraduate students considering graduate school . A set of links from the College of William and Mary. You probably can.
Many students find graduate school difficult.
Not all successful classicists are language whizzes, of course, but it sure don't hurt any. You won't take courses much like our general introductory courses, FL 101 and FL 230. I'd say your aptitude, interest, and drive are more important than your knowledge base. The graduate schools may ask for more than this, but most of them won't really mean it. If you have a better foundation, so much the better.