Most physics department specialize in just two or three different areas, and it makes no sense to go to a very good university that has no faculty in the areas that excite you. Third, think about whether the geographical region and environment are important to you. This shouldn't be the most important criterion, but if you are going to spend up to 7 or 8 years in a place it would be nice to be reasonably happy during the times when you're not in the lab. A useful reference is Graduate Programs in Physics, Astronomy, and Related Fields, published annually by the American Institute of Physics. This book provides detailed, quantitative information about most physics departments in the country, including research specialties of the faculty and some information on admissions standards.
Most departments also have Web pages with varying amounts of information.
However, your most useful resource is likely to be the physics faculty at Penn. Finally, you should make a serious effort to visit the colleges that interest you. You can get a very good idea about whether a place attracts you by wandering around, talking to professors and graduate students, and generally absorbing the atmosphere of the place. Incidentally, although Penn has a first-rate graduate program and we are proud of our undergraduate program, we do not in general encourage undergraduates at Penn to enter our Ph.D. program. There are real benefits to moving to a different institution where the culture and academic priorities are different, and where you will be exposed to a different set of professors.