Smaller is definitely better here. Refractors can be extremely good amateur instruments, highly portable and easy to care for. Most important, while affordable refractors have little aperture, they usually have excellent optics, far better than commercial reflectors or catadioptrics. Refractors should not be written off by amateurs, especially beginners. Try one you might like it.
Magnification is bad low power, wide views are more desirable.
Experienced astronomers know that increasing the power on deep-sky objects increases the contrast, sometimes dramatically. And, of course, on the planets, you want to use the highest power possible consistent with seeing. Kick up the power if you feel like it and your mount can handle it. The eyepiece is half the telescope. This is a myth that results from large aperture telescopes with short focal ratios.