Or the spindle could be a router or better yet a trim router. The trim routers turn at like at 30,000 rpm, great for a small cutter. A drill press does not work the greatest because it turns too slow, and the bearings are not designed for a side load that the milling gives it. With now that you have this spindle that you are going to cut with you need a way to hold it. Adjustable hose clamps work great.
Hose clamps have all the rigidity ones needs.
Your weakest point or points is where your machine is going to vibrate, bend, shimmy, chatter, slop, and all other kinds of nasty problems that will separate your homebuilt from a professional modal. Try to keep the machine parts and tools short as possible and matched. Back to holding the spindle. Imagine if you will your arm and hand holding this spindle. There is a design to start with.