While getting started, you will likely keep your present job, and it may make sense to work from your home. Costs here are the same as in an offline business, and will be a function of your needs. Don't overlook software. For your accounting, Quicken is good. Then there are other things, such as supplies.
Printers chew up enormous gobs of paper.
For competitive prices on consumables, try Office Depot. An email address is not enough. You need a phone and someone to answer it. Even if you expect to receive few calls, this is a must. People often call just to see if you're for real.