I'm shocked at the prices I'm getting. You intentions may be modest, but that does not change the fact that construction is expensive and going up all the time. The relative cost of construction may not be as bad as all that. A new study by the Federal Reserve Bank says that owning a home is more affordable now than it was in the 1950's. All of which does not help you get a house built for a lower price.
Construction projects require hundreds of decisions.
Be clear about your choices, and have everything in writing. Be well organized, and watch every penny on every item. Standard choices are less expensive than special items. Be a smart customer, and spend a lot of time to pre-qualify your contractor. Demand quality work, but show the contractor that you are a reasonable person.