A road bike, for example, is designed for speed over smooth surfaces. If you were to ride one over bumpy terrain, it would feel horrible. To cope with this harsher environment, mountain bikes are made to be more durable, have a more comfortable ride, and be able to cope with repeated shocks.
There are also significant design differences as mountain bikes having a more upright seating position, flat handlebars, and much thicker tires.