MP4 files are made up of two parts: an MP4 wrapper and the actual media content (video or audio). The wrapper contains metadata that describes how the media content should be displayed. This includes how it should be scaled, its aspect ratio, brightness levels, frame rate, etc.
The MP4 format was created because other video formats like AVI were not capable of handling multiple resolutions or bitrates within one single file. They also did not have good support for different formats such as H264, AAC or AC3 audio codecs.