The 5 o'clock shadow refers to the shadowy appearance of the first stubble that shows through after some time on shaved skin.