Some older themes might support featured image captions out of the box. However, using a plugin like Image Source Control to manage and display image captions for all images is more reliable.