I think you're making this more complicated than it needs to be by supporting JFIF.  I think it would be fine to support EXIF only here and have only one type of metadata.