Objective Image Quality Assessment: Facing The Real World Challenges

There has been a growing interest in recent years in the development of objective image quality assessment (IQA) models, whose roles are not only to monitor image quality degradations and benchmark image processing systems, but also to optimize various image and video processing algorithms and systems. While the past achievement is worth celebrating, a number of major challenges remain when we apply existing IQA models in realworld applications. These include obvious ones such as the challenges to largely reduce the complexity of existing IQA algorithms and to make them easy to use and easy to understand. There are also challenges regarding the applicability of existing IQA models in many real world problems where image quality needs to be evaluated and compared across dimensionality, across viewing environment, and across the form of representations specific examples include quality assessment for image resizing, color togray image conversion, multi exposure image fusion, image retargeting, and high dynamic range image tone mapping. Here we will first elaborate these challenges, and then concentrate on a specific one, namely the generalization challenge, which we believe is a more fundamental issue in the development, validation and application of IQA models. Specifically, the challenge is about the generalization capability of existing IQA models, which achieve superior quality prediction performance in lab testing environment using a limited number of subject rated test images, but the performance may not extend to the real world where we are working with images of a much greater diversity in terms of content and complexity. We will discuss some principle ideas and related work that might help us meet the challenges in the future.  Over the past decades, a growing number of researchers and engineers in the image processing community have started to realize the importance of image/video quality assessment (IQA/VQA) [40, 29, 4] This is not surprising because no matter what image/video processing problems we are working on, the same issues repeatedly come up How should we evaluate the images generated from our algorithms/systems? How do we know our algorithm/system is creating an improvement between the input and output images, and by how much? How can we know one algorithm/system performs better than another, and by how much? What should be the quality criterion for which the design of our algorithms/systems should be optimized? Since the human eyes are the ultimate receivers in most image processing applications, human subjective visual testing would be a reliable solution. However, with the exponential increase of the volume of image/video data being generated daily, it becomes impossible to address these quality issues in a timely manner by subjective visual testing, which is slow, cumbersome and expensive. Instead, only trusted objective IQA models may potentially meet these needs. In academia, objective IQA has been a hot research topic, especially in the past 15 years [35, 4, 29] First, the commonly used numerical disotrtion/quality measures in the past the mean squared error (MSE) and the peak signal to noise ratio (PSNR) have been shown to correlate poorly with perceived image quality [28, 30] Second, a large number of perceptually more meaningful IQA models have been proposed, including full reference (where a perfect quality reference image is available when evaluating a distorted image) [35, 4, 29] no reference (where the reference image is not accessible) [34, 24, 31] and reduced reference (where only partial information about the reference image is available) models [39, 36, 31, 29] Third, several design principles have been discovered and repeatedly demonstrated to be useful in the design and improvement of IQA models. These include psychophysical and physiological visibility models [35, 4] the structural similarity (SSIM) approaches [28, 32, 33, 20, 49] the natural scene statistics (NSS) and information theoretic approaches [36, 39, 21, 31] the visual saliency based approaches [50] and the machine learning based approaches  Fourth, a number of subject rated image quality databases have been created and made publicly available [22, 7, 8, 17, 16, 47] They provide a common benchmark platform for the evaluation and comparison of IQA models, among which several algorithms have achieved high correlations with the subjective mean opinion scores (MOSs) of the test images [23, 38, 33, 49] In the video delivery industry, perceptual objective IQA methods such as the SSIM algorithm have been incorporated into many practical hardware and software systems to monitor image/video quality degradations and to test/compare image/video encoders and transcoders [27, 25, 26] The wide use of SSIM has resulted in a Primetime Engineering Emmy Award given by the Academy of Television Arts and Sciences  The remarkable development and successful deployment of modern IQA methods are definitely worth celebrating. Nevertheless, this does not necessarily mean that the existing IQA models have already met the real world challenges. Otherwise, they should have made a much stronger impact and become a gamechanging factor in the industry. Using the video delivery industry as an example, even now most practitioners are still equating bitrate with quality in the practical design of video delivery architectures. However, using the same bitrate to encode different video content could result in dramatically different visual quality. Clearly, the perceptual quality of the video itself, which is presumably the ultimate evaluation criterion of the whole video delivery system, has not been placed at the driver's seat. While it is understandable that quality degradation is inevitable at many stages in the video delivery chain due to practical constraints, the real concern here is that there is no existing protocol to monitor and control such quality degradation. As a result, various tricks have been used to manipulate the video content and network resources are allocated in suboptimal ways, leaving the creative intent of the content producers unprotected. While it is certain that the industry needs to be better informed about the great potentials of making the best use of IQA/VQA models, we believe that an equally important aspect that slows down the process is that the existing IQA/VQA models still do not meet many real world challenges. In the following sections, we will elaborate some of these challenges and then focus on a specific one, namely the generalization challenge. We wish our discussions on some fundamental ideas could provide some useful insights for the future development of IQA models that may meet these real world challenges. The Real World Challenges Here we make a list of real world challenges, many of which are described in more details through examples of practical scenarios. 1. It is highly desirable to reduce the complexity of the IQA/VQA algorithms so that they can be computed in realtime or in an even faster speed. This is especially useful in time sensitive applications such as live broadcasting and videoconferencing. Many existing models are far from meeting this challenge. 2. It is essential to make the IQA/VQA scores easy to use and easy to understand. For example, the raw SSIM score does not have an explicit perceptual meaning, making it difficult to determine what level of SSIM index can warrant an excellent video quality and how much improvement in the SSIM index is sufficient to create visible quality improvement. Mapping the raw scores into a perceptually linear domain that is easily linked to human expressions about image quality is desirable. 3. The same video stream shown on different display devices could result in very different perceptual quality. For example, a strongly compressed video that exhibits very annoying artifacts on a large TV could appear to have fine quality when viewed on the screen of a smartphone. The quality may also change significantly when the video is watched on the same TV but at two different viewing distances, one at the default distance and the other at a very close distance. However, existing IQA/VQA models give the same score based on the video stream only, completely ignorant of the viewing device and viewing condition. 4. In a video on demand application, a high quality highresolution (e.g. 4K) source video may be encoded into multiple video streams of different resolutions (e.g. 1080p, 720p, 360p, 240p, etc. and different bit rates, aiming for satisfying a variety of user needs. In order to measure the quality of the encoded videos, most existing VQA models cannot be computed because the source (reference) and test videos have different spatial resolutions. 5. An image or video may need to be displayed on a screen that has a spatial resolution higher than that of the image resolution. As a result, spatial interpolation is performed. Again, most existing VQA models are not applicable because the reference and test images have different spatial resolutions. 6. An image or video of imperfect quality (e.g. being compressed at an earlier stage) is received and then transcoded to multiple images or videos with different bitrates and resolutions. Most existing IQA/VQA models are not applicable not only because they do not allow for cross resolution quality assessment, but also because they assume the original reference image/video to have perfect quality, which is not the case here. How to carry out "degraded reference" IQA/VQA is a major challenge. 7. A high dynamic range (HDR) image (e.g. the pixels are in 10 or more bit depths) is tone mapped to a standard dynamic range (SDR) image (8 bits per pixel) in order to be visualized on an SDR display. There is certainly information loss that we would like to capture. However, most existing IQA models do not apply because they cannot compare images/videos with different dynamic ranges.