Pixel Co-Occurence Based Loss Metrics for Super Resolution Texture Recovery

Single Image Super Resolution ( SISR ) has significantly improved with Convolutional Neural Networks ( CNNs ) and Generative Adversarial Networks ( GANs ) , often achieving order of magnitude better pixelwise accuracies ( distortions ) and state-of-the-art perceptual accuracy . Due to the stochastic nature of GAN reconstruction and the ill-posed nature of the problem , perceptual accuracy tends to correlate inversely with pixelwise accuracy which is especially detrimental to SISR , where preservation of original content is an objective . GAN stochastics can be guided by intermediate loss functions such as the VGG featurewise loss , but these features are typically derived from biased pre-trained networks . Similarly , measurements of perceptual quality such as the human Mean Opinion Score ( MOS ) and no-reference measures have issues with pre-trained bias . The spatial relationships between pixel values can be measured without bias using the Grey Level Co-occurence Matrix ( GLCM ) , which was found to match the cardinality and comparative value of the MOS while reducing subjectivity and automating the analytical process . In this work , the GLCM is also directly used as a loss function to guide the generation of perceptually accurate images based on spatial collocation of pixel values . We compare GLCM based loss against scenarios where ( 1 ) no intermediate guiding loss function , and ( 2 ) the VGG feature function are used . Experimental validation is carried on X-ray images of rock samples , characterised by significant number of high frequency texture features . We find GLCM-based loss to result in images with higher pixelwise accuracy and better perceptual scores . 1 INTRODUCTION . A super resolution ( SR ) image is generated from a single low resolution image ( LR ) ( with or without variable blur and noise ) such that the result closely matches the true high resolution counterpart ( whether it exists or not ) ( Park et al. , 2003 ) . There thus exists a vast number of possible solutions ( Dong et al. , 2014 ) for any given LR image , and by an extension , there are many techniques to recover SR details with varying degrees of accuracy . These methods range from the simple and blurry interpolation methods ( bicubic , linear , etc . ) that can not recover contextual features to more complex model-based methods that utilise prior knowledge of certain domain characteristics to generate sharper details ( Dai et al. , 2009 ; Jian Sun et al. , 2008 ; Yan et al. , 2015 ) . The contextual dependence of SR images is addressed with Example-based approaches including Markov Random Field ( MRF ) ( Freeman et al. , 2002 ) , Neighbour Embedding ( Hong Chang et al. , 2004 ) , Sparse Coding ( Yang et al. , 2010 ) and Random Forest methods ( Schulter et al. , 2015 ) . These can be generalised ( Dong et al. , 2014 ) into Super Resolution Convolutional Neural Networks ( SRCNN ) for photographic images ( Dong et al. , 2014 ; Wang et al. , 2016 ; Lim et al. , 2017 ; Yu et al. , 2018 ; Kim et al. , 2015 ; Ledig et al. , 2017 ) , medical images ( Umehara et al. , 2018 ; You et al. , 2018 ) , and digital rock images ( Wang et al. , 2019b ; 2018 ; 2019a ) . The original SRCNN network utilised as a backbone 3-5 convolutional layers ( Dong et al. , 2014 ) that sharpened a bicubically upsampled image . More recent models contain dedicated upsampling layers ( Dong et al. , 2016 ; Shi et al. , 2016 ; Odena et al. , 2016 ) , skip connections to improve gradient scaling ( Kim et al. , 2015 ; Ledig et al. , 2017 ) , removal of batch normalisation ( Lim et al. , 2017 ) , and use of the sharper L1 loss function ( Yu et al. , 2018 ; Zhu et al. , 2017 ) . These changes have all contributed gradually to improving the pixelwise accuracy of the super resolution methods . Despite the high pixelwise accuracy achieved by SRCNN networks 1 , the resulting super resolved images are often perceptually unsatisfying , and easily identifiable by a human observer as `` blurry '' . It is due to the SRCNN can recover accurately features spanning several pixels such as larger scale edges , texture and high frequency features are lost as the network attempts to maximise the pixelwise accuracy over a wide range of possible HR counterparts . The local minima problem thus manifests itself clearly in the ill-posed problem of Super Resolution . Perceptual results can be improved by training the network with a hybrid loss function that combines a pixelwise loss ( L1 or L2 ) with a feature-wise loss that is the L2 loss of features extracted from some intermediate convolutional layer of a pre-trained model ( Johnson et al. , 2016 ) . The most effective method of perceptual texture generation thus far has been the use of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , with SRGAN generated images from low resolution photographs score highly on human surveys of image quality ( Ledig et al. , 2017 ; Dosovitskiy & Brox , 2016 ) as they can recover high frequency textures that are perceived as realistic , both for photographic features ( Ledig et al. , 2017 ) and textural quality in X-ray images ( You et al. , 2018 ) . However , SRGAN generated images are stochastic in the way that high frequency features are regenerated , with a tendency to cause pixel mismatch , which is especially exacerbated by the use of the VGG feature loss ( You et al. , 2018 ; Ledig et al. , 2017 ; Sajjadi et al. , 2017 ) that further tends to result in further distortion , leading to higher pixelwise loss . For natural images this loss in pixelwise accuracy may be secondary , but plays an important role in applications where texture carries actual information , such as in X-ray images . This trade-off between the pixelwise accuracy ( distortion ) and the perceptual accuracy ( Sajjadi et al. , 2017 ; Mechrez et al. , 2018 ; Vasu et al. , 2018 ) , is a consistently emergent limitation in SRGAN performance , whereby a high pixelwise accuracy causes over-smoothing , while a high perceptual accuracy causes pixel mismatch and distortion in some features . While both pixelwise accuracy and perceptual accuracy are important , SISR aims to preserve as much content/characteristics from the original image , while GAN essentially `` makes up '' content that is perceptually satisfying at the expense of pixelwise measures . While pixel-wise distortions can be quantitatively measured , in order to evaluate perceptual performance one typically requires human subjective evaluations of Mean Opinion Scores ( MOS ) , or by leveraging the proxy score produced by `` pre-trained '' models ( which could introduce their own biases ) ( Ma et al. , 2016 ; Salimans et al. , 2016 ) . SRGANs typically obtain superior scores in terms of subjective metrics ( Ledig et al. , 2017 ; Zhu et al. , 2017 ; Vasu et al. , 2018 ) , however , an objective study of the differences in the high frequency SR textures compared to the original HR images has not yet been carried . This is especially of interest in imaging areas requiring expert judgements , such as in radiographic images . These are most often benchmarked with a combination of pixelwise metrics and opinion score surveys despite the smaller sample sizes and logistical challenge . The overall aim of this study is the introduction of Grey Level Co-occurence Matrix ( GLCM ) method ( Haralick et al. , 1973 ) as both an auxiliary loss function , and as an addition to the PSNR metric for evaluating perceptual and textural accuracy of super resolution techniques ( GLCM better correlates with the subjective scores of human evaluations ) . GLCM has been sucessfully used in the characterisation of texture in medical images ( Makaju et al. , 2018 ; Madero Orozco et al. , 2015 ; Sivaramakrishna et al. , 2002 ; Pratiwi et al. , 2015 ; Liao et al. , 2011 ; Yang et al. , 2012 ) and CT rock images ( Singh et al. , 2019 ; Becker et al. , 2016 ; Jardine et al. , 2018 ) . In essence , the GLCM transforms an image into a representation of the spatial relationship between the grey colours present in the image . The GLCM is not a pixelwise measurement , and so can be used to evaluate texture recovery that may not be a pixelwise match . The GLCM is particularly well suited for automatic perceptual/textural in-domain comparisons , as it does not require time-consuming expert MOS evaluations , and does not pose inherent data-bias when scoring with auxiliary models . We use the DeepRock-SR ( Wang et al. , 2019a ) dataset to train and validate the results of an SRGAN model , which in this study is a modified Enhanced Deep Super Resolution ( EDSR ) network ( Lim et al. , 2017 ) coupled to a GAN discriminator ( EDSRGAN ) . The resulting performance in generating SR images is quantitatively analysed , both based on the traditional pixelwise approach , as well as using the GLCM spatial texture accuracy method introduced in this work . GLCM texture analyses of the SRGAN and SRCNN results indicate quantitatively that SRGAN produces images with a GLCM error that are an order of magnitude lower ( more texturally similar ) than SRCNN images , that tend to 1Henceforth , unless stated otherwise , all generative convolutional networks ( and the corresponding generated images ) will be referred to as `` SRCNN '' have a higher PSNR , but also a higher GLCM error and lower MOS scores . Overall , the use of the GLCM offers fast and more agnostic evaluation metric when compared to carrying MOS evaluations , and is easier to reproduce and analyse due to its algorithmic data-driven nature . The GLCM can be also used as an auxiliary loss function to guide the generation of spatially accurate texture , resulting in further reductions in the texture error while also improving the pixelwise accuracy . 2 METHODS . 2.1 GREY LEVEL CO-OCCURRENCE MATRIX FOR TEXTURE ANALYSIS . The Grey Level Co-occurrence Matrix characterises the texture of an image by calculating the number of pairs of pixels that could be spatially related within some pre-defined regions of interest . For a given image with N grey levels , an N by N co-occurrence matrix P is constructed , in which the ( i , j ) position denotes the number of times a grey level i is spatially related to a corresponding grey level j . An example of spatial relationship setting in an image of size ( Nx , Ny ) could be all locations of ( x , y ) and ( x+ 1 , y ) ( i.e . their horizontal adjacency ) . In this case , the value at location ( i , j ) in P is the sum of all occurrences where grey level i and grey level j occur horizontally adjacent to each other within the image . Since the GLCM does not compare spatially matching pixel values but instead compares the spatial distribution of pixel values , it is a good measure of texture similarity to complement the pixel-by-pixel similarity PSNR metric . In general , multiple GLCMs with an encompassing set of spatial relationships are constructed to fully characterise the texture of an image . Aside from the default ( x , y ) to ( x+ 1 , y ) relationship , the offset value can be generalised to an omnidirectional pixel distance d ∈ Z such that the spatial relationships are ( x , y ) to ( x+ dx , y+ dy ) . Some of the the GLCMs used in this study are calculated in 8 directions with a 45 degree offset , to a distance of up to 10 pixels with a 4-bit precision ( 16 grey levels after quantisation ) . This results in a 8x10x16x16 GLCM tensor P , or 80 16x16 GLCMS , one for each spatial relationship setting . This raw transformation can then be analysed for a variety of statistical measures , or a pixelwise comparison can be carried on GLCMs for different images to comparatively quantify texture similarity . If a generated SR image is texturally similar to the original HR image , then it can be expected that the corresponding GLCMs are closer to each other w.r.t L1 or L2 distances . The L1-GLCM error is computed as : GLCMLoss = ∑ ij |PSRij − PHRij |∑ ij ij ( 1 )

Summary: The paper considers the problem of generating a high-resolution image from a low-resolution one. The paper introduces the Grey Level Co-occurrence Matrix Method (GLCM) for evaluating the performance of super-resolution techniques and as an auxiliary loss function for training neural networks to perform well for super-resolution. The GLCM was originally introduced in a 1973 paper and has been used in a few papers in the computer vision community. The paper trains and validates a super-resolution GAN (SRGAN)  and a super-resolution CNN (SRCNN) on the DeepRock-SR dataset. Specifically, for the SRGAN, the paper uses the EDSRGAN network trained on loss function particular to the paper: The loss function consists of the addition of the L1 pixel-wise loss plus the VGG19 perceptual objective plus the proposed GLCM loss.