A new approach for super-resolution and classification applications on neonatal thermal images

ABSTRACTABSTRACTThermal imaging systems are harmless to human health and enable contactless heat measurements. The thermal cameras are used in many public sectors where it is necessary to detect the change of temperature values. However, thermal cameras are costly and produce images with low edge information. This situation prevents the widespread use of thermal cameras. Therefore, in recent years, the researches to advance the quality of thermal images have increased. Within the scope of the studies in this paper, first of all, three different datasets consisting of thermal images in the colourful format of neonates were created. Also, TSRGAN+ deep network model was presented for super-resolution studies. The super-resolution images obtained visually approached ground truth images to a great extent. In addition, these results were compared using the peak signal to noise ratio (PSNR) and the structural similarity index measure (SSIM) image quality metrics. The proposed model showed a superior success in terms of the values of PSNR and SSIM compared to the state-of-the-art models. Here, the PSNR value of the proposed TSRGAN+ model increased by 1–1.5 dB compared to the TSRGAN network architecture, while the SSIM value increased by around 2–3%. Finally, the unhealthy-healthy image classification applications were performed on all thermal image sets in order to implement both the task-based evaluation and a real-life application. Thus, a new method is presented to evaluate the results of the super-resolution studies. Here, firstly, a CNN-based classifier was designed and the classification metrics were obtained for all three datasets. Then, transfer learning was applied using state-of-the-art models (ResNet101, Xception) to increase classification success. Here, the most successful results were obtained in applications using the ResNet101 model. Also, the developed model outperformed the TSRGAN, which achieved the second-best result. When all the obtained results are evaluated, it has been observed that the super-resolution models increase the success of unhealthy-healthy classification by about 10% compared to the low resolution images. In other words, the effects of super-resolution techniques on classification applications are clearly seen. In summary, the logical use of the super-resolution research will enable the common use of low-cost thermal cameras in the application fields such as medicine.KEYWORDS: Thermal image processingdeep learning modelssuper-resolutionimage classificationtransfer learning Disclosure statementThe authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.Additional informationFundingThe work was supported by the Scientific and Technological Research Council of Turkey [215E019]; Scientific Research Project Fund of Konya Technical University [201102001].Notes on contributorsFatih Mehmet SenalpFatih Mehmet Senalp received his B.Sc. degree in Electrical Electronics Engineering from Selcuk University in 2014. He received his M.Sc. degree in Electrical Electronics Engineering from Selcuk University in 2017. He received his Ph.D. in Electrical and Electronics Engineering from Konya Technical University, Institute of Science and Sciences in 2022.His research interests include image processing, thermal imaging, super resolution and deep learning techniques.Murat CeylanMurat Ceylan received his Ph.D. from Selcuk University, Institute of Natural Sciences, Electrical-Electronics Engineering. He currently teaches at Konya Technical University with the title of Associate Professor. His scientific work is mainly on artificial intelligence solutions using image processing in the biomedical field. He is also the CEO of AIVISIONTECH, a company that produces innovation in health.