Forged text detection in video, scene, and document images

IET Image ProcessingVolume 14, Issue 17 p. 4744-4755 Research ArticleFree Access Forged text detection in video, scene, and document images Lokesh Nandanwar, Corresponding Author Lokesh Nandanwar lokeshnandanwar150@gmail.com Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, MalaysiaSearch for more papers by this authorPalaiahnakote Shivakumara, Palaiahnakote Shivakumara Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, MalaysiaSearch for more papers by this authorPrabir Mondal, Prabir Mondal Indian Statistical Institute, Kolkata, IndiaSearch for more papers by this authorKarpuravalli Srinivas Raghunandan, Karpuravalli Srinivas Raghunandan Department of Studies in Computer Science, University of Mysore, Karnataka, IndiaSearch for more papers by this authorUmapada Pal, Umapada Pal orcid.org/0000-0002-5426-2618 Indian Statistical Institute, Kolkata, IndiaSearch for more papers by this authorTong Lu, Tong Lu National Key Lab for Novel Software Technology, Nanjing University, Nanjing, People's Republic of ChinaSearch for more papers by this authorDaniel Lopresti, Daniel Lopresti Computer Science and Engineering, Lehigh University, Bethlehem, PA, USASearch for more papers by this author Lokesh Nandanwar, Corresponding Author Lokesh Nandanwar lokeshnandanwar150@gmail.com Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, MalaysiaSearch for more papers by this authorPalaiahnakote Shivakumara, Palaiahnakote Shivakumara Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, MalaysiaSearch for more papers by this authorPrabir Mondal, Prabir Mondal Indian Statistical Institute, Kolkata, IndiaSearch for more papers by this authorKarpuravalli Srinivas Raghunandan, Karpuravalli Srinivas Raghunandan Department of Studies in Computer Science, University of Mysore, Karnataka, IndiaSearch for more papers by this authorUmapada Pal, Umapada Pal orcid.org/0000-0002-5426-2618 Indian Statistical Institute, Kolkata, IndiaSearch for more papers by this authorTong Lu, Tong Lu National Key Lab for Novel Software Technology, Nanjing University, Nanjing, People's Republic of ChinaSearch for more papers by this authorDaniel Lopresti, Daniel Lopresti Computer Science and Engineering, Lehigh University, Bethlehem, PA, USASearch for more papers by this author First published: 25 February 2021 https://doi.org/10.1049/iet-ipr.2020.0590Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Rapid advances in artificial intelligence have made it possible to produce forgeries good enough to fool an average user. As a result, there is growing interest in developing robust methods to counter such forgeries. This study presents a new Fourier spectrum-based method for detecting forged text in video images. The authors' premise is that brightness distribution and the spectrum shape exhibit irregular patterns (inconsistencies) for forged text, while appearing more regular for original text. The method divides the spectrum of an input image into sectors and tracks to highlight these effects. Specifically, positive and negative coefficients for sectors and tracks are extracted to quantify the brightness distribution. Variations in the shape of the spectrum are analysed by determining the angular relationship between the principal axes and the sectors/tracks of the spectrum. Next, it combines these two features to detect forged text in the images of IMEI (International Mobile Equipment Identity) numbers and document. For evaluation, the following datasets are used: own video dataset and standard datasets, namely, IMEI number, ICPR 2018 Fraud Document Contest, and a natural scene text dataset. Experimental results show that the proposed method outperforms existing methods in terms of average classification rate and F-score. 1 Introduction Most popular social media services, such as Twitter, Facebook, Instagram, provide first-hand news provided by traditional journalists and ordinary users [1, 2]. At the same time, with the rapid development of multimedia technology and user-friendly image editing software, such as Adobe Photoshop and Premiere, and Mokey by Imagineer systems, manipulating video news by altering the content is becoming significantly easier. As a result, Facebook, Twitter, and Instagram report many cases of altered content in images and videos [1, 2]. Elsewhere, there are other sensitive applications involving traditional news media, military matters, and law enforcement, for which forged data detection is vital. Hence, there is an urgent need for new methods capable of detecting forged information in images and videos. Video forensics can be broadly classified into two categories: active forensic and passive forensic [1]. In the case of the former, while generating a video, the system can include validation information such as a digital watermark or signature, and/or fingerprinting which can be used for authentication. In the case of the latter, the veracity and integrity of video information is authenticated without requiring any embedded validation information, making it more practical than active forensics [1]. This has made passive video forensics hot research topic. Numerous methods have been developed for detecting forged images and videos created through copy-paste operations and splicing. For instance, Selvaraj and Karuppiah [3] proposed a technique based on SIFT features. Soni et al. [4] addressed forgery detection in images using density-based clustering and noise closeting algorithms. Fadi et al. [5] proposed an approach for inter-frame forgery detection in videos using spatio-temporal information. Chen et al. [1] explored automatic detection of object-based forgeries in videos. D'Amiano et al. [6] proposed a match-based dense field algorithm for copy-move forgery detection and localisation. Feng et al. [7] discussed motion adaptive frame deletion detection for digital video forensics. Pun et al. [8] proposed image forgery detection using adaptive over-segmentation and feature point matching. Yang et al. [9] explored copy-paste forgery detection based on hybrid features. Tian et al. [10] proposed robust independent elementary features and a similarity metric for detecting copy-move forgeries in images. We note that most of the above methods focus on general visual content, but not on text that appears in images. The scope of this method is limited in this regard. This makes forged text detection an interesting topic for research in forensic applications. Methods [11, 12] developed for text detection in video and natural scene images work well irrespective of the text modality, i.e. caption/superimposed text which is edited into an image versus scene text which is an inherent part of the original image. Consider, for example, the results shown in Figs. 1a and b for video and scene images, respectively. Here the method in [12], which employs deep learning, detects the text correctly, including the indicated forged text. Based on this observation, our work uses the output of text detection as the input for our approach to forged text identification. Throughout our work we assume that the altered text in question has been created using standard image editing operations, namely, copy-paste and insertion. Figs. 1a and b show that using Photoshop software, one can copy desired words from a source image and paste them into a target image, and also insert words to create a fake message. Furthermore, when we look at the original and forged text, it is not easy to differentiate them visually without some assistance. This makes the problem interesting and challenging. Fig. 1Open in figure viewerPowerPoint Example of copy-paste and insertion operations for creating forged text, along with the results of the text detection method (a) Word ‘MirrorNow’ in the source image is copied and pasted into the target image. The text detection method segments the words in source and target images well as indicated by the bounding boxes, (b) Word ‘LIVE’ is inserted using Adobe Photoshop software. The text detection method segments the original and forged text in the natural scene image well as indicated by the bounding boxes 2 Related work Since forged text detection in video images can be seen as related to forgery detection in document images, we review methods for identifying fraud in handwritten and printed documents. Likewise, since caption text in videos is super-imposed, this can be considered a similar alteration of the original video image, with naturally occurring scene text viewed as a non-altered image. For this reason, we also review related past work on caption text disambiguation from natural scene text. Barbosa et al. [13] proposed forged text detection in documents written using ballpoint pens. This method uses the ink as a feature. Elkasrawi and Shafait [14] proposed laser printer identification using supervised learning for document forgery detection. This approach explores using noise produced by different printers for fraudulent document classification. Ahmed and Shafait [15] proposed forgery detection based on intrinsic document contents. This method uses the similarity between blocks of an image to identify forged documents. Shafait et al. [16] proposed automatic ink mismatch detection for forensic document analysis. This method analyses inks from different pens to detect fraudulent documents. Luo et al. [17] proposed localised forgery detection in hyperspectral document images. This is an improved version of the above method, which uses ink quality in the hyperspectral space. Bertrand et al. [18] proposed a system based on intrinsic features for fraudulent document detection. This approach extracts features or characters to match with the ground truth for fraud estimation. Barboza et al. [19] proposed a colour-based model to estimate the age of a document for forensic purposes. This approach uses the ink quality of handwritten documents captured over different time periods. The method [19] helps us to identify a given image is old or new. Halder and Garain [20] proposed a colour-based approach for determining ink age in printed documents. This method uses the same colour features as above for printed text images. Kumar et al. [21] proposed detecting alternations in ball point pen strokes. This approach employs colour and texture features for handwritten documents, and in particular bank cheques. Raghunandan et al. [22] proposed Fourier coefficients for fraudulent handwritten document classification using age analysis. This approach studies positive and negative coefficients for analysing the quality of images, which mark them as old or new. Wang et al. [23] proposed a method for printer identification by analysing the output of different types of laser printers. This is based on the premise that each printer introduces its own unique noise during the printing process. Shivakumara et al. [24] proposed a method for detecting forged IMEI (International Mobile Equipment Identity) numbers using a fusion approach based on colour space. The performance of this method depends on Canny and Sobel edge information. If a forged image contains sufficient distortion as a result of the editing, the method works well. Bibi et al. [25] proposed a text-independent approach for document forgery detection through printer source identification. The method employs convolutional neural networks (CNNs). Kalbitz and Vielhauer [26] proposed forensic ink evaluation in handwritten documents using a clustering approach. This method exploits the idea of detecting the use of multiple inks used in a document as an indicator of forgery. Mukhtar and Malhotra [27] proposed a similar idea for legal handwritten documents. The method analyses the colour of the ink used in the document. Rahiche and Cheriet [28] proposed a technique for forgery detection in hyperspectral document images based on graph orthogonal non-negative matrix factorisation. Their method graphs regularised terms to exploit the geometric information lost in the matricisation of the images. Chen and Gao [29] proposed a method for forged handwriting number detection based on a CNN. This method pools different layers for forgery detection in the images. Then review of existing work shows that the methods focus on one type of image, whereas the proposed work considers different types of images in this paper. Nandanwar et al. [30] proposed method for detecting altered text in document images. Their approach splits the Discrete Cosine Transform (DCT) into positive and negative coefficients and reconstructs images for the respective positive and negative coefficients. However, since the method was developed specifically for document images, it may not work well for videos and natural scene images. As indicated in our review above, there are methods that assume printed text on a plain background, while for forged IMEI number detection, there are methods that take advantage of a stable background. In addition, there are methods that use colour or character stroke components for forgery detection. However, these features may not work well for video images, where one cannot expect a plain or fixed background; rather these images can contain complex backgrounds with a large variety of variation. Finally, none of the previous methods we have surveyed here aims at detecting forged text across videos, natural scenes, and document images with a single unified approach. To overcome the problems of methods that assume a plain background, a few approaches have been developed to detect caption text in videos. Shivakumara et al. [31] proposed a technique to separate graphics and scene text in videos. This method works based on the fact that caption text has high contrast and clarity, while scene text does not. Xu et al. [32] also studied graphics and scene text classification in videos. Their method extracts distinct features through distribution of eigenvalues. Roy et al. [33] proposed new features for scene and caption text classification in video frames. This method explores DCT coefficients to differentiate caption text from scene text. Bhardwaj and Pankjakshan [34] proposed image overlay text detection based on JEPG truncation error analysis. This method extracts error-based features through truncation given by a colour filter array for detecting caption text in videos. However, the above methods are not adequate for forged text created through copy-paste and insertion operations because the forged text might be classified as caption text and vice versa. Hence, in this paper, we present a method which exploits brightness distribution (BD) and the shape of the Fourier spectrum through sectors which divide the spectrum into eight divisions based on eight directions and tracks, which are defined by drawing circle over sectors. The main basis for this work is that when a person performs forgery operations like copy-paste and insertion, the forged text will not perfectly align with the other text in the image and there will be irregularities in its background compared to regions around non-tampered text [8, 9]. Therefore, we expect the Fourier spectrum computed from original text will exhibit a regular pattern, while forged text will exhibit an irregular pattern. The main contributions of this paper are three-fold: (i) exploring the Fourier spectrum and its shape for text forgery detection in videos, natural scenes, and document images. (ii) Combining the spatial information and angular information of the spectrum in a new way for addressing this challenging task, and (iii) the way the proposed work combines features and neural networks (NNs) is new compared to the existing methods. 3 Proposed methodology As mentioned in the previous section, the output of a text detection method is the input for our proposed forged text detection. We assume the forgery is created through copy-paste and insertion operations. Inspired by observations from the literature [8, 9], while forged words may appear authentic at first glance, when we look more closely at the pixel level, the text alignment and background exhibit irregular patterns compared to original text pixels. This can be seen in the examples displayed in Figs. 2a–d, where for the original and forged images, the distribution of pixel values (visualised as histograms) exhibit clear differences between original and forged. This shows that even though the shapes of characters may look the same, there is a significant difference at the pixel level. As can be seen in the figures, text forged via copy-paste appears burred in comparison to original text, while text created through an insertion operation appears bright relative to original text. Fig. 2Open in figure viewerPowerPoint Pixel distribution of original and forged text using copy-paste and insertion operations (a) Original and forged text of copy-paste operation, (b) Intensity distribution of original and forged text using copy-paste operation, (c) Original and forged text of insertion operation, (d) Intensity distribution of original and forged text using insertion operation This observation leads us to explore the distribution of brightness and the shape of the Fourier spectrum for achieving our goal in this work. The proposed method obtains spectrum for the original and forged texts of copy-paste and insertion operations as shown in Figs. 3a–f, where we can see clear differences in BDs and shapes of spectrum between the original text and the forged texts by both the two operations. This is the main basis for feature extraction to detect forged text in this work. At the same time, since the basis is the same for different types of text images (videos, natural scenes, and documents), we believe that the aforementioned observations on BD and spectrum shape analysis should be the same. Therefore, the same set of features should work well without depending on the content of the images. Fig. 3Open in figure viewerPowerPoint Examples of BD and shape of the spectrum for the original and forged text using different operations (a) Original images for copy-paste and insertion operations, (b) Fourier spectrum of the images in (a), (c) Binary form for Fourier spectrum in (b), (d) Forged text corresponding to the above images in (a), (e) Fourier spectrum of the forged images in (d), (f) Binary Fourier spectrum of forged images shown in (e) In Fig. 3, we can see the original and forged words created by copy-paste and insert operations, respectively, where we can see BD of spectrum shown in Figs. 3b and e is different for original and forged images. It is noted from the binary form of spectrum of the original and forged images shown in Figs. 3c and f, respectively, that spectrum shape of the original image appears different from that of the forged word. When we compare binary spectrum in Figs. 3c and f, the binary form of forged word loses pixels as, hence it affects the shape of spectrum (SoP). Note that the binary form for Fourier spectrum is obtained as described well-known steps in [35]. Our proposed method extracts statistical features by dividing the spectrum into sectors and tracks, which represent the BD. At the same time, to strengthen the statistical features, we extract angular-based features for the sectors and tracks through principal component analysis (PCA). The statistical and angular-based features are aggregated for the classification of forged text using a NN classifier. The block diagram of the proposed method can be seen in Fig. 4. Fig. 4Open in figure viewerPowerPoint Block diagram of the proposed method 3.1 Brightness distribution for forged text identification We divide the whole spectrum into eight equal divisions (sectors), and then construct rings with a fixed radius over the sectors, which are called tracks. The radius is determined experimentally and is presented in the experimental results. It is known that positive coefficients usually represent high contrast values (edges), while negative coefficients represent low contrast values (background). We propose to use positive and negative coefficients for classifying bright sectors and blur sectors to study the irregularity in BD. Fig. 5a shows eight equal sectors according to angle information, where we can see red colour indicates sectors of the spectrum. The rings which are called tracks are formed with radius of 5 pixels over 8 sectors for the spectrum as shown in Fig. 5a, where we can see blue colour indicates tracks of the spectrum. This results in a number of patches denoted by the green colour in Fig. 5a. The number of tracks is decided based on the length of the major axis of the spectrum automatically. The radius of 5 pixels is determined empirically in this work. Fig. 5Open in figure viewerPowerPoint Classifying the number of SP and BP from the spectrum to study the BD (a) Constructing patches (green colour) for the sectors (red colour) and tracks (blue colour) of the spectrum, (b) Bar graphs to count the number of SP and BP for the original images shown in Fig. 3a, (c) Bar graphs to count the number of SP and BP for the forged images shown in Fig. 3d The brightness of the spectrum depends on the number of positive and negative coefficients. Therefore, we calculate the percentage of positive coefficients (PPC) and negative coefficients (PNC) for each patch of each track as follows: PPC = c pos c total × 100 PNC = c neg c total × 100 (1) where c pos and c neg represent the number of positive coefficients and negative coefficients in each patch of respective tracks, and C total refers to the total number of coefficients in each patch of a track. As expected, if a patch is bright, PPC should be larger than PNC. With this notion, we derive the condition that if PPC is larger than PNC, then the patch is classified as a sharp patch (SP), otherwise it is classified as a blurred patch (BP) as defined in the following equation: SP if PPC > PNC BP if PPC ≤ PNC (2) Our method uses (2) to classify each patch as sharp or blurred with respect to tracks. The proposed method counts how many SP and BP are there for each track, which is illustrated in Figs. 5b and c for original and forged images using copy-paste and insertion operations. It is observed from Figs. 5b and c that the count that satisfies (2) for SP is higher than that satisfies (2) for BP in case of the original image, while it is opposite for a forged image. In other words, the bar that represents bright marked by red colour is greater than that represents blur marked by yellow colour, and higher for most of the tracks in case of an original image while for a forged image it is vice versa. This shows that the number of SPs is greater for the original image compared to the number of BPs for a forged image. This makes sense because when an image is affected by a forgery operation, it degrades quality and hence it loses brightness. The proposed method records the total number of SP and BP for each track, which results in two sets, say, TSP and TBP, which are represented as follows: T SP = S t 1 , S t 2 , … , S t α T BP = S t 1 , S t 2 , … , S t β where α and β are the number of the members in SP and BP, respectively. The size of both TSP and TBP is the total number of the tracks of the spectrum, that is, α + β. To find the distance between the two sets, we calculate the common mean for both the sets, say, M, and then estimate the standard deviation, say, SM–TSP and SM–TBP, for each set with respect to the common mean. Similarly, we calculate the standard deviation using the common median (Me), say, SMe–TSP and SMe–TBP, for each set. This procedure results in feature vector-1 (FV-1) which contains four standard deviation values as defined in the following equation: S MTSP = STD M TSP S MTBP = STD M TBP S MeTSP = STD Me TSP S MeTBP = STD Me TBP (3) where STD ( ) refers to the standard deviation, M TSP denotes the common mean for the number of SPs, M TBP denotes the common mean for the number of BPs, M e TSP represents the common median for the number of SPs, and M e TBP represents the common median for the number of BPs. In the same way, to find the distance between SP and BP, our method employs K-means clustering with K = 2 on TSP and TBP sets. This outputs two clusters, namely, Max cluster which contains high values and Min cluster which contains low values. Again, we calculate the mean and standard deviation for Max and Min clusters, separately. This procedure gives four more values, which is labelled as feature vector-2 (FV-2). Note that both FV-1 and FV-2 represent the whole spectrum of the input image. The above steps outputs FV-1 with four features and FV-2 with four more features. In total, the proposed method extracts eight features from the spectrum distribution. The motivation to use K-means clustering with K = 2 is as follows. The sets, namely, TSP and TBP, contain the number of SPs and BPs in each track. For the original image, the number of SPs varies from one track to another. However, the number of SPs is greater than the number of BPs for every track. This is due to the complexity of the background and multiple types images. Therefore, the set, TSP contains high and low values. However, the difference in the values is not significant. In case of a forged image, the number of BPs is greater than the number of SPs for every track. The set, TBP contains high and low values for the forged image. However, the difference between the values is significant in contrast to the original image. This is due to the distortion created by forgery operations. Therefore, we can conclude that both sets have high and low values with different degrees of variation in the values. To exploit this observation, we propose to use K-means with K = 2 to obtain two clusters, one called Max which contains the high values, and the other called Min, which contains the low values. As mentioned in the Introduction, the considered problem is complex, the features extracted from spectrum distribution may not be adequate for achieving better results and hence we propose to extract shape-based features for strengthening the above features, which will be presented in the next section. 3.2 Spectrum shape for forged text identification When the BD is affected by forgery operations, one can expect the same effect on the shape of the spectrum as shown in Figs. 6a and b for the original and forged text, respectively. It is observed from Figs. 6a and b that the principal axis drawn for the original text is not the same as the principal axis drawn for the forged text. Therefore, the proposed method considers the binary form of spectrum as the input for studying shape changes. For the binary spectrum, the proposed method divides it into eight equally sized sectors and tracks as shown in Fig. 5a. It then finds the angle for the whole spectrum using PCA, which considers the first eigenvector for drawing principal axis, and labels it as the reference angle (RA). To study the effect of individual sectors and tracks with the whole spectrum shape, the proposed method estimates the angle for all the patches of sector-1 using PCA. Then it estimates the angle using PCA for all the patches of sector-1 and sector-2, and so on. As a result, this process gives eight cumulative angles CA sec j for eight sectors. The proposed method computes the absolute difference of each of the eight cumulative angles with the RA, which gives eight difference values DV sec j as defined in the following equation: DV sec j = CA sec j − RA , ∀ j = 1 , 2 , … , 8(4) Fig. 6Open in figure viewerPowerPoint Principal axes (green colour line) for the original text and the forged text attacked by copy-paste and insertion operations (a) Principal axis (green colour line) for the original images of copy-paste and insertion operations, (b) Principal axis (green colour line) for the forged text attacked by copy-paste and insertion operations In the same way of obtaining cumulative angles for the sectors, the proposed method estimates the angles for all the patches of track-1, the angles for all the patches of track-1 and track-2, and so on. This process outputs cumulative angles of tracks, which is considered as feature vector-3 containing eight features. The number of cumulative angles CA trac j is the number of the tracks of the spectrum. The cumulative angles are compared with the RA to find difference values DV trac j w.r.t. tracks as defined in (5). This is considered as feature vector-4 containing 16 features DV trac j = CA trac j − DV trac j , ∀ j = 1 , 2 , … , 16(5) where 16 is the number of the tracks of the spectrum. The proposed method combines FV-1, FV-2, FV-3, and FV-4, namely, 4 + 4 + 8 + 16 as one feature vector, which gives a vector of 32 features. The effect of the features can be seen in Figs. 7a and b, where it can be seen that the curves that represent the original and forged images of cop