you will summary 

Hacker Factor Logo   The Hacker Factor Blog
Smile... tomorrow could be worseHome
Blog
Swag
About
Dr. Neal Krawetz writes The Hacker Factor Blog. Follow him on Mastodon.
Popular Posts
• How Not to do Image Analysis Part I and Part II
• Looks Like It
• Body By Victoria
• Direct Deposit, Direct Withdrawl
• Point-of-Sale Vulnerabilities
Tools
• FotoForensics: Test your own photos.
• Hintfo: View metadata.
• Gender Guesser: Use your words.
Links
Security
Internet Storm Center
Krebs on Security
Bruce Schneier
Tao Security

Images
Photo Stealers
Awkward Family Photos
Unsplash

Debunking News
iMediaEthics
Poynter

Debunking Politics
FactCheck
PolitiFact

Debunking Other
Snopes
Math with Bad Drawings
Calendar
Archives
Feeds
Categories
Perceptual Searches and Applied Heuristics
Saturday, 22 June 2024
I've been seeing a steady increase in new search-by-image (perceptual search) systems. Much of this work is related to the image provenance problem: find the original version. I've seen a variety of new search technologies, including four new approaches this year. Some identify variations of the same picture, while others can identify instances of the same objects.

A Perceptive Eye
Different perceptual search engines serve different purposes. I often find myself using a combination of Google Image, Bing Image Search, and TinEye to track down variations of pictures and related information. I also have my own private perceptual search engine at FotoForensics (not publicly accessible due to privacy concerns) that I use for finding variations of the same picture based on the uploaded content.

When I see new image search technologies, I immediately try to identify their strengths and capabilities:
Has it indexed pictures from across the internet, or from a small niche market? Google and Bing have indexed large swaths of the internet. TinEye has a smaller collection that covers stock photos and older viral images. The standalone commercial FotoForensics system includes a RootAbout search engine that indexes the client's collection (and not the entire internet).

What kind of resources does it need? My RootAbout engine can compare over 3 billion images per second per CPU, with no special hardware and only a few megabytes of disk space for indexing millions of pictures. (RootAbout is fast and lightweight; it takes about 12 opcodes per comparison, but with pipelining, it runs at the speed of the CPU.) In contrast, Google's Image system is hardware intensive and requires lots of disk space. (If you don't have dedicated hardware, then it probably won't work well.) Some of the other solutions I've seen vary from dedicated distributed computers to multiple CPUs and large local archives.

What does it identify? RootAbout is limited; it only finds variations of the visually same picture. If you take a picture of a chair and scale, crop, or do minor edits, then RootAbout will still match it. However, if you take two photos of a chair from different angles, then they are two visually different pictures and RootAbout won't match them. Other engines, like Google and Bing, do object identification and would match the chair at different angles. But then again, the same chair will likely match other visually similar chairs; I haven't seen any object tracking systems perform unique object identification.

What does it return? Some engines list the files that matched. Others identify the types of objects, any printed text in the image (OCR), decoded barcodes, etc.

How sensitive is it to rotation, cropping, recoloring, and edits? TinEye and Google often miss mirrored images, which is why many scammers flip pictures.
It's not that one search engine is better than any other. Rather, each provides a different range of capabilities. You should choose the correct tool based on the problem you're trying to solve.

What is Authoritative?
When examining new search technologies, I also try to determine the developer's use case and intent. If they intend to create another way to find similar pictures, that's great! But if they think it will be a magical solution for establishing authentication or provenance, then that's a problem.

For example, in one of the chat forums that I frequent, a small company announced their new image search engine. They intend to use it for determining image provenance. One of the people in the forum (not me!) asked, "How do you determine who's the original creator? (by the earliest published timestamp?)" The reply? "Yes, earliest published timestamp from the social posts." Ouch.

Sadly, depending on "the oldest one" or "the first sighting" is very common and a really weak heuristic. (A heuristic or rule of thumb is a shortcut or rough approximation based on partial information.) In fact, there are a bunch of common heuristics that developers often rely on for resolving "which one is most authentic" quandaries. These include:
Oldest is original. The basic belief is that the earliest sighting hasn't had time to go viral or undergo lots of alterations. However, often the original is posted after the controversy begins or is never provided. Additionally, the search engine may not have indexed all content on the internet. "The oldest that we indexed" may not be the oldest instance out there. Relying on something like the "earliest published timestamp from the social posts" assumes that you can index every social media service and cross-posting, have access to private forums, the "original" media was posted to the social network, and there was no delay between image creation and forum disclosure. None of these are good assumptions.

Biggest dimensions. When pictures go viral, they are often resized smaller. While the biggest dimensions usually suggest the highest quality picture, it's not always the case. Scaling a picture larger or outpainting algorithms (where AI extends picture's edges) are examples of larger dimensions not being more authoritative. For example, the GitHub project stablediffusion-infinity has a great example where outpainting was used to make a famous painting ("Girl with a Pearl Earring" by Johannes Vermeer, c. 1665) larger:



This heuristic can cause big problems when used for automatically determining authenticity. For example, Truepic Inspect (a defunct service) used "image quality" (a combination of dimensions and last-saved quality) to automatically assign a trustworthiness value. When I evaluated their service, I found that simply scaling a picture larger made them report that it was more trustworthy.

Biggest file size. In general, higher fidelity requires more data, so a bigger file size may be assumed to reflect better quality and fewer resaves from alterations or viral distribution. However:

PNG files are almost always larger than an equivalent JPEG, but no cameras natively capture photos as PNG. This means that the larger PNG file is never an original photo.

If I have a JPEG that was last saved at 80% quality, and then re-save it at 100% quality, then the file size will be larger even thought the fidelity won't improve.

Some mobile devices, like Apple and Samsung, can include additional images after an in-camera alteration. These depth-maps and preview images almost always makes the file larger, even though the image is not more authentic.

While the file size is a good initial heuristic, it's far from authoritative.

Most metadata. In general, camera original images have a lot of metadata, while viral distribution (Facebook, Twitter/X, Telegram, etc.) typically strip out the original metadata. So if you see a picture with more metadata, then it may not be a viral distribution. However, many applications record changes in metadata, so more alterations can also result in more metadata. Just consider something like XMP; some cameras add a little XMP, but lots of XMP usually indicates an alteration. Good forgeries often copy over camera-original metadata. (The AvtoVAZ forgery method was great at replicating metadata.) Relying on the most metadata could lead you to the most-altered version.

C2PA provenance. In theory, C2PA's metadata should record how a picture was created and where it came from. In practice, it is unreliable at best and completely misleading at worst. On top of this, Adobe and Microsoft will digitally sign anything; they don't validate the metadata before signing. Solutions that rely on any kind of C2PA metadata today are fundamentally flawed. What I tell my customers: if you see C2PA metadata, then you should immediately assume that it was altered by an application and not trust anything without external confirmation.

C2PA timestamp. This also goes toward the belief that the "oldest is most authentic". However, timestamps from local signing are often incorrect, easy to fake, and permit altering without detection.
Heuristics are not intended to be authoritative. (If they were authoritative, then they wouldn't be heuristics.) Some developers think they can combine these heuristics to make a more reliable system. For example, "biggest dimensions and most metadata and C2PA information." The problem is, these attributes don't combine like probabilities; you can't add them up or multiply their metrics. Rather, heuristics combine like a min/max functions or the weakest link in a chain: they are as reliable as the most reliable heuristic. And as a heuristic, none of them are authoritative or very reliable.

Proof By Example
As an example of how these different heuristics work, consider this picture (click to view it at FotoForensics):



On 2024-06-13, FotoForensics received this picture of orange flowers with a white border. I don't know who uploaded it or the background story behind it. The file is 1024x768 and 968,511 bytes.

Using my own perceptual search engine, I found five other versions of this picture at FotoForensics. None of them have the white border. For easy referencing, I assigned a title to each of them based on either the uploaded filename or where it came from. While each picture is visually similar, they are all different variants:

	Title: "Orange". First received at FotoForensics on 2023-01-22. At 99,644 bytes and 698x527, this is the smallest version I've seen.
	Title: "Facebook". First received on 2022-03-21. The image is 1024x768 and 163,950 bytes. The upload came from Facebook. However, Facebook only distributes content that came from elsewhere. This only identifies that it originated somewhere else before it was uploaded to Facebook.
	Title: "WelcomeScan". This is also 1024x768 but is 228,040 bytes. The metadata attributes the picture to "Natphotos/Digital Vision/Getty Images", indicating that it is based on a stock photo. The IPTC metadata doesn't identify the specific stock photo ID, but says it was dated 2004-04-09. (I'm taking all metadata at face value and not assuming malicious alterations.)
	Title: "Fleurs". This is also 1024x768 and attributed to the same 2004-04-09 stock photo, but the file is 516,424 bytes.
	Title: "Bienvenue". While this is the same file size (516,424 bytes) and same dimensions (1024x768) as the Fleurs picture, the metadata is a little different. This file was first uploaded to FotoForensics on 2013-04-06, making it the oldest direct sighting. It is the only version that has been repeatedly uploaded to FotoForensics (92 times!), making it the most popular variant. However, it's still not "original". Besides being a stock image, it contains no camera information in the metadata and was last saved using Adobe Save-for-Web.

Google and Bing image searches also turn up lots of versions of this picture. For example, Google Search found an older version at Flickr:



Flickr received this on 2016-05-12 (nearly 3 years after FotoForensics first saw it). Flickr's "Original" size is 1024x768, but only 228,076 bytes, making it a smaller file than my earlier version. And while the metadata does identify it as being a stock image from Getty, the only date in the metadata denotes when it was re-encoded by Adobe Photoshop Elements for Windows on 2006-02-06 (nearly two years after the other stock image dates seen in the other variants).

TinEye helped me find the picture at IMDB. It's the cover for a short movie called Spring Day by Rouzbeh Rashidi (circa 2000). The odd thing is, it's supposed to be a six-minute black-and-white movie, but this is a very colorful image. I suspect that the cover art was added after the movie came out. (The cover art image is dynamically scaled by a service at Amazon, so there is no informative metadata. We don't know when it was added to IMDB.) This instance of the orange flowers could be a red herring.

While TinEye does identify some stock photo services with this image, none of the stock services seem to have it online. For example, ShutterStock reports, "We're sorry! 1829668118 is no longer available. You might be interested in the related results below." The rest of the links are either unavailable or clearly not authoritative. (There's a common scam where people find pictures online and uploaded them as the author to various stock services, in hopes of making a quick buck from licensing. They rely on the fact that these stock services don't authenticate the uploader's ownership claim.)

Applied Heuristics
Now we can apply the previously defined heuristics to see how they hold up for this one sample image:
Oldest is original. The heuristic's interpretation varies based on how we define "old". For example:

The oldest version at FotoForensics is from 2013; it's also the most popular variant.

The oldest confirmed timestamp online is from Flickr, which says it was received in 2006.

Going by internal metadata, I have three different versions that identify the picture as being from 2004-04-09. But since they are all different variants with the same information, so I don't know which is the official version.

There's an unconfirmed sighting at IMDB, linking the image to a movie that came out in 2000. While I don't think this is accurate, it is the oldest attribution.

In each of these case, none are the "original" picture. They are all re-encoded versions that lack original metadata. This heuristic doesn't identify the original image.

Biggest dimensions: Many of these variants have the same big dimensions: 1024x768. However, I can use Amazon's dynamic image generator to render the picture larger. (How about 2000x1500!) Although the source "white border" picture has the same dimensions as Facebook, WelcomeScan, Fleurs, and Bienvenue, it has a big white border. If we exclude the border, then the visually similar content is smaller in the white border version than these other variants. So this heuristic suggests that the white border version isn't the most authoritative. Unfortunately, since we have multiple picture with the same "biggest dimensions" (1024x768), we don't know which (if any) is the original.

Biggest file size. The white border version is the only PNG and it's almost twice the size of the largest JPEG. The largest JPEG is Bienvenue, which is also the most popular variant. However, it was encoded as a JPEG at 100% quality. The other JPEG images are smaller, but were also saved at lower qualities, which results in smaller file sizes. (Fleurs is at 99%, WelcomeScan at 94%, and Facebook at 81%.) Saving the image at a higher JPEG quality doesn't improve the image; it only mitigates further degradation. Based on the JPEG entropy (amount of noise) in the picture, Bienvenue has a cumulative degradation equivalent to a JPEG saved one time at 55% quality. It may be a big picture, but it's really low quality. (Nobody uses JPEG entropy as a heuristic because edits can increase the value. Based on the entropy, WelcomeScan is the highest quality at 58%, and that's still very low.)

Most metadata. Fleurs has the most metadata because it was imported into Windows Photo Gallery and then manually labeled. Excluding the color profile information, the Flickr version has the most metadata because Photoshop Elements added in post-processing information. And if we evaluated based on metadata size, then the white border version has the most metadata because it includes C2PA information and that has a huge overhead.

C2PA provenance and C2PA timestamp. Only the white border picture has C2PA metadata. According the C2PA metadata, this picture came from Microsoft Designer on 2024-06-13. Keep in mind, Microsoft Designer came out around October 2022, so this is definitely not the original source for the picture. This file contains no other information about the image's provenance. Since we found variants of this picture dating back over a decade, we can be certain that the C2PA provenance information is incomplete and misleading.

(Again, I'm taking all of the metadata at face value. I'm ignoring the fact that the C2PA signed timestamp is trivial to alter and the signer's clock is likely wrong.)

There's only one sighting with C2PA metadata. But let's pretend that all of these picture have accurate C2PA provenance information that is consistent with the other metadata that exist in each file. We still don't have any consistency and don't know which one is the 'authentic' version. C2PA does not help here. As heuristics go, relying on C2PA is the worst option.
Heuristics are not authoritative. Analysts use heuristics to sift through large amounts of data, but it requires an understanding of the material, reviewing the content, and perseverance to get near the ground truth.

How to 'Unsuckify' Heuristics
When it comes to analysis, you need to use the clues you have. While none of these pictures identify the original source, these heuristics do provide great clues. For example:
Minimum age. When was it first seen? This sets a "not more recent than" time period. This may not be the absolute first appearance. In this case, FotoForensics says that this picture has been around since at least 2013. TinEye found their first sighting on 2008-02-04, and Flicker shows that it has existed since at least 2006. If we trust the metadata, then that further backdates the age to "at least 2004", making it 20 years old!

Popularity. Based on how many sightings there are, we can approximate whether it is widely known or viral. In this case, there are relatively few sightings. (A viral distribution would be like pictures from One Direction fans or followers of Kate, where the variants came in massive floods from a wide range of sources.) With a lot of digging, you might get lucky and find copies of the picture before it went viral.

Stock Imagery. Does it appear on any stock photo services? This doesn't prevent stock photo services from indexing viral or public images, or from providing false attribution. But it's a good starting point. Also, many stock sites identify when it was first available, further backdating the minimum age.

Alterations. Are there any differences or edits between versions? It's really subtle, but someone used a clone tool to remove a rectangular region from the top right corner of the WelcomeScan picture. If I had to guess, they removed an attribution. In the other variants, the colors have been altered. So at minimum, I know that the undiscovered original had slightly different coloring. And along one of the viral paths, someone added in an attribution and then it was later removed.

Context. Similar versions of the image may be on web pages that provide background information. In this case, many of the filenames are in foreign languages: Fleurs (French), Bienvenue (Spanish), and from the popular Bienvenue uploads, I've also seen names like "Digitalização de Boas-vindas" (Portugese), "Begrüßungsscan" (German), "Skenirana datoteka dobrodošlice" (Croatian), and more. Many of these names translate as "Welcome Scan". A few of the other common names for this file include "Test Picture" and "Garden" (both in a variety of languages). This tells me that the picture comes from something international.
Using the 'context' heuristic, I was able to find the ground truth. A quick Google search for the text "Welcome Scan" immediately identified a page at the Internet Archive: Welcome Scan and it also shows the picture!



The expert archivists at the Internet Archive recorded this information in 2019:
Publication date: 2004-04-09
Topics: windows, vista, fax, and, scan, fax and scan, garden, sample, pictures, sample pictures

Taken from Windows Vista's Fax and Scan program. On Windows 7 and beyond the image is replaced with the sample version, which is 1024x768.

The sample image is called Garden. The "high quality" version is called Welcome Scan.
The highest quality "Welcome Scan.jpg" version at the Internet Archive is 2154x1437 and 709,832 bytes (biggest dimensions and JPEG size found so far).
The picture has almost no informative metadata besides the abstract name "454054.TIF" and being saved by Adobe Photoshop 6.0 on a Mac. (Adobe Photoshop 6.0 was released in September 2000, so that works with a 2004 attribution date.)

It was last saved using Adobe Save-As quality 8 (equivalent to the JPEG standard at 80%).

The colors were still altered (made brighter), but not as much as the later viral version seen in Bienvenue and Facebook.

This highest quality version at the Internet Archive does not mention Getty or other stock photo services. However, that attribution is mentioned in their smaller "Garden.jpg" 1024x768 version.
Windows is international software and their "fax/scan" software explains the file name "Welcome Scan".

Additional Information
Evaluating pictures and tracking origins is often done on a case-by-case basis. Perceptual searches are one type of tool in the analyst's collection. Based on the results, we might be able to identify the art or artist (especially if the work is famous), possibly copyright information, or similar objects or products, such as clothing or furniture. (Bing Image Search is pretty good at this.) If the content is linked to a country or region, then that can further help identify the source.

However, perceptual searches are far from perfect. The accuracy depends on the search engine, amount of indexed media, and quality of the image. Very unique or unpopular images may have no search results. Additionally, there may be privacy concerns related to submitting an image to someone else's service.

While I am absolutely thrilled to see a larger variety of perceptual search systems becoming available, I'm concerned about the focus on making them do something that they just can't do. These search engines can find similar pictures from a known collection. They cannot tell you which one is more authentic.
Read more about Authentication, Forensics, FotoForensics, Image Analysis | Comments (3) | Direct Link

Comments

#1 Alex Chan (Homepage) on 2024-06-22 15:54 (Reply)

And even if you can get a timestamp from a social media site, you’re trusting that timestamp to be (1) correct and (2) mean what you think it means.

Taking Flickr as an example, it’s possible for a user to replace their photo after the initial upload. This preserves the original upload date and the fact that the image has changed is non-obvious. The upload date is technically correct because it says when a photo with that ID first appeared, but it doesn’t mean that’s when the current photo first appeared.

Here’s an example of a photo I replaced: https://www.flickr.com/photos/alexwlchan/53513831787

Here are some notes on how this appears in the Flickr API: https://alexwlchan.net/til/2024/what-happens-when-you-replace-a-photo/
#1.1 Dr. Neal Krawetz (Homepage) on 2024-06-22 16:45 (Reply)

Hi <PRESIDIO_ANONYMIZED_PERSON>,

First:
Wow! I didn't know that about Flickr! Excellent find.

Second:
You're absolutely right that trusting timestamps on remote web services can be misleading. For example:
- If you have a reason to think that someone has altered timestamps on servers, then it's clearly a bad assumption.
- If you think files were unpacked from some kind of archive, then the time zone may not be corrected.
- If you notice a significant drift from the clock (if the web server returns the current time), then it's definitely problematic.

However, (ironically) as heuristics go, taking server timestamps at face value is usually a good starting point.
#2 Jessamyn West (Homepage) on 2024-06-24 09:33 (Reply)

This is so helpful, seeing this all written out.

I run a small image-sharing site (MLTSHP) and often will see people who are interested in crediting the original image when they share but who also have limited resources/ability to track down original images in the vastness of the internet. When I've done similar searching, I often start with TinEye and Bing but I find that TinEye tends to find a lot of reshared Tumblr stuff which is next to useless for finding the original. And yes, as you've discussed neither size nor age is definitive for authority. As a library worker, I wish there was a way to nail down authority better.

Library and archive collections, which are usually authoritative at this current time, are often poorly-surfaced in general web search engines. And they do tend to have a lot of "unique or unpopular" images. I'd love to see some sort of search option--have yet to explore RootAbout or FotoForensics--which can weight certain sources higher or lower (Reddit is more likely to have better information about the image than Tumblr for example) which I think would lead to better outcomes.

Add Comment

Code of conduct
Name calling and anti-social comments will not be posted.
Comments must be related to the topic. Unrelated comments will not be posted. Make sure you are submitting your comment to the correct blog entry; Yes, people have submitted great comments to the wrong blog entries.
Comments should be rational and logical, citing findings as appropriate.
Opinions and speculations are desired and welcome, but if they are represented as fact then they may be moderated or censored.
The moderator reserves the right to end tangential discussions and censor offensive or inappropriate content.
Name	
Email	
Homepage	
In reply to	
[ Top level ]
Comment	

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
 	Remember Information?
Submitted comments will be reviewed by moderators before being displayed.
 	 
Copyright 2002-2024 Hacker Factor. All rights reserved., using simple but precise words, towards a person not having specifical knownlogy in this area.

your answer should use two languages, which are English and chinese on the two sides.