# Using Multimodal Foundation Models for Detecting Fake Images on the Internet with Explanations
## 3 Results and Discussion
### Removal-Based Explanations  
Another approach to understanding model explainability is the removal-based explanation approach. This approach involves systematically occluding varying regions of an image and observing how these changes reflect in the model's predictions. This approach can help identify the most and least influential regions of an image that contribute to the model's decision making. The steps for generating removal-based explanations are as follows:  
1. Define the prediction function. 2. Generate the baseline predictions for the image using the original prediction function. 3. Define a function to occlude regions in an image and create a list of regions to occlude. Change the region size and step size of the occlusion as needed. 4. For each region, measure the change in probabilities from the baseline prediction to the new prediction of the occluded image. 5. Output the most and least influential regions of the image. The exact formula for calculating the most and least influential regions of the image is simply using the change in the Euclidean distance between the array of baseline probabilities and the array of new probabilities. Euclidean distance allows us to consider the difference across all dimensions, or elements in the array. In addition, it allows the comparison of probability vectors in their entirety. The Euclidean distance can be used to effectively compare the relative importance of each feature and is an efficient computation. The format of the probability logits for the Llava 1.6 model is as follows when rounded to three significant figures:  
\[ \begin{bmatrix}5.628\times 10^{-9}&6.580\times 10^{-9}&5.843\times 10^{-7}&8.40 7\times 10^{-7}&8.424\times 10^{-7}&8.423\times 10^{-7}\\ 3.244\times 10^{-9}&2.710\times 10^{-9}&8.083\times 10^{-6}&1.673\times 10^{-5}&1.66 7\times 10^{-5}&1.676\times 10^{-5}\\ 7.354\times 10^{-9}&7.297\times 10^{-9}&3.910\times 10^{-5}&5.414\times 10^{-5}&5. 399\times 10^{-5}&5.411\times 10^{-5}\\ 3.539\times 10^{-12}&3.148\times 10^{-12}&6.416\times 10^{-4}&3.512\times 10^{-8}&3. 483\times 10^{-8}&3.512\times 10^{-8}\\ 7.207\times 10^{-11}&1.090\times 10^{-10}&4.333\times 10^{-4}&1.199\times 10^{-6}&1. 188\times 10^{-6}&1.196\times 10^{-6}\\ 6.364\times 10^{-11}&5.660\times 10^{-11}&2.718\times 10^{-5}&4.118\times 10^{-7}&4. 065\times 10^{-7}&4.085\times 10^{-7}\\ \end{bmatrix} \]  
The resulting array of all Euclidean distances generated by the repeated occlusion of portions of the image is shown in Table 3. \begin{table}
\begin{tabular}{c c} \hline
**Model** & **Jaccard Score** \\ \hline Llava & 0.0 \\ \hline CLIP & 0.0 \\ \hline Moondream2 & 0.758 \\ \hline Gemini & 0.0 \\ \hline \end{tabular}
\end{table} 
Table 2: Jaccard scores for different models. Since CLIP is a pure image classification model, the probability logits outputted are much simpler to understand. It is the probability that the image satisfies the first prompt followed by the probability that the image follows the second prompt. The format is as follows:  
Baseline Probabilities: [0.43507442, 0.5649257]  
Since the Euclidean distance calculations are carried out on an array of two numbers instead of several probability logits like with Llava, the calculations are much simpler and more efficient. The results are shown in Table 4.  
This explainability technique does have limitations. Firstly, the technique systematically occludes parts of the image based on region size and step size. This means that these metrics are crucial in generating the regions that are occluded, and the optimization of these metrics is key. Next, because of this, the most and least influential regions remain consistent, which means that we cannot use removal-based explanations to identify the effectiveness of zero- vs. n-shot prompting. The most/least influential regions will remain the same, similar to what can be realized with LIME.