arman-chopikyan/generated-image-quality-assessment

Report: Generated Image Quality Assessment Model

1. Introduction

This report details the development of a model for evaluating the quality of AI-generated human images. The evaluation criteria include:

• Relevance to the text prompt
• Visual quality and aesthetics
• Presence of artifacts and distortions
• Overall Quality

The workflow consists of three key stages:

1. Dataset Preparation using ChatGPT VLM and CLIP embeddings.
2. Model Training of distilled model leveraging CLIP features.
3. Evaluation of model performance.

---

2. Dataset Preparation (Dataset.ipynb)

Due to cost and resource constraints, only 10,000 images were sourced from DiffusionDB, filtered to retain only human-related images. After removing invalid or problematic images, the final dataset size was 9,432.

Data Processing Workflow

1. Filtering Prompts: Extracting images with human-related keywords (e.g., "man", "woman", "child").
2. Downloading Images: Retrieving images from DiffusionDB based on filtered metadata.
3. Generating Annotations Using ChatGPT VLM:
   • GPT-4o Mini was used to assess:
     • Relevance to the prompt (0.0 - 1.0 scale)
     • Visual quality (composition, clarity, aesthetics)
     • Presence of artifacts (missing parts, distortions, unnatural anatomy). Important note: 1 indicates artifact presence, 0 indicates absence. This approach was chosen as detecting artifact presence is significantly easier for the network compared to detecting their absence.
4. Generating CLIP Embeddings:
   • Image and text embeddings were extracted using CLIP ViT-B/32.
5. Dataset Splitting:
   • 80% training, 15% validation, and 5% test split.

Dataset Analysis

A histogram-based analysis was performed to examine score distributions. The dataset exhibited high imbalance, with some quality scores appearing far more frequently than others. To address this, weighted binary cross-entropy was employed during training.

The final dataset used in this study can be accessed at:

• Dataset (data.pkl): See here

---

3. Model Training (Training.ipynb)

A lightweight regression model was trained using CLIP embeddings to predict quality scores for new images.

Model Architecture

• Input: Concatenated CLIP image and text embeddings.
• Output: Predictions for four quality metrics.

Network Structure:

Linear(1024, 256) → ReLU
Linear(256, 4) → Sigmoid

Training Setup

• Optimizer: Adam (learning rate = 1e-4)
• Loss Function: Weighted Binary Cross-Entropy (compensating for class imbalance)
• Epochs: 100 (early stopping applied)
• Batch Size: 512

---

4. Evaluation (Evaluation.ipynb)

The trained model was tested on a held-out test dataset, and its performance was analyzed across multiple metrics.

Model Performance Metrics

The evaluation was conducted using multiple metrics to assess accuracy, precision, recall, and F1-score across different categories.

Training Results

Metric	Overall
Accuracy	0.8326
Precision	0.9930
Recall	0.9576
F1 Score	0.9746

Per-Category Metrics:

Category	Accuracy	Precision	Recall	F1 Score
Relevance	0.9633	0.9887	0.9712	0.9799
Visual Quality	0.9895	0.9990	0.9903	0.9947
Artifacts	0.8860	0.9895	0.8881	0.9360
Final Probability	0.9765	0.9948	0.9808	0.9878

Validation Results

Metric	Overall
Accuracy	0.7603
Precision	0.9699
Recall	0.9466
F1 Score	0.9579

Per-Category Metrics:

Category	Accuracy	Precision	Recall	F1 Score
Relevance	0.9151	0.9553	0.9508	0.9530
Visual Quality	0.9682	0.9883	0.9790	0.9836
Artifacts	0.8571	0.9607	0.8840	0.9208
Final Probability	0.9505	0.9755	0.9726	0.9740

Test Results

Overall Metrics

Metric	Value
Accuracy	0.7611
Precision	0.9741
Recall	0.9480
F1 Score	0.9606

Per-Category Metrics

Category	Accuracy	Precision	Recall	F1 Score
Relevance	0.9281	0.9652	0.9563	0.9607
Visual Quality	0.9767	0.9913	0.9848	0.9881
Artifacts	0.8478	0.9577	0.8750	0.9145
Final Probability	0.9598	0.9822	0.9757	0.9790

The model corresponding to the reported results can be accessed at:

• Best Model (model.pth): See here

---

5. Conclusion

The model achieved promising results given the imposed constraints on cost, time and computational resources. Despite these limitations, the evaluation metrics indicate that the approach is effective for assessing AI-generated human images.

Constraints and Future Improvements

• GPT-4o Mini was chosen over GPT-4o due to its significantly lower cost and faster processing speed. However, GPT-4o is a much more powerful model and could improve results significantly if used.
• The dataset size was capped at 10,000 images due to computational and financial constraints. A larger dataset would likely enhance performance.
• A lightweight head network was selected to balance computational feasibility with effective learning. A more complex model could further improve results but requires more computational power.

By addressing these areas, the model can achieve significantly better results in real-world applications.