BERTopic_Multimodal

This is a BERTopic model. BERTopic is a flexible and modular topic modeling framework that allows for the generation of easily interpretable topics from large datasets.

This model was trained on 8000 images from Flickr without the captions. This demonstrates how BERTopic can be used for topic modeling using images as input only.

A few examples of generated topics:

Usage

To use this model, please install BERTopic:

pip install -U bertopic[vision]
pip install -U safetensors

You can use the model as follows:

from bertopic import BERTopic
topic_model = BERTopic.load("MaartenGr/BERTopic_Multimodal")

topic_model.get_topic_info()

You can view all information about a topic as follows:

topic_model.get_topic(topic_id, full=True)

Topic overview

• Number of topics: 29
• Number of training documents: 8091

Click here for an overview of all topics.

Topic ID	Topic Keywords	Topic Frequency	Label
-1	while - air - the - in - jumping	34	-1_while_air_the_in
0	bench - sitting - people - woman - street	1132	0_bench_sitting_people_woman
1	grass - running - dog - grassy - field	1693	1_grass_running_dog_grassy
2	boy - girl - little - young - holding	1290	2_boy_girl_little_young
3	dog - frisbee - running - water - mouth	1224	3_dog_frisbee_running_water
4	skateboard - ramp - doing - trick - cement	415	4_skateboard_ramp_doing_trick
5	snow - dog - covered - running - through	309	5_snow_dog_covered_running
6	mountain - range - slope - standing - person	205	6_mountain_range_slope_standing
7	pool - blue - boy - toy - water	189	7_pool_blue_boy_toy
8	trail - bike - down - riding - person	166	8_trail_bike_down_riding
9	snowboarder - mid - jump - air - after	126	9_snowboarder_mid_jump_air
10	rock - climbing - up - wall - tree	124	10_rock_climbing_up_wall
11	wave - surfboard - top - riding - of	112	11_wave_surfboard_top_riding
12	beach - surfboard - people - with - walking	102	12_beach_surfboard_people_with
13	jumping - track - horse - racquet - dog	98	13_jumping_track_horse_racquet
14	snowboard - snow - girl - hill - slope	95	14_snowboard_snow_girl_hill
15	game - being - football - played - professional	91	15_game_being_football_played
16	soccer - kicking - team - ball - player	80	16_soccer_kicking_team_ball
17	dirt - bike - person - rider - going	75	17_dirt_bike_person_rider
18	soccer - boys - field - ball - kicking	69	18_soccer_boys_field_ball
19	baseball - player - bat - swinging - into	63	19_baseball_player_bat_swinging
20	basketball - up - and - playing - jumping	59	20_basketball_up_and_playing
21	bird - body - flying - over - long	55	21_bird_body_flying_over
22	motorcycle - track - race - racer - racing	55	22_motorcycle_track_race_racer
23	boat - sitting - water - lake - hose	53	23_boat_sitting_water_lake
24	street - riding - down - bike - woman	52	24_street_riding_down_bike
25	paddle - suit - paddling - water - in	49	25_paddle_suit_paddling_water
26	pair - scissors - stage - white - shirt	42	26_pair_scissors_stage_white
27	tennis - court - racket - racquet - swinging	34	27_tennis_court_racket_racquet

Training Procedure

The data was retrieved as follows:

import os
import glob
import zipfile
import numpy as np
import pandas as pd
from tqdm import tqdm
from sentence_transformers import util

# Flickr 8k images
img_folder = 'photos/'
caps_folder = 'captions/'
if not os.path.exists(img_folder) or len(os.listdir(img_folder)) == 0:
    os.makedirs(img_folder, exist_ok=True)

    if not os.path.exists('Flickr8k_Dataset.zip'):   #Download dataset if does not exist
        util.http_get('https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_Dataset.zip', 'Flickr8k_Dataset.zip')
        util.http_get('https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_text.zip', 'Flickr8k_text.zip')

    for folder, file in [(img_folder, 'Flickr8k_Dataset.zip'), (caps_folder, 'Flickr8k_text.zip')]:
        with zipfile.ZipFile(file, 'r') as zf:
            for member in tqdm(zf.infolist(), desc='Extracting'):
                zf.extract(member, folder)
images = list(glob.glob('photos/Flicker8k_Dataset/*.jpg'))

Then, to perform topic modeling on multimodal data with BERTopic:

from bertopic import BERTopic
from bertopic.backend import MultiModalBackend
from bertopic.representation import VisualRepresentation, KeyBERTInspired

# Image embedding model
embedding_model = MultiModalBackend('clip-ViT-B-32', batch_size=32)

# Image to text representation model
representation_model = {
    "Visual_Aspect": VisualRepresentation(image_to_text_model="nlpconnect/vit-gpt2-image-captioning", image_squares=True),
    "KeyBERT": KeyBERTInspired()
}

# Train our model with images only
topic_model = BERTopic(representation_model=representation_model, verbose=True, embedding_model=embedding_model, min_topic_size=30)
topics, probs = topic_model.fit_transform(documents=None, images=images)

The above demonstrates that the input were only images. These images are clustered and from those clusters a small subset of representative images are extracted. The representative images are captioned using "nlpconnect/vit-gpt2-image-captioning" to generate a small textual dataset over which we can run c-TF-IDF and the additional KeyBERTInspired representation model.

Training hyperparameters

• calculate_probabilities: False
• language: None
• low_memory: False
• min_topic_size: 30
• n_gram_range: (1, 1)
• nr_topics: None
• seed_topic_list: None
• top_n_words: 10
• verbose: True

Framework versions

• Numpy: 1.23.5
• HDBSCAN: 0.8.29
• UMAP: 0.5.3
• Pandas: 1.5.3
• Scikit-Learn: 1.2.2
• Sentence-transformers: 2.2.2
• Transformers: 4.29.2
• Numba: 0.56.4
• Plotly: 5.14.1
• Python: 3.10.10