SlowGuess/NeurIPS_2023_OCR huggingface dataset:
a photo of a car is parked on the porch of a house with a covered porch. The porch is parked in front of it
a photo of a bicycle is parked on the porch of a house with a covered porch. The porch is parked in front of it.
Predictions:  
patio (57.82%)  
mobile home (18.06%)  
picket fence (9.84%)  
sliding door (7.36%)  
rocking chair (1.21%)
Predictions:  
mobile home (29.76%)  
patlo (16.50%)  
picket fence (10.33%)  
gas pump (8.23%)  
bicycle-for-two (4.94%)
Edit object
a photo of a woman in black leotard
a photo of a group of a photo of a group ofchairs sitting on top chairs sitting on topof a cement floor next of a marble floorto a palm tree next to a palm tree
Predictions: balance bean (61.49%) horizontal bar (16.58%) pole (9.16%) parallel bars (8.77%) maillot (1.01%)
Predictions: dining table (29.20%) limousine (27.44%) patio (11.99%) home theater (5.29%) restaurant (3.43%)
Predictions: dining table (41.14%) tub (10.64%) patio (9.99%) washbaskin (7.18%) bathtub (3.62%)
a photo of a woman in a red jacket is pulling two dogs on a leash in the woods
Predictions: Eskimo dog (31.83%) dogsled (16.98%) malamute (12.72%) Siberian husky (5.94%) Saluki (3.34%)
Predictions: toy terrier (44.61%) Eskimo dog (12.72%) malamute (6.38%) Siberian husky (3.96%) French bulldog (2.77%)
a photo of a woman in a red jacket is pulling two dogs on a leash in the misty woods
a photo of a group of young men playing a game of ice hockey on an indoor ice rink
photo of a group of young men playing a game of ice hockey or an outdoor ice hockey
Predictions: puck (98.36%) scoreboard (0.92%) broom (0.16%) ballplayer (0.07%) digital clock (0.06%)
Predictions: puck (88.25%)ski (1.44%)scoreboard (1.43%)go-kart (0.72%)unicycle (0.41%)
a photo of a man holding a baseball bat on top of a lush green field
A photo of a man holding a baseballbat on top of a lush green soccer field
Predictions: ballplayer (96.95%) soccer ball (92.90%) baseball (2.18%) ballplayer (6.17%) football helmet (0.32%) baseball (0.58%) plunger (0.17%) football helmet (0.12%) croquet ball (0.10%) golf ball (0.10%)
a photo of a man riding a dog sled pulled by two dogs on a snow covered field with trees in the background
a drawing of a man riding a dog sled pulled by two dogs or a snow covered field with trees in the background
Predictions: dogsled (99.32%) Eskimo dog (0.46%) Siberian husky (0.17%) malamute (0.03%) timber wolf (0.00%)
Predictions: dogsled (99.11%) Eskimo dog (0.31%) Siberian husky (0.19%) malamute (0.12%) horse cart (0.08%)
a photo of a woman standing in the snow with skis on her feet and a look at her head
a painting of a woman standing in the snow with skirts and a hat on her head
Predictions:  
crutch (82.80%)  
sktf (16.11%)  
pole (0.32%)  
snorkel (0.11%)  
parallel bars (0.08%)
Predictions: skl (99.64%) pole (0.09%) crutch (0.08%) shovel (0.04%) alp (0.03%)
a photo of a pair of sunglasses and a stuffed animal
a drawing of a pair of sunglasses and a stuffed animal
Predictions:  
Pencil box (25.39%)  
Purse (7.92%)  
Stethoscope (7.92%)  
Hair slide (6.73%)  
Sunglasses (4.85%)
Predictions:  
Hair slide (9.92%)  
Spindle (8.34%)  
Pencil box (8.02%)  
Thimble (6.01%)  
Wooden spoon (5.81%)
a photo of a man insunglasses andahoodie takinga selfie in the
a photo of a man insunglasses andafour hoodie taking aselfie in the
Predictions: sunglasses (52,59%) sunglass (27.44%) alp (3.11%) sweatshirt (2.54%) sunscreen (1.79%)
Predictions: sunglasses (44.52%) sunglass (24.31%) alp (6.57%) valley (1.83%) ski (1.70%)
a photo of a seat beltand a cell phone
a photo of a penguin and a cell phone
Predictions: seat belt (97.08%) buckle (0.82%) digital watch (0.50%) scabbard (0.39%) cellular telephone (0.30%)
Figure 10: We visualize the counterfactual images generated by LANCE for images from the HardImageNet [62] dataset. Each row corresponds to a specific edit type. Above each image, we display its generated caption, highlighting the original and edited word(s). Below each image, we display the top-5 classes predicted by a ResNet-50 model and the associated model confidence, coloring the ground truth class in green.
a photo of a man with his black eyes and green sunglasses and an
Predictions:  
sunglass (41.61%)  
binoculars (16.89%)  
sunglasses (11.79%)  
web site (7.41%)  
bubble (1.57%)
a photo of a man with a pair of sunglasses and an
Predictions: sunglass (55.14%) sunglasses (23.37%) binoculars (11.75%) web site (2.42%) cowboy hat (0.79%)
Figure 11: Beyond conventional diagnosis. LANCE can uncover nuanced model bias, e.g. predictive confidence for "dog sled" drops significantly even if the dog breed is changed to other popular sled dog breeds.
Table 5: Prompts used for programmatically collecting perturbations of captions using GPT-3.5 turbo [18]. We use the same prompts without examples as instructions when instruction fine-tuning LLAMA-7B [53] with LoRA [54]
Perturbation type	Prompt
Edit subject	Generate all possible variations by changing only the subject of the provided sentence. For example: Change "A man walking a dog" to "A woman walking a dog". <caption>
Edit object	In English grammar, the subject is the person, thing, or idea that performs the action of the verb in a sentence, while the object is the person, thing, or idea that receives the action of the verb. Generate all possible variations by changing only the object of the provided sentence. For example: Change "A man walking a dog" to "A man walking a horse". <caption>
Edit background	[leftmargin=*]Generate all possible variations of the provided sentence by changing or adding background or location details without altering the foreground or its attributes. For example: Change "A man walking a dog" and "A man walking a dog with mountains in the background", or change "A man walking a dog on grass". Change "A man walking a dog on the road" to "A man walking a dog with mountains by the beach". <caption>Generate all possible variations of the provided sentence by only changing the weather conditions, or adding a description of the weather if not already present. For example: change "A man walking a dog" to "A man walking a dog in the rain", and change "A man walking a dog in the rain" to "A man walking a dog in the snow". <caption>
Edit domain	Generate a few variations by only changing the data domain of the provided sentence without changing the content. For example: Change "A photo of a man" to "A painting of a man" or "A sketch of a man". A photo of <caption>
Edit adjective	Generate all possible variations of the provided sentence by only adding or altering a single adjective or attribute. For example: change "A man walking a brown dog" to "A tall man walking a brown dog", "A man walking a black dog" or "A man walking a cheerful brown dog". <caption>
Figure 12: We visualize the different modes of failure of LANCE for genenerating counterfactuals on the HardImageNet dataset. Each row corresponds to a specific edit type. Above each image, we display its generated caption, highlighting the original and edited word(s). Below each image, we display the top-5 classes predicted by a ResNet-50 model and the associated model confidence, coloring the ground truth class in green.
Figure 13: Human study. A screenshot of the interface we deploy to evaluate the quality of images generated by LANCE.
Table 6: Hyperparameter values used for caption (top left), LLAMA finetuning (top right) and image editing (bottom).
config	value	config	value
learning rate	3e-4
effective batch size	128
grad. accumulation steps	32
weight decay	0.0
max. sequence len	1024
lora rank [54]	8
lora α [54]	16
lora dropout [54]	0.05
lora weights [54]	qkv
(a) Caption Generation and Clustering
(b) LLAMA Finetuning
config	value
Stable Diffusion [16] version	1.4
image resolution	512x512
LR scheduler	DDIM scheduler [60]
beta_start [60]	0.00085
beta_end [60]	0.012
beta_schedule [60]	scaled linear
diffusion steps [19]	50
attention_replace_edit [19]	2
cross_replace_steps [19]	0.8
self_replace_steps [19]	[0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
guidance scale	7.5
τ [42]	0.2
img_sim. threshold [42]	0.7
img-text sim. threshold [42]	0.2
(c) Image Editing