CogCoM: Train Large Vision-Language Models Diving into Details  through Chain of Manipulations

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 10:09:20

Abstract: Vision-Language Models (VLMs) have demonstrated their broad effectiveness thanks to extensive training in aligning visual instructions to responses. However, such training of conclusive alignment leads models to ignore essential visual reasoning, further resulting in failures in meticulous visual problems and unfaithful responses. Drawing inspiration from human cognition in solving visual problems (e.g., marking, zoom in), this paper introduces Chain of Manipulations, a mechanism that enables VLMs to solve problems step-by-step with evidence. After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) and results (e.g., boxes, image) actively without involving external tools, while also allowing users to trace error causes. We study the roadmap to implement this mechanism, including (1) a flexible design of manipulations upon extensive analysis, (2) an efficient automated data generation pipeline, (3) a compatible VLM architecture capable of multi-turn multi-image, and (4) a model training process for versatile capabilities.  With the design, we also manually annotate 6K high-quality samples for the challenging graphical mathematical problems. Our trained model, CogCoM, equipped with this mechanism with 17B parameters achieves state-of-the-art performance across 9 benchmarks from 4 categories, demonstrating the effectiveness while preserving the interpretability. Our code, model weights, and collected data will be publicly available.

Main paper: 
 terminology: 
 We first conduct pilot experiments to investigate the possible manipulations capable of handling diverse visual problems. Using Grounding(the pillar in front of the man at ) to find .., the resultant region is .Using CropZoomIn( , 4) to crop it on , and zoom in it by 4 times to get new image .Using Grounding( the man in black shirt ) to locate .., the position is at .Using OCR( ) recognize the text written on the image , the result is "QUICK DEPOSIT". Using Grounding(the text written on the pillar in ) to find .., the resultant region is . Q v 3 Q j 5 n 9 1 q h a J P h q 6 B k E 1 p R p R 1 Q W l S 6 G 7 o m 5 u / q h K k k N K m I p 7 l M 8 o D r R y 1 G d T a 3 J d u + q t p / M f m q l Q t Q 9 K b o F P d U s a 8 G i K 5 v i g X b N s x 7 J P 6 9 X W Q T n q O W x i C z s 0 z 3 2 0 c I Q T u O Q t 8 I w X v B r H R m r c G 4 N v q j F R< l a t e x i t s h a 1 _ b a s e 6 4 = " l u F z t v g 4 p k m U x M B 0 1 y K J s 5 v i i r E = " > A A A C z n i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b l y 3 Y B 9 Q i k + m 0 D e b F Z F I o p b j 1 B 9 z q Z 4 l / o H / h n T E F t Y h O S H L m 3 H P u z L 3 X S w I / V Y 7 z W r A W F p e W V 4 q r a + s b m 1 v b p Z 3 d Z h p n k o s G j 4 N Y t j 2 W i s C P R E P 5 K h D t R A o W e o F o e X e X O t 4 a C Z n 6 c X S t x o n o h m w Q + X 2 f M 0 V U 5 y Z k a s h Z M K l P b 0 t l p + K Y Z c 8 D N w d l 5 K s W l 1 5 w g x 5 i c G Q I I R B B E Q 7 A k N L T g Q s H C X F d T I i T h H w T F 5 h i j b w Z q Q Q p G L F 3 9 B 3 Q r p O z E e 1 1 z t S 4 O Z 0 S 0 C v J a e O Q P D H p J G F 9 m m 3 i m c m s 2 d 9 y T 0 x O f b c x / b 0 8 V 0 i s w p D Y v 3 w z 5 X 9 9 u h a F P s 5 N D T 7 V l B h G V 8 f z L J n p i r 6 5 / a U q R R k S 4 j T u U V w S 5 s Y 5 6 7 N t P K m p X f e W m f i b U W p W 7 3 m u z f C u b 0 k D d n + O c x 4 0 j y v u a c W t n 5 S r F / m o i 9 j H A Y 5 o n m e o 4 g o 1 N E z H H / G E Z 6 t m j a y p d f 8 p t Q q 5 Z w / f l v X w A Y 5 C k + 4 = < / l a t e x i t > Q < l a t e x i t s h a 1 _ b a s e 6 4 = " L z 9 T 7 0 L Z H 4 j 8 t / W o t J t O L W 8 0 C n 4 = " > A A A C z n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 1 4 7 K C r Y W 2 y G Q 6 r c G 8 m E w K p R S 3 / o B b / S z x D / Q v v D O m o B b R C U n O n H v O n b n 3 e k n g p 8 p x X g v W 3 P z C 4 l J x u b S y u r a + U d 7 c a q Z x J r l o 8 D i I Z c t j q Q j 8 S D S U r w L R S q R g o R e I a + / u X M e v h 0 K m f h x d q V E i u i E b R H 7 f 5 0 w R 1 e 6 E T N 1 y F o x P J z f l i l N 1 z L J n g Z u D C v J V j 8 s v 6 K C H G B w Z Q g h E U I Q D M K T 0 t O H C Q U J c F 2 P i J C H f x A U m K J E 3 I 5 U g B S P 2 j r 4 D 2 r V z N q K 9 z p k a N 6 d T A n o l O W 3 s k S c m n S S s T 7 N N P D O Z N f t b 7 r H J q e 8 2 o r + X 5 w q J V b g l 9 i / f V P l f n 6 5 F o Y 8 T U 4 N P N S W G 0 d X x P E t m u q J v b n + p S l G G h D i N e x S X h L l x T v t s G 0 9 q a t e 9 Z S b + Z p S a 1 X u e a z O 8 6 1 v S g N 2 f 4 5 w F z Y O q e 1 R 1 L w 8 r t b N 8 1 E X s Y B f 7 N M 9 j 1 H C B O h q m 4 4 9 4 w r N V t 4 b W x L r / l F q F 3 L O N b 8 t 6 + A B o M p P e < / l a t e x i t > A < l a t e x i t s h a 1 _ b a s e 6 4 = " L z 9 T 7 0 L Z H 4 j 8 t / W o t J t O L W 8 0 C n 4 = " > A A A C z n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 1 4 7 K C r Y W 2 y G Q 6 r c G 8 m E w K p R S 3 / o B b / S z x D / Q v v D O m o B b R C U n O n H v O n b n 3 e k n g p 8 p x X g v W 3 P z C 4 l J x u b S y u r a + U d 7 c a q Z x J r l o 8 D i I Z c t j q Q j 8 S D S U r w L R S q R g o R e I a + / u X M e v h 0 K m f h x d q V E i u i E b R H 7 f 5 0 w R 1 e 6 E T N 1 y F o x P J z f l i l N 1 z L J n g Z u D C v J V j 8 s v 6 K C H G B w Z Q g h E U I Q D M K T 0 t O H C Q U J c F 2 P i J C H f x A U m K J E 3 I 5 U g B S P 2 j r 4 D 2 r V z N q K 9 z p k a N 6 d T A n o l O W 3 s k S c m n S S s T 7 N N P D O Z N f t b 7 r H J q e 8 2 o r + X 5 w q J V b g l 9 i / f V P l f n 6 5 F o Y 8 T U 4 N P N S W G 0 d X x P E t m u q J v b n + p S l G G h D i N e x S XZ q Q = " > A A A C z n i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 0 a o Y G u h L T K Z T m s w L y a T Q i n F r T / g V j 9 L / A P 9 C + + M K a h F d E K S M + e e c 2 f u v V 4 S + K l y n N e C N T e / s L h U X C 6 t r K 6 t b 5 Q 3 t 5 p p n E k u G j w O Y t n y W C o C P x I N 5 a t A t B I p W O g F 4 t q 7 O 9 P x 6 6 G Q q R 9 H V 2 q U i G 7 I B p H f 9 z l T R L U 7 I V O 3 n A X j i 8 l N u e J U H b P s W e D m o I J 8 1 e P y C z r o I Q Z H h h A C E R T h A A w p P W 2 4 c J A Q 1 8 W Y O E n I N 3 G B C U r k z U g l S M G I v a P vt i p h 4 Q W X 6 C A E R w o f A g E k Y Q 8 M C T 1 N 2 L A Q E d f C k L i Y k K v j A i M s k D a l L E E Z j N g r + v Z o 1 8 z Y g P b K M 9 F q T q d 4 9 M a k N L F N m p D y Y s L q N F P H U + 2 s 2 N + 8 h 9 p T 3 W 1 A f y f z 8 o m V 6 B P 7 l 2 6 c + V + d q k W i i 0 N d g 0 s 1 R Z p R 1 f H M J d V d U T c 3 v 1 Q l y S E i T u E O x W P C X C v H f T a 1 J t G 1 q 9 4B E t i p h 4 Q W X 6 C A E R w o f A g E k Y Q 8 M C T 1 N 2 L A Q E d f C k L i Y k K v j A i M s k D a l L E E Z j N g r + v Z o 1 8 z Y g P b K M 9 F q T q d 4 9 M a k N L F N m p D y Y s L q N F P H U + 2 s 2 N + 8 h 9 p T 3 W 1 A f y f z 8 o m V 6 B P 7 l 2 6 c + V + d q k W i i 0 N d g 0 s 1 R Z p R 1 f H M J d V d U T c 3 v 1 Q l y S E i T u E O x W P C X C v H f T a 1 J t G 1 q 9 4 y H X / T m Y p V e 5 7 l p n h X t 6 Q B 2 z / H O Q l q u y V 7 v 2 S f 7 R X L x 9 m o 8 9 j E F n Z o n g c o 4 x Q V V M n 7 G+ Y h l W q l o u W b h q l 4 4 p Z L u u W Y a r 1 F R S Q r k a Q f 8 E V B g j A k G A M D h + C Y g 8 2 Y n q 6 s G A i J K y H K W E RR a 7 K c 9 w j R 9 q E W J w Y N q E 3 9 B 3 R r p u i P u 2 l Z 6 z U j E 7 x 6 I 1 I q e O A N A H x I o r l a b r K J 8 p Z o r 9 5 T 5 W n v N u E / k 7 q N S Z U 4 J r Q v 3 Q z 5 n 9 1 s h a B I a q q B p d q C h U i q 2 O p S 6 K 6 I m + u f 6 t K k E N I m I w H l I 8 o Z k o 5 6 7 O u N L G q X f b W V v k 3 x Z S o 3 L O U m + B d 3 p I G P D f O + a B V N K y K Y Z 2 X C r W T d N R Z 7 G E f h z T P I 9 R Q R w N N 8 h 7 h E U 9 4 1 u q a r y X a 3 S d V y 6 S a X f x Y 2 s M H F m y Q K w = = <+ Y h l W q l o u W b h q l 4 4 p Z L u u W Y a r 1 F R S Q r k a Q f 8 E V B g j A k G A M D h + C Y g 8 2 Y n q 6 s G A i J K y H K W E RR a 7 K c 9 w j R 9 q E W J w Y N q E 3 9 B 3 R r p u i P u 2 l Z 6 z U j E 7 x 6 I 1 I q e O A N A H x I o r l a b r K J 8 p Z o r 9 5 T 5 W n v N u E / k 7 q N S Z U 4 J r Q v 3 Q z 5 n 9 1 s h a B I a q q B p d q C h U i q 2 O p S 6 K 6 I m + u f 6 t K k E N I m I w H l I 8 o Z k o 5 6 7 O u N L G q X f b W V v k 3 x Z S o 3 L O U m + B d 3 p I G P D f O + a B V N K y K Y Z 2 X C r W T d N R Z 7 G E f h z T P I 9 R Q R w N N 8 h 7 h E U 9 4 1 u q a r y X a 3 S d V y 6 S a X f x Y 2 s M H F m y Q K w = = <+ Y h l W q l o u W b h q l 4 4 p Z L u u W Y a r 1 F R S Q r k a Q f 8 E V B g j A k G A M D h + C Y g 8 2 Y n q 6 s G A i J K y H K W E RR a 7 K c 9 w j R 9 q E W J w Y N q E 3 9 B 3 R r p u i P u 2 l Z 6 z U j E 7 x 6 I 1 I q e O A N A H x I o r l a b r K J 8 p Z o r 9 5 T 5 W n v N u E / k 7 q N S Z U 4 J r Q v 3 Q z 5 n 9 1 s h a B I a q q B p d q C h U i q 2 O p S 6 K 6 I m + u f 6t K k E N I m I w H l I 8 o Z k o 5 6 7 O u N L G q X f b W V v k 3 x Z S o 3 L O U m + B d 3 p I G P D f O + a B V N K y K Y Z 2 X C r W T d N R Z 7 G E f h z T P I 9 R Q R w N N 8 h 7 h E U 9 4 1 u q a r y X a 3 S d V y 6 S a X f x Y 2 s M H F m y Q K w = = < / l a t e x i t >< l a t e x i t s h a 1 _ b a s e 6 4 = " n q u Y d 0 j x d z S E a F 8 B N a u 6 C P w x d h g = " > A A A C y H i c j V H L S s N A F D 3 G 9 7 v q 0 k 2 w C K 7 C p N b Y 7 o p u x J W C q Y K W k q R T H c y L Z K K W 4 s Y f c K t f J v 6 B / o V 3 x h R 0 U X R C k j v n n n N m 7 r 1 + G o p c M v Y + Y U x O T c / M z s 0 v L C 4 t r 6 x W 1 t b b e V J k A X e D J E y y C 9 / L e S h i 7 k o h Q 3 6 R Z t y L / J C f + 7 e H K n 9 + x 7 N c J P G Z H K S 8 E 3 n X s e i L w J M E u b 7 / 0 L W 7 l S q z m k 1 W t x 2 T W X u M 1 Z w m B W y 3 1 n A c 0 7 a Y X l W U 6 y S p v O E K P S Q I U C A C R w x J c Q g P O T 2 X s M G Q E t b B k L C M I q H z H I 9 Y I G 1 B L E 4 M j 9 B b + l 7 T 7 r J E Y 9 o r z 1 y r A z o l p D c j p Y l t 0 i T E y y h W p 5 k 6 X 2 h n h Y 7 z H m p P d b c B / f 3 S K y J U 4 o b Q v 3 Q j 5 n 9 1 q h a J P h q 6 B k E 1 p R p R 1 Q W l S 6 G 7 o m 5 u / q h K k k N K m I p 7 l M 8 o D r R y 1 G d T a 3 J d u + q t p / M f m q l Q t Q 9 K b o F P d U s a 8 G i K 5 v i g X b N s x 7 J P 6 9 X W Q T n q O W x i C z s 0 z 3 2 0 c I Q T u O Q t 8 I w X v B r H R m r c G 4 N v q j F R= " > A A A C y H i c j V H L S s N A F D 3 G 9 7 v q 0 k 2 w C K 7 C p N b Y 7 o p u x J W C q Y K W k q R T H c y L Z K K W 4 s Y f c K t f J v 6 B / o V 3 x h R 0 U X R C k j v n n n N m 7 r 1 + G o p c M v Y + Y U x O T c / M z s 0 v L C 4 t r 6 x W 1 t b b e V J k A X e D J E y y C 9 / L e S h i 7 k o h Q 3 6 R Z t y L / J C f + 7 e H K n 9 + x 7 N c J P G Z H K S 8 E 3 n X s e i L w J M E u b 7 / 0 L W 7 l S q z m k 1 W t x 2 T W X u M 1 Z w m B W y 3 1 n A c 0 7 a Y X l W U 6 y S p v O E K P S Q I U C A C R w x J c Q g P O T 2 X s M G Q E t b B k L C M I q H z H I 9 Y I G 1 B L E 4 M j 9 B b + l 7 T 7 r J E Y 9 o r z 1 y r A z o l p D c j p Y l t 0 i T E y y h W p 5 k 6 X 2 h n h Y 7 z H m p P d b c B / f 3 S K y J U 4 o b Q v 3 Q j 5 n 9 1 q h a J P h q 6 B k E 1 p R p R 1 Q W l S 6 G 7 o m 5 u / q h K k k N K m I p 7 l M 8 o D r R y 1 G d T a 3 J d u + q t p / M f m q l Q t Q 9 K b o F P d U s a 8 G i K 5 v i g X b N s x 7 J P 6 9 X W Q T n q O W x i C z s 0 z 3 2 0 c I Q T u O Q t 8 I w X v B r H R m r c G 4 N v q j FR a j b w a x l P X 9 8 y k U s = < / l a t e x i t > bbx1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 W 6 2 Q + S k A v d K 1 N j l q F l 1 6 k U P 5 h Q = " > A A A C y H i c j V H L T s J A F D 3 U F + I L d e m m k Z i 4 I k N F l B 3 R j X G F i T w S J K Q t A 0 4 o b d N O V U L c + A N u 9 c u M f 6 B / 4 Z 2 x J L o g O k 3 b M + f e c 2 b u v U 7 o i V g y 9 p 4 x F h a X l l e y q 7 m 1 9 Y 3 N r f z 2 T j M O k s j l D T f w g q j t 2 D H 3 h M 8 b U k i P t 8 O I 2 2 P H 4 y 1 n d K 7 i r T s e x S L w r + U k 5 N 2 x P f T F Q L i 2 J K r h O A 8 9 q 5 c v s G K l y q r W k U n g i F m V M o H q c Z l Z z C w V m V 4 F p Kv l f n a p F Y o B T X Y O g m k L N q O r c 1 C X R X V E 3 N 3 9 U J c k h J E 7 h P s U j w q 5 W z v p s a k 2 s a 1 e 9 t X X 8 Q 2 c q V u 3 d N D f B p 7 o l D X g 2 R X M + a F r F U q V Y u i o X a m f p q L P Y w z 4 O a Z 4 n q O E C d T T I W + A Z L 3 g 1 L o 3 Q u D c m 3 6 l G J t X s 4 t c y n r 4 A 1 C u R R g = = < / l a t e x i t > bbx2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 V C l O b I t b F + / n 0 C n X Z n O e b L 3 S y o = " > A A A C y H i c j V H L T s J A F D 3 U F + I L d e m m k Z i 4 a l o E x B 3 R j X G F i Y A J E t K W A S e U t m m n K i F u / A G 3 + m X G P 9 C / 8 M 5 Y j I Y Y n a b t n X P P O T P 3 X i f 0 e C x M 8 z W j z c 0 v L C 5 l l 3 M r q 2 v r G / n N r W Y c J J H L G m 7 g B d G l Y 8 f M 4 z 5 r C C 4 8 d h l G z B 4 5 H m s 5 w x O Z b 9 2 w K O a B f y H G I e u M 7 I H P + 9 y 1 B U E N x 7 n r H n T z B d O w S t V y 0 d J N o 3 R U M c t l 3 T J M t b 6 C A t J V D / I v u E I P A V w k G I H B h 6 D Y g 4 2 Y n j Y s m A g J 6 2 B C W E Q R V 3 m G e + R I m x C L E c M m d E j f A e 3 a K e r T X n r G S u 3 S K R 6 9 E S l 1 7 J E m I F 5 E s T x N V / l E O U v 0 N + + J 8 p R 3 G 9 P f S b 1 G h A p c E / q X b s r 8 r 0 7 W I t B H V d X A q a Z Q I b I 6 N 3 V J V F f k z f V v V Q l y C A m T c Y / y E c W u U k 7 7 r C t N r G q X v b V V / k 0 x J S r 3 b s p N 8 C 5 v S Q O e G e d s 0 C w a V s W w z k u F 2 n E 6 6 i x 2 s I t 9 m u c h a j h F H Q 3 y 5 n j E E 5 6 1 M y 3 U b r X x J 1 X L p J p t / F j a w w e r A J E 0 < / l a t e x i t > bbx3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 W 6 2 Q + S k A v d K 1 N j l q F l 1 6 k U P 5 h Q = " > A A A C y H i c j V H L T s J A F D 3 U F + I L d e m m k Z i 4 I k N F l B 3 R j X G F i T w S J K Q t A 0 4 o b d N O V U L c + A N u 9 c u M f 6 B / 4 Z 2 x J L o g O k 3 b M + f e c 2 b u v U 7 o i V g y 9 p 4 x F h a X l l e y q 7 m 1 9 Y 3 N r f z 2 T j M O k s j l D T f w g q j t 2 D H 3 h M 8 b U k i P t 8 O I 2 2 P H 4 y 1 n d K 7 i r T s e x S L w r + U k 5 N 2 x P f T F Q L i 2 J K r h O A 8 9 q 5 c v s G K l y q r W k U n g i F m V M o H q c Z l Z z C w V m V 4 F p K s e 5 N 9 w g z 4 C u E g w B o c P S d i D j Z i e D k p g C I n r Y k p c R E j o O M c j c q R N K I t T h k 3 s i L 5 D 2 n V S 1 q e 9 8 o y 1 2 q V T P H o j U p o 4 I E 1 A e R F h d Z q p 4 4 l 2 V u w 8 7 6 n 2 V H e b 0 N 9 J v c b E S t w S + 5 d u l v l f n a p F Y o B T X Y O g m k L N q O r c 1 C X R X V E 3 N 3 9 U J c k h J E 7 h P s U j w q 5 W z v p s a k 2 s a 1 e 9 t X X 8 Q 2 c q V u 3 d N D f B p 7 o l D X g 2 R X M + a F r F U q V Y u i o X a m f p q L P Y w z 4 O a Z 4 n q O E C d T T I W + A Z L 3 g 1 L o 3 Q u D c m 3 6 l G J t X s 4 t c y n r 4 A 1 C u R R g = = < / l a t e x i t > bbx2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 V C l O b I t b F + / n 0 C n X Z n O e b L 3 S y o = " > A A A C y H i c j V H L T s J A F D 3 U F + I L d e m m k Z i 4 a l o E x B 3 R j X G F i Y A J E t K W A S e U t m m n K i F u / A G 3 + m X G P 9 C / 8 M 5 Y j I Y Y n a b t n X P P O T P 3 X i f 0 e C x M 8 z W j z c 0 v L C 5 l l 3 M r q 2 v r G / n N r W Y c J J H L G m 7 g B d G l Y 8 f M 4 z 5 r C C 4 8 d h l G z B 4 5 H m s 5 w x O Z b 9 2 w K O a B f y H G I e u M 7 I H P + 9 y 1 B U E N x 7 n r H n T z B d O w S t V y 0 d J N o 3 R U M c t l 3 T J M t b 6 C A t J V D / I v u E I P A V w k G I H B h 6 D Y g 4 2 Y n j Y s m A g J 6 2 B C W E Q R V 3 m G e + R I m x C L E c M m d E j f A e 3 a K e r T X n r G S u 3 S K R 6 9 E S l 1 7 J E m I F 5 E s T x N V / l E O U v 0 N + + J 8 p R 3 G 9 P f S b 1 G h A p c E / q X b s r 8 r 0 7 W I t B H V d X A q a Z Q I b I 6 N 3 V J V F f k z f V v V Q l y C A m T c Y / y E c W u U k 7 7 r C t N r G q X v b V V / k 0 x J S r 3 b s p N 8 C 5 v S Q O e G e d s 0 C w a V s W w z k u F 2 n E 6 6 i x 2 s I t 9 m u c h a j h F H Q 3 y 5 n j E E 5 6 1 M y 3 U b r X x J 1 X L p J p t / F j a w w e r A J E 0 < / l a t e x i t > bbx3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 7 V C l O b I t b F + / n 0 C n X Z n O e b L 3 S y o = " > A A A C y H i c j V H L T s J A F D 3 U F + I L d e m m k Z i 4 a l o E x B 3 R j X G F i Y A J E t K W A S e U t m m n K i F u / A G 3 + m X G P 9 C / 8 M 5 Y j I Y Y n a b t n X P P O T P 3 X i f 0 e C x M 8 z W j z c 0 v L C 5 l l 3 M r q 2 v r G / n N r W Y c J J H L G m 7 g B d G l Y 8 f M 4 z 5 r C C 4 8 d h l G z B 4 5 H m s 5 w x O Z b 9 2 w K O a B f y H G I e u M 7 I H P + 9 y 1 B U E N x 7 n r H n T z B d O w S t V y 0 d J N o 3 R U M c t l 3 T J M t b 6 C A t J V D / I v u E I P A V w k G I H B h 6 D Y g 4 2 Y n j Y s m A g J 6 2 B C W E Q R V 3 m G e + R I m x C L E c M m d E j f A e 3 a K e r T X n r G S u 3 S K R 6 9 E S l 1 7 J E m I F 5 E s T x N V / l E O U v 0 N + + J 8 p R 3 G 9 P f S b 1 G h A p c E / q X b s r 8 r 0 7 W I t B H V d X A q a Z Q I b I 6 N 3 V J V F f k z f V v V Q l y C A m T c Y / y E c W u U k 7 7 r C t N r G q X v b V V / k 0 x J Sc G R q E F R T Y h h d H c t d M t M V f X P 7 S 1 W K H B L i N O 5 S X B J m Rj n p s 2 0 0 q a l d 9 9 Y 3 8 T e T q V m 9 Z 3 l u h n d 9 S x q w + 3 O c 0 6 C x V 3 U P q u 7 5 f q V 2 n I + 6 i C 1 s Y 5 f m e Y g a T l G H R 9 4 C j 3 j C s 3 V m J d a t N f p M t Q q 5 Z h P f l v X w A Y X b k S M = < / l a t e x i t > txt1Visual Annotators (VFMs)< l a t e x i t s h a 1 _ b a s e 6 4 = " b 2 b Q A C R 8 s p o F X U f I D R H a k v G 4 W 4 4 = " > A A A C 7 3 i c j V H L S s N A F D 3 G V 3 1 X X b o J F q G 6 K I m I u h R d 6 L K C V c G W k q T T d D D N x M n E B 8 V / c O d O 3 P o D b v U v x D / Q v / D O G M E H o h O S n D n 3 n j N z 7 / W T i K f K c Z 7 7 r P 6 B w a H h w s j o 2 P j E 5 F R x e m Y / F Z k M W C 0 Q k Z C H v p e y i M e s p r i K 2 G E i m d f 1 I 3 b g H 2 / p + M E p k y k X 8 Z 6 6 S F i j 6 4 U x b / P A U 0 Q 1 i 0 t 1 x c 5 J 1 9 u W I o t b P A 4 v y y p U T X e x L n n Y U Z 6 U 4 s z 2 / f O m 2 y y W n I p j l v 0 T u D k o I V 9 V U X x C H S 0 I B M j Q B U M M R T i C h 5 S e I 7 h w k B D X Q I 8 4 S Y i b O M M l R k m b U R a j D I / Y Y / q G t D v K 2 Z j 2 2 j M 1 6 o B O i e i V p L S x Q B p B e Z K w P s 0 2 8 c w 4 a / Y 3 7 5 7 x 1 H e 7 o L + f e 3 W J V e g Q + 5 f u I / O / O l 2 L Q h v r p g Z O N S W G 0 d U F u U t m u q J v b n + q S p F D Q p z G L Y p L w o F R f v T Z N p r U 1 K 5 7 6 5 n 4 i 8 n U r N 4 H e W 6 G V 3 1 L G r D 7 f Z w / w f 5 y x V 2 t u L s r p Y 3 N f N Q F z G E e Z Z rn G j a w g y p q 5 H 2 F e z z g 0 T q x r q 0 b 6 / Y 9 1 e r L N b P 4 s q y 7 N 0 M p o T o = < / l a t e x i t >Grounding(tgt1) ! bbx1 < l a t e x i t s h a 1 _ b a s e 6 4 = " H 0 O o c 4 Z / Z M j 6 H I y B c o V 6 b 4 A + C U A = " > A A A C 7 3 i c j V H L S s N A F D 2 N 7 / q q u n Q T L I K 6 K E k R d V l 0 o U s F q 4 K V k q R j H E w z c T L R l u I / u H M n b v 0 B t / o X 4 h / o X 3 h n j O A D 0 Q l J z p x 7 z 5 m 5 9 / p J x F P l O M 8 F q 6 9 / Y H B o e K Q 4 O j Y + M V m a m t 5 L R S Y D V g 9 E J O S B 7 6 U s 4 j G r K 6 4 i d p B I 5 r X 9 i O 3 7 p x s 6 v n / O Z M p F v K u 6 C T t q e 2 H M j 3 n g K a K a p a W G Y h 3 S 9 T a l y O I W j 8 P L B R W q Z n W x I X l 4 o j w p x Y X t + 5 1 m t V k q O x X H L P s n c H N Q R r 6 2 R e k J D b Q g E C B D G w w x F O E I H l J 6 D u H C Q U L c E X r E S U L c x B k u U S R t R l m M M j x i T + k b 0 u 4 w Z 2 P a a 8 / U q A M 6 J a J X k t L G P G k E 5 U n C + j T b x D P j r N n f v H v G U 9 + t S 3 8 / 9 2 o T q 3 B C 7 F + 6 j 8 z / 6 n Q t C s d Y M z V w q i k x j K 4 u y F 0 y 0 x V 9 c / t T V Y o c E u I 0 b l F c E g 6 M 8 q P P t t G k p n b d W 8 / E X 0 y m Z v U + y H M z v O p b 0 o D d 7 + P 8 C f a q F X e l 4 u 4 s l 2 v r + a i H M Y s 5 L N A 8 V 1 H D F r Z R J + 8 r 3 O M B j 9 a Z d W 3 d W L f v q V Y h 1 8 z g y 7 L u 3 g B H + 6 E 8 < / l a t e x i t > Grounding(tgt2) ! bbx2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 1 Z z d n o X O 7 u p F U 9 S m d B 0 J a 4 2 I 2 o = " > A A A C 7 3 i c j V H L S s N A F D 2 N 7 3 f V p Z t g E d R F S a y o S 9 G F L h V s K 1 g p S T r G w T Q T J x M f F P / B n T t x 6 w + 4 1 b 8 Q / 0 D / w j t j C j 4 Q n Z D k z L n 3 n J l 7 r 5 9 E P F W O 8 1 K w e n r 7 + g c G h 4 Z H R s f G J 4 q T U 7 V U Z D J g 1 U B E Q u 7 7 X s o i H r O q 4 i p i + 4 l k X t u P W N 0 / 2 d T x + h m T K R f x n r p M 2 G H b C 2 N + x A N P E d U s L j Y U u y B d Z 0 u K L G 7 x O L y a V 6 F q V h Y a k o f H y p N S n N u + f 9 G s N I s l p + y Y Z f 8 E b g 5 K y N e O K D 6 j g R Y E A m R o g y G G I h z B Q 0 r P A V w 4 S I g 7 R I c 4 S Y i b O M M V h k m b U R a j D I / Y E / q G t D v I 2 Z j 2 2 j M 1 6 o B O i e i V p L Q x R x p B e Z K w P s 0 2 8 c w 4 a / Y 3 7 4 7 x 1 H e 7 p L + f e 7 W J V T g m 9 i 9 d N / O / O l 2 L w h H W T A 2 c a k o M o 6 s L c p f M d E X f 3 P 5 U l S K H h D i N W x S X h A O j 7 P b Z N p r U 1 K 5 7 6 5 n 4 q 8 n U r N 4 H e W 6 G N 3 1 L G r D 7 f Z w / Q W 2 p 7 K 6 U 3 d 3 l 0 v p G P u p B z G A W 8 z T P V a x j G z u o k v c 1 H v C I J + v U u r F u r b u P V K u Q a 6 b x Z V n 3 7 0 z N o T 4 = < / l a t e x i t > Grounding(tgt3) ! bbx3< l a t e x i t s h a 1 _ b a s e 6 4 = " 7 6 H X Z u x 3 2 y 8 Z 7 Given an original VQA sample, the linguistic annotator (LLMs) taught with usage of manipulations (prompt) is first asked to provide solving steps for the question Q, and the visual foundational models (VFMs) are then engaged to replace the manipulations results, followed by a final traversal on the tree branched by the possible manipulation results to find positive paths terminating to the answer A.I e C X j M C Y R 2 N w s c = " > A A A C 6 X i c j V H L S s N A F D 3 G V 3 1 X X b o J V q F u S i K i L o v d u P O B 1 U I r J U m n d W h e T C Z a K f 6 A O 3 f i 1 h 9 w q z 8 i / o H + h X f G C G o Rm O f 6 Y 2 m V d H F O K g N t 1 v A X i Y = " > A A A C 0 H i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b 3 V W x D 6 i l T K b T N p i X y U Q s p Y h b f 8 C t f p X 4 B / o X 3 h l T U I v o h C R n z r 3 n z N x 7 n c h z E 2 l Z r z l j a n p m d i 4 / v 7 C 4 t L y y W l h b r y V h G n N R 5 a E X x g 2 H J c J z A 1 G V r v R E I 4 o F 8 x 1 P 1 J 2 r Y x W v 3 4 g 4 c c P g Q g 4 i 0 f J Z L 3 C 7 L m e S q N a l z 2 S f M 2 9 4 O m p b 7 U L R K l l 6 m Z P A z k A R 2 a q E h R d c o o M Q H C l 8 C A S Q h D 0 w J P Q 0 Y c N C R F w L Q + J i Q q 6 O C 4 y w Q N q U s g R l M G K v 6 N u j X T N j A 9 o r z 0 S r O Z 3 i 0 R u T 0 s Q 2 a U L K i w mt i p h 4 Q W X 6 C A E R w o f A g E k Y Q 8 M C T 1 N 2 L A Q E d f C k L i Y k K v j A i MB E t i p h 4 Q W X 6 C A E R w o f A g E k Y Q 8 M C T 1 N 2 L A Q E d f C k L i Y k K v j A i M s k D a l L E E Z j N g r + v Z o 1 8 z Y g P b K M 9 F q T q d 4 9 M a k N L F N m p D y Y s L q N F P H U + 2 s 2 N + 8 h 9 p T 3 W 1 A f y f z 8 o m V 6 B P 7 l 2 6 c + V + d q k W i i 0 N d g 0 s 1 R Z p R 1 f H M J d V d U T c 3 v 1 Q l y S E i T u E O x W P C X C v H f T a 1 J t G 1 q 9 4 y H X / T m Y p V e 5 7 l p n h X t 6 Q B 2 z / H O Q l q u y V 7 v 2 S f 7 R X L x 9 m o 8 9 j E F n Z o n g c o 4 x Q V V Mg i N I O t V z v c G 6 T H r K j L S Y = " > A A A C 0 H i c j V H L S s N A F D 2 N r / q u u n Q T L I K r k o i o y 6 I b 3 V W x D 6 i l T K b T N p i X y U Q s p Y h b f 8 C t f p X 4 B / o X 3 h l T U I v o h C R n z r 3 n z N x 7 n c h z E 2 l Z r z l j a n p m d i 4 / v 7 C 4 t L y y W l h b r y V h G n N R 5 a E X x g 2 H J c J z A 1 G V r v R E I 4 o F 8 x 1 P 1 J 2 r Y x W v 3 4 g 4 c c P g Q g 4 i 0 f J Z L 3 C 7 L m e S q N a l z 2 S f M 2 9 4 O m r b 7 U L R K l l 6 m Z P A z k A R 2 a q E h R d c o o M Q H C l 8 C A S Q h D 0 w J P Q 0 Y c N C R F w L Q + J i Q q 6 O C 4 y w Q N q U s g R l M G K v 6 N u j X T N j A 9 o r z 0 S r O Z 3 i 0 R u T 0 s Q 2 a U L K i w mA h 7 h Q 5 c Z K Q b + I C I 8 y T N q M s Q R m M 2 E v 6 n t P u p G A j 2 m v P 1 K g 5 n R L Q K 0 l p Y 5 M 0 M e V J w v o 0 2 8 Q z 4 6 z Z 3 7 x z 4 6 n v N q S / V 3 i F x C p c E P u X b p z 5 X 5 2 u R W G A f V O D T z U l h t H V 8 c I l M 1 3 R N 7 c / VIn this section, we first introduces the automated data generation pipeline (illustrated in Figure 4 ), that employs reliable LLMs as linguistic annotators and VFMs as the visual annotators to produce (e.g., step 2 for finding pillars in Figure 5 ). We then perform a traversal on each produced tree with Depth First Search (DFS) to find all positive paths {P i |P i ∈ T , i = 1, 2, ...} that can terminate with the final answer A from the result of the last manipulation. Based on this method, the generated CoM samples with positive paths are guaranteed to be error-free. We implement this pipeline on 3 existing datasets that require detailed recognition or objects counting, TextVQA (Singh et al., 2019) , ST-VQA (Biten et al., 2019) , and TDIUC (Shrestha et al., 2019) , to build 70K CoM samples2 . The designed prompt, a generated example with linguistic and visual results, and detailed algorithm illustration are available at AppendixC.1.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Appendix B.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
human annotation: 
 The analysis from Fig. 1 of AlphaGeometry (Trinh et al., 2024) shows that outputting auxiliary lines in linguistic reasoning process helps LLMs to solve complex geometry problems. Benefiting from the expressive capability of CoM structure, we have also manually annotated high-quality CoM samples for the graphical mathematical problems to facilitate VLMs in solving this challenging scenario.Similar to the automated pipeline, we engage 10 human experts as the linguistic annotators and visual annotators, where each expert is asked to annotate the linguistic solving steps and the use of manipulations, as well as the results of manipulations on images. We perform this annotation on the MathVista (Lu et al., 2023) and ChartQA (Masry et al., 2022) , which include geometric and chart math problems, resulting in the collection of 6K high-quality CoM math samples.Finally, we adapt the CoM samples to be compatible with VQA-style training samples. For each CoM sample including n images from manipulations outputs (I 0 , Q, C 0 , I 1 , C 1 , ..., I n , A), we convert it into a multi-turn VQA sample segmented by the images[(I 0 , Q, C 0 ), (I 1 , Q, C 1 ), ..., (I n , Q, A)],where EQUATIONwhere Q t ∈ R s×d is query representation of current layer, and the K t , V t ∈ R (s×t)×d refer to the concatenation of accumulated representations and will be further truncated if the sequence length s × t is greater than a predefined threshold. At t > 0, the new image I t will be cropped from I t-1and amplified with the Bicubic Interpolation (Keys, 1981) .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
training: 
 The proposed CogCoM-17B relies on two main stages of training, to develop the capabilities of general multimodal task-solving as well as the visual reasoning. We empirically show that the model can effectively learn the evidential visual reasoning by ingesting this portion of CoM data. We train model with 14,000 iterations and a batch size of 160, where the learning rate reaches 10 -5 after 280 steps of warm-up and then decays linearly. The parameters of 6.5B visual experts are trained with the objective of next token prediction. These two stages of training result in our standard version of CogCoM involving both chat and reasoning capabilities.More training details are available at Appendix D.2.As the results shown in Table 2 , CogCoM achieves the state-of-the-art performance in comparison with all generalist models, and achieves significant improvements over the baseline model.Specifically, compared to the baseline model, our model achieves up to 5.97 and 9.0 percentage points improvement on the benchmarks that requires complex reasoning and detailed recognition, respectively. On GQA and TextVQA, CogCoM also obtains comparable results with the large-scale specialist SOTAs. This result demonstrates the effectiveness of the proposed approach in solving details recognition problem. 
experiments for reasoning accuracy and time complexity: 
 Due to the lack of resource, we build CoM-test, a benchmark with evidential reasoning chains on the TextVQA test set based on the proposed data generation pipeline, and also introduce a keypointsaware metric to validate the correctness of reasoning paths (see Appendix C.3 for detailed statistics).We also evaluate the time complexity for model generation on a held-out benchmark, MM-Vet. From the results we find that compared to baseline model, CogCoM produces information-intensive reasoning content (e.g., detection boxes, auxiliary lines) without incurring infeasible time overhead.For example, without quantitive optimization, CogCoM outputs 262.9 informative tokens in approximately 9 seconds. With the advantages in long-context optimization techniques (Hooper et al., 2024) , we believe that it is crucial for models to produce informative content and accurate responses.The task of visual grounding requires models to precisely provide the corresponding coordinates of regions in an image based on the given target description. Following the existing work (Wang et al., 2023b) , we train our model obtained by the first stage on a mixture of datasets, including an instruction corpus MultiInstruct, a high-quality grounded VQA corpus introduced in CogVLM, and the 70K CoM data. This training results in a generalist grounding model that is excelling at visual grounding while capable of reasoning. For all benchmarks, we prompt CogOM in a chat manner to ask the model to provide grounded coordinates, such as "Where is expr answer in [x0,y0,x1,y1] format.", where the expr refers to the target expression. We use the standard metric, that considers a prediction as correct when the intersection-over-union (IoU) between boxes is greater than 0.5. (Bai et al., 2023) .As shown in Figure 2 , CogCoM achieves the best performance in 6 out of all 8 sub-sets.Based on the training with a mixture of broad capabilities, this result indicates that our model exhibits a superior grounding abilities while offers potential to solve a variety of tasks.We further examine the general multimodal capabilities, and the hallucination issue. We use the generalist VQA model and obtain model predictions by directly asking the original questions in benchmarks. We use the challenging adversarial version and official evaluation scripts for POPE.InstructBLIP (Dai et al., 2023) Vicuna-13B 25.6 77.3 LLaVA (Liu et al., 2023b) LLaMA2-7B 28.1 66.3 DreamLLM (Dong et al., 2023) Vicuna-7B 35.9 76.5 LLaVA-1.5 (Liu et al., 2023a) Vicuna-13B 36.3 84.5 CogVLM (Wang et al., 2023b) Vicuna-7B 45.5 † 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Detailed compute resources are listed in Section D.2.• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discussed the societal impacts in details in Appendix B.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The training data collected from public datasets with research purpose does not face the safety risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The public available datasets used in this paper are cited properly.• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.We simply implement the CropZoomIn referring to human behaviors with a local code interpreter.The success rate of GPT-4 to achieve the positive paths is 0.3555.xi, yi ∈ [000, 999] refer to the normalized pixel coordinates.See Appendix D.1 for examples. 
cogcom: 
 Vicuna-7B 46.1 87.8Table 3 : Evaluation results on the general and hallucination assessment benchmarks. A.1 Large Vision-Langauge Models as FoundationsMost of LVLMs rely on the training on publicly available image-caption pairs, including ALIGN (Jia et al., 2021) , MSCOCO (Lin et al., 2014) , VG Krishna et al. (2017) , CC3M Sharma et al. (2018) , CC12M (Changpinyo et al., 2021) , SBU (Ordonez et al., 2011) , LAION2B (Schuhmann et al., 2022) , LAION400M Schuhmann et al. (2021) . Starting from Flamingo (Alayrac et al., 2022) , a series of LVLMs have focused on training the adaptation layers to align the visual representation to the frozen LLMs on a mixture of image-text pairs with the above corpus, including BLIP2 Li et al. (2023b) , KOSMOS Huang et al. (2023b) , and OpenFlamingo (Awadalla et al., 2023) . Inspired by success of instruction tuning in LLMs (Wang et al., 2022c) , a line of works have devoted efforts to build vision-oriented instruction-answer pairs through GPT4 and train models for imitation, such as LLAVA (Liu et al., 2023b) , Otter (Li et al., 2023a) , VisionLLM (Wang et al., 2023a) , MultiInstruct (Xu et al., 2022) , Lynx (Zeng et al., 2023) , InstructBLIP (Dai et al.) , CleverFlamingo (Chen et al., 2023a) andStableLLaVA (Li et al., 2023d) . Recently, researchers have proven the efficiency of developing LVLMs with two stages of training, the first stage of abundant pretraining on image-caption pairs and the second stage of alignment on image-question-answer triples, such as PALI (Chen et al., 2022) , PaLI-X (Chen et al., 2023d) , Qwen-VL (Bai et al., 2023) , and CogVLM Wang et al. (2023b) .To further enhance the ability of LVLMs in solving high-level visual problems, research focusing on various aspects of reasoning is attracting broad attention. We simply divide existing studies into tree broad categories. The first line of research focus on enhance train models with a mastery of cross-modal grounded reasoning, where grounded instruction-following supervision is build through public visual grounding dataset or GPT4-V for training, including KOSMOS-2 (Peng et al., 2023), Shikra (Chen et al., 2023b), and GPT4ROI (Zhang et al., 2023a) . The second aspect of efforts have been devoted into promoting models to understand artificial visual scenes, such as figures, charts, and receipts. These studies includes CogAgent (Hong et al., 2023) and CHARTVE (Huang et al., 2023a) .Some other studies address the crucial problem of hallucination in LVLMs with counterfactual or interpretable reasoning (Yu et al., 2023a; Yin et al., 2023) . V* (Wu & Xie, 2023 ) also contributes efforts to enhance the details recognition of VLMs based the LLM-guided searching process.Though we try to develop an accurate and robust framework that engages remarkable LLM to provide basic solving steps, adopts reliable visual tools to obtain visual contents, and then acquires feasible paths based on traversal, there are still limitations in our methodology that we hope to improve in the future. First, We find that the diversity of linguistic solving steps is insufficient, and the inaccuracy of visual tools (e.g., the rough granularity of grounding boxes, OCR failures on slant letters) will lead to a large amount of negative paths (effectively utilizing these paths would beneficial). We suggest to promote these limitations with dedicate prompts and improved visual tools. Second, our current model re-input the manipulated images with a set of hard prompts, which may bring speed losses.This is expected to be improved by implementing the physical manipuations into the calculations in vector space. This work presents a general visual reasoning mechanism that alleviate the problems caused by existing conclusion-alignment training for VLMs, introduces a data production framework involving LLMs and visual tools as reliable annotators, and devises a memory-based compatible VLM architecture. We expect this work to bring three benefits to the community. First, the proposed visual reasoning mechanism may push the progress of VLMs in solving complex visual problems. Second, the introduced data production framework may be applied to widespread training scenarios to promote the development of current data-driven machine learning. Third, we hope that the memory-based architecture will be helpful for VLMs in multi-turn long contexts.In this section, we further introduce the details of CoM data production, with the overall algorithm of a pseudo code, an example of the solving steps generation with LLM and corresponding guideline, an example of the reasoning chains completion with visual tools. We also list the details of data statistics for the synthesised training data as well as the evaluation data of CoM-test, followed by a limitation analysis for the current data production method.C.1 Algorithm for the Automate Data Generation PipelineWe provide the pseudocode of the CoM synthesis algorithm to clearly explain the process of data generation, thereby facilitating understanding and reproduction 1.Algorithm 1 Synthesising Chain of Manipulations 1: Define:    M anipulations : {f i : x → y | f i ∈ M} Linguistic Annotator : Ψ L //W eEQUATION5: Define tree T 6: for i = 1 to |ς| do7: Extract x i , y i instantiated with f i in step i 8:Extract referential boxes B from x i 9:for b in B do 10:Leverage Ψ V to acquire corresponding visual content y i = Ψ(x i |I, b), and apply y i to treeEQUATIONend for 12: end for 13: Traverse T to obtain positive chains that leads to given answer with terminal return[ς 1 , ς 2 , ...] = DF S(T |A) (5) 14: Return [ς 1 , ς 2 , ...]To measure the correctness of CoM chains, we introduce a keypoints-aware metric. The intuition is that we care about the key elements including actions (i.e., manipulation name), targets (i.e., manipulation input), and visual contents (i.e., manipulation returns) of each step in the path, as well as the logical execution order of manipulations. Given a pair of chain-answer annotation (c, a) and corresponding model prediction (c , a ), we first sequentially extract the key elements from c and c to construct two ordered lists, and then replace the elements in the lists with their fixed indices in a Bag-of-Elements E = c ∪ c to result in lists of k and k . We thus calculate the score as the normalized Levenshtein Distance s c = Levenshtein(k, k )/N between the two lists, where N is the maximum length between k and k . We adopt this simple discretization strategy with low time complexity to concentrate on the key points as well as the solving order. We further consider the linguistic matching of paragraphs by calculating the BLEU (Papineni et al., 2002) score between two chains s p = BLEU(c, c ), and the final sore is a weighted combination as acc = (0.6 × s c + 0.4 × s p )/2.We develop a strategy to extract predicate phrases based constituency parsing with StandordCoreNLP, in which we extract verb, conjunction-connected verb phrase, preposition-connected verb phrase.Besides the standard CoM data incorporating manipulations with explicit visual evidences, the proposed data synthesising framework is compatible of producing implicit visual reasoning steps step i = (desc i ) without involving the manipulations. We thereby also build this partial CoM data on the corpus consisting of absurd visual questions (i.e., asking unanswerable questions based on the given image) to further resist the toxic hallucinations. Specifically, given an image I with a question Q,we prompt GPT-4V (OpenAI, 2023b) to solve the question step-by-step to acquire the reasoning chains. i-th crop_and_zoomin manipulation which is useful to identify small and subtle objects in image, that first crops the current image using the box `bbx` defined by the top-left and bottom-right coordinates, and then zoom in the cropped image by two times and finally return the resulting image `img_i`. crop_and_zoomin_i(bbx)->img_i: i-th grounding manipulation, that locates the object(s) specified by the target noun phrase `tgt` in current image, and return the resulting bounding box(es) as `bbx_i` where each box is represented by the top-left and bottom-right coordinates.i-th OCR manipulation, that recognize the natural texts written on the target `tgt`, and return the recognized texts `txt_i`. calculate(tgt)->res_i:Question: What number police station is on the building?Step 1: grounding_1(the building)->bbx_1, Identify the number of the police station on the building in box `bbx_1` and return the bounding box of the number as `bbx_2`.Recognize the number in the region `bbx_2` and return the recognized number as `txt_1`.Question: What number police station is on the building?Locate the building in the image and return the bounding box of the building as `bbx_1`.Identify the number of the police station on the building in box `bbx_1` and return the bounding box of the number as `bbx_2`.Recognize the number in the region `bbx_2` and return the recognized number as `txt_1`.< l a t e x i t s h a 1 _ b a s e 6 4 = " E e i p g 0 S a S z O / b M I o 2 f h u 3 x E h W X Y = " > A A A C y X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 o w h S s W k L t U g y n d b Y v E w m Y i 2 u / A G 3 + m P i H + h f e G d M Q S 2 i E 5 K c O f+ q Y = " > A A A C z H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 0 K 0 Y m M r W k q S T u v Q v E g m Q i n d + g N u+ q Y = " > A A A C z H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 4 0 K 0 Y m M r W k q S T u v Q v E g m Q i n d + g N uY C H j F H c B G w V p I y N / Q C 1 v Q G R z L e v G N p x u O o I Y Y J a 4 d u P + I 9 7 r u C q K s z p 1 F 3 G h e d k 0 6 5 Y l U t t c x J Y G t Q g V 7 1 u P y M G 3 Q R w 0 e O E A w R B O E A L j J 6 r m H D Q k J c G y P i U k J c x R n G K J E 2 p y x G G S 6 x A /I o 2 f h u 3 x E h W X Y = " > A A A C y X i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R I R d V l 0 o w h S s W k L t U g y n d b Y v E w m Y i 2 u / A G 3 + m P i H + h f e G d M Q S 2 i E 5 K c O f e e M 3 P v d S L P T Y R h v O a 0 i c m p 6 Z n 8 b G F u f m F x q b i 8 U k / C N G b c Y q E X x k 3 H T r j n B t w S r v B 4 M 4 q 5 7 T s e b z j 9 A x l v 3 P A 4 c c O g J g Y R b / t 2 L 3 C 7 L r M F U f W j k 6 p V O 7 s o l o y y o Z Y + D s w M l J C t a l h 8 w T k 6 C M G Q w g d H A E H Y g 4 2 E n h Z M G I i I a 2 N I X E z I V X G O e x R I m 1 I W p w y b 2 D 5 9 e 7 R r Z W x A e + m Z K D W j U z x 6 Y 1 L q 2 C B N S H k x Y X m a r u K p c p b s b 9 5 D 5 S n v N q C / k 3 n 5 x A p c E v u X b p T 5 X 5 2 s R a C L P V W D S z V F i p H V s c w l V V 2 R N 9 e / V C X I I S J O 4 g 7 F Y 8 J M K U d 9 1 p U m U b X L 3 t o q / q Y y J S v 3 L M t N 8 S 5 v S Q M 2 f 4 5 z H N S 3 y u Z O 2 T z d L l X 2 s 1 H n s Y Z 1 b N I 8 d 1 H B I a q w y P s K j 3 j C s 3 a s X W u 3 2 t 1 n q p b L N K v 4 t r S H D 4 z X k S A = < / l a t e x i t >Locate the building in the image and return the bounding box of the building as `bbx_1`.Step 2: (grounding_2(number police station on the building in box `bbx_1`)->bbx_2,Step 3: (OCR_1(number in region `bbx_2`) ->txt_1, Linguistic AnnotationIn this work, we adopt the GPT4-turbo as the linguistic annotator for generating problems solving steps, and the API call was conducted during the period of 2023.9 -2023.12. For the visual annotators, we leverage the the currently best-performing tools, GroundingDINO and PaddleOCR, to acquire all visual contents requested by the manipulations. For a clear description to the production setting and results, we illustrate the guiding prompt, and an example-based linguistic annotation results as well as the visual annotation results in Figure 8 .For the implemented data framework, we engage the remarkable LLM to provide basic solving steps, adopt two reliable visual tools (i.e., GroundingDINO and PaddleOCR) to acquire corresponding visual contents, and then perform the traversal to achieve feasible reasoning paths, which ensures the correctness and robustness of data synthesizing. However, we also find that there are three major limitations caused by the employed models and could be improved in future:• The lack of diversity in linguistic reasoning steps. The 5-shot prompting to the GPT-4 gains a stable solving steps, but it also results in the descriptions for executing manipulations or general thinking are similar. We suggest that this can be addressed by employing diversified prompts or requirements.• The inaccuracy of visual tools. We find that there are a considerable amount of negative paths caused by the failures of visual tools, such as the rough granularity of bounding boxes and the error recognition of slated letters or long sentences. This issue can be relieved by improving the semantic understanding capabilities of visual tools. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
large language model image tokens: 
 What is written on the pillar in front of the man?No SmokingImage tokens What is written on the pillar in front of the man?First, use GROUNDING(man) to find regions [[4,5,7,12] , [28, 6, 30, 9] ] … Then, use GROUNDING(pillar near [4, 5, 7, 12] ) to find [[8,0,17,13] ] … Zoom into the cropped region [[9,7,11,9] ] by two times and re-input …Based on the new image, the text written is QUICK DEPOSIT. Benefiting from the advantage of Large Language Models (LLMs) in broad world knowledge, large Vision Language Models (VLMs) (Alayrac et al., 2022; Wang et al., 2023b) that are further trained to understand visual inputs have demonstrated viabilities on broad multimodal scenarios, such as visual question answering (Liu et al., 2023b) , visual grounding (Peng et al., 2023) , optical character recognition (Zhang et al., 2023b) . The research employing VLMs as foundation models (Bai et al., 2023; Sun et al., 2023b; Wang et al., 2023b) usually involves two main stages of training, where the first stage develops intrinsic visual understanding ability through exposure to massive imagecaption pairs, and the second stage endows the models with problem-solving capabilities through the instruction tuning.However, existing tuning methods train models to respond to instructions with conclusive language responses upon visual inputs, which leads models to ignore the essential intermediate visual reasoning and further results in failures in meticulous visual problems, unfaithful responses, and even hallucinations. For example in the left subplot of Figure 2 , we test the top-performing model CogVLM (Wang et al., 2023b) about the details in the image (i.e., texts written on a pillar), and it directly responds an incorrect answer (i.e., NO SMOKING), most likely from bias to visual or linguistic priors (i.e., typical scenes with a pillar in office). The absence of the essential reasoning on the visual scene may lead to a rash response (Hwang et al., 2023) .Humans solve problems regarding visual details by marking or processing the given images for convenience and rigor, which we refer to as manipulations. For example, we find targets by sequentially locating references, and concentrate on subtle details by zooming into a corresponding region.Most of VLMs have developed numerous intrinsic capabilities (e.g., grounding boxes, recognizing texts) during the first stage of training. By further imitating the fundamental human behaviours (e.g., cropping, zoom in), models have the potential to perform this cognitive reasoning process. Three major obstacles in eliciting VLMs with such reasoning are (1) flexible definitions of manipulations covering most visual problems, (2) an efficient data collection pipeline capable of producing abundant training data, and (3) a multi-turn multi-image VLM structure compatible with existing models.Inspired by the human cognition in solving visual problems, we introduce Chain of Manipulations (CoM), a mechanism that enables VLMs to solve problems step-by-step with evidence, with each step potentially involving a manipulation on the visual input and its corresponding result, both generated by the model to facilitate the success and fidelity. This paper studies a complete roadmap with manipulations design, data collection, model architecture and training process for training general VLMs with this mechanism. We first formally design 6 basic manipulations upon the pilot experiments, which are capable of handling diverse visual problems. Next, we propose a cascading data generation pipeline based on reliable large language models (e.g., LLMs, the linguistic annotators) and visual foundational models (e.g., VFMs, the visual annotators), which can automatically produce abundant error-free training data. We collect 70K CoM samples with this pipeline. We then devise a multi-turn multi-image model architecture compatible with typical VLMs structures. Based on a data recipe incorporating the curated corpus, we finally train a general VLM equipped with CoM reasoning mechanism, named CogCoM, which possesses capabilities of chat, captioning, grounding and reasoning. Additionally, benefiting from the expressive capability of the proposed mechanism, we further manually annotated 6K high-quality samples of graphical mathematical problems, each accompanied by a CoM reasoning process, to advance the research of VLMs in solving challenging mathematical problems.We conduct extensive experiments on 9 benchmarks from 4 categories, including TextVQA (Singh et al., 2019) , ST-VQA (Biten et al., 2019) , TallyVQA (Acharya et al., 2019) , and GQA Hudson & Manning (2019) for detailed visual question answering, RefCOCO (Yu et al., 2016) , RefCOCO+ (Yu et al., 2016) , and RefCOCOg (Mao et al., 2016) for visual grounding, POPE (Li et al., 2023c) for hallucination validation, and MM-Vet (Yu et al., 2023b) for general multimodal ability. Our model achieves up to 9.0 and 1.09 accuracy improvement on the detailed VQA and grounding benchmarks, respectively, and the superior performance on the general multimodal benchmark. The results demonstrate the effectiveness of the mechanism while maintaining the interpretability of outputs. 
 
 Appendix:  


Decision: The idea of having a chian of manipulation for VLM makes human sense. However, does such a mechanism truly help with VLM reasoning capabilities, as pointed out by reviewers, it is unclear. The authors argue CogVLM is targeting a wider range of tasks, while CogCoM is mainly focusing on the aspects as being discussed and designed by the authors, this makes the performance gain, either on general tasks or claimed reasoning tasks, less valuable, as the proposed method is supposed to focus on the task as it designed for. The draft received diverging opinions without a strong champion of the work from the set of reviewers, given its current form and presentation that are needing further improvement, AC would suggest a reject.

Reviews: review: paper_summary: paper_summary: Drawing inspiration from human cognition to solve visual problems through localizing, zooming, etc., this paper introduces a new framework called CogCom, which solves visual problems by automatically combining six types of basic manipulations. When facing a visual problem, CogCom can use reasoning to solve each step and employ basic tools to aid in the problem-solving process. To achieve this goal, CogCom constructed a data generation pipeline that leverages GPT4 to build the training data for   CogCom. The CogCom leads to performance gains compared to its baseline CogVLM on several benchmarks.
strength_weakness: strength_weakness: 1. The VQA benchmarks reported in Table 1 are not very convincing. It would be beneficial to consider more modern and challenging benchmarks such as MMBench, MathVista, and SeedBench.
2. The comparison of baseline methods seems to be based on relatively outdated approaches. It might be more informative to compare them with more recent LVLMs like LLaVA-1.5, Monkey, and ShareGPT4V.
3. It would be helpful to discuss a closely related work ViperGPT [3] and V* [4]. ViperGPT shares an idea for solving visual problems via planning tool pathways. V* shares the idea of searching and zoom-in progressively.
4. The differences with some other related works should be discussed [5][6]. 

[1] Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

[2] ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

[3] ViperGPT: Visual Inference via Python Execution for Reasoning

[4] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

[5] CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

[6] DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models
questions: questions: 1. The performance improvements of CogCom in comparison to CogVLM appear to be relatively insignificant. Could this be attributed to the low success rate (around 55% in Figure 6)? Further explanations are necessary to address the situation where the planned pathway failed.

I will be pleased to raise my rating if my concerns as in weaknesses and questions are resolved.
limitations: limitations: The limitations have been discussed.
paper_summary: paper_summary: This paper introduces the Chain of Manipulations (CoM) mechanism for data generation to enhance visual reasoning in VLMs. The authors developed a data generation pipeline, producing 70K high-quality samples, and created the CogCoM model. CogCoM achieves state-of-the-art results across nine benchmarks, demonstrating significant improvements in various visual tasks.
strength_weakness: strength_weakness: 1.During data generation, the process relies entirely on GPT-4 for prompting and existing models (GroundingDino, PaddleOCR) for generation. As mentioned in the appendix, inaccuracies in these current visual models can affect the quality of generated data and the model's reasoning capabilities. However, the system lacks validation or filtering mechanisms to enhance data quality.
2.To highlight the specific improvements brought by CoM, it would be helpful to provide results both with and without the incorporation of CoM data. This would clarify the impact of CoM, especially since CogCoM integrates a significant amount of additional data such as MultiInstruct and LLaVAR during the instruction tuning stage as shown in Table 1.
3.The CoM dataset includes 6K high-quality manually annotated math samples, but no test results for math problems are provided. Clarification is needed on whether the purpose of this math data is solely to enhance the model's reasoning capabilities.
4.The paper emphasizes that CogCoM is a model capable of multi-image multi-turn understanding, but no corresponding test results (qualitative or quantitative) are provided.
5.In the model section, some parameters are not specifically explained, such as the maximum turns the model can accept and the predefined threshold.
6.Typos error: Line 288 CogOM->CogCoM
questions: questions: see weakness
limitations: limitations: see weakness
paper_summary: paper_summary: The paper presents CogCoM, a novel approach to training large Vision-Language Models (VLMs) using a mechanism called Chain of Manipulations (CoM). This mechanism enables the model to solve visual problems step-by-step with evidence, inspired by human cognitive processes like marking and zooming into images. CogCoM integrates manipulations such as grounding, zooming, and OCR into the VLM architecture, allowing it to handle various visual problems without external tools. The model is trained using a robust data generation pipeline and evaluated across multiple benchmarks, demonstrating state-of-the-art performance.
strength_weakness: strength_weakness: Weakenss in Points

This paper is generally good but I can still spot the following issues.

1. **Design of Figures and Tables**: The figures in the paper are not well-designed. The first and second figures are repetitive in meaning, and the colors in the first figure are too light (consider adding black outlines to the boxes). The font size in the second figure is too small to be legible on smaller screens. Additionally, the captions for Table 2 and Table 3 are too close to the tables, violating the submission guidelines.

2. **Lack of Discussion on Related Work**: The paper lacks a discussion of existing related work. It should consider citing and comparing with at least other agentic LMMs such as LLAVA-Plus[1] to provide a comprehensive comparison and context.

[1] https://arxiv.org/abs/2311.05437
questions: questions: See above Weakness part.
limitations: limitations: See above Weakness part.

Comments: comment: Thank you for the feedback. The new discussions are valuable.
Dear Reviewer,

We are very pleased to learn that our efforts have addressed your concerns and that you kindly raised the rating. Regarding your additional question about the discussion on how to mitigate the negative effect of incorrect reasoning paths, we have following response:

  - DualFocus adopts a good strategy that compares the PPLs of generated answers to switch the mode between direct-answering and reasoning-before-answering to alleviate the paths with low confidence. In comparison to DualFocus, we add the explicit launching prompts to allow the switch by users actively, and the randomness during training to encourage model switching the mode by itself accordingly with problem scenarios. As the PPL-based method is general, we will compare and implement this strategy into CogCoM in near future work.
  - Using Reinforcement Learning to penalize the negative paths during training is another strategy to mitigate the negative effects, as we have derived multiple negative reasoning paths during our data collection pipeline. However, using the RL-based objective will introduce the unstable training and low-efficiency compared to the straightforward cross-entropy on positive answers, and improving the effectiveness of this solution is a promising topic.
  - Humans solve difficult questions with a period of thinking before outputting the answers. As LLMs/VLMs generate outputs immediately following the question prompt, CoT mechanism serves as an effective substitute of the thinking. Similar to human thinking, the backtracking mechanism might be a reliable way to form correct and concise reasoning paths. However, in our experiments, we found that our integration of backtracking results in heavy time-complexity and relatively marginal benefits. Therefore, developing an effective backtracking strategy to enhance the correctness of reasoning paths while maintaining the efficiency is meaningful for mitigating the negative impact of the path failures.

Thank you once again for your time and valuable comments. We will add the above discussion to our paper.
Dear Reviewer,

I sincerely apologize for disturbing you again. In response to the multiple questions you raised, we hope that our experiments and explanations are through to address your concerns. As the discussion period is nearing its end, please let us know if you have any questions or feedback. Thank you very much.
Thank you for your efforts in solving my concerns. One additional question: since mistakes in reasoning paths are indeed a major factor affecting performance on the general multimodal benchmarks, the author may have some discussions (experiments are not required) about how to compensate for the negative effects of planned pathway failure, as in the concurrent work DualFocus.

Overall, most concerns are solved. I will raise my rating to 6.
Dear Reviewer,

Thank you once again for your time and valuable suggestions. We sincerely hope that our efforts can address your concerns and look forward to your response.
Dear Reviewer,

Thank you once again for your time and for pointing out the shortcomings in our work. We sincerely hope that our additional experiments and explanations can address your concerns, and look forward to your response.
Dear Reviewer,

Thank you once again for your time and valuable suggestions. We sincerely hope that our efforts can address your concerns and look forward to your response.