SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Date | Description | Received | Expenses | Available Funds |
Balance: end of August | | $154.90 |
9/1 | notebook | | $6.95 | $147.95 |
9/15 | walking dogs | $15.00 | $162.95 |
9/21 | basketball | $11.35 | $151.60
Generated Program: ["Row_Lookup", "Solution_Generator", "Answer_Generator"]
Row Lookup:
Date | Description | Received | Expenses | Available Funds  
9/21 | basketball |  $11.35 |$ 151.60
(Step 1) Subtract the cost of the basketball from the available funds before the purchase.
 (Step 2) $151.60 - $11.35 = $140.25
(Step 3) Oliver had $140.25 after he purchased the basketball.
(Step 4) The answer is $140.25.
Answer Generator (Wrong prediction): 140.25
Correct Answer (Ground truth): 151.60
Table 23: An example of failure cases in TabMWP from Chameleon (GPT-4) where the generated programs are suboptimal. In the given example, the simplified table produced by the "Row Lookup" lacks the structural relations present in the original table's context. These relations are crucial for the LLM-based solution generator to accurately comprehend the domain-specific table.
Table:
Tour bus schedule
historic house	10:15 A.M.	10:30 A.M.	10:45 A.M.	11:00 A.M.
downtown	11:00 A.M.	11:15 A.M.	11:30 A.M.	11:45 A.M.
skyscraper	11:30 A.M.	11:45 A.M.	12:00 P.M.	12:15 P.M.
old building	12:30 P.M.	12:45 P.M.	1:00 P.M.	1:15 P.M.
governor's mansion	1:00 P.M.	1:15 P.M.	1:30 P.M.	1:45 P.M.
Table Title: Tour bus schedule
Table: (semi-structured format)
historic house | 10:15 A.M. | 10:30 A.M. | 10:45 A.M. | 11:00 A.M.
downtown | 11:00 A.M. | 11:15 A.M. | 11:30 A.M. | 11:45 A.M.
skyscraper | 11:30 A.M. | 11:45 A.M. | 12:00 P.M. | 12:15 P.M.
old building | 12:30 P.M. | 12:45 P.M. | 1:00 P.M. | 1:15 P.M.
governor's mansion | 1:00 P.M. | 1:15 P.M. | 1:30 P.M. | 1:45 P.M.
Options: (A) 2 hours and 15 minutes (B) 2 hours and 45 minutes (C) 15 minutes (D) 30 minutes  $\triangleright$  Response
historic house | 10:15 A.M. | 10:30 A.M. | 10:45 A.M. | 11:00 A.M.  
governor's mansion | 1:00 P.M. | 1:15 P.M. | 1:30 P.M. | 1:45 P.M.
(Step 1) Find the last departure time from the historic house and the first arrival time at the governor's mansion: Historic house last departure: 11:00 A.M. Governor's mansion first arrival: 1:00 P.M.   
(Step 2) Subtract the departure time from the arrival time: 1:00 P.M. - 11:00 A.M.  $= 2$  hours   
(Step 3) It takes 2 hours to get from the historic house to the governor's mansion.   
(Step 4) The answer is 2 hours and 15 minutes.
Answer Generator (Wrong prediction): (A) 2 hours and 15 minutes
Correct Answer (Ground truth): (B) 2 hours and 45 minutes
Table 24: An example of failure cases in TabMWP from Chameleon (GPT-4) where some modules perform inaccurately and the generated programs are suboptimal. In this instance, the LLM-based solution generator struggles to understand the bus schedule, which incorporates domain-specific knowledge. Furthermore, the LLM planner does not utilize tools like "Table Verbalizer" and "Column Lookup", which could enhance the LLM's comprehension of the tabular context.