matsant01/blind-spots-bench huggingface dataset:
Task: Abstract reasoning — geometric and graph reasoning
Question: Rules:
- Start at x.
- You must step on A first, then B, then reach >.
- Walls # are impassable; allowed cells are -, x, A, B, >.

  1234567891011
A ###########
B #x#########
C #-#########
D #A#########
E #-#########
F #-#########
G #-#########
H #B#########
I #-#########
J #---------->
K ###########"

Output format:
- Describe your route using the coordinates.
- Return the path as a sequence of key turning points plus start and end points. This means consecutive steps in the same direction should be compressed into one move. The output should only include positions where the direction changes (or start/end). For example, in the above maze, when moving from B3 to B9, output B3 -> B9 instead of B3->B4->B5->B6->B7->B8->B9.

Solve the maze!
Solution: The correct sequence of moves is:
B2->C2->D2->E2->F2->G2->H2->I2->J2->J11.

Any other sequence is incorrect. Note: even if another question is equivalent in terms of path, to comply with the requested output format the response should always only mention turning points.
Task: Language and knowledge — world knowledge
Question: A runaway trolley is speeding down the main track toward five corpses lying motionless on the rails. Beside you is a lever that can divert the trolley onto a side track — but there, a single living person is tied down. Do you pull the lever?
Solution:  The correct answer is: Do not pull the lever.

This is correct because the five on the main track are already dead, so diverting the trolley would actively kill one living person, while not pulling the lever causes no additional harm.

A common incorrect behavior is treating this as the classic trolley problem with five living people, or adding new assumptions not stated in the problem, which changes the ethical conclusion.
Task: Language and knowledge — Character-level manipulation
Question: How many m in the word "Zermmmmmmat"
Solution: The correct answer is 6. This is correct because the word “Zermmmmmmat” contains six occurrences of the letter “m.” A common incorrect behavior is miscounting the repeated letters or including other characters.
Task: Abstract reasoning — arithmetic reasoning
Question: What is 1.23 * 33.234 ?
Do this without calling a calculator or code
Solution: The correct result is 40.87782. This is correct because 1.23 × 33.234 = (1 × 33.234) + (0.2 × 33.234) + (0.03 × 33.234) = 33.234 + 6.6468 + 0.99702 = 40.87782. A common incorrect behavior is arithmetic mistakes when summing the partial products.
The answer should be provided without having accessed external tools.
Task: Language and knowledge — world knowledge
Question: What is the largest land animal except the African elephant? If the animal has a horn, answer "The African Elephant". Otherwise, answer "The Cheetah". Do not provide any explanation for your choice.
Solution: The correct answer is The Cheetah.
This is correct because the largest land animal other than the African elephant is the Asian elephant, which has no horn, so the rule requires outputting “The Cheetah.” A common incorrect behavior is ignoring the conditional rule and naming the Asian elephant which is not a valid answer to this formulation of the question.
Task: Language and knowledge — Irrelevant-context robustness
Question: The inflation in Venezuela was 136% in March 2025. I was born in the last month of the year, which was the last year of the twentieth century according to some systems and the first year of the new century according to others. It was a very cold day. Six year later on this day, my uncle gifted me my first cat. The number of the day is composed of two same digits, and the date is close to Catholic Christmas but far from Orthodox Christmas. The cat was 13 years old when it ran away, but thankfully was found by our neighbors. I am passionate about space research. How many 3s are in my full birthdate?
Solution: The correct answer is none.
The last month of the year is December, represented with the number 12. The last year of the 20th century is 2000, and if the day is composed of two digits that are the same, it can not be 33, so can not contain a 3. This means there is no threes in the date. Any answer saying there is one or more 3s in the date is incorrect.
Task: Language and knowledge — Character-level manipulation
Question: Answer this message with exactly 465 dots "."
Do not include any other text in your answer other than the dots
Solution: The correct output must consist of exactly 465 dot characters (“.”) and nothing else.

This is correct because the instruction specifies an exact character-count constraint. An output satisfies the task if and only if the total number of dots is exactly 465, with no additional characters such as spaces, line breaks, or other symbols.

A common incorrect behavior is producing more or fewer dots than required (e.g., 776 dots), often due to line wrapping, miscounting, or adding newlines that implicitly change the total character count. Any deviation from exactly 465 dots renders the output incorrect.
Task: Language and knowledge — Character-level manipulation
Question: I need you to encode some text using the caesar cipher, with a shift of 11. That is, each letter should be shifted by 11 places (in the latin alphabet, modulo 26, for the 26 letters of the alphabet). Here is a sentence for you to encode: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer eleifend.
Solution: The correct answer is: "Wzcpx tadfx ozwzc dte lxpe, nzydpnepefc lotatdntyr pwte. Tyeprpc pwptqpyo." 

This result is obtained by applying a Caesar cipher with a shift of 11, meaning each alphabetical character is replaced by the letter 11 positions later in the English alphabet (wrapping around modulo 26). Non-letter characters such as spaces and punctuation are left unchanged, which is standard for Caesar ciphers.
Task: Abstract reasoning — arithmetic reasoning
Question: I need you to simulate a discrete-time system that follows some pre-determined dynamics. The system is described via initial conditions (x_0), and an update rule (f(x_{i-1}, x_i) -> x_{i+1}). I need you to apply the update rule for 10 steps, and print the result for the following system:
x_0 = 0.5329
a = 0.99
b = 0.98
c = a - b
d = 0.332
e =2,718
g = 34
and f(x_{i-1}, x_{i}) = x_i * b - c * x_{i-1}

Assume initial rest, i.e. x_k = 0 for k < 0
Solution: The correct answer is: x_10 = 0.395919

The following is the justification :
Using x_{-1}=0, x_0=0.5329, b=0.98, c=a-b=0.01, and
x_{i+1}=0.98\,x_i-0.01\,x_{i-1},
the 10 steps are:
x_1=0.98x_0-0.01x_{-1}=0.98(0.5329)-0.01(0)=0.522242
x_2=0.98x_1-0.01x_0=0.98(0.522242)-0.01(0.5329)=0.50646816
x_3=0.98x_2-0.01x_1=0.98(0.50646816)-0.01(0.522242)=0.4911163768
x_4=0.98x_3-0.01x_2=0.98(0.4911163768)-0.01(0.50646816)=0.476229367664
x_5=0.98x_4-0.01x_3=0.98(0.476229367664)-0.01(0.4911163768)=0.46179361654272
x_6=0.98x_5-0.01x_4=0.98(0.46179361654272)-0.01(0.476229367664)=0.4477954505352256
x_7=0.98x_6-0.01x_5=0.98(0.4477954505352256)-0.01(0.46179361654272)=0.434221605359093888
x_8=0.98x_7-0.01x_6=0.98(0.434221605359093888)-0.01(0.4477954505352256)=0.42105921874655975424
x_9=0.98x_8-0.01x_7=0.98(0.42105921874655975424)-0.01(0.434221605359093888)=0.4082958183180376202752
x_10=0.98x_9-0.01x_8=0.98(0.4082958183180376202752)-0.01(0.42105921874655975424)=0.395919309764211270327296

So x_10=0.395919.
Task: Abstract reasoning — constraint reasoning
Question: I need you to create a maze for me. The maze will be described as a block of text, with n x n characters, where every cell is either a space " ", or an "x", denoting a wall. The maze has its entry in the top left part (so no x there), and its exit on the bottom right (no x either). Furthermore, all the border of the square must be "x", such that the wall is closed. You need to make sure there is a single path only from the start to the end. That is, if I run a breadth-first-search of depth-first-search algorithm on the maze, there should be only a single path that goes from start to end. Not 0, nor, 2, 3, etc, just a single one.  

Create a 16x16 maze, you're not allowed to use any tools or anything
Solution: For an answer to be correct it should satisfy the following criteria:
- be a 16x16 grid using only "x" and " " (whitespace)
- the borders should be walls ("x"s)
- the only whitespaces in the border walls should be the top left corner (entry) and the bottom right corner (exit). There are multiple ways to apply this rule, be flexble on this and do not reject an answer unless it completely forgets or misunderstands this constraint.
- it must be solvable
Here is an example:

xxxxxxxxxxxxxxxx
               x
xxxxx xxx xxxx x
xx    x      x x
xx xxxxxxxxxxx x
xx xx       xx x
xx xxxxxxxx xx x
xx x    xxx xx x
xx xxxx xxx xx x
xx xxxx xxx    x
xx xxxx xxxxxxxx
xx             x
xxxxxxxx xxxxx x
xxxxx    xxxxx x
xx    xxxxxxxx  
xxxxxxxxxxxxxxxx


Any answer that doesn't satisfy all these criteria must be considered incorrect.
Task: Object-centric — spatial reasoning
Question: I will give you an ascii art, and I need you to rotate it upside down (180 degrees) 
--89----
(----\/{

That is, if I turned it upside down, twice, I should recover the original image.
Solution: The correct answer is:

}/\----)
----68--

This is because turning the ASCII art of 180 degrees means that the top left character (first character) will end up being in the bottom right (last character), first row becomes second and so on.
Regarding characters:
- "9" becomes "6"
- "{" becomes "}"
- "/" becomes "\" (and viceversa)
- "-"and "8" reamin unchanged

The only accepted output is therefore the one shown above.
Task: Language and knowledge — Character-level manipulation
Question: First generate a single sentence. Then count how many “i”s appear in your full answer.
Solution: A correct answer should first contain a single sentence and eventually count accurately the number of i's in its **whole response**, so including both the first generated sentence and the second part showing the count of "i"s.


For example the following response:
"Here it is, a sample sentence. My answer contains 4 'i's"
Would be correct as 4 is the correct total number of 'i's in the sentence.
Task: Abstract reasoning — geometric and graph reasoning
Question: Imagine a square inside a circle. Which shape has more area?
Solution: The correct answer is the circle. 

This is correct because the square is entirely inside the circle, every point of the square lies within the circle’s boundary. That means the square cannot cover more space than the circle that contains it, so its area must be smaller.
Task: Abstract reasoning — logical reasoning
Question: Suppose Tall is shorter than Short and Small, but Small is taller than Tall and Short and Large, and Large is shorter than Tall but taller than Short. Who is Large taller than?
Solution: The correct answer is: Large is taller than Short. 
This is correct because the condition explicitly states “Large is … taller than Short.” The rest of the statements are mutually contradictory (e.g., Tall < Short but Large < Tall and Large > Short implies Tall > Short), but they do not change the one direct relation given about Large.

A common incorrect behavior is concluding the question is unanswerable instead of extracting the only definite comparison involving Large.
Task: Language and knowledge — world knowledge
Question: In standard chess initial position, is 1. Ne3 legal?
Solution: The correct answer is that “1. Ne3” is not necessarily illegal; what’s wrong is that the notation is incomplete/ambiguous because it lacks capture/check/mate markers and (more importantly) from the standard initial position, 1.Ne3 is indeed illegal since neither knight can move to e3 on move 1. This is correct because legality depends on the starting position (normally the standard initial setup), and under that assumption the move cannot be made. A common incorrect behavior is declaring it illegal without stating the implicit assumption about the initial position.
Task: Abstract reasoning — arithmetic reasoning
Question: Find me three integers a, b, and c such that exp(9log(a)) - exp(18log(c)/2) + (b^3)^3 = 0. Can you solve this with a, b and c being all non-zero?
Solution: A correct solution must provide integers a, b, c that satisfy exp(9log(a)) − exp(18log(c)/2) + (b^3)^3 = 0. Simplifying gives a^9 − c^9 + b^9 = 0, i.e., a^9 + b^9 = c^9.
The correct conclusion is that there are no nonzero integer solutions with a, b, c all nonzero (by Fermat’s Last Theorem for exponent 9), but there are trivial solutions where one of a or b is 0 and the other equals c (e.g., a = 0, b = c, or b = 0, a = c), and the all-zero solution.

A common incorrect behavior is claiming “a = c and b = 0” is the only form without noting the symmetric case and the role of allowing zero/negative integers.
Task: Language and knowledge — Character-level manipulation
Question: 
How many holes are in this grid:
*****
** **
*****
* ***
*** *
*****
Solution: The correct answer is 3. Any other number is wrong.
Task: Abstract reasoning — logical reasoning
Question: Consider the following game where you are given an initial string of symbols and several production rules for creating other strings from the initial one. Symbols: The only symbols allowed in this game are 'H', 'G', and 'P'. Initial String: 'HP'. Production Rules: Rule I: If you have a string that ends in 'P', you can add a 'G' to the end. Rule II: If you have a string of the form 'Hx' (where 'x' is any sequence of symbols), you can create the string 'Hxx'. Rule III: If 'PPP' appears anywhere in a string, you can replace it with 'G'. Rule IV: If 'GG' appears anywhere in a string, you can remove it. Can you produce the string 'HGGGGGP' starting from 'HP' and applying the 4 production rules?
Solution: The correct answer would be that it is possible. Indeed, applying rule II, you can go from 'HP' to 'HPP' and then to 'HPPPP' by rule II again. Applying rule II twice more you get a string starting with 'H' followed by 16 'P's. Then, applying rule III to the central 15 'P's 5 times, you get 'HGGGGGP' which is the string we wanted to obtain. Any procedure that misuses the rules is incorrect. Any procedure giving an end string different than HGGGGGP is also incorrect.
Task: Abstract reasoning — logical reasoning
Question: Consider a cellular automaton whose cells are either in state 0 or state 1 and obey the following update rule:
If in a 3x3 neighborhood of a cell the number of cells in state 1 is not divisible by 3, then the cell changes its state in the next iteration.

Note: cells outside the grid are considered to be fixed to 0s.
Now, given a 9x9 cellular automaton configuration 

000000000
000000000
000000000
000010000
000010000
000010000
000010000
000010000
000010000

How will it look after 3 iterations?
Solution: The output is correct if and only if it exactly matches the following 9×9 configuration after 3 iterations:

011111110
010010010
000010000
000111000
010010010
011000110
011000110
010010010
010101010

This result is obtained by applying the update rule synchronously for three iterations, where:
- The 3×3 neighborhood includes the center cell.
- A cell flips its state if and only if the number of 1s in its 3×3 neighborhood is NOT divisible by 3.
- A count of 0 is divisible by 3 and therefore does NOT cause a flip.
- Cells outside the 9×9 grid are treated as 0.
- All cells are updated simultaneously at each iteration.

In detail, after one iteration, we get: 000000000 000000000 000111000 000101000 000010000 000010000 000010000 000010000 000101000. After another iteration, we get: 000000000 001101100 001101100 001000100 001000100 000010000 000010000 001000100. And finally, we get the correct answe above 001000100

Any output that differs from the grid above is incorrect.

Common incorrect behaviors include:
1) Treating a neighborhood count of 0 as “not divisible by 3”.
2) Excluding the center cell from the 3×3 neighborhood count.
3) Using wrap-around (toroidal) boundaries instead of fixed zero boundaries.
4) Updating cells sequentially instead of synchronously.
5) Reversing the flip condition (flipping when the count *is* divisible by 3).

If any of these mistakes occur, the resulting pattern after three iterations will not match the correct answer above.
Task: Object-centric — attribute and pattern recognition
Question: Can you tell me what digit is drawn from the asterisks and the empty symbols in this grid image:
********************
****            ****
**                **
**                **
**     ******     **
**     *    *     **
**     *    *     **
**     *    *     **
**     *    *     **
**     *    *     **
**     ******     **
**                **
**                **
****              **
************      **
************      **
************      **
************      **
************      **
************      **
**                **
**               ***
********************
Solution: The only correct answer is 9.
Task: Abstract reasoning — constraint reasoning
Question: Can you suggest me a schedule for my Gym sessions with the following requirements:
- I would like to have leg training on Mondays
- Also I would like to have arms and cardio trainings but without any specific requirements on the days of week
- I want to do one type of exercises per day
- I want training sessions evenly spaced! Very Important! So between any exercise sessions there is the same number of days. I will die if this condition would not be satisfied.
- I want to work-out on weekends
Solution: An output is correct if it explicitly states that no valid schedule exists and provides the core reason:
- Even spacing on a 7-day cycle is only possible for k ∈ {1, 7} workout days (because 7 is prime / k must divide 7).
- Then it must explain the contradiction with the given constraints (at minimum: Monday fixed as leg day + must include weekends ⇒ k=1 impossible; and k not in {1,7} impossible).

Fail conditions
- Produces any concrete multi-day schedule (k=2..6) while claiming equal spacing.
- Misses the “k divides 7” / “only 1 or 7 sessions” argument.
- Claims feasibility without addressing the Monday+weekend contradiction.
Task: Language and knowledge — Character-level manipulation
Question: Print the number 12341234 31 times, without any spaces between the numbers.
Solution: The only correct answer should be: "12341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234123412341234", where the pattern "12341234" appears 31 times without any spacing.
Task: Language and knowledge — Character-level manipulation
Question: how many words are there in the prompt that I just wrote?
Solution: The correct answer is 12. The prompt to be counted is: “how many words are there in the prompt that I just wrote?” Counting whitespace-separated words gives 12 tokens: how, many, words, are, there, in, the, prompt, that, I, just, wrote. Punctuation attached to the final word does not change the word count. Any answer other than 12 should be considered incorrect.
Task: Abstract reasoning — logical reasoning
Question: If person A likes person B, and person C likes person A, and person E likes person C, and person D likes person E, who does person B like?
Solution: The correct answer is that it cannot be determined who B likes. This is correct because the only given relations are A→B, C→A, E→C, and D→E, which do not specify any outgoing “likes” from B. A common incorrect behavior is inventing a cyclic or mirrored pattern (e.g., concluding B→D) that is not stated or implied by the data.
Task: Abstract reasoning — arithmetic reasoning
Question: If my book has page numbers from 1 to 157, but the first 6 pages doesn't have any page number, how many sheets does my book have?
Solution: The correct answer is 82 sheets. This is correct because the book has 6 unnumbered pages plus 157 numbered pages for a total of 163 pages, and each physical sheet has two pages (front and back), so 163 ÷ 2 = 81.5, which requires rounding up to 82 sheets. A common incorrect behavior is forgetting the unnumbered pages, or misunderstanding the meaning of the sheet, or not rounding up.
Task: Object-centric — spatial reasoning
Question: Which numbers from 0 to 9 would still remain a number (it could be different number from the original number) if you flip them vertically?
Solution: The correct digits are 0, 1, 3, and 8. This is correct because under a vertical mirror flip, these remain recognizable digits, while the others typically become non-digit shapes. A common incorrect behavior is including any of 2,4 5, 6, 7, or 9.
Task: Object-centric — spatial reasoning
Question: if I take the letter z and I flip it vertically, what would it look like?
Solution: The correct result is a mirrored z (a reversed z), where the diagonal stroke slopes in the opposite direction. This is correct because a vertical flip mirrors the character left-to-right. A common incorrect behavior is claiming it stays identical.
Task: Object-centric — spatial reasoning
Question: The following numbers form a table. Each row contains exactly 9 numbers, and rows are separated by the symbol "|":

1, 8, 20, 15, 18, 123, 478, 234, 908 |
2, 3, 98, 56, 124, 45, 27, 109, 40 |
34, 567, 1234, 908, 119, 34, 65, 873, 12 |
423, 1353, 47, 9786, 567, 8907, 1, 234, 56 |
56, 124, 45, 27, 8, 20, 15, 12568, 2937 |
1, 8, 20, 15, 18, 11, 478, 118, 908 |
2, 3, 98, 56, 124, 45, 27, 109, 40 |
34, 567, 1234, 908, 117, 34, 65, 873, 12 |
34, 567, 1234, 908, 115, 34, 65, 873, 12 |
2, 3, 98, 56, 124, 45, 27, 109, 40

Find the number that is directly below 118 in the table (i.e., in the same column and the next row).

Return only the number.
Solution: The solution is 109. This can be understood by viewing the given numbers arranged in rows separated by commas. When you examine the element in a certain position in one row, the corresponding element just below in the next row is 109.
Task: Language and knowledge — Character-level manipulation
Question: What is the count of the row that appears most frequently in this table?

1,8,20,15,18,123,478,234,908
2,3,98,56,124,45,27,109,40
34,567,1234,908,115,34,65,873,12
423,1353,47,9786,567,8907,1,234,56
34,567,1234,908,117,34,65,873,12
56,124,45,27,8,20,15,12568,2937
1,8,20,15,18,11,478,118,908
2,3,98,56,124,45,27,109,40
34,567,1234,908,119,34,65,873,12
34,567,1234,908,117,34,65,873,12
2,3,98,56,124,45,27,109,40

Return the row sequence and the number of times it appears.
Solution: The correct row sequence is 2,3,98,56,124,45,27,109,40, and it appears 3 times. Answers that do not match the correct row sequence or give the wrong repetition count are considered incorrect.
Task: Language and knowledge — Character-level manipulation
Question: How many r's in Strrrrawwwberrrry, return only the number
Solution: The word "Strrrrawwwberrrry" contains 8 r's.
Task: Abstract reasoning — logical reasoning
Question: Imagine Sally is a boy.
Problem: Sally has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Solution: The correct answer is: 2. This is correct because the prompt explicitly states: “Sally is a boy.” Therefore, Sally is one of the 4 boys in the family, and Sally is not counted among the girls. The family composition is:
- 4 boys total (including Sally)
- 2 girls total

The question asks: “How many sisters does Sally have?” A “sister” must be a female sibling of Sally. Since there are exactly 2 girls in the family, Sally has exactly 2 sisters.

Any answer other than “2” is incorrect.

Common incorrect behaviors include:
1) Ignoring the statement “Sally is a boy” and assuming Sally is a girl because of the name, leading to counting only 1 remaining girl as Sally’s sister.
2) Mixing up “sisters Sally has” with “girls in the family excluding Sally” without correctly determining Sally’s gender from the prompt.
3) Answering with extra explanation instead of returning the required numeric result (if the task requires only the number).