matsant01/blind-spots-bench huggingface dataset:
Task: Object-centric — spatial reasoning
Question: Rotate the table clockwise
I have a 4x4 table. The table is blank. I colored some of the cells. Cells are labeled with 2 coordinates (i,j), with the first increasing from top to bottom and the second increasing from left to right. Cell (1,1) and (1, 3) in Black. (2,2) and (2, 4) in Blue. (2,3) and (3,3) in red. (3,4) and (4,4) in black. Rotate this table 90 degree clockwise. Now describe which cell is in what color.
Solution: The correct output is the given table rotated 90 degrees clockwise, meaning the first column becomes the first row and so on, whereas the first row becomes the last column, the second row becomes the second-to-last column, and so on. 
The correct solution is therefore that cells (1,4), (3,4), (4,2), and (4,1) are black, cells (2,3) and (4,3) are blue, and cells (3,3), (3,2) are red.
This is correct because a clockwise rotation maps cell (i, j) to (j, 4 − i + 1). A common incorrect behavior is transposing without reversing, rotating counter-clockwise, or leaving some entries in their original positions.
Task: Abstract reasoning — geometric and graph reasoning
Question: There is a circle on a line. Three points on the circle make the perimeter of the circle into 3 equal parts. One point is touching the line named A, and reading the points' names anti-clockwise, it would be A, B, C. The circle spins right (clockwise) for 487 degrees. What point is closest to the line after that?
Solution: The correct answer must account for the fact that the points A, B, and C are labeled in anti-clockwise order and the circle is rotated clockwise by 487 degrees. Since 487 = 360 + 120 + 7, this is equivalent to a 127-degree clockwise rotation, or 120 degrees (one third of a turn) plus 7 more degrees. Therefore, the correct answer is "B". The correct solution identifies which labeled point ends up closest to the line after this rotation. A common incorrect behavior is forgetting that the labels were given anti-clockwise or rotating in the wrong direction.
Task: Abstract reasoning — constraint reasoning
Question: Four people (Alex, Brook, Chris and Dusty) want to cross a river in a 
boat that can only carry 100kg. Alex weighs 90kg, Brook weighs 80kg, 
Chris weighs 60kg and Dusty weighs 40kg, and they have 20kg of supplies.
 How do they get across?
Solution: The correct output is a valid sequence of river crossings that transports Alex (90kg), Brook (80kg), Chris (60kg), Dusty (40kg), and the 20kg of supplies across the river using a boat with a strict 100kg capacity per trip (including both people and supplies). 

This is correct because every crossing must respect the 100kg limit, and the goal is achieved only when all four people and all supplies end on the far side. Any proposed sequence that exceeds 100kg on any trip, leaves any person or the supplies behind, or includes an impossible move (e.g., a “return” by someone who is not on the departure bank at that step) is incorrect. For example, one valid solution is:

1) Chris (60) + Dusty (40) cross.  Total = 100.
2) Dusty (40) returns.            Total = 40.
3) Alex (90) crosses alone.       Total = 90.
4) Chris (60) returns.            Total = 60.
5) Chris (60) + Dusty (40) cross. Total = 100.
6) Dusty (40) returns.            Total = 40.
7) Brook (80) + Supplies (20) cross. Total = 100.
8) Chris (60) returns.            Total = 60.
9) Chris (60) + Dusty (40) cross. Total = 100.

A common incorrect behavior is tracking the state incorrectly (e.g., claiming Dusty returns when Dusty is not on the starting side), ignoring the 20kg supplies in capacity calculations, or presenting a sequence that appears plausible but violates the weight constraint or the per-step feasibility.
Task: Language and knowledge — Character-level manipulation
Question: List 50 words and their definitions. The words should all start with the letter g, but the definitions should not contain the letter g.
Solution: The correct answer is a list of exactly 50 words, where each word starts with the letter “g”, and each word is paired with a definition that does not contain the letter “g” (in either uppercase or lowercase form). 

An output satisfies the instruction if and only if all 50 words begin with “g”, and none of the corresponding definitions contain the character “g”. Any output that violates either the word-initial constraint or the definition constraint is incorrect. 

For example, an answer that includes a definition such as “a strong force” is incorrect because the definition contains the letter “g”, even if the associated word starts with “g”. This error commonly occurs when the model correctly enforces the word constraint but fails to exclude the letter “g” from the definitions.
Task: Language and knowledge — Character-level manipulation
Question: Don't use any external tools. Write “yes” if the number of letters in the word “telecommunication” is even, otherwise “no”.
Solution: The correct answer is “no”. An output satisfies the instruction if and only if it outputs exactly one of the two strings: “yes” (only if the number of letters in “telecommunication” is even) or “no” (only if the number of letters is odd), and it does not use any external tools. The word “telecommunication” contains 17 letters, which is odd, so the correct output is “no”. 

Any output of “yes” is incorrect because it corresponds to an even letter count. A common incorrect behavior is miscounting the letters (e.g., counting 16 or 18 instead of 17) and therefore producing the wrong parity decision (“yes” instead of “no”).
Task: Abstract reasoning — constraint reasoning
Question: A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the farmer transport the goat across the river without it being eaten and without the consideration of the cabbage being eaten? 
Solution: The correct answer is any crossing plan that transports the goat to the opposite bank while ensuring the goat is never eaten. In this variant, the cabbage may be eaten and does not need to be preserved; the only forbidden state is leaving the wolf alone with the goat on either bank when the farmer is absent. An output satisfies the instruction if and only if (1) the goat ends up on the far bank, and (2) at no point is the goat left alone with the wolf without the farmer present.

A valid minimal solution is: the farmer takes the goat across the river in the first trip and stops. This is safe because the start bank is left with the wolf and cabbage (the goat is not there), and the far bank contains the goat with the farmer, so the goat cannot be eaten.

A common incorrect behavior is incorrectly enforcing the cabbage constraint (treating “goat eats cabbage” as a failure even though cabbage loss is allowed), or proposing steps that implicitly leave the goat with the wolf unattended (e.g., having the farmer return while the goat remains with the wolf on the same bank).
Task: Object-centric — spatial reasoning
Question: Alan, Bob, Colin, Dave and Emily are standing in a circle. Alan is on Bob’s immediate left. Bob is on Colin’s immediate left. Colin is on Dave’s immediate left. Dave is on Emily’s immediate left. Who is on Alan’s immediate right?
Solution: The correct answer is: Bob is on Alan’s immediate right. Reasoning: “Alan is on Bob’s immediate left” means Bob is on Alan’s immediate right. Any answer saying that Emily is on Alan's immediate right is incorrect because Emily is on Alan's immediate left.
Task: Language and knowledge — world knowledge
Question: You have three switches. One of them turns on a light bulb, but you don't know which one. The light bulb is inside a room with a glass wall and a door. You can open the door only once to enter the room. How can you figure out which switch turns on the light bulb?
Solution: The correct solution is to use the glass wall (window) to directly observe which switch turns on the bulb, since you can see the bulb from outside the room. For example: flip one switch on and look through the glass wall; if the bulb lights, that is the correct switch. If it does not, flip that switch off and flip a second switch on and look again; if it lights, it is the second switch; otherwise it is the third switch. This is correct because the added condition (glass wall) provides direct visual information, making the classic “heat the bulb” strategy unnecessary. A common incorrect behavior is ignoring the glass wall and using the standard approach (turn one switch on to heat the bulb, then enter once to feel it), which does not take advantage of the new information provided by the prompt.
Task: Language and knowledge — Irrelevant-context robustness
Question: Imagine you are in a dark room. In a drawer there are 10 red socks, 10 white socks, and 10 green socks. The light comes on. How many socks do you need to take out to be sure you have at least one pair of the same color?
Solution: The correct answer depends on the fact that the light comes on before choosing. Once the room is lit, you can deliberately pick socks to guarantee a matching pair with just 2 socks (pick any two socks of the same color). This is correct because the original “dark room” constraint is removed by the light turning on. With full visibility, you can choose two socks of the same color directly, so 2 is sufficient. A common incorrect behavior is treating it as the classic “dark room” version and answering 4 using the pigeonhole principle (one of each color plus one extra), which ignores the modified condition.
Task: Abstract reasoning — arithmetic reasoning
Question: Is 1001 prime?
Solution: The correct answer is: 1001 is not prime. This is correct because 1001 has non-trivial factors: 1001 = 7 × 11 × 13. Therefore, it is composite. A common incorrect behavior is incorrectly declaring it prime without testing divisibility, or making an error in factorization arithmetic and hallucinating a proof.
Task: Object-centric — spatial reasoning
Question: suppose i write the letters "odow" on a piece of paper and hold it in front a mirror. what would i read in the mirror?
Solution: The correct output is what the mirrored text would look like under a left-right reflection. A mirror reverses the order of characters and also flips each character’s shape if it is not symmetric. So you would read the reversed sequence, but note that letters like “d” become “b” in a mirror, and “w” remains “w” under left-right reflection. Therefore, the mirrored reading of “odow” is “wobo”. This is correct because the mirror effect is not only reversing the string but also reflecting letter shapes. Applying both effects yields “wobo”. A common incorrect behavior is only reversing the order to “wodo” (ignoring letter-shape reflection), or claiming it stays the same, or producing inconsistent transformations for “d/b”.
Task: Abstract reasoning — logical reasoning
Question: Given the following mappings: 86569 -> 77951, 11552 -> 40403, 25845 -> 25845, 98511 -> 98511, 70080 -> 92320, 93856 -> 93856, 77951 -> 33256, 21574 -> 32569, 08286 -> 17832, 32569 -> 09623, 21574 -> 21574,43046 -> 43046, 11552 -> 11552, 01461 -> 64408, 85839 -> 08286, 09623 -> 09623, 92320 -> 86569, 17832 -> 92320,58562 -> 80341, 12377 -> 58562, 91691 -> 32529, 01461 -> 01461, 32569 -> 32569, 54285 -> 54285, 18452 -> 98511,1 5786 -> 11552, 40403 -> 36615, 89902 -> 40578, 43046 -> 12377, 25845 -> 43046, 15786 -> 15786, 33256 -> 33256, 40578 -> 93856, 64408 -> 21574, 36615 -> 36615, 12377 -> 12377, 32529 -> 18452, 08286 -> 0828, 98511 -> 70080, 36615 -> 54285, 85839 -> 86569, 91691 -> 70080, 25845 -> 80341. Can you complete this sequence 91691 -> ?
Solution: The correct answer is: 33256. This is correct by transitivity over the provided mapping: 91691 → 70080 then 70080 → 92320 then 92320 -> 86569 then 86569 -> 77951 then 77951 -> 33256 .  A common incorrect behavior is missing the chain or giving a sequence which is not correct and doesn't follow the correcyt path.
Task: Language and knowledge — Character-level manipulation
Question: Consider the following string and invert the order of symbols in prime number positions:
'|"^}*{>%?|&:|$/(|>*~}{}*>+>;/*|)|;.{@~';;%~<+)~

We assume positions start from 1
Solution: The correct output is 
')%^;*{>%?*&;|$/{|>*~}(}*>+>:/||)|;.{@~\'};"~<+|~
Starting from the original string, the order of those symbols located at prime-numbered positions (2, 3, 5, 7, ...) is inverted, while all non-prime positions remain exactly unchanged. This is correct because the instruction restricts modification to prime indices only. A common incorrect behavior is permuting characters at non-prime positions, misidentifying primes (e.g., treating 1 as prime), changing the total character multiset, or shifting characters rather than permuting within the prime-index set.
Task: Abstract reasoning — arithmetic reasoning
Question: Give a smallest non-negative number which can be written as sum of cubes of two different pairs where each pair has different integers
Solution: The correct answer is 0. This is correct because the question allows cubes of integers (including negative integers) and asks for the smallest non-negative number expressible as a sum of cubes of two different pairs. We have: 0 = 1^3 + (-1)^3 = 2^3 + (-2)^3, which uses two different pairs of integers, and the sum is non-negative and minimal. A common incorrect behavior is answering 1729 (the smallest number that is a sum of two positive cubes in two ways), which adds an unstated “positive integers only” constraint.
Task: Abstract reasoning — logical reasoning
Question: Answer the question without using any external tools. In any sentence, if the word INVERT appears then the exact opposite of all the equalities or inequalities before the word INVERT are true. For example, consider the sentence "A = B INVERT C !=D INVERT" then at the end of "A = B, C = D" here != denotes not equal to. Keeping this in mind, consider the following sentence A = B, B != C, C!= D, INVERT D != H, G != B, C = D, E != A INVERT C != H, F = G In the end, is F = A ?
Solution: The final solution is F=A. A correct solution must apply the “INVERT” rule to flip the truth value of all equalities/inequalities that appear before each INVERT in the same sentence, and it must compose multiple INVERT operations correctly. This is correct because the prompt defines INVERT as an operator that retroactively changes the truth status of prior statements. Therefore, the final truth values must be computed by tracking which constraints are flipped an odd vs. even number of times, then determining whether F = A is entailed by the resulting set. 
This produces the relations A=B, B!=C, C!=D, D=H, G=B, E=A, C!=H, F=G. From this it can derived that A=B=G=F. Therefore, by transitivity, F = A. Furthermore, none of the relations contradicts this statement.

Common incorrect behaviors include applying INVERT only to the immediately preceding statement, forgetting that earlier constraints are also affected, failing to compose multiple inversions (treating them independently rather than cumulatively), or finally obtaining F != A.
Task: Abstract reasoning — geometric and graph reasoning
Question: Here is a depiction of a graph. How many connected components are there?

(A)-----------(B) 
|                  |
|                  |
(C)-----------(G)

(D)-----------(E)-----------(F)  (H)
(I)-----------(J) 
 |                  |
 |                  |
(L)-----------(K) 
Solution: The correct answer must count connected components in the given graph exactly as drawn: each component is a maximal set of vertices connected by edges. In this case, the number of components is four.

The correct solution must not assume edges that are not explicitly present in the depiction. This is correct because connected components are determined solely by the existence of edges. Hallucinating an edge between two parts of the diagram changes the component count and makes the answer incorrect. A common incorrect behavior is inventing connections due to visual proximity (e.g., assuming two lines cross or touch when they do not), missing a bridging edge that is present, or counting isolated vertices incorrectly.
Task: Language and knowledge — Character-level manipulation
Question: I will give you a string. You should only close each opening bracket with its matching symbol. Examples:

( -> ), { -> }, [ -> ], so ([{ -> }]).



Do the following sequence: {"{\{""(\(\\([[[{\{((["'[\\{\,([
Solution: The correct answer is then ])}]]))}}]]])))}}}. The correct output is the properly closed sequence for the given opening brackets, matching types and respecting last-opened-first-closed order. Given: {{{((([[[{{(([[{([" You must close in reverse order with the matching closers: ] closes [, ) closes (, } closes {. This is correct because bracket matching is a stack discipline: every opener must be closed by its corresponding closer, and the closing order must be the reverse of the opening order. A common incorrect behavior is producing the right bracket types but in the wrong order, missing some required closers, or outputting the correct order but with the wrong counts.
Task: Language and knowledge — Character-level manipulation
Question: generate a text with exact 31 characters
Solution: The correct answer is any text whose total character length is exactly 31. An output satisfies the instruction if and only if the final generated text contains exactly 31 characters. Any output whose length is not 31 violates the instruction and is therefore incorrect. 

The answer "Silent stars drift through velvet sky." is incorrect because it has 38 characters. By removing the last 7 characters, it can be correct.
Task: Language and knowledge — Character-level manipulation
Question: Count the strokes of my surname 魏 (in Chinese)
Solution: The correct stroke count for the Chinese character “魏” (surname) is 17 under the modern standard stroke-count convention.

A candidate step-by-step analysis is as follows: “魏” can be decomposed into “委” and “鬼”, which have 8 and 9 strokes, respectively. Therefore, the correct output is 17.

A frequent failure mode arises from an incorrect decomposition of the character. For example, if “魏” is wrongly split into “韋” and “鬼”, each counted as having 8 strokes, the resulting total is 18, which is incorrect.
Task: Language and knowledge — Character-level manipulation
Question: In this chat, every time you respond, please reverse the letters of all words. What is the exact temperature currently?
Solution: A correct answer must reverse all tokens in the response, including both text and temperature expressions.

Reversal rule:
Every whitespace-delimited token must have its characters reversed. This applies to alphabetic words as well as tokens containing numbers or units (e.g., “16°C” must also be reversed). Any attached punctuation remains part of the token being reversed.

Temperature rule:
If no real-time temperature information is available, the correct response is to state that the exact current temperature is unknown, while still obeying the reversal rule. If an exact temperature is available, it must be reported correctly and also reversed at the token level. 

For example, the correct answer can be "eht tcaxe tnerruc erutarepmet si nwonknu" or "eht erutarepmet si 61 C°".

Common incorrect behavior includes reversing only alphabetic words while leaving numeric or unit tokens unchanged.
Task: Language and knowledge — Character-level manipulation
Question: Count the number of occurrences of the letter ’L’ in the word -

LOLLAPALOOZALLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL
Solution: The correct answer is: 535 occurrences of the letter “L”.

This is correct because counting all instances of the character “L” in the given string yields 535. Any other count is incorrect.

A common incorrect behavior is undercounting due to skipping long repeated segments or relying on approximate counting.
Task: Language and knowledge — Character-level manipulation
Question: Describe Lausanne in exactly 20 words. The sentence should not contain the letter "r". 
Solution: A correct answer must be a description of Lausanne that contains exactly 20 words and does not include any "r". For example, Lausanne, a hilly, lovely Swiss city on Lake Geneva, has elegant buildings. It is home to many fine schools and the IOC.

This is correct because the task imposes a strict word-count constraint and an additional lexical constraint. Any output with more or fewer than 20 words, or containing any forbidden letter, is incorrect.

A common incorrect behavior is producing fluent descriptions that exceed 20 words or accidentally include the forbidden letter "r".
Task: Language and knowledge — Character-level manipulation
Question: Write a sentence that exactly ends with the the character "m"
Solution: The correct output is any grammatical sentence whose final character is the letter “m” (lowercase m). For example, "A large sum of money was found in the old safe room".

This is correct because the instruction constrains "m" as the last character in the sentence. Any sentence that ends with a different character (including punctuation after “m”) is incorrect.

A common incorrect behavior is placing “m” before a period (ending with “m.”) or ending with another letter.
Task: Abstract reasoning — arithmetic reasoning
Question: There are 100 people in a room, each with a unique number from 1 to 100. The lights are initially off. Person 1 comes in and flips the switch once (on), person 2 comes in and flips the switch twice (off, on), person 3 comes in and flips the switch three times (off, on, off), and so on, until person N comes in and flips the switch N times. After the 100th person finishes, is the light on or off?    
Solution: The correct output is that the light is off after the 100th person finishes.

This is correct because each person flips the switch a number of times equal to their number, so the total number of flips is
1 + 2 + 3 + \dots + 100 = \frac{100 \cdot 101}{2} = 5050.
Since 5050 is even, the switch returns to its original state (off).

A common incorrect behavior is giving the right-looking conclusion for the wrong reason, or producing a plausible-but-wrong reasoning like: “Because 100 is even, the 100th person ends with an even number of flips, so the light must be off,” which ignores that the final state depends on the total flips by everyone, not just the last person. Another wrong reasoning is: “Half the people flip an odd number of times and half flip an even number of times, so it cancels out,” which is not a valid parity argument.
Task: Language and knowledge — Character-level manipulation
Question: use 10 chinese characters to describe love (in chinese).
Solution: The correct output is a description of love written in exactly 10 Chinese characters.

A common mistake is providing a meaningful but overlength sentence, such as “爱是心心相印，携手共度人生。”, which exceeds 10 characters and therefore violates the requirement even though the description itself is appropriate.
Task: Language and knowledge — Character-level manipulation
Question: Invert this string ###><##?/####
Solution: The correct output is the inverted string where the character order is reversed; for “###><##?/####” the correct output is “####/?##<>###”. This is correct because inversion here corresponds to reversing the sequence of symbols. A common incorrect behavior is swapping only “<”/“>” or “#”/“/” while keeping order, or reversing only a substring.