matsant01/blind-spots-bench huggingface dataset:
Task: Abstract reasoning — logical reasoning
Question: Design a word ladder from "molten" to "deadly"
Solution: Here is an example of valid word ladder from "molten" to "deadly": 
MOLTEN → MOLTER → MELTER → PELTER → PETTER → SETTER → SETTEE → SETTLE → NETTLE → NETTLY → NEATLY → NEARLY → DEARLY → DEADLY 

This might not be the shortest possible solution.

Here is an example of incorrect solution instead:
MOLTEN → BOLTEN → BOLDER → BALDER → BALLEY → DALLEY → DALLY → DAILY → DAIRY → DEAIR → DEARY → DEADY → DEADLY

This solution is incorrect for multiple reasons:
1. It uses words that are not in the dictionary, such as "BOLTEN"
2. It includes illegal moves such as "BOLTEN → BOLDER" where two letters are changed at once
Failure modes: I can see that it is more logical reasoning because we have a sequence of logics
Task: Abstract reasoning — logical reasoning
Question: Construct a word ladder with the words "bead", "mall", "down", "cyst", and "find" in it
Solution: Here is an example of valid word ladder containing all the requested words (BEAD, MALL, DOWN, CYST and FIND):

**BEAD** → MEAD → MELD → MELL → **MALL** → MOLL → DOLL → DOWL → **DOWN** → DOWT → DOST → COST → **CYST** → CIST → FIST → FISK → FINK → **FIND**

This was constructed by building several word ladders connecting pairs of input words. Note that the order here is preserved but this was not a requirement.

Here is instead an example of incorrect solution:

**BEAD** → HEAD → HEAL → TEAL → TELL → TALL → **MALL** → MALE → MILE → MILD → MINT → MIST → CIST → **CYST** → CIST → MIST → MINT → MIND → **FIND** → FINE → DINE → DONE → DOWE → **DOWN**

This solution is incorrect because it contains invalid moves, such as "MILD → MINT" where two letters are changed in a single step.
Task: Abstract reasoning — arithmetic reasoning
Question: Consider the sequence 1, 11, 31, 1341. This is not the standard look-and-say sequence. Infer the rule from the given terms and write the next two terms.
Solution: The rule counts digit frequencies over the entire sequence seen so far, not only the previous term. Each new term is written as count-digit pairs ordered by digit value from largest to smallest. From 1, 11, 31, 1341, the cumulative counts are 1 four, 2 threes, and 6 ones, giving 142361. After adding that term, the cumulative counts are 1 six, 2 fours, 3 threes, 1 two, and 8 ones, giving 1624331281. The only correct next two terms are 142361 and 1624331281.
Task: Language and knowledge — Character-level manipulation
Question: Solve a cryptic clue with a hidden meaning. 
"Sign of pets covered up by homeowner" (4 letters)
Solution: A clue provides a definition and a separate wordplay that leads to the same answer, often through devices like hidden words, anagrams, or double meanings. Solvers interpret both parts to reach a four-letter solution consistent with the clue’s structure.

Answer: MEOW

Explanation:
- Definition: “Sign of pets” → a meow is a sign/sound that a (pet) cat is around.
- Wordplay: “covered up by homeowner” indicates a hidden word.
        hoME OWner → the letters MEOW are hidden inside “homeowner”.

So the hidden “sign of pets” inside “homeowner” is MEOW.
Task: Object-centric — spatial reasoning
Question: Alice is supposed to meet Bob at 18:00. She wakes up from her afternoon nap, looks at her watch, which seems to say it's already 22:55, and panics for a minute. She then realises something and calms down. Why?
Solution: Alice realized that because she was groggy from her nap, she was looking at her watch the wrong way.

There are multiple ways this can work out. In all of them Alice is wearing an analog watch (one with hands), not a digital one:

1. Option: 
* The Real Time: It was actually 17:25 (5:25 PM). At this time, both the hour hand and minute hand are pointing roughly towards the number 5 (bottom right of the watch face).
* The Mistake: Alice looked at her watch upside down. When the "5" position is flipped 180 degrees, it sits in the "11" position (top left).
* The Illusion: Hands pointing at the 11 look like the time is 22:55 (10:55 PM).

2. Option:
* The Real Time: It was actually 13:05 (1:05 PM). At this time both the hour hand and minute hand are pointing roughly towards the number 1 (top right of the watch face).
* The Mistake: Alice looked at her watch through a mirror, such that in her perspective the left and right sides of the watch were inverted.
* The Illusion: Hands pointing at the 11 (top left of the watch) look like the time is 22:55 (10:55 PM).

The Result: she initially panicked thinking she was nearly 5 hours late. Once she realized the orientation was wrong, she saw it there was enough time left to get to her 18:00 meeting.
Task: Abstract reasoning — logical reasoning
Question: The barber is the "one who shaves all those, and those only, who shave themselves". Who shaves the barber?
Solution: The correct answer is that both options are possibly correct and there is no way to say which one it is.

A common incorrect behavior is stating that it is necessarily the barber who shaves himself, or that there is no logical answer to the question.
Task: Language and knowledge — Irrelevant-context robustness
Question: Sally places a marble in her basket and then leaves the room. While Sally is gone, Anne moves the marble from the basket to a box and seals it. While Anne is gone, Sally moves the box to the kitchen. Where will Sally look for her marble when she returns?
Solution: Sally will look in the basket. This is correct because Sally did not observe Anne moving the marble to the box, so Sally’s belief remains that the marble is where she left it (the basket). Moving the sealed box later does not update Sally’s belief about the marble’s location. A common incorrect behavior is answering “the box” or “the kitchen” by using the true location rather than Sally’s belief.
Task: Language and knowledge — Irrelevant-context robustness
Question: A house has all four walls facing south. A bear walks past the house. What color is the house?
Solution: The correct answer is that the color of the house cannot be determined from the information given. This is correct because the “four walls facing south” constraint implies a location near the North Pole (so the bear is likely a polar bear), but nothing in the prompt specifies the house’s color. A common incorrect behavior is answering “white” by confusing the bear’s color with the house’s color.
Task: Object-centric — spatial reasoning
Question: Norway, Sweden, Finland, Denmark and Russia have borders with each other in a circle. Norway is on the left border of Sweden, Sweden is on the left border of Finland, Finland is on the left border of Russia and Russia is on the left border of Denmark. Who is on the right border of Norway?
Solution: Sweden is on the right border of Norway. This is correct because “Norway is on the left border of Sweden” directly implies Sweden is on the right border of Norway. A common incorrect behavior is trying to complete the cycle and naming a different country instead of using the given relation.
Task: Language and knowledge — Character-level manipulation
Question: What's the 3-th word of your response going to be?
Solution: The third word of my answer is "word".
Task: Abstract reasoning — logical reasoning
Question: A young mother was rushed to the hospital emergency room. The ER doctor saw her and said, “I can’t operate on this girl, she is my mother.” But the doctor was not the girl's mother. How is this possible?
Solution: The answer to this riddle is: the doctor is the patient's son or daughter.

This riddle is a variation of the classic "Surgeon's Riddle" designed to test unconscious biases.
- The Logic: If the doctor says, "She is my mother," the doctor must be the patient's child.
- The Distraction: The text specifies, "The doctor was not the girl's mother." This is a redundant statement (a child cannot be their own mother), but it tricks the brain into looking for a complex relationship or searching for a female relative.
Task: Language and knowledge — world knowledge
Question: What has no holes and holds water?
Solution: The question has a lot of possible correct solutions. Some correct answers could be a balloon, a bucket, a bowl, or a coconut. 
The question is a variation of a famous riddle which asks the opposite question, being “what is full of holes but still holds water?”, for which the correct answer is “sponge”.  The variation asks instead for an object devoid of holes. The answerer might be misled by this variation and respond with the answer to the more famous original question. Although, in this case, “sponge” is the most incorrect answer as sponges are by definition “full of holes”. Any object lacking any kind of hole, but shaped in such a way that it can contain water it’s a valid answer.
Task: Abstract reasoning — logical reasoning
Question: Let's say you are a fourth-generation local in a certain country. Also, let's say that your full blood aunts/uncles from both sides of the family tree are third-generation locals in Britain. Where were you born?
Solution: The correct answer is: Britain.

This is correct because if you are a fourth-generation local in a country, your parents are third-generation locals there, and having full-blood aunts/uncles on both sides who are third-generation locals in Britain forces that country to be Britain.

A common incorrect behavior is saying “it depends on what the ‘certain country’ is”.
Task: Abstract reasoning — arithmetic reasoning
Question: Let's say you are doing grocery shopping. The bread cost 6.99 Euros, the cheese - 2.59 Euros and the fruits/vegetables - 9.98 Euros. How much will you be charged by the cashier? Give the answer first in Euro (France, prices include tax) and then in USD (Massachusetts, considering applicable sales tax rules).
Solution: The correct answer is: €19.56 in France, and about $20.78 in Massachusetts (any close value is acceptable).

This is correct because the total in euros is 6.99 + 2.59 + 9.98 = 19.56 €. In Massachusetts, prices are shown before tax, so adding the 6.25% sales tax gives approximately $20.78, and any close value that accounts for adding tax is correct.

A common incorrect behavior is giving the same numeric total for USD without adding tax, or converting euros to dollars instead of applying sales tax.
Task: Abstract reasoning — arithmetic reasoning
Question: Let's say you have a company with 20 board directors, each with the same executive power. The board chooses the CEO from its members by voting. The CEO service time is 5 years. If one board member gets elected twice, the company will stagnate. Is it possible for the company to not stagnate, given that the board directorate is for life and the CEO has to get elected immediately after the end of the service time of the previous CEO?
Solution: A correct answer must conclude that permanent avoidance of stagnation is impossible. It should recognize that there are only 20 eligible CEOs, that each can serve at most one term without triggering stagnation, and that this yields at most 20 distinct CEO terms, or 100 years total. The answer is incorrect if it claims infinite avoidance is possible, ignores the finite number of board members, or does not infer that the 21st CEO term must necessarily repeat a previous CEO.
Failure modes: more combinatorial reasoning
Task: Language and knowledge — Character-level manipulation
Question: Repeat "a" 38 times
Solution: The correct answer should be aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa where the letter a is repeated 38 times.
Task: Abstract reasoning — constraint reasoning
Question: Suppose you are playing chess. Blacks have a king in h8 a pawn in h7 and a pawn in g7 and a knight in c6 and a bishop in a8. white have a queen in a1 and a pawn in b5 and a king in h2 and a rook in h1. you are playing with the whites. what move to play to win ?
Solution: The correct answer is: Qa8#.

This is correct because after 1. Qa8+, the black king on h8 has no legal escape squares and no piece can capture the queen or block the check, so it is checkmate.

A common incorrect behavior is suggesting 1. Rh7# because the rook on h1 cannot move to h7: squares h2–h6 are not empty (at least h2 is occupied by White’s king), so that move is illegal.
Failure modes: more logical reasoning
Task: Language and knowledge — world knowledge
Question: You are in absolutely empty room - no windows, no doors, no other objects rather than mirrors. How many times will you see your reflection there?
Solution: The correct answer is: 0.

This is correct because in a completely closed room with no light source, you can’t see anything at all, so no reflections are visible.

A common incorrect behavior is answering “infinitely many” by assuming a fully mirrored room and ignoring the missing-light condition; infinite recursion only matters if there’s light to make the reflections possible.
Task: Language and knowledge — Irrelevant-context robustness
Question: The inflation in Venezuela was 136% in March 2025. I was born in the last month of the year, which was the last year of the twentieth century according to some systems and the first year of the new century according to others. It was a very cold day. Six year later on this day, my uncle gifted me my first cat. The number of the day is composed of two same digits, and the date is close to Catholic Christmas but far from Orthodox Christmas. The cat was 13 years old when it ran away, but thankfully was found by our neighbors. I am passionate about space research. How many 3s are in my full birthdate?
Solution: The correct answer is: 0, because the birthdate is 22-12-2000 (December 2000, with day 22), and this full date contains no digit “3”.

A common incorrect behavior is being influenced by unrelated details in the prompt (such as inflation figures, the cat’s age, or personal interests) and using them to alter the birthdate, leading to an incorrect count of the digit 3.
Task: Language and knowledge — Irrelevant-context robustness
Question: You have a shirt and a pants, but you are not sure what type of shirt or pants they are. Your friend runs:
python
import math
math.factorial(3) // math.gcd(8, 12) + sum(range(4))
and says this is related to the clothing. Calculate how many clothing you have in total right now, and answer with an integer inside <answer> tags.
Solution: The correct answer is: 2, because you have one shirt and one pair of pants. The Python code is unrelated to the clothing count.
Task: Abstract reasoning — arithmetic reasoning
Question: You’re on a game show with three doors. Behind one is a car; the others hide goats. You pick a door, and the host (who knows what’s behind each door) opens your door to reveal a goat. Should you switch your choice?
Solution: The correct answer is: Switching is okay, but it’s not guaranteed; the probability is 1/2 vs 1/2.

This is correct because once your chosen door is opened and shown to be a goat, the car is equally likely to be behind either of the two remaining unopened doors.

A possible incorrect behavior is claiming switching guarantees a win because “your door is confirmed a goat,” but after it’s opened you are no longer choosing among three doors—only two unopened doors remain, with no information favoring one over the other.
Task: Abstract reasoning — arithmetic reasoning
Question: In the base 10 numerical system imagine we change the symbols of 3,8,5 to 5,3,8 . It is just the symbols not the actual value that they are presenting, What would be the result of the following expression in the new system, the symbols in the following expression are in the base 10. 284 + 53 ?
Solution: The correct answer is 292. In the new system the label of number 3 is changed to 5, 8 to 3 and 5 to 8. The expression  284+53 in the new system corresponds to the numerical value 254+38=292. The number 292 does not involve 3,5 or 8 so it the same in the new and original system.
Task: Abstract reasoning — arithmetic reasoning
Question: Use exactly sixteen digit 4s to make 1000. You may use only addition signs between terms.
Solution: 1000 = 444+444+44+44+4+4+4+4+4+4. Any response in which sixteen instances of the digit 4 are combined using mathematical operations  to obtain an expression evaluating to 1000 is correct. Any response using a different amount that 16 instances of the digit 4 is incorrect. Any response giving an expression that does not evaluate to 1000 is also incorrect. Other digits except 4 are not allowed.