Idea Track:Deciphering the code of the novel“The Dancing Men”
[Teams name]: atHome
K. Murakami (Fujitsu Ltd.) murakami.ktk@gmail.com
A. Kishida (KCNKMC), H. Ito (Univ. of Helsinki),
S. Matsumoto (Tokyo Tech), K. Takamatsu (Tokyo Tech)0

Copyright 2023 FUJITSU LIMITEDSummary (key points)◼ This study examines a method for deciphering the handwritten
dancing men graphic shown in "The Dancing Men."◼ We attempted to reproduce and explain Holmes's reasoning for thegraphic cipher by using a mathematical programming approach.
◼ Moreover, we studied the validity of Holmes’s reasoning anddemonstrated a procedure to decipher the cipher computationally.
◼ Herein, we propose a mathematical solution to clarify the ambiguityin Holmes' conjecture.
◼ Although the proposed method has not been implemented, its
feasibility is studied through partial data creation and analysis.

1

Copyright 2023 FUJITSU LIMITEDBackground and Policies of the Proposal◼ The objective of this challenge (workshop) is to develop explainableartificial intelligence (XAI).
◼ The task here is to infer the truth of a case (reproducing Holmes'sreasoning), while providing a reasonable explanation based on therelevant open knowledge graph.
◼ Our analysis focused on the decipherment of the handwritten
graphic cipher in the novel "The Dancing Men" by Arthur Conan
Doyle.
◼ We did not use knowledge graphs in this study. Instead, weconsidered a method to reproduce and explain Holmes's reasoning
by a computational and statistical approach.

2Copyright 2023 FUJITSU LIMITEDProblem◼ In the story, the six messages shown below are presented insequence.
◼ The letters of the alphabet written under the men indicate thecorrect answer. The correspondence between one man and one
letter is elucidated later in the story, as shown in the figures.
◼ Holmes deciphered them when he obtained the first five sentences.

(3)

(2)

(1)

(5)

(4)
(6)

Fig. 1 Graphic code (image data from [1])3

Copyright 2023 FUJITSU LIMITEDHolmes' deciphering points◼ Assumption: Symbols are substitutes for letters (“substitution cipher” type).
◼ (Holmes is well versed in cryptography, has written novels and analyzed 160 differentcryptography methods).

◼ Summary of Reasons of the deciphering◼ Reasoning by letter frequency: 2 times (high objectivity)•E is the most commonly used letter in the alphabet•Next used are T, A, O, and I.
◼ Victim's name, appears 3 times at the end: 1 time (high objectivity)◼ Likely as a reply, makes sense, fits the situation: 6 times (low objectivity, low specificity)◼ Examine the following questions first◼ Is the character frequency information really useful as deduced?
◼ how can we tell if they are the same letters in the pattern of the men’s figures.

4

Copyright 2023 FUJITSU LIMITEDIs the probability of alphabetical appearance useful?
(Original count from Wikipedia text 11,055,558,613 characters)• The highest frequency letter is "e," which, as in the novel, stands out infirst place.
• Then two "a", "t", or seven gradually decrease with close frequency (sevene, a, t, i, n, o r, s are high)5

Copyright 2023 FUJITSU LIMITEDSimulation of how many top 4 letters (e,a,t,I) rank in order of frequency
1.

Generation of 15 characters in length as in the first encrypted sentence.
Simulation: Multinomial distribution with prior distribution, 1000 times

2.

Plotting the distribution of rankings

3.

Percentages were verified with pie charts.
frequency

Rank (1~15)
The numbers
are ranks.

Et

A

I

E appears most often only in half of the cases (left figure), and A and T are also in firstplace in many cases (including ties). The method (assumption) that "the most common
letter is E" is half correct, but it cannot be used as it is in general.
6Copyright 2023 FUJITSU LIMITEDDifficulties with image analysis problems: are theseidentical or not?
Example 1

Example 3Legs, it's hard to judge if the body is leaning
the same way.
In the novel, they are considered the same inE, E .

It is difficult to determine if the feet are
floating and the same or if the knees are bent
differently.
In the novel, it is different from I, O (I-5-4, O-523)

Example 2

It is difficult to determine if the feet are floating and thesame, or if the knees are bent differently.
In the novel, the same is deciphered in A (a-1-1, a-1-12)
7

Copyright 2023 FUJITSU LIMITEDProblem setting again◼ The following is a mystery, although it was not explained in the story◼ How do we know that similar letters are the same letter?
• The characteristics of the images that should be distinguished are not known inadvance.
(In other words, it is not obvious that the similar men are the same characters.)• Figure features include those not related to character type (e.g., flags, body lean)• All 26 letters are not appeared.

◼ So, the following problems should be solved simultaneously◼ Mapping between a figure and an alphabet → sentences are determined◼ What are the critical features of dancing poses to identify the “men” images

8

Copyright 2023 FUJITSU LIMITEDOur approach◼ method
1.

Coding partial features of the images (human judgment here)
1.
2.

2.
3.

Binary variable to set whether each image has the feature or not
Binary variables that should determine the correspondence between figure andtext

Create a relatively small dictionary (list of candidate words)
Give higher scores to words that are likely to fit the following situations• idiomatic usage• Words with many letters• Weight the words that might appear.
• Names of people and places unique to the story• Words that indicate monetary value (property, bank, transaction, criminal plan, safe, etc.)• Words that convey emotions (affection, resentment, etc.)• Verb (to convey a command or request)• Bad language, harm (causing a frightened reaction)

4.

Find the solution by optimization problem using the string fitting position
and features as variables.
9Copyright 2023 FUJITSU LIMITED

Similarity coding between images and 13 attributeitems, named “feature table”.
•••••

We have detarmined the features of each figure by humans.
Same characters have the same man's characteristics (right leg up, etc.)
Other character verbs must have a different feature set.
Meaning of the “flag” should be derivable by the method.
Features are used as variables (1, 0) for optimization, so even if there are features that are not used, it
is possible to determine whether to ignore them.
rightarmcorrectcharacter

AAAAAABCCCDEEEEEE

position ofappearance
AboveSongsandandbelow
Yes/N Noble (if (foroany)songs)
1_1
1-1
1_7
1-1
1_12
1-1
2_1
1-1
4_9
1-1
5_10
1-1
1_8
-1
3_1
-1
4_1
-1
4_13-1
5_24
1-1
1_4
1-1
1_14
1-1
2_3
1-1
2_8
1-1
3_5
1-1
3_9
1
-1

one's
left
right
left
orient
arm
foot
foot
ation flag
Yes Songs Above/No andandHip
Knee
Hip
Knee
Noble below flexion/ flexion/ flexion flexion/ Upper(if any) (forextensi extensio /exten extensio &Yes/N Right/songs) onnsionnLower oLeft1-1-1
1-1-1
1-1
1-1
11-1-1
1-1
11-1
11-1-1
1-1
1-1-1
1-1-1
1-1
1-1-1
1-1-1
1-1
1-1
1
1-1-1
1-1-1
11-1
1
1-1
1-1-1
11
11-1
1-1-1
1
1
11-1
1-1-1
1-1
11-1-1-1-1-1-1-1-1
1-1-1-1-1-1
1-1
1-1-1-1-1-1
1-1
1-1-1-1-1-1
1-1
1-1-1-1-1-1
1-1
1-1-1-1-1-1
1-1
1-1-1-1-1-1
1-1
10Copyright 2023 FUJITSU LIMITED

Created Data Table

In total, we have 76 letters (yellow area is a group of letter Aand letter E).

As an integer optimization problem◼ Objective function (maximization)𝑓 𝑥 =𝑀+𝐷◼ 𝑀 is the agreement score based on the correspondence between image features (orsimilarity) and characters.
◼ 𝐷 is the word score based on the used word in the dictionaryvariable.

◼ Variables for M◼ 𝑢𝑘,𝑙 : if an image 𝑘 corresponds to character 𝑙, 𝑢𝑘,𝑙 = 1, (otherwise 0).
(26 alphabets, 76 images)◼ 𝑚𝑖,𝑗 : if feature 𝑗 is used by character 𝑖, 𝑚𝑖,𝑗 = 1, (otherwise 0).
• Constraint: σ𝑛𝑖=1 𝑚𝑖,𝑗 = 1. (One image only represents one character.)◼ Variables for D◼ 𝑤𝑞,𝑟 : if word 𝑞 is adopted at position 𝑟, 𝑤𝑞,𝑟 = 1, (otherwise 0).

◼ 𝑠𝑡,𝑣 : if character 𝑡 is set at position 𝑣, 𝑠𝑡,𝑣 = 1, (otherwise 0).
• These are used for meets the conditions below:
1.
2.

The same code vector can be used for the same character.
All the characters match the word in the dictionary.

◼ 𝑝𝑞 : preferred score of the word 𝑞.
11

Copyright 2023 FUJITSU LIMITEDDiscussion◼ Holmes' reasoning uses various human insights (miscellaneousknowledge), but this proposal is based on computational processing(dictionary search and score calculation) and mathematicalprocedures, which would be highly explanatory◼ Analysis of this method may reveal the following◼ Difficult to decipher with only objectively presented information• The terms of the information presented are vague.
• Conditions and information that greatly aided in deciphering were presented at thesame time as or after the resolution.

12

Copyright 2023 FUJITSU LIMITEDConclusion◼ The proposal outlines a procedure for mechanically decrypting thecipher.
◼ The mathematical procedure part of solving the cryptographic
problem is an existing optimization method.
◼ feature◼ Squeezing the dictionary down to a smaller size squeezed the amount ofcalculations.
◼ Contextual considerations were included in the score of likely words.
◼ It can also answer the question, "What are the features used to identify thealphabets of the images?"◼ Theoretically, Holmes' estimated answer should be obtained as asomewhat high-scoring solution.
◼ At this point, it has not yet been implemented.

13

Copyright 2023 FUJITSU LIMITEDReferences◼ [1] The Arthur Conan Doyle Encyclopedia, "The Adventure ofthe Dancing Men," 1903. https://www.arthur-conandoyle.com/index.php/The_Adventure_of_the_Dancing_Men(accessed Nov. 17, 2022).

14

Copyright 2023 FUJITSU LIMITEDCopyright 2023 FUJITSU LIMITED