Title: CORTICAL-INSPIRED OPEN-BIGRAM REPRESENTA-TION FOR HANDWRITTEN WORD RECOGNITION

Abstract: Recent research in the cognitive process of reading hypothesized that we do not read words by sequentially recognizing letters, but rather by identifing openbigrams, i.e. couple of letters that are not necessarily next to each other. In this paper, we evaluate an handwritten word recognition method based on original open-bigrams representation. We trained Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) to predict open-bigrams rather than characters, and we show that such models are able to learn the long-range, complicated and intertwined dependencies in the input signal, necessary to the prediction. For decoding, we decomposed each word of a large vocabulary into the set of constituent bigrams, and apply a simple cosine similarity measure between this representation and the bagged RNN prediction to retrieve the vocabulary word. We compare this method to standard word recognition techniques based on sequential character recognition. Experiments are carried out on two public databases of handwritten words (Rimes and IAM), an the results with our bigram decoder are comparable to more conventional decoding methods based on sequences of letters.

1. INTRODUCTION:

Taking inspiration in Biology is sometimes very efficient. For example, deep neural networks (NN) -which are outperforming all other methods (including support vector machines, SVM) in image recognition -are based on a series of several (usually 5 to 15) neurons layers, each layer involving sparsity in the activation pattern (a biological trait of the cortical map). The analogy continues with the modeling of the cortex as a hierarchy of cortical maps. Thanks to the analysis of reaction time in cognitive psychology experiments, the minimal number of cortical maps involved in a cognitive process is estimated to about ten, the same order of magnitude as the number of layers in deep neural networks for computer vision tasks. In the case of handwritten word recognition, Dehaene et al. have proposed a biologically plausible model of the cortical organization of reading (Dehaene et al., 2005) that assumes seven successive steps of increasing complexity, from the retinal ganglion cells to a cortical map of the orthographic word forms (Figure 1 ). One of the most recent successes of experimental psychology was the demonstration that human visual word recognition uses an explicit representation of letter position order based on letter pairs: the open-bigram coding (Whitney et al., 2012; Gomez et al., 2008; Grainger & Van Heuven, 2003; Glotin et al., 2010; Dufau, 2008) .

As demonstrated in (Touzet et al., 2014) , open-bigrams (OB) allow an over-coding of the orthographic form of words that facilitates recognition. OB coding favors same length words (i.e., neighbors of similar lengths). In the context of learning to read, the existence of the OB layer just before the orthographic word representation has been used to explain the lack of efficiency of whole language method (today banned from reading teaching) compared to the phonics method which explic-Figure 1 : The cognitive process of reading, a seven steps procedure that includes an open-bigrams representation layer. Additional information helps the organization of levels 4 and 5, when using a phonics method, but not a whole language method (today banned from reading teaching for lack of efficiency, adapted from (Dehaene et al., 2005) and (Touzet, 2015) .

itly supervises the organization of the OB map (with syllables), where the global method does not (Figure 1 ).

Since cognitive psychology has demonstrated the existence of the OB layer, the hypothesis has been put forward (Touzet et al., 2014) that the orthographic representation of words may have evolved in order to take into account the topology of the OB space, instead of the topology of the single letter space. Our goal here is to test this hypothesis, comparing OB vs sequential character recognition for word recognition. A state-of-the-art decoder based on a Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) is used on two public databases of handwritten words (Rimes and IAM).

The remaining of this paper will be divided as follows. In Section 2, we present related methods for handwritten word recognition. Then, we describe the open-bigram representation of words and the proposed decoder in Section 3. The experimental setup, including the data and the bigram prediction model, is explained in Section 4. Finally, we present our results in Section 5, before concluding in Section 6.

2. RELATED WORK:

In this section, we give a brief overview of existing techniques for handwritten word recognition. Historically, the methods may be divided in three broad categories. The first approach is whole word recognition, where the image of the full word is directly classified into word classes, without relying on the character level (e.g. in (Parisse, 1996; Madhvanath & Govindaraju, 1996) ). In the second method, the word image is segmented into parts of characters (stokes or graphemes). The segments are grouped and scored, and character sequences are obtained with a graph search (e.g. in (Bengio et al., 1995) ) or with hidden Markov models (HMMs, e.g. in (Knerr et al., 1998) ). The last method, most popular nowadays, is a segmentation-free approach. The goal is to predict a character sequence from the image without segmenting it first. The techniques include scanning a sliding window to extract features used in an HMM (e.g. in (Kaltenmeier et al., 1993) ), or to feed the image to a neural network able to output sequences of character predictions (e.g. SDNNs (LeCun et al., 1998) or MDLSTM-RNN (Graves & Schmidhuber, 2009) ).

More recently, different approaches have been proposed to recognize words using character bigrams, and therefore closer to the method we propose in this paper. Jaderberg et al. (2014) propose to predict both the characters and ngrams of characters with two distinct convolutional neural networks (CNNs) to recognize text in natural images. Their approach includes a conditional random field as decoder. Similarly, Poznanski & Wolf (2016) train a CNN with a cross-entropy loss to detect common unigrams, bigrams or trigrams of character in a handwritten word image. The output of the network is matched against the lexicon using canonical correlation analysis. Almazán et al. (2014) use Fisher vectors from images and pyramidal character histograms, to learn a feature space shared by the word images and labels, for word spotting, also using canonical correlation analysis.

3. PROPOSED METHOD:



3.1. AN OPEN-BIGRAM REPRESENTATION OF WORDS:

The letter bigrams of a word w is the set of pairs of consecutive letters. The open-bigram of order d is the set of pairs of letters separated by d other letters in the word, which we call B d (w):

B d (w) = {w i w i+d : i ∈ {1 . . . |w| -d}}.

(1)

The usual bigrams are open-bigrams of order 1. By extension, we call B 0 (w) the set of letters in the word w. For example, for word word, we have:

B 1 (word) = {or, rd, wo} ; B 2 (word) = {od, wr} ; B 3 (word) = {wd}.

The general open-bigram representation of a word is the union of

EQUATION

For example, B 1,2,3 (word) = {od, or, rd, wd, wo, wr}.

We extend B into B by including special bigrams for the letters at the beginning and end of a word:

EQUATION

So, for example, B 1,2,3 (word) = { w, d , od, or, rd, wd, wo, wr}.

In this paper, we will call B the set of all bigrams, and W the set of all words. We will represent a word of the vocabulary w ∈ W as a normalized binary vector

EQUATION

i.e. the vector with 0 everywhere and 1/ |B(w)| at indices corresponding to bigrams of the word. The stacking of the vector representation of all the words in the vocabulary yields the vocabulary matrix V ∈ |W |×|B| .

Note that in this representation, the bigrams form an unordered set. We do not know: (i) where the bigrams are, (ii) what is the order of a given bigram, (iii) how many times it occurs. The goal is to build a word recognition decoder in the bigram space.

3.2. AN OPEN-BIGRAM DECODER:

While the trivial representation of a word is an ordered sequence of letters, the order in the bigram space is locally embedded in the bigram representation. Most state-of-the-art word recognition systems recognize sequences of letters, and organize the vocabulary for a constrained search as directed graphs, such as prefix trees, or Finite-State Transducers. On the other hand, we can interpret the bigram representation as encoding directed edges in a graph, although we will not explicitly build such a graph for decoding.

On Figure 2 , we show the graph for a representation of the word into a sequence of letters. Gray edges show the potential risk of a misrecognition in the letter sequences. On Figure 2 (b), we display the conceptual representation of bigrams as edges. We observe that a global order of letters can emerge from the local representation. Moreover, the constituent information of a word in the bigram space is redundant, potentially making this representation more robust to mispredictions of the optical model.

The optical model is the system which provides the predictions of bigrams from the image (or, in the classical approach sequences of character predictions). That is, it provides a confidence measure that each bigram b is present in image x: 0 ≤ p b (x) ≤ 1. This is transformed into a vector in the bigram space:

EQUATION

)

For decoding, we chose the very simple cosine similarity between the query (q x ) and a vocabulary word (v w ). Since we normalized both vectors, this is simply the dot product:

EQUATION

so the similarity with all words of the vocabulary can be computed with a matrix-vector product:

EQUATION

The recognized word is the one with maximum similarity with the query:

EQUATION

We carried out a few preliminary experiments to justify the open-bigram decoder. First, we considered the famous sentence with mixed up letters:

"aoccdrnig to a rscheearch at cmabrigde uinervtisy it deos not mttaer in waht oredr the ltteers in a wrod are the olny iprmoatnt tihng is taht the frist and lsat ltteers be at the rghit pclae the rset can be a toatl mses and you can sitll raed it wouthit porbelm tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef but the wrod as a wlohe".

Although the origin and validity of this statement when letters are put in the right order has been discussed1 , it is true that most of us can read it without trouble. For each word of more than one letter in this sentence, we computed the open-bigram representation (d = 0..3), and replaced it with the word having the highest cosine similarity in the English vocabulary described in the next section. The result was:

"according to a researcher at abridged university it does not matter in what ordered the letters in a word are the only important thing is that the first and last letters be at the right place the rest can be a total messes and you can still read it outwith problem this is because the human mind does not read every letter by itself but the word as a whole".

Note that the word "cambridge" was not in the vocabulary. Although the task in this paper is not to recognize mixed up words, it shows the ability of our decoder to perform a reading task that we naturally do. We carried out the experiments on two public handwritten word databases: Rimes (Augustin et al., 2006) (French), and IAM (Marti & Bunke, 2002 ) (English). We simplified the problem by limiting ourselves to words of at least two lowercase characters (a to z). This selection removed approximately 30% of the words. The number of words and bigrams of different orders in the different sets are reported on Table 4 , in Appendix A.1.

We applied deslanting (Buse et al., 1997) , contrast enhancement, and padded the images with 10px of white pixels to account for empty context on the left and right of words. From the preprocessed images, we extracted sequences of feature vectors with a sliding window of width 3px. The features are geometrical and statistical features described in (Bianne et al., 2011) , which give state-of-the-art results in handwritten text line recognition (Bluche et al., 2014) .

We downloaded word frequency lists for French and English2 . These lists were built from film subtitles3 written by many contributors, and they contain many misspellings. We removed the misspelled words using GNU Aspell (Atkinson) .

We selected 50,000 words for each language. They are the most frequent words (length ≥ 2) and made only of lowercase characters between a and z, making sure to also include all the words of the database. For example, the 50,000 most frequent French words fulfilling these condition miss about 200 words of the Rimes database, so we selected the most frequent 49,800 and added the missing 200. Note that most of the words that were removed from the dataset are not shorter words, but words with special or upper case characters. The distribution of lengths of filtered-out words is shown on Figure 5 in Appendix A.1.

4.2. RECOGNITION OF OPEN-BIGRAMS WITH RECURRENT NEURAL NETWORKS (RNNS):

To predict bigrams, we chose Bidirectional Long Short-Term Memory RNNs (BLSTM-RNNs) for their ability to consider the whole sequence of input vectors to make predictions. We trained one RNN for each order-d bigram, with the Connectionist Temporal Classification (CTC (Graves et al., 2006) ) criterion. The CTC framework defines a sequence labeling problem, with an output sequence of labels, of smaller length than the input sequence of observations.

We built the target sequences for training as sequences of bigrams, ordered according to the first letter of the bigram. For example, for d = 2, the target for example is ea-xm-ap-ml-pe. The CTC training criterion optimizes the Negative Log-Likelihood (NLL) of the correct label sequence.

We set the learning rate to 0.001, and stopped the training when the NLL on the validation set did not decrease for 20 epochs. We kept the network yielding the best NLL on the validation set. We trained one RNN for each order d = 0 to 3, including the special bigrams for word extremities or not. We will refer to each of these RNNs with rnn d for order d (rnn d when extremities are included). The architecture of the networks is described in Appendix A.3. These RNNs are trained to predict sequences of fixed order bigrams. Here, we are interested in a word representation as a bag of bigrams, which does not carry any information about the sequence in which the bigrams appear, the number of times each bigram appears, or the order of each individual bigram. That is, we are interested in a decoder which considers an unordered set of bigrams predictions across bigram orders.

We forget the temporal aspect of bigram predictions by taking the maximum value of a given bigram prediction by the RNN (where rnn d (x, t) if the output of the RNN for order d, input image

EQUATION

and we forget the bigram order by taking the maximum output across different values of d:

EQUATION

)

It would have been more satisfying for this experiment to train an optical model to predict a set of bigrams for all orders. However, this work is focused on the decoder. Moreover, even the simpler task of predicting a sequence of bigrams of fixed order is challenging (the sequence error rates of these networks are detailed in Appendix B.2). On Figure 4 , we show the hypothetical context needed to make two consecutive predictions, for bigram order d = 0..3. RNNs are popular for handwriting recognition, and can consider a context size of variable length -but still local -to predict characters (d = 0).

For d = 1, the required context is still local (and would span two consecutive characters), but overlap, because each character is involved in two bigrams. For d > 1, the context is even split into two areas (covering the involved characters) that might be far apart depending on d. Contexts for different predictions are entangled: the whole area between two characters forming a bigram is not relevant for this bigram (and might be of varying size), but will be important to predict other bigrams. It means that the RNN will have to remember a character observation for some time, until it sees the second character of the bigram, while ignoring the area in between for this bigram prediction, but remembering it since it will be useful in order to predict other bigrams. The number of classes for bigrams is also 26 times larger than the number of characters, making the classification problem harder, and the number of examples per class in training smaller.

5. RESULTS:

In this paper, we focused on a subset of Rimes and IAM word databases, which makes the comparison with published results difficult. Instead, we compared the bigram decoder approach to decoding with standard models, consisting of a beam search with Viterbi algorithm in the lexicon. However, these standard models yield state-of-the art results on the reference task for the two considered database (Bluche et al., 2014) .

5.1. BASELINE SYSTEMS BASED ON HMMS AND VITERBI DECODING:

We built several models and used the same vocabulary as for the bigram decoder, and no language model (all words have the same prior probability). These baseline systems are based on HMMs, with emission models made either of Gaussian mixtures (GMM/HMM), Multi-Layer Perceptrons (MLP/HMM) or Recurrent Neural Networks (rnn 0 /HMM). They are almost identical to those presented in a previous work (Bluche et al., 2014) , where a comparison is made with state-of-the-art systems for handwritten text line recognition. More details about these models and their training procedure are presented in Appendix A.2. On Table 1 , we report the percentages of word errors on the validation and test sets of Rimes and IAM. The best word error rates are around 10% (17.5% on the test set of IAM), and constitute the baseline performance to which the bigram approach is to be compared.

5.2. MEASURING THE QUALITY OF BIGRAM PREDICTIONS:

Since we keep a confidence value for all bigrams in the prediction vector, rather than using a binary vector (cf. Eq. 6), we modified the formulation of precision and recall. 

which are the usual ones when p b (x) ∈ {0, 1}. The F-measure is calculated from precision and recall with the usual formula. The results for all RNNs, and for the combination of orders, are reported on Table 2 . We observe that the precision and recall results are correlated to the performance in terms of edit distance or sequence error rates. Namely, they decrease as the bigram order increases, which is not surprising, given that higher order bigrams are more difficult to recognize with these sequence models. We also see that including the special bigrams for word beginnings and endings generally improves the results. This is not surprising either: the RNNs are good at recognizing them.

Despite this performance decrease, the precision remains above 70%, which limits the amount of noise that will be included in the bigram representation for recognition. Combining the recognition across orders, we obtain a precision of around 84% on Rimes and 80% on IAM. The recall tends to be higher than the precision, staying around or above 80% in all configurations. Across orders, the Under review as a conference paper at ICLR 2017 recall is above 88% on Rimes and 86% on IAM. The high recall will limit the amount of missing information in the bigram representation.

Overall, the F-measure for bigram recognition is above 80%, which is a good starting point, given that (i) the vocabulary used in decoding will add constraints and may help recovering from some mistakes in the bigram recognition, and (ii) the redundancy and order encoded in the bigram may limit the impact of misrecognitions.

5.3. WORD RECOGNITION USING BIGRAM PREDICTIONS:

On Table 3 , we report the results of bigram decoding. For each word image in the validation and test sets, we computed the bigram predictions with the RNNs described above. We combined the different orders as explained previously, and either added the special bigrams for word boundaries and/or the single character predictions or not. We computed the cosine similarity to the bigram decomposition of all words in the vocabularies in the same representation space (i.e. same orders, and same choices for the inclusion of special bigrams and single characters) by computing the product of the vocabulary matrix V by the recognition vector. We counted the number of times the correct word was not the most similar one. We see that adding the special bigrams for word boundaries improves the results, especially when single characters are not included in the representation. A possible explanation, besides the fact that they tend to be recognized more easily, could be that they provide a very useful information to disambiguate words having a similar bigram representation (e.g. them and theme). Adding single characters also improves the performance of the decoder, especially when the boundary bigrams are not included in the representation. The gain obtained with the single characters is about the samesometimes a little better -as the gain with boundaries. It might be due to the much better recognition of the RNN for single characters (precision and recall over 90%), as well as the added redundancy and complementary information provided. The results of decoding with different combinations of orders are presented in the appendices in Table 7 . They confirm those observations. The best performance is achieved with both single characters and word boundaries, although the gain compared to adding only one of them is slight. The error rates are competitive or better than the best error rates obtained by classical character sequence modeling and Viterbi decoding.

6. CONCLUSION:

State-of-the-art systems, as well as most of the systems for handwritten word recognition found in the literature, either try to model words as a whole, or as a sequence of characters. The latter, which currently gives the best results, is widely adopted by the community, and benefits from a lot of attention. In this paper, we have proposed a simple alternative model, inspired by the recent findings in cognitive neurosciences research on reading.

We focused on the representation of words in the open-bigram space and built an handwritten word recognition system operating in that space. We were interested in observing how a simple decoding scheme, based on a mere cosine similarity measure in the bigram space, compared to traditional methods. The main apparent difficulty arises from the fact that the global ordering of characters and the distance between bigram constituents are lost in this representation.

The qualitative results presented in the first section showed that the envisioned approach was viable. With the letter reordering example, we have seen that the correct orthographic form of words can be retrieved with a limited and local knowledge of character orders. Moreover, we validated that words that are close in orthographic form are also close in the bigram space. Thus, we demonstrated that the open-bigram representation shows interesting and competitive metric properties for the word recognition. Current work consists in learning most discriminant open-bigram at different order, possibly higher than three according to the length of the word and its similarity to others.

A DATA AND MODELS:

A.1 DATA We built several models and used the same vocabulary as for the bigram decoder, and no language model (all words have the same prior probability). The first one is a Hidden Markov Model, with 5 (Rimes) or 6 (IAM) states per characters, and an emission model based on Gaussiam Mixture Models. This system is trained with the Maximum Likelihood criterion, following the usual Expectation-Maximization procedure. At each iteration, the number of Gaussians is increased, until no improvement is observed on the validation set. The forced alignments with the GMM/HMM system are used to build a labeled dataset for training a Multi-Layer Perceptron (MLP) with 4 hidden layers of 1,024 sigmoid units. We optimized the cross-entropy criterion to train the network to predict the HMM states from the concatenation of 11 input frames, with a learning rate of 0.008. The learning rate was halved when the relative improvement was smaller than 1% from one epoch to the next.

The MLP is integrated in the hybrid NN/HMM scheme by dividing the predicted state posteriors p(s|x) by the state priors p(s), estimated from the forced alignments. It is then further trained with a sequence-discriminative criterion: state-level Minimum Bayes Risk (Kingsbury, 2009) (sMBR) for 5 epochs. Finally, we also performed the sequential decoding with vocabulary constraints using rnn 0 . The input features for all systems are the same, described in 4.1. The results of similar systems for handwritten text line recognition can for example be found in a previous work (Bluche et al., 2014) , where a comparison is made with state-of-the-art systems.

A.3 RECURRENT NEURAL NETWORK The networks are trained with stochastic gradient descent to minimize the Connectionist Temporal Classification (CTC (Graves et al., 2006) ) criterion, i.e. the Negative Log-Likelihood (NLL) of the correct label sequence. We set the learning rate to 0.001, and stopped the training when the NLL on the validation set did not decrease for 20 epochs, and kept the network yielding the best NLL on the validation set. On Figure 7 , we randomly selected pairs of words, and pairs of words with high cosine similarity in the French and English dictionary, and plotted their cosine similarity in the bigram space against the edit distance between the two words, normalized by the length of the longest word. We note that words with high cosine similarity also have short edit distance, supporting the idea that the bigram representation encode some global letter order, and therefore might favor word recognition. 

B.2 EVALUATION OF PERFORMANCE OF RNN ALONE:

We trained RNNs to predict sequences of bigrams of a given order. Their performance to accomplish this task can be measured, on the validation set, with the edit distance (edit.dist) between the recognized sequence and the true sequence, and with the percentage of sequences with at least one error: the Sequence Error Rate (SER). The results are reported on Table 6 . We observe that the performance decreases as d increase. Yet the errors remain in a reasonable range compared to the simple case d = 0, and one should keep in mind that a small degradation should be expected from the larger number of classes and smaller number of training examples per class.

We have shown that RNNs can perform the apparently difficult task of recognizing sequences of open-bigrams. Thus, we may use the RNN bigram predictions in our decoder, rather than building bigram predictions from letter ones, which would have been less satisfying considering the supposed interest of the redundancy of the bigram representation that allows error corrections.

B.3 EVALUATION OF PERFORMANCE OF THE DECODER:

In Table 7 , we report the results of the cosine decoder applied to the outputs of OB RNNs with different bigram orders. approaches. We observe the same behaviour: short words tend to be more difficult to recognize.

:

http://www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/

http://invokeit.wordpress.com/frequency-word-lists/.

http://opensubtitles.org

Figures and Tables:

Figure 2: Word representation as an explicit sequence of letters (a), and as a set of bigrams (b). Grey edges show the potential impact of misrecognitions.

Figure 3: Visualization of the bigram representation of the English vocabulary, for d = 1..3 (Touzet et al., 2014) (left), vs after t-SNE (Van der Maaten & Hinton, 2008) (right). Our complete bigramic map of English: https://youtu.be/OR2vjj8MNeM?t=197.

Figure 4: Hypothetical context needed in the input image to make two consecutive (yellow and blue) bigram predictions, for d = 0 (left, to predict c, then t) to 3 (right, to predict ai, then cc). As d increases, the contexts become more complex to model: they involve long range dependencies and are highly intertwined.

A bigram b ∈ B(w) is correctly retrieved with confidence p b (x), and missed with confidence (1 -p b (x)). Similarly, a bigram not in the representation B(w) of word w is falsely recognized with confidence p b (x), and correctly ignored with confidence (1 -p b (x)). It gives us the following expressions for

Figure 5: Distribution of lengths of filtered-out words

Figure 6: Neural network architecture. A sequence of feature vectors is extracted with a sliding window, and fed to a multi-layer BLSTM network, with subsampling after the first BLSTM layer. The output is a sequence of open-bigram prediction.

Figure 7: Relation between cosine similarity in bigram space and normalized edit distance, for the English (a) and French (b) vocabularies.

Figure 8: Distribution of word errors vs. word length for the proposed (left) and sequential (right) approaches. We observe the same behaviour: short words tend to be more difficult to recognize.

Word Error Rates (%) with baseline systems and Viterbi decoding of character sequences.

Table content: <table><tr><td/><td/><td/><td>Models</td><td/></tr><tr><td/><td>Dataset</td><td colspan="3">GMM/HMM MLP/HMM rnn0/HMM</td></tr><tr><td>Rimes</td><td>Valid.</td><td>37.38</td><td>14.82</td><td>10.79</td></tr><tr><td>Viterbi (Char. seq.)</td><td>Test</td><td>36.24</td><td>14.45</td><td>10.03</td></tr><tr><td>IAM</td><td>Valid.</td><td>27.64</td><td>11.73</td><td>10.21</td></tr><tr><td>Viterbi (Char. seq.)</td><td>Test</td><td>37.96</td><td>19.97</td><td>17.49</td></tr></table>

Precision, Recall and F-measure of OB detection by RNNs with different orders d.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>d</td><td/></tr><tr><td/><td/><td>0</td><td>1</td><td>1'</td><td>2</td><td>2'</td><td>3</td><td>3'</td><td>1,2,3 1',2',3'</td></tr><tr><td/><td>Precision</td><td>95.0</td><td colspan="6">89.9 91.2 79.8 82.8 74.8 82.6</td><td>84.5</td><td>84.0</td></tr><tr><td>Rimes</td><td>Recall</td><td>93.4</td><td colspan="6">87.6 89.3 84.8 85.8 83.4 80.9</td><td>86.7</td><td>88.5</td></tr><tr><td/><td>F-measure</td><td>0.94</td><td colspan="6">0.89 0.90 0.82 0.84 0.79 0.82</td><td>0.89</td><td>0.86</td></tr><tr><td/><td>Precision</td><td>93.5</td><td colspan="6">87.3 89.3 77.7 81.6 62.3 76.2</td><td>80.5</td><td>81.0</td></tr><tr><td>IAM</td><td>Recall</td><td>92.5</td><td colspan="6">86.2 88.5 82.3 84.0 77.5 78.6</td><td>84.3</td><td>86.4</td></tr><tr><td/><td>F-measure</td><td>0.93</td><td colspan="6">0.87 0.89 0.80 0.83 0.69 0.77</td><td>0.82</td><td>0.84</td></tr></table>

Decoding results (% of word errors).

Table content: <table><tr><td>Rimes</td><td>IAM</td></tr></table>

Number of words/bigrams in different sets of Rimes and IAM databases. (in parentheses, the number of distinct tokens).

Table content: <table><tr><td/><td/><td>Train</td><td>Valid.</td><td>Test</td></tr><tr><td>Rimes</td><td>Words</td><td>33,947</td><td>3,772</td><td>5,695</td></tr><tr><td/><td>Bigram d = 0</td><td>161,203 (26)</td><td>17,887 (25)</td><td>26,828 (25)</td></tr><tr><td/><td colspan="4">Bigram d = 1 127,256 (312) 14,115 (240) 21,133 (258)</td></tr><tr><td/><td>Bigram d = 2</td><td colspan="3">93,309 (425) 10,343 (336) 15,438 (352)</td></tr><tr><td/><td>Bigram d = 3</td><td>68,747 (436)</td><td>7,651 (327)</td><td>11,372 (368)</td></tr><tr><td/><td colspan="4">Bigram d = 1..3 289,312 (517) 32,109 (420) 47,943 (452)</td></tr><tr><td>IAM</td><td>Words</td><td>38,584</td><td>5,977</td><td>18,394</td></tr><tr><td/><td>Bigram d = 0</td><td>177,101 (26)</td><td>26,133 (26)</td><td>82,320 (26)</td></tr><tr><td/><td colspan="4">Bigram d = 1 138,517 (428) 20,156 (364) 63,926 (417)</td></tr><tr><td/><td>Bigram d = 2</td><td colspan="3">99,933 (550) 14,179 (477) 45,532 (537)</td></tr><tr><td/><td>Bigram d = 3</td><td>68,961 (543)</td><td>9,361 (457)</td><td>30,537 (522)</td></tr><tr><td/><td colspan="4">Bigram d = 1..3 307,411 (598) 43,696 (551) 139,995 (592)</td></tr></table>

Number of parameters in the RNNs

Table content: <table><tr><td colspan="2">d Num. Parameters</td></tr><tr><td>0</td><td>1,805,827</td></tr><tr><td>1</td><td>2,066,477</td></tr><tr><td>1'</td><td>2,087,329</td></tr><tr><td>2</td><td>2,066,477</td></tr><tr><td>2'</td><td>2,087,329</td></tr><tr><td>3</td><td>2,066,477</td></tr><tr><td>3'</td><td>2,087,329</td></tr><tr><td>1,2,3</td><td>6,199,431</td></tr><tr><td>1',2',3'</td><td>6,261,987</td></tr><tr><td>B RESULTS</td><td/></tr></table>

RNN results

Table content: <table><tr><td/><td colspan="6">d edit.dist(%) SER(%) Precision(%) Recall(%) F-measure</td></tr><tr><td/><td>0</td><td>8.3</td><td>22.7</td><td>95.00</td><td>93.42</td><td>0.942021</td></tr><tr><td>Rimes</td><td>1</td><td>11.4</td><td>20.8</td><td>89.86</td><td>87.61</td><td>0.8872</td></tr><tr><td/><td>1'</td><td>9.6</td><td>21.7</td><td>91.17</td><td>89.25</td><td>0.901998</td></tr><tr><td/><td>2</td><td>13.4</td><td>23.2</td><td>79.78</td><td>84.84</td><td>0.822315</td></tr><tr><td/><td>2'</td><td>11.1</td><td>26.2</td><td>82.84</td><td>85.84</td><td>0.843128</td></tr><tr><td/><td>3</td><td>14.8</td><td>23.7</td><td>74.80</td><td>83.37</td><td>0.788523</td></tr><tr><td/><td>3'</td><td>12.5</td><td>27.4</td><td>82.57</td><td>80.93</td><td>0.817402</td></tr><tr><td/><td>1,2,3</td><td>-</td><td>-</td><td>84.53</td><td>86.68</td><td>0.855922</td></tr><tr><td/><td>1',2',3'</td><td>-</td><td>-</td><td>84.03</td><td>88.48</td><td>0.86197</td></tr><tr><td/><td>0</td><td>8.0</td><td>21.7</td><td>93.51</td><td>92.54</td><td>0.930205</td></tr><tr><td>IAM</td><td>1</td><td>13.1</td><td>23.2</td><td>87.34</td><td>86.20</td><td>0.867681</td></tr><tr><td/><td>1'</td><td>10.4</td><td>23.3</td><td>89.28</td><td>88.48</td><td>0.888813</td></tr><tr><td/><td>2</td><td>16.7</td><td>26.2</td><td>77.71</td><td>82.33</td><td>0.799537</td></tr><tr><td/><td>2'</td><td>12.7</td><td>28.3</td><td>81.57</td><td>83.95</td><td>0.827408</td></tr><tr><td/><td>3</td><td>20.8</td><td>31.2</td><td>62.29</td><td>77.54</td><td>0.69081</td></tr><tr><td/><td>3'</td><td>14.6</td><td>31.8</td><td>76.17</td><td>78.56</td><td>0.773436</td></tr><tr><td/><td>1,2,3</td><td>-</td><td>-</td><td>80.53</td><td>84.26</td><td>0.823527</td></tr><tr><td/><td>1',2',3'</td><td>-</td><td>-</td><td>81.04</td><td>86.40</td><td>0.836315</td></tr></table>

Decoding result (word error rate) with different bigram ordersIn Table8, we evaluate the cosine decoder by applying it to the ground-truth OB decomposition with varying order of the validation sets, i.e. the performance assuming the RNN optical models are perfect.

Table content: <table><tr><td colspan="3">d Rimes IAM</td></tr><tr><td>1</td><td>14.5</td><td>16.4</td></tr><tr><td>0,1</td><td>13.0</td><td>14.1</td></tr><tr><td>1'</td><td>12.9</td><td>13.1</td></tr><tr><td>0,1'</td><td>11.9</td><td>12.3</td></tr><tr><td>1,2</td><td>26.4</td><td>13.7</td></tr><tr><td>1',2'</td><td>12.7</td><td>11.8</td></tr><tr><td>0,1,2</td><td>11.7</td><td>12.3</td></tr><tr><td>0,1',2'</td><td>10.1</td><td>11.3</td></tr><tr><td>1,2,3</td><td>25.6</td><td>13.5</td></tr><tr><td>1',2',3'</td><td>12.4</td><td>11.8</td></tr><tr><td>0,1,2,3</td><td>11.0</td><td>12.0</td></tr><tr><td>0,1',2',3'</td><td>9.8</td><td>11.1</td></tr></table>

Decoding result (word error rate) with different bigram orders COMPARISON OF ERRORS OF THE SEQUENTIAL AND OPEN-BIGRAM MODELS

Table content: <table><tr><td colspan="3">d Rimes IAM</td></tr><tr><td>1</td><td>0.64</td><td>1.86</td></tr><tr><td>0,1</td><td>0.90</td><td>1.62</td></tr><tr><td>1'</td><td>0.03</td><td>0.02</td></tr><tr><td>0,1'</td><td>0.03</td><td>0.02</td></tr><tr><td>1,2</td><td>0.21</td><td>0.05</td></tr><tr><td>1',2'</td><td>0.03</td><td>0.00</td></tr><tr><td>0,1,2</td><td>0.13</td><td>0.05</td></tr><tr><td>0,1',2'</td><td>0.29</td><td>0.00</td></tr><tr><td>1,2,3</td><td>0.21</td><td>0.08</td></tr><tr><td>1',2',3'</td><td>0.03</td><td>0.00</td></tr><tr><td>0,1,2,3</td><td>0.03</td><td>0.03</td></tr><tr><td>0,1',2',3'</td><td>0.03</td><td>0.00</td></tr><tr><td>B.4 ERROR ANALYSIS</td><td/><td/></tr><tr><td>B.4.1</td><td/><td/></tr></table>

Peer review (iclr2017):
This paper uses an LSTM model to predict what it calls *open bigrams* (bigrams of characters that may or may not have letters inbetween) from handwriting data. These open bigrams are subsequently used to predict the written word in a decoding step. The experiments indicate that the system does slightly better than a baseline model that uses Viterbi decoding. I have some major concerns about this paper: - I find the *cortical inspired* claim troublesome. If anything, it is psychology/cognitive science inspired, in the sense that open bigrams appear to help for word recognition (Touzet et al. 2014). But the implied cortical characteristics, implicitly referred to e.g. by pointing to analogies between deep neural nets for object recognition and in that case the visual cortex, is unfounded. Is there any direct evidence from neuroscience that open-bigrams constitute a wholly separate layer in the cortex for a handwriting recognition task? Dehaene*s work is a proposal, so you*ll need to describe more *findings in cognitive neurosciences [sic] research on reading* (p. 8) to substantiate those claims. I am further worried by the fact that the authors seem to think that *deep neural networks are based on a series of about five pairs of neurons [sic] layers*. Unless I misunderstand something, you are specifically referring to Krizhevsky*s AlexNet here (which you should probably have cited there)? I hope you don*t mean to imply that all deep neural nets need five layers. It is also not true that ten is *quite close to the number of layers of an efficient deep NN* -- what network? what task? etc. - The model is not clearly explained. There is a short paragraph in Appendix A.3. that roughly describes the setup, but this does not include e.g. the objective function, or answer why the network output is only considered each two consecutive time steps, rather than at each time step (or so it seems?). This is probably because the paper argues that it *is focused on the decoder* (p. 6), rather than on the whole problem. I find this problematic, because in that case we*re effectively measuring how easy it is to reconstruct a word from its open bigrams, which has very little to do with handwriting recognition (it could have been evaluated on any text corpus). In fact, as the example on page 4 shows, handwriting is not necessary to illustrate the open bigram hypothesis. Which leads me to wonder why these particular tasks were chosen, if we are only interested in the decoding mechanism? - The comparison is not really fair. The Viterbi decoder only has access to unigrams, as far as I can tell. The only model that does better than that baseline has access to a lot more information, and does not do that much better. Did the Viterbi model have access to the word boundary information (at one point rather confusingly called *extremities*) that pushed the open bigram model over the edge in terms of performance? Why is there no comparison to e.g. rnn_0,1* (unigram+bigram+boundary markers)? The dataset also appears to be biased in favor of the proposed approach (longer words, only ). I am not convinced that this paper really shows that open bigrams help. I very much like the idea of the paper, but I am simply not convinced by its claims. Minor points: - There are quite a few typos. Just a sample: *independant* (Fig.1), *we evaluate an handwritten*, *, hand written words [..], an the results*, *their approach include*, *the letter bigrams of a word w is*, *for the two considered database* - Wouldn*t it be easy to add how many times a bigram occurs, which would improve the decoding process? You can just normalize over the full counts instead of the binary occurrence counts. - The results in Table 5 are the same (but different precision) as the results in Table 2, except that edit distance and SER are added, this is confusing.