Title: Cortical-Inspired Open-Bigram Representation for Handwritten Word Recognition
Authors: Théodore Bluche
Christopher Kermorvant
Claude Touzet
Hervé Glotin
Venue: ICLR 2017

Abstract: Recent research in the cognitive process of reading hypothesized that we do
not read words by sequentially recognizing letters, but rather by identifing
open-bigrams, i.e. couple of letters that are not necessarily next
to each other. 
In this paper, we evaluate an handwritten word recognition method based on original
open-bigrams representation. We trained Long Short-Term Memory Recurrent Neural Networks
(LSTM-RNNs) to predict open-bigrams rather than characters, and we show that
such models are able to learn the long-range, complicated and intertwined dependencies
in the input signal, necessary to the prediction. 
For decoding, we decomposed each word of a large vocabulary into the set of
constituent bigrams, and apply a simple cosine similarity measure between this 
representation and the bagged RNN prediction to retrieve the vocabulary word. 
We compare this method to standard word recognition techniques based on 
sequential character recognition.
Experiments are carried out on two public databases of handwritten words
(Rimes and IAM), an the results with our bigram decoder are comparable 
to more conventional decoding methods based on sequences of letters.

TL;DR: We propose an handwritten word recognition method based on an open-bigram representation of words, inspired from the research in cognitive psychology

Review discussion:
The performance numbers provided for IAM and RIMES in Table 1 do not seem to be optimal. The best results known to me were published in [Poznansky & Wolf, CVPR 2016], with 3.9% WER for RIMES, and 6.44% WER for IAM.
On the other hand, the model complexity chosen here, as described in Appendix A.3, seems to be limited, with only 100 hidden units per layer. Did you also check larger models, and does this influence the comparison between standard and proposed open-bigram approach?
Also, when comparing the standard approach with the open-bigram approach: what is the number of parameters in each case? It seems, that in case of the open-bigram, not only multiple RNNs are used, but also, the actual bigram RNNs (order >=1) do have a much larger output layer.
On the other hand, the standard approach would benfit from using context-dependent targets and might lead to a different result of the comparison. Did you check this?
At the beginning of Sec. 5.1 you state that no language models were used. As far as I understand this applies to both baseline/standard and open-bigram approach, right? What would be the performance of the baseline model when using a language model?
For your comparative experiments you use only 70% of the data by choosing longer words only.
On the other hand, it is well known that the shorter words are more prone to result in misrecognitions. The question is, if this choice is advantageous for one of the tasks, or not. Can you comment on this?
I do see the motivation for introducing open-bigrams in an unordered way due to the corresponding evidence from cognitive research.
However, decision theoretically I wonder, why the order should be given up, if the underlying sequential classification problem clearly is of a monotonous nature. Ignoring the sequential order, words being anagram of each other would only be distinguishable via their probability of occurrence.
Even if including word boundary symbols into the open-bigrams, anagrams indistinguishable from unordered open-bigrams would remain, e.g. in English: "parental" vs. "paternal". Did you check the performance of a combination of your bigram features with an ordered generation/search process?
The number of target for the RNNs modeling order 0 (unigrams effectively) and the RNNs modeling order 1 and larger are very much different.
Therefore the precision and recall numbers in Table 2 do not seem to be readily comparable between order 0 and orders >=1. Could you comment on this?
Thank you for the interesting and relevant comments and feedback.
> The performance numbers provided for IAM and RIMES in Table 1 do not seem to be optimal. The best results known to me were published in [Poznansky & Wolf, CVPR 2016], with 3.9% WER for RIMES, and 6.44% WER for IAM.
As mentioned in the introduction of Sections 4 & 5, we only used a subset of these databases and could not compare with published results directly.
Regarding [Poznansky & Wolf, CVPR 2016] the vocabulary size is not mentioned (from the contents of Rimes and IAM, I guess it is around 10-15k words) whereas our vocabulary contains 50k words. If we are to compare the results although they are not computed on the same datasets, those we report are close or better than the older results reported in Table I of [Poznansky & Wolf, CVPR 2016] (which were without data augmentation, as our system. And I should point out that most of these results are obtained at line -not word- level with out-of-vocabulary words in the test set).
Finally, Poznansky & Wolf state that the current state of the art at the time of writing was Bluche et al. [10]. In Table 1, we report the results of the systems of that paper, trained on our subset, and with the same vocabulary, in order to have a fair comparison.
> On the other hand, the model complexity chosen here, as described in Appendix A.3, seems to be limited, with only 100 hidden units per layer. Did you also check larger models, and does this influence the comparison between standard and proposed open-bigram approach?
For the standard approach we tried larger models for text-line recognition : the were slightly better on that task. For this work we selected the network architecture offering a good compromise between performance and complexity. We can probably get better results with larger models and more data, but we decided to focus the paper on validating the decoding approach.
> Also, when comparing the standard approach with the open-bigram approach: what is the number of parameters in each case? It seems, that in case of the open-bigram, not only multiple RNNs are used, but also, the actual bigram RNNs (order >=1) do have a much larger output layer.
That is correct.
However, we did not focus on the "optical model" but rather on the decoder in this paper. In that perspective, the open-bigram decodeur gets predictions from the RNNs that are not as good as those in the standard approach.
> On the other hand, the standard approach would benfit from using context-dependent targets and might lead to a different result of the comparison. Did you check this?
On Rimes and IAM we do not get much better results with context dependent models. See for example:
* Bluche, T., Ney, H., & Kermorvant, C. (2013). Tandem HMM with convolutional neural network for handwritten word recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 2390-2394). IEEE.
* Bluche, T. (2015). Deep Neural Networks for Large Vocabulary Handwritten Text Recognition (Doctoral dissertation, Université Paris Sud-Paris XI). (Table 4.3)
> At the beginning of Sec. 5.1 you state that no language models were used. As far as I understand this applies to both baseline/standard and open-bigram approach, right?
That is correct
> What would be the performance of the baseline model when using a language model?
Both models would benefit from a language model, although the benefits would depend on the corpus on which they are estimated. In (Bluche et al. 2013) we observed about 15% reduction of the error rate with a language model. On text lines (Bluche, 2015) we get about 25% improvement.
> For your comparative experiments you use only 70% of the data by choosing longer words only.
On the other hand, it is well known that the shorter words are more prone to result in misrecognitions. The question is, if this choice is advantageous for one of the tasks, or not. Can you comment on this?
In both cases, there are more errors for short words.
However, I should point out that we did not choose the 70% longer words. We discarded words of length 1 (<< 30% of data) and words containing capital letters, punctuation/apostrophes, and accents.
> I do see the motivation for introducing open-bigrams in an unordered way due to the corresponding evidence from cognitive research.
However, decision theoretically I wonder, why the order should be given up, if the underlying sequential classification problem clearly is of a monotonous nature. Ignoring the sequential order, words being anagram of each other would only be distinguishable via their probability of occurrence.
Even if including word boundary symbols into the open-bigrams, anagrams indistinguishable from unordered open-bigrams would remain, e.g. in English: "parental" vs. "paternal". Did you check the performance of a combination of your bigram features with an ordered generation/search process?
Our goal was to evaluate a “pure” open bigrams approach.
Therefore, we didn't try to improve the performance of our unordered bigrams model by introducing additional ordered information (apart from the boundary symbols).
It is true that anagrams are then indistinguishable. Getting into the details of biological vision, cognitive psychology told us that the quality of image depends on the distance to the eye focus point (viewing position effect - usually a little on the left of the middle of the word). Long words may require 2 eye saccades, i.e., two different focus points. By taking into account this parameter (letter distance to the focus point), it is then possible to code bigrams with continuous values ([0,1]) instead of “Absent/Present” ({0,1}). Anagrams would then become distinguishable.
Moreover, the RNNs are trained with CTC to output the bigrams of a given order sequentially. In Appendix B.2, the edit distance between predicted and true sequences of bigrams, and the sequence error are reported on Table 5 and correspond to an ordered search.
> The number of target for the RNNs modeling order 0 (unigrams effectively) and the RNNs modeling order 1 and larger are very much different.
Therefore the precision and recall numbers in Table 2 do not seem to be readily comparable between order 0 and orders >=1. Could you comment on this?
Indeed, they are not comparable.
However, from the perspective of the decoder, they indicate the difficulty to some extend (more errors to be corrected by the open-bigram decoder). Taken indivudually, they give an idea of the quality of the RNNs predicting open-bigrams.
This submission investigates the usability of cortical-inspired distant bigram representations for handwritten word recognition.
Instead of generating neural network based posterior features for character (optionally in local context), sets posterior for character bigrams of different length are used to represent words.  The aim here is to investigate the viability of this approach and to compare to the standard approach.
Overall, the submission is well written, although information is missing w.r.t. to the comparison between the proposed approach and the standard approach, see below.
It would be desirable to see the model complexity of all the different models used here, i.e. the number of parameters used.
Language models are not used here. Since the different models utilize different levels of context, language models can be expected to have a different effect on the different approaches.
Therefore I suggest to include the use of language models into the evaluation.
For your comparative experiments you use only 70% of the data by choosing longer words only.
On the other hand, it is well known that the shorter words are more prone to result in misrecognitions. The question remains, if this choice is advantageous for one of the tasks, or not - corresponding quantitative results should be provided to be able to better evaluate the effect of using this constrained corpus. Without clarification of this I would not readily agree that the error rates are competitive or better than the standard approach, as stated at the end of Sec. 5.
I do see the motivation for introducing open-bigrams in an unordered way due to the corresponding evidence from cognitive research.
However, decision theoretically I wonder, why the order should be given up, if the underlying sequential classification problem clearly is of a monotonous nature. It would be interesting to see an experiment, where only the use of the order is varied, to differentiate the effect of the order from the effect of other aspects of the approach.
End of page 1: "whole language method" - please explain what is meant by this.
Page 6: define your notation for rnn_d(x,t).
The number of target for the RNNs modeling order 0 (unigrams effectively) and the RNNs modeling order 1 and larger are very much different.
Therefore the precision and recall numbers in Table 2 do not seem to be readily comparable between order 0 and orders >=1.
At least, the column for order 0 should be visually separated to highlight this.
Minor comments: a spell check is recommended
p. 2: state-of-art -> state-of-the-art
p. 2: predict character sequence -> predict a character sequence
p. 3, top: Their approach include -> Their approach includes
p. 3, top: an handwritten -> a handwritten
p. 3, bottom: consituent -> constituent
p. 4, top: in classical approach -> in the classical approach
p. 4, top: transformed in a vector -> transformed into a vector
p. 5: were build -> were built
References: first authors name written wrongly: Thodore Bluche -> Theodore Bluche
Thank you for the comments and feedback, and for pointing out the typos.
> It would be desirable to see the model complexity of all the different models used here, i.e. the number of parameters used.
We added the number of parameters of the RNNs used for the experiments in Appendix A (Table 5., about 2M for each of the RNNs).
> For your comparative experiments you use only 70% of the data by choosing longer words only.
That is not quite correct. We chose the words written with lowercase letters and single-letter words. It indeed filters out the frequent "a" in English, and punctuation marks, which are of length 1, but also many words with capital letters or accent. The distribution of word lengths of filtered words will be added in the appendices.
> I do see the motivation for introducing open-bigrams in an unordered way due to the corresponding evidence from cognitive research.
However, decision theoretically I wonder, why the order should be given up, if the underlying sequential classification problem clearly is of a monotonous nature. It would be interesting to see an experiment, where only the use of the order is varied, to differentiate the effect of the order from the effect of other aspects of the approach.
The goal of this research paper was to evaluate a decoder based on the findings in cognitive research. We wanted to focus on the ability to recognize a word from a set of unordered bigram predictions.
Thus the order is not expected to be present or available at any point. At the implementation level, we use RNNs with a notion of order, but this is not the focus of the paper. In the ideal case (in the hypothesis that reading is achieved via bigram coding and not explicit ordering), the recognition system should output a set of bigram and not model the order.
However, in Table 6 (prev. Table 5), we report the edit distance (in character/bigram sequences) and sequence error rate (~= word error rates) when only the order is varied.
> End of page 1: "whole language method" - please explain what is meant by this.
The "whole language" method is a teaching method for learning how to read (https://en.wikipedia.org/wiki/Whole_language), often opposed to the phonics method.
To simplify, in the former method, children are taught to focus on the word globally, whereas in the latter, they learn to associate phonemes with graphemes, or spelling.
> Page 6: define your notation for rnn_d(x,t).
Added.
> The number of target for the RNNs modeling order 0 (unigrams effectively) and the RNNs modeling order 1 and larger are very much different.
Therefore the precision and recall numbers in Table 2 do not seem to be readily comparable between order 0 and orders >=1.
At least, the column for order 0 should be visually separated to highlight this.
Thanks for the suggestion, the column 0 is now separated.
This paper uses an LSTM model to predict what it calls "open bigrams" (bigrams of characters that may or may not have letters inbetween) from handwriting data. These open bigrams are subsequently used to predict the written word in a decoding step. The experiments indicate that the system does slightly better than a baseline model that uses Viterbi decoding. I have some major concerns about this paper:
- I find the "cortical inspired" claim troublesome. If anything, it is psychology/cognitive science inspired, in the sense that open bigrams appear to help for word recognition (Touzet et al. 2014).
But the implied cortical characteristics, implicitly referred to e.g. by pointing to analogies between deep neural nets for object recognition and in that case the visual cortex, is unfounded. Is there any direct evidence from neuroscience that open-bigrams constitute a wholly separate layer in the cortex for a handwriting recognition task? Dehaene's work is a proposal, so you'll need to describe more "findings in cognitive neurosciences [sic] research on reading" (p. 8) to substantiate those claims. I am further worried by the fact that the authors seem to think that "deep neural networks are based on a series of about five pairs of neurons [sic] layers".
Unless I misunderstand something, you are specifically referring to Krizhevsky's AlexNet here (which you should probably have cited there)? I hope you don't mean to imply that all deep neural nets need five layers. It is also not true that ten is "quite close to the number of layers of an efficient deep NN" -- what network? what task? etc.
- The model is not clearly explained. There is a short paragraph in Appendix A.3. that roughly describes the setup, but this does not include e.g. the objective function, or answer why the network output is only considered each two consecutive time steps, rather than at each time step (or so it seems?). This is probably because the paper argues that it "is focused on the decoder" (p. 6), rather than on the whole problem. I find this problematic, because in that case we're effectively measuring how easy it is to reconstruct a word from its open bigrams, which has very little to do with handwriting recognition (it could have been evaluated on any text corpus).
In fact, as the example on page 4 shows, handwriting is not necessary to illustrate the open bigram hypothesis. Which leads me to wonder why these particular tasks were chosen, if we are only interested in the decoding mechanism?
- The comparison is not really fair. The Viterbi decoder only has access to unigrams, as far as I can tell. The only model that does better than that baseline has access to a lot more information, and does not do that much better. Did the Viterbi model have access to the word boundary information (at one point rather confusingly called "extremities") that pushed the open bigram model over the edge in terms of performance? Why is there no comparison to e.g. rnn_0,1' (unigram+bigram+boundary markers)? The dataset also appears to be biased in favor of the proposed approach (longer words, only ). I am not convinced that this paper really shows that open bigrams help.
I very much like the idea of the paper, but I am simply not convinced by its claims.
Minor points:
- There are quite a few typos. Just a sample: "independant" (Fig.1), "we evaluate an handwritten", ", hand written words [..], an the results", "their approach include", "the letter bigrams of a word w is", "for the two considered database"
- Wouldn't it be easy to add how many times a bigram occurs, which would improve the decoding process? You can just normalize over the full counts instead of the binary occurrence counts.
- The results in Table 5 are the same (but different precision) as the results in Table 2, except that edit distance and SER are added, this is confusing.
Thank you for the detailed review and comments
> - I find the "cortical inspired" claim troublesome. If anything, it is psychology/cognitive science inspired, in the sense that open bigrams appear to help for word recognition (Touzet et al. 2014).
But the implied cortical characteristics, implicitly referred to e.g. by pointing to analogies between deep neural nets for object recognition and in that case the visual cortex, is unfounded.
Our title is: "Cortical inspired open-bigram representation for handwritten word recognition".  "Cortical inspired" refers to the hypothesis that there exists a map (or layer) in the cortex that is devoted to code for bigrams.
It is true that we know about this hypothesis thanks to cognitive psychology experiments and models, but it seems more logical to refer to the subject's study (bigrams' map) instead of the research domain (cognitive prychology).
As for the recurrent question about the legitimacy of a cortical map (or layer) devoted to code for bigrams, recent results (Glasser 2015) establish that the cortex is made of about 360 maps (180 per hemisphere), and that indeed there is a igram representation (Vinckier 2011). It is enough to get the inspiration that one of this 360 maps codes for bigrams...
- Glasser, M. F., Coalson, T., Robinson, E., Hacker, C., Harwell, J., Yacoub, E., ... & Smith, S. M. (2015). A Multi-modal parcellation of human cerebral cortex. Nature.
- Vinckier, Fabien, Emilie Qiao, Christophe Pallier, Stanislas Dehaene & Laurent Cohen (2011). The impact of letter spacing on reading: a test of the bigram coding hypothesis. Journal of Vision 11: 1-21.
> Is there any direct evidence from neuroscience that open-bigrams constitute a wholly separate layer in the cortex for a handwriting recognition task? Dehaene's work is a proposal, so you'll need to describe more "findings in cognitive neurosciences [sic] research on reading" (p. 8) to substantiate those claims.
Yes, there are direct evidences (Vinckier 2011, Grainger 2014).
- Vinckier, Fabien, Emilie Qiao, Christophe Pallier, Stanislas Dehaene & Laurent Cohen (2011). The impact of letter spacing on reading: a test of the bigram coding hypothesis. Journal of Vision 11: 1-21.
- J Grainger, T Hannagan (2014). What is special about orthographic processing? Written Language & Literacy 17 (2), 225-252.
> I am further worried by the fact that the authors seem to think that "deep neural networks are based on a series of about five pairs of neurons [sic] layers".
We will change the formulation. We don't claim that all deep neural nets need five hidden layers, but rather that the number of layers is larger than one or two. I agree that this sentence is misleading, we will reformulate the analogy.
> Unless I misunderstand something, you are specifically referring to Krizhevsky's AlexNet here (which you should probably have cited there)?
No, not specifically. LeNet-5 is another example, the RNNs for handwriting recognition by Graves (2008) too.
> I hope you don't mean to imply that all deep neural nets need five layers.
We don't.
> It is also not true that ten is "quite close to the number of layers of an efficient deep NN" -- what network? what task? etc.
Well, most deep nets in the litterature (for computer vision, speech recognition, ...) have between 5 and 15 layers.
Of course, it depends on the task.
We will reformulate the analogy between the cortex and deep (artificial) neural networks.
> - The model is not clearly explained. There is a short paragraph in Appendix A.3. that roughly describes the setup,
We will extend A.3 to provide more details
> but this does not include e.g. the objective function,
Sect. 4.2 : "We trained one RNN for each order-d bigram, with the Connectionist Temporal Classification (CTC (Graves et al.,2006)) criterion. [...] The CTC training criterion optimizes the Negative Log-Likelihood~(NLL) of the correct label sequence. We set the learning rate to 0.001, [...]"
> or answer why the network output is only considered each two consecutive time steps, rather than at each time step (or so it seems?).
I guess you refer to "The first LSTM layers have 100 hidden LSTM units. The outputs of two consecutive timesteps of both directions are fed to a feed-forward layer with 100 nodes,". We will reformulate : the "outputs" there are the outputs of the first BLSTM ; the "two consecutive timesteps" allow a subsampling, but both timesteps are used as inputs of the next layer. We will clarify this in a more detailed version of A.3.
> This is probably because the paper argues that it "is focused on the decoder" (p. 6), rather than on the whole problem.
We present a system for the whole problem of recognizing words from a visual input. The novelty only lies in the decoder so we decided to focus the paper on the decoder. We will add more details on the neural networks in appendix A.3.
but this part won't contain any scientific contribution.
> I find this problematic, because in that case we're effectively measuring how easy it is to reconstruct a word from its open bigrams
The motivation of this approach lies in theories of reading mechanisms in the brain (i.e. reconstruct a word from a visual input).
Open bigrams are a way of representing the words (alternative to character sequences).
We will add the results of the decoder only, i.e. assume we know already the open-bigrams to measure how easy it is to reconstruct a word from its open bigrams
> which has very little to do with handwriting recognition
Offline handwriting recognition consists in recognizing words from a visual input.
> (it could have been evaluated on any text corpus).
In fact, as the example on page 4 shows, handwriting is not necessary to illustrate the open bigram hypothesis. Which leads me to wonder why these particular tasks were chosen, if we are only interested in the decoding mechanism?
They were chosen because they are "reading" tasks.
Indeed, a text corpus could be used to evaluate the decoder, but in that case the open-bigrams will be known perfectly, and I fear that the results would be too high to have an interest, just like evaluating a prefix tree decoder on a text corpus would have a limited interest.
> - The comparison is not really fair. The Viterbi decoder only has access to unigrams, as far as I can tell. The only model that does better than that baseline has access to a lot more information, and does not do that much better.
The Viterbi decoder also has access to all character positions and ordering. The OB decoder only knows which bigrams are present, but not the order, count, or position.
> Did the Viterbi model have access to the word boundary information (at one point rather confusingly called "extremities") that pushed the open bigram model over the edge in terms of performance?
Of course. The Viterbi decoder sees a sequence of prediction, so the first character prediction is also the first boundary.
> Why is there no comparison to e.g. rnn_0,1' (unigram+bigram+boundary markers)?
We added the comparisons with different orders (Table 7).
> The dataset also appears to be biased in favor of the proposed approach (longer words, only ).
That is not true. See added figures 5 and 7.
> I am not convinced that this paper really shows that open bigrams help. I very much like the idea of the paper, but I am simply not convinced by its claims.
Note that we don't claim that the OB decoder is a better choice than the sequential decoder.
Even though we sometimes got better empirical results, we conclude by saying that the approach is viable, and that it is possible to recognize words with a decoder operating on the OB representation.
This paper explores the use of Open Bigrams as a target representation of words, for application to handwriting image recognition.
Pros:
- The use of OBs is novel and interesting.
- Clearly written and explained.
Cons:
- No comparison to previous state of the art, only with author-generated results.
- More ablation studies needed -- i.e. fill in Table3 with rnn0,1 rnn0,1,2 rnn0,1' etc etc. It is not clear where the performance is coming from, as it seems that it is single character modelling (0) and word endings (') that are actually beneficial.
- While the use of Open bigrams is novel, there are works which use bag of bigrams and ngrams as models which are not really compared to or explored. E.g. https://arxiv.org/abs/1406.2227 [1] and https://arxiv.org/abs/1412.5903 [2]. Both use bag of ngrams models and achieve state of the art results, so it would be interesting to see whether open bigrams in the same experimental setup as [1] would yield better results.
- Why not use a graph-based decoder like in Fig 2 b?
Overall an interesting paper but the lack of comparisons and benchmarks makes it difficult to assess the reality of the contributions.
Thanks for the feedback and relevant comments.
> - No comparison to previous state of the art, only with author-generated results.
No fair comparison is possible since we limited ourselves to ([a-z]+) words.
Moreover, we don't claim to improve handwriting recognition with this method, but rather to learn a representation of words inspired from neuroscience, and to build a decoder operative on that representation.
> - More ablation studies needed -- i.e. fill in Table3 with rnn0,1 rnn0,1,2 rnn0,1' etc etc. It is not clear where the performance is coming from, as it seems that it is single character modelling (0) and word endings (') that are actually beneficial.
We will add those experiments and results.
> - While the use of Open bigrams is novel, there are works which use bag of bigrams and ngrams as models which are not really compared to or explored. E.g. https://arxiv.org/abs/1406.2227 [1] and https://arxiv.org/abs/1412.5903 [2]. Both use bag of ngrams models and achieve state of the art results, so it would be interesting to see whether open bigrams in the same experimental setup as [1] would yield better results.
I agree, it would be interesting, although the goal of the COGNILEGO project was to "[develop] novel systems for handwriting recognition based on the latest advances in cognitive perception research." (http://cognilego.univ-tln.fr/doku.php).
It would be also interesting to see how [1] and [2] perform in our setup.
> - Why not use a graph-based decoder like in Fig 2 b?
The assumption in the approach is that we only have a set of bigrams. I don't see a simple and efficient way (compared to the cosine similarity) to build a decoder from the hypothetical graph of Fig. 2b.

Meta-review:
role: PC
data: comment: There is consistent agreement towards the originality of this work and that the topic here is "interesting". Additionally there is consensus that the work is "clearly written", and (excepting questions of the word "cortical") all would be primed to accept this style of work. 
 
 However there is a shared concern about the quality and potential impact of the work, in particularly in terms of the validity of empirical evaluations. Reviewers are generally not inclined to believe that the current empirical evidence validates the conclusions of the word. Suggestions are to: make greater use of a language model, compare to external baselines, or remove the handwriting aspects.

Decision: Reject