Schreuder et al. Listen, you are writing! FIGURE 2 | Visualization of iteration-speciﬁc decision threshold. The conﬁdence measure p is plotted for correct (P +) and incorrect (P −) decisions as a function of the number of iterations j. The black line F is ﬁtted to the maximum values of P −. The decision threshold T is deﬁned as the maximum of F and the median of P +. a conservative threshold is that any false positive is costly, as correcting an error takes up to four selections. For applications where a false selection is less costly, the threshold can be set less conservative by taking a smaller value for R. DATA ANALYSES The BCI performance evaluation was done online, unless stated otherwise. Three evaluation metrics were calculated for each sub- ject and each sentence: accuracy (acc), characters per minute (char/min), and information transfer rate (ITR; Schlögl et al., 2007). We deﬁne accuracy as the number of correct selections divided by the total amount of selections. Accuracy assesses the performance of the BCI, irrespective of the application interface. For assessing the BCI in combination with the spelling appli- cation we used char/min as the preferred metric for writing pro- ﬁciency. Char/min is deﬁned as the number of correctly written letters divided by the time it took to write all letters (including incorrect letters). Here, it gives a particular conservative estimate, as subjects had to correct any mistakes. Thus, all letters are in the end correct at the cost of longer writing time (and lower char/min). This is a more realistic assessment of the system’s usefulness, as it includes the error recovery strategy. In a real user setting, errors will occur and may have to be corrected. For the sake of comparison, the ITR is also reported. We use the calculation method proposed by Schlögl et al. (2007), as there are systematic confusions between directions. ITR combines the accu- racy, number of possible decisions, and the number of selections per minute into a single number. Originating from informa- tion theory, it assesses information transfer over a noisy channel. Although often used in BCI, it is a theoretical measure and its interpretability in terms of practical relevance of a BCI is difﬁcult. DIRECTIONAL PREFERENCE The confusion matrix (CM) provides a lot of information about the discriminability of the different classes. From it, the pairwise F -scores were calculated to assess the confusion between each (1) (2) (3) pair of directions. This should expose any systematic directional preference, if present. Given a confusion matrix CM N × N , where N is the number of classes and each row is normalized, we deﬁne the sensitivity and recall for pairs of classes i, j ∈ {1,. . .,6} as follows CM(i,i) sensitivity(i,j) = CM(i,i) + CM(i,j) ∗ (N − 1) CM(i,i) recall(i,j) = CM(i,i) + CM(j,i) ∗ (N − 1) The matrix F N × N of pairwise F -scores is then deﬁned as sensitivity(i,j) ∗ recall(i,j) F(i,j) = 2 × sensitivity(i,j) + recall(i,j) where 1 ≤ i ≤ C and 1 ≤ j ≤ C. It is thus the harmonic mean of sensitivity (the ability to identify a target) and recall (the ability to reject a non-target). RESULTS Of the 21 subjects, 16 subjects (or 76%) were able to write a full German sentence at ﬁrst try (Session 1). The other ﬁve subjects got at most one word correct, before their experiment was stopped. In Session 2, 14 out of 15 returning subjects wrote both full German sentences. Subject VPfch was able to write the ﬁrst sentence, but took so long that the session had to be aborted after this. Sub- ject VPfax was unable to return for the second session, although successful in the ﬁrst session. In the following, sentence 1 refers to the sentence written in the ﬁrst session, and sentence 2a and 2b to those written in the second session. Individual performance measures for each subject can be found in Table 2. Movie S1 in Supplementary Material shows an example of a user operating the system. BCI PERFORMANCE Brain–computer interface performance did neither correlate with age (r < 0.01, p = 0.75), nor with gender (Kolmogorov–Smirnov test: p = 0.42). Average accuracy per sentence was 77.4, 84.3, and 86.1% for sentence 1, 2a, and 2b, respectively. The increase in performance from Session 1 to Session 2 is due to the drop out of less-performing subjects. This is shown in Figure 3A, which gives the overall accuracy for each subject and each session. Three means were calculated; ﬁrst, the black line indicates the mean when considering all subjects for a session, as given above. However, as those subjects that were unable to write in the ﬁrst session did not participate in the second session, the gray line includes only those subjects (N = 14) that ﬁnished all three sentences, which is a fairer comparison of the sessions (0.88, 0.86, and 0.86 for sentence 1, 2a, and 2b respectively). No signiﬁcant difference existed (p = 0.97) between performance in sentence 2a (m = 0.86, SD = 0.08) and 2b (m = 0.86, SD = 0.11). The difference between these conditions is that for sentence 2b the visually presented support labels were switched off. This shows that the spelling tree can easily be learned and thus need not be presented visually. As the number of iterations varied in Session 2, an ofﬂine analy- sis was performed to objectively assess any learning effect between www.frontiersin.org October 2011 | Volume 5 | Article 112 | 5