<figure><img src="image_1.png" /></figure>

**HAL**
open science

# Unitex Getting Started

Denis Maurel, Cvetana Krstev

► To cite this version:

Denis Maurel, Cvetana Krstev. Unitex Getting Started. Independently published, 2024, 9798338757444. hal-04769290

HAL Id: hal-04769290

https://hal.science/hal-04769290v1

Submitted on 6 Nov 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

Language Resources and Technologies Society JeRTeh and University of Belgrade
<figure><img src="image_4.png" /></figure>
# Summary

<table><tbody><tr><td>Chapter 1: introduction</td><td>1</td></tr><tr><td>1 A short history</td><td>1</td></tr><tr><td>2 A quick overview of the book</td><td>1</td></tr><tr><td>3 Organization of correct files</td><td>2</td></tr><tr><td>4 A workbook?</td><td>3</td></tr><tr><td>5 Acknowledgement</td><td>6</td></tr><tr><td>Chapter 2: first steps</td><td>7</td></tr><tr><td>1 Discovering Unitex</td><td>7</td></tr><tr><td>2 A detailed example</td><td>16</td></tr><tr><td>3 The .snt file</td><td>35</td></tr><tr><td>Chapter 3: corpus annotation</td><td>45</td></tr><tr><td>1 First example: compound verbs using the verb will</td><td>45</td></tr><tr><td>2 Second example: numbers written with words</td><td>59</td></tr><tr><td>3 Third example: annotating Roman numerals</td><td>63</td></tr><tr><td>Chapter 4: CasSys</td><td>71</td></tr><tr><td>1 First example: named entity recognition in a plain text</td><td>71</td></tr><tr><td>2 Second example: named entity recognition in an XML text</td><td>84</td></tr><tr><td>3 Third example: measure recognition</td><td>88</td></tr><tr><td>4 Additional possibilities (only for confident users)</td><td>94</td></tr><tr><td>Chapter 5: dictionary creation</td><td>107</td></tr><tr><td>1 Introduction</td><td>107</td></tr><tr><td>2 Inflection of monolexical words</td><td>107</td></tr><tr><td>3 Inflection of multi-word units</td><td>118</td></tr><tr><td>4 Some additional remarks</td><td>123</td></tr><tr><td>5 Dictionary graphs</td><td>128</td></tr><tr><td>6 Additional possibilities (only for confident users)</td><td>132</td></tr><tr><td>Chapter 6: other tools</td><td>137</td></tr><tr><td>1 Lexicon-grammar</td><td>137</td></tr><tr><td>2 XAlign</td><td>141</td></tr><tr><td>Chapter 7: scripts</td><td>149</td></tr><tr><td>1 Introduction</td><td>149</td></tr><tr><td>2 Creation of the linguistic package</td><td>149</td></tr><tr><td>3 The command line</td><td>157</td></tr><tr><td>4 An inventory of tag occurrences</td><td>158</td></tr><tr><td>5 Two small remarks on the optimization of a cascade</td><td>164</td></tr></tbody></table>
<figure><img src="image_5.png" /></figure>
# Chapter 1: introduction

## 1 A short history

Before introducing the Unitex software, we must go back in time and talk about the *LADL*, *Laboratoire d'automatique documentaire et linguistique*, established in 1968 by the late Professor Maurice Gross.<fnref n="1" />

Maurice Gross introduced the notion of *lexicon-grammar*: he asserted that grammar could not be formalized without its dependence on the lexicon.<fnref n="2" /> In collaboration with Professor Dominique Perrin,<fnref n="3" /> he proposed the use of the theory of finite state automata (FSA) for some aspects of linguistic description.<fnref n="4" /> At the same time, he had entrusted one of his doctoral students, Max Silberztein, with the task of creating the computer tool allowing such use of FSA. As part of his thesis, this student designed software called Intex.<fnref n="5" />

Another doctoral student of Maurice Gross,<fnref n="6" /> Sébastien Paumier, wanted to test in his thesis<fnref n="7" /> a different algorithm<fnref n="8" /> from the one implemented in Intex. As was customary in the nineties, the license to use Intex was not free and its code was not open. Therefore Sébastien Paumier had the idea of creating the free and open software, Unitex, based on this new algorithm, in addition to the features already developed by Max Silberztein.

Since then, Unitex has been enhanced by additions and improvements made by many different developers and users. The main contributors are listed in the Unitex user manual.<fnref n="9" />

## 2 A quick overview of the book

The aim of this book is to help its readers to get started with Unitex. In this, it differs from the Unitex user manual whose objective is to list the different possibilities of use that Unitex

<footnote n="1">¹ https://en.wikipedia.org/wiki/Maurice_Gross</footnote>

<footnote n="2">² Gross M. (1975), *Méthodes en syntaxe*, Hermann, Paris.</footnote>

<footnote n="3">³ https://en.wikipedia.org/wiki/Dominique_Perrin</footnote>

<footnote n="4">⁴ Gross M., Perrin D. (1989), *Electronic Dictionaries and Automata in Computational Linguistics*, LNCS 377.</footnote>

<footnote n="5">⁵ Silberztein M. (1989), *Dictionnaires électroniques et reconnaissance lexicales automatique*, Thèse de doctorat (Université Paris VII).</footnote>

<footnote n="6">⁶ Unfortunately, Maurice Gross passed away before the defense of the Sébastien Paumier's thesis.</footnote>

<footnote n="7">⁷ Paumier S. (2003), *De la Reconnaissance de Formes Linguistiques à l'Analyse Syntaxique*, Thèse de doctorat (Université de Marne-la-Vallée).</footnote>

<footnote n="8">⁸ Paumier S. (2003), A time-efficient token representation for parsers, *FSMNLP 2003*, Budapest.</footnote>

<footnote n="9">⁹ The manual is automatically integrated when downloading Unitex (see Chapter 2, Section 1.1.2, page 7). It is also available at the address: https://unitexgramlab.org/releases/3.3/man/Unitex-GramLab-3.3-usermanual-en.pdf</footnote>
offers. The objective of this work is an educational approach to Unitex. It covers most of its possibilities, although not all of them. Namely:

**Chapter 2** (*first steps*), page 7:
Installing and starting to work with Unitex; use of dictionaries; creation of simple graphs.

**Chapter 3** (*corpus annotation*), page 45:
Creation of complex graphs with output; use of weights; morphological mode.

**Chapter 4** (*CasSys*), page 71:
Creation of cascades of graphs; generalization graphs, negative right context, use of variables; testing variables.

**Chapter 5** (*dictionary creation*), page 107:
Creation of dictionaries; inflectional graphs; dictionary graphs.

**Chapter 6** (*other tools*), page 137:
Creation of graphs based on a lexicon-grammar; aligning texts.

**Chapter 7** (*scripts*), page 149:
Creation of scripts.

This book proposes many exercises and micro-projects that can be implemented step by step by following the text.<fnref n="10" /> The different chapters are relatively independent. Screenshots are taken from the Unitex Windows interface. Some minor differences are possible with Mac or Linux interfaces.

## 3 Organization of correct files

The book comes with a zipped file (*MyUnitex.zip*) including all correct files (graphs, cascades, dictionaries, scripts and so on) answering the exercises. If the reader does not wish to construct her/his own graphs or wishes to verify them, she/he may consult them. The early chapters also include videos detailing the step-by-step construction of the graphs.

<figure><img src="image_6.png" /></figure>

<footnote n="10">¹⁰ Work with Elag Grammars and text automata (or FST) is not covered.</footnote>
Since these files often have the same name in one section and another, they are stored in folders named by the number of the section. Also, before consulting them, the reader must copy an entire section folder and paste it into the folder hierarchy just above.

For example, the first graph built is in the folder named:

MyUnitex\English\Graphs\UnitexGettingStarted\Will\3.1.1

with an illustrative video. The reader must copy this graph and paste it in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Will

before opening it with Unitex.

# 4 A workbook?

For a user already familiar with Unitex, this book can be used differently, as a book of correct examples.

## 4.1 Graph exercises

Before starting graph exercises

Open the text:MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt

### Exercise 1

Create a graph that recognizes all forms of the verb *meet*, including verbs with auxiliaries and modal auxiliaries, also including negative and interrogative forms.

The correct solution is in Chapter 2, Section 2.3.6, page 28.

### Exercise 2

Create a graph that recognizes and annotates compound verbs using the verbs *will* and *shall*.
Use Verb, Adverb and Pronoun tags. Verb tags can have an attribute *type* for annotation.<fnref n="11" />

The correct solution is in Chapter 3, Section 1.9.3, page 58.

### Exercise 3

Create a graph that recognizes and annotates numbers written with words, from 2 to 999 999.
Use non-nested *Number* tags.

The correct solution is in Chapter 3, Section 2.4.2, page 62.

### Exercise 4

Create a graph that recognizes and annotates Roman numerals from 1 to 3 999. Use non-nested *RomanNumeral* tags.

The correct solution is in Chapter 3, Section 3.5.2, page 69.

## 4.2 Cascade exercises

### Exercise 5

Before starting exercise 5

Open the text:MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt

<footnote n="11">¹¹ Affirmative, negative, interrogative and interro-negative forms.</footnote>
Create a cascade of graphs that recognizes and annotates measures with six different types: *currencies* (pence, sixpence, <fnref n="12" /> shilling, pound), *duration* (minute, hour, day, week, month, year), *lengths* (mile, foot, yard), *speed* (mile an hour), *temperature* (degree) and *weight* (ton). Use *Number* and *Measure* tags.

Modify the synthesis cascade to have the *Number* tags only inside the *Measure* tags.

The correct solution is in *Chapter 4, Section 3*, page 88.

### Exercise 6

Before starting exercise 6

Open the text: `MyUnitex\English\Corpus\UnitexGettingStarted\jubileePlaintext\jubileePlaintext.txt`

Create a cascade of graphs that recognizes and annotates absolute dates (the year is specified), relative dates (the year is not specified), names of towns or cities and names of streets or avenues. Use *AbsoluteDate*, *RelativeDate*, *Town* and *Address* tags.

Modify the synthesis cascade to have non-nested *Address* tags.

The correct solution is in *Chapter 4, Section 1*, page 71.

### Exercise 7

Before starting exercise 7

Open the text: `MyUnitex\English\Corpus\UnitexGettingStarted\jubileePlaintext\jubileeXmlText.xml`

Modify the previous cascades so that the annotation is only in the `<body></body>` part.

The correct solution is in *Chapter 4, Section 2*, page 84.

## 4.3 Dictionary exercises

Before starting Dictionary exercises

Open the text: `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt`

Create a dictionary that contains all forms of the following nouns and verbs. Use it to parse the text.

### Exercise 8

The beginning of the list:

* *advantage, agent, firm:* nouns with suffix *s*.

* *branch, bus, box:* nouns with suffix *es*.

* *accept, connect, work:* verbs with suffixes *s*, *ed* and *ing*.

The correct solution is in *Chapter 5, Section 2.1*, page 107.

Continuation of the list:

* *company, complexity, family:* nouns with suffix *ies*.

* *leaf, thief, calf:* nouns with suffix *ves*.

* *wife, knife, life:* nouns with suffix *ves*.

* *mouse, louse:* nouns with suffix *ice*.

* *narcissus, terminus, tumulus:* nouns with suffix *i*.

<footnote n="12">¹² Sixpence is the name of an old coin.</footnote>
* *phenomenon, criterion, datum*: nouns with suffix *a*.

* *love, live, give*: verbs with suffixes *s, d* and *ing*.

* *accompany, try, worry*: verbs with suffixes *ies, ied* and *ing*.

The correct solution is in *Chapter 5, Section 2.2*, page 113.

Continuation of the list:

* *goose, tooth*: nouns with suffix *eese or eeth*.

* *foot*: noun with suffix *eet*.

* *man, woman, hypothesis*: nouns in which the penultimate letter changes to *e*.

* *jog, plan, stop, prefer*: verbs with double consonant.

* *slit, split*: verbs with suffixes *s* and *ting*.

* *quiz, gas*: nouns with suffix *zes* or *sses*.

* *shoot, speed, meet*: verbs for which the double letter (*oo, ee*) disappears in the simple past and the past participle.

* *kneel, sleep, creep*: verbs with suffixes *s, elt or ept* and *ing*.

The correct solution is in *Chapter 5, Section 2.3*, page 116.

## Exercise 9

End of the list:

* *air of mystery, sort of thing, tree of knowledge*: MWUs in which only the first component is inflected.

* *crow's nest, death's-head, hair's breadth*: MWUs in which only the last component is inflected.

* *boa constrictor, prince regent, secretary general*: MWUs in which only the first component is inflected.

* *altar rail, Anglo-Norman, bad boy*: MWUs in which only the last component is inflected.

* *after-life, eye-cup, sky-blue*: MWUs in which only the last component is inflected; word tokens can be written together.

* *three quarter, blood red, self glorification*: MWUs in which only the last component is inflected; hyphen can be added.

* *carry out, take off, wash out*: MWUs in which only the last component is inflected; hyphen can be added, space can be omitted.

* *attorney general, notary public*: MWUs in which either the first or the second component (but not both!) are inflected.

The correct solution is in *Chapter 5, Section 3*, page 118.

## Exercise 10

Create a dictionary graph that recognizes all Roman numbers from 1 to 3 999. Use it to create a *Word List*.

The correct solution is in *Chapter 5, Section 5.1*, page 128.
## 4.4 Lexicon grammar exercise

### Exercise 11

Before starting Lexicon exercise 10

Open the text: `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt`

Open the `particleVerbTable.xlsx` Lexicon-grammar in the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\LexiconGrammar\ParticleVerbs`

Create a graph that recognizes all positive forms from the lexicon-grammar `particleVerbTable.xlsx`.

The correct solution is in Chapter 6, Section 1, page 137.

## 4.5 XAlign exercise

### Exercise 12

Align the text:

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng\DombeyAndSonEng.txt`

with its translation:

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonFra\DombeyAndSonFra.txt`

The correct solution is in Chapter 6, Section 2, page 141.

## 4.6 Script exercises

### Exercise 13

Create a script to perform the cascade from Exercise 7, page 4.

The correct solution is in Chapter 7, Section 2, page 149.

### Exercise 14

Complete the previous cascade to get an inventory of tag occurrences with the name of the file and the information inside `title` tags.

The correct solution is in Chapter 7, Section 4, page 158.

# 5 Acknowledgement

The authors sincerely thank all those who helped and advised them while preparing this work: Laurence Danlos, Bérengère David, Anne Dister, Nathalie Friburger, Cristian Martínez, Alexis Neme, Benoît Sagot and Duško Vitas. Authors are especially grateful to Éric Laporte and Anne-Lyse Minard who carefully read the entire manuscript.
# 1 Discovering Unitex

## 1.1 Unitex installation

The Unitex installation file can be found at the URL:

https://unitexgramlab.org

Unitex is a cross-platform software that can be installed on all widely used operating systems (Windows, Mac, Unix and so on). In the following text, we will describe the installation under Windows. Some details shown here may be slightly different on a non-Windows system. For other systems, refer to Section 1.3 of the Unitex user manual downloadable at the URL:<fnref n="13" />

https://unitexgramlab.org/releases/3.3/man/Unitex-GramLab-3.3-usermanual-en.pdf

### 1.1.1 Creating a private working folder

Before launching Unitex for the first time, we must create a private working folder in which we will work. For example, let us go to the *Documents* folder and create a new folder, called for example *MyUnitex* (No spaces and case sensitive):<fnref n="14" />

C:\Documents\MyUnitex

### 1.1.2 Installation process

We start by launching the installation file:

<figure><img src="image_7.png" /></figure>

<footnote n="13">¹³ There is also a French version of the Unitex user manual, downloadable at the URL: https://unitexgramlab.org/releases/3.3/man/Unitex-GramLab-3.3-usermanual-fr.pdf</footnote>

<footnote n="14">¹⁴ Unitex is a cross-platform software, so filenames should not contain spaces and diacritics, which only Windows accepts. Also, calls to these filenames may be case-sensitive.</footnote>
We will click the Next button, and then twice the I Agree button.<fnref n="15" /> In the Choose Components window, we will get this screen and then we will follow the steps:

<figure><img src="image_8.png" /></figure>

1. Click on the check box next to the Visual Integrated Environments line to uncheck it, then on the + sign placed on the left of this line to show possibilities and select the Unitex Java IDE line. Click on the — sign to the left of the Visual Integrated Environments line to hide the possibilities.

2. Click on the check box next to the *Languages Resources* line to uncheck it, then on the + sign placed to the left of this line to show possibilities and select *English* and *French*.<fnref n="16" /> Click on the — sign to the left of the *Languages Resources* line to hide.

3. Click on the check box next to the *User Manual* line to uncheck it, then on the + sign placed to the left of this line to show possibilities and select the line *English*.<fnref n="17" /> Click on the — sign to the left of the *User Manual* line to hide.

Unitex programs are written in C language, while its interface is written in Java language. Java must therefore be installed on the computer, which is frequently the case. In fact, Unitex favors a certain version of Java, JRE, and tests if it is installed on the computer. However, if it is not, the following window appears:

<figure><img src="image_9.png" /></figure>

We have three choices: install Java ourselves, let Unitex install JRE or continue without installing it. If we know that Java is installed on the computer and we do not want to

<footnote n="15">¹⁵ The two licenses (for the programs and for the resources) are LGPL licenses that authorize their free use including the use in commercial products.</footnote>

<footnote n="16">¹⁶ In addition, some other languages, of course, if you want. We will use French in the chapter dealing with aligned texts (Chapter 6, Section 2, page 141).</footnote>

<footnote n="17">¹⁷ You can select the *French* manual as well if you prefer or check only *French*.</footnote>
additionally install the JRE version, we should choose the third option. Otherwise, we advise
you to choose the second option and let Unitex try to install it during the setup process.

After selecting the second option click on the Next button. If you are the administrator of your computer, Unitex offers you an installation only for you or an installation for all users of the computer:

<figure><img src="image_10.png" /></figure>

If you choose the *just for me* option, Unitex offers the installation in the User folder:

C:\Users\user_name\AppData\Local\Unitex-GramLab<fnref n="18" />

<figure><img src="image_11.png" /></figure>

If you choose the *for anyone* option, Unitex offers the installation in the Program Files (x86) folder:<fnref n="19" />

C:\Program Files (x86)\Unitex-GramLab\App

<footnote n="18">¹⁸，《user_name》 is your name under the Windows system.</footnote>

<footnote n="19">¹⁹ To choose this option, you must be an IT administrator.</footnote>
<figure><img src="image_12.png" /></figure>

Both options are fine. If you want to change this setting, don't install Unitex in the *MyUnitex* folder that you have created: this folder must remain separate from the installation folder. After choosing the folder for installation that is appropriate for your computer, click on the *Next* button, and after that on the *Finish* button:

<figure><img src="image_13.png" /></figure>

Unitex starts automatically; if that does not happen, see Section 1.2.1 that follows.

### 1.1.3 Initial choices

Unitex will then ask you to choose your private working directory (or working folder):

<figure><img src="image_14.png" /></figure>

Select then the *MyUnitex* folder that you have already created (see Section 1.1.1, page 7):

`C:\Documents\MyUnitex`

And then choose the language (in this case, *English*) and click on the *OK* button:

<figure><img src="image_15.png" /></figure>
### 1.1.4 Check or modify your private working directory

If you want to check your private working directory, you can open the `Info/Preferences.../Directories` menu.

<figure><img src="image_16.png" /></figure>

To modify your private working directory, click the `Set` button to choose a new one, and click on the `OK` button. Unitex must then be closed and reopened to take the changes into account.

<figure><img src="image_17.png" /></figure>

It is thus possible to have several private working directories, one per project.

The name of your private working directory is saved in a specific file which depends on where Unitex is installed.

If you chose the *just for me* option, Unitex save this name in the file:

`C:\Users\user_name\AppData\Local\Unitex\GramLab\Users\user_name.cfg`

Else, if you chose the *for anyone* option, Unitex save this name in the file:

`C:\Users\user_name\.unitex.cfg`

To avoid opening and closing Unitex to change the private working directory, it is possible to directly modify this file. You can also create a shortcut to it on the desktop.

## 1.2 Using Unitex

### 1.2.1 Starting Unitex

It can happen that Unitex is not launched at the end of the setup or that you have accidentally closed Unitex. It is also possible that you are using a computer on which Unitex is already installed. In these three cases, you must launch Unitex yourself, as you will always do when
wanting to work with it. The easiest way is to use the shortcut placed on the desktop during installation and double click it:<fnref n="20" />

<figure><img src="image_18.png" /></figure>

If you do not have this shortcut or it does not work, it is possible to launch Unitex by going to the folder where the Unitex program has been installed:

C:\Program Files (x86)\Unitex-GramLab\App

or

C:\Users\user_name\AppData\Local\Unitex-GramLab\App

Double-click on the file named Unitex having the type Executable Jar File. A permanent solution should be to send this file to the desktop as a shortcut by right clicking on this file and choosing the Send to/Desktop option. This will create a shortcut of the Unitex executable jar file.

### 1.2.2 The Unitex interface

Unitex processes different types of files: texts, dictionaries, graphs and so on. In the Unitex interface, each menu is specific to one of them and therefore has, for instance, an Open sub-menu (or option).

<table><thead><tr><td>Menus</td><td>types of files</td></tr></thead><tbody><tr><td>Text</td><td>text to analyze</td></tr><tr><td>DELA</td><td>dictionary</td></tr><tr><td>FSGraph</td><td>graph</td></tr><tr><td>Lexicon-Grammar</td><td>lexicon-grammar</td></tr><tr><td>XAlign</td><td>parallel texts</td></tr><tr><td>File Edition</td><td>textual file to edit</td></tr></tbody></table>

In addition, there are three more menus: Windows, Help and Info. The Unitex user manual can be opened directly from Unitex, via the Help menu.

<footnote n="20">²⁰ Whenever you update Java, you will need to redo this shortcut by restarting the installation or changing its target.</footnote>
1.2.3 Opening a text

Before starting this section

Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted`

Create a new folder named `DombeyAndSon` (without space).

Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon`<fnref n="21" />

Download the `DombeyAndSon.txt` file.

We will click the Text/Open... menu to open the file named:

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt`

and answer No to the question Do you want to preprocess the text?

<figure><img src="image_19.png" /></figure>

Unitex will create a new file: <fnref n="22" />

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.snt`

<figure><img src="image_20.png" /></figure>

The top bar displays some numbers: `0 sentence delimiter`, `803 028 (16 453 diff)` tokens, `363 065 (16 434) simple forms`, `0 (0) digits`. <fnref n="23" /> Their meaning is:

sentence delimiter: Unitex can insert the special symbol `{S}` at the end of a sentence during the preprocessing (see Section 3.3.1, page 37). Since we skipped the preprocessing there are no such symbols.

<footnote n="21">²¹ This file is the novel *Dombey and Son*, by English author Charles Dickens. According to [Wikipedia](https://en.wikipedia.org/wiki/Dombey_and_Son): "It follows the fortunes of a shipping firm owner, who is frustrated at the lack of a son to follow him in his footsteps; he initially rejects his daughter's love before eventually becoming reconciled with her before his death".</footnote>

<footnote n="22">²² Unitex never works on an original text file.</footnote>

<footnote n="23">²³ These numbers are recorded in the file named: `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\stats.n`</footnote>
*   **token**: it is either a string of letters<sup>24</sup> or any other single character (including spaces). There are 803 028 tokens in this novel, but only 16 453 different.

*   **simple form**: it is a string of letters. There are 363 065 of them in this novel, but only 16 434 different.

At the bottom left is a reduced window, the *Token list*, that lists the tokens of the text and their number of appearances. By double-clicking on it, we obtain this list in decreasing order of frequency. You can also have it in the alphabetical order by clicking the *Char Order* button.<fnref n="25" />

<figure><img src="image_21.png" /></figure>

We can notice that this *Token list* begins with space, punctuation characters and functional words, which is normal, because they are the most frequently used tokens.

### 1.2.4 Applying dictionaries

We will open the *Text/Apply Lexical Resources...* menu. There we will find available dictionaries, on the left those in the private working folder, on the right those that come with the Unitex distribution.<fnref n="26" /> The default dictionaries are preselected (`dela-en-public.bin` and `Dnum.fst2`).

If these files are not present, Unitex will not work. In this case you have to open the folder where Unitex is installed:

`C:\Users\maurel\AppData\Local\Unitex-GramLab\English` or

`C:\Program Files (x86)\Unitex-GramLab\English`

and copy the three files `Alphabet.txt`, `Alphabet_sort.txt` and `Norm.txt`, and then paste them in the private folder `English`:

<footnote n="24">24 Letters of a language are defined in the file: `MyUnitex\English\Alphabet.txt`
A sorting order, for arranging concordances and dictionaries, is specified in the file: `MyUnitex\English\Alphabet_sort.txt`</footnote>

<footnote n="25">25 These lists are recorded in the three files:
*   `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\tokens.txt`
*   `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\tok_by_freq.txt`
*   `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\tok_by_alph.txt`</footnote>

<footnote n="26">26 We have not yet dictionaries in the private working folder, see *Chapter 5*, page 107.</footnote>
<figure><img src="image_22.png" /></figure>

By clicking the Apply button, we apply the dictionaries to the text and we obtain the WordList.

<figure><img src="image_23.png" /></figure>

The WordList is displayed in three windows:<fnref n="27" /> DLF (22 604 simple forms), DLC (937 compounds or multiword units) and ERR (580 words that are not in applied dictionaries).

### 1.2.5 The DLF window

We will first look at the format of the eleventh line of the DLF window (the first line not displayed in the above screenshot):

abandoning, abandon.V:G

At the beginning of the line is the word from the text, followed by its heading form (or lemma), after the comma and before the period. After the period comes a word's part of speech (V for verb), and it is followed, after the colon, by the inflection code (G for gerund). This word from the text is recognized as possibly being the gerund of the verb to abandon. If the lemma form

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\dlf

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\dlc

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\err

They come with three additional files:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\dlf.n

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\dlc.n

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\err.n

These contain just the number of entries in the corresponding files.

<footnote n="27">²⁷ These windows correspond to the following files:</footnote>
does not appear, which is the case on the line just above the analyzed one where the period comes right after the comma, it means that the lemma is identical to the text word itself; in this case the lemma of the noun *abandoning* is simply *abandoning*.

It may seem strange to see two lines beginning with *abandoning*. We must keep in mind that words out of context are ambiguous, and *abandoning* can be the noun *abandoning* or the gerund of the verb *to abandon*.

It is possible to force one analysis rather than another by modifying the text. For instance, by replacing *after abandoning herself* with *after {abandoning, abandon.V:G} herself*, we explicitly assign the verb category to the word *abandoning*. This expression between curly braces is counted as a single token and is called a *lexical tag* (However, if you want to do that manually, you shall have to close the text with Unitex, edit the .txt file, save it and reopen it with Unitex).

A line can also contain one or more features for the entries, which are preceded by a plus sign. One example can be seen in the first line of the *DLF* window where the *Dind* features is used to indicate that the determiner is indefinite:

a,. DET+Dind:s

### 1.2.6 The *DLC* window

The *DLC* window has the same format as the *DLF* window. If the multiword form contains a character used as a delimiter (comma, period, plus, colon and slash),<fnref n="28" /> it must be preceded by a backslash. Such a case does not occur in our *DLC* window.

### 1.2.7 The *ERR* window

The words that were not recognized by the applied dictionary are mostly proper names (*Aldgate*, *Alfred*, *Alraschid*...); but also foreign language words: *Anno* (Latin), Roman numbers, unusual orthography (*adventures*), parts of contracted verbal forms (*dïn*, *hïsn...*) or simply words that were not recorded in the applied dictionaries (*nevyless*,<fnref n="29" /> *unbusiness*, *undauntable...*).

## 2 A detailed example

Before starting this section

Go to the folder: `MyUnitex\English\Graphs`

Create a new folder named `UnitexGettingStarted` (without space).

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted`

Create a new folder named `Meet`.

### 2.1 Concordances with one word query

Suppose, for example, that we want to study the use of the verb *to meet* in our novel. In order to do that we will open the *Text/Locate Pattern* menu, select the *Regular expression* option and type *meet* in the corresponding field.

<footnote n="28">²⁸ The slash enables comments, see *Chapter 5, Section 4.3.1*, page 125.</footnote>

<footnote n="29">²⁹ The word *nevy* is an old form of *nephew*.</footnote>
<figure><img src="image_24.png" /></figure>

We will launch the search by clicking the **Search** button, which will yield 58 matches.

<figure><img src="image_25.png" /></figure>

Clicking the **OK** button will open the *Located sequences* dialog box. In order to obtain concordances, we need to click the **Build concordance** button (situated at the bottom right of the *Located sequences* dialog box).

<figure><img src="image_26.png" /></figure>
The produced list is a hypertext file in which each recognized sequence leads to the corresponding location in the text.<fnref n="30" /> For instance, if we click on the first line, this opens the following window (right):

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

As the verb *to meet* has conjugated forms *meets*, *met* and *meeting*, we would have to repeat what we have just done three more times in order to obtain a concordance of these three forms as well.

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

If we look closely at the concordance of the word *meeting*, we can see that some lines represent a form of the verb *to meet* (*Mrs Miff*, *meeting* Mr Dombey's eye...), while others represent the noun *meeting* (*This is indeed a meeting!*). You will remember that in *Section 1.2.5*, page 15, we explained that some words can have several interpretations.

## 2.2 Concordances with a lemma

Instead of launching four different searches, we could have obtained the same result with one query in the form of the regular expression *meet+meets+meeting+met*. With this query, we

<footnote n="30">³⁰ These concordances are registered in the file: *MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon_snt\concord.html*
With each new search and concordance produced, this file is replaced, but you can save it using a different name if you want to keep it. The saved concordance can be reopened by the *Text/Open Concordance* menu.</footnote>
would obtain a list of 131 concordance lines, containing all the lines previously obtained in four subsequent runs (58+1+46+26). However, all these queries ignore the fact that Unitex uses dictionaries, containing among other entries, all forms of the verb *to meet*.<fnref n="31" />

...
meet,.A
meet,.N:s
meet,.V:W:P1s:P2s:P1p:P2p:P3p
meeting,.N:s
meeting,meet.V:G
meets,meet.N:p
meets,meet.V:P3s
...
met,meet.V:K:I1s:I2s:I3s:I1p:I2p:I3p
...

Instead of launching four searches, or listing all forms of the verb, we can use one query with the lemma form of the verb, <meet>.<fnref n="32" />

<figure><img src="image_32.png" /></figure>

With this query, we will obtain the same result, a list of 131 concordance lines.

## 2.3 Concordances with a graph query

Until now, only simple forms of the verb *to meet*, one of the four possible forms, appeared in the concordances. But we might like to highlight longer sequences in concordance lines: the sequences containing, beside forms of the verb *to meet*, also words that we see in the left context of concordance lines: *did meet*, *should meet*, *would meet*, *shall meet*, *will meet*, *would seldom meet*, *may ever meet*, *had met*, *have met*, *being met*, *had been met*, *have been met*, *had ever met*, *had never met* and *has met*. The reason to switch from regular expressions to using Unitex graphs will become obvious with these examples.<fnref n="33" />

<footnote n="31">³¹ Note that the noun *meetings* (plural form of *meeting*) is not present in this text.</footnote>

<footnote n="32">³² Note that this does not remove the ambiguity, because `<meet>` is also the lemma of a noun and an adjective.</footnote>

<footnote n="33">³³ For the sake of simplicity, we will omit the modal form (*may ever meet*) and the emphatic past (*did meet*).</footnote>
### 2.3.1 A first graph

Let us open the FSGraph/New menu. We will get a window with a starting point on the left and an ending point on the right.

<figure><img src="image_33.png" /></figure>

We will now create in five steps a graph that recognizes four word forms meet, meets, met and meeting:

1. Click on the starting point (start box).

2. Position the cursor a little more to the right, click the right mouse button and select Create box from the displayed menu.

<figure><img src="image_34.png" /></figure>

3. Type without any space meet+meets+met+meeting<fnref n="34" /> and finish with clicking the Enter key.

<footnote n="34">³⁴ These words will appear in the formula bar, just below the task bar of the graph window. They will replace the <E> symbol, which means empty. The plus sign means or.</footnote>
<figure><img src="image_35.png" /></figure>

4. Click on the *meet+meets+met+meeting* box and then the ending point (the end box).

<figure><img src="image_36.png" /></figure>

5. Click on the **Save** button (the leftmost in the task bar). Unitex will open the Graph folder to save it.<fnref n="35" />

Go to the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Meet

And save this graph in it using the name *meet*. Unitex will add the **.grf** extension.

<footnote n="35">³⁵ The whole process is shown in the video 2.2.3.1.meet.mp4 which is in the folder: MyUnitex\English\Graphs\UnitexGettingStarted\Meet\2.2.3.1</footnote>
**Notes**

1. Clicking on a box, then on another one creates a path between them or, if it already exists, removes it.

2. To delete a box, you must delete its content (by using the Delete key, for example), then validate with clicking the Enter key.

### 2.3.2 Concordances

In order to produce a concordance with our first graph, we will open the *Text/Locate Pattern* menu and click the *Set* button next to the *Graph* field. The *Graphs* folder will open and we will move to the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Meet

where we will find and select the *meet.grf* file.

<figure><img src="image_37.png" /></figure>

We will launch the search, as before, by clicking the **SEARCH** button. Finally, we will obtain the same list of 131 concordance lines.<fnref n="36" />

As before, we can replace four word forms with their common lemma.

1. Select the meet+meets+met+meeting box, type `<meet>` instead of meet+meets+met+meeting then press the Enter key.

2. Click the diskette icon, at the left end of the task bar, to save the graph.

<footnote n="36">³⁶ Note that *Locate Pattern* creates a new file, *meet.fst2*: it is a compiled version of the *meet.grf* graph. If we used the compiled version of a graph in *Locate Pattern* after modifying it, the modified version would not be used for search.</footnote>
<figure><img src="image_38.png" /></figure>

Now, we will begin to highlight longer sequences in concordance lines by adding the future tense and the conditional, i.e. four examples in our concordance: *shall meet*, *will meet*, *should meet* and *would meet*.

### 2.3.3 Adding the future tense and the conditional

The future and the conditional use the forms of the auxiliaries, *will*, *would*, *shall* and *should*, which can be represented by two lemmas `<will>` and `<shall>`. We will add two boxes to our graph:

1. Click on the starting box, then move the cursor up and to the right, click the right mouse button and select *Create box* from the displayed menu.

2. Type `<will> + <shall>`, then press the *Enter* key.

3. Click on the `<will> + <shall>` box, then move the cursor right, click the right mouse button and select *Create box* from the displayed menu.

4. Type *meet*, then press the *Enter* key.<fnref n="37" />

5. Click on the *meet* box and then on the final box.

6. Rearrange the graph.

7. Click the *Save* button.

<footnote n="37">³⁷ Pay attention: *meet* without angle brackets.</footnote>
<figure><img src="image_39.png" /></figure>

The concordance produced with this new *meet.grf* graph contains again 131 lines and among them lines with highlighted *shall meet, should meet, will meet and would meet*. You will find these occurrences at the end of the concordance list.

<figure><img src="image_40.png" /></figure>

Let us move to the perfect tenses. They consist of the verb *to have* followed by the past participle, *met*. Three such forms can be spotted in our concordance, in the left context of the form *met: had met, have met and has met*.

### 2.3.4 Adding the perfect tenses

The perfect tenses use the auxiliary forms *have, has, had* and *having*, which can be represented by the lemma <have>. We will enhance our graph by using the copy-paste
technique. We will add four boxes to our graph, one for the `<have>` lemma, one for the *have* form, and two for the *met* form:

1. Click on the starting box, then move the cursor down and to the right, click the right mouse button and select *Create box* from the displayed menu.

2. Type `<have>`, then press the *Enter* key.

3. Click on the `<have>` box, then move the cursor right, click the right mouse button and select *Create box* from the displayed menu.

4. Type *met*, then press the *Enter* key.

5. Click on the *met* box and then on the end box.

6. Select the `<have>` box and the *met* box by drawing with the mouse a rectangle that encloses both boxes, then click the *Copy* button (or use `CTRL-C` on the keyboard).

7. Click the *Paste* button (or use `CTRL-V` on the keyboard): the two boxes appear twice.

8. Move the copied boxes to the top of the graph. Click outside the graph to deselect them.

9. Click on the top `<have>` box and delete the angular brackets (*have* replaces `<have>`), then press the *Enter* key.

10. Click on the `<will> + <shall>` box and then on the *have* box.

11. Click on the *met* box and then on the end box.

12. Rearrange the graph.

13. Click the *Save* button.

<figure><img src="image_41.png" /></figure>

The concordance produced with this graph contains the lines with five highlighted sequences *had met*, two sequences *has met* and three sequences *have met*.
<figure><img src="image_42.png" /></figure>

## Two important notes on how copy-paste functions in Unitex

1. If only one box is selected, the box is not copied, only its content is. In order to copy-paste a single box, you have to copy the box, create a new one and paste the content.

2. Copying and pasting multiple boxes does not use the usual clipboard. Therefore, there may be competition between pasting text and pasting boxes. If some operation pastes boxes when you wanted to paste text, just erase the unwanted boxes by clicking the *Delete* key, then the *Enter* key. If you have copied both boxes and text and if you want to paste the text into some box, there is a trick that ensures that what is pasted is the text and not the boxes: just type or erase some character in the box before pasting the text.

It remains to insert the passive forms to our graph. They consist of the verb *to be* followed by the past participle, *met*. Three such forms can be found in our concordance, if we look at the left context of *met*: *being met*, *had been met* and *have been met*.

### 2.3.5 Adding the passive forms

The passive forms use the forms of the auxiliary *to be* (*am*, *are*, *was*, *were*, *be* and *being*) which can be represented by their lemma <be>.
We will add five boxes to our graph, one for the `<be>` lemma, one for the *be* form and one for the *met* form, two for the *been* form:

1. Select the whole graph by drawing with the mouse a rectangle that encloses all the boxes (or use `CTRL-A`<fnref n="38" /> on the keyboard) and then move it down. Click somewhere outside the graph to deselect it.

2. Click on the starting box, then move the cursor down and to the right, click the right mouse button and select *Create box* from the displayed menu.

3. Type `<be>`, then press the `Enter` key.

4. Click on the `<be>` box and then on the *met* box.

5. Click on the `<have>` box, then move the cursor up on the right, click the right mouse button and select *Create box* from the displayed menu.

6. Type *been*, then press the `Enter` key.

7. Click on the *been* box, then click on the *met* box.

8. Click on the *have* box, then move the cursor up and to the right, click the right mouse button and select *Create box* from the displayed menu.

9. Click the *Paste* button (or use `CTRL-V` on the keyboard): *been* is pasted. Press the `Enter` key.

10. Click on the *been* box and then on the *met* box.

11. Click on the `<will> + <shall>` box, then move the cursor down and to the right, click the right mouse button and select *Create box* from the displayed menu.

12. Type *be*, then press the `Enter` key.

13. Click on the *be* box, then move the cursor to the right, click the right mouse button and select *Create box* from the displayed menu.

14. Type *met*, then press the `Enter` key.

15. Click on the *met* box and then on the *end* box.

16. Click the *Save* button.

<figure><img src="image_43.png" /></figure>

The concordance produced with this graph highlights sequences *being met*, *had been met* and twice *have been met*.

<footnote n="38">³⁸After the selection of two boxes by drawing with the mouse a rectangle that encloses both.</footnote>
<figure><img src="image_44.png" /></figure>

If we look closer at our concordance, we will notice that sometimes an adverb is inserted between an auxiliary and a main verb, as in the sequences *would seldom meet*, *had ever met* and *had never met* which are therefore not recognized.

### 2.3.6 Adding adverbs

In order to retrieve these sequences as well, we can insert in our graph a box containing *seldom*, *ever* and *never*. However, we should try to be more general because almost any adverb can occur at this position. If we look closer at the Word List (see Section 1.2.4, page 14), we will see that its third line is:

aback, ADV

We learn from it that the code for adverbs used in dictionaries is *ADV*. We will therefore add a box containing `<ADV>` at appropriate positions in our graph.
1. Click on the <be> box, then move the cursor to the right, click the right mouse button and select Create box from the displayed menu.

2. Type <ADV>, then press the Enter key.

3. Click on the <ADV> box and on the met box.

4. Rearrange the graph.

5. Click on the <have> box, then click the Copy button (or use CTRL-C on the keyboard). Click somewhere in the blank space to deselect it.

6. Click on the start box, then move the cursor down and to the right, click the right mouse button and select Create box from the displayed menu.

7. Click the Paste button (or use CTRL-V on the keyboard): <have> is pasted. Then press the Enter key.

8. Click on the previous <have> box and type <E>.<fnref n="39" />

9. Click on the new <have> box and on the empty box.

10. Click on the start box and on the empty box. <fnref n="40" />

11. Click on the <have> box, then move the cursor up and to the right, click the right mouse button and select Create box from the displayed menu.

12. Type <ADV>, then press the Enter key.

13. Click on this new <ADV> box and on the empty box.

14. Rearrange the graph.

15. Click on the <will> + <shall> box, then click the Copy button (or use CTRL-C on the keyboard). Click somewhere in the blank space to deselect it.

16. Click on the start box, then move the cursor up and to the right, click the right mouse button and select Create box from the displayed menu.

17. Click the Paste button (or use CTRL-V on the keyboard): <will> + <shall> is typed. Then press the Enter key.

18. Click on the previous <will> + <shall> box and type <E>.

19. Click on the start box and on the new empty box.

20. Click on the <will> + <shall> box and on the new empty box.

21. Click on the <will> + <shall> box, then move the cursor up and to the right, click the right mouse button and select Create box from the displayed menu.

22. Type <ADV>, then press the Enter key.

23. Click on the new <ADV> box and on the new empty box.

24. Rearrange the graph.

25. Click the Save button.

<figure><img src="image_45.png" /></figure>

<footnote n="39">³⁹ Remember that the <E> symbol means empty.</footnote>

<footnote n="40">⁴⁰ This will remove the existing path.</footnote>
Finally, the concordance produced with this graph highlights the sequences *had ever met*, *had never met* and *would seldom meet*.

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

## 2.4 Contexts

### 2.4.1 Left context

#### 2.4.1.1 The verb to meet preceded by a pronoun

Suppose that instead of displaying all occurrences of the verb *to meet*, we only want to display occurrences in which the verb is preceded on the left by a pronoun (or a pronoun followed by an adverb). We will erase one box and add two boxes in our graph:

1. Open the FSGraph/Save as... menu to rename the graph to *meetLeft.grf*.

2. Select the whole graph by drawing with the mouse a rectangle that encloses all the boxes, except the starting box, and then move it to the right. Click somewhere in the blank space to deselect it.<fnref n="41" />

3. Rearrange the graph to align vertically the `<will> + <shall>` box, the `<meet>` box, the `<have>` box and the `<be>` box.<fnref n="42" />

<figure><img src="image_48.png" /></figure>

4. Select the `<will> + <shall>` box, the `<have>` box and the `<be>` box, then click the *Reversed link between boxes* button in the tool bar, in the third group of options, fifth in the group. Click on the starting box (this will remove three paths).

<figure><img src="image_49.png" /></figure>

5. Click the *Normal editing mode* button in the same third group of options, first in the group.

6. Click on the starting box, then move the cursor to the right, click the right mouse button and select *Create box* from the displayed menu.

<footnote n="41">⁴¹ Or select any two boxes by drawing with the mouse a rectangle that encloses both, then select the whole graph by typing `CTRL-A` on the keyboard and then move it right. Click somewhere in the blank space to deselect it. Click on the starting box, then move it left.</footnote>

<footnote n="42">⁴² To align boxes vertically (or horizontally), we can also select them and right-click to open the *Format/Alignment...* menu.</footnote>
7. Click on the newly created box and type `I+you+he+she+it+we+they`, then press the Enter key.<fnref n="43" />

8. Click on the `I+you+he+she+it+we+they` box, then move the cursor to the right, click the right mouse button and select `Create box` from the displayed menu. This box will remain empty.

9. Click on the `I+you+he+she+it+we+they` box, move the cursor up and to the right, then click the right mouse button and select `Create box` from the displayed menu.

10. Click on the newly created box and type `<ADV>`, then press the Enter key.

11. Click on the `<ADV>` box and on the empty box.

12. Select the `<will> + <shall>` box, the `<meet>` box, the `<have>` box and the `<be>` box.

13. Click the *Reversed link between boxes* button. Click on the empty box (this will create three paths).

14. Click the *Normal editing mode* button.

15. Click the *Save* button.

<figure><img src="image_50.png" /></figure>

This graph will produce the concordance with forty matched sequences.<fnref n="44" />

<figure><img src="image_51.png" /></figure>

<footnote n="43">⁴³ If you wish to use the corpus *Ivanhoe.txt*, distributed with Unitex, you must add to this list the second person singular, *thou*. However, *thou* never precedes a form of the verb *to meet* in the text we use.</footnote>

<footnote n="44">⁴⁴ Do not forget to use the new *meetLeft.grf* graph in *Locate pattern*.</footnote>
### 2.4.1.2 The verb to meet preceded by a pronoun and a left context

The concordance we obtained is not like the previous ones, because pronouns are part of the highlighted sequences. It is possible to avoid this if we use the so-called left context:

1. Select the `<will>`+`<shall>` box, the `<meet>` box, the `<have>` box and the `<be>` box.

2. Click the *Reversed link between boxes* button in the tool bar, in the third group of options, forth in the group.

<figure><img src="image_52.png" /></figure>

3. Click the first green button (`$*`, *Inserts left context mark before box selection*) situated at the right end of the graph editor toolbar.

4. Rearrange the graph.

5. Click the *Save* button.

<figure><img src="image_53.png" /></figure>

We obtain nearly the same concordance: pronouns are not highlighted anymore, but occurrences are still displayed only when the pronoun is present.

<figure><img src="image_54.png" /></figure>

### 2.4.2 Right context

It is possible to do the same thing on the right. It is done in a slightly different way. We will start from the *meet.grf* graph and suppose that we want to highlight only the left context of the verb *to meet* (the auxiliaries) and not the verb itself.
1. Open the *meet.grf* graph.

2. Open the FSGraph/Save as... menu to rename the graph as *meetRight.grf*.

3. Click on the <meet> box and type *Delete* and *Enter*.

<figure><img src="image_55.png" /></figure>

4. Select the *meet* box and click the second green button ($\$[$, *Surround box selection with right context tags*) situated at the right end of the graph editor toolbar.

5. Select the *met* box and click again the *Surround box selection with right context tags* button (three times).

6. Rearrange the graph.

7. Click the *Save* button.

<figure><img src="image_56.png" /></figure>

The concordance obtained by applying this graph contains 28 matches (among 131 matches obtained by the *meet.grf* graph, 103 recognize only the main verb *to meet* in its various forms). In the recognized sequences, the verb *to meet* is not highlighted.

<figure><img src="image_57.png" /></figure>
### 2.4.3 Lexical masks

We have already introduced two types of lexical masks: for instance, `<meet>` matches all forms of the lemma *meet* (see Section 2.2, page 18) while `<ADV>` matches all adverbs (see Section 2.3.6, page 28). These masks rely on the dictionaries used.

There are other masks depending on the use of upper and lower case letters:<fnref n="45" /> `<UPPER>` for a word in uppercase, `<FIRST>` for a word starting with uppercase and `<LOWER>` for a word in lowercase. Besides them, the mask `<WORD>` corresponds to a sequence of letters; the mask `<NB>` to a sequence of digits; the mask `<DIC>` to any dictionary entry; and the mask `<TOKEN>` to any token (see Section 1.2.3, page 13).<fnref n="46" />

We will now transform the *meetLeft.grf* graph to recognize all forms of the verb *to meet* that are preceded by a word starting with uppercase. We will name this graph *meetLeftRight.grf*:

1. Open the *meetLeft.grf* graph.

2. Open the FSGraph/Save as... menu to rename the graph as *meetLeftRight.grf*.

3. Click on the `I+you+he+she+it+we+they` box and type `<FIRST>`.

4. Select the `<ADV>` box and the empty box by drawing the rectangle over these boxes, then type *Delete* and *Enter*.

5. Click on the `<FIRST>` box and the `$*` box.

6. Rearrange the graph.

7. Click the *Save* button.

<figure><img src="image_58.png" /></figure>

Now, the concordance displays only twenty-one lines.

<footnote n="45">⁴⁵ The uppercase and lowercase letters are defined in the Alphabet.txt file, see Section 3.3.3.1, page 41.</footnote>

<footnote n="46">⁴⁶ There are some other masks, see the Unitex user manual, Section 4.3.1. Beware that for the preprocessing, the topic of the next section, different masks are valid (see Section 2.5.2 in the Unitex user manual).</footnote>
<figure><img src="image_59.png" /></figure>

# 3 The .snt file

## 3.1 Document transformation

### 3.1.1 Standard normalization

Before splitting the text into tokens (see Chapter 2, Section 1.2.3, page 13), Unitex normalizes the text. This new text is saved with the same name as the original text, but with the .snt extension. It is constructed as follows:

1. Any sequence of spaces and tabulation characters is replaced by a single space (recognized by a path between two boxes).

2. Any series of line breaks (possibly with spaces and tabs) is replaced by a single line break (also recognized by a path between two boxes).<fnref n="47" />

3. Braces are replaced by square brackets, unless they correspond to a dictionary entry (a lexical tag, see Section 1.2.5, page 15).

### 3.1.2 The Norm.txt file

The *Norm.txt* file in the Unitex distribution is important:

MyUnitex\English\Norm.txt

We are going to open this file using the *File Edition/Open/Other files* menu. It only contains two lines.

<figure><img src="image_60.png" /></figure>

<footnote n="47">⁴⁷ The *No separator normalization* option, in the *Preprocessing* dialog box, only works for preprocessing graphs. For other graphs, a path on a graph between two boxes may pass not only through one space or one line break, but also through a series of spaces, tabs and line breaks.</footnote>
These two lines define the rule for transforming braces into brackets. Each line consists of a character string (the text to be replaced), followed by the tabulation character, then another character string (the replacement text) and finally a line break (Enter).

Note that this file is strictly speaking not necessary, because this rule applies anyway, even if the *Norm.txt* file is not present. It is actually there to indicate how to add more lines.

### 3.1.2.1 The standardization of dashes and quotation marks

There are two types of dashes in our text: 2 034 hyphens (-) and 2 378 em dashes (—). Similarly, there are three types of quotation marks in our text: 7 766 left double quotation marks ("), 7 824 right double quotation marks (”) and 2 quotation marks (").<fnref n="48" /> If we would like to normalize them to hyphens (-) and quotation marks ("), we can add three lines to our *Norm.txt* file:<fnref n="49" />

1. Open the *Norm.txt* file.

2. Type `-`, `Tab`, `-` and `Enter`.<fnref n="50" />

3. Type `", Tab, "` and `Enter`.

4. Type `", Tab, "` and `Enter`.

5. Click the `Save` button.

<figure><img src="image_61.png" /></figure>

Now, close Unitex and reopen it. Open the text (the original `.txt` text, not the already processed `.snt` text) and skip all preprocessing to get 4 412 hyphens (-) and 15 592 quotation marks (""). However, if we look at the number of different tokens, we can notice that we have three unique tokens less than when processing the text with the old *Norm.txt* file: 16 450. Three unique characters were replaced with two existing unique characters.

### 3.1.2.2 The standardization of apostrophes

Looking closer, we see that the novel uses the *right single quotation mark* (') as an apostrophe and not the keyboard character, the *punctuation apostrophe* ('). The novel also uses the left single quotation mark ('). If we want to normalize these two single quotation marks to punctuation apostrophe, we can add two new lines to our *Norm.txt* file:

1. Open the *Norm.txt* file.

2. Type `', Tab, '` and `Enter`.

3. Type `', Tab, '` and `Enter`.

4. Click the `Save` button.

<footnote n="48">⁴⁸ There are more than two types of dashes and three types of quotation marks according to the Unicode standard; however only those mentioned here appear in our text. If you do not know how to type these characters, the best thing is to locate them in the text and copy and paste them.</footnote>

<footnote n="49">⁴⁹ In order to apply the modified *Norm.txt* file, it is necessary to close Unitex and then to reopen it.</footnote>

<footnote n="50">⁵⁰ For example, by copying and pasting non-ASCII characters from the novel.</footnote>
<figure><img src="image_62.png" /></figure>

Now, close Unitex and reopen it. Open the text (the original .txt text, not the already processed .snt text) and skip all preprocessing.

<figure><img src="image_63.png" /></figure>

If we compare the .snt text from Section 1.2.3, page 13, we can see that five tokens are missing (16 453 different tokens versus 16 448).

## 3.2 The preprocessing

Unitex offers two possibilities of preprocessing:

1. Switch on one or two graphs, one working in *Merge mode* and one in *Replace mode*. These graphs can recognize a line break, which is represented by <^> in a box. After normalization, the line break will not be distinguished from the space and will be recognized by a path between two boxes.

2. Launch the default dictionaries (see Section 1.2.4, page 14).

By default, switching on *Merge mode* will use the graph:

MyUnitex\English\Graphs\Preprocessing\Sentence\Sentence.grf

while switching on *Replace mode* will use the graph:

MyUnitex\English\Graphs\Preprocessing\Replace\Replace.grf

## 3.3 Material (only for confident users)

### 3.3.1 The Sentence.grf graph

The *Sentence.grf* graph inserts the Unitex symbol {S} (*sentence delimiter*) at the end of the sentence.

We will open the file:

MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt
by using the *Text/Open...* menu and then answer *Yes* to the question *Do you want to preprocess the text?* By default, two boxes are checked. If we check only the box next to *Apply graph in MERGE mode*

<figure><img src="image_64.png" /></figure>

the displayed information for the file:

MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.snt

will change.

<figure><img src="image_65.png" /></figure>

Unitex has inserted 16 515 sentence delimiters.<fnref n="51" />

A *sentence delimiter* can be placed and recognized in a box of a graph. For instance, create the *sentenceWithoutDot.grf* graph and record it in the same folder as the *Meet.grf* graph:

1. Open the FSGraph/New menu.

2. Click on the starting box, then move the cursor to the right, click the right mouse button and select *Create box* from the displayed menu.

3. Click on the newly created box and type `?+!`, followed by *Enter*.

4. Click on the `?+!` box, then move the cursor to the right, click the right mouse button and select *Create box* from the displayed menu.

5. Click on the newly created box and type `{S}`, followed by *Enter*.

6. Click on the `{S}` box and on the final box.

7. Click the *Save* button.

<footnote n="51">⁵¹ Since the last sentence is not followed by the *sentence delimiter*, there are therefore 16 516 recognized sentences. The number of tokens is now 803 028+16 515=819 543, because `{S}` counts as one token. There is also one more different token.</footnote>
<figure><img src="image_66.png" /></figure>

In order to produce a concordance with this graph, we will open the *Text/Locate Pattern* menu and click the *Set* button next to the *Graph* field. The *Graphs* folder will open, and we will move to the folder

MyUnitex\English\Graphs\UnitexGettingStarted\Meet

where we will find and select `sentenceWithoutDot.grf`. The search with this graph has more than 200 matches; therefore, we must modify the *Search limitation* option by checking *Index all occurrences in text*. When we click the *SEARCH* button, we obtain 1 316 sentence delimiters preceded by a question mark or an exclamation mark.

<figure><img src="image_67.png" /></figure>

If we select the *Regular expression* option and type `?+!` in the search field, we obtain 4 469 question marks or exclamation marks in the text, difference due to the presence of direct speech.
<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

This may seem confusing, but if we look at the following sequence: *Dear Mama! what is...*
(among the first matched sequence in the concordance), we can see that the *sentence delimiter* is not inserted after each question or exclamation mark by the applied *Sentence.grf*.
The users may modify the sentence graph supplied by the Unitex distribution or write their
own sentence graph.

### 3.3.2 The Replace.grf graph

The *Replace.grf* graph modifies the text by replacing the contracted verb forms with the
corresponding full forms, for instance *I'm* is replaced by *I am*.

We will open the file

*MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt*

by using the *Text/Open...* menu and then answer *Yes* to the question *Do you want to preprocess the text?* If we check only the box next to *Apply graph in REPLACE mode*

<figure><img src="image_70.png" /></figure>

the displayed information for the file:

*MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.snt*

does not change from the text without preprocessing ([Section 3.1.2.2](#), page 36), because the
number of tokens remains the same; however, frequencies of some tokens change. We can
see, at the thirteenth paragraph, the first transformation: *couldn't* is replaced by *could not*.
<figure><img src="image_71.png" /></figure>

The both sequences have the same number of tokens, three; that is the reason that the number of tokens in the whole novel has not changed. On the other hand, could already existed in the text, so the number of different tokens decreases (16 448 to 16 434).

### 3.3.3 The two alphabet files

#### 3.3.3.1 The Alphabet.txt file

Recall (see Section 1.2.3, page 13) that a token is either a string of letters or any other single character. What is considered a letter is defined by the *Alphabet.txt* file:

MyUnitex\English\Alphabet.txt

Since there are two cases in English, all letters are presented in pairs: uppercase and lowercase. We can open this file using the *File Edition/Open/Other Files* menu.

<figure><img src="image_72.png" /></figure>

The pairs mean that a word in a box written in lowercase recognizes the same word with uppercase letters in a text. The reverse is not true; that is, if we place uppercase letters in a
box, only words with uppercase at that position are recognized. It is also possible to enforce case-sensitive matching using quotation marks.<fnref n="52" />

<table><thead><tr><th>This box</th><th>recognizes</th></tr></thead><tbody><tr><td>word</td><td>word, Word, WORD, but also wOrd, etc.</td></tr><tr><td>Word</td><td>Word, WORD, WOrd..., but not word, wORD, etc.</td></tr><tr><td>WORD</td><td>WORD and nothing else</td></tr><tr><td>"Word"</td><td>Word and nothing else</td></tr></tbody></table>

The same is true when a dictionary is consulted; the entry:

`word,..,N:s`

matches words `word`, `Word`, `WORD`, but also `wOrd`, etc., while the imaginary entry

`WORD,..,N:s`

would match only the word `WORD`.

### 3.3.3.2 The `Alphabet_sort.txt` file

The `Alphabet_sort.txt` file:

`MyUnitex\English\Alphabet_sort.txt`

defines the sort order of the letters of the alphabet. It is used to sort concordances and dictionaries, when presented in alphabetical order. The alphabetical sort implemented in Unitex is very fast, even on very large files. We will open this file by using the File Edition/Open/Other Files menu.

<figure><img src="image_73.png" /></figure>

<footnote n="52">⁵² Or, as we have seen in Section 2.4.3, page 34, by using lexical masks.</footnote>
In this file, letters are also represented in pairs, uppercase and lowercase. The sort order is defined in two levels. The first level is defined by lines, meaning that, for instance, each A is preceding each B, whether it is written in uppercase or not, whether it is accented or not. The second level is defined by each line. Consider the line:

ÉÉÈÊËeéèêë

It imposes that an *E* without an accent precedes an *É* with an acute accent, and also that an *É* with an acute accent precedes an *È* with a grave accent, and so on. For instance, in French, *PÊCHER* or *pêcher* (*to fish*) and *PÉCHER* or *pécher* (*to sin*) are ranked as follows:

*PÉCHER*, *PÊCHER*, *pécher*, *pêcher*

However, the whole group of letters has its own order in relation to other groups of letters. Thus, the words *ÉTÉ* or *été* (*summer*) and *ÈRE* or *ère* (*era*) and *EUROPE* or *Europe* are ranked:

*ÈRE*, *ère*, *ÉTÉ*, *été*, *EUROPE*, *Europe*

because the *Rr* group precedes the *Tt* group and the *Tt* group precedes the *Uu* group.
<figure><img src="image_74.png" /></figure>
Before starting this chapter

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted`

Create three new folders named *Will*, *Numbers* and *RomanNumerals*.

# 1 First example: compound verbs using the verb will

We will start this chapter by opening the text *DombeyAndSon.txt* (`Text/Open` menu) in the folder:

`MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted\DombeyAndSon`

and by answering *Yes* to the question *Do you want to preprocess the text*. A window will open in which we will remove the first two checkmarks, leaving just the last one *Apply All default Dictionaries*.

<figure><img src="image_75.png" /></figure>

## 1.1 First step: the simplest graph

Let us create a first graph that we will save, like all graphs in this chapter, in the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\Will`

We will name it *willV.grf*. This graph will recognize all forms of the verb *will* (*forms will* and *would*) followed by a verb in the infinitive (the future tense) and insert the XML tags `<verb>...</verb>` for the recognized forms. To insert an annotation into a box, add a trailing slash and type the desired annotation. This annotation will be placed just before the text recognized by the box. To place an annotation after the box, you must therefore add an empty box after it (with the annotation).

Generally, the lexical mask `<will>` represents all forms of the lexeme *will*. However, there are two lexemes `<will>`, a verb (*will*) and a noun (*a will*, *wills*). In order to avoid the recognition of nominal forms of *will*, we have to precise its part-of-speech with an appropriate code `<will.V>`.<fnref n="53" />

<footnote n="53">⁵³ Pay attention to the fact that information inside a lexical mask is case sensitive.</footnote>
In the English dictionary distributed with Unitex, the code for infinitive forms of verbs is *W*.
The lexical mask that recognizes an infinitive form of any verb is <V:W>.

<figure><img src="image_76.png" /></figure>

Start the search (*Text/Locate patterns*) using the *willV.grf* graph (*Set* button in the *Graph* line) with options *Merge with input text* and *Index all occurrences in text*. Click the *Search* button and then click the *OK* button. We get 1 146 matches. To make the concordance easier to read, we are going to slightly modify the display parameters by choosing to display 20 characters on the left and 255 characters on the right of the matched sequences. Finally, click on the *Build concordance* button.

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

You will see that at the beginning of the concordance list there are some false recognitions.
Namely, in a text segment *could I do more than render up my whole will and being to you, will*
is a form of a noun and *and* is a conjunction. This is because, in the English dictionary
distributed with Unitex a form *will* has two interpretations: a noun and a verb, while *and* has
also two interpretations: a conjunction and a verb.<fnref n="54" /> We will see in the section devoted to dictionaries how some cases of ambiguities can be avoided.

## 1.2 Second step: future passive and future perfect tense

When looking at the produced concordance we observe that in several cases *will be* and *will have* were recognized as a form of future tense, but the recognition was not complete. Now we will modify our graph so that it recognizes also passive tenses and forms of the future perfect tense. In the English dictionary distributed with *Unitex*, past participles are encoded with *K* and a lexical mask for retrieval of past participle word forms is `<V:K>`. In order to extend our graph, we will add a new path that exits from the `<will:V>` box. In this path, a form of a past participle must be preceded by *be* or *have*. Note that in this case, we use strings and not lexical masks, because these are exact forms that have to be recognized. We could have also used `<be.V:W>` and `<have.V:W>` but these masks correspond to exactly one form each.

<figure><img src="image_79.png" /></figure>

Open the *Text/Locate Pattern...* menu and click the *Search* button. We will obtain again 1 146 matches. Instead of producing concordances, we will click the *Show differences with previous concordances* button in the *Locate sequences* dialog box. We will obtain the parallel list of matches obtained by the previous graph and matches obtained by the modified graph.

<figure><img src="image_80.png" /></figure>

Open the *Text/Located sequences...* menu and click the *Build concordance* button. You will obtain the new concordance list.

<footnote n="54">⁵⁴ An intransitive verb with dialectal use, meaning: to breathe; whisper; devise; imagine (source: Wiktionary).</footnote>
<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

You should note that, in the *Locate pattern* dialog box, we used the default option *Index longest matches*, which means that shortest matches at the same position will not be indexed or listed in concordances. In order to understand what it means, open the *Text/Locate pattern...* menu, change the default option to *Index all matches* and click the *Build concordance* button. Now, you will obtain 1 382 matches and among them are both shorter and longer matches, for instance *will be associated* and *will be*. You may wonder why two concordance lines containing these two matches are far apart in the concordance list. When building concordances, we used the default option for sorting concordance lines (*Located sequences/Sort according to*) which is *Center, Left*. This option means that concordances are first sorted according to matches (from left to right) and then according to the left context (from right to left). In this case, the occurrence `<verb>will be associated</verb>` will be listed before `<verb>will be</verb>` associated.

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

## 1.3 Third step: recognizing the future perfect passive

We will modify our graph to recognize more compound verb forms with the lexeme `<will.V>`. Now we must split the *be+have* box because the forms *been* and *had* can follow *have* but not be.
<figure><img src="image_85.png" /></figure>

Open the *Text/Locate Pattern...* menu, return to the original indexing option: *Longest matches* and click the *Search* button. By clicking the *Build concordances* button in the *Locate sequences* dialog box, you will obtain the new concordance. This list again contains 1 146 lines. Note that the added forms *been* and *had* are also past participles of verbs *to be* and *to have* respectively, so now some longer (complete) compound forms are recognized instead of shorter ones: for instance, *would have been* (before) is replaced by *would have been added* (now).<fnref n="55" />

<figure><img src="image_86.png" /></figure>

1.4 Fourth step: recognizing continuous forms

In order to recognize continuous forms: future continuous and future perfect continuous, we
will modify our graph again. Continuous forms are built using the present participle for which
we use the lexical mask <V:G>. This mask will be used in two new boxes, once for the future
continuous tense and once for the future perfect continuous tense. We must split the
*been+had* box because the present participle can follow *will have been*, but not *will have had*.

<footnote n="55">⁵⁵ You can analyze all differences obtained by this new graph if you use the *Show differences with previous concordances* option in the *Located sequences* dialog box.</footnote>
<figure><img src="image_87.png" /></figure>

Now the concordance highlights four new sequences:<fnref n="56" /> *will be giving, would be dancing, would be enchanting and would be obliging*.

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

## 1.5 Fifth step: insertion of an adverb after *will* or *would*

### 1.5.1 A single adverb

Now we will take into account that an adverb can be inserted after the verb <will>. Since the code used for adverbs in the English dictionary distributed with Unitex is **ADV**, we will use for them the lexical mask <ADV>. The tags around them in a text will be <adverb>...</adverb>.

<figure><img src="image_90.png" /></figure>

Save the modified graph and use it in *Locate pattern*. You will obtain now 1 330 concordance lines. This time, they contain new matches, because occurrences with an adverb after <will>.V> could not be retrieved with the previous version of the graph. Some of the occurrences can be seen in the following figure.

<footnote n="56">⁵⁶ The number of matches remains the same because the previous graph recognized *will be*.</footnote>
<figure><img src="image_91.png" /></figure>

1.5.2 Several adverbs

However, it sometimes happens that an adverb is followed by another adverb or even more.
We will add a loop between the two boxes concerned.

<figure><img src="image_92.png" /></figure>

The new concordance shows 13 new matches. You can also use the *Show differences with previous concordance* option in the *Located sequences* menu to spot them.

<figure><img src="image_93.png" /></figure>
## 1.6 Sixth step: insertion of an adverb after *be*

The position after the verb <will.V> is not the only one where an adverb can occur. It can also be inserted between the verb *to be* and the present or past participle.

This should be added as optional after the *been* and *be* boxes in our previous graph. Therefore, we will modify our graph once again. However, rather than duplicating the boxes and loop of the previous section, we will create a subgraph to recognize adverbs.

### 1.6.1 A graph that recognizes and annotates adverbs

We observe that adverbs can appear at two positions in a verb group, so the same boxes for recognition and annotation would appear twice. In such situations, it is good to use subgraphs. First, the same problem of recognizing and annotating adverbs can appear in some other graph and then the subgraph can be reused.<fnref n="57" /> Second, the graph that recognizes verb groups with *will/would* becomes less complex and easier to read and to develop further.

The easiest way to create a subgraph named *adverb.grf* from the existing graph is to use the *Export as new graph* option (select boxes that will belong to a subgraph and use the right mouse button). Here we select the `<ADV>/<adverb>` and `<E>/<adverb>` boxes, then we right-click to choose the *Export as new graph* option. We call this graph *adverb.grf*.

The boxes of the main graph, which recognize and annotate the adverbs, must then be replaced by a call to the subgraph (name preceded by a colon, here :adverb). In the main graph, boxes that call a subgraph have a gray background.

<figure><img src="image_94.png" /></figure>

The concordance obviously remains the same.

### 1.6.2 The main graph

Let us modify our graph again to recognize occurrences of adverbs after the verb *to be* (in two places).

<footnote n="57">⁵⁷ See *Chapter 4, Section 4.3*, page 95 (only for confident users).</footnote>
<figure><img src="image_95.png" /></figure>

We obtain the same number of concordance lines, but longer sequences are recognized and
highlighted.

<figure><img src="image_96.png" /></figure>

1.6.3 A new subgraph

Before continuing to improve our graph, we will simplify it by creating a new subgraph. Since
the next steps will enhance the beginning of the graph without changing anything in its second
part, we will create a new subgraph from it. In order to make the main graph easy to
understand we will leave the insertion of the end-tag </verb> in the main graph. We will give
the name infinitiveVerb.grf to the new subgraph.

<figure><img src="image_97.png" /></figure>

If you apply this modified graph to the text, you will again obtain the same concordances,
because we have only reorganized the graph.
# 1.7 Seventh step: negative adverbs

## 1.7.1 The main graph

In the concordance, you can see two special adverbs, *never* and *not*, which mark negative forms.

<figure><img src="image_98.png" /></figure>

We will now treat them separately, as a negation. They usually follow another adverb, although sometimes they can precede it. We will create a subgraph for negation, but before doing that we will put a call to a future subgraph named *negation.grf* at the right place in the main graph. This graph does not exist yet, so this box has a red background.

<figure><img src="image_99.png" /></figure>

## 1.7.2 The negation subgraph

We can easily create this new subgraph from the main graph: we need to click on the **:negation** box, click the right mouse button and select **Open subgraph** from the menu displayed, then we need to click the **Yes** button to create the new graph.

After creating and saving the negation graph, the background of the **:negation** box in the main graph becomes gray.

<figure><img src="image_100.png" /></figure>

**Notes**

A call to a subgraph has a red background if:

1. The subgraph exists, but the main graph has not yet been saved.

2. The subgraph exists but it is not in the same folder as the main graph.

3. The subgraph does not exist, it will be created later.

4. The subgraph does not exist because the call contains a spelling error.

If not, the call to a subgraph has a grey background.
1.7.3 Negative right context

If we look closely at the concordance obtained, we may be surprised to see 93 more lines, a
total of 1 436 lines.<fnref n="58" /> These are lines in which *never* and *not* are annotated as *negation*; they
appear in addition to the 93 lines annotated as *adverb*. The *Longest matches* option does not
allow for choosing between the two annotations; they are therefore both displayed.

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

If we look at the list of differences between the two concordances (see Section 1.2, page 47),
we will see that the new concordance includes all forms of adverbs *never* and *not* labeled also
as *negation*.

<figure><img src="image_103.png" /></figure>

This is normal, because *never* and *not* are both adverbs and negations, but that is not what we
want. To force the negation tag, we will use a *negative right context* in the *adverb.grf*
subgraph, which is not to be confused with the *right context* seen in *Chapter 2, Section 2.4.2*,
page 32.

<figure><img src="image_104.png" /></figure>

<footnote n="58">⁵⁸ There is the same number of matches as before, 1 343, but more concordance lines, 1 436.</footnote>
Now, *never* and *not* are always annotated as negation. There are again 1 343 lines in the produced concordance.

<figure><img src="image_105.png" /></figure>

## 1.8 Eighth step: adding the verb *shall*

The verb *shall* follows the same rules as the verb *will*, making it easy to add it in the main graph. Since this verb is not ambiguous with a noun or some other category, as *will* is, it is not necessary to add its category in a lexical mask, so we will use *<shall>*.<fnref n="59" />

<figure><img src="image_106.png" /></figure>

The new concordance contains 1 948 lines.

<figure><img src="image_107.png" /></figure>

<footnote n="59">⁵⁹ Keep in mind that the lexical mask *<shall>V* would not be incorrect; these two masks produce the same results.</footnote>
## 1.9 Continuation of this graph (only for confident users)

Before starting this section

In order to retrieve all results you should use the *Norm.txt* file introduced in *Chapter 2*, *Section 3.1.2*, page 35. If you have not used it when you preprocessed the text, download it, then close and reopen Unitex.

Open the original *DombeyAndSon.txt* file, process it by applying the default dictionaries (do not use *Sentence* or *Replace* graphs).

### 1.9.1 Addition of contracted forms

In the *will* graph, we have omitted the recognition of contracted forms *won't*, *wouldn't*, *shan't* and *shouldn't*. We need a separate path for them because, in the case of contractions with *not*, additional negations are not allowed.

<figure><img src="image_108.png" /></figure>

If you now apply the new *will* graph to the text, you will get 2 069 lines, including 121 contracted forms, and among them all four possibilities.

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

### 1.9.2 Affirmative and negative constructions

We can now distinguish between affirmative and negative forms. We will add the attribute *construction* with possible values *affirmative/negative* to the XML start tag `<verb>`.
<figure><img src="image_113.png" /></figure>

When you apply this graph to the text, you will obtain the same number of concordance lines, 2 069. The output is different since recognized verb groups are now annotated as affirmative or negative.

<figure><img src="image_114.png" /></figure>

### 1.9.3 Interrogative constructions

In order to recognize interrogative constructions as well it is enough to create a subgraph for personal pronouns<fnref n="60" /> and to duplicate several paths. The attribute type will now have two more possible values: *interro-negative* and *interrogative*.

<figure><img src="image_115.png" /></figure>

<footnote n="60">⁶⁰ We add to the list of common pronouns the old form of the second person singular, *thou*.</footnote>
If you apply this graph to the text you will obtain 2 224 concordance lines, some of which are represented below.

<figure><img src="image_116.png" /></figure>

## 2 Second example: numbers written with words

Our second exercise will deal with the annotation of numerals written using only words (not digits). We will proceed step-by-step using subgraphs in each of them. We will save this graph in the *Numbers* folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Numbers

### 2.1 First step: from 2 to 9

We will start with numbers *two* to *nine*. We will skip the number *one* because it is ambiguous with the pronoun *one*. We will save this graph using the name *NB2-9.grf*.

<figure><img src="image_117.png" /></figure>

When we apply this graph to our text *Dombey and Son* in *Merge* mode and without the search limitation (*Index all occurrences in text*), we obtain 565 concordance lines, some of which are displayed in the screenshot.

<figure><img src="image_118.png" /></figure>

### 2.2 Second step: from 2 to 99

#### 2.2.1 A subgraph without output

We would like to use the previous graph as a subgraph. It is possible to do it for numbers from 2 to 9; for numbers from 21 to 29 it is also possible to use the previous graph as a subgraph
but the result would contain some unwanted nesting: for instance, *twenty-two* would be annotated as `<number>twenty-<number>two</number></number>`. In order to avoid this, we will create, based on NB2-9 graph, an NB1-9-subgraph graph that produces no output and contains the number one. Note that *one* that follows other numbers written in words is no longer ambiguous. This graph annotates *twenty-two* as `<number>twenty-two</number>`.

<figure><img src="image_119.png" /></figure>

### 2.2.2 The NB2-99.grf graph

Now we can produce a graph that recognizes numerals from 2 to 99. It will have one path for the recognition of numbers *two* to *nine*: this path will consist of a box that calls the *NB2-9.grf* subgraph, which means that the annotation will also be performed by the subgraph. Another path will contain the box that recognizes numbers *ten* to *nineteen*. The box that recognizes these numbers will insert the start tag `<number>`. The third path will recognize numbers *twenty* to *ninety*, which can be followed by numbers *one* to *nine*. For this recognition, the subgraph *NB1-9-subgraph.grf* will be used. A hyphen can be used between numbers written in words. The box that will recognize numbers *twenty* to *ninety* will insert the start tag `<number>`, while the end tag `</number>` will be produced by an empty box common to the second and third paths (but not the first one). We will save this graph under the name *NB2-99.grf*.

<figure><img src="image_120.png" /></figure>

A few of the 709 concordance lines produced when applying this graph to the text are displayed in the screenshot.
<figure><img src="image_121.png" /></figure>

## 2.3 Third step: from 2 to 999

### 2.3.1 The NB1-99-subgraph.grf graph

For the same reason as before, we will produce a graph that recognizes numbers *two* to *ninety-nine* but does not insert any tag. Namely, we want to use it as a subgraph of the graph that will recognize numbers from *two* to *nine hundred ninety-nine*. We will base this graph on the already developed *NB2-99* graph and name it *NB2-99-subgraph.grf*.

<figure><img src="image_122.png" /></figure>

### 2.3.2 The NB2-999.grf graph

In this graph we will use as a subgraph *NB2-99.grf* to recognize and annotate all numbers in the interval from 2 to 99. For numbers in the interval from 100 to 199, the word *hundred* must be present and can be preceded by an indefinite article *a* or by a number *one*. For numbers in the interval 200 to 999, the word *hundred* has to be preceded by a number from 1 to 9 (possibly followed by a hyphen); for that, we will use the appropriate subgraph that does not insert tags. The previously produced *NB1-99-subgraph.grf* subgraph recognizes numbers from 1 to 99, without annotating them. Note that numbers from 1 to 99 that follow hundreds can be preceded with the conjunction *and*.
<figure><img src="image_123.png" /></figure>

The application of the main graph to the text produces 719 concordance lines.

<figure><img src="image_124.png" /></figure>

## 2.4 Fourth step: from 2 to 999 999

### 2.4.1 The NB100-999-subgraph.grf graph

We will produce a graph named NB100-999-subgraph.grf that recognizes numbers from 100 to 999 without annotating them. It presents the modification of the previously developed NB2-999.grf graph, but the path recognizing numbers from 2 to 99 has been deleted. The reason why this path was deleted will become clear when observing the next graph, which recognizes numbers from 2 to 999 999.

<figure><img src="image_125.png" /></figure>

### 2.4.2 The NB2-999999.grf graph

This graph will look similar to the NB2-999.grf graph, except that here the word thousand has to be used, optionally preceded by a or a number between 1 to 99 (NB1-99-subgraph.grf, see Section 2.3.1, page 61) or between 100 to 999 (NB100-999-subgraph.grf). The word thousand can be followed by a number between 1 to 99 (subgraph NB1-99-subgraph.grf), which can be preceded by a conjunction and. The word thousand can also be followed by a number between 100 to 999 (NB100-999-subgraph.grf), but in that case, the conjunction and is not
used and that is the reason that we separated the cases of numbers between 1 to 99 and between 100 to 999.

<figure><img src="image_126.png" /></figure>

If we apply this graph to the text, we will obtain 736 concordance lines for numbers from 2 to 999 999.

<figure><img src="image_127.png" /></figure>

<figure><img src="image_128.png" /></figure>

## 3 Third example: annotating Roman numerals

We will now continue with Roman numerals which we want to tag as *romanNumeral*. We want to describe them, as number written in words in the previous section, with graphs. However, we encounter a problem if we use graphs in a similar way to before. Namely, with three successive boxes containing the letter *l*, we can recognize three successive words *l l l* and not one word *ll*. In order to solve this problem, we use the morphological mode that allows for matching a word, that is, a sequence of letters.

### 3.1 First step: from 2 to 9

We will start with the graph that recognizes Roman numerals with values 2 to 9. As before, we will skip the numeral *l* because it is ambiguous with the pronoun *l*. We will name this graph *RN2-9.grf*. We will build this graph bearing in mind the rules for constructing Roman numerals. Once we have done that we will select all boxes, except the start box and the end box. Then we will click the *Surround box selection with morphological mode tags* button (you will find this button at the right end of the toolbar represented as violet angular brackets). The graph is now surrounded with two boxes that contain codes $< and $> inside this *morphological part*, the content of boxes is read as concatenated.

In order to obtain a tidy graph, we will align the left and right parts of the graph. This can be done by opening the *FSGraph/Format/Alignment/Horizontal/Center* menu option. This menu can be accessed by the menu bar or by a click with the right mouse button.
<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

When applying this graph to the text we obtain a concordance with the numbers of the first eight chapters of our novel *Dombey and Son* (except the first one, which we omitted on purpose).

<figure><img src="image_131.png" /></figure>

## 3.2 Second step: from 2 to 99

### 3.2.1 The subgraph from 1 to 9

As before, we will use the previously created *RN2-9.grf* graph to create a subgraph that does not insert tags. We will also cancel the morphological mode (by deleting the boxes that contain $< and $>), because it will have been introduced by the invoking graph, and nesting of morphological modes is not allowed. We will add a path that recognizes the roman numeral I, because in the context it will be used. We will name this graph *RN1-9-subgraph.grf*.

<figure><img src="image_132.png" /></figure>

### 3.2.2 The graph from 2 to 99

Roman numerals are constructed very regularly. Therefore, in order to recognize numbers greater than 10 and up to 99, we can use the preceding *RN2-9.grf* graph, and replace in it *I* with *X*, *V* with *L* and *X* with *C*. Then we have to add a path for *X*, a call to the *RN2-9.grf* subgraph outside the morphological mode (this to recognize numerals from 2 to 9), and a call to the *RN1-9-subgraph.grf* (inside the morphological mode) for numerals from 1 to 9 when they are added to numerals greater or equal to ten. We will name this graph *RN2-99.grf*.
<figure><img src="image_133.png" /></figure>

When we apply this graph to our text, we obtain 63 concordance lines. Note that this text (novel) has 64 chapters designated with Roman numerals. With the *RN2-99.grf* graph we recognize 63 of them (we do not recognize the first chapter “I”).

<figure><img src="image_134.png" /></figure>

## 3.3 Third step: from 2 to 999

### 3.3.1 The subgraph from 1 to 99

We will repeat the same procedure to produce the *RN2-9-subgraph.grf*, namely, we will cancel the morphological mode and the insertion of tags. We should not forget to replace the call to the *RN2-9.grf* subgraph with *RN1-9-subgraph.grf* because the later subgraph does not insert tags, does not impose morphological mode and recognizes *I* (this subgraph is never used independently).
<figure><img src="image_135.png" /></figure>

### 3.3.2 The graph from 2 to 999

In order to recognize Roman numerals between 2 and 999, we will again use the already developed *RN2-99.grf* graph in which we will replace X with C, L with D and C with M. Again, we will replace the call to the *RN2-9.grf* subgraph with *RN2-99.grf* outside the morphological mode (for numerals between 2 and 99), and a call to the subgraph with *RN1-99-subgraph.grf* (inside the morphological mode) for numerals between 1 to 99, when they are added to hundreds. We will name this graph *RN2-999.grf*.

<figure><img src="image_136.png" /></figure>

When we apply this graph to our text, we will obtain 76 concordance lines. But there are no Roman numerals greater than 62 (the last chapter) in this novel, so these thirteen new lines are due to false recognitions: The Roman numerals for 100 and 500 are confused:

- With the letters C and D.<fnref n="61" />

- With abbreviations, i.e. K.C.B.: *Knight Commander of the Bath* (see the [Wikipedia](https://en.wikipedia.org/wiki/Knight Commander of the Bath) ).<fnref n="62" />

<footnote n="61">⁶¹ He acquitted himself very well, nevertheless; and Miss Blimber, commending him as giving promise of getting on fast, immediately provided him with subject B; from which he passed to C, and even D before dinner.</footnote>

<footnote n="62">⁶² If that had been Joe's character, Joe might have been, by this time, Lieutenant-General Sir Joseph Bagstock, K.C.B., and might have received you in very different quarters.</footnote>
<figure><img src="image_137.png" /></figure>

Unfortunately, without linguistic analysis, the ambiguity of the language prevents us from distinguishing a letter from a Roman numeral. However, one could assume that in general (except at the end of a sentence!), a Roman numeral is not followed by a period or an apostrophe (as *D'ye want* for *Do you want*). We could add this rule to our graph, with a negative right context (see Section 1.7.3, page 55). In fact, we were lucky so far, because we could have had in the concordances, from the first graph, a confusion either with the letters V or X, possibly used in an abbreviation. Let us finish our complete recognition of all the Roman numerals and we will see later what we can do about ambiguities (Section 3.5, page 68).

## 3.4 Fourth step: from 2 to 3 999

The numbers greater than 3 999 are generally not written as Roman numerals because there are no symbols for 5 000 and 10 000.

### 3.4.1 The subgraph from 1 to 999

We will repeat the same procedure for the third time. Starting with the *RN2-999.grf* graph we will modify it by canceling the morphological mode and the insertion of tags. We should not forget to replace the call to the *RN2-99.grf* subgraph with *RN1-99-subgraph.grf* because the later subgraph does not insert tags, does not impose morphological mode and recognizes *l*.

<figure><img src="image_138.png" /></figure>

### 3.4.2 The graph from 2 to 3 999

In order to recognize thousands (one, two and three thousand) it is enough to take the graph *RN2-999.grf*, to replace in it, *C* with *M*, and to delete all other paths. As before, we will replace
a call to the subgraph with *RN2-999.grf* outside the morphological mode (for numerals between 2 to 999), and a call to the subgraph with *RN1-999-subgraph.grf* (inside the morphological mode) for numerals between 1 to 999 when they are added to thousands. We will name this graph *RN2-3999.grf*.

<figure><img src="image_139.png" /></figure>

The application of this graph to our text yields 81 matches. Five new false recognitions are added to those previously detected by the occurrence of “M” with abbreviations, i.e. *M.A..: Master of Arts* (see the [Wikipedia](https://en.wikipedia.org/wiki/Master_of_Arts)).*<sup>63</sup>

<figure><img src="image_140.png" /></figure>

The continuation of this exercise (Section 3.5) is only for confident users.

## 3.5 Ambiguities of Roman numerals (only for confident users)

Before starting this section

If you have not modified the *Norm.txt* file as explained in *Chapter 2, Section 3.1.2*, page 35, do so, or download it, then close and reopen Unitex.

As we said in *Section 3.3.2*, page 66, Roman numerals are ambiguous with letters or abbreviations; and it is almost impossible to disambiguate them. We can reduce these ambiguities by assuming that a Roman numeral is not followed by a full stop or an apostrophe. However, false annotations cannot be completely avoided, because a Roman numeral can occur at the end of a sentence and thus be followed by a period. Let us also add a list of seven common words that are ambiguous with a number written in Roman numerals: *CD* (compact

<footnote n="63"><sup>63</sup> the Reverend Alfred Feeder, M.A., who was to perform the ceremony.</footnote>
disc), *CI* (criminal investigation), *CM* (College of Medicine), *CV* (curriculum vitae), *DC* (District of Columbia), *MI* (Michigan) and *MM* (abbreviation of month or of millimeter).<fnref n="64" />

### 3.5.1 The subgraph of ambiguous Roman numerals

Let us create a graph based on the above remarks. This graph, named *ambiguousRN.grf* does not insert tags.

<figure><img src="image_141.png" /></figure>

### 3.5.2 The use of weights

If we create a graph with two subgraphs, *RN2-3999* and *ambiguousRN*, and if we use it to recognize, for instance, the sequence *CD*, we do not know if the result will be `<romanNumeral>CD</romanNumeral> (:RN2-3999 path) or *CD* (:ambiguousRN path). We would not like to annotate *CD* as a Roman numeral. The *Longest matches* option cannot help here, because both annotations are equally long. In order to get what we want, we have to use weights. We will save this graph as *nonAmbiguousRN.grf*.

One can assign a weight to a box by inserting `${weight}$` in its output where *weight* is a positive integer or zero (*weight* ≥ 0). The path with the highest last weight has the priority. The use of weights is interesting for graphs that produce output because it enables choosing an output when there are several possibilities (as in our case). One must keep in mind that weights apply only to cases when two (or more) graph paths recognize exactly the same sequence of tokens; otherwise, the *Longest matches* option reduces the range of possibilities to those paths that recognize the longest sequence. If one path has several weights, only the last one is applied. A path with a weight always has priority over a path without any weight.

<figure><img src="image_142.png" /></figure>

If you apply the *nonAmbiguousRN* graph to the text, you will obtain 81 concordance lines, but only 65 with the *romanNumeral* tags. Of these, only the seven capital letters are errors while

<footnote n="64">⁶⁴ This list is language dependent. For example, for French, DIX is the number ten, and so on.</footnote>
the other tags correspond to 58 chapters (the first is not counted, but also the fifth, tenth and
fiftieth, due to the period after the Roman numeral).

<figure><img src="image_143.png" /></figure>
Before starting this chapter

1. Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted`

Create three new folders named (without space): JubileePlainText and JubileeXmlText.

Go to the folder:

`MyUnitex\English\Corpus\UnitexGettingStarted\UnitexGettingStarted\JubileePlainText`

Download the `jubileePlainText.txt` file.

Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\JubileeXmlText`

Download the `JubileeXmlText.xml` file.<fnref n="65" />

2. Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted`

Create two new folders named `Entities` and `Measures`.

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted\Entities`

Create two new folders named `Analysis` and `Synthesis`.

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted\Measures`

Create two new folders named `Analysis` and `Synthesis`.

3. Go to the folder: `MyUnitex\English\CasSys`

Create a new folder named `UnitexGettingStarted` (without space).

Go to the folder: `MyUnitex\English\CasSys\UnitexGettingStarted`

Create three new folders named `PlainTextEntities`, `XmlTextEntities` and `Measures`.

# 1 First example: named entity recognition in a plain text

We will introduce cascades that analyze texts with aim to annotate named entities in a text (dates and places). First, we will use the text `jubileePlainText.txt`.

This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on June 2, 2022 and end on June 5. The Queen ascended the British throne on February 6, 1952.

Our small town of Nearlondon will also celebrate this event. Nearlondon has always been a loyal subject of Her Majesty. Although the city of London is dear to us, on Thursday June 2 we will change the name of London Street to Queen Elizabeth Street.

Everyone will fondly remember 2022!

God Save the Queen!

We will open this text in Unitex and answer Yes to the question *Do you want to preprocess the text*. A window will open in which we will remove the first two checkmarks, leaving just the last one, *Apply All default Dictionaries*.

## 1.1 Analysis cascade

All graphs in the Section 1.1 will be saved in the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Analysis`

<footnote n="65">⁶⁵ These two texts were prepared specifically to illustrate this chapter.</footnote>
1.1.1 The absoluteDate.grf graph

Let us create a first graph, which will recognize dates where the year is specified; we will call
such dates absolute. In order to simplify this graph, we will not describe the days of months
and years in detail but will instead use the lexical mask <NB>, which recognizes strings
consisting of digits. We will put the recognized date between symbols { and . AbsoluteDate}.
By doing so, we are producing a lexical tag.<fnref n="66" /> Assuming that a date in a text is June 23, 2022,
the graph we are building will recognize it and replace it with a lexical tag {June 23\,
2022, .AbsoluteDate}. This text segment does not consist of eight tokens anymore, but just
one, June 23, 2022 and this token is classified as AbsoluteDate; in other words, the part-of-
speech field in the lexical tag contains AbsoluteDate, in the same way as June was classified
before as N (noun) by the application of dictionaries.<fnref n="67" />

In this graph, we will duplicate the box containing the names of months in order to avoid
opening braces twice in the same path. We will add a tracing feature to the recognized
sequence: the name of the graph itself +grf=absoluteDate, which can be useful in the
discovering process. We will save this graph as absoluteDate.grf. In order to use the graph in a
cascade (see Section 1.1.3, page 73), the saved graph must be compiled by clicking the
Compile graph button at the left corner of the graph processing toolbar, next to the Save graph
button.

<figure><img src="image_144.png" /></figure>

We will create a concordance with this graph. After clicking the *SEARCH* and *OK* buttons, the *Located sequences* window appears. At the bottom right of this window, we will open the *Sort according to* drop-down menu and choose the *Text order* option, instead of *Center, Left*. We will click on the *Build concordance* button and the concordance lines, in this case two lines, will be displayed in the order in which matched sequences appear in the text.

<footnote n="66">⁶⁶ See Chapter 2, Section 1.2.5, page 15. The common way to insert a `{` character is by the box that recognizes the beginning of the phrase you want to tag. If the `{` character is inserted just before, by a separate `<E>` box, the inserted `{` character can be followed by a space in the output, which is unsatisfactory because the braces are supposed to delimit a precise phrase.</footnote>

<footnote n="67">⁶⁷ Remember that each alphabetic string is one token (*June* in this case), a punctuation one token (, in this case), and each digit is a separate token (2, 3, 2, 0, 2, 2 in this case).</footnote>
<figure><img src="image_145.png" /></figure>

### 1.1.2 The `relativeDate.grf` Graph

We will now create the graph that recognizes relative, or incomplete, dates, when the year is not mentioned. We will use the previously constructed `absoluteDate.grf` graph, remove it from the boxes for year and separating comma and save it under the new name `relativeDate.grf` (we will use the `FSGraph/Save as...` menu). Do not forget to modify the output as well: `..AbsoluteDate+grf=absoluteDate}` should be replaced by `..RelativeDate+grf=relativeDate}`. After compiling the graph, we will apply it to the text.

<figure><img src="image_146.png" /></figure>

Now we will obtain four concordance lines in which all dates will be annotated as relative dates, including two recognized by the previous graph (note that in these cases, the years are outside the scope of the lexical tags). In order to correct this, we will apply these two graphs one after the other, using what is known as a cascade of graphs.

<figure><img src="image_147.png" /></figure>

### 1.1.3 The analysis.csc cascade

Now we will build a small cascade containing both graphs. We will open the `Text/Apply CasSys Cascade...` menu and click the `New` button. The Cascade configuration frame will open. Then we must find the location of the graphs that we want to put into the cascade, in this case:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Analysis
We will notice that the left window shows only files with the `.fst2` extension that is only graphs that have been compiled. It means that we can put into a cascade only already compiled graphs.<fnref n="68" />

With the mouse, we can drag and drop the `absoluteDate.fst2` graph into the right window; and we will repeat the same with the `relativeDate.fst2` graph. Alternatively, we can select a graph and click the **Add** button when we want to add it to a cascade. We will save this cascade as `analysis.csc` in the folder:

MyUnitex\English\CasSys\UnitexGettingStarted\PlainTextEntities

by using the **Save** button.

<figure><img src="image_148.png" /></figure>

We will close this frame by clicking the **Close** button.

Now we will select our cascade and click the **Launch** button, which will initiate the analysis. When analysis is over, we can build concordances. We will obtain four concordance lines, but now the dates will be correctly annotated. Why is it so? Because the first graph in the cascade, `absoluteDate.grf`, has annotated all absolute dates and produced lexical tags. As we explained before, a lexical tag is treated as one token, and the subsequently applied graph `relativeDate.grf` cannot analyze inside its content.

<figure><img src="image_149.png" /></figure>

### 1.1.4 The town.grf graph

We will now prepare the graph that will recognize names of towns and cities and annotate them as *Town*.<fnref n="69" /> We will presume that any sequence of letters of the English alphabet

<footnote n="68">⁶⁸ If we use subgraphs, only the main graph needs to be compiled, not the subgraphs.</footnote>

<footnote n="69">⁶⁹ One has to be careful when choosing the annotation tags in order to avoid ambiguity with dictionary lemmas. For instance, if we had chosen *town* instead of *Town* as a tag for inhabited places, then the lexical mask `<town>` would correspond to recognized names of towns, but also the lemma *town*, that is, its forms *town* and *towns*. On the contrary, the lexical mask `<Town>` corresponds only to the annotated names of towns (remember that lexical masks are case sensitive).</footnote>
`<LETTER>` starting with an uppercase letter (`<UPPER>`) and following the sequences `city of` or `town of` is the name we want to annotate. In order to recognize such a sequence, we must use the morphological mode in order to describe the structure of a single word.<fnref n="70" /> We will save this graph as `town.grf` and compile it.

<figure><img src="image_150.png" /></figure>

We will drag and drop this graph to the last position in the cascade. As a reminder: open the *Text/Apply CasSys Cascade...* menu, then, find the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Analysis`

select the *analysis.csc* cascade, click the *Edit* button, move to the folder containing the graphs, then drag and drop the `town.fst2` graph to the list of graphs. Save the cascade.

<figure><img src="image_151.png" /></figure>

After launching this cascade two new concordance lines will be produced by the `town.grf` graph with *Nearlondon* and *London* annotated as *Town*.

<figure><img src="image_152.png" /></figure>

### 1.1.5 Generalization graphs

If you look at the short text we are annotating, you will notice that *Nearlondon* in the sequence *Nearlondon has always been...* was not annotated as *Town* because it was preceded neither with *town of* nor *city of*. Moreover, *Nearlondon* is not even in the dictionary of the text, it is an unknown word.

In order to solve this problem we will use a generalization graph which will annotate a word (or a sequence of words) even when it does not appear in a required context, if the same word appeared somewhere else in the same text in the required context, and was annotated by some graph in a cascade. We have such a situation with the word *Nearlondon* in our text: it was annotated once as *Town* by the `town.grf` graph, because it occurred in the required context, so its second occurrence will be annotated as *Town* by the `townGeneralization.grf` graph.

<figure><img src="image_153.png" /></figure>

In order to create a simple generalization graph it is enough to introduce a box with the output consisting of the annotation that has to be generalized (in our case, `.Town`), select this box and

<footnote n="70">⁷⁰ See *Chapter 3, Section 3.1*, page 63. In CasSys cascades, the `<FIRST>` or `<UPPER>` lexical mask does not always detect if a word has an uppercase initial letter, because some words or expressions may have been replaced by lexical tags, as in the preceding examples. This is why we show another way of detecting words with an uppercase initial.</footnote>
click to the *Insert generic graph mark before the selected box* button at the far right end of the toolbar.<fnref n="71" />

<figure><img src="image_154.png" /></figure>

After compiling this graph, we will add it as the last one in the list of graphs in the *analysis.csc* cascade (remember that it has to be in the list after the *town.grf* graph) in the same way as we have done before.<fnref n="72" /> We must do one more thing: the configuration panel offers options for the modes of graph application. By default, this option is *Merge* mode, which suited us so far for all the graphs in our list. However, for this last graph we have also to check the *Generaliz...* box in order to signal that this is a generalization graph.

<figure><img src="image_155.png" /></figure>

After launching our cascade, we will obtain a concordance with two new lines, one of them with *Nearlondon* annotated although it was preceded neither with *city of* nor *town of*. We can see that the second occurrence of *London* was annotated as well, although we did not want that, since the context is *change the name of London Street to Queen Elizabeth Street*.

<figure><img src="image_156.png" /></figure>

### 1.1.6 The street.grf graph

We will now create a graph that recognizes names of streets with the *Address* category. We will recognize two patterns before the words *street* and *avenue*: (1) a date or a town name already recognized by the cascade and tagged as `<AbsoluteDate>`, `<RelativeDate>` or `<Town>`,<fnref n="73" /> (2) one or more upper-case words. In order to make the loop for uppercase words, we can duplicate two boxes (the big angular bracket and the lexical mask `<UPPER>`), but without the opening braces as output. Alternatively, we can add the new `<UPPER>` box, surround it with morphological mode angular brackets by clicking the *Surround box selection with morphological mode tags* button, and then delete the unnecessary closing bracket and connect the box with `<LETTER>`.

<footnote n="71">⁷¹ Generalization graphs for annotation can be more sophisticated; see the Section 4.5.2, page 104.</footnote>

<footnote n="72">⁷² The *townGeneralization.grf* file must be compiled before we place it in the cascade, but the program uses also the *.grf* file which must therefore also be present in the folder, whereas this is not obligatory for other graphs.</footnote>

<footnote n="73">⁷³ This enables recognition of *May 8, 1945 Avenue*.</footnote>
<figure><img src="image_157.png" /></figure>

After saving the graph with the name `street.grf` we will compile it and add it as the last graph in our `analysis.csc` cascade to be used in the *Merge* mode. After launching the cascade and producing a concordance, we will obtain nine concordance lines. One new line contains the match *Queen Elizabeth Street* recognized by the second rule in the `street.grf` graph (two upper-case words). The first rule is applied to *London Street* since *London* was already tagged as *Town*. If we look at that line in the concordance list (it is the second-to-last line in the *Text Order* option), we will see that there we have nested tagging, that is a lexical tag inside another lexical tag.<fnref n="74" />

<figure><img src="image_158.png" /></figure>

### 1.1.7 Created files

The role of the Cassys program is not only to produce concordances; it produces as output numerous files. If we open the folder:

`MyUnitex\English\Corpus\UnitexGettingStarted\jubileePlaintext`

we will see in it, besides the original `jubileePlaintext.txt` file, also `jubileePlaintext.snt` and the `jubileePlaintext_snt` folder, as for all texts processed by Unitex. Besides this, we will see four more files with the suffix `_csc` and one new folder.

<table><thead><tr><th>Nom</th><th>Modifié le</th><th>Type</th><th>Taille</th></tr></thead><tbody><tr><td>jubileePlaintext_cscraw</td><td>22/03/2023 09:53</td><td>Fichier RAW</td><td>1 Ko</td></tr><tr><td>jubileePlaintext_csc.txt</td><td>22/03/2023 09:53</td><td>Fichier TXT</td><td>2 Ko</td></tr><tr><td>jubileePlaintext_csc_raw_offset.txt</td><td>22/03/2023 09:53</td><td>Fichier TXT</td><td>1 Ko</td></tr><tr><td>jubileePlaintext_csc_txt_offset.txt</td><td>22/03/2023 09:53</td><td>Fichier TXT</td><td>1 Ko</td></tr><tr><td>jubileePlaintext.snt</td><td>20/03/2023 16:12</td><td>Fichier SNT</td><td>1 Ko</td></tr><tr><td>jubileePlaintext.txt</td><td>26/10/2022 09:19</td><td>Fichier TXT</td><td>1 Ko</td></tr><tr><td>jubileePlaintext_snt</td><td>22/03/2023 09:53</td><td>Dossier de fichiers</td><td></td></tr><tr><td>jubileePlaintext_csc</td><td>22/03/2023 09:53</td><td>Dossier de fichiers</td><td></td></tr></tbody></table>

#### 1.1.7.1 The _csc files

The `jubileePlaintext_cscraw` file contains the text modified by the cascade. Namely, all sequences recognized by cascade graphs are replaced by lexical tags enclosed in braces.

<footnote n="74">⁷⁴ In order to do that the delimiting characters of the internal lexical tag (`{.+}`) are protected by the backslash character.</footnote>
This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on {June 2, 2022, AbsoluteDate+grf=absoluteDate} and end on {June 5, RelativeDate+grf=relativeDate}. The Queen ascended the British throne on {February 6, 1952, AbsoluteDate+grf=absoluteDate}.

Our small town of {Nearlondon, Town+grf=town} will also celebrate this event. {Nearlondon, Town+grf=townGeneralization} has always been a loyal subject of Her Majesty. Although the city of {London, Town+grf=town} is dear to us, on {Thursday June 2, RelativeDate+grf=relativeDate} we will change the name of {\{London, \.Town\+grf=townGeneralization\} Street, Address+grf=street} to {Queen Elizabeth Street, Address+grf=street}.

Everyone will fondly remember 2022!

God Save the Queen!

The jubilee PlainText_csc.txt file is a quasi XML version of the same output in which the braces are replaced by XML tags `<csc></csc>`, the recognized text is surrounded by tags `<form></form>`, while the code in the POS field and its additional features appear in tags `<code></code>`.<fnref n="75" />

This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on <csc><form>June 2, 2022</form><code>AbsoluteDate</code><code>grf=absoluteDate</code></csc> and end on <csc><form>June 5</form><code>RelativeDate</code><code>grf=relativeDate</code></csc>.

The Queen ascended the British throne on <csc><form>February 6, 1952</form><code>AbsoluteDate</code><code>grf=absoluteDate</code></csc>.

Our small town of <csc><form>Nearlondon</form><code>Town</code><code>grf=town</code></csc> will also celebrate this event.

<csc><form>Nearlondon</form><code>Town</code><code>grf=townGeneralization</code></csc> has always been a loyal subject of Her Majesty. Although the city of

<csc><form>London</form><code>Town</code><code>grf=town</code></csc> is dear to us, on <csc><form>Thursday June 2</form><code>RelativeDate</code><code>grf=relativeDate</code></csc> we will change the name of

<csc><form><csc><form>London</form><code>Town</code><code>grf=townGeneralization</code></csc> Street</form><code>Address</code><code>grf=street</code></csc> to <csc><form>Queen Elizabeth Street</form><code>Address</code><code>grf=street</code></csc>.

Everyone will fondly remember 2022!

God Save the Queen!

The remaining two files contain offsets corresponding to two previously described files.<fnref n="76" />

### 1.1.7.2 The _csc folder

The folder with the name ending with _csc contains all intermediary files, which are very useful in debugging cascades.

<footnote n="75">⁷⁵ `<lemma></lemma>` is one more pair of XML tag that can appear in the content of `<csc></csc>`. It does not appear in our case because in the description of all lexical tags the lemma field (between the comma and the period) is empty. If it were not the case, this field would be copied here enclosed in `<lemma></lemma>` tags.</footnote>

<footnote n="76">⁷⁶ See the Unitex user manual, Section 15.13.10.</footnote>
<figure><img src="image_159.png" /></figure>

The original text is copied in this folder and renamed as jubileePlainText_0_0.txt. The result of the first graph in the cascade is given the name jubileePlainText_1_0.txt, and so on. If some graph in the cascade is used in the until fix point mode (this will be explained in the following sections) it means that it will be repeatedly used and the second number in a file's name tells which iteration produced the file (in our present example it is always 0).

## 1.2 Synthesis cascade

We have seen in the previous section that there are two outputs of an analysis cascade.

### 1. The_cscraw

In the _cscraw version of the text the recognized matches are transformed into lexical tags that can be subsequently used in Unitex for further text exploration. It can be used in Unitex for searching in the usual way. We can open the text jubileePlainText_cscraw, without preprocessing, and then launch with Locate pattern a search using classical as well as new categories.<fnref n="77" /> For instance, a pattern <Address> retrieves two matches:

<figure><img src="image_160.png" /></figure>

### 2. The_csc.txt

The_csc.txt version of the text is the resulting text where the lexical tags are replaced by XML tags. This format is used to transform the output of an analysis cascade into whatever other format needed.

We will now open (with the Text/Open menu) the file:

MyUnitex\MyUnitex\English\Corpus\UnitexGettingStarted\jubileePlainText\jubileePlainText_csc.txt

and reply No to the question Do you want to preprocess the text.

We will delete tags <csc>, <form> and so on, and replace them with tags that we have chosen, namely, <AbsoluteDate></AbsoluteDate>, <RelativeDate></RelativeDate>, <Town></Town> and <Address></Address>.

#### 1.2.1 The tag.grf graph

Let us create the first graph, which we will save in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Synthesis

<footnote n="77">⁷⁷ That is to say <N>, <V> or <ADV>, but also, <AbsoluteDate>, <Town> or <Address>.</footnote>
In this folder we will save the rest of the graphs for synthesis that we will introduce in this
section. The aim of this graph that we will use in *Replace* mode is:

1. delete tags *csc*, *form* and *code*;

2. save the matched text in a variable `$text$` and its category in a variable `$tag$`;

3. replace all that with the tagged text: `<$tag$>$text$</$tag$>`; that is, everything recognized is replaced (thus the graph must be used in the *Replace* mode) by the recognized entity (`$text$`) enclosed in appropriate tags (`<$tag$></$tag$>`).

<figure><img src="image_161.png" /></figure>

An input variable in Unitex stores the part of a text recognized by boxes inside (big red) parentheses. These parentheses are placed in the same way as context and morphological mode brackets by clicking on the *Surround box selection with an input variable* button (red parentheses at the right-hand side of the toolbar).

The user can freely choose the name of the variable. If the value of a variable is used in output, then it is surrounded by a pair of $ character.

### 1.2.1.1 A first attempt

In order to collect the text we will use a loop that recognizes a continuous sequence of tokens (lexical mask: `<TOKEN>`) except that it will exclude `</form>` (the tag at the end of the recognized text).<fnref n="78" /> After the loop, the graph will match one of the used categories, followed by everything until the end of the coded sequence that we are analyzing (`</csc>`).

A very important note: in Unitex queries, angle brackets have the special meaning, so if we want to recognize an angle bracket in a text, we have to *protect* it by preceding it with a backslash.<fnref n="79" /> For instance, if we want to recognize `<csc>` in a text we have to write `\<csc>` (otherwise it would be interpreted as a lexical mask recognizing all forms of the lemma *csc*). Similarly, since a slash also has a special meaning in Unitex graph boxes (it marks the beginning of an output) if we want to recognize it in a text, it also has to be preceded by a backslash. For instance, in order to recognize `</csc>` we must write `\<\csc>`.

In order to make this complex graph easier to understand, we will provide an example inside a comment box (a box starting with a slash and not connected with any other boxes). Text in such a box will appear in green.

<footnote n="78">⁷⁸ The negative right contexts (see Chapter 3, Section 1.7.3, page 55) are interpreted like this: `<TOKEN>` is recognized if it is not the < character, followed by /form> (first negative right context), respectively /csc> (second negative right context).</footnote>

<footnote n="79">⁷⁹ See the Unitex user manual, Section 5.2.7. We do not have to take care about this in the output text - no protection is needed there.</footnote>
<figure><img src="image_162.png" /></figure>

We will now save the graph as `tag.grf`; compile it; create the new cascade in which we will add in the first position this graph; and choose the *Replace* mode by checking the appropriate box. We will call this cascade `synthesis.csc` and we will save it in the same folder as the `analysis.csc` cascade.

<figure><img src="image_163.png" /></figure>

After launching this cascade, we will obtain a new text file:<fnref n="80" />

MyUnitex\MyUnitex\English\Corpus\UnitexGettingStarted\jubileePlainText\jubileePlainText_csc_csc.txt

This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on `<AbsoluteDate>June 2, 2022</AbsoluteDate>` and end on `<RelativeDate>June 5</RelativeDate>`. The Queen ascended the British throne on `<AbsoluteDate>February 6, 1952</AbsoluteDate>`.

Our small town of `<Town>Nearlondon</Town>` will also celebrate this event. `<Town>Nearlondon</Town>` has always been a loyal subject of Her Majesty. Although the city of `<Town>London</Town>` is dear to us, on `<RelativeDate>Thursday June 2</RelativeDate>` we will change the name of `<Town><csc><form>London</Town>` Street</form><code>Address</code><code>grf=street</code></csc> to <Address>Queen Elizabeth Street</Address>.

Everyone will fondly remember 2022!

God Save the Queen!

We can observe in the resulting text that the transformation of annotations of non-nested elements was done successfully. For instance, the occurrence of the first date in the text became `<AbsoluteDate>June 2, 2022</AbsoluteDate>`. However, for nested elements, the result is surprising: `<Town><csc><form>London</Town> Street</form><code>Address</code><code>grf=street</code></csc>`! What happened is that the graph generated the opening tag using the internal markup `<Town>`.

<footnote n="80">⁸⁰ Because the graph in this cascade is used in the *Replace* mode, it is difficult to visualize the output as a concordance. We will therefore from now on display the file obtained as a result, which is in this case *jubileePlainText_csc_csc.txt*.</footnote>
### 1.2.1.2 A second attempt

To avoid this, we will add `<csc>` (the beginning of a nested coded sequence) in the first negative context box in order to avoid deletion and replacement in cases when an entity has nested entities in its content.

<figure><img src="image_164.png" /></figure>

We will compile the graph and relaunch the cascade on jubileePlainText_csc.snt.

This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on `<AbsoluteDate>June 2, 2022</AbsoluteDate>` and end on `<RelativeDate>June 5</RelativeDate>`. The Queen ascended the British throne on `<AbsoluteDate>February 6, 1952</AbsoluteDate>`.

Our small town of `<Town>Nearlondon</Town>` will also celebrate this event. `<Town>Nearlondon</Town>` has always been a loyal subject of Her Majesty. Although the city of `<Town>London</Town>` is dear to us, on `<RelativeDate>Thursday June 2</RelativeDate>` we will change the name of `<csc><form><Town>London</Town>`

`Street</form><code>Address</code><code>grf=street</code></csc>` to `<Address>Queen Elizabeth Street</Address>`.

Everyone will fondly remember 2022!

God Save the Queen!

We can now observe in the resulting text that, this time, the nested element was well transformed: `<csc><form><Town>London</Town> Street</form><code>Address</code><code>grf=street</code></csc>`.

### 1.2.1.3 The Until fix point option

In order to transform the annotations of the external element it would be enough to launch the same graph for the second time (add it to the cascade). However, if we had a three-level nesting, then we would have to launch (and add to the cascade) the graph a third time. In order to avoid this, we will check the Until Fix Point box that will launch the tag.fst2 graph as long as it makes some changes to the text.

<figure><img src="image_165.png" /></figure>

After launching again this cascade (applying it to the text jubileePlainText_csc.txt) the resulting text jubileePlainText_csc_csc.txt becomes correct, and all annotations have been correctly transformed: `<Address><Town>London</Town> Street</Address>`.
This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on `<AbsoluteDate>June 2, 2022</AbsoluteDate>` and end on `<RelativeDate>June 5</RelativeDate>`. The Queen ascended the British throne on `<AbsoluteDate>February 6, 1952</AbsoluteDate>`.

Our small town of `<Town>Nearlondon</Town>` will also celebrate this event. `<Town>Nearlondon</Town>` has always been a loyal subject of Her Majesty. Although the city of `<Town>London</Town>` is dear to us, on `<RelativeDate>Thursday June 2</RelativeDate>` we will change the name of `<Address><Town>London</Town> Street</Address>` to `<Address>Queen Elizabeth Street</Address>`.

Everyone will fondly remember 2022!

God Save the Queen!

If we open the `JubileePlaintext_csc_csc` folder, we will find in it four files (and four corresponding folders): the original text `jubileePlaintext_csc_0_0.snt`, the result of the first run of the graph (resolves outer-level and one-level annotations), `jubileePlaintext_csc_1_0.snt`, the result of the second run of the graph (resolves second-level annotations), `jubileePlaintext_csc_1_1.snt`, the result of the third run of the graph, `jubileePlaintext_csc_1_2.snt`, that establishes that the text was not modified (the last two files are identical).

Nom

<figure><img src="image_166.png" /></figure>

jubileePlaintext_csc_1_0.snt

<figure><img src="image_167.png" /></figure>

jubileePlaintext_csc_1_1.snt

<figure><img src="image_168.png" /></figure>

jubileePlaintext_csc_1_2.snt

<figure><img src="image_169.png" /></figure>

jubileePlaintext_csc_0_0.snt

<figure><img src="image_170.png" /></figure>

jubileePlaintext_csc_1_0.snt

<figure><img src="image_171.png" /></figure>

jubileePlaintext_csc_1_1.snt

<figure><img src="image_172.png" /></figure>

jubileePlaintext_csc_1_2.snt

<figure><img src="image_173.png" /></figure>

jubileePlaintext_csc_0_0.snt

Modifié le

24/03/2023 13:12

24/03/2023 13:12

24/03/2023 13:12

24/03/2023 13:12

24/03/2023 13:12

24/03/2023 13:12

24/03/2023 13:12

24/03/2023 13:12

Type

Fichier SNT

Fichier SNT

Fichier SNT

Fichier SNT

Dossier de fichiers

Dossier de fichiers

Dossier de fichiers

Dossier de fichiers

Taille

1 Ko

1 Ko

1 Ko

2 Ko

### 1.2.2 The internalDeletion.grf graph

In some cases, we do not want nested annotations. For instance, annotating a date or a city occurring in an address is not obligatorily relevant. While recognizing dates and inhabited places has been essential for recognizing addresses, we would like to remove the date and town tags now that we have used them.

The `internalDeletion.grf` graph that will remove nested tags in the `<Address>` element begins with the recognition of the start tag `<Address>`, which is followed by the recognition of a date tag or a city tag; the content of this second element (start tag and end tag excluded) becomes the value of the variable `$inside$`. The end tag `</Address>` is then searched for and everything found between the end tag of the nested annotation and the end tag `</Address>` is remembered in the variable `$outside$`. The output of this graph, which works in the `Replace` mode, are tags `<Address></Address>` surrounding the content of the nested element (`$inside$`) followed by the content of the `<Address>` element itself (`$outside$`).
<figure><img src="image_174.png" /></figure>

After saving and compiling the graph, we will add it as the second graph in the synthesis.csc cascade (do not forget to check the Replace box).

<figure><img src="image_175.png" /></figure>

After launching this cascade, we will get the desired result.

This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on `<AbsoluteDate>June 2, 2022</AbsoluteDate>` and end on `<RelativeDate>June 5</RelativeDate>`. The Queen ascended the British throne on `<AbsoluteDate>February 6, 1952</AbsoluteDate>`.

Our small town of `<Town>Nearlondon</Town>` will also celebrate this event. `<Town>Nearlondon</Town>` has always been a loyal subject of Her Majesty. Although the city of `<Town>London</Town>` is dear to us, on `<RelativeDate>Thursday June 2</RelativeDate>` we will change the name of `<Address>London Street</Address>` to `<Address>Queen Elizabeth Street</Address>`.

Everyone will fondly remember 2022!

God Save the Queen!

## 2 Second example: named entity recognition in an XML text

We will work now with the same text but prepared in the XML format, jubileeXmlText.xml. This text is in the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\jubileeXmlText

Please note that despite being an XML text it should not have the `.xml` extension because Unitex automatically removes in preprocessing all XML tags from a text having an `.xml` extension. In order to avoid this, we will rename our text to `jubileeXmlText.xml.txt`.
```xml
<xml>
  <title>The municipal newspaper of the city of Nearlondon</title>
  <date>Posted on Wednesday June 1, 2022</date>
  <body>
    <p>This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on June 2, 2022 and end on June 5. The Queen ascended the British throne on February 6, 1952.</p>
    <p>Our small town of Nearlondon will also celebrate this event. Nearlondon has always been a loyal subject of Her Majesty. Although the city of London is dear to us, on Thursday June 2 we will change the name of London Street to Queen Elizabeth Street.</p>
    <p>Everyone will fondly remember 2022!</p>
    <p>God Save the Queen!</p>
  </body>
</xml>
```

We will open the text jubileeXmlText.xml.txt in Unitex (Text/Open) and answer Yes to the question Do you want to preprocess the text. A window will open in which we will remove the first two checkmarks, leaving just the last one Apply All default Dictionaries. We will launch the analysis.csc cascade (Text/Apply CasSys Cascade), then open the resulting text, jubileeXmlText.xml_csc.txt, and launch the synthesis.csc cascade. We will get as a result the jubileeXmlText.xml_csc_csc.txt file. Let us look at it:

```xml
<xml>
  <title>The municipal newspaper of the city of <Town>Nearlondon</Town></title>
  <date>Posted on <AbsoluteDate>Wednesday June 1, 2022</AbsoluteDate></date>
  <body>
    <p>This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on <AbsoluteDate>June 2, 2022</AbsoluteDate> and end on <RelativeDate>June 5</RelativeDate>. The Queen ascended the British throne on <AbsoluteDate>February 6, 1952</AbsoluteDate>.</p>
    <p>Our small town of <Town>Nearlondon</Town> will also celebrate this event. <Town>Nearlondon</Town> has always been a loyal subject of Her Majesty. Although the city of <Town>Nearlondon</Town> is dear to us, on <RelativeDate>Thursday June 2</RelativeDate> we will change the name of <Address>London Street</Address> to <Address>Queen Elizabeth Street</Address>.</p>
    <p>Everyone will fondly remember 2022!</p>
    <p>God Save the Queen!</p>
  </body>
</xml>
```

The result of annotating the element <body> was the same as before. However, we see that Nearlondon and Wednesday June 1, 2022 appearing in the header were annotated as well although they are not part of the text. We would like to avoid this, because we do not want to annotate the header.

We will use the cascades developed before to which we will add some new graphs. We will copy two cascades (analysis.csc and synthesis.csc) from the folder:

MyUnitex\English\CasSys\UnitexGettingStarted\PlainTextEntities

and paste them in the folder:

MyUnitex\English\CasSys\UnitexGettingStarted\XmlTextEntities
## 2.1 Analysis cascade

We will open the text jubileeXmlText.xml.txt in Unitex (Text/Open) and answer No to the question Do you want to preprocess the text.

We will build two new graphs, saved in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Analysis

### 2.1.1 The toolXml.grf graph

We will assume that the name in an XML tag is a sequence of letters that can be recognized in Unitex with the lexical mask `<WORD>`.<fnref n="81" /> The toolXml.grf graph recognizes XML tags, transforms them into brace-enclosed lexical tags, with *Xml* in the part-of-speech (POS) field and with as features the name of the recognized tag and the indication whether it was an opening or closing tag.<fnref n="82" />

<figure><img src="image_176.png" /></figure>

### 2.1.2 The toolHidden.grf graph

The role of the *toolHidden.grf* graph is to hide and protect the XML elements that can be found in the text header.<fnref n="83" /> By doing this, we disable the annotation of entities in the content of hidden elements. This graph transforms the recognized content, together with the opening and closing tags, into a lexical tag categorized as *Hidden* in the POS field.

<figure><img src="image_177.png" /></figure>

We will put these two new graphs at the beginning of the *analysis.csc* cascade, which will now consist of seven graphs. These two graphs will work in the *Merge* mode like the rest of the analysis graphs. Do not forget to apply the *analysis.csc* cascade to the original *jubileeXmlText.xml.snt* file.

<footnote n="81">⁸¹ Of course, XML names can be more complex, but we want to keep our example simple. It would be possible to call in this box a subgraph that describes possible names more precisely.</footnote>

<footnote n="82">⁸² This graph is designed for the simple XML document that we are using in our examples. A general graph that would recognize all possible XML tags, including their attributes, would be more complex.</footnote>

<footnote n="83">⁸³ This graph only works if the text to hide is not too long. Otherwise, see Section 4.4.1, page 98 (for confident users).</footnote>
<figure><img src="image_178.png" /></figure>

The result of the first graph in the cascade (`toolXml.grf`), `jubileeXmlText.xml_1_0.snt` file, is recorded in the folder:

`MyUnitex\English\Corpus\UnitexGettingStarted\JubileeXmlText\jubileeXmlText.xml_csc`

We can see that all XML tags in the text have been transformed into lexical tags with `Xml` in the POS field (only the first line is displayed here).

`{<xml>,.Xml+name=xml+openTag+grf=toolXml}`

The result of the second graph is recorded in the `jubileeXmlText.xml_2_0.snt` file. We can see that two tag-enclosed XML elements, `<title></title>` and `<date></date>`, which appear in the header, have been transformed into lexical tags *Hidden* (only the second and third lines are displayed here). Note that in order to preserve the literal meaning of the special characters (period, comma, parentheses and so on), they are protected by an escape character backslash when they appear inside the first field of a lexical tag. These escape characters are inserted before the special characters only if the application of the graph has been launched as an item in a cascade.

`{\{<title>\,\.Xml\+name(title\+openTag\+grf=toolXml\}\}The municipal newspaper of the city of Nearlondon\{</title>\,\.Xml\+name(title\+closeTag\+grf=toolXml\},.Hidden+grf=toolHidden}\{\{<date>\,\.Xml\+name(date\+openTag\+grf=toolXml\}\}Posted on Wednesday June 1\, 2022\{</date>\,\.Xml\+name(date\+closeTag\+grf=toolXml\},.Hidden+grf=toolHidden}\}`

## 2.2 Synthesis cascade

Now, we will open the text `jubileeXmlText.xml_csc.txt` in Unitex (`Text/Open`) and answer `No` to the question *Do you want to preprocess the text*.

We will open (`Graphs/Open` menu or `Graphs/Open Recent` menu) the existing `tag.grf` graph that lists all annotation categories that must remain (the output is `<$tag$>$text$</$tag>`). We will modify this graph by adding a path where we list the annotation categories to be removed (`Xml` and `Hidden`); the output is `$text$`.
<figure><img src="image_179.png" /></figure>

We will save the *tag.grf* graph, compile it and launch the synthesis.csc cascade. The resulting jubileeXmlText.xml_csc_csc.txt file will contain exactly what we expected.

```xml
<xml>
  <title>The municipal newspaper of the city of Nearlondon</title>
  <date>Posted on Wednesday June 1, 2022</date>
  <body>
    <p>This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on <AbsoluteDate>June 2, 2022</AbsoluteDate> and end on <RelativeDate>June 5</RelativeDate>. The Queen ascended the British throne on <AbsoluteDate>February 6, 1952</AbsoluteDate>.</p>
    <p>Our small town of <Town>Nearlondon</Town> will also celebrate this event. <Town>Nearlondon</Town> has always been a loyal subject of Her Majesty. Although the city of <Town>London</Town> is dear to us, on <RelativeDate>Thursday June 2</RelativeDate> we will change the name of <Address>London Street</Address> to <Address>Queen Elizabeth Street</Address>.</p>
    <p>Everyone will fondly remember 2022!</p>
    <p>God Save the Queen!</p>
  </body>
</xml>
```

## 3 Third example: measure recognition

We will now go back to the text *DombeyAndSon* in the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon

with the idea of recognizing measures in it. We will do that in two phases: first, we will recognize numbers and then units. In order to keep things simple, we will produce graphs that recognize only units that occur in the novel: currencies (pence, sixpence, shilling and pound), duration (minute, hour, day, week, month and year), lengths (mile, foot and yard), speed (mile an hour), temperature (degree) and weight (ton).
*Sixpence* is the name of an old coin. We will check its presence in the Unitex distribution dictionary. We will open the *Dela/Lookup...* menu, select the *dela-en-public.bin* dictionary and type *sixpence* in the *Word* field. Unitex displays:

`sixpence, N:s:p`

Unitex considers *sixpence* as both singular and plural, but Charles Dickens uses a plural form with an s. Sure enough, if we use the *Dela/Lookup...* menu again for the form *sixpences*, we will not find it.<fnref n="84" />

Therefore, in the graph of Section 3.1.3, page 90, we cannot use the `<sixpence>` mask, but `sixpence+sixpences`.<fnref n="85" />

## 3.1 Analysis cascade

Most probably, we will be able to use the *Text/Open Recent* menu and open directly *DombeyAndSon.snt*. If this file is not in the offered list, we will use the *Text/Open* menu, find the *DombeyAndSon* folder, and choose the *All Files* option in the *Files of Type* drop-down menu. If *DombeyAndSon.snt* exists in it, we will open it because it means that it has already been preprocessed. Otherwise, we will open *DombeyAndSon.txt* and answer *Yes* to the question *Do you want to preprocess the text*. A window will open in which we will remove the first two checkmarks, leaving just the last one *Apply All default Dictionaries*.

### 3.1.1 The number.grf graph

We will copy the *NB2-999999.grf* graph and all the subgraphs (`*-subgraph.grf`) from the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\Numbers`

and paste them in the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\Measures\Analysis`

Then we will rename *NB2-999999.grf* as *number.grf*. We should remember that this graph recognizes all numbers between 2 and 999 999.<fnref n="86" />

We must make a final modification to the *number.grf* graph.

1. We replace the call to the *NB2-999* graph by a call to the two subgraphs, *NB1-99-subgraph* and *NB100-999-subgraph* (these two subgraphs produce no output and the first one recognizes number one).

2. We modify the two paths leading to and from this box (because the subgraphs don't produce output).

3. We remove the box *NB100-999-subgraph* after the box *thousand* and replace the subgraph *NB1-99-subgraph* by the subgraph *NB100-999-subgraph*.

4. Since we are going to use it in a cascade, we will change its output to produce lexical tags belonging to the category *Number*: the opening brace will replace the opening

<footnote n="84">⁸⁴ The Merriam-Webster Dictionary records that this noun has two forms of plural: *sixpence* and *sixpences* (https://www.merriam-webster.com/dictionary/sixpence). In the Unitex distribution dictionary only the first possibility is recorded.</footnote>

<footnote n="85">⁸⁵ The `<sixpence>` mask only matches the *sixpence* form present in the dictionary.</footnote>

<footnote n="86">⁸⁶ See Chapter 3, Section 2.4.2, page 62.</footnote>
XML tag `<number>` and the rest of the lexical tag including the closing brace will replace the closing XML tag `</number>`.

<figure><img src="image_180.png" /></figure>

This graph recognizes all numbers between 1 and 999 999. We will compile it.

### 3.1.2 Cascade

We will now produce a new analysis cascade that will initially contain only one graph, `number.grf`. Remember that in order to add it to the cascade this graph must be compiled. We will save the cascade in the folder

`MyUnitex\English\CasSys\UnitexGettingStarted\Measures`

and name it `analysis.csc`. The graph will work in the `Merge` mode.

When we launch this cascade, we will obtain 1 652 lines: the same 736 matches that we obtained with the `NB2-999999.grf` graph (the only difference is the output, the matches are in lexical tags now) and 916 matches for the ambiguous word `one`, which we previously omitted.

### 3.1.3 The measure.grf graph

We will now build a graph that recognizes and classifies measurement entities, that is numbers followed by measurement units. As explained before, it will recognize only those units that occur in the novel. The graph classifies the recognized entities as *Measure* and sub-classifies them as *Currencies*, *Durations*, *Lengths*, *Speed*, *Temperature* and *Weight*. Note that we have added the Part-Of-Speech code *N* in the lexical masks for the units when corresponding lemmas are ambiguous with verbs or adjectives.
<figure><img src="image_181.png" /></figure>

We will now compile this graph and add it as the second graph in the analysis cascade. This graph will also be applied in the *Merge* mode.

<table><thead><tr><th>#</th><th>Disabled</th><th>Name</th><th>Merge</th><th>Replace</th><th>Until Fix Point</th><th>Generaliz.</th></tr></thead><tbody><tr><td>1</td><td></td><td>number.fst2</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>2</td><td></td><td>measure.fst2</td><td>✓</td><td></td><td></td><td></td></tr></tbody></table>

After launching this cascade, the result will be again a list of 1 652 concordance lines, some of which contain measurement entities and the other just numbers or the word *one* (tagged as a number as well).

<figure><img src="image_182.png" /></figure>

## 3.2 Synthesis cascade

Our next step is to produce the cascade that will transform lexical masks into XML tags. We will open the text *DombeyAndSon_csc.txt* from the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon

without preprocessing.
### 3.2.1 The tag.grf graph

We will create a *tag.grf* graph that will be similar to the graph with the same name that we produced for the named entity recognition problem. So, we can copy this graph from the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Synthesis

and paste it in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Measures\Synthesis

We will delete the upper part of the previous *tag.grf* (the part used to recognize *Hidden* and *Xml* codes) and modify the rest. We will replace the old names of the tags (*Measure* and *Number* will replace *AbsoluteDate*, *RelativeDate*, *Town* and *Street*) and add the recognition of subclasses that will be values of the attribute *type*. Note that subclasses are optional (class *Measure* have them, while class *Number* does not) and that these codes are also inside tags `<code>` and `</code>` produced by the *analysis.csc* cascade. Two paths that lead to the ending node after the recognition of codes for classes (*Measure* and *Number*) need to have weights, otherwise the recognition may just randomly skip everything up to the closing tag of a matched `</csc>`.<fnref n="87" /> That is why the path for the recognition of subclasses has the higher priority $2$.

<figure><img src="image_183.png" /></figure>

### 3.2.2 Cascade

We will now create a new cascade containing only the *tag.grf* graph that we have just built and compiled. This graph will work in the *Replace* mode and it will iterate until it comes to a fixed point (no change made). Therefore, we must check these two options. We will save this cascade in the folder:

MyUnitex\English\CasSys\UnitexGettingStarted\Measures

<footnote n="87">⁸⁷ See the Chapter 3, Section 3.5.2, page 69.</footnote>
giving it the name *synthesis.csc*.

After launching this cascade, we will get the resulting *DombeyAndSon_csc_csc.txt* file in the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon

Here is an excerpt from this file in which we see that numbers are tagged both by themselves and inside `<Measure></Measure>` tags. In addition, *Measure* tags have the attribute *type* with a value indicating the type of measurement unit.

We will take the following sentence as an example:

Dombey was about <csc><form>eight</form><code>Number</code><code>grf=number</code>
</csc>-and-<csc><form><csc><form>forty</form><code>Number</code>
<code>grf=number</code></csc> years</form><code>Measure</code>
<code>Duration</code><code>grf=measure</code></csc> of age.

After applying the cascade, this sentence becomes:

Dombey was about <Number>eight</Number>-and-<Measure type="Duration"><Number>forty</Number> years</Measure> of age.

### 3.2.3 The numberDeletion.grf graph

We would like to annotate only measurement entities and not numbers unless they are part of them. We will thus create a *numberDeletion.grf* graph that removes *Number* tags found outside *Measure* tags.

<figure><img src="image_184.png" /></figure>

We will compile this graph and add it as the second in the *synthesis.csc* cascade to be used in the *Replace* mode.

This cascade applied to the text *DombeyAndSon_csc.txt* gives us the desired result. Now, the annotation of the first number is deleted.

Dombey was about eight-and-<Measure type="Duration"><Number>forty</Number> years</Measure> of age.

However, we see that a *number-and-number* sequence actually represents one number, while only the second number has been included in the *Measure* entity. We need to correct this.

### 3.2.4 The numberAndNumber.grf graph

We will just add a new graph, *numberAndNumber.grf*, between the two previous graphs of the *synthesis* cascade, to include the whole sequence in *Measure* tags. We have to take care to recognize the type of the *Measure* entity.
<figure><img src="image_185.png" /></figure>

<figure><img src="image_186.png" /></figure>

The previous example is now correctly annotated. The text obtained as a result of the tag.fst2 graph:

Dombey was about <Number>eight</Number>-and-<Measure type="Date"><Number>forty</Number> years</Measure> of age.

becomes now:

Dombey was about <Measure type="Duration"><Number>eight-and-forty</Number> years</Measure> of age.

# 4 Additional possibilities (only for confident users)

## 4.1 How does a generalization graph work?

The generalization graph developed in Section 1.1.5, page 75, enables the automatic creation of a graph which will replace it in the cascade and which is constructed on the basis of occurrences found in a text. This graph, having the same name townGeneralization.grf, is in the folder (this graph is the fourth graph in the analysis cascade):

MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\jubileePlaintext_csc\jubileePlaintext_4_0_snt

After rearrangement, this graph looks like this:

<figure><img src="image_187.png" /></figure>

As compared to the generalization graph, the automatically generated graph contains an additional box with the list of all words that were previously tagged as Town (London and Nearlondon). The G box is replaced by a negative right context in order to avoid tagging again as Town words that were already tagged Town.

## 4.2 Testing the variables

In the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Measures\Synthesis
we can observe in the *numberDeletion.grf* graph, Section 3.2.3, page 93, that some part is written twice (recognition of the element tags and the content). In order to avoid this, we could create a subgraph. However, we will instead explore another solution, the possibility to test variables.

We will create a new graph named *numberDeletionWithTest.grf* from the previous *numberDeletion.grf* graph. In this graph we recognize the element only once, but we decide what output will be produced based on the variable `$tag$` (value of the open tag of a measurement entity with its attribute). In this graph the `$tag.SET$` box is successful only if the `$tag$` variable is not empty, that is, the *Number* tag is inside a *Measure* tag. On the other hand, the path through the `$tag.UNSET$` box is successful only if the `$tag$` variable is empty, that is, the *Number* tag is outside a *Measure* tag.

<figure><img src="image_188.png" /></figure>

If we compile this graph and replace the second graph in the *synthesis.csc* cascade with it, the cascade will produce the same result as before when applied to the text *DombeyAndSon_csc.txt*. To do the substitution, we can use the *Disabled* column that enables switching on and off some graphs in a cascade.

<table><thead><tr><th>#</th><th>Disabled</th><th>Name</th><th>Merge</th><th>Replace</th><th>Until Fix Point</th><th>Generaliz.</th></tr></thead><tbody><tr><td>1</td><td>☐</td><td>tag.fst2</td><td>☐</td><td>☑</td><td>☑</td><td></td></tr><tr><td>2</td><td>☑</td><td>numberAndNumber.fst2</td><td>☐</td><td>☑</td><td>☑</td><td></td></tr><tr><td>3</td><td>☑</td><td>numberDeletion.fst2</td><td>☐</td><td>☑</td><td>☑</td><td></td></tr><tr><td></td><td>☑</td><td>numberDeletionWithTest.fst2</td><td>☐</td><td>☑</td><td>☑</td><td></td></tr></tbody></table>

## 4.3 Reuse the same graph

Redears can recall that we have constructed the *number.grf* graph from the *NB2-999999.grf* graph (see Section 3.1.1, page 89). We will now propose another solution, a single graph that can be used to annotate both with XML tags and with lexical tags. We will copy the *number.grf* graph and paste it in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Numbers

We will name it *numberXmlOrLt.grf* and we will remove the insertion of an opening tag `<E>/` and replace it with an empty box (`<E>`); and similarly, we will remove the insertion of a closing tag `<E>/`, *Number+grf=number* and replace it by an empty box (`<E>`).
<figure><img src="image_189.png" /></figure>

We will now describe how to use this graph for both types of annotation: We will use output variables.

### 4.3.1 Output variable

We will create a new graph, `numberTest.grf`, and place it in the same folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Numbers

In this graph, the call to the `numberXmlOrLt.grf` graph follows a test on an output variable `$test$`.

<figure><img src="image_190.png" /></figure>

A new graph named `numberXml.grf` will replace the `NB2-999999.grf` graph created in Chapter 3, Section 2.4.2, page 62. In this graph, the `<E>/XML` empty box is surrounded by special blue output parentheses, which you can insert by clicking the *Surround box selection with an output variable* button. The value of the output variable `$test$` is the output of the path enclosed in the blue parentheses, here XML.<fnref n="88" /> To tag only numbers greater than 1, as we saw in Chapter 3, Section 3.4.2, page 67, we will use weights.

<figure><img src="image_191.png" /></figure>

If we apply this graph to the novel text, we will obtain 1658 concordance lines, but only 736 lines with `<Number></Number>` tags, as in the precedent chapter.

<footnote n="88">⁸⁸ See Unitex user manual, Section 6.8.</footnote>
<figure><img src="image_192.png" /></figure>

In the same way, a new graph named *numberLt.grf* will replace the *number.grf* graph, created in *Section 3.1.1*, page 89.

<figure><img src="image_193.png" /></figure>

### 4.3.2 Graph repository

This solution can be improved by using a graph repository. For this, let us create a new folder, *Repository*, inside the folder:<fnref n="89" />

*MyUnitex\English\Graphs\UnitexGettingStarted*

In order to make this folder our *graph repository*, we have to open the *Info/Preferences/Directories* menu, click the *Set* button next to the *Graph repository* option in order to choose this specific folder, and finish the action by clicking the *OK* button.<fnref n="90" />

<figure><img src="image_194.png" /></figure>

We will copy the *numberXmlOrLt.grf* graph and its three subgraphs<fnref n="91" /> from the folder:

*MyUnitex\English\Graphs\UnitexGettingStarted\Numbers*

and paste them in the folder:

*MyUnitex\English\Graphs\UnitexGettingStarted\Repository*

<footnote n="89">⁸⁹ This folder is not necessarily located in the *MyUnitex* folder. It can be common to several projects, each corresponding to a particular private working directory. See *Chapter 2, Section 1.1.4*, page 11.</footnote>

<footnote n="90">⁹⁰ This action adds a line to the configuration file located in the folder:</footnote>

<footnote>*MyUnitex\English*</footnote>

<footnote n="91">⁹¹ *NB1-9-subgraph.grf*, *NB1-99-subgraph.grf* and *NB100-999-subgraph.grf*.</footnote>
A call to a subgraph that is in the repository is obtained by preceding its name with two colons;
the box will have yellow background. It doesn't matter in which folder the main graph is, since
it calls a subgraph that is in the repository.

We will create two new graphs in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Numbers

The first graph, named *numberXmlRepository.grf*, annotates numbers with `<Number></Number>` tags, while the second graph named *numberLtRepository.grf* transforms numbers in *lexical tags*.

<figure><img src="image_195.png" /></figure>

## 4.4 Hiding or deleting part of a text

Before starting this section

Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted`

Create a new folder named (without space): `JubileeLoremIpsum`.

Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\JubileeLoremIpsum`

Download the *JubileeLoremIpsum.xml* file.

### 4.4.1 Hiding part of a text

As mentioned in Section 2.1.2, page 86, the *toolHidden.grf* graph causes an error if the text to be hidden is too long. Actually, a Unitex graph analyzes a text by traversing all the possible paths (between the initial box and the final box) before deciding which path to choose. However, the maximum length of this run is set at 1,000 tokens. This value can be increased by using a script (see Chapter 7, page 149), but there is still a limit.

We will use a text that we prepared specifically for this exercise, *JubileeLoremIpsum.xml*. You will find it in the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\JubileeLoremIpsum

This file contains 596 sequences of letters and 750 other tokens between tags `<title></title>`.

As we explained before, we will rename this file as *JubileeLoremIpsum.xml.txt*, open it in Unitex and answer *Yes* to the question *Do you want to preprocess the text*. A window will open in which we will remove the first two checkmarks, leaving just the last one *Apply All default Dictionaries*.

Now we will open the *Text/Apply CasSys Cascade...* menu and select the *analysis.csc* file in the folder:

MyUnitex\English\CasSys\UnitexGettingStarted\XmlTextEntities

When we click the *Launch* button, we obtain an error message.
<figure><img src="image_196.png" /></figure>

We have to modify the *toolHidden.grf* graph that caused the error. We will open it (*Graphs/Open* menu) in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Analysis

and save it under the new name *toolHiddenDateTag.grf* (we will use the *FSGraph/Save as...* menu). We should not forget to modify the tracing feature as well: **,Hidden+grf=toolHidden** should be replaced by **,Hidden+grf=toolHiddenDateTag**. We will delete the first path, leaving only the path that hides the content between the `<date></date>` tags. We will compile this graph.

<figure><img src="image_197.png" /></figure>

We will create another new graph, named *toolHiddenTitleTag.grf*, to hide the content between the `<title></title>` tags.

<figure><img src="image_198.png" /></figure>

The notation `$[1,300]$` above the box that calls this subgraph controls the number of iterations (from one to 300 tokens).<fnref n="92" /> The *patternNotHiddenNotTitleCloseTag.grf* subgraph uses the negative right context to recognize a token only if it is neither a hidden text (token classified as “Hidden”), nor the *title* closing tag.<fnref n="93" />

<figure><img src="image_199.png" /></figure>

We will modify the analysis cascade in which we will replace the previous *toolHidden.grf* graph with the two graphs: the *toolHiddenDateTag.grf* graph followed by the *toolHiddenTitleTag* graph. The latter graph will be applied until the fixed point is reached.

<footnote n="92">⁹² Like all output, these interval boundaries are written after a slash character. The box actually contains `:patternNotHiddenNotTitleCloseTag/$[1,300]$`.</footnote>

<footnote n="93">⁹³ See *Chapter 3, Section 1.7.3*, page 55, for more details about negative right context.</footnote>
<figure><img src="image_200.png" /></figure>

We will explain in detail how the `toolHiddenTitleTag.grf` graph works: Unitex detects the opening `title` tag, then it iterates and recognizes 300 tokens at most. If it does not come upon the closing `title` tag, as will be the case with our example text, it tags these 300 tokens as a text to be hidden (`Hidden`) with an additional feature (`Hidden=insideTag`) stating that the end of a text was not yet reached. The graph is launched again. In the second run, the graph detects the new token that we created in the previous step, followed by some regular tokens and after that it loops again over at most 300 tokens.<fnref n="94" /> When the closing `title` tag is recognized before reaching the upper limit of iterations 300, the text collected as a title is categorized as `Hidden` with the additional attribute `Hidden=endTag`.

As we saw in Section 1.1.7.2, page 78, the cascade created a new folder:

MyUnitex\English\Corpus\UnitexGettingStarted\JubileeLoremIpsum\JubileeLoremIpsum.xml_csc

containing a new file for each graph or iteration of graph.

The `toolHiddenTitleTag.grf` graph is the third graph of the cascade and it is iterated four times. So the cascade created four new files named:

1. JubileeLoremIpsum.xml_3_0.snt,

2. JubileeLoremIpsum.xml_3_1.snt,

3. JubileeLoremIpsum.xml_3_2.snt,

4. JubileeLoremIpsum.xml_3_3.snt.

To check the result, we will open these files. If we search with `name=title` to gain insight only into interesting categories, we will find:

<table><tr><td>JubileeLoremIpsum.xml_3_0.snt</td><td>{name=title+openTag}{Hidden=insideTag} Aliquam convallis sollicitudin purus...{name=title+closeTag}</td></tr><tr><td>JubileeLoremIpsum.xml_3_1.snt</td><td>{name=title+openTag}{Hidden=insideTag}{Hidden=insideTag} augue. Integer id felis...</td></tr><tr><td>JubileeLoremIpsum.xml_3_2.snt</td><td>{name=title+openTag}{Hidden=insideTag}{Hidden=insideTag}{Hidden=endTag}{name=title+closeTag}<fnref n="95" /></td></tr></table>

The `JubileeLoremIpsum.xml_3_2.snt` and `JubileeLoremIpsum.xml_3_3.snt` files are identical, which means that the fixed point has been reached and that the iteration of the `toolHiddenTitleTag` graph stops.

The synthesis cascade (the same as before, see Section 2.2, page 87) will rewrite this text without any internal annotation, whereas, if the hiding graphs had not been used, Wednesday

<footnote n="94">⁹⁴ Actually, the 300 tokens surrounded with braces have become a single token of a category *Hidden*.</footnote>

<footnote n="95">⁹⁵ Which means that at this point our text, inside `<title></title>` tags, consists of only 5 tokens.</footnote>
June 1, 2022 would be tagged as an *AbsoluteDate* inside the `<date></date>` tags and *Nearlondon* would be tagged as a *Town* inside the `<title></title>` tags.<fnref n="96" />

```xml
<xml>
  <date>Posted on Wednesday June 1, 2022</date>
  <title><p>From Nearlondon:</p><p> Lorem ipsum...</p></title>
```

### 4.4.2 Deleting part of a text

#### 4.4.2.1 The deleting graphs

Instead of hiding the *date* and *title* tags, suppose we want to remove their content.

For the *date* tag, presumably with a short content, we can write a graph that replaces everything recognized with `<date></date>`. We will name this graph `toolEraseDateTag.grf`.

<figure><img src="image_201.png" /></figure>

For the *title* tag, which can have a very long content in the number of tokens, we will use a loop limited to 300 tokens. We will name this graph `toolEraseTitleTag.grf`.

<figure><img src="image_202.png" /></figure>

This graph uses a subgraph that we will name `patternNotCloseTitle.grf`.

<figure><img src="image_203.png" /></figure>

These two graphs are similar to graphs developed in the previous section; they differ only in the produced output. We will build now the cascade. The `toolEraseDateTag` graph and the `toolEraseTitleTag` graph are in *Replace* mode; the `toolEraseTitleTag` graph is repeatedly used until a fixed point.

<figure><img src="image_204.png" /></figure>

We will open the *JubileeLoremIpsum.xml.snt* text and launch the analysis cascade; we will open the *JubileeLoremIpsum.xml_csc.txt* text (without applying dictionaries) and launch the

<footnote n="96">⁹⁶ Now that we no longer have to care about the length of the text, we can directly hide the whole preamble, between the `<xml>` and `<body>` tags, instead of hiding inside the `<date>` and `<title>` tags separately.</footnote>
synthesis cascade. The beginning of the resulting text *JubileeLoremIpsum.xml_csc_csc.txt* is
now:

```xml
<xml>
<date></date>
<title></title>
<body>...
```

#### 4.4.2.2 Comparing the variables

To stick to common XML practice, we will add to the synthesis cascade a new graph that transforms two successive tags, opening and closing, into a self-closing tag. We will name this graph `selfClosing.grf`. We will use two variables `$tag1$` and `$tag2$` and compare them with the formula `$tag1.EQUAL=tag2$`.<sup>97</sup> A self-closing tag is output only if two successive opening and closing tag contain the same element name.

<figure><img src="image_205.png" /></figure>

The new graph is added as the third in the synthesis cascade and marked to be used in *Replace* mode.

<table><thead><tr><th>#</th><th>Disabled</th><th>Name</th><th>Merge</th><th>Replace</th><th>Until Fix Point</th><th>Generalize</th></tr></thead><tbody><tr><td>1</td><td>☐</td><td>tag.fst2</td><td>☐</td><td>☑</td><td>☑</td><td>☐</td></tr><tr><td>2</td><td>☐</td><td>internalDeletion.fst2</td><td>☐</td><td>☑</td><td>☑</td><td>☐</td></tr><tr><td>3</td><td>☐</td><td>selfClosing.fst2</td><td>☐</td><td>☑</td><td>☑</td><td>☐</td></tr></tbody></table>

After applying the synthesis graph to *JubileeLoremIpsum.xml_csc.snt*, we will get what we wanted.

```xml
<xml>
<date/>
<title/>
<body>...
```

## 4.5 Sub-categorization with cascades

We will work here on the *JubileePlainText* text.

### 4.5.1 Three modified graphs

We will take the first two graphs of [Section 1.1](#), page 71, named `absoluteDate.grf` and `relativeDate.grf`; we will modify them by introducing sub-categories. Either `DayName`, `DayNumer` (as in graphs) or name of a day, number of a day.<sup>98</sup>

<footnote n="97"><sup>97</sup> It is also possible to compare a variable to a character string either in a case-sensitive or case-insensitive way. See the Unitex user manual, Section 6.9.2.</footnote>

<footnote n="98"><sup>98</sup> Care must be taken that, on each path, a closing brace corresponds to an opening brace. In a complex graph, we can verify this by using the FSGraph/Tools/Verify braces option.</footnote>
<figure><img src="image_206.png" /></figure>

Then, we must also modify the *tag* graph of the synthesis cascade, adding the new categories.<fnref n="99" />

<figure><img src="image_207.png" /></figure>

When we run the analysis and synthesis cascades with these three new graphs (don't forget to compile the modified graphs!), we get the desired result, a subcategorization of dates.

<footnote n="99">⁹⁹ When the categories have been added in the corresponding box, it is possible to sort them in alphabetical order by the FSGraph/Tools/Sort Node Label option.</footnote>
This week we celebrate the Platinum Jubilee of the Queen of England: seventy years of reign! The party will start on `<AbsoluteDate><Month>June</Month><DayNumber>2</DayNumber>`, `<Year>2022</Year></AbsoluteDate>` and end on `<RelativeDate><Month>June</Month><DayNumber>5</DayNumber></RelativeDate>`. The Queen ascended the British throne on `<AbsoluteDate><Month>February</Month><DayNumber>6</DayNumber>`, `<Year>1952</Year></AbsoluteDate>`.

Our small town of `<Town>Nearlondon</Town>` will also celebrate this event. `<Town>Nearlondon</Town>` has always been a loyal subject of Her Majesty. Although the city of `<Town>London</Town>` is dear to us, on `<RelativeDate><DayName>Thursday</DayName><Month>June</Month><DayNumber>2</DayNumber></RelativeDate>` we will change the name of `<Address>London Street</Address>` to `<Address>Queen Elizabeth Street</Address>`.

Everyone will fondly remember 2022!

God Save the Queen!

## 4.5.2 Generalization graphs with restrictions

### 4.5.2.1 Without substitution of a category

A single number should not be recognized as a year, even if it has 4 digits. This could cause too many errors. This is why 2022 in the penultimate sentence *Everyone will fondly remember 2022* was not tagged as a *Year* (nor as a *date*). We will use a generalization graph to decide that since 2022 has already been recognized as a *date*, it has to be recognized again as a *date*.<fnref n="100" /> We will name this graph *yearGeneralization.grf*. Note that the subcategory *Year* is in the box, while category *AbsoluteDate* is the output of the box.

<figure><img src="image_208.png" /></figure>

We will place this graph in the third position of the *analysis* cascade. Don't forget to check the *Generaliz...* box.

<figure><img src="image_209.png" /></figure>

When we run this new analysis cascade and the same synthesis cascade, the desired result is as expected (We display only the penultimate line).

Everyone will fondly remember `<AbsoluteDate><Year>2022</Year></AbsoluteDate>`!

As explained in Section 4.1, page 94, the generalization graph is automatically replaced in the cascade by a generated graph, having the same name *yearGeneralization.grf*, which is in the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlainText\jubileezb PlainText_csc\jubileezb PlainText_3_0_snt

After rearrangement, this graph looks like this:

<figure><img src="image_210.png" /></figure>

<footnote n="100">¹⁰⁰ See Section 1.1.5, page 75. Remember that the .grf file must stay in the same folder as the .fst2 file.</footnote>
### 4.5.2.2 With substitution of a category

It is also possible to modify the category obtained. For example, if we will want to categorize `YearDate` years recognized in this way, we must write in the box `Year.YearDate/,.AbsoluteDate`. Save this graph.

<figure><img src="image_211.png" /></figure>

Of course, this will only work if the `tag.grf` graph of the synthesis cascade is also modified to recognize the new tag `YearDate`.

<figure><img src="image_212.png" /></figure>

`<csc><form>June 2, 2022</form><code>AbsoluteDate</code><grf=absoluteDate></csc>`

The result is now different (don't forget to compile the modified graphs!).

Everyone will fondly remember `<YearDate><Year>2022</Year></YearDate>!

After rearrangement, the replaced generalization graph looks like this:

<figure><img src="image_213.png" /></figure>
<figure><img src="image_214.png" /></figure>
# Chapter 5: dictionary creation

Before starting this chapter

Go to the folder: `MyUnitex\English\Inflection`

Select and copy the two files named *Equivalences.txt* and *Morphology.txt*.

Create a new folder named *UnitexGettingStarted* (without space).

Go to the folder: `MyUnitex\English\Inflection\UnitexGettingStarted`

Paste the two files *Equivalences.txt* and *Morphology.txt*.<fnref n="101" />

## 1 Introduction

We have already discussed the use of the dictionaries from the Unitex distribution in Chapter 2, Section 1.2.4, page 14.

Before we begin, we will open in Unitex our novel, *DombeyAndSon.txt*, located in the folder:

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon`

We will answer *No* to the question *Do you want to preprocess the text*.

In this chapter, we will create a new dictionary following these three steps:

1. We will manually create a dictionary of representative (principal) forms (or lemmas) and rules for their inflection;

2. We will create automatically a dictionary of inflected forms, that is, all forms that can occur in texts, like singular and plural forms of nouns;

3. We will automatically create a dictionary in format `.bin` that will appear in the *User resources* field.

For the first step, we will use the Unitex editor. We will open the *File Edition/New File* menu and save the file in the *Dela* folder using the name *myDictionary dik*. We will make a difference between the inflection of monolexical words and polylexical words.<fnref n="102" />

## 2 Inflection of monolexical words

### 2.1 Inflection by simple suffixation

#### 2.1.1 The nouns with s in the plural

We will enter the following three lines in the *myDictionary dik* file. Note that these entries should not contain spaces. In addition, we will finish the third line (the last line) by clicking the *Enter* key.

<footnote n="101">¹⁰¹ These two files define the morphological codes used by Unitex for a specific language. It is not recommended to modify them, except if a user wants to use different morphological codes or wants to create resources for a new language.</footnote>

<footnote n="102">¹⁰² It is also possible to create the inflection rules based on word stems, which is convenient for the Semitic languages; see the Unitex user manual, Section 3.5.4.</footnote>
advantage,N1
agent,N1
firm,N1

Each line represents a dictionary entry consisting of a lemma (representative form), followed after a comma by the associated inflectional graph (in the case of our entries N1.grf). In the name of this graph, *N* represents the grammatical category (Part-Of-Speech) noun. We will now create this graph and save it in the folder:

MyUnitex\English\Inflexion\UnitexGettingStarted

We will save and close the *myDictionary dik* file.

<figure><img src="image_215.png" /></figure>

The *N1* graph enables the inflection of nouns for which the plural is obtained by adding an *s* to the singular form. We will use, although it is not mandatory, the same codes used by the dictionary provided with the Unitex distribution *dela-en-public.bin*. One can see these codes in the *Word List* window: *s* is used for the singular forms and *p* for the plural forms. We will add a comment (a box starting with the slash `/`) with some examples of nouns that can be inflected by this graph.

This inflectional graph is read like this: in the upper part, there is an empty box, signifying that a word (lemma) to which it applies does not change; in the lower part, there is a box containing an *s*, signifying that it is concatenated to the word to which it applies. The output of these boxes are values of grammatical categories associated with inflected forms.

<figure><img src="image_216.png" /></figure>

This graph must be compiled, so we will click the *Compile graph* button. We will obtain an error message: *Main graph matches epsilon! Error: the main graph N0 recognize <E>.* The meaning of this message is that we cannot use this graph for search (the *Locate pattern* option) because it recognizes an empty symbol.<fnref n="103" /> Since we have not developed this graph for search, we can ignore this message. We will simply click the *OK* button.

We will now launch the inflection program. This program will create a dictionary of all inflected forms, named *myDictionaryflx.dic* (*flx* is automatically added to the file name). In order to do this, we will open the *myDictionary.dic* file in the *DELA* menu, which gives access to programs

<footnote n="103">¹⁰³ Which would cause an infinite loop and crash the computer.</footnote>
dealing with dictionaries. Note that now we will not use the *File Edition* menu, because we are not going to create or modify a dictionary of lemmas.<fnref n="104" />

We will now inflect the *myDictionary.dic* dictionary by using the *DELA/Inflect* menu option. First, we will set the folder in which the inflectional graphs are stored. Click the *SET* button, select the folder:<fnref n="105" />

*MyUnitex\English\Inflexion\UnitexGettingStarted*

and click the *Open* button. Next, click the *Inflect Dictionary* button.

<figure><img src="image_217.png" /></figure>

The *myDictionaryflx.dic* dictionary is created and displayed on the screen.

<figure><img src="image_218.png" /></figure>

If we want to use this dictionary on our novel (or any other text), one more step is necessary. We must choose the *DELA/Compress into FST* menu option and confirm by clicking *OK*. This will create two more files: *myDictionaryflx.bin* and *myDictionaryflx.inf*.<fnref n="106" /> Note that if you want to use this dictionary on some other computer you need to copy both files.

We will now close both windows containing the *myDictionary.dic* and *myDictionaryflx.dic* dictionaries. We want to apply our newly created dictionary to the novel (already open). We will open the *Text/Apply Lexical Resources* menu and click the *Clear* button (in order to cancel all previous selections). We will choose the *myDictionaryflx.bin* file (appearing in the left field) and click the *Apply* button. As a result, the *Word List* window will be displayed.

<footnote n="104">¹⁰⁴ If you detect an error in your *myDictionary.dic* file, you have to close it in the *DELA* menu and reopen it in the *File Edition* menu.</footnote>

<footnote n="105">¹⁰⁵ If you are not going to go through this entire section in one session, remember to set the folder containing the inflectional graphs when you start to work again.</footnote>

<footnote n="106">¹⁰⁶ The *myDictionaryflx.bin* file contains words, but it is a binary file and a text editor cannot read it. The *myDictionaryflx.inf* file contains the factorization of tags associated with entries: it is readable.</footnote>
<figure><img src="image_219.png" /></figure>

Since five of our six inflected forms (two for each lemma) appear in this novel, they will be listed in the *DLF* area. *DLC* area is empty, while the *ERR* area lists 16 427 unknown simple words. This is natural since we applied only our tiny dictionary to the text.

We will close the *Word List* window.

### 2.1.2 Two more examples

We will open the *myDictionary.dic* file again, using the *File Edition/Open.../Dictionaries* menu and answer *OK* to the warning *This is not necessarily the text being processed by Unitex stating that this file cannot be used for the text analysis.*<sup>107</sup> We will add to our dictionary of lemmas some more illustrative examples:

* for the inflection of the nouns *branch*, *bus*, *box* having plural forms produced by adding *es* to the singular form we will use the *N2* graph;

* for the inflection of the verbs *accept*, *connect*, *work* having forms produced by adding *s*, *ed* and *ing* to the infinitive form we will use the *V1* graph.<sup>108</sup>

We will use the same format as before – remember that you should not use spaces and to press the *Enter* key after the last entry.

advantage,N1
agent,N1
firm,N1
branch,N2
bus,N2
box,N2
accept,V1
connect,V1
work,V1

We will save and close the *myDictionary.dic* file.

<footnote n="107"><sup>107</sup> Except if we close it and reopen using the menu *Text/Open*.</footnote>

<footnote n="108"><sup>108</sup> We saw in the *Chapter 2, Section 1.2.5*, page 15, that *G* codes the gerund of a verb. We saw in the *Chapter 3, Section 1.2*, page 47, that *K* codes the past participle and *W* codes the infinitive of a verb. We also need to know that the code *P* is used for the present tense, *I* for the perfect tense, while *1*, *2* and *3* are codes for persons.</footnote>
We will create and compile the *N2* et *V1* graphs. We can notice that the *N2* graph is very similar to the previous *N1* graph (the content of the lower box is *es* instead of *s*); so we can facilitate our work by opening *N1.grf*, using the *FSGraphs/Save as...* menu to rename it as *N2.grf*; and then changing the content of this box. Do not forget to replace the comment as well by typing new examples and to compile graphs.

<figure><img src="image_220.png" /></figure>

We can now open the *myDictionary dik* dictionary by using the *DELA/Open Recent* menu and inflect it using the *DELA/Inflect* option. The new dictionary named *myDictionaryflx.dic* will be displayed in a separate window with its 57 entries. You can see below the content of this window with some of the produced inflected forms.

<figure><img src="image_221.png" /></figure>

We will verify the inflected forms and, if we find them correct, we can sort them by using the *DELA/Sort Dictionary* option.<fnref n="109" />

<footnote n="109">¹⁰⁹ Note that we could have inflected this dictionary after saving the first graph, *N1.grf*. The inflection program issue error notifications for all entries using graphs that do not exist and produces a dictionary of inflected forms only for those entries for which inflectional graphs exist.</footnote>
<figure><img src="image_222.png" /></figure>

When inflecting a dictionary of lemmas, we can check the *Sort and factorize inflectional codes* option in the *DELA/Inflect* menu. A more compact dictionary of inflected forms will be produced. Namely, there will be only one entry for each form of a lemma, followed by a sequence of all possible values of grammatical codes. If we apply this to our *myDictionary dik* dictionary, we will get the *myDictionaryflx.dic* dictionary of inflected forms consisting of 24 entries, some of which are displayed in the figure below.

<figure><img src="image_223.png" /></figure>

In order to make this dictionary functional, we must apply the *DELA/Compress into FST* option. We will now close the windows containing *myDictionary.dic* and *myDictionaryflx.dic*. We will test this new dictionary by applying it to our novel (do not forget to click the *Clear* button in the *Text/ApplyLexical Resources*) menu. In the *DLF* zone of the *Word List* window, 19 entries will be listed (the word forms *accepts*, *bus* and *buses* do not occur in the novel) compared to the five entries obtained by the previous version of the dictionary. Therefore, there are less entries in the *ERR* zone: 16 411. Close the *Word List* window.
<figure><img src="image_224.png" /></figure>

## 2.2 The L operator

In order to concatenate the suffix to a substring of a word, that is, not after its last letter (e.g. *compan* from *company*, see below), we use the *L* operator (standing for *Left*) in order to move from right to left starting from the end of a word.

### 2.2.1 The nouns with final y

Plural forms of the nouns *company*, *complexity*, *family* are produced by replacing the final y with the suffix *ies*. Starting from the end of the word, we move one letter to the left (operator *L*) and then we add *ies*. This rule will be written as *Lies* in the box for plural forms in the *N3* graph.<fnref n="110" />

We will now add three more lines to the *myDictionary.dic* file (having now 12 entries). Remember to open it in the Unitex editor by using the *File Edition/Open.../Dictionaries* menu.

...
company,N3
complexity,N3
family,N3

We will save this dictionary and close the editor. Now we will create the *N3* graph, using as a base the *N1.grf* graph, modifying it and saving with the *Save as...* menu option under the name *N3.grf*. Remember to replace the comment with new examples and compile the graph.

<figure><img src="image_225.png" /></figure>

In order to use this new version of the dictionary we have to inflect and compress it. These are the steps to follow: open the *myDictionary.dic* file (the *DELA/Open Recent* menu) and inflect it (the *DELA/Inflect* menu, *Sort and factorize inflectional codes* option). The new version of the *myDictionaryflx.dic* dictionary will be displayed containing 30 lines. If inflected forms

<footnote n="110">¹¹⁰ Since the inflection endings are written in lower-case letters, the upper-case letters are reserved for operators.</footnote>
are correct, we can proceed with the production of the dictionary version that can be used in text analysis: the DELA/Compress into FST menu. We will close the windows displaying `myDictionary dik` and `myDictionaryflx.dik`. We can now apply the new dictionary to our text: the Text/ApplyLexical Resources menu (do not forget to click the Clear button). The DLF zone of the Word List window shows that 23 words have been recognized by the `myDictionaryflx.dik` dictionary. We can also see that the word *complexity* does not occur in our text. Before proceeding with new examples, we will close the Word List window.

### 2.2.2 Several more examples

We will again open the `myDictionary.dik` file in the Unitex editor (File Edition/Open.../Dictionaries). We will add to the dictionary some more examples of inflections requiring the L operator once or several times.

* leaf, thief, calf are nouns in which the final f is replaced by ves in plural (N4.grf graph).

* wife, knife, life are nouns in which the final fe is replaced by ves in plural (N5.grf graph, the L operator is applied twice).

* mouse, louse are nouns in which the final ouse is replaced by ice in plural (N6.grf graph, the L operator is applied four times).

* narcissus, terminus, tumulus are nouns in which the final us is replaced by i in plural (N7.grf graph, the L operator is applied twice).

* phenomenon, criterion, datum are nouns in which the two final letters are replaced by a in plural (N8.grf graph, the L operator is applied twice).

* love, live, give are verbs for which the final e is deleted before adding the ending ing (V2.grf graph, the L operator is applied once for this inflected form).

* accompany, try, worry are verbs for which the final letter y is replaced by i in all inflected forms (V3.grf graph, the L operator is applied once for all inflected forms).

We will now add entries for these nouns and verbs into the `myDictionary.dik` dictionary (do not forget to press the Enter key after the last entry). This dictionary now has 32 entries.

...leaf,N4
...worry,V3

We will save the dictionary and close the Unitex editor. After that, we will create and compile new graphs for the inflection of nouns and verbs.

<figure><img src="image_226.png" /></figure>

N4

<figure><img src="image_227.png" /></figure>

N5
<figure><img src="image_228.png" /></figure>

N6

narcissus
terminus
tumulus

<figure><img src="image_229.png" /></figure>

N7

phenomenon
cntenion
datum

<figure><img src="image_230.png" /></figure>

N8

love
live
give

<figure><img src="image_231.png" /></figure>

V2

accompany
try
worry

<figure><img src="image_232.png" /></figure>

V3

As the last step, we will open the *myDictionary.dic* file using the *DELA/Open Recent* option and inflect it choosing *DELA/Inflect*. The new *myDictionaryflx.dic* dictionary contains 82 lines (if the *Sort and factorize inflectional codes* option is on). After compressing this dictionary (*DELA/Compress into FST* menu), we will close both windows (*myDictionary.dic* and *myDictionaryflx.dic*). We will open the *Text/ApplyLexical Resources* menu (do not forget to clear all preselected dictionaries by clicking the *Clear* button and select the *myDictionary.bin* dictionary). The *Word List* now displays 57 recognized words. We will close *Word List*.
## 2.3 Operators R, C and D

We use the *R* operator (**Right**) in combination with the *L* operator, when we need to substitute part of a lemma while preserving another part on the right.

What do the nouns *goose* and *tooth* have in common? In the plural, the last two letters remain as they are while the string *oo* is replaced by *ee*. It means that starting from the end of a lemma, we have to move to the left by four letters (*LLLL*), then write *ee*, and then move to the right by two letters (*RR*). This procedure is used by the *N9.grf* graph.

<figure><img src="image_233.png" /></figure>

Let us see how it works. The vertical bar indicates the current position in the processed word (the position of the cursor) at each step in the sequence of operations.

<table><tr><td>|LLLLeeRR</td><td>goose|</td><td>tooth|</td></tr><tr><td>L|LLLLeeRR</td><td>goos|e</td><td>toot|h</td></tr><tr><td>LL|LLeeRR</td><td>goo|se</td><td>too|th</td></tr><tr><td>LLL|LeeRR</td><td>go|ose</td><td>to|oth</td></tr><tr><td>LLLL|eeRR</td><td>g|oose</td><td>t|ooth</td></tr><tr><td>LLLLe|eRR</td><td>ge|ose</td><td>te|oth</td></tr><tr><td>LLLLee|RR</td><td>gee|se</td><td>tee|th</td></tr><tr><td>LLLLeeR|R</td><td>gees|e</td><td>teet|h</td></tr><tr><td>LLLLeeRR|</td><td>geese|</td><td>teeth|</td></tr></table>

Other examples:

* *foot* is a noun for which the last letter remains in the plural while the string *oo* is replaced by *ee* (*N10.grf* graph).

* *man*, *woman*, *hypothesis* are nouns for which the penultimate letter becomes *e* in the plural (*N11.grf* graph).

<figure><img src="image_234.png" /></figure>

N10

<figure><img src="image_235.png" /></figure>

N11
The C operator (Copy) duplicates the current letter, that is the letter situated on the left of the cursor, which permits the inflection with a single graph of the verbs jog, plan, stop, prefer for which the final consonant is duplicated in the past tense, the past participle and the gerund (*V4.grf* graph).

Other examples:

* past tense is identical to the present tense for verbs *slit*, *split* while their final consonant is duplicated in the gerund (*V5.grf* graph).

* the final consonant is duplicated in the plural before adding *es* for the nouns *quiz* or *gas* (*N12.grf* graph).

<figure><img src="image_236.png" /></figure>

The D operator (Delete) deletes the current letter (and shifts the right-hand part of the word to the left), that is the letter situated to the left of the cursor, which permits the inflection with a single graph of the verbs *shoot*, *speed*, *meet* for which the double letter (*oo*, *ee*) is simplified in the past tense and the past participle (*V6.grf* graph).<fnref n="111" /> We will demonstrate step by step how the pattern *LDR* used in the *V6.grf* graph works.

<footnote n="111">¹¹¹ Note that some other operators exist that are described in the Unitex user manual, Section 3.5.1.</footnote>
<table><tr><td>LDR</td><td>shoot</td><td>meet</td></tr><tr><td>LDR</td><td>shoot</td><td>meet</td></tr><tr><td>LD | R</td><td>shoot</td><td>meet</td></tr><tr><td>LDR |</td><td>shoot</td><td>meet</td></tr></table>

Other examples:

* the verbs *kneel*, *sleep*, *creep* for which the double letter (*oo*, *ee*) is simplified in the past tense and the past participle and the letter *t* is added (*V7.grf* graph).

<figure><img src="image_237.png" /></figure>

We will now add these eight nouns and twelve verbs into the *myDictionary.dic* dictionary (do not forget to press the *Enter* key after the last one). This dictionary now has 52 entries.

... 
  goose,N9 
  ...
  creep,V7

We will save the dictionary and close the Unitex editor.

After opening the *myDictionary.dic* file, using the *DELA/Open Recent* menu, and inflecting it, choosing the *DELA/Inflect* menu, the new *myDictionaryflx.dic* dictionary contains 144 lines (if the *Sort and factorize inflectional codes* option is on). After compressing this dictionary (*DELA/Compress into FST* menu), we will close both windows (*myDictionary.dic* and *myDictionaryflx.dic*). We will apply the new *myDictionaryflx.bin* dictionary (*Text/ApplyLexical Resources* menu). The *Word List* now displays 103 recognized words. We will close the *Word List*.

## 3 Inflection of multi-word units

Before starting this section

If you have not modified the *Norm.txt* file as explained in *Chapter 2, Section 3.1.2*, page 35, do so, or download it, then close and reopen Unitex.
In order to inflect multi-word units (MWU), or compound words, we will create graphs that have as many boxes as there are tokens in an MWU with the addition of one box for the result. One has to remember that processing units, or *tokens*, for Unitex are strings of alphabetic characters (letters) and any other individual character, including the space.<fnref n="112" />

## 3.1 Example of the MWU *air of mystery*

MWUs *air of mystery*, *sort of thing* and *tree of knowledge*, each comprised of five tokens (three alphabetic strings and two spaces), of which only the first one is inflected to obtain the plural form. We will inflect these MWUs with the graph named *NC_XWW1*.<fnref n="113" />

We will now add three entries to the *myDictionary.dic* file (and click *ENTER* after the third line) using the Unitex editor (*File Edition/Open.../Dictionaries*; answer *OK*). These entries contain the information about the inflection of the MWU as a whole (in this case *NC_XWW1*) and between parentheses (without any space) information about the inflection of each word token (if it inflects): lemma, inflectional graph and inflectional information of a simple word form used in a MWU being described.

* ...
* air(air.N1:s) of mystery,NC_XWW1
* sort(sort.N1:s) of thing,NC_XWW1
* tree(tree.N1:s) of knowledge,NC_XWW1

The *NC_XWW1.grf* graph consists of six boxes: the first five boxes refer to the five tokens in a MWU, while the sixth box is empty, it is used only for output.

air of mystery
sort of thing
tree of knowledge

<figure><img src="image_238.png" /></figure>

The second (<$2>) and the fourth (<$4>) token correspond to spaces, so they do not inflect. The third (<$3>) and the fifth (<$5>) token refer to word tokens that do not inflect in described MWUs. Only the first token inflects in number. In order to indicate this, we will use the keyword *Nb* representing the grammatical category of number.<fnref n="114" /> The formula *Nb=$n* means that the variable *$n* is instantiated to all possible values of the keyword *Nb*, in this case, *s* and *p*. This means that the single path in the graph corresponds to both singular and plural forms of a MWU being inflected. This formula in the first box indicates that all forms (actually two) of the first word token will be taken in turn, while its use in the sixth box indicates that the corresponding values will be assigned to the inflected MWU forms in the output.

<footnote n="112">¹¹² See *Chapter 2, Section 1.2.3*, page 13.</footnote>

<footnote n="113">¹¹³ The prefix *NC_* in this name indicates that a noun to be inflected is a compound. The suffix is free.</footnote>

<footnote n="114">¹¹⁴ The keywords and their possible values are given for each language in the *Morphology.txt* file that has to be in the same folder as the inflectional graphs.</footnote>