<figure><img src="image_239.png" /></figure>

We will close this graph after compiling it. We will open the *myDictionary.dic* file using the *DELA/Open Recent* menu and inflect it using the inflectional graphs in the folder:

MyUnitex\English\Inflection\UnitexGettingStarted

The new *myDictionaryflx.dic* dictionary has 150 entries, two for each of the newly added MWU lemmas. After its transformation (*DELA/Compress into FST* menu), we will close the windows with the *myDictionary.dic* and *myDictionaryflx.dic* dictionaries; and open the *Text/ApplyLexical Resources* menu, click the *Clear* button, select the *myDictionaryflx.dic* dictionary and then click the *Apply* button. The number of recognized simple words listed in *Word List* is still 103 (*DLF*), but there are also three MWUs (*DLC*).

<figure><img src="image_240.png" /></figure>

We will close the *Word List*.

## 3.2 Some other examples

We will open for the last time the *myDictionary.dic* file using the *File Edition/Open.../Dictionaries* menu, and clicking the *OK* button. We will add to it lemmas for some other illustrative examples of MWUs.

* *crow's nest, death's-head, hair's breadth*; these are MWUs composed, like before, of five tokens, but this time only the last one is inflected (*NC_WWX1.grf* graph); the only difference between this graph and the *NC_XWW1.grf* graph is that now the fifth instead of the first box contains the formula $Nb=$n indicating the inflection in number.

* *boa constrictor, prince regent, secretary general*; these are three token MWUs in which only the first word token inflects (*NC_XW1.grf* graph).

* *altar rail, Anglo-Norman, bad boy*; only the last word token inflects (*NC_WX1.grf* graph).

* *after-life, eye-cup, sky-blue*; these MWUs inflect like the previous except that the first and the third word tokens can be written together (*NC_WX2.grf* graph); the fact that two words can be written without the hyphen is indicated by the new path that connects directly the first and the third token.
* three quarter, blood red, self glorification; these MWUs inflect like the previous except that the first and the third token can be connected with a hyphen (*NC_WX3.grf* graph); the new path is added to the *NC_WX3.grf* graph which connects the *<$1>* and *<$3>* boxes with a hyphen in between.

* carry out, take off, wash out are MWUs which unites two previous cases, namely, two word tokens can be either written together or separated by a hyphen or a space, while only the last word inflects (*NC_WX4.grf* graph).

* attorney general, notary public are MWUs that allow inflection of either the first or the second word (but not both!) (*NC_XX1.grf* graph); this graph has two paths, the upper taken from the *NC_XW1.grf* graph and the lower from the *NC_WX1.grf* graph.<fnref n="115" />

We will get the final version of the *myDictionary.dic* dictionary by adding these twenty entries:

...
crow's nest(nest.N1:s),NC_WWX1
death's-head(head.N1:s),NC_WWX1
hair's breadth(breadth.N1:s),NC_WWX1
boa(boa.N1:s) constrictor,NC_XW1
prince(prince.N1:s) regent,NC_XW1
secretary(secretary.N3:s) general,NC_XW1
altar rail(rail.N1:s),NC_WX1
Anglo-Norman(Norman.N1:s),NC_WX1
bad boy(boy.N1:s),NC_WX1
after-life(life.N5:s),NC_WX2
eye-cup(cup.N1:s),NC_WX2
sky-blue(blue.N1:s),NC_WX2
three quarter(quarter.N1:s),NC_WX3
blood red(red.N1:s),NC_WX3
self glorification(glorification.N1:s),NC_WX3
carry out(out.N1:s),NC_WX4
take off(off.N1:s),NC_WX4
wash out(out.N1:s),NC_WX4
attorney(attorney.N1:s) general(general.N1:s),NC_XX1
notary(notary.N3:s) public(public.N1:s),NC_XX1

We will construct and compile these seven inflectional graphs.

<figure><img src="image_241.png" /></figure>

NC_WWX1

<footnote n="115">¹¹⁵ We thank Agata Savary for these three examples: *carry out*, *take off* and *attorney general*.</footnote>
<figure><img src="image_242.png" /></figure>

NC_XW1

<figure><img src="image_243.png" /></figure>

NC_WX1

<figure><img src="image_244.png" /></figure>

NC_WX2

<figure><img src="image_245.png" /></figure>

NC_WX3

<figure><img src="image_246.png" /></figure>

NC_WX4

<figure><img src="image_247.png" /></figure>

NC_XX1
We will now close all graphs using the *Graphs/Close all* option. Like before, we will open the *myDictionary.dic* file using the *DELA/Open Recent* menu and inflect it (DELA/Inflect menu). The final *myDictionaryflx.dic* dictionary consists of 216 entries. After transforming it (DELA/Compress into FST menu), we will close both *myDictionary.dic* and *myDictionaryflx.dic* windows. We will apply our dictionary to the text (*Text/Apply Lexical Resources*, clear all pre-selections and select *myDictionaryflx.dic*). The number of recognized MWUs listed in the *Word List* is now 21 (DLC).

# 4 Some additional remarks

## 4.1 Adding features in dictionary entries

Features with additional information can be added to Unitex dictionaries. In the Unitex user manual, these features are referred to as *semantic codes*. In a dictionary of lemmas these features follow an inflectional class (a name of an inflectional graph) and are preceded by the `+` character. Features that are added to the entries in a dictionary of lemmas are transferred to the entries of a dictionary of inflected forms.

Let us turn again to our dictionary of lemmas (*myDictionary.dic*) that we will modify by adding semantic codes: *abstract* for abstract nouns, *animal* for nouns denoting animal, *concrete* for concrete inanimate nouns and *human* for nouns denoting human.

advantage,N1+abstract
agent,N1+human
firm,N1+concrete
...
wash out(out.N1:s),NC_WX4+abstract
notary(notary.N3:s) public(public.N1:s),NC_XX1+human
attorney(attorney.N1:s) general(general.N1:s),NC_XX1+human

If you add one of these semantic codes to all noun entries in the lemma dictionary, then our form dictionary (*myDictionaryflx.dic*) will have 133 semantically described entries.

Anglo-Norman,Anglo-Norman.N+human:s
Anglo-Normans,Anglo-Norman.N+human:p
...
advantage,advantage.N+abstract:s
advantages,advantage.N+abstract:p
after-life,after-life.N+abstract:s
after-lifes,after-life.N+abstract:p
afterlife,after-life.N+abstract:s
afterlifes,after-life.N+abstract:p
agent,agent.N+human:s
agents,agent.N+human:p
...

These semantic codes can be used to refine a lexical mask, for instance by writing `<N+abstract>`.

## 4.2 Direct creation of inflected entries

It is possible to add entries directly into the *myDictionaryflx.dic* dictionary, which can be useful, for instance, for words that do not inflect. However, one has to be careful because if
the `myDictionaryflx dik` dictionary is recreated from `myDictionary.dic`, these manually added entries would be lost. It is better thus to have a new inflectional graph, named N13.grf, describing nouns that do not change in plural, like `moose`, `sheep` or `aircraft`.

moose
sheep
aircraft

<figure><img src="image_248.png" /></figure>

...moose, N13+animal
sheep, N13+animal
aircraft, N13+concrete

On the other hand, one can directly create a dictionary of inflected forms, which can be useful, for instance, for a dictionary of proper names that usually do not inflect in number. For instance, a `myProperNameDictionary.dic` dictionary can be created directly containing the following entries, collected on the text as you go.

Dombey, N+surname:ms:fs
Paul, N+forename:ms
Florence, N+forename:fs
Peps, N+surname:ms:fs

This dictionary can then be sorted by the *Dela/Sort Dictionary* menu.

Dombey, N+surname:ms:fs
Florence, N+forename:fs
Paul, N+forename:ms
Peps, N+surname:ms:fs

After transforming it (`DELA/Compress into FST` menu), we will close the `myProperNameDictionary.dic` window. We will apply our two dictionaries to the text (`Text/Apply Lexical Resources`, clear all pre-selections and select `myDictionaryflx.dic` and `myProperNameDictionary.dic` by CTRL-clicking the mouse button). The number of recognized simple word listed in the *Word List* is now 108 (DLF).
<figure><img src="image_249.png" /></figure>

<figure><img src="image_250.png" /></figure>

## 4.3 Adding comments

### 4.3.1 Adding comments to entries

It is possible to add comments to entries in a dictionary of inflected forms. They are placed at the end of a line, after a slash “/”. For instance, we can supply some explanatory information in the *myProperNameDictionary.dic* dictionary. This information does not appear in the *Word List*.

- Dombey, N+surname:ms:fs

- Florence, N+forename:fs/the daughter of Mr Dombey

- Paul, N+forename:ms/the son of Mr Dombey

- Peps, N+surname:ms:fs/the doctor

It is also possible to add comments to the dictionary of lemmas, but these comments are not transferred to the inflected dictionary. It can be sometimes important to memorize who added an entry and when. If we use instead of *myDictionary.dic* the following dictionary of lemmas with added comments, we would still obtain the same *flx* dictionary. Let us modify, for example, the first three lines, then inflect the dictionary. The comments are not in the dictionary of inflected forms.

- advantage, N1+abstract/ Cvetana, 2022/07/12

- agent, N1+human/ Denis, 2022/07/13

- firm, N1+concrete/ Cvetana, 2022/07/14

- ...

<figure><img src="image_251.png" /></figure>
### 4.3.2 Documenting dictionaries

Finally, we can provide a description of our two dictionaries in two files, `myDictionaryflx.txt` and `myProperNameDictionary.txt`, which placed in the same folder as dictionaries themselves. For instance (for both):

Unitex Getting Started
Denis Maurel
University of Tours
Cvetana Krstev
University of Belgrade

This information can be displayed by accessing the `Text/ApplyLexical Resources` menu and right-clicking the name of the dictionary.

## 4.4 Automatically verifying the format of a dictionary

### 4.4.1 Inflected dictionary

After adding new entries to a dictionary, we can verify if it is well formed, that is, in the format expected by Unitex. Let us take, for example, the `myDictionaryflx dik` dictionary and open it using the `DELA/Open Recent` menu. If we select the `DELA/Check Format` menu and click the `Check Dictionary` button, the file named `CHECK_DIK.TXT` will appear in the `DELA` folder and its content will be displayed in a window.

The content of this file in the case of our dictionary will be:

1. Quantitative information: 219 lines read, 159 simple entries for 61 distinct lemmas, 60 compound entries for 23 distinct lemmas.

2. The list of used characters and their UTF8 code.

3. The used grammatical and semantic codes: 6 grammatical/semantic codes used in dictionary: N, human, V, abstract, concrete and animal.

4. The used morphological codes: 17 inflectional codes used in dictionary, s, p, P1p, P1s...

However, if we have not manually added anything to the `myDictionaryflx.dik` dictionary, that is, if it has been automatically produced from `myDictionary.dik`, there is no need to verify its format, since it must be correct.

### 4.4.2 Lemma dictionary

What we can do is to check the correctness of the dictionary of lemmas (`myDictionary.dik`). After opening it in the `DELA` menu and choosing `DELA/Check Format` we can verify its format by checking the `DELAS/DELAC` option.<fnref n="116" /> After applying this option to our dictionary of lemmas, the content of the `CHECK_DIK.TXT` file displayed on the screen will contain the similar information (and replace the previous one):

1. Quantitative information: 78 lines read, 55 simple entries for 55 distinct lemmas, 23 compound entries for 23 distinct lemmas.

2. The list of used characters and their UTF8 code.

3. The used grammatical and semantic codes: 32 grammatical/semantic codes used in dictionary: N1, abstract, human, concrete...

<footnote n="116">¹¹⁶ This option corresponds to dictionaries of lemmas, while the `DELAC/DELACF` option that is selected by default corresponds to dictionaries of inflected forms.</footnote>
4. The used morphological codes: *O inflectional codes used in dictionary*.

**4.5 Morphological-mode dictionary (only for confident users)**

Before starting this section

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted`

Create a new folder named *Dictionaries*.

Dictionary information can be inserted into a text by a graph, for example, to tag it. To do this, you must know how to insert the morphological mode (see *Chapter 3, Section 3*, page 63) and use variables (see *Chapter 4, Section 1.2*, page 79).

We will use the *myProperNameDictionary.dic* dictionary to markup proper names.

First (don't forget this point!), this dictionary must be allowed for use in the morphological mode: open the *Info/Preferences.../Morphological mode dictionaries* menu and click the *Add* button to select the *myProperNameDictionary.bin* dictionary.

We will also use the *myProperNameDictionaryTag.grf* graph below, saved in the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\Dictionaries`

$PN$ is a dictionary-entry variable, placed as output text. In the second output, $PN.CODE.SEM$ is the semantic code corresponding to the dictionary entry (*forename* or *surname*); $PN.INFLECTED$ is the inflected form corresponding to the dictionary entry (i.e. the proper noun itself).<fnref n="117" />

<figure><img src="image_252.png" /></figure>

This graph will produce, in *Replace* mode and with the *Text Order* option, the concordance with 3 493 proper names displayed below. If graphs that you will produce and use later no longer use this dictionary in morphological mode, remember to remove it from the list.

<figure><img src="image_253.png" /></figure>

**4.6 Priorities**

If we use multiple dictionaries, we can set up three priority levels. To do this, we will add the symbols - (maximum priority) or + (minimum priority) to the name of the dictionary file.<fnref n="118" />

<footnote n="117">¹¹⁷ See the Unitex user manual, Section 6.4.4, for more information on dictionary-entry variables.</footnote>

<footnote n="118">¹¹⁸ We have to change the name of two files (`.bin` and `.inf`). If it exists, we have to do the same for the `.txt` file.</footnote>
For instance, if we want to use our two dictionaries, `myDictionary.dic` and `myProperNameDictionary.dic`:

<table><tr><td>With the dictionary selection:</td><td>A word present in both dictionaries will be labeled</td></tr><tr><td>myDictionaryflx.bin<br>myProperNameDictionary.bin</td><td>by each dictionary</td></tr><tr><td>myDictionaryflx.bin<br>myProperNameDictionary+.bin</td><td>only by the myDictionary.bin dictionary</td></tr><tr><td>myDictionaryflx.bin<br>myProperNameDictionary-.bin</td><td>only by the myProperNameDictionary-.bin dictionary</td></tr></table>

## 4.7 Additional notes

The inflectional morphology of English is relatively simple. In French, common nouns inflect, not only in number, but some of them also in gender, and adjectives inflect too. In Serbian, nouns have gender, animacy, number and case, while all these categories inflect for adjectives, in addition to definiteness and comparison. All this can be modeled using the inflectional graphs introduced before.

When inflection also touches the prefix, then the more complex rules introduced in Section 6, page 132 must be used.

For Semitic languages, like Arabic, Hebrew and so on, in which inflection often operates between consonants in the root, specific rules were developed (see the Unitex user manual, Section 3.5.4).

# 5 Dictionary graphs

## 5.1 Example of Roman Numeral dictionary

You will recall that we have constructed the `RN2-3999.grf` graph to tag Roman numerals (see Chapter 3, Section 3.4.2, page 67). We now propose to create a dictionary graph for Roman numerals. We will name it `RomanNumeralDictionary.grf` and we will base it on the `RN2-3999.grf` graph. So, we can copy the four graphs: `RN1-9-subgraph.grf`, `RN1-99-subgraph.grf`, `RN1-999-subgraph.grf` and `RN2-3999.grf` from the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\RomanNumerals`

and paste them in the folder:

`MyUnitex\English\Dela`

then rename the `RN2-3999.grf` graph as `RomanNumeralDictionary.grf`.

We are going to remove the insertions of an opening tag `<romanNumeral>` and a closing tag `</romanNumeral>`; and we replace them with empty boxes (`<E>`). We will also replace the call to `:RN2-999` with a call to `:RN1-999-subgraph` and place this call inside morphological tags as well. We will add a new box at the end of the graph with the output `.RN`.
<figure><img src="image_254.png" /></figure>

We will compile this graph, open the *Text/Apply Lexical Resources* menu and click the *Clear* button (in order to cancel all previous selections). We will choose the *RomanNumeralDictionary.fst2* file (appearing in the left field) and click the *Apply* button.<fnref n="119" /> As a result the *Word List* window will be displayed.

<figure><img src="image_255.png" /></figure>

## 5.2 Morphological extension of the used dictionaries (only for confident users)

### 5.2.1 Morphological-mode dictionary declaration

It is also possible to extend the used dictionaries by creating morphological extensions of some entries. The root dictionary must be declared as a morphological-mode dictionary (don't forget this point!). See Section 4.5, page 127.

If we consider that the default English dictionary, *dela-en-public*, is the root dictionary, we have to open the *Info/Preferences.../Morphological mode dictionaries* menu and click the *Add* button to select the *dela-en-public.bin* file. This file is located in your installation folder (see Chapter 2, Section 1.1.2, page 7):

C:\Users\username\AppData\Local\Unitex-GramLab<fnref n="120" />

or

C:\Program Files (x86)\Unitex-GramLab\App

<footnote n="119">¹¹⁹ We can also create a text file to write a documentation of the dictionary graph, *RomanNumeralDictionary.txt*, see Section 4.3.2, page 126.</footnote>

<footnote n="120">¹²⁰ *username* is your name under the Windows system.</footnote>
### 5.2.2 Example of the -less suffix

Generally, a noun can generate an adjective with the addition of the -less suffix. For instance, in our novel some such adjectives occur: *remorseless* from *remorse*, *restless* from *rest*, *colourless* from *colour* and so on. Many of these adjectives are entries in the default English dictionary, *dela-en-public*, as we can see if we use the *Dela/Lookup...* menu (see *Chapter 4*, *Section 3*, page 88). However, in our novel, three adjectives derived from nouns with -less suffix occur that are not in this dictionary: *brotherless*, *nevyless*<fnref n="121" /> and *whiskerless*. The *Nless.grf* dictionary graph recognizes and tags them appropriately. In addition to the POS tag A, it adds two tags: *Derivative* to indicate that this is an input constructed by a graph and *Root=$n.LEMMA$* to specify the root.<fnref n="122" />

<figure><img src="image_256.png" /></figure>

This graph should be added in the list of resources used to analyze the text: we must save and compile it in the folder:

MyUnitex\English\Dela

It recognizes adjectives composed of a noun (in a morphological-mode dictionary, not necessarily present in the applied dictionaries) and the -less suffix. We will now apply lexical resources to our text, this time adding to preselected dictionaries the *NLess.fst2* dictionary graph.<fnref n="123" /> The number of simple-word lexical entries is now 22 674 (compared to 22 604 obtained before).<fnref n="124" />

<figure><img src="image_257.png" /></figure>

<footnote n="121">¹²¹ The word *nevy* is an old form of *nephew*.</footnote>

<footnote n="122">¹²² The dictionary-entry variables are explained in *Section 4.5*, page 127.</footnote>

<footnote n="123">¹²³ We can also create a text file to document the dictionary graph, *NLess.txt*, see *Section 4.3.2*, page 126.</footnote>

<footnote n="124">¹²⁴ See *Chapter 2, Section 1.2.3*, page 13.</footnote>
<figure><img src="image_258.png" /></figure>

We notice that our graph recognized two of the three afore-mentioned adjectives. It could not recognize the adjective *neveless* because the noun *nevy* itself is not in the applied dictionary.

This dictionary graph adds seventy entries to the *Word List*. We found two, what are the other sixty eight?

First, two errors, from the *b, N* entry.

bless,bless.A+Derivative+Root=b
Bless,bless.A+Derivative+Root=b

Second, four entries starting with an uppercase letter, although already present in lowercase.<fnref n="125" />

Homeless,homeless.A+Derivative+Root=home
Restless,restless.A+Derivative+Root=rest
Speechless,speechless.A+Derivative+Root=speech
Supperless,supperless.A+Derivative+Root=supper

Third, sixty four duplications of entries already present.

artless,.A
artless,artless.A+Derivative+Root=art
...
voiceless,.A
voiceless,voiceless.A+Derivative+Root=voice

We will now make a small addition to our graph. The *NlessCorrected.grf* dictionary graph only recognizes the two adjectives ending with *-less* that are not already in applied dictionaries (*brotherless* and *whiskerless*). It uses a negative right context (see *Chapter 3, Section 1.7.3*, page 55) and a `<DIC>` mask (see *Chapter 2, Section 2.4.3*, page 34).

<figure><img src="image_259.png" /></figure>

We will now apply lexical resources to our text, this time adding to preselected dictionaries the *NlessCorrected.fst2* dictionary graph.

<footnote n="125">¹²⁵ If the graph does not add a lemma (the output, .A instead of the output, $n.LEMMA$less.A), the graph creates four new lemmas, *Homeless*, *Restless*, *Speechless* and *Supperless*, which is best avoided.</footnote>
<figure><img src="image_260.png" /></figure>

The number of simple-word lexical entries is now 22 606 entries.

# 6 Additional possibilities (only for confident users)

## 6.1 A test on a suffix

By testing the last letter of a verb, we can join the `V2.grf` and `V3.grf` graphs into one graph, named `V101.grf`. In order to illustrate this, we will create the `myAdvancedDictionary dik` file containing the following six entries (remember that there should be no spaces, and that you should press the *Enter* key at the end of the last line). You will remember that the first three of these verbs were inflected with the `V2.grf` graph in our `myDictionary dik` dictionary, while the last three verbs were inflected with the `V3.grf` graph.

love,V101
live,V101
give,V101
accompany,V101
try,V101
worry,V101

In the new `V101.grf` graph, we will use tests on the verb suffixes and inflect all six verbs. The occurrence of `<e>` in an inflectional graph means that the test is made on the verb suffix; if this suffix is *e*, it is removed and the inflection proceeds further. For instance, in order to produce forms of the third person singular of the present tense:

* the `<e>` box reads the suffix *e*, removes it and the next box adds `es:P3s`

* the `<y>` box reads the suffix *y*, removes it and the following boxes write first *i* and then `es:P3s`

which yields, for instance, `lovees:P3s` and `accompanyies:P3s`.
<figure><img src="image_261.png" /></figure>

Now we can inflect the *myAdvancedDictionary.dic* dictionary by choosing the *DELA* menu and the *Inflect* option. The file containing the *myAdvancedDictionaryflx.dic* inflected dictionary will be created and displayed on the screen.

<figure><img src="image_262.png" /></figure>

## 6.2 Use of variables

### 6.2.1 Example of *goose*, *tooth*, *foot*

Let us look at these three nouns: *goose*, *tooth*, *foot*. With the *N9.grf* graph we can inflect the first two nouns, but not the third for which a new graph should be written in which the rule *LLLLeeRR* would be replaced by *LLLLeeR* (there is one consonant, instead of two at the end of the noun). There is another way to formulate the rules that is more complex but also more powerful. It is based on two variables, *£n* and *$n*, that memorize the longest and the shortest strings, respectively, that occur where they are placed in the pattern (put in angular brackets)
in a box of the graph. This pattern is matched with the input word, starting from the right end.<fnref n="126" /> Several such variables can be used and numbered from n=1 to n=9.

In the *N101.grf* graph the first box contains the pattern `<£1oo£2>` which means that the variable £2 will be assigned the longest string preceded by *oo* in the processed word (that is, *se*, *th*, *t*, respectively), then the variable £1 will be assigned the longest string before *oo* (that is, *g*, *t*, *f*, respectively). In the case of the singular, the word is rewritten, while for the plural *oo* is replaced by *ee* keeping everything before and after it as it was, which seems as the most natural way to express this rule.<fnref n="127" />

<figure><img src="image_263.png" /></figure>

### 6.2.2 Example of bring, fight, think

The use of variables can be further illustrated with the inflection of the verbs *bring, fight, think* for which the letters before *i* are suffixed by *ought* in the past tense and the past participle (*V102.grf* graph).

<figure><img src="image_264.png" /></figure>

<footnote n="126">¹²⁶ For detailed explanation of the meaning and use of these variables, see the Unitex user manual, Section 3.5.2.</footnote>

<footnote n="127">¹²⁷ Note that in the string £1oo£2, which is not inside angular brackets, variables £1 and £2 are not assigned values, but their previously assigned values are used.</footnote>
We will add to our `myAdvancedDictionary dik` dictionary the following six entries (without spaces and with clicking the Enter key at the end of the last line):

goose,N101
tooth, N101
foot, N101
bring,V102
fight,V102
think,V102

We will follow the preceding steps to inflect the *myAdvancedDictionary.dic* dictionary. The created file named *myAdvancedDictionaryflx.dic* will be displayed on the screen.

<figure><img src="image_265.png" /></figure>
<figure><img src="image_266.png" /></figure>
# 1 Lexicon-grammar

Before starting this section

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted`

Create a new folder named `LexiconGrammar` (without space).

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted\LexiconGrammar`

Create a new folder named `ParticleVerbs` (without space).

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted\LexiconGrammar\ParticleVerbs`

Download the `particleVerbTable.xlsx` file.

## 1.1 Example: particle verbs

We will click the `Text/Open...` menu to open the file:

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSon\DombeyAndSon.txt`

Then answer `Yes` to the question `Do you want to preprocess the text?` A window will open in which we will remove the first two checkmarks, leaving just the last one (`Apply All default Dictionaries`).

The aim of this section is to automatically transform a lexicon-grammar table into a graph. We will prepare in some spreadsheet program a table that describes how some verbs combine with a set of particles to obtain particle verbs. This table `particleVerbTable.xlsx` is in the folder:

`MyUnitex\English\Graphs\UnitexGettingStarted\LexiconGrammar\ParticleVerbs`

<table><thead><tr><th></th><th>A</th><th>B</th><th>C</th><th>D</th><th>E</th><th>F</th><th>G</th><th>H</th><th>I</th><th>J</th><th>K</th></tr></thead><tbody><tr><td>1</td><td colspan="11">Particle verbs</td></tr><tr><td>2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>3</td><td>break</td><td>+</td><td>-</td><td>+</td><td>+</td><td>+</td><td>+</td><td>off</td><td>on</td><td>out</td><td>up</td></tr><tr><td>4</td><td>carry</td><td>+</td><td>+</td><td>-</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>-</td></tr><tr><td>5</td><td>get</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td></tr><tr><td>6</td><td>give</td><td>+</td><td>-</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td></tr><tr><td>7</td><td>put</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td></tr><tr><td>8</td><td>speak</td><td>+</td><td>-</td><td>+</td><td>+</td><td>-</td><td>-</td><td>+</td><td>+</td><td>+</td><td>+</td></tr><tr><td>9</td><td>turn</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td></tr><tr><td>10</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

The A column in the table contains particle verbs. The plus sign in B column indicates that the verb can be used without particles (all verbs in our table can be used without a particle as well). The C to L columns signal particles with which a verb can be used. The plus sign in a cell indicates that the verb and the particle form a particle verb, while the minus sign means the opposite.

## 1.2 Transformation of the spreadsheet table

The table that we prepared in a spreadsheet program must be transformed into a Unitex lexicon-grammar table.

First, in the `particleVerbTable.xlsx` spreadsheet, verbs in A column must be enclosed in angular brackets so that they should match all forms of the listed verbs. B2 cell is empty because it is the header of the column about the use of the verbs without a particle; we will write here the symbol `<E>` that corresponds to an empty box in a graph. This file should consist only of table
data with only one header line. We will therefore select A2:K9 (the 2 to 9 lines and the A to K columns) and copy its content. We will now open the Unitex editor (the File Edition/New File menu) and paste the content from the table.<fnref n="128" />

We will save this file with the name *particleVerbTable.txt* in the same folder:

MyUnitex\English\Graphs\UnitexGettingStarted\LexiconGrammar\ParticleVerbs

<figure><img src="image_267.png" /></figure>

## 1.3 Parameterized graphs

We will write a graph that retrieves in a text the constructions described in the table and we will save it in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\LexiconGrammar\ParticleVerbs

using the name *particleVerbParameterized.grf*. This graph uses variables @A, @B, @C... in which the content of cells A, B, C... will be stored. If the variables contain a text it will be copied in the working graph produced from this parameterized graph. If the value of the variable is a plus or a minus sign, the box functions as a test: with +, the test is passed and the graph proceeds with what follows, while with - the path is interrupted. There is one more variable, @%, that contains the number of the line, which we will use for the name of subgraphs.

<figure><img src="image_268.png" /></figure>

## 1.4 Generation of graphs

We will make Unitex generate a graph from this parameterized graph and the table. This graph will invoke one subgraph for each line in the table. In order to do this, we must open the

<footnote n="128">¹²⁸ By using the Unitex editor, we are sure that the lines and columns are in the Tabulation/Line break format and that the text is in UTF-8.</footnote>
Lexicon-Grammar/Open menu and select the *particleVerbTable.txt* table. Next, we must use the Lexicon-Grammar/Compile to GRF... menu to open (by clicking the SET button in the first line) the *particleVerbParameterized.grf* parameterized graph. In the second line, we can choose the name of this graph and modify it to *particleVerb.grf*. In the third line, we will choose the previous graph name and change it to *particleVerb@%.grf* (remember that @% is the special variable containing the line number). Finally, we will click the Compile button.

<figure><img src="image_269.png" /></figure>

Now we can open the *particleVerb.grf* graph. It invokes seven subgraphs, one for each line in our table (except the header line) which are numbered 0001 to 0007.

<figure><img src="image_270.png" /></figure>

We can open the first subgraph by selecting it and clicking the right mouse-button and choosing the *Open subgraph* option. We see that this graph has six paths for six particles used by the verb *break* and one path for the verb without a particle.

<figure><img src="image_271.png" /></figure>

By using the produced *particleVerb.grf* graph in the *Text/Locate Pattern* menu we will get 1 983 concordance lines.
<figure><img src="image_272.png" /></figure>

## 1.5 Annotations

The parameterized graph can be more complex than the previous example. For instance, the *particleVerbAnnotationParameterized.grf* graph annotates the recognized verbs (with particles or not).

<figure><img src="image_273.png" /></figure>

After repeating the process of graph generation, and its application to the text in the *Merge* mode, we will obtain a new concordance.

<figure><img src="image_274.png" /></figure>
# 2 XAlign

## 2.1 Installation

The aim of this software that is integrated into Unitex is to align two similar texts, for instance two drafts of the same document or a text and its translation. We will demonstrate how to use this software with the example of a text and its translation.

It is important to note that in order to reproduce the exercises from this chapter one should have downloaded the French resources for Unitex. If that is not the case, one should repeat the process of the installation of Unitex. However, it is not necessary to repeat the whole installation.

<figure><img src="image_275.png" /></figure>

One should only add a language.

<figure><img src="image_276.png" /></figure>

## 2.2 The French folder

After automatically creating the *French* folder:

* if Unitex is closed, we should open it by choosing *French* as the working language;

* if it is already open, we will choose *French* as the working language by using the *Text/Change Language...* menu.

It is of course possible to create the French folder yourself, before launching Unitex. But, in this case, Unitex will not copy the three files necessary for its operation (*Norm.txt*,
Alphabet.txt and Alphabet_sort.txt).<fnref n="129" /> You must do this yourself before running Unitex (or after closing and restarting it).

Before proceeding with this section

1. Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted`
Create a new folder named `DombeyAndSonEng` (without space).
Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng`
Download the `DombeyAndSonEng.txt` file.

2. Go to the folder: `MyUnitex\French\Corpus\UnitexGettingStarted`
Create a new folder named `DombeyAndSonFra` (without space).
Go to the folder: `MyUnitex\French\Corpus\UnitexGettingStarted\DombeyAndSonFra`
Download the `DombeyAndSonFra.txt` file.

3. Go to the folder: `MyUnitex`
Create a new folder named `XAlign` (without space).
Go to the folder: `MyUnitex\XAlign`
Create two new folders named `Corpus` and `Sentence`.
Go to the folder: `MyUnitex\XAlign\Corpus\UnitexGettingStarted`
Create a new folder named `DombeyAndSonEngFra`.

## 2.3 File preparation

We will use files from three folders: `English`, `French` and `XAlign`. We will choose `English` as the working language by using the `Text/Change Language...` menu.

### 2.3.1 English folder

We will open the file (`File Edition/Open/Text files` menu):<fnref n="130" />

`MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng\DombeyAndSonEng.txt`

We will ignore the message *This is not necessarily the text being processed by Unitex*. We will save this file using the name `DombeyAndSonEng.xml` (we will change from the `.txt` format to `.xml` format).

We will add TEI tags at the beginning and at the end of this file:<fnref n="131" />

```xml
<tei>
<teiHeader/>
<body>
<text>
<div>
Dombey and Son by Charles Dickens
...
A transient flush of faint surprise overspread the sick lady's face as she raised her eyes towards
him.
</div>
</text>
</body>
</tei>
```

<footnote n="129">¹²⁹ About the *Norm.txt* file, see *Chapter 2, Section 3.1.2*, page 35; about the *Alphabet.txt* and *Alphabet_sort.txt* files, see *Chapter 2, Section 3.3.3*, page 41.</footnote>

<footnote n="130">¹³⁰ We produced these samples of text in English and French just in order to illustrate the alignment process.</footnote>

<footnote n="131">¹³¹ `XAlign` does not handle correctly files with a TEI header, since it will delete it and replace it with the empty tag `<teiHeader/>`. If we want that our result files have a header, we can add it after the alignment.</footnote>
We will save this file and close the editor.

### 2.3.2 French folder

Now we will repeat the process for the *French* version of the text: we will open the file (*File Edition/Open/Text files* menu):

MyUnitex\French\Corpus\UnitexGettingStarted\DombeyAndSonFra\DombeyAndSonFra.txt

We will save this file using the name `DombeyAndSonFra.xml`. Finally, we will add the same tags at the beginning and the end of this file.

```xml
<tei>
  <teiHeader/>
  <body>
    <text>
      <div>
        Dombey et fils traduit de Charles Dickens
        ...
        Une rougeur passagère, causée par la surprise, colora légèrement les joues de la malade qui leva les yeux vers son mari.
      </div>
    </text>
  </body>
</tei>
```

We will save this file and close the editor.

### 2.3.3 XAlign folder

We will now open a new file using the editor `File Edition/New File`. In this file, we will copy the following lines:

```xml
<tei>
  <teiHeader/>
  <tei>
```

We will save this file (Save text button), using the name `DombeyAndSonEngFra.xml`, in the folder:

MyUnitex\XAlign\Corpus\UnitexGettingStarted\DombeyAndSonEngFra

During the alignment XAlign uses a file for sentence splitting for the source language (which is, in our case, *English*); we will use the graph that comes with the Unitex distribution. We will copy the `Sentence.fst2` file from the folder

MyUnitex\English\Graphs\Preprocessing\Sentence

And paste it in the folder:

MyUnitex\XAlign\Sentence

Once this is done, we will rename this file into `SentenceXAlign.fst2`.

## 2.4 Automatic alignment

We will click the `Text/Open...` menu to open the file:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng\DombeyAndSonEng.txt
and answer **Yes** to the question *Do you want to preprocess the text*. A window will open in which we will remove the first two checkmarks, leaving just the last one *Apply All default Dictionaries*.

We will open the Xalign/Open Files menu and fill the first line (by selecting the Set button) with the file:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng\DombeyAndSonEng.txt

And we fill the second line with the file:

MyUnitex\French\Corpus\UnitexGettingStarted\DombeyAndSonFra\DombeyAndSonFra.txt

<figure><img src="image_277.png" /></figure>

We will click the **OK** button. A window will ask us to choose the XML file that corresponds to the text file in the first line (**source file**). After clicking **OK**, we will choose the first of the previously developed files:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng\DombeyAndSonEng.xml

Again, another window will open asking us to select an XML file corresponding to the text file given in the second line (**target file**). We will choose the second previously developed file:

MyUnitex\French\Corpus\UnitexGettingStarted\DombeyAndSonFra\DombeyAndSonFra.xml

The new window will open:

<figure><img src="image_278.png" /></figure>

We will click the **Align** button that will initiate the alignment process. A new window will ask us to select the XML file for the result. We will choose the third and the last file that we created before, namely the file:

MyUnitex\XAlign\Corpus\UnitexGettingStarted\DombeyAndSonEngFra\DombeyAndSonEngFra.xml
After the alignment process is finished, some error messages will be displayed, which we will ignore, and click the OK button

<figure><img src="image_279.png" /></figure>

Click the *Save alignment* button.

You should note that if you close Unitex after saving the alignment, you can open them again by filling all three lines with the appropriate XML files.

<figure><img src="image_280.png" /></figure>

## 2.5 Reading and manual correction

We have done the first alignment. It is necessary now to read and correct it. It is enough to click the *English* sentence (or phrase) and then the *French* sentence (or phrase) to establish the link, if it does not exist, or to remove it if it exists.<fnref n="132" />

<footnote n="132">¹³² In order to remove the link it is enough to simply click on it.</footnote>
remove 4-6;
add 4-5;
remove 5-7;
add 5-6;
remove 6-8;
remove 6-9;
add 6-7;
add 7-8;
add 8-14;
remove 9-14;
add 9-15;
add 10-16;
remove 11-16;
remove 11-17;
add 12-15.

Click the *Save alignment* button.

Generally, a sentence is translated into another sentence: (0-0), (5-6), (6-7) (9-15) and (13-18).
However, there are exceptions:

* three sentences in the source file, 1, 2 and 10, are translated into two sentences: (1-1, 1-2), (2-3, 2-4) and (10-15, 10-16);

* three sentences in the source file, 7, 8 and 12, are translated into three sentences: (7-8, 7-10, 7-11), (8-12, 8-13, 8-14) and (12-15, 12-16, 12-17);

* a sentence in the target file, 9, is added;

* a sentence in the source file, 11, is not translated;

* two sentences in the source file, 3 and 4, are translated together into a single sentence: (3-5, 4-5);

* two sentences in the source file, 10 and 12, are reversed: (10-16, 12-15).

## 2.6 Search performed on one of two texts

In XAlign we can use Unitex search functions via *Text/Locate Pattern* and get as a result the visualized alignment. For instance, we can be interested in whether the name of Mr Dombey was systematically represented in the French translation. We will click the *Locate...* button, that can be found on the left side of the *XAlign* window, and then answer *Yes* to the question that appears. Unitex proposes to open a text deduced from the XML file, text which it names with the suffix `_xalign` (*DombeyAndSonEng_xalign.txt*).

We will type *Dombey* in the *Regular expression* line and click the *SEARCH* button. The `_xalign` file will be processed by Unitex and the result will appear in the *XAlign* window, after checking the *All sentences/HTML* option (at the left-hand side). If the *Matched sentences* option is chosen only the sentences that correspond to our search query will be displayed.
<figure><img src="image_281.png" /></figure>

In the obtained results we can see that, in the translation (12-16), the proper name was translated by *son épouse* (*his wife*).

We will choose the *All sentences/Plain text* option at the left side of the *XAlign* window in order to start a new search. This time we will use a graph, for instance the *willV.grf* graph that we developed in *Chapter 3, Section 1.8*, page 56.<fnref n="133" /> We will search for this graph (*Set* button) in the *Graph* line and click the *SEARCH* button. We will select the *Matched sentences* option and we will get only one match in the source file sentence 7.

The same options exist on the right of the *XAlign* window. It is therefore possible to search in the target language.

However, it is not possible to launch a cascade on an aligned text, except if we proceed as explained in the next section (only for confident users).

## 2.7 Cascade search and XAlign (only for confident users)

The main idea is to launch cascades on the *DombeyAndSonEng.xml* file without touching the `<p>` and `
` tags placed by XAlign.

### 2.7.1 Launching cascades

We will go to the folder:

MyUnitex\English\Corpus\UnitexGettingStarted\DombeyAndSonEng

and rename the *DombeyAndSonEng.xml* file as *DombeyAndSonEng.xml.txt* (see *Chapter 4, Section 2*, page 84). We will open this text (*Text/Open* menu) and answer *Yes* to the question *Do you want to preprocess the text*. A window will open in which we will remove the first two checkmarks, leaving just the last one *Apply All default Dictionaries*. We will find the *analysis* cascade (*Apply CasSys Cascade...* menu) in the folder:

MyUnitex\English\CasSys\UnitexGettingStarted\Measures

And we will launch this cascade. The result file is the *DombeyAndSonEng.xml_csc.txt* file. We will open this text with Unitex, without processing it, and launch the *synthesis* cascade. This second result file is the *DombeyAndSonEng.xml_csc_csc.txt* file with two recognized measures (sentences 4 and 5).

<footnote n="133">¹³³ Or *Chapter 3, Section 1.9.3*, page 58, if you have done this section.</footnote>
<s id="n4" xml:id="d1p1s4">Dombey was about <Measure type="Duration"><Number>eight-and-forty</Number> years</Measure> of age. 
<s id="n5" xml:id="d1p1s5">Son about <Measure type="Duration"><Number>eight-and-forty</Number> minutes</Measure>. 


2.7.2 File transformation

We will rename the *DombeyAndSonEng.xml_csc_csc.txt* file as *DombeyAndSonEng.xml*. If we open this file with XAlign, the tags disappear. To keep them, we will use the HTML codes `&lt;` for `<` and `&gt;` for `>`. We will pay attention to modify only our tags (*Measure* and *Number*) and not the `<p>` and `
` tags produced by XAlign.

<s id="n4" xml:id="d1p1s4">Dombey was about &lt;Measure type="Duration"&gt;&lt;Number&gt;eight-and-forty&lt;/Number&gt; years&lt;/Measure&gt; of age. 
<s id="n5" xml:id="d1p1s5">Son about &lt;Measure type="Duration"&gt;&lt;Number&gt;eight-and-forty&lt;/Number&gt; minutes&lt;/Measure&gt;.


We will save this file.

2.7.3 Using XAlign

If we open XAlign with this new file:

<figure><img src="image_282.png" /></figure>

We obtain the desired result.

<figure><img src="image_283.png" /></figure>
# Chapter 7: scripts

Before reading this chapter, please, read this caution:

This chapter requires some computer knowledge.

You must at least be comfortable with the notion of scripts.

Before starting this chapter:<fnref n="134" />

1. Go to the folder: `MyUnitex`
   Create a new folder named `Scripts`.
   Go to the folder: `MyUnitex\Scripts`
   Create a new folder named `ScriptEntities` (without space).
   Go to the folder: `MyUnitex\Scripts\ScriptEntities`
   Create three new folders named `Input`, `Output` and `ScriptEntitiesLingpkg`.
   Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg`
   Create two new folders named `Resources` and `Scripts`.

2. Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext`
   Copy the `jubileePlaintext.txt` file.
   Go to the folder: `MyUnitex\Scripts\ScriptEntities\Input`
   Paste this file.

3. Go to the folder: `MyUnitex\English\Corpus\UnitexGettingStarted\JubileeXmlText`
   Copy the `jubileeXmlText.xml` file.
   Go to the folder: `MyUnitex\Scripts\ScriptEntities\Input`
   Paste this file.

## 1 Introduction

The Unitex interface is a tool for developing graphs and graph cascades. Once these resources have been developed and tested, it is possible to create Unitex scripts which apply the developed tools on a corpus of several documents stored in several files. These Unitex scripts are launched from a command line and they access a compressed file that can contain all types of resources: Unitex scripts, graphs, cascades and dictionaries. We will take as an example the cascades developed in *Chapter 4, Section 1*, page 71 and *Section 2*, page 84. If you have allowed a dictionary for use in the morphological mode, please, remove them now from the list (see *Chapter 5, Section 4.5*, page 127).

In general, users will copy in the `Input` folder all texts they want to process, as we have done.

## 2 Creation of the linguistic package

The linguistic package (`scriptEntitiesLingpkg` folder) will contain everything needed to launch the analysis and synthesis cascades.

<footnote n="134">¹³⁴ Remember that file names should not contain spaces and are case sensitive. The same goes for folder names. See *Chapter 2, Section 1.1.1*, page 7.</footnote>
## 2.1 The Resources folder

We will replicate our personal working folder in the Resources folder.

1. Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources`
Create a new folder named `English`.

Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English`
Create three new folders named `CasSys`, `Dela` and `Graphs`.

2. Go to the folder: `MyUnitex\English`

Copy the three `Alphabet.txt`, `Alphabet_sort.txt` and `Norm.txt` files.

Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English`
Paste these files.

In general, one will copy in the CasSys folder the cascades that one wants to use for the processing.

Go to the folder: `MyUnitex\English\CasSys\UnitexGettingStarted\XmlTextEntities`

Copy the two `analysis.csc` and `synthesis.csc` files.

Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\CasSys`

Paste these files.

In general, users copy the graphs used by cascades in the Graphs folder. In fact, it is enough to copy the compiled files (`.fst2`). However, the presence of source graph files (`.grf`) makes it possible to use the language package as a backup of one's work.

1. Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\Graphs`

Create a new folder named `UnitexGettingStarted`.

2. Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted`

Copy the folder `Entities`.

Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\Graphs\UnitexGettingStarted`

Paste this folder.

Neither of the cascades use dictionaries, so we do not need to copy anything to the folder:

`MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resource\English\Dela`

Otherwise, if the cascades used dictionaries, see Section 2.4.1, page 155.

## 2.2 The scripts folder

### 2.2.1 Using the console

The scripts folder will contain the Unitex scripts. The scripts can be developed manually, but the Unitex console will help us.

If Unitex is already open, we need to close and reopen it in order to clean the console. After that, everything we do with the menu will be recorded by the console. We will launch our two cascades.

1. We will use the `Text/Open` menu (answering `No` to the question `Do you want to preprocess the text`) to open the file:

`MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlainText\jubileezb PlainText.txt`
2. We will use the *Text/ApplyCasSysCascade...* menu, then subsequently double-click the *UnitexGettingStarted* and *XmlTextEntities* folders to open them; we will select the *analysis.csc* cascade and click the *Launch* button (do not build the concordance).<fnref n="135" />

3. We will use the *Text/Open* menu (answering *No* to the question *Do you want to preprocess the text*) to open the file:

MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlainText\JubileePlainText_csc.txt

4. We will use the *Text/ApplyCasSysCascade...* menu, then select the *synthesis.csc* cascade and click the *Launch* button (do not build the concordance).

5. We will use the *Info/Console* menu and copy its content (*CTRL-C*).

6. We will use the *File Edition/New File* menu, then paste the commands recorded in the console.

7. We will use the *Save text* button and save this file with the name *scriptEntities.uniscript*, placed in the folder:

MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Scripts

<footnote n="135">¹³⁵ We will use the *XmlTextEntities* cascade, as we will later parse also the *XmlTextEntities.xml* file.</footnote>
This file contains eight commands, which look like this:<fnref n="136" />

```
mkdir "C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_snt"
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" Normalize
"C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext.txt"
"-rC:\Documents\MyUnitex\English\Norm.txt" "--output offsets=C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_snt\normalize.out offsets" -qutf8-no-bom
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" Tokenize
"C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext.snt"
"-aC:\Documents\MyUnitex\English\Alphabet.txt" -qutf8-no-bom
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" CasSys
"-aC:\Documents\MyUnitex\English\Alphabet.txt"
"-tC:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext.snt"
"-IC:\Documents\MyUnitex\English\CasSys\UnitexGettingStarted\XmlTextEntities\analysis.csc" -v
"-rC:\Documents\MyUnitex\English\Graphs\"
"--
input offsets=C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_snt\normalize.out offsets" -qutf8-no-bom
mkdir "C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_csc_snt"
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" Normalize
"C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_csc.txt"
"-rC:\Documents\MyUnitex\English\Norm.txt"
"--
output offsets=C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_csc_snt\normalize.out offsets" -qutf8-no-bom
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" Tokenize
"C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_csc.snt"
"-aC:\Documents\MyUnitex\English\Alphabet.txt" -qutf8-no-bom
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" CasSys
"-aC:\Documents\MyUnitex\English\Alphabet.txt"
"-tC:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_csc.snt"
"-IC:\Documents\MyUnitex\English\CasSys\UnitexGettingStarted\XmlTextEntities\synthesis.csc" -v
"-rC:\Documents\MyUnitex\English\Graphs\"
"--
input offsets=C:\Documents\MyUnitex\English\Corpus\UnitexGettingStarted\JubileePlaintext\JubileePlaintext_csc_snt\normalize.out offsets" -qutf8-no-bom
```

<footnote n="136">136 Users can obtain a different content if the location and names of the folders in question differ. Do not forget this when preparing a script.</footnote>
### 2.2.2 File adjustment

We must adjust this file because of differences between the operating systems Windows, MacOS and Unix:

1. We will remove the 56 quotation marks;

2. We will replace the 148 backslashes (\) with slashes (/).<fnref n="137" />

3. We will add two lines at the beginning of the script to create a virtual working folder where the files will be processed one after the other (`CURRENT WORK DIR`). The virtual folder and its virtual files will not be stored on the disk, but in random-access memory, which will speed up the processing.

```
CURRENT WORK DIR = {CORPUS WORK DIR}/{UNIQUE_VALUE}
DuplicateFile -p {CURRENT WORK DIR}
```

4. We will delete the two lines starting with `mkdir` command and replace them by these two lines at the beginning of the script, after the previous two inserted lines.

```
DuplicateFile --make-dir {CURRENT WORK DIR}/corpus_snt
DuplicateFile --make-dir {CURRENT WORK DIR}/corpus_csc_snt
```

### 2.2.3 Script generalization

In order to make this script more general, we will transform this sequence of commands in the following way:

1. We will add a line (it will become the fifth line) to process a file from our folder.

```
DuplicateFile -i {INPUT_FILE_1} {CURRENT WORK DIR}/corpus.txt
```

2. We will replace the ten occurrences of the beginning of the text file name:

`C:/Documents/MyUnitex/English/Corpus/JubileePlaintext/jubileePlaintext`

with:

`{CURRENT WORK DIR}/corpus`

3. We will add a line (the penultimate of the script) to save the result file.

```
DuplicateFile -i {CURRENT WORK DIR}/corpus_csc_csc.txt {OUTPUT_FILE_1}
```

4. We will remove the six calls to the `UnitexToolLogger` program and the path that leads to it:

`C:/Program Files (x86)/Unitex-GramLab/App/UnitexToolLogger.exe`

5. We will replace the ten paths that lead to our personal Unitex working folder:

`C:/Documents/MyUnitex`

with the folder of resources of the package

<footnote n="137">¹³⁷ This is due to the inversion of these symbols between Windows, on the one hand, and MacOS or Unix, on the other. If you are not using Windows, you can therefore ignore this step. You may have a different number of backslashes if the hierarchy of folders on your computer is different.</footnote>
`{PACKAGE_DIR}/Resources`

6. We will modify the two paths leading to our cascades (the two *Cassys* lines) by removing (before *analysis.csc* and *synthesis.csc*):

UnitexGettingStarted/XmITextEntities/

7. We will add a line at the end of the script, to free up some space before processing the next file in the input folder.

`DuplicateFile --recursive-delete {CURRENT WORK DIR}`

Our script file (thirteen lines) will now look like this:

```
CURRENT WORK DIR = {CORPUS WORK DIR}/{UNIQUE_VALUE}
DuplicateFile -p {CURRENT WORK DIR}
DuplicateFile --make-dir {CURRENT WORK DIR}/corpus_snt
DuplicateFile --make-dir {CURRENT WORK DIR}/corpus_csc_snt
DuplicateFile -i {INPUT_FILE_1} {CURRENT WORK DIR}/corpus.txt
Normalize {CURRENT WORK DIR}/corpus.txt -r{PACKAGE_DIR}/Resources/English/Norm.txt --
output offsets={CURRENT WORK DIR}/corpus_snt/normalize.out offsets -qutf8-no-bom
Tokenize {CURRENT WORK DIR}/corpus.snt -a{PACKAGE_DIR}/Resources/English/Alphabet.txt -
qutf8-no-bom
Cassys -a{PACKAGE_DIR}/Resources/English/Alphabet.txt -t{CURRENT WORK DIR}/corpus.snt -
I{PACKAGE_DIR}/Resources/English/CasSys/analysis.csc -v -
r{PACKAGE_DIR}/Resources/English/Graphs/ --
input offsets={CURRENT WORK DIR}/corpus_snt/normalize.out offsets -qutf8-no-bom
Normalize {CURRENT WORK DIR}/corpus_csc.txt -r{PACKAGE_DIR}/Resources/English/Norm.txt --
output offsets={CURRENT WORK DIR}/corpus_csc_snt/normalize.out offsets -qutf8-no-bom
Tokenize {CURRENT WORK DIR}/corpus_csc.snt -
a{PACKAGE_DIR}/Resources/English/Alphabet.txt -qutf8-no-bom
Cassys -a{PACKAGE_DIR}/Resources/English/Alphabet.txt -
t{CURRENT WORK DIR}/corpus_csc.snt -
I{PACKAGE_DIR}/Resources/English/CasSys/synthesis.csc -v -
r{PACKAGE_DIR}/Resources/English/Graphs/ --
input offsets={CURRENT WORK DIR}/corpus_csc_snt/normalize.out offsets -qutf8-no-bom
DuplicateFile -i {CURRENT WORK DIR}/corpus_csc_csc.txt {OUTPUT_FILE_1}
DuplicateFile --recursive-delete {CURRENT WORK DIR}
```

This script successively launches two cascades on a file and will produce the final result without writing on the disk numerous intermediate files, which are processed in RAM instead. It is the *DuplicateFile* command which allows access to a virtual folder and its virtual files.

### 2.2.4 Final touches

Since the *corpus.txt* file has already been normalized, the file resulting from the first cascade, *corpus_csc.txt*, is also normalized. Therefore, it is not necessary to do it again, although it would not be wrong. Since the second cascade uses the *corpus_csc.snt* file, we must rename the *corpus_csc.txt* file produced by the first cascade.
We are therefore going to replace the second *Normalize* command which follows the first *CasSys* command, on line 9, by:

```
DuplicateFile -i {CURRENT WORK DIR}/corpus_csc.txt {CURRENT WORK DIR}/corpus_csc.snt
```

In this case, the offset file is that of the first, and now only, *Normalize* command.

```
Normalize {PACKAGE_DIR}/Resources/corpus.txt
-r{PACKAGE_DIR}/Resources/English/Norm.txt
--output offsets={CURRENT WORK DIR}/corpus_snt/normalize.out offsets
-qutf8-no-bom
```

We need to replace the *input offsets* option of the second *CasSys* command with the *output offsets* option of the *Normalize* command.

```
CasSys -a{PACKAGE_DIR}/Resources/English/Alphabet.txt
-t{PACKAGE_DIR}/Resources/corpus_csc.snt
-l{PACKAGE_DIR}/Resources/English/CasSys/synthesis.csc -v
-r{PACKAGE_DIR}/Resources/English/Graphs/
--input offsets={CURRENT WORK DIR}/corpus_snt/normalize.out offsets
-qutf8-no-bom
```

With these modifications we obtain the last version of our script.

## 2.3 Compression

The *scriptEntitiesLingpkg* folder is now complete. The only thing that remains to be done is to create the compressed file. In order to do that we have to open this folder, select the two subfolders, named *Resources* and *Scripts*, compress them together and name the compressed *scriptEntitiesLingpkg.zip* file.<fnref n="138" /> We will cut this compressed file and paste it in the *MyUnitex\Scripts\ScriptEntities* folder.

## 2.4 Another possibility (only for confident users)

### 2.4.1 The use of dictionaries

#### 2.4.1.1 The default English dictionary

If the cascades use the default English dictionary, *dela-en-public*:

Go to the folder where Unitex is installed:<fnref n="139" />
C:\Users\username\AppData\Local\Unitex\GramLab
or
C:\Program Files (x86)\Unitex\GramLab\App
Copy the three files named *dela-en-public.bin*, *dela-en-public.inf* and *dela-en-public.txt*.
Go to the folder: *MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\Dela*
Paste these files.

We must add to our script five new lines between the first *Tokenize* command and the first *CasSys* command. The first one for the *Dico* command and the other four for the *SortTxt* commands that create four files *dlf.n*, *dlc.n*, *err.n* and *tags_err.n*.

<footnote n="138">¹³⁸ It must be a .zip file, .rar files are not accepted.</footnote>

<footnote n="139">¹³⁹ See Chapter 2, Section 1.1.2, page 7.</footnote>
Dico -t{CURRENT WORK_DIR}/corpus.snt -a{PACKAGE_DIR}/Resources/English/Alphabet.txt
{PACKAGE_DIR}/Resources/English/Dela/dela-en-public.bin
SortTxt {CURRENT WORK_DIR}/corpus_snt/dlf -l{CURRENT WORK_DIR}/corpus_snt/dlf.n -
o{PACKAGE_DIR}/Resources/English/Alphabet_sort.txt
SortTxt {CURRENT WORK_DIR}/corpus_snt/dlc -l{CURRENT WORK_DIR}/corpus_snt/dlc.n -
o{PACKAGE_DIR}/Resources/English/Alphabet_sort.txt
SortTxt {CURRENT WORK_DIR}/corpus_snt/err -l{CURRENT WORK_DIR}/corpus_snt/err.n -
o{PACKAGE_DIR}/Resources/English/Alphabet_sort.txt
SortTxt {CURRENT WORK_DIR}/corpus_snt/tags_err -
I{CURRENT WORK_DIR}/corpus_snt/tags_err.n -
o{PACKAGE_DIR}/Resources/English/Alphabet_sort.txt

### 2.4.1.2 A private dictionary

If the cascades use a private dictionary of forms, for example the *myProperNameDictionary.dic* dictionary, created in *Chapter 5, Section 4.2*, page 123.

Go to the folder `MyUnitex\English\Dela`
Copy the two files named `myProperNameDictionary.bin` and `myProperNameDictionary.inf`.
Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\Dela`
Paste these files.<fnref n="140" />

Just the *Dico* command changes.

Dico -t{CURRENT WORK_DIR}/corpus.snt -a{PACKAGE_DIR}/Resources/English/Alphabet.txt
{PACKAGE_DIR}/Resources/English/Dela/myProperNameDictionary.bin

### 2.4.1.3 Several dictionaries

If the cascades use both the default English dictionary and the private dictionary, the both dictionaries are placed in the same line.

Dico -t{CURRENT WORK_DIR}/corpus.snt -a{PACKAGE_DIR}/Resources/English/Alphabet.txt
{PACKAGE_DIR}/Resources/English/Dela/dela-en-public.bin {PACKAGE_DIR}/Resources/English/
Dela/myProperNameDictionary.bin

### 2.4.1.4 A morphological dictionary

It is also possible to allow the use of dictionaries in the morphological mode (see *Chapter 5, Section 4.5*, page 127). For example, if the private dictionary used in the previous section, *myProperNameDictionary.dic*, is to be declared also as a morphological dictionary, its name must be written a second time, preceded by the `-m` prefix.

Dico -t{CURRENT WORK_DIR}/corpus.snt -a{PACKAGE_DIR}/Resources/English/Alphabet.txt
{PACKAGE_DIR}/Resources/English/Dela/dela-en-public.bin {PACKAGE_DIR}/Resources/English/
Dela/myProperNameDictionary.bin -m{PACKAGE_DIR}/Resources/English/Dela/
myProperNameDictionary.bin

In any case, if the user's cascades use dictionaries, the user can always repeat the procedure described in *Section 2.2.1*, page 150, which consists of applying the cascades through the Unitex interface, copying the console to the editor and adjusting the resulting commands.

<footnote n="140">¹⁴⁰ Optionally, you can also copy and paste the *myProperNameDictionary.dic* file and, if it exists, *myProperNameDictionary.txt*, if you want the language package to be also a backup of your work.</footnote>
### 2.4.2 The last file from the _csc folder

It is possible to retrieve the last file from the _csc folder. This is useful because a bug has crept into the stable version 3.3 of Unitex: the output files are not correct if the cascade annotates categories with graphs, then deletes them with other graphs. This rarely happens, but in order to be sure that we have the correct output, the right file to recover is the last file from the _csc folder.

For that:

1. Add, at the beginning of the script, a new *DuplicateFile* line.

```
DuplicateFile --make-dir {CURRENT WORK DIR}/corpus_csc_csc
```

2. Modify the output file.

```
DuplicateFile -i {CURRENT WORK DIR}/corpus_csc_csc/corpus_0_0.snt {OUTPUT_FILE_1}
```

Since intermediate files are not kept, the number of the last file processed is always 0_0.

# 3 The command line

## 3.1 The batch launching file

We will use the *File Edition/New File* menu to create a file that will contain the following command line (it has to be adapted for each computer, depending on where the Unitex software is installed):<fnref n="141" />

```
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" {BatchRunScript -i .\Input -e -o .\Output -t1 .\scriptEntitiesLingpkg.zip -f -s Scripts\scriptEntities.uniscript }
```

We will save this file, naming it *multiLaunchEntities.bat*, in the folder:

MyUnitex\Scripts\ScriptEntities

## 3.2 Running the script

We will now open the command prompt window and choose as the working folder:

MyUnitex\Scripts\ScriptEntities

And then launch the *multiLaunchEntities.bat* batch file.

<figure><img src="image_284.png" /></figure>

As a result, we will obtain two files (*jubileePlainText.result.txt* and *JubileePlainText.xml.result.txt*) in the folder:

MyUnitex\Scripts\ScriptEntities\Output

<footnote n="141">¹⁴¹ A space is obligatory after the opening brace and before the closing brace. The t1 option means that you are working with one core. If your computer has several and your corpus is very large, you can work by parsing the texts of the corpus in parallel, by modifying this option: t2, t3...</footnote>
## 3.3 A command line with comments and several possible outputs

The command line above has two drawbacks: errors are not reported and only one file is possible as output. To overcome this, it is possible to launch the files one by one with the command line below which can be saved under the name `launchEntities.bat` in the folder:<fnref n="142" />

MyUnitex\Scripts\ScriptEntities

```
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe"
{ SelectOutput --output=on }<fnref n="143" /> { InstallLingResourcePackage
-p C:\Documents\MyUnitex\Scripts\ScriptEntities\scriptEntitiesLingpkg.zip
-x $:UnitexPkgResource -v } { RunScript -v
-a INPUT_FILE_1=C:\Documents\MyUnitex\Scripts\ScriptEntities\Input\jubileePlaintext.txt
-a CORPUS WORK DIR=$:UnitexPkgWork -a PACKAGE_DIR=$:UnitexPkgResource
-a OUTPUT_FILE_1=
C:\Documents\MyUnitex\Scripts\ScriptEntities\Output\jubileePlaintext.result1.txt
-a OUTPUT_FILE_2=
C:\Documents\MyUnitex\Scripts\ScriptEntities\Output\jubileePlaintext.result2.txt
$:$UnitexPkgResource\Scripts\scriptEntities.uniscript } { InstallLingResourcePackage
-p C:\Documents\MyUnitex\Scripts\ScriptEntities\scriptEntitiesLingpkg.zip
-x $:UnitexPkgResource -u -v }
```

If you want to get a single output file, but without comments, you need to remove the `-v` option. It can also be used to have comments with a single output file by removing the `-a OUTPUT_FILE_2` command. Finally, we can also add the commands `-a OUTPUT_FILE_3`, and so on, to have more than two output files.

But the use of this script is not mandatory.

# 4 An inventory of tag occurrences

In a cascade that processes XML-like input text, the standoff option launches the creation of an inventory of tag occurrences. For example, a standoff cascade can be used after the annotation with two cascades (analysis and synthesis). These standoff cascades are not integrated in the interface version of Unitex and can therefore be launched only by a script. To illustrate their use, we will create a new script by modifying the previously developed script.

The data used for this step is a cascade of graphs (which may possibly consist of only one graph) and a template file for formatting the results.

In order not to lose the results of the previous section (the annotated files), we will create a new folder for the results of the standoff cascade: *StandoffOutput* in the *Scripts* folder:

MyUnitex\Scripts\ScriptEntities

## 4.1 The standoff cascade

The *standoff* cascade graphs indicate which tags we would like to count. For these graphs we will create a *Standoff* folder in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities

<footnote n="142">¹⁴² Do not forget to adapt it for your computer, e.g. the paths to Unitex and to your personal Unitex folder.</footnote>

<footnote n="143">¹⁴³ This script can also be used without comment with the `{ SelectOutput --output=off }` option.</footnote>
If we want to process the same examples where we have inserted the tags `<AbsoluteDate></AbsoluteDate>`, `<RelativeDate></RelativeDate>`, `<Town></Town>` and `<Address></Address>`,<fnref n="144" /> we need four graphs that we can combine into one, with four subgraphs, which we will name `grfStandoff.grf`.<fnref n="145" /> The graph and its subgraphs will be recorded in the folder:

MyUnitex\English\Graphs\UnitexGettingStarted\Entities\Standoff

<figure><img src="image_285.png" /></figure>

Each subgraph recognizes the start tag, the end tag and the content between them, and at the same time prevents nesting of tags of the same type.

<figure><img src="image_286.png" /></figure>

We will now create a cascade that consists of only one graph (`grfStandoff.fst2` in `Merge` mode). We will record it using the name `standoff.csc` in the folder:

MyUnitex\English\Cassys\UnitexGettingStarted\XmlTextEntities

<table><thead><tr><th>#</th><th>Disabled</th><th>Name</th><th>Merge</th><th>Replace</th><th>Until Fix Po.</th><th>Generaliz.</th></tr></thead><tbody><tr><td>1</td><td>☐</td><td>grfStandoff.fst2</td><td>☑</td><td>☐</td><td>☐</td><td>☐</td></tr></tbody></table>

## 4.2 The template file

The template file sets the design of the report that inventories the tag occurrences. It is divided into three parts, the second part being itself divided into two.

<footnote n="144">¹⁴⁴ Chapter 4, Section 1, page 71 and Section 2, page 84.</footnote>

<footnote n="145">¹⁴⁵ This grouping makes it possible to process the text only once, which saves time. There may be a bias in case of nesting.</footnote>
Introduction

Inventory (#LINE... #REST): to be repeated for each type and type+subtype combination

Inventoried item: {TYPE} or {TYPE} <<and {SUBTYPE}>>><fnref n="146" />

Inventoried term: {TERM} and its number of occurrences in the document: {COUNT}

Conclusion

We will use the *File Edition* menu to create a new file with the name *standoffPattern.txt* in the folder:<fnref n="147" />

MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English

<xml>

#LINE

<listAnnotation type="{TYPE}" <<subtype="{SUBTYPE}" >>>>

#BLOCK

<term frequency="{COUNT}">{TERM}</term>

#END

</listAnnotation>

#REST

</xml>

## 4.3 The script

We will use the final version of the script we developed in *Section 2.2.4*, page 154. We will rename this file under the name *standoffScriptEntities.uniscript* in the folder:

MyUnitex\Scripts\ScriptEntities\scriptEntitiesLingpkg\Scripts

We will add it in a line before *Normalize* to indicate the location of the template file:

RunScript {PACKAGE_DIR}/Resources/English/standoffPattern.txt -R -o {CURRENT WORK_DIR}/patternResult.txt --noTranslate_separator

This line uses a specific Unitex function *RunScript*, which loads the *standoffPattern.txt* file under the name *patternResult.txt*. The *noTranslate_separator* option prohibits replacement of slashes by backslashes.

We will copy the last three lines *DuplicateFile*, *Tokenize* and *Cassys*; we will paste them right after. We will modify them: the working file is now *corpus_csc_csc.snt*; the *standoff* option replaces the *input offsets* option; and the name of cascade to be used is *standoff.csc*. The result of the standoff cascade, the *corpus_csc_csc_standoff.txt* file, replace the previous output in the ultimate line of the new script.

DuplicateFile -i {CURRENT WORK_DIR}/corpus_csc_csc.txt

{CURRENT WORK_DIR}/corpus_csc_csc.snt

Tokenize {CURRENT WORK_DIR}/corpus_csc_csc.snt

-a{PACKAGE_DIR}/Resources/English/Alphabet.txt -qutf8-no-bom

Cassys -a{PACKAGE_DIR}/Resources/English/Alphabet.txt

-t{CURRENT WORK_DIR}/corpus_csc_csc.snt

-I{PACKAGE_DIR}/Resources/English/CasSys/standoff.csc -v

-r{PACKAGE_DIR}/Resources/English/Graphs/

--standoff={CURRENT WORK_DIR}/patternResult.txt -qutf8-no-bom

DuplicateFile -i {CURRENT WORK_DIR}/corpus_csc_csc_standoff.txt {OUTPUT_FILE_1}

<footnote n="146">¹⁴⁶ The text between double angle brackets will only be written in the presence of a subtype.</footnote>

<footnote n="147">¹⁴⁷ This example of template file is in XML format, which is not the obligatory format.</footnote>
## 4.4 The new linguistic package

We will now copy graphs in the linguistic package.

Go to the folder: `MyUnitex\English\Graphs\UnitexGettingStarted\Entities`

Copy the *Standoff* folder.

Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\Graphs\UnitexGettingStarted\Entities`

Paste this folder.

As well as the new cascade.

Go to the folder: `MyUnitex\English\Cassys\UnitexGettingStarted\XmlTextEntities`

Copy the *standoff.csc* file.

Go to the folder: `MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English\CasSys`

Paste this file.

We will create the *scriptEntitiesLingpkg.zip* file in the same way as in *Section 2.3*, page 155.

## 4.5 The new batch file

We will save the *multiLaunchEntities.bat* file under the name *multiLaunchStandoffEntities.bat*.

We need to change in this file the name of the script and name of the folder for results (bold and underlined):

```
"C:\Program Files (x86)\Unitex-GramLab\App\UnitexToolLogger.exe" {BatchRunScript -i \Input -e -o \StandoffOutput -t1 \scriptEntitiesLingpkg.zip -f -s Scripts\standoffScriptEntities.uniscript }
```

As a result of running this script, we will obtain two identical files (*JubileePlaintext.result.txt* and *JubileePlaintext.xml.result.txt*) in the folder:<fnref n="148" />

`MyUnitex\Scripts\ScriptEntities\StandoffOutput`

Note that with the command line with several possible outputs in *Section 3.3*, page 158, you could merge both the script of *Section 2.2.4*, page 154, and the script of *Section 4.3*, page 160, to output the tagged file and the inventory of tag occurrences file.

<footnote n="148">¹⁴⁸ These two files are identical since the entities are the same in both.</footnote>
```xml
<?xml>
  <listAnnotation type="AbsoluteDate">
    <term frequency="1">June 2, 2022</term>
    <term frequency="1">February 6, 1952</term>
  </listAnnotation>
  <listAnnotation type="Address">
    <term frequency="1">Queen Elizabeth Street</term>
    <term frequency="1">London Street</term>
  </listAnnotation>
  <listAnnotation type="RelativeDate">
    <term frequency="1">Thursday June 2</term>
    <term frequency="1">June 5</term>
  </listAnnotation>
  <listAnnotation type="Town">
    <term frequency="2">Nearlondon</term>
    <term frequency="1">London</term>
  </listAnnotation>
</xml>
```

## 4.6 Some additional information

It is possible to add information to the *standoff* file by using specific variables. We can retrieve the name of the file and, if it is in XML format, also information from its header.<fnref n="149" />

### 4.6.1 The name of the input file

The file name can be included in the template file. To do this, we must add the `{FILENAME}` variable to the *standoffPattern.txt* file, which is in the folder:

`MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Resources\English`

```xml
<?xml>
  <sourceFile>
    <filename>{FILENAME}</filename>
  </sourceFile>
  #LINE
  <listAnnotation type="{TYPE}" <<subtype="{SUBTYPE}">>>
  #BLOCK
    <term frequency="{COUNT}">{TERM}</term>
  #END
  <listAnnotation>
  #REST
</xml>
```

Besides this, we must add the option `-a FILENAME={INPUT_FILE_1}` to the `RunScript` line of the file:

`MyUnitex\Scripts\ScriptEntities\ScriptEntitiesLingpkg\Scripts\standoffScriptEntities.uniscript`

<footnote n="149">¹⁴⁹ It is also possible to retrieve the Unitex version and its data by using a specific Unitex function `VersionInfo`, but we will not go into details here.</footnote>
After repeating the compression process and launching the script, we get two different files, one with empty `<title>{TITLE}</title>` tags and the other with the extracted text, enclosed in `<title></title>` tags. For the `jubileeXmlText.xml` file:

```xml
<xml>
  <sourceFile>
    <filename>.\Input\jubileeXmlText.xml</filename>
    <title>The municipal newspaper of the city of Nearlondon</title>
  </sourceFile>
  <listAnnotation type="AbsoluteDate">
    <term frequency="1">June 2, 2022</term>
    <term frequency="1">February 6, 1952</term>
  </listAnnotation>
  <listAnnotation type="Address">
    <term frequency="1">Queen Elizabeth Street</term>
    <term frequency="1">London Street</term>
  </listAnnotation>
  <listAnnotation type="RelativeDate">
    <term frequency="1">Thursday June 2</term>
    <term frequency="1">June 5</term>
  </listAnnotation>
  <listAnnotation type="Town">
    <term frequency="2">Nearlondon</term>
    <term frequency="1">London</term>
  </listAnnotation>
</xml>
```

# 5 Two small remarks on the optimization of a cascade

Let us say we want to launch our cascade over a very large number of texts, all placed in the same directory. Cascade time, even with multiple cores, can become an important parameter.

## 5.1 Number of graphs

Each time a graph of a cascade is applied, the whole text is reread. If two graphs are not ambiguous or if the first does not prepare data to be processed by the second, it is therefore better to merge them into a single graph which calls two subgraphs.

For instance, the five graphs of the analysis cascade of *Chapter 4, Section 1.1*, page 71, cannot be merged.

<table><thead><tr><th>#</th><th>Disabled</th><th>Name</th><th>Merge</th><th>Replace</th><th>Until Fix Point</th><th>Generalize</th></tr></thead><tbody><tr><td>1</td><td></td><td>absoluteDate.fst2</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>2</td><td></td><td>relativeDate.fst2</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>3</td><td></td><td>town.fst2</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>4</td><td></td><td>townGeneralization.fst2</td><td>✓</td><td></td><td></td><td></td></tr><tr><td>5</td><td></td><td>street.fst2</td><td>✓</td><td></td><td></td><td>✓</td></tr></tbody></table>

The two `absoluteDate.grf` and `relativeDate.grf` graphs are ambiguous and they must be applied in this order; the `town.grf` graph uses the previous two; the `townGeneralization.grf` graph uses the previous one; and the `street.grf` graph uses all previous graphs.

## 5.2 Number of boxes

It takes a little longer to read two boxes of a graph than one, it is therefore better to merge them, if it is possible.
For instance, the *month.grf* graph below:

<figure><img src="image_287.png" /></figure>

could be replaced by the *optimizedMonth.grf* graph:

<figure><img src="image_288.png" /></figure>

Note that such adjustments reducing the number of graphs or boxes for optimization may hinder the development of the entire work. It is therefore recommended to do it near the end of the elaboration, only if the number of files to process is large.
<figure><img src="image_289.png" /></figure>
# Contents

Chapter 1: introduction.................................................................................................................. 1

1 A short history................................................................................................................................. 1

2 A quick overview of the book........................................................................................................ 1

3 Organization of correct files ........................................................................................................ 2

4 A workbook? ................................................................................................................................. 3

4.1 Graph exercises................................................................................................................................. 3

Exercise 1 .................................................................................................................................................. 3

Exercise 2 .................................................................................................................................................. 3

Exercise 3 .................................................................................................................................................. 3

Exercise 4 .................................................................................................................................................. 3

4.2 Cascade exercises................................................................................................................................. 3

Exercise 5 .................................................................................................................................................. 3

Exercise 6 .................................................................................................................................................. 4

Exercise 7 .................................................................................................................................................. 4

4.3 Dictionary exercises................................................................................................................................. 4

Exercise 8 .................................................................................................................................................. 4

Exercise 9 .................................................................................................................................................. 5

Exercise 10 .................................................................................................................................................. 5

4.4 Lexicon grammar exercise .................................................................................................................. 6

Exercise 11 .................................................................................................................................................. 6

4.5 XAlign exercise................................................................................................................................. 6

Exercise 12 .................................................................................................................................................. 6

4.6 Script exercises................................................................................................................................. 6

Exercise 13 .................................................................................................................................................. 6

Exercise 14 .................................................................................................................................................. 6

5 Acknowledgement................................................................................................................................. 6

Chapter 2: first steps ................................................................................................................................. 7

1 Discovering Unitex................................................................................................................................. 7

1.1 Unitex installation................................................................................................................................. 7

1.1.1 Creating a private working folder.................................................................................. 7

1.1.2 Installation process.................................................................................................................. 7

1.1.3 Initial choices.................................................................................................................................. 10

1.1.4 Check or modify your private working directory.................................................................. 11

1.2 Using Unitex................................................................................................................................. 11

1.2.1 Starting Unitex................................................................................................................................. 11

1.2.2 The Unitex inteface.................................................................................................................. 12

1.2.3 Opening a text................................................................................................................................. 13

1.2.4 Applying dictionaries.................................................................................................................. 14

1.2.5 The DLF window................................................................................................................................. 15

1.2.6 The DLC window................................................................................................................................. 16

1.2.7 The ERR window................................................................................................................................. 16

2 A detailed example................................................................................................................................. 16

2.1 Concordances with one word query.................................................................................. 16

2.2 Concordances with a lemma........................................................................................................ 18

2.3 Concordances with a graph query.................................................................................. 19

2.3.1 A first graph................................................................................................................................. 20

2.3.2 Concordances................................................................................................................................. 22

2.3.3 Adding the future tense and the conditional.................................................................. 23
<table>
  <tbody>
    <tr>
      <td>2.3.4 Adding the perfect tenses</td>
      <td>24</td>
    </tr>
    <tr>
      <td>2.3.5 Adding the passive forms</td>
      <td>26</td>
    </tr>
    <tr>
      <td>2.3.6 Adding adverbs</td>
      <td>28</td>
    </tr>
    <tr>
      <td>2.4 Contexts</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.4.1 Left context</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.4.1.1 The verb to meet preceded by a pronoun</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.4.1.2 The verb to meet preceded by a pronoun and a left context</td>
      <td>32</td>
    </tr>
    <tr>
      <td>2.4.2 Right context</td>
      <td>32</td>
    </tr>
    <tr>
      <td>2.4.3 Lexical masks</td>
      <td>34</td>
    </tr>
    <tr>
      <td>3 The .snt file</td>
      <td>35</td>
    </tr>
    <tr>
      <td>3.1 Document transformation</td>
      <td>35</td>
    </tr>
    <tr>
      <td>3.1.1 Standard normalization</td>
      <td>35</td>
    </tr>
    <tr>
      <td>3.1.2 The Norm.txt file</td>
      <td>35</td>
    </tr>
    <tr>
      <td>3.1.2.1 The standardization of dashes and quotation marks</td>
      <td>36</td>
    </tr>
    <tr>
      <td>3.1.2.2 The standardization of apostrophes</td>
      <td>36</td>
    </tr>
    <tr>
      <td>3.2 The preprocessing</td>
      <td>37</td>
    </tr>
    <tr>
      <td>3.3 Material (only for confident users)</td>
      <td>37</td>
    </tr>
    <tr>
      <td>3.3.1 The Sentence.grf graph</td>
      <td>37</td>
    </tr>
    <tr>
      <td>3.3.2 The Replace.grf graph</td>
      <td>40</td>
    </tr>
    <tr>
      <td>3.3.3 The two alphabet files</td>
      <td>41</td>
    </tr>
    <tr>
      <td>3.3.3.1 The Alphabet.txt file</td>
      <td>41</td>
    </tr>
    <tr>
      <td>3.3.3.2 The Alphabet_sort.txt file</td>
      <td>42</td>
    </tr>
    <tr>
      <td>Chapter 3: corpus annotation</td>
      <td>45</td>
    </tr>
    <tr>
      <td>1 First example: compound verbs using the verb will</td>
      <td>45</td>
    </tr>
    <tr>
      <td>1.1 First step: the simplest graph</td>
      <td>45</td>
    </tr>
    <tr>
      <td>1.2 Second step: future passive and future perfect tense</td>
      <td>47</td>
    </tr>
    <tr>
      <td>1.3 Third step: recognizing the future perfect passive</td>
      <td>48</td>
    </tr>
    <tr>
      <td>1.4 Fourth step: recognizing continuous forms</td>
      <td>49</td>
    </tr>
    <tr>
      <td>1.5 Fifth step: insertion of an adverb after will or would</td>
      <td>50</td>
    </tr>
    <tr>
      <td>1.5.1 A single adverb</td>
      <td>50</td>
    </tr>
    <tr>
      <td>1.5.2 Several adverbs</td>
      <td>51</td>
    </tr>
    <tr>
      <td>1.6 Sixth step: insertion of an adverb after be</td>
      <td>52</td>
    </tr>
    <tr>
      <td>1.6.1 A graph that recognizes and annotates adverbs</td>
      <td>52</td>
    </tr>
    <tr>
      <td>1.6.2 The main graph</td>
      <td>52</td>
    </tr>
    <tr>
      <td>1.6.3 A new subgraph</td>
      <td>53</td>
    </tr>
    <tr>
      <td>1.7 Seventh step: negative adverbs</td>
      <td>54</td>
    </tr>
    <tr>
      <td>1.7.1 The main graph</td>
      <td>54</td>
    </tr>
    <tr>
      <td>1.7.2 The negation subgraph</td>
      <td>54</td>
    </tr>
    <tr>
      <td>1.7.3 Negative right context</td>
      <td>55</td>
    </tr>
    <tr>
      <td>1.8 Eighth step: adding the verb shall</td>
      <td>56</td>
    </tr>
    <tr>
      <td>1.9 Continuation of this graph (only for confident users)</td>
      <td>57</td>
    </tr>
    <tr>
      <td>1.9.1 Addition of contracted forms</td>
      <td>57</td>
    </tr>
    <tr>
      <td>1.9.2 Affirmative and negative constructions</td>
      <td>57</td>
    </tr>
    <tr>
      <td>1.9.3 Interrogative constructions</td>
      <td>58</td>
    </tr>
    <tr>
      <td>2 Second example: numbers written with words</td>
      <td>59</td>
    </tr>
    <tr>
      <td>2.1 First step: from 2 to 9</td>
      <td>59</td>
    </tr>
    <tr>
      <td>2.2 Second step: from 2 to 99</td>
      <td>59</td>
    </tr>
    <tr>
      <td>2.2.1 A subgraph without output</td>
      <td>59</td>
    </tr>
    <tr>
      <td>2.2.2 The NB2-99.grf graph</td>
      <td>60</td>
    </tr>
    <tr>
      <td>2.3 Third step: from 2 to 999</td>
      <td>61</td>
    </tr>
    <tr>
      <td>2.3.1 The NB1-99-subgraph.grf graph</td>
      <td>61</td>
    </tr>
    <tr>
      <td>2.3.2 The NB2-999.grf graph</td>
      <td>61</td>
    </tr>
    <tr>
      <td>2.4 Fourth step: from 2 to 999 999</td>
      <td>62</td>
    </tr>
    <tr>
      <td>2.4.1 The NB100-999-subgraph.grf graph</td>
      <td>62</td>
    </tr>
    <tr>
      <td>2.4.2 The NB2-999999.grf graph</td>
      <td>62</td>
    </tr>
    <tr>
      <td>3 Third example: annotating Roman numerals</td>
      <td>63</td>
    </tr>
  </tbody>
</table>
Contents

<table><tr><td>3.1 First step: from 2 to 9</td><td>63</td></tr><tr><td>3.2 Second step: from 2 to 99</td><td>64</td></tr><tr><td>3.2.1 The subgraph from 1 to 9</td><td>64</td></tr><tr><td>3.2.2 The graph from 2 to 99</td><td>64</td></tr><tr><td>3.3 Third step: from 2 to 999</td><td>65</td></tr><tr><td>3.3.1 The subgraph from 1 to 99</td><td>65</td></tr><tr><td>3.3.2 The graph from 2 to 999</td><td>66</td></tr><tr><td>3.4 Fourth step: from 2 to 3 999</td><td>67</td></tr><tr><td>3.4.1 The subgraph from 1 to 999</td><td>67</td></tr><tr><td>3.4.2 The graph from 2 to 3 999</td><td>67</td></tr><tr><td>3.5 Ambiguities of Roman numerals (only for confident users)</td><td>68</td></tr><tr><td>3.5.1 The subgraph of ambiguous Roman numerals</td><td>69</td></tr><tr><td>3.5.2 The use of weights</td><td>69</td></tr><tr><td>Chapter 4: CasSys</td><td>71</td></tr><tr><td>1 First example: named entity recognition in a plain text</td><td>71</td></tr><tr><td>1.1 Analysis cascade</td><td>71</td></tr><tr><td>1.1.1 The absoluteDate.grf graph</td><td>72</td></tr><tr><td>1.1.2 The relativeDate.grf graph</td><td>73</td></tr><tr><td>1.1.3 The analysis.csc cascade</td><td>73</td></tr><tr><td>1.1.4 The town.grf graph</td><td>74</td></tr><tr><td>1.1.5 Generalization graphs</td><td>75</td></tr><tr><td>1.1.6 The street.grf graph</td><td>76</td></tr><tr><td>1.1.7 Created files</td><td>77</td></tr><tr><td>1.1.7.1 The _csc files</td><td>77</td></tr><tr><td>1.1.7.2 The _csc folder</td><td>78</td></tr><tr><td>1.2 Synthesis cascade</td><td>79</td></tr><tr><td>1.2.1 The tag.grf graph</td><td>79</td></tr><tr><td>1.2.1.1 A first attempt</td><td>80</td></tr><tr><td>1.2.1.2 A second attempt</td><td>82</td></tr><tr><td>1.2.1.3 The Until fix point option</td><td>82</td></tr><tr><td>1.2.2 The internalDeletion.grf graph</td><td>83</td></tr><tr><td>2 Second example: named entity recognition in an XML text</td><td>84</td></tr><tr><td>2.1 Analysis cascade</td><td>86</td></tr><tr><td>2.1.1 The toolXml.grf graph</td><td>86</td></tr><tr><td>2.1.2 The toolHidden.grf graph</td><td>86</td></tr><tr><td>2.2 Synthesis cascade</td><td>87</td></tr><tr><td>3 Third example: measure recognition</td><td>88</td></tr><tr><td>3.1 Analysis cascade</td><td>89</td></tr><tr><td>3.1.1 The number.grf graph</td><td>89</td></tr><tr><td>3.1.2 Cascade</td><td>90</td></tr><tr><td>3.1.3 The measure.grf graph</td><td>90</td></tr><tr><td>3.2 Synthesis cascade</td><td>91</td></tr><tr><td>3.2.1 The tag.grf graph</td><td>92</td></tr><tr><td>3.2.2 Cascade</td><td>92</td></tr><tr><td>3.2.3 The numberDeletion.grf graph</td><td>93</td></tr><tr><td>3.2.4 The numberAndNumber.grf graph</td><td>93</td></tr><tr><td>4 Additional possibilities (only for confident users)</td><td>94</td></tr><tr><td>4.1 How does a generalization graph work?</td><td>94</td></tr><tr><td>4.2 Testing the variables</td><td>94</td></tr><tr><td>4.3 Reuse the same graph</td><td>95</td></tr><tr><td>4.3.1 Output variable</td><td>96</td></tr><tr><td>4.3.2 Graph repository</td><td>97</td></tr><tr><td>4.4 Hiding or deleting part of a text</td><td>98</td></tr><tr><td>4.4.1 Hiding part of a text</td><td>98</td></tr><tr><td>4.4.2 Deleting part of a text</td><td>101</td></tr><tr><td>4.4.2.1 The deleting graphs</td><td>101</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>4.4.2.2 Comparing the variables</td>
      <td>102</td>
    </tr>
    <tr>
      <td>4.5 Sub-categorization with cascades</td>
      <td>102</td>
    </tr>
    <tr>
      <td>4.5.1 Three modified graphs</td>
      <td>102</td>
    </tr>
    <tr>
      <td>4.5.2 Generalization graphs with restrictions</td>
      <td>104</td>
    </tr>
    <tr>
      <td>4.5.2.1 Without substitution of a category</td>
      <td>104</td>
    </tr>
    <tr>
      <td>4.5.2.2 With substitution of a category</td>
      <td>105</td>
    </tr>
    <tr>
      <td>Chapter 5: dictionary creation</td>
      <td>107</td>
    </tr>
    <tr>
      <td>1 Introduction</td>
      <td>107</td>
    </tr>
    <tr>
      <td>2 Inflection of monolexical words</td>
      <td>107</td>
    </tr>
    <tr>
      <td>2.1 Inflection by simple suffixation</td>
      <td>107</td>
    </tr>
    <tr>
      <td>2.1.1 The nouns with s in the plural</td>
      <td>107</td>
    </tr>
    <tr>
      <td>2.1.2 Two more examples</td>
      <td>110</td>
    </tr>
    <tr>
      <td>2.2 The L operator</td>
      <td>113</td>
    </tr>
    <tr>
      <td>2.2.1 The nouns with final y</td>
      <td>113</td>
    </tr>
    <tr>
      <td>2.2.2 Several more examples</td>
      <td>114</td>
    </tr>
    <tr>
      <td>2.3 Operators R, C and D</td>
      <td>116</td>
    </tr>
    <tr>
      <td>3 Inflection of multi-word units</td>
      <td>118</td>
    </tr>
    <tr>
      <td>3.1 Example of the MWU air of mystery</td>
      <td>119</td>
    </tr>
    <tr>
      <td>3.2 Some other examples</td>
      <td>120</td>
    </tr>
    <tr>
      <td>4 Some additional remarks</td>
      <td>123</td>
    </tr>
    <tr>
      <td>4.1 Adding features in dictionary entries</td>
      <td>123</td>
    </tr>
    <tr>
      <td>4.2 Direct creation of inflected entries</td>
      <td>123</td>
    </tr>
    <tr>
      <td>4.3 Adding comments</td>
      <td>125</td>
    </tr>
    <tr>
      <td>4.3.1 Adding comments to entries</td>
      <td>125</td>
    </tr>
    <tr>
      <td>4.3.2 Documenting dictionaries</td>
      <td>126</td>
    </tr>
    <tr>
      <td>4.4 Automatically verifying the format of a dictionary</td>
      <td>126</td>
    </tr>
    <tr>
      <td>4.4.1 Inflected dictionary</td>
      <td>126</td>
    </tr>
    <tr>
      <td>4.4.2 Lemma dictionary</td>
      <td>126</td>
    </tr>
    <tr>
      <td>4.5 Morphological-mode dictionary (only for confident users)</td>
      <td>127</td>
    </tr>
    <tr>
      <td>4.6 Priorities</td>
      <td>127</td>
    </tr>
    <tr>
      <td>4.7 Additional notes</td>
      <td>128</td>
    </tr>
    <tr>
      <td>5 Dictionary graphs</td>
      <td>128</td>
    </tr>
    <tr>
      <td>5.1 Example of Roman Numeral dictionary</td>
      <td>128</td>
    </tr>
    <tr>
      <td>5.2 Morphological extension of the used dictionaries (only for confident users)</td>
      <td>129</td>
    </tr>
    <tr>
      <td>5.2.1 Morphological-mode dictionary declaration</td>
      <td>129</td>
    </tr>
    <tr>
      <td>5.2.2 Example of the -less suffix</td>
      <td>130</td>
    </tr>
    <tr>
      <td>6 Additional possibilities (only for confident users)</td>
      <td>132</td>
    </tr>
    <tr>
      <td>6.1 A test on a suffix</td>
      <td>132</td>
    </tr>
    <tr>
      <td>6.2 Use of variables</td>
      <td>133</td>
    </tr>
    <tr>
      <td>6.2.1 Example of goose, tooth, foot</td>
      <td>133</td>
    </tr>
    <tr>
      <td>6.2.2 Example of bring, fight, think</td>
      <td>134</td>
    </tr>
    <tr>
      <td>Chapter 6: other tools</td>
      <td>137</td>
    </tr>
    <tr>
      <td>1 Lexicon-grammar</td>
      <td>137</td>
    </tr>
    <tr>
      <td>1.1 Example: particle verbs</td>
      <td>137</td>
    </tr>
    <tr>
      <td>1.2 Transformation of the spreadsheet table</td>
      <td>137</td>
    </tr>
    <tr>
      <td>1.3 Parameterized graphs</td>
      <td>138</td>
    </tr>
    <tr>
      <td>1.4 Generation of graphs</td>
      <td>138</td>
    </tr>
    <tr>
      <td>1.5 Annotations</td>
      <td>140</td>
    </tr>
    <tr>
      <td>2 XAlign</td>
      <td>141</td>
    </tr>
    <tr>
      <td>2.1 Installation</td>
      <td>141</td>
    </tr>
    <tr>
      <td>2.2 The French folder</td>
      <td>141</td>
    </tr>
    <tr>
      <td>2.3 File preparation</td>
      <td>142</td>
    </tr>
    <tr>
      <td>2.3.1 English folder</td>
      <td>142</td>
    </tr>
    <tr>
      <td>2.3.2 French folder</td>
      <td>143</td>
    </tr>
    <tr>
      <td>2.3.3 XAlign folder</td>
      <td>143</td>
    </tr>
  </tbody>
</table>
Contents

<table><tr><td>2.4 Automatic alignment</td><td>143</td></tr><tr><td>2.5 Reading and manual correction</td><td>145</td></tr><tr><td>2.6 Search performed on one of two texts</td><td>146</td></tr><tr><td>2.7 Cascade search and XAlign (only for confident users)</td><td>147</td></tr><tr><td>2.7.1 Launching cascades</td><td>147</td></tr><tr><td>2.7.2 File transformation</td><td>148</td></tr><tr><td>2.7.3 Using XAlign</td><td>148</td></tr><tr><td>Chapter 7: scripts</td><td>149</td></tr><tr><td>1 Introduction</td><td>149</td></tr><tr><td>2 Creation of the linguistic package</td><td>149</td></tr><tr><td>2.1 The Resources folder</td><td>150</td></tr><tr><td>2.2 The scripts folder</td><td>150</td></tr><tr><td>2.2.1 Using the console</td><td>150</td></tr><tr><td>2.2.2 File adjustment</td><td>153</td></tr><tr><td>2.2.3 Script generalization</td><td>153</td></tr><tr><td>2.2.4 Final touches</td><td>154</td></tr><tr><td>2.3 Compression</td><td>155</td></tr><tr><td>2.4 Another possibility (only for confident users)</td><td>155</td></tr><tr><td>2.4.1 The use of dictionaries</td><td>155</td></tr><tr><td>2.4.1.1 The default English dictionary</td><td>155</td></tr><tr><td>2.4.1.2 A private dictionary</td><td>156</td></tr><tr><td>2.4.1.3 Several dictionaries</td><td>156</td></tr><tr><td>2.4.1.4 A morphological dictionary</td><td>156</td></tr><tr><td>2.4.2 The last file from the _csc folder</td><td>157</td></tr><tr><td>3 The command line</td><td>157</td></tr><tr><td>3.1 The batch launching file</td><td>157</td></tr><tr><td>3.2 Running the script</td><td>157</td></tr><tr><td>3.3 A command line with comments and several possible outputs</td><td>158</td></tr><tr><td>4 An inventory of tag occurrences</td><td>158</td></tr><tr><td>4.1 The standoff cascade</td><td>158</td></tr><tr><td>4.2 The template file</td><td>159</td></tr><tr><td>4.3 The script</td><td>160</td></tr><tr><td>4.4 The new linguistic package</td><td>161</td></tr><tr><td>4.5 The new batch file</td><td>161</td></tr><tr><td>4.6 Some additional information</td><td>162</td></tr><tr><td>4.6.1 The name of the input file</td><td>162</td></tr><tr><td>4.6.2 Extracting information from the XML header</td><td>163</td></tr><tr><td>5 Two small remarks on the optimization of a cascade</td><td>164</td></tr><tr><td>5.1 Number of graphs</td><td>164</td></tr><tr><td>5.2 Number of boxes</td><td>164</td></tr></table>
<figure><img src="image_290.png" /></figure>
Index

<table><tr><td>Alphabet.txt</td><td>41</td></tr><tr><td>Build concordance</td><td>17</td></tr><tr><td>CODE.SEM</td><td>127</td></tr><tr><td>Copy operator</td><td>117</td></tr><tr><td>correct files</td><td>2</td></tr><tr><td>DELA/Compress into FST</td><td>109</td></tr><tr><td>DELA/Inflect</td><td>109</td></tr><tr><td>Dela/Lookup</td><td>89</td></tr><tr><td>dela-en-public.bin</td><td>14</td></tr><tr><td>Delete operator</td><td>117</td></tr><tr><td>DIC</td><td>34</td></tr><tr><td>dictionary graph</td><td>128</td></tr><tr><td>DLC</td><td>15</td></tr><tr><td>DLF</td><td>15</td></tr><tr><td>Dominique Perrin</td><td>1</td></tr><tr><td>EQUAL</td><td>102</td></tr><tr><td>ERR</td><td>15</td></tr><tr><td>File Edition</td><td>35</td></tr><tr><td>FIRST</td><td>34</td></tr><tr><td>FSA</td><td>1</td></tr><tr><td>FSGraph/New</td><td>20</td></tr><tr><td>generalization graph</td><td>75</td></tr><tr><td>graph repository</td><td>97</td></tr><tr><td>INFLECTED</td><td>127</td></tr><tr><td>input variable</td><td>80</td></tr><tr><td>installation</td><td>7</td></tr><tr><td>Intex</td><td>1</td></tr><tr><td>inventory of tag occurrences</td><td>158</td></tr><tr><td>LADL</td><td>1</td></tr><tr><td>left context</td><td>30</td></tr><tr><td>Left operator</td><td>113</td></tr><tr><td>LEMMA</td><td>130</td></tr><tr><td>lexical mask</td><td>34</td></tr><tr><td>lexicon-grammar</td><td>1, 137</td></tr><tr><td>linguistic package</td><td>149</td></tr><tr><td>Located sequences</td><td>17</td></tr><tr><td>LOWER</td><td>34</td></tr><tr><td>Maurice Gross</td><td>1</td></tr><tr><td>Max Silberstein</td><td>1</td></tr><tr><td>Menus</td><td>12</td></tr></table>

<table><tr><td>morphological mode</td><td>63</td></tr><tr><td>Morphological mode dictionaries</td><td>127</td></tr><tr><td>NB</td><td>34</td></tr><tr><td>negative right context</td><td>55</td></tr><tr><td>Norm.txt file</td><td>35</td></tr><tr><td>output variable</td><td>96</td></tr><tr><td>preprocessing</td><td>37</td></tr><tr><td>private working directory</td><td>10</td></tr><tr><td>private working folder</td><td>7</td></tr><tr><td>Regular expression</td><td>16</td></tr><tr><td>Replace.grf</td><td>40</td></tr><tr><td>Reversed link between boxes</td><td>30</td></tr><tr><td>right context</td><td>32</td></tr><tr><td>Right operator</td><td>116</td></tr><tr><td>script</td><td>149</td></tr><tr><td>Sébastien Paumier</td><td>1</td></tr><tr><td>semantic code</td><td>123</td></tr><tr><td>sentence delimiter</td><td>13</td></tr><tr><td>Sentence.grf</td><td>37</td></tr><tr><td>SET</td><td>95</td></tr><tr><td>Show differences with previous concordances</td><td>47</td></tr><tr><td>simple form</td><td>14</td></tr><tr><td>standoff</td><td>158</td></tr><tr><td>subgraph</td><td>52</td></tr><tr><td>template file</td><td>159</td></tr><tr><td>Text/Apply CasSys Cascade</td><td>73</td></tr><tr><td>Text/Apply Lexical Resources</td><td>14</td></tr><tr><td>Text/Change Language</td><td>141</td></tr><tr><td>Text/Locate Pattern</td><td>16</td></tr><tr><td>Text/Open</td><td>13</td></tr><tr><td>token</td><td>13</td></tr><tr><td>TOKEN</td><td>34</td></tr><tr><td>Token list</td><td>14</td></tr><tr><td>UNSET</td><td>95</td></tr><tr><td>Until Fix Point</td><td>82</td></tr><tr><td>UPPER</td><td>34</td></tr><tr><td>variable</td><td>80</td></tr><tr><td>weight</td><td>69</td></tr><tr><td>WORD</td><td>34</td></tr><tr><td>WordList</td><td>15</td></tr></table>
<figure><img src="image_291.png" /></figure>
<figure><img src="image_292.png" /></figure>
**Denis Maurel**

is emeritus professor at the University of Tours, member of Lifat (Laboratory of Fundamental and Applied Computer Science of Tours) and associate member of the LLL (Laboratory of Nigerian Linguistics).

He defended his thesis in 1989 under the supervision of Professor Maurice Gross.

His research focuses on NLP (Natural Language Processing), mainly using Unitex, as well as the creation of linguistic resources: Proper Names Processing (Prolex project), Named Entity Recognition (CasEN project), scientific documents mining (Istex and Abliss projects).

He taught Unitex in linguistics and digital humanity masters courses, as well as in computer science engineering schools. He gave several tutorials to doctoral students and researchers.

<figure><img src="image_293.png" /></figure>

**Cvetana Krstev**

served as a professor at the University of Belgrade, Faculty of Philology from 1998 to 2020. She is a co-founder and current president of the Language Resources and Technologies Society JeRTeh.

Her research focuses on the development of NLP resources and tools. She developed e-dictionaries of Serbian as well as numerous tools for Unitex. These tools, which target Serbian texts, include the named entity recognition system, correction of OCR errors, restoration of diacritics, extraction of definitions, and many more.

She gave tutorials on using Unitex to researchers and PhD students studying Linguistics, Philology and Information Sciences.

<figure><img src="image_294.png" /></figure>