<table>
  <tbody>
    <tr>
      <td>http://r4ds.had.co.nz/transform.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/transform.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/transform.html</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>https://r4ds.hadley.nz/data-tidy</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
    <tr>
      <td>http://r4ds.had.co.nz/data-import.html</td>
    </tr>
  </tbody>
</table>
<table><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr><tr><td>http://r4ds.had.co.nz/data-import.html</td></tr></table>

### 4.8.6.3 Tokens

`tokens()` segments texts in a corpus into tokens (words or sentences) by word boundaries. We can remove punctuations or not

```
web_pages_txt_corpus_tokens <- tokens(web_pages_txt_corpus)
web_pages_txt_corpus_tokens
```

#### 4.8.6.3.1 With punctuations

<table><tr><td>## Tokens consisting of 9 documents and 4 docvars.</td></tr><tr><td>## text1 :</td></tr><tr><td>## [1] "Skip" "to" "content" "ggplot24.0.1" "Get"</td></tr><tr><td>## [6] "started" "Reference" "News" "Releases" "Version"</td></tr><tr><td>## [11] "4.0.0" "Version"</td></tr><tr><td>## [ ... and 785 more ]</td></tr><tr><td>## text2 :</td></tr><tr><td>## [1] "Skip" "to" "content" "dplyr1.1.4" "Get"</td></tr><tr><td>## [6] "started" "Reference" "Articles" "Grouped" "data"</td></tr><tr><td>## [11] "Two-table" "verbs"</td></tr><tr><td>## [ ... and 1,246 more ]</td></tr><tr><td>## text3 :</td></tr><tr><td>## [1] "Skip" "to" "content" "tidyr1.3.1" "Tidy"</td></tr><tr><td>## [6] "data" "Reference" "Articles" "Pivoting" "Rectangling"</td></tr><tr><td>## [11] "Nested" "data"</td></tr><tr><td>## [ ... and 717 more ]</td></tr><tr><td>## text4 :</td></tr><tr><td>## [1] "Skip" "to" "content" "readr2.1.6" "Get"</td></tr><tr><td>## [6] "started" "Reference" "Articles" "Column" "type"</td></tr><tr><td>## [11] "Locales" "News"</td></tr><tr><td>## [ ... and 1,739 more ]</td></tr></table>
##
## text5 :
## [1] "Skip" "to" "content" "purrr1.2.0" "Get"
## [6] "started" "Reference" "Articles" "purrr" "<"
## [11] "-" ">"
## [ ... and 487 more ]
##
## text6 :
## [1] "Skip" "to" "content" "tibble3.3.0" "Get"
## [6] "started" "Reference" "Articles" "Column" "types"
## [11] "Controlling" "display"
## [ ... and 705 more ]
##
## [ reached max_ndoc ... 3 more documents ]

web_pages_txt_corpus_tokens <- tokens(web_pages_txt_corpus, remove_punct = TRUE)
web_pages_txt_corpus_tokens

4.8.6.3.2 Without punctuations

## Tokens consisting of 9 documents and 4 docvars.
## text1 :
## [1] "Skip" "to" "content" "ggplot24.0.1" "Get"
## [6] "started" "Reference" "News" "Releases" "Version"
## [11] "4.0.0" "Version"
## [ ... and 652 more ]
##
## text2 :
## [1] "Skip" "to" "content" "dplyr1.1.4" "Get"
## [6] "started" "Reference" "Articles" "Grouped" "data"
## [11] "Two-table" "verbs"
## [ ... and 988 more ]
##
## text3 :
## [1] "Skip" "to" "content" "tidyr1.3.1" "Tidy"
## [6] "data" "Reference" "Articles" "Pivoting" "Rectangling"
## [11] "Nested" "data"
## [ ... and 547 more ]
##
## text4 :
## [1] "Skip" "to" "content" "readr2.1.6" "Get"
## [6] "started" "Reference" "Articles" "Column" "type"
## [11] "Locales" "News"
## [ ... and 1,370 more ]
##
## text5 :
[1] "Skip" "to" "content" "purrr1.2.0" "Get"
[6] "started" "Reference" "Articles" "purrr" "<"
[11] ">" "base"
[ ... and 380 more ]
##
## text6 :
[1] "Skip" "to" "content" "tibble3.3.0" "Get"
[6] "started" "Reference" "Articles" "Column" "types"
[11] "Controlling" "display"
[ ... and 536 more ]
##
[ reached max_ndoc ... 3 more documents ]

4.8.6.4 Types

We can also generate types on the tokenised corpus (without punctuations)

web_pages_txt_corpus tok_no_punct_types <- types(web_pages_txt_corpus tok_no_punct)
web_pages_txt_corpus tok_no_punct_types

[1] "Skip"
[2] "to"
[3] "content"
[4] "ggplot24.0.1"
[5] "Get"
[6] "started"
[7] "Reference"
[8] "News"
[9] "Releases"
[10] "Version"
[11] "4.0.0"
[12] "3.5.0"
[13] "3.4.0"
[14] "3.3.0"
[15] "3.2.0"
[16] "3.1.0"
[17] "3.0.0"
[18] "2.2.0"
[19] "2.1.0"
[20] "2.0.0"
[21] "1.0.0"
[22] "Changelog"
[23] "Articles"
[24] "Aesthetic"
[25] "specifications"
[26] "Developer"
[27] "Extending"
[28] "ggplot2"
[29] "Using"
[30] "in"
[31] "packages"
[32] "Profiling"
[33] "Performance"
[34] "FAQ"
[35] "Axes"
[36] "Faceting"
[37] "Customising"
[38] "Annotation"
[39] "Reordering"
[40] "Barplots"
[41] "Ask"
[42] "AI"
[43] ""
[44] "Overview"
[45] "is"
[46] "a"
[47] "system"
[48] "for"
[49] "declaratively"
[50] "creating"
[51] "graphics"
[52] "based"
[53] "on"
[54] "The"
[55] "Grammar"
[56] "of"
[57] "Graphics"
[58] "You"
[59] "provide"
[60] "the"
[61] "data"
[62] "tell"
[63] "how"
[64] "map"
[65] "variables"
[66] "aesthetics"
[67] "what"
[68] "graphical"
[69] "primitives"
[70] "use"
[71] "and"
[72] "it"
<table><tr><td>[73]</td><td>takes</td></tr><tr><td>[74]</td><td>care</td></tr><tr><td>[75]</td><td>details</td></tr><tr><td>[76]</td><td>Installation</td></tr><tr><td>[77]</td><td>easiest</td></tr><tr><td>[78]</td><td>way</td></tr><tr><td>[79]</td><td>get</td></tr><tr><td>[80]</td><td>install</td></tr><tr><td>[81]</td><td>whole</td></tr><tr><td>[82]</td><td>tidyverse</td></tr><tr><td>[83]</td><td>install.packages</td></tr><tr><td>[84]</td><td>Alternatively</td></tr><tr><td>[85]</td><td>just</td></tr><tr><td>[86]</td><td>Or</td></tr><tr><td>[87]</td><td>development</td></tr><tr><td>[88]</td><td>version</td></tr><tr><td>[89]</td><td>from</td></tr><tr><td>[90]</td><td>GitHub</td></tr><tr><td>[91]</td><td>pak</td></tr><tr><td>[92]</td><td>Cheatsheet</td></tr><tr><td>[93]</td><td>Usage</td></tr><tr><td>[94]</td><td>It's</td></tr><tr><td>[95]</td><td>hard</td></tr><tr><td>[96]</td><td>succinctly</td></tr><tr><td>[97]</td><td>describe</td></tr><tr><td>[98]</td><td>works</td></tr><tr><td>[99]</td><td>because</td></tr><tr><td>[100]</td><td>embodies</td></tr><tr><td>[101]</td><td>deep</td></tr><tr><td>[102]</td><td>philosophy</td></tr><tr><td>[103]</td><td>visualisation</td></tr><tr><td>[104]</td><td>However</td></tr><tr><td>[105]</td><td>most</td></tr><tr><td>[106]</td><td>cases</td></tr><tr><td>[107]</td><td>you</td></tr><tr><td>[108]</td><td>start</td></tr><tr><td>[109]</td><td>with</td></tr><tr><td>[110]</td><td>ggplot</td></tr><tr><td>[111]</td><td>supply</td></tr><tr><td>[112]</td><td>dataset</td></tr><tr><td>[113]</td><td>aesthetic</td></tr><tr><td>[114]</td><td>mapping</td></tr><tr><td>[115]</td><td>aes</td></tr><tr><td>[116]</td><td>then</td></tr><tr><td>[117]</td><td>add</td></tr><tr><td>[118]</td><td>layers</td></tr></table>
[119] "like"
[120] "geom_point"
[121] "or"
[122] "geom_histogram"
[123] "scales"
[124] "scaleColourBrewer"
[125] "faceting"
[126] "facetWrap"
[127] "coordinate"
[128] "systems"
[129] "coordFlip"
[130] "library"
[131] "mpg"
[132] "displ"
[133] "hwy"
[134] "colour"
[135] "="
[136] "class"
[137] "+"
[138] "Lifecycle"
[139] "now"
[140] "over"
[141] "10"
[142] "years"
[143] "old"
[144] "used"
[145] "by"
[146] "hundreds"
[147] "thousands"
[148] "people"
[149] "make"
[150] "millions"
[151] "plots"
[152] "That"
[153] "means"
[154] "by-and-large"
[155] "itself"
[156] "changes"
[157] "relatively"
[158] "little"
[159] "When"
[160] "we"
[161] "do"
[162] "they"
[163] "will"
[164] "be"
## [165] "generally"
## [166] "new"
## [167] "functions"
## [168] "arguments"
## [169] "rather"
## [170] "than"
## [171] "changing"
## [172] "behaviour"
## [173] "existing"
## [174] "if"
## [175] "them"
## [176] "compelling"
## [177] "reasons"
## [178] "If"
## [179] "are"
## [180] "looking"
## [181] "innovation"
## [182] "look"
## [183] "s"
## [184] "rich"
## [185] "ecosystem"
## [186] "extensions"
## [187] "See"
## [188] "community"
## [189] "maintained"
## [190] "list"
## [191] "at"
## [192] "https://exts.ggplot2.tidyverse.org gallery/"
## [193] "Learning"
## [194] "better"
## [195] "off"
## [196] "starting"
## [197] "systematic"
## [198] "introduction"
## [199] "trying"
## [200] "learn"
## [201] "reading"
## [202] "individual"
## [203] "documentation"
## [204] "pages"
## [205] "Currently"
## [206] "there"
## [207] "several"
## [208] "good"
## [209] "places"
## [210] "Data"
[211] "Visualization"

[212] "Communication"

[213] "chapters"

[214] "R"

[215] "Science"

[216] "designed"

[217] "give"

[218] "comprehensive"

[219] "these"

[220] "two"

[221] "up"

[222] "speed"

[223] "essentials"

[224] "as"

[225] "quickly"

[226] "possible"

[227] "you'd"

[228] "take"

[229] "an"

[230] "online"

[231] "course"

[232] "try"

[233] "With"

[234] "Kara"

[235] "Woo"

[236] "follow"

[237] "webinar"

[238] "Plotting"

[239] "Anything"

[240] "Thomas"

[241] "Lin"

[242] "Pedersen"

[243] "want"

[244] "dive"

[245] "into"

[246] "making"

[247] "common"

[248] "I"

[249] "recommend"

[250] "Cookbook"

[251] "Winston"

[252] "Chang"

[253] "It"

[254] "provides"

[255] "set"

[256] "recipes"
<table><tr><td>[257]</td><td>"solve"</td></tr><tr><td>[258]</td><td>"problems"</td></tr><tr><td>[259]</td><td>"you've"</td></tr><tr><td>[260]</td><td>"mastered"</td></tr><tr><td>[261]</td><td>"basics"</td></tr><tr><td>[262]</td><td>"more"</td></tr><tr><td>[263]</td><td>"read"</td></tr><tr><td>[264]</td><td>"Elegant"</td></tr><tr><td>[265]</td><td>"Analysis"</td></tr><tr><td>[266]</td><td>"describes"</td></tr><tr><td>[267]</td><td>"theoretical"</td></tr><tr><td>[268]</td><td>"underpinnings"</td></tr><tr><td>[269]</td><td>"shows"</td></tr><tr><td>[270]</td><td>"all"</td></tr><tr><td>[271]</td><td>"pieces"</td></tr><tr><td>[272]</td><td>"fit"</td></tr><tr><td>[273]</td><td>"together"</td></tr><tr><td>[274]</td><td>"This"</td></tr><tr><td>[275]</td><td>"book"</td></tr><tr><td>[276]</td><td>"helps"</td></tr><tr><td>[277]</td><td>"understand"</td></tr><tr><td>[278]</td><td>"theory"</td></tr><tr><td>[279]</td><td>"that"</td></tr><tr><td>[280]</td><td>"underpins"</td></tr><tr><td>[281]</td><td>"help"</td></tr><tr><td>[282]</td><td>"create"</td></tr><tr><td>[283]</td><td>"types"</td></tr><tr><td>[284]</td><td>"specifically"</td></tr><tr><td>[285]</td><td>"tailored"</td></tr><tr><td>[286]</td><td>"your"</td></tr><tr><td>[287]</td><td>"needs"</td></tr><tr><td>[288]</td><td>"For"</td></tr><tr><td>[289]</td><td>"articles"</td></tr><tr><td>[290]</td><td>"about"</td></tr><tr><td>[291]</td><td>"announcements"</td></tr><tr><td>[292]</td><td>"deep-dives"</td></tr><tr><td>[293]</td><td>"can"</td></tr><tr><td>[294]</td><td>"visit"</td></tr><tr><td>[295]</td><td>"blog"</td></tr><tr><td>[296]</td><td>"Getting"</td></tr><tr><td>[297]</td><td>"There"</td></tr><tr><td>[298]</td><td>"main"</td></tr><tr><td>[299]</td><td>"Posit"</td></tr><tr><td>[300]</td><td>"Community"</td></tr><tr><td>[301]</td><td>"formerly"</td></tr><tr><td>[302]</td><td>"RStudio"</td></tr></table>
[303] "friendly"

[304] "place"

[305] "ask"

[306] "any"

[307] "questions"

[308] "Stack"

[309] "Overflow"

[310] "great"

[311] "source"

[312] "answers"

[313] "also"

[314] "once"

[315] "have"

[316] "created"

[317] "reproducible"

[318] "example"

[319] "illustrates"

[320] "problem"

[321] "Links"

[322] "View"

[323] "CRAN"

[324] "Browse"

[325] "code"

[326] "Report"

[327] "bug"

[328] "Learn"

[329] "Extensions"

[330] ""

[331] "License"

[332] "Full"

[333] "license"

[334] "MIT"

[335] "file"

[336] "LICENSE"

[337] "Contributing"

[338] "guide"

[339] "Code"

[340] "conduct"

[341] "Citation"

[342] "Citing"

[343] "Developers"

[344] "Hadley"

[345] "Wickham"

[346] "Author"

[347] "Lionel"

[348] "Henry"
<table>
  <tbody>
    <tr>
      <td>[349]</td>
      <td>"maintainer"</td>
    </tr>
    <tr>
      <td>[350]</td>
      <td>"Kohske"</td>
    </tr>
    <tr>
      <td>[351]</td>
      <td>"Takahashi"</td>
    </tr>
    <tr>
      <td>[352]</td>
      <td>"Claus"</td>
    </tr>
    <tr>
      <td>[353]</td>
      <td>"Wilke"</td>
    </tr>
    <tr>
      <td>[354]</td>
      <td>"Hiroaki"</td>
    </tr>
    <tr>
      <td>[355]</td>
      <td>"Yutani"</td>
    </tr>
    <tr>
      <td>[356]</td>
      <td>"Dewey"</td>
    </tr>
    <tr>
      <td>[357]</td>
      <td>"Dunnington"</td>
    </tr>
    <tr>
      <td>[358]</td>
      <td>"Teun"</td>
    </tr>
    <tr>
      <td>[359]</td>
      <td>"van"</td>
    </tr>
    <tr>
      <td>[360]</td>
      <td>"den"</td>
    </tr>
    <tr>
      <td>[361]</td>
      <td>"Brand"</td>
    </tr>
    <tr>
      <td>[362]</td>
      <td>"Copyright"</td>
    </tr>
    <tr>
      <td>[363]</td>
      <td>"holder"</td>
    </tr>
    <tr>
      <td>[364]</td>
      <td>"funder"</td>
    </tr>
    <tr>
      <td>[365]</td>
      <td>"Developed"</td>
    </tr>
    <tr>
      <td>[366]</td>
      <td>"Site"</td>
    </tr>
    <tr>
      <td>[367]</td>
      <td>"built"</td>
    </tr>
    <tr>
      <td>[368]</td>
      <td>"pkgdown"</td>
    </tr>
    <tr>
      <td>[369]</td>
      <td>"dplyr1.1.4"</td>
    </tr>
    <tr>
      <td>[370]</td>
      <td>"Grouped"</td>
    </tr>
    <tr>
      <td>[371]</td>
      <td>"Two-table"</td>
    </tr>
    <tr>
      <td>[372]</td>
      <td>"verbs"</td>
    </tr>
    <tr>
      <td>[373]</td>
      <td>"dplyr"</td>
    </tr>
    <tr>
      <td>[374]</td>
      <td>"<"</td>
    </tr>
    <tr>
      <td>[375]</td>
      <td>">"</td>
    </tr>
    <tr>
      <td>[376]</td>
      <td>"base"</td>
    </tr>
    <tr>
      <td>[377]</td>
      <td>"Automation"</td>
    </tr>
    <tr>
      <td>[378]</td>
      <td>"Column-wise"</td>
    </tr>
    <tr>
      <td>[379]</td>
      <td>"operations"</td>
    </tr>
    <tr>
      <td>[380]</td>
      <td>"Row-wise"</td>
    </tr>
    <tr>
      <td>[381]</td>
      <td>"Programming"</td>
    </tr>
    <tr>
      <td>[382]</td>
      <td>"More"</td>
    </tr>
    <tr>
      <td>[383]</td>
      <td>"1.1.0"</td>
    </tr>
    <tr>
      <td>[384]</td>
      <td>"0.8.3"</td>
    </tr>
    <tr>
      <td>[385]</td>
      <td>"0.8.2"</td>
    </tr>
    <tr>
      <td>[386]</td>
      <td>"0.8.1"</td>
    </tr>
    <tr>
      <td>[387]</td>
      <td>"0.8.0"</td>
    </tr>
    <tr>
      <td>[388]</td>
      <td>"0.7.5"</td>
    </tr>
    <tr>
      <td>[389]</td>
      <td>"grammar"</td>
    </tr>
    <tr>
      <td>[390]</td>
      <td>"manipulation"</td>
    </tr>
    <tr>
      <td>[391]</td>
      <td>"providing"</td>
    </tr>
    <tr>
      <td>[392]</td>
      <td>"consistent"</td>
    </tr>
    <tr>
      <td>[393]</td>
      <td>"challenges"</td>
    </tr>
    <tr>
      <td>[394]</td>
      <td>"mutate"</td>
    </tr>
  </tbody>
</table>
[395] "adds"
[396] "select"
[397] "picks"
[398] "their"
[399] "names"
[400] "filter"
[401] "values"
[402] "summarise"
[403] "reduces"
[404] "multiple"
[405] "down"
[406] "single"
[407] "summary"
[408] "arrange"
[409] "ordering"
[410] "rows"
[411] "These"
[412] "combine"
[413] "naturally"
[414] "group_by"
[415] "which"
[416] "allows"
[417] "perform"
[418] "operation"
[419] "group"
[420] "vignette"
[421] "As"
[422] "well"
[423] "single-table"
[424] "variety"
[425] "two-table"
[426] "best"
[427] "transformation"
[428] "chapter"
[429] "Backends"
[430] "In"
[431] "addition"
[432] "frames"
[433] "tibbles"
[434] "makes"
[435] "working"
[436] "other"
[437] "computational"
[438] "backends"
[439] "accessible"
[440] "efficient"
<table><tr><td>[441]</td><td>"Below"</td></tr><tr><td>[442]</td><td>"alternative"</td></tr><tr><td>[443]</td><td>"arrow"</td></tr><tr><td>[444]</td><td>"larger-than-memory"</td></tr><tr><td>[445]</td><td>"datasets"</td></tr><tr><td>[446]</td><td>"including"</td></tr><tr><td>[447]</td><td>"remote"</td></tr><tr><td>[448]</td><td>"cloud"</td></tr><tr><td>[449]</td><td>"storage"</td></tr><tr><td>[450]</td><td>"AWS"</td></tr><tr><td>[451]</td><td>"S3"</td></tr><tr><td>[452]</td><td>"using"</td></tr><tr><td>[453]</td><td>"Apache"</td></tr><tr><td>[454]</td><td>"Arrow"</td></tr><tr><td>[455]</td><td>"C"</td></tr><tr><td>[456]</td><td>"engine"</td></tr><tr><td>[457]</td><td>"Acero"</td></tr><tr><td>[458]</td><td>"dtplyr"</td></tr><tr><td>[459]</td><td>"large"</td></tr><tr><td>[460]</td><td>"in-memory"</td></tr><tr><td>[461]</td><td>"Translates"</td></tr><tr><td>[462]</td><td>"high"</td></tr><tr><td>[463]</td><td>"performance"</td></tr><tr><td>[464]</td><td>"data.table"</td></tr><tr><td>[465]</td><td>"dbplyr"</td></tr><tr><td>[466]</td><td>"stored"</td></tr><tr><td>[467]</td><td>"relational"</td></tr><tr><td>[468]</td><td>"database"</td></tr><tr><td>[469]</td><td>"SQL"</td></tr><tr><td>[470]</td><td>"duckplyr"</td></tr><tr><td>[471]</td><td>"duckdb"</td></tr><tr><td>[472]</td><td>"zero"</td></tr><tr><td>[473]</td><td>"extra"</td></tr><tr><td>[474]</td><td>"copies"</td></tr><tr><td>[475]</td><td>"queries"</td></tr><tr><td>[476]</td><td>"automatic"</td></tr><tr><td>[477]</td><td>"fallback"</td></tr><tr><td>[478]</td><td>"when"</td></tr><tr><td>[479]</td><td>"translation"</td></tr><tr><td>[480]</td><td>"isn't"</td></tr><tr><td>[481]</td><td>"still"</td></tr><tr><td>[482]</td><td>"small"</td></tr><tr><td>[483]</td><td>"enough"</td></tr><tr><td>[484]</td><td>"computer"</td></tr><tr><td>[485]</td><td>"sparklyr"</td></tr><tr><td>[486]</td><td>"very"</td></tr></table>
[487] "Spark"
[488] "Development"
[489] "To"
[490] "fix"
[491] "feature"
[492] "Cheat"
[493] "Sheet"
[494] "starwars"
[495] "species"
[496] "Droid"
[497] "A"
[498] "tibble"
[499] "6"
[500] "x"
[501] "14"
[502] "name"
[503] "height"
[504] "mass"
[505] "hair_color"
[506] "skin_color"
[507] "eye_color"
[508] "birth_year"
[509] "sex"
[510] "gender"
[511] "chr"
[512] "int"
[513] "dbl"
[514] "1"
[515] "C-3PO"
[516] "167"
[517] "75"
[518] "NA"
[519] "gold"
[520] "yellow"
[521] "112"
[522] "none"
[523] "masculi"
[524] "2"
[525] "R2-D2"
[526] "96"
[527] "32"
[528] "white"
[529] "blue"
[530] "red"
[531] "33"
[532] "3"
<table><tr><td>[533]</td><td>"R5-D4"</td></tr><tr><td>[534]</td><td>"97"</td></tr><tr><td>[535]</td><td>"4"</td></tr><tr><td>[536]</td><td>"IG-88"</td></tr><tr><td>[537]</td><td>"200"</td></tr><tr><td>[538]</td><td>"140"</td></tr><tr><td>[539]</td><td>"metal"</td></tr><tr><td>[540]</td><td>"15"</td></tr><tr><td>[541]</td><td>"5"</td></tr><tr><td>[542]</td><td>"R4-P17"</td></tr><tr><td>[543]</td><td>"silver"</td></tr><tr><td>[544]</td><td>"feminine"</td></tr><tr><td>[545]</td><td>""</td></tr><tr><td>[546]</td><td>"row"</td></tr><tr><td>[547]</td><td>"homeworld"</td></tr><tr><td>[548]</td><td>"films"</td></tr><tr><td>[549]</td><td>"vehicles"</td></tr><tr><td>[550]</td><td>"starships"</td></tr><tr><td>[551]</td><td>"ends_with"</td></tr><tr><td>[552]</td><td>"color"</td></tr><tr><td>[553]</td><td>"87"</td></tr><tr><td>[554]</td><td>"Luke"</td></tr><tr><td>[555]</td><td>"Skywalker"</td></tr><tr><td>[556]</td><td>"blond"</td></tr><tr><td>[557]</td><td>"fair"</td></tr><tr><td>[558]</td><td>"Darth"</td></tr><tr><td>[559]</td><td>"Vader"</td></tr><tr><td>[560]</td><td>"Leia"</td></tr><tr><td>[561]</td><td>"Organa"</td></tr><tr><td>[562]</td><td>"brown"</td></tr><tr><td>[563]</td><td>"light"</td></tr><tr><td>[564]</td><td>"82"</td></tr><tr><td>[565]</td><td>"bmi"</td></tr><tr><td>[566]</td><td>"100"</td></tr><tr><td>[567]</td><td>"^"</td></tr><tr><td>[568]</td><td>"172"</td></tr><tr><td>[569]</td><td>"77"</td></tr><tr><td>[570]</td><td>"26.0"</td></tr><tr><td>[571]</td><td>"26.9"</td></tr><tr><td>[572]</td><td>"34.7"</td></tr><tr><td>[573]</td><td>"202"</td></tr><tr><td>[574]</td><td>"136"</td></tr><tr><td>[575]</td><td>"33.3"</td></tr><tr><td>[576]</td><td>"150"</td></tr><tr><td>[577]</td><td>"49"</td></tr><tr><td>[578]</td><td>"21.8"</td></tr></table>
## [579] "desc"
## [580] "Jabba"
## [581] "De"
## [582] "175"
## [583] "1358"
## [584] "green-tan"
## [585] "orange"
## [586] "600"
## [587] "herm"
## [588] "mascu"
## [589] "Grievous"
## [590] "216"
## [591] "159"
## [592] "wh"
## [593] "green"
## [594] "y"
## [595] "male"
## [596] "Va"
## [597] "41.9"
## [598] "Tarfful"
## [599] "234"
## [600] "n"
## [601] "mean"
## [602] "na.rm"
## [603] "TRUE"
## [604] "50"
## [605] "9"
## [606] "69.8"
## [607] "Gungan"
## [608] "74"
## [609] "Human"
## [610] "35"
## [611] "81.3"
## [612] "Kaminoan"
## [613] "88"
## [614] "Mirialan"
## [615] "53.1"
## [616] "encounter"
## [617] "clear"
## [618] "please"
## [619] "issue"
## [620] "minimal"
## [621] "discussion"
## [622] "community.rstudio.com"
## [623] "manipulatr"
## [624] "mailing"
<table><tr><td>[625]</td><td>"Please"</td></tr><tr><td>[626]</td><td>"note"</td></tr><tr><td>[627]</td><td>"this"</td></tr><tr><td>[628]</td><td>"project"</td></tr><tr><td>[629]</td><td>"released"</td></tr><tr><td>[630]</td><td>"Contributor"</td></tr><tr><td>[631]</td><td>"Conduct"</td></tr><tr><td>[632]</td><td>"By"</td></tr><tr><td>[633]</td><td>"participating"</td></tr><tr><td>[634]</td><td>"agree"</td></tr><tr><td>[635]</td><td>"abide"</td></tr><tr><td>[636]</td><td>"its"</td></tr><tr><td>[637]</td><td>"terms"</td></tr><tr><td>[638]</td><td>"Romain"</td></tr><tr><td>[639]</td><td>"François"</td></tr><tr><td>[640]</td><td>"Kirill"</td></tr><tr><td>[641]</td><td>"Müller"</td></tr><tr><td>[642]</td><td>"Davis"</td></tr><tr><td>[643]</td><td>"Vaughan"</td></tr><tr><td>[644]</td><td>"2.1.3"</td></tr><tr><td>[645]</td><td>"tidyr1.3.1"</td></tr><tr><td>[646]</td><td>"Tidy"</td></tr><tr><td>[647]</td><td>"Pivoting"</td></tr><tr><td>[648]</td><td>"Rectangling"</td></tr><tr><td>[649]</td><td>"Nested"</td></tr><tr><td>[650]</td><td>"1.3.0"</td></tr><tr><td>[651]</td><td>"1.2.0"</td></tr><tr><td>[652]</td><td>"0.7.0"</td></tr><tr><td>[653]</td><td>"0.6.0"</td></tr><tr><td>[654]</td><td>"0.5.0"</td></tr><tr><td>[655]</td><td>"0.4.0"</td></tr><tr><td>[656]</td><td>"0.3.0"</td></tr><tr><td>[657]</td><td>"0.2.0"</td></tr><tr><td>[658]</td><td>"0.1.0"</td></tr><tr><td>[659]</td><td>"tidyr"</td></tr><tr><td>[660]</td><td>"goal"</td></tr><tr><td>[661]</td><td>"tidy"</td></tr><tr><td>[662]</td><td>"where"</td></tr><tr><td>[663]</td><td>"Each"</td></tr><tr><td>[664]</td><td>"variable"</td></tr><tr><td>[665]</td><td>"column"</td></tr><tr><td>[666]</td><td>"each"</td></tr><tr><td>[667]</td><td>"observation"</td></tr><tr><td>[668]</td><td>"value"</td></tr><tr><td>[669]</td><td>"cell"</td></tr><tr><td>[670]</td><td>"standard"</td></tr></table>
[671] "storing"
[672] "wherever"
[673] "throughout"
[674] "ensure"
[675] "you'll"
[676] "spend"
[677] "less"
[678] "time"
[679] "fighting"
[680] "tools"
[681] "analysis"
[682] "tidy-data"
[683] "fall"
[684] "five"
[685] "categories"
[686] "converts"
[687] "between"
[688] "long"
[689] "wide"
[690] "forms"
[691] "introduces"
[692] "pivot_longer"
[693] "pivot_wider"
[694] "replacing"
[695] "older"
[696] "spread"
[697] "gather"
[698] "pivot"
[699] "turns"
[700] "deeply"
[701] "nested"
[702] "lists"
[703] "JSON"
[704] "unnest_longer"
[705] "unnest_wider"
[706] "hoist"
[707] "rectangle"
[708] "Nesting"
[709] "grouped"
[710] "form"
[711] "becomes"
[712] "containing"
[713] "frame"
[714] "unnesting"
[715] "does"
[716] "opposite"
## [717] "nest"
## [718] "unnest"
## [719] "Splitting"
## [720] "combining"
## [721] "character"
## [722] "columns"
## [723] "Use"
## [724] "separate_wider_delim"
## [725] "separate_wider_position"
## [726] "separate_wider match"
## [727] "pull"
## [728] "unite"
## [729] "Make"
## [730] "implicit"
## [731] "missing"
## [732] "explicit"
## [733] "complete"
## [734] "drop_na"
## [735] "replace"
## [736] "next"
## [737] "previous"
## [738] "fill"
## [739] "known"
## [740] "replace_na"
## [741] "Related"
## [742] "work"
## [743] "supersedes"
## [744] " reshape2"
## [745] "2010-2014"
## [746] " reshape"
## [747] "2005-2010"
## [748] "Somewhat"
## [749] "counterintuitively"
## [750] "iteration"
## [751] "package"
## [752] "has"
## [753] "done"
## [754] "tidying"
## [755] "not"
## [756] "general"
## [757] "reshaping"
## [758] "aggregation"
## [759] "high-performance"
## [760] "implementations"
## [761] "melt"
## [762] "dcast"
[763] "CS"

[764] "perspective"

[765] "I'd"

[766] "following"

[767] "three"

[768] "papers"

[769] "Wrangler"

[770] "Interactive"

[771] "visual"

[772] "specification"

[773] "scripts"

[774] "An"

[775] "interactive"

[776] "framework"

[777] "cleaning"

[778] "Potter's"

[779] "wheel"

[780] "On"

[781] "efficiently"

[782] "implementing"

[783] "SchemaSQL"

[784] "here's"

[785] "terminology"

[786] "different"

[787] "longer"

[788] "wider"

[789] "cast"

[790] "spreadsheets"

[791] "unpivot"

[792] "databases"

[793] "fold"

[794] "unfold"

[795] "github"

[796] "contributing"

[797] "Maximilian"

[798] "Girlich"

[799] "authors"

[800] "readr2.1.6"

[801] "Column"

[802] "type"

[803] "Locales"

[804] "1.4.0"

[805] "1.3.1"

[806] "readr"

[807] "fast"

[808] "rectangular"
## [809] "delimited"
## [810] "files"
## [811] "such"
## [812] "comma-separated"
## [813] "CSV"
## [814] "tab-separated"
## [815] "TSV"
## [816] "parse"
## [817] "many"
## [818] "found"
## [819] "wild"
## [820] "while"
## [821] "informative"
## [822] "report"
## [823] "parsing"
## [824] "leads"
## [825] "unexpected"
## [826] "results"
## [827] "import"
## [828] "part"
## [829] "core"
## [830] "so"
## [831] "load"
## [832] ""
## [833] "Attaching"
## [834] ""
## [835] "1.1.4"
## [836] "2.1.5.9000"
## [837] "forcats"
## [838] "1.0.1"
## [839] "stringr"
## [840] "1.5.2"
## [841] "lubridate"
## [842] "1.9.4"
## [843] "purrr"
## [844] "Conflicts"
## [845] "tidyverse_conflicts"
## [846] ""
## [847] "masks"
## [848] "stats"
## [849] "lag"
## [850] "conflicted"
## [851] "http://conflicted.r-lib.org/"
## [852] "force"
## [853] "conflicts"
## [854] "become"
[855] "errors"
[856] "Of"
[857] "function"
[858] "parses"
[859] "lines"
[860] "fields"
[861] "supports"
[862] "formats"
[863] "read_"
[864] "read_csv"
[865] "read_tsv"
[866] "read_csv2"
[867] "semicolon-separated"
[868] "decimal"
[869] "mark"
[870] "read_delim"
[871] "important"
[872] "special"
[873] "readfwf"
[874] "fixed-width"
[875] "read_table"
[876] "whitespace-separated"
[877] "read_log"
[878] "web"
[879] "log"
[880] "should"
[881] "converted"
[882] "vector"
[883] "specific"
[884] "e.g"
[885] "numeric"
[886] "datetime"
[887] "etc"
[888] "absence"
[889] "guess"
[890] "column-types"
[891] "gives"
[892] "detail"
[893] "guesses"
[894] "guessing"
[895] "handy"
[896] "especially"
[897] "during"
[898] "exploration"
[899] "but"
[900] "it's"
## [901] "remember"
## [902] "matures"
## [903] "past"
## [904] "exploratory"
## [905] "phase"
## [906] "strategy"
## [907] "loads"
## [908] "sample"
## [909] "bundled"
## [910] "chickens"
## [911] "readr_example"
## [912] "chickens.csv"
## [913] "Rows"
## [914] "Columns"
## [915] "Delimiter"
## [916] "chicken"
## [917] "motto"
## [918] "eggs_laid"
## [919] "^"
## [920] "spec"
## [921] "retrieve"
## [922] "full"
## [923] "Specify"
## [924] "show_col_types"
## [925] "FALSE"
## [926] "quiet"
## [927] "message"
## [928] "Foghorn"
## [929] "Leghorn"
## [930] "rooster"
## [931] "0"
## [932] "That's"
## [933] "joke"
## [934] "ah"
## [935] "say"
## [936] "that's"
## [937] "jok"
## [938] "Chicken"
## [939] "Little"
## [940] "hen"
## [941] "sky"
## [942] "falling"
## [943] "Ginger"
## [944] "12"
## [945] "Listen"
## [946] "We'll"
## [947] "either"
## [948] "die"
## [949] "free"
## [950] "chick"
## [951] "Camilla"
## [952] "7"
## [953] "Bawk"
## [954] "buck"
## [955] "ba-gawk"
## [956] "Ernie"
## [957] "Giant"
## [958] "Put"
## [959] "Captain"
## [960] "Solo"
## [961] "cargo"
## [962] "hold"
## [963] "Note"
## [964] "prints"
## [965] "guessed"
## [966] "case"
## [967] "useful"
## [968] "check"
## [969] "been"
## [970] "expect"
## [971] "haven't"
## [972] "need"
## [973] "sounds"
## [974] "lot"
## [975] "trouble"
## [976] "luckily"
## [977] "affords"
## [978] "nice"
## [979] "workflow"
## [980] "initial"
## [981] "effort"
## [982] "cols"
## [983] "col_character"
## [984] "col_double"
## [985] "Now"
## [986] "copy"
## [987] "paste"
## [988] "tweak"
## [989] "call"
## [990] "expresses"
## [991] "desired"
## [992] "Here"
## [993] "express"
## [994] "factor"
## [995] "levels"
## [996] "order"
## [997] "integer"
## [998] "col_types"
## [999] "col_factor"
## [1000] "c"
## [1001] "col_integer"
## [1002] "fct"
## [1003] "expanded"
## [1004] "Editions"
## [1005] "got"
## [1006] "July"
## [1007] "2021"
## [1008] "so-called"
## [1009] "second"
## [1010] "edition"
## [1011] "calls"
## [1012] "vroom"
## [1013] "default"
## [1014] "versions"
## [1015] "prior"
## [1016] "called"
## [1017] "first"
## [1018] "you're"
## [1019] "access"
## [1020] "via"
## [1021] "with_edition"
## [1022] "local_edition"
## [1023] "And"
## [1024] "obviously"
## [1025] "definition"
## [1026] "that's"
## [1027] "We"
## [1028] "continue"
## [1029] "support"
## [1030] "number"
## [1031] "releases"
## [1032] "overall"
## [1033] "uniformly"
## [1034] "Therefore"
## [1035] "plan"
## [1036] "eventually"
## [1037] "deprecated"
## [1038] "remove"
[1039] "New"

[1040] "actively-maintained"

[1041] "workarounds"

[1042] "offered"

[1043] "pragmatic"

[1044] "patch"

[1045] "legacy"

[1046] "temporary"

[1047] "solution"

[1048] "infelicities"

[1049] "identified"

[1050] "Alternatives"

[1051] "alternatives"

[1052] "data.table's"

[1053] "fread"

[1054] "differences"

[1055] "discussed"

[1056] "below"

[1057] "Base"

[1058] "Compared"

[1059] "corresponding"

[1060] "naming"

[1061] "scheme"

[1062] "parameters"

[1063] "col_names"

[1064] "header"

[1065] "colClasses"

[1066] "Are"

[1067] "much"

[1068] "faster"

[1069] "10x-100x"

[1070] "depending"

[1071] "Leave"

[1072] "strings"

[1073] "automatically"

[1074] "date"

[1075] "Have"

[1076] "helpful"

[1077] "progress"

[1078] "bar"

[1079] "loading"

[1080] "going"

[1081] "All"

[1082] "exactly"

[1083] "same"

[1084] "regardless"
<table><tr><td>[1085]</td><td>"current"</td></tr><tr><td>[1086]</td><td>"locale"</td></tr><tr><td>[1087]</td><td>"override"</td></tr><tr><td>[1088]</td><td>"US-centric"</td></tr><tr><td>[1089]</td><td>"defaults"</td></tr><tr><td>[1090]</td><td>"similar"</td></tr><tr><td>[1091]</td><td>"sometimes"</td></tr><tr><td>[1092]</td><td>"slower"</td></tr><tr><td>[1093]</td><td>"particularly"</td></tr><tr><td>[1094]</td><td>"heavy"</td></tr><tr><td>[1095]</td><td>"Can"</td></tr><tr><td>[1096]</td><td>"some"</td></tr><tr><td>[1097]</td><td>"basically"</td></tr><tr><td>[1098]</td><td>"encourage"</td></tr><tr><td>[1099]</td><td>"delimiter"</td></tr><tr><td>[1100]</td><td>"skipped"</td></tr><tr><td>[1101]</td><td>"Follow"</td></tr><tr><td>[1102]</td><td>"tidyverse-wide"</td></tr><tr><td>[1103]</td><td>"conventions"</td></tr><tr><td>[1104]</td><td>"returning"</td></tr><tr><td>[1105]</td><td>"approach"</td></tr><tr><td>[1106]</td><td>"repair"</td></tr><tr><td>[1107]</td><td>"mini-language"</td></tr><tr><td>[1108]</td><td>"selection"</td></tr><tr><td>[1109]</td><td>"Acknowledgements"</td></tr><tr><td>[1110]</td><td>"Thanks"</td></tr><tr><td>[1111]</td><td>"Joe"</td></tr><tr><td>[1112]</td><td>"Cheng"</td></tr><tr><td>[1113]</td><td>"showing"</td></tr><tr><td>[1114]</td><td>"me"</td></tr><tr><td>[1115]</td><td>"beauty"</td></tr><tr><td>[1116]</td><td>"deterministic"</td></tr><tr><td>[1117]</td><td>"finite"</td></tr><tr><td>[1118]</td><td>"automata"</td></tr><tr><td>[1119]</td><td>"teaching"</td></tr><tr><td>[1120]</td><td>"why"</td></tr><tr><td>[1121]</td><td>"write"</td></tr><tr><td>[1122]</td><td>" tokenizer"</td></tr><tr><td>[1123]</td><td>"JJ"</td></tr><tr><td>[1124]</td><td>"Allaire"</td></tr><tr><td>[1125]</td><td>"helping"</td></tr><tr><td>[1126]</td><td>"come"</td></tr><tr><td>[1127]</td><td>"design"</td></tr><tr><td>[1128]</td><td>"few"</td></tr><tr><td>[1129]</td><td>"easy"</td></tr><tr><td>[1130]</td><td>"extend"</td></tr></table>
[1131] "Dirk"

[1132] "Eddelbuettel"

[1133] "coming"

[1134] "Jim"

[1135] "Hester"

[1136] "Jennifer"

[1137] "Bryan"

[1138] "purrr1.2.0"

[1139] "Functional"

[1140] "programming"

[1141] "languages"

[1142] "0.2.3"

[1143] "enhances"

[1144] "R's"

[1145] "functional"

[1146] "FP"

[1147] "toolkit"

[1148] "vectors"

[1149] "never"

[1150] "heard"

[1151] "before"

[1152] "family"

[1153] "allow"

[1154] "loops"

[1155] "both"

[1156] "succinct"

[1157] "easier"

[1158] "uses"

[1159] "fairly"

[1160] "realistic"

[1161] "split"

[1162] "model"

[1163] "piece"

[1164] "compute"

[1165] "extract"

[1166] "R2"

[1167] "mtcars"

[1168] "|"

[1169] "$"

[1170] "cyl"

[1171] "df"

[1172] "lm"

[1173] "~"

[1174] "wt"

[1175] "map_db1"

[1176] "r.squared"
## [1177] "8"
## [1178] "0.5086326"
## [1179] "0.4645102"
## [1180] "0.4229655"
## [1181] "advantages"
## [1182] "equivalents"
## [1183] "argument"
## [1184] "always"
## [1185] "pipe"
## [1186] "type-stable"
## [1187] "They"
## [1188] "return"
## [1189] "advertised"
## [1190] "output"
## [1191] "returns"
## [1192] "double"
## [1193] "throw"
## [1194] "error"
## [1195] "accept"
## [1196] "named"
## [1197] "anonymous"
## [1198] "lambda"
## [1199] "components"
## [1200] "position"
## [1201] "obvious"
## [1202] "easily"
## [1203] "track"
## [1204] "running"
## [1205] "jobs"
## [1206] "in_parallel"
## [1207] "computation"
## [1208] "across"
## [1209] "cores"
## [1210] "machines"
## [1211] "network"
## [1212] "tibble3.3.0"
## [1213] "Controlling"
## [1214] "display"
## [1215] "numbers"
## [1216] "Comparing"
## [1217] "Invariants"
## [1218] "behavior"
## [1219] "2.1.1"
## [1220] "2.0.1"
## [1221] "pre-announcement"
## [1222] "1.4.2"
[1223] "1.4.1"
[1224] "tbl DF"
[1225] "modern"
[1226] "reimagining"
[1227] "data.frame"
[1228] "keeping"
[1229] "proven"
[1230] "effective"
[1231] "throwing"
[1232] "out"
[1233] "Tibbles"
[1234] "data fr ams"
[1235] "la zy"
[1236] "surly"
[1237] "i.e"
[1238] "don't"
[1239] "change"
[1240] "partial"
[1241] "matching"
[1242] "complain"
[1243] "exist"
[1244] "forces"
[1245] "confront"
[1246] "earlier"
[1247] "typically"
[1248] "leading"
[1249] "cleaner"
[1250] "expressive"
[1251] "enhanced"
[1252] "print"
[1253] "method"
[1254] "complex"
[1255] "objects"
[1256] "science"
[1257] "Create"
[1258] "object"
[1259] "as_tibble"
[1260] "b"
[1261] "letters"
[1262] "Sys.Date"
[1263] "2025-03-18"
[1264] "2025-03-17"
[1265] "2025-03-16"
[1266] "reasonable"
[1267] "inputs"
[1268] "already"
## [1269] "matrices"
## [1270] "tables"
## [1271] "x"
## [1272] "z"
## [1273] "17"
## [1274] "26"
## [1275] "keeps"
## [1276] "only"
## [1277] "recycles"
## [1278] "length"
## [1279] "creates"
## [1280] "row.names"
## [1281] "features"
## [1282] "define"
## [1283] "row-by-row"
## [1284] "tribble"
## [1285] "3.6"
## [1286] "8.5"
## [1287] "draws"
## [1288] "inspiration"
## [1289] "Like"
## [1290] "doesn't"
## [1291] "rownames"
## [1292] "2.1.3.9000"
## [1293] "stringr1.6.0"
## [1294] "From"
## [1295] "Locale"
## [1296] "sensitive"
## [1297] "Regular"
## [1298] "expressions"
## [1299] "1.6.0"
## [1300] "1.5.0"
## [1301] "Strings"
## [1302] "glamorous"
## [1303] "high-profile"
## [1304] "play"
## [1305] "big"
## [1306] "role"
## [1307] "preparation"
## [1308] "tasks"
## [1309] "cohesive"
## [1310] "familiar"
## [1311] "top"
## [1312] "stringi"
## [1313] "ICU"
## [1314] "correct"
[1315] "string"

[1316] "manipulations"

[1317] "focusses"

[1318] "commonly"

[1319] "whereas"

[1320] "covering"

[1321] "almost"

[1322] "anything"

[1323] "imagine"

[1324] "find"

[1325] "Both"

[1326] "share"

[1327] "similarly"

[1328] "str_"

[1329] "video"

[1330] "cross"

[1331] "deal"

[1332] "authority"

[1333] "str_length"

[1334] "str_c"

[1335] "collapse"

[1336] "str_sub"

[1337] "vi"

[1338] "cr"

[1339] "ex"

[1340] "de"

[1341] "au"

[1342] "Most"

[1343] "regular"

[1344] "concise"

[1345] "language"

[1346] "describing"

[1347] "patterns"

[1348] "text"

[1349] "expression"

[1350] "aeiou"

[1351] "matches"

[1352] "vowel"

[1353] "str_subset"

[1354] "str_count"

[1355] "seven"

[1356] "str_detect"

[1357] "pattern"

[1358] "tells"

[1359] "there's"

[1360] "match"
## [1361] "counts"
## [1362] "extracts"
## [1363] "str locate"
## [1364] "end"
## [1365] "str extract"
## [1366] "i"
## [1367] "o"
## [1368] "e"
## [1369] "str match"
## [1370] "parts"
## [1371] "defined"
## [1372] "parentheses"
## [1373] "characters"
## [1374] "side"
## [1375] "vid"
## [1376] "v"
## [1377] "d"
## [1378] "ros"
## [1379] "r"
## [1380] "dea"
## [1381] "aut"
## [1382] "t"
## [1383] "str replace"
## [1384] "replacement"
## [1385] "replaces"
## [1386] "deo"
## [1387] "ss"
## [1388] "xtra"
## [1389] "al"
## [1390] "authority"
## [1391] "str split"
## [1392] "splits"
## [1393] "engines"
## [1394] "fixed"
## [1395] "exact"
## [1396] "bytes"
## [1397] "coll"
## [1398] "human"
## [1399] "boundary"
## [1400] "boundaries"
## [1401] "Addin"
## [1402] "RegExplain"
## [1403] "addin"
## [1404] "interface"
## [1405] "interactively"
## [1406] "build"
[1407] "regexp"
[1408] "consult"
[1409] "included"
[1410] "resources"
[1411] "installed"
[1412] "devtools"
[1413] "install_github"
[1414] "gadenbuie"
[1415] "regexplain"
[1416] "solid"
[1417] "grown"
[1418] "organically"
[1419] "inconsistent"
[1420] "Additionally"
[1421] "behind"
[1422] "things"
[1423] "Ruby"
[1424] "Python"
[1425] "Uses"
[1426] "modify"
[1427] "conjunction"
[1428] "str_pad"
[1429] "right"
[1430] "11"
[1431] "f"
[1432] "g"
[1433] "h"
[1434] "j"
[1435] "k"
[1436] "Simplifies"
[1437] "eliminating"
[1438] "options"
[1439] "95"
[1440] "Produces"
[1441] "outputs"
[1442] "includes"
[1443] "ensuring"
[1444] "result"
[1445] "from-base"
[1446] "R4DS"
[1447] "copyright"
[1448] "forcats1.0.1"
[1449] "factors"
[1450] "handle"
[1451] "categorical"
[1452] "Factors"
## [1453] "reordering"
## [1454] "improve"
## [1455] "suite"
## [1456] "Some"
## [1457] "examples"
## [1458] "include"
## [1459] "fct_reorder"
## [1460] "another"
## [1461] "fct_infreq"
## [1462] "frequency"
## [1463] "fct_relevel"
## [1464] "Changing"
## [1465] "hand"
## [1466] "fct_lump"
## [1467] "Collapsing"
## [1468] "least"
## [1469] "frequent"
## [1470] "is.na"
## [1471] "count"
## [1472] "sort"
## [1473] "37"
## [1474] "Twi'lek"
## [1475] "Wookiee"
## [1476] "Zabrak"
## [1477] "Aleena"
## [1478] "Besalisk"
## [1479] "27"
## [1480] "Other"
## [1481] "39"
## [1482] "geom_bar"
## [1483] "history"
## [1484] "stringsAsFactors"
## [1485] "unauthorized"
## [1486] "biography"
## [1487] "Roger"
## [1488] "Peng"
## [1489] "sigh"
## [1490] "Lumley"
## [1491] "approaches"
## [1492] "Wrangling"
## [1493] "Amelia"
## [1494] "McNamara"
## [1495] "Nicholas"
## [1496] "Horton"
## [1497] "Github"
## [1498] "https://forum.posit.co/."
[1499] "lubridate1.9.4"
[1500] "1.7.0"
[1501] "Date-time"
[1502] "frustrating"
[1503] "commands"
[1504] "date-times"
[1505] "unintuitive"
[1506] "date-time"
[1507] "being"
[1508] "Moreover"
[1509] "methods"
[1510] "must"
[1511] "robust"
[1512] "zones"
[1513] "leap"
[1514] "days"
[1515] "daylight"
[1516] "savings"
[1517] "times"
[1518] "related"
[1519] "quirks"
[1520] "lacks"
[1521] "capabilities"
[1522] "situations"
[1523] "Lubridate"
[1524] "Features"
[1525] "warn.conflicts"
[1526] "Easy"
[1527] "ymd"
[1528] "ymd_hms"
[1529] "dmy"
[1530] "dmy_hms"
[1531] "mdy"
[1532] "20101215"
[1533] "2010-12-15"
[1534] "2017-04-01"
[1535] "Simple"
[1536] "year"
[1537] "month"
[1538] "mday"
[1539] "hour"
[1540] "minute"
[1541] "bday"
[1542] "1979"
[1543] "wday"
[1544] "label"
## [1545] "Sun"
## [1546] "Levels"
## [1547] "Mon"
## [1548] "Tue"
## [1549] "Wed"
## [1550] "Thu"
## [1551] "Fri"
## [1552] "Sat"
## [1553] "2016"
## [1554] "Helper"
## [1555] "handling"
## [1556] "with_tz"
## [1557] "force_tz"
## [1558] "2010-12-13"
## [1559] "30"
## [1560] "UTC"
## [1561] "Changes"
## [1562] "printing"
## [1563] "America"
## [1564] "Chicago"
## [1565] "09"
## [1566] "CST"
## [1567] "expands"
## [1568] "mathematical"
## [1569] "performed"
## [1570] "span"
## [1571] "classes"
## [1572] "borrowed"
## [1573] "https://www.joda.org"
## [1574] "durations"
## [1575] "measure"
## [1576] "amount"
## [1577] "points"
## [1578] "periods"
## [1579] "accurately"
## [1580] "clock"
## [1581] "despite"
## [1582] "seconds"
## [1583] "day"
## [1584] "intervals"
## [1585] "protean"
## [1586] "information"
## [1587] "GPL"
## [1588] "Vitalie"
## [1589] "Spinu"
## [1590] "Garrett"
## [1591] "Grolemund"

### 4.8.6.5 Keyword-in-context (kwic)

#### 4.8.6.5.1 Pattern
You can see how keywords are used in the actual contexts in a concordance view produced by kwic(). Pattern is used to search for the pattern we are interested in (with * as a wildcard). Window used to display number of words/tokens around it.

web_pages_txt_corpus_tokens_no_punct_types_tidy <- kwic(web_pages_txt_corpus_tokens_no_punct, head(web_pages_txt_corpus_tokens_no_punct_types_tidy, 10))

## Keyword-in-context with 10 matches.

## [text1, 113] ggplot2 is to install the whole | tidyverse |

## [text1, 115] to install the whole tidyverse install.packages | tidyverse |

## [text1, 132] from GitHub install.packages pak pak pak | tidyverse |

## [text1, 351] you a comprehensive introduction to the | tidyverse |

## [text1, 505] and deep-dives you can visit the | tidyverse |

## [text2, 318] dplyr is to install the whole | tidyverse |

## [text2, 320] to install the whole tidyverse install.packages | tidyverse |

## [text2, 357] from GitHub install.packages pak pak pak | tidyverse |

## [text3, 4] Skip to content | tidyr1.3.1 |

## [text3, 5] Skip to content tidyr1.3.1 | Tidy |

##

## install.packages tidyverse Alternatively install just ggplot2

## Alternatively install just ggplot2 install.packages ggplot2

## ggplot2 Cheatsheet Usage It's hard to

## and these two chapters will get

## blog Getting help There are two

## install.packages tidyverse Alternatively install just dplyr

## Alternatively install just dplyr install.packages dplyr

## dplyr Cheat Sheet Usage library dplyr

## Tidy data Reference Articles Pivoting Rectangling

## data Reference Articles Pivoting Rectangling Nested

web_pages_txt_corpus_tokens_no_punct_phrase <- kwic(web_pages_txt_corpus_tokens_no_punct, head(web_pages_txt_corpus_tokens_no_punct_phrase, 10))

#### 4.8.6.5.2 Phrase

## Keyword-in-context with 4 matches.

## [text1, 350:351] give you a comprehensive introduction to |

## [text1, 504:505] announcements and deep-dives you can visit |

## [text3, 109:110] that is used wherever possible throughout |

## [text3, 477:478] please use community.rstudio.com Please note that |

##
## text6 :
## [1] "Skip" "content" "tibble3.3.0" "started" "Reference"
## [6] "Articles" "Column" "types" "Controlling" "display"
## [11] "Comparing" "display"
## [ ... and 338 more ]
##
## [ reached max_ndoc ... 3 more documents ]

4.8.6.7 wordstem

To be able to extract the stems of each of the given words, we use the function
`tokens_wordstem`, `char_wordstem` or `dfm_wordstem`

web_pages_txt_corpus tok_no_punct_no_Stop_stem <- tokens_wordstem(web_pages_txt_corpus web_pages_txt_corpus tok_no_punct_no_Stop_stem

4.8.6.7.1 tokens_wordstem

## Tokens consisting of 9 documents and 4 docvars.

text1 :

## [1] "Skip" "content" "ggplot24.0.1" "start" "Refer"
## [6] "New" "Releas" "Version" "4.0.0" "Version"
## [11] "3.5.0" "Version"
## [ ... and 349 more ]

##

text2 :

## [1] "Skip" "content" "dplyr1.1.4" "start" "Refer"
## [6] "Articl" "data" "Two-tabl" "verb" "dplyr"
## [11] "<" ">"
## [ ... and 733 more ]

##

text3 :

## [1] "Skip" "content" "tidyr1.3.1" "Tidi" "data"
## [6] "Refer" "Articl" "Pivot" "Rectangl" "Nest"
## [11] "data" "articl"
## [ ... and 322 more ]

##

text4 :

## [1] "Skip" "content" "readr2.1.6" "start" "Refer"
## [6] "Articl" "Column" "type" "Local" "New"
## [11] "Releas" "Version"
## [ ... and 890 more ]

##

text5 :

## [1] "Skip" "content" "purrr1.2.0" "start" "Refer"
## [6] "Articl" "purrr" "<" ">" "base"
## [11] "Function" "program"
## [ ... and 230 more ]
##
## text6 :
## [1] "Skip" "content" "tibble3.3.0" "start" "Refer"
## [6] "Article" "Column" "type" "Control" "display"
## [11] "Compar" "display"
## [ ... and 338 more ]
##
## [ reached max_ndoc ... 3 more documents ]

4.8.6.7.2 dfm_wordstem Here we can use the dfm (for Document Feature Matrix) to obtain details of the wordstems used in each of the texts

web_pages_txt_corpus tok_no_punct_no_STOP_stem dfm <- dfm(web_pages_txt_corpus tok_no_punct_no_STOP_stem dfm)
dfm_wordstem(web_pages_txt_corpus tok_no_punct_no_STOP_stem dfm)

## Document-feature matrix of: 9 documents, 1,022 features (79.66% sparse) and 4 docvars.
## features
## docs skip content ggplot24.0.1 start refer news releas version 4.0.0 3.5.0
## text1 1 1 1 4 1 1 1 12 1 1
## text2 1 1 0 2 1 1 2 10 0 0
## text3 1 1 0 1 1 1 2 13 0 0
## text4 2 1 0 2 1 1 3 9 1 0
## text5 1 1 0 2 1 1 1 6 0 0
## text6 1 1 0 2 1 1 1 7 0 0
## [ reached max_ndoc ... 3 more documents, reached max_nfeat ... 1,012 more features ]

4.9 session info

sessionInfo()

## R version 4.5.2 (2025-10-31 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
##
## Matrix products: default
## LAPACK version 3.12.1
##
## locale:
## [1] LC_COLLATE=English_United Kingdom.utf8
## [2] LC_CTYPE=English_United Kingdom.utf8
## [3] LC_MONETARY=English_United Kingdom.utf8
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United Kingdom.utf8
<table>
  <tr>
    <td>[22]</td>
    <td>abind_1.4-8</td>
    <td>quadprog_1.5-8</td>
    <td>RCurl_1.98-1.17</td>
  </tr>
  <tr>
    <td>[25]</td>
    <td>nnet_7.3-20</td>
    <td>TH.data_1.1-5</td>
    <td>sandwich_3.1-1</td>
  </tr>
  <tr>
    <td>[28]</td>
    <td>labelled_2.16.0</td>
    <td>ggrepel_0.9.6</td>
    <td>tokenizers_0.3.0</td>
  </tr>
  <tr>
    <td>[31]</td>
    <td>listenv_0.10.0</td>
    <td>correlation_0.8.8</td>
    <td>cards_0.7.0</td>
  </tr>
  <tr>
    <td>[34]</td>
    <td>performance_0.15.2</td>
    <td>parallelly_1.45.1</td>
    <td>commonmark_2.0.0</td>
  </tr>
  <tr>
    <td>[37]</td>
    <td>codetools_0.2-20</td>
    <td>pdftools_3.6.0</td>
    <td>xml2_1.5.0</td>
  </tr>
  <tr>
    <td>[40]</td>
    <td>tidyselect_1.2.1</td>
    <td>ndjson_0.9.1</td>
    <td>ggeffects_2.3.1</td>
  </tr>
  <tr>
    <td>[43]</td>
    <td>farver_2.1.2</td>
    <td>broom.mixed_0.2.9.6</td>
    <td>matrixStats_1.5.0</td>
  </tr>
  <tr>
    <td>[46]</td>
    <td>base64enc_0.1-3</td>
    <td>broom.helpers_1.22.0</td>
    <td>Formula_1.2-5</td>
  </tr>
  <tr>
    <td>[49]</td>
    <td>survival_3.8-3</td>
    <td>tools_4.5.2</td>
    <td>pryr_0.1.6</td>
  </tr>
  <tr>
    <td>[52]</td>
    <td>Rcpp_1.1.0</td>
    <td>glue_1.8.0</td>
    <td>gridExtra_2.3</td>
  </tr>
  <tr>
    <td>[55]</td>
    <td>xfun_0.54</td>
    <td>distributional_0.5.0</td>
    <td>withr_3.0.2</td>
  </tr>
  <tr>
    <td>[58]</td>
    <td>numDeriv_2016.8-1.1</td>
    <td>fastmap_1.2.0</td>
    <td>boot_1.3-32</td>
  </tr>
  <tr>
    <td>[61]</td>
    <td>streamR_0.4.5</td>
    <td>litedown_0.8</td>
    <td>digest_0.6.39</td>
  </tr>
  <tr>
    <td>[64]</td>
    <td>timechange_0.3.0</td>
    <td>R6_2.6.1</td>
    <td>estimability_1.5.1</td>
  </tr>
  <tr>
    <td>[67]</td>
    <td>qpdf_1.4.1</td>
    <td>colorspace_2.1-2</td>
    <td> markdown_2.0</td>
  </tr>
  <tr>
    <td>[70]</td>
    <td>utf8_1.2.6</td>
    <td>generics_0.1.4</td>
    <td>data.table_1.17.8</td>
  </tr>
  <tr>
    <td>[73]</td>
    <td>httr_1.4.7</td>
    <td>htmlwidgets_1.6.4</td>
    <td>pkgconfig_2.0.3</td>
  </tr>
  <tr>
    <td>[76]</td>
    <td>gtable_0.3.6</td>
    <td>statsExpressions_1.7.1</td>
    <td>S7_0.2.1</td>
  </tr>
  <tr>
    <td>[79]</td>
    <td>selectr_0.5-0</td>
    <td>furrr_0.3.1</td>
    <td>bookdown_0.45</td>
  </tr>
  <tr>
    <td>[82]</td>
    <td>scales_1.4.0</td>
    <td>attempt_0.3.1</td>
    <td>reformulas_0.4.2</td>
  </tr>
  <tr>
    <td>[85]</td>
    <td>snakecase_0.11.1</td>
    <td>splancs_2.01-45</td>
    <td>rstudioapi_0.17.1</td>
  </tr>
  <tr>
    <td>[88]</td>
    <td>rjson_0.2.23</td>
    <td>tzdb_0.5.0</td>
    <td>reshape2_1.4.5</td>
  </tr>
  <tr>
    <td>[91]</td>
    <td>curl_7.0.0</td>
    <td>coda_0.19-4.1</td>
    <td>checkmate_2.3.3</td>
  </tr>
  <tr>
    <td>[94]</td>
    <td>nloptr_2.2.1</td>
    <td>zoo_1.8-14</td>
    <td>sjlabelled_1.2.0</td>
  </tr>
  <tr>
    <td>[97]</td>
    <td>parallel_4.5.2</td>
    <td>foreign_0.8-90</td>
    <td>pillar_1.11.1</td>
  </tr>
  <tr>
    <td>[100]</td>
    <td>grid_4.5.2</td>
    <td>vctrs_0.6.5</td>
    <td>slam_0.1-55</td>
  </tr>
  <tr>
    <td>[103]</td>
    <td>ucminf_1.2.2</td>
    <td>xtable_1.8-4</td>
    <td>cluster_2.1.8.1</td>
  </tr>
  <tr>
    <td>[106]</td>
    <td>htmlTable_2.4.3</td>
    <td>evaluate_1.0.5</td>
    <td>zeallot_0.2.0</td>
  </tr>
  <tr>
    <td>[109]</td>
    <td>tinytex_0.58</td>
    <td>magick_2.9.0</td>
    <td>readMDTable_0.3.2</td>
  </tr>
  <tr>
    <td>[112]</td>
    <td>mvtnorm_1.3-3</td>
    <td>cli_3.6.5</td>
    <td>compiler_4.5.2</td>
  </tr>
  <tr>
    <td>[115]</td>
    <td>rlang_1.1.6</td>
    <td>crayon_1.5.3</td>
    <td>rstantools_2.5.0</td>
  </tr>
  <tr>
    <td>[118]</td>
    <td>labeling_0.4.3</td>
    <td>rematch2_2.1.2</td>
    <td>plyr_1.8.9</td>
  </tr>
  <tr>
    <td>[121]</td>
    <td>sjmisc_2.8.11</td>
    <td>fs_1.6.6</td>
    <td>gap datasets_0.0.6</td>
  </tr>
  <tr>
    <td>[124]</td>
    <td>stringi_1.8.7</td>
    <td>pander_0.6.6</td>
    <td>bayestestR_0.17.0</td>
  </tr>
  <tr>
    <td>[127]</td>
    <td>rapportools_1.2</td>
    <td>hms_1.1.4</td>
    <td>patchwork_1.3.2</td>
  </tr>
  <tr>
    <td>[130]</td>
    <td>bit64_4.6.0-1</td>
    <td>future_1.68.0</td>
    <td>haven_2.5.5</td>
  </tr>
  <tr>
    <td>[133]</td>
    <td>rbibutils_2.4</td>
    <td>gt_1.1.0</td>
    <td>RcppParallel_5.1.11-1</td>
  </tr>
  <tr>
    <td>[136]</td>
    <td>fastmatch_1.1-6</td>
    <td>bit_4.6.0</td>
    <td></td>
  </tr>
</table>
## 5.2 jane austen

### 5.2.1 Import into Global Environment

Adding data from the first book into the Global Environment

```
data(sensesensibility)
#data(prideprejudice)
#data(mansfieldpark)
#data(emma)
#data(northangerabbey)
#data(persuasion)
```

### 5.2.2 Look into data

And we can get the top 60 rows from "sensesensibility"

```
sensesensibility %>%
  head(n = 60)

## [1] "SENSE AND SENSIBILITY"
## [2]"
## [3] "by Jane Austen"
## [4]"
## [5] "(1811)"
## [6]"
## [7]"
## [8]"
## [9]"
## [10] "CHAPTER 1"
## [11]"
## [12]"
## [13] "The family of Dashwood had long been settled in Sussex. Their estate"
## [14] "was large, and their residence was at Norland Park, in the centre of"
## [15] "their property, where, for many generations, they had lived in so"
## [16] "respectable a manner as to engage the general good opinion of their"
## [17] "surrounding acquaintance. The late owner of this estate was a single"
## [18] "man, who lived to a very advanced age, and who for many years of his"
## [19] "life, had a constant companion and housekeeper in his sister. But her"
## [20] "death, which happened ten years before his own, produced a great"
## [21] "alteration in his home; for to supply her loss, he invited and received"
## [22] "into his house the family of his nephew Mr. Henry Dashwood, the legal"
## [23] "inheritor of the Norland estate, and the person to whom he intended to"
## [24] "bequeath it. In the society of his nephew and niece, and their"
## [25] "children, the old Gentleman's days were comfortably spent. His"
## [26] "attachment to them all increased. The constant attention of Mr. and"
## [27] "Mrs. Henry Dashwood to his wishes, which proceeded not merely from"
```
[28] "interest, but from goodness of heart, gave him every degree of solid"

[29] "comfort which his age could receive; and the cheerfulness of the"

[30] "children added a relish to his existence."

[31] ""

[32] "By a former marriage, Mr. Henry Dashwood had one son: by his present"

[33] "lady, three daughters. The son, a steady respectable young man, was"

[34] "amply provided for by the fortune of his mother, which had been large,"

[35] "and half of which devolved on him on his coming of age. By his own"

[36] "marriage, likewise, which happened soon afterwards, he added to his"

[37] "wealth. To him therefore the succession to the Norland estate was not"

[38] "so really important as to his sisters; for their fortune, independent"

[39] "of what might arise to them from their father's inheriting that"

[40] "property, could be but small. Their mother had nothing, and their"

[41] "father only seven thousand pounds in his own disposal; for the"

[42] "remaining moiety of his first wife's fortune was also secured to her"

[43] "child, and he had only a life-interest in it."

[44] ""

[45] "The old gentleman died: his will was read, and like almost every other"

[46] "will, gave as much disappointment as pleasure. He was neither so"

[47] "unjust, nor so ungrateful, as to leave his estate from his nephew;--but"

[48] "he left it to him on such terms as destroyed half the value of the"

[49] "bequest. Mr. Dashwood had wished for it more for the sake of his wife"

[50] "and daughters than for himself or his son;--but to his son, and his"

[51] "son's son, a child of four years old, it was secured, in such a way, as"

[52] "to leave to himself no power of providing for those who were most dear"

[53] "to him, and who most needed a provision by any charge on the estate, or"

[54] "by any sale of its valuable woods. The whole was tied up for the"

[55] "benefit of this child, who, in occasional visits with his father and"

[56] "mother at Norland, had so far gained on the affections of his uncle, by"

[57] "such attractions as are by no means unusual in children of two or three"

[58] "years old; an imperfect articulation, an earnest desire of having his"

[59] "own way, many cunning tricks, and a great deal of noise, as to outweigh"

[60] "all the value of all the attention which, for years, he had received"

### 5.2.3 Transform to a data frame

```
sensesensibility DF <- sensesensibility %>%
  data.frame()

sensesensibility DF <- sensesensibility DF[-c(1:12),]

sensesensibility DF %>%
  head(60)
```

[1] "The family of Dashwood had long been settled in Sussex. Their estate"

[2] "was large, and their residence was at Norland Park, in the centre of"

[3] "their property, where, for many generations, they had lived in so"

[4] "respectable a manner as to engage the general good opinion of their"
[5] "surrounding acquaintance. The late owner of this estate was a single"

[6] "man, who lived to a very advanced age, and who for many years of his"

[7] "life, had a constant companion and housekeeper in his sister. But her"

[8] "death, which happened ten years before his own, produced a great"

[9] "alteration in his home; for to supply her loss, he invited and received"

[10] "into his house the family of his nephew Mr. Henry Dashwood, the legal"

[11] "inheritor of the Norland estate, and the person to whom he intended to"

[12] "bequeath it. In the society of his nephew and niece, and their"

[13] "children, the old Gentleman's days were comfortably spent. His"

[14] "attachment to them all increased. The constant attention of Mr. and"

[15] "Mrs. Henry Dashwood to his wishes, which proceeded not merely from"

[16] "interest, but from goodness of heart, gave him every degree of solid"

[17] "comfort which his age could receive; and the cheerfulness of the"

[18] "children added a relish to his existence."

[19] ""

[20] "By a former marriage, Mr. Henry Dashwood had one son: by his present"

[21] "lady, three daughters. The son, a steady respectable young man, was"

[22] "amply provided for by the fortune of his mother, which had been large,"

[23] "and half of which devolved on him on his coming of age. By his own"

[24] "marriage, likewise, which happened soon afterwards, he added to his"

[25] "wealth. To him therefore the succession to the Norland estate was not"

[26] "so really important as to his sisters; for their fortune, independent"

[27] "of what might arise to them from their father's inheriting that"

[28] "property, could be but small. Their mother had nothing, and their"

[29] "father only seven thousand pounds in his own disposal; for the"

[30] "remaining moiety of his first wife's fortune was also secured to her"

[31] "child, and he had only a life-interest in it."

[32] ""

[33] "The old gentleman died: his will was read, and like almost every other"

[34] "will, gave as much disappointment as pleasure. He was neither so"

[35] "unjust, nor so ungrateful, as to leave his estate from his nephew;--but"

[36] "he left it to him on such terms as destroyed half the value of the"

[37] "bequest. Mr. Dashwood had wished for it more for the sake of his wife"

[38] "and daughters than for himself or his son;--but to his son, and his"

[39] "son's son, a child of four years old, it was secured, in such a way, as"

[40] "to leave to himself no power of providing for those who were most dear"

[41] "to him, and who most needed a provision by any charge on the estate, or"

[42] "by any sale of its valuable woods. The whole was tied up for the"

[43] "benefit of this child, who, in occasional visits with his father and"

[44] "mother at Norland, had so far gained on the affections of his uncle, by"

[45] "such attractions as are by no means unusual in children of two or three"

[46] "years old; an imperfect articulation, an earnest desire of having his"

[47] "own way, many cunning tricks, and a great deal of noise, as to outweigh"

[48] "all the value of all the attention which, for years, he had received"

[49] "from his niece and her daughters. He meant not to be unkind, however,"

[50] "and, as a mark of his affection for the three girls, he left them a"
## [51] "thousand pounds a-piece."
## [52] ""
## [53] "Mr. Dashwood's disappointment was, at first, severe; but his temper was"
## [54] "cheerful and sanguine; and he might reasonably hope to live many years,"
## [55] "and by living economically, lay by a considerable sum from the produce"
## [56] "of an estate already large, and capable of almost immediate"
## [57] "improvement. But the fortune, which had been so tardy in coming, was"
## [58] "his only one twelvemonth. He survived his uncle no longer; and ten"
## [59] "thousand pounds, including the late legacies, was all that remained for"
## [60] "his widow and daughters."

### 5.2.4 Create a corpus

sensesensibility_corpus <- corpus(sensesensibility DF)
print(sensesensibility_corpus)

## Corpus consisting of 12,612 documents.

## text1 :

## "The family of Dashwood had long been settled in Sussex. The..."

## text2 :

## "was large, and their residence was at Norland Park, in the c..."

## text3 :

## "their property, where, for many generations, they had lived ..."

## text4 :

## "respectable a manner as to engage the general good opinion o..."

## text5 :

## "surrounding acquaintance. The late owner of this estate was..."

## text6 :

## "man, who lived to a very advanced age, and who for many year..."

## [ reached max_ndoc ... 12,606 more documents ]

Given that we have many empty lines in this corpus, we clean it by excluding
any empty line.

sensesensibility_corpus <- corpus_subset(sensesensibility_corpus, ntoken(sensesensibility_corpus)

#### 5.2.4.1 Summary

summary(sensesensibility_corpus, 10)
## Corpus consisting of 10592 documents, showing 10 documents:

##

Text Types Tokens Sentences

<table><tr><td>text1</td><td>13</td><td>13</td><td>2</td></tr><tr><td>text2</td><td>13</td><td>15</td><td>1</td></tr><tr><td>text3</td><td>12</td><td>14</td><td>1</td></tr><tr><td>text4</td><td>12</td><td>12</td><td>1</td></tr><tr><td>text5</td><td>12</td><td>12</td><td>2</td></tr><tr><td>text6</td><td>15</td><td>17</td><td>1</td></tr><tr><td>text7</td><td>14</td><td>14</td><td>2</td></tr><tr><td>text8</td><td>12</td><td>13</td><td>1</td></tr><tr><td>text9</td><td>15</td><td>15</td><td>1</td></tr><tr><td>text10</td><td>13</td><td>15</td><td>1</td></tr></table>

### 5.2.4.2 Accessing parts of corpus

ndoc(sensesensibility_corpus)

## [1] 10592

sensesensibility_corpus[[1]]

## [1] "The family of Dashwood had long been settled in Sussex. Their estate"

## 5.2.5 Advanced manipulations

### 5.2.5.1 Tokens

tokens() segments texts in a corpus into tokens (words or sentences) by word boundaries. We can remove punctuations or not

sensesensibility_corpus_tokens <- tokens(sensesensibility_corpus)
sensesensibility_corpus_tokens

#### 5.2.5.1.1 With punctuations

## Tokens consisting of 10,592 documents.

## text1 :

## [1] "The" "family" "of" "Dashwood" "had" "long"
## [7] "been" "settled" "in" "Sussex" "." "Their"
## [ ... and 1 more ]

##

## text2 :

## [1] "was" "large" "," "and" "their" "residence"
## [7] "was" "at" "Norland" "Park" "," "in"
## [ ... and 3 more ]
##
## text3 :
[1] "their" "property" "," "where" ","
[6] "for" "many" "generations" "," "they"
[11] "had" "lived"
[ ... and 2 more ]
##
## text4 :
[1] "respectable" "a" "manner" "as" "to"
[6] "engage" "the" "general" "good" "opinion"
[11] "of" "their"
##
## text5 :
[1] "surrounding" "acquaintance" "." "The" "late"
[6] "owner" "of" "this" "estate" "was"
[11] "a" "single"
##
## text6 :
[1] "man" "," "who" "lived" "to" "a"
[7] "very" "advanced" "age" "," "and" "who"
[ ... and 5 more ]
##
[ reached max_ndoc ... 10,586 more documents ]

sensesensibility_corpus_tok_no_punct <- tokens(sensesensibility_corpus, remove_punct = TRUE)
sensesensibility_corpus_tok_no_punct

5.2.5.1.2 Without punctuations

## Tokens consisting of 10,592 documents.
## text1 :
[1] "The" "family" "of" "Dashwood" "had" "long"
[7] "been" "settled" "in" "Sussex" "Their" "estate"
##
## text2 :
[1] "was" "large" "and" "their" "residence" "was"
[7] "at" "Norland" "Park" "in" "the" "centre"
[ ... and 1 more ]
##
## text3 :
[1] "their" "property" "where" "for" "many"
[6] "generations" "they" "had" "lived" "in"
[11] "so"
##
## text4 :
[1] "respectable" "a" "manner" "as" "to"
## [6] "engage" "the" "general" "good" "opinion"
## [11] "of" "their"
##
## text5 :
## [1] "surrounding" "acquaintance" "The" "late" "owner"
## [6] "of" "this" "estate" "was" "a"
## [11] "single"
##
## text6 :
## [1] "man" "who" "lived" "to" "a" "very"
## [7] "advanced" "age" "and" "who" "for" "many"
## [ ... and 3 more ]
##
## [ reached max_ndoc ... 10,586 more documents ]

### 5.2.5.2 Compound words

Compound words are multi-word expressions that are relevant for our analysis.
We already found these based the kwic function

sensesensibility_corpus_tokens_no_punct_phrase <- kwic(sensesensibility_corpustok_no_punct_punct_phrase, 10)

#### 5.2.5.2.1 kwic Phrase

## Keyword-in-context with 10 matches.

## [text97, 4:5] right to come | the house |
## [text110, 9:10] the latter she would have quitted | the house |
## [text430, 8:9] had been staying several weeks in | the house |
## [text674, 4:5] dwelling and though | the house |
## [text677, 7:8] her after giving the particulars of | the house |
## [text703, 8:9] mother's intention of removing into Devonshire | The house |
## [text757, 4:5] Mrs Dashwood took | the house |
## [text773, 3:4] to prepare | the house |
## [text776, 9:10] undoubtingly on Sir John's description of | the house |
## [text802, 3:4] alone before | the house |
##
## was her husband's from the moment
## for ever had
## before he engaged
## he now offered her was merely
## and garden to come with
## too as
## for a twelvemonth it was ready
## for their mistress's arrival for as
## as to
on the last evening of their

**5.2.5.2.2 Compounds** However, these are computed as individual words not compound words. To do so, we can use the following code. As you see the phrase we were interested in is now added as compound words

```
sensesensibility_corpus tok no punct comp <- tokens_compound(sensesensibility_corpus tok no punct comp kwic <- kwic(sensesensibility_corpus tok no punct comp head(sensesensibility_corpus tok no punct comp kwic, 10)
```

Keyword-in-context with 10 matches.

<table><tr><td>[text97, 4]</td><td>right to come</td><td>the house</td></tr><tr><td>[text110, 9]</td><td>latter she would have quitted</td><td>the house</td></tr><tr><td>[text430, 8]</td><td>been staying several weeks in</td><td>the house</td></tr><tr><td>[text674, 4]</td><td>dwelling and though</td><td>the house</td></tr><tr><td>[text677, 7]</td><td>after giving the particulars of</td><td>the house</td></tr><tr><td>[text703, 8]</td><td>intention of removing into Devonshire</td><td>The house</td></tr><tr><td>[text757, 4]</td><td>Mrs Dashwood took</td><td>the house</td></tr><tr><td>[text773, 3]</td><td>to prepare</td><td>the house</td></tr><tr><td>[text776, 9]</td><td>on Sir John's description of</td><td>the house</td></tr><tr><td>[text802, 3]</td><td>alone before</td><td>the house</td></tr></table>

### 5.2.5.3 N-grams

N-grams are a subfamily of compound words. They can be named as "bi-grams", "tri-grams", etc. N-grams yield a sequence of tokens from already tokenised text object.

**5.2.5.3.1 Multi-grams** The code below allows to obtain the sequences of consecutive compound words, with 2, 3 or 4 compound words.

```
sensesensibility_corpus tok no punct ngram <- tokens_ngrams(sensesensibility_corpus tok no punct nlist() %>% tolower() %>% table()
```
<table><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td>##</td><td>1</td><td>1</td></tr><tr><td colspan="3">## Last 10 rows</td></tr><tr><td colspan="3">tail(sensesensibility_corpus tok no punct ngram_skip, 10)</td></tr><tr><td>##</td><td></td><td></td></tr><tr><td>## zealous_as what and zealous_as what was</td><td>zealous_to</td><td>zealous_to he</td></tr><tr><td>##</td><td>1</td><td>1</td></tr><tr><td>## zealous_to he and zealous_to he what</td><td>zealous_to what</td><td>zealous_to what and</td></tr><tr><td>##</td><td>1</td><td>1</td></tr><tr><td>## zealous_to what was</td><td>zealously_as</td><td></td></tr><tr><td>##</td><td>1</td><td>1</td></tr></tbody></table>

### 5.2.5.4 Dictionary

If you have a dictionary with various words that fall within a generic word (e.g., variants of pronunciation of a word), then you can look these up. Here, we will create a dictionary that we populate ourselves and we show how to use it to search for items

```python
dict_sensesensibility <- dictionary(list(large = c("large", "big"),
                                      property = c("property", "house"),
                                      good = c("good", "great")))
print(dict_sensesensibility)
```

#### 5.2.5.4.1 Create dictionary

```r
## Dictionary object with 3 key entries.
## - [large]:
## - large, big
## - [property]:
## - property, house
## - [good]:
## - good, great
```

```r
sensesensibility_corpus tok no punct_dict_tokens <- tokens_lookup(sensesensibility_corpus tok no punct dict_tokens)
print(sensesensibility_corpus tok no punct dict_tokens)
```

#### 5.2.5.4.2 Token lookup

```r
## Tokens consisting of 10,592 documents.
## text1 :
## character(0)
##
## text2 :
## [1] "large"
```
##
## text3 :
## [1] "property"
##
## text4 :
## [1] "good"
##
## text5 :
## character(0)
##
## text6 :
## character(0)
##
## [ reached max_ndoc ... 10,586 more documents ]

dfm(sensesensibility_corpus_tokens_no_punct_dict_tokens)

### 5.2.5.4.3 DFM

## Document-feature matrix of: 10,592 documents, 3 features (98.51% sparse) and 0 documents
## features
## docs large property good
## text1 0 0 0
## text2 1 0 0
## text3 0 1 0
## text4 0 0 1
## text5 0 0 0
## text6 0 0 0
## [ reached max_ndoc ... 10,586 more documents ]

### 5.2.5.5 Part of Speech tagging

This section borrows many features from here. Part-of-Speech tagging (or PoS-Tagging) is used to distinguish different part of speech, e.g., the sentence: "Jane likes the girl" can be tagged as "Jane/NNP likes/VBZ the/DT girl/NN", where NNP = proper noun (singular), VBZ = 3rd person singular present tense verb, DT = determiner, and NN = noun (singular or mass). We will use the udpipe package
<table><caption>Overview of Penn English Treebank part-of-speech tags.</caption><thead><tr><td>Tag</td><td>Description</td><td>Examples</td></tr></thead><tbody><tr><td>CC</td><td>Coordinating conjunction</td><td>and, or, but</td></tr><tr><td>CD</td><td>Cardinal number</td><td>one, two, three</td></tr><tr><td>DT</td><td>Determiner</td><td>a, the</td></tr><tr><td>EX</td><td>Existential there</td><td>There/EX was a party in progress</td></tr><tr><td>FW</td><td>Foreign word</td><td>persona/FW non/FW grata/FW</td></tr><tr><td>IN</td><td>Preposition or subordinating con</td><td>uh, well, yes</td></tr><tr><td>JJ</td><td>Adjective</td><td>good, bad, ugly</td></tr><tr><td>JJR</td><td>Adjective, comparative</td><td>better, nicer</td></tr><tr><td>JJS</td><td>Adjective, superlative</td><td>best, nicest</td></tr><tr><td>LS</td><td>List item marker</td><td>a., b., 1., 2.</td></tr><tr><td>MD</td><td>Modal</td><td>can, would, will</td></tr><tr><td>NN</td><td>Noun, singular or mass</td><td>tree, chair</td></tr><tr><td>NNS</td><td>Noun, plural</td><td>trees, chairs</td></tr><tr><td>NNP</td><td>Proper noun, singular</td><td>John, Paul, CIA</td></tr><tr><td>NNPS</td><td>Proper noun, plural</td><td>Johns, Pauls, CIAs</td></tr><tr><td>PDT</td><td>Predeterminer</td><td>all/PDT this marble, many/PDT a soul</td></tr><tr><td>POS</td><td>Possessive ending</td><td>John/NNP 's/POS, the parentss/NNP 'POS distress</td></tr><tr><td>PRP</td><td>Personal pronoun</td><td>I, you, he</td></tr><tr><td>PRP$</td><td>Possessive pronoun</td><td>mine, yours</td></tr><tr><td>RB</td><td>Adverb</td><td>evry, enough, not</td></tr><tr><td>RBR</td><td>Adverb, comparative</td><td>later</td></tr><tr><td>RBS</td><td>Adverb, superlative</td><td>latest</td></tr><tr><td>RP</td><td>Particle</td><td>RP</td></tr><tr><td>SYM</td><td>Symbol</td><td>CO2</td></tr><tr><td>TO</td><td>to</td><td>to</td></tr><tr><td>UH</td><td>Interjection</td><td>uhm, uh</td></tr><tr><td>VB</td><td>Verb, base form</td><td>go, walk</td></tr><tr><td>VBD</td><td>Verb, past tense</td><td>walked, saw</td></tr><tr><td>VBG</td><td>Verb, gerund or present particip</td><td>walking, seeing</td></tr><tr><td>VBN</td><td>Verb, past participle</td><td>walked, thought</td></tr><tr><td>VBP</td><td>Verb, non-3rd person singular pr</td><td>walk, think</td></tr><tr><td>VBZ</td><td>Verb, 3rd person singular presen</td><td>walks, thinks</td></tr><tr><td>WDT</td><td>Wh-determiner</td><td>which, that</td></tr><tr><td>WP</td><td>Wh-pronoun</td><td>what, who, whom (wh-pronoun)</td></tr><tr><td>WP$</td><td>Possessive wh-pronoun</td><td>whose, who (wh-words)</td></tr><tr><td>WRB</td><td>Wh-adverb</td><td>how, where, why (wh-adverb)</td></tr></tbody></table>
5.2.5.5.1 Download and load language model Before using the PoS-
tagger, we need to download a language model. As you can see from typing
?udpipe_download_model, there are 65 languages trained on 101 treebanks
from here

file_to_check <- "models/english-ewt-ud-2.5-191206.udpipe"

if (file.exists(file = file_to_check)){
  m_english <- udpipe_load_model(file = "models/english-ewt-ud-2.5-191206.udpipe")
}else{
  m_english <- udpipe_download_model(model_dir = "models/", language = "english-ewt")
  m_english <- udpipe_load_model(file = "models/english-ewt-ud-2.5-191206.udpipe")
}

sensesensibility_corpus_tokens_no_punct_anndf <- udpipe_ annotate(m_english, x = sensesensibility_corpus_tokens_no_punct_anndf, 10)
as.data.frame()
## inspect
head(sensesensibility_corpus_tokens_no_punct_anndf, 10)

5.2.5.5.2 Tokenise, tag, dependency parsing

## doc_id paragraph_id sentence_id sentence token_id token lemma upos
## 1 doc1 1 1 The 1 The the DET
## 2 doc2 1 1 family 1 family family NOUN
## 3 doc3 1 1 of 1 of of ADP
## 4 doc4 1 1 Dashwood 1 Dashwood Dashwood PROPN
## 5 doc5 1 1 had 1 had have AUX
## 6 doc6 1 1 long 1 long long ADV
## 7 doc7 1 1 been 1 been be AUX
## 8 doc8 1 1 settled 1 settled settle VERB
## 9 doc9 1 1 in 1 in in ADP
## 10 doc10 1 1 Sussex 1 Sussex sussex NOUN
## xpos feats head_token_id dep_rel deps
## 1 DT Definite=Def|PronType=Art 0 root <NA>
## 2 NN Number=Sing 0 root <NA>
## 3 IN <NA> 0 root <NA>
## 4 NNP Number=Sing 0 root <NA>
## 5 VBD Mood=Ind|Tense=Past|VerbForm=Fin 0 root <NA>
## 6 RB Degree=Pos 0 root <NA>
## 7 VBN Tense=Past|VerbForm=Part 0 root <NA>
## 8 VBN Tense=Past|VerbForm=Part 0 root <NA>
## 9 IN <NA> 0 root <NA>
## 10 NN Number=Sing 0 root <NA>
## misc
## 1 SpacesAfter=\\n## 2 SpacesAfter=\\n
## 3 SpacesAfter=\\n
## 4 SpacesAfter=\\n
## 5 SpacesAfter=\\n
## 6 SpacesAfter=\\n
## 7 SpacesAfter=\\n
## 8 SpacesAfter=\\n
## 9 SpacesAfter=\\n
## 10 SpacesAfter=\\n

## parse text

sensesensibility_corpus_sent <- udpipe_ annotate(m_english, x = sensesensibility_corpus[[1]]) %>%
as.data.frame()

## inspect

head(sensesensibility_corpus_sent)

### 5.2.5.5.3 Dependency parsing

## doc_id paragraph_id sentence_id

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr></table>

## sentence token_id token
## 1 The family of Dashwood had long been settled in Sussex. 1 The
## 2 The family of Dashwood had long been settled in Sussex. 2 family
## 3 The family of Dashwood had long been settled in Sussex. 3 of
## 4 The family of Dashwood had long been settled in Sussex. 4 Dashwood
## 5 The family of Dashwood had long been settled in Sussex. 5 had
## 6 The family of Dashwood had long been settled in Sussex. 6 long

## lemma upos xpos feats head_token_id dep_rel
## 1 the DET DT Definite=Def|PronType=Art 2 det
## 2 family NOUN NN Number=Sing 8 nsubj:pass
## 3 of ADP IN <NA> 4 case
## 4 Dashwood PROPN NNP Number=Sing 2 nmod
## 5 have AUX VBD Mood=Ind|Tense=Past|VerbForm=Fin 8 aux
## 6 long ADV RB Degree=Pos 8 advmod

## deps misc

## 1 <NA> <NA>
## 2 <NA> <NA>
## 3 <NA> <NA>
## 4 <NA> <NA>
## 5 <NA> <NA>
## 6 <NA> <NA>
```
sensesensibility_corpus_sent_dplot <- textplot_dependencyparser(sensesensibility_corpus_sent_dplot)
## show plot
sensesensibility_corpus_sent_dplot
```

## Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_142.png" /></figure>

### 5.2.5.6 Feature co-occurrence matrix (FCM)

Feature co-occurrence matrix (FCM) records the number of co-occurrences of tokens

```
sensesensibility_corpus DFmat <- dfm(sensesensibility_corpus_tokens)
sensesensibility_corpus DFmat Trim <- dfm Trim(sensesensibility_corpus DFmat, min_term)
topfeatures_sensesensibility_corpus <- topfeatures(sensesensibility_corpus DFmat Trim)
topfeatures_sensesensibility_corpus
```

#### 5.2.5.6.1 Computing number of co-occurrences

```
## the to of and her a i in was it
## 4105 4104 3570 3488 2543 2067 1998 1948 1861 1755
nfeat(sensesensibility_corpus DFmat Trim)
```

```
## [1] 40
```

```
sensesensibility_corpus fcmat <- fcm(sensesensibility_corpus DFmat Trim)
sensesensibility_corpus fcmat
```
### 5.2.5.6.2 Features co-occurrences

Feature co-occurrence matrix of: 40 by 40 features.

features

features the of had in was and at for they so
## the 807 2343 407 968 799 1439 418 482 198 162
## of 0 388 333 730 560 1226 223 354 148 158
## had 0 0 42 204 148 286 61 141 77 64
## in 0 0 0 138 390 607 109 161 100 103
## was 0 0 0 0 124 644 163 264 50 128
## and 0 0 0 0 0 290 265 386 181 212
## at 0 0 0 0 0 0 25 98 53 60
## for 0 0 0 0 0 0 0 55 59 67
## they 0 0 0 0 0 0 0 0 20 33
## so 0 0 0 0 0 0 0 0 0 48

[ reached max_nfeat ... 30 more features, reached max_nfeat ... 30 more features ]

## 5.3 proustr

### 5.3.1 Import into Global Environment

Adding data from the first book into the Global environment

data(ducotedechezswann)
#data(alombredesjeunesfillesenfleurs)
#data(lecotedeguermantes)
#data(sodomeetgomorrhe)
#data(laprisonniere)
#data(albertinedisparue)
#data(letempretrouve)

### 5.3.2 Look into data

And we can get the top 60 rows from the first one

ducotedechezswann %>%
  head(n = 60)

# A tibble: 60 x 4

| text | book volume year |
|---|---|
| <chr> | <chr> <chr> <db1> |
| 1 | "Longtemps, je me suis couché de bonne heure. Parfois, à ~ Du c~ Premi~ 1913 |
| 2 | "J'appuyais tendrement mes joues contre les belles joues ~ Du c~ Premi~ 1913 |
| 3 | "Je me rendormais, et parfois je n'avais plus que de cour~ Du c~ Premi~ 1913 |
| 4 | "Quelquefois, comme Ève naquit d'une côte d'Adam, une fem~ Du c~ Premi~ 1913 |
| 5 | "Un homme qui dort tient en cercle autour de lui le fil d~ Du c~ Premi~ 1913 |
## 6 "Peut-être l'immobilité des choses autour de nous leur es~ Du c~ Premi~ 1913

## 7 "Puis renaissait le souvenir d'une nouvelle attitude ; le~ Du c~ Premi~ 1913

## 8 "Ces évocations tournoyantes et confuses ne duraient jama~ Du c~ Premi~ 1913

## 9 "Certes, j'étais bien éveillé maintenant : mon corps avai~ Du c~ Premi~ 1913

## 10 "À Combray, tous les jours dès la fin de l'après-midi, lo~ Du c~ Premi~ 1913

## # i 50 more rows

### 5.3.3 Create a corpus

```
ducotedechezswann_corpus <- corpus(ducotedechezswann, text_field = "text")
print(ducotedechezswann_corpus)
```

## Corpus consisting of 1,004 documents and 3 docvars.

## text1 :

## "Longtemps, je me suis couché de bonne heure. Parfois, à pein..."

##

## text2 :

## "J'appuyais tendrement mes joues contre les belles joues de l..."

##

## text3 :

## "Je me rendormais, et parfois je n'avais plus que de courts r..."

##

## text4 :

## "Quelquefois, comme Ève naquit d'une côte d'Adam, une femme n..."

##

## text5 :

## "Un homme qui dort tient en cercle autour de lui le fil des h..."

##

## text6 :

## "Peut-être l'immobilité des choses autour de nous leur est-el..."

##

## [ reached max_ndoc ... 998 more documents ]

#### 5.3.3.1 Summary

```
summary(ducotedechezswann_corpus, 10)
```

## Corpus consisting of 1004 documents, showing 10 documents:

##

<table><thead><tr><th></th><th>Text</th><th>Types</th><th>Tokens</th><th>Sentences</th><th>book</th><th>volume</th></tr></thead><tbody><tr><td>##</td><td>text1</td><td>200</td><td>359</td><td></td><td>6 Du côté de chez Swann Première partie : Combray</td><td></td></tr><tr><td>##</td><td>text2</td><td>110</td><td>166</td><td></td><td>11 Du côté de chez Swann Première partie : Combray</td><td></td></tr><tr><td>##</td><td>text3</td><td>120</td><td>172</td><td></td><td>3 Du côté de chez Swann Première partie : Combray</td><td></td></tr><tr><td>##</td><td>text4</td><td>116</td><td>175</td><td></td><td>6 Du côté de chez Swann Première partie : Combray</td><td></td></tr><tr><td>##</td><td>text5</td><td>202</td><td>370</td><td></td><td>5 Du côté de chez Swann Première partie : Combray</td><td></td></tr><tr><td>##</td><td>text6</td><td>212</td><td>409</td><td></td><td>6 Du côté de chez Swann Première partie : Combray</td><td></td></tr></tbody></table>
<table><tr><td>text7</td><td>120</td><td>191</td><td>6 Du côté de chez Swann Première partie : Combray</td></tr><tr><td>text8</td><td>347</td><td>658</td><td>4 Du côté de chez Swann Première partie : Combray</td></tr><tr><td>text9</td><td>114</td><td>179</td><td>2 Du côté de chez Swann Première partie : Combray</td></tr><tr><td>text10</td><td>130</td><td>204</td><td>4 Du côté de chez Swann Première partie : Combray</td></tr><tr><td>year</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr><tr><td>1913</td><td></td><td></td><td></td></tr></table>

### 5.3.3.2 Accessing parts of corpus

ducotedechezswann_corpus[[1]]

[1] "Longtemps, je me suis couché de bonne heure. Parfois, à peine ma bougie éteinte, mes yeux

### 5.3.3.3 Document-level information

head(docvars(ducotedechezswann_corpus))

<table><thead><tr><th></th><th>book</th><th>volume</th><th>year</th></tr></thead><tbody><tr><td>1</td><td>Du côté de chez Swann Première partie</td><td>Combray</td><td>1913</td></tr><tr><td>2</td><td>Du côté de chez Swann Première partie</td><td>Combray</td><td>1913</td></tr><tr><td>3</td><td>Du côté de chez Swann Première partie</td><td>Combray</td><td>1913</td></tr><tr><td>4</td><td>Du côté de chez Swann Première partie</td><td>Combray</td><td>1913</td></tr><tr><td>5</td><td>Du côté de chez Swann Première partie</td><td>Combray</td><td>1913</td></tr><tr><td>6</td><td>Du côté de chez Swann Première partie</td><td>Combray</td><td>1913</td></tr></tbody></table>

### 5.3.3.4 Unique variable names (for volume)

unique(docvars(ducotedechezswann_corpus, field = "volume"))

[1] "Première partie : Combray"

[2] "Deuxième partie - Un amour de Swan"

[3] "Troisième partie : Nom de pays : le nom"
### 5.3.4 Advanced manipulations

#### 5.3.4.1 Tokens

`tokens()` segments texts in a corpus into tokens (words or sentences) by word boundaries. We can remove punctuations or not

```
ducotedechezswann_corpus_tokens <- tokens(ducotedechezswann_corpus)
ducotedechezswann_corpus_tokens
```

#### 5.3.4.1.1 With punctuations

## Tokens consisting of 1,004 documents and 3 docvars.

text1 :

[1] "Longtemps" "," "je" "me" "suis" "couché"

[7] "de" "bonne" "heure" "." "Parfois" ","

[ ... and 347 more ]

##

text2 :

[1] "J'appuyais" "tendrement" "mes" "joues" "contre"

[6] "les" "belles" "joues" "de" "l'oreiller"

[11] "qui" ","

[ ... and 154 more ]

##

text3 :

[1] "Je" "me" "rendormais" "," "et"

[6] "parfois" "je" "n'avais" "plus" "que"

[11] "de" "courts"

[ ... and 160 more ]

##

text4 :

[1] "Quelquefois" "," "comme" "Ève" "naquit"

[6] "d'une" "côte" "d'Adam" "," "une"

[11] "femme" "naissait"

[ ... and 163 more ]

##

text5 :

[1] "Un" "homme" "qui" "dort" "tient" "en" "cercle" "autour"

[9] "de" "lui" "le" "fil"

[ ... and 358 more ]

##

text6 :

[1] "Peut-être" "l'immobilité" "des" "choses" "autour"

[6] "de" "nous" "leur" "est-elle" "imposée"

[11] "par" "notre"

[ ... and 397 more ]
##
## [ reached max_ndoc ... 998 more documents ]

ducotedechezswann_corpus tok no punct <- tokens(ducotedechezswann_corpus, remove_punct = TRUE)
ducotedechezswann_corpus tok no punct

5.3.4.1.2 Without punctuations

## Tokens consisting of 1,004 documents and 3 docvars.

## text1 :

## [1] "Longtemps" "je" "me" "suis" "couché" "de"
## [7] "bonne" "heure" "Parfois" "à" "peine" "ma"
## [ ... and 306 more ]

##

## text2 :

## [1] "J'appuyais" "tendrement" "mes" "joues" "contre"
## [6] "les" "belles" "joues" "de" "l'oreiller"
## [11] "qui" "pleines"
## [ ... and 133 more ]

##

## text3 :

## [1] "Je" "me" "rendormais" "et" "parfois"
## [6] "je" "n'avais" "plus" "que" "de"
## [11] "courts" "réveils"
## [ ... and 146 more ]

##

## text4 :

## [1] "Quelquefois" "comme" "Ève" "naquit" "d'une"
## [6] "côte" "d'Adam" "une" "femme" "naissait"
## [11] "pendant" "mon"
## [ ... and 144 more ]

##

## text5 :

## [1] "Un" "homme" "qui" "dort" "tient" "en" "cercle" "autour"
## [9] "de" "lui" "le" "fil"
## [ ... and 320 more ]

##

## text6 :

## [1] "Peut-être" "l'immobilité" "des" "choses" "autour"
## [6] "de" "nous" "leur" "est-elle" "imposée"
## [11] "par" "notre"
## [ ... and 336 more ]

##

## [ reached max_ndoc ... 998 more documents ]
### 5.3.4.2 Compound words

```
ducotedechezswann_corpus_tokens_no_punct_phrase <- kwic(ducotedechezswann_corpus_tokens_no_punct_punct_phrase, 10)
head(ducotedechezswann_corpus_tokens_no_punct_phrase, 10)
```

#### 5.3.4.2.1 kwic Phrase

```
## Keyword-in-context with 10 matches.
```

<table><tr><td>[text5, 1:2]</td><td>Un homme</td></tr><tr><td>[text82, 216:217]</td><td>nous avions rencontré près du Pont-Vieux</td></tr><tr><td>[text82, 224:225]</td><td>que mon grand-père ne connaissait pas</td></tr><tr><td>[text82, 268:269]</td><td>rencontré près du Pont-Vieux mon oncle</td></tr><tr><td>[text82, 316:317]</td><td>me disait que vous aviez rencontré</td></tr><tr><td>[text198, 465:466]</td><td>les femmes qui prétendent ne juger</td></tr><tr><td>[text224, 99:100]</td><td>votre mort on fasse de vous</td></tr><tr><td>[text298, 175:176]</td><td>de raisons pour aimer Mais pour</td></tr><tr><td>[text431, 108:109]</td><td>eut besoin d'elle il était comme</td></tr><tr><td>[text449, 15:16]</td><td>cela serait assez triste mais enfin</td></tr></table>

```
ducotedechezswann_corpus_tokens_no_punct_comp <- tokens_compound(ducotedechezswann_corpus_tokens_no_punct_comp, 10)
ducotedechezswann_corpus_tokens_no_punct_comp kwic <- kwic(ducotedechezswann_corpus_tokens_no_punct_comp, 10)
head(ducotedechezswann_corpus_tokens_no_punct_comp, 10)
```

#### 5.3.4.2.2 Compounds

```
## Keyword-in-context with 10 matches.
```

<table><tr><td>[text5, 1]</td><td>Un homme</td></tr><tr><td>[text82, 216]</td><td>avions rencontré près du Pont-Vieux</td></tr><tr><td>[text82, 223]</td><td>mon grand-père ne connaissait pas</td></tr><tr><td>[text82, 266]</td><td>près du Pont-Vieux mon oncle</td></tr><tr><td>[text82, 313]</td><td>disait que vous aviez rencontré</td></tr><tr><td>[text198, 465]</td><td>femmes qui prétendent ne juger</td></tr><tr><td>[text224, 99]</td><td>mort on fasse de vous</td></tr><tr><td>[text298, 175]</td><td>raisons pour aimer Mais pour</td></tr></table>
## [text431, 108] besoin d'elle il était comme | un_homme |
## [text449, 15] serait assez triste mais enfin | un_homme |
##
## qui dort tient en cercle
## que mon grand-père ne connaissait
## que grand-père ne connaissait point
## que vous ne connaissiez point
## que vous ne connaissiez point
## que sur son physique voient
## Monsieur le Curé a toujours
## comme M Vinteuil il devait
## dans la vie de qui
## de génie peut être le

5.3.4.3 N-grams

N-grams are a subfamily of compound words. They can be named as “bi-grams”,
“tri-grams”, etc. N-grams yield a sequence of tokens from already tokenised text
object.

5.3.4.3.1 Multi-grams The code below allows to obtain the sequences of
consecutive compound words, with 2, 3 or 4 compound words.

ducotedechezswann_corpus_tokens_ngrams <- tokens_ngrams(ducotedechezswann_corpus_tokens_ngrams, n=10)

unlist() %>%
tolower() %>%
table()

## Top 10 rows
head(ducotedechezswann_corpus_tokens_ngrams, 10)

## .
## il_est il_est_très il_est_très_tenu
## 1 1 1
## _comment _comment_c'est _comment_c'est_cela
## 1 1 1
## _et _et_la _et_la_joie
## 1 1 1
## _mais
## 3

## Last 10 rows
tail(ducotedechezswann_corpus_tokens_ngrams, 10)

## .
## zoologiques_ou_l'on_voit zut_mais zut_mais_en
## 1 1 1
## zut_mais_en_même zut_zut zut_zut_mais
<table><tr><td>##</td><td>1</td><td>3</td><td>1</td></tr><tr><td>##</td><td>zut_zut_mais_en</td><td>zut_zut_zut</td><td>zut_zut_zut_mais</td></tr><tr><td>##</td><td>1</td><td>2</td><td>1</td></tr><tr><td>##</td><td>zut_zut_zut_zut</td><td></td><td></td></tr><tr><td>##</td><td>1</td><td></td><td></td></tr></table>

### 5.3.4.3.2 Skip-grams
Skip-grams allow to obtain non consecutive n-grams

```
ducotedechezswann_corpus_tokens_no_punct_ngram_skip <- tokens_ngrams(ducotedechezswann_corp
unlist() %>%
to lower() %>%
table()

## Top 10 rows
head(ducotedechezswann_corpus_tokens_no_punct_ngram_skip, 10)
```

<table><tr><td>##</td><td>il_tenu</td><td>il_tenu_a</td><td>il_tenu_a_femme</td></tr><tr><td>##</td><td>1</td><td>1</td><td>1</td></tr><tr><td>##</td><td>il_tenu_a_une</td><td>il_tenu_y</td><td>il_tenu_y_certainement</td></tr><tr><td>##</td><td>1</td><td>1</td><td>1</td></tr><tr><td>##</td><td>il_tenu_y_une</td><td>il_très</td><td>il_très_il</td></tr><tr><td>##</td><td>1</td><td>1</td><td>1</td></tr><tr><td>##</td><td>il_très_il_a</td><td></td><td></td></tr><tr><td>##</td><td>1</td><td></td><td></td></tr></table>

```
## Last 10 rows
tail(ducotedechezswann_corpus_tokens_no_punct_ngram_skip, 10)
```

<table><tr><td>##</td><td>zut_zut</td><td>zut_zut_en</td><td>zut_zut_en_je</td><td>zut_zut_en_temp</td></tr><tr><td>##</td><td>3</td><td>3</td><td>3</td><td>3</td></tr><tr><td>##</td><td>zut_zut_mais</td><td>zut_zut_mais_même</td><td>zut_zut_mais_temp</td><td>zut_zut_même</td></tr><tr><td>##</td><td>1</td><td>1</td><td>1</td><td>2</td></tr><tr><td>##</td><td>zut_zut_même_je</td><td>zut_zut_même_sentis</td><td></td><td></td></tr><tr><td>##</td><td>2</td><td>2</td><td></td><td></td></tr></table>

### 5.3.4.4 Dictionary

If you have a dictionary with various words that fall within a generic word (e.g., variants of pronunciation of a word), then you can look these up. Here, we will create a dictionary that we populate ourselves and we show how to use it to search for items

```
dict_ducotedechezswann <- dictionary(list(famille = c("grand-père", "femme", "homme",
    dormir = c("rendormais", "dors", "dort")))
print(dict_ducotedechezswann)
```
### 5.3.4.4.1 Create dictionary

## Dictionary object with 2 key entries.

## - [famille]:

## - grand-père, femme, homme, oncle

## - [dormir]:

## - rendormais, dors, dort

```
ducotedechezswann_corpus tok no punct dict toks <- tokens lookup (ducotedechezswann_corpus tok no punct dict toks)
print (ducotedechezswann_corpus tok no punct dict toks)
```

### 5.3.4.4.2 Token lookup

## Tokens consisting of 1,004 documents and 3 docvars.

## text1 :

## character(0)

##

## text2 :

## character(0)

##

## text3 :

## [1] "dormir"

##

## text4 :

## [1] "famille" "famille" "famille"

##

## text5 :

## [1] "famille" "dormir" "dormir"

##

## text6 :

## [1] "famille"

##

## [ reached max_ndoc ... 998 more documents ]

```
dfm (ducotedechezswann_corpus tok no punct dict toks)
```

### 5.3.4.4.3 DFM

## Document-feature matrix of: 1,004 documents, 2 features (91.33% sparse) and 3 docvars.

## features

## docs famille dormir

## text1 0 0

## text2 0 0

## text3 0 1

## text4 3 0
```R
## text5      1      2
## text6      1      0
## [ reached max_ndoc ... 998 more documents ]
```

### 5.3.4.5 Part of Speech tagging

Part-of-Speech tagging (or PoS-Tagging) is used to distinguish different part of speech, e.g., the sentence: "Jane likes the girl" can be tagged as "Jane/NNP likes/VBZ the/DT girl/NN", where NNP = proper noun (singular), VBZ = 3rd person singular present tense verb, DT = determiner, and NN = noun (singular or mass). We will use the udpipe package

#### 5.3.4.5.1 Download and load language model
Before using the PoS-tagger, we need to download a language model. As you can see from typing ?udpipe_download_model, there are 65 languages trained on 101 treebanks from here

```R
file_to_check <- "models/french-partut-ud-2.5-191206.udpipe"

if (file.exists(file = file_to_check)){
  m_french <- udpipe_load_model(file = "models/french-partut-ud-2.5-191206.udpipe")
}else{
  m_french <- udpipe_download_model(model_dir = "models/", language = "french-partut")
  m_french <- udpipe_load_model(file = "models/french-partut-ud-2.5-191206.udpipe")
}
```

#### 5.3.4.5.2 Tokenise, tag, dependency parsing
We use the already tokenised text, with no punctuations.

```R
ducotedechezswann_anndf <- udpipe_ annotate(m_french, x = ducotedechezswann_corpus_tokenizer,
                                           as.data.frame())
## inspect
head(ducotedechezswann_anndf, 10)
```

```
##   doc_id paragraph_id sentence_id sentence token_id token lemma upos
## 1   doc1              1              1 Longtemps          1 Longtemps Longtemp NOUN
## 2   doc2              1              1      je              1      je      je PRON
## 3   doc3              1              1      me              1      me      me PRON
## 4   doc4              1              1      suis             1      suis    être AUX
## 5   doc5              1              1      couché           1      couché  couché VERB
## 6   doc6              1              1      de              1      de      de ADP
## 7   doc7              1              1      bonne            1      bonne   bon ADJ
## 8   doc8              1              1      heure            1      heure   heure VERB
## 9   doc9              1              1      Parfois          1      Parfois parfois ADV
## 10  doc10             1              1      à               1      à       à ADP
##   spos
## 1   S
feats head_token_id
Gender=Masc                 0
```
<table><tr><td>## 2</td><td>PE</td><td>Number=Sing|Person=1|PronType=Prs</td><td>0</td></tr><tr><td>## 3</td><td>P</td><td>Person=1|PronType=Prs</td><td>0</td></tr><tr><td>## 4</td><td>V</td><td>Mood=Ind|Number=Sing|Person=1|Tense=Pres|VerbForm=Fin</td><td>0</td></tr><tr><td>## 5</td><td>V</td><td>Gender=Masc|Number=Sing|Tense=Past|VerbForm=Part</td><td>0</td></tr><tr><td>## 6</td><td>E</td><td>&lt;NA&gt;</td><td>0</td></tr><tr><td>## 7</td><td>A</td><td>Gender=Fem|Number=Sing</td><td>0</td></tr><tr><td>## 8</td><td>V</td><td>VerbForm=Inf</td><td>0</td></tr><tr><td>## 9</td><td>B</td><td>&lt;NA&gt;</td><td>0</td></tr><tr><td>## 10</td><td>E</td><td>&lt;NA&gt;</td><td>0</td></tr><tr><td>##</td><td>dep_rel</td><td>deps</td><td>misc</td></tr><tr><td>## 1</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 2</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 3</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 4</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 5</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 6</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 7</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 8</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 9</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>## 10</td><td>root</td><td>&lt;NA&gt; SpacesAfter=\\n</td><td></td></tr></table>

### parse text

```
ducotedechezswann_corpus_sent <- udpipe_ annotate(m_french, x = ducotedechezswann_corpus[[1]]) %>
as.data.frame()
```

### inspect

```
head(ducotedechezswann_corpus_sent)
```

#### 5.3.4.5.3 Dependency parsing

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1 Longtemps, je me suis couché de bonne heure.</td></tr><tr><td>2</td><td>doc1</td><td>1</td><td>1 Longtemps, je me suis couché de bonne heure.</td></tr><tr><td>3</td><td>doc1</td><td>1</td><td>1 Longtemps, je me suis couché de bonne heure.</td></tr><tr><td>4</td><td>doc1</td><td>1</td><td>1 Longtemps, je me suis couché de bonne heure.</td></tr><tr><td>5</td><td>doc1</td><td>1</td><td>1 Longtemps, je me suis couché de bonne heure.</td></tr><tr><td>6</td><td>doc1</td><td>1</td><td>1 Longtemps, je me suis couché de bonne heure.</td></tr><tr><td>1</td><td>token_id</td><td>token</td><td>lemma upos</td></tr><tr><td>2</td><td>1</td><td>Longtemps</td><td>Longtemp</td></tr><tr><td>3</td><td>2</td><td>je</td><td>je</td></tr><tr><td>4</td><td>3</td><td>me</td><td>me</td></tr><tr><td>5</td><td>4</td><td>suis</td><td>être</td></tr><tr><td>6</td><td>5</td><td>couché</td><td>coucher</td></tr><tr><td>1</td><td>feats</td><td>head_token_id dep_rel</td><td></td></tr><tr><td>1</td><td>Gender</td><td>Masc|Number=Plur</td><td>6 obl</td></tr></tbody></table>
<table><tr><td>## 2</td><td>&lt;NA&gt;</td><td>1</td><td>punct</td></tr><tr><td>## 3</td><td>Number=Sing|Person=1|PronType=Prs</td><td>6</td><td>nsubj</td></tr><tr><td>## 4</td><td>Person=1|PronType=Prs</td><td>6</td><td>expl</td></tr><tr><td>## 5</td><td>Mood=Ind|Number=Sing|Person=1|Tense=Pres|VerbForm=Fin</td><td>6</td><td>aux</td></tr><tr><td>## 6</td><td>Gender=Masc|Number=Sing|Tense=Past|VerbForm=Part</td><td>0</td><td>root</td></tr><tr><td>##</td><td>deps</td><td>misc</td><td></td></tr><tr><td>## 1</td><td>&lt;NA&gt; SpaceAfter=No</td><td></td><td></td></tr><tr><td>## 2</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>## 3</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>## 4</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>## 5</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>## 6</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr></table>

```
ducotedechezswann_corpus_sent_dplot <- textplot_dependencyparser(ducotedechezswann_corp
## show plot
ducotedechezswann_corpus_sent_dplot
```

### Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_143.png" /></figure>

#### 5.3.4.6 Feature co-occurrence matrix (FCM)

Feature co-occurrence matrix (FCM) records the number of co-occurrences of tokens

```
ducotedechezswann_corpus DFmat <- dfm(ducotedechezswann_corpus tok_no_punct)
ducotedechezswann_corpus DFmat Trim <- dfm Trim (ducotedechezswann_corpus DFmat, min DFmat

topfeatures_ducotedechezswann_corpus <- topfeatures (ducotedechezswann_corpus DFmat Trin
```
topfeatures_ducotedechezswann_corpus

### 5.3.4.6.1 Computing number of co-occurrences

```R
## de la et à que le les qui il pas
## 7780 3935 3894 3686 3137 3033 2330 2150 1998 1850
```

```
nfeat(ducotedechezswann_corpus DFmat_trim)
```

```R
## [1] 49
```

```
ducotedechezswann_corpus_fcmat <- fcm(ducotedechezswann_corpus DFmat_trim)
ducotedechezswann_corpus_fcmat
```

### 5.3.4.6.2 Features co-occurrences

```R
## Feature co-occurrence matrix of: 49 by 49 features.
```

```R
## features
```

<table><thead><tr><th>features</th><th>je</th><th>me</th><th>de</th><th>à</th><th>ma</th><th>se</th><th>si</th><th>que</th><th>pas</th><th>le</th></tr></thead><tbody><tr><td>je</td><td>5576</td><td>5509</td><td>31908</td><td>15322</td><td>3528</td><td>3201</td><td>3707</td><td>16146</td><td>7601</td><td>12009</td></tr><tr><td>me</td><td>0</td><td>1429</td><td>15601</td><td>7505</td><td>1894</td><td>1390</td><td>1753</td><td>7838</td><td>3461</td><td>5552</td></tr><tr><td>de</td><td>0</td><td>0</td><td>94644</td><td>89769</td><td>13593</td><td>24452</td><td>20964</td><td>73030</td><td>38139</td><td>68045</td></tr><tr><td>à</td><td>0</td><td>0</td><td>0</td><td>22220</td><td>6645</td><td>12505</td><td>10230</td><td>35120</td><td>18360</td><td>32286</td></tr><tr><td>ma</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1084</td><td>1466</td><td>1461</td><td>6514</td><td>2903</td><td>4655</td></tr><tr><td>se</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1941</td><td>2946</td><td>8978</td><td>5404</td><td>9415</td></tr><tr><td>si</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1355</td><td>8396</td><td>4749</td><td>7750</td></tr><tr><td>que</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>15519</td><td>15987</td><td>26398</td></tr><tr><td>pas</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>4662</td><td>13926</td></tr><tr><td>le</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>12745</td></tr></tbody></table>

```R
## [ reached max_nfeat ... 39 more features, reached max_nfeat ... 39 more features ]
```

## 5.4 Inaugural Corpus USA

The readtext package comes with various datasets. We specify the path to where to find the datasets and upload them

```
Data_Dir <- system.file("extdata/", package = "readtext")
```

### 5.4.1 Importing data

```
dat_inaug <- read.csv(paste0(Data_Dir, "/csv/inaugCorpus.csv"))
```
## 5.4.2 Create a corpus

```
dat_inaug_corpus <- corpus.dat_inaug(text_field = "texts")
print.dat_inaug_corpus)
```

```r
## Corpus consisting of 5 documents and 3 docvars.
## text1 :
## "Fellow-Citizens of the Senate and of the House of Representa..."
##
## text2 :
## "Fellow citizens, I am again called upon by the voice of my c..."
##
## text3 :
## "When it was first perceived, in early times, that no middle ..."
##
## text4 :
## "Friends and Fellow Citizens: Called upon to undertake the du..."
##
## text5 :
## "Proceeding, fellow citizens, to that qualification which the..."
```

### 5.4.2.1 Summary

```
summary.dat_inaug_corpus, 10)
```

```r
## Corpus consisting of 5 documents, showing 5 documents:
```

<table><thead><tr><th></th><th>Text Types</th><th>Tokens</th><th>Sentences</th><th>Year</th><th>President</th><th>First Name</th></tr></thead><tbody><tr><td>## text1</td><td>625</td><td>1538</td><td>23</td><td>1789</td><td>Washington</td><td>George</td></tr><tr><td>## text2</td><td>96</td><td>147</td><td>4</td><td>1793</td><td>Washington</td><td>George</td></tr><tr><td>## text3</td><td>826</td><td>2578</td><td>37</td><td>1797</td><td>Adams</td><td>John</td></tr><tr><td>## text4</td><td>717</td><td>1927</td><td>41</td><td>1801</td><td>Jefferson</td><td>Thomas</td></tr><tr><td>## text5</td><td>804</td><td>2381</td><td>45</td><td>1805</td><td>Jefferson</td><td>Thomas</td></tr></tbody></table>

### 5.4.2.2 Editing docnames

```
docid <- paste.dat_inaug$Year,
        dat_inaug$First Name,
        dat_inaug$President, sep = " ")
docnames.dat_inaug_corpus) <- docid
print.dat_inaug_corpus)
```

```r
## Corpus consisting of 5 documents and 3 docvars.
## 1789 George Washington :
## "Fellow-Citizens of the Senate and of the House of Representa..."
##
```
## 1793 George Washington :
## "Fellow citizens, I am again called upon by the voice of my c..."
##
## 1797 John Adams :
## "When it was first perceived, in early times, that no middle ..."
##
## 1801 Thomas Jefferson :
## "Friends and Fellow Citizens: Called upon to undertake the du..."
##
## 1805 Thomas Jefferson :
## "Proceeding, fellow citizens, to that qualification which the..."

### 5.4.2.3 Accessing parts of corpus

```
dat_inaug_corpus[[1]]
```

## [1] "Fellow-Citizens of the Senate and of the House of Representatives:\n\nAmong the vicissitudes of the American Republic:"

dat_inaug_corpus[["1789 George Washington"]]

## [1] "Fellow-Citizens of the Senate and of the House of Representatives:\n\nAmong the vicissitudes of the American Republic:"

### 5.4.2.4 Document-level information

```
head(docvars.dat_inaug_corpus))
```

## Year President First Name
## 1 1789 Washington George
## 2 1793 Washington George
## 3 1797 Adams John
## 4 1801 Jefferson Thomas
## 5 1805 Jefferson Thomas

### 5.4.2.5 Unique variable names (for volume)

```
unique(docvars.dat_inaug_corpus, field = "Year"))
```

## [1] 1789 1793 1797 1801 1805

## 5.4.3 Advanced manipulations

### 5.4.3.1 Tokens

tokens() segments texts in a corpus into tokens (words or sentences) by word boundaries. We can remove punctuations or not
```
dat_inaug_corpus_tokens <- tokens (dat_inaug_corpus)
dat_inaug_corpus_tokens
```

### 5.4.3.1.1 With punctuations

```r
## Tokens consisting of 5 documents and 3 docvars.
## 1789 George Washington :
## [1] "Fellow-Citizens" "of"          "the"          "Senate"
## [5] "and"             "of"          "the"          "House"
## [9] "of"              "Representatives" ":"          "Among"
## [ ... and 1,526 more ]
##
## 1793 George Washington :
## [1] "Fellow" "citizens" "," "I" "am" "again"
## [7] "called" "upon" "by" "the" "voice" "of"
## [ ... and 135 more ]
##
## 1797 John Adams :
## [1] "When" "it" "was" "first" "perceived" ","
## [7] "in" "early" "times" "," "that" "no"
## [ ... and 2,566 more ]
##
## 1801 Thomas Jefferson :
## [1] "Friends" "and" "Fellow" "Citizens" ":" "Called"
## [7] "upon" "to" "undertake" "the" "duties" "of"
## [ ... and 1,915 more ]
##
## 1805 Thomas Jefferson :
## [1] "Proceeding" "," "fellow" "citizens"
## [5] "," "to" "that" "qualification"
## [9] "which" "the" "Constitution" "requires"
## [ ... and 2,369 more ]
```

```
dat_inaug_corpus_tokens_no_punct <- tokens (dat_inaug_corpus, remove_punct = TRUE)
dat_inaug_corpus_tokens_no_punct
```

### 5.4.3.1.2 Without punctuations

```r
## Tokens consisting of 5 documents and 3 docvars.
## 1789 George Washington :
## [1] "Fellow-Citizens" "of"          "the"          "Senate"
## [5] "and"             "of"          "the"          "House"
## [9] "of"              "Representatives" "Among" "the"
## [ ... and 1,418 more ]
##
```
## 1793 George Washington :
## [1] "Fellow" "citizens" "I" "am" "again" "called"
## [7] "upon" "by" "the" "voice" "of" "my"
## [ ... and 123 more ]
##
## 1797 John Adams :
## [1] "When" "it" "was" "first" "perceived" "in"
## [7] "early" "times" "that" "no" "middle" "course"
## [ ... and 2,306 more ]
##
## 1801 Thomas Jefferson :
## [1] "Friends" "and" "Fellow" "Citizens" "Called" "upon"
## [7] "to" "undertake" "the" "duties" "of" "the"
## [ ... and 1,714 more ]
##
## 1805 Thomas Jefferson :
## [1] "Proceeding" "fellow" "citizens" "to"
## [5] "that" "qualification" "which" "the"
## [9] "Constitution" "requires" "before" "my"
## [ ... and 2,154 more ]

### 5.4.3.2 Compound words

```
dat_inaug_corpus_tokens_no_punct_phrase <- kwic.dat_inaug_corpus_tokens_no_punct, pattern = phrase("the head">(dat_inaug_corpus_tokens_no_punct_phrase, 10)
```

#### 5.4.3.2.1 kwic Phrase

## Keyword-in-context with 10 matches.
## [1789 George Washington, 1023:1024]
## [1793 George Washington, 71:72]
## [1797 John Adams, 465:466]
## [1797 John Adams, 688:689]
## [1797 John Adams, 739:740]
## [1797 John Adams, 1537:1538]
## [1797 John Adams, 2225:2226]
## [1801 Thomas Jefferson, 331:332]
## [1805 Thomas Jefferson, 8:9]
## [1805 Thomas Jefferson, 482:483]
##
## delegated by the fifth article of | the Constitution |
## any official act of the President | the Constitution |
## of these transactions I first saw | the Constitution |
## and the State legislatures according to | the Constitution |
## the most serious obligations to support | the Constitution |
## after truth if an attachment to | the Constitution |
same American people pledged to support | the Constitution |
announced according to the rules of | the Constitution |
fellow citizens to that qualification which | the Constitution |
States and a corresponding amendment of | the Constitution |

is rendered expedient at the present
requires an oath of office This
of the United States in a
itself adopt and ordain Returning to
The operation of it has equaled
of the United States and a
of the United States I entertain
all will of course arrange themselves
requires before my entrance on the
be applied in time of peace

dat_inaug_corpus tok no punct comp <- tokens_compound (dat_inaug_corpus tok no punct, p

 Dat-inaug-corpus tok no punct comp kwic <- kwic (dat_inaug_corpus tok no punct comp, p

 head (dat_inaug_corpus tok no punct comp kwic, 10)

5.4.3.2.2 Compounds

Keyword-in-context with 10 matches.

[1789 George Washington, 1023] by the fifth article of
[1793 George Washington, 71] official act of the President
[1797 John Adams, 465] these transactions I first saw
[1797 John Adams, 687] the State legislatures according to
[1797 John Adams, 737] most serious obligations to support
[1797 John Adams, 1534] truth if an attachment to
[1797 John Adams, 2221] American people pledged to support
[1801 Thomas Jefferson, 331] according to the rules of
[1805 Thomas Jefferson, 8] citizens to that qualification which
[1805 Thomas Jefferson, 481] and a corresponding amendment of

the_Constitution | is rendered expedient at the
the_Constitution | requires an oath of office
the_Constitution | of the United States in
the_Constitution | itself adopt and ordain Returning
the_Constitution | The operation of it has
the_Constitution | of the United States and
the_Constitution | of the United States I
the_Constitution | all will of course arrange
the_Constitution | requires before my entrance on
the_Constitution | be applied in time of
### 5.4.3.3 N-grams

N-grams are a subfamily of compound words. They can be named as “bi-grams”, “tri-grams”, etc. N-grams yield a sequence of tokens from already tokenised text object.

#### 5.4.3.3.1 Multi-grams
The code below allows to obtain the sequences of consecutive compound words, with 2, 3 or 4 compound words.

```
dat_inaug_corpus tok no punct ngram <- tokens_ngrams (dat_inaug_corpus tok no punct, n = 2:4) %>
unlist() %>
to lower() %>
table()
## Top 10 rows
head (dat_inaug_corpus tok no punct ngram, 10)
```

```
## .
##                 14th day              14th day of
##                      1                   1
##                 14th day of the        a_benevolent
##                      1                   1
##        a_benevolent human        a_benevolent human mind
##                      1                   1
##        a_benign            a_benign religion
##                      1                   1
## a_benign religion professed    a_bulwark
##                      1                   1
```

```
## Last 10 rows
tail (dat_inaug_corpus tok no punct ngram, 10)
```

```
## .
##  zeal and purity zeal and purity and  zeal and wisdom  zeal and wisdom of
##                 1                   1                   1                   1
##        zeal on        zeal on which    zeal on which to    zeal with
##                 1                   1                   1                   1
## zeal with which  zeal with which it
##                 1                   1
```

#### 5.4.3.3.2 Skip-grams
Skip-grams allow to obtain non consecutive n-grams

```
dat_inaug_corpus tok no punct ngram_skip <- tokens_ngrams (dat_inaug_corpus tok no punct, n = 2:4)
unlist() %>
to lower() %>
table()
## Top 10 rows
head (dat_inaug_corpus tok no punct ngram_skip, 10)
```
## .
## 14th_of 14th_of_month 14th_of_month_one 14th_of_month_the
## 1 1 1 1
## 14th_of present 14th_of present on 14th_of present the 14th_the
## 1 1 1 1
## 14th the month 14th the month one
## 1 1

## Last 10 rows

tail ( dat_inaug_corpus_tokens_no_punct_ngram_skip, 10)

## .
## zeal Which under zeal Which under difficulties
## 1 1
## zeal Which under to zeal wisdom
## 1 1
## zeal wisdom characters zeal wisdom characters selected
## 1 1
## zeal wisdom characters who zeal wisdom the
## 1 1
## zeal wisdom the selected zeal wisdom the thus
## 1 1

5.4.3.4 Dictionary

If you have a dictionary with various words that fall within a generic word (e.g., variants of pronunciation of a word), then you can look these up. Here, we will create a dictionary that we populate ourselves and we show how to use it to search for items

dict_dat_inaug <- dictionary ( list (Population = c("Citizens*", "people"), upperrajah = c("Representatives", "senat")))
print ( dict_dat_inaug)

5.4.3.4.1 Create dictionary

## Dictionary object with 2 key entries.

## - [Population]:

## - citizens*, people

## - [upper\_house]:

## - representatives, senat

dat_inaug\_corpus\_tok\_no\_punct\_dict\_toks <- tokens\_lookup (dat\_inaug\_corpus\_tok\_no\_punct\_toks)
print ( dat\_inaug\_corpus\_tok\_no\_punct\_dict\_toks)

5.4.3.4.2 Token lookup
### Tokens consisting of 5 documents and 3 docvars.

### 1789 George Washington :

## [1] "upper_house" "Population" "Population" "Population" "Population"

## [6] "Population" "Population" "Population" "upper_house" "Population"

##

### 1793 George Washington :

## [1] "Population" "Population"

##

### 1797 John Adams :

## [1] "Population" "upper_house" "Population" "Population" "Population"

## [6] "Population" "upper_house" "Population" "Population" "Population"

## [11] "Population" "Population"

## [ ... and 12 more ]

##

### 1801 Thomas Jefferson :

## [1] "Population" "Population" "Population" "Population" "Population"

## [6] "Population" "Population"

##

### 1805 Thomas Jefferson :

## [1] "Population" "Population" "Population" "Population" "Population"

## [6] "Population" "Population" "Population" "Population" "Population"

dfm.dat_inaug_corpus_tokens_no_punct_dict_tokens)

### 5.4.3.4.3 DFM

### Document-feature matrix of: 5 documents, 2 features (30.00% sparse) and 3 docvars.

### features

### docs population upper_house

## 1789 George Washington 8 2

## 1793 George Washington 2 0

## 1797 John Adams 22 2

## 1801 Thomas Jefferson 7 0

## 1805 Thomas Jefferson 10 0

### 5.4.3.5 Part of Speech tagging

Part-of-Speech tagging (or PoS-Tagging) is used to distinguish different part of speech, e.g., the sentence: "Jane likes the girl" can be tagged as "Jane/NNP likes/VBZ the/DT girl/NN", where NNP = proper noun (singular), VBZ = 3rd person singular present tense verb, DT = determiner, and NN = noun (singular or mass). We will use the udpipe package

#### 5.4.3.5.1 Download and load language model
Before using the PoS- tagger, we need to download a language model. As you can see from typing
?udpipe_download_model, there are 65 languages trained on 101 treebanks
from here

```R
file_to_check <- "models/english-ewt-ud-2.5-191206.udpipe"

if (file.exists(file = file_to_check)){
  m_english <- udpipe_load_model(file = "models/english-ewt-ud-2.5-191206.udpipe")
}else{
  m_english <- udpipe_download_model(model_dir = "models/", language = "english-ewt")
  m_english <- udpipe_load_model(file = "models/english-ewt-ud-2.5-191206.udpipe")
}
```

### 5.4.3.5.2 Tokenise, tag, dependency parsing
We use the already tokenised text, with no punctuations.

```R
dat_inaug_anndf <- udpipe_ annotate(m_english, x = dat_inaug_corpus_token_no_punct[[1]])
as.data.frame()
## inspect
head.dat_inaug_anndf, 10
```

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th><th>lemma</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1 Fellow-Citizens</td><td>1</td><td>Fellow</td><td>fellow</td></tr><tr><td>2</td><td>doc1</td><td>1</td><td>1 Fellow-Citizens</td><td>2</td><td>-</td><td>-</td></tr><tr><td>3</td><td>doc1</td><td>1</td><td>1 Fellow-Citizens</td><td>3</td><td>Citizens</td><td>citizen</td></tr><tr><td>4</td><td>doc2</td><td>1</td><td>1</td><td>of</td><td>1</td><td>of</td></tr><tr><td>5</td><td>doc3</td><td>1</td><td>1</td><td>the</td><td>1</td><td>the</td></tr><tr><td>6</td><td>doc4</td><td>1</td><td>1</td><td>Senate</td><td>1</td><td>Senate</td></tr><tr><td>7</td><td>doc5</td><td>1</td><td>1</td><td>and</td><td>1</td><td>and</td></tr><tr><td>8</td><td>doc6</td><td>1</td><td>1</td><td>of</td><td>1</td><td>of</td></tr><tr><td>9</td><td>doc7</td><td>1</td><td>1</td><td>the</td><td>1</td><td>the</td></tr><tr><td>10</td><td>doc8</td><td>1</td><td>1</td><td>House</td><td>1</td><td>House</td></tr></tbody></table>

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th><th>lemma</th></tr></thead><tbody><tr><td>1</td><td>ADJ JJ</td><td>Degree=Pos</td><td>Degree=Pos</td><td>3</td><td>amod</td><td>&lt;NA&gt;</td></tr><tr><td>2</td><td>PUNCT HYPH</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td>3</td><td>punct</td><td>&lt;NA&gt;</td></tr><tr><td>3</td><td>NOUN NNS</td><td>Number=Plur</td><td>Number=Plur</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>4</td><td>ADP IN</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>5</td><td>DET DT</td><td>Definite=Def|PronType=Art</td><td>Definite=Def|PronType=Art</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>6</td><td>PROPN NNP</td><td>Number=Sing</td><td>Number=Sing</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>7</td><td>CCONJ CC</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>8</td><td>ADP IN</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>9</td><td>DET DT</td><td>Definite=Def|PronType=Art</td><td>Definite=Def|PronType=Art</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td>10</td><td>PROPN NNP</td><td>Number=Sing</td><td>Number=Sing</td><td>0</td><td>root</td><td>&lt;NA&gt;</td></tr><tr><td colspan="7">misc</td></tr><tr><td>1</td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2</td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>3</td><td>SpacesAfter=\\n</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
```
dat_inaug_corpus_sent_dplot <- textplot_dependencyparser (dat_inaug_corpus_sent, size =
## show plot
dat_inaug_corpus_sent_dplot
```

### Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_144.png" /></figure>

#### 5.4.3.6 Feature co-occurrence matrix (FCM)

Feature co-occurrence matrix (FCM) records the number of co-occurrences of tokens

```
dat_inaug_corpus DFmat <- dfm (dat_inaug_corpus tok_no_punct)
dat_inaug_corpus DFmat Trim <- dfm Trim (dat_inaug_corpus DFmat, min_termfreq = 50)
topfeatures dat_inaug_corpus <- topfeatures (dat_inaug_corpus DFmat Trim)
topfeatures dat_inaug_corpus
```

##### 5.4.3.6.1 Computing number of co-occurrences

```
## the of and to in a which that be by
## 565 427 354 269 140 106 105 102 93 90
nfeat (dat_inaug_corpus DFmat Trim)
```

```
## [1] 22
```

```
dat_inaug_corpus fcmat <- fcm (dat_inaug_corpus DFmat Trim)
dat_inaug_corpus fcmat
```
### 5.4.3.6.2 Features co-occurrences

Feature co-occurrence matrix of: 22 by 22 features.

features

<table><thead><tr><th>features</th><th>of</th><th>the</th><th>and</th><th>to</th><th>have</th><th>with</th><th>that</th><th>which</th><th>by</th><th>on</th></tr></thead><tbody><tr><td>of</td><td>22676</td><td>59162</td><td>39447</td><td>28270</td><td>5307</td><td>8355</td><td>10602</td><td>10290</td><td>9528</td><td>5314</td></tr><tr><td>the</td><td>0</td><td>38489</td><td>50613</td><td>37168</td><td>7278</td><td>11184</td><td>14098</td><td>13988</td><td>12462</td><td>7283</td></tr><tr><td>and</td><td>0</td><td>0</td><td>17032</td><td>24334</td><td>4530</td><td>7120</td><td>9111</td><td>8494</td><td>8206</td><td>4540</td></tr><tr><td>to</td><td>0</td><td>0</td><td>0</td><td>8927</td><td>3687</td><td>5512</td><td>6988</td><td>6607</td><td>5932</td><td>3610</td></tr><tr><td>have</td><td>0</td><td>0</td><td>0</td><td>0</td><td>408</td><td>1188</td><td>1499</td><td>1375</td><td>1140</td><td>824</td></tr><tr><td>with</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>824</td><td>2174</td><td>2076</td><td>1756</td><td>1143</td></tr><tr><td>that</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1326</td><td>2540</td><td>2220</td><td>1435</td></tr><tr><td>which</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1373</td><td>2228</td><td>1398</td></tr><tr><td>by</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>977</td><td>1174</td></tr><tr><td>on</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>381</td></tr></tbody></table>

[ reached max_nfeat ... 12 more features, reached max_nfeat ... 12 more features ]

## 5.5 Universal Declaration of Human Rights

We import multiple files containing the Universal Declaration of Human Rights in 13 languages. There are 13 different textfiles

### 5.5.1 Importing data

```
dat_udhr <- readtext(paste0(Data_Dir, "/txt/UDHR/*"),
  docvarsfrom = "filenames",
  docvarnames = c("document", "language"))
```

### 5.5.2 Create a corpus

```
dat_udhr_corpus <- corpusWhole UDHR udhrUDHRcorpus
printWhole UDHRcorpus
```

Corpus consisting of 13 documents and 2 docvars.

UDHR_chinese.txt :

```
"217A(III) 1948 12 10 ..."
```

##

UDHR_czech.txt :

```
"VŠEOBECNÁ DEKLARACE LIDSKÝCH PRÁV Úvod U vědomí toho, že uzn..."
```

##

UDHR_danish.txt :

```
"Den 10. december 1948 vedtog og offentliggjorde FNs tredie g..."
```

##
## UDHR_english.txt :
## "Universal Declaration of Human Rights Preamble Whereas recog..."
##
## UDHR_french.txt :
## "Déclaration universelle des droits de l'homme Préambule Cons..."
##
## UDHR_georgian.txt :
## "FLFVBFYBC EAKT<FSF CF>JDTKSFJ LTRKFHFWBF GHTFV<EKF dbyfblfy ..."
##
## [ reached max_ndoc ... 7 more documents ]

### 5.5.2.1 Summary

summary.dat_udhr_corpustext

## Corpus consisting of 13 documents, showing 13 documents:

<table><thead><tr><th></th><th>Text</th><th>Types</th><th>Tokens</th><th>Sentences</th><th>document</th><th>language</th></tr></thead><tbody><tr><td>##</td><td>UDHR_chinese.txt</td><td>600</td><td>2849</td><td>94</td><td>UDHR</td><td>chinese</td></tr><tr><td>##</td><td>UDHR_czech.txt</td><td>755</td><td>1753</td><td>93</td><td>UDHR</td><td>czech</td></tr><tr><td>##</td><td>UDHR_danish.txt</td><td>697</td><td>2087</td><td>131</td><td>UDHR</td><td>danish</td></tr><tr><td>##</td><td>UDHR_english.txt</td><td>557</td><td>1970</td><td>93</td><td>UDHR</td><td>english</td></tr><tr><td>##</td><td>UDHR_french.txt</td><td>671</td><td>2188</td><td>90</td><td>UDHR</td><td>french</td></tr><tr><td>##</td><td>UDHR_georgian.txt</td><td>770</td><td>2765</td><td>27</td><td>UDHR</td><td>georgian</td></tr><tr><td>##</td><td>UDHR_greek.txt</td><td>735</td><td>2193</td><td>106</td><td>UDHR</td><td>greek</td></tr><tr><td>##</td><td>UDHR_hungarian.txt</td><td>750</td><td>1843</td><td>93</td><td>UDHR</td><td>hungarian</td></tr><tr><td>##</td><td>UDHR_icelandic.txt</td><td>709</td><td>1916</td><td>105</td><td>UDHR</td><td>icelandic</td></tr><tr><td>##</td><td>UDHR.irish.txt</td><td>704</td><td>2197</td><td>117</td><td>UDHR</td><td>irish</td></tr><tr><td>##</td><td>UDHR_japanese.txt</td><td>586</td><td>2553</td><td>97</td><td>UDHR</td><td>japanese</td></tr><tr><td>##</td><td>UDHR_russian.txt</td><td>763</td><td>1875</td><td>94</td><td>UDHR</td><td>russian</td></tr><tr><td>##</td><td>UDHR_vietnamese.txt</td><td>191</td><td>2597</td><td>9</td><td>UDHR</td><td>vietnamese</td></tr></tbody></table>

### 5.5.2.2 Accessing parts of corpus

dat_udhr_corpustext[4]

[1] "Universal Declaration of Human Rights\nPreamble Whereas recognition of the inhe

### 5.5.2.3 Document-level information

head(docvars.dat_udhr_corpustext)

<table><thead><tr><th></th><th>document</th><th>language</th></tr></thead><tbody><tr><td>## 1</td><td>UDHR chinese</td></tr><tr><td>## 2</td><td>UDHR czech</td></tr><tr><td>## 3</td><td>UDHR danish</td></tr></tbody></table>
## UDHR_georgian.txt :
## [1] "FLFVBFYBC" "EAKT" "<" "FSF" "CF"
## [6] ">" "JDTKSFJ" "LTRKFHFWBF" "GHTFV" "<"
## [11] "EKF" "dbyfblfy"
## [ ... and 2,753 more ]
##
## [ reached max_ndoc ... 7 more documents ]

dat_udhr_corpus_tokens <- tokens (dat_udhr_corpus, remove_punct = TRUE)
dat_udhr_corpus_tokens_no_punct

5.5.3.1.2 Without punctuations

## Tokens consisting of 13 documents and 2 docvars.

## UDHR_chinese.txt :
## [1] " " " " " " " " " " " " " "
## [9] " " " " " " " " " " " " "
## [ ... and 2,603 more ]
##

##
## UDHR_czech.txt :
## [1] "VŠEOBECNÁ" "DEKLARACE" "LIDSKÝCH" "PRÁV" "Úvod"
## [6] "U" "vědomí" "toho" "že" "uznání"
## [11] "přirozené" "důstojnosti"
## [ ... and 1,519 more ]

##
## UDHR_danish.txt :
## [1] "Den" "10" "december"
## [4] "1948" "vedtog" "og"
## [7] "offentliggjorde" "FNs" "tredie"
## [10] "generalforsamling" "Verdenserklæringen" "om"
## [ ... and 1,814 more ]

##
## UDHR_english.txt :
## [1] "Universal" "Declaration" "of" "Human" "Rights"
## [6] "Preamble" "Whereas" "recognition" "of" "the"
## [11] "inherent" "dignity"
## [ ... and 1,766 more ]

##
## UDHR_french.txt :
## [1] "Déclaration" "universelle" "des" "droits"
## [5] "de" "l'homme" "Préambule" "Considérant"
## [9] "que" "la" "reconnaissance" "de"
## [ ... and 1,962 more ]

##
## UDHR_georgian.txt :
## [1] "FLFVBFYBC" "EAKT" "<" "FSF" "CF"
## [6] ">" "JDTKSFJ" "LTRKFHFWBF" "GHTFV" "<"
## [11] "EKF" "dbyfblfy"
## [ ... and 2,023 more ]
##
## [ reached max_ndoc ... 7 more documents ]

### 5.5.3.2 Compound words

```
dat_udhr_corpus_tokens_no_punct_phrase <- kwic.dat_udhr_corpus_tokens_no_punct, pattern = phrase("Human rights are the strength of a free and independent nation.")
```

#### 5.5.3.2.1 kwic Phrase

```
## Keyword-in-context with 7 matches.
## [UDHR_english.txt, 4:5] Universal Declaration of
## [UDHR_english.txt, 43:44] world Whereas disregard and contempt for
## [UDHR_english.txt, 116:117] rebellion against tyranny and oppression that
## [UDHR_english.txt, 155:156] Charter reaffirmed their faith in fundamental
## [UDHR_english.txt, 213:214] universal respect for and observance of
## [UDHR_english.txt, 249:250] Assembly Proclaims this Universal Declaration of
## [UDHR_english.txt, 1500:1501] to the strengthening of respect for
##
## | Human Rights | Preamble Whereas recognition of the inherent
## | human rights | have resulted in barbarous acts which
## | human rights | should be protected by the rule
## | human rights | in the dignity and worth of
## | human rights | and fundamental freedoms Whereas a common
## | Human Rights | as a common standard of achievement
## | human rights | and fundamental freedoms It shall promote
```

```
dat_udhr_corpus_tokens_no_punct_comp <- tokens_compound.dat_udhr_corpus_tokens_no_punct, pattern = phrase("Human rights are the strength of a free and independent nation.")
dat_udhr_corpus_tokens_no_punct_comp_kwic <- kwic.dat_udhr_corpus_tokens_no_punct_comp, pattern = phrase("Human rights are the strength of a free and independent nation.")
head.dat_udhr_corpus_tokens_no_punct_comp_kwic, 10)
```

#### 5.5.3.2.2 Compounds

```
## Keyword-in-context with 7 matches.
## [UDHR_english.txt, 4] Universal Declaration of
## [UDHR_english.txt, 42] Whereas disregard and contempt for
## [UDHR_english.txt, 114] against tyranny and oppression that
## [UDHR_english.txt, 152] reaffirmed their faith in fundamental
## [UDHR_english.txt, 209] respect for and observance of
## [UDHR_english.txt, 244] Proclaims this Universal Declaration of
## [UDHR_english.txt, 1494] the strengthening of respect for
```
<table><tr><td>##</td><td>Human_Rights</td><td>Preamble Whereas recognition of the humanRights</td></tr><tr><td>##</td><td>humanRights</td><td>have resulted in barbarous acts</td></tr><tr><td>##</td><td>humanRights</td><td>should be protected by the humanRights</td></tr><tr><td>##</td><td>humanRights</td><td>in the dignity and worth of the humanRights</td></tr><tr><td>##</td><td>humanRights</td><td>and fundamental freedoms Whereas a HumanRights</td></tr><tr><td>##</td><td>humanRights</td><td>as a common standard of humanRights</td></tr><tr><td>##</td><td>humanRights</td><td>and fundamental freedoms It shall</td></tr></table>

### 5.5.3.3 N-grams

N-grams are a subfamily of compound words. They can be named as “bi-grams”, “tri-grams”, etc. N-grams yield a sequence of tokens from already tokenised text object.

#### 5.5.3.3.1 Multi-grams
The code below allows to obtain the sequences of consecutive compound words, with 2, 3 or 4 compound words.

```
dat_udhr_corpus_tok_no_punct_ngram <- tokens_ngrams (dat_udhr_corpus tok_no_punct, n = 2)
unlist() %>%
to lower() %>%
table()
## Top 10 rows
head (dat_udhr_corpus tok_no_punct_ngram, 10)
```

<table><tr><td>##</td><td>#_8_fv</td><td>#_8_fv_ltrkfhfwfib</td></tr><tr><td>##</td><td>1</td><td>1</td></tr><tr><td>##</td><td>#_8_fv_ltrkfhfwfib_fhfathb</td><td>#8_8</td></tr><tr><td>##</td><td>1</td><td>2</td></tr><tr><td>##</td><td>#8_8_jdkt</td><td>#8_8_jdkt_flfvbfc</td></tr><tr><td>##</td><td>2</td><td>2</td></tr><tr><td>##</td><td>#8_fk</td><td>#8_fk_bc</td></tr><tr><td>##</td><td>1</td><td>1</td></tr><tr><td>##</td><td>#8_fk_bc_yt</td><td>#8_fv</td></tr><tr><td>##</td><td>1</td><td>1</td></tr></table>

```
## Last 10 rows
tail (dat_udhr_corpus tok_no_punct_ngram, 10)
```

<table><tr><td>##</td><td>---</td><td>1</td><td>-</td><td>1</td><td>-</td><td>1</td><td>1</td></tr><tr><td>##</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>1</td></tr><tr><td>##</td><td>1</td><td>-</td><td>-</td><td>1</td><td>-</td><td>1</td><td>1</td></tr><tr><td>##</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>1</td></tr></table>
5.5.3.3.2 Skip-grams Skip-grams allow to obtain non consecutive n-grams

```
dat_udhr_corpus_tokens_no_punct_ngram_skip <- tokens_ngrams (dat_udhr_corpus_tokens_no_punct, n = 2:4, s = 10)
unlist() %>%
toower() %>%
table()

## Top 10 rows
head (dat_udhr_corpus_tokens_no_punct_ngram_skip, 10)
```

```
## .
##                 #_8_fhfathb          #_8_fhfathb_eylf
##                 1                   1
##                 #_8_fhfathb_eylf_hjujhw #_8_fhfathb_eylf_vbybzt
##                 1                   1
##                 #_8_fhfathb_ufybvfhnjc7 #_8_fhfathb_ufybvfhnjc7_f
##                 1                   1
##                 #_8_fhfathb_ufybvfhnjc7_vbybzt #_8_ltrkfhfwbfib
##                 1                   1
##                 #_8_ltrkfhfwbfib_eylf   #_8_ltrkfhfwbfib_eylf_hjujhw
##                 1                   1
```

```
## Last 10 rows
tail (dat_udhr_corpus_tokens_no_punct_ngram_skip, 10)
```

```
## .
##                 --  --  --  --  --  --  --  --  --  --  --
##                 1       1       1       1       1       1       1
##                 --  --  --  --  --  --  --  --  --  --  --
##                 1       1       1       1       1       1       1
```

5.5.3.4 Dictionary

If you have a dictionary with various words that fall within a generic word (e.g., variants of pronunciation of a word), then you can look these up. Here, we will create a dictionary that we populate ourselves and we show how to use it to search for items

```
dict_dat_udhr <- dictionary (list (Human = c("Human*", "people"),
                                bad_aspects = c("tyranny", "barbarous", "oppression", "disregard")))
print (dict_dat_udhr)
```

5.5.3.4.1 Create dictionary

```
## Dictionary object with 2 key entries.
## - [Human]:
## - human*, people
## - [bad_aspects]:
```
## - tyranny, barbarous, oppression, disregard

```
dat_udhr_corpus_tokens_no_punct_dict_tokens <- tokens_lookup.dat_udhr_corpus_tokens_no_punct, print.dat_udhr_corpus_tokens_no_punct_dict_tokens)
```

### 5.5.3.4.2 Token lookup

## Tokens consisting of 13 documents and 2 docvars.

## UDHR_chinese.txt :

## character(0)

##

## UDHR_czech.txt :

## character(0)

##

## UDHR_danish.txt :

## character(0)

##

## UDHR_english.txt :

## [1] "Human" "Human" "bad_aspects" "Human" "bad_aspects"

## [6] "Human" "Human" "bad_aspects" "bad_aspects" "Human"

## [11] "Human" "Human"

## [ ... and 7 more ]

##

## UDHR_french.txt :

## character(0)

##

## UDHR_georgian.txt :

## character(0)

##

## [ reached max_ndoc ... 7 more documents ]

```
dfm.dat_udhr_corpus_tokens_no_punct_dict_tokens)
```

### 5.5.3.4.3 DFM

## Document-feature matrix of: 13 documents, 2 features (92.31% sparse) and 2 docvars.

## features

## docs human bad_aspects

## UDHR_chinese.txt 0 0

## UDHR_czech.txt 0 0

## UDHR_danish.txt 0 0

## UDHR_english.txt 15 4

## UDHR_french.txt 0 0

## UDHR_georgian.txt 0 0

## [ reached max_ndoc ... 7 more documents ]
5.5.3.5 Part of Speech tagging

Part-of-Speech tagging (or PoS-Tagging) is used to distinguish different part of
speech, e.g., the sentence: “Jane likes the girl” can be tagged as “Jane/NNP
likes/VBZ the/DT girl/NN”, where NNP = proper noun (singular), VBZ = 3rd
person singular present tense verb, DT = determiner, and NN = noun (singular
or mass). We will use the udpipe package

5.5.3.5.1 Download and load language models
Before using the PoS-
tagger, we need to download a language model. As you can see from typing
?udpipe_download_model, there are 65 languages trained on 101 treebanks
from here

Because we have 13 languages in our document, we can download the language
models for all of the ones available.

These are the models available: - chinese: chinese-gsd - czech: czech-cac - dan-
ish: danish-ddt - english: english-ewt - french: french-partut - georgian: NO
language model! - greek: greek-gdt - hungarian: hungarian-szeged - icelandic:
NO language model! - irish: irish-idt - japanese: japanese-gsd - russian: russian-
gsd - vietnamese: vietnamese-vtb

With the function below, we check if the file exists and if this is the case, we
load it; otherwise, we download it and load it later on

file_to_check <- list("models/chinese-gsd-ud-2.5-191206.udpipe", "models/czech-cac-ud-2.5-191206 udpipe", "models/english-ewt-ud-2.5-191206.udpipe", "models/french-partut-ud-2.5-191206.udpipe", "models/greek-gdt-ud-2.5-191206.udpipe", "models/hungarian-szeged-ud-2.5-191206.udpipe", "models/irish-idt-ud-2.5-191206 udpipe", "models/japanese-gsd-ud-2.5-191206.udpipe", "models/russian-gsd-ud-2.5-191206 udpipe", "models/vietnamese-vtb-ud-2.5-191206 udpipe")

for (x in file_to_check){
  if (file.exists(file = x)){
    m_chinese <- udpipe_load_model(file = "models/chinese-gsd-ud-2.5-191206.udpipe")
    m_czech <- udpipe_load_model(file = "models/czech-cac-ud-2.5-191206.udpipe")
    m_danish <- udpipe_load_model(file = "models/danish-ddt-ud-2.5-191206.udpipe")
    m_english <- udpipe_load_model(file = "models/english-ewt-ud-2.5-191206.udpipe")
    m_french <- udpipe_load_model(file = "models/french-partut-ud-2.5-191206.udpipe")
    m_greek <- udpipe_load_model(file = "models/greek-gdt-ud-2.5-191206.udpipe")
    m_hungarian <- udpipe_load_model(file = "models/hungarian-szeged-ud-2.5-191206.udpipe")
    mIrish <- udpipe_load_model(file = "models/irish-idt-ud-2.5-191206 udpipe")
    m_japanese <- udpipe_load_model(file = "models/japanese-gsd-ud-2.5-191206.udpipe")
    m_russian <- udpipe_load_model(file = "models/russian-gsd-ud-2.5-191206.udpipe")
    m_vietnamese <- udpipe_load_model(file = "models/vietnamese-vtb-ud-2.5-191206 udpipe")
  }else{
    m_chinese <- udpipe_download_model(model_dir = "models/", language = "chinese-gsd")
  }
}
m_chinese <- udpipe_load_model(file = "models/chinese-gsd-ud-2.5-191206.udpipe")
m_czech <- udpipe_download_model(model_dir = "models/", language = "czech-cac")
m_czech <- udpipe_load_model(file = "models/czech-cac-ud-2.5-191206.udpipe")
m_danish <- udpipe_download_model(model_dir = "models/", language = "danish-ddt")
m_danish <- udpipe_load_model(file = "models/danish-ddt-ud-2.5-191206.udpipe")
m_english <- udpipe_download_model(model_dir = "models/", language = "english-ewt")
m_english <- udpipe_load_model(file = "models/english-ewt-ud-2.5-191206.udpipe")
m_french <- udpipe_download_model(model_dir = "models/", language = "french-partut")
m_french <- udpipe_load_model(file = "models/french-partut-ud-2.5-191206.udpipe")
m_greek <- udpipe_download_model(model_dir = "models/", language = "greek-gdt")
m_greek <- udpipe_load_model(file = "models/greek-gdt-ud-2.5-191206.udpipe")
m_hungarian <- udpipe_download_model(model_dir = "models/", language = "hungarian-sz")
m_hungarian <- udpipe_load_model(file = "models/hungarian-szeged-ud-2.5-191206.udpipe")
m_irish <- udpipe_download_model(model_dir = "models/", language = "irish-idt")
m_irish <- udpipe_load_model(file = "models/irish-idt-ud-2.5-191206.udpipe")
m_japanese <- udpipe_download_model(model_dir = "models/", language = "japanese-gsd")
m_japanese <- udpipe_load_model(file = "models/japanese-gsd-ud-2.5-191206.udpipe")
m_russian <- udpipe_download_model(model_dir = "models/", language = "russian-gsd")
m_russian <- udpipe_load_model(file = "models/russian-gsd-ud-2.5-191206.udpipe")
m_vietnamese <- udpipe_download_model(model_dir = "models/", language = "vietnamese-vt")
m_vietnamese <- udpipe_load_model(file = "models/vietnamese-vtb-ud-2.5-191206.udpipe")
}
}

### 5.5.3.5.2 Tokenise, tag, dependency parsing
We use the already tokenised text, with no punctuations.

```
dat_udhr_anndf_chinese <- udpipe_ annotate(m_chinese, x = dat_udhr_corpus_tokens_no_punct[as.data.frame()
## inspect
head(d
```

#### 5.5.3.5.2.1 Chinese

<table><thead><tr><th></th><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th></tr></thead><tbody><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 2</td><td>doc2</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 3</td><td>doc2</td><td>1</td><td>1</td><td></td><td>2</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 4</td><td>doc3</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 5</td><td>doc4</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 6</td><td>doc4</td><td>1</td><td>1</td><td></td><td>2</td><td></td><td>VERB</td><td>VV</td><td></td></tr><tr><td>## 7</td><td>doc5</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 8</td><td>doc6</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>PART</td><td>PFA</td><td></td></tr><tr><td>## 9</td><td>doc6</td><td>1</td><td>1</td><td></td><td>2</td><td></td><td>NOUN</td><td>NN</td><td></td></tr><tr><td>## 10</td><td>doc7</td><td>1</td><td>1</td><td></td><td>1</td><td></td><td>PROPN</td><td>NNP</td><td></td></tr></tbody></table>
<table><thead><tr><th></th><th>feats</th><th>head_token_id</th><th>dep_rel</th><th>deps</th><th>misc</th></tr></thead><tbody><tr><td>##</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 1</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 2</td><td>&lt;NA&gt;</td><td>2</td><td>nmod</td><td>&lt;NA&gt;</td><td>SpaceAfter=No</td></tr><tr><td>## 3</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 4</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 5</td><td>&lt;NA&gt;</td><td>2</td><td>nsubj</td><td>&lt;NA&gt;</td><td>SpaceAfter=No</td></tr><tr><td>## 6</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 7</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 8</td><td>&lt;NA&gt;</td><td>2</td><td>case:pref</td><td>&lt;NA&gt;</td><td>SpaceAfter=No</td></tr><tr><td>## 9</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 10</td><td>&lt;NA&gt;</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr></tbody></table>

```
dat_udhr_anndf_czech <- udpipe_ annotate(m_czech, x = dat_udhr_corpus_token_no_punct[[2]]) %>%
as.data.frame()
```

### inspect

```
head(dat_udhr_anndf_czech, 10)
```

### 5.5.3.5.2.2 Czech

<table><thead><tr><th></th><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th><th>lemma</th></tr></thead><tbody><tr><td>##</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td><td>VŠEOBECNÁ</td><td>1</td><td>VŠEOBECNÁ</td><td>Všeobecnat</td></tr><tr><td>## 2</td><td>doc2</td><td>1</td><td>1</td><td>DEKLARACE</td><td>1</td><td>DEKLARACE</td><td>DEKLARACE</td></tr><tr><td>## 3</td><td>doc3</td><td>1</td><td>1</td><td>LIDSKÝCH</td><td>1</td><td>LIDSKÝCH</td><td>lidský</td></tr><tr><td>## 4</td><td>doc4</td><td>1</td><td>1</td><td>PRÁV</td><td>1</td><td>PRÁV</td><td>právo</td></tr><tr><td>## 5</td><td>doc5</td><td>1</td><td>1</td><td>Úvod</td><td>1</td><td>Úvod</td><td>úvod</td></tr><tr><td>## 6</td><td>doc6</td><td>1</td><td>1</td><td>U</td><td>1</td><td>U</td><td>u</td></tr><tr><td>## 7</td><td>doc7</td><td>1</td><td>1</td><td>vědomí</td><td>1</td><td>vědomí</td><td>vědomí</td></tr><tr><td>## 8</td><td>doc8</td><td>1</td><td>1</td><td>toho</td><td>1</td><td>toho</td><td>ten</td></tr><tr><td>## 9</td><td>doc9</td><td>1</td><td>1</td><td>že</td><td>1</td><td>že</td><td>že</td></tr><tr><td>## 10</td><td>doc10</td><td>1</td><td>1</td><td>uznání</td><td>1</td><td>uznání</td><td>uznání</td></tr><tr><td>##</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>##</td><td>upos</td><td></td><td></td><td>xpos</td><td></td><td></td><td></td></tr><tr><td>## 1</td><td>NOUN</td><td>NNFS1-----A----</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 2</td><td>NOUN</td><td>NNNX-----A---8</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 3</td><td>ADJ</td><td>AAF2-----1A----</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 4</td><td>ADJ</td><td>AUIS1M--------</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 5</td><td>NOUN</td><td>NNIS4-----A----</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 6</td><td>ADP</td><td>RR--2---------</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 7</td><td>NOUN</td><td>NNNS1-----A----</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 8</td><td>DET</td><td>PDZS2---------</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 9</td><td>SCONJ</td><td>J,------------</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 10</td><td>NOUN</td><td>NNNS4-----A----</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>##</td><td></td><td></td><td></td><td></td><td></td><td></td><td>feats</td></tr><tr><td>## 1</td><td></td><td></td><td></td><td></td><td></td><td>Case=Nom|Gender=Fem|Number=Sing|Polarity=Pos</td><td></td></tr><tr><td>## 2</td><td></td><td></td><td></td><td></td><td></td><td>Abbr=Yes|Gender=Neut|Polarity=Pos</td><td></td></tr><tr><td>## 3</td><td></td><td></td><td></td><td></td><td></td><td>Case=Gen|Degree=Pos|Gender=Fem|NameType=Geo|Number=Plur|Polarity=Pos</td><td></td></tr></tbody></table>
<table><thead><tr><th>##</th><th>head_token_id</th><th>dep_rel</th><th>deps</th><th>misc</th></tr></thead><tbody><tr><td>1</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>2</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>3</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>4</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>5</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>6</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>7</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>8</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>9</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>10</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr></tbody></table>

```
dat_udhr_anndf_danish <- udpipe_ annotate(m_danish, x = dat_udhr_corpus_token_no_punct[[3]]
as.data.frame()
```

```
## inspect
```

```
head (dat_udhr_anndf_danish, 10)
```

### 5.5.3.5.2.3 Danish

<table><thead><tr><th>##</th><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence_token_id</th><th>token</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1</td><td>Den</td><td>Den</td></tr><tr><td>2</td><td>doc2</td><td>1</td><td>1</td><td>10</td><td>10</td></tr><tr><td>3</td><td>doc3</td><td>1</td><td>1</td><td>december</td><td>december</td></tr><tr><td>4</td><td>doc4</td><td>1</td><td>1</td><td>1948</td><td>1948</td></tr><tr><td>5</td><td>doc5</td><td>1</td><td>1</td><td>vedtog</td><td>vedtog</td></tr><tr><td>6</td><td>doc6</td><td>1</td><td>1</td><td>og</td><td>og</td></tr><tr><td>7</td><td>doc7</td><td>1</td><td>1</td><td>offentliggjorde</td><td>offentliggjorde</td></tr><tr><td>8</td><td>doc8</td><td>1</td><td>1</td><td>FNs</td><td>FNs</td></tr><tr><td>9</td><td>doc9</td><td>1</td><td>1</td><td>tredie</td><td>tredie</td></tr><tr><td>10</td><td>doc10</td><td>1</td><td>1</td><td>generalforsamling</td><td>generalforsamling</td></tr><tr><td>1</td><td>lemma</td><td>upos</td><td>xpos</td><td></td><td>feats</td></tr><tr><td>2</td><td>den</td><td>DET</td><td>&lt;NA&gt;</td><td>Gender=Com|Number=Sing|PronType=Dem</td><td></td></tr><tr><td>3</td><td>december</td><td>NOUN</td><td>&lt;NA&gt;</td><td>NumType=Card</td><td></td></tr><tr><td>4</td><td>1948</td><td>NUM</td><td>&lt;NA&gt;</td><td>Definite=Ind|Gender=Com|Number=Sing</td><td></td></tr><tr><td>5</td><td>vedtage</td><td>ADV</td><td>&lt;NA&gt;</td><td>NumType=Card</td><td>&lt;NA&gt;</td></tr><tr><td>6</td><td>og</td><td>CCONJ</td><td>&lt;NA&gt;</td><td></td><td>&lt;NA&gt;</td></tr><tr><td>7</td><td>offentliggøre</td><td>VERB</td><td>&lt;NA&gt;</td><td>Mood=Ind|Tense=Past|VerbForm=Fin|Voice=Act</td><td></td></tr></tbody></table>
<table><tr><td>## 8</td><td>FN</td><td>PROPN</td><td>&lt;NA&gt;</td><td>Case=Gen</td></tr><tr><td>## 9</td><td>tredie</td><td>NOUN</td><td>&lt;NA&gt;</td><td>Definite=Ind|Gender=Com|Number=Sing</td></tr><tr><td>## 10</td><td>generalforsamling</td><td>NOUN</td><td>&lt;NA&gt;</td><td>Definite=Ind|Gender=Com|Number=Sing</td></tr><tr><td>##</td><td>head_token_id</td><td>dep_rel</td><td>deps</td><td>misc</td></tr><tr><td>## 1</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 2</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 3</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 4</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 5</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 6</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 7</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 8</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 9</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr><tr><td>## 10</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td></tr></table>

```
dat_udhr_annndf_english <- udpipe_ annotate(m_english, x = dat_udhr_corpus_tokens_no_punct[[4]]) %>%
as.data.frame()
```

## inspect

```
head/dat_udhr_annndf_english, 10)
```

### 5.5.3.5.2.4 English

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence_token_id</th><th>token</th><th>lemma</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1</td><td>Universal</td><td>1 Universal universal</td></tr><tr><td>2</td><td>doc2</td><td>1</td><td>1</td><td>Declaration</td><td>1 Declaration declaration</td></tr><tr><td>3</td><td>doc3</td><td>1</td><td>1</td><td>of</td><td>1 of of</td></tr><tr><td>4</td><td>doc4</td><td>1</td><td>1</td><td>Human</td><td>1 Human human</td></tr><tr><td>5</td><td>doc5</td><td>1</td><td>1</td><td>Rights</td><td>1 Rights right</td></tr><tr><td>6</td><td>doc6</td><td>1</td><td>1</td><td>Preamble</td><td>1 Preamble preamble</td></tr><tr><td>7</td><td>doc7</td><td>1</td><td>1</td><td>Whereas</td><td>1 Whereas whereas</td></tr><tr><td>8</td><td>doc8</td><td>1</td><td>1</td><td>recognition</td><td>1 recognition recognition</td></tr><tr><td>9</td><td>doc9</td><td>1</td><td>1</td><td>of</td><td>1 of of</td></tr><tr><td>10</td><td>doc10</td><td>1</td><td>1</td><td>the</td><td>1 the the</td></tr><tr><td>upos</td><td>xpos</td><td></td><td></td><td>feats</td><td>head_token_id dep_rel deps</td></tr><tr><td>1</td><td>PROPN</td><td>NNP</td><td></td><td>Number=Sing</td><td>0 root &lt;NA&gt;</td></tr><tr><td>2</td><td>NOUN</td><td>NN</td><td></td><td>Number=Sing</td><td>0 root &lt;NA&gt;</td></tr><tr><td>3</td><td>ADP</td><td>IN</td><td></td><td>&lt;NA&gt;</td><td>0 root &lt;NA&gt;</td></tr><tr><td>4</td><td>ADJ</td><td>JJ</td><td></td><td>Degree=Pos</td><td>0 root &lt;NA&gt;</td></tr><tr><td>5</td><td>NOUN</td><td>NNS</td><td></td><td>Number=Plur</td><td>0 root &lt;NA&gt;</td></tr><tr><td>6</td><td>NOUN</td><td>NN</td><td></td><td>Number=Sing</td><td>0 root &lt;NA&gt;</td></tr><tr><td>7</td><td>ADV</td><td>RB</td><td></td><td>&lt;NA&gt;</td><td>0 root &lt;NA&gt;</td></tr><tr><td>8</td><td>NOUN</td><td>NN</td><td></td><td>Number=Sing</td><td>0 root &lt;NA&gt;</td></tr><tr><td>9</td><td>ADP</td><td>IN</td><td></td><td>&lt;NA&gt;</td><td>0 root &lt;NA&gt;</td></tr><tr><td>10</td><td>DET</td><td>DT</td><td>Definite=Def|PronType=Art</td><td></td><td>0 root &lt;NA&gt;</td></tr><tr><td></td><td></td><td></td><td>misc</td><td></td><td></td></tr></tbody></table>
## 1 SpacesAfter=\\n
## 2 SpacesAfter=\\n
## 3 SpacesAfter=\\n
## 4 SpacesAfter=\\n
## 5 SpacesAfter=\\n
## 6 SpacesAfter=\\n
## 7 SpacesAfter=\\n
## 8 SpacesAfter=\\n
## 9 SpacesAfter=\\n
## 10 SpacesAfter=\\n

dat_udhr_anndf_french <- udpipe_ annotate(m_french, x = dat_udhr_corpus_token_no_punct)[5]
as.data.frame()
## inspect
head.dat_udhr_anndf_french, 10)

5.5.3.5.2.5 French

## doc_id paragraph_id sentence_id sentence token_id token lemma
## 1 doc1 1 1 Déclaration 1 Déclaration déclaration
## 2 doc2 1 1 universelle 1 universelle universelle
## 3 doc3 1 1 des 1-2 des <NA>
## 4 doc3 1 1 des 1 de de
## 5 doc3 1 1 des 2 les le
## 6 doc4 1 1 droits 1 droits droits
## 7 doc5 1 1 de 1 de de
## 8 doc6 1 1 l'homme 1 l' le
## 9 doc6 1 1 l'homme 2 homme homme
## 10 doc7 1 1 Préambule 1 Préambule Préambule
## upos xpos feats head_token_id dep_rel deps
## 1 NOUN S Gender=Fem|Number=Sing 0 root <NA>
## 2 PRON PQ Gender=Fem|Number=Sing|PronType=Int 0 root <NA>
## 3 <NA> <NA> <NA> <NA> <NA>
## 4 ADP E <NA> 2 case <NA>
## 5 DET RD Definite=Def|Number=Plur|PronType=Art 0 root <NA>
## 6 NOUN S Gender=Masc|Number=Plur 0 root <NA>
## 7 ADP E <NA> 0 root <NA>
## 8 DET RD Definite=Def|Number=Sing|PronType=Art 2 det <NA>
## 9 NOUN S Gender=Masc|Number=Sing 0 root <NA>
## 10 PROPN SP <NA> 0 root <NA>
## misc
## 1 SpacesAfter=\\n
## 2 SpacesAfter=\\n
## 3 SpacesAfter=\\n
## 4 <NA>
<table><tr><td>## 5</td><td>&lt;NA&gt;</td></tr><tr><td>## 6</td><td>SpacesAfter=\\n</td></tr><tr><td>## 7</td><td>SpacesAfter=\\n</td></tr><tr><td>## 8</td><td>SpaceAfter=No</td></tr><tr><td>## 9</td><td>SpacesAfter=\\n</td></tr><tr><td>## 10</td><td>SpacesAfter=\\n</td></tr></table>

```
dat_udhr_anndf_greek <- udpipe_ annotate(m_greek, x = dat_udhr_corpus_tokens_no_punct[[7]]) %>%
as.data.frame()
## inspect
head.dat_udhr_anndf_greek, 10)
```

### 5.5.3.5.2.6 Greek

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th><th>lemma</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1</td><td>OIKOTMENIKH</td><td>1</td><td>OIKOTMENIKH OIKOTMENIKH</td></tr><tr><td>2</td><td>doc2</td><td>1</td><td>1</td><td>ΔΙΑΚΗΡΤΞΗ</td><td>1</td><td>ΔΙΑΚΗΡΤΞΗ ΔΙΑΚΗΡΤΞΗ</td></tr><tr><td>3</td><td>doc3</td><td>1</td><td>1</td><td>ΓΙΑ</td><td>1</td><td>ΓΙΑ</td></tr><tr><td>4</td><td>doc4</td><td>1</td><td>1</td><td>ΤΑ</td><td>1</td><td>ΤΑ</td></tr><tr><td>5</td><td>doc5</td><td>1</td><td>1</td><td>ΑΝΘΡΩΠΙΝΑ</td><td>1</td><td>ΑΝΘΡΩΠΙΝΑ ΑΝΘΡΩΠΙΝΑ</td></tr><tr><td>6</td><td>doc6</td><td>1</td><td>1</td><td>ΔΙΚΑΙΩΜΑΤΑ</td><td>1</td><td>ΔΙΚΑΙΩΜΑΤΑ ΔΙΚΑΙΩΜΑΤΑ</td></tr><tr><td>7</td><td>doc7</td><td>1</td><td>1</td><td>10</td><td>1</td><td>10</td></tr><tr><td>8</td><td>doc8</td><td>1</td><td>1</td><td>ΔΕΚΕΜΒΡΙΟΤ</td><td>1</td><td>ΔΕΚΕΜΒΡΙΟΤ Δ</td></tr><tr><td>9</td><td>doc9</td><td>1</td><td>1</td><td>1948</td><td>1</td><td>1948</td></tr><tr><td>10</td><td>doc10</td><td>1</td><td>1</td><td>ΠΡΟΟΙΜΙΟ</td><td>1</td><td>ΠΡΟΟΙΜΙΟ Π</td></tr><tr><td colspan="7">upos_xpos feats</td></tr><tr><td>1</td><td>DET</td><td>DET</td><td>Case=Nom|Definite=Def|Gender=Fem|Number=Sing|PronType=Art</td><td></td><td></td><td></td></tr><tr><td>2</td><td>DET</td><td>DET</td><td>Case=Nom|Definite=Def|Gender=Fem|Number=Sing|PronType=Art</td><td></td><td></td><td></td></tr><tr><td>3</td><td>ADP</td><td>ADP</td><td></td><td></td><td></td><td>&lt;NA&gt;</td></tr><tr><td>4</td><td>DET</td><td>DET</td><td>Case=Nom|Definite=Def|Gender=Neut|Number=Plur|PronType=Art</td><td></td><td></td><td></td></tr><tr><td>5</td><td>NOUN</td><td>NOUN</td><td></td><td></td><td></td><td>Abbr=Yes</td></tr><tr><td>6</td><td>NOUN</td><td>NOUN</td><td></td><td></td><td></td><td>Abbr=Yes</td></tr><tr><td>7</td><td>NUM</td><td>NUM</td><td></td><td></td><td></td><td>NumType=Card</td></tr><tr><td>8</td><td>ADV</td><td>ADV</td><td></td><td></td><td></td><td>&lt;NA&gt;</td></tr><tr><td>9</td><td>NUM</td><td>NUM</td><td></td><td></td><td></td><td>NumType=Card</td></tr><tr><td>10</td><td>NOUN</td><td>NOUN</td><td></td><td></td><td></td><td>Abbr=Yes</td></tr><tr><td colspan="7">head_token_id dep_rel deps misc</td></tr><tr><td>1</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>2</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>3</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>4</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>5</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>6</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>7</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr><tr><td>8</td><td></td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td><td></td><td></td></tr></tbody></table>
<table><tr><td>## 2</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 3</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 4</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 5</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 6</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 7</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 8</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 9</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr><tr><td>## 10</td><td>0</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td></tr></table>

```
dat_udhr_anndf.irish <- udpipe_ annotate(m.irish, x = dat_udhr_corpus_token_no_punct[[10]]) %>%
as.data.frame()
```

```
## inspect
```

```
head.dat_udhr_anndf.irish, 10)
```

### 5.5.3.5.2.8 Irish

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>DEARBHÚ</td><td>1</td><td>DEARBHÚ</td></tr><tr><td>2</td><td>doc2</td><td>1</td><td>UILE-CHOITEANN</td><td>1</td><td>UILE-CHOITEANN</td></tr><tr><td>3</td><td>doc3</td><td>1</td><td>CEARTA</td><td>1</td><td>CEARTA</td></tr><tr><td>4</td><td>doc4</td><td>1</td><td>AN</td><td>1</td><td>AN</td></tr><tr><td>5</td><td>doc5</td><td>1</td><td>DUINE</td><td>1</td><td>DUINE</td></tr><tr><td>6</td><td>doc6</td><td>1</td><td>Preamble</td><td>1</td><td>Preamble</td></tr><tr><td>7</td><td>doc7</td><td>1</td><td>De</td><td>1</td><td>De</td></tr><tr><td>8</td><td>doc8</td><td>1</td><td>Bhrí</td><td>1</td><td>Bhrí</td></tr><tr><td>9</td><td>doc9</td><td>1</td><td>gurb</td><td>1</td><td>gurb</td></tr><tr><td>10</td><td>doc10</td><td>1</td><td>é</td><td>1</td><td>é</td></tr><tr><td colspan="6">lemma upos xpos feats head_token_id</td></tr><tr><td>1</td><td>dearbh</td><td>NOUN</td><td>Case=NomAcc|Gender=Masc|Number=Sing</td><td></td><td>0</td></tr><tr><td>2</td><td>Uile-choit</td><td>VERB</td><td>VTI Mood=Ind|Tense=Pres</td><td></td><td>0</td></tr><tr><td>3</td><td>ceart</td><td>NOUN</td><td>Case=NomAcc|Gender=Masc|Number=Sing</td><td></td><td>0</td></tr><tr><td>4</td><td>is</td><td>AUX</td><td>Cop Mood=Int|Tense=Pres|VerbForm=Cop</td><td></td><td>0</td></tr><tr><td>5</td><td>duine</td><td>NOUN</td><td>Case=NomAcc|Gender=Masc|Number=Sing</td><td></td><td>0</td></tr><tr><td>6</td><td>Preamble</td><td>ADP</td><td>Simp &lt;NA&gt;</td><td></td><td>0</td></tr><tr><td>7</td><td>de</td><td>ADP</td><td>Simp &lt;NA&gt;</td><td></td><td>0</td></tr><tr><td>8</td><td>brí</td><td>NOUN</td><td>Noun Form=Len|Gender=Fem|Number=Sing</td><td></td><td>0</td></tr><tr><td>9</td><td>is</td><td>AUX</td><td>Cop Form=VF|Tense=Pres|VerbForm=Cop</td><td></td><td>0</td></tr><tr><td>10</td><td>é</td><td>PRON</td><td>Pers Gender=Masc|Number=Sing|Person=3</td><td></td><td>0</td></tr><tr><td colspan="6">dep_rel deps misc</td></tr><tr><td>1</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td><td></td><td></td></tr><tr><td>2</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td><td></td><td></td></tr><tr><td>3</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td><td></td><td></td></tr><tr><td>4</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td><td></td><td></td></tr><tr><td>5</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\n</td><td></td><td></td></tr></tbody></table>
## 6 root <NA> SpacesAfter=\\n
## 7 root <NA> SpacesAfter=\\n
## 8 root <NA> SpacesAfter=\\n
## 9 root <NA> SpacesAfter=\\n
## 10 root <NA> SpacesAfter=\\n

dat_udhr_anndf_japanese <- udpipe_ annotate(m_japanese, x = dat_udhr_corpus_token_no_punct, as.data.frame())
## inspect
head.dat_udhr_anndf_japanese, 10)

### 5.5.3.5.2.9 Japanese

## doc_id paragraph_id sentence_id sentence_token_id token lemma
## 1 doc1 1 1 1
## 2 doc2 1 1 1
## 3 doc3 1 1 1
## 4 doc4 1 1 1948.12.10 1 1948.12.10 1948.12.10
## 5 doc5 1 1 1
## 6 doc6 1 1 1
## 7 doc7 1 1 1
## 8 doc8 1 1 1
## 9 doc9 1 1 1
## 10 doc10 1 1 1
## upos xpos feats head_token_id dep_rel deps misc
## 1 NOUN NN <NA> 0 root <NA> SpacesAfter=\\n
## 2 NOUN NN <NA> 0 root <NA> SpacesAfter=\\

## 3 NOUN NN <NA> 0 root <NA> SpacesAfter=\\

## 4 NUM CD NumType=Card 0 root <NA> SpacesAfter=\\

## 5 PART XPC <NA> 0 root <NA> SpacesAfter=\\

## 6 NUM CD NumType=Card 0 root <NA> SpacesAfter=\\

## 7 NOUN NN <NA> 0 root <NA> SpacesAfter=\\

## 8 NOUN NN <NA> 0 root <NA> SpacesAfter=\\

## 9 NOUN NN <NA> 0 root <NA> SpacesAfter=\\

## 10 NOUN NN <NA> 0 root <NA> SpacesAfter=\\

dat_udhr_anndf_russian <- udpipe_ annotate(m_russian, x = dat_udhr_corpus_token_no_punct, as.data.frame())
## inspect
head.dat_udhr_anndf_russian, 10)

### 5.5.3.5.2.10 Russian

## doc_id paragraph_id sentence_id sentence_token_id token
## 1 doc1 1 1 1
```
as.data.frame()
## inspect
head.dat_udhr_annndf_vietnamese, 10)
```

### 5.5.3.5.2.11 Vietnamese

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th>sentence</th><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1</td><td>7X</td><td>1</td><td>7X</td><td>7X</td><td>NOUN</td><td>Nb</td></tr><tr><td>2</td><td>doc2</td><td>1</td><td>1</td><td>zQ</td><td>1</td><td>zQ</td><td>zQ</td><td>NOUN</td><td>N</td></tr><tr><td>3</td><td>doc3</td><td>1</td><td>1</td><td>QJ</td><td>1</td><td>QJ</td><td>QJ</td><td>NOUN</td><td>N</td></tr><tr><td>4</td><td>doc4</td><td>1</td><td>1</td><td>Q</td><td>1</td><td>Q</td><td>Q</td><td>NOUN</td><td>N</td></tr><tr><td>5</td><td>doc5</td><td>1</td><td>1</td><td>WR</td><td>1</td><td>WR</td><td>WR</td><td>NOUN</td><td>N</td></tr><tr><td>6</td><td>doc6</td><td>1</td><td>1</td><td>Q</td><td>1</td><td>Q</td><td>Q</td><td>NOUN</td><td>N</td></tr><tr><td>7</td><td>doc7</td><td>1</td><td>1</td><td>WK</td><td>1</td><td>WK</td><td>WK</td><td>NOUN</td><td>Ny</td></tr><tr><td>8</td><td>doc8</td><td>1</td><td>1</td><td>JL</td><td>1</td><td>JL</td><td>JL</td><td>NOUN</td><td>N</td></tr><tr><td>9</td><td>doc9</td><td>1</td><td>1</td><td>±</td><td>1</td><td>±</td><td>±</td><td>SYM</td><td>&lt;NA&gt;</td></tr><tr><td>10</td><td>doc10</td><td>1</td><td>1</td><td>L</td><td>1</td><td>L</td><td>L</td><td>DET</td><td>L</td></tr></tbody></table><table><thead><tr><th>feats</th><th>head_token_id</th><th>dep_rel</th><th>deps</th><th>misc</th></tr></thead><tbody><tr><td>1</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>2</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>3</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>4</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>5</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>6</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>7</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>8</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>9</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr><tr><td>10</td><td>&lt;NA&gt;</td><td>0</td><td>root &lt;NA&gt; SpacesAfter=\\n</td><td></td></tr></tbody></table>

#### 5.5.3.5.3 Dependency parsing
Below, we look into the data and identify when there is a punctuation mark and use that as our limit for generating the dependency parsing plot. We start by obtaining the corpus for hte full textfile and then filter out.

```
## parse text
dat_inaug_corpus_sent_chinese <- udpipe_ annotate(m_chinese, x = dat_udhr_corpus[[1]])
as.data.frame()
## inspect
head.dat_inaug_corpus_sent_chinese)
```

### 5.5.3.5.3.1 Chinese

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td></tr><tr><td>2</td><td>doc1</td><td>1</td></tr></tbody></table>
<table><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>##</td><td></td><td></td><td></td></tr><tr><td>## 1</td><td></td><td>217A(III)</td><td>1948 12 10</td></tr><tr><td>## 2</td><td></td><td>217A(III)</td><td>1948 12 10</td></tr><tr><td>## 3</td><td></td><td>217A(III)</td><td>1948 12 10</td></tr><tr><td>## 4</td><td></td><td>217A(III)</td><td>1948 12 10</td></tr><tr><td>## 5</td><td></td><td>217A(III)</td><td>1948 12 10</td></tr><tr><td>## 6</td><td></td><td>217A(III)</td><td>1948 12 10</td></tr></table>

<table><thead><tr><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th><th>feats</th><th>head_token_id</th><th>dep_rel</th><th>deps</th></tr></thead><tbody><tr><td>## 1</td><td></td><td>1</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>2</td><td>compound &lt;NA&gt;</td></tr><tr><td>## 2</td><td></td><td>2</td><td>PART</td><td>SFN</td><td>&lt;NA&gt;</td><td></td><td>5</td><td>nmod &lt;NA&gt;</td></tr><tr><td>## 3</td><td></td><td>3</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>5</td><td>nmod &lt;NA&gt;</td></tr><tr><td>## 4</td><td></td><td>4</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>5</td><td>nmod &lt;NA&gt;</td></tr><tr><td>## 5</td><td></td><td>5</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>6</td><td>nsubj &lt;NA&gt;</td></tr><tr><td>## 6</td><td></td><td>6</td><td>VERB</td><td>VV</td><td>&lt;NA&gt;</td><td></td><td>0</td><td>root &lt;NA&gt;</td></tr><tr><td>##</td><td></td><td></td><td>misc</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 1</td><td></td><td></td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 2</td><td></td><td></td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 3</td><td></td><td></td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 4</td><td></td><td></td><td>SpacesAfter=\\n</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 5</td><td></td><td></td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>## 6</td><td></td><td></td><td>SpaceAfter=No</td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

dat_inaug_corpus_sent_chinese_short <- dat_inaug_corpus_sent_chinese[1:32,]

dat_inaug_corpus_sent_dplot_chinese <- textplot_dependencyparser.dat_inaug_corpus_sent_chinese_s

## show plot

dat_inaug_corpus_sent_dplot_chinese
<figure><img src="image_145.png" /></figure>

Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_146.png" /></figure>

## parse text
dat_inaug_corpus_sent_danish <- udpipe_ annotate(m_danish, x = dat_udhr_corpus[[3]]) %>%
as.data.frame()
## inspect

head.dat_inaug_corpus_sent_danish)

5.5.3.5.3.3 Danish

<table><thead><tr><th>doc_id</th><th>paragraph_id</th><th>sentence_id</th><th></th></tr></thead><tbody><tr><td>1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>6</td><td>doc1</td><td>1</td><td>1</td></tr></tbody></table>

<table><thead><tr><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th><th>feats</th></tr></thead><tbody><tr><td>1</td><td>Den</td><td>den</td><td>DET</td><td>&lt;NA&gt;</td><td>Gender=Com|Number=Sing|PronType=Dem</td></tr><tr><td>2</td><td>10.</td><td>10.</td><td>ADJ</td><td>&lt;NA&gt;</td><td>NumType=Ord</td></tr><tr><td>3</td><td>3 december</td><td>december</td><td>NOUN</td><td>&lt;NA&gt;</td><td>Definite=Ind|Gender=Com|Number=Sing</td></tr><tr><td>4</td><td>1948</td><td>1948</td><td>NUM</td><td>&lt;NA&gt;</td><td>NumType=Card</td></tr><tr><td>5</td><td>vedtog</td><td>vedtog</td><td>NOUN</td><td>&lt;NA&gt;</td><td>Definite=Ind|Gender=Neut|Number=Plur</td></tr><tr><td>6</td><td>og</td><td>og</td><td>CCONJ</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td></tr><tr><td colspan="6">head_token_id dep_rel deps misc</td></tr><tr><td>1</td><td>3</td><td>det</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>2</td><td>3</td><td>amod</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>3</td><td>7</td><td>nsubj</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>4</td><td>5</td><td>nummod</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>5</td><td>3</td><td>nmod</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr><tr><td>6</td><td>3</td><td>dep</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td></tr></tbody></table>

dat_inaug_corpus_sent_danish_short <- dat_inaug_corpus_sent_danish[1:14,]

dat_inaug_corpus_sent_dplot_danish <- textplot_dependencyparser.dat_inaug_corpus_sent_danish

## show plot

dat_inaug_corpus_sent_dplot_danish
Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_147.png" /></figure>

### parse text

```
dat_udhr_corpus_english <- dat_udhr_corpus[[4]]
dat_udhr_corpus_english_short <- str_extract.dat_udhr_corpus_english, "Whereas recognition of the
```

```
dat_udhr_corpus_sent_english <- udpipe_ annotate(m_english, x = dat_udhr_corpus_english_short) %>
as.data.frame()
```

### inspect

```
head.dat_udhr_corpus_sent_english)
```

5.5.3.5.3.4 English

### doc_id paragraph_id sentence_id

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr></table>

###

```
## 1 Whereas recognition of the inherent dignity and of the equal and inalienable rights of all
## 2 Whereas recognition of the inherent dignity and of the equal and inalienable rights of all
## 3 Whereas recognition of the inherent dignity and of the equal and inalienable rights of all
## 4 Whereas recognition of the inherent dignity and of the equal and inalienable rights of all
## 5 Whereas recognition of the inherent dignity and of the equal and inalienable rights of all
## 6 Whereas recognition of the inherent dignity and of the equal and inalienable rights of all
```

```
## token_id      token          lemma upos xpos          feats
```

```
## 1          1      Whereas      whereas ADJ JJ          Degree=Pos
```
<table>
  <tr>
    <td>## 2</td>
    <td>2</td>
    <td>recognition</td>
    <td>recognition</td>
    <td>NOUN</td>
    <td>NN</td>
    <td>Number=Sing</td>
  </tr>
  <tr>
    <td>## 3</td>
    <td>3</td>
    <td>of</td>
    <td>of</td>
    <td>ADP</td>
    <td>IN</td>
    <td>&lt;NA&gt;</td>
  </tr>
  <tr>
    <td>## 4</td>
    <td>4</td>
    <td>the</td>
    <td>the</td>
    <td>DET</td>
    <td>DT</td>
    <td>Definite=Def|PronType=Art</td>
  </tr>
  <tr>
    <td>## 5</td>
    <td>5</td>
    <td>inherent</td>
    <td>inherent</td>
    <td>ADJ</td>
    <td>JJ</td>
    <td>Degree=Pos</td>
  </tr>
  <tr>
    <td>## 6</td>
    <td>6</td>
    <td>dignity</td>
    <td>dignity</td>
    <td>NOUN</td>
    <td>NN</td>
    <td>Number=Sing</td>
  </tr>
  <tr>
    <td>##</td>
    <td>head_token_id</td>
    <td>dep_rel</td>
    <td>deps</td>
    <td>misc</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>## 1</td>
    <td></td>
    <td>2</td>
    <td>amod</td>
    <td>&lt;NA&gt;</td>
    <td>&lt;NA&gt;</td>
    <td></td>
  </tr>
  <tr>
    <td>## 2</td>
    <td></td>
    <td>23</td>
    <td>nsubj</td>
    <td>&lt;NA&gt;</td>
    <td>&lt;NA&gt;</td>
    <td></td>
  </tr>
  <tr>
    <td>## 3</td>
    <td></td>
    <td>6</td>
    <td>case</td>
    <td>&lt;NA&gt;</td>
    <td>&lt;NA&gt;</td>
    <td></td>
  </tr>
  <tr>
    <td>## 4</td>
    <td></td>
    <td>6</td>
    <td>det</td>
    <td>&lt;NA&gt;</td>
    <td>&lt;NA&gt;</td>
    <td></td>
  </tr>
  <tr>
    <td>## 5</td>
    <td></td>
    <td>6</td>
    <td>amod</td>
    <td>&lt;NA&gt;</td>
    <td>&lt;NA&gt;</td>
    <td></td>
  </tr>
  <tr>
    <td>## 6</td>
    <td></td>
    <td>2</td>
    <td>nmod</td>
    <td>&lt;NA&gt;</td>
    <td>&lt;NA&gt;</td>
    <td></td>
  </tr>
</table>

```
dat_inaug_corpus_sent_dplot_english <- textplot_dependencyparser.dat_inaug_corpus_sent_dplot_english
## show plot
dat_inaug_corpus_sent_dplot_english
```

## Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_148.png" /></figure>

### parse text

```
dat_udhr_corpus_french <- dat_udhr_corpus[[5]]
dat_udhr_corpus_french_short <- str_extract.dat_udhr_corpus_french, "Considérant que l'anglais est un langage de la main, considérant que l'anglais est un langage de la main")
dat_inaug_corpus_sent_french <- udpipe_ annotate(m_french, x = dat_udhr_corpus_french_sents, as.data.frame())
```

### inspect

```
head.dat_inaug_corpus_sent_french
```

5.5.3.5.3.5 French
## doc_id paragraph_id sentence_id

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr></table>

##

## 1 Considérant que la reconnaissance de la dignité inhérente à tous les membres de la famille h
## 2 Considérant que la reconnaissance de la dignité inhérente à tous les membres de la famille h
## 3 Considérant que la reconnaissance de la dignité inhérente à tous les membres de la famille h
## 4 Considérant que la reconnaissance de la dignité inhérente à tous les membres de la famille h
## 5 Considérant que la reconnaissance de la dignité inhérente à tous les membres de la famille h
## 6 Considérant que la reconnaissance de la dignité inhérente à tous les membres de la famille h

## token_id token lemma upos xpos
## 1 1 considérant considérer VERB V
## 2 2 que que SCONJ CS
## 3 3 la le DET RD
## 4 4 reconnaissance reconnaissance NOUN S
## 5 5 de de ADP E
## 6 6 la le DET RD

## feats head_token_id dep_rel dep5
## 1 VerbForm=Ger 0 root <NA>
## 2 <NA> 24 mark <NA>
## 3 Definite=Def|Gender=Fem|Number=Sing|PronType=Art 4 det <NA>
## 4 Gender=Fem|Number=Sing 24 nsubj <NA>
## 5 <NA> 7 case <NA>
## 6 Definite=Def|Gender=Fem|Number=Sing|PronType=Art 7 det <NA>
## misc
## 1 <NA>
## 2 <NA>
## 3 <NA>
## 4 <NA>
## 5 <NA>
## 6 <NA>

dat_inaug_corpus_sent_dplot_french <- textplot_dependencyparser.dat_inaug_corpus_sent_french, size=10
## show plot
dat_inaug_corpus_sent_dplot_french
Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_149.png" /></figure>

### parse text

```
dat_inaug_corpus_sent_greek <- udpipe_ annotate(m_greek, x = dat_udhr_corpus[[7]]) %>%
as.data.frame()
```

### inspect

```
head.dat_inaug_corpus_sent_greek
```

5.5.3.5.3.6 Greek

### doc_id paragraph_id sentence_id

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>2</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>2</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>2</td></tr></table>

##

## 1

## 2

## 3

## 4 ΤΑ ΑΝΘΡΩΠΙΝΑ ΔΙΚΑΙΩΜΑΤΑ 10 ΔΕΚΕΜΒΡΙΟΤ 1948 ΠΡΟΟΙΜΙΟ Ε ,

## 5 ΤΑ ΑΝΘΡΩΠΙΝΑ ΔΙΚΑΙΩΜΑΤΑ 10 ΔΕΚΕΜΒΡΙΟΤ 1948 ΠΡΟΟΙΜΙΟ Ε ,

## 6 ΤΑ ΑΝΘΡΩΠΙΝΑ ΔΙΚΑΙΩΜΑΤΑ 10 ΔΕΚΕΜΒΡΙΟΤ 1948 ΠΡΟΟΙΜΙΟ Ε ,

<table><thead><tr><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th></tr></thead><tbody><tr><td>## 1</td><td>1 ΟΙΚΟΤΜΕΝΙΚΗ</td><td>Ο</td><td>DET</td><td>DET</td></tr><tr><td>## 2</td><td>2 ΔΙΑΚΗΡΤΞΗ</td><td>Δ</td><td>ADJ</td><td>ADJ</td></tr><tr><td>## 3</td><td>3 ΓΙΑ</td><td></td><td>NOUN</td><td>NOUN</td></tr></tbody></table>
<table><thead><tr><th></th><th></th><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td>## 4</td><td>1</td><td>TA</td><td>DET</td><td>DET</td><td></td></tr><tr><td>## 5</td><td>2</td><td>ANΘΡΩΠΙΝΑ</td><td>A</td><td>ADJ</td><td>ADJ</td></tr><tr><td>## 6</td><td>3</td><td>ΔΙΚΑΙΩΜΑΤΑ</td><td>ΔΙΚΑΙΩΜΑΤΑ</td><td>NOUN</td><td>NOUN</td></tr><tr><td colspan="6">##</td><td>feats</td><td>head_token_id</td></tr><tr><td>## 1</td><td>Case=Nom|Definite=Def|Gender=Fem|Number=Sing|PronType=Art</td><td></td><td></td><td></td><td>3</td></tr><tr><td>## 2</td><td></td><td>Case=Nom|Gender=Fem|Number=Sing</td><td></td><td></td><td>3</td></tr><tr><td>## 3</td><td></td><td>Case=Nom|Gender=Fem|Number=Sing</td><td></td><td></td><td>0</td></tr><tr><td>## 4</td><td>Case=Nom|Definite=Def|Gender=Neut|Number=Plur|PronType=Art</td><td></td><td></td><td></td><td>3</td></tr><tr><td>## 5</td><td></td><td>Case=Nom|Gender=Neut|Number=Plur</td><td></td><td></td><td>3</td></tr><tr><td>## 6</td><td></td><td>Case=Nom|Gender=Neut|Number=Plur</td><td></td><td></td><td>0</td></tr><tr><td colspan="6">## dep_rel</td><td>deps</td><td>misc</td></tr><tr><td>## 1</td><td>det</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td><td></td></tr><tr><td>## 2</td><td>amod</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td><td></td></tr><tr><td>## 3</td><td>root</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td><td></td></tr><tr><td>## 4</td><td>det</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td><td></td></tr><tr><td>## 5</td><td>amod</td><td>&lt;NA&gt;</td><td>&lt;NA&gt;</td><td></td><td></td></tr><tr><td>## 6</td><td>root</td><td>&lt;NA&gt;</td><td>SpacesAfter=\\</td><td></td><td></td></tr></tbody></table>

dat_inaugcorpus_sent_greek[4:51,]"

dat_inaug_corp

 Dat_inaug_corp_sent_dplot_greek <- textplot_dependencyparser (dat_inaug_corp_sent_greek_

## show plot

dat_inaug_corp_sent_dplot_greek

Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_150.png" /></figure>

## parse text

dat_inaug_corp_sent_hungarian <- udpipe_ annotate(m_hungarian, x = dat_udhrcorpus[[8]]) %>% as.data.frame()
## inspect

head (dat_inaug_corpus_sent_hungarian)

### 5.5.3.5.3.7 Hungarian

## doc_id paragraph_id sentence_id

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr></table>

##

## 1 Az Emberi Jogok Egyetemes Nyilatkozata Bevezető Tekintettel arra, hogy az emberis
## 2 Az Emberi Jogok Egyetemes Nyilatkozata Bevezető Tekintettel arra, hogy az emberis
## 3 Az Emberi Jogok Egyetemes Nyilatkozata Bevezető Tekintettel arra, hogy az emberis
## 4 Az Emberi Jogok Egyetemes Nyilatkozata Bevezető Tekintettel arra, hogy az emberis
## 5 Az Emberi Jogok Egyetemes Nyilatkozata Bevezető Tekintettel arra, hogy az emberis
## 6 Az Emberi Jogok Egyetemes Nyilatkozata Bevezető Tekintettel arra, hogy az emberis

<table><thead><tr><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th></tr></thead><tbody><tr><td>## 1</td><td>1</td><td>Az</td><td>a</td><td>DET &lt;NA&gt;</td></tr><tr><td>## 2</td><td>2</td><td>Emberi</td><td>emberi</td><td>PROPN &lt;NA&gt;</td></tr><tr><td>## 3</td><td>3</td><td>Jogok</td><td>Jogok</td><td>PROPN &lt;NA&gt;</td></tr><tr><td>## 4</td><td>4</td><td>Egyetemes</td><td>Egyetemes</td><td>PROPN &lt;NA&gt;</td></tr><tr><td>## 5</td><td>5</td><td>Nyilatkozata</td><td>nyilatkozata</td><td>PROPN &lt;NA&gt;</td></tr><tr><td>## 6</td><td>6</td><td>Bevezető</td><td>Bevezető</td><td>PROPN &lt;NA&gt;</td></tr></tbody></table>

##

feats

## 1 Case=Nom|Number=Sing|Number[psed]=None|Number[psor]=None|Person[psor]=None
## 2 Case=Nom|Number=Sing|Number[psed]=None|Number[psor]=None|Person[psor]=None
## 3 Case=Nom|Number=Sing|Number[psed]=None|Number[psor]=None|Person[psor]=None
## 4 Case=Nom|Number=Sing|Number[psed]=None|Number[psor]=None|Person[psor]=None
## 5 Case=Nom|Number=Sing|Number[psed]=None|Number[psor]=None|Person[psor]=None
## 6 Case=Nom|Number=Sing|Number[psed]=None|Number[psor]=None|Person[psor]=None
## head_token_id dep_rel deps misc
## 1 2 det &lt;NA&gt; &lt;NA&gt;
## 2 8 nmod:att &lt;NA&gt; &lt;NA&gt;
## 3 2 flat:name &lt;NA&gt; &lt;NA&gt;
## 4 2 flat:name &lt;NA&gt; &lt;NA&gt;
## 5 2 flat:name &lt;NA&gt; SpacesAfter=\\n## 6 2 flat:name &lt;NA&gt; &lt;NA&gt;

dat_inaug_corpus_sent_hungarian_short <- dat_inaug_corpus_sent_hungarian[1:36,]

dat_inaug_corpus_sent_dplot_hungarian <- textplot_dependencyparser (dat_inaug_corpus_sent_hungarian)

## show plot

dat_inaug_corpus_sent_dplot_hungarian
Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_151.png" /></figure>

### parse text

```
dat_inaug_corpus_sent.irish <- udpipe_ annotate(m_irish, x = dat_udhr_corpus[[10]]) %>
as.data.frame()
```

### inspect

```
head.dat_inaug_corpus_sent.irish
```

5.5.3.5.3.8 Irish

## doc_id paragraph_id sentence_id

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr></table>

##

```
## 1 DEARBHÚ UILE-CHOITEANN CEARTA AN DUINE [Preamble] De Bhrí gurb é aithint dínte dúchais agus
## 2 DEARBHÚ UILE-CHOITEANN CEARTA AN DUINE [Preamble] De Bhrí gurb é aithint dínte dúchais agus
## 3 DEARBHÚ UILE-CHOITEANN CEARTA AN DUINE [Preamble] De Bhrí gurb é aithint dínte dúchais agus
## 4 DEARBHÚ UILE-CHOITEANN CEARTA AN DUINE [Preamble] De Bhrí gurb é aithint dínte dúchais agus
## 5 DEARBHÚ UILE-CHOITEANN CEARTA AN DUINE [Preamble] De Bhrí gurb é aithint dínte dúchais agus
## 6 DEARBHÚ UILE-CHOITEANN CEARTA AN DUINE [Preamble] De Bhrí gurb é aithint dínte dúchais agus
```

<table><thead><tr><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th></tr></thead><tbody><tr><td>## 1</td><td>1</td><td>DEARBHÚ</td><td>dearbh</td><td>NOUN</td></tr><tr><td>## 2</td><td>2</td><td>UILE-CHOITEANN</td><td>Uile-choit</td><td>VERB</td></tr><tr><td>## 3</td><td>3</td><td>CEARTA</td><td>ceart</td><td>PROPN</td></tr></tbody></table>
```
dat_inaug_corpus_sent_japanese <- udpipe_ annotate(m_japanese, x = dat_udhr_corpus_japanese_short)
as.data.frame()
```

```
## inspect
```

```
head (dat_inaug_corpus_sent_japanese)
```

### 5.5.3.5.3.9 Japanese

```
## doc_id paragraph_id sentence_id
```

<table><tr><td>## 1</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 2</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 3</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 4</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 5</td><td>doc1</td><td>1</td><td>1</td></tr><tr><td>## 6</td><td>doc1</td><td>1</td><td>1</td></tr></table>

```
##
```

```
## 1
```

```
## 2
```

```
## 3
```

```
## 4
```

```
## 5
```

```
## 6
```

<table><thead><tr><th>##</th><th>token_id</th><th>token</th><th>lemma</th><th>upos</th><th>xpos</th><th>feats</th><th>head_token_id</th><th>dep_rel</th><th>deps</th></tr></thead><tbody><tr><td>## 1</td><td>1</td><td>1</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>2</td><td>compound</td><td>&lt;NA&gt;</td></tr><tr><td>## 2</td><td>2</td><td>2</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>4</td><td>nmod</td><td>&lt;NA&gt;</td></tr><tr><td>## 3</td><td>3</td><td>3</td><td>ADP</td><td>PN</td><td>&lt;NA&gt;</td><td></td><td>2</td><td>case</td><td>&lt;NA&gt;</td></tr><tr><td>## 4</td><td>4</td><td>4</td><td>NOUN</td><td>NR</td><td>&lt;NA&gt;</td><td></td><td>7</td><td>nmod</td><td>&lt;NA&gt;</td></tr><tr><td>## 5</td><td>5</td><td>5</td><td>ADP</td><td>PN</td><td>&lt;NA&gt;</td><td></td><td>4</td><td>case</td><td>&lt;NA&gt;</td></tr><tr><td>## 6</td><td>6</td><td>6</td><td>NOUN</td><td>NN</td><td>&lt;NA&gt;</td><td></td><td>7</td><td>compound</td><td>&lt;NA&gt;</td></tr></tbody></table>

```
##
```

```
## 1 SpaceAfter=No
```

```
## 2 SpaceAfter=No
```

```
## 3 SpaceAfter=No
```

```
## 4 SpaceAfter=No
```

```
## 5 SpaceAfter=No
```

```
## 6 SpaceAfter=No
```

```
dat_inaug_corpus_sent_dplot_japanese <- textplot_dependencyparser (dat_inaug_corpus_sent_japanese,
## show plot
```

```
dat_inaug_corpus_sent_dplot_japanese
```
as.data.frame()
## inspect
head.dat_inaug_corpus_sent_vietnamese)

5.5.3.5.3.11 Vietnamese

## doc_id paragraph_id sentence_id
## 1 doc1 1 1
## 2 doc1 1 1
## 3 doc1 1 1
## 4 doc1 1 1
## 5 doc1 1 1
## 6 doc1 1 1
##
## 1 7X\\zQ QJ{Q WR\177Q WK\u009b JL±L Y\u0098 QKyQ TX\\\u0098Q F´D /LzQ +²S 4X¬F w}²F L ~@QJ /LzQ +²S 4X¬F WK{QJ TXD Y\177 F{QJ E~ WKHR 1JK£ TX\\
## 2 7X\\zQ QJ{Q WR\177Q WK\u009b JL±L Y\u0098 QKyQ TX\\\u0098Q F´D /LzQ +²S 4X¬F w}²F L ~@QJ /LzQ +²S 4X¬F WK{QJ TXD Y\177 F{QJ E~ WKHR 1JK£ TX\\
## 3 7X\\zQ QJ{Q WR\177Q WK\u009b JL±L Y\u0098 QKyQ TX\\\u0098Q F´D /LzQ +²S 4X¬F w}²F L ~@QJ /LzQ +²S 4X¬F WK{QJ TXD Y\177 F{QJ E~ WKHR 1JK£ TX\\
## 4 7X\\zQ QJ{Q WR\177Q WK\u009b JL±L Y\u0098 QKyQ TX\\\u0098Q F´D /LzQ +²S 4X¬F w}²F L ~@QJ /LzQ +²S 4X¬F WK{QJ TXD Y\177 F{QJ E~ WKHR 1JK£ TX\\
## 5 7X\\zQ QJ{Q WR\177Q WK\u009b JL±L Y\u0098 QKyQ TX\\\u0098Q F´D /LzQ +²S 4X¬F w}²F L ~@QJ /LzQ +²S 4X¬F WK{QJ TXD Y\177 F{QJ E~ WKHR 1JK£ TX\\
## 6 7X\\zQ QJ{Q WR\177Q WK\u009b JL±L Y\u0098 QKyQ TX\\\u0098Q F´D /LzQ +²S 4X¬F w}²F L ~@QJ /LzQ +²S 4X¬F WK{QJ TXD Y\177 F{QJ E~ WKHR 1JK£ TX\\
## token_id token lemma upos xpos feats
## 1 1 7X 7X NUM M NumType=Card
## 2 2 \\zQ QJ{Q WR\177Q \\zQ QJ{Q WR\177Q NOUN N <NA>
## 3 3 WK\u009b WK\u009b NOUN N <NA>
## 4 4 JL±L JL±L DET L <NA>
## 5 5 Y\u0098 QKyQ Y\u0098 QKyQ NOUN N <NA>
## 6 6 TX TX NOUN Ny <NA>
## head_token_id dep_rel deps misc
## 1 2 nummod <NA> SpaceAfter=No
## 2 5 nsubj <NA> <NA>
## 3 2 compound <NA> <NA>
## 4 5 det <NA> <NA>
## 5 0 root <NA> <NA>
## 6 5 compound <NA> SpaceAfter=No

dat_inaug_corpus_sent_vietnamese_short <- dat_inaug_corpus_sent_vietnamese[1:26,]

dat_inaug_corpus_sent_dplot_vietnamese <- textplot_dependencyparser.dat_inaug_corpus_sent_dplot_vietnamese

## show plot

dat_inaug_corpus_sent_dplot_vietnamese
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font
## width unknown for character 0x7f in encoding cp1252
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font
## width unknown for character 0x7f in encoding cp1252
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font
## width unknown for character 0x7f in encoding cp1252

## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## font width unknown for character 0x7f in encoding cp1252

## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font
## width unknown for character 0x7f in encoding cp1252
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font
## width unknown for character 0x7f in encoding cp1252
## Warning in grid.Call(C_textBounds, as.graphicsAnnot(x$label), x$x, x$y, : font
## width unknown for character 0x7f in encoding cp1252

## Warning in grid.Call.graphics(C_text, as.graphicsAnnot(x$label), x$x, x$y, :
## font width unknown for character 0x7f in encoding cp1252

Dependency Parser

tokenisation, parts of speech tagging & dependency relations

<figure><img src="image_152.png" /></figure>

5.5.3.6 Feature co-occurrence matrix (FCM)

Feature co-occurrence matrix (FCM) records the number of co-occurrences of
tokens

5.5.3.6.1 Computing number of co-occurrences
```
dat_udhr_corpus_english <- dat_udhr_corpus[[4]]
dat_udhr_corpus_english DFmat <- dfm Mod tokens Mod dat_udhr_corpus_english, remove_punct = TRUE
```
```
dat_udhr_corpus_english DFmat_trim <- dfm_trim Mod dat_udhr_corpus_english DFmat, min_termlen = 1
```

```
topfeatures dat_udhr_corpus_english <- topfeatures Mod dat_udhr_corpus_english DFmat_trim
```
```
topfeatures dat_udhr_corpus_english
```

### 5.5.3.6.1.1 English

<table><tr><td>##</td><td>the</td><td>and</td><td>of</td><td>to</td><td>in</td><td>right</td><td>be</td><td>article</td></tr><tr><td>##</td><td>120</td><td>106</td><td>90</td><td>84</td><td>43</td><td>33</td><td>31</td><td>30</td></tr><tr><td>##</td><td>everyone</td><td>or</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>##</td><td>30</td><td>30</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></table>

```
nfeat Mod dat_udhr_corpus_english DFmat_trim
```

```
## [1] 30
```

```
dat_udhr_corpus_french <- dat_udhr_corpus[[5]]
dat_udhr_corpus_french DFmat <- dfm Mod tokens Mod dat_udhr_corpus_french, remove_punct = TRUE
```
```
dat_udhr_corpus_french DFmat_trim <- dfm_trim Mod dat_udhr_corpus_french DFmat, min_termlen = 1
```

```
topfeatures dat_udhr_corpus_french <- topfeatures Mod dat_udhr_corpus_french DFmat_trim
```
```
topfeatures dat_udhr_corpus_french
```

### 5.5.3.6.1.2 French

<table><tr><td>##</td><td>de</td><td>et</td><td>la</td><td>à</td><td>des</td><td>droit</td><td>le</td><td>a</td><td>les</td><td>article</td></tr><tr><td>##</td><td>134</td><td>90</td><td>79</td><td>61</td><td>42</td><td>41</td><td>36</td><td>32</td><td>31</td><td>30</td></tr></table>

```
nfeat Mod dat_udhr_corpus_french DFmat_trim
```

```
## [1] 34
```

### 5.5.3.6.2 Features co-occurrences

```
dat_udhr_corpus_english fcmat <- fcm Mod dat_udhr_corpus_english DFmat_trim
```
```
dat_udhr_corpus_english fcmat
```

#### 5.5.3.6.2.1 English

```
## Feature co-occurrence matrix of: 30 by 30 features.
## features
## features of human rights the and equal all is freedom in
## of 4005 1170 1980 10800 9540 990 1080 1170 990 3870
```
## 5.6. TWITTER DATA

1543

```
## human      0   78   286  1560  1378  143  156  169   143  559
## rights    0   0   231  2640  2332  242  264  286   242  946
## the       0   0     0  7140 12720 1320 1440 1560  1320 5160
## and       0   0     0     0  5565 1166 1272 1378  1166 4558
## equal     0   0     0     0     0   55  132  143   121  473
## all       0   0     0     0     0     0   66  156   132  516
## is        0   0     0     0     0     0     0   78   143  559
## freedom   0   0     0     0     0     0     0     0   55  473
## in        0   0     0     0     0     0     0     0     0  903
## [ reached max_nfeat ... 20 more features, reached max_nfeat ... 20 more features ]
```

```
dat_udhr_corpus_french_fcmat <- fcm.dat_udhr_corpus_french DFmat_trim, context = "document")
dat_udhr_corpus_french_fcmat
```

### 5.5.3.6.2.2 French

```
## Feature co-occurrence matrix of: 34 by 34 features.
```

```
## features des droits de l'homme que la à tous les et
## des      861   924  5628    420   840  3318  2562  588 1302  3780
## droits    0   231  2948    220   440  1738  1342  308  682  1980
## de       0     0  8911   1340  2680 10586  8174 1876 4154 12060
## l'homme  0     0     0     45   200   790   610  140  310   900
## que      0     0     0     0   190  1580  1220  280  620  1800
## la       0     0     0     0     0  3081  4819 1106 2449  7110
## à        0     0     0     0     0   0  1830  854 1891  5490
## tous     0     0     0     0     0     0   0   91  434  1260
## les      0     0     0     0     0     0     0   0   465  2790
## et       0     0     0     0     0     0     0     0   0  4005
## [ reached max_nfeat ... 24 more features, reached max_nfeat ... 24 more features ]
```

You can test it on the other languages

## 5.6 Twitter data

We the twitter.json data accessed from here. This is a JSON file (.json) downloaded from the Twitter stream API.

### 5.6.1 Importing data

```
dat_twitter <- readtext("data/twitter.json", source = "twitter")
```
### 5.6.1.1 Create a corpus

```
dat.twitter_corpus <- corpus.dat.twitter)
print.dat.twitter_corpus)
```

```
## Corpus consisting of 7,504 documents and 42 docvars.
## twitter.json.1 :
## "@EFC_Jayy UKIP"
##
## twitter.json.2 :
## "RT @Corbynator2:@jeremycorbyn Reaction from people at the Wa..."
##
## twitter.json.3 :
## "RT @ryvr: Stephen Hawking, the world's smartest man, backs J..."
##
## twitter.json.4 :
## "RT @TheGreenParty: How you cast your vote will shape the fut..."
##
## twitter.json.5 :
## "RT @UKLabour: #VoteLabour today for a fairer Britain. Find o..."
##
## twitter.json.6 :
## "RT @WestmonsterUK: POLLS: FINAL DAY AVERAGE CON 43.8% LAB 35..."
##
## [ reached max_ndoc ... 7,498 more documents ]
```

### 5.6.1.2 Summary

```
summary.dat.twitter_corpus, 10)
```

```
## Corpus consisting of 7504 documents, showing 10 documents:
##
## Text Types Tokens Sentences retweet_count favorite_count favorited
## twitter.json.1      2      2          1          0          0  FALSE
## twitter.json.2     24     30         3         90        108  FALSE
## twitter.json.3     15     16         1         78        104  FALSE
## twitter.json.4     20     23         3        244        218  FALSE
## twitter.json.5     23     23         2        1896       2217  FALSE
## twitter.json.6     18     22         1         55         52  FALSE
## twitter.json.7     28     34         2         65         73  FALSE
## twitter.json.8     23     23         1         30          9  FALSE
## twitter.json.9     23     23         2        1896       2217  FALSE
## twitter.json.10    26     27         2          0          0  FALSE
## truncated id_str in reply to screen name
## FALSE 872596537142116352      EFC_Jayy
## FALSE 872596536869363712      <NA>
```
<table><tr><td>London</td><td>241664322</td></tr><tr><td>AKast UDI RN PRI Evopoli FKast</td><td>811768906301968384</td></tr><tr><td>Liverpool</td><td>2157745426</td></tr><tr><td>Social Affairs Editor: Scisco</td><td>586255942</td></tr><tr><td>&lt;NA&gt;</td><td>4551111388</td></tr><tr><td>Glasgow, Scotland</td><td>704227420</td></tr></table>

## Infovore, atheist, post-, p/t nihilist, lifelong radiophile, aspiring cultural thinker

## REPUBLICA LIBERTARIA NO BUROCRACIA ESTATISTA MONOPOLIA
## LFC. Lets get Labour back in power! Its not left vs right

## Public interest issues, policy, human rights & social sciences. Researcher and
## artist. sick and tired of not being allowed an opinion in a so called free country

<table><thead><tr><th>geo_enabled</th><th>user_created_at</th><th> statuses_count</th><th>followers_count</th></tr></thead><tbody><tr><td>FALSE Mon May 15 01:30:11 +0000 2017</td><td>2930</td><td>367</td></tr><tr><td>FALSE Tue Jun 08 05:05:23 +0000 2010</td><td>10223</td><td>845</td></tr><tr><td>FALSE Tue Mar 29 01:59:32 +0000 2011</td><td>20934</td><td>761</td></tr><tr><td>FALSE Sat Jan 28 22:41:07 +0000 2012</td><td>13603</td><td>321</td></tr><tr><td>TRUE Sat Jan 22 20:53:11 +0000 2011</td><td>13179</td><td>386</td></tr><tr><td>FALSE Thu Dec 22 03:02:43 +0000 2016</td><td>52384</td><td>792</td></tr><tr><td>FALSE Mon Oct 28 21:16:49 +0000 2013</td><td>8572</td><td>361</td></tr><tr><td>FALSE Mon May 21 04:46:43 +0000 2012</td><td>33759</td><td>4475</td></tr><tr><td>TRUE Sun Dec 20 23:27:00 +0000 2015</td><td>171</td><td>21</td></tr><tr><td>TRUE Tue Oct 08 00:09:09 +0000 2013</td><td>21568</td><td>837</td></tr></tbody></table>

<table><thead><tr><th>favours_count</th><th>protected</th><th>user_url</th></tr></thead><tbody><tr><td>1260</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>9813</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>14733</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>5421</td><td>FALSE</td><td>http://www.instagram.com/stuhornett/</td></tr><tr><td>5219</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>6</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>11592</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>18368</td><td>FALSE</td><td>https://kittysjones.wordpress.com/</td></tr><tr><td>120</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>12177</td><td>FALSE</td><td>&lt;NA&gt;</td></tr><tr><td>name</td><td>time_zone</td><td>userlang</td><td>utc_offset</td><td>friends_count</td><td>screen_name</td></tr><tr><td>&lt;NA&gt;</td><td>en</td><td>NA</td><td>304</td><td></td><td>CoysJoji</td></tr><tr><td>Yara-ma-yha-who</td><td>London</td><td>en</td><td>3600</td><td>439</td><td>Unkle_Ken</td></tr><tr><td>Openly classist</td><td>London</td><td>en</td><td>3600</td><td>2761</td><td>OpenlyClassist</td></tr><tr><td>Stu</td><td>Casablanca</td><td>en</td><td>0</td><td>767</td><td>StuHornett</td></tr><tr><td>Ousama</td><td>Dublin</td><td>en</td><td>3600</td><td>257</td><td>ousamaZ18</td></tr><tr><td>Capitán Tatán</td><td>&lt;NA&gt;</td><td>es</td><td>NA</td><td>353</td><td>omentando</td></tr></tbody></table>
## https://t.co/953WYc2p0c

## https://t.co/qYdu0hUeey

### 5.6.1.3 Accessing parts of corpus

```
dat.twitter_corpus[,1]
```

## Corpus consisting of 7,504 documents and 42 docvars.

```
## twitter.json.1 :
```

## "@EFC_Jayy UKIP"

##

## twitter.json.2 :
```

## "RT @Corbynator2:@jeremycorbyn Reaction from people at the Wa..."

##

## twitter.json.3 :
```

## "RT @ryvr: Stephen Hawking, the world's smartest man, backs J..."

##

## twitter.json.4 :
```

## "RT @TheGreenParty: How you cast your vote will shape the fut..."

##

## twitter.json.5 :
```

## "RT @UKLabour: #VoteLabour today for a fairer Britain. Find o..."

##

## twitter.json.6 :
```

## "RT @WestmonsterUK: POLLS: FINAL DAY AVERAGE CON 43.8% LAB 35..."

##

## [ reached max_ndoc ... 7,498 more documents ]

### 5.6.1.4 Document-level information

```
head(docvars.datالفیئرCorpus))
```

## retweet_count favorite_count favorited truncated id_str

## 1 0 0 FALSE FALSE 872596537142116352

## 2 90 108 FALSE FALSE 872596536869363712

## 3 78 104 FALSE FALSE 872596537444093952

## 4 244 218 FALSE FALSE 872596538492637185

## 5 1896 2217 FALSE FALSE 872596538480103425

## 6 55 52 FALSE FALSE 872596538815651841

## in reply to screen_name

## 1 EFC_Jayy

## 2 <NA>

## 3 <NA>

## 4 <NA>

## 5 <NA>

## 6 <NA>
## 4 5421 FALSE http://www.instagram.com/stuhornett/
## 5 5219 FALSE <NA>
## 6 6 FALSE <NA>
## name time_zone userlang utc_offset friends_count screen_name
## 1 <NA> en NA 304 CoysJoji
## 2 Yara-ma-yha-who London en 3600 439 Unkle_Ken
## 3 Openly classist London en 3600 2761 OpenlyClassist
## 4 Stu Casablanca en 0 767 StuHornett
## 5 Ousama Dublin en 3600 257 ousamaZ18
## 6 Capitán Tatán <NA> es NA 353 omentando
## country_code country place_type full_name place_name place_id place_lat
## 1 <NA> <NA> NA <NA> <NA> <NA> NaN
## 2 <NA> <NA> NA <NA> <NA> <NA> NaN
## 3 <NA> <NA> NA <NA> <NA> <NA> NaN
## 4 <NA> <NA> NA <NA> <NA> <NA> NaN
## 5 <NA> <NA> NA <NA> <NA> <NA> NaN
## 6 <NA> <NA> NA <NA> <NA> <NA> NaN
## place_lon lat lon
## 1 NaN NA NA
## 2 NaN NA NA
## 3 NaN NA NA
## 4 NaN NA NA
## 5 NaN NA NA
## 6 NaN NA NA
##
## 1
## 2
## 3 http://www.independent.co.uk/news/science/stephen-hawking-jeremy-corbyn-labour-the
## 4
## 5
## 6
## url
## 1 <NA>
## 2
## 3 https://t.co/2kl3ayLd44
## 4 <NA>
## 5 https://t.co/953WYc2p0c
## 6 https://t.co/6zBHeHWuru

5.6.1.5 Unique variable names (for volume)

unique(docvars.dat_twitter_corpus, field = "lang"))

## [1] "en" "tl" "und" "fr" "ja" "de" "sv" "hu" "ht" "lt" "cy" "ar"
## [13] "pt" "in" "fi" "tr" "es" "hi" "fa" "it" "pl"