# R Programming/Text Processing

R Programming
* R Basics
** Introduction
** Sample Session
** Manage your workspace
** Settings and Installation
** Documentation
** Control Structures
** Working with functions
** Debugging
** Using C or Fortran
** Utilities
** Estimation utilities
** Packages
* Data Management
** Data types
** Working with data frames
** Importing and exporting data
** Text Processing
** Times and Dates
* Graphics
** Grammar of graphics
* Publication quality output
* Descriptive Statistics
* Mathematics
** Optimization
** Probability Distributions
** Random Number Generation
* Statistical Core Methods
** Maximum Likelihood
** Method of Moments
** Bayesian Methods
** Bootstrap
** Multiple Imputation
** Nonparametric Methods
* Regression Models
** Linear Models
** Quantile Regression
** Binomial Models
** Multinomial Models
** Tobit And Selection Models
** Count Data Models
** Duration Analysis
* Time Series
* Factor Analysis
* Classification
** Ordination
** Clustering
* Network Analysis
* High Performance Computing
** Profiling R code
** Parallel computing with R
* Appendix
** Sources
** Index
This page includes all the material you need to deal with strings in R. The section on regular expressions may be useful to understand the rest of the page, even if it is not necessary if you only need to perform some simple tasks.
This page may be useful to :
* perform statistical text analysis.
* collect data from an unformatted text file.
* deal with character variables.
In this page, we learn how to read a text file and how to use R functions for characters. There are two kind of function for characters, simple functions and regular expressions. Many functions are part of the standard R base package.
However, their name and their syntax is not intuitive to all users. Hadley Wickham has developed the stringr package which defines functions with similar behaviour but their names are easier to retain and their syntax much more systematic.
* Keywords : text mining, natural language processing
* See CRAN Task view on Natural Language Processing
* See also the following packages tm, tau, languageR, scrapeR.

## Reading and writing text files

R can read any text file using readLines or scan. It is possible to specify the encoding of the imported text file with readLines. The entire contents of the text file can be read into an R object (e.g., a character vector). scan is more flexible. The kind of data expected can be specified in the second argument (e.g., character(0) for a string).
We can write the content of an R object into a text file using cat or writeLines. By default cat concatenates vectors when writing to the text file. You can change it by adding options sep="\n" or fill=TRUE. The default encoding depends on your computer.
Before reading a text file, you can look at its properties. nlines (parser package) and countLines (R.utils package) count the number of lines in the file. count.chars (parser package) counts the number of bytes and characters in each line of a file. You can also display a text file using file.show.

## Character encoding

R provides functions to deal with various set of encoding schemes. This is useful if you deal with text file which have been created with another operating system and especially if the language is not English and has many accents and specific characters. For instance, the standard encoding scheme in Linux is "UTF-8" whereas the standard encoding scheme in Windows is "Latin1". The Encoding functions returns the encoding of a string. iconv is similar to the unix command iconv and converts the encoding.
* iconvlist gives the list of available encoding scheme on your computer.
* readLines, scan and file.show have also an encoding option.
* is.utf8 (tau) tests if the encoding is "utf8".
* is.locale (tau) tests if encoding is the same as the default encoding on your computer.
* translate (tau) translates the encoding into the current locale.
* fromUTF8 (descr) is less general than iconv.
* utf8ToInt (base)

### Example

The following example was run under Windows. Thus, the default encoding is "latin1".

## Regular Expressions

A regular expression is a specific pattern in a set of strings. For instance, one could have the following pattern : 2 digits, 2 letters and 4 digits. R provides powerful functions to deal with regular expressions. Two types of regular expressions are used in R
* extended regular expressions, used by ‘perl = FALSE’ (the default),
* Perl-like regular expressions used by ‘perl = TRUE’.
There is a also an option called ‘fixed = TRUE’ which can be considered as a literal regular expression. fixed (stringr) is equivalent to fixed=TRUE in the standard regex functions. These functions are by default case sensitive. This can be changed by specifying the option ignore.case = TRUE.
If you are not a specialist in regular expression you may find the glob2rx useful. This function suggests some regular expression for a specific ("glob" or "wildcard") pattern :

### Functions which use regular expressions in R

* sub, gsub, str_replace (stringr) make some substitutions in a string.
* grep, str_extract (stringr) extract some value
* grepl, str_detect (stringr) detect the presence of a pattern.
* see also splitByPattern (R.utils)
* See also gsubfn in the gsubfn package.

### Extended regular expressions (The default)

* "." stands for any character.
* "[ABC]" means A,B or C.
* "[A-Z]" means any upper letter between A and Z.
* "[0-9]" means any digit between 0 and 9.
Here is the list of metacharacters ‘$ * + . ? [ ] ^ { } | ( ) \’. If you need to use one of those characters, precede them with a doubled backslash.
Here are some classes of regular expressions : For numbers :
* ‘[:digit:]’ Digits: ‘0 1 2 3 4 5 6 7 8 9’.
For letters :
* ‘[:alpha:]’ Alphabetic characters: ‘[:lower:]’ and ‘[:upper:]’.
* ‘[:upper:]’ Upper-case letters.
* ‘[:lower:]’ Lower-case letters.
Note that the set of alphabetic characters includes accents such as é è ê which are very common in some languages like French. Therefore, it is more general than "[A-Za-z]" which does not include letters with accent.
For other characters :
* ‘[:punct:]’ Punctuation characters: ‘! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~’.
* ‘[:space:]’ Space characters: tab, newline, vertical tab, form feed, carriage return, and space.
* ‘[:blank:]’ Blank characters: space and tab.
* ‘[:cntrl:]’ Control characters.
For combination of other classes :
* [:alnum:] Alphanumeric characters: ‘[:alpha:]’ and ‘[:digit:]’.
* ‘[:graph:]’ Graphical characters: ‘[:alnum:]’ and ‘[:punct:]’.
* ‘[:print:]’ Printable characters: ‘[:alnum:]’, ‘[:punct:]’ and space.
* ‘[:xdigit:]’ Hexadecimal digits: ‘0 1 2 3 4 5 6 7 8 9 A B C D E F a b c d e f’.
You can quantify the number of repetition by adding after the regular expression the following characters :
* ‘?’ The preceding item is optional and will be matched at most once.
* ‘*’ The preceding item will be matched zero or more times.
* ‘+’ The preceding item will be matched one or more times.
* ‘{n}’ The preceding item is matched exactly ‘n’ times.
* ‘{n,}’ The preceding item is matched ‘n’ or more times.
* ‘{n,m}’ The preceding item is matched at least ‘n’ times, but not more than ‘m’ times.
* ^ to force the regular expression to be at the beginning of the string
* $ to force the regular expression to be at the end of the string
If you want to know more, have a look at the 2 following help files :

### Perl-like regular expressions

It is also possible to use "perl-like" regular expressions. You just need to use the option perl=TRUE.

#### Examples

If you want to remove space characters in a string, you can use the \\s Perl macro.

#### See also

* Perl Programming/Regular Expressions

## Concatenating strings

* paste concatenates strings.
* str_c (stringr) does a similar job.
* cat prints and concatenates strings.

### Examples
## Splitting a string

* strsplit : Split the elements of a character vector ‘x’ into substrings according to the matches to substring ‘split’ within them.
* See also str_split (stringr).
* tokenize (tau) split a string into tokens.

## Counting the number of characters in a string

* nchar gives the length of a string. Note that that for non-ASCII encodings, there is more one way to measure such a length.
* See also str_length (stringr)

## Detecting the presence of a substring
### Detecting a pattern in a string ?

* grepl returns a logical expression (TRUE or FALSE).
* str_detect (stringr) does a similar job.
The 1st one is true and the second one is false since there is only one digit in the first number.

### Counting the occurrence of each pattern in a string ?

* textcnt (tau) counts the occurrence of each pattern or each term in a text.

## Extracting the position of a substring or a pattern in a string
### Extracting the position of a substring ?

* cpos (cwhmisc) returns the position of a substring in a string.
* substring.location (cwhmisc) does the same job but returns the first and the last position.

### Extracting the position of a pattern in a string ?

* regexpr returns the position of the regular expression. str_locate (stringr) does the same job. gregexpr is similar to regexpr but the starting position of every match is returned. str_locate_all (stringr) does the same job.

## Extracting a substring from a string
### Extracting a fixed width substring ?

* substr takes a sub string.
* str_sub (stringr) is similar.

### Extracting the first word in a string ?

* first.word First Word in a String or Expression in the Hmisc package

### Extracting a pattern in a string ?

* grep returns the value of the regular expression if value=T and its position if value=F.
* str_extract, str_extract_all, str_match, str_match_all (stringr) and m (caroline package) are similar to grep. str_extract and str_extract_all return a vector. str_match and str_match_all return a matrix and m a dataframe.
* Named capture regular expressions can be used to define column names in the regular expression (this also serves to document the regular expression). Install the namedCapture package via devtools::install_github("tdhock/namedCapture") to use str_match_all_named. It uses the base function gregexpr(perl=TRUE) to parse a Perl-Compatible Regular Expression, and returns a list of match matrices with column names:

## Making some substitution inside a string
### Substituting a pattern in a string

* sub makes a substitution.
* gsub is similar to sub but replace all occurrences of the pattern whereas sub only replaces the first occurrence.
* str_replace (stringr) is similar to sub, str_replace_all (stringr) is similar to gsub.
In the following example, we have a French date. The regular pattern is the following : 2 digits, a blank, some letters, a blank, 4 digits. We capture the 2 digits with the [[:digit:]]{2} expression, the letters with [[:alpha:]]+ and the 4 digits with [[:digit:]]{4}. Each of these three substrings is surrounded with parenthesis. The first substring is stored in "\\1", the second one in "\\2" and the 3rd one in "\\3".
In the following example, we compare the outcome of sub and gsub. The first one removes the first space whereas the second one removes all spaces in the text.

### Substituting characters in a string ?

* chartr substitutes characters in an expression. It stands for "character translation".
* replacechar (cwhmisc) does the same job ...
* as well as str_replace_all (stringr).

## Converting letters to lower or upper-case

* tolower converts upper-case characters to lower-case.
* toupper converts lower-case characters to upper-case.
* capitalize (Hmisc) capitalize the first letter of a string
* See also cap, capitalize, lower, lowerize and CapLeading in the cwhmisc package.

## Filling a string with some character

* padding (cwhmisc) fills a string with some characters to fit a given length. See also str_pad (stringr).
Note that str_pad is very slow. For instance for a vector of length 10,000, we have a very long computing time. paddingdoes not seem to handle character vectors but the best solution may be to use the sapply and padding functions together.

## Removing leading and trailing spaces

* trimws (memisc package) trim leading and trailing white spaces.
* trim (gdata package) does the same job.
* See also str_trim (stringr)

## Comparing two strings
### Assessing if they are identical

* == returns TRUE if both strings are the same and false otherwise.

### Computing distance between strings

Few packages implement the Levenshtein distance between two strings:
* adist in base package utils
* stringMatch in MiscPsycho
* stringdist in stringdist
* levenshteinDist in RecordLinkage
A benchmark comparing the speed of levenshteinDist and stringdist is available here: .

#### Example with MiscPsycho

stringMatch (MiscPsycho) computes If normalize="YES" the levenshtein distance is divided by the maximum length of each string.

#### Approximate matching

agrep search for approximate matches using the Levenshtein distance.
* If 'value = TRUE', this returns the value of the string
* If 'value = FALSE' this returns the position of the string
* max returns the maximal levenshtein distance.

## Miscellaneous

* deparse : Turn unevaluated expressions into character strings.
* char.expand (base) expands a string with respect to a target.
* pmatch (base) and charmatch (base) seek matches for the elements of their first argument among those of their second.
* make.unique makes a character string unique. This is useful if you want to use a string as an identifier in your data.

## References

* ↑ Hadley Wickham "stringr: modern, consistent string processing" The R Journal, December 2010, Vol 2/2, http://journal.r-project.org/archive/2010-2/RJournal_2010-2_Wickham.pdf
* ↑ http://cran.r-project.org/web/views/NaturalLanguageProcessing.html
* ↑ In former versions (< 2.10) we had also basic regular expressions in R : extended regular expressions, used by extended = TRUE (the default), basic regular expressions, as used by extended = FALSE (obsolete in R 2.10). Since basic regular expressions (‘extended = FALSE’) are now obsolete, the extended option is obsolete in version 2.11.