# NooJ

NooJ is a sophisticated linguistic development environment and corpus processing software created by Max Silberztein. It enables linguists to construct and analyze the four classes of the Chomsky-Schützenberger hierarchy of generative grammars: Regular Grammars, Context-Free Grammars, Context-Sensitive Grammars as well as Unrestricted Grammars. Users can create these grammars using either a text editor for writing regular expressions or a graphical editor for visual representation.
NooJ allows linguists to develop orthographical and morphological grammars, dictionaries of simple words, of compound words as well as discontinuous expressions, local syntactic grammars (such as Named Entities Recognizers), structural syntactic grammars (that produce syntactic trees) as well as Zellig Harris‘ transformational grammars.
All NooJ parsers operate on Atomic Linguistic Units (ALUs) rather than traditional word forms (i.e. sequences of letters between two space characters), allowing for nuanced parsing of phrases like “can not” as well as their contracted forms, such as “cannot” or “can’t”. This feature facilitates the creation of relatively simple syntactic grammars, even for agglutinative languages.
ALUs are represented by annotations stored in the Text Annotation Structure (or TAS) enabling NooJ parsers to add or remove annotations dynamically. A typical analysis in NooJ involves applying a series of elementary grammars in a cascading, bottom-up approach, progressing from spelling to semantics.

## History of NooJ

NooJ originated from the research conducted by Max Silberztein and the INTEX community, a group of linguists focused on the Lexicon-Grammar approach developed by Maurice Gross at the LADL (Laboratoire d’Automatique Documentaire et Linguistique). This approach posits that no grammar rule can be formulated without a precise definition of its domain of application, emphasizing the importance of context in linguistic analysis.
Since its inception, NooJ has been utilized as a powerful corpus processor across various fields, including: Linguistics (for analyzing language structures and patterns), History (for processing historical texts and documents), Psychology (in studies examining language use and cognition), in Literature studies (for textual analysis and interpretation), in sentiment analysis projects (to assess emotional tone in written content), data mining (for extracting useful information from large datasets), and even to analyze and interpret musical notation. Notably, NooJ played a significant role in the MARS 500 experiment, showcasing its versatility in handling complex data. Additionally, several software companies have leveraged NooJ to develop Information Extraction and Information Retrieval systems, further demonstrating its impact on computational linguistics and related disciplines.

## Complexity and application

NooJ’s dictionaries are represented by finite-state transducers and can effectively handle various linguistic constructs, including simple words (e.g. 'table'), compound words (e.g. 'as a matter of fact') as well as discontinuous expressions such as phrasal verbs (e.g. 'to turn … off'), idiomatic expressions (e.g. 'to take the bull by the horns') as well as support verb/predicative noun associations (e.g. 'to take a nap').
NooJ enables linguists to create, edit, debug and maintain a wide number of grammars that fall within the four classes of generative grammars in the Chomsky-Schützenberger hierarchy: regular grammars, context-free grammars, context-sensitive grammars, and unrestricted grammars.
NooJ is capable of applying grammars to texts in linear time. For instance, many NooJ context-free grammars can be derecursived for efficiency. Context-sensitive grammars in NooJ consist of two components: one part is a context-free (or even regular) grammar applied efficiently to texts, while the second part includes a set of constraints applied to matching sequences, each executed in constant time.
Unrestricted grammars in NooJ are context-sensitive grammars that can incorporate variables and modify the input text. These grammars are typically used for transformational analysis and generation (see Zellig Harris). Moreover, several research teams have demonstrated that when combined with multilingual lexicons, NooJ can effectively perform Machine Translation.