# Solon: A Holistic Approach for Modelling, Managing and Mining Legal Sources
## 3 Solon in Depth
### Parsing Process  
The main steps of our approach are: (a) identify the structure of the legal documents, (b) identify legal documents metadata, and (c) validate produced files against the selected schema. As a pre-processing step, in our method, all legal documents are converted into plain text files. Notably, this step does have the downside of discarding valuable style and layout information found in the presentation document format (pdf/word). Several open source content analysis frameworks can efficiently and effectively perform this task, e.g., Apache Tika. Also, scanned text stored as an image in the supplied document can be adequately handled by an OCR process. While an optical character recognition or OCR system transforms a scanned image into text, it cannot offer 100% accuracy. Although the OCR process accuracy has improved notably over the past decades, as a result of more elaborated algorithms, it may still lose valuable information. As such, we do believe that manual content curation process is an importance step towards the completeness of the resulting legal knowledge base, especially in cases where there is limited tolerance for errors. Solon utilizes the Apache Tika framework for converting PDF files into plain text. Tesseract OCR can be seemingly integrated with Apache Tika as to handle scanned text stored in PDF images. Solon architecture and implementation provides two models of operation regarding scanned images: (a) having them stored as images and accompany the resulting XML file and (b) enriching the text file with content acquired from the OCR process. However, in the particular use case presented in this paper, Section 4, it was requested by the program owner that Solon operates without the OCR sub-process. Figure 9 provides a visual overview of our parsing process. The input document is firstly transformed into a token stream by the lexer. In the stream, tokens are associated with a regular expression that define the set of possible character sequences used to form the token. Afterwards, the parser, parses syntactic structure of the token stream creating an Abstract Syntax Tree (AST). AST is a tree data structure representing the abstract syntactic structure of the input stream, which is processed by a tree walker that generates the in-memory document model. Afterwards, a transformer formats the in-memory document model to the chosen legal schema (Section 2.2). Finally, upon successful completion of the semantic checker/validator, the output document is serialized. Figure 10 provides a visual overview of our parsing methodology. It follows a pipeline strategy, utilizing a top-down approach, that can be summarized into the following five steps:  
1. _Document Type Identification_. A first step in our methodology is to properly identify the type of legal document, since the type of the document defines both the document structure, available metadata and the internal semantic organization of the resulting document. Thus, identifying the type of legal document is a fundamental task to effectively model legal documents. This is achieved by scanning the beginning of the text for several predefined keywords that effectively distinguish document types. 2. _Structural Analysis_. Having identified the type of legal document, we distinguish legal blocks in the document (e.g., front matter, body, conclusions, annees). Structural Analysis differs between various document types as predefined in the corresponding grammar and implemented in the appropriate parser. 3. _Legal Blocks Isolation_. Legal blocks, previously identified in the structural analysis step, are iteratively broken down, into distinct elements, according to the corresponding grammar. Within this step, we try to identify and describe the structure of the legal documents; this includes identification and mark-up of each structural unit of the document (title, articles, sections, chapters, paragraphs, annees, etc.).