Abstract: Information distribution by electronic messages is a privileged means of transmission for many businesses and individuals, often under the form of plain-text tables. As their number grows, it becomes necessary to use an algorithm to extract text and numbers instead of a human. Usual methods are focused on regular expressions or on a strict structure in the data, but are not efficient when we have many variations, fuzzy structure or implicit labels. In this paper we introduce SC2T, a totally self-supervised model for constructing vector representations of tokens in semi-structured messages by using characters and context levels that address these issues. It can then be used for an unsupervised labeling of tokens, or be the basis for a semi-supervised information extraction system.

Introduction: Today most of business-related information is transmitted in an electronic form, such as emails. Therefore, converting these messages into an easily analyzable representation could open numerous business opportunities, as a lot of them are not used fully because of the difficulty to build bespoke parsing methods. In particular, a great number of these transmissions are semi-structured text, which doesn’t necessarily follows the classic english grammar. As seen in Fig. 1 , they can be under the form of tables containing diverse elements, words and numbers, afterwards referred to as tokens. These tables are often implicitly defined, which means that there are no special tags between what is or not part of the table, or even between cells. In these cases, the structure is coming from space or tabs alignment and from the relative order of the tokens. The data often are unlabeled, which means that the content must be read with domain-based knowledge. Thus, automatic extraction of structured information is a major challenge because token candidates come in a variety of forms within a fuzzy context. A high level of supervision is hard to obtain as manual labeling requires time that is hardly affordable when receiving thousands of such emails a day, and even more so as databases can become irrelevant over time. That is why training a generalizable model to extract these data should not rely on labeled inputs, but rather on the content itself - a paradigm called self-supervised learning. Many approaches already exist in Natural Language Processing, such as Part-of-Speech (POS) tagging or Named Entity Recognition (NER), but they do not take advantage of the semi-structured data framework. On the contrary, there exists some information extraction algorithms applied to tables, but they necessitate a great amount of manually defined rules and exceptions. Our model aims to reconcile both approaches for an efficient and totally self-supervised take on information extraction in the particular context of semi-structured data. In this paper, we present a neural architecture for token embedding in plain-text tables, which provides a useful lower-dimensional representation for tasks such as unsupervised, or semi-supervised clustering. Intuitively, tokens with a similar meaning should be close in the feature space to ease any further information extraction. Our model aims to combine the better of the context and the character composition of each token, and that is why the neural architecture is designed to learn both context and character-level representations simultaneously. Finally, we can take advantage of the distances between tokens in the feature space to create proper tables from fuzzy input data.

Question: Which regions of the United States do they consider?
Answer: all regions except those that are colored black