Training and database supervision	We propose to train the neural network by referencing candidates extracted by a high-recall candidate-generating parser against a potentially noisy reference source (see Figure FIGREF12 , left panel). In our application, this reference was a database containing historical time series data, which enabled us to check how well the extracted numerical data fit into time series in the database. Concretely, we compute a consistency score INLINEFORM0 that measures the degree of consistency with the database. Depending on the application, the score may for instance be a squared relative error, an absolute error, or a more complex error function. In many applications, the score INLINEFORM1 will be noisy (see below for further discussion). We threshold INLINEFORM2 to obtain binary correctness labels INLINEFORM3 . We then use the binary correctness labels INLINEFORM4 for supervised neural network training, with binary cross-entropy loss as the loss function. This allows us to train a network that can compute a pseudo-likelihood INLINEFORM5 of a given extraction candidate to agree with the database. Thus, INLINEFORM6 estimates how likely the extraction candidate is correct.

Information Extraction with Character-level Neural Networks and Free Noisy Supervision	We present an architecture for information extraction from text that augments an existing parser with a character-level neural network. The network is trained using a measure of consistency of extracted data with existing databases as a form of noisy supervision. Our architecture combines the ability of constraint-based information extraction systems to easily incorporate domain knowledge and constraints with the ability of deep neural networks to leverage large amounts of data to learn complex features. Boosting the existing parser's precision, the system led to large improvements over a mature and highly tuned constraint-based production information extraction system used at Bloomberg for financial language text.

Conclusion	We presented an architecture for information extraction from text using a combination of an existing parser and a deep neural network. The architecture can boost the precision of a high-recall information extraction system. To train the neural network, we use measures of consistency between extracted data and existing databases as a form of noisy supervision. The architecture resulted in substantial improvements over a mature and highly tuned constraint-based information extraction system for financial language text. While we used time series databases to derive measures of consistency for candidate extractions, our set-up can easily be applied to a variety of other information extraction tasks for which potentially noisy reference data is available.

In a production setting, the neural architecture presented here reduced the number of false positive extractions in financial information extraction application by INLINEFORM0 relative to a mature system developed over the course of several years.

At Bloomberg, we found the system for information extraction described in this paper especially useful to extract time series (TS) data. As an example, consider numerical relations of the form ts_tick_abs (TS symbol, numerical value), e.g. ts_tick_abs (US_Unemployment, 4.9%), or ts_tick_rel (TS symbol, change in num. value), e.g. ts_tick_abs (US_Unemployment, -0.2%).

Our contribution	We present an information extraction architecture that augments a candidate-generating parser with a deep neural network. The candidate-generating parser may leverage constraints. At the same time, the architecture gains the neural networks's ability to leverage large amounts of data to learn complex features that are tuned for the application at hand. Our method assumes the existence of a potentially noisy source of supervision INLINEFORM0 , e.g. via consistency checks of extracted data against existing databases, or via human interaction. This supervision is used to train the neural network. Our extraction system has three advantages over earlier work on information extraction with deep neural networks  ,  ,  ,  ,  ,  ,  ,  : Our system leverages “free” data to train a deep neural network, and does not require large-scale manual annotation. The network is trained with noisy supervision provided by measures of consistency with existing databases (e.g. an extraction ts_tick_abs (US_Unemployment, 49%) would be implausible given recent US employment history). With slight modifications, our pipeline could be trained with supervision from human interaction, such as clicks on online advertisements.

Results	The full pipeline, deployed in a production setting, resulted in a reduction in false positives of more than INLINEFORM0 in the extractions produced by our pipeline. The drop in recall relative to the production system was smaller than INLINEFORM1 . We found that even with only 256 hidden LSTM cells, the neural network described in the previous section significantly outperformed a 2-layer fully connected network with n-grams based on document text and parser annotations as input.

Information extraction in finance	Unstructured textual data is abundant in the financial domain (see e.g. Figure FIGREF2 ). This information is by definition not in a format that lends itself to immediate processing. Hence, information extraction is an essential step in business applications that require fast, accurate, and low-cost information processing. In the financial domain, these applications include the creation of time series databases for macroeconomic forecasting or financial analysis, as well as the real-time extraction of time series data to inform algorithmic trading strategies. Bloomberg has had information extraction systems for financial language text for nearly a decade. To meet the application domain's high accuracy requirements, marrying constraints with statistical models is often beneficial, see e.g.  ,  . Many quantities appearing in information extraction problems are by definition constrained in the numerical values they can assume (e.g. unemployment numbers cannot be negative numbers, while changes in unemployment numbers can be negative). The inclusion of such constraints may significantly boost data efficiency. Constraints can be complex in nature, and may involve multiple entities belonging to an extraction candidate generated by the parser.

Learning without explicit annotations is critical in applications where large-scale manual annotation would be prohibitively expensive. If an extractor for the given application has already been built, the neural network boosts its accuracy without the need to re-engineer or discard the existing solution. Even for new systems, the decoupling of candidate-generation and the neural network offers advantages: the candidate-generating parser can easily enforce contraints that would be difficult to support in an algorithm relying entirely on a neural network. Note that, in particular, a carefully engineered candidate-generating parser enforces constraints intelligently, and can in many instances eliminate the need to evaluate computationally expensive constraints, e.g. API calls. We encode the candidate-generating parser's document annotations character-by-character into vectors INLINEFORM0 that also include a one-hot encoding of the character itself. We believe that this encoding makes it easier for the network to learn character-level characteristics of the entities in the semantic relation. Moreover, our encoding lends itself well to processing both by recurrent architectures (processing character-by-character input vectors INLINEFORM1 ) and convolutional architectures (performing 1D convolutions over an input matrix whose columns are vectors INLINEFORM2 ).

Acknowledgments	We would like to thank my managers Alex Bozic, Tim Phelan and Joshwini Pereira for supporting this project, as well as David Rosenberg from the CTO's office for providing access to GPU infrastructure.

We assume that the noise in the source of supervision INLINEFORM0 is limited in magnitude, e.g. INLINEFORM1 . We moreover assume that there are no strong patterns in the distribution of the noise: if the noise correlates with certain attributes of the candidate-extraction, the pseudo-likelihoods INLINEFORM2 might no longer be a good estimate of the candidate extraction's probability of being a correct extraction. There are two sources of noise in our application's database supervision. First, there is a high rate of false positives. It is not rare for the parser to generate an extraction candidate ts_tick_abs (TS symbol, numerical value) in which the numerical value fits into the time series of the time series symbol, but the extraction is nonetheless incorrect. False negatives are also a problem: many financial time series are sparse and are rarely observed. As a result, it is common for differences between reference numerical values and extracted numerical values to be large even for correct extractions. The neural network's training data consists of candidates generated by the candidate-generating parser, and noisy binary consistency labels INLINEFORM0 .

For our application, we include in INLINEFORM3 a vector of indicators specifying whether or not any of the entities appearing in the relations supported by the parser were found in the position of character INLINEFORM4 .

Neural network input and architecture	The neural network processes each input candidate independently. To estimate the correctness of a extracted candidate, the network is provided with two pieces of input (see Figure FIGREF14 for the full structure of the neural network): first, the network is provided with a vector INLINEFORM0 containing global features, such as attributes of the document's author, or word-level n-gram features of the document text. The second piece of input data consists of a sequence of vectors INLINEFORM1 , encoding the document text and the parser's output at a character level. There is one vector INLINEFORM2 for each character INLINEFORM3 of the document section where the extraction candidate was found. The vectors INLINEFORM0 are a concatenation of (i) a one-hot encoding of the character and (ii) information about entities the parser identified at the position of INLINEFORM1 . For (i) we use a restricted character set of size 94, including [a-zA-Z0-9] and several whitespace and special characters, plus an indicator to represent characters not present in our restricted character set. For (ii), INLINEFORM2 contains data representing the parser's output.

Overview	The information extraction pipeline we developed consists of four stages (see right pane of Figure FIGREF12 ). The document is parsed using a potentially constraint-based parser, which outputs a set of candidate extractions. Each candidate extraction consists of the character offsets of all extracted constituent entities, as well as a representation of the extracted relation. It may additionally contain auxilliary information that the parser may have generated, such as part of speech tags. We compute a consistency score INLINEFORM0 for the candidate extraction, measuring if the extracted relation is consistent with (noisy) supervision INLINEFORM1 (e.g. an existing database). Each candidate extraction generated, together with the section of the document it was found in, is encoded into feature data INLINEFORM0 . A deep neural network is used to compute a neural network candidate correctness score INLINEFORM1 for each extraction candidate. A linear classifier classifies extraction candidates as correct and incorrect extractions, based on consistency and correctness scores INLINEFORM0 and INLINEFORM1 and potentially other features. Candidates classified as incorrect are discarded.

Question: by how much did the system improve?
Answer: By more than 90%