Current natural language processing systems lack lexical and grammatical resources with sufficient coverage for unrestricted text . Time flies like an arrow . Fruit flies like a banana . It would be useful for many applications to know that fruit flies is a phrase and time flies is not. Most systems currently do not have access to this kind of information.
Parsers currently operate at the level of parts of speech , without looking at the words.
Ultimately, parsers and other natural language applications will have to make greater use of collocational constraints and other constraints on words. The task may require a monumental effort like Murray's Oxford English Dictionary project . This kind of evidence is suggestive of the desired distinction, though far from conclusive. The long-term research challenge is to derive lexicons and grammars for broad coverage natural language processing applications from corpus evidence . In the short-term, progress can be achieved by improving the infrastructure, i.e. the stock of intermediate resources mentioned in section 12.1 .