Source: https://patents.google.com/patent/US20020091688A1/en
Timestamp: 2018-04-26 23:21:57
Document Index: 254253922

Matched Legal Cases: ['application No. 09', 'application No.09', 'Application No. 60', 'application No. 09', 'application No. 09', 'application No. 09', 'application No. 09', 'application No. 09', 'application No. 09']

US20020091688A1 - Computer method and apparatus for extracting data from web pages - Google Patents
Computer method and apparatus for extracting data from web pages Download PDF
US20020091688A1
US20020091688A1 US09910169 US91016901A US2002091688A1 US 20020091688 A1 US20020091688 A1 US 20020091688A1 US 09910169 US09910169 US 09910169 US 91016901 A US91016901 A US 91016901A US 2002091688 A1 US2002091688 A1 US 2002091688A1
US09910169
US7065483B2 (en )
Computer method and apparatus for extracting information from a Web page is disclosed. The invention apparatus is formed of an extractor coupled to receive Web pages from a source. The extractor uses natural language processing to extract desired information from the Web page. A storage subsystem receives from the extractor the extracted desired information and stores the extracted desired information in a database. The invention method for extracting data from a Web page includes the computer implemented steps of (i) using natural language processing, fmding possible formal names on a given Web page, (ii) using pattern matching, searching the given Web page for formal names not found by the natural language processing, and (iii) refining a combined set of the found formal names to produce a working set of people and organization names extracted from the given Web page. The refining includes determining aliases of respective people and organization names, so as to effectively reduce duplicate names.
http://fi.enn.com/archives/may99/pr37.html
One purpose of the present invention is to collect publicly available information about people and organizations published in the Web. Usually information about organizations is published in Web sites maintained by the organizations themselves and includes the above-mentioned information. However, very often relevant informafion can be collected by press releases, news articles, product reviews and other independent sources.
[0067]FIG. 1 is a flow diagram of an extractor process for processing a document in the preferred embodiment of FIG. 4.
[0068]FIG. 2 is a flow diagram of functions performed during lexical analysis in the extractor process of FIG. 1.
[0069]FIG. 3 is a flow chart of post-processing actions performed on information extracted from a Web page by the extractor of FIG. 1.
[0070]FIG. 4 is a schematic view of a computer system embodying the present invention.
[0071]FIGS. 5a and 5 b are block diagrams of working database records storing infonnation extracted by the extractor of FIG. 1.
In the preferred embodiment, the Crawler 11 is a version of the software robot described in U.S. patent application No. 09/821,908 filed on Mar. 30, 2001 for a “Computer Method and Apparatus for Collecting People and Organization Information from Web Sites” and assigned to the assignee of the present invention. Specific rules are used to identify pages that contain organization information or relevant people information (e.g. name, employment, contact info, etc). For example, pages with a street address of the organization, press release pages, product list pages, pages that contain information about the management team, employee directory, etc. All the interesting pages 12 that the Crawler 11 collects are then passed (through a local storage 48) to the Extractor 41 for further processing and data extraction.
In the preferred embodiment of the present invention, these are obtained through the NLP techniques described in U.S. patent application No.09/585,320 filed on Jun. 2, 2000 for a “Method and Apparatus for Deriving Information from Written Text”.
[0089]FIG. 1 illustrates the document processing performed by the Extractor 41 in the preferred embodiment. First, the Extractor 41 transforms the subject document (Web page) into a canonical or standardized form (step 111). The way information about a person or organization is expressed and the meaning of such information varies according to the kind of Web page on which the information appear and, in particular, to the type of section inside a page (e.g.: board of directors, contacts at the end of a press release, staff list, member list, organization URL, stock ticker symbol, organization mission, product description, etc.). The type of a page is obtained through a process described in Invention 3 as disclosed in the related Provisional Application No. 60/221,750 filed on Jul. 31, 2000 for a “Computer Database Method and Apparatus”, now pending as U.S. patent application No. 09/768,869, filed Jan. 24, 2001. However, for a particular page type, it is necessary to recognize different sections within the page and to discover the overall structure of the page. For this, Applicants have devised various methods as follows.
Noun Phrase Typing: With the use of a noun phrase classifier joiner, the lexical analysis 113 identifies all noun phrases that could potentially correspond to a person's or organization's name (step 25). An example noun phrase classifier joiner is disclosed in U.S. patent application No. 09/585,320 filed Jun. 2, 2000, herein incorporated by reference. In order to identify such noun phrases, rules describing the composition of a NAME must be defined. Those rules define what are the different parts of a name and the different order in which they can appear. In the preferred embodiment, names of people have seven possible parts: Address, FirstName, Initial, MiddleName, NameParticle (e.g.: van, de), LastName, NameSuffix.
Noun Phrase Joining: In some cases, names of people span across more than one noun phrase. In particular, this is the case when commas “,” appear within a name (e.g.: “John Smith, Jr.”, “Smith, John”). With the use of a noun phrase joiner (see patent application No. 09/585,320 filed on Jun. 2, 2000 for a “Method and Apparatus for Deriving Information from Written Text”), rules have been defined to (i) detect such construction, (ii) join the different parts in a single noun phrase and (iii) assign the correct name part to each word.
Noun Phrase Splitting: In many cases, improper punctuation or irregular format create a situation where the name of a person is immediately followed by a title or something else (e.g.: “John Smith Vice-President”). With the use of a noun phrase splitter (see patent application No. 09/585,320 filed on Jun. 2, 2000 for a “Method and Apparatus for Deriving Information from Written Text”), rules are defined to (i) detect such constructions, (ii) split the noun phrase into two parts at the appropriate point, and (iii) reanalyze the name so that correct name parts are assigned.
When pieces of personal information appear in a connected sentence, the logical relationship between each element (e.g.: title, company/organization, date) is expressed through the rules of the English language. In order to understand how those pieces of information are related, Natural Language Processing is employed in information extraction step 118. Sentences are syntactically parsed to obtain lexical frames representing potential relationships between words (see patent application No. 09/585,320 filed on Jun. 2, 2000 for a “Method and Apparatus for Deriving Information from Written Text”). Alternatively, those relationships can also be obtained through other NLP methods such as deterministic parsing. Those syntactical relations (or trees) are then searched for the appearance of pre-defined patterns corresponding to information that is of interest to the Extractor 41. Those patterns are referred to as “Semantic Frames”.
In the preferred embodiment, a list of semantic frames are defined for (a) sentences that express a relationship of employment between a company and a person. This includes, for instance, such semantic frames as “work Subject: [PERSON] as:[TITLE] for: [COMPANY]”, and, (b) sentences that express that a person holds a certain degree, for instance “graduated Subject:[PERSON] from:[INSTITUTION] with:[DEGREE] in:[DISCIPLINE]”. Included in the former are semantic frames that recognize an organization as an object of certain verbs, such as “joined” (as in “. . . joined ACME in 1998”) and “was employed” (as in “. . . was employed by ACME . . . ”). Other semantic frames for other types of personal or organization information can be defined using the same method. Semantic frames can also indicate how the resulting database record 16, 17 should be constructed from elements matching the frame (see patent application No. 09/585,320 filed on Jun. 2, 2000 for a “Method and Apparatus for Deriving Information from Written Text”). Once a sentence has been parsed, all possible semantic frames are applied. Successful matches lead to the creation of database or working records 16, 17 (FIG. 4).
In one embodiment, the database/working records 16, 17 are structured as follows and illustrated in FIGS. 5a and 5 b. There is one working record 16 a, 17 a for each “extracted” person or organization, i.e., extracted data pertaining to an individual or organization, respectively. The working record 16 a in FIG. 5a stores the name of the person using respective fields for a prefix (Dr., Mr., Ms., etc.), first name, middle name, last name and suffix (M.D., Ph.D. . . . ).
Further it is useful to associate a list 17 e (FIG. 5b) of keywords to a particular organization. This enables searches for organizations operating in certain domains or related to certain activities or products. In a preferred embodiment, keywords describing an organization are computed by calculating the recurrence of all noun phrases within interesting pages or all pages. As a function of the trade-off between coverage and accuracy some or all of the following rules are activated:
Referring back to FIG. 4, the loader 43 is responsive to the working records 16, 17 produced by Extractor 41. In the preferred embodiment, loader 43 identifies whether two records 16 relate to the same person at the same current position or whether two records 17 relate to the same organization. In addition, it is very common in a language to use different words and abbreviations that basically mean the same thing. In order for the loader 43 to properly identify people and organizations and to collect all relevant data about them it is necessary to identify and resolve all such aliases, a process called aliasing resolution. The loader 43 accomplishes detection/ deletion of effectively duplicate records 16, 17 and aliasing resolution of people names, organization names and titles as follows.
At the database 45 level, the same process used to conclude that two given organizations are the same can be used to tie a person to an organization as well as to another person. Information about an organization is also stored in the database 45, including the host name, the location of the organization, a description, etc. By storing the database id 19 (FIGS. 5a, 5 b) of the organization that the person works for, sophisticated queries can be run that will search for people based on the organization they work for, in addition to information about the person itself. For example, assume that a person in the database 45, John Smith, is a software engineer who works for Dragon Systems, a speech recognition software company. A search can be performed for software engineers who work for companies involved in speech recognition, even if nowhere in the information about John Smith himself does it mention speech recognition.
searching the given Web page for formal names not found by the natural language processing step of finding, said searching producing a second set of formal names; and
13. A method as claimed in claim 1 wherein the step of searching employs pattern matching.
14. A database having records formed by data extracted from Web pages by the method of claim 1.
15. A method for extracting information from a Web page document comprising the computer implemented steps of:
detecting a regular recurrence of a certain type of element, the detecting producing additional formal names;
16. A method as claimed in claim 15, further comprising the step of transforming the given Web page document into a standardized form, the step of transforming including identifying page structure of the Web page document.
17. A method as claimed in claim 15, further comprising the step of assigning a type to each line in the given Web page document, the step of assigning a type indicating purpose of each line in the given Web page document.
18. A method as claimed in claim 17 wherein the step of performing a lexical analysis further identifies elements of interest on lines of certain assigned types.
19. A method as claimed in claim 17 wherein the step of detecting includes using pattern matching, detecting a regular recurrence of a certain type of line, to produce additional formal names.
20. A method as claimed in claim 15 wherein the step of performing a lexical analysis includes syntactically and grammatically identifying elements of interest.
21. A method as claimed in claim 20 wherein the step of identifying elements of interest identifies noun phrases that correspond to a person or organization named in the given Web page document.
22. A method as claimed in claim 20 wherein the step of performing a lexical analysis includes using natural language processing.
23. A method as claimed in claim 20 wherein the step of performing a lexical analysis includes utilizing rules describing composition of a name.
24. A method as claimed in claim 15 wherein the step of resolving aliases includes employing rules for determining variant versions of a person's name or an organization's name.
25. A method as claimed in claim 15 wherein the step of aliasing includes rejecting names containing predefined forms of common known phrases.
26. A method as claimed in claim 15 further comprising the steps of:
extracting from the formed text units desired information relating to the people or organizations named in the given Web page document wherein the step of grouping identifies boundaries where information about a person or organization is to be found.
27. A method as claimed in claim 26 wherein the step of grouping recognizes elements of information that span across more than one line.
28. A method as claimed in claim 26 wherein the step of extracting includes:
29. A method as claimed in claim 28 wherein the step of determining type of Web page document includes determining structure and organization of contents of the document.
30. A method as claimed in claim 28 wherein the step of extracting includes determining whether the given Web page document is a press release, and if so, identifying organization mentioned in the press release.
31. A method as claimed in claim 26 wherein the step of extracting includes using a parser to recognize the relationship between elements of information.
32. A method as claimed in claim 31 wherein the step of extracting further includes utilizing predefined semantic frames for determining (i) sentences that express a relationship between a person and organization named in the given Web page document and (ii) sentences that express that a person has a certain level of education.
33. A method as claimed in claim 26 wherein the step of extracting includes associating a person or organization with an element of information if said element appears in a non-sentence within a formed text unit for that person or organization.
34. A method as claimed in claim 26 wherein the step of extracting further divides a line that contains multiple names.
35. A method as claimed in claim 26 wherein the step of extracting is rules based.
36. A method as claimed in claim 15 further comprising the step of post-processing to extract further names of organizations and relationships to people named in the given Web page document.
37. A method as claimed in claim 36 wherein the step of post-processing includes:
38. Computer apparatus for extracting information from a Web page comprising:
a storage subsystem coupled to the extractor for storing the extracted desired information in a data store.
39. Computer apparatus as claimed in claim 38 wherein the extractor extracts desired information from a given Web page by:
using pattern matching, searching the given Web page for formal names not found by the natural language processing step of finding, said searching producing a second set of formal names; and
40. Computer apparatus as claimed in claim 39 wherein the extractor further determines aliases of respective people and organization names in the combined set so as to reduce effectively duplicate names.
41. Computer apparatus as claimed in claim 39 wherein the extractor further finds professional titles and determines organization for which a person named on the given Web page holds that title.
42. Computer apparatus as claimed in claim 39 wherein the extractor further determines educational background of a person including at least one of name of institution, degree earned from the institution and date of graduation from the institution.
43. Computer apparatus as claimed in claim 39 wherein the extractor further determines employment history of a person named on the given Web page.
44. Computer apparatus as claimed in claim 38 wherein the extractor is rules based.
45. Computer apparatus as claimed in claim 38 wherein the extractor further determines type of the given Web page, and from the determined type defines contents of different portions of the Web page, such that extraction of desired information is performed as a function of the defined contents.
46. Computer apparatus as claimed in claim 45 wherein the extractor further using the determined type, deduces additional information regarding a named person on the given Web page, the additional information supplementing information found on another Web page of the same Web site as the given Web page.
47. Computer apparatus as claimed in claim 38 wherein the extracted desired information includes names of people or organizations named on the given Web page, addresses, telephone numbers and email addresses relating to the named person or organization.
48. Computer apparatus as claimed in claim 38 wherein the storage subsystem is formed of a loader responsive to the extracted desired information, the loader post-processing the extracted desired information to refine the extracted desired information for storage in the data store.
US09910169 2000-07-31 2001-07-20 Computer method and apparatus for extracting data from web pages Active 2024-04-19 US7065483B2 (en)
US09910169 US7065483B2 (en) 2000-07-31 2001-07-20 Computer method and apparatus for extracting data from web pages
US20020091688A1 true true US20020091688A1 (en) 2002-07-11
US7065483B2 US7065483B2 (en) 2006-06-20
US20100299184A1 (en) * 2009-05-20 2010-11-25 Joseph Ruston Bishop Mining of distributed scientific data for enriched product/contact valuation
US20120030200A1 (en) * 2008-06-25 2012-02-02 Microsoft Corporation Topics in relevance ranking model for web search
EP1669896A2 (en) * 2004-12-03 2006-06-14 Panscient Pty Ltd. A machine learning system for extracting structured records from web pages and other text sources
WO2007038292A2 (en) * 2005-09-22 2007-04-05 Fair Isaac Corporation Method and apparatus for automatic entity disambiguation
WO2007038292A3 (en) * 2005-09-22 2009-04-16 Fair Isaac Corp Method and apparatus for automatic entity disambiguation
US9092524B2 (en) * 2008-06-25 2015-07-28 Microsoft Technology Licensing, Llc Topics in relevance ranking model for web search