Source: https://patents.google.com/patent/JP4007596B2/en
Timestamp: 2020-02-16 18:41:03
Document Index: 707170110

Matched Legal Cases: ['art 13', 'art 11', 'art, 12', 'art, 14', 'art, 15', 'art, 210']

JP4007596B2 - Server and program - Google Patents
Server and program Download PDF
JP4007596B2
JP4007596B2 JP2003047983A JP2003047983A JP4007596B2 JP 4007596 B2 JP4007596 B2 JP 4007596B2 JP 2003047983 A JP2003047983 A JP 2003047983A JP 2003047983 A JP2003047983 A JP 2003047983A JP 4007596 B2 JP4007596 B2 JP 4007596B2
JP2003047983A
JP2004258911A (en
隆文 木下
2003-02-25 Application filed by インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｓｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ filed Critical インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｓｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ
2004-09-16 Publication of JP2004258911A publication Critical patent/JP2004258911A/en
2007-11-14 Publication of JP4007596B2 publication Critical patent/JP4007596B2/en
The present invention relates to a search engine used for information search on the Internet or the like, and more particularly to a search robot that automatically collects information on a network.
Today, with the spread of computer networks represented by the Internet, various services are provided via the network. The types of terminal devices connected to the network also receive and display contents such as mobile phones and PDAs (Personal Digital Assistants) from computer devices with high processing capabilities such as personal computers and workstations for simple processing. There are various things to do only. Therefore, there are sites (servers) that dynamically create content and provide services according to the type (capability) of the connected terminal device (user agent).
In the case of a web server that returns web content according to the type of terminal device, the type of the terminal device can be identified by referring to a User-Agent HTTP header included in an HTTP (Hypertext Transfer Protocol) request from the terminal device. . Therefore, it is possible to generate and return web content according to the capability of the terminal device. To generate content according to the capability of the terminal device, for example, the content is divided according to the memory capacity of the terminal device (that is, the size of data that can be read at one time) and downloaded multiple times. For example, a text-only content excluding an image is generated for a terminal device that does not have an image.
In addition, today, it is a common practice to search for and acquire desired content from a vast amount of content on a network using a search site (search engine). This search site acquires and holds information for accessing content on the network in advance, and returns the information in response to a search request from a terminal device. As today's main search site, it automatically accesses the site on the network while following the hyperlink (hereinafter referred to as the link) of the content, collects information for accessing the content of the site, and from the terminal device There is a robot type search site that responds to the search request (see, for example, Patent Documents 1 and 2).
FIG. 17 is a diagram illustrating a schematic configuration of a robot type search site.
As shown in FIG. 17, the robot-type search site 1710 is acquired by the search robot 1711 that automatically accesses the site 1720 on the network and acquires information for accessing the content of the site, and the search robot 1711. It includes a database 1712 for registering and storing information, and a search engine 1713 that receives a search request from the terminal device 1730, searches the database 1712, and returns a search result.
JP 2002-215642 A
JP 2002-259432 A
As described above, in the robot type search site, the search robot accesses the site on the network and acquires information (for example, URL (uniform resource locators)) for accessing the content. However, as described above, when a search robot accesses a site that dynamically creates and responds according to the type of terminal device that is a client, the site recognizes and responds as an access from the search robot on the site side. The search robot cannot acquire information related to the content of each model corresponding to the site. Depending on the site, an error may be processed as an unknown model, or an appropriate model may be assumed. These can cause a decline in the trust of search sites.
For example, a predetermined site S1 has a content C1 having a size of 5 KB, and for the terminal device A having a data size of 6 KB that can be read at a time, the content of 5 KB is sent as it is, and the data size that can be read at a time is Consider a case in which the 5 KB content is divided into 3 KB contents c01, c02 and c03 of 2 KB, 2 KB, and 1 KB and sent in order to the 2 KB terminal device B.
When the search robot is determined to be the terminal device A at the site during the robot search (search by the search robot), only the original content C1 having a size of 5 KB is registered in the database of the search site S2.
In this situation, it is assumed that the terminal device B searches the search site S2 for content with a predetermined search term and the content C1 is hit. When an HTTP request is sent from the terminal device B to the site S1 according to the search result, the content c01 divided into three is returned from the site S1 as described above. At this time, if the hit search term exists in the content c02 or the content c03, the terminal device B cannot obtain desired information even though it is accessed based on the search result of the search site S2.
In the above-mentioned Patent Document 2, the content of the content is analyzed, and the degree of appropriateness of the predetermined content according to the type of the terminal device is calculated and evaluated. A technique for selecting a search result is disclosed. However, even the technique disclosed in Patent Document 2 does not acquire content assuming the type of terminal device when the search robot collects content.
Therefore, according to the present invention, when information is collected by a search robot, information regarding content of each model corresponding to the site can be acquired for a site that dynamically generates and returns content according to the type of the terminal device. The purpose is to do so.
The present invention that achieves the above object is realized as a server that is configured as follows. This server includes a transmission / reception unit that relays data exchange between a site on the network and a search robot that accesses this site and collects information, and a response from the site to an information acquisition request from the search robot to the site. Is received and the link destination of the link included in this response is a content that returns an individual response according to the type of the user agent that is the access source, the link is expanded, and the link destination content is The link expansion unit that acquires information about the corresponding user agent, and the user agent included in the information acquisition request when the transmission / reception unit receives an information acquisition request from the search robot made based on the expanded link Who convert the information of the user to the information of the user agent supported by the linked content Characterized in that it comprises a Jento information editing unit.
The server further includes a database in which location information of content that returns an individual response according to the type of user agent that is an access source, and information of a user agent that the content corresponds to and registered. can do. In this case, the link expansion unit searches the database using the link destination position information of the link included in the response from the site as an argument, and acquires user agent information. The user agent information editing unit converts the information of the information acquisition request using the user agent information obtained from the database.
The other server of the present invention is received by the first transmission / reception means for exchanging data with a search robot that accesses a site on the network and collects information, and the first transmission / reception means. The editing means for converting the user agent information included in the information acquisition request from the search robot to the site into other user agent information for pretending to be an information acquisition request issued from a predetermined user agent, and the site And a second transmission / reception unit that exchanges data and transmits an information acquisition request in which the user agent information is converted by the editing unit to a destination site.
The server may further include an analysis unit that analyzes a response from the site received by the second transmission / reception unit to the search robot. In this case, the editing unit specifies a predetermined user agent based on the information obtained by the analysis by the analyzing unit, and converts the user agent information of the information acquisition request into the user agent information of the specified user agent.
More preferably, in this server, when the link destination of the link included in the response is a content that returns an individual response according to the type of the user agent that is the access source, the link destination information of the link is displayed. The dummy information including identification information for identifying the user agent corresponding to the linked content is replaced with the first transmission / reception means to be transmitted to the search robot. Then, when the first transmission / reception means receives the information acquisition request issued from the search robot with the dummy information as the destination, the editing means returns the dummy information to the original link destination information, and the information acquisition request The included user agent information is converted into user agent information of the user agent specified by the identification information included in the dummy information.
Still another server of the present invention includes a transmission / reception unit that relays data exchange between a website on the network and a search robot that accesses the website and collects information, and a website from the search robot. When the web site receives a response from this website in response to an information acquisition request, and the link destination of the link included in this response is a web content that returns an individual response according to the type of user agent that is the access source In addition, a link expansion unit that replaces the link destination URL of the link included in this response with an alternative URL that individually corresponds to the user agent that this web content supports, and is issued from the search robot with this alternative URL as a destination. When the transmission / reception unit receives an information acquisition request, the alternative URL is changed to the original link. Information on the user agent in the HTTP (Hypertext Transfer Protocol) header of the information acquisition request when the transmission / reception unit receives the URL conversion unit that replaces the previous URL and the information acquisition request issued from the search robot with the alternative URL as the destination And a user agent information editing unit that converts the information into user agent information corresponding to the alternative URL.
The above servers may be physically built on the same server machine as the server having the search robot, or may be built on the same server machine as the server having the site. Furthermore, it can also be set as the structure built in the server machine provided on the network separately from these servers.
Still another embodiment of the present invention for achieving the above object can be realized as the following information collecting method for collecting information from a site on a network. That is, this information collection method receives a response from this site in response to an information acquisition request to a predetermined site, and the link destination of the link included in the received response is individually determined according to the type of user agent that is the access source. In the case of content that returns a response, the user agent is expanded based on the first step of expanding the link and acquiring the information of the user agent corresponding to the linked content, and the obtained user agent information. A second step of transmitting an information acquisition request to which information for impersonating an information acquisition request issued from is added to the link destination of the link included in the response, and a response according to the type of the user agent from the link destination And a third step of acquiring.
Another information collecting method according to the present invention receives a response from a site to an information acquisition request from a predetermined search robot to a predetermined site, and a user agent whose link destination is included in the received response is an access source If the web content returns an individual response according to the type of the URL, the URL of the link destination of the link included in the response is replaced with an alternative URL that individually corresponds to the user agent supported by the web content, The first step of transmitting the response to the search robot and the information acquisition request issued from the search robot with the alternative URL as the destination are received, the alternative URL is replaced with the original link destination URL, Replaces the user agent information in the HTTP (Hypertext Transfer Protocol) header of the information acquisition request The second step of converting the information to the user agent corresponding to the RL and transmitting the information to the link destination, and the response from the link destination to the information acquisition request converted into the user agent information are received. And a third step of adding the identification information and transmitting it to the search robot.
Furthermore, the present invention is also realized as a program that controls a computer to realize the above-described function as a server, or a program that causes a computer to execute processing corresponding to each step of the information acquisition method. This program can be provided by being stored and distributed in a magnetic disk, optical disk, semiconductor memory, or other recording device, or distributed via a network.
FIG. 1 is a diagram for explaining the concept of robot search according to the present invention.
In the present invention, a conversion proxy 10 that converts a request from the search robot 310 and a response from the site 210 is interposed between the site 210 that provides the content and the search robot 310 that performs the robot search. Thereby, when collecting information by robot search, it is possible to acquire information related to content of each model type from a site that dynamically generates and returns content according to the type of the terminal device.
2 to 4 are diagrams illustrating an embodiment of the present invention.
In the example of FIG. 2, the conversion proxy 10 is provided in a search server (a server that provides a search site function) 300 together with the search robot 310. In this configuration, the search robot 310 accesses each site 210 on the network via the conversion proxy 10.
The example of FIG. 3 is a configuration in which the conversion proxy 10 is provided independently in a predetermined proxy server (intermediate site) 100 on the network. In this configuration, the search robot accesses the intermediate site, and accesses each site 210 via the conversion proxy 10.
In the example of FIG. 4, the conversion proxy 10 is provided in the content server 200 together with the site 210. In this configuration, each search robot 310 on the network trying to access the site 210 of the content server 200 accesses the site 210 via the conversion proxy 10.
FIG. 5 schematically shows an example of a hardware configuration of a computer apparatus suitable for realizing the server (search server 300, proxy server 100, and content server 200) provided with the conversion proxy shown in FIGS. It is a figure.
The computer apparatus shown in FIG. 5 includes a CPU (Central Processing Unit) 101 which is a calculation means, a main memory 103 connected to the CPU 101 via an M / B (motherboard) chipset 102 and a CPU bus, Similarly, a video card 104 connected to the CPU 101 via the M / B chipset 102 and AGP (Accelerated Graphics Port), and a hard disk 105 connected to the M / B chipset 102 via a PCI (Peripheral Component Interconnect) bus, A floppy disk drive 109 and a keyboard / computer connected to the M / B chipset 102 via a network interface 106 and a USB port 107, and a low-speed bus such as a bridge circuit 108 and an ISA (Industry Standard Architecture) bus from the PCI bus. And a mouse 110.
Note that FIG. 5 merely illustrates the hardware configuration of the computer apparatus that implements the present embodiment, and various other configurations can be employed as long as the present embodiment is applicable. For example, instead of providing the video card 104, only the video memory may be mounted and the image data may be processed by the CPU 101, or a CD-ROM (Compact Disc Read Only) via an interface such as ATA (AT Attachment). Memory (DVD) and DVD-ROM (Digital Versatile Disc Read Only Memory) drives may be provided.
In the present embodiment described below, a web-based network system typified by the Internet will be described as an example. That is, assume that the site 210 in FIG. 1 holds various web contents (web pages and objects thereof), and the search robot 310 sends an HTTP request to the site 210 and receives an HTTP response.
In the present embodiment, the site 210 is a normal website, and the search robot 310 is also a normal search robot conventionally used for web search.
FIG. 6 is a block diagram showing a functional configuration of the conversion proxy 10 according to the present embodiment.
As shown in FIG. 6, the conversion proxy 10 includes a transmission / reception unit 11 for mediating a request (HTTP request) from the search robot 310 to the site 210 and a response (HTTP response) from the site 210 to the search robot 310, and a terminal Diversity URL database (DB) 12 that registers and holds information on a site 210 that dynamically generates content according to the type of device, or URLs of dynamically generated content (hereinafter referred to as diversity URLs). A link expansion unit 13 as an analysis unit that expands a link of content included in a response sent from the site 210, a URL conversion unit 14 that converts a destination URL as an editing unit for a request sent from the search robot 310, and User agent information for editing the user agent information in the request And a collection unit 15.
The link expansion unit 13, the URL conversion unit 14, and the user agent information editing unit 15 described above are virtual software blocks realized by controlling the CPU 101 with the program expanded in the main memory 103 shown in FIG. is there. A program that controls the CPU 101 to realize these functions is provided by being stored and distributed in a magnetic disk, an optical disk, a semiconductor memory, or other storage media, or distributed via a network. In the present embodiment, the program is input via the network interface 106, floppy disk drive 109, CD-ROM drive (not shown) and the like shown in FIG. Then, the program stored in the hard disk 105 is read into the main memory 103, expanded, and executed by the CPU 101, thereby realizing these functions.
The diversity URL database 12 is realized by the main memory 103, the hard disk 105, and the program-controlled CPU 101 shown in FIG. The transmission / reception unit 11 is realized by the network interface 106 and the program-controlled CPU 101 shown in FIG.
In the above configuration, the transmission / reception unit 11 receives a request from the search robot 310, passes the request to the URL conversion unit 14 and the user agent information editing unit 15, and transmits the processed request to the site 210. Further, after receiving a response from the site 210 and passing it to the link expansion unit 13 for processing, the processed response is transmitted to the search robot 310.
The diversity URL database 12 registers a diversity URL, which is content location information, and also relates to the diversity URL, the type of terminal device to be identified, that is, the content specified by the diversity URL (or User agent information for specifying the type of terminal device corresponding to (site) and alternative URLs individually corresponding to the terminal devices specified by this user agent information are registered.
The diversity URL is data whose data format is a character string and whose size is variable length.
The user agent information is data whose character format is a character string and whose size is variable length, and is a value (User-Agent value) described in the User-Agent HTTP header of the HTTP request.
The alternative URL is data with a data format of a character string and a variable size, and is a URL of content (or site) individually corresponding to the terminal device specified by the user agent information.
FIG. 7 is a diagram illustrating an example of information (registered values) registered in the diversity URL database 12.
Referring to FIG. 7, for example, the diversity URL “www.cp_X.com/page_1.html” corresponds to two terminal devices, device A and device B, and corresponding to these user agent information, “ Two alternative URLs “www.cp_X.com/page_1.html.for.device_A” and “www.cp_X.com/page_1.html.for.device_B” are registered.
These pieces of information are acquired and registered in advance from the site 210 or its manager. If a diversity URL is newly set, information related to the diversity URL is given from the site 210 or its administrator and additionally registered. In addition, when the types of terminal devices corresponding to the diversity URL site 210 are increased or changed, new user agent information is given and the registered contents are updated.
Furthermore, the control means of the diversity URL database 12 includes a link expansion list acquisition function, a URL conversion function, and a user agent as access functions to be used by the link expansion unit 13, URL conversion unit 14, and user agent information editing unit 15. An information acquisition function is provided.
The link expansion list acquisition function is a function used when referring to by the link expansion unit 13. When the original URL is passed as an argument, a record having a matching column value is selected from the table shown in FIG. Returns a list of corresponding user agent information and alternative URLs.
The URL conversion function is a function used for reference by the URL conversion unit 14, and when an alternative URL is passed as an argument, a record having a matching column value is selected from the table as shown in FIG. URL (diversity URL) is returned.
The user agent information acquisition function is a function that is used when the user agent information editing unit 15 refers, and when an alternative URL is passed as an argument, a record having a matching column value is selected from the table shown in FIG. Return the corresponding user agent information.
The link expansion unit 13 analyzes the response sent from the site 210, detects the diversity URL from the link included in the response, and selects the link for each corresponding user agent (terminal device) (hereinafter, dummy). Expand to link). For example, let us consider a case where a web page sent as a response from the site 210 has a link to dynamic content corresponding to two types of terminal devices, model A and B. In this case, the diversity URL database 12 is searched by the link expansion list acquisition function using the URL that is the link destination of the link as an argument. When the URL of the link is a diversity URL (when hit), the user agent information of the models A and B registered in the diversity URL database 12 and the alternative URL are read, and the URL of the link To each alternative URL. As a result, the link is replaced with a dummy link.
FIG. 8 is a flowchart for explaining the operation of the link expansion unit 13.
As shown in FIG. 8, when the response from the site 210 is received by the transmission / reception unit 11, the link expansion unit 13 reads the header part and the body part of the response, for example, in the main memory 103 shown in FIG. Store in the work area (step 801). Then, it is determined whether the data type of the body part is text based on the read Content-Type of the header part information (step 802).
If the data type of the body part of the response is text, then the link expansion unit 13 checks whether a link (hyperlink) exists in the body part (web page) (step 803). Whether there is a link is anchor tag "<a> ”or form tag“ It can be checked by detecting a tag carrying a link such as <form>". If a link exists, the link is expanded into a dummy link (step 804). Details of the processing will be described later (FIG. 9).
After the expansion process to the dummy link is completed for all the links existing in the body part, the link expansion part 13 rewrites the response received from the site 210 based on the editing result by the process (step 805). 806). Then, the rewritten response is transmitted to the search robot 310 by the transmission / reception unit 11.
If it is determined in step 802 that the data type of the body part of the response is not text, and if no link exists in the body part in step 803, the response received by the transmission / reception unit 11 is used as it is. Is transmitted to the search robot 310.
FIG. 9 is a flowchart for explaining the detailed contents of the process of expanding to a dummy link in step 804 of FIG. The processing of FIG. 9 is executed for each link detected in step 803.
The link expansion unit 13 first selects an anchor tag “ <a> ”or form tag“ The link destination URL is cut out from the tag carrying the link such as <form>"and stored in the work area of the main memory 103 shown in FIG. 5, for example (step 901). (Step 902) The link expansion list acquisition function described above is used for this process.
If the URL cut out from the link is registered in the diversity URL database 12, the link expansion unit 13 acquires the user agent information of each user agent corresponding to the URL and a list of alternative URLs (step 903). Based on this alternative URL, a dummy URL is generated (step 904). The dummy URL is obtained by adding information (screen display size, color tone, etc.) specific to the user agent to the alternative URL of the corresponding user agent registered in the diversity URL database 12. Therefore, since the alternative URL is included in the dummy URL, in the following description, the alternative URL will be described as a dummy URL without distinction unless otherwise required.
Since these pieces of information added to the alternative URL are usually described by defining an extension header in the contents individually corresponding to the user agent, they can be read and acquired from this extension header. If the alternative URL is defined and registered in the diversity URL database 12 including the information specific to the user agent, the alternative URL read from the diversity URL database 12 becomes a dummy URL as it is, and the link expansion unit In step 13, it is not necessary to generate a dummy URL.
After generating dummy URLs for all user agents corresponding to URLs cut out from links (hereinafter referred to as original URLs), link expansion unit 13 generates original URLs stored in the work area of main memory 103 The dummy URL group is replaced, and the development process to the dummy link is completed (steps 905 and 906). At this time, the return value after normal processing is set in the return code, and the processing returns to the main processing (FIG. 8). As described above, the link destination URL described in the response is rewritten to a dummy URL group (see step 806).
On the other hand, if the URL cut out from the link is not registered in the diversity URL database 12 as a result of the process in step 902, it is not necessary to expand the link, and the process ends as it is. At this time, a return value that does not require processing is set in the return code, and the processing returns to the main processing (FIG. 8).
As described above, when the response from the site 210 includes a link to the diversity URL depending on the type of the user agent, the link is expanded into a link (dummy link) for each corresponding individual user agent. (That is, the original URL (diversity URL) of the link destination is replaced with a dummy URL group). Then, the previous response is transmitted to the search robot 310 in a form including the expanded link.
When the request sent from the search robot 310 is issued based on the dummy link replaced by the link expansion unit 13, the URL conversion unit 14 originally sets the destination (dummy link link destination) of this request. Return to the destination. Specifically, the alternative URL in the dummy URL that is the destination of the request is used as an argument, and the diversity URL database 12 is searched by the URL conversion function. Then, the original URL (original URL) corresponding to the alternative URL registered in the diversity URL database 12 is read, and the destination of the request is rewritten to the read URL.
When the request sent from the search robot 310 is issued based on the dummy link replaced by the link expansion unit 13, the user agent information editing unit 15 describes the user agent information described in the header of this request. Is replaced with the user agent information of the user agent corresponding to the dummy link. Specifically, the alternative URL in the dummy URL that is the destination of the request is used as an argument, and the diversity URL database 12 is searched by the user agent information acquisition function. Then, the user agent information corresponding to the alternative URL registered in the diversity URL database 12 is read and written in the User-Agent HTTP header.
FIG. 10 is a flowchart for explaining operations performed by the URL conversion unit 14 and the user agent information editing unit 15.
As shown in FIG. 10, when the transmission / reception unit 11 receives a request from the search robot 310, the URL conversion unit 14 reads the header part and the body part of the request, for example, the main memory 103 shown in FIG. (Step 1001). At this time, the header portion of the request is cut out and further stored in the work area of the main memory 103. Then, the URL conversion function checks whether the destination URL of the header part is registered in the diversity URL database 12 (step 1002).
If the URL is a dummy URL expanded by the link expansion unit 13, since it is registered in the diversity URL database 12, the corresponding original URL is returned. Therefore, the URL conversion unit 14 replaces the destination URL of the header part extracted from the request stored in the work area of the main memory 103 with the original URL acquired from the diversity URL database 12 (steps 1003 and 1004). . Then, the process proceeds to the user agent information editing unit 15.
The user agent information editing unit 15 acquires user agent information corresponding to a dummy URL (alternate URL) by using a user agent information acquisition function. The user acquired from the diversity URL database 12 is the user agent information (information indicating that the search robot 310 is a user agent) included in the header extracted from the request stored in the work area of the main memory 103. The processing is returned to the URL conversion unit 14 in place of the agent information (step 1005).
The URL conversion unit 14 receives from the search robot 310 stored in the work area of the main memory 103 based on the above URL conversion processing and the result of the user agent information editing processing by the user agent information editing unit 15. The request header information (destination and user agent information) is rewritten (step 1006). Then, the rewritten request is transmitted to the site 210 by the transmission / reception unit 11.
As described above, a request from the search robot 310 is transmitted to the site 210 as if it were a request from each user agent corresponding to the diversity URL. As a result, a response corresponding to each user agent is returned from the site 210 and stored in the search server 300. When the response corresponding to the user agent is received from the site 210 and transmitted to the search robot 310, the conversion proxy 10 stores the user agent information of the user agent corresponding to the response with the meta tag “ <meta> ” <meta name = "keywords" content = "devinfo = model A"> ”,“ It is embedded in the response like <meta name = "keywords" content = "devinfo = model B">. Thereby, the search server 300 can store the user agent and the response in association with each other. For content that does not return different contents depending on the type of user agent (without diversity), for example, “ Embed information such as <meta name = "keywords" content = "devinfo = ANY">".
Next, the operation of the conversion proxy 10 according to this exemplary embodiment will be described with a specific operation example.
FIG. 11 is a diagram illustrating a configuration example of content collected in the present embodiment.
As shown in FIG. 11, a content composed of two web pages (HTML files) called page_1.html and page_2.html is assumed. The page_1.html has a link to page_2.html, and the page_2.html is a web page that is dynamically generated corresponding to two types of user agents, model A and model B.
FIG. 12 is a diagram for explaining processing and data flow in the case where the content information shown in FIG. 11 is collected by the search robot 310.
As shown in FIG. 12, it is assumed that an acquisition request (request 1) for page_1.html is first issued from the search robot 310 to the site 210. This request 1 is sent to the site 210 via the conversion proxy 10. Then, the response 1 to the request 1 is returned from the site 210. The content of the response 1 is page_1.html.
The conversion proxy 10 processes this page_1.html, has a link to page_2.html, and this page_2.html is content depending on the type of user agent (that is, the URL of page_2.html is a diversity URL). Therefore, link expansion processing is performed by the link expansion unit 13. As a result, the response 1 is a response 1 including a web page having a link (page_2_for_A.html) to page_2.html corresponding to the model A and a link (page_2_for_B.html) corresponding to page_2.html corresponding to the model B. Is converted to 'and sent to the search robot 310.
Since the search robot 310 has two links to page_2_for_A.html and page_2_for_B.html on the web page that is the content of the received response 1 ′, the retrieval robot 310 sequentially issues an acquisition request for each.
First, it is assumed that an acquisition request (request 2) for page_2_for_A.html is issued. Using the URL conversion unit 14 and the user agent information editing unit 15, the conversion proxy 10 returns the URL that is the destination of the request 2 to the original URL, and converts the user agent value in the HTTP header into a user agent value indicating the model A. . As a result, the request 2 is sent to the site 210 while pretending to be a request issued from the model A (request 2 ′). In response to this request 2 ′, the site 210 returns a response 2 ′ containing the content page_2.html for the model A.
In the example shown in FIG. 11, since the link to the diversity URL is not included in the page_2.html, the link expansion process for the response 2 ′ is not performed in the conversion proxy 10, and the model A is supported as described above. User agent information for indicating that the response is a meta tag is added with a meta tag, and the response 2 ′ is sent to the search robot 310.
Next, the search robot 310 issues an acquisition request (request 3) for page_2_for_B.html, and a response 3 ′ corresponding to the model B is returned to the search robot 310 through the same processing.
FIG. 13 is a diagram for explaining processing and data flow when link expansion is performed in response to a response from the site 210 to the search robot 310.
As shown in FIG. 13, first, when the transmission / reception unit 11 of the conversion proxy 10 receives a response transmitted from the site 210 (1), a reception notification is sent from the transmission / reception unit 11 to the link expansion unit 13 (2). The expansion unit 13 acquires the response from the transmission / reception unit 11 (3). Then, the link of the body part of the response is searched (4). The obtained link is held in the work area of the main memory 103 in FIG. 5, for example.
The link expansion unit 13 searches the diversity URL database 12 using the link destination URL (original URL) existing in the body part of the response as an argument, and acquires an alternative URL for each user agent corresponding to the original URL. (5). Then, a dummy URL including the acquired alternative URL is generated, and link expansion processing is performed to replace the original URL held in the work area of the main memory 103 with the acquired alternative URL (6). At this time, if there is no link to the content that returns a response corresponding to the type of user agent in the body part of the response (only a link to content that returns a uniform response regardless of the type of user agent) If the URL does not exist, the corresponding URL is not registered in the diversity URL database 12, so the alternative URL is not acquired and the link expansion process is not performed.
Finally, the link expansion unit 13 rewrites the URL (original URL) described in the response body part into a dummy URL group replaced on the work area of the main memory 103 (7), and displays the edited response. It returns to the transmission / reception part 11 (8). Then, the edited response is sent from the transmission / reception unit 11 to the search robot 310.
FIG. 14 is a diagram for explaining processing and data flow in response to a request from the search robot 310 to the site 210.
As shown in FIG. 14, first, when the transmission / reception unit 11 of the conversion proxy 10 receives a request transmitted from the search robot 310 (1), a reception notification is sent from the transmission / reception unit 11 to the URL conversion unit 14 (2). The URL conversion unit 14 acquires the request from the transmission / reception unit 11 (3). Then, the diversity URL database 12 is searched by using the destination URL of the request as an argument (4). If the destination URL is a dummy URL generated by the previous link expansion process, the alternative URL is registered in the diversity URL database 12, and the corresponding original URL is returned.
If the original URL is returned from the diversity URL database 12, the destination URL is converted into the original URL in the work area of the main memory 103 (5). Then, the process proceeds to the user agent information editing unit 15 (6).
Similarly, the user agent information editing unit 15 searches the diversity URL database 12 using the destination URL of the request as an argument (7). Then, the user agent information of the corresponding user agent is acquired, and the editing process for converting the user agent information included in the header information of the request into the acquired user agent information on the work area of the main memory 103 is performed (8). .
Thereafter, the URL conversion unit 14 causes the URL conversion and user agent information conversion performed on the work area of the main memory 103 to be reflected in the header portion of the request received from the search robot 310 (9). The completed request is returned to the transmission / reception unit 11 (10). Then, the processed request is sent from the transmission / reception unit 11 to the site 210.
Through the above operation, the search robot 310 can obtain a response according to the type of each corresponding user agent from the site 210 that responds with different contents according to the type of user agent. In the search server (search site) 300 provided with the search robot 310, index information used for search is created from the content collected in this way and registered in the database. The index information includes information indicating the type of user agent corresponding to the position information of the content and information such as text in the content.
FIG. 15 is a diagram schematically showing the configuration of such a search server 300.
As shown in FIG. 15, the search server 300 includes a search robot 310, a database 320 that stores index information for information search, and a search engine 330 that searches the database 320. Although not particularly illustrated, it is needless to say that a transmission / reception unit for receiving a search request from a terminal device (user agent) via the network and returning a search result by the search engine 330 is provided. As described above, the database 320 stores index information created from content collected by the search robot 310. In the present embodiment, in the conversion proxy 10, the meta tag “ Since the user agent information is embedded using “<meta>”, information such as which index information corresponds to which user agent can be included in the index information.
FIG. 16 is a flowchart for explaining the operation of the search engine 330 when a search request is received from a predetermined terminal device (user agent).
First, a search request is transmitted from the predetermined terminal device to the search server 300. In the HTTP header or the extension header of the search request, user agent information for specifying the type of search term (keyword) and the own device (user agent) is recorded.
As shown in FIG. 16, when a search request from a terminal device is received and read (step 1601), the search engine 330 cuts out a search term (keyword) and user agent information in the input search request (step 1602). . Then, the contents are narrowed down using these pieces of information as constraint conditions, and the database 320 is searched (step 1603). In other words, the search is performed using the search terms by narrowing down to the content including the character string “devinfo = user agent information” or “devinfo = ANY” in the keyword information of the meta tag.
Next, the search engine 330 creates a response page describing the search results in a predetermined format (step 1604), and returns the response page to the terminal device that made the search request (step 1605).
As described above, the terminal device can obtain a search result suitable for the terminal device from the search server 300. This search result is a result of narrowing down and searching for content corresponding to the terminal device out of the content that can be searched by the search server 300, and thus information on the content created for the model is appropriately included. .
As described above, according to the present invention, when information is collected by the search robot, each site type corresponding to the site corresponding to the site that dynamically generates and returns content according to the type of the terminal device is returned. It becomes possible to acquire information related to the contents of the.
FIG. 2 is a diagram showing a configuration in which a conversion proxy according to the present invention is provided in a search server together with a search robot.
FIG. 3 is a diagram showing a configuration in which a conversion proxy according to the present invention is provided in a predetermined proxy server on a network.
FIG. 4 is a diagram showing a configuration in which a conversion proxy according to the present invention is provided in a content server together with a site.
FIG. 5 is a diagram schematically showing an example of a hardware configuration of a computer apparatus suitable for realizing a server provided with a conversion proxy in the present embodiment.
FIG. 6 is a block diagram showing a functional configuration of a conversion proxy according to the present embodiment.
FIG. 7 is a diagram illustrating an example of information registered in a diversity URL database according to the present embodiment.
FIG. 8 is a flowchart for explaining the operation of a link expansion unit according to the present embodiment.
FIG. 9 is a flowchart for explaining the detailed contents of the process of expanding to a dummy link in FIG. 8;
FIG. 10 is a flowchart illustrating an operation performed by a URL conversion unit and a user agent information editing unit according to the present embodiment.
12 is a diagram for explaining processing and data flow when content information shown in FIG. 11 is collected by a search robot; FIG.
FIG. 13 is a diagram for explaining processing and data flow when link expansion is performed in response to a response from a site to a search robot in the present embodiment.
FIG. 14 is a diagram for explaining processing and data flow in response to a request from a search robot to a site in the present embodiment.
FIG. 15 is a diagram schematically showing a configuration of a search server.
FIG. 16 is a flowchart illustrating an operation of a search engine when a search request is received from a predetermined terminal device.
DESCRIPTION OF SYMBOLS 10 ... Conversion proxy, 11 ... Transmission / reception part, 12 ... Diversity URL database, 13 ... Link expansion part, 14 ... URL conversion part, 15 ... User agent information edit part, 210 ... Site, 300 ... Search server, 310 ... Search robot 320 ... database, 330 ... search engine
A transmission / reception unit that relays data exchange between a site on the network and a search robot that accesses the site and collects information;
The transmission / reception unit receives a response from the search robot to the information acquisition request from the search robot to the site, and sends an individual response according to the type of the user agent whose link destination is the access source. A link expansion unit that replaces the link with a plurality of links individually corresponding to the user agent to which the linked content corresponds, in the case of content to be returned;
When the transmission / reception unit receives an information acquisition request from the search robot made based on a specific link among the plurality of links, the user link information included in the information acquisition request is changed to the specific link. And a user agent information editing unit for converting the information into the user agent information corresponding to the linked content.
Further comprising a database in which location information of content that returns an individual response according to the type of user agent that is the access source, and information of the user agent that the content corresponds to and registered.
The link expansion unit specifies the user agent to which the content corresponds by searching the database by using the position information of the link destination of the link included in the response as an argument,
The user agent information editing unit converts the user agent information included in the information acquisition request into the user agent information obtained by searching the database using the position information of the link destination of the specific link as an argument. The server according to claim 1, wherein:
The server according to claim 1, wherein the server is constructed on the same server machine as the server including the search robot.
The server according to claim 1, wherein the server is constructed on the same server machine as the server including the site.
The server according to claim 1, wherein the server is built on a server machine that is separate from the server including the search robot and the server including the site.
A first transmission / reception means for exchanging data with a search robot that accesses a site on the network and collects information;
A second transmitting / receiving means for exchanging data with the site;
When the link destination of the link included in the response from the site received by the second transmission / reception means to the search robot is a content that returns an individual response according to the type of the user agent that is the access source, Analysis means for replacing the link destination information of the link with dummy information including identification information for identifying the user agent corresponding to the linked content, and causing the first transmitting / receiving means to transmit to the search robot ; ,
Another user agent for pretending the user agent information included in the information acquisition request from the search robot to the site received by the first transmission / reception means as an information acquisition request issued from a predetermined user agent Editing means for converting to information ,
When the first transmitting / receiving unit receives an information acquisition request issued from the search robot with the dummy information as a destination, the editing unit returns the dummy information to the original link destination information, and the dummy information Identifying the predetermined user agent based on the identification information included in the information, converting the user agent information of the information acquisition request into the user agent information of the predetermined user agent ,
The server according to claim 2, wherein the second transmitting / receiving unit transmits the information acquisition request obtained by converting the user agent information by the editing unit to the destination site .
A transmission / reception unit that relays data exchange between a website on the network and a search robot that accesses the website and collects information;
The transmission / reception unit receives a response from the website to the information acquisition request from the search robot to the website, and the link destination of the link included in the response is individual according to the type of the user agent that is the access source. In the case of Web content that returns a response, a link expansion unit that replaces the URL (Uniform Resource Locators) of the link included in the response with an alternative URL that individually corresponds to the user agent that the Web content corresponds to When,
A URL conversion unit that replaces the alternative URL with the URL of the original link destination when the transmission / reception unit receives an information acquisition request issued from the search robot with the alternative URL as a destination;
When the transmission / reception unit receives an information acquisition request issued from the search robot with the alternative URL as a destination, user agent information in an HTTP (Hypertext Transfer Protocol) header of the information acquisition request corresponds to the alternative URL. A server comprising: a user agent information editing unit that converts the information into the user agent information.
Corresponds to the URL of the web content that returns an individual response according to the type of the user agent that is the access source, the information of the user agent that the web content corresponds to, and the alternative URL that corresponds to the user agent individually It is further equipped with a registered database,
The link expansion unit searches the database using a link destination URL of a link included in a response from the website in response to an information acquisition request from the search robot to the website, and acquires the alternative URL.
The URL conversion unit acquires the original link destination URL by searching the database using the alternative URL in the information acquisition request issued from the search robot as an argument, with the alternative URL as a destination,
The user agent information editing unit searches the database using the alternative URL in the information acquisition request issued from the search robot with the alternative URL as a destination as an argument to acquire the corresponding user agent information. The server according to claim 7 .
A program for controlling a computer connected to a network to realize a predetermined function,
Transmission / reception means for relaying data exchange between a site on the network and a search robot that accesses the site and collects information;
A response from the site to the information acquisition request from the search robot to the site is received by the transmission / reception means, and an individual response according to the type of user agent whose link destination is the access source Link expansion means for replacing the link with a plurality of links individually corresponding to the user agent to which the link destination content corresponds,
When an information acquisition request from the search robot made based on a specific link of the plurality of links is received by the transmission / reception means, the user agent information included in the information acquisition request is As user agent information editing means for converting the information of the link destination of the link to the information of the user agent that is supported,
A program for causing the computer to function.
A response from the website to the information acquisition request from the search robot to the website is received by the transmission / reception means, and the link destination of the link included in the response is individually determined according to the type of user agent that is the access source Link that replaces the link destination URL (Uniform Resource Locators) of the link included in the response with an alternative URL that individually corresponds to the user agent that the web content supports. Means,
URL conversion means for replacing the alternative URL with the original link destination URL when the transmission / reception means receives an information acquisition request issued from the search robot with the alternative URL as a destination;
When an information acquisition request issued from the search robot with the alternative URL as a destination is received by the transmission / reception means, the user agent information in the HTTP (Hypertext Transfer Protocol) header of the information acquisition request corresponds to the alternative URL As user agent information editing means for converting the information into the user agent information,
The link expansion means realized by the program is stored in a predetermined storage device, and the URL of the web content that returns an individual response according to the type of the user agent that is the access source corresponds to the web content. Included in the response from the website to the information acquisition request from the search robot to the database in which the information on the user agent and the alternative URL individually corresponding to the user agent are registered in association with each other Search for the link destination URL of the link to be obtained as an argument to obtain the alternative URL,
The URL converting means realized by the program searches the database using the alternative URL in the information acquisition request issued from the search robot as an argument and acquires the original link destination URL. And
The user agent information editing means realized by the program searches the database with the alternative URL in the information acquisition request issued from the search robot as an argument with the alternative URL as a destination and searches for the corresponding user agent. Information is acquired, The program of Claim 10 characterized by the above-mentioned.
JP2003047983A 2003-02-25 2003-02-25 Server and program Expired - Fee Related JP4007596B2 (en)
US10/751,767 US7536445B2 (en) 2003-02-25 2004-01-05 Enabling a web-crawling robot to collect information from web sites that tailor information content to the capabilities of accessing devices
JP2004258911A JP2004258911A (en) 2004-09-16
JP4007596B2 true JP4007596B2 (en) 2007-11-14
JP2003047983A Expired - Fee Related JP4007596B2 (en) 2003-02-25 2003-02-25 Server and program
JP5705114B2 (en) * 2009-07-28 2015-04-22 インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ Information processing apparatus, information processing method, program, and web system
JP5602678B2 (en) * 2011-05-26 2014-10-08 日本電信電話株式会社 WEB information prefetching method and prefetching proxy server device
JP3714879B2 (en) 2001-02-27 2005-11-09 ヤフー株式会社 WEB search service system and method
US7536445B2 (en) 2009-05-19
2007-08-23 RD14 Notification of resignation of power of sub attorney