Steps after a user finds an attractive product until he or she buys it are successively performed so that he or she can buy it stressfreely. The user photographs an attractive product with a portable camera telephone terminal 1 and transmits image data 7 to a server 3 through Internet 2. The server 3 has a DB 4 in which images such as various types of products, persons, and logo marks are registered and a DB 5 in which related information of image data registered in the DB 4 is registered. The server 3 performs an image recognizing process for the image data 7, retrieves data that match the image data 7 with a high match rate from the image data registered in the DB 4, and extracts corresponding related information from the DB 5. The related information is provided to the user through the Internet 2. By only transmitting the image data 7, the user can obtain related information of an attractive product. When the user accesses a site 6 for the product corresponding to the related information, he or she can obtain detailed information and buy the product.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Japanese Patent Document Nos. P2004-020852 filed on Jan. 29, 2004, the disclosure of which is herein incorporated by reference.

BACKGROUND

The present invention relates to an information retrieving system, an information retrieving method, an information retrieving apparatus, an information retrieving program, an image recognizing apparatus, an image recognizing method, an image recognizing program, and a sales system that allow a user to retrieve his or her interested information and for example easily order a product according to the retrieved information.

It is assumed that a user found his or her interested product in advertisement of print media such as posters and magazines and advertisement of broadcast media such as commercials of television broadcasts and radio broadcasts. At this point, if the advertisement contains store information indicating a store that the user can buy the product, he or she can directly go to the store and buy the product at the store. In contrast, if the advertisement does not contain store information indicating a store that he or she can buy the product, he or she needs to estimate a store that he or she can buy the product and look for it.

In recent years, networks such as the Internet have been outspread. The user can easily search for detailed information about his or her desired product and store information indicating a store that he or she can buy product from a network. The user accesses a search site on the Internet with for example a personal computer and retrieves desired information with a keyword that he or she thinks that it indicates the product. If the retrieved information contains store information that indicates a store that he or she can buy the product, he or she can go to the store and buy the product at the store.

EC (Electronic Commerce) has emerged as the Internet has grown. Thus, the user can order his or her favorite product and pay the money for the product through the Internet according to the retrieved result from the information search site on the Internet without need to actually go to the store. As a result, the user can obtain his or her favorite product at home.

Moreover, in recent years, portable telephone terminals having an Internet connection function have been widespread. In addition, portable telephone terminals having a camera function (hereinafter sometimes referred to as portable camera telephone terminals) have been widespread. The user can easily photograph an object and transmit image data of the photographed object to the Internet. In the circumstance, a method of obtaining store information indicating a store that a user can buy his or her desired product according to image data that he or she photographed with a camera has been proposed in Japanese Patent Application Unexamined Publication No. 2003-122757.

In this method, the user photographs his or her desired product with a portable camera telephone terminal and transmits the photographed image to a server through the Internet. The server analyzes the received photographed image, obtains feature amount information, retrieves detailed information of the product, store location information of the product, and so forth from a database according to the obtained feature amount information, outputs them, and returns them to user's portable camera telephone terminal. Even if the user does not know the name of the product, he or she can obtain information necessary for ordering the product.

With respect to information retrieval for contents of a television broadcast and a radio broadcast that are on the air, a method of receiving a television broadcast with an Internet connection type television receiver and retrieving data transmitted through the Internet corresponding to the received television broadcast has been practically implemented for a digital BS broadcast and a terrestrial digital television broadcast.

For a regular television receiver, a system that uses a personal computer or a portable telephone terminal having an Internet communication function has been practically implemented. In this system, a user transmits time information indicating time at which his or her interested program or a commercial was on the air to a server through a network such as the Internet with the personal computer or the portable telephone terminal having the Internet communication function. The server side retrieves information from a prepared database that stores related information (program name, commercial information, performers, song names, artist names, and so forth) of broadcast contents at each time of each broadcasting station according to the received time information, and returns the retrieved result to the user.

In the method of which the user sees advertisement and goes to a store according to information in the advertisement and he or she retrieves his or her desired product from a search site on the Internet (hereinafter this method is referred to as the first method), in many cases, the user needs to remember information about a product as character information such as the name of the product. If the user does not know the character information, it is very difficult to obtain necessary information with which he or she can buy his or her desired product.

Now, it is assumed that the user saw a poster of a newly released movie and wanted to know a movie theater and show times at which it performs the movie. Conventionally, the user needs to memorize or take a note of character information such as the title of the movie on the poster and retrieve necessary information from an information search site according to the character information. In this case, if the user retrieves the necessary information according to information he or she memorizes, he or she may not retrieve the necessary information or may retrieve an improper result because he or she mistakenly memorizes character information or forgot them. If the user takes a note of the character information, he or she needs to take a time to prepare a writing tool and a notebook.

When the user found his or her favorite shoes that a person in advertisement of a magazine worn, if the user does not know the maker and the model of the shoes, he or she cannot obtain even character information as a keyword with which he or she retrieve information from the information search site.

When the user buys wine, he or she may not determine whether to buy it without necessary information about what taste the win has, where it was produced, and what grape species was used for the wine. If character information on the label of the wine bottle is written in for example French, it is difficult for a non-French-speaking user to input a keyword in an input field in the foregoing conventional retrieving method. Moreover, in the conventional method, it is difficult to satisfy needs of the user to quickly obtain information on the spot.

FIG. 1AandFIG. 1Bshow an example of a procedure of retrieving information and buying a product according to the first method. First, a user obtains information about a product from so-called push media such as paper media, a television broadcast, or a radio broadcast (at step S100). In this stage, the user is interested in the product in an “attractive” level. The user checks detailed information about the product such as the price and specifications with so-called pull media such as paper media e.g. an information magazine or an information search site on the Internet (at step S101).

When the user has obtained the detailed information about the product, he or she checks the information. If the user has obtained information about a plurality of stores for the product, he or she compares their prices (at step S102) and checks the location of the store that he or she can buy the product (at step S103). When the store is a real store, the user checks the location of the store on a map. If the store supports the EC and is a store as a web site on the Internet, the user checks the URL (Uniform Resource Locator) of the store. The user determines whether to actually buy the product according to his or her budget and the priority of the product in a product list (wish list) (at step S104). When the user has decided to buy the product, he or she performs the buying process for it (at step S105).

When the user buys a product according to the first method, he or she needs to retrieve information and make a decision at each step from S100to S105. Thus, the method involves the user's effort. Further, since these steps are incontinuous and independent processes, it takes time to buy the product after the user finds an attractive product. Thus, before the user buys the product, he or she may be tired, which may discourage the user from buying the product. This means that the sales side may lose a business chance.

In contrast, in the method described in Japanese Patent Application Unexamined Publication No. 2003-122757, where the user photographs his or her desired product with the portable camera telephone terminal, transmits the photographed image to the server, and obtains product information, (hereinafter this method is referred to as the second method), there is a problem of which the user needs to actually photograph the product for example his or her acquaintance has the product and he or she can photograph it or a show window exhibits the product. Of course, an image of advertisement of paper media and a commercial of a television broadcast may be used. In this case, the product needs to appear independently or nearly independently. Thus, the second method also has many restrictions.

On the other hand, in the method using the network connection type television receiver (hereinafter this method is referred to as the third method), there is a problem of which a dedicated television receiver is required. In addition, since related information is broadcast using a data channel, when a recorded television broadcast is reproduced, a unit that records and reproduces the television broadcast needs to correspond to recording of the data channel. In addition, the user can retrieve only information that the broadcasting station has prepared. Moreover, since the related information is broadcast through the data channel, the related information that was recorded is not updated. Thus, when the recorded information is reproduced, it may become old and useless.

On the other hand, in the method of which while the user is seeing a television broadcast received by a regular television receiver, he or she transmits time information to a server using a personal computer or a portable telephone terminal and receives information from the server (this method is referred to as the fourth method), while he or she is accessing the server using the personal computer or the portable telephone terminal, the desired image may disappear. In addition, the user needs to register information about the broadcasting station that he or she sees to the server in advance. Thus, the user needs to take time. In addition, when the user wants to obtain related information from a reproduced image of a recorded broadcast, it is necessary to record information about the broadcasting station and broadcasting time of the image along with the image in advance. Thus, the user needs to prepare a special recording unit.

On the other hand, when the product related information retrieving and buying system is used through the Internet, for example the following services have been performed.

(1) A service of which a related information provider provides information to a pay user who retrieves the information.

(2) A business that publishes information that a particular company requests and gets compensation for an advertisement effect of the disclosed information from the particular company regardless of whether the disclosed information is concerned with related information that a user requests.

(3) A business that licenses a company to use a retrieving system and related databases and gains compensation from the licensee company.

(4) A business that introduces a user who has retrieved information and who wants to buy a product according to the retrieved information to a mail order company and gets a commission from the mail order company.

(5) A business of which a user who retrieves information sells a product to another user.

However, conventionally, there are many problems as described above. Thus, these services and businesses cannot be smoothly carried out. As a result, many companies and users lose their business chances.

SUMMARY

Therefore, an object of the present invention is to provide an information retrieving system, an information retrieving method, an information retrieving apparatus, an information retrieving program, an image recognizing apparatus, an image recognizing method, an image recognizing program, and a sales system that allow steps after a user finds an attractive product until he or she buys it to be successively performed so that he or she can buy it stressfreely.

To solve the foregoing problem, the present invention in an embodiment is an information retrieving system, which retrieves information according to image data, comprising a terminal unit having image data obtaining means for obtaining image data; transmitting means for transmitting the image data obtained by the image data obtaining means to a communication network, and a server unit having a first database in which retrievable recognizing image data are registered; a second database in which meta data corresponding to the recognizing image data registered to the first database are registered; receiving means for receiving the image data transmitted from the terminal unit through the network; retrieving means for retrieving the recognizing image data that match an image element of the image data with a high match rate from the first database according to the image data received by the receiving means; and meta data extracting means for extracting the meta data from the second database according to a retrieved result of the retrieving means, wherein the server unit allows the terminal unit to browse the meta data extracted by the meta data extracting means through the communication network.

In addition, the present invention in an embodiment is an information retrieving method of retrieving information according to image data, comprising the steps of causing a terminal unit to obtain image data; transmitting the image data obtained at the image data obtaining step from the terminal unit to a communication network; receiving the image data transmitted at the transmitting step through the network; retrieving recognizing image data that match an image element of the image data with a high match rate from a first database according to the image data received at the receiving step, retrievable recognizing image data being registered in the first database; and extracting meta data from a second database according to a retrieved result at the retrieving step, the meta data corresponding to the recognizing image data registered in the first database being registered in the second database, wherein the terminal unit can browse the meta data extracted at the meta data extracting step through the communication network.

In addition, the present invention in an embodiment is an information retrieval apparatus which retrieves information according to image data, comprising a first database in which retrievable recognizing image data are registered; a second database in which meta data corresponding to the recognizing image data registered to the first database are registered; receiving means for receiving image data transmitted from a terminal unit through a communication network; retrieving means for retrieving the recognizing image data that match an image element of the image data with a high match rate from the first database according to the image data received by the receiving means; and meta data extracting means for extracting the meta data from the second database according to a retrieved result of the retrieving means, wherein the terminal unit can browse the meta data extracted by the meta data extracting means through the communication network.

In addition, the present invention in an embodiment is an information retrieval method of retrieving information according to image data, comprising the steps of receiving image data transmitted from a terminal unit through a communication network; retrieving recognizing image data that match an image element of the image data with a high match rate from a first database according to the image data received at the receiving step, retrievable recognizing image data being registered in the first database; and extracting meta data from a second database according to a retrieved result at the retrieving step, the meta data corresponding to the recognizing image data registered to the first database being registered in the second database, wherein the terminal unit can browse the meta data extracted at the meta data extracting step through the communication network.

In addition, the present invention in an embodiment is an information retrieving program which causes a computer unit to execute an information retrieving method of retrieving information according to image data, the information retrieving method comprising the steps of receiving image data transmitted from a terminal unit through a communication network; retrieving recognizing image data that match an image element of the image data with a high match rate from a first database according to the image data received at the receiving step, retrievable recognizing image data being registered in the first database; and extracting meta data from a second database according to a retrieved result at the retrieving step, the meta data corresponding to the recognizing image data registered to the first database being registered in the second database, wherein the terminal unit can browse the meta data extracted at the meta data extracting step through the communication network.

In addition, the present invention in an embodiment is an image recognizing apparatus which recognizes a predetermined shape from image data, comprising a first database in which retrievable recognizing image data are registered; a second database in which meta data of the recognizing image data registered to the first database are registered; retrieving means for retrieving the recognizing image data that match an image element of the image data with a high match rate from the first database according to input image data; and meta data extracting means for extracting the meta data from the second database according to a retrieved result of the retrieving means.

In addition, the present invention in an embodiment is an image recognizing method of recognizing a predetermined shape from image data, comprising the steps of retrieving recognizing image data that match an image element of input image data with a high match rate from a first database according to the input image data, the retrievable recognizing image data being registered in the first database; and extracting meta data from a second database according to a retrieved result at the retrieving step, the meta data of the recognizing image data registered to the first database being registered in the second database.

In addition, the present invention in an embodiment is an image recognizing program which causes a computer unit to execute an image recognizing method of recognizing a predetermined shape from image data, comprising the steps of retrieving recognizing image data that match an image element of input image data with a high match rate from a first database according to the input image data, the retrievable recognizing image data being registered in the first database; and extracting meta data from a second database according to a retrieved result at the retrieving step, the meta data of the recognizing image data registered to the first database being registered in the second database.

In addition, the present invention in an embodiment is a sales system, comprising a terminal unit which obtains image data and transmits them to a communication network; a server unit which can communicate with the communication network and outputs meta data of image data according to the image data; and a product sales section which can provide related information of the meta data and/or a product, wherein the server unit charges the terminal unit, outputs the meta data according to the image data transmitted from the terminal unit so that it can brose the meta data, and introduces the terminal unit as a customer to the product sales section, wherein the product sales section provides related information that the terminal unit can browse and/or a product to the terminal unit introduced as the customer, and wherein the terminal unit pays the product sales section for the provided information and/or product.

In addition, the present invention in an embodiment is a sales system, comprising a terminal unit which obtains image data and transmits them to a communication network; a server unit which can communicate with the communication network and outputs meta data of image data according to the image data; and a product sales section which can provide related information of the meta data and/or a product, wherein the server unit outputs the meta data according to the image data transmitted from the terminal unit so that it can brose the meta data, wherein the terminal unit pays the server unit for related information of the meta data that the terminal unit can brose and/or a product, wherein the server unit pays the product sales section for a part of the related information and/or the product, and wherein the product sales section provides the related information and/or the product to the terminal unit.

In addition, the present invention in an embodiment is a sales system, comprising a terminal unit which obtains image data and transmits them to a communication network; a server unit which can communicate with the communication network and outputs meta data of image data according to the image data; and a product sales section which can provide related information of the meta data and/or a product, wherein the terminal unit transmits the image data to the product sales section, wherein the product sales section obtains meta data according to the image data transmitted from the terminal unit through the server unit and pays the server unit for a rental fee, wherein the product sales section allows the terminal unit to browse the meta data corresponding to the image data transmitted from the terminal unit through the server unit, and wherein the terminal unit pays the product sales section for related information of the meta data that the terminal unit can browse and/or a product.

As was described above, according to an embodiment of the present invention, image data transmitted from a terminal unit are received through a communication network. Recognizing image data that match an image element of the image data with a high match rate are retrieved from a first database according to the received image data. Retrievable recognizing image data are registered in the first database. Meta data are extracted from a second database according to a retrieved result. The meta data corresponding to the recognizing image data registered to the first database are registered in the second database. The terminal unit can browse the extracted meta data through the communication network.

Thus, by only transmitting image data from the terminal unit, the user can browse meta data corresponding to the transmitted image data on the terminal unit. Thus, the user can quickly obtain necessary information without need to have the knowledge of the object on the spot.

Moreover, according to an embodiment of the present invention, recognizing image data that match an image element of input image data with a high match rate are retrieved from a first database according to the input image data. The retrievable recognizing image data are registered in the first database. Meta data are extracted from a second database according to a retrieved result. The meta data of the recognizing image data registered to the first database are registered in the second database. Thus, when image data are input, meta data corresponding to the input image data can be obtained. Thus, the user can obtain related information of the image data without need to have the knowledge of the image data.

Moreover, according to an embodiment the present invention, the sales system has a terminal unit which obtains image data and transmits them to a communication network, a server unit which can communicate with the communication network and outputs meta data of image data according thereto, and a product sales section which can provide related information of the meta data and/or a product. The server unit charges the terminal unit, outputs the meta data according to the image data transmitted from the terminal unit so that it can brose the meta data, and introduces the terminal unit as a customer to the product sales section. The product sales section provides related information that the terminal unit can browse and/or a product to the terminal unit introduced as the customer. The terminal unit pays the product sales section for the provided information and/or product. Thus, when the user of the terminal unit pays the server unit for a fee, he or she can transmit image data photographed by the terminal unit to the server unit and obtain a product and information corresponding to the image data. The server unit can obtain a fee from the user. The product sales section can obtain a fee for a product and information from a customer introduced by the server unit. In addition, the user can obtain the product and information without need to have the knowledge of the photographed product and information. The product sales section can obtain a new customer who does not have the knowledge of the product and information the product sales section sells.

Moreover, according to an embodiment of the present invention, the sales system has a terminal unit which obtains image data and transmits them to a communication network, a server unit which can communicate with the communication network and outputs meta data of image data according thereto, and a product sales section which can provide related information of the meta data and/or a product. The server unit outputs the meta data according to the image data transmitted from the terminal unit so that it can brose the meta data. The terminal unit pays the server unit for related information of the meta data that the terminal unit can brose and/or a product. The server unit pays the product sales section for a part of the related information and/or the product. The product sales section provides the terminal unit with the related information and/or the product. Thus, when the user of the terminal unit pays the server unit for a fee, he or she can transmit image data photographed by the terminal unit to the server unit and obtain a product and information relating to the image data. The server unit can obtain a part of the fee of the product and information that the server unit has provided to the user. The product sales section can obtain the rest of the fee of the product and the information that the product sales section has provided to the user. In addition, the user can obtain the product and information without the knowledge of the photographed product and information. The product sales section can obtain a new customer who does not have the knowledge of the product and information the product sales section sells.

Moreover, according to an embodiment of the present invention, the sales system has a terminal unit which obtains image data and transmits them to a communication network, a server unit which can communicate with the communication network and outputs meta data of image data according thereto, and a product sales section which can provide related information of the meta data and/or a product. The terminal unit transmits the image data to the product sales section. The product sales section obtains meta data corresponding to the image data transmitted from the terminal unit through the server unit and pays the server unit for a rental fee. The product sales section allows the terminal unit to browse the meta data corresponding to the image data transmitted from the terminal unit through the server unit. The terminal unit pays the product sales section for related information of the meta data that the terminal unit can browse and/or a product. Thus, when the user of the terminal unit pays the server unit for a fee, he or she can transmit image data photographed by the terminal unit to the product sales section and obtain a product and information relating to the image data. The product sales section can obtain a fee for the product and information that the product sales section has provided to the user. The server unit can obtain a function rental fee from the product sales section. In addition, the user can obtain the product and information without the knowledge of the photographed product and information. The product sales section can obtain a new customer who does not have the knowledge of the product and information the product sales section sells.

DETAILED DESCRIPTION

Next, an embodiment of the present invention will be described. Before the present invention is described, a service that the applicant of the present invention is carrying out will be described in brief.

In this service, related information of songs and commercials that will be aired in television broadcasts and radio broadcasts are correlated with broadcast time information and registered on the server side in advance. When an attractive song or commercial is aired in a television broadcast or a radio broadcast that a user is watching or listening to outdoor or at home, he or she transmits time information to the server using for example a portable telephone terminal having an Internet communication function. The server retrieves related information about the song or commercial that was aired according to the received time information and returns the retrieved result to him or her. Thus, the user can easily obtain the related information about an attractive song and commercial. In addition, the server side can promote a business associated with song and commercial information that the server side has provided to the user.

The present invention aims to apply the on-going service to image data and provide a successive buying process to the user. In other words, in the first method described in the related art section, step S100to step S105after the user obtains information until he or she buys a product are not successive as shown inFIG. 1AandFIG. 1B. In contrast, as shown inFIG. 2AandFIG. 2B, according to the present invention, steps of obtaining information (at step S1), retrieving information (at step S2), checking information and comparing prices (at step S3), retrieving store or site that deals with the product (at step S4), checking budget (at step S5), and buying or downloading the product (at step S6) are performed as successive processes.

FIG. 3shows an example of a structure of a system according to the embodiment of the present invention. When the user found attractive information, he or she photographs it with for example a portable camera telephone terminal1. For example, it is assumed that the user found that a cover of a magazine “ABCD” had a photo of an attractive product. The user photographs the cover of the magazine ABCD with the portable camera telephone terminal1and transmits the obtained image data7to a server3through the Internet2.

The server3is a unit that has the same structure as for example an ordinary computer unit. The server3has an image data analyzing function. In addition, the server3has an image database4and a related information database5. Various types of image data such as images of various types of products, images of persons, images of logo marks, and images indicating character information are registered in the image database4. Hereinafter, images registered in the image database4are referred to as registered images. Related information of images registered in the image database4are registered as meta data of the images in the related information database5.

The server3is not limited to a unit composed of a single computer unit. Instead, the server3may be composed of a plurality of computers connected through a network. Functions of the server3are executed by a program installed therein. The program is supplied through a record medium such as a CD-ROM (Compact Disc-Read Only Memory) or a network.

When the server3receives the image data7, the server3performs an image recognizing process for the received image data7. In the image recognizing process, the server3checks whether an image that matches or is very similar to an image element contained in the image data7has been registered in the image database4. When the image has been registered in the image database4, the server3retrieves related information corresponding to the registered image of the image database4from the related information database5. The server3transmits information according to the related information as the retrieved result to the portable camera telephone terminal1through the Internet2.

An image element is a meaningful element that composes one image. Assuming that one image is composed of a person who wears a cap and an image of a logo mark, an image of the cap, an image of the person, and the image of the logo mark are image elements of the image.

Related information registered in the related information database5may contain the URL (Uniform Resource Locator) of a related site6on the Internet2. The user can access the related site6with the URL contained in the related information transmitted from the server3. When the related site6supports the EC and the user can buy a product through the Internet2, he or she can access the related site6according to the URL contained in the related information transmitted from the server3and buy the product from the related site6.

Next, the structure of the system according to the embodiment of the present invention will be described in detail.FIG. 4AandFIG. 4Bshow an example of a structure of a user side system and a server side system. On the user side, the portable camera telephone terminal1has a camera section10, a display section11, and an interface (I/F) section12. The camera section10uses for example a CCD (Charge Coupled Device) as an image device. The camera section10converts a photographed signal into image data in a predetermined manner and outputs the image data. The image data are supplied to the display section11that uses for example an LCD (Liquid Crystal Display) as a display device.

In addition, the image data, which are output from the camera section10, are gray-scaled and supplied to the interface section12. The image data can be gray-scaled for example by extracting only a luminance component from the image data.

The interface section12controls communication between the portable camera telephone terminal1and the Internet2according to a predetermined protocol. The image data supplied to the interface section12are transmitted as image data7to the server3through the Internet2. At this point, it is preferred that ID information that identifies the user be transmitted along with the image data.

As denoted by a dotted line block in the drawing, a feature extraction processing section13may be disposed so as to extract a feature from the image data that are output from the camera section10in a predetermined manner and supply only the extracted feature data to the interface section12. In this case, only the feature data are transmitted to the Internet2.

On the user side, means that obtains image data and transmits them is not limited to the portable camera telephone terminal1. Instead, as long as an image supplied from the information provider side can be obtained, any means may bee used.

For example, when image data photographed by a digital camera1B are transferred to a personal computer (PC)15and then the personal computer15communicates with the Internet2, image data7can be transmitted to the server3. In this case, a feature extracting section16pre-extracts a feature from image data photographed by the digital camera1B and transmits only the extracted feature data as image data7to the server3. The feature extracting section16may be composed of software on the personal computer15.

In addition, image data of a digital television broadcast received by for example a digital television receiver1C can be transmitted as image data7to the server3. A digital television broadcast signal received by an antenna20is processed by a receiving section21in a predetermined manner. As a result, a digital video signal is obtained. The digital video signal is displayed on a displaying section22. In addition, frames for a predetermined period for which the user sets an operating section (not shown) or one particular frame is extracted from the digital video signal. An interface section23transmits the extracted frames or frame as image data7to the server3through the Internet2.

Like the portable camera telephone terminal1, the digital television receiver1C may have a feature extraction processing section24as denoted by a dotted line block in the drawing so as to pre-extract a feature of image data that are output from the receiving section21and transmit only the feature data as image data7to the server3.

In addition, a digital video signal received by a digital television receiver25and recorded by a video recorder26can be transmitted as image data7to the server3in the same manner. The video recorder26reproduces frames in a particular period or one particular frame of the digital video signal. When the frames or frame reproduced by the video recorder26is transferred to for example a personal computer27and then the personal computer27communicates with the Internet2, the frames or frame can be transmitted as image data7to the server3.

In this case, a feature extracting section28pre-extracts a feature from the frames or frame that are output from the video recorder26and transmits only the extracted feature data as image data7to the server3. The feature extracting section28may be composed of software on the personal computer27.

In stead, on the user side, means that obtains the image data and transmits them may be a home robot having an image recognizing function.

In any of the foregoing examples, gray-scaled image data are transmitted as image data7to the server3. The user side extracts a feature from image data that has been gray-scaled.

In the foregoing examples, the digital television receiver1C receives a television broadcast as a digital television broadcast. Of course, an analog television broadcast signal that an analog television receiver receives as an analog television broadcast may be converted into a digital signal.

Next, the server side system will be described. The server side system has the server3, an image database4, a related information database5, and a data registering section. Interface sections40and41control communication between the server3and the Internet2according to a predetermined protocol in a predetermined manner. The interface section40receives image data transmitted from the user side through the Internet2and supplies the image data to the server3.

The data registering section collects commercials extracted by an automatic CM (commercial) extracting system51from television broadcasts of individual channels, information published in regular web sites52on the Internet2, and so forth. A data registering tool50registers images to the image database4. When images are registered to the image database4, the images are assigned unique image IDs. The data registering tool50is software structured so that data are effectively registered to the image database4and the related information database5.

Images that the user may retrieve are pre-registered to the image database4. For example images of logo marks, partial or whole images of articles and buildings, images of persons, and so forth are registered to the image database4. Character information is also registered to the image database4so that character strings can be analyzed.

In addition, the data registering section collects related information that relates to images registered to the image database4from the automatic CM extracting system51and the general web sites52. The data registering tool50correlates the collected related information, the images, and the image IDs and register the correlated data to the related information database5.

The server3has an image feature extracting section30, an image feature comparing section31, a related information retrieving section32, and a related information outputting section33. The image data7from which a feature has not been extracted on the user side are received by the interface section40and supplied to the image feature extracting section30. In contrast, the image data7from which a feature has been extracted on the user side are received by the interface section40and supplied to the image feature comparing section31. The feature extracting sections13,16,24, and28on the user side perform the same feature extracting process as the image feature extracting section30on the server side.

The image feature extracting section30extracts a feature from the supplied image data7and outputs the obtained feature data. The image feature extracting section30may perform the feature extracting process in various manners. For example, the image feature extracting section30may perform an edge detecting process for the image data in such a manner that the image feature extracting section30extracts sharp edge portions as feature points and uses the relationship of the positions of the extracted feature points as feature data.

The feature data are supplied to the image feature comparing section31. The image feature comparing section31extracts a feature from an image registered in the image database4and compares the extracted feature data with the feature data supplied from the image feature extracting section30. The image feature comparing section31retrieves an image from the image database4according to the compared result in such a manner that the image matches the whole or a part of the feature data supplied from the image feature extracting section30with a high much rate. At this point, if the image feature comparing section31has retrieved a plurality of images that have a higher match rate, the image feature comparing section31outputs the image IDs of the registered images.

The image ID that is output from the image feature comparing section31is supplied to the related information retrieving section32. The related information retrieving section32retrieves related information from the related information database5according to the supplied image ID. The related information corresponding to the image identified by the image ID is supplied to the related information outputting section33.

The related information outputting section33converts related information supplied from the related information retrieving section32into data so that the user can browse them. The converted meta data are supplied to the interface section41and returned to the user through the Internet2. For example, when the image data7is transmitted from the portable camera telephone terminal1to the server, related information of the image data7is returned to the portable camera telephone terminal1through the Internet2.

In the foregoing example, the related information outputting section33transmits related information to the user. However, the present invention is not limited to such an example. For example, the related information outputting section33may create a web site that publishes related information supplied from the related information retrieving section32to the user through the Internet2. The server3informs the user (portable camera telephone terminal1) of the URL of the created web site so as to present the web site to him or her. When the user who received the information accesses the URL, he or she can browse the related information of the image data7that he or she transmitted.

In addition, in the foregoing, the image feature extracting section30performs the feature extracting process of gray-scaled image data. However, the present invention is not limited to such an example. In other words, the image feature extracting section30may perform a feature extracting process for image data containing color information. In this case, the user side transmits image data that are not gray-scaled to the server3. When the user side performs the feature extracting process using the feature extracting section13,16,24, or28, it can perform the feature extracting process for image data containing color information. In addition, when the user side performs the feature extracting process, it can perform the feature extracting process for gray-scaled data. When the server side3performs the feature extracting process, it may perform the feature extracting process for image data containing color information.

Next, data registration to the image database4and the related information database5will be described. First, with reference toFIG. 5AandFIG. 5B, a method of collecting image data and related information according to a commercial of a television broadcast will be described. The automatic CM extracting system51has a plurality of tuners511,511, . . . , corresponding to individual channels of television broadcasts and a CM detector512. Television broadcasts received by the tuners511,511, . . . are supplied to the CM detector512. The CM detector512automatically detects and extracts commercials from the supplied television broadcasts. A commercial of a television broadcast can be detected from the difference between audio data of a non-commercial broadcast program and that of a commercial, a scene change, a program guide, and so forth.

The automatic CM extracting system51creates a list of new commercials aired on the day according to the detected commercials. The list of new commercials is supplied to a data center60along with image data (video data) of new commercials detected on the day by the automatic CM extracting system51. The automatic CM extracting system51also extracts program guides of broadcasting stations, event announcements from the television broadcasts, and so forth and supplies the extracted guides, announcements, and so forth to the data center60.

The data center60checks related information about individual commercials according to the list of new commercials and video data of the new commercials supplied from the automatic CM extracting system51. For example, the data center60checks a sponsor, a sales representative, a production company, a music production company as a copyright owner for a song that is used for a commercial, a related web site, a general web site that publishes related information, and so forth for each commercial. The checked results are registered as related information of commercials to a commercial (CM) information database61along with video data.

In the example shown inFIG. 5AandFIG. 5B, a first broadcast time, a broadcasting station, a commercial duration, and version information are registered as related information (meta data) of a broadcast to the CM information database61. As information about a sponsor, a company name, a company category, and a URL of a company's web site are registered. As information about a product, a product name, a product category, and product information (or a URL of a related web site that publishes product information) are registered to the CM information database61. As information about a performer, a performer's name and his or her word are registered to the CM information database61. As information about direction of a picture, a catch copy, a scene, a feature of a picture, and so forth are registered to the CM information database61. As information about a song, a song name, a performer, a song writer, a composer, and if the song is recorded on a CD, a CD number are registered to the CM information database61.

A match retrieving file is registered to the CM information database61. Features of images and audio of video data supplied from the data center60are extracted. As a result, feature data files are obtained. Feature data files of images and sound are correlated with commercials and registered as a match retrieving file.

Each unit of information registered in the CM information database61is registered in the related information database5. Recorded video data of commercials are registered to the image database4. It is preferred that the feature data file of image data be registered to the image database4because image data can be more easily retrieved.

Next, with reference toFIG. 6AandFIG. 6B, a method of collecting image data and related information from media other than television broadcasts will be described. In this example, it is assumed that information sources are web sites on the Internet. For example, web sites that publish various types of advertisement messages, various news sites, various types of information sites, web sites of newspaper companies, web sites of publishing companies (in particular, web sites related to magazines they publish), related web sites, general web sites may be information sources. Instead, paper media such as various types of magazines and newspapers may be information sources.

The data center60observes these web sites for example on daily basis. When the data center60found a new product, a topical product, an event, and so forth that were released on the day, the data center60creates a list of information that it found. The data center60inquires of information sources and sales representatives detailed information and checks related web sites and general web sites that publish the related information according to the list. The data center60registers information to a product (content) information database62according to the inquired results and checked results.

In the example shown inFIG. 6AandFIG. 6B, as related information (meta data) of an original seller, an original seller company, its category, and its logo mark image are registered to the product (content) information database62. As related information of a sales period and a sales region, a sales start date and a sales area are registered to the product (content) information database62. As related information of a selling store, a selling store name, a category of the selling store, and a URL of a web site of the selling store are registered to the product (content) information database62. As related information of a product (content), a product name, a category of the product, a product number, a JAN code, information about a related product, and a URL of a web site of the relevant product are registered to the product (content) information database62. As related information of a price, a sales price is registered to the product (content) information database62.

An image of each company's log mark, image data of each product, and related image data of each product are registered to the product (content) information database62.

Each unit of information registered in the product (content) information database62is registered to the related information database5. Each image data registered in the product (content) information database62are registered to the image database4. It is preferred that a feature data file as a feature extracted from each image be registered to the image database4because image data can be easily retrieved.

The structures of the foregoing CM information database61and the product (content) information database62are just examples. Thus, items registered to these databases are not limited to the foregoing examples. In addition, in the foregoing examples, the CM information database61, the product (content) information database62, the image database4, and the related information database5are described as independent databases. However, the structures of these databases are not limited to these examples. Instead, these four databases may be integrated into one database.

Next, with reference toFIG. 7, a process that recognizes image data transmitted from the user and extracting meta data therefrom will be described. InFIG. 7, an image recognizing application71and a meta data extracting application73are application programs that are executable on the server3.

The user transmits image data70,70, . . . to the server3. In the server3, the image recognizing application71performs a feature extracting process for the image data70,70, . . . transmitted from the user and extracts feature data. The image recognizing application71retrieves from an image data master72(corresponding to the image database4) an image that matches or is very similar to the feature data extracted from the image data70and outputs an image ID as a retrieved result. Character information may be output as text data. If the image data70contain a plurality of image elements such as a person and a product, the image recognizing application71outputs an image ID of an image that matches or is very similar to each of the plurality of image elements. The image recognizing application71supplies the image ID to the meta data extracting application73.

The meta data extracting application73retrieves meta data from the meta data master74corresponding to the received ID and outputs meta data75of the match image. When the meta data extracting application73receives a plurality of image IDs, the meta data extracting application73retrieves meta data from the meta data master74corresponding to the plurality of image IDs and outputs meta data75,75, . . . of the match images. The meta data master74corresponds to the related information database5. The meta data correspond to related information.

Next, with reference toFIG. 8AandFIG. 8B, the image recognizing process and the meta data extracting process will be described in more detail. When moving picture data70′ composed of a plurality of frames have been transmitted from the user, the moving picture data70′ are processed as image data70,70, . . . of individual frames. At this point, when necessary, frames may be partly removed from the moving picture data70′ in a predetermined manner so as to obtain image data70,70, . . . for individual frames.

In this example, image elements contained in the image data70can be categorized as three types that are different in characteristics, namely a shape of a general article, a person (physiognomy), and characters. The image recognizing application71has a pattern recognizing engine71A, a person recognizing engine71B, and a character recognizing engine71C to perform recognizing processes for these three different types of image elements. In addition, the image data master72has a general image master72A, a person image master72B, and a character string master72C that support these three types of data elements. The image data70,70, . . . are input to the pattern recognizing engine71A, the person recognizing engine71B, and the character recognizing engine71C, respectively.

The pattern recognizing engine71A performs an image recognizing process for the shape of a general article. The pattern recognizing engine71A compares a pattern of feature points of an image registered in the general image master72A with a pattern of feature points of the image data70transmitted from the user and extracts from the general image master72A an image that matches or is similar to the image data70, namely, an image that has a match rate higher than a predetermined value. For example, the pattern recognizing engine71A compares feature points of a registered image with those of the image data70and calculates a score of a match rate of a registered image. When the score of a registered image exceeds the predetermined value, the pattern recognizing engine71A extracts the image as a registered image with a high match rate. If the pattern recognizing engine71A has obtained a plurality of registered images having a high match rate exceeding the predetermined value corresponding to the image data70, the pattern recognizing engine71A extracts all the obtained registered images from the general image master72A.

The pattern recognizing engine71A outputs image IDs76,76, . . . of the registered images that have been extracted. According to these image IDs76,76, . . . , the names of alternatives of a general article contained in the image data70can be obtained.

In the image data70transmitted from the user, an object may not have been photographed at a proper angle. Thus, when the image recognizing application71compares patterns, it calculates the relationship of the positions of a plurality of feature points. Thus, even if the angle of the image photographed by the user is different from the angle of a registered image by a predetermined angle, for example 45°, the pattern recognizing engine71A can retrieve the image according to the foregoing pattern-matching method.

In addition, as was described above, since the image data7transmitted from the user have been gray-scaled on the user side, the image data7are not affected by a camera unit that has photographed it and lighting color, the image data7can be accurately recognized.

In addition, the pattern recognizing engine71A compares the image data70transmitted from the user with a registered image, calculates a score of a match rate, and determines a match rate of a feature point according to the score. Thus, even if a part of the image data7is different from a registered image, the pattern recognizing engine71A can extract the image as an image with a high match rate. Even if the image data70that the user photographed contains for example a signboard that a corresponding registered image does not contain, the pattern recognizing engine71A can extract the image as an image with a high match rate. In addition, even if the range of a photographed object of the image data70transmitted from the user is different from that of a registered image, the pattern recognizing engine71A can extract the registered image as an image with a high match rate.

The person recognizing engine71B performs a person recognizing process for the image data70. When the person recognizing engine71B recognizes a face-like image, the person recognizing engine71B recognizes physiognomy. Several technologies that recognize a face-like image from the image data70have been practically implemented. When the person recognizing engine71B recognizes physiognomy, it assigns a plurality of feature points to a portion recognized as a human face and calculates a score of a face-like image registered in the person image master72B according to the relationship of positions of the feature points. The person recognizing engine71B extracts a registered image having a score that exceeds a predetermined value as a physiognomy image with a high match rate. If the person recognizing engine71B has obtained a plurality of physiognomy images having a match rate exceeding the predetermined value according to the image data70, the person recognizing engine71B extracts all the plurality of registered images that have been obtained.

The person recognizing engine71B outputs image IDs77,77, . . . of the registered images that have been extracted. According to the image IDs77,77, . . . , the names of alternatives of a person contained in the image data70can be obtained.

Several physiognomy image extracting technologies have been practically implemented. For example, a face authenticating technology using both Gabor wavelet transform and graph matching method or another face authenticating technology using both perturbation space method and adaptive regional blend matching method may be used.

Like the foregoing, since the image data70transmitted from the user have been gray-scaled on the user side, the image data7are not affected by a camera unit that has photographed it and lighting color, the image data7can be accurately recognized.

In addition, the pattern recognizing engine71A compares the image data70transmitted from the user with a registered image, calculates a score of a match rate, and determines a match rate according to the score. Thus, even if a part of the image data70is different from a registered image, the person recognizing engine71B can extract the image as an image with a high match rate. Even if the image data70that the user photographed contains for example a person who wears a cap and glasses that a corresponding registered image does not contain, the person recognizing engine71B can extract the image as an image with a high match rate.

The character recognizing engine71C performs a character recognizing process for the image data70and extracts character information. Character information can be extracted using an ordinary OCR (Optical Character Reader) technology. The character string master72C may compose for example a word dictionary and detect words according to detected characters. The character recognizing engine71C references the character string master72C according to the recognized results and determines whether the image data70contain a character string having a match rate exceeding a predetermined value. When the image data70contain a character string that matches a character string registered in the character string master72C, the character recognizing engine71C outputs character string data78. If the character recognizing engine71C has obtained a plurality of character strings having a match rate exceeding the predetermined value from the image data70, the character recognizing engine71C outputs the plurality of character string data78,78, . . . as alternatives.

Since the image IDs76,76, . . . that have been output from the pattern recognizing engine71A, the image IDs77,77, . . . that have been output from the person recognizing engine71B, and the character strings78,78, . . . as alternatives that have been output from the character recognizing engine71C may be alternatives of one image, it can be said that they are ambiguous results. A meta data extracting engine73A retrieves meta data75,75, . . . from the meta data master74according to the plurality of recognized ambiguous results recognized by the engines71A,71B, and71C. The meta data extracting engine73A determines that the meta data75,75, . . . that have a common part correspond to an image contained in the image data70.

When the moving picture data70′ has been transmitted from the user and divided into the image data70,70, . . . according to frames, the meta data extracting engine73A determines that a recognized result contained in many of the image data70,70, . . . correspond to an image contained in the moving picture data70′. For example, the meta data extracting engine73A determines that a recognized result contained in the image data70,70, . . . of frames with a ratio exceeding a predetermined value correspond to an image contained in the moving picture data70′.

FIG. 9A,FIG. 9B,FIG. 9C,FIG. 9D, andFIG. 9Eshow examples of information obtained from the image data70by the engines71A,71B, and71C.FIG. 9Ashows an example of which the image data70contain a logo mark image and a text image as image elements. The pattern recognizing engine71A recognizes a logo mark image. However, the pattern recognizing engine71A does not recognize the shape of an article. Since the image data70do not contain a face-like image, the person recognizing engine71B does not output a recognized result. The character recognizing engine71C recognizes a character string at a lower portion of the logo mark image.

FIG. 9Bis an example of which the image data70contain an image of an article (shoes) as an image element and the article is accompanied by a logo mark image. The pattern recognizing engine71A recognizes that the shape of the article is shoes. In addition, the pattern recognizing engine71A recognizes a logo mark image on the front surface of the article recognized as shoes. Since the image data70do not contain characters, the character recognizing engine71C outputs “none” as a recognized result.FIG. 9Cis an example of which the image data70contain a part of a building as an image element and its appurtenance describes a logo mark image. The pattern recognizing engine71A recognizes that the image data70contain an image element that is a building. In addition, the pattern recognizing engine71A recognizes that the building contains a logo mark image. The pattern recognizing engine71A may identify the building with an image element of the recognized building. In any ofFIG. 9BandFIG. 9C, since the image data70do not contain a face-like image, the person recognizing engine71B does not recognize the image data70. In the examples shown inFIG. 9BandFIG. 9C, the pattern recognizing engine71A may recognize image elements in the reveres order.

FIG. 9Dis an example of which the image data70contain an image of a person as an image element. The person recognizing engine71B recognizes a face-like image contained in the image data70. In addition, the person recognizing engine71B recognizes physiognomy. Since the image data70do not contain characters, the character recognizing engine71C does not output a meaningfully recognized result. Since the image data70do not contain an image element other than a person and an article she wears, the pattern recognizing engine71A does not output a meaningfully recognized result. In the example shown inFIG. 9D, the pattern recognizing engine71A may output a recognized result of an article that a person wears.

FIG. 9Eis an example of which the image data70contain an image of an article and a character string as image elements. The front surface of the article contains a logo mark image. The character string is hand-written or written with a hand-written style typeface. The pattern recognizing engine71A recognizes that an article contained in an image element of the image data70is a game machine made by OO company. In addition, the pattern recognizing engine71A recognizes a logo mark image on the front surface of the recognized article. The character recognizing engine71C recognizes a character string contained in the image data70. Thus, even if characters are not perfectly recognized because they are written with a hand-written style typeface, a correct character string can be estimated with the word dictionary of the character string master72C. Since the image data70do not contain a face-like image, the person recognizing engine71B does not recognize the image data70.

FIG. 10A,FIG. 10B, andFIG. 10Cshow a more specific example of the image recognizing process and the meta data extracting process shown inFIG. 8AandFIG. 8B. It is assumed that the image data70transmitted from the user contains image elements of a plurality of human images, logo mark images, and product images. The image data70are supplied to the pattern recognizing engine71A, the person recognizing engine71B, and the character recognizing engine71C.

The pattern recognizing engine71A retrieves images from the general image master72A according to feature data of the image data70. Various types of product images, various types of logo mark images, partial or whole building images (not shown), and so forth are registered in the general image master72A as registered images as shown inFIG. 10A,FIG. 10B, andFIG. 10C. The pattern recognizing engine71A compares feature data of the image data70with feature data of a registered image of the general image master72A and retrieves a registered image that matches or is very similar to a part or whole of the image data70from the general image master72A. In the examples shown inFIG. 10A,FIG. 10B, andFIG. 10C, the logo mark image “ABC” and the image of the DVD recorder made by ZZZ company contained in the image data70are retrieved from the general image master72A. Image IDs76,76, . . . of the retrieved images are supplied to the meta data extracting engine73.

InFIG. 10A,FIG. 10B, andFIG. 10C, for convenience, as the image data IDs76,76, . . . , the names of the corresponding article (DVD recorder made by ZZZ company) and logo mark (“ABC”) are described.

The person recognizing engine71B recognizes a person according to feature data of the image data70. The person recognizing engine71B retrieves a face-like image from the person image master72B and recognizes physiognomy. As shown inFIG. 10A,FIG. 10B, andFIG. 10C, physiognomy images of various persons are registered in the person image master72B. The person recognizing engine71B compares the recognized face-like image with a physiognomy image registered in the person image master72B and retrieves a registered physiognomy image that matches or is very similar to the face-like image therefrom. In the example shown inFIG. 10A,FIG. 10B, andFIG. 10C, two human images are contained in the image data70and retrieved from the person image master72B. Image IDs77,77, . . . of the retrieved images are supplied to the meta data extracting engine73.

The character recognizing engine71C performs a character recognizing process for the image data70and detects characters from the image data70. In the example shown inFIG. 10A,FIG. 10B, andFIG. 10C, “DVD” and “recorder” are retrieved as character strings contained in the image data70. The retrieved character strings78,78, . . . are supplied to the meta data extracting engine73.

The meta data extracting engine73identifies the contents of the image data70according to the image IDs76,76, . . . and the image IDs77,77, . . . supplied from the pattern recognizing engine71A and the person recognizing engine71B and character strings supplied from the character recognizing engine71C.

In the example shown inFIG. 10A,FIG. 10B, andFIG. 10C, the meta data extracting engine73retrieves meta data from the meta data master74with six keywords of two image IDs76,76according to the recognized results of the pattern recognizing engine71A, two image IDs77,77according to the recognized results of the person recognizing engine71B, and two character strings78,78according to the recognized results of the character recognizing engine71C.

When meta data that are in common with (or that include all) these six keywords have been registered in the meta data master74, the meta data extracting engine73outputs the meta data as meta data corresponding to the image data70transmitted from the user. In the example shown inFIG. 10A,FIG. 10B, andFIG. 10C, meta data that contain all the logo mark image “ABC”, the image of the DVD recorder made by ZZZ company, the persons “XXX MASAKAZU” and “YY MIKI”, and the character strings “DVD” and “recorder” are meta data corresponding to the image data70.

Instead, meta data75,75, . . . of the logo mark image “ABC”, the image of the DVD recorder made by ZZZ company, the persons “XX MASAKAZU” and “YY MIKI”, and the character strings “DVD” and “recorder” may be separately output.

In the example shown inFIG. 10A,FIG. 10B, andFIG. 10C, meta data corresponding to the image data70contain a URL of a web site that publishes the model number, price, introduction, and information of the DVD recorder made by ZZZ company and a URL of a web site that sells the product. When the image data70are an image of a commercial of a television broadcast, meta data corresponding to the image data70contain the names of performers, song name, information about performers of the song, and so forth. When meta data correspond to for example a product image, meta data75,75, . . . of image elements contain a URL of a web site that publishes the model number, price, introduction, and information of a product and a URL of a web site that sells the product. When meta data correspond to a person, the meta data contain the name of the person and profile thereof.

According to the embodiment of the present invention, even if the meta data extracting engine73cannot recognize some (for example, one, two, or three) of the six input keywords, the meta data extracting engine73can estimate image elements that are contained in the image data70and that the engines71A,71B, and71C have not recognized and output the estimated image elements.

Now, it is assumed that the logo mark image “ABC”, the person “XX MASAKAZU”, and the person “YY MIKI” of the foregoing six keywords obtained as the searched results of the engines71A,71B, and71C were recognized with a high match rate and the other three keywords were not recognized because their match rate was low. In this case, the meta data extracting engine73AND-retrieves meta data from the meta data master74with the recognized three keywords. It can be expected that the meta data extracting engine73can obtain meta data for an advertisement image of the DVD recorder made by ZZZ company with a high probability. The meta data extracting engine73references a portion other than information corresponding to the recognized image elements of the meta data of the image data70. As a result, the meta data extracting engine73can obtain meta data of image elements that were not recognized by the engines71A,71B, and71C.

Next, services that can be provided according to the embodiment of the present invention will be described. According to the present invention, as was described above, the server3pre-registers image data and meta data thereof to databases. The server3performs the image recognizing process for image data transmitted from the user, references the databases, and extracts meta data therefrom. According to the present invention, using meta data, various services can be provided to the user.

Next, with reference toFIG. 11AandFIG. 11B, these services will be described using more specific examples. The user photographs an attractive photo or article in a magazine or a newspaper with the portable camera telephone terminal1, the digital camera2, or the like and transmits the photo or article as image data7A to the server3. The server3performs an image recognizing process for the received image data7A, recognizes character information, a logo mark image, and an image of an article contained as image elements in the image data7A and extracts meta data corresponding to individual image elements of the image data7A from the meta data master74. Thus, the server3can provide services for a news flash of the photographed article, information about a product introduced in the photographed magazine, and mail order of the product. When the photographed magazine or newspaper describes information about for example resort information, ticket sales information, and so forth, the server3can provide these information to the user.

Likewise, when image data7B photographed by the user and transmitted to the server are a part of a building such as a restaurant, the server3recognizes the shape of the building, a logo mark image, and so forth contained as image elements in the image data7B, identifies for example the name of the restaurant, and provides other information about the restaurant such as menu information, business hours, photos of the inside of the restaurant, and reservation information (telephone number) to the user along with the restaurant name.

When image data7C photographed by the user and transmitted to the server3contain a label of a drink for example wine or a food as an image element, the server3recognizes a logo mark image and character information of the label image of the image data7C, identifies for example the name of the wine, and provides information about the taste of the wine, information about retailers, information about prices, and so forth to the user. In addition, the server may provide a wine order service to the user.

When image data7D photographed by the user and transmitted by the server3contain a person as an image element, the server3recognizes the person contained in the image data7D and identifies the name of the person. When the person is for example an athlete, the server3can provide information about the team to which the athlete belongs, the score of a match of the team, and so forth to the user along with the information about the athlete. When there are character goods of the athlete and the team, the server3may provide a character goods order service to the user.

When image data7E photographed and transmitted to the server3by the user contain an advertisement image of a new release CD (Compact Disc) or a new release movie as image elements, the server3identifies the title of the CD that will be released, the artist names, and the movie title according to the advertisement image contained in the image data7E. When the image data7E are an advertisement image of for example a new release movie, the server can provide the story of the movie, information about the performers, information about the director, and information about theaters and show times to the user along with the title name. In addition, the server3may provide a movie ticket order service to the user. When a DVD (Digital Versatile Disc) that contains the movie is released, the server3may provide a DVD order service to the user.

When image data7F photographed and transmitted to the server3by the user contain a product advertisement image of a poster or a commercial of a television broadcast, the server3identifies an image of the product contained in the image data7F, identifies the name of the product, and provides information about the product and a product order service to the user. When the advertisement image contains for example an image of a person as an image element as well as the product as an advertisement object, the server can recognize the person and provide information about the person to the user. When the advertisement image is a commercial of a television broadcast, the server can provide music information used in the commercial to the user.

Now, it is assumed that the server3has created a web site that provides to the user a service corresponding to the image data7transmitted from the user and he or she can access the web site through the Internet2. The server3transmits the URL of the web site to the portable camera telephone terminal1that transmitted the image data7using for example an electronic mail function.

On the other hand, the portable camera telephone terminal1has an electronic mail transmitting and receiving function and an accessing function for web sites on the Internet2. The portable camera telephone terminal1has browser software with which the user can browse web sites. When the user designates the URL of a web site, he or she can access the web site. At this point, the user can select an item with keys on the portable camera telephone terminal1, input data, and transmit them. When the user accesses a URL received by the portable camera telephone terminal1, he or she can receive a service that the server3provide to him or her. As the interface on the user side, the user can use a personal computer that can communicate with the Internet2instead of the portable camera telephone terminal1. In this case, the user may transmit the image data7from the personal computer to the server3. Instead, the user may transmit the image data7to the server3by the portable camera telephone terminal1and receive the service from the server3by the personal computer.

FIG. 12A,FIG. 12B,FIG. 13A,FIG. 13B, andFIG. 14correspond to the example of the image data7A. In this example, the user photographs an attractive photo or article in a magazine or a newspaper, transmits it to the server3, and receives a service from the server3. InFIG. 12AandFIG. 12B, the user photographs the cover of the magazine that contains the attractive photo or article with the portable camera telephone terminal1, attaches the photographed image data7A to electronic mail, and transmits the electronic mail to the server3. In this case, the image data7A may not be a photo of the magazine, but a copy thereof, image data of a television broadcast, or the like. A screen101is an example of a screen of a display section of the portable camera telephone terminal1. Hereinafter, an example of a screen is an example of a screen of the display section of the portable camera telephone terminal1. When the user transmits the image data7A, he or she does not need to input data in the subject field. When the user presses a send button of the portable camera telephone terminal1, the image data7A are transmitted to the server3.

The server3receives the image data7A, performs an image recognizing process for the image data7A, and identifies image elements contained therein, and extracts meta data corresponding to the identified image elements. Thereafter, the server3creates a web site that provides a service according to the extracted meta data to the user, and returns electronic mail describing the URL of the web site to the portable camera telephone terminal1. As shown in a screen102, the returned electronic mail describes a URL102A of the web site that provides the service to the user. When the user designates the URL102A in a predetermined manner, the portable camera telephone terminal1accesses the web site and displays a screen103.

On the screen103, an image7A′ of the magazine of the image data7A and buttons103A and103B appear. In the example of which the user photographed a magazine100, the server3recognizes a logo mark image of a magazine name “ABCD” and a portable telephone terminal from a photo of the cover of the magazine100and provides a service regarding the magazine “ABCD” and a service regarding the portable telephone terminal to the user.

The user can select buttons appearing on the screen by a predetermined operation on the portable camera telephone terminal1. When the user confirms a selected button by a predetermined operation, selection information according to the selected button is transmitted to the server3. In the following description, an operation that the user selects a button appearing on a screen by a predetermined manner and confirms the selected button is sometimes referred to as “press a button”. The server3transmits data according to the received selection information to the portable camera telephone terminal1. The portable camera telephone terminal1displays a screen according to the received data.

When the user wants to receive the service regarding the magazine “ABCD” and he or she selects a button103A, selection information of the button103A is transmitted to the server3. As a result, the screen103is changed to a screen104shown inFIG. 13AandFIG. 13B. An outlined description of the magazine “ABCD” appears on the screen104according to meta data of the magazine “ABCD”. In addition, buttons104A,104B,104C, and104D appear on the screen104. When the user selects the button104A, the portable camera telephone terminal1displays a screen105. The user can know more detailed information about the magazine “ABCD” on the screen105.

When the user presses the button104B on the screen104, it is changed to a screen106. The user can mail-order the magazine “ABCD” on the screen106. When the user inputs necessary information and presses a button106A on the screen106, it is changed to a screen107on which he or she can confirm the input information. When the user presses a button107A, he or she can pay for the mail order. When the user presses the button104C on the screen104, it is changed to a screen on which he or she can retrieve similar information.

When the user presses the button103B on the screen103shown inFIG. 12AandFIG. 12B, the screen103is changed to a screen108(refer toFIG. 14) on which he or she can receive a service regarding the portable telephone terminal recognized from the image data7A of the photographed magazine100. Outlined information of the portable telephone terminal in the magazine appears on the screen108according to meta data. In addition, buttons108A,108B, and108C appear on the screen108. When the user presses the button108A, the portable camera telephone terminal1displays a screen (not shown) on which he or she can browse more detailed information about the portable telephone terminal.

When the user presses the button108B on the screen108, it is changed to a screen109on which he or she can retrieve similar information. For example, the server3retrieves meta data with a keyword of “portable telephone terminal” and extracts alternatives of similar information. The alternatives of similar information appear on the screen109. When the user selects an alternative and presses the button109A on the screen109, it is changed to a screen (not shown) on which he or she can browse the selected alternative. The selected alternative has the same structure as the portable telephone terminal appearing on the screen108, but different therefrom in model. When the user presses a predetermined button on the screen, it is changed to a screen on which he or she can browse detailed information or retrieve similar information.

In the foregoing description, for example an information screen and a mail order screen that appear after the screen104may be provided by the server3. Instead, an original seller or the like of the product may provide these screens as a web site on the Internet2. For example, the button103A on the screen103may be linked to a web site of the magazine “ABCD” of its publisher. When the button103A is pressed, the web site of the magazine “ABCD” may be accessed.

The button104D on the screen104, the button106B on the screen106, the button107B on the screen107, the button108C on the screen108, and the button109B on the screen109are buttons with which the current screen is returned to the immediately preceding screen.

FIG. 15A,FIG. 15B,FIG. 16A, andFIG. 16Bshow an example of the foregoing image data7B. In this case, the user photographs the whole or a part of a building, transmits the photographed image to the server3, and receives a service therefrom. InFIG. 15AandFIG. 15B, the user photographs a portion about an entrance of an attractive restaurant “XXX” with the portable camera telephone terminal1, attaches the photographed image data7B to electronic mail on the screen101, and transmits the electronic mail to the server3. The image data7B may not be a photo of the restaurant “XXX”, but a copy of a magazine, image data of a television broadcast, or the like.

The server3performs an image recognizing process and a meta data extracting process for the received image data7B, identifies image elements contained in the image data7B, and retrieves meta data corresponding to the image elements. The server3creates a web site that provides a service according to the extracted meta data to the user and returns electronic mail describing the URL of the web site to the portable camera telephone terminal1. When the user designates the URL102A appearing on the screen102in a predetermined manner, the portable camera telephone terminal1accesses the web site and displays a screen111.

An image7B′ of the restaurant “XXX” according to the image data7B and a button111A appear on the screen111. When the user presses the button111A on the screen111, it is changed to a screen112shown inFIG. 16AandFIG. 16B. An outlined description of the restaurant “XXX” and buttons112A,112B,112C, and112D appear on the screen112. When the user presses the button112A on the screen112, it is changed to a screen113. Detailed information about the restaurant “XXX” appears on the screen113.

When the user presses the button112B on the screen112, it is changed to a screen114on which a map indicating the location of the restaurant “XXX” appears. When the user presses a button114A on the screen114, the user can call the restaurant “XXX” with the telephone function of the portable camera telephone terminal1. The user can talk to a person of the restaurant “XXX” for a reservation or the like. Instead, when the user selects a telephone number portion on the screen112, he or she can call the restaurant “XXX” using the telephone function of the portable camera telephone terminal1.

When the user presses the button112C on the screen112, it is changed to a screen on which he or she can retrieve similar information. For example, the server3retrieves meta data according to the category (for example, “Japanese restaurants”) to which the restaurant “XXX” belongs. A list of restaurants that belong to categories similar to the category of the restaurant “XXX” appears on the screen.

In the foregoing example, information appears after for example the screen112may be provided by the server3. Instead, the information may be provide by the restaurant “XXX” side as a web site on the Internet2. In this case, when the user presses the button111A on the screen111, the portable camera telephone terminal1is linked to the web site.

The button112D on the screen112and the button114B on the screen114are buttons with which the current screen is returned to the immediately preceding screen.

FIG. 17A,FIG. 17B,FIG. 18A, andFIG. 18Bshow an example of the foregoing image data7C. In this case, the user photographs for example a label of a wine bottle, transmits the photographed image to the server3, and receives a service therefrom. InFIG. 17AandFIG. 17B, the user photographs a label of an attractive wine bottle with the portable camera telephone terminal1, attaches the photographed image as the image data7C to electronic mail on the screen101, and transmits the electronic mail to the server3. The image data7C may not be a photo of the label, but a photo of a magazine, image data of a television broadcast, or the like.

The server3performs an image recognizing process and a meta data extracting process for the received image data7C, identifies image elements contained in the image data7C, and retrieves meta data corresponding to the image elements. The server3creates a web site that provides a service according to the meta data to the user and returns electronic mail describing the URL of the web site to the portable camera telephone terminal1. When the user designates a URL102A of the return electronic mail on the screen102in a predetermined manner, the portable camera telephone terminal1accesses the web site and displays a screen120.

An image7C′ of the label of the image data7C and a button120A appear on the display screen120. When the user presses the button120A on the screen120, it is changed to a screen121shown inFIG. 18AandFIG. 18B. An outlined description of the wine and buttons121A,121B,121C, and121D appear on the screen121. When the user presses the button121A on the screen121, it is changed to a screen122on which detailed information of the wine appears.

When the user presses the button121B on the screen121, it is changed to a screen123on which he or she can mail-order the wine. When the user inputs necessary information on the screen123and presses the button123A, the screen123is changed to a screen124on which he or she can confirm the input information. When the user presses the button124A on the screen124, he or she can pay for the mail order.

When the user presses the button121C on the screen121, it is changed to a screen on which similar information appears. For example, the server3retrieves meta data of other wines according to the meta data of the wine and obtains a list of wines similar to the wine. The list of similar wines appears on the screen.

Information that appears after for example the screen121may be provided by the server3. Instead, the information may be provided by the original seller, importer, or the like of the wine as a web site on the Internet2. In this case, when the user presses the button120A on the screen120, the portable camera telephone terminal1is linked to the web site.

The button121D on the screen121, the button123B on the screen123, and the button124B on the screen124are buttons with which the current screen is changed to the immediately preceding screen.

FIG. 19A,FIG. 19B,FIG. 20A, andFIG. 20Bshow an example of the image data7D. In this case, the user photographs a person (professional athlete), transmits the photographed image to the server3, and receives a service therefrom. InFIG. 19AandFIG. 19B, the user photographs an attractive athlete with the portable camera telephone terminal1, attaches the photographed image as the image data7D to electronic mail on the screen101, and transmits the electronic mail to the server3. The image data7D may not be a photo of a person, but image data of a television broadcast, a photo of a magazine, or the like.

The server3performs an image recognizing process, a physiognomy process, and a meta data extracting process for the received image data7D, identifies a person of an image element contained in the image data7D, and retrieves meta data corresponding to the person. The server3creates a web site that provides a service according to the meta data to the user and returns electronic mail describing the URL of the web site to the portable camera telephone terminal1. When the user designates a URL102A of the return electronic mail on the screen102in a predetermined manner, the portable camera telephone terminal1accesses the web site and displays a screen130.

An image7D′ corresponding to the image data7D and buttons130A and130B appear on the screen130. In the example of which the user photographed an athlete, the server3recognizes the image of the athlete and a logo mark image of a uniform that he wears and provides a service regarding the athlete and a service regarding the uniform that he wears to the user.

When the user wants to receive the service regarding the athlete and the user presses the button130A on the screen130, it is changed to a screen131(refer toFIG. 20A). An outlined description about the athlete appears on the screen131. In addition, buttons131A,131B,131C, and131D appear on the screen131. When the user presses the button131A on the screen131, it is changed to a screen132on which detailed information about the athlete appears on the screen132.

When the user presses the button131C on the screen131, it is changed to a screen133on which similar information is retrieved. For example, the server3retrieves meta data with a keyword of the name of the athlete. Alternatives of similar information appear on the screen133. In this example of which an object is an athlete, information about a game of the team to which the athlete belongs and information about other athletes of the team appear as similar information on the screen133. When the user selects his or her desired similar information and presses the button133A on the screen133, it is changed to a screen corresponding to the selected similar information.

Wallpaper image data using photographed image data of the athlete (image data used as a background of a screen of the display section) and call alert audio data of his voice may be used as similar information. In this case, when the user presses the button133A on the screen133, it is changed to a screen (not shown) on which the portable camera telephone terminal1downloads the wallpaper image data and call alert audio data through the Internet2.

When the user presses the button131B on the screen133, it is changed to a screen134on which he or she can mail-order a product. Instead, when the user wants to receive a service regarding a uniform and presses the button130B on the screen130, it is changed to the screen134.

As described above, in the example shown inFIG. 19A,FIG. 19B,FIG. 20A, andFIG. 20B, in addition to the image of the athlete, the logo mark image of the uniform that he wears is recognized from the image data7D. Thus, the user can mail-order sportsware relating to the recognized logo mark image. In this example, images134A and134B of uniforms that the user can mail-order also function as buttons. When the user presses an image of his or her desired sportswear on the screen134, it is changed to the screen135on which he or she can mail-order the selected sportswear.

When the user inputs necessary information and presses the button135A on the screen135, it is changed to a screen (not shown) on which he or she confirms his or her input information. When the user presses for example a pay button on the screen, he or she can pay for the mail order.

Information that appears after for example the screen131may be provided by the server3. Instead, the information may be provided by the contractor of the athlete, original seller of the sportsware recognized by the logo mark image, or the like as a web site on the Internet2. In this case, when the user presses the button130A or the button130B on the screen130, the portable camera telephone terminal1is linked to the web site.

The button131D on the screen131, the button133B on the screen133, and the button135B on the screen135are buttons with which the current screen is changed to the immediately preceding screen.

FIG. 21A,FIG. 21B,FIG. 22A, andFIG. 22Bshow an example of the image data7E. In this example, the user photographs an advertisement image of a new release CD or a new release movie, transmits the photographed image to the server3, and receives a service therefrom. In this example, it is assumed that the image data7E are data of which an advertisement poster of a new release movie was photographed. InFIG. 21AandFIG. 21B, the user photographs an advertisement poster of an attractive new release movie with the portable camera telephone terminal1, attaches the photographed image as the image data7E to electronic mail on the screen101, and transmits the electronic mail to the server3. The image data7E may be image data of a commercial of a television broadcast.

The server3performs an image recognizing process and a meta data extracting process for the received image data7E, identifies image elements contained in the image data7E, and retrieves meta data corresponding to the image elements. The server3creates a web site that provides a service according to the meta data to the user and returns electronic mail describing the URL of the web site to the portable camera telephone terminal1. When the user designates a URL102A of the return electronic mail on the screen102in a predetermined manner, the portable camera telephone terminal1accesses the web site and displays a screen140.

A poster image7E′ corresponding to the image data7E and a button140A appear on the screen140. When the user presses the button140A on the screen140, it is changed to a screen141(refer toFIG. 22A). An outlined description of the movie advertised by the poster and buttons141A,141B,141C, and141D appear on the screen141. When the user presses the button141A on the screen141, it is changed to a screen142on which detailed information of the movie appears. In this example, information about shows such as theaters and show times appear as the detailed information.

When the user presses the button142A on the screen142, it is changed to a screen144on which he or she can mail-order a movie ticket. Instead, when the user presses the button141B on the screen141, it is changed to the screen144on which he or she can mail-order a movie ticket. When the user inputs necessary information and presses the button144A on the screen144, it is changed to a screen145on which he or she can confirm the input information. When the user presses the button145A on the screen145, he or she can pay for the mail order.

When the user presses the button141C on the screen141, it is changed to a screen143on which similar information appears. Order information of a DVD of the movie, related information of the movie such as story and cast, download information of wallpaper data using an image of the movie, news about the movie, and so forth appear as the similar information. When the user selects his or her desired similar information and presses the button143A on the screen143, it is changed to a screen on which the selected information appears. When the user selects order information about the DVD and presses the button143A on the screen, it is changed to a screen (not shown) on which a DVD mail order page appears. When the user selects wallpaper data and presses the button143A on the screen143, it is changed to a screen (not shown) on which he or she can download wallpaper data through the Internet2.

Information that appears after for example the screen141may be provided by the server3. Instead, the information may be provided by the distributor or the like as a web site on the Internet2. In this case, when the user presses the button140A on the screen140, the portable camera telephone terminal1is linked to the web site.

The button141D on the screen141, the button142B on the screen142, the button143B on the screen143, the button144B on the screen144, and the button145B on the screen145are buttons with which the current screen is changed to the immediately preceding screen.

FIG. 23A,FIG. 23B,FIG. 24A,FIG. 24B, andFIG. 25show an example of the image data7F. In this example, the user photographs a poster or the like that advertises a product, transmits the photographed image to the server3, and receives a service therefrom. InFIG. 23AandFIG. 23B, the user photographs an attractive product with the portable camera telephone terminal1, attaches the photographed image as the image data7F to electronic mail on the screen101, and transmits the electronic mail to the server3. The image data7F may be image data of a commercial of a television broadcast or the like.

The server3performs an image recognizing process and a meta data extracting process for the received image data7F, identifies image elements contained in the image data7F, and retrieves meta data corresponding to the image elements. The server3creates a web site that provides a service according to the meta data to the user and returns electronic mail describing the URL of the web site to the portable camera telephone terminal1. When the user designates a URL102A of the return electronic mail on the screen102in a predetermined manner, the portable camera telephone terminal1accesses the web site and displays a screen150.

An image7F′ corresponding to the image data7F and buttons150A and150B appear on the screen150. In the example of which the user photographed a poster adverting a product, the server3recognizes the image of the advertised product, a logo mark image of the product, and an image of a model used in the advertisement and provides a service regarding the product and a service regarding the model.

When the user presses the button150A on the screen150, it is changed to a screen151(refer toFIG. 24A). An outlined description of the product advertised by the poster and buttons151A,151B,151C, and151D appear on the screen151. When the user presses the button151A on the screen151, it is changed to a screen152on which detailed information of the product appears.

When the user presses the button152A on the screen152, it is changed to a screen154on which he or she can mail-order the product. Instead, when the user presses the button151B on the screen151, it is changed to the screen154on which he or she can mail-order the product. When the user inputs necessary information and presses the button154A on the screen154, it is changed to a screen (not shown) on which he or she can confirm the input information and pay for the mail order.

When the user presses the button151C on the screen151, it is changed to a screen153on which similar information appears. Other products of the same series of the relevant product, information about commercials advertising the relevant product, related news thereof, and so forth appear on the screen153as the similar information according to the meta data of the relevant product. When the user selects desired similar information and presses the button153A on the screen153, it is changed to a screen corresponding to the selected similar information.

When the user presses the button150B on the screen150, it is changed to a screen155(refer toFIG. 25). Outlined information about the model in the poster appears on the screen155. When the user presses the button155A on the screen155, it is changed to a screen (not shown) on which detailed information about the model appears. When the user presses the button155B, it is changed to a screen on which he or she can mail-order a product corresponding to the model. When the user presses the button155C on the screen155, it is changed to a screen (not shown) on which similar information about the model appears. In other words, information that appears after the screen155on which detailed information about the model appears has the same structure as information that appears after the screen131on which detailed information about the athlete appears.

Information that appears after for example the screen151may be provided by the server3. Instead, the information may be provided by the sponsor of the product, the model agency to which the model belongs, or the like as a web site on the Internet2. In this case, when the user presses the button150A or the button150B on the screen150, the portable camera telephone terminal1is linked to the web site.

The button151D on the screen151, the button152B on the screen152, the button153B on the screen153, the button154B on the screen154, and the button155D on the screen155are buttons with which the current screen is changed to the immediately preceding screen.

Thus, using the system according to the embodiment of the present invention, when the user transmits only the photographed image data7to the server3, he or she can receive various services according to the contents of the photographed image data7. The server3extracts meta data according to the result of the image recognizing process and determines the contents of the service. Thus, even if the user does not have the knowledge of an object that he or she photographs, he or she can properly receive a service corresponding thereto.

In addition, when the server3performs an image recognizing process for the image data7transmitted form the user, the server3recognizes a plurality of image elements contained in the image data7and extracts meta data corresponding to the image elements. Thus, the user can also receive services for other than the object that he or she photographed. In addition, the user can receive services corresponding to image elements of an image (a commercial image of a poster or a television broadcast, a magazine, or the like) other than an image of an object that he or she photographs.

Thus, when the system according to the embodiment of the present invention is used, the following things can be accomplished.

(1) It is assumed that the user sees a poster of a new release movie and wants to know a theater and show times at which the movie will be performed. When the user photographs the poster with his or her portable camera telephone terminal1and transmits the image data7of the poster to the server3, he or she can obtain his or her desired information. When the user obtains his or her desired information, he or she does not need to input characters. In addition, even if the user forgot or incorrectly memorizes necessary information, he or she can retrieve it.

(2) It is assumed that the user wants shoes that a person on a photo of an advertisement of a magazine wears. In this case, even if the user does not know the maker and model of the shoes, when he or she photographs the photo with the portable camera telephone terminal1and transmits image data of the photo to the server3, he or she can obtain his or her desired information.

(3) When the user buys a bottle of wine, he or she may not determine whether to buy it because he or she does not know necessary information about what taste the wine has, where it was produced, and what grape species was used for the wine. In this case, when the user photographs the label of the bottle of wine with the portable camera telephone terminal1and transmits an image of the photo of the label to the server3, he or she can obtain the necessary information. In addition, since the user only needs to transmit an image that he or she photographed, even if he or she cannot read characters of the label because they are written in French or Italian, he or she can quickly retrieve his or her necessary information on the spot.

Next, the case that the embodiment of the present invention is applied to contents of a television broadcast will be considered. With a network connection type television receiver that has been practically implemented, information corresponding to a television broadcast can be obtained through a network. However, the embodiment of the present invention has the following advantages against such a network connection type television receiver.

(1) When the user receives services of the network connection type television receiver, he or she needs to prepare a dedicated television receiver. In contrast, when the user receives services according to the embodiment of the present invention, he or she only needs to photograph an image that appears on a television receiver with the portable camera telephone terminal1and transmit the photographed image to the server3. Thus, when the user receives the services, he or she does need to prepare a dedicated television receiver.

(2) When the user receives services of the network connection type television receiver, he or she can receive only services that a television broadcasting station has prepared. In contrast, when the user receives services according to the embodiment of the present invention, since information is provided according to meta data extracted from an image, he or she can retrieve various types of information.

(3) When the user receives services of the network connection type television receiver, since data are transmitted through a data channel when a broadcast is aired, related information is not updated after the broadcast is aired. Thus, if the user records a broadcast and then reproduces it, since information is transmitted when the broadcast is aired, the information may be old and useless. In contrast, when the user receives the services according to the embodiment of the present invention, since related information is updated, he or she can retrieve the latest information with an old image.

(4) When the user receives services of the network connection type television receiver, if he or she reproduces a recorded broadcast and wants to receive a service regarding the broadcast, he or she needs to prepare a special recording unit that can record a data channel corresponding to the broadcast. In contrast, when the user receives the services according to the embodiment of the present invention, since information is retrieved according to an image, he or she can use a regular recording unit.

Many of television broadcasting stations have their web sites on the Internet2. In addition, there are many sites for television broadcast programs and commercials on the Internet2. While the user is watching a television broadcast on a regular television receiver that does not have a network connection function, if he or she found attractive information, he or she can connect his or her personal computer, portable telephone terminal having an Internet connection function, or the like to the Internet2and retrieve the information therefrom. The embodiment of the present invention has the following advantages against the information retrieving method based on such a television broadcast.

(1) While the user is watching a television broadcast, if he or she finds attractive information, he or she accesses a predetermined system (information search site or the like on the Internet) with a personal computer or the like. While the user is accessing the system, an attractive image may disappear. In contrast, when the user receives the services according to the embodiment of the present invention, if an attractive image appears, when he or she photographs the screen with a portable camera telephone terminal and transmits the photographed image of the screen to the server3, he or she can obtain attractive information. Instead, after the user photographed the screen, he or she can access the predetermined system with a personal computer or the like. Thus, even if the attractive image disappears while he or she is accessing the predetermined system, when he or she transmits the photographed image data7to the server3, he or she can access the predetermined system.

(2) When the user accesses the predetermined system with a personal computer or the like and retrieves information according to a television broadcast from the predetermined system, he or she needs to input information about the broadcasting station. In contrast, when the user receives services according to the embodiment of the present invention, he or she does not need to consider the broadcasting station. This is because when the user photographs an image of a television broadcast and transmits the photographed image to the server3, he or she can obtain information.

(3) When the user accesses the predetermined system with a personal computer or the like and retrieves information corresponding to a recorded television broadcast, he or she needs to record information about a broadcasting station that aired the television broadcast and information of air time of the television broadcast along with an image of the television broadcast. Thus, the user needs to prepare a special recording unit. In contrast, when the user receives the services according to the embodiment of the present invention, he or she can use a regular recording unit. This is because when the user photographs an image of a television broadcast reproduced from a recording unit and transmits the photographed image to the server3, he or she can obtain information.

Thus, since the services according to the embodiment of the present invention has higher advantages than the conventional information retrieving method, the services can be applied to various types of business. Next, a business model that uses the services according to the embodiment of the present invention will be described.

FIG. 26A,FIG. 26B, andFIG. 26Cconceptually show the relationship among the services according to the embodiment of the present invention (hereinafter, these services will be referred to as the image marker service), the user, and various companies. An image marker service200corresponds to for example the server side system described with reference toFIG. 4AandFIG. 4B. The image marker service200has various types of image masters such as a logo mark image master214, a regular image master215, and a person image master (for example, they correspond to the image database4shown inFIG. 4AandFIG. 4B), various types of meta data corresponding to various types of images registered in the image masters, such as content meta data211, product meta data212, and other meta data213(for example, they correspond to the related information database5shown inFIG. 4AandFIG. 4B), and an image marker service engine composed of an image recognizing engine210.

The image recognizing engine210corresponds to the image feature extracting section30, the image feature comparing section31, and the related information retrieving section32described with reference to for exampleFIG. 4AandFIG. 4B. The image recognizing engine210performs an image recognizing process for input image data, retrieves an image that matches the input image data with a high match rate from each of the image masters, extracts meta data corresponding to an image that matches the input image data with the high match rate as related information of the input image data.

The image masters and meta data are automatically collected by the automatic CM extracting system51from commercials of television broadcasts. In addition, they are retrieved from the general web sites52. The collected masters and meta data are registered by a data registering center50using a registering tool.

The user201photographs his or her attractive product, advertisement, or the like and transmits the photographed image to the image marker service200through the Internet. In the image marker service200, the image marker service engine retrieves related information of the image data transmitted from the user201. The image marker service200transmits the retrieved related information to the user201.

The user201can buy the photographed product or the like through the Internet according to the related information transmitted from the image marker service200. At this point, another company202may perform a buying process and so forth instead of the image marker service200.

The company202is an EC company that performs an electronic trade on the Internet using for example a web site at which the user can brose and buy a product. Hereinafter, the company202is referred to as the EC company202. For example, the image marker service200rents the image marker service engine to the EC company202. In other words, the image marker service200is an ASP (Application Service Provider) that rents the image marker service engine to the EC company202.

Conventionally, the user201accesses the web site of the EC company202, retrieves and selects his or her desired product from many produces, and buys it. When the EC company202uses the image marker service engine, the user directly designates a product and buys it from the EC company202without accessing the web site of the EC company202and selecting a product. The user201who buys a product from the EC company202through the image marker service engine is a prospect of the EC company202. Thus, it can be expected that the user201may be led to the web site or the like of the EC company202in the future.

On the other hand, when a general company203such as a production maker that produces products that the user can buy through the image marker service engine actively registers product information, its URL, and so forth to the image marker service engine, the general company203can expect to increase its sales. The image marker service200rents or sells a registering tool used in a data center50to the general company203so that it can register information to the image marker service engine.

Next, with reference toFIG. 27toFIG. 30, a business model of the image marker service200will be described. In the following, the user201corresponds to for example the portable camera telephone terminal1and a user who uses it. The image marker service200corresponds to for example the server3. A seller or the like204corresponds to the EC company202and the general company203shown inFIG. 26C. The user201and the image marker service200can communicate with each other through the Internet2. For example, as a server unit connected to the Internet2, the seller or the like204, the user201, and the image marker service200may mutually communicate with each other through the Internet2.

FIG. 27shows an example of a membership pay service that charges the user with for services. The example shown inFIG. 27is referred to as a first stage of the business model of the image marker service200. The image marker service200charges the user201for an information service that the image marker service200provides to him or her. In other words, the user201makes a membership contract with the image marker service200, pays the image marker service200for the information service, and receives it from the image marker service200. When the user201found a product that he or she wants to buy in information that the information service provides to him or her, the image marker service200introduces the customer who wants to buy the product (user201) to the seller or the like204. The seller or the like204provides product information to the user201that the image marker service200introduced to the seller or the like204. When the user pays the seller or the like204the money for the product, the seller or the like204ships the product and information to the user.

FIG. 28is an example that the image marker service200sells a product to the user and charges him or her for it instead of the seller or the like204. The example shown inFIG. 28is referred to as a second stage of the business model of the image marker service200. The image marker service200performs an information service that provides product information and so forth to the user201. The user buys a product according to the provided information. At this point, the user201pays the image marker service200for the product. The image marker service200deducts a service commission from the product fee and pays the remaining to the seller or the like204. The service commission becomes the profit of the image marker service200. When the seller or the like204receives the deducted money from the image marker service200, the seller or the like204ships the product to the user201.

FIG. 29is an example of which the image marker service200becomes a functional ASP that provides the function of the image marker service engine to the seller or the like204. The example shown inFIG. 29is referred to a third stage of the business model of the image marker service200. The image marker service200provides the function of the image marker service engine to the seller or the like204. The seller or the like204pays the image marker service200for the function rental fee of the image marker service engine. As a result, the seller or the like204can use the image recognizing function and the meta data extracting function for image data transmitted from the user. The seller or the like204side needs to collect meta data themselves.

The user201transmits photographed image data of a favorite product or the like to the seller or the like204. The seller or the like204transmits related information of the transmitted image data to the user201using the function of the image marker service engine. When the user201pays the seller or the like204for the product according to the related information, the seller or the like204ships the product to the user201.

FIG. 30is an example of which the image marker service200licenses the seller or the like204to use the image marker service engine. The example shown inFIG. 30is referred to as fourth stage of the business model of the image marker service200. The image marker service200licenses the seller or the like204to use the technology of the image marker service engine and provides meta data to the seller or the like204. The seller or the like204pays the image marker service200for the technology and meta data. Thus, the seller or the like204can freely use the image recognizing function and the meta data extracting function for image data transmitted from the user under licensed conditions. In addition, the seller or the like204can freely use the meta data under the licensed conditions. The seller or the like204side does not need to collect meta data.

The user201transmits photographed image data of his or her favorite product or the like to the seller or the like204. The seller or the like204transmits related information of the transmitted image data to the user201using the function of the image marker service engine. When the user201pays the seller or the like204for the product according to the related information, the seller or the like204ships the product to the user201.

In the foregoing four business models, as the stage advances from the first state to the second stage, and so forth, it can be expected that the profit of the image marker service200will increase. In addition, as the stage advances, the seller or the like204side can more feely use the function of the image marker service engine. In addition, as the stage advances, the flows of information and money are simplified. As a result, the relationship among the image marker service200, the user201, and the seller or the like204becomes more simplified. This means that business is more effectively carried out among the image marker service200, the user201, and the seller or the like204.

According to the present invention, the server has the image recognizing function. In addition, the serve pre-registers image data and related information thereof to databases. The server performs the image recognizing process for image data transmitted from the user, compares it with image data registered in the databases, and transmits related information of registered image data that match the image data transmitted from the user with a high match rate to the user.

Thus, as an effect of the embodiment of the present invention, by only transmitting photographed image data to the server, the user can receive various types of services according to the photographed contents. In addition, since the server extracts related information from databases according to the result of the image recognizing process and determines the type of a service, as an effect of the embodiment of the present invention, the user can properly receive a service of the server without the knowledge of the photographed object. Thus, as an effect of the embodiment of the present invention, the service provider side can obtain a user who does not have the knowledge about services as a new customer.

In addition, when the server performs the image recognizing process for image data transmitted from the user, the server recognizes a plurality of image elements contained in image data transmitted from the user and extracts related information for each image element. Thus, as an effect of the embodiment of the present invention, the user can receive services for portions other than the object that he or she photographed. In addition, as an effect of the embodiment of the present invention, the user can receive services corresponding to image elements of an image (a commercial image of a poster or a television broadcast, a magazine, or the like) other than an image of an object that he or she photographs.

In addition, according to the embodiment of the present invention, by only photographing image data and transmitting them to the server, the user can obtain related information of the object that he or she photographed. Thus, when the user obtains his or her desired information, as an effect of the embodiment of the present invention, he or she can omit to input characters. In addition, even if the user incorrectly memorizes or forgot a keyword, as an effect of the embodiment of the present invention, he or she can retrieve his or her desired information.

In addition, since the user can obtain related information of image data according to the result of the image recognizing process for the image data, even if he or she does not know the production maker and model of the object, as an effect of the embodiment of the present invention, by only photographing the object and transmitting the photographed image data to the server, the user can obtain his or her desired information. When the object is a label or the like, even if characters on the label are written in a foreign language that he or she cannot read, as an effect of the embodiment of the present invention, he or she can retrieve the information without difficulty.

In the case that the present invention is applied to image data of a television broadcast, as an effect of the embodiment of the present invention, by only photographing an image on a television receiver and transmitting the photographed image to the server, the user can obtain necessary information. Thus, when the user receives services from the server, he or she does not need to prepare a dedicated television receiver such as a network connection type television receiver. Since the user can obtain related information according to an image itself, when he or she wants to receive a service corresponding to a recorded image, he or she can use a regular recording unit. Thus, as an effect of the embodiment of the present invention, the user does not need to prepare a special recording unit that records a data channel along with a television broadcast.

In addition, when the user receives services of the network connection type television receiver, he or she can receive only services that a television broadcasting station has prepared. In contrast, when the user receives services according to the present invention, since information is provided according to related data extracted from an image, as an effect of the embodiment of the present invention, he or she can retrieve various types of information.

In addition, when the user receives services of the network connection type television receiver, since data are transmitted through a data channel when a broadcast is aired, related information is not updated after the broadcast is aired. Thus, if the user records a broadcast and then reproduces it, since information is transmitted when the broadcast is aired, the information may be old and useless. In contrast, when the user receives the services according to the present invention, since related information is updated, as an effect of the embodiment of the present invention, he or she can retrieve the latest information with an old image.

In addition, when the user receives services according to the present invention, even if the user finds attractive information in a television broadcast that he or she is watching, by only photographing the screen and transmitting the photographed image to the server, he or she can obtain necessary information. Thus, as an effect of the embodiment of the present invention, the user does not need to input information about a broadcasting station unlike the case that he or she retrieves information from an information search site on the Internet using for example a personal computer. In addition, as an effect of the embodiment of the present invention, there is no risk of which while the user is accessing the information search site, the broadcast ends and he or she loses information. In addition, by only photographing a reproduced image of a recorded television broadcast and transmitting the photographed image to the server, he or she can obtain necessary information. Thus, as an effect of the embodiment of the present invention, the user does not need to prepare a special recording unit that records information about broadcast time and broadcasting station as well as a television broadcast.

In addition, from the foregoing effects, the user can obtain his or her desired information with high probability. In addition, the user is stressfreely guided from obtaining information to buying a product. Thus, as an effect of the embodiment of the present invention, the information providing side does not lose business chance.