Patent Description:
The disclosed exemplary embodiments relate to a device, information processing apparatus and method. For example, embodiments relate to a terminal device, an information processing device, an object identifying method, a program, and an object identifying system.

With the advancement of image recognition technology in recent years, it has become possible to identify the position and attitude of an object in an image input from a camera, through matching of image feature quantities. One application of such object identification is an augmented reality (AR) application. In the AR application, a variety of information (e.g., advertising information, navigation information, or information for games) can be additionally displayed in an image of a building, a road, or other objects existing in the real world such that the information is associated with the object.

Japanese Patent Application Publication No. <CIT> proposes a feature extraction algorithm for identifying objects that has increased robustness against changes in the viewpoint, changes in luminance, and noise. Furthermore, Oezuysal proposes a feature extraction algorithm called "Random Ferns" that can operate at fast speed with a lower processing cost (See <NPL>).

As described above, feature extraction algorithms for identifying an object in an image come in a variety of types. However, typically, the higher the identification performance that an algorithm can realize, the higher the processing cost. Therefore, when object identification is performed on a device with a small amount of processing resources, such as a portable terminal, for example, there is a restriction on the identification performance (e.g., the accuracy of identification and the number of objects that can be identified concurrently). Meanwhile, when an image in each frame is transferred to a server with abundant processing resources to cause it to perform object identification, a delay caused by the wait time for a response from the server could hinder the rapid response of the application.

<NPL>, discloses an outdoors augmented reality system for mobile phones that matches camera-phone images against a large database of location-tagged images using an image retrieval algorithm. The system is built as a client-server system in which relevant features are selected and transmitted from the server to the client device to enable recognizing objects proximate to the client device.

In light of the foregoing, it is desirable to provide a terminal device, an information processing device, an object identifying method, a program, and an object identifying system that can achieve higher object identification performance in a device with a small amount of processing resources.

As described above, it is possible to achieve higher object identification performance in a device with a small amount of processing resources.

Hereinafter, exemplary embodiments will be described in detail with reference to the appended drawings.

The exemplary embodiments will be described in the following order.

<FIG> is an explanatory diagram illustrating an overview of an object identifying system to which the technology disclosed in this specification can be applied. Referring to <FIG>, an object identifying system <NUM> in accordance with an exemplary embodiment is shown. The object identifying system <NUM> includes a terminal device <NUM> and a dictionary server <NUM>.

The terminal device <NUM> is a device that identifies an object in an image captured by an imaging device. The terminal device <NUM> can be a portable terminal carried by a user, such as a smart phone or a PDA (Personal Digital Assistant). Alternatively, the terminal device <NUM> can be other types of device such as a PC (Personal Computer), a digital information home appliance, a game machine, or a robot used for operations. The imaging device can be incorporated in the terminal device <NUM>. Alternatively, the imaging device can be provided outside the terminal device <NUM> and connected to the terminal device <NUM> by a cable or radio.

The terminal device <NUM>, in identifying an object in an image, checks a feature quantity extracted from the image against a feature dictionary that is a set of known feature quantities for one or more objects. Then, the terminal device <NUM>, on the basis of a score calculated by the checking (hereinafter referred to as a "checked score"), identifies which object is in the image. Note that in this specification, if a checked score for the feature quantity of a known object is "high," it means that there is a high possibility that the object is in the input image. For example, when the difference between a known feature quantity and the feature quantity of an input image at a particular position and attitude is close to zero, there is a high possibility that an object corresponding to the feature quantity is in the input image at that position and attitude. Such a circumstance will be referred to as a "high" checked score (even though the evaluated value of the difference is small). That is, the terminal device <NUM> can even identify the position and attitude of an object in an image. Various applications that use the result of such object identification can be mounted on the terminal device <NUM>. This specification will mainly describe an example in which an AR application that uses the result of the object identification is mounted on the terminal device <NUM>. However, in the terminal device <NUM>, an application having a different objective (e.g., monitoring, recognizing the environment, or assisting in operations) can use the result of object identification.

The dictionary server <NUM> may be an information processing device that provides a feature dictionary for object identification to the terminal device <NUM>. The dictionary server <NUM> communicates with the terminal device <NUM> over a network <NUM>. The network <NUM> can be any types of network, such as the Internet, a provider network, or an intranet. In this exemplary embodiment, the dictionary server <NUM> receives an image from the terminal device <NUM>. Then, the dictionary server <NUM> identifies an object in the received image, and provides a feature dictionary in accordance with the result of identification to the terminal device <NUM>.

<FIG> is an explanatory diagram illustrating an image that can be displayed on a screen of the terminal device <NUM>, consistent with an exemplary embodiment. For example, the image illustrated in <FIG> may be an image of an AR application. Referring to <FIG>, an image of a building <NUM>, which exists in the real space, is displayed on the screen of the terminal device <NUM>. In addition, additive information <NUM> is overlaid on the image. The additive information <NUM> is information indicating the name and rating of a restaurant operated in the building <NUM>. Such additive information is selected on the basis of the result of object identification in the terminal device <NUM>, and is then overlaid on the image at a position corresponding to the object in the image. In this exemplary embodiment, a database of additive information that is overlaid on the image in this manner is also provided from the dictionary server <NUM> to the terminal device <NUM>.

<FIG> is a block diagram showing an exemplary hardware configuration of the terminal device <NUM> in accordance with this exemplary embodiment. Referring to <FIG>, the terminal device <NUM> includes an imaging unit <NUM>, a sensor unit <NUM>, an input unit <NUM>, a tangible, non-transitory computer-readable medium, an example of which is a storage unit <NUM>, a display unit <NUM>, a communication unit <NUM>, a bus <NUM>, and a control unit <NUM>.

The imaging unit <NUM> is a camera module that captures images. The imaging unit <NUM> generates an input image for object identification by imaging the real space using an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

The sensor unit <NUM> is a sensor group that assists in the recognition of the position and attitude of the terminal device <NUM>. For example, the sensor unit <NUM> can include a GPS sensor that receives a GPS (Global Positioning System) signal and measures the latitude, longitude, and altitude of the terminal device <NUM>. In addition, the sensor unit <NUM> can include a positioning sensor that measures the position of the terminal device <NUM> on the basis of the intensity of a radio signal received from a wireless access point. Further, the sensor unit <NUM> can include a gyro sensor that measures the tilt angle of the terminal device <NUM>, an accelerometer that measures the three-axis acceleration, or a geomagnetic sensor that measures the orientation. Note that when the terminal device <NUM> has a position estimation function and an attitude estimation function based on the image recognition, the sensor unit <NUM> can be omitted from the configuration of the terminal device <NUM>.

The input unit <NUM> is an input device used for a user to operate the terminal device <NUM> or to input information to the terminal device <NUM>. The input device <NUM> can include a keyboard, a keypad, a mouse, a button, a switch, a touch panel, or the like, for example. The input unit <NUM> can also include a gesture recognition module that recognizes a gesture of a user in an input image. Further, the input unit <NUM> can also include a line-of-sight detection module that detects the direction of the line of sight of a user wearing an HMD (Head Mounted Display) as a user input.

The storage unit <NUM> includes a storage medium such as semiconductor memory or a hard disk, and stores programs and data to be used for processes performed by the terminal device <NUM>. For example, the storage unit <NUM> temporarily stores an input image generated by the imaging unit <NUM> and sensor data measured by the sensor unit <NUM>. The storage unit <NUM> also stores data received form the dictionary server <NUM> via the communication unit <NUM>. Examples of data received from the dictionary server <NUM> are described in detail below.

The display unit <NUM> is a display module including an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or a CRT (Cathode Ray Tube). The display unit <NUM> displays an input image captured by the imaging unit <NUM>, or an image of an application that uses the result of object identification (e.g., an image of an AR application exemplarily shown in <FIG>) on the screen. The display unit <NUM> can be a part of the terminal device <NUM> or can be provided outside the terminal device <NUM>. Alternatively, the display unit <NUM> can be an HMD worn by a user.

The communication unit <NUM> is a communication interface that mediates the communication between the terminal device <NUM> and the dictionary server <NUM>. The communication unit <NUM> supports a given radio communication protocol or wire communication protocol, and establishes a communication connection with the dictionary server <NUM>. Accordingly, it becomes possible for the terminal device <NUM> to transmit an image to the dictionary server <NUM> and to receive a feature dictionary from the dictionary server <NUM>.

The bus <NUM> mutually connects the imaging unit <NUM>, the sensor unit <NUM>, the input unit <NUM>, the storage unit <NUM>, the display unit <NUM>, the communication unit <NUM>, and the control unit <NUM>.

The control unit <NUM> corresponds to a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The control unit <NUM> causes a variety of functions of the terminal device <NUM> described below to operate by executing the programs stored in the storage unit <NUM> or another storage medium.

<FIG> is a block diagram showing an exemplary configuration of the logical function implemented by the storage unit <NUM> and the control unit <NUM> of the terminal device <NUM> shown in <FIG>. Referring to <FIG>, the terminal device <NUM> includes an image acquisition unit <NUM>, a transmitting unit <NUM>, a receiving unit <NUM>, a dictionary cache <NUM>, an identification unit <NUM>, an additive information cache <NUM>, and a display control unit <NUM>.

The image acquisition unit <NUM> acquires an input image generated by the imaging unit <NUM>. Then, the image acquisition unit <NUM> sequentially outputs the acquired input image to the transmitting unit <NUM> and the identification unit <NUM>.

The transmitting unit <NUM>, when a predetermined trigger event is detected, transmits the input image input from the image acquisition unit <NUM> to the dictionary server <NUM> via the communication unit <NUM>. The dictionary server <NUM> is a server that holds a feature dictionary, which is a set of image feature quantities for a known object, as described above.

The trigger event that is a trigger for the transmission of the input image from the transmitting unit <NUM> can be one or more of the following events, for example:.

Periodic transmission of the input images can be adopted when it is desirable to continuously perform object identification independently of the content of the images. Transmission of the input image in response to a user instruction can be adopted when, for example, a user desires that an object displayed on the screen be identified or tracked. The other trigger events are events that are based on the presumption that there is a high possibility that a new object is in the image; when the input image is transmitted in response to such trigger event and a feature dictionary is provided from the dictionary server <NUM>, it becomes possible to adequately identify a new object.

The transmitting unit <NUM> can also transmit to the dictionary server <NUM> assistance information for assisting in the acquisition of a feature dictionary by the dictionary server <NUM>, together with the input image. Examples of the assistance information can include at least one of the position (of the terminal device <NUM> or the imaging device) or the date and time of when the input image was captured, and the capability information of the terminal device <NUM>. The position and the date and time can be used in filtering a feature dictionary in the dictionary server <NUM>. The capability information of the terminal device <NUM> can be used in determining the data volume of a feature dictionary to be provided to the terminal device <NUM> from the dictionary server <NUM>. Utilization of such assistance information is described in further detail below.

The receiving unit <NUM>, after the input image is transmitted from the transmitting unit <NUM> to the dictionary server <NUM>, receives from the dictionary server <NUM> a feature dictionary acquired in the dictionary server <NUM> in accordance with the result of identification of an object in the input image. The feature dictionary received by the receiving unit <NUM> is a dictionary with a less data volume than the feature dictionary of the dictionary server <NUM>. How the feature dictionary provided to the terminal device <NUM> is acquired in the dictionary server <NUM> is described in further detail below.

The receiving unit <NUM>, upon receiving a feature dictionary, causes the dictionary cache <NUM> to store the received feature dictionary. In this exemplary embodiment, each feature quantity included in the feature dictionary is associated with an identifier for uniquely identifying an object (hereinafter referred to as an "object ID"). If the receiving unit <NUM> has newly received a feature quantity with the same object ID as that of the feature quantity stored in the dictionary cache <NUM>, the feature quantity in the dictionary cache <NUM> can be updated to the newly received feature quantity. In addition, the receiving unit <NUM> can add a reception time stamp to each feature quantity received, and automatically delete from the dictionary cache <NUM> a feature quantity that has been stored over a predetermined period of time since the addition of the reception time stamp. Alternatively, a feature quantity can be deleted from the dictionary cache <NUM> in accordance with, as a trigger, a specific amount of a movement of the terminal device <NUM> or a frame-out movement of the associated object going out of the image.

Further, in this exemplary embodiment, the receiving unit <NUM> receives from the dictionary server <NUM> an additive information database acquired in the dictionary server <NUM> in accordance with the result of objet identification. The additive information database received by the receiving unit <NUM> is a database with a less data volume than the additive information database stored in the dictionary server <NUM> in advance. The receiving unit <NUM> causes the additive information cache <NUM> to sore the received additive information database.

The dictionary cache <NUM> stores a feature dictionary received by the receiving unit <NUM>, using the storage unit <NUM> shown in <FIG>. The feature dictionary stored in the dictionary cache <NUM> is referred to when object identification is performed by the identification unit <NUM>.

The identification unit <NUM> extracts the feature quantity of an input image input from the image acquisition unit <NUM>, and checks the extracted feature quantity against the feature dictionary stored in the dictionary cache <NUM>, thereby identifying an object in the input image. A feature extraction algorithm used by the identification unit <NUM> can be, for example, Random Ferns described in the aforementioned document or SURF described in "<NPL>). Such algorithms are "lightweight" algorithms that can operate at fast speed with a lower processing cost. As a result of object identification performed by the identification unit <NUM>, the object ID of an object in the input image, and the position and attitude of the object in the input image are derived. Then, the identification unit <NUM> outputs the result of object identification to the display control unit <NUM>.

The additive information cache <NUM> stores an additive information database received by the receiving unit <NUM>, using the storage unit <NUM> shown in <FIG>. The display control unit <NUM> described next selects additive information to be overlaid on the input image from the additive information database stored in the additive information cache <NUM>.

The display control unit <NUM> acquires additive information associated with the object identified by the identification unit <NUM> from the additive information database stored in the additive information cache <NUM>, and overlays the acquired additive information on the input image, thereby generating an output image. Then, the display control unit <NUM> outputs the generated output image to the display unit <NUM>.

The additive information overlaid on the input image can be any information. For example, the additive information overlaid on the input image can be advertising information, rating information, and the like associated with a building in the input image as exemplarily shown in <FIG>. Other examples of the additive information are described in further detail below.

<FIG> is a block diagram showing an exemplary hardware configuration of the dictionary server <NUM> in accordance with this exemplary embodiment. Referring to <FIG>, the dictionary server <NUM> includes a tangible, non-transitory computer-readable medium, an example of which is storage unit <NUM>, a communication unit <NUM>, a bus <NUM>, and a control unit <NUM>.

The storage unit <NUM> includes a tangible, non-transitory storage medium such as semiconductor memory or a hard disk, and stores programs and data to be for processes performed by the dictionary server <NUM>. The storage unit <NUM> can have a higher storage capacity than the storage unit <NUM> of the terminal device <NUM>. The storage unit <NUM> stores in advance a feature dictionary and an additive information database described below.

The communication unit <NUM> is a communication interface that mediates the communication between the dictionary server <NUM> and the terminal device <NUM>. The communication unit <NUM> supports a given radio communication protocol or wire communication protocol, and establishes a communication connection with the terminal device <NUM>. Accordingly, it becomes possible for the dictionary server <NUM> to receive an image from the terminal device <NUM> and to transmit a feature dictionary and an additive information database to the terminal device <NUM>.

The bus <NUM> mutually connects the storage unit <NUM>, the communication unit <NUM> and the control unit <NUM>.

The control unit <NUM> corresponds to a processor such as a CPU or a DSP. The control unit <NUM> can have higher operation performance than the control unit <NUM> of the terminal device <NUM>. The control unit <NUM> causes a variety of functions of the dictionary server <NUM> described below to operate by executing the programs stored in the storage unit <NUM> or another storage medium.

<FIG> is a block diagram showing an exemplary configuration of the logical function implemented by the storage unit <NUM> and the control unit <NUM> of the dictionary server <NUM> shown in <FIG>. Referring to <FIG>, the dictionary server <NUM> includes a receiving unit <NUM>, a feature dictionary <NUM> for a first algorithm (Arg <NUM>), a feature dictionary <NUM> for a second algorithm (Arg <NUM>), an identification unit <NUM>, a dictionary acquisition unit <NUM>, an additive information database (DB) <NUM>, an additive information acquisition unit <NUM>, and a transmitting unit <NUM>.

The receiving unit <NUM> waits for an input image transmitted from the terminal device <NUM>. The receiving unit <NUM>, upon receiving an input image via the communication unit <NUM>, outputs the received input image to the identification unit <NUM>. In addition, the receiving unit <NUM>, upon receiving the aforementioned assistance information together with the input image, outputs the assistance information to the identification unit <NUM> and the dictionary acquisition unit <NUM>.

Each of the feature dictionary (Arg <NUM>) <NUM> and the feature dictionary (Arg <NUM>) <NUM> is a set of feature quantities stored in the storage unit <NUM> in advance. Each feature quantity in the feature dictionary (Arg <NUM>) <NUM> is extracted from a known object image in accordance with a first algorithm. Likewise, each feature quantity in the feature dictionary (Arg <NUM>) <NUM> is extracted from the known object image in accordance with a second algorithm. Typically, the first algorithm is a feature extraction algorithm that enables object identification with higher accuracy than the second algorithm. Meanwhile, the second algorithm is a feature extraction algorithm that can be executed at faster speed than the first algorithm. The first algorithm can be, for example, the feature extraction algorithm described in <CIT> above. Alternatively, the first algorithm may be an algorithm described in, for example, "<NPL>), or an algorithm described in "<NPL>). The second algorithm may a feature extraction algorithm (e.g., Random Ferns or SURF) that is also used for object identification performed by the identification unit <NUM> of the terminal device <NUM> described above. In the following description, the first algorithm will be referred to as a high-accuracy algorithm and the second algorithm will be referred to as a "lightweight" algorithm.

A feature quantity in the feature dictionary (Arg <NUM>) <NUM> and a feature quantity in the feature dictionary (Arg <NUM>) <NUM> are linked together using a common object ID. That is, a feature quantity for an identical object ID is included in both the feature dictionary (Arg <NUM>) <NUM> and the feature dictionary (Arg <NUM>) <NUM>.

<FIG> is an explanatory diagram illustrating an exemplary feature dictionary stored in the dictionary server <NUM>. Referring to <FIG>, the feature dictionary (Arg <NUM>) <NUM> includes a feature quantity for each of a plurality of objects including eight objects B<NUM> to B<NUM>, extracted from known object images in accordance with a high-accuracy algorithm. Each object is assigned a name. Likewise, the feature dictionary (Arg <NUM>) <NUM> includes a feature quantity for each of the plurality of objects including eight objects B<NUM> to B<NUM>, extracted in accordance with a "lightweight" algorithm. The object ID of each object is common to the two feature dictionaries. That is, the feature quantity for the object B<NUM> in the feature dictionary <NUM> is the same as the feature quantity for the object B<NUM> in the feature dictionary <NUM>, namely, a feature quantity extracted from an image of a building A.

The feature dictionaries <NUM> and <NUM> are not limited to the examples shown in <FIG>, and can include additional data. In some of the examples described below, the feature dictionary <NUM> includes additional data for assisting in the efficient acquisition of a feature dictionary to be provided to the terminal device <NUM>. Note that instead of (or in addition to) the feature dictionary <NUM>, the feature dictionary <NUM> can include such additional data.

The identification unit <NUM> extracts the feature quantity of an input image received by the receiving unit <NUM> in accordance with a high-accuracy algorithm, and checks the extracted feature quantity against the feature dictionary (Arg <NUM>) <NUM>, thereby identifying one or more objects in the input image. Then, the identification unit <NUM> outputs the object ID and the checked score of the identified object(s) to the dictionary acquisition unit <NUM> and the additive information acquisition unit <NUM>.

The dictionary acquisition unit <NUM> acquires a feature dictionary to be provided to the terminal device <NUM> in accordance with the result of identification performed by the identification unit <NUM>. The feature dictionary acquired by the dictionary acquisition unit <NUM> is a subset of the feature dictionary (Arg <NUM>) <NUM> that has a less data volume than the feature dictionary (Arg <NUM>) <NUM> and the feature dictionary (Arg <NUM>) <NUM> described above. Hereinafter, four examples of the acquisition of a dictionary subset by the dictionary acquisition unit <NUM> will be described with reference to <FIG>. The First Example and the Fourth Example are embodiments not covered by the present invention defined in the appended claims.

<FIG> is an explanatory diagram illustrating a first example of a dictionary subset acquired by the dictionary acquisition unit <NUM>. Referring to <FIG>, the rank of a checked score, which is obtained as a result of identification performed by the identification unit <NUM>, is shown for each object ID in the feature dictionary (Arg <NUM>) <NUM>. In the example of <FIG>, the checked score of the object B<NUM> is the highest and ranks first. The checked score of the object B<NUM> is the second highest and ranks second. The checked score of the object B<NUM> ranks k-th. The dictionary acquisition unit <NUM> acquires from the feature dictionary (Arg <NUM>) <NUM> the feature quantities for the objects associated with ranks that exceed a threshold value, for example, whose checked scores rank first to k-th. Then, the dictionary acquisition unit <NUM> outputs, as a feature dictionary to be provided to the terminal device <NUM>, a subset 242a of a feature dictionary including the acquired feature quantities to the transmitting unit <NUM>.

Note that the volume of data (e.g., the number k of feature quantities) to be included in the subset 242a of the feature dictionary can be dynamically determined in accordance with the capability information of the terminal device <NUM> received as the assistance information from the terminal device <NUM>. Capability of the terminal device <NUM> can be expressed by, for example, the number of pieces of processable data, the number of cores of the processor, the memory capacity, or the like.

<FIG> is an explanatory diagram illustrating a second example of a dictionary subset acquired by the dictionary acquisition unit <NUM>. In the second example, the feature dictionary (Arg <NUM>) <NUM> has, in addition to the "object ID," "name," and "feature quantity" for each object, predefined data called "co-occurring object. " The "co-occurring object" represents a list of objects that have a high possibility of co-occurring with each object. In this specification, a state in which a second object exists near a first object is rephrased as: the first object and the second object "co-occur. " In the example of <FIG>, co-occurring objects of the object B<NUM> are the object B<NUM> and the object B<NUM>. This means that when an input image is identified as including the object B<NUM> (a traffic light D), it is highly probable that the object B<NUM> (a vehicle E) or the object B<NUM> (a road sign) appear in the following input image. Using such data, the dictionary acquisition unit <NUM> can acquire not only the feature quantity for an object that is already in the input image but also the feature quantity for an object that is predicted to appear in the following input image. In the example of <FIG>, the dictionary acquisition unit <NUM> acquires, in addition to the feature quantity for the object B<NUM> whose checked score ranks high, the feature quantities for the objects B<NUM> and B<NUM> that are predicted to appear in the following input image from the feature dictionary (Arg <NUM>) <NUM>. Then, the dictionary acquisition unit <NUM> outputs a subset 242b of a feature dictionary including the acquired feature quantities to the transmitting unit <NUM>.

<FIG> is an explanatory diagram illustrating a third example of a dictionary subset acquired by the dictionary acquisition unit <NUM>. In the third example also, the dictionary acquisition unit <NUM> acquires not only the feature quantity for an object that is already in the input image but also the feature quantity for an object that is predicted to appear in the following input image. In the third example, the object that is predicted to appear in the following input image is an object that is determined, from positional data, to be located near the object that is already in the input image. Referring to <FIG>, the feature dictionary (Arg <NUM>) <NUM> has positional data (latitude and longitude, or other coordinate data) for each object. For example, the position of the object B<NUM> is X<NUM>, the position of the object B<NUM> is X<NUM>, and the position of the object B<NUM> is X<NUM>. Among them, the distance between the position X<NUM> and the position X<NUM> is less than a threshold value, e.g., threshold D. The dictionary acquisition unit <NUM>, when the checked score of the object B<NUM> ranks high, acquires not only the feature quantity for the object B<NUM> but also the feature quantity for the object B<NUM> located near the object B<NUM> from the feature dictionary (Arg <NUM>) <NUM> on the basis of the positional data. Then, the dictionary acquisition unit <NUM> outputs a subset 242c of a feature dictionary including the acquired feature quantities to the transmitting unit <NUM>.

Note that the positional data exemplarily shown in <FIG> can also be used for filtering the feature dictionary. For example, the dictionary acquisition unit <NUM> can acquire only the feature quantity for an object located near the terminal device <NUM> among objects whose checked scores rank first to k-th. Alternatively, the identification unit <NUM> can use only the feature quantity for an object located near the terminal device <NUM> as the target to be checked against the feature quantity extracted from the input image. The position of the terminal device <NUM> can be recognized from the assistance information received from the terminal device <NUM>.

The exemplary processes described above are not limited to the identification of objects included within the feature dictionary and disposed within a threshold distance of an identified object, e.g., object B<NUM>. For example, as depicted in <FIG>, dictionary acquisition unit <NUM> may identify object B<NUM> associated with "Building A," determine that the checked score of the object B<NUM> ranks high, and subsequently output object B<NUM> as a portion of the subset 242c. In additional embodiments, dictionary acquisition unit <NUM> may obtain information associated with additional objects of potential relevance to object B<NUM>, or that are related to object B<NUM>. The obtained information may include, but is not limited to, feature quantities of the additional objects, object identifiers associated with the additional objects, and positional data associated with the additional objects. For example, such additional objects may be landmarks near object B<NUM>, buildings related to occupants of Building A, infrastructure elements disposed near object B<NUM>, and any additional or alternate object related to object B<NUM>, as would be apparent to one of skill in the art.

In such an embodiment, dictionary acquisition unit <NUM> may select one or more of the additional objects for inclusion within subset 242c, and may output information associated with the additional objects (e.g., feature quantities and object identifiers) to transmitting unit <NUM>. Additionally or alternatively, dictionary acquisition unit <NUM> may determine whether geographic positions of the additional objects fall within the threshold distance of object B<NUM>, and may subsequently incorporate, into subset 242c, one or more of the additional elements that are disposed within the threshold distance of object B<NUM>.

In such embodiments, dictionary acquisition unit <NUM> may obtain feature identifiers associated with the additional objects from the obtained information, as outlined above. In additional embodiments, dictionary acquisition unit <NUM> may initially determine whether information associated with the additional objects is included within the feature dictionary. Dictionary acquisition unit <NUM> may subsequently rely on the obtained information with the feature dictionary does not include the additional objects. In such an embodiment, dictionary acquisition unit <NUM> may update the feature dictionary to include the information associated with one or more of the additional objects.

<FIG> is an explanatory diagram illustrating a fourth example of a dictionary subset acquired by the dictionary acquisition unit <NUM>. Referring to <FIG>, the feature dictionary (Arg <NUM>) <NUM> has, in addition to the "object ID," "name," and "feature quantity" for each object, data called "luminance conditions. " The "luminance conditions" can be a classification indicating the luminance conditions of when a known object image was captured. The luminance conditions are distinguished from each other in accordance with the time-related conditions, i.e., a time period or season of when an image was captured, or the weather-related conditions. The feature dictionary (Arg <NUM>) <NUM> can include a plurality of types of feature quantities extracted from images that have been obtained by capturing an identical object under different luminance conditions. In the example of <FIG>, for the object B<NUM>, a feature quantity corresponding to a luminance condition L1 (e.g., "morning" or "sunny"), a feature quantity corresponding to a luminance condition L2 (e.g., "daytime" or "cloudy"), and a feature quantity corresponding to a luminance condition L3 (e.g., "late afternoon" or "rainy") are included in the feature dictionary (Arg <NUM>) <NUM>. Likewise, for the object B<NUM>, feature quantities corresponding to the luminance conditions L1, L2, and L3 are also included in the feature dictionary (Arg <NUM>) <NUM>. As described above, when a plurality of feature quantities for an identical object captured under different luminance conditions are included in the feature dictionary (Arg <NUM>) <NUM>, object identification performed by the identification unit <NUM> will be less susceptible to the influence of the difference in the way in which an object looks different due to the difference in luminance conditions. In the example of <FIG>, when an input image including the object B<NUM> is received, for example, a score obtained by checking the feature quantity of the input image against the feature quantity corresponding to each of the luminance conditions L1 and L2 is low, but a score obtained by checking the feature quantity of the input image against the feature quantity corresponding to the luminance condition L3 is high. Thus, the feature quantity for the object B<NUM> is appropriately included in a subset 242d of a feature dictionary.

Note that the luminance condition data exemplarily shown in <FIG> can also be used for filtering the feature dictionary. For example, the dictionary acquisition unit <NUM> can exclude from the subset 242d of feature quantities a feature quantity corresponding to a luminance condition that is different from the luminance condition to which the date and time of when the input image was captured belong among the feature quantities of objects whose checked scores rank first to k-th. Alternatively, the identification unit <NUM> can use only a feature quantity corresponding to the luminance condition to which the date and time of when the input image was captured belong as the target to be checked against the feature quantity extracted from the input image. The date and time of when the input image was captured can be recognized from the assistance information received from the terminal device <NUM>.

The additive information DB <NUM> is a set of additive information associated with objects existing in the real space. In the field of AR, additive information is also referred to as "annotation. " <FIG> is an explanatory diagram illustrating exemplary data stored in the additive information DB. Referring to <FIG>, in the additive information DB <NUM>, additive information including two data items: "type" and "content" are associated with the object ID of each object. The "type" refers to the type of individual additive information. The "content" can be text data, graphic data, image data, or the like as the entity of the individual additive information. In the example of <FIG>, advertising information and rating information are associated with the object B<NUM>. In addition, advertising information, attention-seeking information, and vehicle type information are associated with the objects B<NUM>, B<NUM>, and B<NUM>, respectively.

The additive information acquisition unit <NUM> acquires from the additive information DB <NUM> additive information to be provided to the terminal device <NUM> in accordance with the result of identification performed by the identification unit <NUM>, and generates a subset of an additive information database with a less data volume. Then, the additive information acquisition unit <NUM> outputs the generated subset of the additive information database to the transmitting unit <NUM>. The additive information acquisition unit <NUM> typically acquires from the additive information DB <NUM> a set of additive information including object IDs that are common to those of the subset of the feature dictionary acquired by the dictionary acquisition unit <NUM>. That is, the additive information acquisition unit <NUM> can also acquire from the additive information DB <NUM> a set of additive information corresponding to the objects whose checked scores rank first to k-th. Further, the additive information acquisition unit <NUM> can also acquire from the additive information DB <NUM> additive information corresponding to an object that is predicted to appear in the following input image.

The transmitting unit <NUM> transmits the subset of the feature dictionary acquired by the dictionary acquisition unit <NUM> to the terminal device <NUM> via the communication unit <NUM>. In that case, the transmitting unit <NUM> can determine if the identified object includes a new object that is different from the objects identified in the past and can, only when a new object is identified, transmit to the terminal device <NUM> a subset of a feature dictionary for the new object. Accordingly, when an identical object continuously appears in the input images, redundant transmission of feature dictionaries is omitted, whereby the traffic load is reduced. In addition, the transmitting unit <NUM> transmits to the terminal device <NUM> a subset of the additive information database generated by the additive information acquisition unit <NUM>. The subset of the additive information database can also be transmitted only when a new object is identified.

Next, two variations of the dictionary server <NUM> will be described.

<FIG> is a block diagram showing an exemplary configuration of the logical function of the dictionary server <NUM> in accordance with a first variation. Referring to <FIG>, the dictionary server <NUM> includes a receiving unit <NUM>, a feature dictionary <NUM> for a high-accuracy algorithm (Arg <NUM>), a feature dictionary <NUM> for a "lightweight" algorithm (Arg <NUM>), an identification unit <NUM>, a dictionary acquisition unit <NUM>, an additive information DB <NUM>, an additive information acquisition unit <NUM>, and a transmitting unit <NUM>.

The receiving unit <NUM> waits for an input image transmitted from the terminal device <NUM>. The receiving unit <NUM>, upon receiving an input image via the communication unit <NUM>, outputs the received input image to the identification unit <NUM> and the dictionary acquisition unit <NUM>.

The identification unit <NUM> extracts the feature quantity of the input image received by the receiving unit <NUM> in accordance with a high-accuracy algorithm, and checks the extracted feature quantity against the feature dictionary (Arg <NUM>) <NUM>, thereby identifying one or more objects in the input image. In addition, the identification unit <NUM> identifies the position and attitude of the object(s) in the input image. Then, the identification unit <NUM> outputs the object ID, position, and attitude of the identified object(s) to the dictionary acquisition unit <NUM>. In addition, the identification unit <NUM> outputs the object ID of the identified object(s) to the additive information acquisition unit <NUM>.

The dictionary acquisition unit <NUM> acquires a feature dictionary to be provided to the terminal device <NUM> in accordance with the result of identification performed by the identification unit <NUM>. More specifically, the dictionary acquisition unit <NUM> first recognizes the position of the object identified by the identification unit <NUM> in the input image, and cuts a partial image of an area including the object out of the input image. Then, the dictionary acquisition unit <NUM> extracts a feature quantity from the cut-out partial image in accordance with a "lightweight" algorithm. The dictionary acquisition unit <NUM> associates the object ID input from the identification unit <NUM> with the thus extracted feature quantity of each object, and generates a feature dictionary for the "lightweight" algorithm. In this case, the feature dictionary <NUM> for the "lightweight" algorithm (Arg <NUM>) can be omitted from the configuration of the dictionary server <NUM>. Instead, the dictionary acquisition unit <NUM> can generate a new feature dictionary by adding the feature quantity extracted from the partial image (e.g., additionally learned feature quantity) to a subset of feature quantities acquired from the feature dictionary <NUM>. The dictionary acquisition unit <NUM> outputs the thus generated feature quantity dictionary to the transmitting unit <NUM>, and causes the feature dictionary to be transmitted from the transmitting unit <NUM> to the terminal device <NUM>.

Further, the dictionary acquisition unit <NUM> can generate variations of the feature quantity extracted in accordance with the "lightweight" algorithm by varying a parameter such as the color, luminance, or the degree of blur of the feature quantity. Such variations of the feature quantity can also form a new feature dictionary.

<FIG> and <FIG> are explanatory diagrams each illustrating the generation of a feature dictionary with the dictionary acquisition unit <NUM> in accordance with the first variation. Referring to <FIG>, objects B<NUM> and B<NUM> in an input image Im1 are identified using the feature dictionary <NUM> and a high-accuracy algorithm. Then, as shown in <FIG>, the dictionary acquisition unit <NUM> cuts a partial image A1 including the object B<NUM> and a partial image A2 including the object B<NUM> out of the input image Im1. Then, the dictionary acquisition unit <NUM> extracts a feature quantity from each of the partial image A1 and the partial image A2 in accordance with a "lightweight" algorithm. In addition, the dictionary acquisition unit <NUM> generates variations of the extracted feature quantity by varying a parameter such as the color or luminance of the feature quantity. Further, the dictionary acquisition unit <NUM> forms a new feature dictionary 242d to be provided to the terminal device <NUM> by adding an object ID to each feature quantity.

According to the first variation, a feature dictionary that is dynamically generated from the input image by the dictionary server <NUM> is provided to the terminal device <NUM>. Such a feature dictionary is a feature dictionary with a less data volume, including feature quantities that are particularly adapted to the environment (e.g., imaging environment or luminance environment) in which the terminal device <NUM> is located. Therefore, the terminal device <NUM> can identify an object in the input image as well as the position and attitude of the object with high accuracy and a low processing cost.

In the aforementioned example, a subset of a feature dictionary for a "lightweight" algorithm is provided to the terminal device <NUM> from the dictionary server <NUM>. However, as in a second variation described below, the dictionary server <NUM> can provide a subset of a feature dictionary for a high-accuracy algorithm to the terminal device <NUM>.

<FIG> is a block diagram showing an exemplary configuration of the logical function of the dictionary server <NUM> in accordance with the second variation. Referring to <FIG>, the dictionary server <NUM> includes a receiving unit <NUM>, a feature dictionary <NUM> for a high-accuracy algorithm (Arg1), an identification unit <NUM>, a dictionary acquisition unit <NUM>, an additive information DB <NUM>, an additive information acquisition unit <NUM>, and a transmitting unit <NUM>.

The dictionary acquisition unit <NUM> acquires from the feature dictionary (Arg <NUM>) <NUM> a subset of a feature dictionary to be provided to the terminal device <NUM> in accordance with the result of identification performed by the identification unit <NUM>. For example, <FIG> again shows the ranks of checked scores obtained as a result of identification performed by the identification unit <NUM>. In the example of <FIG>, the checked score of the object B<NUM> ranks first, the checked score of the object B<NUM> ranks second, and the checked score of the object B<NUM> ranks k-th. The dictionary acquisition unit <NUM> acquires from the feature dictionary (Arg <NUM>) <NUM> the feature quantities for the objects whose checked scores rank first to k-th, for example, and forms a subset 240a of a feature dictionary including the acquired feature quantities. Then, the transmitting unit <NUM> transmits the subset 240a of the feature dictionary to the terminal device <NUM>.

When the second variation is adopted, the identification unit <NUM> of the terminal device <NUM> extracts a feature quantity from the input image in accordance with a high-accuracy algorithm, and checks the extracted feature quantity against a subset of a feature dictionary provided from the dictionary server <NUM>. In this case, in comparison with an example in which a "lightweight" algorithm is used, the processing cost of the terminal device <NUM> needed for extraction of feature quantities is higher. However, the dictionary cache <NUM> stores not the entire feature dictionary of the dictionary server <NUM> but only a subset of the feature dictionary. Therefore, in comparison with a case in which the terminal device <NUM> has the entire feature dictionary, the processing cost for checking feature quantities in the terminal device <NUM> and the consumed memory resources can be significantly lower.

Heretofore, an example in which the transmitting unit <NUM> of the terminal device <NUM> transmits an input image to the dictionary server <NUM> has been mainly described. However, the aforesaid example is not covered by the present invention defined in the appended claims. In the present invention, the transmitting unit <NUM> of the terminal device <NUM>, instead of transmitting an input image, transmits to the dictionary server <NUM> a feature quantity extracted from the input image by the identification unit <NUM>. In that case, the identification unit <NUM> of the dictionary server <NUM> checks the feature quantity of the input image received by the receiving unit <NUM> against the feature dictionary (Arg <NUM>) <NUM>.

<FIG> is a flowchart showing an exemplary flow of processes performed by the terminal device <NUM> in accordance with this exemplary embodiment.

Referring to <FIG>, first, the image acquisition unit <NUM> of the terminal device <NUM> acquires an input image (step S102). Next, the transmitting unit <NUM> determines if a predetermined trigger event described above (e.g., arrival of a periodic timing or user instruction) has been detected (step S104). Herein, if a trigger event has not been detected, the processes of the following steps S106 to S110 are skipped. Meanwhile, if a trigger event has been detected, the transmitting unit <NUM> transmits the input image (and assistance information if necessary) to the dictionary server <NUM> (step S106). Then, the receiving unit <NUM> receives a feature dictionary from the dictionary server <NUM> (step S108). The feature dictionary received herein is stored in the dictionary cache <NUM>. In addition, the receiving unit <NUM> receives an additive information DB from the dictionary server <NUM> (step S110). The additive information DB received herein is stored in the additive information cache <NUM>. Next, the identification unit <NUM> identifies an object in the input image using the feature dictionary in the dictionary cache <NUM> (step S112). Next, the display control unit <NUM> acquires from the additive information cache <NUM> additive information associated with the object identified by the identification unit <NUM>, and overlays the acquired additive information on the input image, thereby generating an output image (step S114). The position and attitude of the additive information in the input image can be adjusted in accordance with the position and attitude of the object identified by the identification unit <NUM>, for example. Then, the display control unit <NUM> causes the generated output image to be displayed on the display unit <NUM> (step S116).

Such processes are repeated for each of a series of input images acquired by the image acquisition unit <NUM>.

<FIG> is a flowchart showing an exemplary flow of processes performed by the dictionary server <NUM> in accordance with this exemplary embodiment.

Referring to <FIG>, first, the receiving unit <NUM> of the dictionary server <NUM> waits for the reception of an input image from the terminal device <NUM> (step S202). Then, when an input image is received by the receiving unit <NUM>, the identification unit <NUM> extracts a feature quantity from the input image in accordance with a high-accuracy algorithm (step S204). Next, the identification unit <NUM> checks the extracted feature quantity of the input image against each feature quantity in the feature dictionary (Arg <NUM>) <NUM>, and identifies an object in the input image (S206). Herein, if a new object that is different from the objects identified in the previously received input images is identified, the process proceeds to step S210 (S208). Meanwhile, if a new object is not identified, the processes of the following steps S210 to S214 can be skipped. If a new object is identified by the identification unit <NUM>, a subset of a feature dictionary is acquired in accordance with the result of identification (or a new feature dictionary with a less data volume is generated) (step S210). Next, the additive information acquisition unit <NUM> acquires from the additive information DB <NUM> a subset of an additive information DB in accordance with the result of object identification performed by the identification unit <NUM> (step S212). Next, the transmitting unit <NUM> transmits the subset of the feature dictionary and the subset of the additive information DB to the terminal device <NUM> (steps S214).

The feature dictionary and the additive information DB, which are provided to the terminal device <NUM> from the dictionary server <NUM> through the aforementioned processes, are used for object identification in the terminal device <NUM>.

An exemplary embodiment and two variations of the technology disclosed in this specification have been described above with reference to <FIG>. According to the aforementioned exemplary embodiment, a feature dictionary used for identification of an object in an input image by the terminal device <NUM> is provided with the terminal device <NUM> from the dictionary server <NUM> that stores a feature dictionary with more abundant feature quantities in advance. The feature dictionary provided to the terminal device <NUM> is a dictionary that is acquired in the dictionary server <NUM> in accordance with the result of identification of an object in the input image. Thus, even if the terminal device <NUM> with a small amount of processing resources does not have a feature dictionary with a large volume in advance, the terminal device <NUM> can identify an object with higher accuracy using a feature dictionary that is suitable for the conditions in which the terminal device <NUM> is located.

In addition, according to the aforementioned exemplary embodiment, an object can be identified using a high-accuracy feature extraction algorithm in the dictionary server <NUM>, and the object can be identified using a "lightweight" feature extraction algorithm in the terminal device <NUM>. Thus, even in the terminal device <NUM> with a small amount of processing resources, an application that involves object identification, for which real-time properties are required, such as an AR application can be operated with high accuracy at fast speed.

Further, according to the aforementioned exemplary embodiment, a database of additive information that can be overlaid on an image with an AR application is stored in the dictionary server <NUM> in advance, and a subset thereof is provided to the terminal device <NUM>. Additive information provided to the terminal device <NUM> from the dictionary server <NUM> is also acquired in accordance with the result of identification of an object in the input image with the dictionary server <NUM>. Thus, resources used for storing and processing the additive information in the terminal device <NUM> can also be saved.

Furthermore, according to the aforementioned exemplary embodiment, a feature dictionary provided to the terminal device <NUM> from the dictionary server <NUM> includes not only the feature quantity for an object in the latest input image but also the feature quantity for an object that is predicted to appear in the following input image. Thus, in the terminal device <NUM>, a feature dictionary that is once provided from the dictionary server <NUM> can be continuously used for a certain period of time. Accordingly, once a feature dictionary is provided, there will be no need thereafter to wait for the reception of data for object identification in the terminal device <NUM>. Thus, the real-time properties of an application operating on the terminal device <NUM> can be improved. Further, as the terminal device <NUM> need not transmit an input image to the dictionary server <NUM> for each frame, the traffic load can also be reduced.

Moreover, according to the first variation, a new feature dictionary that is generated in the dictionary server <NUM> using a partial image of an input image is provided to the terminal device <NUM>. In this case, in comparison with a case in which a subset of a feature dictionary that is prepared in advance in a normal environment is provided, it becomes possible for the terminal device <NUM> to use a feature dictionary that is particularly adapted to the environment (e.g., imaging environment or luminance environment) in which the terminal device <NUM> is located. Therefore, the processing cost for checking feature quantities in the terminal device <NUM> and the consumed memory resources can also be reduced in the second variation.

Note that the aforementioned object identification technique can be used not only for an AR application or applications having other objectives, but also for the initialization or calibration of a coordinate system in estimating the position and attitude of the terminal device <NUM> with the SLAM (Simultaneous Localization and Mapping) technology. For the SLAM technology, see "<NPL>).

Claim 1:
A computer-implemented method comprising:
acquiring, by a terminal device (<NUM>), a first image including a first real object;
in response to a detected trigger event, transmitting, from the terminal device (<NUM>) to a server (<NUM>), first information associated with the first image, the first information including a feature quantity extracted from the first image;
identifying, by the server (<NUM>), the first real object by checking the feature quantity extracted from the first image against a feature quantity dictionary comprising feature quantities of a plurality of known objects;
obtaining, by the server (<NUM>), from the feature quantities included in the feature quantity dictionary and based on the result of identification, a data set comprising a feature quantity for a known object (B4) identified as the first real object and a feature quantity for a second real object (B5, B9) that is predicted to appear in a second image acquired subsequent to the first image;
wherein the second real object (B5, B9) is different from the first real object, and
wherein the second real object (B5, B9) predicted to appear is
an object determined on the basis of positional data to be located near the object (B4) identified as the first real object, or
an object associated in advance with the object (B4) identified as the first real object as having a high possibility of co-occurring with the object (B4) identified as the first real object;
acquiring, by the server (<NUM>), additive information corresponding to the second real object (B5, B9) that is predicted to appear;
receiving, by the terminal device (<NUM>), the data set and the additive information from the server (<NUM>);
recognizing, by the terminal device (<NUM>) and based on the received data set, an object (B5, B9) in an input image, and
controlling, by the terminal device (<NUM>), a display to display, overlayed on the input image, the additive information corresponding to the recognized object.