Information display terminal, information display system, and information display method

In a wearable information display terminal, information related to an object is displayed at a timing required by a user in an easily recognized form. Photography is performed in a field-of-vision direction, a first object is detected from a first image obtained by the photography, relevant information related to the first object is acquired from a network, and a second image related to the relevant information is generated. A second object different from the first object is detected from the first image, the second object is used as a display trigger, and the second image is displayed when the second object is close to the first object.

TECHNICAL FIELD

The present invention relates to an information display terminal, an information display system, and an information display method.

BACKGROUND ART

Various kinds of methods have been proposed as an information display method related to a so-called wearable information display terminal that displays predetermined information using a goggle-type or glasses-type head mounted display.

For example, Patent Document 1 discloses a method of switching an information display mode between a summary mode and a detailed mode according to a situation of a user. Further, for example, Patent Document 2 discloses a method of combining information input by the user through an input unit with information photographed by an imaging unit and displaying combined information according to a situation of a real world specified by a current position or object information.

CITATION LIST

Patent Document

Patent Document 1: JP 2007-163634 A

Patent Document 2: JP 2011-28763 A

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

A difference between a glasses-type wearable information display terminal and a mobile terminal such as a smartphone lies in that, when predetermined information is acquired, in the mobile terminal, the user should actively takes a look at display content provided by the mobile terminal, whereas in the wearable information display terminal, the user can confirm information display passively within a field of vision when it is in an operation state. However, in the wearable information display terminal, since the information display starts and ends regardless of the will of the user, particularly, the information display is performed even when unnecessary, and thus the user may feel annoyed according to circumstances.

In the technique of the related art disclosed in Patent Document 1, the information display mode is switched according to a situation such as whether the user is walking, viewing a display video, or speaking but how to determine a timing at which the user desires information related to an arbitrary object to be displayed and how to start or end the display of the information related to the object were not particularly taken into account.

In the technique of the related art disclosed in Patent Document 2, the information related to the object input by the user is combined with the information photographed by the imaging unit, the combined information is displayed, but similarly, how to start or end the display of the information related to the object at an effective timing was not particularly taken into account.

The present invention was made in light of the foregoing, and it is an object of the present invention to provide an information display terminal and an information display method, which are capable of starting or ending display of information related to an object at an appropriate timing and displaying the information in a form easily recognized by the user in a wearable information display terminal or the like.

Solutions to Problems

The object is achieved, for example, by the inventions stated in claims.

Effects of the Invention

According to the present invention, there is an effect that display of information related to an object can be started or ended at an appropriate timing, display information can be displayed in a form easily recognized by the user, and convenience of the user in, for example, the wearable information display terminal can be improved.

MODE FOR CARRYING OUT THE INVENTION

First Embodiment

FIG. 1is a block diagram illustrating an example of a configuration of an information display terminal101. The first embodiment will be described using this.

101denotes an information display terminal,102denotes a control unit that controls the information display terminal101in general,103denotes an imaging unit,104denotes an image processing unit,105denotes a display control unit,106denotes an audio input/output unit corresponding to a microphone or an earphone,107denotes an audio processing unit,108denotes a communication processing unit,109denotes a communication input/output unit,110denotes a position detecting unit including a global positioning system (GPS),111denotes a sensor input/output unit, and112denotes an various kinds of sensors.

The communication input/output unit109is connected with a network through wireless communication. In the present embodiment, an example in which the information display terminal101is connected directly to the Internet or the like and acquires information will be described. However, the information display terminal101may perform communication with an information terminal (for example, a smartphone) serving as a separate device through near filed communication (NFC) such as Bluetooth (a registered trademark), and the information terminal may perform a connection with the Internet or the like and a main process.

The information display terminal101is a head mounted display equipped with at least the imaging unit103and the display control unit105. As various kinds of sensors112, a plurality of sensors such as an inclination sensor that detects a posture of the user, an acceleration sensor that detects a motion of the user, a visual line sensor or a temperature sensor that detects a body state of the user, and a pressure sensor or a capacitance sensor used as an input/output interface (I/F) that detects an instruction input of the user can be mounted.

FIG. 2is a sketch diagram illustrating an example in which the user wears the information display terminal101. Components that are common to those ofFIG. 1are denoted by the same reference numerals.

201denotes the user,202schematically indicates an area of a first image captured by the imaging unit103, and203schematically indicates an area of a second image displayed through the display control unit105. The area of the second image is not limited to a size or a range indicated by203and may have a size or a range capable of covering a field of vision widely. The information display terminal101can have any form as long as it is worn on the head, and the display control unit105that projects, for example, light according to information within the field of vision of the user201and displays information and the imaging unit103capable of imaging the field-of-vision direction of the user201are arranged. For example, the information display terminal101may be a glasses-type information display terminal, a goggle-type information display terminal, an information display terminal worn on one eye, or an information display terminal worn on both eyes and is not limited to the exemplary form illustrated inFIG. 2.

The image processing unit104analyzes the first image of the area202photographed by the imaging unit103and detects a predetermined object included in the first image. Information indicating a feature of the detected object, for example, information indicating a feature related to a pattern which will be described later is transmitted to the network through the communication processing unit108, and the information related to the object accordingly acquired from the network is supplied to the image processing unit104. The image processing unit104generates the second image based on the information, and supplies the second image to the display control unit105. The display control unit105causes the second image to be displayed on the area203.

For example, the display control unit105causes an image to be displayed on a half mirror installed at a position corresponding to the area203or displays the second image by projecting the second image directly onto a retina of the user201, but when the information display terminal101is of a transmissive type as at least a head mounted display, and the second image is not displayed, the area203is set not to block the field of vision of the user201.

FIG. 3is an operation flowchart according to the first embodiment.

FIGS. 4A and 4Bare diagrams illustrating an example of relevant information display according to the first embodiment, and illustrate the field of vision of the user.

An operation of the information display terminal101according to the present embodiment will be described in detail with reference toFIG. 3andFIGS. 4A and 4B. The present embodiment will be described in connection with an example in which, particularly, when the user shops around for products in stores or the like, the information display terminal101displays relevant information related to a predetermined object as the second image.

First, in step301, a process starts. The whole process may be initiated by a touch input of the pressure sensor in a sensor112or a voice input of the user using the audio input/output unit106. The process may start automatically when the user is recognized to stay in a store based on position information detected by the position detecting unit110or an image photographed by the imaging unit103.

Then, in step302, a predetermined object is detected. Specifically, the image processing unit104detects a predetermined object by performing image recognition on images consecutively photographed by the imaging unit103and performing matching between a plurality of clipped objects and a shape or a color of a predetermined pattern or a transform prediction value thereof. The predetermined pattern may be a default pattern (for example, a genre such as a “suit”) according to store information or may be acquired from the Internet or the like in association with the store information.

When a desired production is decided, the user may designate a product name, a model number, a genre, and the like in advance, a predetermined pattern may be acquired from a server on the Internet or other public information, and predetermined object detection may be more specifically performed. The number of predetermined objects detected in step302is not limited to one, and a plurality of objects may be simultaneously detected in an imaging area.

When the predetermined object is detected, in step303, relevant information about the predetermined object is acquired. Specifically, the communication processing unit108acquires information related to the predetermined object from the Internet or the like through the communication input/output unit109. When the product name, the model number, and the like can be decided on based on a product tag or the like from an image of the predetermined object photographed by the imaging unit103, the relevant information can be also acquired based on the product name, the model number, and the like.

Then, in step304, determination of a display trigger is performed. The display trigger is a specific object for initiating the display of the second image based on the relevant information, and in this case, a “hand” is used as the display trigger. When the predetermined object is detected in step302, although the relevant information of the predetermined object is acquired in step303, the display of the second image is not performed unless the display trigger is photographed through the imaging unit103and recognized through the image processing unit104. A method of recognizing a “hand” through the image processing unit104is the same method as the object recognition in step302.

For example, as illustrated inFIG. 4A, although402to405in an imaging area401are detected as a predetermined object, since there is not display trigger in the imaging area401, the display of the second image is not performed (No in step304).

On the other hand, as illustrated inFIG. 4B, when a specific object407serving as the display trigger is recognized in the imaging area401and determined to approach a predetermined object (Yes in step304), in step305, the second image based on the relevant information of the specified predetermined object402is generated through the image processing unit104. Further, the second image is displayed through the display control unit105in a form that can be recognized as information related to the predetermined object402.

In this case, content displayed as the second image406is product information related to the predetermined object402. Here, for example, reviews information or market price information related to the product are displayed in the form recognized by the user. The displayed content is preferably information useful for the user and not particularly limited. A broken line of the imaging area401is illustrated for the sake of convenience and not displayed within the field of vision of the user.

In step306, the determination of the display trigger is continuously performed, and the relevant information is displayed while the display trigger is being recognized. Further, when the position of the display trigger is changed, and a predetermined object is changed, replacing to a second image related to a changed predetermined object is performed (Yes in step306). When the specific object40serving as the display trigger is hardly recognized in the imaging area401, in step307, the display of the second image ends, and the process returns to the detection of the predetermined object in step302(No in step306).

FIG. 5is a block diagram illustrating a system configuration on a network. The acquisition of the relevant information through the communication processing unit108will be briefly described with reference toFIG. 5.

The information display terminal101acquires the relevant information of the predetermined object402from the Internet501through the above-described configuration. For example, unspecified third parties transmit the information via the Internet501using a personal computer502, a mobile terminal503, an information display terminal504, and the like, and the information is accumulated in a storage506by a server505. The information display terminal101transmits information identifying the predetermined object402detected from the image photographed by the imaging unit103, and the server505supplies information related to the predetermined object402among the information accumulated in the storage506to the information display terminal101.

The server505collects information about predetermined stores, predetermined regions, and the like in advance, and supplies the information to the information display terminal. However, when a specific server505has no collected relevant information, the information display terminal101may searches for and acquire information related to the predetermined object402from information disclosed by a plurality of servers connected to the Internet.

As described above, according to the first embodiment of the present invention, the wearable information display terminal starts or ends the display of the second image for the relevant information of the predetermined object, for example, using the display trigger of causing the hand to approach the object. Thus, the display of the second image can be performed at a timing and a period of time in which a will or action of the user (for example, an action of picking up a product in a hand=a will desiring to see information related to a product) is reflected. The user can use both hands freely in the state in which the information display terminal is worn and need not operate the information display terminal101to select an object, and thus convenience is improved. Since a situation in which the second image is displayed at an unnecessary timing or a situation in which information about a plurality of objects is displayed simultaneously does not happen, there is an effect that a display annoying the user can be prevented.

Second Embodiment

Next, a second embodiment of the information display terminal101will be described with reference toFIGS. 6 to 8B. A configuration of the information display terminal101is basically the same as that illustrated inFIG. 1, and thus the description will proceed using the reference numerals allocated to the respective components ofFIG. 1. Similarly to the first embodiment, the description will proceed with an example in which, when the user shops around a product in a store or the like, the information display terminal101displays the second image as relevant information about a predetermined object.

FIG. 6is an operation flowchart according to the second embodiment.

FIG. 7is a diagram illustrating a display example at the time of display trigger registration.

One of features of the present embodiment lies in that the user can register the display trigger.FIG. 6illustrates an operation flow at the time of display trigger registration, andFIG. 7illustrates the field of vision of the user at the time of display trigger registration.

First, in step601, the process starts. The process may be initiated according to a user instruction given by a touch input of the pressure sensor or a voice input. Then, in step602, a frame in which the display trigger is registered is displayed. Specifically, the image processing unit104generates an image indicating a registration frame, and the display control unit105causes a registration frame702to be displayed within a range of an imaging area701of the imaging unit103, that is, in the field-of-vision direction of the user as illustrated inFIG. 7.

In step603, the user positions a specific object that is desired to be registered as the display trigger to fall within the registration frame702. In this case, a “palm” is registered as the specified object703. Then, in step604, when the image processing unit104recognizes that there is the specified object703within the range of the registration frame702from the image photographed through the imaging unit103, an inquiry indicating whether or not the object is registered as the display trigger is given to the user. The inquiry given to the user may be a text display as illustrated inFIG. 7or may be a voice output from the audio input/output unit106.

When a registration instruction is given by an instruction input of the user (Yes in step604), the specified object703is stored in a memory (not illustrated) in the information display terminal101as the display trigger. The memory is installed to be accessible from the control unit102or the image processing unit104but may be installed in the control unit102or the image processing unit104. Thereafter, in step606, the process ends. Further, in step604, in a state in which the presence of the specified object703is recognized by the image processing unit104, an image indicating a contour of the specified object703may be generated and displayed by the display control unit105, and the user may be notified of the fact that the specified object703has been recognized through a sound output from the audio input/output unit106.

FIGS. 8A and 8Bare diagrams illustrating an example of the relevant information display according to the second embodiment. An example in which, when the user shops around a product in a store or the like, the information display terminal101displays the second image as relevant information about a predetermined object, similarly to the first embodiment will be described with reference toFIGS. 8A and 8B. Here, a “palm” is assumed to have been registered in step605as the specified object703.

The operation flow is basically the same as that ofFIG. 3, and thus a detailed description thereof is omitted. In the determination of the display trigger of step304, as illustrated inFIG. 8A, there is a predetermined object802in an imaging area801, and the relevant information is acquired from the Internet or the like, but since a specific object803is a “back of hand” and not recognized as the registered display trigger, the display of the relevant information is not performed. The image processing unit104performs the determination as to whether or not the specific object is identical to the registered display trigger using matching of the shape or the color of the object, the transform prediction value thereof, or the like. InFIG. 8B, a specific object804is a “palm” and determined to be the display trigger registered in step605ofFIG. 6.

In step305, the image processing unit104generates the relevant information of the predetermined object802specified by the display trigger as a second image805. The display control unit105causes the second image805to be displayed in the form that can be recognized as information related to the predetermined object802.

Similarly to the above description, content displayed as the second image805is preferably product information that relates to the predetermined object802and useful for the user and not particularly limited. Broken lines of the imaging area701and801are illustrated for the sake of convenience and not displayed within the field of vision of the user.

As described above, according to the second embodiment of the present invention, a specific object is registered as the display trigger in the wearable information display terminal, and thus it is possible to start or end the display of the second image according to the user's intention. Similarly to the first embodiment, the user can use both hands freely in the state in which the information display terminal is worn and need not operate the information display terminal101to select an object. The display trigger can be appropriately changed according to the user's desire, and thus convenience is improved.

As a modified example of the present embodiment, an embodiment in which the process is initiated by the registered display trigger in step301ofFIG. 3is considered. In other words, as an operation of the information display terminal101, first, in step301, only the determination of the display trigger is continuously performed. Then, when the object recognized by the image processing unit104from the image photographed through the imaging unit103is determined to be identical to the registered display trigger, the operations of step302and subsequent steps are performed. In the example of the present embodiment, a series of operations can be performed by showing a “palm” in the imaging area401. Since the start of the operation is simple, there is an effect that the convenience of the user can be further improved.

Third Embodiment

Next, a third embodiment of the information display terminal101will be described with reference toFIG. 9andFIGS. 10A and 10B. A configuration of the information display terminal101is basically the same as that illustrated inFIG. 1, and thus the description will proceed using the reference numerals allocated to the respective components ofFIG. 1. In the present embodiment, similarly to the first embodiment, the description will proceed with an example in which, when the user shops around a product in a store or the like, the information display terminal101displays the second image as relevant information about a predetermined object.

FIG. 9is an operation flowchart according to the third embodiment.

FIGS. 10A and 10Bare diagrams illustrating an example of the relevant information display according to the third embodiment, and illustrate the field of vision of the user.

Operations of step901to step903are the same as the operations of step301to step303inFIG. 3, and a detailed description thereof is omitted. When the process starts in step901, in step902, a predetermined object is detected, and in step903, the relevant information related to the predetermined object is acquired from the Internet or the like. The number of predetermined objects detected in step902is not limited to one, and a plurality of objects may be simultaneously detected in an imaging area.

Then, in step904, a visual line of the user is determined. A gazing point of the visual line of the user wearing the information display terminal101is detected through the visual line sensor included in the sensor112. For the visual line sensor, for example, a method using infrared rays disclosed in Patent Document 1, a method using a complementary metal-oxide semiconductor (CMOS) sensor, and the like are known. When the gazing point falls within a range to which the detected predetermined object pertains (No in step904), the display of the second image related to the relevant information is not performed. For example, as illustrated inFIG. 10A, although1002to1005in an imaging area1001are detected as the predetermined object, when a gazing point1006falls at none of the predetermined objects, the display of the second image related to the relevant information is not performed.

On the other hand, as illustrated inFIG. 10B, when the gazing point1006falls at a predetermined object1002, and this state is continued during a predetermined period of time (for example, 3 seconds) or more (Yes in step904), in step905, the image processing unit104generates a second image1007as the relevant information of the predetermined object1002, and the display control unit105causes the second image1007to be displayed in the form that can be recognized as information related to the predetermined object1002. Similarly to the above embodiments, content displayed as the second image1007is the product information related to the predetermined object1002, and for example, the reviews information or the market price information related to the product is displayed in the form easily recognized by the user.

The displayed content is preferably information useful for the user and not particularly limited. A broken line of the imaging area1001and the gazing point1006are illustrated for the sake of convenience and not displayed within the field of vision of the user. However, a mode in which an image is displayed as the mark of the gazing point1006may be provided so that the mark of the gazing point1006can be recognized by the user, and in this case, the shape of the mark is not limited to the shape of the gazing point1006.

In step906, the determination of the visual line is continuously performed, and the second image1007is displayed while the gazing point1006falls within the range to which the predetermined object1002or the second image1007pertains (Yes in step906). It is because the gazing point1006moves to the second image1007side at a point in time at which the second image1007is displayed. Further, when the gazing point1006deviates from the range to which the predetermined object1002or the second image1007pertains or when the state in which the gazing point1006deviates is continued during a predetermined period of time (for example, 3 seconds) or more, the display of the second image1007ends (No in step906), and the process returns to the detection of the predetermined object in step902.

As described above, according to the third embodiment, in the wearable information display terminal101, the display of the second image starts or ends according to the gazing point of the user on the relevant information of the predetermined object, and thus the display of the image can be performed at a timing and a period of time in which a will or action of the user (for example, an action of drawing attention to a product=a will desiring to see information related to a product) is reflected. Similarly to the above embodiments, the user can use both hands freely in the state in which the information display terminal101is worn and need not operate the information display terminal101to select an object, and thus convenience is improved. Since a situation in which the second image is displayed at an unnecessary timing or a situation in which information about a plurality of objects is displayed simultaneously does not happen, there is an effect that a display annoying the user can be prevented.

Fourth Embodiment

Next, a fourth embodiment of the information display terminal101will be described with reference toFIG. 11andFIGS. 12A and 12B. A configuration of the information display terminal101is basically the same as that illustrated inFIG. 1, and thus the description will proceed using the reference numerals allocated to the respective components ofFIG. 1. In the present embodiment, similarly to the first embodiment, the description will proceed with an example in which, when the user shops around a product in a store or the like, the information display terminal101displays the second image as relevant information about a predetermined object.

FIG. 11is an operation flowchart according to the fourth embodiment.

FIGS. 12A and 12Bare diagrams illustrating an example of the relevant information display according to the fourth embodiment, and illustrate the field of vision of the user.

Operations of step1101to step1103are the same as the operations of step301to step303inFIG. 3, and a detailed description thereof is omitted. When the process starts in step1101, in step1102, a predetermined object is detected, and in step903, the relevant information related to the predetermined object is acquired from the Internet or the like. In the present embodiment, the description will proceed with an example in which a plurality of predetermined objects are simultaneously detected in an imaging area in step1102.

When the plurality of predetermined objects are detected, and the relevant information is acquired, in step1104, a candidate mark is displayed. The candidate mark refers to a mark displayed when effective relevant information is acquired for the detected predetermined object. For example,FIG. 12Aillustrates an example in which1202to1205in an imaging area1201are detected as a predetermined object in step1102, and effective relevant information is acquired for1202,1204, and1205in step1103.1206to1208are the candidate marks, the image processing unit104generates images as the candidate marks, and the display control unit105displays the candidate marks to overlap the corresponding predetermined objects. The shape of the candidate mark is not limited to the shape illustrated inFIG. 12Aas long as the candidate mark has the size that can be recognized by the user, and an associated predetermined object can be determined.

Then, in step1105, determination of a display trigger is performed. In this case, similarly to the above embodiments, a “hand” is registered as the display trigger. InFIG. 12A, since there is no display trigger in the imaging area1201, only the candidate marks1206,1207, and1208are displayed, and the display of the second image related to the relevant information of each predetermined object is not performed.

On the other hand, as illustrated inFIG. 12B, when a specific object1209serving as the display trigger is recognized in the imaging area1201and detected to approach the predetermined object1202, in step1106, the image processing unit104generates a second image1210as the relevant information of the predetermined object1202specified by the display trigger, and the second image1210is displayed through the display control unit105instead of the candidate mark1206.

In this case, the predetermined object1202are close to the other objects1203to1205. For this reason, the second image1210is displayed to overlap a part of the predetermined object1202. Thus, an object associated with the second image1210is explicitly displayed. In the other embodiments, in order to explicitly display an object associated with the second image, the second image may be displayed to overlap a part of an associated object.

Similarly to the above embodiments, content displayed as the second image1210is the product information related to the predetermined object1202, and for example, the reviews information or the market price information related to the product is displayed in the form easily recognized by the user. The displayed content is preferably information useful for the user and not particularly limited. The candidate marks1207and1208indicate the state in which the associated objects are not selected, and thus the candidate marks are continuously displayed without change. A broken line of the imaging area1201is illustrated for the sake of convenience and not displayed within the field of vision of the user.

In step1107, the determination of the display trigger is continuously performed, and the relevant information is displayed while the display trigger is being recognized (Yes in step1107). Further, when the position of the display trigger is changed, and a predetermined object close to the display trigger is changed, the candidate mark for the changed predetermined object is replaced to the display of the second image related to the relevant information. Further, the predetermined object before the change is replaced from the second image related to the relevant information to the candidate mark display. When the specific object1209serving as the display trigger is hardly recognized in the imaging area1201(No in step1107), in step1108, the second image display related to the relevant information ends, and the process returns to the detection of the predetermined object in step1102. However, the candidate mark displayed in the imaging area1201is continuously displayed.

As described above, according to the fourth embodiment, in the wearable information display terminal101, the candidate mark is displayed before the relevant information display of the predetermined object, and thus the user can recognize an object whose relevant information can be displayed in advance. For the relevant information, the display of the second image starts or ends using the display trigger, and thus the display of the second image can be performed at a timing and a period of time in which the will or action of the user is reflected. The user can use both hands freely in the state in which the information display terminal is worn and cause the display of the second image to be performed while recognizing an object whose relevant information can be displayed. Further, the user need not operate the information display terminal101to select an object, and thus convenience is improved.

As a modified example of the present embodiment, an embodiment in which the size or the color of the candidate marks1206to1208inFIG. 12Ais changed according to validity of content of the relevant information is considered. Since each relevant information is acquired at a point in time at which the candidate mark is displayed, for example, when an amount of additional information is large or when a product is designated by the user in advance, a display mark is enlarged or framed in a reddish color. As a result, the user can easily recognize it in advance.

In the present embodiment, the example in which the operation of displaying the candidate mark is performed in combination with the first embodiment has been described, but the operation of displaying the candidate mark may be performed in combination with the other embodiments.

All the above embodiments have been described in connection with the example in which, when the user shops around a product in a store, particularly, a clothing store, the information display terminal displays the relevant information about the predetermined object as the second image. However, the application scene is not limited thereto, and the present invention can be applied to information display in different categories of stores or outdoors. The example in which the “hand” is used as the display trigger has been described, but any other things may be designated as the display trigger.

The present invention is not limited to the above embodiments, and various modified examples are included. For example, the above embodiments have been described to facilitate understanding of the present invention, and the present invention is not limited to a configuration necessarily including all components described above. All or some of the above-described functions or the processing units may be implemented by hardware or may be implemented by software by interpreting and executing a program for implementing the functions through a processor. A control line or an information line is considered to be necessary for description and illustrated, and all control lines or information lines are not necessarily illustrated.

REFERENCE SIGNS LIST