Patent ID: 12235444

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Throughout the drawings, the same components are provided with the same reference signs, and the repetitive explanation therefor are omitted.

First Embodiment

FIG.1illustrates a configuration of a linked display system for an information device and a head mount display (HMD) according to the present embodiment.

FIG.1illustrates a user1, an information device2, an HMD3, an access point4, communication signals2a,3a,4a, a network5, and a web server6.

The user1browses or views web content in one of two scenes of SCENE1and SCENE2. The transition between two scenes can be made at an arbitrary timing. In the following, an action in which the user1searches and receives the web content by using the information device2to watch it on a display screen of the information device2is referred to as “browsing”, while an action in which the user1watches the web content on a virtual screen of the HMD is referred to as “viewing”.

In SCENE1, the user1uses the information device2to browse the web content. The information device2is connected to the network5via the communication signals2a,4aand the access point4. The information device2is configured to search the web content on the web server6connected to the network5, receive the desired web content, and display it thereon, thereby allowing the user1to view the web content.

Preferably, the communication signals2a,4aare wireless communication signals for LANs, whereby browsing of the web content can be performed by utilizing TCP/IP and HTTP protocols that are widely used on the Internet.

The network5stands for an integration of the Internet and the Intranet.FIG.1does not illustrate a proxy server or the like provided between the Internet and the Intranet.

The user1searches and invokes the web content, for example, by inputting a keyword or a URL on the information device2, which restricts a motion of the user's hand. Thus, when the user1attempts to do a task requiring a motion of his or her hand while browsing the web content, transition from the SCENE1to the SCENE2is made. The transition from the SCENE1to the SCENE2is made, for example, in the case of actually cooking while searching and watching the web content that explains how to cook such as a recipe, and in the case of handling a household appliance while searching an instruction manual thereof on the Internet and watching it.

The user1makes connection for wireless communication between the information device2(corresponding to the “first information device”) and the HMD3(corresponding to the “second information device”), temporarily stops displaying the web content, and wears the HMD3. The SCENE2expresses the user1who has worn the HMD3.

The HMD3is, for example, an optical see-through type HMD or a video see-through type HMD. In the following, the case employing the optical see-through type HMD3will be described.

By using the HMD3, the user1can watch a virtual screen projected by a display optical system at the same time while seeing a real view through an optical screen. The virtual screen displays a part of the web content which was viewed by the user1on the information device2until immediately before the scene transition. In the present embodiment, the virtual screen displays an image containing only text data.

The HMD3is connected to the network5via the communication signals3a,4aand the access point4, and is configured to receive the web content from the web server6. The HMD3may be configured to receive the web content via the information device2.

The communication signals2a,3aare signals for connecting the information device2to the HMD3to establish wireless communication therebetween, which allows them to transmit and receive data with each other. Here, the communication signals2a,3amay be wireless LAN signals or Bluetooth (registered mark) signals. The Bluetooth (registered mark) signals implement one-to-one communication between the information device2and the HMD3, and thus enables pairing of the information device2and the HMD3with a series of protocols. The linked display system10is a system capable of display control by connecting the information device2to the HMD3to establish communication therebetween.

The information device2transmits, to the HMD3, data such as the URL of the web content which was watched by the user1until immediately before the scene transition and an elapsed time from the start of watching thereof. The HMD3receives the web content corresponding to the received data from the network5and causes a display optical system to display it so that the user1can watch the web content continuously.

(Information Device)

With reference toFIG.2andFIG.3, an example of the information device2will be described.FIG.2is a front view of an appearance of the information device2, andFIG.3is a hardware configuration diagram of the information device.

FIG.2illustrates a home button2b, an ambient light and proximity sensor2c, an earpiece2d, an in-camera2e, a notification lamp2f, and a display area2g(also referred to as the virtual screen) of a touch screen display (corresponding to the “first display”). In addition, although not illustrated inFIG.2, for example, a volume button and a power button are provided on the side of the information device2, a speaker, a microphone, a power and/or communication connector, and an earphone jack are provided on the bottom thereof, and an out-camera and a fingerprint sensor are provided on the back surface thereof. These components are not necessarily arranged in the same manner as the example illustrated inFIG.2.

FIG.3illustrates a camera201(including at least two cameras, namely, the in-camera and the out-camera), a group of sensors202including such as the ambient sensor, the proximity sensor, a GPS sensor, a gyro sensor, a geomagnetic sensor, and an acceleration sensor, a graphics processing unit203, a touch screen display204(the screen2ginFIG.2), a microphone and speaker205, a wireless communication unit (a wireless communication transmitter and receiver)206(corresponding to the “first communication unit”), a CPU207(corresponding to the “first processor”), a RAM208, a Flash ROM (FROM)209, an external interface (I/F)212, and an internal bus213.

The FROM209includes a basic operation program210of the information device2and a program of an HMD linking application211(installed in the information device), and the CPU207loads and executes these programs onto the RAM208. Furthermore, the FROM209can store data generated in the process and as a result of execution of the programs.

The wireless communication unit206includes a plurality of communication functions, for example, a mobile communication function such as 4G, a wireless LAN communication function, and a Bluetooth (registered mark) communication function.

The touch screen display204is configured to cause a display part to display an image generated by the graphics processing unit203, and receive an input operation by a user through a touch panel part. The user uses a keyboard screen displayed on the display part and enters such as the URL to browse the web content.

(HMD)

With reference toFIG.4andFIG.5, an example of the HMD3will be described.FIG.4illustrates an appearance of the HMD3, andFIG.5is a hardware configuration diagram of the HMD.

FIG.4illustrates a camera30, a ranging device (a ranging sensor)31, projection optical systems32a,32b, a display optical system33such as a lens or a screen, a nose pad34, a processor35, a speaker36, a microphone37, and frame housings38a,38b,38c.

The user1wears the HMD3on his or her own face by putting the frame housings38a,38band the nose pad34thereon.

The camera30is mounted on the HMD3to capture an image of the front of the user's line of sight. The ranging device31is configured to measure a distance to a real object in the real space which is included in the captured image (corresponding to the “front image”) of the camera30.

The ranging device31may be the one configured to calculate the distance with respect to feature points such as the outline of the real object in the same method as that of a stereo camera, or may be the one configured to measure the distance by irradiating a light beam two-dimensionally in a method like a TOF (Time of Flight) method, as long as a ranging device to be employed can measure a distance to the real object in the captured image. The ranging device31includes the ranging device31and a distance calculation unit351(seeFIG.5).

The projection optical systems32a,32band the display optical system33form a display section (corresponding to the “second display”) of the HMD3. The projection optical systems32a,32bare configured to project and display video of the virtual object on the display optical system33, as an image to be visually recognized by the left eye and an image to be visually recognized by the right eye. Since the user1can see a scene and the real object in his or her front through the display optical system33, the video of the virtual object projected from the projection optical systems32a,32bcan be displayed at any position on the display optical system33as long as the position allows the user1to easily distinguish the virtual object from the real object in the real space. Note that the ranging device31may not be provided. In this case, the virtual screen is displayed at a position preset in the real space.

The processor35(corresponding to the “second processor”) is configured to take in the image of the real space captured by the camera30and the position data of the real space of the real object acquired by the ranging device31, and supply an internal memory and the CPU with them. Furthermore, the processor35includes the group of sensors such as a gyro sensor, a direction sensor, a position sensor, a contact sensor, and a line-of-sight detection sensor.

The processor35is further configured to create an image to be projected by the projection optical systems32a,32band a sound to be output to the speaker36. The processor35, the camera30, the ranging device31, the speaker36, and the microphone37are arranged on the frame housings38a,38b,38c. These components are not necessarily arranged in the same manner as the example illustrated inFIG.4.

InFIG.5, the components which are the same as those inFIG.4are provided with the same reference signs.FIG.5further illustrates an internal bus350, the distance calculation unit (a distance calculator)351, a group of sensors352, a wireless communication unit (a wireless communication transmitter and receiver)354, a CPU355, a RAM356, an image RAM357, and a FROM358.

The projection optical system32corresponds to the projection optical systems32a,32binFIG.4. The projection optical system32is configured to cause the display optical system33to separately project an image for the left eye and an image for the right eye in the same manner as the projection optical systems32a,32b. Alternatively, the projection optical system32may be the one configured to project an image for the left eye and an image for the right eye which have been interleaved in one projector, and cause a shutter optical system to transmit the image for the left eye and the image for the right eye through the respective eyes. In addition, the projection optical system32may be an optical system using a holographic lens.

The wireless communication unit354(corresponding to the “second communication unit”) connects the HMD3to the network5via the access point4on the wireless LAN. Furthermore, the wireless communication unit354implements one-to-one communication with the information device2, for example, by Bluetooth (registered trademark) communication.

The FROM358includes, as processing programs, a basic operation program580and a program581of the HMD linking application. The CPU355loads and executes these processing programs onto the RAM356. Furthermore, the FROM358can store data generated in the process and as a result of execution of the programs.

The image data to be transmitted to the projection optical system32is stored in the image RAM357and read out from the image RAM357.

FIG.6is a block diagram illustrating program functions of the linked display system10. The CPU207of the information device2loads and executes the HMD linking application211onto the RAM208, whereby a display control section401, a content analysis section402, and a communication control section403are developed. The CPU355of the HMD3loads and executes the HMD linking application581onto the RAM356, whereby a display control section501, a content analysis section502, a communication control section503, and a line-of-sight detection section504are developed. The functions of the respective sections will be described later with reference to a flowchart ofFIG.9and that ofFIG.10.

EachFIG.7andFIG.8illustrates a display switch example by the linked display system10.

In a display state A (the all-content display state) ofFIG.7andFIG.8, when connection between the information device2and the HMD3is established and then the information device2transmits the text data to the HMD3, transition of the display state A1to a display state B1(seeFIG.7) or a display state B2(seeFIG.8) is made. When the information device2or the HMD3receives an instruction for terminating the linkage, transition from the display state B1or the display state B2to the display state A is made.

In the display state A, that is, in the display state where the content is displayed only by the information device, the content containing the text data and an image A is displayed on the information device2. At the right end of the screen2gof the touch screen display204, a scroll bar2g1for moving the display screen in the front and rear directions is provided.

In the display state B1, only the image A is displayed on the information device2while only the text data is displayed on the HMD3. The display size of the image A in the display state B1is larger than the display size of the image A in the display state A. In this connection, although it is often the case to refer an object displayed on the HMD3as an “AR”, a virtual object (corresponding to the text data inFIG.7) displayed on the HMD3is not necessarily associated with a real object or a position.

In the display state B1, the text data is gathered and displayed on the left side of the display area (virtual screen) of the HMD3so as to ensure the visibility of the center of the visual field of the wearer (user1) of the HMD3.

As illustrated in the display state B2, the text data may be gathered and displayed near the top of the virtual screen of the HMD3so as to ensure the visibility of the center of the visual field of the wearer of the HMD3. In particular, in the case where the wearer of the HMD3is working while looking down at his or her hand or a real object, such as for cooking, displaying a virtual object only on the upper part of the virtual screen of the HMD3can ensure the visibility during cooking. Depending on working states, the position on which the virtual object is to be displayed may be set to either the lower part or the left or right part. The display position may be arbitrarily determined by the wearer, or may be automatically set by the HMD3. In the case where the HMD3automatically sets the position, the out-camera detects a moving range of the wearer's hand so that the virtual object can be displayed on a position of the display area of the HMD3where the virtual object does not overlap the detected moving range of the wearer's hand.

In each of the display states B1and B2, in the case where the number of characters in the text data is so large that they cannot fit within the virtual screen, the display control section501displays a virtual object M having an arrow shape for scrolling the virtual screen together with the text data on the virtual screen of the HMD3. An arrow selection area M1is formed by a closed area including the virtual object M. The arrow selection area M1is an area in which the display control section501determines that the line of sight has selected the arrow.

The line-of-sight detection section504is configured to cause the in-camera to capture an image of the face of the wearer of the HMD3so as to detect a line-of-sight direction based on the captured face image. Then, upon detecting that the line of sight of the wearer has reached the arrow selection area M1, the display control section501makes a displayable area of the HMD3move in the direction of the arrow. The HMD3notifies the information device2that the screen has been scrolled. When an image associated with newly displayed text data as a result of scroll is an image B which is different from the image A, the information device2switches the image A to the image B.

EachFIG.9andFIG.10explains a flow of processing of the linked display system. In step S100, an application is started. The following processing is started from a state in which a main power of the information device2and that of the HMD3are turned on, the HMD linking application is started in each of the information device2and the HMD3, and pairing therebetween is completed.

FIG.9illustrates a flowchart of processing of analyzing the content on the side of the information device2.

In a state where the text data and the image A are displayed (display state A) on the information device2, a user operates the information device2to switch to a mode for browsing the content on the HMD3(step S101).

The communication control section403of the information device2confirms whether connection with the HMD has been established (step S102).

The content analysis section402of the information device2extracts only text from the displayed content to generate text data (step S103).

The content analysis section402outputs the generated text data to the communication control section403. The communication control section403transmits the text data via the wireless communication unit206to the HMD3(step S104).

The communication control section503of the HMD3receives the text data via the wireless communication unit354, and transmits a reception completion signal to the information device2(step S105).

The display control section501of the HMD3adjusts the received text data, for example, fonts, size, and colors thereof to those for HMD display (step S106), and causes the projection optical system32and the display optical system33to display the text data on the HMD3(step S107).

The display control section501causes the wireless communication unit354to transmit the text which has been displayed to the information device2(step S108).

The content analysis section402determines an image corresponding to the text which has been displayed on the HMD3, and the display control section401causes the touch screen display204of the information device2to enlarge and display the image (step S109).

In the case where the entire text is not displayed (step S110/NO), the display control section501continues to display the text on the HMD3(step S107).

In the case where the entire text is displayed (step S110/YES) but display of the text on the HMD3is requested to continue (step S111/NO), the display control section501continues to display the text on the HMD3(step S107).

In the case where display of the text on the HMD3is requested to be terminated (step S111/YES), the HMD3transmits a display termination signal to the information device2(step S112).

Upon receiving the display termination signal, the display control section401of the information device2restores the original display state of the information device2(step S113).

FIG.10illustrates a flowchart of the processing of analyzing the content on the side of the HMD3. The same steps as those inFIG.9are provided with the same reference signs, and a repetitive explanation therefor is omitted.

After confirming that connection with the HMD3is established (step S102), the information device2transmits the content data (text data and image data) to the HMD3(step S201).

The HMD3receives the content data, and causes the communication control section503to transmit the reception completion signal to the information device2(step S202).

The content analysis section502of the HMD3extracts the text part from the received content data to generate the text data (step S203), performs the processing for displaying the text data on the HMD3(step S106), and displays the text (step S107). Thereafter, in the same manner as the flow ofFIG.9, the information device2enlarges and displays the image.

In the steps described above, there may be various methods in which the content analysis section502of the HMD3and the content analysis section402of the information device2discriminate between text and images. For example, in the case of a website or e-book data in EPUB format, the content analysis section502and the content analysis section402may analyze the sources of HTML, XML, and XHTML to separate the text to be displayed from the image files, and extract the text being displayed (excluding a part of the text that is not being displayed on the information device2, such as comments) to generate the text data.

On the other hand, since the image has the directory information of storage location in its tag, the content analysis section502and the content analysis section402may be configured to directly acquire the corresponding address therefrom so that the image can be displayed on the information device2.

In the case of PDF data, the content analysis section502and the content analysis section402may extract a portion that can be written out by such as the OCR as the text data.

In addition, as a method of displaying only an image on the information device2, for example, in the case where a user is using the information device2to browse a site, since the URL (storage location of the image file on the server) has been already read at the time of displaying the image, it may be only needed to display its address.

On the other hand, in the case where the content data does not include any image, the displayed content may be displayed as it is on the information device2. This case is an exception of the above-described processing using the content including text and an image.

According to the first embodiment, at the time of displaying the content including text and an image by linking the information device2with the HMD3, it is possible to display only the text data on the HMD3while enlarging and displaying the image on the information device2. With this configuration, since the image is not displayed on the HMD3, in the state where the user wears the HMD3and visually recognizes the text information displayed thereon, the visibility of the text information displayed on the HMD3can be improved. Furthermore, since the image is enlarged and displayed on the information device2, the user who wants to watch the image is only required to look at the information device2to visually recognize the enlarged image as a real image, whereby the visibility of the image can be improved.

Second Embodiment

The difference between the second embodiment and the first embodiment can be found in content. In particular, the linked display system according to the second embodiment is configured to link and display content in which a part of text and an image are associated with other for each operation process (procedure), which is different from the linked display system according to the first embodiment.

FIG.11illustrates an example of content used in the second embodiment.FIG.12illustrates a display example on the HMD according to the second embodiment.FIG.13illustrates an example of content used in the first embodiment.FIG.14illustrates a display example on the HMD3according to the second embodiment.

The content1ofFIG.11is content in which an image such as a figure or a picture is associated with a part of text in one-to-one relation. The content1includes content involving activities, for example, cooking recipes, guides for sewing and gardening, training manuals of stretching and workout, and methods for dressing kimonos and tying obi belts for kimonos. The image associated with the part of the text may contain a still image or video. The image and the part of the text in the content are placed such that, in the case of the cooking recipes, an Image1showing a preliminary cooking procedure is displayed on the left side of the screen while the part of the text corresponding to the Image1describing the procedure is displayed on the right side of the Image1. An Image2showing the next procedure is arranged on the lower side of the Image1while another part of the text corresponding to the Image2is arranged on the lower side of the text corresponding to the Image1.

As a criterion in which the content analysis section502of the HMD3and the content analysis section402of the information device2recognize that the parts of the content are displayed along the processes, at least one of the following criteria or any combination thereof may be used, for example.(Criteria 1) Each part of text and each image are placed on a frame or a table per paragraph.(Criteria 2) A number or alphabet indicating the order is provided at the top of each text.(Criteria 3) A number or alphabet indicating the order is provided on a caption of each image (in the case of a web site, an <ALT> tag may be used).(Criteria 4) Each title sentence contains a word such as “procedure”, “recipe”, “how to”, or “method”, which means the activity procedure.

Upon determining that the content is divided for each process based on one of the criteria such as a frame, table, and paragraph, or any combination thereof, the content analysis section502of the HMD3and the content analysis section402of the information device2switch from the display state A to the display state B1or the display state B2. In this case, as illustrated inFIG.12, the content analysis section502of the HMD3extracts and displays only the text portion from the part of the content included in one process. The content analysis section502of the HMD3further displays the arrow selection area M1on a position near the AR of the text. Upon detecting that the user's line of sight has stayed in the arrow selection area M1for a predetermined time, the line-of-sight detection unit504may output a detection result to the content analysis section502so that the content analysis section502causes the display control section501to display the content of the next procedure (step). For example, the screen of the HMD3illustrated inFIG.12is updated and displays the text “step2. Boil the potatoes.” ofFIG.11.

On the other hand, the content2ofFIG.13is an example of content which is determined to be not divided for each process by the content analysis section502of the HMD3and the content analysis section402of the information device2.

The content2is content in which an image such as a figure or a picture is not associated with a part of text in one-to-one relation, that is, content only to be “read”, for example, news, columns, and blogs.

In the content2, below a title sentence of an article or column, an image such as a figure or a picture (hereinafter, referred to as <Fig.>) and a part of the text are not placed in a one-to-one manner. This type of content, in which the <Fig.> is inserted in the middle of the text, is mainly provided to be “read” by a user. In this case, there is no regularity at the place where the <FIG.> is to be inserted while the author appropriately inserts the <FIG.> depending on the type of the content. The number of <FIG.> to be inserted may be one or more.

When not recognizing that the content shows a procedure (content divided for each process), the content analysis section502of the HMD3and the content analysis section402of the information device2determine that the content is provided to be read such as a news article, a column, or a blog.

In the case of a web site (news, blog, etc.), an image inserted in the content is displayed in accordance with a description position of an image insertion tag, or, displayed by reading a link destination such as a URL where the image is stored.

In the case of displaying a Word file or a PDF file, in the same manner as described above, when processes can be recognized, an image and a part of text may be displayed for each process. When the processes cannot be recognized, the image and the part of the text may be displayed separately in the same manner as the first embodiment so that the image is updated and displayed in accordance with scroll of the screen on which the text is being displayed. On the other hand, in the second embodiment, instead of processes, an image is used as an indicator for determination of a break in the content. For example, the content analysis section502determines a portion from the start to “The Maxell Group . . . (hereafter, referred to as “We”)” as one block (being composed of partial data of content), and displays it on the HMD3. At this time, in the case where one block does not fit within the display area of the HMD3, as illustrated inFIG.14, the content analysis section502displays the arrow selection area M1on a position near the AR of the text. Upon detecting that the user's line of sight has stayed in the arrow selection area M1for a predetermined time, the line-of-sight detection unit504may output a detection result to the content analysis section502so that the content analysis section502causes the display control section501to display the content in the next block.

Linking and displaying content that is not related to processes as illustrated inFIG.13benefits a user who wants to watch the content that can be easily read such as news and columns while doing daily work such as brushing teeth and folding laundry or doing tasks without requiring thoughts. In such a situation, the user cannot hold the information device2since he or she has to move his or her hand for the work, or when doing the work while putting on the information device2nearby and watching the screen, he or she often needs to stop the work. On the other hand, displaying a text portion (main part of the content) on the HMD3allows the user to read the content without stopping the work.

The second embodiment for displaying content for each process also benefits, for example in the case of cooking, the user who wants to watch a recipe during cooking. Since the text portion of a recipe is displayed on the HMD3, the user wearing the HMD3can check the recipe in any place while keeping his or her hand clean even in a situation where there is no space to put a recipe book under an opened state, the user feels trouble about frequently going to check the recipe book, the user feels that tapping and scrolling a display by a finger is insanitary at the time of watching the recipe on a smartphone or a tablet, or the user cannot watch the recipe on the information device2as a screen is locked (display is hidden) after a certain time has elapsed even when he or she wants to check the recipe immediately.

In addition, the user can check a finished state by displaying (enlarging) an image with full screen on the information device2.

Furthermore, the present invention solves a problem that, although the user wants to read the descriptive text while enlarging and displaying the image, it is difficult to check them at the same time on the information device2alone.

Still further, as a method of updating and displaying the next procedure (process) or content, or content of the next block, the arrow selection area M1is displayed to make the user's line of sight stay therein for a predetermined time, whereby making it possible to update and display the content even in a situation where both hands are occupied, for example, during doing tasks.

Each of the above-described embodiments does not limit the present invention, and a part of the configuration of one of the embodiments can be replaced with the other one of the embodiments. In addition, a configuration of one of the embodiments can be added to the other one of the embodiments. These modifications are included in the scope of the present invention. Furthermore, numerical values and messages appearing in the text and drawings are merely examples, and thus the advantageous effects of the present invention are not impaired even if different ones are used.

A part or all of the functions of the present invention may be implemented in hardware, for example, by designing them in an integrated circuit, or may be implemented in software by causing such as the microprocessor unit or the CPU to interpret and execute the operation program. In addition, the hardware and the software may be used in combination without limiting the scope of software implementation.

For example, the image (still image or video) may be displayed on the information device2on which the user watched it firstly, or displayed on other display devices, which are different from the information device2, such as a television and a monitor, by transmitting the content thereto.

Still further, the present invention may be configured to capture a front view including the visual field of the wearer by using the camera of the HMD3to generate front image data, analyze a state of a subject in the front image data by using the content analysis section502, notify the display control section501when determining that the state of the subject is included in the next process, and update the text data being displayed on the virtual screen to display the text data associated with the next process by using the display control section501.

REFERENCE SIGNS LIST

1: user2a: communication signal2g: screen2g1: scroll bar3: HMD3a,4a: communication signal4: access point5: network6: web server10: linked display system30: camera31: ranging device32,32a,32b: projection optical system33: display optical system34: nose pad35: processor36: speaker37: microphone38a,38b,38c: frame housing203: graphics processing unit204: touch screen display206,354: wireless communication unit207,355: CPU208,356: RAM210,580: basic operation program211,358: HMD linking application351: distance calculation unit357: image RAM401,501: display control section402,502: content analysis section403,503: communication control section504: line-of-sight detection section581: programM: virtual objectM1: arrow selection area