Systems including mobile devices and head-mountable displays that selectively display content, such mobile devices, and computer-readable storage media for controlling such mobile devices

A system includes a head-mountable display and a mobile device. The mobile device includes a position acquisition device that acquires positional information indicating a position of the mobile device. The mobile device includes a first orientation acquisition device that acquires first orientation information indicating an orientation of the mobile device with respect to a reference direction. The mobile device includes a first display that displays content based on the positional information and the first orientation information on the mobile device when a condition is satisfied. The mobile device includes a first receiver that receives second orientation information indicating an orientation of the head-mountable display with respect to one or more of the reference direction and the orientation of the mobile device. The mobile device includes a first transmitter that transmits content based on the positional information and the second orientation information when the condition is not satisfied.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present disclosure generally relates to displaying content on mobile devices and head-mountable displays, and more specifically to systems including mobile devices and head-mountable displays that selectively display content, such mobile devices, and non-transitory, computer-readable storage media storing instructions for controlling such mobile devices.

2. Description of Related Art

Known mobile devices, such as mobile phones, include compact liquid crystal displays (“LCDs”), which may serve as display devices. The mobile devices are functionally enhanced with one or more of a Global Positioning System (“GPS”) function and a communication function for communicating via the Internet. The mobile devices acquire positional information by using the GPS function. In addition, the mobile devices acquire display content by using the communication function. The mobile devices are configured to display content related to their current positions on the LCDs, based on the positional information, such that the acquired content is put into practical use. The displayed content related to the positional information includes, for example, map information and information concerning surrounding facilities.

Mobile devices provided with bearing sensors are also known. The bearing sensors each detect an orientation of the mobile device. Content based on the detected orientation of the mobile devices is displayed on the LCDs in a user-friendly manner. For example, when the displayed content is map information, a function of aligning a direction of the mobile device with the upward direction of the map that is displayed is provided. When the display content is information concerning surrounding facilities, a function of extracting only the information about the direction of the mobile device and displaying the extracted information is provided. Nevertheless, a burden is imposed on the user when the user moves while visually recognizing the displayed content on the LCD of the mobile device.

A known head-mountable display displays sonic content similar to that displayed by the mobile device and may function as a display device. Specifically, a known technology displays the map information, which is the same as the displayed content displayed on the LCD of the mobile device, on the head-mountable display.

In addition, technologies to provide the hearing sensors in both mobile devices and headphones worn by the users on their heads are known. Specifically, a technology for varying an output condition of a musical sound based on a difference in orientation between the current bearing toward a destination and the current bearing of the head of the user is known. The technology further varies an output condition of a musical sound depending on a difference in orientation between the current bearing from an intermediate point that is not the current position toward the destination and the current bearing of the head of the user.

SUMMARY OF THE INVENTION

Only the difference in orientation between the current bearing from an intermediate point that is not the current position toward a destination or the difference in orientation between the current bearing toward the destination and the current bearing of the head of the user may be indicated to the user with the musical sound in the known technology. The map information, such as a route image, displayed on the LCD in the mobile device may be based on the bearing of the mobile device with respect to a set of reference directions (e.g., north, east, south, and west). Accordingly, when the head-mountable display is used as the display device, the map information may be based on the bearing of the mobile device with respect to the set of reference directions and may be displayed on the head-mountable display. Consequently, the map information displayed on the head-mountable device may not correspond to the current bearing of the front direction of the head of the actual user or the current bearing of the head-mountable device. The displayed map information may not be recognized by the user with the head-mountable display in such circumstances. To display the map information corresponding to the current bearing of the head of the user and the current bearing of the head-mountable device, the orientation of the reference directions displayed on the mobile device may be adjusted to match the current bearing of the front direction of the head of the user and the current bearing of the head-mountable device. In addition, the user may be required to use the mobile device while carrying the mobile device.

According to an embodiment of the disclosure, a system disclosed herein may comprise a head-mountable display and a mobile device. The mobile device may comprise a first processor and a first memory. The first memory may be configured to store first computer-readable instructions therein. The first computer-readable instructions may instruct the first processor to execute certain processes. The first computer-readable instructions may instruct the first processor to execute a process of acquiring positional information that indicates a position of the mobile device. The first computer-readable instructions may instruct the first processor to execute a process of acquiring first orientation information that indicates an orientation of the mobile device with respect to a reference direction. The first computer-readable instructions may instruct the first processor to execute a process of displaying content based on the positional information and the first orientation information on the mobile device when a particular condition is satisfied. The first computer-readable instructions may instruct the first processor to execute a process of receiving from the head-mountable display second orientation information that indicates art orientation of the head-mountable display with respect to one or more of the reference direction and the orientation of the mobile device. The first computer-readable instructions may instruct the first processor to execute a process of transmitting to the head-mountable display content based on the positional information and the second orientation information when the particular condition is not satisfied.

According to another embodiment of the disclosure, a mobile device disclosed herein may comprise a processor and a memory configured to store computer-readable instructions therein. The computer-readable instructions may instruct the processor to execute certain processes. The computer-readable instructions may instruct the processor to execute a process of acquiring positional information indicative of a position of the mobile device. The computer-readable instructions may instruct the processor to execute a process of acquiring first orientation information that indicates an orientation of the mobile device with respect to a reference direction. The computer-readable instructions may instruct the processor to execute a process of displaying content based on the positional information and the first orientation information on the mobile device when a particular condition is satisfied. The computer-readable instructions may instruct the processor to execute a process of receiving second orientation information that indicates an orientation of a head-mountable display with respect to one or more of the reference direction and the orientation of the mobile device. The computer-readable instructions may instruct the processor to execute a process of transmitting to the head-mountable display content based on the positional information and the second orientation information when the particular condition is not satisfied.

According to still another embodiment of the disclosure, a non-transitory, computer-readable storage medium disclosed herein may store computer-readable instructions thereon. When executed, the computer-readable instructions may instruct a mobile device to execute certain processes. The computer-readable instructions may instruct the mobile device to execute a process of acquiring positional information indicative of a position of the mobile device. The computer-readable instructions may instruct the mobile device to execute a process of acquiring first orientation information that indicates an orientation of the mobile device with respect to a reference direction. The computer-readable instructions may instruct the mobile device to execute a process of displaying content based on the positional information and the first orientation information on the mobile device when a particular condition is satisfied. The computer-readable instructions may instruct the mobile device to execute a process of receiving second orientation information that indicates an orientation of a head-mountable display with respect to one or more of the reference direction and the orientation of the mobile device. The computer-readable instructions may instruct the mobile device to execute a process of transmitting to the head-mountable display content based on the positional information and the second orientation information when the particular condition is not satisfied.

According to yet another embodiment of the disclosure, a system disclosed herein may comprise a head-mountable display and a mobile device. The mobile device may comprise a position acquisition device configured to acquire positional information that indicates a position of the mobile device. The mobile device may comprise a first orientation acquisition device configured to acquire first orientation information that indicates an orientation of the mobile device with respect to a reference direction. The mobile device may comprise a first display configured to display content based on the positional information and the first orientation information on the mobile device when a particular condition is satisfied. The mobile device may comprise a first receiver configured to receive from the head-mountable display second orientation information that indicates an orientation of the head-mountable display with respect to one or more of the reference direction and the orientation of the mobile device. The mobile device may comprise a first transmitter configured to transmit to the head-mountable display content based on the positional information and the second orientation information when the particular condition is not satisfied.

Other objects, features, and advantages will be apparent to persons of ordinary skill in the art from the following detailed description of embodiments of the invention and the accompanying drawings.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the present disclosure now are described with reference to the attached drawings.

[1. Outline of Navigation System]

FIG. 1depicts an outline of a navigation system, which is an example of a position-related display system. The navigation system depicted inFIG. 1may comprise a mobile phone10, which is a mobile device, a head-mountable display (“HMD”)20, and a connection cable30. The HMD20may be worn by a user2on the head., The connection cable30may be used to connect the HMD20and the mobile phone10. In alternative, configurations, mobile phone10may be replaced by a mobile device other than a mobile phone and cable30may be replaced by a wireless connection, implemented by transmitters and receivers comprised in mobile phone30and HMD20, for example, which may transmit wireless signals therebetween.

The mobile phone10may comprise the above-described UPS function and the above-described communication function. The UPS fraction may be used to acquire positional information about the mobile phone10. The mobile phone110may determine positional information about the mobile phone10itself with the GPS function. The mobile phone10may acquire display content that may be based on the positional information by utilizing the communication function. In the navigation system ofFIG. 1, the display content may be, for example, map information. The mobile phone10may communicate with, for example, a UPS satellite40using the UPS function to determine the positional information about the mobile phone10, such as, for example, a longitude and a latitude. The mobile phone10may acquire the map information within a certain range, based on the positional information determined with the UPS function, from an external server52connected to the Internet50via, for example, a relay station51. The map information within the certain range may be, for example, the map information corresponding to locations within a range of 2 km in the directions of north, south, east, and west around a location corresponding to the positional information about the mobile phone10. The range of the map information to be acquired may be arbitrarily varied based on input into the mobile phone10. Even when the mobile phone10is moved, the acquisition of the map information based on the positional information about the mobile phone110may allow the map information based on the most recent positional information acquired after the movement, in certain configurations, the map information may not be acquired from the external server52connected to the Internet50via, the relay station51. For example, the map information may be stored in advance in a storage device (not depicted), such as, for example, a memory card, or a hard disk drive (“HMD”) in the mobile phone10.

The user may use the HMD20to recognize visually an image generated on the basis of image information, such as the map information. The HMD20may be, for example, a see-through HMD. When the see-through HMD20is used, a polarization member, such as a half minor, a prism lens, or a hologram lens, each of which may be configured to transmit at least a portion of light external to the HMD20, may be arranged in front of the user. The user may use the HMD20, which is the see-through HMD, to visually recognize the image overlapped on an external scene (e.g., environmental objects in the user's line of sight). The HMD20may be, for example, a retinal-scanning HMD, in which laser light that is scanned may be incident on the eyes of the user. Alternatively, the HMD20may comprise a spatial light modulation element, such as an LCD or an organic light emitting display (“OLED”). In the retinal-scanning HMD, the laser light may comprise an intensity and a color corresponding to each pixel in an image corresponding to image information and may be emitted sequentially. The emitted laser light may be two-dimensionally scanned, The laser light that is two-dimensionally scanned may be projected onto the retina of one or more eye of the user wearing the HMD20. In the HMD provided with the LCD as the spatial light modulation element, the image corresponding to the image information may be displayed on the LCD. The image displayed on the LCD may be formed on the retina of one or more eye of the user wearing the HMD20. The HMD20may not be limited to the retinal-scanning HMD and the HMD provided with the LCD, but rather the HMD20may be selected from many varieties of HMDs that may be configured to display the image that corresponds to the image information, such as the map information, and that may be configured to display the image overlapping the external scene without blocking the view of the user2.

The HMD20may connect to the mobile phone10via the connection cable30, as described above. The connection cable30may be removable from the mobile phone10. When using the navigation system depicted inFIG. 1, the mobile phone10may be connected to the HMD20via the connection cable30. The connection between the mobile phone10and the HMD20via the connection cable30may allow the map information acquired by the mobile phone10to be transmitted to the HMD20via a mobile phone communication device15(depicted inFIG. 2), as described below, and the map information may be displayed on the HMD20. The connection for communication between the HMD20and the mobile phone10may be via a wireless connection, such as, for example, Bluetooth communication, instead of the connection via the wired connection cable30. The Bluetooth communication may be one of short-distance wireless communication technologies used for connection between, for example, mobile information devices that are apart from each other by several meters. Other known wireless connections may be suitable.

The mobile phone10may comprise means for acquiring first bearing information indicating an orientation of the mobile phone10. The HMD20may comprise means for acquiring second bearing information indicating the orientation of the HMD20. When the map information acquired by the mobile phone10is displayed in the mobile phone10, mobile device map information on the basis of the first bearing information with respect to the mobile phone10may be displayed. In contrast, when the map information acquired by the mobile phone10is displayed on the HMD20, head-mountable display map information (“HMD map information”) based on the second bearing information indicating the orientation of the HMD20may be displayed, instead of the mobile device map information based on the first bearing information indicating the orientation of the mobile phone10.

As described above, in the navigation system ofFIG. 1, when the map information is viewed with the mobile phone10, the mobile phone10may display, for example, the current position of the mobile phone10and the geography around the mobile phone10with the map information based on the orientation of the mobile phone10. When the map information is viewed with the HMD20, the mobile phone10may display, for example, the current position of the HMD20and the geography around the HMD20with the map information based on the orientation of the HMD20.

FIG. 2andFIG. 3depict in detail the configurations of the mobile phone10and the HMD20in the navigation system depicted inFIG. 1. As depicted inFIG. 2, the mobile phone10may comprise a casing comprising, for example, a display device11on a side face and a casing comprising a mobile phone operation device19on another side face. The casings may be foldably attached to each other. As depicted inFIG. 2, the display device11may be made visible in a state in which both the casings are configured to be opened. As depicted inFIG. 3, the mobile phone10may comprise a first bearing sensor12, a mobile phone communication device15, a map information acquirer16, a positional information acquirer17, an acceleration sensor18, and a mobile phone controller10a. The mobile phone10may also have a well-known call function and a mail transmission and reception function.

The display device11may be, for example, a rectangular LCD. For example, the mobile device map information may be displayed on the display device11. Information for executing the call function and the mail transmission and reception function may be displayed on the display device11, in addition to the mobile device map information.

The first bearing sensor12may acquire the first bearing information indicating the direction of the mobile phone10. The first bearing sensor12may be disposed at a certain position in the mobile phone10. In the configuration inFIG. 2, the first bearing sensor12may be arranged at an upper right corner of the display device11. In certain configurations, the first bearing sensor12may be disposed at other locations. The first bearing sensor12may be configured to detect the orientation of the long sides of the display device11(e.g., an orientation of a direction parallel to one or more of the long sides of mobile phone10relative to north, east, south, and west directions) as the orientation of the mobile phone10in a state in which the user holds the mobile phone10in a substantially horizontal position (e.g., horizontal with respect to gravity) to make the display device11visible. For example, a magnetic sensor may be used as the first bearing sensor12. The magnetic sensor may be, for example, a compact and light-weight Magneto Impedance (“MI”) sensor that may be installed in the mobile phone and that may use an MI effect of a magnetic body having a soft magnetic property.

The mobile phone communication device15may function as a mobile device side communication device. Specifically, the mobile phone communication device15may be used to transmit and receive a variety of data between the mobile phone10and the HMD20. As depicted inFIG. 3, the mobile phone communication device15may comprise a map information processor13and a second bearing information receiver14. The map information processor13may transmit the HMD map information generated in the mobile phone10to the HMD20. The second bearing information receiver14may receive the second bearing information indicating the direction of the HMD20from the HMD20. The data received by the second bearing information receiver14may comprise, for example, the operation states of a power ON-OFF button26aand a display switching button26bprovided in the HMD20(described below) in addition to the second bearing information indicating the direction of the HMD20. The data received by the second bearing information receiver14may also comprise, for example, the output state of laser light emitted from a projection part27in the HMD20(described below). The operation states of the power ON-OFF button26aand the display switching button26bprovided in the HMD20may be the state detection means for detecting the state of one or more of the mobile phone10and the HMD20. The output state of the laser light emitted from the projection part27in the HMD20may also be the state detection means for detecting the state of one or more of the mobile phone10and the HMD20.

The positional information acquirer17may function as means for acquiring the positional information about the mobile phone10. The positional information acquirer17may communicate with the GPS satellite40to determine, for example, the positional information about the mobile phone10, such as a longitude and a latitude of the mobile phone10. The positional information acquirer17may also function as movement detecting means for the mobile phone10(described below).

The map information acquirer16may acquire the map information based on the positional information about the mobile phone10determined by the positional information acquirer17. The map information may be acquired from the external server52connected to the Internet50via the relay station51. Alternatively, the map information acquirer16may read out, for example, the map information stored in advance in a storage device, such as a memory card or a built-in HDD, in the mobile phone10based on the positional information about the mobile phone10, instead of acquiring the map information by the external communication.

The acceleration sensor18may function as the movement detecting means for the mobile phone10. The acceleration sensor18may detect movement of the mobile phone10at an acceleration greater than a predetermined value. Start of movement from a stopped state or stop of movement from a moving state of the mobile phone10may be estimated based on the detected acceleration. In addition, a variation in the positional information acquired by the positional information acquirer17may be monitored for a predetermined time period. The start of movement and the stop from the moving state of the mobile phone10may be identified based on the result of the monitoring and the result of the detection of by the acceleration sensor18The acceleration sensor18may function as the movement detecting means for detecting that the position of the mobile phone10is moving based on the variation in the positional information acquired by the positional information acquirer17and the result of the detection by the acceleration sensor18. Although the acceleration sensor18may function as the movement detecting means, the movement detecting means may not be limited to the acceleration sensor18and a speed sensor, for example, or other known movement detecting means, may be used.

The mobile phone operation device19may comprise various buttons used to operate the mobile phone10. The various buttons may be operated to execute the call function and the mail transmission and reception function of the mobile phone10. A certain operation with the mobile phone operation device19may activate the navigation system. Specifically, monitoring of the operation of the mobile phone operation device19may function as operation determining means for determining whether the mobile phone10is being operated.

The mobile phone controller10amay control the entire mobile phone10. The mobile phone controller10amay comprise, for example, a central processing unit (“CPU”) and a non-transitory, computer-readable memory, such as, for example, one or more of a read-only memory (“ROM”) and a random access memory (“RAM”) (not depicted), configured to store computer-readable instructions for instructing the controller to perform certain processes. The mobile phone controller10a. may comprise an Application Specific Integrated Circuit (“ASIC”) or a Field programmable Gate Array (“FPGA”). The CPU in the mobile phone controller10amay read out a mobile device program stored in the ROM to execute a navigation process (depicted inFIG. 7and described below). The execution of the navigation process by the CPU in the mobile phone controller10amay cause the mobile phone controller10ato generate the map information based on the positional information about the mobile phone10and the orientation of the mobile phone10or the orientation of the HMD20.

The HMD20according to the present embodiment may comprise a mounting device23and a control device21, as depicted inFIG. 2. The mounting device23may be a glass frame-type device configured to be worn by the user on the head. The control device21may project the image, which is generated based on the image information, such as the map information, toward the eyes of the user to display the image information overlapping the external scene. The mounting device23may comprise a left-side frame23aand a right-side frame23cconfigured to be worn by the user on the left and right ears, respectively, and a central frame23b. The left-side frame23amay be attached to the right-side frame23cvia the central frame23b. Because the mounting device23may be formed of a glass frame type device in the above manner, the user may easily wear the HMD20on the head as if the HMD20were a pair of glasses.

A second bearing sensor22may be provided at a central position of the central frame23bof the mounting device23as means for acquiring the second bearing information indicating the orientation of the HMD20. The second bearing sensor22may be configured to detect the orientation of the HMD20, which corresponds to the orientation of the front face of the head of the user when the user wears the HMD20on the head. A magnetic sensor, such as a compact and light-weight MI sensor, may be used as the second bearing sensor22in a manner similar to the first bearing sensor12described above.

The control device21may be mountable to the left-side frame23aof the mounting device23and may be positioned at a side of the left eye of the user. Alternatively, the control device21may be mountable to the right-side frame23cof the mounting device23and may be positioned at a side of the right eye of the user (not depicted). The control device21may comprise a half minor29. The laser light emitted from the control device21may be reflected from the half minor29and may be incident on the left or right eye of the user, as appropriate. As depicted inFIG. 3, the control device21may comprise an HMD communication device25, a projection portion27, an HMD operation device26, and an HMD controller20a.

The HMD communication device25may function as communication means at the side of the HMD20. The HMD communication device25may comprise an HMD map information receiver28and a second bearing information transmitter24. The HMD map information receiver28may receive the HMD map information transmitted from the mobile phone10, The second bearing information transmitter24may transmit the second bearing information indicating the orientation of the HMD20relative to the mobile phone10. The second bearing information transmitter24may transmit, for example, an operation state of the HMD operation device26(described below) to the mobile phone10in addition to the second bearing information. In addition, the second bearing information transmitter24may transmit, for example, an output state of the laser light emitted from the projection part27(described below) to the mobile phone10.

The projection part27may project the image corresponding to the received HMD map information onto the eyes of the user, such that the image is overlapped on the external scene. The image overlapped on the external scene may be formed on the retina of the user when the projection part27projects the laser light for displaying the HMD map information onto an eye of the user.

The HMD operation device26may comprise a power ON-OFF button26aand a display switching button26b. The power ON-OFF button26amay be used to turn on the power to use the HMD20. The power ON-OFF button26amay be also used to turn off the power to deactivate the HMD20. The display switching button26bmay be used by the user to switch selectively between display and non-display of the image information, such as the HMD map information, on the HMD)20. As described above, the display switching button26bmay be operated to not display the HMD map information or to display the HMD map information again, even when the HMD map information is configured to be displayed on the HMD20.

The HMD controller20amay control the HMD20. The HMD controller20amay comprise, for example, a CPU and a non-transitory, computer-readable memory, such as, for example, one or more of a ROM, and a RAM (not depicted), configured to store computer-readable instructions for instructing the controller to perform certain processes. The HMD controller20amay comprise an ASIC or an FPGA. Specifically, the HMD controller20amay control the second bearing information transmitter24. The second bearing information transmitter24may acquire the second bearing information about the HMD20from the second bearing sensor22and transmit the acquired second bearing information to the mobile phone10. The HMD controller20amay control the projection part27to emit the received HMD map information to the eye of the user as the laser light. The HMD map information may be transmitted from the mobile phone10and be received by the HMD20via the HMD map information receiver28. The CPU in the HMD controller20amay be configured to execute, for example, an HMD map information displaying process (depicted inFIG. 8and described below) in the above-described manner.

[3. Display Mode of Navigation System]

In the navigation system ofFIG. 1(described above), when the mobile phone10is used to confirm the map information, the mobile device map information based on the positional information about the mobile phone10and the first bearing information indicating the orientation of the mobile phone10in relation to the map information may be displayed on the display device11, In contrast, when the HMD20is used to confirm the map information, the HMD map information based on the positional information about the mobile phone10and the second bearing information indicating the orientation of the HMD20in relation to one or more of the map information and the first bearing information may be displayed on the HMD20. A difference in the display mode of the map information between when the user confirms the map information displayed on the display device11in the mobile phone10and when the user confirms the map information displayed on the HMD20will now be described with reference toFIG. 4andFIG. 5.

When the user confirms the mobile device map information displayed on the display device11in the mobile phone10, as depicted inFIG. 4, a map11amay be displayed on the display device11. The map11amay be a map within a certain range substantially around the current positional information about the mobile phone10. It may be assumed inFIG. 4that the destination may be located in a direction east of the current position within the certain range. An icon11bmay be overlapped on the map11a. The icon11bmay indicate the current positional information about the mobile phone10and the orientation of the mobile phone10in relation to the features on the map11a. Because the mobile device map information with respect to the orientation of the mobile phone10is displayed on the display device11in the mobile phone10as described above, the orientation of the icon11bon the mobile device map information may coincide with the orientation of the mobile phone10and may point in a direction which is the same as a pointing direction of the mobile phone10. For example, when the orientation of the mobile phone10points in the north direction, the direction of the icon11bon the mobile device map information may also point in the north direction.

On the other hand, when using the HMD20to confirm the HMD map information, as depicted inFIG. 5, a map21amay be displayed on the HMD20. The map21amay be a map within a certain range substantially around the current positional information about the mobile phone10. Specifically, the HMD map information with respect to the orientation of the HMD20may be displayed on the HMD20.FIG. 5depicts a state in which the user faces east. An icon21bindicating the current positional information about the mobile phone10and the orientation of the HMD20may be overlapped on the map21ainFIG. 5. The destination may be straight ahead of the current position on the map21a, based on the orientation of the HMD20.

When the user faces wearing the HMD20faces east, the orientation of the HMD20also may be directed to east. Accordingly, the orientation of the icon21bon the HMD map information may point in the east direction, even when the orientation of the mobile phone10is points in the north direction, as depicted inFIG. 5. In other words, when the orientation of the mobile phone10is different from that of the HMD20, the HMD map information with respect to the direction of the HMD20may be displayed on the HMD20. Accordingly, it may not be necessary to pay attention to the direction of the mobile phone10, and the mobile phone10may be stored in a pocket, a handbag, or another storage location while using the HMD20.

In the example depicted inFIG. 5, the map21amay be displayed such that east becomes the upward direction in the map displayed in the HMD20, in accordance with the orientation of the HMD20and, correspondingly, the orientation of the user, as indicated by the orientation of the icon21b. Specifically, the map21amay be displayed in a state in which the map11aon the mobile device map information depicted inFIG. 4is rotated counterclockwise by 90° along with the rectangular display area. In other words, the map11amay be displayed such that the pointing direction of the mobile phone10(e.g., east, south, west, or north) is the upward-pointing direction on the mobile device map information displayed on the display device11in the mobile phone10. The map21amay be displayed such that the pointing direction of the HMD20may be the upward-pointing direction on the HMD map information displayed on the HMD20. Accordingly, it may be possible to easily confirm the current position of the user and the destination regardless of whether the mobile device map information or the HMD map information is used.

The orientations of the map11aand the map21aon the respective displays may not be varied in accordance with the directions of the mobile phone10and the HMD20. For example, the map11aand the map21amay be displayed fixedly such that the north direction constantly is the upward-pointing direction and the orientation of each of the icon11band the icon21bmay be varied in accordance with the respective orientations of the mobile phone10and the HMD20to generate the mobile device map information and the HMD map information.

The navigation system depicted inFIG. 1may be configured such that, when the HMD20is operating, the HMD map information may be displayed on the HMD20only when it is detected that one or more of the HMD20and the mobile phone10are moving. An example of displaying the HMD map information on the HMD20when one or more of the HMD20and the mobile phone10are moving is described below, with reference toFIGS. 6A and 6B.FIGS. 6A and 6Bdepict an example of how the display of the HMD map information on the HMD20may be varied between a state when one or more of the HMD20and the mobile phone10are moving and a state when the HMD20and the mobile phone10are not moving.

When one or more of the HMD20and the mobile phone10are moving, as depicted inFIG. 6A, the map21aand the icon21bindicating the positional information about the mobile phone10and the direction of the HMD20may be displayed in a display area21cfor the HMD map information on the HMD20as the HMD map information. Accordingly, it may be possible for the user to move to the destination while confirming the HMD map information that is varied in accordance with the direction of the movement.

Nevertheless, the display of the HMD map information on the HMD20may be stopped when the user stops, as depicted inFIG. 6B. In other words, the display area21cin which the HMD map information is displayed may be in the non-display state. In this case, the eye of the user may visualize the image of the actual scene around the user. As described above, the navigation system may display the HMD map information on the HMD20when one or more of the HMD20and the mobile phone10are moving. Consequently, for example, when the user stops to operate the mobile phone10, it may be possible to prevent the HMD map information displayed on the HMD20from disturbing the user. When the HMD20and the mobile phone10stop moving, the mobile device map information may be displayed on the display device11in the mobile phone10. Accordingly, the user may view the map information using the mobile phone10, on an as-needed basis.

When one or more of the HMD20and the mobile phone10starts to move again, the map21aand the icon21bindicating the positional information about the mobile phone10and the orientation of the HMD20may be displayed on the display area21cagain as the HMD map information, as depicted inFIG. 6A.

The acceleration sensor18and the positional information acquirer17may be used to detect the movement and the stopping of the mobile phone10, as described above. Specifically, when the acceleration sensor18detects that an acceleration of the mobile phone10is greater than a predetermined value, it may be possible to estimate that the mobile phone10has changed from the stop state to the movement state or from the movement state to the stop state. Then, when the acceleration sensor18detects that an acceleration of the mobile phone10is greater than the predetermined value and the positional information acquirer17detects that the positional information is varied in a predetermined time period, it may be determined that the mobile phone10has started to move from the stop state. When the positional information is not varied in the predetermined time period, it may be determined that the mobile phone10has transitioned from the movement state to the stop state.

[4. Control Processes in Mobile Phone10and HMD20]

Control processes executed by the mobile phone10and the HMD20in the navigation system described above now are described with reference toFIG. 7andFIG. 8.

The navigation process executed by the mobile phone controller10ain the mobile phone10now is described with reference toFIG. 7. When the mobile phone controller10acomprises the CPU, the navigation process may be executed by the CPU. The navigation process may be executed when a certain operation is performed with the mobile phone operation device19in the mobile phone10.

Referring toFIG. 7, at S101, the mobile phone controller10amay communicate with multiple GPS satellites40with the positional information acquirer17to acquire the positional information, such as a longitude and a latitude. The positional information about the mobile phone10may be determined from the acquired positional information. At S102, the mobile phone controller10amay acquire the map information based on the positional information about the mobile phone10with the map information acquirer16from the external server52connected to the Internet50via the relay station51or from a storage device in the mobile phone10. In the process of S102, for example, the positional information about the mobile phone10determined by the positional information acquirer17and a certain range of the map information may be transmitted to the external server52connected to the Internet50via the map information acquirer16. The external server52or the storage device in the mobile phone10may generate the map information corresponding to the certain range substantially around the received positional information about the mobile phone10and may transmit the generated map information to the map information acquirer16. The map information received in the above-described manner may be displayed in, for example, the display device11in the mobile phone10as the map11a (depicted inFIG. 4).

At S103, the mobile phone controller10amay determine whether the second bearing information is received from the HMD20. When the mobile phone controller10adetermines that the second bearing information is not received from the HMD20(S103: NO), the process may proceed to S110. When the mobile phone controller10adetermines that the second bearing information is received from the HMD20(S103: YES), the process may proceed to S104.

At S103, the mobile phone controller10amay determine whether the second bearing information indicating the orientation of the HMD20transmitted from the second bearing information transmitter24in the HMD20(depicted inFIG. 3) is received by the second bearing information receiver14in the mobile phone10(depicted inFIG. 3). The reception of the second bearing information from the HMD20during the navigation process may allow the mobile phone10to acquire the most recent orientation of the HMD20and, correspondingly, the orientation of the user. In addition, the reception of the second bearing information from the HMD20may indicate that the HMD20is connected to the mobile phone10via the connection cable30or via a wireless connection.

At S104, the mobile phone controller10amay acquire information regarding the operation state of the mobile phone operation device19and may proceed to S105thereafter. At S105, the mobile phone controller10amay determine whether the mobile phone operation device19currently is being operated. When the mobile phone controller10adetermines that the mobile phone operation device19currently is being operated (S105: YES), the process may proceed to S110, When the mobile phone controller10adetermines that the mobile phone operation device19currently is not being operated (S105: NO), the process may proceed to S106.

At Step S106, the mobile phone controller10amay confirm the display state on the HMD20to determine whether the HMD map information may be displayed on the HMD20. When the mobile phone controller10adetermines that the HMD map information may be displayed on the HMD20(S106: YES), the process may proceed to S107. When the mobile phone controller10adetermines that the HMD map information may not be displayed on the HMD20(S106: NO), the process may proceed to S110.

The display state of the HMD20determined at Step S106may correspond to, for example, the operation state of the display switching button26bin the HMD20. The operation state of the display switching button26bin the HMD20may be received along with the second bearing information from the HMD20. Alternatively or additionally, the display state of the HMD20determined at S106may correspond to, for example, the output state of the laser light emitted from the projection part27. Specifically, when the mobile phone controller10adetects that the display switching button26bis set to the non-display state or that the output of the laser light is set to a decreased state, the mobile phone controller1.0amay determine that the HMD map information may not be displayed on the HMD20.

At S107, the mobile phone controller10amay determine whether the mobile phone10is moving. When the mobile phone controller10adetermines that the mobile phone10is not moving (S107: NO), the process may proceed to S110. When the mobile phone controller10adetermines that the mobile phone10is moving (S107: YES), the process may proceed to S108.

The determination of whether the user of the mobile phone10is moving at S107may be performed by, for example, the acceleration sensor18and the positional information acquirer17in the mobile phone10. Specifically, when the acceleration sensor18detects that the mobile phone10moves at an acceleration or deceleration greater than a predetermined value, it may be estimated that the mobile phone10transitions from the stop state to the movement state or from the movement state to the stop state. When the positional information acquirer17detects that the positional information is varied in a predetermined time period, it may be determined that the mobile phone10starts to move from the stop state. When the positional information is not varied in the predetermined time period, it may be determined that the mobile phone10makes a transition from the movement state to the stop state.

At S108, the mobile phone controller10amay generate the HMD map information based on the second bearing information received from the HMD20, the positional information acquired by the positional information acquirer17, and the map information acquired by the map information acquirer16, and the process may proceed to S109. At S109, the mobile phone controller10amay transmit the HMD map information generated at S108to the HMD20via the map information processor13. The icon21bindicating the direction corresponding to the orientation of the HMD20may be displayed as the HMD map information, as depicted inFIG. 5. Subsequently, the process may proceed to S112.

At S110, the mobile phone controller10amay acquire the first bearing information indicating the orientation of the mobile phone10with the first bearing sensor12. The mobile phone controller10amay generate the mobile device map information (depicted inFIG. 4) based on the acquired first bearing information, the positional information acquired by the positional information acquirer17, and the map information acquired by the map information acquirer16, and the process may proceed to S111. At S111, the mobile phone controller10amay display the mobile device map information generated at S110on the display device11. Subsequently, the process may proceed to S112.

At S112, the mobile phone controller10amay determine whether to turn off the navigation system. When the mobile phone controller10adetermines to turn off the navigation system (S112: YES), the navigation process may be terminated. When the mobile phone controller10adetermines not to turn off the navigation system (S112: NO), the process may return to S101.

The HMD map information displaying process executed by the HMD controller20anow is described with reference toFIG. 8. When the HMD controller20acomprises the CPU, the HMD map information displaying process may be executed by the CPU. The HMD map information displaying process may be executed when the mobile phone10connects with the HMD20via one or more of the connection cable30and a wireless connection, and the HMD20is turned on.

Referring toFIG. 8, at S201, the HMD controller20amay determine whether the HMD20is connected to the mobile phone10via one or more of the connection cable30and a wireless connection. When the HMD controller20adetermines that the HMD20is connected to the mobile phone10via one or more of the connection cable30and a wireless connection (S201: YES), the process may proceed to S202. When the HMD controller20adetermines that the HMD20is not connected to the mobile phone10(S201: NO), the process may proceed to S207.

At S201, the connection between the HMD20and the mobile phone10may be determined, for example, in the following manner. When the mobile phone controller10adetects a connection between the HMD20and the mobile phone10(S201: YES), the mobile phone controller10amay submit a transmission request for the second bearing information to the HMD20via the mobile phone communication device15in the mobile phone10. When the transmission request for the second bearing information is received by the HMD communication device25, the HMD20may determine that the HMD20is connected to the mobile phone10.

At S202, the HMD controller20amay acquire the second bearing information indicating the orientation of the HMD20from the second bearing sensor22in the HMD20, and the process may proceed to S203. At S203, the HMD controller20amay transmit the acquired second bearing information to the mobile phone10via the second bearing information transmitter24.

At S203, the HMD controller20aalso may transmit information about one or more of the operation state of the display switching button26bin the HMD20and the output state of the laser light emitted from the projection part27via the second bearing information transmitter24. Subsequently, the process may proceed to S204.

At S204, the HMD controller20amay perform a reception waiting process. In the reception waiting process, the HMD controller20amay wait for the reception of the HMD map information from the mobile phone10. After the second bearing information and the display state of the HMD20are transmitted to the mobile phone10, the HMD controller20amay wait a predetermined time for the reception of the HMD map information. After the predetermined time has elapsed, the process may proceed to S205. Alternatively, the process may proceed to S205immediately after reception of the HMD map information at the HMD20.

At S205, the HMD controller20amay determine whether the HMD map information transmitted from the mobile phone10has been received by the HMD20. When the HMD controller20adetermines that the HMD map information has not been received by the HMD20(S205: YES), the process may proceed to S206. When the HMD controller20adetermines that the HMD map information has not been received by the HMD20(S205: NO), the process may proceed to S207.

At S206, the HMD controller20amay display the HMD map information received from the mobile phone10on the HMD20. Specifically, the HMD map information may be emitted from the projection part27toward the eye of the user as image light, such that the HMD map information may be visible to the user. Then, the process may proceed to S208,

At S207, the HMD controller20amay turn off the image display of the HMD map information on the HMD20. Specifically, when the HMD controller20adetermines that the HMD20is not connected to the mobile phone10(S201: NO) or the the HMD map information has not been received in a predetermined time period (S205: NO), the display of the HMD map information may be stopped. Subsequently, the process may proceed to S208.

At S208, the HMD controller20amay determine whether the power ON-OFF button26ain the HMD operation device26is operated to turn off the power of the HMD20. When the HMD controller20adetermines that the power of the HMD20is turned off (S208: YES), the HMD map information displaying process may be terminated. When the HMD controller20adetermines that the power of the HMD20is not turned off (S208: NO), the process may return to S201.

As described above, the HMD map information corresponding to the orientation of the HMD20may be displayed on the HMD20when the mobile phone10is connected to the HMD20via the connection cable30or a wireless connection. Accordingly, the current position of the HMD20and the destination from the HMD map information generated based on the orientation of the HMD20may be displayed without regard to the orientation of the mobile phone10.

In addition, a setting of one of a HMD display state and a HMD non-display state, in which the HMD20displays or does not display, respectively, the HMD map information may be selected. Accordingly, for example, in an emergency or when the HMD20reaches the destination, it may be sufficient to set the HMD map information on the HMD20to the non-display state. Therefore, it may not be necessary to remove the HMD20worn by the user on the head or to disconnect the connection cable30or the wireless connection to make the HMD map information invisible.

Furthermore, the mobile phone10may comprise the acceleration sensor18used to detect a moving state and a stop state of the mobile phone10, The acceleration sensor18and the positional information acquirer17may be used to identify the movement state or the stop state of the mobile phone10. When it is detected that the mobile phone10transitions from the movement state to the stop state, the transmission of the HMD map information from the mobile phone10to the HMD20may be stopped, and the display of the HMD map information may be turned off on the HMD20. When it is detected that the mobile phone10transitions from the stop state to the movement state, the transmission of the HMD map information from the mobile phone10to the HMD20may be restarted and the HMD map information may be displayed on the HMD20.

Although the map information displayed on the display device11in the mobile phone10and on the HMD20is described as a two-dimensional planar map, the map information may not be limited to this. For example, the map information may be displayed as a bird's eye view or a panoramic three-dimensional map. Alternatively, a two-dimensional planar map may be displayed in the mobile phone10and a bird's eye view or a panoramic three-dimensional map may be displayed on the HMD20or vice versa. The display mode in the mobile phone10may be differentiated from that on the HMD20in the above manner. Alternatively, one or more of the HMD20and the mobile phone10may selectively display desired map information based on input at the mobile phone operation unit19or elsewhere. When a bird's eye view or a panoramic three-dimensional map is displayed as the map information, the icon11bor21bcorresponding to the current position or the direction of the user may not be displayed.

Although the maps11aand21aand the icons11band21bare displayed as the map information in the present embodiment, the map information may not be limited to the maps11aand21aand the icons11band21b. For example, a route to the destination may be displayed on the maps11aand21a.

The navigation system may be configured as the position-related display system described above. Nevertheless, another position-related display system may be used as long as display content is acquired based on the positional information about the mobile phone10. For example, an Augmented Reality (“AR”) system may be used as the position-related display system. The AR system, for example, may acquire information about facilities around the mobile phone10, rather than or in addition to map information, and the navigation system may use the information about the facilities as the display content, and may display the information about facilities in accordance with the orientation of one of the mobile phone10and the HMD20.

In the above-described navigation system, the mobile phone10may be moving to a certain position and the HMD20may display the HMD map information generated on the basis of the positional information about the mobile phone10and the second bearing information indicating the orientation of the HMD20. In addition, in the above-described navigation system, the mobile phone10may be moving to a certain position and one or more of the HMD20and the mobile phone10may display one or more of the device map information and the HMD map information displayed based on the variation in the usage state of one or more of the mobile phone10and the HMD20. Furthermore, in the above-described navigation system, the mobile phone10arbitrarily may display the mobile device map information on the display device11while the HMD map information is being displayed on the HMD20.

Furthermore, in the above-described navigation system, for example, even though the HMD map information may not be displayed on the HMD20under certain conditions the HMD20is the off state or the HMD20is not connected to the mobile phone10), it may be possible to view the mobile device map information on the mobile phone10, such that functionality of the navigation system may be realized.

Furthermore, in the above-described navigation system, it may be possible to display the HMD map information on the HMD20, for example, when the HMD20is moving.

Furthermore, in the above-described navigation system, the connection between the mobile phone10and the HMD20may allow the HMD map information to be displayed on the HMD20.

Furthermore, in the above-described navigation system, for example, the installation of the mobile device program in the mobile phone may allow the HMD map information to be displayed on the HMD20.

While the invention has been described in connection with various exemplary structures and illustrative embodiments, it will be understood by those skilled in the art that other variations and modifications of the structures, configurations, and embodiments described above may be made without departing from the scope of the invention. For example, this application comprises possible combinations of the various elements and features disclosed herein, and the particular elements and features presented in the claims and disclosed above may be combined with each other in other ways within the scope of the application, such that the application should be recognized as also directed to other embodiments comprising other possible combinations. Other structures, configurations, and embodiments consistent with the scope of the claimed invention will be apparent to those skilled in the art from a consideration of the specification or practice of the invention disclosed herein. It is intended that the specification and the described examples are illustrative with the true scope of the invention being defined by the following claims.