Head mounted display device for displaying image and method thereof

A head mounted display (HMD) device, a method by the HMD of displaying an image, a non-transitory computer readable recording medium, and a chipset are provided. The HMD device includes a color information obtaining unit configured to obtain color information regarding a lens arranged at a portion of the HMD device; and a control unit configured to output an image modified based on the color information via a display unit arranged at the HMD device. The method includes obtaining color information regarding a lens arranged at a portion of the HMD device; and outputting an image modified based on the color information via a display unit arranged at the HMD device.

PRIORITY

This application claims priority under 35 U.S.C. § 119(a) to a Korean Patent Application filed on Jul. 29, 2014 in the Korean Intellectual Property Office and assigned Serial No. 10-2014-0096765 and to a Korean Patent Application filed on Jan. 15, 2015 in the Korean Intellectual Property Office and assigned Serial No. 10-2015-0007447, the entire contents of both of which are incorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention relates generally to a wearable display device, and more particularly, to a Head Mounted Display (HMD) device.

2. Description of the Related Art

Along with reductions in weights and sizes of digital devices, various wearable devices have been developed. From among the wearable devices, Head Mounted Display (HMD) devices refer to various digital devices that are worn by a user on the head and provide multimedia content to the user. An HMD device may be embodied in any of various forms e.g., glasses, a helmet, etc., to be worn on the head.

Since an HMD device is used after being attached to the body of a user, the HMD device provides images to the user from various environments as the user moves. Therefore, it is necessary that the HMD device include lenses with various colors or transmissivity. For example, the HMD device may include a dark-colored lens to block sunlight at an outdoor location where the sun is strong or a transparent-colored lens used at an indoor location. Thus, the user may properly see his/her surroundings.

The HMD device may switch a lens arranged at a side thereof to a lens having another color. Alternatively, if a lens arranged at a side of an HMD device is a smart window capable of changing at least one of color and transmissivity thereof, the HMD device may control the color and transmissivity of the lens. As the color or transmissivity of a lens arranged at a side of an HMD device is modified, the HMD device may provide an image that may be more easily recognized by the user.

SUMMARY

The present invention has been made to address the above-mentioned problems and disadvantages, and to provide at least the advantages described below.

Accordingly, an aspect of the present invention provides an HMD device capable of providing content via user interfaces that may be more clearly recognized by a user and a method of displaying content via the HMD device.

In accordance with an aspect of the present invention, an HMD is provided. The HMD includes a color information obtaining unit configured to obtain color information regarding a lens arranged at a portion of the HMD device; and a control unit configured to output an image modified based on the color information via a display unit arranged at the HMD device.

In accordance with another aspect of the present invention, a method by an HMD device of displaying an image is provided. The method includes obtaining color information regarding a lens arranged at a portion of the HMD device; and outputting an image modified based on the color information via a display unit arranged at the HMD device.

In accordance with another aspect of the present invention, a non-transitory computer readable recording medium having recorded thereon a computer program for implementing a method by an HMD device of displaying an image is provided. The method includes obtaining color information regarding a lens arranged at a portion of the HMD device; and outputting an image modified based on the color information via a display unit arranged at the HMD device.

In accordance with another aspect of the present invention, a chipset for displaying an image is provided. The chipset is configured to obtain color information regarding a lens arranged at a portion of an HMD device, and output an image modified based on the color information via a display unit arranged at the HMD device.

Throughout the present disclosure, it will be understood that when a portion is referred to as being “connected to” another portion, it can be “directly connected to” the other portion or “electrically connected to” the other portion via another element. Furthermore, it will be further understood that the terms “comprises” and/or “comprising” used herein specify the presence of stated features or components, but do not preclude the presence or addition of one or more other features or components.

Throughout the present disclosure, “lens” may refer to an object capable of transmitting light therethrough. A lens is not limited to an object with surfaces for forming an optical image. For example, a lens may be an object with flat surfaces. A lens may be formed of glass, crystal, or plastic. Alternatively, a lens may be a smart window. A smart window may include an element of which at least one of color or transmissivity can be changed based on a signal applied thereto. For example, an electrochromic glass, a Suspended Particle Device (SPD), or a Liquid Crystal (LC) may be used to form a smart window.

Throughout the present disclosure, an image displayed by an HMD device includes not only an image displayed directly on a lens, but also an image recognized by a user like as being displayed on a lens.

Furthermore, throughout the present disclosure, “displaying an image on a lens” includes not only a case where an image is displayed directly on a lens, but also a case where an image is output to be recognized by a user like as being displayed on a lens.

FIG. 1illustrates an HMD device100according to an embodiment of the present invention.

Referring toFIG. 1, the HMD device100may be embodied in the form of eyeglasses that may be fixed to the face of a user (or a wearer) via the ears and the nose of the user. However, a configuration of the HMD device100is not limited thereto. For example, the HMD device100may be attached to a helmet or may be embodied in the form of goggles.

The HMD device100according to an embodiment of the present invention may include a frame110. The HMD device100according to an embodiment of the present invention may include a nosepiece140to be fixed on the nose of a user at the center portion130. According to an embodiment of the present invention, the nosepiece140may include bridge arms131and pads141.

According to an embodiment of the present invention, a detachable lens120may be attached to the frame110. A plurality of lenses120of different colors may be used. For example, the user may use the HMD device100having attached thereto a clear lens120indoors and may use the HMD device100having attached thereto a lens120with a dark color outdoors. For example, clips121arranged at the lens120may be attached to the bridge arms131of the HMD device100.

Furthermore, the HMD device100may include a display unit111for displaying images. The display unit111may be embodied in various forms according to an embodiment of the present invention. For example, the display unit111may be configured to form images on the retina of a user or may be configured to display images on a lens as shown inFIG. 1.

FIG. 2illustrates an HMD device100according to an embodiment of the present invention.

Referring toFIG. 2, the HMD device100may be embodied in the form of eyeglasses that may be fixed to the face of a user (or a wearer) via the ears and the nose of the user. However, the configuration of the HMD device100is not limited thereto. For example, the HMD device100may be attached to a helmet or may be embodied in the form of goggles.

The HMD device100shown inFIG. 2may include a lens frame201, right and left side-arms202and203, a lens204, an on-board computer205, a camera206, a user input unit207, sensors208,209, and210, a haptic module211, a display212, an optical output unit213, a microphone214, right and left speakers215and216, and a power supply unit217.

Some of the components included in the HMD device100may be arranged internally to the HMD device100, whereas the remaining components may be arranged externally to the HMD device100. For example, the sensors208,209, and210may be arranged internally to the HMD device100. The display212may be arranged externally to the HMD device100. Components arranged internally to the HMD device100and components arranged externally to the HMD device100are not limited thereto.

Components included in the HMD device100are not limited to those shown inFIG. 2. For example, the HMD device100may exclude some of the sensors208,209, and210and the haptic module211. For example, the HMD device100may include only the sensor210from among the sensors208,209, and210or may include more sensors at more locations internally or externally to the HMD device100than those shown inFIG. 2.

The lens frame201and the right and left side-arms202and203may be formed of a solid material, such as a plastic and/or a metal, and may have a hollow structure to include wiring for interconnecting components included in the HMD device100.

The lens frame201and the right and left side-arms202and203shown inFIG. 2may be integrated with each other without a connecting member. Therefore, the lens frame201and the right and left side-arms202and203may be referred to collectively as a frame of the HMD device100.

The structure of the lens frame201and the right and left side-arms202and203are not limited to that shown inFIG. 2. For example, the HMD device100may be configured to have a structure in which connecting members are arranged between the lens frame201and the right and left side-arms202and203. In other words, if the HMD device100is configured to have a structure in which a connecting member is arranged between the lens frame201and the right side-arm202and a connecting member is arranged between the lens frame201and the left side-arm203, the right and left side-arms202and203may be folded toward the lens frame201.

If the HMD device100is configured to have a structure in which a connecting member is arranged at the center201′ of the lens frame201, a connecting member is arranged between the lens frame201and the right side-arm202, a connecting member is arranged between the lens frame201and the left side-arm203, and connecting members are arranged at the respective centers202′ and203′ of the right and left side-arms202and203, the lens frame201and the right and left side-arms202and203may be folded inward via the respective connecting members, thereby reducing volume of the HMD device100.

The connecting members may be screw-assembled or may be attached via hooks and grooves. However, the present invention is not limited thereto.

The right and left side-arms202and203may be formed to have shapes to be hung on the ears of a user. The right and left side-arms202and203may be configured to extend to the back of the head of the user. If the right and left side-arms202and203are configured to extend to the back of the head of the user, a band for fixing the right and left side-arms202and203to the head of the user may be added to the right and left side-arms202and203. The band may be an elastic band for fixing the right and left side-arms202and203to the head of the user regardless of the size of the head of the user.

Anti-slip members may be attached to inner portions of the right and left side-arms202and203contacting portions of the head of the user behind the earflaps to prevent the right and left side-arms202and203from slipping.

The lens frame201may be configured such that the lens204is attached thereto and detached therefrom. The lens204shown inFIG. 2is integrated with a nosepiece204′. However, the present invention is not limited thereto. For example, if the lens frame201is integrated with a nosepiece204′, the lens204may include a right lens404and a left lens405as shown inFIG. 4.

The lens204may be formed of a transparent material that allows a user to see therethrough. The lens204may be formed of a material that transmits a light reflected by the display212. Materials for forming the lens204may include a plastic, such as a polycarbonate, and a glass, but is not limited thereto. The lens204may include at least one of an anti-reflection and anti-glare coating, an anti-fog coating, and an UltraViolet (UV) ray block coating.

The on-board computer205may be connected to the HMD device100by a wire or wirelessly. The on-board computer205is located at a portion of the left side-arm203near the lens frame201. However, the present invention is not limited thereto. For example, the on-board computer205may be arranged at another portion of the HMD device100, e.g., a portion of the right side-arm202, another portion of the left side-arm203, etc.

The on-board computer205may receive data from the camera206, the user input unit207, and the sensors208,209, and210, analyze the received data, and generate data to be provided to a user of the HMD device100via at least one of the optical output unit213, the display212, and the right and left speakers215and216. Data to be provided to a user may include at least one of an image, text, video, and audio. However, the present invention is not limited thereto.

The camera206may be arranged at an outer portion of the left side-arm203or may be arranged internally to the left side-arm203. The camera206may be attached to other locations of the HMD device100. For example, the camera206may be arranged at a portion of the right side-arm202or at a portion (e.g., the center) of the lens frame201. The camera206may be a small-sized camera for a smart phone or a webcam. The camera206may be arranged at a location for photographing at least a portion of the surroundings of the user of the HMD device100.

The user input unit207may include at least one of a touch pad that may be operated by fingers of a user and a button that may be push-operated by a user. However, the present invention is not limited thereto. The user input unit207may be arranged at a location of the left side-arm203or other locations of the HMD device100. For example, the user input unit207may be arranged at another portion of the left side-arm203, a portion of the right side-arm202, or a portion of the lens frame201.

The user input unit207may be used by a user to input a command. The user input unit207may include an “ON/OFF” switch for turning on/off the HMD device100.

The sensor208may be a biological information detecting sensor for detecting biological information of a user. The sensor208is arranged to protrude from a lower-right portion of the lens frame201. However, attachment of the sensor208is not limited thereto. For example, the sensor208may be arranged internally to the lens frame201or may be arranged on the lens frame201. The sensor208may be arranged at an arbitrary portion of the HMD device100.

If the sensor208consists of an image sensor based camera, the sensor208may detect a bloodshot eye, eye dryness, eye blinking rate, pupil locations, and a change of the pupils of the eyes of the user. However, the present invention is not limited thereto.

If the sensor208is at least one of an oxygen sensor, an air sensor, and a fine dust sensor, the sensor208may detect eye dryness of the user. However, the present invention is not limited thereto.

The sensor209may be a surrounding environment sensor for detecting the surrounding environment of the HMD device100. The sensor209is arranged at the inner center portion of the lens frame201. However, the present invention is not limited thereto. For example, the sensor209may be arranged at at least one of an inner portion of the lens frame201other than the inner center portion of the lens frame201, an outer portion of the lens frame201, an inner portion of the right and left side-arms202and203, and an outer portion of the right and left side-arms202and203. However, locations and structures for attaching the sensor209are not limited thereto. The sensor209may be a sensor for detecting environmental conditions, such as temperature, light condition, humidity, wind speed, altitude, and atmospheric pressure in the vicinity of the HMD device100. However, the detectable environmental conditions are not limited thereto.

If the sensor208is configured as at least one of an oxygen sensor, an air sensor, and a fine dust sensor as described above, information regarding the surrounding environment of the HMD device100may be based on signals detected by the sensor208. In this case, the HMD device100may exclude the sensor209.

The sensor210may be a sensor for detecting a state of wearing the HMD device100. The sensor210is arranged at an inner portion of the right side-arm202that contacts a surface of the head of a user behind the ear. However, a location of the sensor210is not limited thereto. For example, the sensor210may be arranged at an inner portion of the left side-arm203that contacts a surface of the head of a user behind the ear or inside the power supply unit217that contacts a surface of the head of a user behind the ear.

The sensor210may be a pressure sensor. In this case, the sensor210may detect a pressure and magnitude thereof and transmit signals regarding the pressure to the on-board computer205. Therefore, based on detected values received from the sensor210, the on-board computer205may detect a period of time when a user wears the HMD device100or a period of time when the user executed applications while wearing the HMD device100.

The haptic module211is arranged at an inner portion in the vicinity of the end of the left side-arm203of the HMD device100. However, the location of the haptic module211is not limited thereto. For example, the haptic module211may be arranged at an inner portion of the right side-arm202or at another location of the HMD device100.

The haptic module211may be formed of a material capable of generating various tactile sensations that may be felt by a user. For example, the haptic module211may be formed of a material capable of generating various tactile sensations, such as a material capable of generating a vibration, a stimulation of contacting a skin surface, a stimulation based on jetting or sucking air, a stimulation based on contact of electrodes, a stimulation using electrostatic force, cool/warm sensations based on a heat-absorbing device or a heat-emitting device, etc. However, the present invention is not limited thereto.

The display212is connected to the left side-arm203of the HMD device100and located at the upper-left portion of the lens204. However, the location of the display212is not limited thereto. For example, the display212may be connected to the right side-arm202to be located at the right center of the lens204or may be connected to the left side-arm203to be located at the left center of the lens204. The display212may be formed of a semi-transparent optical waveguide (e.g., a prism). However, the inventive concept is not limited thereto. Therefore, the display212may reflect light output by the optical output unit213and focus an image at the fovea of the retina of an eye of a user of the HMD device100.

The optical output unit213may be configured as a mini-projector. However, the present invention is not limited thereto.

The microphone214may receive a voice signal of a user of the HMD device100and other sounds. The microphone214may be configured to be selectively set to a close-distance mode or a far-distance mode. For example, if the microphone214is set to the close-distance mode, the microphone214may be used to receive sounds regarding biological information (e.g., information regarding the blinking of eyes) of a user of the HMD device100. If the microphone214is set to the far-distance mode, the microphone214may be used to receive the voice of a user of the HMD device100. The close-distance mode of the microphone214may be a mode for receiving sounds generated internally to the HMD device100, whereas the far-distance mode of the microphone214may be a mode for receiving sounds generated externally to the HMD device100. However, operation ranges of the close-distance mode and the far-distance mode may be set differently.

The right and left speakers215and216may be configured as earphones to be mounted on the ears of a user of the HMD device100. The speaker215is located at an inner portion of the right side-arm202to be mounted on the right ear of the user. The left speaker216is located at an inner portion of the left side-arm203to be mounted on the left ear of the user.

The right and left speakers215and216may be fixed to the HMD device100as shown inFIG. 2. However, attachment of the right and left speakers215and216is not limited thereto. For example, the right and left speakers215and216may be detachably attached to the HMD device100, and thus a user of the HMD device100may selectively insert the right and left speakers215and216into the ears of the user.

If the right and left speakers215and216are detachable from the HMD device100, the right and left speakers215and216may be arranged at inner portions of the right and left side-arms202and203that contact portions of the head of the user behind the ears. Therefore, the user may detach the right and left speakers215and216from the respective right and left side-arms202and203and insert the right and left speakers215and216into the right and left ears, respectively, of the user. In the case of detaching the right and left speakers215and216from the respective right and left side-arms202and203and inserting the right and left speakers215and216into the right and left ears, respectively, of a user as described above, connecting wires may be arranged between the right and left speakers215and216and the HMD device100.

Connecting wires between the right and left speakers215and216and the HMD device100may be arranged in the HMD device100when the right and left speakers215and216are attached to the HMD device100. However, the present invention is not limited thereto. The right and left speakers215and216and the HMD device100may be wirelessly connected to each other based on a close-distance communication.

Connecting members based on elastic members (e.g., springs) may be arranged between the right and left speakers215and216and the HMD device100. In this case, the connecting members based on elastic members may be deformed based on locations of the ears of a user. For example, the connecting members may be deformed to be able to tilt the right and left speakers215and216in at least one of a leftward direction, a rightward direction, an upward direction, and a downward direction. If there are connecting members based on elastic members between the right and left speakers215and216and the HMD device100, the right and left speakers215and216and the HMD device100may be configured to communication with each other over wires or wirelessly. According to an embodiment of the present invention, the right and left speakers215and216may be configured as bone conduction speakers.

The power supply unit217is arranged at an end portion of the right side-arm202of the HMD device100. However, the location of the power supply unit217is not limited thereto. For example, the power supply unit217may be arranged at an end portion of the left side-arm203of the HMD device100.

The power supply unit217may be detachably attached to the right side-arm202of the HMD device100. The power supply unit217may include an external power connector that may be connected to an external power supply device to charge the HMD device100after being connected to the right side-arm202or detached from the right side-arm202. The power supply unit217may be integrated with the right side-arm202of the HMD device100. Even when the power supply unit217is integrated with the right side-arm202of the HMD device100, the power supply unit217may include an external power connector that may be connected to an external power supply device to charge the HMD device100.

FIG. 3illustrates an HMD device300according to an embodiment of the present invention. The HMD device300shown inFIG. 3includes a mono-eye lens303that may be embodied via a glasses-type display device as shown inFIG. 2or as a mono-eye display device.

Referring toFIG. 3, the HMD device300according to an embodiment of the present invention is a monocle-like device that is fixed to the face of a user via one of the ears and one side of the nose of the user. The HMD device300may be attached to a helmet structure.

The HMD device300may include a lens frame301, a side-arm302, a mono-eye lens303, an on-board computer304, a camera305, a user input unit306, sensors307,308, and309, a haptic module310, a display311, an optical output unit312, a microphone313, a speaker314, and a power supply unit315.

Components included in the HMD device300are not limited to those shown inFIG. 3. For example, the HMD device300may exclude some of the sensors307,308, and309and the haptic module310. For example, the HMD device300may include only the sensor309from among the sensors307,308, and309or may include more sensors at locations internally or externally to the HMD device300than those shown inFIG. 3. Locations of the sensors307,308, and309attached to the HMD device300may vary.

The lens frame301may be configured to be able to attach and detach from the HMD device300as described above with reference toFIG. 1. Furthermore, as described above with reference toFIG. 2, the HMD device300may include a connecting member between the lens frame301and the side-arm302. Furthermore, as described above with reference toFIG. 2, the HMD device300may include a connecting member at the center of the side-arm302.

The mono-eye lens303, the on-board computer304, the camera305, the user input unit306, the sensors307,308, and309, the haptic module310, the display311, the optical output unit312, the microphone313, the speaker314, and the power supply unit315included in the HMD device300may be located and configured similarly as the lens204, the on-board computer205, the camera206, the user input unit207, the sensors208,209, and210, the haptic module211, the display212, the optical output unit213, the microphone214, the speaker216, and the power supply unit217(when located in the left side-arm203) shown inFIG. 2.

The HMD device100according to an embodiment of the present invention may be configured like an HMD device400shown inFIG. 4. The HMD device400shown inFIG. 4is configured as a binocular display device. The HMD device400may be attached to a helmet structure or may have a glasses-like structure.

FIG. 4illustrates an HMD device400according to an embodiment of the present invention.

Referring toFIG. 4, the HMD device400may include a lens frame401, right and left side-arms402and403, right and left lenses404and405, an on-board computer406, a camera407, a user input unit408, sensors409,410, and411, a haptic module412, right and left displays413and414, a microphone415, right and left speakers416and417, and a power supply unit418.

The lens frame401included in the HMD device400shown inFIG. 4is integrated with a nosepiece401′. The lens frame401may have any of various shapes including rectangular shape, elliptical shape, and elliptical shape. However, the shape of the lens frame401is not limited thereto. The lens frame401may be formed of a material similar to that constituting the lens frame201as shown inFIG. 2.

The right and left lenses404and405may comprise an arbitrary material suitable for displaying data including images or graphics projected by an optical output unit, which is a projector as stated above with reference toFIG. 2. The right and left lenses404and405may comprise a material capable of overlapping data including images or graphics projected by an optical output unit according to the surroundings that a user may see through the right and left lenses404and405. The right and left lenses404and405may be formed of a plastic material or a glass like the lens204described above with reference toFIG. 2. However, materials for forming the right and left lenses404and405are not limited thereto.

Although some of the right and left side-arms402and403, the on-board computer406, the camera407, the user input unit408, the sensors409,410, and411, the haptic module412, the microphone415, the right and left speakers416and417, and the power supply unit418shown inFIG. 4are arranged at different locations from the right and left side-arms202and203, the on-board computer205, the camera206, the user input unit207, the sensors208,209, and210, the haptic module211, the microphone214, the right and left speakers215and216, and the power supply unit217shown inFIG. 2, functions and configurations of the above-stated components ofFIG. 4may be similar to those of the components shown inFIG. 4. However, the present invention is not limited thereto.

The right and left displays413and414shown InFIG. 4may display data including images or graphics projected by the optical output units that may be attached to the inner surfaces of the respective right and left side-arms402and403.

The HMD device400may be transformed to operate the right and left lenses404and405as display units. In this case, the right and left lenses404and405may be configured as transparent display units or semi-transparent display units. If the right and left lenses404and405are configured as semi-transparent display units, the right and left lenses404and405may include at least one optical waveguide (e.g., a prism), an electroluminescent display unit, or a Liquid Crystal Display (LCD) unit. However, the present invention is not limited thereto.

FIG. 5is a block diagram of an HMD device500according to an embodiment of the present invention.

Referring toFIG. 5, the HMD device500according to an embodiment of the present invention may include an input/output unit510, a memory520, a sensor unit525, a battery530, a power management unit535, a communication unit540, a touch sensor550, a camera560, a display unit570, a lens580, and a control unit590.

The input/output unit510includes units for receiving an input from a user, providing information to the user, receiving data externally, and outputting data externally. The input/output unit510may include at least one speaker511, at least one microphone512, at least one of a button, a connector, and a keypad, or a combination thereof. However, the present invention is not limited thereto.

The speaker511may output sounds corresponding to various data from the HMD device500under the control of the control unit590. Furthermore, the speaker511may output sounds corresponding to functions performed by the HMD device500. The speaker511may be arranged at a suitable location of the HMD device500or a plurality of speakers511may be arranged at suitable locations of the HMD device500. According to an embodiment of the present invention, the speaker511may be configured as an earphone as shown inFIG. 2.

The microphone512may receive a voice or a sound external to the HMD device500, generate an electrical signal based on the received voice or the received sound, and output the generated electrical signal to the control unit590. The microphone512may be arranged at a suitable location of the HMD device500or a plurality of microphones512may be arranged at suitable locations of the HMD device500. Throughout the present disclosure, a signal may also be referred to as data and data may also be referred to as a data signal.

The sensor unit525may include at least one sensor for detecting a state of the HMD device500or a state of the surrounding environment. For example, the sensor unit525may include at least one of a proximity sensor for detecting a user that approaches the HMD device500, a motion/direction sensor for detecting motion of the HMD device500(e.g., rotation, acceleration, deceleration, vibration, etc.), an illumination sensor for detecting ambient illumination, a photosensitive sensor for detecting color or a spectrum of light, or a combination thereof. Furthermore, the motion/direction sensor may include at least one of a gravitational sensor, a geomagnetic sensor, a gyro sensor, a shock sensor, a Global Positioning System (GPS) module, a compass, and an acceleration sensor. For example, a GPS module may receive radio signals from a plurality of GPS satellites orbiting the Earth and may calculate a location of the HMD device500by using the times of arrival of radio signals from the GPS satellites to the HMD device500.

The power management unit535may supply power to the HMD device500under the control of the control unit590. The power management unit535may be connected to one or more batteries530. Furthermore, the power management unit535may supply power, which is input from an external power source via a cable, to the HMD device500.

The communication unit540may be a wire communication unit, a wireless communication unit, or a wire/wireless communication unit and may transmit data from the control unit590to an external device over a wire or wirelessly via an external communication network or over the air, receive data from an electronic device over a wire or wirelessly via an external communication network or over the air, and transmit the received data to the control unit590. According to an embodiment of the present invention, the communication unit540may include at least one of a mobile communication module, a wireless Local Area Network (LAN) module, and a close-distance communication module.

The mobile communication module may perform a communication with an external device via a mobile communication network by using at least one antenna under the control of the control unit590. The mobile communication module may transmit or receive wireless signals for a voice call, a video call, a Short Message Service (SMS), or a Multimedia Message Service (MMS) to and from a mobile phone, a smart phone, a tablet Personal Computer (PC), or other communication devices having a network address, such as an Internet Protocol (IP) address, or a phone number.

The wireless LAN module may be connected to the Internet via a wireless Access Point (AP) around the AP under the control of the control unit590. The wireless LAN module may support the wireless LAN standard IEEE802.11x of the Institute of Electrical and Electronics Engineers (IEEE).

The close-distance communication module may perform a wireless close-distance communication with an external communication device under the control of the control unit590. Close-distance communication methods may be Bluetooth™, an Infrared Data Association (IrDA) standard, Wireless Fidelity (Wi-Fi) Direct communication, a Near Field Communication (NFC), or a combination thereof.

The touch sensor550may transmit a signal corresponding to at least one touch input to the control unit590. A user may contact the touch sensor550by using a body part (e.g., a finger) or another touch input unit, and the touch sensor550may receive a touch input by detecting the contact of the user. Furthermore, the touch sensor550may receive an input based on successive touches at a touch location (e.g., a drag input). Touch input information may include touch coordinates and/or a contact state. The contact state may be a state wherein a body part contacts the touch sensor550, a state wherein a body part is separated from the touch sensor550, or a drag state wherein a body part slides on the touch sensor550while the body part contacts the touch sensor550. The control unit590may determine user input information, e.g., selecting or moving a menu item or an item, a handwriting input, etc., based on a touch input. The control unit590may perform a function (e.g., establishing a phone call, photographing, selecting a menu, composing/reading a message, transmitting data, etc.) corresponding to a touch input.

However, in the present disclosure, a touch input is not limited to an input based on a contact between the touch sensor550and a touch input device. A touch input may include a contactless input (e.g., the touch sensor550and a touch input device are separated from each other). Such a contactless input may also be referred to as a hovering input.

The touch sensor550may be embodied as a resistive type, a capacitive type, an infrared type, an acoustic wave type, an ElectroMagnetic Resonance (EMR) type, or a combination thereof.

The camera560may include a lens system and an image sensor and may further include a flash. The camera560may transform light input (or picked up) via the lens system into electrical image signals and output the transformed image signals to the control unit590. A user may acquire moving pictures or still images by using the camera560. Furthermore, the camera560may also be used to receive an input based on a motion or a gesture of a user.

The lens system may form an image of an object by converging light received externally. The lens system includes at least one or more lenses, where the each of the lenses may be a convex lens, an aspheric lens, etc. The lens system is symmetrical with respect to an optical axis extending from the center of the lens system, wherein the optical axis may be defined as the center axis of the lens system. An image sensor may transform an optical image formed based on an external light input via the lens system into electrical image signals. The image sensor may include a plurality of pixel units arranged in a M×N matrix shape. Each of the pixel units may include a photodiode, and a plurality of transistors. The pixel units accumulate charge generated by an input of light, and a voltage based on the accumulated charge may indicate an intensity of brightness of the input of light. In the case of processing a still image or an image constituting moving pictures, an image signal output by an image sensor consists of a set of voltages (that is, pixel values) output by pixel units, and an image signal may indicate one frame (that is, a still image). Furthermore, a frame may include M×N pixels. An image sensor may be a Charge-Coupled Device (CCD) image sensor or a Complementary Metal-Oxide Semiconductor (CMOS) image sensor.

According to a control signal received from the control unit590, an image sensor may operate all pixels of the image sensor or only pixels in an area of interest. Image data output by the pixels may be output to the control unit590.

The control unit590may process an image input from the camera560, an image stored in the memory520, or an image formed by the control unit590using data stored in the memory520frame by frame. The control unit590may output frames of an image externally via the display unit570or store frames of an image in the memory520. In the present disclosure, images output by the display unit570may be visual contents including moving pictures or still images or a Graphical User Interface (GUI).

The display unit570may be embodied in any of various forms according to an embodiment of the present invention. For example, the display unit570may display an image by forming an image on the retina of a user according to various optical system principles. Alternatively, the display unit570may include a transparent display unit and display an image on the transparent display unit.

The lens580according to an embodiment of the present invention may be configured to be attachable to and detachable from the HMD device500as shown inFIG. 1. In other words, the lens580may not be included in the HMD device500, and only a structure for mounting the lens580may be included in the HMD device500. A user may change the color of the lens580of the HMD device500by switching the lens580to one with a different color. Alternatively, the lens580according to an embodiment of the present invention may be a smart window of which color and transmissivity may be modified under the control of the control unit590.

FIG. 6is a diagram of a method of an HMD device600to display an image according to an embodiment of the present invention. The HMD device600according to an embodiment of the present invention may include the display unit570including an optical system. The display unit570according to an embodiment of the present invention may display an image by forming an image on an eye610of a user1by irradiating light via the lens580.

FIG. 7is a diagram of a portion of an HMD device500according to an embodiment of the present invention.

Referring toFIG. 7, the HMD device500may include a frame720for attaching the lens580. According to an embodiment of the present invention, the frame720may have a structure in which the frame720is connected via a hinge722. An opening711, which is a path for outputting a projected light710output by a projector included in a display unit included in the frame720externally from the frame720, may be arranged on the inner surface of the frame720. In this case, a glass member or a transparent plastic member for preventing introduction of dust into the frame720may be arranged at the opening711.

Furthermore, at least one speaker may be arranged on the inner surface of the frame720.

A projector outputs the projected light710for forming a virtual image. The projected light710output by the projector is focused and reflected by the lens580, and a focused and reflected projected light730may form a virtual image on the retina of the eye610of a user. In this case, focusing may refer to the collimation of light. The focusing may include convergence of light to a point or reduction of beam spots of light. The reflected projected light730may converge onto the lens or the pupil of the eye610. An HMD device500may use a plurality of projectors for forming virtual images on both eyes of a user, respectively.

FIG. 8is a block diagram of a display unit according to an embodiment of the present invention. In detail,FIG. 8is a block diagram showing the structure of a projector in a case where the display unit includes the projector.

Referring toFIG. 8, if the display unit570includes a projector, the display unit570includes a light source810, an illumination optical system820for illuminating a display device840by using light output by the light source810, a mirror830for reflecting light transmitted through the illumination optical system820, the display device840for forming a virtual image by reflecting light reflected by the mirror830pixel by pixel, and a projection optical system850for projecting light reflected by the display device840externally.

The illumination optical system820may be arranged along a first optical axis571parallel to an x-axis. The illumination optical system820may include at least one collimating lens, at least one filter, at least one equalization lens, a focusing lens, or a combination thereof.

Optical elements (e.g., a lens, a prism, a filter, etc.) of the illumination optical system820may be aligned along the first optical axis571. Generally, an optical axis may refer to an axis along which no optical change occurs in an optical system when the optical system is rotated around it. When optical elements are aligned along an optical axis the curvature centers of the optical elements constituting the optical system are located on, the optical axis or centers of symmetry of the optical elements are located on the optical axis.

The light source810may output light that travels along the first optical axis571. For example, at least one Light Emitting Diode (LED) that outputs white light, a light of a primary color (i.e., blue, red, and yellow), or light of a combination of primary colors (e.g., purple, green, orange, etc.) may be used as the light source810.

The illumination optical system820may collimate, filter, and/or focus light input from the light source810and output processed light to the mirror830.

The mirror830may reflect light input via the illumination optical system820toward the display device840. The mirror830may have a structure in which a highly-reflective dielectric layer or a highly-reflective metal layer is deposited on a substrate.

The display device840may display an image pixel by pixel based on data input by the control unit590. The display device840may include pixel elements in correspondence to a pre-set resolution. The display device840displays an image by turning the pixel elements “ON” and “OFF.” For example, a Digital Micro-mirror Device (DMD) including micro-mirrors arranged in an M×N matrix shape may be used as the display device840. Each micro-mirror may be rotated to a position corresponding to the “ON” state and a position corresponding to the “OFF” state according to a driving signal. Each micro-mirror may reflect an incident light at an angle that is displayed externally of the display device840in the “ON” state and may reflect an incident light at an angle that is not displayed externally of the display device840in the “OFF” state.

The projection optical system850may be arranged along a second optical axis572that is parallel to a y-axis. The projection optical system850may include a relay lens860and a projection lens870. The relay lens860and the projection lens870may be aligned on the second optical axis572.

The relay lens860may align light reflected by the mirror830toward the display device840in consideration of light overfilling. In other words, the relay lens860may allow a light reflected by the mirror830to be incident on an area larger than an area occupied by pixel elements of the display device840.

Furthermore, the relay lens860may receive light reflected by the display device840, reduce a beam spot size of the light, and output the processed light. Since light reflected by the display device840has a large beam spot size, the light may not be transmitted to the projection lens870due to light loss. The relay lens860focuses light reflected by the display device840and reduces beam spot size thereof, and thus more light may be transmitted to the projection lens870.

The projection lens870may receive light transmitted through the relay lens860, collimate or focus the received light, and project the processed light externally.

FIG. 9is a diagram of a lens capable of changing a color or a transmissivity according to an embodiment of the present invention.

Referring toFIG. 9, the lens580according to an embodiment of the present invention may change a color or a transmissivity thereof. The lens580according to an embodiment of the present invention may include glass581capable of changing transmissivity thereof (that is, color or transmissivity of the glass581is modified based on a signal applied thereto) and a Holographic Optical Element (HOE)582that functions as a concave mirror.

The lens580may transmit an ambient light911input externally therethrough and may reflect and focus a light921output by the display unit570.

A transmitted ambient light911and the light921reflected by the lens580are input to the eye610of a user, and a surrounding environment image910based on a transmitted ambient light912and a displayed image920(based on a reflected light) may be overlappingly formed on the retina613of the eye610. In other words, the user sees an image formed as the surrounding environment image910and the displayed image920overlap each other. A user may recognize as if the displayed image920is floating on a transparent layer on the surrounding environment image910. The displayed image920may include a still GUI including menus and icons corresponding to a phone call, contact information, a message, a main menu, or may include a content image including a still image or a moving picture. Although an area in which the displayed image920is displayed is shown as a non-transparent area inFIG. 9, the area in which the displayed image920is displayed may have a transparency to allow the user to see the surrounding environment image910in the area in which the displayed image920is displayed. In other words, a transparency of the area in which the displayed image920is displayed may vary according to embodiments of the present invention.

The light921output by the display unit570may be a parallel light having a wavelength λ (that is, a collimated light). The light921may be incident to the HOE582to form an angle θ with a normal of the HOE582. The HOE582may be an element with wavelength selectivity. The HOE582may reflect and focus a light of the wavelength λ (that is, the light921according to the present invention) and may transmit light of a wavelength different from the wavelength λ (that is, the ambient light911according to the present invention) therethrough without converging the same.

The HOE582may reflect and focus the input light921. Furthermore, the reflected light922from the HOE582may converge on a location corresponding to the eye610, which is a certain distance (that is, an eye relief) apart from the HOE582. The reflected light922may converge on a location corresponding to a pupil611or an eye lens612of the eye610. In this case, the converged light922may have a converging angle or a viewing angle ϕ. The eye lens612controls focus of a light incident on the eye610, and since the transmitted ambient light912is converged at a location corresponding to the pupil611or the eye lens612, the transmitted ambient light912may be projected onto a retina613, without being converged by the eye lens612, and form the image920.

Although the light921incident to the HOE582is described above as a parallel light, the present invention is not limited thereto. If the reflected light922from the HOE582converges on a location corresponding to the pupil611or the eye lens612, the light921may not be a parallel light, but a diverging light or a converging light.

If a user wearing an HMD device squints his/her eyes to see actual objects in his surroundings, an image displayed by the HMD device (a virtual object) may not be clearly formed on the retina of the user. In this case, the user is unable to clearly see an actual object and a virtual object at the same time, and thus, augmented reality may not be implemented. Furthermore, a focus inconsistency between an actual object and a displayed image may increase the fatigue of the user due to an inconsistency between the convergence of the eyes and a focus adjustment of a single eye.

An HMD device according to an embodiment of the present invention may converge a projected light for forming a virtual image on a pupil, an eye lens, or a location close to the same, thereby reducing a focus change of the projected light based on the eye lens. If a projected light may be directly projected to a retina regardless of an action of an eye lens, an image projected to a retina may be recognized by a user as a clear image regardless of a focus adjustment and aberration of an eye lens.

The HOE582may have a focal distance corresponding to a distance between the pupil611or the eye lens612and the HOE582.

FIG. 10is a diagram of the glass581included in a lens of an HMD device according to an embodiment of the present invention. Transmissivity or color of the glass581according to an embodiment of the present invention may be modified based on a signal or a voltage applied thereto by the control unit590.

An electrochromic glass, a Suspended Particle Device (SPD, or a Liquid Crystal (LC) may be used as the glass581. Alternatively, if necessary, a photochromic glass or a thermochromic glass, of which transmissivity is passively modified based on a wavelength of light or a change of temperature instead of being actively controlled via application of a signal, may be used as the glass581.

The glass581may be fabricated using various methods. For example, the glass581may be fabricated by applying a material of which transmissivity may be controlled onto a glass or attaching a thin film of which transmissivity may be controlled to a glass.

A case in which an electrochromic glass is used as the glass581will be described below with reference toFIG. 10.

The glass581includes first and second insulating substrates1010and1015, a first conductive electrode1020deposited on the top surface of the first insulating substrate1010, a second conductive electrode1025deposited on the bottom surface of the second insulating substrate1015, an insulating spacer1030, which separates the first and second insulating substrates1010and1015from each other and seals a space therebetween, and a electrochromic layer1040and an electrolyte1050, which fills a space between the first and second insulating substrates1010and1015.

The first and second insulating substrates1010and1015may be formed of a transparent glass or a transparent plastic. For example, the transparent plastic may include at least one of polyacrylate, polyethylene ether phthalate, polyethylene naphthalate, polycarbonate, polyarylate, polyetherimide, polyethersulfone, and polyimide.

The first conductive electrode1020may be formed of a transparent conductor. For example, the first conductive electrode1020may contain an inorganic conductive material, such as Indium Tin Oxide (ITO), Fluorine-doped Tin Oxide (FTO), or an Antimony-doped Tin Oxide (ATO), or an organic conductive material, such as polyacetylene or polythiophene.

The second conductive electrode1025may be formed of a transparent or non-transparent conductive material. For example, the second electrode1025may contain ITO, FTO, a metal like aluminum (Al), ATO, or a combination thereof.

The electrochromic layer1040containing an electrochromic material may be arranged on the first conductive electrode1020. The electrochromic layer1040may be arranged on the first electrode1020as a film, but is not limited thereto.

The first insulating substrate1010and the second insulating substrate1015may be fixed to the insulating spacer1030. A space between the first insulating substrate1010and the second insulating substrate1015may be filled with the electrolyte1050. The electrolyte1050may include an oxidizing/reducing material that reacts with the electrochromic material. The electrolyte1050may be a liquid electrolyte or a solid electrolyte. A solution formed by a lithium salt, such as lithium hydroxide (LiOH) or lithium perchlorate LiClO4, a potassium salt, such as potassium hydroxide (KOH), or a sodium salt, such as sodium hydroxide (NaOH), may be used as a liquid electrolyte. Meanwhile, (poly(2-acrylamido-2-methylpropane sulfonic acid) or Poly(ethylene) Oxide (PEO) may be used as a solid electrolyte, for example.

A material constituting the electrochromic layer1040, that is, an electrochromic material, may contain a metal-organic composite, which is a combination of a metal and an organic compound including a functional group capable of forming coordination with the metal. The metal may include a light metal, a transition metal, a lanthanide metal, an alkali metal, or a combination thereof. For example, the metal may include beryllium (Be), barium (Ba), copper (Cu), zinc (Zn), cerium (Ce), magnesium (Mg), aluminum (Al), titanium (Ti), or a combination thereof. Furthermore, for example, the functional group may include a carboxyl group, a pyridine group, an imidazole group, or a combination thereof. Furthermore, for an example, the organic compound may include a viologen derivative, an anthraquinone derivative, or a combination thereof.

FIGS. 11A to 11Care diagrams of a method of an HMD device to display an image according to an embodiment of the present invention.

Referring toFIG. 11A, the display unit570may be arranged between the lens580and an eye of a user. The display unit570may include a transparent display capable of transmitting light therethrough.

Light1110-1of a surrounding environment may be transmitted through the lens580and the display unit570and be incident to an eye of a user, and a light1120-1generated by the display unit570may be directly incident to an eye of a user without being transmitted through the lens580.

Alternatively, with reference toFIG. 11B, the lens580may be arranged between the display unit570and an eye of a user. In this case, light1110-2of a surrounding environment may be transmitted through the lens580and the display unit570and incident to the eye of the user. A light1120-2generated by the display unit570may be transmitted through the lens580and incident to an eye of a user. In this case, since both light1110-2of the surrounding environment and the light1120-2generated by the display unit570are transmitted through the lens580and incident to an eye of a user, an HMD device may modify an image in consideration of colors recognized by the user from the light1120-2generated by the display unit570and transmitted through the lens580.

Alternatively, with reference toFIG. 11C, the display unit570and the lens580may be integrated with each other as a display unit/lens570-1. In this case, a color or transmissivity of the display unit/lens570-1may be controlled and the display unit/lens570-1may be configured as a smart window capable of displaying an image. Light1110-3of a surrounding environment may be transmitted through the display unit/lens570-1and incident to an eye of a user. Furthermore, a light1120-3generated by the display unit/lens570-1may be directly incident to an eye of a user.

FIG. 12is a flowchart of a method of an HMD device to display an image according to an embodiment of the present invention.

Referring toFIG. 12, the HMD device may obtain color information regarding a lens in step S1210. The color information regarding the lens may refer to identification information for the HMD device to identify the color of the lens. For example, the color information regarding the lens may include Red-Green-Blue (RGB) values indicating a color of the lens. The color information may vary according to an embodiment of the present invention. For example, the color information may be information instructing a color of a lens. Alternatively, according to an embodiment of the present invention, the color information may instruct a color that is recognized as a color of a background around an image displayed by an HMD device. In this case, color of a light transmitted through a lens is detected by a photosensitive sensor arranged at the HMD device, and thus the HMD device may obtain color information instructing a color recognized as a background color. In this case, the color information instructing a color recognized as a background color may also be referred to as color information regarding an image display area.

An HMD device may obtain color information regarding a lens using various methods. For example, if the HMD device is lens-interchangeable, the HMD device may obtain the color of a replaced lens when a lens is interchanged. Furthermore, according to an embodiment of the present invention, if a lens is a smart window of which transmissivity or color may be modified under the control of a control unit of the HMD device, the HMD device may control the color of the lens, and thus a current color of the lens may be obtained.

Next, the HMD device may modify an image based on color information obtained in step S1210and display the modified image via a display unit in step S1220. For example, when the HMD device displays an image including text and the obtained color information regarding a lens indicates yellow, the HMD device may modify the color of the text to blue, which is the complementary color of yellow, and display the modified text in blue via a display unit.

If a lens may be attached to and detached from an HMD device or the HMD device is capable of controlling the color of a lens, the HMD device may obtain color information regarding the lens in step S1210. However, if a lens is integrated with an HMD device and the color of the lens is unchangeable, color information regarding the lens may be pre-set. In other words, if a fixed-color lens is integrated with an HMD device, the HMD device may correct the color of an image based on the pre-set color information regarding the lens. For example, if a blue lens is fixed to an HMD device, the HMD device may modify the color of an image to red, which is the complementary color of blue, without obtaining color information regarding the lens and display the modified image.

FIG. 13is a block diagram for modifying and displaying an image according to an embodiment of the present invention.

Referring toFIG. 13, an HMD device according to an embodiment of the present invention may include a color correcting module1330. The color correcting module1330may display a modified image1340generated by modifying the color of an original image1320based on color information1310regarding a lens and the original image1320. In this case, the color correcting module1330according to an embodiment of the present invention may use the control unit590. For example, the color correcting module1330may include a processor loaded with a program for generating the modified image1340.

The color correcting module1330may modify the original image1320using various methods according to an embodiment of the present invention. For example, if the original image1320includes text, the color correcting module1330may modify the original image1320to display the text in a complementary color of the color of a lens.

However, modification of an image by the color correcting module1330is not limited to modification of the color of the original image1320. For example, the color correcting module1330may increase or decrease the size of an object (e.g., a GUI object like an icon) included in the original image1320. As another example, the color correcting module1330may change the shape of an object (e.g., a GUI object like an icon) included in the original image1320.

If a lens is integrated with an HMD device and it is unable to change the color of the lens, the color information1310may not be separately input to the color correcting module1330but may be pre-set. The color correcting module1330may modify an image based on pre-set color information. In other words, if a lens with a fixed color is integrated with an HMD device, the color correcting module1330may modify an image based on the pre-set color information regarding the lens. For example, if a blue lens is fixed to an HMD device, the color correcting module1330may modify the color of an image to red, which is the complementary color of blue, without obtaining color information regarding the lens and display the modified image.

The display unit570may display the modified image1340output by the color correcting module1330.

An HMD device may obtain color information regarding a lens using various methods according to an embodiment of the present invention. For example, an HMD device may obtain color information regarding a lens by using a metal line included in the lens, a code printed on the lens, a photosensitive sensor, or an image picked up by a camera. In this case, obtaining color information regarding a lens may include a case in which color information regarding the lens is input by a user or a case in which, as an HMD device controls color or transmissivity of the lens, the HMD device is capable of recognizing color information regarding the lens.

FIG. 14is a flowchart of a method of obtaining color information regarding a lens according to an embodiment of the present invention.

Referring toFIG. 14, an HMD device may obtain identification information regarding a lens in step S1410. In this case, identification information regarding a lens refers to information for the HMD device to identify the lens. For example, identification information regarding a lens may be a code allocated based on the type of the lens. Methods for the HMD device to obtain identification information regarding a lens may vary according to an embodiment of the present invention. For example, if a metal line is arranged at a lens, identification information regarding the lens may be obtained based on a location of the metal line. As another example, if a code (e.g., a barcode or a Quick Response (QR) code) is printed on a surface of a lens, an HMD device may obtain identification information regarding the lens from the code printed on the lens.

In this case, obtained identification information regarding a lens may be information allocated based on the color of the lens. For example, identification information regarding a black lens may be “01,” and identification information regarding a transparent (e.g. colorless) lens may be “00.” In this case, the HMD device may obtain color information regarding a lens corresponding to the obtained identification information.

Next, the HMD device may modify an image to be displayed based on the obtained color information. The HMD device may modify the image using various methods according to an embodiment of the present invention. For example, the HMD device may modify the image, such that a text included in the image is displayed in a complementary color of the color of the lens.

However, modification of an image by the HMD device is not limited to modification of the color of the image. For example, the HMD device may increase or decrease the size of an object (e.g., a GUI object like an icon) included in the image. For another example, the HMD device may change the shape of an object (e.g., a GUI object like an icon) included in the image.

Next, the HMD device may display the modified image in step S1440.

FIG. 15is a diagram of an HMD device1500for identifying a lens using a metal line according to an embodiment of the present invention.

Referring toFIG. 15, the HMD device1500according to an embodiment of the present invention may include a display unit1511and an electrode1532arranged at a portion of a frame1510. A lens1520may be attached to and detached from the HMD device1500according to an embodiment of the present invention. In this case, the detachable lens1520may include a metal line1531. In this case, the electrode1532and the metal line1531may be arranged at locations at which the electrode1532and the metal line1531may contact each other when the lens1520is attached to the frame1510. When the lens1520is attached to the frame1510, the HMD device1500may detect the location of the metal line1531by using the electrode1532. The HMD device1500may obtain identification information regarding the lens1520. According to an embodiment of the present invention, a magnet1540may be arranged at a portion of the frame1510so that the lens1520may be fixed to the frame1510.

FIG. 16is a diagram of portions of an HMD device1500, which includes a metal line, an electrode, and a lens according to an embodiment of the present invention.

Referring toFIG. 16, the detachable lens1520may include the metal line1531. In this case, the metal line1531may be arranged at a certain location allocated based on the type of the lens1520. When the lens1520is attached to the frame1510, the metal line1531may contact the electrode1532.

When the metal line1531contacts the electrode1532, the HMD device1500may detect the location at which the metal line1531is arranged. The HMD device1500may obtain identification information regarding the lens1520corresponding to the location at which the metal line1531is arranged.

FIG. 17is a diagram of a method of identifying a lens by using a code printed on the lens according to an embodiment of the present invention.

Referring toFIG. 17, an HMD device1700may obtain identification information regarding a lens1720based on a code1721included in the lens1720. AlthoughFIG. 17shows that the code1721is printed on the front surface of the lens1720, location of the code1721may vary according to an embodiment of the present invention.

According to an embodiment of the present invention, the code1721may refer to a symbol provided to be optically read by the HMD device1700. For example, the code1721may be a QR code or a barcode. In this case, the HMD device1700may pick up an image of a location at which the code1721is arranged by using a camera1710and obtain identification information regarding the lens1720from the picked up image.

FIG. 18is a flowchart of a method of obtaining color information regarding a lens by using a photosensitive sensor according to an embodiment of the present invention.

Referring toFIG. 18, an HMD device according to an embodiment of the present invention may include a photosensitive sensor. A photosensitive sensor refers to a sensor capable of detecting light by responding to radiation energy like light. The HMD device may detect light transmitted through a lens via the photosensitive sensor and analyze the detected light in step S1810.

Next, the HMD device may determine color information regarding the lens based on a result of the analysis in step S1820. For example, the HMD device may analyze a spectrum of the light detected by the photosensitive sensor. If the percentage of blue light in the detected light is high, the HMD device may obtain color information corresponding to blue as the color information regarding the lens.

Once color information regarding the lens is obtained, the HMD device may modify an image based on the obtained color information in step S1830and display the modified image in step S1840.

FIGS. 19A and 19Bare diagrams showing a method of obtaining color information regarding a lens by using a photosensitive sensor according to an embodiment of the present invention.

Referring forFIGS. 19A and 19B, an HMD device according to an embodiment of the present invention may include a photosensitive sensor1930at a location where light emitted by a light source1910arranged beyond a lens1920of the HMD device may be received. For example, referring toFIG. 19B, the photosensitive sensor1930may be arranged at a portion of the frame of the HMD device. The HMD device according to an embodiment of the present invention may obtain color information regarding the lens1920as a result of analyzing light transmitted through the lens1920.

FIG. 20is a flowchart of a method of obtaining color information regarding a lens based on a size of a pupil of a user according to another embodiment of the present invention.

Referring toFIG. 20, an HMD device may pick up an image by using a camera in step S2010. In this case, the picked up image may include an image of an eye of a user.

The HMD device may perform image recognition with respect to the picked up image. As a result of the image recognition, the HMD device may obtain a value regarding a size of a pupil included in the image. Based on the obtained value regarding the size of the pupil, a change of the size of the pupil included in the image may be determined in step S2020.

Next, the HMD device may determine color information regarding a lens based on the change of the pupil in step S2030. If transmissivity of the lens is high, a large amount of light may be incident to an eye of a user, and thus the size of the pupil of the user may be reduced. On the contrary, if transmissivity of the lens is low, a small amount of light may be incident to an eye of a user, and thus the size of the pupil of the user may be increased. Based on the mechanism, the HMD device may obtain color information regarding the lens based on the size of the pupil of the user. In other words, if the size of the pupil of the user is large, the HMD device may determine that the lens has a bright color. On the contrary, if the size of the pupil of the user is small, the HMD device may determine that the lens has a dark color.

Once color information regarding the lens is obtained, the HMD device may modify an image based on the obtained color information in step S2040and display the modified image in step S2050.

FIG. 21is a diagram of a method of obtaining color information based on a size of a pupil of a user according to an embodiment of the present invention.

Referring toFIG. 21, an HMD device2100may pick up an image of an eye2120of a user by using a camera2110arranged at a portion of the HMD device2100. AlthoughFIG. 21shows that a lens is arranged between the camera2110and the user, the location of the camera2110may vary according to an embodiment of the present invention. The HMD device2100may detect the size of a pupil from the image of the eye2120. The HMD device2100may obtain color information regarding the detected size of the pupil.

FIG. 22is a diagram of a method of obtaining color information regarding a lens based on settings of a user according to an embodiment of the present invention.

According to an embodiment of the present invention, a user may set color information regarding a lens2220. Referring toFIG. 22, an HMD device2200may display a user interface2210for setting color information regarding the lens2220. The user interface2210may include a list of color information, for example.

The HMD device2200may receive an input for selecting one from among color information regarding lens2220from a user via an input/output unit. For example, referring toFIG. 22, if brown2211is selected via the user interface2210, the HMD device may obtain RGB values corresponding to brown as the color information regarding the lens2220.

FIG. 23is a flowchart of a method of an HMD device to display an image according to an embodiment of the present invention. Furthermore,FIG. 24is a diagram of a method of an HMD device to display an image according to an embodiment of the present invention.

According to an embodiment of the present invention, an HMD device may be a passive device that is unable to process an image. In this case, the HMD device may modify an image by using an active device capable of communicating with the HMD device.

Referring toFIGS. 23 and 24, an HMD device2300may obtain identification information regarding a lens in step S2310. Next, the HMD device2300may transmit the obtained identification information regarding the lens to a control terminal2400, which is an active device, in step2320.

Next, the control terminal2400may modify or generate an image to be displayed by the HMD device2300based on the identification information received from the HMD device2300in step2330.

Next, the control terminal2400may transmit the modified or generated image to the HMD device2300in step2340.

Next, the HMD device2300may display the modified or generated image in step2350.

FIG. 25is a flowchart of a method of an HMD device to recommend a lens according to an embodiment of the present invention.

Referring toFIG. 25, an HMD device may obtain information regarding an external environment in step S2510. The information regarding the external environment may be information indicating conditions in the vicinity of the HMD device detected by at least one sensor. For example, information regarding the external environment may include information regarding brightness in the vicinity of the HMD device, GPS coordinates regarding location of the HMD device, or information regarding sound detected by the HMD device.

Next, the HMD device may determine color information corresponding to the information regarding the external environment obtained in step S2510in step S2520. According to an embodiment of the present invention, the color information regarding the external environment may be raw information regarding the external environment, such as information regarding brightness in the vicinity of the HMD device. For example, the HMD device may pick up an image of the external environment of the HMD device by using the camera560. The HMD device may obtain information regarding the external environment from the picked up image. In this case, step S2520may be omitted. Furthermore, according to an embodiment of the present invention, color information regarding the external environment may be a color code selected from color information included in a color DataBase (DB) included internal or external to the HMD device based on information regarding brightness in the vicinity of the HMD device, GPS coordinates regarding the location of the HMD device, or information regarding sound detected by the HMD device. The HMD device may recommend a lens corresponding to the determined color information regarding the external environment in step S2530. The expression “the HMD device may recommend a lens” indicates that the HMD device displays a message instructing a user to change a lens, for example. For example, if brightness obtained in step S2510is low, the HMD device may recommend a colorless lens. On the contrary, if brightness obtained in step S2510is high, the HMD device may recommend a colored lens, e.g., a brown lens, a blue lens, a black lens, etc.

FIG. 27is a diagram of a method of an HMD device2700to recommend a lens.

Referring toFIG. 27, the HMD device2700may display a message2710for recommending a lens selected based on information regarding the external environment. In this case, the HMD device2700may display the message2710such that a user recognizes that the message2710is displayed on a lens2720.

According to an embodiment of the present invention, if the HMD device2700is capable of controlling the color or transmissivity of the lens2720, step S2530may be replaced with a step in which the HMD device2700controls the color or transmissivity of the lens2720based on the color information regarding the external environment determined in step S2520.

FIG. 26is a flowchart of a method of an HMD device to recommend a lens according to an embodiment of the present invention. Furthermore,FIGS. 28 through 31are diagrams of a method of an HMD device to recommend a lens according to an embodiment of the present invention.

An HMD device may select content to be displayed via a display unit in step S2610. For example, the HMD device may display a content list and select content from the content list based on a user input received via an input/output unit of the HMD device. In this case, the content list may include executable applications or multimedia contents including pictures or moving pictures.

For example, referring toFIG. 28, the HMD device2700may display a content list2810on the lens2720. The HMD device2700may select one from among the contents included in the content list2810. As shown inFIG. 28, the HMD device2700may select a web browser2811from the content list2810.

Next, the HMD device2700may recommend a lens corresponding to the content selected in step S2610in step S2620inFIG. 26. For example, referring toFIG. 29, if the web browser2811is selected, the HMD device2700may display a message2920for recommending a lens having a color suitable for using the web browser2811on a lens2820. Furthermore, the HMD device2700may display a web browser executing screen image2910.

Next, it may be determined whether a recommended lens is attached to the HMD device in step S2630ofFIG. 26. In step S2630, the HMD device may determine whether a recommended lens is attached to the HMD device by using various methods. For example, the HMD device may obtain identification information regarding a lens and determine whether the obtained identification information is identical to identification information of the recommended lens. As another example, the HMD device may obtain color information regarding a lens and determine whether the obtained color information is identical to color information of the recommended lens. As shown inFIG. 30, if the lens2820is replaced with a lens2821having a different color, the HMD device may determine whether the recommended lens is attached to the HMD device. If the recommended lens is not attached to the HMD device, the HMD device may perform step S2620ofFIG. 26again.

According to an embodiment of the present invention, if the HMD device is capable of controlling the color or transmissivity of the lens2820, the steps S2620and S2630may be replaced with a step for controlling the lens2820to change the color or transmissivity of the lens2820to a color or a transmissivity recommended for displaying the selected content.

Next, the HMD device may display the selected content on a lens in step S2640. The content displayed in step S2640may be an image modified based on color information regarding a lens modified in steps S2620and S2630. As shown inFIG. 31, the HMD device2700may display a web browser for displaying a screen image2911with a modified color on the lens2821with a modified color.

FIG. 32is a flowchart of a method of an HMD device capable of changing the color or transmissivity of a lens displaying an image according to an embodiment of the present invention.

The HMD device according to an embodiment of the present invention may control color or transmissivity of a lens in step S3110. For example, the HMD device may measure brightness by using a brightness sensor and control transmissivity of the lens. For example, the brightness sensor may include at least one of an illumination sensor for detecting ambient illumination, a photosensitive sensor for detecting color or a spectrum of light, or a combination thereof. However, the present invention is not limited thereto. As another example, the HMD device may pick up an image of surrounding environments by using a camera and measure brightness based on the picked up image. Alternatively, brightness in the vicinity of the HMD device may be measured by using information indicating the location of the HMD device. If a measured brightness is low, transmissivity of the lens may be increased. On the contrary, if the measured brightness is high, transmissivity of the lens may be reduced to protect the eyes of a user.

As a lens is formed of an electrochromic glass, a Liquid Crystal (LC) panel, or a Suspended Particle device (SPD), the HMD device may control the color or transmissivity of the lens. For example, the lens may include an electrochromic glass including an ElectroChromic material (EC) thin-film transparent electrode, an ITO glass, and an electrolyte. If the lens is formed of an electrochromic glass, the HMD device may control the color or transmissivity of the lens by controlling the magnitude of a voltage applied to the electrochromic glass.

Next, the HMD device may modify an image based on the color or transmissivity of the lens in step S3120. The HMD device may display a modified image in step S3130. For example, if the HMD device controlled a lens to change the color of the lens to blue, the HMD device may display text in red, which is the complementary color of blue.

FIG. 33is a chart of complementary colors that may be used to modify an image according to an embodiment of the present invention.

An HMD device according to an embodiment of the present invention may modify an image by using the complementary color chart. For example, if color information obtained in step S1210ofFIG. 12is FFFF00 (e.g. yellow), the HMD device may display text in blue, which is the complementary color of yellow. However, the present invention is not limited thereto, and methods of modifying an image may vary according to an embodiment of the present invention.

In this case, color information may vary according to an embodiment of the present invention. For example, according to an embodiment of the present invention, color information may be information instructing a color of a lens. Alternatively, according to an embodiment of the present invention, the color information may instruct a color that is recognized as a color of a background around an image displayed by an HMD device. In this case, the color of light transmitted through a lens is detected by a photosensitive sensor arranged at the HMD device, and thus the HMD device may obtain color information instructing a color recognized as a background color. According to an embodiment of the present invention, color information may be information determined based on information regarding the external environment.

FIGS. 34 and 35are diagrams of an HMD device displaying corrected images according to an embodiment of the present invention.

Referring toFIG. 34, an image3310is displayed when a user wearing an HMD device3300sees an indoor landscape. The HMD device3300may measure brightness of a circumference. If the HMD device3300is located at an indoor location and the measured brightness is low, the HMD device3300may increase a transmissivity of a lens3320or may control the lens3320to be colorless. The user of the HMD device3300may recognize that the image3310is displayed on the lens3320.

Referring toFIG. 35, an image3311is displayed when the user wearing the HMD device3300sees an outside landscape3401. As a measured brightness is increased, the HMD device3300may control the color or transmissivity of a lens3321. When the color or transmissivity of the lens3321is modified, the HMD device3300may modify the image3311based on color information corresponding to the modified color or transmissivity of the lens3321. As shown inFIG. 35, the HMD device3300may display the image3311with modified color or modified icon sizes.

FIGS. 36 and 37are diagrams of an HMD device displaying a modified image according to an embodiment of the present invention.

Referring toFIG. 36, an HMD device3500may display an image3510for providing an augmented reality function on a lens3520. The image3510for providing augmented reality function may include information regarding an actual object a user is seeing, for example. As shown inFIG. 37, if the color or transmissivity of the lens3520is modified, the HMD device3500may display a modified image3511. The modified image3511may not only be an image with a modified color, but also an image including modified objects as shown inFIG. 37.

FIG. 38is a block diagram of an HMD device according to an embodiment of the present invention.

The HMD device may include a color information obtaining unit3710, a control unit3720, and a display unit3730.

The color information obtaining unit3710according to an embodiment of the present invention may obtain color information regarding a lens. The color information obtaining unit3710may be configured in various ways according to an embodiment of the present invention. For example, as shown inFIG. 15, the color information obtaining unit3710may include the electrode1532. Alternatively, the color information obtaining unit3710may include a recognizing unit, which recognizes a code by using the camera1710, as shown inFIG. 17, or may obtain color information regarding a lens by using the photosensitive sensor1930, as shown inFIG. 19. Alternatively, if the control unit3720ofFIG. 38controls the color or transmissivity of a lens, the color information obtaining unit3710may be included in the control unit3720.

Furthermore, if a lens may be attached to and detached from the HMD device, the color information obtaining unit3710may determine whether the lens is attached to the HMD device. When a lens attached to the HMD device is recognized, the color information obtaining unit3710may obtain color information regarding the attached lens.

The control unit3720may process information and control components of the HMD device. The control unit3720may modify an image based on color information obtained by the color information obtaining unit3710. The control unit3720may control the display unit3730to display the modified image.

According to an embodiment of the present invention, the control unit3720may control at least one of the color and transmissivity of a lens. In this case, the lens may include an electrochromic glass, an SPD, or an LC. For example, the lens may include a glass having the structure as shown inFIG. 10. If the control unit3720ofFIG. 38controls the color or transmissivity of a lens, the control unit3720may obtain color information regarding the lens corresponding to the modified color or transmissivity. Furthermore, the control unit3720may modify an image based on the obtained color information regarding the lens.

According to an embodiment of the present invention, the control unit3720may select content that may be output via the display unit3730. For example, the HMD device may display a content list and select content from the content list based on a user input received via an input/output unit of the HMD device. In this case, the content list may include executable applications or multimedia contents including pictures or moving pictures.

Furthermore, an HMD device according to an embodiment of the present invention may further include a brightness sensor for obtaining information regarding external brightness. The control unit3720may modify an image in further consideration of information regarding external brightness obtained via the brightness sensor. For example, the control unit3720may increase brightness of an image as external brightness increases.

An HMD device according to an embodiment of the present invention may modify an image in further consideration of colors of a background a user is seeing (e.g. a background of the image).

FIG. 39is a flowchart of an HMD device4000displaying an image according to an embodiment of the present invention. Furthermore,FIGS. 40 through 43are diagrams of the HMD device4000displaying a corrected image4125according to an embodiment of the present invention.

First, the HMD device4000may pick up a background image4020in step S3910. The HMD device4000may include a camera4010for picking up an image in a viewing direction of a user1. The HMD device4000may pick up the background image4020by using the camera4010.

Referring toFIG. 40, the HMD device4000may include the camera4010at a portion of the HMD device4000. If the user1wearing the HMD device4000is seeing a crosswalk, the camera4010may obtain the background image4020in which the crosswalk is captured.

Next, the HMD device4000may determine an image displaying area4120related to a location of an image4110to be displayed by the HMD device4000in the background image4020in step S3920. When the HMD device4000displays the image4110, the image displaying area4120may be a portion of the background image4020, the portion recognized by a user as a region having displayed therein the image4110. According to an embodiment of the present invention, the image displaying area4120may be determined based on a location where the image4110is displayed.

Referring toFIG. 41, if the image4110is displayed at the upper-left portion of the HMD device4000, the HMD device4000may determine that the image displaying area4120is located at the upper-left portion of the background image4020. Referring toFIG. 42, the user1may recognize that the image4110is displayed on the image displaying area4120in the background image4020. In this case, if the colors of the background in the image displaying area4120are similar to colors of the image4110, it may be difficult for the user1to recognize the image4110. Therefore, it is necessary to modify colors of the image4110.

When the image displaying area4120is determined in step S3920inFIG. 39, the HMD device4000may determine color information regarding the image displaying area4120in step S3930. In this case, step S3930may vary according to the embodiment of the present invention. For example, the HMD device4000may determine an average color of pixels included in the image displaying area4120as color information regarding the image displaying area4120. As another example, the HMD device4000may extract color information from a feature point, such as a center point, in the image displaying area4120.

Furthermore, according to an embodiment of the present invention, the HMD device4000may determine color information in consideration of both the color of a lens and the color of the image displaying area4120. If the user1sees an object through a lens, the user1may recognize the object in a color different from an actual color of the object. Therefore, in step S3930, the HMD device4000may determine the color of an object recognized by the user1through the lens as color information regarding the image displaying area4120.

Next, the HMD device4000may modify an image based on color information regarding the image displaying area4120determined in step S3930in step S3940. The HMD device4000may display the modified image via a display unit in step S3950.

For example, the HMD device4000may determine a complementary color with respect to color information regarding the image displaying area4120and modify the color of the image4110to the determined complementary color. Referring toFIG. 43, the HMD device4000may display a modified image4125, such that the modified image4125is easily distinguished from the background image4020.

In step S3940, it is not necessary to modify an entire image. For example, only a color of the borders of an image may be modified or a separate background may be added around an image.

FIG. 44is a flowchart of a method of an HMD device controlling a color or transmissivity of a lens according to an embodiment of the present invention.

First, in step S4410, the HMD device may pick up an image by using a camera included in the HMD device. In this case, the camera may be aimed externally from the HMD device. In other words, the camera may pick up an image by receiving light that is not transmitted through a lens.

Next, in step S4420, the HMD device may determine a color or transmissivity of the lens. In this case, the lens may include a smart window of which transmissivity or color may be modified under the control of a control unit of the HMD device. For example, the HMD device may determine external brightness based on the picked up image. In other words, the HMD device may analyze brightness of the picked up image and determine brightness external to the HMD device based on a result of the analysis. If external brightness is high, the HMD device may change a color of the lens to a dark color, such as brown. On the contrary, if external brightness is low, the HMD device may control the lens to be colorless or determine a color or transmissivity of the lens for the lens to have a high transmissivity.

Next, in step S4430, the HMD device may control a color or transmissivity of the lens, such that the lens has the color or the transmissivity determined in step S4420.

FIG. 45is a diagram of a method of an HMD device picking up an image for controlling a color or transmissivity according to an embodiment of the present invention.

According to an embodiment of the present invention, the HMD device may include a plurality of cameras. Referring toFIG. 45, the HMD device may include a first camera560-1and a second camera560-2. The first camera560-1may be arranged to receive light4520transmitted externally to the HMD device but not through the lens580. The second camera560-2may be arranged to receive light4510transmitted through the lens580.

The HMD device may determine a color or transmissivity of the lens580based on the light4520received via the first camera560-1and the light4510received via the second camera560-2. For example, the HMD device may compare the light4520received via the first camera560-1to the light4510received via the second camera560-2(e.g., a brightness difference or a color difference) and determine a color or transmissivity of the lens580based on a result of the comparison. Alternatively, the HMD device may select one of the light4520received via the first camera560-1and the light4510received via the second camera560-2and determine a color or transmissivity of the lens580based on the brightness or the color of the selected light. According to an embodiment of the present invention, methods of determining a color or transmissivity of a lens may vary.

FIG. 46is a flowchart of a method of an HMD device controlling a color or transmissivity of a lens according to an embodiment of the present invention.

First, the HMD device may pick up an image via the first camera560-1in step S4610. Furthermore, the HMD device may pick up an image via the second camera560-2in step S4615.

Next, in step S4620, the HMD device may analyze the images picked up in steps S4610and S4615. For example, the HMD device may compare the image picked up via the first camera560-1to the image picked up via the second camera560-2. Alternatively, the HMD device may select one of the images picked up via the first camera560-1and the second camera560-2. However, the present invention is not limited thereto.

Next, in step S4630, the HMD device may determine a color or transmissivity of the lens. When a color or transmissivity of the lens is determined, the HMD device may control a color or transmissivity of the lens based on the determined color or transmissivity in step S4640.

As described above, according to an embodiment of the present invention, a user may easily recognize contents or user interfaces.

An embodiment of the present invention may also be implemented as a non-transitory computer readable recording medium comprising instructions, such a program module, executable by a computer. Non-transitory computer readable recording media can be any available media that can be accessed by a computer and include both volatile media (e.g., Random Access Memory (RAM)) and nonvolatile media (e.g., Read Only Memory (ROM)) and both removable and non-removable media. Furthermore, the non-transitory computer readable recording media may include both computer storage media and communication media. Computer storage media include computer readable instructions, data structures, program modules, or volatile, nonvolatile, removable, and non-removable media embodied by using any technique for storing information. Communication media typically include computer readable instructions, data structures, program modules, other data or other transport mechanisms for modulated data signals, and any information delivery media. For example, the non-transitory computer readable recording media may be ROM, RAM, flash memory, a Compact Disc (CD), a Digital Video Disc (DVD), a magnetic disk, or a magnetic tape.

It should be understood that the embodiments of the present invention described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment of the present invention should typically be considered as available for other similar features or aspects in other embodiments.