Patent ID: 12190459

DETAILED DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present disclosure are disclosed herein. However, specific structural and functional details disclosed herein are merely representative for purposes of describing exemplary embodiments of the present disclosure. Thus, exemplary embodiments of the present disclosure may be embodied in many alternate forms and should not be construed as limited to exemplary embodiments of the present disclosure set forth herein.

Accordingly, while the present disclosure is capable of various modifications and alternative forms, specific exemplary embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the present disclosure to the particular forms disclosed, but on the contrary, the present disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure. Like numbers refer to like elements throughout the description of the figures.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being “connected” or “coupled” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (i.e., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and/or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this present disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereinafter, exemplary embodiments of the present disclosure will be described in greater detail with reference to the accompanying drawings. In order to facilitate general understanding in describing the present disclosure, the same components in the drawings are denoted with the same reference signs, and repeated description thereof will be omitted.

FIG.1is an exemplary diagram for describing an image transfer process and luminance definition of a projection display that can be used in a method for providing a mixed reality combined physical space according to an embodiment of the present disclosure.FIG.2is a graph illustrating a result of measuring a change in an ambient contrast ration according to an external illuminance environment of a mixed reality device equipped with the projection display ofFIG.1.

Referring toFIG.1, a luminance LDISPLAYof a projection display10is defined with a background luminance LBby an external environment background20from ambient light, a reflection and glare luminance LRaccording to reflection or glare effect, and an emission luminance LE that transmits an image from a display device.

Unlike the environment of a portable device such as a mobile phone or a smart phone, in the image quality of the projection display10, the influence of ambient light on the deterioration of the color accuracy of an image is large.

Particularly, as shown inFIG.2, in the optical see-through system for augmented reality or mixed reality, a change in color accuracy such as ambient contrast ration by ambient light intensity is large, and in a bright external environment, the background luminance LBof the light becomes dominant, so that the visibility of the virtual image generated by the emission luminance LE is sharply reduced.

FIG.3is an exemplary diagram of a resolution of a virtual image for describing a factor that greatly affects a user's immersion feeling in the mixed reality device equipped with the projection display ofFIG.1.

Referring toFIG.3, an external scene enters the user's eyes50through a shader40and the projection display10, and the image of a micro display installed in a headset for mixed reality (image from p-display) is reflected from the projection display10and enters the user's eye50.

A first light representing the external scene may be first refracted on a first surface of the projection display10and secondarily refracted on a second surface of the projection display10, and then transmitted to the retina through the lens of the user's eye. In addition, the second light representing the image of the micro display may be reflected from the second surface of the projection display10and transmitted to the retina through the lens of the user's eye.

Of course, a portion of the first light may be reflected at the first and second surfaces of the projection display10, and a portion of the second light may be reflected at the second or first surface of the projection display10, another portion of the second light may be reflected off the lens surface of the user's eye.

The image that is finally refracted by the lens of the user's eye and formed on the retina is about 90.16% of the first light and about 3.28% of the second light, excluding the portion that is reflected and lost on a propagation path. In this case, the resolution, color gamut, or ambient contrast ration can be calculated as “90.16/3.28” and expressed as about 27.5.

FIGS.4and5are exemplary diagrams for describing a traditional environment and immersive mixed reality environment of a mixed reality combined physical space.

In the present embodiment, when a display image is output through a mixed reality headset using an optical see-through system in a traditional physical space100aas shown inFIG.4, it is possible to solve the problem of a sharp decrease in the visibility of the virtual image by effectively improving the phenomenon in which the light from ambient lighting is synthesized with the virtual image and the inability to express a black color due to the characteristics of the optical system that is projected using light.

That is, in the present embodiment, as shown inFIG.5, the mixed reality physical space100bmay be configured to improve the color gamut of the image by installing a surrounding environment configuration such as curtains120for minimizing visibility deterioration due to ambient light, and a black blind110for expressing a black color in the front surface.

As an example, it is possible to install low-reflection materials110,120of a dark achromatic color that can convert the environment on the ceiling or inner wall of a physical space, and configure the surface of furniture130in the physical space with the low-reflection material. The low-reflection materials110,120may include a curtain or blind that suppresses light reflection. The unfolding or folding operation of such curtain or blind can be controlled by an actuator. The actuator may include motors in a broad sense.

In such a mixed reality combined physical space environment, the illuminance of the lighting in the physical space can be set higher than a conventional traditional illuminance. In addition, the traditional physical space100amay be converted into the mixed reality physical space100bfor a mixed reality lecture or conference by a switch operation or the like. In this case, the curtains120and the front blind110having a high extinction coefficient of the physical space are unfolded, so that the light generated by a lighting150is absorbed, and the black blind110at the front surface simultaneously performs the role of a display that expresses a black color and a role of a visor that blocks light, thereby effectively providing an immersive augmented reality environment or an immersive mixed reality environment in a physical space.

FIGS.6and7are another exemplary diagrams for describing a traditional environment of a mixed reality combined physical space and an immersive mixed reality environment.

As shown inFIG.6, in the traditional physical space100a, bright illuminance is provided to provide a traditional environment such as a lecture room, and the color gamut and color contrast of the headset for mixed reality are rapidly reduced due to the bright illuminance, and the generation of the background luminance LBis low by configuring the surface of the ceiling, inner wall and furniture with reflective materials, but the light of the lighting may deteriorate the visibility of the virtual image by the reflection and glare luminance LRgenerated by the optical element of the headset for the mixed reality and the light visor on the bottom.

Meanwhile, as shown inFIG.7, in the mixed reality physical space100b, the illuminance of the lighting150in the physical space100bmay be adjusted to 45 lux in order to implement an immersive environment. In addition, in the mixed reality physical space100b, the black curtains120and the front blind110are unfolded to further lower the background luminance LB, and the reflection and glare luminance LRis also generated less due to the sharply dropped illuminance. Accordingly, it is possible to provide an environment in which the optical sensor of the headset for mixed reality is normally operated, and it is also possible to improve the visibility of the virtual image.

FIG.8is an exemplary diagram illustrating measurement results of a spectrometer and an optical probe for analyzing display optical characteristics of a traditional physical space and a mixed reality (MR) physical space that transforms the use of the traditional physical space.

For this experiment, a GL Optics' spectrometer (Spectis 1.0), an optical probe (Optical Probe 5.0 luminance), and analysis software (GL Spectrosoft) for analyzing optical properties can be used.

As shown inFIG.8, as an analysis result of the optical characteristics of the display device of the headset for mixed reality for each physical space environment, the color gamut compared to 72% of the national television system committee (NTSC) was 4.02% in the traditional physical space (TRS) environment and 60.38% in the mixed reality physical space (MRPS) environment, which was about 15.02 times improvement. The experimental results are summarized in Table 1.

TABLE 1Ambient light conditionTraditional physical spaceMR physical spaceLuminance [lux]796.4845.59ColorRed(0.4161, 03631)(0.6498, 0.3007)CoordinateGreen(0.3472, 03825)(0.2902, 0.5067)(x, y)Blue(0.2455, 0.2266)(0.1466, 0.0577)Color gamut [%]4.0260.38

According to the results of this experiment, it can be seen that the color gamut of the headset for mixed reality with a projection display can be significantly and effectively improved in a wide physical space targeting multiple users.

FIG.9is an exemplary diagram for describing a darkroom environment in a traditional physical space as a comparative example.

Referring toFIG.9, in the darkroom environment100c, since all ambient lights are turned off, the reflection and glare luminance LRand background luminance LBof the projection display are close to 0, so the color gamut close to the original image and high color contrast are obtained. However, if the minimum illuminance for recognizing the surrounding environment and motion is not secured, there may be a problem that the device for mixed reality does not operate.

Accordingly, while implementing an environment similar to the darkroom environment100c, this embodiment is configured to provide the same illuminance environment as in the traditional physical space or an illuminance environment lower than the illuminance environment of the traditional physical space, and this embodiment may provide a mixed reality physical space environment having a high immersion under these conditions.

FIG.10is a schematic block diagram of a configuration of an apparatus for providing a mixed reality combined physical space according to another embodiment of the present disclosure.FIG.11is a block diagram of a software configuration that may be mounted on a processor of the apparatus for providing a physical space combined with mixed reality ofFIG.10.

Referring toFIG.10, an apparatus for providing a mixed reality combined physical space1000may include at least one processor1100and a memory1200that are mounted on a display headset or an electronic controller of a physical space. The electronic controller of a physical space may include a remote server connected to the headsets for mixed reality of multiple users in the physical space via a wireless network.

In addition, the apparatus for providing mixed reality combined physical space1000may further include a transceiver1300connected to a network to perform communication. In addition, the apparatus for providing mixed reality combined physical space1000may further include an input interface device1400, an output interface device1500, a storage device1600, or the like. Each of the components included in the apparatus for providing mixed reality combined physical space1000may be connected by a bus1700to communicate with each other.

The processor1100may execute a program command stored in at least one of the memory1200and the storage device1600. The processor1100may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present disclosure are performed.

A program command may include a command for configuring at least one component of the apparatus for providing a mixed reality combined physical space for providing a high immersion feeling of the present embodiment in software, a command for driving the at least some components, a command for executing the functions of the at least some components, or the like.

As an example, the program command may include a command for controlling the black blind in the front surface of the physical space (blind control command), a command for controlling the curtains on the inner wall of the physical space (curtain control command), a command for controlling the operation of a lighting in the physical space (light control command), a command for acquiring content data for mixed reality (MR) (data acquisition command), a command for MR content identification (MR content identification command), a color chart analysis command for analyzing a color chart of MR content, and a command for extracting a corresponding illuminance value from a content-illuminance lookup table according to the type of content or a preset classification (illuminance value extraction command).

As shown inFIG.11, the above-described program command may be implemented with a blind control unit1110, a curtain control unit1110, a lighting control unit1120, a MR content identification unit1140for acquiring and identifying MR data, and a color chart analysis unit1150, which are mounted on the processor1100. Although these units are described as individual units, it is apparent that two or more units may be integrated into one unit. In addition, the content-illuminance lookup table may be stored in the memory1200or storage device1600connected to the processor1100, and the processor1100reads the content-illuminance lookup table1160to control the illuminance of the lighting device in the physical space with an optimal illuminance preset for all or at least a portion of the pre-identified MR content.

The lighting control unit1120may include an exposure control unit. In this case, the processor may be configured to analyze the color chart of the physical space while gradually increasing the minimum illuminance of the lighting in the physical space at a preset interval or gradually lowering the illuminance from the maximum illuminance at a preset interval to obtain an optimal value of a three-dimensional illuminance in the physical space.

Referring back toFIG.10, each of the memory1200and the storage device1600may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory1200may be configured as at least one of a read only memory (ROM) and a random access memory (RAM). The memory1200and/or the storage device1600may be configured to store the content-illuminance lookup table for each MR content type or the analysis result of the color chart for each illuminance of lighting, and may be collectively referred to as a storage unit.

The transceiver1300may include a communication interface or a sub-communication system for a short-range wireless network or cable connection, communication with a satellite, wired or wireless communication with a general-purpose base station, or an ideal backhaul link or non-ideal backhaul link connection with a mobile edge core network or core network, or the like.

The input interface device1400may include at least one selected from input means such as a keyboard, a microphone, a touch pad, and a touch screen, and an input signal processor that maps or processes a signal input through at least one input means with a pre-stored command.

The above-described transceiver1300and/or the input interface device1400may be included in a signal reception unit that receives a signal for converting a traditional physical space into a mixed reality physical space in a mixed reality combined physical space. In this case, it is apparent that the signal reception unit may receive a signal for converting the mixed reality physical space into the traditional physical space.

The output interface device1500may include an output signal processor that maps or processes a signal outputted under the control of the processor1100to a pre-stored signal form or level, and an output means for outputting a signal or information in the form of vibration, light, etc. according to the signal of the output signal processor. At least one output means may include at least one selected from the output means such as a speaker, a display device, a printer, an optical output device, and a vibration output device.

As described above, in the present embodiment, the user's mixed reality immersion can be increased by improving the color gamut from the user's visual point of view viewing the mixed reality combined physical space through the projection display of the headset for mixed reality.

FIG.12is a flowchart illustrating a method for providing a mixed reality combined physical space according to another embodiment of the present disclosure.

Referring toFIG.12, when a preset level or type of signal is input to an apparatus for providing mixed reality combined physical space (S1210), the processor of the apparatus for providing mixed reality combined physical space determines whether to convert a traditional physical space into a mixed reality physical space according to the input of a predetermined signal (S1220). The processor may terminate a current process when a predetermined signal is not input. Also, in order to convert to the mixed reality physical space environment, the processor may perform controlling of blind and curtain (S1230).

Here, the traditional physical space may refer to a physical space in which the illuminance of a lighting is set to 700 lux or more in a state in which a black blind installed on a front surface is not unfolded and a black curtain on a wall surface are not unfolded in the mixed reality combined physical.

In addition, the mixed reality physical space may refer to a physical space in which the front black blind is unfolded to cover the front surface of the physical space in black and the black curtain on the wall surface is unfolded to cover the wall surface in black. In addition, the illuminance of the mixed reality physical space may be set to a specific illuminance that is substantially the same as or lower than the illuminance of a traditional physical space, for example, 45 lux.

Next, the processor may acquire content data for mixed reality (MR) (S1240).

The processor may be mounted on the headset for mixed reality and may be configured to acquire MR content data stored in a memory or storage device of the headset for mixed reality. In addition, the processor may be mounted on the electronic controller of a physical space and may be configured to receive the MR content data from the headset for mixed reality through a wired network or a wireless network.

That is, the acquisition of the MR content data may include acquiring the content itself, acquiring a content type identifier preset for the content, or acquiring some content for a predetermined time preset for the content. In addition, the acquisition of MR content data is processed by accessing the memory or storage device in which corresponding content is stored and extracting the necessary content or information. Alternatively, the acquisition of MR content data may be processed by receiving MR content data from a separate device storing the MR content data.

Next, the processor may selectively identify the MR content (S1250). The identification of the MR content may be configured to identify the type of the MR content, an average illuminance, and the like. Also, the processor may identify the MR content based on all or part of the MR content data. When a part of the MR content data is used as a reference, the processor may be configured to generate or output a plurality of identification values or a plurality of identification information according to the type or content of the MR content or the passage of time as a result of the identification of the MR content. This MR content identification information may be used to determine the illuminance in the physical space based on the content-illuminance lookup table.

Next, the processor may selectively analyze the color chart of the MR content (S1260). The analysis of the color chart may be used to obtain an average illuminance of MR content or to calculate an optimal illuminance. Such color chart analysis may be performed for a predetermined time or a plurality of times while gradually increasing or decreasing the illuminance of the lighting in the physical space with respect to all or part of the MR content data displayed on the projection display.

Next, the processor may control the illuminance of the lighting according to the identified MR content and/or the analysis result of the color chart (S1270). The processor may set a preset optimal illuminance according to the MR content or may control the illuminance of the lighting in the physical space with an optimal illuminance obtained according to the color chart analysis result according to a preset routine according to the type of MR content. The control of lighting illuminance may be configured to control the operation of lighting with preset multiple optimal illuminances according to the progress of an activity such as a mixed reality lecture based on the MR content.

As such, according to the configuration of this embodiment, it is possible to provide a mixed reality combined physical space that can perform conversion between a traditional physical space and a mixed reality physical space, and an environment for a perfect mixed reality immersion in this physical space.

The operations of the method according to the exemplary embodiment of the present disclosure can be implemented as a computer readable program or code in a computer readable recording medium. The computer readable recording medium may include all kinds of recording apparatus for storing data which can be read by a computer system. Furthermore, the computer readable recording medium may store and execute programs or codes which can be distributed in computer systems connected through a network and read through computers in a distributed manner.

The computer readable recording medium may include a hardware apparatus which is specifically configured to store and execute a program command, such as a ROM, RAM or flash memory. The program command may include not only machine language codes created by a compiler, but also high-level language codes which can be executed by a computer using an interpreter.

Although some aspects of the present disclosure have been described in the context of the apparatus, the aspects may indicate the corresponding descriptions according to the method, and the blocks or apparatus may correspond to the steps of the method or the features of the steps. Similarly, the aspects described in the context of the method may be expressed as the features of the corresponding blocks or items or the corresponding apparatus. Some or all of the steps of the method may be executed by (or using) a hardware apparatus such as a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important steps of the method may be executed by such an apparatus.

In some exemplary embodiments, a programmable logic device such as a field-programmable gate array may be used to perform some or all of functions of the methods described herein. In some exemplary embodiments, the field-programmable gate array may be operated with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by a certain hardware device.

The description of the disclosure is merely exemplary in nature and, thus, variations that do not depart from the substance of the disclosure are intended to be within the scope of the disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the disclosure. Thus, it will be understood by those of ordinary skill in the art that various changes in form and details may be made without departing from the spirit and scope as defined by the following claims.