Patent Description:
Augmented reality (AR) is a technology that combines a real space and a virtual space such that a virtual object appears to exist in a real space, and AR may originate from virtual reality that creates a virtual space similar to reality using computer graphics.

Recently, as an electronic device for providing an AR service (hereinafter referred to as an AR device), wearable electronic devices capable of being directly worn on the body have been developed. For example, the wearable electronic devices providing augmented reality may include a head-mounted device (HMD), a head-mounted display (HMD), or AR glasses.

The AR devices described above may enable a user wearing the same to view content provided through an external display device located in the surrounding environment. Patent document <CIT>, discloses a method of outputting a currently displayed content of a digital device to an HMD, and the method includes performing a pairing with a digital device having at least one display unit configured to display the content, detecting a location state of the paired digital device, and outputting the content displayed on the digital device to the HMD, wherein the location state comprises a first state in which a display unit of the digital device is located within a preset view angle region of the HMD and a second state in which the display unit of the digital device is not located within the view angle region.

If an external display device disappears from the view as the user's gaze moves while viewing content provided to the external display device through an AR device, the user can no longer use the content that the user was viewing.

In addition, if an external display device comes into the view while viewing content through an AR device, the user may wish to view the content through the external display device.

The technical problems to be solved in the disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned may be clearly understood by those of ordinary skill in the art to which the disclosure belongs from the description below.

An electronic device according to the disclosure may include at least one processor, at least one camera, a communication circuit, and at least one display, wherein the processor may be configured to connect to an external display device through the communication circuit, receive information related to content reproduced by the external display device, obtain an image including the external display device that reproduces the content through the camera, obtain information about the external display device, if the external display device is identified to have left an identifiable range of the at least one camera, produce a virtual object corresponding thereto, based on the information about the external display device, obtain the content, based on the information related to content, and reproduce the obtained content within the area of the virtual object.

A method of an electronic device according to the disclosure may include connecting to an external display device and receiving information related to content reproduced by the external display device, obtaining an image including the external display device that reproduces the content and obtaining information about the external display device, if the external display device is identified to have left an identifiable range of the at least one camera, producing a virtual object corresponding thereto, based on the information about the external display device, and obtaining the content, based on the information related to content, and reproducing the obtained content within the area of the virtual object.

According to various embodiments, in the case where a display device disappears from the view of a user as the user's gaze moves while wearing an AR device and viewing content provided by an external display device, it is possible to provide the content that the user was viewing on the display device as an AR service through the AR device.

In addition, when an external display device comes into the view while viewing content according to an AR service provided by the AR device, it is possible to provide the corresponding content by the external display device.

Various embodiments may seamlessly and conveniently provide an optimized mixed reality experience in association with an external display device without overlapping between real objects and virtual objects in providing content according to an AR service through an AR device.

In addition, various effects directly or indirectly identified may be provided.

The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which.

Referring to <FIG>, the electronic device <NUM> in the network environment <NUM> may communicate with an electronic device <NUM> via a first network <NUM> (e.g., a short-range wireless communication network), or at least one of an electronic device <NUM> or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network). According to an embodiment, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input 1module <NUM>, a sound output 1module <NUM>, a display 1module <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a connecting terminal <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module (SIM) <NUM>, or an antenna module <NUM>. In some embodiments, at least one of the components (e.g., the 11connecting terminal <NUM>) may be omitted from the electronic device <NUM>, or one or more other components may be added in the electronic device <NUM>. In some embodiments, some of the components (e.g., the sensor module <NUM>, the camera module <NUM>, or the antenna module <NUM>) may be implemented as a single component (e.g., the display module <NUM>).

The auxiliary processor <NUM> may control at least some of functions or states related to at least one component (e.g., the display 1module <NUM>, the sensor module <NUM>, or the communication module <NUM>) among the components of the electronic device <NUM>, instead of the main processor <NUM> while the main processor <NUM> is in an inactive (e.g., sleep) state, or together with the main processor <NUM> while the main processor <NUM> is in an active state (e.g., executing an application).

The input 1module <NUM> may receive a command or data to be used by another component (e.g., the processor <NUM>) of the electronic device <NUM>, from the outside (e.g., a user) of the electronic device <NUM>. The input 1module <NUM> may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output 1module <NUM> may output sound signals to the outside of the electronic device <NUM>. The sound output 1module <NUM> may include, for example, a speaker or a receiver.

The display 1module <NUM> may visually provide information to the outside (e.g., a user) of the electronic device <NUM>. The display 1module <NUM> may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display 1module <NUM> may include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the intensity of force incurred by the touch.

According to an embodiment, the audio module <NUM> may obtain the sound via the input 1module <NUM>, or output the sound via the sound output 1module <NUM> or a headphone of an external electronic device (e.g., an electronic device <NUM>) directly (e.g., wiredly) or wirelessly coupled with the electronic device <NUM>.

The communication module <NUM> may include one or more communication processors that are operable independently from the processor <NUM> (e.g., the AP) and supports a direct (e.g., wired) communication or a wireless communication.

<FIG> is a diagram of an augmented reality (AR) device according to an embodiment. <FIG> is a diagram illustrating a front side of an AR device according to an embodiment. <FIG> is a diagram illustrating a rear side of an AR device according to an embodiment. <FIG> is a first exemplary view showing the front part of the AR device <NUM>, <FIG> is a second exemplary view showing the rear part of the AR device <NUM>, and the internal configuration may be as shown in <FIG>.

In various embodiments, the AR device <NUM> may be worn on a user's head to provide the user with an image related to an AR service. According to an embodiment, the AR device <NUM> may provide the AR service in which at least one virtual object is output to be superimposed on an area determined as a user's field of view (FoV). For example, the area determined as the user's FoV is an area determined to be recognizable through the AR device <NUM> by the user wearing the AR device <NUM>, and it may be an area including the overall or at least a part of a display module (e.g., a display module <NUM> in <FIG>) of the AR device <NUM>. According to an embodiment, the AR device <NUM> may include a plurality of glasses (e.g., a first glass <NUM> and/or a second glass <NUM>) corresponding to both eyes of the user (e.g., left eye and/or right eye). The plurality of glasses may include at least a part of a display module (e.g., a first display module <NUM>-<NUM> and/or a second display module <NUM>-<NUM> in <FIG>). For example, the first glass <NUM> corresponding to the user's left eye may include the first display module <NUM>-<NUM>, and the second glass <NUM> corresponding to the user's right eye may include the second display module <NUM>-<NUM>. The AR device <NUM> may be configured, for example, in the form of at least one of glasses, goggles, a helmet, or a hat, but is not limited thereto.

With reference to <FIG>, the AR device <NUM> according to an embodiment may include a display module <NUM>, a camera module, an audio module, a first support part <NUM>, and/or a second support part <NUM>. According to an embodiment, the display module may include a first display (e.g., the first glass <NUM>) (e.g., the first display module <NUM>-<NUM> in <FIG>) and/or a second display (e.g., the second glass <NUM>) (e.g., the second display module <NUM>-<NUM> in <FIG>). According to an embodiment, at least one camera may include a shooting camera <NUM> for capturing an image corresponding to the user's FoV and/or measuring a distance to an object, an eye tracking camera <NUM> for identifying the direction of a user's gaze, and/or gesture cameras <NUM>-<NUM> and <NUM>-<NUM> for recognizing a certain space. For example, the shooting camera <NUM> may photograph in a front direction of the AR device <NUM>, and the eye tracking camera <NUM> may photograph in a direction opposite to the photographing direction of the shooting camera <NUM>. For example, the eye tracking camera <NUM> may photograph at least in part both eyes of the user. According to an embodiment, the first support part <NUM> and/or the second support part <NUM> may include a PCB <NUM>-<NUM>, <NUM>-<NUM>, a speaker <NUM>-<NUM>, <NUM>-<NUM>, and/or a battery <NUM>-<NUM>, <NUM>-<NUM>.

According to an embodiment, the display module (e.g., the display module <NUM> in <FIG>) may be disposed in a body part (e.g., a body part <NUM> in <FIG>) of the AR device <NUM>, and may include a condensing lens and/or a transparent waveguide on the glass (e.g., the first glass <NUM> and the second glass <NUM>). For example, the transparent waveguide may be positioned at least in part on a portion of the glass. According to an embodiment, the light emitted from the display module may be incident on one end of the glass through the first glass <NUM> and the second glass <NUM>, and the incident light may be transferred to the user through the waveguide formed on the glass. The waveguide may be made of glass, plastic, or polymer, and may include a nano-pattern, e.g., a polygonal or curved grating structure, formed on one inner or outer surface thereof. According to an embodiment, the incident light may be propagated or reflected inside the waveguide by the nano-pattern and provided to the user. According to an embodiment, the waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) and/or a reflective element (e.g., a reflective mirror). According to an embodiment, using the at least one diffractive element or the reflective element, the waveguide may guide the display light emitted from the light source to the user's eyes.

With reference to <FIG>, the first support part <NUM> and/or the second support part <NUM> may include the printed circuit board <NUM>-<NUM>, <NUM>-<NUM> for transmitting an electrical signal to respective components of the AR device <NUM>, the speaker <NUM>-<NUM>, <NUM>-<NUM> for outputting an audio signal, the battery <NUM>-<NUM>, <NUM>-<NUM>, and/or a hinge part <NUM>-<NUM>, <NUM>-<NUM> coupled at least in part to the body part <NUM> of the AR device <NUM>. According to an embodiment, the speaker <NUM>-<NUM>, <NUM>-<NUM> may include a first speaker <NUM>-<NUM> for transferring an audio signal to the user's left ear and a second speaker <NUM>-<NUM> for transferring an audio signal to the user's right ear. According to an embodiment, the AR device <NUM> may include a plurality of batteries <NUM>-<NUM> and <NUM>-<NUM>, which may supply power to the printed circuit boards <NUM>-<NUM> and <NUM>-<NUM> through a power management module.

Referring to <FIG>, the AR device <NUM> may include a microphone <NUM> for receiving a user's voice and ambient sounds. The AR device <NUM> may include at least one light emitting device (illumination light emitting diode (LED)) <NUM> for increasing the accuracy of the at least one camera (e.g., the shooting camera <NUM>, the eye tracking camera <NUM>, and/or the gesture cameras <NUM>-<NUM> and <NUM>-<NUM>). For example, the light emitting device <NUM> may be used as an auxiliary means to increase accuracy when the eye tracking camera <NUM> photographs the user's pupil, and the light emitting device <NUM> may use an IR LED having an infrared wavelength rather than a visible light wavelength. In another example, when the gesture cameras <NUM>-<NUM> and <NUM>-<NUM> take a user's gesture, the light emitting device <NUM> may be used as an auxiliary means if it is not easy to detect a subject to be photographed due to a dark environment or mixing of various light sources and reflected light.

With reference to <FIG> and <FIG>, the AR device <NUM> according to an embodiment may include the body part <NUM> and the support part (e.g., the first support part <NUM> and/or the second support part <NUM>), and the body part <NUM> and the support parts <NUM> and <NUM> may be operatively connected. For example, the body part <NUM> and the support parts <NUM> and <NUM> may be operatively connected through the hinge parts <NUM>-<NUM> and <NUM>-<NUM>. The body part <NUM> may be mounted at least in part on the user's nose, and may include the display module and the camera module. The support parts <NUM> and <NUM> may include a support member mounted on the user's ear, and may include the first support part <NUM> mounted on the left ear and/or the second support part <NUM> mounted on the right ear. According to an embodiment, the first support part <NUM> or the second support part <NUM> may include at least in part the printed circuit board <NUM>-<NUM>, <NUM>-<NUM>, the speaker <NUM>-<NUM>, <NUM>-<NUM>, and/or the battery <NUM>-<NUM>, <NUM>-<NUM>. The battery may be electrically connected to the power management module.

According to an embodiment, the display module may include the first glass <NUM> and/or the second glass <NUM>, and may provide visual information to the user through the first glass <NUM> and the second glass <NUM>. The AR device <NUM> may include the first glass <NUM> corresponding to the left eye and/or the second glass <NUM> corresponding to the right eye. According to an embodiment, the display module may include a display panel and/or a lens (e.g., glass). For example, the display panel may include a transparent material such as glass or plastic.

According to an embodiment, the display module may be formed of a transparent element, so the user may recognize a real space behind the display module through the display module. The display module may display a virtual object on at least a portion of the transparent element so that the virtual object can be seen to the user as if added to at least a portion of the real space. The first glass <NUM> and/or the second glass <NUM> included in the display module may include a plurality of display panels corresponding to both eyes of the user (e.g., the left eye and/or the right eye).

According to an embodiment, the AR device <NUM> may include a virtual reality (VR) device. When the AR device <NUM> is the VR device, the first glass <NUM> may be the first display module <NUM>-<NUM>, and the second glass <NUM> may be the second display module <NUM>-<NUM>.

According to an embodiment, the virtual object output through the display module may include information related to an application program executed in the AR device <NUM> and/or information related to an external object located in the real space corresponding to an area determined as a user's FoV. For example, from image information related to the real space acquired through the camera (e.g., the shooting camera <NUM>) of the AR device <NUM>, the AR device <NUM> may identify an external object included in at least a portion corresponding to the area determined as the user's FoV. The AR device <NUM> may output (or display) a virtual object related to the identified external object through the area determined as the user's FoV in the display area of the AR device <NUM>. The external object may include a thing existing in the real space. According to various embodiments, the display area in which the AR device <NUM> displays the virtual object may include a portion (e.g., at least a portion of the display panel) of the display module (e.g., the first display module <NUM>-<NUM> or the second display module <NUM>-<NUM>). According to an embodiment, the display area may correspond to at least a portion of the first glass <NUM> and/or the second glass <NUM>.

According to an embodiment, the AR device <NUM> may include the shooting camera <NUM> (e.g., an RGB camera) for capturing an image corresponding to the user's FoV and/or measuring a distance to an object, the eye tracking camera <NUM> for identifying the direction of a user's gaze, and/or the recognizing cameras <NUM>-<NUM> and <NUM>-<NUM> (e.g., gesture camera) for recognizing a certain space. According to an embodiment, using the shooting camera <NUM>, the AR device <NUM> may measure a distance to an object located in the front direction of the AR device <NUM>. According to an embodiment, in the AR device <NUM>, a plurality of eye tracking cameras <NUM> may be disposed to correspond to both eyes of the user. For example, the eye tracking camera <NUM> may photograph in a direction opposite to the photographing direction of the shooting camera <NUM>. The eye tracking camera <NUM> may detect the user's gaze direction (e.g., eye movement). For example, the eye tracking camera <NUM> may include a first eye tracking camera <NUM>-<NUM> for tracking the gaze direction of the user's left eye, and a second eye tracking camera <NUM>-<NUM> for tracking the gaze direction of the user's right eye. According to an embodiment, using the recognizing cameras <NUM>-<NUM> and <NUM>-<NUM>, the AR device <NUM> may detect a user gesture within a predetermined distance (e.g., a certain space). For example, there may be a plurality of recognizing cameras <NUM>-<NUM> and <NUM>-<NUM>, which may be disposed on both sides of the AR device <NUM>. Using at least one camera, the AR device <NUM> may detect an eye corresponding to a primary eye and/or an auxiliary eye from among the left eye and/or the right eye. For example, the AR device <NUM> may detect an eye corresponding to the primary eye or the auxiliary eye or both, based on the user's gaze direction with respect to the external object or the virtual object.

According to an embodiment, the shooting camera <NUM> may include a high resolution (HR) camera and/or a photo video (PV) camera. According to an embodiment, the eye tracking camera <NUM> may detect the user's pupil, thereby track the gaze direction, and may be utilized to move the center of a virtual image along the gaze direction. For example, the eye tracking camera <NUM> may be divided into the first eye tracking camera <NUM>-<NUM> corresponding to the left eye and the second eye tracking camera <NUM>-<NUM> corresponding to the right eye, which may have substantially the same performance and/or specifications. According to an embodiment, the recognizing cameras <NUM>-<NUM> and <NUM>-<NUM> may be used for detection of a user's hand (gesture) or spatial recognition or both, and may include a global shutter (GS) camera. For example, in order to detect and track quick hand movements and/or minute finger movements, the recognizing cameras <NUM>-<NUM> and <NUM>-<NUM> may include the GS camera with less screen afterimage such as a rolling shutter (RS) camera.

According to an embodiment, the AR device <NUM> may display together a virtual object related to the AR service, based on image information related to a real space acquired through a camera (e.g., the camera module <NUM> in <FIG>) of the AR device <NUM>. According to an embodiment, the AR device <NUM> may display the virtual object, based on a display module (e.g., the first display module <NUM>-<NUM> corresponding to the left eye and/or the second display module <NUM>-<NUM> corresponding to the right eye) disposed to correspond to both eyes of the user. According to an embodiment, the AR device <NUM> may display the virtual object, based on predetermined setting information (e.g., resolution, frame rate, brightness, and/or display area).

According to an embodiment, the AR device <NUM> may enable the first display panel included in the first glass <NUM> and the second display panel included in the second glass <NUM> to be operated as independent components, respectively. For example, the AR device <NUM> may determine the display performance of the first display panel, based on first setting information, and may determine the display performance of the second display panel, based on second setting information.

The number and location(s) of one or more cameras (e.g., the shooting camera <NUM>, the eye tracking camera <NUM>, or the recognizing camera <NUM>-<NUM>, <NUM>-<NUM>) included in the AR device <NUM> shown in <FIG>, <FIG>, and/or <FIG> may not be limited. For example, based on the form (e.g., shape or size) of the AR device <NUM>, the number and location(s) of one or more cameras (e.g., the shooting camera <NUM>, the eye tracking camera <NUM>, and/or the recognizing camera <NUM>-<NUM>, <NUM>-<NUM>) may vary.

<FIG> is a diagram illustrating a connection relationship of a display device <NUM>, an AR device <NUM> (e.g., the AR device <NUM> in <FIG>), and a server <NUM> according to an embodiment.

Referring to <FIG>, in order to provide content, the AR device <NUM> may be connected to the display device <NUM> using a wireless network <NUM>. The AR device <NUM> may be in the form of glasses capable of being worn on the user as shown in <FIG>.

The display device <NUM> may be an electronic device that performs wireless communication and includes a display for outputting a variety of content including image data. For example, the display device <NUM> may include various electronic devices having a display, such as a television (TV) set, a mobile terminal, a mobile phone, and a monitor. The AR device <NUM> may receive information related to the content, which is being currently reproduced, from the display device <NUM> using the wireless network <NUM>. The content being reproduced in the display device <NUM> may include a variety of content such as moving images, still images, documents, and homepages, but is not limited thereto. The content-related information may include, for example, related application information, content reproduction information, information on the name, capacity, and version of corresponding content, and information for obtaining corresponding content. The information for obtaining the content may include information for connection with the server <NUM> providing the content and/or information on the address in which the content is stored. The content reproduction information may include, for example, audio volume level information according to reproduction of the content in the display device <NUM>, reproduction-related information including the reproduction position such as information on a frame currently being reproduced or a reproduction time, and, in the case of a web page, the address and the reproduction position or time thereof.

The server <NUM> may provide a variety of data for supporting an augmented reality service provided through the AR device <NUM>. The server <NUM> may provide a variety of content output from the AR device <NUM> or the display device <NUM>.

According to an embodiment, the AR device <NUM> may connect with the server <NUM> through the wireless network <NUM> to receive a variety of data for providing an augmented reality service. The server <NUM> may be communicatively connected to the AR device <NUM> using wireless communication. The wireless communication network <NUM> connecting the server <NUM> and the AR device <NUM> may be a long-range wireless communication network (e.g., the second network <NUM> in <FIG>). For example, the server <NUM> and the AR device <NUM> may be connected using at least one wireless communication scheme among a cellular network, a fifth generation (<NUM>) network, a next-generation communication network, and the Internet, but the wireless communication scheme is not limited thereto.

According to an embodiment, the AR device <NUM> may obtain corresponding content data, based on information related to the content reproduced in the display device <NUM>.

According to an embodiment, the AR device <NUM> may receive corresponding content data from the display device <NUM>, based on the content-related information.

According to an embodiment, the AR device <NUM> may communicate with the server <NUM>, based on the information related to the content reproduced in the display device <NUM>, and receive corresponding content data from the server <NUM>. The AR device <NUM> may execute an application, based on the content-related information, and output the received content data. The AR device <NUM> may output the received content data through a virtual object that is produce based on information related to the display device <NUM>. The information related to the display device <NUM> may include image information that includes form information including the size, color, and shape, position information, and depth information of the display device <NUM>. The AR device <NUM> may obtain information related to the display device <NUM> using a sensor or a camera.

The display device <NUM> and the AR device <NUM> may be connected through the wireless communication network <NUM>. The wireless communication network <NUM> connecting the display device <NUM> and the AR device <NUM> may be a short-range wireless network (e.g., the first network <NUM> in <FIG>). According to various embodiments, the display device <NUM> and the AR device <NUM> may be connected using a wireless communication scheme such as WiFi-peer to peer (P2P), Bluetooth, and Bluetooth low energy (BLE), but the wireless communication scheme is not limited to the above-described examples.

According to various embodiments, in order to provide an AR service, the AR device <NUM> may obtain information about actual surrounding environment including position information, size information, distance information, sensor information, and image information of various real objects including the display device <NUM> using a sensor or a camera. The AR device <NUM> may produce a virtual object, based on the information about the surrounding environment, and output the same through a display.

According to an embodiment, the AR device <NUM> may receive a variety of data including content data from the server <NUM> or the display device <NUM> and output the received content data through a virtual object. For example, the AR device <NUM> may output a virtual object and display content data within the area of the output virtual object.

According to an embodiment, in order to provide an AR service, the AR device <NUM> may collect a variety of information, based on communication with the display device <NUM> or the server <NUM>, as well as receiving content data or a variety of information from the display device <NUM> or the server <NUM>.

<FIG> is a diagram of a display device <NUM> according to an embodiment. <FIG> is a diagram of an AR device <NUM> according to an embodiment.

Referring to <FIG>, the display device <NUM> may include a communication module <NUM>, a display <NUM>, a memory <NUM>, and a processor <NUM>. The display device <NUM> may further include at least some of the configurations and/or functions of the electronic device <NUM> in <FIG>.

According to various embodiments, the communication module <NUM> may include a software and/or hardware module (e.g., a CP) for wireless communication with a network or an external electronic device (e.g., the AR device <NUM> in <FIG> and/or the server <NUM> in <FIG>) and include at least some of the configurations and/or functions of the communication module <NUM> in <FIG>. The communication module <NUM> may communicate with an external electronic device through a wireless communication network (e.g., the first network <NUM> or the second network <NUM> in <FIG>).

According to various embodiments, the communication module <NUM> may transmit data provided from another element (e.g., the processor <NUM>) of the display device to an external electronic device, or receive data from an external electronic device and transmit the same to another element of the display device. The communication module <NUM> may communicate with the AR device <NUM> through a wireless communication network (e.g., the first network <NUM> in <FIG> (e.g., a short-range wireless communication network)) or communicate with the server <NUM> through a long-range wireless network (e.g., the second network <NUM> in <FIG>).

The display <NUM> may output and provide content including a visual element, based on content data, and include at least some of functions and/or configurations of the display module <NUM> in <FIG>. For example, the display <NUM> may output data provided from another element (e.g., the processor <NUM>) of the display device to the outside of the display device.

According to various embodiments, the memory <NUM> may store a variety of data used by at least one element (e.g., the processor <NUM>) of a terminal device and include at least some of the configurations and/or functions of the memory <NUM> in <FIG>. The memory <NUM> may store instructions to be executed by the processor <NUM>. For example, the memory <NUM> may store a variety of information such as address information required for the processor <NUM> to connect with the server <NUM> and obtain content data.

The processor <NUM> may process data in the display device, control at least one other element related to the function of the display device, and perform data processing and operation necessary for executing the function. The processor <NUM> may include at least some of functions and/or configurations of the processor <NUM> in <FIG>. The processor <NUM> may be electrically and/or functionally connected to the elements of a terminal device, such as the communication module <NUM>, the display <NUM>, and the memory <NUM>.

According to various embodiments, the processor <NUM> may receive control information from the AR device <NUM> through the communication module <NUM> and, based on the same, perform control operations such as volume control, reproduction, pause, or stop of content, and turning on/off the display <NUM>.

According to various embodiments, the processor <NUM> may provide information related to content currently being reproduced in response to a request of the AR device <NUM>. The processor <NUM> may receive specific content-related information from the AR device <NUM>, obtain corresponding content data using the received content-related information according to control information of the AR device <NUM>, and output the same through the display <NUM>.

Referring to <FIG>, the AR device <NUM> may include a communication module <NUM>, a camera <NUM>, a sensor <NUM>, a display module <NUM>, and a processor <NUM>.

The AR device <NUM> may be in the form of glasses capable of being worn on a user, as shown in <FIG>.

The communication module <NUM> may include a software and/or hardware module (e.g., a CP) for wireless communication with a network or an external electronic device (e.g., the display device <NUM> in <FIG> or the server <NUM> in <FIG>).

According to various embodiments, the communication module <NUM> may transmit data provided from another element (e.g., the processor <NUM>) of the AR device to an external electronic device, or receive data from an external electronic device and transmit the same to another element of the AR device. The communication module <NUM> may communicate with the display device <NUM> through a wireless communication network (e.g., the first network <NUM> in <FIG> (e.g., a short-range wireless communication network)) or communicate with the server <NUM> through a long-range wireless network (e.g., the second network <NUM> in <FIG>).

The camera <NUM> may photograph still images and moving images. According to an embodiment, the camera may include one or more lenses and image sensors. According to various embodiments, the camera <NUM> may include at least some of the functions and/or configurations of the camera module in <FIG>. A recognition camera (gesture camera) for recognizing a certain space, which is able to photograph a real image corresponding to an FoV and/or measure a distance to an object, may be included. According to various embodiments, the camera may photograph a real image corresponding to the FoV of the AR device in order to obtain image information.

The sensor <NUM> may detect movement of the AR device. According to various embodiments, the sensor may detect a physical quantity related to the movement of the AR device, for example, the velocity, acceleration, angular velocity, angular acceleration, and geographic position of the AR device. The sensor <NUM> may include various sensors and include a gyro sensor and a gravity sensor, thereby detecting the movement of the AR device, including the position, speed, and/or posture of the AR device. The sensor <NUM> may include, for example, a tactile sensor to detect wearing of the AR device. The sensor <NUM> may include an ultrasonic sensor or a light detection and ranging (LiDAR) sensor that emits laser pulses for a LiDAR function.

The display module <NUM> may visually output information processed by the processor <NUM> of the AR device. For example, the display module <NUM> may display various virtual objects, based on virtual information received by the AR device from a server or virtual information produced by the processor <NUM>.

According to various embodiments, the display module <NUM> may include the glasses in <FIG> (e.g., the first glass <NUM> and the second glass <NUM>) and emit light to the transparent waveguide included in at least a portion of the glasses, and induce the emitted light to the user's eyes, thereby outputting virtual information. The display module <NUM> may include a first display module <NUM>-<NUM> corresponding to the user's left eye (e.g., the first glass <NUM> in <FIG>) and a second display module <NUM>-<NUM> corresponding to the user's right eye (e.g., the second glass <NUM> in <FIG>). According to an embodiment, the display module <NUM> may include glasses formed of a transparent (or translucent) element, and a user may recognize the real space behind the display module <NUM> through the display module <NUM>. The display module <NUM> may display a virtual object in at least a portion of the transparent element so that the user may see the virtual object added to at least a portion of the real space.

The processor <NUM> may control at least one other element related to the function of the AR device and perform data processing and operation necessary to execute the function. The processor <NUM> may be electrically and/or functionally connected to elements of the AR device such as the communication module <NUM>, the camera, the sensor, and the display module <NUM>.

According to various embodiments, the processor <NUM> may measure physical quantities (e.g., geographic position, velocity, acceleration, angular velocity, and angular acceleration of the AR device) related to the movement of the AR device through the sensor, and obtain movement information of the AR device using the measured physical quantities or a combination thereof.

According to various embodiments, the processor <NUM> may obtain image information and movement information of the AR device. The processor <NUM> may obtain image information by photographing an image through the camera. For example, the processor <NUM> may obtain image information that includes form and size information including the size, color, and shape of the display device <NUM>, and position information and depth information of the display device <NUM> from the image photographed by the camera.

The processor <NUM> may obtain information related to the display device <NUM> such as display brightness (illuminance) information, size information, and position information of the display device <NUM> through sensors such as an illuminance sensor, a LiDAR sensor, and an ultrasonic sensor.

According to various embodiments, the processor <NUM> may obtain virtual information, based on the movement information and/or image information of the AR device, and produce a virtual object, based thereon. For example, the processor <NUM> may extract an image for the display device <NUM> from an image photographed by the camera and obtain image information including position information and depth information of the display device <NUM> existing in the image.

According to various embodiments, the processor <NUM> may produce a virtual object representing the display device <NUM>, based on the image information, and display the same through the display module <NUM>. For example, the processor <NUM> may produce a virtual object representing the display device <NUM>, based on the image information including size information and illuminance information of the display device <NUM>, and position information and depth information of the display device <NUM>.

According to various embodiments, the processor <NUM> may analyzes a real image corresponding to the FoV of the AR device, based on movement information of the AR device and/or image information obtained from the camera, and, if the display device <NUM> has left the FoV, produce a virtual object corresponding to the display device <NUM>, based on information related to the display device <NUM>, thereby displaying the same through the display module <NUM>.

According to various embodiments, the processor <NUM> of the AR device may receive content data from the display device <NUM> or the server <NUM>. For example, the processor <NUM> may receive content data from the display device <NUM>. For example, the processor <NUM> may receive related content data from the server <NUM>, based on content-related information received from the display device <NUM>. The processor <NUM> may receive the content data at a specific frame rate from the server <NUM>.

According to various embodiments, if the display device <NUM> has left the FoV , the processor <NUM> of the AR device may display a content image based on the received content data within a virtual object area corresponding to the display device <NUM>, which is produced based on the information related to the display device <NUM>.

An electronic device (e.g., the electronic device <NUM> in <FIG>, <FIG>, <FIG>, or <FIG> or the AR device <NUM> in <FIG>) according to various embodiments may include at least one processor (e.g., the processor <NUM> in <FIG> or the processor <NUM> in <FIG>), at least one camera (e.g., the camera module <NUM> in <FIG> or the camera <NUM> in <FIG>), a communication circuit (e.g., the communication module <NUM> in <FIG> or the communication module <NUM> in <FIG>), and at least one display (e.g., the display module <NUM> in <FIG>, or the first display module <NUM>-<NUM> and/or the second display module <NUM>-<NUM> in <FIG>), where the processor may connect to an external display device (e.g., the display device <NUM> in <FIG>) through the communication circuit, receive information related to content reproduced by the external display device, obtain an image including the external display device that reproduces the content through the camera, obtain information about the external display device, if the external display device is identified to have left (e.g., being outside of) an identifiable range of the at least one camera, produce a virtual object corresponding thereto, based on the information about the external display device, obtain the content, based on the information related to content, and reproduce the obtained content within the area of the virtual object.

According to various embodiments, the processor may provide an image including the external display device that reproduces the content through the display after the external display device has left the identifiable range until the content is reproduced.

According to various embodiments, the processor may stop providing an image including the external display device if the content starts to be reproduced through the virtual object.

According to various embodiments, the information about the external display device may be obtained from the image including the external display device.

According to various embodiments, the information about the external display device may include at least one piece of size, position, or depth information of the external display device, and the virtual object may be produced based on the information about the external display device.

According to various embodiments, the processor may receive the content from the external display device through the communication circuit, based on the information related to content.

According to various embodiments, the information related to content may include information on the external server providing the content or address information of the content. The processor may connect to the external server, based on the information related to content, and receive the content.

According to various embodiments, the information related to content may include at least one piece of application information for reproducing the content, audio volume level information of the content, and reproduction time information of the content, and the processor may reproduce the content subsequent to the content that was displayed in the external display device within the area of the virtual object using the information related to content.

According to various embodiments, another virtual object for controlling the content reproduced in the area of the virtual object may be displayed near the virtual object produced based on the information about the external display device after the external display device has left the identifiable range of the at least one camera.

According to various embodiments, the processor may perform at least one of an operation of muting the external display device or an operation of switching the external display device to an inactive state if the content starts to be reproduced through the virtual object.

<FIG> is a diagram illustrating a communication flow between a display device <NUM> and an AR device <NUM> according to an embodiment.

Referring to <FIG>, when the AR device <NUM> discovers the display device <NUM> through image information photographed by a camera (e.g., the camera <NUM> in <FIG>) of the AR device after the AR device <NUM> is worn on the user and turned on, a processor (e.g., the processor <NUM> in <FIG>) of the AR device <NUM> may communicate with the display device <NUM> through a communication module (e.g., the communication module <NUM> in <FIG>).

According to various embodiments, the display device <NUM> and the AR device <NUM> may be communicatively connected using short-range wireless communication (e.g., Wi-Fi P2P, Bluetooth, or BLE).

The AR device <NUM> may transmit/receive a variety of control data to and from the display device <NUM> in operation <NUM>. The control data exchanged between the AR device <NUM> and the display device <NUM> may include, for example, information for controlling a connection between the display device <NUM> and the AR device <NUM> and/or information for controlling the operation of the display device <NUM>. The information for controlling the connection between the display device <NUM> and the AR device <NUM> may include, for example, information necessary for establishment of connection, such as internet protocol (IP) information and negotiation information of the display device <NUM> and the AR device <NUM>. The information for controlling the operation of the display device <NUM> may include control operations such as volume control of the display device <NUM>, reproduction, pause, or stop of content, and power-on/off, and control messages such as a request for transmission of specific information. For example, the AR device <NUM> may request information related to content reproduced in the display device <NUM>.

The AR device <NUM> may receive content-related information from the display device <NUM> in operation <NUM>. The content-related information may include, for example, related application information, information about the name, capacity, and version of the corresponding content, reproduction information thereof, and information for obtaining the content. The information for obtaining the content may include information for connection with the server <NUM> that provides the content and/or information on an address in which the content is stored. The content reproduction information may include, for example, audio volume level information according to reproduction of the content in the display device <NUM>, reproduction-related information including the reproduction position such as current reproduction time information or frame information, and, in the case of a web page, the address and the reproduction position or time thereof.

If it is identified that the display device <NUM> has left the FoV, the AR device <NUM> may transmit control data to the display device <NUM> in operation <NUM>. The transmitted control data may include, for example, a control command to mute the display device <NUM>, stop reproduction of content being reproduced in the display device <NUM>, enter an inactive state, or turn off the display device <NUM>.

According to various embodiments, the display device <NUM> may be muted or the audio volume thereof may be adjusted based on at least one piece of information about the degree to which the display device <NUM> falls outside of the FoV of the AR device <NUM> or the amount of change in the audio volume of the display device <NUM> detected by the microphone of the AR device <NUM>. For example, if a change in the audio volume of the display device <NUM> detected by the microphone of the AR device <NUM> is not large even in the case where the display device <NUM> has left the FoV of the AR device <NUM>, the audio volume of the display device <NUM> may remain as it is. In addition, the AR device <NUM> may gradually change the volume of the display device <NUM> and the AR device <NUM> step by step (e.g., incrementally) with respect to each other, thereby providing the user with an effect of naturally switching the audio volume (e.g., a gradual decrease in the volume of the display device and a gradual increase in the volume of the AR device depending on the degree of falling outside of the FoV or the amount of change in the volume).

<FIG> is a diagram illustrating a communication flow for receiving content data from a server (e.g., the server <NUM> in <FIG>) or a display device (e.g., the display device <NUM> in <FIG>) in an AR device (e.g., the AR device <NUM> in <FIG>) according to an embodiment.

The operation of the AR device <NUM> may be performed under the control of a processor (e.g., the processor <NUM> in <FIG>) of the AR device <NUM>. Hereinafter, although an example in which the AR device <NUM> receives content data from the server <NUM> will be described, the disclosure is not limited thereto, and the content data may also be received from the display device <NUM>.

According to an embodiment, the AR device <NUM> may establish a connection with the server <NUM> through a wireless communication network in operation <NUM>, based on the content-related information received from the display device <NUM> in operation <NUM> in <FIG>, may transmit/receive control data for requesting content data to/from the server <NUM>. The content-related information may include, for example, related application information, information on the name, capacity, and version of corresponding content, reproduction information, and information for obtaining corresponding content. The information for obtaining the content may include information for connection with the server <NUM> providing the content and/or information on the address in which the content is stored. The content reproduction information may include, for example, audio volume level information according to reproduction of the content in the display device <NUM>, reproduction-related information including the reproduction position such as current reproduction time or frame information, and, in the case of a web page, the address and the reproduction position or time thereof.

The wireless communication network connecting the AR device <NUM> and the server <NUM> may be a long-range wireless communication network (e.g., the second network <NUM> in <FIG>). For example, the server <NUM> and the AR device <NUM> may be connected using at least one wireless communication scheme among a cellular network, a <NUM> network, a next-generation communication network, and the Internet, but the wireless communication scheme is not limited thereto. The AR device <NUM> may transmit content-related information to the server <NUM> after the connection with the server <NUM> is established.

The AR device <NUM> may receive content data from the server <NUM>, based on the content-related information, in operation <NUM>. For example, the server <NUM> may stream content data images to the AR device <NUM> at a specified frame rate. For example, the AR device <NUM> may transmit transmission control information for controlling video streaming of the content data to the server <NUM> in operation <NUM>. The transmission control information may include information about a transmission frame rate of content video streaming and information for controlling content data transmission such as stopping or ending transmission.

Requested content may be specified based on the information, and data corresponding to the entirety of the specified content data or frames subsequent to the specified frame may be streamed to the AR device <NUM>. The specified frame may be a frame before or after a frame recently output from the display device <NUM> according to the content reproduction information.

<FIG> is a flowchart illustrating the operation of an AR device (e.g., the AR device <NUM> in <FIG>) according to an embodiment. <FIG> is a diagram illustrating a viewing range of an AR device <NUM> according to an embodiment. <FIG> and <FIG> are diagrams illustrating the operation of an AR device <NUM> according to an embodiment. Hereinafter, the operation of the AR device <NUM> will be described with reference to <FIG>. The operation of the AR device <NUM> to be described below may be performed under the control of a processor (e.g., the processor <NUM> in <FIG>).

If the user wears the AR device <NUM> after it is turned on, the AR device <NUM> may detect that it is worn through a sensor (e.g., the sensor <NUM> in <FIG>), and then start an operation of providing an AR service.

The AR device <NUM> may detect an external display device (e.g., the display device <NUM> in <FIG>) in operation <NUM>. The AR device <NUM> may photograph an image including real objects within an FoV through a camera (e.g., the camera <NUM> in <FIG>), and the processor may identify whether or not the display device <NUM> is present in the image, thereby detecting the display device <NUM>. For example, the user may wear the AR device <NUM> and view content displayed in the display device <NUM>, or may wear the AR device <NUM> while viewing content displayed in the display device <NUM>, thereby detecting the display device <NUM>. The AR device <NUM> may detect the display device <NUM> using a sensor such as an ultrasonic sensor or a LiDAR sensor.

The AR device <NUM> may connect to the display device <NUM> in operation <NUM>. The AR device <NUM> may be connected to the display device <NUM> through a wireless communication network. The wireless communication network connecting the AR device <NUM> and the display device <NUM> may be a short-range wireless network (e.g., the first network <NUM> in <FIG>). The AR device <NUM> and the display device <NUM> may be connected using a wireless communication scheme such as WiFi-P2P, Bluetooth, and BLE (Bluetooth low energy), but the wireless communication scheme is not limited to the above-described examples.

The AR device <NUM> may obtain information related to the display device <NUM> in operation <NUM>. The information related to the display device <NUM> may include image information that includes an image including the size, color, and shape of the display device <NUM>, position information, and depth information. The AR device <NUM> may obtain information related to the display device <NUM> using a sensor or a camera. For example, the AR device <NUM> may identify the display device <NUM> from an image obtained through a camera and obtain size and position information and depth information of the display device <NUM> from the image. For example, the AR device <NUM> may identify the position and size of the display device <NUM> and calculate the distance thereof using a sensor such as an ultrasonic sensor or a LiDAR sensor.

The AR device <NUM> may obtain reproduced content-related information from the display device <NUM> in operation <NUM>. The content-related information may include, for example, related application information, information about the name, capacity, and version of the corresponding content, reproduction information thereof, and information for obtaining the content. The information for obtaining the content may include information for connection with the server <NUM> that provides the content and/or information on an address in which the content is stored. The content reproduction information may include, for example, audio volume level information according to reproduction of the content in the display device <NUM>, reproduction-related information including the reproduction position such as current reproduction frame information, and, in the case of a web page, the address and the reproduction position or time thereof.

The AR device <NUM> may obtain an image including the display device <NUM> reproducing the content in operation <NUM>. For example, the AR device <NUM> may photograph an image including the entirety of the current viewing range of the AR device <NUM> through the camera.

The AR device <NUM> may identify whether or not the display device <NUM> has left the viewing range in operation <NUM>. That is, the AR device <NUM> may identify whether or not the display device <NUM> is outside of a range of the camera and/or the AR device <NUM>. Here, the viewing range may indicate a specified range detected by the sensor of the AR device <NUM>. For example, the AR device <NUM> may obtain movement information of the AR device <NUM> using a sensor, if the movement of the AR device <NUM> is detected, photograph an image within the viewing range of the AR device <NUM> using the camera, and if the display device <NUM> is not identified in the image, determine that the display device <NUM> has left the viewing range. For example, the AR device <NUM> may photograph a two-dimensional image and compare position information of the two-dimensional image with an FoV of the AR device <NUM>, thereby identifying whether or not the two-dimensional image falls outside of the viewing range of the AR device <NUM>. In addition, the AR device <NUM> may be configured to compare position information of the display device <NUM> with position information of the two-dimensional image and, based on whether or not the two-dimensional image is covered by the display device <NUM>, identify whether or not the two-dimensional image falls outside of the viewing range of the AR device <NUM>.

Referring to <FIG>, the viewing range of the AR device <NUM> may be a range that the user wearing the AR device <NUM> is able to recognize through the AR device. For example, the viewing range may be an FoV of the AR device. The AR device may identify the FoV of the AR device, based on information on a photographed image and/or movement information of the AR device.

According to various embodiments, the AR device <NUM> may model a virtual space, based on the information on the photographed image, and identify, as an FoV, a range <NUM> between the recognizable leftmost gaze boundary <NUM> and the recognizable rightmost gaze boundary <NUM> of a virtual camera <NUM> at a corresponding position and posture in the virtual space modeled using movement information of the AR device in the virtual space. In this case, the terminal device may compare the position of an object <NUM> with the FoV <NUM> to determine whether or not the object falls outside of the viewing range of the AR device. In addition, if the position or posture of the virtual camera <NUM> in the virtual space changes to the position or posture of the virtual camera <NUM>, a range <NUM> between the recognizable leftmost gaze boundary <NUM> and the recognizable rightmost gaze boundary <NUM> at a corresponding position and posture may be identified as an FoV. In this case, the terminal device may compare the positions of objects <NUM> and <NUM> with the fields of view <NUM> and <NUM> to determine whether or not the objects fall outside of the viewing range of the AR device. For example, the AR device, if the object <NUM> is included in the FoV <NUM> that the camera <NUM> in the virtual space may have, may determine that the object does not fall outside of the viewing range, or, if the object <NUM> is not included in the FoV <NUM> that the camera <NUM> in the virtual space may have, determine that the object falls outside of the viewing range. If the AR device <NUM> identifies that the display device <NUM> has not left the viewing range, the process may end.

Referring back to <FIG>, if the display device <NUM> has left the viewing range, the AR device <NUM> may produce a virtual object, based on information related to the display device <NUM>, in operation <NUM> and output the same through a display (e.g., the display module <NUM> in <FIG>). For example, the AR device <NUM> may produce a virtual object representing the display device <NUM>, based on image information including the size, position information, depth information, and illuminance information of the display device <NUM>.

For example, the AR device <NUM> may display a virtual object replacing the display device <NUM> according to the size and depth information of the display device <NUM> to have the same size and depth within the viewing range after movement, thereby reducing dizziness due to the difference in depth information.

The AR device <NUM> may display the virtual object at the same position within the viewing range after movement according to the position information in the previous viewing range of the display device <NUM>. However, if another object (e.g., a moving person or a configured object of interest) is recognized at the corresponding position, the virtual object may be moved to another position and then displayed.

The AR device <NUM> may obtain content, based on content-related information, in operation <NUM> and reproduce the content through the output virtual object of the display, based on content reproduction information. The AR device <NUM> may receive content from a server (e.g., the server <NUM> in <FIG>) or the display device <NUM>, based on the content-related information. For example, a virtual object produced based on image information of the display device <NUM> may correspond to information such as the size, position, depth, and color of the display device <NUM>, and the obtained content may be output through a content reproduction area of the virtual object in the shape of the display device <NUM>.

The AR device <NUM> may reproduce the content subsequent to the content that was reproduced in the display device <NUM> through the virtual object, based on reproduction information among the content-related information. In addition, based on the reproduction information, the audio volume of the reproduced content may be configured to be the same as the audio volume reproduced in the display device <NUM>.

After the display device <NUM> has left the viewing range, the AR device <NUM> may output an image including the display device <NUM> that reproduces the content through the display at least until the virtual object is produced and output. If the content starts to be reproduced, the AR device <NUM> may stop outputting an image including the display device <NUM> photographed by the camera.

Referring to <FIG>, <FIG>, and <FIG>, in <FIG>, <FIG>, and <FIG>, the AR device <NUM> in the state of being worn on the user may restrict displaying of other virtual objects in the area of the display device <NUM> such that the light emitted from the area of the display device <NUM> is able to pass through the glasses of the AR device <NUM>, so that the user may watch the content (e.g., an image) <NUM> displayed in the display device <NUM>.

The AR device <NUM> may recognize the display device <NUM> together with a plurality of real objects within a viewing range <NUM> between a recognizable leftmost gaze boundary <NUM> and the recognizable rightmost gaze boundary <NUM>.

Thereafter, a viewing range <NUM> may change due to movement of the AR device <NUM> as shown in <FIG>, <FIG>, and <FIG> so that the display device <NUM> may leave the range between a leftmost gaze boundary <NUM> and a rightmost gaze boundary <NUM>. That is, the AR device <NUM> may fail to identify the display device <NUM> in the two-dimensional image <NUM> photographed by the camera within the viewing range <NUM>.

As shown in <FIG>, <FIG> and <FIG>, the AR device <NUM> may produce and display a virtual object <NUM>, based on information related to the display device <NUM>, and may display content <NUM> subsequent to the content <NUM> that was previously reproduced in the display device <NUM> within the area of the displayed virtual object <NUM>, based on the content related information.

The AR device <NUM> may display the virtual object <NUM>, based on related information of the display device <NUM>. For example, the AR device <NUM> may display a virtual object replacing the display device <NUM> according to the size and depth information of the display device <NUM> to have the same size and depth, thereby reducing dizziness due to the difference in depth information.

The AR device <NUM>, according to the position information of the display device <NUM> in the previous viewing range <NUM>, may display the virtual object at the same position of the two-dimensional image <NUM> photographed by the camera within the viewing range after movement, for example, within the viewing range <NUM>. However, if another object (e.g., a moving person or a configured object of interest) is recognized at the corresponding position, the virtual object may be moved to another position and then displayed.

The AR device <NUM> may apply reproduction information of the content <NUM> in the display device <NUM> to the content <NUM> reproduced in the area of the virtual object <NUM>. For example, the content <NUM> reproduced in the area of the virtual object <NUM> may be the same content as the content <NUM> reproduced in the display device <NUM>, and may be continuously reproduced subsequent to the last content reproduced in the display device <NUM>. In addition, the audio volume level of the content <NUM> may be adjusted to be the same as the audio volume level reproduced in the display device <NUM>.

Referring to <FIG>, the AR device <NUM> may display, as a virtual object, a control panel <NUM> for controlling the audio volume of the display device <NUM> on the right side of the display device <NUM> in (a). In (b) after switching the viewing range due to the movement, the AR device <NUM> may display a virtual object <NUM> that replaces the display device <NUM> and reproduce the content <NUM>, and at the same time, may display, as a virtual object, a control panel <NUM> for controlling the audio volume of the content <NUM> on the right side of the virtual object <NUM>. The display of the virtual object <NUM> for audio volume control is an example, and the AR device <NUM> may display, in addition to the audio volume, a variety of additional information related to the reproduced content or the display device <NUM> as a virtual object. For example, a variety of additional information displayed as a virtual object may include text related to content, subtitles in the case where content has an audio caption (subtitles), and text converted from audio data included in content.

Referring to the example in <FIG>, in the case where the display device <NUM> provides an audio caption (subtitles) <NUM> for content displayed in the display device <NUM> in (a), the virtual object <NUM> replacing the display device <NUM> may be displayed and the content <NUM> is reproduced in (b), and at the same time, additional information related to the content <NUM>, for example, an audio caption (subtitles) <NUM> may be display, as an individual virtual object, below the virtual object <NUM>. For example, even if the display device <NUM> does not provide audio captions (subtitles), the voice of the content may be converted into text (e.g., speech to text), thereby providing a virtual object <NUM> including the converted text.

The AR device <NUM> may display an image that is photographed at the time at which the content <NUM> was previously reproduced in the display device <NUM> until the content <NUM> is reproduced in the area of the virtual object <NUM> after the display device <NUM> has left the viewing range. The AR device <NUM> may stop outputting the image if the content <NUM> starts to be reproduced in the area of the virtual object <NUM>. Therefore, even if the display device <NUM> disappears from the viewing range, the user may recognize that the content <NUM> reproduced in the display device <NUM> naturally switches to the virtual object until the content <NUM> is reproduced through the virtual object <NUM>, which enables seamless switching without overlapping between the real object and the virtual object.

If the content <NUM> starts to be reproduced in the area of the virtual object <NUM> after the display device <NUM> has left the viewing range, the AR device <NUM> may mute the display device <NUM>. If the content <NUM> starts to be reproduced in the area of the virtual object <NUM>, the AR device <NUM> may stop or terminate reproduction of the content <NUM> in the display device <NUM>. If the content <NUM> starts to be reproduced in the area of the virtual object <NUM>, the AR device <NUM> may switch the operation state of the display device <NUM> to an inactive state (e.g., sleep) or turn off the power.

<FIG> is a flowchart illustrating the operation of an AR device according to an embodiment.

According to an embodiment, the AR device <NUM> may provide content through a display (e.g., the display module <NUM> in <FIG>), as an AR service, in operation <NUM>. For example, the AR device <NUM> may reproduce an image.

The AR device <NUM> may discover an external display device (e.g., the display device <NUM> in <FIG>) and connect with the same in operation <NUM>. The AR device <NUM> may photograph an image including real objects within an FoV through a camera (e.g., the camera <NUM> in <FIG>), and the processor may identify whether or not the display device <NUM> is present in the image, thereby detecting the display device <NUM>. For example, while the user wears the AR device <NUM> and views an image reproduced through the display, the display device <NUM> entering the viewing range due to movement of the AR device <NUM> may be detected.

The AR device <NUM> may establish a communication connection with the discovered display device <NUM>. The AR device <NUM> may be connected to the display device <NUM> through a wireless communication network. The wireless communication network connecting the AR device <NUM> and the display device <NUM> may be a short-range wireless network (e.g., the first network <NUM> in <FIG>). The AR device <NUM> and the display device <NUM> may be connected using a wireless communication scheme such as WiFi-P2P, Bluetooth, and BLE, but the wireless communication scheme is not limited to the above-described examples.

In operation <NUM>, the AR device <NUM> may identify whether or not to reproduce the content, which was previously reproduced in the AR device <NUM>, in the display device <NUM>. To this end, the AR device <NUM> may display, for example, a pop-up window asking whether or not to reproduce the content in the display device <NUM>. The AR device <NUM> may receive an input for reproducing the content in the display device <NUM> according to a user's selection. When the AR device <NUM> identifies that the content is not to be reproduced in the external display device, the process may end.

In the case where the content is to be reproduced in the display device <NUM>, the AR device <NUM> may transmit reproduced content-related information to the display device <NUM> in operation <NUM>. The content-related information may include, for example, related application information, name, capacity, and version information of corresponding content, content reproduction information, and information for obtaining corresponding content. The information for obtaining the content may include information for connection with the server <NUM> providing the content and/or information on the address in which the content is stored. The content reproduction information may include, for example, audio volume level information according to reproduction of the content in the display device <NUM>, reproduction-related information including the reproduction position such as information on a frame currently being reproduced, and, in the case of a web page, the address and the reproduction position or time thereof.

The AR device <NUM> may identify that the content is reproduced in the display device <NUM> in operation <NUM>. The AR device <NUM> may identify whether or not the content starts to be reproduced by photographing an image including the display device <NUM> through the camera or communicating with the display device <NUM>.

As the display device <NUM> starts to reproduce the content, the AR device <NUM> may terminate reproduction of the content through the display in operation <NUM>. Accordingly, the user may watch the content, which is reproduced in the display device <NUM>, through the AR device <NUM>.

Claim 1:
An electronic device comprising:
at least one camera (<NUM>);
a communication circuit (<NUM>, <NUM>); and
at least one processor (<NUM>, <NUM>) configured to:
receive, by the communication circuit (<NUM>, <NUM>), information related to content reproduced by an external display device (<NUM>),
obtain, by the at least one camera (<NUM>), a first image comprising the external display device (<NUM>),
based on the external display device (<NUM>) being identified to be outside of an identifiable range of the at least one camera (<NUM>), generate a virtual object corresponding to the external display device (<NUM>).
obtain the content reproduced by the external display device (<NUM>), based on the information related to the content reproduced by the external display device (<NUM>), and
reproduce the obtained content within an area of the virtual object;
characterised in that the virtual object is generated based on information about the external display device (<NUM>).