Patent Description:
As an AR device, such as a wearable device or a head mounted display (HMD) is developed technologically, a user may experience AR. For example, the AR device may display a user interface (UI), such as an icon or an execution screen of an application in a real world to which the user belongs.

No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with regard to the disclosure <CIT> describes a method of initiating and conducting voice communication with a contact.

<CIT> describe controls and interfaces for user interactions in virtual spaces.

To experience an augmented reality (AR), a user needs to wear an AR device. Even while wearing the AR device, the user needs to launch an application of a user device, such as a smartphone. For example, the user needs to communicate with family, friends, or acquaintances by launching a message application installed in the user device while the user wears the AR device.

Therefore, a need exists for an AR device including a display, a communication circuit, and at least one processor operatively connected to the display and the communication circuit, and a memory operatively connected to the at least one processor.

Accordingly, an aspect of the disclosure is to provide a technology for communication in AR.

In accordance with an aspect of the disclosure, an AR device supporting an AR is provided. The AR device includes a display, a communication circuit, at least one processor operatively connected to the display and the communication circuit, and a memory operatively connected to the at least one processor. The memory may store instructions that, when executed, cause the at least one processor to establish a connection with a user device storing a contact application and a message application, through the communication circuit, to detect that the message application is executed, and to display a first graphic user interface (GUI) and at least one avatar, which is disposed at a location adjacent to the first GUI and corresponds to at least one contact associated with the contact application or the message application, through the display in the AR.

In accordance with another aspect of the disclosure, an AR system supporting an AR is provided. The AR system includes an AR device and a user device. The AR device may include a first display and a first communication circuit. The user device may include a second display, a second communication circuit, at least one processor operatively connected to the second display and the second communication circuit, and a memory operatively connected to the at least one processor. The memory may store at least one contact associated with a message application or a contact application. The memory may store instructions that, when executed, cause the at least one processor to establish a connection with the AR device through the second communication circuit, to detect that the message application is executed, to render a first GUI and at least one avatar, which are to be displayed in the AR through the first display, as a <NUM> dimensional (3D) image, and to transmit the rendered first GUI and the rendered at least one avatar to the AR device through the second communication circuit. The at least one avatar may correspond to the at least one contact.

According to the embodiments disclosed in this specification, an AR device may provide an environment in which a user is capable of naturally exchanging a message with a counterpart in an AR.

According to the embodiments disclosed in this specification, the AR device may provide an environment in which the user is capable of utilizing a message application of a user device without restriction of actions even while the user wears the AR device.

According to the embodiments disclosed in this specification, the AR device may display an execution screen of an application stored in the user device in a three-dimensional space.

Besides, a variety of effects directly or indirectly understood through the specification may be provided.

Accordingly, those of ordinary sill in the art will recognize that various changes and modifications of the various embodiments described herein can be made.

<FIG> illustrates a GUI and an avatar, which are displayed in an AR according to an embodiment of the disclosure.

Referring to <FIG>, in a real space <NUM>, an AR device <NUM> may be worn on a part (e.g., a head) of a body of a user <NUM>. The AR device <NUM> may refer to a device including a display capable of supporting an AR. For example, the AR device <NUM> may include at least one of a wearable device and a head mounted display (HMD).

According to an embodiment of the disclosure, the AR device <NUM> may output a first GUI <NUM> for displaying a contact stored in a message application or a contact application of a user device (e.g., a smartphone, not illustrated) in <NUM> dimensional (3D) in an AR <NUM>. For example, the first GUI <NUM> may be referred to as a 'nest'.

According to an embodiment of the disclosure, the first GUI <NUM> may be fixed in a specified area of the real space <NUM>. For example, when a user input for placing the first GUI <NUM> on a table <NUM> of the real space <NUM> is received, the AR device <NUM> may display the first GUI <NUM> on the table <NUM>. In this case, even though the AR device <NUM> moves by a movement of the user <NUM>, a location of the first GUI <NUM> may not be changed in the real space <NUM>. According to another embodiment of the disclosure, the first GUI <NUM> may be fixed to a specified location of an area (e.g., a viewing area <NUM> of <FIG>) that the user <NUM> is capable of viewing through the AR device <NUM>. In this case, when the AR device <NUM> moves by the movement of the user <NUM>, the first GUI <NUM> may move together with the movement of the AR device <NUM>.

The first GUI <NUM> may include at least one avatar (e.g., <NUM>, <NUM>, or <NUM>) corresponding to each contact stored in the user device. In this specification, it may be described that the first GUI <NUM> 'includes' an avatar. However, according to various embodiments of the disclosure, the avatars <NUM>, <NUM>, and <NUM> in the AR <NUM> may be positioned at various locations adjacent to the first GUI <NUM>. For example, referring to an enlarged view <NUM> in which the AR <NUM> is enlarged, the avatars <NUM>, <NUM>, and <NUM> may be disposed on the first GUI <NUM>. As another example, although not illustrated in <FIG>, the avatars <NUM>, <NUM>, and <NUM> may be disposed below, side, or inside the first GUI <NUM>.

The avatars <NUM>, <NUM>, and <NUM> may represent contacts stored in the user device, respectively. According to embodiments of the disclosure, a user of the user device (or the AR device <NUM>) may select one of template avatars that have been stored in advance, and thus an appearance of the avatars <NUM>, <NUM>, and <NUM> may be determined, an appearance of the avatars <NUM>, <NUM>, and <NUM> may be arbitrarily determined based on information (e.g., an age, a gender, or a country) included in contact information. Alternatively, an appearance of the avatars <NUM>, <NUM>, and <NUM> may be determined based on settings of a counterpart.

A size of the AR space <NUM> capable of being provided by the AR device <NUM> is restricted as compared to the number of contacts stored in the user device, and thus the AR device <NUM> according to an embodiment may display only an avatar corresponding to at least one pre-selected contact among contacts stored in the user device. For example, the AR device <NUM> may display, on the first GUI <NUM>, a contact (e.g., "favorite") selected by user settings or a contact having a high frequency of communication.

<FIG> illustrates a block diagram illustrating an electronic device in a network environment <NUM> according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> in the network environment <NUM> may communicate with an external electronic device <NUM> via a first network <NUM> (e.g., a short-range wireless communication network), or at least one of an external electronic device <NUM> or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network). According to an embodiment of the disclosure, the electronic device <NUM> may communicate with the external electronic device <NUM> via the server <NUM>. According to an embodiment of the disclosure, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input module <NUM>, a sound output module <NUM>, a display module <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a connecting terminal <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module (SIM) <NUM>, or an antenna module <NUM>. In an embodiment of the disclosure, at least one of the components (e.g., the connecting terminal <NUM>) may be omitted from the electronic device <NUM>, or one or more other components may be added in the electronic device <NUM>. In an embodiment of the disclosure, some of the components (e.g., the sensor module <NUM>, the camera module <NUM>, or the antenna module <NUM>) may be implemented as a single component (e.g., the display module <NUM>).

According to one embodiment of the disclosure, as at least part of the data processing or computation, the processor <NUM> may store a command or data received from another component (e.g., the sensor module <NUM> or the communication module <NUM>) in a volatile memory <NUM>, process the command or the data stored in the volatile memory <NUM>, and store resulting data in a non-volatile memory <NUM>. According to an embodiment of the disclosure, the processor <NUM> may include a main processor <NUM> (e.g., a central processing unit (CPU) or an application processor (AP)), or an auxiliary processor <NUM> (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor <NUM>.

The auxiliary processor <NUM> may control at least some of functions or states related to at least one component (e.g., the display module <NUM>, the sensor module <NUM>, or the communication module <NUM>) among the components of the electronic device <NUM>, instead of the main processor <NUM> while the main processor <NUM> is in an inactive (e.g., a sleep) state, or together with the main processor <NUM> while the main processor <NUM> is in an active state (e.g., executing an application). According to an embodiment of the disclosure, the auxiliary processor <NUM> (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module <NUM> or the communication module <NUM>) functionally related to the auxiliary processor <NUM>. According to an embodiment of the disclosure, the auxiliary processor <NUM> (e.g., the neural processing unit) may include a hardware structure specified for artificial intelligence model processing.

According to an embodiment of the disclosure, the receiver may be implemented as separate from, or as part of the speaker.

According to an embodiment of the disclosure, the display module <NUM> may include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the intensity of force incurred by the touch.

According to an embodiment of the disclosure, the audio module <NUM> may obtain the sound via the input module <NUM>, or output the sound via the sound output module <NUM> or a headphone of an external electronic device (e.g., the external electronic device <NUM>) directly (e.g., wiredly) or wirelessly coupled with the electronic device <NUM>.

According to an embodiment of the disclosure, the sensor module <NUM> may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface <NUM> may support one or more specified protocols to be used for the electronic device <NUM> to be coupled with the external electronic device (e.g., the external electronic device <NUM>) directly (e.g., wiredly) or wirelessly. According to an embodiment of the disclosure, the interface <NUM> may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal <NUM> may include a connector via which the electronic device <NUM> may be physically connected with the external electronic device (e.g., the external electronic device <NUM>). According to an embodiment of the disclosure, the connecting terminal <NUM> may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

According to an embodiment of the disclosure, the haptic module <NUM> may include, for example, a motor, a piezoelectric element, or an electric stimulator.

According to an embodiment of the disclosure, the camera module <NUM> may include one or more lenses, image sensors, image signal processors, or flashes.

According to one embodiment of the disclosure, the power management module <NUM> may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

According to an embodiment of the disclosure, the battery <NUM> may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module <NUM> may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device <NUM> and the external electronic device (e.g., the external electronic device <NUM>, the external electronic device <NUM>, or the server <NUM>) and performing communication via the established communication channel. According to an embodiment of the disclosure, the communication module <NUM> may include a wireless communication module <NUM> (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module <NUM> (e.g., a local area network (LAN) communication module or a power line communication (PLC) module).

The wireless communication module <NUM> may support various requirements specified in the electronic device <NUM>, an external electronic device (e.g., the external electronic device <NUM>), or a network system (e.g., the second network <NUM>). According to an embodiment of the disclosure, the wireless communication module <NUM> may support a peak data rate (e.g., 20Gbps or more) for implementing eMBB, loss coverage (e.g., 164dB or less) for implementing mMTC, or U-plane latency (e.g., <NUM> or less for each of downlink (DL) and uplink (UL), or a round trip of <NUM> or less) for implementing URLLC.

According to an embodiment of the disclosure, the antenna module <NUM> may include an antenna including a radiating element including a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment of the disclosure, the antenna module <NUM> may include a plurality of antennas (e.g., array antennas). According to an embodiment of the disclosure, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module <NUM>.

According to various embodiments of the disclosure, the antenna module <NUM> may form an mmWave antenna module. According to an embodiment of the disclosure, the mmWave antenna module may include a printed circuit board, a RFIC disposed on a first surface (e.g., the bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high-frequency band (e.g., the mmWave band), and a plurality of antennas (e.g., array antennas) disposed on a second surface (e.g., the top or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high-frequency band.

According to an embodiment of the disclosure, commands or data may be transmitted or received between the electronic device <NUM> and the external electronic device <NUM> via the server <NUM> coupled with the second network <NUM>. Each of the external electronic devices <NUM> or <NUM> may be a device of a same type as, or a different type, from the electronic device <NUM>. According to an embodiment of the disclosure, all or some of operations to be executed at the electronic device <NUM> may be executed at one or more of the external electronic devices <NUM>, <NUM>, or <NUM>. The electronic device <NUM> may provide ultra-low-latency services using, e.g., distributed computing or mobile edge computing. In another embodiment of the disclosure, the external electronic device <NUM> may include an internet-of-things (IoT) device. According to an embodiment of the disclosure, the external electronic device <NUM> or the server <NUM> may be included in the second network <NUM>.

The electronic device <NUM> illustrated in <FIG> may be the AR device <NUM> of <FIG> or a device (e.g., a user device) that is electrically connected to the AR device <NUM> and stores contacts.

<FIG> illustrates a block diagram of a user device according to an embodiment of the disclosure.

Referring to <FIG>, a user device <NUM> may refer to a device configured to communicate with a counterpart stored in a message application or a contact application. For example, the user device <NUM> may include a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), a moving picture experts group phase <NUM> or phase <NUM> (MPEG-<NUM> or MPEG-<NUM>) audio layer <NUM> (MP3) player, a mobile medical device, or a home appliance.

An AR device <NUM> may be implemented as a wearable device (e.g., a HMD) capable of being worn by a user. The AR device <NUM> may include a display for providing an AR when the user device <NUM> operates in an AR mode. For example, when the user device <NUM> is mounted on the AR device <NUM>, the user device <NUM> may execute the AR mode. As another example, when the user device <NUM> is not mounted on the AR device <NUM>, the user device <NUM> may execute the AR mode depending on user settings or execution of an application associated with the AR mode. When the AR mode is executed, the user device <NUM> may three-dimensionally render a GUI (e.g., the first GUI <NUM> and the avatars <NUM>, <NUM>, and <NUM> of <FIG>) displayed in the AR through the AR device <NUM>. In this case, two screens corresponding to the user's eyes (e.g., a left eye and a right eye) may be output through a display of the AR device <NUM>.

The user device <NUM> may include a touch screen <NUM>, a controller <NUM>, a memory <NUM>, and a communication circuit <NUM>. As another example, the user device <NUM> may further include at least one of other components included in the electronic device <NUM> of <FIG>.

The touch screen <NUM> may perform a function that is identical or similar to the function of the display module <NUM> of <FIG>. The touch screen <NUM> may include a display panel <NUM> and/or a touch panel <NUM>. Each of the display panel <NUM> and the touch panel <NUM> may be implemented as a separate panel. As another example, the touch screen <NUM> may include only the display panel <NUM> without including the touch panel <NUM>.

The controller <NUM> may perform a function that is identical or similar to the function of the processor <NUM> of <FIG>. The controller <NUM> may include an AR mode processing module <NUM>, an event detection module <NUM>, an event information processing module <NUM>, and an application controller <NUM>. Modules included in the controller <NUM> may mean configurations implemented by hardware or software. When implemented in software, the modules included in the controller <NUM> may be stored in the memory <NUM> as an instruction set. The controller <NUM> may perform operations of the user device <NUM> by executing the instruction set.

When the user device <NUM> operates in the AR mode, the AR mode processing module <NUM> may process operations associated with the AR mode. For example, the AR mode processing module <NUM> may load at least one AR program <NUM> stored in the memory <NUM>.

The event detection module <NUM> may detect an event that occurs when the AR mode is operated. The event detection module <NUM> may determine whether there is information to be displayed on a screen associated with an event occurring when the AR mode is operated. For example, when content, such as a message is received from a counterpart's electronic device through a message application, the event detection module <NUM> may detect that an event occurs.

The event information processing module <NUM> may process information that is to be detected by the event detection module <NUM> and is to be displayed in an AR. For example, the event information processing module <NUM> may convert information (e.g., a text message) associated with a detected event into a 3D image to be displayed in the AR. In this case, the event information processing module <NUM> may convert two-dimensional (2D) information into information corresponding to left and right eyes, may synthesize the converted information, and may render the synthesized information to be displayed on the AR device <NUM>.

The application controller <NUM> may control an application when the user device <NUM> operates in the AR mode. For example, when a message application is executed in the AR mode, the application controller <NUM> may block other applications or operate other applications in the background such that other applications do not interfere with execution of the message application.

The memory <NUM> may perform a function that is identical or similar to the function of the memory <NUM> of <FIG>. The memory <NUM> may store the AR program <NUM> and event-related information <NUM>. The AR program <NUM> may be an application associated with an AR mode operation of the user device <NUM>. The event-related information <NUM> may include information associated with an event occurring in the AR.

The communication circuit <NUM> may perform a function that is identical or similar to the function of the communication module <NUM> of <FIG>. The communication circuit <NUM> may be used to be electrically connected to the AR device <NUM> or an external server. For example, the communication circuit <NUM> may be connected to the AR device <NUM> through Bluetooth, Wi-Fi, or tethering. As another example, the communication circuit <NUM> may include a wired interface.

<FIG> is a block diagram of a program module according to an embodiment of the disclosure.

Referring to <FIG>, a program module <NUM> may include a configuration that is identical or similar to that of the program <NUM> of <FIG>. The program module <NUM> may include a system operating system (e.g., an operating system (OS)) <NUM>, a framework <NUM>, and an application <NUM>.

The system operating system <NUM> may perform a function that is identical or similar to the function of the operating system <NUM> of <FIG>. The system operating system <NUM> may include at least one system resource manager or at least one device driver. The system resource manager may control, allocate, or recover a system resource. The system resource manager may include at least one manager, such as a process manager, a memory manager, or a file system manager. The device driver may include at least one driver, such as a display driver, a camera driver, a Bluetooth driver, a shared memory driver, an USB driver, a keypad driver, a Wi-Fi driver, an audio driver, or an inter-process communication (IPC) driver.

For example, the framework <NUM> may include the middleware <NUM> of <FIG>. The framework <NUM> may provide a function necessary for an application through an application programming interface (API) such that the application is capable of efficiently using limited system resources in an electronic device (e.g., the user device <NUM> in <FIG>). When the electronic device operates in an AR mode, the framework <NUM> may control a function associated with the AR mode. For example, the framework <NUM> may control at least one AR application (e.g., a plurality of AR applications <NUM>).

The application <NUM> may perform a function that is identical or similar to the function of the application <NUM> of <FIG>. The application <NUM> may include a plurality of AR applications (e.g., the plurality of AR applications <NUM>) and a plurality of general applications (e.g., <NUM>). The AR application may refer to an application that operates in the AR mode, and, the general application may refer to an application that operates in a normal mode, not the AR mode.

According to an embodiment of the disclosure, the application <NUM> may further include an AR control application <NUM>. At least one AR application and/or at least one general application <NUM> may be controlled by the AR control application <NUM>. For example, the AR control application <NUM> may perform a function that is identical or similar to the function of at least one of the AR mode processing module <NUM>, the event detection module <NUM>, the event information processing module <NUM>, or the application controller <NUM> of <FIG>.

According to an embodiment of the disclosure, when a user device operates in the AR mode or the normal mode, the system operating system <NUM> may notify the framework <NUM> that an event occurs (e.g., receiving a text message from a counterpart's electronic device). When an event occurs in the case where the user device operates in the normal mode, the framework <NUM> may control execution of the general application such that a screen associated with the event is displayed through a display (e.g., the touch screen <NUM> in <FIG>) of the user device. When an event occurs in the case where the user device operates in the AR mode, the framework <NUM> may control the general application such that a screen associated with the event is displayed only through an AR device (e.g., the AR device <NUM> in <FIG>) and is not displayed through a display of the user device, thereby preventing unnecessary resource consumption and preventing the general application from interfering with an operation in the AR mode. According to another embodiment of the disclosure, when an event occurs in the case where the user device operates in the AR mode, the framework <NUM> may provide an environment in which a user is capable of controlling an AR device through the user device, by controlling an event-related screen to be displayed through the display of the user device as well as the AR device.

<FIG> is a block diagram of an AR device according to an embodiment of the disclosure.

Referring to <FIG>, an AR device <NUM> may include a configuration that is identical or similar to that of the AR device <NUM> of <FIG> or the AR device <NUM> of <FIG>, and may perform a function that is identical or similar to that of the AR device <NUM> of <FIG> or the AR device <NUM> of <FIG>. The AR device <NUM> may include a memory <NUM>, an input/output interface <NUM>, a communication interface <NUM>, a sensor <NUM>, a display <NUM>, and a processor <NUM> that are operatively connected through a bus <NUM>.

The memory <NUM> may store a spatial map <NUM>. For example, the spatial map <NUM> may refer to spatial information about a real space (e.g., the real space <NUM> of <FIG>) surrounding the AR device <NUM>. For example, when the AR device <NUM> is worn by a user, the AR device <NUM> may generate a space surrounding the AR device <NUM> as a 3D map by scanning a surrounding area through the sensor <NUM>. The AR device <NUM> may identify a location of the AR device <NUM> by using the generated spatial map <NUM> or may identify a location where a specific GUI (e.g., the first GUI <NUM> of <FIG> or the avatars <NUM>, <NUM>, and <NUM>) is positioned.

The memory <NUM> may include an API <NUM> for providing a necessary function to applications stored in the AR device <NUM>.

The input/output interface <NUM> may include a hardware or software configuration for interfacing with a user. For example, the input/output interface <NUM> may include a microphone or a speaker.

The communication interface <NUM> may be configured to be connected to a user device (e.g., the user device <NUM> in <FIG>) by wire or wirelessly. The communication interface <NUM> may also be referred to as a "communication module" or a "communication circuit".

The sensor <NUM> may include at least one sensor. For example, the sensor <NUM> may include at least one of a hand pose sensor <NUM> capable of sensing a movement of the user's hand (or a finger), a head pose sensor <NUM> capable of sensing a movement of the user's head, or another sensor <NUM>. For example, the other sensor <NUM> may include at least one of a location measurement sensor capable of identifying a location of the AR device <NUM> in a real space, such as a global positioning system (GPS), a motion sensor capable of sensing a direction or movement of the AR device <NUM>, such as an acceleration sensor, a geomagnetic sensor, and a gyro sensor, a proximity sensor capable of sensing an external object positioned at a location close to the AR device <NUM>, a red green blue (RGB) sensor capable of sensing an intensity of light or color around the AR device <NUM>, an illuminance sensor, or a hall sensor.

The display <NUM> may include a see-through display <NUM> to provide an AR to a user.

The processor <NUM> may be operatively connected to the memory <NUM>, the input/ output interface <NUM>, the communication interface <NUM>, the sensor <NUM>, and the display <NUM>, and may perform overall operations of the AR device <NUM>.

<FIG> is a diagram of a message application <NUM> combined with a hardware configuration of an AR system according to an embodiment of the disclosure.

Referring to <FIG>, an AR device <NUM> and a user device <NUM> may be electrically connected to each other. For example, the user device <NUM> may be mounted in the AR device <NUM>, the AR device <NUM> and the user device <NUM> may be connected through a wired interface. Alternatively, the AR device <NUM> and the user device <NUM> may be connected via wireless communication. A processor <NUM> and a memory <NUM> of the AR device <NUM> may perform functions that are identical or similar to functions of the processor <NUM> and the memory <NUM> of <FIG>, respectively. A processor <NUM> and a memory <NUM> of the user device <NUM> may perform functions that are identical or similar to functions of the controller <NUM> and the memory <NUM> of <FIG>, respectively.

The message application <NUM> may be stored in the memory <NUM> of the user device <NUM>. In this case, the message application <NUM> may perform functions that are identical or similar to those of the modules included in the controller <NUM> of <FIG> or the AR control application <NUM> of <FIG>. For example, the message application <NUM> may render at least one avatar <NUM> (e.g., the avatar <NUM>, <NUM>, or <NUM> of <FIG>) and a first GUI <NUM> (e.g., the first GUI <NUM> of <FIG>) that are to be displayed through a see-through display <NUM> (e.g., the display <NUM> of <FIG>) of the AR device <NUM>, and may transmit the rendered first GUI <NUM> and the rendered at least one avatar <NUM> to the AR device <NUM>. When a user input <NUM> to select one of at least one avatar included in the first GUI <NUM> is received, the message application <NUM> may render a second GUI <NUM> and a selected avatar <NUM> such that the selected avatar <NUM> is displayed on the second GUI <NUM> (e.g., a second GUI <NUM> in <FIG>) indicating a message thread. The message application <NUM> may transmit the rendered second GUI <NUM> and the selected avatar <NUM> to the AR device <NUM>. While the second GUI <NUM> is displayed, the message application <NUM> may receive a user input <NUM> to enter content. For example, the content may include at least one of a text, an image, 3D content, or a voice.

The AR device <NUM> may output the first GUI <NUM>, at least one avatar <NUM>, the selected avatar <NUM>, and the second GUI <NUM>, which are rendered through the message application <NUM>, through the see-through display <NUM>. According to an embodiment of the disclosure, the AR device <NUM> may obtain the user input <NUM> to select an avatar and the user input <NUM> to enter content, through a sensor (e.g., the sensor <NUM> in <FIG>).

<FIG> illustrates an operation flowchart <NUM> of an AR system for displaying a first GUI and an avatar according to an embodiment of the disclosure. An AR system may refer to a system including an AR device (e.g., the AR device <NUM> of <FIG>) and a user device (e.g., the user device <NUM> of <FIG>).

Referring to <FIG>, in operation <NUM>, the AR system may establish a connection between the AR device and the user device. The connection between the AR device and the user device may be established by wire or wirelessly. For example, when the AR device is implemented with an HMD, the user device may be mounted in the AR device. As another example, the AR device and the user device may be tethered through a wired interface or a wireless communication protocol (e.g., Bluetooth or Wi-Fi).

In operation <NUM>, the AR system may detect that a message application of the user device is executed while the connection between the AR device and the user device is established. For example, the message application may include an application capable of communicating with another user device. For example, the message application may be replaced with a chat application, a video call application, a mail application, or a social network service (SNS) application.

In operation <NUM>, the AR system may display a first GUI (e.g., GUI <NUM> of <FIG>) including at least one avatar (e.g., the avatar <NUM>, <NUM>, or <NUM> of <FIG>) in an AR. The first GUI may refer to an object in which a space in which at least one avatar is positioned is visually displayed in the AR. The at least one avatar may be based on at least one contact stored in the executed message application or a contact application linked to the message application.

<FIG> illustrates a first GUI including an avatar according to an embodiment of the disclosure.

Referring to <FIG>, an AR device may display a first GUI <NUM> in an AR. A shape of the first GUI <NUM> is not limited to an example shown in <FIG>, and, a shape of the first GUI <NUM> may be changed by user settings. The first GUI <NUM> may be fixed to a specified location in a real space or may be fixed to a specified area of a display of an AR device. According to an embodiment of the disclosure, the AR device may display at least one avatar <NUM>, <NUM>, or <NUM> at a location (e.g., a top surface of the first GUI) adjacent to the first GUI <NUM>. The AR device may display the specified number of avatars on the first GUI <NUM> based on a size of the first GUI <NUM>. For example, the AR device may display, on the first GUI <NUM>, a contact, which is specified as a favorite depending on user settings, or a contact having a high frequency of contact.

According to an embodiment of the disclosure, the AR device may display icons <NUM>, <NUM>, and <NUM> of a menu for controlling the first GUI <NUM> or the at least one avatar <NUM>, <NUM>, or <NUM> at a location adjacent to the first GUI <NUM>. For example, the menu corresponding to the icons <NUM>, <NUM>, and <NUM> may include at least one of editing of the first GUI <NUM>, favorite settings, editing of an avatar, checking of recent communication, or editing of a contact.

<FIG> illustrates an avatar corresponding to a contact according to an embodiment of the disclosure.

Referring to <FIG>, an AR device may edit at least one avatar <NUM>, <NUM>, or <NUM> or contact information <NUM>, <NUM>, or <NUM> respectively corresponding to the at least one avatar <NUM>, <NUM>, or <NUM>. For example, when an avatar of a contact stored in a user device has not been generated yet, the AR device may apply one of the pre-stored template avatars to a contact by a user's selection. When a counterpart of a contact, to which a template avatar is applied, generates a new avatar, an avatar stored in the user device may be changed from the template avatar to an avatar generated by the counterpart.

<FIG> illustrates a graphic effect of moving an avatar from a first GUI to a second GUI in an AR according to an embodiment of the disclosure.

Referring to <FIG>, an AR device (e.g., the AR device <NUM> of <FIG>) may display a second GUI <NUM> in an AR in addition to a first GUI <NUM>. The first GUI <NUM> may correspond to the first GUI <NUM> of <FIG> or the first GUI <NUM> of <FIG>. The second GUI <NUM> may indicate a message thread of a message application. The second GUI <NUM> may be referred to as a 'nest'. In this case, the first GUI <NUM> and the second GUI <NUM> may be referred to as a 'main nest' and a 'secondary nest', respectively.

According to an embodiment of the disclosure, the second GUI <NUM> may include a first avatar <NUM> corresponding to a user account of an AR system. It may be described that the second GUI <NUM> 'includes' the first avatar <NUM>. However, according to various embodiments of the disclosure, the AR device may place the first avatar <NUM> at a location (e.g., top, bottom, side, or inside) adjacent to the second GUI <NUM>.

According to an embodiment of the disclosure, the AR device may detect a user gesture <NUM> of moving <NUM> a second avatar <NUM> to the second GUI <NUM> among avatars <NUM> and <NUM> included in the first GUI <NUM>. For example, the AR device may detect a user gesture <NUM> of grabbing <NUM> the second avatar <NUM> from the first GUI <NUM> through a hand pose sensor (e.g., the hand pose sensor <NUM> in <FIG>), and dropping the grabbed second avatar <NUM> on the second GUI <NUM>. The AR device may provide a graphic effect of moving the second avatar <NUM> from the first GUI <NUM> to the second GUI <NUM> depending on the detected user gesture <NUM> in the AR. When the second avatar <NUM> is located on the second GUI <NUM>, a user device electrically connected to the AR device may communicate with a counterpart of a contact corresponding to the second avatar <NUM>.

According to an embodiment of the disclosure, the second GUI <NUM> may be displayed before the second avatar <NUM> is selected, the second avatar <NUM> may be displayed in response to detecting a user gesture <NUM> that deviates from the first GUI <NUM>.

According to an embodiment of the disclosure, to provide a sense of reality, such as a talking between the first avatar <NUM> and the second avatar <NUM>, an AR device may change locations and directions of the first avatar <NUM> and the second avatar <NUM>. For example, the AR device may be arranged such that the first avatar <NUM> and the second avatar <NUM> stand while facing each other on opposite sides of the second GUI <NUM>.

According to an embodiment of the disclosure, the AR device may display icons <NUM> and <NUM> of menus for controlling a message thread indicated by the second GUI <NUM> at a location adjacent to the second GUI <NUM>. For example, the menu corresponding to the icons <NUM> and <NUM> may include at least one of a function to terminate the message thread, a function to share content (e.g., at least one of an image, a video, a link, a map, another person's contact, a music, a file, a voice message, or a schedule), or a function to send an emoticon.

According to an embodiment of the disclosure, the AR device may terminate communication with a counterpart of a contact indicated by the second avatar <NUM>, in response to detecting a user gesture <NUM> of moving <NUM> the second avatar <NUM> from the second GUI <NUM> to the first GUI <NUM>.

<FIG> and <FIG> illustrate an operation of detecting a gesture for an avatar according to various embodiments of the disclosure. <FIG> and <FIG> illustrate an operation of detecting a movement of a user's finger (or a hand) by using a hand pose sensor.

Referring to <FIG> and <FIG>, in operation <NUM>, an AR device may detect that a user's finger <NUM> moves toward an avatar <NUM> disposed on a first GUI <NUM> (e.g., the first GUI <NUM> in <FIG>). The AR device may not only detect the movement of the user's finger <NUM>, but may also measure a distance <NUM> between the user's finger <NUM> and the avatar <NUM>.

When the distance <NUM> between the user's finger <NUM> and the avatar <NUM> is less than a threshold value, the AR device may determine that the avatar <NUM> is selected by the user. According to an embodiment of the disclosure, to notify the user that the distance <NUM> between the user's finger <NUM> and the avatar <NUM> is getting closer, the AR device may provide a graphic effect indicating a gesture of the avatar <NUM>. For example, referring to operation <NUM> of <FIG>, the AR device may provide a graphic effect in which the avatar <NUM> raises a hand in a direction, in which the user's finger <NUM> is located, in response to detecting that the distance <NUM> between the user's finger <NUM> and the avatar <NUM> is less than the threshold value. According to an embodiment of the disclosure, to guide a location where the user's finger <NUM> is capable of grabbing the avatar <NUM>, the AR device may output an additional graphic effect (e.g., light) on a hand raised by the avatar <NUM>. For another example, referring to <FIG>, in operation <NUM>, the AR device may provide a visual graphic effect <NUM> (e.g., firecrackers or pollen) at a periphery of the avatar <NUM> in response to an event that the user's finger <NUM> touches a hand of the avatar <NUM>.

In operation <NUM>, the AR device may detect a gesture in which the user's finger <NUM> grabs the avatar <NUM>. For example, when the user's finger <NUM> touches the avatar <NUM> or the distance between the user's finger <NUM> and the avatar <NUM> is less than the threshold value, the AR device may detect a movement in which the user's finger <NUM> is closed. The AR device may provide a graphic effect in which the avatar <NUM> deviates from the first GUI <NUM>, in response to detecting a gesture in which the user's finger <NUM> grabs the avatar <NUM>. The AR device may provide a graphic effect in which the avatar <NUM> moves together with the user's finger <NUM> depending on the movement of the closed user's finger <NUM>.

In operation <NUM>, the AR device may display the avatar <NUM> on a second GUI <NUM> in response to detecting that the user's finger <NUM> is opened or detecting that the avatar <NUM> is located at a location adjacent to the second GUI <NUM> (e.g., the second GUI <NUM> in <FIG>). The second GUI <NUM> may include an avatar <NUM> (e.g., the first avatar <NUM> of <FIG>) corresponding to a user account of the AR device.

<FIG> illustrates an embodiment in which the avatar <NUM> moves from the first GUI <NUM> to the second GUI <NUM>. However, the same principle may be applied to an embodiment in which the avatar <NUM> moves from the second GUI <NUM> to the first GUI <NUM>.

<FIG> is an operation flowchart <NUM> of an AR system for providing a graphic effect in which an avatar moves according to an embodiment of the disclosure.

Referring to <FIG>, in operation <NUM>, an AR system may display a second GUI (e.g., the second GUI <NUM> of <FIG>) indicating a message thread in an AR. According to an embodiment of the disclosure, the second GUI may include an avatar (e.g., the first avatar <NUM> of <FIG>) corresponding to a user account of the AR system.

In operation <NUM>, the AR system may detect a first gesture for an avatar (e.g., the second avatar <NUM> of <FIG>) included in the first GUI (e.g., the first GUI <NUM> of <FIG>). For example, the AR system may detect a movement, in which a user's finger moves to a location close to an avatar and the user's finger is closed toward the avatar, using a hand pose sensor. The AR system may detect a movement in which the user's finger moves from the first GUI to the second GUI while the user's finger is closed.

In operation <NUM>, the AR system may provide a graphic effect, in which the avatar moves from the first GUI to the second GUI, based on the detected first gesture.

According to another embodiment of the disclosure, the AR system may display the second GUI in the AR in response to detecting the first gesture in operation <NUM>. For example, when it is detected that the user's finger grabs the avatar, the AR system may display the second GUI.

<FIG> illustrates an operation of changing a message thread according to an embodiment of the disclosure.

Referring to <FIG>, a user device may generate a plurality of message threads for counterparts of a plurality of contacts. An AR device may display a plurality of GUIs <NUM>, <NUM>, <NUM>, and <NUM> indicating the plurality of message threads in an AR. For example, the AR device may display the plurality of GUIs <NUM>, <NUM>, <NUM>, and <NUM> indicating the plurality of message threads at a location adjacent to a second GUI <NUM> (e.g., the second GUI <NUM> of <FIG>) generated between a first avatar <NUM> and a second avatar <NUM>.

According to an embodiment of the disclosure, the AR device may select a current message thread among the plurality of message threads depending on a user input. For example, the AR device may select the current message thread among the plurality of message threads depending on a movement in which a user finger <NUM> scrolls the plurality of GUIs <NUM>, <NUM>, <NUM>, and <NUM> in a specified direction (e.g., <NUM>) or touches one of the plurality of GUIs <NUM>, <NUM>, <NUM>, and <NUM>.

<FIG> and <FIG> illustrate a user interface for entering a message according to various embodiments of the disclosure.

Referring to <FIG>, an AR device may display a second GUI <NUM> (e.g., the second GUI <NUM> of <FIG>) including a first avatar <NUM> (e.g., the first avatar <NUM> in <FIG>) and a second avatar <NUM> (e.g., the second avatar <NUM> in <FIG>). The AR device may display, at a location adjacent to the second GUI <NUM>, icons <NUM>, <NUM>, and <NUM> for selecting the type of a user input to be transmitted to a counterpart of a contact corresponding to the second avatar <NUM> and an icon <NUM> for transmitting a obtained user input to the counterpart of the contact corresponding to the second avatar <NUM>. For example, the type of an input indicated by the icons <NUM>, <NUM>, and <NUM> may include a voice input, a text input, or a reaction input. For example, the reaction input may refer to a gesture that appears on the first avatar <NUM>.

Referring to <FIG>, for example, the AR device may receive a user's voice input <NUM>. The AR device may obtain the voice input <NUM> through a microphone (e.g., the input/output interface <NUM> of <FIG>) of the AR device. As another example, a user device connected to the AR device may obtain the voice input <NUM> through a microphone (e.g., the input module <NUM> in <FIG>) of the user device, and may transmit the obtained voice input <NUM> to the AR device. As another example, the AR device may obtain a text input through a virtual keyboard <NUM> displayed in an AR. In this case, the AR device may obtain a text input on the virtual keyboard <NUM> by detecting a movement of a user's finger using a hand pose sensor. As another example, the AR device may obtain a text input through a virtual keyboard output on a touch screen of a user device <NUM>. As another example, although not shown in <FIG>, the AR device may obtain a text input through an input device (e.g., a Bluetooth keyboard) separately connected to the user device <NUM>.

According to an embodiment of the disclosure, the AR device may obtain a message input through a plurality of input means. For example, when an application of a user device is operated in an AR mode, the AR device may obtain a message input through the virtual keyboard <NUM> displayed in the AR and, at the same time, may obtain a message input through the keyboard output on a touch screen of the user device <NUM>. According to another embodiment of the disclosure, the AR device may set a specific input means as a default means for obtaining a message input. According to another embodiment of the disclosure, the AR device may obtain a message input through an input means that is last used.

According to an embodiment of the disclosure, the AR device may transmit the obtained message (e.g., a voice or a text) indicated by a user input to a counterpart of a contact corresponding to the second avatar <NUM>.

According to an embodiment of the disclosure, the AR device may provide a sense of reality as if a user of the first avatar <NUM> speaks directly to a user of the second avatar <NUM>, by displaying a text corresponding to the transmitted message or a speech bubble <NUM> including a text on the first avatar <NUM>.

<FIG> illustrates a UI for displaying a message according to an embodiment of the disclosure.

Referring to <FIG>, an AR device may display a second GUI <NUM> (e.g., the second GUI <NUM> of <FIG>) including a first avatar <NUM> (e.g., the first avatar <NUM> in <FIG>) and a second avatar <NUM> (e.g., the second avatar <NUM> in <FIG>). The AR device may display a message received from an electronic device of a counterpart or a message transmitted to the electronic device of the counterpart in an AR in various shapes.

For example, in operation <NUM>, the AR device may display a speech bubble <NUM> including a text corresponding to a message in the AR. For example, the speech bubble <NUM> may be displayed on an avatar (e.g., the first avatar <NUM>) corresponding to a subject (e.g., a user of the AR device) that sends a message. According to an embodiment of the disclosure, the AR device may control the speech bubble <NUM> to disappear after a specified time.

For another example, in operation <NUM>, the AR device may display a UI <NUM> indicating a voice message in the AR. When a user input for selecting the UI <NUM> is received, the AR device may output a voice through a speaker (e.g., the input/output interface <NUM> of <FIG>).

According to an embodiment of the disclosure, the AR device may output a message as a voice or a text depending on user settings. For example, even though the AR device obtains a voice message or a voice input, the AR device may display a text corresponding to a voice in the AR through a speech-to-text (STT) technology. On the other hand, even though the AR device obtains a text message or a text input, the AR device may output a voice corresponding to a text in the AR through a text-to-speech (TTS) technology.

For another example, in operation <NUM>, the AR device may display a UI <NUM> indicating message transmission records in the AR. According to an embodiment of the disclosure, the UI <NUM> may be scrolled in a specified direction (e.g., upward or downward) based on a user gesture. According to an embodiment of the disclosure, when a specified time elapses or the number of texts included in the UI1433 is not less than a specified number, the AR device may reduce visual clutter by making at least part of texts included in the UI <NUM> disappear from the AR.

<FIG> illustrates an operation of outputting a message depending on a viewing area according to various embodiments of the disclosure.

Referring to <FIG>, when an AR device <NUM> (e.g., the AR device <NUM> in <FIG>) is worn on a part (e.g., a head) of the body of a user <NUM>, an AR space that the user <NUM> is capable of viewing may be limited due to a physical size limitation of a display (e.g., the display <NUM> of <FIG>) of the AR device <NUM>. In this specification, an area that the user <NUM> is capable of viewing through the AR device <NUM> may be referred to as a 'viewing area (e.g., the viewing area <NUM>)'. For example, when a second GUI <NUM> (e.g., the second GUI <NUM> in <FIG>) including a first avatar <NUM> (e.g., the first avatar <NUM> in <FIG>) and a second avatar <NUM> (e.g., the second avatar <NUM> in <FIG>) is fixed to a specified location in a real space, the second GUI <NUM> may be positioned within the viewing area <NUM> or may deviate from the viewing area <NUM>, depending on a movement of the AR device <NUM>.

According to an embodiment of the disclosure, the AR device <NUM> may output a message as a text or a voice based on whether the second GUI <NUM> deviates from the viewing area <NUM>. For example, when the second GUI <NUM> is located within the viewing area <NUM> as shown in the left diagram of <FIG>, the AR device <NUM> may output a message in a text form (e.g., <NUM>). As another example, when the second GUI <NUM> deviates from the viewing area <NUM> as shown in the right diagram of <FIG>, the AR device <NUM> may output a message as a voice.

<FIG> illustrates an operation flowchart <NUM> of an AR device for outputting a message depending on a viewing area according to an embodiment of the disclosure.

Referring to <FIG>, in operation <NUM>, the AR device <NUM> may detect a movement of the AR device <NUM>. For example, the AR device <NUM> may detect the movement of the AR device <NUM> through a head pose sensor (e.g., the head pose sensor <NUM> in <FIG>) or a motion sensor.

In operation <NUM>, the AR device <NUM> may determine whether the second GUI <NUM> is located within the viewing area <NUM>, in response to an event that the movement is detected.

When the second GUI <NUM> is located within the viewing area <NUM>, in operation <NUM>, the AR device <NUM> may display a text (e.g., <NUM>) corresponding to a message through the display.

When the second GUI <NUM> is not located within the viewing area <NUM>, in operation <NUM>, the AR device <NUM> may output a voice corresponding to a message through a speaker.

<FIG> illustrates a UI for displaying a received message according to an embodiment of the disclosure.

Referring to <FIG>, after a message thread is generated, a user device (e.g., the user device <NUM> in <FIG>) may receive a message from a counterpart of another contact (e.g., a contact corresponding to a third avatar <NUM>) for which the message thread is not generated. To notify a user of a message received from the counterpart for which the message thread is not generated, an AR device (e.g., the AR device <NUM> of <FIG>) may provide a graphic effect indicating a gesture of the third avatar <NUM> included in a first GUI <NUM> (e.g., the first GUI <NUM> of <FIG>). For example, the AR device may provide a graphic effect indicating a gesture in which the third avatar <NUM> raises a hand.

<FIG> illustrates a second GUI including a plurality of avatars in an AR according to an embodiment of the disclosure.

Referring to <FIG>, while a second GUI <NUM> (e.g., the second GUI <NUM> of <FIG>) including a first avatar <NUM> (e.g., the first avatar <NUM> in <FIG>) and a second avatar <NUM> (e.g., the second avatar <NUM> in <FIG>) is displayed, an AR device (e.g., the AR device <NUM> of <FIG>) may detect a user gesture of moving a third avatar <NUM> among avatars <NUM> and <NUM> included in a first GUI <NUM> (e.g., the first GUI <NUM> of <FIG>) to the second GUI <NUM>. The user gesture of moving the third avatar <NUM> to the second GUI <NUM> may be the same as or similar to the user gesture described with reference to <FIG>. For example, the AR device may detect, through a hand pose sensor (e.g., the hand pose sensor <NUM> of <FIG>), a movement in which a user's finger <NUM> moves the third avatar <NUM> in a specified direction <NUM> while grabbing the third avatar <NUM> and then drops the third avatar <NUM> on the second GUI <NUM>.

According to an embodiment of the disclosure, a user device connected to the AR device may simultaneously communicate with a counterpart of the second avatar <NUM> and a counterpart of the third avatar <NUM>, in response to detecting that the third avatar <NUM> is positioned at a location adjacent to the second GUI <NUM>. <FIG> illustrates an embodiment in which only the third avatar <NUM> is added to the second GUI <NUM>, but the number of avatars added to the second GUI <NUM> is not limited thereto.

According to an embodiment of the disclosure, as the third avatar <NUM> is disposed at a location adjacent to the second GUI <NUM>, the AR device may provide a graphic effect such that the avatars <NUM>, <NUM>, and <NUM> included in the second GUI <NUM> express greeting each other.

<FIG> illustrates an operation of selecting one of a plurality of avatars disposed on a second GUI according to an embodiment of the disclosure.

Referring to <FIG>, an AR device <NUM> (e.g., the AR device <NUM> in <FIG>) may select one of remaining avatars <NUM> and <NUM> disposed on a second GUI <NUM> other than a first avatar <NUM> corresponding to a user account of the AR device <NUM> in an AR <NUM>, based on a gaze direction of a user <NUM> wearing the AR device <NUM>. For example, when a message input is obtained while a gaze direction <NUM> of the user <NUM> points to the second avatar <NUM>, it may be indicated that the obtained message input is used for a counterpart of a contact corresponding to the second avatar <NUM>.

Although not illustrated in <FIG>, according to other embodiments of the disclosure, the AR device <NUM> may select the second avatar <NUM> among the avatars <NUM> and <NUM>, in response to obtaining a user voice for calling an identifier or name of the second avatar <NUM>, or detecting that a finger of the user <NUM> points to the second avatar <NUM>.

<FIG> illustrates an operation of sharing content in an AR according to an embodiment of the disclosure.

Referring to <FIG>, while a second GUI <NUM> (e.g., the second GUI <NUM> in <FIG>) including a first avatar <NUM> and a second avatar <NUM> is displayed, an AR system may obtain a user input for sharing content in an AR. For example, the content may include at least one of an image, a video, a link, a map, another person's contact, music, a file, a voice message, or a schedule.

For example, in operation <NUM>, the AR system may detect a user gesture of sharing content <NUM> output in the AR. For example, the user gesture may include a gesture of grabbing the content <NUM> and dropping the content <NUM> on the second GUI <NUM> similar to the principle described with reference to <FIG>.

As another example, in operation <NUM>, the AR system may obtain a user input for sharing content <NUM> output on a touch screen of a user device <NUM> (e.g., the user device <NUM> in <FIG>) in the AR. For example, the user input may include a swipe input facing in a specific direction (e.g., upward) on the touch screen.

<FIG> illustrates an interaction scenario in a second GUI according to an embodiment of the disclosure.

Referring to <FIG>, it is assumed that a second GUI <NUM> including a first avatar <NUM> and a second avatar <NUM> is displayed in an AR. Operation <NUM> to operation <NUM> are not implemented in order and may be performed independently.

Referring to <FIG>, in operation <NUM>, an AR device (e.g., the AR device <NUM> in <FIG>) may express a user's emotion through a first avatar <NUM> corresponding to a user account. For example, the AR device may display a UI <NUM> for selecting various gestures of the first avatar <NUM>, and may control the first avatar <NUM> to express a gesture selected in the UI <NUM>.

In operation <NUM>, the AR device may display a UI <NUM> indicating that a voice message is being output while the voice message received from a counterpart of a contact corresponding to the second avatar <NUM> is being output.

In operation <NUM>, the AR device may display an input window <NUM> in the AR such that a user is capable of identifying an obtained message input.

<FIG> illustrates an image including avatars according to an embodiment of the disclosure.

Referring to <FIG>, an AR device may provide an image <NUM> of avatars <NUM>, <NUM>, <NUM>, or <NUM> included in a second GUI (e.g., the second GUI <NUM> of <FIG>). The number of avatars included in the image <NUM> is not limited to the example illustrated in <FIG>. For example, the AR device may generate the image <NUM> in response to a user input for requesting image capture.

As described above, according to various embodiments of the disclosure, an AR device (e.g., the AR device <NUM> of <FIG>) supporting an AR may include a display (e.g., the display <NUM> of <FIG>), a communication circuit (e.g., the communication interface <NUM> of <FIG>), a processor (e.g., the processor <NUM> of <FIG>) operatively connected to the display and the communication circuit, and a memory (e.g., the memory <NUM> of <FIG>) operatively connected to the processor. The memory may store instructions that, when executed, cause the processor to establish a connection with a user device (e.g., the user device <NUM> of <FIG>) storing a contact application and a message application, through the communication circuit, to detect that the message application is executed, and to display a first GUI (e.g., <NUM> of <FIG>) and at least one avatar (e.g., <NUM>, <NUM>, or <NUM> of <FIG>), which is disposed at a location adjacent to the first GUI and corresponds to at least one contact associated with the contact application or the message application, through the display in the AR.

According to an embodiment of the disclosure, the instructions may cause the processor to display a second GUI (e.g., the second GUI <NUM> of <FIG>), which indicates a message thread of the message application and is disposed at a location spaced from the first GUI, and a first avatar (e.g., the first avatar <NUM> of <FIG>), which is disposed at a location adjacent to the second GUI and which corresponds to a user account, through the display in the AR.

According to an embodiment of the disclosure, the AR device may further include at least one sensor (e.g., the sensor <NUM> of <FIG>), a microphone (e.g., the input/output interface <NUM> of <FIG>), and a speaker (e.g., the input/output interface <NUM> of <FIG>). The instructions may cause the processor to detect a first gesture for a second avatar (e.g., the second avatar <NUM> of <FIG>) disposed at the location adjacent to the first GUI, through the at least one sensor, and to provide a graphic effect in which the second avatar moves from the first GUI to the second GUI, through the display in response to detecting the first gesture.

According to an embodiment of the disclosure, while the first avatar and the second avatar are displayed at the location adjacent to the second GUI, the instructions may cause the processor to receive a first user input for inputting a message by detecting a second gesture for a virtual keyboard (e.g., the virtual keyboard <NUM> of <FIG>) displayed through the display through the at least one sensor, obtaining a user voice (e.g., the voice input <NUM> of <FIG>) through the microphone, or receiving information from the user device or an accessory device through the communication circuit.

According to an embodiment of the disclosure, the instructions may cause the processor to display the message corresponding to the first user input on the first avatar through the display.

According to an embodiment of the disclosure, while the first avatar and the second avatar are displayed at the location adjacent to the second GUI, the instructions may cause the processor to receive a first message of the second avatar through the communication circuit and to output a voice corresponding to the first message through the speaker or to display a text including the first message on the second avatar through the display.

According to an embodiment of the disclosure, while the first avatar and the second avatar are displayed at the location adjacent to the second GUI, the instructions may cause the processor to detect a movement of the AR device through the at least one sensor, to determine whether the second GUI is positioned within a viewing area (e.g., the viewing area <NUM> of <FIG>) of the display, in response to the detected movement, to display the text corresponding to the first message on the second avatar through the display based on the second GUI being positioned within the viewing area, or to output the voice corresponding to the first message through the speaker based on the second GUI being not positioned within the viewing area.

According to an embodiment of the disclosure, while the first avatar and the second avatar are displayed at the location adjacent to the second GUI, the instructions may cause the processor to detect a third gesture for a third avatar (e.g., the third avatar <NUM> of <FIG>) disposed at the location adjacent to the first GUI, through the at least one sensor and to provide a graphic effect in which the third avatar moves from the first GUI to the second GUI, in response to detecting the third gesture.

According to an embodiment of the disclosure, while the first avatar, the second avatar, and the third avatar are displayed at the location adjacent to the second GUI, the instructions may cause the processor to receive a second user input for selecting one of the second avatar and the third avatar by detecting a user gaze through the at least one sensor, obtaining a user voice through the microphone, or detecting a gesture through the at least one sensor.

According to an embodiment of the disclosure, while the first avatar and the second avatar are displayed at the location adjacent to the second GUI, the instructions may cause the processor to receive a second message of a fourth avatar (e.g., the third avatar <NUM> of <FIG>) disposed at the location adjacent to the first GUI, through the communication circuit and to provide a graphic effect indicating a movement of the fourth avatar through the display in response to receiving the second message.

According to an embodiment of the disclosure, while the message application is executed, the instructions may cause the processor to block execution of another application stored in the memory or to execute the other application in a background.

As described above, according to various embodiments of the disclosure, an AR system supporting an AR may include an AR device (e.g., the AR device <NUM> of <FIG>) and a user device (e.g., the user device <NUM> of <FIG>). The AR device may include a first display (e.g., the display <NUM> of <FIG>) and a first communication circuit (e.g., the communication interface <NUM> of <FIG>). The user device may include a second display (e.g., the touch screen <NUM> of <FIG>), a second communication circuit (e.g., the communication circuit <NUM> of <FIG>), a processor (e.g., the controller <NUM> of <FIG>) operatively connected to the second display and the second communication circuit, and a memory (e.g., the memory <NUM> of <FIG>) operatively connected to the processor. The memory may store at least one contact associated with a message application or a contact application. The memory may store instructions that, when executed, cause the processor to establish a connection with the AR device through the second communication circuit, to detect that the message application is executed, to render a first GUI (e.g., <NUM> of <FIG>) and at least one avatar (e.g., <NUM>, <NUM>, or <NUM> of <FIG>), which are to be displayed in the AR through the first display, as a <NUM> dimensional (3D) image, and to transmit the rendered first GUI and the rendered at least one avatar to the AR device through the second communication circuit. The at least one avatar may correspond to the at least one contact.

According to an embodiment of the disclosure, the instructions may cause the processor to render a second GUI (e.g., the second GUI <NUM> of <FIG>) and a first avatar (e.g., the first avatar <NUM> of <FIG>), which are to be displayed in the AR through the first display, as the 3D image and to transmit the rendered second GUI and the rendered first avatar to the AR device through the second communication circuit. The second GUI may indicate a message thread of the message application and is disposed at a location spaced from the first GUI. The first avatar may be disposed at a location adjacent to the second GUI and corresponds to a user account.

According to an embodiment of the disclosure, the AR system may further include at least one sensor (e.g., the sensor <NUM> of <FIG>) detecting a gesture of a user, a microphone (e.g., the input module <NUM> of <FIG> or the input/output interface <NUM> of <FIG>), and a speaker (e.g., the sound output module <NUM> of <FIG> or the input/output interface <NUM> of <FIG>). The instructions may cause the processor to detect a first gesture for a second avatar (e.g., the second avatar <NUM> of <FIG>) disposed at a location adjacent to the first GUI, through the at least one sensor, to render a first graphic effect in which the second avatar moves from the first GUI to the second GUI, in response to detecting the first gesture, and to transmit the rendered first graphic effect to the AR device through the second communication circuit.

According to an embodiment of the disclosure, the instructions may cause the processor to receive a first user input for inputting a message by receiving a text input through a virtual keyboard displayed on the second display or obtaining a user voice through the microphone.

According to an embodiment of the disclosure, the instructions may cause the processor to render the message corresponding to the first user input as the 3D image to be displayed in the AR through the first display and to transmit the rendered message to the AR device through the second communication circuit.

According to an embodiment of the disclosure, the instructions may cause the processor to receive a first message through the second communication circuit and to output a voice corresponding to the first message through the speaker or to render a text including the first message so as to be displayed in the AR through the first display.

According to an embodiment of the disclosure, the instructions may cause the processor to block execution of another application stored in the memory or to execute the other application in a background, while the message application is executed.

According to an embodiment of the disclosure, the instructions may cause the processor to detect a third gesture for a third avatar disposed at the location adjacent to the first GUI, through the at least one sensor, to render a second graphic effect in which the third avatar moves from the first GUI to the second GUI, in response to detecting the third gesture, and to transmit the rendered second graphic effect to the AR device through the second communication circuit.

According to an embodiment of the disclosure, the instructions may cause the processor to receive a second user input for selecting one of the second avatar and the third avatar by detecting a user gaze through the at least one sensor, obtaining a user voice through the microphone, or detecting a gesture through the at least one sensor.

According to an embodiment of the disclosure, the instructions may cause the processor to receive a second message of a fourth avatar disposed at the location adjacent to the first GUI, through the second communication circuit, to render a third graphic effect indicating a movement of the fourth avatar in response to receiving the second message, and to transmit the rendered third graphic effect to the AR device through the second communication circuit.

The electronic device according to various embodiments disclosed in the disclosure may be various types of devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a mobile medical appliance, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of this specification may not be limited to the above-described electronic devices.

Various embodiments of the disclosure and terms used herein are not intended to limit the technical features described in the disclosure to specific embodiments of the disclosure, and it should be understood that the embodiments and the terms include modification, equivalent, or alternative on the corresponding embodiments described herein. With regard to description of drawings, similar or related components may be marked by similar reference marks/numerals. The singular form of the noun corresponding to an item may include one or more of items, unless interpreted otherwise in context. In the disclosure, the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any and all combinations of one or more of the associated listed items. The terms, such as "first" or "second" may be used to simply distinguish the corresponding component from the other component, but do not limit the corresponding components in other aspects (e.g., importance or order). When a component (e.g., a first component) is referred to as being "coupled with/to" or "connected to" another component (e.g., a second component) with or without the term of "operatively" or "communicatively", it may mean that a component is connectable to the other component, directly (e.g., by wire), wirelessly, or through the third component.

The term "module" used herein may include a unit, which is implemented with hardware, software, or firmware, and may be interchangeably used with the terms "logic", "logical block", "part", or "circuit". The "module" may be a minimum unit of an integrated part or may be a minimum unit of the part for performing one or more functions or a part thereof. For example, according to an embodiment of the disclosure, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the disclosure may be implemented with software (e.g., the program <NUM>) including one or more instructions stored in a storage medium (e.g., an embedded memory <NUM> or an external memory <NUM>) readable by a machine (e.g., the electronic device <NUM>). For example, the processor (e.g., the processor <NUM>) of the machine (e.g., the electronic device <NUM>) may call at least one instruction of the stored one or more instructions from a storage medium and then may execute the at least one instruction. This enables the machine to operate to perform at least one function depending on the called at least one instruction. Herein, 'non-transitory' just means that the storage medium is a tangible device and does not include a signal (e.g., electromagnetic waves), and this term does not distinguish between the case where data is semi-permanently stored in the storage medium and the case where the data is stored temporarily. For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment of the disclosure, a method according to various embodiments disclosed herein may be provided to be included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or may be distributed (e.g., downloaded or uploaded), through an application store (e.g., PlayStore™), directly between two user devices (e.g., smartphones), or online. In the case of on-line distribution, at least part of the computer program product (e.g., a downloadable app) may be at least temporarily stored in the machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server or may be generated temporarily.

According to various embodiments of the disclosure, each component (e.g., a module or a program) of the above-described components may include a single entity or a plurality of entities. According to various embodiments of the disclosure, one or more the components of the above-described components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the manner same as or similar to being performed by the corresponding component of the plurality of components prior to the integration. According to various embodiments of the disclosure, operations executed by modules, programs, or other components may be executed by a successive method, a parallel method, a repeated method, or a heuristic method. Alternatively, at least one or more of the operations may be executed in another order or may be omitted, or one or more operations may be added.

Claim 1:
An augmented reality, AR, device that supports an AR, the AR device comprising:
a display;
a communication circuit;
at least one sensor;
at least one processor (<NUM>) operatively connected to the display and the communication circuit; and
a memory (<NUM>) operatively connected to the at least one processor (<NUM>),
wherein the memory (<NUM>) stores instructions that, when executed, cause the at least one processor (<NUM>) to:
establish a connection with a user device storing a contact application and a message application, through the communication circuit,
detect that the message application is executed, and
display a first graphic user interface (<NUM>), GUI, and at least one avatar, which is disposed at a location adjacent to the first GUI (<NUM>) and corresponds to at least one contact associated with the contact application or the message application, through the display in the AR;
wherein the instructions further cause the at least one processor (<NUM>) to:
display a second GUI (<NUM>) which is disposed at a location spaced from the first GUI (<NUM>), and a first avatar (<NUM>), which is disposed at a location adjacent to the second GUI (<NUM>) and which corresponds to a user account, through the display in the AR;
detect a first gesture for selecting a second avatar (<NUM>) among the at least one avatar which is disposed at the location adjacent to the first GUI (<NUM>), through the at least one sensor, and
provide a graphic effect in which the second avatar (<NUM>) moves from the first GUI (<NUM>) to the second GUI (<NUM>), through the display in response to detecting the first gesture;
wherein when the second avatar (<NUM>) is located on the second GUI (<NUM>), the user device is operable to communicate with a counterpart of a contact corresponding to the second avatar (<NUM>).