Augmented reality device and electronic device interacting with augmented reality device

An augmented reality device may comprise a display, at least one sensor, a communication circuit, and at least one processor operatively connected with the display, the at least one sensor, and the communication circuit. The at least one processor may be configured to identify a first user input specifying a shared area for outputting shared content to be output on at least one other augmented reality device, transmit information about the shared area to a server through the communication circuit, based on the first user input, identify whether the shared area and a personal area for outputting personal content overlap, based on information obtained through the at least one sensor, control the display to output the shared content in the shared area and the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and control the display to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

BACKGROUND

The disclosure relates to an augmented reality device and an electronic device that interacts with the augmented reality device.

2. Description of Related Art

Augmented reality (AR) is technology for overlaying three-dimensional (3D) (or two-dimensional (2D)) virtual images on the real-world image or background and displaying overlaid images. Augmented reality technology which combines the real-world environment with virtual objects enables the user to view the real-world environment, thereby providing a better real-life feel and additional information.

Augmented reality devices may allow images provided through a projector to be incident through a prism to the input grating surface. The user may then view the image passing the output grating surface with his eyes. The user may observe the image together with the real-world environment and identify information about a target object in the environment that he is currently viewing.

SUMMARY

A meeting may be conducted using augmented reality devices. In a meeting using augmented reality devices, several meeting participants may each use their respective augmented reality devices in the same meeting space. Each meeting participant's augmented reality device may display shared data for the meeting.

However, since each meeting participant mainly looks at the data displayed on her augmented reality device, eye contact with other meeting participants may be insufficient. Therefore, in a meeting using augmented reality devices, it may be difficult for a meeting participant to effectively interact with other meeting participants.

Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. According to an embodiment, an augmented reality device may display shared data for a meeting in a position with fixed Cartesian coordinates, received from the server.

According to an embodiment, an augmented reality device may comprise a display, at least one sensor, a communication circuit, and at least one processor operatively connected with the display, the at least one sensor, and the communication circuit. The at least one processor may be configured to identify a first user input specifying a shared area for outputting shared content to be output on at least one other augmented reality device, transmit information about the shared area to a server through the communication circuit, based on the first user input, identify whether the shared area and a personal area for outputting personal content overlap, based on information obtained through the at least one sensor, control the display to output the shared content in the shared area and the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and control the display to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

According to an embodiment, an augmented reality device may comprise a display, at least one sensor, a communication circuit, and at least one processor operatively connected with the display, the at least one sensor, and the communication circuit. The at least one processor may be configured to obtain shared content and information indicating a position of a shared area for outputting the shared content, from a server through the communication circuit, identify whether the shared area and a personal area for outputting personal content overlap, based on information obtained through the at least one sensor, control the display to output the shared content in the shared area and the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and control the display to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

According to an embodiment, an electronic device may comprise a communication circuit and at least one processor operatively connected with the communication circuit. The at least one processor may be configured to obtain shared content and information indicating a position of a shared area for outputting the shared content, from a server through the communication circuit, obtain information obtained through at least one sensor of an augmented reality device, from the augmented reality device through the communication circuit, identify whether the shared area and a personal area for outputting personal content overlap, based on the information obtained through the at least one sensor of the augmented reality device, transmit, to the augmented reality device through the communication circuit, display information to allow the augmented reality device to output the shared content in the shared area and output the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and transmit, to the augmented reality device through the communication circuit, display information to allow the augmented reality device to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

According to an embodiment, there may be provided an augmented reality device and an electronic device that interacts with the augmented reality device. According to an embodiment, an augmented reality device may output shared data for a meeting in a position with fixed Cartesian coordinates, received from the server. Accordingly, since the gazes of several meeting participants are focused within a certain range from a position having fixed Cartesian coordinates, a meeting participant may efficiently interact with other meeting participants.

DETAILED DESCRIPTION

FIG.1illustrates a structure of an augmented reality device according to various embodiments.

According to various embodiments, an augmented reality device100may include one or more first cameras111-1and111-2, one or more second cameras112-1and112-2, and one or more third cameras113. According to various embodiments, an image obtained through the one or more first cameras111-1and111-2may be used for detection of the user's hand gesture, tracking the user's head, and/or spatial recognition. According to various embodiments, the one or more first cameras111-1and111-2may be global shutter (GS) cameras. According to various embodiments, the one or more first cameras111-1and111-2may perform a simultaneous localization and mapping (SLAM) operation through depth capture. According to various embodiments, the one or more first cameras111-1and111-2may perform spatial recognition for 6DoF.

According to various embodiments, an image obtained through the one or more second cameras112-1and112-2may be used to detect and track the user's pupil. According to various embodiments, the one or more second cameras112-1and112-2may be GS cameras. According to various embodiments, the one or more second cameras112-1and112-2may correspond to the left eye and the right eye, respectively, and the one or more second cameras112-1and112-2may have the same performance.

According to various embodiments, the one or more third cameras113may be high-resolution cameras. According to various embodiments, the one or more third cameras113may perform an auto-focusing (AF) function and an image stabilization function. According to various embodiments, the one or more third cameras113may be a GS camera or a rolling shutter (RS) camera.

According to various embodiments, the augmented reality device100may include one or more light emitting elements114-1and114-2. The light emitting elements114-1and114-2are different from a light source, which is described below, for irradiating light to a screen output area of the display. According to various embodiments, the light emitting devices114-1and114-2may irradiate light to facilitate pupil detection in detecting and tracking the user's pupils through the one or more second cameras112-1and112-2. According to various embodiments, each of the light emitting devices114-1and114-2may include a light emitting diode (LED). According to various embodiments, the light emitting devices114-1and114-2may irradiate light in an infrared band. According to various embodiments, the light emitting devices114-1and114-2may be attached around the frame of the augmented reality device100. According to various embodiments, the light emitting devices114-1and114-2may be positioned around the one or more first cameras111-1and111-2and may assist in gesture detection, head tracking, and spatial recognition by the one or more first cameras111-1and111-2when the augmented reality device100is used in a dark environment. According to various embodiments, the light emitting devices114-1and114-2may be positioned around the one or more third cameras113and may assist in obtaining images by the one or more third cameras113when the augmented reality device100is used in a dark environment.

According to various embodiments, the augmented reality device100may include batteries135-1and135-2. The batteries135-1and135-2may store power for operating the remaining components of the augmented reality device100.

According to various embodiments, the augmented reality device100may include a first display151, a second display152, one or more input optical members153-1and153-2, one or more transparent members190-1and190-2, and one or more screen display portions154-1and154-2. According to various embodiments, the first display151and the second display152may include, e.g., a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), or an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). According to various embodiments, when the first display151and the second display152are formed of one of a liquid crystal display device, a digital mirror display device, or a silicon liquid crystal display device, the augmented reality device100may include a light source for irradiating light to a screen output area of the display. According to various embodiments, when the first display151and the second display152may generate light on their own, e.g., when formed of either organic light emitting diodes or micro LEDs, the augmented reality device100may provide a virtual image of good quality to the user even when a separate light source is not included.

According to various embodiments, the one or more transparent members190-1and190-2may be disposed to face the user's eyes when the user wears the augmented reality device100. According to various embodiments, the one or more transparent members190-1and190-2may include at least one of a glass plate, a plastic plate, or a polymer. According to various embodiments, the user may view the outside world through the one or more transparent members190-1and190-2when the user wears the augmented reality device100. According to various embodiments, the one or more input optical members153-1and153-2may guide the light generated by the first display151and the second display152to the user's eyes. According to various embodiments, images based on the light generated by the first display151and the second display152may be formed on one or more screen display portions154-1and154-2on the one or more transparent members190-1and190-2, and the user may view the images formed on the one or more screen display portions154-1and154-2.

According to various embodiments, the augmented reality device100may include one or more optical waveguides (not shown). The optical waveguide may transfer the light generated by the first display151and the second display152to the user's eyes. The augmented reality device100may include one optical waveguide corresponding to each of the left eye and the right eye. According to various embodiments, the optical waveguide may include at least one of glass, plastic, or polymer. According to various embodiments, the optical waveguide may include a nano-pattern formed inside or on one outer surface, e.g., a polygonal or curved grating structure. According to various embodiments, the optical waveguide may include a free-form type prism, and in this case, the optical waveguide may provide incident light to the user through a reflective mirror. According to various embodiments, the optical waveguide may include at least one of at least one diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a reflective mirror) and guide the display light emitted from the light source to the user's eyes using at least one diffractive element or reflective element included in the optical waveguide. According to various embodiments, the diffractive element may include input/output optical elements. According to various embodiments, the reflective element may include a member causing total reflection.

According to various embodiments, the augmented reality device100may include one or more sound input devices162-1,162-2, and162-3and one or more sound output devices163-1and163-2.

According to various embodiments, the augmented reality device100may include a first PCB170-1and a second PCB170-2. The first PCB170-1and the second PCB170-2may transfer electrical signals to components included in the augmented reality device100, such as a first camera211, a second camera212, a third camera213, a display module250, an audio module261, and a sensor280described below with reference toFIG.2. According to various embodiments, the first PCB170-1and the second PCB170-2may be FPCBs. According to various embodiments, the first PCB170-1and the second PCB170-2each may include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate.

FIG.2is a block diagram illustrating an augmented reality device according to various embodiments.

An augmented reality device200may include a first camera211, a second camera212, a third camera213, a processor220, a PMIC230, a battery235, a memory240, a display module250, an audio module261, a sound input device262, a sound output device263, a communication circuit270, and a sensor280.

According to various embodiments, the details of the one or more first cameras111-1and111-2, one or more second cameras112-1and112-2, and one or more third cameras113described above in connection withFIG.1may be equally applied to the first camera211, the second camera212, and the third camera213, respectively. According to various embodiments, the augmented reality device200may include at least one of the first camera211, the second camera212, and the third camera213, in plurality.

According to various embodiments, the processor220may control the other components of the augmented reality device200, e.g., the first camera211, the second camera212, the third camera213, the PMIC230, the memory240, the display module250, the audio module261, the communication circuit270, and the sensor280and may perform various data processing or computations.

According to various embodiments, the PMIC230may convert the power stored in the battery235to have the current or voltage required by the other components of the augmented reality device200and supply it to the other components of the augmented reality device200.

According to various embodiments, the memory240may store various data used by at least one component (e.g., the processor220or a sensor280) of the augmented reality device200.

According to various embodiments, the display module250may display a screen to be provided to the user. According to various embodiments, the display module250may include the first display151, the second display152, one or more input optical members153-1and153-2, one or more transparent members190-1and190-2, and one or more screen display portions154-1and154-2, which are described above in connection withFIG.1.

According to various embodiments, the audio module261may be connected to the sound input device262and the sound output device263and may convert the data input through the sound input device262and may convert data to be output to the sound output device263. The sound output device263may include a speaker and an amplifier.

According to various embodiments, the communication circuit270may support establishment of a wireless communication channel with an electronic device outside the augmented reality device200and performing communication through the established communication channel.

According to various embodiments, the sensor280may include a 6-axis sensor281, a magnetic sensor282, a proximity sensor283, and/or an optical sensor284.

FIG.3illustrates an augmented reality device, an electronic device interacting with the augmented reality device, and a server according to various embodiments.

An augmented reality device330and an augmented reality device340may be devices of meeting participants participating in the same meeting. According to various embodiments, at least one of the augmented reality device330and the augmented reality device340may have the authority to control a shared area, which is an area in which shared content to be shared in the meeting is displayed. A server310may store information about which augmented reality device among the plurality of augmented reality devices330and340has the authority to control the shared area.

According to various embodiments, the server310may perform direct communication with the augmented reality device340. According to various embodiments, the server310may perform communication with an electronic device320interacting with the augmented reality device330. According to various embodiments, the electronic device320may perform at least one operation for determining content to be displayed on the augmented reality device330.

According to various embodiments, the server310may receive, from the electronic device320, at least one of position information about the augmented reality device330and information about the space in which the augmented reality device330is positioned. According to various embodiments, the server310may receive at least one of the position information about the augmented reality device340and information about the space in which the augmented reality device340is positioned directly from the augmented reality device340. The server310may configure and store a map of the meeting place based on at least one of information obtained from the electronic device320and information obtained from the augmented reality device340.

The server310may receive information about content to be shared in the meeting from the electronic device320and may transfer it to the augmented reality device340. In another embodiment, the server310may receive information about content to be shared in the meeting from the augmented reality device340and may transfer it to the electronic device320. The information about the content to be shared in the meeting may include at least one of the content to be shared in the meeting and the coordinates of the position where the content to be shared in the meeting is displayed.

According to various embodiments, when the server310receives the information about the content to be shared in the meeting from the electronic device320and transfers it to the augmented reality device340, the augmented reality device340may display the shared content shared in the meeting. According to various embodiments, the augmented reality device340may further display personal content of the augmented reality device340, which is not shared with the augmented reality devices (e.g., the augmented reality device330) of the meeting members, while simultaneously displaying the shared content.

According to various embodiments, when the server310receives the information about the content to be shared in the meeting from the augmented reality device340and transfers it to the electronic device320, the electronic device320may transmit information about content to be displayed on the augmented reality device330(hereinafter “display information”) to the augmented reality device330. The display information may include position information about the shared content, which is expressed as coordinates with respect to the augmented reality device330. According to various embodiments, the display information may include information about the position where the personal content for the user of the augmented reality device330, not shared with the augmented reality devices of the meeting members, is to be displayed, and which is expressed as coordinates with respect to the augmented reality device330.

Operations performed on an augmented reality device are mainly described below in connection withFIGS.5,8,9A,10,11,12A,12B, and14. However, as shown inFIG.3, the augmented reality device330may exchange information with the server310through the electronic device320in which case at least some of the operations performed on the augmented reality device340directly communicating with the server310may be performed on the electronic device320.

FIG.4is a block diagram illustrating an electronic device in a network environment according to various embodiments.

Referring toFIG.4, an electronic device401in a network environment400may communicate with an external electronic device402via a first network498(e.g., a short-range wireless communication network), or an external electronic device404or a server408via a second network499(e.g., a long-range wireless communication network). According to an embodiment, the electronic device401may communicate with the external electronic device404via the server408. According to an embodiment, the electronic device401may include a processor420, a memory430, an input module450, a sound output module455, a display module460, an audio module470, a sensor module476, an interface477, a connecting terminal478, a haptic module479, a camera module480, a power management module488, a battery489, a communication module490, a subscriber identification module (SIM)496, or an antenna module497. In some embodiments, at least one (e.g., the connecting terminal478) of the components may be omitted from the electronic device401, or one or more other components may be added in the electronic device101. According to an embodiment, some (e.g., the sensor module476, the camera module480, or the antenna module497) of the components may be integrated into a single component (e.g., the display module460). According to an embodiment, an electronic device401may include the same components, or perform the same operations, as the augmented reality device100ofFIG.1, the augmented reality device200ofFIG.2, or the augmented reality device330or augmented reality device340ofFIG.3.

The processor420(e.g., the processor220ofFIG.2) may execute, for example, software (e.g., a program440) to control at least one other component (e.g., a hardware or software component) of the electronic device401coupled with the processor420, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor420may store a command or data received from another component (e.g., the sensor module476or the communication module490) in a volatile memory432, process the command or the data stored in the volatile memory432, and store resulting data in a non-volatile memory434. According to an embodiment, the processor420may include a main processor421(e.g., a central processing unit (CPU) or an application processor (AP)), or an auxiliary processor423(e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor121. For example, when the electronic device401includes the main processor421and the auxiliary processor423, the auxiliary processor423may be configured to use lower power than the main processor421or to be specified for a designated function. The auxiliary processor423may be implemented as separate from, or as part of the main processor421.

The auxiliary processor423may control at least some of functions or states related to at least one component (e.g., the display module460, the sensor module476, or the communication module490) among the components of the electronic device401, instead of the main processor421while the main processor421is in an inactive (e.g., sleep) state, or together with the main processor421while the main processor421is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor423(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module480or the communication module490) functionally related to the auxiliary processor123. According to an embodiment, the auxiliary processor423(e.g., the neural processing unit) may include a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be generated via machine learning. Such learning may be performed, e.g., by the electronic device401where the artificial intelligence is performed or via a separate server (e.g., the server408). Learning algorithms may include, but are not limited to, e.g., supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-network or a combination of two or more thereof but is not limited thereto. The artificial intelligence model may, additionally or alternatively, include a software structure other than the hardware structure.

The memory430(e.g., the memory240ofFIG.2) may store various data used by at least one component (e.g., the processor420or the sensor module476) of the electronic device401. The various data may include, for example, software (e.g., the program440) and input data or output data for a command related thereto. The memory430may include the volatile memory432or the non-volatile memory434.

The program440may be stored in the memory430as software, and may include, for example, an operating system (OS)442, middleware444, or an application446.

The input module450may receive a command or data to be used by other component (e.g., the processor420) of the electronic device401, from the outside (e.g., a user) of the electronic device401. The input module450may include, for example, a microphone, a mouse, a keyboard, keys (e.g., buttons), or a digital pen (e.g., a stylus pen).

The sound output module455(e.g., the audio output device263) may output sound signals to the outside of the electronic device401. The sound output module455may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record. The receiver may be used for receiving incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.

The display module460(e.g., the display module250ofFIG.2) may visually provide information to the outside (e.g., a user) of the electronic device401. The display460may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display460may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module470(e.g., the audio module261ofFIG.2) may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module470may obtain the sound via the input module450, or output the sound via the sound output module455or a headphone of an external electronic device (e.g., an electronic device402) directly (e.g., wiredly) or wirelessly coupled with the electronic device401.

The interface477may support one or more specified protocols to be used for the electronic device401to be coupled with the external electronic device (e.g., the electronic device402) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface477may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal478may include a connector via which the electronic device401may be physically connected with the external electronic device (e.g., the external electronic device402). According to an embodiment, the connecting terminal478may include, for example, a HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The camera module480(e.g., the first camera211, the second camera212or the third camera213ofFIG.2) may capture still images and videos. According to an embodiment, the camera module480may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module488(e.g., the PMIC230ofFIG.2) may manage power supplied to the electronic device401. According to one embodiment, the power management module488may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

The battery489(e.g., the battery235ofFIG.2) may supply power to at least one component of the electronic device401. According to an embodiment, the battery489may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module490(e.g., the communication circuit270ofFIG.2) may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device401and the external electronic device (e.g., the electronic device402, the electronic device404, or the server408) and performing communication via the established communication channel. The communication module490may include one or more communication processors that are operable independently from the processor420(e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module490may include a wireless communication module492(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module494(e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device404via a first network498(e.g., a short-range communication network, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or a second network499(e.g., a long-range communication network, such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., local area network (LAN) or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module492may identify or authenticate the electronic device401in a communication network, such as the first network498or the second network499, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module496.

The antenna module497may transmit or receive a signal or power to or from the outside (e.g., the external electronic device). According to an embodiment, the antenna module497may include one antenna including a radiator formed of a conductor or conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module497may include a plurality of antennas (e.g., an antenna array). In this case, at least one antenna appropriate for a communication scheme used in a communication network, such as the first network498or the second network499, may be selected from the plurality of antennas by, e.g., the communication module490. The signal or the power may then be transmitted or received between the communication module490and the external electronic device via the selected at least one antenna. According to an embodiment, other parts (e.g., radio frequency integrated circuit (RFIC)) than the radiator may be further formed as part of the antenna module497. According to various embodiments, the antenna module497may form a mmWave antenna module. According to an embodiment, the mmWave antenna module may include a printed circuit board, a RFIC disposed on a first surface (e.g., the bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high-frequency band (e.g., the mmWave band), and a plurality of antennas (e.g., array antennas) disposed on a second surface (e.g., the top or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high-frequency band.

These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module492may identify or authenticate the electronic device401in a communication network, such as the first network498or the second network499, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module496.

The antenna module497may transmit or receive a signal or power to or from the outside (e.g., the external electronic device). According to an embodiment, the antenna module497may include one antenna including a radiator formed of a conductor or conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module497may include a plurality of antennas (e.g., an antenna array). In this case, at least one antenna appropriate for a communication scheme used in a communication network, such as the first network498or the second network499, may be selected from the plurality of antennas by, e.g., the communication module490. The signal or the power may then be transmitted or received between the communication module490and the external electronic device via the selected at least one antenna. According to an embodiment, other parts (e.g., radio frequency integrated circuit (RFIC)) than the radiator may be further formed as part of the antenna module497. According to various embodiments, the antenna module497may form a mmWave antenna module. According to an embodiment, the mmWave antenna module may include a printed circuit board, a RFIC disposed on a first surface (e.g., the bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high-frequency band (e.g., the mmWave band), and a plurality of antennas (e.g., array antennas) disposed on a second surface (e.g., the top or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high-frequency band.

According to an embodiment, commands or data may be transmitted or received between the electronic device401and the external electronic device404via the server408coupled with the second network499. The external electronic devices402or404each may be a device of the same or a different type from the electronic device401. According to an embodiment, all or some of operations to be executed at the electronic device401may be executed at one or more of the external electronic devices402,404, or408. For example, if the electronic device401should perform a function or a service automatically, or in response to a request from a user or another device, the electronic device401, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service. The one or more external electronic devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request, and transfer an outcome of the performing to the electronic device401. The electronic device401may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device401may provide ultra low-latency services using, e.g., distributed computing or mobile edge computing. In another embodiment, the external electronic device404may include an internet-of-things (IoT) device. The server408may be an intelligent server using machine learning and/or a neural network. According to an embodiment, the server408may include the same components, or perform the same operations, as the server310ofFIG.3. According to an embodiment, the external electronic device404or the server408may be included in the second network499. The electronic device401may be applied to intelligent services (e.g., smart home, smart city, smart car, or health-care) based on 5G communication technology or IoT-related technology.

FIG.5illustrates operations of augmented reality devices and a server according to various embodiments.

InFIG.5, a first augmented reality device501may be a device authorized to control the shared area. For example, the first augmented reality device501may be a device used by a meeting host, and the second augmented reality device503may be a device used by a meeting participant.

In operation510, the first augmented reality device501(e.g., the processor220of the augmented reality device200) may identify a user input specifying a shared area. According to various embodiments, the user input may be an input through a controller connected to the first augmented reality device501. According to various embodiments, the user input may be identified by tracking the gaze of the user of the first augmented reality device501based on an image obtained through at least one camera (e.g., the second camera212) of the first augmented reality device501. According to various embodiments, the user input may be identified by detecting the gesture of the user of the first augmented reality device501based on an image obtained through at least one camera (e.g., the first camera211) of the first augmented reality device501.

In operation520, the first augmented reality device501(e.g., the processor220of the augmented reality device200) may transmit information about the shared area to a server502through a communication circuit (e.g., the communication circuit270). According to various embodiments, the information about the shared area may indicate at least one of shared content to be shared with another augmented reality device (e.g., the second augmented reality device503) or the position of the shared area.

According to various embodiments, the position of the shared area transmitted from the first augmented reality device501to the server502may be expressed as server reference coordinates with respect to a reference point set by the server502. According to various embodiments, the position of the shared area transmitted from the first augmented reality device501to the server502may be expressed as coordinates with respect to the first augmented reality device501. In this case, the first augmented reality device501may transmit, to the server502, space information and/or position information about the first augmented reality device501, identified through SLAM technology, and the server502may convert the position of the shared area, which is expressed as coordinates with respect to the first augmented reality device501and received from the first augmented reality device501, into the server reference coordinates, based on the space information and/or position information about the first augmented reality device501.

In operation530, the server502may transmit the shared content and/or the position of the shared area to the second augmented reality device503. According to various embodiments, the position of the shared area transmitted from the server502to the second augmented reality device503may be expressed as the server reference coordinates. In this case, the second augmented reality device503may convert the position of the shared area expressed as the server reference coordinates into coordinates with respect to the second augmented reality device503, based on at least one of the position information and/or space information about the second augmented reality device503.

According to various embodiments, the position of the shared area transmitted from the server502to the second augmented reality device503may be expressed as coordinates with respect to the second augmented reality device503. In this case, before performing operation530, the server502may receive at least one of the position information and/or space information about the second augmented reality device503from the second augmented reality device503, identify the position of the shared area expressed as the coordinates with respect to the second augmented reality device503based on at least one of the position information and/or space information about the second augmented reality device503, and transmit it to the second augmented reality device503.

According to various embodiments, the server502may generate shared content in the form of a virtual content, based on the shared content received from the first augmented reality device501. For example, the server502may generate a virtual object for data, such as an image or text received from the first augmented reality device501and transmit it to the second augmented reality device503.

In operation540, the second augmented reality device503(e.g., the processor220) may identify whether the shared area and the personal area overlap based on the position of the shared area received in operation530.

According to various embodiments, the personal area of the second augmented reality device503may be an area for displaying content not shared with other meeting participants, preset by the second augmented reality device503.

According to various embodiments, the second augmented reality device503(e.g., the processor220) may perform operation540in response to receiving the position of the shared area according to operation530while displaying the content not shared with the other meeting participants. In this case, the content not shared with other meeting participants may be defined as personal content, and the area where the personal content may be defined as a personal area.

According to various embodiments, while displaying the shared content in the shared area after receiving the position of the shared area according to operation530, the second augmented reality device503(e.g., the processor220) may perform operation540in response to identifying an input for displaying the content not shared with other meeting participants. In this case, the content not shared with other meeting participants may be defined as personal content, and the area for displaying the personal content may be defined as a personal area.

According to various embodiments, the personal area of the second augmented reality device503may be specified by an input through a controller connected with the second augmented reality device503. According to various embodiments, the personal area of the second augmented reality device503may be identified by tracking the gaze of the user of the second augmented reality device503based on an image obtained through at least one camera (e.g., the second camera212) of the second augmented reality device503. According to various embodiments, the personal area of the second augmented reality device503may be identified by detecting the gesture of the user of the second augmented reality device503based on an image obtained through at least one camera (e.g., the first camera211) of the second augmented reality device503. According to various embodiments, the personal area of the second augmented reality device503may be designated as an arbitrary display area without a separate user input. According to various embodiments, the personal area of the second augmented reality device503may have a position set as default, without a separate user input. For example, the position set as default may be determined by the system of the second augmented reality device503or may be set by the user.

According to various embodiments, that the shared area and the personal area overlap may mean that there is an area belonging to both the shared area and the personal area without being limited to when the boundary specifying the shared area and the boundary specifying the personal area completely overlap each other.

According to various embodiments, the shared area may be displayed in a world lock scheme on the second augmented reality device503. For example, the world lock scheme may mean that the position of the shared area is fixed when expressed as the server reference coordinates or Cartesian coordinates with respect to the reference point set by the server502.

According to various embodiments, the personal area in the second augmented reality device503may be displayed in the world lock scheme like the shared area. According to various embodiments, the personal area may be displayed in a body lock scheme on the second augmented reality device503. For example, the body lock scheme may mean that the position of the personal area is fixed when expressed as coordinates with respect to the second augmented reality device503. In other words, the position of the personal area may be fixed within the field of view (FOV) of the user (or the camera (e.g., the first camera211or the second camera212)) of the second augmented reality device503.

Although not shown inFIG.5, according to various embodiments, after performing operation520, the first augmented reality device501(e.g., the processor220of the augmented reality device200) may identify an input for displaying the personal content in the personal area while displaying the shared content in the shared area or perform operations510and520while displaying the personal content. According to various embodiments, the first augmented reality device501(e.g., the processor220of the augmented reality device200) may identify whether the personal area where the personal content is displayed and the shared area overlap, based on the information obtained through at least one sensor280.

According to various embodiments, the personal area may be an area for displaying personal content which is content displayed only on the first augmented reality device501and not displayed on the augmented reality devices of other meeting participants.

According to various embodiments, when an input for displaying personal content in the personal area is identified while displaying the shared content in the shared area, the personal area, as an area for displaying personal content, may be specified by the user or set as default.

According to various embodiments, while the first augmented reality device501(e.g., the processor220of the augmented reality device200) performs operations510and520while displaying personal content, the personal area may mean an area that the personal content used to be displayed.

Details of the personal area of the first augmented reality device501are identical to the details of the personal area of the second augmented reality device503, described above in connection with operation540.

Based on identifying that the shared area and the personal area do not overlap, the first augmented reality device501(e.g., the processor220of the augmented reality device200) may control the display (e.g., the display module250) to display the shared content in the shared area and the personal content in the personal area.

Based on identifying that the shared area and the personal area overlap, the first augmented reality device501(e.g., the processor220of the augmented reality device200) may control the display (e.g., the display module250) to display the shared content in the shared area and the personal content in an area not overlapping the shared area.

To describe the world lock scheme and the body lock scheme, reference may be made toFIGS.6A and6B.

FIGS.6A and6Billustrate an example of displaying body lock content and world lock content according to various embodiments.

Referring toFIG.6A, body lock contents610aand630aand a world lock content620amay be displayed within the FOV of the user650aof the augmented reality device640a. For example, the FOV of the user650aof the augmented reality device640amay mean an area that the user650amay identify through the augmented reality device640aand may substantially coincide with or be contained within the FOV of a camera (e.g., the first camera211or second camera212).

FIG.6Billustrates a case in which the user650bturns her head to the right. As the user650bturns her head to the right as compared withFIG.6A, the FOV of the user650bof the augmented reality device640bmay rotate to the right area as compared toFIG.6A. The body lock contents610band630bmay be displayed in constant positions within the FOV of the user650b. In other words, when expressed as coordinates with respect to the augmented reality device640b, the positions of the body lock contents610band630bmay be the same as the positions of the body lock contents610aand630aofFIG.6A. When expressed as server reference coordinates or Cartesian coordinates, the position of the world lock content620bmay be constant. For example, the world lock content620bmay not be positioned within the FOV of the user650bbased on the direction in which the user650bor the augmented reality device640bfaces.

FIG.7Aillustrates an example in which the shared area and the personal area are identified as overlapping in operation540.

Referring toFIG.7A, a user750amay identify an area through the augmented reality device760aand the augmented reality device760amay define a plurality of personal areas710a,720a, and740aand a shared area730a. The personal area720aand the shared area730amay partially overlap.

When it is identified that the shared area and the personal area overlap in operation540, the second augmented reality device503(e.g., the processor220) may display the shared content in the shared area and personal content in an area that does not overlap the shared area, in operation550. For example, an area in which personal content is displayed may be included in an area other than the shared area in the FOV. Further, the personal area may be an area in which personal content is displayed.

FIG.7Billustrates an example of displaying shared content and personal content according to various embodiments.FIG.7Billustrates an example in which a shared area and a personal area are displayed according to operation550.

Referring toFIG.7B, a user750bmay identify an area through the augmented reality device760band a personal area710displayed on an augmented reality device760bis identical in position to a personal area710a, a personal area740bis identical in position to a personal area740a, and a shared area730bis identical in position to a shared area730a. The position of a personal area720aoverlapping the shared area730amay be changed according to operation550so that a personal area720bmay be displayed in a position not overlapping a shared area730b.

As described above in connection withFIG.3, at least some of the operations of the second augmented reality device503(e.g., the augmented reality device640aor the augmented reality device640b) may be performed on an electronic device (e.g., the electronic device320ofFIG.3) connected to the second augmented reality device503. In this case, at least one processor of the electronic device may obtain shared content from the server502and information indicating the position of the shared area for outputting the shared content, obtain information about the personal area from the second augmented reality device503, obtain at least one of position information and/or space information obtained through at least one sensor of the second augmented reality device503from the second augmented reality device503, identify whether the shared area and the personal area overlap based on at least one of the position information and/or the space information, transmit, to the second augmented reality device503, display information to allow the second augmented reality device503to output the shared content in the shared area and the personal content in the personal area in response to identifying that the shared area and the personal area do not overlap, and transmit, to the second augmented reality device503, display information to allow the second augmented reality device503to output the shared content in the shared area and the personal content in an area different from the personal area and not overlapping the shared area in response to identifying that the shared area and the personal area overlap. The second augmented reality device503may output a screen based on the display information received from the electronic device. According to an embodiment, when the position information and/or space information obtained through at least one sensor of the second augmented reality device503is changed, or periodically, the second augmented reality device503may transmit position information and/or space information to the electronic device (e.g., the electronic device320ofFIG.3) connected with the second augmented reality device503and receive, from the electronic device, and output display information.

As described above in connection withFIG.3, at least some of the operations of the first augmented reality device501may be performed on the electronic device connected with the first augmented reality device501. In this case, the electronic device connected with the first augmented reality device501may receive information about a user input through communication from the first augmented reality device501or identify a user input made as an input to the electronic device and transmit information about the shared area to the server502based on the user input.

FIG.8is a flowchart illustrating operations performed in an augmented reality device according to various embodiments.

In operation810, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify the shared area and the FOV of the user of the augmented reality device200. According to various embodiments, the at least one processor220may identify the shared area based on a user input. As described above with reference to operation510ofFIG.5, the user input may be at least one of an input through a controller, an input identified through eye tracking, or a gesture input. In this case, the augmented reality device200may have the authority to control the shared area.

According to various embodiments, the at least one processor220may identify the shared area based on information received from the server. In this case, the augmented reality device200may not have the authority to control the shared area. The information received from the server may indicate a shared area specified by another augmented reality device having the authority to control the shared area.

According to various embodiments, the at least one processor220may identify the FOV based on the image obtained through at least one of a first camera (e.g., the first camera211) or a second camera (e.g., the second camera212).

In operation820, at least one processor (e.g., processor220) of the augmented reality device (e.g., the augmented reality device200) may identify whether the shared area is included in the field of view (FOV). According to various embodiments, the at least one processor220may identify that the shared area is included in the FOV only when the entire shared area is included in the FOV. According to various embodiments, the at least one processor220may identify that the shared area is included in the FOV when at least a portion of the shared area is included in the FOV.

When it is identified that the shared area is not included in the FOV in operation820, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may display an object indicating the shared area in operation830. According to various embodiments, the object indicating the shared area may indicate a relative position of the shared area with respect to the FOV. According to various embodiments, the object indicating the shared area may be an arrow-shaped indicator. According to various embodiments, the object indicating the shared area may be a message indicating a relative position of the shared area with respect to the FOV. According to various embodiments, instead of operation830, at least one processor (e.g., processor220) of the augmented reality device (e.g., the augmented reality device200) may output, through the sound output device (e.g., the sound output device263), an audio indicating the relative position of the shared area with respect to the FOV.

In operation840, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify whether a movement (or change) of the FOV is detected. According to various embodiments, the at least one processor220may identify whether a movement of the FOV is detected, based on information obtained through at least one of the first camera211, the second camera212, or the six-axis sensor281.

If a movement of the FOV is not detected in operation840, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may repeat operations830and840until a movement of the FOV is detected. According to an embodiment, when a movement of the FOV is not detected within a designated time, the augmented reality device may stop displaying the object indicating the shared area.

When a movement of the FOV is detected in operation840, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may return to operation820.

When it is identified that the shared area is included in the FOV in operation820, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify whether the shared area and the personal area overlap in operation850. Details regarding operation540ofFIG.5may be equally applied to operation850.

When it is identified that the shared area and the personal area overlap in operation850, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may output the shared content in the shared area and personal content in an area that does not overlap the shared area, in operation860. For example, an area different from the private area may include an area other than the shared area within the FOV. Details regarding operation550ofFIG.5may be equally applied to operation860.

When it is identified that the shared area and the personal area do not overlap in operation850, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may output the shared content in the shared area and personal content in the personal area, in operation870. Details regarding operation560ofFIG.5may be equally applied to operation870.

As described above in connection withFIG.3, at least some of the operations ofFIG.8may be performed on an electronic device (e.g., the electronic device401) connected with the augmented reality device200. For example, outputting in operations860and870when operations860and870are performed by the electronic device401connected with the augmented reality device200may be replaced with transmitting display information for outputting content on the augmented reality device200through a communication circuit (e.g., the communication module490), rather than outputting content on the display (e.g., the display module460) of the electronic device401connected with the augmented reality device200. In this case, the augmented reality device200obtaining the display information from the electronic device401may output the content on the display (e.g., the display module250) based on the display information. When at least some of the operations ofFIG.8are performed on the electronic device (e.g., the electronic device401) connected with the augmented reality device200, the electronic device401may be aware of the display information output on the augmented reality device200and the space information and/or position information about the augmented reality device200. For example, the electronic device401may obtain space information and/or position information about the augmented reality device200from the augmented reality device200, identify the FOV based on the space information and/or position information about the augmented reality device200, perform at least one of operations820,840, or850periodically or when the space information and/or position information is changed, and transmit, to the augmented reality device200, display information to allow the augmented reality device200to output the shared content and the personal content.

FIG.9Ais a flowchart illustrating operations performed in an augmented reality device according to various embodiments.

In operation910a, at least one processor (e.g., processor220) of the augmented reality device (e.g., the augmented reality device200) may identify at least one of the position or direction of a predefined external object. According to various embodiments, the predefined external object may be at least one of a portion of a human body, such as a human head, an animal, or a thing, such as a bag.

According to various embodiments, the at least one processor220may analyze an image obtained through the first camera (e.g., the first camera211), thereby identifying the position of the predefined external object included in the image. In this case, the memory (e.g., the memory240) of the augmented reality device200may store data regarding the predefined external object and commands for identifying the position of the external object from the image.

According to various embodiments, the at least one processor220may transmit the image obtained through the first camera211to the electronic device (e.g., the electronic device401) and may receive the position of the predefined external object included in the image from the electronic device401to thereby identify the position of the predefined external object. In this case, the memory (e.g., the memory430) of the electronic device401may store data regarding the predefined external object and commands for identifying the position of the external object from the image.

According to various embodiments, the at least one processor220may transmit the image obtained through the first camera211to the server (e.g., the server310) and receive the position of the predefined external object included in the image from the server310to thereby identify the position of the predefined external object. In this case, the server310may store data regarding the predefined external object and commands for identifying the position of the external object from the image.

According to various embodiments, the at least one processor220may analyze an image obtained through the first camera (e.g., the first camera211), thereby identifying the direction of the predefined external object included in the image. In this case, the memory (e.g., the memory240) of the augmented reality device200may store data regarding the predefined external object and commands for identifying the direction of the external object from the image.

According to various embodiments, the at least one processor220may transmit the image obtained through the first camera211to the electronic device (e.g., the electronic device401) and may receive the direction of the predefined external object included in the image from the electronic device401to thereby identify the direction of the predefined external object. In this case, the memory (e.g., the memory430) of the electronic device401may store data regarding the predefined external object and commands for identifying the direction of the external object from the image.

According to various embodiments, the at least one processor220may transmit the image obtained through the first camera211to the server (e.g., the server310) and receive the direction of the predefined external object included in the image from the server310to thereby identify the direction of the predefined external object. In this case, the server310may store data regarding the predefined external object and commands for identifying the direction of the external object from the image.

In operation920a, at least one processor (e.g., processor220) of the augmented reality device (e.g., the augmented reality device200) may output personal content in the position identified based on at least one of the position or direction of the prespecified external object.

According to various embodiments, when the position of the predefined external object overlaps the personal area, and the direction of the predefined external object is a direction in which it may meet the gaze of the user of the augmented reality device (e.g., the augmented reality device200), the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may output the personal content in an area not overlapping the position of the external object, different from the existing personal area, thereby allowing the user of the augmented reality device (e.g., the augmented reality device200) to avoid eye contact with the person corresponding to the external object.

According to various embodiments, when the position of the predefined external object overlaps the personal area but the direction of the predefined external object is a direction in which it may not meet the gaze (e.g., FOV) of the user of the augmented reality device (e.g., the augmented reality device200), the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may output the personal content in the existing personal area. For example, when the predefined external object is a human head, and the back of the human head faces the augmented reality device (e.g., the augmented reality device200), the augmented reality device (e.g., the augmented reality device200) may output the personal content in the existing personal area.

As described above in connection withFIG.3, at least some of the operations ofFIG.9Amay be performed on an electronic device (e.g., the electronic device401) connected with the augmented reality device200. In this case, before performing operation910a, the electronic device401may receive, from the augmented reality device200, an image obtained through the first camera211of the augmented reality device200and, in operation910a, perform image analysis by referring to the memory (e.g., the memory430) of the electronic device401or transmit the image through the communication circuit (e.g., the communication module490) to the server (e.g., the server310) and receive a result of image analysis from the server310having performed the image analysis. Further, displaying content when operation920ais performed by the electronic device401may be replaced with transmitting display information for displaying content on the augmented reality device200through a communication circuit (e.g., the communication circuit270), rather than displaying content on the display (e.g., the display module460) of the electronic device401connected with the augmented reality device200. For example, the electronic device401may generate display information generated to allow the augmented reality device200to display personal content in a position not overlapping the predefined external object and transmit it to the augmented reality device200. In this case, the augmented reality device200receiving the display information from the electronic device401may display the content on the display (e.g., the display module250) based on the display information.

FIG.9Billustrates an example in which personal content is displayed on an augmented reality device, according to various embodiments. More specifically,FIG.9Billustrates an example of a change in position of personal content when the operations ofFIG.9Aare performed.

Referring toFIG.9B, personal areas910band940band a shared area920bmay be defined in the FOV. A human face950b, which is one of types predefined for external objects, may overlap the personal area940b, and the direction in which the human face950bfaces may be a direction in which it may meet the gaze of the user902bof the augmented reality device901b. In this case, at least one processor (e.g., the processor220) of the augmented reality device901b(e.g., the augmented reality device200) may display personal content in the position930bnot overlapping the human face950baccording to operation920a.

According to an embodiment, personal content configured in a world lock scheme may be displayed in the personal area940b, and the personal content configured in the world lock scheme may overlap an external object. In this case, the augmented reality device may change the attribute of the personal content configured in the world lock scheme into a body lock scheme and display content in a position930bnot overlapping the external object.

FIG.10is a flowchart illustrating operations performed in an augmented reality device according to various embodiments.

In operation1010, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify the position of a predefined external object and the FOV of the external object. According to various embodiments, the predefined external object may be an object that may have a gaze, such as a human head or a camera.

The example described above in connection with operation910aofFIG.9Amay be applied to the method for the at least one processor220to grasp the position of the external object in operation1010.

According to various embodiments, the at least one processor220may analyze an image obtained through the first camera (e.g., the first camera211), thereby identifying the FOV of the predefined external object included in the image. In this case, the memory (e.g., the memory240) of the augmented reality device200may store data regarding the predefined external object and commands for identifying the FOV of the external object from the image.

According to various embodiments, the at least one processor220may transmit the image obtained through the first camera211to the electronic device (e.g., the electronic device401) and may receive the FOV of the predefined external object included in the image from the electronic device401to thereby identify the FOV of the predefined external object. In this case, the memory (e.g., the memory430) of the electronic device401may store data regarding the predefined external object and commands for identifying the FOV of the external object from the image.

According to various embodiments the at least one processor220may transmit the image obtained through the first camera211to the server (e.g., the server310) and receive the FOV of the predefined external object included in the image from the server310to thereby identify the FOV of the predefined external object. In this case, the server310may store data regarding the predefined external object and commands for identifying the FOV of the external object from the image.

In operation1020, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify whether the predefined external object overlaps the personal area based on the position of the predefined external object. Here, that the predefined external object overlaps the personal area may mean that at least a portion of the personal area overlaps the external object.

When it is identified that the predefined external object does not overlap the personal area in operation1020, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may display personal content in the personal area in operation1040.

When it is identified that the predefined external object overlaps the personal area in operation1020, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify whether the augmented reality device200is positioned in the FOV of the external object in operation1030.

When it is identified that the augmented reality device200is positioned in the FOV of the external object in operation1030, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may display personal content in a position not overlapping the predefined external object, different from the personal area, in operation1035.

When it is identified that the augmented reality device200is not positioned in the FOV of the external object in operation1030, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may perform operation1040.

As described above in connection withFIG.3, at least some of the operations ofFIG.10may be performed on an electronic device (e.g., the electronic device401) connected with the augmented reality device200. In this case, before performing operation1010, the electronic device401may receive, from the augmented reality device200, an image obtained through the first camera211of the augmented reality device200and, in operation1010, perform image analysis by referring to the memory (e.g., the memory430) of the electronic device401or transmit the image through the communication circuit (e.g., the communication module490) to the server (e.g., the server310) and receive a result of image analysis from the server310having performed the image analysis. Further, displaying content when operations1035and1040are performed by the electronic device401may be replaced with transmitting display information for displaying content on the augmented reality device200through a communication circuit (e.g., the communication circuit270), rather than displaying content on the display (e.g., the display module460) of the electronic device401connected with the augmented reality device200. For example, when performing operation1035, the electronic device401may generate display information generated to allow the augmented reality device200to display personal content in a position not overlapping the predefined external object and transmit it to the augmented reality device200. In this case, the augmented reality device200receiving the display information from the electronic device401may display the content on the display (e.g., the display module250) based on the display information.

According to an embodiment, the personal content configured in the world lock scheme may overlap the external object, and the augmented reality device may be positioned within the FOV of the external object. In this case, the augmented reality device may change the attribute of the personal content configured in the world lock scheme into a body lock scheme and display content in a position not overlapping the external object.

FIG.11is a flowchart illustrating operations performed in an augmented reality device according to various embodiments.

In operation1110, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify the position of a predefined external object. Details regarding operation910aofFIG.9Amay be equally applied to operation1110.

In operation1120, at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify whether the predefined external object overlaps the shared area. Details regarding operation920aofFIG.9Amay be equally applied to operation1120.

When it is identified that the predefined external object overlaps the shared area in operation1120, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may display a message instructing the user to move in operation1130. According to various embodiments, the message may be a message urging the user of the augmented reality device200to move the external object to a position or direction where it does not overlap the shared area.

When it is identified that the predefined external object does not overlap the shared area in operation1120, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may display shared content in the shared area in operation1140.

As described above in connection withFIG.3, at least some of the operations ofFIG.11may be performed on an electronic device (e.g., the electronic device401) connected with the augmented reality device200. In this case, before performing operation1110, the electronic device401may receive, from the augmented reality device200, an image obtained through the first camera211of the augmented reality device200and, in operation1110, perform image analysis by referring to the memory (e.g., the memory430) of the electronic device401or transmit the image through the communication circuit (e.g., the communication module490) to the server (e.g., the server310) and receive a result of image analysis from the server310having performed the image analysis. Further, displaying content when operations1130and1140are performed by the electronic device401may be replaced with transmitting display information for displaying content on the augmented reality device200through a communication circuit (e.g., the communication circuit270), rather than displaying content on the display (e.g., the display module460) of the electronic device401connected with the augmented reality device200. In this case, the augmented reality device200receiving the display information from the electronic device401may display the content on the display (e.g., the display module250) based on the display information.

FIG.12Aillustrates operations of augmented reality devices and a server according to various embodiments.

In operation1210a, at least one processor (e.g., the processor220) of a first augmented reality device1201amay identify an input to move personal content to the shared area. For example, the first augmented reality device1201amay obtain information including Cartesian coordinates for a specified shared area from a server1202aor may be outputting, in a specified shared area, content shared through the server1202a. According to various embodiments, the input to move the personal content to the shared area may include an input for selecting personal content and an input to move the position of the personal content over the shared area. According to various embodiments, as described above with reference to operation510ofFIG.5, the user input may be at least one of an input through a controller, an input identified through eye tracking, or a gesture input.

In operation1220a, the at least one processor (e.g., the processor220) of the first augmented reality device1201amay transmit personal content and a share request to the server1202a. For example, the personal content may include virtual content being displayed on the first augmented reality device1201aor data stored in the first augmented reality device1201a.

In operation1230a, the server1202amay identify whether the user of the first augmented reality device1201ais a user having the authority to share content. According to various embodiments, the server1202amay store a list of augmented reality devices granted the authority to control the shared area and identify whether the first augmented reality device1201ais included in the stored list to thereby identify whether the user of the first augmented reality device1201ais a user having the authority to share content.

When it is identified that the user of the first augmented reality device1201ais not a user having the authority to share in operation1230a, the server1202amay notify the first augmented reality device1201athat it has no authority to share in operation1240a. In this case, in response to reception of the notification of having no authority, the first augmented reality device1201amay transmit a request for the authority to share to another augmented reality device or the server1202a. When the server1202areceives the request for the authority to share, the server1202amay transfer the request to another augmented reality device having the authority to share and, based on a response from the other augmented reality device, may grant or may not grant the authority to share to the augmented reality device1201a.

When it is identified that the user of the first augmented reality device1201ahas the authority to share in operation1230a, the server1202amay identify the shared area in operation1250a. According to various embodiments, before performing operation1250a, the server1202amay receive information about the shared area from the first augmented reality device1201aor the other augmented reality device granted the authority to control the shared area.

In operation1260a, the server1202amay transmit, to a second augmented reality device1203a, the personal content received in operation1220aand the position of the shared area identified in operation1250a. According to an embodiment, the second augmented reality device1203amay be a device sharing the content, displayed in a designated shared area through the server1202a, with the first augmented reality device1201a. For example, the second augmented reality device1203amay be determined by the first augmented reality device1201abefore transmitting the personal content and share request to the server through operation1220a, designated as a sharing target by the first augmented reality device1201awhen the first augmented reality device1201aperforms operation1220a, or determined by the server1202abefore the server1202aperforms operation1260a.

In operation1270a, the at least one processor (e.g., the processor220) of the second augmented reality device1203a(e.g., the augmented reality device200) may output the content obtained from the server1202ain the shared area based on the position of the shared area obtained from the server1202a.

In operation1280a, the server1202amay notify the first augmented reality device1201athat content sharing has been permitted. According to various embodiments, operation1280amay be omitted.

In operation1290a, the at least one processor (e.g., the processor220) of the first augmented reality device1201a(e.g., the augmented reality device200) may display specific personal content in the shared area in operation1210a. For example, when the first augmented reality device1201amoves personal content, configured in the body lock scheme or personal world lock scheme, to the shared area and displays it in the shared area, the scheme of displaying the personal content may be changed from the body lock scheme or personal world lock scheme to the shared world lock scheme.

As described above in connection withFIG.3, at least some of the operations ofFIGS.12A and12Bmay be performed on an electronic device (e.g., the electronic device401) connected with the augmented reality device200(e.g., the first augmented reality device1201a). In this case, for the electronic device401to perform operation1210amay include for the electronic device401to receive a signal indicating that there was a user's input through the augmented reality device200from the augmented reality device200and to identify that the electronic device401is used as a controller of the augmented reality device200and identify that the input to allow the electronic device401to move the personal content to the shared area has been made through the electronic device401.

Further, displaying content when operations1270aand1290aare performed by the electronic device401may be replaced with transmitting display information for displaying content on the augmented reality device200through a communication circuit (e.g., the communication circuit270), rather than displaying content on the display (e.g., the display module460) of the electronic device401connected with the augmented reality device200. In this case, the augmented reality device200receiving the display information from the electronic device401may display the content on the display (e.g., the display module250) based on the display information.

FIG.12Billustrates operations of an augmented reality device and a server according to various embodiments.

In operation1210b, at least one processor (e.g., the processor220) of an augmented reality device1201bmay receive, from a server1202b, shared content and the position of a shared area for displaying the shared content. The shared content may mean content provided to the server1202bso as to be provided from a device other than the augmented reality device1201bto the augmented reality device1201b.

In operation1220b, the at least one processor (e.g., the processor220) of the augmented reality device1201bmay display the shared content received in operation1210b, in the form of a virtual object, in the shared area, in the world lock scheme.

In operation1230b, the at least one processor (e.g., the processor220) of the augmented reality device1201bmay identify an input to move the shared content displayed in the shared area from the shared area to the personal area. According to various embodiments, the personal area may be an area not overlapping the shared area. According to various embodiments, as described above with reference to operation510ofFIG.5, the user input may be at least one of an input through a controller, an input identified through eye tracking, or a gesture input.

In operation1240b, the at least one processor (e.g., the processor220) of the augmented reality device1201bmay transmit a request for downloading shared content to the server1202b.

In operation1250b, the at least one processor (e.g., the processor220) of the augmented reality device1201bmay download shared content from the server1202b. According to an embodiment, before permitting the augmented reality device1201bto download the shared content, the server1202bmay identify whether the augmented reality device1201bhas the authority to download and, only when the augmented reality device1201bhas the authority to download, permit the augmented reality device1201bto download the shared content.

In operation1260b, the at least one processor (e.g., the processor220) of the augmented reality device1201bmay display the shared content, downloaded from the server1202b, in the personal area in the body lock scheme or personal world lock scheme.

FIG.13illustrates an example of a screen displayed on an electronic device interacting with an augmented reality device according to various embodiments. For example,FIG.13illustrates a screen displayed for an electronic device (e.g., the electronic device320ofFIG.3) interacting with an augmented reality device (e.g., the augmented reality device330ofFIG.3) to identify a user input to specify a shared area as in operation510ofFIG.5.

Referring toFIG.13, an image1310indicating a meeting place may be displayed in a screen1300, and a plurality of candidate areas1321and1322recommended as shared areas in the meeting place may be overlaid and displayed on an image1310. For example, a candidate area1321may be a front wall area, and a candidate area1322may be a middle area of the meeting room desk. According to various embodiments, if one candidate area is selected from among the plurality of candidate areas1321and1322, the selected candidate area may be displayed visually in a different manner from the other candidate areas and, if ‘choose’1330is selected in that state, the candidate area, which is displayed visually in a different manner from the other candidate areas, may be finally selected as a shared area. According to various embodiments, an interface indicating ‘learn areas’1340may be displayed on the screen1300. For example, when ‘learn areas’1340is selected, the electronic device320may capture the ambient environment through a camera (e.g., the camera module480ofFIG.4) of the electronic device320or a camera (e.g., the first camera211or third camera213ofFIG.2) of the augmented reality device330. The electronic device320or the server (e.g., the server310ofFIG.3) may generate candidate areas based on information about the captured ambient environment. According to various embodiments, an interface for starting ‘choose shared content’1350may be displayed on the screen1300. For example, when ‘choose shared content’1350is performed on the electronic device320, and at least one content stored in the electronic device320is shared, the content selected to be shared may be transmitted to the server310, and the server310may generate virtual content corresponding to the selected content and transmit the generated virtual content, along with the information about the shared area selected on the electronic device320, to the sharing target (e.g., the augmented reality device340ofFIG.3).

FIG.13exemplifies a screen displayed on the electronic device320interacting with the augmented reality device330. However, according to various embodiments, similar to the screen1300shown inFIG.13, a plurality of candidate areas recommended as shared areas in the meeting place may be overlaid and displayed also on the augmented reality device340, and at least one area may be finally selected as a shared area from among the plurality of candidate areas based on the user's selection.

FIG.14is a flowchart illustrating operations performed in an augmented reality device according to various embodiments.

In operation1410, at least one processor (e.g., the processor220) of an augmented reality device (e.g., the augmented reality device200) may identify the position of a shared area and shared content. According to various embodiments, the position of the shared area may be a position specified by a user input to the augmented reality device200or a position obtained from the server (e.g., the server310) by the augmented reality device200. According to various embodiments, the shared content may be content shared by the augmented reality device200or may be content shared by an electronic device (e.g., an augmented reality device) other than the augmented reality device200and obtained from the server (e.g., the server310) by the augmented reality device200.

In operation1420, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may identify the distance between the augmented reality device200and the shared area in the Cartesian coordinate system. According to various embodiments, the at least one processor220may identify the distance, in the Cartesian coordinate system, between the augmented reality device200and the shared area based on position information about the augmented reality device200obtained through at least one sensor and the position of the shared area identified in operation1410.

In operation1430, the at least one processor (e.g., the processor220) of the augmented reality device (e.g., the augmented reality device200) may display the shared content in a size different from the size of the shared area, based on the distance identified in operation1420. According to various embodiments, when the identified distance is a predetermined first distance or less, the at least one processor220may shrink and display the shared content in a smaller size than the size of the shared area. According to various embodiments, when the identified distance is a predetermined second distance or more, the at least one processor220may enlarge and display the shared content in a larger size than the size of the shared area. According to various embodiments, the at least one processor220may enlarge or shrink and display the shared content as compared with the shared area so that the size of the shared content recognized by the user of the augmented reality device200is constant regardless of the identified distance. According to an embodiment, the shared content may include information about the center point of the shared area and the spatial size of the shared content. For example, in displaying the shared content based on the spatial size of the shared content and the center point of the shared area of the shared content, the augmented reality device200may enlarge or shrink and display the shared content based on the distance between the user and the center point of the shared area. According to another embodiment, the shared content may include only the center point information about the shared area. For example, in displaying the shared content based on the center point of the shared area of the shared content, if the shared area is included in the user's FOV, the augmented reality device200may display the shared content, in a designated size, in a designated distance from the user.

According to various embodiments, an augmented reality device (e.g., the augmented reality device200) may comprise a display (e.g., the display250), at least one sensor (e.g., the sensor280), a communication circuit (e.g., the communication circuit270), and at least one processor (e.g., the processor220) operatively connected with the display250, the at least one sensor280, and the communication circuit270. The at least one processor220may be configured to identify a first user input specifying a shared area for outputting shared content to be output on at least one other augmented reality device, transmit information about the shared area to a server through the communication circuit270, based on the first user input, identify whether the shared area and a personal area for outputting personal content overlap, based on information obtained through the at least one sensor280, control the display250to output the shared content in the shared area and the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and control the display250to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

According to various embodiments, the at least one processor220may be configured to identify that the shared area is not included in a field of view (FOV), based on the information obtained through the at least one sensor280and control the display250to display an object indicating the shared area, based on identifying that the shared area is not included in the FOV.

According to various embodiments, the shared area may have fixed Cartesian coordinates.

According to various embodiments, the personal area may have a position in a fixed FOV or fixed Cartesian coordinates.

According to various embodiments, the at least one processor220may be configured to identify that the personal area overlaps a prespecified external object, based on the information obtained through the at least one sensor280and control the display250to output the personal content in a position not overlapping the prespecified external object, different from the personal area, based on identifying that the personal area overlaps the prespecified external object.

According to various embodiments, the at least one processor220may be configured to identify that the shared area overlaps a prespecified external object, based on the information obtained through the at least one sensor280and control the display250to output a message instructing to move so that the shared area does not overlap the prespecified external object, based on identifying that the shared area overlaps the prespecified external object.

According to various embodiments, the at least one processor220may be configured to identify an input to move the personal content to the shared area, based on the information obtained through the at least one sensor280, transmit the personal content to the server based on identifying the input to move the personal content to the shared area, and control the display250to output the personal content in the shared area. The shared area may have fixed Cartesian coordinates.

According to various embodiments, the at least one processor220may be configured to control the display250to output a plurality of candidate areas of the shared area before identifying the first user input. The first user input may specify one of the plurality of candidate areas, as the shared area.

According to various embodiments, the at least one processor220may be configured to identify a distance, in a Cartesian coordinate system, between the augmented reality device200and the shared area, based on the information obtained through the at least one sensor280and control the display250to output the shared content in a size different from a size of the shared area, based on the distance.

According to various embodiments, an augmented reality device (e.g., the augmented reality device200) may comprise a display (e.g., the display250), at least one sensor (e.g., the sensor280), a communication circuit (e.g., the communication circuit270), and at least one processor (e.g., the processor220) operatively connected with the display250, the at least one sensor280, and the communication circuit270. The at least one processor220may be configured to obtain shared content and information indicating a position of a shared area for outputting the shared content, from a server through the communication circuit270, identify whether the shared area and a personal area for outputting personal content overlap, based on information obtained through the at least one sensor280, control the display250to output the shared content in the shared area and the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and control the display250to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

According to various embodiments, the at least one processor220may be configured to identify that the shared area is not included in a field of view (FOV), based on the information obtained through the at least one sensor280and control the display250to output an object indicating the shared area, based on identifying that the shared area is not included in the FOV.

According to various embodiments, the shared area may have fixed Cartesian coordinates.

According to various embodiments, the personal area may have a position in a fixed FOV or fixed Cartesian coordinates.

According to various embodiments, the at least one processor220may identify that the personal area overlaps a prespecified external object, based on information obtained through the at least one sensor280and control the display250to output the personal content in a position identified based on a position and/or direction of the prespecified external object, based on identifying that the personal area overlaps the prespecified external object.

According to various embodiments, the at least one processor220may be configured to identify that the shared area overlaps a prespecified external object, based on the information obtained through the at least one sensor280and control the display250to output a message instructing to move so that the shared area does not overlap the prespecified external object, based on identifying that the shared area overlaps the prespecified external object.

According to various embodiments, the at least one processor220may be configured to identify a distance, in a Cartesian coordinate system, between the augmented reality device200and the shared area, based on the information obtained through the at least one sensor280and control the display250to output the shared content in a size different from a size of the shared area, based on the distance.

According to various embodiments, an electronic device (e.g., the electronic device401) may comprise a communication circuit (e.g., the communication circuit490) and at least one processor (e.g., the processor420) operatively connected with the communication circuit. The at least one processor420may be configured to obtain shared content and information indicating a position of a shared area for outputting the shared content, from a server (e.g., the server408) through the communication circuit490, obtain information obtained through at least one sensor (e.g., the sensor280) of an augmented reality device, from the augmented reality device (e.g., the augmented reality device200) through the communication circuit490, identify whether the shared area and a personal area for outputting personal content overlap, based on the information obtained through the at least one sensor280of the augmented reality device200, transmit, to the augmented reality device200through the communication circuit490, display information to allow the augmented reality device200to output the shared content in the shared area and output the personal content in the personal area, based on identifying that the shared area and the personal area do not overlap, and transmit, to the augmented reality device200through the communication circuit490, display information to allow the augmented reality device200to output the shared content in the shared area and the personal content in an area not overlapping the shared area, based on identifying that the shared area and the personal area overlap.

According to various embodiments, the at least one processor420may be configured to identify a first user input specifying the shared area, and transmit information about the shared area to the server408through the communication circuit490, based on the first user input.

According to various embodiments, the at least one processor420may be configured to output a plurality of candidate areas of the shared area before identifying the first user input. The first user input may specify one of the plurality of candidate areas, as the shared area.

According to various embodiments, the shared area may have fixed Cartesian coordinates. The personal area may have a position in a fixed FOV or fixed Cartesian coordinates.